CrackedRuby CrackedRuby

Overview

GitOps defines a set of practices for managing infrastructure and applications where Git repositories serve as the source of truth for declarative system specifications. The methodology emerged from Weaveworks in 2017 as a formalization of deployment patterns observed in cloud-native environments, particularly those using Kubernetes.

The core concept centers on storing all configuration as code in version control, then using automated processes to synchronize the actual system state with the desired state declared in Git. This inverts traditional deployment models where operators push changes to systems. Instead, specialized controllers continuously pull from Git repositories and reconcile any differences between declared and actual state.

GitOps applies to both infrastructure provisioning and application deployment. A Git repository might contain Kubernetes manifests, Terraform configurations, or application deployment specifications. Changes flow through standard Git workflows: developers create branches, submit pull requests, conduct reviews, and merge to main branches. These Git operations trigger automated reconciliation rather than manual deployment steps.

The approach addresses several operational challenges. Configuration drift becomes immediately visible when actual state diverges from Git. Audit trails exist naturally in Git history. Rollbacks reduce to Git reverts. Disaster recovery simplifies to pointing new clusters at existing Git repositories.

# Example GitOps repository structure
├── apps/
│   ├── frontend/
│   │   ├── deployment.yaml
│   │   ├── service.yaml
│   │   └── ingress.yaml
│   └── backend/
│       ├── deployment.yaml
│       └── service.yaml
├── infrastructure/
│   ├── namespaces.yaml
│   ├── network-policies.yaml
│   └── rbac.yaml
└── clusters/
    ├── production/
    │   └── kustomization.yaml
    └── staging/
        └── kustomization.yaml

GitOps gained prominence alongside Kubernetes adoption but extends beyond container orchestration. Teams apply GitOps principles to database migrations, DNS records, cloud resources, and configuration management. The unifying factor remains Git-based declarative specifications with automated reconciliation.

Key Principles

GitOps rests on four foundational principles that distinguish it from related deployment methodologies.

Declarative Description: Systems must be described declaratively rather than through imperative scripts. Declarative specifications state the desired outcome without prescribing steps to achieve it. A Kubernetes Deployment manifest declares "run three replicas of this container" rather than instructing "start container A, then start container B, then start container C." This separation allows reconciliation controllers to determine necessary actions based on current state.

# Declarative: states desired outcome
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80

Declarative specifications enable idempotent operations. Applying the same configuration multiple times produces identical results. Controllers can safely retry operations without side effects. This property proves critical for automated systems that may reprocess configurations during failures or network interruptions.

Versioned and Immutable: Git provides both versioning and immutability guarantees. Every change receives a unique commit hash. History remains permanent and tamper-evident. Tags and branches provide human-readable references to specific system states. This versioning addresses change tracking, audit compliance, and rollback requirements simultaneously.

The immutability aspect extends beyond Git commits. Container images should reference specific tags or digests rather than mutable tags like "latest." Infrastructure code should pin dependency versions. This ensures reproducible deployments where the same Git commit always produces identical system state.

Automatic Pull: Automated agents pull desired state from Git and reconcile actual state continuously. This contrasts with push-based CD where external systems push changes to environments. Pull-based models keep credentials and cluster access within the deployment environment rather than exposing them to external CI systems.

# Conceptual reconciliation loop
loop do
  desired_state = fetch_from_git(repository_url, branch)
  actual_state = query_cluster_state
  
  differences = calculate_diff(desired_state, actual_state)
  
  differences.each do |resource|
    case resource.action
    when :create
      apply_resource(resource)
    when :update
      update_resource(resource)
    when :delete
      delete_resource(resource)
    end
  end
  
  sleep(reconciliation_interval)
end

The pull model also improves security posture. External systems never receive cluster credentials. Authentication flows reverse: clusters authenticate to Git repositories rather than CI systems authenticating to clusters. This reduces credential exposure and simplifies secret rotation.

Continuous Reconciliation: Controllers continuously compare desired state in Git against actual state in systems. Reconciliation occurs on configurable intervals, typically every few minutes. This detects configuration drift from any source: manual changes, external automation, or failed synchronization attempts.

Reconciliation loops implement eventual consistency. Systems converge toward desired state over time despite transient failures. If a node fails during deployment, the next reconciliation cycle recreates missing pods. If someone manually deletes a resource, reconciliation recreates it from the Git specification.

The reconciliation model shifts operational focus from "was the deployment successful" to "does current state match desired state." Success becomes a continuous property rather than a point-in-time event. Monitoring evaluates whether reconciliation maintains convergence rather than whether individual deployments completed.

Implementation Approaches

Organizations implement GitOps through several architectural patterns, each with distinct trade-offs around complexity, flexibility, and operational requirements.

Monorepo Pattern: A single Git repository contains all configuration for all environments and applications. This simplifies discovery and cross-cutting changes. Updating a shared library or base configuration requires modifications in one location. Atomic commits can coordinate changes across multiple applications.

gitops-repo/
├── base/
│   ├── networking/
│   ├── storage/
│   └── monitoring/
├── apps/
│   ├── app-a/
│   ├── app-b/
│   └── app-c/
└── environments/
    ├── dev/
    ├── staging/
    └── production/

The monorepo pattern introduces scaling challenges. Large repositories with many contributors face merge conflicts and slow Git operations. Permissions become coarse-grained: users typically receive access to the entire repository. Build and validation times increase as configuration grows. Organizations using monorepos often implement path-based permissions and selective reconciliation to mitigate these issues.

Polyrepo Pattern: Each application or service maintains its own repository. This aligns with microservices architectures where teams own independent services. Repositories scale independently. Teams manage their own release cadences without coordinating with other teams. Permission boundaries match organizational boundaries naturally.

org/
├── app-a-config/
├── app-b-config/
├── app-c-config/
└── platform-infrastructure/

Polyrepos complicate cross-cutting changes. Updating a shared standard requires changes across multiple repositories. Discovering all applications in an environment requires querying multiple repositories. Organizations address this through standardization tools, repository templates, and aggregation services that catalog all GitOps repositories.

Environment-per-Repository: Repositories correspond to environments rather than applications. The production repository contains all production configuration. The staging repository contains all staging configuration. This pattern simplifies promotion workflows: promoting to production means copying files from the staging repository to the production repository.

├── production-gitops/
├── staging-gitops/
└── development-gitops/

This approach centralizes environment management but complicates application-specific changes. Developers working on a single application must modify multiple repositories to deploy across environments. The pattern works well for platform teams managing shared infrastructure but less effectively for application teams deploying frequently.

Hub and Spoke: A central platform repository manages shared infrastructure and configuration. Individual application repositories contain application-specific configuration. Reconciliation controllers in each environment pull from both the platform repository and relevant application repositories. This separates platform concerns from application concerns while maintaining centralized control where appropriate.

The hub and spoke model requires coordination between platform and application teams. Changes to platform configuration may impact applications. Applications must conform to platform standards. Organizations implementing this pattern typically establish clear interfaces between platform and application layers, treating the platform repository as an API that applications consume.

Tools & Ecosystem

GitOps implementations depend on specialized controllers that automate Git synchronization and state reconciliation. Multiple tools provide these capabilities with varying feature sets and architectural assumptions.

ArgoCD: A declarative GitOps controller for Kubernetes that provides both CLI and web UI interfaces. ArgoCD monitors Git repositories and synchronizes changes to clusters continuously or on-demand. The tool supports multiple configuration formats including raw Kubernetes YAML, Helm charts, Kustomize overlays, and custom configuration management tools.

# Ruby script to interact with ArgoCD API
require 'net/http'
require 'json'
require 'uri'

class ArgoCDClient
  def initialize(server_url, auth_token)
    @server_url = server_url
    @auth_token = auth_token
  end
  
  def sync_application(app_name)
    uri = URI("#{@server_url}/api/v1/applications/#{app_name}/sync")
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    
    request = Net::HTTP::Post.new(uri)
    request['Authorization'] = "Bearer #{@auth_token}"
    request['Content-Type'] = 'application/json'
    request.body = { prune: true, dryRun: false }.to_json
    
    response = http.request(request)
    JSON.parse(response.body)
  end
  
  def get_application_status(app_name)
    uri = URI("#{@server_url}/api/v1/applications/#{app_name}")
    http = Net::HTTP.new(uri.host, uri.port)
    http.use_ssl = true
    
    request = Net::HTTP::Get.new(uri)
    request['Authorization'] = "Bearer #{@auth_token}"
    
    response = http.request(request)
    app_data = JSON.parse(response.body)
    
    {
      health: app_data['status']['health']['status'],
      sync: app_data['status']['sync']['status'],
      revision: app_data['status']['sync']['revision']
    }
  end
end

# Usage
client = ArgoCDClient.new(
  'https://argocd.example.com',
  ENV['ARGOCD_TOKEN']
)

status = client.get_application_status('my-app')
puts "Health: #{status[:health]}, Sync: #{status[:sync]}"

if status[:sync] != 'Synced'
  puts "Triggering synchronization..."
  result = client.sync_application('my-app')
  puts "Sync initiated: #{result['metadata']['name']}"
end

ArgoCD implements application-level abstractions. Each application resource defines a Git repository, target revision, and destination cluster. The controller tracks synchronization state and health status. Automated sync policies can enable hands-off deployments or require manual approval for production changes.

The tool provides diff visualization showing differences between Git state and cluster state before synchronization. This preview capability helps operators understand change impacts. ArgoCD also implements progressive delivery patterns through integration with Argo Rollouts, enabling blue-green and canary deployment strategies.

Flux: A GitOps toolkit providing modular controllers for different reconciliation tasks. Flux decomposes GitOps into specialized components: source-controller manages Git repositories, kustomize-controller applies Kustomize configurations, helm-controller manages Helm releases, and notification-controller sends alerts.

# Flux GitRepository source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: app-source
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/app-config
  ref:
    branch: main
  secretRef:
    name: git-credentials
---
# Flux Kustomization
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: app-deployment
  namespace: flux-system
spec:
  interval: 5m
  sourceRef:
    kind: GitRepository
    name: app-source
  path: ./clusters/production
  prune: true
  wait: true
  timeout: 2m

This modular architecture allows selective deployment. Organizations can install only needed controllers. The source-controller supports Git, Helm repositories, and S3 buckets as sources. Custom controllers can extend the toolkit for organization-specific requirements.

Flux emphasizes GitOps for the GitOps controllers themselves. Flux components are installed and configured through Git-managed manifests. Upgrading Flux involves committing new manifests to Git. This bootstrapping approach ensures that all cluster state, including the GitOps system itself, exists in Git.

Jenkins X: A CI/CD platform built around GitOps principles for Kubernetes. Jenkins X automates the creation of Git repositories, CI pipelines, and GitOps promotion workflows. The platform generates Tekton pipelines that build applications and commit generated Kubernetes manifests to GitOps repositories.

# Ruby script to trigger Jenkins X promotion
require 'octokit'

class JenkinsXPromotion
  def initialize(github_token, org, repo)
    @client = Octokit::Client.new(access_token: github_token)
    @org = org
    @repo = repo
  end
  
  def promote_to_staging(app_name, version)
    # Jenkins X uses pull requests for promotions
    branch_name = "promote-#{app_name}-#{version}-staging"
    
    # Create branch
    main_ref = @client.ref(@repo_full_name, 'heads/main')
    @client.create_ref(@repo_full_name, "heads/#{branch_name}", main_ref.object.sha)
    
    # Update version in staging overlay
    content = @client.contents(
      @repo_full_name,
      path: "env/staging/#{app_name}/values.yaml",
      ref: branch_name
    )
    
    updated_content = content.content.gsub(/version: .*/, "version: #{version}")
    
    @client.update_contents(
      @repo_full_name,
      "env/staging/#{app_name}/values.yaml",
      "Promote #{app_name} to version #{version}",
      content.sha,
      updated_content,
      branch: branch_name
    )
    
    # Create pull request
    @client.create_pull_request(
      @repo_full_name,
      'main',
      branch_name,
      "Promote #{app_name} #{version} to staging",
      "Automated promotion of #{app_name} to version #{version}"
    )
  end
  
  private
  
  def repo_full_name
    "#{@org}/#{@repo}"
  end
end

# Usage
promoter = JenkinsXPromotion.new(
  ENV['GITHUB_TOKEN'],
  'my-org',
  'environment-gitops'
)

promoter.promote_to_staging('payment-service', 'v2.3.1')

Jenkins X integrates preview environments for pull requests. When developers create a pull request, Jenkins X automatically deploys a temporary environment with the proposed changes. This enables testing in realistic environments before merging. Once merged, Jenkins X automatically promotes changes through environments based on configured promotion strategies.

Config Sync: Google Cloud's GitOps operator for GKE and Anthos clusters. Config Sync enforces policies across multiple clusters from a single Git repository. The tool supports hierarchical configuration where cluster-scoped, namespace-scoped, and cluster-selector-scoped policies combine to produce final configuration.

Config Sync implements policy inheritance. Configurations in a root directory apply to all clusters. Configurations in namespace directories apply to specific namespaces. Configurations in cluster directories apply to clusters matching selectors. This hierarchy reduces duplication for organizations managing many clusters with common policies.

Common Patterns

GitOps implementations converge on several recurring patterns that address operational challenges and establish standardized workflows.

Environment Promotion: Applications progress through environments in a controlled sequence. Changes merge to a development branch, deploy to development environments, then promote to staging, and finally to production. Each promotion creates explicit Git operations that record approval and timing.

# Automated promotion script
class EnvironmentPromoter
  def initialize(git_repo_path)
    @repo_path = git_repo_path
    @git = Git.open(@repo_path)
  end
  
  def promote(app_name, from_env, to_env)
    from_file = "environments/#{from_env}/#{app_name}/values.yaml"
    to_file = "environments/#{to_env}/#{app_name}/values.yaml"
    
    # Extract current version from source environment
    from_content = File.read(File.join(@repo_path, from_file))
    version = from_content[/version: (.+)/, 1]
    
    # Update target environment
    to_content = File.read(File.join(@repo_path, to_file))
    updated_content = to_content.gsub(/version: .+/, "version: #{version}")
    
    File.write(File.join(@repo_path, to_file), updated_content)
    
    # Commit and push
    @git.add(to_file)
    @git.commit("Promote #{app_name} #{version} from #{from_env} to #{to_env}")
    @git.push('origin', 'main')
    
    { app: app_name, version: version, from: from_env, to: to_env }
  end
  
  def verify_promotion(app_name, environment, expected_version)
    file = "environments/#{environment}/#{app_name}/values.yaml"
    content = File.read(File.join(@repo_path, file))
    actual_version = content[/version: (.+)/, 1]
    
    actual_version == expected_version
  end
end

# Usage
promoter = EnvironmentPromoter.new('/path/to/gitops-repo')

result = promoter.promote('api-service', 'staging', 'production')
puts "Promoted #{result[:app]} #{result[:version]} to #{result[:to]}"

# Verify promotion
if promoter.verify_promotion('api-service', 'production', result[:version])
  puts "Promotion verified in Git"
else
  puts "Promotion verification failed"
end

Promotion strategies vary by risk tolerance. Conservative approaches require manual pull request approval for production changes. Progressive approaches automate promotion after passing integration tests in lower environments. Some organizations implement time-based promotions where changes automatically promote after a soak period in staging.

Multi-Cluster Management: Organizations managing multiple Kubernetes clusters face challenges distributing configuration consistently. GitOps addresses this through cluster-specific overlays or through multi-cluster capable controllers.

# Kustomize overlay structure for multi-cluster
base/
├── app-deployment.yaml
└── app-service.yaml

overlays/
├── us-east-cluster/
│   └── kustomization.yaml
├── us-west-cluster/
│   └── kustomization.yaml
└── eu-cluster/
    └── kustomization.yaml

# us-east-cluster/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

patchesStrategicMerge:
  - replica-count.yaml
  - resource-limits.yaml

configMapGenerator:
  - name: region-config
    literals:
      - region=us-east
      - endpoint=api-use1.example.com

Multi-cluster patterns often implement geographic distribution for latency optimization or disaster recovery. Configuration bases define common application structure. Overlays customize settings per cluster such as replica counts, resource limits, and region-specific endpoints. Controllers in each cluster pull appropriate overlay configurations.

Secret Management Integration: GitOps repositories should never contain plaintext secrets. Several patterns integrate secret management with GitOps workflows while maintaining Git as the source of truth.

Sealed Secrets encrypts secrets with cluster-specific keys. Developers encrypt secrets locally and commit encrypted versions to Git. Controllers in clusters decrypt secrets during reconciliation. This maintains Git audit trails while preventing secret exposure.

# Ruby script to create sealed secrets
require 'json'
require 'base64'
require 'openssl'

class SealedSecretGenerator
  def initialize(cert_path)
    @public_key = OpenSSL::X509::Certificate.new(
      File.read(cert_path)
    ).public_key
  end
  
  def seal(secret_data, namespace, name)
    encrypted_data = {}
    
    secret_data.each do |key, value|
      # Encrypt each secret value
      encrypted = @public_key.public_encrypt(
        value,
        OpenSSL::PKey::RSA::PKCS1_OAEP_PADDING
      )
      encrypted_data[key] = Base64.strict_encode64(encrypted)
    end
    
    {
      apiVersion: 'bitnami.com/v1alpha1',
      kind: 'SealedSecret',
      metadata: {
        name: name,
        namespace: namespace
      },
      spec: {
        encryptedData: encrypted_data
      }
    }
  end
  
  def write_sealed_secret(sealed_secret, output_path)
    File.write(output_path, sealed_secret.to_yaml)
  end
end

# Usage
generator = SealedSecretGenerator.new('sealed-secrets-cert.pem')

sealed = generator.seal(
  {
    'database-password' => 'super-secret-password',
    'api-key' => 'sk-1234567890abcdef'
  },
  'production',
  'app-secrets'
)

generator.write_sealed_secret(
  sealed,
  'apps/production/sealed-secret.yaml'
)

External secret operators retrieve secrets from external systems like HashiCorp Vault or AWS Secrets Manager. Git repositories contain references to secrets rather than secret values. Controllers synchronize references with actual secrets at runtime. This separates secret storage from configuration management while maintaining GitOps workflows.

Progressive Delivery: GitOps integrates with progressive delivery techniques that gradually roll out changes while monitoring metrics. This combines declarative configuration with controlled release strategies.

# Argo Rollouts canary deployment
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-service
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 30
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 100
      analysis:
        templates:
        - templateName: success-rate
        startingStep: 2
      trafficRouting:
        istio:
          virtualService:
            name: api-service
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: api-service:v2.0.0

Progressive delivery patterns define deployment steps as Git-managed configuration. Rollout strategies specify traffic weighting, pause durations, and analysis criteria. Automated analysis queries metrics systems and promotes or rolls back based on configured thresholds. This codifies deployment processes while maintaining human oversight through Git approval workflows.

Security Implications

GitOps introduces specific security considerations that span Git access control, credential management, and system integrity verification.

Git Repository Security: Git repositories become high-value targets since they control production infrastructure. Repository access must implement principle of least privilege. Developers who deploy applications require write access to application configuration paths but not infrastructure paths. Platform teams require write access to shared infrastructure but not necessarily all applications.

# Script to audit Git repository permissions
require 'octokit'

class GitRepoAuditor
  def initialize(github_token, org)
    @client = Octokit::Client.new(access_token: github_token)
    @org = org
  end
  
  def audit_repository(repo_name)
    repo_full_name = "#{@org}/#{repo_name}"
    
    # Check branch protection
    protection = @client.branch_protection(repo_full_name, 'main')
    
    audit_results = {
      repository: repo_name,
      branch_protection: {
        required_reviews: protection.required_pull_request_reviews&.required_approving_review_count || 0,
        enforce_admins: protection.enforce_admins.enabled,
        status_checks: protection.required_status_checks&.contexts || []
      },
      access_levels: {}
    }
    
    # Check collaborator access
    collaborators = @client.collaborators(repo_full_name)
    collaborators.each do |collab|
      permission = @client.permission_level(repo_full_name, collab.login)
      audit_results[:access_levels][collab.login] = permission.permission
    end
    
    # Check team access
    teams = @client.repository_teams(repo_full_name)
    teams.each do |team|
      audit_results[:access_levels]["team:#{team.name}"] = team.permission
    end
    
    audit_results
  end
  
  def find_security_issues(audit_results)
    issues = []
    
    bp = audit_results[:branch_protection]
    
    issues << "No required reviews" if bp[:required_reviews] < 2
    issues << "Admins can bypass protection" unless bp[:enforce_admins]
    issues << "No status checks required" if bp[:status_checks].empty?
    
    # Check for overly permissive access
    audit_results[:access_levels].each do |user, permission|
      if permission == 'admin' && !user.start_with?('team:')
        issues << "Individual user #{user} has admin access"
      end
    end
    
    issues
  end
end

# Usage
auditor = GitRepoAuditor.new(ENV['GITHUB_TOKEN'], 'my-org')
results = auditor.audit_repository('production-gitops')
issues = auditor.find_security_issues(results)

if issues.empty?
  puts "No security issues found"
else
  puts "Security issues detected:"
  issues.each { |issue| puts "  - #{issue}" }
end

Branch protection rules enforce review requirements before merging to protected branches. This creates approval workflows for production changes. Status checks can require successful builds, security scans, and policy validations before allowing merges. These controls embed governance into Git workflows rather than relying on external approval systems.

Commit signing provides cryptographic verification of change authorship. Organizations can require signed commits to prevent unauthorized modifications. This establishes a chain of custody for all configuration changes. GitOps controllers can verify signatures before applying configurations, rejecting unsigned or invalid commits.

Cluster Credential Management: GitOps controllers require credentials to access Git repositories and apply changes to clusters. These credentials must remain secure while enabling automated operations.

Pull-based GitOps models keep cluster credentials within clusters. Controllers authenticate outbound to Git repositories rather than receiving credentials from external systems. This reduces credential exposure. Git credentials can use deploy keys with read-only access, further limiting potential damage from credential compromise.

Service accounts for GitOps controllers should follow least privilege. Controllers need permissions to create, update, and delete resources they manage but not cluster-admin privileges. Namespace-scoped controllers reduce blast radius by limiting access to specific namespaces. Custom admission controllers can prevent controllers from modifying resources outside designated paths in Git repositories.

Supply Chain Security: GitOps reconciliation applies configurations from Git without verification of provenance by default. Organizations must implement additional controls to ensure configurations originate from trusted sources.

# Verify commit signatures before deployment
require 'git'
require 'gpgme'

class CommitVerifier
  def initialize(repo_path, trusted_keys)
    @repo = Git.open(repo_path)
    @trusted_keys = trusted_keys
  end
  
  def verify_commit(commit_sha)
    commit = @repo.object(commit_sha)
    
    # Check if commit is signed
    unless commit.gpg_signature
      return { verified: false, reason: 'Commit not signed' }
    end
    
    # Verify signature
    crypto = GPGME::Crypto.new
    signature = GPGME::Data.new(commit.gpg_signature)
    signed_text = GPGME::Data.new(commit.contents)
    
    begin
      signatures = crypto.verify(signature, signed_text: signed_text)
      sig = signatures.first
      
      # Check if signing key is trusted
      unless @trusted_keys.include?(sig.fingerprint)
        return {
          verified: false,
          reason: "Untrusted key: #{sig.fingerprint}"
        }
      end
      
      { verified: true, signer: sig.fingerprint }
    rescue GPGME::Error => e
      { verified: false, reason: "Signature verification failed: #{e.message}" }
    end
  end
  
  def verify_recent_commits(count = 10)
    commits = @repo.log(count)
    results = {}
    
    commits.each do |commit|
      results[commit.sha] = verify_commit(commit.sha)
    end
    
    results
  end
end

# Usage
verifier = CommitVerifier.new(
  '/path/to/gitops-repo',
  [
    'ABCD1234EFGH5678',  # Trusted developer key fingerprints
    '1234ABCD5678EFGH'
  ]
)

recent = verifier.verify_recent_commits
unsigned = recent.select { |sha, result| !result[:verified] }

if unsigned.empty?
  puts "All recent commits verified"
else
  puts "Unsigned or unverified commits found:"
  unsigned.each do |sha, result|
    puts "  #{sha[0..7]}: #{result[:reason]}"
  end
  exit 1
end

Policy engines can enforce additional requirements on configurations. Open Policy Agent (OPA) or Kyverno policies can validate resource specifications before controllers apply them. These policies check for required labels, resource limits, security contexts, and other organizational standards. Policy violations prevent configuration application, creating a safeguard against misconfigurations or malicious changes.

Image signing and verification extends supply chain security to container images. Controllers can require images referenced in configurations carry valid signatures from trusted registries. This prevents deployment of unofficial or compromised images even if attackers modify Git configurations.

Secrets in Git History: Accidentally committed secrets create permanent security risks. Git history remains immutable, so secrets cannot be removed without rewriting history across all repository clones. Organizations must implement preventive controls rather than reactive remediation.

Pre-commit hooks scan for secret patterns before allowing commits. These hooks detect API keys, passwords, certificates, and other sensitive patterns. Detection prevents secrets from entering repositories initially. Git guardian tools provide pre-commit hooks that integrate with local Git workflows.

Server-side scanning provides defense in depth. Services like GitHub secret scanning notify administrators when commits contain detected secrets. These notifications enable rapid response to rotate compromised credentials before attackers discover them.

Real-World Applications

GitOps patterns scale from small teams managing single applications to large enterprises coordinating thousands of services across hundreds of clusters. Implementation details vary based on organizational scale and operational maturity.

Microservices Deployment: Organizations with microservices architectures often adopt GitOps for standardizing deployment across diverse services. Each service team maintains configuration repositories following organization-wide templates. Platform teams provide base configurations, monitoring integrations, and security policies that service teams extend.

# Service deployment generator
class ServiceDeployer
  def initialize(service_name, team)
    @service_name = service_name
    @team = team
  end
  
  def generate_base_config
    {
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: {
        name: @service_name,
        labels: {
          app: @service_name,
          team: @team,
          managed_by: 'gitops'
        }
      },
      spec: {
        replicas: 3,
        selector: {
          matchLabels: { app: @service_name }
        },
        template: {
          metadata: {
            labels: {
              app: @service_name,
              team: @team
            },
            annotations: {
              'prometheus.io/scrape': 'true',
              'prometheus.io/port': '8080',
              'prometheus.io/path': '/metrics'
            }
          },
          spec: {
            serviceAccountName: "#{@service_name}-sa",
            containers: [{
              name: @service_name,
              image: "registry.example.com/#{@team}/#{@service_name}:latest",
              ports: [{ containerPort: 8080 }],
              env: [
                { name: 'SERVICE_NAME', value: @service_name },
                { name: 'TEAM', value: @team }
              ],
              resources: {
                requests: { cpu: '100m', memory: '128Mi' },
                limits: { cpu: '500m', memory: '512Mi' }
              },
              livenessProbe: {
                httpGet: { path: '/health', port: 8080 },
                initialDelaySeconds: 30,
                periodSeconds: 10
              },
              readinessProbe: {
                httpGet: { path: '/ready', port: 8080 },
                initialDelaySeconds: 5,
                periodSeconds: 5
              }
            }]
          }
        }
      }
    }
  end
  
  def generate_service_config
    {
      apiVersion: 'v1',
      kind: 'Service',
      metadata: {
        name: @service_name,
        labels: { app: @service_name, team: @team }
      },
      spec: {
        selector: { app: @service_name },
        ports: [{
          protocol: 'TCP',
          port: 80,
          targetPort: 8080
        }]
      }
    }
  end
  
  def write_configs(output_dir)
    deployment = generate_base_config
    service = generate_service_config
    
    FileUtils.mkdir_p(output_dir)
    
    File.write(
      File.join(output_dir, 'deployment.yaml'),
      deployment.to_yaml
    )
    File.write(
      File.join(output_dir, 'service.yaml'),
      service.to_yaml
    )
  end
end

# Generate configurations for new service
deployer = ServiceDeployer.new('payment-processor', 'platform')
deployer.write_configs('apps/payment-processor')

Microservices deployments benefit from GitOps observability. Each service's deployment state remains visible in Git history. Operators can quickly identify which version runs in each environment by examining Git tags and branches. Service interdependencies become explicit when services reference other services' endpoints or shared resources.

Configuration drift detection proves particularly valuable in microservices environments. With dozens or hundreds of services, manual verification of deployment state becomes impractical. GitOps controllers automatically detect and correct drift, maintaining consistency across the service fleet without operator intervention.

Infrastructure as Code Integration: GitOps extends beyond Kubernetes to infrastructure provisioning. Teams store Terraform configurations, CloudFormation templates, or Pulumi programs in Git repositories. Automated systems apply infrastructure changes following the same GitOps workflows used for application deployment.

# Terraform GitOps reconciliation
class TerraformReconciler
  def initialize(workspace_dir, state_backend)
    @workspace_dir = workspace_dir
    @state_backend = state_backend
  end
  
  def reconcile(git_ref)
    Dir.chdir(@workspace_dir) do
      # Pull latest configuration
      system("git fetch origin")
      system("git checkout #{git_ref}")
      
      # Initialize Terraform
      system("terraform init -backend-config='#{@state_backend}'")
      
      # Plan changes
      plan_output = `terraform plan -detailed-exitcode -out=tfplan`
      plan_exitcode = $?.exitcode
      
      case plan_exitcode
      when 0
        { status: 'synced', changes: false }
      when 2
        # Changes detected, apply them
        apply_output = `terraform apply -auto-approve tfplan`
        
        if $?.success?
          { status: 'synced', changes: true, output: apply_output }
        else
          { status: 'failed', error: apply_output }
        end
      else
        { status: 'error', error: plan_output }
      end
    end
  end
  
  def get_drift
    Dir.chdir(@workspace_dir) do
      plan_output = `terraform plan -detailed-exitcode`
      exitcode = $?.exitcode
      
      exitcode == 2 ? parse_drift(plan_output) : []
    end
  end
  
  private
  
  def parse_drift(plan_output)
    drift = []
    current_resource = nil
    
    plan_output.lines.each do |line|
      if line =~ /# (.+) (will be created|will be destroyed|will be updated)/
        drift << {
          resource: $1,
          action: $2.gsub('will be ', '')
        }
      end
    end
    
    drift
  end
end

# Usage
reconciler = TerraformReconciler.new(
  '/path/to/terraform-gitops',
  'backend.hcl'
)

result = reconciler.reconcile('main')
if result[:status] == 'synced'
  puts result[:changes] ? "Applied changes" : "No changes needed"
else
  puts "Reconciliation failed: #{result[:error]}"
end

# Check for drift
drift = reconciler.get_drift
unless drift.empty?
  puts "Drift detected:"
  drift.each { |d| puts "  #{d[:resource]} #{d[:action]}" }
end

Infrastructure GitOps implementations often separate networks, storage, and compute into distinct repositories. This separation allows different teams to manage their domains independently while maintaining relationships through Terraform remote state or explicit resource references.

Drift detection for infrastructure proves more complex than application deployment. Cloud providers may modify resources automatically for maintenance or security patches. GitOps systems must distinguish intentional provider modifications from unauthorized changes. Some infrastructure controllers implement read-only drift detection that alerts operators without automatically correcting drift.

Multi-Tenant Platforms: Organizations providing internal platforms use GitOps to enforce standardization while enabling tenant self-service. Platform teams define base configurations and policies. Tenant teams customize within allowed parameters by modifying their sections of GitOps repositories.

# Multi-tenant GitOps structure
platform-gitops/
├── tenants/
│   ├── team-a/
│   │   ├── namespace.yaml
│   │   ├── resource-quota.yaml
│   │   └── apps/
│   │       ├── service-1/
│   │       └── service-2/
│   ├── team-b/
│   │   ├── namespace.yaml
│   │   ├── resource-quota.yaml
│   │   └── apps/
│   └── team-c/
├── platform/
│   ├── networking/
│   ├── security/
│   └── monitoring/
└── policies/
    ├── resource-limits.rego
    └── security-context.rego

Repository permissions enforce tenant boundaries. Team A members cannot modify Team B's configuration. Platform team members can modify all sections. Policy engines validate tenant modifications comply with platform standards before accepting pull requests. This creates self-service within guardrails.

Multi-tenant platforms often implement namespace-per-tenant models where each team receives isolated Kubernetes namespaces. GitOps controllers reconcile tenant-specific configurations to appropriate namespaces. Resource quotas defined in Git prevent individual tenants from consuming excessive cluster resources.

Reference

Core Components

Component Purpose Common Implementations
Git Repository Source of truth for system state GitHub, GitLab, Bitbucket, Gitea
Reconciliation Controller Synchronizes desired and actual state ArgoCD, Flux, Config Sync, Jenkins X
Declarative Configuration Machine-readable system specifications Kubernetes YAML, Helm, Kustomize, Terraform
Diff Engine Compares desired vs actual state Built into controllers
Notification System Alerts on sync status and failures Slack, email, PagerDuty, webhooks

GitOps Principles Checklist

Principle Implementation Requirements
Declarative All configuration expressed as desired state
Versioned Every change recorded with commit history
Immutable Past states preserved and recoverable
Pull-based Systems pull configuration from Git
Automated Controllers handle synchronization without manual intervention

Repository Organization Patterns

Pattern Structure Best For
Monorepo Single repository for all configuration Small teams, shared infrastructure
Polyrepo Repository per application or service Microservices, independent teams
Environment-based Repository per environment Promotion-focused workflows
Hub and Spoke Platform repo plus app repos Shared platform with multiple tenants

Reconciliation Loop Phases

Phase Action Failure Handling
Fetch Pull latest from Git repository Retry with backoff
Parse Read and validate configuration Alert on syntax errors
Diff Compare desired vs actual state Log differences
Apply Create or update resources Retry individual resources
Prune Delete resources removed from Git Optional, often requires explicit enable
Report Update status and send notifications Best effort

Common Sync Policies

Policy Behavior Use Case
Automatic Apply changes immediately Non-production environments
Manual Require operator approval Production environments
Prune Enabled Delete resources not in Git Strict state enforcement
Prune Disabled Never delete resources Conservative deployments
Self-heal Revert manual changes Prevent configuration drift

Tool Feature Comparison

Feature ArgoCD Flux Jenkins X Config Sync
Web UI Yes Limited Yes No
Multi-cluster Yes Yes Yes Yes
Helm Support Yes Yes Yes No
Kustomize Support Yes Yes Yes Yes
Progressive Delivery Via Rollouts Via Flagger Built-in No
SSO Integration Yes No Yes No
RBAC Yes Via Kubernetes Yes Via Kubernetes

Secret Management Patterns

Approach How It Works Trade-offs
Sealed Secrets Encrypt secrets with cluster public key Requires key management
External Secrets Reference secrets in external vaults Depends on external system availability
SOPS Encrypt files with age or PGP More complex tooling
Git-crypt Transparent encryption in Git Requires local key distribution
Secret Injection Mount secrets at runtime Secrets not in Git at all

Health Status Values

Status Meaning Common Causes
Healthy All resources running correctly Normal operation
Progressing Deployment in progress Rollout ongoing, replica scaling
Degraded Some resources unhealthy Container crashes, failed probes
Suspended Sync temporarily disabled Manual intervention, maintenance
Missing Expected resources not found Misconfiguration, pruning issues
Unknown Health cannot be determined Resource status unclear

Sync Status Values

Status Meaning Action Required
Synced Git matches cluster state None
OutOfSync Git differs from cluster Review changes, sync if appropriate
Failed Sync attempt failed Check logs, fix configuration errors
Unknown Sync status unclear Investigate controller health

ArgoCD CLI Common Commands

Command Purpose
argocd app list List all applications
argocd app get APP Show application details
argocd app sync APP Trigger synchronization
argocd app diff APP Show pending changes
argocd app rollback APP Revert to previous version
argocd app history APP Show deployment history
argocd app set APP Update application settings

Flux CLI Common Commands

Command Purpose
flux get sources git List Git repositories
flux get kustomizations List kustomization resources
flux reconcile source git NAME Force repository update
flux reconcile kustomization NAME Force reconciliation
flux suspend kustomization NAME Pause reconciliation
flux resume kustomization NAME Resume reconciliation
flux logs Stream controller logs

Performance Characteristics

Metric Typical Value Optimization Strategy
Sync Frequency 1-5 minutes Balance freshness vs load
Diff Calculation Sub-second for small apps Cache computed diffs
Apply Duration Seconds to minutes Parallel resource application
Webhook Latency Sub-second Use webhooks for immediate sync
Git Poll Interval 1-3 minutes Use webhooks to reduce polling
Scale Limit Hundreds of applications Split across multiple controllers

Troubleshooting Common Issues

Symptom Likely Cause Investigation Steps
Continuous sync failures Invalid configuration Check resource validation, examine logs
OutOfSync but no changes Computed fields differ Enable resource ignore rules
Slow synchronization Large manifests or many resources Enable parallel sync, optimize manifests
Secret access failures Incorrect credentials or permissions Verify deploy keys, check RBAC
Webhook not triggering Incorrect webhook configuration Test webhook delivery, check firewall
Prune not removing resources Prune disabled or ownership issues Enable prune, check resource ownership labels