GitOps Explained Desired State Drift and Reconciliation
Explains GitOps principles, the declarative desired state model, reconciliation loop, drift detection and self healing, and the four pillars enabling secure auditable continuous delivery.
Welcome to the GitOps and Continuous Delivery lesson.Scenario: It’s 2 a.m. and your pager goes off — production is down. You SSH into the cluster to investigate, but something changed and you have no record of what, when, or who. You’re flying blind. This is the world before GitOps.In this lesson you’ll learn:
The problems GitOps solves,
The four pillars of GitOps,
How the reconciliation loop functions, and
How drift detection and self-healing work.
Why incidents like this occurMany Kubernetes environments grant developers direct kubectl access to production. That enables ad-hoc, imperative changes from developers’ laptops. Those changes are applied directly, often without a centralized audit trail or version history.Example (imperative action):
# example of an imperative client actionkubectl apply -f ./manifest.yaml
When multiple people make ad-hoc changes—especially during high-pressure incidents—conflicting modifications can be introduced simultaneously. With no single source of truth and no formal change history, teams cannot track who changed what, when, or why.
Consequences: configuration drift, unpredictable behavior, and prolonged outagesUncoordinated changes create configuration drift between environments, unpredictable production behavior, and outages caused by incompatible or unintended changes. This makes root cause analysis difficult and puts production stability in the hands of human coordination and memory rather than an auditable system.
Imperative vs. DeclarativeRoot cause: the imperative model. Running imperative commands (scale, set image, etc.) immediately changes cluster state and leaves only ephemeral terminal history.Imperative examples:
# Imperative examples (state-changing commands run ad-hoc)kubectl scale deployment api --replicas=5kubectl set image deployment/web web=nginx:1.21
There is no integrated version history, no central audit trail, and no automatic detection when the same resource changes later.By contrast, the declarative model records desired state in files (YAML/JSON) committed to Git. Git automatically provides a versioned history so you can see who changed what, when, and why (when commit messages are used).Example (declarative snippet):
The key insight: GitOps replaces imperative “do this now” commands with declarative desired state and automation that converges actual cluster state to that declared state.
What is GitOps?GitOps is an operational model where Git repositories store declarative descriptions of infrastructure and applications. Automated controllers running in the cluster ensure the cluster state matches those Git-declared states.
The four pillars of GitOps
Pillar
What it means
Why it matters
Declarative
Desired state defined as files (YAML/JSON)
Describes the intended system rather than individual commands
Versioned
All manifests stored in Git
Complete commit history, easy rollbacks and audits
Pull-based
Agents inside the cluster pull from Git
No CI with persistent cluster creds; reduces secret sprawl and attack surface
Reconciled
Controllers continuously converge state
Detects and corrects drift; enables self-healing
If it’s not in Git, it does not exist. If it’s in Git, it should be in the cluster.How the reconciliation loop worksThe reconciliation loop is the engine of GitOps. Controllers such as Argo CD and Flux operate continuously in three core phases:
Watch: The controller detects Git changes via polling, webhooks, or provider notifications.
Compare: It compares the declared desired state in Git against the actual cluster state resource-by-resource.
Sync: When differences are found, the controller applies the necessary changes (create/update/delete) to align the cluster with Git.
This loop integrates with developer workflows: Code & Push → Detect & Compare → Sync & Deploy → Continuous Watch.
Example: you change a Deployment’s image from 1.0 to 2.0 and push the commit. Within the controller’s detection window (seconds to minutes), it sees the Git commit, compares the cluster’s running 1.0 deployment, and applies the update to 2.0. The loop is continuous; it’s not a one-time deployment.Drift detection and self-healingDrift occurs when the cluster diverges from Git-declared desired state. Controllers handle drift in several ways depending on policy and environment sensitivity.
Response
Behavior
Use case
Auto-sync (self-healing)
Controller automatically reverts unintended changes to match Git
Standard GitOps for most environments
Alert + manual sync
Controller alerts team and waits for approval to apply changes
Production systems requiring human checks
Diff + PR workflow
Controller generates a diff or PR to update Git when cluster changes are intentional
When cluster-initiated changes must be reflected back into Git
Common sources of drift
Manual kubectl edits and ad-hoc changes.
Autoscalers (HPA/VPA) adjusting replicas or resource requests/limits.
Other controllers or operators adding annotations/labels or modifying resources.
Principle to follow: Git is authoritative. Either the controller syncs the cluster to Git, or you update Git to reflect an intentional change in the cluster. The cluster is not the source of truth.
Using a pull-based model (agents inside the cluster pulling from Git) reduces secret sprawl. CI/CD systems do not need persistent cluster credentials to apply changes, improving security and auditability.
Recap — core takeaways
Git is the single source of truth: desired state belongs in version-controlled repositories.
Prefer declarative manifests over imperative commands: state is described, automation handles convergence.
Continuous reconciliation prevents unintentional drift: controllers compare and correct constantly.
Pull-based deployments are more secure and auditable because the cluster pulls from Git and CI/CD systems avoid persistent credentials.
