Delivery Troubleshooting Drift Permissions and Bad Configs

Everything we’ve covered so far assumes sync will work. But in real clusters, sync often fails. When an Argo CD Application is OutOfSync, the root cause can be one of several distinct categories: drift, permission (RBAC) issues, or invalid manifests. This guide builds a systematic troubleshooting toolkit that covers the three categories responsible for roughly 90% of GitOps failures:

Drift caused by controllers or manual edits
Role-based access control (RBAC) permission problems
Invalid or misconfigured manifests

For each category you’ll find diagnostic commands and concrete fixes so you can resolve issues quickly.

A presentation slide titled "Learning Objectives" with a blue gradient left panel and four colorful numbered markers down the center. The right side lists objectives about ArgoCD: diagnosing perpetual OutOfSync from drift, resolving RBAC permission issues, validating manifests before deployment, and using argocd app diff/kubectl auth can‑i/--dry-run=server.

Real-world example: perpetual OutOfSync loop

A platform team spent four hours debugging an Argo CD Application stuck OutOfSync even after repeated manual syncs. Git config was correct, cluster connectivity was fine, and manifests were valid — yet every sync reverted to OutOfSync within seconds. Root cause: an HPA controlling replicas. Git declared replicas: 3; the cluster’s HPA kept the Deployment at 7. Argo CD synced to 3, the HPA scaled to 7, and the app returned to OutOfSync—resulting in a perpetual drift loop. Key lesson: not all differences are errors. Some drift is expected when controllers (HPAs, operators) manage fields. Recognize that pattern and decide whether to tolerate or reconcile that drift.

Common symptoms and likely causes

Symptom	Likely causes
Perpetual OutOfSync (never reaches Synced)	HPA scaling replicas, operators/controllers modifying resources, repeated manual kubectl edits
Sync failed (errors during sync)	RBAC/permission errors preventing create/update/delete; missing verbs for cluster-scoped resources
Degraded / Progressing (resources applied but unhealthy)	Bad configuration: image pull errors, insufficient resources, missing dependencies, wrong probe configuration
Unknown health	CRDs missing, custom health checks undefined, unsupported resource types for health assessment

Diagnosing drift

If your application is always OutOfSync even after syncing, start with an exact comparison between Git and the cluster.

CLI diff:

argocd app diff my-app

Or use the Diff tab in the Argo CD UI.

This shows field-by-field differences: e.g. spec.replicas desired: 3 vs actual: 7, or extra annotations present only in the cluster. Common drift sources:

HPA modifies spec.replicas
Operators or controllers add fields, annotations, or labels
Manual kubectl edits or patches
API server defaulting (fields populated by the server that you didn’t set)

A presentation slide titled "Diagnosing Drift" showing "Common Drift Sources" in four numbered boxes. The boxes list: 1) HPA modifies replicas, 2) Operators add fields, 3) Manual kubectl edits, and 4) Defaulting by API server.

If the drift is expected (for example the HPA manages replicas), tell Argo CD to ignore those differences by adding ignoreDifferences to the Application spec:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  # ... source / destination / syncPolicy ...
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: my-app
      jsonPointers:
        - /spec/replicas

Use ignoreDifferences sparingly. Ignoring fields hides drift from Argo CD — acceptable for controller-managed fields (like replicas managed by an HPA), but avoid ignoring fields that should be reconciled by Git.

Practical tips:

Prefer scoping ignores narrowly (resource kind, name, and specific JSON pointers).
Consider documenting ignored differences in your repo so platform consumers understand which fields are controlled by cluster controllers.

Diagnosing RBAC / permission issues

When Argo CD reports forbidden errors, fails to prune resources, or produces partial syncs, RBAC is a common cause. Symptoms:

forbidden errors shown in sync logs
Partial creation: some resources created while others fail (e.g., Deployments succeed but CRDs or cluster-scoped resources fail)
Resources persist despite auto-prune (delete permission missing)

Diagnostic steps:

Check effective permissions of the Argo CD controller service account with kubectl auth can-i. Run this as the Argo CD controller SA to emulate what Argo CD can do:

# Example: check whether Argo CD controller can create a Deployment
kubectl auth can-i create deployment \
  --as=system:serviceaccount:argocd:argocd-application-controller

Inspect controller logs for detailed error messages:

kubectl logs -n argocd deploy/argocd-application-controller

Common causes and fixes:

Missing ClusterRole or ClusterRoleBinding: grant cluster-scoped permissions if Argo CD must manage cluster resources.
Role bound instead of ClusterRole for resources that require cluster scope.
Application Project restricts kinds or namespaces — update the Project allow-list if appropriate.

Be careful when granting cluster-wide permissions. Follow least privilege: only add verbs and API groups required for the resources your Argo CD instance must reconcile. Review ClusterRoleBindings and avoid granting system:admin unless absolutely necessary.

Quick examples:

Granting create/update/delete on a cluster-scoped resource (ClusterRole + ClusterRoleBinding).
Allowing specific resource kinds in an Argo CD Project:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  # allow specific kinds
  clusterResourceWhitelist:
    - group: "apiextensions.k8s.io"
      kind: "CustomResourceDefinition"

Diagnosing bad manifests and validating before Argo CD sees them

Bad manifests cause SyncFailed or Degraded states. Catch them early in CI/CD using server-side dry-run validation to test against the cluster API. Common manifest issues:

Invalid YAML (indentation, missing punctuation)
Schema validation errors (wrong field type, unknown fields)
Referencing CRs when CRDs are not installed
Target namespace not created or mismatched

Validation commands (run in CI or locally):

# Dry-run against cluster API server (validates schema, CRDs, and permissions)
kubectl apply --dry-run=server -f .

# Render Helm chart and validate against cluster
helm template . | kubectl apply --dry-run=server -f -

# Render Kustomize and validate against cluster
kubectl kustomize . | kubectl apply --dry-run=server -f -

Pro tip: run these validations on every PR to block merges that would introduce invalid manifests. Integrate with CI tools (GitHub Actions, GitLab CI, Tekton) to fail early. Additional helpful tool: use argocd app diff my-app to preview what Argo CD will change before committing to Git or initiating a sync.

Quick reference: diagnostic commands

Purpose	Command
Compare Git vs live cluster	`argocd app diff my-app`
Check Argo CD SA permission to create a Deployment	`kubectl auth can-i create deployment --as=system:serviceaccount:argocd:argocd-application-controller`
Inspect Argo CD application controller logs	`kubectl logs -n argocd deploy/argocd-application-controller`
Validate manifests server-side	`kubectl apply --dry-run=server -f .`
Validate Helm rendered templates	`helm template . \| kubectl apply --dry-run=server -f -`
Validate Kustomize output	`kubectl kustomize . \| kubectl apply --dry-run=server -f -`

Five things to remember

Use argocd app diff to see exact differences between Git and the cluster.
Use ignoreDifferences to exclude expected controller-managed drift (narrowly scoped).
Use kubectl auth can-i as the Argo CD service account to verify permissions.
Use kubectl apply --dry-run=server in CI to catch schema and CRD issues early.
Most GitOps issues fall into drift, permissions, or configuration errors — learning to diagnose each saves hours.

This lesson concludes the module. You have learned GitOps principles, repo design, configuration templating, Argo CD usage, and Kubernetes-native delivery patterns. References and further reading:

Argo CD documentation: https://argo-cd.readthedocs.io/
Kubernetes RBAC: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
kubectl apply dry-run: https://kubernetes.io/docs/reference/using-api/server-side-apply/

Use argocd app diff to identify drift between Git and cluster.
Use ignoreDifferences to exclude expected drift from sync status.
Use kubectl auth can-i to verify Argo CD permissions.
Use kubectl apply --dry-run=server to catch schema issues early.
Most GitOps issues fall into drift, permissions, or configuration errors.

Course Introduction

Platform Architecture and Infrastructure

GitOps and Continuous Delivery

Platform APIs and Self-Service Capabilities

Observability and Operations

Security and Policy Enforcement

Conclusion

Delivery Troubleshooting Drift Permissions and Bad Configs

Real-world example: perpetual OutOfSync loop

Common symptoms and likely causes

Diagnosing drift

Diagnosing RBAC / permission issues

Diagnosing bad manifests and validating before Argo CD sees them

Quick reference: diagnostic commands

Five things to remember

Watch Video

Practice Lab

​Real-world example: perpetual OutOfSync loop

​Common symptoms and likely causes

​Diagnosing drift

​Diagnosing RBAC / permission issues

​Diagnosing bad manifests and validating before Argo CD sees them

​Quick reference: diagnostic commands

​Five things to remember

Watch Video

Practice Lab

Real-world example: perpetual OutOfSync loop

Common symptoms and likely causes

Diagnosing drift

Diagnosing RBAC / permission issues

Diagnosing bad manifests and validating before Argo CD sees them

Quick reference: diagnostic commands

Five things to remember