Delivery Troubleshooting Drift Permissions and Bad Configs

Everything we’ve covered so far assumes sync will succeed. When it doesn’t, diagnosing the failure quickly is critical. Argo CD applications reported as OutOfSync or SyncFailed usually fall into three categories:

Drift (external controllers or manual changes)
Permissions (RBAC / service account limitations)
Invalid configurations (bad manifests, missing CRDs, missing namespaces)

This guide provides a compact, systematic troubleshooting toolkit with the key diagnostic commands and remediation steps to resolve most GitOps delivery failures. Three categories cover about ninety percent of GitOps failures. For each category you’ll find: the diagnostic question, quick commands to inspect the problem, and typical remediation patterns.

The image lists four learning objectives related to resolving technical issues, validating manifests, and using specific tools and commands effectively. The objectives are visually highlighted with numbered tags on a gradient background.

Real-world example A platform team repeatedly synced an Argo CD application. Git matched the desired manifests, the cluster was reachable, and each sync initially succeeded — but within seconds the app returned to OutOfSync. Root cause: an HPA controlling replicas. Git declared spec.replicas: 3, while the HPA scaled the Deployment to 7. Argo CD reconciled the Deployment back to 3; the HPA then bumped it to 7, causing a continuous drift loop. Key lesson: not every difference between Git and the cluster is an error. Controllers (HPAs, operators) and the API server often mutate objects. Recognize expected drift and tell Argo CD what to ignore. Common symptoms and usual causes

Perpetual OutOfSync — often drift from HPAs, operators, or manual edits that continuously change cluster state.
SyncFailed — typically permission problems: Argo CD cannot create/update/delete resources.
Degraded / Progressing — resources were applied but are not becoming healthy; usually bad configurations (image pull errors, insufficient resources, missing dependencies).
Unknown health — Argo CD cannot determine health for a resource type, often because CRDs or health checks are missing.

The image outlines four common symptoms of sync failures: Perpetual OutOfSync, SyncFailed, Degraded/Progressing, and Unknown, along with their potential causes.

Quick reference: symptoms → likely cause → first diagnostic command

Symptom	Likely cause	First diagnostic command
Perpetual OutOfSync	Drift (HPA, operator, controller, manual edits)	`argocd app diff <app>`
SyncFailed	RBAC / permissions	`kubectl auth can-i ... --as=system:serviceaccount:<ns>:<sa>`
Degraded / Progressing	Bad manifests, image pull, resources	`kubectl describe <resource>` / `kubectl logs`
Unknown health	Missing CRD / custom health checks	`kubectl get crd` / check Argo CD health overrides

Diagnosis and remediation — start with the most common: drift Drift (most common cause of perpetual OutOfSync)

Diagnostic question: Why does my app stay OutOfSync even after syncing?
Diagnostic tool: Argo CD diff (UI or CLI Diff tab) to see field-by-field differences between Git and live cluster.

Example CLI:

# Compare application manifests in Git vs cluster
argocd app diff my-app

The diff highlights differences such as spec.replicas (Git: 3, live: 7) or extra annotations/fields present only in the cluster. Common drift sources:

HPA modifying spec.replicas.
Operators adding fields or status subtrees.
Controllers adding annotations or labels.
Manual kubectl edits or patches.
API server server-side defaulting.

The image outlines four common sources of drift in Kubernetes configurations: HPA modifying replicas, operators adding fields, manual kubectl edits, and defaulting by the API server.

When drift is expected: ignore it If a controller is intentionally managing a field (for example, an HPA managing spec.replicas), add an ignoreDifferences entry in the Application spec so Argo CD does not treat that as drift. Example Application snippet:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  # ... source / destination / syncPolicy ...
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: my-app
      jsonPointers:
        - /spec/replicas

Only ignore fields that are purposefully controlled by external controllers. Overusing ignoreDifferences can mask genuine drift or configuration mistakes.

Permissions (RBAC) issues

Symptoms: SyncFailed with forbidden, partial syncs (some resources created while others fail), or inability to delete resources during auto-prune.
Cause: Argo CD’s service account lacks the required create/update/delete permissions for certain resource kinds, or the ApplicationProject restricts allowed kinds.

Diagnostic tools:

kubectl auth can-i with impersonation to simulate Argo CD controller permissions.
Controller logs for detailed error messages (argocd-application-controller).

Commands:

# Example: check whether the Argo CD application-controller can create Deployments
kubectl auth can-i create deploy --as=system:serviceaccount:argocd:argocd-application-controller

# View Argo CD application-controller logs for permission errors
kubectl logs -n argocd deploy/argocd-application-controller

Typical fixes:

Grant missing permissions using ClusterRole / ClusterRoleBinding for cluster-scoped needs.
Configure namespace-scoped Role / RoleBinding for limited privileges.
Add required resource kinds to the ApplicationProject allow list if the project policy is blocking them.
If CRDs cause permission or create errors, ensure CRDs are installed before Argo CD tries to manage CR instances.

The image illustrates permission issues related to ArgoCD, highlighting missing ClusterRole bindings, namespace-scoped limitations, and CRDs not being allowed, along with a suggested fix.

Bad manifests and validation

Symptoms: SyncFailed or resources stuck in Degraded/Progressing because YAML is invalid, fields have wrong types, CRDs are missing, or target namespaces don’t exist.
Best practice: validate manifests before they reach Argo CD (CI pre-merge checks).

Common issues to check:

Invalid YAML (indentation, missing colons).
Schema validation errors (wrong types or unknown fields).
CRDs not installed in the cluster.
Target namespaces don’t exist or lack access.

Validation (server-side dry-run validates against the API server and catches schema / CRD / permission issues):

# Dry-run apply against the cluster API server
kubectl apply --dry-run=server -f .

# Render Helm or Kustomize and validate against the cluster
helm template . | kubectl apply --dry-run=server -f -
kubectl kustomize . | kubectl apply --dry-run=server -f -

Use argocd app diff locally too:

# Compare rendered local manifests with the live cluster before committing
argocd app diff my-app --local ./manifests

Add kubectl apply --dry-run=server (or rendered Helm/Kustomize validation) to CI to block PRs that would fail validation in the cluster. This prevents the most common sync failures caused by invalid manifests or missing CRDs.

Additional troubleshooting tips

Check controller logs and kubectl describe / kubectl logs for failing pods.
For custom resource types, confirm CRDs exist and are compatible with the manifests being applied.
Use Argo CD UI Diff/Sync history to see when drift started and what changed.
When using operators, check operator documentation for which fields they manage so you can ignore them in Argo CD.

Summary — Five key points to remember

Use argocd app diff to see exact differences between Git and the cluster.
Use ignoreDifferences for fields intentionally managed by external controllers (e.g., HPA-driven replicas).
Use kubectl auth can-i (impersonating the Argo CD service account) and argocd-application-controller logs to diagnose RBAC problems.
Add kubectl apply --dry-run=server (or rendered Helm/Kustomize validation) to CI to catch schema/CRD/permission issues before merge.
Most GitOps issues fall into drift, permissions, or bad configurations. Learning to diagnose each category reduces debugging time from hours to minutes.

Course Introduction

Platform Architecture and Infrastructure

GitOps and Continuous Delivery

Platform APIs and Self-Service Capabilities

Observability and Operations

Security and Policy Enforcement

Conclusion

Delivery Troubleshooting Drift Permissions and Bad Configs

Watch Video

Practice Lab