> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Delivery Troubleshooting Drift Permissions and Bad Configs

> Guide to troubleshooting Argo CD delivery failures by diagnosing drift, RBAC permission problems, and invalid manifests with commands and fixes to resolve OutOfSync, sync errors, and degraded resources

Everything we've covered so far assumes sync will work. But in real clusters, sync often fails. When an Argo CD Application is OutOfSync, the root cause can be one of several distinct categories: drift, permission (RBAC) issues, or invalid manifests. This guide builds a systematic troubleshooting toolkit that covers the three categories responsible for roughly 90% of GitOps failures:

* Drift caused by controllers or manual edits
* Role-based access control (RBAC) permission problems
* Invalid or misconfigured manifests

For each category you'll find diagnostic commands and concrete fixes so you can resolve issues quickly.

![A presentation slide titled "Learning Objectives" with a blue gradient left panel and four colorful numbered markers down the center. The right side lists objectives about ArgoCD: diagnosing perpetual OutOfSync from drift, resolving RBAC permission issues, validating manifests before deployment, and using argocd app diff/kubectl auth can‑i/--dry-run=server.](https://mintcdn.com/kodekloud-c4ac6d9a/8pYzlc1aD6cgTw8b/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/GitOps-and-Continuous-Delivery/Delivery-Troubleshooting-Drift-Permissions-and-Bad-Configs/argocd-learning-objectives-outofsync-rbac-diff.jpg?fit=max\&auto=format\&n=8pYzlc1aD6cgTw8b\&q=85\&s=52ea62174b5569985b751605969a0acc)

## Real-world example: perpetual OutOfSync loop

A platform team spent four hours debugging an Argo CD Application stuck OutOfSync even after repeated manual syncs. Git config was correct, cluster connectivity was fine, and manifests were valid — yet every sync reverted to OutOfSync within seconds.

Root cause: an HPA controlling replicas. Git declared `replicas: 3`; the cluster's HPA kept the Deployment at `7`. Argo CD synced to `3`, the HPA scaled to `7`, and the app returned to OutOfSync—resulting in a perpetual drift loop.

Key lesson: not all differences are errors. Some drift is expected when controllers (HPAs, operators) manage fields. Recognize that pattern and decide whether to tolerate or reconcile that drift.

## Common symptoms and likely causes

| Symptom | Likely causes |
| - | - |
| Perpetual OutOfSync (never reaches Synced) | HPA scaling replicas, operators/controllers modifying resources, repeated manual kubectl edits |
| Sync failed (errors during sync) | RBAC/permission errors preventing create/update/delete; missing verbs for cluster-scoped resources |
| Degraded / Progressing (resources applied but unhealthy) | Bad configuration: image pull errors, insufficient resources, missing dependencies, wrong probe configuration |
| Unknown health | CRDs missing, custom health checks undefined, unsupported resource types for health assessment |

## Diagnosing drift

If your application is always OutOfSync even after syncing, start with an exact comparison between Git and the cluster.

* CLI diff:

```bash theme={null}
argocd app diff my-app
```

* Or use the Diff tab in the Argo CD UI.

This shows field-by-field differences: e.g. `spec.replicas` desired: `3` vs actual: `7`, or extra annotations present only in the cluster.

Common drift sources:

* HPA modifies `spec.replicas`
* Operators or controllers add fields, annotations, or labels
* Manual `kubectl` edits or patches
* API server defaulting (fields populated by the server that you didn’t set)

![A presentation slide titled "Diagnosing Drift" showing "Common Drift Sources" in four numbered boxes. The boxes list: 1) HPA modifies replicas, 2) Operators add fields, 3) Manual kubectl edits, and 4) Defaulting by API server.](https://mintcdn.com/kodekloud-c4ac6d9a/8pYzlc1aD6cgTw8b/images/Prep-Course-Certified-Cloud-Native-Platform-Engineering-CNPE/GitOps-and-Continuous-Delivery/Delivery-Troubleshooting-Drift-Permissions-and-Bad-Configs/diagnosing-drift-hpa-operators-kubectl-defaults.jpg?fit=max\&auto=format\&n=8pYzlc1aD6cgTw8b\&q=85\&s=eb5eb33077fb9300e7fe5a10d45e8ee8)

If the drift is expected (for example the HPA manages replicas), tell Argo CD to ignore those differences by adding `ignoreDifferences` to the Application spec:

```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  # ... source / destination / syncPolicy ...
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: my-app
      jsonPointers:
        - /spec/replicas
```

Use `ignoreDifferences` sparingly. Ignoring fields hides drift from Argo CD — acceptable for controller-managed fields (like replicas managed by an HPA), but avoid ignoring fields that should be reconciled by Git.

Practical tips:

* Prefer scoping ignores narrowly (resource kind, name, and specific JSON pointers).
* Consider documenting ignored differences in your repo so platform consumers understand which fields are controlled by cluster controllers.

## Diagnosing RBAC / permission issues

When Argo CD reports forbidden errors, fails to prune resources, or produces partial syncs, RBAC is a common cause.

Symptoms:

* `forbidden` errors shown in sync logs
* Partial creation: some resources created while others fail (e.g., Deployments succeed but CRDs or cluster-scoped resources fail)
* Resources persist despite auto-prune (delete permission missing)

Diagnostic steps:

* Check effective permissions of the Argo CD controller service account with `kubectl auth can-i`. Run this as the Argo CD controller SA to emulate what Argo CD can do:

```bash theme={null}
# Example: check whether Argo CD controller can create a Deployment
kubectl auth can-i create deployment \
  --as=system:serviceaccount:argocd:argocd-application-controller
```

* Inspect controller logs for detailed error messages:

```bash theme={null}
kubectl logs -n argocd deploy/argocd-application-controller
```

Common causes and fixes:

* Missing ClusterRole or ClusterRoleBinding: grant cluster-scoped permissions if Argo CD must manage cluster resources.
* Role bound instead of ClusterRole for resources that require cluster scope.
* Application Project restricts kinds or namespaces — update the Project allow-list if appropriate.

Be careful when granting cluster-wide permissions. Follow least privilege: only add verbs and API groups required for the resources your Argo CD instance must reconcile. Review ClusterRoleBindings and avoid granting system:admin unless absolutely necessary.

Quick examples:

* Granting create/update/delete on a cluster-scoped resource (ClusterRole + ClusterRoleBinding).
* Allowing specific resource kinds in an Argo CD Project:

```yaml theme={null}
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: default
  namespace: argocd
spec:
  # allow specific kinds
  clusterResourceWhitelist:
    - group: "apiextensions.k8s.io"
      kind: "CustomResourceDefinition"
```

## Diagnosing bad manifests and validating before Argo CD sees them

Bad manifests cause SyncFailed or Degraded states. Catch them early in CI/CD using server-side dry-run validation to test against the cluster API.

Common manifest issues:

* Invalid YAML (indentation, missing punctuation)
* Schema validation errors (wrong field type, unknown fields)
* Referencing CRs when CRDs are not installed
* Target namespace not created or mismatched

Validation commands (run in CI or locally):

```bash theme={null}
# Dry-run against cluster API server (validates schema, CRDs, and permissions)
kubectl apply --dry-run=server -f .

# Render Helm chart and validate against cluster
helm template . | kubectl apply --dry-run=server -f -

# Render Kustomize and validate against cluster
kubectl kustomize . | kubectl apply --dry-run=server -f -
```

Pro tip: run these validations on every PR to block merges that would introduce invalid manifests. Integrate with CI tools (GitHub Actions, GitLab CI, Tekton) to fail early.

Additional helpful tool: use `argocd app diff my-app` to preview what Argo CD will change before committing to Git or initiating a sync.

## Quick reference: diagnostic commands

| Purpose | Command |
| - | - |
| Compare Git vs live cluster | `argocd app diff my-app` |
| Check Argo CD SA permission to create a Deployment | `kubectl auth can-i create deployment --as=system:serviceaccount:argocd:argocd-application-controller` |
| Inspect Argo CD application controller logs | `kubectl logs -n argocd deploy/argocd-application-controller` |
| Validate manifests server-side | `kubectl apply --dry-run=server -f .` |
| Validate Helm rendered templates | `helm template . \| kubectl apply --dry-run=server -f -` |
| Validate Kustomize output | `kubectl kustomize . \| kubectl apply --dry-run=server -f -` |

## Five things to remember

* Use `argocd app diff` to see exact differences between Git and the cluster.
* Use `ignoreDifferences` to exclude expected controller-managed drift (narrowly scoped).
* Use `kubectl auth can-i` as the Argo CD service account to verify permissions.
* Use `kubectl apply --dry-run=server` in CI to catch schema and CRD issues early.
* Most GitOps issues fall into drift, permissions, or configuration errors — learning to diagnose each saves hours.

This lesson concludes the module. You have learned GitOps principles, repo design, configuration templating, Argo CD usage, and Kubernetes-native delivery patterns.

References and further reading:

* Argo CD documentation: [https://argo-cd.readthedocs.io/](https://argo-cd.readthedocs.io/)
* Kubernetes RBAC: [https://kubernetes.io/docs/reference/access-authn-authz/rbac/](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
* kubectl apply dry-run: [https://kubernetes.io/docs/reference/using-api/server-side-apply/](https://kubernetes.io/docs/reference/using-api/server-side-apply/)

```text theme={null}
Use argocd app diff to identify drift between Git and cluster.
Use ignoreDifferences to exclude expected drift from sync status.
Use kubectl auth can-i to verify Argo CD permissions.
Use kubectl apply --dry-run=server to catch schema issues early.
Most GitOps issues fall into drift, permissions, or configuration errors.
```

> **[Watch Video](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/dff5382b-dbe7-4cac-bd2b-d5a47028945e/lesson/6a1ab174-fb6a-4448-8b53-689c6ea9fccc)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/prep-course-certified-cloud-native-platform-engineer-cnpe/module/dff5382b-dbe7-4cac-bd2b-d5a47028945e/lesson/9581df5b-9e92-49bc-87e8-2e256be4e549)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.