Skip to main content
In this lesson you’ll learn how to raise an Alertmanager alert whenever an Argo CD application becomes OutOfSync. We assume Alertmanager and Prometheus are already installed and accessible (Alertmanager via NodePort in this environment). This guide covers:
  • Creating a Prometheus rule for Argo CD application drift
  • Where to add the rule (Prometheus Operator PrometheusRule)
  • How to test by introducing drift in an Argo CD application
  • Viewing the fired alert in Alertmanager and forwarding notifications (Slack example)

​
Define the Prometheus alert rule

Create a PrometheusRule group that triggers when Argo CD reports an OutOfSync application. The PromQL expression uses the argocd_app_info metric exposed by Argo CD. 
# PrometheusRule group for Argo CD
- name: ArgoCD Rules
  rules:
    - alert: ArgoApplicationOutOfSync
      expr: argocd_app_info{sync_status="OutOfSync"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "'{{ $labels.name }}' Application has synchronization issue"

​
What each field means

FieldPurposeExample / Notes
nameGroup name that organizes related alertsArgoCD Rules
alertUnique alert identifier shown in Alertmanager and dashboardsArgoApplicationOutOfSync
exprPromQL expression that evaluates the conditionargocd_app_info{sync_status="OutOfSync"} == 1
forMinimum duration the condition must hold before firing5m to avoid transient noise
labelsMetadata for alert routing and severityseverity: warning (use critical for high priority)
annotationsHuman-readable descriptions inserted into notificationssummary using {{ $labels.name }} to reference the app name

​
Where to add this rule

Add the group into an existing PrometheusRule resource managed by the Prometheus Operator (namespace often monitoring). Example commands to inspect and edit the PrometheusRule: 
kubectl -n monitoring get prometheusrules.monitoring.coreos.com kode-kloud-prometheus-stac-alertmanager.rules -o yaml
kubectl -n monitoring edit prometheusrules.monitoring.coreos.com kode-kloud-prometheus-stac-alertmanager.rules
Insert the ArgoCD Rules group inside the spec.groups array. Example fragment showing the ArgoCD group placed at the top of spec.groups: 
spec:
  groups:
  - name: ArgoCD Rules
    rules:
    - alert: ArgoApplicationOutOfSync
      expr: argocd_app_info{sync_status="OutOfSync"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "'{{ $labels.name }}' Application has synchronization issue"
  # ... other groups and rules follow ...
After saving, the Prometheus Operator will reconcile the PrometheusRule; Prometheus will reload rules automatically, which can take a short moment.

​
Test the alert by causing drift

To test the rule, make an Argo CD application go OutOfSync. A simple approach is to modify the live cluster resource so it no longer matches the Git desired state. Open the Argo CD application UI and select an application to test. You can also directly edit a Deployment to introduce drift.
The image displays the Argo CD application's dashboard, showing the synchronization status and health of the "highway-animation" application with its deployment and replica sets visualized.
Example: edit the live Deployment to change replicas or a container env value: 
kubectl -n highway-animation edit deployment highway-animation
Example deployment fragment (this was edited live to create drift): 
spec:
  replicas: 1
  template:
    spec:
      containers:
        - name: highway-animation
          image: siddharth67/highway-animation:blue
          env:
            - name: POD_COUNT
              value: "8"
          ports:
            - containerPort: 3000
              protocol: TCP
Once the application reports OutOfSync and the rule evaluates true for for: 5m, the alert will fire in Alertmanager. While waiting, Alertmanager may display other currently active alerts.
The image shows a web interface listing alerts for various Kubernetes jobs such as "kube-controller-manager," "kube-etcd," and others, with options to expand groups and silence alerts.

​
Viewing the fired alert

After the for window elapses and the alert fires, Alertmanager will show the new alert. The alert details include labels such as job="argocd-metrics", namespace, the application name, repository URL, and the severity label.
The image shows an Alertmanager interface displaying an alert related to "argocd-metrics," with details like severity, sync status, and other metadata tags.

​
Forwarding alerts (Slack example)

Once the alert appears in Alertmanager you can route it to external receivers (Slack, PagerDuty, email, etc.) by configuring Alertmanager receivers and routes.
Ensure the argocd_app_info metric is being scraped by Prometheus. Argo CD exposes metrics via the argocd-metrics endpoint; if Prometheus isn’t scraping argocd-metrics, the rule cannot evaluate true.
Example Alertmanager Slack configuration (in Alertmanager YAML): 
global:
  resolve_timeout: 1m
  slack_api_url: 'https://hooks.slack.com/services/TSUJ1MIHQ/BT7J5TR5/5eZMpDbKk8wk2'
route:
  receiver: 'slack-notifications'
receivers:
  - name: 'slack-notifications'
    slack_configs:
      - channel: '#monitoring-instances'
        send_resolved: true
Example Slack message template for Alertmanager (use in templates files): 
{{- /* Example Slack message template for Alertmanager */ -}}
text: >
{{ range .Alerts -}}
*Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

*Description:* {{ .Annotations.description }}

*Details:*
{{ range .Labels.SortedPairs -}}
* {{ .Name }}: `{{ .Value }}`
{{ end }}
{{ end }}
The image is a screenshot from a Grafana Labs blog on setting up Slack alerts using Prometheus. It shows steps in Slack to manage apps and search for Incoming WebHooks, alongside a sidebar listing blog contents and a prompt to create a Grafana Cloud account.

​
Notes and next steps

  • After confirming the alert works, refine labels and routing in your Alertmanager configuration to match your on-call and escalation workflows.
  • Use severity labels to differentiate notification channels (e.g., warning -> pager muted, critical -> paged).
  • Integrate Grafana or other visualization tools to display Argo CD metrics and alert status.
Useful references: That’s how you create an alert for Argo CD application drift, see it in Alertmanager, and route notifications to external systems like Slack.

Watch Video

Practice Lab