ArgoCD Metrics Monitoring 2

In this guide, we explore how Prometheus can leverage ArgoCD metrics to raise alerts using Alertmanager—automating the detection of synchronization issues in ArgoCD applications.

The image contains text that reads "ArgoCD Metrics Prometheus + AlertManager" highlighted in yellow.

Alertmanager handles alerts generated by client applications, such as the Prometheus server. In this article, we explain how to configure alerting rules based on ArgoCD metrics within a Prometheus instance. Previously, we examined how Prometheus scrapes these metrics; now, we will use them to define alert rules for detecting synchronization problems in ArgoCD applications.

As part of the Prometheus Operator, configuration details are maintained for both Alertmanager and Prometheus rules. The Prometheus Rules custom resource enables you to define both recording and alerting rules for your Prometheus instance.

Below is an example of a PrometheusRule configuration containing a single alerting rule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-argocd-rules
spec:
- name: ArgoCD Rules
  rules:
  - alert: ArgoApplicationOutOfSync
    expr: argocd_app_info{sync_status="OutOfSync"} == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "{{ $labels.name }} Application has synchronization issue"

To verify the applied configuration, run the following commands. These commands connect to the Prometheus pod and display the contents of the rules configuration file:

$ kubectl exec -it prometheus-0 -c config-reloader -- /bin/sh
$ cat /etc/prometheus/rules/prometheus-rulefiles-0/argocd-app-sync

Understanding Alert Rule Components

Each alert rule configuration includes:

Alert name: The identifier for the alert (e.g., "ArgoApplicationOutOfSync").
Expression: The condition that triggers the alert. In this example, the alert fires when an ArgoCD application's synchronization status is out of sync (argocd_app_info{sync_status="OutOfSync"} == 1).
Duration: The period (5 minutes in this instance) the condition must persist before the alert triggers.
Labels: Metadata tags for routing and categorizing alerts, such as a severity level set to "warning".
Annotations: Additional information, including a summary message to describe the alert.

When the Prometheus rule is created, the Prometheus Operator automatically generates the corresponding configuration for Prometheus using the Prometheus Rule custom resource definition (CRD). It then uses the Config Reloader component to update the rules file accordingly.

Below is an updated version of the PrometheusRule configuration (functionally identical to the previous example but formatted properly):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: example
    role: alert-rules
  name: prometheus-argocd-rules
spec:
- name: ArgoCD Rules
  rules:
  - alert: ArgoApplicationOutOfSync
    expr: argocd_app_info{sync_status="OutOfSync"} == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "'{{ $labels.name }}' Application has synchronization issue"

You can inspect the active configuration with these commands:

$ kubectl exec -it prometheus-0 -c config-reloader -- /bin/sh
$ cat /etc/prometheus/rules/prometheus-rulefiles-0/argocd-app-sync

Verify Alerting Rules

After updating the rules file, check the Rules tab in the Prometheus UI. If an ArgoCD application's synchronization status deviates from the expected state, the alert will trigger and be visible in the Alertmanager UI.

Watch Video

Watch video content