Skip to main content
In this guide, you’ll learn how to integrate Prometheus with Flux v2 to scrape controller metrics and visualize them in Grafana. We’ll leverage the monitoring folder from the fluxcd/flux2 GitHub repo and deploy a PodMonitor CRD and Grafana dashboards via Flux Kustomizations.

​
1. Explore the monitoring directory

Clone or browse the Flux repository and locate the monitoring folder:
The image shows a GitHub repository page for "fluxcd/flux2" with a focus on the "monitoring" directory, containing folders like "kube-prometheus-stack" and "loki-stack."
Inside monitoring/ you’ll find:
  • PodMonitor YAML for scraping all Flux controllers.
  • A dashboards/ folder with two Grafana JSON files:
    • cluster.json
    • controlplane.json
The image shows a GitHub repository page for "fluxcd/flux2" with a focus on the "dashboards" directory, displaying JSON files related to monitoring configurations.

​
2. Apply the Flux PodMonitor

The PodMonitor CRD instructs Prometheus to scrape metrics from Flux controllers in the flux-system namespace: 
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  namespaceSelector:
    matchNames:
      - flux-system
  selector:
    matchExpressions:
      - key: app
        operator: In
        values:
          - helm-controller
          - source-controller
          - kustomize-controller
          - notification-controller
          - image-automation-controller
          - image-reflector-controller
  podMetricsEndpoints:
    - port: http-prom
      relabelings: []
Ensure the Prometheus Operator and its CRDs (including PodMonitor) are installed (for example via the kube-prometheus-stack).
Apply it directly or via Flux: 
kubectl apply -f monitoring/manifests/monitoring-config/podmonitor.yaml

​
3. Create a Flux Kustomization

Automate the deployment by defining a Flux Kustomization that points to your Git source: 
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring-config
  namespace: flux-system
spec:
  dependsOn:
    - name: monitoring-kustomization-prometheus-stack
  interval: 1h0m0s
  path: ./manifests/monitoring/monitoring-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: monitoring-source-prometheus-stack
You can export this with the Flux CLI: 
flux create kustomization monitoring-config \
  --namespace flux-system \
  --depends-on monitoring-kustomization-prometheus-stack \
  --interval 1h0m0s \
  --path "./manifests/monitoring/monitoring-config" \
  --prune=true \
  --source GitRepository/monitoring-source-prometheus-stack \
  --export > monitoring-config.yaml
Commit and push: 
git add manifests/monitoring/monitoring-config
git commit -m "Add Flux PodMonitor and Grafana dashboards"
git push
Reconcile the Git source and kustomization: 
flux reconcile source git flux-system --namespace flux-system
flux reconcile kustomization monitoring-config --namespace flux-system
Verify the PodMonitor is ready: 
kubectl get podmonitor -n flux-system

​
4. Validate in Prometheus

Open your Prometheus UI and go to Status → Targets. Within seconds, the Flux controller endpoints should appear as UP:
The image shows a Prometheus monitoring dashboard displaying the status of various targets, with endpoints, states, labels, and scrape durations. All listed targets are marked as "UP," indicating they are functioning properly.

​
Common gotk_ Queries

QueryDescription
gotk_reconcile_condition{type="Ready", status="True"}Count of successful reconciliations
gotk_reconcile_condition{type="Ready", status="False"}Count of failed reconciliations
gotk_suspend_statusSuspension state of Git sources/controllers
gotk_reconcile_duration_seconds_bucketHistogram buckets for reconcile durations
gotk_reconcile_duration_seconds_sumTotal reconcile duration
gotk_reconcile_duration_seconds_countNumber of reconcile operations
Run any query in Graph view to see real-time metrics:
The image shows a Prometheus monitoring interface displaying query results related to the "gotk_reconcile_condition" metric, with details about containers, endpoints, and statuses.
Switch to Graph mode for time-series visualization:
The image shows a Prometheus monitoring dashboard with a graph displaying multiple colored lines representing different metrics over time. A tooltip provides detailed information about a specific data point on the graph.

​
5. Explore Grafana Dashboards

After Flux applies the dashboard JSON, refresh Grafana. Two dashboards are now available:
  • Flux Control Plane
  • Flux Cluster Stats

​
Flux Control Plane

Tracks each controller’s queue lengths, CPU/memory usage, API request rates, and reconciliation durations:
The image shows a dashboard interface displaying metrics for a Flux Control Plane, including controllers, max work queue time, memory usage, API requests, and resource usage graphs for CPU and memory.
The image shows a dashboard with graphs displaying metrics related to a Flux control plane, including successful and failed reconciliations, Git pulls, and Helm stats.

​
Flux Cluster Stats

Provides cluster-wide health: counts of reconcilers, failing controllers, manifest source statuses, operation durations, and readiness tables:
The image shows a dashboard interface displaying metrics for a Flux Control Plane, including controllers, max work queue time, memory usage, API requests, and resource usage graphs.
The image shows a Flux Cluster Stats dashboard displaying metrics such as cluster reconcilers, failing reconcilers, Kubernetes manifest sources, and their statuses. It includes tables and graphs indicating reconciliation readiness and operation durations.
These dashboards ship out of the box—just apply them to get comprehensive visibility into your Flux GitOps workflows.

Watch Video

Practice Lab