DEMO Monitor Flux using Prometheus Grafana

In this guide, you’ll learn how to integrate Prometheus with Flux v2 to scrape controller metrics and visualize them in Grafana. We’ll leverage the monitoring folder from the fluxcd/flux2 GitHub repo and deploy a PodMonitor CRD and Grafana dashboards via Flux Kustomizations.

1. Explore the monitoring directory

Clone or browse the Flux repository and locate the monitoring folder:

The image shows a GitHub repository page for "fluxcd/flux2" with a focus on the "monitoring" directory, containing folders like "kube-prometheus-stack" and "loki-stack."

Inside monitoring/ you’ll find:

PodMonitor YAML for scraping all Flux controllers.
A dashboards/ folder with two Grafana JSON files:
- cluster.json
- controlplane.json

The image shows a GitHub repository page for "fluxcd/flux2" with a focus on the "dashboards" directory, displaying JSON files related to monitoring configurations.

2. Apply the Flux PodMonitor

The PodMonitor CRD instructs Prometheus to scrape metrics from Flux controllers in the flux-system namespace:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  namespaceSelector:
    matchNames:
      - flux-system
  selector:
    matchExpressions:
      - key: app
        operator: In
        values:
          - helm-controller
          - source-controller
          - kustomize-controller
          - notification-controller
          - image-automation-controller
          - image-reflector-controller
  podMetricsEndpoints:
    - port: http-prom
      relabelings: []

Ensure the Prometheus Operator and its CRDs (including PodMonitor) are installed (for example via the kube-prometheus-stack).

Apply it directly or via Flux:

kubectl apply -f monitoring/manifests/monitoring-config/podmonitor.yaml

3. Create a Flux Kustomization

Automate the deployment by defining a Flux Kustomization that points to your Git source:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring-config
  namespace: flux-system
spec:
  dependsOn:
    - name: monitoring-kustomization-prometheus-stack
  interval: 1h0m0s
  path: ./manifests/monitoring/monitoring-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: monitoring-source-prometheus-stack

You can export this with the Flux CLI:

flux create kustomization monitoring-config \
  --namespace flux-system \
  --depends-on monitoring-kustomization-prometheus-stack \
  --interval 1h0m0s \
  --path "./manifests/monitoring/monitoring-config" \
  --prune=true \
  --source GitRepository/monitoring-source-prometheus-stack \
  --export > monitoring-config.yaml

Commit and push:

git add manifests/monitoring/monitoring-config
git commit -m "Add Flux PodMonitor and Grafana dashboards"
git push

Reconcile the Git source and kustomization:

flux reconcile source git flux-system --namespace flux-system
flux reconcile kustomization monitoring-config --namespace flux-system

Verify the PodMonitor is ready:

kubectl get podmonitor -n flux-system

4. Validate in Prometheus

Open your Prometheus UI and go to Status → Targets. Within seconds, the Flux controller endpoints should appear as UP:

The image shows a Prometheus monitoring dashboard displaying the status of various targets, with endpoints, states, labels, and scrape durations. All listed targets are marked as "UP," indicating they are functioning properly.

Common `gotk_` Queries

Query	Description
`gotk_reconcile_condition{type="Ready", status="True"}`	Count of successful reconciliations
`gotk_reconcile_condition{type="Ready", status="False"}`	Count of failed reconciliations
`gotk_suspend_status`	Suspension state of Git sources/controllers
`gotk_reconcile_duration_seconds_bucket`	Histogram buckets for reconcile durations
`gotk_reconcile_duration_seconds_sum`	Total reconcile duration
`gotk_reconcile_duration_seconds_count`	Number of reconcile operations

Run any query in Graph view to see real-time metrics:

The image shows a Prometheus monitoring interface displaying query results related to the "gotk_reconcile_condition" metric, with details about containers, endpoints, and statuses.

Switch to Graph mode for time-series visualization:

The image shows a Prometheus monitoring dashboard with a graph displaying multiple colored lines representing different metrics over time. A tooltip provides detailed information about a specific data point on the graph.

5. Explore Grafana Dashboards

After Flux applies the dashboard JSON, refresh Grafana. Two dashboards are now available:

Flux Control Plane
Flux Cluster Stats

Flux Control Plane

Tracks each controller’s queue lengths, CPU/memory usage, API request rates, and reconciliation durations:

The image shows a dashboard with graphs displaying metrics related to a Flux control plane, including successful and failed reconciliations, Git pulls, and Helm stats.

Flux Cluster Stats

Provides cluster-wide health: counts of reconcilers, failing controllers, manifest source statuses, operation durations, and readiness tables:

The image shows a Flux Cluster Stats dashboard displaying metrics such as cluster reconcilers, failing reconcilers, Kubernetes manifest sources, and their statuses. It includes tables and graphs indicating reconciliation readiness and operation durations.

These dashboards ship out of the box—just apply them to get comprehensive visibility into your Flux GitOps workflows.

GitOps Overview

Flux Overview

Source and Kustomize Controller

Helm Controller and OCI Registry

Image Automation Controller

Secret Management & Sign Verification

Notification Controller

Monitoring & User Interface

DEMO Monitor Flux using Prometheus Grafana

1. Explore the monitoring directory

2. Apply the Flux PodMonitor

3. Create a Flux Kustomization

4. Validate in Prometheus

Common `gotk_` Queries

5. Explore Grafana Dashboards

Flux Control Plane

Flux Cluster Stats

Links and References

Watch Video

Practice Lab

​1. Explore the monitoring directory

​2. Apply the Flux PodMonitor

​3. Create a Flux Kustomization

​4. Validate in Prometheus

​Common gotk_ Queries

​5. Explore Grafana Dashboards

​Flux Control Plane

​Flux Cluster Stats

​Links and References

Watch Video

Practice Lab

1. Explore the monitoring directory

2. Apply the Flux PodMonitor

3. Create a Flux Kustomization

4. Validate in Prometheus

Common `gotk_` Queries

5. Explore Grafana Dashboards

Flux Control Plane

Flux Cluster Stats

Links and References