Demo Monitor Flux using Prometheus Grafana

In this lesson you will configure Prometheus to scrape metrics exported by Flux controllers and visualize those metrics in Grafana dashboards. This gives you visibility into reconciliation status, durations, suspended resources, and resource usage for the Flux GitOps control plane.

Repository layout

The Flux v2 repository contains a ready-made monitoring configuration that includes a PodMonitor CRD and prebuilt Grafana dashboards. The PodMonitor tells Prometheus (via the Prometheus Operator) which Flux controller pods to scrape, while the JSON dashboard files can be auto-provisioned into Grafana.

A dark-themed GitHub repository page for the fluxcd/flux2 project showing the path manifests/monitoring/monitoring-config/dashboards and a list of files including cluster.json, control-plane.json, and logs.json. The header shows repo actions (Code, Issues, Pull requests) and a recent commit/author area on the directory.

For reference:

Flux v2 repository: https://github.com/fluxcd/flux2
Flux metrics and documentation: https://fluxcd.io/docs/

PodMonitor — add Flux controllers as scrape targets

A PodMonitor resource (managed by the Prometheus Operator) selects pods and exposes metrics endpoints for Prometheus to scrape. The example below selects pods in the flux-system namespace and matches the app label values for the Flux controllers. It configures Prometheus to scrape the http-prom metrics port.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: flux-system
  namespace: flux-system
  labels:
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/component: monitoring
spec:
  namespaceSelector:
    matchNames:
      - flux-system
  selector:
    matchExpressions:
      - key: app
        operator: In
        values:
          - helm-controller
          - source-controller
          - kustomize-controller
          - notification-controller
          - image-automation-controller
          - image-reflector-controller
  podMetricsEndpoints:
    - port: http-prom
      relabelings: []

Ensure Prometheus is installed with the Prometheus Operator (for example via the kube-prometheus-stack Helm chart). PodMonitor is a CRD provided by that operator, and Prometheus will only discover targets if the operator is present.

Useful links:

Prometheus Operator: https://github.com/prometheus-operator/prometheus-operator
kube-prometheus-stack chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

Apply the monitoring config with Flux Kustomization

Use a Flux Kustomization to point to the monitoring manifests stored in Git. The Kustomization below references the repository path containing the PodMonitor and dashboard files and declares a dependency on your Prometheus stack kustomization.

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring-config
  namespace: flux-system
spec:
  dependsOn:
    - name: monitoring-kustomization-prometheus-stack
  interval: 1h0m0s
  path: ./manifests/monitoring/monitoring-config
  prune: true
  sourceRef:
    kind: GitRepository
    name: monitoring-source-prometheus-stack

Typical workflow:

Add the PodMonitor and dashboard JSON files to your monitoring path in Git.
Commit and push the changes.
Reconcile the Kustomization manually or wait for the next sync interval:
- Example: flux reconcile kustomization monitoring-config --namespace flux-system

Validation commands

Purpose	Command
Confirm Kustomization exists	`flux get kustomizations -n flux-system`
Confirm PodMonitor resource	`kubectl get podmonitor -n flux-system`
Describe a PodMonitor	`kubectl describe podmonitor flux-system -n flux-system`

Prometheus targets

Once Flux creates the PodMonitor, the Prometheus Operator will convert it into scrape configurations and create targets for each controller. In the Prometheus UI, visit Status → Targets to verify a podMonitor/monitoring/flux-system entry with an endpoint per controller.

Prometheus metrics and PromQL examples

Flux exports metrics prefixed with gotk_ that cover reconciliation conditions, durations, and suspend status. Use PromQL in the Prometheus expression browser to explore these timeseries. Common Flux metrics

gotk_reconcile_condition{type="Ready", status="<status>"} — reconciliation Ready condition (statuses: True, False, Unknown, Deleted)
gotk_suspend_status — whether resources are suspended
gotk_reconcile_duration_seconds_* — histogram metrics for reconcile durations

Example PromQL expressions:

# Reconciliation conditions (Ready)
gotk_reconcile_condition{type="Ready", status="True"}
gotk_reconcile_condition{type="Ready", status="False"}
gotk_reconcile_condition{type="Ready", status="Unknown"}
gotk_reconcile_condition{type="Ready", status="Deleted"}

# Suspend status
gotk_suspend_status

# Reconciliation duration histogram
gotk_reconcile_duration_seconds_bucket
gotk_reconcile_duration_seconds_sum
gotk_reconcile_duration_seconds_count

Example output (abridged):

gotk_reconcile_condition{container="manager", endpoint="http-prom", exported_namespace="flux-system", instance="10.1.0.218:8080", job="monitoring/flux-system", kind="Alert", name="notification-alert-slack", namespace="flux-system", pod="notification-controller-57fb4fcb68-c8lnl", status="True", type="Ready"} 1

gotk_reconcile_condition{container="manager", endpoint="http-prom", exported_namespace="flux-system", instance="10.1.0.220:8080", job="monitoring/flux-system", kind="HelmRelease", name="5-demo-helm-release-git-helm-bb-app", namespace="flux-system", pod="helm-controller-644cc6fdfb-tn7kw", status="True", type="Ready"} 1

These timeseries are suitable for:

Building Grafana visualizations of controller health and performance
Defining alerting rules for reconciliation failures or long reconcile durations

Grafana dashboards and provisioning

The Flux repository provides dashboard JSON files such as control-plane.json and cluster.json. If your Grafana is configured to provision dashboards from a directory (or if you import them manually), these files will give you immediate visualizations for Flux metrics.

A screenshot of Grafana's "Dashboards" browse view in dark mode, showing a scrollable list of monitoring dashboards (Alertmanager, CoreDNS, Flux, Kubernetes, etc.) with checkboxes, tag labels, and action buttons (Move, Delete). The left navigation bar and browser tabs are also visible.

Grafana must be configured to provision dashboards from the location where Flux places the dashboard JSON files, or you must import the dashboards manually. Check your Grafana provisioning settings or the Grafana UI to ensure the dashboards appear after you apply the monitoring manifests.

Control plane dashboard

This dashboard focuses on controller-level metrics: controller counts, work queue sizes, CPU/memory usage, and Kubernetes API request rates. Use it to spot overloaded controllers or API hotspots.

A Grafana monitoring dashboard for a Flux control plane showing key metrics — 6 controllers, 1.46 hour max work queue, 220 Mb memory, and ~23.4 API requests — with time-series charts for Kubernetes API requests, CPU usage and memory.

Cluster stats / readiness dashboard

This dashboard summarizes reconcilers, manifest sources, failing sources, reconciliation durations, artifact readiness, and includes tables showing resource readiness across namespaces. It’s useful for assessing cluster-wide health of Flux-managed resources.

A Grafana dashboard showing Flux cluster statistics and readiness, with large metric panels (e.g., 15 cluster reconcilers, 11 manifest sources, 1 failing source) and charts for reconciler/source durations plus tables of cluster and source readiness. The lower section lists namespaces, resource kinds and readiness statuses for many Flux resources.

Summary & next steps

Add the provided PodMonitor so Prometheus (via the Prometheus Operator) can scrape Flux controller metrics.
Use a Flux Kustomization or your preferred deployment method to apply the monitoring manifests from Git.
Verify the PodMonitor and Prometheus targets, then explore Flux metrics with PromQL in Prometheus.
Provision or import the Grafana dashboards to visualize the Flux control plane and cluster metrics, and consider creating alerts based on gotk_ metrics.

GitOps Overview

Flux Overview

Source and Kustomize Controller

Helm Controller and OCI Registry

Image Automation Controller

Secret Management & Sign Verification

Notification Controller

Monitoring & User Interface

Demo Monitor Flux using Prometheus Grafana

Repository layout

PodMonitor — add Flux controllers as scrape targets

Apply the monitoring config with Flux Kustomization

Prometheus targets

Prometheus metrics and PromQL examples

Grafana dashboards and provisioning

Control plane dashboard

Cluster stats / readiness dashboard

Summary & next steps

Watch Video

Practice Lab

​Repository layout

​PodMonitor — add Flux controllers as scrape targets

​Apply the monitoring config with Flux Kustomization

​Prometheus targets

​Prometheus metrics and PromQL examples

​Grafana dashboards and provisioning

​Control plane dashboard

​Cluster stats / readiness dashboard

​Summary & next steps

Watch Video

Practice Lab

Repository layout

PodMonitor — add Flux controllers as scrape targets

Apply the monitoring config with Flux Kustomization

Prometheus targets

Prometheus metrics and PromQL examples

Grafana dashboards and provisioning

Control plane dashboard

Cluster stats / readiness dashboard

Summary & next steps