Demo Install Kube Prometheus Stack

This tutorial shows how to install the kube-prometheus-stack on a Kubernetes cluster using Flux v2 components and the Prometheus Community Helm chart. Follow these steps to create the Flux Git source and Kustomization, add a HelmRepository and HelmRelease for the chart, and then expose Prometheus and Grafana for a local demo. This guide follows Flux documentation patterns and includes sample manifests and commands you can adapt.

Flux example commands

The Flux documentation provides example commands to create a Git source and a Kustomization that points to a monitoring path in a Git repository. Use these as a starting point for a GitOps deployment:

flux create source git flux-monitoring \
  --interval=30m \
  --url=https://github.com/fluxcd/flux2 \
  --branch=main

flux create kustomization kube-prometheus-stack \
  --interval=1h \
  --prune \
  --source=flux-monitoring \
  --path="./manifests/monitoring/kube-prometheus-stack" \
  --health-check-timeout=5m \
  --wait

These example commands create a Git source and Kustomization in your cluster. Replace the URL, branch, and path with your repository and manifest locations. For more Flux examples and options, see the Flux docs: https://fluxcd.io/docs/.

Below we adapt that flow to install the kube-prometheus-stack by creating a Git source that points to a repository containing the kube-prometheus-stack manifests and then creating a Kustomization pointing to the specific path.

Helm repository and HelmRelease (chart configuration)

The Flux examples typically include a HelmRepository that points to the Prometheus Community Helm charts. Example:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: prometheus-community
spec:
  interval: 120m
  # OCI builds for kube-prometheus-stack have been temporarily disabled
  # (see https://github.com/prometheus-community/helm-charts/issues/2940).
  type: default
  url: https://prometheus-community.github.io/helm-charts

A HelmRelease installs the kube-prometheus-stack chart and configures values. In this demo Alertmanager is disabled and Prometheus retention is set:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
spec:
  interval: 5m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "45.x"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
      interval: 60m
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
  # https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
  values:
    alertmanager:
      enabled: false
    prometheus:
      prometheusSpec:
        retention: 24h
        resources:
          requests:
            cpu: 200m
            memory: 200Mi
    podMonitorNamespaceSelector: {}
    podMonitorSelector:
      matchLabels:
        app.kubernetes.io/component: monitoring

Note: when authoring MDX content, avoid placing raw curly-brace snippets inline; keep JSON/YAML examples in fenced code blocks as shown above.

Create Flux Git source and Kustomization

On your workstation (with Flux CLI configured for the target cluster) create the Git source pointing to the repository that contains the kube-prometheus-stack manifests. The following example uses the Flux project’s repo for kube-prometheus-stack:

flux create source git monitoring-source-prometheus-stack \
  --interval=30m \
  --url=https://github.com/fluxcd/kube-prometheus-stack \
  --branch=main \
  --export > monitoring-source-prometheus-stack.yml

The exported GitRepository YAML will look like:

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: monitoring-source-prometheus-stack
  namespace: flux-system
spec:
  interval: 30m0s
  ref:
    branch: main
  url: https://github.com/fluxcd/kube-prometheus-stack

Create (export) a Kustomization that points at the monitoring path inside that repository:

flux create kustomization monitoring-kustomization-prometheus-stack \
  --interval=1h \
  --source=monitoring-source-prometheus-stack \
  --path="./manifests/monitoring/kube-prometheus-stack" \
  --export > monitoring-kustomization-prometheus-stack.yml

Exported Kustomization YAML:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring-kustomization-prometheus-stack
  namespace: flux-system
spec:
  interval: 1h0m0s
  path: ./manifests/monitoring/kube-prometheus-stack
  prune: false
  sourceRef:
    kind: GitRepository
    name: monitoring-source-prometheus-stack

After exporting these manifests, either commit and push them to the Git repository Flux watches or apply them directly with:

kubectl apply -f monitoring-source-prometheus-stack.yml
kubectl apply -f monitoring-kustomization-prometheus-stack.yml

What Flux creates (quick summary)

Once Flux reconciles the Kustomization the monitoring namespace is created and the chart manifests are applied. Typical components created by the kube-prometheus-stack include:

Resource Type	Purpose	Example
StatefulSet	Prometheus storage and replicas	`statefulset.apps/prometheus-kube-prometheus-stack-prometheus`
DaemonSet	node-exporter for host metrics	`daemonset.apps/kube-prometheus-stack-prometheus-node-exporter`
Deployment	Grafana, kube-state-metrics	`deployment.apps/kube-prometheus-stack-grafana`
Service	Cluster networking for Prometheus/Grafana	`service/kube-prometheus-stack-prometheus`
HelmRelease / HelmRepository	Flux-managed Helm install	See `HelmRelease` and `HelmRepository` YAML above

Inspect what was created in the monitoring namespace

A sample kubectl -n monitoring get all output soon after install (trimmed for clarity):

$ kubectl -n monitoring get all
NAME                                                              READY   STATUS             RESTARTS      AGE
pod/kube-prometheus-stack-grafana-7f8b9dc64b-vfqvn                3/3     Running            0             14s
pod/kube-prometheus-stack-kube-state-metrics-5464cfdf86-vjhkg     1/1     Running            0             14s
pod/kube-prometheus-stack-operator-fc96b6df7-bvwp6                 1/1     Running            0             14s
pod/kube-prometheus-stack-prometheus-node-exporter-lwzct          0/1     CrashLoopBackOff   1 (13s ago)   14s
pod/prometheus-kube-prometheus-stack-prometheus-0                 1/2     Running            0             12s

NAME                                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/kube-prometheus-stack-grafana                             ClusterIP   10.99.232.155    <none>        80/TCP      14s
service/kube-prometheus-stack-kube-state-metrics                  ClusterIP   10.108.229.49    <none>        8080/TCP    14s
service/kube-prometheus-stack-operator                            ClusterIP   10.111.241.52    <none>        443/TCP     14s
service/kube-prometheus-stack-prometheus                          ClusterIP   10.97.179.234    <none>        9090/TCP    14s

NAME
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter     1          1               0          1          0          <none>   14s

NAME                                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-prometheus-stack-grafana                      1/1     1            1           14s
deployment.apps/kube-prometheus-stack-kube-state-metrics           1/1     1            1           14s
deployment.apps/kube-prometheus-stack-operator                     1/1     1            1           14s

NAME                                                          READY   AGE
statefulset.apps/prometheus-kube-prometheus-stack-prometheus  0/1     12s

You will typically see a Prometheus StatefulSet, node-exporter DaemonSet, and multiple deployments and services for Grafana and state metrics.

Expose Prometheus and Grafana (NodePort for local access)

By default services are created as ClusterIP. For a local demo you can temporarily change Grafana and Prometheus to NodePort so they are accessible from your workstation browser. Warning: exposing monitoring services with NodePort is convenient for demos but not recommended for production. Use Ingress, LoadBalancer, or proper authentication in production.

Exposing Prometheus and Grafana via NodePort is suitable for local testing only. For production, secure these endpoints via ingress/load balancer and enable authentication.

Edit the services:

kubectl -n monitoring edit svc kube-prometheus-stack-prometheus
kubectl -n monitoring edit svc kube-prometheus-stack-grafana

Change type: ClusterIP to type: NodePort and save. Verify the services:

$ kubectl -n monitoring get svc
NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
kube-prometheus-stack-grafana                   NodePort    10.99.232.155    <none>        80:31921/TCP        2m9s
kube-prometheus-stack-kube-state-metrics        ClusterIP   10.108.229.49    <none>        8080/TCP            2m9s
kube-prometheus-stack-operator                  ClusterIP   10.111.241.52    <none>        443/TCP             2m9s
kube-prometheus-stack-prometheus                NodePort    10.97.179.234    <none>        9090:30753/TCP      2m9s
kube-prometheus-stack-prometheus-node-exporter  ClusterIP   10.108.183.71    <none>        9100/TCP            2m9s
prometheus-operated                              ClusterIP   None             <none>        9090/TCP            2m7s

Open Prometheus at http://localhost:<PROMETHEUS_NODEPORT> (e.g., http://localhost:30753) and Grafana at http://localhost:<GRAFANA_NODEPORT> (e.g., http://localhost:31921) — replace ports with the NodePort values shown on your node.

Grafana credentials

Grafana’s admin password is stored in a Kubernetes Secret. Retrieve and decode it:

kubectl -n monitoring get secret kube-prometheus-stack-grafana -o yaml

Example secret YAML (data fields are base64-encoded):

apiVersion: v1
data:
  admin-password: cHJvbS1vcGVyYXRvcg==
  admin-user: YWRtaW4=
  ldap-toml: ""
kind: Secret
metadata:
  name: kube-prometheus-stack-grafana
  namespace: monitoring
type: Opaque

Decode the password:

echo 'cHJvbS1vcGVyYXRvcg==' | base64 -d
echo 'YWRtaW4=' | base64 -d
# => admin

Default credentials in this demo:

Username: admin
Password: prom-operator

Log in to Grafana and explore the preinstalled dashboards (Kubernetes overview, Node metrics, Prometheus overview). Flux-specific dashboards are not included by default.

Prometheus UI — Targets for scraping

Prometheus will be running after the chart install, but you may not immediately see Flux-related targets. Open the Prometheus “Targets” page to view discovered targets and scraping status. If Flux components are not listed, add PodMonitor or ServiceMonitor resources that match Flux component labels so Prometheus can scrape them.

A screenshot of the Prometheus "Targets" page showing multiple serviceMonitors and their endpoints with labels, last scrape times, scrape durations and status indicators. One entry shows a red "DOWN" error for a kube-controller-manager endpoint while others are marked "UP."

If you need to add scraping for Flux, create a PodMonitor or ServiceMonitor that targets the Flux component pods or services and matches the selectors defined in your HelmRelease (or add new selectors).

Grafana dashboards — add Flux dashboards

Grafana displays dashboards installed by the chart. To visualize Flux metrics you must import or provision dashboards that target Flux metrics and the service/PodMonitor labels used for scraping.

A dark-themed Grafana dashboard browse page showing a list of monitoring dashboards (Alertmanager, CoreDNS, Grafana Overview, various Kubernetes dashboards) with tags and search/filter controls. The screenshot displays the sidebar and a scrollable list of dashboard entries.

Next steps:

Add a PodMonitor or ServiceMonitor to enable scraping of Flux components.
Provision or import Grafana dashboards for Flux metrics.
For production exposure, switch from NodePort to Ingress/LoadBalancer and secure access with authentication.

References

Flux documentation: https://fluxcd.io/docs/
Prometheus Community Helm charts: https://prometheus-community.github.io/helm-charts
kube-prometheus-stack chart values and dashboards: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

That’s all for this lesson — you now have kube-prometheus-stack installed via Flux and can extend scraping and dashboards to monitor Flux components.

GitOps Overview

Flux Overview

Source and Kustomize Controller

Helm Controller and OCI Registry

Image Automation Controller

Secret Management & Sign Verification

Notification Controller

Monitoring & User Interface

Demo Install Kube Prometheus Stack

Flux example commands

Helm repository and HelmRelease (chart configuration)

Create Flux Git source and Kustomization

What Flux creates (quick summary)

Inspect what was created in the monitoring namespace

Expose Prometheus and Grafana (NodePort for local access)

Grafana credentials

Prometheus UI — Targets for scraping

Grafana dashboards — add Flux dashboards

References

Watch Video

​Flux example commands

​Helm repository and HelmRelease (chart configuration)

​Create Flux Git source and Kustomization

​What Flux creates (quick summary)

​Inspect what was created in the monitoring namespace

​Expose Prometheus and Grafana (NodePort for local access)

​Grafana credentials

​Prometheus UI — Targets for scraping

​Grafana dashboards — add Flux dashboards

​References

Watch Video

Flux example commands

Helm repository and HelmRelease (chart configuration)

Create Flux Git source and Kustomization

What Flux creates (quick summary)

Inspect what was created in the monitoring namespace

Expose Prometheus and Grafana (NodePort for local access)

Grafana credentials

Prometheus UI — Targets for scraping

Grafana dashboards — add Flux dashboards

References