> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Install Kube Prometheus Stack

> Tutorial to install kube-prometheus-stack on Kubernetes using Flux v2, Helm charts and Kustomize, configure HelmRelease and expose Prometheus and Grafana for local demos

This tutorial shows how to install the kube-prometheus-stack on a Kubernetes cluster using Flux v2 components and the Prometheus Community Helm chart. Follow these steps to create the Flux Git source and Kustomization, add a HelmRepository and HelmRelease for the chart, and then expose Prometheus and Grafana for a local demo. This guide follows Flux documentation patterns and includes sample manifests and commands you can adapt.

## Flux example commands

The Flux documentation provides example commands to create a Git source and a Kustomization that points to a monitoring path in a Git repository. Use these as a starting point for a GitOps deployment:

```bash theme={null}
flux create source git flux-monitoring \
  --interval=30m \
  --url=https://github.com/fluxcd/flux2 \
  --branch=main

flux create kustomization kube-prometheus-stack \
  --interval=1h \
  --prune \
  --source=flux-monitoring \
  --path="./manifests/monitoring/kube-prometheus-stack" \
  --health-check-timeout=5m \
  --wait
```

These example commands create a Git source and Kustomization in your cluster. Replace the URL, branch, and path with your repository and manifest locations. For more Flux examples and options, see the Flux docs: [https://fluxcd.io/docs/](https://fluxcd.io/docs/).

Below we adapt that flow to install the kube-prometheus-stack by creating a Git source that points to a repository containing the kube-prometheus-stack manifests and then creating a Kustomization pointing to the specific path.

## Helm repository and HelmRelease (chart configuration)

The Flux examples typically include a HelmRepository that points to the Prometheus Community Helm charts. Example:

```yaml theme={null}
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: prometheus-community
spec:
  interval: 120m
  # OCI builds for kube-prometheus-stack have been temporarily disabled
  # (see https://github.com/prometheus-community/helm-charts/issues/2940).
  type: default
  url: https://prometheus-community.github.io/helm-charts
```

A HelmRelease installs the kube-prometheus-stack chart and configures values. In this demo Alertmanager is disabled and Prometheus retention is set:

```yaml theme={null}
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kube-prometheus-stack
spec:
  interval: 5m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "45.x"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
      interval: 60m
  install:
    crds: Create
  upgrade:
    crds: CreateReplace
  # https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
  values:
    alertmanager:
      enabled: false
    prometheus:
      prometheusSpec:
        retention: 24h
        resources:
          requests:
            cpu: 200m
            memory: 200Mi
    podMonitorNamespaceSelector: {}
    podMonitorSelector:
      matchLabels:
        app.kubernetes.io/component: monitoring
```

Note: when authoring MDX content, avoid placing raw curly-brace snippets inline; keep JSON/YAML examples in fenced code blocks as shown above.

## Create Flux Git source and Kustomization

On your workstation (with Flux CLI configured for the target cluster) create the Git source pointing to the repository that contains the kube-prometheus-stack manifests. The following example uses the Flux project’s repo for kube-prometheus-stack:

```bash theme={null}
flux create source git monitoring-source-prometheus-stack \
  --interval=30m \
  --url=https://github.com/fluxcd/kube-prometheus-stack \
  --branch=main \
  --export > monitoring-source-prometheus-stack.yml
```

The exported GitRepository YAML will look like:

```yaml theme={null}
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: monitoring-source-prometheus-stack
  namespace: flux-system
spec:
  interval: 30m0s
  ref:
    branch: main
  url: https://github.com/fluxcd/kube-prometheus-stack
```

Create (export) a Kustomization that points at the monitoring path inside that repository:

```bash theme={null}
flux create kustomization monitoring-kustomization-prometheus-stack \
  --interval=1h \
  --source=monitoring-source-prometheus-stack \
  --path="./manifests/monitoring/kube-prometheus-stack" \
  --export > monitoring-kustomization-prometheus-stack.yml
```

Exported Kustomization YAML:

```yaml theme={null}
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: monitoring-kustomization-prometheus-stack
  namespace: flux-system
spec:
  interval: 1h0m0s
  path: ./manifests/monitoring/kube-prometheus-stack
  prune: false
  sourceRef:
    kind: GitRepository
    name: monitoring-source-prometheus-stack
```

After exporting these manifests, either commit and push them to the Git repository Flux watches or apply them directly with:

```bash theme={null}
kubectl apply -f monitoring-source-prometheus-stack.yml
kubectl apply -f monitoring-kustomization-prometheus-stack.yml
```

## What Flux creates (quick summary)

Once Flux reconciles the Kustomization the `monitoring` namespace is created and the chart manifests are applied. Typical components created by the kube-prometheus-stack include:

| Resource Type | Purpose | Example |
| - | - | - |
| StatefulSet | Prometheus storage and replicas | `statefulset.apps/prometheus-kube-prometheus-stack-prometheus` |
| DaemonSet | node-exporter for host metrics | `daemonset.apps/kube-prometheus-stack-prometheus-node-exporter` |
| Deployment | Grafana, kube-state-metrics | `deployment.apps/kube-prometheus-stack-grafana` |
| Service | Cluster networking for Prometheus/Grafana | `service/kube-prometheus-stack-prometheus` |
| HelmRelease / HelmRepository | Flux-managed Helm install | See `HelmRelease` and `HelmRepository` YAML above |

## Inspect what was created in the monitoring namespace

A sample `kubectl -n monitoring get all` output soon after install (trimmed for clarity):

```console theme={null}
$ kubectl -n monitoring get all
NAME                                                              READY   STATUS             RESTARTS      AGE
pod/kube-prometheus-stack-grafana-7f8b9dc64b-vfqvn                3/3     Running            0             14s
pod/kube-prometheus-stack-kube-state-metrics-5464cfdf86-vjhkg     1/1     Running            0             14s
pod/kube-prometheus-stack-operator-fc96b6df7-bvwp6                 1/1     Running            0             14s
pod/kube-prometheus-stack-prometheus-node-exporter-lwzct          0/1     CrashLoopBackOff   1 (13s ago)   14s
pod/prometheus-kube-prometheus-stack-prometheus-0                 1/2     Running            0             12s

NAME                                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
service/kube-prometheus-stack-grafana                             ClusterIP   10.99.232.155    <none>        80/TCP      14s
service/kube-prometheus-stack-kube-state-metrics                  ClusterIP   10.108.229.49    <none>        8080/TCP    14s
service/kube-prometheus-stack-operator                            ClusterIP   10.111.241.52    <none>        443/TCP     14s
service/kube-prometheus-stack-prometheus                          ClusterIP   10.97.179.234    <none>        9090/TCP    14s

NAME
daemonset.apps/kube-prometheus-stack-prometheus-node-exporter     1          1               0          1          0          <none>   14s

NAME                                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kube-prometheus-stack-grafana                      1/1     1            1           14s
deployment.apps/kube-prometheus-stack-kube-state-metrics           1/1     1            1           14s
deployment.apps/kube-prometheus-stack-operator                     1/1     1            1           14s

NAME                                                          READY   AGE
statefulset.apps/prometheus-kube-prometheus-stack-prometheus  0/1     12s
```

You will typically see a Prometheus StatefulSet, node-exporter DaemonSet, and multiple deployments and services for Grafana and state metrics.

## Expose Prometheus and Grafana (NodePort for local access)

By default services are created as `ClusterIP`. For a local demo you can temporarily change Grafana and Prometheus to `NodePort` so they are accessible from your workstation browser.

Warning: exposing monitoring services with NodePort is convenient for demos but not recommended for production. Use Ingress, LoadBalancer, or proper authentication in production.

Exposing Prometheus and Grafana via NodePort is suitable for local testing only. For production, secure these endpoints via ingress/load balancer and enable authentication.

Edit the services:

```bash theme={null}
kubectl -n monitoring edit svc kube-prometheus-stack-prometheus
kubectl -n monitoring edit svc kube-prometheus-stack-grafana
```

Change `type: ClusterIP` to `type: NodePort` and save. Verify the services:

```console theme={null}
$ kubectl -n monitoring get svc
NAME                                            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
kube-prometheus-stack-grafana                   NodePort    10.99.232.155    <none>        80:31921/TCP        2m9s
kube-prometheus-stack-kube-state-metrics        ClusterIP   10.108.229.49    <none>        8080/TCP            2m9s
kube-prometheus-stack-operator                  ClusterIP   10.111.241.52    <none>        443/TCP             2m9s
kube-prometheus-stack-prometheus                NodePort    10.97.179.234    <none>        9090:30753/TCP      2m9s
kube-prometheus-stack-prometheus-node-exporter  ClusterIP   10.108.183.71    <none>        9100/TCP            2m9s
prometheus-operated                              ClusterIP   None             <none>        9090/TCP            2m7s
```

Open Prometheus at `http://localhost:<PROMETHEUS_NODEPORT>` (e.g., `http://localhost:30753`) and Grafana at `http://localhost:<GRAFANA_NODEPORT>` (e.g., `http://localhost:31921`) — replace ports with the NodePort values shown on your node.

## Grafana credentials

Grafana's admin password is stored in a Kubernetes Secret. Retrieve and decode it:

```bash theme={null}
kubectl -n monitoring get secret kube-prometheus-stack-grafana -o yaml
```

Example secret YAML (data fields are base64-encoded):

```yaml theme={null}
apiVersion: v1
data:
  admin-password: cHJvbS1vcGVyYXRvcg==
  admin-user: YWRtaW4=
  ldap-toml: ""
kind: Secret
metadata:
  name: kube-prometheus-stack-grafana
  namespace: monitoring
type: Opaque
```

Decode the password:

```bash theme={null}
echo 'cHJvbS1vcGVyYXRvcg==' | base64 -d
echo 'YWRtaW4=' | base64 -d
# => admin
```

Default credentials in this demo:

* Username: `admin`
* Password: `prom-operator`

Log in to Grafana and explore the preinstalled dashboards (Kubernetes overview, Node metrics, Prometheus overview). Flux-specific dashboards are not included by default.

## Prometheus UI — Targets for scraping

Prometheus will be running after the chart install, but you may not immediately see Flux-related targets. Open the Prometheus "Targets" page to view discovered targets and scraping status. If Flux components are not listed, add PodMonitor or ServiceMonitor resources that match Flux component labels so Prometheus can scrape them.

![A screenshot of the Prometheus "Targets" page showing multiple serviceMonitors and their endpoints with labels, last scrape times, scrape durations and status indicators. One entry shows a red "DOWN" error for a kube-controller-manager endpoint while others are marked "UP."](https://mintcdn.com/kodekloud-c4ac6d9a/AyhVWHpoMIYCWiei/images/GitOps-with-FluxCD/Monitoring-User-Interface/Demo-Install-Kube-Prometheus-Stack/prometheus-targets-kube-controller-manager-down.jpg?fit=max\&auto=format\&n=AyhVWHpoMIYCWiei\&q=85\&s=dc048366af09ee436dd78cb831af087c)

If you need to add scraping for Flux, create a PodMonitor or ServiceMonitor that targets the Flux component pods or services and matches the selectors defined in your HelmRelease (or add new selectors).

## Grafana dashboards — add Flux dashboards

Grafana displays dashboards installed by the chart. To visualize Flux metrics you must import or provision dashboards that target Flux metrics and the service/PodMonitor labels used for scraping.

![A dark-themed Grafana dashboard browse page showing a list of monitoring dashboards (Alertmanager, CoreDNS, Grafana Overview, various Kubernetes dashboards) with tags and search/filter controls. The screenshot displays the sidebar and a scrollable list of dashboard entries.](https://mintcdn.com/kodekloud-c4ac6d9a/AyhVWHpoMIYCWiei/images/GitOps-with-FluxCD/Monitoring-User-Interface/Demo-Install-Kube-Prometheus-Stack/dark-grafana-dashboard-browse.jpg?fit=max\&auto=format\&n=AyhVWHpoMIYCWiei\&q=85\&s=e1d932ec34f2ab4b8e36ad440f43d01c)

Next steps:

* Add a PodMonitor or ServiceMonitor to enable scraping of Flux components.
* Provision or import Grafana dashboards for Flux metrics.
* For production exposure, switch from NodePort to Ingress/LoadBalancer and secure access with authentication.

## References

* Flux documentation: [https://fluxcd.io/docs/](https://fluxcd.io/docs/)
* Prometheus Community Helm charts: [https://prometheus-community.github.io/helm-charts](https://prometheus-community.github.io/helm-charts)
* kube-prometheus-stack chart values and dashboards: [https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)

That’s all for this lesson — you now have kube-prometheus-stack installed via Flux and can extend scraping and dashboards to monitor Flux components.

> **[Watch Video](https://learn.kodekloud.com/user/courses/gitops-with-fluxcd/module/fb9e59dc-9dee-4532-92a8-553d0df1df27/lesson/8402737e-1205-4320-8bf5-453600a08706)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.