Prometheus Chart Overview

We installed the Prometheus Helm chart (kube-prometheus-stack). Let’s inspect what it created and where the important configuration and rule files live. This guide walks through the main resources, how they map to Prometheus components, and the exact commands to inspect and extract configuration for editing.

Quick summary (what the Helm chart installs)

StatefulSets: Prometheus server, Alertmanager
Deployments: Prometheus Operator, kube-state-metrics, Grafana
ReplicaSets: associated with the Deployments above
DaemonSet: node-exporter (one pod per node)
Services: ClusterIP services for internal access
Secrets / ConfigMaps: generated Prometheus config and rule files

The Helm chart also installs Grafana and configures it to use the Prometheus datasource out of the box. Grafana provides a graphical UI to visualize Prometheus metrics (installed automatically by the chart).

Inspect the cluster resources

Run:

kubectl get all

Example output (trimmed for readability):

NAME                                               READY   STATUS    RESTARTS   AGE
pod/alertmanager-kube-prometheus-alertmanager-0    2/2     Running   1          158m
pod/prometheus-grafana-d978dcd9-xhvgw              3/3     Running   0          158m
pod/prometheus-prometheus-kube-prometheus-0        2/2     Running   0          158m
pod/prometheus-prometheus-node-exporter-5ptg       1/1     Running   1          158m

NAME                                       TYPE        CLUSTER-IP       PORT(S)
service/prometheus-grafana                 ClusterIP   10.100.235.247   443/TCP
service/prometheus-kube-prometheus        ClusterIP   10.100.53.131    9093/TCP
service/prometheus-kube-state-metrics     ClusterIP   10.100.196.32    9090/TCP

NAME                                                       DESIRED   CURRENT   READY
daemonset.apps/prometheus-prometheus-node-exporter         2         2         2

NAME                                               READY   UP-TO-DATE   AVAILABLE
deployment.apps/prometheus-grafana                 1/1     1            1
deployment.apps/prometheus-kube-prometheus-operator 1/1     1            1

Resource summary table

Resource Type	Purpose	Example from chart
StatefulSet	Runs stateful components (persistent storage)	Prometheus server, Alertmanager
Deployment	Managed, scalable pods	`prometheus-grafana`, `prometheus-kube-prometheus-operator`
ReplicaSet	Deployment’s replica controller	`prometheus-grafana-<hash>`
DaemonSet	Runs a pod per node	`prometheus-prometheus-node-exporter`
Service	Internal networking to pods	`prometheus-prometheus` (ClusterIP)
ConfigMap / Secret	Prometheus config and rulefiles	`prometheus...-rulefiles-0`, `prometheus...` secret

StatefulSets (Prometheus & Alertmanager)

Start from the bottom of kubectl get all output: the StatefulSets are the Prometheus server and Alertmanager. List StatefulSets:

kubectl get statefulset

Example:

NAME                                                     READY   AGE
alertmanager-prometheus-kube-prometheus-alertmanager     1/1     168m
prometheus-prometheus-kube-prometheus-prometheus         1/1     168m

Describe the Prometheus StatefulSet to inspect containers, args, mounts, and volumes:

kubectl describe statefulset prometheus-prometheus-kube-prometheus-prometheus

You can pipe the full description to a file for easier viewing in an editor:

kubectl describe statefulset prometheus-prometheus-kube-prometheus-prometheus > prometheus-statefulset.txt

Key sections you’ll see in the StatefulSet description:

Prometheus container args (web UI, storage path, config file)
Init container: prometheus-config-reloader (creates initial config)
Sidecar/container: config reloader (watches config changes and triggers Prometheus reload)
Mounts and Volumes (where Prometheus YAML and rule files come from)

Example container args extracted from the StatefulSet (trimmed):

containers:
  - name: prometheus
    image: quay.io/prometheus/prometheus:v2.39.1
    ports:
      - containerPort: 9090
    args:
      - --web.console.templates=/etc/prometheus/consoles
      - --web.console.libraries=/etc/prometheus/console_libraries
      - --storage.tsdb.retention.time=10d
      - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
      - --web.external-url=http://prometheus-kube-prometheus-prometheus.default:9090

Init container (responsible for initial config):

initContainers:
  - name: init-config-reloader
    image: quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
    command:
      - /bin/prometheus-config-reloader
    args:
      - --watch-interval=0
      - --listen-address=:8080
      - --config-file=/etc/prometheus/config/prometheus.yaml.gz
      - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      - --watched-dir=/etc/prometheus/rules/

The StatefulSet mounts show where the config, rule files, and web config live:

mounts:
  - /etc/prometheus/config from config (rw)         # Prometheus YAML (from Secret)
  - /etc/prometheus/config_out from config-out (rw) # generated config
  - /etc/prometheus/rules/... from <rulefiles> (rw) # ConfigMap with rule files
  - /etc/prometheus/web_config/web-config.yaml from web-config (rw)
volumes:
  config:
    type: Secret
    secretName: prometheus-prometheus-kube-prometheus
  prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0:
    type: ConfigMap
    name: prometheus-prometheus-kube-prometheus-rulefiles-0
  web-config:
    type: Secret
    secretName: prometheus-prometheus-kube-prometheus-web-config

Deployments and ReplicaSets

Key deployments:

prometheus-kube-prometheus-operator — the Prometheus Operator container that manages Prometheus instances and their lifecycle.
prometheus-grafana — Grafana UI.
prometheus-kube-state-metrics — exposes Kubernetes object metrics (deployments, pods, etc).

Describe the operator deployment:

kubectl describe deployment prometheus-kube-prometheus-operator > operator-deployment.txt

Important flags you’ll see for the operator:

containers:
  - name: kube-prometheus-stack
    image: quay.io/prometheus-operator/prometheus-operator:v0.60.1
    args:
      - --kubelet-service=kube-system/prometheus-kube-prometheus-kubelet
      - --localhost=127.0.0.1
      - --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
      - --config-reloader-cpu-request=200m
      - --config-reloader-memory-request=50Mi
      - --thanos-default-base-image=quay.io/thanos/thanos:v0.28.1
      - --web.enable-tls=true
      - --web.cert-file=/cert/cert
      - --web.key-file=/cert/key
      - --web.listen-address=:10250

The operator primarily watches Kubernetes API objects (Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, etc.) and generates the Prometheus configuration, rule files, and Secrets/ConfigMaps used by the Prometheus StatefulSet.

DaemonSet — node-exporter

The chart creates a DaemonSet called something like prometheus-prometheus-node-exporter. DaemonSets ensure one node-exporter pod runs on each cluster node and expose host metrics (CPU, memory, filesystem) for Prometheus to scrape. Check nodes and daemonset status:

kubectl get nodes
kubectl get daemonset -n default prometheus-prometheus-node-exporter

Pods and Services

Pods section shows all pods created (Prometheus, Alertmanager, Grafana, Operator, kube-state-metrics, two node-exporters). Services created are typically ClusterIP (internal) by default:

kubectl get svc

If you want to access Prometheus or Grafana externally you can:

Create an Ingress
Change the Service type to LoadBalancer
Use kubectl port-forward or kubectl proxy for quick local access

Where the Prometheus configuration and rule files live

The Prometheus Operator generates the Prometheus configuration and stores it in a Secret (gzipped prometheus.yaml.gz) that is mounted into the Prometheus pod. Rule files are stored in a ConfigMap. To inspect the Secret containing the generated Prometheus config:

kubectl describe secret prometheus-prometheus-kube-prometheus > prometheus.yaml

Example Secret description:

Name:         prometheus-prometheus-kube-prometheus
Namespace:    default
Labels:       managed-by=prometheus-operator
Annotations:  generated: true
Type: Opaque

Data
====
prometheus.yaml.gz:  1723 bytes

To inspect rule files (ConfigMap with alerting/recording rules):

kubectl describe configmap prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 > rulefile.yaml

A rulefile excerpt will look like:

groups:
- name: kubernetes-system
  rules:
  - alert: KubeVersionMismatch
    annotations:
      description: There are {{ $value }} different semantic versions of Kubernetes components running.
      summary: Different semantic versions of Kubernetes components running.
    expr: count by (cluster) (count by (git_version, cluster) (label_replace(kubernetes_build_info{job~"kube-dns|coredns"},"git_version","$1","git_version", "(v[0-9]*.[0-9]*).*")) > 1
    for: 15m
    labels:
      severity: warning

If you prefer to view the raw generated file locally, dump the secret and extract the gzipped YAML:

kubectl get secret prometheus-prometheus-kube-prometheus -o jsonpath="{.data.prometheus\.yaml\.gz}" | base64 --decode > prometheus.yaml.gz
gunzip prometheus.yaml.gz
less prometheus.yaml

Do not manually edit the generated Secret or ConfigMap created by the operator. Changes will be overridden by the operator. Instead, use Kubernetes-native objects supported by the Prometheus Operator (ServiceMonitor, PodMonitor, PrometheusRule, Prometheus CR) to configure scraping and rules.

How the config reload workflow works

The Prometheus Operator generates prometheus.yaml and writes it into a Secret.
The Prometheus StatefulSet mounts that Secret into /etc/prometheus/config.
An init container (init-config-reloader) and a sidecar prometheus-config-reloader watch the config/rule mounts (ConfigMap / Secret / EmptyDir).
When a change is detected, the reloader triggers Prometheus’ /-/reload endpoint so Prometheus applies the updated configuration without manual restarts.

Reloader config example (from the StatefulSet):

config-reloader:
  image: quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
  command:
    - /bin/prometheus-config-reloader
  args:
    - --listen-address=:8080
    - --reload-url=http://127.0.0.1:9090/-/reload
    - --config-file=/etc/prometheus/config/prometheus.yaml.gz
    - --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    - --watched-dir=/etc/prometheus/rules/

PrometheusRule objects vs editing the generated files

Use Prometheus Operator resources (PrometheusRule) to add or modify alerting and recording rules. The operator will render these into the rule ConfigMap(s) and mount them into Prometheus. This is the recommended pattern — it avoids manual edits to the generated artifacts. Example: create a PrometheusRule manifest and apply with kubectl apply -f myrule.yaml. The operator merges it into the rule ConfigMap.

Quick commands cheat-sheet

# Show all cluster resources
kubectl get all

# List StatefulSets and inspect Prometheus StatefulSet
kubectl get statefulset
kubectl describe statefulset prometheus-prometheus-kube-prometheus-prometheus

# Dump generated Prometheus config secret to file
kubectl get secret prometheus-prometheus-kube-prometheus -o jsonpath="{.data.prometheus\.yaml\.gz}" | base64 --decode > prometheus.yaml.gz
gunzip prometheus.yaml.gz

# Dump rule ConfigMap to file
kubectl describe configmap prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 > rulefile.yaml

# Inspect operator deployment
kubectl describe deployment prometheus-kube-prometheus-operator > operator-deployment.txt

# View node-exporter daemonset
kubectl get daemonset prometheus-prometheus-node-exporter -o wide

Useful links and references

Prometheus Operator docs: https://github.com/prometheus-operator/prometheus-operator
Prometheus Operator runbooks and rules: https://prometheus-operator.dev/
Kubernetes kubectl docs: https://kubernetes.io/docs/reference/kubectl/overview/

If you’d like, I can:

Show an example PrometheusRule manifest and walk you through adding an alert, or
Provide a sample ServiceMonitor to scrape a custom app and demonstrate how the operator renders the config.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Prometheus Chart Overview

Quick summary (what the Helm chart installs)

Inspect the cluster resources

Resource summary table

StatefulSets (Prometheus & Alertmanager)

Deployments and ReplicaSets

DaemonSet — node-exporter

Pods and Services

Where the Prometheus configuration and rule files live

How the config reload workflow works

PrometheusRule objects vs editing the generated files

Quick commands cheat-sheet

Useful links and references

Watch Video

​Quick summary (what the Helm chart installs)

​Inspect the cluster resources

​Resource summary table

​StatefulSets (Prometheus & Alertmanager)

​Deployments and ReplicaSets

​DaemonSet — node-exporter

​Pods and Services

​Where the Prometheus configuration and rule files live

​How the config reload workflow works

​PrometheusRule objects vs editing the generated files

​Quick commands cheat-sheet

​Useful links and references

Watch Video

Quick summary (what the Helm chart installs)

Inspect the cluster resources

Resource summary table

StatefulSets (Prometheus & Alertmanager)

Deployments and ReplicaSets

DaemonSet — node-exporter

Pods and Services

Where the Prometheus configuration and rule files live

How the config reload workflow works

PrometheusRule objects vs editing the generated files

Quick commands cheat-sheet

Useful links and references