Prometheus Chart Overview

You’ve installed the Helm chart for the kube-prometheus stack. The next step is to inspect the Kubernetes resources the chart created so you can understand how Prometheus, Alertmanager, Grafana, the operator, and node exporters are laid out inside the cluster.

List resources created by the Helm chart

Run this to list pods, services, daemonsets, deployments, statefulsets, and replicasets in the current namespace:

kubectl get all

Example condensed output (trimmed for clarity):

pod/alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running     1 (158m ago)   158m
pod/prometheus-grafana-d978ddcb9-xhvgw                      3/3     Running     0               158m
pod/prometheus-kube-prometheus-operator-66c5f6b89b-xxsdt     1/1     Running     0               158m
pod/prometheus-kube-state-metrics-649f8795d4-fltnq          1/1     Running     0               158m
pod/prometheus-prometheus-kube-prometheus-prometheus-0      2/2     Running     0               158m
pod/prometheus-prometheus-node-exporter-5pst9                1/1     Running     0               158m
pod/prometheus-prometheus-node-exporter-xntb2                1/1     Running     0               158m

service/alertmanager-operated                  ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP
service/prometheus-grafana                     ClusterIP   10.100.235.247   <none>        80/TCP
service/prometheus-kube-prometheus-alertmanager ClusterIP  10.100.53.114    <none>        9093/TCP
service/prometheus-kube-prometheus-operator    ClusterIP   10.100.196.32    <none>        443/TCP
service/prometheus-kube-prometheus-prometheus  ClusterIP   10.100.54.169    <none>        9090/TCP
service/prometheus-kube-state-metrics          ClusterIP   10.100.133.149   <none>        8080/TCP
service/prometheus-prometheus-node-exporter    ClusterIP   10.100.248.61    <none>        9100/TCP

daemonset.apps/prometheus-prometheus-node-exporter             2         2         2       2            2
deployment.apps/prometheus-grafana                             1/1       1            1
deployment.apps/prometheus-kube-prometheus-operator            1/1       1            1
deployment.apps/prometheus-kube-state-metrics                  1/1       1            1
replicaset.apps/prometheus-grafana-d978ddcb9                   1         1         1
replicaset.apps/prometheus-kube-prometheus-operator-66c5f6b89b 1         1         1
replicaset.apps/prometheus-kube-state-metrics-649f8795d4       1         1         1

statefulset.apps/alertmanager-prometheus-kube-prometheus-alertmanager    1/1
statefulset.apps/prometheus-prometheus-kube-prometheus-prometheus       1/1

High-level mapping of resources

Resource Type	Purpose / What it runs	Example resources from the chart
StatefulSet	Long-lived, stable identities and persistent storage for core services	Prometheus server, Alertmanager
Deployment	Managed, scalable workloads	Grafana, kube-prometheus-operator, kube-state-metrics
ReplicaSet	Backing controller for Deployments	ReplicaSets for Grafana and the operator
DaemonSet	One pod per node to expose host metrics	prometheus-node-exporter (port 9100)
Service	Internal networking for components	`prometheus-kube-prometheus-prometheus` (ClusterIP), `prometheus-grafana` (ClusterIP)

The Pods list should show Prometheus, Alertmanager, Grafana, the operator, kube-state-metrics, and one node-exporter pod per node.
The Services created by the chart are primarily type ClusterIP (reachable only inside the cluster by default).

If you need to access Prometheus or Grafana from outside the cluster, you must either:

Change a service to LoadBalancer, or
Add an Ingress that routes to the service, or
Use kubectl port-forward as a quick, local option.

Inspecting the Prometheus StatefulSet

To review the Prometheus StatefulSet and see how the Prometheus process and its sidecars are configured:

List StatefulSets:

kubectl get statefulset

Save a describe to a file for easier review:

kubectl describe statefulset prometheus-prometheus-kube-prometheus-prometheus > prometheus.yaml
# then open prometheus.yaml in your editor (e.g., code prometheus.yaml)

Key excerpts from the StatefulSet describe output: Prometheus main container arguments and flags:

Args:
  --web.console.templates=/etc/prometheus/consoles
  --web.console.libraries=/etc/prometheus/console_libraries
  --storage.tsdb.retention.time=10d
  --config.file=/etc/prometheus/config_out/prometheus.env.yaml
  --storage.tsdb.path=/prometheus
  --web.enable-lifecycle
  --web.external-url=http://prometheus-kube-prometheus-prometheus.default:9090
  --web.route-prefix=/
  --storage.tsdb.wal-compression
  --web.config.file=/etc/prometheus/web_config/web-config.yaml

Liveness:   http-get http://:http-web/-/healthy delay=0s timeout=3s period=5s #success=1 #failure=6
Readiness:  http-get http://:http-web/-/ready delay=0s timeout=3s period=5s #success=1 #failure=3
Startup:    http-get http://:http-web/-/ready delay=0s timeout=3s period=15s #success=1 #failure=60

Mounts:
  /etc/prometheus/certs from tls-assets (ro)
  /etc/prometheus/config_out from config-out (ro)
  /etc/prometheus/rules/... from prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 (rw)
  /etc/prometheus/web_config/web-config.yaml from web-config (ro,path="web-config.yaml")
  /prometheus from prometheus-prometheus-kube-prometheus-prometheus-db (rw)

Init container (generates initial config files):

Init Containers:
  init-config-reloader:
    Image: quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
    Port: 8080/TCP
    Command:
      /bin/prometheus-config-reloader
    Args:
      --watch-interval=0
      --listen-address=:8080
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
      --watched-dir=/etc/prometheus/rules/
      prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
    Limits:
      cpu: 200m
      memory: 50Mi
    Requests:
      cpu: 200m

Config reloader sidecar (keeps Prometheus config in sync and triggers reloads):

config-reloader:
  Image: quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
  Port: 8080/TCP
  Command:
    /bin/prometheus-config-reloader
  Args:
    --listen-address=:8080
    --reload-url=http://127.0.0.1:9090/-/reload
    --config-file=/etc/prometheus/config/prometheus.yaml.gz
    --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    --watched-dir=/etc/prometheus/rules/
    prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0
  Limits:
    cpu: 200m
    memory: 50Mi
  Requests:
    cpu: 200m
    memory: 50Mi
  Environment:
    POD_NAME:    (v1:metadata.name)
    SHARD:       0
  Mounts:
    /etc/prometheus/config from config (rw)
    /etc/prometheus/config_out from config-out (rw)
    /etc/prometheus/rules/... from prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 (rw)

Where Prometheus config and rules come from

The Prometheus configuration is generated by the Prometheus Operator and injected into the pod via a Secret mounted at /etc/prometheus/config.
Alerting and recording rules are rendered into ConfigMaps and mounted under /etc/prometheus/rules/....

Inspect the Secret that contains the generated Prometheus YAML:

kubectl describe secret prometheus-prometheus-kube-prometheus-prometheus

Example output (Secret contains a gzipped prometheus.yaml):

Name:        prometheus-prometheus-kube-prometheus-prometheus
Namespace:   default
Labels:      managed-by=prometheus-operator
Annotations: generated: true
Type:        Opaque

Data
====
prometheus.yaml.gz:  1723 bytes

Inspect the ConfigMap containing rule files (save to file for easier reading):

kubectl describe configmap prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 > rulefile.yaml
# open rulefile.yaml in your editor to read the full contents

Example rule group excerpt (from the generated ConfigMap):

groups:
- name: kubernetes-system
  rules:
  - alert: KubeVersionMismatch
    annotations:
      description: There are {{ $value }} different semantic versions of Kubernetes components running.
      runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeversionmismatch
      summary: Different semantic versions of Kubernetes components running.
    expr: >-
      count by (cluster) (
        count by (git_version, cluster) (
          label_replace(
            kubernetes_build_info{job!~"kube-dns|coredns"},
            "git_version","$1","git_version","(v[0-9]*.[0-9]*.*)"
          )
        )
      ) > 1
    for: 15m
    labels:
      severity: warning
  - alert: KubeClientErrors
    annotations:
      description: Kubernetes API server client '{{ $labels.job }}/{{ $labels.instance }}' is experiencing {{ $value | humanizePercentage }} errors.
      runbook_url: https://runbooks.prometheus-operator.dev/runbooks/kubernetes/kubeclienterrors
      summary: Kubernetes API server client is experiencing errors.
    expr: >-
      (
        sum(rate(rest_client_requests_total{code=~"5.."}[5m])) by (cluster, instance, job, namespace)
        /
        sum(rate(rest_client_requests_total[5m])) by (cluster, instance, job, namespace)
      ) > 0.01
    for: 15m
    labels:
      severity: warning

Note: The Prometheus Operator is responsible for generating these Secrets and ConfigMaps. Do not edit the generated Secrets or ConfigMaps directly. Instead, modify scraping targets and alerts declaratively using the Operator CRDs (for example, ServiceMonitor, PodMonitor, PrometheusRule).

Inspecting the Prometheus Operator deployment

List deployments in the namespace:

kubectl get deployment

Save the operator describe to a file for review:

kubectl describe deployment prometheus-kube-prometheus-operator > operator.yaml
# open operator.yaml to review the full spec

Key operator container args, TLS and mounts:

kube-prometheus-stack:
  Image: quay.io/prometheus-operator/prometheus-operator:v0.60.1
  Args:
    --kubelet-service=kube-system/prometheus-kube-prometheus-kubelet
    --localhost=127.0.0.1
    --prometheus-config-reloader=quay.io/prometheus-operator/prometheus-config-reloader:v0.60.1
    --config-reloader-cpu-request=200m
    --config-reloader-cpu-limit=200m
    --config-reloader-memory-request=50Mi
    --config-reloader-memory-limit=50Mi
    --thanos-default-base-image=quay.io/thanos/thanos:v0.28.1
    --web.enable-tls=true
    --web.cert-file=/cert/cert
    --web.key-file=/cert/key
    --web.listen-address=:10250
    --web.tls-min-version=VersionTLS13
  Mounts:
    /cert from tls-secret (ro)

Volumes:
  tls-secret:
    Type: Secret
    SecretName: prometheus-kube-prometheus-admission

Operator responsibilities:

Reconcile Prometheus, Alertmanager, and related resources based on higher-level CRDs.
Generate the Prometheus YAML configuration (stored in Secrets) and rule ConfigMaps from CRD manifests.
Manage configuration reloads and restarts (via config-reloader sidecars).
Provide extensibility points such as ServiceMonitor, PodMonitor, and PrometheusRule CRDs for declarative configuration.

Best practices and wrap-up

Prefer using the Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule) to add scrape targets and alerting/recording rules.
Do not edit generated Secrets or ConfigMaps directly; changes will be overwritten by the Operator.
To access Prometheus or Grafana externally, expose services via LoadBalancer, set up an Ingress, or use kubectl port-forward for quick local access.
Monitor the prometheus StatefulSet and the prometheus-operator Deployment for health and restart events.

Links and references

Prometheus Operator / kube-prometheus documentation: https://github.com/prometheus-operator/kube-prometheus
Prometheus Operator CRDs and docs: https://prometheus-operator.dev/
Prometheus project: https://prometheus.io/
Kubernetes documentation: https://kubernetes.io/docs/
Helm charts / Artifact Hub: https://artifacthub.io/

If you want, I can provide example ServiceMonitor and PrometheusRule YAML to add a basic scrape target and an alert rule using the Operator CRDs.

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Prometheus Chart Overview

List resources created by the Helm chart

High-level mapping of resources

Inspecting the Prometheus StatefulSet

Where Prometheus config and rules come from

Inspecting the Prometheus Operator deployment

Best practices and wrap-up

Links and references

Watch Video

​List resources created by the Helm chart

​High-level mapping of resources

​Inspecting the Prometheus StatefulSet

​Where Prometheus config and rules come from

​Inspecting the Prometheus Operator deployment

​Best practices and wrap-up

​Links and references

Watch Video

List resources created by the Helm chart

High-level mapping of resources

Inspecting the Prometheus StatefulSet

Where Prometheus config and rules come from

Inspecting the Prometheus Operator deployment

Best practices and wrap-up

Links and references