Prometheus Configuration

This article walks through the Prometheus configuration that the kube-prometheus-stack Helm chart installs by default and explains how Kubernetes Service Discovery (SD) is used to find scrape targets dynamically. You’ll see which SD role the chart uses, sample scrape configurations produced from ServiceMonitor resources, and the relabeling logic that filters endpoints to the intended targets.

Kubernetes service discovery roles

Prometheus supports multiple Kubernetes SD roles. Below is a concise reference of the most common roles and when to use them.

SD Role	What it discovers	Typical use
`node`	All cluster nodes and node-related metadata	Scrape node-level exporters or node metadata
`service`	Kubernetes Services and their ports	Discover Services directly (less common for metrics endpoints)
`pod`	Pods directly	Scrape pods when you want pod-level discovery without relying on Service endpoints
`endpoints`	Endpoints associated with Services (IP:port pairs)	Most flexible — discovers pod, service, and node endpoints. Works well with relabeling to pick exact targets

The kube-prometheus-stack Helm chart uses role: endpoints by default. This gives Prometheus access to all endpoints, and relabeling rules are applied so ServiceMonitor-derived jobs only scrape the intended endpoints.

Representative global configuration

Below is a cleaned excerpt of the default Prometheus configuration produced by the Helm chart. Note the kubernetes_sd_configs block is set to role: endpoints, and alerting is wired to Alertmanager discovered via ServiceMonitor.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s

external_labels:
  prometheus: default/prometheus-kube-prometheus-prometheus
  prometheus_replica: prometheus-prometheus-kube-prometheus-prometheus-0

alerting:
  alert_relabel_configs:
    - separator: ;
      regex: prometheus_replica
      replacement: $1
      action: labeldrop

alertmanagers:
  - follow_redirects: true
    enable_http2: true
    scheme: http
    path_prefix: /
    timeout: 10s
    api_version: v2
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        separator: ;
        regex: prometheus-kube-prometheus-alertmanager
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        separator: ;
        regex: http-web
        replacement: $1
        action: keep

kubernetes_sd_configs:
  - role: endpoints
    kubeconfig_file: ""
    follow_redirects: true
    enable_http2: true
    namespaces:
      own_namespace: false
      names:
        - default

rule_files:
  - /etc/prometheus/rules/prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0/*.yaml

Key points:

role: endpoints is used so Prometheus discovers IP:port pairs for every Service.
Alertmanager is discovered via labels and endpoint port selection, then referenced in alertmanagers.
Rule files are mounted into the rule_files section.

How ServiceMonitor resources become scrape jobs

Helm (via the Prometheus Operator flow) converts ServiceMonitor objects into Prometheus scrape_configs. Each ServiceMonitor results in a scrape job configured to search the endpoints SD stream and then apply relabeling to select only the endpoints that match the ServiceMonitor’s labels and port name. Below is a representative scrape_config for the Alertmanager ServiceMonitor and the relabeling used to keep only the intended Alertmanager endpoint.

# Example scrape_config for Alertmanager ServiceMonitor
scrape_configs:
  - job_name: serviceMonitor/default/prometheus-kube-prometheus-alertmanager/0
    honor_timestamps: true
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    follow_redirects: true
    enable_http2: true
    relabel_configs:
      - source_labels: [job]
        separator: ;
        regex: (.*)
        target_label: __tmp_prometheus_job_name
        replacement: $1
        action: replace

      - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
        separator: ;
        regex: (kube-prometheus-stack-alertmanager);true
        replacement: $1
        action: keep

      - source_labels: [__meta_kubernetes_service_label_release, __meta_kubernetes_service_labelpresent_release]
        separator: ;
        regex: (prometheus);true
        replacement: $1
        action: keep

      - source_labels: [__meta_kubernetes_service_label_self_monitor, __meta_kubernetes_service_labelpresent_self_monitor]
        separator: ;
        regex: (true);true
        replacement: $1
        action: keep

      - source_labels: [__meta_kubernetes_endpoint_port_name]
        separator: ;
        regex: http-web
        replacement: $1
        action: keep

      - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
        separator: ;
        regex: Node;(.*)
        target_label: node
        replacement: $1
        action: replace

      - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
        separator: ;
        regex: Pod;(.*)
        target_label: pod
        replacement: $1
        action: replace

      - source_labels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        replacement: $1
        action: replace

Relabel rules explained:

Copy the job label into a temporary label for bookkeeping (__tmp_prometheus_job_name).
The keep rules ensure the discovered endpoint belongs to a Service with specific labels:
- app=kube-prometheus-stack-alertmanager
- release=prometheus
- self-monitor=true These combined label checks ensure only the Alertmanager Service is kept for this job.
__meta_kubernetes_endpoint_port_name: http-web ensures the endpoint port is the intended HTTP port.
Node/Pod relabels extract the target name into node/pod labels for easier identification.
The namespace is recorded on the target label set.

Prometheus uses the same SD block and pattern for all ServiceMonitor-derived jobs (kubelet, apiserver, kube-controller-manager, kube-proxy, kube-state-metrics, node-exporter, CoreDNS, etc.). The relabeling is what narrows the endpoints stream down to the exact endpoints each job should scrape.

Example: API server ServiceMonitor (HTTPS + service account auth)

Some components (like the API server) require TLS and token-based authorization. The ServiceMonitor-derived scrape job includes authorization and tls_config to securely access /metrics over HTTPS:

# Example scrape_config for apiserver ServiceMonitor
- job_name: serviceMonitor/default/prometheus-kube-prometheus-apiserver/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  authorization:
    type: Bearer
    credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    server_name: kubernetes
    insecure_skip_verify: false
  follow_redirects: true
  enable_http2: true
  relabel_configs:
    - source_labels: [job]
      separator: ;
      regex: (.*)
      target_label: __tmp_prometheus_job_name
      replacement: $1
      action: replace

    - source_labels: [__meta_kubernetes_service_label_component, __meta_kubernetes_service_labelpresent_component]
      separator: ;
      regex: (apiserver);true
      replacement: $1
      action: keep

    - source_labels: [__meta_kubernetes_service_label_provider, __meta_kubernetes_service_labelpresent_provider]
      separator: ;
      regex: (kubernetes);true
      replacement: $1
      action: keep

    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: https
      replacement: $1
      action: keep

    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: $1
      action: replace

    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: $1
      action: replace

Inspecting Service Discovery and active targets

You can verify how the SD and relabel rules behave in the Prometheus UI:

Status → Service Discovery shows the raw discovered endpoints from Kubernetes.
Status → Targets shows the final active targets Prometheus is scraping (with their UP/Down state, last scrape, and scrape duration).

When configured by the Helm chart and Prometheus Operator, most of these scrape jobs are created automatically from ServiceMonitor resources — you rarely need to write relabel rules from scratch unless you require custom scraping behavior.

A screenshot of the Prometheus "Targets" page showing monitored service endpoints with their UP state, labels, last scrape times, and scrape durations. Multiple serviceMonitor entries (alertmanager, apiserver, coredns, kube-proxy) with IP/metrics endpoints are listed.

Summary

The kube-prometheus-stack Helm chart configures Prometheus to use Kubernetes endpoints SD (role: endpoints) for broad discovery.
ServiceMonitor resources are translated into scrape_configs that filter the endpoints stream using relabel_configs, ensuring each job scrapes only its intended targets.
Use the Prometheus UI (Service Discovery and Targets) to inspect discovered endpoints and active scrapes.
Custom relabeling is available when you need to alter defaults or implement more selective scraping.

Links and references

Prometheus Service Discovery: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
Kubernetes Service Discovery in Prometheus: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config
Prometheus Operator / kube-prometheus-stack: https://github.com/prometheus-operator/kube-prometheus
Kubernetes documentation: https://kubernetes.io/docs/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Prometheus Configuration

Kubernetes service discovery roles

Representative global configuration

How ServiceMonitor resources become scrape jobs

Example: API server ServiceMonitor (HTTPS + service account auth)

Inspecting Service Discovery and active targets

Summary

Links and references

Watch Video

​Kubernetes service discovery roles

​Representative global configuration

​How ServiceMonitor resources become scrape jobs

​Example: API server ServiceMonitor (HTTPS + service account auth)

​Inspecting Service Discovery and active targets

​Summary

​Links and references

Watch Video

Kubernetes service discovery roles

Representative global configuration

How ServiceMonitor resources become scrape jobs

Example: API server ServiceMonitor (HTTPS + service account auth)

Inspecting Service Discovery and active targets

Summary

Links and references