Service Monitors

In this lesson we’ll learn how to add scrape targets to Prometheus by using the ServiceMonitor custom resource provided by the Prometheus Operator. ServiceMonitors let you declare Prometheus scrape configuration in Kubernetes objects instead of editing Prometheus config files directly. The Prometheus Operator installs several Custom Resource Definitions (CRDs) that provide higher-level abstractions for deploying and configuring Prometheus components (Prometheus instances, Alertmanagers, ServiceMonitors, PodMonitors, etc.).

The image contains text explaining that the Prometheus operator includes several custom resource definitions that provide high-level abstraction for deploying and configuring Prometheus.

List the installed CRDs:

kubectl get crd

Example truncated output:

NAME                                           CREATED AT
alertmanagerconfigs.monitoring.coreos.com      2022-11-18T01:18:55Z
alertmanagers.monitoring.coreos.com            2022-11-18T01:18:55Z
podmonitors.monitoring.coreos.com              2022-11-18T01:18:56Z
prometheuses.monitoring.coreos.com             2022-11-18T01:18:56Z
prometheusrules.monitoring.coreos.com          2022-11-18T01:18:56Z
servicemonitors.monitoring.coreos.com          2022-11-18T01:18:57Z
thanosrulers.monitoring.coreos.com             2022-11-18T01:18:57Z

Key CRDs and their use cases:

Resource Type	Use Case	Example
Alertmanager	Manage Alertmanager instances	`alertmanagers.monitoring.coreos.com`
Prometheus	Manage Prometheus instances	`prometheuses.monitoring.coreos.com`
ServiceMonitor	Declare service-level scrape targets	`servicemonitors.monitoring.coreos.com`
PodMonitor	Declare pod-level scrape targets	`podmonitors.monitoring.coreos.com`
PrometheusRule	Define alerting and recording rules	`prometheusrules.monitoring.coreos.com`

What is a ServiceMonitor?

A ServiceMonitor is a CRD that defines a set of targets for Prometheus to monitor and scrape.
It expresses scrape configuration (endpoints, path, interval, job label) in Kubernetes objects so you don’t edit Prometheus config files directly.
The Prometheus Operator translates ServiceMonitor resources into Prometheus scrape_configs.

The image explains that service monitors define targets for Prometheus and provide a declarative Kubernetes syntax to avoid direct configuration changes.

Example flow

Deploy your application (Deployment + Service).
Create a ServiceMonitor that selects the Service (via labels) and specifies endpoints (port, path, interval).
The Prometheus Operator discovers the ServiceMonitor and adds targets to Prometheus’ scrape configuration.

Example manifests

Service exposes the application on port 3000 with port name web and labels job: node-api and app: api.
ServiceMonitor selects the Service by matchLabels, sets jobLabel: job (so the Prometheus job name is taken from the Service job label), and configures endpoints (scrape interval and metrics path).

Service manifest (api-service.yaml):

# api-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: api-service
  labels:
    job: node-api
    app: api
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
    - name: web
      protocol: TCP
      port: 3000
      targetPort: 3000

ServiceMonitor manifest (api-service-monitor.yaml):

# api-service-monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-service-monitor
  labels:
    release: prometheus
    app: prometheus
spec:
  jobLabel: job
  endpoints:
    - interval: 30s
      port: web
      path: /swagger-stats/metrics
  selector:
    matchLabels:
      app: api

Prometheus discovers ServiceMonitors based on the Prometheus custom resource configuration. Check the Prometheus CR’s serviceMonitorSelector and serviceMonitorNamespaceSelector to know which ServiceMonitors (labels and namespaces) your Prometheus instance will use. When using the kube-prometheus-stack Helm chart, ServiceMonitors often need the label release: prometheus.

Verify Prometheus CR configuration Inspect the Prometheus custom resource to see how it discovers ServiceMonitors:

kubectl get prometheuses.monitoring.coreos.com -o yaml

Look for:

serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
  matchLabels:
    release: prometheus

If the Prometheus CR selects ServiceMonitors with release: prometheus, ensure your ServiceMonitor includes that label. Apply the manifests Apply the Service and ServiceMonitor (together or separately):

kubectl apply -f api-deploy.yaml

Example output:

deployment.apps/api-deployment unchanged
service/api-service unchanged
servicemonitor.monitoring.coreos.com/api-service-monitor created

Check ServiceMonitors:

kubectl get servicemonitor

Verify targets in Prometheus Open the Prometheus UI and navigate to Status → Targets. You should see a job corresponding to your ServiceMonitor, for example: serviceMonitor/default/api-service-monitor/0. A Deployment with multiple replicas will show multiple endpoints (one per pod).

The image shows a Prometheus monitoring dashboard displaying the status of several service endpoints, with details such as their state, labels, last scrape time, and scrape duration.

Querying metrics Confirm metrics ingestion by querying Prometheus for metrics associated with your job label (job="node-api"). Example metric lines returned from Prometheus:

api_request_duration_milliseconds_bucket{code="304", container="api", endpoint="web", instance="192.168.45.233:3000", job="node-api", le="25", method="GET", namespace="default", path="/comments", pod="api-deployment-85cb98d64f-pk7pz", service="api-service"}  ...
api_request_duration_milliseconds_bucket{code="404", container="api", endpoint="web", instance="192.168.45.233:3000", job="node-api", le="5", method="GET", namespace="default", path="/messages", pod="api-deployment-85cb98d64f-pk7pz", service="api-service"}  ...

A useful quick check in Prometheus: up{job="node-api"} — a value of 1 indicates a healthy scrape target. Inspect the generated Prometheus configuration The Prometheus Operator converts ServiceMonitors into scrape_configs. In the Prometheus UI → Status → Configuration, search for the job name (e.g., serviceMonitor/default/api-service-monitor/0) to view the generated config. This shows how your ServiceMonitor settings translate into Prometheus scrape settings and relabel rules.

The image shows a Prometheus interface displaying a list of metrics related to API performance, with options for viewing configuration and various statistics.

Example generated snippet:

scrape_configs:
- job_name: serviceMonitor/default/api-service-monitor/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 1m
  metrics_path: /swagger-stats/metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
    - source_labels: [job]
      regex: (.*)
      target_label: __tmp_prometheus_job_name
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
      separator: ;
      regex: (api);true
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: web
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Node;(.*)
      target_label: node
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
      separator: ;
      regex: Pod;(.*)
      target_label: pod
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_namespace]
      regex: (.*)
      target_label: namespace
      replacement: $1
      action: replace
    - source_labels: [__meta_kubernetes_service_name]
      regex: (.*)
      target_label: service
      replacement: $1
      action: replace

All configuration declared in the ServiceMonitor (port name, path, interval, jobLabel and selector) will be reflected in the generated Prometheus scrape config and relabel rules.

Namespaces and labels matter. Prometheus discovers ServiceMonitors according to the Prometheus CR’s selectors—if your ServiceMonitor is in a different namespace or lacks the expected label, Prometheus will not pick it up. Also ensure required RBAC permissions are in place for Prometheus to read ServiceMonitor resources in the target namespaces.

Summary

ServiceMonitors provide a declarative method to register Prometheus scrape targets in Kubernetes.
Make sure the Prometheus CR’s serviceMonitorSelector and serviceMonitorNamespaceSelector include the labels and namespaces of your ServiceMonitors.
Create Services with meaningful labels and named ports, then create ServiceMonitors that select those Services and define scrape parameters (path, port, interval, jobLabel).
Verify in Prometheus UI → Status → Targets that the job and endpoints appear and metrics are being scraped.

Links and references

Prometheus Operator: https://prometheus-operator.dev
kube-prometheus-stack Helm chart: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Prometheus documentation: https://prometheus.io/docs/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Watch Video