Service Monitors

In this lesson we’ll add targets to Prometheus using the Kubernetes-native ServiceMonitor resource provided by the Prometheus Operator. First, a quick note about Custom Resource Definitions (CRDs): the Prometheus Operator installs multiple CRDs that expose high-level Kubernetes abstractions for deploying and configuring Prometheus and its related resources.

A dark presentation slide titled "CRDs" showing text that explains the Prometheus operator includes several custom resource definitions to provide a high-level abstraction for deploying and configuring Prometheus. The phrase "custom resource definitions" is highlighted.

To list all CRDs installed in your cluster:

$ kubectl get crd
NAME                                         CREATED AT
alertmanagerconfigs.monitoring.coreos.com    2022-11-18T01:18:55Z
alertmanagers.monitoring.coreos.com          2022-11-18T01:18:55Z
eniconfigs.crd.k8s.amazonaws.com             2022-11-18T01:04:11Z
podmonitors.monitoring.coreos.com            2022-11-18T01:18:56Z
probes.monitoring.coreos.com                 2022-11-18T01:18:56Z
prometheuses.monitoring.coreos.com           2022-11-18T01:18:56Z
prometheusrules.monitoring.coreos.com        2022-11-18T01:18:56Z
securitygrouppolicies.vpcresources.k8s.aws   2022-11-18T01:04:14Z
servicemonitors.monitoring.coreos.com        2022-11-18T01:18:57Z
thanosrulers.monitoring.coreos.com           2022-11-18T01:18:57Z

Two CRDs to focus on for adding Prometheus scrape targets:

CRD (resource)	Purpose
`prometheuses.monitoring.coreos.com`	Defines Prometheus instances managed by the Prometheus Operator (controls discovery & scrape behavior).
`servicemonitors.monitoring.coreos.com`	Declares Kubernetes Services for Prometheus to scrape; translated into Prometheus `scrape_configs` by the operator.

A presentation slide titled "Service Monitors" explaining that service monitors define targets for Prometheus to monitor and scrape. It notes they let you avoid editing Prometheus configs by using a declarative Kubernetes syntax, with "Service monitors" and "declarative" highlighted.

What is a ServiceMonitor?

A ServiceMonitor is a CRD that declares which Kubernetes Services Prometheus should scrape.
It enables declarative discovery: create Service and ServiceMonitor resources and the Prometheus Operator converts them into Prometheus scrape_configs, avoiding manual edits to Prometheus config files.
See the ServiceMonitor API docs for full fields and examples: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#servicemonitor

How ServiceMonitor and Service relate

The ServiceMonitor selects Kubernetes Services via selector.matchLabels.
Endpoints in the ServiceMonitor refer to named ports on the Service (port: <name>) and can set path, interval, scheme, etc.
jobLabel (optional) maps a Service label to the Prometheus job name.

Example: Service + ServiceMonitor (combined YAML)

apiVersion: v1
kind: Service
metadata:
  name: api-service
  labels:
    job: node-api
    app: api
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
    - name: web
      protocol: TCP
      port: 3000
      targetPort: 3000
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: api-service-monitor
  labels:
    release: prometheus    # must match Prometheus' serviceMonitorSelector
    app: prometheus
spec:
  jobLabel: job
  endpoints:
    - interval: 30s
      port: web
      path: '/swagger-stats/metrics'
  selector:
    matchLabels:
      app: api

Notes on the example:

The Service has labels job: node-api and app: api.
The Service exposes a named port web on port 3000.
The ServiceMonitor selects Services with app: api, scrapes the web port at /swagger-stats/metrics every 30s.
jobLabel: job tells Prometheus to use the Service label job (value: node-api) as the Prometheus job name.

Ensure the ServiceMonitor includes a label that matches the Prometheus instance’s serviceMonitorSelector (commonly release: prometheus). If the ServiceMonitor does not match that selector, the Prometheus instance will not discover it automatically.

Prometheus instance configuration

The Prometheus CRD configures which ServiceMonitors it should discover through serviceMonitorSelector. Check your Prometheus custom resource to confirm the selector:

# output of: kubectl get prometheuses.monitoring.coreos.com -o yaml
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
  matchLabels:
    release: prometheus

If your Prometheus has a selector like release: prometheus, make sure your ServiceMonitor includes metadata.labels.release: prometheus. Apply the resources

Add the Service and ServiceMonitor YAML (for example, append to api-deploy.yaml) and apply:

$ kubectl apply -f api-deploy.yaml
deployment.apps/api-deployment unchanged
service/api-service unchanged
servicemonitor.monitoring.coreos.com/api-service-monitor created

Verify the ServiceMonitor exists:

$ kubectl get servicemonitor
NAME                     AGE
api-service-monitor      19s
prometheus-grafana       16h
prometheus-kube-...      16h
...                      ...

Prometheus discovery and verification

In the Prometheus UI go to Status → Targets. You should see a job that corresponds to your ServiceMonitor. In the example above the job will be node-api, since the Service label job: node-api is used.
If the Deployment has multiple replicas, each pod appears as a separate target.
To confirm metrics ingestion, run queries in Prometheus. Example (truncated):

api_request_duration_milliseconds_bucket{code="304", container="api", endpoint="web", instance="192.168.45.233:3000", job="node-api", le="25", method="GET", namespace="default", path="/comments", pod="api-deployment-85cb98d64f-pk7pz", service="api-service"} 25
api_request_duration_milliseconds_bucket{code="304", container="api", endpoint="web", instance="192.168.45.233:3000", job="node-api", le="250", method="GET", namespace="default", path="/comments", pod="api-deployment-85cb98d64f-pk7pz", service="api-service"} 25
...

Generated scrape config

The Prometheus Operator translates each ServiceMonitor into a Prometheus scrape_config. An excerpt of the generated configuration for this ServiceMonitor:

- job_name: serviceMonitor/default/api-service-monitor/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /swagger-stats/metrics
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (api);true
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: web
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Node;(.*)
    target_label: node
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
    separator: ;
    regex: Pod;(.*)
    target_label: pod
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    action: replace

Summary

ServiceMonitors let you declaratively add Prometheus scrape targets using Kubernetes resources instead of editing Prometheus config files directly.
Key checks when using ServiceMonitors:
- The ServiceMonitor selector.matchLabels must select the intended Service.
- The ServiceMonitor endpoints must reference the Service’s named ports.
- The ServiceMonitor must carry the label(s) (e.g. release: prometheus) expected by your Prometheus instance’s serviceMonitorSelector.

Links and references

Prometheus Operator — ServiceMonitor API: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#servicemonitor
Prometheus Operator GitHub: https://github.com/prometheus-operator/prometheus-operator
Prometheus scrape_config docs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
Kubernetes CRDs docs: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

Watch Video