- Filter out unwanted discovered targets (for example, keep only
prodtargets). - Rename or transform labels (for example, store only the IP by removing the port).
- Drop unwanted discovery labels or metric labels.
relabel_configs(under ascrape_config): runs before scraping and only has access to labels provided by service discovery.metric_relabel_configs(under ascrape_config): runs after scraping and has access to all scraped metrics and their labels (including__name__).
relabel_configs:
__meta_ec2_tag_env can be dev or prod. We will use that to demonstrate filtering and transformation.
When you inspect targets in the Prometheus UI (for example up{instance="172.31.1.156:80", job="ec2"}) you typically see the final (normal) target labels such as instance and job. Discovery labels that begin with double underscores are metadata and are removed automatically at the end of relabeling unless you map them into regular target labels.
Note: Labels beginning with double underscores (e.g.,
__meta_*, __address__, __metrics_path__) are internal discovery labels and are removed automatically after relabeling unless you convert them into normal target labels (for example via replace or labelmap).relabel_configs (pre-scrape)
relabel_configs runs before scraping and operates only on discovery-provided labels. Common fields:
source_labels: list of labels whose values are concatenated and tested byregex(default concatenation separator:;).regex: regular expression matched against the concatenated value.action: what to do on match (keep,drop,replace,labelmap,labelkeep,labeldrop, etc.).target_labelandreplacement: used by actions such asreplace.
__meta_ec2_tag_env equals prod:
keep, any targets that do not match that rule will be excluded. Using only keep rules that match a subset results in all other targets being dropped.
If you use
keep, ensure you understand the resulting set of targets. For example, a keep rule that matches only prod will exclude non-prod targets.dev:
Combining multiple source labels
Whensource_labels has multiple entries, Prometheus concatenates their values with ; by default. Example: keep only targets where __meta_ec2_tag_env is dev and __meta_ec2_tag_team is marketing:
separator:
Extracting and saving a target label (replace)
Discovery labels like__address__="172.31.1.156:80" are metadata. To save the IP portion (without the port) as a normal target label ip, use a replace action with a capture group:
ip="172.31.1.156" available on all scraped metrics.
Dropping and keeping labels
- Drop a specific discovery label by name (e.g.,
__meta_ec2_owner_id) usinglabeldrop:
- Keep only specific labels (for example
instanceandjob) and drop everything else usinglabelkeep:
Mapping discovery labels into target labels (labelmap)
To convert many discovery labels (like__meta_ec2_*) into regular target labels at once, use labelmap. This rewrites label names using a regex; values stay the same.
Example: convert discovery labels starting with __meta_ec2_ into target labels prefixed with ec2_:
__meta_ec2_tag_env="dev" becomes ec2_tag_env="dev" (the original __meta_ec2_tag_env is removed).
metric_relabel_configs (post-scrape)
metric_relabel_configs runs after metrics have been scraped and can operate on metric names (available as the __name__ label) and metric labels. Use this for filtering, renaming, or transforming metrics based on their actual scraped content.
Drop a metric named http_errors_total:
http_errors_total to http_failures_total:
code from scraped metrics:
path="/cars" into endpoint="cars" (strip the leading slash and rename):
Common relabel actions — quick reference
When to use which config
This covers the most common relabeling operations: filtering targets, extracting and renaming labels, mapping discovery metadata to target labels, and post-scrape metric/label transformations. Use
relabel_configs for decisions based only on discovery metadata and metric_relabel_configs for operations that need access to scraped metric names and label values.