AWS

When running Prometheus in cloud environments (especially with auto-scaling), infrastructure is dynamic: instances are created and terminated frequently. Prometheus EC2 service discovery automates target discovery so Prometheus can always find the EC2 instances it should scrape.

A slide titled "AWS EC2" explaining that cloud infrastructure is dynamic and that EC2 Service Discovery lets Prometheus find all available EC2 instances. A simple diagram shows a bordered box with multiple server icons and an "EC2" label.

How EC2 service discovery works

Prometheus queries the EC2 Describe APIs to receive a list of available instances. For each discovered instance it exposes a rich set of metadata labels that you can use for filtering, relabeling, and target selection. To enable EC2 service discovery, add an ec2_sd_configs block to your scrape_configs in prometheus.yml. The minimal required fields are:

the AWS region to query,
AWS credentials (access key and secret key) or a secure alternative (recommended),
optionally filters to limit discovered instances.

Example minimal job snippet:

scrape_configs:
  - job_name: "ec2"
    ec2_sd_configs:
      - region: "us-east-1"
        access_key: "<ACCESS_KEY>"
        secret_key: "<SECRET_KEY>"

Prometheus exposes many metadata labels per discovered target. Example discovered labels:

__address__="172.31.1.156:80"
__meta_ec2_ami__="ami-026b57f3c383c2eec"
__meta_ec2_architecture__="x86_64"
__meta_ec2_availability_zone__="us-east-1b"
__meta_ec2_availability_zone_id__="use1-az4"
__meta_ec2_instance_id__="i-02e3a91e06ca198db"
__meta_ec2_instance_state__="running"
__meta_ec2_instance_type__="t2.micro"
__meta_ec2_owner_id__="040497317401"
__meta_ec2_private_dns_name__="ip-172-31-1-156.ec2.internal"
__meta_ec2_private_ip__="172.31.1.156"
__meta_ec2_public_dns_name__="ec2-3-80-117-102.compute-1.amazonaws.com"
__meta_ec2_public_ip__="3.80.117.102"
__meta_ec2_subnet_id__="subnet-0fde5f37cd6c80f41"
__meta_ec2_tag_Name__="web"
__meta_ec2_vpc_id__="vpc-294f5553"
__metrics_path__="/metrics"
__scheme__="http"
__scrape_interval__="15s"
__scrape_timeout__="10s"
job="ec2"
instance="172.31.1.156:80"

Important: the default instance label (and __address__) typically use the private IP and port, because Prometheus is commonly deployed inside the same VPC. If Prometheus runs outside the VPC or cannot reach private IPs, use relabeling to change the target address to the public IP (for example set __address__ to __meta_ec2_public_ip__:9100). See the official docs for ec2_sd_config and relabel_config:

Prometheus exposes useful EC2 metadata labels for filtering and relabeling, such as __meta_ec2_tag_Name__, instance type, VPC ID, private/public IPs, and AMI IDs — use these to limit scrapes to relevant instances.

Common EC2 metadata labels

Label	Meaning
`__meta_ec2_tag_Name__`	The EC2 instance Name tag (useful for human-readable grouping).
`__meta_ec2_instance_type__`	Instance type (e.g., `t2.micro`, `m5.large`).
`__meta_ec2_vpc_id__`	The VPC where the instance is running.
`__meta_ec2_private_ip__`	The instance private IP address (often used as the default scrape address).
`__meta_ec2_public_ip__`	The instance public IP address (useful when Prometheus runs outside the VPC).
`__meta_ec2_instance_state__`	Instance lifecycle state (e.g., `running`, `stopped`).

Setting up IAM credentials for Prometheus

Prometheus needs permission to call EC2 Describe APIs. Create an IAM user or role with read-only EC2 permissions. Recommended steps:

Open the AWS Console: IAM → Users → Create user.
Create a user for programmatic access only (no console sign-in).
Attach the managed policy AmazonEC2ReadOnlyAccess or a custom scoped policy that permits the EC2 read-only actions Prometheus requires.
Save the Access key ID and Secret access key — the secret is shown only once. Use secure mechanisms to provide credentials to Prometheus (see warning below).

See AWS docs: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html

A screenshot of the AWS IAM "Add user" permissions screen with "Attach existing policies directly" selected and a search filter for "ec2", showing a scrollable list of AWS-managed EC2 policies (e.g., AmazonEC2ReadOnlyAccess) with checkboxes. Navigation buttons like "Previous" and "Next: Tags" are visible at the bottom.

After creation you will see the keys once. Example success message:

Success
You successfully created the users shown below. You can view and download user security credentials. This is the last time these credentials will be available to download. However, you can create new credentials at any time.

Users with AWS Management Console access can sign-in at: https://040497317401.signin.aws.amazon.com/console

Download .csv

User    Access key ID             Secret access key
prometheus    AKIA...EXAMPLE       wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Do not store long-lived AWS access keys directly in prometheus.yml in production. Prefer IAM roles (instance profiles), short-lived credentials (via STS), environment variables, or a secrets manager.

Recommended Prometheus configuration example

Replace the placeholders with your AWS region and preferred credential configuration. For production, prefer IAM roles or environment-based credentials instead of embedding keys in the YAML.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: []

rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node"
    static_configs:
      - targets: ["192.168.1.168:9100"]

  - job_name: "ec2"
    ec2_sd_configs:
      - region: "us-east-1"
        access_key: "<ACCESS_KEY>"
        secret_key: "<SECRET_KEY>"

After updating prometheus.yml, restart Prometheus:

sudo systemctl restart prometheus

Then verify discovery in the Prometheus UI:

Status → Service Discovery: confirm EC2 is listed and shows the number of discovered targets.
Status → Targets: inspect each discovered target, its labels, and scrape status.

If a target is DOWN, check:

Network connectivity (security groups, VPC routing, NAT/Gateway),
Whether Prometheus can reach the instance private IP (or switch to public IP via relabeling),
That the node exporter or other exporter is running on the expected port.

As EC2 instances scale up and down, Prometheus will refresh discovered targets automatically, keeping your monitoring accurate in dynamic AWS environments.

Additional references

Prometheus EC2 service discovery: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config
Prometheus relabeling: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
AWS IAM users: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html
Amazon EC2 read-only policy: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-policies-for-amazon-ec2.html

Introduction

Observability Fundamentals

Prometheus Fundamentals

PromQL

Dashboarding & Visualization

Application Instrumentation

Service Discovery

Push Gateway

Alerting

Monitoring Kubernetes

Conclusion

Scaling & Long Term Storage

How EC2 service discovery works

Common EC2 metadata labels

Setting up IAM credentials for Prometheus

Recommended Prometheus configuration example

Additional references

Watch Video

​How EC2 service discovery works

​Common EC2 metadata labels

​Setting up IAM credentials for Prometheus

​Recommended Prometheus configuration example

​Additional references

Watch Video

How EC2 service discovery works

Common EC2 metadata labels

Setting up IAM credentials for Prometheus

Recommended Prometheus configuration example

Additional references