- Add Prometheus to Docker Compose and enable the remote write receiver
- Provide a Prometheus configuration suitable for receiving pushed metrics
- Configure the OpenTelemetry Collector to export metrics via
prometheusremotewrite - Start the stack and validate metrics in the Prometheus UI
This demonstration uses Prometheus’ remote write API so the OpenTelemetry Collector can push metrics directly into Prometheus. Use this when you want the Collector to push metrics rather than having Prometheus scrape targets.
Overview
- Add a Prometheus service to your Docker Compose file and enable the remote-write HTTP receiver.
- Mount a simple
prometheus.ymlthat sets global intervals (noscrape_configsneeded if the Collector pushes metrics). - Configure the Collector with a
prometheusremotewriteexporter that targets Prometheus athttp://prometheus:9090/api/v1/write. - Start the stack and verify metrics at
http://localhost:9090.
1) Add Prometheus to Docker Compose
Updatedocker-compose.yml to include Jaeger and Prometheus services, and configure Prometheus to enable the remote write receiver:
| Service | Purpose / UI | Port (host:container) |
|---|---|---|
| Jaeger | UI / tracing | 16686:16686 |
| Prometheus | UI / metrics | 9090:9090 |
| Jaeger | gRPC | 14250:14250 |
| Jaeger | HTTP | 14268:14268 |
| Jaeger | Agent (UDP) | 6831:6831/udp, 6832:6832/udp |
- The
--web.enable-remote-write-receiverflag enables Prometheus to accept pushed metrics at/api/v1/write. - We mount a local
prometheus.ymlto control Prometheus behavior from your project directory.
2) Create Prometheus configuration (prometheus.yml)
If the Collector is pushing metrics to Prometheus, you do not needscrape_configs. Create prometheus.yml beside your docker-compose.yml:
remote_write section? The remote_write stanza in Prometheus is for sending metrics out to remote storage. In this setup, Prometheus acts as the receiver — the HTTP endpoint /api/v1/write — which is enabled by the CLI flag above, not by prometheus.yml.
Helpful reference:
- Prometheus remote write receiver docs: https://prometheus.io/docs/prometheus/latest/storage/#remote-endpoints-and-storage
3) Configure the OpenTelemetry Collector to export metrics via Prometheus remote_write
Add aprometheusremotewrite exporter to your Collector configuration (example filename: collector-config.yaml or otel-collector-config.yaml) and include it in the metrics pipeline.
Example Collector config:
- Exporter name:
prometheusremotewrite(this is the Collector exporter that POSTs to Prometheus’/api/v1/write). - Endpoint:
http://prometheus:9090/api/v1/write. When running under Docker Compose, the Collector can resolve theprometheushostname (service name) on the default network. - Keep a
debugexporter if you want to inspect metrics in the Collector logs while simultaneously pushing them to Prometheus. - The
otlp/jaegerexporter forwards traces to Jaeger on port14250.
4) Bring up the stack
Start the services in detached mode:| Action | Command |
|---|---|
| Prometheus logs | docker-compose logs prometheus |
| Collector logs | docker-compose logs <collector-service-name> |
| Show running containers | docker-compose ps |
Jaeger UI: http://localhost:16686
5) Validate metrics in Prometheus UI
- Open http://localhost:9090.
- In the “Expression” input, search for metrics emitted by your application. Example metric names from a sample app:
current_temperature_fahrenheit_degFcurrent_humidity_percentage
- Run queries to confirm metrics are visible and being updated.
If metrics are missing, check:
- The Collector is running and its metrics pipeline includes the
prometheusremotewriteexporter. - The
endpointin the Collector exporter ishttp://prometheus:9090/api/v1/write. - Prometheus was started with
--web.enable-remote-write-receiverand the mountedprometheus.ymlis valid. - Services are on the same Docker Compose network so hostnames resolve between containers.
- Use the Collector debug exporter (configured above) to see metric payloads in logs.
- Inspect Prometheus logs for errors related to
/api/v1/write. - Confirm the Collector and Prometheus containers can reach each other by running a quick
curlfrom within the Collector container:- Enter the container:
docker exec -it <collector-container> /bin/sh - Test connectivity:
curl -v http://prometheus:9090/api/v1/write
- Enter the container:
- Ensure no firewall or host port conflicts exist on
9090.
- OpenTelemetry Collector: https://opentelemetry.io/docs/collector/
- Prometheus remote write docs: https://prometheus.io/docs/prometheus/latest/storage/
- Jaeger: https://www.jaegertracing.io/