> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Demo Internal Metrics of the Collector

> How to expose, inspect, and interpret OpenTelemetry Collector internal metrics in Prometheus format and configure scraping and scaling

This guide shows how to expose and inspect the OpenTelemetry Collector's internal telemetry (metrics) in Prometheus exposition format. You will learn how to enable the collector to serve Prometheus-format metrics and how to interpret the most useful metric families for debugging and scaling.

* OpenTelemetry Collector documentation: [https://opentelemetry.io/docs/collector/](https://opentelemetry.io/docs/collector/)
* Prometheus exposition format: [https://prometheus.io/docs/instrumenting/exposition\_formats/](https://prometheus.io/docs/instrumenting/exposition_formats/)

## What you will do

1. Optionally enable the zpages extension for quick debugging.
2. Add a metrics pipeline that includes the Prometheus receiver.
3. Configure `telemetry.metrics.readers` to expose the Collector's internal metrics in Prometheus format on a chosen host:port.

## Minimal example configuration

The following corrected YAML config shows the minimal settings to expose internal Collector metrics via a Prometheus reader. Place this in your Collector config file (e.g., `collector-config.yaml`) and restart the Collector.

```yaml theme={null}
zpages:
  endpoint: 0.0.0.0:55679

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [attributes, resourcedetection]
      exporters: [otlphttp/dynatrace, otlp/collector2]

    # Metrics pipeline - internal collector metrics
    metrics:
      receivers: [prometheus]
      processors: [resourcedetection, cumulativetodelta]
      exporters: [otlphttp/dynatrace, debug]

telemetry:
  logs:
    level: "INFO" # DEBUG | INFO | WARN | ERROR
  metrics:
    level: detailed
    readers:
      - pull:
          exporter:
            prometheus:
              host: "0.0.0.0"
              port: 8888
```

Notes:

* This example exposes internal metrics on port `8888`. Adjust `host` and `port` to match your network and security requirements.
* After applying the configuration, visit `http://<collector-ip>:8888/metrics` to view the Prometheus-format metrics.

<Callout icon="lightbulb" color="#1CB2FE">
  Exposed metrics are presented in Prometheus exposition format. Some metric names are marked `[alpha]`, which indicates experimental metrics that may change across Collector releases.
</Callout>

## Typical Prometheus-format metric excerpts

Below are representative metric families and sample lines you will commonly see from the Collector. The examples are cleaned up for clarity.

* Instrumentation scope metadata and exporter queue capacity/size:

```text theme={null}
# HELP otelcol_scope_info Instrumentation Scope metadata
# TYPE otelcol_scope_info gauge
otelcol_scope_info{otel_scope_name="github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer",otel_scope_version=""} 1
otelcol_scope_info{otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper",otel_scope_version=""} 1
otelcol_scope_info{otel_scope_name="go.opentelemetry.io/collector/processor/batchprocessor",otel_scope_version=""} 1

# HELP otelcol_exporter_queue_capacity Fixed capacity of the retry queue (in batches) [alpha]
# TYPE otelcol_exporter_queue_capacity gauge
otelcol_exporter_queue_capacity{data_type="logs",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 2048
otelcol_exporter_queue_capacity{data_type="metrics",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 2048
otelcol_exporter_queue_capacity{data_type="traces",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 2048

# HELP otelcol_exporter_queue_size Current size of the retry queue (in batches) [alpha]
# TYPE otelcol_exporter_queue_size gauge
otelcol_exporter_queue_size{data_type="logs",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 0
otelcol_exporter_queue_size{data_type="metrics",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 0
otelcol_exporter_queue_size{data_type="traces",exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 0
```

* Exporter send/failed counters and fileconsumer metrics:

```text theme={null}
# HELP otelcol_exporter_send_failed_metric_points_total Number of metric points in failed attempts to send to destination. [alpha]
# TYPE otelcol_exporter_send_failed_metric_points_total counter
otelcol_exporter_send_failed_metric_points_total{exporter="debug",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 0
otelcol_exporter_send_failed_metric_points_total{exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 4885

# HELP otelcol_exporter_sent_metric_points_total Count of metric points successfully sent to destination. [alpha]
# TYPE otelcol_exporter_sent_metric_points_total counter
otelcol_exporter_sent_metric_points_total{exporter="otlp/failing",otel_scope_name="go.opentelemetry.io/collector/exporter/exporterhelper"} 0

# TYPE otelcol_fileconsumer_open_files_ratio gauge
otelcol_fileconsumer_open_files_ratio{otel_scope_name="github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer"} 0
```

* Process-level resource metrics (CPU, memory, heap, uptime):

```text theme={null}
# HELP otelcol_process_cpu_seconds_total Total CPU time used in seconds [alpha]
# TYPE otelcol_process_cpu_seconds_total counter
otelcol_process_cpu_seconds_total{service="go.opentelemetry.io/collector/service"} 2.546875

# HELP otelcol_process_memory_rss_bytes Total physical memory (resident set size) [alpha]
# TYPE otelcol_process_memory_rss_bytes gauge
otelcol_process_memory_rss_bytes{otel_scope_name="go.opentelemetry.io/collector/service"} 30443776

# HELP otelcol_process_runtime_heap_alloc_bytes Bytes of allocated heap objects (see 'go doc runtime.MemStats')
# TYPE otelcol_process_runtime_heap_alloc_bytes gauge
otelcol_process_runtime_heap_alloc_bytes{otel_scope_name="go.opentelemetry.io/collector/service"} 140424

# HELP otelcol_process_uptime_seconds_total Uptime of the process [alpha]
# TYPE otelcol_process_uptime_seconds_total counter
otelcol_process_uptime_seconds_total{otel_scope_name="go.opentelemetry.io/collector/service"} 125979
```

* Processor counters (incoming/outgoing items) and exporter sent log counts:

```text theme={null}
# HELP otelcol_processor_incoming_items_items_total Number of items passed to the processor. [alpha]
# TYPE otelcol_processor_incoming_items_items_total counter
otelcol_processor_incoming_items_items_total{otel_signal="logs",otel_scope_name="go.opentelemetry.io/collector/service"} 125979
otelcol_processor_incoming_items_items_total{otel_signal="metrics",otel_scope_name="go.opentelemetry.io/collector/service"} 214406

# HELP otelcol_exporter_sent_log_records__records__total Number of log records successfully sent to destinations
# TYPE otelcol_exporter_sent_log_records__records__total counter
otelcol_exporter_sent_log_records__records__total{exporter="otlp/collector2",otel_scope_name="go.opentelemetry.io/collector"} 214369
```

## Interpreting these metrics

Use the table below for a quick reference of metric types and what they typically represent.

| Metric Type             | Meaning                                                                                      | Example usage                                                                               |
| ----------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| `counter`               | Cumulative value that only increases (monotonic). Useful for totals and rates.               | Count of log records successfully sent: `otelcol_exporter_sent_log_records__records__total` |
| `gauge`                 | Instantaneous value that can go up or down. Useful for current resource usage or queue size. | Current retry queue size: `otelcol_exporter_queue_size`                                     |
| `[alpha]` label in HELP | Experimental metric that may change between Collector releases.                              | `otelcol_exporter_queue_capacity` marked `[alpha]`                                          |

Common metric labels:

* `data_type` — the signal type: `logs`, `metrics`, or `traces`
* `exporter` — which exporter produced the metric (useful to tie metrics to a destination)
* `otel_scope_name` — the instrumentation or component scope

## Scaling and collection strategies

* Manually curling each Collector's `/metrics` endpoint (e.g., `http://<collector-ip>:8888/metrics`) is feasible for a few instances but does not scale for large deployments.
* Recommended production pattern:
  * Configure a central Prometheus server (or a centralized scraping pipeline) to scrape each Collector instance's `/metrics` endpoint.
  * Forward aggregated data from Prometheus (or the scraping pipeline) to a long-term backend for visualization and alerting.
  * Alternatively, use the Collector’s Prometheus receiver itself to scrape other Collector instances and forward their internal metrics to a central backend.

Useful references:

* Prometheus overview: [https://prometheus.io/docs/introduction/overview/](https://prometheus.io/docs/introduction/overview/)
* OpenTelemetry Collector components: [https://opentelemetry.io/docs/collector/](https://opentelemetry.io/docs/collector/)

<Callout icon="warning" color="#FF6B6B">
  Be cautious when exposing internal metrics publicly—ensure your network and authentication policies prevent unauthorized access to the `/metrics` endpoint. Limit exposure to trusted networks and enforce firewall and authentication controls as appropriate.
</Callout>

## Summary and quick checklist

* Enable zpages (optional) for quick debugging.
* Add a metrics pipeline that includes the Prometheus receiver.
* Add a Prometheus-format reader under `telemetry.metrics.readers` and set `host`/`port` (e.g., `8888`).
* Confirm by visiting `http://<collector-ip>:8888/metrics`.
* For scale: configure a central Prometheus scrape or a centralized scraping pipeline and forward to a backend for aggregation, alerting, and storage.

<CardGroup>
  <Card title="Watch Video" icon="video" cta="Learn more" href="https://learn.kodekloud.com/user/courses/prep-course-opentelemetry-certified-associate-certification-otca/module/9c72c1a7-4e0b-4541-8811-755843e69659/lesson/f0507d60-5de4-4ed8-a6dd-3c4e04249ce1" />
</CardGroup>
