Demo Collector Internal Logs and Debug Exporter

This guide walks through a live OpenTelemetry Collector (contrib) configuration — the exact config used on the host at /etc/otelcol-contrib/config.yaml. It explains how receivers, processors, exporters, extensions, service pipelines, and the telemetry / debug exporter interact. You’ll also learn how to control Collector log verbosity via the telemetry section and how to adjust the debug exporter verbosity to inspect metric/log payloads. To edit the config file:

sudo nano /etc/otelcol-contrib/config.yaml

Receivers

This Collector uses two receivers:

filelog — tails a JSON file and emits log entries.
prometheus — exposes/scrapes internal Collector metrics for pipelines.

Quick receiver summary:

Receiver	Purpose	Typical use-case
`filelog`	Tail file-based logs	Read application or device JSON logs (e.g., aircraft.json)
`prometheus`	Expose/collect metrics	Scrape internal Collector metrics or application metrics

Example receiver section:

receivers:
  filelog:
    include: ["/run/dump1090-mutability/aircraft.json"]
    start_at: beginning
    include_file_path: true
    multiline:
      line_start_pattern: '^\{'

  # Prometheus receiver to scrape internal collector metrics
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otelcol-internal-metrics'
          scrape_interval: 30s
          static_configs:
            - targets: ['localhost:8888']
              labels:
                collector_instance: 'rpi24-adsb-collector'
                environment: 'production'

Processors

Processors enrich, transform, or otherwise manipulate telemetry data before export. This configuration includes:

resource_detection — auto-detect system/host attributes.
attributes — upsert or transform log/metric attributes.
cumulativetodelta — convert cumulative metrics to delta form (useful for backends expecting deltas).
transform/fix_names — optional transform processor declared but not attached to pipelines by default.

Example processor declarations:

processors:
  resource_detection:
    detectors: [env, system]
    timeout: 5s
    override: false
    system:
      hostname_sources: ["os", "dns"]
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        host.arch:
          enabled: true
        host.cpu.vendor.id:
          enabled: true

  attributes:
    actions:
      - key: log.source
        value: opentelemetry-iot-dump1090-collector
        action: upsert
      - key: iot-device
        value: rpi24-adsb-receiver
        action: upsert
      - key: loglevel
        value: INFO
        action: upsert
      - key: device.address
        value: IJWJHUD768H
        action: upsert

  cumulativetodelta: {}

  transform/fix_names:
    error_mode: ignore
    metric_statements:
      - context: metric
        statements:
          - replace_pattern(name, "\\{", "-")
          - replace_pattern(name, "\\}", "-")

Note: To use the transform/fix_names processor, add it to a pipeline’s processors list (e.g., processors: [resource_detection, transform/fix_names]).

Exporters

This configuration uses two OTLP exporters and the debug exporter for local inspection.

exporters:
  otlphttp/dynatrace:
    endpoint: "${DT_ENDPOINT}"
    headers:
      Authorization: "Api-Token ${DT_API_TOKEN}"

  otlp/collector2:
    endpoint: "192.168.1.88:44317"  # Update with actual collector-2 address
    tls:
      insecure: true  # In production, configure proper TLS

  debug:
    verbosity: detailed  # options: basic | normal | detailed

Tips:

Replace ${DT_ENDPOINT} and ${DT_API_TOKEN} with actual Dynatrace OTLP HTTP endpoint and API token.
Avoid tls.insecure: true in production; configure proper TLS for OTLP exporters.

Extensions

Common extensions enable health checks, profiling, and zPages for debugging:

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

Service and Pipelines

The service section wires receivers, processors, and exporters into pipelines. Important: the debug exporter must be listed in a pipeline for it to emit debug output for that signal.

service:
  extensions: [health_check, pprof, zpages]
  pipelines:
    logs:
      receivers: [filelog]
      processors: [attributes, resource_detection]
      exporters: [otlphttp/dynatrace, otlp/collector2]

    # Metrics pipeline - internal collector metrics
    metrics:
      receivers: [prometheus]
      processors: [resource_detection, cumulativetodelta]
      exporters: [otlphttp/dynatrace, debug]

Telemetry (Collector internal logs & metrics)

The telemetry section controls the Collector’s own logs and how internal metrics are exposed to readers (e.g., Prometheus). Setting telemetry.logs.level adjusts the verbosity of the Collector’s operational logs. Example telemetry block with metrics reader:

telemetry:
  logs:
    # level: "ERROR"  # DEBUG | INFO | WARN | ERROR
  metrics:
    level: detailed
    readers:
      - pull:
          exporter:
            prometheus:
              host: "0.0.0.0"
              port: 8888

Notes:

If telemetry.logs.level is not set, the Collector defaults to INFO.
The Prometheus reader above exposes internal metrics on the configured host/port so the Prometheus receiver (or other collectors) can scrape them.

Viewing Collector logs

Follow the Collector service logs with systemd/journalctl:

sudo journalctl -u otelcol-contrib -f

When telemetry.logs.level is not explicitly set, you’ll typically see info-level messages about receivers, exporters, and pipeline processing. If the debug exporter is included in a pipeline and telemetry.logs.level permits, you’ll also see debug exporter output in the logs.

Changing Collector telemetry log level

To reduce log noise temporarily, set telemetry.logs.level to "ERROR" and restart the Collector:

telemetry:
  logs:
    level: "ERROR" # DEBUG | INFO | WARN | ERROR
  metrics:
    level: detailed
    readers:
      - pull:
          exporter:
            prometheus:
              host: "0.0.0.0"
              port: 8888

Restart the service:

sudo systemctl restart otelcol-contrib

With level: "ERROR", routine informational lines no longer appear in journalctl; only error-level telemetry is shown. Revert to "INFO" (or remove the setting) and restart to restore normal operational logs.

Debug exporter verbosity

The debug exporter prints telemetry payloads to the Collector logs and has three verbosity levels:

basic — minimal detail
normal — more detail
detailed — full metric data including histogram buckets and attributes

Example debug exporter configuration:

debug:
  verbosity: detailed  # basic | normal | detailed

To see debug exporter output for a given signal (e.g., metrics):

Ensure debug is declared under exporters.
Add debug to the pipeline’s exporters list for the signal you want to inspect.
Make sure telemetry.logs.level is at least INFO so the debug exporter prints to logs.

Example metrics pipeline that sends internal metrics to both Dynatrace and the debug exporter:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [resource_detection, cumulativetodelta]
      exporters: [otlphttp/dynatrace, debug]

Common runtime commands

Task	Command
Edit config	`sudo nano /etc/otelcol-contrib/config.yaml`
Restart Collector	`sudo systemctl restart otelcol-contrib`
Follow logs	`sudo journalctl -u otelcol-contrib -f`

Callouts

Ensure the debug exporter is declared under exporters and also listed in the pipeline exporters for the signal you want to inspect. Also keep telemetry.logs.level at or above INFO so debug exporter output is visible in logs.

Do not use tls.insecure: true in production. Configure proper TLS for OTLP exporters and for any remote collectors to protect telemetry data.

Summary

Receivers collect logs and internal metrics (filelog, prometheus).
Processors enrich and transform telemetry (resource_detection, attributes, cumulativetodelta, optional transform).
Exporters send data to remote backends (otlphttp/dynatrace, otlp/collector2) and debug for local inspection.
telemetry controls the Collector’s own logs and internal metrics exposure; adjust telemetry.logs.level to control noise.
Use debug exporter with verbosity: detailed to inspect full payloads; use basic or normal to reduce output.
After changing config, restart the service and follow logs with journalctl -u otelcol-contrib -f.

Links and References

OpenTelemetry Collector documentation: https://opentelemetry.io/docs/collector/
Prometheus scrape configs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
Dynatrace OTLP ingestion (reference your Dynatrace docs for the exact endpoint and headers)

If you need an example to attach the transform/fix_names processor into a pipeline or want a sample service block with more pipelines (traces, logs, metrics) let me know and I’ll add a fully annotated example.

Watch Video

Debug Exporter

Internal Metrics of the Collector

​Receivers

​Processors

​Exporters

​Extensions

​Service and Pipelines

​Telemetry (Collector internal logs & metrics)

​Viewing Collector logs

​Changing Collector telemetry log level

​Debug exporter verbosity

​Common runtime commands

​Callouts

​Summary

​Links and References

Watch Video

Receivers

Processors

Exporters

Extensions

Service and Pipelines

Telemetry (Collector internal logs & metrics)

Viewing Collector logs

Changing Collector telemetry log level

Debug exporter verbosity

Common runtime commands

Callouts

Summary

Links and References