Skip to main content
This lesson demonstrates how to implement sampling in an OpenTelemetry Collector pipeline. You’ll learn how to:
  • Send traces to a Collector with no sampling.
  • Apply probabilistic sampling for simple percentage-based reduction.
  • Use tail sampling to make decisions after a trace has completed.
  • Combine policies (composite tail sampling) to always keep errors while sampling non-error traces.
Overview of the starting Collector configuration
  • The Collector accepts OTLP over gRPC and HTTP.
  • Exporters are configured for Jaeger (OTLP) and a detailed debug exporter.
  • For clarity this example shows traces only; metrics and logs use the same concepts and similar configuration.
Initial Collector config
Send sample traces without any sampling
  • Generate 50 traces using telemetrygen and send them to the Collector via OTLP HTTP (insecure).
  • Adjust GOBIN if needed.
Commands:
Trimmed sample telemetrygen output:
Because no sampling is active, all 50 traces should be processed and forwarded to the configured exporters (debug output and Jaeger).
The image shows a trace analysis interface from Jaeger UI displaying telemetry data with a timeline graph and a list of recent traces for the "telemetrygen" service.
Probabilistic sampling (simple percentage-based sampling)
  • Use the probabilistic sampler processor when you only need a fixed-percentage reduction of trace volume (for example, keep 10% of traces).
  • This sampler is fast and low-overhead but cannot inspect full trace content to preferentially keep errors or high-latency traces.
Reference: Probabilistic Sampling Processor (GitHub)
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor
The image shows a GitHub page for the "Probabilistic Sampling Processor" within the OpenTelemetry collector contributions. It includes status details, code ownership, and a description of the processor's functionality.
Minimal probabilistic sampler example
Extended options (attribute source, trace ID selection)
Wire probabilistic sampling into the traces pipeline (example: proportional mode with 10%):
Restart the Collector after updating its config. When you generate 50 traces again, roughly 10% (about 5 traces) should be forwarded to Jaeger — subject to random variance. Limitations of probabilistic sampling
  • Samples purely by probability; cannot preferentially keep error traces or traces with special attributes.
  • Not suitable when you need content-aware decisions that require seeing the entire trace.
Detailed documentation: Probabilistic Sampling Processor (GitHub)
The image shows a GitHub page displaying documentation for the "probabilisticsamplerprocessor" within the OpenTelemetry Collector contrib project. It includes detailed text about proportional and equalizing sampling modes.
Tail sampling (sample after trace completion)
  • Tail sampling waits until a trace is complete (or until a configured decision timeout) so the processor can evaluate the whole trace when deciding to keep or drop it.
  • Useful when you must always keep errors, high-latency traces, or apply complex composite rules.
Reference: tailsamplingprocessor docs
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor Key configuration options
  • decision_wait: How long to wait for late spans before making a decision (e.g., 10s). Longer waits can improve accuracy but increase memory usage.
  • num_traces: Max number of traces to keep in memory (eviction safety limit).
  • expected_new_traces_per_sec: Hint for internal optimizations.
  • decision_cache: Cache sizes for sampled and non-sampled decisions to avoid recalculating decisions.
Basic tail_sampling example (keep all traces with ERROR status):
Wire tail sampling into the traces pipeline:
Callout: tuning and memory considerations
Tail sampling lets you make accurate, content-aware decisions (for example, always keep errors). Tune decision_wait, num_traces, and cache sizes based on your traffic profile to balance accuracy and memory usage.
Behavior examples and tests
  1. No error status in traces
  • Generate 50 traces without ERROR status. With the all-errors policy above, the tail sampler will drop them (they don’t match the ERROR policy).
Expected result: Jaeger receives no traces (tail sampler dropped them).
  1. All traces set to ERROR
  • Generate 50 traces with ERROR status. Tail sampling will keep them.
Expected result: Jaeger receives all 50 traces (policy matched). Composite policies: keep all errors, but probabilistically sample non-error traces
  • Real-world pattern:
    • Always keep traces with ERROR status.
    • For OK or UNSET traces, keep around 10% to reduce volume.
Composite policy example (AND + status + probabilistic):
How this works:
  • all-errors preserves any trace with status ERROR.
  • sample-success is an AND composite policy: it only samples when both a non-error status is detected and the probabilistic sub-policy passes.
Testing the composite policy
  • Restart the Collector after applying the composite policy.
  • Generate 50 ERROR traces -> all should be kept in Jaeger.
  • Generate 100 OK/UNSET traces -> about 10% (~10) should be kept and forwarded.
Callout: resource and correctness warning
Tail sampling holds traces in memory until a decision is made. Misconfigured decision_wait, num_traces, or cache sizes can cause high memory usage or eviction of traces. Monitor memory and tune policies for production workloads.
Comparison: Probabilistic vs Tail Sampling
FeatureProbabilistic (processor)Tail Sampling (processor)
Decision timeAt ingestion (early)After trace completion (late)
Can prioritize errors?NoYes
Memory overheadMinimalHigher (stores traces until decision)
Typical use caseSimple percentage reductionKeep errors/high-latency and apply complex rules
Latency impactNoneAdds decision_wait delay before export
Links and references Wrap-up
  • Use probabilistic sampling for low-cost, uniform reductions.
  • Use tail sampling when you need content-aware decisions (always keep errors, high-latency traces, or apply composite policies).
  • Carefully tune decision_wait, num_traces, and decision caches for tail sampling to avoid excessive memory usage and evictions.
This concludes the lesson on implementing sampling strategies in the OpenTelemetry Collector.

Watch Video