Tail Sampling - KodeKloud

Tail sampling defers the sampling decision until a trace completes — after all spans for that trace have been collected. This lets you make decisions using the trace’s final outcome (errors, latency, span counts, attributes) instead of sampling early without full context. When traces arrive at an OpenTelemetry Collector, the tail sampler buffers each trace until it finishes (or the configured decision wait expires). The sampler evaluates policies in order, retains traces that match a policy, and drops or further samples the remainder. Selected traces are then exported to the configured backend. Example high-level tail-sampling configuration:

# collector-config.yaml (high-level example)
processors:
  tail_sampling:
    policies:
      - name: errors-policy
        type: status_code
      - name: latency-policy
        type: latency
      - name: probabilistic
        type: probabilistic
        sampling_percentage: 10

How tail sampling works (internal flow):

Spans arrive at the OpenTelemetry Collector and are buffered until the trace completes or decision_wait expires.
The tail sampler evaluates configured policies in order.
The first matching policy determines whether the trace is kept; otherwise fallback rules (rate allocation, probabilistic rules) apply.
Kept traces are exported; others are dropped.

The image is a flowchart showing how tail sampling works internally, with stages labeled as "Spans in," "Buffer," "Tail Sampler Policies," and "Export."

Policy examples, decision_wait, and behavior The decision_wait parameter controls how long the collector waits for late-arriving spans before making a sampling decision. Typical values are a few seconds (for example, 5s). Policies are evaluated in the configured order — the first matching policy decides to keep a trace. Detailed example showing multiple policy types, a decision wait, and rate allocation:

# collector-config.yaml (detailed tail-sampling example)
processors:
  tail_sampling:
    decision_wait: 5s
    policies:
      - status_code:
          status_codes: ["ERROR"]
      - latency:
          threshold_ms: 1200
      - spans_count:
          min_spans: 50
      - string_attribute:
          key: tenant
          values: ["vip"]
      - rate_limiting:
          spans_per_second: 2000
    rate_allocation:
      policies:
        - name: remaining
          percentage: 5

Common policy types and when to use them:

Policy Type	Purpose	Example snippet
Status code	Retain traces with error/failed status	`status_codes: ["ERROR"]`
Latency	Capture slow requests exceeding a threshold	`threshold_ms: 1200`
Spans count	Keep high-complexity traces with many spans	`min_spans: 50`
Attribute match	Retain traces from important tenants/services	`key: tenant` / `values: ["vip"]`
Rate limiting	Cap throughput to limit resource usage	`spans_per_second: 2000`
Rate allocation	Sample a percentage of remaining traces	`percentage: 5`

The status code policy retains traces containing errors.
The latency policy retains traces slower than 1200 ms.
The spans count policy retains traces with many spans (e.g., >= 50).
The attribute policy retains traces containing tenant: "vip".
The rate-limiting policy caps throughput (e.g., 2000 spans/sec).
The rate allocation policy samples a percentage of traces that didn’t match earlier rules (e.g., 5%).

Set an appropriate decision_wait (for example, decision_wait: 5s) to balance waiting for late spans against export latency. Monitor late-span patterns and tune this value to match your environment.

Trade-offs of tail sampling Tail sampling is powerful for retaining outcome-relevant traces, but it introduces trade-offs you must manage:

Trade-off	Impact
Memory pressure	Traces are buffered until completion; memory usage increases with traffic and trace duration.
Added latency	Export decisions are delayed until the trace completes (plus `decision_wait`).
Correctness	All spans for a trace must reach the same collector for accurate decisions.
Late spans / timeouts	Spans arriving after buffer expiry are dropped, potentially causing incomplete traces.

Monitor collector metrics (buffer sizes, dropped spans, memory usage, sampling decisions) to detect and address these impacts.

The image outlines trade-offs to watch in tail sampling, including memory pressure, latency, correctness, and timeout. Each trade-off has an associated consequence, such as increasing memory pressure or dropped late traces.

Scaling: combine head and tail sampling For high-volume environments, combine head (early) and tail (late) sampling:

Head sampler (application or edge collector): perform early probabilistic sampling to reduce telemetry volume and cost.
Tail sampler (downstream collector): retain outcome-relevant traces such as errors, slow traces, or traces from important tenants or services.

This hybrid approach balances cost control with the ability to preserve high-value traces for debugging and analysis.

The image illustrates a flowchart showing a "Head Sampler" leading to a "Tail Sampler," emphasizing cost control and outcome relevance in optimizing pipeline efficiency.

Distributed collectors and consistent hashing When collectors are horizontally scaled, spans for the same trace can be routed to different collector instances. For correct tail sampling, ensure all spans for a given trace are routed to the same downstream collector. A common pattern:

Deploy a first layer of stateless collectors (or edge collectors).
Use a load-balancing exporter with consistent hashing (trace ID as the routing key) to route spans for the same trace to the same downstream stateful collector instance (e.g., a StatefulSet).
The downstream collectors perform tail sampling on complete traces.

If a single collector handles all traffic, extra routing is not required. At scale, also add memory limiting and batching processors to stabilize resource usage:

# processors for resource control and batching
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 3500
    spike_limit_mib: 500
  batch:
    send_batch_size: 2000
    timeout: 2s

Monitor collector metrics (dropped spans, memory usage, sampling decisions, buffer sizes) and tune memory_limiter, batch, and decision_wait accordingly.

If spans for a trace are routed to different collectors, tail sampling decisions will be inaccurate. Use consistent hashing (trace ID) in the load-balancing exporter to maintain trace integrity.

Summary and best practices

Use head sampling to control telemetry volume early and reduce costs; use tail sampling to retain traces that matter based on full trace outcomes.
Tail sampling is especially valuable where retaining all traces is infeasible but outcome-relevant traces must be preserved.
For correct tail sampling at scale, route all spans of a trace to the same collector using consistent hashing on the trace ID.
Monitor and tune: decision_wait, memory_limiter, and batching parameters to balance memory usage, latency, and sampling accuracy.
Consider hybrid architectures (head + tail) to achieve both cost efficiency and diagnostic fidelity.

The image outlines various use cases for sampling, emphasizing head sampling for controlling telemetry volume, tail sampling for retaining high-value traces, and combining both for optimal results. It also suggests using tail sampling with load-balanced collectors to maintain accuracy.

This means every trace must be routed (via consistent hashing on trace ID) to the same downstream collector so tail-based sampling decisions are accurate and trace integrity is preserved. Links and references

OpenTelemetry Collector documentation: https://opentelemetry.io/docs/collector/
Consistent hashing overview: https://en.wikipedia.org/wiki/Consistent_hashing
OpenTelemetry sampling concepts: https://opentelemetry.io/docs/concepts/sampling/

Watch Video

Head Sampling

Demo Sampling Strategies