Skip to content
SRE & DevOpsDeep Dive Published Updated 4 min readViews unavailable

OpenTelemetry Tail Sampling: Making Trace Decisions After the Outcome Is Known

How tail sampling buffers complete traces, evaluates outcome-aware policies, controls memory, and preserves representative telemetry during failures.

Head sampling decides whether to keep a trace when it begins. It is cheap and scalable, but the root span cannot yet know that a downstream request will fail or take ten seconds. Tail sampling delays the decision until spans have arrived, allowing policies based on final status, duration, attributes, or combinations—but that context costs memory, latency, and stateful operation.

The Collector becomes an assembly point

The OpenTelemetry Collector Contrib tail-sampling processor groups spans by trace ID, waits for a decision interval, and evaluates policies. It therefore needs to see all spans for a trace. If a load balancer sends sibling spans to different collector instances without trace-aware routing, each instance sees an incomplete trace and may decide inconsistently.

A common architecture uses stateless gateway collectors to route by trace ID to a stateful sampling tier. The exact load-balancing exporter and processor availability depend on the Collector distribution; verify component maturity and configuration against the pinned release.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    expected_new_traces_per_sec: 1000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000

This is an illustrative capacity point, not a safe universal configuration. Arrival delay, span count per trace, attribute size, and burstiness determine actual memory.

Decision timing is an accuracy tradeoff

decision_wait gives late spans time to arrive. A short wait reduces memory and export latency but can decide before the error span appears. A long wait improves completeness while increasing buffered state and the time before a trace reaches the backend.

Asynchronous messaging can create traces that last minutes or hours; buffering every such trace in one processor is rarely practical. Define where trace boundaries belong, use links when causality crosses long-lived jobs, and keep representative head sampling or application-level business events for workflows that cannot fit the tail window.

Capacity limits create their own sampling

num_traces bounds tracked traces. When arrival exceeds capacity, the processor may evict or drop state before a normal policy decision. If overload preferentially occurs during incidents, the system can lose exactly the traces operators wanted to preserve.

Monitor accepted spans, traces in memory, dropped or evicted traces, policy decisions, decision latency, exporter queue, process memory, and GC. Load-test the sampling tier with failure bursts, not only average traffic. Use memory limiting and queues as coordinated controls; an out-of-memory restart loses every undecided trace on that instance.

Policies can introduce bias and privacy risk

Keeping all errors and a percentage of successes is useful, but it does not create an unbiased dataset. Error-heavy endpoints, large traces, and policies evaluated in order can distort service comparisons. Record sampling probability or decision metadata where supported and ensure the backend’s rate calculations account for the sampling design.

Attribute policies see telemetry content. Do not route or retain traces based on raw user IDs, authorization headers, personal data, or high-cardinality secrets. Redact sensitive values before they reach the stateful sampling tier; sampling is not a privacy filter.

Composite and rate-allocation policies can reserve throughput for classes of traces. Their budgets must reflect business importance and incident needs, with a fallback probabilistic sample so an unexpected failure mode does not disappear because no policy named it.

Distributed sampling needs one coherent decision

W3C Trace Context carries sampling flags, but tail sampling may receive spans that were recorded despite an earlier undecided or sampled indication depending on SDK configuration. Align SDK head behavior, collector routing, and backend expectations. If an upstream service never records a span, no tail policy can recover it later.

Document which layer makes the authoritative decision and how that decision propagates. Mixed vendors or independently configured services can otherwise produce partial traces that look like application gaps.

Test failure behavior before production

Exercise complete success traces, errors arriving last, spans beyond the decision window, oversized traces, missing parents, collector restart, routing imbalance, backend outage, and capacity overflow. Confirm the kept population, not merely that the pipeline exports something.

Tail sampling earns its complexity when outcome-aware evidence materially improves incident diagnosis and the sampling tier itself is observable, capacity-tested, and recoverable. It should never be the only record that a transaction failed.

Related:

Sources:

Comments