OpenTelemetry tail sampling through a checkout failure

OpenTelemetry tail sampling chooses which recorded traces to retain using evidence from their spans, the records of individual operations. In the Collector’s default trace-complete strategy, the sampler buffers incoming spans before deciding. A later payment error can therefore cause earlier checkout work to be retained with it. That can reduce ordinary traces stored in the backend, but the evidence has to reach the sampler before its decision.

That bargain is useful only if the traces left behind still answer the investigation. Start with one request path whose missing evidence matters, reproduce a known failure, and retrieve it under the proposed sampling policy before expanding. The pilot in this guide follows a checkout timeout through collection, selection and backend lookup, so storage savings can be judged against the evidence they preserve.

Can the error reach the decision at all?

Consider a hypothetical checkout that calls a payment service and receives a timeout. Its checkout, payment-client and payment-server spans share a trace ID. If the timeout status reaches the collector before the sampling decision, an error policy can keep the associated trace, including the successful operations that preceded the failure. That is the attraction of waiting: the later result can determine whether earlier context is worth retaining.

A head sampler makes its choice earlier, often in the application SDK. It can avoid some recording and export work, but cannot use a failure that has yet to occur. Tail sampling makes the choice after recorded spans arrive; it cannot recover those discarded before arrival. Put an error-retention policy after a 10% head sample and it can keep qualifying errors within that sample, not every application error. The OpenTelemetry sampling overview explains these different points of decision.

For the checkout trial, establish that the collector receives the evidence before asking it to select it. Inspect SDK settings and parent sampling behavior across the payment call. With recording and export enabled for every request on the pilot path, capture a controlled request’s trace ID and find every expected span. A missing payment-server span at this stage points to instrumentation or propagation; no later retention rule can supply it.

A staging checkout with synthetic transactions and no real payment data keeps that baseline manageable. Full collection on one path lets the team inspect how checkout and payment connect without increasing retention across the fleet. The tracing implementation guide explains choosing and instrumenting such a path. Once its complete trace is retrievable, the pilot has something meaningful to compare with the sampled result.

One trace needs one place to be judged

Now consider what happens if batches go to arbitrary collector instances. One sampler might receive the payment error while another sees only successful checkout work. Each can make a locally reasonable decision and still leave the investigator with an incomplete story: the error survives, but the work leading to it disappears.

A common solution separates intake from sampling and routes by trace ID between them. The OpenTelemetry load-balancing exporter supports that arrangement, provided intake instances resolve a consistent destination set. The documentation also notes that changing that set reroutes some trace IDs. Adding a healthy sampler is therefore a change to where in-flight trace state belongs, not simply extra stateless capacity.

One sampling instance removes this variable from the first experiment. It is useful for learning, not a high-availability design. Before scaling the pilot, send the same request cases through several intake instances and change sampling membership deliberately in staging. Comparing retrieved spans before, during and after the change shows the evidence cost of that topology transition.

Attribute processing is another part of the route that the policy must understand. Enrich resources and remove sensitive data before a policy that depends on their resulting values. If privacy processing removes a customer attribute, a later rule cannot match the original value. Use the gateway deployment documentation to separate these responsibilities and show on the deployment diagram which hop transforms attributes and which decides retention.

Error, latency and ordinary-request policies

An initial policy can retain recorded errors, retain slow traces and keep a modest probabilistic selection of ordinary traffic for comparison. Each branch has a job: failures expose what went wrong, slow successes show another problematic path, and ordinary requests provide context. The sampling percentage is a trial input whose cost and usefulness must be measured, not a recommended value for every service.

This fragment illustrates that combination in a Collector distribution containing the tail-sampling processor. Merge it into an existing traces pipeline without losing receivers, memory protection, authentication or exporters, and validate it against the exact build used for the pilot. It is intentionally a fragment rather than a complete production configuration; no runtime benchmark is claimed here.

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 20000
    expected_new_traces_per_sec: 300
    policies:
      - name: checkout-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1500
      - name: ordinary-comparison
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

The processor reference explains the policy types and combined decision rules. In this simple combination, the error and latency branches can retain a qualifying trace independently of the 5% branch. Adding explicit drop policies changes the decision logic, so do not assume that a larger policy list behaves like this example. Its capacity and arrival-rate values are illustrative as well.

Send three identifiable requests: a fast successful checkout, a slow successful checkout and a checkout whose error status is explicitly recorded. Preserve each unique trace ID in the test output. After decision and export delay, the slow and error traces should be retrievable. The fast one may be absent by design. A larger set of distinct IDs, rather than one lucky lookup, is needed to inspect the probabilistic branch’s approximate selection rate.

Open the spans to verify what the rules actually saw. An HTTP error does not automatically mark every span as an error, and application code may represent a caught timeout differently from an uncaught exception. Likewise, the latency policy evaluates received spans, timestamps and its threshold; it is not a promise to preserve every request that violated a service-level objective. The retrieved trace connects the configuration to the request you meant to investigate.

Waiting for the answer uses memory

While those decisions are pending, the sampler must hold traces. New trace IDs per second multiplied by the wait gives a useful first capacity estimate. With the illustrative 300-per-second arrival rate and a 30-second interval, about 9,000 traces may be waiting at steady load. A burst of 900 per second for that interval produces an estimate of 27,000, exceeding the fragment’s 20,000-trace setting before any claim about memory safety can be made.

A count is not yet a RAM budget. Six small checkout spans occupy a different amount of memory from hundreds of tool spans in an agent execution. Measure resident memory and CPU with representative attributes and span counts, allowing for exporter queues and other processors in the same container. The capacity margin protects the very evidence the wait is meant to collect: an out-of-memory restart can discard pending traces.

A longer wait can give the payment error time to arrive, but also holds more data and postpones the trace’s availability. Measure from the first received span to the last span required to explain the timeout. The request’s own duration omits SDK batching and transport delay, so it cannot alone tell you how long the sampler needs to wait.

If decisive evidence normally arrives after the window, extending this short-request design indefinitely is the wrong bargain. Jobs that run for hours need a separately designed policy or tracing path with appropriate context connecting their stages. A pilot that serves synchronous checkout well should not be taken as evidence that it will preserve background-work failures.

Late spans, bursts and sampler restarts

Once normal requests behave as expected, challenge each assumption separately. Delay the payment span past the decision window, create a safe burst of new traces, then test a restart while requests are outstanding. Separate experiments let the team explain a missing span; combining all three creates a failure without revealing which limit caused it.

Late spans, decision caches and premature removal are documented processor behaviors to inspect here. Remembering a decision can let later spans follow it, but cannot let earlier selection use evidence that had not arrived. Pair the pinned version’s early-drop and late-span telemetry with the known trace IDs. That prevents a quiet component dashboard from substituting for proof that the failed checkout can actually be found.

Experiment	Inspect	What the result changes
Payment error arrives on time	Checkout and payment spans under the saved trace ID	Confirms the intended investigation survives the normal path
Error span arrives after the window	Initial decision, late-span behavior and final retrieved trace	Shows whether the wait or workload split needs changing
Arrival burst	Known trace retrieval, memory, refusals and early removals	Tests capacity rather than average-volume assumptions
Sampler restart or membership change	Missing span sets and recovery time	Establishes the evidence loss to expect during maintenance

Selection is only one stop on the way to the backend. An accepted trace can still fail during export. The Collector resiliency guidance covers queues and retries at that later stage; they do not protect the same state as a pending sampling decision. Observe both stages, then perform the final backend lookup that a responder would use.

For our checkout, success means retrieving the failed request and distinguishing time spent before the payment call from the payment timeout itself. A trace that shows only “checkout was slow” has preserved a symptom without the explanatory boundary. Working backward from that investigation gives every routing, waiting and export check a reason to exist.

Storage savings and collection costs

Compare retained bytes and backend ingestion cost with the full-retention baseline, then account for collector resources and maintenance. Tail sampling can save downstream storage while leaving substantial application-to-collector traffic. Use the deployment’s real billing units, compression and prices; a 5% ordinary-trace policy is not a 95% reduction in the complete observability bill.

The other side of the comparison is what responders can still learn. Report which controlled failures retained the required span relationships, how long lookup took, and what the burst and restart removed. An operational lead can then decide whether the savings justify both another collector tier and its measured loss behavior.

Do not use that deliberately selected dataset to estimate the service’s error fraction. Preferentially retaining errors makes them a larger share of stored traces than of all requests. Population-level metrics provide the request and error denominators; sampled traces explain particular executions. The guide to reading distributed traces also covers how missing observations and overlapping spans limit what an investigator can conclude.

For an illustrative calculation, suppose 1,000 recorded checkouts contain 20 errors and 980 fast successes. All error spans arrive in time, and only the ordinary 5% branch applies to the successful traces. Retaining all 20 errors and an expected 49 successes leaves about 69 traces; errors make up roughly 29% of that retained set, although they were 2% of the original requests. This is expected-count arithmetic, not a Collector result, and a finite probabilistic sample will vary. The same policy that preserves failures for diagnosis changes their share of the dataset.

Discarding a trace later does not prevent its attributes from passing through the application, network and sampler first. Remove secrets and unnecessary personal data at instrumentation or the earliest appropriate processing point, limit collector access, and use authenticated encrypted transport across trust boundaries. Review the OpenTelemetry sensitive-data guidance with whoever is responsible for emitted attributes before treating lower retention as a privacy solution.

When tail sampling is worth operating

A pilot earns adoption when its savings justify the measured collection effort and the known failure still yields useful evidence. Full retention may remain simpler for an affordable low-volume service. If decisive spans cannot reach the same sampler in time, repair that limitation or choose another design before relying on the policy. The cheaper dataset is not a substitute for the investigation it must support.

Preserve the previous Collector configuration and rehearse a scoped route back in staging. Returning to full retention increases backend traffic; check ingestion limits so a recovery action does not create another telemetry failure. A rollback for the pilot path is easier to reason about than a fleet-wide emergency switch.

For checkout, the comparison ends in the tool a responder actually uses: can they retrieve the controlled timeout and distinguish the payment failure from the work preceding it? A limited production slice and representative load show whether that result survives ordinary operation. The storage saving then has a concrete meaning: fewer retained traces without losing the request that explains this failure.