All postsEngineering

Tail sampling and trace retention for a Next.js and Go stack

Keep the traces that explain incidents and drop the rest: context propagation, a tested OTel Collector config, and a retention budget you can compute.

Johnny Unar11 min read

01

head sampling keeps the wrong traces

A Next.js frontend that calls a Go API produces a lot of spans. In Next.js 16 with @vercel/otel, a single App Router page render that makes one backend call emits about a dozen spans in the default, non-verbose mode, and the Go service adds its own. Multiply that by every request, including bots and health checks, and the bill from whichever tracing backend you use grows faster than traffic does. The usual reaction is to set the SDK's sample rate to 10% and move on.

That's head sampling: the decision is made when the root span starts, before anyone knows whether the request will fail, take four seconds or touch checkout. At 10%, nine out of ten failed checkouts are gone before they happen, while the trace store fills up with fast, successful requests that nobody will ever open. Tail sampling turns this around. The OpenTelemetry Collector buffers every span of a trace for a few seconds, looks at the finished trace, and keeps it if it matches a rule you care about, such as an error, a slow request or a route that moves money, plus a small random baseline for comparison.

This post covers what that takes in a Next.js and Go stack: getting one trace across both services (which fails silently by default on both sides), a Collector config that was run against real spans rather than just validated, and how to turn sampling rules into a retention budget before the invoice does it for you.

02

one trace across next.js and go

Tail sampling decides per trace, so a trace that's broken in two is worse than useless: the Next.js half might be kept because it was slow while the Go half, which holds the actual slow query, is dropped as an unremarkable request. Both services need to propagate the W3C traceparent header, and both have a default that quietly doesn't.

On the Next.js side, @vercel/otel instruments fetch and injects trace context, but its README says that by default the context is propagated only to Vercel deployment URLs, and every other URL has to be enabled explicitly. Without the option below, the Go service receives no traceparent and starts a new trace for every call. Running the same page with and without the option against a local Collector shows it plainly: without the option, the Next.js fetch span and the Go server span carry different trace IDs.

ts
1import { registerOTel } from '@vercel/otel'
2
3export function register() {
4 registerOTel({
5 serviceName: 'web',
6 instrumentationConfig: {
7 fetch: {
8 // By default context is only propagated to Vercel deployment URLs.
9 propagateContextUrls: [process.env.API_ORIGIN!],
10 },
11 },
12 })
13}

On the Go side, otelhttp.NewHandler extracts the incoming context using the global propagator, and the global propagator in the Go SDK is a no-op until you set one. Forgetting otel.SetTextMapPropagator gives exactly the same symptom as the missing Next.js option, which makes it hard to tell which side is at fault. The setup below also sets the sampler explicitly to ParentBased(AlwaysSample()). That's the SDK default, but writing it down documents the intent: the SDK sends every span it gets, and the Collector makes the real decision.

go
1func setupTracing(ctx context.Context) (func(context.Context) error, error) {
2 // Endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT.
3 exp, err := otlptracehttp.New(ctx)
4 if err != nil {
5 return nil, err
6 }
7 tp := sdktrace.NewTracerProvider(
8 sdktrace.WithBatcher(exp),
9 // Keep every span the caller sampled; the collector makes the real decision.
10 sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.AlwaysSample())),
11 sdktrace.WithResource(resource.NewSchemaless(semconv.ServiceName("orders-api"))),
12 )
13 otel.SetTracerProvider(tp)
14 // Without this line otelhttp extracts nothing and every request starts a new trace.
15 otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
16 propagation.TraceContext{}, propagation.Baggage{},
17 ))
18 return tp.Shutdown, nil
19}

There's one more gap worth closing if you sample by route. Next.js sets http.route on its root span, but otelhttp v0.72 with a Go 1.22+ ServeMux only puts the matched pattern into the span name (GET /v1/orders/{id}), not into an attribute, so a Collector rule on http.route can't see Go routes. A small wrapper copies the pattern over:

go
1// withRoute copies the matched ServeMux pattern onto the server span as
2// http.route, so collector policies can match on it.
3func withRoute(mux *http.ServeMux) http.Handler {
4 return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
5 if _, pattern := mux.Handler(r); pattern != "" {
6 _, path, _ := strings.Cut(pattern, " ")
7 trace.SpanFromContext(r.Context()).SetAttributes(semconv.HTTPRoute(path))
8 }
9 mux.ServeHTTP(w, r)
10 })
11}

Mount it inside the instrumentation, as otelhttp.NewHandler(withRoute(mux), "orders-api"), so the server span already exists when the wrapper runs.

03

a collector config that keeps what matters

The config below receives OTLP, computes request metrics from every span, and then tail-samples. Policies are evaluated together: a trace is kept if any policy says sample, and a drop policy wins over everything else, which is how health checks are excluded even when they error. All of it was run on otelcol-contrib 0.165 with a test sender: a fast, successful span tagged as a cold start was kept, an erroring /healthz span was dropped, an erroring span on a normal route and a span on a money route were kept, and a plain fast span was dropped.

yaml
1receivers:
2 otlp:
3 protocols:
4 grpc:
5 endpoint: 0.0.0.0:4317
6 http:
7 endpoint: 0.0.0.0:4318
8
9processors:
10 memory_limiter:
11 check_interval: 1s
12 limit_percentage: 80
13 spike_limit_percentage: 20
14
15 tail_sampling:
16 decision_wait: 10s
17 num_traces: 100000
18 expected_new_traces_per_sec: 1000
19 policies:
20 - name: drop-health-checks
21 type: drop
22 drop:
23 drop_sub_policy:
24 - name: health-route
25 type: string_attribute
26 string_attribute:
27 key: http.route
28 values: ["/healthz", "/readyz"]
29 - name: errors
30 type: status_code
31 status_code:
32 status_codes: [ERROR]
33 - name: slow
34 type: latency
35 latency:
36 threshold_ms: 1000
37 - name: cold-starts
38 type: boolean_attribute
39 boolean_attribute:
40 key: faas.coldstart
41 value: true
42 - name: money-routes
43 type: string_attribute
44 string_attribute:
45 key: http.route
46 values: ["/api/checkout", "/v1/commands/consume-credits"]
47 - name: baseline
48 type: probabilistic
49 probabilistic:
50 sampling_percentage: 2
51
52 batch: {}
53
54connectors:
55 span_metrics:
56 histogram:
57 explicit:
58 buckets: [50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s]
59 dimensions:
60 - name: http.route
61
62exporters:
63 otlphttp/traces:
64 endpoint: ${env:TRACES_BACKEND_URL}
65 otlphttp/metrics:
66 endpoint: ${env:METRICS_BACKEND_URL}
67
68service:
69 pipelines:
70 traces/all:
71 receivers: [otlp]
72 processors: [memory_limiter]
73 exporters: [span_metrics]
74 traces/sampled:
75 receivers: [otlp]
76 processors: [memory_limiter, tail_sampling, batch]
77 exporters: [otlphttp/traces]
78 metrics:
79 receivers: [span_metrics]
80 processors: [batch]
81 exporters: [otlphttp/metrics]

The cold-start policy is where copied configs usually go wrong. faas.coldstart is a boolean in the OpenTelemetry semantic conventions, so it needs the boolean_attribute policy. A string_attribute policy with values: ["true"] passes otelcol validate without complaint and then never matches, because the string policy doesn't compare against booleans. In the test run, swapping in the string version silently stopped keeping the cold-start trace. This is the general lesson about Collector configs: validation checks the shape, so send a few known spans through before trusting the rules.

The span_metrics connector sits in its own pipeline before sampling, which matters more than it looks. Request rate, error rate and latency histograms per route are computed from 100% of spans (they emit as traces.span.metrics.calls and traces.span.metrics.duration), so dashboards and alerts stay accurate even though only a few percent of traces are stored. In the test, the metrics counted the /healthz and plain spans that the sampler threw away. If you derive RED metrics from stored traces instead, they inherit the sampling bias: errors are stored at 100% and ordinary successes at 2%, so an error rate computed from stored traces is inflated many times over.

decision_wait is how long the Collector holds a trace before deciding, so it has to be longer than your slowest normal request, including the Go service's downstream calls, but every second of it costs memory, because num_traces traces are held at once. If num_traces is too small for your traffic multiplied by the wait, the processor evicts traces before deciding and counts them in otelcol_processor_tail_sampling_sampling_trace_dropped_too_early, which is the first metric to alert on.

Tail sampling has one hard requirement, which the processor's README states plainly: all spans of a trace must reach the same Collector instance. With one Collector that's free. With several replicas behind a normal load balancer, spans from the Next.js service and the Go service land on different instances and each sees half a trace. The documented fix is a two-tier setup where a first layer only routes by trace ID with the load_balancing exporter, and a second layer, reached through a headless service, runs the config above:

yaml
1receivers:
2 otlp:
3 protocols:
4 grpc:
5 endpoint: 0.0.0.0:4317
6 http:
7 endpoint: 0.0.0.0:4318
8
9exporters:
10 load_balancing:
11 protocol:
12 otlp:
13 tls:
14 insecure: true
15 resolver:
16 dns:
17 hostname: otel-sampler-headless.observability.svc.cluster.local
18 port: "4317"
19
20service:
21 pipelines:
22 traces:
23 receivers: [otlp]
24 exporters: [load_balancing]

The routing key defaults to traceID for traces, so it doesn't need setting.

04

sdk samplers decide too early

Any sampling done in an SDK happens at span start, which is the same limitation tail sampling exists to avoid. Sentry's tracesSampler is a common example. According to Sentry's sampling docs it runs when a span starts and receives the span name, the attributes passed at creation and the parent's decision. Attributes set later in the request are not there yet, so a sampler rule like "keep everything where faas.coldstart is true" never fires when the attribute is set inside the handler, which is the only place it can be set. Use the SDK sampler for decisions that are knowable from the route alone, and let it inherit the upstream decision so a trace isn't kept in Next.js and dropped in a downstream service:

ts
1import * as Sentry from '@sentry/nextjs'
2
3Sentry.init({
4 dsn: process.env.SENTRY_DSN,
5 tracesSampler: ({ name, inheritOrSampleWith }) => {
6 if (name.includes('/healthz')) return 0
7 if (name.includes('/api/checkout')) return 1
8 return inheritOrSampleWith(0.05)
9 },
10})

inheritOrSampleWith arrived in version 9 of the Sentry JavaScript SDK, and older versions read parentSampled instead. If the same traces also pass through an OTel Collector, decide which layer owns sampling, because whatever an SDK drops at the head never reaches the Collector's tail sampler.

The cold-start tag itself is a few lines. It marks the first request each server process handles, and it must be called from request code, because the Next.js docs note that spans created while instrumentation.ts loads don't join request traces:

ts
1import { trace } from '@opentelemetry/api'
2
3const state = globalThis as { __coldStartSeen?: boolean }
4
5// Tags the first request a server process handles. Call it from request code,
6// not from instrumentation.ts, so there is an active request span to tag.
7export function tagColdStart() {
8 trace.getActiveSpan()?.setAttribute('faas.coldstart', !state.__coldStartSeen)
9 state.__coldStartSeen = true
10}

Called at the top of a page, it tags the page's render span with true on the first request after a start and false afterwards, and since the tail sampler checks every span in a trace, that's enough for the policy to match. Whether cold starts deserve a policy depends on where Next.js runs. On a serverless platform they're a real source of tail latency, and on a long-running container they happen once per deploy and the rule costs almost nothing.

05

turning rules into a retention budget

Tracing backends bill by spans or bytes ingested and by how long they keep them, so the number to estimate is spans stored per month. The kept fraction follows from the policies: everything the rules match, plus the baseline share of the rest. Summing the rule shares ignores overlap (a slow checkout that also errors is counted three times), which makes the estimate an upper bound, and for a budget that's the right direction to be wrong in.

python
1traces_per_month = 50_000_000
2spans_per_trace = 15
3bytes_per_span = 900 # placeholder: measure your own
4retention_days = 14
5
6# Share of all traces each policy matches. Read these off your own data.
7policies = {
8 "errors": 0.004,
9 "slow": 0.010,
10 "cold_starts": 0.002,
11 "money_routes": 0.030,
12}
13baseline = 0.02
14
15matched = min(sum(policies.values()), 1.0) # upper bound, ignores overlap
16kept = matched + (1 - matched) * baseline
17spans = traces_per_month * kept * spans_per_trace
18stored_gb = spans * bytes_per_span * retention_days / 30 / 1e9
19
20print(f"kept fraction {kept:.2%}")
21print(f"spans/month {spans / 1e6:,.1f} M")
22print(f"stored at once {stored_gb:,.1f} GB")

With these made-up inputs it prints a kept fraction of 6.51%, 48.8 million spans a month instead of 750 million, and about 20.5 GB held at any time. The inputs are where the work is, and the Collector gives you most of them once it's running. otelcol_processor_tail_sampling_count_traces_sampled is labelled by policy and decision, so you can see how many traces each rule keeps, and dividing otelcol_processor_tail_sampling_global_count_traces_sampled with sampled="true" by otelcol_processor_tail_sampling_new_trace_id_received gives the real kept fraction. These are exposed on the Collector's own metrics endpoint, port 8888 by default. Span size is easiest to get from your backend's usage page, as ingested bytes divided by ingested spans over a day.

The budget also tells you which rule to tune when it's over. In the example the money-route rule dominates because it keeps every successful checkout too, and that's often the wrong trade: errors and slow requests on those routes are already kept by the other rules, so replacing it with a 20% probabilistic rule scoped to those routes (an and policy combining the route match with a probabilistic sub-policy) keeps plenty of normal checkouts for comparison at a fraction of the volume.

Retention doesn't have to be uniform either. Sampled traces are mostly read within days of an incident, while the span metrics built from 100% of traffic are what you look at over months, so a short trace retention with longer metric retention is usually the better use of the same money. If your backend lets you set retention per dataset or environment, preview deployments rarely need more than a couple of days.

06

when tail sampling isn't worth it

If the whole stack produces a few million spans a month, keep everything. Tail sampling adds a stateful component that has to be sized, monitored and scaled in two tiers, and that only pays off once the volume of spans costs more than running it. Below that, a backend's free or entry tier with 100% of traces is simpler and you never lose the one trace you needed.

Some backends can also sample after ingest, which moves the operational work to the vendor. Whether that saves money depends on whether they bill on ingested or retained data, so read the pricing page before relying on it. If ingest is what you pay for, the decision has to happen before the data leaves your network, and that means a Collector you run.

Thanks for reading.
Johnny Unar

Written by

Johnny Unar

Keep reading

Stop patching.Start building.

Tell us what's slowing you down. We'll bring coffee and opinions.