Distributed Tracing Common Mistakes and How to Fix Them

You instrumented your services, connected a tracing backend, and opened the trace view — only to find traces that stop halfway through the request, spans with no useful attributes, and a span list so noisy you cannot find what you are looking for. Distributed tracing is one of the highest-value observability investments you can make, but several common setup mistakes can make it nearly worthless.

Here are the patterns that consistently cause tracing problems in production and how to fix each one.

Mistake 1: Broken Context Propagation at Async Boundaries

This is the most common and most damaging mistake. When a request crosses an async boundary — a message queue, a background job, an event stream — the trace context must be explicitly serialized and deserialized. If it is not, traces appear to end at the boundary and the downstream work appears as orphaned root spans with no connection to the originating request.

What it looks like: A trace that shows the API layer but has no visibility into what happens in the workers that process the resulting jobs. You see the request that enqueued the job but not the job execution.

How to fix it: Treat trace context as a first-class citizen in any message or event payload. When enqueuing:

const carrier = {};
propagation.inject(context.active(), carrier);
await queue.publish({ body: payload, headers: carrier });

When consuming:

const parentContext = propagation.extract(context.active(), message.headers);
context.with(parentContext, () => processJob(message.body));

Do this consistently across every async boundary: Kafka, RabbitMQ, SQS, Redis streams, and custom job runners. There is no automatic instrumentation for these unless the specific library has an OpenTelemetry plugin that handles it.

Mistake 2: Missing Spans for the Most Important Operations

Auto-instrumentation captures the infrastructure layer: HTTP calls, database queries, cache operations. It does not capture business logic. If your traces show an HTTP handler and a SQL query but nothing in between, you cannot answer the question "what was the application doing for the 400ms between receiving the request and hitting the database?"

What it looks like: Traces with large timing gaps between auto-instrumented spans. The total request took 800ms but the only spans you see account for 200ms.

How to fix it: Add custom spans for operations that take meaningful time and have business significance. Validation pipelines, pricing calculation, external enrichment calls, complex aggregations — any work that takes more than a few milliseconds and might need to be debugged deserves a span. Name spans with a dot-hierarchy that reflects your domain: order.price.calculate, user.permissions.evaluate.

Mistake 3: Spans With No Attributes

A span with a name and a duration tells you something happened. A span with attributes tells you what happened. During debugging, the difference between "this span was called" and "this span processed order_id=9f2k1 for user_id=4821 with 3 line items totaling 4999 cents" is the difference between knowing where to look and knowing the answer.

What it looks like: Traces where spans have names but no attached key-value data. You can see the shape of the request but not the substance.

How to fix it: Every custom span should set attributes for the entity IDs and key business values involved in that operation. Before ending the span, add:

span.setAttribute('order.id', order.id);
span.setAttribute('order.total_cents', order.totalCents);
span.setAttribute('order.item_count', order.items.length);
span.setAttribute('customer.tier', customer.tier);

Follow the OpenTelemetry semantic conventions for standard attributes — http.method, db.system, db.statement — so that your tooling can automatically render and filter them correctly.

Mistake 4: Unclosed Spans

Spans that are started but never ended are silently dropped by most SDKs after a timeout. If you have code paths that throw exceptions before reaching span.end(), those spans disappear from your traces — often the most important spans, since they represent error cases.

What it looks like: Intermittent traces that appear to succeed even when requests fail, or traces that seem shorter than expected during error conditions.

How to fix it: Always end spans in a finally block:

const span = tracer.startSpan('my.operation');
try {
  const result = await doWork();
  span.setStatus({ code: SpanStatusCode.OK });
  return result;
} catch (err) {
  span.recordException(err);
  span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
  throw err;
} finally {
  span.end();
}

Using startActiveSpan with a callback handles this more cleanly in many SDKs, as the span is automatically ended when the callback returns or throws.

Mistake 5: Sampling Everything or Sampling Nothing

Two failure modes at opposite ends of the spectrum.

Sampling 100% of requests in high-throughput production systems generates trace volumes that are expensive to store and slow to query. Most traces from successful fast requests provide no debugging value.

Sampling too aggressively — keeping only 1% of traces — means that low-frequency events, like specific error types or edge-case slow requests, may never be captured at all.

How to fix it: Use tail-based sampling rather than head-based sampling. Head-based sampling decides whether to keep a trace when the request starts, before you know whether it will be interesting. Tail-based sampling decides after the trace is complete, allowing you to keep all errors and slow requests while dropping fast successful ones.

Tools like the OpenTelemetry Collector support tail-based sampling via the tail_sampling processor. A simple policy: keep all traces where any span has an error status, keep all traces where the root span duration exceeds a threshold (e.g., 500ms), and sample the remainder at a lower rate.

Mistake 6: Span Name Cardinality Explosion

Span names that include dynamic values — user IDs, order IDs, request parameters — create high-cardinality span name spaces that break trace search performance and cause problems in backends that index span names.

What it looks like: Tens of thousands of unique span names in your tracing backend. Slow queries when searching by operation name.

How to fix it: Span names must be static strings or low-cardinality templates. Dynamic values belong in span attributes, not names.

Wrong: span name = "process order ord_9f2k1 for user 4821" Right: span name = "order.process", attributes order.id = "ord_9f2k1", user.id = 4821

For HTTP server spans, many auto-instrumentation libraries default to using the full URL path as the span name, which causes cardinality issues for routes with path parameters. Configure the instrumentation to use the route template (/orders/:id) rather than the resolved path (/orders/9f2k1).

Mistake 7: Instrumenting Without a Retention and Sampling Budget

Traces are expensive to store. A service handling 10,000 requests per minute, with 20 spans per trace, each carrying a dozen attributes, generates significant data volume. Teams that instrument fully and ship everything to their backend without modeling the cost are often surprised by the invoice.

How to fix it: Decide on your retention needs (1 day? 7 days? 30 days?) and your sampling policy before you go to production. Model the span volume. Choose a backend that fits your budget at that volume. Configure sampling in the collector layer so application code does not need to know about it.

Getting distributed tracing right is a systems design problem, not just a library configuration problem. If you want tracing that actually works when incidents happen, Clixo can help you design and implement the right setup for your system.