Trace Visualization and Root Cause Analysis

This lesson assumes you already understand traces, spans, OpenTelemetry, and Jaeger. Here we make those concepts practical: how to read a Jaeger waterfall, spot latency patterns, and follow a repeatable triage workflow that takes you from “something is slow” to “this is exactly why.”

​

Core skill: reading the waterfall

The waterfall (timeline) view is your primary diagnostic tool. Each row is a span; each bar shows when the operation started and how long it ran. Read it like this:

• Start at the root span (top).
• Find the widest child span — that’s the current critical path.
• Drill into that child and repeat until you find the operation that dominates latency.

Example: an API gateway span lasts 100 ms. Inside it, auth-service takes 12 ms while order-service takes 82 ms — clearly most time is spent in the order service.

​

Five common latency patterns

Most latency incidents fit into one of these patterns. Commit these to memory — they speed up triage. After the next section on the first three patterns, see the illustrated example below.

1. Serial bottleneck
   One span dominates the timeline. Example: a DB call is 800 ms in a 900 ms request. Fixes: caching, asynchronous processing, or optimizing the slow operation.
2. Fan-out amplification
   A service issues many parallel calls. The slowest child determines total latency. Fixes: set timeouts, degrade functionality, add fallbacks.
3. Retry storm
   A failed call is retried multiple times. Each retry consumes its timeout — three retries with 2 s timeouts add 6 s to latency. Fixes: address the root failure, implement exponential backoff, limit retries, add circuit breakers.

4. Missing spans (gaps)
   The parent span duration is longer than the sum of child spans. Example: parent shows 500 ms but children add to 200 ms — the remaining 300 ms is usually uninstrumented work (queue wait, DNS lookup, connection pool wait). Fix: add instrumentation or capture timing for the missing operations.

5. Error cascade
   One service fails, its caller retries, and upstream callers retry — a tree of failures. Fix the root error and add circuit breakers, timeouts, and graceful degradation to stop the cascade.

​

A systematic triage workflow

Use a repeatable process rather than guessing. This workflow takes you from a symptom to a concrete root cause.

1. Search and filter
   • Open Jaeger, select the relevant service and operation.
   • Set a minimum duration filter so you only see slow traces (e.g., > 1s).
   • Use tags to narrow results, for example: http.status_code = 500.
2. Sort and pick
   • Sort traces by duration and pick the slowest trace.
   • When possible, find a corresponding fast trace for the same operation and compare them.
3. Identify and drill down
   • In the waterfall, find the widest span and click it.
   • Inspect the span’s tags, logs/events, and process info.
   • Follow the critical path: at each level, pick the widest child.
4. Correlate with metrics and logs
   • Use the service name, pod/host, and timestamp from the span to search metrics (Prometheus/Grafana) and logs (ELK/Loki) for that instance.
   • Look for CPU, memory, thread/connection pool saturation, GC pauses, or network errors at the same timestamp.

Pro tip: always compare a slow and a fast trace of the same operation — the delta usually points to the root cause.

​

Span details: what to look for

When you click a span in Jaeger, examine these sections — they tell you “why,” not just “where”:

• Tags — e.g., http.method, http.status_code, db.statement. These describe the request and its outcome.
• Logs / events — timestamped internal events like connection acquired at 0 ms or query started at 5 ms. These reveal timing inside the span.
• Process information — hostname, pod name, namespace; these identify which instance emitted the span so you can correlate logs/metrics.

The waterfall shows where the problem is; span details show why it happened.

​

Practical tips that save time

• Set a minimum duration when searching to avoid triaging healthy traces. Example: duration > 1s or > 500ms depending on your SLA.
• Use tag filters to focus on errors: http.status_code = 500.
• At each level of the waterfall, follow the widest child (the critical path).
• Watch for gaps: if children don’t add up to the parent, instrument the missing interval (queue, DNS, connection pool).
• Repeated adjacent spans usually indicate retries; inspect the first failure event and the retry policy.

​

Quick checklist for live triage

• Did you filter by duration and tags?
• Did you compare slow vs fast traces?
• Did you follow the critical path to the widest spans?
• Did you inspect tags, events, and the process that emitted the span?
• Did you correlate with metrics/logs for the same timestamp and instance?

​

Wrap-up

• The waterfall is your primary diagnostic tool: find the widest bar and follow it.
• Remember the five latency patterns — they cover almost every issue you’ll encounter.
• Use the triage workflow (search, filter, identify, drill down, correlate) instead of ad hoc guessing.
• Span tags, logs/events, and process details point you to the next action and the fix.

​

Links and references

• Jaeger documentation: https://www.jaegertracing.io/docs/
• OpenTelemetry: https://opentelemetry.io/
• Distributed tracing best practices: https://opentelemetry.io/docs/concepts/best-practices/