Skip to main content
This lesson assumes you already understand traces, spans, OpenTelemetry, and Jaeger. Here we make those concepts practical: how to read a Jaeger waterfall, spot latency patterns, and follow a repeatable triage workflow that takes you from “something is slow” to “this is exactly why.”
The image lists five learning objectives related to using Jaeger for identifying critical paths, recognizing latency patterns, following a triage workflow, connecting traces to metrics and logs, and using Jaeger effectively.

​
Core skill: reading the waterfall

The waterfall (timeline) view is your primary diagnostic tool. Each row is a span; each bar shows when the operation started and how long it ran. Read it like this:
  • Start at the root span (top).
  • Find the widest child span — that’s the current critical path.
  • Drill into that child and repeat until you find the operation that dominates latency.
Example: an API gateway span lasts 100 ms. Inside it, auth-service takes 12 ms while order-service takes 82 ms — clearly most time is spent in the order service.
The image is a waterfall chart illustrating the timeline of different services in a Jaeger trace, highlighting API-gateway, Auth-service, Order-service, Inventory-svc, Payment-svc, and Payment-gv with their respective latencies. A note indicates that the payment-svc is a critical path, making up 70% of the total latency.

​
Five common latency patterns

Most latency incidents fit into one of these patterns. Commit these to memory — they speed up triage. After the next section on the first three patterns, see the illustrated example below.
  1. Serial bottleneck
    One span dominates the timeline. Example: a DB call is 800 ms in a 900 ms request. Fixes: caching, asynchronous processing, or optimizing the slow operation.
  2. Fan-out amplification
    A service issues many parallel calls. The slowest child determines total latency. Fixes: set timeouts, degrade functionality, add fallbacks.
  3. Retry storm
    A failed call is retried multiple times. Each retry consumes its timeout — three retries with 2 s timeouts add 6 s to latency. Fixes: address the root failure, implement exponential backoff, limit retries, add circuit breakers.
The image lists five common latency patterns: Serial Bottleneck, Fan-Out Amplification, Retry Storm, Missing Spans (Gaps), and Error Cascade, with a focus on Retry Storm and suggestions for fixes.
  1. Missing spans (gaps)
    The parent span duration is longer than the sum of child spans. Example: parent shows 500 ms but children add to 200 ms — the remaining 300 ms is usually uninstrumented work (queue wait, DNS lookup, connection pool wait). Fix: add instrumentation or capture timing for the missing operations.
The image lists five common latency patterns: Serial Bottleneck, Fan-Out Amplification, Retry Storm, Missing Spans (Gaps), and Error Cascade, with a note about unexplained gaps in the timeline and a suggested fix.
  1. Error cascade
    One service fails, its caller retries, and upstream callers retry — a tree of failures. Fix the root error and add circuit breakers, timeouts, and graceful degradation to stop the cascade.
The image illustrates "Five Common Latency Patterns," including Serial Bottleneck, Fan-Out Amplification, Retry Storm, Missing Spans (Gaps), and Error Cascade, with corrective actions suggested below.

​
A systematic triage workflow

Use a repeatable process rather than guessing. This workflow takes you from a symptom to a concrete root cause.
  1. Search and filter
    • Open Jaeger, select the relevant service and operation.
    • Set a minimum duration filter so you only see slow traces (e.g., > 1s).
    • Use tags to narrow results, for example: http.status_code = 500.
  2. Sort and pick
    • Sort traces by duration and pick the slowest trace.
    • When possible, find a corresponding fast trace for the same operation and compare them.
  3. Identify and drill down
    • In the waterfall, find the widest span and click it.
    • Inspect the span’s tags, logs/events, and process info.
    • Follow the critical path: at each level, pick the widest child.
  4. Correlate with metrics and logs
    • Use the service name, pod/host, and timestamp from the span to search metrics (Prometheus/Grafana) and logs (ELK/Loki) for that instance.
    • Look for CPU, memory, thread/connection pool saturation, GC pauses, or network errors at the same timestamp.
Pro tip: always compare a slow and a fast trace of the same operation — the delta usually points to the root cause.
The image illustrates a "Systematic Triage Workflow" process, detailing steps such as Search, Identify, Correlate, Filter, and Drill Down for analyzing and troubleshooting service requests. It includes a pro tip about comparing slow and fast traces to identify root causes.

​
Span details: what to look for

When you click a span in Jaeger, examine these sections — they tell you “why,” not just “where”:
  • Tags — e.g., http.method, http.status_code, db.statement. These describe the request and its outcome.
  • Logs / events — timestamped internal events like connection acquired at 0 ms or query started at 5 ms. These reveal timing inside the span.
  • Process information — hostname, pod name, namespace; these identify which instance emitted the span so you can correlate logs/metrics.
The waterfall shows where the problem is; span details show why it happened.
The image provides details of what to look for in span data, including tags, logs/events, and process information related to an operation, such as HTTP method, status code, timestamps of key events, and process identifiers.

​
Practical tips that save time

  • Set a minimum duration when searching to avoid triaging healthy traces. Example: duration > 1s or > 500ms depending on your SLA.
  • Use tag filters to focus on errors: http.status_code = 500.
  • At each level of the waterfall, follow the widest child (the critical path).
  • Watch for gaps: if children don’t add up to the parent, instrument the missing interval (queue, DNS, connection pool).
  • Repeated adjacent spans usually indicate retries; inspect the first failure event and the retry policy.
The image provides tips and tricks for using Jaeger, focusing on setting minimum duration, using tags in search, adjusting time range, and using trace compare.
Compare a slow trace with a fast trace of the same operation — the difference is often the quickest path to the root cause.
The image provides tips and tricks for using Jaeger in practice, including noting span count, looking for retries, following the critical path, and checking for gaps, all centered around a graphic of a stylized brain.
If spans are missing (gaps), don’t assume the slow child is the cause — the uninstrumented interval might be the true source. Add instrumentation before concluding.

​
Quick checklist for live triage

  • Did you filter by duration and tags?
  • Did you compare slow vs fast traces?
  • Did you follow the critical path to the widest spans?
  • Did you inspect tags, events, and the process that emitted the span?
  • Did you correlate with metrics/logs for the same timestamp and instance?

​
Wrap-up

  • The waterfall is your primary diagnostic tool: find the widest bar and follow it.
  • Remember the five latency patterns — they cover almost every issue you’ll encounter.
  • Use the triage workflow (search, filter, identify, drill down, correlate) instead of ad hoc guessing.
  • Span tags, logs/events, and process details point you to the next action and the fix.

Watch Video