Skip to main content
This lesson shows how to use traces, spans, OpenTelemetry, and Jaeger to diagnose latency and find root causes. It’s a hands-on guide for reading the Jaeger waterfall, recognizing common latency patterns, and following a repeatable triage workflow that gets you from “something is slow” to “this is exactly why.” Core SEO keywords: traces, spans, OpenTelemetry, Jaeger, waterfall view, critical path, latency patterns, root cause analysis, observability. Core skill: reading the waterfall Each row in the Jaeger waterfall is a span. The horizontal bar shows when the span started and how long it lasted. The root span (for example, an API gateway span) covers the full request duration. Child spans (auth service, order service, etc.) appear as subranges within that bar. Technique: start at the root span, locate the widest child span (the one consuming the most elapsed time), click through, and repeat. Two or three clicks will usually reveal the bottleneck.
A Jaeger "waterfall view" timeline showing different microservices (API-gateway, auth-service, order-service, inventory-svc, payment-svc, etc.) as horizontal colored bars with their latencies in milliseconds. A callout notes the critical path, highlighting that payment-svc is the main bottleneck (70% of total latency).
Five common latency patterns Almost every latency issue you’ll investigate fits one of these five patterns. Recognizing the pattern quickly points you toward the right remediation.
A slide titled "Five Common Latency Patterns" showing a timeline of icons labeled Serial Bottleneck, Fan-Out Amplification, Retry Storm, Missing Spans (Gaps), and Error Cascade. A boxed note below explains that one slow span can dominate latency and suggests fixes like caching, async processing, or optimizing the slow operation.
Summary table: patterns, symptoms, quick mitigations Detailed descriptions and recommendations
  • Serial bottleneck
    One span dominates the request. Example: a DB query takes 800 ms in a 900 ms request. Typical fixes: caching, moving work to asynchronous flows, or optimizing the slow operation itself.
  • Fan-out amplification
    A service calls many downstream services in parallel. If the parent aggregates or waits for results, a single slow child (e.g., 500 ms) sets the latency. Add timeouts, consider hedged requests, and provide graceful degradation to avoid one slow dependency blocking everything.
  • Retry storm
    A call fails and is retried multiple times. Each retry may include its own timeout, so three retries of a 2-second timeout can add 6 seconds. The correct fix is to address the root failure, not just increase timeouts. Use exponential backoff, jitter, and sensible retry limits.
Retry storms amplify latency and can overload downstream systems. Ensure retries include jitter and exponential backoff, and monitor retry counts as a first alerting signal.
  • Missing spans (gaps)
    When child spans add up to less than the parent span, the remaining time is usually uninstrumented work: queue waits, DNS resolution, connection pool waits, network stalls, or processing done outside traced code. Add instrumentation at these boundaries.
  • Error cascade
    One downstream error causes retries or fallbacks upstream, producing a cascade. Fix the originating error and mitigate propagation using circuit breakers, bulkheads, and defensive timeouts.
Systematic triage workflow Use a repeatable process to investigate slow traces:
  1. Search and filter — open Jaeger, select the affected service, and set a minimum duration filter (for example >1s for slow checkout requests).
  2. Sort and identify — sort traces by duration (descending) and open the slowest traces.
  3. Compare — if possible, compare a slow trace with a fast trace performing the same operation; differences often point to the root cause.
  4. Drill down — in the waterfall, find the widest child span and click it to inspect tags, logs/events, and process info.
  5. Correlate — use the service name, pod/hostname, and timestamp to pull matching metrics and logs for more context.
An infographic titled "Systematic Triage Workflow" showing a row of colorful chevron arrows and labeled steps (Search, Identify, Correlate, Filter, Drill Down) with brief explanatory notes. The slide is branded © KodeKloud.
Compare a slow trace with a fast trace for the same operation. The differences between them typically highlight the root cause quickly.
Span details: tags, logs/events, and process info Clicking a span in Jaeger will typically reveal three helpful areas:
  • Tags: structured key/value pairs such as HTTP method, status code, DB type, and the SQL statement.
  • Logs/Events: timestamped events inside the span, for example “connection acquired” or “query started”.
  • Process information: hostname, pod name, namespace — which identifies the instance that generated the span.
Example consolidated span detail (tags, logs, and process info):
Practical tips that save time
  • Always set a minimum duration filter when searching — it filters out healthy traces and surfaces problematic cases.
  • Use tags to narrow results, for example http.status_code=500 or error=true.
  • Follow the critical path: the widest child span is usually your next target.
  • Check for gaps: if child spans don’t add up to the parent duration, look for uninstrumented waits or external work.
  • Repeated identical spans back-to-back usually indicate retries; inspect the first failure to find the root cause.
  • Correlate trace timestamps with logs and metrics — that combined view often reveals environmental or deployment-related causes.
An infographic titled "Jaeger in Practice: Tips and Tricks" with a stylized brain illustration at the center. Around it are four analysis tips: Note the span count, Follow the critical path, Look for retries, and Check for gaps.
Wrap-up The Jaeger waterfall is your primary diagnostic tool: find the widest bar, follow the critical path, recognize which of the five common patterns matches the behavior, and use the systematic triage workflow (search → identify → drill down → correlate) to reach the root cause. Inspect span tags, logs/events, and process information to determine why a span was slow or failed. Links and references

Watch Video