Skip to main content
In this lesson we’ll walk through real distributed traces so you can quickly see what trace data looks like, how to read spans, and how to use traces to troubleshoot production issues. The demo uses an application instrumented with OpenTelemetry that exports traces to Jaeger. Grafana is connected to the Jaeger datasource so we can search and inspect traces inside Grafana’s Explore view.
This guide assumes you have traces being exported to a Jaeger-compatible backend and Grafana configured with a Jaeger datasource. If you need setup guidance, see the OpenTelemetry, Jaeger, and Grafana docs linked in References.
How to query traces in Grafana
  • Open Grafana → Explore.
  • Select the Jaeger datasource.
  • Enter filters such as service or operation (span name), and optional attribute matches or min/max duration.
  • Limit the time range (e.g., last 5 minutes) before running the query.
Below I start from the frontend service because user-initiated flows originate there.
The image shows a Grafana interface with a Jaeger query panel where a service name is being selected. The user is choosing between "Frontend Service," "Product Backend Service," and "jaeger-all-in-one."
After selecting the service you can inspect available operation names (span names) such as getUserCart, getIndividualProduct, getRecommendedProduct, requestToTheProductsAPI, updateProduct. For this walkthrough I select all operations to capture a representative trace.
The image shows a Grafana interface with Jaeger integration, displaying an operation name dropdown for querying services like "Delete Product," "Get User Cart," and "Update Product."
Run the query with a narrow time window (last 5 minutes). Grafana lists traces with metadata such as trace name, start time, and total duration. Open an interesting trace and undock the details panel for more room. This selected trace shows:
  • Trace ID and start timestamp.
  • Total duration (~80 ms).
  • Involved services: frontend and backend.
  • Total spans: four.
    Expand each span to view service-level or span-level details. The first span is the frontend root span representing the initial update product action.
The image shows a Jaeger UI interface used for tracing operations in a "Frontend Service: Update Product" process, displaying span details and timings for various service operations.
Reading the frontend root span
  • Root span: starts at 0 relative to the trace.
  • Duration: ~79 ms (matches trace total because the root waits for children).
  • Resource attributes: host, pod, region, etc. (this demo instruments a minimal set).
  • Span attributes: in-application metadata such as service name and version.
The frontend span has a child that issues an HTTP request to the product backend. That child span includes richer HTTP attributes.
The image shows a Jaeger UI interface displaying trace details for a "Frontend Service: Update Product" operation, including span attributes like HTTP method and status code.
Typical HTTP client span details
  • Request URL and path (backend endpoint).
  • HTTP method (PATCH) and status code (200).
  • Span kind: client.
  • Events: timestamped logs such as request.sent. Events are ideal for attaching exceptions or custom logs to spans.
The image shows an interface from Jaeger, a distributed tracing system, displaying a trace for a "Frontend Service: Update Product" operation. It includes details like HTTP method, status code, URL, and service duration.
The backend receives the request and performs the update-product operation.
The image shows a Jaeger UI displaying tracing information for a service called "Frontend Service: Update Product," including query types, trace IDs, and detailed span information.
Backend span highlights
  • Service name: product-backend-service.
  • Backend span total: ~73 ms.
  • A DB child span: ~1.23 ms.
  • Start times are relative to the trace root; e.g., frontend root = 0 µs, backend starts at ~21 µs, DB child at ~1.23 ms — these offsets show timing relationships between steps.
Common backend span and resource attributes
AttributeMeaning / Example
span.kindRole of span: server for incoming HTTP
net.peer.ip / client.ipClient address (may be loopback in local demos)
http.methodHTTP verb (e.g., PATCH, GET)
http.target / http.urlPath/URL called (e.g., /products)
db.statement / SQL_QUERYSQL query text or signature (if instrumented)
service.versionApp version for troubleshooting regressions
The image shows a Jaeger UI interface displaying trace details for a "Frontend Service: Update Product" operation, including span attributes such as client IP, HTTP method, and service attributes.
Instrumenting database calls I instrumented the SQL executed by the backend so the DB query appears as its own span. Including either the full SQL or a query signature in a span attribute makes it far easier to find slow or incorrect queries.
Recording the SQL in a span attribute speeds escalation to DB or query-optimization teams with exact text and timing. Why traces matter for performance troubleshooting Traces provide a step-by-step timeline across services. Use duration filters to find slow requests (e.g., search for traces with min duration > 1s). In the next example I filter for traces longer than 1 second and open a trace that took 1.61 s. The initial spans are microseconds–milliseconds, but a final DB call to fetch recommended products consumed the full 1.61 s — isolating the query as the latency source.
The image shows a Jaeger UI screen displaying the trace details of a "Frontend Service: Get recommended Products" operation, including various span attributes and service durations.
Example span attributes for the slow DB client span:
  • otel.scope.name: “charge.py”
  • otel.scope.version: “0.5.0”
  • span.kind: “client”
  • SQL_QUERY: “select * from recommended_products”
These attributes let you hand off the exact problematic query and timing to the team that owns the database.
Use duration filters when troubleshooting performance problems: start broad (for example, min duration > 1s) and then drill into individual traces to identify the single offending span(s).
Traces and errors Traces capture errors and exceptions. When a request fails, the span may be marked error = true and include an exception event with a stack trace. Searching for traces where error = true helps you quickly surface failing requests. Below is an example exception recorded as a span event. The frontend failed to connect to the cart service (connection refused). The exception payload includes the HTTP client error message and the Python stack trace, which drastically reduces time-to-fix.
The image is a screenshot of a monitoring dashboard, likely from Jaeger, showing trace details for a "Frontend Service: Get User Cart" operation, including span attributes and an error indication.
Embedding exceptions and stack traces into spans eliminates guesswork: you can see the exact failing call and error, enabling faster diagnosis of network or application faults. Summary and next steps
  • Traces provide correlated, time-ordered telemetry across service boundaries to pinpoint latency hotspots and failing calls.
  • Instrument your services with the OpenTelemetry SDK for your language and configure an exporter (e.g., Jaeger) so services emit spans like the ones shown above.
  • Focus on these practical steps when troubleshooting:
    1. Filter traces by duration to find slow flows.
    2. Drill into a trace and inspect per-span attributes and events.
    3. Look for error = true spans and exception events for root cause details.
    4. If available, use db.statement or SQL_QUERY to escalate DB problems.
References

Watch Video

Practice Lab