Timeouts Retries Idempotency

You know the feeling: you tap Pay, the spinner keeps spinning, and you aren’t sure whether the payment went through. Tap again and you risk paying twice. Or maybe the payment never happened at all. This article shows how to design distributed systems so the Pay button (and similar user actions) are safe to press more than once. We cover the three core tools you should combine: timeouts, retries with backoff, and idempotency. All components in this design communicate via API calls, and every call crosses a network. Networks are inherently unreliable: calls can be lost, delayed, duplicated, or reordered. Robust systems anticipate these worst-case scenarios rather than assuming responses always arrive.

This image is a network diagram illustrating a digital transaction process involving a mobile device, load balancer, app server with cache, service, and database, all connected via APIs. It highlights the flow of information and the elements of data management in an online purchase scenario.

A common mistake is to write code that assumes a response will always arrive. In distributed systems, “no response” is normal. From the caller’s perspective, a slow service and a dead service look the same: no reply. Since you generally cannot reliably distinguish the two, set clear timeouts and design behavior for timeout events.

Timeouts

When your application server calls another service, it cannot wait forever. If the downstream service is slow or down, blocking indefinitely ties up threads, sockets, and other resources and causes requests to pile up. That slows the server and can eventually crash it. Best practices for timeouts:

Set a sensible per-call timeout (e.g., 1–5 seconds for user-facing APIs). Tune based on observed latency.
Prefer short timeouts for synchronous user flows; allow longer timeouts for internal background workflows.
Fail fast: when a timeout triggers, return a clear error so caller logic can decide to retry or take an alternate path.
Propagate deadline information where possible (for example, include an explicit deadline header) so downstream services can respect caller expectations.

Example: pseudo-code showing a client call with a timeout.

// Pseudo-code (Node/Fetch-like)
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 2000); // 2s timeout

try {
  const resp = await fetch(url, { signal: controller.signal });
  // handle response
} catch (err) {
  // timeout or network error
} finally {
  clearTimeout(timeout);
}

Important: a timeout only stops the caller from waiting. It does not tell you whether the remote service completed the work. The request may have failed before reaching the service, or the service may have completed the work but its response was lost. Treat timeouts as “unknown” outcomes until you have an application-level mechanism (like idempotency or a transaction trace) to determine final state.

Retries and Backoff

Many errors are transient: a short network blip, a DNS hiccup, or a service restart. Retrying failed calls can often recover automatically. But retries must be performed carefully.

The image is a flowchart illustrating a network process involving a user purchasing headphones for $10.00, with components like a load balancer, app server, service, cache, and database, highlighting a "network blip" issue.

The dangerous pattern is immediate, aggressive retries from many callers. If a database or service is struggling and every client retries at once, retries amplify load and can push the system into complete failure — the classic retry storm.

Do not retry blindly. Immediate repeated retries across many clients can worsen outages (retry storms). Limit retries and add backoff and jitter.

The image is a diagram showing a flow of data from a user's phone through a load balancer, app server, and service to a struggling database. It illustrates retries in a network architecture.

Recommended retry pattern:

Try immediately once (fast-fail).
On failure, retry with exponential backoff (e.g., wait 1s, then 2s, then 4s).
Add jitter (randomized small variance) to avoid synchronized retries across clients.
Cap the maximum wait and the number of attempts (for example, max 5 tries).
Only retry on transient errors (network errors, 5xx server errors). Avoid retrying on client errors like 4xx unless you know they are safe to retry.

Example exponential backoff with jitter (pseudo-code):

# Pseudo-code
attempt = 0
max_attempts = 5
base_delay = 1.0  # seconds

while attempt < max_attempts:
    attempt += 1
    try:
        return call_remote_service()
    except TransientError:
        sleep_time = min(base_delay * (2 ** (attempt - 1)), 30)
        # Add small random jitter
        sleep_time *= (0.5 + random.random() * 0.5)
        sleep(sleep_time)
# if we get here, return error to caller

Note: Reads (GETs) are usually safe to retry because they do not change state. Writes (POST/PUT/PATCH) are potentially dangerous to retry unless you can guarantee the side effects happen at most once — which is where idempotency comes in.

The image is an illustration showing a flowchart of a payment failure and retry mechanism involving a load balancer, app server, and service, with a timeline indicating retry attempts.

Idempotency

Idempotency guarantees that performing the same operation multiple times has the same effect as performing it once. This property makes retries safe for state-changing operations. The elevator-button metaphor is helpful: pressing a call button once or ten times only causes the elevator to come once. A common approach is to attach an idempotency key to each client-initiated request. The service stores the state of processed keys and uses them to deduplicate work. Example behavior with idempotency key:

Client sends a write request with Idempotency-Key: A7Q.
On first receipt of A7Q: the service performs the action, records that A7Q has been completed (and optionally stores the response), and returns success.
On subsequent receipts of A7Q: the service detects A7Q is already processed and returns the stored success result without performing the action again.

The image illustrates a system architecture flowchart for idempotency in processing a "like" request in a social media app. It shows the interaction between a mobile interface, load balancer, app server, service, and database.

Implementation considerations:

Where to store idempotency records: use a durable store that supports fast lookups (a database table, a distributed cache with persistence, etc.).
TTL for idempotency records: keep them long enough to cover retries (minutes to days depending on use-case) but not indefinitely.
Store both the outcome and any meaningful response payload you want to replay to the client.
Ensure the idempotency check and the action are atomic or use a transactional pattern so race conditions cannot lead to double processing.

Example idempotency record (JSON):

{
  "idempotency_key": "A7Q",
  "status": "completed",
  "response": { "payment_id": "pay_12345", "amount": 1000 },
  "created_at": "2025-01-01T12:00:00Z",
  "expires_at": "2025-01-08T12:00:00Z"
}

When to use idempotency keys:

Payments and billing operations
Order creation and inventory reservations
Any state-changing operation where duplicates are harmful

When not to use them:

Non-critical, cheap operations where duplicates are acceptable
Operations where deduplication logic would be more complex or costly than the impact of duplicates

Quick Reference

Topic	Recommendation	Example / Notes
Timeout	Set per-call timeout and fail fast	`2s` for user-facing calls; propagate deadlines downstream
Retries	Exponential backoff + jitter; cap attempts	Try immediately, then 1s, 2s, 4s, … up to 5 tries
Idempotency	Use unique idempotency keys for writes	Store `idempotency_key`, `status`, and `response` with TTL

Summary

Combine these three tools:

Timeouts so slow calls fail fast and don’t exhaust resources.
Retries with exponential backoff and jitter, and a capped attempt count so transient errors can recover without creating retry storms.
Idempotency so retries of state-changing operations do not create duplicate side-effects.

Timeouts, retries (with backoff), and idempotency are complementary. Timeouts prevent resource exhaustion, backoff-controlled retries handle transient failures, and idempotency makes retries safe for writes.

Foundations & One Server

Data: Reads and Writes

Media, APIs, and Async Work

Failure, Protection, and Operations

Put It All Together

Timeouts Retries Idempotency

Timeouts

Retries and Backoff

Idempotency

Quick Reference

Summary

Links and References

Watch Video

​Timeouts

​Retries and Backoff

​Idempotency

​Quick Reference

​Summary

​Links and References

Watch Video

Timeouts

Retries and Backoff

Idempotency

Quick Reference

Summary

Links and References