> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Timeouts Retries Idempotency
> How to design distributed systems using timeouts, retries with backoff and idempotency to safely handle network failures and avoid duplicate side effects
You know the feeling: you tap Pay, the spinner keeps spinning, and you aren't sure whether the payment went through. Tap again and you risk paying twice. Or maybe the payment never happened at all.
This article shows how to design distributed systems so the Pay button (and similar user actions) are safe to press more than once. We cover the three core tools you should combine: timeouts, retries with backoff, and idempotency.
All components in this design communicate via API calls, and every call crosses a network. Networks are inherently unreliable: calls can be lost, delayed, duplicated, or reordered. Robust systems anticipate these worst-case scenarios rather than assuming responses always arrive.
A common mistake is to write code that assumes a response will always arrive. In distributed systems, “no response” is normal. From the caller’s perspective, a slow service and a dead service look the same: no reply. Since you generally cannot reliably distinguish the two, set clear timeouts and design behavior for timeout events.
## Timeouts
When your application server calls another service, it cannot wait forever. If the downstream service is slow or down, blocking indefinitely ties up threads, sockets, and other resources and causes requests to pile up. That slows the server and can eventually crash it.
Best practices for timeouts:
* Set a sensible per-call timeout (e.g., 1–5 seconds for user-facing APIs). Tune based on observed latency.
* Prefer short timeouts for synchronous user flows; allow longer timeouts for internal background workflows.
* Fail fast: when a timeout triggers, return a clear error so caller logic can decide to retry or take an alternate path.
* Propagate deadline information where possible (for example, include an explicit deadline header) so downstream services can respect caller expectations.
Example: pseudo-code showing a client call with a timeout.
```javascript theme={null}
// Pseudo-code (Node/Fetch-like)
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), 2000); // 2s timeout
try {
const resp = await fetch(url, { signal: controller.signal });
// handle response
} catch (err) {
// timeout or network error
} finally {
clearTimeout(timeout);
}
```
Important: a timeout only stops the caller from waiting. It does not tell you whether the remote service completed the work. The request may have failed before reaching the service, or the service may have completed the work but its response was lost. Treat timeouts as "unknown" outcomes until you have an application-level mechanism (like idempotency or a transaction trace) to determine final state.
## Retries and Backoff
Many errors are transient: a short network blip, a DNS hiccup, or a service restart. Retrying failed calls can often recover automatically. But retries must be performed carefully.
The dangerous pattern is immediate, aggressive retries from many callers. If a database or service is struggling and every client retries at once, retries amplify load and can push the system into complete failure — the classic retry storm.
Do not retry blindly. Immediate repeated retries across many clients can worsen outages (retry storms). Limit retries and add backoff and jitter.
Recommended retry pattern:
* Try immediately once (fast-fail).
* On failure, retry with exponential backoff (e.g., wait 1s, then 2s, then 4s).
* Add jitter (randomized small variance) to avoid synchronized retries across clients.
* Cap the maximum wait and the number of attempts (for example, max 5 tries).
* Only retry on transient errors (network errors, 5xx server errors). Avoid retrying on client errors like 4xx unless you know they are safe to retry.
Example exponential backoff with jitter (pseudo-code):
```python theme={null}
# Pseudo-code
attempt = 0
max_attempts = 5
base_delay = 1.0 # seconds
while attempt < max_attempts:
attempt += 1
try:
return call_remote_service()
except TransientError:
sleep_time = min(base_delay * (2 ** (attempt - 1)), 30)
# Add small random jitter
sleep_time *= (0.5 + random.random() * 0.5)
sleep(sleep_time)
# if we get here, return error to caller
```
Note: Reads (GETs) are usually safe to retry because they do not change state. Writes (POST/PUT/PATCH) are potentially dangerous to retry unless you can guarantee the side effects happen at most once — which is where idempotency comes in.
## Idempotency
Idempotency guarantees that performing the same operation multiple times has the same effect as performing it once. This property makes retries safe for state-changing operations.
The elevator-button metaphor is helpful: pressing a call button once or ten times only causes the elevator to come once.
A common approach is to attach an idempotency key to each client-initiated request. The service stores the state of processed keys and uses them to deduplicate work.
Example behavior with idempotency key:
* Client sends a write request with `Idempotency-Key: A7Q`.
* On first receipt of `A7Q`: the service performs the action, records that `A7Q` has been completed (and optionally stores the response), and returns success.
* On subsequent receipts of `A7Q`: the service detects `A7Q` is already processed and returns the stored success result without performing the action again.
Implementation considerations:
* Where to store idempotency records: use a durable store that supports fast lookups (a database table, a distributed cache with persistence, etc.).
* TTL for idempotency records: keep them long enough to cover retries (minutes to days depending on use-case) but not indefinitely.
* Store both the outcome and any meaningful response payload you want to replay to the client.
* Ensure the idempotency check and the action are atomic or use a transactional pattern so race conditions cannot lead to double processing.
Example idempotency record (JSON):
```json theme={null}
{
"idempotency_key": "A7Q",
"status": "completed",
"response": { "payment_id": "pay_12345", "amount": 1000 },
"created_at": "2025-01-01T12:00:00Z",
"expires_at": "2025-01-08T12:00:00Z"
}
```
When to use idempotency keys:
* Payments and billing operations
* Order creation and inventory reservations
* Any state-changing operation where duplicates are harmful
When not to use them:
* Non-critical, cheap operations where duplicates are acceptable
* Operations where deduplication logic would be more complex or costly than the impact of duplicates
## Quick Reference
| Topic | Recommendation | Example / Notes |
| ----------- | ------------------------------------------ | ---------------------------------------------------------- |
| Timeout | Set per-call timeout and fail fast | `2s` for user-facing calls; propagate deadlines downstream |
| Retries | Exponential backoff + jitter; cap attempts | Try immediately, then 1s, 2s, 4s, … up to 5 tries |
| Idempotency | Use unique idempotency keys for writes | Store `idempotency_key`, `status`, and `response` with TTL |
## Summary
Combine these three tools:
* Timeouts so slow calls fail fast and don’t exhaust resources.
* Retries with exponential backoff and jitter, and a capped attempt count so transient errors can recover without creating retry storms.
* Idempotency so retries of state-changing operations do not create duplicate side-effects.
Timeouts, retries (with backoff), and idempotency are complementary. Timeouts prevent resource exhaustion, backoff-controlled retries handle transient failures, and idempotency makes retries safe for writes.
## Links and References
* [Exponential backoff (Wikipedia)](https://en.wikipedia.org/wiki/Exponential_backoff)
* [Stripe: Idempotent requests](https://stripe.com/docs/api/idempotent_requests)
* [AWS Architecture: Exponential backoff and jitter](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)
* [Microsoft: Resiliency best practices](https://learn.microsoft.com/azure/architecture/best-practices/resiliency)