Retry Strategies: Exponential Backoff, Jitter, and Retry Budgets
Retries turn a brief blip into an invisible non-event, or they turn it into an outage that outlasts the original failure by an hour. The difference is when you retry, how many clients retry together, and who's allowed to retry at all. Here's how to retry without taking down the thing you're waiting on.
A Three-Second Failover, a Forty-Minute Outage
The database fails over to its standby. That's routine: about three seconds of connection errors, and then the new primary is up and accepting queries.
Except the site stays down for forty minutes. Every API server saw its queries fail and retried right away. Their callers timed out and retried too. So did the mobile apps. When the new primary came up, the first thing it received was every retry from every layer at once, many times its normal load. It fell over, which caused more failures, which caused more retries.
Retries are the most common resilience technique and also one of the most common causes of cascading failure. A retry assumes the failure is temporary, and the retry itself can be what keeps it from ending.
What's Worth Retrying
A retry only helps when the next attempt has a reasonable chance of turning out differently. That rules out more errors than you might expect.
| Failure | Retry? | Why |
|---|---|---|
| Connection refused / reset | Yes | Typically a restart, failover, or dropped connection that will clear |
| 503 Service Unavailable, 502, 504 | Yes, with backoff | The server or a proxy is overloaded or between deploys |
| 429 Too Many Requests | Yes, after Retry-After | The server told you exactly when to come back, so wait that long |
| Timeout | Only if the operation is idempotent | The request may have succeeded; you just didn't hear back |
| 500 Internal Server Error | Rarely | Usually a bug, and bugs don't fix themselves in 200ms |
| 400, 401, 403, 404, 422 | Never | The request itself is wrong; it will fail the same way every time |
| 409 Conflict | Not blindly | Re-read the current state and decide again |
The timeout row matters most. After a timeout you don't know whether the server did the work. Retrying a read is harmless. Retrying "charge this card" without an idempotency key is how customers get charged twice. A retry is only safe when the operation is idempotent, either naturally (GET, PUT, DELETE) or because you made it idempotent.
Retries Multiply
The first problem with retries is simple arithmetic. If a client makes 3 retries, one logical request can turn into up to 4 attempts. That's fine at one layer. But in a real call chain, every layer usually has its own retry policy:
Three layers with three retries each means one user tap can produce 64 queries against a database that's already struggling. Add a fourth layer and it's 256. Nobody designed this; each team made a sensible local decision.
🔴 Retry at one layer
Choose a single layer to own retries for a given call path, usually the one closest to the failure that knows whether the operation is idempotent. Every other layer should fail fast and pass the error up. The AWS Builders' Library and Google's SRE book both give this advice, and it's the cheapest fix in this post.
Exponential Backoff: Giving the System Room to Recover
The second problem is timing. Retrying immediately means the struggling service gets hit again while it's still struggling. A fixed delay helps a little, but if the outage lasts longer than the delay, every client is back at the same rate.
Exponential backoff doubles the wait after each failure: 100ms, 200ms, 400ms, 800ms, and so on. A short blip costs almost nothing, since the first retry happens quickly. A long outage gets progressively less load, because clients back off further the longer it lasts. You always pair it with a cap on the delay, so one client doesn't wait eight minutes, and a maximum number of attempts, so a request eventually fails and the caller can decide what to do.
base = 100ms
cap = 10s
max_attempts = 5
for attempt in 0 .. max_attempts - 1:
result = operation()
if result.ok or not retryable(result):
return result
sleep(min(cap, base * 2^attempt)) # 100ms, 200ms, 400ms, 800ms...
return last_errorThat looks correct. It isn't, and the reason is the most useful idea in this post.
The Problem Backoff Doesn't Solve: Synchronization
Failures are rarely independent. When the database fails over, every client that had a request in flight fails in the same instant. With plain exponential backoff, they all compute the same schedule: all retry at +100ms, all at +300ms, all at +700ms. You haven't spread out the load. You've moved the whole stampede to a later time, and it arrives in waves.
Each wave hits the recovering server at many times its capacity. Most requests time out, and the clients back off, in step with each other, until the next wave. The server keeps receiving spikes it can't handle, separated by quiet periods it can't use.
Jitter fixes this by randomizing each client's delay, so a synchronized burst turns into a steady flow the server can actually work through.
Model: 8 new requests every 250ms, plus a burst of clients that all fail at the moment the outage starts. Over capacity, each request's chance of success drops with the square of the overload, standing in for queueing and timeouts.
Start on Exponential and look at the chart. After the outage, you get tall red spikes (the synchronized waves), each far above the capacity line, and nearly flat gaps between them where the server is idle. Now switch to Exp + Full Jitter. The same clients with the same attempt limit spread out into a lower, wider curve, and most of them succeed. A few more things to try:
- Immediate retries use up all their attempts during the outage itself. Those clients fail, and the server did extra work for nothing.
- No Retry has the lowest load, but every request that hit the outage fails. Sometimes that really is the right answer; more on this below.
- Drag the burst size up. Unjittered strategies get worse fast, because every wave grows with the burst.
💡 About the model
The simulator is a deliberately simple model: fixed capacity, a burst of synchronized failures, and success rates that fall as overload grows. Real systems have queues, timeouts, and autoscaling, so the exact numbers don't carry over. The shape does. AWS's analysis of the same problem, with a more careful simulation, reached the same conclusion: backoff without jitter barely reduces contention.
Three Kinds of Jitter
Marc Brooker's 2015 AWS Architecture Blog post compared the common variants. Here they are side by side (n is the attempt number, base and cap as before):
| Variant | Delay | Behavior |
|---|---|---|
| Full jitter | random(0, min(cap, base · 2ⁿ)) | Spreads retries across the whole window. The least total work in AWS's comparison. A good default. |
| Equal jitter | d = min(cap, base · 2ⁿ); d/2 + random(0, d/2) | Guarantees a minimum wait, but spreads less. Did worst of the jittered variants. |
| Decorrelated jitter | min(cap, random(base, previous · 3)) | Each delay is based on the previous one, not the attempt count. About as good as full jitter. |
Use full jitter unless you have a reason not to. The worry that it sometimes retries almost immediately doesn't hold up: some clients retrying early is fine, as long as they aren't all doing it together.
✅ Jitter applies to more than retries
Any time many machines do the same thing on the same schedule, add randomness: cron jobs at midnight, cache TTLs set during a deploy (the thundering herd problem), client reconnects after a server restart, health-check intervals. Synchronization is the underlying bug in all of these, and retries are just one place it shows up.
Retry Budgets: A Limit on Total Retries
Backoff and jitter control when a client retries. They don't control how much retry traffic the whole fleet sends. During a long outage, even well-jittered retries can double or triple the load. A retry budget puts a ceiling on it.
The idea is to allow retries only while they're a small fraction of total traffic. Google's SRE book describes a per-client budget: retry only while retries are below about 10% of requests. Once most requests are failing, the budget runs out and clients stop retrying, which is exactly when retrying would have made things worse. The same idea shows up elsewhere:
| Mechanism | How it limits retries |
|---|---|
| Per-client retry ratio (Google SRE) | Retry only while retries are under about 10% of this client's requests |
| gRPC retry throttling | A token bucket: failures drain tokens, successes refill them, and retries stop when the bucket is below half |
| AWS SDK standard retry mode | A retry quota that each retry draws from and successful requests refill |
| Attempt-count header | Clients mark retries, so an overloaded server can see how much of its load is retries and reject those first |
A retry budget and a circuit breaker solve related problems. The budget limits retries while letting normal traffic through. The breaker stops all traffic to a dependency that's clearly down. Most mature client libraries use both, alongside timeouts.
Deadlines: Knowing When to Give Up
A retry policy measured only in attempts ignores the user who's waiting. If your API has a 2-second SLA and the first attempt took 1.8 seconds to time out, a retry with 400ms of backoff can't possibly finish in time. It's wasted work.
Give each request a deadline, an absolute time by which the answer stops being useful, and pass it down the call chain (gRPC does this natively). Before each retry, check the remaining time: if it's less than a typical successful call takes, stop retrying and fail. Deadlines also stop the amplification from earlier: a downstream service that knows its caller gave up 300ms ago shouldn't still be retrying on that caller's behalf.
⚠️ Metastable failures: when the retries keep the outage going
The worst version of this has a name. A metastable failure is when a system stays down after the original cause is gone, held there by its own reaction to the failure. The trigger was a three-second failover. What kept it going was retry load, cold caches, and queues full of requests whose clients had already given up. Bronson et al. described the pattern in a 2021 paper, and it explains many long outages. Jitter, retry budgets, deadlines, and load shedding all exist to stop that feedback loop.
Should This Call Be Retried?
Is the failure plausibly transient? (connection error, 503, 429, timeout)
Key Takeaways
- Retry only what can succeed next time. Transient network errors, 503s, and 429s qualify. 4xx errors and bugs don't.
- Only retry idempotent operations. A timeout means you don't know whether the work happened. Idempotency keys make retrying writes safe.
- Retries multiply across layers. Three layers with three retries each is 64× amplification. One layer should own retries.
- Backoff without jitter just moves the stampede. Synchronized clients retry in synchronized waves. Full jitter spreads them into load the server can absorb.
- Limit total retry volume with a budget. When most requests are failing, stop retrying and let the system recover.
- Retry within a deadline, not just a count. A retry that can't finish before the caller gives up is wasted work.
- Jitter applies to anything scheduled. Cron jobs, TTLs, reconnects, and health checks all synchronize the same way.
References
- Marc Brooker, Exponential Backoff and Jitter (AWS Architecture Blog), the source of the jitter comparison
- AWS Builders' Library: Timeouts, retries, and backoff with jitter
- Google SRE Book: Handling Overload and Addressing Cascading Failures, on retry budgets and cascading failures
- gRPC proposal A6: Client Retries, including retry throttling
- Bronson et al., Metastable Failures in Distributed Systems (HotOS 2021)
- The Circuit Breaker Pattern: Failing Fast to Stay Alive