Your payment service calls a provider. The provider times out. Your code tries again, and again, and within seconds every instance of your service is hammering a system that was already struggling. The retry that was meant to add resilience has become the outage.
This is one of the oldest failure patterns in distributed systems, and it keeps recurring because retries feel free. They are not. Each one is extra load, extra latency and, if you are careless, an extra side effect. Here is how we think about doing them properly.
Decide what is worth retrying
Not every failure deserves another attempt. Retrying a request that failed because the input was invalid just produces the same failure more slowly.
Retry when the failure is likely to be transient:
- Network timeouts and connection resets.
- Server errors that signal temporary trouble, such as a 503 or a rate limit response.
- Lock conflicts and deadlock errors in your own database.
Do not retry validation errors, authentication failures, or anything that tells you the request itself is wrong. If a provider returns a client error, a second identical request will get a second identical answer.
The harder case is the timeout. A timeout does not tell you the request failed. It tells you that you stopped waiting. The operation may have completed on the other side. This is why retries and idempotency are inseparable: if the operation is not safe to repeat, you have no business repeating it automatically.
Backoff: give the other side room
The simplest retry policy is to try again immediately. It is also the worst. If the dependency is overloaded, instant retries add pressure at exactly the wrong moment.
Exponential backoff fixes the first half of this. After the first failure, wait a short time. After the second, wait twice as long. After the third, twice again. The delay grows quickly enough that a struggling system gets breathing space.
delay = min(cap, base * 2 ** attempt)The cap matters. Without it, the tenth retry might wait for minutes, and your user has long since given up. Pick a base delay that suits the call, perhaps a hundred milliseconds for an internal service, and a cap that matches how long the caller can sensibly wait.
Jitter: stop everyone retrying together
Backoff alone has a flaw. If a hundred clients all fail at the same instant, they will all wait the same delay and all retry at the same instant. The load arrives in synchronised waves.
Jitter adds randomness to break the pattern. Instead of waiting exactly the computed delay, wait a random amount up to it:
delay = random_between(0, min(cap, base * 2 ** attempt))This is often called full jitter, and it spreads retries across the window rather than stacking them at the edges. It looks slightly odd, since some retries will fire almost immediately, but the aggregate behaviour is far gentler on the dependency. There are variations, such as keeping a minimum delay and jittering only above it. The principle is the same: desynchronise your clients.
Budgets: limit the total damage
Backoff and jitter govern the timing of one request. They say nothing about the volume of retries across the whole service. That is where a retry budget helps.
Imagine each service allows retries only up to a fixed fraction of its normal traffic, say a small percentage of requests in a recent window. When the dependency is healthy, almost nothing is retried and the budget is never touched. When it fails broadly, the budget runs out quickly, and the service stops adding retry load rather than multiplying it.
Without a budget, the maths turns against you. If every request is allowed three retries, a complete outage multiplies traffic to the dependency by four at the worst possible moment. And if the call passes through several layers, each retrying three times, the multiplication compounds. A budget turns "up to N attempts per request" into "at most this much extra load overall", which is the number you actually care about.
Retry in one place only
A subtle trap is layered retries. Your HTTP client retries. Your SDK retries. Your service wrapper retries. Your job queue retries the whole job. A single failed call can fan into dozens of attempts, none of which any layer knows about.
Pick one layer to own retries for each call, usually the one closest to the business meaning of the operation, and disable the rest. Document it. When someone debugging an incident asks how many times a payment call is attempted, the answer should be one number, not a research project.
Pair retries with timeouts and breakers
Retries need a deadline. Set an overall time limit for the operation, and make every attempt and every wait respect it. A request that has already taken too long to be useful should fail, not queue another try.
A circuit breaker complements this. After enough consecutive failures, it stops sending calls to the dependency for a short period and fails fast instead. That protects the dependency, and it protects your own threads and connections from piling up behind a dead service. Retries handle brief faults. Breakers handle sustained ones.
One more caution: honour what the dependency tells you. If a response carries a hint about when to come back, such as a Retry-After header on a rate limit, use that instead of your own schedule. The server knows more about its own recovery than your formula does. And never retry in a tight loop inside a request that a user is waiting on; fail quickly and let a background process pick the work up.
Make retries observable
If you cannot see retries, you cannot tune them. Log or count each retry with the reason, the attempt number and the dependency involved. Watch the retry rate as a signal in its own right: a quiet climb in retries is often the first sign of a degrading provider, well before errors reach users.
A short checklist
Before you ship a retry policy, check that:
- The operation is idempotent, or carries an idempotency key.
- Only transient errors are retried.
- Delays grow exponentially, are capped, and include jitter.
- A budget or breaker limits total retry volume.
- Exactly one layer is responsible for retrying.
- There is an overall deadline.
Think of a retry as spending someone else's capacity on your behalf, and ask whether you would spend it that freely if the bill came to you.
Retries are a loan against the future health of your system. Used sparingly and with limits, they hide small faults. Used carelessly, they turn small faults into large ones. The regret comes from having never set the terms.