Retries, backpressure and load shedding: what to do when a queue backs up
When work arrives faster than you can finish it, you can slow the sender, drop work, or let it wait. How to choose, and how retries make it worse.
5 min read
When work arrives faster than a service can finish it, there are only three things that can happen: the sender slows down (backpressure), some work is dropped on purpose (load shedding), or the work waits in a queue that grows until something breaks. Good designs choose the first two deliberately and bound the third. Retries decide how bad the overload gets, so they belong in the same decision.
Why queues stop helping
A queue absorbs short bursts. It cannot absorb a sustained gap between arrivals and capacity: if 1,200 jobs arrive per second and you finish 1,000, the queue grows by 200 every second, forever.
The cost is simple arithmetic. A job that joins a queue of 60,000 thumbnail jobs, drained at 1,000 a second, waits a full minute before anyone starts it. If users give up after ten seconds, most of that queue is work nobody wants any more, and processing it delays the requests people are still waiting for.
Waiting time also rises steeply as a service nears full use. In the simplest queueing model, a service at 50% of capacity makes requests wait about as long as the work itself; at 90% they wait about nine times as long; near 100% the wait has no limit. That is why systems feel fine until they suddenly do not.
Bound the queue by time, not by memory
Set the queue’s limit from the longest wait that is still useful, not from how much memory you have:
max useful queue length = drain rate × longest useful wait
At 1,000 jobs a second and a ten-second limit, the queue should hold about 10,000 jobs. Anything beyond that is work that will finish too late to matter. Stating the limit this way also tells you what the cost is: past 10,000, new work is refused or dropped.
Backpressure: make the sender slow down
Backpressure pushes the overload back to whoever is sending. Common forms:
- a bounded queue that refuses new items when full;
- an HTTP 429 or 503 response with a
Retry-Afterheader; - a client library that limits how many requests are in flight at once.
Backpressure works when the sender can wait or slow down: a batch job, an internal service, a data pipeline. It works poorly when the sender is a person who will press the button again, or a fleet of devices that will all retry at the same moment.
What it costs: the sender has to handle refusals, and work is delayed at the edge instead of in the middle. State who sees the refusal and what they do next.
Load shedding: drop work on purpose
Load shedding accepts that not everything can be served and chooses what to drop so that the rest finishes in time. The choice is which work to drop:
- Newest first (refuse at the door): simple and fair to work already accepted.
- Oldest first: during a long backlog, the oldest items are often already useless, because the user has left. Some large systems switch a queue to last-in, first-out under overload for this reason.
- By priority: keep payments and sign-ins, drop analytics and prefetches.
- By deadline: drop anything whose caller’s deadline has already passed.
What it costs: some users get an error during overload. Say how many, and which ones. “At twice normal traffic, about half of image previews fail and show a placeholder; uploads themselves are never dropped” is a priced decision.
How retries turn a slowdown into an outage
Retries are a good defence against brief, random failures. During overload they multiply the traffic. If each client tries up to three times, a dependency that starts failing receives up to three times its normal load at the moment it can least handle it. The extra load keeps it failing, which keeps the retries coming. This is a retry storm.
Four rules keep retries safe:
- Retry only what is safe to repeat. Reads, and writes made safe with an idempotency key.
- Back off exponentially, with jitter. Wait longer after each attempt, and randomize the wait so thousands of clients do not retry in the same second.
- Cap retries with a budget. For example, retries may add at most 10% to normal traffic; when the budget is spent, fail fast.
- Respect the deadline. Do not retry if the attempt cannot finish before the caller gives up.
Retries at several layers multiply too. If the client, the gateway and the service each try up to three times, one user action can become 27 calls to the database. Retry at one layer, usually the one closest to the user that knows whether the operation is safe to repeat.
Choosing between them
| Situation | Usually best | Why |
|---|---|---|
| Sender is a batch job or internal service | Backpressure | It can wait, and nothing is lost |
| Sender is people or a large fleet of devices | Load shedding plus backoff on the client | They retry anyway; drop what is least valuable |
| Work loses value quickly | Shed by deadline or oldest first | Late work is wasted work |
| Some work matters more | Shed by priority | Protect the requests the business depends on |
| Failures are brief and random | Retries with backoff and a budget | Most retries succeed |
Most real systems combine them: a bounded queue sized by useful wait, priority shedding when it fills, and clients that back off with jitter inside a retry budget.
Common questions
What is the difference between backpressure and load shedding?
Backpressure slows the sender down by refusing or delaying new work, so nothing is lost but work waits at the edge. Load shedding drops some work on purpose so that the work it keeps finishes in time. Backpressure suits senders that can wait; shedding suits senders that cannot.
How big should a queue be?
Big enough to hold the work you can finish within the longest wait that is still useful: drain rate times that wait. A queue sized by memory alone fills with work that finishes too late.
Why do retries cause outages?
During overload, every failed request that is retried adds load to the system that is already failing. Without backoff, jitter and a retry budget, that extra load keeps the system failing, which causes more retries.
Should I drop the oldest or the newest requests?
Drop the newest when accepted work is still useful and fairness matters. Drop the oldest, or anything past its deadline, when waiting work quickly loses value because the user has already left.