How to choose a timeout for a service call

Choose a timeout from the dependency’s measured latency and from the time your own caller is willing to wait. Start a little above the dependency’s p99 latency, make sure the timeout plus any retry fits inside your caller’s deadline, and pass the remaining deadline down so the next service does not keep working on a request nobody is waiting for.

What a timeout actually decides

Every timeout answers one question: how long will we wait before we treat a slow answer as a failure? Too long, and one slow dependency ties up your threads, connections and memory until your own service stops answering. Too short, and you fail requests that would have succeeded, and possibly retry them, adding load to a dependency that is already slow.

Many services ship with library defaults, such as 30 or 60 seconds, or no timeout at all. Those numbers were not chosen for your system. A request that takes 30 seconds to fail is, for most users, worse than one that fails in one second with a clear error.

Step 1: start from the dependency’s latency

Look at the dependency’s latency distribution under normal load, not its average. Averages hide the slow tail that timeouts exist for.

• If the p99 is 120 ms and the p99.9 is 300 ms, a timeout around 300 to 500 ms lets almost every healthy request finish and cuts off the ones that are stuck.
• If the tail is long and spread out, a single timeout will always cut off some healthy requests. That is a cost to state, not to hide: “about 0.1% of healthy requests time out and are retried.”

Measure the latency from the caller’s side. It includes the network, the connection pool wait and serialization, which the dependency’s own dashboard does not show.

Step 2: fit inside the caller’s deadline

Your service has a caller, and that caller has a timeout too. Your timeout to the dependency must be shorter than the time your caller will wait for you, with room for your own work.

Work backwards. If the browser waits 2 seconds for a search results page, the API gateway might allow 1.8 seconds, the search service 1.5 seconds, and the search service’s call to the ranking service 400 ms. If the ranking timeout were 3 seconds, the browser would already have given up while search was still waiting, and every slow ranking call would waste work nobody sees.

A simple rule: each hop’s timeout is smaller than the timeout of the hop that called it.

Step 3: leave room for one retry, or none

A retry only helps if it can finish before the caller’s deadline. With a 1.5 second budget and a 400 ms timeout, you can afford one retry with some backoff. With a 500 ms budget and a 400 ms timeout, a retry can never succeed in time; it only adds load.

Retries also multiply load during exactly the moment the dependency is struggling. If every caller retries twice, a dependency that slows down receives up to three times its normal traffic. Retry only operations that are safe to repeat, use backoff with jitter, and cap retries as a share of traffic (a retry budget, for example no more than 10% extra requests).

Step 4: propagate the deadline

A fixed timeout per hop wastes time when an earlier hop was slow. Deadline propagation fixes this: the first service sets an absolute deadline (“this request must finish by 12:00:01.500”), passes it with the request, and each service sets its timeout from the time that remains. gRPC does this with deadlines; over HTTP you can pass a header with the remaining budget.

With propagation, a service that receives a request with 50 ms left can refuse it immediately instead of starting work that cannot finish.

Step 5: decide what happens when it fires

A timeout is half a decision. The other half is what the user gets:

• Fail fast with a clear error when the result is essential and the call changed nothing, such as loading an account page.
• Check before you answer when the call may have changed something. A payment that times out may still have gone through: look up its status by its idempotency key before you tell the user it failed, and never charge again blindly.
• Serve a fallback when the result is optional, such as recommendations, or results in a simpler order.
• Accept the request and finish later when the work can be queued, and tell the user so.

Write down which one each call uses. “Ranking times out at 400 ms; search then returns the matches newest first and counts the fallback on a dashboard” is a design. “Ranking has a timeout” is not.

Common questions

Should a timeout be based on the average latency?

No. Base it on the tail, typically the p99 or p99.9 measured from the caller. Averages hide the slow requests that timeouts exist to cut off.

What is the difference between a connection timeout and a request timeout?

A connection timeout limits how long you wait to open a connection; it is usually short, because a healthy host accepts connections quickly. A request (or read) timeout limits how long you wait for the answer once connected. Set both; a missing request timeout is the one that hangs services.

Should every service in a chain use the same timeout?

No. Each hop needs a shorter timeout than the one that called it, or better, a timeout derived from the deadline passed down with the request.

How do I know my timeout is right?

Watch two numbers after you set it: the rate of timeouts while the dependency is healthy (it should be tiny and stable) and the time your service takes to recover when the dependency is slow (it should recover as soon as the dependency does, without a pile of stuck requests).