Retrying a Failing Dependency Without a Circuit Breaker Makes the Outage Worse
Key takeaway: Naive retry logic assumes failures are transient blips, and during a genuine sustained outage that assumption actively harms the struggling dependency by adding retry load exactly when it can least absorb it. A circuit breaker recognises sustained failure and stops making it worse.
Why Naive Retries Backfire During a Real Outage
A dependency starts failing due to overload — too many requests for its current capacity. Every caller’s retry logic, built on the reasonable assumption that failures are usually brief transient blips, responds to each failure by retrying shortly afterward.
This means the struggling dependency, already failing because it has too much load, now receives the original request volume plus a wave of retries from every failed attempt, which is additional load precisely when it has the least capacity to handle it — the retries do not help it recover; they actively make recovery harder or impossible, extending an outage that might otherwise have been brief into one that persists because the retry storm itself is now sustaining the overload.
What a Circuit Breaker Does Differently
A circuit breaker tracks the failure rate of calls to a specific dependency and, once failures exceed a threshold, stops sending requests to it entirely for a defined period — failing fast locally instead, without even attempting the call, which removes load from the struggling dependency rather than adding to it.
| State | Behaviour | Purpose |
|---|---|---|
| Closed (normal) | Requests pass through normally | Default operation |
| Open (tripped) | Requests fail immediately, no call attempted | Stops adding load to a struggling dependency |
| Half-open (testing) | A small number of requests allowed through | Tests whether the dependency has recovered |
The half-open state is what allows automatic recovery without requiring a human to manually decide when to resume traffic — after the open period elapses, a small number of test requests are allowed through, and if they succeed, the circuit closes and normal traffic resumes; if they still fail, the circuit reopens for another period, all without needing anyone to watch and intervene manually.
Why Failing Fast Is Actually the Better Outcome
Failing fast locally, the instant the circuit opens, is faster and less resource-intensive for the calling service than waiting for a call to the struggling dependency to time out, which additionally consumes the calling service’s own threads or connections while waiting — under load, a slow dependency without a circuit breaker can exhaust the calling service’s own resources purely through everything waiting on slow calls, which can then cascade the outage into a service that was otherwise perfectly healthy on its own.
This cascading failure mode — one struggling dependency causing an otherwise healthy dependent service to also fail, purely because it exhausted its own resources waiting on slow calls to the struggling one — is one of the most common ways a localised problem in a distributed system becomes a much larger outage than the original failure alone would have caused, and a circuit breaker is specifically the mechanism that prevents this particular escalation.
Combining With a Sensible Fallback
A circuit breaker on its own converts a slow failure into a fast failure, which is an improvement and still a failure the calling code needs to handle sensibly — pairing the circuit breaker with a defined fallback behaviour for when the circuit is open (serving cached or default data, degrading a feature gracefully rather than failing the entire request, queueing the work for later) turns a fast failure into an actual degraded-but-functioning user experience rather than merely a faster error.
Setting the Thresholds Sensibly
The failure rate threshold and the open-state duration should be tuned to the actual dependency’s typical behaviour rather than copied from a generic default — a dependency with naturally higher baseline latency variance needs a more tolerant threshold than one that is normally extremely consistent, and setting these values without understanding the dependency’s real behaviour produces either a circuit that trips on normal variance or one that fails to trip during a genuine sustained problem.
The Bottom Line
Wrap calls to external dependencies in a circuit breaker rather than relying on naive per-call retry logic alone, since retries during a genuine sustained outage add load exactly when the dependency can least absorb it. Pair the circuit breaker with a defined fallback for the open state, and tune thresholds to the dependency’s actual observed behaviour rather than an untested default.
