What is Circuit Breaker?
A circuit breaker stops calls to a failing dependency after a threshold of failures, returning an error immediately instead of waiting for a timeout. It prevents a failing service from consuming its callers' resources and taking them down as well.
How a circuit breaker works
The breaker has three states. Closed is normal operation, with calls passing through while failures are counted. When failures exceed a threshold, the breaker opens and subsequent calls fail immediately without contacting the dependency.
After a cooldown, the breaker moves to half open and allows a limited number of trial calls through. If they succeed, it closes and normal operation resumes. If they fail, it opens again. The half open state is what lets the breaker recover automatically without a human resetting it.
Why circuit breakers matter
The failure they prevent is the one that turns a single service problem into a system-wide outage.
- Resource protection: callers stop holding threads and connections waiting on a dead dependency.
- Cascade prevention: the failure stops at the breaker rather than propagating upward.
- Recovery assistance: a struggling service is not buried under continued traffic.
- Fast failure: immediate errors are better than timeouts for both users and callers.
Limits and tuning difficulties
Thresholds are genuinely hard to set. Too sensitive and the breaker opens on normal transient errors, cutting off a healthy dependency. Too tolerant and it opens long after the damage is done.
A breaker that opens silently is also dangerous. It converts a visible failure into an invisible one, so opening must emit a metric and usually an alert. Partial failures complicate things further, since a dependency healthy for most requests and failing for one category will trip a breaker that then blocks the requests that would have succeeded.
Circuit breaker vs retry
Retries assume the failure is transient and that trying again will work. They are correct for brief network blips and momentary contention.
Circuit breakers assume the failure is persistent and that continuing to call makes things worse. They are correct for a dependency that is down or overloaded. Applied to the wrong situation, each is harmful: retries against an overloaded service add load to an overloaded service, and a breaker on a transient blip cuts off a dependency that was about to recover. Production systems typically use both, with limited retries inside a breaker that trips when retries stop helping.
Key takeaways
- The breaker cycles through closed, open, and half open, with half open enabling automatic recovery.
- It protects callers from exhausting threads and connections on a dead dependency.
- Threshold tuning is difficult in both directions, and an over sensitive breaker cuts off healthy dependencies.
- Retries suit transient faults and breakers suit persistent ones; each is harmful in the other situation.