What is Circuit Breaker?

#How a circuit breaker works

The breaker has three states. Closed is normal operation, with calls passing through while failures are counted. When failures exceed a threshold, the breaker opens and subsequent calls fail immediately without contacting the dependency.

After a cooldown, the breaker moves to half open and allows a limited number of trial calls through. If they succeed, it closes and normal operation resumes. If they fail, it opens again. The half open state is what lets the breaker recover automatically without a human resetting it.

#Why circuit breakers matter

The failure they prevent is the one that turns a single service problem into a system-wide outage.

• Resource protection: callers stop holding threads and connections waiting on a dead dependency.
• Cascade prevention: the failure stops at the breaker rather than propagating upward.
• Recovery assistance: a struggling service is not buried under continued traffic.
• Fast failure: immediate errors are better than timeouts for both users and callers.

#Limits and tuning difficulties

Thresholds are genuinely hard to set. Too sensitive and the breaker opens on normal transient errors, cutting off a healthy dependency. Too tolerant and it opens long after the damage is done.

A breaker that opens silently is also dangerous. It converts a visible failure into an invisible one, so opening must emit a metric and usually an alert. Partial failures complicate things further, since a dependency healthy for most requests and failing for one category will trip a breaker that then blocks the requests that would have succeeded.

#Circuit breaker vs retry

Retries assume the failure is transient and that trying again will work. They are correct for brief network blips and momentary contention.

Circuit breakers assume the failure is persistent and that continuing to call makes things worse. They are correct for a dependency that is down or overloaded. Applied to the wrong situation, each is harmful: retries against an overloaded service add load to an overloaded service, and a breaker on a transient blip cuts off a dependency that was about to recover. Production systems typically use both, with limited retries inside a breaker that trips when retries stop helping.

#Key takeaways

• The breaker cycles through closed, open, and half open, with half open enabling automatic recovery.
• It protects callers from exhausting threads and connections on a dead dependency.
• Threshold tuning is difficult in both directions, and an over sensitive breaker cuts off healthy dependencies.
• Retries suit transient faults and breakers suit persistent ones; each is harmful in the other situation.

#Frequently asked

What should a service return when the breaker is open?

A cached or degraded response where one is meaningful, otherwise a fast explicit error. Fast failure is better than a slow timeout.

How do you tune the threshold?

From observed error rates in normal operation. The threshold should sit clearly above normal transient failure rates and well below the level that indicates real trouble.

Should breaker state be visible?

Always. An open breaker is a significant system state, and a breaker that trips silently hides an active failure.