Skip to content

Circuit Breaker

1. What is it? (2-3 lines)

A Circuit Breaker is a software stability pattern that monitors out-of-process service calls. If downstream failure rates exceed a safety threshold, the breaker "trips" (opens), instantly failing subsequent calls locally without calling the failing downstream service, preventing cascading failures across microservice architectures.

2. Why does it exist?

When a downstream microservice (e.g., Payment Gateway or Recommendation Engine) slows down or crashes, calling threads block waiting for network timeouts (e.g. 30 seconds). Un-isolated calls cause thread pool exhaustion in calling services, cascading a single sub-system outage into a total enterprise system crash.

3. Where is it used?

  • Microservice Mesh & Resilience Libraries: Resilience4j (Java standard), Hystrix (legacy), Istio / Envoy Proxy, Resilience4go.
  • API Gateways: Kong, Ambassador, AWS API Gateway.

4. How does it work? (State Machine Flow)

  1. CLOSED State (Normal Operation):
  2. All requests pass through to the downstream service.
  3. Breaker tracks success/failure metrics over a sliding window (e.g. last 100 requests).
  4. TRIPPED \(\to\) OPEN State (Failure Threshold Exceeded):
  5. If failure rate exceeds threshold (e.g. \(>50\%\) failures or high latency), breaker transitions to OPEN.
  6. Fast-Fail Behavior: All incoming calls fail INSTANTLY locally (CallNotPermittedException / HTTP 503), consuming 0ms and zero downstream network bandwidth.
  7. OPEN \(\to\) HALF-OPEN State (Probe Recovery):
  8. After a wait timer elapses (e.g. 30 seconds), breaker transitions to HALF-OPEN.
  9. Permits a small trial batch of requests (e.g. 5 requests) to pass through to test downstream health.
  10. State Transition:
  11. If trial requests Succeed \(\implies\) Transition back to CLOSED (System Restored!).
  12. If trial requests Fail \(\implies\) Transition back to OPEN (Reset 30s wait timer).
   +-----------------------------------------+
   |                                         |
   v                                         | (Success Rate > Threshold)
[CLOSED] --(Failure Rate > Threshold)--> [OPEN]
   ^                                         |
   |                                         | (Wait Timer Elapses)
   |                                         v
   +-------------- (Success) ---------- [HALF-OPEN]

5. Advantages

  • Prevents cascading microservice failures by fast-failing calls when dependencies are down.
  • Protects thread pools and CPU resources in calling services from socket blocking.
  • Gives failing downstream systems room to recover without being overwhelmed by requests.

6. Limitations

  • Requires configuring fallback strategies (e.g., returning cached data, default values, or empty lists).
  • Complex threshold tuning (sliding window size, failure rate %, half-open trial count).

7. Common interview questions (10)

  1. Walk through the 3 states (CLOSED, OPEN, HALF-OPEN) of a Circuit Breaker state machine.
  2. How does a Circuit Breaker prevent cascading failures in a microservice dependency graph?
  3. What is the difference between Count-Based and Time-Based sliding windows in Resilience4j?
  4. How do Fallback functions (e.g., returning stale cache) work when a Circuit Breaker is OPEN?
  5. How does Istio / Envoy enforce Outlier Detection (Circuit Breaking) at the Service Mesh sidecar layer?
  6. How do Circuit Breakers interact with Retry with Exponential Backoff policies?
  7. What metrics (Prometheus/Grafana) should be monitored to detect tripped Circuit Breakers in production?
  8. What happens if a Circuit Breaker is configured with a sliding window that is too small or too large?
  9. How does Netflix Hystrix's Thread Isolation differ from Resilience4j's Semaphore isolation?
  10. How do you implement a Circuit Breaker for asynchronous reactive streams (RxJava / Project Reactor)?

8. Common misconceptions

  • Misconception: Circuit Breaker retries failed requests. Fact: Circuit Breakers do NOT retry; they block and fast-fail requests to protect systems. Retries are handled by a separate Retry module.
  • Misconception: Circuit Breaker is configured on the downstream server. Fact: Circuit Breaker is implemented on the CLIENT / CALLER side (or service mesh proxy) to isolate call sites.

10. 30-second cheat sheet

  • States: CLOSED (Normal) \(\to\) OPEN (Fast-Fail 0ms) \(\to\) HALF-OPEN (Probe Test).
  • Core Benefit: Fast-Fails broken downstream dependencies to save thread pools and prevent total system collapse.
  • Rule of Thumb: Pair Circuit Breakers with Fallback Responses (e.g. return cached data or default payload).
  • Service Mesh: Envoy implements circuit breaking via outlier_detection without changing application code.