How do I set up a Circuit Breaker for a flaky third-party API dependency?
Question
Our app fetches shipping prices from an external carrier API. When the carrier slows down or goes down, our requests hit the timeout, the Octane/PHP-FPM workers block, and the whole system locks up. How do I set up a Circuit Breaker with Guzzle middleware or a Go library to stop this cascade? How should the system behave in the Open/Closed/Half-Open states?
Answer
Short answer: a circuit breaker alone won’t save you — it saves you when you pair it with an aggressive timeout. Because what kills your workers isn’t the error itself, it’s the 30 seconds of waiting that lead up to it.
What you’re hitting is a classic cascading failure: a slow dependency makes every request wait on the timeout, the pool fills, the system locks. The breaker’s job is to stop calling a failing dependency entirely.
- Get the three states right.
CLOSED: calls pass, failures are counted. Once the threshold trips, goOPEN: don’t call the dependency at all, return a fallback immediately, and stay that way for a cooldown. When the cooldown ends, goHALF-OPEN: let a few trial calls through — on success go back toCLOSED, on failure flip back toOPEN. That’s the heart of it. - Cut the timeout first — that’s the real killer. Keep
connect_timeoutandtimeoutin milliseconds, not seconds (e.g. 800ms-2s). A breaker is useless without a short timeout; set both or the pool fills anyway. - Isolate that dependency with a bulkhead. Cap the number of concurrent requests allowed to the carrier API. That way, when it slows down it consumes only a small slice reserved for it, not all your workers, and the core path (cart, checkout) stays up.
- Define the fallback in advance. Return the last cached shipping price, and if there’s none, say “estimate unavailable right now.” Serve a degraded-but-working experience instead of an error page.
- Pick the right tool. In Laravel wrap the Guzzle client in middleware and keep the state in Redis. In Go don’t hand-roll it —
sony/gobreakeralready gives you these three states.
Bottom line: I’d never leave the breaker alone: I set it up as a trio of short timeout + bulkhead + cached fallback. Workers blocking while they wait on a timeout hits especially hard in a persistent-process architecture (Octane), so cut the timeout first, then add the breaker.
Related Reading
Comments
Sign in with your GitHub account to join the discussion. Comments are stored in GitHub Discussions.