Circuit Breaker Pattern: Stop Hammering a Service That Is Already Down

Your payment provider starts timing out. Every request in your app waits 10 seconds, retries three times, then fails anyway. PHP-FPM workers pile up waiting on a service that is already down, and now your app is down too. Retries with backoff help with short blips, but when a dependency is truly broken, the kindest thing you can do is stop calling it for a while. That is what a circuit breaker does.

The Three States

A circuit breaker wraps calls to a dependency and tracks how they go. It lives in one of three states:

• Closed: normal operation. Calls go through, failures are counted. A success resets the counter.
• Open: the threshold was hit. Calls fail immediately with no network round trip, so your workers are freed in microseconds instead of seconds.
• Half-Open: the cooldown has passed. One trial call is allowed through. If it succeeds, the circuit closes; if it fails, it opens again for another cooldown.

Implementation

The clock is injected so the breaker can be tested without sleeping.

final class CircuitOpenException extends RuntimeException {}

final class CircuitBreaker
{
    private string $state = 'closed';
    private int $failures = 0;
    private ?float $openedAt = null;

    public function __construct(
        private int $failureThreshold = 5,
        private float $cooldownSeconds = 30.0,
        private ?Closure $clock = null,
    ) {
        $this->clock ??= fn (): float => microtime(true);
    }

    public function call(callable $operation): mixed
    {
        if ($this->state === 'open') {
            if (($this->clock)() - $this->openedAt < $this->cooldownSeconds) {
                throw new CircuitOpenException('Circuit is open, failing fast.');
            }

            $this->state = 'half_open';
        }

        try {
            $result = $operation();
        } catch (Throwable $e) {
            $this->recordFailure();

            throw $e;
        }

        $this->reset();

        return $result;
    }

    public function state(): string
    {
        return $this->state;
    }

    private function recordFailure(): void
    {
        $this->failures++;

        if ($this->state === 'half_open' || $this->failures >= $this->failureThreshold) {
            $this->state = 'open';
            $this->openedAt = ($this->clock)();
        }
    }

    private function reset(): void
    {
        $this->state = 'closed';
        $this->failures = 0;
        $this->openedAt = null;
    }
}

import time
from typing import Callable


class CircuitOpenError(Exception):
    pass


class CircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        cooldown_seconds: float = 30.0,
        clock: Callable[[], float] = time.monotonic,
    ) -> None:
        self.failure_threshold = failure_threshold
        self.cooldown_seconds = cooldown_seconds
        self.clock = clock
        self.state = "closed"
        self.failures = 0
        self.opened_at: float | None = None

    def call[T](self, operation: Callable[[], T]) -> T:
        if self.state == "open":
            if self.clock() - self.opened_at < self.cooldown_seconds:
                raise CircuitOpenError("Circuit is open, failing fast.")
            self.state = "half_open"

        try:
            result = operation()
        except Exception:
            self._record_failure()
            raise

        self._reset()
        return result

    def _record_failure(self) -> None:
        self.failures += 1
        if self.state == "half_open" or self.failures >= self.failure_threshold:
            self.state = "open"
            self.opened_at = self.clock()

    def _reset(self) -> None:
        self.state = "closed"
        self.failures = 0
        self.opened_at = None

Sharing State in Laravel

The class above keeps its state in memory. In a long-running Python worker that is fine, but under PHP-FPM every request starts with a fresh object, so the breaker never remembers a single failure. In Laravel, keep the state in a shared cache (Redis) instead:

use Illuminate\Support\Facades\Cache;

final class CachedCircuitBreaker
{
    public function __construct(
        private string $name,
        private int $failureThreshold = 5,
        private int $cooldownSeconds = 30,
        private int $failureWindowSeconds = 60,
    ) {}

    public function call(callable $operation): mixed
    {
        $openKey = "circuit:{$this->name}:open";
        $failKey = "circuit:{$this->name}:failures";

        if (Cache::has($openKey)) {
            throw new CircuitOpenException("Circuit {$this->name} is open.");
        }

        try {
            $result = $operation();
        } catch (Throwable $e) {
            Cache::add($failKey, 0, $this->failureWindowSeconds);

            if (Cache::increment($failKey) >= $this->failureThreshold) {
                Cache::put($openKey, true, $this->cooldownSeconds);
                Cache::forget($failKey);
            }

            throw $e;
        }

        Cache::forget($failKey);

        return $result;
    }
}

$breaker = new CachedCircuitBreaker('payment-gateway', failureThreshold: 5, cooldownSeconds: 30);

try {
    $response = $breaker->call(
        fn () => Http::timeout(3)->post($url, $payload)->throw()
    );
} catch (CircuitOpenException) {
    return response()->json([
        'message' => 'The payment provider is unavailable, please try again shortly.',
    ], 503);
}

This shared version is simpler than the in-memory one: when the open key expires, all traffic flows again instead of a single trial call. For most apps that is good enough. If the dependency is fragile, add a short Cache::lock() so only one request makes the trial call.

Testing the Breaker

Because the clock is injected, you can move time forward instead of sleeping:

it('opens after the threshold and recovers after the cooldown', function () {
    $now = 1000.0;
    $breaker = new CircuitBreaker(3, 30, function () use (&$now) { return $now; });
    $fail = fn () => throw new RuntimeException('down');

    foreach (range(1, 3) as $_) {
        try { $breaker->call($fail); } catch (RuntimeException) {}
    }

    expect($breaker->state())->toBe('open')
        ->and(fn () => $breaker->call(fn () => 'ok'))->toThrow(CircuitOpenException::class);

    $now += 31;

    expect($breaker->call(fn () => 'ok'))->toBe('ok')
        ->and($breaker->state())->toBe('closed');
});

Choosing the Thresholds

• Failure threshold: 5 failures is a common start. Too low and one hiccup trips the breaker; too high and you keep hammering a dead service.
• Cooldown: 15 to 60 seconds. Long enough for the dependency to restart or scale, short enough that users do not wait minutes after it recovers.
• Timeouts: a breaker only helps if calls fail fast. Always set a client timeout (Http::timeout(3)); a 30-second default timeout means 30 seconds of blocked workers per call.

Common Pitfalls

• Counting the wrong errors. A 422 Validation failed is the caller's fault, not a sign the service is down. Only count timeouts, connection errors and 5xx responses.
• One breaker for everything. Use one breaker per dependency (payment-gateway, sms-provider), so an SMS outage does not block payments.
• No fallback. Decide what the user sees when the circuit is open: a cached value, a queued job to retry later, or a clear 503 message.
• Retrying inside the breaker without limits. Put backoff retries inside the operation and keep them small, or the breaker sees one failure for what was actually five slow calls.

When to Reach for This

Use a circuit breaker around any remote dependency that can be slow or down and that sits on a hot path: payment gateways, SMS and email providers, third-party APIs, even an internal microservice. Pair it with exponential backoff: backoff handles short blips, the breaker handles real outages.