Health Checks Done Right: Liveness vs Readiness vs Startup Probes

At 02:14 the primary database fails over. It takes about 40 seconds. Annoying, but the app is built to retry. Except that someone added a database query to the liveness probe last quarter, "to make the health check more thorough". Kubernetes sees every pod failing liveness at the same moment and restarts all of them. Now the database is back, but every pod is cold-booting, running config caching and warming connections at the same time, and the 40-second blip becomes a 9-minute outage with a thundering herd on top.

Health checks look like a one-line YAML concern, but each probe type answers a different question and triggers a different action. Mix them up and the platform that is supposed to heal your app amplifies every incident instead.

Three Probes, Three Questions

Kubernetes has three probe types. Docker has one HEALTHCHECK. What matters is the question each one asks and what happens when the answer is "no":

• Liveness: "Is this process broken beyond self-repair?" On failure the kubelet kills the container and restarts it. Only deadlocks, wedged event loops or corrupted in-memory state justify this. A restart fixes nothing outside the container.
• Readiness: "Should this pod receive traffic right now?" On failure the pod is removed from the Service endpoints. Nothing is restarted. When the probe passes again, traffic comes back. This is where dependency awareness belongs, carefully.
• Startup: "Has the app finished booting?" While it has not passed, liveness and readiness are not run. It protects slow starters (cache warmup, migration checks, JIT warmup) from being killed by a liveness probe tuned for steady state.

The defaults are periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1. That one-second timeout bites PHP apps under load more often than any other setting.

The Probe Lifecycle

Read it as two separate loops after startup: liveness decides whether to restart, readiness decides whether to route. A pod can be alive and unready (draining, dependency down, warming up) for as long as needed without being restarted, which is exactly the state you want during a database failover.

What to Check, and What Not To

Liveness should check almost nothing. The request reaching your code and getting a response is the check. In Laravel that is the built-in /up route: it boots the framework, fires a DiagnosingHealth event, and returns 200, or 500 if something throws. No database, no Redis, no external API. If PHP-FPM is out of workers the probe times out, which is a fair signal that the container is stuck.

Readiness may check dependencies this pod cannot work without, and only those. If every request needs MySQL, a failing MySQL check is a reason to stop routing to this pod. But think about the fleet: if all pods share the same database, all of them go unready together and the Service has zero endpoints. Users get connection errors from the load balancer instead of a clean 503 from your app. For shared dependencies, many teams deliberately keep readiness green and let the app return proper errors, circuit-break, or serve cached data.

Never check third-party APIs in any probe. A payment provider outage should degrade checkout, not pull your whole API out of rotation.

Keep probes cheap and bounded. A probe runs every few seconds on every pod. A SELECT COUNT(*) on a large table, or an uncapped connection attempt that waits for the default TCP timeout, turns the probe into load and makes it time out on its own.

Laravel: /up for Liveness, /ready for Readiness

Laravel 11 and later register the health route in bootstrap/app.php. Keep it as the liveness endpoint:

// bootstrap/app.php
use Illuminate\Foundation\Application;
use Illuminate\Foundation\Configuration\Exceptions;
use Illuminate\Foundation\Configuration\Middleware;

return Application::configure(basePath: dirname(__DIR__))
    ->withRouting(
        web: __DIR__.'/../routes/web.php',
        commands: __DIR__.'/../routes/console.php',
        health: '/up', // liveness: boots the framework, touches no dependency
    )
    ->withMiddleware(function (Middleware $middleware): void {})
    ->withExceptions(function (Exceptions $exceptions): void {})
    ->create();

For readiness, add a separate invokable controller that runs fast checks and returns 503 when a critical one fails:

namespace App\Http\Controllers;

use Illuminate\Http\JsonResponse;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Redis;
use Throwable;

final class ReadinessController
{
    public function __invoke(): JsonResponse
    {
        $checks = [
            'database' => ['critical' => true, 'run' => fn () => DB::select('select 1')],
            'redis' => ['critical' => true, 'run' => fn () => Redis::connection()->ping()],
        ];

        $results = [];
        $ready = true;

        foreach ($checks as $name => $check) {
            try {
                ($check['run'])();
                $results[$name] = 'ok';
            } catch (Throwable $e) {
                report($e);
                $results[$name] = 'fail';
                $ready = $ready && ! $check['critical'];
            }
        }

        return response()->json(
            ['status' => $ready ? 'ready' : 'not_ready', 'checks' => $results],
            $ready ? 200 : 503,
        );
    }
}

Register it with Route::get('/ready', ReadinessController::class); outside any auth or session middleware, and exclude both routes from rate limiting. The critical flag is the important design choice: a non-critical dependency (a search index, a recommendations service) can fail without taking the pod out of rotation. Reporting the exception with report() gives you the cause in your error tracker instead of a bare 503 in the kubelet events.

Kubernetes Configuration

Point liveness and startup at /up, readiness at /ready, and size the startup window from measured boot time:

containers:
  - name: app
    image: registry.example.com/shop-api:2026.11.04
    ports:
      - containerPort: 8080
    startupProbe:            # gate: liveness/readiness wait until this passes
      httpGet: { path: /up, port: 8080 }
      periodSeconds: 5
      failureThreshold: 24   # up to 120s to boot
    livenessProbe:           # "is the process wedged?" -> restart
      httpGet: { path: /up, port: 8080 }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:          # "can it serve traffic right now?" -> remove from Service
      httpGet: { path: /ready, port: 8080 }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
    lifecycle:
      preStop:
        exec: { command: ["sleep", "5"] }  # let endpoints update before SIGTERM

failureThreshold * periodSeconds on the startup probe is the maximum boot time you allow: 24 x 5 = 120 seconds here. The preStop sleep covers the gap between the pod being marked terminating and every node's proxy dropping it from the endpoints; without it, a rolling deploy sends a few requests to a container that is already shutting down. That command needs a sleep binary in the image.

Docker and Compose Healthchecks

Plain Docker only has one healthcheck, and it is informational: Docker marks the container healthy or unhealthy but does not restart it (Swarm and Compose depends_on conditions do act on it). Use the liveness endpoint:

# Dockerfile
HEALTHCHECK --interval=15s --timeout=3s --start-period=30s --retries=3 \
  CMD curl -fsS http://127.0.0.1:8080/up || exit 1

--start-period is Docker's version of a startup probe: failures during that window do not count against --retries. Use 127.0.0.1 rather than localhost to avoid IPv6 resolution surprises, and make sure curl (or wget) actually exists in your final image; distroless and slim images often lack both.

When something goes wrong, look at what the platform saw, not what you think the endpoint returns:

docker inspect --format '{{json .State.Health}}' shop-api | jq '.Status, .Log[-1]'
kubectl describe pod shop-api-7c9f8 | grep -A3 -E 'Liveness|Readiness|Startup'
kubectl get endpoints shop-api   # unready pods disappear from this list

Testing Health Endpoints

Health endpoints are code, and the most important property to test is the negative one: liveness must stay green when dependencies fail. A Pest feature test with facade mocks covers it:

use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Redis;

it('keeps /up green even when the database is down', function () {
    DB::shouldReceive('select')->andThrow(new RuntimeException('db down'));

    $this->get('/up')->assertOk();
});

it('reports not ready when a critical dependency fails', function () {
    DB::shouldReceive('select')->andThrow(new RuntimeException('db down'));
    Redis::shouldReceive('connection->ping')->andReturn(true);

    $this->getJson('/ready')
        ->assertStatus(503)
        ->assertJsonPath('checks.database', 'fail')
        ->assertJsonPath('checks.redis', 'ok');
});

it('reports ready when every check passes', function () {
    DB::shouldReceive('select')->andReturn([]);
    Redis::shouldReceive('connection->ping')->andReturn(true);

    $this->getJson('/ready')->assertOk()->assertJsonPath('status', 'ready');
});

The first test is the one that would have prevented the 02:14 incident. If someone later adds a database call to a DiagnosingHealth listener, it fails in CI instead of in production. Back it up with one end-to-end check in your deploy pipeline: start the container, wait for the healthcheck to report healthy, and fail the pipeline if it never does.

Common Pitfalls

Dependencies in liveness. The classic cascading failure. A shared dependency fails, every pod restarts at once, and recovery gets slower and noisier. Liveness checks the process, nothing else.

Same endpoint for liveness and readiness. It forces you to choose between restarts on dependency blips and no traffic control at all. Two endpoints, two questions.

Timeouts tighter than real latency. A one-second timeout on a pod at 90% CPU fails intermittently, the pod flaps between ready and unready, and the load shifts to the others, which then flap too. Measure p99 of the probe under load and set the timeout well above it.

No startup probe for slow boots. Teams compensate with a huge initialDelaySeconds on liveness, which also delays detection of real hangs forever after a restart. Use a startup probe instead.

Probes behind auth, sessions or rate limits. A probe that hits session middleware writes a session per call; one behind a rate limiter fails during traffic spikes, exactly when you need it to be accurate.

Health checks that log every call. At three probes every five seconds across 30 pods, access logs fill with noise. Exclude the probe paths from access logging or sample them.

When to Use Each Probe, and When Not To

Use readiness on every service that receives traffic through a Service or load balancer. It is the probe with the best risk/benefit ratio, because failing it never destroys anything.

Use liveness only when you have a realistic failure mode that a restart fixes: memory leaks that wedge workers, deadlocks, an event loop that stops processing. If you cannot name one, leaving liveness off is a valid choice; a missing liveness probe is safer than a wrong one.

Use startup whenever boot time is longer or more variable than your liveness budget, which is typical for apps that warm caches or wait for config at boot.

Skip dependency checks in readiness when the dependency is shared by the whole fleet and the app can return meaningful errors on its own. Skip HTTP probes entirely for queue workers that expose no port; an exec probe that checks a heartbeat file written by the worker loop is a better fit there.