Rolling vs Blue-Green vs Canary: Deploying Without Breaking the Database

The deploy looked routine: rename customers.name to customers.full_name, update the model, ship. The migration ran first, the rolling update started second, and for the four minutes it took Kubernetes to replace eight pods, every old pod still running was querying a column that no longer existed. Checkout returned 500s for a third of users. Rolling back the code did not help either, because the old code needed the old column and the migration had already dropped it.

Picking a deployment strategy is not just about how traffic moves from version N to N+1. It is about what happens while both versions run at the same time, against the same database, the same queues and the same caches. Every strategy has that window. The differences are how long it lasts, how much traffic sees the new version, and how fast you can get out.

Rolling Updates: The Default

A rolling update replaces pods a few at a time. It is the Kubernetes default and costs no extra infrastructure:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
spec:
  replicas: 8
  minReadySeconds: 20          # a new pod must stay ready this long before it counts
  progressDeadlineSeconds: 600
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%            # up to 2 extra pods during the rollout
      maxUnavailable: 0        # never drop below 8 ready pods

maxUnavailable: 0 with a non-zero maxSurge keeps full capacity during the rollout. minReadySeconds stops a pod that passes readiness and then crashes ten seconds later from counting as "available", which otherwise lets a broken version roll through the whole fleet. If the rollout stalls past progressDeadlineSeconds, the Deployment is marked as failed, and kubectl rollout undo deployment/shop-api brings back the previous ReplicaSet.

The trade-off: during the rollout, users bounce between versions request by request. Old and new code must be compatible with each other's data, sessions and queued jobs. Rollback is another rolling update, so it takes as long as the deploy did.

Blue-Green: Two Full Stacks, One Switch

Blue-green runs the new version (green) next to the old one (blue) at full size, verifies it, then flips traffic in one step:

# Both stacks run side by side; the Service selector decides who gets traffic.
kubectl apply -f shop-api-green.yaml
kubectl rollout status deployment/shop-api-green --timeout=5m

# Smoke test green through its own internal Service before the switch.
curl -fsS http://shop-api-green.internal/ready

# Switch: one atomic change to the selector.
kubectl patch service shop-api -p '{"spec":{"selector":{"app":"shop-api","color":"green"}}}'

# Rollback is the same command with color=blue. Keep blue running until you are sure.

The switch is near-instant and so is the rollback, as long as blue is still running. You can test green with production config and real dependencies before any user sees it. The costs: double capacity during the deploy, and long-lived connections (WebSockets, streaming responses) that stay on blue until they close. The database is still shared, so blue-green does nothing to make a breaking migration safe. It only makes the moment of the switch sharper.

Canary: Real Traffic, Small Blast Radius

A canary sends a small slice of real traffic to the new version, compares its metrics with the stable version, and widens the slice step by step: 5%, 25%, 50%, 100%. Service meshes and tools like Argo Rollouts or Flagger automate this; the core idea fits in a few lines of nginx:

# nginx: send ~5% of clients to the canary, sticky per client address
split_clients "${remote_addr}" $shop_backend {
    5%      canary;
    *       stable;
}

upstream stable { server 10.0.1.10:8080; server 10.0.1.11:8080; }
upstream canary { server 10.0.2.10:8080; }

server {
    listen 80;
    location / { proxy_pass http://$shop_backend; }
}

Canaries catch what staging never will: real data shapes, real client versions, real load. They need two things to work. First, enough traffic for the metrics to mean anything; 5% of a service with 20 requests per minute tells you nothing in ten minutes. Second, an automated, pre-agreed decision rule. A human staring at a dashboard at 5% will rationalise a small error spike every time.

Choosing a Strategy

Notice that the database question comes first. No traffic strategy fixes a schema change that breaks the version still running.

Database Compatibility: Expand, Migrate, Contract

The rule that makes every strategy safe: each release must work with the schema of the release before and the release after it. That means breaking changes are split across several deploys:

1. Expand: add the new column or table, nullable, and backfill it. Old code ignores it.
2. Dual write: ship code that writes both name and full_name and reads the old one.
3. Switch reads: ship code that reads full_name. Rolling back to step 2 is still safe.
4. Contract: once nothing reads or writes name, drop it in a later release.

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;

// Release 1 (expand): add the new column, nullable, no reads depend on it yet.
return new class extends Migration
{
    public function up(): void
    {
        Schema::table('customers', function (Blueprint $table): void {
            $table->string('full_name')->nullable()->after('name');
        });

        // Backfill in chunks so the migration never locks the table for long.
        DB::table('customers')->whereNull('full_name')->chunkById(1000, function ($rows): void {
            foreach ($rows as $row) {
                DB::table('customers')->where('id', $row->id)->update(['full_name' => $row->name]);
            }
        });
    }
};

The same thinking applies beyond SQL. Queued jobs serialized by version N will be processed by version N+1, so job payloads need to stay backward compatible for at least one release. Cache keys that change format should get a new prefix rather than overwrite old entries. API responses consumed by mobile apps outlive every deploy you will ever do.

Feature Flags: Deploy Is Not Release

Feature flags separate shipping code from turning behaviour on. With Laravel Pennant, the new checkout can be deployed to 100% of pods while only staff and 5% of users see it:

namespace App\Providers;

use App\Models\User;
use Illuminate\Support\ServiceProvider;
use Laravel\Pennant\Feature;

final class FeatureServiceProvider extends ServiceProvider
{
    public function boot(): void
    {
        // Code ships dark; the flag decides who sees it. Stable per user.
        Feature::define('new-checkout', function (User $user): bool {
            return $user->is_staff || $user->id % 100 < 5;
        });
    }
}

// In a controller:
// return Feature::active('new-checkout') ? $this->newCheckout($cart) : $this->legacyCheckout($cart);

Now the risky part (the new behaviour) is controlled by a flag that changes in seconds, and the deploy itself becomes boring. Flags are also the fastest rollback you have: turning one off does not need a build, a pipeline, or a database migration. Hashing on a stable key (here the user id) matters: users should not flip between old and new checkout on every request.

Automating the Canary Decision

Write the decision rule as code, agree on thresholds before the deploy, and let the pipeline call it after each step:

namespace App\Deploy;

final class CanaryJudge
{
    public function __construct(
        private int $minRequests = 500,
        private float $maxErrorDelta = 0.01,
        private float $maxLatencyRatio = 1.2,
    ) {}

    /** @param array{requests: int, errors: int, p95_ms: int} $baseline */
    public function decide(array $baseline, array $canary): string
    {
        if ($canary['requests'] < $this->minRequests) {
            return 'wait';
        }

        if ($this->errorRate($canary) - $this->errorRate($baseline) > $this->maxErrorDelta) {
            return 'rollback';
        }

        if ($baseline['p95_ms'] > 0 && $canary['p95_ms'] > $baseline['p95_ms'] * $this->maxLatencyRatio) {
            return 'rollback';
        }

        return 'promote';
    }

    private function errorRate(array $m): float
    {
        return $m['requests'] > 0 ? $m['errors'] / $m['requests'] : 0.0;
    }
}

Three outcomes, not two. wait stops the pipeline from promoting on a statistically meaningless sample. Comparing against the live baseline instead of a fixed number keeps the rule honest during an incident that affects both versions equally.

Testing the Deploy Gate

The judge decides whether broken code reaches every user, so it deserves tests that pin down each branch. Pest keeps them readable:

use App\Deploy\CanaryJudge;

$baseline = ['requests' => 20_000, 'errors' => 20, 'p95_ms' => 180];

it('waits until the canary has enough traffic', function () use ($baseline) {
    $canary = ['requests' => 120, 'errors' => 0, 'p95_ms' => 150];

    expect((new CanaryJudge())->decide($baseline, $canary))->toBe('wait');
});

it('rolls back when the error rate jumps', function () use ($baseline) {
    $canary = ['requests' => 1_000, 'errors' => 40, 'p95_ms' => 170];

    expect((new CanaryJudge())->decide($baseline, $canary))->toBe('rollback');
});

it('rolls back on a latency regression even with zero errors', function () use ($baseline) {
    $canary = ['requests' => 1_000, 'errors' => 0, 'p95_ms' => 260];

    expect((new CanaryJudge())->decide($baseline, $canary))->toBe('rollback');
});

it('promotes a healthy canary', function () use ($baseline) {
    $canary = ['requests' => 1_000, 'errors' => 2, 'p95_ms' => 190];

    expect((new CanaryJudge())->decide($baseline, $canary))->toBe('promote');
});

Beyond the unit tests, rehearse the rollback itself: in staging, deploy, roll back, and check that the previous version still works against the migrated schema. A rollback path you have never exercised is a hope, not a plan.

Common Pitfalls

Destructive migrations in the same release as the code change. Dropping or renaming a column while old pods still run is the most common deploy outage. Expand first, contract later.

Readiness that passes too early. If /ready returns 200 before caches are warm or config is loaded, the rolling update moves on and a broken version spreads. Combine a real readiness check with minReadySeconds.

Canary without session or cache isolation. A canary that writes a new cache format can poison the cache for stable pods. Version cache keys and session payloads.

Rollback that assumes the database rolls back too. It usually cannot. Data written by the new version must be readable by the old one, which is exactly what expand and contract give you.

Flags that never get removed. Every flag doubles the code paths to test. Put a removal ticket in the same pull request that adds the flag.

When to Use Which, and When Not To

Rolling is right for most stateless web services with backward-compatible changes. Do not use it when two versions must never serve traffic at the same time, for example a protocol change between client and server.

Blue-green fits when you need an instant, atomic switch and a fast rollback, and can afford double capacity for the deploy window. It is a poor fit for large stateful systems where duplicating the stack is expensive.

Canary pays off for high-traffic, high-risk changes where real traffic is the only meaningful test. Skip it for low-traffic services where the metrics will never reach significance; use flags and a careful rolling update instead.

Whatever you pick, the database rule does not change: two versions will overlap, so design every release to coexist with its neighbours.