PHP Generators for Memory-Efficient Processing: yield, yield from, send() and LazyCollection

The nightly import worked fine for a year. Then a partner started sending their full order history instead of deltas: one 2.1 GB CSV. The job died on line 1 with Allowed memory size of 536870912 bytes exhausted. Someone bumped memory_limit to 4G, the job got further, and then the container got OOM-killed and took the queue worker with it.

The code was the usual one: read the file into an array, map it into another array, loop. Memory grows with input size, so every larger file is a new incident. Generators flip that around: you process one row at a time and memory stays flat no matter how big the file gets. They have been in PHP since 5.5 and are still underused, so here is the full toolbox: yield, keys, yield from, send(), and how Laravel's LazyCollection builds on them.

Why the Naive Version Explodes

// Loads the whole file, then builds a second full copy as arrays.
function importOrders(string $path): int
{
    $lines = file($path, FILE_IGNORE_NEW_LINES);   // 2 GB of strings
    $header = str_getcsv(array_shift($lines), ',', '"', '');
    $rows = array_map(
        fn (string $line) => array_combine($header, str_getcsv($line, ',', '"', '')),
        $lines,
    );                                              // another few GB of arrays

    foreach ($rows as $row) {
        // insert...
    }

    return count($rows);
}

PHP strings and arrays carry overhead. A 2 GB CSV loaded with file() costs more than 2 GB, and turning each line into an associative array multiplies that several times. Even array_map over a smaller file keeps the source and the result alive at the same time. The fix is not more RAM. The fix is never holding more than one row.

How a Generator Works

Any function that contains yield becomes a generator function. Calling it runs no code; it returns a Generator object. Each time the consumer asks for the next value, the function body resumes from where it paused, runs until the next yield, hands over a value, and freezes again with its local variables intact.

Only the current row exists in memory. The file handle, the header and the line counter live in the generator's paused stack frame. When the consumer stops (a break, an exception, or the generator going out of scope), PHP destroys the generator and runs any pending finally blocks, so the file still gets closed.

Implementation: Streaming a CSV

/**
 * Stream a CSV file one row at a time.
 *
 * @return Generator<int, array<string, string>> line number => associative row
 */
function readCsv(string $path): Generator
{
    $handle = fopen($path, 'rb');
    if ($handle === false) {
        throw new RuntimeException("Cannot open $path");
    }

    try {
        $header = fgetcsv($handle, null, ',', '"', '');
        $line = 1;

        while (($fields = fgetcsv($handle, null, ',', '"', '')) !== false) {
            $line++;
            if ($fields === [null]) {
                continue; // blank line
            }
            yield $line => array_combine($header, $fields);
        }
    } finally {
        fclose($handle); // runs even if the consumer stops early
    }
}

foreach (readCsv('/data/orders.csv') as $line => $row) {
    if ($row['status'] === 'cancelled') {
        continue;
    }
    // process one row; previous rows are already garbage
}

echo round(memory_get_peak_usage() / 1024 / 1024, 1), " MB\n";

A few details that matter in production:

• Keys. yield $key => $value sets the key. Without it, keys are auto-incremented integers like a list. Using the line number as the key gives you precise error messages for free.
• try/finally. Put cleanup in finally, not after the loop. If the consumer breaks early, code after the loop never runs, but finally does when the generator is destroyed.
• Explicit CSV escape. Since PHP 8.4, relying on the default $escape argument of fgetcsv() and str_getcsv() is deprecated. Pass '' to get RFC 4180 behaviour.
• Blank lines. fgetcsv() returns [null] for an empty line. Skip it explicitly or array_combine() will throw a ValueError about mismatched counts.

With this version the peak memory for the 2.1 GB file was about 2 MB, the same as for a 2 KB file.

Composing Generators with yield from

yield from delegates to another generator (or any iterable) and passes its values straight through. Its value as an expression is the inner generator's return value, which is a neat way to aggregate stats while streaming:

/** Concatenate several CSV exports into one stream and report the total. */
function readAll(array $paths): Generator
{
    $total = 0;
    foreach ($paths as $path) {
        // yield from forwards every key/value of the inner generator
        // and evaluates to the inner generator's return value.
        $total += yield from countingReader($path);
    }

    return $total;
}

function countingReader(string $path): Generator
{
    $n = 0;
    foreach (readCsv($path) as $line => $row) {
        $n++;
        yield $line => $row;
    }

    return $n;
}

$stream = readAll(['/data/2026-10.csv', '/data/2026-11.csv']);
foreach ($stream as $line => $row) {
    // keys restart at 2 for each file: they are NOT unique across files
}
echo $stream->getReturn(), " rows\n";

// Pitfall: iterator_to_array() keeps keys by default, so duplicate keys overwrite.
$rows = iterator_to_array(readAll(['/data/a.csv', '/data/b.csv']), preserve_keys: false);

$generator->getReturn() is available only after the generator has finished; calling it earlier throws. And watch the keys: yield from forwards inner keys unchanged, so two files both produce key 2. iterator_to_array() preserves keys by default and silently overwrites, which is how "the import only kept the last file's rows" bugs are born. Pass preserve_keys: false or do not materialize at all.

Two-Way Generators with send()

Generators can also receive values. $gen->send($value) resumes the generator and makes the paused yield expression evaluate to $value. That turns a generator into a small coroutine, which is handy for a batching sink:

/**
 * A coroutine: receives rows through send(), flushes them in batches,
 * and returns the number of rows written when it receives null.
 */
function batchWriter(Closure $flush, int $size = 1000): Generator
{
    $buffer = [];
    $written = 0;

    while (true) {
        $row = yield;              // pauses here until send($row)
        if ($row === null) {
            break;
        }
        $buffer[] = $row;
        if (count($buffer) >= $size) {
            $flush($buffer);
            $written += count($buffer);
            $buffer = [];
        }
    }

    if ($buffer !== []) {
        $flush($buffer);
        $written += count($buffer);
    }

    return $written;
}

$writer = batchWriter(fn (array $rows) => printf("flush %d\n", count($rows)), 2);
$writer->current();               // run to the first yield
foreach ([['id' => 1], ['id' => 2], ['id' => 3]] as $row) {
    $writer->send($row);
}
$writer->send(null);              // flush 2, flush 1
echo $writer->getReturn(), "\n";  // 3

The first send() implicitly runs the generator to its first yield before delivering the value, but calling current() first makes the intent explicit. Most application code never needs send(); reach for it when a consumer-side state machine (batching, rate counting, a parser that receives chunks) reads more clearly as one function than as a class with fields.

Laravel: LazyCollection

LazyCollection wraps a generator and gives you the Collection API (filter, map, chunk, take, each, reduce) while staying lazy: each operation pulls items through one at a time instead of building intermediate arrays. Here is the import job rewritten as an Artisan command:

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\LazyCollection;

final class ImportOrders extends Command
{
    protected $signature = 'orders:import {path}';

    protected $description = 'Stream a large CSV into the orders table';

    public function handle(): int
    {
        $path = $this->argument('path');

        $imported = self::rows($path)
            ->filter(fn (array $row) => $row['status'] !== 'cancelled')
            ->map(fn (array $row) => [
                'external_id' => $row['id'],
                'email' => strtolower(trim($row['email'])),
                'total_cents' => (int) round(((float) $row['total']) * 100),
                'created_at' => $row['created_at'],
            ])
            ->chunk(1000)
            ->reduce(function (int $count, LazyCollection $chunk) {
                DB::table('orders')->upsert($chunk->values()->all(), ['external_id']);

                return $count + $chunk->count();
            }, 0);

        $this->info("Imported {$imported} rows, peak "
            . round(memory_get_peak_usage(true) / 1048576) . ' MB');

        return self::SUCCESS;
    }

    /** @return LazyCollection<int, array<string, string>> */
    public static function rows(string $path): LazyCollection
    {
        return LazyCollection::make(function () use ($path) {
            $handle = fopen($path, 'rb');
            try {
                $header = fgetcsv($handle, null, ',', '"', '');
                while (($fields = fgetcsv($handle, null, ',', '"', '')) !== false) {
                    if ($fields !== [null]) {
                        yield array_combine($header, $fields);
                    }
                }
            } finally {
                fclose($handle);
            }
        });
    }
}

chunk(1000) yields LazyCollection chunks, so you batch the database writes (one upsert per 1000 rows instead of 1000 queries) while still holding at most 1000 rows in memory. upsert keyed on external_id makes the import safe to rerun after a crash halfway through.

Eloquent has lazy readers too, and they are not interchangeable:

use App\Models\Order;

// One query, models hydrated one at a time.
foreach (Order::query()->where('status', 'paid')->cursor() as $order) {
    // ...
}

// Many small queries keyed by id: stable memory, safe while updating rows.
Order::query()->where('status', 'paid')->lazyById(1000)->each(function (Order $order) {
    $order->update(['exported_at' => now()]);
});

cursor() runs one query and hydrates models one by one, but on MySQL the PDO driver buffers the entire result set by default, so very large results can still eat memory. lazy() and lazyById() page through the table with many small queries; lazyById() is the safe choice when you update the rows you are iterating, because it pages by primary key instead of offset.

Testing It

Two things deserve a test: the stream really is lazy, and memory stays bounded on a large input. Generating a 200k-row file in the test takes about a second and catches the regression where someone adds ->all() or ->collect() in the middle of the pipeline, which forces the whole file into memory.

// tests/Feature/ImportOrdersTest.php
use App\Console\Commands\ImportOrders;
use Illuminate\Support\Facades\DB;

function writeCsv(int $rows): string
{
    $path = tempnam(sys_get_temp_dir(), 'orders');
    $h = fopen($path, 'wb');
    fwrite($h, "id,email,total,status,created_at\n");
    for ($i = 1; $i <= $rows; $i++) {
        $status = $i % 10 === 0 ? 'cancelled' : 'paid';
        fwrite($h, "{$i}, User{$i}@Example.com ,19.99,{$status},2026-11-01 10:00:00\n");
    }
    fclose($h);

    return $path;
}

it('streams rows lazily', function () {
    $path = writeCsv(5);
    $rows = ImportOrders::rows($path);

    expect($rows->first())->toMatchArray(['id' => '1', 'status' => 'paid'])
        ->and($rows->take(3)->count())->toBe(3);
});

it('imports a big file in bounded memory', function () {
    DB::table('orders')->delete();
    $path = writeCsv(200_000);
    $before = memory_get_usage();

    $this->artisan('orders:import', ['path' => $path])->assertSuccessful();

    expect(DB::table('orders')->count())->toBe(180_000)
        ->and(DB::table('orders')->where('external_id', '1')->value('email'))->toBe('[email protected]')
        ->and(memory_get_peak_usage() - $before)->toBeLessThan(64 * 1024 * 1024);
});

The memory assertion is deliberately generous. For unit-level checks of pure generator functions, assert on iterator_to_array($gen, false) with small inputs and feed an infinite generator to prove that only the needed items are pulled.

Common Pitfalls

• Iterating twice. A generator is single-use. A second foreach throws Exception: Cannot traverse an already closed generator. LazyCollection::make(fn () => ...) takes a closure exactly so it can create a fresh generator for every pass, which also means each pass re-reads the file.
• Calling count() or ->all() on the stream. Anything that needs the whole sequence materializes it. LazyCollection::count() iterates everything; ->all() builds the full array.
• iterator_to_array() with duplicate keys. Values get overwritten silently. Use preserve_keys: false.
• Cleanup after the loop instead of in finally. Early break skips it and you leak file handles or database cursors.
• Expecting laziness from cursor() on MySQL. The query is lazy, the result buffer is not. Use lazyById() for huge tables.
• Holding references. Pushing each row into $this->seen[] for deduplication rebuilds the memory problem. Keep a set of ids or hashes, or deduplicate in the database.

When to Use Generators and When Not To

Use them when:

• input size is unbounded or large: files, exports, API pagination, database tables;
• you want a pipeline (read, filter, transform, batch, write) without intermediate arrays;
• you need to stop early: take(10) on a lazy stream reads only what it needs;
• you want to separate "how to fetch" from "what to do with each item" without callbacks.

Avoid them when:

• the data is small and you need random access, sorting, or multiple passes; a plain array is simpler and faster;
• you need the total count up front, for example for a progress bar, unless you can get it cheaply from somewhere else;
• the work is CPU-bound per item and you want parallelism; generators are sequential, so use queued jobs per chunk instead;
• debugging lazy chains confuses your team more than the memory saving helps.

Memory usage should depend on the batch size you pick, not on the size of the input file.