Finding memory leaks in long-running processes

The idea

A memory leak in a long-running process — a CLI batch job, a queue worker, a daemon — is usually code written for a short life. "Register this callback, it gets cleaned up when the request ends" is fine for one request, but the same code inside a process that runs for hours never gets that cleanup. And the way to find one isn't "watch memory go up". It's to pick a specific number to watch and prove what's driving it, rather than guess.

The method: reproduce it fast (make it fail in minutes, not hours), add logging at a checkpoint the process already hits over and over, narrow from a general signal down to one named cause, then confirm with a differential test — change one thing, rerun the same input, compare.

How it shows up

Two leaks found this way in one debugging session, both with the same root cause: a callback registered once per loop iteration instead of once, and never removed.

// leaks: registers a NEW closure every call, all of it captured in the closure's `use`
function record( $data ) {
    add_filter( 'some_filter', function( $x ) use ( $data ) { ... } );
}

// fixed: accumulates into a shared store, registers the filter exactly once
private static $store = [];
private static $filter_added = false;

function record( $data ) {
    self::$store[] = $data;
    if ( ! self::$filter_added ) {
        add_filter( 'some_filter', function( $x ) { /* reads self::$store */ } );
        self::$filter_added = true;
    }
}

To catch this instead of assuming it, hook into a checkpoint the process already runs on a schedule. In WordPress, ElasticPress runs a stop_the_insanity()-style cleanup after every batch — a good place to log memory_get_usage() next to a breakdown of $wp_filter by hook name, sorted by callback count. When one hook's count climbs steadily while the rest stay flat, that's your leak, named.

Read more

• PHP manual: debug_zval_refcount and reference counting
• Valgrind Memcheck (for non-PHP long-running processes)
• WordPress: WP_Hook and how $wp_filter is structured

Exercises

1. Reproduce a toy leak under a hard ceiling. Write a small script (any language) that registers a closure onto a global list in a loop of 100,000 iterations, each closure capturing a ~1KB string. Run it under a memory limit low enough to fail in under 10 seconds (e.g. php -d memory_limit=64M script.php). Done when: it fails with an out-of-memory error, and you can point to the exact line causing growth.
2. Instrument instead of guessing. Add a checkpoint every 10,000 iterations that logs current memory usage and the length of the global list. Confirm the growth is linear, not just "increasing." Done when: you have a printed table of iteration → memory, and can state the approximate bytes-per-iteration cost.
3. Differential test a fix. Change the script to register the closure once (reading from a shared array) instead of once per iteration, matching the "fixed" pattern above. Rerun with the same input size and hard ceiling. Done when: memory stays flat instead of climbing, and you can say by what factor growth dropped.