Finding memory leaks in long-running processes

The idea

A memory leak in a long-running process (a CLI batch job, a queue worker, a daemon) is usually code that was written assuming a short lifetime — "register this callback, it'll get cleaned up when the request ends" — running instead inside something with no natural end for hours. The fix for finding one isn't "watch memory go up," it's picking a specific number to watch and proving, not guessing, what's driving it.

The core method: reproduce fast (make the leak fail in minutes, not hours), instrument at a checkpoint the target process already visits repeatedly, narrow from an aggregate signal to a specific named cause, then confirm with a differential test — change one thing, rerun the identical input, compare.

How it shows up

Two examples found this way in the same debugging session, same root pattern (a callback registered once per loop iteration instead of once total, never removed):

// leaks: registers a NEW closure every call, all of it captured in the closure's `use`
function record( $data ) {
    add_filter( 'some_filter', function( $x ) use ( $data ) { ... } );
}

// fixed: accumulates into a shared store, registers the filter exactly once
private static $store = [];
private static $filter_added = false;

function record( $data ) {
    self::$store[] = $data;
    if ( ! self::$filter_added ) {
        add_filter( 'some_filter', function( $x ) { /* reads self::$store */ } );
        self::$filter_added = true;
    }
}

To actually catch this rather than assume it: hook into a periodic checkpoint the process already has (in WordPress, ElasticPress calls stop_the_insanity()-style cleanup after every batch — a perfect place to log memory_get_usage() alongside a breakdown of $wp_filter by hook name, sorted by callback count). When one hook's count climbs steadily while everything else stays flat, that's the leak, named.

Read more

• PHP manual: debug_zval_refcount and reference counting
• Valgrind Memcheck (for non-PHP long-running processes)
• WordPress: WP_Hook and how $wp_filter is structured

Exercises

1. Reproduce a toy leak under a hard ceiling. Write a small script (any language) that registers a closure onto a global list in a loop of 100,000 iterations, each closure capturing a ~1KB string. Run it under a memory limit low enough to fail in under 10 seconds (e.g. php -d memory_limit=64M script.php). Done when: it fails with an out-of-memory error, and you can point to the exact line causing growth.
2. Instrument instead of guessing. Add a checkpoint every 10,000 iterations that logs current memory usage and the length of the global list. Confirm the growth is linear, not just "increasing." Done when: you have a printed table of iteration → memory, and can state the approximate bytes-per-iteration cost.
3. Differential test a fix. Change the script to register the closure once (reading from a shared array) instead of once per iteration, matching the "fixed" pattern above. Rerun with the same input size and hard ceiling. Done when: memory stays flat instead of climbing, and you can say by what factor growth dropped.