cache.add() as an atomic lock

The idea

Django's cache framework (from django.core.cache import cache) is a key/value store that works the same across backends. Three core methods look alike but behave very differently when several callers hit them at once:

• cache.set(key, value) — always writes. No race protection.
• cache.get(key) — reads. Two callers reading at once both see the same state.
• cache.add(key, value, timeout=...) — atomic set-if-absent. Returns True if the key was new, False if it already existed. The atomicity is the whole point.

That makes cache.add a one-line distributed lock with a built-in expiry. On Redis it becomes SET key value NX EX <timeout>.

The classic mistake is if not cache.get(key): cache.set(key, ...). Two workers can both pass the get before either one sets, and now both think they hold the lock. cache.add folds both steps into one atomic operation that Redis (or the database) arbitrates.

How it shows up

The per-host politeness floor in crawl_feeds:

lock_key = f"crawl:host:{host}"
if not cache.add(lock_key, 1, timeout=1800):
    # another fetch claimed this host in the last 30 min — defer
    UserFeed.objects.filter(pk=feed.pk).update(
        next_check_at=now() + HOST_POLITENESS_INTERVAL,
    )
    continue

Two Celery workers crawling at the same time both call cache.add. Exactly one gets True, the other gets False, so we never double-fetch a host. No separate lock object needed.

Read more

• Django cache framework — low-level API
• Redis SET NX — Redis docs

Exercises

1. Watch it fail without add — rewrite the lock as cache.get/cache.set and write a test that simulates two workers (just two consecutive calls in one thread is enough — get returns None for both). Confirm both pass the check. Done when: you have a test that proves get+set is racy and cache.add isn't.
2. Set a TTL by feature, not by guess — find another place in this repo where a temporary key would help (rate limit? dedupe? short-lived flag?) and pick a TTL that maps to the feature's natural lifetime, not a round number. Done when: you can justify the TTL in one sentence ("30 min because the politeness window is 30 min").