Skip to content

feat(cache): run the handler once for concurrent misses with coalesce=True - #440

Merged
allen0099 merged 1 commit into
masterfrom
feat/252-coalesce-misses
Oct 7, 2026
Merged

allen0099 merged 1 commit into
masterfrom
feat/252-coalesce-misses

Conversation

@allen0099

Copy link
Copy Markdown
Owner

Closes #252

What

@cache(coalesce=True) (off by default) coalesces concurrent misses of one cache key within a process:

  • The first GET to miss renders as usual (the leader).
  • Requests that miss the same key while it runs wait for it (through asyncio.shield, so a cancelled follower does not cancel the others), then read the backend again and are served the stored entry, with ETag, Age and 304 handling like any hit.
  • If nothing was stored (cookie, private, error status, streamed response, handler raised, cancelled), each waiting request runs the handler itself, concurrently, with no queueing.
  • HEAD waits for a running GET but never leads, since its response is never stored.
  • A request whose backend read fails (fail-open) does not wait: the leader could not have stored either.
  • The leader always releases its followers in a finally, however the request ended.

The registry lives in the new fastapi_cachex/_coalesce.py and is keyed by event loop, backend and cache key, so different apps or loops never wait on each other. Each worker process still runs the handler once per cold key; the docs point to get_or_set() stampede protection for a cross-worker lock.

Validation

  • coalesce must be a bool (CacheXError).
  • It needs a positive ttl and is rejected with private or no_cache, which never serve a stored entry (CacheXError).
  • With no_store it is listed in the existing "ignored arguments" UserWarning.

Docs

  • HTTP_CACHING.md: new "Concurrent misses" section.
  • APP_CACHE.md: cross-link from stampede protection.
  • CACHE_FLOW.md: module list, decision logic and the validation list.
  • All in EN and zh-TW. Changelog fragment changelog.d/252.added.md.

Tests

tests/test_cache_coalesce.py (19 tests) covers:

  • one handler run for concurrent misses;
  • followers rendering at once when nothing was stored, when the leader fails and when it is cancelled;
  • a cancelled follower leaving the others waiting;
  • 304 for followers;
  • HEAD following but never leading;
  • separate keys;
  • failed backend reads;
  • coalesce=False;
  • validation.

Each guard was mutation-checked: removing finish(), the shield, the re-read, the registry cleanup, the HEAD rule, the failed-read rule or the validation makes a test fail. The tests wait on events with a 5 s timeout instead of sleeping.

…=True

`@cache(coalesce=True)` coalesces concurrent misses of one key within a
process. The first GET to miss renders as usual; requests that miss the
same key meanwhile wait for it, then read the backend again and are served
the stored entry. If nothing was stored, each waiting request renders
itself, all at once. HEAD waits for a running GET but never leads, and a
request whose backend read fails does not wait.

Off by default. coalesce must be a bool, needs a positive ttl and is
rejected with private or no_cache; no_store ignores it with the usual
warning.

Closes #252
@allen0099 allen0099 added this to the 0.4.2 milestone Oct 5, 2026
@allen0099 allen0099 added enhancement New feature or request http-cache The @cache decorator, cache keys and Cache-Control handling labels Oct 5, 2026
@allen0099
allen0099 merged commit a801db3 into master Oct 7, 2026
15 checks passed
@allen0099
allen0099 deleted the feat/252-coalesce-misses branch October 7, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request http-cache The @cache decorator, cache keys and Cache-Control handling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

@cache: optional stampede protection on a miss

1 participant