Skip to content

fix(tracing): bound earlier-turn history in generation inputs to stop Stop-hook heap OOM - #120

Draft
hassiebp wants to merge 1 commit into
mainfrom
hassiebbot/bounded-generation-history-569a
Draft

hassiebp wants to merge 1 commit into
mainfrom
hassiebbot/bounded-generation-history-569a

Conversation

@hassiebp

Copy link
Copy Markdown
Collaborator

Fixes #119 (Linear LFE-17701, langfuse/langfuse#18995).

Problem

Every generation's input was built from the entire history of all earlier turns, and each input is serialized onto its own span. Memory therefore grew with steps × session size. On long sessions, the Stop hook crashed with a V8 heap OOM. That crash is a hard abort() that skips the fail-open try/catch, and it happens before markTurnUploaded, so every later Stop hook had an even bigger backlog to process and crashed too.

Fix

  • New option max_history_chars (default 200000), set in langfuse.json or with LANGFUSE_CODEX_MAX_HISTORY_CHARS. For each turn, the history from earlier turns is built once. It walks back from the most recent message until the budget is used, and replaces everything older with a single [N earlier message(s) omitted …] system marker.
  • Generations whose history was cut get codex.history_omitted_messages metadata, so the loss of detail is visible in the trace.
  • The current turn's own user message and earlier steps are never dropped.
  • The old code eagerly copied the full history prefix for every turn ([...seen]). That is replaced by per-turn message arrays and a bounded history computed lazily, only for turns that are actually emitted.
  • Invalid values (negative numbers, non-numbers) are ignored and the default is used. The rest of the config file is still kept.
  • The new option is documented in the README.

Verification

  • pnpm test: 125 tests pass, including new ones for the truncation and the marker/metadata and for config parsing. pnpm lint passes.
  • Offline repro with the endpoint unreachable, on a synthetic 21 MB rollout (24 turns × 15 steps × 60 KB tool outputs, all turns pending):
Build Heap limit Result
main 4096 MB completes, peak RSS 3.4 GB, 58 s
main 1024 MB SIGABRT, heap OOM after 5 s
this PR 4096 MB completes, peak RSS 746 MB, 7.6 s
this PR 1024 MB completes, peak RSS 744 MB

Not in this PR

Some follow-ups from the issue are left out: flushing between pending turns, and capping individual tool outputs. Memory within a single very long turn still grows with that turn's own steps.

Linear Issue: LFE-17701

Open in Web Open in Cursor 

…story_chars)

Every generation's input carried the full history of all earlier turns, so
memory grew with steps x session size and long sessions hit a V8 heap OOM in
the Stop hook, which aborts before turns are marked uploaded and fails on every
later Stop. Keep the most recent earlier-turn messages within a character
budget (default 200k, LANGFUSE_CODEX_MAX_HISTORY_CHARS), replace the rest with
an omission marker, and record the count in codex.history_omitted_messages.

Fixes #119

Co-authored-by: Hassieb Pakzad <hassiebp@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stop hook V8 heap OOM on long sessions: every generation rebuilds its full input from the entire session history (O(turns × history))

2 participants