Skip to content

fix: bound reflexion and judge prompts - #168

Merged
SkyeAv merged 1 commit into
net/llm-retryfrom
net/prompt-bounds
Sep 15, 2026
Merged

SkyeAv merged 1 commit into
net/llm-retryfrom
net/prompt-bounds

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Hard-bounds the reflexion and judge prompts so a pathological article cannot outgrow the model's context window. Fleet distill telemetry: the three largest of 534 LLM calls were all reflexion prompts of 1.6M and 2.3M characters (689,241 and 524,336 input tokens) — past every model's window, failing outright after paying for the serialization. Cause: llm_propose_config_edit interpolated the full coverage report and context with no cap; _build_judge_prompt interpolated the raw build report.

Bounds

  • agent.compact_coverage_report (pure, total, non-mutating): caps every reachable unresolved list (top level, sections[i], sections[i].per_column[col], back-compat top-level per_column) at the first 20 terms (UNRESOLVED_CAP) plus one visible +N more marker — the same convention as compact_audit_report — and adds unresolved_count siblings naming the ORIGINAL lengths so the scale signal survives. Beyond max_chars, per-column unresolved lists are dropped entirely (counts/fractions retained); the serialized block is bounded by the caller, never by parsing truncated JSON back.
  • agent._truncate_for_prompt(text, limit, *, what) with a verbatim …[X truncated: N of M chars omitted] marker; limit <= 0 raises.
  • Budgets: config 8,000 (MAX_PROMPT_CONFIG_CHARS) + coverage 8,000 (COVERAGE_PROMPT_CHARS) + context 40,000 (MAX_PROMPT_CONTEXT_CHARS) + report 8,000 (MAX_PROMPT_REPORT_CHARS) ≈ worst case 58 K chars ≈ 15 K tokens — a ≥ 27× reduction, room left for the ~16 K-token output.
  • Public APIs unchanged: map_coverage, _measure_section, build_and_audit, and the supervisor's tier-1 deterministic proposer still receive full uncapped reports; compaction happens only at the tier-2 prompt-serialization boundary. Never-raises contract and the validate_table_config gate preserved.

Testing

  • uv run pytest tests/test_agent_prompt_bounds.py -q --no-cov -n 013 passed in 1.16s: caps-every-list, purity/non-mutation, totality on hostile shapes, verbatim markers, unresolved_count, 50,000-term regression (< 60 K prompt, 3 sections × 2 columns), overflow drop path, 400-section survival, judge compaction (kgx_path absent, both reverse modes), determinism
  • Full gate on the stack tip: uv run pytest -q1565 passed, 15 skipped; ruff/format/pyright clean; CI-equivalent base env passes
  • Tier-2 review caught and fixed a Blocker in the first round: the overflow path parsed a mid-JSON-truncated string back (json.loads always raised, silently disabling reflexion via the caller's blanket except); the parse-back pass was deleted and pinned by overflow tests, re-reviewed PASS (31/31 adversarial checks)

@SkyeAv
SkyeAv added this pull request to stack #170 September 15, 2026 17:40
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 9d791768-fd3e-4266-8de7-78853faccf92


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SkyeAv SkyeAv changed the title fix: [US-006] bound reflexion and judge prompts fix: bound reflexion and judge prompts Sep 15, 2026
@SkyeAv
SkyeAv marked this pull request as ready for review September 15, 2026 18:07
Problem
=======
Fleet distill telemetry showed the three largest of 534 LLM calls were
all reflexion prompts of 1.6M-2.3M characters (689,241 / 524,336 input
tokens) — past every model's context window, so each failed outright
after paying for the serialization. Two uncapped interpolations caused
it: `llm_propose_config_edit` embedded the full
`json.dumps(coverage_report)` (whose `unresolved` lists grow with table
rows, repeated per column, per section, and in the top-level union) and
the full article/table context; `_build_judge_prompt` embedded the raw
`build_and_audit` report whose `unresolved` key is likewise unbounded.

Change
======
- New pure `compact_coverage_report(report, *, max_terms=UNRESOLVED_CAP,
  max_chars=COVERAGE_PROMPT_CHARS)`: caps every reachable `unresolved`
  list (top level, `sections[i]`, `sections[i].per_column[col]`, and the
  single-section back-compat top-level `per_column`) with the first N
  terms in existing sorted order plus ONE visible `f"+{n-N} more"`
  marker — the same convention as `compact_audit_report` — and adds an
  `unresolved_count` sibling naming each ORIGINAL length so the scale
  signal survives. Non-mutating (fresh capped lists, shallow-copied
  owners) and TOTAL (non-dict -> {}, hostile shapes degrade, never
  raise). When the term-capped serialization still exceeds `max_chars`,
  per-column `unresolved` lists are dropped entirely (counts and
  coverage fractions retained); the serialized block itself is bounded
  by the caller — a dict cannot faithfully represent truncated JSON, so
  the function never parses a truncated string back.
- New pure `_truncate_for_prompt(text, limit, *, what)`: fitting text
  passes through; overflow carries
  `"\n…[{what} truncated: {omitted} of {total} chars omitted]"`;
  `limit <= 0` raises ValueError.
- New budgets near `UNRESOLVED_CAP` (reused as the per-list term cap):
  `COVERAGE_PROMPT_CHARS = 8_000`, `MAX_PROMPT_CONFIG_CHARS = 8_000`,
  `MAX_PROMPT_CONTEXT_CHARS = 40_000`, `MAX_PROMPT_REPORT_CHARS =
  8_000`.
- `llm_propose_config_edit` bounds all three interpolated blocks (config
  8K, compacted coverage 8K, context 40K) through `_truncate_for_prompt`
  with visible `what` markers; worst case ≈58K chars ≈15K tokens, down
  from 1.6M+ (≥27x). Never-raises contract and
  `validate_table_config` gate unchanged.
- `_build_judge_prompt` now serializes `compact_audit_report(report)`
  with `json.dumps(..., default=str)` (previously the raw dict via
  f-string) truncated at 8K. `config_yaml` and `metrics` unchanged.
- PUBLIC APIs UNCHANGED: `map_coverage`, `_measure_section`,
  `build_and_audit`, and `compact_audit_report` keep returning full
  uncapped lists; the supervisor's tier-1 deterministic proposer still
  receives the FULL `cov_report` — compaction happens only at the
  tier-2 prompt-serialization boundary. `render_task_context` and the
  inner agent's task prompt are untouched.

Verification
============
- `uv run pytest tests/test_agent_prompt_bounds.py -q --no-cov -n 0`
  -> 13 passed (all <0.03s; pure/offline; plain-callable fake model, no
  smolagents)
- `uv run pytest tests/test_agent_propose.py tests/test_agent_coverage.py
  tests/test_agent_build.py tests/test_agent_eval.py
  tests/test_cover_agent_eval.py tests/test_agent_supervisor.py -q
  --no-cov -n 0` -> 130 passed, 10 skipped
- `uv run pytest -q` -> 1476 passed, 15 skipped (94% coverage)
- `uv run ruff check .` / `uv run ruff format --check .` /
  `uv run pyright` -> clean / 92 files / 0 errors
- CI-equivalent base env (no [agent] extra):
  `UV_PROJECT_ENVIRONMENT=/tmp/us006-baseenv uv run --frozen
  --no-default-groups --group ci --extra qc --extra log pytest
  tests/test_agent_prompt_bounds.py -q --no-cov -n 0` -> 13 passed
- Independent Tier-2 CODE_REVIEWER review: FAIL on first pass (overflow
  path parsed truncated JSON back — always raised, silently disabling
  reflexion via the caller's blanket except); repaired (drop the
  parse-back pass; caller bounds the serialized string) and re-reviewed:
  PASS, 31/31 adversarial checks, no remaining Blocker/Should-fix.
@SkyeAv
SkyeAv merged commit 54f37e2 into main Sep 15, 2026
5 of 9 checks passed
SkyeAv added a commit that referenced this pull request Sep 15, 2026
Cut 19.0.0 and bump the package version in pyproject.toml, CITATION.cff,
and uv.lock.

Major: five breaking changes ship. The `tablassert` console command now
requires the optional `[cli]` extra, since `cyclopts` and `rich` left the
base install (#187). Level-one normalization redefines fullmap keys as
cleaned Unicode-lowercase, Porter2-normalized, byte-ordered token sets and
moves the database to schema v6, so every schema-v5 fullmap is rejected and
must be rebuilt (#171, #172, #174, #175, #176). `build-fullmap --aria2c`
is gone -- aria2c is used automatically whenever the `[aria2]` extra is
installed (#178) -- and `--taxon-allowlist` is gone because the built-in
top-100 experimental-taxon allowlist now applies to every build, guarding
both reuse paths by `META.taxon_allowlist` identity (#153). Logging now
requires the `[log]` extra and is fully disabled without it (#152).

Features: `tablassert.net` is a new stdlib-only transient/permanent
classification and retry seam shared by the agent and the BABEL downloader
(#163, #164), with one bounded jittered retry layer across the inner agent,
reflexion, and judge (#167) and machine-readable `error_code` values on
skipped checkpoint records (#166). `build-kg` gained an automatic ephemeral
shared-prefix TCode cache keyed by content-addressed XXH64 op digests
(#155-#159, #161). The `--distill` corpus became schema-uniform v2 with a
sibling `outcomes.ndjson`, a deterministic `RewardConfig`-tunable reward,
and the new zero-dependency `tablassert distill-weigh` command (#180-#183).
`CLASS_FIELD_OVERRIDES` grants the DAKP sparse qualifier stack on the pinned
association classes (#188).

Fixes: `distill-export` partitions its corpus by content and unions the
schema across record files, so a mixed v1/v2 directory no longer CastErrors
or silently stringifies a column (#184); BABEL retry warnings survive the
loguru sink; PMC downloads are idempotent, atomic, and bounded-parallel
(#165); reflexion and judge prompts are hard-bounded (#168).

Changelog:
- versioned the Unreleased section as 19.0.0 - 2026-09-15
- merged three duplicate `### Added` blocks into one
- moved the two `BREAKING:` entries from `### Changed` into
  `### Breaking Changes` and added a `**Migration:**` note to all five
- added PR links to all 17 entries, which carried none
- added the missing entries: the distill v2 corpus/reward/distill-weigh
  stack (#180-#183), the distill-export schema-drift fix (#184), the
  TCode run-cache detail (#155-#161), and the README badge removal (#154)

Docs: none needed here. Every shipped doc change landed with its own PR,
and the docs source-of-truth gate passes against the bumped tree.

Testing:
- make check -> exit 0
- uv run ruff check . -> All checks passed!
- uv run pyright -> 0 errors, 0 warnings, 0 informations
- uv run pytest -> 1622 passed, 52 skipped
- cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed
- cargo clippy --all-targets -- -D warnings -> clean
- uv lock --check -> resolved 169 packages, lock current
- uv run mkdocs build --strict -> exit 0
- docs SSOT + CLI coverage after the changelog edit -> 204 passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant