fix: bound reflexion and judge prompts - #168
Merged
Merged
Conversation
SkyeAv
added this pull request to stack #170
September 15, 2026 17:40
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
force-pushed
the
net/prompt-bounds
branch
from
September 15, 2026 17:47
39557cd to
5be3f59
Compare
SkyeAv
force-pushed
the
net/prompt-bounds
branch
from
September 15, 2026 17:53
5be3f59 to
28f4908
Compare
SkyeAv
marked this pull request as ready for review
September 15, 2026 18:07
SkyeAv
force-pushed
the
net/prompt-bounds
branch
from
September 15, 2026 18:19
28f4908 to
5186566
Compare
Problem
=======
Fleet distill telemetry showed the three largest of 534 LLM calls were
all reflexion prompts of 1.6M-2.3M characters (689,241 / 524,336 input
tokens) — past every model's context window, so each failed outright
after paying for the serialization. Two uncapped interpolations caused
it: `llm_propose_config_edit` embedded the full
`json.dumps(coverage_report)` (whose `unresolved` lists grow with table
rows, repeated per column, per section, and in the top-level union) and
the full article/table context; `_build_judge_prompt` embedded the raw
`build_and_audit` report whose `unresolved` key is likewise unbounded.
Change
======
- New pure `compact_coverage_report(report, *, max_terms=UNRESOLVED_CAP,
max_chars=COVERAGE_PROMPT_CHARS)`: caps every reachable `unresolved`
list (top level, `sections[i]`, `sections[i].per_column[col]`, and the
single-section back-compat top-level `per_column`) with the first N
terms in existing sorted order plus ONE visible `f"+{n-N} more"`
marker — the same convention as `compact_audit_report` — and adds an
`unresolved_count` sibling naming each ORIGINAL length so the scale
signal survives. Non-mutating (fresh capped lists, shallow-copied
owners) and TOTAL (non-dict -> {}, hostile shapes degrade, never
raise). When the term-capped serialization still exceeds `max_chars`,
per-column `unresolved` lists are dropped entirely (counts and
coverage fractions retained); the serialized block itself is bounded
by the caller — a dict cannot faithfully represent truncated JSON, so
the function never parses a truncated string back.
- New pure `_truncate_for_prompt(text, limit, *, what)`: fitting text
passes through; overflow carries
`"\n…[{what} truncated: {omitted} of {total} chars omitted]"`;
`limit <= 0` raises ValueError.
- New budgets near `UNRESOLVED_CAP` (reused as the per-list term cap):
`COVERAGE_PROMPT_CHARS = 8_000`, `MAX_PROMPT_CONFIG_CHARS = 8_000`,
`MAX_PROMPT_CONTEXT_CHARS = 40_000`, `MAX_PROMPT_REPORT_CHARS =
8_000`.
- `llm_propose_config_edit` bounds all three interpolated blocks (config
8K, compacted coverage 8K, context 40K) through `_truncate_for_prompt`
with visible `what` markers; worst case ≈58K chars ≈15K tokens, down
from 1.6M+ (≥27x). Never-raises contract and
`validate_table_config` gate unchanged.
- `_build_judge_prompt` now serializes `compact_audit_report(report)`
with `json.dumps(..., default=str)` (previously the raw dict via
f-string) truncated at 8K. `config_yaml` and `metrics` unchanged.
- PUBLIC APIs UNCHANGED: `map_coverage`, `_measure_section`,
`build_and_audit`, and `compact_audit_report` keep returning full
uncapped lists; the supervisor's tier-1 deterministic proposer still
receives the FULL `cov_report` — compaction happens only at the
tier-2 prompt-serialization boundary. `render_task_context` and the
inner agent's task prompt are untouched.
Verification
============
- `uv run pytest tests/test_agent_prompt_bounds.py -q --no-cov -n 0`
-> 13 passed (all <0.03s; pure/offline; plain-callable fake model, no
smolagents)
- `uv run pytest tests/test_agent_propose.py tests/test_agent_coverage.py
tests/test_agent_build.py tests/test_agent_eval.py
tests/test_cover_agent_eval.py tests/test_agent_supervisor.py -q
--no-cov -n 0` -> 130 passed, 10 skipped
- `uv run pytest -q` -> 1476 passed, 15 skipped (94% coverage)
- `uv run ruff check .` / `uv run ruff format --check .` /
`uv run pyright` -> clean / 92 files / 0 errors
- CI-equivalent base env (no [agent] extra):
`UV_PROJECT_ENVIRONMENT=/tmp/us006-baseenv uv run --frozen
--no-default-groups --group ci --extra qc --extra log pytest
tests/test_agent_prompt_bounds.py -q --no-cov -n 0` -> 13 passed
- Independent Tier-2 CODE_REVIEWER review: FAIL on first pass (overflow
path parsed truncated JSON back — always raised, silently disabling
reflexion via the caller's blanket except); repaired (drop the
parse-back pass; caller bounds the serialized string) and re-reviewed:
PASS, 31/31 adversarial checks, no remaining Blocker/Should-fix.
SkyeAv
force-pushed
the
net/prompt-bounds
branch
from
September 15, 2026 18:41
5186566 to
424ed10
Compare
SkyeAv
added a commit
that referenced
this pull request
Sep 15, 2026
Cut 19.0.0 and bump the package version in pyproject.toml, CITATION.cff, and uv.lock. Major: five breaking changes ship. The `tablassert` console command now requires the optional `[cli]` extra, since `cyclopts` and `rich` left the base install (#187). Level-one normalization redefines fullmap keys as cleaned Unicode-lowercase, Porter2-normalized, byte-ordered token sets and moves the database to schema v6, so every schema-v5 fullmap is rejected and must be rebuilt (#171, #172, #174, #175, #176). `build-fullmap --aria2c` is gone -- aria2c is used automatically whenever the `[aria2]` extra is installed (#178) -- and `--taxon-allowlist` is gone because the built-in top-100 experimental-taxon allowlist now applies to every build, guarding both reuse paths by `META.taxon_allowlist` identity (#153). Logging now requires the `[log]` extra and is fully disabled without it (#152). Features: `tablassert.net` is a new stdlib-only transient/permanent classification and retry seam shared by the agent and the BABEL downloader (#163, #164), with one bounded jittered retry layer across the inner agent, reflexion, and judge (#167) and machine-readable `error_code` values on skipped checkpoint records (#166). `build-kg` gained an automatic ephemeral shared-prefix TCode cache keyed by content-addressed XXH64 op digests (#155-#159, #161). The `--distill` corpus became schema-uniform v2 with a sibling `outcomes.ndjson`, a deterministic `RewardConfig`-tunable reward, and the new zero-dependency `tablassert distill-weigh` command (#180-#183). `CLASS_FIELD_OVERRIDES` grants the DAKP sparse qualifier stack on the pinned association classes (#188). Fixes: `distill-export` partitions its corpus by content and unions the schema across record files, so a mixed v1/v2 directory no longer CastErrors or silently stringifies a column (#184); BABEL retry warnings survive the loguru sink; PMC downloads are idempotent, atomic, and bounded-parallel (#165); reflexion and judge prompts are hard-bounded (#168). Changelog: - versioned the Unreleased section as 19.0.0 - 2026-09-15 - merged three duplicate `### Added` blocks into one - moved the two `BREAKING:` entries from `### Changed` into `### Breaking Changes` and added a `**Migration:**` note to all five - added PR links to all 17 entries, which carried none - added the missing entries: the distill v2 corpus/reward/distill-weigh stack (#180-#183), the distill-export schema-drift fix (#184), the TCode run-cache detail (#155-#161), and the README badge removal (#154) Docs: none needed here. Every shipped doc change landed with its own PR, and the docs source-of-truth gate passes against the bumped tree. Testing: - make check -> exit 0 - uv run ruff check . -> All checks passed! - uv run pyright -> 0 errors, 0 warnings, 0 informations - uv run pytest -> 1622 passed, 52 skipped - cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed - cargo clippy --all-targets -- -D warnings -> clean - uv lock --check -> resolved 169 packages, lock current - uv run mkdocs build --strict -> exit 0 - docs SSOT + CLI coverage after the changelog edit -> 204 passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hard-bounds the reflexion and judge prompts so a pathological article cannot outgrow the model's context window. Fleet distill telemetry: the three largest of 534 LLM calls were all reflexion prompts of 1.6M and 2.3M characters (689,241 and 524,336 input tokens) — past every model's window, failing outright after paying for the serialization. Cause:
llm_propose_config_editinterpolated the full coverage report and context with no cap;_build_judge_promptinterpolated the raw build report.Bounds
agent.compact_coverage_report(pure, total, non-mutating): caps every reachableunresolvedlist (top level,sections[i],sections[i].per_column[col], back-compat top-levelper_column) at the first 20 terms (UNRESOLVED_CAP) plus one visible+N moremarker — the same convention ascompact_audit_report— and addsunresolved_countsiblings naming the ORIGINAL lengths so the scale signal survives. Beyondmax_chars, per-columnunresolvedlists are dropped entirely (counts/fractions retained); the serialized block is bounded by the caller, never by parsing truncated JSON back.agent._truncate_for_prompt(text, limit, *, what)with a verbatim…[X truncated: N of M chars omitted]marker;limit <= 0raises.MAX_PROMPT_CONFIG_CHARS) + coverage 8,000 (COVERAGE_PROMPT_CHARS) + context 40,000 (MAX_PROMPT_CONTEXT_CHARS) + report 8,000 (MAX_PROMPT_REPORT_CHARS) ≈ worst case 58 K chars ≈ 15 K tokens — a ≥ 27× reduction, room left for the ~16 K-token output.map_coverage,_measure_section,build_and_audit, and the supervisor's tier-1 deterministic proposer still receive full uncapped reports; compaction happens only at the tier-2 prompt-serialization boundary. Never-raises contract and thevalidate_table_configgate preserved.Testing
uv run pytest tests/test_agent_prompt_bounds.py -q --no-cov -n 0→13 passed in 1.16s: caps-every-list, purity/non-mutation, totality on hostile shapes, verbatim markers,unresolved_count, 50,000-term regression (< 60 K prompt, 3 sections × 2 columns), overflow drop path, 400-section survival, judge compaction (kgx_pathabsent, bothreversemodes), determinismuv run pytest -q→1565 passed, 15 skipped; ruff/format/pyright clean; CI-equivalent base env passesjson.loadsalways raised, silently disabling reflexion via the caller's blanketexcept); the parse-back pass was deleted and pinned by overflow tests, re-reviewed PASS (31/31 adversarial checks)