Skip to content

test: pin the token estimators' arithmetic and history signature - #284

Merged
Zongwei9888 merged 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:test/token-estimator-cjk
Oct 28, 2026
Merged

Zongwei9888 merged 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:test/token-estimator-cjk

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Why

estimate_prompt_tokens and estimate_message_tokens decide when the pressure gate compacts or discards history, and history_signature is the contract by which the token meter's anchor and the compaction memo agree on what "unchanged" means. All three were exercised only indirectly, through the meter, so a silent change here would surface as a context overflow or an anchor that quietly stops anchoring.

What

New file tests/test_token_estimator.py (13 tests, no production code touched):

  • Offline estimator arithmetic — locks len(payload) // 4 + 4 * messages and the 4-token floor, with the tokenizer forced off so every value is exact on any machine.
  • Field collection — each field a tool turn can carry (tool_calls, reasoning_content, name, tool_call_id) and the tools schema is asserted to move the price; text-parts are priced like the equivalent string, non-text parts are not.
  • CJK — the offline estimator is script-blind (equal length, equal price); under cl100k_base a CJK character costs more than ASCII. Both behaviours are pinned so a future script-aware change is a reviewed decision.
  • Signature prefix property — the anchor keeps its sample only while new history starts with the signature it took; a shape/weight change that broke the prefix would silently drop every anchor.

Test plan

  • pytest tests/test_token_estimator.py — 13 passed.
  • pytest tests/test_token_meter.py tests/test_token_meter_wiring.py tests/test_tool_result_spill.py — all green (the tests that exercise these helpers indirectly).
  • pre-commit run --all-files — all hooks passed.
  • pip check — only unrelated ML-package conflicts from the local global env; CI installs from the lockfile in a clean venv.

estimate_prompt_tokens and estimate_message_tokens decide when the
pressure gate compacts or discards history, and history_signature is the
contract by which the token meter's anchor and the compaction memo agree
on what "unchanged" means. All three were exercised only indirectly,
through the meter.

Add direct tests that lock the offline estimator's length-over-four
arithmetic, the per-field collection that lets a tool turn's price
move, the script-blindness that prices a CJK prompt like an ASCII one
of equal length, and the signature prefix property the anchor relies
on. The tokenizer is forced off so every value is exact regardless of
what the machine happens to have installed.
@Zongwei9888
Zongwei9888 merged commit 3d0bfe8 into HKUDS:main Oct 28, 2026
12 checks passed
@Zongwei9888

Copy link
Copy Markdown
Collaborator

Thanks @raymondginger2018-sudo, merged as 3d0bfe8. Having these helpers pinned directly is useful.

One note for later: the ("user", 0) in the prefix test reflects a gap in history_signature, which counts list content as zero length. If we fix that, we'll update the assertion with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants