Skip to content

test(quota): raise multi-agent quota JSON ceiling after candidate preservation - #7592

Closed
xiaoan17 wants to merge 1 commit into
loopx-project:mainfrom
xiaoan17:codex/hot-path-budget-evidence-20261028
Closed

xiaoan17 wants to merge 1 commit into
loopx-project:mainfrom
xiaoan17:codex/hot-path-budget-evidence-20261028

Conversation

@xiaoan17

@xiaoan17 xiaoan17 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Scope

One ceiling in the real-CLI output budget, plus the measurement that justifies it.

4b2b97b7b (#7557, "Preserve complete context in shared quota diagnostics")
replaced candidate-collection compaction in
loopx/control_plane/quota/cli_projection.py with preservation. The synthetic
model-portfolio budget for that contract was already re-qualified upstream in
98ca82923 (#7583), which raised 40,000 → 90,000 and added preflight checks that
candidates survive. This PR covers the remaining surface that still measures the
old contract.

Current gap

On main at e19d94ca0:

tests/control_plane/test_cli_output_budget.py::test_real_cli_output_stays_inside_baseline_and_growth_contracts
FAILED
AssertionError: quota_should_run/multi_agent/json emitted 25836 chars; baseline ceiling is 25500

The real quota should-run packet grew because the hot path now keeps the
capability candidate context. Nothing bounds that growth except this ceiling,
which was calibrated against the compaction contract #7557 removed.

Measurement (same fixture, base vs head)

Base is 4b2b97b7b^; head is e19d94ca0. Both arms run the existing budget
fixture.

Scenario Base JSON Head JSON Delta Ceiling
small 22,618 23,265 +647 25,000 (unchanged, 1,735 headroom)
crowded 34,732 35,836 +1,104 42,000 (unchanged, 6,164 headroom)
multi-agent 24,925 25,836 +911 25,500 → 27,000

Markdown is unchanged for crowded and multi-agent (8,148 / 6,439) and moves
7,479 → 7,481 for small, all inside their unchanged ceilings. The per-Todo
allowance (300 chars/unit) and fixed semantic allowance (9,700) are untouched
and still cover crowded-minus-small growth (12,571 measured).

Change

loopx/control_plane/testing/cli_output_budget.py — raise only the exceeded
multi-agent JSON ceiling, 25,500 → 27,000, with the base/head measurement
recorded at the spec. No other surface, growth allowance or line budget changes.

No runtime behavior changes: the constant governs a qualification fixture, not
production transport.

Validation

tests/control_plane/test_cli_output_budget.py                38 passed, 1 skipped
tests/control_plane/test_actual_default_model_behavior_portfolio.py
                                                            50 passed (upstream fix)
ruff check (changed file)                                    passed

The base/head arms were measured on the same fixture; the original 25,500
overflow is recorded here as a failure, not reclassified as a pass.

@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @xiaoan17.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 28, 2026
…servation

loopx-project#7557 preserved capability candidate collections on the hot-path projection.
The real-CLI budget fixture re-measures the same `quota should-run` workload
at 24,925 -> 25,836 JSON characters for the multi-agent scenario (+911),
which exceeds the unchanged 25,500 ceiling.

Markdown is unchanged at 6,439. Small (23,265 against 25,000) and crowded
(35,836 against 42,000) stay inside their ceilings, the per-Todo allowance
(300 chars/unit) and the fixed semantic allowance (9,700) still cover
crowded-minus-small growth, and every other surface ceiling is untouched.

Raise only this ceiling to 27,000 and record the base/head measurement at the
spec. The original overflow remains a recorded failure, not a reclassified
pass. Runtime payload size and production transport limits are unchanged.

Validation: tests/control_plane/test_cli_output_budget.py 38 passed, 1
skipped; ruff clean.

Signed-off-by: xiaoan17 <anbc1224@gmail.com>
@xiaoan17
xiaoan17 force-pushed the codex/hot-path-budget-evidence-20261028 branch from 85a8372 to 722f2b9 Compare October 28, 2026 16:36
@xiaoan17 xiaoan17 changed the title test(quota): re-qualify hot-path budgets after candidate preservation test(quota): raise multi-agent quota JSON ceiling after candidate preservation Oct 28, 2026
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 28, 2026
@mergify

mergify Bot commented Oct 28, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @xiaoan17.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 28, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max

未发现当前精确 head 722f2b9 的阻断问题,结论为 APPROVE。

动机 / Motivation

运行 CLI 输出回归检查、维护 Agent 决策上下文的开发者。

同一组三 Agent fixture 的完整 JSON 已有25,836字符,原25,500上限使基线回归检查失败;本 PR 只把该上限调到27,000,保留实际返回的全部内容。

实际基线失败已复现,当前原矩阵通过;六组真实console的字段、正文和命令保持,其他预算及负向拒绝仍有效。

本次不调整生产传输硬限制、模型执行预算、冻结性能标准,也不声称真实模型效果或长程收益已经通过。 这里恢复的是有意义的回归警戒线,不把测试变绿当作生产性能或模型效果的证据。

改动思路 / Approach

这是测试回归预算的单点校准;完整上下文已有消费者,删除候选正文来满足旧阈值会损失信息,沿既有预算owner调整一个数值即可。

本 PR 只交付多 Agent JSON回归预算和配对实测注释,未改变CLI运行时、fixture、行数或增长预算。

实际调用关系是完整console输出进入measure_cli_output,再由原baseline、行数、语义键和增长门判断;没有新owner、开关或执行效果。用文字足以说明这一处校准,不额外绘图。保留25500会持续报告已实测的同一失败,统一提高所有预算或删除正文都没有本次证据支持。

具体改动 / Changes

spec_ref: docs/development/testing-and-quality.md;spec_revision: e19d94c。对应既有 Budget Failure Decisions:用相同工作负载和计量口径归因,保留有用上下文及原失败,区别回归预算与硬/冻结边界。本项为implemented;不是改写实验通过线。

完整差异为一个既有模块+5/-1,仅一个常量和实测注释。生产AST在27000替回25500后逐项相同;fixture和原断言均未改。

关键代码讲解

  • CLI_OUTPUT_BUDGET_SPECS 仅将multi_agent JSON阈值从25500改到27000,Markdown仍7000;行数650、每Todo增长300及固定语义增长9700均保持。
  • measure_cli_output 按实际stdout记录字符、UTF-8字节和行数并解析JSON,本身未变,不先删候选或正文。本组合fixture当前均为ASCII,所以字符数与字节数恰好相同,不能因此混用一般口径。
  • assert_cli_output_baseline 继续独立拒绝超字符、超行数和缺语义键;原矩阵仍检查JSON/Markdown、规模增长及完整bootstrap路由。

同物理fixture、同运行环境、三个不可变revision的实际console测量如下;这张表只表达测量,不证明模型收益:

场景/口径 候选保留前5a7cc8195 当前基线e19d94ca0 PR head
small JSON字符 22618 23265 23265
crowded JSON字符 34732 35836 35836
multi_agent JSON字符 24925 25836 25836
multi_agent Markdown字符 6439 6439 6439

历史到当前multi_agent增量911已复现,原25500超限失败也由原测试在基线复现。当前base与head的六对完整输出仅有逐字段列明的生成时钟、rollout event id和File导入版本摘要差异;选择、候选正文、完整诊断和命令及其他字段保持。历史版本的完整原文与当前投影元数据差异保留,不冒称历史到当前全部等价。

对主干的风险 / Risks

最强风险是用提上限掩盖无价值膨胀。当前实际headroom为1164字符,只调整这一场景;未减少输入规模或输出内容,所有其他AST与预算保持。真实输出上构造27001字符、652行和删除selected_todo三种变体,均由原helper拒绝。这些是预算helper的负例,不能冒充新的生产故障注入。

PASS:预算与actual-default portfolio模块88项、标准premerge六direct及11selected、改动文件Ruff、配置17文件mypy、完整TS typecheck、diff/DCO/双语门和严格精确CQR。原矩阵在基线明确FAIL为25836>25500,在head通过。原测试另有1个native temp namespace无法容纳depth-12布局的条件skip;其fixture、条件和断言未变,不把skip或一次本地绿变成原生Windows资格。未查询或等待CI。

私有验证驱动初次用了不存在的pytest节点,未执行测试;另一次误以为所有场景都有可选capability_gate而KeyError。修正读取实际结构后才完成上述六对证据;这些驱动失败保留,不算产品测试或PR回归。

语义与 CI 对齐

沿用既有测试预算owner,保持local_only,无新状态词汇、配置或authority。既有候选完整性测试和真实console对照承担语义证据;预算门通过本身不证明文本充分或模型采用。没有松动生产硬限制、执行预算或冻结实验标准。

我的整体评价 / Verdict

APPROVE,当前单点回归预算维护目标达到。long_horizon与user_experience均为preserved:原CLI接受、选择、诊断与恢复入口保持,无新增设置、输入或确认轮次。实际CLI上下文未缩水,尚未测量模型长期效果或总成本收益。

这是测试回归预算的单点校准;完整上下文已有消费者,删除候选正文来满足旧阈值会损失信息,沿既有预算owner调整一个数值即可。 本 PR 只交付多 Agent JSON回归预算和配对实测注释,未改变CLI运行时、fixture、行数或增长预算。

有界维护已应用,不需要新框架或人为后继。已检查 quota CLI、预算consumer及现有Markdown输出;没有页面、渲染组件或交互布局变化,视觉评审不适用。 原生Windows、安装态和真实模型采用/长程效果未在本轮验证。 合并、安装及其他公共发布权仍由各自owner决定。

English verdict: APPROVE — 722f2b9. Matched real-console measurements reproduce the 25,836-character baseline failure and justify the single 27,000-character regression ceiling. Other guards remain effective; current validation passes with one preserved environment-dependent skip. Model and long-horizon qualification remain separate.

@xiaoan17

Copy link
Copy Markdown
Contributor Author

Superseded by main — closing

The change this PR proposes is already on main, from a different session:

  • 791f297d2 — test: calibrate complete multi-agent quota fixture budget —
    raised the same multi_agent.json ceiling in
    loopx/control_plane/testing/cli_output_budget.py from 25,500 to 26,500, on the
    same unchanged 18-Todo / 3-Agent fixture, with the same measurement: 25,836
    JSON characters across 96 matched surface/mode measurements and no semantic or
    route differences.
  • 544aa0fee (test(heartbeat): calibrate complete thin JSON scope budget #7614) then calibrated the related heartbeat max_chars surface on
    top of it (4,500 → 4,600).

This PR proposes 27,000 for the same fixture instead. Keeping both would leave
a duplicate ceiling for the same measurement and would re-litigate a number that
main already carries with its own evidence.

The remaining CI failure on this PR was not caused by it: chat-bundle-browser (1)
fails on the shared base because
apps/presentation/dashboard/smoke/delegation-search-browser-smoke.mjs intercepts
/api/chat/app-steward and hides the fixture's later source refusal, so the
expected alert can never appear. That is fixed separately in #7599.

Closing rather than rebasing. Reopen if the 26,500 ceiling proves too tight for
the fixture; the measurement and the rationale are recorded in 791f297d2.


Superseded by main — closing

This PR's change is already on main: 791f297d2 raised the same
multi_agent.json ceiling from 25,500 to 26,500 on the same fixture, with the
same measurement (25,836 JSON characters, 96 matched surface/mode measurements,
no semantic or route differences), and #7614 calibrated the related heartbeat
max_chars surface on top. This PR proposes 27,000 for the same fixture instead.

The one failing check here was a base failure, not this PR's: the delegation
search smoke hides the fixture's source refusal behind a page route, so the
expected alert can never appear. Fixed separately in #7599.

Reopen if 26,500 proves too tight; the measurement lives in 791f297d2.

@xiaoan17

Copy link
Copy Markdown
Contributor Author

Superseded by 791f297 on main (same fixture, same 25,836-char measurement, ceiling 26,500). Base CI failure fixed separately in #7599.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Mergify: the pull request has merge conflicts with its base branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants