Skip to content

test(benchmark): qualify context delivery through joint checkpoints - #7518

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-context-mode-20261028
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-context-mode-20261028

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

The joint checkpoint qualifier previously exercised context delivery only with the transport off. It now accepts explicit --context-delivery on and verifies complete native input delivery through source wakes, a sealed pre-decision point, fresh recovery and automatic recapture. The existing qualifier still defaults to off. Runtime activation is already on main through #7516; this PR contains qualification tests and documentation.

The source must actually read a complete full packet followed by deltas across settled worker wakes. Fresh receivers read their current full packet before candidate work. The shared byte-map oracle independently checks model-visible coverage and retained delta bases. Existing capture/recovery semantics, private configuration permissions and the daemon archive boundary remain intact.

Validation and scope:

  • Current head ec90739bfc4cbbcd255e970ad0453ffe4269231c is rebased onto target main 9558f75ae52e8a0c9f74ad7df0f47a69e0d413ad. All seven proposal files are byte-identical to prior head 307c3340b; the single commit retains its DCO sign-off.
  • At this head, the shared runtime and native handoff suites report 117 passed, 67 skipped: 48 require optional Harbor and 19 require an explicitly attached native host/transport/supervisor. These skips do not qualify those native paths.
  • Risk-based premerge executed five direct checks and 19 selected checks with zero failures or warnings. The benchmark-sensitive maintainer-review hold remains; the result is manual review required, not merge readiness or authorization.
  • Earlier complete new-source qualification at frozen integration 4bd580a28775f103089a21536713603e34299e3e passed all six stages in each of context-on and context-off, zero skips. Both modes created a new source point; on delivered one full packet plus seven deltas, and every fresh receiver began with full. This qualification used native Codex CLI 0.162.1, a disposable Linux daemon, fresh guests, canonical SQLite, real tools and synthetic local inference/native evaluation fixtures. It is retained at that exact integration source and is not relabeled as a new-source run at the current rebased head.
  • The previous head 307c3340b completed all 27 Linux packaged workspace scenarios, including the goal map (4,673 ms), then failed at the delegation-search fixture entry. The current rebase includes the existing mainline fixture and reader repair. Earlier hidden-help and intermittent map failures remain historical evidence; new-head CI and required review remain separate. The map timeout’s original cause has not been established.

No production default, scoring, task input, evaluation opportunity, cooldown, budget or execution permission changes. No scientific jobs or paid inference are launched by this proposal. Engineering delivery and checkpoint integrity do not establish scientific outcomes or billed-efficiency gains.

@mergify

mergify Bot commented Oct 27, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 27, 2026
@loopx-agent
loopx-agent force-pushed the codex/benchmark-context-mode-20261028 branch from 24c04de to 0deb4de Compare October 27, 2026 20:33
@loopx-agent loopx-agent changed the title fix(benchmark): activate context delivery with the default heartbeat mode test(benchmark): qualify context delivery through joint checkpoints Oct 27, 2026
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Oct 27, 2026
@loopx-agent
loopx-agent force-pushed the codex/benchmark-context-mode-20261028 branch from 0deb4de to 307c334 Compare October 27, 2026 23:24
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent
loopx-agent force-pushed the codex/benchmark-context-mode-20261028 branch from 307c334 to ec90739 Compare October 27, 2026 23:57

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

APPROVE,精确 head ec90739bfc4cbbcd255e970ad0453ffe4269231c 的七文件测试与文档改动未发现阻塞项。该判断不把测试覆盖、历史 native 资格或作者自审变成当前集成或科学验收。

动机 / Motivation

维护联合 checkpoint 和可选差分传输的开发者,需要知道源码存在的功能是否穿过真实 source、封存和接收流程。 以前 joint qualifier 只覆盖传输关闭,开启差分后仍可能没有完整读取就开始候选工作;现在显式选择 on 后,source 必须从完整包进入差分,fresh receiver 必须先读当前全量。 七个提案文件与已通过六阶段 on/off 资格的 frozen source 完全相同;本轮额外验证共享 oracle 的十八组 base/head 输入和反例均一致,真实 resolved-mode 两例通过。 不改变生产默认、任务输入、机会、冷却、预算、评分或执行许可;测试中的静态 inference 不能证明真实模型理解或账单节省。 新生产集成在启动真实研究前仍须固定自己的 source 并完成实际接收资格;历史 native 结果不被改称当前 head 新跑的结果。

改动思路 / Approach

复用现有 joint qualifier、worker、原生 hooks 和独立 byte-map oracle;新增的是可显式选择的验证路径,生产策略、权限和评分继续由既有 owner 决定。 本 PR 只交付可重复的上下文传输与联合 checkpoint 资格测试,不宣告当前集成、真实模型行为、科学效果或完整费用已通过。 只检查 capture 文件存在无法确认模型实际收到全部字节,纯 codec 单测也覆盖不到 worker 退出/resume 和 fresh authority。扩展现有六阶段,比另造 harness 更小;把原 native 测试的 byte-map oracle 移成共享 fixture,保留各阶段独立的来源、候选和权限断言。已有命令默认 off,只有显式选 on 才加入这一验证,不增加普通产品用户的准备或确认。

具体改动 / Changes

变更前规范 benchmark/README.md,固定 revision 9558f75ae52e8a0c9f74ad7df0f47a69e0d413ad 的 Inert checkpoint capture 和 joint qualification 段落。J1(真实 native/SQLite/worker/supervisor/transport,仅 inference 与 grading 合成)、J2(历史源不授予 fresh receiver 权限,recapture 后实际工作)及 J3(skip/reuse 不冒称源执行通过,source fence 与 inert 科学边界保留)均 implemented。新增说明不能替代这些原验收条件。

关键代码讲解

  • delivered_context 在 shared fixture 对实际 native request 的工具输出建立 UTF-8 byte map,完整覆盖、重叠一致和原摘要同时满足才接受,不用 EOF 或文件存在代替读取。
  • NativeContextReader 只跟随模型可见 cursor;完整读取后才检查原件,差分还要在当前 request 中找到原来完整保留的基线。该静态 responder 不证明真实模型理解。
  • qualify_checkpoint_joint.main 把显式模式传给既有 source/ordinary/stride/recapture/grandreceiver,核验实际报告。源码保持冻结;源报告复用既有 daemon archive seam,未通过放开 guest 私有目录来绕过权限。

本轮独立 corpus 使用37,744字节 Unicode 输入:三种 native output framing 各检查完整、相同重叠、缺前缀、缺中段、截断和冲突重叠,十八组 base/head 与独立预期一致,其中十二组故意损坏均拒绝。当前 resolved heartbeat mode 两例 PASS。已核同 head 既有117 PASS/67显式skip记录及 premerge:5 direct +19 selected PASS、零失败/警告,一个 benchmark_sensitive maintainer hold 保留。该 hold 不是 merge 就绪或授权。

七个提案文件与 frozen 4bd580a28775f103089a21536713603e34299e3e 完全相同;那里实际执行的 native on 六阶段与 off 六阶段均通过、零skip,on 为一次完整包加七次差分,fresh receivers 均先全量。这个历史结果保留原 source;没有宣称在当前 ec907 新跑了同一矩阵。67项skip中的48项需可选Harbor,19项需显式native/transport/supervisor,不能改写为通过。

对主干的风险 / Risks

最强反例是最终页和文件摘要都存在,却有正文未进入 native request,或者 fresh receiver 错借 source 连续性。共享 oracle 的缺页/截断/冲突反例逐一拒绝;原 receiver 继续验证新权威目录、历史源不得准入、实际候选、原机会/冷却及后续 recapture。读取完成后才允许 fixture 访问期望原件,没有先读旧全量来造差分基线。新增 helper 只属于测试,不按自然语言给真实 agent 授权。

语义与 CI 对齐

复用现有 Execution off/on、context page schema 和 checkpoint/provider fence,没有新生产词汇、schema 或权限分支。初次独立 corpus 提取漏了 base 函数的 hashlib global,尚未执行断言即失败;修正诊断 harness 后才得到十八组结果,未把该失败归给产品。遵守 wait_for_ci=false,未查询、轮询或等待远端CI。历史浏览器地图时序原因保持未知。

我的整体评价 / Verdict

这是 justified_increment,long_horizon 与 user_experience preserved。复用现有 joint qualifier、worker、原生 hooks 和独立 byte-map oracle;新增的是可显式选择的验证路径,生产策略、权限和评分继续由既有 owner 决定。 本 PR 只交付可重复的上下文传输与联合 checkpoint 资格测试,不宣告当前集成、真实模型行为、科学效果或完整费用已通过。 有价值之处是防止资格测试把局部可用错当真实接收成功,并减少重复测试 oracle;不强求此覆盖提交完成整个研究。已检查七文件、开发者 CLI、README 和未改的产品 activation owner;没有渲染页面或设置变化,visual_review 不适用。未验证范围:当前 head 新跑的 Linux native 六阶段、真实模型理解、正式研究资格与全程费用;旧 frozen 资格保留原 source。维护者敏感边界审查、exact-head merge readiness 和合并授权仍分别保留。

English verdict: APPROVE ec90739bfc4cbbcd255e970ad0453ffe4269231c as a reusable test/docs increment. Independent byte-map counterfactuals preserve the moved oracle, explicit native qualification stays source-pinned, and default-off/runtime authority remain unchanged; current scientific and full-cost acceptance is separate.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant