Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
Answer every question before implementation and PR submission, including drafts.
Reuse explicit, still-applicable human answers with a traceable source reference;
ask the human for anything missing and wait. An agent must not invent answers.
Blank fields, placeholders, or an agent-written rationale do not satisfy this gate.
Share only information and source links appropriate for this PR's audience.
What outcome are you trying to achieve?
Answer: Flip rates and probability shift (total variation) for isolated vs bundled
decide_manyon this repository's own records, without mitigation. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127.What is the highest-leverage variable for that outcome?
Answer: A measurement on the same state under two protocols (isolated heads vs one bundled
decide_many, including reversed question-name order and a same-order bundled repeat as the non-determinism control). Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127 proposed work.How will this work move that variable rather than create busywork?
Answer: It adds
s1a probe --bundleover a committed multi-question file (ticket-like Choice+Noul, a ticket-router dummy-question negative control, and a modest browser-shaped frozen tick in production head order), reports per-head key/binary flip rates and mean TV, and keeps both answers in the record. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127 proposed work and acceptance criteria.Who is the customer or user, and what do they actually need?
Answer: Agents contributors who might publish isolated probe numbers while production browser ticks already bundle every head of the action space in one
decide_many. Ticket-router is single-question today and is only a negative control here. Related closed work ([Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90 option order, [Decisions] Makeconfidencemean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89 confidence) is not reopened. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127.After answering the four questions: is a PR needed at all to achieve this outcome?
Human decision and rationale (or explicit source reference): Yes. Issue #127 asks for the measurement, committed cases, tests, docs, and a results write-up; that has to land in the repo. Danyang asked scheduled coding runs to fill these intent items from the issue and not wait for a separate human answer.
For a bug fix: link actual observed failure evidence.
Not applicable: this is a new measurement, not a bug fix.
Evidence: N/A
Why
Delaunay's audit (arXiv:2611.12521) finds that bundling several System 1 questions in one request is not the same as asking them one at a time. Browser ticks in this repo already send every head of the action space in one
decide_many. We have measured option-order sensitivity (#90) and unified confidence (#89); we had not measured cross-question interference on our own path. This PR is measurement only: no mitigation, and the paper's Tev / Clef percentages are not Agents results.How
s1a probe --bundlereads a multi-question JSONL file. For each case it:decide_many, in file / production order.--order same:N).It reports, per head, the key or binary flip rate versus the isolated answer (a Noul flips when
p >= 0.5changes) and the mean total variation between the distributions. Both answers stay in the record and in--json.--bundleand--ordercannot be used together. Without--bundle, the fit probe anddecide_manyare unchanged.Review first:
s1a/probe_bundle.py,evals/probe/interference.jsonl,docs/decide-many-interference.md,tests/test_probe_bundle.py.What
s1a probe --bundleand--jsonon that pathevals/probe/interference.jsonl(six ticket-like Choice+Noul items, one ticket-router negative control with a dummy second question, one constructed browser-shaped frozen tick:operation,click_target,type_text_target). Recordeddecision_ticks.jsonfiles do not keep the original questions or observation, so this is not a replay of a recorded episode.docs/decide-many-interference.md, a results placeholder, a sentence indocs/architecture.md, adocs/skills.mdnote, a row inevals/README.md, CHANGELOG Unreleaseddecide_manyitself. No mitigation.Verification
uv run ruff format --check . && uv run ruff check . && uv run ty checkuv run pytest -q(1365 passed, 132 skipped) andscripts/smoke.shCHANGELOG.mdand the docs say what the code does nowOffline tests: an interfering fake backend that reads earlier question text flips the later head at 100% when bundled and 0% when heads are asked alone (and the opposite head under reversed order); a content-only fake never flips; the same-order repeat is the control for ordinary non-determinism.
Demo / evidence
uv sync --extra laya/convaiinnovations/laya) nor a Jev key. Follow-up is a results folder underdocs/results/decide-many-interference/once a run is recorded (command, Agents SHA, checkpoint, device, date, case count). Implementation is otherwise complete, so this is ready for review rather than left as a draft. Do not copy arXiv:2611.12521 / Tev / Clef percentages as Agents results.57de60dconissue-127-decide-many-interference, against mainc4da7c3.Closes #127