Skip to content

[Eval] Measure cross-question interference when decide_many bundles questions - #128

Open
Birfy wants to merge 1 commit into
mainfrom
issue-127-decide-many-interference
Open

Birfy wants to merge 1 commit into
mainfrom
issue-127-decide-many-interference

Conversation

@Birfy

@Birfy Birfy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

Answer every question before implementation and PR submission, including drafts.
Reuse explicit, still-applicable human answers with a traceable source reference;
ask the human for anything missing and wait. An agent must not invent answers.
Blank fields, placeholders, or an agent-written rationale do not satisfy this gate.
Share only information and source links appropriate for this PR's audience.

  1. What outcome are you trying to achieve?

    Answer: Flip rates and probability shift (total variation) for isolated vs bundled decide_many on this repository's own records, without mitigation. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127.

  2. What is the highest-leverage variable for that outcome?

    Answer: A measurement on the same state under two protocols (isolated heads vs one bundled decide_many, including reversed question-name order and a same-order bundled repeat as the non-determinism control). Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127 proposed work.

  3. How will this work move that variable rather than create busywork?

    Answer: It adds s1a probe --bundle over a committed multi-question file (ticket-like Choice+Noul, a ticket-router dummy-question negative control, and a modest browser-shaped frozen tick in production head order), reports per-head key/binary flip rates and mean TV, and keeps both answers in the record. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127 proposed work and acceptance criteria.

  4. Who is the customer or user, and what do they actually need?

    Answer: Agents contributors who might publish isolated probe numbers while production browser ticks already bundle every head of the action space in one decide_many. Ticket-router is single-question today and is only a negative control here. Related closed work ([Eval] Measure option-order sensitivity of each decision backend (flip rate under reversed/shuffled options) #90 option order, [Decisions] Make confidence mean the same thing on every backend (Laya reports normalized entropy, others max p or Jev's margin) #89 confidence) is not reopened. Source: issue [Eval] Measure cross-question interference when decide_many bundles questions #127.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes. Issue #127 asks for the measurement, committed cases, tests, docs, and a results write-up; that has to land in the repo. Danyang asked scheduled coding runs to fill these intent items from the issue and not wait for a separate human answer.

For a bug fix: link actual observed failure evidence.
Not applicable: this is a new measurement, not a bug fix.

Evidence: N/A

Why

Delaunay's audit (arXiv:2611.12521) finds that bundling several System 1 questions in one request is not the same as asking them one at a time. Browser ticks in this repo already send every head of the action space in one decide_many. We have measured option-order sensitivity (#90) and unified confidence (#89); we had not measured cross-question interference on our own path. This PR is measurement only: no mitigation, and the paper's Tev / Clef percentages are not Agents results.

How

s1a probe --bundle reads a multi-question JSONL file. For each case it:

  1. Asks every head in its own request (isolated).
  2. Asks the same questions in one decide_many, in file / production order.
  3. Asks them again with the question-name order reversed (directional interference).
  4. Repeats the bundled request in the same order once (the non-determinism control, the same idea as --order same:N).

It reports, per head, the key or binary flip rate versus the isolated answer (a Noul flips when p >= 0.5 changes) and the mean total variation between the distributions. Both answers stay in the record and in --json. --bundle and --order cannot be used together. Without --bundle, the fit probe and decide_many are unchanged.

Review first: s1a/probe_bundle.py, evals/probe/interference.jsonl, docs/decide-many-interference.md, tests/test_probe_bundle.py.

What

  • s1a probe --bundle and --json on that path
  • Cases: evals/probe/interference.jsonl (six ticket-like Choice+Noul items, one ticket-router negative control with a dummy second question, one constructed browser-shaped frozen tick: operation, click_target, type_text_target). Recorded decision_ticks.json files do not keep the original questions or observation, so this is not a replay of a recorded episode.
  • Docs: docs/decide-many-interference.md, a results placeholder, a sentence in docs/architecture.md, a docs/skills.md note, a row in evals/README.md, CHANGELOG Unreleased
  • No change to decide_many itself. No mitigation.

Verification

  • uv run ruff format --check . && uv run ruff check . && uv run ty check
  • uv run pytest -q (1365 passed, 132 skipped) and scripts/smoke.sh
  • CHANGELOG.md and the docs say what the code does now

Offline tests: an interfering fake backend that reads earlier question text flips the later head at 100% when bundled and 0% when heads are asked alone (and the opposite head under reversed order); a content-only fake never flips; the same-order repeat is the control for ordinary non-determinism.

Demo / evidence

  • Video link, or exemption/blocker with follow-up: no live run in this PR. This box had neither in-process Laya weights (uv sync --extra laya / convaiinnovations/laya) nor a Jev key. Follow-up is a results folder under docs/results/decide-many-interference/ once a run is recorded (command, Agents SHA, checkpoint, device, date, case count). Implementation is otherwise complete, so this is ready for review rather than left as a draft. Do not copy arXiv:2611.12521 / Tev / Clef percentages as Agents results.
  • Application/task → agent actions → independently checked result: not yet; the harness is ready to record isolated vs bundled answers.
  • System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: this branch is 57de60dc on issue-127-decide-many-interference, against main c4da7c3.

Closes #127

)

Measure isolated heads against a bundled decide_many in file order and
in reversed question-name order, plus a same-order bundled repeat as the
non-determinism control. Offline tests cover an interfering fake and a
content-only fake. No live row yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Eval] Measure cross-question interference when decide_many bundles questions

1 participant