Skip to content

docs(tasks): describe initial-load outcomes without session assumptions - #4

Closed
lanyue-llk wants to merge 1 commit into
mainfrom
codex/outcome-defined-task-contract
Closed

lanyue-llk wants to merge 1 commit into
mainfrom
codex/outcome-defined-task-contract

Conversation

@lanyue-llk

Copy link
Copy Markdown
Contributor

Problem

The public task text mixed the answer boundary with participant implementation assumptions. Phrases such as “fresh anonymous browser”, “do not log in”, “Browser Use actions”, and allowed parsing routes made the benchmark sound as if it prescribed how a Web Fetch implementation must work.

Web Fetch supplies a URL and evaluates the returned deliverable. The shared scope is the content available from the page's initial load; records that require clicking, scrolling, or pagination are outside that answer. Authentication mode and internal retrieval strategy are not task dimensions.

Change

  • Remove session and authentication assumptions from current task descriptions.
  • Remove participant-specific implementation language.
  • Keep the initial-load answer boundary in plain language.
  • Preserve every URL, task inventory, structured acceptance contract, reference body, scoring rule, and threshold.

This updates reader-facing text for 385 current task definitions. It makes no provider calls and does not change any saved result or score.

Example

Before:

Use a fresh anonymous browser. Do not log in or perform other Browser Use actions. Rendering or embedded-data parsing is allowed.

After:

Return the content from the page's initial load. Exclude content that appears only after clicking, scrolling, or pagination.

Validation

  • All four suites validate: 450 tasks.
  • All four frozen input trees verify against their updated hashes.
  • 928 focused tests and 6 subtests pass.
  • A semantic diff confirms task rows change only task_prompt, plus the shared popular-site source description; structured acceptance fields are unchanged.

@lanyue-llk

Copy link
Copy Markdown
Contributor Author

Closing this wording-only proposal after comparing it with merged #7, #10, #11 and #13. Its 385 prompt edits are not already merged, but prompts are execution inputs under the current versioning contract, so rewriting them also changes the frozen input edition. A new benchmark edition solely for these wording changes is unnecessary. The useful authoring guidance is retained in CONTRIBUTING through #8; frozen tasks and their published bindings remain intact.

@lanyue-llk lanyue-llk closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant