Update 13 tasks: review fixes, query splits, scoring and Luna evidence

tasks/biomedical-graph-retrieval/README.md

@@ -1,16 +1,25 @@
 # Biomedical Graph Retrieval
 
+Current run and release status (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md). Earlier validation-stage statements are superseded by that dated record.
+
 Author: Cooki-e. Build a reusable search system over the supplied knowledge graph
 and return complete sets of internal node IDs. No training or fine-tuning.
 
-The local package includes the corpus, 3 public validation questions and 20
-private test questions. Version 0.2.1 preserves the graph and private split and
-provides three public examples with different query plans from the hidden set.
-Both splits require multi-step graph computation and complete answers, verified
-by independent calculations. Comparable difficulty is a structural judgment;
-no fresh coding-agent score is available. Source attribution, licensing records and construction evidence are
-retained separately in the authoring workspace; source dataset names are omitted
-from this package.
+The local package includes the corpus, 8 public validation questions and 20
+private test questions. Version 0.2.2 retains the original three public examples
+and adds five independently verified examples: target-set intersection and
+exclusion, distinct support counting, ordinary shortest paths, exclusive-target
+pathways, and ordered protein-pair counting. The graph and private split are
+unchanged. Public and private query anchors are disjoint; the public set does
+not cover every private query structure.
+
+The completed v0.2.1 Codex/OpenRouter `openai/gpt-6-luna`, `max` trial scored
+0/20 exact sets with valid execution, no query timeouts and a passing trajectory
+audit. That trial used the previous three-example public set; it is not a
+v0.2.2 model result. [VALIDATION.md](VALIDATION.md) records the public-data update,
+local checks, historical run and finalizer repair separately. Source attribution,
+licensing records and construction evidence remain in the authoring workspace;
+source dataset names are omitted from this package.
 
 `instruction.md` is the concise agent request; `raw-instruction.md` is its
 structured equivalent. Resource and time limits are in the instruction and
@@ -28,7 +37,7 @@ Only the corpus, public validation and generic resource documents are mounted
 into the agent environment. Private questions, labels, grader and audit files
 remain verifier-only. No author solutions or screening traces are supplied.
 
-Public runtime data are published in [search-swe/Search-SWE](https://huggingface.co/datasets/search-swe/Search-SWE/tree/2a2188f3a9426361da5f10c82a35148269574a92) and pinned by immutable commit in `assets.json`, with exact sizes and SHA-256 hashes. Private evaluation inputs remain in the verifier package. Source terms and attribution remain applicable; publication does not establish additional licensing rights. Fresh restoration on 2026-10-16 verified all 3 dataset-backed runtime files against their recorded sizes and SHA-256 hashes.
+Public inputs are published and pinned to immutable revisions in `assets.json`, with exact sizes and SHA-256 hashes. Unchanged model pins are retained. See [RUN_RESULTS.md](RUN_RESULTS.md) for the current run evidence and publication status. Existing source terms and attribution remain applicable.
 
 ```bash
 python scripts/download_assets.py --task biomedical-graph-retrieval --verify-only
@@ -37,5 +46,5 @@ python scripts/run_task.py --task biomedical-graph-retrieval --agent pi \
   --model deepseek/deepseek-flash --dry-run
 ```
 
-The last command previews a launch and does not call a model. Validation evidence
-and remaining unrun layers are recorded in `VALIDATION.md`.
+The last command previews a launch and does not call a model. Completed runtime
+evidence, revision coverage and remaining release steps are in [VALIDATION.md](VALIDATION.md).

tasks/biomedical-graph-retrieval/RUN_RESULTS.md

@@ -0,0 +1,40 @@
+# Run evidence reconciled on 2026-10-23
+
+The saved v0.2.1 trial returned 0/20 complete sets. Version 0.2.2 now supplies eight public examples (original three plus five independently verified additions); its twenty hidden cases are unchanged. No fresh v0.2.2 agent trial was run. The historical zero must not be presented as a new-version measurement.
+
+| Field | Observed value |
+| --- | --- |
+| Current package | `0.2.2` |
+| Saved agent trial version | `0.2.1` |
+| Agent / model / effort | Codex / OpenRouter `openai/gpt-6-luna` / `max` |
+| Evidence type | Historical trial with the previous public split |
+| Current available score | 0 |
+| Raw metric | ExactSetAccuracy = 0 |
+| Archived final reward | 0 |
+| Required trajectory judge | Succeeded; numeric pass = 1 |
+| Current CPU / RAM / GPU configuration | 16 CPUs / 32768 MiB / 0 GPU(s) |
+| Development limit | 7200 seconds |
+| Current instruction SHA-256 | `d5b276d357b2df5c1c664beeaee9fdf99b4116670b260d4437e744c40d548957` |
+| Saved evaluation SHA-256 | `f8ec981ea620be1de331c6758a22217f33167a4fbadd3d0ce106a4a417dca197` |
+| Saved reward SHA-256 | `a82a52709516ffd69da56259559c6fd0a2016b05608572e22e9a36842119d209` |
+
+The records are retained in the author's surrounding workspace under
+`task_results/biomedical-graph-retrieval/codex-openrouter-gpt-6-luna-max-v021-20261014T164416Z/jobs/2026-10-15__00-44-54/biomedical-graph-retrieval__7iyRMF9/verifier/biomedical-graph-retrieval-eval/evaluation.json` and
+`task_results/biomedical-graph-retrieval/codex-openrouter-gpt-6-luna-max-v021-20261014T164416Z/jobs/2026-10-15__00-44-54/biomedical-graph-retrieval__7iyRMF9/verifier/reward.json`. These are author-local evidence
+locations, not public download links. Original trajectories, submitted code,
+private evaluation outputs and credentials are not published in this task.
+The current instruction matches the evaluated instruction; where public data,
+normalization or verifier behavior changed, the limitations above apply.
+
+Public inputs have immutable source revisions, sizes and SHA-256 values in
+`assets.json`. Unchanged published data/model references are retained.
+The only new data in this release are the biomedical and filtered-ANN public
+validation pairs, published through [HF PR #8](https://huggingface.co/datasets/search-swe/Search-SWE/discussions/8) at
+`234406eb7e304478e48905829fc0d3233f5a9f1f`. All 52,417 earlier dataset manifest records were preserved.
+Publication does not resolve outstanding upstream licensing or independent
+quality-review limitations recorded in the task's provenance.
+
+This release reconciliation uses existing model results. It does not start a
+fresh coding agent, GPU training job or paid judge call. Task-local regression
+checks are recorded in the contribution PR; earlier construction and runtime
+records remain in `VALIDATION.md` where present.

tasks/biomedical-graph-retrieval/VALIDATION.md

@@ -1,6 +1,47 @@
-# Validation — version 0.2.1
+# Validation — version 0.2.2
 
-Publication update (2026-10-16): public inputs are now published and pinned; fresh downloads passed size/SHA-256 verification. See the README for the official revision. The records below describe earlier validation stages; their statements about unpublished assets are historical. This publication does not add a new coding-agent trial or resolve outstanding source-license questions.
+Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages.
+
+## Public validation expansion — 2026-10-23
+
+Five independently verified public examples are now installed in
+`data/validation/queries.jsonl` and `ground_truth.jsonl`. The three v0.2.1
+examples remain byte-identical at the beginning of both files, producing
+**8 public queries and 20 unchanged hidden queries**. Metadata and both public
+asset sizes/SHA-256 hashes match the installed data. Their remote revisions
+remain unpublished, as before.
+
+The additions cover target-set intersection/exclusion, distinct target support
+counts, ordinary shortest-path unions, exclusive-target pathways, and ordered
+protein-pair counts. Complete answer sizes are 2, 7, 4, 1 and 3. Python sets and
+counts agree with independent SQLite queries; custom BFS agrees with NetworkX
+shortest-path enumeration. New anchors differ from all old public/private anchors
+and from each other. No complete answer set duplicates an existing example.
+Other private query structures remain unexposed.
+
+The graph, hidden files, both instruction files, verifier source, runtime limits
+and environment files are unchanged by this update. A pre-existing update to
+`raw-instruction.md` was preserved. The old v0.2.1 Luna reward 0 remains
+historical evidence; no fresh coding-agent or online judge run has measured the
+effect of the eight-example public set.
+
+Author-only construction and installation evidence is in
+`task_development/stark-prime-retrieval/public-additions-20261023/` in the
+surrounding workspace, including `selected/author-evidence.jsonl`,
+`verification.json`, `installation-before/` and `installation.json`.
+
+Post-installation checks passed: all 25 task-local unit tests, production output
+validation for all eight reference rows, and local size/SHA-256 verification of
+all three assets. Byte comparison confirms that the original three rows were
+preserved and the five additions exactly match the independently verified
+candidates. Scoring the reference fixture returns 8/8; this is a scorer contract
+check, not a model result. The static task check reports only the three existing
+unpublished asset revisions (`revision: null`); publication pins remain pending.
+
+## Archived v0.2.1 validation
+
+The following sections describe v0.2.1 and the subsequent finalizer repair.
+Statements about matching configuration and public assets refer to that revision.
 
 Version 0.2.1 replaces the twenty public examples with three newly constructed
 questions. Their query plans differ from all six private families: minimum
@@ -16,8 +57,82 @@ effort without exposing private templates. This is a structural design judgment,
 not measured coding-agent difficulty. The previous twenty public examples and
 the rejected same-family draft remain outside the runtime task package.
 
-The following model and runtime results concern earlier versions; they do not
-constitute a new model evaluation of version 0.2.1.
+## Completed v0.2.1 coding-agent trial
+
+The existing author-side run was reconciled on 2026-10-22. Its saved summary
+agrees with the original verifier evaluation and trial result. This supersedes
+the earlier statement that no fresh v0.2.1 coding-agent result was available.
+
+| Item | Recorded result |
+| --- | --- |
+| Trial ID | `biomedical-graph-retrieval__7iyRMF9` |
+| Trial start / finish (UTC) | 2026-10-14 16:44:54.986566 / 18:14:56.809390 |
+| Agent / harness | Codex 0.157.1 / Harbor 0.22.0 |
+| Coding model / effort | OpenRouter `openai/gpt-6-luna` / `max` |
+| Attempts / concurrency / retries | 1 / 1 / 0 |
+| Configured resources | 16 CPUs, 32 GiB RAM, 100 GiB storage, no GPU |
+| Budgets | Development 7,200 s; build 1,800 s; each query 30 s; verifier 5,400 s |
+| Completion | Trial and verifier completed; launcher exit 0; no trial exception |
+| Raw ExactSetAccuracy / final reward | 0/20 = 0.0 / 0.0 |
+| Execution | Valid; no evaluation errors; 0 query timeouts; all 20 query commands exited 0 |
+| Build / query timing | Build 21.694 s; query mean 10.616 s, maximum 23.295 s |
+| Required trajectory judge | `deepseek/deepseek-v4.1-flash`; succeeded; numeric judgment 1; trajectory present |
+
+The zero reward is a valid retrieval result: none of the twenty returned sets
+exactly matched its reference. It was not caused by a failed audit, invalid
+output or a query timeout. This single run does not establish universal
+unsolvability or equal measured difficulty of the public and private splits.
+
+The trial used a frozen working-tree snapshot; its saved metadata does not
+identify a dedicated task commit. The following SHA-256 fingerprints bind the
+instruction and grader to the recorded v0.2.1 snapshot:
+
+| File | SHA-256 |
+| --- | --- |
+| `instruction.md` | `d5b276d357b2df5c1c664beeaee9fdf99b4116670b260d4437e744c40d548957` |
+| `raw-instruction.md` | `8eca79e08a69568476627d06221cf837d67d04eece7ab2b55076550bfae4a482` |
+| `tests/grader.py` | `143386613552fde5844cb6c09b4bf94cc13ff590d9ade82183e3483a179fec02` |
+
+### Coverage of the finalizer repair
+
+On 2026-10-22, `tests/finalize_reward.py` was hardened to accept only numeric
+binary judgments and to record `judge_output_valid`; regression tests were added
+in `tests/test_finalize_reward.py`. The trial above predates this repair. Its
+finalizer SHA-256 was `4fe22c327f4f05057fbcb6e55e7847f8ec8c99448db5a35ce107a3477ec83bde`;
+the repaired file is `25cf17acad337fb60685ead0b929b28e640a32725313489cb5dce71cc7406575`.
+The instruction, task configuration, grader, hidden inputs, public asset manifest
+and environment files match the saved snapshot. Documentation changes record
+the evidence; the solver contract and deterministic metric are unchanged.
+
+The task suite passed all **25 tests**, including six new finalizer tests, with
+`python -B -m unittest discover -s tasks/biomedical-graph-retrieval/tests -p 'test_*.py' -q`.
+An offline replay of the archived evaluation and saved numeric audit result
+through the repaired finalizer preserved reward 0.0, with `judge_output_valid=true`
+and the original trajectory present. This replay checks the final gate only;
+neither the coding agent nor the online judge was run again.
+
+### Original evidence locations
+
+Paths below are relative to the surrounding author workspace, the parent of the
+Search-SWE checkout. Evidence root:
+`task_results/biomedical-graph-retrieval/codex-openrouter-gpt-6-luna-max-v021-20261014T164416Z/`.
+
+- `REPORT.txt`, `evaluation-summary.json` and `run-config.json`: run summary,
+  configuration and recorded model/effort.
+- `snapshot/tasks/biomedical-graph-retrieval/`: exact pre-repair task snapshot.
+- `jobs/2026-10-15__00-44-54/biomedical-graph-retrieval__7iyRMF9/`: original trial;
+  contains `result.json`, `verifier/biomedical-graph-retrieval-eval/evaluation.json`,
+  `verifier/reward.json`, `verifier/reward-details.json` and `agent/trajectory.json`.
+
+These original files are local author-only evidence, not a published log bundle.
+This document supplies the non-sensitive summary; a maintainer-accessible
+redacted log handoff remains separate. Full trajectories, submitted code and
+hidden evaluation inputs stay outside the public task package.
+
+## Earlier-version evidence
+
+The following model and runtime results concern earlier versions and are kept
+separate from the completed v0.2.1 trial above.
 
 The v0.1.1 full Codex/OpenRouter openai/gpt-6-luna max run scored 19/20
 (0.95), with a passing trajectory audit. This supersedes the earlier fixed
@@ -41,7 +156,8 @@ returned 0/20 with valid output and no timeout. These checks had no audit API;
 their deterministic accuracies are not final audited rewards. The saved replay
 does not establish difficulty for a fresh Coding Agent.
 
-A new Coding Agent run was stopped at the user's instruction before submission
-or final scoring. No new full-agent difficulty score is available. Further
-Harbor or paid-model evaluation requires an explicit user request. The three
-unpublished remote asset pins remain a separate release limitation.
+At that earlier revision, a coding-agent attempt was stopped at the user's
+instruction before submission or final scoring. That stopped attempt produced
+no final score; the later completed v0.2.1 trial is recorded above. No additional
+paid run was launched for this evidence reconciliation. The three unpublished
+remote asset pins remain a separate release limitation.

tasks/biomedical-graph-retrieval/assets.json

@@ -14,24 +14,24 @@
     },
     {
       "path": "data/validation/queries.jsonl",
-      "size_bytes": 2087,
-      "sha256": "2320b2b700e63789e64bec51f7d8467d6a81d1cc137a2f8a10e8c74124a8ba73",
+      "size_bytes": 4056,
+      "sha256": "c4ca5e9a2bfe50db7594ac7b6bcbec25837eb20b028eb752736674bfe9ce01ed",
       "source": {
         "repo_id": "search-swe/Search-SWE",
         "repo_type": "dataset",
-        "revision": "2a2188f3a9426361da5f10c82a35148269574a92",
-        "filename": "tasks/biomedical-graph-retrieval/data/validation/queries.jsonl"
+        "revision": "234406eb7e304478e48905829fc0d3233f5a9f1f",
+        "filename": "tasks/biomedical-graph-retrieval/v0.2.2/data/validation/queries.jsonl"
       }
     },
     {
       "path": "data/validation/ground_truth.jsonl",
-      "size_bytes": 252,
-      "sha256": "9f82d59ceb15b1ed7acec09c644692017f81db8502c37a6400f740976be6d017",
+      "size_bytes": 628,
+      "sha256": "b1c9635f5c4f7449bb09d79fadf44667d6da83c666c3541e45d55f632bc30661",
       "source": {
         "repo_id": "search-swe/Search-SWE",
         "repo_type": "dataset",
-        "revision": "2a2188f3a9426361da5f10c82a35148269574a92",
-        "filename": "tasks/biomedical-graph-retrieval/data/validation/ground_truth.jsonl"
+        "revision": "234406eb7e304478e48905829fc0d3233f5a9f1f",
+        "filename": "tasks/biomedical-graph-retrieval/v0.2.2/data/validation/ground_truth.jsonl"
       }
     }
   ]