Fix task contracts, backbone audit, and invalid evaluation labels

tasks/chunqiu-temporal-retrieval/README.md

@@ -12,7 +12,7 @@
 | Agent / verifier time limits | 21600 s / 3000 s |
 | Dataset | ChunQiuTR, audited event-only derivative |
 | Public queries | 0 |
-| Hidden queries | 778 |
+| Hidden queries | 775 |
 
 Retrieve original event records for reign-based dates and time windows in
 Classical Chinese annals. Return up to ten records from the complete corpus.
@@ -30,7 +30,7 @@ commit `392d6ec8d62de8256b3b45578f7b4484c22042ce`.
 | Shared corpus | 18,662 |
 | Public labeled queries | 6,617 |
 | Public training-month document associations | 19,452 |
-| Hidden test queries | 778 |
+| Hidden test queries | 775 |
 | Public validation queries | 0 |
 
 The corpus preserves every original event/comment text after removing all
@@ -68,7 +68,7 @@ See [LICENSE.data.md](LICENSE.data.md) for source and model notices. Public inpu
 
 | Date | Version | Description |
 | --- | --- | --- |
-| 2026-10-23 | 1.0.0 | Added initial task |
+| 2026-10-28 | 1.0.0 | Removed three hidden queries with unsupported April labels; retained 775 queries |
 
 ## Difficulty Explanation
 

tasks/chunqiu-temporal-retrieval/RUN_RESULTS.md

@@ -1,5 +1,9 @@
 # Run evidence reconciled on 2026-10-23
 
+The results below describe the historical 778-query evaluation. The 2026-10-28
+revision retains 775 queries; see [VALIDATION.md](VALIDATION.md). These historical
+model scores have not been recalculated for that revision.
+
 The verifier produced valid nDCG@10 over 778 queries. A command-timeout exception is retained in the original run. This low score does not establish that training a temporal representation is infeasible; that training route remains unvalidated.
 
 | Field | Observed value |

tasks/chunqiu-temporal-retrieval/VALIDATION.md

@@ -1,4 +1,35 @@
-# Local validation — 2026-10-14
+# Local validation
+
+## Query removal — 2026-10-28, v1.0.0
+
+Removed these three hidden queries for 魯隱公二年四月 and their qrels rows:
+
+- `query_0a0f0c586a62ce91a86f4ee6`
+- `query_f1aa6ed34a522f97c291e5e3`
+- `query_2c213f84637f1595e3f2e81f`
+
+All three used `doc_086389fc7d08e192db320dcb` and
+`doc_6ee1a8ee1c6f5c792a1b060e` as positives. The first says
+"夏，莒人入向，以姜氏還" without specifying April; the corpus also contains
+"夏，五月，莒人入向" for the same event. The second passage describes 無駭入極
+without specifying a month. The user selected removal of the three precise-month
+queries. The passages remain in the corpus and in the qrels of retained queries.
+
+The hidden set now contains **775 queries**, with the retained rows unchanged.
+Corpus, public examples, dated examples, runtime limits and the nDCG@10 formula
+are unchanged. The grader averages over the 775 retained queries; this task has
+no baseline subtraction or normalization anchor to recalculate. The package
+version remains **1.0.0**. Historical model scores below use the earlier
+778-query set and have not been recalculated; no new model or judge run was made.
+
+Validation: all ten task-local tests passed, including production data loading
+and query-count agreement with `task.toml`. The task static checker reported zero
+errors. Comparison with the previous revision confirmed that each private JSONL
+file lost exactly the three specified rows and retained every other row unchanged.
+The corpus and both public example files still match their published SHA-256
+checksums. `git diff --check` passed.
+
+## Archived local validation — 2026-10-14
 
 Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages.
 

tasks/chunqiu-temporal-retrieval/task.toml

@@ -17,7 +17,7 @@ metric_k = 10
 corpus_document_count = 18662
 example_count = 6617
 public_query_count = 0
-hidden_query_count = 778
+hidden_query_count = 775
 base_model = "Qwen/Qwen3-Embedding-0.6B"
 index_build_timeout_seconds = 1200
 inference_timeout_seconds = 120

tasks/chunqiu-temporal-retrieval/tests/data/ground_truth.jsonl

@@ -1,6 +1,4 @@
 {"query_id":"query_3792d88c9dc9f0666280440e","relevant_doc_ids":["doc_2093698915f31cc7361d1735","doc_d66fa4a5d755f4ed3223b6f0","doc_0c0382b122d3b47c05c09807","doc_40256812cbc515d6e7ce8b6d"]}
-{"query_id":"query_0a0f0c586a62ce91a86f4ee6","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]}
-{"query_id":"query_f1aa6ed34a522f97c291e5e3","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]}
 {"query_id":"query_d2c507b3f98cac5c011100dd","relevant_doc_ids":["doc_75db28eaee83486bd3083d2e","doc_627aa862a7be1280c0c7ea42","doc_d39f4de34239e40c809a5d3e","doc_32f0cf562fec7a85c55d1243","doc_bcb9c94f32c67e61cdce83d1","doc_9bb7f43e50cf092b2178f1ec"]}
 {"query_id":"query_62a6ac7f2d7b5a0c09cb2c8e","relevant_doc_ids":["doc_247e59b071489e83629e9087"]}
 {"query_id":"query_b03117896ba1acccf8ac48bc","relevant_doc_ids":["doc_d5448628d9a685c14c8edc16","doc_4afcff7215061b804e179aa3","doc_0a46502f8f628c19797f4dd3"]}
@@ -205,7 +203,6 @@
 {"query_id":"query_c00b6de051d93716651ef510","relevant_doc_ids":["doc_86bc3cdf45100656b993c92b","doc_b1abed920d20bcdad1ab19f4","doc_4f5bfcece9c3ef04a97bb601","doc_073542a3a10de8fe52f23eda","doc_55d7750f90137a87435923ee","doc_dd12972425fce8f78e2b43b4","doc_62f058fdfbe6b5f469734fca"]}
 {"query_id":"query_7cdd67fd1b72f4541d4fcc26","relevant_doc_ids":["doc_86bc3cdf45100656b993c92b","doc_b1abed920d20bcdad1ab19f4","doc_4f5bfcece9c3ef04a97bb601","doc_073542a3a10de8fe52f23eda","doc_55d7750f90137a87435923ee","doc_dd12972425fce8f78e2b43b4","doc_62f058fdfbe6b5f469734fca"]}
 {"query_id":"query_7d3b0f151aecfac1d9b3b1a4","relevant_doc_ids":["doc_2093698915f31cc7361d1735","doc_d66fa4a5d755f4ed3223b6f0","doc_0c0382b122d3b47c05c09807","doc_40256812cbc515d6e7ce8b6d"]}
-{"query_id":"query_2c213f84637f1595e3f2e81f","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]}
 {"query_id":"query_eb7e78d44d5f536ceda3e643","relevant_doc_ids":["doc_75db28eaee83486bd3083d2e","doc_627aa862a7be1280c0c7ea42","doc_d39f4de34239e40c809a5d3e","doc_32f0cf562fec7a85c55d1243","doc_bcb9c94f32c67e61cdce83d1","doc_9bb7f43e50cf092b2178f1ec"]}
 {"query_id":"query_9f1e246877830448c296bf16","relevant_doc_ids":["doc_247e59b071489e83629e9087"]}
 {"query_id":"query_f4fcc00b88fa5d81ab20688b","relevant_doc_ids":["doc_d5448628d9a685c14c8edc16","doc_4afcff7215061b804e179aa3","doc_0a46502f8f628c19797f4dd3"]}

tasks/chunqiu-temporal-retrieval/tests/data/queries.jsonl

@@ -1,6 +1,4 @@
 {"query_id":"query_3792d88c9dc9f0666280440e","text":"請列出魯隱公二年正月在《春秋》中的記事。"}
-{"query_id":"query_0a0f0c586a62ce91a86f4ee6","text":"魯隱公二年四月在《春秋》裡究竟有無條目？"}
-{"query_id":"query_f1aa6ed34a522f97c291e5e3","text":"查詢《春秋》，魯隱公二年四月這一月有什麼重要事件？"}
 {"query_id":"query_d2c507b3f98cac5c011100dd","text":"請問魯隱公二年五月這一月，《春秋》有沒有書寫事件？"}
 {"query_id":"query_62a6ac7f2d7b5a0c09cb2c8e","text":"《春秋》在魯隱公二年七月是否另立一條記事？"}
 {"query_id":"query_b03117896ba1acccf8ac48bc","text":"魯隱公二年八月在《春秋》裡究竟有無條目？"}
@@ -205,7 +203,6 @@
 {"query_id":"query_c00b6de051d93716651ef510","text":"魯哀公八年十二月，《春秋》經傳有無記載？"}
 {"query_id":"query_7cdd67fd1b72f4541d4fcc26","text":"請列出魯哀公八年十二月在《春秋》中的記事。"}
 {"query_id":"query_7d3b0f151aecfac1d9b3b1a4","text":"請列出魯隱公二年正月在《春秋》中的記事。"}
-{"query_id":"query_2c213f84637f1595e3f2e81f","text":"魯隱公二年四月發生了什麼事？"}
 {"query_id":"query_eb7e78d44d5f536ceda3e643","text":"魯隱公二年五月，《春秋》經文對此月是記事還是無事可書？"}
 {"query_id":"query_9f1e246877830448c296bf16","text":"魯隱公二年七月發生了什麼事？"}
 {"query_id":"query_f4fcc00b88fa5d81ab20688b","text":"《春秋》在魯隱公二年八月是否另立一條記事？"}

tasks/chunqiu-temporal-retrieval/tests/grader.py

@@ -24,7 +24,7 @@
 LOGS = Path("/logs/verifier/chunqiu-temporal-retrieval-eval")
 UID = GID = 10001
 CORPUS_COUNT = 18662
-QUERY_COUNT = 778
+QUERY_COUNT = 775
 BUILD_TIMEOUT_SECONDS = 1200
 QUERY_TIMEOUT_SECONDS = 120
 

tasks/chunqiu-temporal-retrieval/tests/test_contract.py

@@ -3,6 +3,7 @@
 import os
 from pathlib import Path
 import tempfile
+import tomllib
 import unittest
 from unittest import mock
 
@@ -112,7 +113,9 @@ def test_frozen_private_data_integrity(self):
         with mock.patch.object(grader, "TASK_DATA", task / "data"), mock.patch.object(grader, "PRIVATE", task / "tests/data"):
             corpus, queries, gold = grader.load_data()
         self.assertEqual(len(corpus), 18662)
-        self.assertEqual(len(queries), 778)
+        self.assertEqual(len(queries), 775)
+        metadata = tomllib.loads((task / "task.toml").read_text())["metadata"]
+        self.assertEqual(len(queries), metadata["hidden_query_count"])
         examples = grader.read_jsonl(task / "data/examples.jsonl")
         self.assertEqual(len(examples), 6617)
         self.assertFalse({q["query_id"] for q in examples} & gold.keys())

tasks/code-context-retrieval/README.md

@@ -20,7 +20,9 @@ Build a reusable search engine that returns the source regions needed to underst
 
 The shared `data/repos/` directory contains 51,920 complete eligible text files from 30 pinned snapshots across 22 repositories, totaling 324,317,309 bytes. Each snapshot has an anonymous `repo_<id>` root directory; internal relative paths, file bytes and line breaks are preserved. Each query specifies exactly one snapshot to search. Repository names and source-commit mappings are not supplied as runtime metadata; source text and issue content are unchanged and may still identify their projects.
 
-Public validation contains 10 original issues from seven repositories. The verifier holds 20 original issues from 15 different repositories. Repositories do not overlap between splits, and no repository contributes more than three questions to a split. Both original issue text and `read_core_regions` are preserved; the source's outer JSON string encoding is decoded where applicable. Queries identify their target snapshot.
+Public validation contains 10 original issues from seven repositories. The verifier holds 20 original issues from 15 different repositories. Repositories do not overlap between splits, and no repository contributes more than three questions to a split. Original issue text is preserved; `read_core_regions` are preserved except for the correction below. The source's outer JSON string encoding is decoded where applicable. Queries identify their target snapshot.
+
+On 2026-10-28, the standalone `include/nlohmann/detail/meta/type_traits.hpp:1–5` reference region was removed from `test-009`. These lines contain only the project banner, version and URL, with no implementation relevant to the unsigned-enum serialization issue. The query retains its other four regions, totaling 201 reference lines. Source files and the 20-query test set are unchanged.
 
 Corpus eligibility matches the screening experiment: regular nonsymlink UTF-8 files up to 1 MiB, nonempty physical line content, no NUL in the first 4096 bytes, excluding `.git`, `node_modules` and `__pycache__`. Original file paths and line breaks remain unchanged. Source-defined Pro file overrides were reproduced without executing setup commands; audited export omissions were restored at the pinned revisions.
 
@@ -30,22 +32,23 @@ The solver supplies executable `/app/build.sh` and `/app/run.sh` with the comman
 
 | Date | Version | Description |
 | --- | --- | --- |
-| 2026-10-23 | 1.0.0 | Added initial task |
+| 2026-10-28 | 1.0.0 | Removed an unrelated five-line header region from test-009 reference labels |
 
 ## Difficulty Explanation
 
 ### Difficulty evidence
 
 One packaged CPU/API Codex/OpenRouter `openai/gpt-6-luna`, `max` trial produced
-mean line F1 and reward **0.17146875932337885**. The coding agent reached its
+mean line F1 and reward **0.17146875932337885** under the labels before the
+2026-10-28 correction. The coding agent reached its
 7,200-second development limit (`AgentTimeoutError`); the subsequent verifier
 completed all 20 queries with valid outputs and no query timeouts. This is a
 scored submission after development-budget exhaustion, not an exception-free
 agent completion. [VALIDATION.md](VALIDATION.md) records the trial, matching
 instruction fingerprint, timings and original evidence locations. No online
 trajectory judge is configured for this task.
 
-Author-side screening evaluated the same selected issues with a common five-region/500-line output budget and 180-second query limit. Dense retrieval and reranking used local Qwen3 8B models; agents used local Qwen3.5-9B without thinking, at most 25 exploration steps plus a final call. Local GPU pilot latency is not a measurement of the packaged CPU/API environment. Index/model startup was excluded; enhanced-agent timing summed separately measured retrieval and exploration.
+Author-side screening evaluated the same selected issues with the labels before the 2026-10-28 correction, a common five-region/500-line output budget and a 180-second query limit. Dense retrieval and reranking used local Qwen3 8B models; agents used local Qwen3.5-9B without thinking, at most 25 exploration steps plus a final call. Local GPU pilot latency is not a measurement of the packaged CPU/API environment. Index/model startup was excluded; enhanced-agent timing summed separately measured retrieval and exploration.
 
 | Method | Test mean F1 | Validation mean F1 |
 | --- | ---: | ---: |
@@ -66,7 +69,7 @@ This is a deliberately selected challenge subset, not a benchmark-wide estimate.
 
 ### Metric and limits
 
-A result contains up to five source regions and 500 distinct physical lines. Per-query line F1 is `2 * intersection / (predicted_lines + gold_lines)`, matching `(path, line)` in the requested snapshot. Overlaps are deduplicated; ordering has no effect. Every selected original gold fits the budget and has an attainable oracle score of 1. The reward is the mean over all 20 hidden questions, with no baseline rescaling. Precision and recall are diagnostics.
+A result contains up to five source regions and 500 distinct physical lines. Per-query line F1 is `2 * intersection / (predicted_lines + gold_lines)`, matching `(path, line)` in the requested snapshot. Overlaps are deduplicated; ordering has no effect. Every packaged reference set fits the budget and has an attainable oracle score of 1. The reward is the mean over all 20 hidden questions, with no baseline rescaling. Precision and recall are diagnostics.
 
 Invalid or failed query outputs and timeouts score zero for that query. Build failure scores zero overall. A confirmed integrity violation invalidates the submission. Scoring is deterministic; no subjective answer judge is used. The API model allowlist is a task policy; the line scorer alone does not certify model/provider compliance.
 

tasks/code-context-retrieval/RUN_RESULTS.md

@@ -1,5 +1,7 @@
 # Run evidence reconciled on 2026-10-23
 
+Label correction (2026-10-28, package version 1.0.0): `test-009` now excludes the unrelated `include/nlohmann/detail/meta/type_traits.hpp:1–5` region. The recorded scores below and the README difficulty results use the earlier labels and have not been recomputed. They remain historical run evidence.
+
 The agent reached its development deadline; subsequent verifier scoring completed. Mean line-level F1 is a valid retrieval result, not a clean agent-completion claim.
 
 | Field | Observed value |

tasks/code-context-retrieval/VALIDATION.md

@@ -1,4 +1,17 @@
-# Local validation, version 0.1.0
+# Local validation
+
+## Reference-label correction — 2026-10-28, version 1.0.0
+
+Removed only `include/nlohmann/detail/meta/type_traits.hpp:1–5` from the `test-009` reference regions. These five lines contain a project banner, version and URL; they do not implement unsigned-enum serialization. The other four regions remain unchanged, reducing the reference set from 206 to 201 distinct physical lines. All 20 hidden queries remain in the test set.
+
+- Compared the packaged labels against the previous Git revision: only this region changed; the other 19 reference rows are byte-identical. Query text, source files, task metadata, grader and scoring formula are unchanged.
+- Loaded the complete repository corpus with the production scorer and validated all 20 reference outputs: every output satisfies the five-region/500-line contract, stays within actual file bounds and scores F1 = 1 against its current labels.
+- Scored a synthetic output containing the four retained regions: the old labels gave F1 = 0.9877149877149877; the corrected labels give F1 = 1. This is a scorer check, not an agent result.
+- All eight existing scoring contract tests passed. `python scripts/check_tasks.py tasks/code-context-retrieval` reported zero errors.
+
+Historical model and baseline scores below and in `RUN_RESULTS.md` use the earlier labels; they have not been recomputed. No new agent or model run was performed. Scoring remains direct mean line F1, so no baseline normalization constant requires updating.
+
+## Archived validation, version 0.1.0
 
 Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages.
 

tasks/code-context-retrieval/tests/data/ground_truth.jsonl

@@ -6,7 +6,7 @@
 {"query_id":"test-006","regions":[{"path":"openlibrary/plugins/worksearch/schemes/works.py","start":160,"end":250},{"path":"openlibrary/plugins/worksearch/code.py","start":350,"end":430},{"path":"openlibrary/solr/updater/work.py","start":220,"end":224},{"path":"openlibrary/plugins/worksearch/tests/test_worksearch.py","start":57,"end":61}]}
 {"query_id":"test-007","regions":[{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/TypeExtractor.java","start":260,"end":330},{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/TypeExtractor.java","start":400,"end":450},{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/DefaultVisitorAdapter.java","start":205,"end":209},{"path":"javaparser-core/src/main/java/com/github/javaparser/ast/expr/TextBlockLiteralExpr.java","start":34,"end":34}]}
 {"query_id":"test-008","regions":[{"path":"py/vm.c","start":940,"end":1085},{"path":"py/runtime.c","start":700,"end":940},{"path":"tests/basics/fun_calldblstar4.py","start":11,"end":15},{"path":"py/objfun.c","start":230,"end":234},{"path":"py/map.c","start":205,"end":209}]}
-{"query_id":"test-009","regions":[{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":1,"end":150},{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":292,"end":332},{"path":"include/nlohmann/detail/meta/type_traits.hpp","start":1,"end":5},{"path":"include/nlohmann/json.hpp","start":319,"end":323},{"path":"single_include/nlohmann/json.hpp","start":4767,"end":4771}]}
+{"query_id":"test-009","regions":[{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":1,"end":150},{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":292,"end":332},{"path":"include/nlohmann/json.hpp","start":319,"end":323},{"path":"single_include/nlohmann/json.hpp","start":4767,"end":4771}]}
 {"query_id":"test-010","regions":[{"path":"crates/nu-command/src/system/run_external.rs","start":1,"end":220},{"path":"crates/nu-command/src/system/run_external.rs","start":440,"end":700},{"path":"crates/nu-cmd-lang/src/core_commands/hide_env.rs","start":31,"end":35},{"path":"crates/nu-protocol/src/engine/stack.rs","start":280,"end":284}]}
 {"query_id":"test-011","regions":[{"path":"src/PhpSpreadsheet/Style/NumberFormat/BaseFormatter.php","start":1,"end":25},{"path":"src/PhpSpreadsheet/Shared/StringHelper.php","start":21,"end":25},{"path":"src/PhpSpreadsheet/Style/NumberFormat/Formatter.php","start":117,"end":121},{"path":"src/PhpSpreadsheet/Style/NumberFormat/NumberFormatter.php","start":132,"end":136},{"path":"src/PhpSpreadsheet/Style/NumberFormat/PercentageFormatter.php","start":42,"end":46}]}
 {"query_id":"test-012","regions":[{"path":"src/diff/children.js","start":50,"end":65},{"path":"src/diff/index.js","start":31,"end":35}]}