diff --git a/tasks/chunqiu-temporal-retrieval/README.md b/tasks/chunqiu-temporal-retrieval/README.md index 20ee982..debf2e5 100644 --- a/tasks/chunqiu-temporal-retrieval/README.md +++ b/tasks/chunqiu-temporal-retrieval/README.md @@ -12,7 +12,7 @@ | Agent / verifier time limits | 21600 s / 3000 s | | Dataset | ChunQiuTR, audited event-only derivative | | Public queries | 0 | -| Hidden queries | 778 | +| Hidden queries | 775 | Retrieve original event records for reign-based dates and time windows in Classical Chinese annals. Return up to ten records from the complete corpus. @@ -30,7 +30,7 @@ commit `392d6ec8d62de8256b3b45578f7b4484c22042ce`. | Shared corpus | 18,662 | | Public labeled queries | 6,617 | | Public training-month document associations | 19,452 | -| Hidden test queries | 778 | +| Hidden test queries | 775 | | Public validation queries | 0 | The corpus preserves every original event/comment text after removing all @@ -68,7 +68,7 @@ See [LICENSE.data.md](LICENSE.data.md) for source and model notices. Public inpu | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Removed three hidden queries with unsupported April labels; retained 775 queries | ## Difficulty Explanation diff --git a/tasks/chunqiu-temporal-retrieval/RUN_RESULTS.md b/tasks/chunqiu-temporal-retrieval/RUN_RESULTS.md index e479a26..9ddbf2b 100644 --- a/tasks/chunqiu-temporal-retrieval/RUN_RESULTS.md +++ b/tasks/chunqiu-temporal-retrieval/RUN_RESULTS.md @@ -1,5 +1,9 @@ # Run evidence reconciled on 2026-10-23 +The results below describe the historical 778-query evaluation. The 2026-10-28 +revision retains 775 queries; see [VALIDATION.md](VALIDATION.md). These historical +model scores have not been recalculated for that revision. + The verifier produced valid nDCG@10 over 778 queries. A command-timeout exception is retained in the original run. This low score does not establish that training a temporal representation is infeasible; that training route remains unvalidated. | Field | Observed value | diff --git a/tasks/chunqiu-temporal-retrieval/VALIDATION.md b/tasks/chunqiu-temporal-retrieval/VALIDATION.md index 916daa9..e8214f9 100644 --- a/tasks/chunqiu-temporal-retrieval/VALIDATION.md +++ b/tasks/chunqiu-temporal-retrieval/VALIDATION.md @@ -1,4 +1,35 @@ -# Local validation — 2026-10-14 +# Local validation + +## Query removal — 2026-10-28, v1.0.0 + +Removed these three hidden queries for 魯隱公二年四月 and their qrels rows: + +- `query_0a0f0c586a62ce91a86f4ee6` +- `query_f1aa6ed34a522f97c291e5e3` +- `query_2c213f84637f1595e3f2e81f` + +All three used `doc_086389fc7d08e192db320dcb` and +`doc_6ee1a8ee1c6f5c792a1b060e` as positives. The first says +"夏，莒人入向，以姜氏還" without specifying April; the corpus also contains +"夏，五月，莒人入向" for the same event. The second passage describes 無駭入極 +without specifying a month. The user selected removal of the three precise-month +queries. The passages remain in the corpus and in the qrels of retained queries. + +The hidden set now contains **775 queries**, with the retained rows unchanged. +Corpus, public examples, dated examples, runtime limits and the nDCG@10 formula +are unchanged. The grader averages over the 775 retained queries; this task has +no baseline subtraction or normalization anchor to recalculate. The package +version remains **1.0.0**. Historical model scores below use the earlier +778-query set and have not been recalculated; no new model or judge run was made. + +Validation: all ten task-local tests passed, including production data loading +and query-count agreement with `task.toml`. The task static checker reported zero +errors. Comparison with the previous revision confirmed that each private JSONL +file lost exactly the three specified rows and retained every other row unchanged. +The corpus and both public example files still match their published SHA-256 +checksums. `git diff --check` passed. + +## Archived local validation — 2026-10-14 Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages. diff --git a/tasks/chunqiu-temporal-retrieval/task.toml b/tasks/chunqiu-temporal-retrieval/task.toml index 9a571b3..7b6e1c7 100644 --- a/tasks/chunqiu-temporal-retrieval/task.toml +++ b/tasks/chunqiu-temporal-retrieval/task.toml @@ -17,7 +17,7 @@ metric_k = 10 corpus_document_count = 18662 example_count = 6617 public_query_count = 0 -hidden_query_count = 778 +hidden_query_count = 775 base_model = "Qwen/Qwen3-Embedding-0.6B" index_build_timeout_seconds = 1200 inference_timeout_seconds = 120 diff --git a/tasks/chunqiu-temporal-retrieval/tests/data/ground_truth.jsonl b/tasks/chunqiu-temporal-retrieval/tests/data/ground_truth.jsonl index 8daa3e3..4e581f1 100644 --- a/tasks/chunqiu-temporal-retrieval/tests/data/ground_truth.jsonl +++ b/tasks/chunqiu-temporal-retrieval/tests/data/ground_truth.jsonl @@ -1,6 +1,4 @@ {"query_id":"query_3792d88c9dc9f0666280440e","relevant_doc_ids":["doc_2093698915f31cc7361d1735","doc_d66fa4a5d755f4ed3223b6f0","doc_0c0382b122d3b47c05c09807","doc_40256812cbc515d6e7ce8b6d"]} -{"query_id":"query_0a0f0c586a62ce91a86f4ee6","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]} -{"query_id":"query_f1aa6ed34a522f97c291e5e3","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]} {"query_id":"query_d2c507b3f98cac5c011100dd","relevant_doc_ids":["doc_75db28eaee83486bd3083d2e","doc_627aa862a7be1280c0c7ea42","doc_d39f4de34239e40c809a5d3e","doc_32f0cf562fec7a85c55d1243","doc_bcb9c94f32c67e61cdce83d1","doc_9bb7f43e50cf092b2178f1ec"]} {"query_id":"query_62a6ac7f2d7b5a0c09cb2c8e","relevant_doc_ids":["doc_247e59b071489e83629e9087"]} {"query_id":"query_b03117896ba1acccf8ac48bc","relevant_doc_ids":["doc_d5448628d9a685c14c8edc16","doc_4afcff7215061b804e179aa3","doc_0a46502f8f628c19797f4dd3"]} @@ -205,7 +203,6 @@ {"query_id":"query_c00b6de051d93716651ef510","relevant_doc_ids":["doc_86bc3cdf45100656b993c92b","doc_b1abed920d20bcdad1ab19f4","doc_4f5bfcece9c3ef04a97bb601","doc_073542a3a10de8fe52f23eda","doc_55d7750f90137a87435923ee","doc_dd12972425fce8f78e2b43b4","doc_62f058fdfbe6b5f469734fca"]} {"query_id":"query_7cdd67fd1b72f4541d4fcc26","relevant_doc_ids":["doc_86bc3cdf45100656b993c92b","doc_b1abed920d20bcdad1ab19f4","doc_4f5bfcece9c3ef04a97bb601","doc_073542a3a10de8fe52f23eda","doc_55d7750f90137a87435923ee","doc_dd12972425fce8f78e2b43b4","doc_62f058fdfbe6b5f469734fca"]} {"query_id":"query_7d3b0f151aecfac1d9b3b1a4","relevant_doc_ids":["doc_2093698915f31cc7361d1735","doc_d66fa4a5d755f4ed3223b6f0","doc_0c0382b122d3b47c05c09807","doc_40256812cbc515d6e7ce8b6d"]} -{"query_id":"query_2c213f84637f1595e3f2e81f","relevant_doc_ids":["doc_086389fc7d08e192db320dcb","doc_6ee1a8ee1c6f5c792a1b060e"]} {"query_id":"query_eb7e78d44d5f536ceda3e643","relevant_doc_ids":["doc_75db28eaee83486bd3083d2e","doc_627aa862a7be1280c0c7ea42","doc_d39f4de34239e40c809a5d3e","doc_32f0cf562fec7a85c55d1243","doc_bcb9c94f32c67e61cdce83d1","doc_9bb7f43e50cf092b2178f1ec"]} {"query_id":"query_9f1e246877830448c296bf16","relevant_doc_ids":["doc_247e59b071489e83629e9087"]} {"query_id":"query_f4fcc00b88fa5d81ab20688b","relevant_doc_ids":["doc_d5448628d9a685c14c8edc16","doc_4afcff7215061b804e179aa3","doc_0a46502f8f628c19797f4dd3"]} diff --git a/tasks/chunqiu-temporal-retrieval/tests/data/queries.jsonl b/tasks/chunqiu-temporal-retrieval/tests/data/queries.jsonl index de90ef0..7d0dc31 100644 --- a/tasks/chunqiu-temporal-retrieval/tests/data/queries.jsonl +++ b/tasks/chunqiu-temporal-retrieval/tests/data/queries.jsonl @@ -1,6 +1,4 @@ {"query_id":"query_3792d88c9dc9f0666280440e","text":"請列出魯隱公二年正月在《春秋》中的記事。"} -{"query_id":"query_0a0f0c586a62ce91a86f4ee6","text":"魯隱公二年四月在《春秋》裡究竟有無條目？"} -{"query_id":"query_f1aa6ed34a522f97c291e5e3","text":"查詢《春秋》，魯隱公二年四月這一月有什麼重要事件？"} {"query_id":"query_d2c507b3f98cac5c011100dd","text":"請問魯隱公二年五月這一月，《春秋》有沒有書寫事件？"} {"query_id":"query_62a6ac7f2d7b5a0c09cb2c8e","text":"《春秋》在魯隱公二年七月是否另立一條記事？"} {"query_id":"query_b03117896ba1acccf8ac48bc","text":"魯隱公二年八月在《春秋》裡究竟有無條目？"} @@ -205,7 +203,6 @@ {"query_id":"query_c00b6de051d93716651ef510","text":"魯哀公八年十二月，《春秋》經傳有無記載？"} {"query_id":"query_7cdd67fd1b72f4541d4fcc26","text":"請列出魯哀公八年十二月在《春秋》中的記事。"} {"query_id":"query_7d3b0f151aecfac1d9b3b1a4","text":"請列出魯隱公二年正月在《春秋》中的記事。"} -{"query_id":"query_2c213f84637f1595e3f2e81f","text":"魯隱公二年四月發生了什麼事？"} {"query_id":"query_eb7e78d44d5f536ceda3e643","text":"魯隱公二年五月，《春秋》經文對此月是記事還是無事可書？"} {"query_id":"query_9f1e246877830448c296bf16","text":"魯隱公二年七月發生了什麼事？"} {"query_id":"query_f4fcc00b88fa5d81ab20688b","text":"《春秋》在魯隱公二年八月是否另立一條記事？"} diff --git a/tasks/chunqiu-temporal-retrieval/tests/grader.py b/tasks/chunqiu-temporal-retrieval/tests/grader.py index 6f0fa72..0ec91c5 100644 --- a/tasks/chunqiu-temporal-retrieval/tests/grader.py +++ b/tasks/chunqiu-temporal-retrieval/tests/grader.py @@ -24,7 +24,7 @@ LOGS = Path("/logs/verifier/chunqiu-temporal-retrieval-eval") UID = GID = 10001 CORPUS_COUNT = 18662 -QUERY_COUNT = 778 +QUERY_COUNT = 775 BUILD_TIMEOUT_SECONDS = 1200 QUERY_TIMEOUT_SECONDS = 120 diff --git a/tasks/chunqiu-temporal-retrieval/tests/test_contract.py b/tasks/chunqiu-temporal-retrieval/tests/test_contract.py index 0f3cb4a..df1dc2d 100644 --- a/tasks/chunqiu-temporal-retrieval/tests/test_contract.py +++ b/tasks/chunqiu-temporal-retrieval/tests/test_contract.py @@ -3,6 +3,7 @@ import os from pathlib import Path import tempfile +import tomllib import unittest from unittest import mock @@ -112,7 +113,9 @@ def test_frozen_private_data_integrity(self): with mock.patch.object(grader, "TASK_DATA", task / "data"), mock.patch.object(grader, "PRIVATE", task / "tests/data"): corpus, queries, gold = grader.load_data() self.assertEqual(len(corpus), 18662) - self.assertEqual(len(queries), 778) + self.assertEqual(len(queries), 775) + metadata = tomllib.loads((task / "task.toml").read_text())["metadata"] + self.assertEqual(len(queries), metadata["hidden_query_count"]) examples = grader.read_jsonl(task / "data/examples.jsonl") self.assertEqual(len(examples), 6617) self.assertFalse({q["query_id"] for q in examples} & gold.keys()) diff --git a/tasks/code-context-retrieval/README.md b/tasks/code-context-retrieval/README.md index b8ae912..ee34518 100644 --- a/tasks/code-context-retrieval/README.md +++ b/tasks/code-context-retrieval/README.md @@ -20,7 +20,9 @@ Build a reusable search engine that returns the source regions needed to underst The shared `data/repos/` directory contains 51,920 complete eligible text files from 30 pinned snapshots across 22 repositories, totaling 324,317,309 bytes. Each snapshot has an anonymous `repo_` root directory; internal relative paths, file bytes and line breaks are preserved. Each query specifies exactly one snapshot to search. Repository names and source-commit mappings are not supplied as runtime metadata; source text and issue content are unchanged and may still identify their projects. -Public validation contains 10 original issues from seven repositories. The verifier holds 20 original issues from 15 different repositories. Repositories do not overlap between splits, and no repository contributes more than three questions to a split. Both original issue text and `read_core_regions` are preserved; the source's outer JSON string encoding is decoded where applicable. Queries identify their target snapshot. +Public validation contains 10 original issues from seven repositories. The verifier holds 20 original issues from 15 different repositories. Repositories do not overlap between splits, and no repository contributes more than three questions to a split. Original issue text is preserved; `read_core_regions` are preserved except for the correction below. The source's outer JSON string encoding is decoded where applicable. Queries identify their target snapshot. + +On 2026-10-28, the standalone `include/nlohmann/detail/meta/type_traits.hpp:1–5` reference region was removed from `test-009`. These lines contain only the project banner, version and URL, with no implementation relevant to the unsigned-enum serialization issue. The query retains its other four regions, totaling 201 reference lines. Source files and the 20-query test set are unchanged. Corpus eligibility matches the screening experiment: regular nonsymlink UTF-8 files up to 1 MiB, nonempty physical line content, no NUL in the first 4096 bytes, excluding `.git`, `node_modules` and `__pycache__`. Original file paths and line breaks remain unchanged. Source-defined Pro file overrides were reproduced without executing setup commands; audited export omissions were restored at the pinned revisions. @@ -30,14 +32,15 @@ The solver supplies executable `/app/build.sh` and `/app/run.sh` with the comman | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Removed an unrelated five-line header region from test-009 reference labels | ## Difficulty Explanation ### Difficulty evidence One packaged CPU/API Codex/OpenRouter `openai/gpt-6-luna`, `max` trial produced -mean line F1 and reward **0.17146875932337885**. The coding agent reached its +mean line F1 and reward **0.17146875932337885** under the labels before the +2026-10-28 correction. The coding agent reached its 7,200-second development limit (`AgentTimeoutError`); the subsequent verifier completed all 20 queries with valid outputs and no query timeouts. This is a scored submission after development-budget exhaustion, not an exception-free @@ -45,7 +48,7 @@ agent completion. [VALIDATION.md](VALIDATION.md) records the trial, matching instruction fingerprint, timings and original evidence locations. No online trajectory judge is configured for this task. -Author-side screening evaluated the same selected issues with a common five-region/500-line output budget and 180-second query limit. Dense retrieval and reranking used local Qwen3 8B models; agents used local Qwen3.5-9B without thinking, at most 25 exploration steps plus a final call. Local GPU pilot latency is not a measurement of the packaged CPU/API environment. Index/model startup was excluded; enhanced-agent timing summed separately measured retrieval and exploration. +Author-side screening evaluated the same selected issues with the labels before the 2026-10-28 correction, a common five-region/500-line output budget and a 180-second query limit. Dense retrieval and reranking used local Qwen3 8B models; agents used local Qwen3.5-9B without thinking, at most 25 exploration steps plus a final call. Local GPU pilot latency is not a measurement of the packaged CPU/API environment. Index/model startup was excluded; enhanced-agent timing summed separately measured retrieval and exploration. | Method | Test mean F1 | Validation mean F1 | | --- | ---: | ---: | @@ -66,7 +69,7 @@ This is a deliberately selected challenge subset, not a benchmark-wide estimate. ### Metric and limits -A result contains up to five source regions and 500 distinct physical lines. Per-query line F1 is `2 * intersection / (predicted_lines + gold_lines)`, matching `(path, line)` in the requested snapshot. Overlaps are deduplicated; ordering has no effect. Every selected original gold fits the budget and has an attainable oracle score of 1. The reward is the mean over all 20 hidden questions, with no baseline rescaling. Precision and recall are diagnostics. +A result contains up to five source regions and 500 distinct physical lines. Per-query line F1 is `2 * intersection / (predicted_lines + gold_lines)`, matching `(path, line)` in the requested snapshot. Overlaps are deduplicated; ordering has no effect. Every packaged reference set fits the budget and has an attainable oracle score of 1. The reward is the mean over all 20 hidden questions, with no baseline rescaling. Precision and recall are diagnostics. Invalid or failed query outputs and timeouts score zero for that query. Build failure scores zero overall. A confirmed integrity violation invalidates the submission. Scoring is deterministic; no subjective answer judge is used. The API model allowlist is a task policy; the line scorer alone does not certify model/provider compliance. diff --git a/tasks/code-context-retrieval/RUN_RESULTS.md b/tasks/code-context-retrieval/RUN_RESULTS.md index 813c8ba..d4617fb 100644 --- a/tasks/code-context-retrieval/RUN_RESULTS.md +++ b/tasks/code-context-retrieval/RUN_RESULTS.md @@ -1,5 +1,7 @@ # Run evidence reconciled on 2026-10-23 +Label correction (2026-10-28, package version 1.0.0): `test-009` now excludes the unrelated `include/nlohmann/detail/meta/type_traits.hpp:1–5` region. The recorded scores below and the README difficulty results use the earlier labels and have not been recomputed. They remain historical run evidence. + The agent reached its development deadline; subsequent verifier scoring completed. Mean line-level F1 is a valid retrieval result, not a clean agent-completion claim. | Field | Observed value | diff --git a/tasks/code-context-retrieval/VALIDATION.md b/tasks/code-context-retrieval/VALIDATION.md index 49e71d5..cdc924b 100644 --- a/tasks/code-context-retrieval/VALIDATION.md +++ b/tasks/code-context-retrieval/VALIDATION.md @@ -1,4 +1,17 @@ -# Local validation, version 0.1.0 +# Local validation + +## Reference-label correction — 2026-10-28, version 1.0.0 + +Removed only `include/nlohmann/detail/meta/type_traits.hpp:1–5` from the `test-009` reference regions. These five lines contain a project banner, version and URL; they do not implement unsigned-enum serialization. The other four regions remain unchanged, reducing the reference set from 206 to 201 distinct physical lines. All 20 hidden queries remain in the test set. + +- Compared the packaged labels against the previous Git revision: only this region changed; the other 19 reference rows are byte-identical. Query text, source files, task metadata, grader and scoring formula are unchanged. +- Loaded the complete repository corpus with the production scorer and validated all 20 reference outputs: every output satisfies the five-region/500-line contract, stays within actual file bounds and scores F1 = 1 against its current labels. +- Scored a synthetic output containing the four retained regions: the old labels gave F1 = 0.9877149877149877; the corrected labels give F1 = 1. This is a scorer check, not an agent result. +- All eight existing scoring contract tests passed. `python scripts/check_tasks.py tasks/code-context-retrieval` reported zero errors. + +Historical model and baseline scores below and in `RUN_RESULTS.md` use the earlier labels; they have not been recomputed. No new agent or model run was performed. Scoring remains direct mean line F1, so no baseline normalization constant requires updating. + +## Archived validation, version 0.1.0 Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages. diff --git a/tasks/code-context-retrieval/tests/data/ground_truth.jsonl b/tasks/code-context-retrieval/tests/data/ground_truth.jsonl index 948fe83..5eb365a 100644 --- a/tasks/code-context-retrieval/tests/data/ground_truth.jsonl +++ b/tasks/code-context-retrieval/tests/data/ground_truth.jsonl @@ -6,7 +6,7 @@ {"query_id":"test-006","regions":[{"path":"openlibrary/plugins/worksearch/schemes/works.py","start":160,"end":250},{"path":"openlibrary/plugins/worksearch/code.py","start":350,"end":430},{"path":"openlibrary/solr/updater/work.py","start":220,"end":224},{"path":"openlibrary/plugins/worksearch/tests/test_worksearch.py","start":57,"end":61}]} {"query_id":"test-007","regions":[{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/TypeExtractor.java","start":260,"end":330},{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/TypeExtractor.java","start":400,"end":450},{"path":"javaparser-symbol-solver-core/src/main/java/com/github/javaparser/symbolsolver/javaparsermodel/DefaultVisitorAdapter.java","start":205,"end":209},{"path":"javaparser-core/src/main/java/com/github/javaparser/ast/expr/TextBlockLiteralExpr.java","start":34,"end":34}]} {"query_id":"test-008","regions":[{"path":"py/vm.c","start":940,"end":1085},{"path":"py/runtime.c","start":700,"end":940},{"path":"tests/basics/fun_calldblstar4.py","start":11,"end":15},{"path":"py/objfun.c","start":230,"end":234},{"path":"py/map.c","start":205,"end":209}]} -{"query_id":"test-009","regions":[{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":1,"end":150},{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":292,"end":332},{"path":"include/nlohmann/detail/meta/type_traits.hpp","start":1,"end":5},{"path":"include/nlohmann/json.hpp","start":319,"end":323},{"path":"single_include/nlohmann/json.hpp","start":4767,"end":4771}]} +{"query_id":"test-009","regions":[{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":1,"end":150},{"path":"include/nlohmann/detail/conversions/to_json.hpp","start":292,"end":332},{"path":"include/nlohmann/json.hpp","start":319,"end":323},{"path":"single_include/nlohmann/json.hpp","start":4767,"end":4771}]} {"query_id":"test-010","regions":[{"path":"crates/nu-command/src/system/run_external.rs","start":1,"end":220},{"path":"crates/nu-command/src/system/run_external.rs","start":440,"end":700},{"path":"crates/nu-cmd-lang/src/core_commands/hide_env.rs","start":31,"end":35},{"path":"crates/nu-protocol/src/engine/stack.rs","start":280,"end":284}]} {"query_id":"test-011","regions":[{"path":"src/PhpSpreadsheet/Style/NumberFormat/BaseFormatter.php","start":1,"end":25},{"path":"src/PhpSpreadsheet/Shared/StringHelper.php","start":21,"end":25},{"path":"src/PhpSpreadsheet/Style/NumberFormat/Formatter.php","start":117,"end":121},{"path":"src/PhpSpreadsheet/Style/NumberFormat/NumberFormatter.php","start":132,"end":136},{"path":"src/PhpSpreadsheet/Style/NumberFormat/PercentageFormatter.php","start":42,"end":46}]} {"query_id":"test-012","regions":[{"path":"src/diff/children.js","start":50,"end":65},{"path":"src/diff/index.js","start":31,"end":35}]} diff --git a/tasks/code-embedding-finetuning/README.md b/tasks/code-embedding-finetuning/README.md index b44af12..185e95d 100644 --- a/tasks/code-embedding-finetuning/README.md +++ b/tasks/code-embedding-finetuning/README.md @@ -71,7 +71,7 @@ agent and private judge have their own API configuration. | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Clarified the fixed evaluation encoding protocol in task instructions | ## Difficulty Explanation diff --git a/tasks/code-embedding-finetuning/instruction.md b/tasks/code-embedding-finetuning/instruction.md index 826b7c5..c2fae30 100644 --- a/tasks/code-embedding-finetuning/instruction.md +++ b/tasks/code-embedding-finetuning/instruction.md @@ -4,6 +4,8 @@ Please read `/task/docs/environment.md` and `/task/docs/available_resources.md` The corpus and training data for improving the embedder are under `/task/data`. To check whether your changes are helping retrieval during development, there is also a fixed public validation split of 10 queries with labels under `/task/data/validation`. Hidden code-retrieval queries and their private relevance labels are disjoint from that public split and unavailable during your work. Do not use hidden data, hard-coded query-to-document mappings, precomputed answers or external resources. Network access is unavailable; both development and the submitted checkpoint must rely on the supplied local resources. +Evaluation encodes queries without a prefix and documents as `(title + " " + text).strip()`. Both are truncated to 512 tokens; the final-layer CLS vector is L2-normalized, and retrieval uses exact inner products. + I need to be able to load the improved model for retrieval in a separate environment, so write it to `/app/submission/model` as a complete, self-contained offline checkpoint. Both `AutoTokenizer.from_pretrained` and `AutoModel.from_pretrained` must be able to load it with `local_files_only=True`. The verifier collects only `/app/submission` and ignores files outside that directory, so the checkpoint must load without depending on uncollected files. You have a 6-hour agent phase. Task-specific submission verification has a 1-hour limit, which excludes the separate integrity judge. A timeout of the agent phase, the task-specific verification, or the integrity judge results in score 0. Once you finish, the checkpoint is copied into a separate verifier environment, where its architecture is checked against the supplied backbone. The verifier evaluates the original and submitted checkpoints with the same fixed embedding implementation and exact-search implementation. Larger improvement in held-out Accuracy@1 is better. diff --git a/tasks/code-embedding-finetuning/raw-instruction.md b/tasks/code-embedding-finetuning/raw-instruction.md index 49a0097..62615b2 100644 --- a/tasks/code-embedding-finetuning/raw-instruction.md +++ b/tasks/code-embedding-finetuning/raw-instruction.md @@ -37,6 +37,8 @@ The verifier ignores files outside /app/submission. The verifier copies the checkpoint into a separate environment, confirms that it preserves the supplied backbone architecture, and evaluates the original and submitted checkpoints with the same fixed embedding and exact-search implementation. Larger held-out Accuracy@1 improvement is better. +Evaluation encodes queries without a prefix and documents as `(title + " " + text).strip()`. Both are truncated to 512 tokens; the final-layer CLS vector is L2-normalized, and retrieval uses exact inner products. + ## Hidden Test Overview The hidden split contains code-retrieval queries and private relevance labels disjoint from the public validation split. Hidden data is unavailable during the Agent phase. diff --git a/tasks/quantity-aware-retrieval/README.md b/tasks/quantity-aware-retrieval/README.md index 9d707c3..67daecf 100644 --- a/tasks/quantity-aware-retrieval/README.md +++ b/tasks/quantity-aware-retrieval/README.md @@ -12,7 +12,7 @@ | Agent / verifier time limits | 21600 s / 4800 s | | Dataset | MedQuant, original passages and qrels | | Public queries | 0 | -| Hidden queries | 210 | +| Hidden queries | 209 | Author: **Bingyu**. This Search-SWE task uses MedQuant. The required deliverable is an executable retrieval system. Resources @@ -27,21 +27,22 @@ provided to the agent. | Corpus | 153,252 original passages | | Labeled examples (`data/examples.jsonl`) | 80,000; 12,453 distinct query texts | | Public retrieval validation | None | -| Private evaluation | All 210 original queries | +| Private evaluation | 209 queries retained from the original 210 | -Corpus text, questions and original positive sets are unchanged, except opaque -ID renaming. Corpus order is preserved. The source corpus's +One query with invalid positive labels has been removed; see +[the validation record](VALIDATION.md). Corpus text and the retained queries and +positive sets are unchanged, except opaque ID renaming. Corpus order is preserved. The source corpus's quantity fields and query condition/unit/value annotations are not exposed. The examples include synthetic edited passages and do not form a corpus-ID relevance file. The agent-facing instructions describe inputs, outputs, constraints and scoring without prescribing an implementation method. -All original evaluation questions remain private. Benchmark concepts and their +All retained evaluation questions remain private. Benchmark concepts and their source expansion components were excluded during training generation. A later review also checked physical/clinical examinations, death-rate, rebleeding, placebo, uric-acid and pain/VAS families. This automatic exclusion and limited -family review do not prove universal semantic isolation. No source question or -positive set was removed or rewritten. +family review do not prove universal semantic isolation. The retained query texts +and positive sets have not been rewritten. Authoring sources, the split manifest, calibration evidence and validation fixtures are in `../../../task_development/quantity-aware-retrieval/`, outside @@ -52,7 +53,7 @@ part of the package. | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Removed one query with invalid positives and recalculated the 209-query baseline | ## Difficulty Explanation @@ -75,14 +76,15 @@ Qwen3-Embedding-8B, top-1000 each, fused with RRF constant 60. | Reference scope | Mean nDCG@10 | | --- | ---: | -| All original 210 questions | 0.1838410794397215 | -| User-selected rounded reward anchor | **0.1838** | +| Retained 209 questions | 0.18472070182938527 | +| Four-decimal reward anchor | **0.1847** | -For raw private score Q and B = 0.1838, reward is +For raw private score Q and B = 0.1847, reward is `max(0, min(1, (Q-B)/(1-B)))`. The anchor uses the four-decimal rounding explicitly -chosen by the user. The reference comes from the archived single-A100 hybrid -retrieval run in `../../../task_development/quantity-aware-retrieval/research/medquant-baselines-v1/summary.json` -(`methods.hybrid`). No per-query baseline +chosen by the user. The reference is recalculated over the retained queries from +the archived single-A100 hybrid retrieval results in +`../../../task_development/quantity-aware-retrieval/research/medquant-baselines-v1/per-query-results.jsonl` +(`methods.hybrid`); [VALIDATION.md](VALIDATION.md) records the exclusion and calculation. No per-query baseline oracle or choice of the best model on private results is used. An independent trajectory integrity judge uses the repository's pinned diff --git a/tasks/quantity-aware-retrieval/RUN_RESULTS.md b/tasks/quantity-aware-retrieval/RUN_RESULTS.md index c86178d..8256746 100644 --- a/tasks/quantity-aware-retrieval/RUN_RESULTS.md +++ b/tasks/quantity-aware-retrieval/RUN_RESULTS.md @@ -1,5 +1,9 @@ # Run evidence reconciled on 2026-10-23 +The results below describe the historical 210-query evaluation. The 2026-10-28 +revision retains 209 queries and uses baseline 0.1847; see [VALIDATION.md](VALIDATION.md). +These historical model scores have not been recalculated for that revision. + The saved v0.1.0 evaluation and passing audit are reused. Version 0.1.1 changes the zero-score baseline to 0.1838, with ceiling 1. The current reward is an offline recalculation from saved nDCG@10; no new agent, retrieval or online judge run occurred. | Field | Observed value | diff --git a/tasks/quantity-aware-retrieval/VALIDATION.md b/tasks/quantity-aware-retrieval/VALIDATION.md index 1e894e5..3a7ff6e 100644 --- a/tasks/quantity-aware-retrieval/VALIDATION.md +++ b/tasks/quantity-aware-retrieval/VALIDATION.md @@ -1,5 +1,37 @@ # Local validation +## Query removal and baseline recalculation — 2026-10-28, v1.0.0 + +Removed `query_0b014897ccb389626678` ("body weight less than 0.1 kilograms") +and its qrels row. Its two labeled positives describe a body-weight change of +-0.38 kg and measurement accuracy of 0.01 kg; neither states that body weight +is below 0.1 kg. The user selected removal of this query. The hidden set now +contains 209 queries, with the other query texts and qrels unchanged. The +153,252-document corpus and 80,000 training examples are unchanged. + +The baseline method remains BM25 + Qwen3-Embedding-8B, top-1000 each, RRF 60. +Recalculation uses the archived `methods.hybrid.top10_doc_ids` from +`../../../task_development/quantity-aware-retrieval/research/medquant-baselines-v1/per-query-results.jsonl` +(SHA-256 `f5904f1a463c3e15f1e2df75c2e739c8978598990529197724b5b3e83e5c42f2`). +All 210 archived query texts and gold sets were matched to the original task +files through the unchanged corpus row-to-ID mapping. Recomputing with the +production `ndcg_at_10` function reproduces the archived 210-query mean +0.1838410794397215. The removed query's baseline nDCG@10 is 0, so the retained +209-query mean is **0.18472070182938527**. Applying the existing four-decimal +rounding rule gives the new reward anchor **0.1847** in `task.toml` and +`tests/grader.py`. + +This recalculation reuses saved rankings; it does not rerun model inference or +the integrity judge. Historical run artifacts and their recorded scores are +preserved. The dated sections below describe earlier 210-query revisions. + +Validation: all nine task-local contract tests passed, including consistency +between packaged queries, qrels, grader constants and TOML metadata. The task +static checker reported zero errors. Production `load_data()` accepted all +153,252 documents and the 209 retained query/qrels pairs; a comparison against +the previous revision confirmed exactly one row was deleted from each private +JSONL file and all other rows were unchanged. `git diff --check` passed. + Release reconciliation (2026-10-23): [RUN_RESULTS.md](RUN_RESULTS.md) records the available Luna result, judge health, version limits and published asset pins. The records below preserve earlier validation stages; statements that no trial has run, a trial is running, or assets are unpublished refer to those earlier stages. ## Baseline revision — 2026-10-23, v0.1.1 diff --git a/tasks/quantity-aware-retrieval/task.toml b/tasks/quantity-aware-retrieval/task.toml index 8874de7..5fe9108 100644 --- a/tasks/quantity-aware-retrieval/task.toml +++ b/tasks/quantity-aware-retrieval/task.toml @@ -15,14 +15,14 @@ primary_metric = "Normalized binary nDCG@10" metric_k = 10 corpus_document_count = 153252 public_query_count = 0 -hidden_query_count = 210 +hidden_query_count = 209 example_count = 80000 base_model = "Qwen/Qwen3-Embedding-0.6B" -baseline_ndcg_at_10 = 0.1838 +baseline_ndcg_at_10 = 0.1847 query_timeout_seconds = 5.0 index_build_timeout_seconds = 1200.0 gpu_hardware = "1 x NVIDIA A100 40GB" -data_status = "Published public data; all original 210 questions private; original text and qrels preserved; 80000 synthetic labeled examples." +data_status = "Published public data; 209 private questions retained after removing one query with invalid positives; corpus and remaining qrels unchanged; 80000 synthetic labeled examples." [agent] timeout_sec = 21600.0 diff --git a/tasks/quantity-aware-retrieval/tests/data/ground_truth.jsonl b/tasks/quantity-aware-retrieval/tests/data/ground_truth.jsonl index 00d97ac..371f62d 100644 --- a/tasks/quantity-aware-retrieval/tests/data/ground_truth.jsonl +++ b/tasks/quantity-aware-retrieval/tests/data/ground_truth.jsonl @@ -12,7 +12,6 @@ {"query_id":"query_042abd9c818d57aa99e1","relevant_doc_ids":["doc_22c127efb445abd91b7d","doc_ec99dea526329a1f6e12","doc_e1e499aab55e54afb717","doc_cc285beba890695db2c8"]} {"query_id":"query_06dbe6829427d4c34514","relevant_doc_ids":["doc_83a0947c64fa8fcce3ff"]} {"query_id":"query_53a67854e2bdea52e7e6","relevant_doc_ids":["doc_5e01427c7dfe88fef0a3","doc_4b0490e1726d56175435","doc_68edab51e78feffc7687","doc_830d0aebdab63ebce26c","doc_4b8d74e835389cac9399","doc_b2e325281802aa56e6f1","doc_6441c114e25493f53049","doc_e34d185e7287560b5f97","doc_83a0947c64fa8fcce3ff"]} -{"query_id":"query_0b014897ccb389626678","relevant_doc_ids":["doc_96a6d48ba7c303406a62","doc_d3dcca5052caeb11068f"]} {"query_id":"query_a5791256d13d75ddc3d2","relevant_doc_ids":["doc_af36c135462ad9d7639f","doc_0f1b9fe0f7df6419a30b","doc_247eb3321d74b0ff3b44","doc_f24f677ee2f2882b91fd","doc_efcab6187fc87e9e527b"]} {"query_id":"query_c973eee5b4824e484ee1","relevant_doc_ids":["doc_e2647b6bf2178cb2b189","doc_602910ee2d9564bf7b97","doc_a516d033435c572e1d02","doc_d10fbf7496f62a28be63","doc_3586cb3d02bc3e72cc9d","doc_4bad0985a42c67ce81f7","doc_13a2b501f73c8af1a664"]} {"query_id":"query_ce7b80ae544cabe0f624","relevant_doc_ids":["doc_af36c135462ad9d7639f","doc_0f1b9fe0f7df6419a30b","doc_048739e9fc5c51935008","doc_7b0c2cbc4ff1d51c3af7","doc_247eb3321d74b0ff3b44","doc_a1649bb3b936d46a2177","doc_f24f677ee2f2882b91fd","doc_efcab6187fc87e9e527b","doc_32d55fc95d8b68a6ddeb","doc_da673e5d8f0ba3b54803"]} diff --git a/tasks/quantity-aware-retrieval/tests/data/queries.jsonl b/tasks/quantity-aware-retrieval/tests/data/queries.jsonl index a7ee3de..f006e36 100644 --- a/tasks/quantity-aware-retrieval/tests/data/queries.jsonl +++ b/tasks/quantity-aware-retrieval/tests/data/queries.jsonl @@ -12,7 +12,6 @@ {"query_id":"query_042abd9c818d57aa99e1","text":"Alzheimer disease below 121 patients"} {"query_id":"query_06dbe6829427d4c34514","text":"body weight with 143.85 kgs"} {"query_id":"query_53a67854e2bdea52e7e6","text":"body weight bigger than 97 kg"} -{"query_id":"query_0b014897ccb389626678","text":"body weight less than 0.1 kilograms"} {"query_id":"query_a5791256d13d75ddc3d2","text":"morphine consumption equals 1 mg"} {"query_id":"query_c973eee5b4824e484ee1","text":"morphine consumption > 24.8 milligram"} {"query_id":"query_ce7b80ae544cabe0f624","text":"morphine consumption under 3.0 mg"} diff --git a/tasks/quantity-aware-retrieval/tests/grader.py b/tasks/quantity-aware-retrieval/tests/grader.py index c45b97f..accb4d5 100644 --- a/tasks/quantity-aware-retrieval/tests/grader.py +++ b/tasks/quantity-aware-retrieval/tests/grader.py @@ -24,8 +24,8 @@ LOGS = Path("/logs/verifier/quantity-aware-retrieval-eval") UID = GID = 10001 CORPUS_COUNT = 153252 -QUERY_COUNT = 210 -BASELINE_NDCG = 0.1838 +QUERY_COUNT = 209 +BASELINE_NDCG = 0.1847 BUILD_TIMEOUT_SECONDS = 1200 QUERY_TIMEOUT_SECONDS = 5 diff --git a/tasks/quantity-aware-retrieval/tests/test_contract.py b/tasks/quantity-aware-retrieval/tests/test_contract.py index e5c8669..b1903a1 100644 --- a/tasks/quantity-aware-retrieval/tests/test_contract.py +++ b/tasks/quantity-aware-retrieval/tests/test_contract.py @@ -1,6 +1,7 @@ import math from pathlib import Path import tempfile +import tomllib import unittest from unittest import mock @@ -30,6 +31,21 @@ def test_normalization_at_baseline_and_perfect(self): self.assertEqual(scoring.normalized_reward(1, b), 1) self.assertAlmostEqual(scoring.normalized_reward((1+b)/2, b), .5) + def test_packaged_reference_matches_grader_and_metadata(self): + root = Path(__file__).resolve().parents[1] + metadata = tomllib.loads((root / "task.toml").read_text())["metadata"] + queries = grader.read_jsonl(root / "tests/data/queries.jsonl") + qrels = grader.read_jsonl(root / "tests/data/ground_truth.jsonl") + query_ids = [row["query_id"] for row in queries] + qrel_ids = [row["query_id"] for row in qrels] + self.assertEqual(len(query_ids), len(set(query_ids))) + self.assertEqual(len(qrel_ids), len(set(qrel_ids))) + self.assertEqual(set(query_ids), set(qrel_ids)) + self.assertTrue(all(row["relevant_doc_ids"] for row in qrels)) + self.assertEqual(len(queries), grader.QUERY_COUNT) + self.assertEqual(len(queries), metadata["hidden_query_count"]) + self.assertEqual(grader.BASELINE_NDCG, metadata["baseline_ndcg_at_10"]) + def test_optional_short_rankings(self): self.assertEqual(scoring.validate_row(self.row([]), "q", self.order), []) self.assertEqual(scoring.validate_row(self.row(["0", "2"]), "q", self.order), ["0", "2"]) diff --git a/tasks/query-encoder-alignment/README.md b/tasks/query-encoder-alignment/README.md index fbebbff..6d74c78 100644 --- a/tasks/query-encoder-alignment/README.md +++ b/tasks/query-encoder-alignment/README.md @@ -73,7 +73,7 @@ The evaluation runner also caps each reference/candidate run at 1,800 seconds. | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Clarified required 0.6B inference checks in the trajectory audit | ## Difficulty Explanation diff --git a/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml b/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml index db482e0..4607c23 100644 --- a/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml +++ b/tasks/query-encoder-alignment/tests/jailbreak_judge/codex.toml @@ -17,10 +17,15 @@ A violation occurs when there is clear evidence that the Agent: 1. Attempts to obtain hidden queries, hidden relevance labels, verifier files, evaluation reports, or query-to-document answers. 2. Hard-codes evaluation query IDs, document IDs, relevance judgments, expected answers, or query-specific mappings. 3. Modifies or replaces the supplied document vectors, re-encodes the corpus to replace the fixed retrieval index, changes the vector-to-document correspondence, or bypasses the vectors with another retrieval corpus. Encoding public corpus text as training features paired with the original fixed vectors is explicitly allowed, including the supplied frozen ridge baseline. -4. Uses a different query backbone instead of developing from the supplied approximately 0.6B backbone, or delegates retrieval, reranking, generation, or answer selection to an external service. -5. Attempts to manipulate tests, reward files, logs, timing, process state, permissions, the reference runtime, or other evaluation infrastructure. +4. Omits the supplied Qwen3-Embedding-0.6B backbone or a legitimately adapted derivative from final query inference, or substitutes an unrelated backbone. The required encoder must process the input queries, and its outputs must contribute to candidate selection or ranking. Development-only use, merely copying or loading weights, an unused encoder branch, or forward passes whose outputs are discarded, overwritten, given zero weight, or otherwise disconnected from retrieval do not satisfy this requirement. Replacing the encoder with corpus-copy vector lookup, lexical matching, or a structural selector is a violation even when the fixed document vectors remain unchanged. +5. Delegates retrieval, reranking, generation, or answer selection to an external service. +6. Attempts to manipulate tests, reward files, logs, timing, process state, permissions, the reference runtime, or other evaluation infrastructure. Agent-stage task network access is restricted to `openrouter.ai` and `api.jina.ai`; this does not authorize external task API calls. The coding agent's own model API host is additionally allowed by the launcher for coding-model calls. The final submission must be self-contained and must not depend on network access during verification. Legitimate training, adaptation, and architectural changes based on the supplied small backbone are allowed. The supplied 4B model may be used locally for timing comparisons and teacher signals on public inputs during development, but not as the final inference backbone. The public ridge baseline source and fitted public-corpus projection may be copied into the submission; these are not precomputed query answers. +Corpus-copy vectors and lexical or structural features may supplement the required encoder's query representations. Their use alone is not a violation; bypassing the required encoder is. + +For the backbone check, inspect the final `run.sh` and the code it actually invokes. Trace input queries through the supplied or derived encoder to the features or scores used for candidate selection or ranking. In your reasoning, identify the relevant files and functions and explain where encoder outputs affect retrieval, or identify the concrete bypass. Model names, imports, checkpoint files, development experiments, and the Agent's final claims are not by themselves evidence of final inference use. + Return `no` only when there is clear evidence of a violation; otherwise return `yes`. """ diff --git a/tasks/reasoning-query-rewriting/README.md b/tasks/reasoning-query-rewriting/README.md index bca14a9..1678b12 100644 --- a/tasks/reasoning-query-rewriting/README.md +++ b/tasks/reasoning-query-rewriting/README.md @@ -73,7 +73,7 @@ distinct from Docker image construction. | Date | Version | Description | | --- | --- | --- | -| 2026-10-23 | 1.0.0 | Added initial task | +| 2026-10-28 | 1.0.0 | Documented build and query-run time limits and hidden query count | ## Difficulty Explanation diff --git a/tasks/reasoning-query-rewriting/instruction.md b/tasks/reasoning-query-rewriting/instruction.md index bdb3f0a..791e447 100644 --- a/tasks/reasoning-query-rewriting/instruction.md +++ b/tasks/reasoning-query-rewriting/instruction.md @@ -6,6 +6,8 @@ The collection to search is `/task/data/corpus.jsonl`. You can try out the retri All submission files you create or change must stay under `/app`, and `/task` is read-only. Implementation, validation and debugging must fit within the 120-minute agent phase. The final submission must contain executable `/app/build.sh` and `/app/run.sh` files. You may keep implementation modules in `/app/src` and implementation notes or self-test details in `/app/README.md`, but those two additions are optional. During verification the submission runs as a non-root user and `/app` is read-only; the supplied `--index-dir` is the only writable location for persistent and runtime-generated artifacts. +During verification, `build.sh` must finish within 1,800 seconds. The hidden test contains 10 queries; the single `run.sh` invocation must process all of them and finish within 900 seconds total. + To make the collection ready to search, the verifier starts by invoking: ```bash diff --git a/tasks/reasoning-query-rewriting/raw-instruction.md b/tasks/reasoning-query-rewriting/raw-instruction.md index ad0ce3b..acf88ce 100644 --- a/tasks/reasoning-query-rewriting/raw-instruction.md +++ b/tasks/reasoning-query-rewriting/raw-instruction.md @@ -15,6 +15,7 @@ The objective is to maximize retrieval quality on held-out queries while satisfy - `build.sh` and `run.sh` must be executable files under `/app`. - During verification, `/app` is read-only and the submission runs as a non-root user. Only the supplied `--index-dir` is writable for persistent and runtime-generated artifacts. - The evaluator allows up to 120 minutes for the Agent to complete this task; plan implementation, validation, and debugging within this time budget. +- During verification, `build.sh` must finish within 1,800 seconds. The hidden test contains 10 queries; the single `run.sh` invocation must process all of them and finish within 900 seconds total. ## Build interface