From ded1135b1d025e17cdc7ffd9ecc37352019b598c Mon Sep 17 00:00:00 2001 From: Haoran Jin <57559447+hr-jin@users.noreply.github.com> Date: Fri, 23 Oct 2026 17:33:23 +0800 Subject: [PATCH] Align raw task instructions with reference format Standardize sections and examples for thirteen tasks, clarify existing failure handling, and remove duplicated or undefined descriptions. Runtime instructions, interfaces and scoring rules are unchanged. --- .../composed-image-search/raw-instruction.md | 97 +++++++++++-- tasks/document-page-search/raw-instruction.md | 99 +++++++++++-- .../raw-instruction.md | 132 +++++++++++++++--- tasks/financial-report-qa/raw-instruction.md | 90 ++++++++++-- tasks/icsi-meeting-memory/raw-instruction.md | 110 ++++++++++++--- .../raw-instruction.md | 100 +++++++++++-- tasks/layout-page-search/raw-instruction.md | 92 ++++++++++-- .../narrative-clue-search/raw-instruction.md | 89 ++++++++++-- .../raw-instruction.md | 96 +++++++++++-- tasks/sketch-text-search/raw-instruction.md | 102 ++++++++++++-- tasks/spoken-query-search/raw-instruction.md | 92 ++++++++++-- tasks/video-moment-search/raw-instruction.md | 95 +++++++++++-- tasks/visual-document-qa/raw-instruction.md | 86 ++++++++++-- 13 files changed, 1109 insertions(+), 171 deletions(-) diff --git a/tasks/composed-image-search/raw-instruction.md b/tasks/composed-image-search/raw-instruction.md index 53c3d62..9df8678 100644 --- a/tasks/composed-image-search/raw-instruction.md +++ b/tasks/composed-image-search/raw-instruction.md @@ -1,25 +1,102 @@ -# Task: Composed Image Search +# Composed Image Search ## Task Description A user starts with a photograph and describes a change they want, such as a different arrangement or a different quantity of an object. Build an image search system that finds matching photographs in the supplied gallery while retaining the parts of the reference that the request leaves unchanged. -## Inputs and Available Validation Data +## Requirements + +### Resource limits and workload + +- You have two hours, 8 CPU cores, 32 GiB RAM and 100 GiB working storage and one 40 GiB GPU. +- The build command has 3,600 seconds. +- The system must handle up to 4,400 requests in two calls of at most 2,200 requests, with an independent 14,400-second allowance for each call. +- At most 100 candidate images per request may undergo additional query-dependent visual inspection. +- Corpus-wide global-vector screening and gallery-only preprocessing are allowed. +- The local model and permitted APIs are available as described in `/task/docs/available_resources.md`; also read `/task/docs/environment.md`. + +- Treat data, models and documentation as read-only. +- The submitted `/app` files must work in a fresh environment with the supplied inputs, using the passed index and output paths for generated files. +- Query calls may be repeated after a build. +- Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful. +- A command failure other than timeout invalidates that call. +- Aim to return the requested images near the start of each ranking across the supplied requests. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface + +Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is: + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface + +The query interface is: + +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` + +Honor the supplied paths. + +### Input and output contract + +Each input JSONL row contains `query_id`, `reference_image_id`, `modification` and `shared_concept`. The reference belongs to the gallery. `shared_concept` provides an additional description when available and can be empty; the reference and modification still define the request. + +Return one JSONL row for each request with its unchanged `query_id` and a `results` list of at most 50 distinct gallery images. Each result contains `image_id` and a finite numeric `score`, ordered by nonincreasing score. Exclude the reference image. Write diagnostics to stderr. + +## Available Validation Data The gallery is at `/task/data/corpus/`. Each row of `records.jsonl` contains `image_id` and `image_path`; paths are relative to the corpus directory. Gallery images are supplied. Local image and text encoder weights are available at `/task/models/clip/`. -## Expected Artifacts and Interfaces +Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels map each `query_id` to `qrels`, which may contain several matching images. You may use these examples to develop and check the system. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality uses average precision and recall to measure the ranking and coverage of annotated matching images. The applicable cutoffs, query groups and equal-weight aggregation are defined in [README.md — Search Quality](README.md#search-quality). -Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is `/app/build.sh --corpus /task/data/corpus --index-dir /work/index`. The query interface is `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl`. Honor the supplied paths. Each input JSONL row contains `query_id`, `reference_image_id`, `modification` and `shared_concept`. The reference belongs to the gallery. `shared_concept` provides an additional description when available and can be empty; the reference and modification still define the request. +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. -Return one JSONL row for each request with its unchanged `query_id` and a `results` list of at most 50 distinct gallery images. Each result contains `image_id` and a finite numeric `score`, ordered by nonincreasing score. Exclude the reference image. Write diagnostics to stderr. Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels map each `query_id` to `qrels`, which may contain several matching images. You may use these examples to develop and check the system. +## Hidden Test Overview -## Resources and Workload +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. -You have two hours, 8 CPU cores, 32 GiB RAM and 100 GiB working storage and one 40 GiB GPU. The build command has 3,600 seconds. The system must handle up to 4,400 requests in two calls of at most 2,200 requests, with an independent 14,400-second allowance for each call. At most 100 candidate images per request may undergo additional query-dependent visual inspection. Corpus-wide global-vector screening and gallery-only preprocessing are allowed. The local model and permitted APIs are available as described in `/task/docs/available_resources.md`; also read `/task/docs/environment.md`. +## Environment and available resources -## Execution and File Permissions +Before implementing the system, read the following task-provided documents: -During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -Treat data, models and documentation as read-only. The submitted `/app` files must work in a fresh environment with the supplied inputs, using the passed index and output paths for generated files. Query calls may be repeated after a build. Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful. A command failure other than timeout invalidates that call. Aim to return the requested images near the start of each ranking across the supplied requests. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/document-page-search/raw-instruction.md b/tasks/document-page-search/raw-instruction.md index 0018ed7..d652f5f 100644 --- a/tasks/document-page-search/raw-instruction.md +++ b/tasks/document-page-search/raw-instruction.md @@ -1,28 +1,101 @@ -# Task: Multilingual Visual Document Page Search +# Multilingual Visual Document Page Search ## Task Description -Reports in several languages contain tables, charts and dense page layouts. I need to find the pages that support a question so that I can open the original material and check it. Build a page search system over the supplied collection. +Reports in several languages contain tables, charts and dense page layouts. The system must find the pages that support a question so that the original material can be opened and checked. Build a page search system over the supplied collection. -## Document Collection +## Requirements -The collection is in `/task/data/corpus`. Each line of `records.jsonl` describes a page with `page_id`, `domain`, `doc_id`, `image_path`, `width`, `height`, `markdown` and `page_number_in_doc`. Image paths are relative to the collection directory. The Markdown is a supplied transcription; the original page images are also available. Questions may be in a different language from the document. Treat `/task` as read-only and place your implementation under `/app`. +### Resource limits and workload + +- You have 8 CPU cores, 32 GiB RAM, 100 GiB storage, one 40 GiB GPU and two hours to implement and check the system. +- Local model weights and processors are at `/task/models/page-encoder/` and `/task/models/vision-backbone/`. +- Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed tools and permitted API resources. +- Use only those providers, endpoints and generative model IDs. +- Do not obtain evaluation questions, answers or relevance labels from outside the provided inputs, use external search services, or store API credentials in code, indexes or logs. + +- The system must handle workloads of up to 14,500 requests in 2 calls of at most 7,250 requests. +- Each call has an independent 2,700-second allowance; time used by one call does not reduce another. +- Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. +- A failed query command other than a timeout invalidates that call. -## Expected Artifacts and Interfaces +### Execution and file permissions + +- The complete implementation under `/app` is transferred to a fresh environment. +- Your build command has 10,800 seconds. +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +- Include any additional dependencies needed at runtime under `/app`; development processes and packages installed elsewhere are not transferred. +- Aim to rank the relevant pages as highly as possible on new questions across all supplied domains and languages. + +### Build interface Provide executable `/app/build.sh` and `/app/run.sh`. The build command prepares the index and returns when it is ready: -`/app/build.sh --corpus /task/data/corpus --index-dir /work/index` + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface + The query command accepts a JSONL file and writes a JSONL result file: -`/app/run.sh --index-dir /work/index --queries /path/to/questions.jsonl --output /path/to/results.jsonl` -Honor the passed paths. Each question has `query_id`, `question`, `domain` and `language`. Search the complete page collection for that domain. For every input question, return one object with its unchanged `query_id` and `results`, containing up to ten distinct pages in relevance order. Each result contains `page_id` and a finite numeric `score`; scores must be nonincreasing. Return page IDs from the requested domain. No answer or bounding box is required. + +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /path/to/questions.jsonl \ + --output /path/to/results.jsonl +``` + +Honor the passed paths. + +### Input and output contract + +Each question has `query_id`, `question`, `domain` and `language`. Search the complete page collection for that domain. For every input question, return one object with its unchanged `query_id` and `results`, containing up to ten distinct pages in relevance order. Each result contains `page_id` and a finite numeric `score`; scores must be nonincreasing. Return page IDs from the requested domain. No answer or bounding box is required. + +## Available Validation Data + +The collection is in `/task/data/corpus`. Each line of `records.jsonl` describes a page with `page_id`, `domain`, `doc_id`, `image_path`, `width`, `height`, `markdown` and `page_number_in_doc`. Image paths are relative to the collection directory. The Markdown is a supplied transcription; the original page images are also available. Questions may be in a different language from the document. Treat `/task` as read-only and place your implementation under `/app`. + Public questions and their page-relevance annotations are in `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use these for development and self-checking. Additional questions and annotations are withheld. -## Environment and Available Resources +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is nDCG@10 with linear relevance gains, averaged within each domain and then equally across the eight domains. Missing or invalid query results remain in the denominator. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. -You have 8 CPU cores, 32 GiB RAM, 100 GiB storage, one 40 GiB GPU and two hours to implement and check the system. Local model weights and processors are at `/task/models/page-encoder/` and `/task/models/vision-backbone/`. Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed tools and permitted API resources. Use only those providers, endpoints and generative model IDs. Do not obtain evaluation questions, answers or relevance labels from outside the provided inputs, use external search services, or store API credentials in code, indexes or logs. +## Environment and available resources -## Execution, Permissions and Workload +Before implementing the system, read the following task-provided documents: -The complete implementation under `/app` is transferred to a fresh environment. Your build command has 10,800 seconds. During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. Include any additional dependencies needed at runtime under `/app`; development processes and packages installed elsewhere are not transferred. Aim to rank the relevant pages as highly as possible on new questions across all supplied domains and languages. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -The system must handle workloads of up to 14,500 requests in 2 calls of at most 7,250 requests. Each call has an independent 2,700-second allowance; time used by one call does not reduce another. Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. A failed query command other than a timeout invalidates that call. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/evolving-personal-memory/raw-instruction.md b/tasks/evolving-personal-memory/raw-instruction.md index 7508e40..f47d5e1 100644 --- a/tasks/evolving-personal-memory/raw-instruction.md +++ b/tasks/evolving-personal-memory/raw-instruction.md @@ -1,39 +1,135 @@ -# Task: Personal-History Search and Answering +# Personal-History Search and Answering ## Task Description Build a personal-memory search system that answers questions using a person's conversation history up to a specified session. It will be used repeatedly as the person's circumstances change, so answers need to reflect the history available at the requested cutoff. -## Inputs and Available Validation Data +## Requirements -The complete conversation collection is at `/task/data/corpus/records.jsonl`. Each record contains `record_id`, `persona_id`, `session_id`, `date` and `text`. Public questions and their answer criteria are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use these examples to check your system. +### Resource limits and workload -## Expected Artifacts +- The environment provides 8 CPU cores, 32 GiB RAM, 100 GiB storage and no GPU. +- You have two hours for development. +- Index construction has a 30-minute limit. +- A search call may contain up to 300 questions and has a 30-minute limit. +- Each answer process has up to ten minutes after launch, with up to eight answer processes running concurrently and a shared two-hour answer-phase budget. +- Write each output to the requested path. +- The system must support repeated calls using the same index. -Submit executable `/app/build.sh`, `/app/search.sh` and `/app/answer.sh`, together with any supporting files under `/app`. Put supporting answer-side code and configuration under `/app/answer/`. Build and search may use the corpus and index; answer must work from the current request alone. Answer-side files must contain reusable code and configuration, not stored conversation facts or answers. - -## Build and Search Contract +### Build interface Build the index once with: -`/app/build.sh --corpus /task/data/corpus --index-dir /path/to/index` -The index directory persists across search calls. During verification, build and search run as a non-root user with `/app`, the corpus and query inputs read-only. The passed index directory already exists and is writable, but its parent is not writable. Put generated files, search caches and build temporary directories inside `--index-dir` or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The index remains writable across search calls. Write search results to the passed output path, whose parent is writable, and honor all supplied paths. Each search call receives a JSONL file of questions and writes one JSONL result per question: -`/app/search.sh --index-dir /path/to/index --queries /path/to/queries.jsonl --output /path/to/retrievals.jsonl` + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /path/to/index +``` + +The index directory persists across search calls. During verification, build and search run as a non-root user with `/app`, the corpus and query inputs read-only. The passed index directory already exists and is writable, but its parent is not writable. Put generated files, search caches and build temporary directories inside `--index-dir` or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The index remains writable across search calls. Write search results to the passed output path, whose parent is writable, and honor all supplied paths. + +### Search interface and output contract + +Each search call receives a JSONL file of questions and writes one JSONL result per question: + +```bash +/app/search.sh \ + --index-dir /path/to/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/retrievals.jsonl +``` + For example, an input row and its corresponding output row are: -`{"query_id":"QUERY_PLACEHOLDER","persona_id":"PERSONA_PLACEHOLDER","cutoff_session":10,"question":"QUESTION_PLACEHOLDER"}` `{"query_id":"QUERY_PLACEHOLDER","record_ids":["RECORD_PLACEHOLDER"]}` + +```json +{"query_id":"QUERY_PLACEHOLDER","persona_id":"PERSONA_PLACEHOLDER","cutoff_session":10,"question":"QUESTION_PLACEHOLDER"} +``` + +```json +{"query_id":"QUERY_PLACEHOLDER","record_ids":["RECORD_PLACEHOLDER"]} +``` + Return only `query_id` and `record_ids`. Return at most 350 unique record IDs for the specified person, with session numbers no greater than `cutoff_session`. An empty list is allowed. -## Answer Contract and Isolation +### Answer interface and isolation The returned IDs are checked and used to assemble an answer request containing the complete original records. For example: -`{"query_id":"QUERY_PLACEHOLDER","persona_id":"PERSONA_PLACEHOLDER","cutoff_session":10,"question":"QUESTION_PLACEHOLDER","records":[{"record_id":"RECORD_PLACEHOLDER","persona_id":"PERSONA_PLACEHOLDER","session_id":3,"date":"DATE_PLACEHOLDER","text":"COMPLETE_RECORD_TEXT_PLACEHOLDER"}]}` + +```json +{ + "query_id": "QUERY_PLACEHOLDER", + "persona_id": "PERSONA_PLACEHOLDER", + "cutoff_session": 10, + "question": "QUESTION_PLACEHOLDER", + "records": [ + { + "record_id": "RECORD_PLACEHOLDER", + "persona_id": "PERSONA_PLACEHOLDER", + "session_id": 3, + "date": "DATE_PLACEHOLDER", + "text": "COMPLETE_RECORD_TEXT_PLACEHOLDER" + } + ] +} +``` + Answer this request with: -`/app/answer.sh --input /path/to/request.json --output /path/to/answer.json` + +```bash +/app/answer.sh \ + --input /path/to/request.json \ + --output /path/to/answer.json +``` + The output is a JSON object containing only the same `query_id` and a nonempty natural-language `answer`: -`{"query_id":"QUERY_PLACEHOLDER","answer":"ANSWER_PLACEHOLDER"}` + +```json +{ + "query_id": "QUERY_PLACEHOLDER", + "answer": "ANSWER_PLACEHOLDER" +} +``` + Each answer runs as a separate non-root process with its own writable output and temporary directories. Use the supplied `TMPDIR` and `XDG_CACHE_HOME` for temporary files and caches. The answer process can read its read-only current request, `/app/answer.sh`, supporting files under `/app/answer/`, installed system libraries and its temporary directory. It cannot read the complete corpus, index, search outputs or other requests. Do not use links in the answer directory to access files outside it. The process receives the permitted generative API credential; it does not receive retrieval API credentials. You may process the supplied records in batches and make multiple permitted LLM calls. Record text is supplied in full and may exceed one model call's context capacity. -## Resources and Workload +## Available Validation Data + +The complete conversation collection is at `/task/data/corpus/records.jsonl`. Each record contains `record_id`, `persona_id`, `session_id`, `date` and `text`. Public questions and their answer criteria are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use these examples to check your system. + +## Expected Artifacts + +Submit executable `/app/build.sh`, `/app/search.sh` and `/app/answer.sh`, together with any supporting files under `/app`. Put supporting answer-side code and configuration under `/app/answer/`. Build and search may use the corpus and index; answer must work from the current request alone. Answer-side files must contain reusable code and configuration, not stored conversation facts or answers. + +```text +/app/ +├── build.sh +├── search.sh +├── answer.sh +└── answer/ # supporting answer-side code and configuration +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. Missing required executables or a failed or timed-out build gives zero for the entire submission. A search timeout preserves valid completed rows; any other search-command failure invalidates that call's results. Missing, duplicate or invalid retrieval results, and missing, invalid, failed or timed-out answers, score zero for the affected questions. Valid results for other questions remain usable, and all held-out questions remain in the average. +4. **Search quality.** Three independent judge configurations vote on each memory or forgetting criterion. MPA and FAA are the respective pass rates, and lambda is the fraction of forgetting criteria. Raw quality is the mean of FAMA = max(0, MPA - lambda * (1 - FAA)) across held-out questions. Retrieved IDs select the answer inputs; they are not scored as a separate retrieval metric. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -The environment provides 8 CPU cores, 32 GiB RAM, 100 GiB storage and no GPU. You have two hours for development. Index construction has a 30-minute limit. A search call may contain up to 300 questions and has a 30-minute limit. Each answer process has up to ten minutes after launch, with up to eight answer processes running concurrently and a shared two-hour answer-phase budget. Write each output to the requested path. The system must support repeated calls using the same index. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. -Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed environment and permitted APIs. These documents and the supplied data are read-only. Answers are assessed against the supplied style of answer criteria; prioritize correct, useful answers grounded in the available history. +Answers are assessed against the supplied style of answer criteria; prioritize correct, useful answers grounded in the available history. diff --git a/tasks/financial-report-qa/raw-instruction.md b/tasks/financial-report-qa/raw-instruction.md index c41755f..cce9c01 100644 --- a/tasks/financial-report-qa/raw-instruction.md +++ b/tasks/financial-report-qa/raw-instruction.md @@ -1,27 +1,95 @@ -# Task: Financial-Report Evidence Retrieval +# Financial-Report Evidence Retrieval ## Task Description Analysts need to check the figures behind financial questions across a shared collection of annual reports. They know what they want to investigate, but may not know which report contains the information. Build a reusable search system that finds the relevant reports and returns the original passages supporting each question, so the figures can be checked in context. -## Inputs and Available Validation Data +## Requirements -The complete collection is in `/task/data/corpus/reports.jsonl`. Each row contains a `document_id` and the report's full `text`; all 885 reports are available. Example questions and their annotated evidence locations are in `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use the supplied collection and examples to develop and check the system. Treat `/task` as read-only and put the implementation under `/app`. +### Resource limits and workload + +- Development and checking have a two-hour budget, with eight CPU cores, 32 GiB RAM, 100 GiB working storage and no GPU. +- The system must handle workloads of up to 8,000 requests in two calls of at most 4,000 requests. +- Each call has its own 3,600-second allowance. +- Completed valid JSONL rows remain usable if a call times out; missing or invalid query results are unsuccessful. +- A command failure other than a timeout invalidates that call. +- The index must remain usable across repeated calls after one build. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface + +Provide executable `/app/build.sh` and `/app/run.sh`, together with any supporting files under `/app`. Build the shared collection once with: -## Expected Artifacts and Interfaces +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /path/to/index +``` -Provide executable `/app/build.sh` and `/app/run.sh`, together with any supporting files under `/app`. Build the shared collection once with `/app/build.sh --corpus /task/data/corpus --index-dir /path/to/index`. The command must finish successfully within 1,800 seconds and prepare everything needed for subsequent queries after a fresh start. Then support repeated calls to `/app/run.sh --index-dir DIR --queries JSONL --output JSONL`. Honor the supplied paths. +The command must finish successfully within 1,800 seconds and prepare everything needed for subsequent queries after a fresh start. -## Execution and File Permissions +### Query interface -During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +Then support repeated calls to: -## Query and Evidence Contract +```bash +/app/run.sh \ + --index-dir DIR \ + --queries JSONL \ + --output JSONL +``` + +Honor the supplied paths. + +### Input and output contract Each input row contains `query_id` and `question`. A target report identifier is not supplied. For each query, write one JSONL row containing exactly `query_id` and `spans`, with the same query identifier. The `spans` array may contain up to five evidence locations, ordered from most to least relevant. Each location contains exactly `document_id`, `start` and `end`. Use a report identifier from the supplied collection and integer Unicode-character offsets into that report's original `text`: `start` is inclusive, `end` is exclusive, and `0 <= start < end <= len(text)`. Each span may contain at most 512 Unicode characters. Find the passages that cover the supporting facts, preserving their original report and positions. An empty list is permitted when the system finds no evidence. -## Workload and Available Resources +## Available Validation Data + +The complete collection is in `/task/data/corpus/reports.jsonl`. Each row contains a `document_id` and the report's full `text`; all 885 reports are available. Example questions and their annotated evidence locations are in `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use the supplied collection and examples to develop and check the system. Treat `/task` as read-only and put the implementation under `/app`. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is EvidenceRecall@5, averaged equally across held-out questions. A reference fact is covered when returned spans from the correct report jointly cover at least one complete accepted location. Repeated hits on one fact do not add credit. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -The system must handle workloads of up to 8,000 requests in two calls of at most 4,000 requests. Each call has its own 3,600-second allowance. Completed valid JSONL rows remain usable if a call times out; missing or invalid query results are unsuccessful. A command failure other than a timeout invalidates that call. The index must remain usable across repeated calls after one build. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. -Read `/task/docs/environment.md` and `/task/docs/available_resources.md` before starting. Development and checking have a two-hour budget, with eight CPU cores, 32 GiB RAM, 100 GiB working storage and no GPU. The documented OpenRouter and Jina APIs are available under their provider, model and endpoint restrictions. Use the supplied reports and permitted resources, and leave all files required by the build and query commands under `/app`. +The documented OpenRouter and Jina APIs are available under their provider, model and endpoint restrictions. Use the supplied reports and permitted resources, and leave all files required by the build and query commands under `/app`. diff --git a/tasks/icsi-meeting-memory/raw-instruction.md b/tasks/icsi-meeting-memory/raw-instruction.md index eed7eec..492d350 100644 --- a/tasks/icsi-meeting-memory/raw-instruction.md +++ b/tasks/icsi-meeting-memory/raw-instruction.md @@ -1,47 +1,115 @@ -# Task: Compact Conversational Memory Question Answering +# Compact Conversational Memory Question Answering ## Task Description -I need to ask questions about past meetings without keeping the full transcripts in the answering system. Please prepare a compact memory and an executable search-and-answer pipeline: build an index from that memory, retrieve the relevant notes for a question, and use those notes to produce a concise answer. The goal is to maximize evidence-grounded answer correctness on held-out questions. You may choose the memory structure, index and retrieval method within the interfaces and limits below. +Build a system for asking questions about past meetings without keeping the full transcripts in the answering system. Prepare a compact memory and an executable search-and-answer pipeline: build an index from that memory, retrieve the relevant notes for a question, and use those notes to produce a concise answer. The goal is to maximize evidence-grounded answer correctness on held-out questions. You may choose the memory structure, index and retrieval method within the interfaces and limits below. -## Inputs and Available Validation Data +## Requirements -The complete history is available at `/task/data/history.jsonl`. Its records contain `meeting_id`, `series_id`, `date`, `speaker`, `start_seconds`, `end_seconds`, `text` and `source_ids`; a null speaker is unidentified. To check the system while developing it, use the five public questions at `/task/data/validation/queries.jsonl`, the reference answers and required facts at `/task/data/validation/golden_answers.jsonl`, and their supporting transcript excerpts at `/task/data/validation/evidence.jsonl`. The reference files are keyed by `query_id`, but the program receives only each row's `question` as a single command-line argument, without an ID or meeting metadata. The remaining public-query fields are for analysis. The final questions concern the same 67 meetings, so prepare the memory from the supplied history rather than from a list of question-to-answer mappings. +### Resource limits + +- You have 120 minutes for implementation and validation, with eight CPU cores, 8 GiB RAM, 8 GiB storage and no GPU. +- Development runs as root, and no local model weights are supplied under `/opt/models`. +- Treat `/task` and its documentation as read-only. +- Do not obtain held-out labels, change the evaluator, use external answer services or embed answers to particular evaluation questions. + +### Memory file + +The memory must be readable UTF-8 JSON, with no duplicate object keys or non-finite numbers, and no larger than 183,981 bytes. This is five percent of the history's 3,679,621 UTF-8 dialogue-text bytes, including one newline per utterance and excluding the source JSON metadata. The limit counts the actual bytes of `memory.json`, including its formatting and metadata. Choose its JSON structure freely, but store directly readable memory text rather than compressed or encoded dialogue. Script bytes and the index generated from this memory are outside that memory-file budget. + +### Execution and file permissions + +- Honor every supplied path, which may vary between invocations, and return exit status `0` from all three scripts. +- Submitted commands run as a non-root user in a fresh working directory for each invocation. +- Submitted files, stage inputs and installed runtime files are read-only; only that stage's working directory is writable. +- Hidden evaluation files, grading outputs and judge credentials are inaccessible to the submitted programs. +- The stage budgets sit within a 5,400-second overall verifier phase and do not extend it. + +### Build interface + +To prepare the memory for repeated questions, provide the build interface: + +```bash +/app/build.sh \ + --memory /app/memory.json \ + --output /path/to/index +``` + +It runs once and has 300 seconds, including Jina calls. Create the requested index directory, which does not initially exist; its parent exists and is writable for temporary work. This stage can read `memory.json`, `build.sh`, the supplied Jina transport and installed runtime dependencies, but not the other scripts, original transcripts or evaluation data. Only the resulting index is retained for retrieval, and it becomes read-only. Its entries must be regular files or directories without links. + +### Search interface and output contract + +For each question, the search interface is: + +```bash +/app/run.sh \ + --index /path/to/index \ + --question "$question" \ + --output /path/to/memories.json +``` + +Write a JSON array of at most ten strings in relevance order, drawn from the indexed memory; an empty array is allowed. Search can read only the index, `run.sh`, the supplied Jina transport and installed runtime dependencies. It cannot read `memory.json`, other scripts, the original transcripts or other requests. Use the output file's parent for temporary work. All searches together have 150 seconds, including Jina calls. + +### Answer interface and output contract -## Environment and Resources +For each search result, the answering interface is: -Before implementing the system, read `/task/docs/environment.md` for the installed Python environment, runtime and tools, and `/task/docs/available_resources.md` for API endpoints, credential handling, exact model allowlists and usage restrictions. Local computation and the supplied Jina embedding and reranking APIs are available for memory preparation, indexing and retrieval. Generative helper APIs are not permitted for those operations. Only the answering component may use the OpenRouter generation API, with the current question and retrieved strings, choosing among the four models in the resource policy. An unlisted model, provider or endpoint is not authorized. Harbor supplies the documented credentials at runtime; do not put their values in code, memories, indexes, prompts or logs. +```bash +/app/answer.sh \ + --question "$question" \ + --memories /path/to/memories.json \ + --output /path/to/answer.txt +``` -You have 120 minutes for implementation and validation, with eight CPU cores, 8 GiB RAM, 8 GiB storage and no GPU. Development runs as root, and no local model weights are supplied under `/opt/models`. Treat `/task` and its documentation as read-only. Do not obtain held-out labels, change the evaluator, use external answer services or embed answers to particular evaluation questions. +Its memories file contains exactly the string array returned by search. The answerer may read only `answer.sh`, this question's recalled strings, the supplied generation transport and installed runtime dependencies. It cannot consult the full memory, index, other scripts, original transcripts or previous requests. Write nonempty UTF-8 answer text of at most 2,000 Unicode characters, without a JSON wrapper or a list of recalled notes. If the notes are insufficient, say so in plain text. All answers together have 1,800 seconds; use the output file's parent for temporary work. + +## Available Validation Data + +The complete history is available at `/task/data/history.jsonl`. Its records contain `meeting_id`, `series_id`, `date`, `speaker`, `start_seconds`, `end_seconds`, `text` and `source_ids`; a null speaker is unidentified. To check the system while developing it, use the five public questions at `/task/data/validation/queries.jsonl`, the reference answers and required facts at `/task/data/validation/golden_answers.jsonl`, and their supporting transcript excerpts at `/task/data/validation/evidence.jsonl`. The reference files are keyed by `query_id`, but the program receives only each row's `question` as a single command-line argument, without an ID or meeting metadata. The remaining public-query fields are for analysis. The final questions concern the same 67 meetings, so prepare the memory from the supplied history rather than from a list of question-to-answer mappings. + +Check the memory's JSON validity and byte size, then run build, search and answer on the public questions using the development API route where needed. ## Expected Artifacts Deliver exactly four regular files under `/app`: the finished `memory.json` and executable `build.sh`, `run.sh` and `answer.sh` scripts. Links are not supported. Each script must be self-contained, using embedded code or installed libraries rather than additional submitted modules. The scripts must contain corpus-independent implementation code, without additional memories, transcript excerpts or question-to-answer tables. Keep other development outputs outside `/app`. Only these four files are transferred to the separate evaluation environment; files in your home directory, additional installed packages, indexes and running services are not transferred. -The memory must be readable UTF-8 JSON, with no duplicate object keys or non-finite numbers, and no larger than 183,981 bytes. This is five percent of the history's 3,679,621 UTF-8 dialogue-text bytes, including one newline per utterance and excluding the source JSON metadata. The limit counts the actual bytes of `memory.json`, including its formatting and metadata. Choose its JSON structure freely, but store directly readable memory text rather than compressed or encoded dialogue. Script bytes and the index generated from this memory are outside that memory-file budget. +```text +/app/ +├── memory.json +├── build.sh +├── run.sh +└── answer.sh +``` -## Build, Search and Answer Interfaces +## Verification -To prepare the memory for repeated questions, provide the build interface `/app/build.sh --memory /app/memory.json --output /path/to/index`. It runs once and has 300 seconds, including Jina calls. Create the requested index directory, which does not initially exist; its parent exists and is writable for temporary work. This stage can read `memory.json`, `build.sh`, the supplied Jina transport and installed runtime dependencies, but not the other scripts, original transcripts or evaluation data. Only the resulting index is retained for retrieval, and it becomes read-only. Its entries must be regular files or directories without links. +After you finish, Harbor transfers the four files to a separate verifier and runs the same interfaces on 143 held-out questions. -For each question, the search interface is `/app/run.sh --index /path/to/index --question "$question" --output /path/to/memories.json`. Write a JSON array of at most ten strings in relevance order, drawn from the indexed memory; an empty array is allowed. Search can read only the index, `run.sh`, the supplied Jina transport and installed runtime dependencies. It cannot read `memory.json`, other scripts, the original transcripts or other requests. Use the output file's parent for temporary work. All searches together have 150 seconds, including Jina calls. +A failed process, invalid submission or invalid output receives zero for the submission. -For each search result, the answering interface is `/app/answer.sh --question "$question" --memories /path/to/memories.json --output /path/to/answer.txt`. Its memories file contains exactly the string array returned by search. The answerer may read only `answer.sh`, this question's recalled strings, the supplied generation transport and installed runtime dependencies. It cannot consult the full memory, index, other scripts, original transcripts or previous requests. Write nonempty UTF-8 answer text of at most 2,000 Unicode characters, without a JSON wrapper or a list of recalled notes. If the notes are insufficient, say so in plain text. All answers together have 1,800 seconds; use the output file's parent for temporary work. +An answer earns credit only when both its retrieved evidence and its answer are correct. The retrieved notes must preserve at least one question-relevant factual point from the supporting excerpts; faithful summaries and paraphrases count, while topic overlap alone does not. The answer must agree with the reference and include the required facts without contradictions. Evidence is judged without seeing the submitted answer, and the answer is judged without seeing the retrieved notes. EvidenceGroundedAnswerAccuracy is the mean of these joint binary outcomes, and the final reward applies the fixed task-specific piecewise normalization documented in README.md to that mean. A separate trajectory and file audit checks task and resource compliance; a violation sets the entire score to zero, while an incomplete audit or a failed judge service is an infrastructure failure. -## Execution and File Permissions +## Hidden Test Overview -Honor every supplied path, which may vary between invocations, and return exit status `0` from all three scripts. Submitted commands run as a non-root user in a fresh working directory for each invocation. Submitted files, stage inputs and installed runtime files are read-only; only that stage's working directory is writable. Hidden evaluation files, grading outputs and judge credentials are inaccessible to the submitted programs. The stage budgets sit within a 5,400-second overall verifier phase and do not extend it. +The held-out questions, reference answers and evidence are disjoint from the public examples and are not provided in the development environment. -## Evaluation API Transports +## Environment and available resources -For Jina calls during evaluation, `build.sh` and `run.sh` receive `TASK_JINA_CLIENT` and `TASK_JINA_FD`. Load the supplied client module and use its `embeddings` or `rerank` function, preserving the named file descriptor when starting a subprocess. The client forwards the operation and provider request body while the real key stays outside the submitted process; direct network access is disabled. Each serialized request, including its newline, must fit within 128 KiB, so batch larger inputs. There is no separate Jina call-count limit, but calls consume the current stage's time budget. Use only the permitted stage inputs. During development, when this transport is absent, use the documented Jina endpoints with `JINA_API_KEY` and text from the supplied history. The answerer does not receive the Jina transport. +Before implementing the system, read the following task-provided documents: -For generation during evaluation, `answer.sh` receives `TASK_LLM_CLIENT` and `TASK_LLM_FD`. Load that client module and call `chat_completion`, preserving its file descriptor across subprocesses. When the client is present, use it without requiring `OPENROUTER_API_KEY` or direct network access. During development the transport is absent, and the answerer may send the same request to the documented OpenRouter Chat Completions endpoint using `OPENROUTER_API_KEY`. The resource policy includes examples of both routes. Each request must explicitly select an allowed model and use only the current question and retrieved notes; a failed call does not permit another provider or endpoint. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -Each question allows at most two generation calls, with 1–2,000 output tokens per call. The accepted request options are `model`, `messages`, `temperature`, `top_p`, `max_tokens` and `response_format`. A request may contain 1–32 messages, each with exactly the string fields `role` and `content`; roles may be `system`, `user` or `assistant`. The serialized request, including its newline, must fit within 128 KiB. Streaming, tool calls and alternate endpoints are not supported. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. -## Validation and Answer Requirements +Local computation and the supplied Jina embedding and reranking APIs are available for memory preparation, indexing and retrieval. Generative helper APIs are not permitted for those operations. Only the answering component may use the OpenRouter generation API, with the current question and retrieved strings, choosing among the four models in the resource policy. An unlisted model, provider or endpoint is not authorized. Harbor supplies the documented credentials at runtime; do not put their values in code, memories, indexes, prompts or logs. -Check the memory's JSON validity and byte size, then run build, search and answer on the public questions using the development API route where needed. After you finish, Harbor transfers the four files to a separate verifier and runs the same interfaces on 143 held-out questions. Their questions, reference answers and evidence are disjoint from the public examples and are not provided in your environment. A failed process, invalid submission or invalid output receives zero for the submission. +### Retrieval API transport -An answer earns credit only when both its retrieved evidence and its answer are correct. The retrieved notes must preserve at least one question-relevant factual point from the supporting excerpts; faithful summaries and paraphrases count, while topic overlap alone does not. The answer must agree with the reference and include the required facts without contradictions. Evidence is judged without seeing the submitted answer, and the answer is judged without seeing the retrieved notes. EvidenceGroundedAnswerAccuracy is the mean of these joint binary outcomes, and the final reward applies the fixed task-specific piecewise normalization documented in README.md to that mean. A separate trajectory and file audit checks task and resource compliance; a violation sets the entire score to zero, while an incomplete audit or a failed judge service is an infrastructure failure. +For Jina calls during evaluation, `build.sh` and `run.sh` receive `TASK_JINA_CLIENT` and `TASK_JINA_FD`. Load the supplied client module and use its `embeddings` or `rerank` function, preserving the named file descriptor when starting a subprocess. The client forwards the operation and provider request body while the real key stays outside the submitted process; direct network access is disabled. Each serialized request, including its newline, must fit within 128 KiB, so batch larger inputs. There is no separate Jina call-count limit, but calls consume the current stage's time budget. Use only the permitted stage inputs. During development, when this transport is absent, use the documented Jina endpoints with `JINA_API_KEY` and text from the supplied history. The answerer does not receive the Jina transport. + +### Answer API transport + +For generation during evaluation, `answer.sh` receives `TASK_LLM_CLIENT` and `TASK_LLM_FD`. Load that client module and call `chat_completion`, preserving its file descriptor across subprocesses. When the client is present, use it without requiring `OPENROUTER_API_KEY` or direct network access. During development the transport is absent, and the answerer may send the same request to the documented OpenRouter Chat Completions endpoint using `OPENROUTER_API_KEY`. The resource policy includes examples of both routes. Each request must explicitly select an allowed model and use only the current question and retrieved notes; a failed call does not permit another provider or endpoint. + +Each question allows at most two generation calls, with 1–2,000 output tokens per call. The accepted request options are `model`, `messages`, `temperature`, `top_p`, `max_tokens` and `response_format`. A request may contain 1–32 messages, each with exactly the string fields `role` and `content`; roles may be `system`, `user` or `assistant`. The serialized request, including its newline, must fit within 128 KiB. Streaming, tool calls and alternate endpoints are not supported. diff --git a/tasks/khmer-embedding-training/raw-instruction.md b/tasks/khmer-embedding-training/raw-instruction.md index f217fe5..1d890ca 100644 --- a/tasks/khmer-embedding-training/raw-instruction.md +++ b/tasks/khmer-embedding-training/raw-instruction.md @@ -1,37 +1,107 @@ -# Task: Khmer-to-English Embedding Retrieval +# Khmer-to-English Embedding Retrieval ## Task Description -I need an embedding model and a search system for finding English documents with Khmer questions. Train the supplied multilingual-e5-large model for this collection and provide an executable indexing and retrieval pipeline. +Build an embedding model and a search system for finding English documents with Khmer questions. Train the supplied multilingual-e5-large model for this collection and provide an executable indexing and retrieval pipeline. -## Inputs and Available Validation Data +## Requirements -Read `/task/docs/environment.md` and `/task/docs/available_resources.md` first. The collection at `/task/data/corpus.parquet` has `_id`, `title` and `text` columns. Three examples are available in `/task/data/examples.jsonl`, with `id`, `question` and `answers` fields; the questions are in Khmer and the answers are in English. No other labeled training data is supplied. The embedding backbone is at `/task/models/multilingual-e5-large`, and an optional local generative model is at `/task/models/Qwen3.5-9B`. +### Resource limits -## Environment and Resources +- You have 6 hours, one A100 with 40 GiB GPU memory, 8 CPU cores, 32 GiB RAM and 100 GiB working storage. +- Network access is unavailable. +- Use only the supplied local resources. +- Treat `/task` as read-only and put your work under `/app`. -You have 6 hours, one A100 with 40 GiB GPU memory, 8 CPU cores, 32 GiB RAM and 100 GiB working storage. Network access is unavailable. Use only the supplied local resources. Treat `/task` as read-only and put your work under `/app`. +### Execution and file permissions -## Trained Checkpoint +- `build.sh` must create the requested index directory from the supplied collection. +- The index directory does not initially exist; its parent exists and is writable. +- Construction has a 12-hour limit. +- A previously built collection index or cached retrieval results must not be used as the submission. +- `run.sh` must use the resulting index and support repeated invocations, with a 30-minute limit per invocation. +- The index is read-only during retrieval. +- Honour the supplied paths rather than hard-coding the example paths. +- Build and retrieval run as separate non-root users, each with a writable working directory and `TMPDIR`; `/app` and task inputs are read-only. +- Use those writable directories for temporary files, honor the supplied output path, and do not write caches beside the checkpoint or assume a writable system temporary directory. +- The indexing and retrieval environment provides the same embedding backbone, one GPU, 8 CPU cores and 32 GiB RAM, without the optional generative model or network access. -Save the trained weights to `/app/submission/model/model.safetensors`, overwriting that file if it exists. Put `config.json` and the unchanged tokenizer files in the same directory with their original filenames. Preserve the backbone architecture, parameter names, shapes and architecture configuration. The directory must be a complete, self-contained Hugging Face checkpoint that `AutoTokenizer.from_pretrained` and `AutoModel.from_pretrained` can load with `local_files_only=True` and `trust_remote_code=False`. +### Build interface -## Build, Retrieval and File Permissions +The build command accepts the following arguments: -Provide executable `/app/build.sh` and `/app/run.sh`, with any supporting files they need under `/app`. They must accept the following arguments: +```bash +/app/build.sh \ + --corpus /task/data/corpus.parquet \ + --index-dir /path/to/index +``` -`/app/build.sh --corpus /task/data/corpus.parquet --index-dir /path/to/index` `/app/run.sh --index-dir /path/to/index --queries /path/to/queries.jsonl --output /path/to/results.jsonl` +### Query interface -`build.sh` must create the requested index directory from the supplied collection. The index directory does not initially exist; its parent exists and is writable. Construction has a 12-hour limit. A previously built collection index or cached retrieval results must not be used as the submission. `run.sh` must use the resulting index and support repeated invocations, with a 30-minute limit per invocation. The index is read-only during retrieval. Honour the supplied paths rather than assuming the example paths above. Build and retrieval run as separate non-root users, each with a writable working directory and `TMPDIR`; `/app` and task inputs are read-only. Use those writable directories for temporary files, honor the supplied output path, and do not write caches beside the checkpoint or assume a writable system temporary directory. The indexing and retrieval environment provides the same embedding backbone, one GPU, 8 CPU cores and 32 GiB RAM, without the optional generative model or network access. +```bash +/app/run.sh \ + --index-dir /path/to/index \ + --queries /path/to/queries.jsonl \ + --output /path/to/results.jsonl +``` -## Query and Output Contract +### Input and output contract Each query is a JSONL row with `id` and `question`: -`{"id":"query-001","question":"..."}` +```json +{"id":"query-001","question":"..."} +``` For each query, write one JSONL row in the same order, containing the same `id`, `indices` and `scores`. Return 100 unique document positions, ranked by decreasing relevance score. Positions are zero-based row numbers in the supplied Parquet file. Scores must be finite numbers; ties are ordered by ascending document position. If a collection has fewer than 100 documents, return all its positions. -`{"id":"query-001","indices":[42,7,103],"scores":[0.91,0.85,0.72]}` +```json +{"id":"query-001","indices":[42,7,103],"scores":[0.91,0.85,0.72]} +``` The abbreviated example illustrates the fields and ranking order. Do not write progress messages to the result file. + +## Available Validation Data + +The collection at `/task/data/corpus.parquet` has `_id`, `title` and `text` columns. Three examples are available in `/task/data/examples.jsonl`, with `id`, `question` and `answers` fields; the questions are in Khmer and the answers are in English. No other labeled training data is supplied. The embedding backbone is at `/task/models/multilingual-e5-large`, and an optional local generative model is at `/task/models/Qwen3.5-9B`. + +## Expected Artifacts + +Provide executable `/app/build.sh` and `/app/run.sh`, with any supporting files they need under `/app`. + +Save the trained weights to `/app/submission/model/model.safetensors`, overwriting that file if it exists. Put `config.json` and the unchanged tokenizer files in the same directory with their original filenames. Preserve the backbone architecture, parameter names, shapes and architecture configuration. The directory must be a complete, self-contained Hugging Face checkpoint that `AutoTokenizer.from_pretrained` and `AutoModel.from_pretrained` can load with `local_files_only=True` and `trust_remote_code=False`. + +```text +/app/ +├── build.sh +├── run.sh +└── submission/ + └── model/ + ├── model.safetensors + ├── config.json + └── ... # unchanged tokenizer files +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. An invalid checkpoint, a failed or timed-out build or retrieval command, or invalid retrieval output gives zero for the entire submission. +4. **Search quality.** Raw quality is Recall@100: a query succeeds when at least one of its first 100 retrieved documents contains an accepted answer under the answer-token matching rule. Recall@20 is reported separately. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. + +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/layout-page-search/raw-instruction.md b/tasks/layout-page-search/raw-instruction.md index 3c4586e..9458d1c 100644 --- a/tasks/layout-page-search/raw-instruction.md +++ b/tasks/layout-page-search/raw-instruction.md @@ -1,25 +1,97 @@ -# Task: Visual-Document Evidence Page Retrieval +# Visual-Document Evidence Page Retrieval ## Task Description -I need to locate the pages that support a question about a report's figures, tables or layout, so I can inspect the original material. Build a search system that returns a compact set of supporting pages within the specified document. Include the pages needed to check the request, while avoiding unrelated pages. +Locate the pages that support a question about a report's figures, tables or layout so that the original material can be inspected. Build a search system that returns a compact set of supporting pages within the specified document. Include the pages needed to check the request, while avoiding unrelated pages. -## Inputs and Available Validation Data +## Requirements + +### Resource limits and workload + +- You have six hours for implementation and checking, eight CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. +- The supplied encoders are fixed. +- An optional scoring or adaptation component may have up to five million trainable parameters and use up to 45 minutes of training. +- Per request, query-dependent page-token or region comparisons and learned pairwise scoring may inspect at most 40 candidate pages; corpus-wide text processing, global representations and gallery-only preprocessing are allowed. +- No external model, dataset or API is available. + +- The query workload contains at most 600 requests in two calls, each with an independent 2,700-second allowance. +- Completed valid rows remain usable if a call times out; missing or malformed rows are unsuccessful. +- Honor the supplied paths. +- The submitted system must work after a fresh start with the same inputs. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface + +Provide executable `/app/build.sh` and `/app/run.sh`, with all supporting code and learned files under `/app`. Index construction uses: + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir DIR +``` + +The build command must return successfully within 3,600 seconds. + +### Query interface + +Requests use: + +```bash +/app/run.sh \ + --index-dir DIR \ + --queries JSONL \ + --output JSONL +``` + +### Input and output contract + +Each input row contains `query_id`, `doc_id`, `question`, `domain` and `language`. Return one JSONL object per input, preserving `query_id`, with a `results` list of at most ten distinct pages from the specified document. Each item contains `page_id` and a finite numeric `score`, in nonincreasing score order. No answer text or bounding box is required. Write diagnostics to stderr. + +## Available Validation Data The page collection is at `/task/data/corpus/`. Its `records.jsonl` lists `page_id`, `domain`, `doc_id`, `page_number`, `image_path`, `markdown`. Paths are relative to that directory. The images are rendered document pages and the text is uniformly extracted from the documents. Local model weights are available at `/task/models/page-encoder/` and its underlying model at `/task/models/vision-backbone/`. These resources are read-only. Public requests and page-relevance annotations are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use these examples to check the complete build and query path. The remaining requests and labels are not supplied. -## Expected Artifacts and Interfaces +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is set F1 between returned pages and the complete reference evidence-page set, averaged across held-out questions. Both missing evidence pages and extra pages reduce this metric. Domain-level results are reported separately. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. -Provide executable `/app/build.sh` and `/app/run.sh`, with all supporting code and learned files under `/app`. Index construction uses `/app/build.sh --corpus /task/data/corpus --index-dir DIR` and must return successfully within 3,600 seconds. Requests use `/app/run.sh --index-dir DIR --queries JSONL --output JSONL`. Each input row contains `query_id`, `doc_id`, `question`, `domain` and `language`. Return one JSONL object per input, preserving `query_id`, with a `results` list of at most ten distinct pages from the specified document. Each item contains `page_id` and a finite numeric `score`, in nonincreasing score order. No answer text or bounding box is required. Write diagnostics to stderr. +## Hidden Test Overview -## Environment and Resource Limits +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. -You have six hours for implementation and checking, eight CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. The supplied encoders are fixed. An optional scoring or adaptation component may have up to five million trainable parameters and use up to 45 minutes of training. Per request, query-dependent page-token or region comparisons and learned pairwise scoring may inspect at most 40 candidate pages; corpus-wide text processing, global representations and gallery-only preprocessing are allowed. No external model, dataset or API is available. +## Environment and available resources -## Execution, Permissions and Workload +Before implementing the system, read the following task-provided documents: -The query workload contains at most 600 requests in two calls, each with an independent 2,700-second allowance. Completed valid rows remain usable if a call times out; missing or malformed rows are unsuccessful. Honor the supplied paths. The submitted system must work after a fresh start with the same inputs. Before starting, read `/task/docs/environment.md` and `/task/docs/available_resources.md`. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/narrative-clue-search/raw-instruction.md b/tasks/narrative-clue-search/raw-instruction.md index cb6ae22..d8d087c 100644 --- a/tasks/narrative-clue-search/raw-instruction.md +++ b/tasks/narrative-clue-search/raw-instruction.md @@ -1,32 +1,95 @@ -# Task: Narrative-Clue Search +# Narrative-Clue Search ## Task Description Build a search system for users who recall a work through scattered details. A request may contain a long description, comparisons, uncertain memories and background information. Return documents from the supplied collection. -## Inputs and Available Validation Data +## Requirements -The collection is at `/task/data/corpus/records.jsonl`. Each row contains `doc_id`, `title` and the full article `text`. +### Resource limits and workload -Public queries and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels`, a mapping of document IDs to relevance grades. Original sentence-level annotations for these public queries are at `/task/data/training/sentence_labels.jsonl`. The remaining queries and labels are not supplied. +- The environment has 8 CPU cores, 32 GiB RAM and 100 GiB storage, without a GPU. +- The OpenRouter and Jina APIs listed in the resource document are available. +- Development has a two-hour budget, index construction up to 60 minutes, and the query workload described below. +- Calls may be repeated with different input files. +- Data and documentation mounts are read-only; write generated indexes to the supplied index directory. + +- The system must handle workloads of up to 450 requests in 2 calls of at most 225 requests. +- Each call has an independent 2,700-second allowance; time used by one call does not reduce another. +- Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. +- A failed query command other than a timeout invalidates that call. -## Expected Artifacts and Interfaces +### Execution and file permissions + +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface Create executable `/app/build.sh` and `/app/run.sh`, with their supporting files under `/app`: -`/app/build.sh --corpus /task/data/corpus --index-dir /work/index` `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl` -Each input JSONL row contains `query_id` and `question`. Return the same `query_id` and a `results` list of at most ten distinct documents, each with `doc_id` and a finite numeric `score`, in descending score order. Write diagnostics to stderr. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` -## Resources, Permissions and Workload +### Query interface -The environment has 8 CPU cores, 32 GiB RAM and 100 GiB storage, without a GPU. The OpenRouter and Jina APIs listed in the resource document are available. Development has a two-hour budget, index construction up to 60 minutes, and the query workload described below. Calls may be repeated with different input files. Data and documentation mounts are read-only; write generated indexes to the supplied index directory. +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` -During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +### Input and output contract -The system must handle workloads of up to 450 requests in 2 calls of at most 225 requests. Each call has an independent 2,700-second allowance; time used by one call does not reduce another. Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. A failed query command other than a timeout invalidates that call. +Each input JSONL row contains `query_id` and `question`. Return the same `query_id` and a `results` list of at most ten distinct documents, each with `doc_id` and a finite numeric `score`, in descending score order. Write diagnostics to stderr. + +## Available Validation Data -## Environment and Development Checks +The collection is at `/task/data/corpus/records.jsonl`. Each row contains `doc_id`, `title` and the full article `text`. -Read `/task/docs/environment.md` and `/task/docs/available_resources.md` before starting. Evaluation checks executable behavior, result validity and relevance on held-out queries, together with the resource and API rules. It runs in a separate environment with the submitted `/app` files and supplied corpus. +Public queries and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels`, a mapping of document IDs to relevance grades. Original sentence-level annotations for these public queries are at `/task/data/training/sentence_labels.jsonl`. The remaining queries and labels are not supplied. You may use the public examples to implement and check the complete build and query path during development. Include all required implementation files under `/app` so the system runs after a fresh start with the supplied inputs. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is nDCG@10 using the original relevance judgments, averaged equally across held-out questions. Category-level results are diagnostics and do not change query weights. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. + +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/same-place-image-search/raw-instruction.md b/tasks/same-place-image-search/raw-instruction.md index 8b46785..74d5183 100644 --- a/tasks/same-place-image-search/raw-instruction.md +++ b/tasks/same-place-image-search/raw-instruction.md @@ -1,25 +1,101 @@ -# Task: Same-Place Image Search +# Same-Place Image Search ## Task Description A cropped architectural photograph may show only a small part of a building. Build an image search system that finds additional photographs of that physical place, including views with different scale, viewpoint and occlusion. Each request supplies image IDs to exclude from its results. -## Inputs and Available Validation Data +## Requirements + +### Resource limits and workload + +- You have two hours, 8 CPU cores, 32 GiB RAM, 100 GiB working storage and one 40 GiB GPU. +- The encoder is fixed. +- The build command has 3,600 seconds. +- The system must handle up to 140 requests in two calls of at most 70 requests, with an independent 2,700-second allowance per call. +- Additional pixel-based geometric verification may examine at most 20 candidate gallery images per request. +- Corpus-wide feature screening and gallery-only preprocessing are allowed within the runtime limits. + +- Process requests independently. +- Data, models and documentation are read-only. +- Generate the index at the passed path and include required runtime files under `/app`. +- The submitted files must work after a fresh start with the supplied gallery, fixed encoder and current queries. +- Completed valid JSONL rows remain usable after a timeout; missing or invalid rows are unsuccessful. +- A command failure other than timeout invalidates that call. +- Read `/task/docs/environment.md` and `/task/docs/available_resources.md`. +- Aim to return photographs of the same physical place rather than photographs that merely depict a similar style of building. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. +- The passed index directory already exists and is writable during construction, but its parent directory is not writable. +- Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. +- The completed index becomes read-only for retrieval. +- Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. +- Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface + +Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is: + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface + +The query interface is: + +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` + +### Input and output contract + +Each input JSONL row contains `query_id`, `image_path` and `exclude_image_ids`. The image path is an absolute path to the current query image. Search the gallery excluding that request's listed IDs. + +Return the same `query_id` and a `results` list of at most five distinct eligible images. Each result contains `image_id` and a finite numeric `score`, in nonincreasing score order. Write diagnostics to stderr. + +## Available Validation Data The gallery is at `/task/data/corpus/`. Each row of `records.jsonl` contains `image_id`, `collection` and `image_path`. Image paths are relative to that directory. Original gallery photographs are supplied. Local image encoder weights are available at `/task/models/clip/`. -## Expected Artifacts and Interfaces +Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels` for matching images. Public query images are under `/task/data/queries/`. You may use these examples to develop and check the system. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is binary nDCG@5, which rewards placing labeled matching images near the top of the ranking. Query scores are averaged within each gallery collection, then equally across the two collections. Excluded images are not valid retrieval results. The relevance-label protocol is described in [README.md — Search Quality](README.md#search-quality). -Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is `/app/build.sh --corpus /task/data/corpus --index-dir /work/index`. The query interface is `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl`. Each input JSONL row contains `query_id`, `image_path` and `exclude_image_ids`. The image path is an absolute path to the current query image. Search the gallery excluding that request's listed IDs. +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. -Return the same `query_id` and a `results` list of at most five distinct eligible images. Each result contains `image_id` and a finite numeric `score`, in nonincreasing score order. Write diagnostics to stderr. Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels` for matching images. Public query images are under `/task/data/queries/`. You may use these examples to develop and check the system. +## Hidden Test Overview -## Resources and Workload +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. -You have two hours, 8 CPU cores, 32 GiB RAM, 100 GiB working storage and one 40 GiB GPU. The encoder is fixed. The build command has 3,600 seconds. The system must handle up to 140 requests in two calls of at most 70 requests, with an independent 2,700-second allowance per call. Additional pixel-based geometric verification may examine at most 20 candidate gallery images per request. Corpus-wide feature screening and gallery-only preprocessing are allowed within the runtime limits. +## Environment and available resources -## Execution and File Permissions +Before implementing the system, read the following task-provided documents: -Process requests independently. Data, models and documentation are read-only. Generate the index at the passed path and include required runtime files under `/app`. The submitted files must work after a fresh start with the supplied gallery, fixed encoder and current queries. Completed valid JSONL rows remain usable after a timeout; missing or invalid rows are unsuccessful. A command failure other than timeout invalidates that call. Read `/task/docs/environment.md` and `/task/docs/available_resources.md`. Aim to return photographs of the same physical place rather than photographs that merely depict a similar style of building. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. The passed index directory already exists and is writable during construction, but its parent directory is not writable. Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The completed index becomes read-only for retrieval. Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/sketch-text-search/raw-instruction.md b/tasks/sketch-text-search/raw-instruction.md index 273b094..f46a450 100644 --- a/tasks/sketch-text-search/raw-instruction.md +++ b/tasks/sketch-text-search/raw-instruction.md @@ -1,29 +1,109 @@ -# Task: Sketch-and-Text Image Search +# Sketch-and-Text Image Search ## Task Description An image search request may contain a rough sketch of only part of the desired composition and a description of the complete scene. Build a system that uses these inputs to find the matching artwork in the supplied gallery. -## Inputs, Training and Validation Data +## Requirements + +### Resource limits and workload + +- You have six hours, 8 CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. +- The supplied encoders are fixed; scoring or fusion components may use up to 30 minutes of adaptation. +- The build command has 3,600 seconds. +- The system must handle up to 1,260 requests in three calls of at most 420 requests, with an independent 1,800-second allowance for each call. +- Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful. +- A command failure other than timeout invalidates that call. + +- Data, models and documentation are read-only. +- Generate the index at the passed index path. +- Include required runtime files and learned weights under `/app` so the system works after a fresh start with the supplied gallery, model and current queries. +- Training and validation labels are not runtime indexing inputs. +- Use the local resources described in `/task/docs/available_resources.md` and read `/task/docs/environment.md`. +- Aim to place the matching artwork among the first three results, especially first, for both broad and detailed descriptions. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. +- The passed index directory already exists and is writable during construction, but its parent directory is not writable. +- Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. +- The completed index becomes read-only for retrieval. +- Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. +- Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface + +Provide executable `/app/build.sh` and `/app/run.sh`, together with their implementation and learned files under `/app`. The build interface is: + +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface + +The query interface is: + +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` + +### Input and output contract + +Each input JSONL row contains `query_id`, `description` and `sketch_path`. The sketch path is an absolute path to the current sketch image. + +Return the same `query_id` and a `results` list of at most ten distinct gallery images. Each result contains `image_id` and a finite numeric `score`, in nonincreasing score order. Write diagnostics to stderr. Process each sketch-description pair independently; descriptions and results from other requests are not additional context. Computations depending only on the gallery or an image may be cached. + +## Available Validation Data The gallery is at `/task/data/corpus/`. Each row of `records.jsonl` contains `image_id` and `image_path`, with paths relative to that directory. Original gallery images are available. Local image and text encoder weights are at `/task/models/clip/`. Sketches are partial views; they need not depict every element mentioned in the description. Training pairs and their relevance labels are in `/task/data/training/queries.jsonl` and `/task/data/training/labels.jsonl`. Public validation pairs are at the corresponding paths under `/task/data/validation/`. Labels contain `query_id` and `qrels`, a mapping of gallery image IDs to relevance labels. You may use these inputs to develop and check the system. -## Expected Artifacts and Interfaces +A development-only permissions check is available at `/task/docs/check_runtime.py`; it runs your scripts under these permissions without scoring or reading labels. For example, use: -Provide executable `/app/build.sh` and `/app/run.sh`, together with their implementation and learned files under `/app`. The build interface is `/app/build.sh --corpus /task/data/corpus --index-dir /work/index`. The query interface is `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl`. Each input JSONL row contains `query_id`, `description` and `sketch_path`. The sketch path is an absolute path to the current sketch image. +```bash +/opt/conda/bin/python /task/docs/check_runtime.py build \ + --corpus /task/data/corpus \ + --index-dir /work/runtime-check/index +``` -Return the same `query_id` and a `results` list of at most ten distinct gallery images. Each result contains `image_id` and a finite numeric `score`, in nonincreasing score order. Write diagnostics to stderr. Process each sketch-description pair independently; descriptions and results from other requests are not additional context. Computations depending only on the gallery or an image may be cached. +For a query check, use the `query` stage with that index, a public `--queries` path and an `--output` path under `/work/runtime-check/output/`. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Per-query raw quality is 0.5 * Recall@3 + 0.5 * MRR@3. Scores are averaged within each of the three description levels, then equally across the levels. The interface accepts up to ten results, but only the first three affect the metric. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. -## Resources and Workload +## Hidden Test Overview -You have six hours, 8 CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. The supplied encoders are fixed; scoring or fusion components may use up to 30 minutes of adaptation. The build command has 3,600 seconds. The system must handle up to 1,260 requests in three calls of at most 420 requests, with an independent 1,800-second allowance for each call. Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful. A command failure other than timeout invalidates that call. +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. -## Execution, Permissions and Development Checks +## Environment and available resources -Data, models and documentation are read-only. Generate the index at the passed index path. Include required runtime files and learned weights under `/app` so the system works after a fresh start with the supplied gallery, model and current queries. Training and validation labels are not runtime indexing inputs. Use the local resources described in `/task/docs/available_resources.md` and read `/task/docs/environment.md`. Aim to place the matching artwork among the first three results, especially first, for both broad and detailed descriptions. +Before implementing the system, read the following task-provided documents: -During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. The passed index directory already exists and is writable during construction, but its parent directory is not writable. Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The completed index becomes read-only for retrieval. Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. -A development-only permissions check is available at `/task/docs/check_runtime.py`; it runs your scripts under these permissions without scoring or reading labels. For example, use `/opt/conda/bin/python /task/docs/check_runtime.py build --corpus /task/data/corpus --index-dir /work/runtime-check/index`. For a query check, use the `query` stage with that index, a public `--queries` path and an `--output` path under `/work/runtime-check/output/`. +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/spoken-query-search/raw-instruction.md b/tasks/spoken-query-search/raw-instruction.md index 7b80419..b4758b9 100644 --- a/tasks/spoken-query-search/raw-instruction.md +++ b/tasks/spoken-query-search/raw-instruction.md @@ -1,34 +1,98 @@ -# Task: Spoken-Query Document Search +# Spoken-Query Document Search ## Task Description Build a document search system for spoken requests whose transcription may contain mistaken words or names. Each request provides the output of a fixed speech recognizer, including its best transcription, alternative hypotheses and relative hypothesis scores. -## Inputs and Available Validation Data +## Requirements -The supplied document collections are at `/task/data/corpus/`. Each collection has a JSONL file with `doc_id`, `domain`, `language`, `title` and `text`. Local text encoder weights are available at `/task/models/encoder/`. +### Resource limits and workload -Public queries and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels`, a mapping of document IDs to relevance grades. Clean question text and query audio are not provided. The remaining transcriptions and labels are not supplied. +- The environment has 8 CPU cores, 32 GiB RAM and 100 GiB storage and one 40 GiB GPU. +- Development has a two-hour budget, index construction up to 180 minutes, and query calls follow the workload below. +- Calls may be repeated with different input files. +- Use at most five retrieval formulations and at most 50 query-document pairs for further reranking per request. +- The speech recognizer is fixed; additional speech recognition is not available. + +- Process requests independently; do not combine hypotheses or results from different requests. +- Data, model and documentation mounts are read-only. +- Place all other required runtime files under `/app`. + +- The system must handle workloads of up to 4,600 requests in 4 calls of at most 1,200 requests. +- Each call has an independent 1,350-second allowance; time used by one call does not reduce another. +- Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. +- A failed query command other than a timeout invalidates that call. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. +- The passed index directory already exists and is writable during construction, but its parent directory is not writable. +- Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. +- The completed index becomes read-only for retrieval. +- Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. +- Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. -## Expected Artifacts and Interfaces +### Build interface Create executable `/app/build.sh` and `/app/run.sh` and their supporting files under `/app`: -`/app/build.sh --corpus /task/data/corpus --index-dir /work/index` `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl` -Each input JSONL row contains `query_id`, `domain`, `language`, `one_best` and `hypotheses`. Each hypothesis has `text`, `mean_log_probability`, `confidence` and `words`. Word entries give the word, its character span in that hypothesis and its confidence. Return the same `query_id` and a `results` list of at most ten distinct documents from that domain, each containing `doc_id` and a finite numeric `score`, in descending score order. Write diagnostics to stderr. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` -## Resources and Execution Permissions +### Query interface -The environment has 8 CPU cores, 32 GiB RAM and 100 GiB storage and one 40 GiB GPU. Development has a two-hour budget, index construction up to 180 minutes, and query calls follow the workload below. Calls may be repeated with different input files. Use at most five retrieval formulations and at most 50 query-document pairs for further reranking per request. The speech recognizer is fixed; additional speech recognition is not available. +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` -Process requests independently; do not combine hypotheses or results from different requests. Data, model and documentation mounts are read-only. Place all other required runtime files under `/app`. +### Input and output contract -During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. The passed index directory already exists and is writable during construction, but its parent directory is not writable. Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The completed index becomes read-only for retrieval. Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. +Each input JSONL row contains `query_id`, `domain`, `language`, `one_best` and `hypotheses`. Each hypothesis has `text`, `mean_log_probability`, `confidence` and `words`. Word entries give the word, its character span in that hypothesis and its confidence. Return the same `query_id` and a `results` list of at most ten distinct documents from that domain, each containing `doc_id` and a finite numeric `score`, in descending score order. Write diagnostics to stderr. -## Workload and Development Checks +## Available Validation Data -Read `/task/docs/environment.md` and `/task/docs/available_resources.md`. Evaluation checks executable behavior, ranked relevance under transcription uncertainty, output validity and resource/API rules. It runs in a separate environment with the submitted files, supplied collections and current transcription inputs. +The supplied document collections are at `/task/data/corpus/`. Each collection has a JSONL file with `doc_id`, `domain`, `language`, `title` and `text`. Local text encoder weights are available at `/task/models/encoder/`. -The system must handle workloads of up to 4,600 requests in 4 calls of at most 1,200 requests. Each call has an independent 1,350-second allowance; time used by one call does not reduce another. Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. A failed query command other than a timeout invalidates that call. +Public queries and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels contain `query_id` and `qrels`, a mapping of document IDs to relevance grades. Clean question text and query audio are not provided. The remaining transcriptions and labels are not supplied. You may use the public examples to implement and check the complete build and query path during development. Include all required implementation files under `/app` so the system runs after a fresh start with the supplied inputs. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality is graded nDCG@10, averaged within each collection and acoustic condition and then equally across the six collections and four conditions. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. + +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/video-moment-search/raw-instruction.md b/tasks/video-moment-search/raw-instruction.md index 86354ed..a63fd9c 100644 --- a/tasks/video-moment-search/raw-instruction.md +++ b/tasks/video-moment-search/raw-instruction.md @@ -1,32 +1,101 @@ -# Task: Video-Moment Search +# Video-Moment Search ## Task Description Build a search system that locates the time interval matching a natural-language description within a specified video. -## Inputs, Training and Validation Data +## Requirements -The supplied videos are represented by frozen frame vectors at `/task/data/corpus/`. Its `records.jsonl` contains `video_id`, `duration` in seconds, `feature_path` and `grid_path`. Paths are relative to that directory. Each feature row corresponds to one successive interval between adjacent time-grid endpoints. The video vectors were generated with CLIP ViT-B/32. The same fixed image and text encoder is available at `/task/models/clip/`; raw video files are not supplied. +### Resource limits and workload -Public training requests and interval labels are at `/task/data/training/queries.jsonl` and `/task/data/training/labels.jsonl`. Five validation requests and their labels are at the corresponding paths under `/task/data/validation/`. Labels contain `query_id`, `video_id` and `windows`, a list of acceptable start/end pairs. The remaining requests and labels are not provided. +- The environment has 8 CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. +- Development has a six-hour budget. +- The supplied encoders are fixed. +- An optional temporal/scoring component may have up to 5 million trainable parameters and use up to 45 minutes of training. +- Per request, at most 64 candidate intervals may receive learned or token-level fine scoring; inexpensive frame-level screening and boundary proposals are allowed. +- Index construction has up to 60 minutes, and query calls follow the workload below. +- Calls may be repeated with different input files. + +- Process each request independently. +- Data, models and documentation are read-only. +- Put all other required runtime files under `/app`. +- Read `/task/docs/environment.md` and `/task/docs/available_resources.md`. + +- The system must handle workloads of up to 500 requests in 2 calls of at most 250 requests. +- Each call has an independent 2,700-second allowance; time used by one call does not reduce another. +- Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. +- A failed query command other than a timeout invalidates that call. + +### Execution and file permissions + +- During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. +- The passed index directory already exists and is writable during construction, but its parent directory is not writable. +- Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. +- The completed index becomes read-only for retrieval. +- Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. +- Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. -## Expected Artifacts and Interfaces +### Build interface Create executable `/app/build.sh` and `/app/run.sh` and their supporting files under `/app`: -`/app/build.sh --corpus /task/data/corpus --index-dir /work/index` `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl` -Each input JSONL row contains `query_id`, `video_id` and `description`, the original natural-language description. For each input row return the same `query_id` and numeric `start` and `end` values in seconds. Return exactly one nonempty interval whose endpoints belong to that video's supplied grid. Write diagnostics to stderr. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface -## Resources and Execution Permissions +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/results.jsonl +``` -The environment has 8 CPU cores, 32 GiB RAM, 100 GiB storage and one 40 GiB GPU. Development has a six-hour budget. The supplied encoders are fixed. An optional temporal/scoring component may have up to 5 million trainable parameters and use up to 45 minutes of training. Per request, at most 64 candidate intervals may receive learned or token-level fine scoring; inexpensive frame-level screening and boundary proposals are allowed. Index construction has up to 60 minutes, and query calls follow the workload below. Calls may be repeated with different input files. +### Input and output contract -Process each request independently. Data, models and documentation are read-only. Put all other required runtime files under `/app`. Read `/task/docs/environment.md` and `/task/docs/available_resources.md`. +Each input JSONL row contains `query_id`, `video_id` and `description`, the original natural-language description. For each input row return the same `query_id` and numeric `start` and `end` values in seconds. Return exactly one nonempty interval whose endpoints belong to that video's supplied grid. Write diagnostics to stderr. -During verification, build and query commands run as a non-root user with `/app` and supplied inputs read-only. The passed index directory already exists and is writable during construction, but its parent directory is not writable. Create build files and temporary directories inside the index directory or the supplied `TMPDIR`; do not create siblings or rename the index directory itself. The completed index becomes read-only for retrieval. Query results, caches and temporary files belong in the writable output directory or the supplied `TMPDIR` and `XDG_CACHE_HOME`. Honor the passed paths rather than writing to `/app` or assuming a writable system temporary directory. +## Available Validation Data -## Workload and Development Checks +The supplied videos are represented by frozen frame vectors at `/task/data/corpus/`. Its `records.jsonl` contains `video_id`, `duration` in seconds, `feature_path` and `grid_path`. Paths are relative to that directory. Each feature row corresponds to one successive interval between adjacent time-grid endpoints. The video vectors were generated with CLIP ViT-B/32. The same fixed image and text encoder is available at `/task/models/clip/`; raw video files are not supplied. -The system must handle workloads of up to 500 requests in 2 calls of at most 250 requests. Each call has an independent 2,700-second allowance; time used by one call does not reduce another. Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. A failed query command other than a timeout invalidates that call. +Public training requests and interval labels are at `/task/data/training/queries.jsonl` and `/task/data/training/labels.jsonl`. Five validation requests and their labels are at the corresponding paths under `/task/data/validation/`. Labels contain `query_id`, `video_id` and `windows`, a list of acceptable start/end pairs. The remaining requests and labels are not provided. You may use the public examples to implement and check the complete build and query path during development. Include all required implementation files under `/app` so the system runs after a fresh start with the supplied inputs. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Let x be the best IoU between the predicted interval and the acceptable reference intervals. Its utility is u(x) = 0.7 * mean(I[x >= 0.5], I[x >= 0.7], I[x >= 0.9]) + 0.3 * x. Divide that utility by the best utility attainable with one legal interval on the supplied time grid. Average within each source, then equally across sources. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. + +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions. diff --git a/tasks/visual-document-qa/raw-instruction.md b/tasks/visual-document-qa/raw-instruction.md index 8dbe8e3..7f72569 100644 --- a/tasks/visual-document-qa/raw-instruction.md +++ b/tasks/visual-document-qa/raw-instruction.md @@ -1,30 +1,92 @@ -# Task: Visual-Document Question Answering +# Visual-Document Question Answering ## Task Description Build a system that answers questions using the supplied long documents. Relevant information may occur in body text, tables, charts or several pages. If the document does not contain the requested information, say so. -## Inputs and Available Validation Data +## Requirements -The collection is at `/task/data/corpus/`. Its `records.jsonl` contains `doc_id`, `pdf_path`, `text_path` and `page_count`. Paths are relative to that directory. The PDFs are the original documents; the text files contain uniformly extracted text for each page. You may render and read the PDFs directly. +### Resource limits and workload -Public requests and reference answers are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. The remaining questions are not provided. The original corpus and public examples are read-only. +- The environment provides 8 CPU cores, 32 GiB RAM and 100 GiB storage, with no GPU. +- Development has a two-hour budget. +- Index construction may take up to four hours. +- The system must handle up to 2,500 requests in two calls of at most 1,250 requests, with an independent 10,800-second allowance per call. +- Completed valid JSONL rows remain usable after timeout; missing or invalid rows are unsuccessful. +- A command failure other than timeout invalidates that call. +- Honor the supplied index and output paths. + +### Execution and file permissions -## Expected Artifacts and Interfaces +- During verification, build and query commands run as a non-root user. +- The implementation under `/app` and the supplied inputs are read-only. +- The passed index directory already exists and is writable, but its parent directory is not writable. +- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. +- The index persists and remains writable across query calls. +- Write results at the passed output path, whose parent directory is writable. +- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. + +### Build interface Create executable `/app/build.sh` and `/app/run.sh`, with the code and files they need under `/app`: -`/app/build.sh --corpus /task/data/corpus --index-dir /work/index` `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/answers.jsonl` -Each input JSONL row contains `query_id`, `doc_id` and `question`. Return one JSONL row for each query, with the same `query_id`, an `answer` string and a `pages` list of the supporting document's page numbers. Page numbers start at one. Give the final answer directly, including units where applicable; a list answer may be expressed as a list in the string. Use an empty `pages` list when no supporting page exists. Write diagnostics to stderr. +```bash +/app/build.sh \ + --corpus /task/data/corpus \ + --index-dir /work/index +``` + +### Query interface -## Resources and Execution Permissions +```bash +/app/run.sh \ + --index-dir /work/index \ + --queries /work/input/queries.jsonl \ + --output /work/output/answers.jsonl +``` -The environment provides 8 CPU cores, 32 GiB RAM and 100 GiB storage, with no GPU. Development has a two-hour budget. Index construction may take up to four hours. The system must handle up to 2,500 requests in two calls of at most 1,250 requests, with an independent 10,800-second allowance per call. Completed valid JSONL rows remain usable after timeout; missing or invalid rows are unsuccessful. A command failure other than timeout invalidates that call. Honor the supplied index and output paths. +### Input and output contract -During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. +Each input JSONL row contains `query_id`, `doc_id` and `question`. Return one JSONL row for each query, with the same `query_id`, an `answer` string and a `pages` list of the supporting document's page numbers. Page numbers start at one. Give the final answer directly, including units where applicable; a list answer may be expressed as a list in the string. Use an empty `pages` list when no supporting page exists. Write diagnostics to stderr. + +## Available Validation Data -## Environment and Development Checks +The collection is at `/task/data/corpus/`. Its `records.jsonl` contains `doc_id`, `pdf_path`, `text_path` and `page_count`. Paths are relative to that directory. The PDFs are the original documents; the text files contain uniformly extracted text for each page. You may render and read the PDFs directly. -Use only the external resources described in `/task/docs/available_resources.md`. Before starting, also read `/task/docs/environment.md`. Evaluation checks executable behavior, output validity and answers on held-out questions, together with the task's resource and API rules. It runs in a separate environment with the submitted `/app` files and the supplied corpus. +Public requests and reference answers are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. The remaining questions are not provided. The original corpus and public examples are read-only. You may use the public examples to implement and check the complete build and query path during development. Include all required implementation files under `/app` so the system runs after a fresh start with the supplied inputs. + +## Expected Artifacts + +Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`. + +```text +/app/ +├── build.sh +└── run.sh +``` + +## Verification + +After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items: + +1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers. +2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls. +3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above. +4. **Search quality.** Raw quality measures answer correctness against the reference answers, including numeric, list and unanswerable questions. The answer-comparison rules and grouping used for the final average are defined in [README.md — Search Quality](README.md#search-quality). Evidence-page precision, recall and F1 are reported separately and do not affect the answer score. + +The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure. + +## Hidden Test Overview + +The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above. + +## Environment and available resources + +Before implementing the system, read the following task-provided documents: + +- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools. +- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling. + +These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions.