Skip to content

Align raw task instructions with reference format #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hanhainebula merged 1 commit into from
Oct 23, 2026
Merged

Align raw task instructions with reference format #26

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 87 additions & 10 deletions tasks/composed-image-search/raw-instruction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,102 @@
# Task: Composed Image Search
# Composed Image Search

## Task Description

A user starts with a photograph and describes a change they want, such as a different arrangement or a different quantity of an object. Build an image search system that finds matching photographs in the supplied gallery while retaining the parts of the reference that the request leaves unchanged.

## Inputs and Available Validation Data
## Requirements

### Resource limits and workload

- You have two hours, 8 CPU cores, 32 GiB RAM and 100 GiB working storage and one 40 GiB GPU.
- The build command has 3,600 seconds.
- The system must handle up to 4,400 requests in two calls of at most 2,200 requests, with an independent 14,400-second allowance for each call.
- At most 100 candidate images per request may undergo additional query-dependent visual inspection.
- Corpus-wide global-vector screening and gallery-only preprocessing are allowed.
- The local model and permitted APIs are available as described in `/task/docs/available_resources.md`; also read `/task/docs/environment.md`.

- Treat data, models and documentation as read-only.
- The submitted `/app` files must work in a fresh environment with the supplied inputs, using the passed index and output paths for generated files.
- Query calls may be repeated after a build.
- Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful.
- A command failure other than timeout invalidates that call.
- Aim to return the requested images near the start of each ranking across the supplied requests.

### Execution and file permissions

- During verification, build and query commands run as a non-root user.
- The implementation under `/app` and the supplied inputs are read-only.
- The passed index directory already exists and is writable, but its parent directory is not writable.
- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself.
- The index persists and remains writable across query calls.
- Write results at the passed output path, whose parent directory is writable.
- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory.

### Build interface

Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is:

```bash
/app/build.sh \
--corpus /task/data/corpus \
--index-dir /work/index
```

### Query interface

The query interface is:

```bash
/app/run.sh \
--index-dir /work/index \
--queries /work/input/queries.jsonl \
--output /work/output/results.jsonl
```

Honor the supplied paths.

### Input and output contract

Each input JSONL row contains `query_id`, `reference_image_id`, `modification` and `shared_concept`. The reference belongs to the gallery. `shared_concept` provides an additional description when available and can be empty; the reference and modification still define the request.

Return one JSONL row for each request with its unchanged `query_id` and a `results` list of at most 50 distinct gallery images. Each result contains `image_id` and a finite numeric `score`, ordered by nonincreasing score. Exclude the reference image. Write diagnostics to stderr.

## Available Validation Data

The gallery is at `/task/data/corpus/`. Each row of `records.jsonl` contains `image_id` and `image_path`; paths are relative to the corpus directory. Gallery images are supplied. Local image and text encoder weights are available at `/task/models/clip/`.

## Expected Artifacts and Interfaces
Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels map each `query_id` to `qrels`, which may contain several matching images. You may use these examples to develop and check the system.

## Expected Artifacts

Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`.

```text
/app/
├── build.sh
└── run.sh
```

## Verification

After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items:

1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers.
2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls.
3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above.
4. **Search quality.** Raw quality uses average precision and recall to measure the ranking and coverage of annotated matching images. The applicable cutoffs, query groups and equal-weight aggregation are defined in [README.md — Search Quality](README.md#search-quality).

Provide executable `/app/build.sh` and `/app/run.sh` with their supporting files under `/app`. The build interface is `/app/build.sh --corpus /task/data/corpus --index-dir /work/index`. The query interface is `/app/run.sh --index-dir /work/index --queries /work/input/queries.jsonl --output /work/output/results.jsonl`. Honor the supplied paths. Each input JSONL row contains `query_id`, `reference_image_id`, `modification` and `shared_concept`. The reference belongs to the gallery. `shared_concept` provides an additional description when available and can be empty; the reference and modification still define the request.
The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure.

Return one JSONL row for each request with its unchanged `query_id` and a `results` list of at most 50 distinct gallery images. Each result contains `image_id` and a finite numeric `score`, ordered by nonincreasing score. Exclude the reference image. Write diagnostics to stderr. Public requests and relevance labels are at `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Labels map each `query_id` to `qrels`, which may contain several matching images. You may use these examples to develop and check the system.
## Hidden Test Overview

## Resources and Workload
The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above.

You have two hours, 8 CPU cores, 32 GiB RAM and 100 GiB working storage and one 40 GiB GPU. The build command has 3,600 seconds. The system must handle up to 4,400 requests in two calls of at most 2,200 requests, with an independent 14,400-second allowance for each call. At most 100 candidate images per request may undergo additional query-dependent visual inspection. Corpus-wide global-vector screening and gallery-only preprocessing are allowed. The local model and permitted APIs are available as described in `/task/docs/available_resources.md`; also read `/task/docs/environment.md`.
## Environment and available resources

## Execution and File Permissions
Before implementing the system, read the following task-provided documents:

During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory.
- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools.
- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling.

Treat data, models and documentation as read-only. The submitted `/app` files must work in a fresh environment with the supplied inputs, using the passed index and output paths for generated files. Query calls may be repeated after a build. Completed valid JSONL rows remain usable if a call times out; missing or invalid rows are unsuccessful. A command failure other than timeout invalidates that call. Aim to return the requested images near the start of each ranking across the supplied requests.
These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions.
99 changes: 86 additions & 13 deletions tasks/document-page-search/raw-instruction.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,101 @@
# Task: Multilingual Visual Document Page Search
# Multilingual Visual Document Page Search

## Task Description

Reports in several languages contain tables, charts and dense page layouts. I need to find the pages that support a question so that I can open the original material and check it. Build a page search system over the supplied collection.
Reports in several languages contain tables, charts and dense page layouts. The system must find the pages that support a question so that the original material can be opened and checked. Build a page search system over the supplied collection.

## Document Collection
## Requirements

The collection is in `/task/data/corpus`. Each line of `records.jsonl` describes a page with `page_id`, `domain`, `doc_id`, `image_path`, `width`, `height`, `markdown` and `page_number_in_doc`. Image paths are relative to the collection directory. The Markdown is a supplied transcription; the original page images are also available. Questions may be in a different language from the document. Treat `/task` as read-only and place your implementation under `/app`.
### Resource limits and workload

- You have 8 CPU cores, 32 GiB RAM, 100 GiB storage, one 40 GiB GPU and two hours to implement and check the system.
- Local model weights and processors are at `/task/models/page-encoder/` and `/task/models/vision-backbone/`.
- Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed tools and permitted API resources.
- Use only those providers, endpoints and generative model IDs.
- Do not obtain evaluation questions, answers or relevance labels from outside the provided inputs, use external search services, or store API credentials in code, indexes or logs.

- The system must handle workloads of up to 14,500 requests in 2 calls of at most 7,250 requests.
- Each call has an independent 2,700-second allowance; time used by one call does not reduce another.
- Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful.
- A failed query command other than a timeout invalidates that call.

## Expected Artifacts and Interfaces
### Execution and file permissions

- The complete implementation under `/app` is transferred to a fresh environment.
- Your build command has 10,800 seconds.
- During verification, build and query commands run as a non-root user.
- The implementation under `/app` and the supplied inputs are read-only.
- The passed index directory already exists and is writable, but its parent directory is not writable.
- Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself.
- The index persists and remains writable across query calls.
- Write results at the passed output path, whose parent directory is writable.
- Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory.
- Include any additional dependencies needed at runtime under `/app`; development processes and packages installed elsewhere are not transferred.
- Aim to rank the relevant pages as highly as possible on new questions across all supplied domains and languages.

### Build interface

Provide executable `/app/build.sh` and `/app/run.sh`. The build command prepares the index and returns when it is ready:
`/app/build.sh --corpus /task/data/corpus --index-dir /work/index`

```bash
/app/build.sh \
--corpus /task/data/corpus \
--index-dir /work/index
```

### Query interface

The query command accepts a JSONL file and writes a JSONL result file:
`/app/run.sh --index-dir /work/index --queries /path/to/questions.jsonl --output /path/to/results.jsonl`
Honor the passed paths. Each question has `query_id`, `question`, `domain` and `language`. Search the complete page collection for that domain. For every input question, return one object with its unchanged `query_id` and `results`, containing up to ten distinct pages in relevance order. Each result contains `page_id` and a finite numeric `score`; scores must be nonincreasing. Return page IDs from the requested domain. No answer or bounding box is required.

```bash
/app/run.sh \
--index-dir /work/index \
--queries /path/to/questions.jsonl \
--output /path/to/results.jsonl
```

Honor the passed paths.

### Input and output contract

Each question has `query_id`, `question`, `domain` and `language`. Search the complete page collection for that domain. For every input question, return one object with its unchanged `query_id` and `results`, containing up to ten distinct pages in relevance order. Each result contains `page_id` and a finite numeric `score`; scores must be nonincreasing. Return page IDs from the requested domain. No answer or bounding box is required.

## Available Validation Data

The collection is in `/task/data/corpus`. Each line of `records.jsonl` describes a page with `page_id`, `domain`, `doc_id`, `image_path`, `width`, `height`, `markdown` and `page_number_in_doc`. Image paths are relative to the collection directory. The Markdown is a supplied transcription; the original page images are also available. Questions may be in a different language from the document. Treat `/task` as read-only and place your implementation under `/app`.

Public questions and their page-relevance annotations are in `/task/data/validation/queries.jsonl` and `/task/data/validation/labels.jsonl`. Use these for development and self-checking. Additional questions and annotations are withheld.

## Environment and Available Resources
## Expected Artifacts

Submit executable `/app/build.sh` and `/app/run.sh` together with every implementation file and learned parameter required at runtime under `/app`.

```text
/app/
├── build.sh
└── run.sh
```

## Verification

After development, Harbor transfers the declared artifacts to a separate verifier and evaluates the system on the held-out split. The verifier checks the following items:

1. **Task and resource compliance.** The submission must use the supplied inputs and permitted resources, follow stage isolation, and avoid hidden labels or hard-coded evaluation answers.
2. **Executable behavior.** The required commands must follow the interfaces, filesystem permissions and time limits above and support the specified repeated calls.
3. **Output validity.** Results must satisfy the task-specific schema, identifiers, bounds and ordering rules. The failure and timeout behavior follows the contracts above.
4. **Search quality.** Raw quality is nDCG@10 with linear relevance gains, averaged within each domain and then equally across the eight domains. Missing or invalid query results remain in the denominator.

The final reward applies the fixed task-specific piecewise score mapping documented in `README.md` to raw quality. Raw quality is also reported separately. A confirmed trajectory-audit violation sets the entire reward to zero; an incomplete audit or a failed trusted judge is an infrastructure failure.

## Hidden Test Overview

The held-out questions and reference labels concern the supplied collection and are disjoint from the public validation examples. They are not copied into the Agent-visible environment. Evaluation uses the same input and output contracts, resources and execution limits described above.

You have 8 CPU cores, 32 GiB RAM, 100 GiB storage, one 40 GiB GPU and two hours to implement and check the system. Local model weights and processors are at `/task/models/page-encoder/` and `/task/models/vision-backbone/`. Read `/task/docs/environment.md` and `/task/docs/available_resources.md` for the installed tools and permitted API resources. Use only those providers, endpoints and generative model IDs. Do not obtain evaluation questions, answers or relevance labels from outside the provided inputs, use external search services, or store API credentials in code, indexes or logs.
## Environment and available resources

## Execution, Permissions and Workload
Before implementing the system, read the following task-provided documents:

The complete implementation under `/app` is transferred to a fresh environment. Your build command has 10,800 seconds. During verification, build and query commands run as a non-root user. The implementation under `/app` and the supplied inputs are read-only. The passed index directory already exists and is writable, but its parent directory is not writable. Create build files, caches and temporary directories inside that index directory or the supplied `TMPDIR`; do not create sibling directories or rename the index directory itself. The index persists and remains writable across query calls. Write results at the passed output path, whose parent directory is writable. Honor `TMPDIR` and `XDG_CACHE_HOME` rather than writing to `/app` or assuming a writable system temporary directory. Include any additional dependencies needed at runtime under `/app`; development processes and packages installed elsewhere are not transferred. Aim to rank the relevant pages as highly as possible on new questions across all supplied domains and languages.
- [`/task/docs/environment.md`](/task/docs/environment.md) — installed runtime, Python packages and system tools.
- [`/task/docs/available_resources.md`](/task/docs/available_resources.md) — available local models or APIs, network restrictions and runtime credential handling.

The system must handle workloads of up to 14,500 requests in 2 calls of at most 7,250 requests. Each call has an independent 2,700-second allowance; time used by one call does not reduce another. Completed valid JSONL rows remain usable if a query call times out; missing or malformed rows are unsuccessful. A failed query command other than a timeout invalidates that call.
These documents and the supplied inputs are read-only. Follow the task-specific resource restrictions.
Loading
Loading