[hanhainebula]

Summary

Replace the lengthy manual evaluation setup with a search-swe CLI that guides users through model configuration, credentials, task selection and execution. Add an installable skill so users can also ask their coding agent to configure and run evaluations. Automate bounded-storage preparation for two tasks and remove two tasks with hardware-dependent requirements, leaving 47 tasks.

Changes

• Add the Python package and console entry point, with dependencies declared in pyproject.toml. Support editable installation, wheel/sdist builds, and run, configure, list and doctor commands.
• Guide configuration through harness → provider → model → reasoning effort → credentials → tasks → execution settings. Search the OpenRouter model catalog by name or ID, with caching, offline support and manual model IDs. Mask credential input and optionally save configuration to a private .env file.
• Run multiple selected tasks with one model setting, sequentially or concurrently. Reuse asset verification and the Harbor backend, allocate CPU/GPU resources within each batch, and record per-task logs, output paths and a batch manifest. Provide noninteractive options and dry runs.
• Add skills/run-searchswe-evaluation, with installation instructions for npx skills add and gh skill install. Update English/Chinese READMEs, evaluation documentation, CI and contributor skills to use the new installation and execution workflow.
• Automate host storage preparation and cleanup for large-scale-formula-retrieval and budgeted-bimetric-retrieval, including host checks, CPU affinity and scoped privilege elevation.
• Remove deletion-resilient-vector-search and resource-bounded-multivector-search, and update the task inventory and related references.

Compatibility

For the two retained storage tasks, agent-visible instructions, verifier logic, assets, resource budgets and scoring remain unchanged. These setup changes do not require rerunning existing valid results. Their storage setup requires native Linux with local rootful Docker and FUSE support.

The package is prepared for future PyPI publication; current installation uses pip install -e . and an existing repository checkout. Resource allocation coordinates tasks within one CLI invocation.

Validation

• Targeted CLI, model catalog, storage, CPU/GPU selection, network and release-check suites passed; all 47 tasks passed launcher configuration preflights.
• Real terminal checks covered model search, hidden credential entry, configuration saving, cancellation and offline fallback.
• Small Docker/FUSE probes verified both storage tasks' setup and cleanup; concurrent Docker probes verified disjoint CPU allocation and preserved resource limits.
• Editable installation, wheel/sdist builds, wheel execution outside the checkout, and local skill installation through both installers passed.
• Release validation reported 0 errors and 0 warnings; git diff --check passed.

Validation used configuration checks and infrastructure smoke tests. No paid model inference or complete benchmark evaluation was run.