Skip to content

[feat][evaluation] retry completed items within the current experiment run - #699

Open
xueyizheng wants to merge 31 commits into
mainfrom
feat/experiment-running-retry
Open

xueyizheng wants to merge 31 commits into
mainfrom
feat/experiment-running-retry

Conversation

@xueyizheng

@xueyizheng xueyizheng commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

feat

Check the PR title

  • The title follows the [<type>][<scope>] <description> format.
  • The title describes the user-visible behavior.
  • This PR is written in English.

More detailed description

Allow users to retry terminal items while an offline experiment is still running. Selected items are requeued in the active run, while unfinished items continue executing. Retrying an already finished experiment preserves the existing behavior.

  • Extend RetryExperiment with optional current-run and request identities and a per-item admission receipt.
  • Update item/turn results, run logs, references, statistics, and the receipt in a transaction. Failure retries reuse successful stages; explicit item retries rerun the selected items.
  • Propagate item execution identity through dispatch, callbacks, result publication, and completion checks for both legacy and centralized scheduling. Stale work cannot overwrite a retried item or finish the run prematurely.
  • Run the database and lease regressions in CI with disposable MySQL 8.0 and Redis 8.2 services. Apply the existing schema and configure the integration-test environment variables.

Additional changes included in this branch: preserve canonical multi-turn dataset rows and target history when resuming execution, and add ARM Mac sandbox-mode recognition with explicit rejection of unsupported Windows execution. The topology additions are tracked separately in the scope-review notes.

Validation:

  • On source commit 43010eea3bd2, all 17 CI checks passed. Patch coverage was 82.22%, and evaluation-domain patch coverage was 81.94%, against an 80% target. The CI run included the real database fixtures.
  • Earlier end-to-end acceptance covered 27 cases, 17 database verification chains, and 53 distinct retry/state operations. This evidence is retained; the entire matrix was not rerun after the final rebase. Targeted regressions for intersecting upstream changes and builds passed.
  • A fresh integration smoke test with the deployed frontend and commercial backend retried one successful item in a running 10-item experiment: Success → Queueing → Processing → Success. The run ID stayed unchanged, the receipt accepted one item with no rejections, and other items were not reset. No API responses or UI states were mocked in that smoke test.

Two independent code reviews and a follow-up verification covered 226 non-generated files across the related repositories. No P0/P1 defect was confirmed. The review records separate follow-ups for specialized database-test CI coverage and the cost of per-item quota completion transactions. This PR has not been merged or released to production.

@xueyizheng
xueyizheng force-pushed the feat/experiment-running-retry branch from 37c6950 to 384397a Compare October 22, 2026 12:15
@codecov

codecov Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.22123% with 397 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...luation/domain/service/expt_run_item_event_impl.go 64.91% 60 Missing and 40 partials ⚠️
...ation/domain/service/expt_manage_execution_impl.go 52.20% 49 Missing and 16 partials ⚠️
...dules/evaluation/domain/service/expt_run_finish.go 57.57% 31 Missing and 25 partials ⚠️
...valuation/domain/service/expt_manual_retry_impl.go 91.16% 19 Missing and 15 partials ⚠️
...ation/domain/service/expt_current_result_source.go 91.64% 17 Missing and 15 partials ⚠️
...aluation/domain/service/expt_result_filter_sync.go 89.95% 12 Missing and 10 partials ⚠️
...ules/evaluation/domain/entity/expt_manual_retry.go 86.91% 7 Missing and 7 partials ⚠️
...es/evaluation/domain/service/expt_run_item_impl.go 84.61% 7 Missing and 7 partials ⚠️
...ules/evaluation/domain/service/expt_result_impl.go 72.91% 9 Missing and 4 partials ⚠️
...luation/domain/entity/expt_item_current_request.go 66.66% 8 Missing and 4 partials ⚠️
... and 8 more

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #699      +/-   ##
==========================================
+ Coverage   78.98%   79.39%   +0.40%     
==========================================
  Files         707      721      +14     
  Lines       88191    90982    +2791     
==========================================
+ Hits        69657    72234    +2577     
- Misses      14553    14580      +27     
- Partials     3981     4168     +187     
Flag Coverage Δ
unittests 79.39% <82.22%> (+0.40%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...valuation/application/convertor/experiment/expt.go 83.78% <100.00%> (+0.40%) ⬆️
...modules/evaluation/application/eval_openapi_app.go 83.68% <100.00%> (+0.11%) ⬆️
backend/modules/evaluation/domain/entity/expt.go 85.50% <ø> (ø)
...nd/modules/evaluation/domain/entity/expt_result.go 71.33% <100.00%> (+1.53%) ⬆️
...ckend/modules/evaluation/domain/entity/expt_run.go 98.86% <100.00%> (+0.06%) ⬆️
...tion/domain/entity/target_builtin_sandbox_agent.go 100.00% <100.00%> (ø)
...dules/evaluation/domain/repo/expt_item_dispatch.go 100.00% <ø> (ø)
...odules/evaluation/domain/service/evaluator_impl.go 86.95% <100.00%> (+0.02%) ⬆️
...ules/evaluation/domain/service/expt_manage_impl.go 67.62% <100.00%> (+0.67%) ⬆️
...on/domain/service/expt_run_scheduler_event_impl.go 82.44% <ø> (+0.52%) ⬆️
... and 31 more

... and 9 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e913af8...43010ee. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xueyizheng
xueyizheng force-pushed the feat/experiment-running-retry branch from 4aff09a to db733db Compare October 26, 2026 17:39
xueyizheng and others added 27 commits October 28, 2026 12:07
Use the frozen experiment binding for single-set retries and preserve unknown-run receipt replay semantics.
@xueyizheng
xueyizheng force-pushed the feat/experiment-running-retry branch from 8583995 to 43010ee Compare October 28, 2026 04:26
@xueyizheng
xueyizheng marked this pull request as ready for review October 28, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants