Skip to content

Add parallel live simulation coverage - #3820

Draft
anth-volk wants to merge 3 commits into
masterfrom
feat/2400-parallel-live-simulations
Draft

Add parallel live simulation coverage#3820
anth-volk wants to merge 3 commits into
masterfrom
feat/2400-parallel-live-simulations

Conversation

@anth-volk

@anth-volk anth-volk commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Fixes #2400

Summary

  • keep candidate health, metadata, and the stateful v2 policy lifecycle checks in the serial staging integration job
  • run the non-stateful live test selection inside one GitHub Actions job using two pytest-xdist worker processes with dynamic per-test scheduling
  • limit concurrency explicitly with -n 2, use --dist load, and schedule one test at a time with --maxschedchunk=1
  • use the exact tagged candidate URL and collision-resistant probe identifiers containing the deployment tag, workflow run, run attempt, and a generated worker-session UUID
  • add fixed 2025 household assertions for federal EITC, federal CTC, CalEITC, and Universal Credit for a household in Scotland
  • add society-wide California CalEITC and Scotland Universal Credit reforms while retaining the existing Utah and budget-window coverage
  • remove the obsolete Enhanced CPS configuration requirement from the linked issue; society-wide requests use the currently certified default dataset
  • require both the serial staging checks and the pytest-xdist live-test job to pass before staging promotion

Validation

  • make format
  • Ruff checks on all changed Python files
  • pytest tests/unit/test_cloud_run_deploy_scripts.py -q (102 passed)
  • executed the live-test runner locally with pytest-xdist: 2 workers distributed all 11 tests; all skipped because no deployed API URL was supplied
  • validated .github/workflows/push.yml with actionlint
  • validated the live-test runner with bash -n
  • parsed all new JSON fixtures with jq
  • resolved both reform parameter paths against the pinned US and UK packages
  • calculated the expected household results locally with the pinned US and UK packages
  • ran the changelog-fragment and diff checks

The deployed live checks were not run locally because they require a newly deployed tagged staging candidate.

@codecov

codecov Bot commented Sep 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.18%. Comparing base (37e168a) to head (4b44816).

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #3820       +/-   ##
===========================================
+ Coverage   46.26%   87.18%   +40.92%     
===========================================
  Files         135      150       +15     
  Lines        7963     8935      +972     
  Branches     1326     1542      +216     
===========================================
+ Hits         3684     7790     +4106     
+ Misses       3990      703     -3287     
- Partials      289      442      +153     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk

Copy link
Copy Markdown
Collaborator Author

Deployment validation for this PR currently depends on resolving the pre-existing failure in tests/integration/test_live_phase10_staging.py::test_live_phase10_activation_failure_and_retry on master (failed workflow run). Anthony is working on that failure; this PR should remain blocked until the existing deployment sequence completes successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expand live simulation integration coverage across US and UK

1 participant