Speed up CI: split unit and integration tests, shard simulations, add caching #12

Why

The Tests job in .github/workflows/ci.yml takes about 1h15m to 1h30m on every run, including PRs that don't touch simulation code.

Past develop runs:

PR #10 only adds a JSON-parsing unit spec, and its run was still going after 1h20m. This slows review, delays merges, and wastes runner time.

Cause

• spec/tests/integration/write_and_run_osws_spec.rb writes and runs a full baseline workflow for about 20 BuildingSync fixtures, one after another. Each run builds a model with openstudio-standards, sizes it, and runs a full-year EnergyPlus simulation.
• All specs run as one job (bundle exec rspec), so every PR waits for every simulation.
• Setup repeats work on each run: bundle install has no cache, and rake measures:install_external re-clones comstock at main.
• There is no concurrency: group, so pushing again to a PR doesn't cancel the run already in progress.
• Several fixtures overlap and probably add little coverage:
  • Example_ASHRAE_L2_Report_-_with_Energy_Use_Data and its _2 variant
  • L100_Audit-1.0.0 and L100_Audit-1.0.0_and_BSyncr-1.0.0
  • the two Example_NYC_Energy_Efficiency_Report_Property files

Proposed changes

1. Split the workflow into two jobs.
   • unit runs bundle exec rspec spec/tests/unit on every PR.
   • integration runs spec/tests/integration on pushes to develop, on a nightly schedule, and on manual dispatch. PRs can opt in with a label such as run-integration.
2. Run a smoke subset on PRs. Run 2-3 fixtures (for example building_151.xml and example-smalloffice-level1.xml) so PRs still get a real write-and-run check.
3. Shard the full integration run. Use a job matrix of 4-5 shards. The spec filters its fixture list by BOSS_SHARD and BOSS_SHARDS, so each job runs a share of the files in parallel.
4. Cache dependencies.
   • Cache .bundle/install, keyed on Gemfile.lock.
   • Cache the external measure checkout, keyed on the pinned ref.
   • Pin comstock to a tag or commit instead of main for reproducibility.
5. Cancel stale runs. Add concurrency: { group: ci-${{ github.ref }}, cancel-in-progress: true }.
6. Remove redundant fixtures from the simulation list, or keep them as parse-only checks in the unit specs.
7. Report the slowest tests. Run rspec with --profile 10 so test duration is visible in CI logs.

Sketch

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

jobs:
  unit:
    # same container + setup
    steps:
      - run: bundle exec rspec spec/tests/unit
  integration:
    if: github.event_name == 'push' || contains(github.event.pull_request.labels.*.name, 'run-integration')
    strategy:
      matrix:
        shard: [0, 1, 2, 3]
    env:
      BOSS_SHARD: ${{ matrix.shard }}
      BOSS_SHARDS: 4
    steps:
      - run: bundle exec rspec spec/tests/integration

Acceptance criteria

• PR feedback from the unit job (plus the smoke subset, if kept on PRs) arrives in under 10 minutes.
• The full integration suite finishes in under 25 minutes on develop and nightly.
• Total fixture coverage on develop is unchanged, apart from agreed removals.
• A new push to a PR cancels the previous run.
• The README testing section documents how to run unit, smoke and full integration locally.

Out of scope

• Changing what the integration tests assert.
• Changing simulation or baseline workflow behavior.
• Fixing the known BuildingEQ-1.0.0_gemtest.xml sizing failure, which is already commented out in the spec.

Notes

• Branch protection: if Tests is a required check, update it to the new job names (unit, and integration if it should be required).
• If the nightly schedule is used, make sure failures notify someone, so full-suite regressions aren't found late.