[chiliec]

consensus_sim is one Boost.Test binary that CI runs serially on one core of a 4-core runner. It's the longest step in two required jobs:

job	serial consensus_sim
consensus_sim + pm unit tests	~15.5 min (933 s on #200)
Testnet config + consensus_sim (1 validator)	~19 min (1141–1167 s)

What changes

• New tests/consensus_sim/run_parallel.py: lists enabled cases via --list_content, runs each as its own process (--run_test=suite/case), -j N at a time. On failure it prints that case's full output; it always prints a slowest-cases table; it exits non-zero if any case failed. --filter limits to given top-level suites; extra args after -- are passed to every run.
• tests.yml: both jobs call it with -j$(nproc) from build/tests/consensus_sim, the directory ctest used, so failures/ logs and the upload-artifact path stay the same. The 2-validator arm keeps its three suites via --filter. The other ctest suites (pm_, node_admin_, …) still run through ctest -E '^consensus_sim$'.
• README: how to run it locally.

Why it's safe to split. Every simulated_node opens its own unique_path temp dir (64 MB shared file), failure logs are named <seed>-<scenario>.log, and there's no global fixture or shared mutable state across cases. Same set of cases runs as before — disabled cases stay skipped.

Verified locally. The --list_content parser was checked against real output from Boost 1.83 (the Ubuntu floor) and 1.92 using a test binary with nested suites, a fixture suite and a disabled case: the right leaves are found, disabled ones skipped, --filter and pass-through args work, a failing case prints its output and gives exit 1. I couldn't run the real ASan-instrumented consensus_sim_tests here (the box has a 2 GB memory cap), so this PR's CI run is the real check.

Expected effect. Up to ~4× on the simulator step, bounded by the slowest single case — the table at the end of the log will show it. If one case dominates (e.g. a big pm_lifecycle_suite scenario), that's the next thing to look at. Independent of #202 (ccache); together they cut both compile and test time.

[On1x]

Reviewed the driver against the harness — the parallelism premise holds:

• cases are isolated on disk: simulated_node.cpp:16 builds each node's data/shm dirs with bfs::unique_path("consensus_sim-<label>-%%%%%%%%"), so concurrent processes cannot share a chainbase;
• failure artifacts do not collide either: harness/failure_log.cpp:13 writes <cwd>/failures/<seed>-<scenario>.log, which is unique per case;
• the 2-validator filter keeps exactly the three suites that arm ran before, and ASAN_OPTIONS/UBSAN_OPTIONS are set at step level so the children inherit them;
• rc aggregation is correct: any non-zero case exits 1.

Two things I'd want before merging:

1. A count cross-check. The whole run now rests on parsing --list_content. If the parse silently returns a subset (different Boost indentation, a suite header that looks like a leaf), CI goes green having run fewer cases — a shrinking safety net that looks exactly like a healthy one. Cheap fix: compare len(cases) against an independent count (e.g. leaves found by a --run_test=* dry run, or simply assert len(cases) >= a floor recorded in the README), and print skipped (disabled): N explicitly instead of dropping them silently.

2. Rebase after #202. ci: make ccache actually persist landed in master (022dd398) and edits the same two workflow files, so the combination of the two has never been built. Your current verdict describes a tests.yml that no longer exists — please rebase and let CI run again; then I'll merge.

Not blocking the approach: the ~15 min serial binary on one core while 3 cores idle is real waste, and the driver is the right shape.

[chiliec]

Thanks, both done. I rebased on master (now includes #202's 022dd398) and pushed three commits on top:

1. Count cross-check. The case list now comes from --list_content=DOT, where test cases are identified by id (the TUT_CASE bit), not layout. It has to match the indented listing exactly, or the run aborts. On top of that, --min-cases sets a hard floor (117 regular / 172 testnet, checked before --filter), disabled cases are printed as skipped (disabled): N with their names, an unknown --filter suite is an error, and a --run_test that matches nothing counts as a failure instead of exit 0.
2. The real bottleneck. The last run showed equivocation_suite/seed_sweep_one_hundred_all_fire alone taking 822–927 s of ~930 s wall. I split it into 10 data-driven chunks of 10 seeds. Same seeds and assertions, and a static_assert checks that the chunks cover all 100.
3. concurrency / cancel-in-progress for PR runs of tests.yml and docker-pr-build.yml. Master pushes are never cancelled.

I moved the PR to draft until this CI run is green. Details and local checks are in the description.

[On1x]

Re-reviewed on the new head ea188e08 (rebased on 022dd398). Both blockers from my earlier comment are resolved, and I checked the claims against the run rather than the description:

Case-count cross-check. run_parallel: 117 cases, 4 at a time; skipped (disabled): 0 in the log, and --min-cases 117 passes — the floor and the measured count agree, so the number is not a copy that can rot. The DOT listing is built from the TUT_CASE bit and its path from the edge list, which is layout-independent, and a disagreement with the indented listing aborts instead of running a subset. --run_test matching nothing is now a failure rather than a Boost-version-dependent pass — that closes the "green over an empty set" hole.

Parallelism safety, verified beyond the two properties I checked earlier:

• per-node data dirs stay unique (bfs::unique_path in simulated_node.cpp), failure logs are <seed>-<scenario>.log and the sweep split keeps every seed in exactly one chunk (static_assert, seeds 0..99 partitioned into 10×10), so no two concurrent cases can write the same artifact;
• no case binds a socket — the endpoint/p2p cases are string predicates over host:port, not listeners, so -j4 cannot collide on a port;
• ctest still owns the other suites (-E '^consensus_sim$'), and the driver runs from build/tests/consensus_sim, the directory ctest used.

Measured effect. wall 250s, sum of cases 920s (3.7× on 4 cores) vs 1070 s for the serial consensus_sim on my #205 run — about 14 minutes off the longest step. The 2-validator arm: 11 cases after --filter, --min-cases 172 checked before the filter.

Concurrency block. Correct on both paths: pull_request groups by PR number and cancels the superseded run; push groups by run_id so master runs are never cancelled. tests.yml does not listen to labeled, so a label that re-triggers the docker workflow cannot cancel the test run.

One follow-up, pre-existing and not a blocker: the artifact step uploads tests/consensus_sim/failures/ — that path is relative to the workspace root, i.e. the SOURCE tree, where it holds only .gitkeep; the logs are written under build/tests/consensus_sim/failures/. With if-no-files-found: ignore the artifact is empty and silent. Since this PR is what makes the directory choice explicit, it is the natural place for the one-line fix (path: build/tests/consensus_sim/failures/); otherwise it is worth an issue of its own — a failure artifact nobody can open turns "logs uploaded" into a claim.

Approved from my side. Merge is the owner's, per the repo policy for public VIZ repositories.

[chiliec]

CI is green on the rebased branch (with #202), so this is ready again.

• consensus_sim + pm: 117/117 cases, simulator step 250 s wall (was ~930 s serial). Slowest case is now a sweep chunk at ~82 s.
• Testnet (1 validator): 172/172 cases, 280 s (was ~1150 s). Job total 15m39s, down from ~30 min.
• Testnet (2 validators): 11/11, job 1m32s with ccache restored (49% hits on the first warm run).
• No disabled cases skipped; both listings agreed on every run.

[chiliec]

Good catch, thanks. Fixed in 94e0b108: the upload step now uses build/tests/consensus_sim/failures/.

I checked the path in the harness: failure_log.cpp writes to bfs::current_path() / "failures", and both jobs run the binary from build/tests/consensus_sim. The source-tree directory only holds .gitkeep, so the artifact used to be empty.

[chiliec]

Rebase-free follow-up after #205 landed: it adds db_invariants_suite/healthy_chain_reconciles_and_divergence_is_detected, built in both configurations, so I raised the --min-cases floors 117 → 118 and 172 → 173 (67e5e7e8). PR CI runs on the merge with master, so this run already counts the new case.