Skip to content

feat(run): add --parallel flag to run tests across CPU cores - #17

Open
vladdoster wants to merge 39 commits into
masterfrom
feat/run-tests-in-parallel-support
Open

feat(run): add --parallel flag to run tests across CPU cores#17
vladdoster wants to merge 39 commits into
masterfrom
feat/run-tests-in-parallel-support

Conversation

@vladdoster

@vladdoster vladdoster commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

Parallel test execution — new src/parallel.zsh. -p/--parallel (or parallel: true in .zunit.yml) spreads test files, or slices of a single file, across worker subshells. Workers record results rather than printing them, and the parent replays them in serial order, so output, TAP and report files stay byte for byte identical to a serial run. --fail-fast stops the other workers, and a crashed worker is reported as an error instead of vanishing.

Progress bar — drawn during parallel runs when stderr is a terminal and TAP was not requested. Follows a resize, never fills the last column, restores terminal echo and the cursor on exit, and is suppressed by --no-progress or progress: false.

--verbose output framing — captured output was printed raw, flush left, and above the result line it belonged to, so a test which runs zunit emitted a results table indistinguishable from the outer run's. It is now indented beneath its result, and marked as a TAP diagnostic under --tap.

zunit init — the generated .zunit.yml carried the source file's indentation, so every key parsed one level too deep and none of the names the runner reads were ever set. Templates dedented, and the keys added since they were last touched (parallel, progress, verbose, revolver) included.

Completion and docs — _zunit was missing --revolver, declared --time-limit as a valueless flag, and passed -A without its pattern argument, silently swallowing the following -h/--help spec. README now documents both commands, every option, the three test-argument forms and the .zunit.yml keys.

vladdoster added 30 commits July 18, 2026 01:57
@vladdoster
feat(run): add --parallel flag to run tests in parallel
e9eb070 
Add a -p/--parallel option (and 'parallel: true' config key) to
'zunit run' that distributes work across CPU cores:

- multiple test files run concurrently, one worker per file
- a single test file is split into evenly sized contiguous groups,
  one worker per core

Workers record result events and the parent replays them through the
existing event handlers in serial order, so output, numbering, TAP,
text/HTML reports, fail-fast behaviour and exit codes are identical
to a serial run. Crashed workers are reported as errors and fail the
run.

Also fix two latent issues surfaced by worker isolation:

- testdir in _zunit_run_testfile was expanded from the caller's
  variable of the same name; compute it after testfile is assigned
- extract the verbose-output echo into _zunit_verbose_output so it
  can be recorded as an ordered event
@vladdoster
fix(parallel): harden parallel runner based on review findings
c8b2011 
Correctness:

- wait only for tracked worker PIDs, so background processes left by
  a bootstrap script no longer hang the run
- make --fail-fast stop remaining parallel work via an abort sentinel
  checked by the spawn loop and by workers between tests
- fail the run when file-level errors were reported (e.g. an
  unparseable @setup), in serial mode as well
- detect truncated or empty worker state files with an end-of-file
  marker and report them as crashed workers
- report setup/teardown parse errors once instead of once per core in
  single-file mode
- count a crashed worker as an errored test so TAP numbering, the
  plan line and the summary stay consistent
- quote recorded event fields so payloads containing the 0x1f
  separator byte survive the replay round-trip

Robustness and efficiency:

- abort with a clear error when mktemp fails instead of emitting a
  bogus crash report per core
- exit through the EXIT trap on INT/TERM/HUP so the temporary
  directory is not leaked when the run is interrupted
- track worker liveness via the shell's job table instead of kill -0,
  avoiding PID-reuse races, and poll with the zselect builtin instead
  of forking sleep
- bound single-file fan-out by the number of tests in the file
- detect CPU cores by running the commands directly: the first access
  to zsh's $commands hash scans every $PATH directory, which costs
  hundreds of milliseconds on hosts with slow filesystem mounts

Tests:

- assert known content markers before the checksum parity comparison
  so the tests cannot pass vacuously
- prove config-enabled parallel mode actually spawns workers using
  the crash fixture
- cover fail-fast side-effect prevention with a marker-file fixture
- deduplicate the file header into _zunit_testfile_header
@vladdoster
fix(run): quote basename argument when skipping underscore-prefixed t…
53bef55 
…ests
@vladdoster
chore: bump version to 0.11.0
e02c5e7
@vladdoster
refactor(run): use zsh :t modifier instead of basename when skipping …
e918bc8 
…underscore-prefixed tests
@vladdoster
feat(parallel): draw a progress bar during parallel runs
eea11ca 
Parallel runs buffer each worker's output and replay it once every
worker has finished, so a long run gave no feedback at all until the
very end. Workers now append a single character per completed test to
a shared file, and the parent redraws a bar from those ticks while it
waits.

Single byte appends are atomic, so the workers need no locking between
them. The bar is a terminal affordance only: it is skipped when stderr
is not a terminal, so piped output and report files stay byte for byte
identical to a serial run, and it is skipped under --tap, which is a
machine readable format.

It can also be turned off explicitly with --no-progress, or with
'progress: false' in .zunit.yml.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(parallel): cover progress bar output on a terminal
4ab8248 
The bar is only drawn when stderr is a terminal, so these cases cannot
be exercised by capturing output the usual way. A zpty helper runs the
binary on a pseudo terminal instead; zpty is used rather than script(1),
whose options differ between platforms.

Covers the bar being drawn with the expected test count, --no-progress
suppressing it, and piped output staying free of it.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.12.0
06a69c9 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: ignore the Makefile generated by configure
b2b352d 
./configure writes Makefile from Makefile.in with the local prefix
baked in, and 'make clean' deletes it again, so it is a build artifact
rather than something to track. Makefile.in stays checked in.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(parallel): keep the progress bar on a single line
d11f632 
The bar sized itself from $COLUMNS, which a non interactive shell
never refreshes during a run, so a window resized mid run left it
drawing a line wider than the terminal. Such a line wraps, a wrapped
line cannot be erased with a single escape sequence, and every redraw
after it stranded another copy on the screen. The width is now
measured from the terminal itself, before anything is put on it, and
the line is never allowed to reach the final column.

Terminal echo is turned off while the bar is up, so that a newline
typed during a run no longer scrolls the bar away and leaves a stale
copy behind, and the bar is only redrawn when it has something new to
say. The terminal mode and the cursor are restored on every exit
path, including an interrupted run.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(parallel): make the crash fixture assert something real
68a178d 
The fixture ended in `assert 1 equals 1`, which could never fail and
never reported anything either: the test body runs in a fork of the
worker, so after it SIGKILLs its parent the orphaned process writes
its verdict into a dead pipe, and the worker never gets to record it.

Assert the two preconditions before the kill instead, while the worker
is still alive to report a failure - the fixture is only meaningful in
a parallel worker with no time limit, since a serial run would kill the
runner itself and a time limit puts the async wrapper in between. The
trailing `fail` is only reachable when the kill missed.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(verbose): indent captured output beneath its result
bf3fd8b 
Verbose output was printed raw and flush left, and before the result
line it belonged to. A test which runs zunit itself therefore emitted a
whole results table of its own, indistinguishable from the outer run's,
sitting above the name of the test which produced it.

The result is now printed first, and each line of captured output is
indented by two spaces, matching the indent already used for error and
failure detail. Under --tap the lines are marked as diagnostics instead,
so a verbose run still produces a parseable stream. print -r replaces
echo, which interpreted backslash escapes in the captured output.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(init): write a config file the parser can read
f72596f 
The four templates were zsh string literals whose continuation lines
carried the indentation of the source file, and that indentation was
written into the generated files verbatim. For .zunit.yml this was
fatal: _zunit_parse_yaml derives the nesting of a key from its indent,
so every key landed two levels below the first one, yielding names such
as zunit_config_tap__tests where the runner reads
zunit_config_directories_tests. A generated config set none of the
variables it was meant to.

Dedent all four templates, and add the keys which have been added to the
runner since the config template was last touched - parallel, progress,
verbose and revolver.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
docs: describe the CLI, and complete every flag it accepts
dcefe6d 
The completion had drifted from the runner. --revolver was missing
entirely; --time-limit was declared as a plain flag, so no argument was
offered after it; and -v/--version was absent from both subcommands
although the top level zparseopts strips it before dispatch.

Three calls also passed -A without the pattern it takes, which quietly
consumed the following -h/--help spec as that pattern. -A is wrong here
in any case, since options may be given either side of the test paths,
so it has been dropped rather than corrected. The specs now live in two
arrays shared by the top level and per command calls, -C is passed so
that curcontext is honoured, -s allows single letter options to be
stacked, and test paths complete *.zunit rather than every file.

The README documented no CLI surface at all, deferring to zunit.xyz,
which predates this fork and covers neither --parallel nor
--no-progress. It now lists both commands, every option, the three forms
a test argument can take, the .zunit.yml keys and the exit codes.

tests/completion.zunit derives the option set from the usage text and
compares it against the completion in both directions, so the two cannot
drift apart again unnoticed.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.12.1
b9b7822 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
feat(parallel): source the bootstrap script in each worker
aa16396 
The bootstrap script used to be sourced once, into the parent shell,
before the workers forked. Every worker therefore inherited one shared
copy of whatever the script built - for a bootstrap which prepares an
on-disk environment, all workers raced each other inside the same
directories, and test files which pass in isolation failed under
--parallel.

Each worker now sources the script itself, after its recorders are
installed and before it runs its share of the tests, so every worker
builds its own environment, exactly as a standalone run of its test
file would. The script is sourced at the subshell's top level rather
than inside a helper function, because source runs a script in the
enclosing function scope - a bare typeset in the script would become a
local of the helper and be gone before the tests run.

A bootstrap which returns non-zero is recorded as an error, with the
worker's log attached, and the worker runs no tests; one which exits
outright is reported by the existing crashed-worker path. Serial runs
still source the script once, into the runner, as before. The
zunit_config_parallel fallback moves above the bootstrap handling
because the parent-or-worker decision needs the resolved flag.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(parallel): cover per-worker bootstrap sourcing
58484ec 
The fixture bootstrap appends its shell's PID to a log file on every
source - $sysparams[pid], because zsh keeps the parent's PID in $$
inside subshells - so counting distinct PIDs distinguishes one parent
source from one source per worker. It also declares a variable with a
bare typeset, which only survives into the tests when the script is
sourced at the worker's top level rather than inside a helper.

Covered: one source per worker across files, one source per slice of a
single file, the unchanged single parent source in serial mode, and a
failing bootstrap surfacing as an error which fails the run.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
docs: describe per-worker bootstrap sourcing in parallel runs
7080656 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.13.0
7ddc71a 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
feat(parallel): make single-file test slicing opt-in
8b172b2 
A parallel run of a single test file split that file's tests into one
contiguous slice per core. The tests in a file run in the order they
are declared, and share whatever state they leave on disk, so pulling
them apart changes what the file means - and since each worker sources
the bootstrap script itself, every slice also builds its own copy of
whatever environment that script prepares. A file whose later tests
read what its earlier ones set up therefore failed under --parallel
while passing in serial.

A test file is now a single unit of work. Slicing is still available,
behind --slice and the parallel_slice config key, for a file whose
tests are genuinely independent of each other and of their order. It
implies --parallel, having nothing to split a file across otherwise.

Two existing pieces move with the behaviour rather than after it. The
completion gains the flag alongside the usage text, because the suite
asserts that every option the CLI accepts is offered by the
completion. The test which covers slicing asks for it explicitly, so
that this commit leaves the suite green on its own.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(parallel): cover opt-in slicing
6df2a92 
The bootstrap fixture logs the PID of every shell it is sourced into,
which makes the number of workers a run used directly observable: one
for a single file by default, one per slice under --slice.

The ordering fixture is the regression itself, in miniature. Its
second test reads a marker the first writes, so it passes serially
and under --parallel, and fails when --slice splits the two apart.
The first test sleeps before writing, so the sliced run fails on the
ordering rather than on a race the scheduler might win.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
docs: describe --slice and file-level parallel granularity
2c346a4 
Say what the unit of work in a parallel run is, since that is what
decides whether a test file which shares state between its tests is
safe to run with -p, and record --slice and parallel_slice in the
options and config tables.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.14.0
6a94609 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(completion): offer test files after a run option
86f651c 
A rest argument declared as '*::arg:->args' hands the test paths to a
second _arguments call, and that call is never reached once an option
follows the command. `zunit run --verbose <TAB>` offered nothing at all,
and neither did any other line with a flag after the command. The file
parsed cleanly throughout, so nothing caught it.

List the test arguments in the single _arguments call instead. `init` is
dispatched ahead of it, since its options are the only ones that differ
and it takes no arguments of its own.

Three smaller places where the completion did not match the runner: a
path whose name begins with an underscore is skipped when the tests are
collected, so the output and support directories are no longer offered;
the name in 'file.zunit@test name' now completes from the @test lines
the file declares; and --time-limit guards its value, which is a count
of seconds.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(completion): drive real completion in a pty
5d2ff78 
The existing tests read the flag names out of the completion file and
compare them against the usage text, which says nothing about what a tab
actually offers. The completion offered nothing after a run option for
as long as those tests were green.

Type a line into a zsh running under a pseudo terminal, press tab, and
read back what the completion system draws. That covers the regression,
the paths the runner skips, and the test names a file declares.

Test bodies run with ERR_EXIT set. A wait that times out and a read that
finds nothing both report failure and neither is an error, so the helper
turns the option off the way evl does.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
feat(parallel): announce how a run was split
45b63f8 
Print the group count and the worker cap once, before the first
worker is forked. Neither number is knowable from the command line:
the group count depends on what was queued and whether slicing
applied, and the worker count comes from the machine. Left out of
TAP output, which is machine readable.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(parallel): cover the group count banner
edad334 
The four serial-versus-parallel parity checks compare output byte
for byte, so they filter the banner out alongside the timing line -
it is the one line a parallel run prints which a serial run has no
equivalent for. The count itself is asserted directly instead: a
literal count for file level groups, the shape only for slices,
since that number belongs to the machine.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
docs: describe the parallel group count banner
fef0e45 
Piped output is no longer byte for byte identical to a serial run,
so the parity claim is narrowed to report files rather than left
standing as written.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.15.0
e11dfed 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(cli): accept an option before the command it belongs to
ece25cf 
The command was read from the first word of the line, before a single
option had been parsed, so `zunit --verbose run tests` left the word
`run` behind to be parsed as a test path and died with

    Test file or directory 'run' could not be found

Every option the runner accepts was affected, and `--help` printed the
general usage rather than the usage of the command it was given to.

The command is now the first word which is neither an option nor the
value of one, and it is removed by position, so an option written
before it is still passed along with the rest of the line.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(init): keep reading options past one it does not know
3d4bfe9 
zparseopts stops at the first word it does not recognise unless -E is
given, so now that an option may be written before the command,
`zunit --verbose init --travis` bootstrapped a project without a
.travis.yml and reported success.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(run): read a time limit joined to its option by an equals sign
519d513 
zparseopts leaves the '=' attached to the value in the
`--time-limit=5` form and not in the `--time-limit 5` form, so the
arithmetic which works out the kill time read '=5' as no number of
seconds at all and the limit was never enforced.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(completion): find the command the way the runner finds it
9384fba 
The command was read from the second word, so `zunit --verbose init`
completed as a run and offered options `init` has no use for.

The command word is now looked for the same way the runner looks for
it, and it takes the place of the `zunit` word so that _arguments
reads everything written before it as an option given to it.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
build: make the built runner executable
96208b5 
The build target writes bin/zunit with a plain redirect, which leaves
a newly created file without its execute bit, and the tests target
runs that file directly. `make build tests` on a clean checkout stopped
at `Permission denied`.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
docs: describe options on either side of the command
689232e 
Every option now reads the same before the command as after it, and
-t takes its meaning from the command it is given to rather than from
where it was written.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
test(cli): cover options written before the command
6c708dd 
Nothing exercised the split between the runner's own options and the
command - tests/run.zunit covers the `run` helper, not the subcommand.
Every case in tests/cli.zunit fails against the runner as it was,
apart from the two which guard behaviour that had to stay put.

The completion tests which asserted that a run option was withheld
before a command now assert that it is offered, and one more covers a
command found behind an option.

Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
chore: bump version to 0.15.1
a815c46 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix(make): install command & duplicate test targets
b1b66f8 
Signed-off-by: Vladislav Doster <mvdoster@gmail.com>
@vladdoster
fix: install command by removing redundant flag
369345f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vladdoster