Repository navigation
Conversation
`rspec --parallel[=COUNT]` (or `config.parallel_workers`) forks COUNT workers from the process that loaded the suite and hands them top-level example groups from a shared queue, longest first when `.rspec_status` recorded their timings. Workers forward every notification -- including custom ones published through the reporter -- as id-based, Marshal-safe frames over a pipe; the coordinator maps them back to the real `Example` and `ExampleGroup` objects and replays them onto the ordinary `Reporter`, so existing formatters work unchanged and the run keeps one seed, one summary, one profile, one `.rspec_status` and the usual exit codes. Formatters that follow group boundaries see one group at a time, with bounded latency; the others get every notification as it arrives. Exceptions are never marshaled: workers ship class name, message, detailed message, backtrace, cause chain and aggregate-failure details, and the coordinator rebuilds a subclass of the real class under the original name. A worker that finishes but never exits cannot hang the run: it arms a watchdog before its `after(:suite)` hooks that dumps every thread's backtrace and aborts after `config.parallel_shutdown_timeout`, and the coordinator independently escalates TERM -> KILL -> give up. Cancelled workers (fail-fast, interrupt) and crashed workers are bounded and reported the same way. Suite hooks run once per worker, which sees `config.parallel_worker_number` and a parallel_tests-style `TEST_ENV_NUMBER`. Without `fork`, or inside another tool's parallel worker, the run is serial with a warning. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
`setup_aruba` wipes the working directory, which pulled it out from under the `cd` of other workers when the suite ran under `--parallel`. The directory is now namespaced by `TEST_ENV_NUMBER`, chosen in a `before(:suite)` hook because workers are forked after the suite loads, and the two specs that spelled out `tmp/aruba` use the helper instead. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ly interrupts The "spec files" shared context wrote `_passing_spec.rb` and `_failing_spec.rb` into `spec/support` and deleted them in `after(:all)`; under `--parallel`, nine spec files raced over those two paths and the library-wide checks globbed them while they existed. They now live under `tmp/`, named per worker. Integration specs cover a worker that receives an interrupt on its own: it retires after the group it was running while the others finish, and a run where every worker retires reports the groups nobody was left to run. The changelog spells out what reaches formatters instead of promising that they work unchanged. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…crash, wedge or lose their coordinator Found by a real-subprocess edge-case matrix and by running SerpApi's suite (Rails, Mongoid, Ruby 2.7, 151k examples) on a 3.12 backport of the runner: - The example a worker was running when it crashed or had to be terminated now fails with a Parallel::WorkerError naming the cause, so it shows under "Failed examples", in .rspec_status and in --only-failures; worker problems outside examples count as errors outside of examples. - Workers exit within half a second when the coordinating process goes away, and SIGTERM/SIGHUP to the coordinator -- or to its process group, as timeout(1) sends -- stop every worker, including ones that cannot tear down, before the coordinator exits the way a serial run would. A second SIGTERM used to cut the cleanup short and leave such workers running. - Listeners registered inside a worker (before(:suite) register_listener) are notified, and an example's result is forwarded after its outcome notification so metadata they add (extra_failure_lines) is kept. - Rebuilt exceptions use Exception's own backtrace accessors; a rebuilt Minitest::UnexpectedError used to crash dump_failures. - --[no-]parallel[=COUNT]: a malformed COUNT is a usage error instead of a backtrace, 0 runs serially, --no-parallel overrides .rspec, and the reproduction command --bisect prints runs serially. - Queued presentation is also flushed past 8 MB, and the parallel specs pass on MRI 3.0-4.0 and JRuby. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… under --parallel Fixes from an independent review of the previous commit, each reproduced first with real processes: - A run started under nohup no longer dies on SIGHUP. The coordinator only takes over TERM/HUP when they would end the process; ignored signals and handlers the suite installed stay in force, and workers get back exactly the handlers the coordinator replaced instead of the defaults. - Once a worker is given up on (it did not stop in time, crashed or sent garbage), nothing more is read from it. A worker that survived SIGTERM could otherwise report its example a second time, counting it twice or crashing the failure summary. - Under --fail-fast, a worker that leaves cleanly right after its group, before the coordinator asked it to stop, is no longer reported as an error outside of examples. - --bisect also takes --parallel out of SPEC_OPTS, which win over the command line, for its runs and the reproduction command it prints. - Two coordinator specs no longer race under load. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… are enabled, and finish parallel runs without an idle tick - The reproduction command --bisect prints now always ends with --no-parallel. Bisect cannot see config.parallel_workers set in a helper that the spec files require, and in a parallel run the groups of an order-dependent failure no longer share a process, so the command did not reproduce it. --parallel options are also always taken out of SPEC_OPTS while bisecting. - A signal the suite ignores with trap(signal, nil) stays ignored under --parallel, as with "IGNORE". - The coordinator no longer sleeps a full 0.25 s tick when every remaining worker is already exiting: a 24-example suite goes from 0.35 s to 0.09 s with --parallel=4 (serial: 0.07 s), and on 34 CI shards of a large Rails app the runs went from 0.13 s slower to 0.10 s faster than flatware (medians). This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… and JRuby Found by running rspec-core's own suite under --parallel in CPU-starved containers (MRI 3.0-4.0, JRuby 9.4-10.1): - The coordinator specs' fake workers run their examples on threads of the spec's process, side by side; those examples now skip the rspec-mocks lifecycle, whose scopes the spec's own stubs live in (on JRuby the threads raced on them and stubs vanished). - Deadline specs either use a single worker or a timeout long enough for the well-behaved worker, and the fake grace periods are long unless a spec is about them, so a slow thread death is not escalated. - The integration specs wait for the worker pid files to be written, not just created; the --fail-fast spec waits for the failure instead of racing worker start-up; the requeue spec no longer depends on a 0.2 s timeout; the watchdog spec waits for its blocked thread to block. - The SyntaxError spec no longer depends on syntax_suggest finishing its search twice within its 1 s timeout, and the signal spec accepts how JRuby reports a signal ignored with nil. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l run is stopped A run stopped from outside -- a CI job's timeout sending SIGTERM to the process group, say -- was most often stuck in an example, but the output only said "Received SIGTERM; stopping the workers.". Found running a large Rails app's suite on Ruby 3.3 on macOS, where a worker hung inside an example (in getaddrinfo, after fork) and finding the example took a native stack sample. The coordinator now reads what the workers already sent and adds a line per busy worker: "Parallel worker 1 (pid 10559) was running ./spec/lib/x_spec.rb:42 (X does y) for 14 minutes 47.3 seconds." Results are unchanged: interrupted examples are not turned into failures. Signalling a worker (escalation step, deadline, kill) moves to WorkerHandle. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… --no-parallel-preload Workers forked after the spec files load inherit whatever loading did. On macOS, after a large Rails suite's spec files have loaded, forked workers crash (SIGABRT, Ruby 2.7) or hang at 100% CPU (Ruby 3.3) in getaddrinfo -> Network.framework -> os_log; a probe without RSpec reproduces it, and workers forked before loading were fine in every run. Load-time database clients are inherited too. With `--no-parallel-preload` (`config.parallel_preload = false`) the workers are forked once the `--require`d files are loaded, before the spec files, and each loads them itself with its TEST_ENV_NUMBER already set; the coordinating process loads them too, for its reporting. Default behaviour is unchanged. - A worker gets work only after reporting a fingerprint of the examples it loaded (ids, locations, descriptions, after filtering) that matches the coordinator's, and no load error the coordinator did not get. Otherwise it is not used and its groups run elsewhere; a load error every process gets goes on like a serial run. - A worker still loading well after the coordinator finished (its load time plus parallel_shutdown_timeout) is terminated; workers there are more of than groups, still loading when the queue is empty, or not needed because the run ended early or is serial, are dismissed with SIGKILL -- they ran nothing. A worker given no work exits without suite hooks or at_exit. - Groups are dispatched by id as well as index. Forking, dismissing and terminating workers move to a WorkerLifecycle module. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…and warn when preload settings come too late With `--no-parallel-preload`, a worker is used if it loaded the same example ids and locations as the coordinating process; descriptions computed at load time (from the time or a temporary file's path) no longer get every worker rejected. A worker that loaded other examples is reported with the first spec file that differs. `parallel_workers` or `parallel_preload = false` set while the spec files load (in a file they require) comes too late to fork before loading: the workers are forked after loading, with a warning saying where to set it instead. `--only-failures` without a status file forks nothing early, since that run aborts with `exit!`. The worker count is no longer capped by the number of spec files. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ption their matcher generated
Examples without a docstring (`it { is_expected.to eq(5) }`) get their
description from the matcher once they have run, in the worker. The
coordinating process reported them as "example at ./spec/x_spec.rb:3"
in the documentation, failures, JSON and summary output; workers now
send each example's description and full description with its result,
and each group's with its start, and the coordinating process reports
them as the worker had them. With --no-parallel-preload that also
makes a description computed while a worker loaded the spec files the
worker's own, rather than the coordinating process's.
This is by Claude Opus 5.5
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rnel.rand seed, and never fork after loading Ruby reseeds `Kernel.rand` in a forked process, so workers that load the spec files themselves drew other values while loading than the coordinating process, even in a suite that seeds `Kernel.rand` from `--seed`, and ran other examples than it reported. All of them now start loading from the coordinating process's seed, so the values are those of a serial run with that seed. `parallel_workers` or `parallel_preload = false` set while the spec files load now gives a serial run with a warning, instead of forking after loading, which is what `parallel_preload = false` is there to avoid. No more workers than spec files (two for a single file) are forked before loading, so a small run does not load the suite once per processor, and workers forked before a `fork` that fails are dismissed rather than left behind. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Spec never calls srand Seeding every process from the coordinating process's seed, so that values drawn while loading matched, rewound that process's sequence (values drawn before the workers were forked were drawn again), gave every worker the same sequence while running examples (random ports and keys collided between workers), and broke RSpec's rule that it never calls `srand`. RSpec no longer touches `Kernel.rand`: each worker keeps the seed of its own that Ruby gives a forked process, while loading and running alike. Examples are reported as the worker that ran them described them, so a description drawn at random while loading is reported as it ran, and a worker whose examples differ in ids or locations is still not used. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… lookup per example `File.expand_path` of a relative path asks the OS for the working directory: about 17 microseconds on macOS. Filtering called it for every example even with no location filter (3.3 s for 151,132 examples in one process, 12.5 s while eight parallel workers did the same at once), and loading called it for every example and group to re-expand the relative form of a path RSpec had just loaded by its absolute path. Filtering now skips the lookup when no location or id filter is set, and loading expands the path it was given, which is also right when the working directory changed after `Metadata.relative_path_regex` was memoized. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rge suites Measured on a 151,132-example suite with 8 workers: - Workers spent ~95 microseconds per result frame, almost all of it marshaling `started_at`/`finished_at`: a local `Time` asks libc for its UTC offset, and in a process forked on macOS every such call reloads the time zone file. They now send whole seconds and nanoseconds and the coordinator rebuilds the `Time`s (~8 microseconds per frame). - The coordinator copied the rest of its read buffer (up to a megabyte) for every frame it took out of it; it now keeps an offset and compacts once per read. - It called `waitpid` on every worker at every wake-up of its loop; it now does so every 50 ms, and at once when a worker's results pipe closes. The coordinator's CPU time fell from ~78 s to ~45 s, with workers blocked less on its full pipes. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…'t count a stopped run against their deadlines - A worker's pipes are now opened just before it is forked, so the coordinator holds 2 file descriptors per worker rather than 4: 24 workers fit in a 64-descriptor limit. When the system refuses a worker (EMFILE, ENFILE, EAGAIN, ENOMEM) the run goes on with the workers it got, and serially if it got none, with a warning; it used to end with the error and 0 examples run. - Deadlines exclude time the coordinator's loop was not running beyond its own waits, and a worker's exit watchdog counts only time the worker ran: a run stopped with Ctrl-Z (or a machine asleep) while workers ran their after(:suite) hooks reported each one as stuck and failed. - The watchdog aborts the worker even when an example closed the stream it dumps the threads to. - Warnings no longer end in two periods. - The README says how to get coverage from the workers with SimpleCov (`merge_subprocesses`): rspec-core's suite measures 100% under `--parallel=8` with it, 61% without. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ter it was set Two gaps in the previous commit, found in review and reproduced with SIGSTOP/SIGCONT on the process group: - A pause in the first turn of the coordinator's loop was not counted, so a Ctrl-Z just after it loaded the spec files still killed every worker that was loading them itself. - A deadline set after a pause, later in the same turn (a --fail-fast stop, a worker's results pipe closing), was postponed by the whole pause, so a worker stuck in an example could outlive its timeout by as long as the run had been stopped. Each deadline now records when it was set and is postponed only by the part of a pause that came after it, counted from the start of the loop. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Deciding whether formatters follow group boundaries compared their methods with BaseFormatter's, which only a built-in formatter or a subclass of it loads. With a single custom formatter that does not inherit it, `rspec --parallel` raised NameError before running anything; rspec-core's own spec for custom formatters passed only when an earlier spec in the same process had loaded BaseFormatter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker that did not exit in time printed its timeout followed by
"seconds" whatever the value ("within 1 seconds"); the coordinator's
messages already pluralize.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found by running CI's steps in Linux containers (MRI 3.0-4.0, JRuby 9.4-10.1) on this branch merged into main: - A worker still loading once every group has been handed out is dismissed, so on these few-example suites one worker could take every group first. The specs that expect both workers to load, or the second to be turned away, failed on Ruby 3.0 and 3.4. They now keep groups queued until both workers have loaded, or let the second report first; the --no-parallel-preload scenario checks only what its example proves. - coordinator_spec's fake workers are threads, and Ruby 3.0 warns when a thread requires digest/sha2 while another is loading it. - A loading allowance of exactly one second reads "1 second"; two specs expected "seconds". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…hreads Their workers run on threads of the spec's process, and each one told to stop arms its exit watchdog, which sleeps for up to the shutdown timeout: 60 such threads were still alive after these two files (Ruby 3.4.8). A later example stubbing Parallel.sleep counted their calls too, so the sleep_running specs failed on JRuby 10.0 when worker_spec ran first (1 run in 3). Each example now kills and joins the threads it started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its example passes only in a worker that loaded the spec file itself, but a worker still loading once every group is handed out is dismissed, so the scenario cannot show that every worker loaded the files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On macOS, reading a worker's pipe while that worker copies a large frame into it fails with EINTR if a signal arrives, and Ruby takes SIGCHLD without SA_RESTART, so another worker exiting was enough. The run then crashed, printed a summary without the failure it was receiving and exited 1. The read moved no bytes, so it is now retried. A 5 MB failure message under a SIGCHLD storm: 10 of 10 runs crashed before, 0 of 10 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rspec-core loaded all 12 files of the parallel runner, which serial runs never use. Now only parallel.rb and plan.rb load with it, and the coordinator with the rest once a run is going to fork workers, as bisect does; its specs require it. Serial `rspec` with one example, n=100 interleaved medians: main 93.3 ms, all loaded 96.1 ms, now 93.1 ms. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves `parallel_workers_from` above `add_tag_filter`, so that changes to `configure_only_failures` merge without touching it, and lets the --fail-fast spec accept further segments of the summary line (counts that other changes add) while it still checks for the one failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker whose code under test called `exit` skipped arming its watchdog and got no deadline from the coordinator, so a stuck `after(:suite)` hook hung the run (a repro still ran after 75 s; now it ends in 2.9 s). On every way out of its job loop a worker now arms the watchdog and tells the coordinator before any `after(:suite)` hook runs. `exit` in a worker ends the run with its status, as it does serially, instead of failing the example. A crashed worker's example still fails and is not run again; the examples of its group it had not started run in another worker, and a crash in a context hook fails that group's unstarted examples as a raising hook would. Workers that run out of work wait while others are busy, so that one is there to take them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run is never shorter than its longest top-level group: rspec-core's own suite under --parallel=8 spent 10.62 of its 10.64 s in one. A group tagged with a part count now runs as that many slices of the order its examples run in, balanced by recorded run times, each a filtered run of the group in whichever worker is free. It is opt-in because context hooks run once per part and parts can run at the same time; groups with a custom ordering are not split. rspec-core tags its two longest groups. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once a worker's `exit` has ended the run, its status stands only if every worker then leaves on its own: another worker that is lost, fails or has to be terminated now makes the run end with the exit code its results call for, instead of the `exit` status. And an `after(:suite)` hook that calls `exit` with another status decides the status, as it does serially, instead of being reported as a worker that did not exit in time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The coordinator's report of a worker lost between example groups was reached only when the ECHILD spec happened to reap an idle worker: on Linux MRI 3.1 (aarch64 container) that line was missed in 3 of 6 runs of CI's run_rspec, failing its 100% coverage minimum. Since workers now wait while another one is busy, a worker killed in that wait is a real case: a spec now covers it deterministically (30/30 runs reached it). The `exit` and SIGTERM specs matched the summary line up to its end, so upstream's open rspec#302 (examples not run) and rspec#305 (expectation counts) broke them by appending to it; they now allow a suffix, as the fail-fast spec already did. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JRuby's Example#run rescues SystemExit (example.rb), so there `exit` in an example fails it, serially too, and never leaves the worker's job loop. The two worker specs that call it from an example sent no stop message after the job, so on JRuby 9.4 the worker waited for one forever and CI's run_rspec step hung (JVM thread dump: Channel#receive in the job loop, inside the specs' raise_error). They now skip on JRuby, where --parallel runs serially anyway, and queue a stop so that an `exit` that does not propagate fails them instead of hanging the suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A worker past its shutdown timeout after its last group is sent SIGTERM, but its frames were still read: if an `after(:suite)` hook took the signal by calling `exit 0`, that became the run's status, and a run with an error (or failures) exited 0. Like every other path that terminates a worker, that one now stops reading it. A coordinator spec reproduces it: it got `exit(0)` in 3 of 3 runs before the fix and passes 5 of 5 after (2 of 2 fail again with only the library change taken out). When the same worker's later `exit` replaces the status, the run now says so instead of only naming the first status. The SIGTERM spec, whose summary match allows a suffix, also checks that no error was reported. Found by an independent review of the last three commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every `require "rspec/core"` loaded parallel.rb and parallel/plan.rb (about 400 lines) and every run built a plan, though only a run that asks for workers -- `--parallel`, `config.parallel_workers` or `config.parallel_preload = false`, set before the spec files load or by them -- needs either. The runner now loads them for such a run only. A one-example `rspec` process, each tree under /tmp, n=100 per arm interleaved in one session, macOS Ruby 3.4.8 at load ~10, median (IQR): branch base df877b7 69.6 ms (67.7-71.4), before this commit 70.4 (68.7-72.3), with it 70.1 (68.4-71.1), and 69.7 (68.3-71.2) for a second arm of the same code. The same code run from the worktree's path under /Users took 71.0 ms, so only runs from one location compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pec#305 The spec that checks a --no-parallel-preload run leaves Kernel.rand alone drew its value after its own `expect`. With rspec#305 merged, every `expect` calls `RSpec.configuration`, which in this process may create a configuration and so draw from Kernel.rand: on a trial merge with rspec#305 the check failed after an `expect` for a serial run too, and passed when drawn first, both serially and with --parallel=2 (4/4 pass without rspec#305). It now draws first, and still fails when the coordinator draws (checked by making `Coordinator#start` call Kernel.rand). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A still-loading worker's deadline is twice the coordinating process's own load time plus the shutdown timeout. At load 313 that stretched past the 2.1 s of work the other worker had, so worker 2 was dismissed unreported and the spec failed (1 of 5 serial suite runs). The busy worker now waits until worker 2 is gone, capped at 15 s. With the coordinating process's loading slowed by 0.5 s per file: before 3/3 failed, after 3/3 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The busy worker could read loading.pid after it was created and before
its contents were written, and fail on Integer(""). Worker 2 now writes
a temporary file and renames it. Slowed-coordinator reproduction: 3/3
passed; the file passes 45/0 serially and with --parallel=4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…t count them rspec#305 has rspec-expectations call the configured reporter's increment_expectation_count for every expectation. In a worker that is WorkerReporter, so the counts never reached the summary. Workers now send [:expectations, n] before each unit_finished and after their after(:suite) hooks, and the coordinator replays them if its reporter has that method. Without rspec#305 nothing calls it and nothing is sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…xit` From an independent review: counts were sent only before unit_finished and after the suite hooks, so an `exit` in an example or after(:suite) hook dropped the unit's and the hooks' counts, which a serial run keeps. Workers now also send them on leaving the job loop and from an ensure, ignoring a channel nobody reads, as send_exiting does. The slow-loading spec's 15 s cap is now one deadline per worker, not one per example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…e run Found by a trial merge at load ~300: a worker deadlocked inside fork (rb_thread_atfork waiting on a mutex, before any Ruby code ran) got a group, looked like one running a long example, and the run waited until `timeout` killed it after 600 s; with its coordinator-watch thread never started, it was then orphaned. Workers now send [:started] first; until then they get no work, are killed once nothing is left, and are sent TERM, then KILL, if they miss the start deadline (the shutdown timeout). Before: the new integration spec hung until killed at 90 s. After: 3/3 passed, the stuck worker killed and both examples run by the other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
0780bdd held work back until a worker's first frame, so on a busy machine the first worker to start took every small group: two of rspec-core's parallel specs that expect each worker to run one failed (1 of 3 dogfood runs). Workers again get a group at fork; one that misses its start deadline has that group requeued as is. A reworked start deadline then hung the suite under --parallel=8 (2 of 2 runs): a late [:started] cleared the deadline fail-fast had set. Only the start deadline itself is cleared now; any later one replaces it. New spec: with the old clearing, the run did not end within 10 s. Dogfood --parallel=8: 3/3 passed, 8-12 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…lsewhere From an independent review: when the run was stopping early, the report still said the other workers would run the stuck worker's share, but its group was not handed on. It now says the group runs in another worker, or that it did not run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… they start Fork CI failed one coordinator spec on 4 of 44 jobs: a fake worker whose thread started after `dismiss` had closed its pipe raised sending [:started] and died, so only 1 of 2 workers "survived SIGKILL". The fakes now ignore a pipe nobody reads, as the processes they stand for would go on. With their start delayed 0.05 s: before 5/5 failed, after 5/5 passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…der rspec#305 On a trial merge with rspec#305, the spec that checks a loading worker sends nothing else got [:expectations, 1]: its own `expect`, made while the worker ran, counted on the worker's reporter. It now asserts once the worker has finished. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Bisect started its runs with `-I` paths joined by `:`, which splits `C:/...` on Windows: 8 bisect specs failed there. The rake task spec for a clean environment now passes whether or not ANSICON is active. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
From the repository's root `rspec` found no examples, so the job passed without running any. In each library's directory, as the Linux jobs run: rspec-core 2,465 / 0 failures, rspec-expectations 2,365 / 0, rspec-mocks 1,740 / 0, rspec-support 595 / 0 (Ruby 3.4, windows-latest). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… changed Keys an example, its hooks or a worker listener set, changed in place or deleted reach the coordinator's copy before formatters see the example (screenshot paths, retry counts). RSpec's own keys stay the coordinator's; a value that cannot be marshaled is named in a warning once per key and the formatters see the value from before the example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…rinted it A worker's standard output and error go to files in a private temporary directory; before each frame it says how far they reach, and the coordinator shows the new bytes in the worker's sequenced stream, so `puts`, warnings and subprocess output land under their group as in a serial run instead of between progress dots. What is left once a worker has gone (a crash report, `at_exit` output) is read then. Workers loading the spec files themselves skip their loading output, which the coordinator already printed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Where Ruby has no `fork`, each worker is a new process running the `rspec` command the run was started with: same arguments, `.rspec` files and `SPEC_OPTS`, no formatters, the run's seed (adopted without choosing an ordering). It loads the spec files itself, as with `--no-parallel-preload`, and takes jobs and sends results over its standard input and output in binary mode. Workers notice a gone coordinator by its pid where their parent pid never changes (Windows), get SIGKILL where SIGTERM cannot be sent, and a run that was not started by the `rspec` command stays serial with a warning. Probed first on windows-latest (Ruby 3.0, 3.4), JRuby 9.4/10.0/10.1 and TruffleRuby: pipes, binary frames, capture files, exit statuses, signals, inheritance and parent death, 18/18 checks each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…it to run A worker lost while it ran groups (crashed, killed, terminated for being stuck, or stuck in `fork`) is replaced by one with its number and `TEST_ENV_NUMBER`, which runs what is left; its failed example stays failed. Not once the run stops early or ends by `exit` or a signal, not for a worker lost while loading, and no more often than the run has workers. Under `--no-parallel-preload` with `fork` the replacement is a new `rspec` process: forking the loaded coordinator is what that mode avoids. Output files are per worker generation, so a replacement never truncates a predecessor's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Windows CI showed: `$stdout` replaced by a StringIO broke showing worker output (no `binmode?`); a spawned worker's start was reported only after its `--require`d files loaded, so a slow one was called "stuck in a fork handler"; and a failure while stopping the workers could leave the run's SIGTERM handler installed. Workers now size their output files by path and keep no handles to them, the coordinator restores its signal handlers whatever stopping the workers does, and a spawned worker that never reports back is described as not having started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
JRuby keeps `Dir.chdir` inside the JVM, so a process it spawns starts in the JVM's directory: spawned workers now get `:chdir => Dir.pwd`, as the spec helper that runs `rspec` in a process of its own does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…illed A worker that cannot notice its coordinator is gone -- stopped, or stuck before running any of RSpec's code, as a deadlock in a fork handler leaves it -- kept CI steps open when the coordinator was killed with SIGKILL. A small process, spawned before the first worker is forked, watches the coordinator; when it goes it gives registered workers 1.5 s to leave, sends SIGTERM, then SIGKILL 5 s later, and removes their output files. It signals only registered pids still in the run's process group, ignores INT/TERM/HUP/QUIT and exits when told the run is done. Not covered: a worker forked moments before the coordinator was killed and not yet registered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
On Windows the surviving fake worker finished every queued group before the crash was noticed, so nothing waited to run and no replacement was due. The remaining groups now take 0.5 s each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…speed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Metadata a worker could marshal but the coordinator could not load (a class defined only in the worker) made the frame undecodable, so the worker was terminated and a passing example failed. Each changed key now travels as its own dump; one that fails to load is warned about and left out. - A spawned worker's `--require`d files ran `add_formatter` with a path, truncating the file the coordinator writes. Its formatter loader now opens no files. - Output read between two frames could pile up past the sequencer's 8 MB bound until the next loop turn; queued events are now flushed as soon as a bound is crossed. - Worker output with non-ASCII bytes raised Encoding errors when `$stdout` had an external encoding; it is written untranscoded. - A spawned worker that never reported back while loading was replaced; workers lost while loading are not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Workers that load the spec files themselves reset their reporter before loading, which put back a formatter loader that opens files: a helper the spec files require that adds a formatter writing to a file truncated the coordinator's report (NUL bytes where its first lines were, deterministically for a replacement worker). They now keep a loader that opens nothing. Found in the re-review of the five fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Windows writes CRLF to a text-mode file and can still find a pid for a moment after it was reaped, which two new specs assumed it would not. The docs no longer name TruffleRuby among the platforms that spawn workers: only the process mechanics were probed there, not RSpec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
On JRuby the surviving fake worker finished every queued group before the crash was noticed, so no replacement was due. The groups now take 0.5 s each, as in the other replacement spec. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Author
|
Opus 5.5 xhigh ran these and wrote this comment. SerpApi's app suite runs in 140 s with
Apple M4 Max, the app's CI Ruby, Memory comes from separate runs, one per arm, sampled every ~5 s with macOS Why the memory: every worker loads the whole suite. Booting the app takes 0.6 GB, and loading all 151,725 examples brings a process to 2.0 GB. flatware's workers load only the files of their current batch. Doing the same here would change when a spec file's global configuration and shared examples take effect, so this PR doesn't. When an example never returns. This happened in one extra run of each runner: a worker's threads stayed blocked on a lock for over 8 minutes. I then reproduced it with a 6-file fixture (one example joins a thread waiting on a mutex nobody releases, 2 workers, 3 runs per cell):
The SerpApi runs agreed: after SIGTERM to the main process, flatware printed nothing and left its stuck worker running. Preloading doesn't work for this app on macOS. Preloading is the default: workers fork after the spec files load. In that mode 12 workers crashed (SIGABRT after a segfault in a thread), and 8 replacements were started (the cap). Every example a crashed worker hadn't started still ran in another worker. I checked the recorded ids against a full run: 18 are missing. Matched file by file against the run's failure list, they are the 12 examples that were running when their worker crashed (they fail), plus the stuck example and the 5 queued behind it that the SIGTERM above cut off. That's why the app uses Not shown here: spec names, failure messages, the environment and its services. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opus 5.5 xhigh wrote this and ran every check in it (who wrote and reviewed what: last section).
rspec --parallelruns a suite in worker processes and reports it as one ordinary RSpec run. You get the same formatters, one seed, one summary, one.rspec_statusand the usual exit codes. It's opt-in and adds no gem. Workers are forked where Ruby can fork; on Windows and JRuby they are newrspecprocesses. This is for review here only; nothing has been proposed to rspec/rspec.Before / after
rspecrspec --parallel-w 8: 199.5 safter(:suite)hook never returnsexit 3$stdoutfork(Windows, JRuby)rspecworkersMedians of 5 interleaved runs on 29b48c6, macOS, Ruby 3.4.8, load 9–15. Windows: one run each, Ruby 3.4. SerpApi: median of 3 cold rounds with the same examples and results in each runner (details and memory). Chart: 364655614; this round's A/B on those fixtures shows no change (below).
Formatters,
--seed/--order, filters,path:line,--only-failures,--next-failureand--fail-fastgive the same results as a serial run;--bisectbisects serially.Since the last description (7c924a3 → 29b48c6), its "Not covered" list:
fork: warned and ran seriallyrspecworkers; rspec-core's suite passes that way on Windows and JRuby$stdoutwasn't ordered with the reportbefore(:context)hooks once per partNot covered
--no-parallel-preload: every worker loads the whole suite. On SerpApi's suite that's 22 GB at peak against flatware's 8.5 GB (one sampled run each; 4 workers: 12 GB, 283 s). Loading only each worker's files changes when a spec file's global configuration takes effect, so this PR doesn't do it. Preloading shares one copy, but in that app on macOS the workers then crashed.binding.irb,debug) need a serial run, because a worker's$stdoutisn't a terminal.rspeccommand: a script callingRSpec::Core::Runner.runwarns and runs serially on Windows and JRuby.fork, so they're skipped on Windows and JRuby (109–110 pending there, 7 on macOS).--bisectspec's DRb connection was reset in 1 of 13--parallelruns of rspec-core's suite (0 of 40 reruns).Try it (Ruby 3.0+):
How it works
:parallel_split) to whichever worker is idle. Each worker has its own pair of pipes carrying length-prefixedMarshalframes, and everywaitpidisWNOHANG.ExampleandExampleGroupobjects. Exceptions are rebuilt, never marshaled. Metadata an example changed travels key by key.$stdoutand$stderrgo to a file of its own. The coordinator, the only reader, shows each piece with the group that printed it.fork, workers areruby -I<load path> <the rspec command>processes started in the run's directory. They load the spec files themselves and adopt the coordinator's seed.config.parallel_shutdown_timeout(30 s). A worker that doesn't report back, or doesn't exit after its last group, gets TERM, then KILL 5 s later. A worker leaving its job loop arms a watchdog that dumps its threads and aborts. SIGTERM or SIGHUP to the run stops every worker and prints what each was running. If the coordinator itself is killed, a small watcher process (ruby --disable-gems) stops the workers it registered.config.parallel_worker_number(1, 2, …) and parallel_tests'TEST_ENV_NUMBER("","2", …)..github/workflows/rspec.yml.Benchmarks
rspec-core's own suite on 29b48c6, serial and
--parallel=8alternating, n=5, macOS, Ruby 3.4.8, load 9–15. Every run exited 0 with 2,527 examples, 0 failures, 7 pending.rspecrspec --parallel=8Runner overhead this round (bench fixtures, n=5 interleaved, every run checked for its count and 0 failures, 30/30):
Runner comparison (the chart), measured on 364655614 (456bb51 merged into main), Ruby 3.4.8, Apple M4 Max, load 23–57. Seconds, median (min–max) of 5 runs, runners interleaved, 8 workers, 120/120 runs with the right count and 0 failures:
Windows and JRuby, rspec-core's suite, GitHub runners, one run each (probe run 38839752160, rspec-core identical to d3eb83a):
--parallel=4Checks
GitHub Actions on this fork, run 38841282309 on 29b48c6: 44 of 44 jobs passed.
Windows and JRuby probe, run 38839752160 (rspec-core identical to d3eb83a):
Local, CI's environment variables, macOS, Ruby 3.4.8, on 3b14b6d38 (d3eb83a merged into rspec/rspec main 4af3571, which contains this PR's base 14c8804; 29b48c6 changes one spec, run by the CI above):
Edge cases in real processes, on 29b48c6, Ruby 3.4.8: 107/107 preloaded, 107/107
--no-parallel-preload, 105/105 withforkremoved (spawned workers). Earlier spawned runs failed a worker-count check twice, once in each of two scenarios: the harness expected every worker to run a group, but the groups were so short that one worker ran them all, which is valid scheduling. Those checks now hold each worker's examples until the expected workers have started, for at most 10 s (5 more runs: 3/3 each).An example that never returns (6-file fixture, 2 workers, 3 runs per cell), then SIGTERM to the main process, SIGTERM to the process group, or GNU
timeout 12around the run:rspec --parallelnames the stuck example, prints the summary and leaves no process in every case. flatware 2.4.0 prints nothing, and leaves the stuck worker running when only the main process is signalled (details).Bugs found and fixed this round:
add_formattertruncated the coordinator's report file;--bisectjoined$LOAD_PATHwith:(b39bb9a), and two specs failed only there (d3eb83a).Who wrote and reviewed what
Opus 5.5 wrote this description and ran every check in it: xhigh effort this round, high and max in earlier ones. Of the 58 commits, Fable 5.1 Max wrote the first 3 and Opus 5.5 the rest; this round's 16 at xhigh. An Opus 5.5 xhigh code-review agent reviewed 7c924a3..f92bc3e and then the fixes through 42a596e; d3eb83a and 29b48c6 (spec and Windows fixes) were not reviewed. The reviewer is the writer's model family, so treat it as a self-review. GPT-6 Astra (high and xhigh) advised on the plan and on how this round was measured. Nobody has reviewed it by hand yet.
Machine-readable summary