diff --git a/.github/workflows/rspec.yml b/.github/workflows/rspec.yml index b898cb180..4ce1fcbd5 100644 --- a/.github/workflows/rspec.yml +++ b/.github/workflows/rspec.yml @@ -85,6 +85,12 @@ jobs: ruby: - '3.4' fail-fast: false + # In the library's directory, as the other jobs: from the repository's + # root `rspec` finds no examples. + defaults: + run: + shell: bash + working-directory: ${{ inputs.library }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -92,6 +98,6 @@ jobs: - uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0 with: ruby-version: ${{ matrix.ruby }} - bundler-cache: true - run: choco install ansicon + - run: ../script/bundle - run: bundle exec rspec --backtrace diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 39ccc5787..672c8b2ba 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -1,6 +1,51 @@ ### Development [Full Changelog](https://github.com/rspec/rspec/compare/rspec-core-v3.13.6...main) +Enhancements: + +* Add `--[no-]parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run + top-level example groups across worker processes forked after the suite has loaded. + Formatters run in the coordinating process only and receive one coherent stream of + notifications with real `Example` objects -- results, timing, pending details, + exceptions rebuilt from their class name, message, backtrace and cause chain, messages, + deprecations, marshalable custom notifications and the metadata an example set, changed + or deleted -- so existing formatters keep working and the run has one seed, one summary, + one profile and one `.rspec_status` file. What workers print (also from processes they + start, crash reports and `at_exit` handlers) appears with the example group that printed + it, as in a serial run. Listeners + registered inside a worker, e.g. in a `before(:suite)` hook, still see that worker's + example notifications. Workers run the suite hooks once each, expose + `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way + parallel_tests does. A worker that finishes but does not exit is aborted after + `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its + threads, so a stuck `at_exit` handler can no longer hang a run (nor can one that runs + after code under test called `exit`); neither can a worker stuck in `fork`, even once the + coordinating process is killed with SIGKILL. A worker + that crashes fails the example it was running instead of taking the run down, while + the examples of its group it had not started run in another worker, and a new worker + with its `TEST_ENV_NUMBER` takes its place. Code that calls + `exit` ends the run with that status, as it does serially. A top-level group tagged + `:parallel_split => N` runs as N parts in + different workers. A run stopped with SIGTERM or SIGHUP says which example each worker + was running and for how long. With `--no-parallel-preload` + (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec + files are loaded and each loads them itself, so nothing the spec files do while loading + is shared with the workers; a worker that does not load the examples the coordinating + process has is not used. Where Ruby has no `fork` (Windows, JRuby) the + workers are new processes running the same `rspec` command with the run's seed, and load + the spec files themselves. + `--bisect` runs serially, and the reproduction command it prints now ends with + `--no-parallel`. + (Illia Zub) +* Load and filter large suites faster by no longer looking up the working directory once per + example and example group, which took seconds on a suite of 150,000 examples, and more when + several processes did it at once. (Illia Zub) + +Bug Fixes: + +* Fix `--bisect` on Windows, which passed its load path to the processes it starts joined + with `:` instead of `;`. (Illia Zub) + # 4.0.0.beta1 / 2026-02-18 Breaking Changes: diff --git a/rspec-core/README.md b/rspec-core/README.md index 1798a5198..8a7287747 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -275,6 +275,217 @@ which you'll use to run rspec. The `rspec` command comes with many useful options. Run `rspec --help` to see the complete list. +## Running Examples in Parallel + +`rspec --parallel` runs the top-level example groups of your suite across +worker processes, one per processor by default (`--parallel=4` for four). +RSpec loads the suite once, forks the workers from the process that loaded it, +hands out groups to whichever worker is free -- longest first when a previous +run recorded timings in the `example_status_persistence_file_path` file -- and +streams results back as examples finish. Your formatters run only in the +coordinating process and see the notifications of a normal run, so there is +one seed, one summary, one profile and one exit code, and `--fail-fast`, +`--only-failures`, `--next-failure`, `--seed`, filters and `path:line` +selection keep working. Once the `--fail-fast` limit is reached no further +groups are handed out and busy workers stop after their current example, so +examples already running in other workers can still add failures. You can make +it the default with: + +```ruby +RSpec.configure do |config| + config.parallel_workers = :auto # or an Integer +end +``` + +Each worker runs the `before(:suite)` and `after(:suite)` hooks once. Inside a +worker, `RSpec.configuration.parallel_worker_number` is its 1-based number and +`ENV['TEST_ENV_NUMBER']` follows the convention of parallel_tests (`""` for the +first worker, then `"2"`, `"3"`, ...), so anything you already namespace per +worker for that tool keeps working. Each worker draws from `Kernel.rand` with +a seed of its own, as forked processes do, so `Kernel.srand config.seed` does +not make what examples draw reproducible under `--parallel`. + +Workers inherit everything that happened while the suite was loaded. For a +Rails application that means `rails_helper` has already evaluated +`database.yml` and connected to the database in the coordinating process +before any worker exists: setting `TEST_ENV_NUMBER` afterwards does **not** +give workers separate databases, so such Rails suites run under `--parallel` +share one test database until rspec-rails adds per-worker database support. +Anything the spec files open while they load -- connections, servers, +threads -- is inherited by every worker; acquire such resources in +`before(:suite)` to get one per worker. A database client that connected while +the spec files loaded (a query at load time is enough) hands the same +connection to every worker unless it reconnects after `fork` by itself: as with +app servers that preload, make each worker reconnect in a `before(:suite)` +hook (with Mongoid, `Mongoid.disconnect_clients`), or have the workers load the +spec files themselves (below). On macOS, the system networking libraries do not +survive `fork` well either: when loading the spec files does a lot of work +(resolving host names, say), workers can crash in `getaddrinfo` or hang. + +Or have every worker load the spec files itself: with `--no-parallel-preload` +(or `config.parallel_preload = false` in a `--require`d file) the workers are +forked as soon as the `--require`d files are loaded, before the spec files. +This and the worker count must be known by then: set in a helper that the spec +files require, they come too late, and the run is serial, with a warning -- +forking after loading is what this setting avoids. There are no more workers +than spec files (but two for a single file). Each worker then loads the spec +files with its `TEST_ENV_NUMBER` already set, and the coordinating process +loads them too, for its reporting. RSpec never calls `srand`, so each worker +draws from `Kernel.rand` with a seed of its own while loading too, whatever +the coordinating process seeded it with. Nothing the spec files do while +loading is shared then -- connections, threads, system libraries they +initialize -- and helpers they require (rather than `.rspec`) load in every +worker, so a Rails app whose spec files `require 'rails_helper'` boots in each +worker with its own `TEST_ENV_NUMBER`. The cost is loading the spec files once +per process: more CPU and memory, though they load at the same time, and +whatever they do at load time is done that many times. A worker is given work +only once it has loaded the examples the coordinating process has (the same +ids and locations), and what it runs is reported as it described it, so a +description computed while loading -- from the time or `rand`, say -- is the +worker's. One that loads other examples, gets a load error the coordinating +process did not get, or is still loading long after the coordinating process +finished is not used: that counts as an error outside of examples, and its +groups run in the other workers -- or, when no worker is left, do not run. + +Formatters receive examples with their status, timing, pending details, +exceptions (class name, message, backtrace, cause chain and aggregated +failures; rebuilt as a subclass of the original class when that class exists +in the coordinating process) and the group and example notifications in the +order the worker produced them, plus `reporter.message`, deprecations and +custom notifications whose options can be marshaled. What an example, its +hooks or a worker listener do to the example's metadata -- keys set, changed in +place or deleted, and the description the matcher of a one-liner like +`it { is_expected.to eq(5) }` generates -- reaches the coordinating process +before formatters see the example. RSpec's own keys stay the coordinating +process's; a value that cannot be marshaled is named in a warning, once per +key, and formatters see the value it had before the example. Changes to a +group's metadata are not forwarded. + +What workers print -- `puts` in an example, warnings, the output of processes +they start, a crash report, `at_exit` handlers -- appears on the coordinating +process's standard output and error with the example group that printed it, +as in a serial run: workers write it to files in a private temporary +directory, which the coordinating process reads and then removes. So a +worker's standard output and error are not terminals (`$stdout.tty?` is +false), what an example prints appears once the worker next reports (when the +example finishes, at the latest), and an interactive debugger (`binding.irb`, +`debug`) needs a serial run. With `--no-parallel-preload`, what workers print +while they load the spec files is left out: the coordinating process printed +the same when it loaded them. Listeners registered from +inside a worker (for example with `reporter.register_listener` in a +`before(:suite)` hook) are notified of that worker's example and group +notifications before they are forwarded; run-level notifications (`start`, +`dump_summary`, `close`, ...) only reach the coordinating process. + +A worker that has finished its last example group but does not exit (a +non-daemon thread, a browser that will not quit, ...) is given +`config.parallel_shutdown_timeout` seconds (30 by default) to run its +`after(:suite)` hooks and `at_exit` handlers; after that it dumps the +backtraces of all its threads to stderr and is aborted, and the run exits +with `config.error_exit_code`. The coordinator never waits unboundedly for a +finished worker -- not even one whose watchdog cannot run and that ignores +SIGTERM: the run waits for it about the shutdown timeout plus 11 seconds at +most (a second of grace, then 5 seconds after SIGTERM and 5 after SIGKILL). +A worker that never gets past `fork` (deadlocked in a fork handler, in Ruby or +in a library) is terminated the same way once it has not reported back within +the shutdown timeout of being forked, and the group it was given runs in +another worker. + +A worker that crashes or is killed (`exit!`, a segfault, the OOM killer, ...) +fails the example it was running with the reason, so the example is listed +under "Failed examples" and picked up by `--only-failures`, and is not run +again -- whatever it did may be why the worker died; the examples of its group +it had not started yet run in another worker. One lost in a `before(:context)` +or `after(:context)` hook fails the examples of that group it had not started, +as such a hook raising an error would, and the rest of its group runs +elsewhere. While groups wait to run, a lost worker is replaced by a new one +with its number and `TEST_ENV_NUMBER` -- not once the run stops early (fail-fast, +an interrupt, `exit`), not for a worker lost while loading the spec files, and +no more often than the run has workers; under `--no-parallel-preload` with +`fork` the new one is a new `rspec` process, as without `fork`, since forking +the coordinating process after it loaded the spec files is what that setting +avoids. Workers that run out of work wait for the busy ones, so that there is +a worker to run what a lost one leaves; if none remains, the run says which +examples did not run and how to run them. Problems between example groups count +as errors outside of examples. + +RSpec does not rescue `SystemExit`, so in a serial run code that calls `exit` +(or `abort`) ends the run with that status. Under `--parallel` it ends the run +the same way: no further examples start, busy workers stop after the example +they are running, the summary covers what ran and the run exits with the status +passed to `exit` -- unless a worker then had to be aborted or killed for not +exiting in time. Which examples ran before that depends on timing. To make such +an `exit` fail the example instead, rescue `SystemExit` in an `around` hook. + +Workers also exit on their own if the +coordinating process disappears. One that cannot -- stopped, or stuck before +it runs any of RSpec's code, as a deadlock in a `fork` handler leaves it -- is +stopped by a small process the run starts before forking its first worker, +which outlives a coordinating process killed with SIGKILL: it gives the +workers it was told about a moment to leave, then sends them SIGTERM and 5 +seconds later SIGKILL, and removes their output files. It signals only those +workers, and only while they are in the run's process group; one forked +moments before the coordinating process was killed may not have been +registered yet. SIGTERM or SIGHUP to the coordinating +process, as `timeout` or a cancelled CI job sends, stops every worker -- even +one that cannot tear down -- before the process ends the way a serial run +would, and the output says which example each worker was running and for how +long, which is usually why the run had to be stopped; a signal the run was +started ignoring (`nohup`), or that your suite installs a handler for, is left +alone. Time the whole run spends stopped (Ctrl-Z) or the machine asleep does +not count towards these timeouts. + +A run is never shorter than its longest top-level example group, which a +single worker runs. `--profile` shows the slowest groups; tag one whose +examples can run at the same time in different processes with the number of +parts to run it as, and workers take the parts like any other group: + +```ruby +RSpec.describe Pricing, :parallel_split => 4 do + # ... +end +``` + +Each part is a slice of the order the group's examples run in, of about the +same total run time when a previous run recorded run times in the +`example_status_persistence_file_path` file (of the same number of examples +otherwise), and runs as `rspec spec/pricing_spec.rb[1:2,1:3]` would: +`before(:context)` and `after(:context)` hooks run once per part, so a group +that sets up something expensive or shared in them may be better left whole. +There are no more parts than examples, the number of parts does not depend on +the machine, and a group whose examples (or those of a group in it) use a +custom ordering is not split, with a warning. With a formatter that follows +example groups (documentation, HTML), each part reads as a block of its own, +under the group's description; `--profile` adds up the parts. + +Where Ruby has no `fork` (Windows, JRuby), the workers are new +processes running the `rspec` command the run was started with -- the same +arguments, `.rspec` files and `SPEC_OPTS`, without the formatters, and the +run's seed -- and each loads the spec files itself, as with +`--no-parallel-preload`. Their jobs and results travel over their standard +input and output, so what they print goes through the coordinating process +as described above. Configuration done in-process before +`RSpec::Core::Runner.run` cannot reach such workers, so there a run that was +not started by the `rspec` command runs serially, with a warning. Each worker +starts its own Ruby (seconds on JRuby), and on Windows workers that have to +be stopped get SIGKILL, which is all Windows can send. When `TEST_ENV_NUMBER` +is already set because another tool is running RSpec in parallel, RSpec +prints a warning and runs serially. When the system refuses a worker -- too many open +files or processes -- the run goes on with the workers it got, and serially if +it got none, with a warning. `--bisect` and `--dry-run` always run serially, +and the reproduction command `--bisect` prints includes `--no-parallel`, which +(like `--parallel=0`) overrides `--parallel` in `.rspec` and +`config.parallel_workers` in a helper, and leaves `--parallel` out of its +`SPEC_OPTS`. + +Each worker is a separate process, so a coverage tool records only the +coordinating process unless it merges forked processes. With SimpleCov, set +`merge_subprocesses true` in the `SimpleCov.start` block (SimpleCov 1.3; it +was `enable_for_subprocesses` before): rspec-core's own suite then measures +the same coverage with `--parallel=8`, preloaded or not, as it does serially. +Workers started as new processes (no `fork`) start SimpleCov from your spec +helper themselves, and SimpleCov 1.3.2 merges their results. + ## Store Command Line Options `.rspec` You can store command line options in a `.rspec` file in the project's root diff --git a/rspec-core/features/.nav b/rspec-core/features/.nav index 009cf30c6..05e2d8888 100644 --- a/rspec-core/features/.nav +++ b/rspec-core/features/.nav @@ -9,6 +9,7 @@ - line_number_appended_to_path.feature - exit_status.feature - order.md (`--order` option) + - parallel.feature - rake_task.feature - pending_and_skipped_examples: - pending_examples.feature diff --git a/rspec-core/features/command_line/bisect.feature b/rspec-core/features/command_line/bisect.feature index 82de8d133..7803a5de8 100644 --- a/rspec-core/features/command_line/bisect.feature +++ b/rspec-core/features/command_line/bisect.feature @@ -52,7 +52,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" + Bisect started using options: "--seed 1234 --no-parallel" Running suite to find failures... (0.16755 seconds) Starting bisect with 1 failing example and 9 non-failing examples. Checking that failure(s) are order-dependent... failure appears to be order-dependent @@ -64,7 +64,7 @@ Feature: Bisect Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds. The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234` Then the output should contain "2 examples, 1 failure" @@ -73,7 +73,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect` and abort in the middle with ctrl-c Then bisect should fail with output like: """ - Bisect started using options: "--seed 1234" + Bisect started using options: "--seed 1234 --no-parallel" Running suite to find failures... (0.17102 seconds) Starting bisect with 1 failing example and 9 non-failing examples. Checking that failure(s) are order-dependent... failure appears to be order-dependent @@ -85,7 +85,7 @@ Feature: Bisect Bisect aborted! The most minimal reproduction command discovered so far is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234` Then the output should contain "3 examples, 1 failure" @@ -94,7 +94,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect=verbose` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" and bisect runner: :fork + Bisect started using options: "--seed 1234 --no-parallel" and bisect runner: :fork Running suite to find failures... (0.16528 seconds) - Failing examples (1): - ./spec/calculator_1_spec.rb[1:1] @@ -109,11 +109,11 @@ Feature: Bisect - ./spec/calculator_8_spec.rb[1:1] - ./spec/calculator_9_spec.rb[1:1] Checking that failure(s) are order-dependent.. - - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel (n.nnnn seconds) - Failure appears to be order-dependent Round 1: bisecting over non-failing examples 1-9 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 --no-parallel (0.15302 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 --no-parallel (0.19708 seconds) - Examples we can safely ignore (4): - ./spec/calculator_6_spec.rb[1:1] - ./spec/calculator_7_spec.rb[1:1] @@ -126,8 +126,8 @@ Feature: Bisect - ./spec/calculator_4_spec.rb[1:1] - ./spec/calculator_5_spec.rb[1:1] Round 2: bisecting over non-failing examples 1-5 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 --no-parallel (0.15836 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel (0.19065 seconds) - Examples we can safely ignore (2): - ./spec/calculator_4_spec.rb[1:1] - ./spec/calculator_5_spec.rb[1:1] @@ -136,15 +136,15 @@ Feature: Bisect - ./spec/calculator_2_spec.rb[1:1] - ./spec/calculator_3_spec.rb[1:1] Round 3: bisecting over non-failing examples 1-3 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 --no-parallel (0.21028 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel (0.1975 seconds) - Examples we can safely ignore (1): - ./spec/calculator_2_spec.rb[1:1] - Remaining non-failing examples (2): - ./spec/calculator_10_spec.rb[1:1] - ./spec/calculator_3_spec.rb[1:1] Round 4: bisecting over non-failing examples 1-2 - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel (0.17173 seconds) - Examples we can safely ignore (1): - ./spec/calculator_3_spec.rb[1:1] - Remaining non-failing examples (1): @@ -152,7 +152,7 @@ Feature: Bisect Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds. The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234` Then the output should contain "2 examples, 1 failure" @@ -171,8 +171,8 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect=verbose` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" and bisect runner: :shell + Bisect started using options: "--seed 1234 --no-parallel" and bisect runner: :shell # ... The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature new file mode 100644 index 000000000..f502e4563 --- /dev/null +++ b/rspec-core/features/command_line/parallel.feature @@ -0,0 +1,202 @@ +Feature: `--parallel` option + + Use the `--parallel[=COUNT]` option to run the top-level example groups of + your suite across several worker processes. RSpec loads the suite once, forks + the workers from the process that loaded it, and hands groups out to whichever + worker is free. Results stream back as examples finish, so you get a single + report with one seed, one summary, one profile and one exit code -- produced + by the same formatters that a serial run uses. + + Without a count, one worker per processor is used. Set + `config.parallel_workers` in `RSpec.configure` to make it the default. + + Each worker runs the `before(:suite)` and `after(:suite)` hooks once. In a + worker, `RSpec.configuration.parallel_worker_number` is its 1-based number + and `ENV['TEST_ENV_NUMBER']` follows the convention of parallel_tests (`""` + for the first worker, then `"2"`, `"3"`, ...), so resources that are already + namespaced per worker for that tool keep working. Everything that happens + while the suite is loaded -- such as a Rails `rails_helper` evaluating + `database.yml` and connecting -- happens once, in the coordinating process, + and is inherited by every worker. + + With `--no-parallel-preload` the workers are forked before the spec files + are loaded instead, and each loads them itself with its `TEST_ENV_NUMBER` + already set, so nothing the spec files do while loading is shared. A worker + is given work only once it has loaded the same examples as the coordinating + process. + + A worker that crashes fails the example it was running, and the examples of + its group it had not started run in another worker. Code that calls `exit` + ends the run with that status, as it does in a serial run. A top-level group + tagged `:parallel_split => N` runs as N parts in different workers, which + helps when it is the longest one of the suite. + + `--parallel` needs `fork`. Where it is not available (Windows, JRuby), or + when `TEST_ENV_NUMBER` is already set because another tool is running RSpec + in parallel, RSpec prints a warning and runs serially. + + Background: + Given a file named "spec/alpha_spec.rb" with: + """ruby + RSpec.configure do |config| + config.before(:suite) do + File.write("worker_#{config.parallel_worker_number}", ENV['TEST_ENV_NUMBER'].inspect) + end + end + + RSpec.describe "Alpha" do + it "passes" do + expect(1).to eq(1) + end + + it "fails" do + expect(1).to eq(2) + end + end + """ + And a file named "spec/beta_spec.rb" with: + """ruby + RSpec.describe "Beta" do + it "passes" do + expect(2).to eq(2) + end + + context "nested" do + it "passes too" do + expect(3).to eq(3) + end + end + end + """ + + Scenario: Running a suite across two workers produces one ordinary report + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 1 + And the output should contain "4 examples, 1 failure" + And the output should contain "rspec ./spec/alpha_spec.rb:12 # Alpha fails" + And the output should contain "expected: 2" + And the file "worker_1" should contain exactly: + """ + "" + """ + And the file "worker_2" should contain exactly: + """ + "2" + """ + + Scenario: Formatters that follow group boundaries see each group whole + When I run `rspec --parallel=2 --order defined --format documentation` + Then the exit status should be 1 + And the output should contain: + """ + Beta + passes + nested + passes too + """ + And the output should contain: + """ + Alpha + passes + fails (FAILED - 1) + """ + + Scenario: A worker that finishes but never exits cannot hang the run + Given a file named "spec/hang_spec.rb" with: + """ruby + RSpec.configure do |config| + config.parallel_shutdown_timeout = 1 + + config.after(:suite) do + if config.parallel_worker_number == 2 + thread = Thread.new { sleep } + at_exit { thread.join } + end + end + end + + RSpec.describe "Gamma" do + it "passes" do + expect(4).to eq(4) + end + end + """ + When I run `rspec --parallel=2 --error-exit-code 3 spec/beta_spec.rb spec/hang_spec.rb` + Then the exit status should be 3 + And the output should contain "3 examples, 0 failures" + And the output should match /Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/ + And the stderr should contain "did not exit within 1 second of finishing its last example group" + And the stderr should contain "Threads at this point:" + And the stderr should contain "join" + + Scenario: A worker that crashes leaves the rest of its group to another worker + Given a file named "spec/crash_spec.rb" with: + """ruby + RSpec.describe "Crashing" do + it "dies" do + exit!(7) + end + + it "runs in another worker" do + expect(5).to eq(5) + end + end + """ + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 1 + And the output should contain "6 examples, 2 failures" + And the output should contain "exited unexpectedly (exit status 7) while running this example; 1 example of Crashing" + And the output should contain "rspec ./spec/crash_spec.rb:2 # Crashing dies" + + Scenario: Code that calls `exit` ends the run with that status, as in a serial run + Given a file named "spec/exit_spec.rb" with: + """ruby + RSpec.describe "Exiting" do + it "exits" do + exit 3 + end + end + """ + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 3 + And the output should contain "is ending the run: the code it ran called `exit` or `abort` (status 3)" + + Scenario: A long group tagged `:parallel_split` runs as parts in several workers + Given a file named "spec/long_spec.rb" with: + """ruby + RSpec.describe "Long", :parallel_split => 2 do + 4.times do |i| + it "runs part of the group #{i}" do + File.write("ran_#{i}_in_#{Process.pid}", "") + sleep 0.2 + end + end + end + """ + When I run `rspec --parallel=3 --order defined spec/long_spec.rb spec/beta_spec.rb` + Then the exit status should be 0 + And the output should contain "6 examples, 0 failures" + And the files the examples wrote should name 2 processes + + Scenario: With `--no-parallel-preload` workers run the groups they loaded themselves + Given a file named "spec/gamma_spec.rb" with: + """ruby + File.write("loaded_by_#{ENV.fetch('TEST_ENV_NUMBER', 'the coordinating process')}", "") + + RSpec.describe "Gamma" do + it "runs in a process that loaded this file" do + expect(File).to exist("loaded_by_#{ENV['TEST_ENV_NUMBER']}") + end + end + """ + When I run `rspec --parallel=2 --no-parallel-preload --order defined` + Then the exit status should be 1 + And the output should match /^5 examples, 1 failure$/ + And the file "loaded_by_the coordinating process" should exist + + Scenario: Inside another tool's parallel worker RSpec runs serially + Given I set the environment variable "TEST_ENV_NUMBER" to "2" + When I run `rspec --parallel=2 --order defined spec/beta_spec.rb` + Then the exit status should be 0 + And the output should contain "2 examples, 0 failures" + And the stderr should contain "Ignoring `--parallel`: this process is already a worker of a parallel test run" diff --git a/rspec-core/features/step_definitions/additional_cli_steps.rb b/rspec-core/features/step_definitions/additional_cli_steps.rb index 30850222b..6e8a567a2 100644 --- a/rspec-core/features/step_definitions/additional_cli_steps.rb +++ b/rspec-core/features/step_definitions/additional_cli_steps.rb @@ -278,3 +278,9 @@ def normalize_failure_output(text) World(Normalization) World(FormatterSupport) + +Then(/^the files the examples wrote should name (\d+) processes$/) do |count| + cd('.') do + expect(Dir['ran_*_in_*'].map { |file| file[/\d+\z/] }.uniq.size).to eq(Integer(count)) + end +end diff --git a/rspec-core/lib/rspec/core/bisect/shell_command.rb b/rspec-core/lib/rspec/core/bisect/shell_command.rb index 483a7b291..a77980aa4 100644 --- a/rspec-core/lib/rspec/core/bisect/shell_command.rb +++ b/rspec-core/lib/rspec/core/bisect/shell_command.rb @@ -100,7 +100,7 @@ def parsed_original_cli_options end def load_path - @load_path ||= "-I#{$LOAD_PATH.map { |p| escape(p) }.join(':')}" + @load_path ||= "-I#{$LOAD_PATH.map { |p| escape(p) }.join(File::PATH_SEPARATOR)}" end # Path to the currently running Ruby executable, borrowed from Rake: diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 7cb484433..3057c8a2c 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -231,6 +231,75 @@ def fail_fast=(value) # @return [Boolean] add_setting :fail_if_no_examples + # @macro define_reader + # The number of worker processes to run top-level example groups across + # (default: `nil`, run everything in this process). `:auto` uses one + # worker per processor. Set from the command line with + # `--parallel[=COUNT]`. + # + # Workers are forked from this process once the suite has been loaded, + # so they inherit everything `spec_helper` and the spec files set up at + # load time -- including connections opened then -- unless + # {#parallel_preload} is off. `before(:suite)` and `after(:suite)` + # hooks run once in each worker, where {#parallel_worker_number} and + # `ENV['TEST_ENV_NUMBER']` identify the worker. Formatters run in this + # process only and see the notifications of a normal run. + # + # Needs `fork`; where it is unavailable (Windows, JRuby) the run is + # serial and a warning is printed, as it is when `TEST_ENV_NUMBER` is + # already set because another tool is parallelizing the run. + # + # @return [Integer, :auto, nil] + define_reader :parallel_workers + + # @see parallel_workers + def parallel_workers=(value) + if value.nil? || value == :auto || (value.is_a?(Integer) && value >= 1) + @parallel_workers = value + else + RSpec.warning "Cannot set `RSpec.configuration.parallel_workers` to `#{value.inspect}`; " \ + "it must be a positive integer, `:auto` or `nil`. Running serially" + @parallel_workers = nil + end + end + + # @macro add_setting + # Seconds a parallel worker is given to exit once it has finished its + # last example group -- that is, to run its `after(:suite)` hooks and + # `at_exit` handlers (default: 30). A worker that takes longer dumps the + # backtraces of all its threads to stderr and is aborted, and the run + # exits with {#error_exit_code}. It is also how long a worker forked + # after loading has to report back: one stuck in `fork` (a deadlock in + # a fork handler) is terminated, and its group runs in another worker. + # @return [Numeric] + add_setting :parallel_shutdown_timeout + + # @macro add_setting + # Whether the spec files are loaded before the parallel workers are + # forked (default: `true`), so that they share them. When `false` + # (`--no-parallel-preload`) the workers -- no more than there are spec + # files, but two for one -- are forked as soon as the `--require`d + # files are loaded, and each loads the spec files itself -- once + # `ENV['TEST_ENV_NUMBER']` is set -- as this process does for its + # reporting. Nothing the spec files do while loading (connections, + # threads, system libraries they initialize) is then shared with the + # workers, at the cost of loading them once per worker. A worker that + # does not load the same examples as this process is not used. It, and + # {#parallel_workers}, have to be set before the spec files load (on + # the command line, in `.rspec` or in a `--require`d file); set while + # they load, they come too late, and the run is serial, with a warning. + # @return [Boolean] + add_setting :parallel_preload + + # The 1-based number of the parallel worker this process is, or `nil` + # in the coordinating process and in serial runs. In workers, + # `ENV['TEST_ENV_NUMBER']` is also set, following the convention of + # parallel_tests: `""` for the first worker, then `"2"`, `"3"`, ... so + # that per-worker resource naming written for that tool keeps working. + # @return [Integer, nil] + # @note The writer is for the parallel runner's use. + attr_accessor :parallel_worker_number + # @macro define_reader # Indicates files configured to be required. # @return [Array] @@ -458,6 +527,10 @@ def initialize @failure_exit_code = 1 @error_exit_code = nil # so it can be overridden by failure exit code @fail_if_no_examples = false + @parallel_workers = nil + @parallel_shutdown_timeout = 30 + @parallel_preload = true + @parallel_worker_number = nil @spec_files_loaded = false @backtrace_formatter = BacktraceFormatter.new @@ -527,6 +600,16 @@ def reset_reporter @output_wrapper = nil end + # @private + # Used by the parallel runner to install the reporter that forwards a + # worker's notifications to the coordinating process. + attr_writer :reporter + + # @private + # Used by the parallel runner so that a spawned worker's formatters, + # which never run, do not open the files the coordinating process's do. + attr_writer :formatter_loader + # @private def reset_filters self.filter_manager = FilterManager.new @@ -1924,7 +2007,10 @@ def around(scope=nil, *meta, &block) end # @private - def with_suite_hooks + # `before_teardown` is called first once the suite is done (however it + # ends), before any `after(:suite)` hook: a parallel worker arms its + # exit watchdog there. + def with_suite_hooks(before_teardown=nil) return yield if dry_run? begin @@ -1932,6 +2018,7 @@ def with_suite_hooks run_suite_hooks("a `before(:suite)` hook", @before_suite_hooks) yield ensure + before_teardown&.call RSpec.current_scope = :after_suite_hook run_suite_hooks("an `after(:suite)` hook", @after_suite_hooks) RSpec.current_scope = :suite diff --git a/rspec-core/lib/rspec/core/example.rb b/rspec-core/lib/rspec/core/example.rb index 1477aebe1..ba0d0c2bc 100644 --- a/rspec-core/lib/rspec/core/example.rb +++ b/rspec-core/lib/rspec/core/example.rb @@ -441,6 +441,27 @@ def skip_with_exception(reporter, exception) finish(reporter) end + # @private + # + # Used by the parallel runner to copy onto this instance the result of + # running the same example in a worker process. Exceptions are the + # rebuilt stand-ins described in `Parallel::RemoteException`. + def assign_execution_result(attributes) + result = execution_result + result.status = attributes[:status] + result.started_at = attributes[:started_at] + result.finished_at = attributes[:finished_at] + result.run_time = attributes[:run_time] + result.pending_message = attributes[:pending_message] + result.pending_fixed = attributes[:pending_fixed] + result.pending_exception = attributes[:pending_exception] + result.exception = @exception = attributes[:exception] + + [:description, :full_description, :pending, :skip, :extra_failure_lines].each do |key| + metadata[key] = attributes[key] unless attributes[key].nil? + end + end + # @private def instance_exec(*args, &block) @example_group_instance.instance_exec(*args, &block) diff --git a/rspec-core/lib/rspec/core/filter_manager.rb b/rspec-core/lib/rspec/core/filter_manager.rb index 80b3a3304..ff199128d 100644 --- a/rspec-core/lib/rspec/core/filter_manager.rb +++ b/rspec-core/lib/rspec/core/filter_manager.rb @@ -89,9 +89,14 @@ def add_path_to_arrays_filter(filter_key, path, values) # and there is a `:slow => true` exclusion filter), but only for specs # defined in the same file as the location filters. Excluded specs in # other files should still be excluded. + # + # The emptiness checks come first because `File.expand_path` of a + # relative path asks the OS for the working directory, which is slow + # enough (tens of microseconds on macOS, more when several processes + # do it at once) to dominate filtering a large suite. def file_scoped_include?(ex_metadata, ids, locations) - no_id_filters = ids[ex_metadata[:rerun_file_path]].empty? - no_location_filters = locations[ + no_id_filters = ids.empty? || ids[ex_metadata[:rerun_file_path]].empty? + no_location_filters = locations.empty? || locations[ File.expand_path(ex_metadata[:rerun_file_path]) ].empty? diff --git a/rspec-core/lib/rspec/core/invocations.rb b/rspec-core/lib/rspec/core/invocations.rb index 4719085b3..33f438846 100644 --- a/rspec-core/lib/rspec/core/invocations.rb +++ b/rspec-core/lib/rspec/core/invocations.rb @@ -33,15 +33,39 @@ def call(options, err, out) out, runner.configuration.bisect_runner ) - success = RSpec::Core::Bisect::Coordinator.bisect_with( - runner, options.args, formatter - ) + success = without_parallel_spec_opts do + RSpec::Core::Bisect::Coordinator.bisect_with(runner, cli_args_for(options), formatter) + end runner.exit_code(success) end private + PARALLEL_OPTION = /\A--(?:no-)?parallel(?:=|\z)/ + + # Bisection runs the suite serially, and so does the reproduction + # command it prints -- even when parallel runs are enabled where this + # process cannot see it, in a helper that the spec files require. + def cli_args_for(options) + options.args.reject { |arg| arg =~ PARALLEL_OPTION } + ['--no-parallel'] + end + + # `SPEC_OPTS` win over the command line, and the reproduction command + # and the runs of `--bisect-runner=shell` reuse them. + def without_parallel_spec_opts + original = ENV.fetch('SPEC_OPTS', nil) + if original + require 'shellwords' + args = Shellwords.split(original) + remaining = args.reject { |arg| arg =~ PARALLEL_OPTION } + ENV['SPEC_OPTS'] = remaining.empty? ? nil : Shellwords.join(remaining) unless remaining == args + end + yield + ensure + ENV['SPEC_OPTS'] = original + end + def bisect_formatter_klass_for(argument) return Formatters::BisectDebugFormatter if argument == "verbose" Formatters::BisectProgressFormatter diff --git a/rspec-core/lib/rspec/core/metadata.rb b/rspec-core/lib/rspec/core/metadata.rb index ed47bc07b..db9fbb9de 100644 --- a/rspec-core/lib/rspec/core/metadata.rb +++ b/rspec-core/lib/rspec/core/metadata.rb @@ -143,7 +143,10 @@ def populate_location_attributes end relative_file_path = Metadata.relative_path(file_path) - absolute_file_path = File.expand_path(relative_file_path) + # The path as given rather than the relative one: spec files are + # loaded by their absolute path, which expands without asking the + # OS for the working directory, once per example and group. + absolute_file_path = File.expand_path(file_path) metadata[:file_path] = relative_file_path metadata[:line_number] = line_number.to_i metadata[:location] = "#{relative_file_path}:#{line_number}" diff --git a/rspec-core/lib/rspec/core/option_parser.rb b/rspec-core/lib/rspec/core/option_parser.rb index 57af01707..2667e852c 100644 --- a/rspec-core/lib/rspec/core/option_parser.rb +++ b/rspec-core/lib/rspec/core/option_parser.rb @@ -75,6 +75,14 @@ def parser(options) options[:runner] = RSpec::Core::Invocations::Bisect.new end + parser.on('--[no-]parallel[=COUNT]', 'Run example groups across COUNT worker processes forked after', + ' loading the suite (default: one per processor; 0 runs serially). Needs `fork`.') do |argument| + options[:parallel_workers] = parallel_workers_from(argument) + end + + parser.on('--[no-]parallel-preload', 'Load the spec files before forking the --parallel workers (the default),', + ' or fork first and have every worker load them itself.') { |value| options[:parallel_preload] = value } + parser.on('--[no-]fail-fast[=COUNT]', 'Abort the run after a certain number of failures (1 by default).') do |argument| if argument == true value = 1 @@ -300,6 +308,17 @@ def parser(options) # rubocop:enable Metrics/CyclomaticComplexity # rubocop:enable Metrics/PerceivedComplexity + def parallel_workers_from(argument) + case argument + when true then :auto + when false, /\A0+\z/ then nil + when /\A\d+\z/ then Integer(argument, 10) + else + abort "Expected a non-negative integer for `--parallel`, got: #{argument.inspect}\n\n" \ + "Please use --help for a listing of valid options" + end + end + def add_tag_filter(options, filter_type, tag_name, value=true) (options[filter_type] ||= {})[tag_name] = value end diff --git a/rspec-core/lib/rspec/core/ordering.rb b/rspec-core/lib/rspec/core/ordering.rb index a451c5c6f..d430bbcd3 100644 --- a/rspec-core/lib/rspec/core/ordering.rb +++ b/rspec-core/lib/rspec/core/ordering.rb @@ -160,6 +160,13 @@ def seed=(seed) @seed = seed.to_i end + # @private + # The seed of the parallel run a spawned worker belongs to, taken + # without choosing an ordering: the run's configuration does that. + def adopt_seed(seed) + @seed = seed.to_i unless @seed_forced + end + def order=(type) order, seed = type.to_s.split(':') @seed = seed.to_i if seed diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb new file mode 100644 index 000000000..ed8f7383c --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -0,0 +1,310 @@ +# The runner itself -- parallel/coordinator and the files it requires -- is +# loaded once a run is going to fork workers, so that serial runs skip it. +RSpec::Support.require_rspec_core "parallel/plan" + +module RSpec + module Core + # Runs top-level example groups across worker processes forked from the + # process that loaded the suite (`rspec --parallel[=COUNT]`, + # `config.parallel_workers`). Formatters run in the coordinating process + # only and receive the notifications of a normal run; see {Coordinator}. + # @private + module Parallel + DURATION = /(?:(\d+) minutes? )?([\d.]+) seconds?/ + + # Raised -- as the failure of the example a worker was running, or as + # an error outside of examples -- when a parallel worker is lost: + # it crashed, was killed, or had to be terminated because it did not + # stop or exit in time. + class WorkerError < StandardError; end + + # What a worker is given to run: a top-level example group, or only the + # examples of it with these ids -- the ones a lost worker had not + # started -- which it runs as `rspec path[ids]` would: those examples, + # and the context hooks of the groups that contain them. + # @private + Unit = Struct.new(:group, :example_ids, :retried) do + # @return [Array] the ids of the examples to run + def ids + example_ids || group.descendant_filtered_examples.map(&:id) + end + + def description + "#{group.description} (#{group.location})" + end + end + + # @return [Coordinator, nil] a coordinator for the loaded groups, its + # workers to be forked once it runs, or `nil` when the run should + # proceed serially (not requested, not possible or not worth it). + # Warns when a requested parallel run is not possible. See {Plan} for + # workers forked before loading. + def self.coordinator_for(runner, example_groups) + requested = requested_worker_count(runner.configuration) + return nil unless requested + + units = units_for(example_groups, runner.configuration) + count = [requested, units.size].min + return nil if count < 2 + + RSpec::Support.require_rspec_core "parallel/coordinator" + Coordinator.new(runner, count).tap { |coordinator| coordinator.assign(example_groups, units) } + end + + # Fingerprints of the examples a process loaded -- ids and locations of + # those that will run, in load order -- per spec file, so that a worker + # that loaded the spec files itself is used only if it loaded the same + # examples as the coordinating process, and otherwise the files that + # differ can be named. Descriptions are left out: results are reported + # on the coordinating process's examples, so one computed at load time + # (from the time, a temporary file's path, ...) is simply its own. + # @return [Hash{String => String}] by the file of the top-level group + def self.suite_digest(world) + require 'digest/sha2' + digests = Hash.new { |hash, file| hash[file] = Digest::SHA256.new } + world.example_groups.each do |group| + digest = digests[group.metadata[:file_path]] + group.descendant_filtered_examples.each { |example| digest << example.id << "\0" << example.location << "\n" } + end + digests.transform_values(&:hexdigest) + end + + # @private + def self.requested_worker_count(configuration) + requested = configuration.parallel_workers + return nil if requested.nil? || configuration.dry_run? + + if spawning? && !Runner.command_line + RSpec.warning "Ignoring `--parallel`: without `fork` (#{RUBY_ENGINE}, #{RUBY_PLATFORM}) its workers are new " \ + "`rspec` processes, which takes a run started by the `rspec` command rather than by " \ + "`RSpec::Core::Runner.run`. Running serially", :call_site => nil + return nil + end + + if ENV.key?('TEST_ENV_NUMBER') + RSpec.warning "Ignoring `--parallel`: this process is already a worker of a parallel test run " \ + "(TEST_ENV_NUMBER is set). Running serially", :call_site => nil + return nil + end + + requested == :auto ? default_worker_count : requested + end + + # Without `fork`, workers are new `rspec` processes, which load the spec + # files themselves. + def self.spawning? + !RSpec::Support::RubyFeatures.fork_supported? + end + + # @return [Integer] the number of processors + def self.default_worker_count + require 'etc' + Etc.nprocessors + end + + # The units to dispatch: top-level groups with at least one example to + # run -- or, for a group with `parallel_split` metadata, that many + # parts of it -- longest first when a previous run recorded their + # timings in the example status persistence file, so that the slowest + # unit is never the one that starts last. Units without timings go + # first because nothing is known about them. + # @return [Array] + # @private + def self.units_for(example_groups, configuration) + run_times = recorded_run_times(configuration) + units = example_groups.flat_map do |group| + group.descendant_filtered_examples.empty? ? [] : split(group, run_times) + end + return units if run_times.empty? + + timed = {} + units.each do |unit| + ids = unit.ids + timed[unit] = ids.sum { |id| run_times.fetch(id, 0.0) } if ids.any? { |id| run_times.key?(id) } + end + + unknown, known = units.partition { |unit| !timed.key?(unit) } + unknown + known.each_with_index.sort_by { |unit, index| [-timed[unit], index] }.map(&:first) + end + + # The orderings that order part of a group's examples the way they + # order all of them. + SPLITTABLE_ORDERINGS = [Ordering::Identity, Ordering::Random, Ordering::RecentlyModified].freeze + + # A group tagged `parallel_split: N` is run as N parts -- no more than it + # has examples -- each a slice of the order its examples run in, + # balanced by their recorded run times (by their number, without). + # @return [Array] + # @private + def self.split(group, run_times) + parts = group.metadata[:parallel_split] + return [Unit.new(group)] unless parts + + unless parts.is_a?(Integer) && parts >= 2 + RSpec.warning "Ignoring `parallel_split: #{parts.inspect}` on #{group.description} (#{group.location}): it is " \ + "the number of parts to run the group's examples in, at least 2", :call_site => nil + return [Unit.new(group)] + end + + unless group.descendants.all? { |each| SPLITTABLE_ORDERINGS.any? { |kind| each.ordering_strategy.is_a?(kind) } } + RSpec.warning "Not splitting #{group.description} (#{group.location}) into parts: it orders examples with a " \ + "custom ordering, which may not order part of them the way it orders all of them", + :call_site => nil + return [Unit.new(group)] + end + + ids = execution_order(group) + return [Unit.new(group)] if ids.size < 2 + + slices(ids, [parts, ids.size].min, run_times).map { |slice| Unit.new(group, slice) } + end + + # @return [Array] the ids of the group's examples in the order a + # run of the group runs them in + # @private + def self.execution_order(group) + strategy = group.ordering_strategy + strategy.order(group.filtered_examples).map(&:id) + + strategy.order(group.children).flat_map { |child| execution_order(child) } + end + + # Cuts `ids` into `count` consecutive, non-empty slices of about the + # same total run time -- each example goes to the slice its midpoint + # falls in -- where an example without a recorded time counts as the + # mean of those with one, and with no time recorded at all every + # example counts the same. + # @private + def self.slices(ids, count, run_times) + recorded = ids.map { |id| run_times[id] }.compact + mean = recorded.empty? ? 0.0 : recorded.sum / recorded.size + weights = ids.map { |id| run_times.fetch(id, mean) } + weights = Array.new(ids.size, 1.0) unless weights.sum > 0 + total = weights.sum + + slices = [[]] + done = 0.0 + ids.each_with_index do |id, index| + more = count - slices.size + wanted = ((done + (weights[index] / 2.0)) * count / total).floor + slices << [] if !slices.last.empty? && more > 0 && (wanted >= slices.size || ids.size - index <= more) + slices.last << id + done += weights[index] + end + slices + end + + # @private + def self.recorded_run_times(configuration) + path = configuration.example_status_persistence_file_path + return {} unless path + + ExampleStatusPersister.load_from(path).each_with_object({}) do |example, hash| + seconds = parse_duration(example[:run_time]) + hash[example[:example_id]] = seconds if seconds + end + rescue SystemCallError + {} + end + + # @private + def self.parse_duration(string) + match = DURATION.match(string.to_s) + return nil unless match + + (match[1].to_i * 60) + match[2].to_f + end + + # @private + def self.seconds(duration) + Formatters::Helpers.pluralize(Formatters::Helpers.format_seconds(duration), 'second') + end + + # @private + def self.monotonic_time + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end + + # Longer than any wait of the coordinator's loop or of a worker's + # watchdog: a longer gap means the process was not running -- stopped + # with the whole run (Ctrl-Z), the machine asleep -- and that time does + # not count towards a deadline. + PAUSE_TOLERANCE = 1.0 + + # Sleeps until this process has been running for `seconds`. + # @private + def self.sleep_running(seconds, slice=0.25) + remaining = seconds + while remaining > 0 + step = [remaining, slice].min + started = monotonic_time + sleep step + elapsed = monotonic_time - started + remaining -= elapsed > step + PAUSE_TOLERANCE ? step : elapsed + end + end + + # A `Time` as whole seconds and nanoseconds, the way workers send it: + # marshaling a local `Time` asks libc for its UTC offset, and in a + # process forked on macOS each such call reloads the time zone file + # (tens of microseconds, twice per example). + # @private + def self.time_to_wire(time) + time && [time.tv_sec, time.tv_nsec] + end + + # @private + def self.time_from_wire(value) + value.is_a?(Array) ? ::Time.at(value[0], value[1], :nsec) : value + end + + # Whatever the formatters wrote before `start` (seed, run options) may + # still be buffered in streams that are not `$stdout`; forked workers + # would write it again when they exit. + # @private + def self.flush_output(configuration) + streams = configuration.formatters.map { |formatter| formatter.output if formatter.respond_to?(:output) } + streams.push(configuration.output_stream, configuration.error_stream, configuration.deprecation_stream) + streams.push($stdout, $stderr) + streams.each { |stream| stream.flush if stream.respond_to?(:flush) } + end + + # Handlers that end the process: Ruby's and the system's defaults. Any + # other (`IGNORE`, `nil` -- also an ignored signal --, `EXIT`, a proc) + # stays in force. + ENDS_PROCESS = %w[DEFAULT SYSTEM_DEFAULT].freeze + + # Installs `handler` for each signal the platform has that would end the + # process. Ignored signals (as under `nohup`) and handlers the suite + # installed are left in place. + # @return [Hash{String => Object}] the handlers it replaced + # @private + def self.trap_signals(signals, &handler) + signals.each_with_object({}) do |signal, replaced| + next unless Signal.list.key?(signal) + + previous = trap(signal) { handler.call(signal) } + if ENDS_PROCESS.include?(previous) + replaced[signal] = previous + else + trap(signal, previous) + end + end + end + + # @private + def self.restore_signals(handlers) + handlers.each { |signal, previous| trap(signal, previous) } + end + + # @return the value when `Marshal` can serialize it, otherwise what the + # block makes of it + # @private + def self.marshal_safe(value) + Marshal.dump(value) + value + rescue StandardError + yield value + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/channel.rb b/rspec-core/lib/rspec/core/parallel/channel.rb new file mode 100644 index 000000000..e11815b84 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/channel.rb @@ -0,0 +1,175 @@ +module RSpec + module Core + module Parallel + # Length-prefixed `Marshal` frames over an anonymous pipe, exchanged only + # between the coordinating process and the worker processes it forked + # (the trust boundary `Bisect::Channel` already relies on). The reader + # side never blocks the caller unless asked to: frames are reassembled + # from whatever bytes are available, so a worker stalled mid-frame cannot + # stall the coordinator. + # @private + module Channel + ENCODING = Marshal.dump("").encoding + CHUNK_SIZE = 65_536 + + # Raised when the peer wrote something that is not a frame. + class ProtocolError < StandardError; end + + # @return [Array(Reader, Writer)] the two ends of a new pipe + def self.pipe + read_io, write_io = IO.pipe + [Reader.new(read_io), Writer.new(write_io)] + end + + # @private + class Writer + attr_reader :io + + def initialize(io) + @io = io + # No newline conversion on Windows, where a spawned worker's + # standard input and output start in text mode. + @io.binmode + @io.set_encoding ENCODING + @io.sync = true + end + + def send(message) + packet = Marshal.dump(message) + @io.write("#{packet.bytesize}\n#{packet}") + end + + def closed? + @io.closed? + end + + def close + @io.close unless @io.closed? + end + end + + # @private + class Reader + attr_reader :io, :bytes_read + + def initialize(io) + @io = io + @io.binmode + @io.set_encoding ENCODING + @buffer = String.new(:encoding => ENCODING) + @offset = 0 + @pending = [] + @eof = false + @bytes_read = 0 + end + + def eof? + @eof + end + + def closed? + @io.closed? + end + + def close + @io.close unless @io.closed? + end + + # Reads what is available without blocking. + # @return [Array] decoded messages with their frame size in bytes + def drain + read_available + messages = [] + while (frame = next_frame) + messages << frame + end + messages + end + + # Puts a message obtained from {#drain} back so that {#receive} + # returns it first. + def unread(message) + @pending << message + end + + # Blocks until a whole message has arrived. + # @return [Object, nil] the message, or `nil` once the writer is gone + def receive + return @pending.shift unless @pending.empty? + + loop do + frame = next_frame + return frame.first if frame + return nil if @eof + + # `IO#wait_readable` would need `io/wait`, which rspec-core does not load. + IO.select([@io]) # rubocop:disable Lint/IncompatibleIoSelectWithFiberScheduler + read_available + end + end + + private + + # Reads what the kernel has buffered, a bounded number of chunks at a + # time so that one chatty worker cannot monopolize the coordinator. + def read_available + compact + 16.times do + break if @eof + + chunk = read_chunk + case chunk + when nil then @eof = true + when :wait_readable then break + when :interrupted then next + else + @bytes_read += chunk.bytesize + @buffer << chunk + end + end + end + + # On macOS a read that waits for the pipe while the writer copies a + # large frame into it fails with EINTR when a signal arrives, such as + # the SIGCHLD of a worker exiting. Nothing was read, so it is retried. + def read_chunk + @io.read_nonblock(CHUNK_SIZE, :exception => false) + rescue Errno::EINTR + :interrupted + end + + # Drops the frames already taken from the buffer: once per read + # rather than once per frame, which would copy the rest of up to a + # megabyte of buffered frames for each one. + def compact + return if @offset.zero? + + @buffer = @buffer.byteslice(@offset, @buffer.bytesize - @offset) + @offset = 0 + end + + # rubocop:disable-next Security/MarshalLoad + def next_frame + newline = @buffer.index("\n", @offset) + return nil unless newline + + size = frame_size(@buffer.byteslice(@offset, newline - @offset)) + frame_end = newline + 1 + size + return nil if @buffer.bytesize < frame_end + + packet = @buffer.byteslice(newline + 1, size) + consumed = frame_end - @offset + @offset = frame_end + [Marshal.load(packet), consumed] + end + + def frame_size(header) + Integer(header, 10) + rescue ArgumentError, TypeError + raise ProtocolError, "Expected a frame length, got #{header.inspect}" + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb new file mode 100644 index 000000000..f0795f884 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -0,0 +1,452 @@ +RSpec::Support.require_rspec_core "parallel" +RSpec::Support.require_rspec_core "parallel/channel" +RSpec::Support.require_rspec_core "parallel/remote_exception" +RSpec::Support.require_rspec_core "parallel/worker_reporter" +RSpec::Support.require_rspec_core "parallel/worker" +RSpec::Support.require_rspec_core "parallel/spawned_worker" +RSpec::Support.require_rspec_core "parallel/sequencer" +RSpec::Support.require_rspec_core "parallel/results" +RSpec::Support.require_rspec_core "parallel/suite_index" +RSpec::Support.require_rspec_core "parallel/worker_output" +RSpec::Support.require_rspec_core "parallel/reaper" +RSpec::Support.require_rspec_core "parallel/worker_handle" +RSpec::Support.require_rspec_core "parallel/worker_lifecycle" +RSpec::Support.require_rspec_core "parallel/worker_exits" + +module RSpec + module Core + module Parallel + # The coordinating side of a parallel run. It forks the workers -- after + # the suite has been loaded, so they inherit every example group, filter + # and configuration, or before, so they load the spec files themselves + # (`config.parallel_preload = false`) -- hands out top-level example + # groups from a shared queue, streams the results onto the + # configuration's reporter -- the one and only place formatters run -- + # and makes sure that no worker, however it misbehaves, can keep the run + # from finishing. + # @private + class Coordinator + include WorkerLifecycle + include WorkerExits + + TICK = 0.25 + TERM_GRACE = 5.0 + KILL_GRACE = 5.0 + # Added to the shutdown timeout on the coordinator's side so that the + # worker's own watchdog fires first and the thread dump comes from + # the process that owns the threads. + WATCHDOG_GRACE = 1.0 + # Handled by stopping the workers -- in time, even ones that cannot + # tear down -- before the coordinator goes the way a serial run would. + TERMINATION_SIGNALS = %w[TERM HUP].freeze + + attr_reader :worker_count + + def initialize(runner, worker_count, options={}) + @runner = runner + @configuration = runner.configuration + @world = runner.world + @worker_count = worker_count + @process = options.fetch(:process) { Process } + @spawn = options.fetch(:spawn) { method(Parallel.spawning? ? :spawn_worker : :fork_worker) } + @tick = options.fetch(:tick, TICK) + @term_grace = options.fetch(:term_grace, TERM_GRACE) + @kill_grace = options.fetch(:kill_grace, KILL_GRACE) + @watchdog_grace = options.fetch(:watchdog_grace, WATCHDOG_GRACE) + @reaper_class = options.fetch(:reaper) { Reaper } + @reaper = nil + @workers = [] + @results = Results.new + @quitting = false + @termination_signal = nil + @signal_handlers = {} + @started_at = nil + @reap_due = true + @next_reap_at = 0.0 + @last_turn_at = nil + @output_dir = nil + @boot_allowance = 0.0 + @load_themselves = false + @replacements = 0 + end + + # The groups to run. Workers forked before loading that are more than + # the groups to run are dismissed. + def assign(example_groups, units) + @example_groups = example_groups + @units = units + @queue = (0...@units.size).to_a + @unit_states = Array.new(@units.size, :pending) + @workers.drop(@units.size).each { |worker| dismiss_worker(worker) } + end + + # @return [Integer] the exit code for the run + def run + reporter = @configuration.reporter + @signal_handlers = Parallel.trap_signals(TERMINATION_SIGNALS) { |signal| @termination_signal ||= signal } + loading = !@workers.empty? + start_workers(@units) unless loading + return run_serially if @workers.empty? + + passed = reporter.report(@world.example_count(@example_groups)) do + @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result), :output => WorkerOutput.method(:write)) + + begin + @last_turn_at = now + expect_loading_workers if loading + event_loop + ensure + @winding_down = true + terminate_remaining_workers + @sequencer.finish + @results.apply_all + report_unrun_units + end + + @unit_states.all? { |state| state == :passed } + end + + # Terminated like a serial run would have been, once the workers are gone. + raise SignalException, @termination_signal if @termination_signal + # Ended by `exit` in a worker, as a serial run would have been, + # unless one of them then had to be aborted. + exit(@exit_status) if @exit_status && !@exit_spoiled + + @runner.exit_code(passed) + ensure + begin + terminate_remaining_workers + ensure + clean_up + end + end + + private + + # No worker could be started at all. + def run_serially + Parallel.restore_signals(@signal_handlers) + @signal_handlers = {} + @runner.run_specs(@example_groups) + end + + def event_loop + dispatch_work + + until @workers.all?(&:done?) + return stop_for_signal if @termination_signal + + wait_for_results + reap_workers_when_due + request_quit if @world.wants_to_quit && !@quitting + dispatch_work + account_for_pauses + enforce_deadlines + @sequencer.tick + end + end + + # A run stopped from outside -- a CI job's timeout, say -- most often + # was stuck in an example, so it says what each worker was running. + def stop_for_signal + @winding_down = true + @workers.each { |worker| receive_from(worker) unless worker.done? } + report "Received SIG#{@termination_signal}; stopping the workers." + @workers.each { |worker| (running = worker.running_example_description) && report(running) } + end + + def wait_for_results + readers = @workers.reject { |worker| worker.done? || worker.results.eof? || worker.results.closed? } + + if readers.empty? + # Each worker left is exiting (its results pipe is at its end) or + # was given up on: all there is to do is reap it or enforce its + # deadline, and an exiting worker is reapable within moments. + @reap_due = true + sleep [@tick, 0.01].min + return + end + + ready, = IO.select(readers.map { |worker| worker.results.io }, nil, nil, @tick) + Array(ready).each do |io| + receive_from(readers.find { |worker| worker.results.io.equal?(io) }) + end + end + + def receive_from(worker) + return if worker.results.closed? + + messages = drain(worker) + messages.each { |message, bytes| handle_message(worker, message, bytes) } + + return unless worker.results.eof? + + # A worker whose results pipe closed while it was still running is + # either about to be reaped or wedged; bound the wait for it. + @reap_due = true + worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace + end + + def drain(worker) + worker.results.drain + rescue Channel::ProtocolError, TypeError, ArgumentError, IOError => e + worker.results.close + terminate(worker, "sent data the coordinator could not decode (#{e.class}: #{e.message})") + [] + end + + def handle_message(worker, message, bytes) + kind, *rest = message + number = worker.number + worker_started(worker) if worker.starting + case kind + when :group_started then index.with_group(rest[0]) { |g| group_started(worker, g, rest[1], bytes) } + when :group_finished then index.with_group(rest[0]) { |g| @sequencer.group_finished(number, g, rest[1], bytes) } + when :example_started then index.with_example(rest[0]) { |e| example_started(worker, e, bytes) } + when :example_finished then example_finished(worker, rest[0], rest[1], bytes) + when :message then @sequencer.message(number, rest[0], bytes) + when :deprecation then @sequencer.deprecation(number, rest[0], bytes) + when :custom then @sequencer.custom(number, rest[0], rest[1], bytes) + else handle_control_message(worker, kind, rest) + end + end + + def handle_control_message(worker, kind, rest) + case kind + when :unit_finished then unit_finished(worker, *rest) + when :non_example_failure then non_example_failure(*rest) + when :exiting then exiting(worker, *rest) + when :loaded then loaded(worker, *rest) + when :expectations then expectations(*rest) + when :output then output(worker, *rest) + when :started then nil + else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." + end + end + + # Expectations counted in a worker, for a reporter that counts them + # (`increment_expectation_count`, from an rspec-expectations that + # reports every expectation to it). + def expectations(count) + reporter = @configuration.reporter + return unless reporter.respond_to?(:increment_expectation_count) + + count.times { reporter.increment_expectation_count } + end + + def non_example_failure(payload, context_description) + @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) + end + + # Described as the worker that runs it described it. + def group_started(worker, group, descriptions, bytes) + group.metadata.update(descriptions) if descriptions + worker.last_started_group = group + @sequencer.group_started(worker.number, group, bytes) + end + + def example_started(worker, example, bytes) + worker.current_example = example + worker.started_example_ids[example.id] = true + @sequencer.example_started(worker.number, example, bytes) + end + + def example_finished(worker, example_id, payload, bytes) + index.with_example(example_id) do |example| + worker.current_example = nil + payload[:metadata] = @results.metadata_changes(worker.description, payload[:metadata]) + record_result(example, payload) + @sequencer.example_finished(worker.number, example, bytes) + end + end + + def index + @index ||= SuiteIndex.new(@world) { |warning| report warning } + end + + # Recorded as soon as it arrives: failure counting and `.rspec_status` + # persistence never wait for presentation. + def record_result(example, payload) + @results.record(example, payload) + request_quit if fail_fast_limit_met? + end + + def fail_fast_limit_met?(failed_count=@results.failed_count) + limit = @configuration.fail_fast + return false unless limit + + limit == true ? failed_count >= 1 : failed_count >= limit + end + + def apply_result(example) + @results.apply(example) + end + + # A worker whose world wants to quit would only skip the groups it is + # given (`ExampleGroup.run` returns at once), so it gets no more. The + # events that should end the whole run reach the coordinator by other + # means: an interrupt hits every process in the group, fail-fast is + # counted here and suite hook failures are reported as such. + def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) + @unit_states[index] = passed ? :passed : :failed if worker.current_unit == index + worker.current_unit = nil + @world.non_example_failure = true if non_example_failure + return unless wants_to_quit && !@quitting && !@world.wants_to_quit + + report "#{worker.description} stopped after #{unit_description(index)} because `RSpec.world.wants_to_quit` was " \ + "set while it ran (an interrupt, a suite hook failure or a spec setting it); it runs no further groups." + stop(worker) + end + + # A worker that runs out of work -- or is still loading -- waits while + # another one is busy: if that one is lost, what it had not started + # goes back to the queue. + def dispatch_work + busy = @workers.count(&:busy?) + @workers.each do |worker| + if worker.loading + # Nothing is left to give it once it has loaded. + dismiss_worker(worker) if worker.state == :running && nothing_left?(busy) + elsif worker.idle? + if !@quitting && !@queue.empty? + dispatch(worker, @queue.shift) + busy += 1 if worker.busy? + elsif nothing_left?(busy) + stop(worker) + end + end + end + end + + # Nor will be: no busy worker is left to hand any back. + def nothing_left?(busy) + @quitting || (@queue.empty? && busy.zero?) + end + + def dispatch(worker, index) + worker.current_unit = index + worker.started_example_ids = {} + worker.last_started_group = nil + @unit_states[index] = :running + unit = @units[index] + return if write(worker, [:run, index, unit.group.id, unit.example_ids]) + + @unit_states[index] = :pending + @queue.unshift(index) + worker.current_unit = nil + worker.state = :stopping + end + + def stop(worker) + worker.state = :stopping + worker.deadline = now + @configuration.parallel_shutdown_timeout + @watchdog_grace + write(worker, [:stop]) + end + + # Busy workers are asked to stop between examples and, like workers + # that have finished, get a deadline: a cancelled run ends even if a + # worker is stuck inside an example. + def request_quit + return if @quitting + + @quitting = true + @world.wants_to_quit = true + @workers.each do |worker| + next if worker.quit_sent || !worker.busy? + + worker.quit_sent = true + worker.deadline = now + @configuration.parallel_shutdown_timeout + write(worker, [:quit]) + end + end + + # Control messages are a few bytes; a worker that is not reading them is + # dead or wedged: reaping or the deadline set here deals with it, and + # `judge_exit` still sees the state it was in. A worker that is merely + # exiting gets the usual grace period to do so. + def write(worker, message) + worker.jobs.send(message) + true + rescue Errno::EPIPE, IOError + worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace + false + end + + def reap_workers + @workers.each do |worker| + next if worker.done? || worker.pid.nil? + + pid, status = worker.reap(@process) + next unless pid + + receive_from(worker) + drain_output(worker) + worker.exit_status = status + judge_exit(worker, status) + worker.state = :exited + @reaper&.forget(worker.pid) + replace(worker) + end + end + + def enforce_deadlines + @workers.each do |worker| + next if worker.done? || worker.deadline.nil? || now < worker.deadline + + escalate(worker) + end + end + + def escalate(worker) + case worker.escalation + when nil + spoil_exit + overdue(worker) + worker.state = :terminating + worker.signal(@process, 'TERM', :term, now + @term_grace) + when :term + report "#{worker.description} is still alive #{Parallel.seconds(@term_grace)} after SIGTERM; sending SIGKILL." + worker.signal(@process, 'KILL', :kill, now + @kill_grace) + when :kill + report "#{worker.description} is still alive #{Parallel.seconds(@kill_grace)} after SIGKILL; giving up on it." + drain_output(worker) + abort_unit(worker) + worker.state = :abandoned + replace(worker) + end + end + + # Counted as an error outside of examples, like a failing suite hook. + def error(text) + spoil_exit + @configuration.reporter.notify_non_example_exception(WorkerError.new(text), 'An error occurred in a parallel worker.', + :failure_lines => [text]) + end + + def abort_unit(worker) + if (index = worker.current_unit) + @unit_states[index] = :aborted + worker.current_unit = nil + end + @sequencer&.abandon(worker.number) + end + + def unit_description(index) + @units[index].description + end + + def pluralize(count, noun) + Formatters::Helpers.pluralize(count, noun) + end + + def report(message) + @configuration.reporter.message(message) + end + + def now + Parallel.monotonic_time + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb new file mode 100644 index 000000000..6ecfde38f --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -0,0 +1,93 @@ +module RSpec + module Core + module Parallel + # Decides whether a run is parallel and when its workers are forked: + # once the spec files have been loaded, as by default, or before, when + # each worker is to load them itself (`config.parallel_preload = false`). + # It keeps what it decided before loading for when they have loaded, + # and the workers it forked until the run uses or dismisses them. + # @private + class Plan + def initialize(runner) + @runner = runner + @configuration = runner.configuration + @coordinator = nil + @decided = false + @postponed_by = nil + end + + # Called once the run is configured, before the spec files are loaded. + def before_loading + return if @configuration.parallel_preload? && !Parallel.spawning? + return @postponed_by = '`config.parallel_workers`' if @configuration.parallel_workers.nil? + # Such a run aborts, with `exit!`, as soon as the spec files have loaded. + if @configuration.only_failures_but_not_configured? + return @postponed_by = '`config.example_status_persistence_file_path` (for `--only-failures`)' + end + + @decided = true + count = worker_count + return unless count && count >= 2 + + RSpec::Support.require_rspec_core "parallel/coordinator" + # Assigned first, so that workers forked before a `fork` that fails + # are dismissed too. + @coordinator = Coordinator.new(@runner, count) + @coordinator.start + @coordinator = nil unless @coordinator.started? + end + + # @return [Coordinator, nil] the coordinator to run the loaded groups, + # or `nil` for a serial run + def coordinator_for(example_groups) + return assign(example_groups) if @decided + return Parallel.coordinator_for(@runner, example_groups) if @configuration.parallel_preload? && !Parallel.spawning? + + # Too late to fork before loading, and forking after loading is + # what this run was configured not to do. + requested = Parallel.requested_worker_count(@configuration) + warn_late if requested && requested >= 2 + nil + end + + # Ends the workers forked before loading that the run did not use. + def dismiss + @coordinator&.dismiss + end + + private + + # Each worker loads every spec file, so there are no more of them + # than spec files -- a spec file usually defines one top-level group + # -- but two for a single file, which may define several. + def worker_count + requested = Parallel.requested_worker_count(@configuration) + return requested unless requested && requested > 2 + + @configuration.files_to_run.uniq.size.clamp(2, requested) + end + + def assign(example_groups) + return nil unless @coordinator + + units = Parallel.units_for(example_groups, @configuration) + if units.size < 2 + dismiss + return nil + end + + @coordinator.assign(example_groups, units) + @coordinator + end + + def warn_late + what = @postponed_by || (Parallel.spawning? ? 'a run without `fork`' : '`config.parallel_preload = false`') + RSpec.warning "Ignoring `--parallel`: #{what} was set while the spec files were loading, too late to fork " \ + "the workers before them as `config.parallel_preload = false` asks. Set it before they " \ + "load: on the command line, in `.rspec` or in a `--require`d file. Running serially", + :call_site => nil + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/reaper.rb b/rspec-core/lib/rspec/core/parallel/reaper.rb new file mode 100644 index 000000000..4e63d716b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/reaper.rb @@ -0,0 +1,122 @@ +module RSpec + module Core + module Parallel + # A small process that outlives the coordinator, which it watches: if + # the coordinator is killed (SIGKILL, the OOM killer) while a worker + # cannot notice by itself -- one stuck before running any of RSpec's + # code, as a deadlock in a `fork` handler leaves it, or stopped -- it + # sends the workers it was told about SIGTERM, then SIGKILL, and + # removes their output files. It is started before the first worker is + # forked, by `spawn` (no fork handlers run), and only where workers are + # forked: spawned workers watch the coordinator's pid themselves. A + # worker it was not told about yet -- the coordinator was killed + # between forking it and telling the reaper -- is not covered. + # @private + class Reaper + POLL = 0.25 + # Workers that can leave by themselves do, within their own poll. + OWN_EXIT_GRACE = 3 * Worker::COORDINATOR_POLL + TERM_GRACE = 5.0 + + # Ignores the signals that a terminal or a job's timeout sends to + # the whole process group, which would end it with the coordinator. + # Polls its parent; when that changes, gives the workers still + # registered whose process group is the run's a moment to leave by + # themselves, signals those that did not, then exits. + SCRIPT = <<-'RUBY'.freeze + %w[INT TERM HUP QUIT].each { |signal| trap(signal, 'IGNORE') if Signal.list.key?(signal) } + coordinator, group, output_dir = ARGV[0].to_i, ARGV[1].to_i, ARGV[2] + poll, own_exit, grace = ARGV[3].to_f, ARGV[4].to_f, ARGV[5].to_f + workers = {} + buffer = +'' + loop do + if IO.select([STDIN], nil, nil, poll) + chunk = (STDIN.read_nonblock(4096) rescue nil) + break if chunk.nil? + buffer << chunk + while (line = buffer.slice!(/\A[^\n]*\n/)) + exit!(0) if line == "done\n" + pid = line[1..-1].to_i + line.start_with?('+') ? workers[pid] = true : workers.delete(pid) + end + end + break if Process.ppid != coordinator + end + exit!(0) if Process.ppid == coordinator + ours = lambda do |pid| + Process.getpgid(pid) == group + rescue SystemCallError + false + end + waiting = workers.keys.select(&ours) + until_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) + own_exit + sleep 0.05 while waiting.any?(&ours) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < until_time + left = waiting.select(&ours) + left.each { |pid| Process.kill('TERM', pid) rescue nil } + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + grace + sleep 0.05 while left.any?(&ours) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline + left.select(&ours).each { |pid| Process.kill('KILL', pid) rescue nil } + unless left.empty? + STDERR.puts "RSpec's parallel run (pid #{coordinator}) ended without stopping its workers; stopped " \ + "#{left.size} still running (pids #{left.join(', ')})." + end + if output_dir && !output_dir.empty? + require 'fileutils' + FileUtils.remove_entry(output_dir, true) + end + exit!(0) + RUBY + + # @return [Reaper, nil] one watching this process, or `nil` when it + # could not be started -- the run goes on without it + def self.start(output_dir, process=Process) + read, write = IO.pipe + pid = process.spawn(RbConfig.ruby, '--disable-gems', '-e', SCRIPT, Process.pid.to_s, Process.getpgrp.to_s, + output_dir.to_s, POLL.to_s, OWN_EXIT_GRACE.to_s, TERM_GRACE.to_s, + :in => read, :out => File::NULL, :close_others => true) + new(pid, write, process) + rescue SystemCallError, NotImplementedError + write&.close + nil + ensure + read&.close + end + + attr_reader :pid + + def initialize(pid, input, process=Process) + @pid = pid + @input = input + @input.sync = true + @process = process + end + + def watch(pid) + tell("+#{pid}") + end + + def forget(pid) + tell("-#{pid}") + end + + # Ends it, and reaps it if it has gone within a second. + def finish + tell('done') + @input.close + deadline = Parallel.monotonic_time + 1.0 + sleep 0.01 until @process.waitpid(@pid, Process::WNOHANG) || Parallel.monotonic_time > deadline + rescue SystemCallError + nil + end + + private + + def tell(line) + @input.write("#{line}\n") unless @input.closed? + rescue SystemCallError, IOError + nil # it is gone; the run goes on without it + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/remote_exception.rb b/rspec-core/lib/rspec/core/parallel/remote_exception.rb new file mode 100644 index 000000000..18eec697c --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/remote_exception.rb @@ -0,0 +1,213 @@ +module RSpec + module Core + module Parallel + # Carries the parts of an exception that formatters consume from a worker + # process to the coordinator without ever marshaling the exception itself + # (user exceptions routinely hold IO objects, anonymous classes or large + # object graphs). The coordinator rebuilds an exception whose class + # reports the original name -- and, when that class is loaded in the + # coordinating process, is a subclass of it, so `rescue`-style and + # `is_a?` checks in formatters keep working. + # @private + module RemoteException + MAX_CAUSE_DEPTH = 10 + CONSTANT_NAME = /\A[A-Z]\w*(::[A-Z]\w*)*\z/ + + # @return [Hash, nil] a `Marshal`-safe description of `exception` + def self.payload_for(exception, depth=0) + return nil if exception.nil? + + payload = { + :class_name => exception.class.name, + :message => message_for(exception), + :detailed_message => detailed_message_for(exception), + :backtrace => backtrace_for(exception), + :cause => cause_payload_for(exception, depth) + } + + if MultipleExceptionError::InterfaceTag === exception + payload[:multiple] = multiple_payload_for(exception, depth) + end + + payload + end + + # @return [Exception, nil] an exception rebuilt from `payload_for`'s output + def self.rebuild(payload) + return nil unless payload + + exception = class_for(payload).allocate + ::Exception.instance_method(:initialize).bind_call(exception, payload[:message]) + exception.set_backtrace(payload[:backtrace]) if payload[:backtrace] + exception.instance_variable_set(:@__rspec_remote_payload, payload) + exception + end + + # @private + def self.message_for(exception) + exception.message.to_s + rescue Exception => e # rubocop:disable Lint/RescueException + "A #{exception.class} for which `exception.message.to_s` raises #{e.class}." + end + + # @private + def self.detailed_message_for(exception) + return nil unless exception.respond_to?(:detailed_message) + + exception.detailed_message(:highlight => false).to_s + rescue Exception # rubocop:disable Lint/RescueException + nil + end + + # @private + def self.backtrace_for(exception) + exception.backtrace&.map(&:to_s) + end + + # @private + def self.cause_payload_for(exception, depth) + return nil if depth >= MAX_CAUSE_DEPTH + + cause = exception.cause + cause && payload_for(cause, depth + 1) + end + + # @private + def self.multiple_payload_for(exception, depth) + { + :all_exceptions => exception.all_exceptions.map { |e| payload_for(e, depth + 1) }, + :aggregation_metadata => Parallel.marshal_safe(exception.aggregation_metadata) do |metadata| + { :hide_backtrace => !!(metadata && metadata[:hide_backtrace]) } + end, + :aggregation_block_label => exception.aggregation_block_label&.to_s, + :summary => exception.summary.to_s, + :exception_count_description => exception.exception_count_description.to_s + } + end + + # @private + def self.class_for(payload) + name = payload[:class_name] + base = resolve(name) || StandardError + multiple = payload.key?(:multiple) + + @classes ||= {} + @classes[[name, base, multiple]] ||= build_class(name, base, multiple) + end + + # @private + # Classes defined while loading the suite exist in the coordinator too, + # since workers are forked from it; classes defined while running an + # example do not and fall back to a `StandardError` stand-in. + def self.resolve(name) + return nil unless name && CONSTANT_NAME =~ name + + klass = Object.const_get(name) + klass if klass.is_a?(Class) && klass <= ::Exception + rescue StandardError, ScriptError + nil + end + + # @private + def self.build_class(name, base, multiple) + Class.new(base) do + include Remote + + if multiple + include MultipleExceptionError::InterfaceTag unless base <= MultipleExceptionError::InterfaceTag + include RemoteMultiple + end + + define_singleton_method(:name) { name } + define_singleton_method(:to_s) { name || super() } + define_singleton_method(:inspect) { name || super() } + end + end + + # Instance methods of every rebuilt exception. + # @private + module Remote + BACKTRACE = ::Exception.instance_method(:backtrace) + BACKTRACE_LOCATIONS = ::Exception.instance_method(:backtrace_locations) + SET_BACKTRACE = ::Exception.instance_method(:set_backtrace) + + def message + remote_payload[:message] + end + + def to_s + remote_payload[:message] + end + + def inspect + "#<#{self.class.name}: #{message}>" + end + + def detailed_message(*) + remote_payload[:detailed_message] || message + end + + def cause + @cause ||= RemoteException.rebuild(remote_payload[:cause]) + end + + # The original class may keep its backtrace in state the rebuilt + # instance does not have (`Minitest::UnexpectedError` delegates to + # the error it wraps), so use the one set from the payload -- or by + # `set_backtrace` later on. + def backtrace + BACKTRACE.bind_call(self) + end + + def set_backtrace(backtrace) # rubocop:disable Naming/AccessorMethodName + SET_BACKTRACE.bind_call(self, backtrace) + end + + def backtrace_locations + BACKTRACE_LOCATIONS.bind_call(self) + end + + private + + def remote_payload + @__rspec_remote_payload + end + end + + # Instance methods of rebuilt exceptions that aggregate several failures. + # @private + module RemoteMultiple + def all_exceptions + @all_exceptions ||= remote_payload[:multiple][:all_exceptions].map do |payload| + RemoteException.rebuild(payload) + end + end + + def failures + all_exceptions.select { |e| e.class.name =~ /RSpec/ } + end + + def other_errors + all_exceptions.reject { |e| e.class.name =~ /RSpec/ } + end + + def aggregation_metadata + remote_payload[:multiple][:aggregation_metadata] + end + + def aggregation_block_label + remote_payload[:multiple][:aggregation_block_label] + end + + def summary + remote_payload[:multiple][:summary] + end + + def exception_count_description + remote_payload[:multiple][:exception_count_description] + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/results.rb b/rspec-core/lib/rspec/core/parallel/results.rb new file mode 100644 index 000000000..9c1c776d6 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/results.rb @@ -0,0 +1,87 @@ +module RSpec + module Core + module Parallel + # The results workers report, kept from the moment they arrive -- so + # failure counting and `.rspec_status` persistence never wait for + # presentation -- and copied onto the examples when their notifications + # are delivered (or at the end of the run for whatever was not). + # @private + class Results + attr_reader :failed_count + + def initialize + @payloads = {} + @failed_count = 0 + @unsent_metadata = {} + end + + def record(example, payload) + @payloads[example] = payload + @failed_count += 1 if payload[:status] == :failed + end + + def apply(example) + payload = @payloads.delete(example) + return unless payload + + apply_metadata(example.metadata, payload[:metadata]) + example.assign_execution_result( + payload.merge( + :started_at => Parallel.time_from_wire(payload[:started_at]), + :finished_at => Parallel.time_from_wire(payload[:finished_at]), + :exception => RemoteException.rebuild(payload[:exception]), + :pending_exception => RemoteException.rebuild(payload[:pending_exception]) + ) + ) + end + + # What a worker sent of an example's metadata changes, loaded one by + # one: a value whose class is not loaded here is left out -- with a + # warning, once per key -- like one the worker could not marshal. + # @return [Hash, nil] the keys set (`:set`) and deleted (`:deleted`) + def metadata_changes(worker_description, changes) + return nil unless changes + + unsent = changes.fetch(:unsent, {}).dup + load = lambda do |(key, dump)| + # rubocop:disable-next Security/MarshalLoad + Marshal.load(dump) # dumped by the worker's own reporter + rescue StandardError => e + unsent[key] = "it cannot be loaded here (#{e.class}: #{e.message[0, 200]})" + nil + end + set = changes.fetch(:set, []).map(&load).compact.to_h + deleted = changes.fetch(:deleted, []).map(&load).compact + warn_unsent_metadata(worker_description, unsent) + { :set => set, :deleted => deleted } + end + + # What the example's run in a worker did to its metadata, done to the + # coordinator's copy before formatters are told about it. + def apply_metadata(metadata, changes) + return unless changes + + changes.fetch(:set, {}).each { |key, value| metadata[key] = value } + changes.fetch(:deleted, []).each { |key| metadata.delete(key) } + end + + def apply_all + @payloads.keys.each { |example| apply(example) } + end + + private + + def warn_unsent_metadata(worker_description, unsent) + unsent.each do |key, reason| + next if @unsent_metadata.key?(key) + + @unsent_metadata[key] = true + RSpec.warning "#{worker_description} set example metadata #{key} to a value that cannot reach the " \ + "formatters (#{reason}), so they see the value it had before the example ran. Later such " \ + "values of #{key} are not reported", :call_site => nil + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb new file mode 100644 index 000000000..f252b5a24 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -0,0 +1,306 @@ +module RSpec + module Core + module Parallel + # Replays worker notifications onto the coordinator's reporter. + # + # Formatters that follow example group boundaries (documentation, HTML, + # ...) need every notification of a top-level group to arrive between + # that group's `example_group_started` and `example_group_finished`. + # With several workers running at once this holds only if their streams + # are interleaved at group boundaries, so when such a formatter is + # registered one in-flight group at a time is "live" and delivered as it + # happens while the others are queued and replayed whole once they + # finish -- the output then reads like a serial run of the groups in + # completion order. Formatters that do not care about group boundaries + # (progress, JSON, failures, ...) get every notification immediately. + # + # Liveness is bounded in both modes: when the live group has been silent + # for {SILENCE_BOUND} seconds, when a finished group has been waiting + # for {WAIT_BOUND} seconds behind a live group that keeps producing + # output, or when more than {QUEUE_BOUND} notifications or + # {QUEUE_BYTES_BOUND} bytes of them are queued, waiting notifications + # are delivered right away. Formatters then see + # the live group closed and re-opened around them (its headings + # repeat), which is the price of showing results promptly. + # + # Only presentation is sequenced: results, failure counts and example + # statuses are recorded by the coordinator as they arrive. + # @private + class Sequencer + SILENCE_BOUND = 2.0 + WAIT_BOUND = 10.0 + QUEUE_BOUND = 5_000 + QUEUE_BYTES_BOUND = 8 * 1024 * 1024 + CONTEXT_FREE = [:message, :deprecation, :custom].freeze + # Shown under the groups it was printed in; outside any, as it comes. + OUTSIDE_GROUPS = (CONTEXT_FREE + [:output]).freeze + + Event = Struct.new(:kind, :target, :payload, :bytes) + + # The notifications of one top-level group run by one worker. + # @private + class Stream + attr_reader :worker, :events, :open_groups + attr_accessor :depth, :complete, :completed_at + + def initialize(worker) + @worker = worker + @events = [] + @open_groups = [] + @depth = 0 + @complete = false + @completed_at = nil + end + end + + attr_reader :stats + + def initialize(reporter, options={}) + @reporter = reporter + @apply_result = options.fetch(:apply_result) + @write_output = options.fetch(:output) { lambda { |_stream, _bytes| } } + @clock = options.fetch(:clock) { Parallel.method(:monotonic_time) } + @sequenced = options.fetch(:sequenced) { self.class.group_aware_listeners?(reporter) } + @streams = {} + @completed = [] + @live = nil + @visible = nil + @visible_groups = [] + @last_delivery_at = @clock.call + @group_durations = {} + @stats = { :queued => 0, :peak_queued => 0, :queued_bytes => 0, :peak_queued_bytes => 0, :flushes => 0 } + end + + # @return [Boolean] whether a registered formatter reacts to group + # boundaries. `BaseFormatter#example_group_started` only records the + # current group and RSpec's profiler keeps its own per-group tally, + # so neither needs sequencing. + def self.group_aware_listeners?(reporter) + # Not loaded when every formatter is a custom one that does not inherit it. + base_formatter = Formatters::BaseFormatter if defined?(Formatters::BaseFormatter) + [:example_group_started, :example_group_finished].any? do |event| + reporter.registered_listeners(event).any? do |listener| + !listener.is_a?(Profiler) && listener.method(event).owner != base_formatter + end + end + end + + def sequenced? + @sequenced + end + + def group_started(worker, group, bytes=0) + push(worker, Event.new(:group_started, group, nil, bytes)) { |stream| stream.depth += 1 } + end + + def group_finished(worker, group, duration, bytes=0) + push(worker, Event.new(:group_finished, group, duration, bytes)) do |stream| + stream.depth -= 1 + stream.complete = true if stream.depth <= 0 + end + end + + def example_started(worker, example, bytes=0) + push(worker, Event.new(:example_started, example, nil, bytes)) + end + + def example_finished(worker, example, bytes=0) + push(worker, Event.new(:example_finished, example, nil, bytes)) + end + + def message(worker, message, bytes=0) + push(worker, Event.new(:message, nil, message, bytes)) + end + + def deprecation(worker, hash, bytes=0) + push(worker, Event.new(:deprecation, nil, hash, bytes)) + end + + def custom(worker, event, options, bytes=0) + push(worker, Event.new(:custom, event, options, bytes)) + end + + # What the worker printed to `stream` (`:out` or `:err`). + def output(worker, stream, bytes) + push(worker, Event.new(:output, stream, bytes, bytes.bytesize)) + end + + # The worker died (or was stopped) in the middle of a group: deliver + # what it reported and balance the group notifications formatters saw. + def abandon(worker) + stream = @streams.delete(worker) + return unless stream + + replay(stream) + close_visible if @visible.equal?(stream) + @live = nil if @live.equal?(stream) + promote + end + + # Called periodically by the coordinator. + def tick(now=@clock.call) + return unless @sequenced + return unless @completed.any? + + flush_completed if now - @last_delivery_at >= SILENCE_BOUND || now - @completed.first.completed_at >= WAIT_BOUND + end + + # Delivers everything still queued and closes any open group. + def finish + flush_everything + @streams.keys.each { |worker| abandon(worker) } + close_visible + end + + private + + def push(worker, event) + stream = @streams[worker] + + # Messages sent outside any group (suite hooks) are not sequenced. + if OUTSIDE_GROUPS.include?(event.kind) && (stream.nil? || stream.depth <= 0) + return deliver(Stream.new(worker), event) + end + + stream ||= (@streams[worker] = Stream.new(worker)) + yield stream if block_given? + + if live?(stream) + deliver(stream, event) + else + enqueue(stream, event) + end + + complete(stream) if stream.complete + end + + def live?(stream) + return true unless @sequenced + + @live = stream if @live.nil? + @live.equal?(stream) + end + + def complete(stream) + @streams.delete(stream.worker) + + if stream.events.empty? + @live = nil if @live.equal?(stream) + promote + else + stream.completed_at = @clock.call + @completed << stream + end + end + + def enqueue(stream, event) + stream.events << event + @stats[:queued] += 1 + @stats[:queued_bytes] += event.bytes + @stats[:peak_queued] = [@stats[:peak_queued], @stats[:queued]].max + @stats[:peak_queued_bytes] = [@stats[:peak_queued_bytes], @stats[:queued_bytes]].max + # At once: output read in chunks between two frames must not pile up. + flush_everything if @stats[:queued] >= QUEUE_BOUND || @stats[:queued_bytes] >= QUEUE_BYTES_BOUND + end + + def deliver(stream, event) + if CONTEXT_FREE.include?(event.kind) + deliver_context_free(event) + elsif event.kind == :output && stream.open_groups.empty? + @write_output.call(event.target, event.payload) + else + switch_to(stream) unless @visible.equal?(stream) + deliver_in_context(stream, event) + end + + @last_delivery_at = @clock.call + end + + def deliver_in_context(stream, event) + case event.kind + when :group_started + @reporter.example_group_started(event.target) + stream.open_groups.push(event.target) + @visible_groups.push(event.target) + when :group_finished + @reporter.example_group_finished(event.target) + stream.open_groups.pop + @visible_groups.pop + record_duration(event.target, event.payload) if event.payload + when :example_started + @reporter.example_started(event.target) + when :output + @write_output.call(event.target, event.payload) + when :example_finished + @apply_result.call(event.target) + @reporter.example_finished(event.target) + @reporter.__send__(:"example_#{event.target.execution_result.status}", event.target) + end + end + + # Measured in the worker that ran it; a group split across workers + # took as long as its parts did together. + def record_duration(group, duration) + total = @group_durations[group] = @group_durations.fetch(group, 0.0) + duration + @reporter.record_example_group_duration(group, total) + end + + def deliver_context_free(event) + case event.kind + when :message then @reporter.message(event.payload) + when :deprecation then @reporter.deprecation(event.payload) + when :custom then @reporter.publish(event.target, event.payload) + end + end + + def replay(stream) + until stream.events.empty? + event = stream.events.shift + @stats[:queued] -= 1 + @stats[:queued_bytes] -= event.bytes + deliver(stream, event) + end + end + + def switch_to(stream) + close_visible + stream.open_groups.each { |group| @reporter.example_group_started(group) } + @visible_groups = stream.open_groups.dup + @visible = stream + end + + def close_visible + @visible_groups.reverse_each { |group| @reporter.example_group_finished(group) } + @visible_groups = [] + @visible = nil + end + + # The live group finished: show the groups that finished meanwhile, + # then let the longest-waiting in-flight group go live. + def promote + flush_completed + return unless @live.nil? + + waiting = @streams.values.select { |stream| stream.events.any? } + return if waiting.empty? + + @live = waiting.first + replay(@live) + end + + def flush_completed + @stats[:flushes] += 1 if @completed.any? + while (stream = @completed.shift) + replay(stream) + end + end + + def flush_everything + flush_completed + @stats[:flushes] += 1 if @streams.values.any? { |stream| stream.events.any? } + @streams.values.each { |stream| replay(stream) } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb new file mode 100644 index 000000000..062878906 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -0,0 +1,77 @@ +module RSpec + module Core + module Parallel + # Where Ruby has no `fork` (Windows, JRuby), a worker is a + # new process running the `rspec` command the run was started with: it + # configures itself from the same arguments, `.rspec` files and + # `SPEC_OPTS` -- without formatters, which are the coordinator's -- and + # the run's seed, then loads the spec files and goes on like a worker + # forked before loading. Its jobs arrive on its standard input and its + # results leave on its standard output, so what it prints goes to its + # output files from the start. Configuration done in-process before + # `RSpec::Core::Runner.run` cannot reach it, which is why only a run + # started by the `rspec` command starts workers this way. + # @private + module SpawnedWorker + VARIABLES = %w[ + RSPEC_PARALLEL_WORKER RSPEC_PARALLEL_COORDINATOR RSPEC_PARALLEL_SEED + RSPEC_PARALLEL_STDOUT RSPEC_PARALLEL_STDERR + ].freeze + # Options that open files the coordinator writes to. + NOT_FOR_WORKERS = [:formatters, :deprecation_stream].freeze + + # @return [Hash{String => String}] what a worker numbered `number` is + # told through its environment + def self.environment(number, coordinator, seed, output) + env = { 'RSPEC_PARALLEL_WORKER' => number.to_s, 'RSPEC_PARALLEL_COORDINATOR' => coordinator.to_s, + 'RSPEC_PARALLEL_SEED' => seed.to_s, 'TEST_ENV_NUMBER' => Worker.test_env_number(number) } + env.update('RSPEC_PARALLEL_STDOUT' => output[:out], 'RSPEC_PARALLEL_STDERR' => output[:err]) if output + env + end + + # Taken out of the environment, so that the processes it starts are + # not workers. + # @return [Hash] + def self.settings(env=ENV) + number, coordinator, seed, stdout, stderr = VARIABLES.map { |name| env.delete(name) } + { :number => Integer(number), :coordinator => Integer(coordinator), :seed => seed && Integer(seed), + :output => stdout && stderr && { :out => stdout, :err => stderr } } + end + + def self.configure(args, settings, configuration=RSpec.configuration) + configuration.parallel_worker_number = settings[:number] + configuration.ordering_manager.adopt_seed(settings[:seed]) if settings[:seed] + # Also for formatters its `--require`d files add. + configuration.formatter_loader = Worker::FormatterLoader.new(Reporter.new(configuration)) + options = ConfigurationOptions.new(args) + NOT_FOR_WORKERS.each { |key| options.options.delete(key) } + options.configure(configuration) + end + + # Never returns. + # :nocov: - runs in the spawned worker, covered by integration specs + def self.run(args) + settings = settings() + # rubocop:disable Style/GlobalStdStream + # The process's own streams, which the coordinator connected. + jobs = Channel::Reader.new(STDIN.dup) + results = Channel::Writer.new(STDOUT.dup) + STDIN.reopen(File::NULL) + # Its standard output must not be where its results go. + STDOUT.reopen(STDERR) unless settings[:output] + # rubocop:enable Style/GlobalStdStream + Runner.disable_autorun! + loading = Worker::LoadReporter.new(RSpec.configuration) + worker = Worker.new(settings[:number], nil, jobs, results, + :output => settings[:output], :coordinator => settings[:coordinator], :loading => loading) + worker.capture_output + worker.report_started + RSpec.configuration.reporter = loading + configure(args, settings) + worker.run + end + # :nocov: + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/suite_index.rb b/rspec-core/lib/rspec/core/parallel/suite_index.rb new file mode 100644 index 000000000..31e8d06a8 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/suite_index.rb @@ -0,0 +1,47 @@ +module RSpec + module Core + module Parallel + # Finds the examples and example groups of the loaded suite by the ids + # workers report them under. + # @private + class SuiteIndex + # @param on_unknown [Proc] called with a warning, once for each id the + # suite does not have + def initialize(world, &on_unknown) + @world = world + @on_unknown = on_unknown + @unknown_ids = [] + end + + def with_example(id, &block) + with_known(examples_by_id[id], id, &block) + end + + def with_group(id, &block) + with_known(groups_by_id[id], id, &block) + end + + private + + # Examples and groups defined while running (rather than loading) the + # suite exist only in the worker that defined them. + def with_known(target, id) + return yield(target) if target + return if @unknown_ids.include?(id) + + @unknown_ids << id + @on_unknown.call("A parallel worker reported results for #{id}, which was not defined when the suite was " \ + "loaded; examples and groups defined at run time cannot be reported by `--parallel`.") + end + + def examples_by_id + @examples_by_id ||= @world.all_examples.each_with_object({}) { |example, hash| hash[example.id] = example } + end + + def groups_by_id + @groups_by_id ||= @world.all_example_groups.each_with_object({}) { |group, hash| hash[group.id] = group } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb new file mode 100644 index 000000000..7f0b3db13 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -0,0 +1,353 @@ +module RSpec + module Core + module Parallel + # Runs in a forked worker process: executes the example groups the + # coordinator dispatches over the jobs channel and streams results back + # over the results channel. Suite hooks run once per worker, around all + # the groups it executes; a worker given none exits without running + # them. A worker forked before the spec files were loaded (no + # `example_groups`) loads them itself and reports what it loaded first. + # + # A worker that has finished its work but does not exit (a non-daemon + # thread in `at_exit`, a browser that will not quit, ...) is aborted by + # its own watchdog thread after `parallel_shutdown_timeout` seconds, + # with a dump of every thread's backtrace on stderr, because the process + # holding the stacks is the only one that can print them. The coordinator + # independently bounds the wait from its side. + # @private + class Worker + # Seconds between checks that the coordinating process is still there. + COORDINATOR_POLL = 0.5 + + attr_reader :number + + def initialize(number, example_groups, jobs, results, options={}) + @number = number + @example_groups = example_groups + @jobs = jobs + @results = results + @configuration = options.fetch(:configuration) { RSpec.configuration } + @world = options.fetch(:world) { RSpec.world } + @stderr = options[:stderr] + @exit = options.fetch(:exit) { lambda { |status| exit(status) } } + @exit_now = options.fetch(:exit_now) { lambda { |status| exit!(status) } } + @trap = options.fetch(:trap) do + lambda { |signal, command=nil, &handler| command ? trap(signal, command) : trap(signal, &handler) } + end + @ppid = options.fetch(:ppid) { Process.method(:ppid) } + @alive = options.fetch(:alive) { method(:process_alive?) } + @coordinator = options[:coordinator] + @loading = options[:loading] + @watch_coordinator = options.fetch(:watch_coordinator, true) + @signal_handlers = options.fetch(:signal_handlers, {}) + @output = options[:output] + end + + # @return [String] the `TEST_ENV_NUMBER` value for this worker, following + # the convention established by parallel_tests: `""` for the first + # worker, then `"2"`, `"3"`, ... + def test_env_number + self.class.test_env_number(@number) + end + + # @private + def self.test_env_number(number) + number == 1 ? "" : number.to_s + end + + # Never returns: ends the process when the coordinator has no more work. + def run + capture_output + report_started + watch_coordinator if @watch_coordinator + prepare + load_spec_files unless @example_groups + + # One that ran nothing has no suite hooks or `at_exit` handlers to run. + run_units ? @exit.call(0) : @exit_now.call(0) + rescue Errno::EPIPE, IOError + # The coordinator is gone; there is nobody left to report to. + @exit_now.call(1) + end + + # First, so the coordinator knows it got past `fork` -- or, for a + # spawned worker, that it got as far as RSpec before its `--require`d + # files load. + # @private + def report_started + return if @reported_started + + @reported_started = true + @results.send([:started]) + end + + # @private + def capture_output + return if @captured || !@output + + @captured = true + # For when there is no coordinator left to show the files. + @direct_stderr = STDERR.dup # rubocop:disable Style/GlobalStdStream + @results = WorkerOutput::Capture.new(@output, @results) + rescue SystemCallError, IOError + # It writes where the run does. + @direct_stderr&.close + @direct_stderr = nil + end + + # A worker notices that the coordinator is gone when it next reports + # to it -- which a worker stuck in a long example does not do. Its + # parent changes when the coordinator dies, or (on Windows, where it + # does not) the coordinator's pid stops naming a process, so it does + # not outlive the coordinator by more than {COORDINATOR_POLL} seconds. + # @private + def watch_coordinator + coordinator = @coordinator || @ppid.call + thread = Thread.new do + sleep COORDINATOR_POLL while @ppid.call == coordinator && @alive.call(coordinator) + stderr = @stderr || @direct_stderr || $stderr + begin + stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) is exiting because the " \ + "coordinating process (pid #{coordinator}) is gone." + rescue IOError, SystemCallError + nil + end + # The coordinator would have removed them. + WorkerOutput.remove_directory(File.dirname(@output[:out])) if @output + @exit_now.call(1) + end + thread.report_on_exception = false + thread + end + + # @private + def process_alive?(pid) + Process.kill(0, pid) + true + rescue Errno::ESRCH, Errno::EPERM + false + end + + # @private + def prepare + Runner.disable_autorun! + trap_interrupt + # Inherited from the coordinator: its handlers, which a worker swaps + # back for the ones they replaced. + @signal_handlers.each { |signal, previous| @trap.call(signal, previous) } + ENV['TEST_ENV_NUMBER'] = test_env_number + @configuration.parallel_worker_number = @number + @configuration.reset_reporter + # For formatters that helpers the spec files require add. + @configuration.formatter_loader = FormatterLoader.new(Reporter.new(@configuration)) unless @example_groups + @reporter = WorkerReporter.new(@configuration, @results, @jobs, @world) + @configuration.reporter = @reporter + end + + # Same contract as `Runner.handle_interrupt` -- stop between examples, + # force-quit on a second interrupt -- without the message, which the + # coordinator prints once for the run. + # @private + def trap_interrupt + world = @world + exit_now = @exit_now + @trap.call('INT') do + if world.wants_to_quit + exit_now.call(1) + else + world.wants_to_quit = true + end + end + end + + # Loads the spec files as the coordinator does after forking, and + # tells it which examples that defined and the first load error, if + # any: it uses the worker only if its own loading went the same way. + # What loading reports is for the coordinator to report. + # @private + def load_spec_files + forwarding = @configuration.reporter + # A spawned worker's also saw its `--require`d files load. + @configuration.reporter = loading = @loading || LoadReporter.new(@configuration) + @configuration.load_spec_files + @configuration.reporter = forwarding + @groups_by_id = @world.example_groups.to_h { |group| [group.id, group] } + @results.send([:loaded, Parallel.suite_digest(@world), loading.failure]) + end + + # @return [Boolean] whether it was given anything to run + # @private + def run_units + message = @jobs.receive + return false unless message && message.first == :run + + @configuration.with_suite_hooks(method(:leaving)) do + while message && message.first == :run + run_unit(*message.drop(1)) + message = @jobs.receive + end + end + + true + rescue SystemExit => e + # Raised by an `after(:suite)` hook once it had left the loop, unless + # it is what made it leave: its status is the process's, as serially. + unless e.status == @announced_status + @exit_cause = exit_cause(e, 'an `after(:suite)` hook') + send_exiting(e) + end + raise + ensure + # Those its `after(:suite)` hooks made, which a serial run counts too. + forward_expectation_count + end + + # Arms the watchdog and tells the coordinator -- which then bounds the + # wait from its side too -- before any `after(:suite)` hook runs, + # however the worker leaves its job loop: done, told to stop, or + # because the code it ran called `exit` (not rescued, as in a serial + # run) or the process got a signal. `$!` is the exception ending the + # process, if any. + # @private + def leaving + return if @leaving + + @leaving = true + exception = $! + @exit_cause = exit_cause(exception, 'the code it ran') + arm_watchdog + # Those of a unit that `exit` or a signal cut short, too. + forward_expectation_count + send_exiting(exception) + end + + # @private + def forward_expectation_count + @reporter.forward_expectation_count + rescue Errno::EPIPE, IOError + nil # the coordinator is gone or no longer reads it + end + + # `[:exiting, cause, exit_status]`: sent on leaving the job loop, and + # again for an `exit` that comes later. + # @private + def send_exiting(exception) + status = exception.status if exception.is_a?(SystemExit) + @announced_status = status if status + @results.send([:exiting, @exit_cause, status]) + rescue Errno::EPIPE, IOError + nil # the coordinator is gone; the watchdog and its own watch still end this process + end + + # @return [String, nil] why the process is ending early, as the end of + # a sentence: "the code it ran called `exit` or `abort` (status 0)" + # @private + def exit_cause(exception, source) + case exception + when nil then nil + when SystemExit then "#{source} called `exit` or `abort` (status #{exception.status})" + when SignalException then "it received #{exception.signm}" + else "#{source} raised #{exception.class} (#{exception.message.to_s[0, 200]})" + end + end + + # @private + def run_unit(index, id, example_ids=nil) + group = groups_by_id.fetch(id) + passed = example_ids ? run_only(group, example_ids) : group.run(@reporter) + @reporter.forward_expectation_count + @results.send([:unit_finished, index, passed ? true : false, + !!@world.wants_to_quit, !!@world.non_example_failure]) + end + + # Runs the group the way a run filtered to these examples would + # (`rspec path[ids]`): only they run, and only the context hooks of + # the groups that contain them. Groups filter their examples through + # `RSpec.world`, and every group of the tree caches which of its and + # its descendants' examples run, so each one is reset, before and + # after. + # @private + def run_only(group, example_ids) + wanted = example_ids.to_h { |id| [id, true] } + filtered = RSpec.world.filtered_examples + groups = group.descendants + kept = groups.map { |each| filtered[each] } + groups.each { |each| filtered[each] = filtered[each].select { |example| wanted.key?(example.id) } } + groups.each(&:reset_memoized) + group.run(@reporter) + ensure + if kept + groups.each_with_index { |each, index| filtered[each] = kept[index] } + groups.each(&:reset_memoized) + end + end + + # Those of the units it was forked with, unless it loaded the spec + # files itself: what it is given later is part of one of them. + # @private + def groups_by_id + @groups_by_id ||= @example_groups.to_h { |unit| [unit.group.id, unit.group] } + end + + # @private + def arm_watchdog + timeout = @configuration.parallel_shutdown_timeout + thread = Thread.new do + Parallel.sleep_running(timeout) + begin + dump_threads(timeout) + rescue IOError, SystemCallError + nil # stderr was closed or replaced; aborting matters more than the dump + end + @exit_now.call(1) + end + thread.report_on_exception = false + thread + rescue ThreadError + nil # no thread for it; the coordinator's deadline still ends this process + end + + # @private + def dump_threads(timeout) + stderr = @stderr || $stderr + stopping = @exit_cause ? "stopping because #{@exit_cause}" : 'finishing its last example group' + stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) did not exit within " \ + "#{Parallel.seconds(timeout)} of #{stopping}; aborting it. Its `after(:suite)` hooks or " \ + "`at_exit` handlers may be stuck, or need a longer `config.parallel_shutdown_timeout`. " \ + "Threads at this point:" + Thread.list.each do |thread| + stderr.puts " #{thread.inspect}#{' (this watchdog)' if thread.equal?(Thread.current)}" + Array(thread.backtrace).each { |line| stderr.puts " #{line}" } + end + stderr.flush + end + + # Opens no files: a worker's formatters never run, and the + # coordinating process writes to the files they would. + # @private + class FormatterLoader < Formatters::Loader + private + + # @private + def open_stream(_path_or_wrapper) + File.open(File::NULL, "w") + end + end + + # Reports nothing: while a worker loads the spec files, what they + # report is for the coordinating process, which loads them too, to + # report. Remembers the first error. + # @private + class LoadReporter < Reporter + attr_reader :failure + + def notify_non_example_exception(exception, context_description, _presenter_options={}) + return if @failure + + @failure = "#{context_description} #{exception.class}: #{exception.message}" + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb new file mode 100644 index 000000000..7d861179d --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -0,0 +1,284 @@ +module RSpec + module Core + module Parallel + # The part of the {Coordinator} that deals with a worker that leaves + # before its work is done: lost -- it crashed, was killed or had to be + # terminated -- when the example it was running fails and what it had + # not started goes to another worker, or ending the run because the + # code it ran called `exit`, as a serial run would end. + # @private + module WorkerExits + SENT_TERM = ', so it was sent SIGTERM' + RERUN_ARGUMENTS_SHOWN = 50 + RERUN_IDS_PER_UNIT = 20 + + private + + def judge_exit(worker, status) + case worker.state + when :running + # Asked to stop early, a worker may leave as soon as its group ends. + return if status&.success? && worker.exiting && worker.current_unit.nil? + return left_by_exit(worker, status) if worker.called_exit + + lost(worker, "exited unexpectedly (#{WorkerHandle.describe_status(status)})") + when :stopping + return left_by_exit(worker, status) if worker.called_exit + return if status&.success? + + error("#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ + "If its exit watchdog fired, its thread dump is on stderr above.") + end + end + + def overdue(worker) + timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) + if worker.called_exit + error("#{worker.description} did not exit within #{timeout} of ending the run because " \ + "#{worker.exit_cause} (its `after(:suite)` hooks or `at_exit` handlers may be stuck), so it was " \ + "sent SIGTERM. Raise `config.parallel_shutdown_timeout` if they legitimately need longer.") + abort_unit(worker) + worker.results.close + elsif worker.starting + never_started(worker) + elsif worker.state == :stopping + error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ + "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ + "`config.parallel_shutdown_timeout` if it legitimately needs longer.") + # Nothing it sends from now on is read: an `exit` in how it takes SIGTERM must not decide the status. + worker.results.close + elsif worker.loading + lost(worker, "took longer than #{Parallel.seconds(@load_allowance)}", SENT_TERM) + elsif worker.quit_sent + lost(worker, "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)", SENT_TERM) + elsif worker.exiting + lost(worker, "did not exit within #{timeout} of leaving its job loop (its `after(:suite)` hooks or " \ + "`at_exit` handlers may be stuck; raise `config.parallel_shutdown_timeout` if they " \ + "legitimately need longer)", SENT_TERM) + else + lost(worker, "stopped reporting for #{timeout}", SENT_TERM) + end + end + + def terminate(worker, reason) + lost(worker, reason, SENT_TERM) + worker.state = :terminating + worker.signal(@process, 'TERM', :term, now + @term_grace) + end + + # The worker has left its job loop and runs its `after(:suite)` hooks + # and `at_exit` handlers: as long as that may take is bounded here + # too, also for a worker that left in the middle of a group -- from + # now, so that its own watchdog, which it armed on leaving, fires + # first, also when it was asked to stop earlier. + def exiting(worker, cause, exit_status=nil) + worker.exiting = true + worker.exit_cause ||= cause + teardown = now + @configuration.parallel_shutdown_timeout + @watchdog_grace + worker.deadline = teardown unless worker.escalation || (worker.deadline && worker.deadline >= teardown) + ended_by_exit(worker, exit_status, cause) if exit_status + end + + # RSpec does not rescue `SystemExit`, so in a serial run `exit` ends + # the run: here too no example starts from now on, busy workers stop + # after the one they are running and the run exits with that status + # -- the first one, if more workers call `exit`. + def ended_by_exit(worker, exit_status, cause) + worker.called_exit = exit_status + # Leaving anyway: it is not asked to stop. + worker.quit_sent = true + if @exit_status + # A later `exit` in the same process decides its status. + replace_exit_status(worker, exit_status, cause) if worker.equal?(@exit_worker) + return + end + + @exit_status = exit_status + @exit_worker = worker + running = worker.current_example + report "#{worker.description} is ending the run: #{worker.exit_cause}" \ + "#{" in #{running.full_description} (#{running.location})" if running}. As in a serial run, no " \ + "further examples start and the run exits with status #{exit_status}." + request_quit + end + + def replace_exit_status(worker, exit_status, cause) + return if exit_status == @exit_status + + @exit_status = exit_status + report "#{worker.description}: then #{cause}, so the run exits with status #{exit_status}." unless @exit_spoiled + end + + # Stuck between `fork` and RSpec's code, as a fork-time deadlock (in + # Ruby, or a library's fork handler) leaves a process: it has run + # nothing, so the group it was given goes to another worker as is. + def never_started(worker) + worker.lost = !worker.loading + worker.loading = false + index = worker.current_unit + requeue = index && accepting_work? + stuck = if Parallel.spawning? + "started (Ruby did not get as far as loading RSpec)" + else + "forked (stuck before running any of RSpec's code, as a deadlock in a fork handler would leave it)" + end + report "#{worker.description} never reported back after it was #{stuck}, so it was sent SIGTERM. It ran " \ + "nothing#{never_started_outcome(index, requeue)}." + if requeue + @unit_states[index] = :pending + @queue.unshift(index) + worker.current_unit = nil + end + abort_unit(worker) + worker.results.close + end + + def never_started_outcome(index, requeue) + return "" unless index + + requeue ? "; the group it was given runs in another worker" : "; the group it was given did not run" + end + + # The worker can no longer be relied on, and nothing it sends from now + # on is read: it would report the example a second time. What it was + # running is reported as lost -- in a way that the summary, `Failed + # examples:` and `.rspec_status` point at -- and what it had not + # started of its unit goes to another worker. + def lost(worker, what, consequence='') + spoil_exit + because = "#{consequence}#{worker.exit_cause && ": #{worker.exit_cause}"}" + worker.lost = !worker.loading + if worker.loading + worker.loading = false + error("#{worker.description} #{what} while loading the spec files#{consequence}.") + elsif worker.current_example + lost_in_example(worker, "#{worker.description} #{what} while running this example#{because}") + elsif worker.current_unit + lost_in_unit(worker, "#{worker.description} #{what}", because) + else + error("#{worker.description} #{what} between example groups#{because}.") + end + abort_unit(worker) + worker.results.close + end + + # Its example fails and never runs again -- what it did may be why the + # worker was lost. + def lost_in_example(worker, reason) + unit = @units[worker.current_unit] + rest = unit.ids.reject { |id| worker.started_example_ids.key?(id) } + requeue = !rest.empty? && accepting_work? && !fail_fast_limit_met?(@results.failed_count + 1) + example, result = worker.lost_example_result("#{reason}#{rest_clause(unit, rest, requeue)}.") + record_result(example, result) + @sequencer.example_finished(worker.number, example, 0) + queue_first(Unit.new(unit.group, rest)) if requeue + end + + # Lost in the `before(:context)` or `after(:context)` hooks of a group + # or between its examples: the examples of the group it started last + # that had not started fail, as they do when such a hook raises, and + # the rest goes to another worker. (Its groups may have reported + # finishing as the process unwound.) Lost before it even started the + # unit's group, it is given one more go. + def lost_in_unit(worker, who, because) + unit = @units[worker.current_unit] + unstarted = unit.ids.reject { |id| worker.started_example_ids.key?(id) } + inside = worker.last_started_group + if inside + within = inside.descendant_filtered_examples.each_with_object({}) { |example, ids| ids[example.id] = true } + failing, rest = unstarted.partition { |id| within.key?(id) } + where = "while running #{inside.metadata[:full_description]} (#{inside.location}), outside of any example" + else + retry_it = worker.started_example_ids.empty? && !unit.retried + failing, rest = retry_it ? [[], unstarted] : [unstarted, []] + where = "before it started #{unit.description}" + end + + requeue = !rest.empty? && accepting_work? + if failing.empty? + error("#{who} #{where}#{because}#{rest_clause(unit, rest, requeue)}.") + else + fail_unstarted(worker, failing, "#{who} #{where}#{because}, so this example could not run" \ + "#{rest_clause(unit, rest, requeue)}.") + end + queue_first(Unit.new(unit.group, rest, !inside)) if requeue + end + + def fail_unstarted(worker, ids, reason) + ids.each do |id| + index.with_example(id) do |example| + @sequencer.example_started(worker.number, example, 0) + record_result(example, WorkerHandle.failed_result(example, reason)) + @sequencer.example_finished(worker.number, example, 0) + end + end + end + + def rest_clause(unit, rest, requeued) + return "" if rest.empty? + + examples = pluralize(rest.size, 'example') + if requeued + "; #{examples} of #{unit.description} that it had not started yet will run in another worker" + else + "; #{examples} of #{unit.description} that it had not started did not run" + end + end + + # Once a worker's `exit` ended the run, its status stands only if every + # worker then leaves on its own: one lost, failing or forced to stop + # makes it end with the exit code the results call for. + def spoil_exit + @exit_spoiled = true if @exit_status + end + + def accepting_work? + !@quitting && !@winding_down + end + + def queue_first(unit) + @units << unit + @unit_states << :pending + @queue.unshift(@units.size - 1) + end + + # A worker that called `exit` left the way a serial run does: its + # example is neither passed nor failed. One that then exited + # otherwise was aborted by its watchdog or killed. + def left_by_exit(worker, status) + unless status.nil? || (status.exited? && status.exitstatus == worker.called_exit) + ended = "#{status.signaled? ? 'was' : 'exited with'} #{WorkerHandle.describe_status(status)}" + error("#{worker.description} ended the run because #{worker.exit_cause}, then #{ended}: if its " \ + "watchdog aborted it, its `after(:suite)` hooks or " \ + "`at_exit` handlers did not finish within #{Parallel.seconds(@configuration.parallel_shutdown_timeout)} " \ + "(its thread dump is on stderr above).") + end + abort_unit(worker) + worker.results.close + end + + def report_unrun_units + return if @quitting || @queue.empty? + + units = @queue.map { |index| @units[index] } + examples = units.sum { |unit| unit.ids.size } + groups = units.map(&:group).uniq.size + report "#{pluralize(examples, 'example')} of #{pluralize(groups, 'example group')} did not run because no " \ + "worker remained to run them. To run them: rspec #{rerun_arguments(units)}" + end + + # Quoted as RSpec quotes the examples to rerun after a `--bisect`; a + # long list of examples of one group is given as the whole group. + def rerun_arguments(units) + ids = units.flat_map do |unit| + unit.example_ids && unit.example_ids.size <= RERUN_IDS_PER_UNIT ? unit.example_ids : [unit.group.id] + end + arguments = Formatters::Helpers.organize_ids(ids) + hidden = arguments.size - RERUN_ARGUMENTS_SHOWN + shown = arguments.first(RERUN_ARGUMENTS_SHOWN).join(' ') + hidden > 0 ? "#{shown} (and #{pluralize(hidden, 'more spec file')})" : shown + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb new file mode 100644 index 000000000..59a7cf726 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -0,0 +1,186 @@ +module RSpec + module Core + module Parallel + # Coordinator-side bookkeeping for one worker process. + # @private + class WorkerHandle + attr_reader :number, :jobs, :results, :child_jobs, :child_results, :current_example, :current_example_started_at + attr_accessor :pid, :current_unit, :state, :escalation, :exit_status, :quit_sent, :exiting, :exit_cause + # The status it was asked to `exit` with; the examples and the last + # group it started of its current unit. + attr_accessor :called_exit, :started_example_ids, :last_started_group + # When the deadline was set: only a pause after that postpones it. + attr_reader :deadline, :deadline_set_at + # Forked before the spec files were loaded and has not yet reported + # what it loaded. + attr_accessor :loading + # Forked, and has not sent anything yet. + attr_accessor :starting + # Its standard output and error, or `nil` when it writes where the run does. + attr_reader :output + # Lost while it ran groups: crashed, killed or terminated. + attr_accessor :lost + + # A worker started in place of a lost one has its number and a later + # `generation`. + def initialize(number, output_dir=nil, generation=1) + @number = number + @output = output_dir && WorkerOutput.new(output_dir, generation == 1 ? number : "#{number}-#{generation}") + @lost = false + @child_jobs, @jobs = Channel.pipe + begin + @results, @child_results = Channel.pipe + rescue SystemCallError + [@child_jobs, @jobs].each(&:close) + raise + end + @state = :running + @quit_sent = false + @exiting = false + @loading = false + @starting = @start_deadline = false + @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = @exit_cause = @called_exit = nil + @started_example_ids = {} + @last_started_group = nil + @current_example = @current_example_started_at = nil + end + + def deadline=(value) + @deadline = value + @deadline_set_at = value && Parallel.monotonic_time + @start_deadline = false + end + + # The deadline for its first frame, which any other deadline replaces. + def start_deadline=(value) + self.deadline = value + @start_deadline = true + end + + def clear_start_deadline + self.deadline = nil if @start_deadline + end + + # Moves the deadline back without changing when it was set. + def postpone(seconds) + @deadline += seconds if @deadline + end + + def current_example=(example) + @current_example = example + @current_example_started_at = example && RSpec::Core::Time.now + end + + # @return [String, nil] the example this worker is in the middle of, + # and for how long it has been running + def running_example_description + return nil unless @current_example + + "#{description} was running #{@current_example.location_rerun_argument || @current_example.id} " \ + "(#{@current_example.full_description}) for " \ + "#{Parallel.seconds(RSpec::Core::Time.now - @current_example_started_at)}." + end + + # @return [Array(Example, Hash)] the example this worker was running + # when it was lost, and a result failing it with `text` + def lost_example_result(text) + example = @current_example + result = self.class.failed_result(example, text, @current_example_started_at) + self.current_example = nil + [example, result] + end + + # @return [Hash] a result failing `example` with a {WorkerError} + def self.failed_result(example, text, started_at=RSpec::Core::Time.now) + exception = WorkerError.new(text) + exception.set_backtrace([example.location]) + finished_at = RSpec::Core::Time.now + { :status => :failed, :started_at => started_at, :finished_at => finished_at, + :run_time => finished_at - started_at, :exception => RemoteException.payload_for(exception) } + end + + # @return [Integer] the pid of the worker process forked to run `units`, + # or to load the spec files itself when there are none yet + def fork_worker(process, units, handles, worker_options={}) + process.fork do + # :nocov: - runs in the forked worker process, covered by integration specs + handles.each { |other| other.close_all unless other.equal?(self) } + close_parent_ends + Worker.new(@number, units, @child_jobs, @child_results, + worker_options.merge(:output => @output&.paths)).run + # :nocov: + end + end + + # @return [Integer] the pid of a new process running `command` in + # this process's directory (which JRuby keeps apart from the + # process's own), its jobs on its standard input and its results on + # its standard output + def spawn_worker(process, command, environment) + process.spawn(environment, *command, :in => @child_jobs.io, :out => @child_results.io, :chdir => Dir.pwd) + end + + def close_parent_ends + @jobs.close + @results.close + end + + def close_child_ends + @child_jobs.close + @child_results.close + end + + def close_all + close_parent_ends + close_child_ends + end + + def idle? + @state == :running && @current_unit.nil? && !@loading + end + + def busy? + @state == :running && !@current_unit.nil? + end + + def done? + @state == :exited || @state == :abandoned + end + + def description + "Parallel worker #{@number} (pid #{@pid})" + end + + # @return [Array(Integer, Process::Status), nil] like `waitpid2` with + # `WNOHANG`; a process reaped by someone else counts as exited + def reap(process) + process.waitpid2(@pid, Process::WNOHANG) + rescue Errno::ECHILD + [@pid, nil] + end + + # Sends `signal_name` as the `escalation` step that is followed up at + # `deadline`. + def signal(process, signal_name, escalation, deadline) + @escalation = escalation + self.deadline = deadline + begin + process.kill(signal_name, @pid) if @pid + rescue Errno::EINVAL + # Windows cannot send SIGTERM. + process.kill('KILL', @pid) + end + rescue Errno::ESRCH, Errno::EPERM + nil + end + + def self.describe_status(status) + return "already reaped" unless status + return "killed by SIG#{Signal.signame(status.termsig)}" if status.signaled? + + "exit status #{status.exitstatus}" + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb new file mode 100644 index 000000000..8f4e646ea --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -0,0 +1,287 @@ +module RSpec + module Core + module Parallel + # The part of the {Coordinator} that forks and ends the worker + # processes. Workers are forked after the spec files have been loaded, + # or before (`config.parallel_preload = false`): then each loads the + # spec files itself and is given work only once it has reported that + # its loading went the way the coordinator's did. Workers that never + # got to run anything are dismissed, and every exit path terminates the + # rest within the TERM/KILL grace periods. + # @private + module WorkerLifecycle + # What the system answers when it will not give this process another + # worker right now: no file descriptors for its pipes, no process for + # it. The run goes on with the workers it has. + OUT_OF_RESOURCES = [Errno::EMFILE, Errno::ENFILE, Errno::EAGAIN, Errno::ENOMEM].freeze + + # Between checks for workers that exited without closing their + # results pipe (a child of theirs holds it open, say); one whose pipe + # closed is checked for at once. + REAP_INTERVAL = 0.05 + + # Forks the workers before the spec files are loaded; each loads them + # itself and reports what it loaded. + def start + @started_at = now + @load_themselves = true + # A spawned worker starts Ruby and loads the `--require`d files first. + @boot_allowance = Parallel.spawning? ? Process.times.then { |times| times.utime + times.stime } : 0.0 + start_workers(nil) + end + + # @return [Boolean] whether any worker could be started + def started? + !@workers.empty? + end + + # Ends the workers that are not done -- workers forked before loading + # when the run ends early or goes on serially: they have run nothing. + def dismiss + live = live_workers + live.each { |worker| dismiss_worker(worker) } + give_up_unless_gone(live, @kill_grace) + clean_up + end + + private + # Whatever stopping the workers did, also when it failed. + def clean_up + Parallel.restore_signals(@signal_handlers) + WorkerOutput.remove_directory(@output_dir) + @reaper&.finish + end + + # Rather than a `waitpid` for every worker on every wake-up of the + # loop, which comes every few frames while results stream in. + def reap_workers_when_due + return unless @reap_due || now >= @next_reap_at + + @reap_due = false + @next_reap_at = now + REAP_INTERVAL + reap_workers + end + + # Moves each deadline back by the time the coordinator's loop was not + # running, beyond its own waits, since the deadline was set: the run + # was stopped (Ctrl-Z) or the machine slept, its workers with it. + def account_for_pauses + current = now + turn_started = @last_turn_at + @last_turn_at = current + + @workers.each do |worker| + next unless worker.deadline + + since = [turn_started, worker.deadline_set_at].compact.max + paused = current - since - @tick - Parallel::PAUSE_TOLERANCE + worker.postpone(paused) if paused > 0 + end + end + + # Without `units` the workers load the spec files themselves. Each + # worker's pipes are opened just before it is forked, so that the + # parent never holds more than its own ends of the earlier ones. + def start_workers(units) + Parallel.flush_output(@configuration) + # Shares more memory with forked workers; spawned ones share none. + @process.warmup if @process.respond_to?(:warmup) && !Parallel.spawning? + @output_dir ||= WorkerOutput.make_directory + # Spawned workers watch the coordinator's pid themselves. + @reaper ||= @reaper_class&.start(@output_dir) unless Parallel.spawning? + + @worker_count.times do |index| + break unless start_worker(index + 1, units) + end + end + + # @return [WorkerHandle, nil] the worker, if it started + def start_worker(number, units, starter=@spawn, generation=1) + worker = WorkerHandle.new(number, @output_dir, generation) + worker.loading = units.nil? + @workers << worker + worker.pid = starter.call(worker, @workers) + @reaper&.watch(worker.pid) + worker.close_child_ends + worker.starting = true + # A worker forked before loading gets the loading deadline instead. + worker.start_deadline = now + @configuration.parallel_shutdown_timeout unless worker.loading + worker + rescue *OUT_OF_RESOURCES => e + if worker + @workers.delete(worker) + worker.close_all + end + RSpec.warning "Started #{@workers.size} of #{@worker_count} parallel workers: the next could not be started " \ + "(#{e.class}: #{e.message}). #{@workers.empty? ? 'Running serially' : 'Going on without it'}", + :call_site => nil + nil + end + + # A worker lost while it ran groups is replaced by one with its + # number -- and so its `TEST_ENV_NUMBER` -- while groups wait to run + # and the run goes on: not once it stops early or ends by `exit` or a + # signal, and no more often than there are workers. The examples the + # lost worker failed stay failed. + def replace(worker) + return unless worker.lost && accepting_work? && !@queue.empty? && @exit_status.nil? && @termination_signal.nil? + return no_more_replacements if @replacements >= @worker_count + return unless (starter = replacement_starter) + + @replacements += 1 + replacement = start_worker(worker.number, @load_themselves ? nil : @units, starter, @replacements + 1) + return unless replacement + + replacement.deadline = now + @load_allowance if replacement.loading + report "#{replacement.description} was started in place of the lost worker (pid #{worker.pid})" \ + "#{' and loads the spec files first' if replacement.loading}." + end + + # Workers that loaded the spec files themselves are replaced the same + # way, by a new process when they were forked: forking this process, + # which has loaded them, is what the run is configured not to do. + def replacement_starter + return @spawn if !@load_themselves || Parallel.spawning? + + method(:spawn_worker) if Runner.command_line + end + + def no_more_replacements + return if @replacements_exhausted + + @replacements_exhausted = true + report "No further lost workers are replaced: the run has started as many in place of lost ones as it " \ + "has workers (#{@replacements})." + end + + def fork_worker(worker, workers) + worker.fork_worker(@process, @units, workers, :signal_handlers => @signal_handlers) + end + + # The `rspec` command this process runs, run again by the same Ruby + # with this process's load path. + def spawn_worker(worker, _workers) + script, args = Runner.command_line + command = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", script, *args] + environment = SpawnedWorker.environment(worker.number, Process.pid, @configuration.seed, worker.output&.paths) + worker.spawn_worker(@process, command, environment) + end + + # What a worker that loaded the spec files itself printed while it + # did is what this process printed when it loaded them. + def output(worker, sizes) + return unless worker.output + return worker.output.skip(sizes) if worker.loading + + worker.output.read(sizes) { |stream, bytes| show_output(worker, stream, bytes) } + end + + # What is left in a worker's output files once it has gone: what it + # printed after it last reported, such as a crash report or what its + # `at_exit` handlers printed. + def drain_output(worker) + return unless worker.output + + worker.output.read { |stream, bytes| show_output(worker, stream, bytes) } + worker.output.close + end + + # A worker dismissed before the run used it has only loaded the spec + # files, which this process did too; before the run reports, no + # worker has run anything. + def show_output(worker, stream, bytes) + return if worker.state == :dismissed || @sequencer.nil? + + @sequencer.output(worker.number, stream, bytes) + end + + # Anything it sends shows it got past forking and runs RSpec's code. + def worker_started(worker) + worker.starting = false + worker.clear_start_deadline + end + + # Including those a failed `fork` left without a process. + def live_workers + @workers.each { |worker| worker.state = :exited if worker.pid.nil? } + @workers.reject(&:done?) + end + + # Workers started loading the spec files when this process did, so + # they get as long again as it took, and the shutdown timeout on top, + # to report what they loaded; they get work once they have. + def expect_loading_workers + @suite_digest = Parallel.suite_digest(@world) + @load_failed = @world.non_example_failure + deadline = now + (now - @started_at) + @boot_allowance + @configuration.parallel_shutdown_timeout + @load_allowance = deadline - @started_at + @workers.each { |worker| worker.deadline = deadline if worker.loading } + end + + # A worker that loaded the spec files itself gets work only if its + # loading went the way this process's did: no error this process did + # not get, and the same examples. + def loaded(worker, digest, failure) + worker.loading = false + worker.deadline = nil + if failure && !@load_failed + not_used(worker, "could not load the spec files: #{failure}") + elsif digest != @suite_digest + files = (digest.keys | @suite_digest.keys).reject { |file| digest[file] == @suite_digest[file] } + not_used(worker, "loaded other examples than this process from #{files.first}" \ + "#{" and #{Formatters::Helpers.pluralize(files.size - 1, 'more spec file')}" if files.size > 1} " \ + "(examples defined depending on the process, the time or randomness?); with " \ + "`--parallel-preload` workers are forked after the spec files are loaded") + end + end + + def not_used(worker, reason) + error("#{worker.description} #{reason}, so it was not used.") + dismiss_worker(worker) + end + + # Ends a worker that has run nothing -- no example group, suite hook or + # `at_exit` handler of this run -- so it has nothing to finish. + def dismiss_worker(worker) + worker.loading = false + worker.state = :dismissed + worker.results.close + worker.signal(@process, 'KILL', :kill, now + @kill_grace) + end + + # Runs on every exit path, so an exception in the coordinator can never + # leave workers behind. Bounded by the same TERM/KILL grace periods. + def terminate_remaining_workers + live = live_workers + return if live.empty? + + live.each do |worker| + worker.state = :terminating + worker.signal(@process, 'TERM', :term, now + @term_grace) + end + wait_for_exit(live, @term_grace) + live.reject(&:done?).each { |worker| worker.signal(@process, 'KILL', :kill, now + @kill_grace) } + give_up_unless_gone(live, @kill_grace) + end + + def give_up_unless_gone(workers, grace) + wait_for_exit(workers, grace) + workers.reject(&:done?).each do |worker| + report "#{worker.description} survived SIGKILL; giving up on it." + drain_output(worker) + abort_unit(worker) + worker.state = :abandoned + end + end + + def wait_for_exit(workers, grace) + deadline = now + grace + until workers.all?(&:done?) || now >= deadline + sleep [@tick, 0.05].min + reap_workers + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_output.rb b/rspec-core/lib/rspec/core/parallel/worker_output.rb new file mode 100644 index 000000000..bcacfaefc --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_output.rb @@ -0,0 +1,125 @@ +module RSpec + module Core + module Parallel + # A worker's standard output and error -- what its examples print, + # warnings, the output of processes they start, a crash report -- go to + # two files that only the coordinator reads. Before each frame it sends, + # the worker says how far the files reach, so the coordinator shows + # what was printed in the order the worker reported its examples: with + # the example that printed it, as in a serial run. What is still in the + # files when the worker has gone is read then. + # @private + class WorkerOutput + STREAMS = [:out, :err].freeze + # Read at a time, so that the sequencer's byte bound applies to a + # flood of output too. + CHUNK = 1024 * 1024 + + # @return [String, nil] a directory only this user can read, or `nil` + # when none can be made: workers then write where the run does + def self.make_directory + require 'tmpdir' + Dir.mktmpdir('rspec-parallel-') + rescue SystemCallError, IOError + nil + end + + # Shows what a worker printed where this process prints. + def self.write(stream, bytes) + io = stream == :err ? $stderr : $stdout + # What formatters printed before it comes first. + $stdout.flush if stream == :err + # Text-mode output on Windows adds a carriage return to every newline. + bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !(io.respond_to?(:binmode?) && io.binmode?) + # As written, not transcoded: they are in whatever encoding the worker wrote. + encoding = io.respond_to?(:external_encoding) && io.external_encoding + bytes = bytes.dup.force_encoding(encoding) if encoding + io.write(bytes) + io.flush + rescue IOError, SystemCallError, EncodingError + nil + end + + def self.remove_directory(dir) + return unless dir + + require 'fileutils' + FileUtils.remove_entry(dir, true) + end + + attr_reader :paths + + def initialize(dir, number) + @paths = STREAMS.to_h { |stream| [stream, File.join(dir, "worker-#{number}.#{stream}")] } + @paths.each_value { |path| File.binwrite(path, '') } + @readers = {} + @read = { :out => 0, :err => 0 } + end + + # Yields `stream, bytes` for what the worker wrote up to `sizes` + # (`{:out => n, :err => m}`), or to the end of the files. + def read(sizes=nil, &block) + STREAMS.each { |stream| read_stream(stream, sizes && sizes[stream], &block) } + end + + # Moves on past what the worker wrote up to `sizes` without showing it. + def skip(sizes) + STREAMS.each { |stream| @read[stream] = [@read[stream], sizes[stream]].max } + end + + def close + @readers.each_value(&:close) + @readers.clear + end + + private + + def read_stream(stream, limit) + reader = (@readers[stream] ||= File.open(@paths[stream], 'rb')) + loop do + wanted = limit ? limit - @read[stream] : CHUNK + break if wanted <= 0 + + reader.pos = @read[stream] + bytes = reader.read([wanted, CHUNK].min) + break if bytes.nil? || bytes.empty? + + @read[stream] += bytes.bytesize + yield stream, bytes + end + rescue SystemCallError, IOError + nil + end + + # The worker's side: its standard output and error moved into the + # files, and the results channel it reports through, which announces + # new output before each frame. + # @private + class Capture + def initialize(paths, results) + @results = results + @paths = paths + # The process's own streams, which processes it starts inherit, + # not whatever `$stdout` and `$stderr` point to. + # rubocop:disable-next Style/GlobalStdStream + { STDOUT => paths[:out], STDERR => paths[:err] }.each do |io, path| + File.open(path, 'ab') { |file| io.reopen(file) } + io.binmode + io.sync = true + end + @announced = { :out => 0, :err => 0 } + end + + def send(message) + sizes = @paths.transform_values { |path| File.size(path) } + unless sizes == @announced + @results.send([:output, sizes]) + @announced = sizes + end + @results.send(message) + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb new file mode 100644 index 000000000..1f594e8bc --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -0,0 +1,223 @@ +module RSpec + module Core + module Parallel + # The reporter installed in a worker process. It has no formatters; + # it forwards every notification -- RSpec's own and custom ones + # published through `Reporter#publish` -- to the coordinator as + # `Marshal`-safe frames referring to examples and groups by id. The + # coordinator replays them onto its own reporter, so formatters run + # exclusively in the coordinating process and receive real `Example` + # and `ExampleGroup` objects. + # + # Listeners registered on this reporter from inside the worker (for + # instance in a `before(:suite)` hook) are notified as usual before the + # notification is forwarded. + # @private + class WorkerReporter < Reporter + FORWARDED_AS_PAYLOAD = [:message, :deprecation].freeze + EXAMPLE_OUTCOMES = [:example_passed, :example_failed, :example_pending].freeze + RUN_LIFECYCLE = [:seed, :start, :stop, :start_dump, :dump_pending, :dump_failures, :deprecation_summary, + :dump_profile, :dump_summary, :close].freeze + # Example metadata that is not compared before and after the example: + # RSpec's own, which the coordinator keeps, and what the result + # payload carries anyway. + NOT_COMPARED = (Metadata::RESERVED_KEYS + [:pending, :skip, :extra_failure_lines]).to_h { |key| [key, true] }.freeze + + def initialize(configuration, results, jobs, world=RSpec.world) + super(configuration) + @results = results + @jobs = jobs + @world = world + @group_started_at = nil + @unforwarded_expectations = 0 + @metadata_before = nil + end + + # @private + # Called for every expectation by an rspec-expectations that counts + # them; the count reaches the coordinator through + # {#forward_expectation_count}. + def increment_expectation_count + @unforwarded_expectations += 1 + end + + # @private + # Sends what was counted since the last call, if anything: the + # coordinator adds it to its own reporter's count, if that keeps one. + def forward_expectation_count + return if @unforwarded_expectations.zero? + + @results.send([:expectations, @unforwarded_expectations]) + @unforwarded_expectations = 0 + end + + # @private + # Suite hook failures and the like: the coordinator rebuilds the + # exception and reports it through its own reporter, which keeps the + # run's exit code and "errors occurred outside of examples" count right. + def notify_non_example_exception(exception, context_description) + @world.non_example_failure = true + @results.send([:non_example_failure, RemoteException.payload_for(exception), context_description.to_s]) + end + + # @private + # An example's result is forwarded with its outcome notification + # (`example_passed`, `example_failed` or `example_pending`), which + # follows `example_finished`, so that metadata worker listeners add + # on either (such as `:extra_failure_lines`) reaches the coordinator. + def notify(event, notification) + super + + case event + when :example_group_started then forward_group_started(notification.group) + when :example_group_finished then forward_group_finished(notification.group) + when :example_started then forward_example_started(notification.example) + when *EXAMPLE_OUTCOMES then forward_example_finished(notification.example) + when :example_finished, *RUN_LIFECYCLE then nil + when *FORWARDED_AS_PAYLOAD then forward(event, notification) + else forward_custom(event, notification) + end + end + + private + + def forward_group_started(group) + @group_started_at = Parallel.monotonic_time if group.top_level? + @results.send([:group_started, group.id, descriptions(group.metadata)]) + end + + def forward_group_finished(group) + duration = nil + if group.top_level? && @group_started_at + duration = Parallel.monotonic_time - @group_started_at + @group_started_at = nil + end + @results.send([:group_finished, group.id, duration]) + end + + def forward_example_started(example) + @metadata_before = metadata_snapshot(example.metadata) + @results.send([:example_started, example.id]) + end + + def forward_example_finished(example) + @results.send([:example_finished, example.id, result_payload(example)]) + poll_for_quit + end + + # The coordinator asks workers to stop between examples (fail-fast, + # interrupt) by writing `[:quit]` on the jobs channel. Anything else + # is left for the worker's job loop. + def poll_for_quit + @jobs.drain.each do |message, _size| + if message.first == :quit + @world.wants_to_quit = true + else + @jobs.unread(message) + end + end + end + + def forward(event, notification) + if event == :message + @results.send([:message, Parallel.marshal_safe(notification.message, &:to_s)]) + else + hash = notification.to_h + @results.send([:deprecation, Parallel.marshal_safe(hash) { hash.transform_values { |v| v && v.to_s } }]) + end + end + + def forward_custom(event, notification) + options = notification.respond_to?(:to_h) ? notification.to_h : {} + Marshal.dump(options) + @results.send([:custom, event, options]) + rescue TypeError, ArgumentError, NoMethodError + @results.send([:message, "Parallel worker #{@configuration.parallel_worker_number} could not forward " \ + "the custom `#{event}` notification because its options cannot be marshaled: " \ + "#{options.inspect[0, 500]}"]) + end + + def result_payload(example) + result = example.execution_result + metadata = example.metadata + + { + :status => result.status, + :started_at => Parallel.time_to_wire(result.started_at), + :finished_at => Parallel.time_to_wire(result.finished_at), + :run_time => result.run_time, + :pending_message => Parallel.marshal_safe(result.pending_message, &:to_s), + :pending_fixed => result.pending_fixed, + :exception => RemoteException.payload_for(result.exception), + :pending_exception => RemoteException.payload_for(result.pending_exception), + :pending => Parallel.marshal_safe(metadata[:pending], &:to_s), + :skip => Parallel.marshal_safe(metadata[:skip], &:to_s), + :extra_failure_lines => Parallel.marshal_safe(metadata[:extra_failure_lines]) { |lines| Array(lines).map(&:to_s) }, + :metadata => metadata_changes(metadata) + }.merge(descriptions(metadata)) + end + + # Each value with its hash, so that a value changed in place counts as + # changed too. + def metadata_snapshot(metadata) + metadata.each_with_object({}) do |(key, value), snapshot| + snapshot[key] = [value, hash_of(value)] unless NOT_COMPARED.key?(key) + end + end + + # @return [Hash, nil] what the example, its hooks or a worker listener + # did to its metadata: keys set (`:set`) or deleted (`:deleted`), + # each as the key's `inspect` and its own `Marshal` dump -- so that + # one the coordinator cannot load (its class is not loaded there) + # spoils only itself -- and keys set to values that cannot be + # marshaled (`:unsent`, with the reason), which the coordinator + # leaves as they were. + def metadata_changes(metadata) + before = @metadata_before || {} + @metadata_before = nil + changes = {} + metadata.each do |key, value| + next if NOT_COMPARED.key?(key) || unchanged?(before[key], value) + + if (dump = marshaled([key, value])) + (changes[:set] ||= []) << [key.inspect, dump] + else + (changes[:unsent] ||= {})[key.inspect] = "#{value.class.name} cannot be marshaled" + end + end + deleted = deleted_keys(before, metadata) + changes[:deleted] = deleted unless deleted.empty? + changes.empty? ? nil : changes + end + + def deleted_keys(before, metadata) + before.keys.reject { |key| metadata.key?(key) }.filter_map { |key| (dump = marshaled(key)) && [key.inspect, dump] } + end + + def unchanged?(snapshot, value) + snapshot && snapshot[0].equal?(value) && snapshot[1] == hash_of(value) + end + + def marshaled(value) + Marshal.dump(value) + rescue StandardError + nil + end + + def hash_of(value) + value.hash + rescue StandardError + nil + end + + # As this worker has them: generated by the matcher of a one-liner + # (`it { is_expected.to ... }`) once the example ran, or computed + # while the worker loaded the spec files itself. + def descriptions(metadata) + { :description => String.new(metadata[:description].to_s), + :full_description => String.new(metadata[:full_description].to_s) } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/profiler.rb b/rspec-core/lib/rspec/core/profiler.rb index 5e6527974..8f3697a87 100644 --- a/rspec-core/lib/rspec/core/profiler.rb +++ b/rspec-core/lib/rspec/core/profiler.rb @@ -29,6 +29,15 @@ def example_started(notification) group = notification.example.example_group.parent_groups.last @example_groups[group][:count] += 1 end + + # Used when the group ran in another process and its duration was + # measured there. + def record_group_duration(group, duration) + return unless group.top_level? + + @example_groups[group][:description] = group.top_level_description + @example_groups[group][:total_time] = duration + end end end end diff --git a/rspec-core/lib/rspec/core/reporter.rb b/rspec-core/lib/rspec/core/reporter.rb index d513f3f9f..0c8da7ca7 100644 --- a/rspec-core/lib/rspec/core/reporter.rb +++ b/rspec-core/lib/rspec/core/reporter.rb @@ -156,16 +156,25 @@ def deprecation(hash) notify :deprecation, Notifications::DeprecationNotification.from_hash(hash) end + # @private + # Records the duration of a top-level example group that ran in another + # process (see `Parallel`), in place of the time observed here. + def record_example_group_duration(group, duration) + ensure_listeners_ready + @profiler.record_group_duration(group, duration) + end + # @private # Provides a way to notify of an exception that is not tied to any # particular example (such as an exception encountered in a :suite hook). - # Exceptions will be formatted the same way they normally are. - def notify_non_example_exception(exception, context_description) + # Exceptions will be formatted the same way they normally are, unless + # `presenter_options` say otherwise (`:failure_lines`). + def notify_non_example_exception(exception, context_description, presenter_options={}) @configuration.world.non_example_failure = true @non_example_exception_count += 1 example = Example.new(AnonymousExampleGroup, context_description, {}) - presenter = Formatters::ExceptionPresenter.new(exception, example, :indentation => 0) + presenter = Formatters::ExceptionPresenter.new(exception, example, { :indentation => 0 }.merge(presenter_options)) message presenter.fully_formatted(nil) end diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index 16d07efea..0d4a8b82f 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -42,10 +42,26 @@ def self.perform_at_exit # code. def self.invoke disable_autorun! + # A worker of a parallel run on a platform without `fork`, started as + # a new `rspec` process; it ends the process. + if ENV.key?('RSPEC_PARALLEL_WORKER') + RSpec::Support.require_rspec_core "parallel/coordinator" + Parallel::SpawnedWorker.run(ARGV) + end + @command_line = [File.file?($0) ? File.expand_path($0) : RSpec::Core.path_to_executable, ARGV.dup] status = run(ARGV, $stderr, $stdout).to_i exit(status) if status != 0 end + # @private + # @return [Array(String, Array), nil] the script and arguments + # of the `rspec` command this process runs, which starts the workers + # of a parallel run on a platform without `fork`; `nil` when the run + # was not started by `.invoke` + def self.command_line + @command_line ||= nil + end + # Run a suite of RSpec examples. Does not exit. # # This is used internally by RSpec to run a suite, but is available @@ -76,6 +92,7 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) @options = options @configuration = configuration @world = world + @parallel_plan = nil end # Configures and runs a spec suite. @@ -83,22 +100,30 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) # @param err [IO] error stream # @param out [IO] output stream def run(err, out) - setup(err, out) + setup(err, out) { parallel_plan&.before_loading } return @configuration.reporter.exit_early(exit_code) if RSpec.world.wants_to_quit - run_specs(@world.ordered_example_groups).tap do + example_groups = @world.ordered_example_groups + coordinator = parallel_plan&.coordinator_for(example_groups) + + (coordinator ? coordinator.run : run_specs(example_groups)).tap do persist_example_statuses end + ensure + # Workers forked before loading that the run did not get to use. + @parallel_plan&.dismiss end # Wires together the various configuration objects and state holders. # # @param err [IO] error stream # @param out [IO] output stream + # @yield once configured, before the spec files are loaded def setup(err, out) configure(err, out) return if RSpec.world.wants_to_quit + yield if block_given? @configuration.load_spec_files ensure @world.announce_filters @@ -200,6 +225,16 @@ def exit_code(examples_passed=false) private + # Only a run configured to be parallel -- before the spec files load or + # by them -- loads the parallel runner. + def parallel_plan + return @parallel_plan if @parallel_plan + return nil if @configuration.parallel_workers.nil? && @configuration.parallel_preload? + + RSpec::Support.require_rspec_core "parallel" + @parallel_plan = Parallel::Plan.new(self) + end + def persist_example_statuses return if @configuration.dry_run return unless (path = @configuration.example_status_persistence_file_path) diff --git a/rspec-core/spec/integration/bisect_spec.rb b/rspec-core/spec/integration/bisect_spec.rb index 87f9e4610..155e8542d 100644 --- a/rspec-core/spec/integration/bisect_spec.rb +++ b/rspec-core/spec/integration/bisect_spec.rb @@ -40,6 +40,32 @@ def bisect(cli_args, expected_status=nil) ]) expect(output).to include("Bisect complete!", "rspec ./spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb[1:1] ./spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb[1:1]") end + + it 'returns a reproduction command that runs serially when parallel runs are configured' do + output = bisect(%w[ + --order defined --parallel=2 + spec/rspec/core/resources/bisect/consistently_ordered_1_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_4_specs.rb + ]) + expect(output).to include("Bisect complete!", "consistently_ordered_3_specs.rb[1:1] --order defined --no-parallel") + expect(output).not_to include("--parallel=2") + end + + it 'leaves the parallel options out of the SPEC_OPTS of the reproduction command, since they would win over it' do + output = nil + with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do + output = bisect(%w[ + spec/rspec/core/resources/bisect/consistently_ordered_1_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_4_specs.rb + ]) + end + expect(output).to include("Bisect complete!", 'SPEC_OPTS="--order defined" rspec ./spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb[1:1] ./spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb[1:1] --no-parallel') + expect(output).not_to include("--parallel=2") + end end context "when the bisect command saturates the pipe" do diff --git a/rspec-core/spec/integration/location_rerun_spec.rb b/rspec-core/spec/integration/location_rerun_spec.rb index 38c54c8fa..404935f26 100644 --- a/rspec-core/spec/integration/location_rerun_spec.rb +++ b/rspec-core/spec/integration/location_rerun_spec.rb @@ -67,7 +67,7 @@ around { |ex| with_env_vars('SHELL' => '/usr/local/bin/zsh', &ex) } it 'prints the example id of the failed assertion' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -81,7 +81,7 @@ context "and the shared examples are defined in a separate file" do it 'prints the example id of the failed assertion' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/non_local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/non_local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -103,7 +103,7 @@ context "when the shared examples are defined in the same file as the spec" do it 'prints the line number where the assertion failed in the local file' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -118,7 +118,7 @@ context "and the shared examples are defined in a separate file" do it 'prints the line number where the `it_behaves_like` was called in the local file' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/non_local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/non_local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb new file mode 100644 index 000000000..f0949be55 --- /dev/null +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -0,0 +1,1446 @@ +require 'rspec/core/parallel/coordinator' +require 'support/aruba_support' +require 'json' + +# Runs `rspec` in a process of its own, where the `--require`d files load +# afresh. Given at most 60 seconds. +module ParallelRunProcess + def rspec_process(*args) + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + # Absolute, and in this directory: JRuby keeps its own apart from the process's. + out, err = %w[stdout.txt stderr.txt].map { |name| File.expand_path(name) } + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => out, :err => err, :chdir => Dir.pwd) + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 + sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + Process.kill(:KILL, pid) && Process.wait(pid) unless status + [status, File.read(out), File.read(err)] + end + end +end + +# The longest group of rspec-core's suite: under `--parallel` it runs as parts in +# several workers (each has its own aruba directory). +RSpec.describe 'Parallel execution (--parallel)', :slow, :parallel_split => 4, + :skip => !RSpec::Support::RubyFeatures.fork_supported? do + include_context "aruba support" + include ParallelRunProcess + + # The coordinator refuses to nest inside another parallel worker, which is + # what this very process is when rspec-core's suite runs under `--parallel`. + # Aruba swaps in its own environment while a command runs, so the variable + # is removed from (or, for the fallback test, added to) that environment. + def run_rspec(cmd) + if @inside_another_parallel_worker + set_environment_variable("TEST_ENV_NUMBER", "2") + else + delete_environment_variable("TEST_ENV_NUMBER") + end + run_command(cmd) + end + + # Written by the workers' `before(:suite)` hooks: a file can exist before + # its pid has been written to it. + def worker_pids + Dir['worker_*.pid'].map { |file| File.read(file) }.reject(&:empty?).map { |pid| Integer(pid) } + end + + def write_fixed_suite + write_file_formatted 'spec/alpha_spec.rb', " + RSpec.describe 'Alpha' do + it('passes') { expect(1).to eq(1) } + it('fails') { expect(1).to eq(2) } + it('is pending') { pending 'not yet'; expect(1).to eq(2) } + it('is skipped') { skip 'later' } + context 'nested' do + it('fails with a cause') { begin; raise 'root'; rescue; raise ArgumentError, 'wrapped'; end } + end + end + " + + write_file_formatted 'spec/beta_spec.rb', " + RSpec.describe 'Beta' do + it('aggregates', :aggregate_failures) { expect(1).to eq(2); expect(2).to eq(3) } + it('passes too') { expect(true).to be(true) } + it('fails in an after hook') { } + after { |ex| raise 'in after' if ex.description == 'fails in an after hook' } + end + " + + write_file_formatted 'spec/gamma_spec.rb', " + RSpec.describe 'Gamma' do + 3.times { |i| it(\"passes \#{i}\") { expect(i).to be >= 0 } } + end + " + + # Described by their matchers once they have run. + write_file_formatted 'spec/delta_spec.rb', " + RSpec.describe 'Delta' do + subject { 5 } + it { is_expected.to eq(5) } + it { is_expected.to eq(6) } + end + " + end + + # Timing differs between runs, and failures are numbered in the order they + # are reported, which for a parallel run is the order groups finish in. + # Frames below the spec files differ too because these specs run RSpec + # inside RSpec (the worker's stack goes through the coordinator's fork). + def normalize(output) + output. + gsub(/Finished in .*$/, 'Finished in X'). + gsub(/\d+\.\d+ seconds/, 'X seconds'). + gsub(/\d+(\.\d+)? seconds? average.*$/, 'X average'). + gsub(/\(FAILED - \d+\)/, '(FAILED - N)'). + gsub(/^(\s*)\d+(\.\d+)?\) /, '\\1N) '). + lines.reject { |line| line =~ /^\s*# / && line !~ %r{# \./spec/} }.join + end + + def group_blocks(documentation_output) + body = documentation_output.split(/^Failures:$|^Pending:/).first + body.split(/\n(?=\S)/).map(&:strip).reject(&:empty?).reject { |block| block.start_with?("Randomized") } + end + + it 'runs the groups across workers and reports one coherent run with the serial exit code' do + write_fixed_suite + + run_rspec "--parallel=2 --order defined" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("13 examples, 5 failures, 2 pending") + expect(last_cmd_stdout).to include("Alpha fails", "Alpha nested fails with a cause", "Beta aggregates", "Beta fails in an after hook", + "rspec ./spec/delta_spec.rb:4 # Delta is expected to eq 6") + expect(last_cmd_stdout).to include("Got 2 failures:", "--- Caused by: ---", "root") + expect(last_cmd_stdout.scan("Randomized with seed").size).to eq(0) + expect(last_cmd_stderr).to eq("") + end + + it 'gives each worker its number and a parallel_tests-style TEST_ENV_NUMBER, and runs suite hooks once per worker' do + write_file_formatted 'spec/env_spec.rb', " + RSpec.configure do |c| + c.before(:suite) { File.write(\"before_suite_\#{RSpec.configuration.parallel_worker_number}\", ENV['TEST_ENV_NUMBER'].inspect) } + c.after(:suite) { File.write(\"after_suite_\#{RSpec.configuration.parallel_worker_number}\", Process.pid) } + end + 3.times do |i| + RSpec.describe(\"G\#{i}\") do + it('records its worker') { File.write(\"example_\#{i}\", RSpec.configuration.parallel_worker_number.to_s) } + end + end + " + + test_env_number_before = ENV['TEST_ENV_NUMBER'] + run_rspec "--parallel=2 spec/env_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("3 examples, 0 failures") + cd '.' do + expect(File.read("before_suite_1")).to eq('""') + expect(File.read("before_suite_2")).to eq('"2"') + expect(File.exist?("after_suite_1")).to be true + expect(File.exist?("after_suite_2")).to be true + expect(Dir["example_*"].map { |f| File.read(f) }.sort).to all(match(/\A[12]\z/)) + end + expect(RSpec.configuration.parallel_worker_number).to be_nil + expect(ENV['TEST_ENV_NUMBER']).to eq(test_env_number_before) + end + + it 'notifies listeners registered in a before(:suite) hook and shows the failure lines they add, like a serial run' do + write_file_formatted 'spec/listener_spec.rb', " + class RefListener + def example_failed(notification) + notification.example.metadata[:extra_failure_lines] = ['Ref: added by a before(:suite) listener'] + end + end + RSpec.configure do |c| + c.before(:suite) { RSpec.configuration.reporter.register_listener(RefListener.new, :example_failed) } + end + RSpec.describe('One') { it('fails') { expect(1).to eq(2) } } + RSpec.describe('Two') { it('fails too') { expect(2).to eq(3) } } + " + + run_rspec "--order defined spec/listener_spec.rb" + serial_stdout = last_cmd_stdout + run_rspec "--parallel=2 --order defined spec/listener_spec.rb" + + expect(last_cmd_exit_status).to eq(1) + expect(serial_stdout.scan('Ref: added by a before(:suite) listener').size).to eq(2) + expect(last_cmd_stdout.scan('Ref: added by a before(:suite) listener').size).to eq(2) + end + + describe 'formatter parity with a serial run' do + before { write_fixed_suite } + + it 'produces the same progress output' do + run_rspec "--order defined --format progress" + serial = normalize(last_cmd_stdout) + run_rspec "--order defined --format progress --parallel=2" + parallel = normalize(last_cmd_stdout) + + serial_dots, serial_rest = serial.split("\n", 2) + parallel_dots, parallel_rest = parallel.split("\n", 2) + expect(parallel_dots.chars.sort).to eq(serial_dots.chars.sort) + expect(parallel_rest.lines.sort).to eq(serial_rest.lines.sort) + end + + it 'produces the same documentation blocks for every group, with the serial order inside each group' do + run_rspec "--order defined --format documentation" + serial = normalize(last_cmd_stdout) + run_rspec "--order defined --format documentation --parallel=3" + parallel = normalize(last_cmd_stdout) + + expect(group_blocks(parallel)).to match_array(group_blocks(serial)) + expect(parallel.lines.sort).to eq(serial.lines.sort) + end + + it 'produces the same JSON' do + run_rspec "--order defined --format json --out serial.json" + run_rspec "--order defined --format json --out parallel.json --parallel=2" + + cd '.' do + serial, parallel = %w[serial.json parallel.json].map do |file| + json = JSON.parse(File.read(file)) + json.delete("summary_line") + json["summary"].delete("duration") + json["examples"].each do |e| + e.delete("run_time") + e["exception"]["backtrace"].select! { |frame| frame.start_with?("./spec/") } if e["exception"] + end + json["examples"] = json["examples"].sort_by { |e| e["id"] } + json + end + + expect(parallel).to eq(serial) + expect(parallel["examples"].select { |e| e["exception"] }.map { |e| e["exception"]["class"] }).to include( + "RSpec::Expectations::ExpectationNotMetError", "ArgumentError", "RSpec::Expectations::MultipleExpectationsNotMetError", "RuntimeError" + ) + end + end + + it 'produces the same failures list' do + run_rspec "--order defined --format failures" + serial = last_cmd_stdout + run_rspec "--order defined --format failures --parallel=2" + + expect(last_cmd_stdout.lines.sort).to eq(serial.lines.sort) + end + + it 'produces structurally equivalent HTML' do + run_rspec "--order defined --format html --out serial.html" + run_rspec "--order defined --format html --out parallel.html --parallel=2" + + cd '.' do + serial, parallel = %w[serial.html parallel.html].map { |file| File.read(file) } + expect(parallel.scan(/class="example (\w+)"/).sort).to eq(serial.scan(/class="example (\w+)"/).sort) + expect(parallel.scan(/]*>/).size).to eq(serial.scan(/]*>/).size) + expect(parallel.scan('').size).to eq(serial.scan('').size) + expect(parallel).to include("13 examples, 5 failures, 2 pending") + end + end + + it 'merges the profile with the durations measured in the workers' do + run_rspec "--order defined --profile 2 --parallel=2" + + expect(last_cmd_stdout).to include("Top 2 slowest examples", "Top 2 slowest example groups") + expect(last_cmd_stdout).to match(/\d+\.\d+ seconds average \(\d+\.\d+ seconds \/ \d+ examples?\) \.\/spec\/\w+_spec\.rb:1/) + end + end + + # As rspec-expectations does for every expectation when its reporter counts + # them (rspec/rspec#305), the examples here call the configured reporter's + # `increment_expectation_count`, which the formatter adds to the reporter + # of the coordinating process (or of the serial run). + it 'adds up the expectations its workers count, as a serial run does, for a reporter that counts them' do + write_file_formatted 'spec/support/counted_expectations.rb', " + class CountedExpectations + RSpec::Core::Formatters.register self, :start, :dump_summary + + def initialize(output) + @output = output + end + + def start(_notification) + counted = 0 + reporter = RSpec.configuration.reporter + reporter.define_singleton_method(:increment_expectation_count) { counted += 1 } + reporter.define_singleton_method(:counted_expectations) { counted } + end + + def dump_summary(_notification) + @output.puts \"Counted \#{RSpec.configuration.reporter.counted_expectations} expectations\" + end + end + " + 4.times do |i| + write_file_formatted "spec/c#{i}_spec.rb", " + RSpec.describe('C#{i}') do + before(:context) { RSpec.configuration.reporter.increment_expectation_count } + it('counts twice') { 2.times { RSpec.configuration.reporter.increment_expectation_count } } + it('counts once') { RSpec.configuration.reporter.increment_expectation_count } + end + " + end + options = '--require ./spec/support/counted_expectations --format progress --format CountedExpectations' + + run_rspec options + expect(last_cmd_stdout).to include("8 examples, 0 failures", "Counted 16 expectations") + + run_rspec "#{options} --parallel=2" + expect(last_cmd_stdout).to include("8 examples, 0 failures", "Counted 16 expectations") + expect(last_cmd_exit_status).to eq(0) + end + + it 'shows formatters the metadata an example set, changed or deleted in a worker, as a serial run does' do + write_file_formatted 'spec/support/metadata_formatter.rb', " + class MetadataFormatter + RSpec::Core::Formatters.register self, :example_passed + + def initialize(output) + @output = output + end + + def example_passed(notification) + metadata = notification.example.metadata + @output.puts \"META \#{notification.example.description}: \#{metadata[:screenshot].inspect} \" \\ + \"\#{metadata[:tags].inspect} \#{metadata.key?(:gone)} \#{metadata[:callback].class} \#{metadata[:lazy].class}\" + end + end + " + 4.times do |i| + write_file_formatted "spec/m#{i}_spec.rb", " + RSpec.describe('M#{i}', :gone => true) do + it('e#{i}', :tags => ['a']) do |ex| + ex.metadata[:screenshot] = 'shot-#{i}.png' + ex.metadata[:tags] << 'b' + ex.metadata.delete(:gone) + end + it('callback#{i}') { |ex| ex.metadata[:callback] = lambda {} } + it('lazy#{i}') { |ex| Object.const_set(:ParallelSpecLazyThing, Class.new) unless defined?(ParallelSpecLazyThing); ex.metadata[:lazy] = ParallelSpecLazyThing.new } + end + " + end + options = '--require ./spec/support/metadata_formatter --format progress --format MetadataFormatter' + + # In parallel first: a serial run in this process would define the class here too. + run_rspec "#{options} --parallel=2" + parallel = last_cmd_stdout.scan(/META .*$/).sort + expect(last_cmd_stdout).to include("12 examples, 0 failures") + expect(last_cmd_exit_status).to eq(0) + stderr = last_cmd_stderr + + run_rspec options + serial = last_cmd_stdout.scan(/META .*$/).sort + expect(serial.size).to eq(12) + expect(serial.first).to eq("META callback0: nil nil true Proc NilClass") + expect(serial).to include("META e3: \"shot-3.png\" [\"a\", \"b\"] false NilClass NilClass", + "META lazy0: nil nil true NilClass ParallelSpecLazyThing") + + # A Proc cannot be marshaled; the class is defined only in the worker that ran the example. + expect(parallel).to eq(serial.map { |line| line.sub(' Proc ', ' NilClass ').sub(/ ParallelSpecLazyThing$/, ' NilClass') }) + expect(stderr.scan('set example metadata :callback to a value that cannot reach the formatters (Proc cannot be marshaled)').size).to eq(1) + expect(stderr.scan('set example metadata :lazy to a value that cannot reach the formatters (it cannot be loaded here (ArgumentError: undefined class/module ParallelSpecLazyThing').size).to eq(1) + ensure + Object.send(:remove_const, :ParallelSpecLazyThing) if defined?(ParallelSpecLazyThing) + end + + it 'shows what workers print with the example group that printed it, as a serial run does' do + write_file_formatted 'spec/support/at_exit.rb', " + RSpec.configure { |c| c.after(:suite) { at_exit { puts \"at_exit of worker \#{c.parallel_worker_number}\" } } } + " + 3.times do |i| + write_file_formatted "spec/p#{i}_spec.rb", " + puts 'loading p#{i}' + RSpec.describe('P#{i}') do + it('prints') do + puts 'stdout of P#{i}' + warn 'stderr of P#{i}' + system('echo subprocess of P#{i}') + end + it('is quiet') { } + end + " + end + write_file_formatted 'spec/killed_spec.rb', " + RSpec.describe('Killed') do + it('says goodbye') { $stderr.puts 'last words'; Process.kill(:KILL, Process.pid) } + end + " + options = %w[--require ./spec/support/at_exit --format documentation --order defined --parallel=2] + + status, stdout, stderr = rspec_process(*options) + # Without what the coordinator says when it replaces the killed worker. + shown = stdout.gsub(/^ *Parallel worker \d \(pid \d+\) was started in place .*\n/, '') + %w[P0 P1 P2].each do |name| + expect(shown).to include("#{name}\nstdout of #{name}\nsubprocess of #{name}\n prints\n") + end + # The worker killed with SIGKILL runs no `at_exit` handler. + expect(stdout.scan(/^at_exit of worker \d$/)).not_to be_empty + expect(stderr).to include("stderr of P0\n", "stderr of P1\n", "stderr of P2\n", "last words\n") + expect(stdout).to include("7 examples, 1 failure") + expect(status.exitstatus).to eq(1) + + status, stdout, _stderr = rspec_process(*options, '--no-parallel-preload', *%w[spec/p0_spec.rb spec/p1_spec.rb spec/p2_spec.rb]) + expect(stdout.scan(/^loading p\d$/)).to eq(["loading p0", "loading p1", "loading p2"]) + expect(stdout).to include("6 examples, 0 failures") + expect(status.exitstatus).to eq(0) + end + + it 'starts a worker with the same TEST_ENV_NUMBER in place of one that crashed, which runs what was left' do + 4.times do |i| + write_file_formatted "spec/r#{i}_spec.rb", " + RSpec.describe('R#{i}') do + it('records') do + File.write('ran_r#{i}', [Process.pid, ENV['TEST_ENV_NUMBER'].inspect].join(' ')) + Process.kill(:KILL, Process.pid) if #{i} == 0 + sleep 1 if #{i} == 1 + end + end + " + end + + run_rspec '--parallel=2 --order defined' + + expect(last_cmd_stdout).to include("4 examples, 1 failure", "exited unexpectedly", "was started in place of the lost worker") + expect(last_cmd_exit_status).to eq(1) + ran = cd('.') { Dir['ran_r*'].to_h { |file| [file, File.read(file).split] } } + expect(ran.keys).to eq(%w[ran_r0 ran_r1 ran_r2 ran_r3]) + crashed_pid, crashed_env = ran['ran_r0'] + replacement = ran.values.find { |pid, env| env == crashed_env && pid != crashed_pid } + expect(replacement).not_to be_nil + end + + it 'leaves the file of a formatter that a helper the spec files require adds to the coordinating process' do + write_file_formatted 'spec/helper.rb', "RSpec.configure { |c| c.add_formatter 'documentation', 'out.txt' }" + 4.times do |i| + write_file_formatted "spec/f#{i}_spec.rb", " + require_relative 'helper' + RSpec.describe('F#{i}') { it('runs') { #{i == 1 ? 'Process.kill(:KILL, Process.pid)' : 'sleep 0.5'} } } + " + end + + # Workers that load the spec files require the helper too; the one started in place of the killed one, mid-run. + status, _stdout, _stderr = rspec_process('--parallel=2', '--no-parallel-preload', '--order', 'defined') + + expect(status.exitstatus).to eq(1) + report = cd('.') { File.binread('out.txt') } + expect(report).not_to include("\0") + expect(report).to include("F0", "F2", "F3", "was started in place of the lost worker", "4 examples, 1 failure") + end + + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do + write_file_formatted 'spec/support/my_formatter.rb', " + class MyAppError < StandardError; end + + class MyFormatter + RSpec::Core::Formatters.register self, :example_group_started, :example_group_finished, :example_failed, :example_passed, :my_event, :message + + def initialize(output) + @output = output + @depth = 0 + end + + def example_group_started(notification) + @output.puts \"\#{' ' * @depth}> \#{notification.group.description} (\#{notification.group.metadata[:kind]})\" + @depth += 1 + end + + def example_group_finished(notification) + @depth -= 1 + @output.puts \"\#{' ' * @depth}< \#{notification.group.description}\" + end + + def example_passed(notification) + result = notification.example.execution_result + timed = result.started_at.is_a?(Time) && result.finished_at.is_a?(Time) && result.finished_at >= result.started_at + @output.puts \"\#{' ' * @depth}ok \#{notification.example.metadata[:flag].inspect} timed=\#{timed}\" + end + + def example_failed(notification) + e = notification.exception + @output.puts \"\#{' ' * @depth}FAILED \#{e.class.name} app_error=\#{e.is_a?(MyAppError)} std=\#{e.is_a?(StandardError)} msg=\#{e.message}\" + @output.puts notification.fully_formatted(1) + end + + def my_event(notification) + @output.puts \"custom: \#{notification.payload.inspect}\" + end + + def message(notification) + @output.puts \"message: \#{notification.message}\" + end + end + " + + write_file_formatted 'spec/custom_spec.rb', " + RSpec.describe 'One', :kind => :unit do + it('passes', :flag => 7) { RSpec.configuration.reporter.publish(:my_event, :payload => { :from => 'worker' }) } + it('raises an app error') { raise MyAppError, 'app broke' } + it('publishes something that cannot be marshaled') { RSpec.configuration.reporter.publish(:my_event, :payload => proc {}) } + end + + RSpec.describe 'Two', :kind => :feature do + it('passes') { } + end + " + + run_rspec "--order defined -r ./spec/support/my_formatter --format MyFormatter --parallel=2 spec/custom_spec.rb" + + expect(last_cmd_stdout).to include( + "> One (unit)", + " ok 7 timed=true", + " FAILED MyAppError app_error=true std=true msg=app broke", + "Failure/Error: it('raises an app error') { raise MyAppError, 'app broke' }", + "< One", + "> Two (feature)", + " ok nil timed=true", + "< Two", + "custom: #{{ :from => 'worker' }.inspect}", + "message: Parallel worker 1 could not forward the custom `my_event` notification because its options cannot be " \ + "marshaled: #{{ :payload => 0 }.inspect.chomp('0}')}# deadline + end + 10.times { |i| it(\"example \#{i}\") { sleep 0.15 } } + end + " + + run_rspec "--order defined --fail-fast --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + # Later segments of the summary line (pending, errors, ...) are not this spec's business. + expect(last_cmd_stdout).to match(/^\d+ examples?, 1 failure(, .+)?$/) + expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 12 + end + + it 'persists example statuses from every worker and reruns only the failures with --only-failures' do + write_fixed_suite + + RSpec.configuration.example_status_persistence_file_path = 'statuses.txt' + run_rspec "--order defined --parallel=2" + expect(last_cmd_stdout).to include("13 examples, 5 failures, 2 pending") + cd '.' do + statuses = File.read("statuses.txt") + expect(statuses.scan('| passed ').size).to eq(6) + expect(statuses.scan('| failed ').size).to eq(5) + expect(statuses.scan('| pending').size).to eq(2) + expect(statuses).to match(/\d+\.\d+ seconds/) + end + + RSpec.configuration.example_status_persistence_file_path = 'statuses.txt' + run_rspec "--order defined --parallel=2 --only-failures" + expect(last_cmd_stdout).to include("5 examples, 5 failures") + end + + it 'forwards deprecations and messages from the workers into the single report' do + write_file_formatted 'spec/deprecation_spec.rb', " + RSpec.configure { |c| c.before(:suite) { RSpec.configuration.reporter.message(\"hello from worker \#{RSpec.configuration.parallel_worker_number}\") } } + RSpec.describe('A') { it('deprecates') { RSpec.deprecate('old_thing', :replacement => 'new_thing') } } + RSpec.describe('B') { it('deprecates too') { RSpec.deprecate('other_thing') } } + " + + run_rspec "--parallel=2 --deprecation-out deprecations.txt spec/deprecation_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("hello from worker 1", "hello from worker 2") + expect(last_cmd_stdout).to include("2 deprecations logged to deprecations.txt") + cd '.' do + deprecations = File.read("deprecations.txt") + expect(deprecations).to include("old_thing is deprecated. Use new_thing instead.", "other_thing is deprecated.") + end + end + + it 'reports suite hook failures from workers like a serial run does, with the error exit code' do + write_file_formatted 'spec/suite_hook_spec.rb', " + RSpec.configure do |c| + c.after(:suite) { raise 'cleanup broke' if RSpec.configuration.parallel_worker_number == 2 } + end + RSpec.describe('A') { it('passes') { } } + RSpec.describe('B') { it('passes') { } } + " + + run_rspec "--parallel=2 --error-exit-code 3 spec/suite_hook_spec.rb" + + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include("An error occurred in an `after(:suite)` hook.", "cleanup broke") + expect(last_cmd_stdout).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it 'does not replay output buffered before the fork into files written by formatters' do + write_fixed_suite + + run_rspec "--order defined --format documentation --out doc.txt --parallel=3 spec/gamma_spec.rb spec/beta_spec.rb" + + cd '.' do + expect(File.read("doc.txt").scan("Randomized with seed").size).to eq(0) + expect(File.read("doc.txt").scan(/^Gamma$/).size).to eq(1) + end + end + + describe 'a worker whose own world wants to quit (an interrupt delivered to that worker only)' do + def write_self_interrupting_group(name, examples_after=2) + write_file_formatted "spec/#{name.downcase}_spec.rb", " + RSpec.describe '#{name}' do + it('interrupts itself') { Process.kill('INT', Process.pid); sleep 0.2 } + #{examples_after}.times { |i| it(\"never runs \#{i}\") { } } + end + " + end + + it 'retires that worker after the group it was running and lets the others finish, failing the run' do + write_self_interrupting_group('Interrupted') + write_file_formatted 'spec/other_spec.rb', " + RSpec.describe('Other') { 2.times { |i| it(\"passes \#{i}\") { sleep 0.05 } } } + " + write_file_formatted 'spec/third_spec.rb', " + RSpec.describe('Third') { it('passes') { } } + " + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to match(/Parallel worker \d \(pid \d+\) stopped after Interrupted .* because `RSpec.world.wants_to_quit` was set/) + expect(last_cmd_stdout).to include("4 examples, 0 failures") + expect(last_cmd_stdout).not_to include("RSpec is shutting down") + end + + it 'reports the groups nobody was left to run when every worker retires' do + %w[One Two Three Four].each { |name| write_self_interrupting_group(name, 0) } + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stdout).to match( + /^2 examples of 2 example groups did not run because no worker remained to run them\. To run them: rspec '?\.\/spec\/\w+_spec\.rb\[1\]'? '?\.\/spec\/\w+_spec\.rb\[1\]'?$/ + ) + end + end + + it 'fails the example a worker dies in, with the reason, so it can be rerun, and runs the rest of its group in another worker' do + write_file_formatted 'spec/crash_spec.rb', " + RSpec.configure { |c| c.example_status_persistence_file_path = 'status.txt' } + RSpec.describe 'Crashing' do + it('finishes') { } + it('dies') { File.write(\"dies_\#{Process.pid}\", ''); exit!(7) } + it('runs in another worker') { File.write(\"rest_\#{Process.pid}\", '') } + end + RSpec.describe('Fine') { it('passes') { } } + " + + run_rspec "--order defined --parallel=2 --error-exit-code 3 spec/crash_spec.rb" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to match( + /Failure\/Error: it\('dies'\) \{ File\.write.*exit!\(7\) \}\n\s+Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running this example; 1 example of Crashing \(\.\/spec\/crash_spec\.rb:2\) that it had not started yet will run in another worker\./ + ) + expect(last_cmd_stdout).to include("4 examples, 1 failure", "rspec ./spec/crash_spec.rb:4 # Crashing dies") + cd('.') do + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:2\] \| failed }) + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:3\] \| passed }) + expect(Dir['dies_*'].size).to eq(1) + expect(Dir['rest_*'].map { |file| file[/\d+/] }).not_to include(Dir['dies_*'].first[/\d+/]) + end + end + + it 'fails the examples of a group whose context hook a worker dies in, as a raising hook would, and runs the rest elsewhere' do + write_file_formatted 'spec/crash_spec.rb', " + RSpec.configure { |c| c.example_status_persistence_file_path = 'status.txt' } + RSpec.describe 'Crashing' do + context 'broken' do + before(:context) { exit!(7) } + it('cannot run') { } + end + context 'fine' do + it('runs in another worker') { } + end + end + RSpec.describe('Fine') { it('passes') { sleep 0.5 } } + " + + run_rspec "--order defined --parallel=2 --error-exit-code 3 spec/crash_spec.rb" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to match( + /Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running Crashing broken \(\.\/spec\/crash_spec\.rb:3\), outside of any example, so this example could not run; 1 example of Crashing \(\.\/spec\/crash_spec\.rb:2\) that it had not started yet will run in another worker\./ + ) + expect(last_cmd_stdout).to include("3 examples, 1 failure", "rspec ./spec/crash_spec.rb:5 # Crashing broken cannot run") + expect(last_cmd_stdout).not_to include("outside of examples") + cd('.') do + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:1:1\] \| failed }) + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:2:1\] \| passed }) + end + end + + describe 'when the code an example runs calls `exit`' do + # A run that a worker's `exit` ends raises `SystemExit` out of + # `Runner.run`, as a serial run does -- which would end this very run. + def run_rspec(cmd) + super + rescue SystemExit => e + @last_cmd_exit_status = e.status + end + + def write_exiting_suite(status, after_suite="") + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.after(:suite) { #{after_suite} } + end + RSpec.describe 'Exits' do + it('exits') do + sleep 0.2 + $exiting = true + exit #{status} + end + it('never runs') { File.write('ran_after_exit', '') } + end + RSpec.describe 'Keeps going' do + 20.times { |i| it(\"waits \#{i}\") { sleep 0.05 } } + end + " + end + + [0, 3].each do |status| + it "ends the run as a serial run does, with status #{status}: no further examples start" do + write_exiting_suite(status) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(status) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\) is ending the run: the code it ran called `exit` or `abort` \(status #{status}\) in Exits exits \(\.\/spec\/exit_spec\.rb:6\)\. As in a serial run, no further examples start and the run exits with status #{status}\.$/ + ) + expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 22 + expect(last_cmd_stdout).to match(/^\d+ examples?, 0 failures(, .+)?$/) + cd('.') { expect(File).not_to exist('ran_after_exit') } + end + end + + it "ends with the status of an `after(:suite)` hook's later exit, as a serial run does" do + write_exiting_suite(4, "exit 5 if $exiting") + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(5) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\): then an `after\(:suite\)` hook called `exit` or `abort` \(status 5\), so the run exits with status 5\./ + ) + expect(last_cmd_stdout).not_to include("An error occurred in a parallel worker") + end + + it 'fails the run as usual when another worker then has to be stopped by force' do + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure { |c| c.parallel_shutdown_timeout = 0.5 } + RSpec.describe('Exits') { it('exits') { sleep 0.5; exit 0 } } + RSpec.describe('Stuck') { it('sleeps') { sleep 30 } } + " + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 15 + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include( + "is ending the run: the code it ran called `exit` or `abort` (status 0)", + "did not stop within 0.5 seconds of being asked to stop early (fail-fast or interrupt) while running this example" + ) + end + + it 'is bounded when the worker then does not exit, failing the run' do + # Leaves the worker's watchdog plenty of room to fire before the + # coordinator steps in, even on a loaded machine. + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_exiting_suite(0, "if $exiting; $stderr = File.open('worker_stderr.txt', 'w'); sleep 30; end") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2 --error-exit-code 3" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\) ended the run because the code it ran called `exit` or `abort` \(status 0\), then exited with exit status 1: if its watchdog aborted it, its `after\(:suite\)` hooks or `at_exit` handlers did not finish within 0\.5 seconds/ + ) + expect(cd('.') { File.read('worker_stderr.txt') }).to include( + "did not exit within 0.5 seconds of stopping because the code it ran called `exit` or `abort` (status 0); aborting it." + ) + end + + it 'is bounded by the coordinator when the worker cannot abort itself and ignores SIGTERM' do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 0.1) + # Inherited by the forked workers: a watchdog that never gets to run. + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:arm_watchdog) + write_exiting_suite(0, "if $exiting; trap('TERM') { }; File.write('stuck.pid', Process.pid.to_s); sleep 30; end") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2 --error-exit-code 3" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + + # The shutdown timeout, the watchdog grace and both signal graces, with room for a loaded machine. + expect(elapsed).to be < 0.5 + 0.1 + 0.3 + 0.3 + 5 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include( + "did not exit within 0.5 seconds of ending the run because the code it ran called `exit` or `abort` (status 0)", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + stuck = cd('.') { Integer(File.read('stuck.pid')) } + expect { Process.kill(0, stuck) }.to raise_error(Errno::ESRCH) + end + + it 'ends the run with that status from a before(:suite) hook too, bounded when the worker then hangs' do + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.before(:suite) do + c.after(:suite) do + $stderr = File.open(\"worker_stderr_\#{Process.pid}.txt\", 'w') + sleep 30 + end + exit 4 + end + end + RSpec.describe('One') { it('never runs') { File.write('ran', '') } } + RSpec.describe('Two') { it('never runs') { File.write('ran', '') } } + " + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include("is ending the run: the code it ran called `exit` or `abort` (status 4)") + dumps = cd('.') { Dir['worker_stderr_*.txt'].map { |file| File.read(file) } } + expect(dumps).not_to be_empty + expect(dumps).to all(include("of stopping because the code it ran called `exit` or `abort` (status 4); aborting it.")) + cd('.') { expect(File).not_to exist('ran') } + end + end + + describe 'a group tagged `parallel_split: N`' do + def write_split_suite(extra="") + write_file_formatted 'spec/split_spec.rb', " + RSpec.describe 'Split', :parallel_split => 3 do + before(:context) { File.write(\"context_\#{Process.pid}_\#{rand(1_000_000)}\", '') } + 6.times { |i| it(\"e\#{i}\") { File.write(\"example_\#{i}_\#{Process.pid}\", ''); sleep 0.1 } } + context 'nested' do + 2.times { |i| it(\"n\#{i}\") { File.write(\"nested_\#{i}_\#{Process.pid}\", '') } } + end + #{extra} + end + RSpec.describe('Whole') { it('runs once') { } } + " + end + + it 'runs it as that many parts in different workers, every example once, with the counts of a serial run' do + write_split_suite + run_rspec "--order defined --format json --out serial.json" + cd('.') { Dir['{context,example,nested}_*'].each { |file| File.delete(file) } } + + run_rspec "--order defined --parallel=3 --format json --out parallel.json --format progress" + + expect(last_cmd_exit_status).to eq(0) + cd('.') do + serial, parallel = %w[serial.json parallel.json].map { |file| JSON.parse(File.read(file)) } + ids = parallel['examples'].map { |example| example['id'] } + expect(ids.size).to eq(ids.uniq.size) + expect(ids.sort).to eq(serial['examples'].map { |example| example['id'] }.sort) + expect(parallel['summary']).to include('example_count' => 9, 'failure_count' => 0) + expect(Dir['context_*'].size).to eq(3) + pids = Dir['{example,nested}_*'].map { |file| file[/\d+\z/] } + expect(pids.uniq.size).to be >= 2 + end + end + + it 'runs the rest of a part whose worker dies in another worker' do + write_split_suite("context('crashing') { it('dies') { exit!(7) }; it('after the crash') { File.write('after_crash', '') } }") + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("11 examples, 1 failure", "Split crashing dies") + cd('.') { expect(File).to exist('after_crash') } + end + end + + it 'stops its workers, even ones that cannot tear down, when it and its process group get SIGTERM, saying what they ran' do + write_file_formatted 'spec/wedged_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + 2.times do |i| + RSpec.describe(\"Wedged \#{i}\") do + it('keeps a thread that delays exit') do + File.write(\"running_\#{i}\", '') + Thread.new { begin; sleep; ensure; sleep 60; end } + sleep 60 + end + end + end + " + deadline_for = lambda { |seconds| Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds } + running = lambda do |pid| + state = `ps -o stat= -p #{pid}`.strip + !state.empty? && !state.start_with?('Z') + end + + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/wedged_spec.rb', + :out => 'stdout.txt', :err => File::NULL, :pgroup => true) + deadline = deadline_for.call(20) + sleep 0.05 until (worker_pids.size == 2 && Dir['running_*'].size == 2) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + workers = worker_pids + expect(workers.size).to eq(2) + + # What `timeout` does: the command first, then its whole process group. + Process.kill(:TERM, pid) + Process.kill(:TERM, -pid) + _, status = Process.wait2(pid) + + expect(status.termsig).to eq(Signal.list.fetch('TERM')) + expect(File.read('stdout.txt')).to include("Received SIGTERM; stopping the workers.") + expect(File.read('stdout.txt').scan(%r{^Parallel worker [12] \(pid \d+\) was running \./spec/wedged_spec\.rb:\d+ \(Wedged [01] keeps a thread that delays exit\) for [\d.]+ seconds?\.$}).size).to eq(2) + deadline = deadline_for.call(10) + sleep 0.05 until workers.none?(&running) || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + expect(workers.select(&running)).to eq([]) + end + end + + it 'leaves SIGHUP ignored when it starts ignoring it, as under nohup, like a serial run' do + write_file_formatted 'spec/hangup_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('hangs up on its own process') { Process.kill('HUP', Process.pid); sleep 0.1 } } + RSpec.describe('Two') do + it('outlasts a hangup of the whole run') do + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 20 + sleep 0.05 until File.exist?('hung_up') || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + end + end + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + ignoring_hangups = [RbConfig.ruby, '-e', 'trap("HUP", "IGNORE"); exec(*ARGV)'] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *ignoring_hangups, *rspec, '--parallel=2', 'spec/hangup_spec.rb', + :out => 'stdout.txt', :err => File::NULL) + expect(wait_until.call(20) { Dir['worker_*.pid'].size == 2 }).to be true + + Process.kill(:HUP, pid) + File.write('hung_up', '') + status = nil + finished = wait_until.call(30) { status ||= Process.waitpid2(pid, Process::WNOHANG)&.last } + Process.kill(:KILL, pid) && Process.wait(pid) unless finished + + expect(status&.exitstatus).to eq(0) + expect(File.read('stdout.txt')).to include("2 examples, 0 failures") + expect(File.read('stdout.txt')).not_to include("SIGHUP") + end + end + + it 'stops workers that cannot notice the coordinating process is gone once it is killed, and removes their files' do + write_file_formatted 'spec/stopped_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('stops itself') { Process.kill(:STOP, Process.pid) } } + RSpec.describe('Two') { it('stops itself') { Process.kill(:STOP, Process.pid) } } + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + state = lambda { |pid| `ps -o stat= -p #{pid}`.strip } + # A stopped worker runs none of its code -- like one deadlocked in a fork + # handler -- and keeps every descriptor it inherited. + stopped = lambda { |pid| state.call(pid).start_with?('T') } + running = lambda { |pid| !state.call(pid).empty? && !state.call(pid).start_with?('Z') } + + cd('.') do + Dir.mkdir('tmp') + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil, 'TMPDIR' => File.expand_path('tmp') }, *rspec, '--parallel=2', + 'spec/stopped_spec.rb', :out => File::NULL, :err => 'stderr.txt') + expect(wait_until.call(20) { worker_pids.size == 2 && worker_pids.all?(&stopped) }).to be true + workers = worker_pids + + Process.kill(:KILL, pid) + Process.wait(pid) + + expect(wait_until.call(15) { workers.none?(&running) }).to be true + expect(File.read('stderr.txt')).to include("ended without stopping its workers; stopped 2 still running") + expect(wait_until.call(5) { Dir['tmp/*'].empty? }).to be true + end + end + + it 'stops workers stuck in an example once the coordinating process is gone' do + write_file_formatted 'spec/stuck_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('sleeps') { sleep 60 } } + RSpec.describe('Two') { it('sleeps') { sleep 60 } } + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + # Orphans are adopted by whatever reaps them here -- a container's init may + # not, leaving zombies, which no longer run. + running = lambda do |pid| + state = `ps -o stat= -p #{pid}`.strip + !state.empty? && !state.start_with?('Z') + end + + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + Dir.mkdir('tmp') + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil, 'TMPDIR' => File.expand_path('tmp') }, *rspec, '--parallel=2', + 'spec/stuck_spec.rb', :out => File::NULL, :err => 'stderr.txt') + expect(wait_until.call(20) { worker_pids.size == 2 }).to be true + workers = worker_pids + expect(Dir['tmp/rspec-parallel-*'].size).to eq(1) + + Process.kill(:KILL, pid) + Process.wait(pid) + + expect(wait_until.call(10) { workers.none?(&running) }).to be true + expect(File.read('stderr.txt').scan("is exiting because the coordinating process (pid #{pid}) is gone").size).to eq(2) + # Where the workers' output went, which the coordinator would have removed. + expect(Dir['tmp/*']).to eq([]) + end + end + + describe 'a worker that finishes but does not exit' do + def write_hanging_suite(extra="") + write_file_formatted 'spec/hang_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.after(:suite) do + if RSpec.configuration.parallel_worker_number == 2 + $stderr = File.open('worker_stderr.txt', 'w') + #{extra} + thread = Thread.new { sleep } + at_exit { thread.join } + end + end + end + RSpec.describe('H1') { it('passes') { } } + RSpec.describe('H2') { it('passes') { } } + " + end + + def pids_in(output) + output.scan(/\(pid (\d+)\)/).flatten.map(&:to_i).uniq + end + + def alive?(pid) + Process.kill(0, pid) + Process.waitpid(pid, Process::WNOHANG).nil? + rescue Errno::ESRCH, Errno::ECHILD + false + end + + it 'is aborted by its own watchdog with a thread dump, within the shutdown timeout, and fails the run' do + # Leaves the worker's watchdog plenty of room to fire before the + # coordinator steps in, even on a loaded machine. + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_hanging_suite + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3 spec/hang_spec.rb" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + worker_stderr = cd('.') { File.read('worker_stderr.txt') } + + expect(last_cmd_exit_status).to eq(3) + expect(elapsed).to be < 10 + expect(last_cmd_stdout).not_to include("sending SIGTERM") + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/) + expect(worker_stderr).to include("did not exit within 0.5 seconds of finishing its last example group", "Threads at this point:") + expect(worker_stderr).to match(/[`'](Thread#)?join'/) + expect(pids_in(last_cmd_stdout)).not_to be_empty + pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } + end + + it 'is terminated, then killed, by the coordinator when its own watchdog cannot fire' do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 0.1) + # Forked workers inherit this stub, which stands in for a worker whose + # watchdog thread never gets to run (native code holding the GVL, ...). + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:arm_watchdog) + write_hanging_suite("trap('TERM') { }") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3 spec/hang_spec.rb" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + + expect(last_cmd_exit_status).to eq(3) + expect(elapsed).to be < 5 + expect(last_cmd_stdout).to include( + "did not exit within 0.5 seconds of finishing its last example group", + "so it was sent SIGTERM", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } + end + end + + # As a worker deadlocked in a fork handler (in Ruby, or a library's) is: + # it never runs any of RSpec's code, and SIGTERM does not end it. + describe 'a worker stuck before it runs any of its code' do + before do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + # Forked workers inherit this stub. + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:run).and_wrap_original do |original, *args| + if original.receiver.number == 2 + trap('TERM', 'IGNORE') + sleep + end + original.call(*args) + end + end + + def write_suite + write_file_formatted 'spec/stuck_at_fork_spec.rb', " + RSpec.configure { |c| c.parallel_shutdown_timeout = 0.5 } + RSpec.describe('S1') { it('passes') { } } + RSpec.describe('S2') { it('passes') { } } + " + end + + def pids_in(output) + output.scan(/\(pid (\d+)\)/).flatten.map(&:to_i).uniq + end + + it 'is terminated, then killed, once it misses its start deadline, and its group runs in another worker' do + write_suite + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --order defined spec/stuck_at_fork_spec.rb" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_stdout).to include( + "2 examples, 0 failures", + "never reported back after it was forked", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + expect(last_cmd_exit_status).to eq(0) + pids = pids_in(last_cmd_stdout) + expect(pids.size).to eq(1) + expect { Process.kill(0, pids.first) }.to raise_error(Errno::ESRCH) + end + end + + it 'falls back to a serial run with a warning inside another parallel worker' do + write_file_formatted 'spec/a_spec.rb', " + RSpec.describe('A') { it('passes') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('B') { it('passes') { } } + " + + @inside_another_parallel_worker = true + run_rspec "--parallel=2 spec/a_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("Ignoring `--parallel`: this process is already a worker of a parallel test run") + end + + describe 'with --no-parallel-preload' do + # A worker still loading once every group has been handed out is + # dismissed, so on a suite this small one worker can take every group + # before the other has loaded. With `wait_for_both_workers` each example + # first waits for both workers to have loaded the last file, which keeps + # groups queued until they have. + def write_marking_spec_files(wait_for_both_workers: false) + wait = "deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 30 + sleep 0.05 until Dir['loaded_m2_*'].size == 3 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline" + 3.times do |i| + write_file_formatted "spec/m#{i}_spec.rb", " + File.write(\"loaded_m#{i}_\#{Process.pid}_\#{ENV.fetch('TEST_ENV_NUMBER', 'none')}\", '') + RSpec.describe('M#{i}') do + it('runs in a process that loaded its file') do + #{wait if wait_for_both_workers} + expect(File).to exist(\"loaded_m#{i}_\#{Process.pid}_\#{ENV['TEST_ENV_NUMBER']}\") + end + end + " + end + end + + def loaders + cd('.') { Dir['loaded_*'].map { |file| file.split('_', 4).last }.tally } + end + + it 'has every worker load the spec files itself, once TEST_ENV_NUMBER is set, and run what it loaded' do + write_marking_spec_files(:wait_for_both_workers => true) + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(last_cmd_exit_status).to eq(0) + expect(loaders).to eq("none" => 3, "" => 3, "2" => 3) + end + + it 'does not use a worker that loads other examples than the coordinating process' do + write_marking_spec_files + # Worker 1 loads a second late, so worker 2 reports what it loaded while + # there are still groups to hand out, rather than being dismissed unchecked. + write_file_formatted 'spec/z_spec.rb', " + sleep 1 if ENV['TEST_ENV_NUMBER'] == '' + RSpec.describe('Z') { it('passes') { }; it('only in the second worker') { } if ENV['TEST_ENV_NUMBER'] == '2' } + " + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to match(%r{Parallel worker 2 \(pid \d+\) loaded other examples than this process from \./spec/z_spec\.rb \(}) + expect(last_cmd_stdout).to include("4 examples, 0 failures, 1 error occurred outside of examples") + expect(last_cmd_exit_status).to eq(1) + end + + it 'terminates a worker still loading long after the coordinating process finished, and runs its groups elsewhere' do + write_file_formatted 'spec/spec_helper.rb', "RSpec.configure { |c| c.parallel_shutdown_timeout = 1 }" + # Worker 1 stays busy until worker 2 is gone, however long this + # process's own loading (and so worker 2's deadline) took. + 3.times do |i| + write_file_formatted "spec/s#{i}_spec.rb", " + (File.write('loading.pid.tmp', Process.pid.to_s); File.rename('loading.pid.tmp', 'loading.pid'); sleep 600) if ENV['TEST_ENV_NUMBER'] == '2' + RSpec.describe('S#{i}') do + it('keeps the other worker busy past the deadline') do + $give_up ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) + 15 + loading = lambda do + pid = File.exist?('loading.pid') && Integer(File.read('loading.pid')) + !pid || Process.kill(0, pid) + rescue Errno::ESRCH + false + end + sleep 0.05 while loading.call && Process.clock_gettime(Process::CLOCK_MONOTONIC) < $give_up + end + end + " + end + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec '--require ./spec/spec_helper --parallel=2 --no-parallel-preload' + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 20 + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds? while loading the spec files, so it was sent SIGTERM\./) + expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") + end + + it 'warns, and runs serially rather than fork after loading, when turned off from a spec file, too late' do + write_file_formatted 'spec/late_spec.rb', " + RSpec.configure { |c| c.parallel_preload = false } + RSpec.describe('Late') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('Later') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + " + + run_rspec '--parallel=2 spec/late_spec.rb' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("Ignoring `--parallel`: `config.parallel_preload = false` was set while the spec " \ + "files were loading", "Running serially.") + end + + it 'warns, and runs serially rather than fork after loading, when the worker count is set from a spec file, too late' do + write_file_formatted 'spec/late_spec.rb', " + RSpec.configure { |c| c.parallel_workers = 2 } + RSpec.describe('Late') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('Later') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + " + + run_rspec '--no-parallel-preload spec/late_spec.rb' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("`config.parallel_workers` was set while the spec files were loading", "Running serially.") + end + + it 'leaves Kernel.rand alone in the coordinating process, never calling srand' do + write_marking_spec_files + previous = Kernel.srand(1234) + + run_rspec '--parallel=2 --no-parallel-preload' + # Drawn first: an `expect` may itself draw, by creating `RSpec.configuration` (rspec/rspec#305 does). + drawn = Kernel.rand(2**62) + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(drawn).to eq(Random.new(1234).rand(2**62)) + ensure + Kernel.srand(previous) + end + + it 'has every worker draw from a seed of its own, as forked processes do' do + %w[a b].each do |name| + write_file_formatted "spec/#{name}_spec.rb", " + RSpec.describe('#{name.upcase}') do + it('draws once every worker runs an example') do + number = RSpec.configuration.parallel_worker_number + File.write(\"running_\#{number}\", '') + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 20 + sleep 0.05 until Dir['running_*'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + File.write(\"drew_\#{number}\", rand(2**62).to_s) + end + end + " + end + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + drawn = cd('.') { Dir['drew_*'].map { |file| File.read(file) } } + expect(drawn.size).to eq(2) + expect(drawn.uniq.size).to eq(2) + end + + it 'reports the groups and examples as the worker that ran them described them' do + write_file_formatted 'spec/pid_spec.rb', " + 2.times do |i| + RSpec.describe(\"G\#{i} loaded in \#{Process.pid}\") do + it(\"ran in \#{Process.pid}\") { File.write(\"ran_\#{Process.pid}\", '') } + end + end + " + + run_rspec '--parallel=2 --no-parallel-preload --format documentation' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + described = last_cmd_stdout.scan(/(?:loaded|ran) in (\d+)/).flatten.uniq + ran = cd('.') { Dir['ran_*'].map { |file| file.delete_prefix('ran_') } } + expect(described).not_to be_empty + expect(described).to all(satisfy { |pid| ran.include?(pid) }) + expect(described).not_to include(Process.pid.to_s) + end + + it 'runs the groups of a single spec file across the workers' do + write_file_formatted 'spec/one_spec.rb', " + 3.times { |i| RSpec.describe(\"G\#{i}\") { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } } + " + + run_rspec '--parallel=2 --no-parallel-preload spec/one_spec.rb' + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(last_cmd_stderr).to eq("") + end + + it 'forks nothing for an --only-failures run that is about to abort for want of a status file' do + write_marking_spec_files + + # A real process: the abort ends it with `exit!`. + status, out, err = rspec_process('--parallel=2', '--no-parallel-preload', '--only-failures') + + expect(status.exitstatus).to eq(1) + expect(err + out).to include("example_status_persistence_file_path") + expect(loaders).to eq("none" => 3) + end + + it 'goes on like a serial run when a spec file fails to load in every process' do + write_marking_spec_files + write_file_formatted 'spec/broken_spec.rb', "RSpec.describe('Broken') { it('a') { }" + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("While loading ./spec/broken_spec.rb a `raise SyntaxError` occurred") + expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(last_cmd_stdout).not_to include("Parallel worker") + end + end +end + +# Where Ruby has no `fork` (Windows, JRuby) the workers are new +# `rspec` processes; elsewhere a `--require`d file takes `fork` away. +RSpec.describe 'Parallel execution without fork (spawned workers)', :slow do + include_context "aruba support" + include ParallelRunProcess + + def without_fork + return [] unless RSpec::Support::RubyFeatures.fork_supported? + + write_file 'no_fork.rb', "module Process\n class << self\n undef_method :fork\n end\nend\n" + %w[--require ./no_fork] + end + + # Each group's first example waits for both workers, so that neither runs + # every group before the other has started. + def write_suite(helper="") + write_file 'spec/spec_helper.rb', " + RSpec.configure do |c| + c.before(:suite) { File.write(\"worker_\#{c.parallel_worker_number}.txt\", [Process.pid, ENV['TEST_ENV_NUMBER'].inspect, c.seed].join(' ')) } + end + def both_workers_started + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 30 + sleep 0.05 until Dir['worker_*.txt'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + end + #{helper} + " + 3.times do |i| + write_file "spec/s#{i}_spec.rb", " + RSpec.describe('S#{i}') do + it('passes') { both_workers_started; puts 'printed by S#{i}' } + it('#{i == 1 ? 'fails' : 'passes too'}') { expect(#{i}).not_to eq(1) } + end + " + end + end + + it 'runs the groups in new rspec processes, with the run’s seed, and reports one run' do + write_suite + options = without_fork + %w[--require ./spec/spec_helper --parallel=2 --seed 1234 --format progress + --format json --out results.json] + + status, stdout, stderr = rspec_process(*options) + + expect(stdout).to include("6 examples, 1 failure", "Randomized with seed 1234", "printed by S0", "printed by S2") + expect(stderr).not_to include("Ignoring `--parallel`") + expect(status.exitstatus).to eq(1) + results = cd('.') { JSON.parse(File.read('results.json')) } + expect(results['summary']).to include('example_count' => 6, 'failure_count' => 1) + workers = cd('.') { Dir['worker_*.txt'].map { |file| File.read(file).split } } + expect(workers.size).to eq(2) + expect(workers.map(&:first).uniq.size).to eq(2) + expect(workers.map { |worker| worker.drop(1) }).to eq([['""', '1234'], ['"2"', '1234']]) + end + + it 'fails the example a spawned worker was running when it is killed, and runs the rest' do + write_suite "RSpec.describe('Killed') { it('dies') { both_workers_started; Process.kill(:KILL, Process.pid) } }" + + status, stdout, _stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') + + expect(stdout).to include("7 examples, 2 failures", "exited unexpectedly", "while running this example") + expect(status.exitstatus).to eq(1) + end + + it 'aborts spawned workers that do not exit within the shutdown timeout, and shows their thread dumps' do + # In the workers, whose watchdogs it sets: the coordinator keeps the default for their loading. + write_suite "RSpec.configure { |c| (c.parallel_shutdown_timeout = 1) if c.parallel_worker_number; c.after(:suite) { sleep } }" + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + status, stdout, stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') + + # Its `after(:suite)` hooks never return: that it ends at all is the bound (JRuby's startup is most of the time). + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 55 + expect(stdout).to include("6 examples, 1 failure") + expect(stderr.scan("did not exit within 1 second of finishing its last example group").size).to eq(2) + expect(status.exitstatus).to eq(1) + end +end diff --git a/rspec-core/spec/integration/spec_file_load_errors_spec.rb b/rspec-core/spec/integration/spec_file_load_errors_spec.rb index 0e58a12e3..c740b71e7 100644 --- a/rspec-core/spec/integration/spec_file_load_errors_spec.rb +++ b/rspec-core/spec/integration/spec_file_load_errors_spec.rb @@ -219,7 +219,7 @@ def foo # Extra blank lines were added in 3.4.0; strip out for compat with 3.2/3.3 expect(formatted_output.gsub("\n\n", "\n")).to include unindent(<<-EOS) SyntaxError: - --> ./tmp/aruba/broken_file.rb + --> ./#{aruba_working_directory}/broken_file.rb Unmatched keyword, missing `end' ? 1 class WorkInProgress > 2 def initialize(arg) @@ -227,7 +227,7 @@ def foo 4 end 5 end EOS - expect(formatted_output).to include %r{./tmp/aruba/broken_file.rb:\d: syntax error} + expect(formatted_output).to include %r{./#{aruba_working_directory}/broken_file.rb:\d: syntax error} expect(formatted_output).to include unindent(<<-EOS) Finished in n.nnnn seconds (files took n.nnnn seconds to load) @@ -249,7 +249,7 @@ def foo EOS # This is subset of the formatted_output, it continues slightly but differs on different Rubies - expect(formatted_output).to include "SyntaxError:\n ./tmp/aruba/broken_file.rb:5: syntax error" + expect(formatted_output).to include "SyntaxError:\n ./#{aruba_working_directory}/broken_file.rb:5: syntax error" expect(formatted_output).to include unindent(<<-EOS) Finished in n.nnnn seconds (files took n.nnnn seconds to load) diff --git a/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb b/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb index 850b35811..e2453a8f7 100644 --- a/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb +++ b/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb @@ -112,27 +112,27 @@ def expect_formatters_to_be_excluded it 'includes the current load path as an option to `ruby`, not as an option to `rspec`' do cmd = command_for([], :load_path => %W[ lp/foo lp/bar ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'lp/foo':'lp/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'lp/foo'#{File::PATH_SEPARATOR}'lp/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include("-Ilp/foo:lp/bar").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Ilp/foo#{File::PATH_SEPARATOR}lp/bar").then_include(RSpec::Core.path_to_executable) end end it 'escapes the load path entries' do cmd = command_for([], :load_path => ['l p/foo', 'l p/bar' ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'l p/foo':'l p/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'l p/foo'#{File::PATH_SEPARATOR}'l p/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include('-Il\ p/foo:l\ p/bar').then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Il\\ p/foo#{File::PATH_SEPARATOR}l\\ p/bar").then_include(RSpec::Core.path_to_executable) end end it 'supports Pathnames in the load path' do cmd = command_for([], :load_path => [Pathname('l p/foo'), Pathname('l p/bar') ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'l p/foo':'l p/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'l p/foo'#{File::PATH_SEPARATOR}'l p/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include('-Il\ p/foo:l\ p/bar').then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Il\\ p/foo#{File::PATH_SEPARATOR}l\\ p/bar").then_include(RSpec::Core.path_to_executable) end end end diff --git a/rspec-core/spec/rspec/core/configuration_spec.rb b/rspec-core/spec/rspec/core/configuration_spec.rb index 4dbad52a0..1ed6c23e3 100644 --- a/rspec-core/spec/rspec/core/configuration_spec.rb +++ b/rspec-core/spec/rspec/core/configuration_spec.rb @@ -35,6 +35,74 @@ module RSpec::Core end end + describe "#parallel_workers" do + it "defaults to `nil`, meaning a serial run" do + expect(config.parallel_workers).to be_nil + end + + it "accepts a positive integer, `:auto` or `nil`" do + config.parallel_workers = 4 + expect(config.parallel_workers).to eq(4) + config.parallel_workers = :auto + expect(config.parallel_workers).to eq(:auto) + config.parallel_workers = nil + expect(config.parallel_workers).to be_nil + end + + it "warns and runs serially for anything else" do + [0, -1, "4", 2.5, true].each do |value| + expect(RSpec).to receive(:warning).with(a_string_including("Cannot set `RSpec.configuration.parallel_workers` to `#{value.inspect}`")) + config.parallel_workers = value + expect(config.parallel_workers).to be_nil + end + end + + it "prefers the value forced from the command line" do + config.parallel_workers = 2 + config.force(:parallel_workers => :auto) + expect(config.parallel_workers).to eq(:auto) + end + end + + describe "#parallel_shutdown_timeout" do + it "defaults to 30 seconds" do + expect(config.parallel_shutdown_timeout).to eq(30) + end + + it "can be changed" do + config.parallel_shutdown_timeout = 2.5 + expect(config.parallel_shutdown_timeout).to eq(2.5) + end + end + + describe "#parallel_preload" do + it "defaults to loading the spec files before forking the workers" do + expect(config.parallel_preload?).to be true + end + + it "prefers the value forced from the command line" do + config.parallel_preload = true + config.force(:parallel_preload => false) + expect(config.parallel_preload?).to be false + end + end + + describe "#parallel_worker_number" do + it "is nil until the parallel runner sets it in a worker" do + expect(config.parallel_worker_number).to be_nil + config.parallel_worker_number = 3 + expect(config.parallel_worker_number).to eq(3) + end + end + + describe "#reporter=" do + it "replaces the reporter the configuration hands out" do + reporter = Reporter.new(config) + config.reporter = reporter + expect(config.reporter).to be(reporter) + end + end + describe "#fail_fast=" do context 'when true' do it 'is set to true' do diff --git a/rspec-core/spec/rspec/core/example_spec.rb b/rspec-core/spec/rspec/core/example_spec.rb index 213d370b0..1c21c7b90 100644 --- a/rspec-core/spec/rspec/core/example_spec.rb +++ b/rspec-core/spec/rspec/core/example_spec.rb @@ -972,4 +972,44 @@ def expect_pending_result(example) expect(example.reporter).to be reporter end end + + describe "#assign_execution_result" do + let(:example) { RSpec.describe("group").example("example", :pending => "declared") { } } + + it "copies the result of running the example elsewhere onto the example" do + started_at = Time.now + finished_at = started_at + 1 + error = RuntimeError.new("remote failure") + + example.assign_execution_result( + :status => :failed, :started_at => started_at, :finished_at => finished_at, :run_time => 1.0, + :pending_message => nil, :pending_fixed => false, :exception => error, :pending_exception => nil, + :pending => nil, :skip => true, :extra_failure_lines => ["more"], + :description => "is expected to fail", :full_description => "group is expected to fail" + ) + + expect(example.execution_result).to have_attributes( + :status => :failed, :started_at => started_at, :finished_at => finished_at, :run_time => 1.0, + :pending_message => nil, :pending_fixed => false, :exception => error, :pending_exception => nil + ) + expect(example.exception).to be(error) + expect(example.metadata).to include(:pending => "declared", :skip => true, :extra_failure_lines => ["more"]) + expect(example).to have_attributes(:description => "is expected to fail", :full_description => "group is expected to fail") + expect(RSpec::Core::Notifications::ExampleNotification.for(example)).to be_a(RSpec::Core::Notifications::FailedExampleNotification) + end + + it "marks pending results so that notifications are built like after a local run" do + pending_error = RuntimeError.new("still broken") + + example.assign_execution_result( + :status => :pending, :pending_message => "declared", :pending_fixed => false, + :pending_exception => pending_error, :pending => true + ) + + expect(example.exception).to be_nil + expect(example.execution_result.pending_exception).to be(pending_error) + expect(RSpec::Core::Notifications::ExampleNotification.for(example)).to be_a(RSpec::Core::Notifications::FailedExampleNotification) + expect(example.execution_result).not_to be_example_skipped + end + end end diff --git a/rspec-core/spec/rspec/core/filter_manager_spec.rb b/rspec-core/spec/rspec/core/filter_manager_spec.rb index babb0b506..de91b4ce4 100644 --- a/rspec-core/spec/rspec/core/filter_manager_spec.rb +++ b/rspec-core/spec/rspec/core/filter_manager_spec.rb @@ -116,6 +116,13 @@ def example_with(*args) RSpec.describe("group", *args).example("example") end + it "does not expand each example's path when no location or id filter is set" do + examples = Array.new(3) { example_with } + expect(File).not_to receive(:expand_path) + + expect(prune(examples)).to eq(examples) + end + shared_examples_for "example identification filter preference" do |type| it "prefers #{type} filter to exclusion filter" do group = RSpec.describe("group") diff --git a/rspec-core/spec/rspec/core/invocations_spec.rb b/rspec-core/spec/rspec/core/invocations_spec.rb index 8b4476f1a..db5c9fdb0 100644 --- a/rspec-core/spec/rspec/core/invocations_spec.rb +++ b/rspec-core/spec/rspec/core/invocations_spec.rb @@ -83,16 +83,47 @@ def run_invocation allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with).and_return(success) end - it "starts the bisection coordinator" do + it "starts the bisection coordinator, with a reproduction command that runs serially" do run_invocation expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( an_instance_of(Runner), - configuration_options.args, + configuration_options.args + ["--no-parallel"], an_instance_of(Formatters::BisectProgressFormatter) ) end + context "when parallel runs are configured" do + let(:original_cli_args) { %w[--bisect --parallel=3 --seed 1234] } + + it "replaces the option in the reproduction command" do + run_invocation + + expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( + an_instance_of(Runner), + %w[--bisect --seed 1234 --no-parallel], + an_instance_of(Formatters::BisectProgressFormatter) + ) + end + end + + it "takes the parallel options out of SPEC_OPTS, which win over the command line, while it bisects" do + during = [] + allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with) do + during << ENV.fetch('SPEC_OPTS', nil) + success + end + + with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do + run_invocation + expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--parallel=2 --order defined') + end + with_env_vars('SPEC_OPTS' => '--no-parallel --parallel') { run_invocation } + with_env_vars('SPEC_OPTS' => '--example "two words" --order defined') { run_invocation } + + expect(during).to eq(['--order defined', nil, '--example "two words" --order defined']) + end + context "when the bisection is successful" do it "returns 0" do exit_code = run_invocation @@ -129,7 +160,7 @@ def run_invocation expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( an_instance_of(Runner), - configuration_options.args, + configuration_options.args + ["--no-parallel"], an_instance_of(Formatters::BisectDebugFormatter) ) end diff --git a/rspec-core/spec/rspec/core/metadata_spec.rb b/rspec-core/spec/rspec/core/metadata_spec.rb index 7e1621e82..0c4046b81 100644 --- a/rspec-core/spec/rspec/core/metadata_spec.rb +++ b/rspec-core/spec/rspec/core/metadata_spec.rb @@ -633,6 +633,20 @@ def group_value_for(*args) end end + describe ":absolute_file_path" do + it "is the path of the file that defined the group, not its relative path expanded in another directory" do + dir = File.join(Dir.tmpdir, "loaded_from_here") + allow(Metadata).to receive(:relative_path_regex).and_return(%r{(\A|\s)#{Regexp.escape(dir)}(/|\s|\Z)}) + + group = instance_eval <<-EOS, File.join(dir, "defined_here_spec.rb"), 1 + RSpec.describe("group") { } + EOS + + expect(group.metadata).to include(:file_path => "./defined_here_spec.rb", + :absolute_file_path => File.join(dir, "defined_here_spec.rb")) + end + end + describe ":line_number" do def value_for(*args) value = nil diff --git a/rspec-core/spec/rspec/core/option_parser_spec.rb b/rspec-core/spec/rspec/core/option_parser_spec.rb index af3edf4ec..6eb0bb984 100644 --- a/rspec-core/spec/rspec/core/option_parser_spec.rb +++ b/rspec-core/spec/rspec/core/option_parser_spec.rb @@ -335,6 +335,43 @@ module RSpec::Core end end + describe "--parallel" do + it "requests one worker per processor without an argument" do + expect(Parser.parse(%w[ --parallel ])[:parallel_workers]).to eq(:auto) + end + + it "requests the given number of workers" do + expect(Parser.parse(%w[ --parallel=4 ])[:parallel_workers]).to eq(4) + end + + it "does not swallow a following path as its argument" do + options = Parser.parse(%w[ --parallel spec/models ]) + expect(options[:parallel_workers]).to eq(:auto) + expect(options[:files_or_directories_to_run]).to eq(%w[ spec/models ]) + end + + it "runs serially with a count of 0, like --no-parallel, overriding an earlier --parallel" do + expect(Parser.parse(%w[ --parallel=0 ])).to include(:parallel_workers => nil) + expect(Parser.parse(%w[ --parallel=4 --no-parallel ])).to include(:parallel_workers => nil) + end + + %w[ lots 2.5 -1 ].push("").each do |count| + it "aborts with a usage message for the count #{count.inspect}" do + parser = Parser.new(["--parallel=#{count}"]) + message = "Expected a non-negative integer for `--parallel`, got: #{count.inspect}" + expect(parser).to receive(:abort).with(a_string_including(message, 'use --help')) + parser.parse + end + end + end + + describe "--[no-]parallel-preload" do + it "chooses whether the spec files are loaded before forking the workers or by each of them" do + expect(Parser.parse(%w[ --parallel-preload ])).to include(:parallel_preload => true) + expect(Parser.parse(%w[ --no-parallel-preload ])).to include(:parallel_preload => false) + end + end + describe "--bisect" do it "sets the `:bisect` option" do options = Parser.parse(%w[ --bisect ]) diff --git a/rspec-core/spec/rspec/core/parallel/channel_spec.rb b/rspec-core/spec/rspec/core/parallel/channel_spec.rb new file mode 100644 index 000000000..dcbb3eedf --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/channel_spec.rb @@ -0,0 +1,140 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe Channel do + let(:ends) { Channel.pipe } + let(:reader) { ends.first } + let(:writer) { ends.last } + + after do + reader.close + writer.close + end + + it "round-trips objects as length-prefixed Marshal frames" do + writer.send([:example_finished, "./spec/a_spec.rb[1:1]", { :status => :passed, :run_time => 0.5 }]) + + messages = reader.drain + expect(messages.size).to eq(1) + message, bytes = messages.first + expect(message).to eq([:example_finished, "./spec/a_spec.rb[1:1]", { :status => :passed, :run_time => 0.5 }]) + expect(bytes).to eq(reader.bytes_read) + end + + it "carries binary data without re-encoding it" do + writer.send("caf\xC3\xA9".b) + writer.send("café") + + expect(reader.drain.map(&:first)).to eq(["caf\xC3\xA9".b, "café"]) + end + + it "decodes several frames that arrived at once" do + 3.times { |i| writer.send(i) } + + expect(reader.drain.map(&:first)).to eq([0, 1, 2]) + end + + it "keeps a partial frame that follows whole ones, and reports each frame's own size" do + packets = [:first, "second" * 100, :third].map { |message| Marshal.dump(message) } + frames = packets.map { |packet| "#{packet.bytesize}\n#{packet}" } + writer.io.write(frames[0] + frames[1] + frames[2][0, 5]) + + expect(reader.drain).to eq([[:first, frames[0].bytesize], ["second" * 100, frames[1].bytesize]]) + + writer.io.write(frames[2][5..-1]) + expect(reader.drain).to eq([[:third, frames[2].bytesize]]) + expect(reader.bytes_read).to eq(frames.sum(&:bytesize)) + end + + it "keeps a partial frame buffered without blocking until the rest arrives" do + packet = Marshal.dump(:whole) + writer.io.write("#{packet.bytesize}\n#{packet[0, 3]}") + + expect(reader.drain).to eq([]) + expect(reader.eof?).to be false + + writer.io.write(packet[3..-1]) + expect(reader.drain.map(&:first)).to eq([:whole]) + end + + it "returns nothing when no data is available, without blocking" do + expect(reader.drain).to eq([]) + end + + it "reports EOF once the writer is closed and everything has been read" do + writer.send(:last) + writer.close + + expect(reader.drain.map(&:first)).to eq([:last]) + expect(reader.eof?).to be true + end + + it "raises a ProtocolError when the peer writes something that is not a frame" do + writer.io.write("garbage\n") + + expect { reader.drain }.to raise_error(Channel::ProtocolError, /Expected a frame length/) + end + + it "retries a read that a signal interrupted before it read anything" do + writer.send(:after_the_signal) + interrupted = false + allow(reader.io).to receive(:read_nonblock).and_wrap_original do |original, *args, **options| + unless interrupted + interrupted = true + raise Errno::EINTR + end + original.call(*args, **options) + end + + expect(reader.drain.map(&:first)).to eq([:after_the_signal]) + expect(interrupted).to be true + end + + describe "#receive" do + it "blocks until a whole message is available" do + thread = Thread.new do + sleep 0.05 + writer.send(:late) + end + + expect(reader.receive).to eq(:late) + thread.join + end + + it "returns nil at EOF" do + writer.close + expect(reader.receive).to be_nil + end + + it "returns messages that were put back with #unread first" do + writer.send(:drained) + writer.send(:next) + drained = reader.drain.map(&:first) + expect(drained).to eq([:drained, :next]) + + reader.unread(:next) + writer.send(:after) + + expect(reader.receive).to eq(:next) + expect(reader.receive).to eq(:after) + end + + it "returns buffered messages before reading more" do + writer.send(:first) + writer.send(:second) + + expect(reader.receive).to eq(:first) + expect(reader.receive).to eq(:second) + end + end + + it "closes both ends idempotently" do + reader.close + writer.close + expect { reader.close }.not_to raise_error + expect { writer.close }.not_to raise_error + expect(reader).to be_closed + expect(writer).to be_closed + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb new file mode 100644 index 000000000..bf9ec5f24 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -0,0 +1,1342 @@ +require 'rspec/core/parallel/coordinator' + +# Loaded here: the fake workers below are threads of this process, and Ruby +# 3.0 warns ("loading in progress, circular require considered harmful") when +# a thread requires a file another thread is loading, which fails this suite. +require 'digest/sha2' + +module RSpec::Core::Parallel + # Stands in for `Process` so that workers can be plain threads: `kill` + # ends the thread unless the "signal" is one the fake worker is said to + # survive, and `waitpid2` reports a status once the thread is dead. + class FakeProcess + Status = Struct.new(:exitstatus, :termsig) do + def success? + termsig.nil? && exitstatus == 0 + end + + def signaled? + !termsig.nil? + end + end + + attr_reader :signals, :warmups, :waits + + def threads + @threads.values + end + + def initialize(survives=[]) + @threads = {} + @statuses = {} + @signals = [] + @survives = survives + @warmups = 0 + @waits = 0 + @next_pid = 40_000 + end + + def register(thread, &exit_status) + pid = (@next_pid += 1) + @threads[pid] = thread + @statuses[pid] = exit_status + pid + end + + def warmup + @warmups += 1 + end + + def waitpid2(pid, _flags) + @waits += 1 + thread = @threads.fetch(pid) + return nil if thread.alive? + + [pid, @statuses[pid].call || Status.new(1, nil)] + end + + def kill(name, pid) + @signals << [name, pid] + return if @survives.include?(name) + + @statuses[pid] = lambda { Status.new(nil, Signal.list[name]) } + @threads.fetch(pid).kill + end + end + + # Fake workers run their examples on threads of this process, side by side: + # those examples stay off rspec-mocks' global space, which this spec's own + # stubs live in and whose scopes are not meant to be shared between threads. + module WithoutMocksLifecycle + def setup_mocks_for_rspec; end + + def verify_mocks_for_rspec; end + + def teardown_mocks_for_rspec; end + end + + RSpec.describe Coordinator, :parallel_split => 2 do + let(:config) { RSpec.configuration } + let(:world) { RSpec.world } + let(:output) { StringIO.new } + let(:runner) { RSpec::Core::Runner.new(RSpec::Core::ConfigurationOptions.new([]), config, world) } + let(:process) { FakeProcess.new } + let(:worker_stderr) { StringIO.new } + let(:worker_exits) { {} } + let(:worker_behaviours) { {} } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + # Fake workers are threads of this process, and one told to stop arms its + # exit watchdog, a thread sleeping through `Parallel.sleep_running` for up + # to the shutdown timeout: it would outlive its example and call a later + # example's stubs of `Parallel.sleep` too. + around do |example| + threads = Thread.list + example.run + (Thread.list - threads).each { |thread| thread.kill.join(1) } + end + + before do + config.output_stream = output + config.add_formatter("progress", output) + end + + # Runs a real Worker on a thread, with its own configuration object so + # that it cannot replace the coordinator's reporter the way a forked + # worker replaces its inherited copy, and with its own copies of the + # pipe ends since the coordinator closes the child's ends after spawning + # (as it must after a fork). + # A worker started in place of a lost one with the same number behaves as + # `worker_behaviours[[number, :replacement]]` says, normally by default. + def spawn(handle, _handles) + behaviour = behaviour_of(handle) + jobs = Channel::Reader.new(handle.child_jobs.io.dup) + results = Channel::Writer.new(handle.child_results.io.dup) + exit_status = nil + + thread = Thread.new do + Thread.current.report_on_exception = false + send_started(results, behaviour) + case behaviour + when :garbage then stall_after { results.io.write("garbage\n") } + when :partial_frame then stall_after { results.io.write("999\nabc") } + when :closes_results then stall_after { results.close } + when :leaves_and_hangs then stall_after { leave_for_a_signal(jobs, results) } + when :never_loads, :never_starts, :starts_once_asked_to_quit then stall(behaviour, handle, results) + when :dies_loading then Thread.current.kill + else + load_spec_files(handle, results, behaviour) + send_extra_frames(results, behaviour, handle) + run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } + end + ensure + # Like the end of a process: its pipes close, and it is gone a moment later. + [jobs, results].each(&:close) + sleep 0.2 if behaviour == :slow_exit + end + + @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } + end + + def behaviour_of(handle) + replacement = (@started_numbers ||= []).include?(handle.number) + @started_numbers << handle.number + worker_behaviours.fetch(replacement ? [handle.number, :replacement] : handle.number, :normal) + end + + # What a real worker sends first, which the fakes standing in for one send + # too -- and then go on as they would, even if nothing reads it any more + # (dismissed before their thread got going, say). + def send_started(results, behaviour) + fakes = [:garbage, :partial_frame, :closes_results, :leaves_and_hangs, :never_loads, :dies_loading] + results.send([:started]) if fakes.include?(behaviour) + rescue Errno::EPIPE, IOError + nil + end + + # One that reports back only once it has been asked to stop early stalls then. + def stall(behaviour, handle, results) + if behaviour == :starts_once_asked_to_quit + sleep 0.01 until handle.quit_sent + results.send([:started]) + end + sleep + end + + def stall_after + yield + sleep + end + + # Takes its first job, then says it is leaving its job loop because of a + # signal -- and never does. + def leave_for_a_signal(jobs, results) + jobs.receive + results.send([:exiting, "it received SIGHUP", nil]) + end + + # Dies as it asks for its second job, as a process killed while it waits + # for work would. + def die_on_second_job(jobs) + asked = 0 + jobs.define_singleton_method(:receive) do + asked += 1 + Thread.current.kill if asked == 2 + super() + end + end + + # What a worker forked before loading reports once it has loaded the spec + # files: by default, the examples this process has. + def report_loaded(results, behaviour) + digest = RSpec::Core::Parallel.suite_digest(world) + digest = digest.merge("./spec/other_spec.rb" => "other examples") if behaviour == :loads_other_examples + digest = digest.merge("./spec/a_spec.rb" => "a", "./spec/b_spec.rb" => "b") if behaviour == :loads_other_files + failure = "An error occurred while loading ./spec/x_spec.rb. RuntimeError: boom" if behaviour == :fails_to_load + results.send([:loaded, digest, failure]) + end + + def send_extra_frames(results, behaviour, handle) + send_unknown_ids(results) if behaviour == :unknown_ids + results.send([:expectations, 3]) if behaviour == :counts_expectations + print_to(handle, results, "printed by worker #{handle.number}\n", "warned by worker #{handle.number}\n") if behaviour == :prints + print_to(handle, results, "left behind by worker #{handle.number}\n", "", :announce => false) if behaviour == :prints_unannounced + end + + # As a worker's own output goes to the files once it has moved its + # standard output and error there; the frame is what it sends next. + def print_to(handle, results, out, err, announce: true) + File.open(handle.output.paths[:out], 'ab') { |file| file.write(out) } + File.open(handle.output.paths[:err], 'ab') { |file| file.write(err) } + results.send([:output, handle.output.paths.transform_values { |path| File.size(path) }]) if announce + end + + def load_spec_files(handle, results, behaviour) + print_while_loading(handle, results, behaviour) + report_loaded(results, behaviour) if handle.loading + end + + def print_while_loading(handle, results, behaviour) + print_to(handle, results, "noise while loading\n", "") if behaviour == :prints_while_loading + print_to(handle, results, "loaded quietly\n", "", :announce => false) if behaviour == :prints_unannounced_while_loading + end + + def send_unknown_ids(results) + results.send([:example_started, "./nope_spec.rb[1:1]"]) + results.send([:example_started, "./nope_spec.rb[1:1]"]) + results.send([:group_started, "./nope_spec.rb[1]"]) + results.send([:weird]) + end + + def run_fake_worker(number, jobs, results, behaviour, &record_exit) + die_on_second_job(jobs) if behaviour == :dies_between_groups + worker_config = RSpec::Core::Configuration.new + worker_config.parallel_shutdown_timeout = 60 + Thread.current[:fake_worker_world] = RSpec::Core::World.new(worker_config) + Worker.new(number, units_of(@units), jobs, results, + :configuration => worker_config, + :world => Thread.current[:fake_worker_world], + :stderr => worker_stderr, + :exit => lambda { |status| + record_exit.call(status) + sleep if behaviour == :never_exits + exit_once_sent_term(results) if behaviour == :exits_once_sent_term + }, + :exit_now => record_exit, + :trap => lambda { |*| }, + :watch_coordinator => false).run + end + + # Its teardown hangs until it is sent SIGTERM, which an `after(:suite)` + # hook takes by calling `exit 0` -- and hangs again. + def exit_once_sent_term(results) + sleep 0.01 until @process.signals.any? { |name, _pid| name == 'TERM' } + begin + results.send([:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 0)", 0]) + rescue Errno::EPIPE, IOError + nil # as Worker#send_exiting does once nothing reads it + end + sleep + end + + # A fake worker killed with a signal is gone once its thread has died, + # which a loaded machine can delay: the grace periods are long unless a + # spec is about them, so that a slow death is not escalated. + def new_coordinator(units, worker_count, options) + units.each { |group| group.include(WithoutMocksLifecycle) } + @units = units + @process = options.fetch(:process, process) + Coordinator.new(runner, worker_count, { :process => @process, :spawn => method(:spawn), :tick => 0.01, :reaper => nil, + :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) + end + + def coordinator(groups, worker_count=2, options={}) + new_coordinator(groups, worker_count, options).tap { |instance| instance.assign(groups, units_of(groups)) } + end + + def units_of(groups) + groups.map { |group| Unit.new(group) } + end + + # Started the way `--no-parallel-preload` starts it: the workers are forked + # before the groups to run are known, and load the spec files themselves. + def started_coordinator(units, worker_count=2, options={}) + new_coordinator(units, worker_count, options).tap(&:start) + end + + # The workers here are threads standing in for forked processes, also + # where Ruby has no `fork`. + before { allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(false) } + + let(:short_graces) { { :term_grace => 0.1, :kill_grace => 0.1 } } + + def passing_group(name) + RSpec.describe(name) { it("passes") {} } + end + + def failing_group(name) + RSpec.describe(name) { it("fails") { raise "boom" } } + end + + def messages + output.string + end + + it "checks for exited workers a bounded number of times while results stream in, not on every wake-up" do + groups = %w[A B].map { |name| RSpec.describe(name) { 400.times { |i| it("passes #{i}") {} } } } + started = RSpec::Core::Parallel.monotonic_time + + expect(coordinator(groups).run).to eq(0) + + elapsed = RSpec::Core::Parallel.monotonic_time - started + expect(messages).to include("800 examples, 0 failures") + expect(process.waits).to be <= 2 * ((elapsed / WorkerLifecycle::REAP_INTERVAL).ceil + 6) + end + + it "runs every group across the workers, reports through the configured formatters and exits 0 when all pass" do + groups = %w[A B C D E].map { |name| passing_group(name) } + + exit_code = coordinator(groups, 3).run + + expect(exit_code).to eq(0) + expect(messages).to include("5 examples, 0 failures") + expect(messages).to match(/^\.\.\.\.\.$/) + expect(process.signals).to be_empty + expect(process.warmups).to eq(1) + expect(groups.flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + it "reaps workers as soon as they are gone rather than a tick after their results pipe closed" do + worker_behaviours[1] = worker_behaviours[2] = :slow_exit + started = RSpec::Core::Parallel.monotonic_time + + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :tick => 5).run + + expect(exit_code).to eq(0) + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 2 + end + + it "returns the failure exit code when an example fails" do + exit_code = coordinator([passing_group("A"), failing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(world.non_example_failure).to be_falsey + expect(messages).to include("2 examples, 1 failure", "boom") + end + + it "persists results on the real examples as they arrive so statuses survive even if replay is cut short" do + groups = [passing_group("A"), failing_group("B")] + coordinator(groups).run + + expect(groups[0].examples.first.execution_result.status).to eq(:passed) + expect(groups[1].examples.first.execution_result.status).to eq(:failed) + expect(groups[1].examples.first.exception.message).to eq("boom") + expect(groups[1].examples.first.exception).to be_a(RuntimeError) + end + + it "honours the configured exit codes" do + config.failure_exit_code = 7 + expect(coordinator([passing_group("A"), failing_group("B")]).run).to eq(7) + end + + describe "fail-fast" do + it "stops dispatching and asks busy workers to stop as soon as the limit is met, reporting a failure exit code" do + config.fail_fast = true + slow = RSpec.describe("Slow") do + it("waits") { sleep 0.3 } + it("never runs") {} + end + groups = [failing_group("Fails first"), slow, passing_group("Never dispatched")] + + exit_code = coordinator(groups, 2).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.execution_result.status).to be_nil + expect(world.wants_to_quit).to be true + end + + it "accepts a worker leaving as soon as its group ends once it was asked to stop" do + config.fail_fast = true + config.error_exit_code = 5 + # Holds the coordinator up so that the worker has finished its group + # before the failure is handled and has exited before that pass ends. + slow_formatter = Class.new do + RSpec::Core::Formatters.register self, :example_started, :example_failed + + def initialize(_output); end + + def example_started(_notification) + sleep 0.1 + end + + def example_failed(_notification) + sleep 0.2 + end + end + config.add_formatter(slow_formatter, output) + groups = [failing_group("Fails"), passing_group("Never dispatched")] + + exit_code = coordinator(groups, 1).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("1 example, 1 failure") + expect(messages).not_to include("exited unexpectedly", "outside of examples") + end + end + + it "bounds the wait for a worker that left its job loop for a signal and does not exit, and runs its group elsewhere" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = :leaves_and_hangs + groups = [passing_group("A"), passing_group("B")] + + exit_code = coordinator(groups, 2, short_graces).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to include("TERM") + expect(messages).to include( + "did not exit within 0.05 seconds of leaving its job loop (its `after(:suite)` hooks or `at_exit` handlers may " \ + "be stuck; raise `config.parallel_shutdown_timeout` if they legitimately need longer) before it started A", + ": it received SIGHUP; 1 example of A", + "2 examples, 0 failures, 1 error occurred outside of examples" + ) + end + + describe "a worker that exits while running a group" do + it "fails the example it was running with the reason, never runs it again, and runs the rest of the group in another worker" do + config.error_exit_code = 5 + runs = Hash.new(0) + crashing = RSpec.describe("Crashing") do + it("finishes") { runs[:finishes] += 1 } + it("dies") do + runs[:dies] += 1 + Thread.current.kill + end + it("runs elsewhere") { runs[:elsewhere] += 1 } + end + groups = [crashing, passing_group("B")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + dies = crashing.examples[1] + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, :failed, :passed]) + expect(runs).to eq(:finishes => 1, :dies => 1, :elsewhere => 1) + expect(dies.exception).to be_a(WorkerError) + expect(dies.exception.message).to match( + /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running this example; 1 example of Crashing \(.+\) that it had not started yet will run in another worker\.\z/ + ) + expect(dies.exception.backtrace).to eq([dies.location]) + expect(dies.execution_result.run_time).to be >= 0 + expect(messages).to include("Crashing dies", "4 examples, 1 failure", "rspec #{dies.location_rerun_argument}") + end + + it "fails the unstarted examples of the group it was in when it was not running one, as a failing context hook does" do + config.error_exit_code = 5 + crashing = RSpec.describe("Crashing") do + before(:context) { Thread.current.kill } + it("cannot run") {} + it("cannot run either") {} + end + groups = [crashing, passing_group("B")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:failed, :failed]) + expect(crashing.examples.first.exception.message).to match( + /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing \(.+\), outside of any example, so this example could not run\.\z/ + ) + expect(messages).to include("3 examples, 2 failures", "rspec #{crashing.examples.first.location_rerun_argument}") + expect(messages).not_to include("outside of examples") + end + end + + it "reports a worker that exits between groups, while it waits for a busy one, as an error of the run" do + worker_behaviours[1] = :dies_between_groups + reported = output + busy = RSpec.describe("Busy") do + it("passes once worker 1 is reported lost") do + give_up_at = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 until reported.string.include?("between example groups") || RSpec::Core::Parallel.monotonic_time > give_up_at + end + end + groups = [passing_group("A"), busy] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.map { |group| group.examples.first.execution_result.status }).to eq([:passed, :passed]) + expect(messages).to match(/Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) between example groups\./) + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + # A single worker, so that no well-behaved one can miss a deadline this + # short on a loaded machine. + describe "a worker that does not exit after finishing" do + before do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = :never_exits + end + + it "is sent SIGTERM after the shutdown timeout and the run fails with the error exit code" do + config.error_exit_code = 3 + + exit_code = coordinator([passing_group("A"), passing_group("B")], 1).run + + expect(exit_code).to eq(3) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not exit within 0.05 seconds of finishing its last example group", "so it was sent SIGTERM") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "is sent SIGKILL when it survives SIGTERM" do + process = FakeProcess.new(["TERM"]) + + exit_code = coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).to include("still alive 0.1 seconds after SIGTERM; sending SIGKILL") + end + + it "is given up on, so the run still ends, when it survives SIGKILL" do + process = FakeProcess.new(%w[TERM KILL]) + started = RSpec::Core::Parallel.monotonic_time + + exit_code = coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).to include("still alive 0.1 seconds after SIGKILL; giving up on it") + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 5 + end + + it "does not let an `exit` it calls once sent SIGTERM decide the run's exit status" do + config.error_exit_code = 3 + worker_behaviours[1] = :exits_once_sent_term + process = FakeProcess.new(["TERM"]) + + # Rescued: an `exit` deciding the status would end this suite here. + exit_code = + begin + coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run + rescue SystemExit => e + "exit(#{e.status})" + end + + expect(exit_code).to eq(3) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).not_to include("ending the run") + end + end + + # Long enough a timeout for the well-behaved worker to stop within it even + # on a loaded machine. + describe "a worker that is asked to stop early but does not" do + before { config.parallel_shutdown_timeout = 0.5 } + + it "is terminated after the shutdown timeout even when it reports back only after it was asked to stop" do + config.fail_fast = true + worker_behaviours[2] = :starts_once_asked_to_quit + groups = [failing_group("Fails"), passing_group("Held by the late worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + run = Thread.new { coordinator(groups).run } + + expect(run.join(10)).to be_truthy + expect(run.value).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not stop within 0.5 seconds of being asked to stop early") + end + + it "is terminated after the shutdown timeout so a cancelled run still ends" do + config.fail_fast = true + worker_behaviours[2] = :partial_frame + groups = [failing_group("Fails"), passing_group("Held by the stalled worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not stop within 0.5 seconds of being asked to stop early") + expect(messages).to include("1 example, 1 failure") + end + + # Fake workers share `RSpec.world`: a group started after fail-fast set + # `wants_to_quit` would be skipped, so the failure waits for it to start. + def failure_once_running(name, &body) + started = Queue.new + running = RSpec.describe(name) do + it("runs") do + started << true + instance_exec(&body) + end + end + fails = RSpec.describe("Fails") do + it("fails once #{name} runs") do + deadline = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 while started.empty? && RSpec::Core::Parallel.monotonic_time < deadline + raise "boom" + end + end + [fails, running] + end + + it "fails the example it is stuck in with the reason" do + config.fail_fast = true + groups = failure_once_running("Stuck") { sleep } + stuck = groups.last + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(stuck.examples.first.exception.message).to match( + /\AParallel worker 2 \(pid \d+\) did not stop within 0\.5 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM\.\z/ + ) + expect(messages).to include("2 examples, 2 failures") + end + + it "ignores what it sends once it was given up on, so that its example is reported once" do + config.fail_fast = true + groups = failure_once_running("Late") { sleep 1.5 } + + exit_code = coordinator(groups, 2, :process => FakeProcess.new(["TERM"]), :term_grace => 5).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.exception).to be_a(WorkerError) + expect(messages).to include("2 examples, 2 failures") + expect(messages).not_to include("sending SIGKILL") + end + end + + describe "with workers forked before the spec files are loaded" do + def run_started(units, worker_count=2, options={}) + started_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units_of(units)) }.run + end + + it "gives work to the workers that report having loaded the examples this process has" do + groups = %w[A B C].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + expect(process.signals).to be_empty + end + + it "does not use a worker that loaded other examples, and runs its share elsewhere" do + config.error_exit_code = 5 + worker_behaviours[2] = :loads_other_examples + groups = %w[A B C].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(5) + expect(messages).to match(%r{Parallel worker 2 \(pid \d+\) loaded other examples than this process from \./spec/other_spec\.rb \(.*--parallel-preload.*, so it was not used\.}) + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(process.signals.map(&:first)).to eq(["KILL"]) + end + + it "names the first spec file a worker loaded differently and counts the others" do + worker_behaviours[2] = :loads_other_files + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to match(%r{loaded other examples than this process from \./spec/a_spec\.rb and 1 more spec file \(}) + end + + it "does not use a worker whose loading failed when this process's did not" do + worker_behaviours[2] = :fails_to_load + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to include("Parallel worker 2 (pid #{process.signals.first.last}) could not load the spec files: " \ + "An error occurred while loading ./spec/x_spec.rb. RuntimeError: boom, so it was not used.") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "uses a worker whose loading failed the way this process's did, like a serial run goes on" do + worker_behaviours[2] = :fails_to_load + groups = %w[A B].map { |name| passing_group(name) } + world.non_example_failure = true + + run_started(groups) + + expect(messages).not_to include("could not load") + expect(process.signals).to be_empty + expect(groups.flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + it "terminates a worker still loading well after this process finished, and runs its share elsewhere" do + config.parallel_shutdown_timeout = 0.1 + worker_behaviours[2] = :never_loads + groups = %w[A B C].map { |name| RSpec.describe(name) { it("takes a while") { sleep 0.2 } } } + + run_started(groups) + + expect(messages).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds? while loading the spec files, so it was sent SIGTERM\./) + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(process.signals.map(&:first)).to eq(["TERM"]) + end + + it "dismisses a worker still loading once nothing is left for it" do + worker_behaviours[2] = :never_loads + groups = %w[A B].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(0) + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(messages).not_to include("Parallel worker") + end + + it "reports a worker that exits while loading" do + worker_behaviours[2] = :dies_loading + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to match(/Parallel worker 2 \(pid \d+\) exited unexpectedly \(exit status 1\) while loading the spec files\./) + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "dismisses the workers there are more of than groups to run" do + groups = %w[A B].map { |name| passing_group(name) } + + exit_code = run_started(groups, 3) + + expect(exit_code).to eq(0) + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(messages).to include("2 examples, 0 failures") + end + + it "dismisses every worker when the run does not need them, giving up on one that survives SIGKILL" do + worker_behaviours[1] = worker_behaviours[2] = :never_loads + survivor = FakeProcess.new(["KILL"]) + instance = started_coordinator([passing_group("A")], 2, :process => survivor, :kill_grace => 0.1) + + instance.dismiss + + expect(survivor.signals.map(&:first)).to eq(%w[KILL KILL]) + expect(messages.scan("survived SIGKILL; giving up on it.").size).to eq(2) + instance.dismiss + expect(survivor.signals.size).to eq(2) + ensure + survivor.threads.each(&:kill) + end + + it "goes on with the workers it could fork when a later fork fails, and dismisses only those" do + worker_behaviours[1] = :never_loads + fork_fails = lambda do |handle, handles| + raise Errno::EAGAIN, "fork(2)" if handle.number == 2 + + spawn(handle, handles) + end + instance = new_coordinator([passing_group("A")], 3, :spawn => fork_fails) + expect(RSpec).to receive(:warning).with("Started 1 of 3 parallel workers: the next could not be started " \ + "(Errno::EAGAIN: Resource temporarily unavailable - fork(2)). Going on without it", + :call_site => nil) + + instance.start + expect(instance).to be_started + started = RSpec::Core::Parallel.monotonic_time + instance.dismiss + + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(process.threads).to all(satisfy { |thread| !thread.alive? }) + expect(messages).not_to include("survived SIGKILL") + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 4 + end + end + + describe "pauses of the whole run (Ctrl-Z, a sleeping machine)" do + let(:clock) { [10.0] } + let(:instance) { coordinator([passing_group("A")]) } + let(:workers) { Array.new(2) { |i| RSpec::Core::Parallel::WorkerHandle.new(i + 1) } } + + before do + allow(RSpec::Core::Parallel).to receive(:monotonic_time) { clock.first } + instance.instance_variable_get(:@workers).concat(workers) + instance.instance_variable_set(:@last_turn_at, clock.first) + end + + after { workers.each(&:close_all) } + + def turn(seconds) + clock[0] += seconds + instance.send(:account_for_pauses) + end + + it "move a deadline back by the time the loop was paused beyond its own waits" do + workers[0].deadline = 50.0 + turn(0.2) + turn(20.0) + + expect(workers[0].deadline).to be_within(0.001).of(50.0 + 20.0 - 0.01 - RSpec::Core::Parallel::PAUSE_TOLERANCE) + end + + it "leave a deadline set after the pause, in the same turn, as it is" do + workers[0].deadline = 50.0 + clock[0] += 20.0 + workers[1].deadline = 60.0 + turn(0.1) + + expect(workers[0].deadline).to be > 65.0 + expect(workers[1].deadline).to eq(60.0) + end + + it "leave deadlines alone in turns of normal length" do + workers[0].deadline = 50.0 + 5.times { turn(0.25) } + + expect(workers[0].deadline).to eq(50.0) + end + end + + it "terminates a worker that sends data it cannot decode" do + worker_behaviours[2] = :garbage + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("sent data the coordinator could not decode (RSpec::Core::Parallel::Channel::ProtocolError") + end + + it "warns once about ids it does not know and about unexpected frames" do + worker_behaviours[2] = :unknown_ids + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages.scan("./nope_spec.rb[1:1], which was not defined when the suite was loaded").size).to eq(1) + expect(messages.scan("./nope_spec.rb[1], which was not defined").size).to eq(1) + expect(messages).to include("sent an unexpected :weird frame") + expect(messages).to include("2 examples, 0 failures") + end + + it "adds the expectations a worker counted to its reporter's count, when that keeps one" do + worker_behaviours[1] = :counts_expectations + counted = 0 + config.reporter.define_singleton_method(:increment_expectation_count) { counted += 1 } + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + # Before this spec's own expectations, which an rspec-expectations that counts them adds to. + counted_by_the_run = counted + + expect(exit_code).to eq(0) + expect(counted_by_the_run).to eq(3) + expect(messages).not_to include("unexpected") + end + + it "ignores the expectations a worker counted when its reporter keeps no count" do + worker_behaviours[1] = :counts_expectations + allow(config.reporter).to receive(:respond_to?).and_call_original + allow(config.reporter).to receive(:respond_to?).with(:increment_expectation_count).and_return(false) + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages).not_to include("unexpected") + expect(messages).to include("2 examples, 0 failures") + end + + it "shows what workers printed, where they reported it, and what they left unannounced once they are gone" do + worker_behaviours[1] = :prints + worker_behaviours[2] = :prints_unannounced + exit_code = nil + + stdout, stderr = standard_streams { exit_code = coordinator([passing_group("A"), passing_group("B")]).run } + + expect(exit_code).to eq(0) + expect(stdout).to include("printed by worker 1\n", "left behind by worker 2\n") + expect(stderr).to eq("warned by worker 1\n") + end + + it "does not show what a worker that loaded the spec files itself printed while it did" do + worker_behaviours[1] = :prints_while_loading + instance = started_coordinator([passing_group("A"), passing_group("B")]) + instance.assign(@units, units_of(@units)) + + exit_code = nil + stdout, _stderr = standard_streams { exit_code = instance.run } + + expect(exit_code).to eq(0) + expect(stdout).to eq("") + end + + it "does not show what a worker dismissed before the run used it left in its files" do + worker_behaviours[3] = :prints_unannounced_while_loading + instance = started_coordinator([passing_group("A"), passing_group("B")], 3) + instance.assign(@units, units_of(@units)) + + exit_code = nil + stdout, _stderr = standard_streams { exit_code = instance.run } + + expect(exit_code).to eq(0) + expect(stdout).to eq("") + end + + it "goes on when it cannot show what a worker printed" do + worker_behaviours[1] = :prints + exit_code = nil + + standard_streams(StringIO.new.tap(&:close_write)) { exit_code = coordinator([passing_group("A"), passing_group("B")]).run } + + expect(exit_code).to eq(0) + expect(messages).to include("2 examples, 0 failures") + end + + # What the coordinator writes to them; the formatters write to `output`. + def standard_streams(stdout=StringIO.new) + streams = [$stdout, $stderr] + $stdout = stdout + $stderr = StringIO.new + yield + [$stdout.string, $stderr.string] + ensure + $stdout, $stderr = streams + end + + it "starts workers as new `rspec` processes where there is no `fork`, giving them longer to load" do + allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(true) + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", %w[--tag fast]]) + spawned = [] + killed = [] + fake = double(:process) + allow(fake).to receive(:spawn) { |env, *command, redirects| (spawned << [env, command, redirects.keys]) && (4242 + spawned.size) } + allow(fake).to receive(:kill) { |_signal, pid| killed << pid } + allow(fake).to receive(:waitpid2) { |pid, _flags| killed.include?(pid) ? [pid, nil] : nil } + config.seed = 99 + + instance = Coordinator.new(runner, 2, :process => fake, :tick => 0.01, :kill_grace => 1) + instance.start + instance.dismiss + + expect(spawned.size).to eq(2) + env, command, redirects = spawned.first + expect(command).to eq([RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", "/app/bin/rspec", "--tag", "fast"]) + expect(env).to include("RSPEC_PARALLEL_WORKER" => "1", "RSPEC_PARALLEL_COORDINATOR" => Process.pid.to_s, + "RSPEC_PARALLEL_SEED" => "99", "TEST_ENV_NUMBER" => "") + expect(env["RSPEC_PARALLEL_STDOUT"]).to end_with("worker-1.out") + expect(redirects).to eq([:in, :out, :chdir]) + expect(killed).to eq([4243, 4244]) + expect(instance.instance_variable_get(:@boot_allowance)).to be > 0 + end + + it "reports the groups that never ran when no worker remains, once as many workers as the run has were replaced" do + crashing = Array.new(4) { |i| RSpec.describe("Crashing #{i}") { it("dies") { Thread.current.kill } } } + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(crashing + [passing_group("C"), passing_group("D")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages.scan("was started in place of the lost worker").size).to eq(2) + expect(messages).to include("No further lost workers are replaced", "4 examples, 4 failures", + "2 example groups did not run because no worker remained to run them") + end + + it "starts a worker with the lost one's number in its place, which runs the remaining groups" do + crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } + # Still running when the crash is noticed, so that groups wait to run. + slow = %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.5 } } } + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator([crashing] + slow).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(@started_numbers.sort).to eq([1, 1, 2]) + expect(messages).to match(/Parallel worker 1 \(pid \d+\) was started in place of the lost worker \(pid \d+\)\./) + expect(messages).to include("4 examples, 1 failure", "exited unexpectedly") + end + + context "when the workers loaded the spec files themselves" do + # The others still running when the crash is noticed, so that groups wait to run. + let(:groups) do + [RSpec.describe("Crashing") { it("dies") { Thread.current.kill } }] + + %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.5 } } } + end + + before { config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) } + + it "starts a new `rspec` process in place of a lost one, since forking this process would load them twice" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", []]) + instance = started_coordinator(groups) + allow(instance).to receive(:spawn_worker) { |handle, handles| spawn(handle, handles) } + instance.assign(@units, units_of(@units)) + + expect(instance.run).to eq(config.failure_exit_code) + expect(instance).to have_received(:spawn_worker).once + expect(messages).to match(/Parallel worker 1 \(pid \d+\) was started in place of the lost worker \(pid \d+\) and loads the spec files first\./) + expect(messages).to include("4 examples, 1 failure") + end + + it "starts none in place of one that never reported back while loading, which would likely not either" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", []]) + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :never_starts + # Still running when the other misses its deadline, so that groups wait to run. + slow = %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.3 } } } + instance = started_coordinator(slow) + allow(instance).to receive(:spawn_worker) { |handle, handles| spawn(handle, handles) } + instance.assign(@units, units_of(@units)) + + instance.run + + expect(instance).not_to have_received(:spawn_worker) + expect(messages).to include("never reported back") + end + + it "starts none in place of a lost one when the run was not started by the `rspec` command" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(nil) + instance = started_coordinator(groups) + instance.assign(@units, units_of(@units)) + + expect(instance.run).to eq(config.failure_exit_code) + expect(@started_numbers.sort).to eq([1, 2]) + expect(messages).to include("4 examples, 1 failure") + end + end + + it "starts no worker in place of one lost once the run stops early" do + config.fail_fast = 1 + quitting = world + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + failing = RSpec.describe("Failing") { it("fails") { raise "nope" } } + crashing = RSpec.describe("Crashing") do + it("dies once the run stops") do + sleep 0.01 until quitting.wants_to_quit + Thread.current.kill + end + end + + exit_code = coordinator([crashing, failing, passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(@started_numbers.sort).to eq([1, 2]) + expect(messages).not_to include("was started in place") + end + + it "stops dispatching when the world wants to quit, as on Ctrl-C" do + groups = [RSpec.describe("Interrupting") { it("interrupts") { RSpec.world.wants_to_quit = true } }, passing_group("B"), passing_group("C")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups, 1).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.execution_result.status).to be_nil + end + + it "reports a suite hook failure in a worker as an error of the run, counted in the summary" do + failing = RSpec.describe("Reports a suite failure") do + it("fails the suite") do + error = RuntimeError.new("after suite broke") + error.set_backtrace(["./spec/support/hooks.rb:7"]) + RSpec.configuration.reporter.notify_non_example_exception(error, "An error occurred in an `after(:suite)` hook.") + end + end + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + config.error_exit_code = 9 + + exit_code = coordinator([failing, passing_group("B")], 1).run + + expect(exit_code).to eq(9) + expect(world.non_example_failure).to be true + expect(messages).to include("An error occurred in an `after(:suite)` hook.", "after suite broke", "./spec/support/hooks.rb:7") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "lets a formatter's own exception surface instead of blaming the worker" do + broken = Class.new do + RSpec::Core::Formatters.register self, :example_passed + def initialize(_output); end + + def example_passed(_notification) + raise ArgumentError, "formatter bug" + end + end + config.add_formatter(broken, output) + + expect { coordinator([passing_group("A"), passing_group("B")]).run }.to raise_error(ArgumentError, "formatter bug") + expect(messages).not_to include("could not decode") + end + + it "flushes the formatters' and configuration's streams before forking so workers do not replay buffered output" do + file_output = StringIO.new + allow(file_output).to receive(:flush).and_call_original + config.add_formatter(RSpec::Core::Formatters::ProgressFormatter.new(file_output)) + + coordinator([passing_group("A"), passing_group("B")]).run + + expect(file_output).to have_received(:flush).at_least(:once) + expect(file_output.string).to include("2 examples, 0 failures") + end + + it "runs every group in the workers it could start when the system refuses more" do + failing_spawn = lambda do |handle, handles| + raise Errno::EMFILE if handle.number > 1 + + spawn(handle, handles) + end + expect(RSpec).to receive(:warning).with(a_string_starting_with("Started 1 of 3 parallel workers: the next could not " \ + "be started (Errno::EMFILE: Too many open files)"), + :call_site => nil) + + exit_code = coordinator(%w[A B C].map { |name| passing_group(name) }, 3, :spawn => failing_spawn).run + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + expect(process.signals).to be_empty + end + + it "runs the groups in this process, like a serial run, when it cannot start any worker" do + ran_in = [] + groups = %w[A B].map { |name| RSpec.describe(name) { it("records its process") { ran_in << Process.pid } } } + config.before(:suite) { ran_in << :before_suite } + expect(RSpec).to receive(:warning).with(a_string_including("Started 0 of 2 parallel workers").and(ending_with("Running serially")), + :call_site => nil) + + exit_code = coordinator(groups, 2, :spawn => lambda { |*| raise Errno::EAGAIN, "fork" }).run + + expect(exit_code).to eq(0) + expect(ran_in).to eq([:before_suite, Process.pid, Process.pid]) + expect(messages).to include("2 examples, 0 failures") + expect(Signal.trap("TERM", "DEFAULT")).not_to be_a(Proc) + end + + it "treats a non-example failure reported by a worker as one of the run" do + failing = RSpec.describe("Reports a suite failure") do + it("fails the suite") { RSpec.configuration.reporter.notify_non_example_exception(RuntimeError.new("suite"), "in a hook") } + end + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + config.error_exit_code = 9 + + exit_code = coordinator([failing, passing_group("B"), passing_group("C")], 1).run + + expect(exit_code).to eq(9) + expect(world.non_example_failure).to be true + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + end + + it "stops giving groups to a worker whose own world wants to quit and lets the others finish the queue" do + quitting = RSpec.describe("Quits") do + it("sets the flag on the worker's world") { Thread.current[:fake_worker_world].wants_to_quit = true } + end + groups = [quitting] + %w[B C D E].map { |name| passing_group(name) } + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups, 2).run + + expect(exit_code).to eq(0) + expect(messages).to match(/Parallel worker \d \(pid \d+\) stopped after Quits .* because `RSpec.world.wants_to_quit` was set/) + expect(messages).to include("5 examples, 0 failures") + expect(groups.drop(1).flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + describe "a worker that never reports back after it was forked" do + it "is terminated once it misses its start deadline, and the group it was given runs in another worker" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :never_starts + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages).to include("2 examples, 0 failures") + expect(messages.scan("never reported back after it was forked").size).to eq(1) + expect(messages).to include("the group it was given runs in another worker") + expect(process.signals.map(&:first)).to eq(["TERM"]) + end + + it "says Ruby did not get as far as RSpec when it was started as a new process" do + allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(true) + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :never_starts + + coordinator([passing_group("A"), passing_group("B")]).run + + expect(messages).to include("never reported back after it was started (Ruby did not get as far as loading RSpec)") + end + + it "says the group it was given did not run when the run stops early" do + config.parallel_shutdown_timeout = 0.05 + config.fail_fast = true + worker_behaviours[2] = :never_starts + groups = [failing_group("Fails"), passing_group("Held by the stuck worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("never reported back after it was forked", "the group it was given did not run") + expect(messages).to include("1 example, 1 failure") + end + + it "is terminated once it misses its start deadline, and what no worker ran is reported" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = worker_behaviours[2] = :never_starts + worker_behaviours[[1, :replacement]] = worker_behaviours[[2, :replacement]] = :never_starts + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages.scan("never reported back after it was forked").size).to eq(4) + expect(messages).to include("No further lost workers are replaced", "did not run because no worker remained to run them") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM TERM TERM]) + end + end + + it "requeues a group when its worker cannot be reached and lets another worker run it" do + config.parallel_shutdown_timeout = 5 + # Its pipe closes as the first job is written to it: the write fails, + # and the worker sees the end of it, as when the coordinator is gone. + spawn_with_closed_jobs = lambda do |handle, handles| + pid = spawn(handle, handles) + if handle.number == 2 + handle.jobs.define_singleton_method(:send) do |message| + io.close_write + super(message) + end + end + pid + end + groups = [passing_group("A"), passing_group("B"), passing_group("C")] + + exit_code = coordinator(groups, 2, :spawn => spawn_with_closed_jobs).run + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + end + + it "terminates the workers that are still running when the coordinator itself fails" do + worker_behaviours[1] = :partial_frame + worker_behaviours[2] = :partial_frame + instance = coordinator([passing_group("A"), passing_group("B")]) + allow(instance).to receive(:dispatch_work).and_raise("coordinator bug") + + expect { instance.run }.to raise_error("coordinator bug") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM]) + end + + it "gives up on live workers that survive both signals when the coordinator itself fails" do + worker_behaviours[1] = :partial_frame + worker_behaviours[2] = :partial_frame + process = FakeProcess.new(%w[TERM KILL]) + instance = coordinator([passing_group("A"), passing_group("B")], 2, short_graces.merge(:process => process)) + allow(instance).to receive(:dispatch_work).and_raise("coordinator bug") + + expect { instance.run }.to raise_error("coordinator bug") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM KILL KILL]) + expect(messages).to include("survived SIGKILL; giving up on it") + end + + it "stops the workers when it is sent SIGTERM, saying what they were running, then goes the way a serial run would" do + handler = nil + allow(RSpec::Core::Parallel).to receive(:trap_signals) do |signals, &block| + expect(signals).to eq(%w[TERM HUP]) + handler = block + { "TERM" => "DEFAULT" } + end + allow(RSpec::Core::Parallel).to receive(:restore_signals) + stuck_started = Queue.new + stuck = RSpec.describe("Stuck") do + it("sleeps") do + stuck_started << true + sleep + end + end + groups = [passing_group("A"), stuck] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + signaller = Thread.new do + sleep 0.01 until handler && process.signals.empty? && messages.include?(".") && !stuck_started.empty? + handler.call("TERM") + end + + expect { coordinator(groups).run }.to raise_error(SignalException, "SIGTERM") + signaller.join + # The worker that ran A may not have been reaped yet, and gets one too. + expect(process.signals.map(&:first).uniq).to eq(["TERM"]) + expect(messages).to include("Received SIGTERM; stopping the workers.") + # Not defined in a loaded spec file, so it is named by its id rather than its line. + expect(messages).to match( + /^Parallel worker 2 \(pid \d+\) was running #{Regexp.escape(stuck.examples.first.id)} \(Stuck sleeps\) for [\d.]+ seconds?\.$/ + ) + expect(messages).to match(/^2 examples, 0 failures(, .+)?$/) + expect(messages).not_to include("outside of examples") + expect(stuck.examples.first.execution_result.status).to be_nil + expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") + end + + it "copes with a worker that is gone by the time it is signalled" do + worker_behaviours[2] = :never_exits + config.parallel_shutdown_timeout = 0.05 + allow(process).to receive(:kill).and_raise(Errno::ESRCH) + + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, short_graces).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("so it was sent SIGTERM", "sending SIGKILL", "giving up on it") + end + + describe WorkerHandle do + it "owns both ends of its two pipes and can close the coordinator's or all of them" do + handle = WorkerHandle.new(1) + expect([handle.jobs, handle.results, handle.child_jobs, handle.child_results]).to all(satisfy { |channel| !channel.closed? }) + + handle.close_parent_ends + expect([handle.jobs, handle.results]).to all(be_closed) + expect([handle.child_jobs, handle.child_results]).to all(satisfy { |channel| !channel.closed? }) + + handle.close_all + expect([handle.jobs, handle.results, handle.child_jobs, handle.child_results]).to all(be_closed) + end + end + + it "treats a worker reaped by someone else as exited with an unknown status" do + allow(process).to receive(:waitpid2).and_raise(Errno::ECHILD) + config.parallel_shutdown_timeout = 0.05 + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("exited unexpectedly (already reaped)") + end + + it "bounds the wait for a worker whose results pipe closed while it was running" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = :closes_results + + coordinator([passing_group("A"), passing_group("B")], 1).run + + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("stopped reporting for 0.05 seconds", "so it was sent SIGTERM") + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb new file mode 100644 index 000000000..fe52ca373 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -0,0 +1,199 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core + RSpec.describe Parallel::Plan do + let(:config) { RSpec.configuration } + let(:runner) { Runner.new(ConfigurationOptions.new([]), config, RSpec.world) } + let(:plan) { Parallel::Plan.new(runner) } + let(:started) { instance_double(Parallel::Coordinator, :start => nil, :started? => true, :assign => nil, :dismiss => nil) } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + def groups(count) + Array.new(count) { |i| RSpec.describe("G#{i}") { it("e") {} } } + end + + describe "where there is no `fork`" do + before do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(["exe/rspec", []]) + allow(config).to receive(:files_to_run).and_return(%w[./spec/a_spec.rb ./spec/b_spec.rb ./spec/c_spec.rb]) + end + + it "starts the workers before loading, as new processes that load the spec files, whatever `parallel_preload` says" do + config.parallel_workers = 2 + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(started) + + plan.before_loading + + all = groups(3) + expect(plan.coordinator_for(all)).to eq(started) + expect(started).to have_received(:assign).with(all, anything) + end + + it "warns, and runs serially, when the worker count is set while the spec files load" do + plan.before_loading + config.parallel_workers = 2 + expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_workers` was set while the spec files"), + :call_site => nil) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + end + + describe "by default, the spec files being loaded before the workers are forked" do + it "forks nothing before loading, then leaves the decision to Parallel.coordinator_for" do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) + config.parallel_workers = 2 + expect(Parallel::Coordinator).not_to receive(:new) + plan.before_loading + + all = groups(3) + expect(Parallel).to receive(:coordinator_for).with(runner, all).and_return(:coordinator) + expect(plan.coordinator_for(all)).to eq(:coordinator) + end + end + + context "when every worker is to load the spec files itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + before { config.parallel_preload = false } + + def spec_files(count) + allow(config).to receive(:files_to_run).and_return(Array.new(count) { |i| "./spec/f#{i}_spec.rb" }) + end + + it "forks the requested workers before loading, and gives them the groups" do + config.parallel_workers = 3 + spec_files(5) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(started) + plan.before_loading + expect(started).to have_received(:start) + + all = groups(1).tap { |list| list.first.describe("nested") { it("e") {} } } + groups(2) + expect(plan.coordinator_for(all)).to be(started) + expect(started).to have_received(:assign).with(all, all.map { |group| Parallel::Unit.new(group) }) + end + + it "forks no more workers than there are spec files to load" do + config.parallel_workers = 8 + spec_files(3) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(started) + + plan.before_loading + end + + it "forks two workers for a single spec file, which may define several groups" do + config.parallel_workers = 8 + spec_files(1) + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(started) + + plan.before_loading + end + + it "dismisses the workers it forked when starting the others fails unexpectedly" do + config.parallel_workers = 2 + failing = instance_double(Parallel::Coordinator, :dismiss => nil) + allow(failing).to receive(:start).and_raise(RuntimeError, "unexpected") + allow(Parallel::Coordinator).to receive(:new).and_return(failing) + + expect { plan.before_loading }.to raise_error(RuntimeError, "unexpected") + plan.dismiss + expect(failing).to have_received(:dismiss) + end + + it "runs serially when no worker could be started" do + config.parallel_workers = 2 + allow(Parallel::Coordinator).to receive(:new).and_return(started) + allow(started).to receive(:started?).and_return(false) + + plan.before_loading + + expect(plan.coordinator_for(groups(3))).to be_nil + expect(started).not_to have_received(:assign) + end + + it "dismisses them, and runs serially, when fewer than two groups are to run" do + config.parallel_workers = 3 + allow(Parallel::Coordinator).to receive(:new).and_return(started) + plan.before_loading + + expect(plan.coordinator_for(groups(1))).to be_nil + expect(started).to have_received(:dismiss) + end + + it "runs serially, without forking or warning again, when a parallel run is not possible" do + config.parallel_workers = 2 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(nil) + expect(RSpec).to receive(:warning).once.with(a_string_including("without `fork`"), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "forks nothing for one worker" do + config.parallel_workers = 1 + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "forks nothing before an --only-failures run without a status file, which aborts once the files are loaded" do + config.parallel_workers = 2 + config.force(:only_failures => true) + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + end + + it "warns, and runs serially, when the worker count is set while the spec files load" do + plan.before_loading + config.parallel_workers = 2 + expect(RSpec).to receive(:warning).with(a_string_including( + "Ignoring `--parallel`: `config.parallel_workers` was set while the spec files were loading, too late to fork " \ + "the workers before them as `config.parallel_preload = false` asks." + ).and(ending_with("Running serially")), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "warns, and runs serially, when the status file --only-failures needs is configured while the spec files load" do + config.parallel_workers = 2 + config.force(:only_failures => true) + plan.before_loading + config.example_status_persistence_file_path = "examples.txt" + expect(RSpec).to receive(:warning).with(a_string_including("`config.example_status_persistence_file_path` " \ + "(for `--only-failures`) was set while the spec files were loading"), + :call_site => nil) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + end + + it "warns, and runs serially, when told while the spec files load that each worker is to load them itself", + :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + plan.before_loading + config.parallel_preload = false + expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_preload = false` was set while the spec " \ + "files were loading"), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "does not warn about the fork point of a run that is not parallel" do + plan.before_loading + config.parallel_preload = false + expect(RSpec).not_to receive(:warning) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "has nothing to dismiss when it forked nothing" do + expect { plan.dismiss }.not_to raise_error + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/reaper_spec.rb b/rspec-core/spec/rspec/core/parallel/reaper_spec.rb new file mode 100644 index 000000000..ee6a7b94d --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/reaper_spec.rb @@ -0,0 +1,72 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe Reaper, :skip => !RSpec::Support::RubyFeatures.fork_supported? do + def alive?(pid) + state = `ps -o stat= -p #{pid}`.strip + !state.empty? && !state.start_with?('Z') + end + + def wait_until(seconds) + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until yield || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + yield + end + + it "ends when told the run is done, without signalling anything" do + reaper = Reaper.start(nil) + sleeper = Process.spawn('sleep', '30') + reaper.watch(sleeper) + + reaper.finish + + expect { Process.kill(0, reaper.pid) }.to raise_error(Errno::ESRCH) + expect(alive?(sleeper)).to be true + ensure + Process.kill('KILL', sleeper) && Process.wait(sleeper) if sleeper + end + + it "stops only the registered processes of the run's process group once its parent is gone without saying so" do + script = <<-RUBY + #{$LOAD_PATH.map { |path| "$LOAD_PATH.unshift(#{path.to_s.inspect})" }.join("\n")} + require 'rspec/core' + require 'rspec/core/parallel/coordinator' + registered = Process.spawn('sleep', '30', :out => File::NULL) + elsewhere = Process.spawn('sleep', '30', :out => File::NULL, :pgroup => true) + unregistered = Process.spawn('sleep', '30', :out => File::NULL) + reaper = RSpec::Core::Parallel::Reaper.start(nil) + reaper.watch(registered) + reaper.watch(elsewhere) + puts [registered, elsewhere, unregistered].join(' ') + STDOUT.flush + exit!(0) + RUBY + registered, elsewhere, unregistered = IO.popen([RbConfig.ruby, '-e', script], :err => File::NULL, &:read).split.map(&:to_i) + + expect(wait_until(15) { !alive?(registered) }).to be true + expect([alive?(elsewhere), alive?(unregistered)]).to eq([true, true]) + ensure + [registered, elsewhere, unregistered].compact.each do |pid| + Process.kill('KILL', pid) if alive?(pid) + rescue Errno::ESRCH + nil + end + end + + it "is not started when the system refuses it a process, and the run goes on" do + process = double(:process) + allow(process).to receive(:spawn).and_raise(Errno::EAGAIN) + + expect(Reaper.start(nil, process)).to be_nil + end + + it "is told nothing more once it is gone" do + reaper = Reaper.start(nil) + Process.kill('KILL', reaper.pid) + Process.wait(reaper.pid) + + expect { 100.times { reaper.watch(1) } }.not_to raise_error + expect { reaper.finish }.not_to raise_error + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb new file mode 100644 index 000000000..309c6753b --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb @@ -0,0 +1,283 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe RemoteException do + include FormatterSupport + + def round_trip(exception) + payload = Marshal.load(Marshal.dump(RemoteException.payload_for(exception))) + RemoteException.rebuild(payload) + end + + def raise_with_cause + begin + raise "the cause" + rescue + raise ArgumentError, "the effect" + end + rescue ArgumentError => e + e + end + + it "returns nil for nil" do + expect(RemoteException.payload_for(nil)).to be_nil + expect(RemoteException.rebuild(nil)).to be_nil + end + + it "rebuilds an exception whose class reports the original name and is a subclass of the real class" do + original = begin + raise ArgumentError, "bad arg" + rescue ArgumentError => e + e + end + + rebuilt = round_trip(original) + + expect(rebuilt).to be_a(ArgumentError) + expect(rebuilt.class).not_to equal(ArgumentError) + expect(rebuilt.class.name).to eq("ArgumentError") + expect(rebuilt.class.to_s).to eq("ArgumentError") + expect(rebuilt.class.inspect).to eq("ArgumentError") + expect(rebuilt.message).to eq("bad arg") + expect(rebuilt.to_s).to eq("bad arg") + expect(rebuilt.inspect).to eq("#") + expect(rebuilt.backtrace).to eq(original.backtrace) + expect(rebuilt.cause).to be_nil + end + + it "reuses one generated class per original class" do + a = round_trip(ArgumentError.new("a")) + b = round_trip(ArgumentError.new("b")) + + expect(a.class).to equal(b.class) + end + + it "rebuilds the cause chain" do + rebuilt = round_trip(raise_with_cause) + + expect(rebuilt.cause).to be_a(RuntimeError) + expect(rebuilt.cause.message).to eq("the cause") + expect(rebuilt.cause.cause).to be_nil + end + + it "bounds the depth of the cause chain it ships" do + exception = (1..(RemoteException::MAX_CAUSE_DEPTH + 5)).inject(nil) do |cause, i| + error = RuntimeError.new("level #{i}") + error.instance_variable_set(:@__cause, cause) + error.define_singleton_method(:cause) { @__cause } + error + end + + payload = RemoteException.payload_for(exception) + depth = 0 + while payload + depth += 1 + payload = payload[:cause] + end + expect(depth).to eq(RemoteException::MAX_CAUSE_DEPTH + 1) + end + + it "falls back to a StandardError stand-in named like a class that does not exist here" do + original = Class.new(StandardError) { def self.name = "Only::InWorker" }.new("gone") + rebuilt = round_trip(original) + + expect(rebuilt).to be_a(StandardError) + expect(rebuilt.class.name).to eq("Only::InWorker") + expect(rebuilt.message).to eq("gone") + end + + it "does not resolve names that are not constant paths or not exception classes" do + expect(RemoteException.resolve("String")).to be_nil + expect(RemoteException.resolve("not a constant")).to be_nil + expect(RemoteException.resolve("Kernel")).to be_nil + expect(RemoteException.resolve(nil)).to be_nil + expect(RemoteException.resolve("RSpec::Core::ExampleGroup")).to be_nil + expect(RemoteException.resolve("RSpec::Expectations::ExpectationNotMetError")).to eq(RSpec::Expectations::ExpectationNotMetError) + end + + it "rescues errors raised while resolving a constant" do + stub_const("Explosive", Module.new { def self.const_missing(_) = raise("boom") }) + expect(RemoteException.resolve("Explosive::Nope")).to be_nil + end + + it "preserves an anonymous error class as such" do + rebuilt = round_trip(Class.new(StandardError).new("anon")) + + expect(rebuilt.class.name).to be_nil + expect(rebuilt.class.to_s).to match(/# "b") + payload = RemoteException.payload_for(original) + expect(payload[:message]).to eq("A Needs::Arguments for which `exception.message.to_s` raises NoMethodError.") + + rebuilt = RemoteException.rebuild(payload) + expect(rebuilt).to be_a(klass) + expect(rebuilt.message).to eq("A Needs::Arguments for which `exception.message.to_s` raises NoMethodError.") + end + + it "ships the detailed message so that SyntaxError formatting matches", + :skip => !Exception.method_defined?(:detailed_message) && "needs Exception#detailed_message (Ruby 3.2+)" do + original = begin + eval("}", binding, __FILE__, __LINE__) + rescue SyntaxError => e + e + end + + # syntax_suggest's part of it depends on a search that can time out. + detailed = original.detailed_message(:highlight => false) + allow(original).to receive(:detailed_message).and_return(detailed) + + rebuilt = round_trip(original) + expect(rebuilt).to be_a(SyntaxError) + expect(rebuilt.detailed_message).to eq(detailed) + end + + it "falls back to the message when no detailed message was shipped" do + rebuilt = RemoteException.rebuild(:class_name => "RuntimeError", :message => "plain", :backtrace => nil, :cause => nil) + expect(rebuilt.detailed_message).to eq("plain") + expect(rebuilt.backtrace).to be_nil + end + + it "tolerates a detailed_message that raises" do + original = RuntimeError.new("ok") + def original.detailed_message(*) = raise("nope") + expect(RemoteException.payload_for(original)[:detailed_message]).to be_nil + end + + it "keeps `set_backtrace` working on the rebuilt exception so common backtrace truncation applies" do + rebuilt = round_trip(RuntimeError.new("x").tap { |e| e.set_backtrace(%w[a b c]) }) + copy = rebuilt.dup + copy.set_backtrace(%w[a]) + + expect(copy.backtrace).to eq(%w[a]) + expect(rebuilt.backtrace).to eq(%w[a b c]) + end + + it "uses the backtrace it was sent even when the original class computes its own from state it lacks" do + # Like `Minitest::UnexpectedError`, which rspec-rails specs can raise. + wrapping = Class.new(StandardError) do + attr_accessor :error + + def initialize(error) + super("Unexpected exception") + self.error = error + end + + def backtrace + error.backtrace + end + + def set_backtrace(backtrace) # rubocop:disable Naming/AccessorMethodName + error.set_backtrace(backtrace) + end + end + stub_const("WrappingError", wrapping) + inner = RuntimeError.new("inner") + inner.set_backtrace(["./spec/inner_spec.rb:3"]) + + rebuilt = round_trip(WrappingError.new(inner)) + copy = rebuilt.dup + copy.set_backtrace(["./spec/other_spec.rb:1"]) + + expect(rebuilt).to be_a(WrappingError) + expect(rebuilt.backtrace).to eq(["./spec/inner_spec.rb:3"]) + expect(rebuilt.backtrace_locations).to be_nil + expect(copy.backtrace).to eq(["./spec/other_spec.rb:1"]) + presented = RSpec::Core::Formatters::ExceptionPresenter.new(rebuilt, new_example).fully_formatted(1) + expect(presented).to include("Unexpected exception", "./spec/inner_spec.rb:3") + end + + context "with an exception that aggregates several failures" do + def aggregate_failures_error + RSpec::Expectations::FailureAggregator.new("checking things", :hide_backtrace => true).aggregate do + expect(1).to eq(2) + expect(3).to eq(4) + raise "boom" + end + rescue RSpec::Expectations::MultipleExpectationsNotMetError => e + e + end + + it "rebuilds the multiple exception interface" do + original = aggregate_failures_error + rebuilt = round_trip(original) + + expect(RSpec::Core::MultipleExceptionError::InterfaceTag === rebuilt).to be true + expect(rebuilt).to be_a(RSpec::Expectations::MultipleExpectationsNotMetError) + expect(rebuilt.all_exceptions.map(&:message)).to eq(original.all_exceptions.map(&:message)) + expect(rebuilt.failures.size).to eq(original.failures.size) + expect(rebuilt.other_errors.map { |e| e.class.name }).to eq(["RuntimeError"]) + expect(rebuilt.aggregation_metadata).to eq(original.aggregation_metadata) + expect(rebuilt.aggregation_block_label).to eq(original.aggregation_block_label) + expect(rebuilt.summary).to eq(original.summary) + expect(rebuilt.exception_count_description).to eq(original.exception_count_description) + end + + it "gives the stand-in the interface when the class does not exist here" do + payload = RemoteException.payload_for(RSpec::Core::MultipleExceptionError.new(RuntimeError.new("a"))) + payload[:class_name] = "Unknown::Aggregate" + + rebuilt = RemoteException.rebuild(payload) + expect(RSpec::Core::MultipleExceptionError::InterfaceTag === rebuilt).to be true + expect(rebuilt.all_exceptions.map(&:message)).to eq(["a"]) + end + + it "replaces aggregation metadata that cannot be marshaled with the backtrace flag only" do + original = RSpec::Core::MultipleExceptionError.new(RuntimeError.new("a")) + original.aggregation_metadata[:proc] = lambda {} + + payload = RemoteException.payload_for(original) + expect(payload[:multiple][:aggregation_metadata]).to eq(:hide_backtrace => true) + end + end + + describe "presenter parity" do + def formatted(exception) + example = new_example(:description => "the example") + example.metadata[:absolute_file_path] = __FILE__ + example.execution_result.status = :failed + example.execution_result.exception = exception + RSpec::Core::Formatters::ExceptionPresenter::Factory.new(example).build.fully_formatted(1) + end + + it "produces the same fully formatted failure for a plain exception with a cause" do + original = raise_with_cause + expect(formatted(round_trip(original))).to eq(formatted(original)) + end + + it "produces the same fully formatted failure for an expectation failure" do + original = begin + expect(1).to eq(2) + rescue RSpec::Expectations::ExpectationNotMetError => e + e + end + + expect(formatted(round_trip(original))).to eq(formatted(original)) + expect(formatted(original)).to include("Failure/Error") + expect(formatted(original)).not_to include("ExpectationNotMetError") + end + + it "produces the same fully formatted failure for aggregated failures" do + original = RSpec::Core::MultipleExceptionError.new( + RuntimeError.new("first").tap { |e| e.set_backtrace(["./spec/a_spec.rb:1", "./spec/a_spec.rb:9"]) }, + RSpec::Expectations::ExpectationNotMetError.new("second").tap { |e| e.set_backtrace(["./spec/a_spec.rb:2", "./spec/a_spec.rb:9"]) } + ) + original.set_backtrace(["./spec/a_spec.rb:9"]) + + rebuilt = round_trip(original) + expect(formatted(rebuilt)).to eq(formatted(original)) + expect(formatted(rebuilt)).to include("Got 1 failure and 1 other error") + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/results_spec.rb b/rspec-core/spec/rspec/core/parallel/results_spec.rb new file mode 100644 index 000000000..34e4e03ac --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/results_spec.rb @@ -0,0 +1,34 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe Results do + let(:example) do + RSpec.describe("G", :gone => true, :kept => "k") { it("e", :tags => ["a"]) {} }.examples.first + end + let(:results) { Results.new } + + def payload(metadata) + now = RSpec::Core::Parallel.time_to_wire(::Time.now) + { :status => :passed, :started_at => now, :finished_at => now, :run_time => 0.0, :metadata => metadata } + end + + it "applies the metadata the worker's run set and deleted when it applies the result" do + results.record(example, payload(:set => { :tags => %w[a b], :screenshot => "x.png" }, :deleted => [:gone])) + expect(example.metadata).to include(:gone => true, :tags => ["a"]) + + results.apply(example) + + expect(example.metadata).to include(:tags => %w[a b], :screenshot => "x.png", :kept => "k") + expect(example.metadata).not_to have_key(:gone) + expect(example.execution_result.status).to eq(:passed) + end + + it "leaves the metadata alone when the worker's run changed none, or only sent names of unsent values" do + before = example.metadata.dup + results.record(example, payload(:unsent => { ":callback" => "Proc" })) + results.apply(example) + + expect(example.metadata.except(:execution_result)).to eq(before.except(:execution_result)) + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb new file mode 100644 index 000000000..243330dfe --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -0,0 +1,443 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + class RecordingListener + attr_reader :events + + def initialize + @events = [] + end + + RSpec::Core::Reporter::RSPEC_NOTIFICATIONS.each do |name| + define_method(name) { |notification| @events << [name, label(notification)] } + end + + def my_event(notification) + @events << [:my_event, notification.to_h] + end + + private + + def label(notification) + if notification.respond_to?(:group) + notification.group.description + elsif notification.respond_to?(:example) + notification.example.description + elsif notification.respond_to?(:message) + notification.message + end + end + end + + RSpec.describe Sequencer do + let(:reporter) { RSpec::Core::Reporter.new(RSpec.configuration) } + let(:listener) { RecordingListener.new } + let(:applied) { [] } + let(:now) { [0.0] } + let(:clock) { lambda { now.first } } + + let(:g1) do + RSpec.describe("G1") do + it("a") {} + context("N") { it("b") {} } + end + end + let(:g2) { RSpec.describe("G2") { it("c") {} } } + let(:g3) { RSpec.describe("G3") { it("d") {} } } + + def ex(group, description) + (group.examples + group.children.flat_map(&:examples)).find { |e| e.description == description } + end + + def sequencer(sequenced) + reporter.register_listener(listener, *RSpec::Core::Reporter::RSPEC_NOTIFICATIONS, :my_event) + Sequencer.new(reporter, :apply_result => lambda { |example| applied << example }, :clock => clock, + :sequenced => sequenced, + :output => lambda { |stream, bytes| listener.events << [:output, "#{stream}:#{bytes}"] }) + end + + def run_example(seq, worker, example, status=:passed) + example.execution_result.status = status + seq.example_started(worker, example) + seq.example_finished(worker, example) + end + + def run_group(seq, worker, group, duration=1.5) + seq.group_started(worker, group) + group.examples.each { |example| run_example(seq, worker, example) } + group.children.each { |child| run_group(seq, worker, child, nil) } + seq.group_finished(worker, group, duration) + end + + def events + listener.events + end + + describe "immediate mode (no formatter follows group boundaries)" do + it "delivers every notification as it arrives, re-emitting group boundaries so the current group is always right" do + seq = sequencer(false) + + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 1, ex(g1, "a")) + run_example(seq, 2, ex(g2, "c"), :failed) + seq.group_finished(2, g2, 0.5) + seq.group_finished(1, g1, 1.0) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:example_group_finished, "G1"], [:example_group_started, "G2"], + [:example_group_finished, "G2"], [:example_group_started, "G1"], + [:example_started, "a"], [:example_finished, "a"], [:example_passed, "a"], + [:example_group_finished, "G1"], [:example_group_started, "G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_failed, "c"], + [:example_group_finished, "G2"], + [:example_group_started, "G1"], + [:example_group_finished, "G1"] + ]) + expect(seq.stats[:peak_queued]).to eq(0) + expect(seq).not_to be_sequenced + end + + it "applies the result right before delivering example_finished" do + seq = sequencer(false) + seq.group_started(1, g1) + run_example(seq, 1, ex(g1, "a")) + + expect(applied).to eq([ex(g1, "a")]) + end + + it "records the parts of a group split across workers as taking as long as they did together" do + seq = sequencer(false) + recorded = [] + allow(reporter).to receive(:record_example_group_duration) { |group, total| recorded << [group, total] } + + run_group(seq, 1, g1, 1.5) + run_group(seq, 2, g1, 2.0) + + expect(recorded).to eq([[g1, 1.5], [g1, 3.5]]) + end + + it "records the worker-measured duration of top-level groups for the profiler" do + seq = sequencer(false) + expect(reporter).to receive(:record_example_group_duration).with(g1, 1.5) + expect(reporter).not_to receive(:record_example_group_duration).with(g1.children.first, anything) + + run_group(seq, 1, g1) + end + end + + describe "sequenced mode" do + it "streams the first group live and replays the others whole, in completion order, once it finishes" do + seq = sequencer(true) + + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.group_started(3, g3) + run_example(seq, 2, ex(g2, "c")) + run_example(seq, 1, ex(g1, "a")) + run_example(seq, 3, ex(g3, "d")) + seq.group_finished(3, g3, 0.1) + seq.group_finished(2, g2, 0.2) + expect(events.map(&:last)).not_to include("G2", "G3", "c", "d") + expect(seq.stats[:queued]).to eq(8) + expect(seq.stats[:peak_queued]).to eq(8) + + seq.group_started(1, g1.children.first) + run_example(seq, 1, ex(g1, "b")) + seq.group_finished(1, g1.children.first, nil) + seq.group_finished(1, g1, 0.3) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:example_started, "a"], [:example_finished, "a"], [:example_passed, "a"], + [:example_group_started, "N"], + [:example_started, "b"], [:example_finished, "b"], [:example_passed, "b"], + [:example_group_finished, "N"], + [:example_group_finished, "G1"], + [:example_group_started, "G3"], + [:example_started, "d"], [:example_finished, "d"], [:example_passed, "d"], + [:example_group_finished, "G3"], + [:example_group_started, "G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_passed, "c"], + [:example_group_finished, "G2"] + ]) + expect(seq.stats[:queued]).to eq(0) + expect(seq.stats[:flushes]).to eq(1) + expect(seq).to be_sequenced + end + + it "promotes the longest-waiting unfinished group when the live one finishes and streams it live from then on" do + seq = sequencer(true) + + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.group_started(3, g3) + seq.group_finished(1, g1, 0.1) + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"] + ]) + + run_example(seq, 3, ex(g3, "d")) + run_example(seq, 2, ex(g2, "c")) + expect(events.last).to eq([:example_passed, "c"]) + expect(events.map(&:last)).not_to include("d") + end + + it "treats a group with no queued notifications as finished when its final notification arrives live" do + seq = sequencer(true) + run_group(seq, 1, g2) + run_group(seq, 2, g3) + + expect(events.map(&:last)).to eq(%w[G2 c c c G2 G3 d d d G3]) + end + + it "flushes finished groups when the live group has been silent for #{Sequencer::SILENCE_BOUND} seconds" do + seq = sequencer(true) + seq.group_started(1, g1) + run_group(seq, 2, g2) + expect(events.map(&:last)).to eq(%w[G1]) + + now[0] = Sequencer::SILENCE_BOUND - 0.1 + seq.tick + expect(events.map(&:last)).to eq(%w[G1]) + + now[0] = Sequencer::SILENCE_BOUND + seq.tick + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c G2]) + + run_example(seq, 1, ex(g1, "a")) + expect(events.map(&:last)[7..-1]).to eq(%w[G1 a a a]) + expect(seq.stats[:flushes]).to eq(1) + end + + it "flushes finished groups once one has waited #{Sequencer::WAIT_BOUND} seconds behind a live group that keeps talking" do + seq = sequencer(true) + seq.group_started(1, g1) + run_group(seq, 2, g2) + + (1..Sequencer::WAIT_BOUND.to_i).each do |second| + now[0] = second.to_f + run_example(seq, 1, ex(g1, "a")) + seq.tick + break if events.map(&:last).include?("G2") + end + + expect(now.first).to eq(Sequencer::WAIT_BOUND) + expect(events.map(&:last)).to include("G2", "c") + end + + it "does nothing on tick while nothing finished is waiting" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + now[0] = 100.0 + + expect { seq.tick }.not_to(change { events }) + end + + it "falls back to delivering everything queued, finished or not, past #{Sequencer::QUEUE_BOUND} notifications" do + stub_const("RSpec::Core::Parallel::Sequencer::QUEUE_BOUND", 3) + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + ex(g2, "c").execution_result.status = :passed + seq.example_started(2, ex(g2, "c")) + expect(events.map(&:last)).to eq(%w[G1]) + + seq.example_finished(2, ex(g2, "c")) + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c]) + expect(seq.stats[:queued]).to eq(0) + + seq.group_finished(2, g2, 0.1) + expect(seq.stats[:queued]).to eq(1) + end + + it "also delivers everything queued once the queued notifications reach #{Sequencer::QUEUE_BYTES_BOUND} bytes" do + stub_const("RSpec::Core::Parallel::Sequencer::QUEUE_BYTES_BOUND", 1_000) + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.message(2, "big", 600) + expect(events.map(&:last)).to eq(%w[G1]) + expect(seq.stats[:queued_bytes]).to eq(600) + + # As it is queued, not at the next tick: output read in chunks would pile up until then. + seq.message(2, "bigger", 600) + expect(events.map(&:last)).to eq(%w[G1 G1 G2 big bigger]) + expect(seq.stats).to include(:queued => 0, :queued_bytes => 0, :peak_queued_bytes => 1_200) + end + + it "delivers messages sent outside any group immediately, without touching the open group" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.message(2, "from a suite hook") + seq.deprecation(2, :message => "old", :deprecated => "x", :replacement => nil, :call_site => nil) + seq.custom(2, :my_event, :a => 1) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:message, "from a suite hook"], + [:deprecation, "old"], + [:my_event, { :a => 1 }] + ]) + end + + it "queues messages sent from inside a queued group with that group" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.message(2, "inside G2") + expect(events.map(&:last)).to eq(%w[G1]) + + seq.group_finished(2, g2, 0.1) + seq.group_finished(1, g1, 0.1) + expect(events.map(&:last)).to eq(["G1", "G1", "G2", "inside G2", "G2"]) + end + + it "tracks the bytes of queued frames" do + seq = sequencer(true) + seq.group_started(1, g1, 10) + seq.group_started(2, g2, 20) + seq.example_started(2, ex(g2, "c"), 30) + + expect(seq.stats).to include(:queued => 2, :queued_bytes => 50, :peak_queued_bytes => 50) + end + end + + describe "#output" do + it "shows what a worker printed in a group with that group, queued with it while another group is live" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.output(2, :out, "printed in G2") + run_example(seq, 2, ex(g2, "c")) + seq.output(2, :err, "warned in G2") + seq.group_finished(2, g2, 0.1) + expect(events).to eq([[:example_group_started, "G1"]]) + expect(seq.stats[:queued_bytes]).to eq("printed in G2".bytesize + "warned in G2".bytesize) + + seq.group_finished(1, g1, 0.1) + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"], [:output, "out:printed in G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_passed, "c"], + [:output, "err:warned in G2"], [:example_group_finished, "G2"] + ]) + end + + it "re-opens the group output was printed in when another group was shown since" do + seq = sequencer(false) + seq.group_started(1, g1) + seq.group_started(2, g2) + + seq.output(1, :out, "printed in G1") + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"], [:example_group_finished, "G2"], + [:example_group_started, "G1"], [:output, "out:printed in G1"] + ]) + end + + it "writes what a worker printed outside any group as it comes, without touching the groups shown" do + seq = sequencer(true) + seq.group_started(1, g1) + + seq.output(2, :out, "from before(:suite)") + + expect(events).to eq([[:example_group_started, "G1"], [:output, "out:from before(:suite)"]]) + end + end + + describe "#abandon" do + it "balances the group notifications of a live group that will never finish" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(1, g1.children.first) + + seq.abandon(1) + + expect(events.map(&:last)).to eq(%w[G1 N N G1]) + seq.group_started(2, g2) + expect(events.last).to eq([:example_group_started, "G2"]) + end + + it "delivers what a queued group reported before closing it" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 2, ex(g2, "c")) + + seq.abandon(2) + + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c G2]) + end + + it "ignores workers without a stream" do + seq = sequencer(true) + expect { seq.abandon(9) }.not_to(change { events }) + end + end + + describe "#finish" do + it "delivers everything queued and closes the groups still open" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 2, ex(g2, "c")) + seq.group_started(3, g3) + seq.group_finished(3, g3, 0.1) + + seq.finish + + expect(events.map(&:last)).to eq(%w[G1 G1 G3 G3 G2 c c c G2]) + expect(seq.stats[:queued]).to eq(0) + end + end + + describe ".group_aware_listeners?" do + def reporter_with(formatter, *args) + RSpec.configuration.add_formatter(formatter, *args) + RSpec.configuration.reporter + end + + it "is false for formatters that only inherit BaseFormatter's bookkeeping, like progress and JSON" do + expect(Sequencer.group_aware_listeners?(reporter_with("progress", StringIO.new))).to be false + expect(Sequencer.group_aware_listeners?(reporter_with("json", StringIO.new))).to be false + end + + it "is true for the documentation and HTML formatters" do + expect(Sequencer.group_aware_listeners?(reporter_with("documentation", StringIO.new))).to be true + end + + it "is true for the HTML formatter, which reacts to group starts" do + expect(Sequencer.group_aware_listeners?(reporter_with("html", StringIO.new))).to be true + end + + it "is true for a custom listener of group finishes" do + reporter.register_listener(Class.new { def example_group_finished(_); end }.new, :example_group_finished) + expect(Sequencer.group_aware_listeners?(reporter)).to be true + end + + it "is true for a custom formatter in a process that never loaded BaseFormatter" do + hide_const("RSpec::Core::Formatters::BaseFormatter") + reporter.register_listener(Class.new { def example_group_started(_); end }.new, :example_group_started) + expect(Sequencer.group_aware_listeners?(reporter)).to be true + end + + it "ignores RSpec's own profiler" do + reporter.register_listener(RSpec::Core::Profiler.new, :example_group_started, :example_group_finished) + expect(Sequencer.group_aware_listeners?(reporter)).to be false + end + + it "is consulted by default" do + reporter.register_listener(Class.new { def example_group_finished(_); end }.new, :example_group_finished) + expect(Sequencer.new(reporter, :apply_result => lambda { |_| })).to be_sequenced + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb new file mode 100644 index 000000000..16d60d247 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb @@ -0,0 +1,97 @@ +require 'fileutils' +require 'tmpdir' +require 'tempfile' +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe SpawnedWorker do + describe ".environment" do + it "tells a worker its number, the coordinator, the seed and its output files, and sets TEST_ENV_NUMBER" do + env = SpawnedWorker.environment(2, 4242, 1234, :out => "/tmp/w.out", :err => "/tmp/w.err") + + expect(env).to eq("RSPEC_PARALLEL_WORKER" => "2", "RSPEC_PARALLEL_COORDINATOR" => "4242", + "RSPEC_PARALLEL_SEED" => "1234", "TEST_ENV_NUMBER" => "2", + "RSPEC_PARALLEL_STDOUT" => "/tmp/w.out", "RSPEC_PARALLEL_STDERR" => "/tmp/w.err") + expect(SpawnedWorker.environment(1, 4242, 1234, nil)).to include("TEST_ENV_NUMBER" => "") + expect(SpawnedWorker.environment(1, 4242, 1234, nil)).not_to have_key("RSPEC_PARALLEL_STDOUT") + end + end + + describe ".settings" do + it "takes what the coordinator told it out of the environment, so that processes it starts are not workers" do + env = SpawnedWorker.environment(3, 4242, 99, :out => "o", :err => "e").merge("OTHER" => "kept") + + expect(SpawnedWorker.settings(env)).to eq(:number => 3, :coordinator => 4242, :seed => 99, + :output => { :out => "o", :err => "e" }) + expect(env).to eq("TEST_ENV_NUMBER" => "3", "OTHER" => "kept") + end + + it "has no output files when it was given none" do + env = SpawnedWorker.environment(3, 4242, 99, nil) + expect(SpawnedWorker.settings(env)[:output]).to be_nil + end + end + + describe ".configure" do + let(:configuration) { RSpec::Core::Configuration.new } + # Instead of this project's `.rspec` and the user's. + let(:options_file) { Tempfile.new('rspec-options').tap(&:close).path } + + def configure(args, settings) + SpawnedWorker.configure(['-O', options_file] + args, settings, configuration) + end + + it "configures the worker from the command line, without the formatters and the files the coordinator writes" do + configure(%w[--format json --out results.json --deprecation-out deprecations.txt --tag fast], + :number => 2, :seed => 1234) + + expect(configuration.formatters).to be_empty + expect(configuration.deprecation_stream).not_to eq("deprecations.txt") + expect(configuration.inclusion_filter.rules).to eq(:fast => true) + expect(configuration.parallel_worker_number).to eq(2) + expect(File.exist?("results.json")).to be false + end + + it "opens no file for a formatter its `--require`d files add, which the coordinator writes to" do + dir = Dir.mktmpdir + helper = File.join(dir, 'helper.rb') + File.write(helper, "RSpec.configure { |c| c.add_formatter 'documentation', #{File.join(dir, 'out.txt').inspect} }\n") + File.write(File.join(dir, 'out.txt'), "the coordinator's report") + + # The run's configuration, which `RSpec.configure` in the helper configures. + SpawnedWorker.configure(['-O', options_file, '--require', helper], { :number => 2, :seed => 1 }, RSpec.configuration) + + expect(File.read(File.join(dir, 'out.txt'))).to eq("the coordinator's report") + ensure + FileUtils.remove_entry(dir) if dir + end + + it "takes the run's seed without choosing an ordering, which the run's configuration does" do + configure(%w[--order defined], :number => 2, :seed => 1234) + + expect(configuration.seed).to eq(1234) + expect(configuration.ordering_registry.fetch(:global)).to be_a(RSpec::Core::Ordering::Identity) + end + + it "keeps a seed given on the command line, which is the run's too" do + configure(%w[--seed 77], :number => 2, :seed => 77) + + expect(configuration.seed).to eq(77) + expect(configuration.ordering_registry.fetch(:global)).to be_a(RSpec::Core::Ordering::Random) + end + end + end + + RSpec.describe RSpec::Core::Ordering::ConfigurationManager do + it "adopts a seed without changing the ordering, unless a seed was forced" do + manager = RSpec::Core::Ordering::ConfigurationManager.new + manager.adopt_seed(12) + expect(manager.seed).to eq(12) + expect(manager.seed_used?).to be false + + manager.force(:seed => 34) + manager.adopt_seed(56) + expect(manager.seed).to eq(34) + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb new file mode 100644 index 000000000..2d74df336 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -0,0 +1,69 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe WorkerHandle do + it "closes the pipe it opened when the system refuses it the second one" do + first = Channel.pipe + allow(Channel).to receive(:pipe).and_invoke(lambda { first }, lambda { raise Errno::EMFILE }) + + expect { WorkerHandle.new(1) }.to raise_error(Errno::EMFILE) + expect(first.map(&:closed?)).to eq([true, true]) + end + + describe "#signal" do + let(:handle) { WorkerHandle.new(1).tap { |h| h.pid = 4242 } } + let(:sent) { [] } + + after { handle.close_all } + + it "sends SIGKILL where SIGTERM cannot be sent (Windows)" do + process = double(:process) + allow(process).to receive(:kill) do |name, pid| + raise Errno::EINVAL if name == 'TERM' + + sent << [name, pid] + end + + handle.signal(process, 'TERM', :term, 1.0) + + expect(sent).to eq([['KILL', 4242]]) + expect(handle.escalation).to eq(:term) + end + end + + describe "#spawn_worker" do + it "starts the command with its jobs on standard input and its results on standard output" do + handle = WorkerHandle.new(2) + process = double(:process) + expect(process).to receive(:spawn).with({ "A" => "1" }, "ruby", "rspec", :in => handle.child_jobs.io, + :out => handle.child_results.io, :chdir => Dir.pwd).and_return(4242) + + expect(handle.spawn_worker(process, %w[ruby rspec], "A" => "1")).to eq(4242) + handle.close_all + end + end + + describe "the deadline for its first frame" do + let(:handle) { WorkerHandle.new(1) } + + after { handle.close_all } + + it "is cleared by its first frame, postponed or not" do + handle.start_deadline = 5.0 + handle.postpone(1.0) + handle.clear_start_deadline + + expect(handle.deadline).to be_nil + end + + it "is replaced by any other deadline, which its first frame then leaves alone" do + handle.start_deadline = 5.0 + handle.deadline = 2.0 # asked to stop early before it reported back, say + + handle.clear_start_deadline + + expect(handle.deadline).to eq(2.0) + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb new file mode 100644 index 000000000..aa03913a8 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb @@ -0,0 +1,108 @@ +require 'fileutils' +require 'tmpdir' +require 'rspec/core/parallel/coordinator' + +# These specs stand in for the process's own streams, which a worker moves. +# rubocop:disable-next Style/GlobalStdStream +module RSpec::Core::Parallel + RSpec.describe WorkerOutput do + let!(:dir) { Dir.mktmpdir } + let!(:output) { WorkerOutput.new(dir, 3) } + + after do + output.close + FileUtils.remove_entry(dir) + end + + def read(sizes=nil) + chunks = [] + output.read(sizes) { |stream, bytes| chunks << [stream, bytes] } + chunks + end + + it "creates an empty file per stream, named after the worker" do + expect(output.paths.values.map { |path| File.basename(path) }).to eq(%w[worker-3.out worker-3.err]) + expect(output.paths.values.map { |path| File.size(path) }).to eq([0, 0]) + end + + it "yields what was written up to the given sizes, each byte once, and the rest when asked for everything" do + File.write(output.paths[:out], "one two") + File.write(output.paths[:err], "warn") + + expect(read(:out => 3, :err => 0)).to eq([[:out, "one"]]) + expect(read(:out => 3, :err => 0)).to eq([]) + expect(read).to eq([[:out, " two"], [:err, "warn"]]) + expect(read).to eq([]) + end + + it "moves past what it was told to skip" do + File.write(output.paths[:out], "loading...done") + output.skip(:out => 10, :err => 0) + + expect(read).to eq([[:out, "done"]]) + end + + it "reads a flood of output in chunks" do + stub_const("RSpec::Core::Parallel::WorkerOutput::CHUNK", 4) + File.write(output.paths[:out], "0123456789") + + expect(read.map(&:last)).to eq(%w[0123 4567 89]) + end + + it "yields nothing for files that are gone" do + output.paths.each_value { |path| File.delete(path) } + expect(read).to eq([]) + end + + it "writes what a worker printed as it was written, also to a stream with an encoding of its own" do + # A file, which (unlike a StringIO) transcodes what is written to it; binary, so that Windows adds no CR. + path = File.join(dir, 'stdout') + original = $stdout + $stdout = File.open(path, 'wb:UTF-8') + begin + WorkerOutput.write(:out, "caf\xC3\xA9 \xFF\n".b) + ensure + $stdout.close + $stdout = original + end + + expect(File.binread(path)).to eq("caf\xC3\xA9 \xFF\n".b) + end + + it "goes without a directory when none can be made" do + allow(Dir).to receive(:mktmpdir).and_raise(Errno::EACCES) + expect(WorkerOutput.make_directory).to be_nil + expect { WorkerOutput.remove_directory(nil) }.not_to raise_error + end + + describe WorkerOutput::Capture do + let(:results) { Channel.pipe } + let(:stdout) { File.open(File.join(dir, 'stdout'), 'w') } + let(:stderr) { File.open(File.join(dir, 'stderr'), 'w') } + + before do + stub_const('STDOUT', stdout) + stub_const('STDERR', stderr) + end + + after do + results.each(&:close) + [stdout, stderr].each(&:close) + end + + it "moves standard output and error into the files and says how far they reach before a frame that follows output" do + capture = WorkerOutput::Capture.new(output.paths, results.last) + STDOUT.write("printed") + STDERR.write("warned") + + capture.send([:first]) + capture.send([:second]) + + frames = results.first.drain.map(&:first) + expect(frames).to eq([[:output, { :out => 7, :err => 6 }], [:first], [:second]]) + expect(read(frames.first.last)).to eq([[:out, "printed"], [:err, "warned"]]) + expect([STDOUT.sync, STDERR.sync]).to eq([true, true]) + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb new file mode 100644 index 000000000..8ad9b5091 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -0,0 +1,318 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe WorkerReporter do + let(:results) { Channel.pipe } + let(:jobs) { Channel.pipe } + let(:reporter) { WorkerReporter.new(RSpec.configuration, results.last, jobs.first) } + + def frames + results.first.drain.map(&:first) + end + + # What the reporter dumped itself. + def load_dump(dump) + Marshal.load(dump) # rubocop:disable Security/MarshalLoad + end + + after do + results.each(&:close) + jobs.each(&:close) + end + + it "forwards group and example notifications as frames with ids and result payloads" do + group = RSpec.describe("Outer") do + it("passes") { expect(1).to eq(1) } + it("fails") { raise ArgumentError, "nope" } + it("is pending") do + pending "later" + raise "x" + end + it("is skipped") { skip "not now" } + context("Inner") { it("nested") {} } + end + inner = group.children.first + + group.run(reporter) + + kinds = frames + expect(kinds.map(&:first)).to eq([ + :group_started, + :example_started, :example_finished, + :example_started, :example_finished, + :example_started, :example_finished, + :example_started, :example_finished, + :group_started, :example_started, :example_finished, :group_finished, + :group_finished + ]) + + expect(kinds[0]).to eq([:group_started, group.id, { :description => "Outer", :full_description => "Outer" }]) + expect(kinds[9]).to eq([:group_started, inner.id, { :description => "Inner", :full_description => "Outer Inner" }]) + expect(kinds[12]).to eq([:group_finished, inner.id, nil]) + expect(kinds[13][0..1]).to eq([:group_finished, group.id]) + expect(kinds[13][2]).to be_a(Float) + + payloads = kinds.select { |f| f.first == :example_finished }.to_h { |f| [f[1], f[2]] } + passed, failed, pending, skipped = group.examples.map { |example| payloads.fetch(example.id) } + + expect(passed).to include(:status => :passed, :exception => nil, :pending_exception => nil, :pending => nil, :skip => nil, + :description => "passes", :full_description => "Outer passes") + expect(passed[:started_at]).to eq([group.examples.first.execution_result.started_at.tv_sec, + group.examples.first.execution_result.started_at.tv_nsec]) + expect(passed[:finished_at]).to all(be_an(Integer)) + expect(passed[:run_time]).to be_a(Float) + + expect(failed[:status]).to eq(:failed) + expect(failed[:exception]).to include(:class_name => "ArgumentError", :message => "nope") + + expect(pending).to include(:status => :pending, :pending_message => "later", :pending_fixed => false, :pending => true) + expect(pending[:pending_exception]).to include(:class_name => "RuntimeError", :message => "x") + + expect(skipped).to include(:status => :pending, :pending_message => "not now", :skip => true, :pending_exception => nil) + end + + it "ships the example ids the coordinator can look up" do + group = RSpec.describe("G") { it("e") {} } + group.run(reporter) + + expect(frames.select { |f| f.first == :example_started }).to eq([[:example_started, group.examples.first.id]]) + end + + it "ships the description a one-liner's matcher generated while it ran" do + group = RSpec.describe("G") { it { expect(1).to eq(1) } } + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload).to include(:description => "is expected to eq 1", :full_description => "G is expected to eq 1") + end + + it "forwards extra failure lines and stringifies result parts that cannot be marshaled" do + group = RSpec.describe("G") do + it("fails") do |ex| + ex.metadata[:extra_failure_lines] = ["more", lambda {}] + raise "x" + end + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:extra_failure_lines].size).to eq(2) + expect(payload[:extra_failure_lines].first).to eq("more") + expect(payload[:extra_failure_lines].last).to match(/Proc/) + end + + it "forwards messages and deprecations" do + reporter.message("hello") + reporter.message(:not_a_string) + reporter.deprecation(:deprecated => "foo", :message => "msg", :replacement => "bar", :call_site => "here") + + expect(frames).to eq([ + [:message, "hello"], + [:message, :not_a_string], + [:deprecation, { :deprecated => "foo", :message => "msg", :replacement => "bar", :call_site => "here" }] + ]) + end + + it "stringifies messages and deprecation parts that cannot be marshaled" do + reporter.message(lambda {}) + reporter.deprecation(:deprecated => lambda {}, :message => "msg", :replacement => nil, :call_site => nil) + + message, deprecation = frames + expect(message.first).to eq(:message) + expect(message.last).to match(/Proc/) + expect(deprecation.last[:deprecated]).to match(/Proc/) + expect(deprecation.last).to include(:message => "msg", :replacement => nil) + end + + it "forwards custom notifications published through the reporter" do + reporter.publish(:my_event, :a => 1, :b => "two") + reporter.publish(:bare_event) + + expect(frames).to eq([[:custom, :my_event, { :a => 1, :b => "two" }], [:custom, :bare_event, {}]]) + end + + it "warns instead of crashing when custom notification options cannot be marshaled" do + RSpec.configuration.parallel_worker_number = 3 + reporter.publish(:my_event, :callback => lambda {}) + + message = frames.first + expect(message.first).to eq(:message) + expect(message.last).to include("Parallel worker 3 could not forward the custom `my_event` notification", "Proc") + end + + it "forwards non-example failures as a frame with the exception payload and the context description" do + error = RuntimeError.new("suite setup failed") + error.set_backtrace(["./spec/spec_helper.rb:3"]) + + reporter.notify_non_example_exception(error, "An error occurred in a `before(:suite)` hook.") + + kind, payload, description = frames.first + expect(kind).to eq(:non_example_failure) + expect(payload).to include(:class_name => "RuntimeError", :message => "suite setup failed", :backtrace => ["./spec/spec_helper.rb:3"]) + expect(description).to eq("An error occurred in a `before(:suite)` hook.") + expect(RSpec.world.non_example_failure).to be true + end + + it "ignores the notifications that only the coordinator's reporter emits" do + reporter.start(1) + reporter.finish + + expect(frames).to eq([]) + end + + it "does not forward example_passed/failed/pending separately" do + RSpec.describe("G") { it("e") {} }.run(reporter) + + expect(frames.map(&:first)).not_to include(:example_passed) + end + + it "notifies listeners registered inside the worker, such as from a `before(:suite)` hook" do + events = [] + notifications = [:example_group_started, :example_started, :example_finished, :example_failed, :message, :my_event] + listener = Object.new + notifications.each do |event| + listener.define_singleton_method(event) { |_notification| events << event } + end + reporter.register_listener(listener, *notifications) + + RSpec.describe("G") { it("fails") { raise "x" } }.run(reporter) + reporter.message("hi") + reporter.publish(:my_event) + + expect(events).to eq(notifications) + expect(frames.map(&:first)).to eq([:group_started, :example_started, :example_finished, :group_finished, :message, :custom]) + end + + it "forwards metadata that a worker listener adds when notified of the example's outcome" do + listener = Object.new + def listener.example_failed(notification) + notification.example.metadata[:extra_failure_lines] = ["Ref: added in the worker"] + end + reporter.register_listener(listener, :example_failed) + + RSpec.describe("G") { it("fails") { raise "x" } }.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:extra_failure_lines]).to eq(["Ref: added in the worker"]) + end + + it "forwards the metadata an example set, changed in place or deleted, and nothing for one that changed none" do + group = RSpec.describe("G", :kept => "k", :gone => true) do + it("changes", :tags => ["a"], :same => "s") do |ex| + ex.metadata[:added] = { :path => "shot.png" } + ex.metadata[:tags] << "b" + ex.metadata.store(:same, ex.metadata[:same]) + ex.metadata.delete(:gone) + end + it("changes nothing") {} + end + group.run(reporter) + + payloads = frames.select { |f| f.first == :example_finished }.to_h { |f| [f.last[:description], f.last[:metadata]] } + changed, unchanged = payloads.values_at("changes", "changes nothing") + expect(changed[:set].map { |key, dump| [key, load_dump(dump)] }).to contain_exactly( + [":added", [:added, { :path => "shot.png" }]], [":tags", [:tags, %w[a b]]] + ) + expect(changed[:deleted].map { |key, dump| [key, load_dump(dump)] }).to eq([[":gone", :gone]]) + expect(unchanged).to be_nil + end + + it "leaves RSpec's own metadata and what the payload carries anyway out of the comparison" do + group = RSpec.describe("G") do + it("skips") do |ex| + ex.metadata[:extra_failure_lines] = ["more"] + ex.metadata[:last_run_status] = "changed" + skip "later" + end + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to be_nil + expect(payload).to include(:skip => true, :pending_message => "later", :extra_failure_lines => ["more"]) + end + + it "names metadata set to values that cannot be marshaled instead of sending them" do + group = RSpec.describe("G") do + it("sets a proc") { |ex| ex.metadata[:callback] = lambda {} } + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to eq(:unsent => { ":callback" => "Proc cannot be marshaled" }) + end + + it "copes with values whose `hash` raises and with deleted keys that cannot be marshaled" do + unhashable = Object.new + def unhashable.hash + raise "no hash" + end + anonymous_key = Class.new + group = RSpec.describe("G", anonymous_key => true, :odd => unhashable) do + it("deletes") { |ex| ex.metadata.delete(anonymous_key) } + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to be_nil + end + + it "forwards metadata a worker listener sets once the example finished" do + listener = Object.new + def listener.example_passed(notification) + notification.example.metadata[:screenshot] = "after.png" + end + reporter.register_listener(listener, :example_passed) + + RSpec.describe("G") { it("passes") {} }.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata][:set].map { |key, dump| [key, load_dump(dump)] }).to eq([[":screenshot", [:screenshot, "after.png"]]]) + end + + it "sets the world's `wants_to_quit` when the coordinator asks it to quit between examples" do + jobs.last.send([:quit]) + group = RSpec.describe("G") do + it("first") {} + it("second") {} + end + + group.run(reporter) + + expect(RSpec.world.wants_to_quit).to be true + expect(frames.count { |f| f.first == :example_finished }).to eq(1) + end + + it "leaves other control messages for the worker's job loop while polling" do + jobs.last.send([:run, 7]) + RSpec.describe("G") { it("e") {} }.run(reporter) + + expect(RSpec.world.wants_to_quit).to be_falsey + expect(jobs.first.receive).to eq([:run, 7]) + end + + it "forwards the expectations counted since it last did, and nothing when there were none" do + reporter.forward_expectation_count + 3.times { reporter.increment_expectation_count } + reporter.forward_expectation_count + reporter.forward_expectation_count + reporter.increment_expectation_count + reporter.forward_expectation_count + + expect(frames).to eq([[:expectations, 3], [:expectations, 1]]) + end + + it "keeps the reporter's own failure bookkeeping so fail-fast works inside the worker" do + RSpec.configuration.fail_fast = true + group = RSpec.describe("G") do + it("fails") { raise "x" } + it("never runs") {} + end + + group.run(reporter) + + expect(reporter.failed_examples.size).to eq(1) + expect(RSpec.world.wants_to_quit).to be true + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb new file mode 100644 index 000000000..ab161ab7c --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -0,0 +1,632 @@ +require 'fileutils' +require 'tmpdir' +require 'rspec/core/parallel/coordinator' + +# These specs stand in for the process's own streams, which a worker moves. +# rubocop:disable-next Style/GlobalStdStream +module RSpec::Core::Parallel + RSpec.describe Worker do + let(:jobs) { Channel.pipe } + let(:results) { Channel.pipe } + let(:exits) { [] } + let(:stderr) { StringIO.new } + let(:groups) do + [ + RSpec.describe("First") { it("passes") {} }, + RSpec.describe("Second") do + it("fails") { raise "nope" } + end + ] + end + + let(:traps) { {} } + + def worker(number=2, options={}) + Worker.new(number, groups.map { |group| Unit.new(group) }, jobs.first, results.last, + { :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |signal, command=nil, &handler| traps[signal] = command || handler }, + :watch_coordinator => false }.merge(options)) + end + + def frames + results.first.drain.map(&:first) + end + + after do + jobs.each(&:close) + results.each(&:close) + end + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + # A worker told to stop arms its exit watchdog, a thread that sleeps + # through `Parallel.sleep_running` for up to the shutdown timeout. Workers + # here are threads of this process, so it would outlive its example and + # call a later example's stubs of `Parallel.sleep` too. + around do |example| + threads = Thread.list + example.run + (Thread.list - threads).each { |thread| thread.kill.join(1) } + end + + describe "#test_env_number" do + it "follows the parallel_tests convention: blank for the first worker, then the worker number" do + expect(worker(1).test_env_number).to eq("") + expect(worker(2).test_env_number).to eq("2") + expect(worker(10).test_env_number).to eq("10") + end + end + + describe "#watch_coordinator" do + it "exits at once, saying why, when its parent (the coordinating process) changes" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + parents = [100, 100, 100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include("worker 2 (pid #{Process.pid}) is exiting because the coordinating process (pid 100) is gone") + end + + it "removes the directory of its output files, which the coordinator would have removed" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + dir = Dir.mktmpdir + parents = [100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, :output => WorkerOutput.new(dir, 2).paths, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(File.exist?(dir)).to be false + end + + it "exits when the coordinator's pid no longer names a process, its parent unchanged (Windows)" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + answers = [true, true, false] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, :coordinator => 100, :ppid => lambda { 100 }, + :alive => lambda { |pid| pid == 100 && answers.shift }, + :exit_now => lambda { |status| exits << [:exit_now, status] }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include("the coordinating process (pid 100) is gone") + end + + it "tells a live process from one that is gone" do + expect(worker.process_alive?(Process.pid)).to be true + gone = Process.spawn(RbConfig.ruby, '-e', 'exit') + Process.wait(gone) + # Windows can still find a pid for a moment after it was reaped. + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 5 + sleep 0.05 while worker.process_alive?(gone) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline + expect(worker.process_alive?(gone)).to be false + end + + it "still exits when it cannot say why (its stderr is gone too)" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + parents = [100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => StringIO.new.tap(&:close), + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + end + + it "is started by #run unless disabled" do + instance = Worker.new(2, [], jobs.first, results.last, + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :ppid => lambda { 100 }) + allow(instance).to receive(:watch_coordinator).and_return(nil) + jobs.last.send([:stop]) + + instance.run + + expect(instance).to have_received(:watch_coordinator) + expect(exits).to eq([[:exit_now, 0]]) + end + end + + describe "#prepare" do + it "identifies the worker, disables autorun and installs the forwarding reporter" do + worker(3).prepare + + expect(ENV['TEST_ENV_NUMBER']).to eq("3") + expect(RSpec.configuration.parallel_worker_number).to eq(3) + expect(RSpec.configuration.reporter).to be_a(WorkerReporter) + expect(RSpec::Core::Runner).to be_autorun_disabled + end + + it "handles interrupts like the runner does, but silently: stop between examples, force-quit the second time" do + worker.prepare + + expect { traps.fetch('INT').call }.not_to output.to_stderr + expect(RSpec.world.wants_to_quit).to be true + expect(exits).to eq([]) + + traps.fetch('INT').call + expect(exits).to eq([[:exit_now, 1]]) + end + + it "reinstates the handlers the coordinating process replaced with its own, and only those" do + worker(2, :signal_handlers => { "TERM" => "SYSTEM_DEFAULT", "HUP" => "DEFAULT" }).prepare + + expect(traps).to include("TERM" => "SYSTEM_DEFAULT", "HUP" => "DEFAULT") + + traps.clear + worker.prepare + + expect(traps.keys).to eq(["INT"]) + end + + it "drops the formatters inherited from the coordinating process" do + RSpec.configuration.formatter = "documentation" + expect(RSpec.configuration.formatters).not_to be_empty + + worker.prepare + + expect(RSpec.configuration.formatters).to be_empty + end + end + + describe "#run" do + it "runs the dispatched groups inside the suite hooks, reports each unit and exits normally when told to stop" do + order = [] + RSpec.configuration.before(:suite) { order << :before_suite } + RSpec.configuration.after(:suite) { order << :after_suite } + groups.first.before(:context) { order << :group } + + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:run, 1, groups[1].id]) + jobs.last.send([:stop]) + + worker.run + + expect(order).to eq([:before_suite, :group, :after_suite]) + expect(exits).to eq([[:exit, 0]]) + + all = frames + expect(all.select { |f| f.first == :unit_finished }).to eq([ + [:unit_finished, 0, true, false, false], + [:unit_finished, 1, false, false, false] + ]) + expect(all.last).to eq([:exiting, nil, nil]) + expect(all.map(&:first)).to include(:group_started, :example_started, :example_finished, :group_finished) + end + + it "forwards the expectations counted in each unit before reporting it, and those of `after(:suite)` hooks last" do + count = lambda { RSpec.configuration.reporter.increment_expectation_count } + groups.first.before(:context) { count.call } + groups.first.example("counts twice") { 2.times { count.call } } + RSpec.configuration.after(:suite) { count.call } + + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:run, 1, groups[1].id]) + jobs.last.send([:stop]) + + worker.run + + expect(frames.select { |f| [:expectations, :unit_finished, :exiting].include?(f.first) }).to eq([ + [:expectations, 3], + [:unit_finished, 0, true, false, false], + [:unit_finished, 1, false, false, false], + [:exiting, nil, nil], + [:expectations, 1] + ]) + end + + context "with output files" do + let(:dir) { Dir.mktmpdir } + let(:stdout_stand_in) { File.open(File.join(dir, 'stdout'), 'w') } + let(:stderr_stand_in) { File.open(File.join(dir, 'stderr'), 'w') } + + before do + stub_const('STDOUT', stdout_stand_in) + stub_const('STDERR', stderr_stand_in) + end + + after do + [stdout_stand_in, stderr_stand_in].each(&:close) + # Windows does not remove a file the worker still has open (its copy of stderr). + FileUtils.remove_entry(dir, RSpec::Support::OS.windows?) + end + + it "moves its standard output and error into them and says how far they reach before the next frame" do + output = WorkerOutput.new(dir, 2) + groups.first.example("prints") { STDOUT.write("printed") } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker(2, :output => output.paths).run + + announced = frames.select { |f| f.first == :output } + expect(announced).to eq([[:output, { :out => 7, :err => 0 }]]) + expect(File.read(output.paths[:out])).to eq("printed") + output.close + end + + it "writes where the run does when it cannot open them" do + missing = { :out => File.join(dir, 'gone', 'out'), :err => File.join(dir, 'gone', 'err') } + jobs.last.send([:stop]) + + worker(2, :output => missing).run + + expect(frames).to eq([[:started]]) + end + end + + it "exits at once, without running suite hooks or at_exit handlers, when its first job is not a group to run" do + hooks = [] + RSpec.configuration.before(:suite) { hooks << :before_suite } + RSpec.configuration.after(:suite) { hooks << :after_suite } + + [[:stop], [:quit]].each do |job| + exits.clear + jobs.last.send(job) + worker.run + expect(exits).to eq([[:exit_now, 0]]) + end + + expect(hooks).to eq([]) + expect(frames).to eq([[:started], [:started]]) + end + + it "exits at once when the coordinator goes away before giving it anything" do + jobs.last.close + + worker.run + + expect(exits).to eq([[:exit_now, 0]]) + end + + it "reports a quitting world and non-example failures with the unit" do + RSpec.configuration.before(:suite) { raise "suite setup failed" } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker.run + + unit = frames.find { |f| f.first == :unit_finished } + expect(unit).to eq([:unit_finished, 0, false, true, true]) + end + + it "exits immediately without reporting when the results pipe is gone" do + results.first.close + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker.run + + expect(exits).to eq([[:exit_now, 1]]) + end + + it "arms the watchdog before the after(:suite) hooks run" do + armed_during_after_suite = nil + instance = worker + RSpec.configuration.after(:suite) { armed_during_after_suite = instance.instance_variable_get(:@watchdog_armed) } + allow(instance).to receive(:arm_watchdog) { instance.instance_variable_set(:@watchdog_armed, true) } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + instance.run + + expect(armed_during_after_suite).to be true + end + end + + describe "leaving its job loop" do + # JRuby's `Example#run` rescues `SystemExit` (see example.rb), so there, + # serially too, `exit` in an example fails it rather than ending the run. + jruby_rescues_exit = RSpec::Support::Ruby.jruby? && "JRuby's Example#run rescues SystemExit" + + def leaving_frames + frames.select { |frame| frame.first == :exiting } + end + + it "arms the watchdog and says so before any after(:suite) hook, even one a before(:suite) hook added before calling exit" do + instance = worker + armed = [] + allow(instance).to receive(:arm_watchdog) { armed << :watchdog } + config = RSpec.configuration + config.before(:suite) do + config.after(:suite) { armed << :late_after_suite } + exit 4 + end + jobs.last.send([:run, 0, groups[0].id]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(4) } + + expect(armed).to eq([:watchdog, :late_after_suite]) + expect(leaving_frames).to eq([[:exiting, "the code it ran called `exit` or `abort` (status 4)", 4]]) + end + + it "says which status the code it ran called `exit` with in an example, and runs nothing after it", :skip => jruby_rescues_exit do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + ran = [] + exiting = RSpec.describe("Exits") do + it("exits") { exit 3 } + it("never runs") { ran << :after } + end + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(3) } + + expect(ran).to eq([]) + expect(leaving_frames).to eq([[:exiting, "the code it ran called `exit` or `abort` (status 3)", 3]]) + expect(instance).to have_received(:arm_watchdog).once + end + + it "says so again, with the status, when an after(:suite) hook calls exit" do + instance = worker + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { exit 5 } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(5) } + + expect(leaving_frames).to eq([[:exiting, nil, nil], [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5]]) + end + + it "says so again when an after(:suite) hook calls exit with another status than the example did", :skip => jruby_rescues_exit do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + exiting = RSpec.describe("Exits") { it("exits") { exit 4 } } + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { exit 5 } + jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(5) } + + expect(leaving_frames).to eq([ + [:exiting, "the code it ran called `exit` or `abort` (status 4)", 4], + [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5] + ]) + end + + it "forwards the expectations of a unit `exit` cut short before saying it is leaving, and those of `after(:suite)` hooks after", :skip => jruby_rescues_exit do + exiting = RSpec.describe("Exits") do + it("exits") do + 2.times { RSpec.configuration.reporter.increment_expectation_count } + exit 3 + end + end + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { RSpec.configuration.reporter.increment_expectation_count } + jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) + + expect(frames.select { |f| [:expectations, :exiting].include?(f.first) }).to eq([ + [:expectations, 2], + [:exiting, "the code it ran called `exit` or `abort` (status 3)", 3], + [:expectations, 1] + ]) + end + + it "still exits normally when nothing reads the expectations its `after(:suite)` hooks counted" do + reader = results.first + RSpec.configuration.after(:suite) do + reader.close + RSpec.configuration.reporter.increment_expectation_count + end + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker.run + + expect(exits).to eq([[:exit, 0]]) + end + + it "names a signal or another exception that ends the process" do + expect(worker.exit_cause(SignalException.new("TERM"), "the code it ran")).to eq("it received SIGTERM") + expect(worker.exit_cause(NoMemoryError.new("too much"), "the code it ran")).to eq( + "the code it ran raised NoMemoryError (too much)" + ) + end + + it "goes on without a watchdog when no thread can be started for it" do + instance = worker + allow(Thread).to receive(:new).and_raise(ThreadError, "can't create Thread: Resource temporarily unavailable") + + expect(instance.arm_watchdog).to be_nil + end + + it "names why it is stopping in its thread dump" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + instance = worker + instance.instance_variable_set(:@exit_cause, "the code it ran called `exit` or `abort` (status 0)") + + instance.dump_threads(0.05) + + expect(stderr.string).to include( + "did not exit within 0.05 seconds of stopping because the code it ran called `exit` or `abort` (status 0); aborting it." + ) + end + end + + describe "given a unit of some examples of a group" do + let(:hooks) { [] } + let(:group) do + log = hooks + RSpec.describe("Outer") do + before(:context) { log << :outer } + it("o1") { log << :o1 } + context("A") do + before(:context) { log << :a } + it("a1") { log << :a1 } + it("a2") { log << :a2 } + end + context("B") do + before(:context) { log << :b } + it("b1") { log << :b1 } + it("b2") do |example| + log << :b2 + raise "b2 fails" if example.metadata[:fail] + end + end + end + end + + def ids(*descriptions) + group.descendants.flat_map(&:examples).select { |example| descriptions.include?(example.description) }.map(&:id) + end + + def unit_worker + Worker.new(2, [Unit.new(group)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, :watch_coordinator => false) + end + + it "runs only those examples and the context hooks of the groups containing them, as `rspec path[ids]` would, and then the whole group as before" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + jobs.last.send([:run, 0, group.id, ids("a2")]) + jobs.last.send([:run, 1, group.id, ids("o1", "b1")]) + jobs.last.send([:run, 2, group.id]) + jobs.last.send([:stop]) + + unit_worker.run + + expect(hooks).to eq([:outer, :a, :a2, :outer, :o1, :b, :b1, :outer, :o1, :a, :a1, :a2, :b, :b1, :b2]) + started = frames.select { |frame| frame.first == :group_started }.map { |frame| frame[1] } + expect(started.first(2)).to eq([group.id, group.children[0].id]) + expect(group.descendant_filtered_examples.size).to eq(5) + expect(group.children.map { |child| child.descendant_filtered_examples.size }).to eq([2, 2]) + end + + it "restores what the group runs when a selected example fails" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + group.children[1].examples.last.metadata[:fail] = true + jobs.last.send([:run, 0, group.id, ids("b2")]) + jobs.last.send([:stop]) + + unit_worker.run + + expect(frames.find { |frame| frame.first == :unit_finished }).to eq([:unit_finished, 0, false, false, false]) + expect(hooks).to eq([:outer, :b, :b2]) + expect(group.descendant_filtered_examples.map(&:description)).to eq(%w[o1 a1 a2 b1 b2]) + expect(RSpec.world.filtered_examples[group.children[1]].size).to eq(2) + end + end + + describe "forked before the spec files were loaded" do + def loading_worker + Worker.new(2, nil, jobs.first, results.last, + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :watch_coordinator => false) + end + + def write_spec_file(dir, content) + File.join(dir, "loaded_spec.rb").tap do |path| + File.write(path, content) + RSpec.configuration.files_or_directories_to_run = [path] + end + end + + # The worker blocks on the jobs channel once it has loaded the files. + def run_in_thread(instance) + Thread.new { instance.run }.tap { |thread| thread.report_on_exception = false } + end + + it "loads them itself, reports what it loaded, then runs the groups it is given by id" do + Dir.mktmpdir do |dir| + write_spec_file(dir, "RSpec.describe('Loaded here') { it('passes') { } }") + thread = run_in_thread(loading_worker) + + expect(results.first.receive).to eq([:started]) + loaded = results.first.receive + group = RSpec.world.example_groups.first + expect(loaded).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world), nil]) + expect(group.description).to eq("Loaded here") + + jobs.last.send([:run, 0, group.id]) + jobs.last.send([:stop]) + expect(thread.join(10)).to be_truthy + + expect(exits).to eq([[:exit, 0]]) + expect(frames).to include([:unit_finished, 0, true, false, false]) + end + end + + it "reports the first load error with what it loaded, and nothing else loading reported" do + Dir.mktmpdir do |dir| + write_spec_file(dir, "RSpec.deprecate('old_thing')\nraise 'boom at load'\n") + thread = run_in_thread(loading_worker) + + # Expectations made while the worker runs would count as its own + # with an rspec-expectations that counts them (rspec/rspec#305). + started = results.first.receive + kind, digest, failure = results.first.receive + jobs.last.send([:stop]) + expect(thread.join(10)).to be_truthy + + expect(started).to eq([:started]) + expect([kind, digest]).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world)]) + expect(failure).to match(/\AAn error occurred while loading .*loaded_spec\.rb\. RuntimeError: boom at load\z/) + expect(frames).to eq([]) + expect(RSpec.world.non_example_failure).to be_falsey + expect(exits).to eq([[:exit_now, 0]]) + end + end + end + + describe "the watchdog" do + it "dumps every thread's backtrace to stderr and aborts the process once the timeout passes" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + instance = worker(4) + blocked_line = __LINE__ + 1 + blocked = Thread.new { sleep } + deadline = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 until blocked.status == "sleep" || RSpec::Core::Parallel.monotonic_time > deadline + + instance.arm_watchdog.join + + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include( + "RSpec parallel worker 4 (pid #{Process.pid}) did not exit within 0.05 seconds", + "(this watchdog)" + ) + expect(stderr.string).to match(/worker_spec\.rb:#{blocked_line}:in /) + expect(stderr.string.scan(/^ #= 3 + ensure + blocked&.kill + end + + it "still aborts the process when an example closed the stream it would dump the threads to" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + stderr.close + + worker(4).arm_watchdog.join + + expect(exits).to eq([[:exit_now, 1]]) + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb new file mode 100644 index 000000000..76cfab288 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -0,0 +1,420 @@ +require 'rspec/core/parallel/coordinator' +require 'rspec/support/spec/shell_out' + +module RSpec::Core + RSpec.describe Parallel do + let(:config) { RSpec.configuration } + let(:runner) { Runner.new(ConfigurationOptions.new([]), config, RSpec.world) } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + def groups(count) + Array.new(count) { |i| RSpec.describe("G#{i}") { it("e") {} } } + end + + describe ".coordinator_for" do + it "returns nil when parallel execution was not requested" do + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "returns a coordinator for the requested number of workers", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + expect(Parallel.coordinator_for(runner, groups(3))).to be_a(Parallel::Coordinator) + end + + it "uses one worker per processor for :auto", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = :auto + allow(Parallel).to receive(:default_worker_count).and_return(2) + coordinator = instance_double(Parallel::Coordinator, :assign => nil) + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(coordinator) + + expect(Parallel.coordinator_for(runner, groups(5))).to eq(coordinator) + end + + it "never starts more workers than there are groups with examples to run", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 8 + empty = RSpec.describe("empty") + coordinator = instance_double(Parallel::Coordinator) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(coordinator) + expect(coordinator).to receive(:assign).with(anything, satisfy { |units| units.size == 3 }) + + Parallel.coordinator_for(runner, groups(3) + [empty]) + end + + it "runs serially when fewer than two groups would be dispatched", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 4 + expect(Parallel.coordinator_for(runner, groups(1))).to be_nil + expect(Parallel.coordinator_for(runner, [])).to be_nil + end + + it "runs serially for one worker or a non-positive count forced from the command line", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.force(:parallel_workers => 1) + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + + config.force(:parallel_workers => 0) + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "runs serially, without a warning, under --dry-run" do + config.parallel_workers = 2 + config.dry_run = true + expect(RSpec).not_to receive(:warning) + + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "warns and runs serially where fork is not available and the run was not started by the `rspec` command" do + config.parallel_workers = 2 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(nil) + expect(RSpec).to receive(:warning).with(a_string_including("without `fork`", "started by the `rspec` command", + "Running serially"), :call_site => nil) + + expect(Parallel.requested_worker_count(config)).to be_nil + end + + it "has the workers started as new `rspec` processes where fork is not available" do + config.parallel_workers = 3 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(["exe/rspec", []]) + expect(RSpec).not_to receive(:warning) + + expect(Parallel.requested_worker_count(config)).to eq(3) + expect(Parallel).to be_spawning + end + + it "warns and runs serially inside another tool's parallel worker", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + with_env_vars("TEST_ENV_NUMBER" => "") do + expect(RSpec).to receive(:warning).with(a_string_including("already a worker of a parallel test run"), :call_site => nil) + + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + end + end + + describe ".suite_digest" do + it "fingerprints the ids and locations of the examples to run, in load order, but not their descriptions" do + RSpec.describe("A") do + it("one") {} + it("two") {} + end + RSpec.describe("B") { it("three") {} } + first = Parallel.suite_digest(RSpec.world) + + expect(Parallel.suite_digest(RSpec.world)).to eq(first) + RSpec.world.example_groups.last.examples.first.metadata[:full_description] = "B at #{Time.now}" + expect(Parallel.suite_digest(RSpec.world)).to eq(first) + RSpec.world.example_groups.last.examples.first.metadata[:location] = "./elsewhere_spec.rb:1" + expect(Parallel.suite_digest(RSpec.world)).not_to eq(first) + end + + it "changes with the examples defined" do + RSpec.describe("A") { it("one") {} } + first = Parallel.suite_digest(RSpec.world) + RSpec.describe("B") { it("one more") {} } + + expect(Parallel.suite_digest(RSpec.world)).not_to eq(first) + end + + it "is kept per spec file" do + RSpec.describe("A") { it("one") {} } + RSpec.describe("B") { it("two") {} } + + expect(Parallel.suite_digest(RSpec.world).keys).to eq([RSpec.world.example_groups.first.metadata[:file_path]]) + end + + it "leaves out the examples that are filtered out" do + RSpec.describe("A") { it("one") {} } + unfiltered = Parallel.suite_digest(RSpec.world) + RSpec.describe("B") { it("skipped by the filter", :slow) {} } + RSpec.world.filter_manager.exclude(:slow => true) + + expect(Parallel.suite_digest(RSpec.world)).to eq(unfiltered) + end + end + + describe ".default_worker_count" do + it "is the number of processors" do + require 'etc' + expect(Parallel.default_worker_count).to eq(Etc.nprocessors) + end + end + + describe ".units_for" do + it "skips groups without examples to run and keeps the given order without timing data" do + all = groups(3) + empty = RSpec.describe("empty") + expect(Parallel.units_for([all[1], empty, all[0], all[2]], config).map(&:group)).to eq([all[1], all[0], all[2]]) + end + + describe "of a group tagged `parallel_split: N`" do + def tagged(parts, count=6, &block) + RSpec.describe("Tagged", :parallel_split => parts) do + count.times { |i| it("e#{i}") {} } + instance_exec(&block) if block + end + end + + it "runs it as N parts, each a consecutive slice of the order its examples run in" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(3) { context("nested") { it("n0") {} } } + ids = group.descendant_filtered_examples.map(&:id) + + units = Parallel.units_for([group, groups(1).first], config) + + expect(units.first(3).map(&:group)).to all(eq(group)) + expect(units.first(3).map(&:example_ids)).to eq([ids[0, 2], ids[2, 3], ids[5, 2]]) + expect(units.last.example_ids).to be_nil + end + + it "slices the order a random ordering runs the examples in, which orders any part of them the same way" do + config.seed = 1234 + group = tagged(2) + order = Parallel.execution_order(group) + + parts = Parallel.units_for([group], config).map(&:example_ids) + + expect(parts.flatten).to eq(order) + expect(order).not_to eq(group.examples.map(&:id)) + expect(config.ordering_registry.fetch(:global).order(group.examples.select { |e| parts.first.include?(e.id) }).map(&:id)).to eq(parts.first) + end + + it "makes no more parts than there are examples" do + expect(Parallel.units_for([tagged(8, 3)], config).map(&:example_ids).map(&:size)).to eq([1, 1, 1]) + end + + it "leaves a group with a single example whole" do + expect(Parallel.units_for([tagged(4, 1)], config).map(&:example_ids)).to eq([nil]) + end + + [true, 1, 0, "3", 2.5].each do |value| + it "warns and leaves the group whole for `parallel_split: #{value.inspect}`" do + expect(RSpec).to receive(:warning).with(/Ignoring `parallel_split: #{Regexp.escape(value.inspect)}` on Tagged \(.+\): it is the number of parts/, :call_site => nil) + expect(Parallel.units_for([tagged(value)], config).map(&:example_ids)).to eq([nil]) + end + end + + it "warns and leaves the group whole when it, or a group in it, uses a custom ordering" do + config.register_ordering(:reversed) { |items| items.reverse } + group = tagged(2) { context("nested", :order => :reversed) { it("n0") {} } } + expect(RSpec).to receive(:warning).with(/Not splitting Tagged \(.+\) into parts: it orders examples with a custom ordering/, :call_site => nil) + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([nil]) + end + + context "with run times recorded", :isolated_directory do + before { config.example_status_persistence_file_path = "statuses.txt" } + + def record(times) + rows = times.map { |id, time| "#{id} | passed | #{time} |\n" }.join + File.write("statuses.txt", "example_id | status | run_time |\n---------- | ------ | -------- |\n#{rows}") + end + + it "balances the parts by them, counting an example without one as the mean of the others" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(2, 4) + ids = group.examples.map(&:id) + record(ids[0] => "3 seconds", ids[1] => "1 second", ids[2] => "1 second") + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([[ids[0]], ids[1, 3]]) + end + + it "balances them by the number of examples when every recorded time is zero" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(2, 4) + ids = group.examples.map(&:id) + record(ids.to_h { |id| [id, "0 seconds"] }) + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([ids[0, 2], ids[2, 2]]) + end + end + end + + context "with timings recorded in the example status persistence file", :isolated_directory do + let(:all) { groups(4) } + + before do + config.example_status_persistence_file_path = "statuses.txt" + ExampleStatusPersister.persist([], "statuses.txt") + File.write("statuses.txt", <<-STATUSES.gsub(/^ +\|/, '')) + |example_id | status | run_time | + |---------- | ------ | ---------------- | + |#{all[0].examples.first.id} | passed | 0.5 seconds | + |#{all[1].examples.first.id} | passed | 1 minute 2 seconds | + |#{all[3].examples.first.id} | failed | 3 seconds | + STATUSES + end + + it "puts groups with no recorded timing first, then the others longest first" do + expect(Parallel.units_for(all, config).map(&:group)).to eq([all[2], all[1], all[3], all[0]]) + end + + it "keeps the given order for groups with equal timings" do + File.write("statuses.txt", <<-STATUSES.gsub(/^ +\|/, '')) + |example_id | status | run_time | + |---------- | ------ | --------- | + |#{all[0].examples.first.id} | passed | 1 second | + |#{all[1].examples.first.id} | passed | 1 second | + |#{all[2].examples.first.id} | passed | 1 second | + |#{all[3].examples.first.id} | passed | 1 second | + STATUSES + + expect(Parallel.units_for(all.reverse, config).map(&:group)).to eq(all.reverse) + end + + it "ignores an unreadable status file" do + allow(ExampleStatusPersister).to receive(:load_from).and_raise(Errno::EACCES) + expect(Parallel.units_for(all, config).map(&:group)).to eq(all) + end + end + end + + describe ".parse_duration" do + it "reads the durations written by the example status persister" do + expect(Parallel.parse_duration("0.00012 seconds")).to eq(0.00012) + expect(Parallel.parse_duration("1 second")).to eq(1.0) + expect(Parallel.parse_duration("2 minutes 3.5 seconds")).to eq(123.5) + expect(Parallel.parse_duration("")).to be_nil + expect(Parallel.parse_duration(nil)).to be_nil + end + end + + describe ".marshal_safe" do + it "returns values Marshal can dump and the block's value otherwise" do + expect(Parallel.marshal_safe([1, "two"]) { :fallback }).to eq([1, "two"]) + expect(Parallel.marshal_safe(lambda {}) { |value| value.class }).to eq(Proc) + end + end + + describe ".monotonic_time" do + it "never goes backwards" do + first = Parallel.monotonic_time + expect(Parallel.monotonic_time).to be >= first + end + end + + describe ".sleep_running" do + it "sleeps until the process has run that long, counting a slice it was paused through as the slice" do + clock = 100.0 + slices = [] + allow(Parallel).to receive(:monotonic_time) { clock } + allow(Parallel).to receive(:sleep) do |seconds| + slices << seconds + clock += slices.size == 2 ? 30.0 : seconds + end + + Parallel.sleep_running(1.0, 0.25) + + expect(slices).to eq([0.25, 0.25, 0.25, 0.25]) + expect(clock).to eq(100.0 + 0.75 + 30.0) + end + + it "counts a slice that merely ran late, within the pause tolerance, as the time it took" do + clock = 0.0 + slices = [] + allow(Parallel).to receive(:monotonic_time) { clock } + allow(Parallel).to receive(:sleep) do |seconds| + slices << seconds + clock += seconds + 0.5 + end + + Parallel.sleep_running(1.0, 0.25) + + expect(slices).to eq([0.25, 0.25]) + end + end + + describe ".time_to_wire and .time_from_wire" do + it "carry a local Time as integers and rebuild one equal to it, to the nanosecond" do + time = ::Time.at(1_791_231_154, 100_773_123, :nsec) + + wire = Parallel.time_to_wire(time) + rebuilt = Parallel.time_from_wire(Marshal.load(Marshal.dump(wire))) + + expect(wire).to eq([1_791_231_154, 100_773_123]) + expect(rebuilt).to eq(time) + expect([rebuilt.nsec, rebuilt.utc_offset]).to eq([time.nsec, time.utc_offset]) + end + + it "leave nil, and values that are not on the wire, as they are" do + time = ::Time.now + + expect(Parallel.time_to_wire(nil)).to be_nil + expect(Parallel.time_from_wire(nil)).to be_nil + expect(Parallel.time_from_wire(time)).to be(time) + end + end + + describe "signal handling", :skip => (Signal.list.key?("HUP") ? false : "needs SIGHUP") do + around do |example| + original = trap("HUP", "DEFAULT") + begin + example.run + ensure + trap("HUP", original || "DEFAULT") + end + end + + it "takes over a signal that would end the process and returns the handler it replaced" do + received = [] + trap("HUP", "SYSTEM_DEFAULT") + + replaced = Parallel.trap_signals(%w[HUP NOSUCHSIGNAL]) { |signal| received << signal } + trap("HUP", "DEFAULT").call + + expect(replaced).to eq("HUP" => "SYSTEM_DEFAULT") + expect(received).to eq(["HUP"]) + end + + it "leaves an ignored signal ignored, as under nohup, and a handler the suite installed in place" do + suite_handler = proc {} + trap("HUP", "IGNORE") + + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", nil)).to eq("IGNORE") + + # How this Ruby reports a signal ignored with `nil` (MRI: `nil`, JRuby: "IGNORE"). + ignored_with_nil = trap("HUP", nil) + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", suite_handler)).to eq(ignored_with_nil) + + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", "DEFAULT")).to equal(suite_handler) + end + + it "reinstates the handlers it replaced" do + Parallel.restore_signals("HUP" => "SYSTEM_DEFAULT") + expect(trap("HUP", "IGNORE")).to eq("SYSTEM_DEFAULT") + + Parallel.restore_signals("HUP" => "DEFAULT") + expect(trap("HUP", "IGNORE")).to eq("DEFAULT") + end + end + + describe "loading" do + include RSpec::Support::ShellOut + + def run_in_new_process(args) + code = <<-RUBY + require "rspec/core" + require "stringio" + 2.times { |i| RSpec.describe("G\#{i}") { it("passes") {} } } + status = RSpec::Core::Runner.run(["--options", File::NULL, *#{args.inspect}], $stderr, StringIO.new) + print [status, defined?(RSpec::Core::Parallel::Coordinator), defined?(RSpec::Core::Parallel)].inspect + RUBY + without_env_vars("SPEC_OPTS") { run_ruby_with_current_load_path(code) } + end + + it "leaves the parallel runner unloaded in a serial run", :slow do + out, err, _status = run_in_new_process([]) + expect([out, err]).to eq(["[0, nil, nil]", ""]) + end + + it "loads the runner for a run that forks workers", :slow, :skip => !RSpec::Support::RubyFeatures.fork_supported? do + out, err, _status = run_in_new_process(%w[--parallel=2]) + expect([out, err]).to eq(['[0, "constant", "constant"]', ""]) + end + end + end +end diff --git a/rspec-core/spec/rspec/core/profiler_spec.rb b/rspec-core/spec/rspec/core/profiler_spec.rb index 7bd370a5c..90a1b0a7e 100644 --- a/rspec-core/spec/rspec/core/profiler_spec.rb +++ b/rspec-core/spec/rspec/core/profiler_spec.rb @@ -39,6 +39,32 @@ end end + describe '#record_group_duration' do + it 'records a duration measured elsewhere, in place of the one observed here' do + profiler.example_group_started group_notification group + allow(::RSpec::Core::Time).to receive(:now) { now + 1 } + profiler.example_group_finished group_notification group + + profiler.record_group_duration(group, 7.5) + + expect(profiler.example_groups[group]).to include(:total_time => 7.5, :description => "My Group") + end + + it 'records groups it has not seen start' do + profiler.record_group_duration(group, 2.0) + expect(profiler.example_groups[group]).to include(:total_time => 2.0, :count => 0) + end + + context "when the group is not a top-level group" do + let(:group) { super().describe "nested" } + + it 'no-ops since we only consider top-level groups for profiling' do + profiler.record_group_duration(group, 2.0) + expect(profiler.example_groups).to be_empty + end + end + end + describe '#example_group_finished' do before do profiler.example_group_started group_notification group diff --git a/rspec-core/spec/rspec/core/rake_task_spec.rb b/rspec-core/spec/rspec/core/rake_task_spec.rb index da8b8e1f9..de26b6886 100644 --- a/rspec-core/spec/rspec/core/rake_task_spec.rb +++ b/rspec-core/spec/rspec/core/rake_task_spec.rb @@ -161,7 +161,8 @@ def silence_output(&block) it "removes the environment variables", :skip => RSpec::Support::Ruby.jruby? do with_env_vars 'MY_ENV' => 'ABC' do if RSpec::Support::OS.windows? - essential_shell_variables = /\["ANSICON", "ANSICON_DEF", "HOME", "TMPDIR", "USER"\]/ + # ANSICON's variables are there only when it is active in the shell. + essential_shell_variables = /\[(?:"ANSICON", "ANSICON_DEF", )?"HOME", "TMPDIR", "USER"\]/ else essential_shell_variables = /\["PWD"(?:, "SHLVL")?(?:, "_")?(?:, "__CF_USER_TEXT_ENCODING")?\]/ end diff --git a/rspec-core/spec/rspec/core/reporter_spec.rb b/rspec-core/spec/rspec/core/reporter_spec.rb index 6c0331156..ba1d15271 100644 --- a/rspec-core/spec/rspec/core/reporter_spec.rb +++ b/rspec-core/spec/rspec/core/reporter_spec.rb @@ -304,6 +304,24 @@ module RSpec::Core end end + describe "#record_example_group_duration" do + it "hands the duration to the profiler, setting it up if needed" do + slow, fast = RSpec.describe("slow"), RSpec.describe("fast") + config.profile_examples = 2 + formatter = double("formatter") + reporter.register_listener(formatter, :dump_profile) + + reporter.record_example_group_duration(slow, 4.0) + reporter.record_example_group_duration(fast, 1.0) + + expect(formatter).to receive(:dump_profile) do |notification| + expect(notification.slowest_groups.map(&:last)).to include(a_hash_including(:total_time => 4.0), a_hash_including(:total_time => 1.0)) + end + reporter.start(0) + reporter.finish + end + end + describe "#notify_non_example_exception" do it "sends a `message` notification that contains the formatted exception details" do formatter_out = StringIO.new diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index 7bcabc485..90c82126b 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -1,4 +1,5 @@ require 'rspec/core/drb' +require 'rspec/core/parallel/coordinator' require 'support/runner_support' module RSpec::Core @@ -296,6 +297,37 @@ def interrupt expect(runner).to receive(:exit).with(123) runner.invoke end + + context "remembering the script and arguments it runs, which a parallel run without `fork` starts its workers with" do + before do + allow(runner).to receive(:run) { 0 } + stub_const("ARGV", %w[--tag fast]) + allow(File).to receive(:file?).and_call_original + end + + after { runner.instance_variable_set(:@command_line, nil) } + + it "takes the script that started the process" do + allow(File).to receive(:file?).with($0).and_return(true) + runner.invoke + expect(runner.command_line).to eq([File.expand_path($0), %w[--tag fast]]) + end + + it "takes rspec-core's `rspec` when no script started it (`ruby -e`)" do + allow(File).to receive(:file?).with($0).and_return(false) + runner.invoke + expect(runner.command_line).to eq([RSpec::Core.path_to_executable, %w[--tag fast]]) + end + end + + it "runs as the parallel worker a coordinator without `fork` started it as" do + RSpec::Support.require_rspec_core "parallel/coordinator" + stub_const("ARGV", %w[--tag fast]) + expect(RSpec::Core::Parallel::SpawnedWorker).to receive(:run).with(%w[--tag fast]) { throw :exited } + expect(runner).not_to receive(:run) + + with_env_vars('RSPEC_PARALLEL_WORKER' => '2') { catch(:exited) { runner.invoke } } + end end describe ".run" do @@ -363,6 +395,76 @@ def interrupt end describe "#run" do + it 'hands the ordered example groups to a parallel coordinator when one applies, then persists example statuses' do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) + config.parallel_workers = 2 + coordinator = instance_double(Parallel::Coordinator, :run => 17) + expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups).and_return(coordinator) + config.example_status_persistence_file_path = "statuses.txt" + expect(ExampleStatusPersister).to receive(:persist).with(world.all_examples, "statuses.txt") + + runner = Runner.new(ConfigurationOptions.new([]), config, world) + expect(runner).not_to receive(:run_specs) + + expect(runner.run(err, out)).to eq(17) + end + + it 'plans no parallel run unless one is configured' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) + expect(Parallel::Plan).not_to receive(:new) + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + + it 'plans a parallel run that the spec files configure, once they have loaded' do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) { config.parallel_workers = 2 } + expect(Parallel).to receive(:coordinator_for).and_return(nil) + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + + context 'with a parallel plan' do + let(:plan) { instance_double(Parallel::Plan, :before_loading => nil, :dismiss => nil) } + + before do + config.parallel_workers = 2 + allow(Parallel::Plan).to receive(:new).and_return(plan) + end + + it 'lets it fork workers before the spec files load, then hands it the groups, and dismisses what it did not use' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + coordinator = instance_double(Parallel::Coordinator, :run => 0) + expect(plan).to receive(:before_loading).ordered + expect(config).to receive(:load_spec_files).ordered + expect(plan).to receive(:coordinator_for).with(world.ordered_example_groups).ordered.and_return(coordinator) + expect(plan).to receive(:dismiss).ordered + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + + it 'dismisses the workers it forked when the run ends early' do + allow(config).to receive(:load_spec_files) { world.wants_to_quit = true } + expect(plan).not_to receive(:coordinator_for) + + Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) + + expect(plan).to have_received(:dismiss) + end + + it 'dismisses the workers it forked when loading raises' do + allow(config).to receive(:load_spec_files).and_raise(SystemExit) + + expect { Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) }.to raise_error(SystemExit) + expect(plan).to have_received(:dismiss) + end + end + it 'supports a test-queue like subclass that can perform setup once and run different sets of example groups multiple times' do order = [] diff --git a/rspec-core/spec/rspec/core/suite_hooks_spec.rb b/rspec-core/spec/rspec/core/suite_hooks_spec.rb index cc21d93c7..92669ca6a 100644 --- a/rspec-core/spec/rspec/core/suite_hooks_spec.rb +++ b/rspec-core/spec/rspec/core/suite_hooks_spec.rb @@ -94,6 +94,23 @@ module RSpec::Core }.to raise_error("boom").and yield_control end + it 'calls a given callable before any `after(:suite)` hook, however the suite ends' do + order = [] + RSpec.configuration.after(:suite) { order << :after_suite } + RSpec.configuration.before(:suite) do + RSpec.configuration.after(:suite) { order << :added_by_a_before_suite_hook } + end + exiting = lambda { order << [:before_teardown, $!.class] } + + RSpec.configuration.with_suite_hooks(exiting) { order << :suite } + expect { RSpec.configuration.with_suite_hooks(exiting) { exit 2 } }.to raise_error(SystemExit) + + expect(order).to eq([ + :suite, [:before_teardown, NilClass], :added_by_a_before_suite_hook, :after_suite, + [:before_teardown, SystemExit], :added_by_a_before_suite_hook, :added_by_a_before_suite_hook, :after_suite + ]) + end + describe "the runner" do include_context "Runner support" diff --git a/rspec-core/spec/support/aruba_support.rb b/rspec-core/spec/support/aruba_support.rb index c3bca31dc..2b9292787 100644 --- a/rspec-core/spec/support/aruba_support.rb +++ b/rspec-core/spec/support/aruba_support.rb @@ -6,6 +6,21 @@ module ArubaLoader with_isolated_stderr do require 'aruba/api' end + + # Each parallel worker (`rspec --parallel`, parallel_tests, ...) gets its own + # working directory: `setup_aruba` wipes it, which would pull the directory + # out from under the other workers' `cd`. + def self.working_directory + "tmp/aruba#{ENV['TEST_ENV_NUMBER']}" + end +end + +RSpec.configure do |c| + # In a `before(:suite)` hook because `rspec --parallel` forks its workers, + # which set `TEST_ENV_NUMBER`, after the suite has loaded. + c.before(:suite) do + Aruba.configure { |config| config.working_directory = ArubaLoader.working_directory } + end end RSpec.shared_context "aruba support" do @@ -15,6 +30,10 @@ module ArubaLoader let(:stderr) { StringIO.new } let(:stdout) { StringIO.new } + def aruba_working_directory + ArubaLoader.working_directory + end + before { setup_aruba } attr_reader :last_cmd_stdout, :last_cmd_stderr, :last_cmd_exit_status diff --git a/rspec-core/spec/support/formatter_support.rb b/rspec-core/spec/support/formatter_support.rb index fe1735fe4..94eca0d3a 100644 --- a/rspec-core/spec/support/formatter_support.rb +++ b/rspec-core/spec/support/formatter_support.rb @@ -272,7 +272,8 @@ module RSpec::Core class Reporter module EnforceRSpecNotificationsListComplete def notify(event, *args) - return super if caller_locations(1, 1).first.label =~ /publish/ + # Two frames, so that reporters overriding `notify` and calling `super` are covered too. + return super if caller_locations(1, 2).any? { |location| location.label =~ /publish/ } return super if RSPEC_NOTIFICATIONS.include?(event) raise "#{event.inspect} must be added to `RSPEC_NOTIFICATIONS`" diff --git a/rspec-core/spec/support/spec_files.rb b/rspec-core/spec/support/spec_files.rb index a9f89f16e..590abe4f2 100644 --- a/rspec-core/spec/support/spec_files.rb +++ b/rspec-core/spec/support/spec_files.rb @@ -1,13 +1,21 @@ RSpec.shared_context "spec files" do + # Written outside `spec/` so that the library-wide checks never glob them, + # and named per parallel worker so that one worker's `after(:all)` cannot + # delete the files another worker is loading. + def spec_file_path(basename) + File.expand_path("../../../tmp/#{basename}#{ENV['TEST_ENV_NUMBER']}.rb", __FILE__) + end + def failing_spec_filename - @failing_spec_filename ||= File.expand_path(File.dirname(__FILE__)) + "/_failing_spec.rb" + @failing_spec_filename ||= spec_file_path("_failing_spec") end def passing_spec_filename - @passing_spec_filename ||= File.expand_path(File.dirname(__FILE__)) + "/_passing_spec.rb" + @passing_spec_filename ||= spec_file_path("_passing_spec") end def create_passing_spec_file + FileUtils.mkdir_p(File.dirname(passing_spec_filename)) File.open(passing_spec_filename, 'w') do |f| f.write %q{ RSpec.describe "passing spec" do @@ -20,6 +28,7 @@ def create_passing_spec_file end def create_failing_spec_file + FileUtils.mkdir_p(File.dirname(failing_spec_filename)) File.open(failing_spec_filename, 'w') do |f| f.write %q{ RSpec.describe "failing spec" do