From 974d4027330dd9eb29d9ca9aa7766870517e83dc Mon Sep 17 00:00:00 2001 From: ilyazub Date: Sun, 4 Oct 2026 22:45:08 +0200 Subject: [PATCH 01/58] [rspec-core] Run example groups in parallel with --parallel[=COUNT] `rspec --parallel[=COUNT]` (or `config.parallel_workers`) forks COUNT workers from the process that loaded the suite and hands them top-level example groups from a shared queue, longest first when `.rspec_status` recorded their timings. Workers forward every notification -- including custom ones published through the reporter -- as id-based, Marshal-safe frames over a pipe; the coordinator maps them back to the real `Example` and `ExampleGroup` objects and replays them onto the ordinary `Reporter`, so existing formatters work unchanged and the run keeps one seed, one summary, one profile, one `.rspec_status` and the usual exit codes. Formatters that follow group boundaries see one group at a time, with bounded latency; the others get every notification as it arrives. Exceptions are never marshaled: workers ship class name, message, detailed message, backtrace, cause chain and aggregate-failure details, and the coordinator rebuilds a subclass of the real class under the original name. A worker that finishes but never exits cannot hang the run: it arms a watchdog before its `after(:suite)` hooks that dumps every thread's backtrace and aborts after `config.parallel_shutdown_timeout`, and the coordinator independently escalates TERM -> KILL -> give up. Cancelled workers (fail-fast, interrupt) and crashed workers are bounded and reported the same way. Suite hooks run once per worker, which sees `config.parallel_worker_number` and a parallel_tests-style `TEST_ENV_NUMBER`. Without `fork`, or inside another tool's parallel worker, the run is serial with a warning. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 13 + rspec-core/README.md | 55 ++ rspec-core/features/.nav | 1 + .../features/command_line/parallel.feature | 125 +++++ rspec-core/lib/rspec/core.rb | 1 + rspec-core/lib/rspec/core/configuration.rb | 58 +++ rspec-core/lib/rspec/core/example.rb | 21 + rspec-core/lib/rspec/core/option_parser.rb | 5 + rspec-core/lib/rspec/core/parallel.rb | 124 +++++ rspec-core/lib/rspec/core/parallel/channel.rb | 148 ++++++ .../lib/rspec/core/parallel/coordinator.rb | 440 ++++++++++++++++ .../rspec/core/parallel/remote_exception.rb | 193 +++++++ rspec-core/lib/rspec/core/parallel/results.rb | 40 ++ .../lib/rspec/core/parallel/sequencer.rb | 283 ++++++++++ .../lib/rspec/core/parallel/suite_index.rb | 32 ++ rspec-core/lib/rspec/core/parallel/worker.rb | 126 +++++ .../lib/rspec/core/parallel/worker_handle.rb | 73 +++ .../rspec/core/parallel/worker_reporter.rb | 122 +++++ rspec-core/lib/rspec/core/profiler.rb | 9 + rspec-core/lib/rspec/core/reporter.rb | 8 + rspec-core/lib/rspec/core/runner.rb | 5 +- rspec-core/spec/integration/parallel_spec.rb | 446 ++++++++++++++++ .../spec/rspec/core/configuration_spec.rb | 56 ++ rspec-core/spec/rspec/core/example_spec.rb | 38 ++ .../spec/rspec/core/option_parser_spec.rb | 20 + .../spec/rspec/core/parallel/channel_spec.rb | 111 ++++ .../rspec/core/parallel/coordinator_spec.rb | 485 ++++++++++++++++++ .../core/parallel/remote_exception_spec.rb | 242 +++++++++ .../rspec/core/parallel/sequencer_spec.rb | 360 +++++++++++++ .../core/parallel/worker_reporter_spec.rb | 186 +++++++ .../spec/rspec/core/parallel/worker_spec.rb | 171 ++++++ rspec-core/spec/rspec/core/parallel_spec.rb | 155 ++++++ rspec-core/spec/rspec/core/profiler_spec.rb | 26 + rspec-core/spec/rspec/core/reporter_spec.rb | 18 + rspec-core/spec/rspec/core/runner_spec.rb | 15 + 35 files changed, 4210 insertions(+), 1 deletion(-) create mode 100644 rspec-core/features/command_line/parallel.feature create mode 100644 rspec-core/lib/rspec/core/parallel.rb create mode 100644 rspec-core/lib/rspec/core/parallel/channel.rb create mode 100644 rspec-core/lib/rspec/core/parallel/coordinator.rb create mode 100644 rspec-core/lib/rspec/core/parallel/remote_exception.rb create mode 100644 rspec-core/lib/rspec/core/parallel/results.rb create mode 100644 rspec-core/lib/rspec/core/parallel/sequencer.rb create mode 100644 rspec-core/lib/rspec/core/parallel/suite_index.rb create mode 100644 rspec-core/lib/rspec/core/parallel/worker.rb create mode 100644 rspec-core/lib/rspec/core/parallel/worker_handle.rb create mode 100644 rspec-core/lib/rspec/core/parallel/worker_reporter.rb create mode 100644 rspec-core/spec/integration/parallel_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/channel_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/coordinator_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/sequencer_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel/worker_spec.rb create mode 100644 rspec-core/spec/rspec/core/parallel_spec.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 39ccc5787..49ccdb773 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -1,6 +1,19 @@ ### Development [Full Changelog](https://github.com/rspec/rspec/compare/rspec-core-v3.13.6...main) +Enhancements: + +* Add `--parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run + top-level example groups across worker processes forked after the suite has loaded. + Formatters run in the coordinating process only and receive one coherent stream of + notifications with real `Example` objects, so existing formatters work unchanged and the + run has one seed, one summary, one profile and one `.rspec_status` file. Workers run the + suite hooks once each, expose `RSpec::Core::Configuration#parallel_worker_number` and set + `TEST_ENV_NUMBER` the way parallel_tests does. A worker that finishes but does not exit is + aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of + its threads, so a stuck `at_exit` handler can no longer hang a run. Where `fork` is + unavailable the run is serial with a warning. (Illia Zub) + # 4.0.0.beta1 / 2026-02-18 Breaking Changes: diff --git a/rspec-core/README.md b/rspec-core/README.md index 1798a5198..d1a039b46 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -275,6 +275,61 @@ which you'll use to run rspec. The `rspec` command comes with many useful options. Run `rspec --help` to see the complete list. +## Running Examples in Parallel + +`rspec --parallel` runs the top-level example groups of your suite across +worker processes, one per processor by default (`--parallel=4` for four). +RSpec loads the suite once, forks the workers from the process that loaded it, +hands out groups to whichever worker is free -- longest first when a previous +run recorded timings in the `example_status_persistence_file_path` file -- and +streams results back as examples finish. Your formatters run only in the +coordinating process and see the notifications of a normal run, so there is +one seed, one summary, one profile and one exit code, and `--fail-fast`, +`--only-failures`, `--next-failure`, `--seed`, filters and `path:line` +selection keep working. You can make it the default with: + +```ruby +RSpec.configure do |config| + config.parallel_workers = :auto # or an Integer +end +``` + +Each worker runs the `before(:suite)` and `after(:suite)` hooks once. Inside a +worker, `RSpec.configuration.parallel_worker_number` is its 1-based number and +`ENV['TEST_ENV_NUMBER']` follows the convention of parallel_tests (`""` for the +first worker, then `"2"`, `"3"`, ...), so anything you already namespace per +worker for that tool keeps working. + +Workers inherit everything that happened while the suite was loaded. For a +Rails application that means `rails_helper` has already evaluated +`database.yml` and connected to the database in the coordinating process +before any worker exists: setting `TEST_ENV_NUMBER` afterwards does **not** +give workers separate databases, so Rails suites run under `--parallel` share +one test database until rspec-rails adds per-worker database support. Open +connections, servers and similar resources acquired at load time are shared +the same way; acquire them in `before(:suite)` to get one per worker. + +Formatters receive examples with their status, timing, pending details, +exceptions (class name, message, backtrace, cause chain and aggregated +failures; rebuilt as a subclass of the original class when that class exists +in the coordinating process) and the group and example notifications in the +order the worker produced them, plus `reporter.message`, deprecations and +custom notifications whose options can be marshaled. Metadata that a worker +mutates while running an example is not forwarded, and anything workers write +to `$stdout`/`$stderr` directly is not sequenced with the formatter output. + +A worker that has finished its last example group but does not exit (a +non-daemon thread, a browser that will not quit, ...) is given +`config.parallel_shutdown_timeout` seconds (30 by default) to run its +`after(:suite)` hooks and `at_exit` handlers; after that it dumps the +backtraces of all its threads to stderr and is aborted, and the run exits +with `config.error_exit_code`. The coordinator never waits unboundedly for a +finished worker. + +`--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is +already set because another tool is running RSpec in parallel, RSpec prints a +warning and runs serially. `--bisect` and `--dry-run` always run serially. + ## Store Command Line Options `.rspec` You can store command line options in a `.rspec` file in the project's root diff --git a/rspec-core/features/.nav b/rspec-core/features/.nav index 009cf30c6..05e2d8888 100644 --- a/rspec-core/features/.nav +++ b/rspec-core/features/.nav @@ -9,6 +9,7 @@ - line_number_appended_to_path.feature - exit_status.feature - order.md (`--order` option) + - parallel.feature - rake_task.feature - pending_and_skipped_examples: - pending_examples.feature diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature new file mode 100644 index 000000000..25f94ee65 --- /dev/null +++ b/rspec-core/features/command_line/parallel.feature @@ -0,0 +1,125 @@ +Feature: `--parallel` option + + Use the `--parallel[=COUNT]` option to run the top-level example groups of + your suite across several worker processes. RSpec loads the suite once, forks + the workers from the process that loaded it, and hands groups out to whichever + worker is free. Results stream back as examples finish, so you get a single + report with one seed, one summary, one profile and one exit code -- produced + by the same formatters that a serial run uses. + + Without a count, one worker per processor is used. Set + `config.parallel_workers` in `RSpec.configure` to make it the default. + + Each worker runs the `before(:suite)` and `after(:suite)` hooks once. In a + worker, `RSpec.configuration.parallel_worker_number` is its 1-based number + and `ENV['TEST_ENV_NUMBER']` follows the convention of parallel_tests (`""` + for the first worker, then `"2"`, `"3"`, ...), so resources that are already + namespaced per worker for that tool keep working. Everything that happens + while the suite is loaded -- such as a Rails `rails_helper` evaluating + `database.yml` and connecting -- happens once, in the coordinating process, + and is inherited by every worker. + + `--parallel` needs `fork`. Where it is not available (Windows, JRuby), or + when `TEST_ENV_NUMBER` is already set because another tool is running RSpec + in parallel, RSpec prints a warning and runs serially. + + Background: + Given a file named "spec/alpha_spec.rb" with: + """ruby + RSpec.configure do |config| + config.before(:suite) do + File.write("worker_#{config.parallel_worker_number}", ENV['TEST_ENV_NUMBER'].inspect) + end + end + + RSpec.describe "Alpha" do + it "passes" do + expect(1).to eq(1) + end + + it "fails" do + expect(1).to eq(2) + end + end + """ + And a file named "spec/beta_spec.rb" with: + """ruby + RSpec.describe "Beta" do + it "passes" do + expect(2).to eq(2) + end + + context "nested" do + it "passes too" do + expect(3).to eq(3) + end + end + end + """ + + Scenario: Running a suite across two workers produces one ordinary report + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 1 + And the output should contain "4 examples, 1 failure" + And the output should contain "rspec ./spec/alpha_spec.rb:12 # Alpha fails" + And the output should contain "expected: 2" + And the file "worker_1" should contain exactly: + """ + "" + """ + And the file "worker_2" should contain exactly: + """ + "2" + """ + + Scenario: Formatters that follow group boundaries see each group whole + When I run `rspec --parallel=2 --order defined --format documentation` + Then the exit status should be 1 + And the output should contain: + """ + Beta + passes + nested + passes too + """ + And the output should contain: + """ + Alpha + passes + fails (FAILED - 1) + """ + + Scenario: A worker that finishes but never exits cannot hang the run + Given a file named "spec/hang_spec.rb" with: + """ruby + RSpec.configure do |config| + config.parallel_shutdown_timeout = 1 + + config.after(:suite) do + if config.parallel_worker_number == 2 + thread = Thread.new { sleep } + at_exit { thread.join } + end + end + end + + RSpec.describe "Gamma" do + it "passes" do + expect(4).to eq(4) + end + end + """ + When I run `rspec --parallel=2 --error-exit-code 3 spec/beta_spec.rb spec/hang_spec.rb` + Then the exit status should be 3 + And the output should contain "3 examples, 0 failures" + And the output should match /Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/ + And the stderr should contain "did not exit within 1 seconds of finishing its last example group" + And the stderr should contain "Threads at this point:" + And the stderr should contain "Thread#join" + + Scenario: Inside another tool's parallel worker RSpec runs serially + Given I set the environment variable "TEST_ENV_NUMBER" to "2" + When I run `rspec --parallel=2 --order defined spec/beta_spec.rb` + Then the exit status should be 0 + And the output should contain "2 examples, 0 failures" + And the stderr should contain "Ignoring `--parallel`: this process is already a worker of a parallel test run" diff --git a/rspec-core/lib/rspec/core.rb b/rspec-core/lib/rspec/core.rb index 86a300c6a..fd1ecd2cc 100644 --- a/rspec-core/lib/rspec/core.rb +++ b/rspec-core/lib/rspec/core.rb @@ -33,6 +33,7 @@ example shared_example_group example_group + parallel ].each { |name| RSpec::Support.require_rspec_core name } # Namespace for all core RSpec code. diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 7cb484433..e0eab121b 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -231,6 +231,56 @@ def fail_fast=(value) # @return [Boolean] add_setting :fail_if_no_examples + # @macro define_reader + # The number of worker processes to run top-level example groups across + # (default: `nil`, run everything in this process). `:auto` uses one + # worker per processor. Set from the command line with + # `--parallel[=COUNT]`. + # + # Workers are forked from this process once the suite has been loaded, + # so they inherit everything `spec_helper` and the spec files set up at + # load time -- including connections opened then. `before(:suite)` and + # `after(:suite)` hooks run once in each worker, where + # {#parallel_worker_number} and `ENV['TEST_ENV_NUMBER']` identify the + # worker. Formatters run in this process only and see the notifications + # of a normal run. + # + # Needs `fork`; where it is unavailable (Windows, JRuby) the run is + # serial and a warning is printed, as it is when `TEST_ENV_NUMBER` is + # already set because another tool is parallelizing the run. + # + # @return [Integer, :auto, nil] + define_reader :parallel_workers + + # @see parallel_workers + def parallel_workers=(value) + if value.nil? || value == :auto || (value.is_a?(Integer) && value >= 1) + @parallel_workers = value + else + RSpec.warning "Cannot set `RSpec.configuration.parallel_workers` to `#{value.inspect}`; " \ + "it must be a positive integer, `:auto` or `nil`. Running serially." + @parallel_workers = nil + end + end + + # @macro add_setting + # Seconds a parallel worker is given to exit once it has finished its + # last example group -- that is, to run its `after(:suite)` hooks and + # `at_exit` handlers (default: 30). A worker that takes longer dumps the + # backtraces of all its threads to stderr and is aborted, and the run + # exits with {#error_exit_code}. + # @return [Numeric] + add_setting :parallel_shutdown_timeout + + # The 1-based number of the parallel worker this process is, or `nil` + # in the coordinating process and in serial runs. In workers, + # `ENV['TEST_ENV_NUMBER']` is also set, following the convention of + # parallel_tests: `""` for the first worker, then `"2"`, `"3"`, ... so + # that per-worker resource naming written for that tool keeps working. + # @return [Integer, nil] + # @note The writer is for the parallel runner's use. + attr_accessor :parallel_worker_number + # @macro define_reader # Indicates files configured to be required. # @return [Array] @@ -458,6 +508,9 @@ def initialize @failure_exit_code = 1 @error_exit_code = nil # so it can be overridden by failure exit code @fail_if_no_examples = false + @parallel_workers = nil + @parallel_shutdown_timeout = 30 + @parallel_worker_number = nil @spec_files_loaded = false @backtrace_formatter = BacktraceFormatter.new @@ -527,6 +580,11 @@ def reset_reporter @output_wrapper = nil end + # @private + # Used by the parallel runner to install the reporter that forwards a + # worker's notifications to the coordinating process. + attr_writer :reporter + # @private def reset_filters self.filter_manager = FilterManager.new diff --git a/rspec-core/lib/rspec/core/example.rb b/rspec-core/lib/rspec/core/example.rb index 1477aebe1..661ae3f44 100644 --- a/rspec-core/lib/rspec/core/example.rb +++ b/rspec-core/lib/rspec/core/example.rb @@ -441,6 +441,27 @@ def skip_with_exception(reporter, exception) finish(reporter) end + # @private + # + # Used by the parallel runner to copy onto this instance the result of + # running the same example in a worker process. Exceptions are the + # rebuilt stand-ins described in `Parallel::RemoteException`. + def assign_execution_result(attributes) + result = execution_result + result.status = attributes[:status] + result.started_at = attributes[:started_at] + result.finished_at = attributes[:finished_at] + result.run_time = attributes[:run_time] + result.pending_message = attributes[:pending_message] + result.pending_fixed = attributes[:pending_fixed] + result.pending_exception = attributes[:pending_exception] + result.exception = @exception = attributes[:exception] + + [:pending, :skip, :extra_failure_lines].each do |key| + metadata[key] = attributes[key] unless attributes[key].nil? + end + end + # @private def instance_exec(*args, &block) @example_group_instance.instance_exec(*args, &block) diff --git a/rspec-core/lib/rspec/core/option_parser.rb b/rspec-core/lib/rspec/core/option_parser.rb index 57af01707..d602fa6c9 100644 --- a/rspec-core/lib/rspec/core/option_parser.rb +++ b/rspec-core/lib/rspec/core/option_parser.rb @@ -75,6 +75,11 @@ def parser(options) options[:runner] = RSpec::Core::Invocations::Bisect.new end + parser.on('--parallel[=COUNT]', Integer, 'Run example groups across COUNT worker processes forked after', + ' loading the suite (default: one per processor). Needs `fork`.') do |count| + options[:parallel_workers] = count || :auto + end + parser.on('--[no-]fail-fast[=COUNT]', 'Abort the run after a certain number of failures (1 by default).') do |argument| if argument == true value = 1 diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb new file mode 100644 index 000000000..2784dee43 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -0,0 +1,124 @@ +RSpec::Support.require_rspec_core "parallel/channel" +RSpec::Support.require_rspec_core "parallel/remote_exception" +RSpec::Support.require_rspec_core "parallel/worker_reporter" +RSpec::Support.require_rspec_core "parallel/worker" +RSpec::Support.require_rspec_core "parallel/sequencer" +RSpec::Support.require_rspec_core "parallel/results" +RSpec::Support.require_rspec_core "parallel/suite_index" +RSpec::Support.require_rspec_core "parallel/worker_handle" +RSpec::Support.require_rspec_core "parallel/coordinator" + +module RSpec + module Core + # Runs top-level example groups across worker processes forked from the + # process that loaded the suite (`rspec --parallel[=COUNT]`, + # `config.parallel_workers`). Formatters run in the coordinating process + # only and receive the notifications of a normal run; see {Coordinator}. + # @private + module Parallel + DURATION = /(?:(\d+) minutes? )?([\d.]+) seconds?/ + + # @return [Coordinator, nil] a coordinator for the groups, or `nil` when + # the run should proceed serially (not requested, not possible or not + # worth it). Warns when a requested parallel run is not possible. + def self.coordinator_for(runner, example_groups) + requested = requested_worker_count(runner.configuration) + return nil unless requested + + units = units_for(example_groups, runner.configuration) + count = [requested, units.size].min + return nil if count < 2 + + Coordinator.new(runner, example_groups, units, count) + end + + # @private + def self.requested_worker_count(configuration) + requested = configuration.parallel_workers + return nil if requested.nil? || configuration.dry_run? + + unless RSpec::Support::RubyFeatures.fork_supported? + RSpec.warning "Ignoring `--parallel`: it needs `fork`, which is not available on #{RUBY_ENGINE} " \ + "(#{RUBY_PLATFORM}). Running serially.", :call_site => nil + return nil + end + + if ENV.key?('TEST_ENV_NUMBER') + RSpec.warning "Ignoring `--parallel`: this process is already a worker of a parallel test run " \ + "(TEST_ENV_NUMBER is set). Running serially.", :call_site => nil + return nil + end + + requested == :auto ? default_worker_count : requested + end + + # @return [Integer] the number of processors + def self.default_worker_count + require 'etc' + Etc.nprocessors + end + + # The groups to dispatch: top-level groups with at least one example to + # run, longest first when a previous run recorded their timings in the + # example status persistence file, so that the slowest group is never + # the one that starts last. Groups without timings go first because + # nothing is known about them. + # @private + def self.units_for(example_groups, configuration) + units = example_groups.select { |group| group.descendant_filtered_examples.any? } + run_times = recorded_run_times(configuration) + return units if run_times.empty? + + timed = {} + units.each do |group| + ids = group.descendant_filtered_examples.map(&:id) + timed[group] = ids.sum { |id| run_times.fetch(id, 0.0) } if ids.any? { |id| run_times.key?(id) } + end + + unknown, known = units.partition { |group| !timed.key?(group) } + unknown + known.each_with_index.sort_by { |group, index| [-timed[group], index] }.map(&:first) + end + + # @private + def self.recorded_run_times(configuration) + path = configuration.example_status_persistence_file_path + return {} unless path + + ExampleStatusPersister.load_from(path).each_with_object({}) do |example, hash| + seconds = parse_duration(example[:run_time]) + hash[example[:example_id]] = seconds if seconds + end + rescue SystemCallError + {} + end + + # @private + def self.parse_duration(string) + match = DURATION.match(string.to_s) + return nil unless match + + (match[1].to_i * 60) + match[2].to_f + end + + # @private + def self.seconds(duration) + Formatters::Helpers.pluralize(Formatters::Helpers.format_seconds(duration), 'second') + end + + # @private + def self.monotonic_time + Process.clock_gettime(Process::CLOCK_MONOTONIC) + end + + # @return the value when `Marshal` can serialize it, otherwise what the + # block makes of it + # @private + def self.marshal_safe(value) + Marshal.dump(value) + value + rescue StandardError + yield value + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/channel.rb b/rspec-core/lib/rspec/core/parallel/channel.rb new file mode 100644 index 000000000..27d14803b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/channel.rb @@ -0,0 +1,148 @@ +module RSpec + module Core + module Parallel + # Length-prefixed `Marshal` frames over an anonymous pipe, exchanged only + # between the coordinating process and the worker processes it forked + # (the trust boundary `Bisect::Channel` already relies on). The reader + # side never blocks the caller unless asked to: frames are reassembled + # from whatever bytes are available, so a worker stalled mid-frame cannot + # stall the coordinator. + # @private + module Channel + ENCODING = Marshal.dump("").encoding + CHUNK_SIZE = 65_536 + + # Raised when the peer wrote something that is not a frame. + class ProtocolError < StandardError; end + + # @return [Array(Reader, Writer)] the two ends of a new pipe + def self.pipe + read_io, write_io = IO.pipe + [Reader.new(read_io), Writer.new(write_io)] + end + + # @private + class Writer + attr_reader :io + + def initialize(io) + @io = io + @io.set_encoding ENCODING + @io.sync = true + end + + def send(message) + packet = Marshal.dump(message) + @io.write("#{packet.bytesize}\n#{packet}") + end + + def closed? + @io.closed? + end + + def close + @io.close unless @io.closed? + end + end + + # @private + class Reader + attr_reader :io, :bytes_read + + def initialize(io) + @io = io + @io.set_encoding ENCODING + @buffer = String.new(:encoding => ENCODING) + @pending = [] + @eof = false + @bytes_read = 0 + end + + def eof? + @eof + end + + def closed? + @io.closed? + end + + def close + @io.close unless @io.closed? + end + + # Reads what is available without blocking. + # @return [Array] decoded messages with their frame size in bytes + def drain + read_available + messages = [] + while (frame = next_frame) + messages << frame + end + messages + end + + # Puts a message obtained from {#drain} back so that {#receive} + # returns it first. + def unread(message) + @pending << message + end + + # Blocks until a whole message has arrived. + # @return [Object, nil] the message, or `nil` once the writer is gone + def receive + return @pending.shift unless @pending.empty? + + loop do + frame = next_frame + return frame.first if frame + return nil if @eof + + # `IO#wait_readable` would need `io/wait`, which rspec-core does not load. + IO.select([@io]) # rubocop:disable Lint/IncompatibleIoSelectWithFiberScheduler + read_available + end + end + + private + + # Reads what the kernel has buffered, a bounded number of chunks at a + # time so that one chatty worker cannot monopolize the coordinator. + def read_available + 16.times do + break if @eof + + chunk = @io.read_nonblock(CHUNK_SIZE, :exception => false) + case chunk + when nil then @eof = true + when :wait_readable then break + else + @bytes_read += chunk.bytesize + @buffer << chunk + end + end + end + + # rubocop:disable-next Security/MarshalLoad + def next_frame + newline = @buffer.index("\n") + return nil unless newline + + size = frame_size(@buffer.byteslice(0, newline)) + frame_end = newline + 1 + size + return nil if @buffer.bytesize < frame_end + + packet = @buffer.byteslice(newline + 1, size) + @buffer = @buffer.byteslice(frame_end, @buffer.bytesize - frame_end) + [Marshal.load(packet), frame_end] + end + + def frame_size(header) + Integer(header, 10) + rescue ArgumentError, TypeError + raise ProtocolError, "Expected a frame length, got #{header.inspect}" + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb new file mode 100644 index 000000000..d65a6e12b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -0,0 +1,440 @@ +module RSpec + module Core + module Parallel + # The coordinating side of a parallel run. It forks the workers after the + # suite has been loaded (so they inherit every example group, filter and + # configuration), hands out top-level example groups from a shared queue, + # streams the results onto the configuration's reporter -- the one and + # only place formatters run -- and makes sure that no worker, however it + # misbehaves, can keep the run from finishing. + # @private + class Coordinator + TICK = 0.25 + TERM_GRACE = 5.0 + KILL_GRACE = 5.0 + # Added to the shutdown timeout on the coordinator's side so that the + # worker's own watchdog fires first and the thread dump comes from + # the process that owns the threads. + WATCHDOG_GRACE = 1.0 + + def initialize(runner, example_groups, units, worker_count, options={}) + @runner = runner + @configuration = runner.configuration + @world = runner.world + @example_groups = example_groups + @units = units + @worker_count = worker_count + @process = options.fetch(:process) { Process } + @spawn = options.fetch(:spawn) { method(:fork_worker) } + @tick = options.fetch(:tick, TICK) + @term_grace = options.fetch(:term_grace, TERM_GRACE) + @kill_grace = options.fetch(:kill_grace, KILL_GRACE) + @watchdog_grace = options.fetch(:watchdog_grace, WATCHDOG_GRACE) + @workers = [] + @queue = (0...@units.size).to_a + @unit_states = Array.new(@units.size, :pending) + @results = Results.new + @quitting = false + @unknown_ids = [] + end + + # @return [Integer] the exit code for the run + def run + reporter = @configuration.reporter + + passed = reporter.report(@world.example_count(@example_groups)) do + @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) + + begin + start_workers + event_loop + ensure + terminate_remaining_workers + @sequencer.finish + @results.apply_all + report_unrun_units + end + + passed? + end + + @runner.exit_code(passed) + end + + # @private + def passed? + @unit_states.all? { |state| state == :passed } + end + + private + + def start_workers + flush_output + @process.warmup if @process.respond_to?(:warmup) + @worker_count.times { |index| @workers << WorkerHandle.new(index + 1) } + + @workers.each do |worker| + worker.pid = @spawn.call(worker, @workers) + worker.close_child_ends + end + end + + # Whatever the formatters wrote before `start` (seed, run options) is + # still buffered in streams that are not `$stdout`; forked workers + # would write it again when they exit. + def flush_output + streams = @configuration.formatters.map { |formatter| formatter.output if formatter.respond_to?(:output) } + streams.push(@configuration.output_stream, @configuration.error_stream, @configuration.deprecation_stream) + streams.push($stdout, $stderr) + streams.each { |stream| stream.flush if stream.respond_to?(:flush) } + end + + def fork_worker(worker, workers) + @process.fork do + # :nocov: - runs in the forked worker process, covered by integration specs + workers.each { |other| other.close_all unless other.equal?(worker) } + worker.close_parent_ends + Worker.new(worker.number, @units, worker.child_jobs, worker.child_results).run + # :nocov: + end + end + + def event_loop + dispatch_work + + until @workers.all?(&:done?) + wait_for_results + reap_workers + request_quit if @world.wants_to_quit && !@quitting + dispatch_work + enforce_deadlines + @sequencer.tick + end + end + + def wait_for_results + readers = @workers.reject { |worker| worker.done? || worker.results.eof? || worker.results.closed? } + + if readers.empty? + sleep @tick + return + end + + ready, = IO.select(readers.map { |worker| worker.results.io }, nil, nil, @tick) + Array(ready).each do |io| + receive_from(readers.find { |worker| worker.results.io.equal?(io) }) + end + end + + def receive_from(worker) + return if worker.results.closed? + + messages = drain(worker) + messages.each { |message, bytes| handle_message(worker, message, bytes) } + + # A worker whose results pipe closed while it was still running is + # either about to be reaped or wedged; bound the wait for it. + worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace if worker.results.eof? + end + + def drain(worker) + worker.results.drain + rescue Channel::ProtocolError, TypeError, ArgumentError, IOError => e + worker.results.close + terminate(worker, "sent data the coordinator could not decode (#{e.class}: #{e.message})") + [] + end + + def handle_message(worker, message, bytes) + kind, *rest = message + case kind + when :group_started then with_group(rest[0]) { |g| @sequencer.group_started(worker.number, g, bytes) } + when :group_finished then with_group(rest[0]) { |g| @sequencer.group_finished(worker.number, g, rest[1], bytes) } + when :example_started then with_example(rest[0]) { |e| @sequencer.example_started(worker.number, e, bytes) } + when :example_finished then example_finished(worker, rest[0], rest[1], bytes) + when :message then @sequencer.message(worker.number, rest[0], bytes) + when :deprecation then @sequencer.deprecation(worker.number, rest[0], bytes) + when :custom then @sequencer.custom(worker.number, rest[0], rest[1], bytes) + else handle_control_message(worker, kind, rest) + end + end + + def handle_control_message(worker, kind, rest) + case kind + when :unit_finished then unit_finished(worker, *rest) + when :non_example_failure then non_example_failure(*rest) + when :exiting then worker.exiting = true + else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." + end + end + + def non_example_failure(payload, context_description) + @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) + end + + def example_finished(worker, example_id, payload, bytes) + with_example(example_id) do |example| + record_result(example, payload) + @sequencer.example_finished(worker.number, example, bytes) + end + end + + def with_example(id, &block) + with_known(index.example(id), id, &block) + end + + def with_group(id, &block) + with_known(index.group(id), id, &block) + end + + def index + @index ||= SuiteIndex.new(@world) + end + + # Examples and groups defined while running (rather than loading) the + # suite exist only in the worker that defined them. + def with_known(target, id) + return yield(target) if target + return if @unknown_ids.include?(id) + + @unknown_ids << id + report "A parallel worker reported results for #{id}, which was not defined when the suite was loaded; " \ + "examples and groups defined at run time cannot be reported by `--parallel`." + end + + # Recorded as soon as it arrives: failure counting and `.rspec_status` + # persistence never wait for presentation. + def record_result(example, payload) + @results.record(example, payload) + request_quit if fail_fast_limit_met? + end + + def fail_fast_limit_met? + limit = @configuration.fail_fast + return false unless limit + + limit == true ? @results.failed_count >= 1 : @results.failed_count >= limit + end + + def apply_result(example) + @results.apply(example) + end + + # A worker whose world wants to quit would only skip the groups it is + # given (`ExampleGroup.run` returns at once), so it gets no more. The + # events that should end the whole run reach the coordinator by other + # means: an interrupt hits every process in the group, fail-fast is + # counted here and suite hook failures are reported as such. + def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) + @unit_states[index] = passed ? :passed : :failed if worker.current_unit == index + worker.current_unit = nil + @world.non_example_failure = true if non_example_failure + return unless wants_to_quit && !@quitting && !@world.wants_to_quit + + report "#{worker.description} stopped after #{unit_description(index)} because `RSpec.world.wants_to_quit` was " \ + "set while it ran (an interrupt, a suite hook failure or a spec setting it); it runs no further groups." + stop(worker) + end + + def dispatch_work + @workers.each do |worker| + next unless worker.idle? + + if @quitting || @queue.empty? + stop(worker) + else + dispatch(worker, @queue.shift) + end + end + end + + def dispatch(worker, index) + worker.current_unit = index + @unit_states[index] = :running + return if write(worker, [:run, index]) + + @unit_states[index] = :pending + @queue.unshift(index) + worker.current_unit = nil + worker.state = :stopping + end + + def stop(worker) + worker.state = :stopping + worker.deadline = now + @configuration.parallel_shutdown_timeout + @watchdog_grace + write(worker, [:stop]) + end + + # Busy workers are asked to stop between examples and, like workers + # that have finished, get a deadline: a cancelled run ends even if a + # worker is stuck inside an example. + def request_quit + return if @quitting + + @quitting = true + @world.wants_to_quit = true + @workers.each do |worker| + next if worker.quit_sent || !worker.busy? + + worker.quit_sent = true + worker.deadline = now + @configuration.parallel_shutdown_timeout + write(worker, [:quit]) + end + end + + # Control messages are a few bytes; a worker that is not reading them is + # dead or wedged: reaping or the deadline set here deals with it, and + # `judge_exit` still sees the state it was in. + def write(worker, message) + worker.jobs.send(message) + true + rescue Errno::EPIPE, IOError + worker.deadline ||= now + false + end + + def reap_workers + @workers.each do |worker| + next if worker.done? || worker.pid.nil? + + pid, status = worker.reap(@process) + next unless pid + + receive_from(worker) + worker.exit_status = status + judge_exit(worker, status) + worker.state = :exited + end + end + + def judge_exit(worker, status) + case worker.state + when :running + where = worker.current_unit ? "while running #{unit_description(worker.current_unit)}" : "between example groups" + report "#{worker.description} exited unexpectedly (#{WorkerHandle.describe_status(status)}) #{where}; " \ + "the examples it had not finished did not run." + @world.non_example_failure = true + abort_unit(worker) + when :stopping + return if status&.success? + + report "#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ + "If its exit watchdog fired, its thread dump is on stderr above." + @world.non_example_failure = true + end + end + + def enforce_deadlines + @workers.each do |worker| + next if worker.done? || worker.deadline.nil? || now < worker.deadline + + escalate(worker) + end + end + + def escalate(worker) + case worker.escalation + when nil + report "#{worker.description} #{overdue_description(worker)}; sending SIGTERM. Raise " \ + "`config.parallel_shutdown_timeout` if it legitimately needs longer." + @world.non_example_failure = true + abort_unit(worker) + worker.state = :terminating + signal(worker, 'TERM', :term, @term_grace) + when :term + report "#{worker.description} is still alive #{Parallel.seconds(@term_grace)} after SIGTERM; sending SIGKILL." + signal(worker, 'KILL', :kill, @kill_grace) + when :kill + report "#{worker.description} is still alive #{Parallel.seconds(@kill_grace)} after SIGKILL; giving up on it." + abort_unit(worker) + worker.state = :abandoned + end + end + + def overdue_description(worker) + timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) + if worker.state == :stopping + "did not exit within #{timeout} of finishing its last example group (its `after(:suite)` hooks or " \ + "`at_exit` handlers may be stuck)" + elsif worker.quit_sent + "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)" + else + "stopped reporting #{timeout} ago" + end + end + + def terminate(worker, reason) + report "#{worker.description} #{reason}; sending SIGTERM." + @world.non_example_failure = true + abort_unit(worker) + worker.state = :terminating + signal(worker, 'TERM', :term, @term_grace) + end + + def signal(worker, name, escalation, grace) + worker.escalation = escalation + worker.deadline = now + grace + worker.kill(@process, name) + end + + def abort_unit(worker) + if (index = worker.current_unit) + @unit_states[index] = :aborted + worker.current_unit = nil + end + @sequencer.abandon(worker.number) + end + + # Runs on every exit path, so an exception in the coordinator can never + # leave workers behind. Bounded by the same TERM/KILL grace periods. + def terminate_remaining_workers + @workers.each { |worker| worker.state = :exited if worker.pid.nil? } + live = @workers.reject(&:done?) + return if live.empty? + + live.each do |worker| + worker.state = :terminating + signal(worker, 'TERM', :term, @term_grace) + end + wait_for_exit(live, @term_grace) + live.reject(&:done?).each { |worker| signal(worker, 'KILL', :kill, @kill_grace) } + wait_for_exit(live, @kill_grace) + + live.reject(&:done?).each do |worker| + report "#{worker.description} survived SIGKILL; giving up on it." + abort_unit(worker) + worker.state = :abandoned + end + end + + def wait_for_exit(workers, grace) + deadline = now + grace + until workers.all?(&:done?) || now >= deadline + sleep [@tick, 0.05].min + reap_workers + end + end + + def report_unrun_units + return if @quitting || @queue.empty? + + report "#{Formatters::Helpers.pluralize(@queue.size, 'example group')} did not run because no worker " \ + "remained to run them." + end + + def unit_description(index) + group = @units[index] + "#{group.description} (#{group.location})" + end + + def report(message) + @configuration.reporter.message(message) + end + + def now + Parallel.monotonic_time + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/remote_exception.rb b/rspec-core/lib/rspec/core/parallel/remote_exception.rb new file mode 100644 index 000000000..8568cb811 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/remote_exception.rb @@ -0,0 +1,193 @@ +module RSpec + module Core + module Parallel + # Carries the parts of an exception that formatters consume from a worker + # process to the coordinator without ever marshaling the exception itself + # (user exceptions routinely hold IO objects, anonymous classes or large + # object graphs). The coordinator rebuilds an exception whose class + # reports the original name -- and, when that class is loaded in the + # coordinating process, is a subclass of it, so `rescue`-style and + # `is_a?` checks in formatters keep working. + # @private + module RemoteException + MAX_CAUSE_DEPTH = 10 + CONSTANT_NAME = /\A[A-Z]\w*(::[A-Z]\w*)*\z/ + + # @return [Hash, nil] a `Marshal`-safe description of `exception` + def self.payload_for(exception, depth=0) + return nil if exception.nil? + + payload = { + :class_name => exception.class.name, + :message => message_for(exception), + :detailed_message => detailed_message_for(exception), + :backtrace => backtrace_for(exception), + :cause => cause_payload_for(exception, depth) + } + + if MultipleExceptionError::InterfaceTag === exception + payload[:multiple] = multiple_payload_for(exception, depth) + end + + payload + end + + # @return [Exception, nil] an exception rebuilt from `payload_for`'s output + def self.rebuild(payload) + return nil unless payload + + exception = class_for(payload).allocate + ::Exception.instance_method(:initialize).bind_call(exception, payload[:message]) + exception.set_backtrace(payload[:backtrace]) if payload[:backtrace] + exception.instance_variable_set(:@__rspec_remote_payload, payload) + exception + end + + # @private + def self.message_for(exception) + exception.message.to_s + rescue Exception => e # rubocop:disable Lint/RescueException + "A #{exception.class} for which `exception.message.to_s` raises #{e.class}." + end + + # @private + def self.detailed_message_for(exception) + return nil unless exception.respond_to?(:detailed_message) + + exception.detailed_message(:highlight => false).to_s + rescue Exception # rubocop:disable Lint/RescueException + nil + end + + # @private + def self.backtrace_for(exception) + exception.backtrace&.map(&:to_s) + end + + # @private + def self.cause_payload_for(exception, depth) + return nil if depth >= MAX_CAUSE_DEPTH + + cause = exception.cause + cause && payload_for(cause, depth + 1) + end + + # @private + def self.multiple_payload_for(exception, depth) + { + :all_exceptions => exception.all_exceptions.map { |e| payload_for(e, depth + 1) }, + :aggregation_metadata => Parallel.marshal_safe(exception.aggregation_metadata) do |metadata| + { :hide_backtrace => !!(metadata && metadata[:hide_backtrace]) } + end, + :aggregation_block_label => exception.aggregation_block_label&.to_s, + :summary => exception.summary.to_s, + :exception_count_description => exception.exception_count_description.to_s + } + end + + # @private + def self.class_for(payload) + name = payload[:class_name] + base = resolve(name) || StandardError + multiple = payload.key?(:multiple) + + @classes ||= {} + @classes[[name, base, multiple]] ||= build_class(name, base, multiple) + end + + # @private + # Classes defined while loading the suite exist in the coordinator too, + # since workers are forked from it; classes defined while running an + # example do not and fall back to a `StandardError` stand-in. + def self.resolve(name) + return nil unless name && CONSTANT_NAME =~ name + + klass = Object.const_get(name) + klass if klass.is_a?(Class) && klass <= ::Exception + rescue StandardError, ScriptError + nil + end + + # @private + def self.build_class(name, base, multiple) + Class.new(base) do + include Remote + + if multiple + include MultipleExceptionError::InterfaceTag unless base <= MultipleExceptionError::InterfaceTag + include RemoteMultiple + end + + define_singleton_method(:name) { name } + define_singleton_method(:to_s) { name || super() } + define_singleton_method(:inspect) { name || super() } + end + end + + # Instance methods of every rebuilt exception. + # @private + module Remote + def message + remote_payload[:message] + end + + def to_s + remote_payload[:message] + end + + def inspect + "#<#{self.class.name}: #{message}>" + end + + def detailed_message(*) + remote_payload[:detailed_message] || message + end + + def cause + @cause ||= RemoteException.rebuild(remote_payload[:cause]) + end + + private + + def remote_payload + @__rspec_remote_payload + end + end + + # Instance methods of rebuilt exceptions that aggregate several failures. + # @private + module RemoteMultiple + def all_exceptions + @all_exceptions ||= remote_payload[:multiple][:all_exceptions].map do |payload| + RemoteException.rebuild(payload) + end + end + + def failures + all_exceptions.select { |e| e.class.name =~ /RSpec/ } + end + + def other_errors + all_exceptions.reject { |e| e.class.name =~ /RSpec/ } + end + + def aggregation_metadata + remote_payload[:multiple][:aggregation_metadata] + end + + def aggregation_block_label + remote_payload[:multiple][:aggregation_block_label] + end + + def summary + remote_payload[:multiple][:summary] + end + + def exception_count_description + remote_payload[:multiple][:exception_count_description] + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/results.rb b/rspec-core/lib/rspec/core/parallel/results.rb new file mode 100644 index 000000000..a18b9701b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/results.rb @@ -0,0 +1,40 @@ +module RSpec + module Core + module Parallel + # The results workers report, kept from the moment they arrive -- so + # failure counting and `.rspec_status` persistence never wait for + # presentation -- and copied onto the examples when their notifications + # are delivered (or at the end of the run for whatever was not). + # @private + class Results + attr_reader :failed_count + + def initialize + @payloads = {} + @failed_count = 0 + end + + def record(example, payload) + @payloads[example] = payload + @failed_count += 1 if payload[:status] == :failed + end + + def apply(example) + payload = @payloads.delete(example) + return unless payload + + example.assign_execution_result( + payload.merge( + :exception => RemoteException.rebuild(payload[:exception]), + :pending_exception => RemoteException.rebuild(payload[:pending_exception]) + ) + ) + end + + def apply_all + @payloads.keys.each { |example| apply(example) } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb new file mode 100644 index 000000000..a67f641c4 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -0,0 +1,283 @@ +module RSpec + module Core + module Parallel + # Replays worker notifications onto the coordinator's reporter. + # + # Formatters that follow example group boundaries (documentation, HTML, + # ...) need every notification of a top-level group to arrive between + # that group's `example_group_started` and `example_group_finished`. + # With several workers running at once this holds only if their streams + # are interleaved at group boundaries, so when such a formatter is + # registered one in-flight group at a time is "live" and delivered as it + # happens while the others are queued and replayed whole once they + # finish -- the output then reads like a serial run of the groups in + # completion order. Formatters that do not care about group boundaries + # (progress, JSON, failures, ...) get every notification immediately. + # + # Liveness is bounded in both modes: when the live group has been silent + # for {SILENCE_BOUND} seconds, when a finished group has been waiting + # for {WAIT_BOUND} seconds behind a live group that keeps producing + # output, or when more than {QUEUE_BOUND} notifications are queued, + # waiting notifications are delivered right away. Formatters then see + # the live group closed and re-opened around them (its headings + # repeat), which is the price of showing results promptly. + # + # Only presentation is sequenced: results, failure counts and example + # statuses are recorded by the coordinator as they arrive. + # @private + class Sequencer + SILENCE_BOUND = 2.0 + WAIT_BOUND = 10.0 + QUEUE_BOUND = 5_000 + CONTEXT_FREE = [:message, :deprecation, :custom].freeze + + Event = Struct.new(:kind, :target, :payload, :bytes) + + # The notifications of one top-level group run by one worker. + # @private + class Stream + attr_reader :worker, :events, :open_groups + attr_accessor :depth, :complete, :completed_at + + def initialize(worker) + @worker = worker + @events = [] + @open_groups = [] + @depth = 0 + @complete = false + @completed_at = nil + end + end + + attr_reader :stats + + def initialize(reporter, options={}) + @reporter = reporter + @apply_result = options.fetch(:apply_result) + @clock = options.fetch(:clock) { Parallel.method(:monotonic_time) } + @sequenced = options.fetch(:sequenced) { self.class.group_aware_listeners?(reporter) } + @streams = {} + @completed = [] + @live = nil + @visible = nil + @visible_groups = [] + @last_delivery_at = @clock.call + @stats = { :queued => 0, :peak_queued => 0, :queued_bytes => 0, :peak_queued_bytes => 0, :flushes => 0 } + end + + # @return [Boolean] whether a registered formatter reacts to group + # boundaries. `BaseFormatter#example_group_started` only records the + # current group and RSpec's profiler keeps its own per-group tally, + # so neither needs sequencing. + def self.group_aware_listeners?(reporter) + [:example_group_started, :example_group_finished].any? do |event| + reporter.registered_listeners(event).any? do |listener| + !listener.is_a?(Profiler) && listener.method(event).owner != Formatters::BaseFormatter + end + end + end + + def sequenced? + @sequenced + end + + def group_started(worker, group, bytes=0) + push(worker, Event.new(:group_started, group, nil, bytes)) { |stream| stream.depth += 1 } + end + + def group_finished(worker, group, duration, bytes=0) + push(worker, Event.new(:group_finished, group, duration, bytes)) do |stream| + stream.depth -= 1 + stream.complete = true if stream.depth <= 0 + end + end + + def example_started(worker, example, bytes=0) + push(worker, Event.new(:example_started, example, nil, bytes)) + end + + def example_finished(worker, example, bytes=0) + push(worker, Event.new(:example_finished, example, nil, bytes)) + end + + def message(worker, message, bytes=0) + push(worker, Event.new(:message, nil, message, bytes)) + end + + def deprecation(worker, hash, bytes=0) + push(worker, Event.new(:deprecation, nil, hash, bytes)) + end + + def custom(worker, event, options, bytes=0) + push(worker, Event.new(:custom, event, options, bytes)) + end + + # The worker died (or was stopped) in the middle of a group: deliver + # what it reported and balance the group notifications formatters saw. + def abandon(worker) + stream = @streams.delete(worker) + return unless stream + + replay(stream) + close_visible if @visible.equal?(stream) + @live = nil if @live.equal?(stream) + promote + end + + # Called periodically by the coordinator. + def tick(now=@clock.call) + return unless @sequenced + + if @stats[:queued] >= QUEUE_BOUND + flush_everything + elsif @completed.any? && (now - @last_delivery_at >= SILENCE_BOUND || now - @completed.first.completed_at >= WAIT_BOUND) + flush_completed + end + end + + # Delivers everything still queued and closes any open group. + def finish + flush_everything + @streams.keys.each { |worker| abandon(worker) } + close_visible + end + + private + + def push(worker, event) + stream = @streams[worker] + + # Messages sent outside any group (suite hooks) are not sequenced. + if CONTEXT_FREE.include?(event.kind) && (stream.nil? || stream.depth <= 0) + return deliver(Stream.new(worker), event) + end + + stream ||= (@streams[worker] = Stream.new(worker)) + yield stream if block_given? + + if live?(stream) + deliver(stream, event) + else + enqueue(stream, event) + end + + complete(stream) if stream.complete + end + + def live?(stream) + return true unless @sequenced + + @live = stream if @live.nil? + @live.equal?(stream) + end + + def complete(stream) + @streams.delete(stream.worker) + + if stream.events.empty? + @live = nil if @live.equal?(stream) + promote + else + stream.completed_at = @clock.call + @completed << stream + end + end + + def enqueue(stream, event) + stream.events << event + @stats[:queued] += 1 + @stats[:queued_bytes] += event.bytes + @stats[:peak_queued] = [@stats[:peak_queued], @stats[:queued]].max + @stats[:peak_queued_bytes] = [@stats[:peak_queued_bytes], @stats[:queued_bytes]].max + end + + def deliver(stream, event) + if CONTEXT_FREE.include?(event.kind) + deliver_context_free(event) + else + switch_to(stream) unless @visible.equal?(stream) + deliver_in_context(stream, event) + end + + @last_delivery_at = @clock.call + end + + def deliver_in_context(stream, event) + case event.kind + when :group_started + @reporter.example_group_started(event.target) + stream.open_groups.push(event.target) + @visible_groups.push(event.target) + when :group_finished + @reporter.example_group_finished(event.target) + stream.open_groups.pop + @visible_groups.pop + @reporter.record_example_group_duration(event.target, event.payload) if event.payload + when :example_started + @reporter.example_started(event.target) + when :example_finished + @apply_result.call(event.target) + @reporter.example_finished(event.target) + @reporter.__send__(:"example_#{event.target.execution_result.status}", event.target) + end + end + + def deliver_context_free(event) + case event.kind + when :message then @reporter.message(event.payload) + when :deprecation then @reporter.deprecation(event.payload) + when :custom then @reporter.publish(event.target, event.payload) + end + end + + def replay(stream) + until stream.events.empty? + event = stream.events.shift + @stats[:queued] -= 1 + @stats[:queued_bytes] -= event.bytes + deliver(stream, event) + end + end + + def switch_to(stream) + close_visible + stream.open_groups.each { |group| @reporter.example_group_started(group) } + @visible_groups = stream.open_groups.dup + @visible = stream + end + + def close_visible + @visible_groups.reverse_each { |group| @reporter.example_group_finished(group) } + @visible_groups = [] + @visible = nil + end + + # The live group finished: show the groups that finished meanwhile, + # then let the longest-waiting in-flight group go live. + def promote + flush_completed + return unless @live.nil? + + waiting = @streams.values.select { |stream| stream.events.any? } + return if waiting.empty? + + @live = waiting.first + replay(@live) + end + + def flush_completed + @stats[:flushes] += 1 if @completed.any? + while (stream = @completed.shift) + replay(stream) + end + end + + def flush_everything + flush_completed + @stats[:flushes] += 1 if @streams.values.any? { |stream| stream.events.any? } + @streams.values.each { |stream| replay(stream) } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/suite_index.rb b/rspec-core/lib/rspec/core/parallel/suite_index.rb new file mode 100644 index 000000000..ed529319b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/suite_index.rb @@ -0,0 +1,32 @@ +module RSpec + module Core + module Parallel + # Finds the examples and example groups of the loaded suite by the ids + # workers report them under. + # @private + class SuiteIndex + def initialize(world) + @world = world + end + + def example(id) + examples_by_id[id] + end + + def group(id) + groups_by_id[id] + end + + private + + def examples_by_id + @examples_by_id ||= @world.all_examples.each_with_object({}) { |example, hash| hash[example.id] = example } + end + + def groups_by_id + @groups_by_id ||= @world.all_example_groups.each_with_object({}) { |group, hash| hash[group.id] = group } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb new file mode 100644 index 000000000..ab0df9492 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -0,0 +1,126 @@ +module RSpec + module Core + module Parallel + # Runs in a forked worker process: executes the example groups the + # coordinator dispatches over the jobs channel and streams results back + # over the results channel. Suite hooks run once per worker, around all + # the groups it executes. + # + # A worker that has finished its work but does not exit (a non-daemon + # thread in `at_exit`, a browser that will not quit, ...) is aborted by + # its own watchdog thread after `parallel_shutdown_timeout` seconds, + # with a dump of every thread's backtrace on stderr, because the process + # holding the stacks is the only one that can print them. The coordinator + # independently bounds the wait from its side. + # @private + class Worker + attr_reader :number + + def initialize(number, example_groups, jobs, results, options={}) + @number = number + @example_groups = example_groups + @jobs = jobs + @results = results + @configuration = options.fetch(:configuration) { RSpec.configuration } + @world = options.fetch(:world) { RSpec.world } + @stderr = options[:stderr] + @exit = options.fetch(:exit) { lambda { |status| exit(status) } } + @exit_now = options.fetch(:exit_now) { lambda { |status| exit!(status) } } + @trap = options.fetch(:trap) { lambda { |signal, &handler| trap(signal, &handler) } } + end + + # @return [String] the `TEST_ENV_NUMBER` value for this worker, following + # the convention established by parallel_tests: `""` for the first + # worker, then `"2"`, `"3"`, ... + def test_env_number + @number == 1 ? "" : @number.to_s + end + + # Never returns: ends the process when the coordinator has no more work. + def run + prepare + run_units + @exit.call(0) + rescue Errno::EPIPE, IOError + # The coordinator is gone; there is nobody left to report to. + @exit_now.call(1) + end + + # @private + def prepare + Runner.disable_autorun! + trap_interrupt + ENV['TEST_ENV_NUMBER'] = test_env_number + @configuration.parallel_worker_number = @number + @configuration.reset_reporter + @reporter = WorkerReporter.new(@configuration, @results, @jobs, @world) + @configuration.reporter = @reporter + end + + # Same contract as `Runner.handle_interrupt` -- stop between examples, + # force-quit on a second interrupt -- without the message, which the + # coordinator prints once for the run. + # @private + def trap_interrupt + world = @world + exit_now = @exit_now + @trap.call('INT') do + if world.wants_to_quit + exit_now.call(1) + else + world.wants_to_quit = true + end + end + end + + # @private + def run_units + @configuration.with_suite_hooks do + while (message = @jobs.receive) + break unless message.first == :run + + run_unit(message[1]) + end + + # Armed before `after(:suite)` hooks so that a hook that never + # returns is bounded too. + arm_watchdog + end + + @results.send([:exiting]) + end + + # @private + def run_unit(index) + passed = @example_groups.fetch(index).run(@reporter) + @results.send([:unit_finished, index, passed ? true : false, + !!@world.wants_to_quit, !!@world.non_example_failure]) + end + + # @private + def arm_watchdog + timeout = @configuration.parallel_shutdown_timeout + thread = Thread.new do + sleep timeout + dump_threads(timeout) + @exit_now.call(1) + end + thread.report_on_exception = false + thread + end + + # @private + def dump_threads(timeout) + stderr = @stderr || $stderr + stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) did not exit within #{timeout} " \ + "seconds of finishing its last example group; aborting it. Threads at this point:" + Thread.list.each do |thread| + stderr.puts " #{thread.inspect}#{' (this watchdog)' if thread.equal?(Thread.current)}" + Array(thread.backtrace).each { |line| stderr.puts " #{line}" } + end + stderr.flush + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb new file mode 100644 index 000000000..25311c87c --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -0,0 +1,73 @@ +module RSpec + module Core + module Parallel + # Coordinator-side bookkeeping for one worker process. + # @private + class WorkerHandle + attr_reader :number, :jobs, :results, :child_jobs, :child_results + attr_accessor :pid, :current_unit, :state, :deadline, :escalation, :exit_status, :quit_sent, :exiting + + def initialize(number) + @number = number + @child_jobs, @jobs = Channel.pipe + @results, @child_results = Channel.pipe + @state = :running + @quit_sent = false + @exiting = false + end + + def close_parent_ends + @jobs.close + @results.close + end + + def close_child_ends + @child_jobs.close + @child_results.close + end + + def close_all + close_parent_ends + close_child_ends + end + + def idle? + @state == :running && @current_unit.nil? + end + + def busy? + @state == :running && !@current_unit.nil? + end + + def done? + @state == :exited || @state == :abandoned + end + + def description + "Parallel worker #{@number} (pid #{@pid})" + end + + # @return [Array(Integer, Process::Status), nil] like `waitpid2` with + # `WNOHANG`; a process reaped by someone else counts as exited + def reap(process) + process.waitpid2(@pid, Process::WNOHANG) + rescue Errno::ECHILD + [@pid, nil] + end + + def kill(process, signal_name) + process.kill(signal_name, @pid) if @pid + rescue Errno::ESRCH, Errno::EPERM + nil + end + + def self.describe_status(status) + return "already reaped" unless status + return "killed by SIG#{Signal.signame(status.termsig)}" if status.signaled? + + "exit status #{status.exitstatus}" + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb new file mode 100644 index 000000000..500628db3 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -0,0 +1,122 @@ +module RSpec + module Core + module Parallel + # The reporter installed in a worker process. Instead of notifying + # formatters it forwards every notification -- RSpec's own and custom + # ones published through `Reporter#publish` -- to the coordinator as + # `Marshal`-safe frames referring to examples and groups by id. The + # coordinator replays them onto its own reporter, so formatters run + # exclusively in the coordinating process and receive real `Example` + # and `ExampleGroup` objects. + # @private + class WorkerReporter < Reporter + FORWARDED_AS_PAYLOAD = [:message, :deprecation].freeze + DERIVED_BY_COORDINATOR = [:example_passed, :example_failed, :example_pending].freeze + RUN_LIFECYCLE = [:seed, :start, :stop, :start_dump, :dump_pending, :dump_failures, :deprecation_summary, + :dump_profile, :dump_summary, :close].freeze + + def initialize(configuration, results, jobs, world=RSpec.world) + super(configuration) + @results = results + @jobs = jobs + @world = world + @group_started_at = nil + end + + # @private + # Suite hook failures and the like: the coordinator rebuilds the + # exception and reports it through its own reporter, which keeps the + # run's exit code and "errors occurred outside of examples" count right. + def notify_non_example_exception(exception, context_description) + @world.non_example_failure = true + @results.send([:non_example_failure, RemoteException.payload_for(exception), context_description.to_s]) + end + + # @private + def notify(event, notification) + case event + when :example_group_started then forward_group_started(notification.group) + when :example_group_finished then forward_group_finished(notification.group) + when :example_started then @results.send([:example_started, notification.example.id]) + when :example_finished then forward_example_finished(notification.example) + when *DERIVED_BY_COORDINATOR, *RUN_LIFECYCLE then nil + when *FORWARDED_AS_PAYLOAD then forward(event, notification) + else forward_custom(event, notification) + end + end + + private + + def forward_group_started(group) + @group_started_at = Parallel.monotonic_time if group.top_level? + @results.send([:group_started, group.id]) + end + + def forward_group_finished(group) + duration = nil + if group.top_level? && @group_started_at + duration = Parallel.monotonic_time - @group_started_at + @group_started_at = nil + end + @results.send([:group_finished, group.id, duration]) + end + + def forward_example_finished(example) + @results.send([:example_finished, example.id, result_payload(example)]) + poll_for_quit + end + + # The coordinator asks workers to stop between examples (fail-fast, + # interrupt) by writing `[:quit]` on the jobs channel. Anything else + # is left for the worker's job loop. + def poll_for_quit + @jobs.drain.each do |message, _size| + if message.first == :quit + @world.wants_to_quit = true + else + @jobs.unread(message) + end + end + end + + def forward(event, notification) + if event == :message + @results.send([:message, Parallel.marshal_safe(notification.message, &:to_s)]) + else + hash = notification.to_h + @results.send([:deprecation, Parallel.marshal_safe(hash) { hash.transform_values { |v| v && v.to_s } }]) + end + end + + def forward_custom(event, notification) + options = notification.respond_to?(:to_h) ? notification.to_h : {} + Marshal.dump(options) + @results.send([:custom, event, options]) + rescue TypeError, ArgumentError, NoMethodError + @results.send([:message, "Parallel worker #{@configuration.parallel_worker_number} could not forward " \ + "the custom `#{event}` notification because its options cannot be marshaled: " \ + "#{options.inspect[0, 500]}"]) + end + + def result_payload(example) + result = example.execution_result + metadata = example.metadata + + { + :status => result.status, + :started_at => result.started_at, + :finished_at => result.finished_at, + :run_time => result.run_time, + :pending_message => Parallel.marshal_safe(result.pending_message, &:to_s), + :pending_fixed => result.pending_fixed, + :exception => RemoteException.payload_for(result.exception), + :pending_exception => RemoteException.payload_for(result.pending_exception), + :pending => Parallel.marshal_safe(metadata[:pending], &:to_s), + :skip => Parallel.marshal_safe(metadata[:skip], &:to_s), + :extra_failure_lines => Parallel.marshal_safe(metadata[:extra_failure_lines]) { |lines| Array(lines).map(&:to_s) } + } + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/profiler.rb b/rspec-core/lib/rspec/core/profiler.rb index 5e6527974..8f3697a87 100644 --- a/rspec-core/lib/rspec/core/profiler.rb +++ b/rspec-core/lib/rspec/core/profiler.rb @@ -29,6 +29,15 @@ def example_started(notification) group = notification.example.example_group.parent_groups.last @example_groups[group][:count] += 1 end + + # Used when the group ran in another process and its duration was + # measured there. + def record_group_duration(group, duration) + return unless group.top_level? + + @example_groups[group][:description] = group.top_level_description + @example_groups[group][:total_time] = duration + end end end end diff --git a/rspec-core/lib/rspec/core/reporter.rb b/rspec-core/lib/rspec/core/reporter.rb index d513f3f9f..2a6e97db4 100644 --- a/rspec-core/lib/rspec/core/reporter.rb +++ b/rspec-core/lib/rspec/core/reporter.rb @@ -156,6 +156,14 @@ def deprecation(hash) notify :deprecation, Notifications::DeprecationNotification.from_hash(hash) end + # @private + # Records the duration of a top-level example group that ran in another + # process (see `Parallel`), in place of the time observed here. + def record_example_group_duration(group, duration) + ensure_listeners_ready + @profiler.record_group_duration(group, duration) + end + # @private # Provides a way to notify of an exception that is not tied to any # particular example (such as an exception encountered in a :suite hook). diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index 16d07efea..ab1350c99 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -86,7 +86,10 @@ def run(err, out) setup(err, out) return @configuration.reporter.exit_early(exit_code) if RSpec.world.wants_to_quit - run_specs(@world.ordered_example_groups).tap do + example_groups = @world.ordered_example_groups + coordinator = Parallel.coordinator_for(self, example_groups) + + (coordinator ? coordinator.run : run_specs(example_groups)).tap do persist_example_statuses end end diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb new file mode 100644 index 000000000..58911faa0 --- /dev/null +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -0,0 +1,446 @@ +require 'support/aruba_support' +require 'json' + +RSpec.describe 'Parallel execution (--parallel)', :slow, :skip => !RSpec::Support::RubyFeatures.fork_supported? do + include_context "aruba support" + + # The coordinator refuses to nest inside another parallel worker, which is + # what this very process is when rspec-core's suite runs under `--parallel`. + # Aruba swaps in its own environment while a command runs, so the variable + # is removed from (or, for the fallback test, added to) that environment. + def run_rspec(cmd) + if @inside_another_parallel_worker + set_environment_variable("TEST_ENV_NUMBER", "2") + else + delete_environment_variable("TEST_ENV_NUMBER") + end + run_command(cmd) + end + + def write_fixed_suite + write_file_formatted 'spec/alpha_spec.rb', " + RSpec.describe 'Alpha' do + it('passes') { expect(1).to eq(1) } + it('fails') { expect(1).to eq(2) } + it('is pending') { pending 'not yet'; expect(1).to eq(2) } + it('is skipped') { skip 'later' } + context 'nested' do + it('fails with a cause') { begin; raise 'root'; rescue; raise ArgumentError, 'wrapped'; end } + end + end + " + + write_file_formatted 'spec/beta_spec.rb', " + RSpec.describe 'Beta' do + it('aggregates', :aggregate_failures) { expect(1).to eq(2); expect(2).to eq(3) } + it('passes too') { expect(true).to be(true) } + it('fails in an after hook') { } + after { |ex| raise 'in after' if ex.description == 'fails in an after hook' } + end + " + + write_file_formatted 'spec/gamma_spec.rb', " + RSpec.describe 'Gamma' do + 3.times { |i| it(\"passes \#{i}\") { expect(i).to be >= 0 } } + end + " + end + + # Timing differs between runs, and failures are numbered in the order they + # are reported, which for a parallel run is the order groups finish in. + # Frames below the spec files differ too because these specs run RSpec + # inside RSpec (the worker's stack goes through the coordinator's fork). + def normalize(output) + output. + gsub(/Finished in .*$/, 'Finished in X'). + gsub(/\d+\.\d+ seconds/, 'X seconds'). + gsub(/\d+(\.\d+)? seconds? average.*$/, 'X average'). + gsub(/\(FAILED - \d+\)/, '(FAILED - N)'). + gsub(/^(\s*)\d+(\.\d+)?\) /, '\\1N) '). + lines.reject { |line| line =~ /^\s*# / && line !~ %r{# \./spec/} }.join + end + + def group_blocks(documentation_output) + body = documentation_output.split(/^Failures:$|^Pending:/).first + body.split(/\n(?=\S)/).map(&:strip).reject(&:empty?).reject { |block| block.start_with?("Randomized") } + end + + it 'runs the groups across workers and reports one coherent run with the serial exit code' do + write_fixed_suite + + run_rspec "--parallel=2 --order defined" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("11 examples, 4 failures, 2 pending") + expect(last_cmd_stdout).to include("Alpha fails", "Alpha nested fails with a cause", "Beta aggregates", "Beta fails in an after hook") + expect(last_cmd_stdout).to include("Got 2 failures:", "--- Caused by: ---", "root") + expect(last_cmd_stdout.scan("Randomized with seed").size).to eq(0) + expect(last_cmd_stderr).to eq("") + end + + it 'gives each worker its number and a parallel_tests-style TEST_ENV_NUMBER, and runs suite hooks once per worker' do + write_file_formatted 'spec/env_spec.rb', " + RSpec.configure do |c| + c.before(:suite) { File.write(\"before_suite_\#{RSpec.configuration.parallel_worker_number}\", ENV['TEST_ENV_NUMBER'].inspect) } + c.after(:suite) { File.write(\"after_suite_\#{RSpec.configuration.parallel_worker_number}\", Process.pid) } + end + 3.times do |i| + RSpec.describe(\"G\#{i}\") do + it('records its worker') { File.write(\"example_\#{i}\", RSpec.configuration.parallel_worker_number.to_s) } + end + end + " + + test_env_number_before = ENV['TEST_ENV_NUMBER'] + run_rspec "--parallel=2 spec/env_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("3 examples, 0 failures") + cd '.' do + expect(File.read("before_suite_1")).to eq('""') + expect(File.read("before_suite_2")).to eq('"2"') + expect(File.exist?("after_suite_1")).to be true + expect(File.exist?("after_suite_2")).to be true + expect(Dir["example_*"].map { |f| File.read(f) }.sort).to all(match(/\A[12]\z/)) + end + expect(RSpec.configuration.parallel_worker_number).to be_nil + expect(ENV['TEST_ENV_NUMBER']).to eq(test_env_number_before) + end + + describe 'formatter parity with a serial run' do + before { write_fixed_suite } + + it 'produces the same progress output' do + run_rspec "--order defined --format progress" + serial = normalize(last_cmd_stdout) + run_rspec "--order defined --format progress --parallel=2" + parallel = normalize(last_cmd_stdout) + + serial_dots, serial_rest = serial.split("\n", 2) + parallel_dots, parallel_rest = parallel.split("\n", 2) + expect(parallel_dots.chars.sort).to eq(serial_dots.chars.sort) + expect(parallel_rest.lines.sort).to eq(serial_rest.lines.sort) + end + + it 'produces the same documentation blocks for every group, with the serial order inside each group' do + run_rspec "--order defined --format documentation" + serial = normalize(last_cmd_stdout) + run_rspec "--order defined --format documentation --parallel=3" + parallel = normalize(last_cmd_stdout) + + expect(group_blocks(parallel)).to match_array(group_blocks(serial)) + expect(parallel.lines.sort).to eq(serial.lines.sort) + end + + it 'produces the same JSON' do + run_rspec "--order defined --format json --out serial.json" + run_rspec "--order defined --format json --out parallel.json --parallel=2" + + cd '.' do + serial, parallel = %w[serial.json parallel.json].map do |file| + json = JSON.parse(File.read(file)) + json.delete("summary_line") + json["summary"].delete("duration") + json["examples"].each do |e| + e.delete("run_time") + e["exception"]["backtrace"].select! { |frame| frame.start_with?("./spec/") } if e["exception"] + end + json["examples"] = json["examples"].sort_by { |e| e["id"] } + json + end + + expect(parallel).to eq(serial) + expect(parallel["examples"].select { |e| e["exception"] }.map { |e| e["exception"]["class"] }).to include( + "RSpec::Expectations::ExpectationNotMetError", "ArgumentError", "RSpec::Expectations::MultipleExpectationsNotMetError", "RuntimeError" + ) + end + end + + it 'produces the same failures list' do + run_rspec "--order defined --format failures" + serial = last_cmd_stdout + run_rspec "--order defined --format failures --parallel=2" + + expect(last_cmd_stdout.lines.sort).to eq(serial.lines.sort) + end + + it 'produces structurally equivalent HTML' do + run_rspec "--order defined --format html --out serial.html" + run_rspec "--order defined --format html --out parallel.html --parallel=2" + + cd '.' do + serial, parallel = %w[serial.html parallel.html].map { |file| File.read(file) } + expect(parallel.scan(/class="example (\w+)"/).sort).to eq(serial.scan(/class="example (\w+)"/).sort) + expect(parallel.scan(/]*>/).size).to eq(serial.scan(/]*>/).size) + expect(parallel.scan('').size).to eq(serial.scan('').size) + expect(parallel).to include("11 examples, 4 failures, 2 pending") + end + end + + it 'merges the profile with the durations measured in the workers' do + run_rspec "--order defined --profile 2 --parallel=2" + + expect(last_cmd_stdout).to include("Top 2 slowest examples", "Top 2 slowest example groups") + expect(last_cmd_stdout).to match(/\d+\.\d+ seconds average \(\d+\.\d+ seconds \/ \d+ examples?\) \.\/spec\/\w+_spec\.rb:1/) + end + end + + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do + write_file_formatted 'spec/support/my_formatter.rb', " + class MyAppError < StandardError; end + + class MyFormatter + RSpec::Core::Formatters.register self, :example_group_started, :example_group_finished, :example_failed, :example_passed, :my_event, :message + + def initialize(output) + @output = output + @depth = 0 + end + + def example_group_started(notification) + @output.puts \"\#{' ' * @depth}> \#{notification.group.description} (\#{notification.group.metadata[:kind]})\" + @depth += 1 + end + + def example_group_finished(notification) + @depth -= 1 + @output.puts \"\#{' ' * @depth}< \#{notification.group.description}\" + end + + def example_passed(notification) + @output.puts \"\#{' ' * @depth}ok \#{notification.example.metadata[:flag].inspect}\" + end + + def example_failed(notification) + e = notification.exception + @output.puts \"\#{' ' * @depth}FAILED \#{e.class.name} app_error=\#{e.is_a?(MyAppError)} std=\#{e.is_a?(StandardError)} msg=\#{e.message}\" + @output.puts notification.fully_formatted(1) + end + + def my_event(notification) + @output.puts \"custom: \#{notification.payload.inspect}\" + end + + def message(notification) + @output.puts \"message: \#{notification.message}\" + end + end + " + + write_file_formatted 'spec/custom_spec.rb', " + RSpec.describe 'One', :kind => :unit do + it('passes', :flag => 7) { RSpec.configuration.reporter.publish(:my_event, :payload => { :from => 'worker' }) } + it('raises an app error') { raise MyAppError, 'app broke' } + it('publishes something that cannot be marshaled') { RSpec.configuration.reporter.publish(:my_event, :payload => proc {}) } + end + + RSpec.describe 'Two', :kind => :feature do + it('passes') { } + end + " + + run_rspec "--order defined -r ./spec/support/my_formatter --format MyFormatter --parallel=2 spec/custom_spec.rb" + + expect(last_cmd_stdout).to include( + "> One (unit)", + " ok 7", + " FAILED MyAppError app_error=true std=true msg=app broke", + "Failure/Error: it('raises an app error') { raise MyAppError, 'app broke' }", + "< One", + "> Two (feature)", + " ok nil", + "< Two", + "custom: {from: \"worker\"}", + "message: Parallel worker 1 could not forward the custom `my_event` notification because its options cannot be marshaled: {payload: # 'new_thing') } } + RSpec.describe('B') { it('deprecates too') { RSpec.deprecate('other_thing') } } + " + + run_rspec "--parallel=2 --deprecation-out deprecations.txt spec/deprecation_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("hello from worker 1", "hello from worker 2") + expect(last_cmd_stdout).to include("2 deprecations logged to deprecations.txt") + cd '.' do + deprecations = File.read("deprecations.txt") + expect(deprecations).to include("old_thing is deprecated. Use new_thing instead.", "other_thing is deprecated.") + end + end + + it 'reports suite hook failures from workers like a serial run does, with the error exit code' do + write_file_formatted 'spec/suite_hook_spec.rb', " + RSpec.configure do |c| + c.after(:suite) { raise 'cleanup broke' if RSpec.configuration.parallel_worker_number == 2 } + end + RSpec.describe('A') { it('passes') { } } + RSpec.describe('B') { it('passes') { } } + " + + run_rspec "--parallel=2 --error-exit-code 3 spec/suite_hook_spec.rb" + + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include("An error occurred in an `after(:suite)` hook.", "cleanup broke") + expect(last_cmd_stdout).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it 'does not replay output buffered before the fork into files written by formatters' do + write_fixed_suite + + run_rspec "--order defined --format documentation --out doc.txt --parallel=3 spec/gamma_spec.rb spec/beta_spec.rb" + + cd '.' do + expect(File.read("doc.txt").scan("Randomized with seed").size).to eq(0) + expect(File.read("doc.txt").scan(/^Gamma$/).size).to eq(1) + end + end + + it 'reports a worker that dies and fails the run with the error exit code' do + write_file_formatted 'spec/crash_spec.rb', " + RSpec.describe 'Crashing' do + it('finishes') { } + it('dies') { exit!(7) } + it('never runs') { } + end + RSpec.describe('Fine') { it('passes') { } } + " + + run_rspec "--order defined --parallel=2 --error-exit-code 3 spec/crash_spec.rb" + + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to match(/Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running Crashing/) + expect(last_cmd_stdout).to include("0 failures") + end + + describe 'a worker that finishes but does not exit' do + def write_hanging_suite(extra="") + write_file_formatted 'spec/hang_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.after(:suite) do + if RSpec.configuration.parallel_worker_number == 2 + $stderr = File.open('worker_stderr.txt', 'w') + #{extra} + thread = Thread.new { sleep } + at_exit { thread.join } + end + end + end + RSpec.describe('H1') { it('passes') { } } + RSpec.describe('H2') { it('passes') { } } + " + end + + def pids_in(output) + output.scan(/\(pid (\d+)\)/).flatten.map(&:to_i).uniq + end + + def alive?(pid) + Process.kill(0, pid) + Process.waitpid(pid, Process::WNOHANG).nil? + rescue Errno::ESRCH, Errno::ECHILD + false + end + + it 'is aborted by its own watchdog with a thread dump, within the shutdown timeout, and fails the run' do + # Leaves the worker's watchdog plenty of room to fire before the + # coordinator steps in, even on a loaded machine. + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_hanging_suite + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3 spec/hang_spec.rb" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + worker_stderr = cd('.') { File.read('worker_stderr.txt') } + + expect(last_cmd_exit_status).to eq(3) + expect(elapsed).to be < 10 + expect(last_cmd_stdout).not_to include("sending SIGTERM") + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/) + expect(worker_stderr).to include("did not exit within 0.5 seconds of finishing its last example group", "Threads at this point:", "Thread#join") + expect(pids_in(last_cmd_stdout)).not_to be_empty + pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } + end + + it 'is terminated, then killed, by the coordinator when its own watchdog cannot fire' do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 0.1) + # Forked workers inherit this stub, which stands in for a worker whose + # watchdog thread never gets to run (native code holding the GVL, ...). + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:arm_watchdog) + write_hanging_suite("trap('TERM') { }") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3 spec/hang_spec.rb" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + + expect(last_cmd_exit_status).to eq(3) + expect(elapsed).to be < 5 + expect(last_cmd_stdout).to include( + "did not exit within 0.5 seconds of finishing its last example group", + "sending SIGTERM", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } + end + end + + it 'falls back to a serial run with a warning inside another parallel worker' do + write_file_formatted 'spec/a_spec.rb', " + RSpec.describe('A') { it('passes') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('B') { it('passes') { } } + " + + @inside_another_parallel_worker = true + run_rspec "--parallel=2 spec/a_spec.rb" + + expect(last_cmd_exit_status).to eq(0) + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("Ignoring `--parallel`: this process is already a worker of a parallel test run") + end +end diff --git a/rspec-core/spec/rspec/core/configuration_spec.rb b/rspec-core/spec/rspec/core/configuration_spec.rb index 4dbad52a0..99df572a8 100644 --- a/rspec-core/spec/rspec/core/configuration_spec.rb +++ b/rspec-core/spec/rspec/core/configuration_spec.rb @@ -35,6 +35,62 @@ module RSpec::Core end end + describe "#parallel_workers" do + it "defaults to `nil`, meaning a serial run" do + expect(config.parallel_workers).to be_nil + end + + it "accepts a positive integer, `:auto` or `nil`" do + config.parallel_workers = 4 + expect(config.parallel_workers).to eq(4) + config.parallel_workers = :auto + expect(config.parallel_workers).to eq(:auto) + config.parallel_workers = nil + expect(config.parallel_workers).to be_nil + end + + it "warns and runs serially for anything else" do + [0, -1, "4", 2.5, true].each do |value| + expect(RSpec).to receive(:warning).with(a_string_including("Cannot set `RSpec.configuration.parallel_workers` to `#{value.inspect}`")) + config.parallel_workers = value + expect(config.parallel_workers).to be_nil + end + end + + it "prefers the value forced from the command line" do + config.parallel_workers = 2 + config.force(:parallel_workers => :auto) + expect(config.parallel_workers).to eq(:auto) + end + end + + describe "#parallel_shutdown_timeout" do + it "defaults to 30 seconds" do + expect(config.parallel_shutdown_timeout).to eq(30) + end + + it "can be changed" do + config.parallel_shutdown_timeout = 2.5 + expect(config.parallel_shutdown_timeout).to eq(2.5) + end + end + + describe "#parallel_worker_number" do + it "is nil until the parallel runner sets it in a worker" do + expect(config.parallel_worker_number).to be_nil + config.parallel_worker_number = 3 + expect(config.parallel_worker_number).to eq(3) + end + end + + describe "#reporter=" do + it "replaces the reporter the configuration hands out" do + reporter = Reporter.new(config) + config.reporter = reporter + expect(config.reporter).to be(reporter) + end + end + describe "#fail_fast=" do context 'when true' do it 'is set to true' do diff --git a/rspec-core/spec/rspec/core/example_spec.rb b/rspec-core/spec/rspec/core/example_spec.rb index 213d370b0..dcd030c9b 100644 --- a/rspec-core/spec/rspec/core/example_spec.rb +++ b/rspec-core/spec/rspec/core/example_spec.rb @@ -972,4 +972,42 @@ def expect_pending_result(example) expect(example.reporter).to be reporter end end + + describe "#assign_execution_result" do + let(:example) { RSpec.describe("group").example("example", :pending => "declared") { } } + + it "copies the result of running the example elsewhere onto the example" do + started_at = Time.now + finished_at = started_at + 1 + error = RuntimeError.new("remote failure") + + example.assign_execution_result( + :status => :failed, :started_at => started_at, :finished_at => finished_at, :run_time => 1.0, + :pending_message => nil, :pending_fixed => false, :exception => error, :pending_exception => nil, + :pending => nil, :skip => true, :extra_failure_lines => ["more"] + ) + + expect(example.execution_result).to have_attributes( + :status => :failed, :started_at => started_at, :finished_at => finished_at, :run_time => 1.0, + :pending_message => nil, :pending_fixed => false, :exception => error, :pending_exception => nil + ) + expect(example.exception).to be(error) + expect(example.metadata).to include(:pending => "declared", :skip => true, :extra_failure_lines => ["more"]) + expect(RSpec::Core::Notifications::ExampleNotification.for(example)).to be_a(RSpec::Core::Notifications::FailedExampleNotification) + end + + it "marks pending results so that notifications are built like after a local run" do + pending_error = RuntimeError.new("still broken") + + example.assign_execution_result( + :status => :pending, :pending_message => "declared", :pending_fixed => false, + :pending_exception => pending_error, :pending => true + ) + + expect(example.exception).to be_nil + expect(example.execution_result.pending_exception).to be(pending_error) + expect(RSpec::Core::Notifications::ExampleNotification.for(example)).to be_a(RSpec::Core::Notifications::FailedExampleNotification) + expect(example.execution_result).not_to be_example_skipped + end + end end diff --git a/rspec-core/spec/rspec/core/option_parser_spec.rb b/rspec-core/spec/rspec/core/option_parser_spec.rb index af3edf4ec..0d598d9b8 100644 --- a/rspec-core/spec/rspec/core/option_parser_spec.rb +++ b/rspec-core/spec/rspec/core/option_parser_spec.rb @@ -335,6 +335,26 @@ module RSpec::Core end end + describe "--parallel" do + it "requests one worker per processor without an argument" do + expect(Parser.parse(%w[ --parallel ])[:parallel_workers]).to eq(:auto) + end + + it "requests the given number of workers" do + expect(Parser.parse(%w[ --parallel=4 ])[:parallel_workers]).to eq(4) + end + + it "does not swallow a following path as its argument" do + options = Parser.parse(%w[ --parallel spec/models ]) + expect(options[:parallel_workers]).to eq(:auto) + expect(options[:files_or_directories_to_run]).to eq(%w[ spec/models ]) + end + + it "rejects a non-numeric count" do + expect { Parser.parse(%w[ --parallel=lots ]) }.to raise_error(OptionParser::InvalidArgument) + end + end + describe "--bisect" do it "sets the `:bisect` option" do options = Parser.parse(%w[ --bisect ]) diff --git a/rspec-core/spec/rspec/core/parallel/channel_spec.rb b/rspec-core/spec/rspec/core/parallel/channel_spec.rb new file mode 100644 index 000000000..606c5cf98 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/channel_spec.rb @@ -0,0 +1,111 @@ +module RSpec::Core::Parallel + RSpec.describe Channel do + let(:ends) { Channel.pipe } + let(:reader) { ends.first } + let(:writer) { ends.last } + + after do + reader.close + writer.close + end + + it "round-trips objects as length-prefixed Marshal frames" do + writer.send([:example_finished, "./spec/a_spec.rb[1:1]", { :status => :passed, :run_time => 0.5 }]) + + messages = reader.drain + expect(messages.size).to eq(1) + message, bytes = messages.first + expect(message).to eq([:example_finished, "./spec/a_spec.rb[1:1]", { :status => :passed, :run_time => 0.5 }]) + expect(bytes).to eq(reader.bytes_read) + end + + it "carries binary data without re-encoding it" do + writer.send("caf\xC3\xA9".b) + writer.send("café") + + expect(reader.drain.map(&:first)).to eq(["caf\xC3\xA9".b, "café"]) + end + + it "decodes several frames that arrived at once" do + 3.times { |i| writer.send(i) } + + expect(reader.drain.map(&:first)).to eq([0, 1, 2]) + end + + it "keeps a partial frame buffered without blocking until the rest arrives" do + packet = Marshal.dump(:whole) + writer.io.write("#{packet.bytesize}\n#{packet[0, 3]}") + + expect(reader.drain).to eq([]) + expect(reader.eof?).to be false + + writer.io.write(packet[3..-1]) + expect(reader.drain.map(&:first)).to eq([:whole]) + end + + it "returns nothing when no data is available, without blocking" do + expect(reader.drain).to eq([]) + end + + it "reports EOF once the writer is closed and everything has been read" do + writer.send(:last) + writer.close + + expect(reader.drain.map(&:first)).to eq([:last]) + expect(reader.eof?).to be true + end + + it "raises a ProtocolError when the peer writes something that is not a frame" do + writer.io.write("garbage\n") + + expect { reader.drain }.to raise_error(Channel::ProtocolError, /Expected a frame length/) + end + + describe "#receive" do + it "blocks until a whole message is available" do + thread = Thread.new do + sleep 0.05 + writer.send(:late) + end + + expect(reader.receive).to eq(:late) + thread.join + end + + it "returns nil at EOF" do + writer.close + expect(reader.receive).to be_nil + end + + it "returns messages that were put back with #unread first" do + writer.send(:drained) + writer.send(:next) + drained = reader.drain.map(&:first) + expect(drained).to eq([:drained, :next]) + + reader.unread(:next) + writer.send(:after) + + expect(reader.receive).to eq(:next) + expect(reader.receive).to eq(:after) + end + + it "returns buffered messages before reading more" do + writer.send(:first) + writer.send(:second) + + expect(reader.receive).to eq(:first) + expect(reader.receive).to eq(:second) + end + end + + it "closes both ends idempotently" do + reader.close + writer.close + expect { reader.close }.not_to raise_error + expect { writer.close }.not_to raise_error + expect(reader).to be_closed + expect(writer).to be_closed + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb new file mode 100644 index 000000000..7ad58770a --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -0,0 +1,485 @@ +module RSpec::Core::Parallel + # Stands in for `Process` so that workers can be plain threads: `kill` + # ends the thread unless the "signal" is one the fake worker is said to + # survive, and `waitpid2` reports a status once the thread is dead. + class FakeProcess + Status = Struct.new(:exitstatus, :termsig) do + def success? + termsig.nil? && exitstatus == 0 + end + + def signaled? + !termsig.nil? + end + end + + attr_reader :signals, :warmups + + def initialize(survives=[]) + @threads = {} + @statuses = {} + @signals = [] + @survives = survives + @warmups = 0 + @next_pid = 40_000 + end + + def register(thread, &exit_status) + pid = (@next_pid += 1) + @threads[pid] = thread + @statuses[pid] = exit_status + pid + end + + def warmup + @warmups += 1 + end + + def waitpid2(pid, _flags) + thread = @threads.fetch(pid) + return nil if thread.alive? + + [pid, @statuses[pid].call || Status.new(1, nil)] + end + + def kill(name, pid) + @signals << [name, pid] + return if @survives.include?(name) + + @statuses[pid] = lambda { Status.new(nil, Signal.list[name]) } + @threads.fetch(pid).kill + end + end + + RSpec.describe Coordinator do + let(:config) { RSpec.configuration } + let(:world) { RSpec.world } + let(:output) { StringIO.new } + let(:runner) { RSpec::Core::Runner.new(RSpec::Core::ConfigurationOptions.new([]), config, world) } + let(:process) { FakeProcess.new } + let(:worker_stderr) { StringIO.new } + let(:worker_exits) { {} } + let(:worker_behaviours) { {} } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + before do + config.output_stream = output + config.add_formatter("progress", output) + end + + # Runs a real Worker on a thread, with its own configuration object so + # that it cannot replace the coordinator's reporter the way a forked + # worker replaces its inherited copy, and with its own copies of the + # pipe ends since the coordinator closes the child's ends after spawning + # (as it must after a fork). + def spawn(handle, _handles) + behaviour = worker_behaviours.fetch(handle.number, :normal) + jobs = Channel::Reader.new(handle.child_jobs.io.dup) + results = Channel::Writer.new(handle.child_results.io.dup) + exit_status = nil + + thread = Thread.new do + Thread.current.report_on_exception = false + case behaviour + when :garbage then stall_after { results.io.write("garbage\n") } + when :partial_frame then stall_after { results.io.write("999\nabc") } + when :closes_results then stall_after { results.close } + else + send_unknown_ids(results) if behaviour == :unknown_ids + run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } + end + end + + @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } + end + + def stall_after + yield + sleep + end + + def send_unknown_ids(results) + results.send([:example_started, "./nope_spec.rb[1:1]"]) + results.send([:example_started, "./nope_spec.rb[1:1]"]) + results.send([:group_started, "./nope_spec.rb[1]"]) + results.send([:weird]) + end + + def run_fake_worker(number, jobs, results, behaviour, &record_exit) + worker_config = RSpec::Core::Configuration.new + worker_config.parallel_shutdown_timeout = 60 + Thread.current[:fake_worker_world] = RSpec::Core::World.new(worker_config) + Worker.new(number, @units, jobs, results, + :configuration => worker_config, + :world => Thread.current[:fake_worker_world], + :stderr => worker_stderr, + :exit => lambda { |status| + record_exit.call(status) + sleep if behaviour == :never_exits + }, + :exit_now => record_exit, + :trap => lambda { |*| }).run + end + + def coordinator(units, worker_count=2, options={}) + @units = units + @process = options.fetch(:process, process) + Coordinator.new(runner, units, units, worker_count, + { :process => @process, :spawn => method(:spawn), :tick => 0.01, + :term_grace => 0.1, :kill_grace => 0.1, :watchdog_grace => 0.05 }.merge(options)) + end + + def passing_group(name) + RSpec.describe(name) { it("passes") {} } + end + + def failing_group(name) + RSpec.describe(name) { it("fails") { raise "boom" } } + end + + def messages + output.string + end + + it "runs every group across the workers, reports through the configured formatters and exits 0 when all pass" do + groups = %w[A B C D E].map { |name| passing_group(name) } + + exit_code = coordinator(groups, 3).run + + expect(exit_code).to eq(0) + expect(messages).to include("5 examples, 0 failures") + expect(messages).to match(/^\.\.\.\.\.$/) + expect(process.signals).to be_empty + expect(process.warmups).to eq(1) + expect(groups.flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + it "returns the failure exit code when an example fails" do + exit_code = coordinator([passing_group("A"), failing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(world.non_example_failure).to be_falsey + expect(messages).to include("2 examples, 1 failure", "boom") + end + + it "persists results on the real examples as they arrive so statuses survive even if replay is cut short" do + groups = [passing_group("A"), failing_group("B")] + coordinator(groups).run + + expect(groups[0].examples.first.execution_result.status).to eq(:passed) + expect(groups[1].examples.first.execution_result.status).to eq(:failed) + expect(groups[1].examples.first.exception.message).to eq("boom") + expect(groups[1].examples.first.exception).to be_a(RuntimeError) + end + + it "honours the configured exit codes" do + config.failure_exit_code = 7 + expect(coordinator([passing_group("A"), failing_group("B")]).run).to eq(7) + end + + describe "fail-fast" do + it "stops dispatching and asks busy workers to stop as soon as the limit is met, reporting a failure exit code" do + config.fail_fast = true + slow = RSpec.describe("Slow") do + it("waits") { sleep 0.3 } + it("never runs") {} + end + groups = [failing_group("Fails first"), slow, passing_group("Never dispatched")] + + exit_code = coordinator(groups, 2).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.execution_result.status).to be_nil + expect(world.wants_to_quit).to be true + end + end + + describe "a worker that exits while running a group" do + it "reports it, leaves the examples it had not finished unknown and fails the run with the error exit code" do + config.error_exit_code = 5 + crashing = RSpec.describe("Crashing") do + it("finishes") {} + it("dies") { raise NoMemoryError, "simulated worker death" } + it("never runs") {} + end + groups = [crashing, passing_group("B")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(5) + expect(messages).to match(/Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing/) + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, nil, nil]) + expect(messages).to include("3 examples, 0 failures") + end + end + + describe "a worker that does not exit after finishing" do + before { config.parallel_shutdown_timeout = 0.05 } + + it "is sent SIGTERM after the shutdown timeout and the run fails with the error exit code" do + worker_behaviours[2] = :never_exits + config.error_exit_code = 3 + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(3) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not exit within 0.05 seconds of finishing its last example group", "sending SIGTERM") + expect(messages).to include("2 examples, 0 failures") + end + + it "is sent SIGKILL when it survives SIGTERM" do + worker_behaviours[2] = :never_exits + process = FakeProcess.new(["TERM"]) + + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :process => process).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).to include("still alive 0.1 seconds after SIGTERM; sending SIGKILL") + end + + it "is given up on, so the run still ends, when it survives SIGKILL" do + worker_behaviours[2] = :never_exits + process = FakeProcess.new(%w[TERM KILL]) + started = RSpec::Core::Parallel.monotonic_time + + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :process => process).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).to include("still alive 0.1 seconds after SIGKILL; giving up on it") + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 5 + end + end + + describe "a worker that is asked to stop early but does not" do + it "is terminated after the shutdown timeout so a cancelled run still ends" do + config.parallel_shutdown_timeout = 0.1 + config.fail_fast = true + worker_behaviours[2] = :partial_frame + groups = [failing_group("Fails"), passing_group("Held by the stalled worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not stop within 0.1 seconds of being asked to stop early") + expect(messages).to include("1 example, 1 failure") + end + end + + it "terminates a worker that sends data it cannot decode" do + worker_behaviours[2] = :garbage + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("sent data the coordinator could not decode (RSpec::Core::Parallel::Channel::ProtocolError") + end + + it "warns once about ids it does not know and about unexpected frames" do + worker_behaviours[2] = :unknown_ids + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages.scan("./nope_spec.rb[1:1], which was not defined when the suite was loaded").size).to eq(1) + expect(messages.scan("./nope_spec.rb[1], which was not defined").size).to eq(1) + expect(messages).to include("sent an unexpected :weird frame") + expect(messages).to include("2 examples, 0 failures") + end + + it "reports the groups that never ran when no worker remains" do + crashing = RSpec.describe("Crashing") { it("dies") { raise NoMemoryError, "simulated worker death" } } + groups = [crashing, RSpec.describe("Also crashing") { it("dies") { raise NoMemoryError, "simulated worker death" } }, passing_group("C"), passing_group("D")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("2 example groups did not run because no worker remained to run them") + end + + it "stops dispatching when the world wants to quit, as on Ctrl-C" do + groups = [RSpec.describe("Interrupting") { it("interrupts") { RSpec.world.wants_to_quit = true } }, passing_group("B"), passing_group("C")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups, 1).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.execution_result.status).to be_nil + end + + it "reports a suite hook failure in a worker as an error of the run, counted in the summary" do + failing = RSpec.describe("Reports a suite failure") do + it("fails the suite") do + error = RuntimeError.new("after suite broke") + error.set_backtrace(["./spec/support/hooks.rb:7"]) + RSpec.configuration.reporter.notify_non_example_exception(error, "An error occurred in an `after(:suite)` hook.") + end + end + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + config.error_exit_code = 9 + + exit_code = coordinator([failing, passing_group("B")], 1).run + + expect(exit_code).to eq(9) + expect(world.non_example_failure).to be true + expect(messages).to include("An error occurred in an `after(:suite)` hook.", "after suite broke", "./spec/support/hooks.rb:7") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "lets a formatter's own exception surface instead of blaming the worker" do + broken = Class.new do + RSpec::Core::Formatters.register self, :example_passed + def initialize(_output); end + + def example_passed(_notification) + raise ArgumentError, "formatter bug" + end + end + config.add_formatter(broken, output) + + expect { coordinator([passing_group("A"), passing_group("B")]).run }.to raise_error(ArgumentError, "formatter bug") + expect(messages).not_to include("could not decode") + end + + it "flushes the formatters' and configuration's streams before forking so workers do not replay buffered output" do + file_output = StringIO.new + allow(file_output).to receive(:flush).and_call_original + config.add_formatter(RSpec::Core::Formatters::ProgressFormatter.new(file_output)) + + coordinator([passing_group("A"), passing_group("B")]).run + + expect(file_output).to have_received(:flush).at_least(:once) + expect(file_output.string).to include("2 examples, 0 failures") + end + + it "copes with workers that could not be spawned when the coordinator fails" do + failing_spawn = lambda do |handle, handles| + raise Errno::EAGAIN, "fork" if handle.number == 2 + + spawn(handle, handles) + end + + expect { coordinator([passing_group("A"), passing_group("B")], 2, :spawn => failing_spawn).run }.to raise_error(Errno::EAGAIN) + expect(process.signals.map(&:first)).to eq(["TERM"]) + end + + it "treats a non-example failure reported by a worker as one of the run" do + failing = RSpec.describe("Reports a suite failure") do + it("fails the suite") { RSpec.configuration.reporter.notify_non_example_exception(RuntimeError.new("suite"), "in a hook") } + end + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + config.error_exit_code = 9 + + exit_code = coordinator([failing, passing_group("B"), passing_group("C")], 1).run + + expect(exit_code).to eq(9) + expect(world.non_example_failure).to be true + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + end + + it "stops giving groups to a worker whose own world wants to quit and lets the others finish the queue" do + quitting = RSpec.describe("Quits") do + it("sets the flag on the worker's world") { Thread.current[:fake_worker_world].wants_to_quit = true } + end + groups = [quitting] + %w[B C D E].map { |name| passing_group(name) } + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups, 2).run + + expect(exit_code).to eq(0) + expect(messages).to match(/Parallel worker \d \(pid \d+\) stopped after Quits .* because `RSpec.world.wants_to_quit` was set/) + expect(messages).to include("5 examples, 0 failures") + expect(groups.drop(1).flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + it "requeues a group when its worker cannot be reached and lets another worker run it" do + config.parallel_shutdown_timeout = 0.2 + spawn_with_closed_jobs = lambda do |handle, handles| + pid = spawn(handle, handles) + handle.jobs.io.close_write if handle.number == 2 + pid + end + groups = [passing_group("A"), passing_group("B"), passing_group("C")] + + exit_code = coordinator(groups, 2, :spawn => spawn_with_closed_jobs).run + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + end + + it "terminates the workers that are still running when the coordinator itself fails" do + worker_behaviours[1] = :partial_frame + worker_behaviours[2] = :partial_frame + instance = coordinator([passing_group("A"), passing_group("B")]) + allow(instance).to receive(:dispatch_work).and_raise("coordinator bug") + + expect { instance.run }.to raise_error("coordinator bug") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM]) + end + + it "gives up on live workers that survive both signals when the coordinator itself fails" do + worker_behaviours[1] = :partial_frame + worker_behaviours[2] = :partial_frame + process = FakeProcess.new(%w[TERM KILL]) + instance = coordinator([passing_group("A"), passing_group("B")], 2, :process => process) + allow(instance).to receive(:dispatch_work).and_raise("coordinator bug") + + expect { instance.run }.to raise_error("coordinator bug") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM KILL KILL]) + expect(messages).to include("survived SIGKILL; giving up on it") + end + + it "copes with a worker that is gone by the time it is signalled" do + worker_behaviours[2] = :never_exits + config.parallel_shutdown_timeout = 0.05 + allow(process).to receive(:kill).and_raise(Errno::ESRCH) + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("sending SIGTERM", "sending SIGKILL", "giving up on it") + end + + describe WorkerHandle do + it "owns both ends of its two pipes and can close the coordinator's or all of them" do + handle = WorkerHandle.new(1) + expect([handle.jobs, handle.results, handle.child_jobs, handle.child_results]).to all(satisfy { |channel| !channel.closed? }) + + handle.close_parent_ends + expect([handle.jobs, handle.results]).to all(be_closed) + expect([handle.child_jobs, handle.child_results]).to all(satisfy { |channel| !channel.closed? }) + + handle.close_all + expect([handle.jobs, handle.results, handle.child_jobs, handle.child_results]).to all(be_closed) + end + end + + it "treats a worker reaped by someone else as exited with an unknown status" do + allow(process).to receive(:waitpid2).and_raise(Errno::ECHILD) + config.parallel_shutdown_timeout = 0.05 + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("exited unexpectedly (already reaped)") + end + + it "bounds the wait for a worker whose results pipe closed while it was running" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :closes_results + + coordinator([passing_group("A"), passing_group("B")]).run + + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("stopped reporting 0.05 seconds ago") + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb new file mode 100644 index 000000000..c631e24e7 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb @@ -0,0 +1,242 @@ +module RSpec::Core::Parallel + RSpec.describe RemoteException do + include FormatterSupport + + def round_trip(exception) + payload = Marshal.load(Marshal.dump(RemoteException.payload_for(exception))) + RemoteException.rebuild(payload) + end + + def raise_with_cause + begin + raise "the cause" + rescue + raise ArgumentError, "the effect" + end + rescue ArgumentError => e + e + end + + it "returns nil for nil" do + expect(RemoteException.payload_for(nil)).to be_nil + expect(RemoteException.rebuild(nil)).to be_nil + end + + it "rebuilds an exception whose class reports the original name and is a subclass of the real class" do + original = begin + raise ArgumentError, "bad arg" + rescue ArgumentError => e + e + end + + rebuilt = round_trip(original) + + expect(rebuilt).to be_a(ArgumentError) + expect(rebuilt.class).not_to equal(ArgumentError) + expect(rebuilt.class.name).to eq("ArgumentError") + expect(rebuilt.class.to_s).to eq("ArgumentError") + expect(rebuilt.class.inspect).to eq("ArgumentError") + expect(rebuilt.message).to eq("bad arg") + expect(rebuilt.to_s).to eq("bad arg") + expect(rebuilt.inspect).to eq("#") + expect(rebuilt.backtrace).to eq(original.backtrace) + expect(rebuilt.cause).to be_nil + end + + it "reuses one generated class per original class" do + a = round_trip(ArgumentError.new("a")) + b = round_trip(ArgumentError.new("b")) + + expect(a.class).to equal(b.class) + end + + it "rebuilds the cause chain" do + rebuilt = round_trip(raise_with_cause) + + expect(rebuilt.cause).to be_a(RuntimeError) + expect(rebuilt.cause.message).to eq("the cause") + expect(rebuilt.cause.cause).to be_nil + end + + it "bounds the depth of the cause chain it ships" do + exception = (1..(RemoteException::MAX_CAUSE_DEPTH + 5)).inject(nil) do |cause, i| + error = RuntimeError.new("level #{i}") + error.instance_variable_set(:@__cause, cause) + error.define_singleton_method(:cause) { @__cause } + error + end + + payload = RemoteException.payload_for(exception) + depth = 0 + while payload + depth += 1 + payload = payload[:cause] + end + expect(depth).to eq(RemoteException::MAX_CAUSE_DEPTH + 1) + end + + it "falls back to a StandardError stand-in named like a class that does not exist here" do + original = Class.new(StandardError) { def self.name = "Only::InWorker" }.new("gone") + rebuilt = round_trip(original) + + expect(rebuilt).to be_a(StandardError) + expect(rebuilt.class.name).to eq("Only::InWorker") + expect(rebuilt.message).to eq("gone") + end + + it "does not resolve names that are not constant paths or not exception classes" do + expect(RemoteException.resolve("String")).to be_nil + expect(RemoteException.resolve("not a constant")).to be_nil + expect(RemoteException.resolve("Kernel")).to be_nil + expect(RemoteException.resolve(nil)).to be_nil + expect(RemoteException.resolve("RSpec::Core::ExampleGroup")).to be_nil + expect(RemoteException.resolve("RSpec::Expectations::ExpectationNotMetError")).to eq(RSpec::Expectations::ExpectationNotMetError) + end + + it "rescues errors raised while resolving a constant" do + stub_const("Explosive", Module.new { def self.const_missing(_) = raise("boom") }) + expect(RemoteException.resolve("Explosive::Nope")).to be_nil + end + + it "preserves an anonymous error class as such" do + rebuilt = round_trip(Class.new(StandardError).new("anon")) + + expect(rebuilt.class.name).to be_nil + expect(rebuilt.class.to_s).to match(/# "b") + payload = RemoteException.payload_for(original) + expect(payload[:message]).to eq("A Needs::Arguments for which `exception.message.to_s` raises NoMethodError.") + + rebuilt = RemoteException.rebuild(payload) + expect(rebuilt).to be_a(klass) + expect(rebuilt.message).to eq("A Needs::Arguments for which `exception.message.to_s` raises NoMethodError.") + end + + it "ships the detailed message so that SyntaxError formatting matches", :if => SyntaxError.method_defined?(:detailed_message) do + original = begin + eval("}", binding, __FILE__, __LINE__) + rescue SyntaxError => e + e + end + + rebuilt = round_trip(original) + expect(rebuilt).to be_a(SyntaxError) + expect(rebuilt.detailed_message).to eq(original.detailed_message(:highlight => false)) + end + + it "falls back to the message when no detailed message was shipped" do + rebuilt = RemoteException.rebuild(:class_name => "RuntimeError", :message => "plain", :backtrace => nil, :cause => nil) + expect(rebuilt.detailed_message).to eq("plain") + expect(rebuilt.backtrace).to be_nil + end + + it "tolerates a detailed_message that raises" do + original = RuntimeError.new("ok") + def original.detailed_message(*) = raise("nope") + expect(RemoteException.payload_for(original)[:detailed_message]).to be_nil + end + + it "keeps `set_backtrace` working on the rebuilt exception so common backtrace truncation applies" do + rebuilt = round_trip(RuntimeError.new("x").tap { |e| e.set_backtrace(%w[a b c]) }) + copy = rebuilt.dup + copy.set_backtrace(%w[a]) + + expect(copy.backtrace).to eq(%w[a]) + expect(rebuilt.backtrace).to eq(%w[a b c]) + end + + context "with an exception that aggregates several failures" do + def aggregate_failures_error + RSpec::Expectations::FailureAggregator.new("checking things", :hide_backtrace => true).aggregate do + expect(1).to eq(2) + expect(3).to eq(4) + raise "boom" + end + rescue RSpec::Expectations::MultipleExpectationsNotMetError => e + e + end + + it "rebuilds the multiple exception interface" do + original = aggregate_failures_error + rebuilt = round_trip(original) + + expect(RSpec::Core::MultipleExceptionError::InterfaceTag === rebuilt).to be true + expect(rebuilt).to be_a(RSpec::Expectations::MultipleExpectationsNotMetError) + expect(rebuilt.all_exceptions.map(&:message)).to eq(original.all_exceptions.map(&:message)) + expect(rebuilt.failures.size).to eq(original.failures.size) + expect(rebuilt.other_errors.map { |e| e.class.name }).to eq(["RuntimeError"]) + expect(rebuilt.aggregation_metadata).to eq(original.aggregation_metadata) + expect(rebuilt.aggregation_block_label).to eq(original.aggregation_block_label) + expect(rebuilt.summary).to eq(original.summary) + expect(rebuilt.exception_count_description).to eq(original.exception_count_description) + end + + it "gives the stand-in the interface when the class does not exist here" do + payload = RemoteException.payload_for(RSpec::Core::MultipleExceptionError.new(RuntimeError.new("a"))) + payload[:class_name] = "Unknown::Aggregate" + + rebuilt = RemoteException.rebuild(payload) + expect(RSpec::Core::MultipleExceptionError::InterfaceTag === rebuilt).to be true + expect(rebuilt.all_exceptions.map(&:message)).to eq(["a"]) + end + + it "replaces aggregation metadata that cannot be marshaled with the backtrace flag only" do + original = RSpec::Core::MultipleExceptionError.new(RuntimeError.new("a")) + original.aggregation_metadata[:proc] = lambda {} + + payload = RemoteException.payload_for(original) + expect(payload[:multiple][:aggregation_metadata]).to eq(:hide_backtrace => true) + end + end + + describe "presenter parity" do + def formatted(exception) + example = new_example(:description => "the example") + example.metadata[:absolute_file_path] = __FILE__ + example.execution_result.status = :failed + example.execution_result.exception = exception + RSpec::Core::Formatters::ExceptionPresenter::Factory.new(example).build.fully_formatted(1) + end + + it "produces the same fully formatted failure for a plain exception with a cause" do + original = raise_with_cause + expect(formatted(round_trip(original))).to eq(formatted(original)) + end + + it "produces the same fully formatted failure for an expectation failure" do + original = begin + expect(1).to eq(2) + rescue RSpec::Expectations::ExpectationNotMetError => e + e + end + + expect(formatted(round_trip(original))).to eq(formatted(original)) + expect(formatted(original)).to include("Failure/Error") + expect(formatted(original)).not_to include("ExpectationNotMetError") + end + + it "produces the same fully formatted failure for aggregated failures" do + original = RSpec::Core::MultipleExceptionError.new( + RuntimeError.new("first").tap { |e| e.set_backtrace(["./spec/a_spec.rb:1", "./spec/a_spec.rb:9"]) }, + RSpec::Expectations::ExpectationNotMetError.new("second").tap { |e| e.set_backtrace(["./spec/a_spec.rb:2", "./spec/a_spec.rb:9"]) } + ) + original.set_backtrace(["./spec/a_spec.rb:9"]) + + rebuilt = round_trip(original) + expect(formatted(rebuilt)).to eq(formatted(original)) + expect(formatted(rebuilt)).to include("Got 1 failure and 1 other error") + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb new file mode 100644 index 000000000..35853d8cd --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -0,0 +1,360 @@ +module RSpec::Core::Parallel + class RecordingListener + attr_reader :events + + def initialize + @events = [] + end + + RSpec::Core::Reporter::RSPEC_NOTIFICATIONS.each do |name| + define_method(name) { |notification| @events << [name, label(notification)] } + end + + def my_event(notification) + @events << [:my_event, notification.to_h] + end + + private + + def label(notification) + if notification.respond_to?(:group) + notification.group.description + elsif notification.respond_to?(:example) + notification.example.description + elsif notification.respond_to?(:message) + notification.message + end + end + end + + RSpec.describe Sequencer do + let(:reporter) { RSpec::Core::Reporter.new(RSpec.configuration) } + let(:listener) { RecordingListener.new } + let(:applied) { [] } + let(:now) { [0.0] } + let(:clock) { lambda { now.first } } + + let(:g1) do + RSpec.describe("G1") do + it("a") {} + context("N") { it("b") {} } + end + end + let(:g2) { RSpec.describe("G2") { it("c") {} } } + let(:g3) { RSpec.describe("G3") { it("d") {} } } + + def ex(group, description) + (group.examples + group.children.flat_map(&:examples)).find { |e| e.description == description } + end + + def sequencer(sequenced) + reporter.register_listener(listener, *RSpec::Core::Reporter::RSPEC_NOTIFICATIONS, :my_event) + Sequencer.new(reporter, :apply_result => lambda { |example| applied << example }, :clock => clock, :sequenced => sequenced) + end + + def run_example(seq, worker, example, status=:passed) + example.execution_result.status = status + seq.example_started(worker, example) + seq.example_finished(worker, example) + end + + def run_group(seq, worker, group, duration=1.5) + seq.group_started(worker, group) + group.examples.each { |example| run_example(seq, worker, example) } + group.children.each { |child| run_group(seq, worker, child, nil) } + seq.group_finished(worker, group, duration) + end + + def events + listener.events + end + + describe "immediate mode (no formatter follows group boundaries)" do + it "delivers every notification as it arrives, re-emitting group boundaries so the current group is always right" do + seq = sequencer(false) + + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 1, ex(g1, "a")) + run_example(seq, 2, ex(g2, "c"), :failed) + seq.group_finished(2, g2, 0.5) + seq.group_finished(1, g1, 1.0) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:example_group_finished, "G1"], [:example_group_started, "G2"], + [:example_group_finished, "G2"], [:example_group_started, "G1"], + [:example_started, "a"], [:example_finished, "a"], [:example_passed, "a"], + [:example_group_finished, "G1"], [:example_group_started, "G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_failed, "c"], + [:example_group_finished, "G2"], + [:example_group_started, "G1"], + [:example_group_finished, "G1"] + ]) + expect(seq.stats[:peak_queued]).to eq(0) + expect(seq).not_to be_sequenced + end + + it "applies the result right before delivering example_finished" do + seq = sequencer(false) + seq.group_started(1, g1) + run_example(seq, 1, ex(g1, "a")) + + expect(applied).to eq([ex(g1, "a")]) + end + + it "records the worker-measured duration of top-level groups for the profiler" do + seq = sequencer(false) + expect(reporter).to receive(:record_example_group_duration).with(g1, 1.5) + expect(reporter).not_to receive(:record_example_group_duration).with(g1.children.first, anything) + + run_group(seq, 1, g1) + end + end + + describe "sequenced mode" do + it "streams the first group live and replays the others whole, in completion order, once it finishes" do + seq = sequencer(true) + + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.group_started(3, g3) + run_example(seq, 2, ex(g2, "c")) + run_example(seq, 1, ex(g1, "a")) + run_example(seq, 3, ex(g3, "d")) + seq.group_finished(3, g3, 0.1) + seq.group_finished(2, g2, 0.2) + expect(events.map(&:last)).not_to include("G2", "G3", "c", "d") + expect(seq.stats[:queued]).to eq(8) + expect(seq.stats[:peak_queued]).to eq(8) + + seq.group_started(1, g1.children.first) + run_example(seq, 1, ex(g1, "b")) + seq.group_finished(1, g1.children.first, nil) + seq.group_finished(1, g1, 0.3) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:example_started, "a"], [:example_finished, "a"], [:example_passed, "a"], + [:example_group_started, "N"], + [:example_started, "b"], [:example_finished, "b"], [:example_passed, "b"], + [:example_group_finished, "N"], + [:example_group_finished, "G1"], + [:example_group_started, "G3"], + [:example_started, "d"], [:example_finished, "d"], [:example_passed, "d"], + [:example_group_finished, "G3"], + [:example_group_started, "G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_passed, "c"], + [:example_group_finished, "G2"] + ]) + expect(seq.stats[:queued]).to eq(0) + expect(seq.stats[:flushes]).to eq(1) + expect(seq).to be_sequenced + end + + it "promotes the longest-waiting unfinished group when the live one finishes and streams it live from then on" do + seq = sequencer(true) + + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.group_started(3, g3) + seq.group_finished(1, g1, 0.1) + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"] + ]) + + run_example(seq, 3, ex(g3, "d")) + run_example(seq, 2, ex(g2, "c")) + expect(events.last).to eq([:example_passed, "c"]) + expect(events.map(&:last)).not_to include("d") + end + + it "treats a group with no queued notifications as finished when its final notification arrives live" do + seq = sequencer(true) + run_group(seq, 1, g2) + run_group(seq, 2, g3) + + expect(events.map(&:last)).to eq(%w[G2 c c c G2 G3 d d d G3]) + end + + it "flushes finished groups when the live group has been silent for #{Sequencer::SILENCE_BOUND} seconds" do + seq = sequencer(true) + seq.group_started(1, g1) + run_group(seq, 2, g2) + expect(events.map(&:last)).to eq(%w[G1]) + + now[0] = Sequencer::SILENCE_BOUND - 0.1 + seq.tick + expect(events.map(&:last)).to eq(%w[G1]) + + now[0] = Sequencer::SILENCE_BOUND + seq.tick + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c G2]) + + run_example(seq, 1, ex(g1, "a")) + expect(events.map(&:last)[7..-1]).to eq(%w[G1 a a a]) + expect(seq.stats[:flushes]).to eq(1) + end + + it "flushes finished groups once one has waited #{Sequencer::WAIT_BOUND} seconds behind a live group that keeps talking" do + seq = sequencer(true) + seq.group_started(1, g1) + run_group(seq, 2, g2) + + (1..Sequencer::WAIT_BOUND.to_i).each do |second| + now[0] = second.to_f + run_example(seq, 1, ex(g1, "a")) + seq.tick + break if events.map(&:last).include?("G2") + end + + expect(now.first).to eq(Sequencer::WAIT_BOUND) + expect(events.map(&:last)).to include("G2", "c") + end + + it "does nothing on tick while nothing finished is waiting" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + now[0] = 100.0 + + expect { seq.tick }.not_to(change { events }) + end + + it "falls back to delivering everything queued, finished or not, past #{Sequencer::QUEUE_BOUND} notifications" do + stub_const("RSpec::Core::Parallel::Sequencer::QUEUE_BOUND", 3) + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 2, ex(g2, "c")) + expect(events.map(&:last)).to eq(%w[G1]) + + seq.tick + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c]) + expect(seq.stats[:queued]).to eq(0) + + seq.group_finished(2, g2, 0.1) + expect(seq.stats[:queued]).to eq(1) + end + + it "delivers messages sent outside any group immediately, without touching the open group" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.message(2, "from a suite hook") + seq.deprecation(2, :message => "old", :deprecated => "x", :replacement => nil, :call_site => nil) + seq.custom(2, :my_event, :a => 1) + + expect(events).to eq([ + [:example_group_started, "G1"], + [:message, "from a suite hook"], + [:deprecation, "old"], + [:my_event, { :a => 1 }] + ]) + end + + it "queues messages sent from inside a queued group with that group" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.message(2, "inside G2") + expect(events.map(&:last)).to eq(%w[G1]) + + seq.group_finished(2, g2, 0.1) + seq.group_finished(1, g1, 0.1) + expect(events.map(&:last)).to eq(["G1", "G1", "G2", "inside G2", "G2"]) + end + + it "tracks the bytes of queued frames" do + seq = sequencer(true) + seq.group_started(1, g1, 10) + seq.group_started(2, g2, 20) + seq.example_started(2, ex(g2, "c"), 30) + + expect(seq.stats).to include(:queued => 2, :queued_bytes => 50, :peak_queued_bytes => 50) + end + end + + describe "#abandon" do + it "balances the group notifications of a live group that will never finish" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(1, g1.children.first) + + seq.abandon(1) + + expect(events.map(&:last)).to eq(%w[G1 N N G1]) + seq.group_started(2, g2) + expect(events.last).to eq([:example_group_started, "G2"]) + end + + it "delivers what a queued group reported before closing it" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 2, ex(g2, "c")) + + seq.abandon(2) + + expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c G2]) + end + + it "ignores workers without a stream" do + seq = sequencer(true) + expect { seq.abandon(9) }.not_to(change { events }) + end + end + + describe "#finish" do + it "delivers everything queued and closes the groups still open" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + run_example(seq, 2, ex(g2, "c")) + seq.group_started(3, g3) + seq.group_finished(3, g3, 0.1) + + seq.finish + + expect(events.map(&:last)).to eq(%w[G1 G1 G3 G3 G2 c c c G2]) + expect(seq.stats[:queued]).to eq(0) + end + end + + describe ".group_aware_listeners?" do + def reporter_with(formatter, *args) + RSpec.configuration.add_formatter(formatter, *args) + RSpec.configuration.reporter + end + + it "is false for formatters that only inherit BaseFormatter's bookkeeping, like progress and JSON" do + expect(Sequencer.group_aware_listeners?(reporter_with("progress", StringIO.new))).to be false + expect(Sequencer.group_aware_listeners?(reporter_with("json", StringIO.new))).to be false + end + + it "is true for the documentation and HTML formatters" do + expect(Sequencer.group_aware_listeners?(reporter_with("documentation", StringIO.new))).to be true + end + + it "is true for the HTML formatter, which reacts to group starts" do + expect(Sequencer.group_aware_listeners?(reporter_with("html", StringIO.new))).to be true + end + + it "is true for a custom listener of group finishes" do + reporter.register_listener(Class.new { def example_group_finished(_); end }.new, :example_group_finished) + expect(Sequencer.group_aware_listeners?(reporter)).to be true + end + + it "ignores RSpec's own profiler" do + reporter.register_listener(RSpec::Core::Profiler.new, :example_group_started, :example_group_finished) + expect(Sequencer.group_aware_listeners?(reporter)).to be false + end + + it "is consulted by default" do + reporter.register_listener(Class.new { def example_group_finished(_); end }.new, :example_group_finished) + expect(Sequencer.new(reporter, :apply_result => lambda { |_| })).to be_sequenced + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb new file mode 100644 index 000000000..d81697edb --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -0,0 +1,186 @@ +module RSpec::Core::Parallel + RSpec.describe WorkerReporter do + let(:results) { Channel.pipe } + let(:jobs) { Channel.pipe } + let(:reporter) { WorkerReporter.new(RSpec.configuration, results.last, jobs.first) } + + def frames + results.first.drain.map(&:first) + end + + after do + results.each(&:close) + jobs.each(&:close) + end + + it "forwards group and example notifications as frames with ids and result payloads" do + group = RSpec.describe("Outer") do + it("passes") { expect(1).to eq(1) } + it("fails") { raise ArgumentError, "nope" } + it("is pending") do + pending "later" + raise "x" + end + it("is skipped") { skip "not now" } + context("Inner") { it("nested") {} } + end + inner = group.children.first + + group.run(reporter) + + kinds = frames + expect(kinds.map(&:first)).to eq([ + :group_started, + :example_started, :example_finished, + :example_started, :example_finished, + :example_started, :example_finished, + :example_started, :example_finished, + :group_started, :example_started, :example_finished, :group_finished, + :group_finished + ]) + + expect(kinds[0]).to eq([:group_started, group.id]) + expect(kinds[9]).to eq([:group_started, inner.id]) + expect(kinds[12]).to eq([:group_finished, inner.id, nil]) + expect(kinds[13][0..1]).to eq([:group_finished, group.id]) + expect(kinds[13][2]).to be_a(Float) + + payloads = kinds.select { |f| f.first == :example_finished }.to_h { |f| [f[1], f[2]] } + passed, failed, pending, skipped = group.examples.map { |example| payloads.fetch(example.id) } + + expect(passed).to include(:status => :passed, :exception => nil, :pending_exception => nil, :pending => nil, :skip => nil) + expect(passed[:started_at]).to be_a(Time) + expect(passed[:finished_at]).to be_a(Time) + expect(passed[:run_time]).to be_a(Float) + + expect(failed[:status]).to eq(:failed) + expect(failed[:exception]).to include(:class_name => "ArgumentError", :message => "nope") + + expect(pending).to include(:status => :pending, :pending_message => "later", :pending_fixed => false, :pending => true) + expect(pending[:pending_exception]).to include(:class_name => "RuntimeError", :message => "x") + + expect(skipped).to include(:status => :pending, :pending_message => "not now", :skip => true, :pending_exception => nil) + end + + it "ships the example ids the coordinator can look up" do + group = RSpec.describe("G") { it("e") {} } + group.run(reporter) + + expect(frames.select { |f| f.first == :example_started }).to eq([[:example_started, group.examples.first.id]]) + end + + it "forwards extra failure lines and stringifies result parts that cannot be marshaled" do + group = RSpec.describe("G") do + it("fails") do |ex| + ex.metadata[:extra_failure_lines] = ["more", lambda {}] + raise "x" + end + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:extra_failure_lines].size).to eq(2) + expect(payload[:extra_failure_lines].first).to eq("more") + expect(payload[:extra_failure_lines].last).to match(/Proc/) + end + + it "forwards messages and deprecations" do + reporter.message("hello") + reporter.message(:not_a_string) + reporter.deprecation(:deprecated => "foo", :message => "msg", :replacement => "bar", :call_site => "here") + + expect(frames).to eq([ + [:message, "hello"], + [:message, :not_a_string], + [:deprecation, { :deprecated => "foo", :message => "msg", :replacement => "bar", :call_site => "here" }] + ]) + end + + it "stringifies messages and deprecation parts that cannot be marshaled" do + reporter.message(lambda {}) + reporter.deprecation(:deprecated => lambda {}, :message => "msg", :replacement => nil, :call_site => nil) + + message, deprecation = frames + expect(message.first).to eq(:message) + expect(message.last).to match(/Proc/) + expect(deprecation.last[:deprecated]).to match(/Proc/) + expect(deprecation.last).to include(:message => "msg", :replacement => nil) + end + + it "forwards custom notifications published through the reporter" do + reporter.publish(:my_event, :a => 1, :b => "two") + reporter.publish(:bare_event) + + expect(frames).to eq([[:custom, :my_event, { :a => 1, :b => "two" }], [:custom, :bare_event, {}]]) + end + + it "warns instead of crashing when custom notification options cannot be marshaled" do + RSpec.configuration.parallel_worker_number = 3 + reporter.publish(:my_event, :callback => lambda {}) + + message = frames.first + expect(message.first).to eq(:message) + expect(message.last).to include("Parallel worker 3 could not forward the custom `my_event` notification", "Proc") + end + + it "forwards non-example failures as a frame with the exception payload and the context description" do + error = RuntimeError.new("suite setup failed") + error.set_backtrace(["./spec/spec_helper.rb:3"]) + + reporter.notify_non_example_exception(error, "An error occurred in a `before(:suite)` hook.") + + kind, payload, description = frames.first + expect(kind).to eq(:non_example_failure) + expect(payload).to include(:class_name => "RuntimeError", :message => "suite setup failed", :backtrace => ["./spec/spec_helper.rb:3"]) + expect(description).to eq("An error occurred in a `before(:suite)` hook.") + expect(RSpec.world.non_example_failure).to be true + end + + it "ignores the notifications that only the coordinator's reporter emits" do + reporter.start(1) + reporter.finish + + expect(frames).to eq([]) + end + + it "does not forward example_passed/failed/pending separately" do + RSpec.describe("G") { it("e") {} }.run(reporter) + + expect(frames.map(&:first)).not_to include(:example_passed) + end + + it "sets the world's `wants_to_quit` when the coordinator asks it to quit between examples" do + jobs.last.send([:quit]) + group = RSpec.describe("G") do + it("first") {} + it("second") {} + end + + group.run(reporter) + + expect(RSpec.world.wants_to_quit).to be true + expect(frames.count { |f| f.first == :example_finished }).to eq(1) + end + + it "leaves other control messages for the worker's job loop while polling" do + jobs.last.send([:run, 7]) + RSpec.describe("G") { it("e") {} }.run(reporter) + + expect(RSpec.world.wants_to_quit).to be_falsey + expect(jobs.first.receive).to eq([:run, 7]) + end + + it "keeps the reporter's own failure bookkeeping so fail-fast works inside the worker" do + RSpec.configuration.fail_fast = true + group = RSpec.describe("G") do + it("fails") { raise "x" } + it("never runs") {} + end + + group.run(reporter) + + expect(reporter.failed_examples.size).to eq(1) + expect(RSpec.world.wants_to_quit).to be true + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb new file mode 100644 index 000000000..f1255792b --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -0,0 +1,171 @@ +module RSpec::Core::Parallel + RSpec.describe Worker do + let(:jobs) { Channel.pipe } + let(:results) { Channel.pipe } + let(:exits) { [] } + let(:stderr) { StringIO.new } + let(:groups) do + [ + RSpec.describe("First") { it("passes") {} }, + RSpec.describe("Second") do + it("fails") { raise "nope" } + end + ] + end + + let(:traps) { {} } + + def worker(number=2) + Worker.new(number, groups, jobs.first, results.last, + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |signal, &handler| traps[signal] = handler }) + end + + def frames + results.first.drain.map(&:first) + end + + after do + jobs.each(&:close) + results.each(&:close) + end + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + describe "#test_env_number" do + it "follows the parallel_tests convention: blank for the first worker, then the worker number" do + expect(worker(1).test_env_number).to eq("") + expect(worker(2).test_env_number).to eq("2") + expect(worker(10).test_env_number).to eq("10") + end + end + + describe "#prepare" do + it "identifies the worker, disables autorun and installs the forwarding reporter" do + worker(3).prepare + + expect(ENV['TEST_ENV_NUMBER']).to eq("3") + expect(RSpec.configuration.parallel_worker_number).to eq(3) + expect(RSpec.configuration.reporter).to be_a(WorkerReporter) + expect(RSpec::Core::Runner).to be_autorun_disabled + end + + it "handles interrupts like the runner does, but silently: stop between examples, force-quit the second time" do + worker.prepare + + expect { traps.fetch('INT').call }.not_to output.to_stderr + expect(RSpec.world.wants_to_quit).to be true + expect(exits).to eq([]) + + traps.fetch('INT').call + expect(exits).to eq([[:exit_now, 1]]) + end + + it "drops the formatters inherited from the coordinating process" do + RSpec.configuration.formatter = "documentation" + expect(RSpec.configuration.formatters).not_to be_empty + + worker.prepare + + expect(RSpec.configuration.formatters).to be_empty + end + end + + describe "#run" do + it "runs the dispatched groups inside the suite hooks, reports each unit and exits normally when told to stop" do + order = [] + RSpec.configuration.before(:suite) { order << :before_suite } + RSpec.configuration.after(:suite) { order << :after_suite } + groups.first.before(:context) { order << :group } + + jobs.last.send([:run, 0]) + jobs.last.send([:run, 1]) + jobs.last.send([:stop]) + + worker.run + + expect(order).to eq([:before_suite, :group, :after_suite]) + expect(exits).to eq([[:exit, 0]]) + + all = frames + expect(all.select { |f| f.first == :unit_finished }).to eq([ + [:unit_finished, 0, true, false, false], + [:unit_finished, 1, false, false, false] + ]) + expect(all.last).to eq([:exiting]) + expect(all.map(&:first)).to include(:group_started, :example_started, :example_finished, :group_finished) + end + + it "treats a quit request received while idle like stop" do + jobs.last.send([:quit]) + + worker.run + + expect(exits).to eq([[:exit, 0]]) + expect(frames).to eq([[:exiting]]) + end + + it "stops when the coordinator goes away" do + jobs.last.close + + worker.run + + expect(exits).to eq([[:exit, 0]]) + end + + it "reports a quitting world and non-example failures with the unit" do + RSpec.configuration.before(:suite) { raise "suite setup failed" } + jobs.last.send([:run, 0]) + jobs.last.send([:stop]) + + worker.run + + unit = frames.find { |f| f.first == :unit_finished } + expect(unit).to eq([:unit_finished, 0, false, true, true]) + end + + it "exits immediately without reporting when the results pipe is gone" do + results.first.close + jobs.last.send([:stop]) + + worker.run + + expect(exits).to eq([[:exit_now, 1]]) + end + + it "arms the watchdog before the after(:suite) hooks run" do + armed_during_after_suite = nil + instance = worker + RSpec.configuration.after(:suite) { armed_during_after_suite = instance.instance_variable_get(:@watchdog_armed) } + allow(instance).to receive(:arm_watchdog) { instance.instance_variable_set(:@watchdog_armed, true) } + jobs.last.send([:stop]) + + instance.run + + expect(armed_during_after_suite).to be true + end + end + + describe "the watchdog" do + it "dumps every thread's backtrace to stderr and aborts the process once the timeout passes" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + instance = worker(4) + blocked = Thread.new { sleep } + + instance.arm_watchdog.join + + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include( + "RSpec parallel worker 4 (pid #{Process.pid}) did not exit within 0.05 seconds", + "(this watchdog)", + "Kernel#sleep" + ) + expect(stderr.string.scan(/^ #= 3 + ensure + blocked&.kill + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb new file mode 100644 index 000000000..0104b01a8 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -0,0 +1,155 @@ +module RSpec::Core + RSpec.describe Parallel do + let(:config) { RSpec.configuration } + let(:runner) { Runner.new(ConfigurationOptions.new([]), config, RSpec.world) } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + def groups(count) + Array.new(count) { |i| RSpec.describe("G#{i}") { it("e") {} } } + end + + describe ".coordinator_for" do + it "returns nil when parallel execution was not requested" do + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "returns a coordinator for the requested number of workers", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + expect(Parallel.coordinator_for(runner, groups(3))).to be_a(Parallel::Coordinator) + end + + it "uses one worker per processor for :auto", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = :auto + allow(Parallel).to receive(:default_worker_count).and_return(2) + expect(Parallel::Coordinator).to receive(:new).with(runner, anything, anything, 2).and_return(:coordinator) + + expect(Parallel.coordinator_for(runner, groups(5))).to eq(:coordinator) + end + + it "never starts more workers than there are groups with examples to run", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 8 + empty = RSpec.describe("empty") + expect(Parallel::Coordinator).to receive(:new).with(runner, anything, anything, 3).and_return(:coordinator) + + Parallel.coordinator_for(runner, groups(3) + [empty]) + end + + it "runs serially when fewer than two groups would be dispatched", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 4 + expect(Parallel.coordinator_for(runner, groups(1))).to be_nil + expect(Parallel.coordinator_for(runner, [])).to be_nil + end + + it "runs serially for one worker or a non-positive count forced from the command line", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.force(:parallel_workers => 1) + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + + config.force(:parallel_workers => 0) + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "runs serially, without a warning, under --dry-run" do + config.parallel_workers = 2 + config.dry_run = true + expect(RSpec).not_to receive(:warning) + + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "warns and runs serially where fork is not available" do + config.parallel_workers = 2 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + expect(RSpec).to receive(:warning).with(a_string_including("needs `fork`", "Running serially"), :call_site => nil) + + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + + it "warns and runs serially inside another tool's parallel worker", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + with_env_vars("TEST_ENV_NUMBER" => "") do + expect(RSpec).to receive(:warning).with(a_string_including("already a worker of a parallel test run"), :call_site => nil) + + expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + end + end + end + + describe ".default_worker_count" do + it "is the number of processors" do + require 'etc' + expect(Parallel.default_worker_count).to eq(Etc.nprocessors) + end + end + + describe ".units_for" do + it "skips groups without examples to run and keeps the given order without timing data" do + all = groups(3) + empty = RSpec.describe("empty") + expect(Parallel.units_for([all[1], empty, all[0], all[2]], config)).to eq([all[1], all[0], all[2]]) + end + + context "with timings recorded in the example status persistence file", :isolated_directory do + let(:all) { groups(4) } + + before do + config.example_status_persistence_file_path = "statuses.txt" + ExampleStatusPersister.persist([], "statuses.txt") + File.write("statuses.txt", <<-STATUSES.gsub(/^ +\|/, '')) + |example_id | status | run_time | + |---------- | ------ | ---------------- | + |#{all[0].examples.first.id} | passed | 0.5 seconds | + |#{all[1].examples.first.id} | passed | 1 minute 2 seconds | + |#{all[3].examples.first.id} | failed | 3 seconds | + STATUSES + end + + it "puts groups with no recorded timing first, then the others longest first" do + expect(Parallel.units_for(all, config)).to eq([all[2], all[1], all[3], all[0]]) + end + + it "keeps the given order for groups with equal timings" do + File.write("statuses.txt", <<-STATUSES.gsub(/^ +\|/, '')) + |example_id | status | run_time | + |---------- | ------ | --------- | + |#{all[0].examples.first.id} | passed | 1 second | + |#{all[1].examples.first.id} | passed | 1 second | + |#{all[2].examples.first.id} | passed | 1 second | + |#{all[3].examples.first.id} | passed | 1 second | + STATUSES + + expect(Parallel.units_for(all.reverse, config)).to eq(all.reverse) + end + + it "ignores an unreadable status file" do + allow(ExampleStatusPersister).to receive(:load_from).and_raise(Errno::EACCES) + expect(Parallel.units_for(all, config)).to eq(all) + end + end + end + + describe ".parse_duration" do + it "reads the durations written by the example status persister" do + expect(Parallel.parse_duration("0.00012 seconds")).to eq(0.00012) + expect(Parallel.parse_duration("1 second")).to eq(1.0) + expect(Parallel.parse_duration("2 minutes 3.5 seconds")).to eq(123.5) + expect(Parallel.parse_duration("")).to be_nil + expect(Parallel.parse_duration(nil)).to be_nil + end + end + + describe ".marshal_safe" do + it "returns values Marshal can dump and the block's value otherwise" do + expect(Parallel.marshal_safe([1, "two"]) { :fallback }).to eq([1, "two"]) + expect(Parallel.marshal_safe(lambda {}) { |value| value.class }).to eq(Proc) + end + end + + describe ".monotonic_time" do + it "never goes backwards" do + first = Parallel.monotonic_time + expect(Parallel.monotonic_time).to be >= first + end + end + end +end diff --git a/rspec-core/spec/rspec/core/profiler_spec.rb b/rspec-core/spec/rspec/core/profiler_spec.rb index 7bd370a5c..90a1b0a7e 100644 --- a/rspec-core/spec/rspec/core/profiler_spec.rb +++ b/rspec-core/spec/rspec/core/profiler_spec.rb @@ -39,6 +39,32 @@ end end + describe '#record_group_duration' do + it 'records a duration measured elsewhere, in place of the one observed here' do + profiler.example_group_started group_notification group + allow(::RSpec::Core::Time).to receive(:now) { now + 1 } + profiler.example_group_finished group_notification group + + profiler.record_group_duration(group, 7.5) + + expect(profiler.example_groups[group]).to include(:total_time => 7.5, :description => "My Group") + end + + it 'records groups it has not seen start' do + profiler.record_group_duration(group, 2.0) + expect(profiler.example_groups[group]).to include(:total_time => 2.0, :count => 0) + end + + context "when the group is not a top-level group" do + let(:group) { super().describe "nested" } + + it 'no-ops since we only consider top-level groups for profiling' do + profiler.record_group_duration(group, 2.0) + expect(profiler.example_groups).to be_empty + end + end + end + describe '#example_group_finished' do before do profiler.example_group_started group_notification group diff --git a/rspec-core/spec/rspec/core/reporter_spec.rb b/rspec-core/spec/rspec/core/reporter_spec.rb index 6c0331156..ba1d15271 100644 --- a/rspec-core/spec/rspec/core/reporter_spec.rb +++ b/rspec-core/spec/rspec/core/reporter_spec.rb @@ -304,6 +304,24 @@ module RSpec::Core end end + describe "#record_example_group_duration" do + it "hands the duration to the profiler, setting it up if needed" do + slow, fast = RSpec.describe("slow"), RSpec.describe("fast") + config.profile_examples = 2 + formatter = double("formatter") + reporter.register_listener(formatter, :dump_profile) + + reporter.record_example_group_duration(slow, 4.0) + reporter.record_example_group_duration(fast, 1.0) + + expect(formatter).to receive(:dump_profile) do |notification| + expect(notification.slowest_groups.map(&:last)).to include(a_hash_including(:total_time => 4.0), a_hash_including(:total_time => 1.0)) + end + reporter.start(0) + reporter.finish + end + end + describe "#notify_non_example_exception" do it "sends a `message` notification that contains the formatted exception details" do formatter_out = StringIO.new diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index 7bcabc485..c66c0a480 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -363,6 +363,21 @@ def interrupt end describe "#run" do + it 'hands the ordered example groups to a parallel coordinator when one applies, then persists example statuses' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) + coordinator = instance_double(Parallel::Coordinator, :run => 17) + expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups).and_return(coordinator) + config.example_status_persistence_file_path = "statuses.txt" + expect(ExampleStatusPersister).to receive(:persist).with(world.all_examples, "statuses.txt") + + runner = Runner.new(ConfigurationOptions.new([]), config, world) + expect(runner).not_to receive(:run_specs) + + expect(runner.run(err, out)).to eq(17) + end + it 'supports a test-queue like subclass that can perform setup once and run different sets of example groups multiple times' do order = [] From 49a893336a5848bdbc5934152f1488cb34acaaee Mon Sep 17 00:00:00 2001 From: ilyazub Date: Sun, 4 Oct 2026 22:45:08 +0200 Subject: [PATCH 02/58] [rspec-core] Give each parallel worker its own aruba directory `setup_aruba` wipes the working directory, which pulled it out from under the `cd` of other workers when the suite ran under `--parallel`. The directory is now namespaced by `TEST_ENV_NUMBER`, chosen in a `before(:suite)` hook because workers are forked after the suite loads, and the two specs that spelled out `tmp/aruba` use the helper instead. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../spec/integration/location_rerun_spec.rb | 8 ++++---- .../integration/spec_file_load_errors_spec.rb | 6 +++--- rspec-core/spec/support/aruba_support.rb | 19 +++++++++++++++++++ 3 files changed, 26 insertions(+), 7 deletions(-) diff --git a/rspec-core/spec/integration/location_rerun_spec.rb b/rspec-core/spec/integration/location_rerun_spec.rb index 38c54c8fa..404935f26 100644 --- a/rspec-core/spec/integration/location_rerun_spec.rb +++ b/rspec-core/spec/integration/location_rerun_spec.rb @@ -67,7 +67,7 @@ around { |ex| with_env_vars('SHELL' => '/usr/local/bin/zsh', &ex) } it 'prints the example id of the failed assertion' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -81,7 +81,7 @@ context "and the shared examples are defined in a separate file" do it 'prints the example id of the failed assertion' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/non_local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/non_local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -103,7 +103,7 @@ context "when the shared examples are defined in the same file as the spec" do it 'prints the line number where the assertion failed in the local file' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: @@ -118,7 +118,7 @@ context "and the shared examples are defined in a separate file" do it 'prints the line number where the `it_behaves_like` was called in the local file' do - run_command("--order defined #{Dir.pwd}/tmp/aruba/non_local_shared_examples_spec.rb") + run_command("--order defined #{Dir.pwd}/#{aruba_working_directory}/non_local_shared_examples_spec.rb") expect(last_cmd_stdout).to include unindent(<<-EOS) Failed examples: diff --git a/rspec-core/spec/integration/spec_file_load_errors_spec.rb b/rspec-core/spec/integration/spec_file_load_errors_spec.rb index 0e58a12e3..c740b71e7 100644 --- a/rspec-core/spec/integration/spec_file_load_errors_spec.rb +++ b/rspec-core/spec/integration/spec_file_load_errors_spec.rb @@ -219,7 +219,7 @@ def foo # Extra blank lines were added in 3.4.0; strip out for compat with 3.2/3.3 expect(formatted_output.gsub("\n\n", "\n")).to include unindent(<<-EOS) SyntaxError: - --> ./tmp/aruba/broken_file.rb + --> ./#{aruba_working_directory}/broken_file.rb Unmatched keyword, missing `end' ? 1 class WorkInProgress > 2 def initialize(arg) @@ -227,7 +227,7 @@ def foo 4 end 5 end EOS - expect(formatted_output).to include %r{./tmp/aruba/broken_file.rb:\d: syntax error} + expect(formatted_output).to include %r{./#{aruba_working_directory}/broken_file.rb:\d: syntax error} expect(formatted_output).to include unindent(<<-EOS) Finished in n.nnnn seconds (files took n.nnnn seconds to load) @@ -249,7 +249,7 @@ def foo EOS # This is subset of the formatted_output, it continues slightly but differs on different Rubies - expect(formatted_output).to include "SyntaxError:\n ./tmp/aruba/broken_file.rb:5: syntax error" + expect(formatted_output).to include "SyntaxError:\n ./#{aruba_working_directory}/broken_file.rb:5: syntax error" expect(formatted_output).to include unindent(<<-EOS) Finished in n.nnnn seconds (files took n.nnnn seconds to load) diff --git a/rspec-core/spec/support/aruba_support.rb b/rspec-core/spec/support/aruba_support.rb index c3bca31dc..2b9292787 100644 --- a/rspec-core/spec/support/aruba_support.rb +++ b/rspec-core/spec/support/aruba_support.rb @@ -6,6 +6,21 @@ module ArubaLoader with_isolated_stderr do require 'aruba/api' end + + # Each parallel worker (`rspec --parallel`, parallel_tests, ...) gets its own + # working directory: `setup_aruba` wipes it, which would pull the directory + # out from under the other workers' `cd`. + def self.working_directory + "tmp/aruba#{ENV['TEST_ENV_NUMBER']}" + end +end + +RSpec.configure do |c| + # In a `before(:suite)` hook because `rspec --parallel` forks its workers, + # which set `TEST_ENV_NUMBER`, after the suite has loaded. + c.before(:suite) do + Aruba.configure { |config| config.working_directory = ArubaLoader.working_directory } + end end RSpec.shared_context "aruba support" do @@ -15,6 +30,10 @@ module ArubaLoader let(:stderr) { StringIO.new } let(:stdout) { StringIO.new } + def aruba_working_directory + ArubaLoader.working_directory + end + before { setup_aruba } attr_reader :last_cmd_stdout, :last_cmd_stderr, :last_cmd_exit_status From e4c8715d34dbf5f231d77f66a4db5c53f109d0b5 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Sun, 4 Oct 2026 22:52:12 +0200 Subject: [PATCH 03/58] [rspec-core] Keep shared spec fixtures per worker and cover worker-only interrupts The "spec files" shared context wrote `_passing_spec.rb` and `_failing_spec.rb` into `spec/support` and deleted them in `after(:all)`; under `--parallel`, nine spec files raced over those two paths and the library-wide checks globbed them while they existed. They now live under `tmp/`, named per worker. Integration specs cover a worker that receives an interrupt on its own: it retires after the group it was running while the others finish, and a run where every worker retires reports the groups nobody was left to run. The changelog spells out what reaches formatters instead of promising that they work unchanged. This is by Claude Fable 5.1 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 7 ++-- rspec-core/spec/integration/parallel_spec.rb | 38 ++++++++++++++++++++ rspec-core/spec/support/spec_files.rb | 13 +++++-- 3 files changed, 54 insertions(+), 4 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 49ccdb773..e7083bfe1 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -6,8 +6,11 @@ Enhancements: * Add `--parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run top-level example groups across worker processes forked after the suite has loaded. Formatters run in the coordinating process only and receive one coherent stream of - notifications with real `Example` objects, so existing formatters work unchanged and the - run has one seed, one summary, one profile and one `.rspec_status` file. Workers run the + notifications with real `Example` objects -- results, timing, pending details, exceptions + rebuilt from their class name, message, backtrace and cause chain, messages, deprecations + and marshalable custom notifications (metadata mutated while an example runs is not + forwarded) -- so existing formatters keep working and the run has one seed, one summary, + one profile and one `.rspec_status` file. Workers run the suite hooks once each, expose `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way parallel_tests does. A worker that finishes but does not exit is aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 58911faa0..d5af7807a 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -338,6 +338,44 @@ def message(notification) end end + describe 'a worker whose own world wants to quit (an interrupt delivered to that worker only)' do + def write_self_interrupting_group(name, examples_after=2) + write_file_formatted "spec/#{name.downcase}_spec.rb", " + RSpec.describe '#{name}' do + it('interrupts itself') { Process.kill('INT', Process.pid); sleep 0.2 } + #{examples_after}.times { |i| it(\"never runs \#{i}\") { } } + end + " + end + + it 'retires that worker after the group it was running and lets the others finish, failing the run' do + write_self_interrupting_group('Interrupted') + write_file_formatted 'spec/other_spec.rb', " + RSpec.describe('Other') { 2.times { |i| it(\"passes \#{i}\") { sleep 0.05 } } } + " + write_file_formatted 'spec/third_spec.rb', " + RSpec.describe('Third') { it('passes') { } } + " + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to match(/Parallel worker \d \(pid \d+\) stopped after Interrupted .* because `RSpec.world.wants_to_quit` was set/) + expect(last_cmd_stdout).to include("4 examples, 0 failures") + expect(last_cmd_stdout).not_to include("RSpec is shutting down") + end + + it 'reports the groups nobody was left to run when every worker retires' do + %w[One Two Three Four].each { |name| write_self_interrupting_group(name, 0) } + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stdout).to include("2 example groups did not run because no worker remained to run them") + end + end + it 'reports a worker that dies and fails the run with the error exit code' do write_file_formatted 'spec/crash_spec.rb', " RSpec.describe 'Crashing' do diff --git a/rspec-core/spec/support/spec_files.rb b/rspec-core/spec/support/spec_files.rb index a9f89f16e..590abe4f2 100644 --- a/rspec-core/spec/support/spec_files.rb +++ b/rspec-core/spec/support/spec_files.rb @@ -1,13 +1,21 @@ RSpec.shared_context "spec files" do + # Written outside `spec/` so that the library-wide checks never glob them, + # and named per parallel worker so that one worker's `after(:all)` cannot + # delete the files another worker is loading. + def spec_file_path(basename) + File.expand_path("../../../tmp/#{basename}#{ENV['TEST_ENV_NUMBER']}.rb", __FILE__) + end + def failing_spec_filename - @failing_spec_filename ||= File.expand_path(File.dirname(__FILE__)) + "/_failing_spec.rb" + @failing_spec_filename ||= spec_file_path("_failing_spec") end def passing_spec_filename - @passing_spec_filename ||= File.expand_path(File.dirname(__FILE__)) + "/_passing_spec.rb" + @passing_spec_filename ||= spec_file_path("_passing_spec") end def create_passing_spec_file + FileUtils.mkdir_p(File.dirname(passing_spec_filename)) File.open(passing_spec_filename, 'w') do |f| f.write %q{ RSpec.describe "passing spec" do @@ -20,6 +28,7 @@ def create_passing_spec_file end def create_failing_spec_file + FileUtils.mkdir_p(File.dirname(failing_spec_filename)) File.open(failing_spec_filename, 'w') do |f| f.write %q{ RSpec.describe "failing spec" do From c786d8d8dc3393a61795291f5c90a773f21782e2 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 02:38:27 +0200 Subject: [PATCH 04/58] [rspec-core] Keep --parallel runs accounted and bounded when workers crash, wedge or lose their coordinator Found by a real-subprocess edge-case matrix and by running SerpApi's suite (Rails, Mongoid, Ruby 2.7, 151k examples) on a 3.12 backport of the runner: - The example a worker was running when it crashed or had to be terminated now fails with a Parallel::WorkerError naming the cause, so it shows under "Failed examples", in .rspec_status and in --only-failures; worker problems outside examples count as errors outside of examples. - Workers exit within half a second when the coordinating process goes away, and SIGTERM/SIGHUP to the coordinator -- or to its process group, as timeout(1) sends -- stop every worker, including ones that cannot tear down, before the coordinator exits the way a serial run would. A second SIGTERM used to cut the cleanup short and leave such workers running. - Listeners registered inside a worker (before(:suite) register_listener) are notified, and an example's result is forwarded after its outcome notification so metadata they add (extra_failure_lines) is kept. - Rebuilt exceptions use Exception's own backtrace accessors; a rebuilt Minitest::UnexpectedError used to crash dump_failures. - --[no-]parallel[=COUNT]: a malformed COUNT is a usage error instead of a backtrace, 0 runs serially, --no-parallel overrides .rspec, and the reproduction command --bisect prints runs serially. - Queued presentation is also flushed past 8 MB, and the parallel specs pass on MRI 3.0-4.0 and JRuby. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 21 +-- rspec-core/README.md | 38 ++++- .../features/command_line/parallel.feature | 2 +- rspec-core/lib/rspec/core/invocations.rb | 10 +- rspec-core/lib/rspec/core/option_parser.rb | 17 +- rspec-core/lib/rspec/core/parallel.rb | 31 ++++ .../lib/rspec/core/parallel/coordinator.rb | 148 +++++++++--------- .../rspec/core/parallel/remote_exception.rb | 20 +++ .../lib/rspec/core/parallel/sequencer.rb | 8 +- .../lib/rspec/core/parallel/suite_index.rb | 23 ++- rspec-core/lib/rspec/core/parallel/worker.rb | 33 +++- .../lib/rspec/core/parallel/worker_handle.rb | 34 +++- .../rspec/core/parallel/worker_reporter.rb | 22 ++- rspec-core/lib/rspec/core/reporter.rb | 7 +- rspec-core/spec/integration/bisect_spec.rb | 12 ++ rspec-core/spec/integration/parallel_spec.rb | 130 ++++++++++++++- .../spec/rspec/core/invocations_spec.rb | 14 ++ .../spec/rspec/core/option_parser_spec.rb | 14 +- .../rspec/core/parallel/coordinator_spec.rb | 89 +++++++++-- .../core/parallel/remote_exception_spec.rb | 37 ++++- .../rspec/core/parallel/sequencer_spec.rb | 16 ++ .../core/parallel/worker_reporter_spec.rb | 30 ++++ .../spec/rspec/core/parallel/worker_spec.rb | 57 ++++++- rspec-core/spec/support/formatter_support.rb | 3 +- 24 files changed, 676 insertions(+), 140 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index e7083bfe1..21be7c8b2 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -3,19 +3,22 @@ Enhancements: -* Add `--parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run +* Add `--[no-]parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run top-level example groups across worker processes forked after the suite has loaded. Formatters run in the coordinating process only and receive one coherent stream of notifications with real `Example` objects -- results, timing, pending details, exceptions rebuilt from their class name, message, backtrace and cause chain, messages, deprecations - and marshalable custom notifications (metadata mutated while an example runs is not - forwarded) -- so existing formatters keep working and the run has one seed, one summary, - one profile and one `.rspec_status` file. Workers run the - suite hooks once each, expose `RSpec::Core::Configuration#parallel_worker_number` and set - `TEST_ENV_NUMBER` the way parallel_tests does. A worker that finishes but does not exit is - aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of - its threads, so a stuck `at_exit` handler can no longer hang a run. Where `fork` is - unavailable the run is serial with a warning. (Illia Zub) + and marshalable custom notifications (of the metadata changed while an example runs only + `:extra_failure_lines`, `:pending` and `:skip` are forwarded) -- so existing formatters + keep working and the run has one seed, one summary, one profile and one `.rspec_status` + file. Listeners registered inside a worker, e.g. in a `before(:suite)` hook, still see + that worker's example notifications. Workers run the suite hooks once each, expose + `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way + parallel_tests does. A worker that finishes but does not exit is aborted after + `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, + so a stuck `at_exit` handler can no longer hang a run, and a worker that crashes fails the + example it was running instead of taking the run down. Where `fork` is unavailable the run + is serial with a warning. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index d1a039b46..4d81c46fb 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -286,7 +286,10 @@ streams results back as examples finish. Your formatters run only in the coordinating process and see the notifications of a normal run, so there is one seed, one summary, one profile and one exit code, and `--fail-fast`, `--only-failures`, `--next-failure`, `--seed`, filters and `path:line` -selection keep working. You can make it the default with: +selection keep working. Once the `--fail-fast` limit is reached no further +groups are handed out and busy workers stop after their current example, so +examples already running in other workers can still add failures. You can make +it the default with: ```ruby RSpec.configure do |config| @@ -307,16 +310,30 @@ before any worker exists: setting `TEST_ENV_NUMBER` afterwards does **not** give workers separate databases, so Rails suites run under `--parallel` share one test database until rspec-rails adds per-worker database support. Open connections, servers and similar resources acquired at load time are shared -the same way; acquire them in `before(:suite)` to get one per worker. +the same way; acquire them in `before(:suite)` to get one per worker. In +particular, a database client that connected while the spec files loaded (a +query at load time is enough) hands the same connection to every worker unless +it reconnects after `fork` by itself -- as with app servers that preload, make +each worker reconnect in a `before(:suite)` hook, e.g. +`Mongoid.disconnect_clients`. On macOS, the system networking libraries do not +survive `fork` well either: when loading the spec files makes real requests +(resolving host names from several threads, say), workers can crash in +`getaddrinfo` or hang. Keep such work out of load time -- in `before(:context)` +hooks it runs in the workers instead. Formatters receive examples with their status, timing, pending details, exceptions (class name, message, backtrace, cause chain and aggregated failures; rebuilt as a subclass of the original class when that class exists in the coordinating process) and the group and example notifications in the order the worker produced them, plus `reporter.message`, deprecations and -custom notifications whose options can be marshaled. Metadata that a worker -mutates while running an example is not forwarded, and anything workers write -to `$stdout`/`$stderr` directly is not sequenced with the formatter output. +custom notifications whose options can be marshaled. Of the metadata a worker +changes while running an example, only `:extra_failure_lines`, `:pending` and +`:skip` are forwarded, and anything workers write to `$stdout`/`$stderr` +directly is not sequenced with the formatter output. Listeners registered from +inside a worker (for example with `reporter.register_listener` in a +`before(:suite)` hook) are notified of that worker's example and group +notifications before they are forwarded; run-level notifications (`start`, +`dump_summary`, `close`, ...) only reach the coordinating process. A worker that has finished its last example group but does not exit (a non-daemon thread, a browser that will not quit, ...) is given @@ -324,11 +341,18 @@ non-daemon thread, a browser that will not quit, ...) is given `after(:suite)` hooks and `at_exit` handlers; after that it dumps the backtraces of all its threads to stderr and is aborted, and the run exits with `config.error_exit_code`. The coordinator never waits unboundedly for a -finished worker. +finished worker. A worker that crashes or is killed (`exit!`, a segfault, the +OOM killer, ...) fails the example it was running with the reason, so the +example is listed under "Failed examples" and picked up by `--only-failures`; +the rest of its example group does not run, and problems outside of examples +count as errors outside of examples. Workers also exit on their own if the +coordinating process disappears. `--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a -warning and runs serially. `--bisect` and `--dry-run` always run serially. +warning and runs serially. `--bisect` and `--dry-run` always run serially, and +the reproduction command `--bisect` prints includes `--no-parallel`, which (like +`--parallel=0`) also overrides a `--parallel` stored in `.rspec`. ## Store Command Line Options `.rspec` diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index 25f94ee65..7641a9a35 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -115,7 +115,7 @@ Feature: `--parallel` option And the output should match /Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/ And the stderr should contain "did not exit within 1 seconds of finishing its last example group" And the stderr should contain "Threads at this point:" - And the stderr should contain "Thread#join" + And the stderr should contain "join" Scenario: Inside another tool's parallel worker RSpec runs serially Given I set the environment variable "TEST_ENV_NUMBER" to "2" diff --git a/rspec-core/lib/rspec/core/invocations.rb b/rspec-core/lib/rspec/core/invocations.rb index 4719085b3..756ad9279 100644 --- a/rspec-core/lib/rspec/core/invocations.rb +++ b/rspec-core/lib/rspec/core/invocations.rb @@ -34,7 +34,7 @@ def call(options, err, out) ) success = RSpec::Core::Bisect::Coordinator.bisect_with( - runner, options.args, formatter + runner, cli_args_for(options, runner.configuration), formatter ) runner.exit_code(success) @@ -42,6 +42,14 @@ def call(options, err, out) private + # Bisection runs the suite serially; the reproduction command it + # prints has to as well, whatever enabled parallel runs. + def cli_args_for(options, configuration) + return options.args unless configuration.parallel_workers + + options.args.reject { |arg| arg =~ /\A--(?:no-)?parallel(?:=|\z)/ } + ['--no-parallel'] + end + def bisect_formatter_klass_for(argument) return Formatters::BisectDebugFormatter if argument == "verbose" Formatters::BisectProgressFormatter diff --git a/rspec-core/lib/rspec/core/option_parser.rb b/rspec-core/lib/rspec/core/option_parser.rb index d602fa6c9..e12ed61e4 100644 --- a/rspec-core/lib/rspec/core/option_parser.rb +++ b/rspec-core/lib/rspec/core/option_parser.rb @@ -75,9 +75,9 @@ def parser(options) options[:runner] = RSpec::Core::Invocations::Bisect.new end - parser.on('--parallel[=COUNT]', Integer, 'Run example groups across COUNT worker processes forked after', - ' loading the suite (default: one per processor). Needs `fork`.') do |count| - options[:parallel_workers] = count || :auto + parser.on('--[no-]parallel[=COUNT]', 'Run example groups across COUNT worker processes forked after', + ' loading the suite (default: one per processor; 0 runs serially). Needs `fork`.') do |argument| + options[:parallel_workers] = parallel_workers_from(argument) end parser.on('--[no-]fail-fast[=COUNT]', 'Abort the run after a certain number of failures (1 by default).') do |argument| @@ -313,6 +313,17 @@ def set_fail_fast(options, value) options[:fail_fast] = value end + def parallel_workers_from(argument) + case argument + when true then :auto + when false, /\A0+\z/ then nil + when /\A\d+\z/ then Integer(argument, 10) + else + abort "Expected a non-negative integer for `--parallel`, got: #{argument.inspect}\n\n" \ + "Please use --help for a listing of valid options" + end + end + def configure_only_failures(options) options[:only_failures] = true add_tag_filter(options, :inclusion_filter, :last_run_status, 'failed') diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index 2784dee43..56f69bfab 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -18,6 +18,12 @@ module Core module Parallel DURATION = /(?:(\d+) minutes? )?([\d.]+) seconds?/ + # Raised -- as the failure of the example a worker was running, or as + # an error outside of examples -- when a parallel worker is lost: + # it crashed, was killed, or had to be terminated because it did not + # stop or exit in time. + class WorkerError < StandardError; end + # @return [Coordinator, nil] a coordinator for the groups, or `nil` when # the run should proceed serially (not requested, not possible or not # worth it). Warns when a requested parallel run is not possible. @@ -110,6 +116,31 @@ def self.monotonic_time Process.clock_gettime(Process::CLOCK_MONOTONIC) end + # Whatever the formatters wrote before `start` (seed, run options) may + # still be buffered in streams that are not `$stdout`; forked workers + # would write it again when they exit. + # @private + def self.flush_output(configuration) + streams = configuration.formatters.map { |formatter| formatter.output if formatter.respond_to?(:output) } + streams.push(configuration.output_stream, configuration.error_stream, configuration.deprecation_stream) + streams.push($stdout, $stderr) + streams.each { |stream| stream.flush if stream.respond_to?(:flush) } + end + + # Installs `handler` for each signal the platform has. + # @return [Hash{String => Object}] the handlers it replaced + # @private + def self.trap_signals(signals, &handler) + signals.select { |signal| Signal.list.key?(signal) }.to_h do |signal| + [signal, trap(signal) { handler.call(signal) }] + end + end + + # @private + def self.restore_signals(handlers) + handlers.each { |signal, previous| trap(signal, previous || 'DEFAULT') } + end + # @return the value when `Marshal` can serialize it, otherwise what the # block makes of it # @private diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index d65a6e12b..6a36a99b8 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -16,6 +16,10 @@ class Coordinator # worker's own watchdog fires first and the thread dump comes from # the process that owns the threads. WATCHDOG_GRACE = 1.0 + SENT_TERM = ', so it was sent SIGTERM' + # Handled by stopping the workers -- in time, even ones that cannot + # tear down -- before the coordinator goes the way a serial run would. + TERMINATION_SIGNALS = %w[TERM HUP].freeze def initialize(runner, example_groups, units, worker_count, options={}) @runner = runner @@ -35,12 +39,13 @@ def initialize(runner, example_groups, units, worker_count, options={}) @unit_states = Array.new(@units.size, :pending) @results = Results.new @quitting = false - @unknown_ids = [] + @termination_signal = nil end # @return [Integer] the exit code for the run def run reporter = @configuration.reporter + handlers = Parallel.trap_signals(TERMINATION_SIGNALS) { |signal| @termination_signal ||= signal } passed = reporter.report(@world.example_count(@example_groups)) do @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) @@ -55,21 +60,21 @@ def run report_unrun_units end - passed? + @unit_states.all? { |state| state == :passed } end - @runner.exit_code(passed) - end + # Terminated like a serial run would have been, once the workers are gone. + raise SignalException, @termination_signal if @termination_signal - # @private - def passed? - @unit_states.all? { |state| state == :passed } + @runner.exit_code(passed) + ensure + Parallel.restore_signals(handlers) if handlers end private def start_workers - flush_output + Parallel.flush_output(@configuration) @process.warmup if @process.respond_to?(:warmup) @worker_count.times { |index| @workers << WorkerHandle.new(index + 1) } @@ -79,30 +84,16 @@ def start_workers end end - # Whatever the formatters wrote before `start` (seed, run options) is - # still buffered in streams that are not `$stdout`; forked workers - # would write it again when they exit. - def flush_output - streams = @configuration.formatters.map { |formatter| formatter.output if formatter.respond_to?(:output) } - streams.push(@configuration.output_stream, @configuration.error_stream, @configuration.deprecation_stream) - streams.push($stdout, $stderr) - streams.each { |stream| stream.flush if stream.respond_to?(:flush) } - end - def fork_worker(worker, workers) - @process.fork do - # :nocov: - runs in the forked worker process, covered by integration specs - workers.each { |other| other.close_all unless other.equal?(worker) } - worker.close_parent_ends - Worker.new(worker.number, @units, worker.child_jobs, worker.child_results).run - # :nocov: - end + worker.fork_worker(@process, @units, workers) end def event_loop dispatch_work until @workers.all?(&:done?) + return report("Received SIG#{@termination_signal}; stopping the workers.") if @termination_signal + wait_for_results reap_workers request_quit if @world.wants_to_quit && !@quitting @@ -147,14 +138,15 @@ def drain(worker) def handle_message(worker, message, bytes) kind, *rest = message + number = worker.number case kind - when :group_started then with_group(rest[0]) { |g| @sequencer.group_started(worker.number, g, bytes) } - when :group_finished then with_group(rest[0]) { |g| @sequencer.group_finished(worker.number, g, rest[1], bytes) } - when :example_started then with_example(rest[0]) { |e| @sequencer.example_started(worker.number, e, bytes) } + when :group_started then index.with_group(rest[0]) { |g| @sequencer.group_started(number, g, bytes) } + when :group_finished then index.with_group(rest[0]) { |g| @sequencer.group_finished(number, g, rest[1], bytes) } + when :example_started then index.with_example(rest[0]) { |e| example_started(worker, e, bytes) } when :example_finished then example_finished(worker, rest[0], rest[1], bytes) - when :message then @sequencer.message(worker.number, rest[0], bytes) - when :deprecation then @sequencer.deprecation(worker.number, rest[0], bytes) - when :custom then @sequencer.custom(worker.number, rest[0], rest[1], bytes) + when :message then @sequencer.message(number, rest[0], bytes) + when :deprecation then @sequencer.deprecation(number, rest[0], bytes) + when :custom then @sequencer.custom(number, rest[0], rest[1], bytes) else handle_control_message(worker, kind, rest) end end @@ -172,34 +164,24 @@ def non_example_failure(payload, context_description) @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) end + def example_started(worker, example, bytes) + worker.current_example = example + @sequencer.example_started(worker.number, example, bytes) + end + def example_finished(worker, example_id, payload, bytes) - with_example(example_id) do |example| + index.with_example(example_id) do |example| + worker.current_example = nil record_result(example, payload) @sequencer.example_finished(worker.number, example, bytes) end end - def with_example(id, &block) - with_known(index.example(id), id, &block) - end - - def with_group(id, &block) - with_known(index.group(id), id, &block) - end - def index - @index ||= SuiteIndex.new(@world) - end - - # Examples and groups defined while running (rather than loading) the - # suite exist only in the worker that defined them. - def with_known(target, id) - return yield(target) if target - return if @unknown_ids.include?(id) - - @unknown_ids << id - report "A parallel worker reported results for #{id}, which was not defined when the suite was loaded; " \ - "examples and groups defined at run time cannot be reported by `--parallel`." + @index ||= SuiteIndex.new(@world) do |id| + report "A parallel worker reported results for #{id}, which was not defined when the suite was loaded; " \ + "examples and groups defined at run time cannot be reported by `--parallel`." + end end # Recorded as soon as it arrives: failure counting and `.rspec_status` @@ -284,12 +266,13 @@ def request_quit # Control messages are a few bytes; a worker that is not reading them is # dead or wedged: reaping or the deadline set here deals with it, and - # `judge_exit` still sees the state it was in. + # `judge_exit` still sees the state it was in. A worker that is merely + # exiting gets the usual grace period to do so. def write(worker, message) worker.jobs.send(message) true rescue Errno::EPIPE, IOError - worker.deadline ||= now + worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace false end @@ -310,17 +293,12 @@ def reap_workers def judge_exit(worker, status) case worker.state when :running - where = worker.current_unit ? "while running #{unit_description(worker.current_unit)}" : "between example groups" - report "#{worker.description} exited unexpectedly (#{WorkerHandle.describe_status(status)}) #{where}; " \ - "the examples it had not finished did not run." - @world.non_example_failure = true - abort_unit(worker) + lost(worker, "exited unexpectedly (#{WorkerHandle.describe_status(status)})") when :stopping return if status&.success? - report "#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ - "If its exit watchdog fired, its thread dump is on stderr above." - @world.non_example_failure = true + error("#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ + "If its exit watchdog fired, its thread dump is on stderr above.") end end @@ -335,10 +313,7 @@ def enforce_deadlines def escalate(worker) case worker.escalation when nil - report "#{worker.description} #{overdue_description(worker)}; sending SIGTERM. Raise " \ - "`config.parallel_shutdown_timeout` if it legitimately needs longer." - @world.non_example_failure = true - abort_unit(worker) + overdue(worker) worker.state = :terminating signal(worker, 'TERM', :term, @term_grace) when :term @@ -351,26 +326,51 @@ def escalate(worker) end end - def overdue_description(worker) + def overdue(worker) timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) if worker.state == :stopping - "did not exit within #{timeout} of finishing its last example group (its `after(:suite)` hooks or " \ - "`at_exit` handlers may be stuck)" + error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ + "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ + "`config.parallel_shutdown_timeout` if it legitimately needs longer.") elsif worker.quit_sent - "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)" + lost(worker, "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)", SENT_TERM) else - "stopped reporting #{timeout} ago" + lost(worker, "stopped reporting for #{timeout}", SENT_TERM) end end def terminate(worker, reason) - report "#{worker.description} #{reason}; sending SIGTERM." - @world.non_example_failure = true - abort_unit(worker) + lost(worker, reason, SENT_TERM) worker.state = :terminating signal(worker, 'TERM', :term, @term_grace) end + # The worker can no longer be relied on. The example it was running, + # if any, fails with the reason -- so the summary, `Failed examples:` + # and `.rspec_status` point at it -- and otherwise the reason is + # reported as an error outside of examples. + def lost(worker, what, consequence='') + unit = worker.current_unit && unit_description(worker.current_unit) + if worker.current_example + example, result = worker.lost_example_result( + "#{worker.description} #{what} while running this example#{consequence}" \ + "#{"; the rest of #{unit} did not run" if unit}." + ) + record_result(example, result) + @sequencer.example_finished(worker.number, example, 0) + else + where = unit ? "while running #{unit}, outside of any example" : 'between example groups' + error("#{worker.description} #{what} #{where}#{consequence}.") + end + abort_unit(worker) + end + + # Counted as an error outside of examples, like a failing suite hook. + def error(text) + @configuration.reporter.notify_non_example_exception(WorkerError.new(text), 'An error occurred in a parallel worker.', + :failure_lines => [text]) + end + def signal(worker, name, escalation, grace) worker.escalation = escalation worker.deadline = now + grace diff --git a/rspec-core/lib/rspec/core/parallel/remote_exception.rb b/rspec-core/lib/rspec/core/parallel/remote_exception.rb index 8568cb811..18eec697c 100644 --- a/rspec-core/lib/rspec/core/parallel/remote_exception.rb +++ b/rspec-core/lib/rspec/core/parallel/remote_exception.rb @@ -127,6 +127,10 @@ def self.build_class(name, base, multiple) # Instance methods of every rebuilt exception. # @private module Remote + BACKTRACE = ::Exception.instance_method(:backtrace) + BACKTRACE_LOCATIONS = ::Exception.instance_method(:backtrace_locations) + SET_BACKTRACE = ::Exception.instance_method(:set_backtrace) + def message remote_payload[:message] end @@ -147,6 +151,22 @@ def cause @cause ||= RemoteException.rebuild(remote_payload[:cause]) end + # The original class may keep its backtrace in state the rebuilt + # instance does not have (`Minitest::UnexpectedError` delegates to + # the error it wraps), so use the one set from the payload -- or by + # `set_backtrace` later on. + def backtrace + BACKTRACE.bind_call(self) + end + + def set_backtrace(backtrace) # rubocop:disable Naming/AccessorMethodName + SET_BACKTRACE.bind_call(self, backtrace) + end + + def backtrace_locations + BACKTRACE_LOCATIONS.bind_call(self) + end + private def remote_payload diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb index a67f641c4..ad88a2907 100644 --- a/rspec-core/lib/rspec/core/parallel/sequencer.rb +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -17,8 +17,9 @@ module Parallel # Liveness is bounded in both modes: when the live group has been silent # for {SILENCE_BOUND} seconds, when a finished group has been waiting # for {WAIT_BOUND} seconds behind a live group that keeps producing - # output, or when more than {QUEUE_BOUND} notifications are queued, - # waiting notifications are delivered right away. Formatters then see + # output, or when more than {QUEUE_BOUND} notifications or + # {QUEUE_BYTES_BOUND} bytes of them are queued, waiting notifications + # are delivered right away. Formatters then see # the live group closed and re-opened around them (its headings # repeat), which is the price of showing results promptly. # @@ -29,6 +30,7 @@ class Sequencer SILENCE_BOUND = 2.0 WAIT_BOUND = 10.0 QUEUE_BOUND = 5_000 + QUEUE_BYTES_BOUND = 8 * 1024 * 1024 CONTEXT_FREE = [:message, :deprecation, :custom].freeze Event = Struct.new(:kind, :target, :payload, :bytes) @@ -128,7 +130,7 @@ def abandon(worker) def tick(now=@clock.call) return unless @sequenced - if @stats[:queued] >= QUEUE_BOUND + if @stats[:queued] >= QUEUE_BOUND || @stats[:queued_bytes] >= QUEUE_BYTES_BOUND flush_everything elsif @completed.any? && (now - @last_delivery_at >= SILENCE_BOUND || now - @completed.first.completed_at >= WAIT_BOUND) flush_completed diff --git a/rspec-core/lib/rspec/core/parallel/suite_index.rb b/rspec-core/lib/rspec/core/parallel/suite_index.rb index ed529319b..bedb231ff 100644 --- a/rspec-core/lib/rspec/core/parallel/suite_index.rb +++ b/rspec-core/lib/rspec/core/parallel/suite_index.rb @@ -5,20 +5,33 @@ module Parallel # workers report them under. # @private class SuiteIndex - def initialize(world) + # @param on_unknown [Proc] called once for each id the suite does not have + def initialize(world, &on_unknown) @world = world + @on_unknown = on_unknown + @unknown_ids = [] end - def example(id) - examples_by_id[id] + def with_example(id, &block) + with_known(examples_by_id[id], id, &block) end - def group(id) - groups_by_id[id] + def with_group(id, &block) + with_known(groups_by_id[id], id, &block) end private + # Examples and groups defined while running (rather than loading) the + # suite exist only in the worker that defined them. + def with_known(target, id) + return yield(target) if target + return if @unknown_ids.include?(id) + + @unknown_ids << id + @on_unknown.call(id) + end + def examples_by_id @examples_by_id ||= @world.all_examples.each_with_object({}) { |example, hash| hash[example.id] = example } end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index ab0df9492..71bab765d 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -14,6 +14,9 @@ module Parallel # independently bounds the wait from its side. # @private class Worker + # Seconds between checks that the coordinating process is still there. + COORDINATOR_POLL = 0.5 + attr_reader :number def initialize(number, example_groups, jobs, results, options={}) @@ -26,7 +29,11 @@ def initialize(number, example_groups, jobs, results, options={}) @stderr = options[:stderr] @exit = options.fetch(:exit) { lambda { |status| exit(status) } } @exit_now = options.fetch(:exit_now) { lambda { |status| exit!(status) } } - @trap = options.fetch(:trap) { lambda { |signal, &handler| trap(signal, &handler) } } + @trap = options.fetch(:trap) do + lambda { |signal, command=nil, &handler| command ? trap(signal, command) : trap(signal, &handler) } + end + @ppid = options.fetch(:ppid) { Process.method(:ppid) } + @watch_coordinator = options.fetch(:watch_coordinator, true) end # @return [String] the `TEST_ENV_NUMBER` value for this worker, following @@ -38,6 +45,7 @@ def test_env_number # Never returns: ends the process when the coordinator has no more work. def run + watch_coordinator if @watch_coordinator prepare run_units @exit.call(0) @@ -46,10 +54,33 @@ def run @exit_now.call(1) end + # A worker notices that the coordinator is gone when it next reports + # to it -- which a worker stuck in a long example does not do. Its + # parent changes when the coordinator dies, so it does not outlive the + # coordinator by more than {COORDINATOR_POLL} seconds. + # @private + def watch_coordinator + coordinator = @ppid.call + thread = Thread.new do + sleep COORDINATOR_POLL while @ppid.call == coordinator + begin + (@stderr || $stderr).puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) is exiting because the " \ + "coordinating process (pid #{coordinator}) is gone." + rescue IOError, SystemCallError + nil + end + @exit_now.call(1) + end + thread.report_on_exception = false + thread + end + # @private def prepare Runner.disable_autorun! trap_interrupt + # The coordinator's own handlers for these are inherited; a worker just goes. + Coordinator::TERMINATION_SIGNALS.each { |signal| @trap.call(signal, 'DEFAULT') if Signal.list.key?(signal) } ENV['TEST_ENV_NUMBER'] = test_env_number @configuration.parallel_worker_number = @number @configuration.reset_reporter diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 25311c87c..1afbf356b 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -4,7 +4,7 @@ module Parallel # Coordinator-side bookkeeping for one worker process. # @private class WorkerHandle - attr_reader :number, :jobs, :results, :child_jobs, :child_results + attr_reader :number, :jobs, :results, :child_jobs, :child_results, :current_example, :current_example_started_at attr_accessor :pid, :current_unit, :state, :deadline, :escalation, :exit_status, :quit_sent, :exiting def initialize(number) @@ -14,6 +14,38 @@ def initialize(number) @state = :running @quit_sent = false @exiting = false + @pid = @current_unit = @deadline = @escalation = @exit_status = nil + @current_example = @current_example_started_at = nil + end + + def current_example=(example) + @current_example = example + @current_example_started_at = example && RSpec::Core::Time.now + end + + # @return [Array(Example, Hash)] the example this worker was running + # when it was lost, and a result failing it with `text` + def lost_example_result(text) + example = @current_example + exception = WorkerError.new(text) + exception.set_backtrace([example.location]) + finished_at = RSpec::Core::Time.now + result = { :status => :failed, :started_at => @current_example_started_at, :finished_at => finished_at, + :run_time => finished_at - @current_example_started_at, + :exception => RemoteException.payload_for(exception) } + self.current_example = nil + [example, result] + end + + # @return [Integer] the pid of the worker process forked to run `units` + def fork_worker(process, units, handles) + process.fork do + # :nocov: - runs in the forked worker process, covered by integration specs + handles.each { |other| other.close_all unless other.equal?(self) } + close_parent_ends + Worker.new(@number, units, @child_jobs, @child_results).run + # :nocov: + end end def close_parent_ends diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 500628db3..8cdfc82c2 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -1,17 +1,21 @@ module RSpec module Core module Parallel - # The reporter installed in a worker process. Instead of notifying - # formatters it forwards every notification -- RSpec's own and custom - # ones published through `Reporter#publish` -- to the coordinator as + # The reporter installed in a worker process. It has no formatters; + # it forwards every notification -- RSpec's own and custom ones + # published through `Reporter#publish` -- to the coordinator as # `Marshal`-safe frames referring to examples and groups by id. The # coordinator replays them onto its own reporter, so formatters run # exclusively in the coordinating process and receive real `Example` # and `ExampleGroup` objects. + # + # Listeners registered on this reporter from inside the worker (for + # instance in a `before(:suite)` hook) are notified as usual before the + # notification is forwarded. # @private class WorkerReporter < Reporter FORWARDED_AS_PAYLOAD = [:message, :deprecation].freeze - DERIVED_BY_COORDINATOR = [:example_passed, :example_failed, :example_pending].freeze + EXAMPLE_OUTCOMES = [:example_passed, :example_failed, :example_pending].freeze RUN_LIFECYCLE = [:seed, :start, :stop, :start_dump, :dump_pending, :dump_failures, :deprecation_summary, :dump_profile, :dump_summary, :close].freeze @@ -33,13 +37,19 @@ def notify_non_example_exception(exception, context_description) end # @private + # An example's result is forwarded with its outcome notification + # (`example_passed`, `example_failed` or `example_pending`), which + # follows `example_finished`, so that metadata worker listeners add + # on either (such as `:extra_failure_lines`) reaches the coordinator. def notify(event, notification) + super + case event when :example_group_started then forward_group_started(notification.group) when :example_group_finished then forward_group_finished(notification.group) when :example_started then @results.send([:example_started, notification.example.id]) - when :example_finished then forward_example_finished(notification.example) - when *DERIVED_BY_COORDINATOR, *RUN_LIFECYCLE then nil + when *EXAMPLE_OUTCOMES then forward_example_finished(notification.example) + when :example_finished, *RUN_LIFECYCLE then nil when *FORWARDED_AS_PAYLOAD then forward(event, notification) else forward_custom(event, notification) end diff --git a/rspec-core/lib/rspec/core/reporter.rb b/rspec-core/lib/rspec/core/reporter.rb index 2a6e97db4..0c8da7ca7 100644 --- a/rspec-core/lib/rspec/core/reporter.rb +++ b/rspec-core/lib/rspec/core/reporter.rb @@ -167,13 +167,14 @@ def record_example_group_duration(group, duration) # @private # Provides a way to notify of an exception that is not tied to any # particular example (such as an exception encountered in a :suite hook). - # Exceptions will be formatted the same way they normally are. - def notify_non_example_exception(exception, context_description) + # Exceptions will be formatted the same way they normally are, unless + # `presenter_options` say otherwise (`:failure_lines`). + def notify_non_example_exception(exception, context_description, presenter_options={}) @configuration.world.non_example_failure = true @non_example_exception_count += 1 example = Example.new(AnonymousExampleGroup, context_description, {}) - presenter = Formatters::ExceptionPresenter.new(exception, example, :indentation => 0) + presenter = Formatters::ExceptionPresenter.new(exception, example, { :indentation => 0 }.merge(presenter_options)) message presenter.fully_formatted(nil) end diff --git a/rspec-core/spec/integration/bisect_spec.rb b/rspec-core/spec/integration/bisect_spec.rb index 87f9e4610..674de6299 100644 --- a/rspec-core/spec/integration/bisect_spec.rb +++ b/rspec-core/spec/integration/bisect_spec.rb @@ -40,6 +40,18 @@ def bisect(cli_args, expected_status=nil) ]) expect(output).to include("Bisect complete!", "rspec ./spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb[1:1] ./spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb[1:1]") end + + it 'returns a reproduction command that runs serially when parallel runs are configured' do + output = bisect(%w[ + --order defined --parallel=2 + spec/rspec/core/resources/bisect/consistently_ordered_1_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_4_specs.rb + ]) + expect(output).to include("Bisect complete!", "consistently_ordered_3_specs.rb[1:1] --order defined --no-parallel") + expect(output).not_to include("--parallel=2") + end end context "when the bisect command saturates the pipe" do diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index d5af7807a..69a536ada 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -107,6 +107,29 @@ def group_blocks(documentation_output) expect(ENV['TEST_ENV_NUMBER']).to eq(test_env_number_before) end + it 'notifies listeners registered in a before(:suite) hook and shows the failure lines they add, like a serial run' do + write_file_formatted 'spec/listener_spec.rb', " + class RefListener + def example_failed(notification) + notification.example.metadata[:extra_failure_lines] = ['Ref: added by a before(:suite) listener'] + end + end + RSpec.configure do |c| + c.before(:suite) { RSpec.configuration.reporter.register_listener(RefListener.new, :example_failed) } + end + RSpec.describe('One') { it('fails') { expect(1).to eq(2) } } + RSpec.describe('Two') { it('fails too') { expect(2).to eq(3) } } + " + + run_rspec "--order defined spec/listener_spec.rb" + serial_stdout = last_cmd_stdout + run_rspec "--parallel=2 --order defined spec/listener_spec.rb" + + expect(last_cmd_exit_status).to eq(1) + expect(serial_stdout.scan('Ref: added by a before(:suite) listener').size).to eq(2) + expect(last_cmd_stdout.scan('Ref: added by a before(:suite) listener').size).to eq(2) + end + describe 'formatter parity with a serial run' do before { write_fixed_suite } @@ -250,8 +273,9 @@ def message(notification) "> Two (feature)", " ok nil", "< Two", - "custom: {from: \"worker\"}", - "message: Parallel worker 1 could not forward the custom `my_event` notification because its options cannot be marshaled: {payload: # 'worker' }.inspect}", + "message: Parallel worker 1 could not forward the custom `my_event` notification because its options cannot be " \ + "marshaled: #{{ :payload => 0 }.inspect.chomp('0}')}# nil }, *rspec, '--parallel=2', 'spec/wedged_spec.rb', + :out => 'stdout.txt', :err => File::NULL, :pgroup => true) + deadline = deadline_for.call(20) + sleep 0.05 until Dir['worker_*.pid'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + workers = Dir['worker_*.pid'].map { |file| Integer(File.read(file)) } + expect(workers.size).to eq(2) + + # What `timeout` does: the command first, then its whole process group. + Process.kill(:TERM, pid) + Process.kill(:TERM, -pid) + _, status = Process.wait2(pid) + + expect(status.termsig).to eq(Signal.list.fetch('TERM')) + expect(File.read('stdout.txt')).to include("Received SIGTERM; stopping the workers.") + deadline = deadline_for.call(10) + sleep 0.05 until workers.none?(&running) || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + expect(workers.select(&running)).to eq([]) + end + end + + it 'stops workers stuck in an example once the coordinating process is gone' do + write_file_formatted 'spec/stuck_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('sleeps') { sleep 60 } } + RSpec.describe('Two') { it('sleeps') { sleep 60 } } + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + # Orphans are adopted by whatever reaps them here -- a container's init may + # not, leaving zombies, which no longer run. + running = lambda do |pid| + state = `ps -o stat= -p #{pid}`.strip + !state.empty? && !state.start_with?('Z') + end + + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/stuck_spec.rb', :out => File::NULL, :err => 'stderr.txt') + expect(wait_until.call(20) { Dir['worker_*.pid'].size == 2 }).to be true + workers = Dir['worker_*.pid'].map { |file| Integer(File.read(file)) } + + Process.kill(:KILL, pid) + Process.wait(pid) + + expect(wait_until.call(10) { workers.none?(&running) }).to be true + expect(File.read('stderr.txt').scan("is exiting because the coordinating process (pid #{pid}) is gone").size).to eq(2) + end end describe 'a worker that finishes but does not exit' do @@ -439,7 +554,8 @@ def alive?(pid) expect(last_cmd_stdout).not_to include("sending SIGTERM") expect(last_cmd_stdout).to include("2 examples, 0 failures") expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/) - expect(worker_stderr).to include("did not exit within 0.5 seconds of finishing its last example group", "Threads at this point:", "Thread#join") + expect(worker_stderr).to include("did not exit within 0.5 seconds of finishing its last example group", "Threads at this point:") + expect(worker_stderr).to match(/[`'](Thread#)?join'/) expect(pids_in(last_cmd_stdout)).not_to be_empty pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } end @@ -461,7 +577,7 @@ def alive?(pid) expect(elapsed).to be < 5 expect(last_cmd_stdout).to include( "did not exit within 0.5 seconds of finishing its last example group", - "sending SIGTERM", + "so it was sent SIGTERM", "still alive 0.3 seconds after SIGTERM; sending SIGKILL" ) pids_in(last_cmd_stdout).each { |pid| expect(alive?(pid)).to be false } diff --git a/rspec-core/spec/rspec/core/invocations_spec.rb b/rspec-core/spec/rspec/core/invocations_spec.rb index 8b4476f1a..c345dcf64 100644 --- a/rspec-core/spec/rspec/core/invocations_spec.rb +++ b/rspec-core/spec/rspec/core/invocations_spec.rb @@ -93,6 +93,20 @@ def run_invocation ) end + context "when parallel runs are configured" do + let(:original_cli_args) { %w[--bisect --parallel=3 --seed 1234] } + + it "makes the reproduction command run serially" do + run_invocation + + expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( + an_instance_of(Runner), + %w[--bisect --seed 1234 --no-parallel], + an_instance_of(Formatters::BisectProgressFormatter) + ) + end + end + context "when the bisection is successful" do it "returns 0" do exit_code = run_invocation diff --git a/rspec-core/spec/rspec/core/option_parser_spec.rb b/rspec-core/spec/rspec/core/option_parser_spec.rb index 0d598d9b8..fc02e19a3 100644 --- a/rspec-core/spec/rspec/core/option_parser_spec.rb +++ b/rspec-core/spec/rspec/core/option_parser_spec.rb @@ -350,8 +350,18 @@ module RSpec::Core expect(options[:files_or_directories_to_run]).to eq(%w[ spec/models ]) end - it "rejects a non-numeric count" do - expect { Parser.parse(%w[ --parallel=lots ]) }.to raise_error(OptionParser::InvalidArgument) + it "runs serially with a count of 0, like --no-parallel, overriding an earlier --parallel" do + expect(Parser.parse(%w[ --parallel=0 ])).to include(:parallel_workers => nil) + expect(Parser.parse(%w[ --parallel=4 --no-parallel ])).to include(:parallel_workers => nil) + end + + %w[ lots 2.5 -1 ].push("").each do |count| + it "aborts with a usage message for the count #{count.inspect}" do + parser = Parser.new(["--parallel=#{count}"]) + message = "Expected a non-negative integer for `--parallel`, got: #{count.inspect}" + expect(parser).to receive(:abort).with(a_string_including(message, 'use --help')) + parser.parse + end end end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 7ad58770a..cad8d5dc0 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -119,7 +119,8 @@ def run_fake_worker(number, jobs, results, behaviour, &record_exit) sleep if behaviour == :never_exits }, :exit_now => record_exit, - :trap => lambda { |*| }).run + :trap => lambda { |*| }, + :watch_coordinator => false).run end def coordinator(units, worker_count=2, options={}) @@ -196,11 +197,34 @@ def messages end describe "a worker that exits while running a group" do - it "reports it, leaves the examples it had not finished unknown and fails the run with the error exit code" do + it "fails the example it was running with the reason and leaves the rest of the group unknown" do config.error_exit_code = 5 crashing = RSpec.describe("Crashing") do it("finishes") {} - it("dies") { raise NoMemoryError, "simulated worker death" } + it("dies") { Thread.current.kill } + it("never runs") {} + end + groups = [crashing, passing_group("B")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + dies = crashing.examples[1] + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, :failed, nil]) + expect(dies.exception).to be_a(WorkerError) + expect(dies.exception.message).to match( + /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running this example; the rest of Crashing \(.+\) did not run\.\z/ + ) + expect(dies.exception.backtrace).to eq([dies.location]) + expect(dies.execution_result.run_time).to be >= 0 + expect(messages).to include("Crashing dies", "3 examples, 1 failure", "rspec #{dies.location_rerun_argument}") + end + + it "reports an error outside of examples when it was not running one" do + config.error_exit_code = 5 + crashing = RSpec.describe("Crashing") do + before(:context) { Thread.current.kill } it("never runs") {} end groups = [crashing, passing_group("B")] @@ -209,9 +233,11 @@ def messages exit_code = coordinator(groups).run expect(exit_code).to eq(5) - expect(messages).to match(/Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing/) - expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, nil, nil]) - expect(messages).to include("3 examples, 0 failures") + expect(crashing.examples.first.execution_result.status).to be_nil + expect(messages).to match( + /An error occurred in a parallel worker\.\nParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing \(.+\), outside of any example\./ + ) + expect(messages).to include("1 example, 0 failures, 1 error occurred outside of examples") end end @@ -226,8 +252,8 @@ def messages expect(exit_code).to eq(3) expect(process.signals.map(&:first)).to eq(["TERM"]) - expect(messages).to include("did not exit within 0.05 seconds of finishing its last example group", "sending SIGTERM") - expect(messages).to include("2 examples, 0 failures") + expect(messages).to include("did not exit within 0.05 seconds of finishing its last example group", "so it was sent SIGTERM") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") end it "is sent SIGKILL when it survives SIGTERM" do @@ -270,6 +296,23 @@ def messages expect(messages).to include("did not stop within 0.1 seconds of being asked to stop early") expect(messages).to include("1 example, 1 failure") end + + it "fails the example it is stuck in with the reason" do + config.parallel_shutdown_timeout = 0.1 + config.fail_fast = true + stuck = RSpec.describe("Stuck") { it("sleeps") { sleep } } + groups = [failing_group("Fails"), stuck] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(stuck.examples.first.exception.message).to match( + /\AParallel worker 2 \(pid \d+\) did not stop within 0\.1 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM; the rest of Stuck \(.+\) did not run\.\z/ + ) + expect(messages).to include("2 examples, 2 failures") + end end it "terminates a worker that sends data it cannot decode" do @@ -295,8 +338,8 @@ def messages end it "reports the groups that never ran when no worker remains" do - crashing = RSpec.describe("Crashing") { it("dies") { raise NoMemoryError, "simulated worker death" } } - groups = [crashing, RSpec.describe("Also crashing") { it("dies") { raise NoMemoryError, "simulated worker death" } }, passing_group("C"), passing_group("D")] + crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } + groups = [crashing, RSpec.describe("Also crashing") { it("dies") { Thread.current.kill } }, passing_group("C"), passing_group("D")] config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) exit_code = coordinator(groups).run @@ -437,6 +480,28 @@ def example_passed(_notification) expect(messages).to include("survived SIGKILL; giving up on it") end + it "stops the workers when it is sent SIGTERM, then goes the way a serial run would" do + handler = nil + allow(RSpec::Core::Parallel).to receive(:trap_signals) do |signals, &block| + expect(signals).to eq(%w[TERM HUP]) + handler = block + { "TERM" => "DEFAULT" } + end + allow(RSpec::Core::Parallel).to receive(:restore_signals) + groups = [passing_group("A"), RSpec.describe("Stuck") { it("sleeps") { sleep } }] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + signaller = Thread.new do + sleep 0.01 until handler && process.signals.empty? && messages.include?(".") + handler.call("TERM") + end + + expect { coordinator(groups).run }.to raise_error(SignalException, "SIGTERM") + signaller.join + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("Received SIGTERM; stopping the workers.", "2 examples, 0 failures") + expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") + end + it "copes with a worker that is gone by the time it is signalled" do worker_behaviours[2] = :never_exits config.parallel_shutdown_timeout = 0.05 @@ -445,7 +510,7 @@ def example_passed(_notification) exit_code = coordinator([passing_group("A"), passing_group("B")]).run expect(exit_code).to eq(config.failure_exit_code) - expect(messages).to include("sending SIGTERM", "sending SIGKILL", "giving up on it") + expect(messages).to include("so it was sent SIGTERM", "sending SIGKILL", "giving up on it") end describe WorkerHandle do @@ -479,7 +544,7 @@ def example_passed(_notification) coordinator([passing_group("A"), passing_group("B")]).run expect(process.signals.map(&:first)).to eq(["TERM"]) - expect(messages).to include("stopped reporting 0.05 seconds ago") + expect(messages).to include("stopped reporting for 0.05 seconds", "so it was sent SIGTERM") end end end diff --git a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb index c631e24e7..663b81f79 100644 --- a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb @@ -123,7 +123,8 @@ def message = "from accessor #{@required.upcase}" expect(rebuilt.message).to eq("A Needs::Arguments for which `exception.message.to_s` raises NoMethodError.") end - it "ships the detailed message so that SyntaxError formatting matches", :if => SyntaxError.method_defined?(:detailed_message) do + it "ships the detailed message so that SyntaxError formatting matches", + :skip => !Exception.method_defined?(:detailed_message) && "needs Exception#detailed_message (Ruby 3.2+)" do original = begin eval("}", binding, __FILE__, __LINE__) rescue SyntaxError => e @@ -156,6 +157,40 @@ def original.detailed_message(*) = raise("nope") expect(rebuilt.backtrace).to eq(%w[a b c]) end + it "uses the backtrace it was sent even when the original class computes its own from state it lacks" do + # Like `Minitest::UnexpectedError`, which rspec-rails specs can raise. + wrapping = Class.new(StandardError) do + attr_accessor :error + + def initialize(error) + super("Unexpected exception") + self.error = error + end + + def backtrace + error.backtrace + end + + def set_backtrace(backtrace) # rubocop:disable Naming/AccessorMethodName + error.set_backtrace(backtrace) + end + end + stub_const("WrappingError", wrapping) + inner = RuntimeError.new("inner") + inner.set_backtrace(["./spec/inner_spec.rb:3"]) + + rebuilt = round_trip(WrappingError.new(inner)) + copy = rebuilt.dup + copy.set_backtrace(["./spec/other_spec.rb:1"]) + + expect(rebuilt).to be_a(WrappingError) + expect(rebuilt.backtrace).to eq(["./spec/inner_spec.rb:3"]) + expect(rebuilt.backtrace_locations).to be_nil + expect(copy.backtrace).to eq(["./spec/other_spec.rb:1"]) + presented = RSpec::Core::Formatters::ExceptionPresenter.new(rebuilt, new_example).fully_formatted(1) + expect(presented).to include("Unexpected exception", "./spec/inner_spec.rb:3") + end + context "with an exception that aggregates several failures" do def aggregate_failures_error RSpec::Expectations::FailureAggregator.new("checking things", :hide_backtrace => true).aggregate do diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index 35853d8cd..d886a59d2 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -239,6 +239,22 @@ def events expect(seq.stats[:queued]).to eq(1) end + it "also delivers everything queued once the queued notifications reach #{Sequencer::QUEUE_BYTES_BOUND} bytes" do + stub_const("RSpec::Core::Parallel::Sequencer::QUEUE_BYTES_BOUND", 1_000) + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.message(2, "big", 600) + seq.tick + expect(events.map(&:last)).to eq(%w[G1]) + expect(seq.stats[:queued_bytes]).to eq(600) + + seq.message(2, "bigger", 600) + seq.tick + expect(events.map(&:last)).to eq(%w[G1 G1 G2 big bigger]) + expect(seq.stats).to include(:queued => 0, :queued_bytes => 0, :peak_queued_bytes => 1_200) + end + it "delivers messages sent outside any group immediately, without touching the open group" do seq = sequencer(true) seq.group_started(1, g1) diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index d81697edb..e2356eb73 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -149,6 +149,36 @@ def frames expect(frames.map(&:first)).not_to include(:example_passed) end + it "notifies listeners registered inside the worker, such as from a `before(:suite)` hook" do + events = [] + notifications = [:example_group_started, :example_started, :example_finished, :example_failed, :message, :my_event] + listener = Object.new + notifications.each do |event| + listener.define_singleton_method(event) { |_notification| events << event } + end + reporter.register_listener(listener, *notifications) + + RSpec.describe("G") { it("fails") { raise "x" } }.run(reporter) + reporter.message("hi") + reporter.publish(:my_event) + + expect(events).to eq(notifications) + expect(frames.map(&:first)).to eq([:group_started, :example_started, :example_finished, :group_finished, :message, :custom]) + end + + it "forwards metadata that a worker listener adds when notified of the example's outcome" do + listener = Object.new + def listener.example_failed(notification) + notification.example.metadata[:extra_failure_lines] = ["Ref: added in the worker"] + end + reporter.register_listener(listener, :example_failed) + + RSpec.describe("G") { it("fails") { raise "x" } }.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:extra_failure_lines]).to eq(["Ref: added in the worker"]) + end + it "sets the world's `wants_to_quit` when the coordinator asks it to quit between examples" do jobs.last.send([:quit]) group = RSpec.describe("G") do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index f1255792b..ef766d419 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -20,7 +20,8 @@ def worker(number=2) :stderr => stderr, :exit => lambda { |status| exits << [:exit, status] }, :exit_now => lambda { |status| exits << [:exit_now, status] }, - :trap => lambda { |signal, &handler| traps[signal] = handler }) + :trap => lambda { |signal, command=nil, &handler| traps[signal] = command || handler }, + :watch_coordinator => false) end def frames @@ -42,6 +43,49 @@ def frames end end + describe "#watch_coordinator" do + it "exits at once, saying why, when its parent (the coordinating process) changes" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + parents = [100, 100, 100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include("worker 2 (pid #{Process.pid}) is exiting because the coordinating process (pid 100) is gone") + end + + it "still exits when it cannot say why (its stderr is gone too)" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + parents = [100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => StringIO.new.tap(&:close), + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + end + + it "is started by #run unless disabled" do + instance = Worker.new(2, [], jobs.first, results.last, + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :ppid => lambda { 100 }) + allow(instance).to receive(:watch_coordinator).and_return(nil) + jobs.last.send([:stop]) + + instance.run + + expect(instance).to have_received(:watch_coordinator) + expect(exits).to eq([[:exit, 0]]) + end + end + describe "#prepare" do it "identifies the worker, disables autorun and installs the forwarding reporter" do worker(3).prepare @@ -63,6 +107,12 @@ def frames expect(exits).to eq([[:exit_now, 1]]) end + it "restores the default handling of the signals the coordinating process handles itself" do + worker.prepare + + expect(traps).to include("TERM" => "DEFAULT", "HUP" => "DEFAULT") + end + it "drops the formatters inherited from the coordinating process" do RSpec.configuration.formatter = "documentation" expect(RSpec.configuration.formatters).not_to be_empty @@ -152,6 +202,7 @@ def frames it "dumps every thread's backtrace to stderr and aborts the process once the timeout passes" do RSpec.configuration.parallel_shutdown_timeout = 0.05 instance = worker(4) + blocked_line = __LINE__ + 1 blocked = Thread.new { sleep } instance.arm_watchdog.join @@ -159,9 +210,9 @@ def frames expect(exits).to eq([[:exit_now, 1]]) expect(stderr.string).to include( "RSpec parallel worker 4 (pid #{Process.pid}) did not exit within 0.05 seconds", - "(this watchdog)", - "Kernel#sleep" + "(this watchdog)" ) + expect(stderr.string).to match(/worker_spec\.rb:#{blocked_line}:in /) expect(stderr.string.scan(/^ #= 3 ensure blocked&.kill diff --git a/rspec-core/spec/support/formatter_support.rb b/rspec-core/spec/support/formatter_support.rb index fe1735fe4..94eca0d3a 100644 --- a/rspec-core/spec/support/formatter_support.rb +++ b/rspec-core/spec/support/formatter_support.rb @@ -272,7 +272,8 @@ module RSpec::Core class Reporter module EnforceRSpecNotificationsListComplete def notify(event, *args) - return super if caller_locations(1, 1).first.label =~ /publish/ + # Two frames, so that reporters overriding `notify` and calling `super` are covered too. + return super if caller_locations(1, 2).any? { |location| location.label =~ /publish/ } return super if RSPEC_NOTIFICATIONS.include?(event) raise "#{event.inspect} must be added to `RSPEC_NOTIFICATIONS`" From 51b6398a1d1d72aae8a2852a2e52650d1a858207 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 03:29:20 +0200 Subject: [PATCH 05/58] [rspec-core] Leave ignored signals alone and report each example once under --parallel Fixes from an independent review of the previous commit, each reproduced first with real processes: - A run started under nohup no longer dies on SIGHUP. The coordinator only takes over TERM/HUP when they would end the process; ignored signals and handlers the suite installed stay in force, and workers get back exactly the handlers the coordinator replaced instead of the defaults. - Once a worker is given up on (it did not stop in time, crashed or sent garbage), nothing more is read from it. A worker that survived SIGTERM could otherwise report its example a second time, counting it twice or crashing the failure summary. - Under --fail-fast, a worker that leaves cleanly right after its group, before the coordinator asked it to stop, is no longer reported as an error outside of examples. - --bisect also takes --parallel out of SPEC_OPTS, which win over the command line, for its runs and the reproduction command it prints. - Two coordinator specs no longer race under load. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/README.md | 9 ++- rspec-core/lib/rspec/core/invocations.rb | 26 +++++-- rspec-core/lib/rspec/core/parallel.rb | 19 ++++- .../lib/rspec/core/parallel/coordinator.rb | 19 ++--- .../lib/rspec/core/parallel/suite_index.rb | 6 +- rspec-core/lib/rspec/core/parallel/worker.rb | 6 +- .../lib/rspec/core/parallel/worker_handle.rb | 4 +- rspec-core/spec/integration/bisect_spec.rb | 14 ++++ rspec-core/spec/integration/parallel_spec.rb | 36 ++++++++++ .../spec/rspec/core/invocations_spec.rb | 19 +++++ .../rspec/core/parallel/coordinator_spec.rb | 69 +++++++++++++++++-- .../spec/rspec/core/parallel/worker_spec.rb | 21 +++--- rspec-core/spec/rspec/core/parallel_spec.rb | 41 +++++++++++ 13 files changed, 254 insertions(+), 35 deletions(-) diff --git a/rspec-core/README.md b/rspec-core/README.md index 4d81c46fb..80483301e 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -346,13 +346,18 @@ OOM killer, ...) fails the example it was running with the reason, so the example is listed under "Failed examples" and picked up by `--only-failures`; the rest of its example group does not run, and problems outside of examples count as errors outside of examples. Workers also exit on their own if the -coordinating process disappears. +coordinating process disappears. SIGTERM or SIGHUP to the coordinating +process, as `timeout` or a cancelled CI job sends, stops every worker -- even +one that cannot tear down -- before the process ends the way a serial run +would; a signal the run was started ignoring (`nohup`), or that your suite +installs a handler for, is left alone. `--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a warning and runs serially. `--bisect` and `--dry-run` always run serially, and the reproduction command `--bisect` prints includes `--no-parallel`, which (like -`--parallel=0`) also overrides a `--parallel` stored in `.rspec`. +`--parallel=0`) also overrides a `--parallel` stored in `.rspec`, and leaves +`--parallel` out of its `SPEC_OPTS`. ## Store Command Line Options `.rspec` diff --git a/rspec-core/lib/rspec/core/invocations.rb b/rspec-core/lib/rspec/core/invocations.rb index 756ad9279..da45f10f3 100644 --- a/rspec-core/lib/rspec/core/invocations.rb +++ b/rspec-core/lib/rspec/core/invocations.rb @@ -33,21 +33,39 @@ def call(options, err, out) out, runner.configuration.bisect_runner ) - success = RSpec::Core::Bisect::Coordinator.bisect_with( - runner, cli_args_for(options, runner.configuration), formatter - ) + success = without_parallel_spec_opts(runner.configuration) do + RSpec::Core::Bisect::Coordinator.bisect_with( + runner, cli_args_for(options, runner.configuration), formatter + ) + end runner.exit_code(success) end private + PARALLEL_OPTION = /\A--(?:no-)?parallel(?:=|\z)/ + # Bisection runs the suite serially; the reproduction command it # prints has to as well, whatever enabled parallel runs. def cli_args_for(options, configuration) return options.args unless configuration.parallel_workers - options.args.reject { |arg| arg =~ /\A--(?:no-)?parallel(?:=|\z)/ } + ['--no-parallel'] + options.args.reject { |arg| arg =~ PARALLEL_OPTION } + ['--no-parallel'] + end + + # `SPEC_OPTS` win over the command line, and the reproduction command + # and the runs of `--bisect-runner=shell` reuse them. + def without_parallel_spec_opts(configuration) + original = ENV.fetch('SPEC_OPTS', nil) + if configuration.parallel_workers && original + require 'shellwords' + remaining = Shellwords.split(original).reject { |arg| arg =~ PARALLEL_OPTION } + ENV['SPEC_OPTS'] = remaining.empty? ? nil : Shellwords.join(remaining) + end + yield + ensure + ENV['SPEC_OPTS'] = original end def bisect_formatter_klass_for(argument) diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index 56f69bfab..9a950f681 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -127,12 +127,25 @@ def self.flush_output(configuration) streams.each { |stream| stream.flush if stream.respond_to?(:flush) } end - # Installs `handler` for each signal the platform has. + # Handlers that end the process: Ruby's and the system's defaults, and + # `nil` -- one installed outside Ruby, which `trap` cannot reinstate. + ENDS_PROCESS = ['DEFAULT', 'SYSTEM_DEFAULT', nil].freeze + + # Installs `handler` for each signal the platform has that would end the + # process. Ignored signals (as under `nohup`) and handlers the suite + # installed are left in place. # @return [Hash{String => Object}] the handlers it replaced # @private def self.trap_signals(signals, &handler) - signals.select { |signal| Signal.list.key?(signal) }.to_h do |signal| - [signal, trap(signal) { handler.call(signal) }] + signals.each_with_object({}) do |signal, replaced| + next unless Signal.list.key?(signal) + + previous = trap(signal) { handler.call(signal) } + if ENDS_PROCESS.include?(previous) + replaced[signal] = previous + else + trap(signal, previous) + end end end diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 6a36a99b8..49e3a942d 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -40,12 +40,13 @@ def initialize(runner, example_groups, units, worker_count, options={}) @results = Results.new @quitting = false @termination_signal = nil + @signal_handlers = {} end # @return [Integer] the exit code for the run def run reporter = @configuration.reporter - handlers = Parallel.trap_signals(TERMINATION_SIGNALS) { |signal| @termination_signal ||= signal } + @signal_handlers = Parallel.trap_signals(TERMINATION_SIGNALS) { |signal| @termination_signal ||= signal } passed = reporter.report(@world.example_count(@example_groups)) do @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) @@ -68,7 +69,7 @@ def run @runner.exit_code(passed) ensure - Parallel.restore_signals(handlers) if handlers + Parallel.restore_signals(@signal_handlers) end private @@ -85,7 +86,7 @@ def start_workers end def fork_worker(worker, workers) - worker.fork_worker(@process, @units, workers) + worker.fork_worker(@process, @units, workers, @signal_handlers) end def event_loop @@ -178,10 +179,7 @@ def example_finished(worker, example_id, payload, bytes) end def index - @index ||= SuiteIndex.new(@world) do |id| - report "A parallel worker reported results for #{id}, which was not defined when the suite was loaded; " \ - "examples and groups defined at run time cannot be reported by `--parallel`." - end + @index ||= SuiteIndex.new(@world) { |warning| report warning } end # Recorded as soon as it arrives: failure counting and `.rspec_status` @@ -293,6 +291,9 @@ def reap_workers def judge_exit(worker, status) case worker.state when :running + # Asked to stop early, a worker may leave as soon as its group ends. + return if status&.success? && worker.exiting && worker.current_unit.nil? + lost(worker, "exited unexpectedly (#{WorkerHandle.describe_status(status)})") when :stopping return if status&.success? @@ -348,7 +349,8 @@ def terminate(worker, reason) # The worker can no longer be relied on. The example it was running, # if any, fails with the reason -- so the summary, `Failed examples:` # and `.rspec_status` point at it -- and otherwise the reason is - # reported as an error outside of examples. + # reported as an error outside of examples. Nothing it sends from now + # on is read: it would report the example a second time. def lost(worker, what, consequence='') unit = worker.current_unit && unit_description(worker.current_unit) if worker.current_example @@ -363,6 +365,7 @@ def lost(worker, what, consequence='') error("#{worker.description} #{what} #{where}#{consequence}.") end abort_unit(worker) + worker.results.close end # Counted as an error outside of examples, like a failing suite hook. diff --git a/rspec-core/lib/rspec/core/parallel/suite_index.rb b/rspec-core/lib/rspec/core/parallel/suite_index.rb index bedb231ff..31e8d06a8 100644 --- a/rspec-core/lib/rspec/core/parallel/suite_index.rb +++ b/rspec-core/lib/rspec/core/parallel/suite_index.rb @@ -5,7 +5,8 @@ module Parallel # workers report them under. # @private class SuiteIndex - # @param on_unknown [Proc] called once for each id the suite does not have + # @param on_unknown [Proc] called with a warning, once for each id the + # suite does not have def initialize(world, &on_unknown) @world = world @on_unknown = on_unknown @@ -29,7 +30,8 @@ def with_known(target, id) return if @unknown_ids.include?(id) @unknown_ids << id - @on_unknown.call(id) + @on_unknown.call("A parallel worker reported results for #{id}, which was not defined when the suite was " \ + "loaded; examples and groups defined at run time cannot be reported by `--parallel`.") end def examples_by_id diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 71bab765d..db9982d1a 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -34,6 +34,7 @@ def initialize(number, example_groups, jobs, results, options={}) end @ppid = options.fetch(:ppid) { Process.method(:ppid) } @watch_coordinator = options.fetch(:watch_coordinator, true) + @signal_handlers = options.fetch(:signal_handlers, {}) end # @return [String] the `TEST_ENV_NUMBER` value for this worker, following @@ -79,8 +80,9 @@ def watch_coordinator def prepare Runner.disable_autorun! trap_interrupt - # The coordinator's own handlers for these are inherited; a worker just goes. - Coordinator::TERMINATION_SIGNALS.each { |signal| @trap.call(signal, 'DEFAULT') if Signal.list.key?(signal) } + # Inherited from the coordinator: its handlers, which a worker swaps + # back for the ones they replaced. + @signal_handlers.each { |signal, previous| @trap.call(signal, previous || 'DEFAULT') } ENV['TEST_ENV_NUMBER'] = test_env_number @configuration.parallel_worker_number = @number @configuration.reset_reporter diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 1afbf356b..8409937e1 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -38,12 +38,12 @@ def lost_example_result(text) end # @return [Integer] the pid of the worker process forked to run `units` - def fork_worker(process, units, handles) + def fork_worker(process, units, handles, signal_handlers={}) process.fork do # :nocov: - runs in the forked worker process, covered by integration specs handles.each { |other| other.close_all unless other.equal?(self) } close_parent_ends - Worker.new(@number, units, @child_jobs, @child_results).run + Worker.new(@number, units, @child_jobs, @child_results, :signal_handlers => signal_handlers).run # :nocov: end end diff --git a/rspec-core/spec/integration/bisect_spec.rb b/rspec-core/spec/integration/bisect_spec.rb index 674de6299..155e8542d 100644 --- a/rspec-core/spec/integration/bisect_spec.rb +++ b/rspec-core/spec/integration/bisect_spec.rb @@ -52,6 +52,20 @@ def bisect(cli_args, expected_status=nil) expect(output).to include("Bisect complete!", "consistently_ordered_3_specs.rb[1:1] --order defined --no-parallel") expect(output).not_to include("--parallel=2") end + + it 'leaves the parallel options out of the SPEC_OPTS of the reproduction command, since they would win over it' do + output = nil + with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do + output = bisect(%w[ + spec/rspec/core/resources/bisect/consistently_ordered_1_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb + spec/rspec/core/resources/bisect/consistently_ordered_4_specs.rb + ]) + end + expect(output).to include("Bisect complete!", 'SPEC_OPTS="--order defined" rspec ./spec/rspec/core/resources/bisect/consistently_ordered_2_specs.rb[1:1] ./spec/rspec/core/resources/bisect/consistently_ordered_3_specs.rb[1:1] --no-parallel') + expect(output).not_to include("--parallel=2") + end end context "when the bisect command saturates the pipe" do diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 69a536ada..1e8322454 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -476,6 +476,42 @@ def write_self_interrupting_group(name, examples_after=2) end end + it 'leaves SIGHUP ignored when it starts ignoring it, as under nohup, like a serial run' do + write_file_formatted 'spec/hangup_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('hangs up on its own process') { Process.kill('HUP', Process.pid); sleep 0.1 } } + RSpec.describe('Two') do + it('outlasts a hangup of the whole run') do + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 20 + sleep 0.05 until File.exist?('hung_up') || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + end + end + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + ignoring_hangups = [RbConfig.ruby, '-e', 'trap("HUP", "IGNORE"); exec(*ARGV)'] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *ignoring_hangups, *rspec, '--parallel=2', 'spec/hangup_spec.rb', + :out => 'stdout.txt', :err => File::NULL) + expect(wait_until.call(20) { Dir['worker_*.pid'].size == 2 }).to be true + + Process.kill(:HUP, pid) + File.write('hung_up', '') + status = nil + finished = wait_until.call(30) { status ||= Process.waitpid2(pid, Process::WNOHANG)&.last } + Process.kill(:KILL, pid) && Process.wait(pid) unless finished + + expect(status&.exitstatus).to eq(0) + expect(File.read('stdout.txt')).to include("2 examples, 0 failures") + expect(File.read('stdout.txt')).not_to include("SIGHUP") + end + end + it 'stops workers stuck in an example once the coordinating process is gone' do write_file_formatted 'spec/stuck_spec.rb', " RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } diff --git a/rspec-core/spec/rspec/core/invocations_spec.rb b/rspec-core/spec/rspec/core/invocations_spec.rb index c345dcf64..315d390a1 100644 --- a/rspec-core/spec/rspec/core/invocations_spec.rb +++ b/rspec-core/spec/rspec/core/invocations_spec.rb @@ -105,6 +105,25 @@ def run_invocation an_instance_of(Formatters::BisectProgressFormatter) ) end + + it "takes the parallel options out of SPEC_OPTS, which win over the command line, while it bisects" do + during = [] + allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with) do + during << ENV.fetch('SPEC_OPTS', nil) + success + end + + with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do + run_invocation + expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--parallel=2 --order defined') + end + with_env_vars('SPEC_OPTS' => '--no-parallel --parallel') do + run_invocation + expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--no-parallel --parallel') + end + + expect(during).to eq(['--order defined', nil]) + end end context "when the bisection is successful" do diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index cad8d5dc0..93f803d6b 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -194,6 +194,34 @@ def messages expect(groups.last.examples.first.execution_result.status).to be_nil expect(world.wants_to_quit).to be true end + + it "accepts a worker leaving as soon as its group ends once it was asked to stop" do + config.fail_fast = true + config.error_exit_code = 5 + # Holds the coordinator up so that the worker has finished its group + # before the failure is handled and has exited before that pass ends. + slow_formatter = Class.new do + RSpec::Core::Formatters.register self, :example_started, :example_failed + + def initialize(_output); end + + def example_started(_notification) + sleep 0.1 + end + + def example_failed(_notification) + sleep 0.2 + end + end + config.add_formatter(slow_formatter, output) + groups = [failing_group("Fails"), passing_group("Never dispatched")] + + exit_code = coordinator(groups, 1).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("1 example, 1 failure") + expect(messages).not_to include("exited unexpectedly", "outside of examples") + end end describe "a worker that exits while running a group" do @@ -297,12 +325,31 @@ def messages expect(messages).to include("1 example, 1 failure") end + # Fake workers share `RSpec.world`: a group started after fail-fast set + # `wants_to_quit` would be skipped, so the failure waits for it to start. + def failure_once_running(name, &body) + started = Queue.new + running = RSpec.describe(name) do + it("runs") do + started << true + instance_exec(&body) + end + end + fails = RSpec.describe("Fails") do + it("fails once #{name} runs") do + deadline = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 while started.empty? && RSpec::Core::Parallel.monotonic_time < deadline + raise "boom" + end + end + [fails, running] + end + it "fails the example it is stuck in with the reason" do config.parallel_shutdown_timeout = 0.1 config.fail_fast = true - stuck = RSpec.describe("Stuck") { it("sleeps") { sleep } } - groups = [failing_group("Fails"), stuck] - config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + groups = failure_once_running("Stuck") { sleep } + stuck = groups.last exit_code = coordinator(groups).run @@ -313,6 +360,19 @@ def messages ) expect(messages).to include("2 examples, 2 failures") end + + it "ignores what it sends once it was given up on, so that its example is reported once" do + config.parallel_shutdown_timeout = 0.1 + config.fail_fast = true + groups = failure_once_running("Late") { sleep 0.5 } + + exit_code = coordinator(groups, 2, :process => FakeProcess.new(["TERM"]), :term_grace => 5).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.last.examples.first.exception).to be_a(WorkerError) + expect(messages).to include("2 examples, 2 failures") + expect(messages).not_to include("sending SIGKILL") + end end it "terminates a worker that sends data it cannot decode" do @@ -497,7 +557,8 @@ def example_passed(_notification) expect { coordinator(groups).run }.to raise_error(SignalException, "SIGTERM") signaller.join - expect(process.signals.map(&:first)).to eq(["TERM"]) + # The worker that ran A may not have been reaped yet, and gets one too. + expect(process.signals.map(&:first).uniq).to eq(["TERM"]) expect(messages).to include("Received SIGTERM; stopping the workers.", "2 examples, 0 failures") expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index ef766d419..dd7e2450e 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -15,13 +15,13 @@ module RSpec::Core::Parallel let(:traps) { {} } - def worker(number=2) + def worker(number=2, options={}) Worker.new(number, groups, jobs.first, results.last, - :stderr => stderr, - :exit => lambda { |status| exits << [:exit, status] }, - :exit_now => lambda { |status| exits << [:exit_now, status] }, - :trap => lambda { |signal, command=nil, &handler| traps[signal] = command || handler }, - :watch_coordinator => false) + { :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |signal, command=nil, &handler| traps[signal] = command || handler }, + :watch_coordinator => false }.merge(options)) end def frames @@ -107,10 +107,15 @@ def frames expect(exits).to eq([[:exit_now, 1]]) end - it "restores the default handling of the signals the coordinating process handles itself" do + it "reinstates the handlers the coordinating process replaced with its own, and only those" do + worker(2, :signal_handlers => { "TERM" => "SYSTEM_DEFAULT", "HUP" => nil }).prepare + + expect(traps).to include("TERM" => "SYSTEM_DEFAULT", "HUP" => "DEFAULT") + + traps.clear worker.prepare - expect(traps).to include("TERM" => "DEFAULT", "HUP" => "DEFAULT") + expect(traps.keys).to eq(["INT"]) end it "drops the formatters inherited from the coordinating process" do diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 0104b01a8..ff6efe941 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -151,5 +151,46 @@ def groups(count) expect(Parallel.monotonic_time).to be >= first end end + + describe "signal handling", :skip => (Signal.list.key?("HUP") ? false : "needs SIGHUP") do + around do |example| + original = trap("HUP", "DEFAULT") + begin + example.run + ensure + trap("HUP", original || "DEFAULT") + end + end + + it "takes over a signal that would end the process and returns the handler it replaced" do + received = [] + trap("HUP", "SYSTEM_DEFAULT") + + replaced = Parallel.trap_signals(%w[HUP NOSUCHSIGNAL]) { |signal| received << signal } + trap("HUP", "DEFAULT").call + + expect(replaced).to eq("HUP" => "SYSTEM_DEFAULT") + expect(received).to eq(["HUP"]) + end + + it "leaves an ignored signal ignored, as under nohup, and a handler the suite installed in place" do + suite_handler = proc {} + trap("HUP", "IGNORE") + + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", suite_handler)).to eq("IGNORE") + + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", "DEFAULT")).to equal(suite_handler) + end + + it "reinstates the replaced handlers, using Ruby's default for one installed outside Ruby" do + Parallel.restore_signals("HUP" => "SYSTEM_DEFAULT") + expect(trap("HUP", "IGNORE")).to eq("SYSTEM_DEFAULT") + + Parallel.restore_signals("HUP" => nil) + expect(trap("HUP", "IGNORE")).to eq("DEFAULT") + end + end end end From 2084cbe2d594b526ada27eb6091de5b96ed1afe0 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 03:54:05 +0200 Subject: [PATCH 06/58] [rspec-core] Make --bisect reproductions serial however parallel runs are enabled, and finish parallel runs without an idle tick - The reproduction command --bisect prints now always ends with --no-parallel. Bisect cannot see config.parallel_workers set in a helper that the spec files require, and in a parallel run the groups of an order-dependent failure no longer share a process, so the command did not reproduce it. --parallel options are also always taken out of SPEC_OPTS while bisecting. - A signal the suite ignores with trap(signal, nil) stays ignored under --parallel, as with "IGNORE". - The coordinator no longer sleeps a full 0.25 s tick when every remaining worker is already exiting: a 24-example suite goes from 0.35 s to 0.09 s with --parallel=4 (serial: 0.07 s), and on 34 CI shards of a large Rails app the runs went from 0.13 s slower to 0.10 s faster than flatware (medians). This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 3 +- rspec-core/README.md | 4 +- .../features/command_line/bisect.feature | 32 ++++++++-------- rspec-core/lib/rspec/core/invocations.rb | 24 ++++++------ rspec-core/lib/rspec/core/parallel.rb | 9 +++-- .../lib/rspec/core/parallel/coordinator.rb | 5 ++- rspec-core/lib/rspec/core/parallel/worker.rb | 2 +- .../spec/rspec/core/invocations_spec.rb | 38 +++++++++---------- .../rspec/core/parallel/coordinator_spec.rb | 14 +++++++ .../spec/rspec/core/parallel/worker_spec.rb | 2 +- rspec-core/spec/rspec/core/parallel_spec.rb | 9 +++-- 11 files changed, 80 insertions(+), 62 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 21be7c8b2..f0081ed74 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -18,7 +18,8 @@ Enhancements: `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, so a stuck `at_exit` handler can no longer hang a run, and a worker that crashes fails the example it was running instead of taking the run down. Where `fork` is unavailable the run - is serial with a warning. (Illia Zub) + is serial with a warning. `--bisect` runs serially, and the reproduction command it prints + now ends with `--no-parallel`. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index 80483301e..5c7ea5c8b 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -356,8 +356,8 @@ installs a handler for, is left alone. already set because another tool is running RSpec in parallel, RSpec prints a warning and runs serially. `--bisect` and `--dry-run` always run serially, and the reproduction command `--bisect` prints includes `--no-parallel`, which (like -`--parallel=0`) also overrides a `--parallel` stored in `.rspec`, and leaves -`--parallel` out of its `SPEC_OPTS`. +`--parallel=0`) overrides `--parallel` in `.rspec` and `config.parallel_workers` +in a helper, and leaves `--parallel` out of its `SPEC_OPTS`. ## Store Command Line Options `.rspec` diff --git a/rspec-core/features/command_line/bisect.feature b/rspec-core/features/command_line/bisect.feature index 82de8d133..7803a5de8 100644 --- a/rspec-core/features/command_line/bisect.feature +++ b/rspec-core/features/command_line/bisect.feature @@ -52,7 +52,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" + Bisect started using options: "--seed 1234 --no-parallel" Running suite to find failures... (0.16755 seconds) Starting bisect with 1 failing example and 9 non-failing examples. Checking that failure(s) are order-dependent... failure appears to be order-dependent @@ -64,7 +64,7 @@ Feature: Bisect Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.26 seconds. The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234` Then the output should contain "2 examples, 1 failure" @@ -73,7 +73,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect` and abort in the middle with ctrl-c Then bisect should fail with output like: """ - Bisect started using options: "--seed 1234" + Bisect started using options: "--seed 1234 --no-parallel" Running suite to find failures... (0.17102 seconds) Starting bisect with 1 failing example and 9 non-failing examples. Checking that failure(s) are order-dependent... failure appears to be order-dependent @@ -85,7 +85,7 @@ Feature: Bisect Bisect aborted! The most minimal reproduction command discovered so far is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234` Then the output should contain "3 examples, 1 failure" @@ -94,7 +94,7 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect=verbose` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" and bisect runner: :fork + Bisect started using options: "--seed 1234 --no-parallel" and bisect runner: :fork Running suite to find failures... (0.16528 seconds) - Failing examples (1): - ./spec/calculator_1_spec.rb[1:1] @@ -109,11 +109,11 @@ Feature: Bisect - ./spec/calculator_8_spec.rb[1:1] - ./spec/calculator_9_spec.rb[1:1] Checking that failure(s) are order-dependent.. - - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 (n.nnnn seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel (n.nnnn seconds) - Failure appears to be order-dependent Round 1: bisecting over non-failing examples 1-9 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 (0.15302 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.19708 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_6_spec.rb[1:1] ./spec/calculator_7_spec.rb[1:1] ./spec/calculator_8_spec.rb[1:1] ./spec/calculator_9_spec.rb[1:1] --seed 1234 --no-parallel (0.15302 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 --no-parallel (0.19708 seconds) - Examples we can safely ignore (4): - ./spec/calculator_6_spec.rb[1:1] - ./spec/calculator_7_spec.rb[1:1] @@ -126,8 +126,8 @@ Feature: Bisect - ./spec/calculator_4_spec.rb[1:1] - ./spec/calculator_5_spec.rb[1:1] Round 2: bisecting over non-failing examples 1-5 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 (0.15836 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.19065 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_4_spec.rb[1:1] ./spec/calculator_5_spec.rb[1:1] --seed 1234 --no-parallel (0.15836 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel (0.19065 seconds) - Examples we can safely ignore (2): - ./spec/calculator_4_spec.rb[1:1] - ./spec/calculator_5_spec.rb[1:1] @@ -136,15 +136,15 @@ Feature: Bisect - ./spec/calculator_2_spec.rb[1:1] - ./spec/calculator_3_spec.rb[1:1] Round 3: bisecting over non-failing examples 1-3 - - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 (0.21028 seconds) - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 (0.1975 seconds) + - Running: rspec ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_2_spec.rb[1:1] --seed 1234 --no-parallel (0.21028 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] ./spec/calculator_3_spec.rb[1:1] --seed 1234 --no-parallel (0.1975 seconds) - Examples we can safely ignore (1): - ./spec/calculator_2_spec.rb[1:1] - Remaining non-failing examples (2): - ./spec/calculator_10_spec.rb[1:1] - ./spec/calculator_3_spec.rb[1:1] Round 4: bisecting over non-failing examples 1-2 - - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 (0.17173 seconds) + - Running: rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel (0.17173 seconds) - Examples we can safely ignore (1): - ./spec/calculator_3_spec.rb[1:1] - Remaining non-failing examples (1): @@ -152,7 +152,7 @@ Feature: Bisect Bisect complete! Reduced necessary non-failing examples from 9 to 1 in 1.47 seconds. The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ When I run `rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234` Then the output should contain "2 examples, 1 failure" @@ -171,8 +171,8 @@ Feature: Bisect When I run `rspec --seed 1234 --bisect=verbose` Then bisect should succeed with output like: """ - Bisect started using options: "--seed 1234" and bisect runner: :shell + Bisect started using options: "--seed 1234 --no-parallel" and bisect runner: :shell # ... The minimal reproduction command is: - rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 + rspec ./spec/calculator_10_spec.rb[1:1] ./spec/calculator_1_spec.rb[1:1] --seed 1234 --no-parallel """ diff --git a/rspec-core/lib/rspec/core/invocations.rb b/rspec-core/lib/rspec/core/invocations.rb index da45f10f3..33f438846 100644 --- a/rspec-core/lib/rspec/core/invocations.rb +++ b/rspec-core/lib/rspec/core/invocations.rb @@ -33,10 +33,8 @@ def call(options, err, out) out, runner.configuration.bisect_runner ) - success = without_parallel_spec_opts(runner.configuration) do - RSpec::Core::Bisect::Coordinator.bisect_with( - runner, cli_args_for(options, runner.configuration), formatter - ) + success = without_parallel_spec_opts do + RSpec::Core::Bisect::Coordinator.bisect_with(runner, cli_args_for(options), formatter) end runner.exit_code(success) @@ -46,22 +44,22 @@ def call(options, err, out) PARALLEL_OPTION = /\A--(?:no-)?parallel(?:=|\z)/ - # Bisection runs the suite serially; the reproduction command it - # prints has to as well, whatever enabled parallel runs. - def cli_args_for(options, configuration) - return options.args unless configuration.parallel_workers - + # Bisection runs the suite serially, and so does the reproduction + # command it prints -- even when parallel runs are enabled where this + # process cannot see it, in a helper that the spec files require. + def cli_args_for(options) options.args.reject { |arg| arg =~ PARALLEL_OPTION } + ['--no-parallel'] end # `SPEC_OPTS` win over the command line, and the reproduction command # and the runs of `--bisect-runner=shell` reuse them. - def without_parallel_spec_opts(configuration) + def without_parallel_spec_opts original = ENV.fetch('SPEC_OPTS', nil) - if configuration.parallel_workers && original + if original require 'shellwords' - remaining = Shellwords.split(original).reject { |arg| arg =~ PARALLEL_OPTION } - ENV['SPEC_OPTS'] = remaining.empty? ? nil : Shellwords.join(remaining) + args = Shellwords.split(original) + remaining = args.reject { |arg| arg =~ PARALLEL_OPTION } + ENV['SPEC_OPTS'] = remaining.empty? ? nil : Shellwords.join(remaining) unless remaining == args end yield ensure diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index 9a950f681..c52082686 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -127,9 +127,10 @@ def self.flush_output(configuration) streams.each { |stream| stream.flush if stream.respond_to?(:flush) } end - # Handlers that end the process: Ruby's and the system's defaults, and - # `nil` -- one installed outside Ruby, which `trap` cannot reinstate. - ENDS_PROCESS = ['DEFAULT', 'SYSTEM_DEFAULT', nil].freeze + # Handlers that end the process: Ruby's and the system's defaults. Any + # other (`IGNORE`, `nil` -- also an ignored signal --, `EXIT`, a proc) + # stays in force. + ENDS_PROCESS = %w[DEFAULT SYSTEM_DEFAULT].freeze # Installs `handler` for each signal the platform has that would end the # process. Ignored signals (as under `nohup`) and handlers the suite @@ -151,7 +152,7 @@ def self.trap_signals(signals, &handler) # @private def self.restore_signals(handlers) - handlers.each { |signal, previous| trap(signal, previous || 'DEFAULT') } + handlers.each { |signal, previous| trap(signal, previous) } end # @return the value when `Marshal` can serialize it, otherwise what the diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 49e3a942d..45109a3b0 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -108,7 +108,10 @@ def wait_for_results readers = @workers.reject { |worker| worker.done? || worker.results.eof? || worker.results.closed? } if readers.empty? - sleep @tick + # Each worker left is exiting (its results pipe is at its end) or + # was given up on: all there is to do is reap it or enforce its + # deadline, and an exiting worker is reapable within moments. + sleep [@tick, 0.01].min return end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index db9982d1a..6a1c78a78 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -82,7 +82,7 @@ def prepare trap_interrupt # Inherited from the coordinator: its handlers, which a worker swaps # back for the ones they replaced. - @signal_handlers.each { |signal, previous| @trap.call(signal, previous || 'DEFAULT') } + @signal_handlers.each { |signal, previous| @trap.call(signal, previous) } ENV['TEST_ENV_NUMBER'] = test_env_number @configuration.parallel_worker_number = @number @configuration.reset_reporter diff --git a/rspec-core/spec/rspec/core/invocations_spec.rb b/rspec-core/spec/rspec/core/invocations_spec.rb index 315d390a1..db5c9fdb0 100644 --- a/rspec-core/spec/rspec/core/invocations_spec.rb +++ b/rspec-core/spec/rspec/core/invocations_spec.rb @@ -83,12 +83,12 @@ def run_invocation allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with).and_return(success) end - it "starts the bisection coordinator" do + it "starts the bisection coordinator, with a reproduction command that runs serially" do run_invocation expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( an_instance_of(Runner), - configuration_options.args, + configuration_options.args + ["--no-parallel"], an_instance_of(Formatters::BisectProgressFormatter) ) end @@ -96,7 +96,7 @@ def run_invocation context "when parallel runs are configured" do let(:original_cli_args) { %w[--bisect --parallel=3 --seed 1234] } - it "makes the reproduction command run serially" do + it "replaces the option in the reproduction command" do run_invocation expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( @@ -105,25 +105,23 @@ def run_invocation an_instance_of(Formatters::BisectProgressFormatter) ) end + end - it "takes the parallel options out of SPEC_OPTS, which win over the command line, while it bisects" do - during = [] - allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with) do - during << ENV.fetch('SPEC_OPTS', nil) - success - end - - with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do - run_invocation - expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--parallel=2 --order defined') - end - with_env_vars('SPEC_OPTS' => '--no-parallel --parallel') do - run_invocation - expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--no-parallel --parallel') - end + it "takes the parallel options out of SPEC_OPTS, which win over the command line, while it bisects" do + during = [] + allow(RSpec::Core::Bisect::Coordinator).to receive(:bisect_with) do + during << ENV.fetch('SPEC_OPTS', nil) + success + end - expect(during).to eq(['--order defined', nil]) + with_env_vars('SPEC_OPTS' => '--parallel=2 --order defined') do + run_invocation + expect(ENV.fetch('SPEC_OPTS', nil)).to eq('--parallel=2 --order defined') end + with_env_vars('SPEC_OPTS' => '--no-parallel --parallel') { run_invocation } + with_env_vars('SPEC_OPTS' => '--example "two words" --order defined') { run_invocation } + + expect(during).to eq(['--order defined', nil, '--example "two words" --order defined']) end context "when the bisection is successful" do @@ -162,7 +160,7 @@ def run_invocation expect(RSpec::Core::Bisect::Coordinator).to have_received(:bisect_with).with( an_instance_of(Runner), - configuration_options.args, + configuration_options.args + ["--no-parallel"], an_instance_of(Formatters::BisectDebugFormatter) ) end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 93f803d6b..cc392ce89 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -89,6 +89,10 @@ def spawn(handle, _handles) send_unknown_ids(results) if behaviour == :unknown_ids run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } end + ensure + # Like the end of a process: its pipes close, and it is gone a moment later. + [jobs, results].each(&:close) + sleep 0.2 if behaviour == :slow_exit end @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } @@ -156,6 +160,16 @@ def messages expect(groups.flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) end + it "reaps workers as soon as they are gone rather than a tick after their results pipe closed" do + worker_behaviours[1] = worker_behaviours[2] = :slow_exit + started = RSpec::Core::Parallel.monotonic_time + + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :tick => 5).run + + expect(exit_code).to eq(0) + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 2 + end + it "returns the failure exit code when an example fails" do exit_code = coordinator([passing_group("A"), failing_group("B")]).run diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index dd7e2450e..08e9df4dd 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -108,7 +108,7 @@ def frames end it "reinstates the handlers the coordinating process replaced with its own, and only those" do - worker(2, :signal_handlers => { "TERM" => "SYSTEM_DEFAULT", "HUP" => nil }).prepare + worker(2, :signal_handlers => { "TERM" => "SYSTEM_DEFAULT", "HUP" => "DEFAULT" }).prepare expect(traps).to include("TERM" => "SYSTEM_DEFAULT", "HUP" => "DEFAULT") diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index ff6efe941..4f66af87c 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -178,17 +178,20 @@ def groups(count) trap("HUP", "IGNORE") expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) - expect(trap("HUP", suite_handler)).to eq("IGNORE") + expect(trap("HUP", nil)).to eq("IGNORE") + + expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) + expect(trap("HUP", suite_handler)).to be_nil expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) expect(trap("HUP", "DEFAULT")).to equal(suite_handler) end - it "reinstates the replaced handlers, using Ruby's default for one installed outside Ruby" do + it "reinstates the handlers it replaced" do Parallel.restore_signals("HUP" => "SYSTEM_DEFAULT") expect(trap("HUP", "IGNORE")).to eq("SYSTEM_DEFAULT") - Parallel.restore_signals("HUP" => nil) + Parallel.restore_signals("HUP" => "DEFAULT") expect(trap("HUP", "IGNORE")).to eq("DEFAULT") end end From 734dc34e7c3f745b959b335833b6b9d3507018a4 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 04:36:36 +0200 Subject: [PATCH 07/58] [rspec-core] Keep the parallel specs deterministic on loaded machines and JRuby Found by running rspec-core's own suite under --parallel in CPU-starved containers (MRI 3.0-4.0, JRuby 9.4-10.1): - The coordinator specs' fake workers run their examples on threads of the spec's process, side by side; those examples now skip the rspec-mocks lifecycle, whose scopes the spec's own stubs live in (on JRuby the threads raced on them and stubs vanished). - Deadline specs either use a single worker or a timeout long enough for the well-behaved worker, and the fake grace periods are long unless a spec is about them, so a slow thread death is not escalated. - The integration specs wait for the worker pid files to be written, not just created; the --fail-fast spec waits for the failure instead of racing worker start-up; the requeue spec no longer depends on a 0.2 s timeout; the watchdog spec waits for its blocked thread to block. - The SyntaxError spec no longer depends on syntax_suggest finishing its search twice within its 1 s timeout, and the signal spec accepts how JRuby reports a signal ignored with nil. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/integration/parallel_spec.rb | 23 +++++-- .../rspec/core/parallel/coordinator_spec.rb | 61 +++++++++++++------ .../core/parallel/remote_exception_spec.rb | 6 +- .../spec/rspec/core/parallel/worker_spec.rb | 2 + rspec-core/spec/rspec/core/parallel_spec.rb | 4 +- 5 files changed, 68 insertions(+), 28 deletions(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 1e8322454..628d02b05 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -17,6 +17,12 @@ def run_rspec(cmd) run_command(cmd) end + # Written by the workers' `before(:suite)` hooks: a file can exist before + # its pid has been written to it. + def worker_pids + Dir['worker_*.pid'].map { |file| File.read(file) }.reject(&:empty?).map { |pid| Integer(pid) } + end + def write_fixed_suite write_file_formatted 'spec/alpha_spec.rb', " RSpec.describe 'Alpha' do @@ -283,11 +289,15 @@ def message(notification) it 'honours --fail-fast across workers by asking them to stop between examples' do write_file_formatted 'spec/fast_fail_spec.rb', " RSpec.describe 'Fails first' do - it('fails') { expect(1).to eq(2) } + it('fails') { File.write('failed', ''); expect(1).to eq(2) } end " write_file_formatted 'spec/slow_spec.rb', " RSpec.describe 'Slow' do + it('waits for the failure') do + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 20 + sleep 0.01 until File.exist?('failed') || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + end 10.times { |i| it(\"example \#{i}\") { sleep 0.15 } } end " @@ -295,7 +305,8 @@ def message(notification) run_rspec "--order defined --fail-fast --parallel=2" expect(last_cmd_exit_status).to eq(1) - expect(last_cmd_stdout).to match(/\b[1-4] examples?, 1 failure/) + expect(last_cmd_stdout).to match(/^\d+ examples?, 1 failure$/) + expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 12 end it 'persists example statuses from every worker and reruns only the failures with --only-failures' do @@ -459,8 +470,8 @@ def write_self_interrupting_group(name, examples_after=2) pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/wedged_spec.rb', :out => 'stdout.txt', :err => File::NULL, :pgroup => true) deadline = deadline_for.call(20) - sleep 0.05 until Dir['worker_*.pid'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline - workers = Dir['worker_*.pid'].map { |file| Integer(File.read(file)) } + sleep 0.05 until worker_pids.size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + workers = worker_pids expect(workers.size).to eq(2) # What `timeout` does: the command first, then its whole process group. @@ -533,8 +544,8 @@ def write_self_interrupting_group(name, examples_after=2) cd('.') do rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/stuck_spec.rb', :out => File::NULL, :err => 'stderr.txt') - expect(wait_until.call(20) { Dir['worker_*.pid'].size == 2 }).to be true - workers = Dir['worker_*.pid'].map { |file| Integer(File.read(file)) } + expect(wait_until.call(20) { worker_pids.size == 2 }).to be true + workers = worker_pids Process.kill(:KILL, pid) Process.wait(pid) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index cc392ce89..b4257f1b1 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -51,6 +51,17 @@ def kill(name, pid) end end + # Fake workers run their examples on threads of this process, side by side: + # those examples stay off rspec-mocks' global space, which this spec's own + # stubs live in and whose scopes are not meant to be shared between threads. + module WithoutMocksLifecycle + def setup_mocks_for_rspec; end + + def verify_mocks_for_rspec; end + + def teardown_mocks_for_rspec; end + end + RSpec.describe Coordinator do let(:config) { RSpec.configuration } let(:world) { RSpec.world } @@ -127,14 +138,20 @@ def run_fake_worker(number, jobs, results, behaviour, &record_exit) :watch_coordinator => false).run end + # A fake worker killed with a signal is gone once its thread has died, + # which a loaded machine can delay: the grace periods are long unless a + # spec is about them, so that a slow death is not escalated. def coordinator(units, worker_count=2, options={}) + units.each { |group| group.include(WithoutMocksLifecycle) } @units = units @process = options.fetch(:process, process) Coordinator.new(runner, units, units, worker_count, { :process => @process, :spawn => method(:spawn), :tick => 0.01, - :term_grace => 0.1, :kill_grace => 0.1, :watchdog_grace => 0.05 }.merge(options)) + :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) end + let(:short_graces) { { :term_grace => 0.1, :kill_grace => 0.1 } } + def passing_group(name) RSpec.describe(name) { it("passes") {} } end @@ -283,14 +300,18 @@ def example_failed(_notification) end end + # A single worker, so that no well-behaved one can miss a deadline this + # short on a loaded machine. describe "a worker that does not exit after finishing" do - before { config.parallel_shutdown_timeout = 0.05 } + before do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = :never_exits + end it "is sent SIGTERM after the shutdown timeout and the run fails with the error exit code" do - worker_behaviours[2] = :never_exits config.error_exit_code = 3 - exit_code = coordinator([passing_group("A"), passing_group("B")]).run + exit_code = coordinator([passing_group("A"), passing_group("B")], 1).run expect(exit_code).to eq(3) expect(process.signals.map(&:first)).to eq(["TERM"]) @@ -299,10 +320,9 @@ def example_failed(_notification) end it "is sent SIGKILL when it survives SIGTERM" do - worker_behaviours[2] = :never_exits process = FakeProcess.new(["TERM"]) - exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :process => process).run + exit_code = coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run expect(exit_code).to eq(config.failure_exit_code) expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) @@ -310,11 +330,10 @@ def example_failed(_notification) end it "is given up on, so the run still ends, when it survives SIGKILL" do - worker_behaviours[2] = :never_exits process = FakeProcess.new(%w[TERM KILL]) started = RSpec::Core::Parallel.monotonic_time - exit_code = coordinator([passing_group("A"), passing_group("B")], 2, :process => process).run + exit_code = coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run expect(exit_code).to eq(config.failure_exit_code) expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) @@ -323,9 +342,12 @@ def example_failed(_notification) end end + # Long enough a timeout for the well-behaved worker to stop within it even + # on a loaded machine. describe "a worker that is asked to stop early but does not" do + before { config.parallel_shutdown_timeout = 0.5 } + it "is terminated after the shutdown timeout so a cancelled run still ends" do - config.parallel_shutdown_timeout = 0.1 config.fail_fast = true worker_behaviours[2] = :partial_frame groups = [failing_group("Fails"), passing_group("Held by the stalled worker")] @@ -335,7 +357,7 @@ def example_failed(_notification) expect(exit_code).to eq(config.failure_exit_code) expect(process.signals.map(&:first)).to eq(["TERM"]) - expect(messages).to include("did not stop within 0.1 seconds of being asked to stop early") + expect(messages).to include("did not stop within 0.5 seconds of being asked to stop early") expect(messages).to include("1 example, 1 failure") end @@ -360,7 +382,6 @@ def failure_once_running(name, &body) end it "fails the example it is stuck in with the reason" do - config.parallel_shutdown_timeout = 0.1 config.fail_fast = true groups = failure_once_running("Stuck") { sleep } stuck = groups.last @@ -370,15 +391,14 @@ def failure_once_running(name, &body) expect(exit_code).to eq(config.failure_exit_code) expect(process.signals.map(&:first)).to eq(["TERM"]) expect(stuck.examples.first.exception.message).to match( - /\AParallel worker 2 \(pid \d+\) did not stop within 0\.1 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM; the rest of Stuck \(.+\) did not run\.\z/ + /\AParallel worker 2 \(pid \d+\) did not stop within 0\.5 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM; the rest of Stuck \(.+\) did not run\.\z/ ) expect(messages).to include("2 examples, 2 failures") end it "ignores what it sends once it was given up on, so that its example is reported once" do - config.parallel_shutdown_timeout = 0.1 config.fail_fast = true - groups = failure_once_running("Late") { sleep 0.5 } + groups = failure_once_running("Late") { sleep 1.5 } exit_code = coordinator(groups, 2, :process => FakeProcess.new(["TERM"]), :term_grace => 5).run @@ -518,7 +538,7 @@ def example_passed(_notification) end it "requeues a group when its worker cannot be reached and lets another worker run it" do - config.parallel_shutdown_timeout = 0.2 + config.parallel_shutdown_timeout = 5 spawn_with_closed_jobs = lambda do |handle, handles| pid = spawn(handle, handles) handle.jobs.io.close_write if handle.number == 2 @@ -546,7 +566,7 @@ def example_passed(_notification) worker_behaviours[1] = :partial_frame worker_behaviours[2] = :partial_frame process = FakeProcess.new(%w[TERM KILL]) - instance = coordinator([passing_group("A"), passing_group("B")], 2, :process => process) + instance = coordinator([passing_group("A"), passing_group("B")], 2, short_graces.merge(:process => process)) allow(instance).to receive(:dispatch_work).and_raise("coordinator bug") expect { instance.run }.to raise_error("coordinator bug") @@ -573,7 +593,8 @@ def example_passed(_notification) signaller.join # The worker that ran A may not have been reaped yet, and gets one too. expect(process.signals.map(&:first).uniq).to eq(["TERM"]) - expect(messages).to include("Received SIGTERM; stopping the workers.", "2 examples, 0 failures") + expect(messages).to include("Received SIGTERM; stopping the workers.") + expect(messages).to match(/^[12] examples?, 0 failures$/) expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") end @@ -582,7 +603,7 @@ def example_passed(_notification) config.parallel_shutdown_timeout = 0.05 allow(process).to receive(:kill).and_raise(Errno::ESRCH) - exit_code = coordinator([passing_group("A"), passing_group("B")]).run + exit_code = coordinator([passing_group("A"), passing_group("B")], 2, short_graces).run expect(exit_code).to eq(config.failure_exit_code) expect(messages).to include("so it was sent SIGTERM", "sending SIGKILL", "giving up on it") @@ -614,9 +635,9 @@ def example_passed(_notification) it "bounds the wait for a worker whose results pipe closed while it was running" do config.parallel_shutdown_timeout = 0.05 - worker_behaviours[2] = :closes_results + worker_behaviours[1] = :closes_results - coordinator([passing_group("A"), passing_group("B")]).run + coordinator([passing_group("A"), passing_group("B")], 1).run expect(process.signals.map(&:first)).to eq(["TERM"]) expect(messages).to include("stopped reporting for 0.05 seconds", "so it was sent SIGTERM") diff --git a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb index 663b81f79..fe06ce3f9 100644 --- a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb @@ -131,9 +131,13 @@ def message = "from accessor #{@required.upcase}" e end + # syntax_suggest's part of it depends on a search that can time out. + detailed = original.detailed_message(:highlight => false) + allow(original).to receive(:detailed_message).and_return(detailed) + rebuilt = round_trip(original) expect(rebuilt).to be_a(SyntaxError) - expect(rebuilt.detailed_message).to eq(original.detailed_message(:highlight => false)) + expect(rebuilt.detailed_message).to eq(detailed) end it "falls back to the message when no detailed message was shipped" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 08e9df4dd..68ad33d72 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -209,6 +209,8 @@ def frames instance = worker(4) blocked_line = __LINE__ + 1 blocked = Thread.new { sleep } + deadline = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 until blocked.status == "sleep" || RSpec::Core::Parallel.monotonic_time > deadline instance.arm_watchdog.join diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 4f66af87c..1a7404163 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -180,8 +180,10 @@ def groups(count) expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) expect(trap("HUP", nil)).to eq("IGNORE") + # How this Ruby reports a signal ignored with `nil` (MRI: `nil`, JRuby: "IGNORE"). + ignored_with_nil = trap("HUP", nil) expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) - expect(trap("HUP", suite_handler)).to be_nil + expect(trap("HUP", suite_handler)).to eq(ignored_with_nil) expect(Parallel.trap_signals(%w[HUP]) {}).to eq({}) expect(trap("HUP", "DEFAULT")).to equal(suite_handler) From 248a02b4af133ff9a09bfb752b7fa19d9a1695a2 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 15:32:00 +0200 Subject: [PATCH 08/58] [rspec-core] Say which example each worker was running when a parallel run is stopped A run stopped from outside -- a CI job's timeout sending SIGTERM to the process group, say -- was most often stuck in an example, but the output only said "Received SIGTERM; stopping the workers.". Found running a large Rails app's suite on Ruby 3.3 on macOS, where a worker hung inside an example (in getaddrinfo, after fork) and finding the example took a native stack sample. The coordinator now reads what the workers already sent and adds a line per busy worker: "Parallel worker 1 (pid 10559) was running ./spec/lib/x_spec.rb:42 (X does y) for 14 minutes 47.3 seconds." Results are unchanged: interrupted examples are not turned into failures. Signalling a worker (escalation step, deadline, kill) moves to WorkerHandle. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 3 ++- rspec-core/README.md | 6 +++-- .../lib/rspec/core/parallel/coordinator.rb | 26 ++++++++++--------- .../lib/rspec/core/parallel/worker_handle.rb | 16 +++++++++++- rspec-core/spec/integration/parallel_spec.rb | 12 ++++++--- .../rspec/core/parallel/coordinator_spec.rb | 20 +++++++++++--- 6 files changed, 60 insertions(+), 23 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index f0081ed74..1dd922059 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -17,7 +17,8 @@ Enhancements: parallel_tests does. A worker that finishes but does not exit is aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, so a stuck `at_exit` handler can no longer hang a run, and a worker that crashes fails the - example it was running instead of taking the run down. Where `fork` is unavailable the run + example it was running instead of taking the run down. A run stopped with SIGTERM or SIGHUP + says which example each worker was running and for how long. Where `fork` is unavailable the run is serial with a warning. `--bisect` runs serially, and the reproduction command it prints now ends with `--no-parallel`. (Illia Zub) diff --git a/rspec-core/README.md b/rspec-core/README.md index 5c7ea5c8b..c404c71cf 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -349,8 +349,10 @@ count as errors outside of examples. Workers also exit on their own if the coordinating process disappears. SIGTERM or SIGHUP to the coordinating process, as `timeout` or a cancelled CI job sends, stops every worker -- even one that cannot tear down -- before the process ends the way a serial run -would; a signal the run was started ignoring (`nohup`), or that your suite -installs a handler for, is left alone. +would, and the output says which example each worker was running and for how +long, which is usually why the run had to be stopped; a signal the run was +started ignoring (`nohup`), or that your suite installs a handler for, is left +alone. `--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 45109a3b0..015409144 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -93,7 +93,7 @@ def event_loop dispatch_work until @workers.all?(&:done?) - return report("Received SIG#{@termination_signal}; stopping the workers.") if @termination_signal + return stop_for_signal if @termination_signal wait_for_results reap_workers @@ -104,6 +104,14 @@ def event_loop end end + # A run stopped from outside -- a CI job's timeout, say -- most often + # was stuck in an example, so it says what each worker was running. + def stop_for_signal + @workers.each { |worker| receive_from(worker) unless worker.done? } + report "Received SIG#{@termination_signal}; stopping the workers." + @workers.each { |worker| (running = worker.running_example_description) && report(running) } + end + def wait_for_results readers = @workers.reject { |worker| worker.done? || worker.results.eof? || worker.results.closed? } @@ -319,10 +327,10 @@ def escalate(worker) when nil overdue(worker) worker.state = :terminating - signal(worker, 'TERM', :term, @term_grace) + worker.signal(@process, 'TERM', :term, now + @term_grace) when :term report "#{worker.description} is still alive #{Parallel.seconds(@term_grace)} after SIGTERM; sending SIGKILL." - signal(worker, 'KILL', :kill, @kill_grace) + worker.signal(@process, 'KILL', :kill, now + @kill_grace) when :kill report "#{worker.description} is still alive #{Parallel.seconds(@kill_grace)} after SIGKILL; giving up on it." abort_unit(worker) @@ -346,7 +354,7 @@ def overdue(worker) def terminate(worker, reason) lost(worker, reason, SENT_TERM) worker.state = :terminating - signal(worker, 'TERM', :term, @term_grace) + worker.signal(@process, 'TERM', :term, now + @term_grace) end # The worker can no longer be relied on. The example it was running, @@ -377,12 +385,6 @@ def error(text) :failure_lines => [text]) end - def signal(worker, name, escalation, grace) - worker.escalation = escalation - worker.deadline = now + grace - worker.kill(@process, name) - end - def abort_unit(worker) if (index = worker.current_unit) @unit_states[index] = :aborted @@ -400,10 +402,10 @@ def terminate_remaining_workers live.each do |worker| worker.state = :terminating - signal(worker, 'TERM', :term, @term_grace) + worker.signal(@process, 'TERM', :term, now + @term_grace) end wait_for_exit(live, @term_grace) - live.reject(&:done?).each { |worker| signal(worker, 'KILL', :kill, @kill_grace) } + live.reject(&:done?).each { |worker| worker.signal(@process, 'KILL', :kill, now + @kill_grace) } wait_for_exit(live, @kill_grace) live.reject(&:done?).each do |worker| diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 8409937e1..2d647f80c 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -23,6 +23,16 @@ def current_example=(example) @current_example_started_at = example && RSpec::Core::Time.now end + # @return [String, nil] the example this worker is in the middle of, + # and for how long it has been running + def running_example_description + return nil unless @current_example + + "#{description} was running #{@current_example.location_rerun_argument || @current_example.id} " \ + "(#{@current_example.full_description}) for " \ + "#{Parallel.seconds(RSpec::Core::Time.now - @current_example_started_at)}." + end + # @return [Array(Example, Hash)] the example this worker was running # when it was lost, and a result failing it with `text` def lost_example_result(text) @@ -87,7 +97,11 @@ def reap(process) [@pid, nil] end - def kill(process, signal_name) + # Sends `signal_name` as the `escalation` step that is followed up at + # `deadline`. + def signal(process, signal_name, escalation, deadline) + @escalation = escalation + @deadline = deadline process.kill(signal_name, @pid) if @pid rescue Errno::ESRCH, Errno::EPERM nil diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 628d02b05..da6c86497 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -450,12 +450,16 @@ def write_self_interrupting_group(name, examples_after=2) expect(last_cmd_stdout).to include("1 example, 0 failures, 1 error occurred outside of examples") end - it 'stops its workers, even ones that cannot tear down, when it and its process group get SIGTERM' do + it 'stops its workers, even ones that cannot tear down, when it and its process group get SIGTERM, saying what they ran' do write_file_formatted 'spec/wedged_spec.rb', " RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } 2.times do |i| RSpec.describe(\"Wedged \#{i}\") do - it('keeps a thread that delays exit') { Thread.new { begin; sleep; ensure; sleep 60; end }; sleep 60 } + it('keeps a thread that delays exit') do + File.write(\"running_\#{i}\", '') + Thread.new { begin; sleep; ensure; sleep 60; end } + sleep 60 + end end end " @@ -470,7 +474,8 @@ def write_self_interrupting_group(name, examples_after=2) pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/wedged_spec.rb', :out => 'stdout.txt', :err => File::NULL, :pgroup => true) deadline = deadline_for.call(20) - sleep 0.05 until worker_pids.size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + sleep 0.05 until (worker_pids.size == 2 && Dir['running_*'].size == 2) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline workers = worker_pids expect(workers.size).to eq(2) @@ -481,6 +486,7 @@ def write_self_interrupting_group(name, examples_after=2) expect(status.termsig).to eq(Signal.list.fetch('TERM')) expect(File.read('stdout.txt')).to include("Received SIGTERM; stopping the workers.") + expect(File.read('stdout.txt').scan(%r{^Parallel worker [12] \(pid \d+\) was running \./spec/wedged_spec\.rb:\d+ \(Wedged [01] keeps a thread that delays exit\) for [\d.]+ seconds?\.$}).size).to eq(2) deadline = deadline_for.call(10) sleep 0.05 until workers.none?(&running) || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline expect(workers.select(&running)).to eq([]) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index b4257f1b1..f2ab15123 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -574,7 +574,7 @@ def example_passed(_notification) expect(messages).to include("survived SIGKILL; giving up on it") end - it "stops the workers when it is sent SIGTERM, then goes the way a serial run would" do + it "stops the workers when it is sent SIGTERM, saying what they were running, then goes the way a serial run would" do handler = nil allow(RSpec::Core::Parallel).to receive(:trap_signals) do |signals, &block| expect(signals).to eq(%w[TERM HUP]) @@ -582,10 +582,17 @@ def example_passed(_notification) { "TERM" => "DEFAULT" } end allow(RSpec::Core::Parallel).to receive(:restore_signals) - groups = [passing_group("A"), RSpec.describe("Stuck") { it("sleeps") { sleep } }] + stuck_started = Queue.new + stuck = RSpec.describe("Stuck") do + it("sleeps") do + stuck_started << true + sleep + end + end + groups = [passing_group("A"), stuck] config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) signaller = Thread.new do - sleep 0.01 until handler && process.signals.empty? && messages.include?(".") + sleep 0.01 until handler && process.signals.empty? && messages.include?(".") && !stuck_started.empty? handler.call("TERM") end @@ -594,7 +601,12 @@ def example_passed(_notification) # The worker that ran A may not have been reaped yet, and gets one too. expect(process.signals.map(&:first).uniq).to eq(["TERM"]) expect(messages).to include("Received SIGTERM; stopping the workers.") - expect(messages).to match(/^[12] examples?, 0 failures$/) + # Not defined in a loaded spec file, so it is named by its id rather than its line. + expect(messages).to match( + /^Parallel worker 2 \(pid \d+\) was running #{Regexp.escape(stuck.examples.first.id)} \(Stuck sleeps\) for [\d.]+ seconds?\.$/ + ) + expect(messages).to match(/^2 examples, 0 failures$/) + expect(stuck.examples.first.execution_result.status).to be_nil expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") end From b2fa100321fcdba8d7d729f65c7c827aaab91a67 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 17:25:21 +0200 Subject: [PATCH 09/58] [rspec-core] Let parallel workers load the spec files themselves with --no-parallel-preload Workers forked after the spec files load inherit whatever loading did. On macOS, after a large Rails suite's spec files have loaded, forked workers crash (SIGABRT, Ruby 2.7) or hang at 100% CPU (Ruby 3.3) in getaddrinfo -> Network.framework -> os_log; a probe without RSpec reproduces it, and workers forked before loading were fine in every run. Load-time database clients are inherited too. With `--no-parallel-preload` (`config.parallel_preload = false`) the workers are forked once the `--require`d files are loaded, before the spec files, and each loads them itself with its TEST_ENV_NUMBER already set; the coordinating process loads them too, for its reporting. Default behaviour is unchanged. - A worker gets work only after reporting a fingerprint of the examples it loaded (ids, locations, descriptions, after filtering) that matches the coordinator's, and no load error the coordinator did not get. Otherwise it is not used and its groups run elsewhere; a load error every process gets goes on like a serial run. - A worker still loading well after the coordinator finished (its load time plus parallel_shutdown_timeout) is terminated; workers there are more of than groups, still loading when the queue is empty, or not needed because the run ended early or is serial, are dismissed with SIGKILL -- they ran nothing. A worker given no work exits without suite hooks or at_exit. - Groups are dispatched by id as well as index. Forking, dismissing and terminating workers move to a WorkerLifecycle module. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 10 +- rspec-core/README.md | 32 ++-- .../features/command_line/parallel.feature | 24 +++ rspec-core/lib/rspec/core/configuration.rb | 24 ++- rspec-core/lib/rspec/core/option_parser.rb | 3 + rspec-core/lib/rspec/core/parallel.rb | 48 +++++- .../lib/rspec/core/parallel/coordinator.rb | 105 +++++-------- rspec-core/lib/rspec/core/parallel/worker.rb | 55 +++++-- .../lib/rspec/core/parallel/worker_handle.rb | 9 +- .../rspec/core/parallel/worker_lifecycle.rb | 122 +++++++++++++++ rspec-core/lib/rspec/core/runner.rb | 10 +- rspec-core/spec/integration/parallel_spec.rb | 68 +++++++++ .../spec/rspec/core/configuration_spec.rb | 12 ++ .../spec/rspec/core/option_parser_spec.rb | 7 + .../rspec/core/parallel/coordinator_spec.rb | 142 +++++++++++++++++- .../spec/rspec/core/parallel/worker_spec.rb | 84 ++++++++++- rspec-core/spec/rspec/core/parallel_spec.rb | 90 ++++++++++- rspec-core/spec/rspec/core/runner_spec.rb | 35 ++++- 18 files changed, 763 insertions(+), 117 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 1dd922059..c24fdbd7f 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -18,9 +18,13 @@ Enhancements: `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, so a stuck `at_exit` handler can no longer hang a run, and a worker that crashes fails the example it was running instead of taking the run down. A run stopped with SIGTERM or SIGHUP - says which example each worker was running and for how long. Where `fork` is unavailable the run - is serial with a warning. `--bisect` runs serially, and the reproduction command it prints - now ends with `--no-parallel`. (Illia Zub) + says which example each worker was running and for how long. With `--no-parallel-preload` + (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec + files are loaded and each loads them itself, so nothing the spec files do while loading is + shared with the workers; a worker that does not load the examples the coordinating process + has is not used. Where `fork` is unavailable the run is serial with a warning. `--bisect` + runs serially, and the reproduction command it prints now ends with `--no-parallel`. + (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index c404c71cf..b28ec940e 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -307,19 +307,33 @@ Workers inherit everything that happened while the suite was loaded. For a Rails application that means `rails_helper` has already evaluated `database.yml` and connected to the database in the coordinating process before any worker exists: setting `TEST_ENV_NUMBER` afterwards does **not** -give workers separate databases, so Rails suites run under `--parallel` share -one test database until rspec-rails adds per-worker database support. Open -connections, servers and similar resources acquired at load time are shared -the same way; acquire them in `before(:suite)` to get one per worker. In -particular, a database client that connected while the spec files loaded (a +give workers separate databases, so such Rails suites run under `--parallel` +share one test database until rspec-rails adds per-worker database support. +Open connections, servers and similar resources acquired at load time are +shared the same way; acquire them in `before(:suite)` to get one per worker. +In particular, a database client that connected while the spec files loaded (a query at load time is enough) hands the same connection to every worker unless it reconnects after `fork` by itself -- as with app servers that preload, make each worker reconnect in a `before(:suite)` hook, e.g. `Mongoid.disconnect_clients`. On macOS, the system networking libraries do not -survive `fork` well either: when loading the spec files makes real requests -(resolving host names from several threads, say), workers can crash in -`getaddrinfo` or hang. Keep such work out of load time -- in `before(:context)` -hooks it runs in the workers instead. +survive `fork` well either: when loading the spec files does a lot of work +(resolving host names, say), workers can crash in `getaddrinfo` or hang. + +Or have every worker load the spec files itself: with `--no-parallel-preload` +(`config.parallel_preload = false` in a `--require`d file) the workers are +forked as soon as the `--require`d files are loaded, before the spec files. +Each worker then loads the spec files with its `TEST_ENV_NUMBER` already set, +and the coordinating process loads them too, for its reporting. Nothing the +spec files do while loading is shared then -- connections, threads, system +libraries they initialize -- and helpers they require (rather than `.rspec`) +load in every worker, so a Rails app whose spec files `require 'rails_helper'` +boots in each worker with its own `TEST_ENV_NUMBER`. The cost is loading the +spec files once per process: more CPU and memory, though they load at the same +time, and whatever they do at load time is done that many times. A worker is +given work only once it has loaded the examples the coordinating process has; +one that loads others (from spec files that define examples depending on the +process, the time or randomness), or is still loading long after the +coordinating process finished, is not used and its groups run elsewhere. Formatters receive examples with their status, timing, pending details, exceptions (class name, message, backtrace, cause chain and aggregated diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index 7641a9a35..d602fcea5 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -19,6 +19,12 @@ Feature: `--parallel` option `database.yml` and connecting -- happens once, in the coordinating process, and is inherited by every worker. + With `--no-parallel-preload` the workers are forked before the spec files + are loaded instead, and each loads them itself with its `TEST_ENV_NUMBER` + already set, so nothing the spec files do while loading is shared. A worker + is given work only once it has loaded the same examples as the coordinating + process. + `--parallel` needs `fork`. Where it is not available (Windows, JRuby), or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a warning and runs serially. @@ -117,6 +123,24 @@ Feature: `--parallel` option And the stderr should contain "Threads at this point:" And the stderr should contain "join" + Scenario: With `--no-parallel-preload` every worker loads the spec files itself + Given a file named "spec/gamma_spec.rb" with: + """ruby + File.write("loaded_by_#{ENV.fetch('TEST_ENV_NUMBER', 'the coordinating process')}", "") + + RSpec.describe "Gamma" do + it "runs in a process that loaded this file" do + expect(File).to exist("loaded_by_#{ENV['TEST_ENV_NUMBER']}") + end + end + """ + When I run `rspec --parallel=2 --no-parallel-preload --order defined` + Then the exit status should be 1 + And the output should contain "5 examples, 1 failure" + And the file "loaded_by_the coordinating process" should exist + And the file "loaded_by_" should exist + And the file "loaded_by_2" should exist + Scenario: Inside another tool's parallel worker RSpec runs serially Given I set the environment variable "TEST_ENV_NUMBER" to "2" When I run `rspec --parallel=2 --order defined spec/beta_spec.rb` diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index e0eab121b..e8a83bb30 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -239,11 +239,11 @@ def fail_fast=(value) # # Workers are forked from this process once the suite has been loaded, # so they inherit everything `spec_helper` and the spec files set up at - # load time -- including connections opened then. `before(:suite)` and - # `after(:suite)` hooks run once in each worker, where - # {#parallel_worker_number} and `ENV['TEST_ENV_NUMBER']` identify the - # worker. Formatters run in this process only and see the notifications - # of a normal run. + # load time -- including connections opened then -- unless + # {#parallel_preload} is off. `before(:suite)` and `after(:suite)` + # hooks run once in each worker, where {#parallel_worker_number} and + # `ENV['TEST_ENV_NUMBER']` identify the worker. Formatters run in this + # process only and see the notifications of a normal run. # # Needs `fork`; where it is unavailable (Windows, JRuby) the run is # serial and a warning is printed, as it is when `TEST_ENV_NUMBER` is @@ -272,6 +272,19 @@ def parallel_workers=(value) # @return [Numeric] add_setting :parallel_shutdown_timeout + # @macro add_setting + # Whether the spec files are loaded before the parallel workers are + # forked (default: `true`), so that they share them. When `false` + # (`--no-parallel-preload`) the workers are forked as soon as the + # `--require`d files are loaded, and each loads the spec files itself + # -- once `ENV['TEST_ENV_NUMBER']` is set -- as this process does for + # its reporting. Nothing the spec files do while loading (connections, + # threads, system libraries they initialize) is then shared with the + # workers, at the cost of loading them once per worker. A worker that + # does not load the same examples as this process is not used. + # @return [Boolean] + add_setting :parallel_preload + # The 1-based number of the parallel worker this process is, or `nil` # in the coordinating process and in serial runs. In workers, # `ENV['TEST_ENV_NUMBER']` is also set, following the convention of @@ -510,6 +523,7 @@ def initialize @fail_if_no_examples = false @parallel_workers = nil @parallel_shutdown_timeout = 30 + @parallel_preload = true @parallel_worker_number = nil @spec_files_loaded = false diff --git a/rspec-core/lib/rspec/core/option_parser.rb b/rspec-core/lib/rspec/core/option_parser.rb index e12ed61e4..2cf9ccd1e 100644 --- a/rspec-core/lib/rspec/core/option_parser.rb +++ b/rspec-core/lib/rspec/core/option_parser.rb @@ -80,6 +80,9 @@ def parser(options) options[:parallel_workers] = parallel_workers_from(argument) end + parser.on('--[no-]parallel-preload', 'Load the spec files before forking the --parallel workers (the default),', + ' or fork first and have every worker load them itself.') { |value| options[:parallel_preload] = value } + parser.on('--[no-]fail-fast[=COUNT]', 'Abort the run after a certain number of failures (1 by default).') do |argument| if argument == true value = 1 diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index c52082686..9b366f269 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -6,6 +6,7 @@ RSpec::Support.require_rspec_core "parallel/results" RSpec::Support.require_rspec_core "parallel/suite_index" RSpec::Support.require_rspec_core "parallel/worker_handle" +RSpec::Support.require_rspec_core "parallel/worker_lifecycle" RSpec::Support.require_rspec_core "parallel/coordinator" module RSpec @@ -27,15 +28,54 @@ class WorkerError < StandardError; end # @return [Coordinator, nil] a coordinator for the groups, or `nil` when # the run should proceed serially (not requested, not possible or not # worth it). Warns when a requested parallel run is not possible. - def self.coordinator_for(runner, example_groups) - requested = requested_worker_count(runner.configuration) + # Workers forked before the spec files were loaded (`started`) are + # used, or dismissed when the run is to be serial or needs fewer. + def self.coordinator_for(runner, example_groups, started=nil) + configuration = runner.configuration + requested = started ? started.worker_count : configuration.parallel_preload? && requested_worker_count(configuration) return nil unless requested - units = units_for(example_groups, runner.configuration) + units = units_for(example_groups, configuration) count = [requested, units.size].min + if count < 2 + started&.dismiss + return nil + end + + (started || Coordinator.new(runner, count)).tap { |coordinator| coordinator.assign(example_groups, units) } + end + + # Forks the workers before the spec files are loaded when they are to + # load the spec files themselves (`config.parallel_preload = false`). + # There is no point in more workers than spec files to load. + # @return [Coordinator, nil] the coordinator of the forked workers + def self.coordinator_before_loading(runner) + configuration = runner.configuration + return nil if configuration.parallel_preload? + + requested = requested_worker_count(configuration) + return nil unless requested + + count = [requested, configuration.files_to_run.uniq.size].min return nil if count < 2 - Coordinator.new(runner, example_groups, units, count) + Coordinator.new(runner, count).tap(&:start) + end + + # A fingerprint of the examples a process loaded -- ids, locations and + # descriptions of those that will run, in load order -- so that a worker + # that loaded the spec files itself is used only if it loaded the same + # examples as the coordinating process. + # @return [String] + def self.suite_digest(world) + require 'digest/sha2' + digest = Digest::SHA256.new + world.example_groups.each do |group| + group.descendant_filtered_examples.each do |example| + digest << example.id << "\0" << example.location << "\0" << example.full_description << "\n" + end + end + digest.hexdigest end # @private diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 015409144..4f685e9da 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -1,14 +1,18 @@ module RSpec module Core module Parallel - # The coordinating side of a parallel run. It forks the workers after the - # suite has been loaded (so they inherit every example group, filter and - # configuration), hands out top-level example groups from a shared queue, - # streams the results onto the configuration's reporter -- the one and - # only place formatters run -- and makes sure that no worker, however it - # misbehaves, can keep the run from finishing. + # The coordinating side of a parallel run. It forks the workers -- after + # the suite has been loaded, so they inherit every example group, filter + # and configuration, or before, so they load the spec files themselves + # (`config.parallel_preload = false`) -- hands out top-level example + # groups from a shared queue, streams the results onto the + # configuration's reporter -- the one and only place formatters run -- + # and makes sure that no worker, however it misbehaves, can keep the run + # from finishing. # @private class Coordinator + include WorkerLifecycle + TICK = 0.25 TERM_GRACE = 5.0 KILL_GRACE = 5.0 @@ -21,12 +25,12 @@ class Coordinator # tear down -- before the coordinator goes the way a serial run would. TERMINATION_SIGNALS = %w[TERM HUP].freeze - def initialize(runner, example_groups, units, worker_count, options={}) + attr_reader :worker_count + + def initialize(runner, worker_count, options={}) @runner = runner @configuration = runner.configuration @world = runner.world - @example_groups = example_groups - @units = units @worker_count = worker_count @process = options.fetch(:process) { Process } @spawn = options.fetch(:spawn) { method(:fork_worker) } @@ -35,12 +39,21 @@ def initialize(runner, example_groups, units, worker_count, options={}) @kill_grace = options.fetch(:kill_grace, KILL_GRACE) @watchdog_grace = options.fetch(:watchdog_grace, WATCHDOG_GRACE) @workers = [] - @queue = (0...@units.size).to_a - @unit_states = Array.new(@units.size, :pending) @results = Results.new @quitting = false @termination_signal = nil @signal_handlers = {} + @started_at = nil + end + + # The groups to run. Workers forked before loading that are more than + # the groups to run are dismissed. + def assign(example_groups, units) + @example_groups = example_groups + @units = units + @queue = (0...@units.size).to_a + @unit_states = Array.new(@units.size, :pending) + @workers.drop(@units.size).each { |worker| dismiss_worker(worker) } end # @return [Integer] the exit code for the run @@ -52,7 +65,7 @@ def run @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) begin - start_workers + @workers.empty? ? start_workers(@units) : expect_loading_workers event_loop ensure terminate_remaining_workers @@ -74,21 +87,6 @@ def run private - def start_workers - Parallel.flush_output(@configuration) - @process.warmup if @process.respond_to?(:warmup) - @worker_count.times { |index| @workers << WorkerHandle.new(index + 1) } - - @workers.each do |worker| - worker.pid = @spawn.call(worker, @workers) - worker.close_child_ends - end - end - - def fork_worker(worker, workers) - worker.fork_worker(@process, @units, workers, @signal_handlers) - end - def event_loop dispatch_work @@ -168,6 +166,7 @@ def handle_control_message(worker, kind, rest) when :unit_finished then unit_finished(worker, *rest) when :non_example_failure then non_example_failure(*rest) when :exiting then worker.exiting = true + when :loaded then loaded(worker, *rest) else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." end end @@ -229,12 +228,11 @@ def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) def dispatch_work @workers.each do |worker| - next unless worker.idle? - - if @quitting || @queue.empty? - stop(worker) - else - dispatch(worker, @queue.shift) + if worker.loading + # Nothing is left to give it once it has loaded. + dismiss_worker(worker) if worker.state == :running && (@quitting || @queue.empty?) + elsif worker.idle? + @quitting || @queue.empty? ? stop(worker) : dispatch(worker, @queue.shift) end end end @@ -242,7 +240,7 @@ def dispatch_work def dispatch(worker, index) worker.current_unit = index @unit_states[index] = :running - return if write(worker, [:run, index]) + return if write(worker, [:run, index, @units[index].id]) @unit_states[index] = :pending @queue.unshift(index) @@ -344,6 +342,8 @@ def overdue(worker) error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ "`config.parallel_shutdown_timeout` if it legitimately needs longer.") + elsif worker.loading + lost(worker, "took longer than #{Parallel.seconds(@load_allowance)}", SENT_TERM) elsif worker.quit_sent lost(worker, "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)", SENT_TERM) else @@ -364,7 +364,10 @@ def terminate(worker, reason) # on is read: it would report the example a second time. def lost(worker, what, consequence='') unit = worker.current_unit && unit_description(worker.current_unit) - if worker.current_example + if worker.loading + worker.loading = false + error("#{worker.description} #{what} while loading the spec files#{consequence}.") + elsif worker.current_example example, result = worker.lost_example_result( "#{worker.description} #{what} while running this example#{consequence}" \ "#{"; the rest of #{unit} did not run" if unit}." @@ -390,37 +393,7 @@ def abort_unit(worker) @unit_states[index] = :aborted worker.current_unit = nil end - @sequencer.abandon(worker.number) - end - - # Runs on every exit path, so an exception in the coordinator can never - # leave workers behind. Bounded by the same TERM/KILL grace periods. - def terminate_remaining_workers - @workers.each { |worker| worker.state = :exited if worker.pid.nil? } - live = @workers.reject(&:done?) - return if live.empty? - - live.each do |worker| - worker.state = :terminating - worker.signal(@process, 'TERM', :term, now + @term_grace) - end - wait_for_exit(live, @term_grace) - live.reject(&:done?).each { |worker| worker.signal(@process, 'KILL', :kill, now + @kill_grace) } - wait_for_exit(live, @kill_grace) - - live.reject(&:done?).each do |worker| - report "#{worker.description} survived SIGKILL; giving up on it." - abort_unit(worker) - worker.state = :abandoned - end - end - - def wait_for_exit(workers, grace) - deadline = now + grace - until workers.all?(&:done?) || now >= deadline - sleep [@tick, 0.05].min - reap_workers - end + @sequencer&.abandon(worker.number) end def report_unrun_units diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 6a1c78a78..e781d3bc9 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -4,7 +4,9 @@ module Parallel # Runs in a forked worker process: executes the example groups the # coordinator dispatches over the jobs channel and streams results back # over the results channel. Suite hooks run once per worker, around all - # the groups it executes. + # the groups it executes; a worker given none exits without running + # them. A worker forked before the spec files were loaded (no + # `example_groups`) loads them itself and reports what it loaded first. # # A worker that has finished its work but does not exit (a non-daemon # thread in `at_exit`, a browser that will not quit, ...) is aborted by @@ -48,8 +50,10 @@ def test_env_number def run watch_coordinator if @watch_coordinator prepare - run_units - @exit.call(0) + load_spec_files unless @example_groups + + # One that ran nothing has no suite hooks or `at_exit` handlers to run. + run_units ? @exit.call(0) : @exit_now.call(0) rescue Errno::EPIPE, IOError # The coordinator is gone; there is nobody left to report to. @exit_now.call(1) @@ -106,13 +110,30 @@ def trap_interrupt end end + # Loads the spec files as the coordinator does after forking, and + # tells it which examples that defined and the first load error, if + # any: it uses the worker only if its own loading went the same way. + # What loading reports is for the coordinator to report. + # @private + def load_spec_files + forwarding = @configuration.reporter + @configuration.reporter = loading = LoadReporter.new(@configuration) + @configuration.load_spec_files + @configuration.reporter = forwarding + @groups_by_id = @world.example_groups.to_h { |group| [group.id, group] } + @results.send([:loaded, Parallel.suite_digest(@world), loading.failure]) + end + + # @return [Boolean] whether it was given anything to run # @private def run_units - @configuration.with_suite_hooks do - while (message = @jobs.receive) - break unless message.first == :run + message = @jobs.receive + return false unless message && message.first == :run - run_unit(message[1]) + @configuration.with_suite_hooks do + while message && message.first == :run + run_unit(message[1], message[2]) + message = @jobs.receive end # Armed before `after(:suite)` hooks so that a hook that never @@ -121,11 +142,13 @@ def run_units end @results.send([:exiting]) + true end # @private - def run_unit(index) - passed = @example_groups.fetch(index).run(@reporter) + def run_unit(index, id) + group = @groups_by_id ? @groups_by_id.fetch(id) : @example_groups.fetch(index) + passed = group.run(@reporter) @results.send([:unit_finished, index, passed ? true : false, !!@world.wants_to_quit, !!@world.non_example_failure]) end @@ -153,6 +176,20 @@ def dump_threads(timeout) end stderr.flush end + + # Reports nothing: while a worker loads the spec files, what they + # report is for the coordinating process, which loads them too, to + # report. Remembers the first error. + # @private + class LoadReporter < Reporter + attr_reader :failure + + def notify_non_example_exception(exception, context_description, _presenter_options={}) + return if @failure + + @failure = "#{context_description} #{exception.class}: #{exception.message}" + end + end end end end diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 2d647f80c..105ef0965 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -6,6 +6,9 @@ module Parallel class WorkerHandle attr_reader :number, :jobs, :results, :child_jobs, :child_results, :current_example, :current_example_started_at attr_accessor :pid, :current_unit, :state, :deadline, :escalation, :exit_status, :quit_sent, :exiting + # Forked before the spec files were loaded and has not yet reported + # what it loaded. + attr_accessor :loading def initialize(number) @number = number @@ -14,6 +17,7 @@ def initialize(number) @state = :running @quit_sent = false @exiting = false + @loading = false @pid = @current_unit = @deadline = @escalation = @exit_status = nil @current_example = @current_example_started_at = nil end @@ -47,7 +51,8 @@ def lost_example_result(text) [example, result] end - # @return [Integer] the pid of the worker process forked to run `units` + # @return [Integer] the pid of the worker process forked to run `units`, + # or to load the spec files itself when there are none yet def fork_worker(process, units, handles, signal_handlers={}) process.fork do # :nocov: - runs in the forked worker process, covered by integration specs @@ -74,7 +79,7 @@ def close_all end def idle? - @state == :running && @current_unit.nil? + @state == :running && @current_unit.nil? && !@loading end def busy? diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb new file mode 100644 index 000000000..c9e98b403 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -0,0 +1,122 @@ +module RSpec + module Core + module Parallel + # The part of the {Coordinator} that forks and ends the worker + # processes. Workers are forked after the spec files have been loaded, + # or before (`config.parallel_preload = false`): then each loads the + # spec files itself and is given work only once it has reported that + # its loading went the way the coordinator's did. Workers that never + # got to run anything are dismissed, and every exit path terminates the + # rest within the TERM/KILL grace periods. + # @private + module WorkerLifecycle + # Forks the workers before the spec files are loaded; each loads them + # itself and reports what it loaded. + def start + @started_at = now + start_workers(nil) + end + + # Ends the workers that are not done -- workers forked before loading + # when the run ends early or goes on serially: they have run nothing. + def dismiss + live = @workers.reject(&:done?) + live.each { |worker| dismiss_worker(worker) } + give_up_unless_gone(live, @kill_grace) + end + + private + + # Without `units` the workers load the spec files themselves. + def start_workers(units) + Parallel.flush_output(@configuration) + @process.warmup if @process.respond_to?(:warmup) + @worker_count.times { |index| @workers << WorkerHandle.new(index + 1) } + + @workers.each do |worker| + worker.loading = units.nil? + worker.pid = @spawn.call(worker, @workers) + worker.close_child_ends + end + end + + def fork_worker(worker, workers) + worker.fork_worker(@process, @units, workers, @signal_handlers) + end + + # Workers started loading the spec files when this process did, so + # they get as long again as it took, and the shutdown timeout on top, + # to report what they loaded; they get work once they have. + def expect_loading_workers + @suite_digest = Parallel.suite_digest(@world) + @load_failed = @world.non_example_failure + deadline = now + (now - @started_at) + @configuration.parallel_shutdown_timeout + @load_allowance = deadline - @started_at + @workers.each { |worker| worker.deadline = deadline if worker.loading } + end + + # A worker that loaded the spec files itself gets work only if its + # loading went the way this process's did: no error this process did + # not get, and the same examples. + def loaded(worker, digest, failure) + worker.loading = false + worker.deadline = nil + if failure && !@load_failed + not_used(worker, "could not load the spec files: #{failure}") + elsif digest != @suite_digest + not_used(worker, "loaded other examples than this process (spec files that define examples depending " \ + "on the process, the time or randomness?); with `--parallel-preload` workers are " \ + "forked after the spec files are loaded") + end + end + + def not_used(worker, reason) + error("#{worker.description} #{reason}, so it was not used.") + dismiss_worker(worker) + end + + # Ends a worker that has run nothing -- no example group, suite hook or + # `at_exit` handler of this run -- so it has nothing to finish. + def dismiss_worker(worker) + worker.loading = false + worker.state = :dismissed + worker.results.close + worker.signal(@process, 'KILL', :kill, now + @kill_grace) + end + + # Runs on every exit path, so an exception in the coordinator can never + # leave workers behind. Bounded by the same TERM/KILL grace periods. + def terminate_remaining_workers + @workers.each { |worker| worker.state = :exited if worker.pid.nil? } + live = @workers.reject(&:done?) + return if live.empty? + + live.each do |worker| + worker.state = :terminating + worker.signal(@process, 'TERM', :term, now + @term_grace) + end + wait_for_exit(live, @term_grace) + live.reject(&:done?).each { |worker| worker.signal(@process, 'KILL', :kill, now + @kill_grace) } + give_up_unless_gone(live, @kill_grace) + end + + def give_up_unless_gone(workers, grace) + wait_for_exit(workers, grace) + workers.reject(&:done?).each do |worker| + report "#{worker.description} survived SIGKILL; giving up on it." + abort_unit(worker) + worker.state = :abandoned + end + end + + def wait_for_exit(workers, grace) + deadline = now + grace + until workers.all?(&:done?) || now >= deadline + sleep [@tick, 0.05].min + reap_workers + end + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index ab1350c99..f58507b47 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -83,25 +83,31 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) # @param err [IO] error stream # @param out [IO] output stream def run(err, out) - setup(err, out) + coordinator = nil + setup(err, out) { coordinator = Parallel.coordinator_before_loading(self) } return @configuration.reporter.exit_early(exit_code) if RSpec.world.wants_to_quit example_groups = @world.ordered_example_groups - coordinator = Parallel.coordinator_for(self, example_groups) + coordinator = Parallel.coordinator_for(self, example_groups, coordinator) (coordinator ? coordinator.run : run_specs(example_groups)).tap do persist_example_statuses end + ensure + # Workers forked before loading that the run did not get to use. + coordinator&.dismiss end # Wires together the various configuration objects and state holders. # # @param err [IO] error stream # @param out [IO] output stream + # @yield once configured, before the spec files are loaded def setup(err, out) configure(err, out) return if RSpec.world.wants_to_quit + yield if block_given? @configuration.load_spec_files ensure @world.announce_filters diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index da6c86497..84a508bf0 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -650,4 +650,72 @@ def alive?(pid) expect(last_cmd_stdout).to include("2 examples, 0 failures") expect(last_cmd_stderr).to include("Ignoring `--parallel`: this process is already a worker of a parallel test run") end + + describe 'with --no-parallel-preload' do + def write_marking_spec_files + 3.times do |i| + write_file_formatted "spec/m#{i}_spec.rb", " + File.write(\"loaded_m#{i}_\#{Process.pid}_\#{ENV.fetch('TEST_ENV_NUMBER', 'none')}\", '') + RSpec.describe('M#{i}') do + it('runs in a process that loaded its file') { expect(File).to exist(\"loaded_m#{i}_\#{Process.pid}_\#{ENV['TEST_ENV_NUMBER']}\") } + end + " + end + end + + def loaders + cd('.') { Dir['loaded_*'].map { |file| file.split('_', 4).last }.tally } + end + + it 'has every worker load the spec files itself, once TEST_ENV_NUMBER is set, and run what it loaded' do + write_marking_spec_files + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(last_cmd_exit_status).to eq(0) + expect(loaders).to eq("none" => 3, "" => 3, "2" => 3) + end + + it 'does not use a worker that loads other examples than the coordinating process' do + write_marking_spec_files + write_file_formatted 'spec/z_spec.rb', " + RSpec.describe('Z') { it('passes') { }; it('only in the second worker') { } if ENV['TEST_ENV_NUMBER'] == '2' } + " + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) loaded other examples than this process/) + expect(last_cmd_stdout).to include("4 examples, 0 failures, 1 error occurred outside of examples") + expect(last_cmd_exit_status).to eq(1) + end + + it 'terminates a worker still loading long after the coordinating process finished, and runs its groups elsewhere' do + write_file_formatted 'spec/spec_helper.rb', "RSpec.configure { |c| c.parallel_shutdown_timeout = 1 }" + 3.times do |i| + write_file_formatted "spec/s#{i}_spec.rb", " + sleep 600 if ENV['TEST_ENV_NUMBER'] == '2' + RSpec.describe('S#{i}') { it('keeps the other worker busy past the deadline') { sleep 0.7 } } + " + end + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec '--require ./spec/spec_helper --parallel=2 --no-parallel-preload' + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 20 + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds while loading the spec files, so it was sent SIGTERM\./) + expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") + end + + it 'goes on like a serial run when a spec file fails to load in every process' do + write_marking_spec_files + write_file_formatted 'spec/broken_spec.rb', "RSpec.describe('Broken') { it('a') { }" + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("While loading ./spec/broken_spec.rb a `raise SyntaxError` occurred") + expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(last_cmd_stdout).not_to include("Parallel worker") + end + end end diff --git a/rspec-core/spec/rspec/core/configuration_spec.rb b/rspec-core/spec/rspec/core/configuration_spec.rb index 99df572a8..1ed6c23e3 100644 --- a/rspec-core/spec/rspec/core/configuration_spec.rb +++ b/rspec-core/spec/rspec/core/configuration_spec.rb @@ -75,6 +75,18 @@ module RSpec::Core end end + describe "#parallel_preload" do + it "defaults to loading the spec files before forking the workers" do + expect(config.parallel_preload?).to be true + end + + it "prefers the value forced from the command line" do + config.parallel_preload = true + config.force(:parallel_preload => false) + expect(config.parallel_preload?).to be false + end + end + describe "#parallel_worker_number" do it "is nil until the parallel runner sets it in a worker" do expect(config.parallel_worker_number).to be_nil diff --git a/rspec-core/spec/rspec/core/option_parser_spec.rb b/rspec-core/spec/rspec/core/option_parser_spec.rb index fc02e19a3..6eb0bb984 100644 --- a/rspec-core/spec/rspec/core/option_parser_spec.rb +++ b/rspec-core/spec/rspec/core/option_parser_spec.rb @@ -365,6 +365,13 @@ module RSpec::Core end end + describe "--[no-]parallel-preload" do + it "chooses whether the spec files are loaded before forking the workers or by each of them" do + expect(Parser.parse(%w[ --parallel-preload ])).to include(:parallel_preload => true) + expect(Parser.parse(%w[ --no-parallel-preload ])).to include(:parallel_preload => false) + end + end + describe "--bisect" do it "sets the `:bisect` option" do options = Parser.parse(%w[ --bisect ]) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index f2ab15123..19d795500 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -15,6 +15,10 @@ def signaled? attr_reader :signals, :warmups + def threads + @threads.values + end + def initialize(survives=[]) @threads = {} @statuses = {} @@ -96,7 +100,10 @@ def spawn(handle, _handles) when :garbage then stall_after { results.io.write("garbage\n") } when :partial_frame then stall_after { results.io.write("999\nabc") } when :closes_results then stall_after { results.close } + when :never_loads then sleep + when :dies_loading then Thread.current.kill else + report_loaded(results, behaviour) if handle.loading send_unknown_ids(results) if behaviour == :unknown_ids run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } end @@ -114,6 +121,14 @@ def stall_after sleep end + # What a worker forked before loading reports once it has loaded the spec + # files: by default, the examples this process has. + def report_loaded(results, behaviour) + digest = behaviour == :loads_other_examples ? "other examples" : RSpec::Core::Parallel.suite_digest(world) + failure = "An error occurred while loading ./spec/x_spec.rb. RuntimeError: boom" if behaviour == :fails_to_load + results.send([:loaded, digest, failure]) + end + def send_unknown_ids(results) results.send([:example_started, "./nope_spec.rb[1:1]"]) results.send([:example_started, "./nope_spec.rb[1:1]"]) @@ -141,13 +156,22 @@ def run_fake_worker(number, jobs, results, behaviour, &record_exit) # A fake worker killed with a signal is gone once its thread has died, # which a loaded machine can delay: the grace periods are long unless a # spec is about them, so that a slow death is not escalated. - def coordinator(units, worker_count=2, options={}) + def new_coordinator(units, worker_count, options) units.each { |group| group.include(WithoutMocksLifecycle) } @units = units @process = options.fetch(:process, process) - Coordinator.new(runner, units, units, worker_count, - { :process => @process, :spawn => method(:spawn), :tick => 0.01, - :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) + Coordinator.new(runner, worker_count, { :process => @process, :spawn => method(:spawn), :tick => 0.01, + :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) + end + + def coordinator(units, worker_count=2, options={}) + new_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units) } + end + + # Started the way `--no-parallel-preload` starts it: the workers are forked + # before the groups to run are known, and load the spec files themselves. + def started_coordinator(units, worker_count=2, options={}) + new_coordinator(units, worker_count, options).tap(&:start) end let(:short_graces) { { :term_grace => 0.1, :kill_grace => 0.1 } } @@ -409,6 +433,116 @@ def failure_once_running(name, &body) end end + describe "with workers forked before the spec files are loaded" do + def run_started(units, worker_count=2, options={}) + started_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units) }.run + end + + it "gives work to the workers that report having loaded the examples this process has" do + groups = %w[A B C].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + expect(process.signals).to be_empty + end + + it "does not use a worker that loaded other examples, and runs its share elsewhere" do + config.error_exit_code = 5 + worker_behaviours[2] = :loads_other_examples + groups = %w[A B C].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(5) + expect(messages).to match(/Parallel worker 2 \(pid \d+\) loaded other examples than this process .*--parallel-preload.*, so it was not used\./) + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(process.signals.map(&:first)).to eq(["KILL"]) + end + + it "does not use a worker whose loading failed when this process's did not" do + worker_behaviours[2] = :fails_to_load + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to include("Parallel worker 2 (pid #{process.signals.first.last}) could not load the spec files: " \ + "An error occurred while loading ./spec/x_spec.rb. RuntimeError: boom, so it was not used.") + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "uses a worker whose loading failed the way this process's did, like a serial run goes on" do + worker_behaviours[2] = :fails_to_load + groups = %w[A B].map { |name| passing_group(name) } + world.non_example_failure = true + + run_started(groups) + + expect(messages).not_to include("could not load") + expect(process.signals).to be_empty + expect(groups.flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) + end + + it "terminates a worker still loading well after this process finished, and runs its share elsewhere" do + config.parallel_shutdown_timeout = 0.1 + worker_behaviours[2] = :never_loads + groups = %w[A B C].map { |name| RSpec.describe(name) { it("takes a while") { sleep 0.2 } } } + + run_started(groups) + + expect(messages).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds while loading the spec files, so it was sent SIGTERM\./) + expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") + expect(process.signals.map(&:first)).to eq(["TERM"]) + end + + it "dismisses a worker still loading once nothing is left for it" do + worker_behaviours[2] = :never_loads + groups = %w[A B].map { |name| passing_group(name) } + + exit_code = run_started(groups) + + expect(exit_code).to eq(0) + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(messages).not_to include("Parallel worker") + end + + it "reports a worker that exits while loading" do + worker_behaviours[2] = :dies_loading + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to match(/Parallel worker 2 \(pid \d+\) exited unexpectedly \(exit status 1\) while loading the spec files\./) + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + + it "dismisses the workers there are more of than groups to run" do + groups = %w[A B].map { |name| passing_group(name) } + + exit_code = run_started(groups, 3) + + expect(exit_code).to eq(0) + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(messages).to include("2 examples, 0 failures") + end + + it "dismisses every worker when the run does not need them, giving up on one that survives SIGKILL" do + worker_behaviours[1] = worker_behaviours[2] = :never_loads + survivor = FakeProcess.new(["KILL"]) + instance = started_coordinator([passing_group("A")], 2, :process => survivor, :kill_grace => 0.1) + + instance.dismiss + + expect(survivor.signals.map(&:first)).to eq(%w[KILL KILL]) + expect(messages.scan("survived SIGKILL; giving up on it.").size).to eq(2) + instance.dismiss + expect(survivor.signals.size).to eq(2) + ensure + survivor.threads.each(&:kill) + end + end + it "terminates a worker that sends data it cannot decode" do worker_behaviours[2] = :garbage diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 68ad33d72..c5d10f535 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -82,7 +82,7 @@ def frames instance.run expect(instance).to have_received(:watch_coordinator) - expect(exits).to eq([[:exit, 0]]) + expect(exits).to eq([[:exit_now, 0]]) end end @@ -153,21 +153,28 @@ def frames expect(all.map(&:first)).to include(:group_started, :example_started, :example_finished, :group_finished) end - it "treats a quit request received while idle like stop" do - jobs.last.send([:quit]) + it "exits at once, without running suite hooks or at_exit handlers, when its first job is not a group to run" do + hooks = [] + RSpec.configuration.before(:suite) { hooks << :before_suite } + RSpec.configuration.after(:suite) { hooks << :after_suite } - worker.run + [[:stop], [:quit]].each do |job| + exits.clear + jobs.last.send(job) + worker.run + expect(exits).to eq([[:exit_now, 0]]) + end - expect(exits).to eq([[:exit, 0]]) - expect(frames).to eq([[:exiting]]) + expect(hooks).to eq([]) + expect(frames).to eq([]) end - it "stops when the coordinator goes away" do + it "exits at once when the coordinator goes away before giving it anything" do jobs.last.close worker.run - expect(exits).to eq([[:exit, 0]]) + expect(exits).to eq([[:exit_now, 0]]) end it "reports a quitting world and non-example failures with the unit" do @@ -183,6 +190,7 @@ def frames it "exits immediately without reporting when the results pipe is gone" do results.first.close + jobs.last.send([:run, 0]) jobs.last.send([:stop]) worker.run @@ -195,6 +203,7 @@ def frames instance = worker RSpec.configuration.after(:suite) { armed_during_after_suite = instance.instance_variable_get(:@watchdog_armed) } allow(instance).to receive(:arm_watchdog) { instance.instance_variable_set(:@watchdog_armed, true) } + jobs.last.send([:run, 0]) jobs.last.send([:stop]) instance.run @@ -203,6 +212,65 @@ def frames end end + describe "forked before the spec files were loaded" do + def loading_worker + Worker.new(2, nil, jobs.first, results.last, + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :watch_coordinator => false) + end + + def write_spec_file(dir, content) + File.join(dir, "loaded_spec.rb").tap do |path| + File.write(path, content) + RSpec.configuration.files_or_directories_to_run = [path] + end + end + + # The worker blocks on the jobs channel once it has loaded the files. + def run_in_thread(instance) + Thread.new { instance.run }.tap { |thread| thread.report_on_exception = false } + end + + it "loads them itself, reports what it loaded, then runs the groups it is given by id" do + Dir.mktmpdir do |dir| + write_spec_file(dir, "RSpec.describe('Loaded here') { it('passes') { } }") + thread = run_in_thread(loading_worker) + + loaded = results.first.receive + group = RSpec.world.example_groups.first + expect(loaded).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world), nil]) + expect(group.description).to eq("Loaded here") + + jobs.last.send([:run, 0, group.id]) + jobs.last.send([:stop]) + expect(thread.join(10)).to be_truthy + + expect(exits).to eq([[:exit, 0]]) + expect(frames).to include([:unit_finished, 0, true, false, false]) + end + end + + it "reports the first load error with what it loaded, and nothing else loading reported" do + Dir.mktmpdir do |dir| + write_spec_file(dir, "RSpec.deprecate('old_thing')\nraise 'boom at load'\n") + thread = run_in_thread(loading_worker) + + kind, digest, failure = results.first.receive + jobs.last.send([:stop]) + expect(thread.join(10)).to be_truthy + + expect([kind, digest]).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world)]) + expect(failure).to match(/\AAn error occurred while loading .*loaded_spec\.rb\. RuntimeError: boom at load\z/) + expect(frames).to eq([]) + expect(RSpec.world.non_example_failure).to be_falsey + expect(exits).to eq([[:exit_now, 0]]) + end + end + end + describe "the watchdog" do it "dumps every thread's backtrace to stderr and aborts the process once the timeout passes" do RSpec.configuration.parallel_shutdown_timeout = 0.05 diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 1a7404163..1a5321f8e 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -22,15 +22,18 @@ def groups(count) it "uses one worker per processor for :auto", :skip => !RSpec::Support::RubyFeatures.fork_supported? do config.parallel_workers = :auto allow(Parallel).to receive(:default_worker_count).and_return(2) - expect(Parallel::Coordinator).to receive(:new).with(runner, anything, anything, 2).and_return(:coordinator) + coordinator = instance_double(Parallel::Coordinator, :assign => nil) + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(coordinator) - expect(Parallel.coordinator_for(runner, groups(5))).to eq(:coordinator) + expect(Parallel.coordinator_for(runner, groups(5))).to eq(coordinator) end it "never starts more workers than there are groups with examples to run", :skip => !RSpec::Support::RubyFeatures.fork_supported? do config.parallel_workers = 8 empty = RSpec.describe("empty") - expect(Parallel::Coordinator).to receive(:new).with(runner, anything, anything, 3).and_return(:coordinator) + coordinator = instance_double(Parallel::Coordinator) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(coordinator) + expect(coordinator).to receive(:assign).with(anything, satisfy { |units| units.size == 3 }) Parallel.coordinator_for(runner, groups(3) + [empty]) end @@ -73,6 +76,87 @@ def groups(count) expect(Parallel.coordinator_for(runner, groups(3))).to be_nil end end + + context "with workers forked before the spec files were loaded" do + let(:started) { instance_double(Parallel::Coordinator, :worker_count => 3, :assign => nil, :dismiss => nil) } + + it "assigns them the groups to run" do + all = groups(5) + + expect(Parallel.coordinator_for(runner, all, started)).to be(started) + expect(started).to have_received(:assign).with(all, all) + end + + it "dismisses them when fewer than two groups are to run" do + expect(Parallel.coordinator_for(runner, groups(1), started)).to be_nil + expect(started).to have_received(:dismiss) + end + + it "runs serially, without warning again, when they could not be forked" do + config.parallel_workers = 2 + config.parallel_preload = false + expect(RSpec).not_to receive(:warning) + + expect(Parallel.coordinator_for(runner, groups(3), nil)).to be_nil + end + end + end + + describe ".coordinator_before_loading" do + before { allow(config).to receive(:files_to_run).and_return(%w[a_spec.rb b_spec.rb c_spec.rb]) } + + it "forks nothing when the spec files are to be loaded first, as by default" do + config.parallel_workers = 2 + expect(Parallel::Coordinator).not_to receive(:new) + + expect(Parallel.coordinator_before_loading(runner)).to be_nil + end + + context "when every worker is to load the spec files itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + before { config.parallel_preload = false } + + it "starts at most as many workers as there are spec files" do + config.parallel_workers = 8 + coordinator = instance_double(Parallel::Coordinator, :start => nil) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(coordinator) + + expect(Parallel.coordinator_before_loading(runner)).to be(coordinator) + expect(coordinator).to have_received(:start) + end + + it "starts none when parallel runs are not requested, or for fewer than two spec files" do + expect(Parallel.coordinator_before_loading(runner)).to be_nil + + config.parallel_workers = 4 + allow(config).to receive(:files_to_run).and_return(%w[a_spec.rb a_spec.rb]) + expect(Parallel.coordinator_before_loading(runner)).to be_nil + end + end + end + + describe ".suite_digest" do + it "fingerprints the ids, locations and descriptions of the examples to run, in load order" do + RSpec.describe("A") do + it("one") {} + it("two") {} + end + RSpec.describe("B") { it("three") {} } + first = Parallel.suite_digest(RSpec.world) + + expect(Parallel.suite_digest(RSpec.world)).to eq(first) + RSpec.world.example_groups.last.examples.first.metadata[:description] = "changed" + RSpec.world.example_groups.last.examples.first.metadata[:full_description] = "B changed" + expect(Parallel.suite_digest(RSpec.world)).not_to eq(first) + end + + it "leaves out the examples that are filtered out" do + RSpec.describe("A") { it("one") {} } + unfiltered = Parallel.suite_digest(RSpec.world) + RSpec.describe("B") { it("skipped by the filter", :slow) {} } + RSpec.world.filter_manager.exclude(:slow => true) + + expect(Parallel.suite_digest(RSpec.world)).to eq(unfiltered) + end end describe ".default_worker_count" do diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index c66c0a480..e1349a2e8 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -367,8 +367,8 @@ def interrupt RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } allow(config).to receive(:load_spec_files) - coordinator = instance_double(Parallel::Coordinator, :run => 17) - expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups).and_return(coordinator) + coordinator = instance_double(Parallel::Coordinator, :run => 17, :dismiss => nil) + expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups, nil).and_return(coordinator) config.example_status_persistence_file_path = "statuses.txt" expect(ExampleStatusPersister).to receive(:persist).with(world.all_examples, "statuses.txt") @@ -378,6 +378,37 @@ def interrupt expect(runner.run(err, out)).to eq(17) end + it 'forks workers that are to load the spec files themselves before loading them, and hands them the groups' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + coordinator = instance_double(Parallel::Coordinator, :run => 0, :dismiss => nil) + expect(Parallel).to receive(:coordinator_before_loading).ordered.and_return(coordinator) + expect(config).to receive(:load_spec_files).ordered + expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups, coordinator). + and_return(coordinator) + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + + it 'dismisses workers forked before loading when the run ends early' do + coordinator = instance_double(Parallel::Coordinator, :dismiss => nil) + allow(Parallel).to receive(:coordinator_before_loading).and_return(coordinator) + allow(config).to receive(:load_spec_files) { world.wants_to_quit = true } + expect(Parallel).not_to receive(:coordinator_for) + + Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) + + expect(coordinator).to have_received(:dismiss) + end + + it 'dismisses workers forked before loading when loading raises' do + coordinator = instance_double(Parallel::Coordinator, :dismiss => nil) + allow(Parallel).to receive(:coordinator_before_loading).and_return(coordinator) + allow(config).to receive(:load_spec_files).and_raise(SystemExit) + + expect { Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) }.to raise_error(SystemExit) + expect(coordinator).to have_received(:dismiss) + end + it 'supports a test-queue like subclass that can perform setup once and run different sets of example groups multiple times' do order = [] From 986d0fab35b0d302b7191d869dd8b2ba1e2f7591 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 18:09:16 +0200 Subject: [PATCH 10/58] [rspec-core] Keep workers whose examples differ only in description, and warn when preload settings come too late With `--no-parallel-preload`, a worker is used if it loaded the same example ids and locations as the coordinating process; descriptions computed at load time (from the time or a temporary file's path) no longer get every worker rejected. A worker that loaded other examples is reported with the first spec file that differs. `parallel_workers` or `parallel_preload = false` set while the spec files load (in a file they require) comes too late to fork before loading: the workers are forked after loading, with a warning saying where to set it instead. `--only-failures` without a status file forks nothing early, since that run aborts with `exit!`. The worker count is no longer capped by the number of spec files. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/README.md | 31 ++--- rspec-core/lib/rspec/core/configuration.rb | 6 +- rspec-core/lib/rspec/core/parallel.rb | 60 ++++------ rspec-core/lib/rspec/core/parallel/plan.rb | 69 +++++++++++ .../rspec/core/parallel/worker_lifecycle.rb | 8 +- rspec-core/lib/rspec/core/runner.rb | 8 +- rspec-core/spec/integration/parallel_spec.rb | 56 ++++++++- .../rspec/core/parallel/coordinator_spec.rb | 15 ++- .../spec/rspec/core/parallel/plan_spec.rb | 108 ++++++++++++++++++ rspec-core/spec/rspec/core/parallel_spec.rb | 78 +++---------- rspec-core/spec/rspec/core/runner_spec.rb | 52 +++++---- 11 files changed, 344 insertions(+), 147 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/plan.rb create mode 100644 rspec-core/spec/rspec/core/parallel/plan_spec.rb diff --git a/rspec-core/README.md b/rspec-core/README.md index b28ec940e..62644b729 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -320,20 +320,25 @@ survive `fork` well either: when loading the spec files does a lot of work (resolving host names, say), workers can crash in `getaddrinfo` or hang. Or have every worker load the spec files itself: with `--no-parallel-preload` -(`config.parallel_preload = false` in a `--require`d file) the workers are +(or `config.parallel_preload = false` in a `--require`d file) the workers are forked as soon as the `--require`d files are loaded, before the spec files. -Each worker then loads the spec files with its `TEST_ENV_NUMBER` already set, -and the coordinating process loads them too, for its reporting. Nothing the -spec files do while loading is shared then -- connections, threads, system -libraries they initialize -- and helpers they require (rather than `.rspec`) -load in every worker, so a Rails app whose spec files `require 'rails_helper'` -boots in each worker with its own `TEST_ENV_NUMBER`. The cost is loading the -spec files once per process: more CPU and memory, though they load at the same -time, and whatever they do at load time is done that many times. A worker is -given work only once it has loaded the examples the coordinating process has; -one that loads others (from spec files that define examples depending on the -process, the time or randomness), or is still loading long after the -coordinating process finished, is not used and its groups run elsewhere. +This and the worker count must be known by then: set in a helper that the spec +files require, they come too late, and the workers are forked after loading, +with a warning. Each worker then loads the spec files with its +`TEST_ENV_NUMBER` already set, and the coordinating process loads them too, +for its reporting. Nothing the spec files do while loading is shared then -- +connections, threads, system libraries they initialize -- and helpers they +require (rather than `.rspec`) load in every worker, so a Rails app whose spec +files `require 'rails_helper'` boots in each worker with its own +`TEST_ENV_NUMBER`. The cost is loading the spec files once per process: more +CPU and memory, though they load at the same time, and whatever they do at +load time is done that many times. A worker is given work only once it has +loaded the examples the coordinating process has (the same ids and locations; +descriptions computed while loading, from the time say, are the coordinating +process's). One that loads others, gets a load error the coordinating process +did not get, or is still loading long after the coordinating process finished +is not used: that counts as an error outside of examples, and its groups run +in the other workers -- or, when no worker is left, do not run. Formatters receive examples with their status, timing, pending details, exceptions (class name, message, backtrace, cause chain and aggregated diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index e8a83bb30..237d037b8 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -281,7 +281,11 @@ def parallel_workers=(value) # its reporting. Nothing the spec files do while loading (connections, # threads, system libraries they initialize) is then shared with the # workers, at the cost of loading them once per worker. A worker that - # does not load the same examples as this process is not used. + # does not load the same examples as this process is not used. It, and + # {#parallel_workers}, have to be set before the spec files load (on the + # command line, in `.rspec` or in a `--require`d file); set while they + # load, they come too late, and the workers are forked after loading, + # with a warning. # @return [Boolean] add_setting :parallel_preload diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index 9b366f269..e0186718e 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -8,6 +8,7 @@ RSpec::Support.require_rspec_core "parallel/worker_handle" RSpec::Support.require_rspec_core "parallel/worker_lifecycle" RSpec::Support.require_rspec_core "parallel/coordinator" +RSpec::Support.require_rspec_core "parallel/plan" module RSpec module Core @@ -25,57 +26,38 @@ module Parallel # stop or exit in time. class WorkerError < StandardError; end - # @return [Coordinator, nil] a coordinator for the groups, or `nil` when - # the run should proceed serially (not requested, not possible or not - # worth it). Warns when a requested parallel run is not possible. - # Workers forked before the spec files were loaded (`started`) are - # used, or dismissed when the run is to be serial or needs fewer. - def self.coordinator_for(runner, example_groups, started=nil) - configuration = runner.configuration - requested = started ? started.worker_count : configuration.parallel_preload? && requested_worker_count(configuration) + # @return [Coordinator, nil] a coordinator for the loaded groups, its + # workers to be forked once it runs, or `nil` when the run should + # proceed serially (not requested, not possible or not worth it). + # Warns when a requested parallel run is not possible. See {Plan} for + # workers forked before loading. + def self.coordinator_for(runner, example_groups) + requested = requested_worker_count(runner.configuration) return nil unless requested - units = units_for(example_groups, configuration) + units = units_for(example_groups, runner.configuration) count = [requested, units.size].min - if count < 2 - started&.dismiss - return nil - end - - (started || Coordinator.new(runner, count)).tap { |coordinator| coordinator.assign(example_groups, units) } - end - - # Forks the workers before the spec files are loaded when they are to - # load the spec files themselves (`config.parallel_preload = false`). - # There is no point in more workers than spec files to load. - # @return [Coordinator, nil] the coordinator of the forked workers - def self.coordinator_before_loading(runner) - configuration = runner.configuration - return nil if configuration.parallel_preload? - - requested = requested_worker_count(configuration) - return nil unless requested - - count = [requested, configuration.files_to_run.uniq.size].min return nil if count < 2 - Coordinator.new(runner, count).tap(&:start) + Coordinator.new(runner, count).tap { |coordinator| coordinator.assign(example_groups, units) } end - # A fingerprint of the examples a process loaded -- ids, locations and - # descriptions of those that will run, in load order -- so that a worker + # Fingerprints of the examples a process loaded -- ids and locations of + # those that will run, in load order -- per spec file, so that a worker # that loaded the spec files itself is used only if it loaded the same - # examples as the coordinating process. - # @return [String] + # examples as the coordinating process, and otherwise the files that + # differ can be named. Descriptions are left out: results are reported + # on the coordinating process's examples, so one computed at load time + # (from the time, a temporary file's path, ...) is simply its own. + # @return [Hash{String => String}] by the file of the top-level group def self.suite_digest(world) require 'digest/sha2' - digest = Digest::SHA256.new + digests = Hash.new { |hash, file| hash[file] = Digest::SHA256.new } world.example_groups.each do |group| - group.descendant_filtered_examples.each do |example| - digest << example.id << "\0" << example.location << "\0" << example.full_description << "\n" - end + digest = digests[group.metadata[:file_path]] + group.descendant_filtered_examples.each { |example| digest << example.id << "\0" << example.location << "\n" } end - digest.hexdigest + digests.transform_values(&:hexdigest) end # @private diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb new file mode 100644 index 000000000..51d8c5c49 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -0,0 +1,69 @@ +module RSpec + module Core + module Parallel + # Decides whether a run is parallel and when its workers are forked: + # once the spec files have been loaded, as by default, or before, when + # each worker is to load them itself (`config.parallel_preload = false`). + # It keeps what it decided before loading for when they have loaded, + # and the workers it forked until the run uses or dismisses them. + # @private + class Plan + def initialize(runner) + @runner = runner + @configuration = runner.configuration + @coordinator = nil + @decided = false + @postponed_by = nil + end + + # Called once the run is configured, before the spec files are loaded. + def before_loading + return if @configuration.parallel_preload? + return @postponed_by = '`config.parallel_workers`' if @configuration.parallel_workers.nil? + # Such a run aborts, with `exit!`, as soon as the spec files have loaded. + if @configuration.only_failures_but_not_configured? + return @postponed_by = '`config.example_status_persistence_file_path` (for `--only-failures`)' + end + + @decided = true + requested = Parallel.requested_worker_count(@configuration) + @coordinator = Coordinator.new(@runner, requested).tap(&:start) if requested && requested >= 2 + end + + # @return [Coordinator, nil] the coordinator to run the loaded groups, + # or `nil` for a serial run + def coordinator_for(example_groups) + unless @decided + coordinator = Parallel.coordinator_for(@runner, example_groups) + warn_late if coordinator && !@configuration.parallel_preload? + return coordinator + end + return nil unless @coordinator + + units = Parallel.units_for(example_groups, @configuration) + if units.size < 2 + dismiss + return nil + end + + @coordinator.assign(example_groups, units) + @coordinator + end + + # Ends the workers forked before loading that the run did not use. + def dismiss + @coordinator&.dismiss + end + + private + + def warn_late + RSpec.warning "The parallel workers are forked after the spec files were loaded, not before them as " \ + "`config.parallel_preload = false` asks: #{@postponed_by || 'it'} was set while they were " \ + "loading. Set it before they load: on the command line, in `.rspec` or in a `--require`d " \ + "file.", :call_site => nil + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index c9e98b403..0cac11c90 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -64,9 +64,11 @@ def loaded(worker, digest, failure) if failure && !@load_failed not_used(worker, "could not load the spec files: #{failure}") elsif digest != @suite_digest - not_used(worker, "loaded other examples than this process (spec files that define examples depending " \ - "on the process, the time or randomness?); with `--parallel-preload` workers are " \ - "forked after the spec files are loaded") + files = (digest.keys | @suite_digest.keys).reject { |file| digest[file] == @suite_digest[file] } + not_used(worker, "loaded other examples than this process from #{files.first}" \ + "#{" and #{Formatters::Helpers.pluralize(files.size - 1, 'more spec file')}" if files.size > 1} " \ + "(examples defined depending on the process, the time or randomness?); with " \ + "`--parallel-preload` workers are forked after the spec files are loaded") end end diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index f58507b47..b4306ac53 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -83,19 +83,19 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) # @param err [IO] error stream # @param out [IO] output stream def run(err, out) - coordinator = nil - setup(err, out) { coordinator = Parallel.coordinator_before_loading(self) } + parallel = Parallel::Plan.new(self) + setup(err, out) { parallel.before_loading } return @configuration.reporter.exit_early(exit_code) if RSpec.world.wants_to_quit example_groups = @world.ordered_example_groups - coordinator = Parallel.coordinator_for(self, example_groups, coordinator) + coordinator = parallel.coordinator_for(example_groups) (coordinator ? coordinator.run : run_specs(example_groups)).tap do persist_example_statuses end ensure # Workers forked before loading that the run did not get to use. - coordinator&.dismiss + parallel&.dismiss end # Wires together the various configuration objects and state holders. diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 84a508bf0..998ce67a6 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -685,7 +685,7 @@ def loaders run_rspec '--parallel=2 --no-parallel-preload' - expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) loaded other examples than this process/) + expect(last_cmd_stdout).to match(%r{Parallel worker 2 \(pid \d+\) loaded other examples than this process from \./spec/z_spec\.rb \(}) expect(last_cmd_stdout).to include("4 examples, 0 failures, 1 error occurred outside of examples") expect(last_cmd_exit_status).to eq(1) end @@ -707,6 +707,60 @@ def loaders expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") end + it 'warns, and forks after loading, when turned off from a spec file, too late' do + write_file_formatted 'spec/late_spec.rb', " + RSpec.configure { |c| c.parallel_preload = false } + RSpec.describe('Late') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + RSpec.describe('Later') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + " + + run_rspec '--parallel=2 spec/late_spec.rb' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("forked after the spec files were loaded, not before them as " \ + "`config.parallel_preload = false` asks: it was set while they were loading") + end + + it 'warns, and forks after loading, when the worker count is set from a spec file, too late' do + write_file_formatted 'spec/late_spec.rb', " + RSpec.configure { |c| c.parallel_workers = 2 } + RSpec.describe('Late') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + RSpec.describe('Later') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + " + + run_rspec '--no-parallel-preload spec/late_spec.rb' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stderr).to include("`config.parallel_workers` was set while they were loading") + end + + it 'runs the groups of a single spec file across the workers' do + write_file_formatted 'spec/one_spec.rb', " + 3.times { |i| RSpec.describe(\"G\#{i}\") { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } } + " + + run_rspec '--parallel=2 --no-parallel-preload spec/one_spec.rb' + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(last_cmd_stderr).to eq("") + end + + it 'forks nothing for an --only-failures run that is about to abort for want of a status file' do + write_marking_spec_files + + # A real process: the abort ends it with `exit!`. + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', '--no-parallel-preload', '--only-failures', + :out => 'stdout.txt', :err => 'stderr.txt') + _, status = Process.wait2(pid) + + expect(status.exitstatus).to eq(1) + expect(File.read('stderr.txt') + File.read('stdout.txt')).to include("example_status_persistence_file_path") + end + expect(loaders).to eq("none" => 3) + end + it 'goes on like a serial run when a spec file fails to load in every process' do write_marking_spec_files write_file_formatted 'spec/broken_spec.rb', "RSpec.describe('Broken') { it('a') { }" diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 19d795500..e2b0c7cec 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -124,7 +124,9 @@ def stall_after # What a worker forked before loading reports once it has loaded the spec # files: by default, the examples this process has. def report_loaded(results, behaviour) - digest = behaviour == :loads_other_examples ? "other examples" : RSpec::Core::Parallel.suite_digest(world) + digest = RSpec::Core::Parallel.suite_digest(world) + digest = digest.merge("./spec/other_spec.rb" => "other examples") if behaviour == :loads_other_examples + digest = digest.merge("./spec/a_spec.rb" => "a", "./spec/b_spec.rb" => "b") if behaviour == :loads_other_files failure = "An error occurred while loading ./spec/x_spec.rb. RuntimeError: boom" if behaviour == :fails_to_load results.send([:loaded, digest, failure]) end @@ -456,11 +458,20 @@ def run_started(units, worker_count=2, options={}) exit_code = run_started(groups) expect(exit_code).to eq(5) - expect(messages).to match(/Parallel worker 2 \(pid \d+\) loaded other examples than this process .*--parallel-preload.*, so it was not used\./) + expect(messages).to match(%r{Parallel worker 2 \(pid \d+\) loaded other examples than this process from \./spec/other_spec\.rb \(.*--parallel-preload.*, so it was not used\.}) expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") expect(process.signals.map(&:first)).to eq(["KILL"]) end + it "names the first spec file a worker loaded differently and counts the others" do + worker_behaviours[2] = :loads_other_files + groups = %w[A B].map { |name| passing_group(name) } + + run_started(groups) + + expect(messages).to match(%r{loaded other examples than this process from \./spec/a_spec\.rb and 1 more spec file \(}) + end + it "does not use a worker whose loading failed when this process's did not" do worker_behaviours[2] = :fails_to_load groups = %w[A B].map { |name| passing_group(name) } diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb new file mode 100644 index 000000000..1bcb6884a --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -0,0 +1,108 @@ +module RSpec::Core + RSpec.describe Parallel::Plan do + let(:config) { RSpec.configuration } + let(:runner) { Runner.new(ConfigurationOptions.new([]), config, RSpec.world) } + let(:plan) { Parallel::Plan.new(runner) } + let(:started) { instance_double(Parallel::Coordinator, :start => nil, :assign => nil, :dismiss => nil) } + + around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + + def groups(count) + Array.new(count) { |i| RSpec.describe("G#{i}") { it("e") {} } } + end + + describe "by default, the spec files being loaded before the workers are forked" do + it "forks nothing before loading, then leaves the decision to Parallel.coordinator_for" do + config.parallel_workers = 2 + expect(Parallel::Coordinator).not_to receive(:new) + plan.before_loading + + all = groups(3) + expect(Parallel).to receive(:coordinator_for).with(runner, all).and_return(:coordinator) + expect(plan.coordinator_for(all)).to eq(:coordinator) + end + end + + context "when every worker is to load the spec files itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do + before { config.parallel_preload = false } + + it "forks the requested workers before loading, whatever the number of spec files, and gives them the groups" do + config.parallel_workers = 3 + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(started) + plan.before_loading + expect(started).to have_received(:start) + + all = groups(1).tap { |list| list.first.describe("nested") { it("e") {} } } + groups(2) + expect(plan.coordinator_for(all)).to be(started) + expect(started).to have_received(:assign).with(all, all) + end + + it "dismisses them, and runs serially, when fewer than two groups are to run" do + config.parallel_workers = 3 + allow(Parallel::Coordinator).to receive(:new).and_return(started) + plan.before_loading + + expect(plan.coordinator_for(groups(1))).to be_nil + expect(started).to have_received(:dismiss) + end + + it "runs serially, without forking or warning again, when a parallel run is not possible" do + config.parallel_workers = 2 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + expect(RSpec).to receive(:warning).once.with(a_string_including("needs `fork`"), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "forks nothing for one worker" do + config.parallel_workers = 1 + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "forks nothing before an --only-failures run without a status file, which aborts once the files are loaded" do + config.parallel_workers = 2 + config.force(:only_failures => true) + expect(Parallel::Coordinator).not_to receive(:new) + + plan.before_loading + end + + it "warns, and forks after loading, when the worker count is set while the spec files load" do + plan.before_loading + config.parallel_workers = 2 + expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_workers` was set while they were loading"), + :call_site => nil) + + expect(plan.coordinator_for(groups(3))).to be_a(Parallel::Coordinator) + end + end + + it "warns, and forks after loading, when told while the spec files load that each worker is to load them itself", + :skip => !RSpec::Support::RubyFeatures.fork_supported? do + config.parallel_workers = 2 + plan.before_loading + config.parallel_preload = false + expect(RSpec).to receive(:warning).with(a_string_including("not before them as `config.parallel_preload = false` asks: " \ + "it was set while they were loading"), :call_site => nil) + + expect(plan.coordinator_for(groups(3))).to be_a(Parallel::Coordinator) + end + + it "does not warn about the fork point of a run that is not parallel" do + plan.before_loading + config.parallel_preload = false + expect(RSpec).not_to receive(:warning) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "has nothing to dismiss when it forked nothing" do + expect { plan.dismiss }.not_to raise_error + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 1a5321f8e..574e1ec9e 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -76,66 +76,10 @@ def groups(count) expect(Parallel.coordinator_for(runner, groups(3))).to be_nil end end - - context "with workers forked before the spec files were loaded" do - let(:started) { instance_double(Parallel::Coordinator, :worker_count => 3, :assign => nil, :dismiss => nil) } - - it "assigns them the groups to run" do - all = groups(5) - - expect(Parallel.coordinator_for(runner, all, started)).to be(started) - expect(started).to have_received(:assign).with(all, all) - end - - it "dismisses them when fewer than two groups are to run" do - expect(Parallel.coordinator_for(runner, groups(1), started)).to be_nil - expect(started).to have_received(:dismiss) - end - - it "runs serially, without warning again, when they could not be forked" do - config.parallel_workers = 2 - config.parallel_preload = false - expect(RSpec).not_to receive(:warning) - - expect(Parallel.coordinator_for(runner, groups(3), nil)).to be_nil - end - end - end - - describe ".coordinator_before_loading" do - before { allow(config).to receive(:files_to_run).and_return(%w[a_spec.rb b_spec.rb c_spec.rb]) } - - it "forks nothing when the spec files are to be loaded first, as by default" do - config.parallel_workers = 2 - expect(Parallel::Coordinator).not_to receive(:new) - - expect(Parallel.coordinator_before_loading(runner)).to be_nil - end - - context "when every worker is to load the spec files itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do - before { config.parallel_preload = false } - - it "starts at most as many workers as there are spec files" do - config.parallel_workers = 8 - coordinator = instance_double(Parallel::Coordinator, :start => nil) - expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(coordinator) - - expect(Parallel.coordinator_before_loading(runner)).to be(coordinator) - expect(coordinator).to have_received(:start) - end - - it "starts none when parallel runs are not requested, or for fewer than two spec files" do - expect(Parallel.coordinator_before_loading(runner)).to be_nil - - config.parallel_workers = 4 - allow(config).to receive(:files_to_run).and_return(%w[a_spec.rb a_spec.rb]) - expect(Parallel.coordinator_before_loading(runner)).to be_nil - end - end end describe ".suite_digest" do - it "fingerprints the ids, locations and descriptions of the examples to run, in load order" do + it "fingerprints the ids and locations of the examples to run, in load order, but not their descriptions" do RSpec.describe("A") do it("one") {} it("two") {} @@ -144,11 +88,27 @@ def groups(count) first = Parallel.suite_digest(RSpec.world) expect(Parallel.suite_digest(RSpec.world)).to eq(first) - RSpec.world.example_groups.last.examples.first.metadata[:description] = "changed" - RSpec.world.example_groups.last.examples.first.metadata[:full_description] = "B changed" + RSpec.world.example_groups.last.examples.first.metadata[:full_description] = "B at #{Time.now}" + expect(Parallel.suite_digest(RSpec.world)).to eq(first) + RSpec.world.example_groups.last.examples.first.metadata[:location] = "./elsewhere_spec.rb:1" expect(Parallel.suite_digest(RSpec.world)).not_to eq(first) end + it "changes with the examples defined" do + RSpec.describe("A") { it("one") {} } + first = Parallel.suite_digest(RSpec.world) + RSpec.describe("B") { it("one more") {} } + + expect(Parallel.suite_digest(RSpec.world)).not_to eq(first) + end + + it "is kept per spec file" do + RSpec.describe("A") { it("one") {} } + RSpec.describe("B") { it("two") {} } + + expect(Parallel.suite_digest(RSpec.world).keys).to eq([RSpec.world.example_groups.first.metadata[:file_path]]) + end + it "leaves out the examples that are filtered out" do RSpec.describe("A") { it("one") {} } unfiltered = Parallel.suite_digest(RSpec.world) diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index e1349a2e8..40cacdebd 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -367,8 +367,8 @@ def interrupt RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } allow(config).to receive(:load_spec_files) - coordinator = instance_double(Parallel::Coordinator, :run => 17, :dismiss => nil) - expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups, nil).and_return(coordinator) + coordinator = instance_double(Parallel::Coordinator, :run => 17) + expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups).and_return(coordinator) config.example_status_persistence_file_path = "statuses.txt" expect(ExampleStatusPersister).to receive(:persist).with(world.all_examples, "statuses.txt") @@ -378,35 +378,37 @@ def interrupt expect(runner.run(err, out)).to eq(17) end - it 'forks workers that are to load the spec files themselves before loading them, and hands them the groups' do - RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } - coordinator = instance_double(Parallel::Coordinator, :run => 0, :dismiss => nil) - expect(Parallel).to receive(:coordinator_before_loading).ordered.and_return(coordinator) - expect(config).to receive(:load_spec_files).ordered - expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups, coordinator). - and_return(coordinator) + context 'with a parallel plan' do + let(:plan) { instance_double(Parallel::Plan, :before_loading => nil, :dismiss => nil) } - expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) - end + before { allow(Parallel::Plan).to receive(:new).and_return(plan) } - it 'dismisses workers forked before loading when the run ends early' do - coordinator = instance_double(Parallel::Coordinator, :dismiss => nil) - allow(Parallel).to receive(:coordinator_before_loading).and_return(coordinator) - allow(config).to receive(:load_spec_files) { world.wants_to_quit = true } - expect(Parallel).not_to receive(:coordinator_for) + it 'lets it fork workers before the spec files load, then hands it the groups, and dismisses what it did not use' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + coordinator = instance_double(Parallel::Coordinator, :run => 0) + expect(plan).to receive(:before_loading).ordered + expect(config).to receive(:load_spec_files).ordered + expect(plan).to receive(:coordinator_for).with(world.ordered_example_groups).ordered.and_return(coordinator) + expect(plan).to receive(:dismiss).ordered - Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end - expect(coordinator).to have_received(:dismiss) - end + it 'dismisses the workers it forked when the run ends early' do + allow(config).to receive(:load_spec_files) { world.wants_to_quit = true } + expect(plan).not_to receive(:coordinator_for) - it 'dismisses workers forked before loading when loading raises' do - coordinator = instance_double(Parallel::Coordinator, :dismiss => nil) - allow(Parallel).to receive(:coordinator_before_loading).and_return(coordinator) - allow(config).to receive(:load_spec_files).and_raise(SystemExit) + Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) - expect { Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) }.to raise_error(SystemExit) - expect(coordinator).to have_received(:dismiss) + expect(plan).to have_received(:dismiss) + end + + it 'dismisses the workers it forked when loading raises' do + allow(config).to receive(:load_spec_files).and_raise(SystemExit) + + expect { Runner.new(ConfigurationOptions.new([]), config, world).run(err, out) }.to raise_error(SystemExit) + expect(plan).to have_received(:dismiss) + end end it 'supports a test-queue like subclass that can perform setup once and run different sets of example groups multiple times' do From c8d5f5497573ba9dcbbc5d75683302289c3f2d74 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 18:56:11 +0200 Subject: [PATCH 11/58] [rspec-core] Report one-liner examples in parallel runs by the description their matcher generated Examples without a docstring (`it { is_expected.to eq(5) }`) get their description from the matcher once they have run, in the worker. The coordinating process reported them as "example at ./spec/x_spec.rb:3" in the documentation, failures, JSON and summary output; workers now send each example's description and full description with its result, and each group's with its start, and the coordinating process reports them as the worker had them. With --no-parallel-preload that also makes a description computed while a worker loaded the spec files the worker's own, rather than the coordinating process's. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 34 ++++++++------- rspec-core/README.md | 19 ++++---- rspec-core/lib/rspec/core/example.rb | 2 +- .../lib/rspec/core/parallel/coordinator.rb | 8 +++- .../rspec/core/parallel/worker_reporter.rb | 12 +++++- rspec-core/spec/integration/parallel_spec.rb | 43 ++++++++++++++++--- rspec-core/spec/rspec/core/example_spec.rb | 4 +- .../core/parallel/worker_reporter_spec.rb | 15 +++++-- 8 files changed, 98 insertions(+), 39 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index c24fdbd7f..1a32c7842 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -6,24 +6,26 @@ Enhancements: * Add `--[no-]parallel[=COUNT]` (and `RSpec::Core::Configuration#parallel_workers`) to run top-level example groups across worker processes forked after the suite has loaded. Formatters run in the coordinating process only and receive one coherent stream of - notifications with real `Example` objects -- results, timing, pending details, exceptions - rebuilt from their class name, message, backtrace and cause chain, messages, deprecations - and marshalable custom notifications (of the metadata changed while an example runs only - `:extra_failure_lines`, `:pending` and `:skip` are forwarded) -- so existing formatters - keep working and the run has one seed, one summary, one profile and one `.rspec_status` - file. Listeners registered inside a worker, e.g. in a `before(:suite)` hook, still see - that worker's example notifications. Workers run the suite hooks once each, expose + notifications with real `Example` objects -- results, timing, pending details, + exceptions rebuilt from their class name, message, backtrace and cause chain, messages, + deprecations and marshalable custom notifications (of the metadata changed while an + example runs only its description, as a one-liner generates it, `:extra_failure_lines`, + `:pending` and `:skip` are forwarded) -- so existing formatters keep working and the run + has one seed, one summary, one profile and one `.rspec_status` file. Listeners + registered inside a worker, e.g. in a `before(:suite)` hook, still see that worker's + example notifications. Workers run the suite hooks once each, expose `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way parallel_tests does. A worker that finishes but does not exit is aborted after - `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, - so a stuck `at_exit` handler can no longer hang a run, and a worker that crashes fails the - example it was running instead of taking the run down. A run stopped with SIGTERM or SIGHUP - says which example each worker was running and for how long. With `--no-parallel-preload` - (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec - files are loaded and each loads them itself, so nothing the spec files do while loading is - shared with the workers; a worker that does not load the examples the coordinating process - has is not used. Where `fork` is unavailable the run is serial with a warning. `--bisect` - runs serially, and the reproduction command it prints now ends with `--no-parallel`. + `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its + threads, so a stuck `at_exit` handler can no longer hang a run, and a worker that + crashes fails the example it was running instead of taking the run down. A run stopped + with SIGTERM or SIGHUP says which example each worker was running and for how long. With + `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are + forked before the spec files are loaded and each loads them itself, so nothing the spec + files do while loading is shared with the workers; a worker that does not load the + examples the coordinating process has is not used. Where `fork` is unavailable the run + is serial with a warning. `--bisect` runs serially, and the reproduction command it + prints now ends with `--no-parallel`. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index 62644b729..ffde56e05 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -333,12 +333,13 @@ files `require 'rails_helper'` boots in each worker with its own `TEST_ENV_NUMBER`. The cost is loading the spec files once per process: more CPU and memory, though they load at the same time, and whatever they do at load time is done that many times. A worker is given work only once it has -loaded the examples the coordinating process has (the same ids and locations; -descriptions computed while loading, from the time say, are the coordinating -process's). One that loads others, gets a load error the coordinating process -did not get, or is still loading long after the coordinating process finished -is not used: that counts as an error outside of examples, and its groups run -in the other workers -- or, when no worker is left, do not run. +loaded the examples the coordinating process has (the same ids and locations), +and what it runs is reported as it described it, so a description computed +while loading -- from the time, say -- is the worker's. One that loads other +examples, gets a load error the coordinating process did not get, or is still +loading long after the coordinating process finished is not used: that counts +as an error outside of examples, and its groups run in the other workers -- +or, when no worker is left, do not run. Formatters receive examples with their status, timing, pending details, exceptions (class name, message, backtrace, cause chain and aggregated @@ -346,8 +347,10 @@ failures; rebuilt as a subclass of the original class when that class exists in the coordinating process) and the group and example notifications in the order the worker produced them, plus `reporter.message`, deprecations and custom notifications whose options can be marshaled. Of the metadata a worker -changes while running an example, only `:extra_failure_lines`, `:pending` and -`:skip` are forwarded, and anything workers write to `$stdout`/`$stderr` +changes while running an example, only its description (generated by the +matcher of a one-liner like `it { is_expected.to eq(5) }`), +`:extra_failure_lines`, `:pending` and `:skip` are forwarded, and anything +workers write to `$stdout`/`$stderr` directly is not sequenced with the formatter output. Listeners registered from inside a worker (for example with `reporter.register_listener` in a `before(:suite)` hook) are notified of that worker's example and group diff --git a/rspec-core/lib/rspec/core/example.rb b/rspec-core/lib/rspec/core/example.rb index 661ae3f44..ba0d0c2bc 100644 --- a/rspec-core/lib/rspec/core/example.rb +++ b/rspec-core/lib/rspec/core/example.rb @@ -457,7 +457,7 @@ def assign_execution_result(attributes) result.pending_exception = attributes[:pending_exception] result.exception = @exception = attributes[:exception] - [:pending, :skip, :extra_failure_lines].each do |key| + [:description, :full_description, :pending, :skip, :extra_failure_lines].each do |key| metadata[key] = attributes[key] unless attributes[key].nil? end end diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 4f685e9da..f7c0739a1 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -150,7 +150,7 @@ def handle_message(worker, message, bytes) kind, *rest = message number = worker.number case kind - when :group_started then index.with_group(rest[0]) { |g| @sequencer.group_started(number, g, bytes) } + when :group_started then index.with_group(rest[0]) { |g| group_started(number, g, rest[1], bytes) } when :group_finished then index.with_group(rest[0]) { |g| @sequencer.group_finished(number, g, rest[1], bytes) } when :example_started then index.with_example(rest[0]) { |e| example_started(worker, e, bytes) } when :example_finished then example_finished(worker, rest[0], rest[1], bytes) @@ -175,6 +175,12 @@ def non_example_failure(payload, context_description) @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) end + # Described as the worker that runs it described it. + def group_started(number, group, descriptions, bytes) + group.metadata.update(descriptions) if descriptions + @sequencer.group_started(number, group, bytes) + end + def example_started(worker, example, bytes) worker.current_example = example @sequencer.example_started(worker.number, example, bytes) diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 8cdfc82c2..17d96db9c 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -59,7 +59,7 @@ def notify(event, notification) def forward_group_started(group) @group_started_at = Parallel.monotonic_time if group.top_level? - @results.send([:group_started, group.id]) + @results.send([:group_started, group.id, descriptions(group.metadata)]) end def forward_group_finished(group) @@ -124,7 +124,15 @@ def result_payload(example) :pending => Parallel.marshal_safe(metadata[:pending], &:to_s), :skip => Parallel.marshal_safe(metadata[:skip], &:to_s), :extra_failure_lines => Parallel.marshal_safe(metadata[:extra_failure_lines]) { |lines| Array(lines).map(&:to_s) } - } + }.merge(descriptions(metadata)) + end + + # As this worker has them: generated by the matcher of a one-liner + # (`it { is_expected.to ... }`) once the example ran, or computed + # while the worker loaded the spec files itself. + def descriptions(metadata) + { :description => String.new(metadata[:description].to_s), + :full_description => String.new(metadata[:full_description].to_s) } end end end diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 998ce67a6..a1a81a215 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -50,6 +50,15 @@ def write_fixed_suite 3.times { |i| it(\"passes \#{i}\") { expect(i).to be >= 0 } } end " + + # Described by their matchers once they have run. + write_file_formatted 'spec/delta_spec.rb', " + RSpec.describe 'Delta' do + subject { 5 } + it { is_expected.to eq(5) } + it { is_expected.to eq(6) } + end + " end # Timing differs between runs, and failures are numbered in the order they @@ -77,8 +86,9 @@ def group_blocks(documentation_output) run_rspec "--parallel=2 --order defined" expect(last_cmd_exit_status).to eq(1) - expect(last_cmd_stdout).to include("11 examples, 4 failures, 2 pending") - expect(last_cmd_stdout).to include("Alpha fails", "Alpha nested fails with a cause", "Beta aggregates", "Beta fails in an after hook") + expect(last_cmd_stdout).to include("13 examples, 5 failures, 2 pending") + expect(last_cmd_stdout).to include("Alpha fails", "Alpha nested fails with a cause", "Beta aggregates", "Beta fails in an after hook", + "rspec ./spec/delta_spec.rb:4 # Delta is expected to eq 6") expect(last_cmd_stdout).to include("Got 2 failures:", "--- Caused by: ---", "root") expect(last_cmd_stdout.scan("Randomized with seed").size).to eq(0) expect(last_cmd_stderr).to eq("") @@ -202,7 +212,7 @@ def example_failed(notification) expect(parallel.scan(/class="example (\w+)"/).sort).to eq(serial.scan(/class="example (\w+)"/).sort) expect(parallel.scan(/]*>/).size).to eq(serial.scan(/]*>/).size) expect(parallel.scan('').size).to eq(serial.scan('').size) - expect(parallel).to include("11 examples, 4 failures, 2 pending") + expect(parallel).to include("13 examples, 5 failures, 2 pending") end end @@ -314,18 +324,18 @@ def message(notification) RSpec.configuration.example_status_persistence_file_path = 'statuses.txt' run_rspec "--order defined --parallel=2" - expect(last_cmd_stdout).to include("11 examples, 4 failures, 2 pending") + expect(last_cmd_stdout).to include("13 examples, 5 failures, 2 pending") cd '.' do statuses = File.read("statuses.txt") - expect(statuses.scan('| passed ').size).to eq(5) - expect(statuses.scan('| failed ').size).to eq(4) + expect(statuses.scan('| passed ').size).to eq(6) + expect(statuses.scan('| failed ').size).to eq(5) expect(statuses.scan('| pending').size).to eq(2) expect(statuses).to match(/\d+\.\d+ seconds/) end RSpec.configuration.example_status_persistence_file_path = 'statuses.txt' run_rspec "--order defined --parallel=2 --only-failures" - expect(last_cmd_stdout).to include("4 examples, 4 failures") + expect(last_cmd_stdout).to include("5 examples, 5 failures") end it 'forwards deprecations and messages from the workers into the single report' do @@ -734,6 +744,25 @@ def loaders expect(last_cmd_stderr).to include("`config.parallel_workers` was set while they were loading") end + it 'reports the groups and examples as the worker that ran them described them' do + write_file_formatted 'spec/pid_spec.rb', " + 2.times do |i| + RSpec.describe(\"G\#{i} loaded in \#{Process.pid}\") do + it(\"ran in \#{Process.pid}\") { File.write(\"ran_\#{Process.pid}\", '') } + end + end + " + + run_rspec '--parallel=2 --no-parallel-preload --format documentation' + + expect(last_cmd_stdout).to include("2 examples, 0 failures") + described = last_cmd_stdout.scan(/(?:loaded|ran) in (\d+)/).flatten.uniq + ran = cd('.') { Dir['ran_*'].map { |file| file.delete_prefix('ran_') } } + expect(described).not_to be_empty + expect(described).to all(satisfy { |pid| ran.include?(pid) }) + expect(described).not_to include(Process.pid.to_s) + end + it 'runs the groups of a single spec file across the workers' do write_file_formatted 'spec/one_spec.rb', " 3.times { |i| RSpec.describe(\"G\#{i}\") { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } } diff --git a/rspec-core/spec/rspec/core/example_spec.rb b/rspec-core/spec/rspec/core/example_spec.rb index dcd030c9b..1c21c7b90 100644 --- a/rspec-core/spec/rspec/core/example_spec.rb +++ b/rspec-core/spec/rspec/core/example_spec.rb @@ -984,7 +984,8 @@ def expect_pending_result(example) example.assign_execution_result( :status => :failed, :started_at => started_at, :finished_at => finished_at, :run_time => 1.0, :pending_message => nil, :pending_fixed => false, :exception => error, :pending_exception => nil, - :pending => nil, :skip => true, :extra_failure_lines => ["more"] + :pending => nil, :skip => true, :extra_failure_lines => ["more"], + :description => "is expected to fail", :full_description => "group is expected to fail" ) expect(example.execution_result).to have_attributes( @@ -993,6 +994,7 @@ def expect_pending_result(example) ) expect(example.exception).to be(error) expect(example.metadata).to include(:pending => "declared", :skip => true, :extra_failure_lines => ["more"]) + expect(example).to have_attributes(:description => "is expected to fail", :full_description => "group is expected to fail") expect(RSpec::Core::Notifications::ExampleNotification.for(example)).to be_a(RSpec::Core::Notifications::FailedExampleNotification) end diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index e2356eb73..36008ecf3 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -39,8 +39,8 @@ def frames :group_finished ]) - expect(kinds[0]).to eq([:group_started, group.id]) - expect(kinds[9]).to eq([:group_started, inner.id]) + expect(kinds[0]).to eq([:group_started, group.id, { :description => "Outer", :full_description => "Outer" }]) + expect(kinds[9]).to eq([:group_started, inner.id, { :description => "Inner", :full_description => "Outer Inner" }]) expect(kinds[12]).to eq([:group_finished, inner.id, nil]) expect(kinds[13][0..1]).to eq([:group_finished, group.id]) expect(kinds[13][2]).to be_a(Float) @@ -48,7 +48,8 @@ def frames payloads = kinds.select { |f| f.first == :example_finished }.to_h { |f| [f[1], f[2]] } passed, failed, pending, skipped = group.examples.map { |example| payloads.fetch(example.id) } - expect(passed).to include(:status => :passed, :exception => nil, :pending_exception => nil, :pending => nil, :skip => nil) + expect(passed).to include(:status => :passed, :exception => nil, :pending_exception => nil, :pending => nil, :skip => nil, + :description => "passes", :full_description => "Outer passes") expect(passed[:started_at]).to be_a(Time) expect(passed[:finished_at]).to be_a(Time) expect(passed[:run_time]).to be_a(Float) @@ -69,6 +70,14 @@ def frames expect(frames.select { |f| f.first == :example_started }).to eq([[:example_started, group.examples.first.id]]) end + it "ships the description a one-liner's matcher generated while it ran" do + group = RSpec.describe("G") { it { expect(1).to eq(1) } } + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload).to include(:description => "is expected to eq 1", :full_description => "G is expected to eq 1") + end + it "forwards extra failure lines and stringifies result parts that cannot be marshaled" do group = RSpec.describe("G") do it("fails") do |ex| From 0449ffec988f246127e68a7b9df4767b9a46350f Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 18:57:28 +0200 Subject: [PATCH 12/58] [rspec-core] Load every --no-parallel-preload worker from the same Kernel.rand seed, and never fork after loading Ruby reseeds `Kernel.rand` in a forked process, so workers that load the spec files themselves drew other values while loading than the coordinating process, even in a suite that seeds `Kernel.rand` from `--seed`, and ran other examples than it reported. All of them now start loading from the coordinating process's seed, so the values are those of a serial run with that seed. `parallel_workers` or `parallel_preload = false` set while the spec files load now gives a serial run with a warning, instead of forking after loading, which is what `parallel_preload = false` is there to avoid. No more workers than spec files (two for a single file) are forked before loading, so a small run does not load the suite once per processor, and workers forked before a `fork` that fails are dismissed rather than left behind. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 10 +-- rspec-core/README.md | 37 +++++----- rspec-core/lib/rspec/core/configuration.rb | 23 +++--- .../lib/rspec/core/parallel/coordinator.rb | 1 + rspec-core/lib/rspec/core/parallel/plan.rb | 58 ++++++++++----- rspec-core/lib/rspec/core/parallel/worker.rb | 11 +-- .../lib/rspec/core/parallel/worker_handle.rb | 4 +- .../rspec/core/parallel/worker_lifecycle.rb | 19 +++-- rspec-core/spec/integration/parallel_spec.rb | 70 ++++++++++++++----- .../rspec/core/parallel/coordinator_spec.rb | 33 +++++++++ .../spec/rspec/core/parallel/plan_spec.rb | 64 ++++++++++++++--- .../spec/rspec/core/parallel/worker_spec.rb | 28 ++++++-- 12 files changed, 265 insertions(+), 93 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 1a32c7842..4a8b8c7a2 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -21,11 +21,11 @@ Enhancements: crashes fails the example it was running instead of taking the run down. A run stopped with SIGTERM or SIGHUP says which example each worker was running and for how long. With `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are - forked before the spec files are loaded and each loads them itself, so nothing the spec - files do while loading is shared with the workers; a worker that does not load the - examples the coordinating process has is not used. Where `fork` is unavailable the run - is serial with a warning. `--bisect` runs serially, and the reproduction command it - prints now ends with `--no-parallel`. + forked before the spec files are loaded and each loads them itself, from the same + `Kernel.rand` seed, so nothing the spec files do while loading is shared with the + workers; a worker that does not load the examples the coordinating process has is not + used. Where `fork` is unavailable the run is serial with a warning. `--bisect` runs + serially, and the reproduction command it prints now ends with `--no-parallel`. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index ffde56e05..999f4115a 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -323,23 +323,26 @@ Or have every worker load the spec files itself: with `--no-parallel-preload` (or `config.parallel_preload = false` in a `--require`d file) the workers are forked as soon as the `--require`d files are loaded, before the spec files. This and the worker count must be known by then: set in a helper that the spec -files require, they come too late, and the workers are forked after loading, -with a warning. Each worker then loads the spec files with its -`TEST_ENV_NUMBER` already set, and the coordinating process loads them too, -for its reporting. Nothing the spec files do while loading is shared then -- -connections, threads, system libraries they initialize -- and helpers they -require (rather than `.rspec`) load in every worker, so a Rails app whose spec -files `require 'rails_helper'` boots in each worker with its own -`TEST_ENV_NUMBER`. The cost is loading the spec files once per process: more -CPU and memory, though they load at the same time, and whatever they do at -load time is done that many times. A worker is given work only once it has -loaded the examples the coordinating process has (the same ids and locations), -and what it runs is reported as it described it, so a description computed -while loading -- from the time, say -- is the worker's. One that loads other -examples, gets a load error the coordinating process did not get, or is still -loading long after the coordinating process finished is not used: that counts -as an error outside of examples, and its groups run in the other workers -- -or, when no worker is left, do not run. +files require, they come too late, and the run is serial, with a warning -- +forking after loading is what this setting avoids. There are no more workers +than spec files (but two for a single file). Each worker then loads the spec +files with its `TEST_ENV_NUMBER` already set, and the coordinating process +loads them too, for its reporting; all of them start from the coordinating +process's `Kernel.rand` seed, so values drawn while loading are the same in +every process, as they are in a serial run with that seed. Nothing the spec +files do while loading is shared then -- connections, threads, system +libraries they initialize -- and helpers they require (rather than `.rspec`) +load in every worker, so a Rails app whose spec files `require 'rails_helper'` +boots in each worker with its own `TEST_ENV_NUMBER`. The cost is loading the +spec files once per process: more CPU and memory, though they load at the same +time, and whatever they do at load time is done that many times. A worker is +given work only once it has loaded the examples the coordinating process has +(the same ids and locations), and what it runs is reported as it described it, +so a description computed while loading -- from the time, say -- is the +worker's. One that loads other examples, gets a load error the coordinating +process did not get, or is still loading long after the coordinating process +finished is not used: that counts as an error outside of examples, and its +groups run in the other workers -- or, when no worker is left, do not run. Formatters receive examples with their status, timing, pending details, exceptions (class name, message, backtrace, cause chain and aggregated diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 237d037b8..189cfe67d 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -275,17 +275,18 @@ def parallel_workers=(value) # @macro add_setting # Whether the spec files are loaded before the parallel workers are # forked (default: `true`), so that they share them. When `false` - # (`--no-parallel-preload`) the workers are forked as soon as the - # `--require`d files are loaded, and each loads the spec files itself - # -- once `ENV['TEST_ENV_NUMBER']` is set -- as this process does for - # its reporting. Nothing the spec files do while loading (connections, - # threads, system libraries they initialize) is then shared with the - # workers, at the cost of loading them once per worker. A worker that - # does not load the same examples as this process is not used. It, and - # {#parallel_workers}, have to be set before the spec files load (on the - # command line, in `.rspec` or in a `--require`d file); set while they - # load, they come too late, and the workers are forked after loading, - # with a warning. + # (`--no-parallel-preload`) the workers -- no more than there are spec + # files, but two for one -- are forked as soon as the `--require`d + # files are loaded, and each loads the spec files itself -- once + # `ENV['TEST_ENV_NUMBER']` is set, from this process's `Kernel.rand` + # seed -- as this process does for its reporting. Nothing the spec + # files do while loading (connections, threads, system libraries they + # initialize) is then shared with the workers, at the cost of loading + # them once per worker. A worker that does not load the same examples + # as this process is not used. It, and {#parallel_workers}, have to be + # set before the spec files load (on the command line, in `.rspec` or + # in a `--require`d file); set while they load, they come too late, and + # the run is serial, with a warning. # @return [Boolean] add_setting :parallel_preload diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index f7c0739a1..ed32ce162 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -44,6 +44,7 @@ def initialize(runner, worker_count, options={}) @termination_signal = nil @signal_handlers = {} @started_at = nil + @load_seed = nil end # The groups to run. Workers forked before loading that are more than diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb index 51d8c5c49..31bf92dde 100644 --- a/rspec-core/lib/rspec/core/parallel/plan.rb +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -26,18 +26,46 @@ def before_loading end @decided = true - requested = Parallel.requested_worker_count(@configuration) - @coordinator = Coordinator.new(@runner, requested).tap(&:start) if requested && requested >= 2 + count = worker_count + return unless count && count >= 2 + + # Assigned first, so that workers forked before a `fork` that fails + # are dismissed too. + @coordinator = Coordinator.new(@runner, count) + @coordinator.start end # @return [Coordinator, nil] the coordinator to run the loaded groups, # or `nil` for a serial run def coordinator_for(example_groups) - unless @decided - coordinator = Parallel.coordinator_for(@runner, example_groups) - warn_late if coordinator && !@configuration.parallel_preload? - return coordinator - end + return assign(example_groups) if @decided + return Parallel.coordinator_for(@runner, example_groups) if @configuration.parallel_preload? + + # Too late to fork before loading, and forking after loading is + # what this run was configured not to do. + requested = Parallel.requested_worker_count(@configuration) + warn_late if requested && requested >= 2 + nil + end + + # Ends the workers forked before loading that the run did not use. + def dismiss + @coordinator&.dismiss + end + + private + + # Each worker loads every spec file, so there are no more of them + # than spec files -- a spec file usually defines one top-level group + # -- but two for a single file, which may define several. + def worker_count + requested = Parallel.requested_worker_count(@configuration) + return requested unless requested && requested > 2 + + @configuration.files_to_run.uniq.size.clamp(2, requested) + end + + def assign(example_groups) return nil unless @coordinator units = Parallel.units_for(example_groups, @configuration) @@ -50,18 +78,12 @@ def coordinator_for(example_groups) @coordinator end - # Ends the workers forked before loading that the run did not use. - def dismiss - @coordinator&.dismiss - end - - private - def warn_late - RSpec.warning "The parallel workers are forked after the spec files were loaded, not before them as " \ - "`config.parallel_preload = false` asks: #{@postponed_by || 'it'} was set while they were " \ - "loading. Set it before they load: on the command line, in `.rspec` or in a `--require`d " \ - "file.", :call_site => nil + what = @postponed_by || '`config.parallel_preload = false`' + RSpec.warning "Ignoring `--parallel`: #{what} was set while the spec files were loading, too late to fork " \ + "the workers before them as `config.parallel_preload = false` asks. Set it before they " \ + "load: on the command line, in `.rspec` or in a `--require`d file. Running serially.", + :call_site => nil end end end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index e781d3bc9..220681ffe 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -37,6 +37,7 @@ def initialize(number, example_groups, jobs, results, options={}) @ppid = options.fetch(:ppid) { Process.method(:ppid) } @watch_coordinator = options.fetch(:watch_coordinator, true) @signal_handlers = options.fetch(:signal_handlers, {}) + @load_seed = options[:load_seed] end # @return [String] the `TEST_ENV_NUMBER` value for this worker, following @@ -110,12 +111,14 @@ def trap_interrupt end end - # Loads the spec files as the coordinator does after forking, and - # tells it which examples that defined and the first load error, if - # any: it uses the worker only if its own loading went the same way. - # What loading reports is for the coordinator to report. + # Loads the spec files as the coordinator does after forking, from + # its `Kernel.rand` seed, and tells it which examples that defined and + # the first load error, if any: it uses the worker only if its own + # loading went the same way. What loading reports is for the + # coordinator to report. # @private def load_spec_files + Kernel.srand(@load_seed) if @load_seed forwarding = @configuration.reporter @configuration.reporter = loading = LoadReporter.new(@configuration) @configuration.load_spec_files diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 105ef0965..2c269a193 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -53,12 +53,12 @@ def lost_example_result(text) # @return [Integer] the pid of the worker process forked to run `units`, # or to load the spec files itself when there are none yet - def fork_worker(process, units, handles, signal_handlers={}) + def fork_worker(process, units, handles, worker_options={}) process.fork do # :nocov: - runs in the forked worker process, covered by integration specs handles.each { |other| other.close_all unless other.equal?(self) } close_parent_ends - Worker.new(@number, units, @child_jobs, @child_results, :signal_handlers => signal_handlers).run + Worker.new(@number, units, @child_jobs, @child_results, worker_options).run # :nocov: end end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 0cac11c90..ee4514215 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -11,16 +11,20 @@ module Parallel # @private module WorkerLifecycle # Forks the workers before the spec files are loaded; each loads them - # itself and reports what it loaded. + # itself and reports what it loaded. Ruby reseeds `Kernel.rand` in a + # forked process, so every process starts loading from this one's + # seed, and values drawn while loading are the same in all of them. def start @started_at = now + @load_seed = Kernel.srand + Kernel.srand(@load_seed) start_workers(nil) end # Ends the workers that are not done -- workers forked before loading # when the run ends early or goes on serially: they have run nothing. def dismiss - live = @workers.reject(&:done?) + live = live_workers live.each { |worker| dismiss_worker(worker) } give_up_unless_gone(live, @kill_grace) end @@ -41,7 +45,13 @@ def start_workers(units) end def fork_worker(worker, workers) - worker.fork_worker(@process, @units, workers, @signal_handlers) + worker.fork_worker(@process, @units, workers, :signal_handlers => @signal_handlers, :load_seed => @load_seed) + end + + # Including those a failed `fork` left without a process. + def live_workers + @workers.each { |worker| worker.state = :exited if worker.pid.nil? } + @workers.reject(&:done?) end # Workers started loading the spec files when this process did, so @@ -89,8 +99,7 @@ def dismiss_worker(worker) # Runs on every exit path, so an exception in the coordinator can never # leave workers behind. Bounded by the same TERM/KILL grace periods. def terminate_remaining_workers - @workers.each { |worker| worker.state = :exited if worker.pid.nil? } - live = @workers.reject(&:done?) + live = live_workers return if live.empty? live.each do |worker| diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index a1a81a215..acab2dc0e 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -677,6 +677,20 @@ def loaders cd('.') { Dir['loaded_*'].map { |file| file.split('_', 4).last }.tally } end + # A process of its own, where the `--require`d files load afresh. Given at + # most 60 seconds. + def rspec_process(*args) + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 + sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + Process.kill(:KILL, pid) && Process.wait(pid) unless status + [status, File.read('stdout.txt'), File.read('stderr.txt')] + end + end + it 'has every worker load the spec files itself, once TEST_ENV_NUMBER is set, and run what it loaded' do write_marking_spec_files @@ -717,31 +731,58 @@ def loaders expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") end - it 'warns, and forks after loading, when turned off from a spec file, too late' do + it 'warns, and runs serially rather than fork after loading, when turned off from a spec file, too late' do write_file_formatted 'spec/late_spec.rb', " RSpec.configure { |c| c.parallel_preload = false } - RSpec.describe('Late') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } - RSpec.describe('Later') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + RSpec.describe('Late') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('Later') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } " run_rspec '--parallel=2 spec/late_spec.rb' expect(last_cmd_stdout).to include("2 examples, 0 failures") - expect(last_cmd_stderr).to include("forked after the spec files were loaded, not before them as " \ - "`config.parallel_preload = false` asks: it was set while they were loading") + expect(last_cmd_stderr).to include("Ignoring `--parallel`: `config.parallel_preload = false` was set while the spec " \ + "files were loading", "Running serially.") end - it 'warns, and forks after loading, when the worker count is set from a spec file, too late' do + it 'warns, and runs serially rather than fork after loading, when the worker count is set from a spec file, too late' do write_file_formatted 'spec/late_spec.rb', " RSpec.configure { |c| c.parallel_workers = 2 } - RSpec.describe('Late') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } - RSpec.describe('Later') { it('runs in a worker') { expect(RSpec.configuration.parallel_worker_number).not_to be_nil } } + RSpec.describe('Late') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } + RSpec.describe('Later') { it('runs serially') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } " run_rspec '--no-parallel-preload spec/late_spec.rb' expect(last_cmd_stdout).to include("2 examples, 0 failures") - expect(last_cmd_stderr).to include("`config.parallel_workers` was set while they were loading") + expect(last_cmd_stderr).to include("`config.parallel_workers` was set while the spec files were loading", "Running serially.") + end + + it 'draws the same values with Kernel.rand while loading as a serial run with the same seed' do + write_file_formatted 'spec/spec_helper.rb', "RSpec.configure { |c| Kernel.srand(c.seed) }" + %w[a b].each do |name| + write_file_formatted "spec/#{name}_spec.rb", " + RSpec.describe('#{name.upcase}') do + 3.times do + n = rand(1000) + it(\"handles \#{n}\") do + File.write(\"ran_\#{RSpec.configuration.parallel_worker_number.inspect}\", '') + expect(n).to be_even + end + end + end + " + end + + _, serial, = rspec_process('--seed', '1234', '--require', './spec/spec_helper') + status, parallel, errors = rspec_process('--seed', '1234', '--require', './spec/spec_helper', '--parallel=2', + '--no-parallel-preload') + + expect(errors).to eq("") + expect(cd('.') { Dir['ran_*'] - ['ran_nil'] }).not_to be_empty + expect(serial.lines.grep(/^rspec /)).not_to be_empty + expect(parallel.lines.grep(/^rspec |examples, /)).to match_array(serial.lines.grep(/^rspec |examples, /)) + expect(status.exitstatus).to eq(1) end it 'reports the groups and examples as the worker that ran them described them' do @@ -778,15 +819,10 @@ def loaders write_marking_spec_files # A real process: the abort ends it with `exit!`. - cd('.') do - rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] - pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', '--no-parallel-preload', '--only-failures', - :out => 'stdout.txt', :err => 'stderr.txt') - _, status = Process.wait2(pid) + status, out, err = rspec_process('--parallel=2', '--no-parallel-preload', '--only-failures') - expect(status.exitstatus).to eq(1) - expect(File.read('stderr.txt') + File.read('stdout.txt')).to include("example_status_persistence_file_path") - end + expect(status.exitstatus).to eq(1) + expect(err + out).to include("example_status_persistence_file_path") expect(loaders).to eq("none" => 3) end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index e2b0c7cec..751845e9a 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -552,6 +552,39 @@ def run_started(units, worker_count=2, options={}) ensure survivor.threads.each(&:kill) end + + it "dismisses the workers it forked when a later fork fails, and does not wait for those it never forked" do + worker_behaviours[1] = :never_loads + fork_fails = lambda do |handle, handles| + raise Errno::EAGAIN, "fork(2)" if handle.number == 2 + + spawn(handle, handles) + end + instance = new_coordinator([passing_group("A")], 3, :spawn => fork_fails) + + expect { instance.start }.to raise_error(Errno::EAGAIN) + started = RSpec::Core::Parallel.monotonic_time + instance.dismiss + + expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(process.threads).to all(satisfy { |thread| !thread.alive? }) + expect(messages).not_to include("survived SIGKILL") + expect(RSpec::Core::Parallel.monotonic_time - started).to be < 4 + end + + it "starts this process's `Kernel.rand` sequence over from its seed, the one the workers load the spec files with" do + previous = Kernel.srand(1234) + first = Kernel.rand(1_000_000) + Kernel.rand(1_000_000) + worker_behaviours[1] = worker_behaviours[2] = :never_loads + + instance = started_coordinator([passing_group("A")]) + + expect(Kernel.rand(1_000_000)).to eq(first) + instance.dismiss + ensure + Kernel.srand(previous) + end end it "terminates a worker that sends data it cannot decode" do diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb index 1bcb6884a..8a173bbad 100644 --- a/rspec-core/spec/rspec/core/parallel/plan_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -26,8 +26,13 @@ def groups(count) context "when every worker is to load the spec files itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do before { config.parallel_preload = false } - it "forks the requested workers before loading, whatever the number of spec files, and gives them the groups" do + def spec_files(count) + allow(config).to receive(:files_to_run).and_return(Array.new(count) { |i| "./spec/f#{i}_spec.rb" }) + end + + it "forks the requested workers before loading, and gives them the groups" do config.parallel_workers = 3 + spec_files(5) expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(started) plan.before_loading expect(started).to have_received(:start) @@ -37,6 +42,33 @@ def groups(count) expect(started).to have_received(:assign).with(all, all) end + it "forks no more workers than there are spec files to load" do + config.parallel_workers = 8 + spec_files(3) + expect(Parallel::Coordinator).to receive(:new).with(runner, 3).and_return(started) + + plan.before_loading + end + + it "forks two workers for a single spec file, which may define several groups" do + config.parallel_workers = 8 + spec_files(1) + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(started) + + plan.before_loading + end + + it "dismisses the workers it forked when a later fork fails" do + config.parallel_workers = 2 + failing = instance_double(Parallel::Coordinator, :dismiss => nil) + allow(failing).to receive(:start).and_raise(Errno::EAGAIN) + allow(Parallel::Coordinator).to receive(:new).and_return(failing) + + expect { plan.before_loading }.to raise_error(Errno::EAGAIN) + plan.dismiss + expect(failing).to have_received(:dismiss) + end + it "dismisses them, and runs serially, when fewer than two groups are to run" do config.parallel_workers = 3 allow(Parallel::Coordinator).to receive(:new).and_return(started) @@ -72,25 +104,41 @@ def groups(count) plan.before_loading end - it "warns, and forks after loading, when the worker count is set while the spec files load" do + it "warns, and runs serially, when the worker count is set while the spec files load" do plan.before_loading config.parallel_workers = 2 - expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_workers` was set while they were loading"), + expect(RSpec).to receive(:warning).with(a_string_including( + "Ignoring `--parallel`: `config.parallel_workers` was set while the spec files were loading, too late to fork " \ + "the workers before them as `config.parallel_preload = false` asks." + ).and(ending_with("Running serially.")), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + + it "warns, and runs serially, when the status file --only-failures needs is configured while the spec files load" do + config.parallel_workers = 2 + config.force(:only_failures => true) + plan.before_loading + config.example_status_persistence_file_path = "examples.txt" + expect(RSpec).to receive(:warning).with(a_string_including("`config.example_status_persistence_file_path` " \ + "(for `--only-failures`) was set while the spec files were loading"), :call_site => nil) - expect(plan.coordinator_for(groups(3))).to be_a(Parallel::Coordinator) + expect(plan.coordinator_for(groups(3))).to be_nil end end - it "warns, and forks after loading, when told while the spec files load that each worker is to load them itself", + it "warns, and runs serially, when told while the spec files load that each worker is to load them itself", :skip => !RSpec::Support::RubyFeatures.fork_supported? do config.parallel_workers = 2 plan.before_loading config.parallel_preload = false - expect(RSpec).to receive(:warning).with(a_string_including("not before them as `config.parallel_preload = false` asks: " \ - "it was set while they were loading"), :call_site => nil) + expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_preload = false` was set while the spec " \ + "files were loading"), :call_site => nil) + expect(Parallel::Coordinator).not_to receive(:new) - expect(plan.coordinator_for(groups(3))).to be_a(Parallel::Coordinator) + expect(plan.coordinator_for(groups(3))).to be_nil end it "does not warn about the fork point of a run that is not parallel" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index c5d10f535..a67e4aeff 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -213,13 +213,13 @@ def frames end describe "forked before the spec files were loaded" do - def loading_worker + def loading_worker(options={}) Worker.new(2, nil, jobs.first, results.last, - :stderr => stderr, - :exit => lambda { |status| exits << [:exit, status] }, - :exit_now => lambda { |status| exits << [:exit_now, status] }, - :trap => lambda { |*| }, - :watch_coordinator => false) + { :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :watch_coordinator => false }.merge(options)) end def write_spec_file(dir, content) @@ -269,6 +269,22 @@ def run_in_thread(instance) expect(exits).to eq([[:exit_now, 0]]) end end + + it "loads them from the coordinating process's `Kernel.rand` seed, which a forked process does not keep" do + previous = Kernel.srand(5) + Dir.mktmpdir do |dir| + write_spec_file(dir, "RSpec.describe(\"drew \#{rand(1_000_000)}\") { it('e') { } }") + thread = run_in_thread(loading_worker(:load_seed => 1234)) + + results.first.receive + jobs.last.send([:stop]) + expect(thread.join(10)).to be_truthy + + expect(RSpec.world.example_groups.first.description).to eq("drew #{Random.new(1234).rand(1_000_000)}") + end + ensure + Kernel.srand(previous) + end end describe "the watchdog" do From 56dafc485644ba2aea3000b15f9a3460e66b5960 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 19:31:15 +0200 Subject: [PATCH 13/58] [rspec-core] Leave Kernel.rand alone with --no-parallel-preload, as RSpec never calls srand Seeding every process from the coordinating process's seed, so that values drawn while loading matched, rewound that process's sequence (values drawn before the workers were forked were drawn again), gave every worker the same sequence while running examples (random ports and keys collided between workers), and broke RSpec's rule that it never calls `srand`. RSpec no longer touches `Kernel.rand`: each worker keeps the seed of its own that Ruby gives a forked process, while loading and running alike. Examples are reported as the worker that ran them described them, so a description drawn at random while loading is reported as it ran, and a worker whose examples differ in ids or locations is still not used. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 10 ++--- rspec-core/README.md | 28 +++++++------ rspec-core/lib/rspec/core/configuration.rb | 17 ++++---- .../lib/rspec/core/parallel/coordinator.rb | 1 - rspec-core/lib/rspec/core/parallel/worker.rb | 11 ++--- .../rspec/core/parallel/worker_lifecycle.rb | 8 +--- rspec-core/spec/integration/parallel_spec.rb | 40 +++++++++++-------- .../rspec/core/parallel/coordinator_spec.rb | 14 ------- .../spec/rspec/core/parallel/worker_spec.rb | 28 +++---------- 9 files changed, 64 insertions(+), 93 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 4a8b8c7a2..1a32c7842 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -21,11 +21,11 @@ Enhancements: crashes fails the example it was running instead of taking the run down. A run stopped with SIGTERM or SIGHUP says which example each worker was running and for how long. With `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are - forked before the spec files are loaded and each loads them itself, from the same - `Kernel.rand` seed, so nothing the spec files do while loading is shared with the - workers; a worker that does not load the examples the coordinating process has is not - used. Where `fork` is unavailable the run is serial with a warning. `--bisect` runs - serially, and the reproduction command it prints now ends with `--no-parallel`. + forked before the spec files are loaded and each loads them itself, so nothing the spec + files do while loading is shared with the workers; a worker that does not load the + examples the coordinating process has is not used. Where `fork` is unavailable the run + is serial with a warning. `--bisect` runs serially, and the reproduction command it + prints now ends with `--no-parallel`. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/README.md b/rspec-core/README.md index 999f4115a..5510bb327 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -301,7 +301,9 @@ Each worker runs the `before(:suite)` and `after(:suite)` hooks once. Inside a worker, `RSpec.configuration.parallel_worker_number` is its 1-based number and `ENV['TEST_ENV_NUMBER']` follows the convention of parallel_tests (`""` for the first worker, then `"2"`, `"3"`, ...), so anything you already namespace per -worker for that tool keeps working. +worker for that tool keeps working. Each worker draws from `Kernel.rand` with +a seed of its own, as forked processes do, so `Kernel.srand config.seed` does +not make what examples draw reproducible under `--parallel`. Workers inherit everything that happened while the suite was loaded. For a Rails application that means `rails_helper` has already evaluated @@ -327,18 +329,18 @@ files require, they come too late, and the run is serial, with a warning -- forking after loading is what this setting avoids. There are no more workers than spec files (but two for a single file). Each worker then loads the spec files with its `TEST_ENV_NUMBER` already set, and the coordinating process -loads them too, for its reporting; all of them start from the coordinating -process's `Kernel.rand` seed, so values drawn while loading are the same in -every process, as they are in a serial run with that seed. Nothing the spec -files do while loading is shared then -- connections, threads, system -libraries they initialize -- and helpers they require (rather than `.rspec`) -load in every worker, so a Rails app whose spec files `require 'rails_helper'` -boots in each worker with its own `TEST_ENV_NUMBER`. The cost is loading the -spec files once per process: more CPU and memory, though they load at the same -time, and whatever they do at load time is done that many times. A worker is -given work only once it has loaded the examples the coordinating process has -(the same ids and locations), and what it runs is reported as it described it, -so a description computed while loading -- from the time, say -- is the +loads them too, for its reporting. RSpec never calls `srand`, so each worker +draws from `Kernel.rand` with a seed of its own while loading too, whatever +the coordinating process seeded it with. Nothing the spec files do while +loading is shared then -- connections, threads, system libraries they +initialize -- and helpers they require (rather than `.rspec`) load in every +worker, so a Rails app whose spec files `require 'rails_helper'` boots in each +worker with its own `TEST_ENV_NUMBER`. The cost is loading the spec files once +per process: more CPU and memory, though they load at the same time, and +whatever they do at load time is done that many times. A worker is given work +only once it has loaded the examples the coordinating process has (the same +ids and locations), and what it runs is reported as it described it, so a +description computed while loading -- from the time or `rand`, say -- is the worker's. One that loads other examples, gets a load error the coordinating process did not get, or is still loading long after the coordinating process finished is not used: that counts as an error outside of examples, and its diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 189cfe67d..1f0f584fa 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -278,15 +278,14 @@ def parallel_workers=(value) # (`--no-parallel-preload`) the workers -- no more than there are spec # files, but two for one -- are forked as soon as the `--require`d # files are loaded, and each loads the spec files itself -- once - # `ENV['TEST_ENV_NUMBER']` is set, from this process's `Kernel.rand` - # seed -- as this process does for its reporting. Nothing the spec - # files do while loading (connections, threads, system libraries they - # initialize) is then shared with the workers, at the cost of loading - # them once per worker. A worker that does not load the same examples - # as this process is not used. It, and {#parallel_workers}, have to be - # set before the spec files load (on the command line, in `.rspec` or - # in a `--require`d file); set while they load, they come too late, and - # the run is serial, with a warning. + # `ENV['TEST_ENV_NUMBER']` is set -- as this process does for its + # reporting. Nothing the spec files do while loading (connections, + # threads, system libraries they initialize) is then shared with the + # workers, at the cost of loading them once per worker. A worker that + # does not load the same examples as this process is not used. It, and + # {#parallel_workers}, have to be set before the spec files load (on + # the command line, in `.rspec` or in a `--require`d file); set while + # they load, they come too late, and the run is serial, with a warning. # @return [Boolean] add_setting :parallel_preload diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index ed32ce162..f7c0739a1 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -44,7 +44,6 @@ def initialize(runner, worker_count, options={}) @termination_signal = nil @signal_handlers = {} @started_at = nil - @load_seed = nil end # The groups to run. Workers forked before loading that are more than diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 220681ffe..e781d3bc9 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -37,7 +37,6 @@ def initialize(number, example_groups, jobs, results, options={}) @ppid = options.fetch(:ppid) { Process.method(:ppid) } @watch_coordinator = options.fetch(:watch_coordinator, true) @signal_handlers = options.fetch(:signal_handlers, {}) - @load_seed = options[:load_seed] end # @return [String] the `TEST_ENV_NUMBER` value for this worker, following @@ -111,14 +110,12 @@ def trap_interrupt end end - # Loads the spec files as the coordinator does after forking, from - # its `Kernel.rand` seed, and tells it which examples that defined and - # the first load error, if any: it uses the worker only if its own - # loading went the same way. What loading reports is for the - # coordinator to report. + # Loads the spec files as the coordinator does after forking, and + # tells it which examples that defined and the first load error, if + # any: it uses the worker only if its own loading went the same way. + # What loading reports is for the coordinator to report. # @private def load_spec_files - Kernel.srand(@load_seed) if @load_seed forwarding = @configuration.reporter @configuration.reporter = loading = LoadReporter.new(@configuration) @configuration.load_spec_files diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index ee4514215..712d9c2fd 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -11,13 +11,9 @@ module Parallel # @private module WorkerLifecycle # Forks the workers before the spec files are loaded; each loads them - # itself and reports what it loaded. Ruby reseeds `Kernel.rand` in a - # forked process, so every process starts loading from this one's - # seed, and values drawn while loading are the same in all of them. + # itself and reports what it loaded. def start @started_at = now - @load_seed = Kernel.srand - Kernel.srand(@load_seed) start_workers(nil) end @@ -45,7 +41,7 @@ def start_workers(units) end def fork_worker(worker, workers) - worker.fork_worker(@process, @units, workers, :signal_handlers => @signal_handlers, :load_seed => @load_seed) + worker.fork_worker(@process, @units, workers, :signal_handlers => @signal_handlers) end # Including those a failed `fork` left without a process. diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index acab2dc0e..c6f9a3fc5 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -758,31 +758,39 @@ def rspec_process(*args) expect(last_cmd_stderr).to include("`config.parallel_workers` was set while the spec files were loading", "Running serially.") end - it 'draws the same values with Kernel.rand while loading as a serial run with the same seed' do - write_file_formatted 'spec/spec_helper.rb', "RSpec.configure { |c| Kernel.srand(c.seed) }" + it 'leaves Kernel.rand alone in the coordinating process, never calling srand' do + write_marking_spec_files + previous = Kernel.srand(1234) + + run_rspec '--parallel=2 --no-parallel-preload' + + expect(last_cmd_stdout).to include("3 examples, 0 failures") + expect(Kernel.rand(2**62)).to eq(Random.new(1234).rand(2**62)) + ensure + Kernel.srand(previous) + end + + it 'has every worker draw from a seed of its own, as forked processes do' do %w[a b].each do |name| write_file_formatted "spec/#{name}_spec.rb", " RSpec.describe('#{name.upcase}') do - 3.times do - n = rand(1000) - it(\"handles \#{n}\") do - File.write(\"ran_\#{RSpec.configuration.parallel_worker_number.inspect}\", '') - expect(n).to be_even - end + it('draws once every worker runs an example') do + number = RSpec.configuration.parallel_worker_number + File.write(\"running_\#{number}\", '') + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 20 + sleep 0.05 until Dir['running_*'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + File.write(\"drew_\#{number}\", rand(2**62).to_s) end end " end - _, serial, = rspec_process('--seed', '1234', '--require', './spec/spec_helper') - status, parallel, errors = rspec_process('--seed', '1234', '--require', './spec/spec_helper', '--parallel=2', - '--no-parallel-preload') + run_rspec '--parallel=2 --no-parallel-preload' - expect(errors).to eq("") - expect(cd('.') { Dir['ran_*'] - ['ran_nil'] }).not_to be_empty - expect(serial.lines.grep(/^rspec /)).not_to be_empty - expect(parallel.lines.grep(/^rspec |examples, /)).to match_array(serial.lines.grep(/^rspec |examples, /)) - expect(status.exitstatus).to eq(1) + expect(last_cmd_stdout).to include("2 examples, 0 failures") + drawn = cd('.') { Dir['drew_*'].map { |file| File.read(file) } } + expect(drawn.size).to eq(2) + expect(drawn.uniq.size).to eq(2) end it 'reports the groups and examples as the worker that ran them described them' do diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 751845e9a..33b2d7dc5 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -571,20 +571,6 @@ def run_started(units, worker_count=2, options={}) expect(messages).not_to include("survived SIGKILL") expect(RSpec::Core::Parallel.monotonic_time - started).to be < 4 end - - it "starts this process's `Kernel.rand` sequence over from its seed, the one the workers load the spec files with" do - previous = Kernel.srand(1234) - first = Kernel.rand(1_000_000) - Kernel.rand(1_000_000) - worker_behaviours[1] = worker_behaviours[2] = :never_loads - - instance = started_coordinator([passing_group("A")]) - - expect(Kernel.rand(1_000_000)).to eq(first) - instance.dismiss - ensure - Kernel.srand(previous) - end end it "terminates a worker that sends data it cannot decode" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index a67e4aeff..c5d10f535 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -213,13 +213,13 @@ def frames end describe "forked before the spec files were loaded" do - def loading_worker(options={}) + def loading_worker Worker.new(2, nil, jobs.first, results.last, - { :stderr => stderr, - :exit => lambda { |status| exits << [:exit, status] }, - :exit_now => lambda { |status| exits << [:exit_now, status] }, - :trap => lambda { |*| }, - :watch_coordinator => false }.merge(options)) + :stderr => stderr, + :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, + :watch_coordinator => false) end def write_spec_file(dir, content) @@ -269,22 +269,6 @@ def run_in_thread(instance) expect(exits).to eq([[:exit_now, 0]]) end end - - it "loads them from the coordinating process's `Kernel.rand` seed, which a forked process does not keep" do - previous = Kernel.srand(5) - Dir.mktmpdir do |dir| - write_spec_file(dir, "RSpec.describe(\"drew \#{rand(1_000_000)}\") { it('e') { } }") - thread = run_in_thread(loading_worker(:load_seed => 1234)) - - results.first.receive - jobs.last.send([:stop]) - expect(thread.join(10)).to be_truthy - - expect(RSpec.world.example_groups.first.description).to eq("drew #{Random.new(1234).rand(1_000_000)}") - end - ensure - Kernel.srand(previous) - end end describe "the watchdog" do From e7a6c1956d9d68ab57b871c1937a88fe8c602698 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 22:29:39 +0200 Subject: [PATCH 14/58] [rspec-core] Load and filter large suites without a working-directory lookup per example `File.expand_path` of a relative path asks the OS for the working directory: about 17 microseconds on macOS. Filtering called it for every example even with no location filter (3.3 s for 151,132 examples in one process, 12.5 s while eight parallel workers did the same at once), and loading called it for every example and group to re-expand the relative form of a path RSpec had just loaded by its absolute path. Filtering now skips the lookup when no location or id filter is set, and loading expands the path it was given, which is also right when the working directory changed after `Metadata.relative_path_regex` was memoized. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 3 +++ rspec-core/lib/rspec/core/filter_manager.rb | 9 +++++++-- rspec-core/lib/rspec/core/metadata.rb | 5 ++++- rspec-core/spec/rspec/core/filter_manager_spec.rb | 7 +++++++ rspec-core/spec/rspec/core/metadata_spec.rb | 14 ++++++++++++++ 5 files changed, 35 insertions(+), 3 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 1a32c7842..4b484a3e1 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -27,6 +27,9 @@ Enhancements: is serial with a warning. `--bisect` runs serially, and the reproduction command it prints now ends with `--no-parallel`. (Illia Zub) +* Load and filter large suites faster by no longer looking up the working directory once per + example and example group, which took seconds on a suite of 150,000 examples, and more when + several processes did it at once. (Illia Zub) # 4.0.0.beta1 / 2026-02-18 diff --git a/rspec-core/lib/rspec/core/filter_manager.rb b/rspec-core/lib/rspec/core/filter_manager.rb index 80b3a3304..ff199128d 100644 --- a/rspec-core/lib/rspec/core/filter_manager.rb +++ b/rspec-core/lib/rspec/core/filter_manager.rb @@ -89,9 +89,14 @@ def add_path_to_arrays_filter(filter_key, path, values) # and there is a `:slow => true` exclusion filter), but only for specs # defined in the same file as the location filters. Excluded specs in # other files should still be excluded. + # + # The emptiness checks come first because `File.expand_path` of a + # relative path asks the OS for the working directory, which is slow + # enough (tens of microseconds on macOS, more when several processes + # do it at once) to dominate filtering a large suite. def file_scoped_include?(ex_metadata, ids, locations) - no_id_filters = ids[ex_metadata[:rerun_file_path]].empty? - no_location_filters = locations[ + no_id_filters = ids.empty? || ids[ex_metadata[:rerun_file_path]].empty? + no_location_filters = locations.empty? || locations[ File.expand_path(ex_metadata[:rerun_file_path]) ].empty? diff --git a/rspec-core/lib/rspec/core/metadata.rb b/rspec-core/lib/rspec/core/metadata.rb index ed47bc07b..db9fbb9de 100644 --- a/rspec-core/lib/rspec/core/metadata.rb +++ b/rspec-core/lib/rspec/core/metadata.rb @@ -143,7 +143,10 @@ def populate_location_attributes end relative_file_path = Metadata.relative_path(file_path) - absolute_file_path = File.expand_path(relative_file_path) + # The path as given rather than the relative one: spec files are + # loaded by their absolute path, which expands without asking the + # OS for the working directory, once per example and group. + absolute_file_path = File.expand_path(file_path) metadata[:file_path] = relative_file_path metadata[:line_number] = line_number.to_i metadata[:location] = "#{relative_file_path}:#{line_number}" diff --git a/rspec-core/spec/rspec/core/filter_manager_spec.rb b/rspec-core/spec/rspec/core/filter_manager_spec.rb index babb0b506..de91b4ce4 100644 --- a/rspec-core/spec/rspec/core/filter_manager_spec.rb +++ b/rspec-core/spec/rspec/core/filter_manager_spec.rb @@ -116,6 +116,13 @@ def example_with(*args) RSpec.describe("group", *args).example("example") end + it "does not expand each example's path when no location or id filter is set" do + examples = Array.new(3) { example_with } + expect(File).not_to receive(:expand_path) + + expect(prune(examples)).to eq(examples) + end + shared_examples_for "example identification filter preference" do |type| it "prefers #{type} filter to exclusion filter" do group = RSpec.describe("group") diff --git a/rspec-core/spec/rspec/core/metadata_spec.rb b/rspec-core/spec/rspec/core/metadata_spec.rb index 7e1621e82..0c4046b81 100644 --- a/rspec-core/spec/rspec/core/metadata_spec.rb +++ b/rspec-core/spec/rspec/core/metadata_spec.rb @@ -633,6 +633,20 @@ def group_value_for(*args) end end + describe ":absolute_file_path" do + it "is the path of the file that defined the group, not its relative path expanded in another directory" do + dir = File.join(Dir.tmpdir, "loaded_from_here") + allow(Metadata).to receive(:relative_path_regex).and_return(%r{(\A|\s)#{Regexp.escape(dir)}(/|\s|\Z)}) + + group = instance_eval <<-EOS, File.join(dir, "defined_here_spec.rb"), 1 + RSpec.describe("group") { } + EOS + + expect(group.metadata).to include(:file_path => "./defined_here_spec.rb", + :absolute_file_path => File.join(dir, "defined_here_spec.rb")) + end + end + describe ":line_number" do def value_for(*args) value = nil From 574f1f8c8d4b402bc4b0a6418b33730bb4332edd Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 22:29:48 +0200 Subject: [PATCH 15/58] [rspec-core] Keep the parallel coordinator ahead of its workers on large suites Measured on a 151,132-example suite with 8 workers: - Workers spent ~95 microseconds per result frame, almost all of it marshaling `started_at`/`finished_at`: a local `Time` asks libc for its UTC offset, and in a process forked on macOS every such call reloads the time zone file. They now send whole seconds and nanoseconds and the coordinator rebuilds the `Time`s (~8 microseconds per frame). - The coordinator copied the rest of its read buffer (up to a megabyte) for every frame it took out of it; it now keeps an offset and compacts once per read. - It called `waitpid` on every worker at every wake-up of its loop; it now does so every 50 ms, and at once when a worker's results pipe closes. The coordinator's CPU time fell from ~78 s to ~45 s, with workers blocked less on its full pipes. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/lib/rspec/core/parallel.rb | 14 +++++++++++ rspec-core/lib/rspec/core/parallel/channel.rb | 21 +++++++++++++--- .../lib/rspec/core/parallel/coordinator.rb | 25 +++++++++++++++++-- rspec-core/lib/rspec/core/parallel/results.rb | 2 ++ .../rspec/core/parallel/worker_reporter.rb | 4 +-- rspec-core/spec/integration/parallel_spec.rb | 8 +++--- .../spec/rspec/core/parallel/channel_spec.rb | 12 +++++++++ .../rspec/core/parallel/coordinator_spec.rb | 15 ++++++++++- .../core/parallel/worker_reporter_spec.rb | 5 ++-- rspec-core/spec/rspec/core/parallel_spec.rb | 21 ++++++++++++++++ 10 files changed, 113 insertions(+), 14 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index e0186718e..e7caf677f 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -138,6 +138,20 @@ def self.monotonic_time Process.clock_gettime(Process::CLOCK_MONOTONIC) end + # A `Time` as whole seconds and nanoseconds, the way workers send it: + # marshaling a local `Time` asks libc for its UTC offset, and in a + # process forked on macOS each such call reloads the time zone file + # (tens of microseconds, twice per example). + # @private + def self.time_to_wire(time) + time && [time.tv_sec, time.tv_nsec] + end + + # @private + def self.time_from_wire(value) + value.is_a?(Array) ? ::Time.at(value[0], value[1], :nsec) : value + end + # Whatever the formatters wrote before `start` (seed, run options) may # still be buffered in streams that are not `$stdout`; forked workers # would write it again when they exit. diff --git a/rspec-core/lib/rspec/core/parallel/channel.rb b/rspec-core/lib/rspec/core/parallel/channel.rb index 27d14803b..605740439 100644 --- a/rspec-core/lib/rspec/core/parallel/channel.rb +++ b/rspec-core/lib/rspec/core/parallel/channel.rb @@ -53,6 +53,7 @@ def initialize(io) @io = io @io.set_encoding ENCODING @buffer = String.new(:encoding => ENCODING) + @offset = 0 @pending = [] @eof = false @bytes_read = 0 @@ -108,6 +109,7 @@ def receive # Reads what the kernel has buffered, a bounded number of chunks at a # time so that one chatty worker cannot monopolize the coordinator. def read_available + compact 16.times do break if @eof @@ -122,18 +124,29 @@ def read_available end end + # Drops the frames already taken from the buffer: once per read + # rather than once per frame, which would copy the rest of up to a + # megabyte of buffered frames for each one. + def compact + return if @offset.zero? + + @buffer = @buffer.byteslice(@offset, @buffer.bytesize - @offset) + @offset = 0 + end + # rubocop:disable-next Security/MarshalLoad def next_frame - newline = @buffer.index("\n") + newline = @buffer.index("\n", @offset) return nil unless newline - size = frame_size(@buffer.byteslice(0, newline)) + size = frame_size(@buffer.byteslice(@offset, newline - @offset)) frame_end = newline + 1 + size return nil if @buffer.bytesize < frame_end packet = @buffer.byteslice(newline + 1, size) - @buffer = @buffer.byteslice(frame_end, @buffer.bytesize - frame_end) - [Marshal.load(packet), frame_end] + consumed = frame_end - @offset + @offset = frame_end + [Marshal.load(packet), consumed] end def frame_size(header) diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index f7c0739a1..8e4f35171 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -14,6 +14,10 @@ class Coordinator include WorkerLifecycle TICK = 0.25 + # Between checks for workers that exited without closing their + # results pipe (a child of theirs holds it open, say); one whose pipe + # closed is checked for at once. + REAP_INTERVAL = 0.05 TERM_GRACE = 5.0 KILL_GRACE = 5.0 # Added to the shutdown timeout on the coordinator's side so that the @@ -44,6 +48,8 @@ def initialize(runner, worker_count, options={}) @termination_signal = nil @signal_handlers = {} @started_at = nil + @reap_due = true + @next_reap_at = 0.0 end # The groups to run. Workers forked before loading that are more than @@ -94,7 +100,7 @@ def event_loop return stop_for_signal if @termination_signal wait_for_results - reap_workers + reap_workers_when_due request_quit if @world.wants_to_quit && !@quitting dispatch_work enforce_deadlines @@ -117,6 +123,7 @@ def wait_for_results # Each worker left is exiting (its results pipe is at its end) or # was given up on: all there is to do is reap it or enforce its # deadline, and an exiting worker is reapable within moments. + @reap_due = true sleep [@tick, 0.01].min return end @@ -133,9 +140,12 @@ def receive_from(worker) messages = drain(worker) messages.each { |message, bytes| handle_message(worker, message, bytes) } + return unless worker.results.eof? + # A worker whose results pipe closed while it was still running is # either about to be reaped or wedged; bound the wait for it. - worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace if worker.results.eof? + @reap_due = true + worker.deadline ||= now + @configuration.parallel_shutdown_timeout + @watchdog_grace end def drain(worker) @@ -289,6 +299,17 @@ def write(worker, message) false end + # `waitpid` for every worker on every wake-up of the loop -- once per + # few frames when results stream in -- cost the coordinator a sixth of + # its time on a large suite. + def reap_workers_when_due + return unless @reap_due || now >= @next_reap_at + + @reap_due = false + @next_reap_at = now + REAP_INTERVAL + reap_workers + end + def reap_workers @workers.each do |worker| next if worker.done? || worker.pid.nil? diff --git a/rspec-core/lib/rspec/core/parallel/results.rb b/rspec-core/lib/rspec/core/parallel/results.rb index a18b9701b..1474a2367 100644 --- a/rspec-core/lib/rspec/core/parallel/results.rb +++ b/rspec-core/lib/rspec/core/parallel/results.rb @@ -25,6 +25,8 @@ def apply(example) example.assign_execution_result( payload.merge( + :started_at => Parallel.time_from_wire(payload[:started_at]), + :finished_at => Parallel.time_from_wire(payload[:finished_at]), :exception => RemoteException.rebuild(payload[:exception]), :pending_exception => RemoteException.rebuild(payload[:pending_exception]) ) diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 17d96db9c..e30000650 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -114,8 +114,8 @@ def result_payload(example) { :status => result.status, - :started_at => result.started_at, - :finished_at => result.finished_at, + :started_at => Parallel.time_to_wire(result.started_at), + :finished_at => Parallel.time_to_wire(result.finished_at), :run_time => result.run_time, :pending_message => Parallel.marshal_safe(result.pending_message, &:to_s), :pending_fixed => result.pending_fixed, diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index c6f9a3fc5..2a8abea6a 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -247,7 +247,9 @@ def example_group_finished(notification) end def example_passed(notification) - @output.puts \"\#{' ' * @depth}ok \#{notification.example.metadata[:flag].inspect}\" + result = notification.example.execution_result + timed = result.started_at.is_a?(Time) && result.finished_at.is_a?(Time) && result.finished_at >= result.started_at + @output.puts \"\#{' ' * @depth}ok \#{notification.example.metadata[:flag].inspect} timed=\#{timed}\" end def example_failed(notification) @@ -282,12 +284,12 @@ def message(notification) expect(last_cmd_stdout).to include( "> One (unit)", - " ok 7", + " ok 7 timed=true", " FAILED MyAppError app_error=true std=true msg=app broke", "Failure/Error: it('raises an app error') { raise MyAppError, 'app broke' }", "< One", "> Two (feature)", - " ok nil", + " ok nil timed=true", "< Two", "custom: #{{ :from => 'worker' }.inspect}", "message: Parallel worker 1 could not forward the custom `my_event` notification because its options cannot be " \ diff --git a/rspec-core/spec/rspec/core/parallel/channel_spec.rb b/rspec-core/spec/rspec/core/parallel/channel_spec.rb index 606c5cf98..1aa3d5e9e 100644 --- a/rspec-core/spec/rspec/core/parallel/channel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/channel_spec.rb @@ -32,6 +32,18 @@ module RSpec::Core::Parallel expect(reader.drain.map(&:first)).to eq([0, 1, 2]) end + it "keeps a partial frame that follows whole ones, and reports each frame's own size" do + packets = [:first, "second" * 100, :third].map { |message| Marshal.dump(message) } + frames = packets.map { |packet| "#{packet.bytesize}\n#{packet}" } + writer.io.write(frames[0] + frames[1] + frames[2][0, 5]) + + expect(reader.drain).to eq([[:first, frames[0].bytesize], ["second" * 100, frames[1].bytesize]]) + + writer.io.write(frames[2][5..-1]) + expect(reader.drain).to eq([[:third, frames[2].bytesize]]) + expect(reader.bytes_read).to eq(frames.sum(&:bytesize)) + end + it "keeps a partial frame buffered without blocking until the rest arrives" do packet = Marshal.dump(:whole) writer.io.write("#{packet.bytesize}\n#{packet[0, 3]}") diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 33b2d7dc5..8763429cb 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -13,7 +13,7 @@ def signaled? end end - attr_reader :signals, :warmups + attr_reader :signals, :warmups, :waits def threads @threads.values @@ -25,6 +25,7 @@ def initialize(survives=[]) @signals = [] @survives = survives @warmups = 0 + @waits = 0 @next_pid = 40_000 end @@ -40,6 +41,7 @@ def warmup end def waitpid2(pid, _flags) + @waits += 1 thread = @threads.fetch(pid) return nil if thread.alive? @@ -190,6 +192,17 @@ def messages output.string end + it "checks for exited workers a bounded number of times while results stream in, not on every wake-up" do + groups = %w[A B].map { |name| RSpec.describe(name) { 400.times { |i| it("passes #{i}") {} } } } + started = RSpec::Core::Parallel.monotonic_time + + expect(coordinator(groups).run).to eq(0) + + elapsed = RSpec::Core::Parallel.monotonic_time - started + expect(messages).to include("800 examples, 0 failures") + expect(process.waits).to be <= 2 * ((elapsed / Coordinator::REAP_INTERVAL).ceil + 6) + end + it "runs every group across the workers, reports through the configured formatters and exits 0 when all pass" do groups = %w[A B C D E].map { |name| passing_group(name) } diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index 36008ecf3..5dae4ec16 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -50,8 +50,9 @@ def frames expect(passed).to include(:status => :passed, :exception => nil, :pending_exception => nil, :pending => nil, :skip => nil, :description => "passes", :full_description => "Outer passes") - expect(passed[:started_at]).to be_a(Time) - expect(passed[:finished_at]).to be_a(Time) + expect(passed[:started_at]).to eq([group.examples.first.execution_result.started_at.tv_sec, + group.examples.first.execution_result.started_at.tv_nsec]) + expect(passed[:finished_at]).to all(be_an(Integer)) expect(passed[:run_time]).to be_a(Float) expect(failed[:status]).to eq(:failed) diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 574e1ec9e..e798aa4fb 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -196,6 +196,27 @@ def groups(count) end end + describe ".time_to_wire and .time_from_wire" do + it "carry a local Time as integers and rebuild one equal to it, to the nanosecond" do + time = ::Time.at(1_791_231_154, 100_773_123, :nsec) + + wire = Parallel.time_to_wire(time) + rebuilt = Parallel.time_from_wire(Marshal.load(Marshal.dump(wire))) + + expect(wire).to eq([1_791_231_154, 100_773_123]) + expect(rebuilt).to eq(time) + expect([rebuilt.nsec, rebuilt.utc_offset]).to eq([time.nsec, time.utc_offset]) + end + + it "leave nil, and values that are not on the wire, as they are" do + time = ::Time.now + + expect(Parallel.time_to_wire(nil)).to be_nil + expect(Parallel.time_from_wire(nil)).to be_nil + expect(Parallel.time_from_wire(time)).to be(time) + end + end + describe "signal handling", :skip => (Signal.list.key?("HUP") ? false : "needs SIGHUP") do around do |example| original = trap("HUP", "DEFAULT") From 0be8e129939c658524de00f9b561c87e2309a93b Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 23:07:25 +0200 Subject: [PATCH 16/58] [rspec-core] Run with the parallel workers the system allows, and don't count a stopped run against their deadlines - A worker's pipes are now opened just before it is forked, so the coordinator holds 2 file descriptors per worker rather than 4: 24 workers fit in a 64-descriptor limit. When the system refuses a worker (EMFILE, ENFILE, EAGAIN, ENOMEM) the run goes on with the workers it got, and serially if it got none, with a warning; it used to end with the error and 0 examples run. - Deadlines exclude time the coordinator's loop was not running beyond its own waits, and a worker's exit watchdog counts only time the worker ran: a run stopped with Ctrl-Z (or a machine asleep) while workers ran their after(:suite) hooks reported each one as stuck and failed. - The watchdog aborts the worker even when an example closed the stream it dumps the threads to. - Warnings no longer end in two periods. - The README says how to get coverage from the workers with SimpleCov (`merge_subprocesses`): rspec-core's suite measures 100% under `--parallel=8` with it, 61% without. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- rspec-core/README.md | 20 ++++-- rspec-core/lib/rspec/core/configuration.rb | 2 +- rspec-core/lib/rspec/core/parallel.rb | 23 ++++++- .../lib/rspec/core/parallel/coordinator.rb | 30 ++++----- rspec-core/lib/rspec/core/parallel/plan.rb | 3 +- rspec-core/lib/rspec/core/parallel/worker.rb | 8 ++- .../lib/rspec/core/parallel/worker_handle.rb | 7 +- .../rspec/core/parallel/worker_lifecycle.rb | 67 +++++++++++++++++-- .../rspec/core/parallel/coordinator_spec.rb | 53 +++++++++++++-- .../spec/rspec/core/parallel/plan_spec.rb | 21 ++++-- .../rspec/core/parallel/worker_handle_spec.rb | 11 +++ .../spec/rspec/core/parallel/worker_spec.rb | 9 +++ rspec-core/spec/rspec/core/parallel_spec.rb | 31 +++++++++ 13 files changed, 239 insertions(+), 46 deletions(-) create mode 100644 rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb diff --git a/rspec-core/README.md b/rspec-core/README.md index 5510bb327..d9a348627 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -379,14 +379,24 @@ one that cannot tear down -- before the process ends the way a serial run would, and the output says which example each worker was running and for how long, which is usually why the run had to be stopped; a signal the run was started ignoring (`nohup`), or that your suite installs a handler for, is left -alone. +alone. Time the whole run spends stopped (Ctrl-Z) or the machine asleep does +not count towards these timeouts. `--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a -warning and runs serially. `--bisect` and `--dry-run` always run serially, and -the reproduction command `--bisect` prints includes `--no-parallel`, which (like -`--parallel=0`) overrides `--parallel` in `.rspec` and `config.parallel_workers` -in a helper, and leaves `--parallel` out of its `SPEC_OPTS`. +warning and runs serially. When the system refuses a worker -- too many open +files or processes -- the run goes on with the workers it got, and serially if +it got none, with a warning. `--bisect` and `--dry-run` always run serially, +and the reproduction command `--bisect` prints includes `--no-parallel`, which +(like `--parallel=0`) overrides `--parallel` in `.rspec` and +`config.parallel_workers` in a helper, and leaves `--parallel` out of its +`SPEC_OPTS`. + +Each worker is a separate process, so a coverage tool records only the +coordinating process unless it merges forked processes. With SimpleCov, set +`merge_subprocesses true` in the `SimpleCov.start` block (SimpleCov 1.3; it +was `enable_for_subprocesses` before): rspec-core's own suite then measures +the same coverage with `--parallel=8`, preloaded or not, as it does serially. ## Store Command Line Options `.rspec` diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 1f0f584fa..e663af919 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -258,7 +258,7 @@ def parallel_workers=(value) @parallel_workers = value else RSpec.warning "Cannot set `RSpec.configuration.parallel_workers` to `#{value.inspect}`; " \ - "it must be a positive integer, `:auto` or `nil`. Running serially." + "it must be a positive integer, `:auto` or `nil`. Running serially" @parallel_workers = nil end end diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index e7caf677f..f1e29d8b4 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -67,13 +67,13 @@ def self.requested_worker_count(configuration) unless RSpec::Support::RubyFeatures.fork_supported? RSpec.warning "Ignoring `--parallel`: it needs `fork`, which is not available on #{RUBY_ENGINE} " \ - "(#{RUBY_PLATFORM}). Running serially.", :call_site => nil + "(#{RUBY_PLATFORM}). Running serially", :call_site => nil return nil end if ENV.key?('TEST_ENV_NUMBER') RSpec.warning "Ignoring `--parallel`: this process is already a worker of a parallel test run " \ - "(TEST_ENV_NUMBER is set). Running serially.", :call_site => nil + "(TEST_ENV_NUMBER is set). Running serially", :call_site => nil return nil end @@ -138,6 +138,25 @@ def self.monotonic_time Process.clock_gettime(Process::CLOCK_MONOTONIC) end + # Longer than any wait of the coordinator's loop or of a worker's + # watchdog: a longer gap means the process was not running -- stopped + # with the whole run (Ctrl-Z), the machine asleep -- and that time does + # not count towards a deadline. + PAUSE_TOLERANCE = 1.0 + + # Sleeps until this process has been running for `seconds`. + # @private + def self.sleep_running(seconds, slice=0.25) + remaining = seconds + while remaining > 0 + step = [remaining, slice].min + started = monotonic_time + sleep step + elapsed = monotonic_time - started + remaining -= elapsed > step + PAUSE_TOLERANCE ? step : elapsed + end + end + # A `Time` as whole seconds and nanoseconds, the way workers send it: # marshaling a local `Time` asks libc for its UTC offset, and in a # process forked on macOS each such call reloads the time zone file diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 8e4f35171..990745921 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -14,10 +14,6 @@ class Coordinator include WorkerLifecycle TICK = 0.25 - # Between checks for workers that exited without closing their - # results pipe (a child of theirs holds it open, say); one whose pipe - # closed is checked for at once. - REAP_INTERVAL = 0.05 TERM_GRACE = 5.0 KILL_GRACE = 5.0 # Added to the shutdown timeout on the coordinator's side so that the @@ -50,6 +46,7 @@ def initialize(runner, worker_count, options={}) @started_at = nil @reap_due = true @next_reap_at = 0.0 + @last_turn_at = nil end # The groups to run. Workers forked before loading that are more than @@ -66,12 +63,15 @@ def assign(example_groups, units) def run reporter = @configuration.reporter @signal_handlers = Parallel.trap_signals(TERMINATION_SIGNALS) { |signal| @termination_signal ||= signal } + loading = !@workers.empty? + start_workers(@units) unless loading + return run_serially if @workers.empty? passed = reporter.report(@world.example_count(@example_groups)) do @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) begin - @workers.empty? ? start_workers(@units) : expect_loading_workers + expect_loading_workers if loading event_loop ensure terminate_remaining_workers @@ -88,11 +88,19 @@ def run @runner.exit_code(passed) ensure + terminate_remaining_workers Parallel.restore_signals(@signal_handlers) end private + # No worker could be started at all. + def run_serially + Parallel.restore_signals(@signal_handlers) + @signal_handlers = {} + @runner.run_specs(@example_groups) + end + def event_loop dispatch_work @@ -103,6 +111,7 @@ def event_loop reap_workers_when_due request_quit if @world.wants_to_quit && !@quitting dispatch_work + account_for_pauses enforce_deadlines @sequencer.tick end @@ -299,17 +308,6 @@ def write(worker, message) false end - # `waitpid` for every worker on every wake-up of the loop -- once per - # few frames when results stream in -- cost the coordinator a sixth of - # its time on a large suite. - def reap_workers_when_due - return unless @reap_due || now >= @next_reap_at - - @reap_due = false - @next_reap_at = now + REAP_INTERVAL - reap_workers - end - def reap_workers @workers.each do |worker| next if worker.done? || worker.pid.nil? diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb index 31bf92dde..acad8230f 100644 --- a/rspec-core/lib/rspec/core/parallel/plan.rb +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -33,6 +33,7 @@ def before_loading # are dismissed too. @coordinator = Coordinator.new(@runner, count) @coordinator.start + @coordinator = nil unless @coordinator.started? end # @return [Coordinator, nil] the coordinator to run the loaded groups, @@ -82,7 +83,7 @@ def warn_late what = @postponed_by || '`config.parallel_preload = false`' RSpec.warning "Ignoring `--parallel`: #{what} was set while the spec files were loading, too late to fork " \ "the workers before them as `config.parallel_preload = false` asks. Set it before they " \ - "load: on the command line, in `.rspec` or in a `--require`d file. Running serially.", + "load: on the command line, in `.rspec` or in a `--require`d file. Running serially", :call_site => nil end end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index e781d3bc9..d09669b40 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -157,8 +157,12 @@ def run_unit(index, id) def arm_watchdog timeout = @configuration.parallel_shutdown_timeout thread = Thread.new do - sleep timeout - dump_threads(timeout) + Parallel.sleep_running(timeout) + begin + dump_threads(timeout) + rescue IOError, SystemCallError + nil # stderr was closed or replaced; aborting matters more than the dump + end @exit_now.call(1) end thread.report_on_exception = false diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 2c269a193..555585521 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -13,7 +13,12 @@ class WorkerHandle def initialize(number) @number = number @child_jobs, @jobs = Channel.pipe - @results, @child_results = Channel.pipe + begin + @results, @child_results = Channel.pipe + rescue SystemCallError + [@child_jobs, @jobs].each(&:close) + raise + end @state = :running @quit_sent = false @exiting = false diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 712d9c2fd..dd9843f4e 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -10,6 +10,16 @@ module Parallel # rest within the TERM/KILL grace periods. # @private module WorkerLifecycle + # What the system answers when it will not give this process another + # worker right now: no file descriptors for its pipes, no process for + # it. The run goes on with the workers it has. + OUT_OF_RESOURCES = [Errno::EMFILE, Errno::ENFILE, Errno::EAGAIN, Errno::ENOMEM].freeze + + # Between checks for workers that exited without closing their + # results pipe (a child of theirs holds it open, say); one whose pipe + # closed is checked for at once. + REAP_INTERVAL = 0.05 + # Forks the workers before the spec files are loaded; each loads them # itself and reports what it loaded. def start @@ -17,6 +27,11 @@ def start start_workers(nil) end + # @return [Boolean] whether any worker could be started + def started? + !@workers.empty? + end + # Ends the workers that are not done -- workers forked before loading # when the run ends early or goes on serially: they have run nothing. def dismiss @@ -27,17 +42,57 @@ def dismiss private - # Without `units` the workers load the spec files themselves. + # Rather than a `waitpid` for every worker on every wake-up of the + # loop, which comes every few frames while results stream in. + def reap_workers_when_due + return unless @reap_due || now >= @next_reap_at + + @reap_due = false + @next_reap_at = now + REAP_INTERVAL + reap_workers + end + + # Moves every deadline back by the time the coordinator's loop was not + # running beyond its own waits: the run was stopped (Ctrl-Z) or the + # machine slept, its workers with it. + def account_for_pauses + current = now + paused = @last_turn_at ? current - @last_turn_at - @tick - Parallel::PAUSE_TOLERANCE : 0.0 + @last_turn_at = current + return unless paused > 0 + + @workers.each { |worker| worker.deadline += paused if worker.deadline } + end + + # Without `units` the workers load the spec files themselves. Each + # worker's pipes are opened just before it is forked, so that the + # parent never holds more than its own ends of the earlier ones. def start_workers(units) Parallel.flush_output(@configuration) @process.warmup if @process.respond_to?(:warmup) - @worker_count.times { |index| @workers << WorkerHandle.new(index + 1) } - @workers.each do |worker| - worker.loading = units.nil? - worker.pid = @spawn.call(worker, @workers) - worker.close_child_ends + @worker_count.times do |index| + break unless start_worker(index + 1, units) + end + end + + # @return [Boolean] whether the worker started + def start_worker(number, units) + worker = WorkerHandle.new(number) + worker.loading = units.nil? + @workers << worker + worker.pid = @spawn.call(worker, @workers) + worker.close_child_ends + true + rescue *OUT_OF_RESOURCES => e + if worker + @workers.delete(worker) + worker.close_all end + RSpec.warning "Started #{@workers.size} of #{@worker_count} parallel workers: the next could not be started " \ + "(#{e.class}: #{e.message}). #{@workers.empty? ? 'Running serially' : 'Going on without it'}", + :call_site => nil + false end def fork_worker(worker, workers) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 8763429cb..2998dbb4d 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -200,7 +200,7 @@ def messages elapsed = RSpec::Core::Parallel.monotonic_time - started expect(messages).to include("800 examples, 0 failures") - expect(process.waits).to be <= 2 * ((elapsed / Coordinator::REAP_INTERVAL).ceil + 6) + expect(process.waits).to be <= 2 * ((elapsed / WorkerLifecycle::REAP_INTERVAL).ceil + 6) end it "runs every group across the workers, reports through the configured formatters and exits 0 when all pass" do @@ -566,7 +566,7 @@ def run_started(units, worker_count=2, options={}) survivor.threads.each(&:kill) end - it "dismisses the workers it forked when a later fork fails, and does not wait for those it never forked" do + it "goes on with the workers it could fork when a later fork fails, and dismisses only those" do worker_behaviours[1] = :never_loads fork_fails = lambda do |handle, handles| raise Errno::EAGAIN, "fork(2)" if handle.number == 2 @@ -574,8 +574,12 @@ def run_started(units, worker_count=2, options={}) spawn(handle, handles) end instance = new_coordinator([passing_group("A")], 3, :spawn => fork_fails) + expect(RSpec).to receive(:warning).with("Started 1 of 3 parallel workers: the next could not be started " \ + "(Errno::EAGAIN: Resource temporarily unavailable - fork(2)). Going on without it", + :call_site => nil) - expect { instance.start }.to raise_error(Errno::EAGAIN) + instance.start + expect(instance).to be_started started = RSpec::Core::Parallel.monotonic_time instance.dismiss @@ -586,6 +590,20 @@ def run_started(units, worker_count=2, options={}) end end + it "moves its workers' deadlines back by the time its loop was paused beyond its own waits" do + instance = coordinator([passing_group("A")]) + worker = RSpec::Core::Parallel::WorkerHandle.new(1) + worker.deadline = 50.0 + instance.instance_variable_get(:@workers) << worker + allow(instance).to receive(:now).and_return(10.0, 10.2, 30.2) + + 3.times { instance.send(:account_for_pauses) } + + expect(worker.deadline).to be_within(0.001).of(50.0 + 20.0 - 0.01 - RSpec::Core::Parallel::PAUSE_TOLERANCE) + ensure + worker&.close_all + end + it "terminates a worker that sends data it cannot decode" do worker_behaviours[2] = :garbage @@ -674,15 +692,36 @@ def example_passed(_notification) expect(file_output.string).to include("2 examples, 0 failures") end - it "copes with workers that could not be spawned when the coordinator fails" do + it "runs every group in the workers it could start when the system refuses more" do failing_spawn = lambda do |handle, handles| - raise Errno::EAGAIN, "fork" if handle.number == 2 + raise Errno::EMFILE if handle.number > 1 spawn(handle, handles) end + expect(RSpec).to receive(:warning).with(a_string_starting_with("Started 1 of 3 parallel workers: the next could not " \ + "be started (Errno::EMFILE: Too many open files)"), + :call_site => nil) - expect { coordinator([passing_group("A"), passing_group("B")], 2, :spawn => failing_spawn).run }.to raise_error(Errno::EAGAIN) - expect(process.signals.map(&:first)).to eq(["TERM"]) + exit_code = coordinator(%w[A B C].map { |name| passing_group(name) }, 3, :spawn => failing_spawn).run + + expect(exit_code).to eq(0) + expect(messages).to include("3 examples, 0 failures") + expect(process.signals).to be_empty + end + + it "runs the groups in this process, like a serial run, when it cannot start any worker" do + ran_in = [] + groups = %w[A B].map { |name| RSpec.describe(name) { it("records its process") { ran_in << Process.pid } } } + config.before(:suite) { ran_in << :before_suite } + expect(RSpec).to receive(:warning).with(a_string_including("Started 0 of 2 parallel workers").and(ending_with("Running serially")), + :call_site => nil) + + exit_code = coordinator(groups, 2, :spawn => lambda { |*| raise Errno::EAGAIN, "fork" }).run + + expect(exit_code).to eq(0) + expect(ran_in).to eq([:before_suite, Process.pid, Process.pid]) + expect(messages).to include("2 examples, 0 failures") + expect(Signal.trap("TERM", "DEFAULT")).not_to be_a(Proc) end it "treats a non-example failure reported by a worker as one of the run" do diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb index 8a173bbad..509e639bc 100644 --- a/rspec-core/spec/rspec/core/parallel/plan_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -3,7 +3,7 @@ module RSpec::Core let(:config) { RSpec.configuration } let(:runner) { Runner.new(ConfigurationOptions.new([]), config, RSpec.world) } let(:plan) { Parallel::Plan.new(runner) } - let(:started) { instance_double(Parallel::Coordinator, :start => nil, :assign => nil, :dismiss => nil) } + let(:started) { instance_double(Parallel::Coordinator, :start => nil, :started? => true, :assign => nil, :dismiss => nil) } around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } @@ -58,17 +58,28 @@ def spec_files(count) plan.before_loading end - it "dismisses the workers it forked when a later fork fails" do + it "dismisses the workers it forked when starting the others fails unexpectedly" do config.parallel_workers = 2 failing = instance_double(Parallel::Coordinator, :dismiss => nil) - allow(failing).to receive(:start).and_raise(Errno::EAGAIN) + allow(failing).to receive(:start).and_raise(RuntimeError, "unexpected") allow(Parallel::Coordinator).to receive(:new).and_return(failing) - expect { plan.before_loading }.to raise_error(Errno::EAGAIN) + expect { plan.before_loading }.to raise_error(RuntimeError, "unexpected") plan.dismiss expect(failing).to have_received(:dismiss) end + it "runs serially when no worker could be started" do + config.parallel_workers = 2 + allow(Parallel::Coordinator).to receive(:new).and_return(started) + allow(started).to receive(:started?).and_return(false) + + plan.before_loading + + expect(plan.coordinator_for(groups(3))).to be_nil + expect(started).not_to have_received(:assign) + end + it "dismisses them, and runs serially, when fewer than two groups are to run" do config.parallel_workers = 3 allow(Parallel::Coordinator).to receive(:new).and_return(started) @@ -110,7 +121,7 @@ def spec_files(count) expect(RSpec).to receive(:warning).with(a_string_including( "Ignoring `--parallel`: `config.parallel_workers` was set while the spec files were loading, too late to fork " \ "the workers before them as `config.parallel_preload = false` asks." - ).and(ending_with("Running serially.")), :call_site => nil) + ).and(ending_with("Running serially")), :call_site => nil) expect(Parallel::Coordinator).not_to receive(:new) expect(plan.coordinator_for(groups(3))).to be_nil diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb new file mode 100644 index 000000000..9087c456d --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -0,0 +1,11 @@ +module RSpec::Core::Parallel + RSpec.describe WorkerHandle do + it "closes the pipe it opened when the system refuses it the second one" do + first = Channel.pipe + allow(Channel).to receive(:pipe).and_invoke(lambda { first }, lambda { raise Errno::EMFILE }) + + expect { WorkerHandle.new(1) }.to raise_error(Errno::EMFILE) + expect(first.map(&:closed?)).to eq([true, true]) + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index c5d10f535..e8d4c5ccd 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -292,6 +292,15 @@ def run_in_thread(instance) ensure blocked&.kill end + + it "still aborts the process when an example closed the stream it would dump the threads to" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + stderr.close + + worker(4).arm_watchdog.join + + expect(exits).to eq([[:exit_now, 1]]) + end end end end diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index e798aa4fb..f5c4e2b5f 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -196,6 +196,37 @@ def groups(count) end end + describe ".sleep_running" do + it "sleeps until the process has run that long, counting a slice it was paused through as the slice" do + clock = 100.0 + slices = [] + allow(Parallel).to receive(:monotonic_time) { clock } + allow(Parallel).to receive(:sleep) do |seconds| + slices << seconds + clock += slices.size == 2 ? 30.0 : seconds + end + + Parallel.sleep_running(1.0, 0.25) + + expect(slices).to eq([0.25, 0.25, 0.25, 0.25]) + expect(clock).to eq(100.0 + 0.75 + 30.0) + end + + it "counts a slice that merely ran late, within the pause tolerance, as the time it took" do + clock = 0.0 + slices = [] + allow(Parallel).to receive(:monotonic_time) { clock } + allow(Parallel).to receive(:sleep) do |seconds| + slices << seconds + clock += seconds + 0.5 + end + + Parallel.sleep_running(1.0, 0.25) + + expect(slices).to eq([0.25, 0.25]) + end + end + describe ".time_to_wire and .time_from_wire" do it "carry a local Time as integers and rebuild one equal to it, to the nanosecond" do time = ::Time.at(1_791_231_154, 100_773_123, :nsec) From 60eb93de74f0d47a602b7261f29f03236256a447 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 5 Oct 2026 23:57:14 +0200 Subject: [PATCH 17/58] [rspec-core] Postpone a parallel worker's deadline only for pauses after it was set Two gaps in the previous commit, found in review and reproduced with SIGSTOP/SIGCONT on the process group: - A pause in the first turn of the coordinator's loop was not counted, so a Ctrl-Z just after it loaded the spec files still killed every worker that was loading them itself. - A deadline set after a pause, later in the same turn (a --fail-fast stop, a worker's results pipe closing), was postponed by the whole pause, so a worker stuck in an example could outlive its timeout by as long as the run had been stopped. Each deadline now records when it was set and is postponed only by the part of a pause that came after it, counted from the start of the loop. This is by Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/coordinator.rb | 1 + .../lib/rspec/core/parallel/worker_handle.rb | 18 +++++-- .../rspec/core/parallel/worker_lifecycle.rb | 17 +++--- .../rspec/core/parallel/coordinator_spec.rb | 54 ++++++++++++++----- 4 files changed, 69 insertions(+), 21 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 990745921..e91b66bdc 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -71,6 +71,7 @@ def run @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) begin + @last_turn_at = now expect_loading_workers if loading event_loop ensure diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 555585521..c12287347 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -5,7 +5,9 @@ module Parallel # @private class WorkerHandle attr_reader :number, :jobs, :results, :child_jobs, :child_results, :current_example, :current_example_started_at - attr_accessor :pid, :current_unit, :state, :deadline, :escalation, :exit_status, :quit_sent, :exiting + attr_accessor :pid, :current_unit, :state, :escalation, :exit_status, :quit_sent, :exiting + # When the deadline was set: only a pause after that postpones it. + attr_reader :deadline, :deadline_set_at # Forked before the spec files were loaded and has not yet reported # what it loaded. attr_accessor :loading @@ -23,10 +25,20 @@ def initialize(number) @quit_sent = false @exiting = false @loading = false - @pid = @current_unit = @deadline = @escalation = @exit_status = nil + @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = nil @current_example = @current_example_started_at = nil end + def deadline=(value) + @deadline = value + @deadline_set_at = value && Parallel.monotonic_time + end + + # Moves the deadline back without changing when it was set. + def postpone(seconds) + @deadline += seconds if @deadline + end + def current_example=(example) @current_example = example @current_example_started_at = example && RSpec::Core::Time.now @@ -111,7 +123,7 @@ def reap(process) # `deadline`. def signal(process, signal_name, escalation, deadline) @escalation = escalation - @deadline = deadline + self.deadline = deadline process.kill(signal_name, @pid) if @pid rescue Errno::ESRCH, Errno::EPERM nil diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index dd9843f4e..6bd12a136 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -52,16 +52,21 @@ def reap_workers_when_due reap_workers end - # Moves every deadline back by the time the coordinator's loop was not - # running beyond its own waits: the run was stopped (Ctrl-Z) or the - # machine slept, its workers with it. + # Moves each deadline back by the time the coordinator's loop was not + # running, beyond its own waits, since the deadline was set: the run + # was stopped (Ctrl-Z) or the machine slept, its workers with it. def account_for_pauses current = now - paused = @last_turn_at ? current - @last_turn_at - @tick - Parallel::PAUSE_TOLERANCE : 0.0 + turn_started = @last_turn_at @last_turn_at = current - return unless paused > 0 - @workers.each { |worker| worker.deadline += paused if worker.deadline } + @workers.each do |worker| + next unless worker.deadline + + since = [turn_started, worker.deadline_set_at].compact.max + paused = current - since - @tick - Parallel::PAUSE_TOLERANCE + worker.postpone(paused) if paused > 0 + end end # Without `units` the workers load the spec files themselves. Each diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 2998dbb4d..2d6656887 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -590,18 +590,48 @@ def run_started(units, worker_count=2, options={}) end end - it "moves its workers' deadlines back by the time its loop was paused beyond its own waits" do - instance = coordinator([passing_group("A")]) - worker = RSpec::Core::Parallel::WorkerHandle.new(1) - worker.deadline = 50.0 - instance.instance_variable_get(:@workers) << worker - allow(instance).to receive(:now).and_return(10.0, 10.2, 30.2) - - 3.times { instance.send(:account_for_pauses) } - - expect(worker.deadline).to be_within(0.001).of(50.0 + 20.0 - 0.01 - RSpec::Core::Parallel::PAUSE_TOLERANCE) - ensure - worker&.close_all + describe "pauses of the whole run (Ctrl-Z, a sleeping machine)" do + let(:clock) { [10.0] } + let(:instance) { coordinator([passing_group("A")]) } + let(:workers) { Array.new(2) { |i| RSpec::Core::Parallel::WorkerHandle.new(i + 1) } } + + before do + allow(RSpec::Core::Parallel).to receive(:monotonic_time) { clock.first } + instance.instance_variable_get(:@workers).concat(workers) + instance.instance_variable_set(:@last_turn_at, clock.first) + end + + after { workers.each(&:close_all) } + + def turn(seconds) + clock[0] += seconds + instance.send(:account_for_pauses) + end + + it "move a deadline back by the time the loop was paused beyond its own waits" do + workers[0].deadline = 50.0 + turn(0.2) + turn(20.0) + + expect(workers[0].deadline).to be_within(0.001).of(50.0 + 20.0 - 0.01 - RSpec::Core::Parallel::PAUSE_TOLERANCE) + end + + it "leave a deadline set after the pause, in the same turn, as it is" do + workers[0].deadline = 50.0 + clock[0] += 20.0 + workers[1].deadline = 60.0 + turn(0.1) + + expect(workers[0].deadline).to be > 65.0 + expect(workers[1].deadline).to eq(60.0) + end + + it "leave deadlines alone in turns of normal length" do + workers[0].deadline = 50.0 + 5.times { turn(0.25) } + + expect(workers[0].deadline).to eq(50.0) + end end it "terminates a worker that sends data it cannot decode" do From 46fbdf05eac3676a5457d36a5326e72677f84f02 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 19:21:00 +0200 Subject: [PATCH 18/58] [rspec-core] Keep --parallel working with only custom formatters Deciding whether formatters follow group boundaries compared their methods with BaseFormatter's, which only a built-in formatter or a subclass of it loads. With a single custom formatter that does not inherit it, `rspec --parallel` raised NameError before running anything; rspec-core's own spec for custom formatters passed only when an earlier spec in the same process had loaded BaseFormatter. Co-Authored-By: Claude Opus 5.5 --- rspec-core/lib/rspec/core/parallel/sequencer.rb | 4 +++- rspec-core/spec/rspec/core/parallel/sequencer_spec.rb | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb index ad88a2907..e1138577f 100644 --- a/rspec-core/lib/rspec/core/parallel/sequencer.rb +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -72,9 +72,11 @@ def initialize(reporter, options={}) # current group and RSpec's profiler keeps its own per-group tally, # so neither needs sequencing. def self.group_aware_listeners?(reporter) + # Not loaded when every formatter is a custom one that does not inherit it. + base_formatter = Formatters::BaseFormatter if defined?(Formatters::BaseFormatter) [:example_group_started, :example_group_finished].any? do |event| reporter.registered_listeners(event).any? do |listener| - !listener.is_a?(Profiler) && listener.method(event).owner != Formatters::BaseFormatter + !listener.is_a?(Profiler) && listener.method(event).owner != base_formatter end end end diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index d886a59d2..736b82bd9 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -362,6 +362,12 @@ def reporter_with(formatter, *args) expect(Sequencer.group_aware_listeners?(reporter)).to be true end + it "is true for a custom formatter in a process that never loaded BaseFormatter" do + hide_const("RSpec::Core::Formatters::BaseFormatter") + reporter.register_listener(Class.new { def example_group_started(_); end }.new, :example_group_started) + expect(Sequencer.group_aware_listeners?(reporter)).to be true + end + it "ignores RSpec's own profiler" do reporter.register_listener(RSpec::Core::Profiler.new, :example_group_started, :example_group_finished) expect(Sequencer.group_aware_listeners?(reporter)).to be false From 7bf79685b26a8df7902213771cbfd6a9509c0692 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 19:21:06 +0200 Subject: [PATCH 19/58] [rspec-core] Say "1 second" in a parallel worker's watchdog message A worker that did not exit in time printed its timeout followed by "seconds" whatever the value ("within 1 seconds"); the coordinator's messages already pluralize. Co-Authored-By: Claude Opus 5.5 --- rspec-core/features/command_line/parallel.feature | 2 +- rspec-core/lib/rspec/core/parallel/worker.rb | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index d602fcea5..50839fdcf 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -119,7 +119,7 @@ Feature: `--parallel` option Then the exit status should be 3 And the output should contain "3 examples, 0 failures" And the output should match /Parallel worker 2 \(pid \d+\) failed while shutting down \(exit status 1\)/ - And the stderr should contain "did not exit within 1 seconds of finishing its last example group" + And the stderr should contain "did not exit within 1 second of finishing its last example group" And the stderr should contain "Threads at this point:" And the stderr should contain "join" diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index d09669b40..f799713d5 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -172,8 +172,8 @@ def arm_watchdog # @private def dump_threads(timeout) stderr = @stderr || $stderr - stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) did not exit within #{timeout} " \ - "seconds of finishing its last example group; aborting it. Threads at this point:" + stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) did not exit within " \ + "#{Parallel.seconds(timeout)} of finishing its last example group; aborting it. Threads at this point:" Thread.list.each do |thread| stderr.puts " #{thread.inspect}#{' (this watchdog)' if thread.equal?(Thread.current)}" Array(thread.backtrace).each { |line| stderr.puts " #{line}" } From aeb38483bce697fc0336e87e999a039037aed884 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 19:21:13 +0200 Subject: [PATCH 20/58] [rspec-core] Make the parallel specs pass on Ruby 3.0 and slow machines Found by running CI's steps in Linux containers (MRI 3.0-4.0, JRuby 9.4-10.1) on this branch merged into main: - A worker still loading once every group has been handed out is dismissed, so on these few-example suites one worker could take every group first. The specs that expect both workers to load, or the second to be turned away, failed on Ruby 3.0 and 3.4. They now keep groups queued until both workers have loaded, or let the second report first; the --no-parallel-preload scenario checks only what its example proves. - coordinator_spec's fake workers are threads, and Ruby 3.0 warns when a thread requires digest/sha2 while another is loading it. - A loading allowance of exactly one second reads "1 second"; two specs expected "seconds". Co-Authored-By: Claude Opus 5.5 --- .../features/command_line/parallel.feature | 4 +--- rspec-core/spec/integration/parallel_spec.rb | 21 +++++++++++++++---- .../rspec/core/parallel/coordinator_spec.rb | 7 ++++++- 3 files changed, 24 insertions(+), 8 deletions(-) diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index 50839fdcf..153db9259 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -136,10 +136,8 @@ Feature: `--parallel` option """ When I run `rspec --parallel=2 --no-parallel-preload --order defined` Then the exit status should be 1 - And the output should contain "5 examples, 1 failure" + And the output should match /^5 examples, 1 failure$/ And the file "loaded_by_the coordinating process" should exist - And the file "loaded_by_" should exist - And the file "loaded_by_2" should exist Scenario: Inside another tool's parallel worker RSpec runs serially Given I set the environment variable "TEST_ENV_NUMBER" to "2" diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 2a8abea6a..6c8e5d81d 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -664,12 +664,22 @@ def alive?(pid) end describe 'with --no-parallel-preload' do - def write_marking_spec_files + # A worker still loading once every group has been handed out is + # dismissed, so on a suite this small one worker can take every group + # before the other has loaded. With `wait_for_both_workers` each example + # first waits for both workers to have loaded the last file, which keeps + # groups queued until they have. + def write_marking_spec_files(wait_for_both_workers: false) + wait = "deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 30 + sleep 0.05 until Dir['loaded_m2_*'].size == 3 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline" 3.times do |i| write_file_formatted "spec/m#{i}_spec.rb", " File.write(\"loaded_m#{i}_\#{Process.pid}_\#{ENV.fetch('TEST_ENV_NUMBER', 'none')}\", '') RSpec.describe('M#{i}') do - it('runs in a process that loaded its file') { expect(File).to exist(\"loaded_m#{i}_\#{Process.pid}_\#{ENV['TEST_ENV_NUMBER']}\") } + it('runs in a process that loaded its file') do + #{wait if wait_for_both_workers} + expect(File).to exist(\"loaded_m#{i}_\#{Process.pid}_\#{ENV['TEST_ENV_NUMBER']}\") + end end " end @@ -694,7 +704,7 @@ def rspec_process(*args) end it 'has every worker load the spec files itself, once TEST_ENV_NUMBER is set, and run what it loaded' do - write_marking_spec_files + write_marking_spec_files(:wait_for_both_workers => true) run_rspec '--parallel=2 --no-parallel-preload' @@ -705,7 +715,10 @@ def rspec_process(*args) it 'does not use a worker that loads other examples than the coordinating process' do write_marking_spec_files + # Worker 1 loads a second late, so worker 2 reports what it loaded while + # there are still groups to hand out, rather than being dismissed unchecked. write_file_formatted 'spec/z_spec.rb', " + sleep 1 if ENV['TEST_ENV_NUMBER'] == '' RSpec.describe('Z') { it('passes') { }; it('only in the second worker') { } if ENV['TEST_ENV_NUMBER'] == '2' } " @@ -729,7 +742,7 @@ def rspec_process(*args) run_rspec '--require ./spec/spec_helper --parallel=2 --no-parallel-preload' expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 20 - expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds while loading the spec files, so it was sent SIGTERM\./) + expect(last_cmd_stdout).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds? while loading the spec files, so it was sent SIGTERM\./) expect(last_cmd_stdout).to include("3 examples, 0 failures, 1 error occurred outside of examples") end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 2d6656887..842d105e5 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -1,3 +1,8 @@ +# Loaded here: the fake workers below are threads of this process, and Ruby +# 3.0 warns ("loading in progress, circular require considered harmful") when +# a thread requires a file another thread is loading, which fails this suite. +require 'digest/sha2' + module RSpec::Core::Parallel # Stands in for `Process` so that workers can be plain threads: `kill` # ends the thread unless the "signal" is one the fake worker is said to @@ -515,7 +520,7 @@ def run_started(units, worker_count=2, options={}) run_started(groups) - expect(messages).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds while loading the spec files, so it was sent SIGTERM\./) + expect(messages).to match(/Parallel worker 2 \(pid \d+\) took longer than [\d.]+ seconds? while loading the spec files, so it was sent SIGTERM\./) expect(messages).to include("3 examples, 0 failures, 1 error occurred outside of examples") expect(process.signals.map(&:first)).to eq(["TERM"]) end From 02d4c4b02ecb7e10f05c6685ec25053300b52a34 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 19:29:00 +0200 Subject: [PATCH 21/58] [rspec-core] Stop the worker and coordinator specs leaking watchdog threads Their workers run on threads of the spec's process, and each one told to stop arms its exit watchdog, which sleeps for up to the shutdown timeout: 60 such threads were still alive after these two files (Ruby 3.4.8). A later example stubbing Parallel.sleep counted their calls too, so the sleep_running specs failed on JRuby 10.0 when worker_spec ran first (1 run in 3). Each example now kills and joins the threads it started. Co-Authored-By: Claude Opus 5.5 --- .../spec/rspec/core/parallel/coordinator_spec.rb | 10 ++++++++++ rspec-core/spec/rspec/core/parallel/worker_spec.rb | 10 ++++++++++ 2 files changed, 20 insertions(+) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 842d105e5..c36dd67e1 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -85,6 +85,16 @@ def teardown_mocks_for_rspec; end around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + # Fake workers are threads of this process, and one told to stop arms its + # exit watchdog, a thread sleeping through `Parallel.sleep_running` for up + # to the shutdown timeout: it would outlive its example and call a later + # example's stubs of `Parallel.sleep` too. + around do |example| + threads = Thread.list + example.run + (Thread.list - threads).each { |thread| thread.kill.join(1) } + end + before do config.output_stream = output config.add_formatter("progress", output) diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index e8d4c5ccd..7efc6c977 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -35,6 +35,16 @@ def frames around { |example| without_env_vars("TEST_ENV_NUMBER", &example) } + # A worker told to stop arms its exit watchdog, a thread that sleeps + # through `Parallel.sleep_running` for up to the shutdown timeout. Workers + # here are threads of this process, so it would outlive its example and + # call a later example's stubs of `Parallel.sleep` too. + around do |example| + threads = Thread.list + example.run + (Thread.list - threads).each { |thread| thread.kill.join(1) } + end + describe "#test_env_number" do it "follows the parallel_tests convention: blank for the first worker, then the worker number" do expect(worker(1).test_env_number).to eq("") From 300aa093cb07addb27f77c4da8c1a60dd30386d3 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 20:04:13 +0200 Subject: [PATCH 22/58] [rspec-core] Name the --no-parallel-preload scenario after what it shows Its example passes only in a worker that loaded the spec file itself, but a worker still loading once every group is handed out is dismissed, so the scenario cannot show that every worker loaded the files. Co-Authored-By: Claude Opus 5.5 --- rspec-core/features/command_line/parallel.feature | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index 153db9259..424761f68 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -123,7 +123,7 @@ Feature: `--parallel` option And the stderr should contain "Threads at this point:" And the stderr should contain "join" - Scenario: With `--no-parallel-preload` every worker loads the spec files itself + Scenario: With `--no-parallel-preload` workers run the groups they loaded themselves Given a file named "spec/gamma_spec.rb" with: """ruby File.write("loaded_by_#{ENV.fetch('TEST_ENV_NUMBER', 'the coordinating process')}", "") From a23add1c9e0609b5867fafcef574bbc1097cd440 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 20:41:07 +0200 Subject: [PATCH 23/58] [rspec-core] Keep --parallel runs going when a signal interrupts a read On macOS, reading a worker's pipe while that worker copies a large frame into it fails with EINTR if a signal arrives, and Ruby takes SIGCHLD without SA_RESTART, so another worker exiting was enough. The run then crashed, printed a summary without the failure it was receiving and exited 1. The read moved no bytes, so it is now retried. A 5 MB failure message under a SIGCHLD storm: 10 of 10 runs crashed before, 0 of 10 after. Co-Authored-By: Claude Opus 5.5 --- rspec-core/lib/rspec/core/parallel/channel.rb | 12 +++++++++++- .../spec/rspec/core/parallel/channel_spec.rb | 15 +++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/rspec-core/lib/rspec/core/parallel/channel.rb b/rspec-core/lib/rspec/core/parallel/channel.rb index 605740439..67ac95a50 100644 --- a/rspec-core/lib/rspec/core/parallel/channel.rb +++ b/rspec-core/lib/rspec/core/parallel/channel.rb @@ -113,10 +113,11 @@ def read_available 16.times do break if @eof - chunk = @io.read_nonblock(CHUNK_SIZE, :exception => false) + chunk = read_chunk case chunk when nil then @eof = true when :wait_readable then break + when :interrupted then next else @bytes_read += chunk.bytesize @buffer << chunk @@ -124,6 +125,15 @@ def read_available end end + # On macOS a read that waits for the pipe while the writer copies a + # large frame into it fails with EINTR when a signal arrives, such as + # the SIGCHLD of a worker exiting. Nothing was read, so it is retried. + def read_chunk + @io.read_nonblock(CHUNK_SIZE, :exception => false) + rescue Errno::EINTR + :interrupted + end + # Drops the frames already taken from the buffer: once per read # rather than once per frame, which would copy the rest of up to a # megabyte of buffered frames for each one. diff --git a/rspec-core/spec/rspec/core/parallel/channel_spec.rb b/rspec-core/spec/rspec/core/parallel/channel_spec.rb index 1aa3d5e9e..e5c44d77c 100644 --- a/rspec-core/spec/rspec/core/parallel/channel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/channel_spec.rb @@ -73,6 +73,21 @@ module RSpec::Core::Parallel expect { reader.drain }.to raise_error(Channel::ProtocolError, /Expected a frame length/) end + it "retries a read that a signal interrupted before it read anything" do + writer.send(:after_the_signal) + interrupted = false + allow(reader.io).to receive(:read_nonblock).and_wrap_original do |original, *args, **options| + unless interrupted + interrupted = true + raise Errno::EINTR + end + original.call(*args, **options) + end + + expect(reader.drain.map(&:first)).to eq([:after_the_signal]) + expect(interrupted).to be true + end + describe "#receive" do it "blocks until a whole message is available" do thread = Thread.new do From a84bd86b919ce8676c86d7fede7850d7d41f2330 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Mon, 12 Oct 2026 21:06:40 +0200 Subject: [PATCH 24/58] [rspec-core] Load the parallel runner only for runs that fork workers rspec-core loaded all 12 files of the parallel runner, which serial runs never use. Now only parallel.rb and plan.rb load with it, and the coordinator with the rest once a run is going to fork workers, as bisect does; its specs require it. Serial `rspec` with one example, n=100 interleaved medians: main 93.3 ms, all loaded 96.1 ms, now 93.1 ms. Co-Authored-By: Claude Opus 5.5 --- rspec-core/lib/rspec/core/parallel.rb | 13 ++------- .../lib/rspec/core/parallel/coordinator.rb | 10 +++++++ rspec-core/lib/rspec/core/parallel/plan.rb | 1 + rspec-core/spec/integration/parallel_spec.rb | 1 + .../spec/rspec/core/parallel/channel_spec.rb | 2 ++ .../rspec/core/parallel/coordinator_spec.rb | 2 ++ .../spec/rspec/core/parallel/plan_spec.rb | 2 ++ .../core/parallel/remote_exception_spec.rb | 2 ++ .../rspec/core/parallel/sequencer_spec.rb | 2 ++ .../rspec/core/parallel/worker_handle_spec.rb | 2 ++ .../core/parallel/worker_reporter_spec.rb | 2 ++ .../spec/rspec/core/parallel/worker_spec.rb | 2 ++ rspec-core/spec/rspec/core/parallel_spec.rb | 28 +++++++++++++++++++ rspec-core/spec/rspec/core/runner_spec.rb | 1 + 14 files changed, 60 insertions(+), 10 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index f1e29d8b4..f6de3317d 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -1,13 +1,5 @@ -RSpec::Support.require_rspec_core "parallel/channel" -RSpec::Support.require_rspec_core "parallel/remote_exception" -RSpec::Support.require_rspec_core "parallel/worker_reporter" -RSpec::Support.require_rspec_core "parallel/worker" -RSpec::Support.require_rspec_core "parallel/sequencer" -RSpec::Support.require_rspec_core "parallel/results" -RSpec::Support.require_rspec_core "parallel/suite_index" -RSpec::Support.require_rspec_core "parallel/worker_handle" -RSpec::Support.require_rspec_core "parallel/worker_lifecycle" -RSpec::Support.require_rspec_core "parallel/coordinator" +# The runner itself -- parallel/coordinator and the files it requires -- is +# loaded once a run is going to fork workers, so that serial runs skip it. RSpec::Support.require_rspec_core "parallel/plan" module RSpec @@ -39,6 +31,7 @@ def self.coordinator_for(runner, example_groups) count = [requested, units.size].min return nil if count < 2 + RSpec::Support.require_rspec_core "parallel/coordinator" Coordinator.new(runner, count).tap { |coordinator| coordinator.assign(example_groups, units) } end diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index e91b66bdc..6492f1b6e 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -1,3 +1,13 @@ +RSpec::Support.require_rspec_core "parallel/channel" +RSpec::Support.require_rspec_core "parallel/remote_exception" +RSpec::Support.require_rspec_core "parallel/worker_reporter" +RSpec::Support.require_rspec_core "parallel/worker" +RSpec::Support.require_rspec_core "parallel/sequencer" +RSpec::Support.require_rspec_core "parallel/results" +RSpec::Support.require_rspec_core "parallel/suite_index" +RSpec::Support.require_rspec_core "parallel/worker_handle" +RSpec::Support.require_rspec_core "parallel/worker_lifecycle" + module RSpec module Core module Parallel diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb index acad8230f..ff3a0cfe0 100644 --- a/rspec-core/lib/rspec/core/parallel/plan.rb +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -29,6 +29,7 @@ def before_loading count = worker_count return unless count && count >= 2 + RSpec::Support.require_rspec_core "parallel/coordinator" # Assigned first, so that workers forked before a `fork` that fails # are dismissed too. @coordinator = Coordinator.new(@runner, count) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 6c8e5d81d..7e5cafa10 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -1,3 +1,4 @@ +require 'rspec/core/parallel/coordinator' require 'support/aruba_support' require 'json' diff --git a/rspec-core/spec/rspec/core/parallel/channel_spec.rb b/rspec-core/spec/rspec/core/parallel/channel_spec.rb index e5c44d77c..dcbb3eedf 100644 --- a/rspec-core/spec/rspec/core/parallel/channel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/channel_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel RSpec.describe Channel do let(:ends) { Channel.pipe } diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index c36dd67e1..c1b844fcb 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + # Loaded here: the fake workers below are threads of this process, and Ruby # 3.0 warns ("loading in progress, circular require considered harmful") when # a thread requires a file another thread is loading, which fails this suite. diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb index 509e639bc..b2e9d0b84 100644 --- a/rspec-core/spec/rspec/core/parallel/plan_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core RSpec.describe Parallel::Plan do let(:config) { RSpec.configuration } diff --git a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb index fe06ce3f9..309c6753b 100644 --- a/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/remote_exception_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel RSpec.describe RemoteException do include FormatterSupport diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index 736b82bd9..64fedd9fe 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel class RecordingListener attr_reader :events diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb index 9087c456d..d84a1b134 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel RSpec.describe WorkerHandle do it "closes the pipe it opened when the system refuses it the second one" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index 5dae4ec16..af894ac94 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel RSpec.describe WorkerReporter do let(:results) { Channel.pipe } diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 7efc6c977..bb50be0e0 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -1,3 +1,5 @@ +require 'rspec/core/parallel/coordinator' + module RSpec::Core::Parallel RSpec.describe Worker do let(:jobs) { Channel.pipe } diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index f5c4e2b5f..1a8d955d4 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -1,3 +1,6 @@ +require 'rspec/core/parallel/coordinator' +require 'rspec/support/spec/shell_out' + module RSpec::Core RSpec.describe Parallel do let(:config) { RSpec.configuration } @@ -293,5 +296,30 @@ def groups(count) expect(trap("HUP", "IGNORE")).to eq("DEFAULT") end end + + describe "loading" do + include RSpec::Support::ShellOut + + def run_in_new_process(args) + code = <<-RUBY + require "rspec/core" + require "stringio" + 2.times { |i| RSpec.describe("G\#{i}") { it("passes") {} } } + status = RSpec::Core::Runner.run(["--options", File::NULL, *#{args.inspect}], $stderr, StringIO.new) + print [status, defined?(RSpec::Core::Parallel::Coordinator)].inspect + RUBY + without_env_vars("SPEC_OPTS") { run_ruby_with_current_load_path(code) } + end + + it "leaves the runner unloaded in a serial run", :slow do + out, err, _status = run_in_new_process([]) + expect([out, err]).to eq(["[0, nil]", ""]) + end + + it "loads the runner for a run that forks workers", :slow, :skip => !RSpec::Support::RubyFeatures.fork_supported? do + out, err, _status = run_in_new_process(%w[--parallel=2]) + expect([out, err]).to eq(['[0, "constant"]', ""]) + end + end end end diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index 40cacdebd..ea1b0b805 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -1,4 +1,5 @@ require 'rspec/core/drb' +require 'rspec/core/parallel/coordinator' require 'support/runner_support' module RSpec::Core From 0aef049151db4a0a514a5df6adb32ba5518fb441 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 16:48:13 +0200 Subject: [PATCH 25/58] [rspec-core] Keep the --parallel option parser and specs merge-friendly Moves `parallel_workers_from` above `add_tag_filter`, so that changes to `configure_only_failures` merge without touching it, and lets the --fail-fast spec accept further segments of the summary line (counts that other changes add) while it still checks for the one failure. Co-Authored-By: Claude Opus 5.5 --- rspec-core/lib/rspec/core/option_parser.rb | 16 ++++++++-------- rspec-core/spec/integration/parallel_spec.rb | 3 ++- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/rspec-core/lib/rspec/core/option_parser.rb b/rspec-core/lib/rspec/core/option_parser.rb index 2cf9ccd1e..2667e852c 100644 --- a/rspec-core/lib/rspec/core/option_parser.rb +++ b/rspec-core/lib/rspec/core/option_parser.rb @@ -308,14 +308,6 @@ def parser(options) # rubocop:enable Metrics/CyclomaticComplexity # rubocop:enable Metrics/PerceivedComplexity - def add_tag_filter(options, filter_type, tag_name, value=true) - (options[filter_type] ||= {})[tag_name] = value - end - - def set_fail_fast(options, value) - options[:fail_fast] = value - end - def parallel_workers_from(argument) case argument when true then :auto @@ -327,6 +319,14 @@ def parallel_workers_from(argument) end end + def add_tag_filter(options, filter_type, tag_name, value=true) + (options[filter_type] ||= {})[tag_name] = value + end + + def set_fail_fast(options, value) + options[:fail_fast] = value + end + def configure_only_failures(options) options[:only_failures] = true add_tag_filter(options, :inclusion_filter, :last_run_status, 'failed') diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 7e5cafa10..cc7ef9974 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -318,7 +318,8 @@ def message(notification) run_rspec "--order defined --fail-fast --parallel=2" expect(last_cmd_exit_status).to eq(1) - expect(last_cmd_stdout).to match(/^\d+ examples?, 1 failure$/) + # Later segments of the summary line (pending, errors, ...) are not this spec's business. + expect(last_cmd_stdout).to match(/^\d+ examples?, 1 failure(, .+)?$/) expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 12 end From e643f1c380fc178e793f9446576de1fdf486a04d Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 16:49:47 +0200 Subject: [PATCH 26/58] [rspec-core] Handle --parallel workers that exit or crash mid-group A worker whose code under test called `exit` skipped arming its watchdog and got no deadline from the coordinator, so a stuck `after(:suite)` hook hung the run (a repro still ran after 75 s; now it ends in 2.9 s). On every way out of its job loop a worker now arms the watchdog and tells the coordinator before any `after(:suite)` hook runs. `exit` in a worker ends the run with its status, as it does serially, instead of failing the example. A crashed worker's example still fails and is not run again; the examples of its group it had not started run in another worker, and a crash in a context hook fails that group's unstarted examples as a raising hook would. Workers that run out of work wait while others are busy, so that one is there to take them. Co-Authored-By: Claude Opus 5.5 --- rspec-core/Changelog.md | 7 +- rspec-core/README.md | 45 +++- .../features/command_line/parallel.feature | 36 +++ rspec-core/lib/rspec/core/configuration.rb | 6 +- rspec-core/lib/rspec/core/parallel.rb | 37 ++- .../lib/rspec/core/parallel/coordinator.rb | 118 +++------ rspec-core/lib/rspec/core/parallel/worker.rb | 99 +++++++- .../lib/rspec/core/parallel/worker_exits.rb | 231 ++++++++++++++++++ .../lib/rspec/core/parallel/worker_handle.rb | 23 +- rspec-core/spec/integration/parallel_spec.rb | 165 +++++++++++-- .../rspec/core/parallel/coordinator_spec.rb | 74 ++++-- .../spec/rspec/core/parallel/plan_spec.rb | 2 +- .../spec/rspec/core/parallel/worker_spec.rb | 160 +++++++++++- rspec-core/spec/rspec/core/parallel_spec.rb | 8 +- .../spec/rspec/core/suite_hooks_spec.rb | 17 ++ 15 files changed, 862 insertions(+), 166 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/worker_exits.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 4b484a3e1..13e3ff8ee 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -17,8 +17,11 @@ Enhancements: `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way parallel_tests does. A worker that finishes but does not exit is aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its - threads, so a stuck `at_exit` handler can no longer hang a run, and a worker that - crashes fails the example it was running instead of taking the run down. A run stopped + threads, so a stuck `at_exit` handler can no longer hang a run (nor can one that runs + after code under test called `exit`), and a worker that crashes fails the example it was + running instead of taking the run down, while the examples of its group it had not + started run in another worker. Code that calls `exit` ends the run with that status, as + it does serially. A run stopped with SIGTERM or SIGHUP says which example each worker was running and for how long. With `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec files are loaded and each loads them itself, so nothing the spec diff --git a/rspec-core/README.md b/rspec-core/README.md index d9a348627..178170727 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -311,13 +311,14 @@ Rails application that means `rails_helper` has already evaluated before any worker exists: setting `TEST_ENV_NUMBER` afterwards does **not** give workers separate databases, so such Rails suites run under `--parallel` share one test database until rspec-rails adds per-worker database support. -Open connections, servers and similar resources acquired at load time are -shared the same way; acquire them in `before(:suite)` to get one per worker. -In particular, a database client that connected while the spec files loaded (a -query at load time is enough) hands the same connection to every worker unless -it reconnects after `fork` by itself -- as with app servers that preload, make -each worker reconnect in a `before(:suite)` hook, e.g. -`Mongoid.disconnect_clients`. On macOS, the system networking libraries do not +Anything the spec files open while they load -- connections, servers, +threads -- is inherited by every worker; acquire such resources in +`before(:suite)` to get one per worker. A database client that connected while +the spec files loaded (a query at load time is enough) hands the same +connection to every worker unless it reconnects after `fork` by itself: as with +app servers that preload, make each worker reconnect in a `before(:suite)` +hook (with Mongoid, `Mongoid.disconnect_clients`), or have the workers load the +spec files themselves (below). On macOS, the system networking libraries do not survive `fork` well either: when loading the spec files does a lot of work (resolving host names, say), workers can crash in `getaddrinfo` or hang. @@ -368,11 +369,31 @@ non-daemon thread, a browser that will not quit, ...) is given `after(:suite)` hooks and `at_exit` handlers; after that it dumps the backtraces of all its threads to stderr and is aborted, and the run exits with `config.error_exit_code`. The coordinator never waits unboundedly for a -finished worker. A worker that crashes or is killed (`exit!`, a segfault, the -OOM killer, ...) fails the example it was running with the reason, so the -example is listed under "Failed examples" and picked up by `--only-failures`; -the rest of its example group does not run, and problems outside of examples -count as errors outside of examples. Workers also exit on their own if the +finished worker -- not even one whose watchdog cannot run and that ignores +SIGTERM: the run waits for it about the shutdown timeout plus 11 seconds at +most (a second of grace, then 5 seconds after SIGTERM and 5 after SIGKILL). + +A worker that crashes or is killed (`exit!`, a segfault, the OOM killer, ...) +fails the example it was running with the reason, so the example is listed +under "Failed examples" and picked up by `--only-failures`, and is not run +again -- whatever it did may be why the worker died; the examples of its group +it had not started yet run in another worker. One lost in a `before(:context)` +or `after(:context)` hook fails the examples of that group it had not started, +as such a hook raising an error would, and the rest of its group runs +elsewhere. Workers that run out of work wait for the busy ones, so that there +is a worker to run what a lost one leaves; if none remains, the run says which +examples did not run and how to run them. Problems between example groups count +as errors outside of examples. + +RSpec does not rescue `SystemExit`, so in a serial run code that calls `exit` +(or `abort`) ends the run with that status. Under `--parallel` it ends the run +the same way: no further examples start, busy workers stop after the example +they are running, the summary covers what ran and the run exits with the status +passed to `exit` -- unless a worker then had to be aborted or killed for not +exiting in time. Which examples ran before that depends on timing. To make such +an `exit` fail the example instead, rescue `SystemExit` in an `around` hook. + +Workers also exit on their own if the coordinating process disappears. SIGTERM or SIGHUP to the coordinating process, as `timeout` or a cancelled CI job sends, stops every worker -- even one that cannot tear down -- before the process ends the way a serial run diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index 424761f68..c330aac10 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -25,6 +25,10 @@ Feature: `--parallel` option is given work only once it has loaded the same examples as the coordinating process. + A worker that crashes fails the example it was running, and the examples of + its group it had not started run in another worker. Code that calls `exit` + ends the run with that status, as it does in a serial run. + `--parallel` needs `fork`. Where it is not available (Windows, JRuby), or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a warning and runs serially. @@ -123,6 +127,38 @@ Feature: `--parallel` option And the stderr should contain "Threads at this point:" And the stderr should contain "join" + Scenario: A worker that crashes leaves the rest of its group to another worker + Given a file named "spec/crash_spec.rb" with: + """ruby + RSpec.describe "Crashing" do + it "dies" do + exit!(7) + end + + it "runs in another worker" do + expect(5).to eq(5) + end + end + """ + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 1 + And the output should contain "6 examples, 2 failures" + And the output should contain "exited unexpectedly (exit status 7) while running this example; 1 example of Crashing" + And the output should contain "rspec ./spec/crash_spec.rb:2 # Crashing dies" + + Scenario: Code that calls `exit` ends the run with that status, as in a serial run + Given a file named "spec/exit_spec.rb" with: + """ruby + RSpec.describe "Exiting" do + it "exits" do + exit 3 + end + end + """ + When I run `rspec --parallel=2 --order defined` + Then the exit status should be 3 + And the output should contain "is ending the run: the code it ran called `exit` or `abort` (status 3)" + Scenario: With `--no-parallel-preload` workers run the groups they loaded themselves Given a file named "spec/gamma_spec.rb" with: """ruby diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index e663af919..145742650 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -2000,7 +2000,10 @@ def around(scope=nil, *meta, &block) end # @private - def with_suite_hooks + # `before_teardown` is called first once the suite is done (however it + # ends), before any `after(:suite)` hook: a parallel worker arms its + # exit watchdog there. + def with_suite_hooks(before_teardown=nil) return yield if dry_run? begin @@ -2008,6 +2011,7 @@ def with_suite_hooks run_suite_hooks("a `before(:suite)` hook", @before_suite_hooks) yield ensure + before_teardown&.call RSpec.current_scope = :after_suite_hook run_suite_hooks("an `after(:suite)` hook", @after_suite_hooks) RSpec.current_scope = :suite diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index f6de3317d..92bd23695 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -18,6 +18,22 @@ module Parallel # stop or exit in time. class WorkerError < StandardError; end + # What a worker is given to run: a top-level example group, or only the + # examples of it with these ids -- the ones a lost worker had not + # started -- which it runs as `rspec path[ids]` would: those examples, + # and the context hooks of the groups that contain them. + # @private + Unit = Struct.new(:group, :example_ids, :retried) do + # @return [Array] the ids of the examples to run + def ids + example_ids || group.descendant_filtered_examples.map(&:id) + end + + def description + "#{group.description} (#{group.location})" + end + end + # @return [Coordinator, nil] a coordinator for the loaded groups, its # workers to be forked once it runs, or `nil` when the run should # proceed serially (not requested, not possible or not worth it). @@ -79,25 +95,28 @@ def self.default_worker_count Etc.nprocessors end - # The groups to dispatch: top-level groups with at least one example to + # The units to dispatch: top-level groups with at least one example to # run, longest first when a previous run recorded their timings in the - # example status persistence file, so that the slowest group is never - # the one that starts last. Groups without timings go first because + # example status persistence file, so that the slowest unit is never + # the one that starts last. Units without timings go first because # nothing is known about them. + # @return [Array] # @private def self.units_for(example_groups, configuration) - units = example_groups.select { |group| group.descendant_filtered_examples.any? } run_times = recorded_run_times(configuration) + units = example_groups.flat_map do |group| + group.descendant_filtered_examples.empty? ? [] : [Unit.new(group)] + end return units if run_times.empty? timed = {} - units.each do |group| - ids = group.descendant_filtered_examples.map(&:id) - timed[group] = ids.sum { |id| run_times.fetch(id, 0.0) } if ids.any? { |id| run_times.key?(id) } + units.each do |unit| + ids = unit.ids + timed[unit] = ids.sum { |id| run_times.fetch(id, 0.0) } if ids.any? { |id| run_times.key?(id) } end - unknown, known = units.partition { |group| !timed.key?(group) } - unknown + known.each_with_index.sort_by { |group, index| [-timed[group], index] }.map(&:first) + unknown, known = units.partition { |unit| !timed.key?(unit) } + unknown + known.each_with_index.sort_by { |unit, index| [-timed[unit], index] }.map(&:first) end # @private diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 6492f1b6e..db4fc4de5 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -7,6 +7,7 @@ RSpec::Support.require_rspec_core "parallel/suite_index" RSpec::Support.require_rspec_core "parallel/worker_handle" RSpec::Support.require_rspec_core "parallel/worker_lifecycle" +RSpec::Support.require_rspec_core "parallel/worker_exits" module RSpec module Core @@ -22,6 +23,7 @@ module Parallel # @private class Coordinator include WorkerLifecycle + include WorkerExits TICK = 0.25 TERM_GRACE = 5.0 @@ -30,7 +32,6 @@ class Coordinator # worker's own watchdog fires first and the thread dump comes from # the process that owns the threads. WATCHDOG_GRACE = 1.0 - SENT_TERM = ', so it was sent SIGTERM' # Handled by stopping the workers -- in time, even ones that cannot # tear down -- before the coordinator goes the way a serial run would. TERMINATION_SIGNALS = %w[TERM HUP].freeze @@ -85,6 +86,7 @@ def run expect_loading_workers if loading event_loop ensure + @winding_down = true terminate_remaining_workers @sequencer.finish @results.apply_all @@ -96,6 +98,9 @@ def run # Terminated like a serial run would have been, once the workers are gone. raise SignalException, @termination_signal if @termination_signal + # Ended by `exit` in a worker, as a serial run would have been, + # unless one of them then had to be aborted. + exit(@exit_status) if @exit_status && !@exit_spoiled @runner.exit_code(passed) ensure @@ -131,6 +136,7 @@ def event_loop # A run stopped from outside -- a CI job's timeout, say -- most often # was stuck in an example, so it says what each worker was running. def stop_for_signal + @winding_down = true @workers.each { |worker| receive_from(worker) unless worker.done? } report "Received SIG#{@termination_signal}; stopping the workers." @workers.each { |worker| (running = worker.running_example_description) && report(running) } @@ -180,7 +186,7 @@ def handle_message(worker, message, bytes) kind, *rest = message number = worker.number case kind - when :group_started then index.with_group(rest[0]) { |g| group_started(number, g, rest[1], bytes) } + when :group_started then index.with_group(rest[0]) { |g| group_started(worker, g, rest[1], bytes) } when :group_finished then index.with_group(rest[0]) { |g| @sequencer.group_finished(number, g, rest[1], bytes) } when :example_started then index.with_example(rest[0]) { |e| example_started(worker, e, bytes) } when :example_finished then example_finished(worker, rest[0], rest[1], bytes) @@ -195,7 +201,7 @@ def handle_control_message(worker, kind, rest) case kind when :unit_finished then unit_finished(worker, *rest) when :non_example_failure then non_example_failure(*rest) - when :exiting then worker.exiting = true + when :exiting then exiting(worker, *rest) when :loaded then loaded(worker, *rest) else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." end @@ -206,13 +212,15 @@ def non_example_failure(payload, context_description) end # Described as the worker that runs it described it. - def group_started(number, group, descriptions, bytes) + def group_started(worker, group, descriptions, bytes) group.metadata.update(descriptions) if descriptions - @sequencer.group_started(number, group, bytes) + worker.last_started_group = group + @sequencer.group_started(worker.number, group, bytes) end def example_started(worker, example, bytes) worker.current_example = example + worker.started_example_ids[example.id] = true @sequencer.example_started(worker.number, example, bytes) end @@ -235,11 +243,11 @@ def record_result(example, payload) request_quit if fail_fast_limit_met? end - def fail_fast_limit_met? + def fail_fast_limit_met?(failed_count=@results.failed_count) limit = @configuration.fail_fast return false unless limit - limit == true ? @results.failed_count >= 1 : @results.failed_count >= limit + limit == true ? failed_count >= 1 : failed_count >= limit end def apply_result(example) @@ -262,21 +270,38 @@ def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) stop(worker) end + # A worker that runs out of work -- or is still loading -- waits while + # another one is busy: if that one is lost, what it had not started + # goes back to the queue. def dispatch_work + busy = @workers.count(&:busy?) @workers.each do |worker| if worker.loading # Nothing is left to give it once it has loaded. - dismiss_worker(worker) if worker.state == :running && (@quitting || @queue.empty?) + dismiss_worker(worker) if worker.state == :running && nothing_left?(busy) elsif worker.idle? - @quitting || @queue.empty? ? stop(worker) : dispatch(worker, @queue.shift) + if !@quitting && !@queue.empty? + dispatch(worker, @queue.shift) + busy += 1 if worker.busy? + elsif nothing_left?(busy) + stop(worker) + end end end end + # Nor will be: no busy worker is left to hand any back. + def nothing_left?(busy) + @quitting || (@queue.empty? && busy.zero?) + end + def dispatch(worker, index) worker.current_unit = index + worker.started_example_ids = {} + worker.last_started_group = nil @unit_states[index] = :running - return if write(worker, [:run, index, @units[index].id]) + unit = @units[index] + return if write(worker, [:run, index, unit.group.id, unit.example_ids]) @unit_states[index] = :pending @queue.unshift(index) @@ -333,21 +358,6 @@ def reap_workers end end - def judge_exit(worker, status) - case worker.state - when :running - # Asked to stop early, a worker may leave as soon as its group ends. - return if status&.success? && worker.exiting && worker.current_unit.nil? - - lost(worker, "exited unexpectedly (#{WorkerHandle.describe_status(status)})") - when :stopping - return if status&.success? - - error("#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ - "If its exit watchdog fired, its thread dump is on stderr above.") - end - end - def enforce_deadlines @workers.each do |worker| next if worker.done? || worker.deadline.nil? || now < worker.deadline @@ -372,52 +382,6 @@ def escalate(worker) end end - def overdue(worker) - timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) - if worker.state == :stopping - error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ - "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ - "`config.parallel_shutdown_timeout` if it legitimately needs longer.") - elsif worker.loading - lost(worker, "took longer than #{Parallel.seconds(@load_allowance)}", SENT_TERM) - elsif worker.quit_sent - lost(worker, "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)", SENT_TERM) - else - lost(worker, "stopped reporting for #{timeout}", SENT_TERM) - end - end - - def terminate(worker, reason) - lost(worker, reason, SENT_TERM) - worker.state = :terminating - worker.signal(@process, 'TERM', :term, now + @term_grace) - end - - # The worker can no longer be relied on. The example it was running, - # if any, fails with the reason -- so the summary, `Failed examples:` - # and `.rspec_status` point at it -- and otherwise the reason is - # reported as an error outside of examples. Nothing it sends from now - # on is read: it would report the example a second time. - def lost(worker, what, consequence='') - unit = worker.current_unit && unit_description(worker.current_unit) - if worker.loading - worker.loading = false - error("#{worker.description} #{what} while loading the spec files#{consequence}.") - elsif worker.current_example - example, result = worker.lost_example_result( - "#{worker.description} #{what} while running this example#{consequence}" \ - "#{"; the rest of #{unit} did not run" if unit}." - ) - record_result(example, result) - @sequencer.example_finished(worker.number, example, 0) - else - where = unit ? "while running #{unit}, outside of any example" : 'between example groups' - error("#{worker.description} #{what} #{where}#{consequence}.") - end - abort_unit(worker) - worker.results.close - end - # Counted as an error outside of examples, like a failing suite hook. def error(text) @configuration.reporter.notify_non_example_exception(WorkerError.new(text), 'An error occurred in a parallel worker.', @@ -432,16 +396,12 @@ def abort_unit(worker) @sequencer&.abandon(worker.number) end - def report_unrun_units - return if @quitting || @queue.empty? - - report "#{Formatters::Helpers.pluralize(@queue.size, 'example group')} did not run because no worker " \ - "remained to run them." + def unit_description(index) + @units[index].description end - def unit_description(index) - group = @units[index] - "#{group.description} (#{group.location})" + def pluralize(count, noun) + Formatters::Helpers.pluralize(count, noun) end def report(message) diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index f799713d5..c89223344 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -130,29 +130,101 @@ def run_units message = @jobs.receive return false unless message && message.first == :run - @configuration.with_suite_hooks do + @configuration.with_suite_hooks(method(:leaving)) do while message && message.first == :run - run_unit(message[1], message[2]) + run_unit(*message.drop(1)) message = @jobs.receive end - - # Armed before `after(:suite)` hooks so that a hook that never - # returns is bounded too. - arm_watchdog end - @results.send([:exiting]) true + rescue SystemExit => e + # Raised by an `after(:suite)` hook once it had left the loop, unless + # that was what made it leave. + unless @exit_announced + @exit_cause ||= exit_cause(e, 'an `after(:suite)` hook') + send_exiting(e) + end + raise + end + + # Arms the watchdog and tells the coordinator -- which then bounds the + # wait from its side too -- before any `after(:suite)` hook runs, + # however the worker leaves its job loop: done, told to stop, or + # because the code it ran called `exit` (not rescued, as in a serial + # run) or the process got a signal. `$!` is the exception ending the + # process, if any. + # @private + def leaving + return if @leaving + + @leaving = true + exception = $! + @exit_cause = exit_cause(exception, 'the code it ran') + arm_watchdog + send_exiting(exception) end + # `[:exiting, cause, exit_status]`: sent on leaving the job loop, and + # again for an `exit` that comes later. # @private - def run_unit(index, id) - group = @groups_by_id ? @groups_by_id.fetch(id) : @example_groups.fetch(index) - passed = group.run(@reporter) + def send_exiting(exception) + status = exception.status if exception.is_a?(SystemExit) + @exit_announced = true if status + @results.send([:exiting, @exit_cause, status]) + rescue Errno::EPIPE, IOError + nil # the coordinator is gone; the watchdog and its own watch still end this process + end + + # @return [String, nil] why the process is ending early, as the end of + # a sentence: "the code it ran called `exit` or `abort` (status 0)" + # @private + def exit_cause(exception, source) + case exception + when nil then nil + when SystemExit then "#{source} called `exit` or `abort` (status #{exception.status})" + when SignalException then "it received #{exception.signm}" + else "#{source} raised #{exception.class} (#{exception.message.to_s[0, 200]})" + end + end + + # @private + def run_unit(index, id, example_ids=nil) + group = groups_by_id.fetch(id) + passed = example_ids ? run_only(group, example_ids) : group.run(@reporter) @results.send([:unit_finished, index, passed ? true : false, !!@world.wants_to_quit, !!@world.non_example_failure]) end + # Runs the group the way a run filtered to these examples would + # (`rspec path[ids]`): only they run, and only the context hooks of + # the groups that contain them. Groups filter their examples through + # `RSpec.world`, and every group of the tree caches which of its and + # its descendants' examples run, so each one is reset, before and + # after. + # @private + def run_only(group, example_ids) + wanted = example_ids.to_h { |id| [id, true] } + filtered = RSpec.world.filtered_examples + groups = group.descendants + kept = groups.map { |each| filtered[each] } + groups.each { |each| filtered[each] = filtered[each].select { |example| wanted.key?(example.id) } } + groups.each(&:reset_memoized) + group.run(@reporter) + ensure + if kept + groups.each_with_index { |each, index| filtered[each] = kept[index] } + groups.each(&:reset_memoized) + end + end + + # Those of the units it was forked with, unless it loaded the spec + # files itself: what it is given later is part of one of them. + # @private + def groups_by_id + @groups_by_id ||= @example_groups.to_h { |unit| [unit.group.id, unit.group] } + end + # @private def arm_watchdog timeout = @configuration.parallel_shutdown_timeout @@ -167,13 +239,18 @@ def arm_watchdog end thread.report_on_exception = false thread + rescue ThreadError + nil # no thread for it; the coordinator's deadline still ends this process end # @private def dump_threads(timeout) stderr = @stderr || $stderr + stopping = @exit_cause ? "stopping because #{@exit_cause}" : 'finishing its last example group' stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) did not exit within " \ - "#{Parallel.seconds(timeout)} of finishing its last example group; aborting it. Threads at this point:" + "#{Parallel.seconds(timeout)} of #{stopping}; aborting it. Its `after(:suite)` hooks or " \ + "`at_exit` handlers may be stuck, or need a longer `config.parallel_shutdown_timeout`. " \ + "Threads at this point:" Thread.list.each do |thread| stderr.puts " #{thread.inspect}#{' (this watchdog)' if thread.equal?(Thread.current)}" Array(thread.backtrace).each { |line| stderr.puts " #{line}" } diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb new file mode 100644 index 000000000..ab58e2e3b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -0,0 +1,231 @@ +module RSpec + module Core + module Parallel + # The part of the {Coordinator} that deals with a worker that leaves + # before its work is done: lost -- it crashed, was killed or had to be + # terminated -- when the example it was running fails and what it had + # not started goes to another worker, or ending the run because the + # code it ran called `exit`, as a serial run would end. + # @private + module WorkerExits + SENT_TERM = ', so it was sent SIGTERM' + RERUN_ARGUMENTS_SHOWN = 50 + RERUN_IDS_PER_UNIT = 20 + + private + + def judge_exit(worker, status) + case worker.state + when :running + # Asked to stop early, a worker may leave as soon as its group ends. + return if status&.success? && worker.exiting && worker.current_unit.nil? + return left_by_exit(worker, status) if worker.called_exit + + lost(worker, "exited unexpectedly (#{WorkerHandle.describe_status(status)})") + when :stopping + return left_by_exit(worker, status) if worker.called_exit + return if status&.success? + + error("#{worker.description} failed while shutting down (#{WorkerHandle.describe_status(status)}). " \ + "If its exit watchdog fired, its thread dump is on stderr above.") + end + end + + def overdue(worker) + timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) + if worker.called_exit + @exit_spoiled = true + error("#{worker.description} did not exit within #{timeout} of ending the run because " \ + "#{worker.exit_cause} (its `after(:suite)` hooks or `at_exit` handlers may be stuck), so it was " \ + "sent SIGTERM. Raise `config.parallel_shutdown_timeout` if they legitimately need longer.") + abort_unit(worker) + worker.results.close + elsif worker.state == :stopping + error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ + "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ + "`config.parallel_shutdown_timeout` if it legitimately needs longer.") + elsif worker.loading + lost(worker, "took longer than #{Parallel.seconds(@load_allowance)}", SENT_TERM) + elsif worker.quit_sent + lost(worker, "did not stop within #{timeout} of being asked to stop early (fail-fast or interrupt)", SENT_TERM) + elsif worker.exiting + lost(worker, "did not exit within #{timeout} of leaving its job loop (its `after(:suite)` hooks or " \ + "`at_exit` handlers may be stuck; raise `config.parallel_shutdown_timeout` if they " \ + "legitimately need longer)", SENT_TERM) + else + lost(worker, "stopped reporting for #{timeout}", SENT_TERM) + end + end + + def terminate(worker, reason) + lost(worker, reason, SENT_TERM) + worker.state = :terminating + worker.signal(@process, 'TERM', :term, now + @term_grace) + end + + # The worker has left its job loop and runs its `after(:suite)` hooks + # and `at_exit` handlers: as long as that may take is bounded here + # too, also for a worker that left in the middle of a group -- from + # now, so that its own watchdog, which it armed on leaving, fires + # first, also when it was asked to stop earlier. + def exiting(worker, cause, exit_status=nil) + worker.exiting = true + worker.exit_cause ||= cause + teardown = now + @configuration.parallel_shutdown_timeout + @watchdog_grace + worker.deadline = teardown unless worker.escalation || (worker.deadline && worker.deadline >= teardown) + ended_by_exit(worker, exit_status) if exit_status + end + + # RSpec does not rescue `SystemExit`, so in a serial run `exit` ends + # the run: here too no example starts from now on, busy workers stop + # after the one they are running and the run exits with that status + # -- the first one, if more workers call `exit`. + def ended_by_exit(worker, exit_status) + worker.called_exit = exit_status + # Leaving anyway: it is not asked to stop. + worker.quit_sent = true + return if @exit_status + + @exit_status = exit_status + running = worker.current_example + report "#{worker.description} is ending the run: #{worker.exit_cause}" \ + "#{" in #{running.full_description} (#{running.location})" if running}. As in a serial run, no " \ + "further examples start and the run exits with status #{exit_status}." + request_quit + end + + # The worker can no longer be relied on, and nothing it sends from now + # on is read: it would report the example a second time. What it was + # running is reported as lost -- in a way that the summary, `Failed + # examples:` and `.rspec_status` point at -- and what it had not + # started of its unit goes to another worker. + def lost(worker, what, consequence='') + because = "#{consequence}#{worker.exit_cause && ": #{worker.exit_cause}"}" + if worker.loading + worker.loading = false + error("#{worker.description} #{what} while loading the spec files#{consequence}.") + elsif worker.current_example + lost_in_example(worker, "#{worker.description} #{what} while running this example#{because}") + elsif worker.current_unit + lost_in_unit(worker, "#{worker.description} #{what}", because) + else + error("#{worker.description} #{what} between example groups#{because}.") + end + abort_unit(worker) + worker.results.close + end + + # Its example fails and never runs again -- what it did may be why the + # worker was lost. + def lost_in_example(worker, reason) + unit = @units[worker.current_unit] + rest = unit.ids.reject { |id| worker.started_example_ids.key?(id) } + requeue = !rest.empty? && accepting_work? && !fail_fast_limit_met?(@results.failed_count + 1) + example, result = worker.lost_example_result("#{reason}#{rest_clause(unit, rest, requeue)}.") + record_result(example, result) + @sequencer.example_finished(worker.number, example, 0) + queue_first(Unit.new(unit.group, rest)) if requeue + end + + # Lost in the `before(:context)` or `after(:context)` hooks of a group + # or between its examples: the examples of the group it started last + # that had not started fail, as they do when such a hook raises, and + # the rest goes to another worker. (Its groups may have reported + # finishing as the process unwound.) Lost before it even started the + # unit's group, it is given one more go. + def lost_in_unit(worker, who, because) + unit = @units[worker.current_unit] + unstarted = unit.ids.reject { |id| worker.started_example_ids.key?(id) } + inside = worker.last_started_group + if inside + within = inside.descendant_filtered_examples.each_with_object({}) { |example, ids| ids[example.id] = true } + failing, rest = unstarted.partition { |id| within.key?(id) } + where = "while running #{inside.metadata[:full_description]} (#{inside.location}), outside of any example" + else + retry_it = worker.started_example_ids.empty? && !unit.retried + failing, rest = retry_it ? [[], unstarted] : [unstarted, []] + where = "before it started #{unit.description}" + end + + requeue = !rest.empty? && accepting_work? + if failing.empty? + error("#{who} #{where}#{because}#{rest_clause(unit, rest, requeue)}.") + else + fail_unstarted(worker, failing, "#{who} #{where}#{because}, so this example could not run" \ + "#{rest_clause(unit, rest, requeue)}.") + end + queue_first(Unit.new(unit.group, rest, !inside)) if requeue + end + + def fail_unstarted(worker, ids, reason) + ids.each do |id| + index.with_example(id) do |example| + @sequencer.example_started(worker.number, example, 0) + record_result(example, WorkerHandle.failed_result(example, reason)) + @sequencer.example_finished(worker.number, example, 0) + end + end + end + + def rest_clause(unit, rest, requeued) + return "" if rest.empty? + + examples = pluralize(rest.size, 'example') + if requeued + "; #{examples} of #{unit.description} that it had not started yet will run in another worker" + else + "; #{examples} of #{unit.description} that it had not started did not run" + end + end + + def accepting_work? + !@quitting && !@winding_down + end + + def queue_first(unit) + @units << unit + @unit_states << :pending + @queue.unshift(@units.size - 1) + end + + # A worker that called `exit` left the way a serial run does: its + # example is neither passed nor failed. One that then exited + # otherwise was aborted by its watchdog or killed. + def left_by_exit(worker, status) + unless status.nil? || (status.exited? && status.exitstatus == worker.called_exit) + @exit_spoiled = true + ended = "#{status.signaled? ? 'was' : 'exited with'} #{WorkerHandle.describe_status(status)}" + error("#{worker.description} ended the run because #{worker.exit_cause}, then #{ended}: if its " \ + "watchdog aborted it, its `after(:suite)` hooks or " \ + "`at_exit` handlers did not finish within #{Parallel.seconds(@configuration.parallel_shutdown_timeout)} " \ + "(its thread dump is on stderr above).") + end + abort_unit(worker) + worker.results.close + end + + def report_unrun_units + return if @quitting || @queue.empty? + + units = @queue.map { |index| @units[index] } + examples = units.sum { |unit| unit.ids.size } + groups = units.map(&:group).uniq.size + report "#{pluralize(examples, 'example')} of #{pluralize(groups, 'example group')} did not run because no " \ + "worker remained to run them. To run them: rspec #{rerun_arguments(units)}" + end + + # Quoted as RSpec quotes the examples to rerun after a `--bisect`; a + # long list of examples of one group is given as the whole group. + def rerun_arguments(units) + ids = units.flat_map do |unit| + unit.example_ids && unit.example_ids.size <= RERUN_IDS_PER_UNIT ? unit.example_ids : [unit.group.id] + end + arguments = Formatters::Helpers.organize_ids(ids) + hidden = arguments.size - RERUN_ARGUMENTS_SHOWN + shown = arguments.first(RERUN_ARGUMENTS_SHOWN).join(' ') + hidden > 0 ? "#{shown} (and #{pluralize(hidden, 'more spec file')})" : shown + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index c12287347..decad33bf 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -5,7 +5,10 @@ module Parallel # @private class WorkerHandle attr_reader :number, :jobs, :results, :child_jobs, :child_results, :current_example, :current_example_started_at - attr_accessor :pid, :current_unit, :state, :escalation, :exit_status, :quit_sent, :exiting + attr_accessor :pid, :current_unit, :state, :escalation, :exit_status, :quit_sent, :exiting, :exit_cause + # The status it was asked to `exit` with; the examples and the last + # group it started of its current unit. + attr_accessor :called_exit, :started_example_ids, :last_started_group # When the deadline was set: only a pause after that postpones it. attr_reader :deadline, :deadline_set_at # Forked before the spec files were loaded and has not yet reported @@ -25,7 +28,9 @@ def initialize(number) @quit_sent = false @exiting = false @loading = false - @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = nil + @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = @exit_cause = @called_exit = nil + @started_example_ids = {} + @last_started_group = nil @current_example = @current_example_started_at = nil end @@ -58,14 +63,18 @@ def running_example_description # when it was lost, and a result failing it with `text` def lost_example_result(text) example = @current_example + result = self.class.failed_result(example, text, @current_example_started_at) + self.current_example = nil + [example, result] + end + + # @return [Hash] a result failing `example` with a {WorkerError} + def self.failed_result(example, text, started_at=RSpec::Core::Time.now) exception = WorkerError.new(text) exception.set_backtrace([example.location]) finished_at = RSpec::Core::Time.now - result = { :status => :failed, :started_at => @current_example_started_at, :finished_at => finished_at, - :run_time => finished_at - @current_example_started_at, - :exception => RemoteException.payload_for(exception) } - self.current_example = nil - [example, result] + { :status => :failed, :started_at => started_at, :finished_at => finished_at, + :run_time => finished_at - started_at, :exception => RemoteException.payload_for(exception) } end # @return [Integer] the pid of the worker process forked to run `units`, diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index cc7ef9974..3ebc1b911 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -421,17 +421,19 @@ def write_self_interrupting_group(name, examples_after=2) expect(last_cmd_exit_status).to eq(1) expect(last_cmd_stdout).to include("2 examples, 0 failures") - expect(last_cmd_stdout).to include("2 example groups did not run because no worker remained to run them") + expect(last_cmd_stdout).to match( + /^2 examples of 2 example groups did not run because no worker remained to run them\. To run them: rspec '?\.\/spec\/\w+_spec\.rb\[1\]'? '?\.\/spec\/\w+_spec\.rb\[1\]'?$/ + ) end end - it 'fails the example a worker dies in, with the reason, so it can be rerun' do + it 'fails the example a worker dies in, with the reason, so it can be rerun, and runs the rest of its group in another worker' do write_file_formatted 'spec/crash_spec.rb', " RSpec.configure { |c| c.example_status_persistence_file_path = 'status.txt' } RSpec.describe 'Crashing' do it('finishes') { } - it('dies') { exit!(7) } - it('never runs') { } + it('dies') { File.write(\"dies_\#{Process.pid}\", ''); exit!(7) } + it('runs in another worker') { File.write(\"rest_\#{Process.pid}\", '') } end RSpec.describe('Fine') { it('passes') { } } " @@ -440,28 +442,163 @@ def write_self_interrupting_group(name, examples_after=2) expect(last_cmd_exit_status).to eq(1) expect(last_cmd_stdout).to match( - /Failure\/Error: it\('dies'\) \{ exit!\(7\) \}\n\s+Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running this example; the rest of Crashing \(\.\/spec\/crash_spec\.rb:2\) did not run\./ + /Failure\/Error: it\('dies'\) \{ File\.write.*exit!\(7\) \}\n\s+Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running this example; 1 example of Crashing \(\.\/spec\/crash_spec\.rb:2\) that it had not started yet will run in another worker\./ ) - expect(last_cmd_stdout).to include("3 examples, 1 failure", "rspec ./spec/crash_spec.rb:4 # Crashing dies") - cd('.') { expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:2\] \| failed }) } + expect(last_cmd_stdout).to include("4 examples, 1 failure", "rspec ./spec/crash_spec.rb:4 # Crashing dies") + cd('.') do + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:2\] \| failed }) + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:3\] \| passed }) + expect(Dir['dies_*'].size).to eq(1) + expect(Dir['rest_*'].map { |file| file[/\d+/] }).not_to include(Dir['dies_*'].first[/\d+/]) + end end - it 'counts a worker that dies outside of any example as an error and fails the run with the error exit code' do + it 'fails the examples of a group whose context hook a worker dies in, as a raising hook would, and runs the rest elsewhere' do write_file_formatted 'spec/crash_spec.rb', " + RSpec.configure { |c| c.example_status_persistence_file_path = 'status.txt' } RSpec.describe 'Crashing' do - before(:context) { exit!(7) } - it('never runs') { } + context 'broken' do + before(:context) { exit!(7) } + it('cannot run') { } + end + context 'fine' do + it('runs in another worker') { } + end end - RSpec.describe('Fine') { it('passes') { } } + RSpec.describe('Fine') { it('passes') { sleep 0.5 } } " run_rspec "--order defined --parallel=2 --error-exit-code 3 spec/crash_spec.rb" - expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_exit_status).to eq(1) expect(last_cmd_stdout).to match( - /Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running Crashing \(\.\/spec\/crash_spec\.rb:1\), outside of any example\./ + /Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 7\) while running Crashing broken \(\.\/spec\/crash_spec\.rb:3\), outside of any example, so this example could not run; 1 example of Crashing \(\.\/spec\/crash_spec\.rb:2\) that it had not started yet will run in another worker\./ ) - expect(last_cmd_stdout).to include("1 example, 0 failures, 1 error occurred outside of examples") + expect(last_cmd_stdout).to include("3 examples, 1 failure", "rspec ./spec/crash_spec.rb:5 # Crashing broken cannot run") + expect(last_cmd_stdout).not_to include("outside of examples") + cd('.') do + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:1:1\] \| failed }) + expect(File.read('status.txt')).to match(%r{^\./spec/crash_spec\.rb\[1:2:1\] \| passed }) + end + end + + describe 'when the code an example runs calls `exit`' do + # A run that a worker's `exit` ends raises `SystemExit` out of + # `Runner.run`, as a serial run does -- which would end this very run. + def run_rspec(cmd) + super + rescue SystemExit => e + @last_cmd_exit_status = e.status + end + + def write_exiting_suite(status, after_suite="") + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.after(:suite) { #{after_suite} } + end + RSpec.describe 'Exits' do + it('exits') do + sleep 0.2 + $exiting = true + exit #{status} + end + it('never runs') { File.write('ran_after_exit', '') } + end + RSpec.describe 'Keeps going' do + 20.times { |i| it(\"waits \#{i}\") { sleep 0.05 } } + end + " + end + + [0, 3].each do |status| + it "ends the run as a serial run does, with status #{status}: no further examples start" do + write_exiting_suite(status) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(status) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\) is ending the run: the code it ran called `exit` or `abort` \(status #{status}\) in Exits exits \(\.\/spec\/exit_spec\.rb:6\)\. As in a serial run, no further examples start and the run exits with status #{status}\.$/ + ) + expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 22 + expect(last_cmd_stdout).to match(/^\d+ examples?, 0 failures$/) + cd('.') { expect(File).not_to exist('ran_after_exit') } + end + end + + it 'is bounded when the worker then does not exit, failing the run' do + # Leaves the worker's watchdog plenty of room to fire before the + # coordinator steps in, even on a loaded machine. + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_exiting_suite(0, "if $exiting; $stderr = File.open('worker_stderr.txt', 'w'); sleep 30; end") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2 --error-exit-code 3" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\) ended the run because the code it ran called `exit` or `abort` \(status 0\), then exited with exit status 1: if its watchdog aborted it, its `after\(:suite\)` hooks or `at_exit` handlers did not finish within 0\.5 seconds/ + ) + expect(cd('.') { File.read('worker_stderr.txt') }).to include( + "did not exit within 0.5 seconds of stopping because the code it ran called `exit` or `abort` (status 0); aborting it." + ) + end + + it 'is bounded by the coordinator when the worker cannot abort itself and ignores SIGTERM' do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 0.1) + # Inherited by the forked workers: a watchdog that never gets to run. + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:arm_watchdog) + write_exiting_suite(0, "if $exiting; trap('TERM') { }; File.write('stuck.pid', Process.pid.to_s); sleep 30; end") + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2 --error-exit-code 3" + elapsed = Process.clock_gettime(Process::CLOCK_MONOTONIC) - started + + # The shutdown timeout, the watchdog grace and both signal graces, with room for a loaded machine. + expect(elapsed).to be < 0.5 + 0.1 + 0.3 + 0.3 + 5 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include( + "did not exit within 0.5 seconds of ending the run because the code it ran called `exit` or `abort` (status 0)", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + stuck = cd('.') { Integer(File.read('stuck.pid')) } + expect { Process.kill(0, stuck) }.to raise_error(Errno::ESRCH) + end + + it 'ends the run with that status from a before(:suite) hook too, bounded when the worker then hangs' do + stub_const("RSpec::Core::Parallel::Coordinator::WATCHDOG_GRACE", 5.0) + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure do |c| + c.parallel_shutdown_timeout = 0.5 + c.before(:suite) do + c.after(:suite) do + $stderr = File.open(\"worker_stderr_\#{Process.pid}.txt\", 'w') + sleep 30 + end + exit 4 + end + end + RSpec.describe('One') { it('never runs') { File.write('ran', '') } } + RSpec.describe('Two') { it('never runs') { File.write('ran', '') } } + " + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --error-exit-code 3" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_exit_status).to eq(3) + expect(last_cmd_stdout).to include("is ending the run: the code it ran called `exit` or `abort` (status 4)") + dumps = cd('.') { Dir['worker_stderr_*.txt'].map { |file| File.read(file) } } + expect(dumps).not_to be_empty + expect(dumps).to all(include("of stopping because the code it ran called `exit` or `abort` (status 4); aborting it.")) + cd('.') { expect(File).not_to exist('ran') } + end end it 'stops its workers, even ones that cannot tear down, when it and its process group get SIGTERM, saying what they ran' do diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index c1b844fcb..cf9bff5d0 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -119,6 +119,7 @@ def spawn(handle, _handles) when :garbage then stall_after { results.io.write("garbage\n") } when :partial_frame then stall_after { results.io.write("999\nabc") } when :closes_results then stall_after { results.close } + when :leaves_and_hangs then stall_after { leave_for_a_signal(jobs, results) } when :never_loads then sleep when :dies_loading then Thread.current.kill else @@ -140,6 +141,13 @@ def stall_after sleep end + # Takes its first job, then says it is leaving its job loop because of a + # signal -- and never does. + def leave_for_a_signal(jobs, results) + jobs.receive + results.send([:exiting, "it received SIGHUP", nil]) + end + # What a worker forked before loading reports once it has loaded the spec # files: by default, the examples this process has. def report_loaded(results, behaviour) @@ -161,7 +169,7 @@ def run_fake_worker(number, jobs, results, behaviour, &record_exit) worker_config = RSpec::Core::Configuration.new worker_config.parallel_shutdown_timeout = 60 Thread.current[:fake_worker_world] = RSpec::Core::World.new(worker_config) - Worker.new(number, @units, jobs, results, + Worker.new(number, units_of(@units), jobs, results, :configuration => worker_config, :world => Thread.current[:fake_worker_world], :stderr => worker_stderr, @@ -185,8 +193,12 @@ def new_coordinator(units, worker_count, options) :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) end - def coordinator(units, worker_count=2, options={}) - new_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units) } + def coordinator(groups, worker_count=2, options={}) + new_coordinator(groups, worker_count, options).tap { |instance| instance.assign(groups, units_of(groups)) } + end + + def units_of(groups) + groups.map { |group| Unit.new(group) } end # Started the way `--no-parallel-preload` starts it: the workers are forked @@ -311,13 +323,34 @@ def example_failed(_notification) end end + it "bounds the wait for a worker that left its job loop for a signal and does not exit, and runs its group elsewhere" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = :leaves_and_hangs + groups = [passing_group("A"), passing_group("B")] + + exit_code = coordinator(groups, 2, short_graces).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to include("TERM") + expect(messages).to include( + "did not exit within 0.05 seconds of leaving its job loop (its `after(:suite)` hooks or `at_exit` handlers may " \ + "be stuck; raise `config.parallel_shutdown_timeout` if they legitimately need longer) before it started A", + ": it received SIGHUP; 1 example of A", + "2 examples, 0 failures, 1 error occurred outside of examples" + ) + end + describe "a worker that exits while running a group" do - it "fails the example it was running with the reason and leaves the rest of the group unknown" do + it "fails the example it was running with the reason, never runs it again, and runs the rest of the group in another worker" do config.error_exit_code = 5 + runs = Hash.new(0) crashing = RSpec.describe("Crashing") do - it("finishes") {} - it("dies") { Thread.current.kill } - it("never runs") {} + it("finishes") { runs[:finishes] += 1 } + it("dies") do + runs[:dies] += 1 + Thread.current.kill + end + it("runs elsewhere") { runs[:elsewhere] += 1 } end groups = [crashing, passing_group("B")] config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) @@ -326,33 +359,36 @@ def example_failed(_notification) expect(exit_code).to eq(config.failure_exit_code) dies = crashing.examples[1] - expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, :failed, nil]) + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:passed, :failed, :passed]) + expect(runs).to eq(:finishes => 1, :dies => 1, :elsewhere => 1) expect(dies.exception).to be_a(WorkerError) expect(dies.exception.message).to match( - /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running this example; the rest of Crashing \(.+\) did not run\.\z/ + /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running this example; 1 example of Crashing \(.+\) that it had not started yet will run in another worker\.\z/ ) expect(dies.exception.backtrace).to eq([dies.location]) expect(dies.execution_result.run_time).to be >= 0 - expect(messages).to include("Crashing dies", "3 examples, 1 failure", "rspec #{dies.location_rerun_argument}") + expect(messages).to include("Crashing dies", "4 examples, 1 failure", "rspec #{dies.location_rerun_argument}") end - it "reports an error outside of examples when it was not running one" do + it "fails the unstarted examples of the group it was in when it was not running one, as a failing context hook does" do config.error_exit_code = 5 crashing = RSpec.describe("Crashing") do before(:context) { Thread.current.kill } - it("never runs") {} + it("cannot run") {} + it("cannot run either") {} end groups = [crashing, passing_group("B")] config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) exit_code = coordinator(groups).run - expect(exit_code).to eq(5) - expect(crashing.examples.first.execution_result.status).to be_nil - expect(messages).to match( - /An error occurred in a parallel worker\.\nParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing \(.+\), outside of any example\./ + expect(exit_code).to eq(config.failure_exit_code) + expect(crashing.examples.map { |e| e.execution_result.status }).to eq([:failed, :failed]) + expect(crashing.examples.first.exception.message).to match( + /\AParallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) while running Crashing \(.+\), outside of any example, so this example could not run\.\z/ ) - expect(messages).to include("1 example, 0 failures, 1 error occurred outside of examples") + expect(messages).to include("3 examples, 2 failures", "rspec #{crashing.examples.first.location_rerun_argument}") + expect(messages).not_to include("outside of examples") end end @@ -447,7 +483,7 @@ def failure_once_running(name, &body) expect(exit_code).to eq(config.failure_exit_code) expect(process.signals.map(&:first)).to eq(["TERM"]) expect(stuck.examples.first.exception.message).to match( - /\AParallel worker 2 \(pid \d+\) did not stop within 0\.5 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM; the rest of Stuck \(.+\) did not run\.\z/ + /\AParallel worker 2 \(pid \d+\) did not stop within 0\.5 seconds of being asked to stop early \(fail-fast or interrupt\) while running this example, so it was sent SIGTERM\.\z/ ) expect(messages).to include("2 examples, 2 failures") end @@ -467,7 +503,7 @@ def failure_once_running(name, &body) describe "with workers forked before the spec files are loaded" do def run_started(units, worker_count=2, options={}) - started_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units) }.run + started_coordinator(units, worker_count, options).tap { |instance| instance.assign(units, units_of(units)) }.run end it "gives work to the workers that report having loaded the examples this process has" do diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb index b2e9d0b84..f1047af93 100644 --- a/rspec-core/spec/rspec/core/parallel/plan_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -41,7 +41,7 @@ def spec_files(count) all = groups(1).tap { |list| list.first.describe("nested") { it("e") {} } } + groups(2) expect(plan.coordinator_for(all)).to be(started) - expect(started).to have_received(:assign).with(all, all) + expect(started).to have_received(:assign).with(all, all.map { |group| Parallel::Unit.new(group) }) end it "forks no more workers than there are spec files to load" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index bb50be0e0..ea13b9ada 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -18,7 +18,7 @@ module RSpec::Core::Parallel let(:traps) { {} } def worker(number=2, options={}) - Worker.new(number, groups, jobs.first, results.last, + Worker.new(number, groups.map { |group| Unit.new(group) }, jobs.first, results.last, { :stderr => stderr, :exit => lambda { |status| exits << [:exit, status] }, :exit_now => lambda { |status| exits << [:exit_now, status] }, @@ -147,8 +147,8 @@ def frames RSpec.configuration.after(:suite) { order << :after_suite } groups.first.before(:context) { order << :group } - jobs.last.send([:run, 0]) - jobs.last.send([:run, 1]) + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:run, 1, groups[1].id]) jobs.last.send([:stop]) worker.run @@ -161,7 +161,7 @@ def frames [:unit_finished, 0, true, false, false], [:unit_finished, 1, false, false, false] ]) - expect(all.last).to eq([:exiting]) + expect(all.last).to eq([:exiting, nil, nil]) expect(all.map(&:first)).to include(:group_started, :example_started, :example_finished, :group_finished) end @@ -191,7 +191,7 @@ def frames it "reports a quitting world and non-example failures with the unit" do RSpec.configuration.before(:suite) { raise "suite setup failed" } - jobs.last.send([:run, 0]) + jobs.last.send([:run, 0, groups[0].id]) jobs.last.send([:stop]) worker.run @@ -202,7 +202,7 @@ def frames it "exits immediately without reporting when the results pipe is gone" do results.first.close - jobs.last.send([:run, 0]) + jobs.last.send([:run, 0, groups[0].id]) jobs.last.send([:stop]) worker.run @@ -215,7 +215,7 @@ def frames instance = worker RSpec.configuration.after(:suite) { armed_during_after_suite = instance.instance_variable_get(:@watchdog_armed) } allow(instance).to receive(:arm_watchdog) { instance.instance_variable_set(:@watchdog_armed, true) } - jobs.last.send([:run, 0]) + jobs.last.send([:run, 0, groups[0].id]) jobs.last.send([:stop]) instance.run @@ -224,6 +224,152 @@ def frames end end + describe "leaving its job loop" do + def leaving_frames + frames.select { |frame| frame.first == :exiting } + end + + it "arms the watchdog and says so before any after(:suite) hook, even one a before(:suite) hook added before calling exit" do + instance = worker + armed = [] + allow(instance).to receive(:arm_watchdog) { armed << :watchdog } + config = RSpec.configuration + config.before(:suite) do + config.after(:suite) { armed << :late_after_suite } + exit 4 + end + jobs.last.send([:run, 0, groups[0].id]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(4) } + + expect(armed).to eq([:watchdog, :late_after_suite]) + expect(leaving_frames).to eq([[:exiting, "the code it ran called `exit` or `abort` (status 4)", 4]]) + end + + it "says which status the code it ran called `exit` with in an example, and runs nothing after it" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + ran = [] + exiting = RSpec.describe("Exits") do + it("exits") { exit 3 } + it("never runs") { ran << :after } + end + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + jobs.last.send([:run, 0, exiting.id]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(3) } + + expect(ran).to eq([]) + expect(leaving_frames).to eq([[:exiting, "the code it ran called `exit` or `abort` (status 3)", 3]]) + expect(instance).to have_received(:arm_watchdog).once + end + + it "says so again, with the status, when an after(:suite) hook calls exit" do + instance = worker + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { exit 5 } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(5) } + + expect(leaving_frames).to eq([[:exiting, nil, nil], [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5]]) + end + + it "names a signal or another exception that ends the process" do + expect(worker.exit_cause(SignalException.new("TERM"), "the code it ran")).to eq("it received SIGTERM") + expect(worker.exit_cause(NoMemoryError.new("too much"), "the code it ran")).to eq( + "the code it ran raised NoMemoryError (too much)" + ) + end + + it "goes on without a watchdog when no thread can be started for it" do + instance = worker + allow(Thread).to receive(:new).and_raise(ThreadError, "can't create Thread: Resource temporarily unavailable") + + expect(instance.arm_watchdog).to be_nil + end + + it "names why it is stopping in its thread dump" do + RSpec.configuration.parallel_shutdown_timeout = 0.05 + instance = worker + instance.instance_variable_set(:@exit_cause, "the code it ran called `exit` or `abort` (status 0)") + + instance.dump_threads(0.05) + + expect(stderr.string).to include( + "did not exit within 0.05 seconds of stopping because the code it ran called `exit` or `abort` (status 0); aborting it." + ) + end + end + + describe "given a unit of some examples of a group" do + let(:hooks) { [] } + let(:group) do + log = hooks + RSpec.describe("Outer") do + before(:context) { log << :outer } + it("o1") { log << :o1 } + context("A") do + before(:context) { log << :a } + it("a1") { log << :a1 } + it("a2") { log << :a2 } + end + context("B") do + before(:context) { log << :b } + it("b1") { log << :b1 } + it("b2") do |example| + log << :b2 + raise "b2 fails" if example.metadata[:fail] + end + end + end + end + + def ids(*descriptions) + group.descendants.flat_map(&:examples).select { |example| descriptions.include?(example.description) }.map(&:id) + end + + def unit_worker + Worker.new(2, [Unit.new(group)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |status| exits << [:exit, status] }, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :trap => lambda { |*| }, :watch_coordinator => false) + end + + it "runs only those examples and the context hooks of the groups containing them, as `rspec path[ids]` would, and then the whole group as before" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + jobs.last.send([:run, 0, group.id, ids("a2")]) + jobs.last.send([:run, 1, group.id, ids("o1", "b1")]) + jobs.last.send([:run, 2, group.id]) + jobs.last.send([:stop]) + + unit_worker.run + + expect(hooks).to eq([:outer, :a, :a2, :outer, :o1, :b, :b1, :outer, :o1, :a, :a1, :a2, :b, :b1, :b2]) + started = frames.select { |frame| frame.first == :group_started }.map { |frame| frame[1] } + expect(started.first(2)).to eq([group.id, group.children[0].id]) + expect(group.descendant_filtered_examples.size).to eq(5) + expect(group.children.map { |child| child.descendant_filtered_examples.size }).to eq([2, 2]) + end + + it "restores what the group runs when a selected example fails" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + group.children[1].examples.last.metadata[:fail] = true + jobs.last.send([:run, 0, group.id, ids("b2")]) + jobs.last.send([:stop]) + + unit_worker.run + + expect(frames.find { |frame| frame.first == :unit_finished }).to eq([:unit_finished, 0, false, false, false]) + expect(hooks).to eq([:outer, :b, :b2]) + expect(group.descendant_filtered_examples.map(&:description)).to eq(%w[o1 a1 a2 b1 b2]) + expect(RSpec.world.filtered_examples[group.children[1]].size).to eq(2) + end + end + describe "forked before the spec files were loaded" do def loading_worker Worker.new(2, nil, jobs.first, results.last, diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 1a8d955d4..fe7ca629f 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -133,7 +133,7 @@ def groups(count) it "skips groups without examples to run and keeps the given order without timing data" do all = groups(3) empty = RSpec.describe("empty") - expect(Parallel.units_for([all[1], empty, all[0], all[2]], config)).to eq([all[1], all[0], all[2]]) + expect(Parallel.units_for([all[1], empty, all[0], all[2]], config).map(&:group)).to eq([all[1], all[0], all[2]]) end context "with timings recorded in the example status persistence file", :isolated_directory do @@ -152,7 +152,7 @@ def groups(count) end it "puts groups with no recorded timing first, then the others longest first" do - expect(Parallel.units_for(all, config)).to eq([all[2], all[1], all[3], all[0]]) + expect(Parallel.units_for(all, config).map(&:group)).to eq([all[2], all[1], all[3], all[0]]) end it "keeps the given order for groups with equal timings" do @@ -165,12 +165,12 @@ def groups(count) |#{all[3].examples.first.id} | passed | 1 second | STATUSES - expect(Parallel.units_for(all.reverse, config)).to eq(all.reverse) + expect(Parallel.units_for(all.reverse, config).map(&:group)).to eq(all.reverse) end it "ignores an unreadable status file" do allow(ExampleStatusPersister).to receive(:load_from).and_raise(Errno::EACCES) - expect(Parallel.units_for(all, config)).to eq(all) + expect(Parallel.units_for(all, config).map(&:group)).to eq(all) end end end diff --git a/rspec-core/spec/rspec/core/suite_hooks_spec.rb b/rspec-core/spec/rspec/core/suite_hooks_spec.rb index cc21d93c7..92669ca6a 100644 --- a/rspec-core/spec/rspec/core/suite_hooks_spec.rb +++ b/rspec-core/spec/rspec/core/suite_hooks_spec.rb @@ -94,6 +94,23 @@ module RSpec::Core }.to raise_error("boom").and yield_control end + it 'calls a given callable before any `after(:suite)` hook, however the suite ends' do + order = [] + RSpec.configuration.after(:suite) { order << :after_suite } + RSpec.configuration.before(:suite) do + RSpec.configuration.after(:suite) { order << :added_by_a_before_suite_hook } + end + exiting = lambda { order << [:before_teardown, $!.class] } + + RSpec.configuration.with_suite_hooks(exiting) { order << :suite } + expect { RSpec.configuration.with_suite_hooks(exiting) { exit 2 } }.to raise_error(SystemExit) + + expect(order).to eq([ + :suite, [:before_teardown, NilClass], :added_by_a_before_suite_hook, :after_suite, + [:before_teardown, SystemExit], :added_by_a_before_suite_hook, :added_by_a_before_suite_hook, :after_suite + ]) + end + describe "the runner" do include_context "Runner support" From 22004b437662fd51aee02bcb86b92cbee37709a9 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 16:50:33 +0200 Subject: [PATCH 27/58] [rspec-core] Run a `:parallel_split => N` group as N parts in workers A run is never shorter than its longest top-level group: rspec-core's own suite under --parallel=8 spent 10.62 of its 10.64 s in one. A group tagged with a part count now runs as that many slices of the order its examples run in, balanced by recorded run times, each a filtered run of the group in whichever worker is free. It is opt-in because context hooks run once per part and parts can run at the same time; groups with a custom ordering are not split. rspec-core tags its two longest groups. Co-Authored-By: Claude Opus 5.5 --- rspec-core/Changelog.md | 17 ++-- rspec-core/README.md | 23 +++++ .../features/command_line/parallel.feature | 21 ++++- .../step_definitions/additional_cli_steps.rb | 6 ++ rspec-core/lib/rspec/core/parallel.rb | 77 +++++++++++++++-- .../lib/rspec/core/parallel/sequencer.rb | 10 ++- rspec-core/spec/integration/parallel_spec.rb | 51 +++++++++++- .../rspec/core/parallel/coordinator_spec.rb | 2 +- .../rspec/core/parallel/sequencer_spec.rb | 11 +++ rspec-core/spec/rspec/core/parallel_spec.rb | 83 +++++++++++++++++++ 10 files changed, 284 insertions(+), 17 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 13e3ff8ee..ea1150ed8 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -21,14 +21,15 @@ Enhancements: after code under test called `exit`), and a worker that crashes fails the example it was running instead of taking the run down, while the examples of its group it had not started run in another worker. Code that calls `exit` ends the run with that status, as - it does serially. A run stopped - with SIGTERM or SIGHUP says which example each worker was running and for how long. With - `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are - forked before the spec files are loaded and each loads them itself, so nothing the spec - files do while loading is shared with the workers; a worker that does not load the - examples the coordinating process has is not used. Where `fork` is unavailable the run - is serial with a warning. `--bisect` runs serially, and the reproduction command it - prints now ends with `--no-parallel`. + it does serially. A top-level group tagged `:parallel_split => N` runs as N parts in + different workers. A run stopped with SIGTERM or SIGHUP says which example each worker + was running and for how long. With `--no-parallel-preload` + (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec + files are loaded and each loads them itself, so nothing the spec files do while loading + is shared with the workers; a worker that does not load the examples the coordinating + process has is not used. Where `fork` is unavailable the run is serial with a warning. + `--bisect` runs serially, and the reproduction command it prints now ends with + `--no-parallel`. (Illia Zub) * Load and filter large suites faster by no longer looking up the working directory once per example and example group, which took seconds on a suite of 150,000 examples, and more when diff --git a/rspec-core/README.md b/rspec-core/README.md index 178170727..4230fa0ca 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -403,6 +403,29 @@ started ignoring (`nohup`), or that your suite installs a handler for, is left alone. Time the whole run spends stopped (Ctrl-Z) or the machine asleep does not count towards these timeouts. +A run is never shorter than its longest top-level example group, which a +single worker runs. `--profile` shows the slowest groups; tag one whose +examples can run at the same time in different processes with the number of +parts to run it as, and workers take the parts like any other group: + +```ruby +RSpec.describe Pricing, :parallel_split => 4 do + # ... +end +``` + +Each part is a slice of the order the group's examples run in, of about the +same total run time when a previous run recorded run times in the +`example_status_persistence_file_path` file (of the same number of examples +otherwise), and runs as `rspec spec/pricing_spec.rb[1:2,1:3]` would: +`before(:context)` and `after(:context)` hooks run once per part, so a group +that sets up something expensive or shared in them may be better left whole. +There are no more parts than examples, the number of parts does not depend on +the machine, and a group whose examples (or those of a group in it) use a +custom ordering is not split, with a warning. With a formatter that follows +example groups (documentation, HTML), each part reads as a block of its own, +under the group's description; `--profile` adds up the parts. + `--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec in parallel, RSpec prints a warning and runs serially. When the system refuses a worker -- too many open diff --git a/rspec-core/features/command_line/parallel.feature b/rspec-core/features/command_line/parallel.feature index c330aac10..f502e4563 100644 --- a/rspec-core/features/command_line/parallel.feature +++ b/rspec-core/features/command_line/parallel.feature @@ -27,7 +27,9 @@ Feature: `--parallel` option A worker that crashes fails the example it was running, and the examples of its group it had not started run in another worker. Code that calls `exit` - ends the run with that status, as it does in a serial run. + ends the run with that status, as it does in a serial run. A top-level group + tagged `:parallel_split => N` runs as N parts in different workers, which + helps when it is the longest one of the suite. `--parallel` needs `fork`. Where it is not available (Windows, JRuby), or when `TEST_ENV_NUMBER` is already set because another tool is running RSpec @@ -159,6 +161,23 @@ Feature: `--parallel` option Then the exit status should be 3 And the output should contain "is ending the run: the code it ran called `exit` or `abort` (status 3)" + Scenario: A long group tagged `:parallel_split` runs as parts in several workers + Given a file named "spec/long_spec.rb" with: + """ruby + RSpec.describe "Long", :parallel_split => 2 do + 4.times do |i| + it "runs part of the group #{i}" do + File.write("ran_#{i}_in_#{Process.pid}", "") + sleep 0.2 + end + end + end + """ + When I run `rspec --parallel=3 --order defined spec/long_spec.rb spec/beta_spec.rb` + Then the exit status should be 0 + And the output should contain "6 examples, 0 failures" + And the files the examples wrote should name 2 processes + Scenario: With `--no-parallel-preload` workers run the groups they loaded themselves Given a file named "spec/gamma_spec.rb" with: """ruby diff --git a/rspec-core/features/step_definitions/additional_cli_steps.rb b/rspec-core/features/step_definitions/additional_cli_steps.rb index 30850222b..6e8a567a2 100644 --- a/rspec-core/features/step_definitions/additional_cli_steps.rb +++ b/rspec-core/features/step_definitions/additional_cli_steps.rb @@ -278,3 +278,9 @@ def normalize_failure_output(text) World(Normalization) World(FormatterSupport) + +Then(/^the files the examples wrote should name (\d+) processes$/) do |count| + cd('.') do + expect(Dir['ran_*_in_*'].map { |file| file[/\d+\z/] }.uniq.size).to eq(Integer(count)) + end +end diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index 92bd23695..ec3d6f0be 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -96,16 +96,17 @@ def self.default_worker_count end # The units to dispatch: top-level groups with at least one example to - # run, longest first when a previous run recorded their timings in the - # example status persistence file, so that the slowest unit is never - # the one that starts last. Units without timings go first because - # nothing is known about them. + # run -- or, for a group with `parallel_split` metadata, that many + # parts of it -- longest first when a previous run recorded their + # timings in the example status persistence file, so that the slowest + # unit is never the one that starts last. Units without timings go + # first because nothing is known about them. # @return [Array] # @private def self.units_for(example_groups, configuration) run_times = recorded_run_times(configuration) units = example_groups.flat_map do |group| - group.descendant_filtered_examples.empty? ? [] : [Unit.new(group)] + group.descendant_filtered_examples.empty? ? [] : split(group, run_times) end return units if run_times.empty? @@ -119,6 +120,72 @@ def self.units_for(example_groups, configuration) unknown + known.each_with_index.sort_by { |unit, index| [-timed[unit], index] }.map(&:first) end + # The orderings that order part of a group's examples the way they + # order all of them. + SPLITTABLE_ORDERINGS = [Ordering::Identity, Ordering::Random, Ordering::RecentlyModified].freeze + + # A group tagged `parallel_split: N` is run as N parts -- no more than it + # has examples -- each a slice of the order its examples run in, + # balanced by their recorded run times (by their number, without). + # @return [Array] + # @private + def self.split(group, run_times) + parts = group.metadata[:parallel_split] + return [Unit.new(group)] unless parts + + unless parts.is_a?(Integer) && parts >= 2 + RSpec.warning "Ignoring `parallel_split: #{parts.inspect}` on #{group.description} (#{group.location}): it is " \ + "the number of parts to run the group's examples in, at least 2", :call_site => nil + return [Unit.new(group)] + end + + unless group.descendants.all? { |each| SPLITTABLE_ORDERINGS.any? { |kind| each.ordering_strategy.is_a?(kind) } } + RSpec.warning "Not splitting #{group.description} (#{group.location}) into parts: it orders examples with a " \ + "custom ordering, which may not order part of them the way it orders all of them", + :call_site => nil + return [Unit.new(group)] + end + + ids = execution_order(group) + return [Unit.new(group)] if ids.size < 2 + + slices(ids, [parts, ids.size].min, run_times).map { |slice| Unit.new(group, slice) } + end + + # @return [Array] the ids of the group's examples in the order a + # run of the group runs them in + # @private + def self.execution_order(group) + strategy = group.ordering_strategy + strategy.order(group.filtered_examples).map(&:id) + + strategy.order(group.children).flat_map { |child| execution_order(child) } + end + + # Cuts `ids` into `count` consecutive, non-empty slices of about the + # same total run time -- each example goes to the slice its midpoint + # falls in -- where an example without a recorded time counts as the + # mean of those with one, and with no time recorded at all every + # example counts the same. + # @private + def self.slices(ids, count, run_times) + recorded = ids.map { |id| run_times[id] }.compact + mean = recorded.empty? ? 0.0 : recorded.sum / recorded.size + weights = ids.map { |id| run_times.fetch(id, mean) } + weights = Array.new(ids.size, 1.0) unless weights.sum > 0 + total = weights.sum + + slices = [[]] + done = 0.0 + ids.each_with_index do |id, index| + more = count - slices.size + wanted = ((done + (weights[index] / 2.0)) * count / total).floor + slices << [] if !slices.last.empty? && more > 0 && (wanted >= slices.size || ids.size - index <= more) + slices.last << id + done += weights[index] + end + slices + end + # @private def self.recorded_run_times(configuration) path = configuration.example_status_persistence_file_path diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb index e1138577f..417477067 100644 --- a/rspec-core/lib/rspec/core/parallel/sequencer.rb +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -64,6 +64,7 @@ def initialize(reporter, options={}) @visible = nil @visible_groups = [] @last_delivery_at = @clock.call + @group_durations = {} @stats = { :queued => 0, :peak_queued => 0, :queued_bytes => 0, :peak_queued_bytes => 0, :flushes => 0 } end @@ -216,7 +217,7 @@ def deliver_in_context(stream, event) @reporter.example_group_finished(event.target) stream.open_groups.pop @visible_groups.pop - @reporter.record_example_group_duration(event.target, event.payload) if event.payload + record_duration(event.target, event.payload) if event.payload when :example_started @reporter.example_started(event.target) when :example_finished @@ -226,6 +227,13 @@ def deliver_in_context(stream, event) end end + # Measured in the worker that ran it; a group split across workers + # took as long as its parts did together. + def record_duration(group, duration) + total = @group_durations[group] = @group_durations.fetch(group, 0.0) + duration + @reporter.record_example_group_duration(group, total) + end + def deliver_context_free(event) case event.kind when :message then @reporter.message(event.payload) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 3ebc1b911..9a17f290c 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -2,7 +2,10 @@ require 'support/aruba_support' require 'json' -RSpec.describe 'Parallel execution (--parallel)', :slow, :skip => !RSpec::Support::RubyFeatures.fork_supported? do +# The longest group of rspec-core's suite: under `--parallel` it runs as parts in +# several workers (each has its own aruba directory). +RSpec.describe 'Parallel execution (--parallel)', :slow, :parallel_split => 4, + :skip => !RSpec::Support::RubyFeatures.fork_supported? do include_context "aruba support" # The coordinator refuses to nest inside another parallel worker, which is @@ -601,6 +604,52 @@ def write_exiting_suite(status, after_suite="") end end + describe 'a group tagged `parallel_split: N`' do + def write_split_suite(extra="") + write_file_formatted 'spec/split_spec.rb', " + RSpec.describe 'Split', :parallel_split => 3 do + before(:context) { File.write(\"context_\#{Process.pid}_\#{rand(1_000_000)}\", '') } + 6.times { |i| it(\"e\#{i}\") { File.write(\"example_\#{i}_\#{Process.pid}\", ''); sleep 0.1 } } + context 'nested' do + 2.times { |i| it(\"n\#{i}\") { File.write(\"nested_\#{i}_\#{Process.pid}\", '') } } + end + #{extra} + end + RSpec.describe('Whole') { it('runs once') { } } + " + end + + it 'runs it as that many parts in different workers, every example once, with the counts of a serial run' do + write_split_suite + run_rspec "--order defined --format json --out serial.json" + cd('.') { Dir['{context,example,nested}_*'].each { |file| File.delete(file) } } + + run_rspec "--order defined --parallel=3 --format json --out parallel.json --format progress" + + expect(last_cmd_exit_status).to eq(0) + cd('.') do + serial, parallel = %w[serial.json parallel.json].map { |file| JSON.parse(File.read(file)) } + ids = parallel['examples'].map { |example| example['id'] } + expect(ids.size).to eq(ids.uniq.size) + expect(ids.sort).to eq(serial['examples'].map { |example| example['id'] }.sort) + expect(parallel['summary']).to include('example_count' => 9, 'failure_count' => 0) + expect(Dir['context_*'].size).to eq(3) + pids = Dir['{example,nested}_*'].map { |file| file[/\d+\z/] } + expect(pids.uniq.size).to be >= 2 + end + end + + it 'runs the rest of a part whose worker dies in another worker' do + write_split_suite("context('crashing') { it('dies') { exit!(7) }; it('after the crash') { File.write('after_crash', '') } }") + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include("11 examples, 1 failure", "Split crashing dies") + cd('.') { expect(File).to exist('after_crash') } + end + end + it 'stops its workers, even ones that cannot tear down, when it and its process group get SIGTERM, saying what they ran' do write_file_formatted 'spec/wedged_spec.rb', " RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index cf9bff5d0..e0c8e70a2 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -75,7 +75,7 @@ def verify_mocks_for_rspec; end def teardown_mocks_for_rspec; end end - RSpec.describe Coordinator do + RSpec.describe Coordinator, :parallel_split => 2 do let(:config) { RSpec.configuration } let(:world) { RSpec.world } let(:output) { StringIO.new } diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index 64fedd9fe..a8ecf3801 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -105,6 +105,17 @@ def events expect(applied).to eq([ex(g1, "a")]) end + it "records the parts of a group split across workers as taking as long as they did together" do + seq = sequencer(false) + recorded = [] + allow(reporter).to receive(:record_example_group_duration) { |group, total| recorded << [group, total] } + + run_group(seq, 1, g1, 1.5) + run_group(seq, 2, g1, 2.0) + + expect(recorded).to eq([[g1, 1.5], [g1, 3.5]]) + end + it "records the worker-measured duration of top-level groups for the profiler" do seq = sequencer(false) expect(reporter).to receive(:record_example_group_duration).with(g1, 1.5) diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index fe7ca629f..d871e5dc0 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -136,6 +136,89 @@ def groups(count) expect(Parallel.units_for([all[1], empty, all[0], all[2]], config).map(&:group)).to eq([all[1], all[0], all[2]]) end + describe "of a group tagged `parallel_split: N`" do + def tagged(parts, count=6, &block) + RSpec.describe("Tagged", :parallel_split => parts) do + count.times { |i| it("e#{i}") {} } + instance_exec(&block) if block + end + end + + it "runs it as N parts, each a consecutive slice of the order its examples run in" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(3) { context("nested") { it("n0") {} } } + ids = group.descendant_filtered_examples.map(&:id) + + units = Parallel.units_for([group, groups(1).first], config) + + expect(units.first(3).map(&:group)).to all(eq(group)) + expect(units.first(3).map(&:example_ids)).to eq([ids[0, 2], ids[2, 3], ids[5, 2]]) + expect(units.last.example_ids).to be_nil + end + + it "slices the order a random ordering runs the examples in, which orders any part of them the same way" do + config.seed = 1234 + group = tagged(2) + order = Parallel.execution_order(group) + + parts = Parallel.units_for([group], config).map(&:example_ids) + + expect(parts.flatten).to eq(order) + expect(order).not_to eq(group.examples.map(&:id)) + expect(config.ordering_registry.fetch(:global).order(group.examples.select { |e| parts.first.include?(e.id) }).map(&:id)).to eq(parts.first) + end + + it "makes no more parts than there are examples" do + expect(Parallel.units_for([tagged(8, 3)], config).map(&:example_ids).map(&:size)).to eq([1, 1, 1]) + end + + it "leaves a group with a single example whole" do + expect(Parallel.units_for([tagged(4, 1)], config).map(&:example_ids)).to eq([nil]) + end + + [true, 1, 0, "3", 2.5].each do |value| + it "warns and leaves the group whole for `parallel_split: #{value.inspect}`" do + expect(RSpec).to receive(:warning).with(/Ignoring `parallel_split: #{Regexp.escape(value.inspect)}` on Tagged \(.+\): it is the number of parts/, :call_site => nil) + expect(Parallel.units_for([tagged(value)], config).map(&:example_ids)).to eq([nil]) + end + end + + it "warns and leaves the group whole when it, or a group in it, uses a custom ordering" do + config.register_ordering(:reversed) { |items| items.reverse } + group = tagged(2) { context("nested", :order => :reversed) { it("n0") {} } } + expect(RSpec).to receive(:warning).with(/Not splitting Tagged \(.+\) into parts: it orders examples with a custom ordering/, :call_site => nil) + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([nil]) + end + + context "with run times recorded", :isolated_directory do + before { config.example_status_persistence_file_path = "statuses.txt" } + + def record(times) + rows = times.map { |id, time| "#{id} | passed | #{time} |\n" }.join + File.write("statuses.txt", "example_id | status | run_time |\n---------- | ------ | -------- |\n#{rows}") + end + + it "balances the parts by them, counting an example without one as the mean of the others" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(2, 4) + ids = group.examples.map(&:id) + record(ids[0] => "3 seconds", ids[1] => "1 second", ids[2] => "1 second") + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([[ids[0]], ids[1, 3]]) + end + + it "balances them by the number of examples when every recorded time is zero" do + config.ordering_registry.register(:global, Ordering::Identity.new) + group = tagged(2, 4) + ids = group.examples.map(&:id) + record(ids.to_h { |id| [id, "0 seconds"] }) + + expect(Parallel.units_for([group], config).map(&:example_ids)).to eq([ids[0, 2], ids[2, 2]]) + end + end + end + context "with timings recorded in the example status persistence file", :isolated_directory do let(:all) { groups(4) } From 37b89f0907a57fdf408beef5564a94667bc1fd7d Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 16:58:51 +0200 Subject: [PATCH 28/58] [rspec-core] Keep `exit` from hiding forced stops or later exits Once a worker's `exit` has ended the run, its status stands only if every worker then leaves on its own: another worker that is lost, fails or has to be terminated now makes the run end with the exit code its results call for, instead of the `exit` status. And an `after(:suite)` hook that calls `exit` with another status decides the status, as it does serially, instead of being reported as a worker that did not exit in time. Co-Authored-By: Claude Opus 5.5 --- .../lib/rspec/core/parallel/coordinator.rb | 2 ++ rspec-core/lib/rspec/core/parallel/worker.rb | 8 +++--- .../lib/rspec/core/parallel/worker_exits.rb | 17 +++++++++--- rspec-core/spec/integration/parallel_spec.rb | 27 +++++++++++++++++++ .../spec/rspec/core/parallel/worker_spec.rb | 18 +++++++++++++ 5 files changed, 65 insertions(+), 7 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index db4fc4de5..86db4b768 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -369,6 +369,7 @@ def enforce_deadlines def escalate(worker) case worker.escalation when nil + spoil_exit overdue(worker) worker.state = :terminating worker.signal(@process, 'TERM', :term, now + @term_grace) @@ -384,6 +385,7 @@ def escalate(worker) # Counted as an error outside of examples, like a failing suite hook. def error(text) + spoil_exit @configuration.reporter.notify_non_example_exception(WorkerError.new(text), 'An error occurred in a parallel worker.', :failure_lines => [text]) end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index c89223344..1cc087709 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -140,9 +140,9 @@ def run_units true rescue SystemExit => e # Raised by an `after(:suite)` hook once it had left the loop, unless - # that was what made it leave. - unless @exit_announced - @exit_cause ||= exit_cause(e, 'an `after(:suite)` hook') + # it is what made it leave: its status is the process's, as serially. + unless e.status == @announced_status + @exit_cause = exit_cause(e, 'an `after(:suite)` hook') send_exiting(e) end raise @@ -170,7 +170,7 @@ def leaving # @private def send_exiting(exception) status = exception.status if exception.is_a?(SystemExit) - @exit_announced = true if status + @announced_status = status if status @results.send([:exiting, @exit_cause, status]) rescue Errno::EPIPE, IOError nil # the coordinator is gone; the watchdog and its own watch still end this process diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index ab58e2e3b..46fb2f087 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -34,7 +34,6 @@ def judge_exit(worker, status) def overdue(worker) timeout = Parallel.seconds(@configuration.parallel_shutdown_timeout) if worker.called_exit - @exit_spoiled = true error("#{worker.description} did not exit within #{timeout} of ending the run because " \ "#{worker.exit_cause} (its `after(:suite)` hooks or `at_exit` handlers may be stuck), so it was " \ "sent SIGTERM. Raise `config.parallel_shutdown_timeout` if they legitimately need longer.") @@ -84,9 +83,14 @@ def ended_by_exit(worker, exit_status) worker.called_exit = exit_status # Leaving anyway: it is not asked to stop. worker.quit_sent = true - return if @exit_status + if @exit_status + # A later `exit` in the same process decides its status. + @exit_status = exit_status if worker.equal?(@exit_worker) + return + end @exit_status = exit_status + @exit_worker = worker running = worker.current_example report "#{worker.description} is ending the run: #{worker.exit_cause}" \ "#{" in #{running.full_description} (#{running.location})" if running}. As in a serial run, no " \ @@ -100,6 +104,7 @@ def ended_by_exit(worker, exit_status) # examples:` and `.rspec_status` point at -- and what it had not # started of its unit goes to another worker. def lost(worker, what, consequence='') + spoil_exit because = "#{consequence}#{worker.exit_cause && ": #{worker.exit_cause}"}" if worker.loading worker.loading = false @@ -178,6 +183,13 @@ def rest_clause(unit, rest, requeued) end end + # Once a worker's `exit` ended the run, its status stands only if every + # worker then leaves on its own: one lost, failing or forced to stop + # makes it end with the exit code the results call for. + def spoil_exit + @exit_spoiled = true if @exit_status + end + def accepting_work? !@quitting && !@winding_down end @@ -193,7 +205,6 @@ def queue_first(unit) # otherwise was aborted by its watchdog or killed. def left_by_exit(worker, status) unless status.nil? || (status.exited? && status.exitstatus == worker.called_exit) - @exit_spoiled = true ended = "#{status.signaled? ? 'was' : 'exited with'} #{WorkerHandle.describe_status(status)}" error("#{worker.description} ended the run because #{worker.exit_cause}, then #{ended}: if its " \ "watchdog aborted it, its `after(:suite)` hooks or " \ diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 9a17f290c..6da0508a7 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -532,6 +532,33 @@ def write_exiting_suite(status, after_suite="") end end + it "ends with the status of an `after(:suite)` hook's later exit, as a serial run does" do + write_exiting_suite(4, "exit 5 if $exiting") + + run_rspec "--order defined --parallel=2" + + expect(last_cmd_exit_status).to eq(5) + expect(last_cmd_stdout).not_to include("An error occurred in a parallel worker") + end + + it 'fails the run as usual when another worker then has to be stopped by force' do + write_file_formatted 'spec/exit_spec.rb', " + RSpec.configure { |c| c.parallel_shutdown_timeout = 0.5 } + RSpec.describe('Exits') { it('exits') { sleep 0.5; exit 0 } } + RSpec.describe('Stuck') { it('sleeps') { sleep 30 } } + " + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--order defined --parallel=2" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 15 + expect(last_cmd_exit_status).to eq(1) + expect(last_cmd_stdout).to include( + "is ending the run: the code it ran called `exit` or `abort` (status 0)", + "did not stop within 0.5 seconds of being asked to stop early (fail-fast or interrupt) while running this example" + ) + end + it 'is bounded when the worker then does not exit, failing the run' do # Leaves the worker's watchdog plenty of room to fire before the # coordinator steps in, even on a loaded machine. diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index ea13b9ada..d81e1d31a 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -278,6 +278,24 @@ def leaving_frames expect(leaving_frames).to eq([[:exiting, nil, nil], [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5]]) end + it "says so again when an after(:suite) hook calls exit with another status than the example did" do + RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + exiting = RSpec.describe("Exits") { it("exits") { exit 4 } } + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { exit 5 } + jobs.last.send([:run, 0, exiting.id]) + + expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(5) } + + expect(leaving_frames).to eq([ + [:exiting, "the code it ran called `exit` or `abort` (status 4)", 4], + [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5] + ]) + end + it "names a signal or another exception that ends the process" do expect(worker.exit_cause(SignalException.new("TERM"), "the code it ran")).to eq("it received SIGTERM") expect(worker.exit_cause(NoMemoryError.new("too much"), "the code it ran")).to eq( From 5a79d054b4ffb93c3283be981fb4924bd51da345 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 17:50:43 +0200 Subject: [PATCH 29/58] [rspec-core] Keep --parallel specs at full coverage and merge-friendly The coordinator's report of a worker lost between example groups was reached only when the ECHILD spec happened to reap an idle worker: on Linux MRI 3.1 (aarch64 container) that line was missed in 3 of 6 runs of CI's run_rspec, failing its 100% coverage minimum. Since workers now wait while another one is busy, a worker killed in that wait is a real case: a spec now covers it deterministically (30/30 runs reached it). The `exit` and SIGTERM specs matched the summary line up to its end, so upstream's open #302 (examples not run) and #305 (expectation counts) broke them by appending to it; they now allow a suffix, as the fail-fast spec already did. Co-Authored-By: Claude Opus 5.5 --- rspec-core/spec/integration/parallel_spec.rb | 2 +- .../rspec/core/parallel/coordinator_spec.rb | 34 ++++++++++++++++++- 2 files changed, 34 insertions(+), 2 deletions(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 6da0508a7..c31beffdb 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -527,7 +527,7 @@ def write_exiting_suite(status, after_suite="") /Parallel worker \d \(pid \d+\) is ending the run: the code it ran called `exit` or `abort` \(status #{status}\) in Exits exits \(\.\/spec\/exit_spec\.rb:6\)\. As in a serial run, no further examples start and the run exits with status #{status}\.$/ ) expect(last_cmd_stdout[/^(\d+) examples?/, 1].to_i).to be < 22 - expect(last_cmd_stdout).to match(/^\d+ examples?, 0 failures$/) + expect(last_cmd_stdout).to match(/^\d+ examples?, 0 failures(, .+)?$/) cd('.') { expect(File).not_to exist('ran_after_exit') } end end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index e0c8e70a2..140952fe9 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -148,6 +148,17 @@ def leave_for_a_signal(jobs, results) results.send([:exiting, "it received SIGHUP", nil]) end + # Dies as it asks for its second job, as a process killed while it waits + # for work would. + def die_on_second_job(jobs) + asked = 0 + jobs.define_singleton_method(:receive) do + asked += 1 + Thread.current.kill if asked == 2 + super() + end + end + # What a worker forked before loading reports once it has loaded the spec # files: by default, the examples this process has. def report_loaded(results, behaviour) @@ -166,6 +177,7 @@ def send_unknown_ids(results) end def run_fake_worker(number, jobs, results, behaviour, &record_exit) + die_on_second_job(jobs) if behaviour == :dies_between_groups worker_config = RSpec::Core::Configuration.new worker_config.parallel_shutdown_timeout = 60 Thread.current[:fake_worker_world] = RSpec::Core::World.new(worker_config) @@ -392,6 +404,26 @@ def example_failed(_notification) end end + it "reports a worker that exits between groups, while it waits for a busy one, as an error of the run" do + worker_behaviours[1] = :dies_between_groups + reported = output + busy = RSpec.describe("Busy") do + it("passes once worker 1 is reported lost") do + give_up_at = RSpec::Core::Parallel.monotonic_time + 10 + sleep 0.01 until reported.string.include?("between example groups") || RSpec::Core::Parallel.monotonic_time > give_up_at + end + end + groups = [passing_group("A"), busy] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(groups.map { |group| group.examples.first.execution_result.status }).to eq([:passed, :passed]) + expect(messages).to match(/Parallel worker 1 \(pid \d+\) exited unexpectedly \(exit status 1\) between example groups\./) + expect(messages).to include("2 examples, 0 failures, 1 error occurred outside of examples") + end + # A single worker, so that no well-behaved one can miss a deadline this # short on a loaded machine. describe "a worker that does not exit after finishing" do @@ -904,7 +936,7 @@ def example_passed(_notification) expect(messages).to match( /^Parallel worker 2 \(pid \d+\) was running #{Regexp.escape(stuck.examples.first.id)} \(Stuck sleeps\) for [\d.]+ seconds?\.$/ ) - expect(messages).to match(/^2 examples, 0 failures$/) + expect(messages).to match(/^2 examples, 0 failures(, .+)?$/) expect(stuck.examples.first.execution_result.status).to be_nil expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") end From 01b0a1383b3c6b275bcf9e692d2b3451781ec575 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 18:00:12 +0200 Subject: [PATCH 30/58] [rspec-core] Keep two worker specs from hanging the suite on JRuby JRuby's Example#run rescues SystemExit (example.rb), so there `exit` in an example fails it, serially too, and never leaves the worker's job loop. The two worker specs that call it from an example sent no stop message after the job, so on JRuby 9.4 the worker waited for one forever and CI's run_rspec step hung (JVM thread dump: Channel#receive in the job loop, inside the specs' raise_error). They now skip on JRuby, where --parallel runs serially anyway, and queue a stop so that an `exit` that does not propagate fails them instead of hanging the suite. Co-Authored-By: Claude Opus 5.5 --- rspec-core/spec/rspec/core/parallel/worker_spec.rb | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index d81e1d31a..02b648b43 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -225,6 +225,10 @@ def frames end describe "leaving its job loop" do + # JRuby's `Example#run` rescues `SystemExit` (see example.rb), so there, + # serially too, `exit` in an example fails it rather than ending the run. + jruby_rescues_exit = RSpec::Support::Ruby.jruby? && "JRuby's Example#run rescues SystemExit" + def leaving_frames frames.select { |frame| frame.first == :exiting } end @@ -246,7 +250,7 @@ def leaving_frames expect(leaving_frames).to eq([[:exiting, "the code it ran called `exit` or `abort` (status 4)", 4]]) end - it "says which status the code it ran called `exit` with in an example, and runs nothing after it" do + it "says which status the code it ran called `exit` with in an example, and runs nothing after it", :skip => jruby_rescues_exit do RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) ran = [] exiting = RSpec.describe("Exits") do @@ -258,6 +262,7 @@ def leaving_frames :trap => lambda { |*| }, :watch_coordinator => false) allow(instance).to receive(:arm_watchdog) jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(3) } @@ -278,7 +283,7 @@ def leaving_frames expect(leaving_frames).to eq([[:exiting, nil, nil], [:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 5)", 5]]) end - it "says so again when an after(:suite) hook calls exit with another status than the example did" do + it "says so again when an after(:suite) hook calls exit with another status than the example did", :skip => jruby_rescues_exit do RSpec.configuration.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) exiting = RSpec.describe("Exits") { it("exits") { exit 4 } } instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, @@ -287,6 +292,7 @@ def leaving_frames allow(instance).to receive(:arm_watchdog) RSpec.configuration.after(:suite) { exit 5 } jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) expect { instance.run }.to raise_error(SystemExit) { |error| expect(error.status).to eq(5) } From d85f8cd027c99943827b7b269428e5fe0eae1296 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 18:28:42 +0200 Subject: [PATCH 31/58] [rspec-core] Keep a forced stop's `exit` from deciding the run's status A worker past its shutdown timeout after its last group is sent SIGTERM, but its frames were still read: if an `after(:suite)` hook took the signal by calling `exit 0`, that became the run's status, and a run with an error (or failures) exited 0. Like every other path that terminates a worker, that one now stops reading it. A coordinator spec reproduces it: it got `exit(0)` in 3 of 3 runs before the fix and passes 5 of 5 after (2 of 2 fail again with only the library change taken out). When the same worker's later `exit` replaces the status, the run now says so instead of only naming the first status. The SIGTERM spec, whose summary match allows a suffix, also checks that no error was reported. Found by an independent review of the last three commits. Co-Authored-By: Claude Opus 5.5 --- .../lib/rspec/core/parallel/worker_exits.rb | 15 +++++++-- rspec-core/spec/integration/parallel_spec.rb | 3 ++ .../rspec/core/parallel/coordinator_spec.rb | 32 +++++++++++++++++++ 3 files changed, 47 insertions(+), 3 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 46fb2f087..92dd018a4 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -43,6 +43,8 @@ def overdue(worker) error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ "`config.parallel_shutdown_timeout` if it legitimately needs longer.") + # Nothing it sends from now on is read: an `exit` in how it takes SIGTERM must not decide the status. + worker.results.close elsif worker.loading lost(worker, "took longer than #{Parallel.seconds(@load_allowance)}", SENT_TERM) elsif worker.quit_sent @@ -72,20 +74,20 @@ def exiting(worker, cause, exit_status=nil) worker.exit_cause ||= cause teardown = now + @configuration.parallel_shutdown_timeout + @watchdog_grace worker.deadline = teardown unless worker.escalation || (worker.deadline && worker.deadline >= teardown) - ended_by_exit(worker, exit_status) if exit_status + ended_by_exit(worker, exit_status, cause) if exit_status end # RSpec does not rescue `SystemExit`, so in a serial run `exit` ends # the run: here too no example starts from now on, busy workers stop # after the one they are running and the run exits with that status # -- the first one, if more workers call `exit`. - def ended_by_exit(worker, exit_status) + def ended_by_exit(worker, exit_status, cause) worker.called_exit = exit_status # Leaving anyway: it is not asked to stop. worker.quit_sent = true if @exit_status # A later `exit` in the same process decides its status. - @exit_status = exit_status if worker.equal?(@exit_worker) + replace_exit_status(worker, exit_status, cause) if worker.equal?(@exit_worker) return end @@ -98,6 +100,13 @@ def ended_by_exit(worker, exit_status) request_quit end + def replace_exit_status(worker, exit_status, cause) + return if exit_status == @exit_status + + @exit_status = exit_status + report "#{worker.description}: then #{cause}, so the run exits with status #{exit_status}." unless @exit_spoiled + end + # The worker can no longer be relied on, and nothing it sends from now # on is read: it would report the example a second time. What it was # running is reported as lost -- in a way that the summary, `Failed diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index c31beffdb..48554acb7 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -538,6 +538,9 @@ def write_exiting_suite(status, after_suite="") run_rspec "--order defined --parallel=2" expect(last_cmd_exit_status).to eq(5) + expect(last_cmd_stdout).to match( + /Parallel worker \d \(pid \d+\): then an `after\(:suite\)` hook called `exit` or `abort` \(status 5\), so the run exits with status 5\./ + ) expect(last_cmd_stdout).not_to include("An error occurred in a parallel worker") end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 140952fe9..d68897553 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -188,12 +188,25 @@ def run_fake_worker(number, jobs, results, behaviour, &record_exit) :exit => lambda { |status| record_exit.call(status) sleep if behaviour == :never_exits + exit_once_sent_term(results) if behaviour == :exits_once_sent_term }, :exit_now => record_exit, :trap => lambda { |*| }, :watch_coordinator => false).run end + # Its teardown hangs until it is sent SIGTERM, which an `after(:suite)` + # hook takes by calling `exit 0` -- and hangs again. + def exit_once_sent_term(results) + sleep 0.01 until @process.signals.any? { |name, _pid| name == 'TERM' } + begin + results.send([:exiting, "an `after(:suite)` hook called `exit` or `abort` (status 0)", 0]) + rescue Errno::EPIPE, IOError + nil # as Worker#send_exiting does once nothing reads it + end + sleep + end + # A fake worker killed with a signal is gone once its thread has died, # which a loaded machine can delay: the grace periods are long unless a # spec is about them, so that a slow death is not escalated. @@ -464,6 +477,24 @@ def example_failed(_notification) expect(messages).to include("still alive 0.1 seconds after SIGKILL; giving up on it") expect(RSpec::Core::Parallel.monotonic_time - started).to be < 5 end + + it "does not let an `exit` it calls once sent SIGTERM decide the run's exit status" do + config.error_exit_code = 3 + worker_behaviours[1] = :exits_once_sent_term + process = FakeProcess.new(["TERM"]) + + # Rescued: an `exit` deciding the status would end this suite here. + exit_code = + begin + coordinator([passing_group("A"), passing_group("B")], 1, short_graces.merge(:process => process)).run + rescue SystemExit => e + "exit(#{e.status})" + end + + expect(exit_code).to eq(3) + expect(process.signals.map(&:first)).to eq(%w[TERM KILL]) + expect(messages).not_to include("ending the run") + end end # Long enough a timeout for the well-behaved worker to stop within it even @@ -937,6 +968,7 @@ def example_passed(_notification) /^Parallel worker 2 \(pid \d+\) was running #{Regexp.escape(stuck.examples.first.id)} \(Stuck sleeps\) for [\d.]+ seconds?\.$/ ) expect(messages).to match(/^2 examples, 0 failures(, .+)?$/) + expect(messages).not_to include("outside of examples") expect(stuck.examples.first.execution_result.status).to be_nil expect(RSpec::Core::Parallel).to have_received(:restore_signals).with("TERM" => "DEFAULT") end From d7c5a351a961fb44772e8d431925c376618d4825 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 18:39:26 +0200 Subject: [PATCH 32/58] [rspec-core] Leave the parallel runner unloaded in serial runs Every `require "rspec/core"` loaded parallel.rb and parallel/plan.rb (about 400 lines) and every run built a plan, though only a run that asks for workers -- `--parallel`, `config.parallel_workers` or `config.parallel_preload = false`, set before the spec files load or by them -- needs either. The runner now loads them for such a run only. A one-example `rspec` process, each tree under /tmp, n=100 per arm interleaved in one session, macOS Ruby 3.4.8 at load ~10, median (IQR): branch base df877b73a 69.6 ms (67.7-71.4), before this commit 70.4 (68.7-72.3), with it 70.1 (68.4-71.1), and 69.7 (68.3-71.2) for a second arm of the same code. The same code run from the worktree's path under /Users took 71.0 ms, so only runs from one location compare. Co-Authored-By: Claude Opus 5.5 --- rspec-core/lib/rspec/core.rb | 1 - .../lib/rspec/core/parallel/coordinator.rb | 1 + rspec-core/lib/rspec/core/runner.rb | 18 +++++++++++---- rspec-core/spec/rspec/core/parallel_spec.rb | 8 +++---- rspec-core/spec/rspec/core/runner_spec.rb | 22 ++++++++++++++++++- 5 files changed, 40 insertions(+), 10 deletions(-) diff --git a/rspec-core/lib/rspec/core.rb b/rspec-core/lib/rspec/core.rb index fd1ecd2cc..86a300c6a 100644 --- a/rspec-core/lib/rspec/core.rb +++ b/rspec-core/lib/rspec/core.rb @@ -33,7 +33,6 @@ example shared_example_group example_group - parallel ].each { |name| RSpec::Support.require_rspec_core name } # Namespace for all core RSpec code. diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 86db4b768..2304cba44 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -1,3 +1,4 @@ +RSpec::Support.require_rspec_core "parallel" RSpec::Support.require_rspec_core "parallel/channel" RSpec::Support.require_rspec_core "parallel/remote_exception" RSpec::Support.require_rspec_core "parallel/worker_reporter" diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index b4306ac53..23c3af44b 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -76,6 +76,7 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) @options = options @configuration = configuration @world = world + @parallel_plan = nil end # Configures and runs a spec suite. @@ -83,19 +84,18 @@ def initialize(options, configuration=RSpec.configuration, world=RSpec.world) # @param err [IO] error stream # @param out [IO] output stream def run(err, out) - parallel = Parallel::Plan.new(self) - setup(err, out) { parallel.before_loading } + setup(err, out) { parallel_plan&.before_loading } return @configuration.reporter.exit_early(exit_code) if RSpec.world.wants_to_quit example_groups = @world.ordered_example_groups - coordinator = parallel.coordinator_for(example_groups) + coordinator = parallel_plan&.coordinator_for(example_groups) (coordinator ? coordinator.run : run_specs(example_groups)).tap do persist_example_statuses end ensure # Workers forked before loading that the run did not get to use. - parallel&.dismiss + @parallel_plan&.dismiss end # Wires together the various configuration objects and state holders. @@ -209,6 +209,16 @@ def exit_code(examples_passed=false) private + # Only a run configured to be parallel -- before the spec files load or + # by them -- loads the parallel runner. + def parallel_plan + return @parallel_plan if @parallel_plan + return nil if @configuration.parallel_workers.nil? && @configuration.parallel_preload? + + RSpec::Support.require_rspec_core "parallel" + @parallel_plan = Parallel::Plan.new(self) + end + def persist_example_statuses return if @configuration.dry_run return unless (path = @configuration.example_status_persistence_file_path) diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index d871e5dc0..323a7bf2a 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -389,19 +389,19 @@ def run_in_new_process(args) require "stringio" 2.times { |i| RSpec.describe("G\#{i}") { it("passes") {} } } status = RSpec::Core::Runner.run(["--options", File::NULL, *#{args.inspect}], $stderr, StringIO.new) - print [status, defined?(RSpec::Core::Parallel::Coordinator)].inspect + print [status, defined?(RSpec::Core::Parallel::Coordinator), defined?(RSpec::Core::Parallel)].inspect RUBY without_env_vars("SPEC_OPTS") { run_ruby_with_current_load_path(code) } end - it "leaves the runner unloaded in a serial run", :slow do + it "leaves the parallel runner unloaded in a serial run", :slow do out, err, _status = run_in_new_process([]) - expect([out, err]).to eq(["[0, nil]", ""]) + expect([out, err]).to eq(["[0, nil, nil]", ""]) end it "loads the runner for a run that forks workers", :slow, :skip => !RSpec::Support::RubyFeatures.fork_supported? do out, err, _status = run_in_new_process(%w[--parallel=2]) - expect([out, err]).to eq(['[0, "constant"]', ""]) + expect([out, err]).to eq(['[0, "constant", "constant"]', ""]) end end end diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index ea1b0b805..6b3272c78 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -368,6 +368,7 @@ def interrupt RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } allow(config).to receive(:load_spec_files) + config.parallel_workers = 2 coordinator = instance_double(Parallel::Coordinator, :run => 17) expect(Parallel).to receive(:coordinator_for).with(an_instance_of(Runner), world.ordered_example_groups).and_return(coordinator) config.example_status_persistence_file_path = "statuses.txt" @@ -379,10 +380,29 @@ def interrupt expect(runner.run(err, out)).to eq(17) end + it 'plans no parallel run unless one is configured' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) + expect(Parallel::Plan).not_to receive(:new) + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + + it 'plans a parallel run that the spec files configure, once they have loaded' do + RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } + allow(config).to receive(:load_spec_files) { config.parallel_workers = 2 } + expect(Parallel).to receive(:coordinator_for).and_return(nil) + + expect(Runner.new(ConfigurationOptions.new([]), config, world).run(err, out)).to eq(0) + end + context 'with a parallel plan' do let(:plan) { instance_double(Parallel::Plan, :before_loading => nil, :dismiss => nil) } - before { allow(Parallel::Plan).to receive(:new).and_return(plan) } + before do + config.parallel_workers = 2 + allow(Parallel::Plan).to receive(:new).and_return(plan) + end it 'lets it fork workers before the spec files load, then hands it the groups, and dismisses what it did not use' do RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } From 559e96f1672d9d6cc00dd680f52e5a13adbcd565 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Tue, 13 Oct 2026 19:20:48 +0200 Subject: [PATCH 33/58] [rspec-core] Keep the Kernel.rand spec from failing under upstream #305 The spec that checks a --no-parallel-preload run leaves Kernel.rand alone drew its value after its own `expect`. With rspec/rspec#305 merged, every `expect` calls `RSpec.configuration`, which in this process may create a configuration and so draw from Kernel.rand: on a trial merge with #305 the check failed after an `expect` for a serial run too, and passed when drawn first, both serially and with --parallel=2 (4/4 pass without #305). It now draws first, and still fails when the coordinator draws (checked by making `Coordinator#start` call Kernel.rand). Co-Authored-By: Claude Opus 5.5 --- rspec-core/spec/integration/parallel_spec.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 48554acb7..be3e45f4e 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -996,9 +996,11 @@ def rspec_process(*args) previous = Kernel.srand(1234) run_rspec '--parallel=2 --no-parallel-preload' + # Drawn first: an `expect` may itself draw, by creating `RSpec.configuration` (rspec/rspec#305 does). + drawn = Kernel.rand(2**62) expect(last_cmd_stdout).to include("3 examples, 0 failures") - expect(Kernel.rand(2**62)).to eq(Random.new(1234).rand(2**62)) + expect(drawn).to eq(Random.new(1234).rand(2**62)) ensure Kernel.srand(previous) end From 5bab6a2494c100eabcdf81ded631c2e7480d4ff5 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Thu, 15 Oct 2026 22:07:16 +0200 Subject: [PATCH 34/58] [rspec-core] Keep the slow-loading worker spec passing on a busy machine A still-loading worker's deadline is twice the coordinating process's own load time plus the shutdown timeout. At load 313 that stretched past the 2.1 s of work the other worker had, so worker 2 was dismissed unreported and the spec failed (1 of 5 serial suite runs). The busy worker now waits until worker 2 is gone, capped at 15 s. With the coordinating process's loading slowed by 0.5 s per file: before 3/3 failed, after 3/3 passed. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/integration/parallel_spec.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index be3e45f4e..769091756 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -949,10 +949,23 @@ def rspec_process(*args) it 'terminates a worker still loading long after the coordinating process finished, and runs its groups elsewhere' do write_file_formatted 'spec/spec_helper.rb', "RSpec.configure { |c| c.parallel_shutdown_timeout = 1 }" + # Worker 1 stays busy until worker 2 is gone, however long this + # process's own loading (and so worker 2's deadline) took. 3.times do |i| write_file_formatted "spec/s#{i}_spec.rb", " - sleep 600 if ENV['TEST_ENV_NUMBER'] == '2' - RSpec.describe('S#{i}') { it('keeps the other worker busy past the deadline') { sleep 0.7 } } + (File.write('loading.pid', Process.pid.to_s); sleep 600) if ENV['TEST_ENV_NUMBER'] == '2' + RSpec.describe('S#{i}') do + it('keeps the other worker busy past the deadline') do + give_up = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 15 + loading = lambda do + pid = File.exist?('loading.pid') && Integer(File.read('loading.pid')) + !pid || Process.kill(0, pid) + rescue Errno::ESRCH + false + end + sleep 0.05 while loading.call && Process.clock_gettime(Process::CLOCK_MONOTONIC) < give_up + end + end " end started = Process.clock_gettime(Process::CLOCK_MONOTONIC) From c8003126e131ac79e1e95398ccc2a323e7db7529 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Thu, 15 Oct 2026 22:25:03 +0200 Subject: [PATCH 35/58] [rspec-core] Publish the loading worker's pid atomically in its spec The busy worker could read loading.pid after it was created and before its contents were written, and fail on Integer(""). Worker 2 now writes a temporary file and renames it. Slowed-coordinator reproduction: 3/3 passed; the file passes 45/0 serially and with --parallel=4. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/integration/parallel_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 769091756..70f8295a1 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -953,7 +953,7 @@ def rspec_process(*args) # process's own loading (and so worker 2's deadline) took. 3.times do |i| write_file_formatted "spec/s#{i}_spec.rb", " - (File.write('loading.pid', Process.pid.to_s); sleep 600) if ENV['TEST_ENV_NUMBER'] == '2' + (File.write('loading.pid.tmp', Process.pid.to_s); File.rename('loading.pid.tmp', 'loading.pid'); sleep 600) if ENV['TEST_ENV_NUMBER'] == '2' RSpec.describe('S#{i}') do it('keeps the other worker busy past the deadline') do give_up = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 15 From 261c1abf1fb1095cdf5d6a9c01ee5ecda269df56 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Thu, 15 Oct 2026 22:29:50 +0200 Subject: [PATCH 36/58] [rspec-core] Count --parallel workers' expectations for reporters that count them rspec/rspec#305 has rspec-expectations call the configured reporter's increment_expectation_count for every expectation. In a worker that is WorkerReporter, so the counts never reached the summary. Workers now send [:expectations, n] before each unit_finished and after their after(:suite) hooks, and the coordinator replays them if its reporter has that method. Without #305 nothing calls it and nothing is sent. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/coordinator.rb | 11 +++++ rspec-core/lib/rspec/core/parallel/worker.rb | 3 ++ .../rspec/core/parallel/worker_reporter.rb | 19 ++++++++ rspec-core/spec/integration/parallel_spec.rb | 44 +++++++++++++++++++ .../rspec/core/parallel/coordinator_spec.rb | 31 ++++++++++++- .../core/parallel/worker_reporter_spec.rb | 11 +++++ .../spec/rspec/core/parallel/worker_spec.rb | 21 +++++++++ 7 files changed, 139 insertions(+), 1 deletion(-) diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 2304cba44..261c4ef43 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -204,10 +204,21 @@ def handle_control_message(worker, kind, rest) when :non_example_failure then non_example_failure(*rest) when :exiting then exiting(worker, *rest) when :loaded then loaded(worker, *rest) + when :expectations then expectations(*rest) else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." end end + # Expectations counted in a worker, for a reporter that counts them + # (`increment_expectation_count`, from an rspec-expectations that + # reports every expectation to it). + def expectations(count) + reporter = @configuration.reporter + return unless reporter.respond_to?(:increment_expectation_count) + + count.times { reporter.increment_expectation_count } + end + def non_example_failure(payload, context_description) @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 1cc087709..58cba0c55 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -136,6 +136,8 @@ def run_units message = @jobs.receive end end + # Those its `after(:suite)` hooks made, which a serial run counts too. + @reporter.forward_expectation_count true rescue SystemExit => e @@ -192,6 +194,7 @@ def exit_cause(exception, source) def run_unit(index, id, example_ids=nil) group = groups_by_id.fetch(id) passed = example_ids ? run_only(group, example_ids) : group.run(@reporter) + @reporter.forward_expectation_count @results.send([:unit_finished, index, passed ? true : false, !!@world.wants_to_quit, !!@world.non_example_failure]) end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index e30000650..2ca733ca4 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -25,6 +25,25 @@ def initialize(configuration, results, jobs, world=RSpec.world) @jobs = jobs @world = world @group_started_at = nil + @unforwarded_expectations = 0 + end + + # @private + # Called for every expectation by an rspec-expectations that counts + # them; the count reaches the coordinator through + # {#forward_expectation_count}. + def increment_expectation_count + @unforwarded_expectations += 1 + end + + # @private + # Sends what was counted since the last call, if anything: the + # coordinator adds it to its own reporter's count, if that keeps one. + def forward_expectation_count + return if @unforwarded_expectations.zero? + + @results.send([:expectations, @unforwarded_expectations]) + @unforwarded_expectations = 0 end # @private diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 70f8295a1..7151dcb20 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -228,6 +228,50 @@ def example_failed(notification) end end + # As rspec-expectations does for every expectation when its reporter counts + # them (rspec/rspec#305), the examples here call the configured reporter's + # `increment_expectation_count`, which the formatter adds to the reporter + # of the coordinating process (or of the serial run). + it 'adds up the expectations its workers count, as a serial run does, for a reporter that counts them' do + write_file_formatted 'spec/support/counted_expectations.rb', " + class CountedExpectations + RSpec::Core::Formatters.register self, :start, :dump_summary + + def initialize(output) + @output = output + end + + def start(_notification) + counted = 0 + reporter = RSpec.configuration.reporter + reporter.define_singleton_method(:increment_expectation_count) { counted += 1 } + reporter.define_singleton_method(:counted_expectations) { counted } + end + + def dump_summary(_notification) + @output.puts \"Counted \#{RSpec.configuration.reporter.counted_expectations} expectations\" + end + end + " + 4.times do |i| + write_file_formatted "spec/c#{i}_spec.rb", " + RSpec.describe('C#{i}') do + before(:context) { RSpec.configuration.reporter.increment_expectation_count } + it('counts twice') { 2.times { RSpec.configuration.reporter.increment_expectation_count } } + it('counts once') { RSpec.configuration.reporter.increment_expectation_count } + end + " + end + options = '--require ./spec/support/counted_expectations --format progress --format CountedExpectations' + + run_rspec options + expect(last_cmd_stdout).to include("8 examples, 0 failures", "Counted 16 expectations") + + run_rspec "#{options} --parallel=2" + expect(last_cmd_stdout).to include("8 examples, 0 failures", "Counted 16 expectations") + expect(last_cmd_exit_status).to eq(0) + end + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do write_file_formatted 'spec/support/my_formatter.rb', " class MyAppError < StandardError; end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index d68897553..5d624393a 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -124,7 +124,7 @@ def spawn(handle, _handles) when :dies_loading then Thread.current.kill else report_loaded(results, behaviour) if handle.loading - send_unknown_ids(results) if behaviour == :unknown_ids + send_extra_frames(results, behaviour) run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } end ensure @@ -169,6 +169,11 @@ def report_loaded(results, behaviour) results.send([:loaded, digest, failure]) end + def send_extra_frames(results, behaviour) + send_unknown_ids(results) if behaviour == :unknown_ids + results.send([:expectations, 3]) if behaviour == :counts_expectations + end + def send_unknown_ids(results) results.send([:example_started, "./nope_spec.rb[1:1]"]) results.send([:example_started, "./nope_spec.rb[1:1]"]) @@ -772,6 +777,30 @@ def turn(seconds) expect(messages).to include("2 examples, 0 failures") end + it "adds the expectations a worker counted to its reporter's count, when that keeps one" do + worker_behaviours[1] = :counts_expectations + counted = 0 + config.reporter.define_singleton_method(:increment_expectation_count) { counted += 1 } + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(counted).to eq(3) + expect(messages).not_to include("unexpected") + end + + it "ignores the expectations a worker counted when its reporter keeps no count" do + worker_behaviours[1] = :counts_expectations + allow(config.reporter).to receive(:respond_to?).and_call_original + allow(config.reporter).to receive(:respond_to?).with(:increment_expectation_count).and_return(false) + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages).not_to include("unexpected") + expect(messages).to include("2 examples, 0 failures") + end + it "reports the groups that never ran when no worker remains" do crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } groups = [crashing, RSpec.describe("Also crashing") { it("dies") { Thread.current.kill } }, passing_group("C"), passing_group("D")] diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index af894ac94..30479a51e 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -212,6 +212,17 @@ def listener.example_failed(notification) expect(jobs.first.receive).to eq([:run, 7]) end + it "forwards the expectations counted since it last did, and nothing when there were none" do + reporter.forward_expectation_count + 3.times { reporter.increment_expectation_count } + reporter.forward_expectation_count + reporter.forward_expectation_count + reporter.increment_expectation_count + reporter.forward_expectation_count + + expect(frames).to eq([[:expectations, 3], [:expectations, 1]]) + end + it "keeps the reporter's own failure bookkeeping so fail-fast works inside the worker" do RSpec.configuration.fail_fast = true group = RSpec.describe("G") do diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 02b648b43..eefd3f58a 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -165,6 +165,27 @@ def frames expect(all.map(&:first)).to include(:group_started, :example_started, :example_finished, :group_finished) end + it "forwards the expectations counted in each unit before reporting it, and those of `after(:suite)` hooks last" do + count = lambda { RSpec.configuration.reporter.increment_expectation_count } + groups.first.before(:context) { count.call } + groups.first.example("counts twice") { 2.times { count.call } } + RSpec.configuration.after(:suite) { count.call } + + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:run, 1, groups[1].id]) + jobs.last.send([:stop]) + + worker.run + + expect(frames.select { |f| [:expectations, :unit_finished, :exiting].include?(f.first) }).to eq([ + [:expectations, 3], + [:unit_finished, 0, true, false, false], + [:unit_finished, 1, false, false, false], + [:exiting, nil, nil], + [:expectations, 1] + ]) + end + it "exits at once, without running suite hooks or at_exit handlers, when its first job is not a group to run" do hooks = [] RSpec.configuration.before(:suite) { hooks << :before_suite } From dcb64792ef70f5302823737425909a01a306abc9 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Thu, 15 Oct 2026 22:55:10 +0200 Subject: [PATCH 37/58] [rspec-core] Keep expectation counts of workers that leave through `exit` From an independent review: counts were sent only before unit_finished and after the suite hooks, so an `exit` in an example or after(:suite) hook dropped the unit's and the hooks' counts, which a serial run keeps. Workers now also send them on leaving the job loop and from an ensure, ignoring a channel nobody reads, as send_exiting does. The slow-loading spec's 15 s cap is now one deadline per worker, not one per example. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/lib/rspec/core/parallel/worker.rb | 14 ++++++- rspec-core/spec/integration/parallel_spec.rb | 4 +- .../spec/rspec/core/parallel/worker_spec.rb | 38 +++++++++++++++++++ 3 files changed, 52 insertions(+), 4 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 58cba0c55..430f571c1 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -136,8 +136,6 @@ def run_units message = @jobs.receive end end - # Those its `after(:suite)` hooks made, which a serial run counts too. - @reporter.forward_expectation_count true rescue SystemExit => e @@ -148,6 +146,9 @@ def run_units send_exiting(e) end raise + ensure + # Those its `after(:suite)` hooks made, which a serial run counts too. + forward_expectation_count end # Arms the watchdog and tells the coordinator -- which then bounds the @@ -164,9 +165,18 @@ def leaving exception = $! @exit_cause = exit_cause(exception, 'the code it ran') arm_watchdog + # Those of a unit that `exit` or a signal cut short, too. + forward_expectation_count send_exiting(exception) end + # @private + def forward_expectation_count + @reporter.forward_expectation_count + rescue Errno::EPIPE, IOError + nil # the coordinator is gone or no longer reads it + end + # `[:exiting, cause, exit_status]`: sent on leaving the job loop, and # again for an `exit` that comes later. # @private diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 7151dcb20..dd1f56021 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -1000,14 +1000,14 @@ def rspec_process(*args) (File.write('loading.pid.tmp', Process.pid.to_s); File.rename('loading.pid.tmp', 'loading.pid'); sleep 600) if ENV['TEST_ENV_NUMBER'] == '2' RSpec.describe('S#{i}') do it('keeps the other worker busy past the deadline') do - give_up = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 15 + $give_up ||= Process.clock_gettime(Process::CLOCK_MONOTONIC) + 15 loading = lambda do pid = File.exist?('loading.pid') && Integer(File.read('loading.pid')) !pid || Process.kill(0, pid) rescue Errno::ESRCH false end - sleep 0.05 while loading.call && Process.clock_gettime(Process::CLOCK_MONOTONIC) < give_up + sleep 0.05 while loading.call && Process.clock_gettime(Process::CLOCK_MONOTONIC) < $give_up end end " diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index eefd3f58a..13d4ee31e 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -323,6 +323,44 @@ def leaving_frames ]) end + it "forwards the expectations of a unit `exit` cut short before saying it is leaving, and those of `after(:suite)` hooks after", :skip => jruby_rescues_exit do + exiting = RSpec.describe("Exits") do + it("exits") do + 2.times { RSpec.configuration.reporter.increment_expectation_count } + exit 3 + end + end + instance = Worker.new(2, [Unit.new(exiting)], jobs.first, results.last, + :stderr => stderr, :exit => lambda { |_| }, :exit_now => lambda { |_| }, + :trap => lambda { |*| }, :watch_coordinator => false) + allow(instance).to receive(:arm_watchdog) + RSpec.configuration.after(:suite) { RSpec.configuration.reporter.increment_expectation_count } + jobs.last.send([:run, 0, exiting.id]) + jobs.last.send([:stop]) + + expect { instance.run }.to raise_error(SystemExit) + + expect(frames.select { |f| [:expectations, :exiting].include?(f.first) }).to eq([ + [:expectations, 2], + [:exiting, "the code it ran called `exit` or `abort` (status 3)", 3], + [:expectations, 1] + ]) + end + + it "still exits normally when nothing reads the expectations its `after(:suite)` hooks counted" do + reader = results.first + RSpec.configuration.after(:suite) do + reader.close + RSpec.configuration.reporter.increment_expectation_count + end + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker.run + + expect(exits).to eq([[:exit, 0]]) + end + it "names a signal or another exception that ends the process" do expect(worker.exit_cause(SignalException.new("TERM"), "the code it ran")).to eq("it received SIGTERM") expect(worker.exit_cause(NoMemoryError.new("too much"), "the code it ran")).to eq( From 0780bdd0ef51a6998905d7b3e2b89a2505bf865c Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 00:02:13 +0200 Subject: [PATCH 38/58] [rspec-core] Keep a --parallel worker stuck in `fork` from hanging the run Found by a trial merge at load ~300: a worker deadlocked inside fork (rb_thread_atfork waiting on a mutex, before any Ruby code ran) got a group, looked like one running a long example, and the run waited until `timeout` killed it after 600 s; with its coordinator-watch thread never started, it was then orphaned. Workers now send [:started] first; until then they get no work, are killed once nothing is left, and are sent TERM, then KILL, if they miss the start deadline (the shutdown timeout). Before: the new integration spec hung until killed at 90 s. After: 3/3 passed, the stuck worker killed and both examples run by the other. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 9 +-- rspec-core/README.md | 4 ++ rspec-core/lib/rspec/core/configuration.rb | 4 +- .../lib/rspec/core/parallel/coordinator.rb | 12 +++- rspec-core/lib/rspec/core/parallel/worker.rb | 2 + .../lib/rspec/core/parallel/worker_exits.rb | 13 ++++ .../lib/rspec/core/parallel/worker_handle.rb | 5 +- .../rspec/core/parallel/worker_lifecycle.rb | 3 + rspec-core/spec/integration/parallel_spec.rb | 59 +++++++++++++++++++ .../rspec/core/parallel/coordinator_spec.rb | 47 ++++++++++++++- .../spec/rspec/core/parallel/worker_spec.rb | 4 +- 11 files changed, 150 insertions(+), 12 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index ea1150ed8..d8689b97d 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -18,10 +18,11 @@ Enhancements: parallel_tests does. A worker that finishes but does not exit is aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, so a stuck `at_exit` handler can no longer hang a run (nor can one that runs - after code under test called `exit`), and a worker that crashes fails the example it was - running instead of taking the run down, while the examples of its group it had not - started run in another worker. Code that calls `exit` ends the run with that status, as - it does serially. A top-level group tagged `:parallel_split => N` runs as N parts in + after code under test called `exit`); neither can a worker stuck in `fork`. A worker + that crashes fails the example it was running instead of taking the run down, while + the examples of its group it had not started run in another worker. Code that calls + `exit` ends the run with that status, as it does serially. A top-level group tagged + `:parallel_split => N` runs as N parts in different workers. A run stopped with SIGTERM or SIGHUP says which example each worker was running and for how long. With `--no-parallel-preload` (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec diff --git a/rspec-core/README.md b/rspec-core/README.md index 4230fa0ca..93037122c 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -372,6 +372,10 @@ with `config.error_exit_code`. The coordinator never waits unboundedly for a finished worker -- not even one whose watchdog cannot run and that ignores SIGTERM: the run waits for it about the shutdown timeout plus 11 seconds at most (a second of grace, then 5 seconds after SIGTERM and 5 after SIGKILL). +A worker that never gets past `fork` (deadlocked in a fork handler, in Ruby or +in a library) is given no work: it is killed once the other workers have run +everything, or terminated the same way once it has not reported back within +the shutdown timeout of being forked. A worker that crashes or is killed (`exit!`, a segfault, the OOM killer, ...) fails the example it was running with the reason, so the example is listed diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 145742650..65ed48bcb 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -268,7 +268,9 @@ def parallel_workers=(value) # last example group -- that is, to run its `after(:suite)` hooks and # `at_exit` handlers (default: 30). A worker that takes longer dumps the # backtraces of all its threads to stderr and is aborted, and the run - # exits with {#error_exit_code}. + # exits with {#error_exit_code}. It is also how long a worker forked + # after loading has to report back: one stuck in `fork` (a deadlock in + # a fork handler) is terminated, and the other workers run its share. # @return [Numeric] add_setting :parallel_shutdown_timeout diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 261c4ef43..5f79efffa 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -186,6 +186,7 @@ def drain(worker) def handle_message(worker, message, bytes) kind, *rest = message number = worker.number + worker_started(worker) if worker.starting case kind when :group_started then index.with_group(rest[0]) { |g| group_started(worker, g, rest[1], bytes) } when :group_finished then index.with_group(rest[0]) { |g| @sequencer.group_finished(number, g, rest[1], bytes) } @@ -205,6 +206,7 @@ def handle_control_message(worker, kind, rest) when :exiting then exiting(worker, *rest) when :loaded then loaded(worker, *rest) when :expectations then expectations(*rest) + when :started then nil else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." end end @@ -219,6 +221,12 @@ def expectations(count) count.times { reporter.increment_expectation_count } end + # Anything it sends shows it got past forking and runs RSpec's code. + def worker_started(worker) + worker.starting = false + worker.deadline = nil unless worker.loading || worker.escalation + end + def non_example_failure(payload, context_description) @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) end @@ -288,8 +296,8 @@ def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) def dispatch_work busy = @workers.count(&:busy?) @workers.each do |worker| - if worker.loading - # Nothing is left to give it once it has loaded. + if worker.loading || worker.starting + # Nothing is left to give it once it has loaded or started. dismiss_worker(worker) if worker.state == :running && nothing_left?(busy) elsif worker.idle? if !@quitting && !@queue.empty? diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 430f571c1..30f6f7c02 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -48,6 +48,8 @@ def test_env_number # Never returns: ends the process when the coordinator has no more work. def run + # First, so the coordinator knows it got past `fork`. + @results.send([:started]) watch_coordinator if @watch_coordinator prepare load_spec_files unless @example_groups diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 92dd018a4..15c5b6eaf 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -39,6 +39,8 @@ def overdue(worker) "sent SIGTERM. Raise `config.parallel_shutdown_timeout` if they legitimately need longer.") abort_unit(worker) worker.results.close + elsif worker.starting + never_started(worker) elsif worker.state == :stopping error("#{worker.description} did not exit within #{timeout} of finishing its last example group (its " \ "`after(:suite)` hooks or `at_exit` handlers may be stuck), so it was sent SIGTERM. Raise " \ @@ -107,6 +109,17 @@ def replace_exit_status(worker, exit_status, cause) report "#{worker.description}: then #{cause}, so the run exits with status #{exit_status}." unless @exit_spoiled end + # Stuck between `fork` and RSpec's code, as a fork-time deadlock (in + # Ruby, or a library's fork handler) leaves a process: it has run + # nothing, so the run's results stay complete without it. + def never_started(worker) + worker.loading = false + report "#{worker.description} never reported back after it was forked (stuck before running any of " \ + "RSpec's code, as a deadlock in a fork handler would leave it), so it was sent SIGTERM. It ran " \ + "nothing; the other workers run its share." + worker.results.close + end + # The worker can no longer be relied on, and nothing it sends from now # on is read: it would report the example a second time. What it was # running is reported as lost -- in a way that the summary, `Failed diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index decad33bf..ddf6bbd7a 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -14,6 +14,8 @@ class WorkerHandle # Forked before the spec files were loaded and has not yet reported # what it loaded. attr_accessor :loading + # Forked, and has not sent anything yet. + attr_accessor :starting def initialize(number) @number = number @@ -28,6 +30,7 @@ def initialize(number) @quit_sent = false @exiting = false @loading = false + @starting = false @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = @exit_cause = @called_exit = nil @started_example_ids = {} @last_started_group = nil @@ -105,7 +108,7 @@ def close_all end def idle? - @state == :running && @current_unit.nil? && !@loading + @state == :running && @current_unit.nil? && !@loading && !@starting end def busy? diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 6bd12a136..c1ae8f09c 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -88,6 +88,9 @@ def start_worker(number, units) @workers << worker worker.pid = @spawn.call(worker, @workers) worker.close_child_ends + worker.starting = true + # A worker forked before loading gets the loading deadline instead. + worker.deadline = now + @configuration.parallel_shutdown_timeout unless worker.loading true rescue *OUT_OF_RESOURCES => e if worker diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index dd1f56021..c1396b3a6 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -911,6 +911,65 @@ def alive?(pid) end end + # As a worker deadlocked in a fork handler (in Ruby, or a library's) is: + # it never runs any of RSpec's code, and SIGTERM does not end it. + describe 'a worker stuck before it runs any of its code' do + before do + stub_const("RSpec::Core::Parallel::Coordinator::TERM_GRACE", 0.3) + stub_const("RSpec::Core::Parallel::Coordinator::KILL_GRACE", 0.3) + # Forked workers inherit this stub. + allow_any_instance_of(RSpec::Core::Parallel::Worker).to receive(:run).and_wrap_original do |original, *args| + if original.receiver.number == 2 + trap('TERM', 'IGNORE') + sleep + end + original.call(*args) + end + end + + def write_suite(first_group_sleeps) + write_file_formatted 'spec/stuck_at_fork_spec.rb', " + RSpec.configure { |c| c.parallel_shutdown_timeout = 0.5 } + RSpec.describe('S1') { it('passes') { sleep #{first_group_sleeps} } } + RSpec.describe('S2') { it('passes') { } } + " + end + + def pids_in(output) + output.scan(/\(pid (\d+)\)/).flatten.map(&:to_i).uniq + end + + it 'gets no work, is killed once it misses its start deadline, and the other workers run everything' do + write_suite(1.5) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 --order defined spec/stuck_at_fork_spec.rb" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 10 + expect(last_cmd_stdout).to include( + "2 examples, 0 failures", + "never reported back after it was forked", + "still alive 0.3 seconds after SIGTERM; sending SIGKILL" + ) + expect(last_cmd_exit_status).to eq(0) + pids = pids_in(last_cmd_stdout) + expect(pids.size).to eq(1) + expect { Process.kill(0, pids.first) }.to raise_error(Errno::ESRCH) + end + + it 'is killed, without waiting for its deadline, once the other workers have run everything' do + write_suite(0) + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + run_rspec "--parallel=2 spec/stuck_at_fork_spec.rb" + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 5 + expect(last_cmd_stdout).to include("2 examples, 0 failures") + expect(last_cmd_stdout).not_to include("never reported back", "SIGTERM") + expect(last_cmd_exit_status).to eq(0) + end + end + it 'falls back to a serial run with a warning inside another parallel worker' do write_file_formatted 'spec/a_spec.rb', " RSpec.describe('A') { it('passes') { expect(RSpec.configuration.parallel_worker_number).to be_nil } } diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 5d624393a..e2fe7b05d 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -115,12 +115,13 @@ def spawn(handle, _handles) thread = Thread.new do Thread.current.report_on_exception = false + send_started(results, behaviour) case behaviour when :garbage then stall_after { results.io.write("garbage\n") } when :partial_frame then stall_after { results.io.write("999\nabc") } when :closes_results then stall_after { results.close } when :leaves_and_hangs then stall_after { leave_for_a_signal(jobs, results) } - when :never_loads then sleep + when :never_loads, :never_starts then sleep when :dies_loading then Thread.current.kill else report_loaded(results, behaviour) if handle.loading @@ -136,6 +137,12 @@ def spawn(handle, _handles) @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } end + # What a real worker sends first, which the fakes standing in for one send too. + def send_started(results, behaviour) + fakes = [:garbage, :partial_frame, :closes_results, :leaves_and_hangs, :never_loads, :dies_loading] + results.send([:started]) if fakes.include?(behaviour) + end + def stall_after yield sleep @@ -783,9 +790,11 @@ def turn(seconds) config.reporter.define_singleton_method(:increment_expectation_count) { counted += 1 } exit_code = coordinator([passing_group("A"), passing_group("B")]).run + # Before this spec's own expectations, which an rspec-expectations that counts them adds to. + counted_by_the_run = counted expect(exit_code).to eq(0) - expect(counted).to eq(3) + expect(counted_by_the_run).to eq(3) expect(messages).not_to include("unexpected") end @@ -928,11 +937,43 @@ def example_passed(_notification) expect(groups.drop(1).flat_map(&:examples).map { |e| e.execution_result.status }).to all(eq(:passed)) end + describe "a worker that never reports back after it was forked" do + it "gets no work, and is killed once the other worker has run everything" do + worker_behaviours[2] = :never_starts + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(0) + expect(messages).to include("2 examples, 0 failures") + expect(messages).not_to include("never reported back") + expect(process.signals.map(&:first)).to eq(["KILL"]) + end + + it "is terminated once it misses its start deadline, and what no worker ran is reported" do + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[1] = worker_behaviours[2] = :never_starts + + exit_code = coordinator([passing_group("A"), passing_group("B")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages.scan("never reported back after it was forked").size).to eq(2) + expect(messages).to include("did not run because no worker remained to run them") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM]) + end + end + it "requeues a group when its worker cannot be reached and lets another worker run it" do config.parallel_shutdown_timeout = 5 + # Its pipe closes as the first job is written to it: the write fails, + # and the worker sees the end of it, as when the coordinator is gone. spawn_with_closed_jobs = lambda do |handle, handles| pid = spawn(handle, handles) - handle.jobs.io.close_write if handle.number == 2 + if handle.number == 2 + handle.jobs.define_singleton_method(:send) do |message| + io.close_write + super(message) + end + end pid end groups = [passing_group("A"), passing_group("B"), passing_group("C")] diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 13d4ee31e..0c94adb3b 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -199,7 +199,7 @@ def frames end expect(hooks).to eq([]) - expect(frames).to eq([]) + expect(frames).to eq([[:started], [:started]]) end it "exits at once when the coordinator goes away before giving it anything" do @@ -480,6 +480,7 @@ def run_in_thread(instance) write_spec_file(dir, "RSpec.describe('Loaded here') { it('passes') { } }") thread = run_in_thread(loading_worker) + expect(results.first.receive).to eq([:started]) loaded = results.first.receive group = RSpec.world.example_groups.first expect(loaded).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world), nil]) @@ -499,6 +500,7 @@ def run_in_thread(instance) write_spec_file(dir, "RSpec.deprecate('old_thing')\nraise 'boom at load'\n") thread = run_in_thread(loading_worker) + expect(results.first.receive).to eq([:started]) kind, digest, failure = results.first.receive jobs.last.send([:stop]) expect(thread.join(10)).to be_truthy From d3e5a1c3b2ca84940a54e269627e7ea4a3264c9d Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 00:34:48 +0200 Subject: [PATCH 39/58] [rspec-core] Give a --parallel worker its first group at fork again 0780bdd0e held work back until a worker's first frame, so on a busy machine the first worker to start took every small group: two of rspec-core's parallel specs that expect each worker to run one failed (1 of 3 dogfood runs). Workers again get a group at fork; one that misses its start deadline has that group requeued as is. A reworked start deadline then hung the suite under --parallel=8 (2 of 2 runs): a late [:started] cleared the deadline fail-fast had set. Only the start deadline itself is cleared now; any later one replaces it. New spec: with the old clearing, the run did not end within 10 s. Dogfood --parallel=8: 3/3 passed, 8-12 s. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/README.md | 6 ++-- rspec-core/lib/rspec/core/configuration.rb | 2 +- .../lib/rspec/core/parallel/coordinator.rb | 6 ++-- .../lib/rspec/core/parallel/worker_exits.rb | 8 ++++- .../lib/rspec/core/parallel/worker_handle.rb | 15 +++++++-- .../rspec/core/parallel/worker_lifecycle.rb | 2 +- rspec-core/spec/integration/parallel_spec.rb | 20 +++--------- .../rspec/core/parallel/coordinator_spec.rb | 32 ++++++++++++++++--- .../rspec/core/parallel/worker_handle_spec.rb | 23 +++++++++++++ 9 files changed, 83 insertions(+), 31 deletions(-) diff --git a/rspec-core/README.md b/rspec-core/README.md index 93037122c..eb3277209 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -373,9 +373,9 @@ finished worker -- not even one whose watchdog cannot run and that ignores SIGTERM: the run waits for it about the shutdown timeout plus 11 seconds at most (a second of grace, then 5 seconds after SIGTERM and 5 after SIGKILL). A worker that never gets past `fork` (deadlocked in a fork handler, in Ruby or -in a library) is given no work: it is killed once the other workers have run -everything, or terminated the same way once it has not reported back within -the shutdown timeout of being forked. +in a library) is terminated the same way once it has not reported back within +the shutdown timeout of being forked, and the group it was given runs in +another worker. A worker that crashes or is killed (`exit!`, a segfault, the OOM killer, ...) fails the example it was running with the reason, so the example is listed diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 65ed48bcb..47453b6ee 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -270,7 +270,7 @@ def parallel_workers=(value) # backtraces of all its threads to stderr and is aborted, and the run # exits with {#error_exit_code}. It is also how long a worker forked # after loading has to report back: one stuck in `fork` (a deadlock in - # a fork handler) is terminated, and the other workers run its share. + # a fork handler) is terminated, and its group runs in another worker. # @return [Numeric] add_setting :parallel_shutdown_timeout diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 5f79efffa..e1c6fd755 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -224,7 +224,7 @@ def expectations(count) # Anything it sends shows it got past forking and runs RSpec's code. def worker_started(worker) worker.starting = false - worker.deadline = nil unless worker.loading || worker.escalation + worker.clear_start_deadline end def non_example_failure(payload, context_description) @@ -296,8 +296,8 @@ def unit_finished(worker, index, passed, wants_to_quit, non_example_failure) def dispatch_work busy = @workers.count(&:busy?) @workers.each do |worker| - if worker.loading || worker.starting - # Nothing is left to give it once it has loaded or started. + if worker.loading + # Nothing is left to give it once it has loaded. dismiss_worker(worker) if worker.state == :running && nothing_left?(busy) elsif worker.idle? if !@quitting && !@queue.empty? diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 15c5b6eaf..872ec1ff3 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -111,12 +111,18 @@ def replace_exit_status(worker, exit_status, cause) # Stuck between `fork` and RSpec's code, as a fork-time deadlock (in # Ruby, or a library's fork handler) leaves a process: it has run - # nothing, so the run's results stay complete without it. + # nothing, so the group it was given goes to another worker as is. def never_started(worker) worker.loading = false report "#{worker.description} never reported back after it was forked (stuck before running any of " \ "RSpec's code, as a deadlock in a fork handler would leave it), so it was sent SIGTERM. It ran " \ "nothing; the other workers run its share." + if (index = worker.current_unit) && accepting_work? + @unit_states[index] = :pending + @queue.unshift(index) + worker.current_unit = nil + end + abort_unit(worker) worker.results.close end diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index ddf6bbd7a..81764994a 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -30,7 +30,7 @@ def initialize(number) @quit_sent = false @exiting = false @loading = false - @starting = false + @starting = @start_deadline = false @pid = @current_unit = @deadline = @deadline_set_at = @escalation = @exit_status = @exit_cause = @called_exit = nil @started_example_ids = {} @last_started_group = nil @@ -40,6 +40,17 @@ def initialize(number) def deadline=(value) @deadline = value @deadline_set_at = value && Parallel.monotonic_time + @start_deadline = false + end + + # The deadline for its first frame, which any other deadline replaces. + def start_deadline=(value) + self.deadline = value + @start_deadline = true + end + + def clear_start_deadline + self.deadline = nil if @start_deadline end # Moves the deadline back without changing when it was set. @@ -108,7 +119,7 @@ def close_all end def idle? - @state == :running && @current_unit.nil? && !@loading && !@starting + @state == :running && @current_unit.nil? && !@loading end def busy? diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index c1ae8f09c..2783659cb 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -90,7 +90,7 @@ def start_worker(number, units) worker.close_child_ends worker.starting = true # A worker forked before loading gets the loading deadline instead. - worker.deadline = now + @configuration.parallel_shutdown_timeout unless worker.loading + worker.start_deadline = now + @configuration.parallel_shutdown_timeout unless worker.loading true rescue *OUT_OF_RESOURCES => e if worker diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index c1396b3a6..2a9755fa0 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -927,10 +927,10 @@ def alive?(pid) end end - def write_suite(first_group_sleeps) + def write_suite write_file_formatted 'spec/stuck_at_fork_spec.rb', " RSpec.configure { |c| c.parallel_shutdown_timeout = 0.5 } - RSpec.describe('S1') { it('passes') { sleep #{first_group_sleeps} } } + RSpec.describe('S1') { it('passes') { } } RSpec.describe('S2') { it('passes') { } } " end @@ -939,8 +939,8 @@ def pids_in(output) output.scan(/\(pid (\d+)\)/).flatten.map(&:to_i).uniq end - it 'gets no work, is killed once it misses its start deadline, and the other workers run everything' do - write_suite(1.5) + it 'is terminated, then killed, once it misses its start deadline, and its group runs in another worker' do + write_suite started = Process.clock_gettime(Process::CLOCK_MONOTONIC) run_rspec "--parallel=2 --order defined spec/stuck_at_fork_spec.rb" @@ -956,18 +956,6 @@ def pids_in(output) expect(pids.size).to eq(1) expect { Process.kill(0, pids.first) }.to raise_error(Errno::ESRCH) end - - it 'is killed, without waiting for its deadline, once the other workers have run everything' do - write_suite(0) - started = Process.clock_gettime(Process::CLOCK_MONOTONIC) - - run_rspec "--parallel=2 spec/stuck_at_fork_spec.rb" - - expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 5 - expect(last_cmd_stdout).to include("2 examples, 0 failures") - expect(last_cmd_stdout).not_to include("never reported back", "SIGTERM") - expect(last_cmd_exit_status).to eq(0) - end end it 'falls back to a serial run with a warning inside another parallel worker' do diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index e2fe7b05d..13ad74b8a 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -121,7 +121,7 @@ def spawn(handle, _handles) when :partial_frame then stall_after { results.io.write("999\nabc") } when :closes_results then stall_after { results.close } when :leaves_and_hangs then stall_after { leave_for_a_signal(jobs, results) } - when :never_loads, :never_starts then sleep + when :never_loads, :never_starts, :starts_once_asked_to_quit then stall(behaviour, handle, results) when :dies_loading then Thread.current.kill else report_loaded(results, behaviour) if handle.loading @@ -143,6 +143,15 @@ def send_started(results, behaviour) results.send([:started]) if fakes.include?(behaviour) end + # One that reports back only once it has been asked to stop early stalls then. + def stall(behaviour, handle, results) + if behaviour == :starts_once_asked_to_quit + sleep 0.01 until handle.quit_sent + results.send([:started]) + end + sleep + end + def stall_after yield sleep @@ -514,6 +523,20 @@ def example_failed(_notification) describe "a worker that is asked to stop early but does not" do before { config.parallel_shutdown_timeout = 0.5 } + it "is terminated after the shutdown timeout even when it reports back only after it was asked to stop" do + config.fail_fast = true + worker_behaviours[2] = :starts_once_asked_to_quit + groups = [failing_group("Fails"), passing_group("Held by the late worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + run = Thread.new { coordinator(groups).run } + + expect(run.join(10)).to be_truthy + expect(run.value).to eq(config.failure_exit_code) + expect(process.signals.map(&:first)).to eq(["TERM"]) + expect(messages).to include("did not stop within 0.5 seconds of being asked to stop early") + end + it "is terminated after the shutdown timeout so a cancelled run still ends" do config.fail_fast = true worker_behaviours[2] = :partial_frame @@ -938,15 +961,16 @@ def example_passed(_notification) end describe "a worker that never reports back after it was forked" do - it "gets no work, and is killed once the other worker has run everything" do + it "is terminated once it misses its start deadline, and the group it was given runs in another worker" do + config.parallel_shutdown_timeout = 0.05 worker_behaviours[2] = :never_starts exit_code = coordinator([passing_group("A"), passing_group("B")]).run expect(exit_code).to eq(0) expect(messages).to include("2 examples, 0 failures") - expect(messages).not_to include("never reported back") - expect(process.signals.map(&:first)).to eq(["KILL"]) + expect(messages.scan("never reported back after it was forked").size).to eq(1) + expect(process.signals.map(&:first)).to eq(["TERM"]) end it "is terminated once it misses its start deadline, and what no worker ran is reported" do diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb index d84a1b134..51e21dabe 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -9,5 +9,28 @@ module RSpec::Core::Parallel expect { WorkerHandle.new(1) }.to raise_error(Errno::EMFILE) expect(first.map(&:closed?)).to eq([true, true]) end + + describe "the deadline for its first frame" do + let(:handle) { WorkerHandle.new(1) } + + after { handle.close_all } + + it "is cleared by its first frame, postponed or not" do + handle.start_deadline = 5.0 + handle.postpone(1.0) + handle.clear_start_deadline + + expect(handle.deadline).to be_nil + end + + it "is replaced by any other deadline, which its first frame then leaves alone" do + handle.start_deadline = 5.0 + handle.deadline = 2.0 # asked to stop early before it reported back, say + + handle.clear_start_deadline + + expect(handle.deadline).to eq(2.0) + end + end end end From 456bb51c445fc0e50656a26e6758ed5c7510d70a Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 00:37:54 +0200 Subject: [PATCH 40/58] [rspec-core] Say whether a worker stuck in `fork` has its group run elsewhere From an independent review: when the run was stopping early, the report still said the other workers would run the stuck worker's share, but its group was not handed on. It now says the group runs in another worker, or that it did not run. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/worker_exits.rb | 12 ++++++++++-- .../spec/rspec/core/parallel/coordinator_spec.rb | 15 +++++++++++++++ 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 872ec1ff3..976b554a2 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -114,10 +114,12 @@ def replace_exit_status(worker, exit_status, cause) # nothing, so the group it was given goes to another worker as is. def never_started(worker) worker.loading = false + index = worker.current_unit + requeue = index && accepting_work? report "#{worker.description} never reported back after it was forked (stuck before running any of " \ "RSpec's code, as a deadlock in a fork handler would leave it), so it was sent SIGTERM. It ran " \ - "nothing; the other workers run its share." - if (index = worker.current_unit) && accepting_work? + "nothing#{never_started_outcome(index, requeue)}." + if requeue @unit_states[index] = :pending @queue.unshift(index) worker.current_unit = nil @@ -126,6 +128,12 @@ def never_started(worker) worker.results.close end + def never_started_outcome(index, requeue) + return "" unless index + + requeue ? "; the group it was given runs in another worker" : "; the group it was given did not run" + end + # The worker can no longer be relied on, and nothing it sends from now # on is read: it would report the example a second time. What it was # running is reported as lost -- in a way that the summary, `Failed diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 13ad74b8a..8793922a2 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -970,9 +970,24 @@ def example_passed(_notification) expect(exit_code).to eq(0) expect(messages).to include("2 examples, 0 failures") expect(messages.scan("never reported back after it was forked").size).to eq(1) + expect(messages).to include("the group it was given runs in another worker") expect(process.signals.map(&:first)).to eq(["TERM"]) end + it "says the group it was given did not run when the run stops early" do + config.parallel_shutdown_timeout = 0.05 + config.fail_fast = true + worker_behaviours[2] = :never_starts + groups = [failing_group("Fails"), passing_group("Held by the stuck worker")] + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(groups).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages).to include("never reported back after it was forked", "the group it was given did not run") + expect(messages).to include("1 example, 1 failure") + end + it "is terminated once it misses its start deadline, and what no worker ran is reported" do config.parallel_shutdown_timeout = 0.05 worker_behaviours[1] = worker_behaviours[2] = :never_starts From b6cd0a362ce48a623fe96592505f796b20bc0f84 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 01:05:25 +0200 Subject: [PATCH 41/58] [rspec-core] Keep fake --parallel workers alive when dismissed before they start Fork CI failed one coordinator spec on 4 of 44 jobs: a fake worker whose thread started after `dismiss` had closed its pipe raised sending [:started] and died, so only 1 of 2 workers "survived SIGKILL". The fakes now ignore a pipe nobody reads, as the processes they stand for would go on. With their start delayed 0.05 s: before 5/5 failed, after 5/5 passed. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/rspec/core/parallel/coordinator_spec.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 8793922a2..7efcfff40 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -137,10 +137,14 @@ def spawn(handle, _handles) @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } end - # What a real worker sends first, which the fakes standing in for one send too. + # What a real worker sends first, which the fakes standing in for one send + # too -- and then go on as they would, even if nothing reads it any more + # (dismissed before their thread got going, say). def send_started(results, behaviour) fakes = [:garbage, :partial_frame, :closes_results, :leaves_and_hangs, :never_loads, :dies_loading] results.send([:started]) if fakes.include?(behaviour) + rescue Errno::EPIPE, IOError + nil end # One that reports back only once it has been asked to stop early stalls then. From 7c924a306b18171aefc15ab6064edc0931e4e92f Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 01:17:42 +0200 Subject: [PATCH 42/58] [rspec-core] Keep a worker spec from counting its own expectations under #305 On a trial merge with rspec/rspec#305, the spec that checks a loading worker sends nothing else got [:expectations, 1]: its own `expect`, made while the worker ran, counted on the worker's reporter. It now asserts once the worker has finished. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/rspec/core/parallel/worker_spec.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 0c94adb3b..2ffa773e9 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -500,11 +500,14 @@ def run_in_thread(instance) write_spec_file(dir, "RSpec.deprecate('old_thing')\nraise 'boom at load'\n") thread = run_in_thread(loading_worker) - expect(results.first.receive).to eq([:started]) + # Expectations made while the worker runs would count as its own + # with an rspec-expectations that counts them (rspec/rspec#305). + started = results.first.receive kind, digest, failure = results.first.receive jobs.last.send([:stop]) expect(thread.join(10)).to be_truthy + expect(started).to eq([:started]) expect([kind, digest]).to eq([:loaded, RSpec::Core::Parallel.suite_digest(RSpec.world)]) expect(failure).to match(/\AAn error occurred while loading .*loaded_spec\.rb\. RuntimeError: boom at load\z/) expect(frames).to eq([]) From b39bb9af6a5a226ed8364bff1c79f974bf8bf6a0 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 01:49:41 +0200 Subject: [PATCH 43/58] [rspec-core] Let `--bisect` find its load path on Windows Bisect started its runs with `-I` paths joined by `:`, which splits `C:/...` on Windows: 8 bisect specs failed there. The rake task spec for a clean environment now passes whether or not ANSICON is active. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/lib/rspec/core/bisect/shell_command.rb | 2 +- .../spec/rspec/core/bisect/shell_command_spec.rb | 12 ++++++------ rspec-core/spec/rspec/core/rake_task_spec.rb | 3 ++- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/rspec-core/lib/rspec/core/bisect/shell_command.rb b/rspec-core/lib/rspec/core/bisect/shell_command.rb index 483a7b291..a77980aa4 100644 --- a/rspec-core/lib/rspec/core/bisect/shell_command.rb +++ b/rspec-core/lib/rspec/core/bisect/shell_command.rb @@ -100,7 +100,7 @@ def parsed_original_cli_options end def load_path - @load_path ||= "-I#{$LOAD_PATH.map { |p| escape(p) }.join(':')}" + @load_path ||= "-I#{$LOAD_PATH.map { |p| escape(p) }.join(File::PATH_SEPARATOR)}" end # Path to the currently running Ruby executable, borrowed from Rake: diff --git a/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb b/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb index 850b35811..e2453a8f7 100644 --- a/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb +++ b/rspec-core/spec/rspec/core/bisect/shell_command_spec.rb @@ -112,27 +112,27 @@ def expect_formatters_to_be_excluded it 'includes the current load path as an option to `ruby`, not as an option to `rspec`' do cmd = command_for([], :load_path => %W[ lp/foo lp/bar ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'lp/foo':'lp/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'lp/foo'#{File::PATH_SEPARATOR}'lp/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include("-Ilp/foo:lp/bar").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Ilp/foo#{File::PATH_SEPARATOR}lp/bar").then_include(RSpec::Core.path_to_executable) end end it 'escapes the load path entries' do cmd = command_for([], :load_path => ['l p/foo', 'l p/bar' ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'l p/foo':'l p/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'l p/foo'#{File::PATH_SEPARATOR}'l p/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include('-Il\ p/foo:l\ p/bar').then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Il\\ p/foo#{File::PATH_SEPARATOR}l\\ p/bar").then_include(RSpec::Core.path_to_executable) end end it 'supports Pathnames in the load path' do cmd = command_for([], :load_path => [Pathname('l p/foo'), Pathname('l p/bar') ]) if uses_quoting_for_escaping? - expect(cmd).to first_include("-I'l p/foo':'l p/bar'").then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-I'l p/foo'#{File::PATH_SEPARATOR}'l p/bar'").then_include(RSpec::Core.path_to_executable) else - expect(cmd).to first_include('-Il\ p/foo:l\ p/bar').then_include(RSpec::Core.path_to_executable) + expect(cmd).to first_include("-Il\\ p/foo#{File::PATH_SEPARATOR}l\\ p/bar").then_include(RSpec::Core.path_to_executable) end end end diff --git a/rspec-core/spec/rspec/core/rake_task_spec.rb b/rspec-core/spec/rspec/core/rake_task_spec.rb index da8b8e1f9..de26b6886 100644 --- a/rspec-core/spec/rspec/core/rake_task_spec.rb +++ b/rspec-core/spec/rspec/core/rake_task_spec.rb @@ -161,7 +161,8 @@ def silence_output(&block) it "removes the environment variables", :skip => RSpec::Support::Ruby.jruby? do with_env_vars 'MY_ENV' => 'ABC' do if RSpec::Support::OS.windows? - essential_shell_variables = /\["ANSICON", "ANSICON_DEF", "HOME", "TMPDIR", "USER"\]/ + # ANSICON's variables are there only when it is active in the shell. + essential_shell_variables = /\[(?:"ANSICON", "ANSICON_DEF", )?"HOME", "TMPDIR", "USER"\]/ else essential_shell_variables = /\["PWD"(?:, "SHLVL")?(?:, "_")?(?:, "__CF_USER_TEXT_ENCODING")?\]/ end From de31530dcbd78c5717e727b03f32807fab293079 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 01:57:22 +0200 Subject: [PATCH 44/58] [CI] Run each library's specs in the Windows job From the repository's root `rspec` found no examples, so the job passed without running any. In each library's directory, as the Linux jobs run: rspec-core 2,465 / 0 failures, rspec-expectations 2,365 / 0, rspec-mocks 1,740 / 0, rspec-support 595 / 0 (Ruby 3.4, windows-latest). Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .github/workflows/rspec.yml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/.github/workflows/rspec.yml b/.github/workflows/rspec.yml index b898cb180..4ce1fcbd5 100644 --- a/.github/workflows/rspec.yml +++ b/.github/workflows/rspec.yml @@ -85,6 +85,12 @@ jobs: ruby: - '3.4' fail-fast: false + # In the library's directory, as the other jobs: from the repository's + # root `rspec` finds no examples. + defaults: + run: + shell: bash + working-directory: ${{ inputs.library }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -92,6 +98,6 @@ jobs: - uses: ruby/setup-ruby@95ef2b042f9d7a56d8268cba8559e2842e2ad01b # v1.321.0 with: ruby-version: ${{ matrix.ruby }} - bundler-cache: true - run: choco install ansicon + - run: ../script/bundle - run: bundle exec rspec --backtrace From d0a3f680a1ecd7ff7a2d64b40e6988dfe75b0ce4 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 01:57:22 +0200 Subject: [PATCH 45/58] [rspec-core] Show formatters the metadata a parallel worker's example changed Keys an example, its hooks or a worker listener set, changed in place or deleted reach the coordinator's copy before formatters see the example (screenshot paths, retry counts). RSpec's own keys stay the coordinator's; a value that cannot be marshaled is named in a warning once per key and the formatters see the value from before the example. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/coordinator.rb | 17 +++++ rspec-core/lib/rspec/core/parallel/results.rb | 10 +++ .../rspec/core/parallel/worker_reporter.rb | 61 +++++++++++++++- rspec-core/spec/integration/parallel_spec.rb | 43 +++++++++++ .../spec/rspec/core/parallel/results_spec.rb | 34 +++++++++ .../core/parallel/worker_reporter_spec.rb | 71 +++++++++++++++++++ 6 files changed, 234 insertions(+), 2 deletions(-) create mode 100644 rspec-core/spec/rspec/core/parallel/results_spec.rb diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index e1c6fd755..15fa55ce4 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -248,10 +248,27 @@ def example_finished(worker, example_id, payload, bytes) index.with_example(example_id) do |example| worker.current_example = nil record_result(example, payload) + warn_unsent_metadata(worker, payload[:metadata]) @sequencer.example_finished(worker.number, example, bytes) end end + # Once per metadata key. + def warn_unsent_metadata(worker, changes) + unsent = changes && changes[:unsent] + return unless unsent + + @unsent_metadata ||= {} + unsent.each do |key, class_name| + next if @unsent_metadata.key?(key) + + @unsent_metadata[key] = true + RSpec.warning "#{worker.description} set example metadata #{key} to a #{class_name}, which cannot be " \ + "marshaled to reach the formatters, so they see the value it had before the example ran. " \ + "Later values of #{key} that cannot be marshaled are not reported", :call_site => nil + end + end + def index @index ||= SuiteIndex.new(@world) { |warning| report warning } end diff --git a/rspec-core/lib/rspec/core/parallel/results.rb b/rspec-core/lib/rspec/core/parallel/results.rb index 1474a2367..b0dae7af6 100644 --- a/rspec-core/lib/rspec/core/parallel/results.rb +++ b/rspec-core/lib/rspec/core/parallel/results.rb @@ -23,6 +23,7 @@ def apply(example) payload = @payloads.delete(example) return unless payload + apply_metadata(example.metadata, payload[:metadata]) example.assign_execution_result( payload.merge( :started_at => Parallel.time_from_wire(payload[:started_at]), @@ -33,6 +34,15 @@ def apply(example) ) end + # What the example's run in a worker did to its metadata, done to the + # coordinator's copy before formatters are told about it. + def apply_metadata(metadata, changes) + return unless changes + + changes.fetch(:set, {}).each { |key, value| metadata[key] = value } + changes.fetch(:deleted, []).each { |key| metadata.delete(key) } + end + def apply_all @payloads.keys.each { |example| apply(example) } end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 2ca733ca4..3969b0f59 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -18,6 +18,10 @@ class WorkerReporter < Reporter EXAMPLE_OUTCOMES = [:example_passed, :example_failed, :example_pending].freeze RUN_LIFECYCLE = [:seed, :start, :stop, :start_dump, :dump_pending, :dump_failures, :deprecation_summary, :dump_profile, :dump_summary, :close].freeze + # Example metadata that is not compared before and after the example: + # RSpec's own, which the coordinator keeps, and what the result + # payload carries anyway. + NOT_COMPARED = (Metadata::RESERVED_KEYS + [:pending, :skip, :extra_failure_lines]).to_h { |key| [key, true] }.freeze def initialize(configuration, results, jobs, world=RSpec.world) super(configuration) @@ -26,6 +30,7 @@ def initialize(configuration, results, jobs, world=RSpec.world) @world = world @group_started_at = nil @unforwarded_expectations = 0 + @metadata_before = nil end # @private @@ -66,7 +71,7 @@ def notify(event, notification) case event when :example_group_started then forward_group_started(notification.group) when :example_group_finished then forward_group_finished(notification.group) - when :example_started then @results.send([:example_started, notification.example.id]) + when :example_started then forward_example_started(notification.example) when *EXAMPLE_OUTCOMES then forward_example_finished(notification.example) when :example_finished, *RUN_LIFECYCLE then nil when *FORWARDED_AS_PAYLOAD then forward(event, notification) @@ -90,6 +95,11 @@ def forward_group_finished(group) @results.send([:group_finished, group.id, duration]) end + def forward_example_started(example) + @metadata_before = metadata_snapshot(example.metadata) + @results.send([:example_started, example.id]) + end + def forward_example_finished(example) @results.send([:example_finished, example.id, result_payload(example)]) poll_for_quit @@ -142,10 +152,57 @@ def result_payload(example) :pending_exception => RemoteException.payload_for(result.pending_exception), :pending => Parallel.marshal_safe(metadata[:pending], &:to_s), :skip => Parallel.marshal_safe(metadata[:skip], &:to_s), - :extra_failure_lines => Parallel.marshal_safe(metadata[:extra_failure_lines]) { |lines| Array(lines).map(&:to_s) } + :extra_failure_lines => Parallel.marshal_safe(metadata[:extra_failure_lines]) { |lines| Array(lines).map(&:to_s) }, + :metadata => metadata_changes(metadata) }.merge(descriptions(metadata)) end + # Each value with its hash, so that a value changed in place counts as + # changed too. + def metadata_snapshot(metadata) + metadata.each_with_object({}) do |(key, value), snapshot| + snapshot[key] = [value, hash_of(value)] unless NOT_COMPARED.key?(key) + end + end + + # @return [Hash, nil] what the example, its hooks or a worker listener + # did to its metadata: keys set (`:set`) or deleted (`:deleted`), + # and those set to values that cannot be marshaled (`:unsent`, the + # values' class names), which the coordinator leaves as they were. + def metadata_changes(metadata) + before = @metadata_before || {} + @metadata_before = nil + changes = {} + metadata.each do |key, value| + next if NOT_COMPARED.key?(key) + + old = before[key] + next if old && old[0].equal?(value) && old[1] == hash_of(value) + + if marshalable?([key, value]) + (changes[:set] ||= {})[key] = value + else + (changes[:unsent] ||= {})[key.inspect] = value.class.name + end + end + deleted = before.keys.reject { |key| metadata.key?(key) || !marshalable?(key) } + changes[:deleted] = deleted unless deleted.empty? + changes.empty? ? nil : changes + end + + def marshalable?(value) + Marshal.dump(value) + true + rescue StandardError + false + end + + def hash_of(value) + value.hash + rescue StandardError + nil + end + # As this worker has them: generated by the matcher of a one-liner # (`it { is_expected.to ... }`) once the example ran, or computed # while the worker loaded the spec files itself. diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 2a9755fa0..9f44197d8 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -272,6 +272,49 @@ def dump_summary(_notification) expect(last_cmd_exit_status).to eq(0) end + it 'shows formatters the metadata an example set, changed or deleted in a worker, as a serial run does' do + write_file_formatted 'spec/support/metadata_formatter.rb', " + class MetadataFormatter + RSpec::Core::Formatters.register self, :example_passed + + def initialize(output) + @output = output + end + + def example_passed(notification) + metadata = notification.example.metadata + @output.puts \"META \#{notification.example.description}: \#{metadata[:screenshot].inspect} \" \\ + \"\#{metadata[:tags].inspect} \#{metadata.key?(:gone)} \#{metadata[:callback].class}\" + end + end + " + 4.times do |i| + write_file_formatted "spec/m#{i}_spec.rb", " + RSpec.describe('M#{i}', :gone => true) do + it('e#{i}', :tags => ['a']) do |ex| + ex.metadata[:screenshot] = 'shot-#{i}.png' + ex.metadata[:tags] << 'b' + ex.metadata.delete(:gone) + end + it('callback#{i}') { |ex| ex.metadata[:callback] = lambda {} } + end + " + end + options = '--require ./spec/support/metadata_formatter --format progress --format MetadataFormatter' + + run_rspec options + serial = last_cmd_stdout.scan(/META .*$/).sort + expect(serial.size).to eq(8) + expect(serial.first).to eq("META callback0: nil nil true Proc") + expect(serial).to include("META e3: \"shot-3.png\" [\"a\", \"b\"] false NilClass") + + run_rspec "#{options} --parallel=2" + parallel = last_cmd_stdout.scan(/META .*$/).sort + expect(parallel).to eq(serial.map { |line| line.sub(/ Proc$/, ' NilClass') }) + expect(last_cmd_stderr.scan(/set example metadata :callback to a Proc/).size).to eq(1) + expect(last_cmd_exit_status).to eq(0) + end + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do write_file_formatted 'spec/support/my_formatter.rb', " class MyAppError < StandardError; end diff --git a/rspec-core/spec/rspec/core/parallel/results_spec.rb b/rspec-core/spec/rspec/core/parallel/results_spec.rb new file mode 100644 index 000000000..7bc024545 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/results_spec.rb @@ -0,0 +1,34 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe Results do + let(:example) do + RSpec.describe("G", :gone => true, :kept => "k") { it("e", :tags => ["a"]) {} }.examples.first + end + let(:results) { Results.new } + + def payload(metadata) + now = RSpec::Core::Parallel.time_to_wire(::Time.now) + { :status => :passed, :started_at => now, :finished_at => now, :run_time => 0.0, :metadata => metadata } + end + + it "applies the metadata the worker's run set and deleted when it applies the result" do + results.record(example, payload(:set => { :tags => ["a", "b"], :screenshot => "x.png" }, :deleted => [:gone])) + expect(example.metadata).to include(:gone => true, :tags => ["a"]) + + results.apply(example) + + expect(example.metadata).to include(:tags => ["a", "b"], :screenshot => "x.png", :kept => "k") + expect(example.metadata).not_to have_key(:gone) + expect(example.execution_result.status).to eq(:passed) + end + + it "leaves the metadata alone when the worker's run changed none, or only sent names of unsent values" do + before = example.metadata.dup + results.record(example, payload(:unsent => { ":callback" => "Proc" })) + results.apply(example) + + expect(example.metadata.reject { |key, _| key == :execution_result }).to eq(before.reject { |key, _| key == :execution_result }) + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index 30479a51e..b4a3fabdc 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -191,6 +191,77 @@ def listener.example_failed(notification) expect(payload[:extra_failure_lines]).to eq(["Ref: added in the worker"]) end + it "forwards the metadata an example set, changed in place or deleted, and nothing for one that changed none" do + group = RSpec.describe("G", :kept => "k", :gone => true) do + it("changes", :tags => ["a"], :same => "s") do |ex| + ex.metadata[:added] = { :path => "shot.png" } + ex.metadata[:tags] << "b" + ex.metadata[:same] = ex.metadata[:same] + ex.metadata.delete(:gone) + end + it("changes nothing") {} + end + group.run(reporter) + + payloads = frames.select { |f| f.first == :example_finished }.to_h { |f| [f.last[:description], f.last[:metadata]] } + changed, unchanged = payloads.values_at("changes", "changes nothing") + expect(changed).to eq(:set => { :added => { :path => "shot.png" }, :tags => ["a", "b"] }, :deleted => [:gone]) + expect(unchanged).to be_nil + end + + it "leaves RSpec's own metadata and what the payload carries anyway out of the comparison" do + group = RSpec.describe("G") do + it("skips") do |ex| + ex.metadata[:extra_failure_lines] = ["more"] + ex.metadata[:last_run_status] = "changed" + skip "later" + end + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to be_nil + expect(payload).to include(:skip => true, :pending_message => "later", :extra_failure_lines => ["more"]) + end + + it "names metadata set to values that cannot be marshaled instead of sending them" do + group = RSpec.describe("G") do + it("sets a proc") { |ex| ex.metadata[:callback] = lambda {} } + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to eq(:unsent => { ":callback" => "Proc" }) + end + + it "copes with values whose `hash` raises and with deleted keys that cannot be marshaled" do + unhashable = Object.new + def unhashable.hash + raise "no hash" + end + anonymous_key = Class.new + group = RSpec.describe("G", anonymous_key => true, :odd => unhashable) do + it("deletes") { |ex| ex.metadata.delete(anonymous_key) } + end + group.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to be_nil + end + + it "forwards metadata a worker listener sets once the example finished" do + listener = Object.new + def listener.example_passed(notification) + notification.example.metadata[:screenshot] = "after.png" + end + reporter.register_listener(listener, :example_passed) + + RSpec.describe("G") { it("passes") {} }.run(reporter) + + payload = frames.find { |f| f.first == :example_finished }.last + expect(payload[:metadata]).to eq(:set => { :screenshot => "after.png" }) + end + it "sets the world's `wants_to_quit` when the coordinator asks it to quit between examples" do jobs.last.send([:quit]) group = RSpec.describe("G") do From 23711973eb2d290042a42d2e0a1c9e5b623e9fc3 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:12:08 +0200 Subject: [PATCH 46/58] [rspec-core] Show what --parallel workers print with the group that printed it A worker's standard output and error go to files in a private temporary directory; before each frame it says how far they reach, and the coordinator shows the new bytes in the worker's sequenced stream, so `puts`, warnings and subprocess output land under their group as in a serial run instead of between progress dots. What is left once a worker has gone (a crash report, `at_exit` output) is read then. Workers loading the spec files themselves skip their loading output, which the coordinator already printed. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 9 +- rspec-core/README.md | 26 ++++- .../lib/rspec/core/parallel/coordinator.rb | 47 +++++++- .../lib/rspec/core/parallel/sequencer.rb | 14 ++- rspec-core/lib/rspec/core/parallel/worker.rb | 20 +++- .../lib/rspec/core/parallel/worker_handle.rb | 8 +- .../rspec/core/parallel/worker_lifecycle.rb | 5 +- .../lib/rspec/core/parallel/worker_output.rb | 107 ++++++++++++++++++ rspec-core/spec/integration/parallel_spec.rb | 78 ++++++++++--- .../rspec/core/parallel/coordinator_spec.rb | 77 ++++++++++++- .../rspec/core/parallel/sequencer_spec.rb | 50 +++++++- .../rspec/core/parallel/worker_output_spec.rb | 91 +++++++++++++++ .../spec/rspec/core/parallel/worker_spec.rb | 54 +++++++++ 13 files changed, 551 insertions(+), 35 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/worker_output.rb create mode 100644 rspec-core/spec/rspec/core/parallel/worker_output_spec.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index d8689b97d..4433ddda9 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -8,10 +8,11 @@ Enhancements: Formatters run in the coordinating process only and receive one coherent stream of notifications with real `Example` objects -- results, timing, pending details, exceptions rebuilt from their class name, message, backtrace and cause chain, messages, - deprecations and marshalable custom notifications (of the metadata changed while an - example runs only its description, as a one-liner generates it, `:extra_failure_lines`, - `:pending` and `:skip` are forwarded) -- so existing formatters keep working and the run - has one seed, one summary, one profile and one `.rspec_status` file. Listeners + deprecations, marshalable custom notifications and the metadata an example set, changed + or deleted -- so existing formatters keep working and the run has one seed, one summary, + one profile and one `.rspec_status` file. What workers print (also from processes they + start, crash reports and `at_exit` handlers) appears with the example group that printed + it, as in a serial run. Listeners registered inside a worker, e.g. in a `before(:suite)` hook, still see that worker's example notifications. Workers run the suite hooks once each, expose `RSpec::Core::Configuration#parallel_worker_number` and set `TEST_ENV_NUMBER` the way diff --git a/rspec-core/README.md b/rspec-core/README.md index eb3277209..0378824a0 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -352,12 +352,26 @@ exceptions (class name, message, backtrace, cause chain and aggregated failures; rebuilt as a subclass of the original class when that class exists in the coordinating process) and the group and example notifications in the order the worker produced them, plus `reporter.message`, deprecations and -custom notifications whose options can be marshaled. Of the metadata a worker -changes while running an example, only its description (generated by the -matcher of a one-liner like `it { is_expected.to eq(5) }`), -`:extra_failure_lines`, `:pending` and `:skip` are forwarded, and anything -workers write to `$stdout`/`$stderr` -directly is not sequenced with the formatter output. Listeners registered from +custom notifications whose options can be marshaled. What an example, its +hooks or a worker listener do to the example's metadata -- keys set, changed in +place or deleted, and the description the matcher of a one-liner like +`it { is_expected.to eq(5) }` generates -- reaches the coordinating process +before formatters see the example. RSpec's own keys stay the coordinating +process's; a value that cannot be marshaled is named in a warning, once per +key, and formatters see the value it had before the example. Changes to a +group's metadata are not forwarded. + +What workers print -- `puts` in an example, warnings, the output of processes +they start, a crash report, `at_exit` handlers -- appears on the coordinating +process's standard output and error with the example group that printed it, +as in a serial run: workers write it to files in a private temporary +directory, which the coordinating process reads and then removes. So a +worker's standard output and error are not terminals (`$stdout.tty?` is +false), what an example prints appears once the worker next reports (when the +example finishes, at the latest), and an interactive debugger (`binding.irb`, +`debug`) needs a serial run. With `--no-parallel-preload`, what workers print +while they load the spec files is left out: the coordinating process printed +the same when it loaded them. Listeners registered from inside a worker (for example with `reporter.register_listener` in a `before(:suite)` hook) are notified of that worker's example and group notifications before they are forwarded; run-level notifications (`start`, diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 15fa55ce4..87497ac92 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -6,6 +6,7 @@ RSpec::Support.require_rspec_core "parallel/sequencer" RSpec::Support.require_rspec_core "parallel/results" RSpec::Support.require_rspec_core "parallel/suite_index" +RSpec::Support.require_rspec_core "parallel/worker_output" RSpec::Support.require_rspec_core "parallel/worker_handle" RSpec::Support.require_rspec_core "parallel/worker_lifecycle" RSpec::Support.require_rspec_core "parallel/worker_exits" @@ -59,6 +60,7 @@ def initialize(runner, worker_count, options={}) @reap_due = true @next_reap_at = 0.0 @last_turn_at = nil + @output_dir = nil end # The groups to run. Workers forked before loading that are more than @@ -80,7 +82,7 @@ def run return run_serially if @workers.empty? passed = reporter.report(@world.example_count(@example_groups)) do - @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result)) + @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result), :output => method(:write_output)) begin @last_turn_at = now @@ -107,6 +109,7 @@ def run ensure terminate_remaining_workers Parallel.restore_signals(@signal_handlers) + WorkerOutput.remove_directory(@output_dir) end private @@ -206,6 +209,7 @@ def handle_control_message(worker, kind, rest) when :exiting then exiting(worker, *rest) when :loaded then loaded(worker, *rest) when :expectations then expectations(*rest) + when :output then output(worker, *rest) when :started then nil else report "#{worker.description} sent an unexpected #{kind.inspect} frame, which was ignored." end @@ -221,6 +225,45 @@ def expectations(count) count.times { reporter.increment_expectation_count } end + # What a worker that loaded the spec files itself printed while it + # did is what this process printed when it loaded them. + def output(worker, sizes) + return unless worker.output + return worker.output.skip(sizes) if worker.loading + + worker.output.read(sizes) { |stream, bytes| show_output(worker, stream, bytes) } + end + + # What is left in a worker's output files once it has gone: what it + # printed after it last reported, such as a crash report or what its + # `at_exit` handlers printed. + def drain_output(worker) + return unless worker.output + + worker.output.read { |stream, bytes| show_output(worker, stream, bytes) } + worker.output.close + end + + # A worker dismissed before the run used it has only loaded the spec + # files, which this process did too; before the run reports, no + # worker has run anything. + def show_output(worker, stream, bytes) + return if worker.state == :dismissed || @sequencer.nil? + + @sequencer.output(worker.number, stream, bytes) + end + + def write_output(stream, bytes) + io = stream == :err ? $stderr : $stdout + # What formatters printed before it comes first. + $stdout.flush if stream == :err + bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !io.binmode? + io.write(bytes) + io.flush + rescue IOError, SystemCallError + nil + end + # Anything it sends shows it got past forking and runs RSpec's code. def worker_started(worker) worker.starting = false @@ -389,6 +432,7 @@ def reap_workers next unless pid receive_from(worker) + drain_output(worker) worker.exit_status = status judge_exit(worker, status) worker.state = :exited @@ -415,6 +459,7 @@ def escalate(worker) worker.signal(@process, 'KILL', :kill, now + @kill_grace) when :kill report "#{worker.description} is still alive #{Parallel.seconds(@kill_grace)} after SIGKILL; giving up on it." + drain_output(worker) abort_unit(worker) worker.state = :abandoned end diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb index 417477067..022554055 100644 --- a/rspec-core/lib/rspec/core/parallel/sequencer.rb +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -32,6 +32,8 @@ class Sequencer QUEUE_BOUND = 5_000 QUEUE_BYTES_BOUND = 8 * 1024 * 1024 CONTEXT_FREE = [:message, :deprecation, :custom].freeze + # Shown under the groups it was printed in; outside any, as it comes. + OUTSIDE_GROUPS = (CONTEXT_FREE + [:output]).freeze Event = Struct.new(:kind, :target, :payload, :bytes) @@ -56,6 +58,7 @@ def initialize(worker) def initialize(reporter, options={}) @reporter = reporter @apply_result = options.fetch(:apply_result) + @write_output = options.fetch(:output) { lambda { |_stream, _bytes| } } @clock = options.fetch(:clock) { Parallel.method(:monotonic_time) } @sequenced = options.fetch(:sequenced) { self.class.group_aware_listeners?(reporter) } @streams = {} @@ -117,6 +120,11 @@ def custom(worker, event, options, bytes=0) push(worker, Event.new(:custom, event, options, bytes)) end + # What the worker printed to `stream` (`:out` or `:err`). + def output(worker, stream, bytes) + push(worker, Event.new(:output, stream, bytes, bytes.bytesize)) + end + # The worker died (or was stopped) in the middle of a group: deliver # what it reported and balance the group notifications formatters saw. def abandon(worker) @@ -153,7 +161,7 @@ def push(worker, event) stream = @streams[worker] # Messages sent outside any group (suite hooks) are not sequenced. - if CONTEXT_FREE.include?(event.kind) && (stream.nil? || stream.depth <= 0) + if OUTSIDE_GROUPS.include?(event.kind) && (stream.nil? || stream.depth <= 0) return deliver(Stream.new(worker), event) end @@ -199,6 +207,8 @@ def enqueue(stream, event) def deliver(stream, event) if CONTEXT_FREE.include?(event.kind) deliver_context_free(event) + elsif event.kind == :output && stream.open_groups.empty? + @write_output.call(event.target, event.payload) else switch_to(stream) unless @visible.equal?(stream) deliver_in_context(stream, event) @@ -220,6 +230,8 @@ def deliver_in_context(stream, event) record_duration(event.target, event.payload) if event.payload when :example_started @reporter.example_started(event.target) + when :output + @write_output.call(event.target, event.payload) when :example_finished @apply_result.call(event.target) @reporter.example_finished(event.target) diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 30f6f7c02..7979c6957 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -37,6 +37,7 @@ def initialize(number, example_groups, jobs, results, options={}) @ppid = options.fetch(:ppid) { Process.method(:ppid) } @watch_coordinator = options.fetch(:watch_coordinator, true) @signal_handlers = options.fetch(:signal_handlers, {}) + @output = options[:output] end # @return [String] the `TEST_ENV_NUMBER` value for this worker, following @@ -48,6 +49,7 @@ def test_env_number # Never returns: ends the process when the coordinator has no more work. def run + capture_output # First, so the coordinator knows it got past `fork`. @results.send([:started]) watch_coordinator if @watch_coordinator @@ -61,6 +63,17 @@ def run @exit_now.call(1) end + # @private + def capture_output + return unless @output + + # For when there is no coordinator left to show the files. + @direct_stderr = STDERR.dup + @results = WorkerOutput::Capture.new(@output, @results) + rescue SystemCallError, IOError + nil # it writes where the run does + end + # A worker notices that the coordinator is gone when it next reports # to it -- which a worker stuck in a long example does not do. Its # parent changes when the coordinator dies, so it does not outlive the @@ -70,12 +83,15 @@ def watch_coordinator coordinator = @ppid.call thread = Thread.new do sleep COORDINATOR_POLL while @ppid.call == coordinator + stderr = @stderr || @direct_stderr || $stderr begin - (@stderr || $stderr).puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) is exiting because the " \ - "coordinating process (pid #{coordinator}) is gone." + stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) is exiting because the " \ + "coordinating process (pid #{coordinator}) is gone." rescue IOError, SystemCallError nil end + # The coordinator would have removed them. + WorkerOutput.remove_directory(File.dirname(@output[:out])) if @output @exit_now.call(1) end thread.report_on_exception = false diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 81764994a..1233505b3 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -16,9 +16,12 @@ class WorkerHandle attr_accessor :loading # Forked, and has not sent anything yet. attr_accessor :starting + # Its standard output and error, or `nil` when it writes where the run does. + attr_reader :output - def initialize(number) + def initialize(number, output_dir=nil) @number = number + @output = output_dir && WorkerOutput.new(output_dir, number) @child_jobs, @jobs = Channel.pipe begin @results, @child_results = Channel.pipe @@ -98,7 +101,8 @@ def fork_worker(process, units, handles, worker_options={}) # :nocov: - runs in the forked worker process, covered by integration specs handles.each { |other| other.close_all unless other.equal?(self) } close_parent_ends - Worker.new(@number, units, @child_jobs, @child_results, worker_options).run + Worker.new(@number, units, @child_jobs, @child_results, + worker_options.merge(:output => @output && @output.paths)).run # :nocov: end end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 2783659cb..1774aaa05 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -38,6 +38,7 @@ def dismiss live = live_workers live.each { |worker| dismiss_worker(worker) } give_up_unless_gone(live, @kill_grace) + WorkerOutput.remove_directory(@output_dir) end private @@ -75,6 +76,7 @@ def account_for_pauses def start_workers(units) Parallel.flush_output(@configuration) @process.warmup if @process.respond_to?(:warmup) + @output_dir ||= WorkerOutput.make_directory @worker_count.times do |index| break unless start_worker(index + 1, units) @@ -83,7 +85,7 @@ def start_workers(units) # @return [Boolean] whether the worker started def start_worker(number, units) - worker = WorkerHandle.new(number) + worker = WorkerHandle.new(number, @output_dir) worker.loading = units.nil? @workers << worker worker.pid = @spawn.call(worker, @workers) @@ -174,6 +176,7 @@ def give_up_unless_gone(workers, grace) wait_for_exit(workers, grace) workers.reject(&:done?).each do |worker| report "#{worker.description} survived SIGKILL; giving up on it." + drain_output(worker) abort_unit(worker) worker.state = :abandoned end diff --git a/rspec-core/lib/rspec/core/parallel/worker_output.rb b/rspec-core/lib/rspec/core/parallel/worker_output.rb new file mode 100644 index 000000000..541a9933b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/worker_output.rb @@ -0,0 +1,107 @@ +module RSpec + module Core + module Parallel + # A worker's standard output and error -- what its examples print, + # warnings, the output of processes they start, a crash report -- go to + # two files that only the coordinator reads. Before each frame it sends, + # the worker says how far the files reach, so the coordinator shows + # what was printed in the order the worker reported its examples: with + # the example that printed it, as in a serial run. What is still in the + # files when the worker has gone is read then. + # @private + class WorkerOutput + STREAMS = [:out, :err].freeze + # Read at a time, so that the sequencer's byte bound applies to a + # flood of output too. + CHUNK = 1024 * 1024 + + # @return [String, nil] a directory only this user can read, or `nil` + # when none can be made: workers then write where the run does + def self.make_directory + require 'tmpdir' + Dir.mktmpdir('rspec-parallel-') + rescue SystemCallError, IOError + nil + end + + def self.remove_directory(dir) + return unless dir + + require 'fileutils' + FileUtils.remove_entry(dir, true) + end + + attr_reader :paths + + def initialize(dir, number) + @paths = STREAMS.to_h { |stream| [stream, File.join(dir, "worker-#{number}.#{stream}")] } + @paths.each_value { |path| File.open(path, 'wb') {} } + @readers = {} + @read = { :out => 0, :err => 0 } + end + + # Yields `stream, bytes` for what the worker wrote up to `sizes` + # (`{:out => n, :err => m}`), or to the end of the files. + def read(sizes=nil, &block) + STREAMS.each { |stream| read_stream(stream, sizes && sizes[stream], &block) } + end + + # Moves on past what the worker wrote up to `sizes` without showing it. + def skip(sizes) + STREAMS.each { |stream| @read[stream] = [@read[stream], sizes[stream]].max } + end + + def close + @readers.each_value(&:close) + @readers.clear + end + + private + + def read_stream(stream, limit) + reader = (@readers[stream] ||= File.open(@paths[stream], 'rb')) + loop do + wanted = limit ? limit - @read[stream] : CHUNK + break if wanted <= 0 + + reader.pos = @read[stream] + bytes = reader.read([wanted, CHUNK].min) + break if bytes.nil? || bytes.empty? + + @read[stream] += bytes.bytesize + yield stream, bytes + end + rescue SystemCallError, IOError + nil + end + + # The worker's side: its standard output and error moved into the + # files, and the results channel it reports through, which announces + # new output before each frame. + # @private + class Capture + def initialize(paths, results) + @results = results + @files = paths.transform_values { |path| File.open(path, 'ab') } + STDOUT.reopen(@files[:out]) + STDERR.reopen(@files[:err]) + [STDOUT, STDERR].each do |io| + io.binmode + io.sync = true + end + @announced = { :out => 0, :err => 0 } + end + + def send(message) + sizes = @files.transform_values(&:size) + unless sizes == @announced + @results.send([:output, sizes]) + @announced = sizes + end + @results.send(message) + end + end + end + end + end +end diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 9f44197d8..d769a10e2 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -21,6 +21,20 @@ def run_rspec(cmd) run_command(cmd) end + # A process of its own, where the `--require`d files load afresh. Given at + # most 60 seconds. + def rspec_process(*args) + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 + sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + Process.kill(:KILL, pid) && Process.wait(pid) unless status + [status, File.read('stdout.txt'), File.read('stderr.txt')] + end + end + # Written by the workers' `before(:suite)` hooks: a file can exist before # its pid has been written to it. def worker_pids @@ -315,6 +329,49 @@ def example_passed(notification) expect(last_cmd_exit_status).to eq(0) end + it 'shows what workers print with the example group that printed it, as a serial run does' do + write_file_formatted 'spec/support/at_exit.rb', " + RSpec.configure { |c| c.after(:suite) { at_exit { puts \"at_exit of worker \#{c.parallel_worker_number}\" } } } + " + 3.times do |i| + write_file_formatted "spec/p#{i}_spec.rb", " + puts 'loading p#{i}' + RSpec.describe('P#{i}') do + it('prints') do + puts 'stdout of P#{i}' + warn 'stderr of P#{i}' + system('echo subprocess of P#{i}') + end + it('is quiet') { } + end + " + end + write_file_formatted 'spec/killed_spec.rb', " + RSpec.describe('Killed') do + it('says goodbye') { $stderr.puts 'last words'; Process.kill(:KILL, Process.pid) } + end + " + options = %w[--require ./spec/support/at_exit --format documentation --order defined --parallel=2] + + status, stdout, stderr = rspec_process(*options) + blocks = stdout.split(/^(?=P\d$)/).select { |block| block =~ /\AP\d$/ } + expect(blocks.size).to eq(3) + blocks.each do |block| + name = block[/\AP\d/] + expect(block).to start_with("#{name}\nstdout of #{name}\nsubprocess of #{name}\n prints\n is quiet\n") + end + # The worker killed with SIGKILL runs no `at_exit` handler. + expect(stdout.scan(/^at_exit of worker \d$/)).not_to be_empty + expect(stderr).to include("stderr of P0\n", "stderr of P1\n", "stderr of P2\n", "last words\n") + expect(stdout).to include("7 examples, 1 failure") + expect(status.exitstatus).to eq(1) + + status, stdout, _stderr = rspec_process(*options, '--no-parallel-preload', *%w[spec/p0_spec.rb spec/p1_spec.rb spec/p2_spec.rb]) + expect(stdout.scan(/^loading p\d$/)).to eq(["loading p0", "loading p1", "loading p2"]) + expect(stdout).to include("6 examples, 0 failures") + expect(status.exitstatus).to eq(0) + end + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do write_file_formatted 'spec/support/my_formatter.rb', " class MyAppError < StandardError; end @@ -866,15 +923,20 @@ def write_split_suite(extra="") cd('.') do rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] - pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, '--parallel=2', 'spec/stuck_spec.rb', :out => File::NULL, :err => 'stderr.txt') + Dir.mkdir('tmp') + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil, 'TMPDIR' => File.expand_path('tmp') }, *rspec, '--parallel=2', + 'spec/stuck_spec.rb', :out => File::NULL, :err => 'stderr.txt') expect(wait_until.call(20) { worker_pids.size == 2 }).to be true workers = worker_pids + expect(Dir['tmp/rspec-parallel-*'].size).to eq(1) Process.kill(:KILL, pid) Process.wait(pid) expect(wait_until.call(10) { workers.none?(&running) }).to be true expect(File.read('stderr.txt').scan("is exiting because the coordinating process (pid #{pid}) is gone").size).to eq(2) + # Where the workers' output went, which the coordinator would have removed. + expect(Dir['tmp/*']).to eq([]) end end @@ -1041,20 +1103,6 @@ def loaders cd('.') { Dir['loaded_*'].map { |file| file.split('_', 4).last }.tally } end - # A process of its own, where the `--require`d files load afresh. Given at - # most 60 seconds. - def rspec_process(*args) - cd('.') do - rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] - pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') - deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 - sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || - Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline - Process.kill(:KILL, pid) && Process.wait(pid) unless status - [status, File.read('stdout.txt'), File.read('stderr.txt')] - end - end - it 'has every worker load the spec files itself, once TEST_ENV_NUMBER is set, and run what it loaded' do write_marking_spec_files(:wait_for_both_workers => true) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 7efcfff40..efdd7f525 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -124,8 +124,10 @@ def spawn(handle, _handles) when :never_loads, :never_starts, :starts_once_asked_to_quit then stall(behaviour, handle, results) when :dies_loading then Thread.current.kill else + print_while_loading(handle, results) if behaviour == :prints_while_loading + print_to(handle, results, "loaded quietly\n", "", false) if behaviour == :prints_unannounced_while_loading report_loaded(results, behaviour) if handle.loading - send_extra_frames(results, behaviour) + send_extra_frames(results, behaviour, handle) run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } end ensure @@ -189,9 +191,23 @@ def report_loaded(results, behaviour) results.send([:loaded, digest, failure]) end - def send_extra_frames(results, behaviour) + def send_extra_frames(results, behaviour, handle) send_unknown_ids(results) if behaviour == :unknown_ids results.send([:expectations, 3]) if behaviour == :counts_expectations + print_to(handle, results, "printed by worker #{handle.number}\n", "warned by worker #{handle.number}\n") if behaviour == :prints + print_to(handle, results, "left behind by worker #{handle.number}\n", "", false) if behaviour == :prints_unannounced + end + + # As a worker's own output goes to the files once it has moved its + # standard output and error there; the frame is what it sends next. + def print_to(handle, results, out, err, announce=true) + File.open(handle.output.paths[:out], 'ab') { |file| file.write(out) } + File.open(handle.output.paths[:err], 'ab') { |file| file.write(err) } + results.send([:output, handle.output.paths.transform_values { |path| File.size(path) }]) if announce + end + + def print_while_loading(handle, results) + print_to(handle, results, "noise while loading\n", "") end def send_unknown_ids(results) @@ -837,6 +853,63 @@ def turn(seconds) expect(messages).to include("2 examples, 0 failures") end + it "shows what workers printed, where they reported it, and what they left unannounced once they are gone" do + worker_behaviours[1] = :prints + worker_behaviours[2] = :prints_unannounced + exit_code = nil + + stdout, stderr = standard_streams { exit_code = coordinator([passing_group("A"), passing_group("B")]).run } + + expect(exit_code).to eq(0) + expect(stdout).to include("printed by worker 1\n", "left behind by worker 2\n") + expect(stderr).to eq("warned by worker 1\n") + end + + it "does not show what a worker that loaded the spec files itself printed while it did" do + worker_behaviours[1] = :prints_while_loading + instance = started_coordinator([passing_group("A"), passing_group("B")]) + instance.assign(@units, units_of(@units)) + + exit_code = nil + stdout, _stderr = standard_streams { exit_code = instance.run } + + expect(exit_code).to eq(0) + expect(stdout).to eq("") + end + + it "does not show what a worker dismissed before the run used it left in its files" do + worker_behaviours[3] = :prints_unannounced_while_loading + instance = started_coordinator([passing_group("A"), passing_group("B")], 3) + instance.assign(@units, units_of(@units)) + + exit_code = nil + stdout, _stderr = standard_streams { exit_code = instance.run } + + expect(exit_code).to eq(0) + expect(stdout).to eq("") + end + + it "goes on when it cannot show what a worker printed" do + worker_behaviours[1] = :prints + exit_code = nil + + standard_streams(StringIO.new.tap(&:close_write)) { exit_code = coordinator([passing_group("A"), passing_group("B")]).run } + + expect(exit_code).to eq(0) + expect(messages).to include("2 examples, 0 failures") + end + + # What the coordinator writes to them; the formatters write to `output`. + def standard_streams(stdout=StringIO.new) + streams = [$stdout, $stderr] + $stdout = stdout + $stderr = StringIO.new + yield + [$stdout.string, $stderr.string] + ensure + $stdout, $stderr = streams + end + it "reports the groups that never ran when no worker remains" do crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } groups = [crashing, RSpec.describe("Also crashing") { it("dies") { Thread.current.kill } }, passing_group("C"), passing_group("D")] diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index a8ecf3801..97b2cb8e5 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -51,7 +51,9 @@ def ex(group, description) def sequencer(sequenced) reporter.register_listener(listener, *RSpec::Core::Reporter::RSPEC_NOTIFICATIONS, :my_event) - Sequencer.new(reporter, :apply_result => lambda { |example| applied << example }, :clock => clock, :sequenced => sequenced) + Sequencer.new(reporter, :apply_result => lambda { |example| applied << example }, :clock => clock, + :sequenced => sequenced, + :output => lambda { |stream, bytes| listener.events << [:output, "#{stream}:#{bytes}"] }) end def run_example(seq, worker, example, status=:passed) @@ -305,6 +307,52 @@ def events end end + describe "#output" do + it "shows what a worker printed in a group with that group, queued with it while another group is live" do + seq = sequencer(true) + seq.group_started(1, g1) + seq.group_started(2, g2) + seq.output(2, :out, "printed in G2") + run_example(seq, 2, ex(g2, "c")) + seq.output(2, :err, "warned in G2") + seq.group_finished(2, g2, 0.1) + expect(events).to eq([[:example_group_started, "G1"]]) + expect(seq.stats[:queued_bytes]).to eq("printed in G2".bytesize + "warned in G2".bytesize) + + seq.group_finished(1, g1, 0.1) + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"], [:output, "out:printed in G2"], + [:example_started, "c"], [:example_finished, "c"], [:example_passed, "c"], + [:output, "err:warned in G2"], [:example_group_finished, "G2"] + ]) + end + + it "re-opens the group output was printed in when another group was shown since" do + seq = sequencer(false) + seq.group_started(1, g1) + seq.group_started(2, g2) + + seq.output(1, :out, "printed in G1") + + expect(events).to eq([ + [:example_group_started, "G1"], [:example_group_finished, "G1"], + [:example_group_started, "G2"], [:example_group_finished, "G2"], + [:example_group_started, "G1"], [:output, "out:printed in G1"] + ]) + end + + it "writes what a worker printed outside any group as it comes, without touching the groups shown" do + seq = sequencer(true) + seq.group_started(1, g1) + + seq.output(2, :out, "from before(:suite)") + + expect(events).to eq([[:example_group_started, "G1"], [:output, "out:from before(:suite)"]]) + end + end + describe "#abandon" do it "balances the group notifications of a live group that will never finish" do seq = sequencer(true) diff --git a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb new file mode 100644 index 000000000..02977eecb --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb @@ -0,0 +1,91 @@ +require 'fileutils' +require 'tmpdir' +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe WorkerOutput do + let!(:dir) { Dir.mktmpdir } + let!(:output) { WorkerOutput.new(dir, 3) } + + after do + output.close + FileUtils.remove_entry(dir) + end + + def read(sizes=nil) + chunks = [] + output.read(sizes) { |stream, bytes| chunks << [stream, bytes] } + chunks + end + + it "creates an empty file per stream, named after the worker" do + expect(output.paths.values.map { |path| File.basename(path) }).to eq(%w[worker-3.out worker-3.err]) + expect(output.paths.values.map { |path| File.size(path) }).to eq([0, 0]) + end + + it "yields what was written up to the given sizes, each byte once, and the rest when asked for everything" do + File.write(output.paths[:out], "one two") + File.write(output.paths[:err], "warn") + + expect(read(:out => 3, :err => 0)).to eq([[:out, "one"]]) + expect(read(:out => 3, :err => 0)).to eq([]) + expect(read).to eq([[:out, " two"], [:err, "warn"]]) + expect(read).to eq([]) + end + + it "moves past what it was told to skip" do + File.write(output.paths[:out], "loading...done") + output.skip(:out => 10, :err => 0) + + expect(read).to eq([[:out, "done"]]) + end + + it "reads a flood of output in chunks" do + stub_const("RSpec::Core::Parallel::WorkerOutput::CHUNK", 4) + File.write(output.paths[:out], "0123456789") + + expect(read.map(&:last)).to eq(%w[0123 4567 89]) + end + + it "yields nothing for files that are gone" do + output.paths.each_value { |path| File.delete(path) } + expect(read).to eq([]) + end + + it "goes without a directory when none can be made" do + allow(Dir).to receive(:mktmpdir).and_raise(Errno::EACCES) + expect(WorkerOutput.make_directory).to be_nil + expect { WorkerOutput.remove_directory(nil) }.not_to raise_error + end + + describe WorkerOutput::Capture do + let(:results) { Channel.pipe } + let(:stdout) { File.open(File.join(dir, 'stdout'), 'w') } + let(:stderr) { File.open(File.join(dir, 'stderr'), 'w') } + + before do + stub_const('STDOUT', stdout) + stub_const('STDERR', stderr) + end + + after do + results.each(&:close) + [stdout, stderr].each(&:close) + end + + it "moves standard output and error into the files and says how far they reach before a frame that follows output" do + capture = WorkerOutput::Capture.new(output.paths, results.last) + STDOUT.write("printed") + STDERR.write("warned") + + capture.send([:first]) + capture.send([:second]) + + frames = results.first.drain.map(&:first) + expect(frames).to eq([[:output, { :out => 7, :err => 6 }], [:first], [:second]]) + expect(read(frames.first.last)).to eq([[:out, "printed"], [:err, "warned"]]) + expect([STDOUT.sync, STDERR.sync]).to eq([true, true]) + end + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 2ffa773e9..2251487c2 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -1,3 +1,5 @@ +require 'fileutils' +require 'tmpdir' require 'rspec/core/parallel/coordinator' module RSpec::Core::Parallel @@ -69,6 +71,19 @@ def frames expect(stderr.string).to include("worker 2 (pid #{Process.pid}) is exiting because the coordinating process (pid 100) is gone") end + it "removes the directory of its output files, which the coordinator would have removed" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + dir = Dir.mktmpdir + parents = [100, 1] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, :output => WorkerOutput.new(dir, 2).paths, + :exit_now => lambda { |status| exits << [:exit_now, status] }, + :ppid => lambda { parents.size > 1 ? parents.shift : parents.first }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(File.exist?(dir)).to be false + end + it "still exits when it cannot say why (its stderr is gone too)" do stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) parents = [100, 1] @@ -186,6 +201,45 @@ def frames ]) end + context "with output files" do + let(:dir) { Dir.mktmpdir } + let(:stdout_stand_in) { File.open(File.join(dir, 'stdout'), 'w') } + let(:stderr_stand_in) { File.open(File.join(dir, 'stderr'), 'w') } + + before do + stub_const('STDOUT', stdout_stand_in) + stub_const('STDERR', stderr_stand_in) + end + + after do + [stdout_stand_in, stderr_stand_in].each(&:close) + FileUtils.remove_entry(dir) + end + + it "moves its standard output and error into them and says how far they reach before the next frame" do + output = WorkerOutput.new(dir, 2) + groups.first.example("prints") { STDOUT.write("printed") } + jobs.last.send([:run, 0, groups[0].id]) + jobs.last.send([:stop]) + + worker(2, :output => output.paths).run + + announced = frames.select { |f| f.first == :output } + expect(announced).to eq([[:output, { :out => 7, :err => 0 }]]) + expect(File.read(output.paths[:out])).to eq("printed") + output.close + end + + it "writes where the run does when it cannot open them" do + missing = { :out => File.join(dir, 'gone', 'out'), :err => File.join(dir, 'gone', 'err') } + jobs.last.send([:stop]) + + worker(2, :output => missing).run + + expect(frames).to eq([[:started]]) + end + end + it "exits at once, without running suite hooks or at_exit handlers, when its first job is not a group to run" do hooks = [] RSpec.configuration.before(:suite) { hooks << :before_suite } From 13c97f6f68ac9a5cd9bfa3f74ba3ba54447d98e2 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:12:08 +0200 Subject: [PATCH 47/58] [rspec-core] Note the Windows `--bisect` fix in the Changelog Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 4433ddda9..fb99bcc1d 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -37,6 +37,11 @@ Enhancements: example and example group, which took seconds on a suite of 150,000 examples, and more when several processes did it at once. (Illia Zub) +Bug Fixes: + +* Fix `--bisect` on Windows, which passed its load path to the processes it starts joined + with `:` instead of `;`. (Illia Zub) + # 4.0.0.beta1 / 2026-02-18 Breaking Changes: From e4619b337822d03d555e70caf46205e330d85a94 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:32:09 +0200 Subject: [PATCH 48/58] [rspec-core] Run --parallel on Windows and JRuby with spawned workers Where Ruby has no `fork`, each worker is a new process running the `rspec` command the run was started with: same arguments, `.rspec` files and `SPEC_OPTS`, no formatters, the run's seed (adopted without choosing an ordering). It loads the spec files itself, as with `--no-parallel-preload`, and takes jobs and sends results over its standard input and output in binary mode. Workers notice a gone coordinator by its pid where their parent pid never changes (Windows), get SIGKILL where SIGTERM cannot be sent, and a run that was not started by the `rspec` command stays serial with a warning. Probed first on windows-latest (Ruby 3.0, 3.4), JRuby 9.4/10.0/10.1 and TruffleRuby: pipes, binary frames, capture files, exit statuses, signals, inheritance and parent death, 18/18 checks each. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 4 +- rspec-core/README.md | 18 ++- rspec-core/lib/rspec/core/ordering.rb | 7 ++ rspec-core/lib/rspec/core/parallel.rb | 13 ++- rspec-core/lib/rspec/core/parallel/channel.rb | 4 + .../lib/rspec/core/parallel/coordinator.rb | 45 +------- rspec-core/lib/rspec/core/parallel/plan.rb | 6 +- .../lib/rspec/core/parallel/spawned_worker.rb | 74 ++++++++++++ rspec-core/lib/rspec/core/parallel/worker.rb | 35 ++++-- .../lib/rspec/core/parallel/worker_handle.rb | 15 ++- .../rspec/core/parallel/worker_lifecycle.rb | 44 ++++++- .../lib/rspec/core/parallel/worker_output.rb | 22 +++- .../rspec/core/parallel/worker_reporter.rb | 9 +- rspec-core/lib/rspec/core/runner.rb | 16 +++ rspec-core/spec/integration/parallel_spec.rb | 109 +++++++++++++++--- .../rspec/core/parallel/coordinator_spec.rb | 48 ++++++-- .../spec/rspec/core/parallel/plan_spec.rb | 32 ++++- .../spec/rspec/core/parallel/results_spec.rb | 6 +- .../core/parallel/spawned_worker_spec.rb | 81 +++++++++++++ .../rspec/core/parallel/worker_handle_spec.rb | 33 ++++++ .../rspec/core/parallel/worker_output_spec.rb | 2 + .../core/parallel/worker_reporter_spec.rb | 4 +- .../spec/rspec/core/parallel/worker_spec.rb | 22 ++++ rspec-core/spec/rspec/core/parallel_spec.rb | 18 ++- rspec-core/spec/rspec/core/runner_spec.rb | 33 ++++++ 25 files changed, 598 insertions(+), 102 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/spawned_worker.rb create mode 100644 rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index fb99bcc1d..25fb9fdf3 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -29,7 +29,9 @@ Enhancements: (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec files are loaded and each loads them itself, so nothing the spec files do while loading is shared with the workers; a worker that does not load the examples the coordinating - process has is not used. Where `fork` is unavailable the run is serial with a warning. + process has is not used. Where Ruby has no `fork` (Windows, JRuby, TruffleRuby) the + workers are new processes running the same `rspec` command with the run's seed, and load + the spec files themselves. `--bisect` runs serially, and the reproduction command it prints now ends with `--no-parallel`. (Illia Zub) diff --git a/rspec-core/README.md b/rspec-core/README.md index 0378824a0..213dda2b0 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -444,9 +444,19 @@ custom ordering is not split, with a warning. With a formatter that follows example groups (documentation, HTML), each part reads as a block of its own, under the group's description; `--profile` adds up the parts. -`--parallel` needs `fork`. On Windows and JRuby, or when `TEST_ENV_NUMBER` is -already set because another tool is running RSpec in parallel, RSpec prints a -warning and runs serially. When the system refuses a worker -- too many open +Where Ruby has no `fork` (Windows, JRuby, TruffleRuby), the workers are new +processes running the `rspec` command the run was started with -- the same +arguments, `.rspec` files and `SPEC_OPTS`, without the formatters, and the +run's seed -- and each loads the spec files itself, as with +`--no-parallel-preload`. Their jobs and results travel over their standard +input and output, so what they print goes through the coordinating process +as described above. Configuration done in-process before +`RSpec::Core::Runner.run` cannot reach such workers, so there a run that was +not started by the `rspec` command runs serially, with a warning. Each worker +starts its own Ruby (seconds on JRuby), and on Windows workers that have to +be stopped get SIGKILL, which is all Windows can send. When `TEST_ENV_NUMBER` +is already set because another tool is running RSpec in parallel, RSpec +prints a warning and runs serially. When the system refuses a worker -- too many open files or processes -- the run goes on with the workers it got, and serially if it got none, with a warning. `--bisect` and `--dry-run` always run serially, and the reproduction command `--bisect` prints includes `--no-parallel`, which @@ -459,6 +469,8 @@ coordinating process unless it merges forked processes. With SimpleCov, set `merge_subprocesses true` in the `SimpleCov.start` block (SimpleCov 1.3; it was `enable_for_subprocesses` before): rspec-core's own suite then measures the same coverage with `--parallel=8`, preloaded or not, as it does serially. +Workers started as new processes (no `fork`) start SimpleCov from your spec +helper themselves, and SimpleCov 1.3.2 merges their results. ## Store Command Line Options `.rspec` diff --git a/rspec-core/lib/rspec/core/ordering.rb b/rspec-core/lib/rspec/core/ordering.rb index a451c5c6f..d430bbcd3 100644 --- a/rspec-core/lib/rspec/core/ordering.rb +++ b/rspec-core/lib/rspec/core/ordering.rb @@ -160,6 +160,13 @@ def seed=(seed) @seed = seed.to_i end + # @private + # The seed of the parallel run a spawned worker belongs to, taken + # without choosing an ordering: the run's configuration does that. + def adopt_seed(seed) + @seed = seed.to_i unless @seed_forced + end + def order=(type) order, seed = type.to_s.split(':') @seed = seed.to_i if seed diff --git a/rspec-core/lib/rspec/core/parallel.rb b/rspec-core/lib/rspec/core/parallel.rb index ec3d6f0be..ed8f7383c 100644 --- a/rspec-core/lib/rspec/core/parallel.rb +++ b/rspec-core/lib/rspec/core/parallel.rb @@ -74,9 +74,10 @@ def self.requested_worker_count(configuration) requested = configuration.parallel_workers return nil if requested.nil? || configuration.dry_run? - unless RSpec::Support::RubyFeatures.fork_supported? - RSpec.warning "Ignoring `--parallel`: it needs `fork`, which is not available on #{RUBY_ENGINE} " \ - "(#{RUBY_PLATFORM}). Running serially", :call_site => nil + if spawning? && !Runner.command_line + RSpec.warning "Ignoring `--parallel`: without `fork` (#{RUBY_ENGINE}, #{RUBY_PLATFORM}) its workers are new " \ + "`rspec` processes, which takes a run started by the `rspec` command rather than by " \ + "`RSpec::Core::Runner.run`. Running serially", :call_site => nil return nil end @@ -89,6 +90,12 @@ def self.requested_worker_count(configuration) requested == :auto ? default_worker_count : requested end + # Without `fork`, workers are new `rspec` processes, which load the spec + # files themselves. + def self.spawning? + !RSpec::Support::RubyFeatures.fork_supported? + end + # @return [Integer] the number of processors def self.default_worker_count require 'etc' diff --git a/rspec-core/lib/rspec/core/parallel/channel.rb b/rspec-core/lib/rspec/core/parallel/channel.rb index 67ac95a50..e11815b84 100644 --- a/rspec-core/lib/rspec/core/parallel/channel.rb +++ b/rspec-core/lib/rspec/core/parallel/channel.rb @@ -27,6 +27,9 @@ class Writer def initialize(io) @io = io + # No newline conversion on Windows, where a spawned worker's + # standard input and output start in text mode. + @io.binmode @io.set_encoding ENCODING @io.sync = true end @@ -51,6 +54,7 @@ class Reader def initialize(io) @io = io + @io.binmode @io.set_encoding ENCODING @buffer = String.new(:encoding => ENCODING) @offset = 0 diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 87497ac92..c5d2e964f 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -3,6 +3,7 @@ RSpec::Support.require_rspec_core "parallel/remote_exception" RSpec::Support.require_rspec_core "parallel/worker_reporter" RSpec::Support.require_rspec_core "parallel/worker" +RSpec::Support.require_rspec_core "parallel/spawned_worker" RSpec::Support.require_rspec_core "parallel/sequencer" RSpec::Support.require_rspec_core "parallel/results" RSpec::Support.require_rspec_core "parallel/suite_index" @@ -46,7 +47,7 @@ def initialize(runner, worker_count, options={}) @world = runner.world @worker_count = worker_count @process = options.fetch(:process) { Process } - @spawn = options.fetch(:spawn) { method(:fork_worker) } + @spawn = options.fetch(:spawn) { method(Parallel.spawning? ? :spawn_worker : :fork_worker) } @tick = options.fetch(:tick, TICK) @term_grace = options.fetch(:term_grace, TERM_GRACE) @kill_grace = options.fetch(:kill_grace, KILL_GRACE) @@ -61,6 +62,7 @@ def initialize(runner, worker_count, options={}) @next_reap_at = 0.0 @last_turn_at = nil @output_dir = nil + @boot_allowance = 0.0 end # The groups to run. Workers forked before loading that are more than @@ -82,7 +84,7 @@ def run return run_serially if @workers.empty? passed = reporter.report(@world.example_count(@example_groups)) do - @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result), :output => method(:write_output)) + @sequencer = Sequencer.new(reporter, :apply_result => method(:apply_result), :output => WorkerOutput.method(:write)) begin @last_turn_at = now @@ -225,45 +227,6 @@ def expectations(count) count.times { reporter.increment_expectation_count } end - # What a worker that loaded the spec files itself printed while it - # did is what this process printed when it loaded them. - def output(worker, sizes) - return unless worker.output - return worker.output.skip(sizes) if worker.loading - - worker.output.read(sizes) { |stream, bytes| show_output(worker, stream, bytes) } - end - - # What is left in a worker's output files once it has gone: what it - # printed after it last reported, such as a crash report or what its - # `at_exit` handlers printed. - def drain_output(worker) - return unless worker.output - - worker.output.read { |stream, bytes| show_output(worker, stream, bytes) } - worker.output.close - end - - # A worker dismissed before the run used it has only loaded the spec - # files, which this process did too; before the run reports, no - # worker has run anything. - def show_output(worker, stream, bytes) - return if worker.state == :dismissed || @sequencer.nil? - - @sequencer.output(worker.number, stream, bytes) - end - - def write_output(stream, bytes) - io = stream == :err ? $stderr : $stdout - # What formatters printed before it comes first. - $stdout.flush if stream == :err - bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !io.binmode? - io.write(bytes) - io.flush - rescue IOError, SystemCallError - nil - end - # Anything it sends shows it got past forking and runs RSpec's code. def worker_started(worker) worker.starting = false diff --git a/rspec-core/lib/rspec/core/parallel/plan.rb b/rspec-core/lib/rspec/core/parallel/plan.rb index ff3a0cfe0..6ecfde38f 100644 --- a/rspec-core/lib/rspec/core/parallel/plan.rb +++ b/rspec-core/lib/rspec/core/parallel/plan.rb @@ -18,7 +18,7 @@ def initialize(runner) # Called once the run is configured, before the spec files are loaded. def before_loading - return if @configuration.parallel_preload? + return if @configuration.parallel_preload? && !Parallel.spawning? return @postponed_by = '`config.parallel_workers`' if @configuration.parallel_workers.nil? # Such a run aborts, with `exit!`, as soon as the spec files have loaded. if @configuration.only_failures_but_not_configured? @@ -41,7 +41,7 @@ def before_loading # or `nil` for a serial run def coordinator_for(example_groups) return assign(example_groups) if @decided - return Parallel.coordinator_for(@runner, example_groups) if @configuration.parallel_preload? + return Parallel.coordinator_for(@runner, example_groups) if @configuration.parallel_preload? && !Parallel.spawning? # Too late to fork before loading, and forking after loading is # what this run was configured not to do. @@ -81,7 +81,7 @@ def assign(example_groups) end def warn_late - what = @postponed_by || '`config.parallel_preload = false`' + what = @postponed_by || (Parallel.spawning? ? 'a run without `fork`' : '`config.parallel_preload = false`') RSpec.warning "Ignoring `--parallel`: #{what} was set while the spec files were loading, too late to fork " \ "the workers before them as `config.parallel_preload = false` asks. Set it before they " \ "load: on the command line, in `.rspec` or in a `--require`d file. Running serially", diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb new file mode 100644 index 000000000..b6cd84451 --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -0,0 +1,74 @@ +module RSpec + module Core + module Parallel + # Where Ruby has no `fork` (Windows, JRuby, TruffleRuby), a worker is a + # new process running the `rspec` command the run was started with: it + # configures itself from the same arguments, `.rspec` files and + # `SPEC_OPTS` -- without formatters, which are the coordinator's -- and + # the run's seed, then loads the spec files and goes on like a worker + # forked before loading. Its jobs arrive on its standard input and its + # results leave on its standard output, so what it prints goes to its + # output files from the start. Configuration done in-process before + # `RSpec::Core::Runner.run` cannot reach it, which is why only a run + # started by the `rspec` command starts workers this way. + # @private + module SpawnedWorker + VARIABLES = %w[ + RSPEC_PARALLEL_WORKER RSPEC_PARALLEL_COORDINATOR RSPEC_PARALLEL_SEED + RSPEC_PARALLEL_STDOUT RSPEC_PARALLEL_STDERR + ].freeze + # Options that open files the coordinator writes to. + NOT_FOR_WORKERS = [:formatters, :deprecation_stream].freeze + + # @return [Hash{String => String}] what a worker numbered `number` is + # told through its environment + def self.environment(number, coordinator, seed, output) + env = { 'RSPEC_PARALLEL_WORKER' => number.to_s, 'RSPEC_PARALLEL_COORDINATOR' => coordinator.to_s, + 'RSPEC_PARALLEL_SEED' => seed.to_s, 'TEST_ENV_NUMBER' => Worker.test_env_number(number) } + env.update('RSPEC_PARALLEL_STDOUT' => output[:out], 'RSPEC_PARALLEL_STDERR' => output[:err]) if output + env + end + + # Taken out of the environment, so that the processes it starts are + # not workers. + # @return [Hash] + def self.settings(env=ENV) + number, coordinator, seed, stdout, stderr = VARIABLES.map { |name| env.delete(name) } + { :number => Integer(number), :coordinator => Integer(coordinator), :seed => seed && Integer(seed), + :output => stdout && stderr && { :out => stdout, :err => stderr } } + end + + def self.configure(args, settings, configuration=RSpec.configuration) + configuration.parallel_worker_number = settings[:number] + configuration.ordering_manager.adopt_seed(settings[:seed]) if settings[:seed] + options = ConfigurationOptions.new(args) + NOT_FOR_WORKERS.each { |key| options.options.delete(key) } + options.configure(configuration) + end + + # Never returns. + # :nocov: - runs in the spawned worker, covered by integration specs + def self.run(args) + settings = settings() + # rubocop:disable Style/GlobalStdStream + # The process's own streams, which the coordinator connected. + jobs = Channel::Reader.new(STDIN.dup) + results = Channel::Writer.new(STDOUT.dup) + STDIN.reopen(File::NULL) + # Its standard output must not be where its results go. + STDOUT.reopen(STDERR) unless settings[:output] + # rubocop:enable Style/GlobalStdStream + Runner.disable_autorun! + loading = Worker::LoadReporter.new(RSpec.configuration) + worker = Worker.new(settings[:number], nil, jobs, results, + :output => settings[:output], :coordinator => settings[:coordinator], :loading => loading) + worker.capture_output + RSpec.configuration.reporter = loading + configure(args, settings) + worker.run + end + # :nocov: + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 7979c6957..0b4779a5d 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -35,6 +35,9 @@ def initialize(number, example_groups, jobs, results, options={}) lambda { |signal, command=nil, &handler| command ? trap(signal, command) : trap(signal, &handler) } end @ppid = options.fetch(:ppid) { Process.method(:ppid) } + @alive = options.fetch(:alive) { method(:process_alive?) } + @coordinator = options[:coordinator] + @loading = options[:loading] @watch_coordinator = options.fetch(:watch_coordinator, true) @signal_handlers = options.fetch(:signal_handlers, {}) @output = options[:output] @@ -44,7 +47,12 @@ def initialize(number, example_groups, jobs, results, options={}) # the convention established by parallel_tests: `""` for the first # worker, then `"2"`, `"3"`, ... def test_env_number - @number == 1 ? "" : @number.to_s + self.class.test_env_number(@number) + end + + # @private + def self.test_env_number(number) + number == 1 ? "" : number.to_s end # Never returns: ends the process when the coordinator has no more work. @@ -65,10 +73,11 @@ def run # @private def capture_output - return unless @output + return if @captured || !@output + @captured = true # For when there is no coordinator left to show the files. - @direct_stderr = STDERR.dup + @direct_stderr = STDERR.dup # rubocop:disable Style/GlobalStdStream @results = WorkerOutput::Capture.new(@output, @results) rescue SystemCallError, IOError nil # it writes where the run does @@ -76,13 +85,14 @@ def capture_output # A worker notices that the coordinator is gone when it next reports # to it -- which a worker stuck in a long example does not do. Its - # parent changes when the coordinator dies, so it does not outlive the - # coordinator by more than {COORDINATOR_POLL} seconds. + # parent changes when the coordinator dies, or (on Windows, where it + # does not) the coordinator's pid stops naming a process, so it does + # not outlive the coordinator by more than {COORDINATOR_POLL} seconds. # @private def watch_coordinator - coordinator = @ppid.call + coordinator = @coordinator || @ppid.call thread = Thread.new do - sleep COORDINATOR_POLL while @ppid.call == coordinator + sleep COORDINATOR_POLL while @ppid.call == coordinator && @alive.call(coordinator) stderr = @stderr || @direct_stderr || $stderr begin stderr.puts "\nRSpec parallel worker #{@number} (pid #{Process.pid}) is exiting because the " \ @@ -98,6 +108,14 @@ def watch_coordinator thread end + # @private + def process_alive?(pid) + Process.kill(0, pid) + true + rescue Errno::ESRCH, Errno::EPERM + false + end + # @private def prepare Runner.disable_autorun! @@ -135,7 +153,8 @@ def trap_interrupt # @private def load_spec_files forwarding = @configuration.reporter - @configuration.reporter = loading = LoadReporter.new(@configuration) + # A spawned worker's also saw its `--require`d files load. + @configuration.reporter = loading = @loading || LoadReporter.new(@configuration) @configuration.load_spec_files @configuration.reporter = forwarding @groups_by_id = @world.example_groups.to_h { |group| [group.id, group] } diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 1233505b3..4464ee3e4 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -102,11 +102,17 @@ def fork_worker(process, units, handles, worker_options={}) handles.each { |other| other.close_all unless other.equal?(self) } close_parent_ends Worker.new(@number, units, @child_jobs, @child_results, - worker_options.merge(:output => @output && @output.paths)).run + worker_options.merge(:output => @output&.paths)).run # :nocov: end end + # @return [Integer] the pid of a new process running `command`, its + # jobs on its standard input and its results on its standard output + def spawn_worker(process, command, environment) + process.spawn(environment, *command, :in => @child_jobs.io, :out => @child_results.io) + end + def close_parent_ends @jobs.close @results.close @@ -151,7 +157,12 @@ def reap(process) def signal(process, signal_name, escalation, deadline) @escalation = escalation self.deadline = deadline - process.kill(signal_name, @pid) if @pid + begin + process.kill(signal_name, @pid) if @pid + rescue Errno::EINVAL + # Windows cannot send SIGTERM. + process.kill('KILL', @pid) + end rescue Errno::ESRCH, Errno::EPERM nil end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 1774aaa05..7db578676 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -24,6 +24,8 @@ module WorkerLifecycle # itself and reports what it loaded. def start @started_at = now + # A spawned worker starts Ruby and loads the `--require`d files first. + @boot_allowance = Parallel.spawning? ? Process.times.then { |times| times.utime + times.stime } : 0.0 start_workers(nil) end @@ -75,7 +77,8 @@ def account_for_pauses # parent never holds more than its own ends of the earlier ones. def start_workers(units) Parallel.flush_output(@configuration) - @process.warmup if @process.respond_to?(:warmup) + # Shares more memory with forked workers; spawned ones share none. + @process.warmup if @process.respond_to?(:warmup) && !Parallel.spawning? @output_dir ||= WorkerOutput.make_directory @worker_count.times do |index| @@ -109,6 +112,43 @@ def fork_worker(worker, workers) worker.fork_worker(@process, @units, workers, :signal_handlers => @signal_handlers) end + # The `rspec` command this process runs, run again by the same Ruby + # with this process's load path. + def spawn_worker(worker, _workers) + script, args = Runner.command_line + command = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", script, *args] + environment = SpawnedWorker.environment(worker.number, Process.pid, @configuration.seed, worker.output&.paths) + worker.spawn_worker(@process, command, environment) + end + + # What a worker that loaded the spec files itself printed while it + # did is what this process printed when it loaded them. + def output(worker, sizes) + return unless worker.output + return worker.output.skip(sizes) if worker.loading + + worker.output.read(sizes) { |stream, bytes| show_output(worker, stream, bytes) } + end + + # What is left in a worker's output files once it has gone: what it + # printed after it last reported, such as a crash report or what its + # `at_exit` handlers printed. + def drain_output(worker) + return unless worker.output + + worker.output.read { |stream, bytes| show_output(worker, stream, bytes) } + worker.output.close + end + + # A worker dismissed before the run used it has only loaded the spec + # files, which this process did too; before the run reports, no + # worker has run anything. + def show_output(worker, stream, bytes) + return if worker.state == :dismissed || @sequencer.nil? + + @sequencer.output(worker.number, stream, bytes) + end + # Including those a failed `fork` left without a process. def live_workers @workers.each { |worker| worker.state = :exited if worker.pid.nil? } @@ -121,7 +161,7 @@ def live_workers def expect_loading_workers @suite_digest = Parallel.suite_digest(@world) @load_failed = @world.non_example_failure - deadline = now + (now - @started_at) + @configuration.parallel_shutdown_timeout + deadline = now + (now - @started_at) + @boot_allowance + @configuration.parallel_shutdown_timeout @load_allowance = deadline - @started_at @workers.each { |worker| worker.deadline = deadline if worker.loading } end diff --git a/rspec-core/lib/rspec/core/parallel/worker_output.rb b/rspec-core/lib/rspec/core/parallel/worker_output.rb index 541a9933b..9fffd5d6b 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_output.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_output.rb @@ -24,6 +24,18 @@ def self.make_directory nil end + # Shows what a worker printed where this process prints. + def self.write(stream, bytes) + io = stream == :err ? $stderr : $stdout + # What formatters printed before it comes first. + $stdout.flush if stream == :err + bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !io.binmode? + io.write(bytes) + io.flush + rescue IOError, SystemCallError + nil + end + def self.remove_directory(dir) return unless dir @@ -35,7 +47,7 @@ def self.remove_directory(dir) def initialize(dir, number) @paths = STREAMS.to_h { |stream| [stream, File.join(dir, "worker-#{number}.#{stream}")] } - @paths.each_value { |path| File.open(path, 'wb') {} } + @paths.each_value { |path| File.binwrite(path, '') } @readers = {} @read = { :out => 0, :err => 0 } end @@ -83,9 +95,11 @@ class Capture def initialize(paths, results) @results = results @files = paths.transform_values { |path| File.open(path, 'ab') } - STDOUT.reopen(@files[:out]) - STDERR.reopen(@files[:err]) - [STDOUT, STDERR].each do |io| + # The process's own streams, which processes it starts inherit, + # not whatever `$stdout` and `$stderr` point to. + STDOUT.reopen(@files[:out]) # rubocop:disable Style/GlobalStdStream + STDERR.reopen(@files[:err]) # rubocop:disable Style/GlobalStdStream + [STDOUT, STDERR].each do |io| # rubocop:disable Style/GlobalStdStream io.binmode io.sync = true end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 3969b0f59..448a1fe46 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -174,10 +174,7 @@ def metadata_changes(metadata) @metadata_before = nil changes = {} metadata.each do |key, value| - next if NOT_COMPARED.key?(key) - - old = before[key] - next if old && old[0].equal?(value) && old[1] == hash_of(value) + next if NOT_COMPARED.key?(key) || unchanged?(before[key], value) if marshalable?([key, value]) (changes[:set] ||= {})[key] = value @@ -190,6 +187,10 @@ def metadata_changes(metadata) changes.empty? ? nil : changes end + def unchanged?(snapshot, value) + snapshot && snapshot[0].equal?(value) && snapshot[1] == hash_of(value) + end + def marshalable?(value) Marshal.dump(value) true diff --git a/rspec-core/lib/rspec/core/runner.rb b/rspec-core/lib/rspec/core/runner.rb index 23c3af44b..0d4a8b82f 100644 --- a/rspec-core/lib/rspec/core/runner.rb +++ b/rspec-core/lib/rspec/core/runner.rb @@ -42,10 +42,26 @@ def self.perform_at_exit # code. def self.invoke disable_autorun! + # A worker of a parallel run on a platform without `fork`, started as + # a new `rspec` process; it ends the process. + if ENV.key?('RSPEC_PARALLEL_WORKER') + RSpec::Support.require_rspec_core "parallel/coordinator" + Parallel::SpawnedWorker.run(ARGV) + end + @command_line = [File.file?($0) ? File.expand_path($0) : RSpec::Core.path_to_executable, ARGV.dup] status = run(ARGV, $stderr, $stdout).to_i exit(status) if status != 0 end + # @private + # @return [Array(String, Array), nil] the script and arguments + # of the `rspec` command this process runs, which starts the workers + # of a parallel run on a platform without `fork`; `nil` when the run + # was not started by `.invoke` + def self.command_line + @command_line ||= nil + end + # Run a suite of RSpec examples. Does not exit. # # This is used internally by RSpec to run a suite, but is available diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index d769a10e2..78d607c07 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -2,11 +2,28 @@ require 'support/aruba_support' require 'json' +# Runs `rspec` in a process of its own, where the `--require`d files load +# afresh. Given at most 60 seconds. +module ParallelRunProcess + def rspec_process(*args) + cd('.') do + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 + sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || + Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + Process.kill(:KILL, pid) && Process.wait(pid) unless status + [status, File.read('stdout.txt'), File.read('stderr.txt')] + end + end +end + # The longest group of rspec-core's suite: under `--parallel` it runs as parts in # several workers (each has its own aruba directory). RSpec.describe 'Parallel execution (--parallel)', :slow, :parallel_split => 4, :skip => !RSpec::Support::RubyFeatures.fork_supported? do include_context "aruba support" + include ParallelRunProcess # The coordinator refuses to nest inside another parallel worker, which is # what this very process is when rspec-core's suite runs under `--parallel`. @@ -21,20 +38,6 @@ def run_rspec(cmd) run_command(cmd) end - # A process of its own, where the `--require`d files load afresh. Given at - # most 60 seconds. - def rspec_process(*args) - cd('.') do - rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] - pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') - deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 - sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || - Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline - Process.kill(:KILL, pid) && Process.wait(pid) unless status - [status, File.read('stdout.txt'), File.read('stderr.txt')] - end - end - # Written by the workers' `before(:suite)` hooks: a file can exist before # its pid has been written to it. def worker_pids @@ -325,7 +328,7 @@ def example_passed(notification) run_rspec "#{options} --parallel=2" parallel = last_cmd_stdout.scan(/META .*$/).sort expect(parallel).to eq(serial.map { |line| line.sub(/ Proc$/, ' NilClass') }) - expect(last_cmd_stderr.scan(/set example metadata :callback to a Proc/).size).to eq(1) + expect(last_cmd_stderr.scan("set example metadata :callback to a Proc").size).to eq(1) expect(last_cmd_exit_status).to eq(0) end @@ -1276,3 +1279,79 @@ def loaders end end end + +# Where Ruby has no `fork` (Windows, JRuby, TruffleRuby) the workers are new +# `rspec` processes; elsewhere a `--require`d file takes `fork` away. +RSpec.describe 'Parallel execution without fork (spawned workers)', :slow do + include_context "aruba support" + include ParallelRunProcess + + def without_fork + return [] unless RSpec::Support::RubyFeatures.fork_supported? + + write_file 'no_fork.rb', "module Process\n class << self\n undef_method :fork\n end\nend\n" + %w[--require ./no_fork] + end + + # Each group's first example waits for both workers, so that neither runs + # every group before the other has started. + def write_suite(helper="") + write_file 'spec/spec_helper.rb', " + RSpec.configure do |c| + c.before(:suite) { File.write(\"worker_\#{c.parallel_worker_number}.txt\", [Process.pid, ENV['TEST_ENV_NUMBER'].inspect, c.seed].join(' ')) } + end + def both_workers_started + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 30 + sleep 0.05 until Dir['worker_*.txt'].size == 2 || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + end + #{helper} + " + 3.times do |i| + write_file "spec/s#{i}_spec.rb", " + RSpec.describe('S#{i}') do + it('passes') { both_workers_started; puts 'printed by S#{i}' } + it('#{i == 1 ? 'fails' : 'passes too'}') { expect(#{i}).not_to eq(1) } + end + " + end + end + + it 'runs the groups in new rspec processes, with the run’s seed, and reports one run' do + write_suite + options = without_fork + %w[--require ./spec/spec_helper --parallel=2 --seed 1234 --format progress + --format json --out results.json] + + status, stdout, stderr = rspec_process(*options) + + expect(stdout).to include("6 examples, 1 failure", "Randomized with seed 1234", "printed by S0", "printed by S2") + expect(stderr).not_to include("Ignoring `--parallel`") + expect(status.exitstatus).to eq(1) + results = cd('.') { JSON.parse(File.read('results.json')) } + expect(results['summary']).to include('example_count' => 6, 'failure_count' => 1) + workers = cd('.') { Dir['worker_*.txt'].map { |file| File.read(file).split } } + expect(workers.size).to eq(2) + expect(workers.map(&:first).uniq.size).to eq(2) + expect(workers.map { |worker| worker.drop(1) }).to eq([['""', '1234'], ['"2"', '1234']]) + end + + it 'fails the example a spawned worker was running when it is killed, and runs the rest' do + write_suite "RSpec.describe('Killed') { it('dies') { both_workers_started; Process.kill(:KILL, Process.pid) } }" + + status, stdout, _stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') + + expect(stdout).to include("7 examples, 2 failures", "exited unexpectedly", "while running this example") + expect(status.exitstatus).to eq(1) + end + + it 'aborts spawned workers that do not exit within the shutdown timeout, and shows their thread dumps' do + write_suite "RSpec.configure { |c| c.parallel_shutdown_timeout = 1; c.after(:suite) { sleep } }" + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + + status, stdout, stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') + + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 30 + expect(stdout).to include("6 examples, 1 failure") + expect(stderr.scan("did not exit within 1 second of finishing its last example group").size).to eq(2) + expect(status.exitstatus).to eq(1) + end +end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index efdd7f525..dfe5870aa 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -124,9 +124,7 @@ def spawn(handle, _handles) when :never_loads, :never_starts, :starts_once_asked_to_quit then stall(behaviour, handle, results) when :dies_loading then Thread.current.kill else - print_while_loading(handle, results) if behaviour == :prints_while_loading - print_to(handle, results, "loaded quietly\n", "", false) if behaviour == :prints_unannounced_while_loading - report_loaded(results, behaviour) if handle.loading + load_spec_files(handle, results, behaviour) send_extra_frames(results, behaviour, handle) run_fake_worker(handle.number, jobs, results, behaviour) { |status| exit_status = status } end @@ -195,19 +193,25 @@ def send_extra_frames(results, behaviour, handle) send_unknown_ids(results) if behaviour == :unknown_ids results.send([:expectations, 3]) if behaviour == :counts_expectations print_to(handle, results, "printed by worker #{handle.number}\n", "warned by worker #{handle.number}\n") if behaviour == :prints - print_to(handle, results, "left behind by worker #{handle.number}\n", "", false) if behaviour == :prints_unannounced + print_to(handle, results, "left behind by worker #{handle.number}\n", "", :announce => false) if behaviour == :prints_unannounced end # As a worker's own output goes to the files once it has moved its # standard output and error there; the frame is what it sends next. - def print_to(handle, results, out, err, announce=true) + def print_to(handle, results, out, err, announce: true) File.open(handle.output.paths[:out], 'ab') { |file| file.write(out) } File.open(handle.output.paths[:err], 'ab') { |file| file.write(err) } results.send([:output, handle.output.paths.transform_values { |path| File.size(path) }]) if announce end - def print_while_loading(handle, results) - print_to(handle, results, "noise while loading\n", "") + def load_spec_files(handle, results, behaviour) + print_while_loading(handle, results, behaviour) + report_loaded(results, behaviour) if handle.loading + end + + def print_while_loading(handle, results, behaviour) + print_to(handle, results, "noise while loading\n", "") if behaviour == :prints_while_loading + print_to(handle, results, "loaded quietly\n", "", :announce => false) if behaviour == :prints_unannounced_while_loading end def send_unknown_ids(results) @@ -273,6 +277,10 @@ def started_coordinator(units, worker_count=2, options={}) new_coordinator(units, worker_count, options).tap(&:start) end + # The workers here are threads standing in for forked processes, also + # where Ruby has no `fork`. + before { allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(false) } + let(:short_graces) { { :term_grace => 0.1, :kill_grace => 0.1 } } def passing_group(name) @@ -910,6 +918,32 @@ def standard_streams(stdout=StringIO.new) $stdout, $stderr = streams end + it "starts workers as new `rspec` processes where there is no `fork`, giving them longer to load" do + allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(true) + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", %w[--tag fast]]) + spawned = [] + killed = [] + fake = double(:process) + allow(fake).to receive(:spawn) { |env, *command, redirects| (spawned << [env, command, redirects.keys]) && (4242 + spawned.size) } + allow(fake).to receive(:kill) { |_signal, pid| killed << pid } + allow(fake).to receive(:waitpid2) { |pid, _flags| killed.include?(pid) ? [pid, nil] : nil } + config.seed = 99 + + instance = Coordinator.new(runner, 2, :process => fake, :tick => 0.01, :kill_grace => 1) + instance.start + instance.dismiss + + expect(spawned.size).to eq(2) + env, command, redirects = spawned.first + expect(command).to eq([RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", "/app/bin/rspec", "--tag", "fast"]) + expect(env).to include("RSPEC_PARALLEL_WORKER" => "1", "RSPEC_PARALLEL_COORDINATOR" => Process.pid.to_s, + "RSPEC_PARALLEL_SEED" => "99", "TEST_ENV_NUMBER" => "") + expect(env["RSPEC_PARALLEL_STDOUT"]).to end_with("worker-1.out") + expect(redirects).to eq([:in, :out]) + expect(killed).to eq([4243, 4244]) + expect(instance.instance_variable_get(:@boot_allowance)).to be > 0 + end + it "reports the groups that never ran when no worker remains" do crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } groups = [crashing, RSpec.describe("Also crashing") { it("dies") { Thread.current.kill } }, passing_group("C"), passing_group("D")] diff --git a/rspec-core/spec/rspec/core/parallel/plan_spec.rb b/rspec-core/spec/rspec/core/parallel/plan_spec.rb index f1047af93..fe52ca373 100644 --- a/rspec-core/spec/rspec/core/parallel/plan_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/plan_spec.rb @@ -13,8 +13,37 @@ def groups(count) Array.new(count) { |i| RSpec.describe("G#{i}") { it("e") {} } } end + describe "where there is no `fork`" do + before do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(["exe/rspec", []]) + allow(config).to receive(:files_to_run).and_return(%w[./spec/a_spec.rb ./spec/b_spec.rb ./spec/c_spec.rb]) + end + + it "starts the workers before loading, as new processes that load the spec files, whatever `parallel_preload` says" do + config.parallel_workers = 2 + expect(Parallel::Coordinator).to receive(:new).with(runner, 2).and_return(started) + + plan.before_loading + + all = groups(3) + expect(plan.coordinator_for(all)).to eq(started) + expect(started).to have_received(:assign).with(all, anything) + end + + it "warns, and runs serially, when the worker count is set while the spec files load" do + plan.before_loading + config.parallel_workers = 2 + expect(RSpec).to receive(:warning).with(a_string_including("`config.parallel_workers` was set while the spec files"), + :call_site => nil) + + expect(plan.coordinator_for(groups(3))).to be_nil + end + end + describe "by default, the spec files being loaded before the workers are forked" do it "forks nothing before loading, then leaves the decision to Parallel.coordinator_for" do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) config.parallel_workers = 2 expect(Parallel::Coordinator).not_to receive(:new) plan.before_loading @@ -94,7 +123,8 @@ def spec_files(count) it "runs serially, without forking or warning again, when a parallel run is not possible" do config.parallel_workers = 2 allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) - expect(RSpec).to receive(:warning).once.with(a_string_including("needs `fork`"), :call_site => nil) + allow(Runner).to receive(:command_line).and_return(nil) + expect(RSpec).to receive(:warning).once.with(a_string_including("without `fork`"), :call_site => nil) expect(Parallel::Coordinator).not_to receive(:new) plan.before_loading diff --git a/rspec-core/spec/rspec/core/parallel/results_spec.rb b/rspec-core/spec/rspec/core/parallel/results_spec.rb index 7bc024545..34e4e03ac 100644 --- a/rspec-core/spec/rspec/core/parallel/results_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/results_spec.rb @@ -13,12 +13,12 @@ def payload(metadata) end it "applies the metadata the worker's run set and deleted when it applies the result" do - results.record(example, payload(:set => { :tags => ["a", "b"], :screenshot => "x.png" }, :deleted => [:gone])) + results.record(example, payload(:set => { :tags => %w[a b], :screenshot => "x.png" }, :deleted => [:gone])) expect(example.metadata).to include(:gone => true, :tags => ["a"]) results.apply(example) - expect(example.metadata).to include(:tags => ["a", "b"], :screenshot => "x.png", :kept => "k") + expect(example.metadata).to include(:tags => %w[a b], :screenshot => "x.png", :kept => "k") expect(example.metadata).not_to have_key(:gone) expect(example.execution_result.status).to eq(:passed) end @@ -28,7 +28,7 @@ def payload(metadata) results.record(example, payload(:unsent => { ":callback" => "Proc" })) results.apply(example) - expect(example.metadata.reject { |key, _| key == :execution_result }).to eq(before.reject { |key, _| key == :execution_result }) + expect(example.metadata.except(:execution_result)).to eq(before.except(:execution_result)) end end end diff --git a/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb new file mode 100644 index 000000000..567e2eee3 --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb @@ -0,0 +1,81 @@ +require 'tempfile' +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe SpawnedWorker do + describe ".environment" do + it "tells a worker its number, the coordinator, the seed and its output files, and sets TEST_ENV_NUMBER" do + env = SpawnedWorker.environment(2, 4242, 1234, :out => "/tmp/w.out", :err => "/tmp/w.err") + + expect(env).to eq("RSPEC_PARALLEL_WORKER" => "2", "RSPEC_PARALLEL_COORDINATOR" => "4242", + "RSPEC_PARALLEL_SEED" => "1234", "TEST_ENV_NUMBER" => "2", + "RSPEC_PARALLEL_STDOUT" => "/tmp/w.out", "RSPEC_PARALLEL_STDERR" => "/tmp/w.err") + expect(SpawnedWorker.environment(1, 4242, 1234, nil)).to include("TEST_ENV_NUMBER" => "") + expect(SpawnedWorker.environment(1, 4242, 1234, nil)).not_to have_key("RSPEC_PARALLEL_STDOUT") + end + end + + describe ".settings" do + it "takes what the coordinator told it out of the environment, so that processes it starts are not workers" do + env = SpawnedWorker.environment(3, 4242, 99, :out => "o", :err => "e").merge("OTHER" => "kept") + + expect(SpawnedWorker.settings(env)).to eq(:number => 3, :coordinator => 4242, :seed => 99, + :output => { :out => "o", :err => "e" }) + expect(env).to eq("TEST_ENV_NUMBER" => "3", "OTHER" => "kept") + end + + it "has no output files when it was given none" do + env = SpawnedWorker.environment(3, 4242, 99, nil) + expect(SpawnedWorker.settings(env)[:output]).to be_nil + end + end + + describe ".configure" do + let(:configuration) { RSpec::Core::Configuration.new } + # Instead of this project's `.rspec` and the user's. + let(:options_file) { Tempfile.new('rspec-options').tap(&:close).path } + + def configure(args, settings) + SpawnedWorker.configure(['-O', options_file] + args, settings, configuration) + end + + it "configures the worker from the command line, without the formatters and the files the coordinator writes" do + configure(%w[--format json --out results.json --deprecation-out deprecations.txt --tag fast], + :number => 2, :seed => 1234) + + expect(configuration.formatters).to be_empty + expect(configuration.deprecation_stream).not_to eq("deprecations.txt") + expect(configuration.inclusion_filter.rules).to eq(:fast => true) + expect(configuration.parallel_worker_number).to eq(2) + expect(File.exist?("results.json")).to be false + end + + it "takes the run's seed without choosing an ordering, which the run's configuration does" do + configure(%w[--order defined], :number => 2, :seed => 1234) + + expect(configuration.seed).to eq(1234) + expect(configuration.ordering_registry.fetch(:global)).to be_a(RSpec::Core::Ordering::Identity) + end + + it "keeps a seed given on the command line, which is the run's too" do + configure(%w[--seed 77], :number => 2, :seed => 77) + + expect(configuration.seed).to eq(77) + expect(configuration.ordering_registry.fetch(:global)).to be_a(RSpec::Core::Ordering::Random) + end + end + end + + RSpec.describe RSpec::Core::Ordering::ConfigurationManager do + it "adopts a seed without changing the ordering, unless a seed was forced" do + manager = RSpec::Core::Ordering::ConfigurationManager.new + manager.adopt_seed(12) + expect(manager.seed).to eq(12) + expect(manager.seed_used?).to be false + + manager.force(:seed => 34) + manager.adopt_seed(56) + expect(manager.seed).to eq(34) + end + end +end diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb index 51e21dabe..c5a0eacee 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -10,6 +10,39 @@ module RSpec::Core::Parallel expect(first.map(&:closed?)).to eq([true, true]) end + describe "#signal" do + let(:handle) { WorkerHandle.new(1).tap { |h| h.pid = 4242 } } + let(:sent) { [] } + + after { handle.close_all } + + it "sends SIGKILL where SIGTERM cannot be sent (Windows)" do + process = double(:process) + allow(process).to receive(:kill) do |name, pid| + raise Errno::EINVAL if name == 'TERM' + + sent << [name, pid] + end + + handle.signal(process, 'TERM', :term, 1.0) + + expect(sent).to eq([['KILL', 4242]]) + expect(handle.escalation).to eq(:term) + end + end + + describe "#spawn_worker" do + it "starts the command with its jobs on standard input and its results on standard output" do + handle = WorkerHandle.new(2) + process = double(:process) + expect(process).to receive(:spawn).with({ "A" => "1" }, "ruby", "rspec", :in => handle.child_jobs.io, + :out => handle.child_results.io).and_return(4242) + + expect(handle.spawn_worker(process, %w[ruby rspec], "A" => "1")).to eq(4242) + handle.close_all + end + end + describe "the deadline for its first frame" do let(:handle) { WorkerHandle.new(1) } diff --git a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb index 02977eecb..30c0d74b0 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb @@ -2,6 +2,8 @@ require 'tmpdir' require 'rspec/core/parallel/coordinator' +# These specs stand in for the process's own streams, which a worker moves. +# rubocop:disable-next Style/GlobalStdStream module RSpec::Core::Parallel RSpec.describe WorkerOutput do let!(:dir) { Dir.mktmpdir } diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index b4a3fabdc..3711900e3 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -196,7 +196,7 @@ def listener.example_failed(notification) it("changes", :tags => ["a"], :same => "s") do |ex| ex.metadata[:added] = { :path => "shot.png" } ex.metadata[:tags] << "b" - ex.metadata[:same] = ex.metadata[:same] + ex.metadata.store(:same, ex.metadata[:same]) ex.metadata.delete(:gone) end it("changes nothing") {} @@ -205,7 +205,7 @@ def listener.example_failed(notification) payloads = frames.select { |f| f.first == :example_finished }.to_h { |f| [f.last[:description], f.last[:metadata]] } changed, unchanged = payloads.values_at("changes", "changes nothing") - expect(changed).to eq(:set => { :added => { :path => "shot.png" }, :tags => ["a", "b"] }, :deleted => [:gone]) + expect(changed).to eq(:set => { :added => { :path => "shot.png" }, :tags => %w[a b] }, :deleted => [:gone]) expect(unchanged).to be_nil end diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 2251487c2..3ae3553f2 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -2,6 +2,8 @@ require 'tmpdir' require 'rspec/core/parallel/coordinator' +# These specs stand in for the process's own streams, which a worker moves. +# rubocop:disable-next Style/GlobalStdStream module RSpec::Core::Parallel RSpec.describe Worker do let(:jobs) { Channel.pipe } @@ -84,6 +86,26 @@ def frames expect(File.exist?(dir)).to be false end + it "exits when the coordinator's pid no longer names a process, its parent unchanged (Windows)" do + stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) + answers = [true, true, false] + instance = Worker.new(2, groups, jobs.first, results.last, + :stderr => stderr, :coordinator => 100, :ppid => lambda { 100 }, + :alive => lambda { |pid| pid == 100 && answers.shift }, + :exit_now => lambda { |status| exits << [:exit_now, status] }) + + expect(instance.watch_coordinator.join(5)).to be_truthy + expect(exits).to eq([[:exit_now, 1]]) + expect(stderr.string).to include("the coordinating process (pid 100) is gone") + end + + it "tells a live process from one that is gone" do + expect(worker.process_alive?(Process.pid)).to be true + gone = Process.spawn(RbConfig.ruby, '-e', 'exit') + Process.wait(gone) + expect(worker.process_alive?(gone)).to be false + end + it "still exits when it cannot say why (its stderr is gone too)" do stub_const("RSpec::Core::Parallel::Worker::COORDINATOR_POLL", 0.01) parents = [100, 1] diff --git a/rspec-core/spec/rspec/core/parallel_spec.rb b/rspec-core/spec/rspec/core/parallel_spec.rb index 323a7bf2a..76cfab288 100644 --- a/rspec-core/spec/rspec/core/parallel_spec.rb +++ b/rspec-core/spec/rspec/core/parallel_spec.rb @@ -63,12 +63,24 @@ def groups(count) expect(Parallel.coordinator_for(runner, groups(3))).to be_nil end - it "warns and runs serially where fork is not available" do + it "warns and runs serially where fork is not available and the run was not started by the `rspec` command" do config.parallel_workers = 2 allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) - expect(RSpec).to receive(:warning).with(a_string_including("needs `fork`", "Running serially"), :call_site => nil) + allow(Runner).to receive(:command_line).and_return(nil) + expect(RSpec).to receive(:warning).with(a_string_including("without `fork`", "started by the `rspec` command", + "Running serially"), :call_site => nil) - expect(Parallel.coordinator_for(runner, groups(3))).to be_nil + expect(Parallel.requested_worker_count(config)).to be_nil + end + + it "has the workers started as new `rspec` processes where fork is not available" do + config.parallel_workers = 3 + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(false) + allow(Runner).to receive(:command_line).and_return(["exe/rspec", []]) + expect(RSpec).not_to receive(:warning) + + expect(Parallel.requested_worker_count(config)).to eq(3) + expect(Parallel).to be_spawning end it "warns and runs serially inside another tool's parallel worker", :skip => !RSpec::Support::RubyFeatures.fork_supported? do diff --git a/rspec-core/spec/rspec/core/runner_spec.rb b/rspec-core/spec/rspec/core/runner_spec.rb index 6b3272c78..90c82126b 100644 --- a/rspec-core/spec/rspec/core/runner_spec.rb +++ b/rspec-core/spec/rspec/core/runner_spec.rb @@ -297,6 +297,37 @@ def interrupt expect(runner).to receive(:exit).with(123) runner.invoke end + + context "remembering the script and arguments it runs, which a parallel run without `fork` starts its workers with" do + before do + allow(runner).to receive(:run) { 0 } + stub_const("ARGV", %w[--tag fast]) + allow(File).to receive(:file?).and_call_original + end + + after { runner.instance_variable_set(:@command_line, nil) } + + it "takes the script that started the process" do + allow(File).to receive(:file?).with($0).and_return(true) + runner.invoke + expect(runner.command_line).to eq([File.expand_path($0), %w[--tag fast]]) + end + + it "takes rspec-core's `rspec` when no script started it (`ruby -e`)" do + allow(File).to receive(:file?).with($0).and_return(false) + runner.invoke + expect(runner.command_line).to eq([RSpec::Core.path_to_executable, %w[--tag fast]]) + end + end + + it "runs as the parallel worker a coordinator without `fork` started it as" do + RSpec::Support.require_rspec_core "parallel/coordinator" + stub_const("ARGV", %w[--tag fast]) + expect(RSpec::Core::Parallel::SpawnedWorker).to receive(:run).with(%w[--tag fast]) { throw :exited } + expect(runner).not_to receive(:run) + + with_env_vars('RSPEC_PARALLEL_WORKER' => '2') { catch(:exited) { runner.invoke } } + end end describe ".run" do @@ -365,6 +396,7 @@ def interrupt describe "#run" do it 'hands the ordered example groups to a parallel coordinator when one applies, then persists example statuses' do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } RSpec.describe("group 2") { example("passing") { expect(1).to eq(1) } } allow(config).to receive(:load_spec_files) @@ -389,6 +421,7 @@ def interrupt end it 'plans a parallel run that the spec files configure, once they have loaded' do + allow(RSpec::Support::RubyFeatures).to receive(:fork_supported?).and_return(true) RSpec.describe("group 1") { example("passing") { expect(1).to eq(1) } } allow(config).to receive(:load_spec_files) { config.parallel_workers = 2 } expect(Parallel).to receive(:coordinator_for).and_return(nil) From 163d45993d0341dddd37f454c29d9bbd93221018 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:39:39 +0200 Subject: [PATCH 49/58] [rspec-core] Replace a --parallel worker that crashes while groups wait to run A worker lost while it ran groups (crashed, killed, terminated for being stuck, or stuck in `fork`) is replaced by one with its number and `TEST_ENV_NUMBER`, which runs what is left; its failed example stays failed. Not once the run stops early or ends by `exit` or a signal, not for a worker lost while loading, and no more often than the run has workers. Under `--no-parallel-preload` with `fork` the replacement is a new `rspec` process: forking the loaded coordinator is what that mode avoids. Output files are per worker generation, so a replacement never truncates a predecessor's. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 3 +- rspec-core/README.md | 10 ++- .../lib/rspec/core/parallel/coordinator.rb | 4 + .../lib/rspec/core/parallel/worker_exits.rb | 2 + .../lib/rspec/core/parallel/worker_handle.rb | 9 +- .../rspec/core/parallel/worker_lifecycle.rb | 49 +++++++++-- rspec-core/spec/integration/parallel_spec.rb | 28 +++++- .../rspec/core/parallel/coordinator_spec.rb | 87 +++++++++++++++++-- 8 files changed, 172 insertions(+), 20 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 25fb9fdf3..533028cf8 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -21,7 +21,8 @@ Enhancements: threads, so a stuck `at_exit` handler can no longer hang a run (nor can one that runs after code under test called `exit`); neither can a worker stuck in `fork`. A worker that crashes fails the example it was running instead of taking the run down, while - the examples of its group it had not started run in another worker. Code that calls + the examples of its group it had not started run in another worker, and a new worker + with its `TEST_ENV_NUMBER` takes its place. Code that calls `exit` ends the run with that status, as it does serially. A top-level group tagged `:parallel_split => N` runs as N parts in different workers. A run stopped with SIGTERM or SIGHUP says which example each worker diff --git a/rspec-core/README.md b/rspec-core/README.md index 213dda2b0..06a8bcd20 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -398,8 +398,14 @@ again -- whatever it did may be why the worker died; the examples of its group it had not started yet run in another worker. One lost in a `before(:context)` or `after(:context)` hook fails the examples of that group it had not started, as such a hook raising an error would, and the rest of its group runs -elsewhere. Workers that run out of work wait for the busy ones, so that there -is a worker to run what a lost one leaves; if none remains, the run says which +elsewhere. While groups wait to run, a lost worker is replaced by a new one +with its number and `TEST_ENV_NUMBER` -- not once the run stops early (fail-fast, +an interrupt, `exit`), not for a worker lost while loading the spec files, and +no more often than the run has workers; under `--no-parallel-preload` with +`fork` the new one is a new `rspec` process, as without `fork`, since forking +the coordinating process after it loaded the spec files is what that setting +avoids. Workers that run out of work wait for the busy ones, so that there is +a worker to run what a lost one leaves; if none remains, the run says which examples did not run and how to run them. Problems between example groups count as errors outside of examples. diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index c5d2e964f..33cf3a4eb 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -63,6 +63,8 @@ def initialize(runner, worker_count, options={}) @last_turn_at = nil @output_dir = nil @boot_allowance = 0.0 + @load_themselves = false + @replacements = 0 end # The groups to run. Workers forked before loading that are more than @@ -399,6 +401,7 @@ def reap_workers worker.exit_status = status judge_exit(worker, status) worker.state = :exited + replace(worker) end end @@ -425,6 +428,7 @@ def escalate(worker) drain_output(worker) abort_unit(worker) worker.state = :abandoned + replace(worker) end end diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 976b554a2..5d27d0f42 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -114,6 +114,7 @@ def replace_exit_status(worker, exit_status, cause) # nothing, so the group it was given goes to another worker as is. def never_started(worker) worker.loading = false + worker.lost = true index = worker.current_unit requeue = index && accepting_work? report "#{worker.description} never reported back after it was forked (stuck before running any of " \ @@ -142,6 +143,7 @@ def never_started_outcome(index, requeue) def lost(worker, what, consequence='') spoil_exit because = "#{consequence}#{worker.exit_cause && ": #{worker.exit_cause}"}" + worker.lost = !worker.loading if worker.loading worker.loading = false error("#{worker.description} #{what} while loading the spec files#{consequence}.") diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 4464ee3e4..6d1622146 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -18,10 +18,15 @@ class WorkerHandle attr_accessor :starting # Its standard output and error, or `nil` when it writes where the run does. attr_reader :output + # Lost while it ran groups: crashed, killed or terminated. + attr_accessor :lost - def initialize(number, output_dir=nil) + # A worker started in place of a lost one has its number and a later + # `generation`. + def initialize(number, output_dir=nil, generation=1) @number = number - @output = output_dir && WorkerOutput.new(output_dir, number) + @output = output_dir && WorkerOutput.new(output_dir, generation == 1 ? number : "#{number}-#{generation}") + @lost = false @child_jobs, @jobs = Channel.pipe begin @results, @child_results = Channel.pipe diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 7db578676..43569f928 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -24,6 +24,7 @@ module WorkerLifecycle # itself and reports what it loaded. def start @started_at = now + @load_themselves = true # A spawned worker starts Ruby and loads the `--require`d files first. @boot_allowance = Parallel.spawning? ? Process.times.then { |times| times.utime + times.stime } : 0.0 start_workers(nil) @@ -86,17 +87,17 @@ def start_workers(units) end end - # @return [Boolean] whether the worker started - def start_worker(number, units) - worker = WorkerHandle.new(number, @output_dir) + # @return [WorkerHandle, nil] the worker, if it started + def start_worker(number, units, starter=@spawn, generation=1) + worker = WorkerHandle.new(number, @output_dir, generation) worker.loading = units.nil? @workers << worker - worker.pid = @spawn.call(worker, @workers) + worker.pid = starter.call(worker, @workers) worker.close_child_ends worker.starting = true # A worker forked before loading gets the loading deadline instead. worker.start_deadline = now + @configuration.parallel_shutdown_timeout unless worker.loading - true + worker rescue *OUT_OF_RESOURCES => e if worker @workers.delete(worker) @@ -105,7 +106,43 @@ def start_worker(number, units) RSpec.warning "Started #{@workers.size} of #{@worker_count} parallel workers: the next could not be started " \ "(#{e.class}: #{e.message}). #{@workers.empty? ? 'Running serially' : 'Going on without it'}", :call_site => nil - false + nil + end + + # A worker lost while it ran groups is replaced by one with its + # number -- and so its `TEST_ENV_NUMBER` -- while groups wait to run + # and the run goes on: not once it stops early or ends by `exit` or a + # signal, and no more often than there are workers. The examples the + # lost worker failed stay failed. + def replace(worker) + return unless worker.lost && accepting_work? && !@queue.empty? && @exit_status.nil? && @termination_signal.nil? + return no_more_replacements if @replacements >= @worker_count + return unless (starter = replacement_starter) + + @replacements += 1 + replacement = start_worker(worker.number, @load_themselves ? nil : @units, starter, @replacements + 1) + return unless replacement + + replacement.deadline = now + @load_allowance if replacement.loading + report "#{replacement.description} was started in place of the lost worker (pid #{worker.pid})" \ + "#{' and loads the spec files first' if replacement.loading}." + end + + # Workers that loaded the spec files themselves are replaced the same + # way, by a new process when they were forked: forking this process, + # which has loaded them, is what the run is configured not to do. + def replacement_starter + return @spawn if !@load_themselves || Parallel.spawning? + + method(:spawn_worker) if Runner.command_line + end + + def no_more_replacements + return if @replacements_exhausted + + @replacements_exhausted = true + report "No further lost workers are replaced: the run has started as many in place of lost ones as it " \ + "has workers (#{@replacements})." end def fork_worker(worker, workers) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 78d607c07..e657d3cb3 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -357,7 +357,9 @@ def example_passed(notification) options = %w[--require ./spec/support/at_exit --format documentation --order defined --parallel=2] status, stdout, stderr = rspec_process(*options) - blocks = stdout.split(/^(?=P\d$)/).select { |block| block =~ /\AP\d$/ } + # The coordinator says so when it replaces the killed worker. + blocks = stdout.gsub(/^ *Parallel worker \d \(pid \d+\) was started in place .*\n/, '').split(/^(?=P\d$)/) + blocks = blocks.select { |block| block =~ /\AP\d$/ } expect(blocks.size).to eq(3) blocks.each do |block| name = block[/\AP\d/] @@ -375,6 +377,30 @@ def example_passed(notification) expect(status.exitstatus).to eq(0) end + it 'starts a worker with the same TEST_ENV_NUMBER in place of one that crashed, which runs what was left' do + 4.times do |i| + write_file_formatted "spec/r#{i}_spec.rb", " + RSpec.describe('R#{i}') do + it('records') do + File.write('ran_r#{i}', [Process.pid, ENV['TEST_ENV_NUMBER'].inspect].join(' ')) + Process.kill(:KILL, Process.pid) if #{i} == 0 + sleep 1 if #{i} == 1 + end + end + " + end + + run_rspec '--parallel=2 --order defined' + + expect(last_cmd_stdout).to include("4 examples, 1 failure", "exited unexpectedly", "was started in place of the lost worker") + expect(last_cmd_exit_status).to eq(1) + ran = cd('.') { Dir['ran_r*'].to_h { |file| [file, File.read(file).split] } } + expect(ran.keys).to eq(%w[ran_r0 ran_r1 ran_r2 ran_r3]) + crashed_pid, crashed_env = ran['ran_r0'] + replacement = ran.values.find { |pid, env| env == crashed_env && pid != crashed_pid } + expect(replacement).not_to be_nil + end + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do write_file_formatted 'spec/support/my_formatter.rb', " class MyAppError < StandardError; end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index dfe5870aa..ff7b12499 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -107,8 +107,10 @@ def teardown_mocks_for_rspec; end # worker replaces its inherited copy, and with its own copies of the # pipe ends since the coordinator closes the child's ends after spawning # (as it must after a fork). + # A worker started in place of a lost one with the same number behaves as + # `worker_behaviours[[number, :replacement]]` says, normally by default. def spawn(handle, _handles) - behaviour = worker_behaviours.fetch(handle.number, :normal) + behaviour = behaviour_of(handle) jobs = Channel::Reader.new(handle.child_jobs.io.dup) results = Channel::Writer.new(handle.child_results.io.dup) exit_status = nil @@ -137,6 +139,12 @@ def spawn(handle, _handles) @process.register(thread) { exit_status && FakeProcess::Status.new(exit_status, nil) } end + def behaviour_of(handle) + replacement = (@started_numbers ||= []).include?(handle.number) + @started_numbers << handle.number + worker_behaviours.fetch(replacement ? [handle.number, :replacement] : handle.number, :normal) + end + # What a real worker sends first, which the fakes standing in for one send # too -- and then go on as they would, even if nothing reads it any more # (dismissed before their thread got going, say). @@ -944,15 +952,77 @@ def standard_streams(stdout=StringIO.new) expect(instance.instance_variable_get(:@boot_allowance)).to be > 0 end - it "reports the groups that never ran when no worker remains" do + it "reports the groups that never ran when no worker remains, once as many workers as the run has were replaced" do + crashing = Array.new(4) { |i| RSpec.describe("Crashing #{i}") { it("dies") { Thread.current.kill } } } + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + + exit_code = coordinator(crashing + [passing_group("C"), passing_group("D")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(messages.scan("was started in place of the lost worker").size).to eq(2) + expect(messages).to include("No further lost workers are replaced", "4 examples, 4 failures", + "2 example groups did not run because no worker remained to run them") + end + + it "starts a worker with the lost one's number in its place, which runs the remaining groups" do crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } - groups = [crashing, RSpec.describe("Also crashing") { it("dies") { Thread.current.kill } }, passing_group("C"), passing_group("D")] config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) - exit_code = coordinator(groups).run + exit_code = coordinator([crashing, passing_group("A"), passing_group("B"), passing_group("C")]).run + + expect(exit_code).to eq(config.failure_exit_code) + expect(@started_numbers.sort).to eq([1, 1, 2]) + expect(messages).to match(/Parallel worker 1 \(pid \d+\) was started in place of the lost worker \(pid \d+\)\./) + expect(messages).to include("4 examples, 1 failure", "exited unexpectedly") + end + + context "when the workers loaded the spec files themselves" do + let(:groups) do + [RSpec.describe("Crashing") { it("dies") { Thread.current.kill } }, passing_group("A"), passing_group("B"), passing_group("C")] + end + + before { config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) } + + it "starts a new `rspec` process in place of a lost one, since forking this process would load them twice" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", []]) + instance = started_coordinator(groups) + allow(instance).to receive(:spawn_worker) { |handle, handles| spawn(handle, handles) } + instance.assign(@units, units_of(@units)) + + expect(instance.run).to eq(config.failure_exit_code) + expect(instance).to have_received(:spawn_worker).once + expect(messages).to match(/Parallel worker 1 \(pid \d+\) was started in place of the lost worker \(pid \d+\) and loads the spec files first\./) + expect(messages).to include("4 examples, 1 failure") + end + + it "starts none in place of a lost one when the run was not started by the `rspec` command" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(nil) + instance = started_coordinator(groups) + instance.assign(@units, units_of(@units)) + + expect(instance.run).to eq(config.failure_exit_code) + expect(@started_numbers.sort).to eq([1, 2]) + expect(messages).to include("4 examples, 1 failure") + end + end + + it "starts no worker in place of one lost once the run stops early" do + config.fail_fast = 1 + quitting = world + config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) + failing = RSpec.describe("Failing") { it("fails") { raise "nope" } } + crashing = RSpec.describe("Crashing") do + it("dies once the run stops") do + sleep 0.01 until quitting.wants_to_quit + Thread.current.kill + end + end + + exit_code = coordinator([crashing, failing, passing_group("A"), passing_group("B")]).run expect(exit_code).to eq(config.failure_exit_code) - expect(messages).to include("2 example groups did not run because no worker remained to run them") + expect(@started_numbers.sort).to eq([1, 2]) + expect(messages).not_to include("was started in place") end it "stops dispatching when the world wants to quit, as on Ctrl-C" do @@ -1102,13 +1172,14 @@ def example_passed(_notification) it "is terminated once it misses its start deadline, and what no worker ran is reported" do config.parallel_shutdown_timeout = 0.05 worker_behaviours[1] = worker_behaviours[2] = :never_starts + worker_behaviours[[1, :replacement]] = worker_behaviours[[2, :replacement]] = :never_starts exit_code = coordinator([passing_group("A"), passing_group("B")]).run expect(exit_code).to eq(config.failure_exit_code) - expect(messages.scan("never reported back after it was forked").size).to eq(2) - expect(messages).to include("did not run because no worker remained to run them") - expect(process.signals.map(&:first)).to eq(%w[TERM TERM]) + expect(messages.scan("never reported back after it was forked").size).to eq(4) + expect(messages).to include("No further lost workers are replaced", "did not run because no worker remained to run them") + expect(process.signals.map(&:first)).to eq(%w[TERM TERM TERM TERM]) end end From f18b0f85d483135d54de730ef61918cdec2978bc Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:45:09 +0200 Subject: [PATCH 50/58] [rspec-core] Fix --parallel output and spawned workers on Windows Windows CI showed: `$stdout` replaced by a StringIO broke showing worker output (no `binmode?`); a spawned worker's start was reported only after its `--require`d files loaded, so a slow one was called "stuck in a fork handler"; and a failure while stopping the workers could leave the run's SIGTERM handler installed. Workers now size their output files by path and keep no handles to them, the coordinator restores its signal handlers whatever stopping the workers does, and a spawned worker that never reports back is described as not having started. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/coordinator.rb | 15 ++++++--------- .../lib/rspec/core/parallel/spawned_worker.rb | 1 + rspec-core/lib/rspec/core/parallel/worker.rb | 18 +++++++++++++++--- .../lib/rspec/core/parallel/worker_exits.rb | 8 ++++++-- .../rspec/core/parallel/worker_lifecycle.rb | 6 ++++++ .../lib/rspec/core/parallel/worker_output.rb | 13 +++++++------ rspec-core/spec/integration/parallel_spec.rb | 14 ++++++-------- .../rspec/core/parallel/coordinator_spec.rb | 9 +++++++++ .../spec/rspec/core/parallel/worker_spec.rb | 3 ++- 9 files changed, 58 insertions(+), 29 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 33cf3a4eb..2820a30c6 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -111,9 +111,12 @@ def run @runner.exit_code(passed) ensure - terminate_remaining_workers - Parallel.restore_signals(@signal_handlers) - WorkerOutput.remove_directory(@output_dir) + begin + terminate_remaining_workers + ensure + Parallel.restore_signals(@signal_handlers) + WorkerOutput.remove_directory(@output_dir) + end end private @@ -229,12 +232,6 @@ def expectations(count) count.times { reporter.increment_expectation_count } end - # Anything it sends shows it got past forking and runs RSpec's code. - def worker_started(worker) - worker.starting = false - worker.clear_start_deadline - end - def non_example_failure(payload, context_description) @configuration.reporter.notify_non_example_exception(RemoteException.rebuild(payload), context_description) end diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb index b6cd84451..2131dd5f8 100644 --- a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -63,6 +63,7 @@ def self.run(args) worker = Worker.new(settings[:number], nil, jobs, results, :output => settings[:output], :coordinator => settings[:coordinator], :loading => loading) worker.capture_output + worker.report_started RSpec.configuration.reporter = loading configure(args, settings) worker.run diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index 0b4779a5d..d3b9c24bf 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -58,8 +58,7 @@ def self.test_env_number(number) # Never returns: ends the process when the coordinator has no more work. def run capture_output - # First, so the coordinator knows it got past `fork`. - @results.send([:started]) + report_started watch_coordinator if @watch_coordinator prepare load_spec_files unless @example_groups @@ -71,6 +70,17 @@ def run @exit_now.call(1) end + # First, so the coordinator knows it got past `fork` -- or, for a + # spawned worker, that it got as far as RSpec before its `--require`d + # files load. + # @private + def report_started + return if @reported_started + + @reported_started = true + @results.send([:started]) + end + # @private def capture_output return if @captured || !@output @@ -80,7 +90,9 @@ def capture_output @direct_stderr = STDERR.dup # rubocop:disable Style/GlobalStdStream @results = WorkerOutput::Capture.new(@output, @results) rescue SystemCallError, IOError - nil # it writes where the run does + # It writes where the run does. + @direct_stderr&.close + @direct_stderr = nil end # A worker notices that the coordinator is gone when it next reports diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 5d27d0f42..8982bdda4 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -117,8 +117,12 @@ def never_started(worker) worker.lost = true index = worker.current_unit requeue = index && accepting_work? - report "#{worker.description} never reported back after it was forked (stuck before running any of " \ - "RSpec's code, as a deadlock in a fork handler would leave it), so it was sent SIGTERM. It ran " \ + stuck = if Parallel.spawning? + "started (Ruby did not get as far as loading RSpec)" + else + "forked (stuck before running any of RSpec's code, as a deadlock in a fork handler would leave it)" + end + report "#{worker.description} never reported back after it was #{stuck}, so it was sent SIGTERM. It ran " \ "nothing#{never_started_outcome(index, requeue)}." if requeue @unit_states[index] = :pending diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index 43569f928..dd648e4af 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -186,6 +186,12 @@ def show_output(worker, stream, bytes) @sequencer.output(worker.number, stream, bytes) end + # Anything it sends shows it got past forking and runs RSpec's code. + def worker_started(worker) + worker.starting = false + worker.clear_start_deadline + end + # Including those a failed `fork` left without a process. def live_workers @workers.each { |worker| worker.state = :exited if worker.pid.nil? } diff --git a/rspec-core/lib/rspec/core/parallel/worker_output.rb b/rspec-core/lib/rspec/core/parallel/worker_output.rb index 9fffd5d6b..052a2c6b7 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_output.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_output.rb @@ -29,7 +29,8 @@ def self.write(stream, bytes) io = stream == :err ? $stderr : $stdout # What formatters printed before it comes first. $stdout.flush if stream == :err - bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !io.binmode? + # Text-mode output on Windows adds a carriage return to every newline. + bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !(io.respond_to?(:binmode?) && io.binmode?) io.write(bytes) io.flush rescue IOError, SystemCallError @@ -94,12 +95,12 @@ def read_stream(stream, limit) class Capture def initialize(paths, results) @results = results - @files = paths.transform_values { |path| File.open(path, 'ab') } + @paths = paths # The process's own streams, which processes it starts inherit, # not whatever `$stdout` and `$stderr` point to. - STDOUT.reopen(@files[:out]) # rubocop:disable Style/GlobalStdStream - STDERR.reopen(@files[:err]) # rubocop:disable Style/GlobalStdStream - [STDOUT, STDERR].each do |io| # rubocop:disable Style/GlobalStdStream + # rubocop:disable-next Style/GlobalStdStream + { STDOUT => paths[:out], STDERR => paths[:err] }.each do |io, path| + File.open(path, 'ab') { |file| io.reopen(file) } io.binmode io.sync = true end @@ -107,7 +108,7 @@ def initialize(paths, results) end def send(message) - sizes = @files.transform_values(&:size) + sizes = @paths.transform_values { |path| File.size(path) } unless sizes == @announced @results.send([:output, sizes]) @announced = sizes diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index e657d3cb3..d3fecefc0 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -357,13 +357,10 @@ def example_passed(notification) options = %w[--require ./spec/support/at_exit --format documentation --order defined --parallel=2] status, stdout, stderr = rspec_process(*options) - # The coordinator says so when it replaces the killed worker. - blocks = stdout.gsub(/^ *Parallel worker \d \(pid \d+\) was started in place .*\n/, '').split(/^(?=P\d$)/) - blocks = blocks.select { |block| block =~ /\AP\d$/ } - expect(blocks.size).to eq(3) - blocks.each do |block| - name = block[/\AP\d/] - expect(block).to start_with("#{name}\nstdout of #{name}\nsubprocess of #{name}\n prints\n is quiet\n") + # Without what the coordinator says when it replaces the killed worker. + shown = stdout.gsub(/^ *Parallel worker \d \(pid \d+\) was started in place .*\n/, '') + %w[P0 P1 P2].each do |name| + expect(shown).to include("#{name}\nstdout of #{name}\nsubprocess of #{name}\n prints\n") end # The worker killed with SIGKILL runs no `at_exit` handler. expect(stdout.scan(/^at_exit of worker \d$/)).not_to be_empty @@ -1370,7 +1367,8 @@ def both_workers_started end it 'aborts spawned workers that do not exit within the shutdown timeout, and shows their thread dumps' do - write_suite "RSpec.configure { |c| c.parallel_shutdown_timeout = 1; c.after(:suite) { sleep } }" + # In the workers, whose watchdogs it sets: the coordinator keeps the default for their loading. + write_suite "RSpec.configure { |c| (c.parallel_shutdown_timeout = 1) if c.parallel_worker_number; c.after(:suite) { sleep } }" started = Process.clock_gettime(Process::CLOCK_MONOTONIC) status, stdout, stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index ff7b12499..36004bd66 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -1155,6 +1155,15 @@ def example_passed(_notification) expect(process.signals.map(&:first)).to eq(["TERM"]) end + it "says Ruby did not get as far as RSpec when it was started as a new process" do + allow(RSpec::Core::Parallel).to receive(:spawning?).and_return(true) + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :never_starts + + expect(coordinator([passing_group("A"), passing_group("B")]).run).to eq(0) + expect(messages).to include("never reported back after it was started (Ruby did not get as far as loading RSpec)") + end + it "says the group it was given did not run when the run stops early" do config.parallel_shutdown_timeout = 0.05 config.fail_fast = true diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 3ae3553f2..5522febaf 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -235,7 +235,8 @@ def frames after do [stdout_stand_in, stderr_stand_in].each(&:close) - FileUtils.remove_entry(dir) + # Windows does not remove a file the worker still has open (its copy of stderr). + FileUtils.remove_entry(dir, RSpec::Support::OS.windows?) end it "moves its standard output and error into them and says how far they reach before the next frame" do From b95258e6ad351a5ba78ed68fcb37b3975afb3981 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:46:56 +0200 Subject: [PATCH 51/58] [rspec-core] Start spawned workers in the run's directory on JRuby JRuby keeps `Dir.chdir` inside the JVM, so a process it spawns starts in the JVM's directory: spawned workers now get `:chdir => Dir.pwd`, as the spec helper that runs `rspec` in a process of its own does. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/lib/rspec/core/parallel/worker_handle.rb | 8 +++++--- rspec-core/spec/integration/parallel_spec.rb | 6 ++++-- rspec-core/spec/rspec/core/parallel/coordinator_spec.rb | 2 +- rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb | 2 +- 4 files changed, 11 insertions(+), 7 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/worker_handle.rb b/rspec-core/lib/rspec/core/parallel/worker_handle.rb index 6d1622146..59a7cf726 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_handle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_handle.rb @@ -112,10 +112,12 @@ def fork_worker(process, units, handles, worker_options={}) end end - # @return [Integer] the pid of a new process running `command`, its - # jobs on its standard input and its results on its standard output + # @return [Integer] the pid of a new process running `command` in + # this process's directory (which JRuby keeps apart from the + # process's own), its jobs on its standard input and its results on + # its standard output def spawn_worker(process, command, environment) - process.spawn(environment, *command, :in => @child_jobs.io, :out => @child_results.io) + process.spawn(environment, *command, :in => @child_jobs.io, :out => @child_results.io, :chdir => Dir.pwd) end def close_parent_ends diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index d3fecefc0..83d143d08 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -8,12 +8,14 @@ module ParallelRunProcess def rspec_process(*args) cd('.') do rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] - pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => 'stdout.txt', :err => 'stderr.txt') + # Absolute, and in this directory: JRuby keeps its own apart from the process's. + out, err = %w[stdout.txt stderr.txt].map { |name| File.expand_path(name) } + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil }, *rspec, *args, :out => out, :err => err, :chdir => Dir.pwd) deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 60 sleep 0.05 until (status = Process.waitpid2(pid, Process::WNOHANG)&.last) || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline Process.kill(:KILL, pid) && Process.wait(pid) unless status - [status, File.read('stdout.txt'), File.read('stderr.txt')] + [status, File.read(out), File.read(err)] end end end diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 36004bd66..0b509cb87 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -947,7 +947,7 @@ def standard_streams(stdout=StringIO.new) expect(env).to include("RSPEC_PARALLEL_WORKER" => "1", "RSPEC_PARALLEL_COORDINATOR" => Process.pid.to_s, "RSPEC_PARALLEL_SEED" => "99", "TEST_ENV_NUMBER" => "") expect(env["RSPEC_PARALLEL_STDOUT"]).to end_with("worker-1.out") - expect(redirects).to eq([:in, :out]) + expect(redirects).to eq([:in, :out, :chdir]) expect(killed).to eq([4243, 4244]) expect(instance.instance_variable_get(:@boot_allowance)).to be > 0 end diff --git a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb index c5a0eacee..2d74df336 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_handle_spec.rb @@ -36,7 +36,7 @@ module RSpec::Core::Parallel handle = WorkerHandle.new(2) process = double(:process) expect(process).to receive(:spawn).with({ "A" => "1" }, "ruby", "rspec", :in => handle.child_jobs.io, - :out => handle.child_results.io).and_return(4242) + :out => handle.child_results.io, :chdir => Dir.pwd).and_return(4242) expect(handle.spawn_worker(process, %w[ruby rspec], "A" => "1")).to eq(4242) handle.close_all From 792ab4f579f4cd52c87c3cb16bfd8926e2f02aea Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:54:47 +0200 Subject: [PATCH 52/58] [rspec-core] Stop stuck --parallel workers after the coordinator is killed A worker that cannot notice its coordinator is gone -- stopped, or stuck before running any of RSpec's code, as a deadlock in a fork handler leaves it -- kept CI steps open when the coordinator was killed with SIGKILL. A small process, spawned before the first worker is forked, watches the coordinator; when it goes it gives registered workers 1.5 s to leave, sends SIGTERM, then SIGKILL 5 s later, and removes their output files. It signals only registered pids still in the run's process group, ignores INT/TERM/HUP/QUIT and exits when told the run is done. Not covered: a worker forked moments before the coordinator was killed and not yet registered. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 3 +- rspec-core/README.md | 10 +- .../lib/rspec/core/parallel/coordinator.rb | 7 +- rspec-core/lib/rspec/core/parallel/reaper.rb | 122 ++++++++++++++++++ .../rspec/core/parallel/worker_lifecycle.rb | 11 +- rspec-core/spec/integration/parallel_spec.rb | 34 +++++ .../rspec/core/parallel/coordinator_spec.rb | 2 +- .../spec/rspec/core/parallel/reaper_spec.rb | 72 +++++++++++ 8 files changed, 255 insertions(+), 6 deletions(-) create mode 100644 rspec-core/lib/rspec/core/parallel/reaper.rb create mode 100644 rspec-core/spec/rspec/core/parallel/reaper_spec.rb diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index 533028cf8..b2d706857 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -19,7 +19,8 @@ Enhancements: parallel_tests does. A worker that finishes but does not exit is aborted after `RSpec::Core::Configuration#parallel_shutdown_timeout` seconds with a dump of its threads, so a stuck `at_exit` handler can no longer hang a run (nor can one that runs - after code under test called `exit`); neither can a worker stuck in `fork`. A worker + after code under test called `exit`); neither can a worker stuck in `fork`, even once the + coordinating process is killed with SIGKILL. A worker that crashes fails the example it was running instead of taking the run down, while the examples of its group it had not started run in another worker, and a new worker with its `TEST_ENV_NUMBER` takes its place. Code that calls diff --git a/rspec-core/README.md b/rspec-core/README.md index 06a8bcd20..e84858319 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -418,7 +418,15 @@ exiting in time. Which examples ran before that depends on timing. To make such an `exit` fail the example instead, rescue `SystemExit` in an `around` hook. Workers also exit on their own if the -coordinating process disappears. SIGTERM or SIGHUP to the coordinating +coordinating process disappears. One that cannot -- stopped, or stuck before +it runs any of RSpec's code, as a deadlock in a `fork` handler leaves it -- is +stopped by a small process the run starts before forking its first worker, +which outlives a coordinating process killed with SIGKILL: it gives the +workers it was told about a moment to leave, then sends them SIGTERM and 5 +seconds later SIGKILL, and removes their output files. It signals only those +workers, and only while they are in the run's process group; one forked +moments before the coordinating process was killed may not have been +registered yet. SIGTERM or SIGHUP to the coordinating process, as `timeout` or a cancelled CI job sends, stops every worker -- even one that cannot tear down -- before the process ends the way a serial run would, and the output says which example each worker was running and for how diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index 2820a30c6..a9a0cfc34 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -8,6 +8,7 @@ RSpec::Support.require_rspec_core "parallel/results" RSpec::Support.require_rspec_core "parallel/suite_index" RSpec::Support.require_rspec_core "parallel/worker_output" +RSpec::Support.require_rspec_core "parallel/reaper" RSpec::Support.require_rspec_core "parallel/worker_handle" RSpec::Support.require_rspec_core "parallel/worker_lifecycle" RSpec::Support.require_rspec_core "parallel/worker_exits" @@ -52,6 +53,8 @@ def initialize(runner, worker_count, options={}) @term_grace = options.fetch(:term_grace, TERM_GRACE) @kill_grace = options.fetch(:kill_grace, KILL_GRACE) @watchdog_grace = options.fetch(:watchdog_grace, WATCHDOG_GRACE) + @reaper_class = options.fetch(:reaper) { Reaper } + @reaper = nil @workers = [] @results = Results.new @quitting = false @@ -114,8 +117,7 @@ def run begin terminate_remaining_workers ensure - Parallel.restore_signals(@signal_handlers) - WorkerOutput.remove_directory(@output_dir) + clean_up end end @@ -398,6 +400,7 @@ def reap_workers worker.exit_status = status judge_exit(worker, status) worker.state = :exited + @reaper&.forget(worker.pid) replace(worker) end end diff --git a/rspec-core/lib/rspec/core/parallel/reaper.rb b/rspec-core/lib/rspec/core/parallel/reaper.rb new file mode 100644 index 000000000..4e63d716b --- /dev/null +++ b/rspec-core/lib/rspec/core/parallel/reaper.rb @@ -0,0 +1,122 @@ +module RSpec + module Core + module Parallel + # A small process that outlives the coordinator, which it watches: if + # the coordinator is killed (SIGKILL, the OOM killer) while a worker + # cannot notice by itself -- one stuck before running any of RSpec's + # code, as a deadlock in a `fork` handler leaves it, or stopped -- it + # sends the workers it was told about SIGTERM, then SIGKILL, and + # removes their output files. It is started before the first worker is + # forked, by `spawn` (no fork handlers run), and only where workers are + # forked: spawned workers watch the coordinator's pid themselves. A + # worker it was not told about yet -- the coordinator was killed + # between forking it and telling the reaper -- is not covered. + # @private + class Reaper + POLL = 0.25 + # Workers that can leave by themselves do, within their own poll. + OWN_EXIT_GRACE = 3 * Worker::COORDINATOR_POLL + TERM_GRACE = 5.0 + + # Ignores the signals that a terminal or a job's timeout sends to + # the whole process group, which would end it with the coordinator. + # Polls its parent; when that changes, gives the workers still + # registered whose process group is the run's a moment to leave by + # themselves, signals those that did not, then exits. + SCRIPT = <<-'RUBY'.freeze + %w[INT TERM HUP QUIT].each { |signal| trap(signal, 'IGNORE') if Signal.list.key?(signal) } + coordinator, group, output_dir = ARGV[0].to_i, ARGV[1].to_i, ARGV[2] + poll, own_exit, grace = ARGV[3].to_f, ARGV[4].to_f, ARGV[5].to_f + workers = {} + buffer = +'' + loop do + if IO.select([STDIN], nil, nil, poll) + chunk = (STDIN.read_nonblock(4096) rescue nil) + break if chunk.nil? + buffer << chunk + while (line = buffer.slice!(/\A[^\n]*\n/)) + exit!(0) if line == "done\n" + pid = line[1..-1].to_i + line.start_with?('+') ? workers[pid] = true : workers.delete(pid) + end + end + break if Process.ppid != coordinator + end + exit!(0) if Process.ppid == coordinator + ours = lambda do |pid| + Process.getpgid(pid) == group + rescue SystemCallError + false + end + waiting = workers.keys.select(&ours) + until_time = Process.clock_gettime(Process::CLOCK_MONOTONIC) + own_exit + sleep 0.05 while waiting.any?(&ours) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < until_time + left = waiting.select(&ours) + left.each { |pid| Process.kill('TERM', pid) rescue nil } + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + grace + sleep 0.05 while left.any?(&ours) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline + left.select(&ours).each { |pid| Process.kill('KILL', pid) rescue nil } + unless left.empty? + STDERR.puts "RSpec's parallel run (pid #{coordinator}) ended without stopping its workers; stopped " \ + "#{left.size} still running (pids #{left.join(', ')})." + end + if output_dir && !output_dir.empty? + require 'fileutils' + FileUtils.remove_entry(output_dir, true) + end + exit!(0) + RUBY + + # @return [Reaper, nil] one watching this process, or `nil` when it + # could not be started -- the run goes on without it + def self.start(output_dir, process=Process) + read, write = IO.pipe + pid = process.spawn(RbConfig.ruby, '--disable-gems', '-e', SCRIPT, Process.pid.to_s, Process.getpgrp.to_s, + output_dir.to_s, POLL.to_s, OWN_EXIT_GRACE.to_s, TERM_GRACE.to_s, + :in => read, :out => File::NULL, :close_others => true) + new(pid, write, process) + rescue SystemCallError, NotImplementedError + write&.close + nil + ensure + read&.close + end + + attr_reader :pid + + def initialize(pid, input, process=Process) + @pid = pid + @input = input + @input.sync = true + @process = process + end + + def watch(pid) + tell("+#{pid}") + end + + def forget(pid) + tell("-#{pid}") + end + + # Ends it, and reaps it if it has gone within a second. + def finish + tell('done') + @input.close + deadline = Parallel.monotonic_time + 1.0 + sleep 0.01 until @process.waitpid(@pid, Process::WNOHANG) || Parallel.monotonic_time > deadline + rescue SystemCallError + nil + end + + private + + def tell(line) + @input.write("#{line}\n") unless @input.closed? + rescue SystemCallError, IOError + nil # it is gone; the run goes on without it + end + end + end + end +end diff --git a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb index dd648e4af..8f4e646ea 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_lifecycle.rb @@ -41,10 +41,16 @@ def dismiss live = live_workers live.each { |worker| dismiss_worker(worker) } give_up_unless_gone(live, @kill_grace) - WorkerOutput.remove_directory(@output_dir) + clean_up end private + # Whatever stopping the workers did, also when it failed. + def clean_up + Parallel.restore_signals(@signal_handlers) + WorkerOutput.remove_directory(@output_dir) + @reaper&.finish + end # Rather than a `waitpid` for every worker on every wake-up of the # loop, which comes every few frames while results stream in. @@ -81,6 +87,8 @@ def start_workers(units) # Shares more memory with forked workers; spawned ones share none. @process.warmup if @process.respond_to?(:warmup) && !Parallel.spawning? @output_dir ||= WorkerOutput.make_directory + # Spawned workers watch the coordinator's pid themselves. + @reaper ||= @reaper_class&.start(@output_dir) unless Parallel.spawning? @worker_count.times do |index| break unless start_worker(index + 1, units) @@ -93,6 +101,7 @@ def start_worker(number, units, starter=@spawn, generation=1) worker.loading = units.nil? @workers << worker worker.pid = starter.call(worker, @workers) + @reaper&.watch(worker.pid) worker.close_child_ends worker.starting = true # A worker forked before loading gets the loading deadline instead. diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 83d143d08..aea08ab3d 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -931,6 +931,40 @@ def write_split_suite(extra="") end end + it 'stops workers that cannot notice the coordinating process is gone once it is killed, and removes their files' do + write_file_formatted 'spec/stopped_spec.rb', " + RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } + RSpec.describe('One') { it('stops itself') { Process.kill(:STOP, Process.pid) } } + RSpec.describe('Two') { it('stops itself') { Process.kill(:STOP, Process.pid) } } + " + wait_until = lambda do |seconds, &condition| + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until condition.call || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + condition.call + end + state = lambda { |pid| `ps -o stat= -p #{pid}`.strip } + # A stopped worker runs none of its code -- like one deadlocked in a fork + # handler -- and keeps every descriptor it inherited. + stopped = lambda { |pid| state.call(pid).start_with?('T') } + running = lambda { |pid| !state.call(pid).empty? && !state.call(pid).start_with?('Z') } + + cd('.') do + Dir.mkdir('tmp') + rspec = [RbConfig.ruby, "-I#{$LOAD_PATH.join(File::PATH_SEPARATOR)}", File.expand_path('../../exe/rspec', __dir__)] + pid = Process.spawn({ 'TEST_ENV_NUMBER' => nil, 'TMPDIR' => File.expand_path('tmp') }, *rspec, '--parallel=2', + 'spec/stopped_spec.rb', :out => File::NULL, :err => 'stderr.txt') + expect(wait_until.call(20) { worker_pids.size == 2 && worker_pids.all?(&stopped) }).to be true + workers = worker_pids + + Process.kill(:KILL, pid) + Process.wait(pid) + + expect(wait_until.call(15) { workers.none?(&running) }).to be true + expect(File.read('stderr.txt')).to include("ended without stopping its workers; stopped 2 still running") + expect(wait_until.call(5) { Dir['tmp/*'].empty? }).to be true + end + end + it 'stops workers stuck in an example once the coordinating process is gone' do write_file_formatted 'spec/stuck_spec.rb', " RSpec.configure { |c| c.before(:suite) { File.write(\"worker_\#{RSpec.configuration.parallel_worker_number}.pid\", Process.pid.to_s) } } diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 0b509cb87..17060721a 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -267,7 +267,7 @@ def new_coordinator(units, worker_count, options) units.each { |group| group.include(WithoutMocksLifecycle) } @units = units @process = options.fetch(:process, process) - Coordinator.new(runner, worker_count, { :process => @process, :spawn => method(:spawn), :tick => 0.01, + Coordinator.new(runner, worker_count, { :process => @process, :spawn => method(:spawn), :tick => 0.01, :reaper => nil, :term_grace => 5, :kill_grace => 5, :watchdog_grace => 0.05 }.merge(options)) end diff --git a/rspec-core/spec/rspec/core/parallel/reaper_spec.rb b/rspec-core/spec/rspec/core/parallel/reaper_spec.rb new file mode 100644 index 000000000..ee6a7b94d --- /dev/null +++ b/rspec-core/spec/rspec/core/parallel/reaper_spec.rb @@ -0,0 +1,72 @@ +require 'rspec/core/parallel/coordinator' + +module RSpec::Core::Parallel + RSpec.describe Reaper, :skip => !RSpec::Support::RubyFeatures.fork_supported? do + def alive?(pid) + state = `ps -o stat= -p #{pid}`.strip + !state.empty? && !state.start_with?('Z') + end + + def wait_until(seconds) + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + seconds + sleep 0.05 until yield || Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + yield + end + + it "ends when told the run is done, without signalling anything" do + reaper = Reaper.start(nil) + sleeper = Process.spawn('sleep', '30') + reaper.watch(sleeper) + + reaper.finish + + expect { Process.kill(0, reaper.pid) }.to raise_error(Errno::ESRCH) + expect(alive?(sleeper)).to be true + ensure + Process.kill('KILL', sleeper) && Process.wait(sleeper) if sleeper + end + + it "stops only the registered processes of the run's process group once its parent is gone without saying so" do + script = <<-RUBY + #{$LOAD_PATH.map { |path| "$LOAD_PATH.unshift(#{path.to_s.inspect})" }.join("\n")} + require 'rspec/core' + require 'rspec/core/parallel/coordinator' + registered = Process.spawn('sleep', '30', :out => File::NULL) + elsewhere = Process.spawn('sleep', '30', :out => File::NULL, :pgroup => true) + unregistered = Process.spawn('sleep', '30', :out => File::NULL) + reaper = RSpec::Core::Parallel::Reaper.start(nil) + reaper.watch(registered) + reaper.watch(elsewhere) + puts [registered, elsewhere, unregistered].join(' ') + STDOUT.flush + exit!(0) + RUBY + registered, elsewhere, unregistered = IO.popen([RbConfig.ruby, '-e', script], :err => File::NULL, &:read).split.map(&:to_i) + + expect(wait_until(15) { !alive?(registered) }).to be true + expect([alive?(elsewhere), alive?(unregistered)]).to eq([true, true]) + ensure + [registered, elsewhere, unregistered].compact.each do |pid| + Process.kill('KILL', pid) if alive?(pid) + rescue Errno::ESRCH + nil + end + end + + it "is not started when the system refuses it a process, and the run goes on" do + process = double(:process) + allow(process).to receive(:spawn).and_raise(Errno::EAGAIN) + + expect(Reaper.start(nil, process)).to be_nil + end + + it "is told nothing more once it is gone" do + reaper = Reaper.start(nil) + Process.kill('KILL', reaper.pid) + Process.wait(reaper.pid) + + expect { 100.times { reaper.watch(1) } }.not_to raise_error + expect { reaper.finish }.not_to raise_error + end + end +end From 54f39647ec9c6193b7090be304705ec68b60998f Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 02:55:50 +0200 Subject: [PATCH 53/58] [rspec-core] Keep the worker replacement spec from depending on speed On Windows the surviving fake worker finished every queued group before the crash was noticed, so nothing waited to run and no replacement was due. The remaining groups now take 0.5 s each. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/rspec/core/parallel/coordinator_spec.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 17060721a..52e11caf9 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -966,9 +966,11 @@ def standard_streams(stdout=StringIO.new) it "starts a worker with the lost one's number in its place, which runs the remaining groups" do crashing = RSpec.describe("Crashing") { it("dies") { Thread.current.kill } } + # Still running when the crash is noticed, so that groups wait to run. + slow = %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.5 } } } config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) - exit_code = coordinator([crashing, passing_group("A"), passing_group("B"), passing_group("C")]).run + exit_code = coordinator([crashing] + slow).run expect(exit_code).to eq(config.failure_exit_code) expect(@started_numbers.sort).to eq([1, 1, 2]) From f92bc3e9f9049c568bf547c0342a2ad65c4c78f2 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 03:27:54 +0200 Subject: [PATCH 54/58] [rspec-core] Keep two spawned-worker specs from depending on JRuby's speed Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/integration/parallel_spec.rb | 3 ++- rspec-core/spec/rspec/core/parallel/coordinator_spec.rb | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index aea08ab3d..f9b8640b7 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -1409,7 +1409,8 @@ def both_workers_started status, stdout, stderr = rspec_process(*without_fork, '--require', './spec/spec_helper', '--parallel=2') - expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 30 + # Its `after(:suite)` hooks never return: that it ends at all is the bound (JRuby's startup is most of the time). + expect(Process.clock_gettime(Process::CLOCK_MONOTONIC) - started).to be < 55 expect(stdout).to include("6 examples, 1 failure") expect(stderr.scan("did not exit within 1 second of finishing its last example group").size).to eq(2) expect(status.exitstatus).to eq(1) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 52e11caf9..e0a68c5a0 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -1162,7 +1162,8 @@ def example_passed(_notification) config.parallel_shutdown_timeout = 0.05 worker_behaviours[2] = :never_starts - expect(coordinator([passing_group("A"), passing_group("B")]).run).to eq(0) + coordinator([passing_group("A"), passing_group("B")]).run + expect(messages).to include("never reported back after it was started (Ruby did not get as far as loading RSpec)") end From a6e1354aa3ebef645d1581e8660b18e2ee1ac4f3 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 03:55:44 +0200 Subject: [PATCH 55/58] [rspec-core] Fix five --parallel issues found in review - Metadata a worker could marshal but the coordinator could not load (a class defined only in the worker) made the frame undecodable, so the worker was terminated and a passing example failed. Each changed key now travels as its own dump; one that fails to load is warned about and left out. - A spawned worker's `--require`d files ran `add_formatter` with a path, truncating the file the coordinator writes. Its formatter loader now opens no files. - Output read between two frames could pile up past the sequencer's 8 MB bound until the next loop turn; queued events are now flushed as soon as a bound is crossed. - Worker output with non-ASCII bytes raised Encoding errors when `$stdout` had an external encoding; it is written untranscoded. - A spawned worker that never reported back while loading was replaced; workers lost while loading are not. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/lib/rspec/core/configuration.rb | 5 +++ .../lib/rspec/core/parallel/coordinator.rb | 18 +--------- rspec-core/lib/rspec/core/parallel/results.rb | 35 +++++++++++++++++++ .../lib/rspec/core/parallel/sequencer.rb | 9 +++-- .../lib/rspec/core/parallel/spawned_worker.rb | 14 ++++++++ .../lib/rspec/core/parallel/worker_exits.rb | 2 +- .../lib/rspec/core/parallel/worker_output.rb | 5 ++- .../rspec/core/parallel/worker_reporter.rb | 24 ++++++++----- rspec-core/spec/integration/parallel_spec.rb | 28 ++++++++++----- .../rspec/core/parallel/coordinator_spec.rb | 16 +++++++++ .../rspec/core/parallel/sequencer_spec.rb | 8 ++--- .../core/parallel/spawned_worker_spec.rb | 16 +++++++++ .../rspec/core/parallel/worker_output_spec.rb | 15 ++++++++ .../core/parallel/worker_reporter_spec.rb | 14 ++++++-- 14 files changed, 160 insertions(+), 49 deletions(-) diff --git a/rspec-core/lib/rspec/core/configuration.rb b/rspec-core/lib/rspec/core/configuration.rb index 47453b6ee..3057c8a2c 100644 --- a/rspec-core/lib/rspec/core/configuration.rb +++ b/rspec-core/lib/rspec/core/configuration.rb @@ -605,6 +605,11 @@ def reset_reporter # worker's notifications to the coordinating process. attr_writer :reporter + # @private + # Used by the parallel runner so that a spawned worker's formatters, + # which never run, do not open the files the coordinating process's do. + attr_writer :formatter_loader + # @private def reset_filters self.filter_manager = FilterManager.new diff --git a/rspec-core/lib/rspec/core/parallel/coordinator.rb b/rspec-core/lib/rspec/core/parallel/coordinator.rb index a9a0cfc34..f0795f884 100644 --- a/rspec-core/lib/rspec/core/parallel/coordinator.rb +++ b/rspec-core/lib/rspec/core/parallel/coordinator.rb @@ -254,28 +254,12 @@ def example_started(worker, example, bytes) def example_finished(worker, example_id, payload, bytes) index.with_example(example_id) do |example| worker.current_example = nil + payload[:metadata] = @results.metadata_changes(worker.description, payload[:metadata]) record_result(example, payload) - warn_unsent_metadata(worker, payload[:metadata]) @sequencer.example_finished(worker.number, example, bytes) end end - # Once per metadata key. - def warn_unsent_metadata(worker, changes) - unsent = changes && changes[:unsent] - return unless unsent - - @unsent_metadata ||= {} - unsent.each do |key, class_name| - next if @unsent_metadata.key?(key) - - @unsent_metadata[key] = true - RSpec.warning "#{worker.description} set example metadata #{key} to a #{class_name}, which cannot be " \ - "marshaled to reach the formatters, so they see the value it had before the example ran. " \ - "Later values of #{key} that cannot be marshaled are not reported", :call_site => nil - end - end - def index @index ||= SuiteIndex.new(@world) { |warning| report warning } end diff --git a/rspec-core/lib/rspec/core/parallel/results.rb b/rspec-core/lib/rspec/core/parallel/results.rb index b0dae7af6..9c1c776d6 100644 --- a/rspec-core/lib/rspec/core/parallel/results.rb +++ b/rspec-core/lib/rspec/core/parallel/results.rb @@ -12,6 +12,7 @@ class Results def initialize @payloads = {} @failed_count = 0 + @unsent_metadata = {} end def record(example, payload) @@ -34,6 +35,27 @@ def apply(example) ) end + # What a worker sent of an example's metadata changes, loaded one by + # one: a value whose class is not loaded here is left out -- with a + # warning, once per key -- like one the worker could not marshal. + # @return [Hash, nil] the keys set (`:set`) and deleted (`:deleted`) + def metadata_changes(worker_description, changes) + return nil unless changes + + unsent = changes.fetch(:unsent, {}).dup + load = lambda do |(key, dump)| + # rubocop:disable-next Security/MarshalLoad + Marshal.load(dump) # dumped by the worker's own reporter + rescue StandardError => e + unsent[key] = "it cannot be loaded here (#{e.class}: #{e.message[0, 200]})" + nil + end + set = changes.fetch(:set, []).map(&load).compact.to_h + deleted = changes.fetch(:deleted, []).map(&load).compact + warn_unsent_metadata(worker_description, unsent) + { :set => set, :deleted => deleted } + end + # What the example's run in a worker did to its metadata, done to the # coordinator's copy before formatters are told about it. def apply_metadata(metadata, changes) @@ -46,6 +68,19 @@ def apply_metadata(metadata, changes) def apply_all @payloads.keys.each { |example| apply(example) } end + + private + + def warn_unsent_metadata(worker_description, unsent) + unsent.each do |key, reason| + next if @unsent_metadata.key?(key) + + @unsent_metadata[key] = true + RSpec.warning "#{worker_description} set example metadata #{key} to a value that cannot reach the " \ + "formatters (#{reason}), so they see the value it had before the example ran. Later such " \ + "values of #{key} are not reported", :call_site => nil + end + end end end end diff --git a/rspec-core/lib/rspec/core/parallel/sequencer.rb b/rspec-core/lib/rspec/core/parallel/sequencer.rb index 022554055..f252b5a24 100644 --- a/rspec-core/lib/rspec/core/parallel/sequencer.rb +++ b/rspec-core/lib/rspec/core/parallel/sequencer.rb @@ -140,12 +140,9 @@ def abandon(worker) # Called periodically by the coordinator. def tick(now=@clock.call) return unless @sequenced + return unless @completed.any? - if @stats[:queued] >= QUEUE_BOUND || @stats[:queued_bytes] >= QUEUE_BYTES_BOUND - flush_everything - elsif @completed.any? && (now - @last_delivery_at >= SILENCE_BOUND || now - @completed.first.completed_at >= WAIT_BOUND) - flush_completed - end + flush_completed if now - @last_delivery_at >= SILENCE_BOUND || now - @completed.first.completed_at >= WAIT_BOUND end # Delivers everything still queued and closes any open group. @@ -202,6 +199,8 @@ def enqueue(stream, event) @stats[:queued_bytes] += event.bytes @stats[:peak_queued] = [@stats[:peak_queued], @stats[:queued]].max @stats[:peak_queued_bytes] = [@stats[:peak_queued_bytes], @stats[:queued_bytes]].max + # At once: output read in chunks between two frames must not pile up. + flush_everything if @stats[:queued] >= QUEUE_BOUND || @stats[:queued_bytes] >= QUEUE_BYTES_BOUND end def deliver(stream, event) diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb index 2131dd5f8..7fd306518 100644 --- a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -41,11 +41,25 @@ def self.settings(env=ENV) def self.configure(args, settings, configuration=RSpec.configuration) configuration.parallel_worker_number = settings[:number] configuration.ordering_manager.adopt_seed(settings[:seed]) if settings[:seed] + # Also for formatters its `--require`d files add. + configuration.formatter_loader = FormatterLoader.new(Reporter.new(configuration)) options = ConfigurationOptions.new(args) NOT_FOR_WORKERS.each { |key| options.options.delete(key) } options.configure(configuration) end + # Opens no files: the formatters of a worker never run, and the + # coordinating process writes to the files they would. + # @private + class FormatterLoader < Formatters::Loader + private + + # @private + def open_stream(_path_or_wrapper) + File.open(File::NULL, "w") + end + end + # Never returns. # :nocov: - runs in the spawned worker, covered by integration specs def self.run(args) diff --git a/rspec-core/lib/rspec/core/parallel/worker_exits.rb b/rspec-core/lib/rspec/core/parallel/worker_exits.rb index 8982bdda4..7d861179d 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_exits.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_exits.rb @@ -113,8 +113,8 @@ def replace_exit_status(worker, exit_status, cause) # Ruby, or a library's fork handler) leaves a process: it has run # nothing, so the group it was given goes to another worker as is. def never_started(worker) + worker.lost = !worker.loading worker.loading = false - worker.lost = true index = worker.current_unit requeue = index && accepting_work? stuck = if Parallel.spawning? diff --git a/rspec-core/lib/rspec/core/parallel/worker_output.rb b/rspec-core/lib/rspec/core/parallel/worker_output.rb index 052a2c6b7..bcacfaefc 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_output.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_output.rb @@ -31,9 +31,12 @@ def self.write(stream, bytes) $stdout.flush if stream == :err # Text-mode output on Windows adds a carriage return to every newline. bytes = bytes.gsub("\r\n".b, "\n".b) if RSpec::Support::OS.windows? && !(io.respond_to?(:binmode?) && io.binmode?) + # As written, not transcoded: they are in whatever encoding the worker wrote. + encoding = io.respond_to?(:external_encoding) && io.external_encoding + bytes = bytes.dup.force_encoding(encoding) if encoding io.write(bytes) io.flush - rescue IOError, SystemCallError + rescue IOError, SystemCallError, EncodingError nil end diff --git a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb index 448a1fe46..1f594e8bc 100644 --- a/rspec-core/lib/rspec/core/parallel/worker_reporter.rb +++ b/rspec-core/lib/rspec/core/parallel/worker_reporter.rb @@ -167,8 +167,11 @@ def metadata_snapshot(metadata) # @return [Hash, nil] what the example, its hooks or a worker listener # did to its metadata: keys set (`:set`) or deleted (`:deleted`), - # and those set to values that cannot be marshaled (`:unsent`, the - # values' class names), which the coordinator leaves as they were. + # each as the key's `inspect` and its own `Marshal` dump -- so that + # one the coordinator cannot load (its class is not loaded there) + # spoils only itself -- and keys set to values that cannot be + # marshaled (`:unsent`, with the reason), which the coordinator + # leaves as they were. def metadata_changes(metadata) before = @metadata_before || {} @metadata_before = nil @@ -176,26 +179,29 @@ def metadata_changes(metadata) metadata.each do |key, value| next if NOT_COMPARED.key?(key) || unchanged?(before[key], value) - if marshalable?([key, value]) - (changes[:set] ||= {})[key] = value + if (dump = marshaled([key, value])) + (changes[:set] ||= []) << [key.inspect, dump] else - (changes[:unsent] ||= {})[key.inspect] = value.class.name + (changes[:unsent] ||= {})[key.inspect] = "#{value.class.name} cannot be marshaled" end end - deleted = before.keys.reject { |key| metadata.key?(key) || !marshalable?(key) } + deleted = deleted_keys(before, metadata) changes[:deleted] = deleted unless deleted.empty? changes.empty? ? nil : changes end + def deleted_keys(before, metadata) + before.keys.reject { |key| metadata.key?(key) }.filter_map { |key| (dump = marshaled(key)) && [key.inspect, dump] } + end + def unchanged?(snapshot, value) snapshot && snapshot[0].equal?(value) && snapshot[1] == hash_of(value) end - def marshalable?(value) + def marshaled(value) Marshal.dump(value) - true rescue StandardError - false + nil end def hash_of(value) diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index f9b8640b7..f6dd328e6 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -303,7 +303,7 @@ def initialize(output) def example_passed(notification) metadata = notification.example.metadata @output.puts \"META \#{notification.example.description}: \#{metadata[:screenshot].inspect} \" \\ - \"\#{metadata[:tags].inspect} \#{metadata.key?(:gone)} \#{metadata[:callback].class}\" + \"\#{metadata[:tags].inspect} \#{metadata.key?(:gone)} \#{metadata[:callback].class} \#{metadata[:lazy].class}\" end end " @@ -316,22 +316,32 @@ def example_passed(notification) ex.metadata.delete(:gone) end it('callback#{i}') { |ex| ex.metadata[:callback] = lambda {} } + it('lazy#{i}') { |ex| Object.const_set(:ParallelSpecLazyThing, Class.new) unless defined?(ParallelSpecLazyThing); ex.metadata[:lazy] = ParallelSpecLazyThing.new } end " end options = '--require ./spec/support/metadata_formatter --format progress --format MetadataFormatter' - run_rspec options - serial = last_cmd_stdout.scan(/META .*$/).sort - expect(serial.size).to eq(8) - expect(serial.first).to eq("META callback0: nil nil true Proc") - expect(serial).to include("META e3: \"shot-3.png\" [\"a\", \"b\"] false NilClass") - + # In parallel first: a serial run in this process would define the class here too. run_rspec "#{options} --parallel=2" parallel = last_cmd_stdout.scan(/META .*$/).sort - expect(parallel).to eq(serial.map { |line| line.sub(/ Proc$/, ' NilClass') }) - expect(last_cmd_stderr.scan("set example metadata :callback to a Proc").size).to eq(1) + expect(last_cmd_stdout).to include("12 examples, 0 failures") expect(last_cmd_exit_status).to eq(0) + stderr = last_cmd_stderr + + run_rspec options + serial = last_cmd_stdout.scan(/META .*$/).sort + expect(serial.size).to eq(12) + expect(serial.first).to eq("META callback0: nil nil true Proc NilClass") + expect(serial).to include("META e3: \"shot-3.png\" [\"a\", \"b\"] false NilClass NilClass", + "META lazy0: nil nil true NilClass ParallelSpecLazyThing") + + # A Proc cannot be marshaled; the class is defined only in the worker that ran the example. + expect(parallel).to eq(serial.map { |line| line.sub(' Proc ', ' NilClass ').sub(/ ParallelSpecLazyThing$/, ' NilClass') }) + expect(stderr.scan('set example metadata :callback to a value that cannot reach the formatters (Proc cannot be marshaled)').size).to eq(1) + expect(stderr.scan('set example metadata :lazy to a value that cannot reach the formatters (it cannot be loaded here (ArgumentError: undefined class/module ParallelSpecLazyThing').size).to eq(1) + ensure + Object.send(:remove_const, :ParallelSpecLazyThing) if defined?(ParallelSpecLazyThing) end it 'shows what workers print with the example group that printed it, as a serial run does' do diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index e0a68c5a0..0c22610b9 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -997,6 +997,22 @@ def standard_streams(stdout=StringIO.new) expect(messages).to include("4 examples, 1 failure") end + it "starts none in place of one that never reported back while loading, which would likely not either" do + allow(RSpec::Core::Runner).to receive(:command_line).and_return(["/app/bin/rspec", []]) + config.parallel_shutdown_timeout = 0.05 + worker_behaviours[2] = :never_starts + # Still running when the other misses its deadline, so that groups wait to run. + slow = %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.3 } } } + instance = started_coordinator(slow) + allow(instance).to receive(:spawn_worker) { |handle, handles| spawn(handle, handles) } + instance.assign(@units, units_of(@units)) + + instance.run + + expect(instance).not_to have_received(:spawn_worker) + expect(messages).to include("never reported back") + end + it "starts none in place of a lost one when the run was not started by the `rspec` command" do allow(RSpec::Core::Runner).to receive(:command_line).and_return(nil) instance = started_coordinator(groups) diff --git a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb index 97b2cb8e5..243330dfe 100644 --- a/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/sequencer_spec.rb @@ -243,10 +243,11 @@ def events seq = sequencer(true) seq.group_started(1, g1) seq.group_started(2, g2) - run_example(seq, 2, ex(g2, "c")) + ex(g2, "c").execution_result.status = :passed + seq.example_started(2, ex(g2, "c")) expect(events.map(&:last)).to eq(%w[G1]) - seq.tick + seq.example_finished(2, ex(g2, "c")) expect(events.map(&:last)).to eq(%w[G1 G1 G2 c c c]) expect(seq.stats[:queued]).to eq(0) @@ -260,12 +261,11 @@ def events seq.group_started(1, g1) seq.group_started(2, g2) seq.message(2, "big", 600) - seq.tick expect(events.map(&:last)).to eq(%w[G1]) expect(seq.stats[:queued_bytes]).to eq(600) + # As it is queued, not at the next tick: output read in chunks would pile up until then. seq.message(2, "bigger", 600) - seq.tick expect(events.map(&:last)).to eq(%w[G1 G1 G2 big bigger]) expect(seq.stats).to include(:queued => 0, :queued_bytes => 0, :peak_queued_bytes => 1_200) end diff --git a/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb index 567e2eee3..16d60d247 100644 --- a/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/spawned_worker_spec.rb @@ -1,3 +1,5 @@ +require 'fileutils' +require 'tmpdir' require 'tempfile' require 'rspec/core/parallel/coordinator' @@ -50,6 +52,20 @@ def configure(args, settings) expect(File.exist?("results.json")).to be false end + it "opens no file for a formatter its `--require`d files add, which the coordinator writes to" do + dir = Dir.mktmpdir + helper = File.join(dir, 'helper.rb') + File.write(helper, "RSpec.configure { |c| c.add_formatter 'documentation', #{File.join(dir, 'out.txt').inspect} }\n") + File.write(File.join(dir, 'out.txt'), "the coordinator's report") + + # The run's configuration, which `RSpec.configure` in the helper configures. + SpawnedWorker.configure(['-O', options_file, '--require', helper], { :number => 2, :seed => 1 }, RSpec.configuration) + + expect(File.read(File.join(dir, 'out.txt'))).to eq("the coordinator's report") + ensure + FileUtils.remove_entry(dir) if dir + end + it "takes the run's seed without choosing an ordering, which the run's configuration does" do configure(%w[--order defined], :number => 2, :seed => 1234) diff --git a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb index 30c0d74b0..4a9e095f6 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb @@ -54,6 +54,21 @@ def read(sizes=nil) expect(read).to eq([]) end + it "writes what a worker printed as it was written, also to a stream with an encoding of its own" do + # A file, which (unlike a StringIO) transcodes what is written to it. + path = File.join(dir, 'stdout') + original = $stdout + $stdout = File.open(path, 'w:UTF-8') + begin + WorkerOutput.write(:out, "caf\xC3\xA9 \xFF\n".b) + ensure + $stdout.close + $stdout = original + end + + expect(File.binread(path)).to eq("caf\xC3\xA9 \xFF\n".b) + end + it "goes without a directory when none can be made" do allow(Dir).to receive(:mktmpdir).and_raise(Errno::EACCES) expect(WorkerOutput.make_directory).to be_nil diff --git a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb index 3711900e3..8ad9b5091 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_reporter_spec.rb @@ -10,6 +10,11 @@ def frames results.first.drain.map(&:first) end + # What the reporter dumped itself. + def load_dump(dump) + Marshal.load(dump) # rubocop:disable Security/MarshalLoad + end + after do results.each(&:close) jobs.each(&:close) @@ -205,7 +210,10 @@ def listener.example_failed(notification) payloads = frames.select { |f| f.first == :example_finished }.to_h { |f| [f.last[:description], f.last[:metadata]] } changed, unchanged = payloads.values_at("changes", "changes nothing") - expect(changed).to eq(:set => { :added => { :path => "shot.png" }, :tags => %w[a b] }, :deleted => [:gone]) + expect(changed[:set].map { |key, dump| [key, load_dump(dump)] }).to contain_exactly( + [":added", [:added, { :path => "shot.png" }]], [":tags", [:tags, %w[a b]]] + ) + expect(changed[:deleted].map { |key, dump| [key, load_dump(dump)] }).to eq([[":gone", :gone]]) expect(unchanged).to be_nil end @@ -231,7 +239,7 @@ def listener.example_failed(notification) group.run(reporter) payload = frames.find { |f| f.first == :example_finished }.last - expect(payload[:metadata]).to eq(:unsent => { ":callback" => "Proc" }) + expect(payload[:metadata]).to eq(:unsent => { ":callback" => "Proc cannot be marshaled" }) end it "copes with values whose `hash` raises and with deleted keys that cannot be marshaled" do @@ -259,7 +267,7 @@ def listener.example_passed(notification) RSpec.describe("G") { it("passes") {} }.run(reporter) payload = frames.find { |f| f.first == :example_finished }.last - expect(payload[:metadata]).to eq(:set => { :screenshot => "after.png" }) + expect(payload[:metadata][:set].map { |key, dump| [key, load_dump(dump)] }).to eq([[":screenshot", [:screenshot, "after.png"]]]) end it "sets the world's `wants_to_quit` when the coordinator asks it to quit between examples" do From 42a596ef251347fd70cd9c7861ba1c69b55e2d89 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 04:00:53 +0200 Subject: [PATCH 56/58] [rspec-core] Keep loading workers from truncating formatter files Workers that load the spec files themselves reset their reporter before loading, which put back a formatter loader that opens files: a helper the spec files require that adds a formatter writing to a file truncated the coordinator's report (NUL bytes where its first lines were, deterministically for a replacement worker). They now keep a loader that opens nothing. Found in the re-review of the five fixes. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../lib/rspec/core/parallel/spawned_worker.rb | 14 +------------- rspec-core/lib/rspec/core/parallel/worker.rb | 14 ++++++++++++++ rspec-core/spec/integration/parallel_spec.rb | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+), 13 deletions(-) diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb index 7fd306518..696a2833d 100644 --- a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -42,24 +42,12 @@ def self.configure(args, settings, configuration=RSpec.configuration) configuration.parallel_worker_number = settings[:number] configuration.ordering_manager.adopt_seed(settings[:seed]) if settings[:seed] # Also for formatters its `--require`d files add. - configuration.formatter_loader = FormatterLoader.new(Reporter.new(configuration)) + configuration.formatter_loader = Worker::FormatterLoader.new(Reporter.new(configuration)) options = ConfigurationOptions.new(args) NOT_FOR_WORKERS.each { |key| options.options.delete(key) } options.configure(configuration) end - # Opens no files: the formatters of a worker never run, and the - # coordinating process writes to the files they would. - # @private - class FormatterLoader < Formatters::Loader - private - - # @private - def open_stream(_path_or_wrapper) - File.open(File::NULL, "w") - end - end - # Never returns. # :nocov: - runs in the spawned worker, covered by integration specs def self.run(args) diff --git a/rspec-core/lib/rspec/core/parallel/worker.rb b/rspec-core/lib/rspec/core/parallel/worker.rb index d3b9c24bf..7f0b3db13 100644 --- a/rspec-core/lib/rspec/core/parallel/worker.rb +++ b/rspec-core/lib/rspec/core/parallel/worker.rb @@ -138,6 +138,8 @@ def prepare ENV['TEST_ENV_NUMBER'] = test_env_number @configuration.parallel_worker_number = @number @configuration.reset_reporter + # For formatters that helpers the spec files require add. + @configuration.formatter_loader = FormatterLoader.new(Reporter.new(@configuration)) unless @example_groups @reporter = WorkerReporter.new(@configuration, @results, @jobs, @world) @configuration.reporter = @reporter end @@ -320,6 +322,18 @@ def dump_threads(timeout) stderr.flush end + # Opens no files: a worker's formatters never run, and the + # coordinating process writes to the files they would. + # @private + class FormatterLoader < Formatters::Loader + private + + # @private + def open_stream(_path_or_wrapper) + File.open(File::NULL, "w") + end + end + # Reports nothing: while a worker loads the spec files, what they # report is for the coordinating process, which loads them too, to # report. Remembers the first error. diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index f6dd328e6..19441e1de 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -410,6 +410,24 @@ def example_passed(notification) expect(replacement).not_to be_nil end + it 'leaves the file of a formatter that a helper the spec files require adds to the coordinating process' do + write_file_formatted 'spec/helper.rb', "RSpec.configure { |c| c.add_formatter 'documentation', 'out.txt' }" + 4.times do |i| + write_file_formatted "spec/f#{i}_spec.rb", " + require_relative 'helper' + RSpec.describe('F#{i}') { it('runs') { #{i == 1 ? 'Process.kill(:KILL, Process.pid)' : 'sleep 0.5'} } } + " + end + + # Workers that load the spec files require the helper too; the one started in place of the killed one, mid-run. + status, _stdout, _stderr = rspec_process('--parallel=2', '--no-parallel-preload', '--order', 'defined') + + expect(status.exitstatus).to eq(1) + report = cd('.') { File.binread('out.txt') } + expect(report).not_to include("\0") + expect(report).to include("F0", "F2", "F3", "was started in place of the lost worker", "4 examples, 1 failure") + end + it 'lets a custom formatter use metadata, exception classes, custom notifications and group boundaries' do write_file_formatted 'spec/support/my_formatter.rb', " class MyAppError < StandardError; end From d3eb83ad19ca7522c82c050cc652bd7f0383d2e9 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 04:18:17 +0200 Subject: [PATCH 57/58] [rspec-core] Fix two parallel specs on Windows; drop an unverified claim Windows writes CRLF to a text-mode file and can still find a pid for a moment after it was reaped, which two new specs assumed it would not. The docs no longer name TruffleRuby among the platforms that spawn workers: only the process mechanics were probed there, not RSpec. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/Changelog.md | 2 +- rspec-core/README.md | 2 +- rspec-core/lib/rspec/core/parallel/spawned_worker.rb | 2 +- rspec-core/spec/integration/parallel_spec.rb | 2 +- rspec-core/spec/rspec/core/parallel/worker_output_spec.rb | 4 ++-- rspec-core/spec/rspec/core/parallel/worker_spec.rb | 3 +++ 6 files changed, 9 insertions(+), 6 deletions(-) diff --git a/rspec-core/Changelog.md b/rspec-core/Changelog.md index b2d706857..672c8b2ba 100644 --- a/rspec-core/Changelog.md +++ b/rspec-core/Changelog.md @@ -31,7 +31,7 @@ Enhancements: (`RSpec::Core::Configuration#parallel_preload`) the workers are forked before the spec files are loaded and each loads them itself, so nothing the spec files do while loading is shared with the workers; a worker that does not load the examples the coordinating - process has is not used. Where Ruby has no `fork` (Windows, JRuby, TruffleRuby) the + process has is not used. Where Ruby has no `fork` (Windows, JRuby) the workers are new processes running the same `rspec` command with the run's seed, and load the spec files themselves. `--bisect` runs serially, and the reproduction command it prints now ends with diff --git a/rspec-core/README.md b/rspec-core/README.md index e84858319..8a7287747 100644 --- a/rspec-core/README.md +++ b/rspec-core/README.md @@ -458,7 +458,7 @@ custom ordering is not split, with a warning. With a formatter that follows example groups (documentation, HTML), each part reads as a block of its own, under the group's description; `--profile` adds up the parts. -Where Ruby has no `fork` (Windows, JRuby, TruffleRuby), the workers are new +Where Ruby has no `fork` (Windows, JRuby), the workers are new processes running the `rspec` command the run was started with -- the same arguments, `.rspec` files and `SPEC_OPTS`, without the formatters, and the run's seed -- and each loads the spec files itself, as with diff --git a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb index 696a2833d..062878906 100644 --- a/rspec-core/lib/rspec/core/parallel/spawned_worker.rb +++ b/rspec-core/lib/rspec/core/parallel/spawned_worker.rb @@ -1,7 +1,7 @@ module RSpec module Core module Parallel - # Where Ruby has no `fork` (Windows, JRuby, TruffleRuby), a worker is a + # Where Ruby has no `fork` (Windows, JRuby), a worker is a # new process running the `rspec` command the run was started with: it # configures itself from the same arguments, `.rspec` files and # `SPEC_OPTS` -- without formatters, which are the coordinator's -- and diff --git a/rspec-core/spec/integration/parallel_spec.rb b/rspec-core/spec/integration/parallel_spec.rb index 19441e1de..f0949be55 100644 --- a/rspec-core/spec/integration/parallel_spec.rb +++ b/rspec-core/spec/integration/parallel_spec.rb @@ -1367,7 +1367,7 @@ def loaders end end -# Where Ruby has no `fork` (Windows, JRuby, TruffleRuby) the workers are new +# Where Ruby has no `fork` (Windows, JRuby) the workers are new # `rspec` processes; elsewhere a `--require`d file takes `fork` away. RSpec.describe 'Parallel execution without fork (spawned workers)', :slow do include_context "aruba support" diff --git a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb index 4a9e095f6..aa03913a8 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_output_spec.rb @@ -55,10 +55,10 @@ def read(sizes=nil) end it "writes what a worker printed as it was written, also to a stream with an encoding of its own" do - # A file, which (unlike a StringIO) transcodes what is written to it. + # A file, which (unlike a StringIO) transcodes what is written to it; binary, so that Windows adds no CR. path = File.join(dir, 'stdout') original = $stdout - $stdout = File.open(path, 'w:UTF-8') + $stdout = File.open(path, 'wb:UTF-8') begin WorkerOutput.write(:out, "caf\xC3\xA9 \xFF\n".b) ensure diff --git a/rspec-core/spec/rspec/core/parallel/worker_spec.rb b/rspec-core/spec/rspec/core/parallel/worker_spec.rb index 5522febaf..ab161ab7c 100644 --- a/rspec-core/spec/rspec/core/parallel/worker_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/worker_spec.rb @@ -103,6 +103,9 @@ def frames expect(worker.process_alive?(Process.pid)).to be true gone = Process.spawn(RbConfig.ruby, '-e', 'exit') Process.wait(gone) + # Windows can still find a pid for a moment after it was reaped. + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + 5 + sleep 0.05 while worker.process_alive?(gone) && Process.clock_gettime(Process::CLOCK_MONOTONIC) < deadline expect(worker.process_alive?(gone)).to be false end From 29b48c65f7f2c7f11c21efb634cf0e69ccc08c24 Mon Sep 17 00:00:00 2001 From: ilyazub Date: Fri, 16 Oct 2026 04:36:13 +0200 Subject: [PATCH 58/58] [rspec-core] Keep the loading-replacement specs from depending on speed On JRuby the surviving fake worker finished every queued group before the crash was noticed, so no replacement was due. The groups now take 0.5 s each, as in the other replacement spec. Co-Authored-By: Claude Opus 5.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- rspec-core/spec/rspec/core/parallel/coordinator_spec.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb index 0c22610b9..bf9ec5f24 100644 --- a/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb +++ b/rspec-core/spec/rspec/core/parallel/coordinator_spec.rb @@ -979,8 +979,10 @@ def standard_streams(stdout=StringIO.new) end context "when the workers loaded the spec files themselves" do + # The others still running when the crash is noticed, so that groups wait to run. let(:groups) do - [RSpec.describe("Crashing") { it("dies") { Thread.current.kill } }, passing_group("A"), passing_group("B"), passing_group("C")] + [RSpec.describe("Crashing") { it("dies") { Thread.current.kill } }] + + %w[A B C].map { |name| RSpec.describe(name) { it("passes") { sleep 0.5 } } } end before { config.ordering_registry.register(:global, RSpec::Core::Ordering::Identity.new) }