diff --git a/.github/actions/setup-mcpp/action.yml b/.github/actions/setup-mcpp/action.yml index 0dee39d..02972b6 100644 --- a/.github/actions/setup-mcpp/action.yml +++ b/.github/actions/setup-mcpp/action.yml @@ -28,14 +28,20 @@ runs: # Linux package is linked against (its interpreter and run paths) are links there, into the # packages. A store restored without it holds programs that do not start, and was taken for # installed (the emsdk compiler, 0.8.4). + # + # One store per job: a job restores what a job of its own saved. The native job of 0.10.0's pull + # request restored the store the Web job had saved first under their common key, which lacked + # the base GCC builds against (glibc, the Linux headers); its first build installed it, the next + # build found the base it had recorded changed and emptied the build directory, and the + # fixtures `mcpp build` had made for the conformance step were gone. - name: Cache the mcpp package store uses: actions/cache@v5 with: path: | ~/.mcpp/registry/data/xpkgs ~/.mcpp/registry/abi - key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }} - restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}- + key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }} + restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}- - name: Install mcpp shell: bash run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index d5d111d..a6867ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,31 @@ # Changelog +## [0.10.0] - 2026-11-19 + +- **A timeout records the program's processes, asks it to end, and then ends it.** Intermittent + timeouts on mcpp's CI (a test ended with status 137) left no state the first time they occurred. + At a program's timeout the engine now records its processes before anything else: on Linux, from + `/proc`, the program, every process it started and every other process of its unit, each with its + parent, group, command, state, the kernel function each of its threads sleeps in, its processor + time, what the request to end does to it (`ends`, `caught`, `ignored`) and, where the system lets + it be read, its kernel stack; on macOS and Windows the program alone. It then asks the program to + end (openkal's request: `SIGTERM` on Linux and macOS; on Windows the program is ended outright), + waits at most the program's grace, records what still runs, and ends the program's unit. The + case's reason says which ended it (`asked to end, it ended within its grace of 2s`, or `... was + still running after its grace of 2s, and was ended`); the record is printed under the case, kept + as the program's `timeout` in the run's `programs.ndjson`, given by `mtest explain`, and shown + below the failures of the Markdown report (docs/commands.md "Timeouts"). +- **`mtest measure` passes the request on.** It starts a program the engine measures (a program + judged by its exit status, on Linux and macOS), and through 0.9.0 the request ended it and not + the test, which its unit's end then ended outright: a test never saw the request. +- **`grace` and `timeout_record`**, keys of `[programs.*]` and `[discover.*]` beside `timeout`, and + `mtest run --grace`. The grace is 2 s by default, the grace of mcpplibs.os's bounded run that the + engine gave through 0.9.0; `timeout_record = false` takes no record. +- CI keeps a package store per job. The native job restored the store the Web job had saved under + their common key, which lacked the base GCC builds against; its first build installed it, the + next build found the base it had recorded changed and emptied the build directory, and the + fixtures `mcpp build` had made for the conformance step were gone. + ## [0.9.0] - 2026-11-15 - **A selection excludes, and selects where declared** (SPEC-017 P4.3). In `MTEST_SELECT`, a line diff --git a/README.md b/README.md index f121c20..b58f68c 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.9.0" +mcpplibs.mtest = "0.10.0" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program diff --git a/README.zh-CN.md b/README.zh-CN.md index e0da9bf..3aae0c3 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.9.0" +mcpplibs.mtest = "0.10.0" ``` 失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。 diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index 6ca5f40..a41302b 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.9.0" +version = "0.10.0" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index 54bebc8..3b7a57b 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 12ecf2c..611298e 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.9.0" +version = "0.10.0" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 2932965..cdd7ef2 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}}); } diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index 32731c3..2916565 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.9.0" +version = "0.10.0" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" diff --git a/docs/commands.md b/docs/commands.md index 1d3d23c..13231d8 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -36,7 +36,7 @@ nothing; a built program's own listing is the authority (`manifest/src/manifest. | Command | Does | |---|---| -| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | +| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--grace 2s` (below), `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | | `run --resume RUN` | the selection, less the cases RUN judged: their results are carried into the new run, so an interrupted long run (a bench over a matrix) ends as one whole run; a case RUN could not start runs again | | `rerun` | the cases that failed in the last run | | `smoke` | the `smoke` set | @@ -49,6 +49,35 @@ A program runs by the protocol when it carries the marker or declares its level; status is its one result. Cases tagged `flaky`, and quarantined ones, are retried alone up to `--retries`; one that then passes is `flaky`. A quarantined case's failure does not fail the run. +### Timeouts + +A program still running at its timeout (`timeout`, `--timeout`; a case's own `timeout` where a +process runs that case alone) is first asked to end and then ended, and its processes are recorded +before the request, so that a hang leaves the state it hung in: + +1. **The record.** On Linux, from `/proc`: the program, every process it started, and every other + process of its unit (the process group it leads, which a process it started keeps after the + program has ended). Each process states its parent, its group, its command, its state, the kernel + function it sleeps in, its processor time, what the request to end does to it (`ends`, `caught`, + `ignored`), each of its threads with its state and its kernel function, and, where the system lets + it be read, its kernel stack. Where the system lists no other process to the engine (macOS, + Windows, a `/proc` the engine cannot read), the record holds the program alone, with its + processor time, and says so. +2. **The request.** openkal's request to end a program: `SIGTERM` on Linux and macOS, which a + program may handle or ignore; on Windows the program is ended outright. The request reaches the + program the engine started; `mtest measure`, which starts a program the engine measures, passes + it on. What the program started receives it from the program, or not at all. +3. **The grace.** The engine waits for the program at most its `grace` (a key of `[programs.*]` and + `[discover.*]`, [config.md](config.md); `--grace`; 2 s by default). +4. **The end.** What still runs in the program's unit is recorded again and ended. + +The case's reason says how the program ended: `asked to end, it ended within its grace of 2s`, or +`asked to end, it was still running after its grace of 2s, and was ended`. The record is printed +under the case, kept in the run's `programs.ndjson` as the program's `timeout` (`limit_ms`, +`grace_ms`, `ended`, `seen`, `processes`, `remaining`), given by `mtest explain`, and shown below the +failures of the Markdown report. A program that states `timeout_record = false` is asked to end and +ended in the same way, with no record. + ## Asking | Command | Answers | diff --git a/docs/config.md b/docs/config.md index 72460d4..0e3e7d1 100644 --- a/docs/config.md +++ b/docs/config.md @@ -13,19 +13,22 @@ jobs = 0 # 0: the machine's processors # A program of any language. `level` states the protocol's level; without it the marker decides (P2.3). [programs.e2e] -command = ["python3", "tests/e2e.py"] -cwd = "." -env = { MODE = "ci" } -sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection) -resources = ["serial:db"] # what it holds alone (P7.1) -tags = ["slow"] -timeout = "10m" -isolation = "process" # or "case": a process per case -kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit) -split = true # whether [run] split may divide its cases among processes -skip_exit = 77 # a program judged by its exit status: the status that says its case did not - # apply here (skip, with the last `SKIP...` line of its output as the reason); - # unset: every status but 0 fails +command = ["python3", "tests/e2e.py"] +cwd = "." +env = { MODE = "ci" } +sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection) +resources = ["serial:db"] # what it holds alone (P7.1) +tags = ["slow"] +timeout = "10m" +grace = "2s" # at its timeout it is asked to end; what still runs this much later is + # ended (commands.md "Timeouts"); 2s by default +timeout_record = true # its processes are recorded at its timeout and after the grace; false: no record +isolation = "process" # or "case": a process per case +kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit) +split = true # whether [run] split may divide its cases among processes +skip_exit = 77 # a program judged by its exit status: the status that says its case did not + # apply here (skip, with the last `SKIP...` line of its output as the reason); + # unset: every status but 0 fails [discover] globs = ["build/tests/*"] # every executable that matches is a program diff --git a/engine/mcpp.toml b/engine/mcpp.toml index 6ec688d..3c80487 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.9.0" +version = "0.10.0" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index 06b8918..01924a8 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -183,6 +183,10 @@ std::expected, answer> programs_of(const context& ctx, if (auto t = text_of(p, "timeout"); !t.empty()) if (auto d = parse_duration(t)) s.timeout = std::chrono::duration_cast(*d); + for (auto& s : *programs) + if (auto t = text_of(p, "grace"); !t.empty()) + if (auto d = parse_duration(t)) + s.grace = *d; if (p.value("isolate", false)) for (auto& s : *programs) s.isolation = "case"; @@ -825,6 +829,8 @@ answer cmd_explain(const std::filesystem::path& cwd, const json& p) { } if (c.contains("output") && !c["output"]["err_tail"].get().empty()) text += " error output (end):\n" + c["output"]["err_tail"].get() + "\n"; + if (c.contains("timeout")) + text += timeout_text(timeout_info_from(c["timeout"]), " ", 400); if (c.contains("diagnosis")) text += " diagnosis: " + c["diagnosis"].get() + "\n"; text += " again: " + c["again"].get() + "\n"; @@ -1799,6 +1805,10 @@ json catalog() { {"retries", {{"type", "integer"}, {"description", "runs again of a failed case"}}}, {"isolate", {{"type", "boolean"}, {"description", "each case in its own process"}}}, {"timeout", {{"type", "string"}, {"description", "a duration for each program"}}}, + {"grace", + {{"type", "string"}, + {"description", "how long a program asked to end at its timeout has before it " + "is ended"}}}, {"env", {{"type", "object"}, {"description", "NAME=VALUE for the programs"}}}, {"run_under", {{"type", "array"}, @@ -1958,7 +1968,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.9.0"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.10.0"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/config.cpp b/engine/src/config.cpp index 2f94d0b..8f74680 100644 --- a/engine/src/config.cpp +++ b/engine/src/config.cpp @@ -43,12 +43,9 @@ void known(const toml::table& t, std::string_view where, } // The keys a program states, in `[programs.*]` and `[discover.*]` alike. -constexpr std::initializer_list kProgramKeys = {"command", "cwd", - "env", "level", - "resources", "sources", - "tags", "timeout", - "isolation", "kind", - "split", "skip_exit"}; +constexpr std::initializer_list kProgramKeys = + {"command", "cwd", "env", "level", "resources", "sources", "tags", + "timeout", "isolation", "kind", "split", "skip_exit", "grace", "timeout_record"}; void read_program(const toml::table& t, program_spec& s) { if (auto v = t["cwd"].value()) @@ -67,6 +64,10 @@ void read_program(const toml::table& t, program_spec& s) { s.tags = strings(t["tags"]); if (auto d = duration_of(t["timeout"])) s.timeout = std::chrono::duration_cast(*d); + if (auto d = duration_of(t["grace"])) + s.grace = *d; + if (auto v = t["timeout_record"].value()) + s.timeout_record = *v; if (auto v = t["isolation"].value()) s.isolation = *v; if (auto v = t["split"].value()) diff --git a/engine/src/exec.cpp b/engine/src/exec.cpp index caac196..09a0e60 100644 --- a/engine/src/exec.cpp +++ b/engine/src/exec.cpp @@ -2,6 +2,7 @@ module mcpplibs.mtest.engine.exec; import std; import mcpplibs.os; +import mcpplibs.mtest.engine.processes; namespace mcpplibs::mtest::engine { @@ -71,6 +72,141 @@ std::filesystem::path resolve_program(const std::string& name) { return p; } +// A RUN WITHIN ITS TIMEOUT (0.10.0). mcpplibs.os's bounded run asks a program past its limit to end +// and ends its unit after a grace, with nothing between; the engine composes the same parts with a +// record between them, so that a hang leaves the state it hung in. The outputs, their limits and +// their drain are that run's. + +// One output, read by a thread of its own in slices, so that the run can stop reading once the +// program has ended and its outputs have had their drain. +struct capture { + std::mutex lock; + std::string data; + std::size_t limit = 0; +}; +struct readers { + std::mutex lock; + std::condition_variable changed; + int open = 2; + std::atomic stop{false}; +}; +constexpr std::chrono::milliseconds read_slice{50}; + +void collect(std::shared_ptr p, std::shared_ptr into, + std::shared_ptr all, bool output) { + while (!all->stop.load(std::memory_order_relaxed)) { + auto chunk = output ? p->read_output_for(read_slice) : p->read_error_for(read_slice); + if (!chunk || (chunk->has_value() && (*chunk)->empty())) + break; // the stream failed, or ended + if (!chunk->has_value()) + continue; + // What passes the limit is read and dropped, so that the program never waits on a full + // pipe. + std::lock_guard g(into->lock); + const auto room = into->limit - std::min(into->limit, into->data.size()); + into->data.append(**chunk, 0, std::min(room, (*chunk)->size())); + } + { + std::lock_guard g(all->lock); + --all->open; + } + all->changed.notify_all(); +} + +struct ended_run { + os::process::exit_status status; + std::string out, err; + std::optional timeout; // past its limit +}; + +// Past its limit: its processes recorded, the program asked to end, and what still runs after its +// grace recorded and ended with its unit. The first record precedes the request, which changes or +// ends what it reaches. +timeout_info end_past_limit(os::process::process& p, const program_spec& spec) { + timeout_info t; + t.limit_ms = std::chrono::duration(spec.timeout).count(); + t.grace_ms = double(spec.grace.count()); + if (spec.timeout_record) { + auto seen = processes_of(p.id()); + t.seen = std::move(seen.seen); + t.processes = std::move(seen.processes); + } + p.terminate(); + const auto within = p.wait_for(spec.grace); + t.ended = within && *within ? "requested" : "terminated"; + if (spec.timeout_record) { + // What still runs: a process that has ended and is not yet collected runs no more. + t.remaining = processes_of(p.id()).processes; + std::erase_if(t.remaining, [](const process_info& x) { + return x.state == "zombie" || x.state == "dead"; + }); + } + p.end_unit(); + return t; +} + +os::result run_within(os::process::spawn_options o, const program_spec& spec) { + const os::process::bounds bounded; + o.out = os::process::stdio::pipe; + o.err = os::process::stdio::pipe; + o.own_unit = true; + auto spawned = os::process::process::spawn(o); + if (!spawned) + return std::unexpected(spawned.error()); + auto p = std::make_shared(std::move(*spawned)); + auto out = std::make_shared(); + out->limit = bounded.out_limit; + auto err = std::make_shared(); + err->limit = bounded.err_limit; + auto all = std::make_shared(); + std::thread(collect, p, out, all, true).detach(); + std::thread(collect, p, err, all, false).detach(); + + ended_run r; + auto waited = p->wait_for(std::chrono::duration_cast(spec.timeout)); + if (!waited) + return std::unexpected(waited.error()); + if (waited->has_value()) { + r.status = **waited; + } else { + r.timeout = end_past_limit(*p, spec); + auto ended = p->wait(); + r.status = ended ? *ended : os::process::exit_status{-1, true}; + } + { + std::unique_lock g(all->lock); + if (!all->changed.wait_for(g, bounded.drain, [&] { return all->open == 0; })) { + // The program has ended, and something it started holds its outputs open. + p->end_unit(); + all->stop.store(true, std::memory_order_relaxed); + all->changed.wait_for(g, read_slice * 4, [&] { return all->open == 0; }); + } + } + { + std::lock_guard g(out->lock); + r.out = out->data; + } + { + std::lock_guard g(err->lock); + r.err = err->data; + } + return r; +} + +std::string duration_words(double ms) { + return std::fmod(ms, 1000) == 0 ? std::format("{}s", ms / 1000) : std::format("{}ms", ms); +} + +// A case's reason for a timeout, `what`, and how the program was then ended. +std::string timeout_reason(std::string_view what, const timeout_info& t) { + const auto grace = duration_words(t.grace_ms); + return t.ended == "requested" + ? std::format("{}; asked to end, it ended within its grace of {}", what, grace) + : std::format("{}; asked to end, it was still running after its grace of {}, " + "and was ended", + what, grace); +} + } // namespace int detect_level(const program_spec& spec) { @@ -245,6 +381,9 @@ list_program(const program_spec& spec, const std::filesystem::path& work, int measure(const std::filesystem::path& out, std::vector argv) { if (argv.empty()) return 2; + // At a timeout the engine asks this process to end, and the test is the program it starts: the + // request is taken over before the start and passed on, and the program ends as it chooses. + const bool passes_on = os::interrupt::available(); os::process::spawn_options o; o.program = resolve_program(argv.front()); o.args.assign(argv.begin() + 1, argv.end()); @@ -266,8 +405,19 @@ int measure(const std::filesystem::path& out, std::vector argv) { // The account of a child is the kernel's once the child is reaped, which closing its // handle does. auto child = std::move(*p); - if (auto s = child.wait()) - st = *s; + for (bool passed = false;;) { + auto s = child.wait_for(std::chrono::milliseconds{50}); + if (!s) + break; + if (s->has_value()) { + st = **s; + break; + } + if (passes_on && !passed && os::interrupt::requested()) { + child.terminate(); + passed = true; + } + } } json j = {{"source", "engine"}}; if (auto u = os::process::resources(os::process::of::children)) { @@ -370,10 +520,8 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { o.env.emplace_back("MTEST_MONITOR_SAMPLE", std::to_string(request.sample_ms)); if (request.seed) o.env.emplace_back("MTEST_SEED", std::to_string(*request.seed)); - os::process::bounds limits; - limits.hard = std::chrono::duration_cast(spec.timeout); const auto began = std::chrono::steady_clock::now(); - auto done = os::process::run(std::move(o), limits); + auto done = run_within(std::move(o), spec); p.ms = std::chrono::duration(std::chrono::steady_clock::now() - began).count(); std::filesystem::remove_all(request.work / "tmp", ec); @@ -390,7 +538,10 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { return out; } p.exit = done->status.shell_code(); - p.timed_out = done->timed_out; + p.timed_out = done->timeout.has_value(); + p.timeout = std::move(done->timeout); + // What a case that the timeout ended says of it. + const auto timed_out = [&](std::string_view what) { return timeout_reason(what, *p.timeout); }; p.out_tail = tail(done->out); p.err_tail = tail(done->err); out.out = std::move(done->out); @@ -430,7 +581,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { if (c.status == "skip") c.reason = skip_reason(out.out, out.err, p.exit); else if (c.status != "pass") - c.reason = p.timed_out ? std::format("still running after {}", spec.timeout) + c.reason = p.timed_out ? timed_out(std::format("still running after {}", spec.timeout)) : std::format("exit {}", p.exit); out.cases.push_back(std::move(c)); } else { @@ -448,7 +599,9 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { if (c.status == "skip") c.reason = skip_reason(out.out, out.err, p.exit); else if (c.status != "pass") - c.reason = std::format("exit {}", p.exit); + c.reason = p.timed_out + ? timed_out(std::format("still running after {}", spec.timeout)) + : std::format("exit {}", p.exit); out.cases.push_back(std::move(c)); } else { for (auto& c : r.cases) { @@ -456,7 +609,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { // P5.4: begun and never ended -- the program ended (or was ended) first. c.status = p.timed_out ? "timeout" : "fail"; c.reason = p.timed_out - ? "the program was still running at its timeout" + ? timed_out("the program was still running at its timeout") : std::format("the program ended (exit {}) before the case did", p.exit); } @@ -474,7 +627,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { c.id = spec.name; c.ms = p.ms; c.status = p.timed_out ? "timeout" : "fail"; - c.reason = p.timed_out ? "the program was still running at its timeout" + c.reason = p.timed_out ? timed_out("the program was still running at its timeout") : std::format("exit {} with no failing case{}", p.exit, r.done ? "" : ", and no `done' event"); out.cases.push_back(std::move(c)); diff --git a/engine/src/main.cpp b/engine/src/main.cpp index b158261..e0573b7 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.9.0") + .version("0.10.0") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.9.0"); + std::println("mtest 0.10.0"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/engine/src/model.cpp b/engine/src/model.cpp index 1ad9ccc..c863564 100644 --- a/engine/src/model.cpp +++ b/engine/src/model.cpp @@ -127,11 +127,91 @@ json to_json(const case_result& c) { j["started_ms"] = c.started_ms; return j; } + +namespace { + +// A process as the timeout recorded it; an empty field is one the system did not tell. +json to_json(const process_info& p) { + json j = {{"id", p.id}, {"parent", p.parent}, {"group", p.group}, {"name", p.name}, + {"command", p.command}, {"state", p.state}, {"cpu_ms", p.cpu_ms}}; + if (!p.waiting.empty()) + j["waiting"] = p.waiting; + if (!p.request.empty()) + j["request"] = p.request; + if (!p.threads.empty()) { + j["threads"] = json::array(); + for (const auto& t : p.threads) { + json one = {{"id", t.id}, {"name", t.name}, {"state", t.state}, {"cpu_ms", t.cpu_ms}}; + if (!t.waiting.empty()) + one["waiting"] = t.waiting; + j["threads"].push_back(std::move(one)); + } + } + if (!p.stack.empty()) + j["stack"] = p.stack; + return j; +} + +std::vector processes_from(const json& j, const char* key) { + std::vector out; + auto it = j.find(key); + if (it == j.end() || !it->is_array()) + return out; + for (const auto& e : *it) { + process_info p; + p.id = e.value("id", std::uint64_t{0}); + p.parent = e.value("parent", std::uint64_t{0}); + p.group = e.value("group", std::uint64_t{0}); + p.name = str(e, "name"); + p.command = strings(e, "command"); + p.state = str(e, "state"); + p.waiting = str(e, "waiting"); + p.request = str(e, "request"); + p.cpu_ms = e.value("cpu_ms", 0.0); + p.stack = str(e, "stack"); + if (auto t = e.find("threads"); t != e.end() && t->is_array()) + for (const auto& x : *t) + p.threads.push_back({x.value("id", std::uint64_t{0}), str(x, "name"), + str(x, "state"), str(x, "waiting"), x.value("cpu_ms", 0.0)}); + out.push_back(std::move(p)); + } + return out; +} + +} // namespace + +json to_json(const timeout_info& t) { + json j = {{"limit_ms", t.limit_ms}, {"grace_ms", t.grace_ms}, {"ended", t.ended}}; + if (!t.seen.empty()) { + j["seen"] = t.seen; + j["processes"] = json::array(); + for (const auto& p : t.processes) + j["processes"].push_back(to_json(p)); + j["remaining"] = json::array(); + for (const auto& p : t.remaining) + j["remaining"].push_back(to_json(p)); + } + return j; +} + +timeout_info timeout_info_from(const json& j) { + timeout_info t; + t.limit_ms = j.value("limit_ms", 0.0); + t.grace_ms = j.value("grace_ms", 0.0); + t.ended = str(j, "ended"); + t.seen = str(j, "seen"); + t.processes = processes_from(j, "processes"); + t.remaining = processes_from(j, "remaining"); + return t; +} + json to_json(const program_result& p) { json j = {{"name", p.name}, {"argv", p.argv}, {"cwd", p.cwd}, {"level", p.level}, {"status", p.status}, {"exit", p.exit}, {"ms", p.ms}}; if (p.timed_out) j["timed_out"] = true; + if (p.timeout) + j["timeout"] = to_json(*p.timeout); if (!p.out_tail.empty()) j["out_tail"] = p.out_tail; if (!p.err_tail.empty()) @@ -276,6 +356,8 @@ program_result program_result_from(const json& j) { p.exit = j.value("exit", 0); p.ms = j.value("ms", 0.0); p.timed_out = j.value("timed_out", false); + if (auto t = j.find("timeout"); t != j.end() && t->is_object()) + p.timeout = timeout_info_from(*t); p.out_tail = str(j, "out_tail"); p.err_tail = str(j, "err_tail"); p.error = str(j, "error"); diff --git a/engine/src/model.cppm b/engine/src/model.cppm index a030309..1318e16 100644 --- a/engine/src/model.cppm +++ b/engine/src/model.cppm @@ -20,6 +20,11 @@ struct program_spec { std::vector sources; // paths whose change affects it (affected selection) std::vector tags; std::chrono::seconds timeout{300}; + // At its timeout it is asked to end, and what still runs `grace` later is ended (0.10.0); the + // default is the grace of mcpplibs.os's bounded run, which the engine gave through 0.9.0. With + // `timeout_record`, its processes are recorded at the timeout and after the grace. + std::chrono::milliseconds grace{std::chrono::seconds{2}}; + bool timeout_record = true; std::string isolation = "process"; // "process" or "case" std::string origin = "config"; // "given" (the command line), "config", "discover", "host" // The kind of an L0 program's one case and of a case that states none: stated in mtest.toml, @@ -118,6 +123,41 @@ struct case_result { double started_ms = -1; }; +// A thread of a process, as the engine saw it at a timeout (0.10.0). +struct thread_info { + std::uint64_t id = 0; + std::string name, state; + std::string waiting; // the kernel function it sleeps in (Linux: wchan); empty: not told + double cpu_ms = 0; +}; + +// A process of a program's unit, as the engine saw it at a timeout (0.10.0). What the system does +// not tell, or does not let the engine read, is empty. +struct process_info { + std::uint64_t id = 0, parent = 0, group = 0; + std::string name; + std::vector command; + std::string state; // running, sleeping, waiting (uninterruptible), stopped, zombie, ... + std::string waiting; // as its first thread's + std::string request; // what the request to end does to it: ends, caught, ignored + double cpu_ms = 0; + std::vector threads; // when it has more than one + std::string stack; // the kernel's stack of its first thread, where readable +}; + +// How a program past its timeout was ended (0.10.0): its processes were recorded, it was asked to +// end, and what still ran after its grace was recorded and ended. +struct timeout_info { + double limit_ms = 0, grace_ms = 0; + std::string ended; // "requested": it ended within its grace; "terminated": ended after it + // "tree": the program, the processes it started and the rest of its unit (Linux); "program": + // the program alone, where the system lists no other process to the engine; empty: not recorded + // (`timeout_record = false`). + std::string seen; + std::vector processes; // at the timeout, before the request + std::vector remaining; // after the grace, before the unit was ended +}; + struct program_result { std::string name; std::vector argv; @@ -127,9 +167,10 @@ struct program_result { int exit = 0; double ms = 0; bool timed_out = false; - std::string out_tail, err_tail; // the last part of each output - std::string error; // why it did not run, for an infra status - std::string origin = "config"; // as program_spec's: how the program is named again + std::optional timeout; // how it was ended at its timeout, and its processes + std::string out_tail, err_tail; // the last part of each output + std::string error; // why it did not run, for an infra status + std::string origin = "config"; // as program_spec's: how the program is named again // When it started, in milliseconds after the run began, and what it held: the run's timeline. double started_ms = 0; std::vector resources; @@ -179,6 +220,7 @@ std::vector bounds_from(const json& j); json to_json(const failure_info& f); json to_json(const metric_info& m); json to_json(const case_result& c); +json to_json(const timeout_info& t); json to_json(const program_result& p); json to_json(const run_record& r); // without programs and cases (run.json) json to_json(const fixture_run& f); @@ -186,6 +228,7 @@ fixture_run fixture_run_from(const json& j); case_info case_info_from(const json& j); case_result case_result_from(const json& j); +timeout_info timeout_info_from(const json& j); program_result program_result_from(const json& j); run_record run_record_from(const json& j); diff --git a/engine/src/output.cpp b/engine/src/output.cpp index 556080d..3e86630 100644 --- a/engine/src/output.cpp +++ b/engine/src/output.cpp @@ -43,8 +43,91 @@ std::string shell_quote(std::string_view t) { return out + "'"; } +// `sleeping in do_wait, ignores the request to end, 3 ms of processor time`. +std::string facts_of(std::string_view state, std::string_view waiting, std::string_view request, + double cpu_ms) { + std::string out; + if (!state.empty()) + out = waiting.empty() ? std::string(state) : std::format("{} in {}", state, waiting); + if (request == "ignored" || request == "caught") + out += std::format("{}{} the request to end", out.empty() ? "" : ", ", + request == "ignored" ? "ignores" : "handles"); + return out + std::format("{}{:.0f} ms of processor time", out.empty() ? "" : ", ", cpu_ms); +} + +std::string process_line(const process_info& p) { + std::string command; + for (const auto& word : p.command) + command += (command.empty() ? "" : " ") + word; + if (command.empty()) + command = p.name; + // A compiler's command line is the record's, not the terminal's. + constexpr std::size_t shown = 160; + if (command.size() > shown) + command = command.substr(0, shown) + " ..."; + return std::format("{}{}{}: {}", p.id, command.empty() ? "" : " ", command, + facts_of(p.state, p.waiting, p.request, p.cpu_ms)); +} + +// Each process under the one that started it, with its threads and its kernel stack. +void add_processes(std::vector& lines, const std::vector& processes, + const std::string& indent) { + std::map depth; + for (const auto& p : processes) { + const auto parent = depth.find(p.parent); + const auto d = parent == depth.end() ? std::size_t{0} : parent->second + 1; + depth[p.id] = d; + const auto at = indent + std::string(2 * d, ' '); + lines.push_back(at + process_line(p)); + for (const auto& t : p.threads) + lines.push_back(std::format("{} thread {} {}: {}", at, t.id, t.name, + facts_of(t.state, t.waiting, {}, t.cpu_ms))); + if (p.stack.empty()) + continue; + for (auto part : std::views::split(std::string_view(p.stack), '\n')) + lines.push_back(std::format("{} {}", at, std::string_view(part.begin(), part.end()))); + } +} + } // namespace +const program_result* timed_out_program(const run_record& r, std::string_view program, + std::string_view id) { + const program_result* any = nullptr; + for (const auto& p : r.programs) { + if (p.name != program || !p.timeout) + continue; + if (p.select.empty() || std::ranges::find(p.select, id) != p.select.end()) + return &p; + any = &p; + } + return any; +} + +std::string timeout_text(const timeout_info& t, std::string_view indent, std::size_t most) { + if (t.seen.empty()) + return {}; + const std::string at(indent); + std::vector lines; + lines.push_back(at + (t.seen == "tree" ? "processes at the timeout:" + : "the program at the timeout (this system lists no " + "other process to mtest):")); + add_processes(lines, t.processes, at + " "); + if (!t.remaining.empty()) { + lines.push_back(at + (t.ended == "requested" + ? "left running by the program, and ended with its unit:" + : "still running after the grace, and ended:")); + add_processes(lines, t.remaining, at + " "); + } + std::string out; + for (std::size_t i = 0; i < lines.size() && i < most; ++i) + out += lines[i] + "\n"; + if (lines.size() > most) + out += std::format("{}... {} more lines: the whole record is in `mtest explain --json`\n", + at, lines.size() - most); + return out; +} + std::string reproduce_command(std::string_view program, std::string_view id) { return std::format("mtest run --program {} --case {}", shell_quote(program), shell_quote(id)); } @@ -76,6 +159,8 @@ std::string summary_text(const run_record& r, bool verbose) { auto it = r.counts.find(k); return it == r.counts.end() ? 0 : it->second; }; + // A program's processes at its timeout, under the first of its cases the timeout ended. + std::set recorded; for (const auto& c : r.cases) { if (!failing(c.status) && c.status != "infra" && !verbose) continue; @@ -90,6 +175,9 @@ std::string summary_text(const run_record& r, bool verbose) { if (!f.expected.empty() || !f.actual.empty()) out += std::format(" expected {}\n actual {}\n", f.expected, f.actual); } + if (const auto* p = c.status == "timeout" ? timed_out_program(r, c.program, c.id) : nullptr; + p && recorded.insert(p).second) + out += timeout_text(*p->timeout, " ", 40); if (failing(c.status)) out += std::format(" again: {}\n", again(c)); } @@ -221,6 +309,10 @@ std::expected explain(const store& s, const run_record& r, {"cwd", p.cwd}, {"level", p.level}, {"exit", p.exit}}; + // How its timeout ended it, and the processes recorded then. + if (const auto* p = + c->status == "timeout" ? timed_out_program(r, c->program, c->id) : nullptr) + item["timeout"] = to_json(*p->timeout); // The case's recent history: a new failure, one that keeps failing, or one that changes. json recent = json::array(); auto all = runs(s); diff --git a/engine/src/output.cppm b/engine/src/output.cppm index 5f4f7f8..d52411b 100644 --- a/engine/src/output.cppm +++ b/engine/src/output.cppm @@ -9,6 +9,14 @@ export import mcpplibs.mtest.engine.store; export namespace mcpplibs::mtest::engine { std::string summary_text(const run_record& r, bool verbose = false); + +// The program of a run whose timeout ended the case `id` (0.10.0): the process that ran it, else +// another of that program's that timed out; null when none did. +const program_result* timed_out_program(const run_record& r, std::string_view program, + std::string_view id); +// The processes a timeout recorded, as lines a person reads, each begun by `indent`: at most `most` +// of them, then how many more the record holds. Empty when no record was taken. +std::string timeout_text(const timeout_info& t, std::string_view indent, std::size_t most); std::string junit_xml(const run_record& r); std::string html_page(const run_record& r); diff --git a/engine/src/processes.cpp b/engine/src/processes.cpp new file mode 100644 index 0000000..0724c07 --- /dev/null +++ b/engine/src/processes.cpp @@ -0,0 +1,241 @@ +module mcpplibs.mtest.engine.processes; + +import std; +import mcpplibs.os; + +namespace mcpplibs::mtest::engine { + +namespace os = mcpplibs::os; + +namespace { + +std::optional number(std::string_view text, int base = 10) { + std::uint64_t v = 0; + const auto [end, ec] = std::from_chars(text.data(), text.data() + text.size(), v, base); + if (ec != std::errc{} || end != text.data() + text.size()) + return std::nullopt; + return v; +} + +// A file of /proc without its trailing line ends; empty where it cannot be read, as the kernel's +// stack of a process is to all but the superuser. +std::string proc_text(const std::filesystem::path& file) { + auto text = os::fs::read(file); + if (!text) + return {}; + while (!text->empty() && (text->back() == '\n' || text->back() == ' ')) + text->pop_back(); + return *text; +} + +// /proc//stat: ` () ...`, user and system time the 12th and +// 13th fields after the name, in ticks of USER_HZ (100), as mcpplibs.os reads a thread's. +struct stat_line { + std::uint64_t id = 0, parent = 0, group = 0; + std::string name; + char state = '?'; + double cpu_ms = 0; +}; + +std::optional read_stat(const std::filesystem::path& file) { + const auto text = proc_text(file); + const auto open = text.find('('); + const auto close = text.rfind(')'); + if (open == std::string::npos || close == std::string::npos || close < open || + close + 2 >= text.size() || open < 2) + return std::nullopt; + std::vector fields; + for (auto part : std::views::split(std::string_view(text).substr(close + 2), ' ')) + fields.emplace_back(part.begin(), part.end()); + const auto id = number(std::string_view(text).substr(0, open - 1)); + if (fields.size() < 13 || fields[0].empty() || !id) + return std::nullopt; + stat_line s; + s.id = *id; + s.name = text.substr(open + 1, close - open - 1); + s.state = fields[0].front(); + s.parent = number(fields[1]).value_or(0); + s.group = number(fields[2]).value_or(0); + s.cpu_ms = double(number(fields[11]).value_or(0) + number(fields[12]).value_or(0)) * 10; + return s; +} + +// The kernel's letters for a state (proc(5)), as a person reads them. +std::string state_word(char state) { + switch (state) { + case 'R': + return "running"; + case 'S': + return "sleeping"; + case 'D': + return "waiting (uninterruptible)"; + case 'T': + return "stopped"; + case 't': + return "traced"; + case 'Z': + return "zombie"; + case 'X': + return "dead"; + case 'I': + return "idle"; + case 'P': + return "parked"; + default: + return std::string(1, state); + } +} + +// The kernel function a thread sleeps in; `0` while it runs, or where the kernel does not say. +std::string waiting_in(const std::filesystem::path& dir) { + auto w = proc_text(dir / "wchan"); + return w == "0" ? std::string{} : w; +} + +// What SIGTERM, the request to end on Linux (openkal's kal_process_terminate), does to a process: +// its disposition, which /proc//status states as masks of the signals ignored and caught. +std::string request_of(const std::filesystem::path& dir) { + const auto status = proc_text(dir / "status"); + const auto mask = [&](std::string_view key) -> std::uint64_t { + for (auto part : std::views::split(std::string_view(status), '\n')) { + std::string_view line(part.begin(), part.end()); + if (!line.starts_with(key)) + continue; + line.remove_prefix(key.size()); + while (!line.empty() && (line.front() == ':' || line.front() == '\t')) + line.remove_prefix(1); + return number(line, 16).value_or(0); + } + return 0; + }; + constexpr std::uint64_t term = std::uint64_t{1} << (15 - 1); + if (status.empty()) + return {}; + return mask("SigIgn") & term ? "ignored" : mask("SigCgt") & term ? "caught" : "ends"; +} + +// Its arguments, NUL-separated; a command past 4 KiB keeps its first arguments and says so, so +// that a compiler's line of thousands of words does not become the record. +std::vector command_of(const std::filesystem::path& dir) { + std::vector out; + const auto text = os::fs::read(dir / "cmdline"); + if (!text) + return out; + constexpr std::size_t keep = 4096; + std::size_t kept = 0; + for (auto part : std::views::split(std::string_view(*text), '\0')) { + std::string_view word(part.begin(), part.end()); + if (kept + word.size() > keep) { + out.push_back(std::format("... ({} bytes in all)", text->size())); + break; + } + kept += word.size(); + out.emplace_back(word); + } + while (!out.empty() && out.back().empty()) + out.pop_back(); + return out; +} + +std::vector threads_of(const std::filesystem::path& dir) { + std::vector out; + std::error_code ec; + for (std::filesystem::directory_iterator it(dir / "task", ec), end; !ec && it != end; + it.increment(ec)) { + if (auto s = read_stat(it->path() / "stat")) + out.push_back( + {s->id, s->name, state_word(s->state), waiting_in(it->path()), s->cpu_ms}); + } + std::ranges::sort(out, {}, &thread_info::id); + // One thread is the process itself, which its own fields already state. + if (out.size() < 2) + out.clear(); + return out; +} + +process_info described(const stat_line& s) { + const std::filesystem::path dir = std::format("/proc/{}", s.id); + process_info p; + p.id = s.id; + p.parent = s.parent; + p.group = s.group; + p.name = s.name; + p.state = state_word(s.state); + p.cpu_ms = s.cpu_ms; + p.command = command_of(dir); + p.waiting = waiting_in(dir); + p.request = request_of(dir); + p.threads = threads_of(dir); + p.stack = proc_text(dir / "stack"); + return p; +} + +// Every process /proc lists, by its stat line. +std::vector process_table() { + std::vector out; + std::error_code ec; + for (std::filesystem::directory_iterator it("/proc", ec), end; !ec && it != end; + it.increment(ec)) { + if (!number(os::fs::utf8(it->path().filename()))) + continue; + if (auto s = read_stat(it->path() / "stat")) + out.push_back(std::move(*s)); + } + return out; +} + +// The program alone, as mcpplibs.os describes it: its identifier and its processor time. +processes_seen alone(std::uint64_t program) { + processes_seen r{"program", {}}; + auto d = os::process::describe(program); + if (!d || d->ended) + return r; + process_info p; + p.id = program; + p.cpu_ms = double(d->cpu_ns) / 1e6; + r.processes.push_back(std::move(p)); + return r; +} + +} // namespace + +processes_seen processes_of(std::uint64_t program) { + if (os::host() != os::platform::linux || program == 0) + return alone(program); + const auto table = process_table(); + if (table.empty()) + return alone(program); + std::map by_id; + std::multimap children; + for (const auto& s : table) { + by_id[s.id] = &s; + children.emplace(s.parent, s.id); + } + // The unit is the group the program leads: a process it started stays there after the program + // has ended and its children are given to another parent. Each is walked with what descends + // from it, which a process that formed a group of its own does too; the program's own first. + std::vector pending; + for (const auto& s : table) + if (s.group == program && s.id != program) + pending.push_back(s.id); + std::ranges::reverse(pending); + pending.push_back(program); + processes_seen r{"tree", {}}; + std::set taken; + while (!pending.empty()) { + const auto id = pending.back(); + pending.pop_back(); + const auto found = by_id.find(id); + if (found == by_id.end() || !taken.insert(id).second) + continue; + r.processes.push_back(described(*found->second)); + const auto [first, last] = children.equal_range(id); + std::vector under; + for (auto c = first; c != last; ++c) + under.push_back(c->second); + pending.insert(pending.end(), under.rbegin(), under.rend()); + } + return r; +} + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/processes.cppm b/engine/src/processes.cppm new file mode 100644 index 0000000..25084e9 --- /dev/null +++ b/engine/src/processes.cppm @@ -0,0 +1,23 @@ +// The processes of a program's unit as the system shows them (0.10.0): what a timeout records +// before it asks the program to end, and again after the program's grace (docs/commands.md +// "Running"). +export module mcpplibs.mtest.engine.processes; + +import std; +export import mcpplibs.mtest.engine.model; + +export namespace mcpplibs::mtest::engine { + +struct processes_seen { + std::string seen; // "tree" or "program", as timeout_info's + std::vector processes; +}; + +// The program the system identifies by `program`, every process it started, and every other process +// of its unit, the program's subtree first: on Linux from /proc, each with its state, what it waits +// on, what the request to end does to it, and its threads. Where the system lists no other process +// to the engine (macOS, Windows, a /proc it cannot read), the program alone, as mcpplibs.os +// describes it; nothing once the program has ended. +processes_seen processes_of(std::uint64_t program); + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/report.cpp b/engine/src/report.cpp index c763c5c..de8e957 100644 --- a/engine/src/report.cpp +++ b/engine/src/report.cpp @@ -212,6 +212,30 @@ void failures(std::string& md, const std::vector& cases) { md += "\nRepeat one with `mtest explain --case ''`.\n\n"; } +// Below the failures, the processes each timeout recorded (0.10.0), once per program. +void timeout_records(std::string& md, const std::vector& runs, + const std::vector& cases) { + const std::set in(cases.begin(), cases.end()); + std::set shown; + std::string blocks; + for (const auto& r : runs) + for (const auto& c : r.cases) { + const auto* p = c.status == "timeout" && in.contains(&c) + ? timed_out_program(r, c.program, c.id) + : nullptr; + if (!p || p->timeout->seen.empty() || !shown.insert(p).second) + continue; + blocks += std::format("
{}: " + "{}\n\n```text\n{}```\n\n
\n\n", + case_label(c.program, c.id), + p->timeout->ended == "requested" ? "it ended when asked" + : "it was ended after its grace", + timeout_text(*p->timeout, "", 200)); + } + if (!blocks.empty()) + md += "The processes of each program a timeout ended, as they were then:\n\n" + blocks; +} + void slowest(std::string& md, std::vector cases, int top) { std::ranges::sort(cases, std::greater{}, [](const case_result* c) { return c->ms; }); md += "## Slowest cases\n\n| case | kind | time | status |\n|---|---|---:|---|\n"; @@ -391,8 +415,10 @@ std::string markdown_report(const store& s, const std::vector& runs, summary(md, runs, o); if (wants(o, "kinds")) kinds(md, cases); - if (wants(o, "failures")) + if (wants(o, "failures")) { failures(md, cases); + timeout_records(md, runs, cases); + } if (wants(o, "slowest")) slowest(md, cases, o.top); if (wants(o, "metrics")) diff --git a/engine/src/service.cpp b/engine/src/service.cpp index a03aa69..11d1feb 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.9.0"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.10.0"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, " diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index 335219c..5d5f53a 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -50,6 +50,13 @@ json json_of(const e2e::outcome& r) { return j; } +// Whether a timeout's record holds every process of a program's unit here: Linux lists its +// processes in /proc, where the engine, as this program, reads it. +bool sees_the_tree() { + return mcpplibs::os::host() == mcpplibs::os::platform::linux && + std::filesystem::exists("/proc/self/stat"); +} + // A protocol script: the cases given as `id:status`, each with a function that does what its status // says. std::string script(std::initializer_list cases, std::string_view extra = {}) { @@ -113,6 +120,80 @@ int main() { auto j = json_of(mtest(root, {"run", "--json", "--timeout", "1s", "bash hang.sh"})); expect(eq(j["counts"].value("timeout", 0), 1)); }; + "a timeout records the program's processes, then asks it to end"_test[kind::e2e][tag{"slow"}] = + [] { + // bash ends on the request; the `sleep` it started does not receive it, and is ended + // with the program's unit. A grace that long is not waited for by a program that ends. + auto root = project({{"hang.sh", "sleep 30\n"}}); + auto r = mtest(root, {"run", "--timeout", "1s", "--grace", "60s", "bash hang.sh"}); + expect(contains(r.out, "asked to end, it ended within its grace of 60s")) << r.out; + auto j = json_of(mtest(root, {"explain", "--json"})); + auto t = j["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << j.dump(); + expect(eq(t.value("ended", ""), std::string("requested"))) << t.dump(); + expect(eq(t.value("grace_ms", 0.0), 60000.0)) << t.dump(); + if (!sees_the_tree()) { + expect(eq(t.value("seen", ""), std::string("program"))) << t.dump(); + expect(le(t["processes"].size(), std::size_t(1))) << t.dump(); + return; + } + expect(eq(t.value("seen", ""), std::string("tree"))) << t.dump(); + expect(contains(r.out, "processes at the timeout:")) << r.out; + const auto& seen = t["processes"]; + const auto sleeping = std::find_if(seen.begin(), seen.end(), [](const json& p) { + return p.value("name", "") == "sleep"; + }); + expect(fatal(sleeping != seen.end())) << t.dump(); + expect(eq(sleeping->value("state", ""), std::string("sleeping"))) << t.dump(); + expect(eq(sleeping->value("request", ""), std::string("ends"))) << t.dump(); + expect(contains(t["remaining"].dump(), "\"sleep\"")) << t.dump(); + }; + "a program that ignores the request to end is ended after its grace"_test[kind::e2e][tag{ + "slow"}] = [] { + auto root = + project({{"stubborn.sh", "trap '' TERM\nsleep 30\n"}, + {"mtest.toml", "[programs.stubborn]\ncommand = [\"bash\", " + "\"stubborn.sh\"]\ntimeout = \"1s\"\ngrace = \"1s\"\n"}}); + const auto began = std::chrono::steady_clock::now(); + auto r = mtest(root, {"run"}); + const std::chrono::duration took = std::chrono::steady_clock::now() - began; + auto j = json_of(mtest(root, {"explain", "--json"})); + auto t = j["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << j.dump(); + expect(eq(j["cases"][0].value("status", ""), std::string("timeout"))); + // openkal's request to end a program is TerminateProcess on Windows, which no + // program declines. + if (mcpplibs::os::host() == mcpplibs::os::platform::windows) { + expect(eq(t.value("ended", ""), std::string("requested"))) << t.dump(); + return; + } + expect(eq(t.value("ended", ""), std::string("terminated"))) << t.dump(); + expect(ge(took.count(), 2.0)) << "the timeout, then the grace"; + expect(contains(r.out, "still running after its grace of 1s, and was ended")) << r.out; + if (!sees_the_tree()) + return; + // bash, and the `sleep` that inherits its disposition, ignore the request. + const auto& left = t["remaining"]; + const auto ignoring = std::count_if(left.begin(), left.end(), [](const json& p) { + return p.value("request", "") == "ignored"; + }); + expect(ge(ignoring, 2)) << t.dump(); + }; + "a protocol program's case says how its timeout ended it; timeout_record = false records " + "nothing"_test[kind::e2e][tag{"slow"}] = [] { + auto root = + project({{"t.sh", script({"quick:pass", "stuck:hang"}, "hang() { sleep 30; }\n")}, + {"mtest.toml", "[programs.t]\ncommand = [\"bash\", \"t.sh\"]\ntimeout = " + "\"1s\"\ntimeout_record = false\n"}}); + auto j = json_of(mtest(root, {"run", "--json"})); + expect(eq(j["counts"].value("timeout", 0), 1)) << j.dump(); + expect(eq(j["counts"].value("pass", 0), 1)) << j.dump(); + expect(contains(j["failing"][0].value("why", ""), "asked to end")) << j.dump(); + auto e = json_of(mtest(root, {"explain", "--json"})); + auto t = e["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << e.dump(); + expect(!t.contains("seen") && !t.contains("processes")) << t.dump(); + }; "a program that does not start is the machine's failure, not the test's"_test[kind::e2e] = [] { auto root = project({{"x.txt", ""}}); auto r = mtest(root, {"run", "--json", "./no-such-program"}); diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index ab285b1..8697a1f 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.9.0"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.10.0"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations @@ -84,7 +84,7 @@ isolation holds for everything inside; the others are the innermost one stated. | `clause{"P3.1"}` | a specification clause the case covers | forms the conformance matrix | | `resource{"serial:db"}`, `resource{"machine"}` | what it holds exclusively | never runs two holders of one `serial:` group, nor a `machine` case beside another, at once | | `artifact{"file:data/x.txt"}` | what it reads | selects it when that changes | -| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time | +| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time; `mtest` first records its processes and asks it to end ([docs/commands.md](../../docs/commands.md) "Timeouts") | | `isolate{}` | that it runs in a process of its own | starts one for it | | `retries{2}` | that a failure may be retried | runs it again, alone; one that then passes is `flaky` | | `diagnose{.env = {{"APP_LOG", "debug"}}}` | how to look at a failure | runs a failing case once more with that environment and keeps the output (`mtest explain`) | diff --git a/lib/mcpp.toml b/lib/mcpp.toml index a8d47ea..2a939ec 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.9.0" +version = "0.10.0" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index bb8ea36..e27fba2 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -338,7 +338,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}, // P4.3: a selection's `!` and `?` lines are understood. {"selection", {"except", "optional"}}, diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 13447df..58fbf56 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.9.0" +version = "0.10.0" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/spec/protocol.md b/spec/protocol.md index ac1799c..dce35c4 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -32,6 +32,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since | B2, runner side | starts programs of any language, by the protocol or by their exit status (P1.1); a program judged by its exit status states the status that skips (`skip_exit`) | 0.1.0; `skip_exit` 0.6.1 | | B4 | keeps every run in the store (P12), and reads the runs a host (`mcpp test`) keeps | 0.1.0; the store's form specified 0.5.0 | | O5 | schedules cases by their resources and history, by pools since 0.5.0; ends a case at its `timeout`, runs `retries` and `diagnose` since 0.3.0 | 0.1.0 | +| O5, the end at a timeout | records the processes of the program a timeout ends, asks it to end, and ends what still runs after its `grace` ([docs/commands.md](../docs/commands.md) "Timeouts") | 0.10.0 | | O6 | serves run fixtures once per run (P11.2, P11.3) | 0.5.0 | | O7 | measures a program that reports no use from a process of its own (P10.4) | 0.5.0 | | P3.3, places | skips a listed case outside its `only` and `except`, and one its program lists skipped with a `reason`, without starting it; takes the program's answers for the capabilities it names (no `[capabilities]` entry needed) and records them in the run; `mtest ci coverage` requires a case to pass on each of its places that a collected run covered | 0.8.0 |