From fd1e223ba0b66d164bb628b64276975f60a6098e Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 19 Nov 2026 14:48:52 +0900 Subject: [PATCH 1/3] A timeout records the program's processes, asks it to end, and ends what still runs after its grace At a program's timeout the engine records the processes of its unit (on Linux from /proc: the program, what it started and the rest of its process group, each with its state, the kernel function it sleeps in, its threads and what SIGTERM does to it; elsewhere the program alone), requests its end through openkal, waits at most its grace, records what still runs, and ends the unit. `mtest measure` passes the request on to the program it starts. `grace` and `timeout_record` are keys of [programs.*] and [discover.*] beside `timeout`; `mtest run --grace` beside `--timeout`. The record is stated in the case's reason, the terminal summary, `mtest explain`, the store and the Markdown report. --- docs/commands.md | 31 ++++- docs/config.md | 29 +++-- engine/src/commands.cpp | 10 ++ engine/src/config.cpp | 13 +- engine/src/exec.cpp | 173 +++++++++++++++++++++++++-- engine/src/model.cpp | 82 +++++++++++++ engine/src/model.cppm | 49 +++++++- engine/src/output.cpp | 92 +++++++++++++++ engine/src/output.cppm | 8 ++ engine/src/processes.cpp | 241 ++++++++++++++++++++++++++++++++++++++ engine/src/processes.cppm | 23 ++++ engine/src/report.cpp | 28 ++++- engine/tests/engine.cpp | 81 +++++++++++++ lib/docs/writing-tests.md | 2 +- spec/protocol.md | 1 + 15 files changed, 828 insertions(+), 35 deletions(-) create mode 100644 engine/src/processes.cpp create mode 100644 engine/src/processes.cppm diff --git a/docs/commands.md b/docs/commands.md index 1d3d23c..13231d8 100644 --- a/docs/commands.md +++ b/docs/commands.md @@ -36,7 +36,7 @@ nothing; a built program's own listing is the authority (`manifest/src/manifest. | Command | Does | |---|---| -| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | +| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--grace 2s` (below), `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) | | `run --resume RUN` | the selection, less the cases RUN judged: their results are carried into the new run, so an interrupted long run (a bench over a matrix) ends as one whole run; a case RUN could not start runs again | | `rerun` | the cases that failed in the last run | | `smoke` | the `smoke` set | @@ -49,6 +49,35 @@ A program runs by the protocol when it carries the marker or declares its level; status is its one result. Cases tagged `flaky`, and quarantined ones, are retried alone up to `--retries`; one that then passes is `flaky`. A quarantined case's failure does not fail the run. +### Timeouts + +A program still running at its timeout (`timeout`, `--timeout`; a case's own `timeout` where a +process runs that case alone) is first asked to end and then ended, and its processes are recorded +before the request, so that a hang leaves the state it hung in: + +1. **The record.** On Linux, from `/proc`: the program, every process it started, and every other + process of its unit (the process group it leads, which a process it started keeps after the + program has ended). Each process states its parent, its group, its command, its state, the kernel + function it sleeps in, its processor time, what the request to end does to it (`ends`, `caught`, + `ignored`), each of its threads with its state and its kernel function, and, where the system lets + it be read, its kernel stack. Where the system lists no other process to the engine (macOS, + Windows, a `/proc` the engine cannot read), the record holds the program alone, with its + processor time, and says so. +2. **The request.** openkal's request to end a program: `SIGTERM` on Linux and macOS, which a + program may handle or ignore; on Windows the program is ended outright. The request reaches the + program the engine started; `mtest measure`, which starts a program the engine measures, passes + it on. What the program started receives it from the program, or not at all. +3. **The grace.** The engine waits for the program at most its `grace` (a key of `[programs.*]` and + `[discover.*]`, [config.md](config.md); `--grace`; 2 s by default). +4. **The end.** What still runs in the program's unit is recorded again and ended. + +The case's reason says how the program ended: `asked to end, it ended within its grace of 2s`, or +`asked to end, it was still running after its grace of 2s, and was ended`. The record is printed +under the case, kept in the run's `programs.ndjson` as the program's `timeout` (`limit_ms`, +`grace_ms`, `ended`, `seen`, `processes`, `remaining`), given by `mtest explain`, and shown below the +failures of the Markdown report. A program that states `timeout_record = false` is asked to end and +ended in the same way, with no record. + ## Asking | Command | Answers | diff --git a/docs/config.md b/docs/config.md index 72460d4..0e3e7d1 100644 --- a/docs/config.md +++ b/docs/config.md @@ -13,19 +13,22 @@ jobs = 0 # 0: the machine's processors # A program of any language. `level` states the protocol's level; without it the marker decides (P2.3). [programs.e2e] -command = ["python3", "tests/e2e.py"] -cwd = "." -env = { MODE = "ci" } -sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection) -resources = ["serial:db"] # what it holds alone (P7.1) -tags = ["slow"] -timeout = "10m" -isolation = "process" # or "case": a process per case -kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit) -split = true # whether [run] split may divide its cases among processes -skip_exit = 77 # a program judged by its exit status: the status that says its case did not - # apply here (skip, with the last `SKIP...` line of its output as the reason); - # unset: every status but 0 fails +command = ["python3", "tests/e2e.py"] +cwd = "." +env = { MODE = "ci" } +sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection) +resources = ["serial:db"] # what it holds alone (P7.1) +tags = ["slow"] +timeout = "10m" +grace = "2s" # at its timeout it is asked to end; what still runs this much later is + # ended (commands.md "Timeouts"); 2s by default +timeout_record = true # its processes are recorded at its timeout and after the grace; false: no record +isolation = "process" # or "case": a process per case +kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit) +split = true # whether [run] split may divide its cases among processes +skip_exit = 77 # a program judged by its exit status: the status that says its case did not + # apply here (skip, with the last `SKIP...` line of its output as the reason); + # unset: every status but 0 fails [discover] globs = ["build/tests/*"] # every executable that matches is a program diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index 06b8918..66e586c 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -183,6 +183,10 @@ std::expected, answer> programs_of(const context& ctx, if (auto t = text_of(p, "timeout"); !t.empty()) if (auto d = parse_duration(t)) s.timeout = std::chrono::duration_cast(*d); + for (auto& s : *programs) + if (auto t = text_of(p, "grace"); !t.empty()) + if (auto d = parse_duration(t)) + s.grace = *d; if (p.value("isolate", false)) for (auto& s : *programs) s.isolation = "case"; @@ -825,6 +829,8 @@ answer cmd_explain(const std::filesystem::path& cwd, const json& p) { } if (c.contains("output") && !c["output"]["err_tail"].get().empty()) text += " error output (end):\n" + c["output"]["err_tail"].get() + "\n"; + if (c.contains("timeout")) + text += timeout_text(timeout_info_from(c["timeout"]), " ", 400); if (c.contains("diagnosis")) text += " diagnosis: " + c["diagnosis"].get() + "\n"; text += " again: " + c["again"].get() + "\n"; @@ -1799,6 +1805,10 @@ json catalog() { {"retries", {{"type", "integer"}, {"description", "runs again of a failed case"}}}, {"isolate", {{"type", "boolean"}, {"description", "each case in its own process"}}}, {"timeout", {{"type", "string"}, {"description", "a duration for each program"}}}, + {"grace", + {{"type", "string"}, + {"description", "how long a program asked to end at its timeout has before it " + "is ended"}}}, {"env", {{"type", "object"}, {"description", "NAME=VALUE for the programs"}}}, {"run_under", {{"type", "array"}, diff --git a/engine/src/config.cpp b/engine/src/config.cpp index 2f94d0b..8f74680 100644 --- a/engine/src/config.cpp +++ b/engine/src/config.cpp @@ -43,12 +43,9 @@ void known(const toml::table& t, std::string_view where, } // The keys a program states, in `[programs.*]` and `[discover.*]` alike. -constexpr std::initializer_list kProgramKeys = {"command", "cwd", - "env", "level", - "resources", "sources", - "tags", "timeout", - "isolation", "kind", - "split", "skip_exit"}; +constexpr std::initializer_list kProgramKeys = + {"command", "cwd", "env", "level", "resources", "sources", "tags", + "timeout", "isolation", "kind", "split", "skip_exit", "grace", "timeout_record"}; void read_program(const toml::table& t, program_spec& s) { if (auto v = t["cwd"].value()) @@ -67,6 +64,10 @@ void read_program(const toml::table& t, program_spec& s) { s.tags = strings(t["tags"]); if (auto d = duration_of(t["timeout"])) s.timeout = std::chrono::duration_cast(*d); + if (auto d = duration_of(t["grace"])) + s.grace = *d; + if (auto v = t["timeout_record"].value()) + s.timeout_record = *v; if (auto v = t["isolation"].value()) s.isolation = *v; if (auto v = t["split"].value()) diff --git a/engine/src/exec.cpp b/engine/src/exec.cpp index caac196..09a0e60 100644 --- a/engine/src/exec.cpp +++ b/engine/src/exec.cpp @@ -2,6 +2,7 @@ module mcpplibs.mtest.engine.exec; import std; import mcpplibs.os; +import mcpplibs.mtest.engine.processes; namespace mcpplibs::mtest::engine { @@ -71,6 +72,141 @@ std::filesystem::path resolve_program(const std::string& name) { return p; } +// A RUN WITHIN ITS TIMEOUT (0.10.0). mcpplibs.os's bounded run asks a program past its limit to end +// and ends its unit after a grace, with nothing between; the engine composes the same parts with a +// record between them, so that a hang leaves the state it hung in. The outputs, their limits and +// their drain are that run's. + +// One output, read by a thread of its own in slices, so that the run can stop reading once the +// program has ended and its outputs have had their drain. +struct capture { + std::mutex lock; + std::string data; + std::size_t limit = 0; +}; +struct readers { + std::mutex lock; + std::condition_variable changed; + int open = 2; + std::atomic stop{false}; +}; +constexpr std::chrono::milliseconds read_slice{50}; + +void collect(std::shared_ptr p, std::shared_ptr into, + std::shared_ptr all, bool output) { + while (!all->stop.load(std::memory_order_relaxed)) { + auto chunk = output ? p->read_output_for(read_slice) : p->read_error_for(read_slice); + if (!chunk || (chunk->has_value() && (*chunk)->empty())) + break; // the stream failed, or ended + if (!chunk->has_value()) + continue; + // What passes the limit is read and dropped, so that the program never waits on a full + // pipe. + std::lock_guard g(into->lock); + const auto room = into->limit - std::min(into->limit, into->data.size()); + into->data.append(**chunk, 0, std::min(room, (*chunk)->size())); + } + { + std::lock_guard g(all->lock); + --all->open; + } + all->changed.notify_all(); +} + +struct ended_run { + os::process::exit_status status; + std::string out, err; + std::optional timeout; // past its limit +}; + +// Past its limit: its processes recorded, the program asked to end, and what still runs after its +// grace recorded and ended with its unit. The first record precedes the request, which changes or +// ends what it reaches. +timeout_info end_past_limit(os::process::process& p, const program_spec& spec) { + timeout_info t; + t.limit_ms = std::chrono::duration(spec.timeout).count(); + t.grace_ms = double(spec.grace.count()); + if (spec.timeout_record) { + auto seen = processes_of(p.id()); + t.seen = std::move(seen.seen); + t.processes = std::move(seen.processes); + } + p.terminate(); + const auto within = p.wait_for(spec.grace); + t.ended = within && *within ? "requested" : "terminated"; + if (spec.timeout_record) { + // What still runs: a process that has ended and is not yet collected runs no more. + t.remaining = processes_of(p.id()).processes; + std::erase_if(t.remaining, [](const process_info& x) { + return x.state == "zombie" || x.state == "dead"; + }); + } + p.end_unit(); + return t; +} + +os::result run_within(os::process::spawn_options o, const program_spec& spec) { + const os::process::bounds bounded; + o.out = os::process::stdio::pipe; + o.err = os::process::stdio::pipe; + o.own_unit = true; + auto spawned = os::process::process::spawn(o); + if (!spawned) + return std::unexpected(spawned.error()); + auto p = std::make_shared(std::move(*spawned)); + auto out = std::make_shared(); + out->limit = bounded.out_limit; + auto err = std::make_shared(); + err->limit = bounded.err_limit; + auto all = std::make_shared(); + std::thread(collect, p, out, all, true).detach(); + std::thread(collect, p, err, all, false).detach(); + + ended_run r; + auto waited = p->wait_for(std::chrono::duration_cast(spec.timeout)); + if (!waited) + return std::unexpected(waited.error()); + if (waited->has_value()) { + r.status = **waited; + } else { + r.timeout = end_past_limit(*p, spec); + auto ended = p->wait(); + r.status = ended ? *ended : os::process::exit_status{-1, true}; + } + { + std::unique_lock g(all->lock); + if (!all->changed.wait_for(g, bounded.drain, [&] { return all->open == 0; })) { + // The program has ended, and something it started holds its outputs open. + p->end_unit(); + all->stop.store(true, std::memory_order_relaxed); + all->changed.wait_for(g, read_slice * 4, [&] { return all->open == 0; }); + } + } + { + std::lock_guard g(out->lock); + r.out = out->data; + } + { + std::lock_guard g(err->lock); + r.err = err->data; + } + return r; +} + +std::string duration_words(double ms) { + return std::fmod(ms, 1000) == 0 ? std::format("{}s", ms / 1000) : std::format("{}ms", ms); +} + +// A case's reason for a timeout, `what`, and how the program was then ended. +std::string timeout_reason(std::string_view what, const timeout_info& t) { + const auto grace = duration_words(t.grace_ms); + return t.ended == "requested" + ? std::format("{}; asked to end, it ended within its grace of {}", what, grace) + : std::format("{}; asked to end, it was still running after its grace of {}, " + "and was ended", + what, grace); +} + } // namespace int detect_level(const program_spec& spec) { @@ -245,6 +381,9 @@ list_program(const program_spec& spec, const std::filesystem::path& work, int measure(const std::filesystem::path& out, std::vector argv) { if (argv.empty()) return 2; + // At a timeout the engine asks this process to end, and the test is the program it starts: the + // request is taken over before the start and passed on, and the program ends as it chooses. + const bool passes_on = os::interrupt::available(); os::process::spawn_options o; o.program = resolve_program(argv.front()); o.args.assign(argv.begin() + 1, argv.end()); @@ -266,8 +405,19 @@ int measure(const std::filesystem::path& out, std::vector argv) { // The account of a child is the kernel's once the child is reaped, which closing its // handle does. auto child = std::move(*p); - if (auto s = child.wait()) - st = *s; + for (bool passed = false;;) { + auto s = child.wait_for(std::chrono::milliseconds{50}); + if (!s) + break; + if (s->has_value()) { + st = **s; + break; + } + if (passes_on && !passed && os::interrupt::requested()) { + child.terminate(); + passed = true; + } + } } json j = {{"source", "engine"}}; if (auto u = os::process::resources(os::process::of::children)) { @@ -370,10 +520,8 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { o.env.emplace_back("MTEST_MONITOR_SAMPLE", std::to_string(request.sample_ms)); if (request.seed) o.env.emplace_back("MTEST_SEED", std::to_string(*request.seed)); - os::process::bounds limits; - limits.hard = std::chrono::duration_cast(spec.timeout); const auto began = std::chrono::steady_clock::now(); - auto done = os::process::run(std::move(o), limits); + auto done = run_within(std::move(o), spec); p.ms = std::chrono::duration(std::chrono::steady_clock::now() - began).count(); std::filesystem::remove_all(request.work / "tmp", ec); @@ -390,7 +538,10 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { return out; } p.exit = done->status.shell_code(); - p.timed_out = done->timed_out; + p.timed_out = done->timeout.has_value(); + p.timeout = std::move(done->timeout); + // What a case that the timeout ended says of it. + const auto timed_out = [&](std::string_view what) { return timeout_reason(what, *p.timeout); }; p.out_tail = tail(done->out); p.err_tail = tail(done->err); out.out = std::move(done->out); @@ -430,7 +581,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { if (c.status == "skip") c.reason = skip_reason(out.out, out.err, p.exit); else if (c.status != "pass") - c.reason = p.timed_out ? std::format("still running after {}", spec.timeout) + c.reason = p.timed_out ? timed_out(std::format("still running after {}", spec.timeout)) : std::format("exit {}", p.exit); out.cases.push_back(std::move(c)); } else { @@ -448,7 +599,9 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { if (c.status == "skip") c.reason = skip_reason(out.out, out.err, p.exit); else if (c.status != "pass") - c.reason = std::format("exit {}", p.exit); + c.reason = p.timed_out + ? timed_out(std::format("still running after {}", spec.timeout)) + : std::format("exit {}", p.exit); out.cases.push_back(std::move(c)); } else { for (auto& c : r.cases) { @@ -456,7 +609,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { // P5.4: begun and never ended -- the program ended (or was ended) first. c.status = p.timed_out ? "timeout" : "fail"; c.reason = p.timed_out - ? "the program was still running at its timeout" + ? timed_out("the program was still running at its timeout") : std::format("the program ended (exit {}) before the case did", p.exit); } @@ -474,7 +627,7 @@ run_outcome run_program(const program_spec& spec, const run_request& request) { c.id = spec.name; c.ms = p.ms; c.status = p.timed_out ? "timeout" : "fail"; - c.reason = p.timed_out ? "the program was still running at its timeout" + c.reason = p.timed_out ? timed_out("the program was still running at its timeout") : std::format("exit {} with no failing case{}", p.exit, r.done ? "" : ", and no `done' event"); out.cases.push_back(std::move(c)); diff --git a/engine/src/model.cpp b/engine/src/model.cpp index 1ad9ccc..c863564 100644 --- a/engine/src/model.cpp +++ b/engine/src/model.cpp @@ -127,11 +127,91 @@ json to_json(const case_result& c) { j["started_ms"] = c.started_ms; return j; } + +namespace { + +// A process as the timeout recorded it; an empty field is one the system did not tell. +json to_json(const process_info& p) { + json j = {{"id", p.id}, {"parent", p.parent}, {"group", p.group}, {"name", p.name}, + {"command", p.command}, {"state", p.state}, {"cpu_ms", p.cpu_ms}}; + if (!p.waiting.empty()) + j["waiting"] = p.waiting; + if (!p.request.empty()) + j["request"] = p.request; + if (!p.threads.empty()) { + j["threads"] = json::array(); + for (const auto& t : p.threads) { + json one = {{"id", t.id}, {"name", t.name}, {"state", t.state}, {"cpu_ms", t.cpu_ms}}; + if (!t.waiting.empty()) + one["waiting"] = t.waiting; + j["threads"].push_back(std::move(one)); + } + } + if (!p.stack.empty()) + j["stack"] = p.stack; + return j; +} + +std::vector processes_from(const json& j, const char* key) { + std::vector out; + auto it = j.find(key); + if (it == j.end() || !it->is_array()) + return out; + for (const auto& e : *it) { + process_info p; + p.id = e.value("id", std::uint64_t{0}); + p.parent = e.value("parent", std::uint64_t{0}); + p.group = e.value("group", std::uint64_t{0}); + p.name = str(e, "name"); + p.command = strings(e, "command"); + p.state = str(e, "state"); + p.waiting = str(e, "waiting"); + p.request = str(e, "request"); + p.cpu_ms = e.value("cpu_ms", 0.0); + p.stack = str(e, "stack"); + if (auto t = e.find("threads"); t != e.end() && t->is_array()) + for (const auto& x : *t) + p.threads.push_back({x.value("id", std::uint64_t{0}), str(x, "name"), + str(x, "state"), str(x, "waiting"), x.value("cpu_ms", 0.0)}); + out.push_back(std::move(p)); + } + return out; +} + +} // namespace + +json to_json(const timeout_info& t) { + json j = {{"limit_ms", t.limit_ms}, {"grace_ms", t.grace_ms}, {"ended", t.ended}}; + if (!t.seen.empty()) { + j["seen"] = t.seen; + j["processes"] = json::array(); + for (const auto& p : t.processes) + j["processes"].push_back(to_json(p)); + j["remaining"] = json::array(); + for (const auto& p : t.remaining) + j["remaining"].push_back(to_json(p)); + } + return j; +} + +timeout_info timeout_info_from(const json& j) { + timeout_info t; + t.limit_ms = j.value("limit_ms", 0.0); + t.grace_ms = j.value("grace_ms", 0.0); + t.ended = str(j, "ended"); + t.seen = str(j, "seen"); + t.processes = processes_from(j, "processes"); + t.remaining = processes_from(j, "remaining"); + return t; +} + json to_json(const program_result& p) { json j = {{"name", p.name}, {"argv", p.argv}, {"cwd", p.cwd}, {"level", p.level}, {"status", p.status}, {"exit", p.exit}, {"ms", p.ms}}; if (p.timed_out) j["timed_out"] = true; + if (p.timeout) + j["timeout"] = to_json(*p.timeout); if (!p.out_tail.empty()) j["out_tail"] = p.out_tail; if (!p.err_tail.empty()) @@ -276,6 +356,8 @@ program_result program_result_from(const json& j) { p.exit = j.value("exit", 0); p.ms = j.value("ms", 0.0); p.timed_out = j.value("timed_out", false); + if (auto t = j.find("timeout"); t != j.end() && t->is_object()) + p.timeout = timeout_info_from(*t); p.out_tail = str(j, "out_tail"); p.err_tail = str(j, "err_tail"); p.error = str(j, "error"); diff --git a/engine/src/model.cppm b/engine/src/model.cppm index a030309..1318e16 100644 --- a/engine/src/model.cppm +++ b/engine/src/model.cppm @@ -20,6 +20,11 @@ struct program_spec { std::vector sources; // paths whose change affects it (affected selection) std::vector tags; std::chrono::seconds timeout{300}; + // At its timeout it is asked to end, and what still runs `grace` later is ended (0.10.0); the + // default is the grace of mcpplibs.os's bounded run, which the engine gave through 0.9.0. With + // `timeout_record`, its processes are recorded at the timeout and after the grace. + std::chrono::milliseconds grace{std::chrono::seconds{2}}; + bool timeout_record = true; std::string isolation = "process"; // "process" or "case" std::string origin = "config"; // "given" (the command line), "config", "discover", "host" // The kind of an L0 program's one case and of a case that states none: stated in mtest.toml, @@ -118,6 +123,41 @@ struct case_result { double started_ms = -1; }; +// A thread of a process, as the engine saw it at a timeout (0.10.0). +struct thread_info { + std::uint64_t id = 0; + std::string name, state; + std::string waiting; // the kernel function it sleeps in (Linux: wchan); empty: not told + double cpu_ms = 0; +}; + +// A process of a program's unit, as the engine saw it at a timeout (0.10.0). What the system does +// not tell, or does not let the engine read, is empty. +struct process_info { + std::uint64_t id = 0, parent = 0, group = 0; + std::string name; + std::vector command; + std::string state; // running, sleeping, waiting (uninterruptible), stopped, zombie, ... + std::string waiting; // as its first thread's + std::string request; // what the request to end does to it: ends, caught, ignored + double cpu_ms = 0; + std::vector threads; // when it has more than one + std::string stack; // the kernel's stack of its first thread, where readable +}; + +// How a program past its timeout was ended (0.10.0): its processes were recorded, it was asked to +// end, and what still ran after its grace was recorded and ended. +struct timeout_info { + double limit_ms = 0, grace_ms = 0; + std::string ended; // "requested": it ended within its grace; "terminated": ended after it + // "tree": the program, the processes it started and the rest of its unit (Linux); "program": + // the program alone, where the system lists no other process to the engine; empty: not recorded + // (`timeout_record = false`). + std::string seen; + std::vector processes; // at the timeout, before the request + std::vector remaining; // after the grace, before the unit was ended +}; + struct program_result { std::string name; std::vector argv; @@ -127,9 +167,10 @@ struct program_result { int exit = 0; double ms = 0; bool timed_out = false; - std::string out_tail, err_tail; // the last part of each output - std::string error; // why it did not run, for an infra status - std::string origin = "config"; // as program_spec's: how the program is named again + std::optional timeout; // how it was ended at its timeout, and its processes + std::string out_tail, err_tail; // the last part of each output + std::string error; // why it did not run, for an infra status + std::string origin = "config"; // as program_spec's: how the program is named again // When it started, in milliseconds after the run began, and what it held: the run's timeline. double started_ms = 0; std::vector resources; @@ -179,6 +220,7 @@ std::vector bounds_from(const json& j); json to_json(const failure_info& f); json to_json(const metric_info& m); json to_json(const case_result& c); +json to_json(const timeout_info& t); json to_json(const program_result& p); json to_json(const run_record& r); // without programs and cases (run.json) json to_json(const fixture_run& f); @@ -186,6 +228,7 @@ fixture_run fixture_run_from(const json& j); case_info case_info_from(const json& j); case_result case_result_from(const json& j); +timeout_info timeout_info_from(const json& j); program_result program_result_from(const json& j); run_record run_record_from(const json& j); diff --git a/engine/src/output.cpp b/engine/src/output.cpp index 556080d..3e86630 100644 --- a/engine/src/output.cpp +++ b/engine/src/output.cpp @@ -43,8 +43,91 @@ std::string shell_quote(std::string_view t) { return out + "'"; } +// `sleeping in do_wait, ignores the request to end, 3 ms of processor time`. +std::string facts_of(std::string_view state, std::string_view waiting, std::string_view request, + double cpu_ms) { + std::string out; + if (!state.empty()) + out = waiting.empty() ? std::string(state) : std::format("{} in {}", state, waiting); + if (request == "ignored" || request == "caught") + out += std::format("{}{} the request to end", out.empty() ? "" : ", ", + request == "ignored" ? "ignores" : "handles"); + return out + std::format("{}{:.0f} ms of processor time", out.empty() ? "" : ", ", cpu_ms); +} + +std::string process_line(const process_info& p) { + std::string command; + for (const auto& word : p.command) + command += (command.empty() ? "" : " ") + word; + if (command.empty()) + command = p.name; + // A compiler's command line is the record's, not the terminal's. + constexpr std::size_t shown = 160; + if (command.size() > shown) + command = command.substr(0, shown) + " ..."; + return std::format("{}{}{}: {}", p.id, command.empty() ? "" : " ", command, + facts_of(p.state, p.waiting, p.request, p.cpu_ms)); +} + +// Each process under the one that started it, with its threads and its kernel stack. +void add_processes(std::vector& lines, const std::vector& processes, + const std::string& indent) { + std::map depth; + for (const auto& p : processes) { + const auto parent = depth.find(p.parent); + const auto d = parent == depth.end() ? std::size_t{0} : parent->second + 1; + depth[p.id] = d; + const auto at = indent + std::string(2 * d, ' '); + lines.push_back(at + process_line(p)); + for (const auto& t : p.threads) + lines.push_back(std::format("{} thread {} {}: {}", at, t.id, t.name, + facts_of(t.state, t.waiting, {}, t.cpu_ms))); + if (p.stack.empty()) + continue; + for (auto part : std::views::split(std::string_view(p.stack), '\n')) + lines.push_back(std::format("{} {}", at, std::string_view(part.begin(), part.end()))); + } +} + } // namespace +const program_result* timed_out_program(const run_record& r, std::string_view program, + std::string_view id) { + const program_result* any = nullptr; + for (const auto& p : r.programs) { + if (p.name != program || !p.timeout) + continue; + if (p.select.empty() || std::ranges::find(p.select, id) != p.select.end()) + return &p; + any = &p; + } + return any; +} + +std::string timeout_text(const timeout_info& t, std::string_view indent, std::size_t most) { + if (t.seen.empty()) + return {}; + const std::string at(indent); + std::vector lines; + lines.push_back(at + (t.seen == "tree" ? "processes at the timeout:" + : "the program at the timeout (this system lists no " + "other process to mtest):")); + add_processes(lines, t.processes, at + " "); + if (!t.remaining.empty()) { + lines.push_back(at + (t.ended == "requested" + ? "left running by the program, and ended with its unit:" + : "still running after the grace, and ended:")); + add_processes(lines, t.remaining, at + " "); + } + std::string out; + for (std::size_t i = 0; i < lines.size() && i < most; ++i) + out += lines[i] + "\n"; + if (lines.size() > most) + out += std::format("{}... {} more lines: the whole record is in `mtest explain --json`\n", + at, lines.size() - most); + return out; +} + std::string reproduce_command(std::string_view program, std::string_view id) { return std::format("mtest run --program {} --case {}", shell_quote(program), shell_quote(id)); } @@ -76,6 +159,8 @@ std::string summary_text(const run_record& r, bool verbose) { auto it = r.counts.find(k); return it == r.counts.end() ? 0 : it->second; }; + // A program's processes at its timeout, under the first of its cases the timeout ended. + std::set recorded; for (const auto& c : r.cases) { if (!failing(c.status) && c.status != "infra" && !verbose) continue; @@ -90,6 +175,9 @@ std::string summary_text(const run_record& r, bool verbose) { if (!f.expected.empty() || !f.actual.empty()) out += std::format(" expected {}\n actual {}\n", f.expected, f.actual); } + if (const auto* p = c.status == "timeout" ? timed_out_program(r, c.program, c.id) : nullptr; + p && recorded.insert(p).second) + out += timeout_text(*p->timeout, " ", 40); if (failing(c.status)) out += std::format(" again: {}\n", again(c)); } @@ -221,6 +309,10 @@ std::expected explain(const store& s, const run_record& r, {"cwd", p.cwd}, {"level", p.level}, {"exit", p.exit}}; + // How its timeout ended it, and the processes recorded then. + if (const auto* p = + c->status == "timeout" ? timed_out_program(r, c->program, c->id) : nullptr) + item["timeout"] = to_json(*p->timeout); // The case's recent history: a new failure, one that keeps failing, or one that changes. json recent = json::array(); auto all = runs(s); diff --git a/engine/src/output.cppm b/engine/src/output.cppm index 5f4f7f8..d52411b 100644 --- a/engine/src/output.cppm +++ b/engine/src/output.cppm @@ -9,6 +9,14 @@ export import mcpplibs.mtest.engine.store; export namespace mcpplibs::mtest::engine { std::string summary_text(const run_record& r, bool verbose = false); + +// The program of a run whose timeout ended the case `id` (0.10.0): the process that ran it, else +// another of that program's that timed out; null when none did. +const program_result* timed_out_program(const run_record& r, std::string_view program, + std::string_view id); +// The processes a timeout recorded, as lines a person reads, each begun by `indent`: at most `most` +// of them, then how many more the record holds. Empty when no record was taken. +std::string timeout_text(const timeout_info& t, std::string_view indent, std::size_t most); std::string junit_xml(const run_record& r); std::string html_page(const run_record& r); diff --git a/engine/src/processes.cpp b/engine/src/processes.cpp new file mode 100644 index 0000000..0724c07 --- /dev/null +++ b/engine/src/processes.cpp @@ -0,0 +1,241 @@ +module mcpplibs.mtest.engine.processes; + +import std; +import mcpplibs.os; + +namespace mcpplibs::mtest::engine { + +namespace os = mcpplibs::os; + +namespace { + +std::optional number(std::string_view text, int base = 10) { + std::uint64_t v = 0; + const auto [end, ec] = std::from_chars(text.data(), text.data() + text.size(), v, base); + if (ec != std::errc{} || end != text.data() + text.size()) + return std::nullopt; + return v; +} + +// A file of /proc without its trailing line ends; empty where it cannot be read, as the kernel's +// stack of a process is to all but the superuser. +std::string proc_text(const std::filesystem::path& file) { + auto text = os::fs::read(file); + if (!text) + return {}; + while (!text->empty() && (text->back() == '\n' || text->back() == ' ')) + text->pop_back(); + return *text; +} + +// /proc//stat: ` () ...`, user and system time the 12th and +// 13th fields after the name, in ticks of USER_HZ (100), as mcpplibs.os reads a thread's. +struct stat_line { + std::uint64_t id = 0, parent = 0, group = 0; + std::string name; + char state = '?'; + double cpu_ms = 0; +}; + +std::optional read_stat(const std::filesystem::path& file) { + const auto text = proc_text(file); + const auto open = text.find('('); + const auto close = text.rfind(')'); + if (open == std::string::npos || close == std::string::npos || close < open || + close + 2 >= text.size() || open < 2) + return std::nullopt; + std::vector fields; + for (auto part : std::views::split(std::string_view(text).substr(close + 2), ' ')) + fields.emplace_back(part.begin(), part.end()); + const auto id = number(std::string_view(text).substr(0, open - 1)); + if (fields.size() < 13 || fields[0].empty() || !id) + return std::nullopt; + stat_line s; + s.id = *id; + s.name = text.substr(open + 1, close - open - 1); + s.state = fields[0].front(); + s.parent = number(fields[1]).value_or(0); + s.group = number(fields[2]).value_or(0); + s.cpu_ms = double(number(fields[11]).value_or(0) + number(fields[12]).value_or(0)) * 10; + return s; +} + +// The kernel's letters for a state (proc(5)), as a person reads them. +std::string state_word(char state) { + switch (state) { + case 'R': + return "running"; + case 'S': + return "sleeping"; + case 'D': + return "waiting (uninterruptible)"; + case 'T': + return "stopped"; + case 't': + return "traced"; + case 'Z': + return "zombie"; + case 'X': + return "dead"; + case 'I': + return "idle"; + case 'P': + return "parked"; + default: + return std::string(1, state); + } +} + +// The kernel function a thread sleeps in; `0` while it runs, or where the kernel does not say. +std::string waiting_in(const std::filesystem::path& dir) { + auto w = proc_text(dir / "wchan"); + return w == "0" ? std::string{} : w; +} + +// What SIGTERM, the request to end on Linux (openkal's kal_process_terminate), does to a process: +// its disposition, which /proc//status states as masks of the signals ignored and caught. +std::string request_of(const std::filesystem::path& dir) { + const auto status = proc_text(dir / "status"); + const auto mask = [&](std::string_view key) -> std::uint64_t { + for (auto part : std::views::split(std::string_view(status), '\n')) { + std::string_view line(part.begin(), part.end()); + if (!line.starts_with(key)) + continue; + line.remove_prefix(key.size()); + while (!line.empty() && (line.front() == ':' || line.front() == '\t')) + line.remove_prefix(1); + return number(line, 16).value_or(0); + } + return 0; + }; + constexpr std::uint64_t term = std::uint64_t{1} << (15 - 1); + if (status.empty()) + return {}; + return mask("SigIgn") & term ? "ignored" : mask("SigCgt") & term ? "caught" : "ends"; +} + +// Its arguments, NUL-separated; a command past 4 KiB keeps its first arguments and says so, so +// that a compiler's line of thousands of words does not become the record. +std::vector command_of(const std::filesystem::path& dir) { + std::vector out; + const auto text = os::fs::read(dir / "cmdline"); + if (!text) + return out; + constexpr std::size_t keep = 4096; + std::size_t kept = 0; + for (auto part : std::views::split(std::string_view(*text), '\0')) { + std::string_view word(part.begin(), part.end()); + if (kept + word.size() > keep) { + out.push_back(std::format("... ({} bytes in all)", text->size())); + break; + } + kept += word.size(); + out.emplace_back(word); + } + while (!out.empty() && out.back().empty()) + out.pop_back(); + return out; +} + +std::vector threads_of(const std::filesystem::path& dir) { + std::vector out; + std::error_code ec; + for (std::filesystem::directory_iterator it(dir / "task", ec), end; !ec && it != end; + it.increment(ec)) { + if (auto s = read_stat(it->path() / "stat")) + out.push_back( + {s->id, s->name, state_word(s->state), waiting_in(it->path()), s->cpu_ms}); + } + std::ranges::sort(out, {}, &thread_info::id); + // One thread is the process itself, which its own fields already state. + if (out.size() < 2) + out.clear(); + return out; +} + +process_info described(const stat_line& s) { + const std::filesystem::path dir = std::format("/proc/{}", s.id); + process_info p; + p.id = s.id; + p.parent = s.parent; + p.group = s.group; + p.name = s.name; + p.state = state_word(s.state); + p.cpu_ms = s.cpu_ms; + p.command = command_of(dir); + p.waiting = waiting_in(dir); + p.request = request_of(dir); + p.threads = threads_of(dir); + p.stack = proc_text(dir / "stack"); + return p; +} + +// Every process /proc lists, by its stat line. +std::vector process_table() { + std::vector out; + std::error_code ec; + for (std::filesystem::directory_iterator it("/proc", ec), end; !ec && it != end; + it.increment(ec)) { + if (!number(os::fs::utf8(it->path().filename()))) + continue; + if (auto s = read_stat(it->path() / "stat")) + out.push_back(std::move(*s)); + } + return out; +} + +// The program alone, as mcpplibs.os describes it: its identifier and its processor time. +processes_seen alone(std::uint64_t program) { + processes_seen r{"program", {}}; + auto d = os::process::describe(program); + if (!d || d->ended) + return r; + process_info p; + p.id = program; + p.cpu_ms = double(d->cpu_ns) / 1e6; + r.processes.push_back(std::move(p)); + return r; +} + +} // namespace + +processes_seen processes_of(std::uint64_t program) { + if (os::host() != os::platform::linux || program == 0) + return alone(program); + const auto table = process_table(); + if (table.empty()) + return alone(program); + std::map by_id; + std::multimap children; + for (const auto& s : table) { + by_id[s.id] = &s; + children.emplace(s.parent, s.id); + } + // The unit is the group the program leads: a process it started stays there after the program + // has ended and its children are given to another parent. Each is walked with what descends + // from it, which a process that formed a group of its own does too; the program's own first. + std::vector pending; + for (const auto& s : table) + if (s.group == program && s.id != program) + pending.push_back(s.id); + std::ranges::reverse(pending); + pending.push_back(program); + processes_seen r{"tree", {}}; + std::set taken; + while (!pending.empty()) { + const auto id = pending.back(); + pending.pop_back(); + const auto found = by_id.find(id); + if (found == by_id.end() || !taken.insert(id).second) + continue; + r.processes.push_back(described(*found->second)); + const auto [first, last] = children.equal_range(id); + std::vector under; + for (auto c = first; c != last; ++c) + under.push_back(c->second); + pending.insert(pending.end(), under.rbegin(), under.rend()); + } + return r; +} + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/processes.cppm b/engine/src/processes.cppm new file mode 100644 index 0000000..25084e9 --- /dev/null +++ b/engine/src/processes.cppm @@ -0,0 +1,23 @@ +// The processes of a program's unit as the system shows them (0.10.0): what a timeout records +// before it asks the program to end, and again after the program's grace (docs/commands.md +// "Running"). +export module mcpplibs.mtest.engine.processes; + +import std; +export import mcpplibs.mtest.engine.model; + +export namespace mcpplibs::mtest::engine { + +struct processes_seen { + std::string seen; // "tree" or "program", as timeout_info's + std::vector processes; +}; + +// The program the system identifies by `program`, every process it started, and every other process +// of its unit, the program's subtree first: on Linux from /proc, each with its state, what it waits +// on, what the request to end does to it, and its threads. Where the system lists no other process +// to the engine (macOS, Windows, a /proc it cannot read), the program alone, as mcpplibs.os +// describes it; nothing once the program has ended. +processes_seen processes_of(std::uint64_t program); + +} // namespace mcpplibs::mtest::engine diff --git a/engine/src/report.cpp b/engine/src/report.cpp index c763c5c..de8e957 100644 --- a/engine/src/report.cpp +++ b/engine/src/report.cpp @@ -212,6 +212,30 @@ void failures(std::string& md, const std::vector& cases) { md += "\nRepeat one with `mtest explain --case ''`.\n\n"; } +// Below the failures, the processes each timeout recorded (0.10.0), once per program. +void timeout_records(std::string& md, const std::vector& runs, + const std::vector& cases) { + const std::set in(cases.begin(), cases.end()); + std::set shown; + std::string blocks; + for (const auto& r : runs) + for (const auto& c : r.cases) { + const auto* p = c.status == "timeout" && in.contains(&c) + ? timed_out_program(r, c.program, c.id) + : nullptr; + if (!p || p->timeout->seen.empty() || !shown.insert(p).second) + continue; + blocks += std::format("
{}: " + "{}\n\n```text\n{}```\n\n
\n\n", + case_label(c.program, c.id), + p->timeout->ended == "requested" ? "it ended when asked" + : "it was ended after its grace", + timeout_text(*p->timeout, "", 200)); + } + if (!blocks.empty()) + md += "The processes of each program a timeout ended, as they were then:\n\n" + blocks; +} + void slowest(std::string& md, std::vector cases, int top) { std::ranges::sort(cases, std::greater{}, [](const case_result* c) { return c->ms; }); md += "## Slowest cases\n\n| case | kind | time | status |\n|---|---|---:|---|\n"; @@ -391,8 +415,10 @@ std::string markdown_report(const store& s, const std::vector& runs, summary(md, runs, o); if (wants(o, "kinds")) kinds(md, cases); - if (wants(o, "failures")) + if (wants(o, "failures")) { failures(md, cases); + timeout_records(md, runs, cases); + } if (wants(o, "slowest")) slowest(md, cases, o.top); if (wants(o, "metrics")) diff --git a/engine/tests/engine.cpp b/engine/tests/engine.cpp index 335219c..5d5f53a 100644 --- a/engine/tests/engine.cpp +++ b/engine/tests/engine.cpp @@ -50,6 +50,13 @@ json json_of(const e2e::outcome& r) { return j; } +// Whether a timeout's record holds every process of a program's unit here: Linux lists its +// processes in /proc, where the engine, as this program, reads it. +bool sees_the_tree() { + return mcpplibs::os::host() == mcpplibs::os::platform::linux && + std::filesystem::exists("/proc/self/stat"); +} + // A protocol script: the cases given as `id:status`, each with a function that does what its status // says. std::string script(std::initializer_list cases, std::string_view extra = {}) { @@ -113,6 +120,80 @@ int main() { auto j = json_of(mtest(root, {"run", "--json", "--timeout", "1s", "bash hang.sh"})); expect(eq(j["counts"].value("timeout", 0), 1)); }; + "a timeout records the program's processes, then asks it to end"_test[kind::e2e][tag{"slow"}] = + [] { + // bash ends on the request; the `sleep` it started does not receive it, and is ended + // with the program's unit. A grace that long is not waited for by a program that ends. + auto root = project({{"hang.sh", "sleep 30\n"}}); + auto r = mtest(root, {"run", "--timeout", "1s", "--grace", "60s", "bash hang.sh"}); + expect(contains(r.out, "asked to end, it ended within its grace of 60s")) << r.out; + auto j = json_of(mtest(root, {"explain", "--json"})); + auto t = j["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << j.dump(); + expect(eq(t.value("ended", ""), std::string("requested"))) << t.dump(); + expect(eq(t.value("grace_ms", 0.0), 60000.0)) << t.dump(); + if (!sees_the_tree()) { + expect(eq(t.value("seen", ""), std::string("program"))) << t.dump(); + expect(le(t["processes"].size(), std::size_t(1))) << t.dump(); + return; + } + expect(eq(t.value("seen", ""), std::string("tree"))) << t.dump(); + expect(contains(r.out, "processes at the timeout:")) << r.out; + const auto& seen = t["processes"]; + const auto sleeping = std::find_if(seen.begin(), seen.end(), [](const json& p) { + return p.value("name", "") == "sleep"; + }); + expect(fatal(sleeping != seen.end())) << t.dump(); + expect(eq(sleeping->value("state", ""), std::string("sleeping"))) << t.dump(); + expect(eq(sleeping->value("request", ""), std::string("ends"))) << t.dump(); + expect(contains(t["remaining"].dump(), "\"sleep\"")) << t.dump(); + }; + "a program that ignores the request to end is ended after its grace"_test[kind::e2e][tag{ + "slow"}] = [] { + auto root = + project({{"stubborn.sh", "trap '' TERM\nsleep 30\n"}, + {"mtest.toml", "[programs.stubborn]\ncommand = [\"bash\", " + "\"stubborn.sh\"]\ntimeout = \"1s\"\ngrace = \"1s\"\n"}}); + const auto began = std::chrono::steady_clock::now(); + auto r = mtest(root, {"run"}); + const std::chrono::duration took = std::chrono::steady_clock::now() - began; + auto j = json_of(mtest(root, {"explain", "--json"})); + auto t = j["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << j.dump(); + expect(eq(j["cases"][0].value("status", ""), std::string("timeout"))); + // openkal's request to end a program is TerminateProcess on Windows, which no + // program declines. + if (mcpplibs::os::host() == mcpplibs::os::platform::windows) { + expect(eq(t.value("ended", ""), std::string("requested"))) << t.dump(); + return; + } + expect(eq(t.value("ended", ""), std::string("terminated"))) << t.dump(); + expect(ge(took.count(), 2.0)) << "the timeout, then the grace"; + expect(contains(r.out, "still running after its grace of 1s, and was ended")) << r.out; + if (!sees_the_tree()) + return; + // bash, and the `sleep` that inherits its disposition, ignore the request. + const auto& left = t["remaining"]; + const auto ignoring = std::count_if(left.begin(), left.end(), [](const json& p) { + return p.value("request", "") == "ignored"; + }); + expect(ge(ignoring, 2)) << t.dump(); + }; + "a protocol program's case says how its timeout ended it; timeout_record = false records " + "nothing"_test[kind::e2e][tag{"slow"}] = [] { + auto root = + project({{"t.sh", script({"quick:pass", "stuck:hang"}, "hang() { sleep 30; }\n")}, + {"mtest.toml", "[programs.t]\ncommand = [\"bash\", \"t.sh\"]\ntimeout = " + "\"1s\"\ntimeout_record = false\n"}}); + auto j = json_of(mtest(root, {"run", "--json"})); + expect(eq(j["counts"].value("timeout", 0), 1)) << j.dump(); + expect(eq(j["counts"].value("pass", 0), 1)) << j.dump(); + expect(contains(j["failing"][0].value("why", ""), "asked to end")) << j.dump(); + auto e = json_of(mtest(root, {"explain", "--json"})); + auto t = e["cases"][0]["timeout"]; + expect(fatal(t.is_object())) << e.dump(); + expect(!t.contains("seen") && !t.contains("processes")) << t.dump(); + }; "a program that does not start is the machine's failure, not the test's"_test[kind::e2e] = [] { auto root = project({{"x.txt", ""}}); auto r = mtest(root, {"run", "--json", "./no-such-program"}); diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index ab285b1..6038e8e 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -84,7 +84,7 @@ isolation holds for everything inside; the others are the innermost one stated. | `clause{"P3.1"}` | a specification clause the case covers | forms the conformance matrix | | `resource{"serial:db"}`, `resource{"machine"}` | what it holds exclusively | never runs two holders of one `serial:` group, nor a `machine` case beside another, at once | | `artifact{"file:data/x.txt"}` | what it reads | selects it when that changes | -| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time | +| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time; `mtest` first records its processes and asks it to end ([docs/commands.md](../../docs/commands.md) "Timeouts") | | `isolate{}` | that it runs in a process of its own | starts one for it | | `retries{2}` | that a failure may be retried | runs it again, alone; one that then passes is `flaky` | | `diagnose{.env = {{"APP_LOG", "debug"}}}` | how to look at a failure | runs a failing case once more with that environment and keeps the output (`mtest explain`) | diff --git a/spec/protocol.md b/spec/protocol.md index ac1799c..dce35c4 100644 --- a/spec/protocol.md +++ b/spec/protocol.md @@ -32,6 +32,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since | B2, runner side | starts programs of any language, by the protocol or by their exit status (P1.1); a program judged by its exit status states the status that skips (`skip_exit`) | 0.1.0; `skip_exit` 0.6.1 | | B4 | keeps every run in the store (P12), and reads the runs a host (`mcpp test`) keeps | 0.1.0; the store's form specified 0.5.0 | | O5 | schedules cases by their resources and history, by pools since 0.5.0; ends a case at its `timeout`, runs `retries` and `diagnose` since 0.3.0 | 0.1.0 | +| O5, the end at a timeout | records the processes of the program a timeout ends, asks it to end, and ends what still runs after its `grace` ([docs/commands.md](../docs/commands.md) "Timeouts") | 0.10.0 | | O6 | serves run fixtures once per run (P11.2, P11.3) | 0.5.0 | | O7 | measures a program that reports no use from a process of its own (P10.4) | 0.5.0 | | P3.3, places | skips a listed case outside its `only` and `except`, and one its program lists skipped with a `reason`, without starting it; takes the program's answers for the capabilities it names (no `[capabilities]` entry needed) and records them in the run; `mtest ci coverage` requires a case to pass on each of its places that a collected run covered | 0.8.0 | From afe0ebc1289360e0f4cb23f9488a644aa713a7b4 Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 19 Nov 2026 14:48:58 +0900 Subject: [PATCH 2/3] 0.10.0: the version in each manifest and in what each part writes, and the changelog --- CHANGELOG.md | 22 ++++++++++++++++++++++ README.md | 2 +- README.zh-CN.md | 2 +- adapters/gtest/mcpp.toml | 2 +- adapters/gtest/src/protocol.cpp | 2 +- capi/mcpp.toml | 2 +- capi/src/mtest.cpp | 2 +- conformance/mcpp.toml | 2 +- engine/mcpp.toml | 2 +- engine/src/commands.cpp | 2 +- engine/src/main.cpp | 4 ++-- engine/src/service.cpp | 2 +- lib/docs/writing-tests.md | 2 +- lib/mcpp.toml | 2 +- lib/src/mtest.cpp | 2 +- manifest/mcpp.toml | 2 +- 16 files changed, 38 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5d111d..73a84fa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,27 @@ # Changelog +## [0.10.0] - 2026-11-19 + +- **A timeout records the program's processes, asks it to end, and then ends it.** Intermittent + timeouts on mcpp's CI (a test ended with status 137) left no state the first time they occurred. + At a program's timeout the engine now records its processes before anything else: on Linux, from + `/proc`, the program, every process it started and every other process of its unit, each with its + parent, group, command, state, the kernel function each of its threads sleeps in, its processor + time, what the request to end does to it (`ends`, `caught`, `ignored`) and, where the system lets + it be read, its kernel stack; on macOS and Windows the program alone. It then asks the program to + end (openkal's request: `SIGTERM` on Linux and macOS; on Windows the program is ended outright), + waits at most the program's grace, records what still runs, and ends the program's unit. The + case's reason says which ended it (`asked to end, it ended within its grace of 2s`, or `... was + still running after its grace of 2s, and was ended`); the record is printed under the case, kept + as the program's `timeout` in the run's `programs.ndjson`, given by `mtest explain`, and shown + below the failures of the Markdown report (docs/commands.md "Timeouts"). +- **`mtest measure` passes the request on.** It starts a program the engine measures (a program + judged by its exit status, on Linux and macOS), and through 0.9.0 the request ended it and not + the test, which its unit's end then ended outright: a test never saw the request. +- **`grace` and `timeout_record`**, keys of `[programs.*]` and `[discover.*]` beside `timeout`, and + `mtest run --grace`. The grace is 2 s by default, the grace of mcpplibs.os's bounded run that the + engine gave through 0.9.0; `timeout_record = false` takes no record. + ## [0.9.0] - 2026-11-15 - **A selection excludes, and selects where declared** (SPEC-017 P4.3). In `MTEST_SELECT`, a line diff --git a/README.md b/README.md index f121c20..b58f68c 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.9.0" +mcpplibs.mtest = "0.10.0" ``` A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program diff --git a/README.zh-CN.md b/README.zh-CN.md index e0da9bf..3aae0c3 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -37,7 +37,7 @@ int main() { ```toml [dev-dependencies] -mcpplibs.mtest = "0.9.0" +mcpplibs.mtest = "0.10.0" ``` 失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。 diff --git a/adapters/gtest/mcpp.toml b/adapters/gtest/mcpp.toml index 6ca5f40..a41302b 100644 --- a/adapters/gtest/mcpp.toml +++ b/adapters/gtest/mcpp.toml @@ -13,7 +13,7 @@ [package] namespace = "mcpplibs.mtest" name = "gtest" -version = "0.9.0" +version = "0.10.0" description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases" license = "Apache-2.0" diff --git a/adapters/gtest/src/protocol.cpp b/adapters/gtest/src/protocol.cpp index 54bebc8..3b7a57b 100644 --- a/adapters/gtest/src/protocol.cpp +++ b/adapters/gtest/src/protocol.cpp @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) { {"protocol", 1}, {"level", 1}, {"framework", "googletest"}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}, {"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/" "writing-tests.md"}}); diff --git a/capi/mcpp.toml b/capi/mcpp.toml index 12ecf2c..611298e 100644 --- a/capi/mcpp.toml +++ b/capi/mcpp.toml @@ -4,7 +4,7 @@ [package] namespace = "mcpplibs.mtest" name = "capi" -version = "0.9.0" +version = "0.10.0" description = "mtest's C ABI: the protocol's events for any language with a C FFI" license = "Apache-2.0" diff --git a/capi/src/mtest.cpp b/capi/src/mtest.cpp index 2932965..cdd7ef2 100644 --- a/capi/src/mtest.cpp +++ b/capi/src/mtest.cpp @@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) { {"protocol", 1}, {"level", std::clamp(level, 1, 3)}, {"framework", text(framework)}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}}); } diff --git a/conformance/mcpp.toml b/conformance/mcpp.toml index 32731c3..2916565 100644 --- a/conformance/mcpp.toml +++ b/conformance/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "conformance" -version = "0.9.0" +version = "0.10.0" description = "The conformance suite of the mtest protocol, for every implementation of its program side" license = "Apache-2.0" diff --git a/engine/mcpp.toml b/engine/mcpp.toml index 6ec688d..3c80487 100644 --- a/engine/mcpp.toml +++ b/engine/mcpp.toml @@ -10,7 +10,7 @@ [package] namespace = "mtest" name = "mtest" -version = "0.9.0" +version = "0.10.0" description = "The mtest engine: run, schedule, store, explain and analyse tests of any language" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/engine/src/commands.cpp b/engine/src/commands.cpp index 66e586c..01924a8 100644 --- a/engine/src/commands.cpp +++ b/engine/src/commands.cpp @@ -1968,7 +1968,7 @@ json catalog() { plain({{"out", {{"type", "string"}}}, {"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}), true); - return {{"schema", "mtest.catalog/1"}, {"version", "0.9.0"}, {"tools", tools}}; + return {{"schema", "mtest.catalog/1"}, {"version", "0.10.0"}, {"tools", tools}}; } } // namespace mcpplibs::mtest::engine diff --git a/engine/src/main.cpp b/engine/src/main.cpp index b158261..e0573b7 100644 --- a/engine/src/main.cpp +++ b/engine/src/main.cpp @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) { cmdline::App application(const json& tools) { auto app = cmdline::App("mtest") - .version("0.9.0") + .version("0.10.0") .description("run tests of any language, and ask about their runs") .option(cmdline::Option("json").global().help("print the answer's data")); for (const auto& t : tools) @@ -253,7 +253,7 @@ int main(int argc, char** argv) { const std::string name(parsed->subcommand_name()); const auto& sub = parsed->subcommand()->get(); if (name == "version") { - std::println("mtest 0.9.0"); + std::println("mtest 0.10.0"); return 0; } const auto cwd = std::filesystem::current_path(); diff --git a/engine/src/service.cpp b/engine/src/service.cpp index a03aa69..11d1feb 100644 --- a/engine/src/service.cpp +++ b/engine/src/service.cpp @@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) { {"result", {{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))}, {"capabilities", {{"tools", {{"listChanged", false}}}}}, - {"serverInfo", {{"name", "mtest"}, {"version", "0.9.0"}}}, + {"serverInfo", {{"name", "mtest"}, {"version", "0.10.0"}}}, {"instructions", "Run tests and ask about runs: run, plan, explain, history, analyze, graph, " "gate, ci. A test session (test/plan, test/run, test/add, test/cancel, " diff --git a/lib/docs/writing-tests.md b/lib/docs/writing-tests.md index 6038e8e..8697a1f 100644 --- a/lib/docs/writing-tests.md +++ b/lib/docs/writing-tests.md @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { / The entry enters the child before any case; the copy runs no case and reports nothing. The files of one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start copies. -- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.9.0"`. +- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.10.0"`. - Identifiers are stable: a selection, a history and a shard name a case by them. ## Expectations diff --git a/lib/mcpp.toml b/lib/mcpp.toml index a8d47ea..2a939ec 100644 --- a/lib/mcpp.toml +++ b/lib/mcpp.toml @@ -17,7 +17,7 @@ [package] namespace = "mcpplibs" name = "mtest" -version = "0.9.0" +version = "0.10.0" description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/lib/src/mtest.cpp b/lib/src/mtest.cpp index bb8ea36..e27fba2 100644 --- a/lib/src/mtest.cpp +++ b/lib/src/mtest.cpp @@ -338,7 +338,7 @@ void write_program() { {"protocol", 1}, {"level", 3}, {"framework", "mcpplibs.mtest"}, - {"version", "0.9.0"}, + {"version", "0.10.0"}, {"isolation", {"process", "case"}}, // P4.3: a selection's `!` and `?` lines are understood. {"selection", {"except", "optional"}}, diff --git a/manifest/mcpp.toml b/manifest/mcpp.toml index 13447df..58fbf56 100644 --- a/manifest/mcpp.toml +++ b/manifest/mcpp.toml @@ -8,7 +8,7 @@ [package] namespace = "mcpplibs.mtest" name = "manifest" -version = "0.9.0" +version = "0.10.0" description = "The static case manifest of mtest: the cases a source file declares, read without building it" license = "Apache-2.0" authors = ["mcpp-community"] From 69417340b22f433b8029ea700baa0f2faacfca1a Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Thu, 19 Nov 2026 15:10:32 +0900 Subject: [PATCH 3/3] CI: each job restores the package store a job of its own saved The native job restored the store the Web job had saved under their common key, which lacked the base GCC builds against; its first build installed it, the next build found the base it had recorded changed and emptied the build directory, and the conformance step found no capi or gtest fixture. The key states the job. --- .github/actions/setup-mcpp/action.yml | 10 ++++++++-- CHANGELOG.md | 4 ++++ 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/actions/setup-mcpp/action.yml b/.github/actions/setup-mcpp/action.yml index 0dee39d..02972b6 100644 --- a/.github/actions/setup-mcpp/action.yml +++ b/.github/actions/setup-mcpp/action.yml @@ -28,14 +28,20 @@ runs: # Linux package is linked against (its interpreter and run paths) are links there, into the # packages. A store restored without it holds programs that do not start, and was taken for # installed (the emsdk compiler, 0.8.4). + # + # One store per job: a job restores what a job of its own saved. The native job of 0.10.0's pull + # request restored the store the Web job had saved first under their common key, which lacked + # the base GCC builds against (glibc, the Linux headers); its first build installed it, the next + # build found the base it had recorded changed and emptied the build directory, and the + # fixtures `mcpp build` had made for the conformance step were gone. - name: Cache the mcpp package store uses: actions/cache@v5 with: path: | ~/.mcpp/registry/data/xpkgs ~/.mcpp/registry/abi - key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }} - restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}- + key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }} + restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}- - name: Install mcpp shell: bash run: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 73a84fa..a6867ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,10 @@ - **`grace` and `timeout_record`**, keys of `[programs.*]` and `[discover.*]` beside `timeout`, and `mtest run --grace`. The grace is 2 s by default, the grace of mcpplibs.os's bounded run that the engine gave through 0.9.0; `timeout_record = false` takes no record. +- CI keeps a package store per job. The native job restored the store the Web job had saved under + their common key, which lacked the base GCC builds against; its first build installed it, the + next build found the base it had recorded changed and emptied the build directory, and the + fixtures `mcpp build` had made for the conformance step were gone. ## [0.9.0] - 2026-11-15