Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/actions/setup-mcpp/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,20 @@ runs:
# Linux package is linked against (its interpreter and run paths) are links there, into the
# packages. A store restored without it holds programs that do not start, and was taken for
# installed (the emsdk compiler, 0.8.4).
#
# One store per job: a job restores what a job of its own saved. The native job of 0.10.0's pull
# request restored the store the Web job had saved first under their common key, which lacked
# the base GCC builds against (glibc, the Linux headers); its first build installed it, the next
# build found the base it had recorded changed and emptied the build directory, and the
# fixtures `mcpp build` had made for the conformance step were gone.
- name: Cache the mcpp package store
uses: actions/cache@v5
with:
path: |
~/.mcpp/registry/data/xpkgs
~/.mcpp/registry/abi
key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }}
restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ env.MCPP_VERSION }}-
key: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}-${{ hashFiles('**/mcpp.toml') }}
restore-keys: mcpp-store-${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-${{ env.MCPP_VERSION }}-
- name: Install mcpp
shell: bash
run: |
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
# Changelog

## [0.10.0] - 2026-11-19

- **A timeout records the program's processes, asks it to end, and then ends it.** Intermittent
timeouts on mcpp's CI (a test ended with status 137) left no state the first time they occurred.
At a program's timeout the engine now records its processes before anything else: on Linux, from
`/proc`, the program, every process it started and every other process of its unit, each with its
parent, group, command, state, the kernel function each of its threads sleeps in, its processor
time, what the request to end does to it (`ends`, `caught`, `ignored`) and, where the system lets
it be read, its kernel stack; on macOS and Windows the program alone. It then asks the program to
end (openkal's request: `SIGTERM` on Linux and macOS; on Windows the program is ended outright),
waits at most the program's grace, records what still runs, and ends the program's unit. The
case's reason says which ended it (`asked to end, it ended within its grace of 2s`, or `... was
still running after its grace of 2s, and was ended`); the record is printed under the case, kept
as the program's `timeout` in the run's `programs.ndjson`, given by `mtest explain`, and shown
below the failures of the Markdown report (docs/commands.md "Timeouts").
- **`mtest measure` passes the request on.** It starts a program the engine measures (a program
judged by its exit status, on Linux and macOS), and through 0.9.0 the request ended it and not
the test, which its unit's end then ended outright: a test never saw the request.
- **`grace` and `timeout_record`**, keys of `[programs.*]` and `[discover.*]` beside `timeout`, and
`mtest run --grace`. The grace is 2 s by default, the grace of mcpplibs.os's bounded run that the
engine gave through 0.9.0; `timeout_record = false` takes no record.
- CI keeps a package store per job. The native job restored the store the Web job had saved under
their common key, which lacked the base GCC builds against; its first build installed it, the
next build found the base it had recorded changed and emptied the build directory, and the
fixtures `mcpp build` had made for the conformance step were gone.

## [0.9.0] - 2026-11-15

- **A selection excludes, and selects where declared** (SPEC-017 P4.3). In `MTEST_SELECT`, a line
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.9.0"
mcpplibs.mtest = "0.10.0"
```

A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program
Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.9.0"
mcpplibs.mtest = "0.10.0"
```

失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。
Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "gtest"
version = "0.9.0"
version = "0.10.0"
description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/src/protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) {
{"protocol", 1},
{"level", 1},
{"framework", "googletest"},
{"version", "0.9.0"},
{"version", "0.10.0"},
{"isolation", {"process", "case"}},
{"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/"
"writing-tests.md"}});
Expand Down
2 changes: 1 addition & 1 deletion capi/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "capi"
version = "0.9.0"
version = "0.10.0"
description = "mtest's C ABI: the protocol's events for any language with a C FFI"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion capi/src/mtest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) {
{"protocol", 1},
{"level", std::clamp(level, 1, 3)},
{"framework", text(framework)},
{"version", "0.9.0"},
{"version", "0.10.0"},
{"isolation", {"process", "case"}}});
}

Expand Down
2 changes: 1 addition & 1 deletion conformance/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "conformance"
version = "0.9.0"
version = "0.10.0"
description = "The conformance suite of the mtest protocol, for every implementation of its program side"
license = "Apache-2.0"

Expand Down
31 changes: 30 additions & 1 deletion docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ nothing; a built program's own listing is the authority (`manifest/src/manifest.

| Command | Does |
|---|---|
| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) |
| `run` | runs the selection; `--jobs N`, `--seed N`, `--fail-fast`, `--retries N`, `--isolate` (a process per case), `--timeout 5m`, `--grace 2s` (below), `--env K=V`, `--run-under CMD` (every start of a program, its listing included: a runner such as an emulator or a machine of the program's system), `--monitor LEVEL` (P10), `--group G` (runs reported together); `--junit`, `--html`, `--report-json FILE`, `--md FILE`; `--coverage` measures the coverage map. A program's cases are divided among workers that would otherwise idle (`[run] split`), and programs run together within the machine's memory (`[run] memory`) |
| `run --resume RUN` | the selection, less the cases RUN judged: their results are carried into the new run, so an interrupted long run (a bench over a matrix) ends as one whole run; a case RUN could not start runs again |
| `rerun` | the cases that failed in the last run |
| `smoke` | the `smoke` set |
Expand All @@ -49,6 +49,35 @@ A program runs by the protocol when it carries the marker or declares its level;
status is its one result. Cases tagged `flaky`, and quarantined ones, are retried alone up to
`--retries`; one that then passes is `flaky`. A quarantined case's failure does not fail the run.

### Timeouts

A program still running at its timeout (`timeout`, `--timeout`; a case's own `timeout` where a
process runs that case alone) is first asked to end and then ended, and its processes are recorded
before the request, so that a hang leaves the state it hung in:

1. **The record.** On Linux, from `/proc`: the program, every process it started, and every other
process of its unit (the process group it leads, which a process it started keeps after the
program has ended). Each process states its parent, its group, its command, its state, the kernel
function it sleeps in, its processor time, what the request to end does to it (`ends`, `caught`,
`ignored`), each of its threads with its state and its kernel function, and, where the system lets
it be read, its kernel stack. Where the system lists no other process to the engine (macOS,
Windows, a `/proc` the engine cannot read), the record holds the program alone, with its
processor time, and says so.
2. **The request.** openkal's request to end a program: `SIGTERM` on Linux and macOS, which a
program may handle or ignore; on Windows the program is ended outright. The request reaches the
program the engine started; `mtest measure`, which starts a program the engine measures, passes
it on. What the program started receives it from the program, or not at all.
3. **The grace.** The engine waits for the program at most its `grace` (a key of `[programs.*]` and
`[discover.*]`, [config.md](config.md); `--grace`; 2 s by default).
4. **The end.** What still runs in the program's unit is recorded again and ended.

The case's reason says how the program ended: `asked to end, it ended within its grace of 2s`, or
`asked to end, it was still running after its grace of 2s, and was ended`. The record is printed
under the case, kept in the run's `programs.ndjson` as the program's `timeout` (`limit_ms`,
`grace_ms`, `ended`, `seen`, `processes`, `remaining`), given by `mtest explain`, and shown below the
failures of the Markdown report. A program that states `timeout_record = false` is asked to end and
ended in the same way, with no record.

## Asking

| Command | Answers |
Expand Down
29 changes: 16 additions & 13 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,22 @@ jobs = 0 # 0: the machine's processors

# A program of any language. `level` states the protocol's level; without it the marker decides (P2.3).
[programs.e2e]
command = ["python3", "tests/e2e.py"]
cwd = "."
env = { MODE = "ci" }
sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection)
resources = ["serial:db"] # what it holds alone (P7.1)
tags = ["slow"]
timeout = "10m"
isolation = "process" # or "case": a process per case
kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit)
split = true # whether [run] split may divide its cases among processes
skip_exit = 77 # a program judged by its exit status: the status that says its case did not
# apply here (skip, with the last `SKIP...` line of its output as the reason);
# unset: every status but 0 fails
command = ["python3", "tests/e2e.py"]
cwd = "."
env = { MODE = "ci" }
sources = ["src/*", "tests/e2e.py"] # a change of these affects it (affected selection)
resources = ["serial:db"] # what it holds alone (P7.1)
tags = ["slow"]
timeout = "10m"
grace = "2s" # at its timeout it is asked to end; what still runs this much later is
# ended (commands.md "Timeouts"); 2s by default
timeout_record = true # its processes are recorded at its timeout and after the grace; false: no record
isolation = "process" # or "case": a process per case
kind = "e2e" # the kind of its cases that state none (default: its directory's, else unit)
split = true # whether [run] split may divide its cases among processes
skip_exit = 77 # a program judged by its exit status: the status that says its case did not
# apply here (skip, with the last `SKIP...` line of its output as the reason);
# unset: every status but 0 fails

[discover]
globs = ["build/tests/*"] # every executable that matches is a program
Expand Down
2 changes: 1 addition & 1 deletion engine/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
[package]
namespace = "mtest"
name = "mtest"
version = "0.9.0"
version = "0.10.0"
description = "The mtest engine: run, schedule, store, explain and analyse tests of any language"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
12 changes: 11 additions & 1 deletion engine/src/commands.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,10 @@ std::expected<std::vector<program_spec>, answer> programs_of(const context& ctx,
if (auto t = text_of(p, "timeout"); !t.empty())
if (auto d = parse_duration(t))
s.timeout = std::chrono::duration_cast<std::chrono::seconds>(*d);
for (auto& s : *programs)
if (auto t = text_of(p, "grace"); !t.empty())
if (auto d = parse_duration(t))
s.grace = *d;
if (p.value("isolate", false))
for (auto& s : *programs)
s.isolation = "case";
Expand Down Expand Up @@ -825,6 +829,8 @@ answer cmd_explain(const std::filesystem::path& cwd, const json& p) {
}
if (c.contains("output") && !c["output"]["err_tail"].get<std::string>().empty())
text += " error output (end):\n" + c["output"]["err_tail"].get<std::string>() + "\n";
if (c.contains("timeout"))
text += timeout_text(timeout_info_from(c["timeout"]), " ", 400);
if (c.contains("diagnosis"))
text += " diagnosis: " + c["diagnosis"].get<std::string>() + "\n";
text += " again: " + c["again"].get<std::string>() + "\n";
Expand Down Expand Up @@ -1799,6 +1805,10 @@ json catalog() {
{"retries", {{"type", "integer"}, {"description", "runs again of a failed case"}}},
{"isolate", {{"type", "boolean"}, {"description", "each case in its own process"}}},
{"timeout", {{"type", "string"}, {"description", "a duration for each program"}}},
{"grace",
{{"type", "string"},
{"description", "how long a program asked to end at its timeout has before it "
"is ended"}}},
{"env", {{"type", "object"}, {"description", "NAME=VALUE for the programs"}}},
{"run_under",
{{"type", "array"},
Expand Down Expand Up @@ -1958,7 +1968,7 @@ json catalog() {
plain({{"out", {{"type", "string"}}},
{"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}),
true);
return {{"schema", "mtest.catalog/1"}, {"version", "0.9.0"}, {"tools", tools}};
return {{"schema", "mtest.catalog/1"}, {"version", "0.10.0"}, {"tools", tools}};
}

} // namespace mcpplibs::mtest::engine
13 changes: 7 additions & 6 deletions engine/src/config.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,12 +43,9 @@ void known(const toml::table& t, std::string_view where,
}

// The keys a program states, in `[programs.*]` and `[discover.*]` alike.
constexpr std::initializer_list<std::string_view> kProgramKeys = {"command", "cwd",
"env", "level",
"resources", "sources",
"tags", "timeout",
"isolation", "kind",
"split", "skip_exit"};
constexpr std::initializer_list<std::string_view> kProgramKeys =
{"command", "cwd", "env", "level", "resources", "sources", "tags",
"timeout", "isolation", "kind", "split", "skip_exit", "grace", "timeout_record"};

void read_program(const toml::table& t, program_spec& s) {
if (auto v = t["cwd"].value<std::string>())
Expand All @@ -67,6 +64,10 @@ void read_program(const toml::table& t, program_spec& s) {
s.tags = strings(t["tags"]);
if (auto d = duration_of(t["timeout"]))
s.timeout = std::chrono::duration_cast<std::chrono::seconds>(*d);
if (auto d = duration_of(t["grace"]))
s.grace = *d;
if (auto v = t["timeout_record"].value<bool>())
s.timeout_record = *v;
if (auto v = t["isolation"].value<std::string>())
s.isolation = *v;
if (auto v = t["split"].value<bool>())
Expand Down
Loading
Loading