Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Changelog

## [0.11.1] - 2026-11-21

- **A command is held to what its case has left.** `time_left()` states what the running case has
left of its `timeout`, the earliest end of its own and of each scope's around it, each counted
from when it began; nothing when none states one. `e2e::run` holds a command to it, less what ending
the command takes (its grace and drain) and a tenth (at most five seconds) the case keeps for its
report: a command that would outlast its case ends
while the case can still say so, with the outputs it wrote, and the outcome's `case_limit` and
the failure's text (`the time its case had left`) say why. A command of a case that states no
time keeps its own limit. Source-compatible with 0.11.0; the engine changes only the version
it states, which the five packages share.

## [0.11.0] - 2026-11-21

What a test system needs to see the cost of the tests it runs (SPEC-017 v2.5; mcpp's test and CI
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.11.0"
mcpplibs.mtest = "0.11.1"
```

A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program
Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.11.0"
mcpplibs.mtest = "0.11.1"
```

失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。
Expand Down
2 changes: 1 addition & 1 deletion adapters/doctest/docs/writing-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ package `mcpplibs.mtest.doctest` supplies doctest's implementation and the progr

```toml
[dev-dependencies]
mcpplibs.mtest.doctest = "0.11.0"
mcpplibs.mtest.doctest = "0.11.1"
```

```cpp
Expand Down
4 changes: 2 additions & 2 deletions adapters/doctest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@
# neither (no `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN`):
#
# [dev-dependencies]
# mcpplibs.mtest.doctest = "0.11.0"
# mcpplibs.mtest.doctest = "0.11.1"
#
# A case's identifier is `<suite>/<name>` (P3.2), or `<name>` outside a `TEST_SUITE`. doctest's
# macros exist only in its header, so this package's one source includes it, and nlohmann/json's
# beside it, as the GoogleTest adapter does.
[package]
namespace = "mcpplibs.mtest"
name = "doctest"
version = "0.11.0"
version = "0.11.1"
description = "doctest speaking the mtest protocol: doctest's implementation and a test main that lists, selects and reports its cases"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion adapters/doctest/src/protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ int mcpplibs_mtest_doctest_main(int argc, char** argv) {
{"protocol", 1},
{"level", 1},
{"framework", "doctest"},
{"version", "0.11.0"},
{"version", "0.11.1"},
{"isolation", {"process", "case"}},
{"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/doctest/docs/"
"writing-tests.md"}});
Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "gtest"
version = "0.11.0"
version = "0.11.1"
description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/src/protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) {
{"protocol", 1},
{"level", 1},
{"framework", "googletest"},
{"version", "0.11.0"},
{"version", "0.11.1"},
{"isolation", {"process", "case"}},
{"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/"
"writing-tests.md"}});
Expand Down
2 changes: 1 addition & 1 deletion capi/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "capi"
version = "0.11.0"
version = "0.11.1"
description = "mtest's C ABI: the protocol's events for any language with a C FFI"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion capi/src/mtest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) {
{"protocol", 1},
{"level", std::clamp(level, 1, 3)},
{"framework", text(framework)},
{"version", "0.11.0"},
{"version", "0.11.1"},
{"isolation", {"process", "case"}}});
}

Expand Down
2 changes: 1 addition & 1 deletion conformance/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "conformance"
version = "0.11.0"
version = "0.11.1"
description = "The conformance suite of the mtest protocol, for every implementation of its program side"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion engine/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
[package]
namespace = "mtest"
name = "mtest"
version = "0.11.0"
version = "0.11.1"
description = "The mtest engine: run, schedule, store, explain and analyse tests of any language"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
2 changes: 1 addition & 1 deletion engine/src/commands.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2009,7 +2009,7 @@ json catalog() {
plain({{"out", {{"type", "string"}}},
{"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}),
true);
return {{"schema", "mtest.catalog/1"}, {"version", "0.11.0"}, {"tools", tools}};
return {{"schema", "mtest.catalog/1"}, {"version", "0.11.1"}, {"tools", tools}};
}

} // namespace mcpplibs::mtest::engine
4 changes: 2 additions & 2 deletions engine/src/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) {

cmdline::App application(const json& tools) {
auto app = cmdline::App("mtest")
.version("0.11.0")
.version("0.11.1")
.description("run tests of any language, and ask about their runs")
.option(cmdline::Option("json").global().help("print the answer's data"));
for (const auto& t : tools)
Expand Down Expand Up @@ -253,7 +253,7 @@ int main(int argc, char** argv) {
const std::string name(parsed->subcommand_name());
const auto& sub = parsed->subcommand()->get();
if (name == "version") {
std::println("mtest 0.11.0");
std::println("mtest 0.11.1");
return 0;
}
const auto cwd = std::filesystem::current_path();
Expand Down
2 changes: 1 addition & 1 deletion engine/src/service.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) {
{"result",
{{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))},
{"capabilities", {{"tools", {{"listChanged", false}}}}},
{"serverInfo", {{"name", "mtest"}, {"version", "0.11.0"}}},
{"serverInfo", {{"name", "mtest"}, {"version", "0.11.1"}}},
{"instructions",
"Run tests and ask about runs: run, plan, explain, history, analyze, graph, "
"gate, ci. A test session (test/plan, test/run, test/add, test/cancel, "
Expand Down
4 changes: 2 additions & 2 deletions lib/docs/writing-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { /
The entry enters the child before any case; the copy runs no case and reports nothing. The files of
one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start
copies.
- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.11.0"`.
- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.11.1"`.
- Identifiers are stable: a selection, a history and a shard name a case by them.

## Expectations
Expand Down Expand Up @@ -84,7 +84,7 @@ isolation holds for everything inside; the others are the innermost one stated.
| `clause{"P3.1"}` | a specification clause the case covers | forms the conformance matrix |
| `resource{"serial:db"}`, `resource{"machine"}` | what it holds exclusively | never runs two holders of one `serial:` group, nor a `machine` case beside another, at once |
| `artifact{"file:data/x.txt"}` | what it reads | selects it when that changes |
| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time; `mtest` first records its processes and asks it to end ([docs/commands.md](../../docs/commands.md) "Timeouts") |
| `timeout{std::chrono::seconds{90}}` | how long it may run | ends the process that runs it at that time; `mtest` first records its processes and asks it to end ([docs/commands.md](../../docs/commands.md) "Timeouts"). Within the case, `time_left()` is what remains of it and of each scope's around it, and `e2e::run` holds a command to that, less what ending it takes and a tenth (at most five seconds) the case keeps for its report (0.11.1) |
| `isolate{}` | that it runs in a process of its own | starts one for it |
| `retries{2}` | that a failure may be retried | runs it again, alone; one that then passes is `flaky` |
| `diagnose{.env = {{"APP_LOG", "debug"}}}` | how to look at a failure | runs a failing case once more with that environment and keeps the output (`mtest explain`) |
Expand Down
2 changes: 1 addition & 1 deletion lib/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
[package]
namespace = "mcpplibs"
name = "mtest"
version = "0.11.0"
version = "0.11.1"
description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
16 changes: 15 additions & 1 deletion lib/src/e2e.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,19 @@ outcome run(command c) {
o.env.push_back(std::move(kv));
o.in = c.input.empty() ? os::process::stdio::null : os::process::stdio::pipe;
os::process::bounds limits;
// A command is held to what its case has left (0.11.1): one that would outlast the case ends
// while the case can still say so, with the outputs it wrote, instead of with the program the
// runner ends at its limit. What ending it takes -- the grace after asking it to end, the
// drain of its outputs -- comes off first, and a tenth of what is left, at most five seconds,
// stays the case's own for its report.
if (const auto left = time_left()) {
const auto keep =
(std::min)(*left / 10, std::chrono::milliseconds{5000}) + limits.grace + limits.drain;
if (const auto usable = *left - keep; usable < c.timeout) {
c.timeout = (std::max)(usable, std::chrono::milliseconds{1});
r.case_limit = true;
}
}
limits.hard = c.timeout;
limits.out_limit = c.out_limit;
limits.err_limit = c.err_limit;
Expand Down Expand Up @@ -351,7 +364,8 @@ comparison about(const outcome& r, bool held, std::string expression, std::strin
if (!r.started)
c.actual = std::format("did not start: {}", r.err);
else if (r.timed_out)
c.actual = std::format("timed out after {} ms", r.duration.count());
c.actual = std::format("timed out after {} ms{}", r.duration.count(),
r.case_limit ? ", the time its case had left" : "");
else
c.actual = std::format("status {}", r.status);
if (!held)
Expand Down
2 changes: 2 additions & 0 deletions lib/src/e2e.cppm
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ struct outcome {
bool out_truncated = false;
bool err_truncated = false;
bool held_open = false;
// Its timeout was what its case had left, not the command's own (0.11.1).
bool case_limit = false;
std::string out, err;
std::chrono::milliseconds duration{0};
std::string line; // the command as a POSIX shell line, to repeat it by hand (0.7.0)
Expand Down
18 changes: 17 additions & 1 deletion lib/src/mtest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ struct running_case {
bool capturing = false;
// Checks made at its end, before its verdict, by key (at_case_end, 0.11.0).
std::vector<std::pair<std::string, std::function<void()>>> end_checks;
// When it began: its `timeout` counts from here (time_left, 0.11.1).
clock::time_point began = clock::now();
};

// The expectations that held, in the whole program: a case's share is the difference over its
Expand Down Expand Up @@ -340,7 +342,7 @@ void write_program() {
{"protocol", 1},
{"level", 3},
{"framework", "mcpplibs.mtest"},
{"version", "0.11.0"},
{"version", "0.11.1"},
{"isolation", {"process", "case"}},
// P4.3: a selection's `!<id>` and `?<id>` lines are understood.
{"selection", {"except", "optional"}},
Expand Down Expand Up @@ -1377,6 +1379,20 @@ std::string current_case() {
return s.stack.empty() || s.stack.back().scope ? std::string{} : s.stack.back().id;
}

std::optional<std::chrono::milliseconds> time_left() {
auto& s = the();
std::lock_guard g(s.lock);
std::optional<clock::time_point> deadline;
for (const auto& r : s.stack)
if (r.attrs.limit && (!deadline || r.began + *r.attrs.limit < *deadline))
deadline = r.began + *r.attrs.limit;
if (!deadline)
return std::nullopt;
const auto left =
std::chrono::duration_cast<std::chrono::milliseconds>(*deadline - clock::now());
return (std::max)(left, std::chrono::milliseconds{0});
}

void produced(const std::filesystem::path& file, std::string_view kind) {
write_event({{"e", "artifact"},
{"id", current_case()},
Expand Down
5 changes: 5 additions & 0 deletions lib/src/mtest.cppm
Original file line number Diff line number Diff line change
Expand Up @@ -846,6 +846,11 @@ private:
std::uint64_t seed();
// The identifier of the case that runs now; empty outside a case.
std::string current_case();
// What the running case has left of the time it may run: the earliest end of its own `timeout` and
// of each scope's around it, each counted from when it began; nothing when none states one, or
// outside a case (0.11.1). mcpplibs.mtest.e2e holds each command to it, less what ending the
// command takes and a tenth (at most five seconds) the case keeps for its report.
std::optional<std::chrono::milliseconds> time_left();
// A file the case produced, reported with the case (P5.2 `artifact`).
void produced(const std::filesystem::path& file, std::string_view kind = "file");

Expand Down
26 changes: 26 additions & 0 deletions lib/tests/children.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ child echo{"echo", [](std::vector<std::string> args) {
return static_cast<int>(args.size()) + 3;
}};

// Sleeps for its first argument's count of seconds.
child sleeper{"sleeper", [](std::vector<std::string> args) {
std::this_thread::sleep_for(std::chrono::seconds(std::stoul(args.at(0))));
return 0;
}};

// Writes its first argument's count of bytes to standard error, and one line to standard output.
child loud{"loud", [](std::vector<std::string> args) {
std::print(std::cerr, "{}", std::string(std::stoul(args.at(0)), 'e'));
Expand Down Expand Up @@ -43,6 +49,26 @@ UTest children{"children", [] {
expect(eq(whole.err.size(), std::size_t{65536}));
expect(!whole.err_truncated);
};
// 0.11.1: a command is held to what its case has left of its `timeout`.
"a command ends when its case's time does"_test[timeout{
std::chrono::seconds{10}}] = [] {
const auto left = time_left();
expect(fatal(left.has_value()));
expect(le(left->count(), 10000LL));
const std::vector<std::string> secs{"30"};
auto o = e2e::run({.argv = child_command("sleeper", secs),
.timeout = std::chrono::seconds{60}});
expect(o.timed_out);
expect(o.case_limit);
expect(lt(o.duration.count(), 10000LL))
<< "it did not leave the case time of its own";
expect(e2e::succeeded(o).actual.contains("the time its case had left"));
};
"a case that states no time leaves the command its own"_test = [] {
expect(!time_left().has_value());
auto o = e2e::run({.argv = child_command("echo", {})});
expect(!o.case_limit);
};
"a name no file declares is refused"_test = [] {
auto argv = self_command();
argv.insert(argv.end(), {"--mtest-child", "none"});
Expand Down
2 changes: 1 addition & 1 deletion manifest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "manifest"
version = "0.11.0"
version = "0.11.1"
description = "The static case manifest of mtest: the cases a source file declares, read without building it"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
1 change: 1 addition & 0 deletions spec/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since
| P3.3, places and capabilities | a `case` event states `only` and `except` (the platforms and architectures of `only{}`, `except{}`), `capabilities` (each `capability` the case requires, as the program answered it), and the `reason` of a case it lists skipped; the program skips a case outside its places or lacking a capability, whichever runner starts it | 0.8.0 |
| B2: P4.3 | a selection's `!<id>` lines leave out a case and the cases it declares, and its `?<id>` lines select a case where it is declared and are not `missing` elsewhere; the `program` event states `selection: ["except", "optional"]` | 0.9.0 |
| P5.6 | `step()` writes a `step` event and `exercise()` an `exercise` event of the running case; `e2e::run` writes a step for each command it runs | 0.11.0 |
| O5 within a case | `time_left()` states what a case has left of its `timeout` and its scopes'; `e2e::run` holds a command to it | 0.11.1 |
| O12 `jitter` | `contention_point()` waits a random time up to `MTEST_JITTER`, the same for the same `MTEST_JITTER_SEED`; a case begins and a shared object is made at one | 0.11.0 |
| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own. Where the system cannot rearrange the program's streams (an ok image), no output is kept for a case, the spec's fallback | 0.6.0 |

Expand Down
Loading