Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,14 @@
# Changelog

## [0.11.2] - 2026-11-21

- **What a scope does names no case.** A case that runs only because it declares a selected case
is that case's scope (P3.4) and is not reported unless a statement of its own fails. A `step`,
`exercise`, `artifact`, `metric` or `fixture` event written while its body ran named it, and the
runner, which never saw it begin, held it as a case that did not end: a capability probe run by
`e2e::run` in a suite's body failed the program whose cases all passed. These events now name no
case. Source-compatible with 0.11.1; the engine changes only the version it states.

## [0.11.1] - 2026-11-21

- **A command is held to what its case has left.** `time_left()` states what the running case has
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.11.1"
mcpplibs.mtest = "0.11.2"
```

A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program
Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.11.1"
mcpplibs.mtest = "0.11.2"
```

失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。
Expand Down
2 changes: 1 addition & 1 deletion adapters/doctest/docs/writing-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ package `mcpplibs.mtest.doctest` supplies doctest's implementation and the progr

```toml
[dev-dependencies]
mcpplibs.mtest.doctest = "0.11.1"
mcpplibs.mtest.doctest = "0.11.2"
```

```cpp
Expand Down
4 changes: 2 additions & 2 deletions adapters/doctest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@
# neither (no `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN`):
#
# [dev-dependencies]
# mcpplibs.mtest.doctest = "0.11.1"
# mcpplibs.mtest.doctest = "0.11.2"
#
# A case's identifier is `<suite>/<name>` (P3.2), or `<name>` outside a `TEST_SUITE`. doctest's
# macros exist only in its header, so this package's one source includes it, and nlohmann/json's
# beside it, as the GoogleTest adapter does.
[package]
namespace = "mcpplibs.mtest"
name = "doctest"
version = "0.11.1"
version = "0.11.2"
description = "doctest speaking the mtest protocol: doctest's implementation and a test main that lists, selects and reports its cases"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion adapters/doctest/src/protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ int mcpplibs_mtest_doctest_main(int argc, char** argv) {
{"protocol", 1},
{"level", 1},
{"framework", "doctest"},
{"version", "0.11.1"},
{"version", "0.11.2"},
{"isolation", {"process", "case"}},
{"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/doctest/docs/"
"writing-tests.md"}});
Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "gtest"
version = "0.11.1"
version = "0.11.2"
description = "GoogleTest speaking the mtest protocol: a test main that lists, selects and reports GoogleTest's cases"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion adapters/gtest/src/protocol.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ int mcpplibs_mtest_gtest_main(int argc, char** argv) {
{"protocol", 1},
{"level", 1},
{"framework", "googletest"},
{"version", "0.11.1"},
{"version", "0.11.2"},
{"isolation", {"process", "case"}},
{"guide", "https://github.com/mcpp-community/mtest/blob/main/adapters/gtest/docs/"
"writing-tests.md"}});
Expand Down
2 changes: 1 addition & 1 deletion capi/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "capi"
version = "0.11.1"
version = "0.11.2"
description = "mtest's C ABI: the protocol's events for any language with a C FFI"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion capi/src/mtest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ void mtest_init(const char* framework, int level) {
{"protocol", 1},
{"level", std::clamp(level, 1, 3)},
{"framework", text(framework)},
{"version", "0.11.1"},
{"version", "0.11.2"},
{"isolation", {"process", "case"}}});
}

Expand Down
2 changes: 1 addition & 1 deletion conformance/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "conformance"
version = "0.11.1"
version = "0.11.2"
description = "The conformance suite of the mtest protocol, for every implementation of its program side"
license = "Apache-2.0"

Expand Down
2 changes: 1 addition & 1 deletion engine/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
[package]
namespace = "mtest"
name = "mtest"
version = "0.11.1"
version = "0.11.2"
description = "The mtest engine: run, schedule, store, explain and analyse tests of any language"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
2 changes: 1 addition & 1 deletion engine/src/commands.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -2009,7 +2009,7 @@ json catalog() {
plain({{"out", {{"type", "string"}}},
{"argv", {{"type", "array"}, {"items", {{"type", "string"}}}}}}),
true);
return {{"schema", "mtest.catalog/1"}, {"version", "0.11.1"}, {"tools", tools}};
return {{"schema", "mtest.catalog/1"}, {"version", "0.11.2"}, {"tools", tools}};
}

} // namespace mcpplibs::mtest::engine
4 changes: 2 additions & 2 deletions engine/src/main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ const json* tool_named(const json& tools, std::string_view name) {

cmdline::App application(const json& tools) {
auto app = cmdline::App("mtest")
.version("0.11.1")
.version("0.11.2")
.description("run tests of any language, and ask about their runs")
.option(cmdline::Option("json").global().help("print the answer's data"));
for (const auto& t : tools)
Expand Down Expand Up @@ -253,7 +253,7 @@ int main(int argc, char** argv) {
const std::string name(parsed->subcommand_name());
const auto& sub = parsed->subcommand()->get();
if (name == "version") {
std::println("mtest 0.11.1");
std::println("mtest 0.11.2");
return 0;
}
const auto cwd = std::filesystem::current_path();
Expand Down
2 changes: 1 addition & 1 deletion engine/src/service.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ int serve(const std::filesystem::path& cwd) {
{"result",
{{"protocolVersion", params.value("protocolVersion", std::string("2025-06-18"))},
{"capabilities", {{"tools", {{"listChanged", false}}}}},
{"serverInfo", {{"name", "mtest"}, {"version", "0.11.1"}}},
{"serverInfo", {{"name", "mtest"}, {"version", "0.11.2"}}},
{"instructions",
"Run tests and ask about runs: run, plan, explain, history, analyze, graph, "
"gate, ci. A test session (test/plan, test/run, test/add, test/cancel, "
Expand Down
2 changes: 1 addition & 1 deletion lib/docs/writing-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ E2E cli{"cli", resource{"serial:home"}, timeout{std::chrono::minutes{2}}, [] { /
The entry enters the child before any case; the copy runs no case and reports nothing. The files of
one directory are then one program (mcpp `[test] programs = "directory"`) however many of them start
copies.
- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.11.1"`.
- The manifest names the library once: `[dev-dependencies] mcpplibs.mtest = "0.11.2"`.
- Identifiers are stable: a selection, a history and a shard name a case by them.

## Expectations
Expand Down
2 changes: 1 addition & 1 deletion lib/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
[package]
namespace = "mcpplibs"
name = "mtest"
version = "0.11.1"
version = "0.11.2"
description = "The C++ test library of mtest: cases, expectations, fixtures, bench, conformance and end-to-end helpers, with no macro and no header, speaking the mtest protocol"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
28 changes: 21 additions & 7 deletions lib/src/mtest.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ struct running_case {
std::vector<std::pair<std::string, std::function<void()>>> end_checks;
// When it began: its `timeout` counts from here (time_left, 0.11.1).
clock::time_point began = clock::now();
// Reported: selected, or declared in a selected case. One that runs only because it declares
// a selected case is that case's scope; its begin is written only if it fails (0.11.2).
bool own = true;
};

// The expectations that held, in the whole program: a case's share is the difference over its
Expand Down Expand Up @@ -342,7 +345,7 @@ void write_program() {
{"protocol", 1},
{"level", 3},
{"framework", "mcpplibs.mtest"},
{"version", "0.11.1"},
{"version", "0.11.2"},
{"isolation", {"process", "case"}},
// P4.3: a selection's `!<id>` and `?<id>` lines are understood.
{"selection", {"except", "optional"}},
Expand Down Expand Up @@ -1052,6 +1055,7 @@ void test::operator=(body b) {
end_capture(s.stack.back()); // the declaring case's output so far is its own
s.stack.push_back({id, false, attrs});
s.stack.back().held_before = g_held.load(std::memory_order_relaxed);
s.stack.back().own = own;
if (capturing) {
s.stack.back().capturing = true;
begin_capture(s.stack.back());
Expand Down Expand Up @@ -1379,6 +1383,16 @@ std::string current_case() {
return s.stack.empty() || s.stack.back().scope ? std::string{} : s.stack.back().id;
}

// The case an event of the running code states: the running case when it is reported, else none.
// A step, an artifact, a metric or a fixture of a case that runs as a scope would otherwise name a
// case the runner never saw begin, and the runner holds such a case as one that did not end.
static std::string event_case() {
auto& s = the();
std::lock_guard g(s.lock);
return s.stack.empty() || s.stack.back().scope || !s.stack.back().own ? std::string{}
: s.stack.back().id;
}

std::optional<std::chrono::milliseconds> time_left() {
auto& s = the();
std::lock_guard g(s.lock);
Expand All @@ -1395,7 +1409,7 @@ std::optional<std::chrono::milliseconds> time_left() {

void produced(const std::filesystem::path& file, std::string_view kind) {
write_event({{"e", "artifact"},
{"id", current_case()},
{"id", event_case()},
{"path", os::fs::generic_utf8(file)},
{"kind", kind}});
}
Expand Down Expand Up @@ -1442,7 +1456,7 @@ void at_case_end(std::string_view key, std::function<void()> check) {
void step(const step_record& r) {
const auto at = std::chrono::duration<double, std::milli>(r.began - the().began).count();
event ev = {{"e", "step"},
{"id", current_case()},
{"id", event_case()},
{"name", r.name},
{"t", at < 0 ? 0.0 : at},
{"ms", std::chrono::duration<double, std::milli>(r.took).count()}};
Expand All @@ -1458,7 +1472,7 @@ void step(const step_record& r) {
void exercise(const std::vector<std::string>& items) {
if (items.empty())
return;
write_event({{"e", "exercise"}, {"id", current_case()}, {"items", items}});
write_event({{"e", "exercise"}, {"id", event_case()}, {"items", items}});
}

void metric(std::string_view name, double value, std::string_view unit,
Expand All @@ -1469,8 +1483,8 @@ void metric(std::string_view name, double value, std::string_view unit,
if (!s.stack.empty() && !s.stack.back().scope)
s.stack.back().metrics[std::string(name)] = {value, std::string(unit)};
}
event ev = {{"e", "metric"}, {"id", current_case()}, {"name", name},
{"value", value}, {"unit", unit}, {"better", better}};
event ev = {{"e", "metric"}, {"id", event_case()}, {"name", name},
{"value", value}, {"unit", unit}, {"better", better}};
if (!samples.empty())
ev["samples"] = std::vector<double>(samples.begin(), samples.end());
write_event(ev);
Expand Down Expand Up @@ -1760,7 +1774,7 @@ attributes within(const attributes& outer, const attributes& inner) {
}

void detail::made_shared(std::string_view name, double ms) {
write_event({{"e", "fixture"}, {"id", current_case()}, {"name", name}, {"ms", ms}});
write_event({{"e", "fixture"}, {"id", event_case()}, {"name", name}, {"ms", ms}});
}

std::vector<std::string> self_command() {
Expand Down
4 changes: 3 additions & 1 deletion lib/src/mtest.cppm
Original file line number Diff line number Diff line change
Expand Up @@ -857,7 +857,9 @@ void produced(const std::filesystem::path& file, std::string_view kind = "file")
// A STEP OF THE RUNNING CASE (P5.6, 0.11.0): one command an end-to-end case ran, when it began, how
// long it took, the processor time of the programs it ran, their exit status, and the record of
// the build it was, when it was one. mcpplibs.mtest.e2e writes one for each command it runs; a kit
// that runs its commands another way writes its own. A step is a record and never a verdict.
// that runs its commands another way writes its own. A step is a record and never a verdict. A
// step, an exercise, an artifact, a metric or a fixture of a case that runs as the scope of a
// selected case (P3.4) names no case (0.11.2).
struct step_record {
std::string name;
std::chrono::steady_clock::time_point began;
Expand Down
7 changes: 7 additions & 0 deletions lib/tests/suites.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ Suite kept{"kept", [] {

Suite scoped{"scoped", [] {
expect(!std::getenv("SCOPE_FAILS")) << "the scope's own statement";
step({.name = "the scope's own command",
.began = std::chrono::steady_clock::now()});
exercise({"scope.item"});
"inner"_test = [] {};
}};

Expand Down Expand Up @@ -242,6 +245,10 @@ int main(int argc, char** argv) {
auto quiet = ended(events_of("run", {"scoped/inner"}));
expect(eq(quiet["scoped/inner"], std::string("pass")));
expect(!quiet.contains("scoped"));
// What the scope's body does names no case: the runner saw none begin (0.11.2).
auto events = events_of("run", {"scoped/inner"});
for (const auto& e : events)
expect(e.value("id", "") != "scoped") << "an event named the scope: " << e.dump();
auto loud = ended(events_of("run", {"scoped/inner"}, {}, {{"SCOPE_FAILS", "1"}}));
expect(eq(loud["scoped"], std::string("fail")));
expect(eq(loud["scoped/inner"], std::string("pass")));
Expand Down
2 changes: 1 addition & 1 deletion manifest/mcpp.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[package]
namespace = "mcpplibs.mtest"
name = "manifest"
version = "0.11.1"
version = "0.11.2"
description = "The static case manifest of mtest: the cases a source file declares, read without building it"
license = "Apache-2.0"
authors = ["mcpp-community"]
Expand Down
1 change: 1 addition & 0 deletions spec/protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ on the spec; this list says which of its clauses mtest's parts answer, and since
| P3.3, places and capabilities | a `case` event states `only` and `except` (the platforms and architectures of `only{}`, `except{}`), `capabilities` (each `capability` the case requires, as the program answered it), and the `reason` of a case it lists skipped; the program skips a case outside its places or lacking a capability, whichever runner starts it | 0.8.0 |
| B2: P4.3 | a selection's `!<id>` lines leave out a case and the cases it declares, and its `?<id>` lines select a case where it is declared and are not `missing` elsewhere; the `program` event states `selection: ["except", "optional"]` | 0.9.0 |
| P5.6 | `step()` writes a `step` event and `exercise()` an `exercise` event of the running case; `e2e::run` writes a step for each command it runs | 0.11.0 |
| P5.6 within a scope | a `step`, `exercise`, `artifact`, `metric` or `fixture` event of a case that runs as the scope of a selected case (P3.4) names no case | 0.11.2 |
| O5 within a case | `time_left()` states what a case has left of its `timeout` and its scopes'; `e2e::run` holds a command to it | 0.11.1 |
| O12 `jitter` | `contention_point()` waits a random time up to `MTEST_JITTER`, the same for the same `MTEST_JITTER_SEED`; a case begins and a shared object is made at one | 0.11.0 |
| O8 `output` | each case's output, kept while it runs under a runner and stated as `output` events; a case declared inside another keeps its own. Where the system cannot rearrange the program's streams (an ok image), no output is kept for a case, the spec's fallback | 0.6.0 |
Expand Down
Loading