Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ jobs:
MTEST_CTS_PROGRAM="$PWD/adapters/script/tests/fixture.py" mcpp test -p conformance
MTEST_CTS_PROGRAM="$PWD/$(bin capi-fixture)" mcpp test -p conformance
MTEST_CTS_PROGRAM="$PWD/$(bin gtest-fixture)" MTEST_CTS_ID_STYLE=gtest mcpp test -p conformance
MTEST_CTS_PROGRAM="$PWD/$(bin doctest-fixture)" MTEST_CTS_ID_STYLE=gtest mcpp test -p conformance
- name: The engine runs mtest's own tests
run: |
set -euo pipefail
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ protocol. This file states the core ideas and rules; the design and its record a
| `manifest` | the static case manifest: the cases a source file declares, read from its text | nothing |
| `engine` | the `mtest` command and its service | `manifest`, `mcpplibs.os`, `mcpplibs.cmdline`, JSON, TOML |
| `conformance` | the protocol's conformance suite | `lib` |
| `adapters/*` | GoogleTest, Bash, Python | the framework adapted, JSON |
| `adapters/*` | GoogleTest, doctest, Bash, Python | the framework adapted, JSON |
| `capi` | `include/mtest.h` for any language with a C FFI | JSON |

## Code
Expand All @@ -42,7 +42,7 @@ protocol. This file states the core ideas and rules; the design and its record a
- Format by [.clang-format](.clang-format) (the same as mcpp's); CI checks it with the pinned
clang-format.
- No macros: modules, `inline constexpr`, `if constexpr`, `std::source_location`. Headers appear
where a library is only a header: GoogleTest's in its adapter (with nlohmann/json's beside it), and
where a library is only a header: GoogleTest's and doctest's in their adapters (with nlohmann/json's beside them), and
the C ABI's `include/mtest.h`.
- Names are readable words; loops nest at most two deep (a helper otherwise).
- Comments and documents are declarative and concise; they carry no local path, user name or
Expand Down
53 changes: 53 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,58 @@
# Changelog

## [0.11.0] - 2026-11-21

What a test system needs to see the cost of the tests it runs (SPEC-017 v2.5; mcpp's test and CI
design of 2026-11-20).

- **A file's first case and expectation instantiate no formatting.** Under GCC a test file of one
case and one expectation took 689 ms to compile: std::format's machinery for `eq()` and
std::function for the case's body were loaded into every file from the standard library's module.
Numbers and truth values are now compared and shown by the library (integers by their values, of
either signedness, as `std::cmp_equal` compares them; NaN is unordered), a case's body is a
reference to its callable (a case runs while it is declared), a case declared in a body keeps its
name and attributes in the library, and a suite of a name and a body that captures nothing passes
a function pointer. The same file takes 265 ms; most of the rest is GCC loading the interface's
classes. `comparison` keeps its fields.
- **The protocol costs a case 0.075 ms.** mcpplibs.os 0.4.7's capture no longer synchronises its
file twice per case: a program of 510 cases took 1.05 to 1.27 s under a runner that keeps each
case's output, and takes 38 ms. Both are now `Bench` cases with bounds (`lib/tests/costs.cpp`).
- **Steps and the surface a case used (P5.6).** `step()` and `exercise()` write the `step` and
`exercise` events; `e2e::run` writes a step for each command it runs, with its status and the
processor time of the programs it ran. The engine keeps a case's `steps`, `exercised`, `cpu_ms`
(its `cpu.user` and `cpu.sys` metrics) and `limit_ms`, and reads a host's `builds.ndjson`.
- **The end of a case checks what its programs did.** `at_case_end()` registers a check the running
case makes before its verdict. The e2e module fails a case whose program left a process holding an
output open, and whose programs printed a line matching a diagnostic pattern of the project
(`e2e::diagnostics`) that the case did not declare (`e2e::expect_diagnostic`), or did not print
one it declared; `MTEST_DIAGNOSTICS=report` prints and counts the lines instead.
- **Golden comparisons with redactions.** `e2e::golden(file, text, rules)` replaces paths,
durations, sizes and digests first; an update prints what it changed, by line.
- **One budget for the machine (O11).** A run makes a directory of one slot file per job; each worker
holds one and states it to its programs (`MTEST_BUDGET`, `MTEST_BUDGET_HELD`), so the parallel
work a program starts (a build's compilers) takes only the slots no worker holds. Inside another
runner's budget the engine uses that one.
- **Jitter (O12).** `mtest run --jitter MS [--jitter-seed N]`; `contention_point()` waits a random
time up to MS, the same for the same seed. A case begins and a shared object is made at one.
- **The machine's class.** `run.json`'s `machine` is the processor's model (mcpplibs.os 0.4.7), else
`<arch>/<processors>`, with `processors` and `memory`; through 0.10.0 it was the host's name.
Estimates prefer what was measured on the same class: hosted runners of one label differ by up to
1.6 times.
- **`[budget]` and `[ledger.*]` are judged by the gate.** `case_cpu` bounds a case's processor time;
a ledger states requirements, the lines of a file or a command's output, each met by a passing
case's clause or used item of the surface, or excused by a waiver with a reason. A waiver that
excuses nothing fails the gate.
- **The coverage map records functions.** A run under `--coverage` keeps, per case, the functions
whose arc counters ran; `mtest coverage spread` gives the median share of cases that execute a
file and a function.
- **The Markdown report** gains the slowest steps, the builds a host recorded, the margin of the
cases near their limit, the budget and the ledgers; the summary names the machine.
- **doctest under the protocol (`mcpplibs.mtest.doctest`, L1).** doctest's implementation and the
test main are compiled once in the package: a test file includes the header and defines no
`DOCTEST_CONFIG_IMPLEMENT*`, which each file compiled again (1.6 s a file under GCC). It passes
the conformance suite.
- mcpplibs.os ^0.4.7.

## [0.10.0] - 2026-11-19

- **A timeout records the program's processes, asks it to end, and then ends it.** Intermittent
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Each part is used alone; together they share one data model.
| C++ library | cases, expectations, fixtures; `bench`, `cts`, `e2e` modules | [lib/](lib/) — `mcpplibs.mtest` |
| Engine | run, plan, explain, history, analyze, graph, gate, trace, bisect, ci | [engine/](engine/) — the `mtest` program |
| Service | the engine's commands as JSON-RPC and MCP tools | `mtest serve` — [docs/service.md](docs/service.md) |
| Adapters | GoogleTest, Bash, Python | [adapters/](adapters/) — [docs/adapters.md](docs/adapters.md) |
| Adapters | GoogleTest, doctest, Bash, Python | [adapters/](adapters/) — [docs/adapters.md](docs/adapters.md) |
| C ABI | the protocol for any language with a C FFI | [capi/](capi/) — `include/mtest.h` |
| Conformance | one suite that checks every implementation of the protocol | [conformance/](conformance/) |

Expand All @@ -38,7 +38,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.10.0"
mcpplibs.mtest = "0.11.0"
```

A failure names its place and both sides (`expected 2, actual 3`). Under a runner the same program
Expand Down Expand Up @@ -92,7 +92,7 @@ Every command takes `--json`, and every answer has a `conclusion`. See
| L2 | metrics | bench verdicts, conformance clauses |
| L3 | resources, artifacts | scheduling, affected selection |

The library, the C ABI and the script adapters are L3; GoogleTest is L1.
The library, the C ABI and the script adapters are L3; GoogleTest and doctest are L1.

## Build

Expand Down
6 changes: 3 additions & 3 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
| C++ 库 | 用例、期望、夹具;`bench`、`cts`、`e2e` 模块 | [lib/](lib/) — `mcpplibs.mtest` |
| 引擎 | run、plan、explain、history、analyze、graph、gate、trace、bisect、ci | [engine/](engine/) — `mtest` 程序 |
| 服务 | 以 JSON-RPC 与 MCP 工具提供引擎的命令 | `mtest serve` — [docs/service.md](docs/service.md) |
| 适配 | GoogleTest、Bash、Python | [adapters/](adapters/) — [docs/adapters.md](docs/adapters.md) |
| 适配 | GoogleTest、doctest、Bash、Python | [adapters/](adapters/) — [docs/adapters.md](docs/adapters.md) |
| C ABI | 供任何有 C FFI 的语言使用的协议 | [capi/](capi/) — `include/mtest.h` |
| 一致性 | 检验协议每一种实现的同一套用例 | [conformance/](conformance/) |

Expand All @@ -37,7 +37,7 @@ int main() {

```toml
[dev-dependencies]
mcpplibs.mtest = "0.10.0"
mcpplibs.mtest = "0.11.0"
```

失败时给出位置与两侧的值(`expected 2, actual 3`)。在运行器之下,同一程序按协议报告;单独运行时每个用例输出一行。
Expand Down Expand Up @@ -83,7 +83,7 @@ mtest serve # 同样的命令,供 Agent(MCP)或 IDE
| L2 | 指标 | bench 判定、一致性条款 |
| L3 | 资源、产物 | 调度、受影响选择 |

库、C ABI 与脚本适配为 L3;GoogleTest 为 L1。
库、C ABI 与脚本适配为 L3;GoogleTest 与 doctest 为 L1。

## 构建

Expand Down
25 changes: 25 additions & 0 deletions adapters/doctest/docs/writing-tests.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Writing tests with doctest under mtest

A program's tests stay doctest's: `TEST_CASE`, `TEST_SUITE`, `CHECK`, `REQUIRE`, `SUBCASE`. The
package `mcpplibs.mtest.doctest` supplies doctest's implementation and the program's `main`, once:

```toml
[dev-dependencies]
mcpplibs.mtest.doctest = "0.11.0"
```

```cpp
#include <doctest/doctest.h>

TEST_SUITE("parse") {
TEST_CASE("an empty line") { CHECK(parse_line("") == 0); }
}
```

- A test file includes the header and defines neither `DOCTEST_CONFIG_IMPLEMENT` nor
`DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN`: the implementation is compiled once, in the package, and not
in each file.
- A case's identifier is `<suite>/<name>` (`parse/an empty line`), or `<name>` outside a
`TEST_SUITE`; it is what `mcpp test --case` and `MTEST_SELECT` name.
- A case skipped by `doctest::skip()` is reported skipped, its `doctest::description` the reason.
- With no runner the program is doctest's own main and takes doctest's options.
24 changes: 24 additions & 0 deletions adapters/doctest/fixture/main.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// The conformance fixture of the doctest adapter (spec/conformance.md), level 1: the cases doctest
// can state, named `<suite>/<name>` (`conformance/passes`), with no tags, kinds or metrics.
// doctest's implementation and the main are the adapter's (src/protocol.cpp, src/main.cpp).
#include <doctest/doctest.h>

TEST_SUITE("conformance") {
TEST_CASE("passes") {}
TEST_CASE("fails") {
CHECK(1 == 2);
}
TEST_CASE("skips" * doctest::skip() * doctest::description("not here")) {}
TEST_CASE("measures") {}
TEST_CASE("covers") {}
TEST_CASE("holds") {}
}
TEST_SUITE("group") {
TEST_CASE("inner") {}
}

int mcpplibs_mtest_doctest_main(int argc, char** argv);

int main(int argc, char** argv) {
return mcpplibs_mtest_doctest_main(argc, argv);
}
46 changes: 46 additions & 0 deletions adapters/doctest/mcpp.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# mcpplibs.mtest.doctest: doctest as an implementation of the mtest protocol, level 1 (spec/protocol.md):
# the cases listed, selected by identifier and reported with their failures' places, by a doctest
# reporter of the package's own. A program's tests stay doctest's (`TEST_CASE`, `CHECK`); the package
# supplies doctest's implementation and `main` once, so a test file includes the header and defines
# neither (no `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN`):
#
# [dev-dependencies]
# mcpplibs.mtest.doctest = "0.11.0"
#
# A case's identifier is `<suite>/<name>` (P3.2), or `<name>` outside a `TEST_SUITE`. doctest's
# macros exist only in its header, so this package's one source includes it, and nlohmann/json's
# beside it, as the GoogleTest adapter does.
[package]
namespace = "mcpplibs.mtest"
name = "doctest"
version = "0.11.0"
description = "doctest speaking the mtest protocol: doctest's implementation and a test main that lists, selects and reports its cases"
license = "Apache-2.0"

[language]
standard = "c++23"
modules = true
import_std = false

[build]
sources = ["src/protocol.cpp"]

[targets.doctest]
kind = "lib"

[targets.main]
kind = "entry"
main = "src/main.cpp"
test-protocol = 1
test-main = true
test-guide = "docs/writing-tests.md"

# The adapter's own conformance fixture (spec/conformance.md), which the conformance suite runs
# (MTEST_CTS_PROGRAM, MTEST_CTS_ID_STYLE=gtest: `<suite>/<name>`).
[targets.doctest-fixture]
kind = "bin"
main = "fixture/main.cpp"

[dependencies]
compat.doctest = "2.4.12"
nlohmann.json = "3.12.0"
7 changes: 7 additions & 0 deletions adapters/doctest/src/main.cpp
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
// The `main` of a doctest program under the mtest protocol: the package's entry, which mcpp begins
// a test file that defines no `main` with (`test-main`).
int mcpplibs_mtest_doctest_main(int argc, char** argv);

int main(int argc, char** argv) {
return mcpplibs_mtest_doctest_main(argc, argv);
}
Loading
Loading