Skip to content

tests: doctest programs speak the mtest protocol (mcpplibs.mtest.doctest) - #8

Open
ZheFeng7110 wants to merge 2 commits into
mcpplibs:mainfrom
ZheFeng7110:test/mtest-doctest
Open

ZheFeng7110 wants to merge 2 commits into
mcpplibs:mainfrom
ZheFeng7110:test/mtest-doctest

Conversation

@ZheFeng7110

Copy link
Copy Markdown
Collaborator

Closes the first items of #7. Upstream mtest 0.11.0 shipped mcpplibs.mtest.doctest, a level-1 adapter that is an entry package (test-protocol = 1, test-main = true). This PR uses it instead of a local adapter. Design record: .agents/docs/2026-11-21-mtest-doctest-adapter-design.md.

Changes

  • The root package and tests/dll take mcpplibs.mtest.doctest = "0.11.1" as a dev-dependency. It depends only on compat.doctest 2.4.12 and nlohmann.json, not on the mtest library, mcpplibs.os or openkal.
  • The 139 doctest files and the two port generators no longer define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN. doctest's implementation and main are compiled once, in the adapter package.
  • mcpp test --cases, --case <id>, --failed, --split case and --message-format json now work on individual doctest cases. Case IDs are <suite>/<name>, or <name> outside a suite.

What had to change for that

mcpp links the adapter into every test, and links its main into every test whose source text defines no main. So the migration has to be done for all files at once, and:

Problem Fix
cobalt/co_main.cpp, test/custom_types.cpp, test/included/{smoke,doc_reference}.cpp get main from a header. mcpp's text scan does not see it, so the entry's main is a second definition Declare int main(int argc, char* argv[]); as the header does
11 files used SIGINT, errno, INT_MAX, FP_ZERO, ::sin, ::uint16_t, which only doctest's implementation had been including port_examples.py adds the C header a case uses (C_NAMES); the generated files have it; three hand-written files include it
asio/custom_types, geometry/api_reference: with GCC 16, CHECK(s.find(..) != std::string::npos) in a TU that only import stds emits npos as a strong symbol, which now conflicts with libstdc++.a(string-inst.o) Use s.contains(..)
Under mcpp, a protocol program reports --list-test-cases to mcpp (the record's listed), not to its output test_coverage.py reads listed. The CI command is unchanged

Local results (Linux x86_64)

  • gcc 16.1.0: all 417 root test programs (136 speak the protocol, 1815 cases) and the 4 tests/dll programs pass. test_coverage.py check passes. With the old script, the new listing reports 1791 missing cases.
  • llvm@22.1.8: spot-checked the 6 most affected tests (json/custom_types, asio/custom_types, geometry/api_reference, cobalt/co_main, test/custom_types, math/examples_2); all pass.
  • Compile time per doctest file (gcc, dev, single job, same command with and without the implementation macro): 1.35–2.09 s saved, 1.55 s on average over 8 files, about 3.6 min of CPU time per gcc leg.
  • A temporary probe (not committed) covered the protocol features:
    • --case matches names containing * and , exactly;
    • --failed reruns only the failed cases;
    • a SIGSEGV is reported against its case, and with --split case the other cases still run;
    • doctest::skip() is reported as skipped;
    • the asio ex_case_names watchdog listener still works alongside the adapter.

Not yet verified

  • The windows llvm-msvc and linux-arm64 (musl) legs: this PR's CI is the first run on them.
  • regen.py output: the generated files were patched with the same rule as the generator, not regenerated (that needs the release archive and the oracle build).

Gaps in the upstream adapter (duplicate case IDs, MTEST_* not cleared, only skip mapped of the P3.3 hints, exception failures located at :0) are listed in the design record §7 and will be reported to mcpp-community/mtest.

…est)

The root package and tests/dll take mcpplibs.mtest.doctest 0.11.1 as a
dev-dependency: doctest's implementation and main are compiled once in the
package, and mcpp test --cases / --case / --failed / --split case act on
each doctest case (mcpplibs#7). The 139 test files and the two port generators drop
DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN (about 1.5 s per file under GCC).

mcpp links the adapter into every test and its main into every test whose
text defines none, so:

- co_main.cpp and the three Boost.Test header-form tests, whose main comes
  from a header, declare it as that header does.
- The ports relied on doctest's implementation for <csignal>, <cerrno>,
  <climits>, <cmath> and <cstdint>; port_examples.py adds the C header a
  case uses (C_NAMES) and the generated files have it; three hand-written
  files include theirs.
- asio/custom_types and geometry/api_reference compare with contains(): a
  CHECK against std::string::npos in a TU that only imports std makes GCC 16
  emit npos as a strong symbol, which now meets libstdc++.a's string-inst.o.

test_coverage.py reads the cases a protocol program lists from mcpp's
record (listed): such a program reports --list-test-cases to mcpp, not to
its output. CI runs the same command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant