Repository navigation
Open
Description
Observation
mcpplibs.mtest.doctest 0.11.1 (adapters/doctest/src/protocol.cpp) runs mcpplibs/boost-m's 136 doctest programs (1815 cases) under mcpp test without a failure (mcpplibs/boost-m#8). A probe program found five places where its reports differ from the protocol (mcpp docs/specs/test.md, SPEC-017) or say less than doctest knows:
- Two cases can have the same identifier (P3.2).
id_ofis<suite>/<name>, and doctest accepts twoTEST_CASE("duplicate")in one program.--caseslistsduplicatetwice,--case duplicateruns both, and--split casestarts one process for the two (6 parts for 7 cases in the program below). P3.2: "同一程序中的标识互不相同". MTEST_*reach the processes a case starts (P2.2).mcpplibs_mtest_doctest_mainreadsMTEST_PROTOCOL,MTEST_REPORT,MTEST_MODEandMTEST_SELECTbut does not remove them from the environment. A case seesMTEST_REPORT(the probe below prints it), so a doctest program of the adapter that the case starts appends its events to the parent's report.- Expected failures are reported as
passorfail, notxfail/xpass(P5.3), and are not listed as expected (P3.3).test_case_endmapsCurrentTestCaseStats::testCaseSuccesstopass/fail. doctest sets it to true when ashould_failcase fails (ShouldHaveFailedAndDid) or anexpected_failures(n)case failsntimes, and to false when ashould_failcase passes (ShouldHaveFailedButDidnt).failure_flagscarries each of these reasons.report_querystatesexpectonly forskip. The library already writesexpect: "fail"and thexfailstatus (lib/src/mtest.cpp, around lines 948 and 1116). doctest::timeout(s)is not stated (O5). doctest checks a timeout only after the case returns (TestCaseFailureReason::Timeout), so a case that hangs is ended only by the runner. Acaseevent withtimeoutwould let a runner that starts a process per case end it at that time.- A failure from an exception or a crash has no place.
test_case_exceptionwrites"file": "","line": 0, somcpp testshowsprobe/throws :0. doctest'sTestCaseExceptionhas no location, but the case's ownm_file/m_lineare known when it starts.
Reproduction
mcpp 2026.11.20.1, gcc 16.1.0, Linux x86_64, mcpplibs.mtest.doctest = "0.11.1" as a dev-dependency:
#include <doctest/doctest.h>
#include <csignal>
#include <cstdio>
#include <cstdlib>
TEST_SUITE("probe") {
TEST_CASE("throws") { throw 42; }
TEST_CASE("should fail" * doctest::should_fail()) { CHECK(false); }
TEST_CASE("timeout" * doctest::timeout(0.5)) {}
TEST_CASE("crashes") { if (std::getenv("PROBE_CRASH")) std::raise(SIGSEGV); }
TEST_CASE("env") {
const char* r = std::getenv("MTEST_REPORT");
std::fprintf(stderr, "MTEST_REPORT=%s\n", r ? r : "(unset)");
}
}
TEST_CASE("duplicate") {}
TEST_CASE("duplicate") {}mcpp test probe --cases:duplicateappears twice, and no case statesexpectortimeout.mcpp test probe --split case: 7 cases in 6 parts, one for bothduplicatecases.mcpp test probe:probe/throws :0.PROBE_CRASH=1 mcpp test probe:probe/crashes :0.mcpp test probe --message-format json:probe/should failhas statuspass(with its failure), andenvprints the report's path.
Proposal
- Identifiers are unique within a program. A case whose
<suite>/<name>is already taken gets a stable suffix in registration order, for exampleduplicate#2. Thecaseevent keepsnameas doctest's name, and the adapter's guide states the rule. Alternatively, the adapter refuses such a program inlistmode with a message naming both places. - After reading them, the adapter removes
MTEST_PROTOCOL,MTEST_REPORT,MTEST_MODE,MTEST_SELECTand the other protocol variables from its environment (unsetenv, or_putenv_s(name, "")on Windows), before any case runs. report_querystatesexpect: "fail"forshould_failandexpected_failures.test_case_endreportsxfailwhenfailure_flagshasShouldHaveFailedAndDidorFailedExactlyNumTimes, andxpasswhen it hasShouldHaveFailedButDidnt.may_failstayspass, since it states no expectation.report_querystatestimeoutfromTestCaseData::m_timeoutwhen it is positive.test_case_exceptiongives the case'sfileandline(from theTestCaseDatasaved intest_case_start).
Items 1, 3 and 4 change only programs that use duplicate names or these decorators. Items 2 and 5 change no passing case.
Also seen (not the adapter's to change)
- With
MTEST_PROTOCOLset, doctest's--list-test-casesreports through the adapter's reporter, so the names go to the report (listedin mcpp's record) and not to standard output. boost-m's coverage check now readslisted. This may be worth a line inadapters/doctest/docs/writing-tests.md. - mcpp links the adapter's
maininto every test whose source text defines nomain. A test whosemaincomes from a header (<boost/cobalt/main.hpp>,<boost/test/included/unit_test.hpp>) therefore ends with two definitions, unless the file also declaresint main(int, char*[]);. This may also be worth a line in the guide.
Metadata
Metadata
Assignees
Labels
No labels
Activity