Skip to content

doctest adapter: duplicate case identifiers, MTEST_* inherited by children, expected failures reported as pass/fail, no timeout hint, exception failures without a place #29

Open

Description

@ZheFeng7110

Observation

mcpplibs.mtest.doctest 0.11.1 (adapters/doctest/src/protocol.cpp) runs mcpplibs/boost-m's 136 doctest programs (1815 cases) under mcpp test without a failure (mcpplibs/boost-m#8). A probe program found five places where its reports differ from the protocol (mcpp docs/specs/test.md, SPEC-017) or say less than doctest knows:

  1. Two cases can have the same identifier (P3.2). id_of is <suite>/<name>, and doctest accepts two TEST_CASE("duplicate") in one program. --cases lists duplicate twice, --case duplicate runs both, and --split case starts one process for the two (6 parts for 7 cases in the program below). P3.2: "同一程序中的标识互不相同".
  2. MTEST_* reach the processes a case starts (P2.2). mcpplibs_mtest_doctest_main reads MTEST_PROTOCOL, MTEST_REPORT, MTEST_MODE and MTEST_SELECT but does not remove them from the environment. A case sees MTEST_REPORT (the probe below prints it), so a doctest program of the adapter that the case starts appends its events to the parent's report.
  3. Expected failures are reported as pass or fail, not xfail / xpass (P5.3), and are not listed as expected (P3.3). test_case_end maps CurrentTestCaseStats::testCaseSuccess to pass/fail. doctest sets it to true when a should_fail case fails (ShouldHaveFailedAndDid) or an expected_failures(n) case fails n times, and to false when a should_fail case passes (ShouldHaveFailedButDidnt). failure_flags carries each of these reasons. report_query states expect only for skip. The library already writes expect: "fail" and the xfail status (lib/src/mtest.cpp, around lines 948 and 1116).
  4. doctest::timeout(s) is not stated (O5). doctest checks a timeout only after the case returns (TestCaseFailureReason::Timeout), so a case that hangs is ended only by the runner. A case event with timeout would let a runner that starts a process per case end it at that time.
  5. A failure from an exception or a crash has no place. test_case_exception writes "file": "", "line": 0, so mcpp test shows probe/throws :0. doctest's TestCaseException has no location, but the case's own m_file / m_line are known when it starts.

Reproduction

mcpp 2026.11.20.1, gcc 16.1.0, Linux x86_64, mcpplibs.mtest.doctest = "0.11.1" as a dev-dependency:

#include <doctest/doctest.h>
#include <csignal>
#include <cstdio>
#include <cstdlib>

TEST_SUITE("probe") {
    TEST_CASE("throws") { throw 42; }
    TEST_CASE("should fail" * doctest::should_fail()) { CHECK(false); }
    TEST_CASE("timeout" * doctest::timeout(0.5)) {}
    TEST_CASE("crashes") { if (std::getenv("PROBE_CRASH")) std::raise(SIGSEGV); }
    TEST_CASE("env") {
        const char* r = std::getenv("MTEST_REPORT");
        std::fprintf(stderr, "MTEST_REPORT=%s\n", r ? r : "(unset)");
    }
}
TEST_CASE("duplicate") {}
TEST_CASE("duplicate") {}
  • mcpp test probe --cases: duplicate appears twice, and no case states expect or timeout.
  • mcpp test probe --split case: 7 cases in 6 parts, one for both duplicate cases.
  • mcpp test probe: probe/throws :0.
  • PROBE_CRASH=1 mcpp test probe: probe/crashes :0.
  • mcpp test probe --message-format json: probe/should fail has status pass (with its failure), and env prints the report's path.

Proposal

  1. Identifiers are unique within a program. A case whose <suite>/<name> is already taken gets a stable suffix in registration order, for example duplicate#2. The case event keeps name as doctest's name, and the adapter's guide states the rule. Alternatively, the adapter refuses such a program in list mode with a message naming both places.
  2. After reading them, the adapter removes MTEST_PROTOCOL, MTEST_REPORT, MTEST_MODE, MTEST_SELECT and the other protocol variables from its environment (unsetenv, or _putenv_s(name, "") on Windows), before any case runs.
  3. report_query states expect: "fail" for should_fail and expected_failures. test_case_end reports xfail when failure_flags has ShouldHaveFailedAndDid or FailedExactlyNumTimes, and xpass when it has ShouldHaveFailedButDidnt. may_fail stays pass, since it states no expectation.
  4. report_query states timeout from TestCaseData::m_timeout when it is positive.
  5. test_case_exception gives the case's file and line (from the TestCaseData saved in test_case_start).

Items 1, 3 and 4 change only programs that use duplicate names or these decorators. Items 2 and 5 change no passing case.

Also seen (not the adapter's to change)

  • With MTEST_PROTOCOL set, doctest's --list-test-cases reports through the adapter's reporter, so the names go to the report (listed in mcpp's record) and not to standard output. boost-m's coverage check now reads listed. This may be worth a line in adapters/doctest/docs/writing-tests.md.
  • mcpp links the adapter's main into every test whose source text defines no main. A test whose main comes from a header (<boost/cobalt/main.hpp>, <boost/test/included/unit_test.hpp>) therefore ends with two definitions, unless the file also declares int main(int, char*[]);. This may also be worth a line in the guide.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions