diff --git a/.agents/docs/2026-11-20-test-architecture-upgrade-design.md b/.agents/docs/2026-11-20-test-architecture-upgrade-design.md new file mode 100644 index 000000000..c4d84fc29 --- /dev/null +++ b/.agents/docs/2026-11-20-test-architecture-upgrade-design.md @@ -0,0 +1,1009 @@ +--- +subject: test +status: active +--- + +# 生态的测试与 CI 方案:mtest 的框架与运行控制、mcpp 自身的测试系统、CI 的形态(mcpp#986、#976、#927、#802) + +**日期:** 2026-11-20 · **版本:** v5(实施于 mcpp #984 等六个仓库,记录在第 14 节。2026-11-20 确定 N1 至 N6,N7 不发出,确定本文为生态级的测试与 CI 方案(1.1),确认评审中补充的 5.16 至 5.22;2026-11-21 确认 N8 至 N10,5.19 不每夜运行,5.22 按树认定而不以合并队列为前提,mcpp 的拉取请求复用 #984) · **基线:** mcpp 2026.11.20.1(`722ec1b6`)、mtest 0.10.0、 +compat.doctest 2.4.12、mcpplibs/boost-m 拉取请求 5(`e7cc105c`) · **前文:** +[CI 与发布的时间成本](2026-11-14-ci-and-release-time-cost-design.md)、 +[生态规则收敛方案](2026-11-19-ecosystem-rule-convergence-design.md)(第 11 节,包的实例)、 +[大型工作区的构建空间与复用](2026-10-31-workspace-space-and-reuse-design.md) · **规范:** SPEC-017(`docs/specs/test.md`) + +## 1. 来源、范围与结论 + +本文处理测试与 CI 的成本。来源有五类:四个 issue、一次 CI 探针、一个下游仓库的 CI、一组框架对照测量,以及对已有机制的 +清点。每一项先核对 `origin/main` 的实现,再判定是否纳入。 + +| 来源 | 判定 | 性质 | 章节 | +|---|---|---|---| +| mcpp#986 | 纳入,范围大于报告 | 包实例的摘要含单元集合;同一类问题另有两处(缓存键含工程路径、含 home 的绝对路径) | 5.1、5.2 | +| mcpp#976 · `trees`、`sets` 超出时限 | 纳入 | 三个因素相乘:机器类别、嵌套并行的争用、平台运行时在每个工程中重新编译 | 3.1、5.2、5.5 | +| mcpp#976 · 选择一个套件的用例时不带套件的时限 | 纳入 | 时限陈述在两处,选择只带其中一处 | 5.8 | +| mcpp#976 · `toolchain/custom` 875 在共享 home 下受干扰 | 纳入 | 共享 home 是未声明的运行夹具 | 5.10 | +| mcpp#976 · 骨架的 mtest 版本 | 纳入 | 版本一致性的检查不含骨架 | 5.14 | +| mcpp#976 · `mingw/182` 的下载用时 | 不纳入 | 下载源的选择属于 xlings;本文只使其成为夹具的一部分(5.10),用例不再各自下载 | — | +| mcpp#927 · 第 2 项(耗时汇总有无名的一段) | 纳入 | 构建在运行记录中是一个整体 | 5.6 | +| mcpp#927 · 第 5 项(升级工具后整个配置重编) | 纳入 | 根工程的产物以修改时间与引擎指纹判定,不以内容寻址 | 5.3 | +| mcpp#927 · 第 4 项(测试程序占配置体积七成) | 纳入 | 程序的大小主要是依赖的调试信息 | 5.15 | +| mcpp#927 · 第 1 项(`programs = "directory"` 对 `tests/` 根无效) | 已有 | `programs = "package"`(2026.11.13.1)与「留下直接位于固定目录的文件时说明其数目」已实现(SPEC-017 B1.2) | — | +| mcpp#802 | 纳入 | 内存准入与处理器预算是同一个准入点 | 5.5 | +| mcpp#293 | 纳入 | 测试的 home 以符号链接指向开发者的载荷 | 5.10 | +| 拉取请求 984(本轮复用它) | 核对后补一处 | 资源脚本的包含目录另有一份推导;修复使其展开 glob,但丢掉了规划时尚不存在的字面目录 | 5.1 | +| 探针(拉取请求 985,三轮) | 纳入 | 见 3.1 | 5.2、5.5、5.9 | +| boost-m 的 CI | 纳入基础设施的部分 | 见 3.3;其自身的用法(两组各测一遍、每个文件一份 doctest 实现)由其维护者决定 | 5.6、5.11、5.13、7 | +| mtest 与 doctest 的对照 | 纳入 | 见 3.4 | 5.12、5.15 | +| 评审中的补充(2026-11-20 确定) | 纳入 | 面的覆盖、预算的门禁、用例结束时的检查、抖动、等待事件、契约性输出、一个提交验证一次 | 5.16 至 5.22 | + +**结论。** + +1. 测试与 CI 的时间主要花在构建上,而测试系统只在用例的粒度上看得见它。mcpp 的 C++ e2e 分片把 2,579 至 7,255 s 的边 + 用时花在平台运行时的编译上;boost-m 的 `mcpp test` 一步有 95% 是构建。 +2. 构建被重复的原因是产物的地址含有不决定其内容的量。已核实三处:包实例的摘要含单元集合(#986);缓存键含构建程序的 + 输出目录,该目录在工程之内;缓存键含 home 的绝对路径。另有一处是缺少寻址:根工程的单元只按修改时间判定。 +3. 嵌套的并行没有共同的预算:四个测试进程各自启动并行构建,4 个处理器上平均 6.8 至 9.4 个进程排队。同一工作的 + 墙钟因此为单独运行的 2.3 倍,时限按墙钟陈述,于是失败。 +4. GitHub 的同一种 runner 是三种处理器,同一组命令的处理器时间相差 1.62 倍。按墙钟陈述的时限与估计在机器之间不可比。 +5. mtest 相对 doctest:断言多的文件编译更快(Clang 上 2 至 4.5 倍),每个文件的首次使用成本在 GCC 上高一个数量级 + (1.1 s 对 0.07 s),链接更慢、程序更大,在协议下每个用例多约 2 ms。四项各有可定位的成因(5.12)。 +6. 一个提交被验证至少两次,而被发布的产物没有被完整验证:CI 测试宿主构建,发布另行构建静态包(5.22)。 +7. 有三类行为今天的测试看不见:没有被任何测试用到的选项与键、通过的用例中未声明的降级提示、没有被固定的次序 + (5.16、5.18、5.19)。 +8. 此前已实施的机制(一次构建、分片、按测量调度、main 的测试构建复用、脚本 e2e 的覆盖图)保留;本文不重复提出, + 并记录其中被测量否定的两项(第 2 节)。 + +**不在范围:** 测试的书写形式(对象名与标识的重复);另行讨论。 + +### 1.1 三个部分 + +本文是生态级的方案:mtest 是各仓库共用的框架,mcpp 的测试系统是它最大的使用者,CI 的形态由二者的能力决定。 + +| 部分 | 内容 | 决定 | +|---|---|---| +| mtest:功能、框架、运行时的控制与配置、数据的粒度 | 协议的步骤、用到的面、构建记录、预算与抖动;机器与用量;时限与预算的门禁;运行夹具;账本;用例结束时的检查;golden;适配器;库自身的成本 | 5.5 至 5.10 的协议与引擎一侧,5.12 至 5.14,5.16 至 5.19,5.21 | +| mcpp 自身的测试系统 | 产物的身份与复用;运行器实现上述协议;测试 kit;按变更选择;测试的规则;以自身的构建与测试为基准 | 5.1 至 5.4,5.5 至 5.11 的运行器一侧,5.15,5.20,第 6 节 | +| CI 的形态 | 档位与树的标记、发布的提升、缓存的写入与传输、分片与预算、夹具的恢复、记录与摘要、下游仓库共用的动作 | 5.22,第 7 节 | + +实施的任务与依赖在第 12 节,自评在第 13 节。 + +## 2. 现状清点 + +下表是 `origin/main` 已有、本文以其为前提的机制。 + +| 机制 | 位置 | 状态 | +|---|---|---| +| 被测程序每个宿主构建一次,各作业取用 | `ci.yml`、`build.yml`、`use-built-mcpp` | 已实施(2026-10-02) | +| 测试构建一次,四个分片运行;按存储测得的用时排序、划分、分片;只重跑失败部分 | `ci-unit.yml`、SPEC-017 O10 | 已实施(2026-11-14) | +| main 的测试构建目录经 artifact 复用,文件修改时间按 blob 恢复(W1) | `mcpp-unit`、`source_mtimes.py` | 已实施(2026-11-15) | +| CI 的测试 profile(`-O0`、无调试信息) | `[profile.ci]` | 已实施 | +| 脚本 e2e 由 mtest 引擎运行;按宿主的历史分片(`mtest ci plan`);每夜以插桩的 mcpp 测得覆盖图,拉取请求按它选择 | `tests/e2e/mtest.toml`、`e2e-coverage.yml`、`e2e-mtest` | 已实施(2026-11-08) | +| 按变更选择:模块导入图、`MCPP_BIN_` 规则、夹具文件 | `mcpp test --since`(docs/08) | 已实施(2026.11.8.1、2026.11.13.1) | +| 运行存储(程序、用例、夹具;`machine`、`monitor` 字段) | SPEC-017 B4 | 已实施;mcpp 写 `machine = ""`、`monitor = "off"` | +| 用量(P10)、运行夹具(P11)、条款矩阵(P6.4)、界限(P6.5) | SPEC-017 O4、O6、O7 | 框架与 mtest 引擎已实现;`mcpp test` 不服务运行夹具,不陈述监测程度 | +| 每次构建一个编译器的 generation,执行器(`__compiler-cap`)从中取槽位 | `detach_codegen.cppm`、`ninja_backend.cppm` | 已实施(2026.11.7.1) | +| 依赖包的全局构建缓存,以包为单位 | `dependency_cache.cpp`、`cache_key.cppm` | 已实施;根工程不进入 | +| 构建引擎的对照基准(mcpp、cmake、xmake) | `bench/`、`bench.yml`(手动) | 已实施 | +| 集合的预算与 `mtest gate`;golden 文件(`MTEST_GOLDEN=update`) | mtest | 已有;没有单个用例的预算、脱敏与更新的摘要;mcpp 的测试没有使用 golden | +| 运行的档位:拉取请求为 `pr`,main 与 `ci:full` 为 `full` | `ci.yml` 的 `changes` | 已实施;只改变 e2e 的集合与一个 macOS 镜像的子集,作业不减少 | + +**此前被测量否定、本文不再提出的两项。** + +- **按文件粒度的覆盖图为 C++ e2e 收窄选择。** 2026-11-14 的记录第 4 节:覆盖图中 `src/` 的文件被执行它的用例覆盖的比例 + 中位数为 99%。本文只把更细的粒度列为一次测量(5.11)。 +- **无条件地按声明的用例划分长套件。** 同一记录 6.3:`sets` 划分为三组后各为 909、1109、1306 s(整体 1375 s),原因是 + 各组重新付出第一个用例填充的共享状态。该记录的结论是缩短这两个套件的途径为共享状态的复用;本文的 5.2 与 5.10 + 即这条途径。 + +## 3. 测量 + +### 3.1 CI 探针(拉取请求 985,2026-11-20,三轮;数据在草稿 release `probe-data`) + +对象是 `e2e/openkal/trees` 的用例 955,在分片中与单独运行两种条件下,各两遍。 + +| 项 | 测量 | +|---|---| +| runner 的处理器 | `ubuntu-24.04`(4 vCPU)出现三种:AMD EPYC 7763、Intel Xeon Platinum 8573C、Intel Xeon 6975P-C | +| 同一组 973 条命令的处理器时间 | EPYC 为 6975P-C 的 1.62 倍(中位数;p10 1.42,p90 1.79);8573C 比 EPYC 快约 15% | +| 分片内的争用 | 四个工作进程各自并行构建;运行队列平均 6.8 至 9.4,峰值 18 | +| 用例 955 的步骤 A(第一次构建) | 单独运行 206 s(EPYC,冷);分片内 478 s(EPYC):墙钟 2.3 倍,处理器时间多 17% | +| 同一 home、新工程的第二遍(单独运行) | 步骤 A 115 s 与 100 s;重新编译的恰是 `x86_64-openkal`、`aarch64-openkal`、`x86_64-windows-gnu` 三个配置。步骤 D(全部命中)7 s | +| 每个配置重新编译的单元 | 约 1,350 个,另加 400 至 500 个(`openkal-musl` 或 `openkal-windows`,与其上的 `llvm-runtime`) | +| 本机的缓存 | `openkal-musl@0.30.0` 的 97 个条目,把输出目录归一后合为 27 个键 | +| 一个分片中平台运行时的边用时 | 2,579 至 7,255 s | + +归档运行的选择另有三个缺陷,均在探针中触发:部分的模式(`e2e/openkal/trees`)在归档运行中选不中任何测试 +(`run_test_archive` 只按程序过滤);`--case 'trees/955*'` 运行整个 `trees`(`selection()` 对已列出用例之下的通配); +`--cases --archive` 以 `no test ... ran on this machine` 结束。 + +### 3.2 mcpp#986 + +报告的最小复现在 2026.11.20.1 上成立。`plan.cppm` 以 `cu.object.generic_string() + flags_text(cu)` 逐单元累加摘要, +设计记录 2026-11-19 第 11.1 节陈述的摘要不含单元集合。在 boost-m 的一条腿上(linux、GCC 16.1.0、A 组),测试步骤的 +进度在 3 分 14 秒到达 1068/1086,重复编译的上限即这一段,约占该步骤 1202 s 的六分之一。 + +### 3.3 boost-m 的 CI(拉取请求 5 的四次运行) + +416 个测试程序,全部使用 doctest,每个文件一个程序,各定义 `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN`。五个宿主各分 A、B +两组,共十条腿,每条腿运行全部 416 个程序。 + +| 项 | 测量 | +|---|---| +| 整次运行墙钟 | 48、53、65、75 分钟 | +| 最慢的腿(44726358869) | linux GCC B 组 45.5 分钟,其中 `mcpp test` 2507 s | +| 测试步骤的构成(linux GCC A 组) | 1202 s = 规划 7 + 构建 1142 + 运行 53 | +| 构建的长尾 | A 组在 3:14 到达 1068/1086,其余 18 条边约 16 分钟;B 组最后 11 条边约 34 分钟;linux LLVM A 组最后 1 条边约 10 分钟 | +| 这段时间的输出 | 无:日志在 16 至 34 分钟内没有一行,CI 不保留构建记录 | +| 同一条腿在四次运行之间 | x86_64 各腿的同一步骤相差 1.4 至 1.9 倍;linux arm64 相差 1.03 至 1.14 倍(四次为同一拉取请求的不同提交) | +| 子工程步骤(`tests/dll`、`tests/test/utf`) | GCC 腿 7 至 21 s;LLVM、macOS、Windows 腿 292 至 618 s | + +长尾对应哪些边没有直接证据。测试文件的大小中位数为 4 KB,超过 100 KB 的有 13 个(最大 500 KB,为合并移植的官方 +示例),与剩余边数相符;这是推断。 + +### 3.4 mtest 0.10.0 与 doctest 2.4.12 + +方法取自 doctest 的 `doc/markdown/benchmarks.md`:三个场景,单线程构建,以工程自己的目标文件的边用时之和为编译成本。 +本机 x86_64,GCC 16.1.0 与 Clang 23.1.3(libc++),`dev`(`-O0 -g`),mcpp 2026.11.20.1;生成脚本与结果在探针的 +工作目录,不入库。doctest 的文件以 `#include` 取得头文件,mtest 的文件以 `import mcpplibs.mtest;`。 + +**场景一:200 个各含一个函数的文件。** + +| 每个文件增加的编译时间 | GCC doctest | GCC mtest | Clang doctest | Clang mtest | +|---|---|---|---|---| +| 只包含或导入 | 39 ms | 26 ms | 256 ms | 4 ms | +| 另声明一个用例、写一条断言 | 71 ms | 1,111 ms | 251 ms | 270 ms | +| 运行器的实现(一次) | 2.3 s | 无(库的编译 9 至 16 s,见下) | 1.2 s | 无(库的编译 4 至 11 s) | + +**场景二:10 个文件,各 50 个用例,各 100 条断言(共 5 万条)。** + +| 编译时间(s,减去基线前) | GCC | Clang | +|---|---|---| +| doctest `CHECK(a == b)` | 40.7 | 33.7 | +| doctest `CHECK(a == b)`,`SUPER_FAST_ASSERTS` | 15.2 | 16.3 | +| doctest `CHECK_EQ(a, b)` | 72.1 | 14.7 | +| doctest `CHECK_EQ(a, b)`,`SUPER_FAST_ASSERTS` | 11.0 | 4.6 | +| mtest `expect(eq(a, b))` | 22.6 | 7.6 | +| mtest `expect(a == b)` | 13.4 | 4.6 | + +**场景三:一个用例中 1000 万条通过的断言(处理器时间,s)。** + +| | GCC | Clang | +|---|---|---| +| doctest `CHECK(i == i)` | 0.67 | 0.79 | +| doctest `INFO(i); CHECK(i == i)` | 0.89 | 2.31 | +| mtest `expect(eq(i, i))` | 2.14 | 2.35 | +| mtest `expect(eq(i, i)) << i` | 2.15 | 2.43 | + +**其余四项。** + +| 项 | doctest | mtest | +|---|---|---| +| 一个测试程序的链接(GCC / Clang) | 0.34 s / 0.06 s | 0.92 s / 0.21 s | +| 一个测试程序的大小(GCC / Clang,`dev`) | 4.1 MB / 1.4 MB | 11.6 MB / 7.2 MB | +| 协议下 510 个用例、5 万条断言的程序 | 不适用(一个程序一个结论) | 独立运行 0.03 s;协议下 1.05 至 1.27 s,其中处理器时间 0.1 s:每个用例建两个文件并 `fsync`(共 4,044 次) | +| 库在每个工程中 | 仅头文件 | 重新编译:`mcpplibs.mtest` 的目标文件单线程 4 至 16 s,其依赖均命中缓存 | + +**`release`(`-O2`,GCC 16.1.0;Clang 未测)。** + +| 项 | doctest | mtest | +|---|---|---| +| 每个文件:只包含或导入 / 另声明一个用例、写一条断言 | 16 ms / 37 ms | 37 ms / 1,600 ms | +| 5 万条断言:缺省形式 | `CHECK` 175.8 s;`CHECK_EQ` 114.6 s | `expect(eq)` 82.5 s;`expect(a == b)` 19.3 s | +| 5 万条断言:doctest 的快速形式 | 48.9 s 与 12.2 s | — | +| 1000 万条通过的断言(处理器时间) | 0.30 s | 0.14 s | +| 库的编译(每个工程,单线程) | 无 | 16 至 33 s | + +**读法。** + +- doctest 的成本集中在实现的那一份:只在一个文件中实现时,其余文件在 GCC 上每个 39 至 71 ms。boost-m 的 416 个文件 + 各实现一次。Clang 上每个文件的 0.25 s 不随断言增加;推断来自 doctest 的头文件包含的标准库头文件与标准库模块并存, + 未单独测量。 +- mtest 的导入几乎不花时间,成本在首次使用:GCC 上声明第一个用例约 0.3 s,第一条 `eq` 再约 0.3 s(单独编译实测), + 在场景一中每个文件 1.1 s。断言本身比 doctest 的缺省宏便宜(GCC 上 22.6 s 对 40.7 s),在 GCC 上仍高于 doctest 的 + 两种快速形式(15.2 s 与 11.0 s)。 +- mtest 的程序更大、链接更慢,来自调试信息:11.5 MB 中调试段 8.5 MB,代码 1.1 MB,其中 `nlohmann.json` 0.13 MB、 + `mcpplibs.os` 与 openkal 0.09 MB(5.15)。 +- mtest 库不由缓存提供:新的构建目录中,2026.11.17.2、2026.11.18.1 与 2026.11.20.1 都重新编译它(mtest 0.9.0 与 + 0.10.0;并行构建约 6 s),也不写入条目;它的依赖 `mcpplibs.os`、`openkal`、`openkal-linux`、`nlohmann.json` 均显示 + `Cached`。缓存中有 `mcpplibs.mtest@0.9.0` 的一个条目,由别的依赖图写入。mcpp 不说明原因。 + +### 3.5 mcpp 的一次 main 运行(44691464513) + +76 个作业,runner 时间 649 分钟。本提交的 mcpp 之外,各作业另以不同的工具链或目标从源码构建 mcpp: +`bootstrap` 1152 s 与 755 s、容器中的 e2e 1408 s、交叉到 `aarch64-linux-musl` 661 s、Windows 宿主交叉到 Linux +2091 s、musl-gcc 657 s、三个 LLVM 自举 310 至 347 s、macOS 自举 389 s。这些构建各自从零开始。四个单元分片中 +三个为 14.5、17.5、27.4 分钟,macOS 的单元步骤 2142 与 2261 s,Windows 1421 s。 + +### 3.6 测试的写法与合入、发布的现状 + +| 项 | 测量 | +|---|---| +| C++ 测试中的 `sleep_for` | 77 处 | +| C++ e2e 中的 `contains(` 与 `e2e::golden` | 2,302 处与 0 处 | +| 一次合入的全量运行 | 至少两次:拉取请求上一次,main 的推送一次 | +| 最近一周每天合入的拉取请求 | 1 至 4 个 | +| CI 测试的程序与发布的程序 | 前者为 `mcpp build --strict` 的宿主构建,后者为 `release.yml` 另行构建的 `x86_64-linux-musl` 静态包 | +| 发布的构建 | 2026.11.20.1 约 30 分钟 | +| main 的规则集(`main-branch-protect`) | 禁止删除与强推、线性历史、签名、拉取请求须两个批准、只允许 squash 与 rebase,两个可绕过的身份;没有必需的状态检查 | + +## 4. 成因 + +| 编号 | 成因 | 表现 | +|---|---|---| +| C1 | 产物的地址含有不决定其内容的量,或缺少由内容决定的地址 | #986;平台运行时在每个工程中重编;键含 home 的绝对路径;mtest 库不进缓存而无提示;每个作业从零构建 mcpp;升级工具后全部重编(#927) | +| C2 | 嵌套的并行没有共同的预算 | 分片内争用使墙钟为 2.3 倍;内存耗尽(#802) | +| C3 | 构建在运行记录中是一个整体 | 16 至 34 分钟无输出;长尾不可定位;汇总有无名的一段(#927);按用时的调度不含构建 | +| C4 | 时限陈述在两处,并以墙钟对所有机器陈述一次 | 套件 40 分钟、kit 的每条命令 10 分钟;选择丢掉套件的时限(#976);同一工作在两类机器上相差 1.6 倍 | +| C5 | 共享状态是隐式的 | 第一个用例付出填充 home 的成本;修改 home 的用例独占机器;875 受干扰;#293 | +| C6 | 框架的固定成本 | mtest 的首次使用、每个用例的 `fsync`;依赖的调试信息进入每个测试程序;doctest 的实现每个文件一份(用法) | +| C7 | 同一个提交被验证多次,被发布的却没有被验证 | 两次全量运行;发布另行构建(3.6) | +| C8 | 测试看不见的行为 | 没有被任何测试用到的选项与键;通过的用例中未声明的降级提示;没有被固定的次序 | + +## 5. 决定 + +每项陈述决定、机制、它取代或关闭的事项,以及验收。验收的数字来自第 3 节;标为「推算」者在实施中以测量替换。 + +### 5.1 包实例的摘要是包的配置的函数(mcpp;关闭 #986) + +**决定。** 一个包的实例由它的配置决定:声明的标志、宏、包含目录、按 glob 的规则、生效的特性、语言标准,以及其依赖的 +实例摘要。实例目录的摘要与缓存键的配置部分由同一个函数产生。单元集合与单元的输出路径不进入。 + +**机制。** `plan.cppm` 中逐单元累加 `cu.object + flags_text(cu)` 的一段改为读取包的配置;`cache_key.cppm` 的 +`fill_package_config` 是这个函数的现有形态,二者合一。按 glob 的规则以规则本身进入,而不是以它匹配到的单元。 + +**后果。** `mcpp build` 与 `mcpp test` 在特性相同时共用一个实例;测试计划以 `[test] features` 打开特性时配置不同, +实例不同,这是应有的区分。模块映射文件的名字含实例目录,随之稳定。`clean --stale` 与 `clean --report` 读取的 +「每个配置一个目录」不变。 + +**包含目录是配置的一部分,只有一份推导(拉取请求 984)。** 一个包展开后的包含目录现有三份推导:依赖包的编译 +(`package_root.cpp`:展开 glob,丢掉不存在的字面目录)、根清单的编译(`plan.cppm` 的 +`expand_manifest_include_entry`:展开 glob,保留不存在的字面目录,因为它可能由稍后的构建步骤建立,#390)、资源脚本 +(`windows_resources.cpp`)。#984 之前第三份不展开 glob,`*/include` 原样到达 `llvm-windres`,在 Windows 上失败; +#984 改为调用 `expand_dir_glob`,方向正确,CI 全部通过。核对代码后有一处:它没有第二份的回退,规划时尚不存在的 +字面目录此前被原样拼接,现在被丢掉,资源脚本包含其中生成的头文件时首次构建找不到、第二次构建找到。这一点来自 +读代码,没有运行复现。本轮把资源脚本改为调用 `expand_manifest_include_entry`,并在用例 837 中增加「稍后建立的字面 +目录仍在 `rc_object` 的边上」;三份推导随配置函数合为一份。 + +**验收。** #986 的最小复现:`mcpp build` 之后 `mcpp test` 只编译 `src/b.cppm` 与 `tests/t.cpp`。SPEC-004 §9 第 17 项 +写明摘要的输入。 + +### 5.2 路径按角色进入键;可缓存的包的构建程序输出在 home 中按身份寻址(mcpp) + +**决定。** 键中的路径以其角色出现:``(每一个存储根)、``、``(构建程序的输出)。绝对路径不进入键。 +来源被命名的包(索引的版本、git 的提交、路径依赖的内容摘要,#974),其构建程序的输出目录位于 +`/build-cache/v1/out/<包>@<版本>/<键>`,键为该包不含输出目录一项的配置键。 + +**成因的细节。** `state.programOutDir` 把输出目录放在 `<工程>/target/.build-mcpp/out//<包>`;构建程序以 +`mcpp:include-dir=` 声明的目录进入包的键,`relativize` 只认识 `` 与 ``。`openkal-musl`(openkal +目标)与 `openkal-windows`(windows 目标)因此每个工程一个键,其上的 `llvm-runtime` 经 `upstream` 一项随之改变。 +另一处:`nlohmann.json` 的包含目录以 `/home/<用户>/.mcpp/registry/data/xpkgs/...` 进入 `mcpplibs.mtest` 的键 +(`mcpp cache info`),`relativize` 只替换第一个存储根。 + +**不采用「在键中抹去路径」。** 构建程序收到 `MCPP_OUT_DIR`,可以把它写进生成的文件;输出的位置因此是内容的一部分。 +把位置固定为身份的函数,内容与地址才一致。输出按 2026-11-19 记录的事务方式发布:两个工程同时运行同一个构建程序时, +先完成者发布,后完成者核对后采用。 + +**验收。** 探针的第二遍:同一 home 的新工程中,三个配置的步骤 A 不编译平台运行时的任何单元。预期用例 955 的步骤 A +由 478 s 降到与步骤 D(7 s)同一量级加打包的时间(推算)。新增 e2e:两个工程依次构建同一 openkal 目标,第二个工程的 +编译边数为它自己的单元数。 + +### 5.3 根工程与未命名来源的单元按内容寻址(mcpp;取代 W1 的修改时间恢复;关闭 #927 第 5 项) + +**决定。** 执行器在运行编译器之前计算这次编译的键,命中时从缓存发布输出,不运行编译器。键的输入是决定输出的全部: +工具链的身份、归一后的命令、源文件的内容、它读取的每个文件的内容、它导入的每个 BMI 的内容。引擎的版本不进入。 + +**机制。** + +- 位置是 `__compiler-cap`:它包住有上限的图中的每一条编译命令(2026.11.7.1 起),持有输出的发布(2026-11-19 记录 + 10.6)。本项使每个图都经由它。 +- 读取的文件在编译之前未知,采用清单的方式:第一级键(命令、源文件内容)指向若干份清单,每份清单是一次编译实际读取 + 的文件及其摘要(来自依赖文件;MSVC 来自 `/sourceDependencies`);清单中的文件都未变时,其结果即命中。 +- 输出为目标文件、BMI、依赖文件与拆分的调试信息。GCC 的 BMI 以模块映射的拼写记录其导入的路径(BMI 目录与包的 + 实例),随 5.1 稳定;Clang 与 MSVC 的 BMI 在验收中各自核对。 +- 源码根的绝对路径进入键,除非工具链的行陈述其输出可重定位。CI 的检出路径不变,命中不受影响;本机的两个工作树之间 + 不共享,这是正确的结果。 +- 条目位于 `/build-cache/v1/unit/`,由 `mcpp cache gc` 按最近使用回收。CI 中只有 main 的推送运行写入(R3,5.22:它登记一次已通过的 + 全量运行的条目,或自己运行全量),拉取请求的运行只读;经 artifact 传输:仓库的 Actions 缓存已用 9.98 GB(2026-11-14 记录 6.1)。 +- ninja 仍决定哪些边要运行;新的检出使全部边变脏,执行器逐条命中并以 `restat` 结束。链接不缓存。 +- 文件的摘要在一次构建中按(路径、大小、修改时间)记忆,一个头文件只计算一次。 +- 修剪按使用:全量运行在上传之前去掉本次没有读取的条目,传输的大小随当前的树,不随历史。 + +**取代。** W1 的 `source_mtimes.py` 与 `.source-blobs`;`bootstrap-mcpp` 恢复 `target/` 后仍运行 830 条边中的 830 条; +第 3.5 节中各作业从零构建 mcpp;工具升级后的全部重编(指纹目录改变,条目命中)。 + +**验收。** 在一个已填充缓存的 home 上,新检出的无改动构建不运行编译器,其用时记入本文(现为 11 分 37 秒);改动一个接口模块时,运行的编译器数为其 +导入闭包的单元数。逐字节核对:同一键的两次冷编译产出相同的目标文件与 BMI(GCC、Clang、MSVC 各自测量;不相同的 +工具链在其行中标为不可缓存,并以 `diag::degraded` 说明)。 + +### 5.4 缓存的每一次拒绝都有原因(mcpp;mtest 库不由缓存提供) + +**决定。** 一个包或一个单元未由缓存提供时,构建记录写出原因标记,`--verbose` 打印它。`serve_or_fill` 现有的五条返回 +路径(无身份、来源未命名、受未命名来源影响、不在存储之内、有不可寻址的单元)各一个标记。 + +**mtest 库。** `mcpplibs.mtest` 作为根工程的 dev 依赖不由缓存提供,也不被写入(3.4)。成因未定:它落在上述五条路径 +之一,而这五条路径今天不可区分,这正是本项要改变的。成因在实施中以原因标记确定,并按成因修复。每个使用 mtest 的 +工程、每个 e2e 用例建立的工程都付出这次编译。 + +**验收。** 新工程第二次构建时 `mcpplibs.mtest` 显示为 `Cached`;新增 e2e 断言 dev 依赖由缓存提供。 + +### 5.5 一台机器一份预算:处理器槽位与内存(mcpp;SPEC-017 新增 O11;关闭 #802) + +**决定。** 运行测试的 mcpp 持有机器的预算。它的工作进程与这些进程启动的每一次构建从同一份预算取槽位。预算另带内存的 +阈值:可用内存低于阈值时不发放新的槽位,已运行的编译器不被结束。 + +**机制。** + +- 预算是一个目录,其中有与槽位数相同的文件;持有一个文件的锁即持有一个槽位。运行器建立它并在环境中陈述其路径 + (SPEC-017 O11 `budget`)。任何程序都能遵守:`mcpplibs.os` 的文件锁,或脚本中的 `flock`。不认识它的程序不受影响。 +- 一个测试进程持有一个槽位;它启动的构建以这个槽位为第一个编译器,其余向预算申请。不构建的测试以自己的槽位运行。 +- mcpp 的执行器在环境陈述了预算时,启动编译器之前先取预算的槽位,再取本次构建的 generation 的槽位;没有陈述时 + 只取后者(现状)。 +- 内存的准入在同一处:执行器在取得槽位后、启动编译器前读取可用内存(`mcpplibs.os.machine`)。阈值是数据 + (`[build] memory_threshold`,四层取值)。`bmi_schedule = "on"` 的分离代码生成经同一执行器,因此被覆盖。 +- mtest 引擎运行脚本 e2e 时同样建立并陈述预算。 + +**验收。** 探针的分片条件重跑:运行队列的平均值不超过处理器数加一;用例 955 的步骤 A 在分片内的墙钟不超过单独运行的 +1.3 倍(现为 2.3 倍)。分片的总墙钟不增加。 + +### 5.6 构建是运行图的节点(mcpp、SPEC-017 B4;关闭 #927 第 2 项) + +**决定。** 运行存储新增 `runs//builds.ndjson`:每条运行过的边一行(输出、所属的包与测试程序、开始、用时、 +处理器时间、最大驻留集);由缓存提供的边按包汇总为一行。`mcpp test` 与 `mcpp build` 的汇总行列出各阶段,并列出最慢的若干条边。 +一条边运行超过一分钟而无输出时,进度行写出正在运行的边及其已用时间。 + +**机制。** ninja 的日志已有每条边的开始与结束,`progress::Attribution` 已把边归属到包;处理器时间与最大驻留集由 +执行器在回收编译器时取得。本项把它们写出。 +P12.3 规定的关键路径由此可以下到边。按测量的调度(O10)在排序测试程序时计入其构建的用时。 + +**验收。** boost-m 的一条腿重跑后,存储回答第 3.3 节的问题:长尾是哪些边、各用多久。`mcpp test -p` 的汇总各段之和 +与总时间之差小于 1 s。 + +### 5.7 步骤是用例的一部分(SPEC-017 P5、mtest、测试 kit) + +**决定。** 协议新增 `step` 事件:属于一个用例,带名字、开始、用时、处理器时间、退出状态。测试 kit 为它运行的每条 +mcpp 命令写一个步骤;该命令是一次构建时,步骤带上那次构建的记录标识。 + +**后果。** 用例 955 的四个步骤各自可见;`mtest analyze path` 从运行下到用例、步骤、边。探针三轮得到的分解成为一次查询。 + +### 5.8 时限只陈述一次,按处理器时间比较(mcpp、mtest;关闭 #976 的一项) + +**决定。** + +- 用例或套件声明的 `timeout` 是唯一的时限。选择一个套件之下的用例时,运行器取它所属的各级作用域声明的时限 + (`--case 'sets/*'` 与 `--failed` 现取运行的缺省 300 s 或 `--timeout`)。 +- kit 的每条命令缺省 10 分钟的时限改为取用例剩余的时间。 +- 时限防止挂起。性能以指标判定:每个用例的处理器时间进入记录(5.9),界限与趋势按它陈述(P6.5),不按墙钟。 +- 报告列出余量:最近若干次运行中用时超过时限五分之四的用例,连同机器的类别。 + +**验收。** #976 的重现:`--case 'sets/*'` 在套件的 30 分钟内运行。 + +### 5.9 机器与用量进入记录(mcpp) + +**决定。** `run.json` 的 `machine` 写出处理器型号、核数、内存(型号取自 `mcpplibs.os.machine` 新增的接口,任务 O1;没有回答的系统上机器的类别取架构与核数);`mcpp test` 缺省以 `basic` 陈述监测程度,并按 P10.4 +从自己的进程测量不写 `usage` 的程序。按测量的调度与分片在估计用时时按机器的类别取历史;没有该类别的历史时取全部。 + +**验收。** 同一测试在两类机器上的估计分开;3.1 的 1.62 倍在报告中表现为类别的差异,而不是回归。 + +### 5.10 共享的 home 是运行夹具(mcpp 实现 O6;测试 kit;关闭 #293,处理 #976 的 875) + +**决定。** `mcpp test` 实现运行夹具(P11):以 `MTEST_SERVE` 服务,把端点交给使用它的程序。测试 kit 把现在隐式共享的 +home 声明为运行夹具:其内容(工具链、平台包、索引)是声明,摘要是它的身份。 + +- 夹具的准备用时记在夹具上(`fixtures.ndjson`),不记在碰巧第一个运行的用例上。 +- 用例陈述它对 home 的使用:只读,或修改。只读的用例共用一个;修改的用例得到自己的 home,其载荷以只读的存储根 + 提供,而不是指向共享载荷的符号链接。`home::shared` 之下以 `machine` 独占来避免干扰的用例改为后者。 +- CI 以夹具的身份为键恢复它;5.2 使其中的构建缓存对每个工程有效。 + +**验收。** 875 与并行的用例同时运行而不受干扰;`sets`、`trees` 的第一个用例不含填充的时间;#293 的复现不改写 +开发者的载荷。 + +### 5.11 按变更选择的两处补充(mcpp) + +- **头文件。** `--since` 现把任何头文件的变更视为影响全部测试。改为:构建目录或存储中有基线提交的依赖记录时, + 头文件影响依赖记录列出它的单元及其导入者;没有记录时保持现状。记录是 5.3 的清单(每个单元实际读取的文件), + 随 main 登记的单元缓存到达拉取请求。对头文件构成的库(boost-m)这是按变更选择能否生效的条件。 +- **C++ e2e。** 文件粒度已被否定(第 2 节)。本轮测量函数粒度:覆盖图记录每个用例执行的函数,统计一个函数被多少 + 用例执行。中位数低于 50% 时实施(变更的函数与用例执行的函数相交),否则把测量记入本文并不实施。 + +### 5.12 mtest 库的四项成本(mtest;其中一项归 mcpp) + +| 项 | 成因 | 决定 | 验收(第 3.4 节的方法) | +|---|---|---|---| +| 每个文件的首次使用(GCC 1.1 s) | 接口中的模板与标准库类型在每个导入者中实例化 | 用例的声明与 `expect` 的常见形式(整数、字符串、布尔)在库中有非模板的入口;接口不要求导入者实例化 `std::function`、流与格式化 | GCC 上每个文件不超过 150 ms | +| 链接与程序大小 | 依赖与运行时库的调试信息(8.5 MB 中的大部分);JSON 库的代码只有 0.13 MB | 不在库中处理:库继续依赖 JSON 库(mtest 的规则:已有的库被依赖,不重写);由 5.15 处理 | 见 5.15 | +| 协议下每个用例约 2 ms | 每个用例为输出归属建两个文件并 `fsync` | 输出经由一个按用例标记的通道;不 `fsync`(进程崩溃不丢失已写入的数据) | 510 个用例的程序在协议下不超过 0.1 s | +| 每个工程重新编译库 | mcpp 不由缓存提供它(5.4) | — | — | + +断言的运行成本(每条 0.2 µs)不处理。 + +### 5.13 需求、用例与理由的账本(mtest) + +**决定。** `mtest.toml` 的 `[ledger.<名>]` 陈述一组需求的来源(一个文件,或一条产生列表的命令)。用例以 `clause{}` +陈述它覆盖的需求。每条需求必须有通过的用例,或一条带理由的豁免;豁免的数目只减不增。`mtest gate` 判定, +`mtest report` 生成表。 + +需求由产品给出、证据取自记录的形式见 5.16。 + +**取代三份手工维护的账本。** mcpp 的 `tests/e2e/coverage-debt.tsv` 与 `coverage-exceptions.tsv`;`docs/specs/` 中 +每条条款的实现状态;boost-m 的 `coverage.toml` 与 508 行的 `test_coverage.py`。 + +### 5.14 其余 + +- **doctest 适配器(mtest)。** 与 GoogleTest 适配器同级(1 级):一个 doctest 的报告器与入口,使现有的 doctest 程序 + 列出、选择并报告用例。测试文件去掉 `DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN` 一行,其余不变:实现与入口由适配器提供 + 一次,每个文件各实现一次的成本(GCC 上约 1.6 s)随之消失。 +- **归档运行的选择(mcpp)。** 3.1 的三个缺陷:归档运行接受部分的模式;已列出用例之下的通配选中匹配的用例; + `--cases --archive` 列出而不报错。 +- **骨架的 mtest 版本(mcpp)。** `kMtestRelease` 进入 `check_version_pins.sh`。 + +### 5.15 依赖的调试信息是 profile 的数据(mcpp;mcpp#927 第 4 项) + +**测量。** 一个 mtest 的测试程序(GCC、`dev`)11.5 MB,其中调试段 8.5 MB(`.debug_info` 6.0),代码 1.1 MB。按符号归属的代码: +标准库与运行时 0.74 MB,mtest 0.14 MB,`nlohmann.json` 0.13 MB,`mcpplibs.os` 与 openkal 0.09 MB。链接已带 +`--gc-sections`。程序的大小与链接用时因此来自依赖与运行时库的调试信息,不来自 JSON 库的代码。#927 的工作区中, +349 个测试程序占 22 GB 中的 15.4 GB,去掉调试信息并剥离后一个程序由 24.5 MB 降为 4.3 MB。 + +**决定。** profile 按包的类别陈述调试信息的程度:工作区的成员一类,来自存储与 git 的依赖一类。`dev` 之下成员保持完整的 +调试信息,依赖缺省为 `debug = "lines"`(已有的取值:只带行表,回溯仍有文件与行号);两者都可在 profile 中覆盖。 +陈述依赖一类的键的拼写与缺省的程度在 SPEC-004 中陈述(N8)。 + +**机制。** 标志是工具链行的数据(GCC `-g1`,Clang `-gline-tables-only`,MSVC 的对应项);它进入包的配置(5.1), +依赖因此在缓存中另有条目。工具链自带的静态运行时库中的调试信息不由编译标志控制,其份额在实施的第一步测量。 + +**验收。** 3.4 的方法重测:程序的大小、链接用时与依赖的编译用时记入本文;#927 的工作区重测测试程序的总大小。 + +### 5.16 产品面的覆盖(mtest 的账本,mcpp 给出面与记录;扩充 5.13) + +**决定。** 账本的需求可以由产品自己给出,证据可以取自运行的记录,而不只取自用例手写的 `clause{}`。 + +- **面。** `mcpp emit surface` 写出 mcpp 的面:每个子命令、每个选项、清单的每个键、每个原因标记,一行一项 + (`command build`、`flag build --target`、`key build.sources`、`reason build/cache`)。命令与选项取自命令行的 + 定义,键取自 `mcpp.manifest`,原因标记取自 `check_reason_tokens.sh` 所核对的那些;本项只是写出。 +- **证据。** 协议新增 `exercise` 事件:属于一个用例,列出它用到的项。测试 kit 为它运行的每条 mcpp 命令写出命令与选项; + 一次调用读取的键与发出的原因标记由 mcpp 写入该次调用的记录(环境变量指明文件),kit 转为 `exercise`。 +- **判定。** 面中没有任何用例用到的项使门禁失败,除非账本中有带理由的豁免;指向已被用到的项的豁免同样失败。 + 不设百分比的门槛。 + +**取代。** `check_e2e_coverage.py` 之外没有检查回答「哪个选项、哪个键从未被测试用到」;AGENTS.md 的「未知的键被报告」 +与本项互为两面。 + +### 5.17 预算是门禁(mtest;mcpp 的分片) + +**决定。** + +- `mtest.toml` 的 `[budget]` 陈述一个用例的处理器时间上限与一个作业的墙钟上限。超过用例上限的用例使 `mtest gate` + 失败,除非有带理由的豁免(5.13 的形式)。处理器时间取 5.9 的记录,按机器的类别比较。 +- 作业的上限决定分片数:`mtest ci plan` 已按宿主的历史为脚本 e2e 计算分片;`mcpp test --shard` 的 C++ 测试改为同一 + 来源(现为固定的四片)。 + +**验收。** 用例 955 与其余超过上限的用例各有一条理由;一个新加入而未豁免的超限用例使门禁失败。 + +### 5.18 用例结束时的检查(mtest 的 e2e 模块;测试 kit) + +**决定。** e2e 的用例结束时检查两件事,任一不满足即失败。 + +- **进程。** 用例启动的进程都已结束。`mcpplibs.os` 的进程单元已能回答一个单元是否为空。 +- **诊断行。** 被运行的程序的输出中,匹配工程声明的诊断模式的行,必须由用例声明;用例声明而没有出现的行同样失败。 + mcpp 的 kit 声明的模式是 `warning:`、`error:` 与降级的提示。 + +**依据。** AGENTS.md:做得少不能静默。今天一个用例通过时,mcpp 在其中打印的降级提示不被任何检查看见。 + +**迁移。** 现有用例中合法出现的诊断行在本轮逐个声明;其数目在实施中以报告模式测得。 + +### 5.19 抖动(SPEC-017 新增 O12;mtest、mcpp 的钩子) + +**决定。** 运行器可以陈述抖动的上限(毫秒)与种子。框架在声明的竞争点、kit 在每条命令的前后、mcpp 在取锁与发布处, +各等待一段不超过上限的随机时间。它在三种场合运行,不每夜运行: + +- **结论翻转时。** 存储中一个用例的结论在同一棵树上发生改变(重试后通过,或两次运行不同)时,CI 以若干个种子带抖动 + 重跑这一个用例,把结果与种子写入摘要。 +- **每周一次**,在 main 上运行全部测试。 +- **手动。** + +**依据。** 偶发失败的来源是没有被固定的次序。这类缺陷只在并发相关的代码改动时新增,每夜扫描在绝大多数夜里没有发现; +已有的证据是 2026-11-15 并行暴露的两个锁的缺陷(#976 的四个偶发失败中,两个与次序无关,一个由 5.10 解决)。 +能力本身很小,定位一个翻转的用例时才用得上它。 + +**边界。** 抖动不进入拉取请求的运行与全量的验证。mcpp 的钩子是一个环境变量与一处等待,位于取锁与发布所经过的 +一个函数中。 + +### 5.20 等待事件,不等待时间(mcpp 的测试) + +**决定。** 测试只等待被测程序给出的事件(一行输出、一个文件、退出)。`sleep_for` 与轮询不出现在新的测试中;现有的 +77 处记入基线文件(`.github/tools/` 中已有两份同类的基线),基线只减不增。时间本身是断言的用例在基线中注明。 +测试需要而产品没有的事件,加到产品中。 + +### 5.21 契约性的输出整段比较(mtest 的 golden;mcpp 只用于契约) + +**决定。** `e2e::golden` 增加脱敏的规则(路径、用时、大小、摘要)与更新时的摘要(哪些块改变、新增、移除)。mcpp 的 +测试把它用于其格式本身是契约的输出:机器可读的记录(docs/50)与带原因标记的拒绝。现有的 2,302 处 `contains(` 不迁移。 + +### 5.22 一棵树验证一次,被使用的就是被验证的(CI;发布规程) + +**现状。** 一次合入至少有两次全量运行:拉取请求上一次,合入后 main 上一次(发布提交的那一次现在被手动取消)。CI 测试的是 +`mcpp build --strict` 的宿主构建;发布的是 `release.yml` 另行构建的 `mcpp pack --target x86_64-linux-musl` 静态包, +它在发布前只经过冒烟,完整的验证在发布之后(`ci-fresh-install`)。main 的规则集没有必需的状态检查:CI 的结论由 +负责人查看后合入(3.6)。 + +**决定。** 验证的对象是源码树,不是提交,也不是推送。 + +- **标记。** 一次全量运行通过后,留下以树的标识命名的标记,连同它的归档、缓存条目与记录。任何之后的使用都按树查找。 +- **全量运行属于检查点。** 拉取请求的推送运行 `pr` 档;工作到达一个检查点时加 `ci:full`,运行全量。实施中的提交是 + 本地的检查点,推送发生在包含多个提交的检查点上。 +- **main 的推送。** 合入的提交的树已有标记时(验证之后 main 没有移动),不运行测试:把那次运行的产物登记为 main 的, + 保存缓存。没有标记时运行全量。供其他运行使用的缓存与历史只由这一步写入(R3 不变)。 +- **发布。** 标签所指的树已有标记时,提升那次运行的归档;没有时按现在的方式构建,并写明这次是构建。 +- **被测的即被发布的。** CI 中被测的二进制改为发布的形态:`build.yml` 产出打包后的归档,测试使用其中的程序。 + +| 时机 | 运行 | 产物 | +|---|---|---| +| 拉取请求的推送 | `pr` 档:每个宿主的构建与门;Linux 的全部单元分片与脚本 e2e 的 `pr` 集合;macOS 与 Windows 各一个单元作业;容器中的 e2e 与各宿主的 e2e | 不被使用 | +| 拉取请求加 `ci:full`(检查点) | `full` 档:全部作业 | 通过时留下树的标记 | +| 推送到 main | 树已有标记:只登记与保存;没有:`full` 档 | main 的缓存、历史、覆盖记录 | +| 发布 | 树已有标记:提升;没有:构建 | release 的归档 | + +**合并队列不是前提。** GitHub 的合并队列在合入之前对「main 加上排队的拉取请求」运行必需的检查,是给出同一个标记的 +又一个来源。工作流接受它的事件,启用与否是仓库的设置:在规则集中加入「要求合并队列」与「要求状态检查」(指向汇总 +作业 `verdict`)两条。每天合入 1 至 4 个拉取请求时不启用;并发的合入使 main 在合入后才发现失败成为常态时,由负责人 +启用,工作流与本文不需改动。 + +**代价。** 拉取请求验证之后 main 又移动时,验证发生在合入之后,与今天相同。次要作业的问题到检查点才暴露。 + +**前提,在实施中核对。** 测试是否依赖宿主构建而在静态二进制上不成立;macOS 与 Windows 的 CI 构建与发布构建的差别; +mcpp 的二进制不含提交的标识(同一棵树的两次构建给出同一个程序)。 + +### 5.23 构建程序的输入由观测给出;提示的统一机制;契约中名字的歧义(mcpp;mcpp#991、#852) + +**决定。** 全文在[子方案](2026-11-21-build-program-observed-inputs-design.md),本节只陈述它在本轮中的位置。 + +- 构建程序的输入分为预测、编译读取、运行读取三类;编译读取取自子构建对程序单元的依赖记录(GNU 的扫描 depfile, + MSVC 的 `sourceDependencies`),写入 `programs.json`,进入程序的身份。工程的快速路径、编译闸门与运行缓存读取这同一份 + 记录。它与 5.3 是同一条原则:能观测的输入以观测为准。 +- 包含标准库头文件与包含本包头文件两种用法在文档中标为不推荐并给出推荐的写法;一个工程第一次构建时各提示一次, + `mcpp clean` 之后重新提示。 +- 读取包外文件在本地构建中警告一次,在 `mcpp publish` 中拒绝(原因标记 `build-program-unpublished-input`)。 +- 提示有一个汇点(`mcpp.diag`)与一张登记表(新包 `modules/notice`);策略为 `always`(缺省)与 `once`(一个工程每个键 + 一次,记在 `target/.mcpp-notices`);直接的 `ui::warning` 与三处各自的「只说一次」并入汇点。 +- 构建程序契约中的名字按发布的变量判定歧义(#852 的根因:按拼写判定,而变量由拼写有损地变换得到);依赖的目录、 + 链接形式、工具与载荷四个通道共用一个分组函数,有歧义的变量不发布,读取者拒绝并说出可用的拼写,构建时不再警告。 + +**验收。** 子方案第 6 节的 T1 至 T10 与 U1 至 U3;#991 的复现在 Linux、macOS 与 Windows 上给出新值;#852 的两种情形 +(共有的末段,变换后相同的名字)构建时都不警告,询问时都被拒绝。 + +### 5.24 离线、宿主输入与工具链的缺省(xlings、xim-pkgindex、mcpp;mcpp#988、#989、#990) + +**决定。** 全文在[第二个子方案](2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md),本节只陈述它在本轮中的位置。 + +- xlings 以 sha256 从有序的来源列表取归档(本地目录在网络之前),并能把一组包连同索引快照导出为目录、从目录安装;mcpp 以 `mcpp fetch` + (`--to`、`--from`)把一个项目所需之物导出与导入,离线是工程与机器两层都可写的 `[build] offline`。 +- 宿主输入逐类显式开启(新增 `[build] allow_host_tools`),身份进入缓存键与锁,每一类在一个工程中提示一次;引导工具按需检查。 +- 工具链的请求只有一个解析函数,写明的请求进入锁;配套载荷的版本取自工具链的声明而非目录的顺序,并可以覆盖;编译器的版本由驱动的 + 机器可读查询给出,三元组的 vendor 段为任意词。 +- 缺省是数据,取上游最新的正式发布并经 CI 验证;本轮刷新 gcc、llvm、binutils、linux-headers、patchelf 与 ninja 的缺省。 +- 命令行的 `@<版本>` 是按书写精度的前缀,`@=<版本>` 是精确的键;清单的版本字符串仍是要求文法。 +- 以 `dlopen` 加载的库的可达性按发起加载的对象判定(带 `DT_RUNPATH` 的对象看不到可执行文件的 `DT_RPATH`),不可达时报告并指出须声明的运行依赖。 + +**验收。** 第二个子方案第 6 节的 X1 至 P1;mcpp 自身在无网络的命名空间中由 bundle 构建。 + +## 6. mcpp 的构建与测试即规模基准 + +基准不另设工作量。每次 CI 已经构建 mcpp(830 条边)并构建、运行它的全部测试;5.6 与 5.9 使这两件事留下记录。 + +| 类别 | 内容 | 判定 | +|---|---|---| +| 不变量(每个拉取请求,小工程的 e2e,带指标与界限) | 无改动的重建运行 0 条编译边;`build` 之后的 `test` 只编译新增的单元(#986);同一 home 的第二个工程编译 0 个平台单元(5.2);新检出在已填充的缓存上运行 0 个编译器(5.3);dev 依赖由缓存提供(5.4) | 超出界限即失败 | +| 规模记录(每次全量运行) | 自建:边数、运行的边数、编译器的处理器时间、最大驻留集、最慢的边;测试构建:同上;测试运行:处理器时间、每个用例的开销;一次无事可做的 mcpp 调用的用时(e2e 调用它成千上万次) | 趋势;拉取请求与同类机器的中位数比较,写入摘要,不判定 | +| 引擎对照(`bench/`,手动) | 不变 | 不变 | + +第一类中的每一项对应本文的一个缺陷:它们在引入时都没有测试能察觉。 + +## 7. CI 的形态与实现 + +### 7.1 形态 + +| 阶段 | 现在 | 本文之后 | +|---|---|---| +| 触发与档位 | 拉取请求与 main 的推送各运行全部作业;`pr` 与 `full` 只改变 e2e 的集合与一个 macOS 镜像的子集 | 拉取请求的推送运行 `pr` 档,检查点(`ci:full`)运行 `full` 档;main 的推送在树已有标记时不运行测试(5.22) | +| 被测程序的构建 | 每个宿主一次,从零;宿主构建 | 每个宿主一次,单元由缓存提供(5.3);发布的形态(5.22) | +| 其他工具链与目标的自举构建 | 各自从零(3.5),每次运行 | 只在 `full` 档;各自的配置在缓存中命中 | +| 测试的构建 | 一次;main 的构建目录按修改时间复用(W1) | 一次;按内容命中(5.3) | +| 分片 | 固定四片,按宿主的历史分配 | 片数来自作业的预算(5.17),按机器类别的历史分配(5.9);一份预算(5.5) | +| 共享状态 | 第一个用例填充 home | 夹具按身份恢复(5.10) | +| 记录 | 程序与用例 | 另有边、步骤、机器、处理器时间、用到的面 | +| 选择 | 脚本 e2e 按覆盖图;C++ e2e 全部 | 不变;头文件按清单;函数粒度视测量(5.11) | +| 门禁 | 测试的结论;每个 e2e 在某处运行 | 另有预算(5.17)、面的覆盖(5.16)、不变量(第 6 节) | +| 定时 | 每夜:覆盖图、金丝雀、已发布版本的安装 | 不变;抖动每周一次,并在结论翻转时运行(5.19) | +| 发布 | 四个平台各自构建,约 30 分钟 | 提升已验证的归档(5.22) | + +### 7.2 实现:逐个文件 + +| 文件 | 改动 | +|---|---| +| `ci.yml` | `changes` 计算本次运行的树的标识;`set` 的含义扩大到作业:`bootstrap`、`cross`、`target-matrix`、`upgrade`、`openkal`、`macos-ios`、`windows-msvc-xlings`、`linux` 中其他工具链的作业、第二个 macOS 镜像只在 `full` 档运行;main 的推送先查树的标记,已有时只运行登记与保存的作业;新增汇总作业 `verdict`,在 `full` 档通过时上传标记 `verified-<树>`;`on:` 接受 `merge_group`(未启用合并队列时不触发) | +| `build.yml` | 构建后以 `mcpp pack` 产出发布形态的归档(Linux 为 `x86_64-linux-musl` 的静态包),`mcpp-built-` 取其中的程序;`full` 档另上传 `mcpp-release-` 与单元缓存中本次用到的条目(5.3 的修剪) | +| `ci-unit.yml`、`mcpp-unit` | 分片的清单来自计划作业的输出,不再是固定的 `[1,2,3,4]`;归档阶段取 main 登记的单元缓存,不再取构建目录与 `.source-blobs`;运行阶段陈述预算(5.5)与监测程度(5.9);重跑仍为 `--failed` | +| `e2e-mtest`、`ci-*-e2e.yml` | mtest 引擎陈述预算;存储照旧上传 | +| `newest_artifact.sh` | 增加按树的标记查找一次运行;「main 最近一次成功的运行」的含义不变(登记之后产物在 main 的运行之下) | +| `release.yml` | 新增 `promote`:由标签得到树,按标记找到运行,下载各宿主的 `mcpp-release-`,核对校验和与版本,创建 release;没有标记时运行现有的构建作业,摘要写明这次是构建而非提升 | +| 新增 `jitter.yml`(每周、手动,及 CI 运行结束后有结论翻转时) | 取 main 登记的归档与测试,带抖动运行;翻转时只运行翻转的用例,若干个种子;结果与种子写入摘要 | +| `e2e-coverage.yml` | 覆盖图记录函数;首次运行给出 5.11 的统计 | +| `test-report` | 报告增加各段:构建的关键路径与最慢的边、步骤、余量、预算的超出、面中未被用到的项、结论翻转的用例 | +| 新增 `.github/actions/project-test` | 下游仓库的动作:安装固定版本的 mcpp,恢复 home(拉取请求只读),`mcpp test`,上传存储,写摘要 | +| `.github/tools/` | 新增 `waits.baseline` 与其检查(5.20);`check_version_pins.sh` 纳入骨架的 mtest 版本;`check_e2e_coverage.py` 与两份 tsv 由账本取代(5.13);`source_mtimes.py` 在固定版本移动后移除 | +| `docs/92-release.md`、`mcpp-release` 技能 | 发布的步骤改为:发布的拉取请求在 `ci:full` 的检查点通过后合入,打标签,`release.yml` 提升;不再取消 main 的运行(它已不运行测试) | + +仓库的设置不需改动。 + +### 7.3 自举版本的约束 + +构建 mcpp 自身的是 `.xlings.json` 固定的已发布版本。因此 5.3 对「构建 mcpp」的各个作业(3.5)生效,要等含有它的版本 +发布并成为固定版本之后;这是发布规程已有的一步(固定版本的移动),W1 的移除与它同一个提交。测试的构建由本提交的 +mcpp 完成的部分(各宿主的单元作业)在本轮即生效。 + +### 7.4 下游仓库 + +`project-test` 之外,下游得到的是引擎与框架的能力本身:实例的复用(5.1)、构建的记录(5.6)、头文件的选择(5.11)、 +doctest 适配器(5.14)、依赖的调试信息(5.15)。boost-m 自己的三项用法(两组各运行全部测试、每个文件一份 doctest +实现、合并移植的示例文件)由其维护者决定。 + +## 8. 各仓库的范围 + +每个仓库一个拉取请求,提交作为检查点。mcpp 的拉取请求是 #984(分支 `fix/rc-include-dir-globs`):它已有的两个提交 +是最早的检查点,本文与各任务的提交接在其后,标题与说明在内容齐备时改写。任务、位置与依赖在第 12 节。 + +| 仓库 | 范围 | +|---|---| +| mcpp | 本文;SPEC-004 与 SPEC-017 的条款;引擎(5.1 至 5.6、5.8 至 5.11、5.14 至 5.16、5.19 的运行器与钩子、5.23、5.24);测试 kit 与测试(5.7、5.10、5.18、5.20、5.21、第 6 节);CI 与发布规程(5.22、第 7 节);docs 两种语言与 CHANGELOG | +| mtest | 协议的实现与一致性套件;5.12、5.13、5.16 至 5.19、5.21 的框架与引擎一侧;doctest 适配器;报告与分析 | +| mcpplibs.os | 处理器的型号(O1) | +| xlings | 按内容取归档的来源;导出与从目录安装(5.24);发版 | +| xlings-res | 5.24 刷新所需的工具链与配套载荷 | +| xim-pkgindex | 5.24 的版本与包版本号的规则;mtest 0.11.x 与 mcpp 的版本条目 | +| boost-m | 不由本文改动 | + +## 9. 预期 + +| 对象 | 基线 | 预期 | 依据 | +|---|---|---|---| +| 用例 955 的步骤 A(分片内,EPYC) | 478 s | 与步骤 D 同一量级 | 5.2;探针第二遍中命中的配置为 7 s | +| 分片中平台运行时的边用时 | 2,579 至 7,255 s | 每个配置在夹具的构建缓存中命中 | 5.2、5.10 | +| 分片内争用 | 墙钟 2.3 倍 | 不超过 1.3 倍 | 5.5 | +| 各作业的 mcpp 自举构建 | 310 至 2091 s,各自从零 | 与改动的单元数成正比 | 5.3(推算) | +| boost-m 的测试步骤 | 916 至 2507 s | #986 的部分减少至多六分之一;长尾在有记录之后定位 | 5.1、5.6 | +| 新工程中的 mtest 库 | 每次 4 至 16 s | 0 | 5.4 | +| mtest 的每个文件(GCC) | 1.1 s | 不超过 0.15 s | 5.12 | +| 一次合入的全量运行 | 至少两次 | 一次 | 5.22 | +| 一个推送三次的拉取请求的 runner 时间 | 约四次全量 | 三次 `pr` 档加一次全量,约少四成 | 5.22;按 3.5 中次要作业约占 649 分钟中的 350 分钟估算 | +| 发布 | 约 30 分钟的构建,产物未经完整验证 | 提升,数分钟;产物即被验证的那一份 | 5.22 | + +拉取请求的整次墙钟不给出预期:它受组织内的排队支配(2026-11-14 记录 6.1),本文改变的是工作量。 + +## 10. 风险与未采纳 + +- **5.3 的正确性取决于键的完备。** 遗漏一个输入会提供错误的目标文件。清单取自编译器自己的依赖记录;逐字节的核对是 + 验收的一部分;不可复现的工具链不缓存。 +- **5.5 改变每一次嵌套构建的并行度。** 单独运行的构建不受影响(没有父 generation)。 +- **5.10 把 home 的内容变成声明。** 现有用例中隐含依赖共享 home 里某个包的,会在迁移时暴露;逐个改为声明。 +- **5.12 的接口改动**不改变已发布名字的含义(SPEC-017 的「只增不改」针对协议;库的源码兼容性另由 mtest 的版本规则约束)。 +- **5.22 之下次要作业的问题到检查点才暴露。** 需要提前看到时加 `ci:full`;验证之后 main 又移动的情形与今天相同。 +- **5.18 使现有用例中未声明的诊断行成为失败。** 其数目未知,在实施中先以报告模式测得。 +- **5.15 改变依赖的缺省调试程度。** 调试进依赖的代码时需要在 profile 中覆盖;缓存中依赖的条目重建一次。 +- **未采纳:以 GNU make 的 jobserver 作为预算的形式。** 它需要命名管道,`mcpplibs.os` 没有;槽位文件以已有的文件锁实现。 +- **未采纳:在 Linux 上交叉构建各平台的测试、到各平台只运行。** macOS 的原生目标与 MSVC 不能从 Linux 交叉;macOS 的 + 瓶颈是 runner 的排队(2026-11-14 记录)。 +- **未采纳:更大的 runner。** 2026-11-15 已定。 +- **未采纳:以 boost-m 作为规模基准。** 规模基准是 mcpp 自己的构建与测试(第 6 节)。 + +## 11. 已定与待定事项 + +**已定(2026-11-20 起)。** + +| 编号 | 事项 | 决定 | +|---|---|---| +| N1 | 5.3(单元的内容寻址)与其余各项同一个拉取请求 | 是 | +| N2 | #802(内存准入)并入 5.5 | 是 | +| N3 | 5.11 的函数粒度选择,阈值取 50%,测量先于实施 | 是 | +| N4 | 5.13 的账本取代 mcpp 的两份 tsv 与规范的状态列;boost-m 的由其维护者决定 | 是 | +| N5 | doctest 适配器进入 mtest | 是 | +| N6 | 下游动作放在 mcpp 仓库 | 是 | +| N7 | 向 boost-m 的维护者提供测量与用法的建议 | 不发出 | +| — | 评审中补充的各项(5.16 至 5.22,第 6 节的调用用时,5.3 的按使用修剪) | 并入 | +| N8 | 5.15:`dev` 之下依赖的缺省调试程度为 `"lines"`;键在 `[profile.<名>]` 之下,名字在 S2 中按 SPEC-004 的命名规则给出 | 是(2026-11-21) | +| N9 | 5.22:只在 `full` 档运行的作业清单按 7.2;容器中的 e2e 与各宿主的 e2e 留在 `pr` 档 | 是(2026-11-21) | +| N10 | `mcpplibs.os` 进入本轮的范围(O1),其形式须符合该仓库的规则并且通用 | 是(2026-11-21) | +| N11 | 合并队列 | 不是前提,不启用;工作流接受其事件,启用与否是仓库的设置(5.22)(2026-11-21) | +| — | 5.19 不每夜运行:结论翻转时、每周、手动 | 是(2026-11-21) | +| — | 5.22 按树认定,全量运行在 `ci:full` 的检查点 | 是(2026-11-21) | +| — | mcpp 的拉取请求复用 #984;实施中的提交是本地的检查点,推送发生在包含多个提交的检查点上 | 是(2026-11-21) | +| N12 | 5.23(mcpp#991)并入本轮,不另立拉取请求;包外读取本地警告、发布拒绝;两种头文件用法文档标为不推荐,一个工程首次构建提示一次;提示的统一机制(策略只有 `always` 与 `once`,直接调用全部迁移,弃用与未知键保持 `always`);#852 本轮按根因修复,与提示机制无关 | 是(2026-11-21) | +| N13 | 5.24(mcpp#988、#989、#990)并入本轮;xlings 与 mcpp 属同一维护范围,xlings 的共用机制以其拉取请求与发版提供;第二个子方案第 9 节取各项的推荐;命令行中 `@` 的文法(其 3.9)与 mcpp-index 中 `compat.slint` 的声明并入 | 是(2026-11-21) | +| N14 | mcpp#992 并入本轮,作为第三个子方案;其第 5 节的 Q1 至 Q5 全部采纳;载荷中的清单在已安装时照常读取,未安装时标明 `not-installed` 并以 `null` 回答 | 是(2026-11-21) | + +没有待定事项。 + +## 12. 实施方案:任务与依赖 + +每个仓库一个拉取请求,下表的每个任务是其中的一个提交或一组相邻的提交。提交是本地的检查点:一组任务在本地做完并 +核对之后才推送,全量的验证(`ci:full`)发生在包含多个提交的检查点上,不随每个提交。任务按依赖排列;同一仓库内没有依赖关系的任务 +次序任意。「位置」写明代码所在:新的机制进入 `modules/` 下的新包(AGENTS.md §4:缩小 `src/` 是方向; +`execute.cppm` 已有 8,649 行),按 `check_modules_wiring.sh` 的四处接入。 + +### 12.1 规范(mcpp 仓库,先于实现) + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| S1 | SPEC-017 v2.3 的条款,状态为未实现:P5 的 `step` 与 `exercise`;B4 的 `builds.ndjson` 与 `machine` 的字段;O5 在选择之下的时限;O11 `budget`;O12 `jitter` | `docs/specs/test.md` | — | +| S2 | SPEC-004 的条款:实例摘要的输入(§9 第 17 项);键中路径的角色;`[build] memory_threshold`;profile 中依赖的调试信息 | `docs/specs/manifest-semantics.md` | — | + +### 12.2 mcpplibs.os + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| O1 | 处理器的型号:与 `machine::memory()` 同一形式,回答为 `std::optional`,各系统一个实现单元并取自其自身的服务,映像不回答;它是机器的一个事实,不为测试而设(基准的报告、诊断同样读取) | `src/machine.cppm` 与各系统的实现单元 | — | +| O2 | 发布 | — | O1 | + +### 12.3 mtest(发布 0.11.0) + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| T1 | 协议:库写出 `step` 与 `exercise`;引擎的模型与存储读写 `builds.ndjson`、步骤、用到的项;一致性套件 | `lib/src/mtest.*`、`engine/src/model.*`、`store.*`、`conformance/` | S1 | +| T2 | 预算与抖动:引擎为所有者,陈述给它运行的程序;库的竞争点 | `engine/src/exec.*`、`schedule.*`、`lib/src/mtest.*` | S1 | +| T3 | 机器与用量:引擎写 `machine`;估计与 `ci plan` 按机器类别取历史 | `engine/src/schedule.*`、`cicheck.*` | S1、O2 | +| T4 | 库的成本:用例声明与常见 `expect` 的非模板入口;输出归属不再每个用例建文件并 `fsync` | `lib/src/mtest.*` | — | +| T5 | 账本与面的覆盖:`[ledger.*]`、豁免、判定与报告 | `engine/src/config.*`、`commands.*`、`report.*` | T1 | +| T6 | 预算的门禁:`[budget]`、豁免;`ci plan` 接受任何存储 | `engine/src/cicheck.*`、`commands.*` | T3 | +| T7 | e2e 模块:用例结束时的进程与诊断行检查;golden 的脱敏与更新摘要 | `lib/src/e2e.*` | — | +| T8 | doctest 适配器(1 级),随一致性套件 | `adapters/doctest/` | — | +| T9 | 覆盖图记录函数 | `engine/src/coverage.*` | — | +| T10 | 报告与分析读取新增记录:关键路径下到边与步骤,余量,机器类别 | `engine/src/dag.*`、`analysis.*`、`report.*` | T1、T3 | +| T11 | 3.4 的三个场景成为 mtest 自己的 `Bench` 用例,带 5.12 的界限 | `lib/tests/`、`engine/tests/` | T4 | +| T12 | 发布 0.11.0 | — | T1 至 T11 | + +### 12.4 mcpp:引擎 + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| M0 | 资源脚本的包含目录调用 `expand_manifest_include_entry`;用例 837 增加稍后建立的字面目录(5.1,接在 #984 的两个提交之后) | `src/build/prepare/windows_resources.cpp`、`tests/e2e/windows/resources.cpp` | — | +| M1 | 包的配置是一个函数;实例摘要与缓存键共用;展开后的包含目录由它给出,三份推导合一(5.1) | `src/build/cache_key.cppm`、`plan.cppm`、`prepare/package_root.cpp`、`prepare/windows_resources.cpp` | S2、M0 | +| M2 | 路径的角色;可缓存的包的构建程序输出在 home 中(5.2) | `cache_key.cppm`、`prepare/graph_provisioning.cpp`、`build_programs.cpp`;docs/25 | M1 | +| M3 | 缓存的拒绝各有原因标记;以它定位并修复 mtest 库(5.4) | `prepare/dependency_cache.cpp`;docs/50 的登记 | — | +| M4 | 构建记录:每条边一行,处理器时间与最大驻留集,各阶段的汇总,长边的进度(5.6) | 新包 `modules/build-record`;`src/build/progress.cppm` 接入 | S1 | +| M5 | 单元的内容寻址:键、清单、条目、发布;每个图经由执行器;工具链行陈述输出是否可复现;按使用修剪(5.3) | 新包 `modules/unit-cache`;`schedule/compiler_transaction.*`、`ninja_backend.cppm` 接入;`modules/toolchain-model/data/builtin.toml` | M1、M2、M4 | +| M6 | 预算:所有者与客户,槽位的目录,内存的阈值;执行器在启动编译器前取槽位(5.5) | 新包 `modules/budget`;`schedule/detach_codegen.cppm` 接入 | S1、S2 | +| M7 | 运行器的记录:`machine`、缺省的监测程度、从自己的进程测量;按机器类别的估计(5.9) | 新包 `modules/test-run`;`src/build/test_schedule.cppm` | S1、O2 | +| M8 | 时限:选择保留作用域声明的时限(5.8 的运行器一侧) | `modules/test-run`;`src/build/test_protocol.cppm` | S1 | +| M9 | 运行夹具的服务(P11)(5.10 的运行器一侧) | `modules/test-run` | S1 | +| M10 | 归档运行的三个选择缺陷;骨架的 mtest 版本进入版本检查(5.14) | `src/build/execute.cppm`、`test_protocol.cppm`;`.github/tools/check_version_pins.sh` | — | +| M11 | 依赖的调试信息(5.15) | `modules/toolchain-model/data/builtin.toml`、`src/build/flags.cppm`、`mcpp.manifest` | S2、M1 | +| M12 | `mcpp emit surface`;一次调用读取的键与发出的原因标记写入记录(5.16) | `src/cli.cppm`、`modules/manifest`、`src/diag.cppm` | S1 | +| M13 | `--since` 的头文件规则读取清单(5.11) | `src/build/test_impact.cppm` | M5 | +| M14 | 抖动的钩子:取锁与发布处的一处等待(5.19) | `src/build/dirlock.cppm` 与发布所经过的函数 | S1 | +| M15 | 提示的统一机制;构建程序的编译读取、三道闸门共用的记录、用法的提示、包外读取的警告与发布的拒绝;名字的歧义按变量判定(5.23;子方案第 7 节的 P0 至 P7) | `src/build/prepare/host_graph.cpp`、`src/build/build_program.cppm`、`src/cli/cmd_publish.cppm`、`docs/specs/build-plugins.md` | M5 | +| M16 | 离线与取齐、宿主输入、工具链的解析与锁、配套载荷、编译器的识别、缺省的数据(5.24;第二个子方案第 3 节、第 7 节的规则) | `src/cli/`、`src/build/prepare/toolchain*.cpp`、`src/toolchain/`、`modules/toolchain-model`、`src/config.cppm`、`src/pm/lock_io.cppm` | M15、xlings 的发版、xim-pkgindex 的发布 | +| M17 | 特性目录项、已安装版本的清单读取、查询的回答与两个输出(第三个子方案第 3 节) | `modules/manifest`、`src/pm/package_manifest.cppm`、`src/pm/index_management.cppm`、`src/doctor.cppm`、`src/wire.cppm` | — | + +### 12.5 mcpp:测试 kit 与测试 + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| K1 | kit:每条命令一个步骤;命令的时限取用例的余量;写出用到的项 | `tests/support/` | T12、M12 | +| K2 | home 的运行夹具;用例陈述只读或修改;修改的用例得到自己的 home 与只读的存储根 | `tests/support/`、`tests/e2e/` | T12、M9、M2 | +| K3 | 第 6 节的不变量用例与规模记录 | `tests/e2e/cache/`、`tests/e2e/self/` | M1 至 M5 | +| K4 | 等待的基线;能以事件替换的即替换 | `tests/`、`.github/tools/waits.baseline` | — | +| K5 | 诊断行的声明:以报告模式测得数目,逐个声明 | `tests/e2e/` | T12 | +| K6 | 契约性输出的 golden | `tests/e2e/`(机器可读输出、拒绝) | T12 | +| K7 | 账本:两份 tsv 与规范的状态列迁入;面的账本与豁免 | `tests/e2e/mtest.toml`、`docs/specs/` | T12、M12 | +| K8 | 函数粒度的统计,写入本文;达到阈值时实施选择 | `e2e-coverage.yml`、`src/build/test_impact.cppm` | T12 | +| K9 | 预算的豁免:超过上限的用例各一条理由 | `tests/e2e/mtest.toml` | T12、M7 | + +### 12.6 CI + +| 编号 | 任务 | 依赖 | +|---|---|---| +| CI1 | 树的标识与标记、档位扩大到作业、汇总作业 `verdict`、main 的推送按标记决定(7.2 的 `ci.yml`) | — | +| CI2 | 被测的二进制为发布的形态;核对测试在其上成立(7.2 的 `build.yml`) | — | +| CI3 | 发布的提升与回退;发布规程与技能的文字 | CI1、CI2 | +| CI4 | 分片来自计划;预算与监测程度的陈述 | M6、M7、T12 | +| CI5 | 单元缓存的传输、登记、查找与修剪 | M5、CI1 | +| CI6 | home 夹具按身份恢复 | K2 | +| CI7 | 报告的各段 | M4、T12 | +| CI8 | 抖动的工作流(每周、手动、结论翻转时);覆盖图的函数 | M14、T12、K8 | +| CI9 | 下游的动作 `project-test` | M4 | +| CI10 | 固定版本移动之后:W1 的移除,自举构建的命中(7.3) | 发布 | + +### 12.7 发布的次序 + +`mcpplibs.os`(O2)→ mtest 0.11.0(T12)→ mcpp(其 dev 依赖指向前两者)→ 固定版本的移动(CI10)。mtest 的拉取请求 +对照 mcpp 拉取请求分支中的 S1 实现;S1 的条款在 mcpp 的实现完成的提交中由未实现改为已实现。 + +### 12.8 关键的依赖链 + +- **身份到缓存到 CI:** S2 → M1 → M2 → M5 → CI5 → CI10。5.3 的收益在这条链的末端。 +- **协议到记录到报告:** S1 → T1、M4 → T10 → CI7。 +- **机器类别:** O1 → O2 → T3、M7 → T6、CI4。 +- **夹具:** S1 → M9 → K2 → CI6;K2 另需 M2,使夹具中的构建缓存对每个工程有效。 +- **树的标记与发布:** CI1、CI2 → CI3。它们不依赖引擎的改动,可以是最早的检查点。 + +## 13. 自评 + +对照 mcpp 的规则(AGENTS.md)、mtest 的规则(其 AGENTS.md)、通用性与成本,逐项核对。下列是核对后改动了本文的各处。 + +| 核对 | 发现 | 改动 | +|---|---|---| +| mtest:依赖已有的库,不重写 | v1 的 5.12 要库自己写出事件、去掉 JSON 库 | 撤回。测量表明 JSON 的代码为 0.13 MB,大小来自调试信息(8.5 MB);改为 5.15 | +| mtest:只经 `mcpplibs.os` 到达系统 | 5.9 需要处理器的型号,`mcpplibs.os.machine` 没有 | 新增 O1,范围多一个仓库 | +| 通用性:预算的形式 | v2 的 5.5 以「父 generation」陈述,是 mcpp 的内部形式,别的程序无从遵守 | 改为规范中的一个目录与其中的槽位文件,持有文件锁即持有槽位;`mcpplibs.os` 的文件锁即可实现,脚本以 `flock` 遵守 | +| 通用性:GNU make 的 jobserver | 它是事实上的标准,但需要命名管道,`mcpplibs.os` 没有 | 不采用,记入第 10 节 | +| 引擎不出现框架的名字 | home 的夹具是测试的概念 | 位于 `tests/support/`,引擎只实现 P11 | +| 每个缺省是数据,且可覆盖 | 内存阈值、依赖的调试信息、输出是否可复现 | 分别为清单的键、profile 的键、工具链行的数据 | +| 做得少不能静默 | 缓存的拒绝、发布的回退、找不到标记 | 5.4 的原因标记;7.2 中「写明」的各处 | +| 成因还是表象 | 把 `trees` 的时限调大、把套件划分得更细,都是表象 | 不提出;5.2、5.5、5.10 处理成因 | +| 已有的机制 | v1 的草稿曾把覆盖图当作新设计,把 mtest 库不进缓存当作回归 | 第 2 节的清点;3.4 改为实测的三个版本 | +| 5.11 的依据 | v2 写头文件的依赖记录来自构建记录,构建记录不含它 | 改为 5.3 的清单 | +| 5.6 的体积 | 一个分片的嵌套构建有上万条边 | 由缓存提供的边按包汇总为一行 | +| 5.3 的开销 | 每条边的摘要计算 | 文件的摘要在一次构建中按(路径、大小、修改时间)记忆;验收增加无改动重建的用时 | +| 5.3 的安全 | 缓存的条目被别的运行使用 | 只有 main 的推送运行登记或写入;拉取请求的运行只读 | +| 5.14 的适配器 | v1 写「不改测试代码」 | 改为去掉每个文件的实现一行 | +| 5.22 的前提 | v3 以合并队列为前提;仓库没有必需的状态检查,每天合入 1 至 4 个,启用它要改规则集与每个会话的合入方式 | 改为按树认定,全量运行在检查点;合并队列是可选的设置 | +| 5.19 的频率 | v3 每夜运行;这类缺陷只在并发相关的改动时新增 | 改为结论翻转时、每周、手动 | +| 已有的修复(#984) | 它称与编译步骤同一展开,但少了字面目录的回退 | 5.1 增加一段,任务 M0 | +| 任务的编号 | CI 的任务以 C 编号,与第 4 节的成因编号相同 | 改为 CI1 至 CI10 | +| 自举 | 构建 mcpp 自身的是已发布的版本 | 7.3 写明哪些收益在固定版本移动之后 | + +仍未核对、列为实施中的前提的各项:5.5 的槽位协议与现有 generation 的衔接(只读了注释);5.3 的 BMI 在 Clang 与 MSVC 上 +是否逐字节可复现;5.22 的两项前提;5.15 中工具链运行时库的调试信息的份额;mtest 库不由缓存提供的成因。 + +## 14. 实施记录 + +### 14.1 各仓库 + +| 仓库 | 内容 | 状态 | +|---|---|---| +| mcpplibs/os | O1:`machine::processor_model()`;0.4.7 | 已发布 | +| mcpp-community/mtest | T1 至 T11;0.11.0。0.11.1 增加 `time_left()`:`e2e::run` 让一条命令在用例余下的时间之内结束,扣除结束它与报告所需的时间(K1 的「命令的时限取用例的余量」) | 已发布 | +| mcpplibs/cmdline | 0.0.5 导出 `snapshot(app)`:M12 由命令行的定义取得命令与选项 | 已发布 | +| mcpplibs/mcpp-index | #865(cmdline 0.0.5)、#866(mtest 0.11.1) | 已合入 | +| openxlings/xim-pkgindex | #1210(xim:mtest 0.11.0);#1211:链接条目(14.3 第 5 项)与 xim:mtest 0.11.1 | #1211 待合入 | +| mcpp-community/mcpp | #984:S1、S2、M0 至 M14、K1 至 K9、CI1 至 CI9 | 本拉取请求 | + +### 14.2 测量 + +| 项 | 之前 | 之后 | 条件 | +|---|---|---|---| +| 新检出的 mcpp 在已填充的单元缓存上构建(5.3) | 164 s | 4.1 s,288 个编译全部由缓存放置 | 同一 home,第二个检出 | +| 依赖的调试信息(5.15):只依赖 mtest 的工程的测试程序 | 43.7 MB,构建 15.9 s | 8.3 MB,构建 7.2 s | `dev`,`dependency_debug` 缺省(`"lines"`) | +| mtest 库:一个文件的首个用例与期望(T4) | 1.1 s | 265 ms | GCC;目标 150 ms,未达到 | + +### 14.3 与第 5 至 12 节的差别 + +1. **M6、M7、M9 的位置。** 预算的客户在 `src/build/schedule/detach_codegen.cppm` 与 `src/home/files.*`,运行器的记录与 + 夹具在 `src/build/execute.cppm` 与新的 `src/build/test_fixture.cppm`,没有成为 `modules/budget`、`modules/test-run`: + 它们接入运行器的工作循环及其类型(`TestRunnable`、分片、`TestResult`),这些类型位于 `execute.cppm`;运行器移出 + `src/` 是一次单独的移动。新包为 `modules/build-record`、`modules/unit-cache`、`modules/surface`。 +2. **K1 的用到的项来自 mcpp 自身。** kit 不解析命令行:解析器知道一个选项属于哪个命令,kit 重新实现它会与之分歧。 + `MCPP_SURFACE_RECORD` 命名一个文件,调用的每个进程追加它用到的命令、选项、键与原因(每项每进程一次);kit 读取后 + 写出 `exercise`。 +3. **M13 的记录来自构建目录。** 头文件的读者取自 ninja 的依赖日志(格式 4)与扫描的 `.ddi.dep`,而不是单元缓存的清单: + 清单只存在于缓存保留的编译(cl.exe 的编译与没有缓存的 mcpp 没有),构建目录记录它构建的每个单元。工程的某个单元 + 没有记录时,头文件影响全部测试,并说明原因。 +4. **K5 以报告的策略声明。** 诊断行按 `warning:`、`error:` 两个模式以 `report` 策略声明,运行给出各用例的数目,不逐个用例 + 声明。 +5. **K2 与上游的链接条目。** 继承的 home 以 `[xlings] upstream` 指向共享 home 与用户的 `~/.xlings`,不再链接载荷、不复制 + subos。实施中发现:上游的版本目录在下游是指向所有者载荷的链接,配方的 `config()` 钩子经由它写入了所有者的文件—— + 用户 home 中 GCC 的 specs 与 binutils 的 `ld` 包装被写入了临时 home 的路径(#293 的机制)。已就地修复,并在配方中 + 修正:链接条目走借用的分支,`ld` 包装跳过链接条目(xim-pkgindex #1211)。此后 914 检查共享 home 在一次构建后没有 + 文件被写入。修正只在索引含有它时生效:用例的 home 链接用户 home 的索引,本地的索引停在 #1211 之前时再次写穿; + CI 恢复的沙箱同样带着保存时的索引,因此使用本提交 mcpp 的每个作业先运行 `mcpp index update`(`use-built-mcpp`)。 +6. **CI6。** home 夹具的恢复即 `bootstrap-mcpp` 的沙箱缓存,其键(`mcpp.toml`、`.xlings.json`)就是夹具的身份;不另设 + 缓存。 +7. **CI2 的档位。** 发布形态只在 `full` 档构建(`release-build.yml`,与 `release.yml` 同一组作业),Linux 的单元与 C++ e2e + 分片测试 Linux 归档中的程序;`pr` 档测试宿主构建。发布形态是又一次自举构建,`pr` 档的目的是时间。macOS、Windows + 与 aarch64 的归档由各自的冒烟步骤核对,与发布相同。 +8. **P10.4。** 不实现:工作线程在同一进程中运行测试程序,系统对子进程的计量按进程汇总,一个程序的用量无法分离 + (SPEC-017 写明)。 +9. **CI10。** 按 7.3,在固定版本移动到含 5.3 的版本之后。 +10. **K7 的两份表。** `coverage-exceptions.tsv`、`coverage-debt.tsv` 与 `host-requirements.tsv` 仍由 + `check_e2e_coverage.py` 判定,没有迁入 `[ci.coverage]`:它的 `job`(工作流按名字运行一个测试并自行断言其结果)与 + `unaffected`(`pr` 集合按变更略去的测试)两种判定在 `mtest ci coverage` 中没有对应,迁入会使这两类测试成为未覆盖。 + 新的账本是面的账本(`[ledger.surface]`)。 +11. **构建记录的 `staged`。** 1122 在实施中失败:缓存提供的包的单元由 `stage_file` 边复制,构建记录把这些边与编译 + 一同列出,第二个工程看起来编译了 mtest 的 13 个输出。记录为这类边增加 `staged`(SPEC-017 B4),它与单元缓存的 + `served` 一样不是编译。余下的一个编译是 mtest 的入口单元:包的入口单元随链接它的映像编译,不取自缓存中包的条目, + 1122 的界限为 1。 +12. **第一次 CI 检查点的发现(运行 44823760291,`pr` 档)。** Linux 的单元分片有 49 个失败,各有成因,均在本拉取请求中修正: + - **嵌套运行的死锁。** 另一个运行器的预算之内(用例中的 `mcpp test`),第一个工作者以持有的槽位运行全部测试,其余 + 工作者等待永远不空出的槽位,嵌套的运行到 600 s 被结束;十余个用例与 macOS 的作业(90 min 被取消)因此失败, + 其余用例在只剩一个编译器的分片中变慢(`sets` 用尽 30 min)。等待槽位的工作者在测试已被取完时退出。 + - **单元缓存与 Clang。** Clang 经搜索路径找到导入的 BMI,其依赖文件不列出它:导入者的条目只记录源文件,修改的 + 模块之后,导入者由缓存提供旧的目标文件(118 输出 41;Windows 的 1111、1112、1114;1113 的编译器崩溃)。 + Clang 的编译得到 `-fmodule-file-deps`(`BmiTraits::moduleFileDepsFlag`),依赖文件列出它读取的 BMI;单元的读取 + 在编译器写出依赖文件之后读取,不列出任何文件的单元不保留。 + - **打包的一遍。** 打包提供者的输出取决于打包请求与成员的暂存树,不在其键中;在 home 中运行时两个工程读到彼此的 + 归档(868)。打包的一遍中构建程序在工程中运行(SPEC-004 §9 第 18 项)。 + - **继承的 home。** 不再复制 subos、不再链接载荷之后,离线或换用替身 xlings 的用例得到未初始化的环境。kit 在建立 + 继承的 home 时以真实的 xlings 运行一次 `mcpp self init`(不到 1 s),由上游按链接安装 patchelf 与 ninja。 + - **其余。** `memory_threshold` 进入成员的分组键;1069 的两处等待恢复(时间前移 2 s 使下一次构建仍晚于它); + 数个用例改读 home 中的输出目录、BMI 旁的生产者键与 gc 的单元条目。 +13. **面的账本的第一次数据。** 429 项中 331 项有通过的用例。未覆盖的 98 项中:命令行在解析之前读取的全局选项 + (`--no-color`、`--frozen`、`--system`、`--protocol-version`)用到而未被记录,调用改为从原始的参数记录它们; + 没有任何用例运行的维护命令与测试选项由新用例 1125、1126 运行;1125 发现 `mcpp index list` 不列出 `mcpp index add` + 加入的注册表(按旧的载荷形式读取提供者的列表),已修正。只由脚本 e2e 用到的项(脚本不写出用到的项)在账本中 + 以豁免陈述,理由指名脚本。 +14. **在同一个 home 中重复运行。** 本地以一个 home 反复运行全量用例,暴露了 CI 的新 home 看不到的四类事实: + - **条目按它编译时所读的接口提供(SPEC-001 §10.4 第 7 条)。** `mcpp cache gc` 移除上游的条目后,同一棵树在另一路径 + 编译出另一份接口(GCC 的 CRC 不同),依赖它的条目仍被提供,构建以 `module 'globbing' CRC mismatch` 失败 + (build-program 311)。条目改为记录所导入接口的摘要,上游在本次构建中编译时依赖者随之编译并重写条目(cache 1133)。 + - **store 按路径判定。** 由上游 home 链接安装的 nasm 被解析到上游的目录,判为宿主的汇编器(build 105);改为按路径。 + - **断言偶然的事实。** 断言「本次编译了某个路径依赖或工具」的用例(features 82、build 50、build-program 698)在条目 + 已存在时不成立:82 改为断言包在构建中,50 与 698 使用用例自己的 home。 + - **运行一次的键按规划陈述。** `[build] offline` 与 `allow_host_tools` 没有进入成员的分组键(SPEC-004 §9 第 10 条), + 工作空间依次规划各组时沿用前一组的陈述;两键进入分组键,每个规划陈述或撤回工程层(单元测试 KeyRegistry、RunKeys)。 +15. **第二次检查点的发现(运行 44897058445,`full` 档)。** + - **等待机器之门的工作者持有预算的槽位。** 每个工作者在整个运行中持有一个槽位,等待持有 `machine` 的测试时也不交回, + 该测试的构建只得到其工作者的一个槽位:234 的编译器峰值为 1(上限 2),860 在 `-j 4` 下三次编译没有重叠。等待门的 + 工作者交回槽位,取得门之后再取(SPEC-017 O11 第 5 条);860 陈述 `machine`,它断言的并行只在预算属于它时成立。 + - **在自己的目录中规划的命令写入了工程的提示记录。** `mcpp emit build-database` 写出 `target/.mcpp-notices`,工程的 + 目录树改变(emit 688、817)。这样的命令读取记录而不写入。 + - **作用域的事件不指名用例**(mtest 0.11.2):运行器把它们归入程序的记录(Windows 的 mingw 97)。 + - **用例探测的工具链依赖缓存。** 本次改变了 xlings 的版本,沙箱缓存的血统随之改变,Linux 分片中需要 llvm 的 14 个 + 用例在任何 runner 上都没有运行,其中 988 是 `toolchain/default-changed` 提示唯一的用例。分片改为显式安装用例探测的 + 工具链(`ci-unit.yml` 的 `toolchains`),与脚本 e2e 的分片相同。 + - **面的账本。** 用例 1127 运行构建命令共有的选项与其他用例没有陈述的 `[build]` 键;bundle 990 在不提供网络命名空间 + 的机器(Ubuntu 24.04)上以引擎的离线为界运行,`ci-offline.yml` 仍在命名空间中构建本仓库。 +16. **第三次检查点的发现(运行 44910330728,`full` 档)。** + - **新的默认工具链暴露了依赖旧载荷的事实。** gcc 16.2.0 第 1 修订版内置的 sysroot 是构建它的 home 的 `subos/current`, + 重映射只认 `subos/default`,未描述环境的链接找不到 home 的 C 库(subos 221、binding 220、flags 141);提供者把两者都 + 视为构建它的 home 的环境。以路径指名的 GCC 使用自己找到的汇编器(mcpp#755),而 store 中 gcc 的 binutils 是单独的 + 载荷,gcc 16.2.0 输出的指令宿主的 binutils 2.42 不能汇编,用例陈述汇编器(bases 925)。替身 xlings 的配置写死旧的 + 默认工具链(local 60),改为沿用 home 已有的默认。 + - **升级通道。** 升级后的第一次运行安装本次提交钉住的引导工具(ninja、patchelf),通道从内建表的默认行读取它们;依赖的 + BMI 暂存目录的拼写在 2026.11.21.1 改变(mcpp#986),通道接受两种拼写。 + - **冷缓存下用例的时间。** 分片安装 llvm 之后,955 第一次在 Linux 上运行(此前在任何 runner 上跳过):第一次打包为两种 + 架构构建 openkal 集合,超过命令的十分钟。无 Visual Studio 的 Windows 上,182 在用例自己的 home 中安装 GNU 行的工具链: + 缓存热时 54 s,xlings 版本改变使缓存冷时超过五分钟。两个用例按测量陈述各自的时间(30 分钟、15 分钟)。 + - **860 在 macOS 上。** 三个处理器的 runner 上三次编译没有全部重叠(macos-15 与 xcode-27 均出现过,不是每次)。临时 + 探测排除了内存阈值:三个、六个编译器运行时两种镜像上的可用内存为 2.9–3.5 GB,远高于 1 GiB。图的全部步骤(失败时 + 报告)表明原因在扫描:每次编译等待其单元自己的扫描,同一文件的三次扫描在图开始后 1.9、3.5、7.7 s 结束,第一次编译 + 在第三次开始前已经结束,而其间有两次同时进行。调度保证的是多于一次编译同时进行,而不是三次同时;用例在 `-j 4` + 下断言同时进行的编译至少两次,在 `-j 1` 下恰为一次。 + - **两处推导的 `-B`。** 以路径指名的 GCC 陈述 `tools.as` 后,`std` 模块的编译指向该汇编器,普通编译与链接仍由归档 + 工具所在的目录推导;这样的 GCC 驱动旁没有归档工具,于是没有 `-B`,驱动从 PATH 运行宿主的 `as`(bases 925 在 CI + 上失败)。编译与链接改为取 + `gcc::binutils_prefix_dir` 的同一个答案。 + - **选中用例的时限。** 182 陈述的时限没有生效:程序的列表只列出顶层的用例,选中其下的用例时,运行器的时限是被列出 + 者陈述的(SPEC-017 O5)。时限改由套件陈述。 + - **面的账本:用到而未记录。** 门禁列出 29 项没有通过的用例。其中数项已被用例运行而没有记录:kit 只为 `mcpp()` + 运行的命令设置记录文件,以会话、在另一程序之下(`unshare -rn`)或由用例自行运行的命令不写出。kit 改为为它组成的 + 每条命令设置记录,`mcpp()` 在命令结束时陈述,其余在用例结束时(mtest 的 `at_case_end`)陈述;`options::wrap` + 使在另一程序之下运行的命令经由 kit(bundle 990、homes 919)。 + - **面的账本:没有用例。** 新用例 1134 至 1140 与已有用例中的步骤覆盖 17 项:加速器的测试与数据库、平台的链接 + 驱动与封装工具及其 `tools`、平台阶段的指令、平台依赖、C 库声明的测量、伴随的 binutils、另一系统的工具链、链接的 + 伴随文件、`fetch --package`、模块图的全量核对、上限与信号量。豁免六项,各陈述理由:只由脚本运行的 `--system`; + 没有输入能到达的 `ambiguous-request` 与 `layer-ordering`;只在 Windows 上到达的两个原因;需要自备载体包的 + `carrier`。 + - **一个答案的两处写法。** `why` 的两种 JSON 对没有记号的拒绝分别报告 `other` 与 `none`;改由拒绝记录的一个函数 + 给出(`take_refused`),`other` 同时记入面的记录。拒绝 `system`、工具链命令的参数错误、`self config` 与 + `doctor` 的提示写死 gcc 16.1.0;改为读取内建表中本机的首次运行,`doctor` 的提示改为 + `mcpp toolchain install --target `。 + +## 15. 子方案 + +本方案的一部分在单独的记录中陈述,因为它们的范围超出测试与 CI,而实施仍属本轮、在同一个拉取请求中(N12)。 +子方案的决定以其自身为准,本方案只陈述它们的位置与依赖。 + +| 子方案 | 范围 | 在本方案中的位置 | +|---|---|---| +| [构建程序的输入由观测给出](2026-11-21-build-program-observed-inputs-design.md)(mcpp#991、#852) | 构建程序的编译读取与三道闸门共用的记录;头文件用法的提示;读取包外文件的警告与发布的拒绝;提示的统一机制(`modules/notice`,策略 `always` 与 `once`);构建程序契约中名字的歧义按变量判定 | 5.23;第 11 节 N12;第 12.4 节 M15;验收随第 14 节的实施记录 | +| [离线、宿主输入与工具链的缺省](2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md)(mcpp#988、#989、#990) | xlings 的按内容取归档的来源与导出导入;mcpp 的 `fetch`、`offline` 与 `allow_host_tools`;宿主输入的身份与提示;工具链的一个解析函数与锁;配套载荷由声明决定并可覆盖;编译器版本与三元组的识别;缺省版本的规则与本轮的刷新 | 5.24;第 11 节 N13;第 12.4 节 M16;涉及 xlings、xlings-res、xim-pkgindex 与 mcpp 四个仓库 | +| [包的特性作为机器可读的回答](2026-11-21-package-features-query-design.md)(mcpp#992) | 一个特性目录项;索引包已安装版本的清单一条读取路径(构建与查询共用);回答的三种状态与不安装;`mcpp search` 的 `features` 与 `mcpp why features`;特性的 `description` | 第 11 节 N14;第 12.4 节 M17;只涉及 mcpp | + +第一个子方案与本方案的联系有三处:编译读取与 5.3 的单元内容寻址是同一条原则(能观测的输入以观测为准),并复用 5.3 的摘要备忘 +与依赖文件的解析;提示的代码与发布的原因标记进入 5.16 的面的账本;它的 e2e 用例按 5.10、5.18 的 kit 编写,并在 +5.22 的全量运行中验证。 + +第二个子方案与本方案的联系:它的离线路径是 5.22「被测的即被发布的」在没有网络的机器上的延伸,mcpp 自身的离线构建成为 CI 的一个作业; +它的新代码与原因标记进入 5.16 的面的账本;它与第一个子方案共用提示的汇点与 `once` 策略;它把本轮的范围扩展到 xlings、xlings-res 与 +xim-pkgindex,各一个拉取请求,次序在该子方案第 4 节。 + +第三个子方案与本方案的联系:它把查询的回答写成机器输出的契约,`--protocol-version` 的变化由 5.21 的黄金文件给出;构建读取已安装版本 +清单的代码与查询合为一处,由 5.22 的全量运行验证构建不变。 diff --git a/.agents/docs/2026-11-21-build-program-observed-inputs-design.md b/.agents/docs/2026-11-21-build-program-observed-inputs-design.md new file mode 100644 index 000000000..252681136 --- /dev/null +++ b/.agents/docs/2026-11-21-build-program-observed-inputs-design.md @@ -0,0 +1,332 @@ +--- +subject: build-program +status: active +--- + +# 构建程序的输入由观测给出:编译读到的文件、三道闸门的同一份记录、头文件的用法提示、提示的统一机制、契约中名字的歧义(mcpp#991、#852) + +**日期:** 2026-11-21 · **版本:** v4(v2 增加 3.6 提示的统一机制;v3 增加 3.7 契约中名字的歧义(#852);v4:旧的回答经过弃用期,被询问时警告。2026-11-21 确定 3.6 的四点:策略只有 `always` 与 `once`,直接的调用全部迁移且检查没有例外,弃用与未知键保持 `always`,#852 本轮按根因修复;此前确定的:本方案并入 +[生态的测试与 CI 方案](2026-11-20-test-architecture-upgrade-design.md) 的本轮,见其 5.23,不另立拉取请求;包外读取在本地构建 +中警告、在 `mcpp publish` 中拒绝;标准库头文件与本包头文件两种用法在文档中标为不推荐,首次构建给出一次提示) · +**基线:** mcpp 2026.11.20.1(`722ec1b6`)与 #984 的分支 · **规范:** SPEC-007 R10(`docs/specs/build-plugins.md`) + +## 1. 问题 + +mcpp#991:`build.mcpp` 以引号路径包含本包的头文件,头文件改变之后 `mcpp build`、`mcpp run` 仍使用旧的程序与旧的指令, +没有任何提示。Linux(gcc 16.1.0)与 Windows(`msvc@system`)相同。本地复现(本方案的基线二进制): + +| 步骤 | 期望 | 实际 | +|---|---|---| +| 首次 `mcpp run` | one | one | +| 头文件改为 two | two | one | +| 删除 `target/.build-mcpp` 之后 | two | one | +| build.mcpp 加入 `rerun-if-changed=support/emit.hpp`(其文本因此改变) | two | two | +| 头文件改为 three | three | two | + +最后一行说明文档给出的声明方式也不能绕过:声明使程序重新运行,运行的仍是旧的二进制。用户侧唯一有效的绕法是把头文件放进 +`[build-dependencies]` 中的 path 包,因为包的树进入身份。用法本身是文档允许的(docs/30:`#include` 照常可用), +缺陷在引擎。 + +## 2. 现状:三道闸门,各有一个预测的输入模型 + +一个头文件的改变要先后经过三处判断,三处都看不见它: + +| 闸门 | 位置 | 依据 | +|---|---|---| +| ① 工程的快速路径(无事可做的构建实测 6 至 9 ms) | `build_program.cppm` 的 `program_inputs_stale` | 源文件的修改时间;构建程序记录中声明的文件、glob 与环境变量 | +| ② 编译闸门 | `host_graph.cpp` 的 `program_identity`,记在 `target/.build-mcpp/host//programs.json` | 引擎版本、工具链、标准、程序文本、库闭包中每个包及可原地变化的树的摘要 | +| ③ 运行缓存(home) | 以 ② 的身份为键,另加声明的输入与契约环境 | 同 ② | + +② 在规划子构建之前判断,所以子构建的 ninja 记录(程序单元的扫描 depfile 列出了 `support/emit.hpp`)从未被询问; +③ 以同一个身份为键,所以删除 `target/.build-mcpp` 之后重新编译的程序,其运行仍被回放。 + +这一类缺陷此前逐项修补:快速路径补入 `build.mcpp` 本身;2026.9.5.4 补入声明的文件(`mcpp.tools.embed`); +#359 与 #765 补入 glob;#748 补入宿主模块提供方的树;#931 补入链接读到的文件。每次补的是某一道闸门上的某一种输入。 +结构上的原因是三道闸门各自预测程序的输入,而主构建早已以观测为准:GNU depfile、MSVC 的 `/sourceDependencies`、 +单元缓存的已读清单(测试方案 5.3)。 + +## 3. 决定 + +### 3.1 输入分三类,各有唯一的来源 + +| 类 | 内容 | 来源 | +|---|---|---| +| 预测 | 引擎版本、工具链、标准、构建程序实例的特性、契约环境、库闭包中每个包的身份 | 本次解析(现状不变) | +| 编译读取 | 编译程序单元时读到的文件 | 观测:子构建为该单元写下的扫描 depfile(`/build.mcpp.ddi.dep`,GNU 的第一条记录),MSVC 为同一单元的 `sourceDependencies`;解析用 `mcpp.build.depfile`,与主构建同一个函数 | +| 运行读取 | 程序运行时打开的文件、读取的环境变量 | 声明:`rerun_if_changed`、`rerun_if_changed_glob`、`rerun_if_env_changed`(现状不变) | + +运行读取不改为观测:跨宿主没有可靠的手段(strace、ETW、`LD_PRELOAD` 各管一个系统,且对静态二进制无效),文档继续要求声明。 + +**编译读取的范围。** 记录排除两类文件:不可变的 store 中的文件(工具链、sysroot 与索引包的载荷,已由工具链与包的身份覆盖; +gcc 的 `-MMD` 本来就不列出系统目录中的头文件),以及 build-dependency 的树中的文件(已由该包的树摘要覆盖)。余下的是 +程序所在包的树中的文件与两者之外的文件,通常只有几个。 + +### 3.2 一份记录,三道闸门共用 + +- **记录。** 一组程序编译之后,引擎读取每个程序单元的编译读取,写入该程序的记录:`programs.json` 的行增加 `reads`, + 每项为路径、大小、修改时间与内容摘要。 +- **身份。** 程序的身份 = 预测部分 + 编译读取的路径与摘要。②、③ 都以它为键,因此一个头文件的改变同时使程序重新编译 + 与重新运行。 +- **快速路径。** `program_inputs_stale` 除构建程序的运行记录外,也读取各组 `programs.json` 的 `reads`;任一项改变即不走快速路径。 +- **比较的次序。** 先比较大小与修改时间,二者之一改变时才计算摘要;摘要相同则更新记录中的大小与时间,不判为改变 + (与单元缓存的摘要备忘相同)。声明的文件(`rerun_if_changed`)的比较改为同一次序:现状每次都计算摘要。 +- **编译失败。** 编译失败的程序不写记录,下次构建再次编译(现状不变)。 + +**成本(实测,Linux,本节的复现工程)。** 主构建无事可做时 6 至 9 ms;在子构建的目录中执行一次无事可做的 +`mcpp build` 为 110 至 280 ms。因此不采用「每次都规划子构建,由 ninja 判断」:它把每次构建的下限提高十倍以上。 +记录的比较只需几次 `stat`。 + +### 3.3 头文件的用法:文档标为不推荐,首次构建提示一次 + +构建程序的推荐写法是导入:标准库用 `import std;`,引擎接口用 `import mcpp;`,自己的共用代码放进一个 path 包的模块, +在 `[build-dependencies]` 中声明并导入。以下两种用法可以使用,文档标为不推荐,并给出推荐的写法: + +| 用法 | 判定 | 推荐 | 理由 | +|---|---|---|---| +| A. 包含标准库或系统的头文件(`#include `) | 程序文本中尖括号形式的 `#include` | `import std;` | 每次编译重新解析头文件;`std` 模块已按工具链与标准备好,构建程序与主构建共用 | +| B. 包含本包或依赖的头文件(`#include "support/emit.hpp"`) | 程序文本中引号形式的 `#include` | 共用代码做成 path 包的模块,列入 `[build-dependencies]` 并导入 | 有包的身份,可在多个构建程序与包之间共用,BMI 只编译一次 | + +判定只读程序自身的文本(`build.mcpp`,注释与字符串中的不算):在 `modules/manifest/src/mangle.cppm` 中,以 +`imported_module_names` 的同一词法新增 `included_headers`,不读它包含的文件,也不依赖 +编译读取:gcc 的 `-MMD` 不列出系统目录中的头文件,A 无法从编译读取中得出。A 与 B 是同一种提示的两种内容,一个程序一条。 + +**提示。** 一个工程第一次构建时,每一种用法给出一条提示(`note`,不是警告),两行,例如: + +``` +note: build.mcpp of `repro` includes headers (, "support/emit.hpp"); this works, and the + recommended form imports them: `import std;`, and a module of a path build-dependency (docs/30, "Headers or modules") +``` + +- **一个工程只提示一次。** 两种用法登记为 `once` 策略的提示(3.6),键为(用法,程序所在的包)。之后的构建不再提示, + 直到 `mcpp clean` 删除 `target/`;出现新的用法,或另一个程序出现同一用法时,只提示新出现的。 +- **合并。** 一次构建中多个程序有同一用法时合并为一条,列出各包名(工作区的成员,3.6)。 +- **范围。** 只提示工程自己能修改的程序:根包、工作区成员与 path 依赖。索引包与 git 依赖的构建程序不提示。 +- **机器输出。** 代码为 `build-program-includes-header`;`--quiet` 不显示(3.6)。 +- 提示不改变构建的结果,不进入任何键。 + +### 3.4 读取包外的文件:本地警告,发布拒绝 + +C. 编译读取中既不在程序所在包的树内、也不在 build-dependency 的树或 store 中的文件(绝对路径、`../shared/x.hpp`), +有两个实际的后果:构建依赖这台机器的文件,违反默认密闭(AGENTS.md §2);发布的归档由 `git archive` 取包的根目录, +这个文件不在其中,消费方取到的 `build.mcpp` 无法编译。 + +- **本地构建。** 给出警告,说明影响与办法(把文件移入本包,或做成 path 包),代码 `build-program-reads-outside`。 + 登记为 `once` 策略(3.6),键为(程序所在的包,文件),`mcpp clean` 之后重新警告。它不改变这次构建的结果, + 影响在别处(其他机器、消费方),所以不属于 `always`。索引包与 git 依赖的程序 + 同样警告,办法改为向包的维护者报告。 +- **发布。** `mcpp publish`(它在打包前调用 `prepare_build`,记录因此是当前的)拒绝发布:编译读取中有文件在包的根目录之外, + 或不在 `HEAD` 的已跟踪文件中(归档不含它)。拒绝说明原因与办法,原因标记为 `build-program-unpublished-input`, + 进入 `check_reason_tokens.sh` 核对的列表与面的账本。 + +### 3.5 构建程序导入的模块与库 + +- **保留。** 程序只能导入 `std`、`std.compat`、`mcpp`、`mcpp.core`,以及 `[build-dependencies]` 导出的模块与它们以 + `reexport = true` 转交的模块;只经由另一个 build-dependency 的依赖到达的模块被拒绝(`host-module-missing`,现状)。 +- **新增的要求即 3.4。** 程序的读取限于它的包、它的 build-dependency 与 store。这是对读取的要求,不是对写法的要求, + 因此同时覆盖头文件、`#embed` 与绝对路径的包含。 +- **不限制。** 库自带构建程序(规则包依赖它,嵌套由 `tool_depth` 处理);库的闭包大小(成本,`mcpp why build-programs` + 已给出)。 +- **宿主的可移植性,在实施中核对。** build-dependency 的闭包只支持某些宿主系统时,宿主图的解析是否与目标图一样按宿主 + 拒绝并给出原因;不成立时在本轮补上同一检查。 + +### 3.6 提示的统一机制:一个汇点、一张登记表、两种策略 + +**现状。** 告诉用户一件事的方式有六种,三种各自实现了「只说一次」: + +| 方式 | 位置 | 策略 | +|---|---|---| +| `mcpp.diag`(`warning`、`degraded`、`note`) | `src/diag.cppm`,89 处调用 | 一个进程内按全文去重;保留机器可读的记录;`--strict` 把 `degraded` 升为错误 | +| 直接调用 `ui::warning`、`ui::note` | 31 个文件中约 100 处;另有 4 处直接写 stderr | 无:不去重,不进入机器输出,`--strict` 看不到 | +| `print_hint_once` | `src/toolchain/compat.cppm` | 自己的静态集合,一个进程一次 | +| `Said` | `src/config.cppm` | 自己的静态集合与互斥量,一个进程一次,直接写 stderr | +| 边的提示(`.mcpp-advice`) | `src/build/advice.cppm` | 边运行时才报告,一个进程一次 | +| 构建程序的 `mcpp::warning` | 构建程序的记录 | 每次缓存命中都回放(docs/30:只说一次会被读成「问题已解决」) | + +弃用登记表(`modules/manifest/src/compat/registry.cppm`)是这里已有的范式:一张表陈述每一行,警告的文字、 +`mcpp why compat`、docs/51 的一致性检查与 `--strict` 的拒绝都由行导出。缺的是同一形式的「提示」:3.3 要求的 +「一个工程只说一次」没有可复用的机制,已有的三处「一个进程一次」各自实现。 + +**决定。** 用户可见的提示只有一个汇点 `mcpp.diag`;提示的种类与策略是数据;「说过」的状态由汇点统一保存。 + +- **汇点。** 所有用户可见的警告、降级与提示经由 `mcpp.diag`。`ui::warning` 与 `ui::note` 只供 `mcpp.diag` 渲染使用; + 上表第二至四行的调用全部改为经由汇点,各自的静态集合删除。边的提示与构建程序的 `mcpp::warning` 是传送通道,保留, + 读到的每一条交给汇点。`.github/tools/check_diag_sink.sh` 检查 `src/`、`modules/` 中 `mcpp.diag` 与 `mcpp.ui` 之外 + 不再有直接的调用。 +- **登记表。** 新包 `modules/notice` 持有提示种类的表,与弃用登记表同一形式。一行陈述:代码(kebab-case,稳定)、 + 严重程度(`note`、`warning`、`degraded`)、策略、领域、文档的位置。只有策略不是缺省的种类需要登记;没有登记的提示 + 按缺省策略处理,与今天相同。 +- **两种策略。** + - `always`(缺省):每次得出该事实时都说,一个进程内按全文去重(现状)。用于这次构建的结果因它而不同的事实 + (`degraded`、构建程序的 `mcpp::warning`、弃用的读法、未知的键)。AGENTS.md §2「做得更少从不沉默」要求这一类不被抑制。 + - `once`:一个工程中每个键只说一次。键由种类与对象(包、文件、成员)组成;对象改变即是新的键,因此新出现的 + 事实仍会被说。「说过」指在终端上显示过:`--quiet` 隐去的与只进入 `--format json` 的不算。说过的键记在 + `target/.mcpp-notices`,每行一个键;`mcpp clean` 删除整个 `target/` 时一并删除,`mcpp clean --stale` 不删除它。 + 不在工程中的命令(`mcpp toolchain install` 等)没有工程,`once` 退化为 `always`。用于不改变结果的建议,3.3 的两种用法即属此类。 +- **防止刷屏。** + - 一次运行中同一种类的多个对象合并为一条,列出对象;超过五个时列出前五个并写明其余的数目。 + - `note` 至多两行:第一行陈述事实,第二行陈述写法与文档的位置(`help:`)。 + - `--quiet` 不显示 `note`;`warning` 与 `degraded` 照常显示。 + - `--verbose` 也显示被 `once` 抑制的提示,标为 `(noted before)`,使作者可以再次看到它们。 +- **机器输出。** `--format json` 的诊断包含每一条提示,显示与否都包含,附 `code` 与 `shown`;机器的读者不受显示策略影响。 +- **面的账本。** 登记的代码作为 `notice ` 进入 `mcpp emit surface`,由测试方案 5.16 的账本覆盖。 +- **并发。** 工程没有覆盖整个 `target/` 的锁(锁在各配置目录与构建程序的组目录上)。写入 `target/.mcpp-notices` 为读出、 + 求并集、写入临时文件后改名;两个进程同时写入时可能丢失对方的键,其后果只是该提示再出现一次,因此不另设锁。 +- **位置。** 登记表、策略的判定、键的保存与合并的格式在 `modules/notice` 中,不依赖 `src/`,单元测试在该包的 + `tests/` 中;`src/diag.cppm` 调用它并负责渲染。 + + +### 3.7 契约中的名字:歧义按变量名判定(mcpp#852) + +**现状。** #852 所报告的情形(`openmui.rules` 与 `gongfu.rules` 的清单都写 `rules`)已在 2026.10.30.1(#862)解决: +两者共有的拼写不属于任何一个包,`MCPP_DEP_AMBIGUOUS_RULES` 列出限定名,`dep_dir("rules")` 结束程序并说出它们; +用例 980(`tests/e2e/deps/graph.cpp`)覆盖它,议题没有关闭。以 host-module 的 build-dependency 复现同一情形,同样不再警告。 + +根本的原因仍在:歧义按**拼写**判定,而契约发布的是由拼写**有损地**变换得到的变量名(字母转大写,其余字符转 `_`)。 +`foo-bar`、`foo_bar` 与 `foo.bar` 是三个拼写、一个变量。本地复现(两个 path 依赖 `foo-bar` 与 `foo_bar`): + +``` +warning: build.mcpp: dependency name collides on MCPP_DEP_FOO_BAR_DIR (kept '.../foo-bar', ignored '.../foo_bar') — rename one dependency to disambiguate +warning: build.mcpp: dependency name collides on MCPP_DEP_MCPPLIBS_FOO_BAR_DIR (kept '.../foo-bar', ignored '.../foo_bar') — ... +``` + +每次构建都重复,并且 `dep_dir("foo_bar")` 得到 `foo-bar` 的目录,是一个错误的回答。同一缺陷出现在四个通道中: + +| 通道 | 变量 | 现状 | +|---|---|---| +| 依赖的目录 | `MCPP_DEP__DIR` | 保留第一个,每次构建警告 | +| 依赖的链接形式 | `MCPP_DEP__LINKAGE` | 同上 | +| 依赖的工具 | `MCPP_DEP__BIN_` | 同上 | +| 声明的载荷 | `MCPP_XPKG__`(不带命名空间的拼写) | 两个命名空间声明同一名字时回答先声明的那个,不提示(docs/30 写明) | + +**决定。** 一条规则:**多个提供者对应的一个已发布变量不属于其中任何一个。** + +- **判定的单位是变量。** 引擎先把每个提供者的每个拼写变换为变量,再按变量分组;一个变量有多于一个提供者时, + 该变量不发布,改为发布 `MCPP_DEP_AMBIGUOUS_`(载荷为 `MCPP_XPKG_AMBIGUOUS_`),值为可用的精确拼写, + 以逗号分隔。四个通道调用同一个函数,取代 `fillDepDirs` 中按拼写的分组与 `contract_env` 中的三处「保留第一个」。 +- **读取者。** `dep_dir` 已按变量读取歧义的记录,不需改动;`dep_linkage`、`dep_bin`、`xpkg_dir` 以同一形式读取, + 遇到歧义时结束程序并说出可用的拼写(与 `dep_dir` 相同,写在标准输出,由引擎在程序非零退出时显示)。 +- **构建时不提示。** 一个有歧义的名字只有被询问时才产生错误的结果,被询问时读取者拒绝;因此删除每次构建重复的 + 两种 `collides on` 警告。`contract_env` 在写出变量时核对没有重复,重复即引擎的缺陷,以错误报告。 +- **没有任何拼写可以区分时。** `foo-bar` 与 `foo_bar` 的每个拼写(包括限定名 `mcpplibs.foo-bar` 与 `mcpplibs.foo_bar`) + 都是同一个变量,构建程序无法通过契约找到其中任何一个。读取者的拒绝说明这两个名字只在变量不能表示的字符上不同, + 办法是改名其中一个包(清单没有给依赖取别名的键)。Cargo 也把 `-` 与 `_` 视为同一个包名。 + +**未采纳。** +- **单射的变换**(例如以 `__` 表示 `.`):改变已发布的变量名,违反 AGENTS.md §3「已发布的键不改名」。 +- **另一个按精确拼写索引的文件通道**:同一事实的第二个通道,类型化接口与直接读取环境变量的程序会得到不同的回答。 + +**旧的回答经过弃用期,不立即拒绝。** 今天回答了的名字(变换后冲突的依赖名回答保留下来的第一个;不带命名空间的载荷名 +被两个命名空间声明时回答先声明的)在一个弃用期内照旧回答,并在被询问时警告,指出统一规则下的写法;弃用期结束后按上面的 +规则拒绝。今天已经拒绝的(#862 的共有末段)不变。 + +- **登记。** 弃用登记表(`modules/manifest/src/compat/registry.cppm`)增加一行 `ambiguous-contract-name`:`since` 为 + 2026.11.21.1,`floor` 按 docs/51 的规则取,`canonical` 为「以精确拼写询问(`xpkg_dir("", "")`、限定名), + 没有可区分的拼写时改名其中一个包」。警告的文字、`mcpp why compat` 的记录、docs/51 的表与 `--strict` 的拒绝都由这一行导出, + 与其余的弃用读法相同。 +- **只在被询问时警告。** 弃用期内引擎照旧发布旧的变量,并同时发布 `*_AMBIGUOUS_`。读取者(`dep_dir`、`dep_linkage`、 + `dep_bin`、`xpkg_dir`)见到二者时返回旧的回答,并写出指令 `mcpp:deprecated=ambiguous-contract-name<名字><可用的拼写>`; + 引擎把它记为该包构建程序的一次弃用读法(`record_use`),以弃用的警告显示。没有询问这个名字的程序不受影响,构建时也不再有 + 每次重复的 `collides on`。指令随引擎接口的协议版本加一。 +- **策略。** 弃用的警告是 `always`(3.6):它改变这次构建的结果(回答的是两个提供者中的一个),并在 `floor` 变为错误。 +- **直接读取环境变量的程序。** 不经类型化接口的程序得不到警告;弃用期内它读到的值与今天相同,弃用期结束后变量不再发布。 + docs/30 写明这一点,并建议使用类型化接口。 +- **`floor` 之后。** 旧的变量不再发布,读取者按本节的规则拒绝;登记表的行按 docs/51 删除,单元测试 `test_pm_compat` 在 + `MCPP_VERSION` 到达 `floor` 时指出它。 + +**兼容。** 原来唯一的变量不变。发生冲突的变量与不带命名空间的载荷名在弃用期内回答不变,被询问时警告;docs/30 的两处陈述 +改为弃用期内的行为与之后的规则,CHANGELOG 记在 Deprecated。 + +## 4. 宿主、跨平台、依赖与升级 + +- **宿主。** 构建程序总以宿主工具链编译;交叉构建多一份 `std` 模块;宿主平台为 openkal 时程序在宿主编译、经 `okrun` + 运行。编译读取随宿主的那次编译,与目标无关。 +- **跨平台。** gcc 与 clang 用扫描 depfile,MSVC 用 `sourceDependencies`;`mcpp.build.depfile` 已处理前缀的改写与 Windows + 的大小写折叠。路径按 AGENTS.md §4 比较 `path`、摘要取 `u8string()`,不经 `.string()`。 +- **工作区。** 一组中有多个程序时编译读取按程序记录;成员共用的 `tools/common.hpp` 是常见写法,修复后被跟踪, + 并属于 3.3 的 B。 +- **升级。** 旧的 `programs.json` 没有 `reads`,视为不当前,每个工程重新编译并运行一次;home 中旧身份的运行缓存不再命中, + 由 `mcpp cache gc` 按未使用的天数回收。没有清单键与命令行的改变。 +- **一致性。** 与主构建、单元缓存使用同一个 depfile 解析函数、同一种摘要备忘、同一条「能观测的以观测为准」。 + +## 5. 规范与文档 + +- **SPEC-007 R10.3** 改为:程序的身份覆盖预测部分与编译读取;编译读取取自子构建对程序单元的依赖记录,不含 store 与 + build-dependency 的树中的文件;运行缓存以身份为键;所有程序的预测部分与编译读取都未变时不规划构建程序图。 + 新增 R10.8(用法的提示,3.3)、R10.9(包外读取,3.4)、R10.10(契约中名字的歧义按变量判定,3.7)。状态标为已实现, + 版本 2026.11.21.1。 +- **docs/30(英、中)。** 「Incremental: declared inputs」改为两段:编译读到的文件由引擎观测,运行时读取的文件与变量仍需声明; + 新增「Headers or modules」一节:两种不推荐的用法、推荐的写法与首次构建的提示;「`import std;`」一节指向它。 +- **docs/50(英、中)。** 新的诊断代码与原因标记。 +- **docs/30** 的「A name two dependencies answer to is no package's」改为按变量陈述,并写明只能改名的情形;`xpkg_dir` 一节中 + 「回答先声明的」改为拒绝。 +- **CHANGELOG** 2026.11.21.1 的 Fixed(#991;#852 的按变量判定)、Added(提示的机制、用法的提示、包外读取的警告与发布的拒绝) + 与 Deprecated(`ambiguous-contract-name`:冲突的依赖名与不带命名空间的载荷名的旧回答,被询问时警告)。 + +## 6. 测试 + +| 编号 | 断言 | 位置 | +|---|---|---| +| T1 | #991 的复现:头文件改变后程序重新编译并重新运行,输出新值;删除 `target/.build-mcpp` 之后同样 | `tests/e2e/build-program/` 的新用例(gcc) | +| T2 | 只改变一个与程序无关的源文件时,程序既不编译也不运行(`--verbose` 的 `cached`) | 同上 | +| T3 | 快速路径:头文件改变后的第一次构建不报告 `Finished ... 0.00s` | 同上 | +| T4 | 提示:第一次构建出现一次,第二次构建没有,`mcpp clean` 之后再出现;工作区中两个成员合并为一条 | 同上 | +| T5 | 包外读取:本地构建警告一次;`mcpp publish --dry-run` 拒绝并给出原因标记;文件未被 git 跟踪时同样拒绝 | `tests/e2e/pack/` 或 `publish` 的用例 | +| T6 | 旧格式的 `programs.json`(无 `reads`):重新编译一次,之后当前 | `tests/e2e/build-program/` | +| T7 | MSVC:T1 与 T2 | `tests/e2e/windows/` 的新用例 | +| T8 | clang(macOS):T1 | `tests/e2e/toolchain/` 或 macOS 的作业已运行的程序 | +| U1 | 编译读取的分类:store、build-dependency 的树、本包、包外;Windows 的大小写与前缀 | `tests/unit/` | +| U2 | 提示的策略:`always` 与 `once` 的判定,键的合并写入(两个写者),合并的格式(五个以上的对象),`--quiet`、`--verbose` | `modules/notice/tests/` | +| T9 | 提示的机器输出:被抑制的提示在 `--format json` 中 `shown` 为假;`check_diag_sink.sh` 在本仓库通过 | `tests/e2e/cli/` 与 CI 的门 | +| T10 | 名字的歧义(3.7):依赖 `foo-bar` 与 `foo_bar` 构建时没有 `collides on`;弃用期内 `dep_dir("foo_bar")` 返回旧的回答并给出弃用的警告,`mcpp why compat` 列出它,`--strict` 拒绝;用例 980 的断言保持;`dep_bin` 与 `xpkg_dir` 的同一情形 | `tests/e2e/deps/graph.cpp` | +| U3 | 分组函数:按变量分组、精确拼写的列表、四个通道的变量形式;载荷的不带命名空间的拼写被两个命名空间声明 | `tests/unit/` | + +面的账本(测试方案 5.16)中新的诊断代码与原因标记由 T4、T5 覆盖。迁移到汇点的约 100 处提示保留原有的文字,已有的 e2e 断言 +继续成立;文字因去重而少出现的地方由完整的 e2e 运行核对。 + +## 7. 实施的任务与依赖 + +| 编号 | 任务 | 位置 | 依赖 | +|---|---|---|---| +| P0 | 提示的机制:`modules/notice`(登记表、策略、键的保存、合并的格式),`mcpp.diag` 接入,按 AGENTS.md §4 的四处接入 | `modules/notice`、`src/diag.cppm` | — | +| P0b | 迁移:直接的 `ui::warning`、`ui::note` 与 stderr 的写入改为经由汇点;删除 `print_hint_once` 与 `Said` 的静态集合;`check_diag_sink.sh` 进入 CI 的门 | 第 3.6 节表中的 31 个文件;`.github/tools/` | P0 | +| P1 | SPEC-007 R10.3、R10.8 至 R10.10 | `docs/specs/build-plugins.md` | — | +| P2 | 编译读取:从子构建取程序单元的读取并分类,写入 `programs.json`,进入身份 | `src/build/prepare/host_graph.cpp`;分类函数进入 `modules/` 中与构建程序记录同一个包 | P1 | +| P3 | 快速路径与运行记录读取同一份 `reads`;声明的文件先比较大小与时间 | `src/build/build_program.cppm` | P2 | +| P4 | 提示(3.3)与包外警告(3.4),登记为 `once` | `src/build/prepare/host_graph.cpp`、`modules/notice` 的登记表 | P0、P2 | +| P5 | 发布的拒绝与原因标记 | `src/cli/cmd_publish.cppm`、`src/build/refusal.cppm` | P2 | +| P5b | 名字的歧义按变量判定:一个分组函数供四个通道使用;`dep_linkage`、`dep_bin`、`xpkg_dir` 读取歧义的记录;删除两种 `collides on` 警告,重复的变量为引擎错误(3.7) | `src/build/prepare/graph_provisioning.cpp`、`sources.cpp`、`src/build/build_program.cppm`、`src/build/hostprogram.cppm` | — | +| P6 | 测试 T1 至 T10、U1 至 U3 | 见第 6 节 | P0 至 P5b | +| P7 | docs/30、docs/50(英、中),CHANGELOG;mcpp-contributing 技能:用户可见的提示经由 `mcpp.diag`,策略不是缺省的提示先登记 | — | P0 至 P5 | + +文件的行数与函数的长度按 AGENTS.md §4:`host_graph.cpp` 现为 1,200 余行,P2 与 P4 的新代码以函数加入,单个函数不超过约 400 行。 + +## 8. 未采纳 + +- **每次规划子构建,由 ninja 判断。** 正确,但无事可做的构建从 6 ms 升到 110 ms 以上(3.2 的测量)。 +- **把程序所在包的整棵树并入身份。** 任何源文件的改变都使程序重新运行,运行缓存失去作用;代码生成类的程序代价最大。 +- **以二进制的内容为运行缓存的键。** MSVC 的二进制默认含时间戳与 PDB 的 GUID,每次重新编译都会改变;它只在重新编译之后 + 才有区别,而重新编译已由编译读取决定,是同一信息的第二个来源。 +- **只对引号包含给出警告(#991 的第二个建议)。** 包含是文档允许的用法;修复之后它能正确工作,警告会把正确的用法报告成问题。 +- **每次构建都提示 3.3 的用法。** 重复的提示使真正的警告被忽略;一个工程提示一次已足够让作者知道。 +- **按时间或次数限制提示(「每天一次」「最多三次」)。** 状态与时间相关,同一工程在两台机器上表现不同;`once` 的键只取决于事实。 +- **把「说过」记在 home 中。** home 为多个工程共用,一个工程的提示会因另一个工程而消失;`mcpp clean` 也无法重置。 +- **`once` 用于 `degraded`。** 降级改变这次构建的结果,抑制它即违反 AGENTS.md §2。 +- **观测运行时的读取。** 见 3.1。 + +## 9. 自评 + +- **架构。** 三道闸门的输入由一份记录给出,以后新增的编译输入(例如 `#embed`)不需要再补任何一道闸门。预测部分仍有 + 一个模型,但它只含观测不到的事实。 +- **稳定性。** 观测的边界与 ninja 相同:depfile 不记录查找失败(包含路径中新出现一个同名文件遮蔽旧文件、`__has_include` + 所查的文件后来出现),`-MMD` 不列出系统目录中的头文件(`gcc@system` 之下 `/usr/include` 的改变不被观测,与主构建相同)。 + 这些边界在 docs/30 中写明。 +- **前提,在实施中核对。** MSVC 的程序单元在子构建中写出 `sourceDependencies`,且其路径可以由 `mcpp.build.depfile` 读出; + 不成立时由 P2 为程序单元加上该选项。快速路径读取各组 `programs.json` 之后,复现工程无事可做的构建的用时(现为 6 至 9 ms) + 不增加 1 ms 以上;测量记录在测试方案的实施记录中。 +- **简洁。** 不新增清单键与命令行选项;新增一个记录字段、一个提示的包(取代三处各自的实现)、两个诊断代码与一个原因标记。 +- **一致性。** 提示与弃用读法同一范式(一张表,行导出一切);所有提示经由一个汇点,`--strict`、机器输出与去重对每一条都成立。 + 契约的四个通道以同一条规则处理名字,判定的单位就是发布的单位(变量)。 +- **用户体验。** 修复是无感的;提示一个工程一次、两行、给出写法;发布的拒绝发生在消费方受影响之前。 +- **兼容性。** 旧记录使每个工程重新编译一次;已发布的包不受影响,除非它读取包外的文件,此时它对消费方本来就是坏的。 +- **跨平台。** gcc、clang、MSVC 三种依赖记录都由已有的解析函数处理,测试覆盖 Linux、macOS 与 Windows。 +- **测试覆盖。** 第 6 节的 T1 至 T10 与 U1 至 U3;面的账本覆盖新的代码。 diff --git a/.agents/docs/2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md b/.agents/docs/2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md new file mode 100644 index 000000000..ae5d28c91 --- /dev/null +++ b/.agents/docs/2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md @@ -0,0 +1,376 @@ +--- +subject: ecosystem +status: active +--- + +# 离线、宿主输入与工具链的缺省:按内容取归档的来源、项目的取齐、宿主输入的声明、工具链的解析与锁、缺省版本的规则(mcpp#988、#989、#990) + +**日期:** 2026-11-21 · **版本:** v3(v3 增加 3.10:`dlopen` 的可达性按发起加载的对象判定。2026-11-21 确定第 9 节,取各项的推荐;增加 3.9 命令行中 `@` 的文法与 mcpp-index 中 `compat.slint` 的声明) · **基线:** mcpp 2026.11.20.1(`722ec1b6`)与 #984 的分支, +xlings 2026.11.17.2,xim-pkgindex `c0023dd` · **规范:** SPEC-004 §2(四层取值)、SPEC-006(工具链管理)、SPEC-002(目标侧五层)、 +SPEC-013(平台与层)、home 的初始化(`docs/specs/home-initialization.md`) · **所属:** +[生态的测试与 CI 方案](2026-11-20-test-architecture-upgrade-design.md) 第 15 节;与 +[构建程序的输入由观测给出](2026-11-21-build-program-observed-inputs-design.md) 共用其 3.6 的提示机制。 + +## 1. 范围与原则 + +本方案处理一台机器得到构建所需之物的方式:联网时从哪里取,离线时从哪里取,哪些可以取自宿主,取什么版本。四条原则: + +1. **每个工具只做它自身视角下通用的机制。** xlings 是包与载荷的管理者:取归档、装载荷、管理索引;mcpp 是项目的构建者: + 从清单与锁得出一个项目需要什么。离线、宿主输入与版本的规则落在拥有该事实的一方,另一方只组合它。 +2. **默认密闭。** 工具链、基础层与引导工具取自索引中的包;宿主只在数据声明或用户显式开启时使用(AGENTS.md §2)。 +3. **每个缺省是数据,且有显式的覆盖**,取值按四层:命令行与环境、工程(`mcpp.toml`)、机器(`config.toml`)、引擎(SPEC-004 §2)。 +4. **做得更少从不沉默,正常的路径保持安静。** 提示经由 `mcpp.diag` 的汇点,按子方案一 3.6 的 `always` 与 `once` 两种策略。 +5. **一种机制只实现一次。** 新的功能扩展已有的机制,不在另一处再写一份: + + | 功能 | 复用的机制 | + |---|---| + | 提示与拒绝 | `mcpp.diag` 的汇点与 `modules/notice` 的登记表(子方案一 3.6);原因标记的登记(`refusal.cppm`、`check_reason_tokens.sh`) | + | 旧回答的弃用 | 弃用登记表(`modules/manifest/src/compat/registry.cppm`) | + | 归档的取得 | xlings 的下载器;本地目录是它的一种来源,不是 mcpp 的第二个下载器 | + | 离线的项目闭包 | 构建自身的解析与目标闭包;`fetch` 调用同一个函数 | + | 部分版本与 `@` | `mcpp.xpkg_version`(寻址文法)与 `mcpp.version_req`(要求文法),各一份 | + | 工具链的解析 | `resolve_version_match` 之上的一个解析函数,三处调用 | + | 配套载荷与 kernel-abi 的覆盖 | xlings 的依赖记录;SPEC-013 的 `layers` 词汇 | + | 以 `dlopen` 加载的库 | `runtime.dlopen_libs` 与其运行期的检查(docs/33);可达性的判定为一个函数,链接后的校验与 `self doctor` 共用(3.10) | + | 宿主输入的身份 | 缓存键与 `mcpp.lock` 的现有写入路径 | + +## 2. 现状 + +| 方面 | 现状 | 问题 | +|---|---|---| +| 离线 | 只有 `--offline`、`MCPP_OFFLINE` 与旧的 `MCPP_NO_AUTO_INSTALL`;`[index] auto_refresh = false` 只停止隐式刷新,下载照常 | 没有工程与机器两层(违反原则 3);没有「联网时取齐、离线时构建」的操作,只能联网完整构建一次 | +| 没有网络的机器 | xlings 的下载器在目标文件已存在且 sha256 相符时不下载;索引可以是本地目录(mcpp 的 `path = "…"`、xlings 的 `file://`);home 可以整体复制或作为 upstream | 可行的做法都要求用户了解内部布局:归档须放在 xlings 的下载目录并按 URL 推出的文件名命名(GitHub 的标签归档同名为 `v<版本>.tar.gz`);复制 home 只在相同的绝对路径下可用 | +| 引导工具 | `check_base_init` 无条件要求 home store 中固定版本的 patchelf 与 ninja | 没有覆盖(违反原则 3);以宿主的工具链与 `glibc@system` 构建时并不需要 patchelf,仍被拒绝 | +| 配套载荷 | 编译器旁的 binutils 与 linux-headers 由 `find_sibling_binary` 与 `find_package_in_xpkgs` 取目录遍历遇到的**第一个**版本 | 选择取决于文件系统的顺序,而非工具链载荷声明的依赖。一个同时装有 binutils 2.42 与 2.42.1 的 store 中,构建取到 2.42(目录遍历先遇到它),gcc 的 recipe 声明的是 2.42.1,而 2.42 没有能看到 subos 库的 `ld` 包装 | +| 配套载荷的版本 | 由 gcc 的 recipe 的依赖固定,mcpp 中没有覆盖 | 违反原则 3 | +| 部分版本 | `gcc@16`:`toolchain install` 取索引最新,`toolchain default` 取已安装的最新,构建取已安装的版本 | 一个请求三个回答;docs/20 陈述的「取最新发布」只对第一处成立 | +| 编译器的版本 | gcc 的版本取横幅中最后一段数字 | Fedora 的 `g++ (GCC) 16.2.1 20261029 (Red Hat 16.2.1-3)` 被读作 `3`(#989) | +| 三元组 | vendor 段只认 `unknown`、`pc`、`w64`、`none` | `x86_64-redhat-linux`、`x86_64-suse-linux` 无法解析,原生 sysroot 的设置不生效,且没有提示 | +| 缺省版本 | 首次运行的缺省是 `triple.cppm` 中的常量 | 不是数据(违反原则 3);多项落后于上游的正式发布(3.7 的表) | +| 包的版本号 | binutils 的 `2.42.1` 与 2.42 字节相同,是为重跑安装钩子而新增的键 | 上游没有发布 2.42.1,而上游确有 `.1` 修订(2.45.1、2.46.1);`revision` 机制已经存在 | + +## 3. 设计 + +### 3.1 xlings:按内容取归档的来源 + +**机制。** 一个归档的身份是索引给出的 sha256。xlings 持有一个有序的**来源**列表;每次需要一个归档时,依次询问各来源, +第一个给出 sha256 相符之文件的来源胜出,文件经校验后进入安装。来源有两类: + +- **目录。** 一个本地目录,其中的文件以任意名字、任意层级存放;xlings 对目录建立 sha256 的索引(按路径、大小与修改时间备忘, + 只在三者改变时重新计算)。 +- **网络。** 现有的镜像(`GLOBAL`、`CN` 与自适应的选择),位于全部目录之后。离线时这一类被跳过。 + +现有的「下载目录中已有且 sha256 相符即不下载」成为列表中第一个目录来源(xlings 自己的下载缓存),行为不变。 + +**配置。** `.xlings.json` 的 `archives`:目录的列表。`xlings config --add-archives ` 与 `--rm-archives ` 修改它。 + +**导出。** `xlings download <包@版本>… --to [--with-index]`:把每个包在本宿主上所需的归档(含依赖的闭包), +以及 `--with-index` 时所用索引的快照,写入目录,并写出清单 `bundle.toml`(格式版本、每个归档的包、版本、sha256 与目标、 +每个索引的名字与修订)。目录本身即是一个可用的来源。 + +**导入。** `xlings install --from <包>…`:以目录为本次的第一个来源,以其中的索引快照为本次的索引。 + +**离线时缺少归档。** 一次列出本次需要而所有来源都没有的归档(包、版本、期望的 sha256),而不是在第一个处停止。 + +### 3.2 mcpp:项目的取齐 + +**`mcpp fetch`** 是构建中「取得所需之物」这一步的命令形式。它与构建共用解析与目标的闭包:依赖(按锁)、工具链与其配套载荷、 +目标的基础层、引导工具、构建程序的依赖。构建在缺少时调用同一个函数。 + +```bash +mcpp fetch [--target ]… # 联网:把所需之物装入本机的 home +mcpp fetch --to [--target ]… # 写出一个自描述的目录(bundle) +mcpp fetch --from # 从目录装入本机的 home;之后目录可以移走 +``` + +- **`--to`** 调用 xlings 的 `download --to --with-index` 写出载荷与索引快照(含 mcpp 的各索引),另以 `git bundle` 写出每个 git 依赖在 + 锁中的提交,并把这些记入同一份 `bundle.toml`。 +- **`--from`** 以目录的索引快照解析,调用 `xlings install --from` 安装,并把 git bundle 导入 `~/.mcpp/git`。 +- **稳定性的承诺。** 一个版本的 `--to` 写出的目录,同一版本与更新的版本的 `--from` 都接受;目录内部的布局不是接口。 + +**离线的键。** `[build] offline = true`,在 `mcpp.toml` 与 `config.toml` 中同名同义(与 `memory_threshold` 相同的形式), +四层取值:`--offline` 与 `MCPP_OFFLINE` 为覆盖层。离线时网络来源被跳过,任何需要网络的动作被拒绝并说明原因。 + +**长期的本地来源。** `config.toml` 的 `[xlings] archives = [""]`(与 `[xlings] upstream` 并列),环境变量 `MCPP_ARCHIVES`; +mcpp 把它传给 xlings。它是机器的事实:写在 `mcpp.toml` 中时可以使用,并按 3.3 给出一次可移植性的提示。 + +**与 upstream 的关系。** `[xlings] upstream` 是已安装载荷的来源(按链接发布,不复制);`archives` 与 bundle 是归档的来源(校验后安装)。 +两者并列,各自不变。 + +**实施修订(3.2)。** + +- **一次运行的键,一个机制。** `[build] offline`、`[build] allow_host_tools`、`[xlings] archives` 作用于整个运行及其 + 启动的程序,各自在开始时陈述为一个变量(`MCPP_OFFLINE`、`MCPP_ALLOW_HOST_TOOLS`、`MCPP_ARCHIVES`):调用者给出的变量为 + 覆盖层,其次工程(prepare 在加载配置之前陈述),再次机器(配置加载时、引导之前陈述)。`mcpp::effects::state_key` + 只在没有更高层陈述时写入,并记下哪一层回答;拒绝的补救由此说明如何回到联网(`offline_remedy`)。 +- **离线的安装来源。** 写明了归档目录时,离线运行把 home 缺少的载荷交给 xlings(`XLINGS_NET_UNREACHABLE`),只询问下载缓存与 + 这些目录;没有写明时不询问任何来源,与离线一向的含义相同。后者使规划性的运行(`mcpp self doctor` 以离线描述一次构建) + 不因下载缓存中恰好有归档而安装;doctor 另以空的 `MCPP_ARCHIVES` 运行。两种拒绝由同一对函数给出 + (`offline_without_sources`、`offline_unavailable`),每条安装路径——载荷、依赖、`[xlings]` 声明、首次运行的工具链——共用。 +- **用到的输入。** 载荷在交出时记录(`Fetcher::resolve_xpkg_path`、依赖的读取、`[xlings]` 声明),git 克隆在作为依赖根时记录; + `--to` 写出这些与构建工具,闭包由 xlings 补全(Linux 上含 patchelf)。 +- **`bundle.toml` 中 mcpp 的表。** `[mcpp]`:`format`、`created_by`、`host`、`targets`、`packages`;`[[git]]`:`url`、 + `commit`、`clone`(`~/.mcpp/git` 下的目录名,其键与宿主无关)、`file`。mcpp 改写时整体替换自己的表,保留其余。 +- **`--from` 的顺序。** 读取并判断(格式、宿主与目标的警告、锁)→ 以不引导的方式加载配置 → 放置索引快照 → 以该目录为归档目录 + 完成 home 的引导(引导工具由快照解析)→ `xlings install --from` → 导入克隆 → 规划。快照放在 home 没有该索引之处,或该处是 + 先前的快照或制品的树;git 工作树与其他目录是 home 自己的,保持不变。已拥有的索引(mcpplibs)此后的刷新照常暂存并交换。 +- **锁的比较。** 锁中来自索引的包与工具链条目,各自须在 bundle 的 `packages` 中有同名同版本的一项,否则为 + `bundle-older-than-lock`,在安装任何东西之前拒绝。 + +### 3.3 宿主输入:声明、身份与提示 + +**宿主输入的类别。** 每一类都是显式的开启,开启的方式在四层中: + +| 类别 | 开启 | +|---|---| +| 宿主的工具链 | `[toolchain] default = { path = … }`、`msvc@system`(行的数据) | +| 宿主的基础层 | `sysroot = "glibc@system"` | +| 宿主的库 | `[build] allow_host_libs = true`、`MCPP_ALLOW_HOST_LIBS` | +| 依赖的工具 | `[tools.overrides]`、`MCPP_TOOL__` | +| **引导工具** | 新增 `[build] allow_host_tools = ["ninja", "patchelf"]`,四层取值;被开启的工具在 `PATH` 中查找 | + +**按需检查。** `check_base_init` 改为在用到时检查:patchelf 只在安装需要重定位的载荷与打包时需要,ninja 只在执行构建图时需要。 +一个以宿主的工具链与 `glibc@system` 构建的工程不再因缺少 patchelf 被拒绝。 + +**身份(实施中修订)。** 宿主的工具链与基础层的身份(驱动的指纹、版本、三元组,C 库的版本)已在缓存键中:它们改变编译命令与对象。 +引导工具(ninja、patchelf)不改变任何编译命令与对象,不进入缓存键。宿主输入的身份不写入 `mcpp.lock`:锁随工程提交, +记入一台机器的路径与版本会使每台机器的构建改写它。身份改变时提示一次(提示的键含身份,见下文)。 + +**提示。** 每一类开启在一个工程中提示一次(`once`,键为类别与对象),两行,陈述来源、风险与回到密闭的写法,例如: + +``` +note: ninja is this machine's (/usr/bin/ninja 1.12.1), not the store's; another machine may build + differently, and it lacks the correction mcpp pins (ninja#2662). help: docs/25, "Host inputs" +``` + +- 风险取自数据:引导工具与工具链的行陈述已知的修正与限制,提示读出它们,代码中不写具体的版本。 +- 开启写在 `mcpp.toml` 时,提示另说明它只在这类机器上成立,建议写入 `config.toml`。 +- 宿主的身份相对锁改变时再提示一次(新的身份即新的键)。 + +### 3.4 工具链的解析与锁 + +- **一个解析函数。** 工具链的请求只有一个解析:部分版本(`gcc@16`、`gcc@16.2`)取**本地索引**中满足它的最新正式发布, + 需要时安装;预发布(`17.0.0-<日期>`)只由完整的版本选中(现状不变)。离线且本地索引不可用时取已安装的最新,并以 `degraded` 说明。 + `toolchain install`、`toolchain default` 与构建调用同一个函数。 +- **写明的部分版本进入锁。** 工程写明的部分版本解析后写入 `mcpp.lock` 的 `[toolchain."<请求>"]`,请求写作它所要求的包 + (`xim:gcc@16`,同一请求在不同目标上对应不同的包),值为解析出的版本。之后的构建按锁取,`mcpp update` 删除记录、重新解析。 + 完整版本即其自身的记录,不写入:一个写明完整版本而没有依赖的工程因此不因升级而多出 `mcpp.lock`。`--toolchain` 与依赖方为宿主 + 工具选择的工具链属于这一次构建,不写入、不比较。 + 旧的锁没有工具链的条目:普通的构建写入一次;`--frozen` 与 `--locked` 之下不写锁,按请求解析并以 `note` 说明锁缺少这一条目, + 不拒绝,使已有的 CI 不因升级而失败。配套载荷(3.5)与宿主输入(3.3)的身份不进入这一条目:前者由工具链载荷的版本与其安装记录 + 决定,后者进入缓存键并按 3.3 提示。 +- **没有写明的跟随缺省。** 工程没有写工具链时,取机器的缺省(`config.toml`,首次运行时由引擎的缺省写入)或目标行的缺省, + 不写入锁。它所给出的工具链与工程上一次所用的不同时(机器的缺省被改变、引擎的缺省随 mcpp 的升级而改变、目标行的缺省改变), + 一个工程提示一次: + + ``` + note: the default toolchain (`mcpp toolchain default`) is now gcc@16.2.0; this project last built with gcc@16.1.0 + help: to keep a version, state it in mcpp.toml: [toolchain] default = "gcc@16.1.0" + ``` + + 「上次所用的缺省」由提示机制记住(子方案一 3.6 的 `target/.mcpp-notices` 增加记住的事实);新工程与 `mcpp clean` 之后没有可比的值,不提示。 + +### 3.5 配套载荷:由工具链声明,可以覆盖 + +- **版本取自声明。** binutils、linux-headers 等配套载荷的版本取自工具链载荷声明并由 xlings 解析的依赖(载荷的安装记录), + 不取目录的顺序。找不到声明的版本时是错误,指名缺少的包与版本。 +- **binutils 属于工具链。** 它是编译器驱动的汇编器与链接器,GCC 在构建时按它探测汇编器的能力。覆盖写在工具链的陈述中 + (9.4 的表形式,`companions = { binutils = "xim:binutils@<版本>" }`);版本低于 GCC 载荷构建时所用的 binutils 时拒绝并说明原因。 +- **linux-headers 不在工程中覆盖(实施中修订)。** 原定以 `layers = { kernel-abi = … }` 覆盖。实施中核对:缺省的 glibc 基础层是 home 的 + subos sysroot,内核头文件以符号链接放入其中,为该 home 的全部工程共用;按工程覆盖需要按工程的 sysroot,而 `layers` 的值是图中的包, + 不是索引的载荷。因此头文件的版本随工具链载荷的声明,经工具链包的修订改变(第 4 节 xim-pkgindex 一行);陈述了目标基础层 + (`sysroot = …`)的工程取该基础层的头文件。 +- binutils 的覆盖是索引中的包,版本固定、密闭,因此不属于宿主输入,不提示。 + +### 3.6 编译器的识别 + +- **版本由驱动的机器可读查询给出。** gcc 用 `-dumpfullversion`(GCC 7 起),不可用时用 `-dumpversion`;clang 用 `-dumpversion`; + 横幅只用于判定家族与驱动的指纹。Fedora 的分支快照因此是 16.2.1,与 GCC 的编号规则一致(`X.Y.1` 为 `X.Y.0` 之后分支上的开发)。 +- **vendor 段是任意的词。** 与 GNU `config.sub` 相同:架构之后、系统段之前的段是 vendor,取任何词;系统段之后的段仍按词表严格判定。 + `x86_64-redhat-linux` 即 `x86_64-linux-gnu`。 +- **本地工具链的三元组无法解析时报告**(`degraded`,说明哪些按目标的设置因此没有生效),不静默地以原文继续。 + +### 3.7 缺省版本的规则 + +**规则。** mcpp 的每个缺省(工具链、配套载荷、基础层、引导工具)取发布时上游最新的正式发布,经 mcpp 的 CI 与 canary 验证; +更新的版本与实验版本由用户经 xlings 的包显式陈述。缺省是 `builtin.toml` 中的数据;发版的规程检查每个缺省与上游的最新正式发布。 + +**包的版本号。** xim-pkgindex 中一个包的版本是上游的版本;同一上游版本的重新打包以 `revision` 表示,不新增上游没有的版本号。 +已发布的键不改名;`binutils@2.42.1` 保留,此后的版本按本规则。 + +**本轮的刷新。** + +| 缺省 | 现状 | 上游最新的正式发布 | 需要的载荷 | +|---|---|---|---| +| gcc(Linux x86_64、Windows 的 MinGW) | 16.1.0 | 16.2.0 | glibc 的 gcc 已有(x86_64、aarch64);musl-gcc、mingw-gcc、mingw-cross-gcc 的 16.2.0 待构建;gcc 16.2.0 以新的 binutils 重建一个修订 | +| gcc(其余的 Linux 架构) | 15.1.0-musl | 16.2.0 | musl-gcc 16.2.0 | +| llvm(macOS、Windows、Linux aarch64) | 23.1.3 | 23.1.6 | 各宿主的 llvm 23.1.6 | +| binutils | 2.42(键 2.42.1) | 2.47 | x86_64、aarch64(aarch64 不含硬链接,Android 拒绝 `link(2)`) | +| linux-headers | 5.11.1 | 6.18 longterm 或 7.3 stable(9.5) | 一个 | +| patchelf | 0.18.0 | 0.19.3 | x86_64、aarch64 | +| ninja | 1.12.1-mcpp.1 | 1.13.2 | 1.13.2 含 ninja#2662 的修正时取上游,否则为 1.13.2 的 mcpp 修订 | + +**实施中修订(2026-11-21)。** 一个包的缺省取它所服务的每个宿主都已发布的最新正式发布,内置表的一行取它所服务的每个宿主 +都能安装的最新正式发布(`check_defaults.sh` 按平台比较)。实测: +- llvm-project 的 llvmorg-23.1.6 没有发布 macOS 的二进制,xim-pkgindex 的 llvm 23.1.6 只有 Linux x86_64、Linux aarch64 与 + Windows x86_64;llvm 的缺省包括 macOS,服务所有宿主的行(bare-metal、openkal、mingw-llvm)也命名 llvm,因此 llvm 的缺省与 + 各行保持 23.1.3,一台机器上只有一个 llvm。 +- musl 的行同时服务自 x86_64 的交叉(`xim:aarch64-linux-musl-gcc`)与 Windows 上的 Canadian cross + (`xim:x86_64-linux-musl-gcc`),二者只发布了 16.1.0,因此该行保持 16.1.0;没有 glibc gcc 的 Linux 宿主的首次缺省取原生的 + musl-gcc 16.2.0。 +- ninja 1.13.2 不含 ninja#2662 的修正(v1.13.2 的 `Plan::EdgeFinished` 与 `Plan::NodeFinished` 仍按迭代器遍历,ninja#2840 + 未合入),缺省为 `ninja@1.13.2-mcpp.1`。 +- 切换的缺省:gcc 16.2.0(Linux x86_64、没有 MSVC 的 Windows、MinGW 的行)、musl gcc 16.2.0(其余的 Linux 架构)、 + ninja 1.13.2-mcpp.1、patchelf 0.19.3;gcc 的配套载荷 binutils 2.47 与 linux-headers 6.18.64 由 gcc 的载荷声明。 + +### 3.9 命令行中 `@` 的文法 + +`@<版本>` 是寻址文法,清单中的版本字符串是要求文法,二者各有一种读法: + +| 写法 | 读法 | 例 | +|---|---|---| +| `<包>@<版本>`(命令行、`--template`、`--list-templates`、工具链、xlings 的引用) | 按书写的精度的前缀:写出的各段相等,更细的段任意,取其中最高者 | `@1.19.0` 选 1.19.0.2;`@16` 选 16.x 的最新 | +| `<包>@=<版本>` | 一个键的精确匹配 | `@=1.19.0.1` | +| `<包>@<运算符><版本>`(`^`、`~`、`>=` 等) | 要求文法,与清单相同 | `@^1.19` | +| 清单的 `"<版本>"` | 要求文法:不带运算符为 caret;等于一个已发布的键时指名该键(现状不变) | `"1.19.0"` | + +- 这与 xlings 的 `@`、mcpp 已有的工具链部分版本、Spack 的 `@` 一致;`cargo install --version` 的精确读法在四段的键之下得不到写出的版本,不采用。 +- 精确匹配落空而存在更细的键时,错误写出它们与两种写法:`1.19.0 is published as 1.19.0.2, 1.19.0.1: @1.19.0 selects 1.19.0.2, @=1.19.0.1 pins one`。 +- 第四段在索引中有两种含义:上游本身的四段版本(Vulkan SDK 的 `1.4.321.0`),与包装层的修订(mwrap 的 `1.19.0.2`)。前缀的读法对二者都成立; + mcpp-index 的贡献说明陈述这两种用法,修订只递增、不复用。 +- SPEC-001 §4.4 的模板表与第 350 行的命令行引用改为本节的表述。 + +### 3.10 以 `dlopen` 加载的库:按发起加载的对象判定可达 + +**加载器的规则。** glibc 为一个对象的依赖与它发起的 `dlopen` 查找库时:该对象有 `DT_RUNPATH` 时,只搜索 `LD_LIBRARY_PATH`、 +它自己的 `DT_RUNPATH` 与系统目录,**不搜索任何 `DT_RPATH`**,包括可执行文件的;没有 `DT_RUNPATH` 时,依次搜索它自己与其加载链上各对象 +(直到可执行文件)的 `DT_RPATH`。以两个最小的对象验证:库带 `DT_RUNPATH` 时,可执行文件 `DT_RPATH` 中的目录对该库的 `dlopen` 不可见; +库不带时可见。 + +**现状的缺陷。** mcpp 把运行期的搜索闭包写入可执行文件的 `DT_RPATH`,subos 的库目录在最后(`flags.cppm` 的 `ldRuntimeFallback`), +并以此为每个对象都可达的前提;`resolution.json` 的校验与 `mcpp self doctor` 的 `dlopen` 检查按全部目录的并集判断。xlings 安装的预构建 +载荷以其 recipe 声明的运行依赖写出 `DT_RUNPATH`,因此一个载荷中的库以 `dlopen` 加载、而其 recipe 未声明的库,在任何 mcpp 构建的程序中都 +找不到,校验却报告通过。`xim:slint` 的 `libslint_cpp.so` 即此情形:它以 `dlopen` 加载 X11、Wayland、xkbcommon 与 GL,`DT_RUNPATH` +只有 slint、fontconfig 与 ABI 的目录;程序在有显示的机器上启动时失败。 + +**决定。** +- **可达性按对象判定。** 运行期的校验对每一个要求(`runtime.dlopen_libs` 的 soname、`DT_NEEDED`)从提出它的对象出发,按上面的规则计算 + 它可见的目录:对象有 `DT_RUNPATH` 时取其 `DT_RUNPATH` 与系统目录,否则取加载链上的 `DT_RPATH`。一个 `dlopen` 要求归属于声明它的包, + 从该包的库出发判定。`resolution.json` 的 `search` 记录每个对象可见的目录;`mcpp self doctor` 与链接后的校验使用同一个函数, + 替换两处各自按并集的判断(原则 5)。 +- **不可达即报告。** 一个必需的 soname 从提出它的对象不可达时,以 `degraded` 报告,代码 `dlopen-unreachable`:写出对象、soname、 + 该对象可见的目录,以及办法——提供该库的包须是声明这个对象的载荷的运行依赖(xlings 据此写入 `DT_RUNPATH`)。不改写载荷, + 不设置 `LD_LIBRARY_PATH`(它被宿主的子进程继承,已测得使 `xdg-open` 等程序退出)。 +- **包的声明。** `compat.slint` 声明 `runtime.dlopen_libs`(第 4 节 mcpp-index 一行所列的 soname)与运行的能力;`xim:slint` 的 recipe 把提供它们的包 + 列为运行依赖。两者分别属于 mcpp-index 与 xim-pkgindex(第 4 节)。 +- **文档。** docs/91 的「the farm is reachable per object (DT_RPATH)」改为本节的规则;docs/33 陈述 `dlopen` 的要求按发起加载的对象判定。 + +### 3.8 提示与拒绝 + +| 情形 | 级别、策略 | 代码或原因标记 | +|---|---|---| +| 离线时所有来源都没有所需的归档 | 错误;一次列出全部,并给出在联网的机器上执行的 `mcpp fetch --to` | `artifact-unavailable-offline` | +| bundle 的索引快照中没有锁所记的版本 | 错误;列出这些版本 | `bundle-older-than-lock` | +| bundle 不含本宿主或所选目标的载荷 | `warning`,`always` | `bundle-lacks-target` | +| 未开启离线而网络不可用,使用了本地的来源 | 沿用现有的「刷新失败、使用本地」的警告,写明来源 | — | +| 宿主输入的开启(3.3) | `note`,`once` | `host-input` | +| 宿主输入的身份相对锁改变 | `note`,`once`(新的身份即新的键) | `host-input-changed` | +| 工程所跟随的缺省工具链改变(3.4) | `note`,`once` | `toolchain/default-changed` | +| 机器的路径写在 `mcpp.toml` 中(`archives`、宿主输入) | `note`,`once` | `machine-path-in-manifest` | +| 离线且本地索引不可用,部分版本取已安装的最新 | `degraded`,`always` | — | +| 本地工具链的三元组无法解析 | `degraded`,`always` | — | +| 必需的 `dlopen` soname 从提出它的对象不可达(3.10) | `degraded`,`always` | `dlopen-unreachable` | +| 覆盖的 binutils 低于 GCC 载荷构建时所用 | 错误 | `companion-too-old` | +| `fetch --to`、`--from` 完成 | 一行汇总:包的数目、大小、目标 | — | + +## 4. 各仓库的范围与次序 + +| 仓库 | 范围 | 依赖 | +|---|---|---| +| xlings | 3.1:来源列表、目录来源的 sha256 索引、`archives` 配置、`download --to --with-index`、`install --from`、离线时一次列出缺少的归档;文档;发版 | — | +| xlings-res(载荷的构建) | 3.7 表中待构建的载荷 | — | +| xim-pkgindex | 3.7 的版本:binutils 2.47、linux-headers、patchelf 0.19.3、gcc 16.2.0 的重建修订与各家族的 16.2.0、llvm 23.1.6、ninja;gcc 的依赖改到这些版本;包版本号的规则写入其贡献说明;`xim:slint` 声明提供上述 `dlopen` 库的包 | xlings-res;xlings 的新版(recipe 若使用新的字段) | +| mcpp(#984) | 3.2 至 3.6、3.8;`builtin.toml` 的缺省(3.7);mcpp 所捆绑与所固定的 xlings 版本 | xlings 的发版;xim-pkgindex 合入并发布 | +| mcpp-index | mtest 0.11.2 的条目;`compat.slint` 声明它以 `dlopen` 加载的库(X11、xcb、Xcursor、Xrandr、Xi、xkbcommon、Wayland、EGL 与 GL 的 soname)与运行的能力,与 `compat.glfw` 相同的形式;贡献说明陈述第四段的两种用法(3.9) | — | + +xlings 与 xlings-res 的工作可以并行;xim-pkgindex 在载荷发布之后合入;mcpp 的缺省在 xim-pkgindex 发布之后切换,其余各项与它们并行实施。 + +## 5. 规范与文档 + +- **SPEC-004 §2** 的键:`[build] offline`、`[build] allow_host_tools`、`[xlings] archives`(机器层)。 +- **SPEC-006** 工具链管理:一个解析函数;写明的请求进入锁;配套载荷取自声明;覆盖;缺省是数据及其规则;编译器版本的查询。 +- **SPEC-002 与 docs/21** 三元组的 vendor 段。 +- **home 的初始化** 的规范:按需检查引导工具;来源与 bundle。 +- **使用文档(英、中)**:docs/05(离线与 `mcpp fetch`)、docs/20(解析、锁、配套载荷、覆盖)、docs/21(vendor 段)、docs/22(kernel-abi 的覆盖与运行时内核)、 + docs/25(来源、bundle、宿主输入)、docs/51(缺省版本的规则与本轮的版本)、docs/90(离线构建 mcpp 自身)、docs/50(新的代码与原因标记)。 +- **技能**:mcpp-release 增加缺省与上游最新正式发布的核对;xlings 的文档陈述来源与 bundle。 +- **CHANGELOG** 各仓库各自的条目。 + +## 6. 测试 + +| 编号 | 断言 | 位置 | +|---|---|---| +| X1 | 目录来源按 sha256 命中,文件名任意;sha256 不符的文件不被采用;来源的次序;目录的索引在文件改变时更新 | xlings 的测试 | +| X2 | `download --to --with-index` 写出的目录可被 `install --from` 在无网络的命名空间中使用;缺少时一次列出全部 | xlings 的测试(`unshare -n`) | +| F1 | `mcpp fetch --to` 后在无网络的命名空间中 `mcpp fetch --from` 与 `mcpp build --offline` 成功;含 git 依赖与构建程序的依赖 | mcpp e2e | +| F2 | 离线时缺少归档:一次列出全部并给出命令;bundle 比锁旧时的错误 | mcpp e2e | +| F3 | mcpp 自身:干净的 home 先 `fetch --to`,再在无网络的命名空间中构建 | CI 的作业 | +| H1 | `allow_host_tools` 使用 `PATH` 中的 ninja 并提示一次;第二次构建不提示;`mcpp clean` 之后再提示 | mcpp e2e | +| H2 | 以宿主的工具链与 `glibc@system` 构建时,store 中没有 patchelf 不被拒绝 | mcpp e2e | +| H3 | 宿主输入的身份改变时提示一次 | mcpp e2e | +| R1 | `gcc@16` 在三处得到同一个回答(索引的最新);写入锁;`mcpp update` 推进 | mcpp e2e 与单元测试 | +| R2 | 没有写工具链的工程在缺省改变后提示一次 | mcpp e2e | +| C1 | 配套载荷取声明的版本:store 中有多个版本时结果与目录顺序无关 | mcpp 单元测试与 e2e | +| C2 | binutils 与 kernel-abi 的覆盖生效;过旧的 binutils 被拒绝 | mcpp e2e | +| V1 | `-dumpfullversion` 的版本;Fedora 横幅的回归;vendor 段为任意词的三元组 | mcpp 单元测试(`modules/toolchain-model/tests`) | +| D1 | 缺省取自 `builtin.toml`;发版规程的核对脚本列出落后于上游的缺省 | mcpp 单元测试与工具的测试 | +| L1 | 带 `DT_RUNPATH` 的库以 `dlopen` 加载一个只在可执行文件 `DT_RPATH` 中可见的库:校验报告 `dlopen-unreachable`;不带 `DT_RUNPATH` 时可达;`mcpp self doctor` 与链接后的校验给出同一结论 | mcpp e2e 与单元测试 | +| A1 | `@1.19.0` 选 1.19.0.2,`@=1.19.0.1` 选 1.19.0.1,`@16` 选 16.x 的最新;精确落空时的错误写出更细的键 | mcpp 单元测试与 e2e(`mcpp new --template`) | +| P1 | 新的代码与原因标记由面的账本覆盖(测试方案 5.16) | 面的账本 | + +## 7. 未采纳 + +- **vendor(把依赖的源码复制进仓库)。** path 依赖与 upstream 已覆盖其用途;复制违背「依赖而不复制」。 +- **每个工程一个 home。** home 是机器的 store,为多个工程共用;工程的需要由键与锁表达。 +- **整体复制 home。** 载荷重定位后带绝对路径,只在相同路径下可用;体积为全部而非所需。 +- **由 mcpp 把归档放进 xlings 的下载目录。** 依赖 xlings 不承诺的内部布局,且只对 mcpp 有用。 +- **按文件名的本地镜像。** 标签归档的文件名在包之间重复;sha256 是索引已经给出的身份。 +- **缺省自动跟随索引的最新。** 用户的缺省会随索引的更新而改变,且未经 mcpp 的 CI 验证。 +- **binutils 作为独立的层。** 它属于工具链,GCC 的能力在构建时依它而定;五层是封闭的集合(SPEC-002)。 +- **按时间或次数限制提示。** 见子方案一第 8 节。 + +## 8. 自评 + +- **架构。** xlings 增加一个来源列表与一对导出导入,mcpp 增加 `fetch` 与三个键;没有一方依赖另一方的内部布局。`fetch` 与构建共用解析, + 离线只是来源的不同。工具链的解析由三处合为一处,配套载荷由目录顺序改为声明。 +- **稳定性与可复现。** 离线取得的字节与联网相同(sha256);锁记录工具链与宿主的身份;配套载荷的选择与机器无关。 +- **简洁。** 用户的操作是 `mcpp fetch --to`、`mcpp fetch --from` 与一个 `offline` 键;不需要了解 store、下载目录与归档的文件名。 +- **用户体验。** 缺少的一次列全并给出命令;宿主输入与缺省的改变各说一次;正常的离线构建没有额外的输出。 +- **兼容与无感升级。** 联网的用户行为不变;`--offline`、`MCPP_OFFLINE` 照旧;锁增加的工具链条目在旧锁缺少时写入一次;缺省的改变有提示, + 需要旧版本的工程写明它。 +- **跨平台。** 来源按内容,不受文件名的大小写与非法字符影响;`--target` 可以重复,一个 bundle 可含多个目标;Windows 与 macOS 走同一条路径。 +- **一致性。** 来源与 upstream 并列;三个键遵循四层;提示经由一个汇点;版本号遵循上游。 +- **测试覆盖。** 第 6 节在 xlings 与 mcpp 两侧覆盖每一项机制,离线的断言在无网络的命名空间中成立;面的账本覆盖新的代码。 +- **代价。** 3.7 的刷新需要构建多个工具链载荷(gcc 的三个家族、llvm 的四个宿主),是本方案中耗时最长的部分。 + +## 9. 已确定(2026-11-21,取各项的推荐) + +9.1 **范围。** xlings 的 PR 与发版、xlings-res 的载荷、xim-pkgindex 的 PR 与 mcpp 的改动都属于本轮(第 4 节)。 + +9.2 **命令与键的名字。** `mcpp fetch [--to|--from]`;`[build] offline`;`[build] allow_host_tools`;`[xlings] archives` 与 `MCPP_ARCHIVES`; +xlings 的 `archives`、`download --to --with-index`、`install --from`。 + +9.3 **bundle 的内容。** 载荷、依赖的源码归档、索引的快照、git 依赖的 `git bundle` 与 `bundle.toml`,即一个目录是完整的离线输入。 + +9.4 **binutils 的覆盖的写法。** 工具链的陈述现有字符串形式(`"gcc@16.2.0"`)与表形式(`{ path = … }`、`{ configure = … }`), +`[toolchain]` 的其余键与子表是系统的名字(`linux`、`windows`)。候选: +- (a) 表形式增加包的一种:`default = { package = "gcc@16.2.0", companions = { binutils = "xim:binutils@2.47" } }`,`[toolchain.linux]` 等处同样可写; +- (b) 与工具链分开的表 `[toolchain.companions]`:与 `[toolchain.<系统>]` 共用一个命名空间,读者无法区分子表与系统名,不采用; +- (c) 不在 mcpp 中提供,以 xlings 包的修订发布配套的组合(以新的 binutils 发布 gcc 的新修订):版本仍是数据,但用户不能在工程中覆盖,违反原则 3。 +推荐 (a):与 `{ path = …, tools = … }` 同为工具链陈述的表形式,`companions` 的值是包引用,与 `layers` 的值同一种写法。 + +9.5 **linux-headers 的缺省。** 6.18 longterm(推荐:与工具链的刷新节奏一致,头文件的接口在长期版本内稳定)或 7.3 stable(最新的接口)。 + +9.6 **锁中的工具链条目。** 只记写明的请求;没有写明的跟随引擎的缺省并提示(3.4)。 + +9.7 **本轮刷新的版本表(3.7)。** 包括 llvm 23.1.6 与 ninja 在内;ninja 取上游还是 mcpp 修订,由 #2662 的修正是否在 1.13.2 中决定,实施中核对。 diff --git a/.agents/docs/2026-11-21-package-features-query-design.md b/.agents/docs/2026-11-21-package-features-query-design.md new file mode 100644 index 000000000..4988e21e4 --- /dev/null +++ b/.agents/docs/2026-11-21-package-features-query-design.md @@ -0,0 +1,108 @@ +--- +subject: ecosystem +status: active +--- + +# 包的特性作为机器可读的回答:一个目录项,两种来源,三种状态(mcpp#992) + +**日期:** 2026-11-21 · **版本:** v1 · **基线:** mcpp 2026.11.21.1 的分支(#984) · **规范:** SPEC-001 §3.5(描述符的两种形式)、 +SPEC-004(清单的键)、SPEC-010 §5(只读查询不安装)、docs/50(`kindVersion` 内只增加字段) · **所属:** +[生态的测试与 CI 方案](2026-11-20-test-architecture-upgrade-design.md) 第 15 节的子方案三。 + +## 1. 问题 + +mcpp-ide 只显示 mcpp 陈述的事实,不读索引的描述符(mcpp-ide#16、#17:`features = [...]` 中的补全、悬停、未知名字的诊断、 +包详情中的勾选)。mcpp 的机器输出不含包的特性:`mcpp search --format json` 只有 `description`、`id`、`index`、`licenses`、 +`name`、`namespace`、`repo`、`versions`;`mcpp xpkg parse` 与 `mcpp why` 也没有。 + +## 2. 现状的测量 + +- **一个模型。** `mcpp.toml`(`toml.cppm`)与描述符的内联表(`xpkg.cppm` 的 `synthesize_from_xpkg_lua`)解析到同一个 + `Manifest`:`featuresMap`(特性 → implies,`default` 项为缺省集合)、`featureForwards`(`<依赖>/<特性>`)、 + `featureRequires`、`featureProvides`。不需要新的读取。 +- **两种来源。** 本机 mcpplibs 索引的 342 个描述符中,197 个为内联形式,特性在索引中(30 个写有特性);81 个为指针形式、 + 66 个未写 `mcpp` 字段(按约定为指针形式),共 147 个,特性在载荷的 `mcpp.toml` 中。内联表对描述符的所有版本共用; + 指针形式的每个版本各有自己的清单。 +- **读取路径已经收敛。** `mcpp.pm.descriptor` 的 `descriptor_payload_present` 判断载荷是否在,`descriptor_manifest` 给出 + 内联表或载荷中清单的位置(mcpp#960)。构建在读取指针形式的清单时另做一步:归档中的 workspace 成员接受其 workspace + 的继承(`graph_load` 的 `loadFrom`)。 +- **没有描述。** 两种来源都没有特性的 `description`;表形式的键检查(#540)把它报为未知键。 + +## 3. 决定 + +### 3.1 一个目录项 + +`mcpp.manifest` 增加 `feature_catalog(const Manifest&)`:每个特性一项,`name`、`default`(在缺省集合中)、`implies` +(本包的特性)、`enables`(`<依赖>/<特性>`)、`description`(可空),按名字排序;`default` 本身不是一项。所有输出由它给出, +JSON 的形状只在一处写出。 + +### 3.2 一条读取路径 + +`mcpp.pm.package_manifest` 的 `installed_manifest` 给出索引包某个已安装版本的清单:`descriptor_manifest` 之后,指针形式的 +清单按构建的方式读取,含归档中 workspace 的继承与描述符的命名空间。构建(`graph_load`)与查询都调用它,二者对同一个包不会 +给出不同的特性。`workspace_listing` 与 `inherit_as_workspace_member` 由 prepare 移入该模块;错误是 `ManifestError` +(`at` 为读不出的清单),由调用方以自己的主语写出,构建的消息不变。 + +### 3.3 三种状态 + +一个包的一个版本的回答: + +```json +"features": { + "version": "0.0.5", + "state": "declared", + "from": "payload", + "list": [ { "name": "gui", "default": true, "implies": ["render"], "enables": ["imgui/docking"], + "description": "..." } ] +} +``` + +| `state` | 含义 | `list` | +|---|---|---| +| `declared` | 回答取自索引(内联形式)或已安装的载荷中的清单 | 特性的列表,没有特性时为 `[]` | +| `not-installed` | 指针形式,该版本的载荷不在 store 中 | `null` | +| `unreadable` | 载荷在,清单读不出;另附一条诊断 | `null` | + +`[]` 只表示「声明了零个特性」。未知的回答为 `null`:只读列表的客户端若把未安装读成没有特性,会把每个合法的名字诊断为 +未知。`from` 为 `index` 或 `payload`。 + +不是 mcpp 包的条目不输出 `features`。判据取自已有的两个事实:描述符写有 `mcpp` 字段(任一形式)的是 mcpp 包;不写该字段的 +描述符,在提供方的官方索引(`official_index_status` 的目录)中是提供方自身的载荷,在其他索引中按 SPEC-001 §3.5 的约定为 +指针形式。本机的目录中,官方索引的 342 个 `xim` 条目与 `luban` 等命名空间的条目不输出;mcpplibs 索引的 66 个未写字段的 +描述符按指针形式回答。 + +### 3.4 不安装 + +查询不下载、不安装(SPEC-010 §5):已安装就读取,未安装就以 `not-installed` 回答。取得载荷的是构建与 `mcpp fetch`。 + +### 3.5 两个输出 + +- **`mcpp search --format json`**:每个包的 `features`,为最新版本(`versions` 的第一项)的回答。一个包的查询即精确名字的 + search。`mcpp.search` 的 `kindVersion` 保持 1(只增加字段)。 +- **`mcpp why features [--format json]`**(`mcpp.why.features`):本工程解析出的每个包,`package`、`root`(是否为根)、 + `manifest_dir`,以及 `features` 的 `list` 与 `active`(本次构建激活的特性)。规划时每个包都已在 home 中,列表总是已知的。 + 数据记录在 `resolution.json` 的 `graph` 中(`features` 为激活的特性,与构建程序读取的图文档同名同形;`feature_list` + 为目录项),命令读取该记录,与 `mcpp why deps` 读取同一记录相同。构建程序的图文档不加目录项:它是构建程序重跑键的一部分, + 描述的改动不应使构建程序重跑。不给话题的 `mcpp why` 不含该话题,与 `flags`、`platform` 相同。 + +`--protocol-version` 的 `features` 增加 `package-features`。 + +### 3.6 描述 + +表形式的特性增加 `description`(字符串),与 `[package] description` 同名同义;描述符的内联表的特性项同样接受。 +短形式(`x = ["a"]`)没有描述。键检查的已知键随之增加。 + +## 4. 不做的 + +- 不在发布时把特性抄进指针形式的描述符:两份事实会分离,描述符也不按版本区分。 +- 不在查询中取得载荷。 + +## 5. 验证 + +| # | 判据 | 层 | 用例 | +|---|---|---|---| +| Q1 | `feature_catalog` 的缺省(含闭包)、implies、enables、description 与排序 | 单元 | `FeatureCatalog/*`(modules/manifest) | +| Q2 | `description` 在两种来源中被读取且为字符串,表形式不再报未知键;两种来源对同一陈述给出同一目录 | 单元 | `FeatureCatalog/*` | +| Q3 | 三种状态:内联形式为 `declared`/`index`;指针形式已安装为 `declared`/`payload`(含归档中 workspace 的继承),未安装为 `not-installed`,清单读不出为 `unreadable`,后两者 `list` 为 `null` | 单元、e2e | `PackageFeatures/*`;`index::features/992 …never installed`(search 前后各一次,中间以 `mcpp fetch` 安装) | +| Q4 | `mcpp why features`:路径依赖的 `list`、`active` 与描述,JSON 与文本 | e2e | `index::features/992 why features…` | +| Q5 | `--protocol-version` 列出 `package-features` 与 `mcpp.why.features` | e2e | `cli::golden/1123` | diff --git a/.agents/docs/README.md b/.agents/docs/README.md index 37e6dce75..eaec1c17d 100644 --- a/.agents/docs/README.md +++ b/.agents/docs/README.md @@ -18,7 +18,7 @@ superseded_by: 2026-09-07-....md # when status is superseded --- ``` -367 records. +371 records. ## By subject @@ -36,6 +36,7 @@ Records that declare one. Everything else is listed by date below. ### build-program +- [构建程序的输入由观测给出:编译读到的文件、三道闸门的同一份记录、头文件的用法提示、提示的统一机制、契约中名字的歧义(mcpp#991、#852)](2026-11-21-build-program-observed-inputs-design.md) — active - [构建依赖的一条管线:配置与产物两轴、包的实例、库接口、兼容系统与工作空间收尾(#797)](2026-10-11-build-dependencies-one-pipeline-design.md) — active - [Four upstream asks from a UI framework: what each one is under mcpp's design, and the combined plan](2026-09-13-four-upstream-asks-from-a-ui-framework.md) — landed @@ -91,6 +92,8 @@ Records that declare one. Everything else is listed by date below. ### ecosystem +- [包的特性作为机器可读的回答:一个目录项,两种来源,三种状态(mcpp#992)](2026-11-21-package-features-query-design.md) — active +- [离线、宿主输入与工具链的缺省:按内容取归档的来源、项目的取齐、宿主输入的声明、工具链的解析与锁、缺省版本的规则(mcpp#988、#989、#990)](2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md) — active - [生态规则收敛方案:2026.11.18.1 之后十个 issue 的五类成因、七项决定与实施范围](2026-11-19-ecosystem-rule-convergence-design.md) — landed - [生态架构优化方案:四个平面的收敛、已合入项的纠正与新系统的准入(mcpp × openkal × xlings)](2026-11-12-ecosystem-architecture-optimization-design.md) — active - [测试与依赖解析补充方案:mcpp、mcpp test、mtest、mcpp-ide](2026-11-10-test-and-resolution-supplement-design.md) — active @@ -173,6 +176,10 @@ Records that declare one. Everything else is listed by date below. - [SDK toolchains, the payload/engine seam, and openkal across iOS, Android and Web](2026-09-11-sdk-toolchains-and-ios-local-verification.md) — landed - [Where a platform's knowledge belongs: iOS, Android and Web across the engine, the index and the plugins](2026-09-11-platform-targets-design-review.md) — active +### test + +- [生态的测试与 CI 方案:mtest 的框架与运行控制、mcpp 自身的测试系统、CI 的形态(mcpp#986、#976、#927、#802)](2026-11-20-test-architecture-upgrade-design.md) — active + ### toolchain - [目标基座与执行宿主:sysroot 的通用模型、运行器契约、自备输入,以及同批的发布产物路径(#832)](2026-10-23-target-base-and-execution-host-design.md) — landed @@ -206,6 +213,10 @@ Records that declare one. Everything else is listed by date below. ### 2026-11 +- [包的特性作为机器可读的回答:一个目录项,两种来源,三种状态(mcpp#992)](2026-11-21-package-features-query-design.md) — active +- [离线、宿主输入与工具链的缺省:按内容取归档的来源、项目的取齐、宿主输入的声明、工具链的解析与锁、缺省版本的规则(mcpp#988、#989、#990)](2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md) — active +- [构建程序的输入由观测给出:编译读到的文件、三道闸门的同一份记录、头文件的用法提示、提示的统一机制、契约中名字的歧义(mcpp#991、#852)](2026-11-21-build-program-observed-inputs-design.md) — active +- [生态的测试与 CI 方案:mtest 的框架与运行控制、mcpp 自身的测试系统、CI 的形态(mcpp#986、#976、#927、#802)](2026-11-20-test-architecture-upgrade-design.md) — active - [程序起点模型:启动者、两层入口与 `main` 的三种陈述(mcpp#845、mcpp#981)](2026-11-20-program-start-model.md) — landed - [生态规则收敛方案:2026.11.18.1 之后十个 issue 的五类成因、七项决定与实施范围](2026-11-19-ecosystem-rule-convergence-design.md) — landed - [第二部分之后的补充方案:带修正的 ninja、模块重编译的级联、工作区发布与本轮遗留项](2026-11-17-ninja-cascade-publish-supplement-design.md) — active diff --git a/.agents/skills/mcpp-contributing/SKILL.md b/.agents/skills/mcpp-contributing/SKILL.md index 0f45b1288..2b52343da 100644 --- a/.agents/skills/mcpp-contributing/SKILL.md +++ b/.agents/skills/mcpp-contributing/SKILL.md @@ -150,6 +150,19 @@ E2E 并不保证完全离线:部分脚本需要工具链、索引或 capabilit 按 CI 等价方式设置 `MCPP_HOME`、镜像和其他 capability 后再运行;不要让缓存命中 或空 workspace 选择冒充行为覆盖。 +C++ 端到端用例的写法(`tests/support`,设计 2026-11-20): + +- 等待被测程序给出的事件(一行输出、一个文件、退出),不等待时间;`sleep_for` 与 shell `sleep ` 的次数只减不增 + (`.github/tools/check_waits.sh`、`waits.baseline`)。只为改变修改时间的等待改用 `s::touch(file)`。 +- 断言只在其 home 不被别人改写时成立的用例(回放的构建、快速路径)调用 `s::own_home()`,在自己的 home 中运行 mcpp, + 而不持有 `machine`;改写 home 配置的用例写自己 home 的 `config.toml`,不写共享 home 的。 +- 继承的 home(`home::inherited`、`_inherit_toolchain.sh`)以用户的 home 为 upstream,不链接其载荷,不复制 subos。 +- 每个 E2E 组带 `s::home_fixture`:共享的 home 是运行夹具,其填充记在夹具上。 +- 形式本身是契约的输出(`--protocol-version`、`emit surface`)以 golden 文件整段比较;改变它的拉取请求以 + `MTEST_GOLDEN=update` 更新文件,审阅读取更新给出的改动。 +- 程序写出的 `warning:` 与 `error:` 行由用例以 `e2e::expect_diagnostic` 声明;kit 以 `report` 方式统计未声明的行。 +- 不变量(无改动的重建、第二个工程)以构建记录(`MCPP_BUILD_RECORD`)计数,以指标与界限陈述。 + ### 4. 提交 PR **提交信息前缀**:`feat:` / `fix:` / `refactor:` / `test:` / `docs:` / `chore:` @@ -357,6 +370,15 @@ UTF-8 ACP(65001),这类用例会静默变成永远绿的装饰品。参见 「只有它自己和它声明的依赖」这个配置里——那是根构建从不产生的配置,也是唯一 能抓到「悄悄依赖了未声明之物」的地方。 ⚠️ `mcpp test -p ` 对没有测试的成员**退 0** +- **用户可见的提示只有一个汇点 `mcpp.diag`**:`warning`、`degraded`、`note`、`notice` + 与收尾的 `closing`/`on_failure`。`mcpp::ui::warning`、`note`、`tip` 只由 `mcpp.diag` + 渲染使用,其他地方直接调用或向 stderr 写 `warning:`/`note:`/`tip:` 都由 + `.github/tools/check_diag_sink.sh` 在 CI 中拒绝。领域即代码:`build-program/reads-outside` + 在机器输出中是 `MCPP_BUILD_PROGRAM_READS_OUTSIDE` +- **策略不是缺省(每次都说)的提示先登记**:`modules/notice` 的 `kKinds` 一行陈述领域、 + 策略(`once`:一个工程每个对象只说一次,记在 `target/.mcpp-notices`)与文档位置,调用 + `mcpp::diag::notice(..., subject, ...)`;多个对象合并为一条时用带 `scope` 与对象列表的形式。 + 改变这次构建结果的事实(降级、弃用的读法、未知的键)不登记为 `once` - E2E 测试应声明所需 capability,并使用隔离的 `MCPP_HOME`;需要网络/索引的脚本 不得被描述为完全离线 - 不确定方向时先在 Issue 讨论再动手 diff --git a/.agents/skills/mcpp-release/SKILL.md b/.agents/skills/mcpp-release/SKILL.md index d4df8a3d3..03d56b26f 100644 --- a/.agents/skills/mcpp-release/SKILL.md +++ b/.agents/skills/mcpp-release/SKILL.md @@ -72,6 +72,14 @@ git checkout main && git pull origin main gh run list --branch main --limit 3 ``` +缺省版本与索引比对(docs/51「缺省版本」):每个 `[[default]]` 应为上游最新的正式发布。 +`behind` 的一项在发版前移动(载荷先进入 xim-pkgindex),或在 CHANGELOG 中陈述保留的原因; +上游的发布页另行核对,索引可能落后于上游。 + +```bash +bash .github/tools/check_defaults.sh +``` + 以分支保护和 `gh pr checks ` 显示的 actual required checks 为准。 在 main 上监控当前运行时,检查 `ci` 这一个工作流的运行(它包含各平台、e2e 分片、 交叉构建与 `e2e-coverage`);known-red 的腿(名字里带 issue 号)允许失败。跳过或 @@ -108,6 +116,11 @@ gh pr create --title "chore: bump version to $NEW_VERSION" --body "..." 然后推送(标签在 push 启动运行时读取)。这样 PR 自己的运行就是全集,结果在 PR 的检查中可见。不要以 `gh workflow run ci.yml --ref <分支>` 另开一次运行,也不要取消 PR 的运行(见 mcpp-contributing 第 5 节)。 +全集运行通过时留下 `verified-`(设计 2026-11-20 §5.22),它的 `release-form` 作业留下四个发布归档。 +版本号的两处、CHANGELOG 与文档中 `@NEXT@` 的替换都在这次运行之前提交:之后的任何改动都改变树,标记随之失效。 +合入(squash)之后 main 的树与被验证的树相同,main 的运行只登记、保存缓存,不再运行测试,**不取消它**。 +合入与打 tag 之间 main 移动过时,发布照常构建,摘要写明原因。 + ### 3. 创建并推送 tag ```bash @@ -115,21 +128,26 @@ git tag "v$NEW_VERSION" git push origin "v$NEW_VERSION" ``` -Tag push 会自动触发 `release.yml` workflow。 +Tag push 会自动触发 `release.yml` workflow。它的 `find` 作业由 tag 得到树:有运行验证过这棵树、且仍保留 +四个 `release-*` 归档(七天)时,`promote` 提升它们(核对 `.sha256` 与 Linux 程序报告的版本),不再构建; +否则运行 `release-build.yml` 的四个构建。运行摘要写明 "Promoted" 或 "Built" 及原因。 ### 4. 监控 Release CI -Release workflow 的四个平台构建**同时开始**,各自把归档留作 artifact;`create-release` -在四个构建都通过后一次创建带全部归档的 GitHub release(设计 2026-11-14): +Release workflow 先由 `find` 决定提升还是构建。构建时四个平台构建(`release-build.yml`)**同时开始**, +各自把归档留作 artifact;`create-release` 在四个构建或 `promote` 通过后一次创建带全部归档的 GitHub +release(设计 2026-11-14、2026-11-20 §5.22): | Job | 平台 | 产物 | 依赖 | |-----|------|------|------| | `canaries` | — | 生态项目(xlings、mcppls)用候选版本构建 | 仅手动发布且勾选 `canaries` 时运行 | -| `build-release` | Linux x86_64 | `mcpp-X.Y.Z-linux-x86_64.tar.gz`、源码包、`mcpp.lua`、`install.sh` | 无 | -| `build-linux-aarch64` | Linux aarch64(交叉) | `mcpp-X.Y.Z-linux-aarch64.tar.gz` | 无 | -| `build-macos` | macOS ARM64 | `mcpp-X.Y.Z-macosx-arm64.tar.gz` | 无 | -| `build-windows` | Windows x86_64 | `mcpp-X.Y.Z-windows-x86_64.zip` | 无 | -| `create-release` | — | GitHub release、覆盖全部归档的 `SHA256SUMS` | 四个构建(及运行时的 canaries) | +| `find` | — | tag、提交、树与验证它的运行 | 无 | +| `promote` | — | 被验证运行的四个归档,作为 `release-promoted` | `find` 找到运行时 | +| `build-release` | Linux x86_64 | `mcpp-X.Y.Z-linux-x86_64.tar.gz`、源码包、`mcpp.lua`、`install.sh` | `find` 未找到运行时(`build`) | +| `build-linux-aarch64` | Linux aarch64(交叉) | `mcpp-X.Y.Z-linux-aarch64.tar.gz` | `find` 未找到运行时(`build`) | +| `build-macos` | macOS ARM64 | `mcpp-X.Y.Z-macosx-arm64.tar.gz` | `find` 未找到运行时(`build`) | +| `build-windows` | Windows x86_64 | `mcpp-X.Y.Z-windows-x86_64.zip` | `find` 未找到运行时(`build`) | +| `create-release` | — | GitHub release、覆盖全部归档的 `SHA256SUMS` | 四个构建或 `promote`(及运行时的 canaries) | | `release-manifest` | — | `mcpp-release.json` | `create-release` | | `publish-ecosystem` | — | 镜像到 xlings-res 双端 + 开索引 bump PR | `release-manifest` | diff --git a/.gitattributes b/.gitattributes index 647206538..116c35487 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,3 +2,5 @@ tests/fixtures/openkal-hosted-threads/src/main.cpp.in text eol=lf # The rows' snapshot is compared line by line with what the engine answers, in its own bytes. tests/fixtures/target-rows.snapshot text eol=lf +# The golden outputs are compared whole with what mcpp writes, in its own bytes (tests/e2e/cli/golden.cpp). +tests/e2e/golden/* text eol=lf diff --git a/.github/actions/bootstrap-mcpp/action.yml b/.github/actions/bootstrap-mcpp/action.yml index fd8b420ff..edf4c576c 100644 --- a/.github/actions/bootstrap-mcpp/action.yml +++ b/.github/actions/bootstrap-mcpp/action.yml @@ -33,7 +33,7 @@ inputs: # `package.name`, so one of the two was simply unreachable — and which one # depended on the machine, which is why CI failed on `compat:lua` on # Windows and `mcpplibs.capi:lua` on Linux. Never pin below that. - default: '2026.11.17.2' + default: '2026.11.21.1' outputs: sandbox-key: diff --git a/.github/actions/e2e-cases/action.yml b/.github/actions/e2e-cases/action.yml index 4ee4bc4a3..ca0fd64d6 100644 --- a/.github/actions/e2e-cases/action.yml +++ b/.github/actions/e2e-cases/action.yml @@ -53,8 +53,7 @@ runs: HOST: ${{ inputs.host }} run: | set -euo pipefail - run=$(gh run list -R "$GITHUB_REPOSITORY" --workflow ci.yml --branch main --status success --limit 1 \ - --json databaseId -q '.[0].databaseId') + run=$(bash .github/tools/main_run.sh) # main's, or the run it registered (design 2026-11-20 §5.22) [ -n "$run" ] || { echo "::error::no successful CI run of main holds the tests"; exit 1; } rm -rf "$RUNNER_TEMP/mcpp-tests" gh run download "$run" -R "$GITHUB_REPOSITORY" -n "mcpp-tests-$HOST" -D "$RUNNER_TEMP/mcpp-tests" @@ -89,4 +88,13 @@ runs: for c in "${cases[@]}"; do tr -d '\r' < "$log" | grep -qxF " ok $c" || { echo "::error::$c did not pass (a skip or a failure; its line above)"; status=1; } done + # A program can fail beside cases that passed (its exit status, a fixture, a case it did + # not report): the run's records say which, and the job keeps no store. + if [ "$status" != 0 ]; then + run=$(ls -t target/mtest/runs 2>/dev/null | head -1) + for f in programs cases fixtures; do + [ -n "$run" ] && [ -f "target/mtest/runs/$run/$f.ndjson" ] || continue + echo "::group::the run's $f.ndjson"; cat "target/mtest/runs/$run/$f.ndjson"; echo "::endgroup::" + done + fi exit "$status" diff --git a/.github/actions/e2e-mtest/action.yml b/.github/actions/e2e-mtest/action.yml index 34c95c5e4..57e445cdb 100644 --- a/.github/actions/e2e-mtest/action.yml +++ b/.github/actions/e2e-mtest/action.yml @@ -41,8 +41,7 @@ runs: HOST: ${{ inputs.host }} run: | set -euo pipefail - run=$(gh run list -R "$GITHUB_REPOSITORY" --workflow ci.yml --branch main --status success --limit 1 \ - --json databaseId -q '.[0].databaseId' 2>/dev/null || true) + run=$(bash .github/tools/main_run.sh) # main's, or the run it registered (design 2026-11-20 §5.22) if [ -z "$run" ]; then echo "no successful run of main: the suite is divided by count"; exit 0; fi rm -rf "$RUNNER_TEMP/e2e-stores" if ! gh run download "$run" -R "$GITHUB_REPOSITORY" -p "e2e-store-$HOST-*" -D "$RUNNER_TEMP/e2e-stores" 2>/dev/null; then diff --git a/.github/actions/mcpp-unit/action.yml b/.github/actions/mcpp-unit/action.yml index eec47323b..2ab02cdfa 100644 --- a/.github/actions/mcpp-unit/action.yml +++ b/.github/actions/mcpp-unit/action.yml @@ -41,6 +41,20 @@ inputs: description: the tests to run, as `mcpp test` patterns separated by spaces; empty for all required: false default: '' + cases: + description: the cases to run, one `mcpp test --case` pattern per line; empty for all + required: false + default: '' + jitter: + description: > + the most a contention point waits, in milliseconds (MTEST_JITTER, SPEC-017 O12; design + 2026-11-20 §5.19), with `jitter-seed`; empty for none + required: false + default: '' + jitter-seed: + description: the seed of the jitter's waits (MTEST_JITTER_SEED) + required: false + default: '' mcpp: description: the mcpp that runs the tests; empty for this commit's (MCPP_FRESH) required: false @@ -70,8 +84,7 @@ runs: run: | set -uo pipefail py=python3; command -v python3 >/dev/null 2>&1 || py=python - run=$(gh run list -R "$GITHUB_REPOSITORY" --workflow ci.yml --branch main --status success --limit 1 \ - --json databaseId -q '.[0].databaseId' 2>/dev/null || true) + run=$(bash .github/tools/main_run.sh) # main's, or the run it registered (design 2026-11-20 §5.22) if [ -z "$run" ]; then echo "no successful run of main: the tests run in the order they are listed"; exit 0; fi d="$RUNNER_TEMP/unit-history" rm -rf "$d" @@ -114,6 +127,9 @@ runs: PATTERNS: ${{ inputs.patterns }} TIMEOUT: ${{ inputs.timeout }} RUNS: ${{ inputs.mcpp }} + CASES: ${{ inputs.cases }} + JITTER: ${{ inputs.jitter }} + JITTER_SEED: ${{ inputs.jitter-seed }} run: | set -euo pipefail args=(test --timeout "$TIMEOUT" --split auto) @@ -137,6 +153,12 @@ runs: [ -z "$SINCE" ] || args+=(--since "$SINCE") # shellcheck disable=SC2206 -- the patterns are words [ -z "$PATTERNS" ] || args+=($PATTERNS) + while IFS= read -r c; do [ -z "$c" ] || args+=(--case "$c"); done <<< "$CASES" + fi + # A search for an order defect, never a verdict of its own (design 2026-11-20 §5.19). + if [ -n "$JITTER" ]; then + export MTEST_JITTER="$JITTER" MTEST_JITTER_SEED="${JITTER_SEED:-0}" + echo "jitter: up to $JITTER ms, seed $MTEST_JITTER_SEED" fi echo "mcpp ${args[*]}" "${RUNS:-$MCPP_FRESH}" "${args[@]}" diff --git a/.github/actions/project-test/action.yml b/.github/actions/project-test/action.yml new file mode 100644 index 000000000..1f0762fd7 --- /dev/null +++ b/.github/actions/project-test/action.yml @@ -0,0 +1,128 @@ +name: mcpp project test +description: > + The tests of a project built with mcpp, as a downstream repository's CI runs them (design + 2026-11-20 §7.4): the mcpp the project pins, its home restored by the project's identity (read + only on a pull request, saved by a push to the default branch), `mcpp test`, the run's store + kept as an artifact, and a summary of the run: its result, its slowest cases and the slowest + edges of its builds (SPEC-017 B4). + + Use: `- uses: mcpp-community/mcpp/.github/actions/project-test@main` after `actions/checkout`. + +inputs: + mcpp: + description: > + the mcpp version to install; empty for the one the project's .xlings.json names, else the + newest release + required: false + default: '' + args: + description: the arguments of `mcpp test`, as words (`-p member --profile release`) + required: false + default: '' + home-key: + description: > + what identifies the project's home beside the system and the mcpp version (its manifests and + lock, by default) + required: false + default: '' + name: + description: the name of the store's artifact + required: false + default: mcpp-test-store + +runs: + using: composite + steps: + - name: The mcpp the project pins + id: version + shell: bash + env: + WANTED: ${{ inputs.mcpp }} + run: | + set -euo pipefail + v="$WANTED" + if [ -z "$v" ] && [ -f .xlings.json ]; then + v=$(sed -n 's/.*"mcpp": *"\([^"]*\)".*/\1/p' .xlings.json | head -1) + fi + echo "version=${v:-latest}" >> "$GITHUB_OUTPUT" + echo "mcpp ${v:-latest}" + + # The home is the tests' run fixture (design 2026-11-20 §5.10): restored by its identity, the + # system, the mcpp version and the project's manifests. A pull request reads it and adds + # nothing; a push to the default branch saves what its run filled. + - name: Restore the project's home + id: home + uses: actions/cache/restore@v4 + with: + path: ~/.mcpp + key: mcpp-project-home-${{ runner.os }}-${{ runner.arch }}-${{ steps.version.outputs.version }}-${{ inputs.home-key || hashFiles('**/mcpp.toml', '**/mcpp.lock', '.xlings.json') }} + restore-keys: | + mcpp-project-home-${{ runner.os }}-${{ runner.arch }}-${{ steps.version.outputs.version }}- + + - name: Install mcpp + shell: bash + env: + VERSION: ${{ steps.version.outputs.version }} + XLINGS_NON_INTERACTIVE: '1' + run: | + set -euo pipefail + if ! command -v xlings > /dev/null 2>&1; then + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + export PATH="$HOME/.xlings/subos/current/bin:$PATH" + fi + spec=mcpp; [ "$VERSION" = latest ] || spec="mcpp@$VERSION" + for attempt in 1 2 3; do + xlings update > /dev/null 2>&1 || true + if xlings install "$spec" -y -g; then break; fi + [ "$attempt" = 3 ] && { echo "::error::$spec could not be installed"; exit 1; } + sleep 30 + done + mcpp --version + + - name: mcpp test + shell: bash + env: + ARGS: ${{ inputs.args }} + run: | + set -euo pipefail + # shellcheck disable=SC2086 -- the arguments are words + mcpp test $ARGS + + - name: Save the project's home + if: ${{ always() && github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) && steps.home.outputs.cache-hit != 'true' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp + key: mcpp-project-home-${{ runner.os }}-${{ runner.arch }}-${{ steps.version.outputs.version }}-${{ inputs.home-key || hashFiles('**/mcpp.toml', '**/mcpp.lock', '.xlings.json') }} + + - name: Keep the run's store + if: always() + uses: actions/upload-artifact@v4 + with: + name: ${{ inputs.name }} + path: target/mtest + if-no-files-found: ignore + include-hidden-files: true + retention-days: 14 + + - name: The run in the summary + if: always() + shell: bash + run: | + set -uo pipefail + run=$(ls -t target/mtest/runs 2>/dev/null | head -1) + [ -n "$run" ] || { echo "no run was kept" >> "$GITHUB_STEP_SUMMARY"; exit 0; } + d="target/mtest/runs/$run" + { + echo "## mcpp test" + jq -r '"\(.status // "?"): \(.counts // {} | to_entries | map("\(.value) \(.key)") | join(", ")) in \((.ms // 0) / 1000 | floor) s on \(.machine // "?")"' "$d/run.json" 2>/dev/null + echo + echo "| slowest case | ms |"; echo "|---|---|" + jq -rs 'sort_by(-(.ms // 0)) | .[:10][] | "| \(.program)/\(.id) | \(.ms // 0 | floor) |"' "$d/cases.ndjson" 2>/dev/null + if [ -s "$d/builds.ndjson" ]; then + echo + echo "| slowest edge | package | ms |"; echo "|---|---|---|" + jq -rs 'map(select(.output)) | sort_by(-(.ms // 0)) | .[:10][] | "| \(.output) | \(.package) | \(.ms // 0 | floor) |"' "$d/builds.ndjson" + fi + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/actions/setup-macos-llvm/action.yml b/.github/actions/setup-macos-llvm/action.yml index eb34dd43a..5a1e6aa6c 100644 --- a/.github/actions/setup-macos-llvm/action.yml +++ b/.github/actions/setup-macos-llvm/action.yml @@ -15,7 +15,7 @@ inputs: # Floor imposed by the index, not a routine bump — see # .github/actions/bootstrap-mcpp/action.yml for why 0.4.69 is required # (two packages named `lua` in one repo need openxlings/xlings#381). - default: '2026.11.17.2' + default: '2026.11.21.1' image: description: > The runner label the job runs on (macos-15, xcode-27). It is part of the diff --git a/.github/actions/use-built-mcpp/action.yml b/.github/actions/use-built-mcpp/action.yml index 7a87ff6fc..eb45b463a 100644 --- a/.github/actions/use-built-mcpp/action.yml +++ b/.github/actions/use-built-mcpp/action.yml @@ -29,16 +29,33 @@ inputs: description: The mirror xlings and mcpp use in this job. required: false default: GLOBAL + form: + description: > + `host`: the binary of build.yml (`mcpp-built-`). `release`: the + program of the archive release-build.yml kept for the host + (`release-`), as a release ships it -- ci.yml's `full` tier + tests it, so that a release promotes what a run tested (design + 2026-11-20 §5.22). + required: false + default: host runs: using: composite steps: - name: Download this commit's mcpp (${{ inputs.host }}) + if: ${{ inputs.form != 'release' }} uses: actions/download-artifact@v4 with: name: mcpp-built-${{ inputs.host }} path: ${{ runner.temp }}/mcpp-built + - name: Download this commit's release archive (${{ inputs.host }}) + if: ${{ inputs.form == 'release' }} + uses: actions/download-artifact@v4 + with: + name: release-${{ inputs.host == 'macos-arm64' && 'macosx-arm64' || inputs.host }} + path: ${{ runner.temp }}/mcpp-release + - name: Use this commit's mcpp shell: bash - run: bash "$GITHUB_ACTION_PATH/use.sh" "${{ inputs.host }}" "${{ inputs.mirror }}" + run: bash "$GITHUB_ACTION_PATH/use.sh" "${{ inputs.host }}" "${{ inputs.mirror }}" "${{ inputs.form }}" diff --git a/.github/actions/use-built-mcpp/use.sh b/.github/actions/use-built-mcpp/use.sh index 5602dd018..d2c0ec0ec 100644 --- a/.github/actions/use-built-mcpp/use.sh +++ b/.github/actions/use-built-mcpp/use.sh @@ -1,17 +1,37 @@ #!/usr/bin/env bash # The body of the use-built-mcpp action; see action.yml for what it provides. # -# Usage: use.sh +# Usage: use.sh [host|release] set -euo pipefail host="$1" mirror="$2" +form="${3:-host}" dir="$RUNNER_TEMP/mcpp-built" case "$host" in windows-*) exe=mcpp.exe; dir="$(cygpath -u "$dir")" ;; *) exe=mcpp ;; esac + +# The release form: the program of the host's versioned archive, as install.sh +# unpacks it, placed where the built binary would be. +if [ "$form" = release ]; then + rel="$RUNNER_TEMP/mcpp-release" + [ "${host#windows-}" != "$host" ] && rel="$(cygpath -u "$rel")" + archive=$(find "$rel" -maxdepth 1 -type f \( -name 'mcpp-*-*.tar.gz' -o -name 'mcpp-*-*.zip' \) \ + | grep -E '/mcpp-[0-9][^/]*-[a-z]' | head -1 || true) + [ -n "$archive" ] || { echo "::error::the release archive of $host is not in the artifact"; ls -la "$rel" || true; exit 1; } + unpacked="$RUNNER_TEMP/mcpp-release-root" + mkdir -p "$unpacked" "$dir" + case "$archive" in + *.zip) (cd "$unpacked" && unzip -q "$archive") ;; + *) tar -xzf "$archive" -C "$unpacked" ;; + esac + wrapper="$(basename "$archive")"; wrapper="${wrapper%.tar.gz}"; wrapper="${wrapper%.zip}" + cp "$unpacked/$wrapper/bin/$exe" "$dir/$exe" + echo "the program under test is the release form: $(basename "$archive")" +fi bin="$dir/$exe" if [ ! -f "$bin" ]; then echo "::error::the artifact mcpp-built-$host holds no $exe" @@ -74,6 +94,14 @@ if [ -n "${XLINGS_BIN:-}" ]; then fi MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" self config --mirror "$mirror" +# THE RECIPES OF THE CURRENT INDEX. A restored sandbox keeps the index it was +# saved with, and the tests' homes read this home's index and name this home an +# upstream (tests/support, tests/e2e/_inherit_toolchain.sh). A recipe older than +# xim-pkgindex c0023dd wrote its `config()` hook through an upstream's link +# entry: measured, gcc's specs and binutils' ld wrapper of this home named a +# case's deleted home after one case, and every later link through them failed. +MCPP_VENDORED_XLINGS="${XLINGS_BIN:-}" "$bin" index update + # THE TOOLCHAIN THE BUILD USED IS INSTALLED, AS IT WAS WHEN EVERY JOB BUILT. # A job that built mcpp itself installed this toolchain as a side effect, and # the steps after the build relied on it without saying so: measured on the diff --git a/.github/tools/check_defaults.sh b/.github/tools/check_defaults.sh new file mode 100755 index 000000000..237e5855b --- /dev/null +++ b/.github/tools/check_defaults.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env bash +# check_defaults.sh — each default of the built-in table against the newest +# release the package index publishes (design 2026-11-21, offline and toolchain +# defaults §3.7; the release procedure, .agents/skills/mcpp-release). +# +# WHY THIS EXISTS +# +# A default is the newest release of its upstream when mcpp is released. The +# defaults are data (`[[default]]` of modules/toolchain-model/data/builtin.toml), +# and nothing compared them with what the index publishes: binutils 2.42 and +# patchelf 0.18.0 stayed defaults through releases that their upstreams had +# superseded (mcpp#988). This lists every default beside the newest release of +# its package in an xim-pkgindex checkout and fails when one is behind. The +# newest release of a package is the newest that every platform its defaults +# name publishes (docs/51): a version one of them lacks is not a default. +# +# A release key is a version without a prerelease part; an mcpp revision of an +# upstream release (`1.12.1-mcpp.1`) counts as that release. A toolchain request +# names its payload: `gcc@-musl` is `musl-gcc@`, and gcc for a +# `*-windows-gnu` target is `mingw-gcc`. +# +# Usage: bash .github/tools/check_defaults.sh [repo_dir] + +set -uo pipefail +# A host pattern (`linux-*`, `*`) is a word, never a file name. +set -f + +INDEX="${1:?usage: check_defaults.sh [repo_dir]}" +ROOT="${2:-$(cd "$(dirname "$0")/../.." && pwd)}" +DATA="$ROOT/modules/toolchain-model/data/builtin.toml" +[ -f "$DATA" ] || { echo "error: $DATA not found"; exit 2; } +[ -d "$INDEX/pkgs" ] || { echo "error: $INDEX is not an xim-pkgindex checkout (no pkgs/)"; exit 2; } + +# The release part of a version: `1.12.1-mcpp.1` -> 1.12.1; a prerelease +# (`17.0.0-20261108`) has none. +release_of() { + case "$1" in + *-mcpp.*) echo "${1%%-mcpp.*}" ;; + *-*) echo "" ;; + *) echo "$1" ;; + esac +} + +# The release keys the descriptor of `pkg` lists, one ` ` per +# line: each version key under `linux = {`, `macosx = {` or `windows = {` of the +# descriptor, with its release part. +releases_by_platform() { + local file="$INDEX/pkgs/${1:0:1}/$1.lua" + [ -f "$file" ] || return + awk ' + { sub(/--.*/, "") } + match($0, /^[ \t]*(linux|macosx|windows)[ \t]*=[ \t]*\{/) { + platform = $0; sub(/^[ \t]*/, "", platform); sub(/[ \t=].*/, "", platform); at = depth + } + platform != "" && match($0, /\["[0-9][0-9A-Za-z.+-]*"\][ \t]*=/) { + key = substr($0, RSTART + 2); sub(/".*/, "", key); print platform, key + } + { + n = gsub(/\{/, "{"); m = gsub(/\}/, "}"); depth += n - m + if (platform != "" && depth <= at) platform = "" + }' "$file" | while read -r platform v; do + r=$(release_of "$v"); [ -n "$r" ] && echo "$platform $r" + done +} + +# The platforms of the descriptor that the hosts of a default name: `linux-*` and +# `linux-x86_64` are linux, `macos-*` macosx, `windows-*` windows, `*` every one. +platforms_of() { + local hosts="$1" pkg="$2" + for h in $hosts; do + case "$h" in + linux-*) echo linux ;; + macos-*) echo macosx ;; + windows-*) echo windows ;; + '*') releases_by_platform "$pkg" | awk '{print $1}' ;; + esac + done | sort -u +} + +# The rows of the built-in table: ` `. +rows() { + awk ' + function flush() { if (use != "") print of, use, (target == "" ? "-" : target), hosts; of = ""; use = ""; target = ""; hosts = "" } + /^\[\[default\]\]/ { flush() } + /^of *=/ { gsub(/"/, "", $3); of = $3 } + /^use *=/ { gsub(/"/, "", $3); use = $3 } + /^target *=/ { gsub(/"/, "", $3); target = $3 } + /^hosts *=/ { h = $0; sub(/^[^[]*\[/, "", h); sub(/\].*/, "", h); gsub(/[",]/, " ", h); hosts = h } + END { flush() }' "$DATA" +} + +# The payload a toolchain request names: `gcc@-musl` is `musl-gcc@`, and gcc +# for a `*-windows-gnu` target is `mingw-gcc`. +payload_of() { + local use="$1" target="$2" pkg="${1%@*}" ver="${1##*@}" + case "$ver" in *-musl) pkg="musl-$pkg"; ver="${ver%-musl}" ;; esac + case "$pkg:$target" in gcc:*-windows-gnu) pkg="mingw-gcc" ;; esac + echo "$pkg $ver" +} + +# ONE VERSION OF A PACKAGE FOR EVERY HOST ITS DEFAULTS SERVE: a package's newest +# release is the newest that each platform its defaults name publishes. llvm +# 23.1.6 has no macOS payload (llvm-project's release publishes no macOS binary), +# and the llvm defaults include macOS, so llvm's is 23.1.3 until one exists. +declare -A platforms +while read -r of use target hosts; do + read -r pkg ver < <(payload_of "$use" "$target") + platforms[$pkg]+=" $(platforms_of "$hosts" "$pkg" | tr '\n' ' ')" +done < <(rows) + +newest_common() { + local pkg="$1" wanted + wanted=$(echo "${platforms[$pkg]}" | tr ' ' '\n' | grep -v '^$' | sort -u) + [ -n "$wanted" ] || return + local count; count=$(echo "$wanted" | wc -l) + releases_by_platform "$pkg" | grep -wFf <(echo "$wanted") | sort -u \ + | awk -v need="$count" '{seen[$2]++} END {for (v in seen) if (seen[v] == need) print v}' \ + | sort -V | tail -1 +} + +fail=0 +printf '%-10s %-24s %-16s %s\n' "default" "use" "newest release" "state" +while read -r of use target hosts; do + read -r pkg ver < <(payload_of "$use" "$target") + have=$(release_of "$ver") + newest=$(newest_common "$pkg") + if [ -z "$newest" ]; then + state="not in the index"; fail=1 + elif [ "$(printf '%s\n%s\n' "$have" "$newest" | sort -V | tail -1)" != "$have" ]; then + state="behind"; fail=1 + else + state="current" + fi + printf '%-10s %-24s %-16s %s\n' "$of" "$pkg@$ver" "${newest:--}" "$state" +done < <(rows | sort -u) + +if [ "$fail" -ne 0 ]; then + echo "a default is behind the index or absent from it: move it (docs/51), or publish the payload first" +fi +exit "$fail" diff --git a/.github/tools/check_diag_sink.sh b/.github/tools/check_diag_sink.sh new file mode 100755 index 000000000..8cd3c7bdc --- /dev/null +++ b/.github/tools/check_diag_sink.sh @@ -0,0 +1,21 @@ +#!/usr/bin/env bash +# Every user-visible warning, note and tip goes through mcpp.diag (design 2026-11-21, build +# programs' inputs §3.6): `mcpp::ui::warning`, `mcpp::ui::note` and `mcpp::ui::tip` are its +# renderers, and nothing else calls them or writes a `warning:` line to standard error itself. +# +# WHY THIS EXISTS. A notice outside the sink is not deduplicated, does not reach the machine +# envelope, is not counted by --strict and cannot have a policy (`once`). Before this check there +# were about a hundred such calls and three private "say it once" sets, and each had to be found +# by reading. +set -u +cd "$(dirname "$0")/../.." + +pattern='mcpp::ui::(warning|note|tip)\(|println\([^;]*stderr[^;]*"(warning|note|tip): ' +hits=$(grep -rnE "$pattern" src modules --include='*.cppm' --include='*.cpp' \ + | grep -vE '^src/(diag|ui)\.cppm:' || true) +if [ -n "$hits" ]; then + echo "::error::user-visible notices outside mcpp.diag (call mcpp::diag::warning/note/notice/closing instead):" + echo "$hits" + exit 1 +fi +echo "OK: every warning, note and tip goes through mcpp.diag" diff --git a/.github/tools/check_version_pins.sh b/.github/tools/check_version_pins.sh index 2238ba297..2505b736d 100755 --- a/.github/tools/check_version_pins.sh +++ b/.github/tools/check_version_pins.sh @@ -13,6 +13,8 @@ # # 3. The launchers a pack builds use mcpp's own mcpplibs.os release. # +# 4. The test skeleton of `mcpp new` uses mcpp's own mcpplibs.mtest release. +# # Why this exists: src/provider/xlings/xlings.cppm used to say "keep in lock-step with the # XLINGS_VERSION pins in release.yml / cross-build-test.yml / ci-linux-e2e.yml" # and that list was ALREADY incomplete — it omitted both composite actions, @@ -191,7 +193,21 @@ elif [ "$os_toml" != "$os_launcher" ]; then bad "src/pack/distribution.cppm builds the launchers on mcpplibs.os '$os_launcher', mcpp.toml builds mcpp on '$os_toml'" fi +# ── 4. A new project's tests use mcpp's own mtest ───────────────────────── +# +# `mcpp new` writes `mcpplibs.mtest = ""` (src/build/test_protocol.cppm), and mcpp's +# own tests use the mtest of mcpp.toml. A skeleton on an older release than mcpp tests itself with +# is a framework nobody here runs: 2026.11.20.1 scaffolded 0.9.0 while mcpp used 0.10.0 (mcpp#976). +mtest_toml=$(awk -F '"' '/^mcpplibs\.mtest[[:space:]]*=/{print $2; exit}' mcpp.toml) +mtest_skeleton=$(grep -oE 'kMtestRelease[[:space:]]*=[[:space:]]*"[^"]+"' src/build/test_protocol.cppm \ + | grep -oE '"[^"]+"' | tr -d '"' | head -1) +if [ -z "$mtest_toml" ] || [ -z "$mtest_skeleton" ]; then + bad "could not read mcpplibs.mtest from mcpp.toml ('$mtest_toml') or kMtestRelease from src/build/test_protocol.cppm ('$mtest_skeleton')" +elif [ "$mtest_toml" != "$mtest_skeleton" ]; then + bad "src/build/test_protocol.cppm scaffolds mcpplibs.mtest '$mtest_skeleton', mcpp.toml tests mcpp with '$mtest_toml'" +fi + if [ "$fail" = 0 ]; then - echo "OK: xlings pins all at $XLINGS_EXPECTED; building mcpp $v_toml, bootstrapping from $v_xl; launchers on mcpplibs.os $os_launcher" >&2 + echo "OK: xlings pins all at $XLINGS_EXPECTED; building mcpp $v_toml, bootstrapping from $v_xl; launchers on mcpplibs.os $os_launcher; skeleton on mcpplibs.mtest $mtest_skeleton" >&2 fi exit "$fail" diff --git a/.github/tools/check_waits.sh b/.github/tools/check_waits.sh new file mode 100644 index 000000000..2ae66f957 --- /dev/null +++ b/.github/tools/check_waits.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# The waits for time in the tests only fall (design 2026-11-20 §5.20). +# +# A test waits for an event the program under test gives -- a line of output, a +# file, its exit -- and not for a time: a sleep long enough on one machine is +# short on a loaded one, and a poll hides how long the event took. Each file's +# count of `sleep_for`, `sleep_until` and shell `sleep ` is at most its line +# in waits.baseline; a file not listed has none. A wait whose time is itself +# what the case asserts says so in a comment on the line above it, and stays. +set -u +cd "$(dirname "$0")/../.." + +baseline=.github/tools/waits.baseline +[ -f "$baseline" ] || { echo "::error::missing $baseline"; exit 1; } + +current=$(grep -rcE 'sleep_for|sleep_until|(^|[;&|[:space:]])sleep [0-9]' tests \ + --include='*.cpp' --include='*.cppm' --include='*.sh' 2>/dev/null \ + | grep -v ':0$' | sed 's/:/ /' | sort) + +rc=0 +while read -r file n; do + [ -n "$file" ] || continue + allowed=$(awk -v f="$file" '$1 == f { print $2 }' "$baseline") + if [ -z "$allowed" ]; then + echo "::error::$file waits for a time $n time(s); a test waits for an event (design 2026-11-20 §5.20)" + rc=1 + elif [ "$n" -gt "$allowed" ]; then + echo "::error::$file waits for a time $n times, more than the $allowed of $baseline" + rc=1 + elif [ "$n" -lt "$allowed" ]; then + echo "note: $file waits $n times, fewer than the $allowed of $baseline; lower its line" + fi +done <<< "$current" +[ "$rc" = 0 ] && echo "OK: no test waits for a time more often than $baseline states" +exit "$rc" diff --git a/.github/tools/main_run.sh b/.github/tools/main_run.sh new file mode 100755 index 000000000..34d9f492f --- /dev/null +++ b/.github/tools/main_run.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# main_run.sh -- print the run whose test artifacts are main's: main's last +# successful ci.yml run, or, when that run registered a verified run instead of +# testing (design 2026-11-20 §5.22), the run it registered. Prints nothing when +# main has no successful run. Reads GITHUB_REPOSITORY and GH_TOKEN. +# +# A push to main whose tree a `full` run of its pull request verified runs no +# test: its `register` job uploads `registered`, a file holding the verifying +# run's id. The stores, the unit archives and the build directories later runs +# read from main are that run's. +set -uo pipefail +repo="${GITHUB_REPOSITORY:?main_run.sh: GITHUB_REPOSITORY is not set}" +run=$(gh run list -R "$repo" --workflow ci.yml --branch main --status success --limit 1 \ + --json databaseId -q '.[0].databaseId' 2>/dev/null || true) +[ -n "$run" ] || exit 0 +tmp=$(mktemp -d) +trap 'rm -rf "$tmp"' EXIT +if gh run download "$run" -R "$repo" -n registered -D "$tmp" > /dev/null 2>&1 && [ -s "$tmp/run" ]; then + tr -d '[:space:]' < "$tmp/run" + echo +else + echo "$run" +fi diff --git a/.github/tools/system-names.baseline b/.github/tools/system-names.baseline index 436f8eea5..304654dd6 100644 --- a/.github/tools/system-names.baseline +++ b/.github/tools/system-names.baseline @@ -14,8 +14,8 @@ haiku 0 dos 0 uefi 0 is_linux 26 -is_windows 126 -is_macos 31 +is_windows 123 +is_macos 28 is_apple 5 is_ios 7 is_ios_simulator 6 diff --git a/.github/tools/upgrade_lane.sh b/.github/tools/upgrade_lane.sh index a6dbcfd84..87277bad8 100644 --- a/.github/tools/upgrade_lane.sh +++ b/.github/tools/upgrade_lane.sh @@ -26,7 +26,7 @@ # mismatch"). # # Asserted, every variant: the first `mcpp new && mcpp run` of the new mcpp -# succeeds and downloads nothing but the ninja this commit pins; a second run +# succeeds and downloads nothing but the bootstrap tools this commit pins; a second run # downloads nothing at all; `mcpp self doctor` # in the project installs nothing and names the toolchain the build uses. # snapshot: `mcpp index update` names the newer packagings, the run after it @@ -117,14 +117,20 @@ run() { echo "::endgroup::" return $rc } -# The one download an upgraded home makes: the ninja this commit pins, when the -# previous release pinned another (mcpp#945: the pin carries a correction, so a -# home takes it with the upgrade, beside the ninja it has). It is taken once -- -# `any_download` holds the runs after the first to nothing at all. -PIN=$(sed -n 's/.*kNinjaVersion *= *"\([^"]*\)".*/\1/p' "$(dirname "$0")/../../src/provider/xlings/xlings.cppm" | head -1) -[ -n "$PIN" ] || { echo "::error::upgrade lane: the pinned ninja is not read from xlings.cppm"; exit 1; } +# The downloads an upgraded home makes: the bootstrap tools this commit pins +# (ninja, patchelf), when the previous release pinned others (mcpp#945: a pin +# carries a correction or a newer release, so a home takes it with the upgrade, +# beside the one it has). They are taken once -- `any_download` holds the runs +# after the first to nothing at all. The pins are data: the `[[default]]` +# entries of the built-in table that are not a toolchain. +TOOLS=$(awk ' + /^\[\[default\]\]/ { of = "" } + /^of *=/ { gsub(/"/, "", $3); of = $3 } + /^use *=/ { gsub(/"/, "", $3); if (of != "toolchain") print "Downloading xim:" $3 } +' "$(dirname "$0")/../../modules/toolchain-model/data/builtin.toml") +echo "$TOOLS" | grep -q "xim:ninja@" || { echo "::error::upgrade lane: the pinned ninja is not read from builtin.toml"; exit 1; } any_download() { grep -o "Downloading [^ ]*" "$W/$1.log" | sort -u | tr '\n' ' '; } -downloads() { grep -o "Downloading [^ ]*" "$W/$1.log" | grep -vxF "Downloading xim:ninja@$PIN" | sort -u | tr '\n' ' '; } +downloads() { grep -o "Downloading [^ ]*" "$W/$1.log" | grep -vxF -f <(echo "$TOOLS") | sort -u | tr '\n' ' '; } new_project() { # (cd "$W" && "$2" new "$1" > /dev/null 2>&1) || mkdir -p "$W/$1" printf '\n[toolchain]\ndefault = "%s"\n' "$TC" >> "$W/$1/mcpp.toml" @@ -256,24 +262,29 @@ fi if [ "$VARIANT" = modules ]; then # A BMI of another spelling is not read: the dependency is rebuilt once (its # cache key carries the spelling), then served from the cache. The spelling - # moved in 2026.10.26.2 (the module map's `$root`) and in 2026.11.19.1 (a - # package instance's directory, mcpp#953); a previous release from the last - # move on spells as this one, and its entry is served at once. - spelled_since=2026.11.19.1 + # moved in 2026.10.26.2 (the module map's `$root`), in 2026.11.19.1 (a + # package instance's directory, mcpp#953) and in 2026.11.21.1 (an instance + # is its configuration's, mcpp#986); a previous release from the last move + # on spells as this one, and its entry is served at once. + spelled_since=2026.11.21.1 + # The directory a served BMI is staged into: `./` before + # 2026.11.21.1, `_//` from it (mcpp#986). Either + # is the dependency's; the staging, not the directory's name, is asserted. + STAGED_OPENKAL='STAGE gcm\.cache/mcpplibs[._]openkal/' old_version=$("$OLD" --version | grep -Eo '[0-9]{4}\.[0-9]+\.[0-9]+\.[0-9]+' | head -1) same_spelling=$(printf '%s\n%s\n' "$old_version" "$spelled_since" | sort -t. -k1,1n -k2,2n -k3,3n -k4,4n | head -1) modules_project m-new1 run new-modules1 "$W/m-new1" "$NEW" build -v || fail "the module project does not build over the previous release's cache" if [ "$same_spelling" = "$spelled_since" ]; then - grep -q "STAGE gcm.cache/mcpplibs.openkal/" "$W/new-modules1.log" \ + grep -qE "$STAGED_OPENKAL" "$W/new-modules1.log" \ || fail "a dependency BMI of the same spelling ($old_version) was not served from the cache" else - grep -q "STAGE gcm.cache/mcpplibs.openkal/" "$W/new-modules1.log" \ + grep -qE "$STAGED_OPENKAL" "$W/new-modules1.log" \ && fail "a dependency BMI of the previous spelling ($old_version) was taken from the cache" fi modules_project m-new2 run new-modules2 "$W/m-new2" "$NEW" build -v || fail "the second module project failed" - grep -q "STAGE gcm.cache/mcpplibs.openkal/" "$W/new-modules2.log" \ + grep -qE "$STAGED_OPENKAL" "$W/new-modules2.log" \ || fail "the rebuilt dependency was not served from the cache" fi if [ "$VARIANT" = programs ]; then diff --git a/.github/tools/waits.baseline b/.github/tools/waits.baseline new file mode 100644 index 000000000..947ef956c --- /dev/null +++ b/.github/tools/waits.baseline @@ -0,0 +1,48 @@ +# One line per test file that waits for a time: its count of sleep_for, sleep_until and shell +# `sleep ` (.github/tools/check_waits.sh). The numbers only fall. +tests/e2e/288_the_openkal_stack_on_a_machine_with_no_os.sh 1 +tests/e2e/actions/checks.cpp 1 +tests/e2e/actions/commands.cpp 2 +tests/e2e/actions/declared.cpp 2 +tests/e2e/actions/generated.cpp 2 +tests/e2e/actions/hooks.cpp 3 +tests/e2e/actions/outputs.cpp 2 +tests/e2e/actions/prepare.cpp 1 +tests/e2e/build/basics.cpp 2 +tests/e2e/build-program/graph.cpp 3 +tests/e2e/build-program/hosttools.cpp 2 +tests/e2e/cache/directories.cpp 3 +tests/e2e/cache/fastpath.cpp 1 +tests/e2e/cdb/root.cpp 2 +tests/e2e/cli/commands.cpp 1 +tests/e2e/cli/output.cpp 1 +tests/e2e/cli/signals.cpp 1 +tests/e2e/cli/terminal.cpp 2 +tests/e2e/deps/advice.cpp 1 +tests/e2e/index/refresh.cpp 4 +tests/e2e/modules/bmi.cpp 1 +tests/e2e/modules/extensions.cpp 1 +tests/e2e/modules/programs.cpp 2 +tests/e2e/run/environment.cpp 1 +tests/e2e/run/program.cpp 1 +tests/e2e/runtime/binding.cpp 2 +tests/e2e/self/homes.cpp 2 +tests/e2e/test/fast.cpp 5 +tests/e2e/test/report.cpp 1 +tests/e2e/test/streams.cpp 1 +tests/e2e/toolchain/llvm.cpp 1 +tests/e2e/windows/images.cpp 1 +tests/e2e/windows/msvc.cpp 2 +tests/e2e/windows/resources.cpp 3 +tests/e2e/workspace/graph.cpp 1 +tests/matrix/scan.sh 1 +tests/release/verify-published.sh 1 +tests/support/src/support.cpp 1 +tests/unit/test_build_legs.cpp 1 +tests/unit/test_build_progress.cpp 3 +tests/unit/test_config.cpp 1 +tests/unit/test_detach_status.cpp 4 +tests/unit/test_network.cpp 3 +tests/unit/test_platform_fs.cpp 3 +tests/unit/test_process_run_exec.cpp 9 +tests/unit/test_xlings_version_pin.cpp 1 diff --git a/.github/versions.env b/.github/versions.env index c571ac8c9..6047f7855 100644 --- a/.github/versions.env +++ b/.github/versions.env @@ -1,5 +1,5 @@ # The engines CI pins, read by the composite actions (.github/actions/mtest). # mtest: an xlings package (xim:mtest) whose sha256 the index states; the release # bundle's sha256 checks the fallback download. -MTEST_VERSION=0.10.0 -MTEST_SHA256=c7187e6211ea15cff23f622c449247508f6f661f0d3b45dcd031f91104b8f499 +MTEST_VERSION=0.11.2 +MTEST_SHA256=88144bacd76ff3af3957b29560e55af813906db2f682f8cefae1869a2be74de9 diff --git a/.github/workflows/bootstrap-macos.yml b/.github/workflows/bootstrap-macos.yml index 7efc90fb6..52d1d9ba4 100644 --- a/.github/workflows/bootstrap-macos.yml +++ b/.github/workflows/bootstrap-macos.yml @@ -17,7 +17,7 @@ jobs: # Dormant (workflow_dispatch only), but kept in step with the rest — # check_version_pins.sh holds it there. Floor: 0.4.69, below which the # index cannot resolve two packages that share a short name. - XLINGS_VERSION: '2026.11.17.2' + XLINGS_VERSION: '2026.11.21.1' steps: - uses: actions/checkout@v4 diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 124433ac4..34fede409 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -69,6 +69,23 @@ jobs: with: image: ${{ inputs.runs-on }} + # THE UNITS MAIN'S BUILDS COMPILED (design 2026-11-20 §5.3, CI5): a compile whose command, + # compiler and the contents of the files it read are those of a compile main kept is placed + # from the home's unit cache. Restored on every run; saved by this job on a push to main + # alone (R3), with the entries no build of this job used removed first, so the cache holds + # what main's build reads. A bootstrap mcpp without the unit cache leaves it as it was. + - name: Restore main's unit cache + uses: actions/cache/restore@v4 + with: + path: ~/.mcpp/build-cache/v1/unit + key: mcpp-units-${{ inputs.host }}-${{ github.sha }} + restore-keys: | + mcpp-units-${{ inputs.host }}- + + - name: The start of this job's use of the units + shell: bash + run: touch "$RUNNER_TEMP/units.mark" + - name: Build mcpp from source (self-host) shell: bash run: | @@ -179,6 +196,27 @@ jobs: path: ~/.xlings key: ${{ steps.boot.outputs.xlings-key }} + - name: Keep the units this job used + if: ${{ github.event_name == 'push' }} + shell: bash + run: | + set -euo pipefail + root="$HOME/.mcpp/build-cache/v1/unit" + [ -d "$root" ] || { echo "no unit cache: this build's mcpp keeps none"; exit 0; } + kept=0 dropped=0 + # An entry is //; a use stamps its manifest.tsv. + while IFS= read -r entry; do + if [ "$entry/manifest.tsv" -nt "$RUNNER_TEMP/units.mark" ]; then kept=$((kept + 1)); else rm -rf "$entry"; dropped=$((dropped + 1)); fi + done < <(find "$root" -mindepth 3 -maxdepth 3 -type d) + echo "units: $kept used by this job and kept, $dropped unused and removed" + + - name: Save main's unit cache + if: ${{ github.event_name == 'push' }} + uses: actions/cache/save@v4 + with: + path: ~/.mcpp/build-cache/v1/unit + key: mcpp-units-${{ inputs.host }}-${{ github.sha }} + # macOS keeps no sandbox cache: setup-macos-llvm builds the sandbox from # the freshly resolved LLVM on every run, which is what the macOS jobs # exist to prove. Its xlings cache has this job as its writer. diff --git a/.github/workflows/ci-bootstrap.yml b/.github/workflows/ci-bootstrap.yml index 679238269..d9bf59ab9 100644 --- a/.github/workflows/ci-bootstrap.yml +++ b/.github/workflows/ci-bootstrap.yml @@ -41,7 +41,7 @@ jobs: os: [ubuntu-24.04, macos-15, windows-2025] env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.11.17.2' + XLINGS_VERSION: '2026.11.21.1' steps: - uses: actions/checkout@v4 @@ -50,7 +50,7 @@ jobs: shell: bash run: | set -euo pipefail - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" echo "$HOME/.xlings/bin" >> "$GITHUB_PATH" @@ -124,7 +124,7 @@ jobs: shell: bash run: | set -euo pipefail - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install the newest release and build the tree with it (mcpp build --strict) diff --git a/.github/workflows/ci-fresh-install.yml b/.github/workflows/ci-fresh-install.yml index 5dbe66872..dbd7babca 100644 --- a/.github/workflows/ci-fresh-install.yml +++ b/.github/workflows/ci-fresh-install.yml @@ -65,7 +65,7 @@ concurrency: jobs: # ────────────────────────────────────────────────────────────────── - # Linux: gcc@16.1.0, musl-gcc@16.1.0, llvm@23.1.3 + # Linux: gcc@16.2.0, musl-gcc@16.1.0, llvm@23.1.3 # ────────────────────────────────────────────────────────────────── # A4: post-release runs race the xim-pkgindex bump (a PR a maintainer # merges asynchronously) — the 0.0.85 fresh-install failed with @@ -164,8 +164,8 @@ jobs: - toolchain: default - toolchain: gcc@16.1.0-musl install: gcc 16.1.0-musl - - toolchain: gcc@16.1.0 - install: gcc 16.1.0 + - toolchain: gcc@16.2.0 + install: gcc 16.2.0 - toolchain: llvm@23.1.3 install: llvm 23.1.3 env: @@ -182,7 +182,7 @@ jobs: env: XLINGS_NON_INTERACTIVE: '1' run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror @@ -311,7 +311,7 @@ jobs: - name: Install xlings + mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 # Deliberately NOT writing to $GITHUB_PATH here. On container # images that declare no PATH in their config (opensuse/ # tumbleweed), appending a single dir to GITHUB_PATH makes the @@ -460,7 +460,7 @@ jobs: # (older ones carry minos=15 and refuse to start). # v0.4.51+: in-process sha256 — this image has no sha256sum # binary, so pinned fetches failed before it. - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - name: Install mcpp and config mirror diff --git a/.github/workflows/ci-linux-e2e.yml b/.github/workflows/ci-linux-e2e.yml index 0e0ba4c48..8194a9a5e 100644 --- a/.github/workflows/ci-linux-e2e.yml +++ b/.github/workflows/ci-linux-e2e.yml @@ -84,8 +84,8 @@ jobs: # explicitly, not assumed: the restored sandbox is the build job's, # and a toolchain that only happens to be in it is a dependency on a # cache. - "$MCPP" toolchain install gcc 16.1.0 - "$MCPP" toolchain default gcc@16.1.0 + "$MCPP" toolchain install gcc 16.2.0 + "$MCPP" toolchain default gcc@16.2.0 # The toolchains whose capabilities the suite probes (run_all.sh): # musl, llvm (which also yields scan-deps and import-std-libcxx) and # the Linux-hosted MinGW cross compiler. Without them their tests @@ -93,7 +93,7 @@ jobs: # nowhere. Warm runs re-install nothing. "$MCPP" toolchain install gcc 16.1.0-musl "$MCPP" toolchain install llvm 23.1.3 - "$MCPP" toolchain install mingw-cross 16.1.0 + "$MCPP" toolchain install mingw-cross 16.2.0 XLINGS_HOME="$MCPP_HOME/registry" "$MCPP_VENDORED_XLINGS" install xim:nasm -y # The index payloads e2e 32, 33 and 40 link from this home into their # own (_inherit_toolchain.sh). Installed in a test home instead, they @@ -190,7 +190,7 @@ jobs: run: | set -euo pipefail export MCPP_HOME=/mnt/nfs/mcpp-home - "$MCPP" toolchain install gcc 16.1.0 + "$MCPP" toolchain install gcc 16.2.0 cd /mnt/nfs "$MCPP" new hello cd hello @@ -386,8 +386,8 @@ jobs: run: | export MCPP MCPP_VENDORED_XLINGS export MCPP_E2E_TOOLCHAIN_MIRROR=GLOBAL - "$MCPP" toolchain install gcc 16.1.0 - "$MCPP" toolchain default gcc@16.1.0 + "$MCPP" toolchain install gcc 16.2.0 + "$MCPP" toolchain default gcc@16.2.0 # Run directly, not through run_all.sh, so a skip cannot pass as a # green: each script either prints its PASS line or this step fails. declare -A pass=( @@ -469,7 +469,7 @@ jobs: - name: Bootstrap xlings + released mcpp run: | - curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.17.2 + curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash -s v2026.11.21.1 export PATH="$HOME/.xlings/subos/current/bin:$PATH" xlings update xlings install mcpp -y -g diff --git a/.github/workflows/ci-linux.yml b/.github/workflows/ci-linux.yml index 0ae71c16a..b3643585a 100644 --- a/.github/workflows/ci-linux.yml +++ b/.github/workflows/ci-linux.yml @@ -128,6 +128,9 @@ jobs: # be the same suite, toolchain and binary as `build-test`. toolchain-gcc: name: "toolchain: gcc (cold self-host)" + # Another toolchain's build of mcpp, or another project's: a checkpoint's (`full`), not each + # push's (design 2026-11-20 §7.2). + if: inputs.set == 'full' runs-on: ubuntu-24.04 timeout-minutes: 45 steps: @@ -150,6 +153,9 @@ jobs: # builder: a runner apiece would cost more in setup than it saves. toolchain-cross: name: "toolchain: musl + llvm" + # Another toolchain's build of mcpp, or another project's: a checkpoint's (`full`), not each + # push's (design 2026-11-20 §7.2). + if: inputs.set == 'full' runs-on: ubuntu-24.04 timeout-minutes: 45 steps: @@ -197,6 +203,9 @@ jobs: # resolves deps through. integration-xlings: name: "integration: mcpp builds & runs xlings" + # Another toolchain's build of mcpp, or another project's: a checkpoint's (`full`), not each + # push's (design 2026-11-20 §7.2). + if: inputs.set == 'full' runs-on: ubuntu-24.04 timeout-minutes: 45 env: diff --git a/.github/workflows/ci-macos-e2e.yml b/.github/workflows/ci-macos-e2e.yml index 9fa3e0be4..d355b341c 100644 --- a/.github/workflows/ci-macos-e2e.yml +++ b/.github/workflows/ci-macos-e2e.yml @@ -41,7 +41,8 @@ jobs: strategy: fail-fast: false matrix: - image: [macos-15, xcode-27] + # The newest image is a checkpoint's (`full`), not each push's (design 2026-11-20 §7.2). + image: ${{ fromJSON(inputs.set == 'full' && '["macos-15", "xcode-27"]' || '["macos-15"]') }} shard: ${{ fromJSON(inputs.shards) }} runs-on: ${{ matrix.image }} timeout-minutes: 60 diff --git a/.github/workflows/ci-macos.yml b/.github/workflows/ci-macos.yml index e869bbeb9..9ebeb454f 100644 --- a/.github/workflows/ci-macos.yml +++ b/.github/workflows/ci-macos.yml @@ -35,12 +35,11 @@ jobs: # behind an input so a PR that could not fix it held no green slot) — the # 23.1.3 line move removed that cause, and a change can change the outcome # here again. + # The newest image is a checkpoint's (`full`), not each push's (design 2026-11-20 §7.2). strategy: fail-fast: false matrix: - include: - - image: macos-15 - - image: xcode-27 + image: ${{ fromJSON(inputs.set == 'full' && '["macos-15", "xcode-27"]' || '["macos-15"]') }} runs-on: ${{ matrix.image }} # The C++ e2e programs run in this job (tests/e2e/*/, design 2026-11-12 # section 5.9); a change to tests/support runs them all, measured at 40 diff --git a/.github/workflows/ci-offline.yml b/.github/workflows/ci-offline.yml new file mode 100644 index 000000000..03d55c603 --- /dev/null +++ b/.github/workflows/ci-offline.yml @@ -0,0 +1,53 @@ +name: ci-offline + +# A repository built on a machine without a network (design +# .agents/docs/2026-11-21-offline-host-inputs-and-toolchain-defaults-design.md +# §3.2, F3; docs/90, "Building on a machine without a network"). A clean home +# writes everything this repository's build needs into a directory with +# `mcpp fetch --to`; a second, fresh home takes it with `mcpp fetch --from` and +# builds mcpp in a network namespace of its own, where any download fails. +# +# Called by ci.yml after the Linux build, with the binary build.yml produced +# (use-built-mcpp, rule R1 of the 2026-10-02 CI record). + +on: + workflow_call: + +jobs: + offline: + name: mcpp builds from a bundle without a network (linux-x86_64) + runs-on: ubuntu-24.04 + timeout-minutes: 60 + steps: + - uses: actions/checkout@v4 + + - uses: ./.github/actions/bootstrap-mcpp + + - uses: ./.github/actions/use-built-mcpp + with: + host: linux-x86_64 + + - name: A clean home writes the repository's inputs into a directory + shell: bash + run: | + set -euo pipefail + MCPP_HOME="$RUNNER_TEMP/home-online" "$MCPP_FRESH" fetch --to "$RUNNER_TEMP/bundle" + test -f "$RUNNER_TEMP/bundle/bundle.toml" + du -sh "$RUNNER_TEMP/bundle" + + - name: A fresh home takes the directory and builds mcpp in a network namespace of its own + shell: bash + run: | + set -euo pipefail + # A network namespace needs root here (the runner restricts unprivileged + # user namespaces); the build inside it runs as the runner's user. + sudo unshare --net -- sudo -u "$(id -un)" env \ + PATH="$PATH" HOME="$HOME" \ + MCPP_HOME="$RUNNER_TEMP/home-offline" \ + MCPP_VENDORED_XLINGS="$MCPP_VENDORED_XLINGS" \ + bash -c 'set -euo pipefail + if curl -sS -m 5 https://github.com > /dev/null 2>&1; then echo "::error::the namespace reaches the network"; exit 1; fi + "$0" fetch --from "$1" --offline + "$0" build --offline + target=$(ls -d target/x86_64-linux-gnu/*/bin/mcpp | head -1) + "$target" --version' "$MCPP_FRESH" "$RUNNER_TEMP/bundle" diff --git a/.github/workflows/ci-target-matrix.yml b/.github/workflows/ci-target-matrix.yml index dce57833e..f320eaa1c 100644 --- a/.github/workflows/ci-target-matrix.yml +++ b/.github/workflows/ci-target-matrix.yml @@ -463,11 +463,11 @@ jobs: # a claim nobody checked; a host that is scanned and never shipped is # coverage spent on a machine no user has. # - # Derived from release.yml's asset names rather than restated here, so + # Derived from release-build.yml's asset names rather than restated here, so # adding a fifth host to the release fails this step until the matrix # covers it. ship=$(grep -oE 'mcpp-\$\{?[A-Za-z_{}. ]*\}?-(linux|macosx|windows)-(x86_64|aarch64|arm64)' \ - .github/workflows/release.yml \ + .github/workflows/release-build.yml \ | sed -E 's/.*-(linux|macosx|windows)-/\1-/' \ | sed 's/^macosx-/macos-/' | sort -u) scanned=$(awk -F'\t' 'NF>=11 {print $2}' tests/matrix/expected.tsv | sort -u) diff --git a/.github/workflows/ci-unit.yml b/.github/workflows/ci-unit.yml index 5fdebc24a..c512022f4 100644 --- a/.github/workflows/ci-unit.yml +++ b/.github/workflows/ci-unit.yml @@ -58,10 +58,35 @@ on: description: the run stage's cases of `mcpp-tests-` that need the MinGW cross toolchain, one per line type: string default: '' + select: + description: the run stage's cases, one `mcpp test --case` pattern per line; empty for every case + type: string + default: '' + jitter: + description: the jitter of the run stage's shards, in milliseconds (jitter.yml); empty for none + type: string + default: '' + jitter-seed: + description: the seed of the jitter + type: string + default: '' + form: + description: the run stage's program under test, `host` or `release` (use-built-mcpp `form`) + type: string + default: host + toolchains: + description: > + The toolchains the run stage's cases probe, installed before the shards run, separated + by ';', each as `mcpp toolchain install` takes it (`llvm 23.1.3`); empty for none. + type: string + default: '' outputs: archived: description: the number of tests the archive holds value: ${{ jobs.archive.outputs.archived }} + shards: + description: the run stage's shards, from the measured work of main's last run + value: ${{ jobs.archive.outputs.shards }} env: MCPP_HOME: /home/runner/.mcpp @@ -77,6 +102,7 @@ jobs: timeout-minutes: 60 outputs: archived: ${{ steps.archive.outputs.archived }} + shards: ${{ steps.shards.outputs.shards }} steps: - uses: actions/checkout@v4 @@ -111,8 +137,7 @@ jobs: HOST: ${{ inputs.host }} run: | set -uo pipefail - run=$(gh run list -R "$GITHUB_REPOSITORY" --workflow ci.yml --branch main --status success --limit 1 \ - --json databaseId -q '.[0].databaseId' 2>/dev/null || true) + run=$(bash .github/tools/main_run.sh) # main's, or the run it registered (design 2026-11-20 §5.22) [ -n "$run" ] || { echo "no successful run of main: the tests are built whole"; exit 0; } if ! gh run download "$run" -R "$GITHUB_REPOSITORY" -n "mcpp-unit-build-$HOST" -D "$RUNNER_TEMP/build" 2>/dev/null; then echo "main's run $run kept no test build for $HOST: the tests are built whole"; exit 0 @@ -165,8 +190,7 @@ jobs: run: | set -uo pipefail mkdir -p "$RUNNER_TEMP/unit/history" - run=$(gh run list -R "$GITHUB_REPOSITORY" --workflow ci.yml --branch main --status success --limit 1 \ - --json databaseId -q '.[0].databaseId' 2>/dev/null || true) + run=$(bash .github/tools/main_run.sh) # main's, or the run it registered (design 2026-11-20 §5.22) if [ -z "$run" ]; then echo "no successful run of main: the shards divide by count"; exit 0; fi d="$RUNNER_TEMP/unit-stores" if ! gh run download "$run" -R "$GITHUB_REPOSITORY" -p "mtest-unit-$KEY*" -D "$d" 2>/dev/null; then @@ -174,6 +198,28 @@ jobs: fi python3 .github/tools/test_history.py take "$RUNNER_TEMP/unit/history" "$d" "$d"/*/ + # THE SHARDS FROM THE PLAN (design 2026-11-20 §5.17, CI4): the measured work of main's last + # run, its shards' programs' durations summed (the history holds that run's shards alone, + # test_history.py), over what one shard does in 30 minutes with its four workers; at least + # one, at most eight. With no history, four, as before. + - name: The run stage's shards + id: shards + shell: bash + run: | + set -uo pipefail + work=$(find "$RUNNER_TEMP/unit/history" -name programs.ndjson -print0 2>/dev/null \ + | xargs -0 -r cat | jq -s 'map(.ms // 0) | add // 0') + if [ -z "$work" ] || [ "${work%.*}" = 0 ]; then + echo 'shards=[1,2,3,4]' >> "$GITHUB_OUTPUT"; echo "no measured work: four shards"; exit 0 + fi + per=$((30 * 60 * 1000 * 4)) + n=$(( (${work%.*} + per - 1) / per )) + [ "$n" -ge 1 ] || n=1 + [ "$n" -le 8 ] || n=8 + list=$(seq -s, 1 "$n") + echo "shards=[$list]" >> "$GITHUB_OUTPUT" + echo "measured work ${work%.*} ms: $n shard(s)" + - name: Upload the tests and their history if: steps.archive.outputs.archived != '0' timeout-minutes: 10 @@ -246,6 +292,7 @@ jobs: - uses: ./.github/actions/use-built-mcpp with: host: ${{ inputs.host }} + form: ${{ inputs.form }} - name: The tests of this run uses: actions/download-artifact@v4 @@ -253,14 +300,35 @@ jobs: name: mcpp-unit-${{ inputs.host }} path: ${{ runner.temp }}/unit - # The program the tests run is this commit's, as build.yml built it: an - # archive written with --no-programs takes it from MCPP_BIN_MCPP. + # The program the tests run is this commit's, as build.yml built it or, in + # ci.yml's `full` tier, as its release archive holds it: an archive + # written with --no-programs takes it from MCPP_BIN_MCPP. - name: The program under test shell: bash run: | echo "MCPP_VENDORED_XLINGS=$XLINGS_BIN" >> "$GITHUB_ENV" echo "MCPP_BIN_MCPP=$MCPP" >> "$GITHUB_ENV" + # THE TOOLCHAINS THE CASES PROBE, installed and not assumed (ci-linux-e2e.yml does the + # same): the restored sandbox holds the ones build.yml's `prewarm` installed only while its + # lineage is main's, and a case whose capability does not hold skips on every runner. On a + # run that changed the xlings version, the fourteen cases that need llvm ran nowhere. + # Warm runs install nothing. + - name: The toolchains the cases probe + if: inputs.toolchains != '' + shell: bash + run: | + set -euo pipefail + IFS=';' read -ra specs <<< "${{ inputs.toolchains }}" + for spec in "${specs[@]}"; do + spec="$(echo "$spec" | xargs)" + [ -n "$spec" ] || continue + echo "::group::mcpp toolchain install $spec" + # shellcheck disable=SC2086 -- a spec is ` ` + "$MCPP" toolchain install $spec + echo "::endgroup::" + done + # THE HOST'S GATE (build.yml): a new program scaffolded, built and run by # this commit's mcpp, seconds of running, from the archive. A red commit # stops here, as the shards start, and the run is cancelled. The first @@ -300,6 +368,9 @@ jobs: name: ${{ inputs.name }}-${{ matrix.shard }} archive: ${{ runner.temp }}/unit/unit.tar.gz shard: ${{ matrix.shard }}/${{ strategy.job-total }} + cases: ${{ inputs.select }} + jitter: ${{ inputs.jitter }} + jitter-seed: ${{ inputs.jitter-seed }} # A compiler that did not start is a state of the home, not of a test: # the loader its payload is bound through (/registry/abi), the @@ -340,13 +411,14 @@ jobs: - uses: ./.github/actions/use-built-mcpp with: host: ${{ inputs.host }} + form: ${{ inputs.form }} # llvm beside it: the case that pairs clang with the MinGW payload reads # the program's resources with the payload's llvm-readobj (`llvm`). - name: Install mingw-cross toolchain shell: bash run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" toolchain install mingw-cross 16.1.0 + "$MCPP" toolchain install mingw-cross 16.2.0 "$MCPP" toolchain install llvm 23.1.3 echo "MCPP_VENDORED_XLINGS=$XLINGS_BIN" >> "$GITHUB_ENV" - uses: ./.github/actions/e2e-cases diff --git a/.github/workflows/ci-windows-e2e.yml b/.github/workflows/ci-windows-e2e.yml index ec73a1434..53838f8a7 100644 --- a/.github/workflows/ci-windows-e2e.yml +++ b/.github/workflows/ci-windows-e2e.yml @@ -89,7 +89,7 @@ jobs: - name: "Toolchain: MinGW payload for the fat-package e2e" shell: bash run: | - # The build job's prewarm installs `mingw 16.1.0` on `main` and saves + # The build job's prewarm installs `mingw 16.2.0` on `main` and saves # it into the sandbox cache, so every PR restores a sandbox that # already holds it. The install call is then a no-op but still pays # the mcpp-toolchain startup cost; measured 68 s on three shards per @@ -99,14 +99,14 @@ jobs: # passing one). export MCPP_VENDORED_XLINGS="$XLINGS_BIN" found="" - for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe \ - "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe; do + for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/16.2.0/bin/g++.exe \ + "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/16.2.0/bin/g++.exe; do [[ -x "$c" ]] && { found="$c"; break; } done if [ -z "$found" ]; then - "$MCPP_SELF" toolchain install mingw 16.1.0 - for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe \ - "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/*/bin/g++.exe; do + "$MCPP_SELF" toolchain install mingw 16.2.0 + for c in "${MCPP_HOME:-$HOME/.mcpp}"/registry/data/xpkgs/xim-x-mingw-gcc/16.2.0/bin/g++.exe \ + "$HOME"/.xlings/data/xpkgs/xim-x-mingw-gcc/16.2.0/bin/g++.exe; do [[ -x "$c" ]] && { found="$c"; break; } done fi diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f33399331..1a152ee2a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,6 +25,9 @@ on: branches: [ main ] pull_request: branches: [ main ] + # A merge queue's checks, when the repository enables one (design 2026-11-20 §5.22); without + # one the event never comes. + merge_group: workflow_dispatch: inputs: bench: @@ -56,9 +59,37 @@ jobs: outputs: code: ${{ steps.classify.outputs.code }} set: ${{ steps.set.outputs.set }} + tree: ${{ steps.tree.outputs.tree }} + verified: ${{ steps.tree.outputs.verified }} + verified_run: ${{ steps.tree.outputs.verified_run }} steps: - uses: actions/checkout@v4 + # A TREE IS VERIFIED ONCE (design 2026-11-20 §5.22). A `full` run that passes leaves the + # artifact `verified-` (the `verdict` job). A push to main whose tree has one -- the + # pull request was verified at its checkpoint and main has not moved since -- runs no test: + # it builds, which saves the caches only main's runs write (R3), and registers the verified + # run's stores as main's (`register`). Without one it runs `full`. + - name: The tree of this run, and whether a run verified it + id: tree + env: + GH_TOKEN: ${{ github.token }} + EVENT: ${{ github.event_name }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + tree=$(git rev-parse 'HEAD^{tree}') + echo "tree=$tree" >> "$GITHUB_OUTPUT" + verified=false run="" + if [ "$EVENT" = push ]; then + run=$(gh api "repos/$REPO/actions/artifacts?name=verified-$tree&per_page=5" \ + --jq '[.artifacts[] | select(.expired | not)][0].workflow_run.id // empty' 2>/dev/null || true) + [ -n "$run" ] && verified=true + fi + echo "verified=$verified" >> "$GITHUB_OUTPUT" + echo "verified_run=$run" >> "$GITHUB_OUTPUT" + echo "tree $tree; verified by a run: $verified${run:+ ($run)}" + # The set of cases this run runs (design 2026-11-12 §5.11): a pull # request's push runs `pr`, the cases its change affects; a pull request # labelled `ci:full`, a push to main and a run started by hand run `full`. @@ -71,6 +102,9 @@ jobs: run: | set=full if [ "$EVENT" = pull_request ] && [[ " $LABELS " != *" ci:full "* ]]; then set=pr; fi + # The tiers widen to jobs (design 2026-11-20 §7.2): `pr` runs each host's build, gates, + # unit shards and the e2e the change affects; the other toolchains' and targets' jobs, the + # bootstrap and the newest macOS image run at a checkpoint (`full`). echo "set=$set" >> "$GITHUB_OUTPUT" echo "the set of this run: $set" @@ -93,6 +127,8 @@ jobs: gh api --paginate "repos/$REPO/pulls/$PR/files" \ --jq '.[] | .filename, (.previous_filename // empty)' > changed.txt \ || { full "the files of the pull request could not be listed"; exit 0; } ;; + merge_group) + full "a merge queue's group"; exit 0 ;; push) if [ -z "$BEFORE" ] || [ "$BEFORE" = 0000000000000000000000000000000000000000 ]; then full "a push with no previous commit"; exit 0 @@ -118,7 +154,7 @@ jobs: e2e-plan: name: the e2e shards of each host needs: changes - if: needs.changes.outputs.code == 'true' + if: ${{ needs.changes.outputs.code == 'true' && needs.changes.outputs.verified != 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 10 outputs: @@ -170,6 +206,9 @@ jobs: - name: Check modules/ wiring run: bash .github/tools/check_modules_wiring.sh + - name: Check that every warning and note goes through mcpp.diag + run: bash .github/tools/check_diag_sink.sh + - name: Check that the engine's sources name no more systems than the baseline run: python3 .github/tools/check_system_names.py @@ -229,6 +268,9 @@ jobs: - name: Reason tokens agree with the engine, and with the mirror run: bash .github/tools/check_reason_tokens.sh + - name: The tests wait for a time no more often than the baseline states + run: bash .github/tools/check_waits.sh + - name: Only the adapter and the composition root name the provider run: python3 .github/tools/check_provider_boundary.py @@ -255,8 +297,9 @@ jobs: host: linux-x86_64 runs-on: ubuntu-24.04 # What the Linux consumers install beyond the build's own toolchain: the - # e2e shards (musl, llvm, mingw-cross), the toolchain legs (musl, llvm). - prewarm: gcc 16.1.0-musl; llvm 23.1.3; mingw-cross 16.1.0 + # first run's default (gcc 16.2.0), the e2e shards (musl, llvm, + # mingw-cross), the toolchain legs (musl, llvm). + prewarm: gcc 16.2.0; gcc 16.1.0-musl; llvm 23.1.3; mingw-cross 16.2.0 # The gate and the tests the Windows cases need are unit-linux-archive's # (ci-unit.yml), beside this build: with GCC the gate held this job 18.6 # minutes after the binary was built, and every Linux job waited for it. @@ -299,7 +342,7 @@ jobs: host: windows-x86_64 runs-on: windows-latest # The MinGW payload every Windows e2e shard installs. - prewarm: mingw 16.1.0 + prewarm: mingw 16.2.0 # The Windows cases the jobs with Visual Studio masked run (no compiler # there), and ci-fresh-install's released mcpp. tests: e2e/windows @@ -308,7 +351,7 @@ jobs: # and so does the newest release while the pin lags it (ci-bootstrap.yml). bootstrap: needs: changes - if: needs.changes.outputs.code == 'true' + if: ${{ needs.changes.outputs.code == 'true' && needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/ci-bootstrap.yml # THE LINUX UNIT AND C++ E2E TESTS, built once beside the build and run in @@ -317,7 +360,7 @@ jobs: # program; the gate and the shards run them against build-linux's mcpp. unit-linux-archive: needs: changes - if: needs.changes.outputs.code == 'true' + if: ${{ needs.changes.outputs.code == 'true' && needs.changes.outputs.verified != 'true' }} # main's measured history is read from its last run. permissions: contents: read @@ -331,8 +374,21 @@ jobs: # The Windows cases the MinGW cross job runs (the run stage's `cases`). tests: e2e/windows + # THE ARCHIVES A RELEASE OF THIS TREE WOULD SHIP (design 2026-11-20 §5.22, CI2), in the `full` + # tier: release.yml's own builds. The Linux shards test the program of the Linux archive, and a + # release of a tree this run verifies promotes the four archives instead of building them. + release-form: + needs: changes + if: ${{ needs.changes.outputs.code == 'true' && needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} + permissions: + contents: read + uses: ./.github/workflows/release-build.yml + unit-linux: - needs: [build-linux, unit-linux-archive, changes] + needs: [build-linux, unit-linux-archive, changes, release-form] + # The release form exists in the `full` tier alone: a skipped `release-form` does not skip the + # shards, a failed one does. + if: ${{ !cancelled() && !failure() && needs.changes.outputs.verified != 'true' && needs.unit-linux-archive.result == 'success' }} # A failed gate cancels the run (ci-unit.yml). permissions: contents: read @@ -344,9 +400,13 @@ jobs: name: linux runs-on: ubuntu-24.04 archived: ${{ needs.unit-linux-archive.outputs.archived }} - # Four shards of four workers: the measured run of 2026-11-13 reached 65 - # of its programs, 3970 s of work, in the 45 minutes of one job. - shards: '[1,2,3,4]' + # The shards the archive stage planned from main's measured work (four + # without a history): the measured run of 2026-11-13 reached 65 of its + # programs, 3970 s of work, in the 45 minutes of one job of four workers. + shards: ${{ needs.unit-linux-archive.outputs.shards || '[1,2,3,4]' }} + form: ${{ needs.release-form.result == 'success' && 'release' || 'host' }} + # The toolchains whose capabilities the cases probe: build-linux's `prewarm`. + toolchains: gcc 16.2.0; gcc 16.1.0-musl; llvm 23.1.3; mingw-cross 16.2.0 # The Windows cases only a MinGW cross toolchain runs, from Linux: # resources (windres -O coff, since GNU ld cannot consume a .res; # mcpp#365) on the GNU and clang + MinGW rows, a Windows program packed @@ -361,6 +421,7 @@ jobs: linux: needs: [build-linux, changes] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-linux.yml with: # A pull request's label is read when a push starts its run; a run @@ -370,6 +431,7 @@ jobs: linux-e2e: needs: [build-linux, changes, e2e-plan] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-linux-e2e.yml with: set: ${{ needs.changes.outputs.set }} @@ -379,12 +441,14 @@ jobs: macos: needs: [build-macos, changes] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-macos.yml with: set: ${{ needs.changes.outputs.set }} macos-e2e: needs: [build-macos, changes, e2e-plan] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-macos-e2e.yml with: set: ${{ needs.changes.outputs.set }} @@ -393,17 +457,20 @@ jobs: work: ${{ needs.e2e-plan.outputs.macos_work != 'false' }} macos-ios: - needs: build-macos + needs: [build-macos, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/ci-macos-ios.yml windows: needs: [build-windows, changes] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-windows.yml with: set: ${{ needs.changes.outputs.set }} windows-e2e: needs: [build-windows, changes, e2e-plan] + if: ${{ needs.changes.outputs.verified != 'true' }} uses: ./.github/workflows/ci-windows-e2e.yml with: set: ${{ needs.changes.outputs.set }} @@ -412,33 +479,42 @@ jobs: work: ${{ needs.e2e-plan.outputs.windows_work != 'false' }} windows-msvc-xlings: - needs: build-windows + needs: [build-windows, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/ci-windows-msvc-xlings.yml cross: - needs: [build-linux, build-windows] + needs: [build-linux, build-windows, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/cross-build-test.yml target-matrix: - needs: [build-linux, build-linux-arm, build-macos, build-windows] + needs: [build-linux, build-linux-arm, build-macos, build-windows, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/ci-target-matrix.yml # A home the previous release made, run by this commit's mcpp (D9 of the # 2026-10-17 upgrade-continuity design). upgrade: - needs: [build-linux, build-linux-arm, build-macos, build-windows] + needs: [build-linux, build-linux-arm, build-macos, build-windows, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/ci-upgrade.yml + offline: + needs: [build-linux, changes] + if: ${{ needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} + uses: ./.github/workflows/ci-offline.yml + # Pull requests and dispatch only, as when it was a workflow of its own. openkal: - needs: [build-linux, build-linux-arm, build-macos, build-windows] - if: github.event_name != 'push' + needs: [build-linux, build-linux-arm, build-macos, build-windows, changes] + if: ${{ github.event_name != 'push' && needs.changes.outputs.verified != 'true' && needs.changes.outputs.set == 'full' }} uses: ./.github/workflows/openkal-cross.yml e2e-coverage: name: every e2e test runs somewhere needs: [changes, linux-e2e, windows-e2e, macos-e2e] - if: ${{ always() && needs.changes.outputs.code == 'true' && !cancelled() }} + if: ${{ always() && needs.changes.outputs.code == 'true' && !cancelled() && needs.changes.outputs.verified != 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 10 steps: @@ -468,11 +544,12 @@ jobs: # THE WORKFLOW RUN'S TESTS AS ONE REPORT (mtest `ci report`): the unit runs # each host kept (test-report, mtest-unit-*) and the e2e shards' stores # (e2e-store-*), the runs of this workflow run's group together -- the - # summary, the failures, the slowest cases and the metrics. + # summary, the failures, the slowest cases, the builds, the steps, the + # margins, the budget, the ledgers and the flips. test-report: name: the run's tests, reported needs: [changes, linux, unit-linux, macos, windows, linux-e2e, macos-e2e, windows-e2e] - if: ${{ always() && needs.changes.outputs.code == 'true' && !cancelled() }} + if: ${{ always() && needs.changes.outputs.code == 'true' && !cancelled() && needs.changes.outputs.verified != 'true' }} runs-on: ubuntu-24.04 timeout-minutes: 15 steps: @@ -487,11 +564,26 @@ jobs: shell: bash run: | set -euo pipefail - for store in stores/*/; do + # In tests/e2e, whose mtest.toml states the budget and the ledgers the report judges. + cd tests/e2e + for store in "$GITHUB_WORKSPACE"/stores/*/; do [ -f "$store/index.ndjson" ] || continue "$MTEST" store import --from "$store" done - "$MTEST" ci report --group "$GITHUB_RUN_ID" --out test-report.md + # The sections of design 2026-11-20 §7.2: beside the summary and the failures, the builds' + # slowest edges, the steps, the cases near their limit, the budget's excesses, the surface's + # ledger and the cases whose verdict flipped. + "$MTEST" ci report --group "$GITHUB_RUN_ID" --out "$GITHUB_WORKSPACE/test-report.md" \ + --sections summary,failures,slowest,builds,steps,margin,budget,ledgers,flaky,path + # THE GATE OVER THE RUN'S GROUP (design 2026-11-20 §5.16, §5.17): no case over the budget + # without a waiver, every item of the surface used by a passing case or waived, and no waiver + # that excuses nothing. + - name: The gate over the run's group + shell: bash + run: | + set -euo pipefail + cd tests/e2e + "$MTEST" gate - uses: actions/upload-artifact@v4 if: always() with: @@ -499,3 +591,73 @@ jobs: path: test-report.md if-no-files-found: ignore retention-days: 14 + + # A PUSH TO MAIN WHOSE TREE IS VERIFIED (design 2026-11-20 §5.22) runs no test: the run that + # verified it is registered as main's, so that the stores, unit archives and build directories + # later runs read from main (.github/tools/main_run.sh) are that run's. The builds above still + # run, and save the caches only main's runs write (R3). + register: + name: the verified run is main's + needs: changes + if: ${{ needs.changes.outputs.verified == 'true' }} + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - name: The run that verified this tree + env: + RUN: ${{ needs.changes.outputs.verified_run }} + TREE: ${{ needs.changes.outputs.tree }} + run: | + set -euo pipefail + mkdir registered + echo "$RUN" > registered/run + { + echo "## Registered, not tested" + echo "The tree \`$TREE\` was verified by run [$RUN](https://github.com/$GITHUB_REPOSITORY/actions/runs/$RUN);" + echo "its stores and archives are main's, and this run builds and saves the caches." + } >> "$GITHUB_STEP_SUMMARY" + - uses: actions/upload-artifact@v4 + with: + name: registered + path: registered/ + retention-days: 90 + + # THE RUN'S VERDICT, one job a ruleset can require: every job it needs succeeded or was skipped + # by the run's tier. A `full` run that passes leaves `verified-` (design 2026-11-20 + # §5.22): a push to main of the same tree registers this run instead of testing again, and a + # release of it promotes this run's archives (release.yml). + verdict: + name: verdict + needs: [changes, docs, build-linux, build-linux-arm, build-macos, build-windows, bootstrap, + unit-linux-archive, unit-linux, linux, linux-e2e, macos, macos-e2e, macos-ios, windows, + windows-e2e, windows-msvc-xlings, cross, target-matrix, upgrade, offline, openkal, e2e-coverage, + test-report, release-form, register] + if: ${{ always() }} + runs-on: ubuntu-24.04 + timeout-minutes: 5 + steps: + - name: Every job passed or was not this tier's + env: + NEEDS: ${{ toJSON(needs) }} + run: | + set -euo pipefail + bad=$(jq -r 'to_entries[] | select(.value.result == "failure" or .value.result == "cancelled") | "\(.key): \(.value.result)"' <<< "$NEEDS") + if [ -n "$bad" ]; then + echo "::error::the run did not pass:"; echo "$bad"; exit 1 + fi + echo "every job passed or was not this run's" + - name: The tree's mark + if: ${{ needs.changes.outputs.set == 'full' && needs.changes.outputs.verified != 'true' }} + env: + TREE: ${{ needs.changes.outputs.tree }} + run: | + set -euo pipefail + mkdir verified + printf 'tree %s\ncommit %s\nrun %s\n' "$TREE" "$GITHUB_SHA" "$GITHUB_RUN_ID" > verified/mark + echo "The tree \`$TREE\` is verified by this run." >> "$GITHUB_STEP_SUMMARY" + - uses: actions/upload-artifact@v4 + if: ${{ needs.changes.outputs.set == 'full' && needs.changes.outputs.verified != 'true' }} + with: + name: verified-${{ needs.changes.outputs.tree }} + path: verified/ + retention-days: 90 diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index e63a742ae..0030f4644 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -11,7 +11,7 @@ name: cross-build-test # target | toolchain | host→target | run # ----------------------|----------------------------------|---------------|----- # aarch64-linux-musl | aarch64-linux-musl-gcc@16.1.0 | x86_64→arm64 | qemu -# x86_64-w64-mingw32 | mingw-cross-gcc@16.1.0 (MSVCRT) | linux→windows | wine +# x86_64-w64-mingw32 | mingw-cross-gcc@16.2.0 (MSVCRT) | linux→windows | wine # x86_64-linux-musl | x86_64-linux-musl-gcc@16.1.0 | windows→linux | linux job # # The mingw row is OS-cross (same arch, different OS/ABI: ELF→PE), so it lives @@ -132,7 +132,7 @@ jobs: # release assets were uploaded in a broken state (records present, # blobs missing → 404 on GET); re-uploaded clean. The stale-INDEX # half is handled by the marker-clear below. - XLINGS_VERSION: '2026.11.17.2' + XLINGS_VERSION: '2026.11.21.1' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ @@ -322,7 +322,7 @@ jobs: - name: Bootstrap mcpp via xlings env: XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.11.17.2' + XLINGS_VERSION: '2026.11.21.1' run: | tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ @@ -353,7 +353,7 @@ jobs: - name: Install mingw-cross toolchain run: | export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - "$MCPP" toolchain install mingw-cross 16.1.0 + "$MCPP" toolchain install mingw-cross 16.2.0 "$MCPP" toolchain install llvm 23.1.3 echo "MCPP_VENDORED_XLINGS=$XLINGS_BIN" >> "$GITHUB_ENV" diff --git a/.github/workflows/e2e-coverage.yml b/.github/workflows/e2e-coverage.yml index 0ba383919..2fc09968e 100644 --- a/.github/workflows/e2e-coverage.yml +++ b/.github/workflows/e2e-coverage.yml @@ -64,6 +64,15 @@ jobs: "$MCPP" self config bash tests/e2e/run_all.sh || true (cd tests/e2e && "$E2E_MTEST" coverage) + # SELECTION BY FUNCTION IS DECIDED BY MEASUREMENT (design 2026-11-20 §5.11, K8): how widely the + # cases share each file and each function they executed. Below a median share of one half, a + # selection by function selects fewer cases than one by file; the figures go to the summary + # and the design record. + - name: How widely the cases share files and functions + run: | + set -euo pipefail + (cd tests/e2e && "$E2E_MTEST" coverage spread) | tee spread.txt + { echo '## How widely the e2e cases share what they executed'; echo '```'; cat spread.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY" - uses: actions/upload-artifact@v4 with: name: e2e-coverage-linux diff --git a/.github/workflows/jitter.yml b/.github/workflows/jitter.yml new file mode 100644 index 000000000..bcca21f82 --- /dev/null +++ b/.github/workflows/jitter.yml @@ -0,0 +1,139 @@ +name: jitter + +# THE SEARCH FOR AN ORDER DEFECT (design 2026-11-20 §5.19, SPEC-017 O12). A case whose verdict +# depends on an order nothing fixes passes most of the time; jitter makes the other orders likely: +# each program waits a random time, up to the jitter, at its contention points -- mtest's at a +# case's start and at a shared object, mcpp's where it takes a lock and where it publishes. A run +# with jitter is a search, never a verdict of the tree: it runs on main's tree, apart from the pull +# requests' runs and the full verification. +# +# weekly every unit and C++ e2e case of Linux, three seeds. +# a flip a CI run of main ended with a case whose verdict changed within it (failed, then +# passed when its shard ran again, or `flaky`): that case alone, five seeds. +# by hand the cases, the jitter and the seeds given. +# +# The summary names each seed's result: a seed that fails repeats the order with +# `MTEST_JITTER= MTEST_JITTER_SEED= mcpp test --case `. + +on: + schedule: + - cron: "41 3 * * 6" + workflow_run: + workflows: [ci] + types: [completed] + branches: [main] + workflow_dispatch: + inputs: + cases: + description: the cases, one `mcpp test --case` pattern per line; empty for every case + type: string + default: '' + jitter: + description: the most a contention point waits, in milliseconds + type: string + default: '50' + seeds: + description: the seeds, a JSON list + type: string + default: '[1, 2, 3]' + +permissions: + contents: read + actions: read + +jobs: + # The cases whose verdict changed within the CI run that triggered this one, from its stores: + # a case with both a failure and a pass in the run's group, or one recorded `flaky`. + plan: + name: what the jitter runs + runs-on: ubuntu-24.04 + timeout-minutes: 10 + if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion != 'cancelled' }} + outputs: + cases: ${{ steps.plan.outputs.cases }} + seeds: ${{ steps.plan.outputs.seeds }} + jitter: ${{ steps.plan.outputs.jitter }} + run: ${{ steps.plan.outputs.run }} + steps: + - uses: actions/checkout@v4 + - name: The cases, the seeds and the jitter + id: plan + env: + GH_TOKEN: ${{ github.token }} + EVENT: ${{ github.event_name }} + RUN: ${{ github.event.workflow_run.id }} + CASES: ${{ inputs.cases }} + JITTER: ${{ inputs.jitter }} + SEEDS: ${{ inputs.seeds }} + run: | + set -euo pipefail + case "$EVENT" in + workflow_dispatch) + { echo "cases<> "$GITHUB_OUTPUT" + echo "seeds=$SEEDS" >> "$GITHUB_OUTPUT"; echo "jitter=$JITTER" >> "$GITHUB_OUTPUT" + echo "run=true" >> "$GITHUB_OUTPUT" ;; + schedule) + { echo "cases<> "$GITHUB_OUTPUT" + echo "seeds=[1, 2, 3]" >> "$GITHUB_OUTPUT"; echo "jitter=50" >> "$GITHUB_OUTPUT" + echo "run=true" >> "$GITHUB_OUTPUT" ;; + workflow_run) + mkdir -p stores + gh run download "$RUN" -R "$GITHUB_REPOSITORY" -p 'mtest-unit-*' -D stores 2>/dev/null || true + # `/` -> every status the run's attempts recorded. + flips=$(find stores -name cases.ndjson -print0 | xargs -0 -r cat \ + | jq -rs 'group_by(.program + "\u0000" + .id) + | map(select((map(.status) | (index("fail") != null and index("pass") != null)) + or (map(.status) | index("flaky") != null))) + | .[] | .[0].id' | sort -u) + { echo "cases<> "$GITHUB_OUTPUT" + echo "seeds=[1, 2, 3, 4, 5]" >> "$GITHUB_OUTPUT"; echo "jitter=50" >> "$GITHUB_OUTPUT" + if [ -n "$flips" ]; then echo "run=true" >> "$GITHUB_OUTPUT"; else echo "run=false" >> "$GITHUB_OUTPUT"; fi + echo "the cases whose verdict changed in run $RUN:"; echo "${flips:-(none)}" ;; + esac + + build-linux: + needs: plan + if: ${{ needs.plan.outputs.run == 'true' }} + permissions: + contents: read + actions: write + uses: ./.github/workflows/build.yml + with: + host: linux-x86_64 + runs-on: ubuntu-24.04 + gate: '' + + unit-archive: + needs: plan + if: ${{ needs.plan.outputs.run == 'true' }} + permissions: + contents: read + actions: read + uses: ./.github/workflows/ci-unit.yml + with: + stage: archive + host: linux-x86_64 + name: linux-jitter + runs-on: ubuntu-24.04 + + # One run per seed, each a shard of its own over the selected cases. + unit: + needs: [plan, build-linux, unit-archive] + strategy: + fail-fast: false + matrix: + seed: ${{ fromJSON(needs.plan.outputs.seeds) }} + permissions: + contents: read + actions: write + uses: ./.github/workflows/ci-unit.yml + with: + stage: run + host: linux-x86_64 + name: linux-jitter-${{ matrix.seed }} + runs-on: ubuntu-24.04 + archived: ${{ needs.unit-archive.outputs.archived }} + gate: '' + select: ${{ needs.plan.outputs.cases }} + jitter: ${{ needs.plan.outputs.jitter }} + jitter-seed: ${{ matrix.seed }} diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml new file mode 100644 index 000000000..b4d038945 --- /dev/null +++ b/.github/workflows/release-build.yml @@ -0,0 +1,762 @@ +name: release-build + +# THE FOUR ARCHIVES OF A RELEASE, BUILT FROM ONE TREE (design 2026-11-20 §5.22, CI2). Called by +# release.yml, when no run verified the tag's tree, and by ci.yml in its `full` tier, whose +# Linux shards then test the program of the Linux archive: what a release promotes is what a run +# tested. Each job keeps its archives as the artifact `release-`; release.yml creates the +# release from them, from its own run or from the run that verified the tree. + +on: + workflow_call: + inputs: + tag: + description: the release's tag; empty for `v` of the tree's mcpp.toml + type: string + required: false + default: '' + ref: + description: the commit to build; empty for the caller's + type: string + required: false + default: '' + outputs: + tag: + value: ${{ jobs.build-release.outputs.tag }} + version: + value: ${{ jobs.build-release.outputs.version }} + +jobs: + build-release: + name: build (linux / x86_64) + runs-on: ubuntu-24.04 + permissions: + contents: read + timeout-minutes: 60 + outputs: + tag: ${{ steps.stage.outputs.tag }} + version: ${{ steps.stage.outputs.version }} + env: + # mcpp resolves MCPP_HOME from the binary's location by default, + # but here we want to share toolchains with the bootstrap sandbox, + # so we pin to a known path. + MCPP_HOME: /home/runner/.mcpp + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref }} + + - name: Resolve target tag + commit + id: resolve + # The caller's tag, else `v` of mcpp.toml: the tree is the + # commit the caller checked out (`ref`). + run: | + if [ -n "${{ inputs.tag }}" ]; then + TAG="${{ inputs.tag }}" + else + VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) + test -n "$VER" || { echo 'failed to read [package].version from mcpp.toml'; exit 1; } + TAG="v$VER" + fi + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" + echo "Resolved tag: $TAG (commit $(git rev-parse --short HEAD))" + + # Cache mcpp's sandbox: musl-gcc 15.1 + binutils + glibc + linux-headers + # + patchelf + ninja is ~800 MB on disk; without this every release + # rebuilds from cold install. Key on the workspace manifest so a + # toolchain change in mcpp.toml refreshes the cache. + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~/.mcpp + key: mcpp-sandbox-${{ runner.os }}-release-${{ hashFiles('mcpp.toml', '.xlings.json') }} + restore-keys: | + mcpp-sandbox-${{ runner.os }}-release- + + # Cache xlings + xim:mcpp install. + - name: Cache xlings + uses: actions/cache@v4 + with: + path: ~/.xlings + key: xlings-${{ runner.os }}-release-xl0462-${{ hashFiles('.xlings.json') }} + restore-keys: | + xlings-${{ runner.os }}-release-xl0462- + + - name: Bootstrap mcpp via xlings + env: + XLINGS_NON_INTERACTIVE: '1' + # Pin xlings to a known-good version. The upstream install + # script always grabs `latest` (no version override), so we + # download + self-install manually to avoid broken releases. + XLINGS_VERSION: '2026.11.21.1' + run: | + # BY VERSION, NOT BY PRESENCE. A restored ~/.xlings holds the + # xlings of an earlier release, and this binary is the one the + # tarball bundles: 2026.10.10.1 shipped xlings 2026.10.8.1 on + # linux-x86_64 because one was present. + have=$("$HOME/.xlings/subos/default/bin/xlings" --version 2>/dev/null | awk '{print $2}') + if [ "$have" != "$XLINGS_VERSION" ]; then + tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" + bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ + "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ + "/tmp/${tarball}" + tar -xzf "/tmp/${tarball}" -C /tmp + "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install + fi + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + xlings --version + test "$(xlings --version | awk '{print $2}')" = "$XLINGS_VERSION" || { + echo "::error::the xlings this release bundles is $(xlings --version), not $XLINGS_VERSION"; exit 1; } + # Pinned to .xlings.json — a bare `xlings install mcpp` resolves + # "newest in this runner's index copy" and put 0.0.105 (below the + # index floor) into this job. See .github/tools/install_pinned_mcpp.sh. + MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" + + - name: Build + pack release artefact (musl static) + id: stage + # Build for the musl-static target, strip the produced ELF, then + # let `mcpp pack` assemble the tarball (binary + top-level wrapper + # + README + LICENSE, contents at archive root). Inject xlings + # afterwards so install.sh consumers get a single self-contained + # bundle. + run: | + TAG="${{ steps.resolve.outputs.tag }}" + VERSION="${{ steps.resolve.outputs.version }}" + TARBALL_NAME="mcpp-${VERSION}-linux-x86_64.tar.gz" + + # Build first so we can strip the ELF before pack copies it. + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + ARTIFACT=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp --target x86_64-linux-musl) + test -n "$ARTIFACT" + file "$ARTIFACT" | grep -q 'statically linked' + # NB: stripping here is pointless — `mcpp pack` below rebuilds the + # binary and overwrites it, which is why every release up to and + # including 2026.7.28.2 shipped an UNSTRIPPED bin/mcpp despite the + # strip that used to live on this line. The payload is slimmed after + # packing instead, in the inject step below. + + # Pack with the freshly-built mcpp (not the bootstrap) so any + # fixes to the pack code path are exercised in the same release + # they ship in. MCPP_HOME is forced so the new binary uses the + # pinned sandbox instead of resolving relative to its own + # location under target/. + MCPP_HOME="$MCPP_HOME" "$ARTIFACT" pack \ + --target x86_64-linux-musl \ + --mode static \ + -o "${TARBALL_NAME}" + + # Inject xlings: extract → add registry/bin/xlings to the wrapper + # dir → re-tar preserving the wrapper. Since 0.0.4 the bundled + # xlings lives at /registry/bin/xlings (= /bin/xlings). + TARBALL="target/dist/${TARBALL_NAME}" + WRAPPER="${TARBALL_NAME%.tar.gz}" + test -f "$TARBALL" + INJECT=$(mktemp -d) + tar -xzf "$TARBALL" -C "$INJECT" + mkdir -p "$INJECT/$WRAPPER/registry/bin" + cp "$XLINGS_BIN" "$INJECT/$WRAPPER/registry/bin/xlings" + chmod +x "$INJECT/$WRAPPER/registry/bin/xlings" + # Slim AFTER pack (pack rebuilds bin/mcpp) and BEFORE tar. Asserts + # the result, so a strip that silently stops working fails the + # release instead of quietly shipping a 34.8MB tarball again. + bash .github/tools/slim_linux_payload.sh "$INJECT/$WRAPPER" + (cd "$INJECT" && tar -czf "$GITHUB_WORKSPACE/${TARBALL}" "$WRAPPER") + rm -rf "$INJECT" + + # Stage final dist/ (tarball + sidecars) for upload. + mkdir -p dist + cp "$TARBALL" "dist/${TARBALL_NAME}" + (cd dist && cp "${TARBALL_NAME}" "mcpp-linux-x86_64.tar.gz") + (cd dist && sha256sum "${TARBALL_NAME}" "mcpp-linux-x86_64.tar.gz" > SHA256SUMS) + (cd dist && sha256sum "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") + (cd dist && sha256sum "mcpp-linux-x86_64.tar.gz" > "mcpp-linux-x86_64.tar.gz.sha256") + + # Top-level install.sh — fetched by `curl | bash`. + cp install.sh dist/install.sh + chmod +x dist/install.sh + + echo "tag=$TAG" >> $GITHUB_OUTPUT + echo "version=$VERSION" >> $GITHUB_OUTPUT + echo "tarball=${TARBALL_NAME}" >> $GITHUB_OUTPUT + ls -la dist/ + + - name: Smoke-test the bundled tarball + # Extract to a scratch dir and run mcpp from there with MCPP_HOME + # unset — proves the release artefact is genuinely self-contained. + run: | + VERSION="${{ steps.stage.outputs.version }}" + TARBALL_NAME="${{ steps.stage.outputs.tarball }}" + # Wrapper dir inside the tarball matches its stem (mcpp pack + # ties the two together). + WRAPPER="${TARBALL_NAME%.tar.gz}" + SMOKE=$(mktemp -d) + tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" + ROOT="$SMOKE/$WRAPPER" + test -x "$ROOT/bin/mcpp" + test -x "$ROOT/registry/bin/xlings" + test -x "$ROOT/mcpp" + file "$ROOT/bin/mcpp" | grep -q 'statically linked' + env -u MCPP_HOME "$ROOT/bin/mcpp" --version + env -u MCPP_HOME "$ROOT/bin/mcpp" --help | head -10 + # Top-level wrapper reports the same version we're shipping. + env -u MCPP_HOME "$ROOT/mcpp" --version | grep -q "$VERSION" + # MCPP_HOME should auto-resolve to the extracted root. + out=$(env -u MCPP_HOME "$ROOT/bin/mcpp" self env) + # R4 (mcpp#832): the binary pack rebuilt with this release's own + # engine trims the paths it carries. + bash .github/tools/check_release_paths.sh "$ROOT/bin/mcpp" + echo "$out" | grep -q "MCPP_HOME *= *$ROOT" + + - name: Generate source tarball + xpkg.lua via mcpp publish + # Use the freshly-built mcpp to produce the source tarball + xpkg + # descriptor for mcpp-index. The release tarball wraps its + # contents in a `/` directory so the extract path + # is $PUB/$WRAPPER/bin/mcpp. + run: | + VERSION="${{ steps.stage.outputs.version }}" + TARBALL_NAME="${{ steps.stage.outputs.tarball }}" + WRAPPER="${TARBALL_NAME%.tar.gz}" + PUB=$(mktemp -d) + tar -xzf "dist/${TARBALL_NAME}" -C "$PUB" + MCPP_BIN="$PUB/$WRAPPER/bin/mcpp" + env -u MCPP_HOME "$MCPP_BIN" publish --dry-run --allow-dirty + test -f "target/dist/mcpp-${VERSION}.tar.gz" + test -f "target/dist/mcpp.lua" + cp "target/dist/mcpp-${VERSION}.tar.gz" dist/ + cp "target/dist/mcpp.lua" dist/ + ls -la dist/ + + - name: Extract release notes from CHANGELOG + id: notes + run: | + TAG="${{ steps.stage.outputs.tag }}" + VERSION="${{ steps.stage.outputs.version }}" + awk -v v="$VERSION" ' + /^## \[/ { + if (in_section) exit + if ($0 ~ "\\[" v "\\]") { in_section=1; next } + } + in_section { print } + ' CHANGELOG.md > dist/RELEASE_NOTES.md || true + # A RELEASE IS OUTWARD-FACING AND PERMANENT: publishing one whose + # notes silently came out empty is not a lesser failure than any + # other defect this workflow guards against, and until now it + # produced none — v2026.9.18.1 went out with its notes body reading + # literally "(no CHANGELOG entry found for 2026.9.18.1)", because + # CHANGELOG.md had no matching `## []` heading and this + # step wrote a placeholder and moved on rather than stopping. Stop + # instead: no matching section is an authoring mistake (the + # section is missing, or its version string does not match what is + # being tagged), and the fix belongs in CHANGELOG.md before the + # release is retried, not in a release whose notes nobody reads. + if [ ! -s dist/RELEASE_NOTES.md ]; then + echo "::error::CHANGELOG.md has no '## [$VERSION]' section — refusing to publish a release with empty notes. Add that section (docs/92-release.md) and re-run, rather than let this placeholder ship: (no CHANGELOG entry found for $VERSION)" + exit 1 + fi + echo "--- RELEASE_NOTES.md ---" + cat dist/RELEASE_NOTES.md + + - name: Keep the Linux archives for the release + uses: actions/upload-artifact@v4 + with: + name: release-linux-x86_64 + if-no-files-found: error + retention-days: 7 + path: | + dist/mcpp-${{ steps.stage.outputs.version }}-linux-x86_64.tar.gz + dist/mcpp-${{ steps.stage.outputs.version }}-linux-x86_64.tar.gz.sha256 + dist/mcpp-linux-x86_64.tar.gz + dist/mcpp-linux-x86_64.tar.gz.sha256 + dist/install.sh + dist/mcpp-${{ steps.stage.outputs.version }}.tar.gz + dist/mcpp.lua + dist/RELEASE_NOTES.md + + build-linux-aarch64: + name: build (linux / aarch64, cross) + runs-on: ubuntu-24.04 + permissions: + contents: read + timeout-minutes: 70 + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref }} + + - name: Install system deps + qemu + run: | + # The runner image carries third-party apt lists (Google Chrome + # among them) that this job does not use, and a transient + # `Hash Sum mismatch` on one of them fails the whole update -- which + # killed two cross-build jobs in setup, before a single byte was + # compiled. Dropping the lists this job has no use for is what makes + # the step's failure mean something about this job. + # + # BY CONTENT, NOT BY FILENAME. The first attempt removed + # `google-chrome.list` and the update failed on the same URL: on + # ubuntu-24.04 the runner writes deb822 `.sources` files, so the + # name was a guess and the guess was wrong. + sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ + /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f + sudo apt-get update -qq + sudo apt-get install -y curl git build-essential qemu-user-static + qemu-aarch64-static --version | head -1 + + - name: Resolve tag + version + id: resolve + run: | + VERSION=$(grep -E '^version' mcpp.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/') + echo "version=$VERSION" >> "$GITHUB_OUTPUT" + echo "tag=v$VERSION" >> "$GITHUB_OUTPUT" + + - name: Bootstrap mcpp via xlings + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: '2026.11.21.1' + run: | + tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" + bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ + "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ + "/tmp/${tarball}" + tar -xzf "/tmp/${tarball}" -C /tmp + "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install + echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" + echo "$HOME/.xlings/bin" >> "$GITHUB_PATH" + echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" + + - name: Bootstrap mcpp + refresh index (latest, GLOBAL) + run: | + xlings config --mirror GLOBAL 2>/dev/null || true + xlings update -y 2>/dev/null || xlings update 2>/dev/null || true + MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + "$MCPP" --version + + - name: Cross-build mcpp -> aarch64-linux-musl (this release's source) + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + # "$MCPP", not a bare `mcpp`: the bare form runs the PATH shim, which + # resolves to whatever version xvm has selected — so pinning the + # bootstrap would have looked correct and changed nothing here. + "$MCPP" self config --mirror GLOBAL 2>/dev/null || true + # The published bootstrap mcpp predates aarch64 cross-build support + # (the feature landed after the last release), so it resolves the + # x86_64 host musl-gcc for an aarch64 target. Two-stage instead: build + # THIS release's x86_64 mcpp first, then cross-build aarch64 with it. + FRESH=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp --target x86_64-linux-musl) + test -n "$FRESH" + BIN=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$FRESH" mcpp --target aarch64-linux-musl) + test -n "$BIN" + file "$BIN" | grep -q 'ARM aarch64' + file "$BIN" | grep -q 'statically linked' + echo "MCPP_AARCH64=$BIN" >> "$GITHUB_ENV" + + - name: Package aarch64 release (+ bundle aarch64 xlings) + id: stage + run: | + VERSION="${{ steps.resolve.outputs.version }}" + TARBALL_NAME="mcpp-${VERSION}-linux-aarch64.tar.gz" + WRAPPER="mcpp-${VERSION}-linux-aarch64" + STAGING=$(mktemp -d) + mkdir -p "$STAGING/$WRAPPER/bin" + cp "$MCPP_AARCH64" "$STAGING/$WRAPPER/bin/mcpp" + # The binary is aarch64, so the host x86_64 `strip` cannot touch it — + # resolve a cross-capable one. Required, not best-effort: the old + # `|| true` here meant a missing tool silently shipped a fat tarball. + # (slim_linux_payload.sh runs after xlings is staged, below.) + STRIP=$(find "$HOME/.mcpp" -name 'aarch64-linux-musl-strip' -type f 2>/dev/null | head -1) + [ -n "$STRIP" ] || STRIP=$(command -v llvm-strip 2>/dev/null || true) + [ -n "$STRIP" ] || STRIP=$(command -v aarch64-linux-gnu-strip 2>/dev/null || true) + [ -n "$STRIP" ] || { echo "no aarch64-capable strip found"; exit 1; } + echo "aarch64 strip: $STRIP" + cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true + cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true + cat > "$STAGING/$WRAPPER/mcpp" << 'LAUNCHER' + #!/bin/sh + exec "$(dirname "$0")/bin/mcpp" "$@" + LAUNCHER + chmod +x "$STAGING/$WRAPPER/mcpp" + # Bundle the aarch64 xlings so install.sh consumers on aarch64 get an + # aarch64 xlings, not the x86_64 bootstrap one. The three literals + # below are pinned to the same version as XLINGS_VERSION; they are + # NOT interpolated from it, so check_version_pins.sh scans for them + # explicitly (they were absent from the old lock-step comment). + XLA="xlings-2026.11.21.1-linux-aarch64.tar.gz" + # NOT fetch_release.sh: this asset is OPTIONAL and the `if` is the + # point — an arch with no prebuilt xlings must fall through quietly, + # while the helper retries a 404 five times before giving up. The one + # flag that matters here is --retry-all-errors: `curl: (52) Empty + # reply from server` is a transport error, so plain --retry does not + # cover it. + if curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ + --connect-timeout 20 --max-time 600 -o "/tmp/$XLA" \ + "https://github.com/openxlings/xlings/releases/download/v2026.11.21.1/$XLA"; then + tar -xzf "/tmp/$XLA" -C /tmp + XLBIN=$(find /tmp/xlings-2026.11.21.1-linux-aarch64 -path '*/bin/xlings' -type f | head -1) + if [ -n "$XLBIN" ]; then + mkdir -p "$STAGING/$WRAPPER/registry/bin" + cp "$XLBIN" "$STAGING/$WRAPPER/registry/bin/xlings" + chmod +x "$STAGING/$WRAPPER/registry/bin/xlings" + fi + fi + # Slim both shipped ELFs with the cross strip resolved above, and + # assert the result (the vendored xlings was 86.9MB unstripped here). + bash .github/tools/slim_linux_payload.sh "$STAGING/$WRAPPER" "$STRIP" + mkdir -p dist + (cd "$STAGING" && tar -czf "$GITHUB_WORKSPACE/dist/${TARBALL_NAME}" "$WRAPPER") + cp "dist/${TARBALL_NAME}" "dist/mcpp-linux-aarch64.tar.gz" + (cd dist && sha256sum "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") + (cd dist && sha256sum "mcpp-linux-aarch64.tar.gz" > "mcpp-linux-aarch64.tar.gz.sha256") + echo "tarball=${TARBALL_NAME}" >> "$GITHUB_OUTPUT" + ls -la dist/ + + - name: Smoke-test the aarch64 tarball (qemu) + run: | + TARBALL_NAME="${{ steps.stage.outputs.tarball }}" + WRAPPER="${TARBALL_NAME%.tar.gz}" + SMOKE=$(mktemp -d) + tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" + ver=$(qemu-aarch64-static "$SMOKE/$WRAPPER/bin/mcpp" --version) + echo "$ver"; echo "$ver" | grep -q 'mcpp' + + - name: Keep the aarch64 archives for the release + uses: actions/upload-artifact@v4 + with: + name: release-linux-aarch64 + if-no-files-found: error + retention-days: 7 + path: | + dist/mcpp-${{ steps.resolve.outputs.version }}-linux-aarch64.tar.gz + dist/mcpp-${{ steps.resolve.outputs.version }}-linux-aarch64.tar.gz.sha256 + dist/mcpp-linux-aarch64.tar.gz + dist/mcpp-linux-aarch64.tar.gz.sha256 + + build-macos: + name: build (macOS / ARM64) + runs-on: macos-15 + permissions: + contents: read + timeout-minutes: 30 + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref }} + + - name: Resolve tag + id: resolve + run: | + if [ -n "${{ inputs.tag }}" ]; then + TAG="${{ inputs.tag }}" + else + VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) + TAG="v$VER" + fi + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" + + - name: Cache xlings + uses: actions/cache@v4 + with: + path: ~/.xlings + key: xlings-macos15-release-xl0462-${{ hashFiles('.xlings.json') }} + restore-keys: | + xlings-macos15-release-xl0462- + + - name: Bootstrap mcpp via xlings + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: '2026.11.21.1' + run: | + # By version, not by presence (see the Linux job). + have=$("$HOME/.xlings/subos/default/bin/xlings" --version 2>/dev/null | awk '{print $2}') + if [ "$have" != "$XLINGS_VERSION" ]; then + WORK=$(mktemp -d) + tarball="xlings-${XLINGS_VERSION}-macosx-arm64.tar.gz" + bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ + "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ + "${WORK}/${tarball}" + tar -xzf "${WORK}/${tarball}" -C "${WORK}" + "${WORK}/xlings-${XLINGS_VERSION}-macosx-arm64/subos/default/bin/xlings" self install + fi + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + xlings --version + test "$(xlings --version | awk '{print $2}')" = "$XLINGS_VERSION" || { + echo "::error::the xlings this release bundles is $(xlings --version), not $XLINGS_VERSION"; exit 1; } + # Pinned to .xlings.json — a bare `xlings install mcpp` resolves + # "newest in this runner's index copy" and put 0.0.105 (below the + # index floor) into this job. See .github/tools/install_pinned_mcpp.sh. + MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" + + - name: Build mcpp from source (two-stage self-host) + env: + # macOS min-version support: target macOS 14 so the release runs + # on 14.0+ instead of only the runner's OS (the official LLVM + # static libc++ archives are built for macOS 14 — going lower + # needs a custom libc++ build, tracked as follow-up). Needs + # static LLVM libc++ — the system libc++ on older macOS lacks + # LLVM-20-era C++23 symbols (std::print's __is_posix_terminal + # etc.; minos-14 + dynamic libc++ dies at launch on macos-14 CI). + # See xlings .agents/docs/2026-06-05-macos-min-version-support.md. + MACOSX_DEPLOYMENT_TARGET: '14.0' + run: | + export PATH="$HOME/.xlings/subos/default/bin:$PATH" + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + + # Stage 1: the bootstrap mcpp builds this release's source. The + # bootstrap's macOS link path predates the staticStdlib + # implementation (hardcoded -lc++), so stage 1 links the system + # libc++ — fine, it only needs to RUN on this runner. + STAGE1=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp) + # --no-cache removes target/, including the stage-1 compiler. Ninja + # invokes that compiler again for dependency scanning, so keep the + # bootstrap outside the tree the second stage is about to clear. + STAGE1_COPY=$(mktemp -d "$RUNNER_TEMP/mcpp-stage1.XXXXXX")/mcpp + cp "$STAGE1" "$STAGE1_COPY" + STAGE1="$STAGE1_COPY" + "$STAGE1" --version + + # Stage 2: this release's mcpp rebuilds itself — flags.cppm's + # native staticStdlib link produces the static minos-14 binary. + # Stage 2 lands in a NEW fingerprint directory (its fingerprint + # includes the deployment target; the bootstrap's did not): the + # build names its own product. + MCPP_BIN=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$STAGE1" mcpp --no-cache) + test -x "$MCPP_BIN" + file "$MCPP_BIN" + otool -L "$MCPP_BIN" + echo "=== LC_BUILD_VERSION (must be minos 14.0) ===" + otool -l "$MCPP_BIN" | grep -A4 LC_BUILD_VERSION | head -6 + otool -l "$MCPP_BIN" | grep -A4 LC_BUILD_VERSION | grep -q "minos 14.0" \ + || { echo "FAIL: expected minos 14.0"; exit 1; } + if otool -L "$MCPP_BIN" | grep -q "libc++"; then + echo "FAIL: still linked against system libc++"; exit 1 + fi + "$MCPP_BIN" --version + echo "MCPP_BIN=$MCPP_BIN" >> "$GITHUB_ENV" + + - name: Package macOS release + id: stage + run: | + VERSION="${{ steps.resolve.outputs.version }}" + TARBALL_NAME="mcpp-${VERSION}-macosx-arm64.tar.gz" + WRAPPER="mcpp-${VERSION}-macosx-arm64" + + # Create release layout + STAGING=$(mktemp -d) + mkdir -p "$STAGING/$WRAPPER/bin" + cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp" + # Strip (Mach-O) + strip "$STAGING/$WRAPPER/bin/mcpp" 2>/dev/null || true + # Copy metadata + cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true + cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true + + # Shell launcher (same as Linux) + cat > "$STAGING/$WRAPPER/mcpp" << 'LAUNCHER' + #!/bin/sh + exec "$(dirname "$0")/bin/mcpp" "$@" + LAUNCHER + chmod +x "$STAGING/$WRAPPER/mcpp" + + # Bundle xlings for install.sh consumers + XLINGS_BIN="$HOME/.xlings/subos/default/bin/xlings" + if [ -x "$XLINGS_BIN" ]; then + mkdir -p "$STAGING/$WRAPPER/registry/bin" + cp "$XLINGS_BIN" "$STAGING/$WRAPPER/registry/bin/xlings" + chmod +x "$STAGING/$WRAPPER/registry/bin/xlings" + fi + + # Create tarball + mkdir -p dist + (cd "$STAGING" && tar -czf "$GITHUB_WORKSPACE/dist/${TARBALL_NAME}" "$WRAPPER") + # Versionless alias + cp "dist/${TARBALL_NAME}" "dist/mcpp-macosx-arm64.tar.gz" + # SHA256 + (cd dist && shasum -a 256 "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") + (cd dist && shasum -a 256 "mcpp-macosx-arm64.tar.gz" > "mcpp-macosx-arm64.tar.gz.sha256") + + echo "tarball=${TARBALL_NAME}" >> "$GITHUB_OUTPUT" + ls -la dist/ + + - name: Smoke-test the tarball + run: | + VERSION="${{ steps.resolve.outputs.version }}" + TARBALL_NAME="${{ steps.stage.outputs.tarball }}" + WRAPPER="${TARBALL_NAME%.tar.gz}" + SMOKE=$(mktemp -d) + tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" + "$SMOKE/$WRAPPER/bin/mcpp" --version + "$SMOKE/$WRAPPER/mcpp" --version | grep -q "$VERSION" + # R4 (mcpp#832): stage 2 is this release's own engine, and it trims + # the paths the binary carries. + bash .github/tools/check_release_paths.sh "$SMOKE/$WRAPPER/bin/mcpp" + + - name: Keep the macOS archives for the release + uses: actions/upload-artifact@v4 + with: + name: release-macosx-arm64 + if-no-files-found: error + retention-days: 7 + path: | + dist/mcpp-${{ steps.resolve.outputs.version }}-macosx-arm64.tar.gz + dist/mcpp-${{ steps.resolve.outputs.version }}-macosx-arm64.tar.gz.sha256 + dist/mcpp-macosx-arm64.tar.gz + dist/mcpp-macosx-arm64.tar.gz.sha256 + + build-windows: + name: build (Windows / x86_64) + runs-on: windows-latest + permissions: + contents: read + timeout-minutes: 45 + env: + MCPP_HOME: C:\Users\runneradmin\.mcpp + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref }} + + - name: Resolve tag + id: resolve + shell: bash + run: | + if [ -n "${{ inputs.tag }}" ]; then + TAG="${{ inputs.tag }}" + else + VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) + TAG="v$VER" + fi + echo "tag=$TAG" >> "$GITHUB_OUTPUT" + echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" + + - name: Cache mcpp sandbox + uses: actions/cache@v4 + with: + path: ~\.mcpp + key: mcpp-sandbox-${{ runner.os }}-release-${{ hashFiles('mcpp.toml', '.xlings.json') }} + restore-keys: | + mcpp-sandbox-${{ runner.os }}-release- + + - name: Cache xlings + uses: actions/cache@v4 + with: + path: ~\.xlings + key: xlings-${{ runner.os }}-release-xl0462-${{ hashFiles('.xlings.json') }} + restore-keys: | + xlings-${{ runner.os }}-release-xl0462- + + - name: Bootstrap mcpp via xlings + shell: bash + env: + XLINGS_NON_INTERACTIVE: '1' + XLINGS_VERSION: '2026.11.21.1' + run: | + # Captured before the `cd` below, in POSIX form: this step never + # returns to the workspace, and GITHUB_WORKSPACE is a backslash + # Windows path that git-bash tools mangle. + REPO_DIR="$(pwd)" + WORK=$(mktemp -d) + zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip" + bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ + "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${zipfile}" \ + "${WORK}/${zipfile}" + cd "${WORK}" + unzip -q "${zipfile}" + "$WORK/xlings-${XLINGS_VERSION}-windows-x86_64/subos/default/bin/xlings.exe" self install + export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH" + echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH" + xlings.exe --version + test "$(xlings.exe --version | awk '{print $2}' | tr -d '\r')" = "$XLINGS_VERSION" || { + echo "::error::the xlings this release bundles is not $XLINGS_VERSION"; exit 1; } + # Pinned + version-scoped lookup. The old `find | head -1` returned + # whichever version the directory walk reached first. + MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR") + echo "MCPP=$MCPP" >> "$GITHUB_ENV" + XLINGS_BIN=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") + echo "XLINGS_BIN=$XLINGS_BIN" >> "$GITHUB_ENV" + echo "XLINGS_BIN_UNIX=$USERPROFILE/.xlings/subos/default/bin/xlings.exe" >> "$GITHUB_ENV" + echo "XLINGS_XPKGS=$USERPROFILE/.xlings/data/xpkgs" >> "$GITHUB_ENV" + + - name: Build mcpp from source (self-host) + shell: bash + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + + "$MCPP" build + # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is + # restored from cache and keeps a directory per build fingerprint, + # so after a version bump the freshly built binary sits alongside + # the previous release's. `find | head -1` returned whichever the + # directory walk hit first — which is how a 0.0.106 build ran the + # 0.0.105 binary and failed 01_help_and_version. + MCPP_BIN=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ + | sort -rn | head -1 | cut -d" " -f2-) + test -n "$MCPP_BIN" || { echo "FAIL: no mcpp.exe in target/"; exit 1; } + MCPP_BIN=$(cd "$(dirname "$MCPP_BIN")" && pwd)/$(basename "$MCPP_BIN") + echo "Self-hosted binary: $MCPP_BIN" + "$MCPP_BIN" --version + echo "MCPP_BIN=$MCPP_BIN" >> "$GITHUB_ENV" + + - name: Package Windows release zip + id: stage + shell: bash + run: | + VERSION="${{ steps.resolve.outputs.version }}" + WRAPPER="mcpp-${VERSION}-windows-x86_64" + ZIPNAME="${WRAPPER}.zip" + + STAGING=$(mktemp -d) + mkdir -p "$STAGING/$WRAPPER/bin" "$STAGING/$WRAPPER/registry/bin" + cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp.exe" + + # Windows batch launcher + printf '@echo off\r\n"%%~dp0bin\\mcpp.exe" %%*\r\n' > "$STAGING/$WRAPPER/mcpp.bat" + cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true + cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true + + # Bundle xlings.exe for install consumers + if [ -f "$XLINGS_BIN_UNIX" ]; then + cp "$XLINGS_BIN_UNIX" "$STAGING/$WRAPPER/registry/bin/xlings.exe" + fi + + # Pack with 7z (available on windows-latest) + mkdir -p dist + (cd "$STAGING" && 7z a -tzip "$ZIPNAME" "$WRAPPER") + cp "$STAGING/$ZIPNAME" "dist/$ZIPNAME" + # Versionless alias + cp "dist/$ZIPNAME" "dist/mcpp-windows-x86_64.zip" + # SHA256 + (cd dist && sha256sum "$ZIPNAME" > "$ZIPNAME.sha256") + (cd dist && sha256sum "mcpp-windows-x86_64.zip" > "mcpp-windows-x86_64.zip.sha256") + + echo "zipname=$ZIPNAME" >> "$GITHUB_OUTPUT" + ls -la dist/ + + - name: Smoke-test the packaged zip + shell: bash + run: | + ZIPNAME="${{ steps.stage.outputs.zipname }}" + WRAPPER="${ZIPNAME%.zip}" + SMOKE=$(mktemp -d) + (cd "$SMOKE" && unzip -q "$GITHUB_WORKSPACE/dist/$ZIPNAME") + "$SMOKE/$WRAPPER/bin/mcpp.exe" --version + "$SMOKE/$WRAPPER/bin/mcpp.exe" --help | head -5 + test -f "$SMOKE/$WRAPPER/registry/bin/xlings.exe" + test -f "$SMOKE/$WRAPPER/mcpp.bat" + echo "Smoke-test passed" + + - name: Keep the Windows archives for the release + uses: actions/upload-artifact@v4 + with: + name: release-windows-x86_64 + if-no-files-found: error + retention-days: 7 + path: | + dist/mcpp-${{ steps.resolve.outputs.version }}-windows-x86_64.zip + dist/mcpp-${{ steps.resolve.outputs.version }}-windows-x86_64.zip.sha256 + dist/mcpp-windows-x86_64.zip + dist/mcpp-windows-x86_64.zip.sha256 diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9f46dc603..19c17d588 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,9 +1,10 @@ name: release -# Self-host release: bootstrap mcpp from xlings (xim:mcpp), build the -# musl-static artefact via `mcpp pack --target x86_64-linux-musl -o ...`, -# inject xlings into the produced tarball for install.sh consumers, -# smoke-test, upload. +# The release of a tag: its four archives, promoted from the run that verified +# the tag's tree or built by release-build.yml (bootstrap mcpp from xlings, +# `mcpp pack --target x86_64-linux-musl`, xlings injected for install.sh +# consumers, smoke tests), then the release, its sealed manifest and the +# ecosystem's publication. on: push: @@ -28,7 +29,9 @@ on: # when all four have passed, so no release exists with a platform missing. The # release v2026.11.12.1 took 109 minutes, its jobs one after another: the # canaries 48, the Linux build 27 (it created the release the others waited -# for), the aarch64 build 29, the seal and the publication 5. +# for), the aarch64 build 29, the seal and the publication 5. The builds are +# release-build.yml's, which ci.yml's `full` tier runs too; a tree such a run +# verified is not built again (design 2026-11-20 §5.22): `find`, `promote`. jobs: # THE ECOSYSTEM'S OWN PROJECTS, BUILT WITH THE CANDIDATE (the 2026-09-28 # design, WS10; .github/release-canaries.toml), when the release is started @@ -39,772 +42,142 @@ jobs: if: github.event_name == 'workflow_dispatch' && inputs.canaries uses: ./.github/workflows/release-canaries.yml - build-release: - name: build (linux / x86_64) + # THE TAG'S TREE, AND THE RUN THAT VERIFIED IT (design 2026-11-20 §5.22, CI3). A `full` run of + # ci.yml that passed left `verified-`, and its `release-form` job kept the four archives + # this workflow would build (release-build.yml). When the tag's tree has that mark and the run's + # archives are still kept, the release promotes them (`promote`): the archives a run tested are + # the ones released. Otherwise it builds them (`build`), and the summary says which it did. + find: + name: the tag's tree and the run that verified it runs-on: ubuntu-24.04 permissions: contents: read - timeout-minutes: 60 + actions: read + timeout-minutes: 10 outputs: - tag: ${{ steps.stage.outputs.tag }} - version: ${{ steps.stage.outputs.version }} - env: - # mcpp resolves MCPP_HOME from the binary's location by default, - # but here we want to share toolchains with the bootstrap sandbox, - # so we pin to a known path. - MCPP_HOME: /home/runner/.mcpp + tag: ${{ steps.find.outputs.tag }} + version: ${{ steps.find.outputs.version }} + ref: ${{ steps.find.outputs.ref }} + run: ${{ steps.find.outputs.run }} steps: # fetch-depth: 0 instead of fetch-tags: true — actions/checkout@v4 # fails on push-tag triggers when both the ref'd tag and # `fetch-tags: true` are set: # "Cannot fetch both and refs/tags/vX.Y.Z to refs/tags/vX.Y.Z" - # Full-history fetch covers the resolve-tag step's needs without - # that contention. - uses: actions/checkout@v4 with: fetch-depth: 0 - - name: Resolve target tag + commit - id: resolve - # Three trigger shapes converge here: - # 1. push: refs/tags/vX.Y.Z → use that tag, build at its commit - # 2. workflow_dispatch with `tag` input set: - # - tag exists on remote → check it out (rebuild scenario) - # - tag doesn't exist → use current HEAD; gh-release - # creates the tag at that commit on upload - # 3. workflow_dispatch with no input → derive `v` from - # mcpp.toml's [package].version, build at current HEAD; - # gh-release creates the tag. + # Three trigger shapes converge here: + # 1. push: refs/tags/vX.Y.Z → that tag, at its commit + # 2. workflow_dispatch with `tag`: + # - the tag exists → its commit (a rebuild) + # - it does not → this run's commit; the release creates the tag there + # 3. workflow_dispatch without `tag` → `v` of mcpp.toml, as in 2. + - name: The tag, its commit and tree, and the run that verified the tree + id: find + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + EVENT: ${{ github.event_name }} + INPUT_TAG: ${{ inputs.tag }} run: | - if [ "${{ github.event_name }}" = "push" ]; then - TAG="${{ github.ref_name }}" - elif [ -n "${{ github.event.inputs.tag }}" ]; then - TAG="${{ github.event.inputs.tag }}" + set -euo pipefail + if [ "$EVENT" = push ]; then + TAG="$GITHUB_REF_NAME" + elif [ -n "$INPUT_TAG" ]; then + TAG="$INPUT_TAG" else VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) test -n "$VER" || { echo 'failed to read [package].version from mcpp.toml'; exit 1; } TAG="v$VER" fi - echo "tag=$TAG" >> "$GITHUB_OUTPUT" - echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" - # If the tag exists on remote AND we're on workflow_dispatch, - # check it out so we rebuild that exact commit. push-tag runs - # already start at the tag commit. - if [ "${{ github.event_name }}" = "workflow_dispatch" ] \ - && git rev-parse --verify "refs/tags/$TAG" >/dev/null 2>&1; then - git checkout --detach "refs/tags/$TAG" - fi - echo "Resolved tag: $TAG (commit $(git rev-parse --short HEAD))" - - # Cache mcpp's sandbox: musl-gcc 15.1 + binutils + glibc + linux-headers - # + patchelf + ninja is ~800 MB on disk; without this every release - # rebuilds from cold install. Key on the workspace manifest so a - # toolchain change in mcpp.toml refreshes the cache. - - name: Cache mcpp sandbox - uses: actions/cache@v4 - with: - path: ~/.mcpp - key: mcpp-sandbox-${{ runner.os }}-release-${{ hashFiles('mcpp.toml', '.xlings.json') }} - restore-keys: | - mcpp-sandbox-${{ runner.os }}-release- - - # Cache xlings + xim:mcpp install. - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~/.xlings - key: xlings-${{ runner.os }}-release-xl0462-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-${{ runner.os }}-release-xl0462- - - - name: Bootstrap mcpp via xlings - env: - XLINGS_NON_INTERACTIVE: '1' - # Pin xlings to a known-good version. The upstream install - # script always grabs `latest` (no version override), so we - # download + self-install manually to avoid broken releases. - XLINGS_VERSION: '2026.11.17.2' - run: | - # BY VERSION, NOT BY PRESENCE. A restored ~/.xlings holds the - # xlings of an earlier release, and this binary is the one the - # tarball bundles: 2026.10.10.1 shipped xlings 2026.10.8.1 on - # linux-x86_64 because one was present. - have=$("$HOME/.xlings/subos/default/bin/xlings" --version 2>/dev/null | awk '{print $2}') - if [ "$have" != "$XLINGS_VERSION" ]; then - tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" - bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ - "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ - "/tmp/${tarball}" - tar -xzf "/tmp/${tarball}" -C /tmp - "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install - fi - export PATH="$HOME/.xlings/subos/default/bin:$PATH" - xlings --version - test "$(xlings --version | awk '{print $2}')" = "$XLINGS_VERSION" || { - echo "::error::the xlings this release bundles is $(xlings --version), not $XLINGS_VERSION"; exit 1; } - # Pinned to .xlings.json — a bare `xlings install mcpp` resolves - # "newest in this runner's index copy" and put 0.0.105 (below the - # index floor) into this job. See .github/tools/install_pinned_mcpp.sh. - MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - - name: Build + pack release artefact (musl static) - id: stage - # Build for the musl-static target, strip the produced ELF, then - # let `mcpp pack` assemble the tarball (binary + top-level wrapper - # + README + LICENSE, contents at archive root). Inject xlings - # afterwards so install.sh consumers get a single self-contained - # bundle. - run: | - TAG="${{ steps.resolve.outputs.tag }}" - VERSION="${{ steps.resolve.outputs.version }}" - TARBALL_NAME="mcpp-${VERSION}-linux-x86_64.tar.gz" - - # Build first so we can strip the ELF before pack copies it. - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - ARTIFACT=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp --target x86_64-linux-musl) - test -n "$ARTIFACT" - file "$ARTIFACT" | grep -q 'statically linked' - # NB: stripping here is pointless — `mcpp pack` below rebuilds the - # binary and overwrites it, which is why every release up to and - # including 2026.7.28.2 shipped an UNSTRIPPED bin/mcpp despite the - # strip that used to live on this line. The payload is slimmed after - # packing instead, in the inject step below. - - # Pack with the freshly-built mcpp (not the bootstrap) so any - # fixes to the pack code path are exercised in the same release - # they ship in. MCPP_HOME is forced so the new binary uses the - # pinned sandbox instead of resolving relative to its own - # location under target/. - MCPP_HOME="$MCPP_HOME" "$ARTIFACT" pack \ - --target x86_64-linux-musl \ - --mode static \ - -o "${TARBALL_NAME}" - - # Inject xlings: extract → add registry/bin/xlings to the wrapper - # dir → re-tar preserving the wrapper. Since 0.0.4 the bundled - # xlings lives at /registry/bin/xlings (= /bin/xlings). - TARBALL="target/dist/${TARBALL_NAME}" - WRAPPER="${TARBALL_NAME%.tar.gz}" - test -f "$TARBALL" - INJECT=$(mktemp -d) - tar -xzf "$TARBALL" -C "$INJECT" - mkdir -p "$INJECT/$WRAPPER/registry/bin" - cp "$XLINGS_BIN" "$INJECT/$WRAPPER/registry/bin/xlings" - chmod +x "$INJECT/$WRAPPER/registry/bin/xlings" - # Slim AFTER pack (pack rebuilds bin/mcpp) and BEFORE tar. Asserts - # the result, so a strip that silently stops working fails the - # release instead of quietly shipping a 34.8MB tarball again. - bash .github/tools/slim_linux_payload.sh "$INJECT/$WRAPPER" - (cd "$INJECT" && tar -czf "$GITHUB_WORKSPACE/${TARBALL}" "$WRAPPER") - rm -rf "$INJECT" - - # Stage final dist/ (tarball + sidecars) for upload. - mkdir -p dist - cp "$TARBALL" "dist/${TARBALL_NAME}" - (cd dist && cp "${TARBALL_NAME}" "mcpp-linux-x86_64.tar.gz") - (cd dist && sha256sum "${TARBALL_NAME}" "mcpp-linux-x86_64.tar.gz" > SHA256SUMS) - (cd dist && sha256sum "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") - (cd dist && sha256sum "mcpp-linux-x86_64.tar.gz" > "mcpp-linux-x86_64.tar.gz.sha256") - - # Top-level install.sh — fetched by `curl | bash`. - cp install.sh dist/install.sh - chmod +x dist/install.sh - - echo "tag=$TAG" >> $GITHUB_OUTPUT - echo "version=$VERSION" >> $GITHUB_OUTPUT - echo "tarball=${TARBALL_NAME}" >> $GITHUB_OUTPUT - ls -la dist/ - - - name: Smoke-test the bundled tarball - # Extract to a scratch dir and run mcpp from there with MCPP_HOME - # unset — proves the release artefact is genuinely self-contained. - run: | - VERSION="${{ steps.stage.outputs.version }}" - TARBALL_NAME="${{ steps.stage.outputs.tarball }}" - # Wrapper dir inside the tarball matches its stem (mcpp pack - # ties the two together). - WRAPPER="${TARBALL_NAME%.tar.gz}" - SMOKE=$(mktemp -d) - tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" - ROOT="$SMOKE/$WRAPPER" - test -x "$ROOT/bin/mcpp" - test -x "$ROOT/registry/bin/xlings" - test -x "$ROOT/mcpp" - file "$ROOT/bin/mcpp" | grep -q 'statically linked' - env -u MCPP_HOME "$ROOT/bin/mcpp" --version - env -u MCPP_HOME "$ROOT/bin/mcpp" --help | head -10 - # Top-level wrapper reports the same version we're shipping. - env -u MCPP_HOME "$ROOT/mcpp" --version | grep -q "$VERSION" - # MCPP_HOME should auto-resolve to the extracted root. - out=$(env -u MCPP_HOME "$ROOT/bin/mcpp" self env) - # R4 (mcpp#832): the binary pack rebuilt with this release's own - # engine trims the paths it carries. - bash .github/tools/check_release_paths.sh "$ROOT/bin/mcpp" - echo "$out" | grep -q "MCPP_HOME *= *$ROOT" - - - name: Generate source tarball + xpkg.lua via mcpp publish - # Use the freshly-built mcpp to produce the source tarball + xpkg - # descriptor for mcpp-index. The release tarball wraps its - # contents in a `/` directory so the extract path - # is $PUB/$WRAPPER/bin/mcpp. - run: | - VERSION="${{ steps.stage.outputs.version }}" - TARBALL_NAME="${{ steps.stage.outputs.tarball }}" - WRAPPER="${TARBALL_NAME%.tar.gz}" - PUB=$(mktemp -d) - tar -xzf "dist/${TARBALL_NAME}" -C "$PUB" - MCPP_BIN="$PUB/$WRAPPER/bin/mcpp" - env -u MCPP_HOME "$MCPP_BIN" publish --dry-run --allow-dirty - test -f "target/dist/mcpp-${VERSION}.tar.gz" - test -f "target/dist/mcpp.lua" - cp "target/dist/mcpp-${VERSION}.tar.gz" dist/ - cp "target/dist/mcpp.lua" dist/ - ls -la dist/ - - - name: Extract release notes from CHANGELOG - id: notes - run: | - TAG="${{ steps.stage.outputs.tag }}" - VERSION="${{ steps.stage.outputs.version }}" - awk -v v="$VERSION" ' - /^## \[/ { - if (in_section) exit - if ($0 ~ "\\[" v "\\]") { in_section=1; next } - } - in_section { print } - ' CHANGELOG.md > dist/RELEASE_NOTES.md || true - # A RELEASE IS OUTWARD-FACING AND PERMANENT: publishing one whose - # notes silently came out empty is not a lesser failure than any - # other defect this workflow guards against, and until now it - # produced none — v2026.9.18.1 went out with its notes body reading - # literally "(no CHANGELOG entry found for 2026.9.18.1)", because - # CHANGELOG.md had no matching `## []` heading and this - # step wrote a placeholder and moved on rather than stopping. Stop - # instead: no matching section is an authoring mistake (the - # section is missing, or its version string does not match what is - # being tagged), and the fix belongs in CHANGELOG.md before the - # release is retried, not in a release whose notes nobody reads. - if [ ! -s dist/RELEASE_NOTES.md ]; then - echo "::error::CHANGELOG.md has no '## [$VERSION]' section — refusing to publish a release with empty notes. Add that section (docs/92-release.md) and re-run, rather than let this placeholder ship: (no CHANGELOG entry found for $VERSION)" - exit 1 + if git rev-parse --verify "refs/tags/$TAG" >/dev/null 2>&1; then + REF=$(git rev-list -n 1 "refs/tags/$TAG") + else + REF="$GITHUB_SHA" fi - echo "--- RELEASE_NOTES.md ---" - cat dist/RELEASE_NOTES.md - - - name: Keep the Linux archives for the release - uses: actions/upload-artifact@v4 - with: - name: release-linux-x86_64 - if-no-files-found: error - retention-days: 7 - path: | - dist/mcpp-${{ steps.stage.outputs.version }}-linux-x86_64.tar.gz - dist/mcpp-${{ steps.stage.outputs.version }}-linux-x86_64.tar.gz.sha256 - dist/mcpp-linux-x86_64.tar.gz - dist/mcpp-linux-x86_64.tar.gz.sha256 - dist/install.sh - dist/mcpp-${{ steps.stage.outputs.version }}.tar.gz - dist/mcpp.lua - dist/RELEASE_NOTES.md - - build-linux-aarch64: - name: build (linux / aarch64, cross) - runs-on: ubuntu-24.04 - permissions: - contents: read - timeout-minutes: 70 - steps: - - uses: actions/checkout@v4 - - - name: Install system deps + qemu - run: | - # The runner image carries third-party apt lists (Google Chrome - # among them) that this job does not use, and a transient - # `Hash Sum mismatch` on one of them fails the whole update -- which - # killed two cross-build jobs in setup, before a single byte was - # compiled. Dropping the lists this job has no use for is what makes - # the step's failure mean something about this job. - # - # BY CONTENT, NOT BY FILENAME. The first attempt removed - # `google-chrome.list` and the update failed on the same URL: on - # ubuntu-24.04 the runner writes deb822 `.sources` files, so the - # name was a guess and the guess was wrong. - sudo grep -rlE 'dl[.]google[.]com|packages[.]microsoft[.]com' \ - /etc/apt/sources.list.d/ 2>/dev/null | xargs -r sudo rm -f - sudo apt-get update -qq - sudo apt-get install -y curl git build-essential qemu-user-static - qemu-aarch64-static --version | head -1 - - - name: Resolve tag + version - id: resolve - run: | - VERSION=$(grep -E '^version' mcpp.toml | head -1 | sed 's/.*"\([^"]*\)".*/\1/') - echo "version=$VERSION" >> "$GITHUB_OUTPUT" - echo "tag=v$VERSION" >> "$GITHUB_OUTPUT" - - - name: Bootstrap mcpp via xlings - env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.11.17.2' - run: | - tarball="xlings-${XLINGS_VERSION}-linux-x86_64.tar.gz" - bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ - "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ - "/tmp/${tarball}" - tar -xzf "/tmp/${tarball}" -C /tmp - "/tmp/xlings-${XLINGS_VERSION}-linux-x86_64/subos/default/bin/xlings" self install - echo "$HOME/.xlings/subos/current/bin" >> "$GITHUB_PATH" - echo "$HOME/.xlings/bin" >> "$GITHUB_PATH" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - - name: Bootstrap mcpp + refresh index (latest, GLOBAL) - run: | - xlings config --mirror GLOBAL 2>/dev/null || true - xlings update -y 2>/dev/null || xlings update 2>/dev/null || true - MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - "$MCPP" --version - - - name: Cross-build mcpp -> aarch64-linux-musl (this release's source) - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - # "$MCPP", not a bare `mcpp`: the bare form runs the PATH shim, which - # resolves to whatever version xvm has selected — so pinning the - # bootstrap would have looked correct and changed nothing here. - "$MCPP" self config --mirror GLOBAL 2>/dev/null || true - # The published bootstrap mcpp predates aarch64 cross-build support - # (the feature landed after the last release), so it resolves the - # x86_64 host musl-gcc for an aarch64 target. Two-stage instead: build - # THIS release's x86_64 mcpp first, then cross-build aarch64 with it. - FRESH=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp --target x86_64-linux-musl) - test -n "$FRESH" - BIN=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$FRESH" mcpp --target aarch64-linux-musl) - test -n "$BIN" - file "$BIN" | grep -q 'ARM aarch64' - file "$BIN" | grep -q 'statically linked' - echo "MCPP_AARCH64=$BIN" >> "$GITHUB_ENV" - - - name: Package aarch64 release (+ bundle aarch64 xlings) - id: stage - run: | - VERSION="${{ steps.resolve.outputs.version }}" - TARBALL_NAME="mcpp-${VERSION}-linux-aarch64.tar.gz" - WRAPPER="mcpp-${VERSION}-linux-aarch64" - STAGING=$(mktemp -d) - mkdir -p "$STAGING/$WRAPPER/bin" - cp "$MCPP_AARCH64" "$STAGING/$WRAPPER/bin/mcpp" - # The binary is aarch64, so the host x86_64 `strip` cannot touch it — - # resolve a cross-capable one. Required, not best-effort: the old - # `|| true` here meant a missing tool silently shipped a fat tarball. - # (slim_linux_payload.sh runs after xlings is staged, below.) - STRIP=$(find "$HOME/.mcpp" -name 'aarch64-linux-musl-strip' -type f 2>/dev/null | head -1) - [ -n "$STRIP" ] || STRIP=$(command -v llvm-strip 2>/dev/null || true) - [ -n "$STRIP" ] || STRIP=$(command -v aarch64-linux-gnu-strip 2>/dev/null || true) - [ -n "$STRIP" ] || { echo "no aarch64-capable strip found"; exit 1; } - echo "aarch64 strip: $STRIP" - cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true - cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true - cat > "$STAGING/$WRAPPER/mcpp" << 'LAUNCHER' - #!/bin/sh - exec "$(dirname "$0")/bin/mcpp" "$@" - LAUNCHER - chmod +x "$STAGING/$WRAPPER/mcpp" - # Bundle the aarch64 xlings so install.sh consumers on aarch64 get an - # aarch64 xlings, not the x86_64 bootstrap one. The three literals - # below are pinned to the same version as XLINGS_VERSION; they are - # NOT interpolated from it, so check_version_pins.sh scans for them - # explicitly (they were absent from the old lock-step comment). - XLA="xlings-2026.11.17.2-linux-aarch64.tar.gz" - # NOT fetch_release.sh: this asset is OPTIONAL and the `if` is the - # point — an arch with no prebuilt xlings must fall through quietly, - # while the helper retries a 404 five times before giving up. The one - # flag that matters here is --retry-all-errors: `curl: (52) Empty - # reply from server` is a transport error, so plain --retry does not - # cover it. - if curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ - --connect-timeout 20 --max-time 600 -o "/tmp/$XLA" \ - "https://github.com/openxlings/xlings/releases/download/v2026.11.17.2/$XLA"; then - tar -xzf "/tmp/$XLA" -C /tmp - XLBIN=$(find /tmp/xlings-2026.11.17.2-linux-aarch64 -path '*/bin/xlings' -type f | head -1) - if [ -n "$XLBIN" ]; then - mkdir -p "$STAGING/$WRAPPER/registry/bin" - cp "$XLBIN" "$STAGING/$WRAPPER/registry/bin/xlings" - chmod +x "$STAGING/$WRAPPER/registry/bin/xlings" + TREE=$(git rev-parse "$REF^{tree}") + RUN=$(gh api "repos/$REPO/actions/artifacts?name=verified-$TREE&per_page=5" \ + --jq '[.artifacts[] | select(.expired | not)][0].workflow_run.id // empty' 2>/dev/null || true) + WHY="" + if [ -n "$RUN" ]; then + kept=$(gh api "repos/$REPO/actions/runs/$RUN/artifacts?per_page=100" \ + --jq '[.artifacts[] | select((.expired | not) and (.name | startswith("release-")))] | length') + if [ "$kept" != 4 ]; then + WHY="run $RUN verified the tree, and keeps $kept of its four archives" + RUN="" fi + else + WHY="no run verified the tree" fi - # Slim both shipped ELFs with the cross strip resolved above, and - # assert the result (the vendored xlings was 86.9MB unstripped here). - bash .github/tools/slim_linux_payload.sh "$STAGING/$WRAPPER" "$STRIP" - mkdir -p dist - (cd "$STAGING" && tar -czf "$GITHUB_WORKSPACE/dist/${TARBALL_NAME}" "$WRAPPER") - cp "dist/${TARBALL_NAME}" "dist/mcpp-linux-aarch64.tar.gz" - (cd dist && sha256sum "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") - (cd dist && sha256sum "mcpp-linux-aarch64.tar.gz" > "mcpp-linux-aarch64.tar.gz.sha256") - echo "tarball=${TARBALL_NAME}" >> "$GITHUB_OUTPUT" - ls -la dist/ - - - name: Smoke-test the aarch64 tarball (qemu) - run: | - TARBALL_NAME="${{ steps.stage.outputs.tarball }}" - WRAPPER="${TARBALL_NAME%.tar.gz}" - SMOKE=$(mktemp -d) - tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" - ver=$(qemu-aarch64-static "$SMOKE/$WRAPPER/bin/mcpp" --version) - echo "$ver"; echo "$ver" | grep -q 'mcpp' - - - name: Keep the aarch64 archives for the release - uses: actions/upload-artifact@v4 - with: - name: release-linux-aarch64 - if-no-files-found: error - retention-days: 7 - path: | - dist/mcpp-${{ steps.resolve.outputs.version }}-linux-aarch64.tar.gz - dist/mcpp-${{ steps.resolve.outputs.version }}-linux-aarch64.tar.gz.sha256 - dist/mcpp-linux-aarch64.tar.gz - dist/mcpp-linux-aarch64.tar.gz.sha256 + { + echo "tag=$TAG"; echo "version=${TAG#v}"; echo "ref=$REF"; echo "run=$RUN" + } >> "$GITHUB_OUTPUT" + { + echo "## $TAG" + echo "Commit \`$REF\`, tree \`$TREE\`." + if [ -n "$RUN" ]; then + echo "Promoted: the archives of run [$RUN](https://github.com/$REPO/actions/runs/$RUN), which verified the tree." + else + echo "Built: $WHY." + fi + } >> "$GITHUB_STEP_SUMMARY" + echo "$TAG at $REF (tree $TREE): ${RUN:+promoted from run $RUN}${WHY:+built, $WHY}" - build-macos: - name: build (macOS / ARM64) - runs-on: macos-15 + build: + needs: find + if: ${{ needs.find.outputs.run == '' }} permissions: contents: read - timeout-minutes: 30 - steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Resolve tag - id: resolve - run: | - if [ "${{ github.event_name }}" = "push" ]; then - TAG="${{ github.ref_name }}" - elif [ -n "${{ github.event.inputs.tag }}" ]; then - TAG="${{ github.event.inputs.tag }}" - else - VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) - TAG="v$VER" - fi - echo "tag=$TAG" >> "$GITHUB_OUTPUT" - echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" - if [ "${{ github.event_name }}" = "workflow_dispatch" ] \ - && git rev-parse --verify "refs/tags/$TAG" >/dev/null 2>&1; then - git checkout --detach "refs/tags/$TAG" - fi - - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~/.xlings - key: xlings-macos15-release-xl0462-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-macos15-release-xl0462- - - - name: Bootstrap mcpp via xlings - env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.11.17.2' - run: | - # By version, not by presence (see the Linux job). - have=$("$HOME/.xlings/subos/default/bin/xlings" --version 2>/dev/null | awk '{print $2}') - if [ "$have" != "$XLINGS_VERSION" ]; then - WORK=$(mktemp -d) - tarball="xlings-${XLINGS_VERSION}-macosx-arm64.tar.gz" - bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ - "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${tarball}" \ - "${WORK}/${tarball}" - tar -xzf "${WORK}/${tarball}" -C "${WORK}" - "${WORK}/xlings-${XLINGS_VERSION}-macosx-arm64/subos/default/bin/xlings" self install - fi - export PATH="$HOME/.xlings/subos/default/bin:$PATH" - xlings --version - test "$(xlings --version | awk '{print $2}')" = "$XLINGS_VERSION" || { - echo "::error::the xlings this release bundles is $(xlings --version), not $XLINGS_VERSION"; exit 1; } - # Pinned to .xlings.json — a bare `xlings install mcpp` resolves - # "newest in this runner's index copy" and put 0.0.105 (below the - # index floor) into this job. See .github/tools/install_pinned_mcpp.sh. - MCPP=$(bash "$GITHUB_WORKSPACE/.github/tools/install_pinned_mcpp.sh" "$GITHUB_WORKSPACE") - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - echo "XLINGS_BIN=$HOME/.xlings/subos/default/bin/xlings" >> "$GITHUB_ENV" - - - name: Build mcpp from source (two-stage self-host) - env: - # macOS min-version support: target macOS 14 so the release runs - # on 14.0+ instead of only the runner's OS (the official LLVM - # static libc++ archives are built for macOS 14 — going lower - # needs a custom libc++ build, tracked as follow-up). Needs - # static LLVM libc++ — the system libc++ on older macOS lacks - # LLVM-20-era C++23 symbols (std::print's __is_posix_terminal - # etc.; minos-14 + dynamic libc++ dies at launch on macos-14 CI). - # See xlings .agents/docs/2026-06-05-macos-min-version-support.md. - MACOSX_DEPLOYMENT_TARGET: '14.0' - run: | - export PATH="$HOME/.xlings/subos/default/bin:$PATH" - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - - # Stage 1: the bootstrap mcpp builds this release's source. The - # bootstrap's macOS link path predates the staticStdlib - # implementation (hardcoded -lc++), so stage 1 links the system - # libc++ — fine, it only needs to RUN on this runner. - STAGE1=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$MCPP" mcpp) - # --no-cache removes target/, including the stage-1 compiler. Ninja - # invokes that compiler again for dependency scanning, so keep the - # bootstrap outside the tree the second stage is about to clear. - STAGE1_COPY=$(mktemp -d "$RUNNER_TEMP/mcpp-stage1.XXXXXX")/mcpp - cp "$STAGE1" "$STAGE1_COPY" - STAGE1="$STAGE1_COPY" - "$STAGE1" --version - - # Stage 2: this release's mcpp rebuilds itself — flags.cppm's - # native staticStdlib link produces the static minos-14 binary. - # Stage 2 lands in a NEW fingerprint directory (its fingerprint - # includes the deployment target; the bootstrap's did not): the - # build names its own product. - MCPP_BIN=$(bash "$GITHUB_WORKSPACE/.github/tools/built_artifact.sh" "$STAGE1" mcpp --no-cache) - test -x "$MCPP_BIN" - file "$MCPP_BIN" - otool -L "$MCPP_BIN" - echo "=== LC_BUILD_VERSION (must be minos 14.0) ===" - otool -l "$MCPP_BIN" | grep -A4 LC_BUILD_VERSION | head -6 - otool -l "$MCPP_BIN" | grep -A4 LC_BUILD_VERSION | grep -q "minos 14.0" \ - || { echo "FAIL: expected minos 14.0"; exit 1; } - if otool -L "$MCPP_BIN" | grep -q "libc++"; then - echo "FAIL: still linked against system libc++"; exit 1 - fi - "$MCPP_BIN" --version - echo "MCPP_BIN=$MCPP_BIN" >> "$GITHUB_ENV" - - - name: Package macOS release - id: stage - run: | - VERSION="${{ steps.resolve.outputs.version }}" - TARBALL_NAME="mcpp-${VERSION}-macosx-arm64.tar.gz" - WRAPPER="mcpp-${VERSION}-macosx-arm64" - - # Create release layout - STAGING=$(mktemp -d) - mkdir -p "$STAGING/$WRAPPER/bin" - cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp" - # Strip (Mach-O) - strip "$STAGING/$WRAPPER/bin/mcpp" 2>/dev/null || true - # Copy metadata - cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true - cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true - - # Shell launcher (same as Linux) - cat > "$STAGING/$WRAPPER/mcpp" << 'LAUNCHER' - #!/bin/sh - exec "$(dirname "$0")/bin/mcpp" "$@" - LAUNCHER - chmod +x "$STAGING/$WRAPPER/mcpp" - - # Bundle xlings for install.sh consumers - XLINGS_BIN="$HOME/.xlings/subos/default/bin/xlings" - if [ -x "$XLINGS_BIN" ]; then - mkdir -p "$STAGING/$WRAPPER/registry/bin" - cp "$XLINGS_BIN" "$STAGING/$WRAPPER/registry/bin/xlings" - chmod +x "$STAGING/$WRAPPER/registry/bin/xlings" - fi - - # Create tarball - mkdir -p dist - (cd "$STAGING" && tar -czf "$GITHUB_WORKSPACE/dist/${TARBALL_NAME}" "$WRAPPER") - # Versionless alias - cp "dist/${TARBALL_NAME}" "dist/mcpp-macosx-arm64.tar.gz" - # SHA256 - (cd dist && shasum -a 256 "${TARBALL_NAME}" > "${TARBALL_NAME}.sha256") - (cd dist && shasum -a 256 "mcpp-macosx-arm64.tar.gz" > "mcpp-macosx-arm64.tar.gz.sha256") - - echo "tarball=${TARBALL_NAME}" >> "$GITHUB_OUTPUT" - ls -la dist/ - - - name: Smoke-test the tarball - run: | - VERSION="${{ steps.resolve.outputs.version }}" - TARBALL_NAME="${{ steps.stage.outputs.tarball }}" - WRAPPER="${TARBALL_NAME%.tar.gz}" - SMOKE=$(mktemp -d) - tar -xzf "dist/${TARBALL_NAME}" -C "$SMOKE" - "$SMOKE/$WRAPPER/bin/mcpp" --version - "$SMOKE/$WRAPPER/mcpp" --version | grep -q "$VERSION" - # R4 (mcpp#832): stage 2 is this release's own engine, and it trims - # the paths the binary carries. - bash .github/tools/check_release_paths.sh "$SMOKE/$WRAPPER/bin/mcpp" - - - name: Keep the macOS archives for the release - uses: actions/upload-artifact@v4 - with: - name: release-macosx-arm64 - if-no-files-found: error - retention-days: 7 - path: | - dist/mcpp-${{ steps.resolve.outputs.version }}-macosx-arm64.tar.gz - dist/mcpp-${{ steps.resolve.outputs.version }}-macosx-arm64.tar.gz.sha256 - dist/mcpp-macosx-arm64.tar.gz - dist/mcpp-macosx-arm64.tar.gz.sha256 - - build-windows: - name: build (Windows / x86_64) - runs-on: windows-latest + uses: ./.github/workflows/release-build.yml + with: + tag: ${{ needs.find.outputs.tag }} + ref: ${{ needs.find.outputs.ref }} + + # The verified run's four archives, checked against their checksums and their version, kept as + # this run's for `create-release`. + promote: + name: promote the verified run's archives + needs: find + if: ${{ needs.find.outputs.run != '' }} + runs-on: ubuntu-24.04 permissions: contents: read - timeout-minutes: 45 - env: - MCPP_HOME: C:\Users\runneradmin\.mcpp + actions: read + timeout-minutes: 15 steps: - - uses: actions/checkout@v4 - with: - fetch-depth: 0 - - - name: Resolve tag - id: resolve - shell: bash - run: | - if [ "${{ github.event_name }}" = "push" ]; then - TAG="${{ github.ref_name }}" - elif [ -n "${{ github.event.inputs.tag }}" ]; then - TAG="${{ github.event.inputs.tag }}" - else - VER=$(awk -F '"' '/^version[[:space:]]*=/{print $2; exit}' mcpp.toml) - TAG="v$VER" - fi - echo "tag=$TAG" >> "$GITHUB_OUTPUT" - echo "version=${TAG#v}" >> "$GITHUB_OUTPUT" - if [ "${{ github.event_name }}" = "workflow_dispatch" ] \ - && git rev-parse --verify "refs/tags/$TAG" >/dev/null 2>&1; then - git checkout --detach "refs/tags/$TAG" - fi - - - name: Cache mcpp sandbox - uses: actions/cache@v4 - with: - path: ~\.mcpp - key: mcpp-sandbox-${{ runner.os }}-release-${{ hashFiles('mcpp.toml', '.xlings.json') }} - restore-keys: | - mcpp-sandbox-${{ runner.os }}-release- - - - name: Cache xlings - uses: actions/cache@v4 - with: - path: ~\.xlings - key: xlings-${{ runner.os }}-release-xl0462-${{ hashFiles('.xlings.json') }} - restore-keys: | - xlings-${{ runner.os }}-release-xl0462- - - - name: Bootstrap mcpp via xlings - shell: bash + - name: The archives, their checksums and their version env: - XLINGS_NON_INTERACTIVE: '1' - XLINGS_VERSION: '2026.11.17.2' - run: | - # Captured before the `cd` below, in POSIX form: this step never - # returns to the workspace, and GITHUB_WORKSPACE is a backslash - # Windows path that git-bash tools mangle. - REPO_DIR="$(pwd)" - WORK=$(mktemp -d) - zipfile="xlings-${XLINGS_VERSION}-windows-x86_64.zip" - bash "$GITHUB_WORKSPACE/.github/tools/fetch_release.sh" \ - "https://github.com/openxlings/xlings/releases/download/v${XLINGS_VERSION}/${zipfile}" \ - "${WORK}/${zipfile}" - cd "${WORK}" - unzip -q "${zipfile}" - "$WORK/xlings-${XLINGS_VERSION}-windows-x86_64/subos/default/bin/xlings.exe" self install - export PATH="$USERPROFILE/.xlings/subos/default/bin:$PATH" - echo "$USERPROFILE/.xlings/subos/default/bin" >> "$GITHUB_PATH" - xlings.exe --version - test "$(xlings.exe --version | awk '{print $2}' | tr -d '\r')" = "$XLINGS_VERSION" || { - echo "::error::the xlings this release bundles is not $XLINGS_VERSION"; exit 1; } - # Pinned + version-scoped lookup. The old `find | head -1` returned - # whichever version the directory walk reached first. - MCPP=$(bash "$REPO_DIR/.github/tools/install_pinned_mcpp.sh" "$REPO_DIR") - echo "MCPP=$MCPP" >> "$GITHUB_ENV" - XLINGS_BIN=$(cygpath -w "$USERPROFILE/.xlings/subos/default/bin/xlings.exe") - echo "XLINGS_BIN=$XLINGS_BIN" >> "$GITHUB_ENV" - echo "XLINGS_BIN_UNIX=$USERPROFILE/.xlings/subos/default/bin/xlings.exe" >> "$GITHUB_ENV" - echo "XLINGS_XPKGS=$USERPROFILE/.xlings/data/xpkgs" >> "$GITHUB_ENV" - - - name: Build mcpp from source (self-host) - shell: bash - run: | - export MCPP_VENDORED_XLINGS="$XLINGS_BIN" - - "$MCPP" build - # Pick the NEWEST mcpp.exe, not an arbitrary one: `target/` is - # restored from cache and keeps a directory per build fingerprint, - # so after a version bump the freshly built binary sits alongside - # the previous release's. `find | head -1` returned whichever the - # directory walk hit first — which is how a 0.0.106 build ran the - # 0.0.105 binary and failed 01_help_and_version. - MCPP_BIN=$(find target -name "mcpp.exe" -path "*/bin/*" -printf "%T@ %p\n" \ - | sort -rn | head -1 | cut -d" " -f2-) - test -n "$MCPP_BIN" || { echo "FAIL: no mcpp.exe in target/"; exit 1; } - MCPP_BIN=$(cd "$(dirname "$MCPP_BIN")" && pwd)/$(basename "$MCPP_BIN") - echo "Self-hosted binary: $MCPP_BIN" - "$MCPP_BIN" --version - echo "MCPP_BIN=$MCPP_BIN" >> "$GITHUB_ENV" - - - name: Package Windows release zip - id: stage - shell: bash - run: | - VERSION="${{ steps.resolve.outputs.version }}" - WRAPPER="mcpp-${VERSION}-windows-x86_64" - ZIPNAME="${WRAPPER}.zip" - - STAGING=$(mktemp -d) - mkdir -p "$STAGING/$WRAPPER/bin" "$STAGING/$WRAPPER/registry/bin" - cp "$MCPP_BIN" "$STAGING/$WRAPPER/bin/mcpp.exe" - - # Windows batch launcher - printf '@echo off\r\n"%%~dp0bin\\mcpp.exe" %%*\r\n' > "$STAGING/$WRAPPER/mcpp.bat" - cp README.md "$STAGING/$WRAPPER/" 2>/dev/null || true - cp LICENSE "$STAGING/$WRAPPER/" 2>/dev/null || true - - # Bundle xlings.exe for install consumers - if [ -f "$XLINGS_BIN_UNIX" ]; then - cp "$XLINGS_BIN_UNIX" "$STAGING/$WRAPPER/registry/bin/xlings.exe" - fi - - # Pack with 7z (available on windows-latest) - mkdir -p dist - (cd "$STAGING" && 7z a -tzip "$ZIPNAME" "$WRAPPER") - cp "$STAGING/$ZIPNAME" "dist/$ZIPNAME" - # Versionless alias - cp "dist/$ZIPNAME" "dist/mcpp-windows-x86_64.zip" - # SHA256 - (cd dist && sha256sum "$ZIPNAME" > "$ZIPNAME.sha256") - (cd dist && sha256sum "mcpp-windows-x86_64.zip" > "mcpp-windows-x86_64.zip.sha256") - - echo "zipname=$ZIPNAME" >> "$GITHUB_OUTPUT" - ls -la dist/ - - - name: Smoke-test the packaged zip - shell: bash + GH_TOKEN: ${{ github.token }} + RUN: ${{ needs.find.outputs.run }} + VERSION: ${{ needs.find.outputs.version }} run: | - ZIPNAME="${{ steps.stage.outputs.zipname }}" - WRAPPER="${ZIPNAME%.zip}" - SMOKE=$(mktemp -d) - (cd "$SMOKE" && unzip -q "$GITHUB_WORKSPACE/dist/$ZIPNAME") - "$SMOKE/$WRAPPER/bin/mcpp.exe" --version - "$SMOKE/$WRAPPER/bin/mcpp.exe" --help | head -5 - test -f "$SMOKE/$WRAPPER/registry/bin/xlings.exe" - test -f "$SMOKE/$WRAPPER/mcpp.bat" - echo "Smoke-test passed" + set -euo pipefail + gh run download "$RUN" -R "$GITHUB_REPOSITORY" -p 'release-*' -D promoted + mkdir dist + for dir in promoted/release-*/; do cp "$dir"* dist/; done + cd dist + for platform in linux-x86_64 linux-aarch64 macosx-arm64 windows-x86_64; do + ls "mcpp-$VERSION-$platform".tar.gz "mcpp-$VERSION-$platform".zip 2>/dev/null | grep -q . || { + echo "::error::run $RUN holds no archive of $VERSION for $platform"; ls -la; exit 1; } + done + for sum in *.sha256; do sha256sum -c "$sum"; done + test -s RELEASE_NOTES.md && test -f install.sh && test -f mcpp.lua && test -f "mcpp-$VERSION.tar.gz" + root=$(mktemp -d) + tar -xzf "mcpp-$VERSION-linux-x86_64.tar.gz" -C "$root" + ver=$(env -u MCPP_HOME "$root/mcpp-$VERSION-linux-x86_64/bin/mcpp" --version) + echo "$ver" + grep -qF "$VERSION" <<< "$ver" || { echo "::error::the promoted program reports $ver, not $VERSION"; exit 1; } + ls -la - - name: Keep the Windows archives for the release + - name: Keep the archives for the release uses: actions/upload-artifact@v4 with: - name: release-windows-x86_64 + name: release-promoted if-no-files-found: error retention-days: 7 - path: | - dist/mcpp-${{ steps.resolve.outputs.version }}-windows-x86_64.zip - dist/mcpp-${{ steps.resolve.outputs.version }}-windows-x86_64.zip.sha256 - dist/mcpp-windows-x86_64.zip - dist/mcpp-windows-x86_64.zip.sha256 + path: dist/ # Seal the complete, non-draft release inventory only after every platform # uploader has finished. The manifest is desired state for downstream @@ -815,14 +188,14 @@ jobs: # release, the four builds and the source. create-release: name: create the release - needs: [canaries, build-release, build-linux-aarch64, build-macos, build-windows] + needs: [canaries, find, build, promote] if: ${{ !cancelled() && !failure() }} runs-on: ubuntu-24.04 permissions: contents: write # required to create releases + push tags timeout-minutes: 15 steps: - - name: The archives of the four builds + - name: The archives of the four builds, built or promoted uses: actions/download-artifact@v4 with: pattern: release-* @@ -840,9 +213,9 @@ jobs: - name: Create GitHub Release uses: softprops/action-gh-release@v2 with: - tag_name: ${{ needs.build-release.outputs.tag }} - name: ${{ needs.build-release.outputs.tag }} - target_commitish: ${{ github.sha }} + tag_name: ${{ needs.find.outputs.tag }} + name: ${{ needs.find.outputs.tag }} + target_commitish: ${{ needs.find.outputs.ref }} body_path: dist/RELEASE_NOTES.md draft: false prerelease: false diff --git a/CHANGELOG.md b/CHANGELOG.md index d7282ec32..fae16a7e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,235 @@ > Each `## []` section is that release's notes. Entries are written in English > from 2026.9.28.3 on; earlier entries remain as written. +## [2026.11.21.1] - 2026-11-21 + +### Added + +- **The home's unit cache** (mcpp#927, docs/26). Before it starts a compiler, the compiler + executor looks up the compile by its command, its compiler and its source's content; when the + files an earlier compile of that key read still hold their contents, its object and BMI are + placed and no compiler starts. A BMI is named by the key of the compile that made it. A fresh + checkout of mcpp on a filled cache built in 4.1 s instead of 164 s, its 288 compiles placed. + The split schedule places its units from the cache too. cl.exe's compiles are not cached. A + Clang compile is given `-fmodule-file-deps`, so its depfile names the BMIs it imports beside + its headers, and ninja and the cache read them from it. +- **A cacheable package's build program runs in the home** (mcpp#986, mcpp#927, SPEC-004 §9 + item 18). A package whose source is named -- an index's version, a git commit, a path + dependency's content digest -- runs its build program into + `build-cache/v1/out/@//out`, keyed by its configuration, and a second + project takes the outputs; a program run by `mcpp pack --pkg-format` runs in the project. A + path enters a cache key by its role (``, ``, ``), never as written. +- **`[profile.] dependency_debug`** (mcpp#927, SPEC-004 §9 item 20): the debug information + of the dependencies from the package store and git, `true`, `false` or `"lines"`. Omitted, + `dev` and `debug` take `"lines"` while their `debug` is on. A test program whose only + dependency is mtest fell from 43.7 MB to 8.3 MB and its build from 15.9 s to 7.2 s. +- **`[build] memory_threshold`** (mcpp#802, SPEC-004 §9 item 19): the available memory below + which the compiler executor waits for a running compiler to end before it starts another; the + engine's value is 1 GiB, `MCPP_MEMORY_THRESHOLD` overrides it, and after a minute below it the + compiler starts and standard error says so. +- **One budget of slots for the machine** (mcpp#976, SPEC-017 O11). `mcpp test` states a + directory of one slot file per job (`MTEST_BUDGET`); a compiler a test starts through mcpp + holds a slot of it, so a test that builds a project takes from the run's jobs. A run inside + another runner's budget uses that one. +- **The record of a build and of a run** (mcpp#927, SPEC-017 B4, P5.6, P12.4). + `MCPP_BUILD_RECORD=` appends a line per edge a build ran: its package, its time, the + compiler's processor time and largest resident set, and whether the unit cache placed it. + `mcpp test` keeps a run's builds in `builds.ndjson`; a run names its machine's class, its + processors and memory, and a case its processor time, its steps, the surface items it used and + the limit it ran within. Estimates prefer the runs of the same class of machine. An edge that + runs a minute is named on a `Running` line; `--verbose` names the five slowest. +- **Run fixtures** (mcpp#293, SPEC-017 O6, P11). A program declares a fixture the run shares and + its cases name it in `uses`; `mcpp test` starts it once, with `MTEST_SERVE`, before the first + program that uses it, gives its endpoint as `MTEST_FIXTURE_`, and ends it after the + last. A fixture that does not become ready is the machine's failure: its users record `infra`. + `fixtures.ndjson` states each fixture's start, readiness and end. +- **A header selects the tests that read it.** `mcpp test --changed`/`--since` takes the units + that read a changed header from the build directory's records (ninja's deps log, the scans' + depfiles) and runs the tests built from them; while a unit of the project has no record, a + header affects every test. +- **`mcpp emit surface`** writes mcpp's surface, one ` ` per line: the commands and + flags of the command line's definition, the manifest's keys and the refusals' reason tokens. + `MCPP_SURFACE_RECORD=` makes each process of an invocation append the items it used. +- `wire` features `test-steps`, `test-budget`, `test-jitter` and `test-fixtures` (docs/50). +- `mcpp-community/mcpp/.github/actions/project-test`, a downstream repository's test job: the mcpp + the project pins, its home restored by the project's identity, `mcpp test`, the run's store as + an artifact and its slowest cases and edges in the summary (docs/08). + +- **A project is told a notice once** (design 2026-11-21, build programs' inputs §3.6). Every + warning, note and closing tip goes through one sink; a notice whose kind is registered `once` + (mcpp.notice) is said once per project and subject, recorded in `target/.mcpp-notices`, and said + again after `mcpp clean`. `--verbose` shows a suppressed one marked `(noted before)`; `--quiet` + hides notes. The build database's diagnostics carry every notice, with `shown: false` for one the + terminal did not show (docs/50); a command that plans in a directory of its own (`mcpp emit + build-database`, doctor, publish's plan) reads the record and does not write it. + `.github/tools/check_diag_sink.sh` keeps notices in the sink. +- **A build program's headers are told once**: a program of the project that includes a standard, + system or package header receives one note per form, with the form that imports it + (`MCPP_BUILD_PROGRAM_INCLUDES_HEADER`, docs/30 "Headers or modules"); a file its compile reads + outside its package, its build dependencies and the store is warned once per file + (`MCPP_BUILD_PROGRAM_READS_OUTSIDE`), and `mcpp publish` refuses one its archive does not carry + (`build-program-unpublished-input`). +- Directive protocol 21: `mcpp:deprecated=`, stated by a typed reader that answers by a deprecated + reading. +- **`[build] allow_host_tools`** (mcpp#990, SPEC-004 §9 item 21, docs/25): the bootstrap tools a + build takes from `PATH` instead of the store, `ninja` and `patchelf`, from + `MCPP_ALLOW_HOST_TOOLS`, the project, then `config.toml`. Every host input -- a bootstrap tool, + a toolchain named by path, `glibc@system`, a tool found on `PATH` -- is a note once per project + and identity (`host-input`). +- **`mcpp fetch`** (mcpp#990, docs/05 "Offline builds and mcpp fetch"): the step of a build that + obtains what it needs, as a command; it plans as `--configure-only` does and compiles nothing. + `mcpp fetch --to ` writes the payloads the build used with their closures, the index + snapshots and a `git bundle` of each git clone into a directory described by `bundle.toml`; + `mcpp fetch --from ` takes such a directory into a home, a fresh one included, after + which the directory may be removed. A bundle that lacks a version `mcpp.lock` records is + refused (`bundle-older-than-lock`); one written on another host or for none of the selected + targets is warned (`bundle-lacks-target`). +- **`[build] offline`** in `mcpp.toml` and `config.toml` (SPEC-004 §9 item 22), below `--offline` + and `MCPP_OFFLINE` (`0` turns it off for a run). A refusal of an offline run names the tier + that made it offline. Workspace members that state different values of it, or of + `[build] allow_host_tools`, are planned apart, each plan with its root's value. +- **`[xlings] archives`** in `config.toml` and `mcpp.toml`, and `MCPP_ARCHIVES` (SPEC-004 §9 + item 23, docs/25 "Archive directories"): directories of archives taken by sha256 before the + network, and in its place offline, where a payload the home lacks is installed from the + download cache or a directory; one none holds is refused (`artifact-unavailable-offline`). + A directory outside the project named in `mcpp.toml` is a note once + (`machine-path-in-manifest`). +- **A package's features are machine-readable** (mcpp#992, SPEC-004 §16, docs/06 "Reading a + package's features"). `mcpp search --format json` states, for each package of an mcpp index, + the features of its newest version (`features`: `version`, `state`, `from`, `list`), + read from the index or the installed payload; a payload that is not installed is + `not-installed` with a `null` list, and the query installs nothing. `mcpp why features + [--format json]` (`mcpp.why.features`) lists each package of the resolved graph with the + features it declares and those the build activates, which `resolution.json` records + (`features`, `feature_list`). `--protocol-version` lists `package-features`. A table-form + feature, and a descriptor's `features.`, accept `description`. + +### Changed + +- **A package's instance is a function of its configuration, not of its units** (mcpp#986): a + test build after `mcpp build` no longer compiles the package again, and each refusal of the + cache names its reason. The cache's key carries the instance, so a dependency an earlier + release cached is compiled once more after the upgrade and served from then on. +- **A case's time is the one its scopes state**: a case selected with `--case` runs within the + `timeout` its listed case states, not the run's. +- **The defaults are the newest releases** (mcpp#988; docs/51): a first run installs gcc 16.2.0 on + Linux x86_64 and on Windows without MSVC (the MinGW row), musl gcc 16.2.0 on the other Linux + architectures, and the bootstrap tools ninja 1.13.2-mcpp.1 and patchelf 0.19.3; gcc 16.2.0 + revision 1 brings binutils 2.47 and linux-headers 6.18.64. llvm stays 23.1.3, since + llvm-project's 23.1.6 publishes no macOS binary, and the musl rows stay 16.1.0, the only version + of their cross payloads. A project that built with the earlier default is told once. + `check_defaults.sh` compares a default with the newest release on every platform it serves. +- mcpplibs.mtest 0.11.2 and mcpplibs.os 0.4.7 throughout, the new project's skeleton and the CI + engine included: a case run as a scope names no case in its events. +- xlings 2026.11.21.1 is the pinned provider: archives by content, `xlings download --to` and + `xlings install --from`. +- **CI verifies a tree once** (design 2026-11-20 §5.22). A pull request's push runs the `pr` tier; + a `ci:full` run that passes leaves `verified-`, a push to main of that tree registers + the run instead of testing again, and `release.yml` promotes the four archives that run built + and tested (`release-build.yml`) instead of building them. The release pull request passes + `ci:full` before it is merged, and main's run is not cancelled (docs/92). + +### Deprecated + +- `ambiguous-contract-name` (until 2027.2): a name whose variable several dependencies answer, + which a build program's typed reader answered by the first. It still answers so, and every build + whose program asks warns; `--strict` refuses it (docs/51). + +### Fixed + +- **A cached dependency is served only against the module interfaces it was compiled with** + (SPEC-001 §10.4 item 7). An entry records the digest of each upstream interface it imported; + when an upstream is compiled again (its entry removed by `mcpp cache gc`, then built at + another path), its dependents are compiled with it and their entries written again, instead of + being served and failing with `module '' CRC mismatch`. Entries an earlier mcpp wrote + for packages that import another package's module are filled again once. +- The store's nasm, installed by link from an upstream home, is no longer reported as the host's + assembler: the store is judged by path, not by where the link leads. +- **`mcpp test --failed` reads the newest run** when two runs started in the same second: the + later line of the store's index is the newer run. +- **What a scope's body does is the program's** (SPEC-017 P5.6, mtest 0.11.2): a `step`, + `exercise` or `metric` event that names no case is kept in the program's record (`steps`, + `exercised`, `metrics`) instead of making a case that never ended, which failed a Windows + program whose suite probed Visual Studio before its cases. +- **A test that holds the machine has the budget's slots** (SPEC-017 O11 item 5): a worker of + `mcpp test` that waits for the machine gives its slot back, so the build of a `machine` test + runs its compilers in parallel instead of on the one slot its worker held. +- **A compiler's version is what its driver states for machines** (mcpp#989): GCC's + `-dumpfullversion` (else `-dumpversion`) and clang's `-dumpversion`, not the last number of the + banner, which read Fedora's `g++ (GCC) 16.2.1 20261029 (Red Hat 16.2.1-3)` as `3`. A triple's + vendor field is any word, as GNU config.sub reads it (`x86_64-redhat-linux` is + `x86_64-linux-gnu`); a toolchain whose target mcpp cannot read is reported as a degradation + instead of applying none of the target's settings in silence. +- **A build no longer requires patchelf** (mcpp#990): a payload's install declares it and the + provider installs it then, and mcpp runs it only when it packs. A build with the machine's + toolchain and `glibc@system` was refused in a home without it. +- **A toolchain request has one answer** (mcpp#988, SPEC-006 §2.6 to §2.8, docs/20). A partial + version (`gcc@16`) is the newest release of the local index whose version it is a prefix of, in + `mcpp toolchain install`, `mcpp toolchain default` and a build alike; `toolchain default` took the + newest installed version and a build the one the payload manager kept active. A partial version + `mcpp.toml` states is recorded in `mcpp.lock` (`[toolchain."xim:gcc@16"]`) and read back until + `mcpp update`; under `--locked` a lock without the record is kept as it is and the build says so. + A project that states no toolchain is told once when the default it follows answers another + toolchain than the one it last built with. +- **A required `dlopen` library is judged from the object that loads it** (design 2026-11-21 + §3.10, docs/33, docs/91). The post-link validation and `mcpp self doctor` looked for it on the + union of the plan's search paths, the executable's `DT_RPATH`; an object with `DT_RUNPATH`, as + every payload xlings installs carries, searches no `DT_RPATH`, so xim:slint's `libslint_cpp.so` + could not load X11, Wayland or GL in any program mcpp built while both checks passed. Each + required soname is now looked up from the requiring package's own shared objects by glibc's + rule, through one function both checks call, and one that cannot be reached is reported + (`runtime/dlopen-unreachable`). The `DT_NEEDED` walk applies the same rule to the farm. +- **A toolchain's companions are the versions its payload declares** (mcpp#990, SPEC-006 §2.9). + binutils, the kernel headers and GCC's runtime were the first version a directory walk of the + store returned: with binutils 2.42 and 2.42.1 installed, GCC 16.1.0 built with 2.42, which its + recipe does not declare. The version is now the one the payload's descriptor declares, else the + newest, and a declared one the store lacks is reported. `default = { package = "gcc@…", + companions = { binutils = "xim:binutils@…" } }` states another binutils; an older one than the + declared is refused (`companion-too-old`). +- **The command line's `@` selects by the written prefix** (SPEC-001 §4.5): `--template + mcpplibs.mwrap.slint@1.19.0` selects the newest key 1.19.0.x (it refused, as no key is + `1.19.0`); `@=` selects one key, `@^1.19` is a requirement, and an exact miss names the + finer keys. +- **An edit of a header a build program includes compiles and runs the program again** (mcpp#991). + The program's identity was predicted at three places -- the fast path, the group record and the + home run cache -- and a header is no package's, so the old program ran with the old directives, + also after `target/.build-mcpp` was deleted. The files the program's compile read are observed, + from the sub-build's dependency records, recorded with the program and compared by size and time, + then content; a declared input (`rerun_if_changed`) is compared the same way. +- **A name in a build program's contract is judged by its variable** (mcpp#852): `foo-bar`, + `foo_bar` and `foo.bar` publish one variable, and a variable several dependencies, tools or + payloads answer belongs to none of them. `MCPP_DEP_AMBIGUOUS_` (`MCPP_XPKG_AMBIGUOUS_`) + lists the spellings that name one, and `dep_dir`, `dep_linkage`, `dep_bin` and `xpkg_dir` asked + by the name end the program naming them. The per-build `collides on` warnings are removed. +- `mcpp why --format json` reports a refusal whose branch has no token as + `other`, as `mcpp why toolchain` does; it reported `none` beside `refused`. Both take the reason + from one function of the refusal record. +- A message that names a toolchain to write or to install names this host's first run, read from + the built-in table: the refusal of `[toolchain] = "system"`, the examples of `mcpp toolchain`'s + argument errors, `mcpp toolchain list` and `mcpp self config` with nothing installed, and + `mcpp self doctor`'s hints for the optional cross toolchains, which name + `mcpp toolchain install --target `. They named gcc 16.1.0 as written in the code. +- A GCC named by path whose project states `tools = { as = ... }` (or `ld`) compiles and links with + that binutils, as its `std` module's compile already did (mcpp#755). The compile and link lines + took their `-B` from the archive tool's directory; such a GCC has no archive tool beside its + driver, so they had none, and the driver ran the host's assembler from PATH, which cannot read + gcc 16.2.0's `.base64` directive (binutils 2.42). Both take the directory from one function now. +- A compiler whose baked sysroot is the building home's `subos/current` links against this home's + C library, as one whose baked sysroot is `subos/default` does. gcc 16.2.0 revision 1 names + `subos/current`; the remap recognised `subos/default` alone, and the link of a program whose + environment no package describes found no C library. +- `mcpp index list` names the registries `mcpp index add` added. It read the provider's list of + registries in an older form, found none in the current one, and showed the configuration's + alone; the text form now marks the default as the JSON form does. +- A program's resource script is compiled with its package's `include_dirs` expanded, as the + compile steps expand them: a relative entry is a glob, and `*/include` names the directories it + matches. It reached `llvm-windres` as written, and LLVM's tools on Windows expand wildcards in + their own command line against files, so the preprocessor saw several inputs and refused + ("cannot specify -o when generating multiple output files"); on other hosts the literal `*` + found nothing. Every program of a package whose `include_dirs` name an archive's wrap layer + failed so on Windows, among them compat.protobuf's `protoc` host tool, which gRPC's code + generation builds. + ## [2026.11.20.1] - 2026-11-20 ### Added diff --git a/bench/mcpp.toml b/bench/mcpp.toml index 20021a9bb..35b850155 100644 --- a/bench/mcpp.toml +++ b/bench/mcpp.toml @@ -21,4 +21,4 @@ windows = "llvm@23.1.3" # small fixture. `mcpp build` ignores [dev-dependencies], so the harness itself # still takes nothing beyond `import std;`. [dev-dependencies] -mcpplibs.mtest = "0.10.0" +mcpplibs.mtest = "0.11.2" diff --git a/docs/01-getting-started.md b/docs/01-getting-started.md index ce92416d4..66165d2a2 100644 --- a/docs/01-getting-started.md +++ b/docs/01-getting-started.md @@ -41,15 +41,17 @@ the host: | host | default | |---|---| -| Linux x86_64 | `gcc@16.1.0` | +| Linux x86_64 | `gcc@16.2.0` | | Linux aarch64 | `llvm@23.1.3` | -| other Linux architectures | `gcc@15.1.0-musl` | +| other Linux architectures | `gcc@16.2.0-musl` | | macOS | `llvm@23.1.3` | | Windows with usable MSVC | `llvm@23.1.3` | -| Windows without it | `gcc@16.1.0` for `x86_64-windows-gnu` | +| Windows without it | `gcc@16.2.0` for `x86_64-windows-gnu` | -The Linux aarch64 default applies from 2026.10.8.1. Existing configured -toolchains and targets remain in effect. +The Linux aarch64 default applies from 2026.10.8.1, and gcc 16.2.0 from +2026.11.21.1. Existing configured toolchains and targets remain in effect; a +project that last built with the earlier default is told once (20 — Toolchains, +"Resolution and the lock"). Full installation instructions, including Windows, are in the ["Installation" section of the README](../README.md#install). @@ -152,6 +154,12 @@ automatically uses the package's only template when none is explicitly marked. Multiple templates without one default are an error that points to `--list-templates`. There is no separate `--variant` vocabulary. +The `@version` of the command line is read as written (2026.11.21.1+): `@1.19.0` +selects the highest published key that starts with it, such as a packaging +revision `1.19.0.2`; `@=1.19.0.1` selects that key alone; `@^1.19` is a +requirement, read as in `mcpp.toml`. A manifest's version string keeps the +manifest's reading (docs/05). + A package template can use a module wrapper for a traditional header library. The module name and the C++ API namespace are independent: importing a wrapper does not rename the upstream types or functions. Use the package's published diff --git a/docs/04-mcpp-toml.md b/docs/04-mcpp-toml.md index 48c19739d..c1c11fff9 100644 --- a/docs/04-mcpp-toml.md +++ b/docs/04-mcpp-toml.md @@ -1096,7 +1096,9 @@ longer updated; the first build that finds it says so once, naming this key. | Key | Chapter | |---|---| | `allow_host_libs` | [20 — Toolchains](20-toolchains.md), [22 — Target Side](22-target-side.md) | +| `allow_host_tools` | [25 — Home, Indices and Store](25-home-indices-and-store.md) | | `ios_deployment_target` | [20 — Toolchains](20-toolchains.md) | +| `offline` | [05 — Dependencies](05-dependencies.md), "Offline builds and mcpp fetch" | | `platform-dependencies` | [22 — Target Side](22-target-side.md) | | `std-module`, `std-compat-module` | [22 — Target Side](22-target-side.md) | @@ -1225,6 +1227,24 @@ its final path with `O_TRUNC` and a reader could observe a half-written file. MSVC is left alone — neither `/ifcOnly`'s cost nor `.ifc` atomicity has been measured, and guessing either wrong is silent. +### Available memory (`memory_threshold`) *(2026.11.21.1+)* + +```toml +[build] +memory_threshold = "2GiB" # a size ("512MiB", "1.5GB") or bytes; "0" waits for nothing +``` + +The compiler executor reads the system's available memory after it holds its +slot and before it starts a compiler. Below this value it waits for a compiler +already running to end; it ends none. The engine's value is 1 GiB, since one +module interface compile peaks at 0.5–1.0 GB (above). The value is taken from +`MCPP_MEMORY_THRESHOLD`, this key, `[build] memory_threshold` in +`~/.mcpp/config.toml`, and the engine, in that order; a value that is not a size +is reported and the next one holds. After a minute below it the executor says so +on standard error and starts the compiler: memory that no compiler of this machine +holds would be waited for forever ([SPEC-004 §9](specs/manifest-semantics.md) +item 19). + ### Module interface extensions (`module_extensions`) mcpp treats `.cppm` as a module interface unit. The C++ ecosystem has not @@ -1786,6 +1806,7 @@ ndebug = true # -DNDEBUG for every unit of the graph (default: on fo trim_paths = true # -ffile-prefix-map per unit (default: on for release/dist) split_debuginfo = "off" # "unpacked": debug information beside the objects (default: dev/debug) bmi_cutoff = "exact" # "ignore-locations" (default): a GCC BMI whose line maps alone moved is kept +dependency_debug = "lines" # the store's and git's dependencies: true | false | "lines" (default under dev/debug) # passthrough escape hatch (fixed keys, open values): cflags = ["-fno-plt"] cxxflags = ["-fno-plt"] @@ -1829,6 +1850,17 @@ anything compiles. into the program (`__builtin_LINE()`, `std::source_location::current()` in a template's body) and a graph compiled with `-fcontracts` are compared exactly under either value. Clang's BMI has no such part and the key has no effect on it. +- **`dependency_debug` (2026.11.21.1+)** states the debug information of the dependencies + that come from the package store and from git: `true` (the profile's `debug`), + `false` (none), `"lines"` (the line tables alone: `-g1` on GCC, + `-gline-tables-only` on Clang, from the toolchain's row). Omitted, `dev` and `debug` + take `"lines"` while their `debug` is on, and every other profile takes its + `debug`. The members of the workspace and path dependencies keep the profile's + `debug`. A backtrace through a dependency still names its files and lines; + measured on a project whose only dependency is mtest, its test program fell from + 43.7 MB to 8.3 MB and its build from 15.9 s to 7.2 s. The flag enters the + dependency's configuration, so each value has its own entry in the cache + ([SPEC-004 §9](specs/manifest-semantics.md) item 20). - **`ndebug` and `trim_paths` (2026.10.24.1+)** follow the industry's convention: on for `release` and `dist`, off for `dev`, `debug` and a custom profile that does not state them. `ndebug` defines `NDEBUG` for every unit, so `assert` compiles out, as in diff --git a/docs/05-dependencies.md b/docs/05-dependencies.md index f42f4ef6c..f5594403b 100644 --- a/docs/05-dependencies.md +++ b/docs/05-dependencies.md @@ -67,8 +67,8 @@ not deprecated: ```toml [dev-dependencies] -mcpplibs.mtest = "0.10.0" -"mcpplibs.mtest.gtest" = "0.10.0" +mcpplibs.mtest = "0.11.2" +"mcpplibs.mtest.gtest" = "0.11.2" ``` ```toml @@ -463,8 +463,9 @@ Controls, in order of precedence: | Control | Effect | |---|---| -| `--offline` (any command) | Never touch the network — no index refresh, no downloads, no toolchain auto-install, no `git ls-remote`/`clone`. Anything already installed still builds, including git deps whose commit is in `mcpp.lock` and whose clone is cached | +| `--offline` (any command) | Never touch the network — no index refresh, no downloads, no toolchain auto-install, no `git ls-remote`/`clone`. Anything already installed still builds, including git deps whose commit is in `mcpp.lock` and whose clone is cached; a payload the home lacks comes from an archive directory, if one is named (see below) | | `MCPP_OFFLINE=1` | Same, for a whole shell session or CI job | +| `[build] offline = true` (2026.11.21.1+) | Same, stated by the project (`mcpp.toml`) or by the machine (`config.toml`); the command line and the variable take precedence, then the project, then the machine. `MCPP_OFFLINE=0` turns a stated `offline` off for one run | | `[index] auto_refresh = false` in `~/.mcpp/config.toml` | Never refresh an index implicitly: not on a dependency miss, not on a target row the index's platform column does not list (2026.11.19.1+), not before installing a package the local index lacks, and not for the first sync of a project's custom index (that build stops and names `mcpp index update`). Downloads still work | `MCPP_NO_AUTO_INSTALL=1` remains accepted as the older, narrower spelling of @@ -479,6 +480,50 @@ terminates the xlings process it started. Run any command with `-v` to see the decision for each dependency and why. +### Offline builds and mcpp fetch (2026.11.21.1+) + +`mcpp fetch` is the step of a build that obtains what the build needs, as a +command of its own. It resolves and plans each configuration as `mcpp build +--configure-only` does — the dependencies by the lock, the toolchain and the +payloads it declares, the target's base, the build programs' dependencies, the +tests' dependencies — so that everything is in the home, and it compiles +nothing. It takes the selection options of `mcpp build` (`--target`, repeatable; +`--toolchain`, `--sysroot`, `--platform`, `-p`, `--workspace`, `--profile`, +`--features`). + +```bash +mcpp fetch # online: put what the build needs in the home +mcpp fetch --to ./bundle # then write it into a directory +mcpp fetch --from ./bundle # take such a directory into this home first +mcpp build --offline # afterwards: no network +``` + +- `--to ` writes every payload the run used, with the closure of each + (xlings `download --to --with-index`), a snapshot of each index that resolved + them, a `git bundle` of each git clone the build used, and `bundle.toml`, + which describes the directory. The directory's layout is not an interface: + a directory one release writes, that release and every later one read. +- `--from ` places the bundle's index snapshots where the home has no index + of its own (an index the home keeps as a git work tree is kept), installs the + packages resolving with those snapshots, imports the clones into + `~/.mcpp/git`, and then plans as `mcpp fetch` does. A home that did not exist + is created and bootstrapped from the bundle. Afterwards the directory may be + removed. +- `--from` refuses a bundle that lacks a version `mcpp.lock` records + (`bundle-older-than-lock`), and warns when the bundle was written on another + host or for none of the selected targets (`bundle-lacks-target`). + +An offline run that needs a payload the home lacks is refused by its cause: + +| The run | Refusal | +|---|---| +| names no archive directory | `offline-download-required`: nothing is asked; the remedy names how the run became offline and `mcpp fetch --to`, `--from` | +| names archive directories (`[xlings] archives`, `MCPP_ARCHIVES`; 25 — Home, Indices and Store) | the download cache and the directories are asked, the network is not; `artifact-unavailable-offline` when none holds the archive, with the provider's list of what it lacked | + +The commit of a git dependency comes from `mcpp.lock` offline, so a project +carried with `mcpp fetch --to` commits its lock. `mcpp self doctor` installs +nothing, whatever archive directories are named. + ## `[dev-dependencies]` — Test Dependencies ```toml @@ -809,9 +854,9 @@ it rather than building it without its patches. ## Current limitations - **Two things need the network, and only two:** resolving a branch that has no - commit in the lock, and cloning a commit that is not cached yet. A `git =` - value naming a local directory or a `file://` URL needs neither and is never - refused offline. + commit in the lock, and cloning a commit that is not cached yet (a clone + `mcpp fetch --from` imported is cached). A `git =` value naming a local + directory or a `file://` URL needs neither and is never refused offline. - The index-refresh window **does not apply to a selector that states a namespace**. `mcpplibs.gtest` misses and stays missed until the next refresh. - `mcpp.lock` records and verifies a resolution; it does not constrain one. See diff --git a/docs/06-features-and-capabilities.md b/docs/06-features-and-capabilities.md index 51488a3c4..0ea1a6eaf 100644 --- a/docs/06-features-and-capabilities.md +++ b/docs/06-features-and-capabilities.md @@ -94,7 +94,7 @@ simd = { sources = ["src/simd/**"], flags = [ ``` - **The table form accepts exactly** `implies`, `forward`, `defines`, `sources`, - `flags`, `requires`, `provides`, `tools`. Anything else is reported as a schema warning + `flags`, `requires`, `provides`, `tools`, `description`. Anything else is reported as a schema warning and ignored (mcpp 2026.9.1.1+); `deps` is reported separately as reserved and points at `[feature-deps.]`. Before that release `[features]` was the one structured section with no schema check at all, so a misplaced @@ -125,6 +125,36 @@ simd = { sources = ["src/simd/**"], flags = [ receives them as if its dependency edge had written `tools = [...]` ([05, build-time dependencies](05-dependencies.md)); a name that is not a `bin` target of the package is refused at load. +- `description` (mcpp 2026.11.21.1+) is one line for a reader of the package's + features, as `[package] description` is for the package. It changes nothing + in a build; an index descriptor's `features.` accepts it too. The array + shorthand has none. + +### Reading a package's features (mcpp 2026.11.21.1+) + +A package's features are machine-readable in two places +([50 — Machine Output](50-machine-output.md)): + +- `mcpp search --format json` states, for each package, the features of its + newest version (the first of `versions`): their names, which are in the default set, what + each implies and enables, and the description. A descriptor that carries its + manifest inline answers from the index. A descriptor whose manifest is inside + the payload answers from the installed payload; when the payload is not + installed, the answer says so and has no list. The query installs nothing. +- `mcpp why features` lists every package of the project's resolved graph with + the features it declares and those the build activates: + +``` +$ mcpp why features +features: + mcpplibs.app@0.1.0 (root) + (declares none) + mcpplibs.widget@0.1.0 + extra active implies simd + Extra helpers + fast default, active implies simd + simd default, active +``` ### A feature that is a build rule (mcpp 2026.9.7.1+) diff --git a/docs/08-testing.md b/docs/08-testing.md index 805971bb5..cf9917af9 100644 --- a/docs/08-testing.md +++ b/docs/08-testing.md @@ -147,8 +147,9 @@ object whole. A run is kept in the form of the mtest store (2026.11.9.1+): `target/mtest/` holds `runs//run.json`, a line per process in `programs.ndjson` (its -status, duration, start, worker and selected cases, and the program's `usage` when it reported -one) and a line per case in `cases.ndjson` (its kind, tags, start after the run began, failures +status, duration, start, worker and selected cases, the program's `usage` when it reported +one, and the steps, used items and metrics it wrote outside every case, as a suite's body does, +2026.11.21.1+) and a line per case in `cases.ndjson` (its kind, tags, start after the run began, failures with their expected and actual values, metrics and bounds, 2026.11.10.1+), and `index.ndjson` lists the runs. A `Bench` case's measurements are therefore in the store, and `mtest analyze trend` shows them beside their bounds over the runs `mcpp test` kept. `mtest store import --from @@ -159,6 +160,30 @@ the slowest cases, the critical path and the schedule). In CI, `commit` and `group` are `GITHUB_SHA` and `GITHUB_RUN_ID`. `[test] store = ""` names another directory, relative to the project; `"none"` keeps nothing. +**What a run records** (2026.11.21.1+, [SPEC-017](specs/test.md) B4, P5.6, P12.4). A run's +`run.json` and its line in `index.ndjson` name the machine's class (`machine`: +the processor's model), its `processors` and `memory`, the monitoring level the +run stated (`MTEST_MONITOR`, `basic` unless the environment states another: each +case's processor time) and the jitter, when one was stated. A case's line adds +its processor time (`cpu_ms`), its steps (each command an end-to-end case ran, +with its time and status), the items of the product's surface it used +(`exercised`) and the limit it ran within (`limit_ms`). `builds.ndjson` holds the +builds of the run: a line per edge that ran, with its package, its time, the +processor time and the largest resident set of the compiler, and whether the unit +cache placed it (`served`) or it copied a unit of a package the cache served +(`staged`), and a line per package the cache served. The +estimates of the order prefer the runs of the same class of machine. + +**One budget for the machine** (2026.11.21.1+, SPEC-017 O11). `mcpp test` makes a +directory of one slot file per job and states it to its programs +(`MTEST_BUDGET`, and `MTEST_BUDGET_HELD`, the slot a worker holds); a compiler a +test starts through mcpp holds a slot of it before it starts, so a test that +builds a project takes from the run's jobs rather than adding its own. A run +inside another runner's budget uses that one. + +**A case's time is the one its scopes state** (2026.11.21.1+). A case selected with +`--case` runs within the `timeout` its listed case states, not the run's. + A suite compiled from several sources of its own is also a package: a workspace member whose `[build] sources` carries the suite and whose one test program drives it, selected with `mcpp test -p `. @@ -335,8 +360,13 @@ the tests a change affects, read from the plan before anything is built: - a test is affected by a change of a file it is built from: its own files, the module units it imports and the units those import in turn, and every unit of its link that belongs to no module; -- a change of a manifest, a lock, a build program, a header, or a source the - plan does not hold affects every test; +- a change of a manifest, a lock, a build program, or a source the plan does + not hold affects every test; +- a header (2026.11.21.1+) affects the tests built from a unit that read it when + the build directory last built it, as ninja's deps log and the scans' + depfiles record it, and the tests that run a program of the package when + such a unit is not a test's own; while a unit of the project has no such + record (a build directory that never built it), a header affects every test; - a test that runs a program of the package (a file it is built from names `MCPP_BIN_`) is affected by a change of any source of the package that is not a test's own (2026.11.13.1+): an end-to-end test runs the program it @@ -612,6 +642,22 @@ The records of `--message-format json` carry the protocol, the cases and their failures ([50](50-machine-output.md)); `--print-launch` states the protocol for a debugger that starts the program. +**Run fixtures** (2026.11.21.1+, SPEC-017 O6). A program may declare a run fixture -- +one object the whole run shares: a sandboxed home, a server, an index -- and its +cases may name it in `uses` (mtest's `run_fixture` and `uses{}`). `mcpp test` +reads both from the programs' listings. Before the first program whose selected +cases use a fixture starts, it starts a program that declares it with +`MTEST_SERVE` naming it and its input a pipe, and waits, at most `--timeout`, for +its line `mtest-ready `; each user receives the endpoint as +`MTEST_FIXTURE_`. After the last user ends, its input is closed, and it is +ended after the same time if it has not ended. `fixtures.ndjson` in the run's +store states each fixture's start, readiness and end, so its preparation is not +counted in the first case that used it; `programs.ndjson` names the fixtures each +program used. A fixture that does not become ready is the machine's failure: the +programs that use it do not run, their records say `infra`, and the run exits +with 2 as for a test that was not run. A program run behind a runner is not +listed and receives no fixture. + ## Writing a test ```bash @@ -649,6 +695,23 @@ Builds the one test the pattern names and prints how `mcpp test` would start it (the `mcpp.launch` envelope, mcpp 2026.10.31.2+), for a debugger to start: the runner, the environment the run adds and the arguments after `--` are in it. +### In a GitHub Actions job (2026.11.21.1+) + +```yaml +- uses: actions/checkout@v4 +- uses: mcpp-community/mcpp/.github/actions/project-test@main + with: + args: --workspace # the words of `mcpp test` +``` + +The action installs the mcpp the project's `.xlings.json` names (else the input +`mcpp`, else the newest release), restores the home by the system, that version +and the project's manifests and lock (`home-key` replaces the last), runs +`mcpp test`, keeps the run's store (`target/mtest`) as an artifact and writes the +run's result, its ten slowest cases and the ten slowest edges of its builds to +the job's summary. A pull request reads the home; a push to the default branch +saves it. + ## Current limitations - A test is one `.cpp` producing one program. The cases inside it are the diff --git a/docs/09-commands-by-scenario.md b/docs/09-commands-by-scenario.md index d0a26d8e5..aabb4e9ad 100644 --- a/docs/09-commands-by-scenario.md +++ b/docs/09-commands-by-scenario.md @@ -231,18 +231,22 @@ build programs, or for both (2026.10.11.1+). `mcpp why build-programs` lists the build programs' graphs: each program, the libraries it uses, the identity it was compiled from and its binary. `mcpp why compat` lists every use of a deprecated reading with the release that removes it (51 — Supported Versions). +`mcpp why features` lists each package with the features it declares and those +the build activates (2026.11.21.1+; docs/06). The same graph is recorded under `graph` in `target///resolution.json`, one entry per package with the root first: `package` (canonical identity, namespace, name, version, source), `root`, `requested_by` (`requester`, `key`, `table`), `manifest_dir` (the -package's root), and, for a library, `link` (`form`, `reason`). +package's root), `features` (those the build activates) and `feature_list` +(those the package declares, 2026.11.21.1+), and, for a library, `link` +(`form`, `reason`). `mcpp why target` lists a request's candidates in order (docs/21), `mcpp why flags []` each word of a unit's command line with the layer that gave it, and `mcpp why platform` the rows a platform defines and the identities it -serves (docs/35); the last two are not part of the report without a topic -(2026.11.9.1+). +serves (docs/35); the last two, and `features`, are not part of the report +without a topic (2026.11.9.1+). The topic is `toolchain`, `runtime`, `deps` or `runners`, and all four report when none is named. `--target` and `--toolchain` turn the report into a query @@ -290,8 +294,13 @@ that stops it. `mcpp self doctor` lists every index whose floor the running mcpp does not meet. Earlier versions printed the E0006 text as `error:` at the start of any run that read such an index, including runs that then succeeded. -`--offline` (or `MCPP_OFFLINE=1`) forbids the network for one invocation and -fails rather than fetch. `--locked` fails when resolution differs from +`--offline` (or `MCPP_OFFLINE=1`, or `[build] offline = true` in `mcpp.toml` or +`config.toml`) forbids the network and fails rather than fetch; a payload the +home lacks comes from a named archive directory, if any. `mcpp fetch` puts +what a build needs in the home beforehand, `mcpp fetch --to ` writes it +into a directory and `mcpp fetch --from ` takes that directory into +another machine's home (2026.11.21.1+; 05 — Dependencies, "Offline builds and +mcpp fetch"). `--locked` fails when resolution differs from `mcpp.lock` instead of rewriting it, which is the shape a CI job wants. `mcpp index pin ` records a commit for a custom index in `mcpp.toml`; `mcpp index unpin` removes it. Both edit an `[indices]` entry @@ -799,7 +808,7 @@ mcpp is involved and its documentation will not help. ## Current limitations - `mcpp why --format json` is defined for the `toolchain`, `sources`, `tool`, - `payload`, `build-programs`, `compat`, `target`, `flags` and `platform` topics. The other topics report + `payload`, `build-programs`, `compat`, `features`, `target`, `flags` and `platform` topics. The other topics report `'' has no machine-readable shape yet` and exit non-zero. - `mcpp search` matches a substring; there is no field selector, and no way to restrict a search to one namespace. diff --git a/docs/20-toolchains.md b/docs/20-toolchains.md index b4d409c20..78695dc53 100644 --- a/docs/20-toolchains.md +++ b/docs/20-toolchains.md @@ -22,14 +22,14 @@ On the first `mcpp build` with no toolchain configured, mcpp installs and persists a default pair for the current host. The choice is host-aware: -- Linux x86_64 uses `gcc@16.1.0` for the native glibc ABI, so X11, OpenGL, and +- Linux x86_64 uses `gcc@16.2.0` for the native glibc ABI, so X11, OpenGL, and system libraries work out of the box. - Linux aarch64 uses `llvm@23.1.3` for the native glibc ABI (2026.10.8.1+). -- Other Linux architectures use `gcc@15.1.0-musl`, a self-contained static +- Other Linux architectures use `gcc@16.2.0-musl`, a self-contained static toolchain. - macOS uses `llvm@23.1.3`. - Windows with a usable MSVC installation uses `llvm@23.1.3` for the MSVC ABI. - Without usable MSVC, it uses `gcc@16.1.0` with target + Without usable MSVC, it uses `gcc@16.2.0` with target `x86_64-windows-gnu` (MinGW-w64, static by default), says which half of the MSVC ABI is missing — Visual Studio's C++ tools or the Windows SDK — and where it looked, and records the pair as the machine's default. Once both halves @@ -77,20 +77,21 @@ this model with a one-line `note:` hint. ## Manual Installation ```bash -mcpp toolchain install gcc 16.1.0 # host target (GNU libc on Linux) +mcpp toolchain install gcc 16.2.0 # host target (GNU libc on Linux) mcpp toolchain install llvm 23.1.3 # LLVM/Clang, default on macOS and Windows with usable MSVC mcpp toolchain install gcc 16 --target x86_64-linux-musl # musl target payload mcpp toolchain install --target x86_64-windows-gnu # family omitted → the - # target's convention pin (gcc@16.1.0) + # target's convention pin (gcc@16.2.0) ``` Explicit installation is mostly for CI cache warm-up and offline prep — `mcpp build --target ` auto-installs whatever the target needs. -Version numbers support partial matching: +Version numbers support partial matching, resolved as described in +[Resolution and the lock](#resolution-and-the-lock-202611211): ```bash -mcpp toolchain install gcc 15 # installs the highest 15.x.y version (15.1.0) +mcpp toolchain install gcc 15 # the newest 15.x.y release of the index mcpp toolchain install gcc@16 # the @ form works too ``` @@ -165,17 +166,84 @@ and the C units as with any compiler. A unit that declares a module, is a module unit or imports one (`import std;` included) is refused at planning, naming the file and what it needs. +## Resolution and the lock (2026.11.21.1+) + +A request for a packaged toolchain has one answer, the same in +`mcpp toolchain install`, `mcpp toolchain default` and a build: + +| Request | Answer | +|---|---| +| a full version (`gcc@16.1.0`) | that version | +| a partial version (`gcc@16`, `gcc@16.2`) | the newest release of the local index whose version it is a prefix of; a build installs it when it is absent | +| a prerelease (`xim:gcc@17.0.0-20261108`) | selected by its full key only; a partial version never selects it | +| any request, with no index entry for the package | the newest installed match, reported as degraded: a newer release is not considered | + +A partial version written in `mcpp.toml` (`[toolchain]`, +`[workspace.toolchain]`, `[target.] toolchain`) is recorded in +`mcpp.lock`, and later builds take the recorded version: + +```toml +[toolchain."xim:gcc@16"] +version = "16.2.0" +``` + +- A full version is its own record and writes none; a toolchain chosen by + `--toolchain`, by the machine's default or by mcpp is not recorded either. +- `mcpp update` drops every record; `mcpp update gcc` (or `mcpp update + xim:gcc@16`) drops that toolchain's. The next build resolves it again. +- Under `--locked` and `--frozen` the lock is kept as it is. A lock written + before the record has none for the request: the build resolves it, says so + in a note, and does not refuse. + +A project that states no toolchain follows the machine's default +(`mcpp toolchain default`) or mcpp's own. When that default answers another +toolchain than the one the project last built with, the build says so once: + +``` +note: the default toolchain (`mcpp toolchain default`) is now gcc@16.2.0; this project last built with gcc@16.1.0 + help: to keep a version, state it in mcpp.toml: [toolchain] default = "gcc@16.1.0" +``` + +The toolchain a project last built with is kept, per target, in +`target/.mcpp-notices`; a project's first build, and the first after +`mcpp clean`, has nothing to compare with. A `--toolchain` build is neither +compared nor remembered. + +## Companions (2026.11.21.1+) + +A managed toolchain's companions -- the binutils GCC assembles and links with, +the Linux kernel headers of the glibc base, GCC's runtime libraries -- are the +versions its payload declares as runtime dependencies for this host. A store +that holds several versions answers with the declared one; without a +declaration, with the newest. A declared version the store does not hold is +reported as a degradation; `mcpp toolchain install` installs it again. + +A project may state the binutils beside a GCC: + +```toml +[toolchain] +default = { package = "gcc@16.2.0", companions = { binutils = "xim:binutils@2.47" } } +``` + +The build installs the package, says `Companion binutils 2.47 ()`, and +passes its directory to the driver. A binutils older than the one the GCC +payload declares is refused (`companion-too-old`): GCC chose its assembler +features by that binutils when it was built. `companions` names binutils only; +the kernel headers follow the toolchain payload's declaration, since the +default base is the home's sysroot, which every project shares. + ## Switching the Default Toolchain The default is a *pair* — toolchain axis + target axis (target omitted = host): ```bash mcpp toolchain default gcc@16.1.0 -mcpp toolchain default gcc 15 # partial version → highest installed match +mcpp toolchain default gcc 15 # partial version → the newest 15.x.y release, which must be installed mcpp toolchain default gcc@16 --target x86_64-linux-musl # "default to fully-static musl" ``` -The pair persists as `[toolchain] default = "gcc@16.1.0"` + +A partial version is resolved as a build resolves it, and the full version is +what is written. The pair persists as `[toolchain] default = "gcc@16.1.0"` + `default_target = "x86_64-linux-musl"` in `~/.mcpp/config.toml`. (Older configs with combined spellings like `default = "gcc@15.1.0-musl"` keep working unchanged.) diff --git a/docs/21-the-target-triple.md b/docs/21-the-target-triple.md index 47f10ee26..9bba9acc9 100644 --- a/docs/21-the-target-triple.md +++ b/docs/21-the-target-triple.md @@ -291,6 +291,20 @@ aarch64-macos → aarch64-unknown-macos A three-field spelling is a four-field one with a field elided, and which field was elided follows from what parses. +The vendor field is any word (2026.11.21.1+), as GNU `config.sub` reads it: the +field after the architecture, when a system field follows, names whoever built +the compiler. A distribution's compiler reports its own (`x86_64-redhat-linux`, +`x86_64-suse-linux`), and that is `x86_64-linux-gnu`. The fields after it are +read strictly. A compiler named by path whose target mcpp cannot read is +reported as a degradation (`toolchain/triple`): the settings that follow the +target are not applied to that build. + +A compiler's version is the one its driver states for machines: GCC's +`-dumpfullversion` (GCC 7 and later, else `-dumpversion`) and clang's +`-dumpversion`. The `--version` banner names the family; a distribution's banner +ends in its own release number (`(Red Hat 16.2.1-3)`), which is not the +compiler's version. + ### GCC and clang do not select targets the same way | | GCC | clang | diff --git a/docs/23-the-project-environment.md b/docs/23-the-project-environment.md index c553ee4e0..087b53bd8 100644 --- a/docs/23-the-project-environment.md +++ b/docs/23-the-project-environment.md @@ -465,6 +465,20 @@ statement, where they come from is the project's. A build reports every override it used, and `--managed-only` refuses a build that uses any: see [20 — Toolchain Management](20-toolchains.md). +### `archives` — directories of archives (2026.11.21.1+) + +```toml +[xlings] +archives = ["vendor/archives"] # relative to mcpp.toml +``` + +The archive directories of this project's builds, asked by sha256 before the +network and in its place offline. Read from the root of a build; `MCPP_ARCHIVES` +takes precedence, and `config.toml`'s `[xlings] archives` applies when the +project states none. A directory inside the project, such as one committed with +it, holds on every machine; one outside it is a note once (25 — Home, Indices +and Store, "Archive directories"). + ### A rule package brings its own environment (2026.9.6.6+) The table above is what a project writes when it has an opinion. Most projects diff --git a/docs/25-home-indices-and-store.md b/docs/25-home-indices-and-store.md index 51e3ed7bf..ea2e0e197 100644 --- a/docs/25-home-indices-and-store.md +++ b/docs/25-home-indices-and-store.md @@ -112,6 +112,61 @@ and source attempt. Provider preparation and process timeouts are reported separately. Mirror selection uses `mcpp self config --mirror auto`, `CN` or `GLOBAL`. +### Host inputs (2026.11.21.1+) + +A build takes its toolchain, its base and its bootstrap tools from the store, +and the machine only where a statement says so: + +| Input | Enabled by | +|---|---| +| the machine's toolchain | `[toolchain] default = { path = … }`, `msvc@system` | +| the machine's C library | `sysroot = "glibc@system"` | +| the machine's libraries | `[build] allow_host_libs = true`, `MCPP_ALLOW_HOST_LIBS` | +| a dependency's tool | `[xlings.overrides]`, `MCPP_XLINGS_OVERRIDE__` | +| a bootstrap tool | `[build] allow_host_tools = ["ninja", "patchelf"]` | + +`allow_host_tools` is read from `MCPP_ALLOW_HOST_TOOLS` (a comma list, `none` +for none), else the project's `[build]`, else `config.toml`'s `[build]`. A tool +it names is taken from `PATH` when it is there, and from the store otherwise. +patchelf is run by mcpp only when it packs; a build does not require it. + +Each input is a note once per project, and again when its path or version +changes: + +``` +note: ninja (bootstrap) is this machine's (1.11.1 /usr/bin/ninja), not the store's + help: another machine may build differently; docs/25, "Host inputs" +``` + +When `mcpp.toml` states the input, the note adds that the statement holds only +on machines like this one, and that a machine's input belongs in `config.toml`. + +### Archive directories (2026.11.21.1+) + +An archive directory holds the archives payloads are installed from, found by +their sha256 whatever their file name: a directory `mcpp fetch --to` wrote, a +mirror on a shared volume. + +```toml +# ~/.mcpp/config.toml +[xlings] +archives = ["/srv/mcpp-archives"] +``` + +The list is read from `MCPP_ARCHIVES` (separated as `PATH` is), else the +project's `[xlings] archives` (relative to `mcpp.toml`), else `config.toml`'s +(relative to it), and is passed to xlings, which asks its download cache, then +these directories, then the network. Offline (`--offline`, `[build] offline`), +the network is not asked: a payload the home lacks is installed from the cache +or a directory, and one no source holds is refused +(`artifact-unavailable-offline`, 05 — Dependencies, "Offline builds and mcpp +fetch"). A directory outside the project named in `mcpp.toml` is a note once: +another machine lacks it, and a machine's directory belongs in `config.toml`. + +An upstream (§3) and an archive directory are sources of different things, and +both hold: an upstream's installed payloads are published by link; an archive +is verified and installed. + ### Retention of build outputs (2026.10.31.3+) The build cache under `build-cache/v1/` is shared by every project of the home; @@ -119,7 +174,7 @@ a project's `target/` is its own. Each has a retention, in config.toml: | key | default | removed | |---|---|---| -| `[cache] unused_days` | 30 | after a build, at most once a day: cache entries (packages, `std` modules, host tools) no build used for that many days | +| `[cache] unused_days` | 30 | after a build, at most once a day: cache entries (packages, `std` modules, host tools, units, build programs' outputs) no build used for that many days | | `[build] keep_days` | 3 | after `mcpp build` or `mcpp test`, at most once a day per project: build directories no configuration used for that many days | | `[build] target_budget` | none | after every `mcpp build` or `mcpp test` whose `target/` holds more than this size (`"20GiB"`, `"512MiB"`): every build directory no configuration records as current, whatever its age (2026.11.8.1+) | diff --git a/docs/26-build-optimizations.md b/docs/26-build-optimizations.md index e8035fb0a..84de0de9c 100644 --- a/docs/26-build-optimizations.md +++ b/docs/26-build-optimizations.md @@ -86,6 +86,41 @@ The note is given when the rebuild started at a module of one file with at least importing units, once a build directory. Beside it, a build that replaced or kept BMIs states which ([04](04-mcpp-toml.md), `bmi_cutoff`). +## What the home's cache places (2026.11.21.1+) + +Beside the entries, the home's build cache (`build-cache/v1/`, [25](25-home-indices-and-store.md)) +keeps what one build made for the next, in any project of the home, whenever `[build] cache` +is the home's (the default): + +- **Units.** Before it starts a compiler, the compiler executor looks up the compile by its + command, its compiler and the content of its source. An entry names the files an earlier + compile of that key read (its depfile, or the scan's depfile and the BMIs it imported) with + their contents; when they all still hold those contents, its object and BMI are placed and + the compiler does not start. After a compile it keeps the outputs and the files read. A BMI + is named by the key of the compile that made it, since GCC writes a different BMI on each + compile of the same source; a file of a package store is named by its path, since the store + does not change. A fresh checkout of mcpp on a filled cache built in 4.1 s instead of 164 s, + its 288 compiles all placed. cl.exe's compiles are not cached. +- **The outputs of build programs.** A package whose source is named -- an index's version, a + git commit, a path dependency's content digest -- runs its build program into + `build-cache/v1/out/@//out` (`MCPP_OUT_DIR`), the key being its + configuration without that directory: a second project takes the outputs, and runs nothing + ([SPEC-004 §9](specs/manifest-semantics.md) item 18). A program run by `mcpp pack + --pkg-format` runs in the project: what it writes depends on the pack's request and the + member's staged tree. + +Entries no build used within `[cache] unused_days` are removed as other entries are. + +## The record of a build (2026.11.21.1+) + +A build records each edge it ran: its package, its time, and, for a compile, the compiler's +processor time and largest resident set and whether the unit cache placed it; and the units +the cache served each package. `MCPP_BUILD_RECORD=` appends the record of every build of +the invocation to the file, a line per edge ([SPEC-017](specs/test.md) B4, `builds.ndjson`); +`mcpp test` keeps the builds of a run in its store. Under `--verbose` a build names its five +slowest edges, each on a `Slowest` line with its processor time and memory. An edge that has +run for a minute is named once on a `Running` line, with the package and source it compiles. + ## Stating an entry otherwise ```toml diff --git a/docs/30-build-mcpp.md b/docs/30-build-mcpp.md index cc79b8fb4..c8a60683c 100644 --- a/docs/30-build-mcpp.md +++ b/docs/30-build-mcpp.md @@ -458,8 +458,11 @@ const char* same = mcpp::xpkg_dir("picolibc-riscv"); // bare name The namespaced form answers only for a package declared under that namespace and is the one to prefer; the bare form is a convenience for the common single -declaration, and when two namespaces claim one name it answers for the first -**declared**. Both return `""` when the package was not declared or is not +declaration. When two namespaces claim one name, the bare form is judged as a +name two dependencies answer to (see "A name two dependencies answer to is no +package's"): until 2027.2 it answers for the first **declared** and the build +warns of the deprecated reading `ambiguous-contract-name`; from 2027.2 it ends +the program naming the namespaced forms. Both return `""` when the package was not declared or is not installed — a program that needs it should say so itself, because only it knows whether the absence is fatal. @@ -1415,8 +1418,35 @@ nothing, the artifact is already there. A cross build (`--target …`) pays for one extra std module, because `build.mcpp` compiles and runs on the *host* while the project targets something else. -`#include` still works and stays the right choice for a program that only -needs `std::fopen`; there is no requirement to modularize a build script. +`#include` works; "Headers or modules" below states what it costs and the form +that replaces it. + +### Headers or modules (2026.11.21.1+) + +The recommended `build.mcpp` imports what it uses: the standard library as +`import std;`, the engine interface as `import mcpp;`, and its own shared code as +a module of a path package listed in `[build-dependencies]`. Two forms work and +are not recommended: + +| Form | Instead | Reason | +|---|---|---| +| a standard or system header, `#include ` | `import std;` | the header is parsed again by every compile of the program; the `std` module is built once per toolchain and standard and is shared with the main build | +| a header of the package or of a dependency, `#include "support/emit.hpp"` | a module of a path package in `[build-dependencies]`, imported | the package has an identity, can be shared by several programs and packages, and its BMI is built once | + +The first build of a project says each form once, as a `note` of two lines +(`MCPP_BUILD_PROGRAM_INCLUDES_HEADER`), naming every package whose program uses +it; later builds do not repeat it until `mcpp clean` removes `target/`. A program +of an index package or a git dependency is not the project's to change and is +not reported. The form is read from the program's text; the note changes nothing +the build does. + +A file the program's compile reads outside its package -- an absolute path, +`../shared/x.hpp` -- makes the build depend on this machine, and the package's +archive does not carry it. A build warns once per package and file +(`MCPP_BUILD_PROGRAM_READS_OUTSIDE`), and `mcpp publish` refuses such a file, and +a file of the package that HEAD does not track, with the reason +`build-program-unpublished-input`. A program's reads are limited to its package, +its `[build-dependencies]` and the store. Every toolchain mcpp can build a host program with can build a `build.mcpp`, including native MSVC — the module handling reads the same tables the main @@ -1710,13 +1740,29 @@ A generated **module interface** is fine here: `.cppm` goes through the same scan as any other source, so a generated file declaring `export module …` can be imported by the package's own TUs. -## Incremental: declared inputs (no needless re-runs) +## Incremental: observed and declared inputs (no needless re-runs) mcpp does **not** re-run `build.mcpp` on every build. It caches the program's -directives and re-runs only when something it depends on changed: - -- the `build.mcpp` source itself, -- the toolchain, +directives and re-runs only when something it depends on changed. Its inputs are +of two kinds. + +**What the program's compile reads is observed** (2026.11.21.1+). The files the +compile of `build.mcpp` read -- a header it includes, a file it embeds -- are +taken from the compile's dependency record (a GNU depfile; cl.exe's +`/sourceDependencies`), as the main build takes them. A file of the store (the +toolchain, an index package) or of a `[build-dependencies]` package is named by +that package's identity already and is not recorded. An edit of a recorded file +compiles the program again and runs it again, on the fast path too and after +`target/.build-mcpp` is deleted. A file is compared by its size and modification +time, and by its content when either moved. The record follows the compiler: +a header found at a new place on the include path, and a change below a system +include directory (`gcc@system` reading `/usr/include`), are not observed, as for +the main build. + +**What the program reads when it runs is declared**: + +- the `build.mcpp` source itself and the toolchain are inputs without a + declaration, - any file declared with `rerun-if-changed`, - any env var declared with `rerun-if-env-changed`, - (or a `generated` output / `source=` selection went missing), @@ -1724,10 +1770,12 @@ directives and re-runs only when something it depends on changed: — the entry carries a format **epoch**, and a foreign one re-runs the program once instead of replaying values under the wrong meaning). -So **declare the inputs**: if the program reads `config.h` or the `USE_FAST` -variable, emit `mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`. -This replaces the old "process exited 0, so assume it's fine" guesswork with an -explicit input/output contract — incremental builds stay correct. +So **declare what the program reads at run time**: if the program opens +`config.h` or reads the `USE_FAST` variable, emit +`mcpp:rerun-if-changed=config.h` / `mcpp:rerun-if-env-changed=USE_FAST`. No +host offers a portable way to observe what a process opens, so this stays a +contract: the program's input/output is explicit, and incremental builds stay +correct. A declared file is compared as an observed one is. A program that runs or fails has one line, written when it finishes (2026.9.30.1+): `build.mcpp ran