Skip to content

fix(cli): run mops bench cells with the incremental GC paused - #889

Closed
timohanke wants to merge 1 commit into
caffeinelabs:mainfrom
research-ag:fix/bench-gc-barrier
Closed

timohanke wants to merge 1 commit into
caffeinelabs:mainfrom
research-ag:fix/bench-gc-barrier

Conversation

@timohanke

Copy link
Copy Markdown
Contributor

Fixes #888

Problem

An incremental GC cycle can span several messages. While one is in progress, the GC's write and allocation barriers are on, and the cell pays for them in its own instruction count. --force-gc starts a new cycle at the end of every message, so with a large heap about every other cell ran with the barriers on. Identical cells differed by 20–40%, depending on the order of the cells. #888 has the details and the compiler source references.

This is about the Instructions table only. Heap stays exactly what the cell allocated (#879), and the GC still never runs inside the measured window.

Change

  • mops bench compiles the bench canister without --force-gc when the GC is incremental, which is the default and the only GC on moc 2. The legacy GCs of moc 1.x (--gc copying|compacting|generational) keep --force-gc: they finish a collection within the message, so nothing carries over to the next cell.
  • New update method collectorInstructions() in bench-canister.mo (and in the declarations), returning Prim.rts_collector_instructions().
  • Before each cell, waitForGcPause calls it until two consecutive calls return the same count: the message in between ended without a GC increment, so no cycle is in progress and the cell runs with the barriers off. It gives up with an error after 1000 messages.
  • Docs (mops bench: how it works) and changelog.

Tests

  • New fixture cli/tests/bench-gc-barrier: 8 identical cells (generic [var ?Nat] writes) on a 1M-object live heap.

    moc 2.0.0 moc 1.16.1
    3.5.0 alternates 17,300,245 / 12,500,245 alternates 17,300,245 / 12,500,245
    this PR all 12,500,245 all 12,500,245
    this PR, --gc copying – all 9,800,183 (still --force-gc)
  • New test "identical cells cost the same instructions, whatever ran before": checks forceGc: false in the saved results and a single distinct instruction count. With 3.5.0's bench.ts it fails with 2 distinct values.

  • tests/bench.test.ts (9 passed), tests/publish-dry-run.test.ts (10 passed), npm run check, prettier.

  • Real-world check: research-ag/sort's benchmark gives the same 36 instruction counts (±300) in both cell orders. On 3.5.0, 23 of its 36 cells were inflated by 23–40%.

🤖 Generated with Claude Code

An incremental GC cycle can span several messages. A cell that starts
while one is in progress pays for the GC's write and allocation barriers,
so identical cells cost up to 40% more instructions depending on what ran
before them. --force-gc made this happen in about every other cell, since
it starts a new cycle after every message.

Compile the bench canister without --force-gc for the incremental GC, and
before each cell call a new collectorInstructions() method until the
collector's instruction count stops changing. The legacy GCs keep
--force-gc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@timohanke
timohanke requested review from a team and automation-sa-sre as code owners October 12, 2026 12:59
Kamirus added a commit that referenced this pull request Oct 12, 2026
Supersedes [#889](#889) by
@timohanke, whose branch on an organization-owned fork doesn't accept
maintainer pushes. The first commit is #889 unchanged, so the squash
keeps @timohanke as co-author. Fixes
[#888](#888).

## Problem

`mops bench` instruction counts depended on the order of the cells. The
bench canister was compiled with `--force-gc`, so the incremental GC
started a new cycle at the end of every message, and on a large heap
that cycle ran on into the next cell. While a cycle is in progress, the
GC's write and allocation barriers are on, and the cell pays for them:
identical cells differed by 20–40%.
[#888](#888) has the compiler
details.

Dropping `--force-gc` isn't enough on its own. A cell that leaves enough
garbage still starts a cycle, and the next cell runs mid-cycle. In the
fixture below, that cell costs 16,100,150 instructions instead of
12,500,150.

## Fix

- The incremental GC is no longer compiled with `--force-gc`. The legacy
GCs of `moc` 1.x keep it, because they finish a collection within the
message.
- Before each cell, `mops bench` calls a new `collectorInstructions()`
method on the bench canister until two calls in a row return the same
value. `rts_collector_instructions` is the GC cost at the end of the
previous message, not a running total: about 330 instructions while the
GC is paused, and hundreds of millions per increment while a cycle runs.
After 1000 messages without a pause, `mops bench` fails.

## Before / after

`cli/tests/bench-gc-barrier`, `moc` 2.0.0: identical `fill` cells, each
after a `garbage` cell, on a 1M-object live heap.

Before (3.5.0):

```
|    |       fill |     garbage |
| :- | ---------: | ----------: |
| a  | 17_300_150 |  75_001_653 |
| b  | 16_100_150 | 158_552_560 |
| c  | 12_500_150 |  76_051_963 |
| d  | 16_100_150 | 157_503_563 |
```

After:

```
|    |       fill |    garbage |
| :- | ---------: | ---------: |
| a  | 12_500_150 | 75_003_204 |
| b  | 12_500_150 | 75_001_630 |
| c  | 12_500_150 | 75_001_681 |
| d  | 12_500_150 | 75_001_428 |
```

On a real benchmark, research-ag/sort, 23 of 36 cells were inflated by
23–40% on 3.5.0. With this fix, it gives the same counts in both cell
orders.

## Migration

Baselines saved with 3.5.0 can show lower instruction counts for cells
that used to start mid-cycle. To compare against fair numbers, re-save
them with `mops bench --save`. Saved and published results now record
`forceGc: false` for the incremental GC.

## What is unchanged

- The Heap table still shows exactly what a cell allocated, and the GC
still never runs inside the measured window.
- `--gc copying|compacting|generational` on `moc` 1.x still compiles
with `--force-gc` and skips the wait.

## Follow-up worth flagging

If two GC increments in a row cost exactly the same number of
instructions, the wait would end mid-cycle. That's unlikely, and the
Motoko runtime exposes no GC phase to check instead.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Timo Hanke <t@timohanke.org>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Kamirus Kamirus closed this Oct 12, 2026
@timohanke
timohanke deleted the fix/bench-gc-barrier branch October 14, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mops bench: instruction counts depend on cell order (incremental GC barriers active when a cycle spans messages)

2 participants