[wojo]

Fixes #1538.

The rows store's write worker no longer drops a save that fits the staging permit on its own, and no longer charges or
writes a block twice while it is still being written. ashhart asked on #1411 for both as their own pull request on
main.

Commits

1. rows: a block still being written is charged and gathered once.
   • newBytes counted every block a state names that r.blocks lacks, and r.blocks learns a job's blocks only
     when reap publishes it. A second save sharing blocks with a job in flight was charged for them and, if staged,
     gathered and wrote them again.
   • stageJob now takes a block as its own to write only when the disk lacks it and no job that can still publish
     writes it. Worker.promised collects those blocks under one lock, before the charge and again after the cap check
     (a fence inside room can cancel jobs). It holds the block files each job writes, so a job that skipped a block
     promises nothing for it, and neither does a fenced job or one that finished with an error. The charge, the file
     count and the gather loop use that one set.
   • publish now checks that every block a state names is on disk or among the job's own files, and fails the job
     (MissingBlock) otherwise, so no state on disk names a block that never reached it. The price: a save that
     skipped a block is lost with the job that failed to write it, where it used to write its own copy. A save staged
     after that job has failed writes the block itself.
   • The I/O worker picked and returned jobs by slot, so a slot freed early went to a newer job that then ran and was
     taken first. Both now go by job id, so a job that skips a block is published after the one that writes it.
2. rows: a save that fits the staging permit waits for room instead of being skipped.
   • stageJob now answers StagingBusy when the charge fits an empty permit and a job is in flight (the eight-job
     limit counts the same way). stage puts the save back where step took it from and remembers it in
     Rows.waiting. step then reports no work and ready treats the queue as idle while that save is the newest, so
     the idle turn does not spin. A reap that takes a job clears the mark, and a newer save is tried at once.
   • A save larger than the permit on its own is skipped as before, and so are refusals by the daily budget, the cap
     and the free-disk floor. An eviction cannot wait (its entry leaves RAM right after the copy), so a full permit
     still skips that copy. A close drains the writes in flight, then stages the saves that waited.
   • A waiting save whose entry leaves RAM first is counted as skipped (StagingFull), so the counters still show it.
3. rows: an older save that waited still counts as skipped when a newer one waits too (from ashhart's review).
   • A queued save carries its own mark (Queued.waited): stage puts a save back with it set, and dropped counts
     the save it removes as skipped when it has waited, unless the store has stopped. Rows.waiting is only the newest
     save refused room, which keeps step from spinning; dropped clears it when its entry leaves.
   • A save that waited and is dropped after a reap, before step tries it again, counts too.
   • The skip dropped records takes a fresh job id; it used to take next_job as it stood, which can be a job's id.
4. CHANGELOG: rows: the write worker charges a block in flight once, and a save that fits the staging permit waits instead of being skipped #1548 under Unreleased.

docs/api.md describes both, including which saves are still skipped.

Tests

Each new test of changed behaviour fails before its fix, checked by reverting that one piece; the larger-than-permit,
eviction and fence tests guard behaviour that is unchanged:

• Charged and gathered once: rows_stage_test.zig:36 fails with expected 30256, found 37360 (the bytes charged
  against the bytes that reached disk). The test also reads the second state back.
• complete() removed from publish: "a state whose earlier job failed to write its block is not published" fails at
  rows_stage_test.zig:57 (expected 0, found 1).
• The promised set matching the blocks a state names instead of the files a job writes: the chain test "a block its
  writer failed to write is written by the next save, though a job that skipped it is in flight" fails at
  rows_stage_test.zig:93, and the promised test at rows_io.zig:372.
• The worker back to slot order: rows_io.zig:355 ("jobs are run and taken in the order they were staged, whatever
  slot each holds").
• Waiting saves: "a full byte permit holds another save that fits it alone until the first job is written" and "the job
  permit holds a ninth small save behind eight blocked writes until one is written" (rows_async_test.zig:365 and
  :388; they used to assert the skip). Also: a newer save that fits is staged ahead of one that waits; a close
  writes the saves that waited; a save larger than the permit and an eviction against a full permit are still
  skipped; a waiting save whose entry leaves RAM is counted as skipped (rows_stage_test.zig:236 with the count in
  dropped removed).
• Two waiting saves (ashhart's test): losing the older one counts it (rows_stage_test.zig:266, expected 1, found 0
  without the count in dropped; the test as posted fails the same way on commit 2). A save that waited, dropped after a reap and after a newer save was staged: not counted
  without the mark (:303), and recorded under the staged job's id without the fresh id (:305).

Host (Zig 0.17.0, an M4 Mac), on the branch's head after the rebase onto main 36470cecc: test-rows-host,
test-rows-prefetch-host, test-learn-recovery, test-cuda-host, test-snapshot-index-host, test-lane-api-host
and test-golden pass (288/288 steps). zig fmt --check and git diff --check are clean, and lean_check.py reports
425 on main and on the branch.

GPU: two GB10s at TP=2

These runs predate the rebase and commit 3, which changes only how a lost waiting save is counted and recorded.

GLM-5.3-Flash on CUDA is the family whose saves showed this (#1538's evidence), and its disk store is #1411. So the run
used a local merge (657c0b43c, not pushed) of this branch onto #1411's head fbea020a2, against #1411 alone. Mia's
EXL3 4bpw checkpoint, MTP drafts, --context 131072 --parallel 8 --prompt-cache-gib 6 --learn-conversations --learn-gib 30 --learn-min-free-gib 15. The cells marked (#1411's run) are from #1411's own run on the same setup.

Case	#1411 alone	#1411 + this branch
A's two states (16,384 and 18,432 tokens) after turn 2	the 18,432-token save skipped (StagingFull) (#1411's run)	both written: 539 MiB, then 377 MiB (only its new bytes)
A's next turn after two fillers evict it	resumed from 16,384 tokens, first token 2.6 s (#1411's run)	read the 18,432-token state, 564 MiB in 466 ms, first token 0.96 s, same token_sha
The fillers' pairs (53,248 and 55,296 tokens)	the 55,296-token save skipped (#1411's run)	both written; the second waited, then wrote its new 416 MiB
Saves over the whole run	(this run of #1411 alone was the decode cell only)	18 written, 0 skipped

Decode while saves stage. A 1,024-token reply, decoded alone and decoded right after two new 20k-token prompts
finish, so their saves stage between its rounds. Three times each, alternating the order, every reply to
max_tokens:

Tree	Alone (tok/s)	Right after two 20k prompts (tok/s)	Saves in the cell
#1411 alone	24.12 (23.90-25.79)	24.12 (23.94-24.18)	9 written, 3 skipped
#1411 + this branch	24.56 (23.94-25.00)	24.51 (24.49-24.72)	13 written, 0 skipped (one is the second filler's waiting save)

Medians, with the range in parentheses. Of the branch's 13 writes, 12 come from the cell's prompts; the other is the
second filler's 55,296-token save, which had waited and was written during the first alone rep. The three saves #1411
skipped, written here, leave decode where it was: -0.2% against +0.0%, inside the spread between runs. Rank 1's logs
have no errors.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)

[ashhart]

Thanks @wojo. We replayed this on main (7c0688951), and both fixes work: the PR's seven host suites pass (4,335 tests), and main fails your two issue cases, as it should. We found one more case before we can land it:

Losing the older of two waiting saves isn't counted. When a second save also has to wait, zig/src/core/rows_write.zig:241 overwrites Rows.waiting with it. dropped (:36) only checks that one pointer. So if the prompt cache evicts the older waiting entry, its queued save is removed without forget, and it's never counted as skipped or recorded as StagingFull. This test fails on the PR (expected 1 skipped, found 0):

//! A queued save that already waited still counts as skipped when a newer waiting save takes the mark.
const std = @import("std");
const rows = @import("rows_store.zig");
const fixture = @import("rows_async_fixture.zig");

test "losing the older of two waiting saves counts it as skipped" {
    const x = try fixture.Fixture.open("two-waiters", .off, false, 1 << 20);
    defer x.close();
    var hold: fixture.Gate = .{};
    defer hold.release.store(true, .release);
    x.r.file_io = hold.io();
    const first = fixture.prompt(41);
    try x.keep(&first, 600);
    try fixture.Gate.wait(&hold.entered);
    x.r.worker.?.limit = x.r.worker.?.bytes();
    const second = fixture.prompt(42);
    try x.keep(&second, 600);
    const older = x.r.waiting.?;
    const third = fixture.prompt(43);
    try x.keep(&third, 600);
    try std.testing.expectEqual(@as(usize, 2), x.r.queue.items.len);
    try std.testing.expect(x.r.waiting.? != older);
    for (x.s.entries.items, 0..) |e, i| if (e == older) {
        x.s.remove(i);
        break;
    };
    try std.testing.expectEqual(@as(usize, 1), x.r.queue.items.len);
    try std.testing.expectEqual(@as(u64, 1), x.r.counts.skipped);
    try std.testing.expectEqual(error.StagingFull, x.r.failures.records[0].?.err);
}

One way to fix it is to mark each queued save that has waited, and have dropped check the queue for that mark. waiting would then only be the newest blocked marker that stops the worker spinning. Also, could you add a line under ## Unreleased in CHANGELOG.md (1.1.9 is now released, so it's a new section at the top)? If you'd rather we add the fix and the line on your branch, say so and we will, keeping you as the author.

[wojo]

Thanks @ashhart, good catch. Fixed on the branch as you suggested, and rebased onto main (36470cecc): with #1543 in, rows_fake.zig conflicted, and the two new fake fields now sit side by side.

• b0604379c: a queued save carries its own mark, Queued.waited. stage puts a save back with it set, and dropped counts the save it removes as skipped (StagingFull) when it has waited, unless the store has stopped. Rows.waiting is now only the newest save refused room, which keeps step from spinning, and dropped clears it when its entry leaves. forget is gone. Two things follow:
  • A save that waited and is dropped after a reap, before step tries it again, now counts too. It did wait, and it was lost unwritten. On main it was counted when it was refused, so the totals match main's.
  • The skip dropped records now takes a fresh job id. It used to take next_job as it stood, which can be the id of a job staged after the save waited, so two failure records could share an id.
• Tests:
  • Your test is in rows_stage_test.zig, with three more checks:
    • the newer save is still Rows.waiting;
    • its queued save is marked;
    • it is written once the job in flight finishes.
  • A second new test drops a save that waited, after a reap and after a newer save was staged. It fails without the mark (not counted) and without the fresh id (the record takes the staged job's id).
• 2a492dd06: the line goes at the top of ## Unreleased. It and docs/api.md now say which saves are still skipped: one larger than the permit, a waiting save whose state leaves RAM unless --spill-gib copies it then, and that copy when the permit is full.

Host (Zig 0.17.0, an M4 Mac), on 2a492dd06:

• The seven suites pass (288/288 steps).
• zig fmt --check and git diff --check are clean.
• lean_check.py reports 425 on main and on the branch.

I've updated the description to match.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)

[wojo]

@ashhart The one red check on 9e3a32db5 looks like a flake. In the desktop job, 854 of 855 tests passed. The failure is electron/agent/engine/hub-endpoint.test.ts, which could not remove its temp directory at cleanup (ENOTEMPTY on rmdir).

This PR changes only zig/, docs/api.md and CHANGELOG.md. The same job passed on main d33dcfc76 and on this branch's previous head 2bd384352, which has the same code on the older base. The other eight checks pass, including zig-windows. Re-running the desktop job should clear it; I can't re-run it from the fork.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)