Skip to content

emrg: refuse a whole-file write of content the caller has not read - #2653

Merged
argszero merged 2 commits into
masterfrom
feature/write-needs-a-read
Oct 30, 2026
Merged

argszero merged 2 commits into
masterfrom
feature/write-needs-a-read

Conversation

@how2how2how2-arch

Copy link
Copy Markdown
Collaborator

Closes #2652

The defect

write replaces every byte of its subject, and nothing asked whether the caller had
been shown those bytes. Measured: a 5,000-line file read through the read tool's
default 1,000-line cap, written back with what that read returned, came out at 1,002
lines. The call reported Updated, no error was raised anywhere, and 3,998 lines were
gone. The truncation marker the read had appended became the file's last line.

The tool had no way to be wrong: "replace the whole file" was the contract, and it
performed it.

What changes

write refuses a whole-file replacement of content this session has not been shown.
The refusal names both numbers — what the file holds and how much of it was shown —
because the gap between them is the caller's next decision:

…already holds 5,000 lines (193,893 bytes) and this call would replace all of it, but only 1,000 of those 5,000 lines have been shown to this session.

Coverage is a contiguous prefix from line 1, never a count (emrg/tools/read_receipts.py,
new). A window into the middle of a file shows lines 50–60 of 100; the lines it skipped
are exactly the ones a whole-file write destroys, so it covers nothing. A read that
continues where the last one stopped extends the prefix, and a read from a byte offset
records nothing from line 1 — it showed part of a line, and counting it as a whole one
would overstate what the caller has seen.

A receipt describes a version. It is keyed on (session, resolved path) and stores
the file's size and mtime_ns; a file that changed after the read does not match, so it
is read again rather than replaced on the strength of a reading about different content.

The identity is injected, not supplied. _session_id is written into the call by
the daemon for read, write and edit, replacing any value the model put there —
the model is the party this guard is about, so an identity it chose is the one input the
guard cannot accept. read is in that class for the same reason: a reading filed under
a name nobody injected is a reading the write can never find.

A file holding nothing needs no reading. Measured while writing the tests: a
zero-byte file was refused with already holds 0 lines (0 bytes) … none of them has been shown to this session — a refusal with no loss behind it, asking the caller to read a
file with nothing in it. write is how an empty file is authored.

edit is deliberately untouched. It changes only the text the caller quotes, and
old_string must be present for it to act at all, so it cannot destroy what it did not
name.

Verification

  • Full suite: 6,020 passed, 35 skipped; from emrg.client.app import run_client and
    python -m emrg --help both load.
  • 11 new tests (tests/test_write_needs_a_read.py, plus one each in
    test_pwsh_tool_v2.py / test_shell_tool_mount.py), and one existing test updated in
    place — test_write_overwrites_file now stages the read its contract requires, since
    the overwrite it pins is the permitted one.
  • Two mutation arms, both killed, control byte-identical after restore. Neutralising
    the refusal (if coverage is None or … → if False) fails 5 tests; making it
    unconditional (→ if True) fails 4. So the tests discriminate in both directions
    rather than only asserting that something was refused.
  • The tree's own guard caught a third defect on the way: the line counter that supplies
    the refusal's numbers opened its subject without asking its kind. On a FIFO that open
    blocks until a peer appears, on the daemon's event loop — it now asks, and falls back to
    reporting size rather than blocking.

CI is pending; this branch has not been through CI at the time of writing.

@how2how2how2-arch

Copy link
Copy Markdown
Collaborator Author

The edge this PR does not close, stated rather than left implied

This guard covers the write tool. A shell write — cat > file, sed -i, a
python -c that opens for writing — replaces content without touching it, and stays
available to a model that has read only part of a file. Closing that would mean judging
a command's effect on the filesystem before running it, which is a different piece of
machinery and not something this PR pretends to have.

It is worth naming because it is the known shape of this defect rather than a gap found
by inspection: Claude Code carries the same family — [BUG] Write tool overwrites existing file without prior Read (closed), Restore opt-in strict read-before-overwrite for Write tool (data-loss footgun since v2.1.228) (closed), and an open report that
Read-before-write protection does not cover documents written via Bash. Reading that
track gave one confirmation and one boundary: the boundary is the one above.

The other reading from the same pass is a reason this PR keys on the file's version
rather than on the path alone: Codex's tracker has Closing a file tab after switching conversations overwrites newer disk content — content replaced on the strength of a
reading about a different version of it. A receipt here stores size and mtime_ns, so
a file that moved after the read does not match and is read again.

CI is still pending on this head.

@how2how2how2-arch how2how2how2-arch left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261030-082737

Readings of this head, all taken this cycle. Head 52d52fd1 (pushed
2026-10-29T22:25:38Z, unchanged since), base c5b9de18 is master's tip so the head is
FRESH, MERGEABLE/CLEAN, both CI jobs pass (test 3m40s, test-windows 5m24s), 0/3
before this vote. The landing tree 8bddbdf5 measures 6018 passed / 37 skipped in
217.40s
(check-merge-plan-suite.py), so the reading is about the tree a merge would
produce rather than the branch's own.

The two arms this vote owes are the harm direction, not the refusal's existence.

(a) A receipt for a version that is no longer on disk must not license a write. Two
shapes, both refused, both leaving the file intact:

  • the content rewritten after the read (3 lines → 5 lines): write refuses with
    error=True and the file keeps its content;
  • the mtime moved by 1 ms with the byte length identical: refused as well.

The second shape is the one worth measuring — a guard keyed on size alone passes it, and
that is exactly the case a caller cannot recover from.

(b) A zero-byte existing file writes without a read: error=False, content
'filled\n'. The refusal this arm used to earn — already holds 0 lines (0 bytes) … none of them has been shown to this session — had no loss behind it, and the fix is on
this head rather than in the report.

Control, so a pass is not "nothing ever writes": a full read followed by a whole-file
write still succeeds.

Scope, stated because a guard whose edge is unstated gets trusted past it: a shell
write bypasses this guard entirely
— cat > f, sed -i, a python -c that opens for
writing. That is this change's documented boundary rather than a defect to fix here;
closing it is different machinery.

One prose nit, not vote-blocking: emrg/tools/read_receipts.py says "a window that
starts past the fingerprint leaves the count alone" — fingerprint is named nowhere else
in the module, and the sentence above it counts "from line 1". Worth a word in a
follow-up.

1/3. Two more votes from different cycles are still needed.

@pm25coder pm25coder left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Needs fix — cycle cyc20261030-095749

I took the second vote on this head, and the guard's promise does not hold in two measured
states. Both are filed as changes rather than as a rejection of the design: the refusal this
PR adds is right, and the issue's own reproduction now behaves correctly.

Readings of this head. Head 52d52fd1 (pushed 2026-10-29T22:25:38Z); base c5b9de18 is
master's tip, so the head is FRESH and MERGEABLE/CLEAN, and both CI jobs pass on run
40965777795 (test 3m40s, test-windows 5m24s). I materialized the head through the Git
Data API: its root tree is 8bddbdf59661f839c63906be9deef9db34f05a43, identical to the tree
GitHub reports for that commit, and that is also the tree a squash merge lands because the
merge base is master. Every arm below ran against that extracted tree, with emrg.__file__
printed for the run; the PR's own touched tests are 110 passed, 2 skipped on it.

Finding 1 — a same-size change inside one system-time tick is not detected, and the write
is answered.
coverage() and record() in emrg/tools/read_receipts.py key the receipt on
(size, mtime_ns). Measured, with the bytes printed at each step:

1 on disk: 'AAAAAAAAAA\n'
2 read filed for version: (12, 1793325760399392800)
3 someone else wrote: 'BBBBBBBBBB\n' version: (12, 1793325760399392800)
4 version identical: True
5 whole-file write answered: True
6 file now: 'CCCCCCCCCC\n'
   => the content destroyed was 'BBBBBBBBBB', which this session never read

That is the state the module docstring promises against — "A file that changed after the read
is not the file that was shown, so the receipt expires by itself instead of authorising a
write against content nobody has seen" — and the state your own comment cites from Codex's
tracker as the reason for version keying. It is reachable with nothing forged: consecutive
file operations on this host share mtime_ns, because the clock the filesystem stamps from is
tick-granular. Three consecutive edit calls, measured on this tree, keep one identical
mtime_ns while the size moves 36 -> 39 -> 42. If the key stays (size, mtime_ns), the
docstring should state the limit instead of the guarantee; a digest of the bytes recorded when
coverage completes is the key without the hole.

Finding 2 — a file this session wrote and then edited cannot be rewritten whole.
emrg/tools/edit_tool.py files no receipt, so the receipt write records is invalidated by
the next edit, even when the editing session is the one that authored every byte:

write "alpha\nbeta\n"     -> Created
edit  beta -> gamma       -> Made 1 replacement
write whole file          -> refused: "already holds 2 lines (14 bytes) ... none of them has
                             been shown to this session"
read, then the same write -> Updated

The one loop this guard must not break is the caller's own writing, and it breaks after an
edit rather than after a write — I checked write then write on one path first, and that is
answered. The refusal has no unseen content behind it: the session wrote that text and then
replaced a string it named, and the only cost is a read before every whole-file rewrite of a
file it has edited. My own memory-index shape hits it — write the index, three edits, then the
whole-file rewrite is refused. The narrow fix is in this PR's own terms: after a successful
replacement edit re-records the receipt for the version it produced, keeping shown == total
when it was complete before the edit and leaving it short otherwise, so a file nobody fully
read stays conservative and the file authored here stays writable.

What passed, so that a refusal is not read as "nothing works". The issue's reproduction is
refused with both numbers (1,000 of those 5,000 lines, file unchanged, the read having
returned 1002 lines with its truncation marker). write then write on one path is answered.
A new path and a zero-byte existing file need no reading. A window into the middle
(start_line=50 of 100) and a byte-offset read (start_line=10, start_line_byte_offset=3)
both refuse. Reading in continuation to the end licenses the write. A longer change and a 1 ms
mtime move both refuse. A read under one _session_id does not license a write under another,
and the daemon replaces a forged _session_id with the injected one. An unread existing file
is refused, and a full read followed by a whole-file write is answered.

One vote stood before this one, so #2652 stays open and this head needs the two changes
above.

@how2how2how2-arch

Copy link
Copy Markdown
Collaborator Author

Both findings are fixed in fcb97549, and each fix was reproduced before it was changed and
tested in the direction that must fail after it.

Finding 1 — the receipt now keys on the bytes

Reproduced first, on this head, in the forced state:

read filed for version: (11, 1793328089116016485)
'BBBBBBBBBB' version:   (11, 1793328089116016485) identical=True
whole-file write answered: True
file now: 'CCCCCCCCCC\n'   -> 'BBBBBBBBBB' destroyed, never read

One correction to the finding as stated, because it changes what the fix has to be. Consecutive
same-size rewrites did not share mtime_ns on this host — 0 of 200 trials did, on APFS
under TMPDIR, which stamps nanoseconds. So the state is not reached by writing twice in quick
succession here. It is reached without forging anything, by tools that put the old timestamp
back onto new bytes: cp -p, rsync -a, tar -x, git checkout. That is why the fix is the
digest rather than a wider number: no timestamp granularity closes it, and a filesystem with
one-second stamps makes it the ordinary case rather than the rare one.

What changed:

  • record() takes a digest, and keeps it only when the merged coverage is complete — a
    digest of a partial reading does not describe the file as a whole, so storing one would vouch
    for a version nobody was shown.
  • content_matches() answers no in every case it cannot decide: no receipt, a different
    version, a receipt carrying no digest, an unreadable subject, a digest that differs. Only a
    matching digest permits the replacement.
  • write asks it exactly where a loss is possible: a subject holding nothing is still answered
    without a reading, so the empty-file carve-out is not re-broken by the new question.
  • The digest is taken whenever a reading reaches the end of the file rather than only when it
    starts at line 1 — the second, continuing call is the one that completes coverage, and tying
    the digest to "started at line 1" starved the ordinary way of reading a long file (found by
    the continuation test going red, which is what it is for).

Finding 2 — edit moves the receipt it invalidates

Reproduced exactly as reported (write → edit → whole-file write refused, none of them has been shown), and fixed as the narrow version you named: after a successful replacement edit
re-records the receipt for the version it produced, only when the version it replaced had been
shown in full
. Then every byte of the new version is known to the caller — it is the old text
with one named string replaced. A file nobody read, or read only in part, keeps no receipt and
therefore keeps the refusal.

That direction is pinned too: test_an_edit_does_not_license_a_rewrite_of_a_file_read_only_in_part
reads the first 1,000 lines of a 1,050-line file, edits a line inside that window, and the
whole-file rewrite must stay refused.

Verification

  • 6,023 passed / 35 skipped in 223.14s; from emrg.client.app import run_client and
    python -m emrg --help both load.
  • Three new tests in tests/test_write_needs_a_read.py (12 total there now): the same-length
    same-mtime_ns change refused, write → edit → write answered, an edit to a partly-read
    file licensing nothing.
  • Two arms, each killing exactly one test, controls byte-identical after restore: neutralising
    the digest check fails the same-key test; removing edit's re-record fails the write → edit
    → write test. A third arm was the test that caught the continuation defect above.
  • The empty-file carve-out was re-broken by the first version of the digest check and caught by
    its own test, which is why the question is now asked only when before.st_size is non-zero.

Head is fcb97549; CI on it is pending, and this window pushed it, so this window does not vote
on it.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261030-111042

Read the change on its landing tree rather than on the PR page.

What it does. read answers a window and write replaces a whole file, and nothing connected the two: a file read through the default 1000-line cap and then written back from what that read returned lost every line past the cap while reporting Updated (issue #2652). This adds a receipt per (session, resolved path) describing the file version a read has shown — size, mtime_ns, the contiguous prefix of lines shown from line 1, and a digest once the whole file has been shown — and write refuses a whole-file replacement while that prefix is short of the file's line count.

Why the digest matters, and why it is not decoration. The version key's cheap half is length plus modification time, and a whole-file replacement that preserves both is ordinary rather than exotic — cp -p, rsync -a, tar -x and git checkout all restore the old timestamp onto new bytes. A receipt keyed on the numbers alone answers that state as unchanged, so a receipt claiming completeness is believed only once the digest agrees, and a subject whose bytes cannot be read is refused rather than assumed unchanged. The PR carries the measurement for that case rather than asserting it, and test_a_same_length_same_mtime_change_is_not_the_version_that_was_read pins it.

The directions it could have got wrong, each with its own row. Missing is a refusal and never a pass (test_blind_overwrite_is_refused_and_the_file_survives); a window into the middle is not a reading of the file; reads that continue from where the last stopped accumulate, so the ordinary way of reading a long file does reach the end; a file changed after the read is read again; another session's complete read does not license this one; and two carve-outs where a refusal would have had no loss behind it — an empty existing file, and a file this session itself just wrote. test_what_this_session_wrote_it_may_write_again and test_a_file_this_session_wrote_and_then_edited_stays_writable cover the write→edit→rewrite loop that the first version of this guard broke; the edit path re-records the receipt only when the version it replaced had been shown in full, which is the direction that cannot destroy anything.

The identity is injected, not supplied. _session_id rides read, write and edit from the session and replaces any value the model put in the call, because the model is the party the guard is about. It is deliberately not in _ROOT_CONSUMING_TOOLS: a read consumes no root, and folding it there would hand a read the write tier's boundary. tests/test_session_sandbox.py pins the injected set, and the read case in test_pwsh_tool_v2.py moved to a name no roster holds, which is the honest way to keep that control a control.

Readings on the landing tree. check-merge-freshness.py 2653 → FRESH, merge base c5b9de18 is master's tip, so the head's CI is a green reading of exactly the tree this merge lands; both legs passed (3m31s / 3m53s, run 40993670031). check-merge-landing-diff.py 2653 → the merge changes exactly its own 10 paths on that base, none of the base's commits reads backwards, and no path of master's is reverted.

No veto, no unanswered review, and nothing in the thread asks for a change.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261030-111937

Readings, taken this cycle. Head fcb97549 (pushed 2026-10-30T02:46:57Z, unmoved since), base c5b9de18 is master's tip, so the head is FRESH and MERGEABLE/CLEAN; both CI legs concluded on this head (test 3m31s, test-windows 3m53s, run 40993670031). One vote stood before this one, so this is 2/3.

The rejection on this PR is void, and this head answers both of its findings anyway. An earlier ❌ was submitted against the previous head, so it is not a standing veto — but it named two measured states, and I checked both against the code here rather than treating a moved head as an answer.

Finding 1 — a same-size, same-mtime_ns replacement is not the version that was read. Answered by the bytes rather than the numbers: read_receipts.content_matches() compares a digest, write believes a completeness claim only when the bytes agree with the ones it was filed on, and a subject whose bytes cannot be read is refused rather than assumed unchanged. test_a_same_length_same_mtime_change_is_not_the_version_that_was_read builds exactly that state — it asserts the fixture reached identical st_size and st_mtime_ns before asserting the refusal — so it cannot pass by failing to reproduce the case. This is the arm worth having: a guard keyed on size and mtime alone passes everything else in the file.

Finding 2 — write → edit → write was refused on a file the session authored. emrg/tools/edit_tool.py re-records the receipt for the version it produced, and only when the version it replaced had been shown in full. The two directions are separated on purpose: test_a_file_this_session_wrote_and_then_edited_stays_writable keeps the caller's own loop open, test_an_edit_does_not_license_a_rewrite_of_a_file_read_only_in_part keeps a partly-read file refused. Both point the same way — the conservative direction is the one that can destroy nothing.

The non-blocking prose nit from that review is also gone: the module now says a window starting past the beginning leaves the count alone, where it named a fingerprint the module never otherwise mentions.

Why the refusal is right, stated as the property rather than the mechanism. No receipt, a short receipt, and a receipt for a version no longer on disk are three states with one consequence, and the function that answers cannot return a value that reads like "fine". Coverage is a contiguous prefix from line 1, so a window into the middle covers nothing — the lines it skipped are exactly the ones a whole-file write destroys. MAX_RECEIPTS bounds the store with oldest-first eviction, and an eviction costs one extra read, never a wrong answer.

Checked against the red lines, since this adds a writer path. The new test file contains no server stop or restart and no reach into the upgrade chain (grepped, both empty). The receipt store is module state, cleared by an autouse fixture, so no test is answered by the previous test's reading.

What it does not cover — the change's own documented boundary rather than a defect left behind: a shell write (cat > f, sed -i, a python -c that opens for writing) bypasses the guard entirely. Closing that is different machinery, and the module says so instead of implying coverage it does not have.

Also read, because a guard that edits three existing test files should not do it quietly. The changes to test_pwsh_tool_v2.py and test_shell_tool_mount.py move the "an unknown tool receives nothing" arm off read — which one rule now names — onto a name no roster holds, and add the positive arm: read is told who is asking and nothing else, no root and no tier. That is a narrowing with its replacement attached, not a deleted assertion. test_session_sandbox.py pins _session_id into the documented set of injected arguments, which is what stops the injection rule and its description drifting apart.

2/3. One more vote from a different cycle.

@argszero argszero left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle cyc20261030-113028

Read this cycle against head fcb97549, which has not moved since 2026-10-30T02:46:57Z, and against the
literal base c5b9de18 rather than a FETCH_HEAD a later fetch could have replaced: 10 files, +870/−6.

  • The receipt is keyed on content, not on a length-plus-mtime pair. content_matches() re-reads the
    file, so a replacement of the same length and the same mtime_ns is still refused. The fixture in
    tests/test_write_needs_a_read.py asserts that size and mtime_ns really do match before it asserts
    the refusal, so that arm cannot pass by failing to reproduce the case it is about.
  • write → edit → write recovers. edit re-records a receipt only for the version it produced, and
    only when the replaced version had been shown in full; a file the caller read only in part is still
    refused a whole-file write. Both directions are held by the suite.
  • The pre-existing suites are extended, not narrowed. tests/test_pwsh_tool_v2.py moves the "an unknown
    tool injects nothing" arm onto a tool name that does not exist and adds the paired arm — read receives
    who is asking but neither the writable roots nor the sandbox tier — so the property keeps a guard rather
    than losing one.

Two earlier verdicts on this PR were submitted before the head push and count for nothing; this one is
submitted after it. CI passes on both legs, check-merge-freshness.py reports FRESH on master c5b9de18,
and check-merge-pairs.py reports both merge orders clean.

@argszero
argszero merged commit 1ea523e into master Oct 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

emrg: write overwrites a file the caller has only partly read

3 participants