Skip to content

fix(data): reject destructive JSON store shrink - #1461

Closed
mikemikimike wants to merge 8 commits into
micro:mainfrom
mikemikimike:fix/protect-json-store-shrinks
Closed

mikemikimike wants to merge 8 commits into
micro:mainfrom
mikemikimike:fix/protect-json-store-shrinks

Conversation

@mikemikimike

@mikemikimike mikemikimike commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

Related issue

Related to #1458

Background

A whole-file JSON store can be overwritten by a stale or partial in-memory view. The existing shrink check preserved a .prev copy but still replaced the live store, so the running instance could immediately lose the data it needed.

Changes

  • Reject destructive JSON store shrinks instead of replacing the live file.
  • Fail closed when the recovery copy cannot be read or written.
  • Serialize the shrink-check, recovery-copy, and live-file replacement sequence within the process.
  • Extend regression coverage for failed recovery copies and preserved live contents.

Compatibility

Normal writes and legitimate changes below the shrink threshold are unchanged. Callers that attempt an unexpectedly destructive JSON write now receive an error and must repair or explicitly handle the operation.

The mutex protects concurrent writers in one process. Cross-process writers and equal-sized stale snapshots would require a file-lock or versioned compare-and-swap protocol and are outside this focused change.

Verification

  • GOPROXY=https://goproxy.cn,direct go test ./internal/data -run 'TestAWriteThatLosesMostOfAStoreKeepsACopy|TestSaveJSONRejectsShrinkWhenPreviousCopyCannotBeWritten|TestSaveJSONPreservesPreviousContentsOnMarshalFailure|TestSaveJSONRoundTripLeavesNoTempFiles' -count=1 — passed.
  • gofmt -w internal/data/data.go internal/data/atomic_test.go — passed.
  • git diff --check — passed.
  • GOPROXY=https://goproxy.cn,direct go test ./internal/data -count=1 — not green on Windows because existing tests expect Unix file modes (0600) and SQLite cleanup encounters a locked file.
  • GOPROXY=https://goproxy.cn,direct go test ./... — not green on Windows due existing platform-specific syscall.Statfs_t/syscall.Statfs build errors, file-lock failures, permission-mode failures, and proxy environment assumptions.

No Docker service is required for this unit-level change.

@mikemikimike
mikemikimike force-pushed the fix/protect-json-store-shrinks branch from fbe6bf8 to 4b3fd75 Compare August 23, 2026 16:44
asim pushed a commit that referenced this pull request Aug 31, 2026
keepIfShrinking exists because the wallet was destroyed by two components each
saving their own stale view over the other's — so when a write is about to halve
a store, the old contents are copied to .prev first. Every failure on the way to
that copy returned silently and let the write through, which is the one case
where the guard was needed and the one case it did nothing: the store shrank,
the copy was not there, and nothing said so.

It refuses the write now. Not being able to take the safety copy is exactly when
not to do the dangerous thing, and a caller that gets an error can be told;
a caller whose data is gone cannot.

And the check is serialised with the write. Stat, read, copy, replace was four
steps with no lock, so two concurrent shrinking writers could interleave and the
copy could be of a file the other had already replaced — the .prev being written
to protect against a lost store would hold the lost store.

The existing test only asserted the copy was kept; it now also asserts the write
that triggered it actually landed, so "preserve the old contents" cannot quietly
become "refuse everything".

Co-authored-by: mikemikimike <186855910+mikemikimike@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P2r4ca9UPPf7FDk7y8eJLr

asim commented Aug 31, 2026

Copy link
Copy Markdown
Member

Merged as a345e79 — thank you, this is a good catch.

The argument that convinced me is that every failure on the way to the copy returned silently and let the write through, so the guard did nothing in exactly the case it exists for: the store shrank, .prev was not there, and nothing said so. Refusing is right — a caller that gets an error can be told, a caller whose data is gone cannot.

The saveMu half is the part I might not have spotted. Stat, read, copy, replace with no lock means two concurrent shrinking writers can interleave and the copy can be of a file the other has already replaced, so the .prev written to protect against a lost store would hold the lost store.

Carried in by hand rather than through the merge button, because the branch was several days behind and I wanted to run the full suite against current main — your commit is credited as co-author. Two notes, neither blocking:

  • saveMu is global, so every JSON store write in the process now serialises on one mutex. json.Marshal is outside it, which is the expensive part, and writeAtomic is a temp write plus a rename — fine at this scale, but a per-key lock is the shape if it ever shows up in a profile.
  • Your addition to TestAWriteThatLosesMostOfAStoreKeepsACopy is the one I'd have asked for: it stops "preserve the old contents" quietly becoming "refuse everything".

Generated by Claude Code

@asim asim closed this Aug 31, 2026

asim commented Nov 17, 2026

Copy link
Copy Markdown
Member

@mikemikimike — thank you for your contribution to Mu. We're requesting your explicit permission to relicense your JSON-store shrink-safety contribution in #1461, incorporated in commit a345e79, including the portions retained or modified in later Mu versions.

The proposal is in #2180: future Mu runtime releases would be source-available, free for personal/internal use, with a paid licence required for commercial resale or hosting. Micro would also offer separate proprietary commercial licences. MUCP and the SDKs remain permissively licensed. Existing AGPL grants would remain valid.

If you agree, please reply with the following confirmation (identifying the rights-holding entity if you are acting for one):

I hold the relevant rights, or am authorised by the rights holder, in my contribution identified above. I grant Micro XYZ Ltd a worldwide, perpetual, irrevocable, non-exclusive, royalty-free copyright licence to use, reproduce, modify, distribute and sublicense that contribution and derivative works as part of Mu under the proposed Mu Runtime Licence and under separate proprietary commercial licences, including granting customers rights to modify, redistribute and provide hosted services. This is an additional permission; it does not revoke existing AGPL grants or transfer my copyright.

This request concerns only the identified contribution, not future contributions. Attribution will be preserved. If any portion belongs to an employer, co-author or another party and you cannot grant permission for it, please identify that limitation instead. If you would prefer a separate agreement or do not agree, please say so; silence will not be treated as consent.

For reference, the current draft is Mu Runtime Licence draft 1.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants