From 526d9bcfe415e2c313441cf4d32e7f85a24151a3 Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 14:20:50 -0600 Subject: [PATCH 01/11] Preserve accepted metadata on implicit durable re-inserts Snapshot accepted row metadata for durable inserts that omit explicit metadata writes. Keep explicit metadata writes authoritative and avoid attributing the preserved metadata to the source. Share completed-reach classification between the controlled and native SQLite receiving drivers. Law: Sync metadata session decisions4,5,6,8,9,14,15,18 in docs/contributing/oracle-reviews/2026-10-22-sync-metadata-session.md, the implicit row-metadata rules, and glossary hydration-seed contract. An inner equal re-insert preserves accepted metadata through persistence and peer reload. Reach: Each controlled/native host campaign completes40 histories in each of three lanes. Fixed seed527962096 reaches2 completed bare-key1 implicit keeps and2 completed non-bare keeps per lane, with21 named/19 fullreload notices. RED: M01 controlled extended owner fails unchanged parent production at seed2281001/path7:0; M06/M07 native SQLite receiving replays/property fail the same metadata law, generated path7:0:0. GREEN: M08/M09 paired40-run campaigns and receiving replays pass. H02 repeats both hosts with the shared helper; H01b proves final-helper equivalence over 2304 histories/102912 comparisons. Final types/lint/format pass. Mutant: Original durable-metadata map and always-delete rival both fail the shared metadata comparison in controlled/native generated histories. An all-explicit-inner grammar ablation fails the completed implicit-keep reaches. Limits: One process/native SQLite with two production coordinators. No OPFS, Electron IPC, transport-loss/leadership-turnover or broader2272 claim. Random scarce reach is measured; fixed guards enforce both scarce branches. Host runs precede only an explicit terminal undefined return, linked by exact equivalence. --- docs/contributing/oracle-coverage.md | 52 +- .../src/persisted.ts | 14 +- .../sync-metadata-session-persisted.test.ts | 211 +++++--- .../sync-metadata-session-sqlite.test.ts | 462 ++++++++++++++++++ packages/db/tests/oracle-config.ts | 2 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../db/tests/sync-metadata-session-oracle.ts | 111 ++++- 7 files changed, 778 insertions(+), 81 deletions(-) create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index afa469b5b0..e63484720e 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4422,24 +4422,50 @@ must equal the model's, which a restart would read. `persisted-peers-full` repeats the lane under full row updates, where an update replaces its row in storage too; `persisted-peers-on-demand` repeats it on demand with a demand for every row. A reset over metadata of a missing -row is too rare in generated histories to declare; a replay pins it. A -random CI campaign (seed `654774679`) found a `persisted-peers-full` history -that reproduces on `main`: an outer transaction inserts over a bare seed, an -inner one re-inserts the row and commits, and a peer's full reload leaves the -row with the cached seed's metadata, where the model expects none (law: -the source reads its row metadata). The maintainer chose to leave it open, -tracked with #2281; an `it.fails` replay through the same model, driver, and -comparison pins its shrunk prefix, and the random campaign can still draw it. -Owner: this oracle. A reset +row is too rare in generated histories to declare; a replay pins it. Durable source equal re-inserts preserve accepted row metadata, including absence +or metadata supplied by a later DbClient hydration seed, without making the keep +a source metadata write (decisions 9, 14, and 18). The constructive +`seedReinsertReloadHistoryArbitrary` and ordinary original replay use the shared +independent model and per-action source metadata comparison. Controlled +partial/full/on-demand lanes and native SQLite receiving lanes each pass fixed +40 and random 40 generated histories, with actual named/full-reload notices in +every native lane. Final fixed seed `527962096` independently reaches 40 inner +commits, 21 named notices, 19 full reloads, 2 completed bare-key1 implicit keeps, +and 2 completed non-bare implicit keeps per lane on both hosts. Every normal run +enforces those completed fixed-branch guards; random branch counts are measured +without a guarantee of reach. Controlled final random seed `-704889444` measures +0 bare/4 non-bare; native `1784284659` measures 2 bare/1 non-bare. Older +commit/notice guards still apply to both campaigns. Original RED and direct +replay seed `2281001` remain preserved. Original-defect and +always-delete-kept-metadata mutants both fail the shared product comparison +through both drivers. An all-explicit-inner grammar ablation passes 40 product +comparisons per host but fails the new completed-branch guards, with all hashes +restored. Native on-demand RED can occur at inner commit before the explicit peer +notice because actual coordination reads storage sooner; eager partial/full RED +reaches explicit peer step 6. + +The completed-branch guards were added in independent review after the first +production edit. M16 revealed zero bare-key1 reach at fixed seed `2281001`, then +measured seed `527962096` supplied both premises. The bare conjunction has +probability 1/24, so the reproducible fixed campaign supplies its durable +assertion while random campaigns measure it. No prior guard, comparison, +minimum, or run budget was removed or lowered. The model, grammar, and comparison +are unchanged; only the new properties' fixed seeds were calibrated. Broader +#2272 transport and host gaps remain open: this receiver covers two real Browser +coordinators over Node BroadcastChannel/Web Locks and native SQLite in one +process. It does not establish Chromium OPFS, Electron, transport gaps, +leadership turnover, every partial-notice shape, or every shared session grammar. +The constructive receiver uses core holds; broad controlled histories retain +store/read holds. Owner: this sync metadata session oracle. A reset may rewrite its rows, which its reload then applies as source writes. `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end (decision 21); a seed that arrived after it stays judged against it (decision 22). Dependent suffixes of an abandoned transaction belong to the -abort-graph oracle. The peer is the driver, not a second -Collection, so the claim stops at the controlled coordinator; a second -production driver with two real coordinators is the open receiving -witness (issue #2272). Missed notices and notices over the targeted key limit are not +abort-graph oracle. The broad peer grammar uses a controlled coordinator. The new constructive +seed/re-insert/reload receiver uses two real Collections and coordinators for +the bounded metadata law above; it does not close the broader host and transport +receiving scope of issue #2272. Missed notices and notices over the targeted key limit are not generated; the persisted oracle's grids cover them. Two held-cleanup Query retention witnesses race on CI; the maintainer deferred the race until a failure reports the timer trace these witnesses now print, and the review diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 666fbf6a1a..0d93565e26 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -4698,6 +4698,18 @@ class PersistedCollectionRuntime< value: operation.value as Record, } }) + // An equal re-insert keeps accepted metadata without writing it in this + // sync run. Its durable replacement must keep that same value, including + // a DbClient seed that never reached storage (decisions 9 and 14). + const rowMetadataWrites = new Map(transaction.rowMetadataWrites) + for (const { key } of transaction.insertOperations) { + if (rowMetadataWrites.has(key)) continue + const value = this.syncControls.metadata?.row.get(key) + rowMetadataWrites.set( + key, + value === undefined ? { type: `delete` } : { type: `set`, value }, + ) + } return { txId: safeRandomUUID(), term: streamPosition.term, @@ -4706,7 +4718,7 @@ class PersistedCollectionRuntime< cacheGenerationClaimId: this.cacheGenerationClaim?.claimId, truncate: transaction.truncate, mutations, - rowMetadataMutations: toMetadataMutations(transaction.rowMetadataWrites), + rowMetadataMutations: toMetadataMutations(rowMetadataWrites), collectionMetadataMutations: toMetadataMutations( transaction.collectionMetadataWrites, ), diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index 7e50570609..b483f91cd2 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -5,6 +5,7 @@ import { absentMarker, cachedMetadata, cachedRow, + completedSeedReinsertReloadRegime, describeMismatch, historyRegimes, persistedCachedLane, @@ -19,13 +20,17 @@ import { persistedReadSeedLane, persistedSeedLane, runSessionHistory, + seedReinsertReloadHistoryArbitrary, + seedReinsertReloadReplay, sessionCollectionKeys, sessionHistoryArbitrary, } from '../../db/tests/sync-metadata-session-oracle' import { oraclePropertyOptions, + oracleReplayTest, oracleRuns, readOracleRunConfig, + recordOracleReplay, } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' @@ -1209,29 +1214,13 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla it.each(peerFullReplays)(`replays under full rows: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerFullLane), ) - // Known gap, pinned from the persisted-peers-full random campaign (seed - // 654774679, shrunk to this prefix): an outer transaction inserts over a - // bare seed, an inner one re-inserts the row and commits, and a peer's full - // reload then leaves row 1 with the cached seed's metadata, where the model - // expects none. This reproduces on main and is tracked with #2281. It - // replays through the same model, driver, and comparison. Remove `fails` - // when the reload stops restoring that metadata. - it.fails( - `replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, - () => - assertPersistedHistory( - [ - { type: `begin` }, - { type: `seed`, key: 1, narrow: false, bare: true }, - { type: `insertOverSeed`, key: 1, metadata: false }, - { type: `begin` }, - { type: `reinsert`, key: 1, metadata: false }, - { type: `commit` }, - { type: `peer`, key: 2, notice: `reload` }, - ], - persistedPeerFullLane, - ), - ) + // Pinned from the persisted-peers-full random campaign (seed 654774679): + // an outer transaction inserts over a bare seed and an inner equal re-insert + // commits first. Its durable replacement must keep the accepted metadata + // (absent here), so a peer's full reload cannot restore the older cache + // metadata. This uses the campaigns' model, driver, and comparison. + it(`replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, () => + assertPersistedHistory(seedReinsertReloadReplay, persistedPeerFullLane)) it.each(peerOnDemandReplays)(`replays on demand: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerOnDemandLane), ) @@ -1241,48 +1230,140 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla ) }) -describe(`sync metadata session oracle through the persisted wrapper: nesting and seeds`, () => { - for (const { - lane, - property: lanePropertyName, - seed, - runs, - reach: states, - } of extraLanes) { - const runLane = (history: Array) => - assertPersistedHistory(history, lane) - if (requestedReplayProperty === undefined) { - const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { - minimum: 5, - }) - fcTest.prop([sessionHistoryArbitrary(lane)], { - numRuns: oracleRuns(runs), - seed, - })( - `keeps the session law on ${lane.name} histories (fixed)`, - (history) => { +// The broad peer campaigns can already find this defect. This constructive +// grammar measures its scarce overlap and adjacent legal histories directly; +// it uses the existing independent model, driver, and per-action comparison. +const seedReloadProperty = `sync-metadata-session.persisted-seed-reinsert-reload` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requestedReplayProperty !== undefined && + (requestedReplayProperty !== seedReloadProperty || campaign === `fixed`) + ) + continue + it(`keeps inserted metadata across peer reloads (${campaign})`, async () => { + const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const states = lanes.flatMap((lane) => [ + `${lane.name}: inner commit`, + `${lane.name}: row`, + `${lane.name}: reload`, + ]) + const completedStates = lanes.flatMap((lane) => [ + `${lane.name}: completed kept bare seed reload`, + `${lane.name}: completed kept non-bare seed reload`, + ]) + const reach = createOracleReach(seedReloadProperty, states) + const completedReach = createOracleReach( + seedReloadProperty, + completedStates, + ) + const result = await fc.check( + fc.asyncProperty(seedReinsertReloadHistoryArbitrary, async (history) => { + const failures: Array = [] + for (const lane of lanes) { reach.beginHistory() - for (const regime of historyRegimes(lane, history)) - if (states.includes(regime)) reach.reach(regime) - return runLane(history) - }, - 240_000, - ) - it(`reaches the declared states in the ${lane.name} fixed campaign`, () => - reach.assertReached()) - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )( - `keeps the session law on ${lane.name} histories (random)`, - runLane, - 120_000, + completedReach.beginHistory() + const result = await runSessionHistory(lane, history, () => + createPersistedDriver( + true, + true, + false, + true, + lane.fullRows === true, + lane.onDemand === true, + ), + ) + const last = history.at(-1)! + const innerCommit = history.findIndex( + (action) => action.type === `commit`, + ) + if (result.steps > innerCommit) + reach.reach(`${lane.name}: inner commit`) + if (last.type === `peer` && result.steps === history.length) + reach.reach(`${lane.name}: ${last.notice}`) + const completed = completedSeedReinsertReloadRegime(history, result) + if (completed !== undefined) + completedReach.reach(`${lane.name}: ${completed}`) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }), + campaign === `fixed` + ? { numRuns: oracleRuns(40), seed: 527962096 } + : oraclePropertyOptions(40, seedReloadProperty), + ) + recordOracleReplay(seedReloadProperty, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property: seedReloadProperty, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, ) - } else if (requestedReplayProperty === lanePropertyName) { - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )(`replays ${lane.name} session histories`, runLane, 120_000) + reach.assertReached() + // The bare-key-1 conjunction has probability 1/24: random40 would miss it + // about 18% of the time. Every normal run enforces the fixed40 branch + // guards; the identical random40 campaign measures them without a promise + // of reach. Existing inner/row/reload guards apply to both campaigns. + if (campaign === `fixed`) completedReach.assertReached() + }, 120_000) +} + +const describeNested = oracleReplayTest( + describe, + ...extraLanes.map(({ property }) => property), +) +describeNested( + `sync metadata session oracle through the persisted wrapper: nesting and seeds`, + () => { + for (const { + lane, + property: lanePropertyName, + seed, + runs, + reach: states, + } of extraLanes) { + const runLane = (history: Array) => + assertPersistedHistory(history, lane) + if (requestedReplayProperty === undefined) { + const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { + minimum: 5, + }) + fcTest.prop([sessionHistoryArbitrary(lane)], { + numRuns: oracleRuns(runs), + seed, + })( + `keeps the session law on ${lane.name} histories (fixed)`, + (history) => { + reach.beginHistory() + for (const regime of historyRegimes(lane, history)) + if (states.includes(regime)) reach.reach(regime) + return runLane(history) + }, + 240_000, + ) + it(`reaches the declared states in the ${lane.name} fixed campaign`, () => + reach.assertReached()) + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )( + `keeps the session law on ${lane.name} histories (random)`, + runLane, + 120_000, + ) + } else if (requestedReplayProperty === lanePropertyName) { + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )(`replays ${lane.name} session histories`, runLane, 120_000) + } } - } -}) + }, +) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts new file mode 100644 index 0000000000..cb4edf4e61 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -0,0 +1,462 @@ +// @vitest-environment node +/** + * The sync metadata session oracle's seed/re-insert/reload law crosses real + * SQLite and coordinator boundaries here. The law, model, legal histories, + * and comparison are imported from sync-metadata-session-oracle.ts. This + * driver supplies two persisted Collections, two production coordinators over + * Node's BroadcastChannel/Web Locks, and one node:sqlite database. Each + * Collection has its own adapter; their shared physical database supplies the + * shared scheduler premise. + * + * A peer's named notice comes from its real source commit. For a full reload, + * that source takes a durable snapshot, changes the requested row, and commits + * a snapshot transaction. This preserves every other durable row and causes + * production to announce requiresFullReload. The receiver records the actual + * notice and waits for SQL/message quiescence before the shared comparison. + * Neither this driver nor the peer invents an expected metadata value. + * + * Limits: one process, one SQLite database, two Collections of one ID, no + * transport loss or leadership turnover, and core holds only. This is a real + * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. + */ +import { DatabaseSync } from 'node:sqlite' +import { fc } from '@fast-check/vitest' +import { it } from 'vitest' +import { DbClient, collectionOptions } from '@tanstack/db' +import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' +import { + absentMarker, + cachedMetadata, + cachedRow, + completedSeedReinsertReloadRegime, + describeMismatch, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + runSessionHistory, + seedReinsertReloadHistoryArbitrary, + seedReinsertReloadReplay, + sessionCollectionKeys, +} from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, + readOracleRunConfig, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + eagerStorageTarget, + managedStorageTarget, + persistedCollectionOptions, +} from '../src' +import type { SQLInputValue } from 'node:sqlite' +import type { SyncConfig } from '@tanstack/db' +import type { + ReceiptState, + SessionAction, + SessionDriver, + SessionLane, +} from '../../db/tests/sync-metadata-session-oracle' +import type { SQLiteDriver } from '../src' + +type Row = { id: number; v: unknown } +type Actions = Parameters[`sync`]>[0] +let run = 0 + +async function createSQLiteDriver(lane: SessionLane): Promise { + const database = new DatabaseSync(`:memory:`) + let activity = 0 + let queue: Promise = Promise.resolve() + const driver: SQLiteDriver = { + exec: async (sql) => { + activity++ + database.exec(sql) + }, + query: async (sql: string, params: ReadonlyArray = []) => { + activity++ + return database + .prepare(sql) + .all(...(params as Array)) as Array + }, + run: async (sql, params = []) => { + activity++ + database.prepare(sql).run(...(params as Array)) + }, + transaction: (work) => { + const next = queue.then(async () => { + database.exec(`BEGIN IMMEDIATE`) + try { + const result = await work(driver) + database.exec(`COMMIT`) + return result + } catch (error) { + database.exec(`ROLLBACK`) + throw error + } + }) + queue = next.catch(() => undefined) + return next + }, + } + Object.defineProperty(driver, SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, { + value: {}, + }) + const adapters = [ + new SQLiteCorePersistenceAdapter({ driver }), + new SQLiteCorePersistenceAdapter({ driver }), + ] + const id = `metadata-items` + const seedClaim = lane.onDemand + ? await adapters[0]!.claimCacheGeneration(id) + : undefined + const target = seedClaim + ? managedStorageTarget(seedClaim.storageCollectionId, seedClaim.claimId) + : eagerStorageTarget(id) + const storageId = seedClaim?.storageCollectionId ?? id + await adapters[0]!.applyCommittedTx(target, { + txId: `cached`, + term: 1, + seq: 1, + rowVersion: 1, + cacheGenerationClaimId: seedClaim?.claimId, + mutations: [{ type: `insert`, key: 1, value: cachedRow }], + rowMetadataMutations: [{ type: `set`, key: 1, value: cachedMetadata }], + }) + const dbName = `sync-metadata-receiving-${process.pid}-${run++}` + const coordinators = adapters.map( + (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), + ) + const notices: Array<{ reload: boolean }> = [] + let durableRows: ReadonlyArray<{ key: string | number; value: unknown }> = [] + const stopNotices = coordinators[0]!.subscribe(storageId, (message) => { + activity++ + const payload = message.payload as { + type: string + requiresFullReload?: boolean + } + if (payload.type === `tx:committed`) + notices.push({ reload: payload.requiresFullReload === true }) + }) + const quiesce = async () => { + let quiet = 0 + for (let turn = 0; turn < 400 && quiet < 5; turn++) { + const before = activity + await new Promise((resolve) => setTimeout(resolve, 2)) + quiet = before === activity ? quiet + 1 : 0 + } + if (quiet < 5) throw new Error(`SQLite metadata hosts did not quiesce`) + durableRows = await adapters[0]!.loadSubset(target, {}) + } + const actions: Array = [] + let resolveMutation: (() => void) | undefined + const clients = [new DbClient(), new DbClient()] + const collections = clients.map((client, index) => + client.collection( + collectionOptions( + persistedCollectionOptions({ + id, + getKey: (row) => row.id, + ...(lane.onDemand ? { syncMode: `on-demand` as const } : {}), + sync: { + rowUpdateMode: lane.fullRows ? `full` : `partial`, + sync: (sync) => { + actions[index] = sync + sync.begin() + sync.commit() + sync.markReady() + return lane.onDemand ? { loadSubset: () => true } : undefined + }, + }, + onInsert: () => + new Promise((resolve) => { + resolveMutation = resolve + }), + persistence: { + adapter: adapters[index]!, + coordinator: coordinators[index]!, + }, + }), + ), + ), + ) + // These direct source histories have no live query to start on-demand sync. + // Start both sources before a follower can route its first demand to its peer. + for (const current of collections) current.startSyncImmediate() + if (!lane.onDemand) + await Promise.all(collections.map((collection) => collection.preload())) + const demands = collections.map(() => ({})) + if (lane.onDemand) + await Promise.all( + collections.map((collection, index) => + collection._sync.loadSubset(demands[index]!), + ), + ) + await quiesce() + const sync = actions[0]! + const metadata = sync.metadata! + const collection = collections[0]! + const receipts: Array> = [] + const receiptStates: Array = [] + const rejections: Array = [] + let depth = 0 + let holdCount = 0 + let blocker: { when: (state: 'settled') => Promise } | undefined + const release = async () => { + resolveMutation?.() + resolveMutation = undefined + await blocker?.when(`settled`) + blocker = undefined + } + const abort = () => { + const controller = new AbortController() + controller.abort() + const receipt = sync.commit(controller.signal) + depth-- + if (receipt !== true) + receipts.push( + receipt.catch((error: unknown) => { + if (!(error instanceof Error) || error.name !== `AbortError`) + rejections.push(String(error)) + }), + ) + } + const read = (value: unknown) => (value === undefined ? absentMarker : value) + const sorted = (entries: ReadonlyArray<{ key: string; value: unknown }>) => + entries + .map(({ key, value }): [string, unknown] => [key, value]) + .sort(([a], [b]) => a.localeCompare(b)) + const rowMetadata = (value: unknown) => + value === undefined ? {} : { metadata: value as Record } + return { + consumerCheckpoint: `final`, + begin: () => { + sync.begin() + depth++ + }, + commit: () => { + const index = receiptStates.length + const receipt = sync.commit() + depth-- + receiptStates.push(receipt === true ? `applied` : `pending`) + if (receipt !== true) + receipts.push( + receipt.then( + () => { + receiptStates[index] = `applied` + }, + () => { + receiptStates[index] = `rejected` + }, + ), + ) + }, + cancel: abort, + rowSet: (key, value) => metadata.row.set(key, value), + rowDelete: (key) => metadata.row.delete(key), + insert: (_key, value, meta) => + sync.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }), + update: (_key, value, meta) => + sync.write({ type: `update`, value: value as Row, ...rowMetadata(meta) }), + delete: (key) => sync.write({ type: `delete`, key }), + truncate: () => sync.truncate(), + collectionSet: (key, value) => metadata.collection.set(key, value), + collectionDelete: (key) => metadata.collection.delete(key), + hold: (store, readHold) => { + if (store || readHold) + throw new Error(`the SQLite seed/re-insert grammar uses core holds`) + blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) + }, + release, + seed: (key, value, meta) => + clients[0]!.hydrate({ + collections: [ + { + collectionId: id, + rows: [ + { + key, + value: value as Row, + metadata: meta as Record, + }, + ], + }, + ], + }), + peer: async (key, notice, value, meta) => { + await quiesce() + const before = notices.length + const peer = actions[1]! + if (notice === `reload`) { + const rows = await adapters[1]!.loadSubset(target, {}) + peer.begin({ snapshot: true }) + for (const row of rows) + if (row.key !== key) + peer.write({ + type: `insert`, + value: row.value as Row, + ...rowMetadata(row.metadata), + }) + } else if (notice === `row`) peer.begin() + else + throw new Error( + `unexpected peer notice in the seed/re-insert grammar: ${notice}`, + ) + peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) + await peer.commit() + await quiesce() + if ( + !notices + .slice(before) + .some((entry) => entry.reload === (notice === `reload`)) + ) + throw new Error( + `the real coordinator did not deliver the ${notice} notice`, + ) + console.info( + `SQLITE_METADATA_NOTICE ${JSON.stringify({ lane: lane.name, notice, received: notices.length - before })}`, + ) + }, + observe: () => ({ + rowMetadata: { + 1: read(metadata.row.get(1)), + 2: read(metadata.row.get(2)), + }, + collection: Object.fromEntries( + sessionCollectionKeys.map((key) => [ + key, + read(metadata.collection.get(key)), + ]), + ), + list: sorted(metadata.collection.list()), + prefixList: sorted(metadata.collection.list(`gc:`)), + base: { + 1: read(collection.base.get(1)), + 2: read(collection.base.get(2)), + }, + receipts: [...receiptStates], + }), + storedRows: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.value), + ]), + ), + flush: quiesce, + settle: async () => { + await release() + while (depth) abort() + await quiesce() + await Promise.all(receipts) + await quiesce() + return rejections + }, + cleanup: async () => { + await release() + while (depth) abort() + await Promise.all(receipts) + if (lane.onDemand) + for (const [index, current] of collections.entries()) + current._sync.unloadSubset(demands[index]!) + await quiesce() + await Promise.all(collections.map((current) => current.cleanup())) + stopNotices() + for (const coordinator of coordinators) coordinator.dispose() + if (seedClaim) + await adapters[0]!.releaseCacheGenerationClaim(seedClaim.claimId) + await queue + database.close() + }, + } +} + +const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, +] +async function check(history: ReadonlyArray, lane: SessionLane) { + return runSessionHistory(lane, history, () => createSQLiteDriver(lane)) +} +oracleReplayTest(it).each(lanes)( + `receives the seed/re-insert/reload replay through SQLite ($name)`, + async (lane) => { + const result = await check(seedReinsertReloadReplay, lane) + if (result.mismatch) + throw new Error( + describeMismatch(seedReinsertReloadReplay, result.mismatch), + ) + }, +) + +const property = `sync-metadata-session.sqlite-seed-reinsert-reload` +const requested = readOracleRunConfig().replayProperty +for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== property || campaign === `fixed`) + ) + continue + it(`receives generated seed/re-insert/reload histories through SQLite (${campaign})`, async () => { + const states = lanes.flatMap((lane) => [ + `${lane.name}: inner commit`, + `${lane.name}: row`, + `${lane.name}: reload`, + ]) + const completedStates = lanes.flatMap((lane) => [ + `${lane.name}: completed kept bare seed reload`, + `${lane.name}: completed kept non-bare seed reload`, + ]) + const reach = createOracleReach(property, states) + const completedReach = createOracleReach(property, completedStates) + const result = await fc.check( + fc.asyncProperty(seedReinsertReloadHistoryArbitrary, async (history) => { + const failures: Array = [] + for (const lane of lanes) { + reach.beginHistory() + completedReach.beginHistory() + const result = await check(history, lane) + const last = history.at(-1)! + const innerCommit = history.findIndex( + (action) => action.type === `commit`, + ) + if (result.steps > innerCommit) + reach.reach(`${lane.name}: inner commit`) + if (last.type === `peer` && result.steps === history.length) + reach.reach(`${lane.name}: ${last.notice}`) + // The peer action has asserted actual requiresFullReload delivery + // before the shared comparison passes. + const completed = completedSeedReinsertReloadRegime(history, result) + if (completed !== undefined) + completedReach.reach(`${lane.name}: ${completed}`) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }), + campaign === `fixed` + ? { numRuns: oracleRuns(40), seed: 527962096 } + : oraclePropertyOptions(40, property), + ) + recordOracleReplay(property, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + // The fixed40 campaign enforces this scarce bare-key-1 premise on every + // normal run. Its probability is 1/24, so random40 would miss it about + // 18% of the time; that identical campaign reports measured branch reach. + // Existing inner/row/reload guards remain enforced in both campaigns. + if (campaign === `fixed`) completedReach.assertReached() + }, 120_000) +} diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index ca9c1a8927..952349fa99 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -238,6 +238,8 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-peers`, `sync-metadata-session.persisted-peers-full`, `sync-metadata-session.persisted-peers-on-demand`, + `sync-metadata-session.persisted-seed-reinsert-reload`, + `sync-metadata-session.sqlite-seed-reinsert-reload`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, `d2-source.exact-retractions`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 5de3ae76b1..fc31946410 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -636,7 +636,12 @@ const ownerGroups: ReadonlyArray< [ `db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts`, `sync-metadata-session`, - `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand`, + `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand persisted-seed-reinsert-reload`, + ], + [ + `db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts`, + `sync-metadata-session`, + `sqlite-seed-reinsert-reload`, ], ] diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 6e7772325a..a942262634 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -582,6 +582,109 @@ export function sessionHistoryArbitrary( }) } +/** + * A source insert can be visible to an inner transaction before it commits. + * The inner equal re-insert keeps metadata; its durable replacement must keep + * the metadata of the accepted row, including a later seed, rather than the + * older cache's metadata. A peer notice then reads that durable replacement. + * + * This constructive subgrammar rebuilds the #2281 metadata prefix and varies + * both keys, seed shape/metadata and arrival, explicit metadata, repeated + * re-inserts, a core hold, the enclosing transaction's ending, and named + * versus full-reload notices. Every action is legal in the peer lanes. Store + * and read holds are excluded because their pending work blocks peer notices. + * The general session grammar continues to cover their independent laws. + */ +export const seedReinsertReloadHistoryArbitrary: fc.Arbitrary< + Array +> = fc + .record({ + key: fc.constantFrom(...sessionRowKeys), + bare: fc.boolean(), + narrow: fc.boolean(), + seedInside: fc.boolean(), + outerMetadata: fc.boolean(), + innerMetadata: fc.boolean(), + repeats: fc.integer({ min: 1, max: 3 }), + hold: fc.boolean(), + ending: fc.constantFrom(`open`, `commit`, `cancel`), + notice: fc.constantFrom(`row` as const, `reload` as const), + }) + .map((input): Array => { + const seed: SessionAction = { + type: `seed`, + key: input.key, + bare: input.bare, + narrow: input.narrow, + } + return [ + ...(input.hold ? [{ type: `hold` as const }] : []), + ...(input.seedInside + ? [{ type: `begin` as const }, seed] + : [seed, { type: `begin` as const }]), + { type: `insertOverSeed`, key: input.key, metadata: input.outerMetadata }, + { type: `begin` }, + ...Array.from({ length: input.repeats }, (): SessionAction => ({ + type: `reinsert`, + key: input.key, + metadata: input.innerMetadata, + })), + { type: `commit` }, + ...(input.ending === `open` + ? [] + : [{ type: input.ending } as SessionAction]), + { type: `peer`, key: input.key === 1 ? 2 : 1, notice: input.notice }, + ] + }) + +/** The source report is input to the same model and comparison as generation. */ +export const seedReinsertReloadReplay: Array = [ + { type: `begin` }, + { type: `seed`, key: 1, narrow: false, bare: true }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `reload` }, +] + +/** + * A completed implicit re-insert must keep the later seed across a full reload + * while the enclosing write remains unaccepted. This reach classifier judges + * only the grammar and completed comparison, never production metadata values. + * Only key 1 starts with cached metadata, so only its bare seed distinguishes + * a stale cache value from absence. Both receiving drivers use these premises. + */ +export function completedSeedReinsertReloadRegime( + history: ReadonlyArray, + result: { mismatch: SessionMismatch | undefined; steps: number }, +): + | `completed kept bare seed reload` + | `completed kept non-bare seed reload` + | undefined { + const last = history.at(-1)! + const seed = history.find((action) => action.type === `seed`) + const keptSeedReload = + last.type === `peer` && + last.notice === `reload` && + history.some((action) => action.type === `reinsert`) && + history.every( + (action) => action.type !== `reinsert` || action.metadata === false, + ) && + history.filter((action) => action.type === `commit`).length === 1 + if ( + !result.mismatch && + result.steps === history.length && + keptSeedReload && + seed?.type === `seed` + ) { + if (seed.bare === true && seed.key === 1) + return `completed kept bare seed reload` + if (seed.bare === false) return `completed kept non-bare seed reload` + } + return undefined +} + /** * The regimes that a legal history reaches, read from the model as it * replays. The campaign reach test requires each law's premise. @@ -2525,5 +2628,11 @@ declareOracle({ runSeedOrderInvariance, ], model: [SessionModel, createSessionModel, modelCheckpoints], - grammar: [sessionActionArbitrary, sessionHistoryArbitrary, historyRegimes], + grammar: [ + sessionActionArbitrary, + sessionHistoryArbitrary, + historyRegimes, + seedReinsertReloadHistoryArbitrary, + completedSeedReinsertReloadRegime, + ], }) From ea769ec8d3b4c1bffc286e1d88b1b0c553c5cc5f Mon Sep 17 00:00:00 2001 From: Test Date: Thu, 29 Oct 2026 14:25:30 -0600 Subject: [PATCH 02/11] Document implicit re-insert metadata repair release --- .changeset/preserve-implicit-reinsert-metadata.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 .changeset/preserve-implicit-reinsert-metadata.md diff --git a/.changeset/preserve-implicit-reinsert-metadata.md b/.changeset/preserve-implicit-reinsert-metadata.md new file mode 100644 index 0000000000..bf6bf1192b --- /dev/null +++ b/.changeset/preserve-implicit-reinsert-metadata.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +--- + +Preserve accepted row metadata when an equal source re-insert omits an explicit metadata write. Prevent peer reloads from restoring older cached metadata after that insert. From a04d85519a4073466b32353e7bfce42314ceadcd Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 11:25:56 -0700 Subject: [PATCH 03/11] test(persistence): separate local metadata keeps from durable writes Law: Decisions 9 and 18 keep tab-local seed metadata separate from durable writes. Decision 15 permits unknown-author storage observations to adopt differing durable metadata; named current-run writes follow decision 11. Reach: Both hosts reach all 36 constructive case/lane states eight times per fixed and random campaign. Native recovery is reached 96 times each. Existing 40-run campaigns and fixed completed-branch guards are preserved. RED: No production defect is claimed on a3c4d067. The original PR diagnosis is withdrawn. Both hosts reject the old serializer materialization at the settled durable metadata comparison, seed 2220031, path 0:0:0. GREEN: 74 controlled and 12 native/helper/work tests pass after extraction; 92 core tests passed before the mechanics-only extraction. Direct replay, scoped types, formatting and lint pass (12 general ESLint warnings). Mutant: All five controls reject the intended assertion: materialization on each host, omitted model recovery input, dropped explicit writes and unscoped root SQL. Temporary edits are restored byte-for-byte. Limits: One-process native SQLite with two Browser coordinators and a bounded correlated notice/recovery schedule. No arbitrary recovery timing, transport/leadership loss, Chromium OPFS, Electron IPC or restart proof. Broader #2281/#2272 and the separate #2317/P8 witness remain open. See docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md. --- docs/contributing/oracle-coverage.md | 80 ++- .../2026-10-30-kept-metadata-durability.md | 165 +++++ .../2026-11-02-p6-metadata-closeout.md | 228 +++++++ .../tests/sync-metadata-session-campaign.ts | 99 +++ .../sync-metadata-session-persisted.test.ts | 238 ++++++-- .../sync-metadata-session-sqlite.test.ts | 564 ++++++++++++++++++ .../tests/sync-metadata-sqlite-driver.test.ts | 43 ++ .../tests/sync-metadata-sqlite-driver.ts | 48 ++ .../tests/sync-metadata-write-work.test.ts | 100 ++++ packages/db/tests/oracle-config.ts | 4 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../db/tests/sync-metadata-session-oracle.ts | 261 +++++++- 12 files changed, 1744 insertions(+), 93 deletions(-) create mode 100644 docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md create mode 100644 docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d1ad895998..fac95b1294 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4727,16 +4727,16 @@ row or its metadata; an equal re-insert keeps metadata without writing it. In the persisted wrapper an insert replaces whatever row it lands on. A peer's notice of a write in this run counts as a source write, and writes during a storage read follow the same rules, the last write for a key -winning. When a stored row's author is unknown, a seed this tab holds stays -unless storage changed it, and other stored rows count as source writes -(decisions 13 and 15). Reloads keep metadata of a missing row (decision 16), +winning. When a stored row's author is unknown, a DbClient seed this tab holds stays, +and other differing stored rows count as source writes (decisions 13 and 15). Reloads keep metadata of a missing row (decision 16), a write over a seed stores the row this tab shows (decision 17), and a reload after a named notice keeps a seed's metadata (decision 18). | Owner | Production paths | Observations and checkpoint | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [sync metadata session oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/sync-metadata-session-oracle.ts) (`flat`, `nested`, `seeds`, `nested-seeds`, `seeds-full`, `nested-conflicts` lanes) | `DbClient` Collection, core hold, `DbClient.hydrate` seeds | `metadata.row.get`, `metadata.collection.get`, `list()`, `list(prefix)`, `collection.base`, each commit receipt's state, thrown errors; after every action and after settlement | -| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, the stored rows after settlement | +| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, stored rows and stored row metadata after settlement | +| [native SQLite/coordinator receiver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts) | two real Collections and coordinators, node:sqlite, BroadcastChannel and Web Locks; partial/full/on-demand lanes | shared model and source reads after correlated commit/notice/recovery cuts; rows, metadata, applied receipts and direct durable rows/metadata after settlement | | paired properties and boundary witnesses in the core driver file | production against itself: holds removed, row 2's actions removed, or seeds performed first; stale run; write without a transaction; cleanup with a held seed | source reads (and row 1's view) at each shared checkpoint, or every observation after settlement for seed order; thrown errors; unhandled rejections | | [row metadata composition oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts) held-window and nested lanes | Collection, core hold, a transaction begun inside an open one, `DbClient.hydrate` | implicit row-metadata rules during a hold, and for an outer transaction whose inserts are reclassified after an inner write applies first, with and without a hold | | [hydration authority oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/db-client-hydration-authority-oracle.test.ts) | `DbClient` initialData, hydration chunks, and an adapter sync transaction (inserts, updates, deletes, truncates, and metadata-only writes) in full and partial row modes | public rows, the synced store, and row metadata through `client.dehydrate()` after each action; D5 per object | @@ -4778,24 +4778,68 @@ must equal the model's, which a restart would read. `persisted-peers-full` repeats the lane under full row updates, where an update replaces its row in storage too; `persisted-peers-on-demand` repeats it on demand with a demand for every row. A reset over metadata of a missing -row is too rare in generated histories to declare; a replay pins it. A -random CI campaign (seed `654774679`) found a `persisted-peers-full` history -that reproduces on `main`: an outer transaction inserts over a bare seed, an -inner one re-inserts the row and commits, and a peer's full reload leaves the -row with the cached seed's metadata, where the model expects none (law: -the source reads its row metadata). The maintainer chose to leave it open, -tracked with #2281; an `it.fails` replay through the same model, driver, and -comparison pins its shrunk prefix, and the random campaign can still draw it. -Owner: this oracle. A reset -may rewrite its rows, which its reload then applies as source writes. +row is too rare in generated histories to declare; a replay pins it. + +An equal source re-insert keeps metadata without writing it (decisions 9 and +14). As with an update, that keep neither makes a DbClient seed durable nor +sends it to peers (decision 18). Direct durable metadata is compared separately +from this tab's source reads. The constructive durability grammar includes all +12 update/equal-reinsert/inner-reinsert × bare/non-bare seed × implicit/explicit +metadata cases in each generated batch. Its partial/full/on-demand lanes run +through both the controlled persisted adapter and the real SQLite/coordinator +receiver, with declared successful reach for every case in both fixed and +random campaigns. Seed row shape, seed timing, repeat count and the final +named/full-reload/no-notice choice vary. Explicit metadata writes are the +neighboring control. The original seed/re-insert/reload grammar, fixed/random +40 budgets and its completed fixed-branch guards remain; the new constructive +batch supplies guaranteed bare-key-1 implicit histories independently of seed. +Both hosts share the 40-run campaign mechanics in +`sync-metadata-session-campaign.ts`; production setup, recovery observations +and direct replay selection remain in their receiving files. + +The native on-demand path can receive a successful recovery response that +crosses an earlier cache version without transaction replay details. The shared +model then applies the existing unknown-write rule (decision 15) to its own +independent durable storage. This is a recorded transport input, not an expected +value inferred from a production reload classifier. A named current-run write +still follows decision 11; a previous-run cached row initially remains a seed. +The native driver finishes each source commit's correlated notice and recovery +before admitting the next authored action. It compares source reads after that +cut and consumers, receipts, stored rows and stored metadata after settlement. +This is one legal schedule; arbitrary response/application interleavings, expiry, +cleanup during recovery and the necessity of gap detection are not established. + +The receiver serializes every root SQL call with transaction admission on its +one physical connection, while scoped transaction calls stay inside that +admission. A rollback/isolation witness rejects the earlier unscoped-call +fixture. The driver waits for actual correlated notices, remote ensure replies +and accepted empty source commits through the wrapper's apply lock, rather than +a quiet-time polling interval. It covers two real Browser coordinators over +Node BroadcastChannel/Web Locks and native SQLite in one process. Chromium +OPFS, Electron IPC, transport loss, leadership turnover, every partial-notice +shape and every shared session grammar remain outside it. The constructive +receiver uses core holds; broad controlled histories retain store/read holds. +Owner: this sync metadata session oracle. A reset may rewrite its rows, which +its reload then applies as source writes. The +[kept-metadata review](oracle-reviews/2026-10-30-kept-metadata-durability.md) +records the D9/D18 durable comparison, the D15 model-input correction, measured +reach, calibration and combined-tree evidence. +The [P6 current-main revalidation](oracle-reviews/2026-11-02-p6-metadata-closeout.md) +keeps the withdrawn production diagnosis distinct from the replacement test/model +patch. The source-facing metadata failure in [#2317](https://github.com/TanStack/db/issues/2317) +remains separate and open under this owner, with P8 investigating: property +`sync-metadata-session.persisted-peers`, seed `346038028`, path +`36:2:4:6:10:14:16:14:14:13:13:13`. P6's durability evidence does not close that +controlled-notice witness or the broader #2281 persistence reports. + `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end (decision 21); a seed that arrived after it stays judged against it (decision 22). Dependent suffixes of an abandoned transaction belong to the -abort-graph oracle. The peer is the driver, not a second -Collection, so the claim stops at the controlled coordinator; a second -production driver with two real coordinators is the open receiving -witness (issue #2272). Missed notices and notices over the targeted key limit are not +abort-graph oracle. The broad peer grammar uses a controlled coordinator. The new constructive +seed/re-insert/reload receiver uses two real Collections and coordinators for +the bounded metadata law above; it does not close the broader host and transport +receiving scope of issue #2272. Missed notices and notices over the targeted key limit are not generated; the persisted oracle's grids cover them. Two held-cleanup Query retention witnesses race on CI; the maintainer deferred the race until a failure reports the timer trace these witnesses now print, and the review diff --git a/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md b/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md new file mode 100644 index 0000000000..7cd0b71bd3 --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md @@ -0,0 +1,165 @@ +# Kept metadata: local reads, durable writes, and receiving evidence + +This record evaluates PR #2303 at published commit +`ea769ec8d3b4c1bffc286e1d88b1b0c553c5cc5f`. The final candidate is that +commit plus an ordinary merge of main +`9cfa11fc712866f8f276e85762865ebb8bd49cbe` and the review repairs below. +It has no production-source difference against that main commit. In +particular, `persisted.ts` retains main's source-progress fields and serializer. +The original metadata-materialization change and its release note are withdrawn. + +## Law and repair location + +Keeping metadata is not writing it. An update or equal re-insert without +explicit metadata must not copy a tab's hydration seed into durable metadata +or a peer metadata write. Authority: decisions 9 and 18 in the +[sync metadata review](2026-10-22-sync-metadata-session.md). Decision 14 fixes +insert kind at commit; it does not authorize another metadata write. + +Cached metadata M0 and a later hydration seed M1 can therefore coexist: this +tab reads M1 while storage retains M0. An explicit metadata write is the +neighboring control and must reach storage. A named current-run notice has +source authority under decision 11. When authorship is unknown, decision 15 +permits differing stored metadata to become source-written. Restoring M0 in +that case is not evidence that M1 should have been stored. + +The original oracle conflated a local keep with a durable write, omitted a +direct durable-metadata comparison, and sampled the distinguishing bare-seed +premise too rarely. The repair changes the independent model, grammar and +observations, then restores the serializer already present on main. It does +not add a production recovery algorithm. + +The native path also exposed a model environment-input gap. A real successful +`pullSince(0)` returned row version 2 without transaction replay details. The +shared model now records that response input and applies the existing D15 rule +to its own storage. It derives no expected metadata from production output or +the wrapper's reload classifier. Earlier suspicion of a production recovery +bug was withdrawn after this input was measured. + +## Histories, receiving path, and checkpoints + +The primary owner remains +[`sync-metadata-session-oracle.ts`](../../../packages/db/tests/sync-metadata-session-oracle.ts). +Each constructive batch includes all 12 combinations of update, equal +re-insert or inner equal re-insert; bare or non-bare seed; and implicit or +explicit metadata. Partial, full and on-demand lanes give 36 declared +successful case/lane states on each host. Seed shape, timing, repeat count and +the final named/full-reload/no-notice choice vary. Notice choices are sampled; +the property does not guarantee every notice choice for every case. + +The controlled persisted driver and native SQLite/coordinator driver use the +same model, histories and comparison. Source metadata reads are compared +after authored actions. Asynchronous consumer rows, receipts, durable rows +and durable metadata are compared after settlement. + +The native driver uses two real Collections and Browser coordinators, Node +BroadcastChannel/Web Locks, and native SQLite. It finishes each source +commit's correlated notice and recovery before admitting the next authored +action. Actual response inputs are consumed before comparison. Accepted empty +source commits drain delivered handlers through the apply lock without +changing rows or metadata. This selects one legal schedule. + +Every root SQL call shares transaction admission on the physical connection; +scoped calls remain inside their owning transaction. A focused rollback/read +isolation test rejects the earlier fixture, whose unscoped calls could join +another adapter's transaction. Actual notices, ensure replies and apply-lock +barriers replace the former quiet-time heuristic. + +## Evidence + +The extended comparisons ran on unchanged PR production before removing +materialization. They also ran on the merged current-main composition before +that same repair. Both controlled and native comparisons failed at the durable +metadata cut: expected cached M0, actual tab seed M1 or absent metadata. These +were assertion failures, not setup failures or timeouts. + +Current-main verification, with one worker and coverage/type checking disabled +inside Vitest: + +| Experiment | Result | +| --- | --- | +| Native receiver, SQL fixture and work files | 12 tests passed | +| Complete controlled persisted session file | 74 tests passed | +| New durability fixed/random campaigns, each host | 8 batches per campaign; every one of the 36 case/lane states reached 8 times | +| Native prior-cache version crossed without replay details | 96 successful histories in each campaign | +| Original seed/re-insert/reload campaigns | Fixed/random 40-run budgets and existing fixed completed-branch guards retained | +| Direct durability replay, both hosts | Seed `2220031`, path `0:0:0`, passed; each case/lane reached once | +| Scoped TypeScript check | Passed on the combined tree | +| Unchanged refresh work | 0 implicit metadata mutations for 2, 100 and 10,000 rows; original PR emitted 2, 100 and 10,000 | + +The work count measures emitted mutations, not native SQLite elapsed time or +a universal performance bound. Original generator sampling measured zero +completed bare-key-1 premises in 40 inputs at seed `-704889444`, and two at +fixed seed `527962096`. Constructive batches now supply the bare implicit +premise independently of seed while preserving the older campaigns. + +Calibration on the combined tree used the same seed/path and comparison: + +| Wrong account | Rejection | +| --- | --- | +| Restore the original PR materialization on top of current main | Both hosts reject M1/absence in durable metadata where M0 must remain | +| Ignore the measured unknown-authorship recovery input in the model | Native source metadata comparison fails at commit; actual response input remains in the trace | +| Drop explicit durable metadata writes too | Native explicit-write controls reject cached M0 where explicit metadata must persist | +| Earlier unscoped SQL fixture | Focused isolation/rollback test observes an uncommitted row and rejects it | + +The model-input control is a wrong model, not a production mutant. The original +serializer and explicit-write controls are production mutants. Every temporary +edit was restored, then direct replays passed on both hosts. The final serializer +SHA-256 is `1a3bc05bbf10646c8d75fb062799c8335a181a8d3a5e91d61a9244c1a4f5027a`; +the shared model is `584d767b43aa7a9f4661c65fa431ac579d0037e5e1fa8415e693817b0dae635c`. +The controlled and native driver hashes are respectively +`0a568a76809363068f885c132ed66ecea74bbbc984d5d27116ec852b9d83bac0` and +`a6387073a9cd3fa08c2d553d0e30f949a0df53c11644080985e2113698b1c47c`. +These identities bind this evidence to the uncommitted combined candidate; +later semantic edits require affected checks again. + +## Oracle review and limits + +Independent code review found contradictory retained prose and an obsolete +release promise. Both were corrected, then the combined candidate was reviewed +again with no new actionable finding. A separate optional campaign-runner +extraction remains unapproved and unapplied; none of these results validate it. + +| Requirement | Outcome for the changed scope | +| --- | --- | +| ORC-001 | D9/D14/D18 authority and D11/D15 neighbors are explicit. | +| ORC-002 | Independent storage/events compute expected metadata. | +| ORC-003 | Contract, model, grammar, receiver and comparison prose agree. | +| ORC-004 | Original witness reconstructs; update/flat/inner, bare/non-bare and implicit/explicit axes distinguish competing rules. Existing legality rejects transactionless writes. Notice coverage stays bounded. | +| ORC-005 | Real entry points, measured receiving inputs and direct durable comparisons reach the named cuts. | +| ORC-006 | Wrong serializer, missing model input, dropped explicit writes and bad SQL admission have distinct rejecting witnesses. | +| ORC-007 | Matching fixed/random properties and budgets; direct seed/path replay verified on both hosts. | +| ORC-008 | A subsequent unknown storage observation distinguishes local metadata from durable metadata, so both remain modeled. | +| ORC-009 | Recovery input explicitly abstracts a successful response without authorship. | +| ORC-010 | Shared mismatch reporting retains recovery inputs and primary failure; cleanup diagnostics remain distinguishable. Broader setup/host teardown is outside this evidence. | +| ORC-011 | No remaining concrete shared semantic-fault hypothesis requiring a second expected-value model was identified. | +| ORC-012 | This record binds contract, witnesses, calibration, review outcomes and limits to exact source identities. | +| ORC-013 | Explicit writes and bare seeds distinguish keeping, materializing and never-writing rivals. | +| ORC-014 | Native receiving reuses the controlled owner's model and comparisons; actual transport input supplies its own premise. | +| ORC-015 | All 36 successful case/lane states and the native recovery state have enforced reach. | +| ORC-016 | Fixed histories use the shared judgment. Supplemental SQL/work examples remain in ordinary test files. | + +The evidence covers one process, two coordinators, partial/full/on-demand +lanes, and the bounded commit-to-notice/recovery schedule. It does not cover +arbitrary response/application interleavings, cleanup/expiry/cancellation during +recovery, leadership turnover, transport loss, Chromium OPFS, Electron IPC, +restart semantics, every shared grammar, intermediate durable cuts hidden by +later writes, or the necessity of gap detection. The native constructive +campaign uses core holds; the broader controlled owner retains store/read +holds. These results do not close the broader receiving scope of #2272. + +Run the two host files from `packages/db-sqlite-persistence-core`: + +```sh +pnpm exec vitest run --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false tests/sync-metadata-session-persisted.test.ts +pnpm exec vitest run --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false tests/sync-metadata-session-sqlite.test.ts tests/sync-metadata-sqlite-driver.test.ts tests/sync-metadata-write-work.test.ts +``` + +For direct replay, set `TANSTACK_DB_ORACLE_PROPERTY` to +`sync-metadata-session.persisted-kept-metadata-durability` or +`sync-metadata-session.sqlite-kept-metadata-durability`, plus +`TANSTACK_DB_ORACLE_SEED=2220031` and `TANSTACK_DB_ORACLE_PATH=0:0:0`. +Select `-t 'tab-local and durable metadata distinct'` in the matching host file. +The fixed campaign stands aside. Local runs use +`pnpm_config_verify_deps_before_run=false` with existing dependency bindings; +no install override or lockfile change belongs to this repair. diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md new file mode 100644 index 0000000000..3d4f78c0ac --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -0,0 +1,228 @@ +# P6 metadata correction on current main + +Baseline: `a3c4d0676621cc7df4d242bdfe31815509add734` (`origin/main`, +November 2, 2026). Branch: `codex/p6-metadata-closeout`. This candidate transfers +the test/model/fixture patch whose SHA-256 is +`3b82a8471cae91982e0085dade060804354904ffa839405856eef7c50ca243a9`, then +corrects one stale comment and extracts the approved shared campaign runner. +It changes no production source or release note. +The earlier [kept-metadata record](2026-10-30-kept-metadata-durability.md) retains +its historical baseline and executions; the results below are new executions. + +## Accepted law and corrected diagnosis + +Keeping metadata in this tab is not writing metadata durably. Decisions 9 and +18 of the [sync metadata review](2026-10-22-sync-metadata-session.md) prohibit +promoting a hydration seed's metadata through an implicit keep. Explicit +metadata writes must still persist. Decision 14 fixes insert kind, not metadata +durability. Decision 11 governs named current-run writes; decision 15 permits +an unknown-author storage observation to adopt differing durable metadata. + +Thus this tab may read seed metadata M1 while storage retains cached metadata +M0. An update or equal re-insert without explicit metadata does not make M1 +durable. A later unknown-author reload may expose M0. The original #2303 +production diagnosis conflated those observations and is withdrawn. + +The current model separately retains durable metadata and tab-local reads. The +native driver records actual successful recovery responses without replay +details; the model applies D15 using its own storage. This repairs a missing +model input, not a demonstrated production recovery defect. Direct durable +metadata comparison, constructive reach and calibrated SQL admission repair +the other instrument gaps. The approved shared completed-history reach helper +remains. The maintainer approved the separate campaign-runner extraction on +November 2; both hosts now call the same 40-run campaign mechanics. + +## Published PR and issue dispositions + +PR [#2303](https://github.com/TanStack/db/pull/2303) was refreshed at head +`ea769ec8d3b4c1bffc286e1d88b1b0c553c5cc5f`, still open. Its published title, +body, serializer materialization and release note describe the withdrawn +production fix. No new review or inline comment was present. The current +replacement candidate contains only tests, model, fixtures and documentation. + +Recommended title: **test(persistence): separate local metadata keeps from durable writes**. +Update the existing PR with ordinary follow-up history: merge current main, +replace its net diff with this validated candidate, remove the obsolete +materialization and release note, and rewrite the body around D9/D18 and the +bounded receiving evidence. Preserve main's source-progress changes. No force +push or new package release is needed. This session has not updated the PR. + +- **#2281:** open. Withdraw the old metadata product-repair claim; the row-loss + and resurrection reports remain separate. +- **#2272:** open. This supplies one bounded native receiving path, not its + broader host, transport or lifecycle closure. +- **#2317 / P8:** open. Its source-facing metadata loss after a controlled + notice is a distinct witness. Neither the withdrawn diagnosis nor this + durability comparison resolves it. + +## Lossless review accounting + +The original six claims and proposed cures, historical assessments, sibling +corrections and seven claim-accounting amendments remain in the append-only +task ledger. The original source was re-read before this evaluation. + +| ID | Claim and current disposition | Preserved value | +| --- | --- | --- | +| META-EXT-2303-001 | PR materialization amplifies metadata writes and copies a map: repaired by omitting the materialization. Selective seed materialization is also unauthorized. | Direct durable comparison and 2/100/10,000-row emitted-mutation controls. | +| META-EXT-2303-002 | Updates need the same materialization: refuted under D18/D15. | Update histories and actual unknown-recovery input remain useful. | +| META-EXT-2303-003 | Materialization conflicts with D18: repaired by preserving main's serializer. | Keep tab-local and durable metadata separate; no restart proof inferred. | +| META-EXT-2303-004 | Completed bare-seed premise was scarce: repaired constructively. | Every batch crosses write form, seed metadata and explicit/implicit writes on both hosts. | +| META-EXT-2303-005 | Root SQL calls can enter another transaction; quiet-time polling is insufficient: fixture repaired. | Root/scoped admission, actual notices/ensure replies and apply-lock barriers. | +| META-EXT-2303-006 | Duplicated campaign runner: repaired after maintainer approval. | Shared mechanics preserve host setup, recovery, registration, replay selection, budgets and comparisons. | + +Raw total: **6 = 5 repaired + 1 refuted**. +The suspected production recovery sibling remains refuted; the missing-model-input +sibling remains repaired. Prior prep prose/release-note corrections and the +seven original-claim corrections retain their identities and qualifications. + +Two fresh independent prep agents found the same stale phrase, “find this +defect,” in the controlled receiver. P6-SIMP-DOC-001 is fixed by changing it to +“generate this interaction”; P6-CR-001 is a duplicate with its source identity +retained. Neither reviewer found another supported behavioral defect. The +original reviewer merits a hire recommendation for finding the D18 and fixture +problems, with contract verification required before applying proposed cures. + +## Current evidence + +The unmodified production on this main baseline is GREEN. No product RED on +this baseline is claimed: this is correction of the PR and its instruments. +The historical pre-removal RED remains in the October 30 record. Fresh hostile +controls below reject the same PR materialization on the selected main source. + +| Check | Current result | +| --- | --- | +| Complete controlled persisted receiver file after extraction | 74 passed, 279.07 seconds | +| Native receiver, SQL fixture and emitted-work files after extraction | 12 passed, 4.90 seconds | +| Core session property/boundary files and row-metadata composition owner before the mechanics-only extraction | 92 passed, 69.26 seconds | +| Scoped TypeScript | Passed | +| Prettier and `pnpm lint:changed` | Passed | +| General ESLint on nine changed TypeScript files | 0 errors, 12 warnings (shadowed test variables and async fixture stubs) | +| Shared campaign direct replay on each host | One requested history ran and passed on each host, seed `527962096`, path `0`; the fixed campaign stood aside | +| Restored direct durability replay on each host | One property ran and passed on each host, seed `2220031`, path `0:0:0` | + +Both normal native durability campaigns reached all 36 successful case/lane +states eight times each. Actual prior-cache recovery without replay details +was reached 96 times per campaign. Fixed seed was `2220031`; the native random +seed was `705218229`. The preserved 40-run native seed/re-insert/reload fixed +campaign (`527962096`) reached inner commits 40, named notices 21, full reloads +19 and both completed kept-seed states twice, per lane. Random seed +`-1759635559` reached inner commits 40, named notices 18, full reloads 22, +completed kept bare seeds twice and non-bare seeds six times, per lane. +Random completed-branch counts remain measured, not promised by that campaign. +The constructive durability matrix supplies that premise independently. + +Both controlled durability campaigns also reached all 36 successful case/lane +states eight times each: fixed seed `2220031`, random seed `1349769898`. +The preserved controlled 40-run campaigns passed at fixed seed `527962096` +and random seed `-1150914171`. Fixed reach matched the native fixed campaign. +The random campaign reached inner commits 40, named notices 20, full reloads +20 and each completed kept-seed branch three times, per lane. The complete +post-extraction file executed and passed all 74 tests. + +`sync-metadata-session-campaign.ts` shares the existing history loop, reach +counters, fast-check configuration and failure reporting. It introduces no +model rule or expected answer. Each host keeps its production driver and +replay filter; the newer constructive durability campaigns remain local. +Direct replay of the extracted runner selected one history on each host; +normal fixed and random runs retained their original 40-history budgets. + +The emitted-work controls observed zero implicit metadata mutations at 2, 100 +and 10,000 unchanged rows. These are mutation counts, not elapsed-SQLite-time +claims. The grammar sampling control again measured zero old completed bare +premises at seed `-704889444` and two at `527962096`, versus 24 constructive +bare/implicit histories across each eight-batch sample. + +All five calibration executions were repeated after extraction and failed at +the intended comparison: + +| Wrong account | Observed rejection | +| --- | --- | +| Original PR materialization, controlled host | Settled durable metadata expected cached M0, got local seed M1 or absence. | +| Original PR materialization, native host | Same durable mismatch; native on-demand also exposes the differing source read after actual recovery. | +| Omit the measured recovery input from the model | Source metadata mismatch at `commit`: expected seed metadata, got cached metadata; actual response crossing version 0 to 2 remains in the trace. This is a model calibration, not a production mutant. | +| Drop explicit durable metadata writes | Explicit-write case expected `{s:3}` but durable storage retained `{cached:true}`. | +| Restore unscoped root SQL calls | Rollback/isolation assertion expected `[]` but read uncommitted row 1. This is fixture calibration. | + +The three metadata controls replayed the same property with seed `2220031`, +path `0:0:0`; the materialization control ran on both hosts. None failed by +setup error, timeout or zero-test selection. Temporary production/model/fixture +edits were restored byte-for-byte, then both direct replays passed. Production +still equals main. No original budget, guard or comparison was removed. + +Commands from `packages/db-sqlite-persistence-core` (all Vitest invocations also +used `--maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false`): + +```sh +pnpm exec vitest run --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false tests/sync-metadata-session-persisted.test.ts +pnpm exec vitest run --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false --silent=false --disableConsoleIntercept tests/sync-metadata-session-sqlite.test.ts tests/sync-metadata-sqlite-driver.test.ts tests/sync-metadata-write-work.test.ts +``` + +The core command from `packages/db` used the same flags and +`tests/sync-metadata-session-oracle.property.test.ts`, +`tests/sync-metadata-session.test.ts`, and +`tests/collection-row-metadata-composition-oracle.test.ts`. +For each metadata calibration and restored replay, set +`TANSTACK_DB_ORACLE_PROPERTY=sync-metadata-session.persisted-kept-metadata-durability` +or `sync-metadata-session.sqlite-kept-metadata-durability`, +`TANSTACK_DB_ORACLE_SEED=2220031`, `TANSTACK_DB_ORACLE_PATH=0:0:0`, and select +`-t 'tab-local and durable metadata distinct'` in the matching host file. +For direct replay of the shared runner, use property +`sync-metadata-session.persisted-seed-reinsert-reload` or +`sync-metadata-session.sqlite-seed-reinsert-reload`, seed `527962096`, path `0`, +and select `-t 'inserted metadata across peer reloads'` or +`-t 'histories through SQLite'` in the matching host file. Each replay ran one +history; it establishes replay selection, not full campaign reach. + +Scoped types used `pnpm exec tsc --project /private/tmp/tanstack-db-p6-2026-11-02/scoped-types.json`: the package's config, +limited to the changed model/receiver/support files, with source aliases for +DB, DB IVM, Store and the broadcast-coordinator entry. Logs, exact calibration +commands/source edits, restored hashes, raw PR/issue snapshots, reviews and the +append-only finding ledger are retained in +`/private/tmp/tanstack-db-p6-2026-11-02/`. Those scratch artifacts are not PR files. +Post-extraction runs use the `runner-` log prefix; repeated calibration logs, +exact mutation commands and restored hashes are in its `runner-calibration/` +directory. The earlier runs and ledger entries remain intact. + +The historical ORC-001 through ORC-016 assessment still describes the unchanged +semantic model/grammar/comparison and receiver behavior. This new baseline was +independently reviewed; current runtime, reach and calibration results above +replace no historical execution. The accepted extraction moves the same +campaign loop into a support module and retains both host-specific production +drivers. Exact SHA-256 identities at this cut: + +- `packages/db-sqlite-persistence-core/src/persisted.ts`: `1a3bc05bbf10646c8d75fb062799c8335a181a8d3a5e91d61a9244c1a4f5027a`. +- `packages/db/tests/sync-metadata-session-oracle.ts`: `584d767b43aa7a9f4661c65fa431ac579d0037e5e1fa8415e693817b0dae635c`. +- `packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts`: `e63459f1dfa5721ab3aa154771f3c0d2ba3c1f3f3ddf2305a5290f5b0527d071`. +- `packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts`: `66cd18bc436d3358cd3f4f4aee577d0dd24154a581df7948b52b2a1bb2987a16`. +- `packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts`: `6c96e1f4b498a8b14fcd9835d6926c1daee4e4c33e3fa5e007f84a2d4b71cc46`. +- `packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts`: `99ec1af0f43f4885558c61296c66f2a42226fae7be58b32efaa970c831cde059`. + +## Limits and readiness + +The shared judgment checks source metadata after authored actions, and rows, +receipts, durable rows and durable metadata after settlement. The constructive +matrix covers update, flat equal re-insert and inner equal re-insert; bare and +metadata-bearing seeds; implicit and explicit metadata; partial/full/on-demand +lanes. Notice choices are sampled rather than a Cartesian guarantee. + +Native evidence uses one process, two Collections and Browser coordinators, +Node BroadcastChannel/Web Locks and one SQLite connection with serialized root +admission. Each correlated commit/notice/recovery finishes before the next +authored action. Arbitrary response/application interleavings, recovery during +cleanup/expiry/cancellation, leadership or transport loss, Chromium OPFS, +Electron IPC, restart semantics and necessity of gap detection remain outside +this evidence. Existing controlled store/read-hold lanes remain intact. + +The published PR is **not ready** while it contains the withdrawn production +change. All optional prep suggestions are now addressed. Replacement-head +readiness remains unproven until the commit/documentation gate, publication +and final-head CI are resolved. No issue closure or merge is authorized by +this record. + +During validation the shared `origin/main` reference advanced to +`30e85464ed2b9b12c5f10981a062427a1fc8daaa` (#2316). The tested baseline remains +`a3c4d0676621cc7df4d242bdfe31815509add734`. That new commit changes the coverage +map and replay configuration as well as other subsystems; integrate it before +publishing the replacement. The persistence production source and shared +metadata model are unchanged between those main commits. The results above +do not claim validation of the future integrated head. diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts new file mode 100644 index 0000000000..66b95b6ba2 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts @@ -0,0 +1,99 @@ +import { fc } from '@fast-check/vitest' +import { + completedSeedReinsertReloadRegime, + describeMismatch, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + seedReinsertReloadHistoryArbitrary, +} from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleRuns, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import type { + SessionAction, + SessionLane, + runSessionHistory, +} from '../../db/tests/sync-metadata-session-oracle' + +/** + * Both receiving drivers execute the same generated campaign. The shared + * oracle owns its histories, completed-state classifier and comparison; + * each host supplies its unchanged runHistory and observation checkpoint. + * Registration and direct replay selection remain in each host test file. + */ +export async function runSeedReinsertReloadCampaign({ + property, + campaign, + runHistory, +}: { + property: string + campaign: `fixed` | `random` + runHistory: ( + history: ReadonlyArray, + lane: SessionLane, + ) => ReturnType +}): Promise { + const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const states = lanes.flatMap((lane) => [ + `${lane.name}: inner commit`, + `${lane.name}: row`, + `${lane.name}: reload`, + ]) + const completedStates = lanes.flatMap((lane) => [ + `${lane.name}: completed kept bare seed reload`, + `${lane.name}: completed kept non-bare seed reload`, + ]) + const reach = createOracleReach(property, states) + const completedReach = createOracleReach(property, completedStates) + const result = await fc.check( + fc.asyncProperty(seedReinsertReloadHistoryArbitrary, async (history) => { + const failures: Array = [] + for (const lane of lanes) { + reach.beginHistory() + completedReach.beginHistory() + const result = await runHistory(history, lane) + const last = history.at(-1)! + const innerCommit = history.findIndex( + (action) => action.type === `commit`, + ) + if (result.steps > innerCommit) + reach.reach(`${lane.name}: inner commit`) + if (last.type === `peer` && result.steps === history.length) + reach.reach(`${lane.name}: ${last.notice}`) + const completed = completedSeedReinsertReloadRegime(history, result) + if (completed !== undefined) + completedReach.reach(`${lane.name}: ${completed}`) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }), + campaign === `fixed` + ? { numRuns: oracleRuns(40), seed: 527962096 } + : oraclePropertyOptions(40, property), + ) + recordOracleReplay(property, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + // The bare-key-1 conjunction has probability 1/24: random40 would miss it + // about 18% of the time. The fixed40 campaign enforces completed branch + // guards; the identical random40 campaign measures them. Existing + // inner/row/reload guards apply to both campaigns. + if (campaign === `fixed`) completedReach.assertReached() +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index 7e50570609..b69d46ce78 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -7,6 +7,7 @@ import { cachedRow, describeMismatch, historyRegimes, + keptMetadataDurabilityHistoriesArbitrary, persistedCachedLane, persistedConflictLane, persistedFlatLane, @@ -19,16 +20,20 @@ import { persistedReadSeedLane, persistedSeedLane, runSessionHistory, + seedReinsertReloadReplay, sessionCollectionKeys, sessionHistoryArbitrary, } from '../../db/tests/sync-metadata-session-oracle' import { oraclePropertyOptions, + oracleReplayTest, oracleRuns, readOracleRunConfig, + recordOracleReplay, } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' +import { runSeedReinsertReloadCampaign } from './sync-metadata-session-campaign' import type { CollectionConfig, SyncConfig } from '@tanstack/db' import type { PeerNotice, @@ -195,6 +200,7 @@ function createHoldingAdapter(cached = false, stores = false) { resetEpoch: () => resetEpoch, /** The row storage holds for a key, if any. */ storedRow: (key: number): Row | undefined => stored.get(key)?.value, + storedRowMetadata: (key: number): unknown => stored.get(key)?.metadata, holdNextStore: () => { holdNext = true }, @@ -475,6 +481,10 @@ async function createPersistedDriver( 1: read(store.storedRow(1)), 2: read(store.storedRow(2)), }), + storedRowMetadata: () => ({ + 1: read(store.storedRowMetadata(1)), + 2: read(store.storedRowMetadata(2)), + }), observe: () => ({ rowMetadata: { 1: read(metadata().row.get(1)), @@ -1209,29 +1219,14 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla it.each(peerFullReplays)(`replays under full rows: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerFullLane), ) - // Known gap, pinned from the persisted-peers-full random campaign (seed - // 654774679, shrunk to this prefix): an outer transaction inserts over a - // bare seed, an inner one re-inserts the row and commits, and a peer's full - // reload then leaves row 1 with the cached seed's metadata, where the model - // expects none. This reproduces on main and is tracked with #2281. It - // replays through the same model, driver, and comparison. Remove `fails` - // when the reload stops restoring that metadata. - it.fails( - `replays under full rows: a reload after an inner re-insert over a seed clears the seed's metadata`, - () => - assertPersistedHistory( - [ - { type: `begin` }, - { type: `seed`, key: 1, narrow: false, bare: true }, - { type: `insertOverSeed`, key: 1, metadata: false }, - { type: `begin` }, - { type: `reinsert`, key: 1, metadata: false }, - { type: `commit` }, - { type: `peer`, key: 2, notice: `reload` }, - ], - persistedPeerFullLane, - ), - ) + // Pinned from the persisted-peers-full random campaign (seed 654774679): + // an outer transaction inserts over a bare seed and an inner equal re-insert + // commits first. Its implicit metadata keep writes no metadata to storage; + // the shared model judges source reads and the later unknown-author reload + // separately under D9, D15 and D18. This uses the campaigns' model, driver, + // and comparison. + it(`replays under full rows: keeps source metadata reads consistent across an inner re-insert and reload`, () => + assertPersistedHistory(seedReinsertReloadReplay, persistedPeerFullLane)) it.each(peerOnDemandReplays)(`replays on demand: %s`, (_name, history) => assertPersistedHistory(history, persistedPeerOnDemandLane), ) @@ -1241,48 +1236,159 @@ describe(`sync metadata session oracle through the persisted wrapper: peer repla ) }) -describe(`sync metadata session oracle through the persisted wrapper: nesting and seeds`, () => { - for (const { - lane, - property: lanePropertyName, - seed, - runs, - reach: states, - } of extraLanes) { - const runLane = (history: Array) => - assertPersistedHistory(history, lane) - if (requestedReplayProperty === undefined) { - const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { - minimum: 5, - }) - fcTest.prop([sessionHistoryArbitrary(lane)], { - numRuns: oracleRuns(runs), - seed, - })( - `keeps the session law on ${lane.name} histories (fixed)`, - (history) => { - reach.beginHistory() - for (const regime of historyRegimes(lane, history)) - if (states.includes(regime)) reach.reach(regime) - return runLane(history) +// The broad peer campaigns can already generate this interaction. This constructive +// grammar measures its scarce overlap and adjacent legal histories directly; +// it uses the existing independent model, driver, and per-action comparison. +const seedReloadProperty = `sync-metadata-session.persisted-seed-reinsert-reload` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requestedReplayProperty !== undefined && + (requestedReplayProperty !== seedReloadProperty || campaign === `fixed`) + ) + continue + it(`keeps inserted metadata across peer reloads (${campaign})`, async () => { + await runSeedReinsertReloadCampaign({ + property: seedReloadProperty, + campaign, + runHistory: (history, lane) => + runSessionHistory(lane, history, () => + createPersistedDriver( + true, + true, + false, + true, + lane.fullRows === true, + lane.onDemand === true, + ), + ), + }) + }, 120_000) +} + +const describeNested = oracleReplayTest( + describe, + ...extraLanes.map(({ property }) => property), +) +describeNested( + `sync metadata session oracle through the persisted wrapper: nesting and seeds`, + () => { + for (const { + lane, + property: lanePropertyName, + seed, + runs, + reach: states, + } of extraLanes) { + const runLane = (history: Array) => + assertPersistedHistory(history, lane) + if (requestedReplayProperty === undefined) { + const reach = createOracleReach(`${lanePropertyName} (fixed)`, states, { + minimum: 5, + }) + fcTest.prop([sessionHistoryArbitrary(lane)], { + numRuns: oracleRuns(runs), + seed, + })( + `keeps the session law on ${lane.name} histories (fixed)`, + (history) => { + reach.beginHistory() + for (const regime of historyRegimes(lane, history)) + if (states.includes(regime)) reach.reach(regime) + return runLane(history) + }, + 240_000, + ) + it(`reaches the declared states in the ${lane.name} fixed campaign`, () => + reach.assertReached()) + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )( + `keeps the session law on ${lane.name} histories (random)`, + runLane, + 120_000, + ) + } else if (requestedReplayProperty === lanePropertyName) { + fcTest.prop( + [sessionHistoryArbitrary(lane)], + oraclePropertyOptions(50, lanePropertyName), + )(`replays ${lane.name} session histories`, runLane, 120_000) + } + } + }, +) +/** + * Direct durability is a separate cut from this tab's metadata read. Each + * generated batch crosses implicit keeps with explicit writes and updates + * with flat/inner equal re-inserts, before and after named/unknown reloads. + */ +const durabilityProperty = `sync-metadata-session.persisted-kept-metadata-durability` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requestedReplayProperty !== undefined && + (requestedReplayProperty !== durabilityProperty || campaign === `fixed`) + ) + continue + it(`keeps tab-local and durable metadata distinct (persisted, ${campaign})`, async () => { + const durabilityLanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const states = durabilityLanes.flatMap((lane) => + [`update`, `reinsert`, `inner-reinsert`].flatMap((write) => + [false, true].flatMap((bare) => + [false, true].map( + (metadata) => + `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, + ), + ), + ), + ) + const reach = createOracleReach(durabilityProperty, states) + const result = await fc.check( + fc.asyncProperty( + keptMetadataDurabilityHistoriesArbitrary, + async (batch) => { + const failures: Array = [] + for (const lane of durabilityLanes) { + for (const { write, bare, metadata, history } of batch) { + reach.beginHistory() + const result = await runSessionHistory(lane, history, () => + createPersistedDriver( + true, + true, + false, + true, + lane.fullRows === true, + lane.onDemand === true, + ), + ) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + else if (result.steps === history.length) + reach.reach( + `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, + ) + } + } + if (failures.length) throw new Error(failures.join(`\n\n`)) }, - 240_000, - ) - it(`reaches the declared states in the ${lane.name} fixed campaign`, () => - reach.assertReached()) - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )( - `keeps the session law on ${lane.name} histories (random)`, - runLane, - 120_000, + ), + campaign === `fixed` + ? { numRuns: oracleRuns(8), seed: 2220031 } + : oraclePropertyOptions(8, durabilityProperty), + ) + recordOracleReplay(durabilityProperty, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property: durabilityProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, ) - } else if (requestedReplayProperty === lanePropertyName) { - fcTest.prop( - [sessionHistoryArbitrary(lane)], - oraclePropertyOptions(50, lanePropertyName), - )(`replays ${lane.name} session histories`, runLane, 120_000) - } - } -}) + reach.assertReached() + }, 240_000) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts new file mode 100644 index 0000000000..31dc36533b --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -0,0 +1,564 @@ +// @vitest-environment node +/** + * The sync metadata session oracle's seed/re-insert/reload law crosses real + * SQLite and coordinator boundaries here. The law, model, legal histories, + * and comparison are imported from sync-metadata-session-oracle.ts. This + * driver supplies two persisted Collections, two production coordinators over + * Node's BroadcastChannel/Web Locks, and one node:sqlite database. Each + * Collection has its own adapter; their shared physical database supplies the + * shared scheduler premise. + * + * A peer's named notice comes from its real source commit. For a full reload, + * that source takes a durable snapshot, changes the requested row, and commits + * a snapshot transaction. This preserves every other durable row and causes + * production to announce requiresFullReload. The receiver records the actual + * notice and waits for SQL/message quiescence before the shared comparison. + * Neither this driver nor the peer invents an expected metadata value. + * + * Limits: one process, one SQLite database, two Collections of one ID, no + * transport loss or leadership turnover, and core holds only. This is a real + * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. + */ +import { DatabaseSync } from 'node:sqlite' +import { fc } from '@fast-check/vitest' +import { it } from 'vitest' +import { DbClient, collectionOptions, whenSyncAccepted } from '@tanstack/db' +import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' +import { + absentMarker, + cachedMetadata, + cachedRow, + describeMismatch, + keptMetadataDurabilityHistoriesArbitrary, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + runSessionHistory, + seedReinsertReloadReplay, + sessionCollectionKeys, +} from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, + readOracleRunConfig, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + eagerStorageTarget, + managedStorageTarget, + persistedCollectionOptions, +} from '../src' +import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' +import { runSeedReinsertReloadCampaign } from './sync-metadata-session-campaign' +import type { SyncConfig } from '@tanstack/db' +import type { + ReceiptState, + SessionAction, + SessionDriver, + SessionLane, + SessionRecoveryInput, +} from '../../db/tests/sync-metadata-session-oracle' + +type Row = { id: number; v: unknown } +type Actions = Parameters[`sync`]>[0] +let run = 0 + +async function createSQLiteDriver(lane: SessionLane): Promise { + const database = new DatabaseSync(`:memory:`) + const { driver, drained } = createMetadataSQLiteDriver(database) + Object.defineProperty(driver, SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, { + value: {}, + }) + const adapters = [ + new SQLiteCorePersistenceAdapter({ driver }), + new SQLiteCorePersistenceAdapter({ driver }), + ] + const id = `metadata-items` + const seedClaim = lane.onDemand + ? await adapters[0]!.claimCacheGeneration(id) + : undefined + const target = seedClaim + ? managedStorageTarget(seedClaim.storageCollectionId, seedClaim.claimId) + : eagerStorageTarget(id) + const storageId = seedClaim?.storageCollectionId ?? id + await adapters[0]!.applyCommittedTx(target, { + txId: `cached`, + term: 1, + seq: 1, + rowVersion: 1, + cacheGenerationClaimId: seedClaim?.claimId, + mutations: [{ type: `insert`, key: 1, value: cachedRow }], + rowMetadataMutations: [{ type: `set`, key: 1, value: cachedMetadata }], + }) + const dbName = `sync-metadata-receiving-${process.pid}-${run++}` + const coordinators = adapters.map( + (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), + ) + // Observe an actual transport input, never a production reload flag or a + // metadata mismatch. Without replay details this response cannot distinguish + // earlier cached writes from this run's writes; D15 governs that information + // gap. The model derives its effect solely from its own durable storage. + const recoveryInputs: Array = [] + const pullSince = coordinators[0]!.pullSince!.bind(coordinators[0]) + coordinators[0]!.pullSince = async (...args) => { + const response = await pullSince(...args) + if (response.ok && (response.requiresFullReload || !response.deltas)) + recoveryInputs.push({ + type: `unknown-storage-observation`, + fromRowVersion: args[1], + latestRowVersion: response.latestRowVersion, + }) + return response + } + const pendingEnsures = new Set>() + for (const coordinator of coordinators) { + const request = coordinator.requestEnsureRemoteSubset.bind(coordinator) + coordinator.requestEnsureRemoteSubset = (...args) => { + const pending = request(...args) + pendingEnsures.add(pending) + const done = () => pendingEnsures.delete(pending) + void pending.then(done, done) + return pending + } + } + let lastPeerTxId: string | undefined + const sourceTxIds: Array = [] + const sourceApply = coordinators[0]!.requestApplyCommittedTx.bind( + coordinators[0], + ) + coordinators[0]!.requestApplyCommittedTx = (...args) => { + sourceTxIds.push(args[1].txId) + return sourceApply(...args) + } + const peerApply = coordinators[1]!.requestApplyCommittedTx.bind( + coordinators[1], + ) + coordinators[1]!.requestApplyCommittedTx = (...args) => { + lastPeerTxId = args[1].txId + return peerApply(...args) + } + const notices: Array<{ reload: boolean; txId: string }> = [] + const noticeWaiters = new Set<() => void>() + const actions: Array = [] + let durableRows: ReadonlyArray<{ + key: string | number + value: unknown + metadata?: unknown + }> = [] + const stopNotices = coordinators[0]!.subscribe(storageId, (message) => { + const payload = message.payload as { + type: string + txId: string + requiresFullReload?: boolean + } + if (payload.type === `tx:committed`) { + notices.push({ + reload: payload.requiresFullReload === true, + txId: payload.txId, + }) + for (const wake of noticeWaiters) wake() + } + }) + const quiesce = async () => { + // Each empty source commit enters the wrapper after already delivered + // notices. Its acceptance drains that wrapper's apply lock without + // waiting for visibility behind a core hold. No write or notice results. + do { + // An on-demand cached load may settle before its upstream ensure. Wait + // for those actual coordinator replies before draining their receiver. + await Promise.all([...pendingEnsures]) + for (const source of actions) { + source.begin() + await whenSyncAccepted(source.commit()) + } + await adapters[0]!.runInRegularScope.afterQueuedWork() + await drained() + } while (pendingEnsures.size > 0) + durableRows = await adapters[0]!.loadSubset(target, {}) + } + const awaitNotice = (after: number, reload: boolean, txId: string) => + new Promise((resolve, reject) => { + const timer = setTimeout(() => { + noticeWaiters.delete(check) + reject( + new Error(`the real coordinator did not deliver the expected notice`), + ) + }, 5_000) + const check = () => { + if ( + !notices + .slice(after) + .some((entry) => entry.reload === reload && entry.txId === txId) + ) + return + clearTimeout(timer) + noticeWaiters.delete(check) + resolve() + } + noticeWaiters.add(check) + check() + }) + let resolveMutation: (() => void) | undefined + const clients = [new DbClient(), new DbClient()] + const collections = clients.map((client, index) => + client.collection( + collectionOptions( + persistedCollectionOptions({ + id, + getKey: (row) => row.id, + ...(lane.onDemand ? { syncMode: `on-demand` as const } : {}), + sync: { + rowUpdateMode: lane.fullRows ? `full` : `partial`, + sync: (sync) => { + actions[index] = sync + sync.begin() + sync.commit() + sync.markReady() + return lane.onDemand ? { loadSubset: () => true } : undefined + }, + }, + onInsert: () => + new Promise((resolve) => { + resolveMutation = resolve + }), + persistence: { + adapter: adapters[index]!, + coordinator: coordinators[index]!, + }, + }), + ), + ), + ) + // These direct source histories have no live query to start on-demand sync. + // Start both sources before a follower can route its first demand to its peer. + for (const current of collections) current.startSyncImmediate() + if (!lane.onDemand) + await Promise.all(collections.map((collection) => collection.preload())) + const demands = collections.map(() => ({})) + if (lane.onDemand) + await Promise.all( + collections.map((collection, index) => + collection._sync.loadSubset(demands[index]!), + ), + ) + await quiesce() + const sync = actions[0]! + const metadata = sync.metadata! + const collection = collections[0]! + const receipts: Array> = [] + const receiptStates: Array = [] + const rejections: Array = [] + let depth = 0 + let holdCount = 0 + let blocker: { when: (state: 'settled') => Promise } | undefined + const release = async () => { + resolveMutation?.() + resolveMutation = undefined + await blocker?.when(`settled`) + blocker = undefined + } + const abort = () => { + const controller = new AbortController() + controller.abort() + const receipt = sync.commit(controller.signal) + depth-- + if (receipt !== true) + receipts.push( + receipt.catch((error: unknown) => { + if (!(error instanceof Error) || error.name !== `AbortError`) + rejections.push(String(error)) + }), + ) + } + const read = (value: unknown) => (value === undefined ? absentMarker : value) + const sorted = (entries: ReadonlyArray<{ key: string; value: unknown }>) => + entries + .map(({ key, value }): [string, unknown] => [key, value]) + .sort(([a], [b]) => a.localeCompare(b)) + const rowMetadata = (value: unknown) => + value === undefined ? {} : { metadata: value as Record } + return { + consumerCheckpoint: `final`, + begin: () => { + sync.begin() + depth++ + }, + commit: async () => { + const noticeStart = notices.length + const transactionStart = sourceTxIds.length + const index = receiptStates.length + const receipt = sync.commit() + depth-- + receiptStates.push(receipt === true ? `applied` : `pending`) + if (receipt !== true) + receipts.push( + receipt.then( + () => { + receiptStates[index] = `applied` + }, + () => { + receiptStates[index] = `rejected` + }, + ), + ) + // This bounded legal schedule completes the source's correlated notice + // and any resulting recovery before the next authored action. In + // particular, a peer write cannot enter the model ahead of a recovery + // that actually read the pre-peer durable state. Core holds may keep the + // applied receipt pending, so wait for acceptance rather than visibility. + await whenSyncAccepted(receipt) + for (const txId of sourceTxIds.slice(transactionStart)) + await awaitNotice(noticeStart, false, txId) + await quiesce() + }, + cancel: abort, + rowSet: (key, value) => metadata.row.set(key, value), + rowDelete: (key) => metadata.row.delete(key), + insert: (_key, value, meta) => + sync.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }), + update: (_key, value, meta) => + sync.write({ type: `update`, value: value as Row, ...rowMetadata(meta) }), + delete: (key) => sync.write({ type: `delete`, key }), + truncate: () => sync.truncate(), + collectionSet: (key, value) => metadata.collection.set(key, value), + collectionDelete: (key) => metadata.collection.delete(key), + hold: (store, readHold) => { + if (store || readHold) + throw new Error(`the SQLite seed/re-insert grammar uses core holds`) + blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) + }, + release, + seed: (key, value, meta) => + clients[0]!.hydrate({ + collections: [ + { + collectionId: id, + rows: [ + { + key, + value: value as Row, + metadata: meta as Record, + }, + ], + }, + ], + }), + peer: async (key, notice, value, meta) => { + await quiesce() + const before = notices.length + const peer = actions[1]! + if (notice === `reload`) { + const rows = await adapters[1]!.loadSubset(target, {}) + peer.begin({ snapshot: true }) + for (const row of rows) + if (row.key !== key) + peer.write({ + type: `insert`, + value: row.value as Row, + ...rowMetadata(row.metadata), + }) + } else if (notice === `row`) peer.begin() + else + throw new Error( + `unexpected peer notice in the seed/re-insert grammar: ${notice}`, + ) + peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) + await peer.commit() + if (!lastPeerTxId) + throw new Error(`the peer issued no durable transaction`) + await awaitNotice(before, notice === `reload`, lastPeerTxId) + await quiesce() + if ( + !notices + .slice(before) + .some((entry) => entry.reload === (notice === `reload`)) + ) + throw new Error( + `the real coordinator did not deliver the ${notice} notice`, + ) + console.info( + `SQLITE_METADATA_NOTICE ${JSON.stringify({ lane: lane.name, notice, received: notices.length - before })}`, + ) + }, + observe: () => ({ + rowMetadata: { + 1: read(metadata.row.get(1)), + 2: read(metadata.row.get(2)), + }, + collection: Object.fromEntries( + sessionCollectionKeys.map((key) => [ + key, + read(metadata.collection.get(key)), + ]), + ), + list: sorted(metadata.collection.list()), + prefixList: sorted(metadata.collection.list(`gc:`)), + base: { + 1: read(collection.base.get(1)), + 2: read(collection.base.get(2)), + }, + receipts: [...receiptStates], + }), + storedRows: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.value), + ]), + ), + storedRowMetadata: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.metadata), + ]), + ), + takeRecoveryInputs: () => recoveryInputs.splice(0), + flush: quiesce, + settle: async () => { + await release() + while (depth) abort() + await quiesce() + await Promise.all(receipts) + await quiesce() + return rejections + }, + cleanup: async () => { + await release() + while (depth) abort() + await Promise.all(receipts) + if (lane.onDemand) + for (const [index, current] of collections.entries()) + current._sync.unloadSubset(demands[index]!) + await quiesce() + await Promise.all(collections.map((current) => current.cleanup())) + stopNotices() + for (const coordinator of coordinators) coordinator.dispose() + if (seedClaim) + await adapters[0]!.releaseCacheGenerationClaim(seedClaim.claimId) + await drained() + database.close() + }, + } +} + +const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, +] +async function check(history: ReadonlyArray, lane: SessionLane) { + return runSessionHistory(lane, history, () => createSQLiteDriver(lane)) +} +oracleReplayTest(it).each(lanes)( + `receives the seed/re-insert/reload replay through SQLite ($name)`, + async (lane) => { + const result = await check(seedReinsertReloadReplay, lane) + if (result.mismatch) + throw new Error( + describeMismatch(seedReinsertReloadReplay, result.mismatch), + ) + }, +) + +const property = `sync-metadata-session.sqlite-seed-reinsert-reload` +const requested = readOracleRunConfig().replayProperty +for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== property || campaign === `fixed`) + ) + continue + it(`receives generated seed/re-insert/reload histories through SQLite (${campaign})`, async () => { + // The peer action asserts actual requiresFullReload delivery before the + // shared comparison passes; the campaign runner changes no host premise. + await runSeedReinsertReloadCampaign({ + property, + campaign, + runHistory: check, + }) + }, 120_000) +} +/** + * Direct durability is a separate cut from this tab's metadata read. Each + * generated batch crosses implicit keeps with explicit writes and updates + * with flat/inner equal re-inserts, before and after named/unknown reloads. + */ +const durabilityProperty = `sync-metadata-session.sqlite-kept-metadata-durability` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== durabilityProperty || campaign === `fixed`) + ) + continue + it(`keeps tab-local and durable metadata distinct (sqlite, ${campaign})`, async () => { + const durabilityLanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const states = durabilityLanes.flatMap((lane) => + [`update`, `reinsert`, `inner-reinsert`].flatMap((write) => + [false, true].flatMap((bare) => + [false, true].map( + (metadata) => + `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, + ), + ), + ), + ) + const reach = createOracleReach(durabilityProperty, [ + ...states, + `on-demand: cached prior-run version crossed without replay details`, + ]) + const result = await fc.check( + fc.asyncProperty( + keptMetadataDurabilityHistoriesArbitrary, + async (batch) => { + const failures: Array = [] + for (const lane of durabilityLanes) { + for (const { write, bare, metadata, history } of batch) { + reach.beginHistory() + const result = await check(history, lane) + if ( + lane.onDemand && + result.recoveryInputs.some( + (input) => + input.fromRowVersion === 0 && input.latestRowVersion >= 2, + ) + ) + reach.reach( + `on-demand: cached prior-run version crossed without replay details`, + ) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + else if (result.steps === history.length) + reach.reach( + `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, + ) + } + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }, + ), + campaign === `fixed` + ? { numRuns: oracleRuns(8), seed: 2220031 } + : oraclePropertyOptions(8, durabilityProperty), + ) + recordOracleReplay(durabilityProperty, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property: durabilityProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + }, 240_000) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts new file mode 100644 index 0000000000..c15efbb6c9 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts @@ -0,0 +1,43 @@ +// @vitest-environment node +import { DatabaseSync } from 'node:sqlite' +import { expect, it } from 'vitest' +import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' + +it(`keeps another adapter's root reads and writes outside a rolled-back transaction`, async () => { + const database = new DatabaseSync(`:memory:`) + const { driver, drained } = createMetadataSQLiteDriver(database) + let entered!: () => void + const started = new Promise((resolve) => { + entered = resolve + }) + let release!: () => void + const gate = new Promise((resolve) => { + release = resolve + }) + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + const failed = driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (1)`) + entered() + await gate + throw new Error(`rollback this transaction only`) + }) + const rejected = expect(failed).rejects.toThrow( + `rollback this transaction only`, + ) + await started + const read = driver.query(`SELECT id FROM rows ORDER BY id`) + const write = driver.run(`INSERT INTO rows VALUES (2)`) + release() + await rejected + expect(await read).toEqual([]) + await write + expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ + { id: 2 }, + ]) + } finally { + release() + await drained() + database.close() + } +}) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts new file mode 100644 index 0000000000..88ebaa2d66 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts @@ -0,0 +1,48 @@ +import type { DatabaseSync, SQLInputValue } from 'node:sqlite' +import type { SQLiteDriver } from '../src' + +/** + * One SQLite connection admits root calls in order. A transaction's scoped + * driver executes its calls inside that admission, so another adapter's read + * cannot observe it before commit or become part of its rollback. + */ +export function createMetadataSQLiteDriver(database: DatabaseSync) { + let queue: Promise = Promise.resolve() + const enqueue = (work: () => Promise): Promise => { + const next = queue.then(work) + queue = next.catch(() => undefined) + return next + } + const scoped: SQLiteDriver = { + exec: async (sql) => database.exec(sql), + query: async (sql: string, params: ReadonlyArray = []) => + database + .prepare(sql) + .all(...(params as Array)) as Array, + run: async (sql, params = []) => { + database.prepare(sql).run(...(params as Array)) + }, + transaction: async (work) => work(scoped), + } + const transaction: SQLiteDriver[`transaction`] = (work) => + enqueue(async () => { + database.exec(`BEGIN IMMEDIATE`) + try { + const result = await work(scoped) + database.exec(`COMMIT`) + return result + } catch (error) { + database.exec(`ROLLBACK`) + throw error + } + }) + const driver: SQLiteDriver = { + exec: (sql) => enqueue(() => scoped.exec(sql)), + query: (sql: string, params?: ReadonlyArray) => + enqueue(() => scoped.query(sql, params)), + run: (sql, params) => enqueue(() => scoped.run(sql, params)), + transaction, + transactionWithDriver: transaction, + } + return { driver, drained: () => queue } +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts new file mode 100644 index 0000000000..880dd11f67 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts @@ -0,0 +1,100 @@ +import { createCollection } from '@tanstack/db' +import { expect, it } from 'vitest' +import { fc } from '@fast-check/vitest' +import { + completedSeedReinsertReloadRegime, + keptMetadataDurabilityHistoriesArbitrary, + seedReinsertReloadHistoryArbitrary, +} from '../../db/tests/sync-metadata-session-oracle' +import { persistedCollectionOptions } from '../src' +import type { SyncConfig } from '@tanstack/db' +import type { PersistedTx, PersistenceAdapter } from '../src' + +type Row = { id: number; v: number } +type Actions = Parameters[`sync`]>[0] + +it(`measures the previous scarce branch and the constructive replacement`, () => { + for (const seed of [-704889444, 527962096, 1, 2, 3]) { + const old = fc.sample(seedReinsertReloadHistoryArbitrary, { + seed, + numRuns: 40, + }) + const count = old.filter( + (history) => + completedSeedReinsertReloadRegime(history, { + mismatch: undefined, + steps: history.length, + }) === `completed kept bare seed reload`, + ).length + const next = fc.sample(keptMetadataDurabilityHistoriesArbitrary, { + seed, + numRuns: 8, + }) + const bareImplicit = next + .flat() + .filter((entry) => entry.bare && !entry.metadata).length + console.info( + `METADATA_GENERATOR ${JSON.stringify({ seed, oldBareCompletedPremise: count, newBareImplicitHistories: bareImplicit })}`, + ) + expect(bareImplicit).toBe(24) + } +}) + +it.each([2, 100, 10_000])( + `counts implicit keeps in an unchanged %i-row refresh`, + async (count) => { + const transactions: Array = [] + const adapter: PersistenceAdapter = { + loadSubset: async () => [], + loadResumeSnapshot: async () => ({ + rows: [], + keySet: { status: `consistent` }, + collectionMetadata: [], + latestTerm: 0, + latestSeq: 0, + latestRowVersion: 0, + resetEpoch: 0, + }), + applyCommittedTx: async (_target, tx) => { + transactions.push(tx) + }, + ensureIndex: async () => {}, + } + let source!: Actions + const collection = createCollection( + persistedCollectionOptions({ + id: `external-work-${count}`, + getKey: (row) => row.id, + sync: { + sync: (actions) => { + source = actions + actions.markReady() + }, + }, + persistence: { adapter }, + }), + ) + try { + await collection.preload() + source.begin() + for (let id = 0; id < count; id++) + source.write({ + type: `insert`, + value: { id, v: id }, + metadata: { v: id }, + }) + await source.commit() + source.begin() + for (let id = 0; id < count; id++) + source.write({ type: `insert`, value: { id, v: id } }) + await source.commit() + const latest = transactions.at(-1)! + console.info( + `METADATA_WORK ${JSON.stringify({ count, rowMutations: latest.mutations.length, metadataMutations: latest.rowMetadataMutations?.length ?? 0 })}`, + ) + expect(latest.rowMetadataMutations?.length ?? 0).toBe(0) + } finally { + await collection.cleanup() + } + }, +) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index f851ae34e8..0c65c4937c 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -241,6 +241,10 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-peers`, `sync-metadata-session.persisted-peers-full`, `sync-metadata-session.persisted-peers-on-demand`, + `sync-metadata-session.persisted-seed-reinsert-reload`, + `sync-metadata-session.sqlite-seed-reinsert-reload`, + `sync-metadata-session.persisted-kept-metadata-durability`, + `sync-metadata-session.sqlite-kept-metadata-durability`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, `d2-source.exact-retractions`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 498cfb0ad8..7b0d172160 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -656,7 +656,12 @@ const ownerGroups: ReadonlyArray< [ `db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts`, `sync-metadata-session`, - `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand`, + `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand persisted-seed-reinsert-reload persisted-kept-metadata-durability`, + ], + [ + `db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts`, + `sync-metadata-session`, + `sqlite-seed-reinsert-reload sqlite-kept-metadata-durability`, ], ] diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 6e7772325a..8977e19209 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -63,6 +63,10 @@ import { declareOracle } from './oracle-declaration.js' * a seed supplied it is an insert. Among seeds of a row that no source * write has written, the latest seed wins whole, metadata included * (`db-client-hydration-authority-oracle.test.ts`). + * - Keeping metadata is not writing it (D9 and D18). Updates and equal + * re-inserts without explicit metadata leave durable metadata alone. A + * named current-run write is source-written (D11); under unknown writes, + * differing stored metadata may become source-written (D15 and D18). * * Authority: maintainer decisions of 2026-10-22 to borrow standard theory and * to place seeds beneath source writes (D4, D5, and its clarification "b"); @@ -71,6 +75,8 @@ import { declareOracle } from './oracle-declaration.js' * share the sync API with the source; the glossary * entries for accepted sync transactions and the exposed authoritative base; * and the row-metadata write rules of 2026-10-05 for implicit metadata. + * Decisions 9, 11, 14, 15 and 18 are recorded in + * `docs/contributing/oracle-reviews/2026-10-22-sync-metadata-session.md`. * * ## Limits * @@ -87,6 +93,12 @@ import { declareOracle } from './oracle-declaration.js' * source update of an absent row is not a legal history (the change-message * protocol), so it is not generated; it arises only when an inner * transaction's update commits ahead of the enclosing write it relied on. + * The native receiver also records actual successful pull responses without + * replay details. It completes each source commit's correlated notice and + * recovery before the next authored action, then compares the same model's + * D15 storage observation. Arbitrary response/application interleavings, + * recovery during cleanup, and the necessity of gap detection are outside + * this bounded schedule. */ // --------------------------------------------------------------------------- @@ -582,6 +594,162 @@ export function sessionHistoryArbitrary( }) } +/** + * A source insert can be visible to an inner transaction before it commits. + * The inner equal re-insert keeps metadata without writing it (D9 and D18). + * A later named or unknown-author reload follows its own rules (D11, D15 and + * D18); keeping a value in this tab does not make that value durable. + * + * This constructive subgrammar rebuilds the #2281 metadata prefix and varies + * both keys, seed shape/metadata and arrival, explicit metadata, repeated + * re-inserts, a core hold, the enclosing transaction's ending, and named + * versus full-reload notices. Every action is legal in the peer lanes. Store + * and read holds are excluded because their pending work blocks peer notices. + * The general session grammar continues to cover their independent laws. + */ +export const seedReinsertReloadHistoryArbitrary: fc.Arbitrary< + Array +> = fc + .record({ + key: fc.constantFrom(...sessionRowKeys), + bare: fc.boolean(), + narrow: fc.boolean(), + seedInside: fc.boolean(), + outerMetadata: fc.boolean(), + innerMetadata: fc.boolean(), + repeats: fc.integer({ min: 1, max: 3 }), + hold: fc.boolean(), + ending: fc.constantFrom(`open`, `commit`, `cancel`), + notice: fc.constantFrom(`row` as const, `reload` as const), + }) + .map((input): Array => { + const seed: SessionAction = { + type: `seed`, + key: input.key, + bare: input.bare, + narrow: input.narrow, + } + return [ + ...(input.hold ? [{ type: `hold` as const }] : []), + ...(input.seedInside + ? [{ type: `begin` as const }, seed] + : [seed, { type: `begin` as const }]), + { type: `insertOverSeed`, key: input.key, metadata: input.outerMetadata }, + { type: `begin` }, + ...Array.from({ length: input.repeats }, (): SessionAction => ({ + type: `reinsert`, + key: input.key, + metadata: input.innerMetadata, + })), + { type: `commit` }, + ...(input.ending === `open` + ? [] + : [{ type: input.ending } as SessionAction]), + { type: `peer`, key: input.key === 1 ? 2 : 1, notice: input.notice }, + ] + }) + +/** The source report is input to the same model and comparison as generation. */ +export const seedReinsertReloadReplay: Array = [ + { type: `begin` }, + { type: `seed`, key: 1, narrow: false, bare: true }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `reload` }, +] + +/** + * D9 separates keeping metadata in this tab from writing it. D18 says that + * storing a row over a DbClient seed stores only the row; D15 lets an unknown + * reload expose differing durable metadata. Cross updates and equal re-inserts + * with implicit keeps and explicit writes before that later reload can erase + * the distinction. Every generated batch contains each conditional premise; + * the seed shape, timing, repeats, and final notice still vary. + */ +export const keptMetadataDurabilityHistoriesArbitrary = fc + .record({ + narrow: fc.boolean(), + seedInside: fc.boolean(), + repeats: fc.integer({ min: 1, max: 3 }), + notice: fc.constantFrom(`none`, `row`, `reload`), + }) + .map((input) => + ([`update`, `reinsert`, `inner-reinsert`] as const).flatMap((write) => + [false, true].flatMap((bare) => + [false, true].map((metadata) => { + const seed: SessionAction = { + type: `seed`, + key: 1, + bare, + narrow: input.narrow, + } + const history: Array = [ + ...(input.seedInside + ? [{ type: `begin` as const }, seed] + : [seed, { type: `begin` as const }]), + { + type: write === `inner-reinsert` ? `insertOverSeed` : `upsert`, + key: 1, + metadata: false, + }, + ] + if (write !== `update`) { + if (write === `reinsert`) history.push({ type: `commit` }) + history.push({ type: `begin` }) + for (let index = 0; index < input.repeats; index++) + history.push({ type: `reinsert`, key: 1, metadata }) + } else if (metadata) history.push({ type: `rowSet`, key: 1 }) + history.push({ type: `commit` }) + if (input.notice !== `none`) + history.push({ type: `peer`, key: 2, notice: input.notice }) + return { write, bare, metadata, history } + }), + ), + ), + ) + +/** + * This classifier records a completed implicit re-insert across a full reload + * while the enclosing write remains unaccepted. "Kept" names that implicit + * operation, not a promise to promote its metadata into storage. The shared + * model separately judges unknown-author observations under D15. This reach + * classifier uses only the grammar and completed comparison, never production + * metadata values. + * Only key 1 starts with cached metadata, so only its bare seed distinguishes + * a stale cache value from absence. Both receiving drivers use these premises. + */ +export function completedSeedReinsertReloadRegime( + history: ReadonlyArray, + result: { mismatch: SessionMismatch | undefined; steps: number }, +): + | `completed kept bare seed reload` + | `completed kept non-bare seed reload` + | undefined { + const last = history.at(-1)! + const seed = history.find((action) => action.type === `seed`) + const keptSeedReload = + last.type === `peer` && + last.notice === `reload` && + history.some((action) => action.type === `reinsert`) && + history.every( + (action) => action.type !== `reinsert` || action.metadata === false, + ) && + history.filter((action) => action.type === `commit`).length === 1 + if ( + !result.mismatch && + result.steps === history.length && + keptSeedReload && + seed?.type === `seed` + ) { + if (seed.bare === true && seed.key === 1) + return `completed kept bare seed reload` + if (seed.bare === false) return `completed kept non-bare seed reload` + } + return undefined +} + /** * The regimes that a legal history reaches, read from the model as it * replays. The campaign reach test requires each law's premise. @@ -1060,9 +1228,9 @@ export class SessionModel { } /** - * A committed transaction stores, for every row it wrote or truncated, the - * row and metadata this tab now shows; for a row whose metadata alone it - * wrote, that metadata. + * A committed transaction stores the row it wrote or truncated and only + * metadata that it wrote. An equal re-insert keeps metadata without writing + * it (D9), as an update does (D18); its kind stays fixed at commit (D14). */ private store(frame: number): void { const truncates = this.truncates(frame) @@ -1083,7 +1251,7 @@ export class SessionModel { (event) => !event.truncate && (event.store === `rowMetadata` || - event.rowWrite === `insert` || + (event.rowWrite === `insert` && event.kind !== `reinsert`) || !event.value.present), ) if (!writesRow && !writesMetadata) continue @@ -1242,6 +1410,16 @@ export class SessionModel { ) return } + this.observeUnknownStorage() + } + + /** + * D15: a storage observation without transaction authorship can establish + * differing durable objects as source writes. A recovery response with no + * replay details supplies the same information gap as an unnamed notice; + * it performs no peer write. The fold uses only this model's storage. + */ + observeUnknownStorage(): void { // A held seed keeps the peer's write from this tab until a notice names // the row; a write of the row by this tab stores this tab's row over it // (decision 17). @@ -1395,6 +1573,20 @@ export class SessionModel { ) } + /** Only metadata attached to a durable row survives a restart (D16). */ + storedRowMetadata(): Record { + return Object.fromEntries( + sessionRowKeys.map((key) => [ + key, + encode( + this.storedRow(key).present + ? this.storage.get(key)!.metadata + : absent, + ), + ]), + ) + } + /** * A seed arrives after every accepted transaction but beneath the open * transactions, whose writes have not yet committed. @@ -1733,7 +1925,7 @@ function encode(value: ModelValue): unknown { */ export type SessionDriver = { begin: () => void - commit: () => void + commit: () => void | Promise cancel: () => void rowSet: (key: SessionRowKey, value: unknown) => void rowDelete: (key: SessionRowKey) => void @@ -1775,6 +1967,10 @@ export type SessionDriver = { * reads them back into this tab. */ storedRows?: () => Record + /** Durable metadata, distinct from the source's tab-local seed reads. */ + storedRowMetadata?: () => Record + /** Actual host recovery inputs since the previous completed comparison. */ + takeRecoveryInputs?: () => Array /** Lets receipt continuations run before an observation. */ flush: () => Promise /** Ends any hold and resolves every receipt; returns unexpected rejections. */ @@ -1789,12 +1985,24 @@ export type SessionDriver = { cleanup: () => Promise } +/** + * Model abstraction: a successful pull response lacks transaction authorship. + * This records the provider input, not production's reload classification or + * observed metadata. Drivers must finish its application before returning it. + */ +export type SessionRecoveryInput = { + type: `unknown-storage-observation` + fromRowVersion: number + latestRowVersion: number +} + export type SessionMismatch = { step: number action: SessionAction | `settle` law: string expected: unknown actual: unknown + recoveryInputs?: ReadonlyArray } // --------------------------------------------------------------------------- @@ -1827,12 +2035,23 @@ export async function runSessionHistory( lane: SessionLane, history: ReadonlyArray, createDriver: () => Promise, -): Promise<{ mismatch: SessionMismatch | undefined; steps: number }> { +): Promise<{ + mismatch: SessionMismatch | undefined + steps: number + recoveryInputs: ReadonlyArray +}> { const model = createSessionModel(lane) const driver = await createDriver() let mismatch: SessionMismatch | undefined let steps = 0 let primary: unknown + const recoveryInputs: Array = [] + const applyRecoveryInputs = () => { + for (const input of driver.takeRecoveryInputs?.() ?? []) { + recoveryInputs.push(input) + model.observeUnknownStorage() + } + } try { for (const [step, action] of history.entries()) { if (!isLegal(lane, model, action)) continue @@ -1857,6 +2076,7 @@ export async function runSessionHistory( break } await driver.flush() + applyRecoveryInputs() mismatch = compare( step, action, @@ -1870,6 +2090,7 @@ export async function runSessionHistory( const rejections = await driver.settle() if (model.isHolding) model.release() while (model.depth > 0) model.cancel() + applyRecoveryInputs() if (rejections.length > 0) mismatch = { step: history.length, @@ -1897,10 +2118,24 @@ export async function runSessionHistory( actual, } } + if (!mismatch && lane.peers === true && driver.storedRowMetadata) { + const expected = model.storedRowMetadata() + const actual = driver.storedRowMetadata() + if (canonical(expected) !== canonical(actual)) + mismatch = { + step: history.length, + action: `settle`, + law: `keeping metadata does not write it to durable storage (decisions 9 and 18)`, + expected, + actual, + } + } } } catch (error) { primary = error } + if (mismatch && recoveryInputs.length) + mismatch.recoveryInputs = recoveryInputs let cleanupFailure: unknown try { await driver.cleanup() @@ -1922,7 +2157,7 @@ export async function runSessionHistory( ) } if (primary !== undefined) throw primary - return { mismatch, steps } + return { mismatch, steps, recoveryInputs } } /** @@ -2511,6 +2746,9 @@ export function describeMismatch( `history: ${trace}`, `expected: ${JSON.stringify(mismatch.expected)}`, `actual: ${JSON.stringify(mismatch.actual)}`, + ...(mismatch.recoveryInputs?.length + ? [`host recovery inputs: ${JSON.stringify(mismatch.recoveryInputs)}`] + : []), ].join(`\n`) } @@ -2525,5 +2763,12 @@ declareOracle({ runSeedOrderInvariance, ], model: [SessionModel, createSessionModel, modelCheckpoints], - grammar: [sessionActionArbitrary, sessionHistoryArbitrary, historyRegimes], + grammar: [ + sessionActionArbitrary, + sessionHistoryArbitrary, + historyRegimes, + seedReinsertReloadHistoryArbitrary, + completedSeedReinsertReloadRegime, + keptMetadataDurabilityHistoriesArbitrary, + ], }) From 9eef6525d663a3ca586f1e0c5e8af4bab1c29dbb Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 11:36:57 -0700 Subject: [PATCH 04/11] docs: record integrated P6 publication evidence --- .../2026-11-02-p6-metadata-closeout.md | 60 ++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md index 3d4f78c0ac..c1cf126b77 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -45,7 +45,8 @@ Update the existing PR with ordinary follow-up history: merge current main, replace its net diff with this validated candidate, remove the obsolete materialization and release note, and rewrite the body around D9/D18 and the bounded receiving evidence. Preserve main's source-progress changes. No force -push or new package release is needed. This session has not updated the PR. +push or new package release is needed. This snapshot predates the approved +publication preparation recorded below. - **#2281:** open. Withdraw the old metadata product-repair claim; the row-loss and resurrection reports remain separate. @@ -226,3 +227,60 @@ map and replay configuration as well as other subsystems; integrate it before publishing the replacement. The persistence production source and shared metadata model are unchanged between those main commits. The results above do not claim validation of the future integrated head. + +## Approved publication preparation + +The maintainer approved the commit and PR-documentation gate on November 2. +Commit `a04d85519` contains the reviewed candidate. Merge `ea43b47c4` integrates +main at `30e85464ed2b9b12c5f10981a062427a1fc8daaa`. Merge +`41f8d99e975aa56ee18d6bca46b7193c602ca287` preserves the published #2303 ancestry +while keeping the corrected candidate tree. It supersedes the withdrawn +materialization and release note. No published history is rewritten. + +The following results test the production/test tree at +`41f8d99e975aa56ee18d6bca46b7193c602ca287`, before this documentation update: + +| Integrated check | Result | +| --- | --- | +| Complete controlled persisted receiver | 74 passed, 284.31 seconds | +| Native receiver, SQL fixture and emitted-work files | 12 passed, 4.53 seconds | +| Core session, boundary, row-metadata composition and replay-registry files | 162 passed, 178.37 seconds | +| Scoped TypeScript and changed-oracle lint | Passed | +| General ESLint | Zero errors, 12 warnings | +| Shared campaign direct replay | One requested history passed per host, seed `527962096`, path `0` | + +The core command adds `tests/oracle-replay.test.ts` to the three core files +listed above. This exercises the merged registry and replay configuration. +The paired constructive campaigns again reached every one of the 36 +case/lane states eight times. Native prior-cache recovery reached 96 histories +in each campaign. Fixed seed remains `2220031`; random seeds are `-985340019` +for the controlled receiver and `622373113` for native SQLite. + +Both 40-run fixed campaigns retain seed `527962096` and the same per-lane +reach: 40 inner commits, 21 named notices, 19 full reloads and two completed +bare and non-bare kept-seed states each. Controlled random seed `570739933` +reached 40 inner commits, 19 named notices, 21 reloads, one completed bare +state and five non-bare states per lane. Native random seed `-237872140` +reached 40 inner commits, 21 named notices, 19 reloads, zero completed bare +states and two non-bare states per lane. That scarce random branch remains +measured; the fixed guards and constructive matrix supply guaranteed reach. + +The dependency refresh used the frozen main lockfile and the public npm +registry after the sandbox blocked registry downloads. It required no version +override or local manifest/lockfile change. Against the integrated main, +production source, manifests, the lockfile and release notes have zero delta. +No package release or changeset is appropriate for this test-only result. + +Integrated logs use the `integrated-` prefix in the same scratch directory. +`integrated-code-freeze.json` records the tested head and changed-file hashes; +`integrated-reach.json` retains every measured state count. The native +receiving limits and separate #2317/P8 witness remain unchanged. + +All five calibration controls were repeated on the integrated tree. They +failed at the same intended assertions listed above, with no setup failure or +timeout. Both restored durability replays passed at seed `2220031`, path +`0:0:0`. The model, production serializer and SQL fixture were restored +byte-for-byte. The `integrated-calibration/` directory retains exact commands, +mutated/restored hashes and assertion classifications. All 248 integrated +tests, replay checks and controls preserve the earlier bounded conclusion. +Final published-head CI and review remain a separate publication check. From d0c4d0dde04208f9aaa66a03dcb7bada9c4f7cdb Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 11:52:02 -0700 Subject: [PATCH 05/11] docs: clarify metadata receiver evidence and helpers --- .../2026-10-30-kept-metadata-durability.md | 4 +-- .../2026-11-02-p6-metadata-closeout.md | 25 +++++++++++++++++++ .../sync-metadata-session-persisted.test.ts | 5 ++++ .../sync-metadata-session-sqlite.test.ts | 5 ++++ .../db/tests/sync-metadata-session-oracle.ts | 5 ++++ 5 files changed, 42 insertions(+), 2 deletions(-) diff --git a/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md b/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md index 7cd0b71bd3..993300badf 100644 --- a/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md +++ b/docs/contributing/oracle-reviews/2026-10-30-kept-metadata-durability.md @@ -29,8 +29,8 @@ premise too rarely. The repair changes the independent model, grammar and observations, then restores the serializer already present on main. It does not add a production recovery algorithm. -The native path also exposed a model environment-input gap. A real successful -`pullSince(0)` returned row version 2 without transaction replay details. The +The native path also exposed a model environment-input gap. A successful +`pullSince(0)` response returned row version 2 without transaction replay details. The shared model now records that response input and applies the existing D15 rule to its own storage. It derives no expected metadata from production output or the wrapper's reload classifier. Earlier suspicion of a production recovery diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md index c1cf126b77..580bbce6ce 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -284,3 +284,28 @@ byte-for-byte. The `integrated-calibration/` directory retains exact commands, mutated/restored hashes and assertion classifications. All 248 integrated tests, replay checks and controls preserve the earlier bounded conclusion. Final published-head CI and review remain a separate publication check. + +## CodeRabbit documentation follow-up + +Review `5741628374` and mutable summary `6507648970` cover published head +`9eef6525d663a3ca586f1e0c5e8af4bab1c29dbb`. The review contains one minor +wording suggestion and no inline comments. The summary also reports 57.14% +docstring coverage against its 80% threshold, without naming deficient functions. + +| ID | Evaluation and repair | Evidence and retained value | +| --- | --- | --- | +| META-CR-5741628374-001 | Clarified the successful `pullSince(0)` response sentence. The proposed sentence preserves the measured input; calling `real` an adverb was unnecessary. | Direct comparison of the historical sentence and replacement; no evidence or contract change. | +| META-CR-6507648970-002 | Added missing function documentation to both receiving-driver constructors and `describeMismatch`. | The comments explain receiving inputs, observation cuts and failure diagnostics. The external coverage percentage needs CodeRabbit's next calculation. | + +These are documentation findings. They require source and formatting checks, +not a new product RED or changed oracle histories. TypeScript emission with +comments removed is identical to the reviewed head for all three edited test +modules. The 248-test integrated result and five calibration controls therefore +still describe the executable tree. Budgets, guards and comparisons are unchanged. + +The raw review, mutable summary and append-only ledger remain in task scratch. +The loss audit accounts for both documentation items; neither reveals a new +behavioral defect. CodeRabbit supplied useful minor cleanup, but this review +does not demonstrate deep behavioral analysis. Hire for assisted documentation +review; do not use this sample as evidence of independent oracle-audit ability. +Final-head CI and the external docstring calculation remain publication checks. diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index b69d46ce78..e07b238f7d 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -219,6 +219,11 @@ const drainTurns = async () => { let run = 0 +/** + * Drives one persisted Collection through the shared session contract. Lane + * options select hydration seeds, cached rows, storage holds and peer notices; + * source reads and durable observations remain separate comparison inputs. + */ async function createPersistedDriver( withClient = false, cached = false, diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts index 31dc36533b..078303dc66 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -67,6 +67,11 @@ type Row = { id: number; v: unknown } type Actions = Parameters[`sync`]>[0] let run = 0 +/** + * Supplies the shared model with two SQLite-backed Collections and real + * coordinator traffic. Each authored action drains its correlated recovery + * before comparison; returned recovery inputs contain no expected metadata. + */ async function createSQLiteDriver(lane: SessionLane): Promise { const database = new DatabaseSync(`:memory:`) const { driver, drained } = createMetadataSQLiteDriver(database) diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 8977e19209..6fa39ec4fe 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -2719,6 +2719,11 @@ function describeError(error: unknown): string { : String(error) } +/** + * Reports the violated law and history prefix at its observation checkpoint. + * Recorded host recovery inputs accompany expected and actual values so an + * input gap remains distinguishable from a wrong model or production result. + */ export function describeMismatch( history: ReadonlyArray, mismatch: SessionMismatch, From adf6a07f97b26116933ad68e3d5e1b2f6fe5093f Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 12:26:34 -0700 Subject: [PATCH 06/11] Strengthen metadata oracle observations and receiving fixtures --- docs/contributing/oracle-coverage.md | 40 ++-- .../2026-11-02-p6-metadata-closeout.md | 200 ++++++++++++++++++ .../tests/sync-metadata-session-campaign.ts | 97 +++++++++ .../sync-metadata-session-persisted.test.ts | 143 +++++++++---- .../sync-metadata-session-sqlite.test.ts | 122 +++++------ .../tests/sync-metadata-sqlite-driver.test.ts | 35 +++ .../tests/sync-metadata-sqlite-driver.ts | 47 +++- .../tests/sync-metadata-write-work.test.ts | 106 ++++++++-- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 2 +- .../db/tests/sync-metadata-session-oracle.ts | 187 ++++++++++++---- .../db/tests/sync-metadata-session.test.ts | 55 +++++ 12 files changed, 821 insertions(+), 214 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index ed8bb2bc23..f23d71ebbb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4765,8 +4765,8 @@ reload after a named notice keeps a seed's metadata (decision 18). | Owner | Production paths | Observations and checkpoint | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [sync metadata session oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/sync-metadata-session-oracle.ts) (`flat`, `nested`, `seeds`, `nested-seeds`, `seeds-full`, `nested-conflicts` lanes) | `DbClient` Collection, core hold, `DbClient.hydrate` seeds | `metadata.row.get`, `metadata.collection.get`, `list()`, `list(prefix)`, `collection.base`, each commit receipt's state, thrown errors; after every action and after settlement | -| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, stored rows and stored row metadata after settlement | -| [native SQLite/coordinator receiver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts) | two real Collections and coordinators, node:sqlite, BroadcastChannel and Web Locks; partial/full/on-demand lanes | shared model and source reads after correlated commit/notice/recovery cuts; rows, metadata, applied receipts and direct durable rows/metadata after settlement | +| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, stored rows after settlement and stored row metadata at every completed durable action cut and after settlement | +| [native SQLite/coordinator receiver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts) | two real Collections and coordinators, node:sqlite, BroadcastChannel and Web Locks; partial/full/on-demand lanes | shared model and source reads after correlated commit/notice/recovery cuts; rows, metadata and applied receipts after settlement; direct durable metadata after each correlated action and durable rows after settlement | | paired properties and boundary witnesses in the core driver file | production against itself: holds removed, row 2's actions removed, or seeds performed first; stale run; write without a transaction; cleanup with a held seed | source reads (and row 1's view) at each shared checkpoint, or every observation after settlement for seed order; thrown errors; unhandled rejections | | [row metadata composition oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts) held-window and nested lanes | Collection, core hold, a transaction begun inside an open one, `DbClient.hydrate` | implicit row-metadata rules during a hold, and for an outer transaction whose inserts are reclassified after an inner write applies first, with and without a hold | | [hydration authority oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/db-client-hydration-authority-oracle.test.ts) | `DbClient` initialData, hydration chunks, and an adapter sync transaction (inserts, updates, deletes, truncates, and metadata-only writes) in full and partial row modes | public rows, the synced store, and row metadata through `client.dehydrate()` after each action; D5 per object | @@ -4814,35 +4814,47 @@ An equal source re-insert keeps metadata without writing it (decisions 9 and 14). As with an update, that keep neither makes a DbClient seed durable nor sends it to peers (decision 18). Direct durable metadata is compared separately from this tab's source reads. The constructive durability grammar includes all -12 update/equal-reinsert/inner-reinsert × bare/non-bare seed × implicit/explicit -metadata cases in each generated batch. Its partial/full/on-demand lanes run +36 update/equal-reinsert/inner-reinsert × bare/non-bare seed × implicit/explicit +metadata × no-notice/named-notice/full-reload cases in each generated batch. Its partial/full/on-demand lanes run through both the controlled persisted adapter and the real SQLite/coordinator -receiver, with declared successful reach for every case in both fixed and -random campaigns. Seed row shape, seed timing, repeat count and the final -named/full-reload/no-notice choice vary. Explicit metadata writes are the +receiver, with 108 declared case/lane/notice states per host, each reached eight times +in both fixed and random campaigns. Seed row shape, seed timing and repeat +count vary; no campaign can omit the full reload. Explicit metadata writes are the neighboring control. The original seed/re-insert/reload grammar, fixed/random 40 budgets and its completed fixed-branch guards remain; the new constructive batch supplies guaranteed bare-key-1 implicit histories independently of seed. -Both hosts share the 40-run campaign mechanics in +Both hosts share the 40-run sibling and 8-run durability campaign mechanics in `sync-metadata-session-campaign.ts`; production setup, recovery observations and direct replay selection remain in their receiving files. The native on-demand path can receive a successful recovery response that crosses an earlier cache version without transaction replay details. The shared model then applies the existing unknown-write rule (decision 15) to its own -independent durable storage. This is a recorded transport input, not an expected -value inferred from a production reload classifier. A named current-run write +independent durable storage. The recorder withholds that transport input until its hydration task completes +a storage read and application. A discarded response fails a guard instead of +advancing the model. Setup inputs are consumed before authored actions. No +expected metadata comes from a production reload classifier. A named current-run write still follows decision 11; a previous-run cached row initially remains a seed. The native driver finishes each source commit's correlated notice and recovery -before admitting the next authored action. It compares source reads after that -cut and consumers, receipts, stored rows and stored metadata after settlement. +before admitting the next authored action. It compares source reads and durable metadata after that cut, and consumers, +receipts and stored rows after settlement. The controlled driver waits for +acceptance before comparing durable metadata; store/read holds postpone that +cut until release. A later explicit write cannot hide an earlier wrong keep. +The controlled adapter also records every completed durable metadata snapshot, +so queued commits drained by one release cannot hide a wrong write. The +comparison preserves the sequence of changes and permits repeated identical +snapshots from empty commits. A constructive immediate/store-held × +bare/metadata-bearing repair campaign declares all four regimes, eight times +each. Native commits are individually awaited, so its action cuts already +separate these writes. No claim covers intermediate SQL statements. This is one legal schedule; arbitrary response/application interleavings, expiry, cleanup during recovery and the necessity of gap detection are not established. The receiver serializes every root SQL call with transaction admission on its one physical connection, while scoped transaction calls stay inside that -admission. A rollback/isolation witness rejects the earlier unscoped-call -fixture. The driver waits for actual correlated notices, remote ensure replies +admission. Nested scopes use savepoints; same-transaction async root reentry +uses that scope. Nested rollback/root-reentry and cross-adapter isolation +witnesses reject missing savepoints and unqueued root calls. The driver waits for actual correlated notices, remote ensure replies and accepted empty source commits through the wrapper's apply lock, rather than a quiet-time polling interval. It covers two real Browser coordinators over Node BroadcastChannel/Web Locks and native SQLite in one process. Chromium diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md index 580bbce6ce..0077f9d9d9 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -309,3 +309,203 @@ behavioral defect. CodeRabbit supplied useful minor cleanup, but this review does not demonstrate deep behavioral analysis. Hire for assisted documentation review; do not use this sample as evidence of independent oracle-audit ability. Final-head CI and the external docstring calculation remain publication checks. + +## Ten-item revision review: enforcement repairs + +The external review targets `9eef6525d663a3ca586f1e0c5e8af4bab1c29dbb`. +Evaluation started at `d0c4d0dde04208f9aaa66a03dcb7bada9c4f7cdb`; its intervening +changes were documentation only. The raw ten claims, grouped experiment plan, +commands, traces and append-only dispositions remain in task scratch under +`revision-review-10-*`, `revision-*-green.log` and `revision-calibration/`. +This section supplements the historical results above. It changes no product +law or production source, and needs no release changeset. + +### Law, experiment and inference + +No new product decision is required. Dated decisions 9, 14, 15 and 18 remain +the authority. A seed's metadata can remain local while cached metadata stays +durable. D18 explicitly limits seed retention to reloads after named notices; +its final paragraph applies D15 to differing metadata under unknown writes. +An old model expectation or open coverage note cannot supersede that decision. + +| Accepted law and affected items | Decisive experiment | Inference and enforcement | +| --- | --- | --- | +| An implicit keep does not write metadata; an explicit write does (D9/D18; 1, 4, 6). | The supplied nested bare-seed trace passed on both hosts in all three lanes, restoring cached M0 under D15 as independently predicted. A materialization mutant made an incorrect first durable write, then an explicit write repaired it; the old settlement-only runner passed. Seed 46 also produced eight old batches without a reload; reload-only corruption survived all 96 histories. | The authority accusation is false, but action-level durable comparison and guaranteed reload reach were missing. Both receiving paths now compare completed durable metadata before a later action can hide it. The controlled path also compares a transaction trace across held releases. Every constructive batch includes all three notice variants. | +| Only completed storage observations advance the model, at their actual order (D15; 2, 3). | The old runner consumed inputs after its first authored action. The six actual factories recorded zero setup inputs, so no existing setup metadata failure was demonstrated. A separate runner-order witness supplies a setup input. | The recorder now withholds qualifying raw responses until hydration completes a storage read and application. Undrained responses fail a guard. Setup observations are consumed before authored actions; expected metadata still comes only from independent model storage. | +| Constructive histories execute their declared actions; fixture transactions preserve scope (5, 7, 8). | All 3,000 sampled sibling lane histories had no skipped actions. The caught-inner-failure SQL probe committed row 2 incorrectly; a root query remained queued behind its own callback. Empty housekeeping commits under two open authored transactions preserved source metadata, consumer rows, receipts and durable metadata. | The reach count mismatch requires illegal actions the constructive grammar excludes; a new guard makes that premise explicit. SQL nested scopes now use savepoints and async-context root reentry, while unrelated callers retain serialized admission. Empty quiescence commits remain a stuttering abstraction; maxDepth bounds authored histories, not the runtime API. | + +The primary law, model and comparison remain in +`packages/db/tests/sync-metadata-session-oracle.ts`. The controlled persisted +and native SQLite/coordinator hosts receive the same generated histories and +judgment. Store/read holds postpone the durable comparison until release. The controlled +adapter preserves each completed metadata snapshot for comparison then; +core holds postpone visibility but do not postpone durability. This preserves +commit order, explicit writes, source reads and consumer settlement as separate +guarantees. The new checkpoint enforces the existing no-write law, rather than +creating an obligation for every internal SQLite statement. + +### Lossless item dispositions + +IDs below abbreviate the scratch prefix `META-REV2-`. These are instrument and +maintenance findings; none establishes a new production defect. + +| ID | Technical verdict and severity | PR action and preserved value | +| --- | --- | --- | +| 001 | Refuted. D18, dated October 27, explicitly selects the reviewed unknown-write outcome; six same-trace host/lane runs agree. | Retain the shared replay and distinguish named versus unknown notices. No model-law or production change. The broader #2281 reports remain open. | +| 002 | Confirmed recorder gap, P2. A qualifying transport response alone did not prove that storage was observed or applied. Reading response.requiresFullReload is reading transport data, so the old comment was imprecise. | Correlate response delivery with successful hydration/storage observation before exposing a model input; guard unapplied responses. Retain the serialized recovery scope and its lifecycle exclusions. | +| 003 | Confirmed latent ordering gap, P2. The old order probe was action/recovery/recovery; actual factories had no setup inputs in the measured runs. | Consume inputs before the authored loop and keep an ordinary runner-order calibration. Do not report an unobserved current metadata failure. | +| 004 | Confirmed generator/reach gap, P2. Seed 46 omitted reload in eight old batches, and reload-only wrong observations survived. | Construct all no-notice/named/full-reload variants and declare 108 case/lane/notice states per host. Preserve the eight-run budgets and neighboring explicit-write cases. | +| 005 | Refuted for this constructive grammar. Every action is legal by construction; sampled model checkpoints execute all 3,000 lane histories without skips. | Add a fail-fast successful-history length check, preserving the legality premise and existing completed-branch guards. The reviewer’s suggested executed-step accounting remains relevant to grammars that can skip. | +| 006 | Confirmed observation gap, P2. A wrong first durable write followed by an explicit repair passed the old settlement comparison. | Compare durable metadata after every action with completed storage I/O and at settlement. Both receiving hosts use the same check. The controlled adapter records every completed durable transaction, so a hold release cannot collapse a wrong write and its later repair. A four-regime generated campaign covers both seed forms and immediate/store-held commits. Coverage-map observations now name these cuts. | +| 007 | Confirmed fixture defect, P2. Caught inner rollback left row 2 committed; awaited root reentry would cycle behind its own callback. | Add nested savepoints and scoped async root reentry. Retain cross-adapter isolation. Reusing preparedDriver alone would lose the root admission that this two-adapter fixture needs. | +| 008 | Refuted as a fidelity defect. Empty commits are real runtime operations but introduce no model writes or recorded receipts; the same-path two-open-transaction probe preserved all compared observations. | Keep quiescence and document its stuttering abstraction. Runtime depth three is legal; the grammar's authored depth two is a finite test bound. | +| 009 | Confirmed duplication, P3. The hosts repeated durability campaign mechanics and state construction. | Extract runKeptMetadataDurabilityCampaign. Hosts retain setup, replay selection and native recovery reach. Source comparison and both receiver suites validate this maintenance repair. | +| 010 | Confirmed weak calibration, P3. The count guarded label cardinality, not whether actions implemented those labels. Its sampling cost was not a product-performance finding. | Replace diagnostic sampling with checks of all 36 unique cells, seed metadata, actual notices, explicit-write actions, reinsert form, nesting and legality in all lanes. Preserve emitted-work controls. | + +### Reviewer assessment + +The review supplied seven useful issues or narrower weaknesses out of ten, +including the important transient-durability and reload omissions. It separated +several test-integrity concerns from cleanup and proposed useful general fixes. +However, the highest-ranked authority accusation missed D18's explicit scoped +decision. The reach and quiescence objections also omitted premises that source +inspection and executable probes resolved. No submitted test or mutant supported +the downstream consequences, and preparedDriver was an incomplete fixture cure. + +Hire for assisted adversarial test review, with required authority verification +and executable follow-through. This sample does not support an unsupervised +final correctness verdict. Its strongest value is identifying distinguishing +experiments, not proving all ten conclusions. + +### Repaired observations and hostile controls + +A second cut audit extended item 006 beyond action-level storage reads. A +store hold can drain several commits during one release. The controlled +adapter now captures raw durable metadata after every completed transaction +and peer write. The model records its own durable states after authored +transitions. The comparison removes only consecutive identical snapshots, +which permits empty transactions while preserving every change and its order. +Expected metadata never comes from a production transaction's classifications. +The native schedule awaits each durable commit separately, so its action cuts +already distinguish those writes. + +`queuedMetadataRepairHistoriesArbitrary` constructs all four bare/non-bare × +immediate/store-held histories in each batch, varying seed shape and reinsert +count. Each makes an implicit keep, then an explicit write. Its controlled +fixed and random campaigns retain eight-run parity, declare every regime and +use the same session model/driver/comparison. Direct replay is registered as +`sync-metadata-session.persisted-queued-metadata-repair`. + +| Deliberately wrong execution | Rejection or distinguishing result | +| --- | --- | +| Materialization followed by a later explicit write, both hosts | The new comparison rejects the first wrong durable commit at step 5. The earlier runner passed the same corrupted history. | +| Wrong source observation only after full reload | The expanded grammar reaches reload and the source-metadata assertion rejects `wrong after reload`. The old seed-46 campaign omitted that cut and accepted the corruption. | +| Keep notice labels but remove peer actions | The grammar calibration rejects the missing actual notice. | +| Consume setup input after the first action | The runner-order assertion rejects action before recovery. | +| Remove nested savepoints | The fixture assertions reject row 2 surviving caught inner rollback, for scoped and root reentry. | +| Remove unrelated-root admission | The isolation assertion reads uncommitted row 1 instead of `[]`. | +| Discard a successful qualifying pull response before hydration | The actual native path fails `recovery response was not applied before comparison`. It never supplies that unapplied input to the model. | +| Ignore completed recovery input | The source-metadata comparison rejects cached M0 where the mutant model still expects the seed. The trace records version 0 to 2. | +| Drop explicit durable metadata writes | The durable comparison rejects the missing explicit value. | +| Materialization with a later repair inside one store hold, trace disabled | GREEN hostile execution: action-level comparison alone misses both bare and metadata-bearing cases. | +| The same held materialization, trace enabled | RED at release: the trace contains the wrong intermediate value before the explicit write. Restored production passes. | +| Replay obsolete durability path `0:0:0` after grammar expansion | Both hosts now reject zero-history execution. The valid path `0` executes the requested batch and passes after restoration. | + +The initial obsolete-path control ran zero histories and returned GREEN. That +is an unreached path, not a surviving recovery mutant. Sibling +`META-REV2-S001` adds zero-run guards to both shared runners. The following +valid-path control reaches the native recovery guard. An intermediate log +validator also searched for `source metadata` instead of the actual law text, +`source reads its row metadata`; inspection found the intended assertion, and +the repeated control validated that exact law. Neither diagnostic mistake +counts as evidence of oracle sensitivity. + +Temporary production, model, driver and fixture edits were restored +byte-for-byte after each control. Logs and restoration hashes remain in +`revision-calibration-v2/`, `revision-calibration-v3/` and +`revision-held-calibration/`. No timeout, setup failure or zero-test run is +credited as a successful value comparison. The temporary probe is absent from +the final diff. + +### Current validation and rubric audit + +The final executable tree passes **253 tests**: 76 controlled receiver, 163 +core/session/composition/registry, and 14 native receiver/SQL/work tests. Scoped +TypeScript, Prettier, changed-oracle lint and diff checks pass. General ESLint +reports zero errors and 12 warnings, from shadowed test variables and async +fixture stubs. The full workspace test command was not run locally. + +The complete runs use the same commands listed earlier, with +`tests/oracle-replay.test.ts` included in the core command. Logs use the +`revision-complete-` prefix. Direct controls and replays use the same Vitest +flags and matching property/seed/path selectors. The ten changed TypeScript +file hashes in `revision-final-code-freeze.json` match the executed tree. Their +sorted path/hash JSON has SHA-256 +`ba6782bae1011f2f7d565157cafa1fe5e0050d63c3c9b761cec5c9e87e8069d6`. +This record accompanies those repairs on parent `d0c4d0dde`. + +The fetched main head is `d9d0a8e50bf8494fe401e2925438b766594c5e26`. +Its only change since integrated `30e85464e` is a disjoint review-document +appendix. Production, manifests, lockfile and changesets still have zero delta +against fetched main. No overlapping implementation integration is outstanding. + +Normal constructive campaigns declare all 108 case/lane/notice states per +host and reach each state eight times. Native prior-cache recovery reaches +288 histories per campaign. The additional controlled queued-repair campaigns +reach all four declared regimes eight times. The older 40-run campaigns keep +their budgets, fixed completed-branch guards, replay selection and comparisons. +Random completed-branch counts remain measured. No run budget decreases. + +Durability replay now uses seed `2220031`, path `0`, because the expanded +grammar has a different shrink tree. Sibling replay retains seed `527962096`, +path `0`. Queued-repair replay uses seed `2220031`, path `0`. Each requested +property executes directly and the fixed campaign stands aside. Obsolete +zero-history paths fail explicitly. + +| Rubric requirement | Current outcome | +| --- | --- | +| ORC-001 | D9/D14/D15/D18 supply authority. Named/unknown authorship, direct durability and finite receiving limits remain explicit. | +| ORC-002 | Expected storage and its trace come from model actions. Transport inputs carry versions, never expected metadata. No production semantic helper computes expected values. | +| ORC-003 | Opening law/limits, grammar prose, model store rule, recorder and comparison describe their separate responsibilities beside executable code. | +| ORC-004 | The grammar rebuilds the supplied prefix, omitting notice actions fails calibration, both seed forms and immediate/held repairs occur constructively, and all authored actions satisfy model legality. Depth overflow and writes without an open transaction remain excluded. | +| ORC-005 | Actual source, peer/coordinator and direct durable paths reach the shared checks. The response guard distinguishes delivery from applied storage observation. Completed durable traces preserve intermediate changes. | +| ORC-006 | The controls above distinguish assertion rejection, an instrument guard, a GREEN hostile subject and an initially unreached replay. Restored subjects pass. | +| ORC-007 | Fixed/random campaigns share histories, driver, judgment and budget. Both receiver properties and the new queued property execute checked direct replays. Registry tests cover their selectors. | +| ORC-008 | The semantic model state and rules are unchanged. The runner's new trace retains a distinguishable history: two different durable writes can end at the same final value. | +| ORC-009 | Existing glossary mappings remain. Raw transport response, completed hydration input, action cut and durable-transaction snapshot name separate observations. Empty housekeeping commits stutter in the model. | +| ORC-010 | Existing primary/cleanup failure handling remains. Controls preserve their intended law/checkpoint, and temporary probes retain raw diagnostics separately from cleanup. | +| ORC-011 | The alleged shared semantic fault in item 001 is disproved by dated D18 and same-path execution. No model-law revision requires another formulation. Snapshot traces challenge the named final-state blind spot directly. | +| ORC-012 | This append-only versioned section records all applicable requirements, all ten raw findings, the replay sibling and evidence limits. | +| ORC-013 | Implicit versus explicit writes, bare versus metadata-bearing seeds, named versus unknown notices and immediate versus queued repairs distinguish the conditional law and nearby wrong designs. | +| ORC-014 | Controlled and native receivers share the same model/grammar/comparison. Native storage, messages and recovery supply the bounded real-host premise. Store holds and their transaction trace belong to the controlled receiver, while native commits are awaited individually. | +| ORC-015 | Both hosts declare 108 successful notice/case/lane states. The native receiver separately declares prior-cache recovery. The queued campaign declares all four immediate/store-held seed regimes. | +| ORC-016 | New fixed/constructive histories reuse the model, driver and comparison. Runner-order, SQL fixture and grammar calibrations remain in ordinary test files. | + +The durable trace covers the peer lanes, where the model owns durable storage, +including queued source commits under store holds. Read-hold/abandonment lanes +retain their source-facing laws and do not establish this direct durable trace. +No evidence covers intermediate SQL statements, arbitrary concurrent recovery, +cleanup/expiry/generation changes during recovery, Chromium OPFS, Electron IPC +or restart. The existing coverage-map owner retains those receiving limits. +The #2317/P8 source-facing witness, the broader #2281 reports and #2272 scope +remain separate and open. + +### Final loss audit + +The raw review was reread after the final diff. All ten original source-order +items survive: **10 = 7 fixed-now + 3 refuted** (001, 005 and 008). No raw item +is deferred, duplicated away or awaiting a product decision. The initial +counts-only calibration's limited value, item 003's absent actual setup input, +item 007's incomplete preparedDriver cure and the authority dispute in 001 +retain their qualifications. The separately discovered replay item S001 is +fixed-now. No sibling counterexample remains hidden in the loss accounting. + +The same current-head CodeRabbit summary reports no actionable comments and +100% docstring coverage at `d0c4d0dde`; the previous external percentage is +resolved. Formal review 5741628374 still covers `9eef6525d` and its wording +item is fixed. CI snapshot 14 has all 25 checks passing for `d0c4d0dde`. +These are historical publication observations, not final-head CI for the new +semantic test/instrument repairs. The new pushed head needs its own checks and +review coverage before a readiness verdict. This evaluation's accounting and +authorized repairs are complete within the stated receiving boundary. It does +not close #2281, #2272 or #2317/P8, or establish whole-library correctness. diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts index 66b95b6ba2..51ad1817ea 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts @@ -2,6 +2,7 @@ import { fc } from '@fast-check/vitest' import { completedSeedReinsertReloadRegime, describeMismatch, + keptMetadataDurabilityHistoriesArbitrary, persistedPeerFullLane, persistedPeerLane, persistedPeerOnDemandLane, @@ -60,6 +61,8 @@ export async function runSeedReinsertReloadCampaign({ reach.beginHistory() completedReach.beginHistory() const result = await runHistory(history, lane) + if (!result.mismatch && result.steps !== history.length) + throw new Error(`constructive history skipped an authored action`) const last = history.at(-1)! const innerCommit = history.findIndex( (action) => action.type === `commit`, @@ -83,6 +86,8 @@ export async function runSeedReinsertReloadCampaign({ : oraclePropertyOptions(40, property), ) recordOracleReplay(property, result) + if (result.numRuns === 0) + throw new Error(`oracle replay executed no history: ${property}`) console.info( `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, ) @@ -97,3 +102,95 @@ export async function runSeedReinsertReloadCampaign({ // inner/row/reload guards apply to both campaigns. if (campaign === `fixed`) completedReach.assertReached() } + +/** + * Every batch reaches every write/seed/metadata/notice premise on each host. + * Drivers own their real entry point and recovery evidence; this runner shares + * only campaign mechanics, the common comparison and successful reach checks. + */ +export async function runKeptMetadataDurabilityCampaign({ + property, + campaign, + runHistory, + extraStates = [], + extraReached = () => [], +}: { + property: string + campaign: `fixed` | `random` + runHistory: ( + history: ReadonlyArray, + lane: SessionLane, + ) => ReturnType + extraStates?: ReadonlyArray + extraReached?: ( + lane: SessionLane, + result: Awaited>, + ) => ReadonlyArray +}): Promise { + const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ] + const name = ( + lane: SessionLane, + write: string, + bare: boolean, + metadata: boolean, + notice: string, + ) => + `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}: ${notice}` + const states = lanes.flatMap((lane) => + [`update`, `reinsert`, `inner-reinsert`].flatMap((write) => + [false, true].flatMap((bare) => + [false, true].flatMap((metadata) => + [`none`, `row`, `reload`].map((notice) => + name(lane, write, bare, metadata, notice), + ), + ), + ), + ), + ) + const reach = createOracleReach(property, [...states, ...extraStates]) + const result = await fc.check( + fc.asyncProperty( + keptMetadataDurabilityHistoriesArbitrary, + async (batch) => { + const failures: Array = [] + for (const lane of lanes) { + for (const { write, bare, metadata, notice, history } of batch) { + reach.beginHistory() + const result = await runHistory(history, lane) + if (result.mismatch) + failures.push( + `${lane.name}\n${describeMismatch(history, result.mismatch)}`, + ) + else { + if (result.steps !== history.length) + throw new Error( + `constructive history skipped an authored action`, + ) + reach.reach(name(lane, write, bare, metadata, notice)) + for (const state of extraReached(lane, result)) reach.reach(state) + } + } + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }, + ), + campaign === `fixed` + ? { numRuns: oracleRuns(8), seed: 2220031 } + : oraclePropertyOptions(8, property), + ) + recordOracleReplay(property, result) + if (result.numRuns === 0) + throw new Error(`oracle replay executed no history: ${property}`) + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index e07b238f7d..9d440ac721 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -1,13 +1,17 @@ import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' -import { DbClient, collectionOptions, createCollection } from '@tanstack/db' +import { + DbClient, + collectionOptions, + createCollection, + whenSyncAccepted, +} from '@tanstack/db' import { absentMarker, cachedMetadata, cachedRow, describeMismatch, historyRegimes, - keptMetadataDurabilityHistoriesArbitrary, persistedCachedLane, persistedConflictLane, persistedFlatLane, @@ -19,6 +23,7 @@ import { persistedReadLane, persistedReadSeedLane, persistedSeedLane, + queuedMetadataRepairHistoriesArbitrary, runSessionHistory, seedReinsertReloadReplay, sessionCollectionKeys, @@ -33,7 +38,10 @@ import { } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' -import { runSeedReinsertReloadCampaign } from './sync-metadata-session-campaign' +import { + runKeptMetadataDurabilityCampaign, + runSeedReinsertReloadCampaign, +} from './sync-metadata-session-campaign' import type { CollectionConfig, SyncConfig } from '@tanstack/db' import type { PeerNotice, @@ -116,6 +124,22 @@ function createHoldingAdapter(cached = false, stores = false) { ) const storedRows = () => stores ? Array.from(stored, ([key, row]) => ({ key, ...row })) : cachedRows + // Capture every completed durable transaction before another queued write + // can replace it. The shared comparison permits identical empty commits. + const metadataSnapshots: Array> = [] + const recordMetadata = () => { + metadataSnapshots.push( + structuredClone( + Object.fromEntries( + [1, 2].map((key) => { + const value = stored.get(key)?.metadata + return [key, value === undefined ? absentMarker : value] + }), + ), + ), + ) + } + recordMetadata() // A reload restores collection metadata from storage, so storage keeps it. const storedCollectionMetadata = new Map() const collectionMetadata = () => @@ -164,6 +188,7 @@ function createHoldingAdapter(cached = false, stores = false) { if (mutation.type === `delete`) storedCollectionMetadata.delete(mutation.key) else storedCollectionMetadata.set(mutation.key, mutation.value) + recordMetadata() }, ensureIndex: () => Promise.resolve(), } @@ -185,6 +210,7 @@ function createHoldingAdapter(cached = false, stores = false) { } resetEpoch++ latest = { term: latest.term + 1, seq: 0, rowVersion: 0 } + recordMetadata() return latest } if (notice === `delete`) stored.delete(key) @@ -195,12 +221,14 @@ function createHoldingAdapter(cached = false, stores = false) { seq: latest.seq + 1, rowVersion: latest.rowVersion + 1, } + recordMetadata() return latest }, resetEpoch: () => resetEpoch, /** The row storage holds for a key, if any. */ storedRow: (key: number): Row | undefined => stored.get(key)?.value, storedRowMetadata: (key: number): unknown => stored.get(key)?.metadata, + storedMetadataTrace: () => metadataSnapshots, holdNextStore: () => { holdNext = true }, @@ -326,6 +354,8 @@ async function createPersistedDriver( const read = (value: unknown) => (value === undefined ? absentMarker : value) let readGate: { resolve: () => void } | undefined let reading: Promise | undefined + let storageHeld = false + const accepted: Array> = [] const release = async () => { readGate?.resolve() readGate = undefined @@ -333,6 +363,7 @@ async function createPersistedDriver( await reading reading = undefined store.releaseStore() + storageHeld = false const pending = blocker blocker = undefined resolveMutation?.() @@ -344,6 +375,7 @@ async function createPersistedDriver( return { consumerCheckpoint: `final`, + storedMetadataTrace: peers ? store.storedMetadataTrace : undefined, begin: () => { sync.begin() depth++ @@ -353,6 +385,9 @@ async function createPersistedDriver( const controller = new AbortController() if (readGate && depth === 1) readCommits.push(controller) const receipt = sync.commit(controller.signal) + accepted.push( + Promise.resolve(whenSyncAccepted(receipt)).catch(() => undefined), + ) depth-- if (receipt === true) { receiptStates.push(`applied`) @@ -391,6 +426,7 @@ async function createPersistedDriver( collectionSet: (key, value) => metadata().collection.set(key, value), collectionDelete: (key) => metadata().collection.delete(key), hold: (holdStore, holdRead) => { + storageHeld = holdStore || holdRead === true if (holdRead) { let open!: () => void const gate = new Promise((resolve) => { @@ -511,7 +547,12 @@ async function createPersistedDriver( }, receipts: [...receiptStates], }), - flush: () => new Promise((resolve) => setTimeout(resolve, 0)), + flush: async () => { + // Acceptance follows durability even when a core hold delays visibility. + // A deliberately held store/read has no completed durable cut yet. + if (!storageHeld) await Promise.all(accepted) + await new Promise((resolve) => setTimeout(resolve, 0)) + }, settle: async () => { await release() while (depth > 0) abort() @@ -1335,65 +1376,75 @@ for (const campaign of [`fixed`, `random`] as const) { ) continue it(`keeps tab-local and durable metadata distinct (persisted, ${campaign})`, async () => { - const durabilityLanes = [ - persistedPeerLane, - persistedPeerFullLane, - persistedPeerOnDemandLane, - ] - const states = durabilityLanes.flatMap((lane) => - [`update`, `reinsert`, `inner-reinsert`].flatMap((write) => - [false, true].flatMap((bare) => - [false, true].map( - (metadata) => - `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, + await runKeptMetadataDurabilityCampaign({ + property: durabilityProperty, + campaign, + runHistory: (history, lane) => + runSessionHistory(lane, history, () => + createPersistedDriver( + true, + true, + false, + true, + lane.fullRows === true, + lane.onDemand === true, ), ), - ), + }) + }, 240_000) +} + +// Storage holds belong to this controlled receiver. The native schedule awaits +// every durable commit separately and uses the same action-level comparison. +const queuedRepairProperty = `sync-metadata-session.persisted-queued-metadata-repair` +for (const campaign of [`fixed`, `random`] as const) { + if ( + requestedReplayProperty !== undefined && + (requestedReplayProperty !== queuedRepairProperty || campaign === `fixed`) + ) + continue + it(`observes durable metadata between queued commits (${campaign})`, async () => { + const name = (bare: boolean, held: boolean) => + `${bare ? `bare` : `metadata`} seed: ${held ? `store hold` : `immediate`}` + const states = [false, true].flatMap((bare) => + [false, true].map((held) => name(bare, held)), ) - const reach = createOracleReach(durabilityProperty, states) + const reach = createOracleReach(queuedRepairProperty, states) const result = await fc.check( fc.asyncProperty( - keptMetadataDurabilityHistoriesArbitrary, + queuedMetadataRepairHistoriesArbitrary, async (batch) => { - const failures: Array = [] - for (const lane of durabilityLanes) { - for (const { write, bare, metadata, history } of batch) { - reach.beginHistory() - const result = await runSessionHistory(lane, history, () => - createPersistedDriver( - true, - true, - false, - true, - lane.fullRows === true, - lane.onDemand === true, - ), - ) - if (result.mismatch) - failures.push( - `${lane.name}\n${describeMismatch(history, result.mismatch)}`, - ) - else if (result.steps === history.length) - reach.reach( - `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, - ) - } + for (const { bare, held, history } of batch) { + reach.beginHistory() + const result = await runSessionHistory( + persistedPeerLane, + history, + () => createPersistedDriver(true, true, false, true), + ) + if (result.mismatch) + throw new Error(describeMismatch(history, result.mismatch)) + if (result.steps !== history.length) + throw new Error(`constructive history skipped an authored action`) + reach.reach(name(bare, held)) } - if (failures.length) throw new Error(failures.join(`\n\n`)) }, ), campaign === `fixed` ? { numRuns: oracleRuns(8), seed: 2220031 } - : oraclePropertyOptions(8, durabilityProperty), + : oraclePropertyOptions(8, queuedRepairProperty), ) - recordOracleReplay(durabilityProperty, result) + recordOracleReplay(queuedRepairProperty, result) console.info( - `ORACLE_REACH ${JSON.stringify({ property: durabilityProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + `ORACLE_REACH ${JSON.stringify({ property: queuedRepairProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, ) + if (result.numRuns === 0) + throw new Error( + `oracle replay executed no history: ${queuedRepairProperty}`, + ) if (result.failed) throw new Error( `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, ) reach.assertReached() - }, 240_000) + }, 120_000) } diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts index 078303dc66..e9c7ed6e7c 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -20,7 +20,6 @@ * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. */ import { DatabaseSync } from 'node:sqlite' -import { fc } from '@fast-check/vitest' import { it } from 'vitest' import { DbClient, collectionOptions, whenSyncAccepted } from '@tanstack/db' import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' @@ -29,7 +28,6 @@ import { cachedMetadata, cachedRow, describeMismatch, - keptMetadataDurabilityHistoriesArbitrary, persistedPeerFullLane, persistedPeerLane, persistedPeerOnDemandLane, @@ -38,13 +36,9 @@ import { sessionCollectionKeys, } from '../../db/tests/sync-metadata-session-oracle' import { - oraclePropertyOptions, oracleReplayTest, - oracleRuns, readOracleRunConfig, - recordOracleReplay, } from '../../db/tests/oracle-config' -import { createOracleReach } from '../../db/tests/oracle-reach' import { SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, SQLiteCorePersistenceAdapter, @@ -53,7 +47,10 @@ import { persistedCollectionOptions, } from '../src' import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' -import { runSeedReinsertReloadCampaign } from './sync-metadata-session-campaign' +import { + runKeptMetadataDurabilityCampaign, + runSeedReinsertReloadCampaign, +} from './sync-metadata-session-campaign' import type { SyncConfig } from '@tanstack/db' import type { ReceiptState, @@ -103,22 +100,43 @@ async function createSQLiteDriver(lane: SessionLane): Promise { const coordinators = adapters.map( (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), ) - // Observe an actual transport input, never a production reload flag or a + // Observe the transport's replay information, not the wrapper's decision or a // metadata mismatch. Without replay details this response cannot distinguish // earlier cached writes from this run's writes; D15 governs that information // gap. The model derives its effect solely from its own durable storage. const recoveryInputs: Array = [] + const pendingRecoveryInputs: Array = [] const pullSince = coordinators[0]!.pullSince!.bind(coordinators[0]) coordinators[0]!.pullSince = async (...args) => { const response = await pullSince(...args) if (response.ok && (response.requiresFullReload || !response.deltas)) - recoveryInputs.push({ + pendingRecoveryInputs.push({ type: `unknown-storage-observation`, fromRowVersion: args[1], latestRowVersion: response.latestRowVersion, }) return response } + const hydrate = adapters[0]!.runInHydrationScope.bind(adapters[0]) + adapters[0]!.runInHydrationScope = (task) => + hydrate(async (adapter) => { + const inputs = pendingRecoveryInputs.splice(0) + const observation = { reads: 0 } + const result = await task({ + ...adapter, + loadSubset: async (...args) => { + const rows = await adapter.loadSubset(...args) + observation.reads++ + return rows + }, + }) + if (inputs.length && observation.reads === 0) + throw new Error(`recovery response completed no storage observation`) + // The hydration task includes application. Merely receiving a response + // that cleanup or claim recovery discards must not advance the model. + recoveryInputs.push(...inputs) + return result + }) const pendingEnsures = new Set>() for (const coordinator of coordinators) { const request = coordinator.requestEnsureRemoteSubset.bind(coordinator) @@ -172,6 +190,8 @@ async function createSQLiteDriver(lane: SessionLane): Promise { // Each empty source commit enters the wrapper after already delivered // notices. Its acceptance drains that wrapper's apply lock without // waiting for visibility behind a core hold. No write or notice results. + // These empty commits stutter in the model, including under two open + // authored transactions; maxDepth bounds authored histories only. do { // An on-demand cached load may settle before its upstream ensure. Wait // for those actual coordinator replies before draining their receiver. @@ -422,7 +442,11 @@ async function createSQLiteDriver(lane: SessionLane): Promise { read(durableRows.find((row) => row.key === key)?.metadata), ]), ), - takeRecoveryInputs: () => recoveryInputs.splice(0), + takeRecoveryInputs: () => { + if (pendingRecoveryInputs.length) + throw new Error(`recovery response was not applied before comparison`) + return recoveryInputs.splice(0) + }, flush: quiesce, settle: async () => { await release() @@ -501,69 +525,19 @@ for (const campaign of [`fixed`, `random`] as const) { ) continue it(`keeps tab-local and durable metadata distinct (sqlite, ${campaign})`, async () => { - const durabilityLanes = [ - persistedPeerLane, - persistedPeerFullLane, - persistedPeerOnDemandLane, - ] - const states = durabilityLanes.flatMap((lane) => - [`update`, `reinsert`, `inner-reinsert`].flatMap((write) => - [false, true].flatMap((bare) => - [false, true].map( - (metadata) => - `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, - ), - ), - ), - ) - const reach = createOracleReach(durabilityProperty, [ - ...states, - `on-demand: cached prior-run version crossed without replay details`, - ]) - const result = await fc.check( - fc.asyncProperty( - keptMetadataDurabilityHistoriesArbitrary, - async (batch) => { - const failures: Array = [] - for (const lane of durabilityLanes) { - for (const { write, bare, metadata, history } of batch) { - reach.beginHistory() - const result = await check(history, lane) - if ( - lane.onDemand && - result.recoveryInputs.some( - (input) => - input.fromRowVersion === 0 && input.latestRowVersion >= 2, - ) - ) - reach.reach( - `on-demand: cached prior-run version crossed without replay details`, - ) - if (result.mismatch) - failures.push( - `${lane.name}\n${describeMismatch(history, result.mismatch)}`, - ) - else if (result.steps === history.length) - reach.reach( - `${lane.name}: ${write}: ${bare ? `bare` : `metadata`} seed: ${metadata ? `explicit` : `implicit`}`, - ) - } - } - if (failures.length) throw new Error(failures.join(`\n\n`)) - }, - ), - campaign === `fixed` - ? { numRuns: oracleRuns(8), seed: 2220031 } - : oraclePropertyOptions(8, durabilityProperty), - ) - recordOracleReplay(durabilityProperty, result) - console.info( - `ORACLE_REACH ${JSON.stringify({ property: durabilityProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, - ) - if (result.failed) - throw new Error( - `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, - ) - reach.assertReached() + const recoveryState = `on-demand: cached prior-run version crossed without replay details` + await runKeptMetadataDurabilityCampaign({ + property: durabilityProperty, + campaign, + runHistory: check, + extraStates: [recoveryState], + extraReached: (lane, result) => + lane.onDemand && + result.recoveryInputs.some( + (input) => input.fromRowVersion === 0 && input.latestRowVersion >= 2, + ) + ? [recoveryState] + : [], + }) }, 240_000) } diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts index c15efbb6c9..4d8013cf20 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts @@ -3,6 +3,41 @@ import { DatabaseSync } from 'node:sqlite' import { expect, it } from 'vitest' import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' +it.each([false, true])( + `rolls back only a failed nested scope (root reentry: %s)`, + async (rootReentry) => { + const database = new DatabaseSync(`:memory:`) + const { driver, drained } = createMetadataSQLiteDriver(database) + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + await driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (1)`) + const nested = rootReentry ? driver : scoped + await expect( + nested.transaction(async (inner) => { + await inner.run(`INSERT INTO rows VALUES (2)`) + throw new Error(`rollback the inner scope`) + }), + ).rejects.toThrow(`rollback the inner scope`) + await nested.transaction(async (inner) => { + await inner.run(`INSERT INTO rows VALUES (3)`) + }) + // Root calls in this async transaction must use its scope, rather than + // queue behind the callback that is awaiting them. + await (rootReentry ? driver : scoped).run(`INSERT INTO rows VALUES (4)`) + }) + expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ + { id: 1 }, + { id: 3 }, + { id: 4 }, + ]) + } finally { + await drained() + database.close() + } + }, +) + it(`keeps another adapter's root reads and writes outside a rolled-back transaction`, async () => { const database = new DatabaseSync(`:memory:`) const { driver, drained } = createMetadataSQLiteDriver(database) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts index 88ebaa2d66..db178aa732 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts @@ -1,3 +1,4 @@ +import { AsyncLocalStorage } from 'node:async_hooks' import type { DatabaseSync, SQLInputValue } from 'node:sqlite' import type { SQLiteDriver } from '../src' @@ -5,9 +6,13 @@ import type { SQLiteDriver } from '../src' * One SQLite connection admits root calls in order. A transaction's scoped * driver executes its calls inside that admission, so another adapter's read * cannot observe it before commit or become part of its rollback. + * Nested scopes use savepoints. Root calls in the transaction's async context + * reenter its scope, rather than queue behind the callback that awaits them. */ export function createMetadataSQLiteDriver(database: DatabaseSync) { let queue: Promise = Promise.resolve() + const context = new AsyncLocalStorage() + let nextSavepoint = 0 const enqueue = (work: () => Promise): Promise => { const next = queue.then(work) queue = next.catch(() => undefined) @@ -22,25 +27,45 @@ export function createMetadataSQLiteDriver(database: DatabaseSync) { run: async (sql, params = []) => { database.prepare(sql).run(...(params as Array)) }, - transaction: async (work) => work(scoped), - } - const transaction: SQLiteDriver[`transaction`] = (work) => - enqueue(async () => { - database.exec(`BEGIN IMMEDIATE`) + transaction: async (work) => { + const savepoint = `metadata_${nextSavepoint++}` + database.exec(`SAVEPOINT ${savepoint}`) try { const result = await work(scoped) - database.exec(`COMMIT`) + database.exec(`RELEASE SAVEPOINT ${savepoint}`) return result } catch (error) { - database.exec(`ROLLBACK`) + database.exec(`ROLLBACK TO SAVEPOINT ${savepoint}`) + database.exec(`RELEASE SAVEPOINT ${savepoint}`) throw error } - }) + }, + } + const transaction: SQLiteDriver[`transaction`] = (work) => + context.getStore() + ? scoped.transaction(work) + : enqueue(async () => { + database.exec(`BEGIN IMMEDIATE`) + try { + const result = await context.run(true, () => work(scoped)) + database.exec(`COMMIT`) + return result + } catch (error) { + database.exec(`ROLLBACK`) + throw error + } + }) const driver: SQLiteDriver = { - exec: (sql) => enqueue(() => scoped.exec(sql)), + exec: (sql) => + context.getStore() ? scoped.exec(sql) : enqueue(() => scoped.exec(sql)), query: (sql: string, params?: ReadonlyArray) => - enqueue(() => scoped.query(sql, params)), - run: (sql, params) => enqueue(() => scoped.run(sql, params)), + context.getStore() + ? scoped.query(sql, params) + : enqueue(() => scoped.query(sql, params)), + run: (sql, params) => + context.getStore() + ? scoped.run(sql, params) + : enqueue(() => scoped.run(sql, params)), transaction, transactionWithDriver: transaction, } diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts index 880dd11f67..55b6cef0d6 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts @@ -2,9 +2,12 @@ import { createCollection } from '@tanstack/db' import { expect, it } from 'vitest' import { fc } from '@fast-check/vitest' import { - completedSeedReinsertReloadRegime, keptMetadataDurabilityHistoriesArbitrary, - seedReinsertReloadHistoryArbitrary, + modelCheckpoints, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + queuedMetadataRepairHistoriesArbitrary, } from '../../db/tests/sync-metadata-session-oracle' import { persistedCollectionOptions } from '../src' import type { SyncConfig } from '@tanstack/db' @@ -13,32 +16,89 @@ import type { PersistedTx, PersistenceAdapter } from '../src' type Row = { id: number; v: number } type Actions = Parameters[`sync`]>[0] -it(`measures the previous scarce branch and the constructive replacement`, () => { +// Calibrate the grammar against its declared premises, not only entry labels. +// Deleting a notice, seed or explicit write must fail even if labels survive. +it(`constructs every declared metadata premise as a legal history`, async () => { + const expected = new Set() + for (const write of [`update`, `reinsert`, `inner-reinsert`]) + for (const bare of [false, true]) + for (const metadata of [false, true]) + for (const notice of [`none`, `row`, `reload`]) + expected.add(`${write}/${bare}/${metadata}/${notice}`) for (const seed of [-704889444, 527962096, 1, 2, 3]) { - const old = fc.sample(seedReinsertReloadHistoryArbitrary, { + for (const batch of fc.sample(queuedMetadataRepairHistoriesArbitrary, { seed, - numRuns: 40, - }) - const count = old.filter( - (history) => - completedSeedReinsertReloadRegime(history, { - mismatch: undefined, - steps: history.length, - }) === `completed kept bare seed reload`, - ).length - const next = fc.sample(keptMetadataDurabilityHistoriesArbitrary, { + numRuns: 8, + })) { + expect(new Set(batch.map(({ bare, held }) => `${bare}/${held}`))).toEqual( + new Set([`false/false`, `false/true`, `true/false`, `true/true`]), + ) + expect(batch).toHaveLength(4) + for (const { bare, held, history } of batch) { + expect(history.find((action) => action.type === `seed`)?.bare).toBe( + bare, + ) + expect( + history.some((action) => action.type === `hold` && action.store), + ).toBe(held) + expect(history.some((action) => action.type === `release`)).toBe(held) + expect( + history.filter((action) => action.type === `commit`), + ).toHaveLength(2) + expect(history.some((action) => action.type === `rowSet`)).toBe(true) + expect((await modelCheckpoints(persistedPeerLane, history)).size).toBe( + history.length, + ) + } + } + for (const batch of fc.sample(keptMetadataDurabilityHistoriesArbitrary, { seed, numRuns: 8, - }) - const bareImplicit = next - .flat() - .filter((entry) => entry.bare && !entry.metadata).length - console.info( - `METADATA_GENERATOR ${JSON.stringify({ seed, oldBareCompletedPremise: count, newBareImplicitHistories: bareImplicit })}`, - ) - expect(bareImplicit).toBe(24) + })) { + expect( + new Set( + batch.map( + (entry) => + `${entry.write}/${entry.bare}/${entry.metadata}/${entry.notice}`, + ), + ), + ).toEqual(expected) + expect(batch).toHaveLength(expected.size) + for (const entry of batch) { + const { history, write, bare, metadata, notice } = entry + const seeds = history.filter((action) => action.type === `seed`) + expect(seeds).toHaveLength(1) + expect(seeds[0]!.bare).toBe(bare) + const peers = history.filter((action) => action.type === `peer`) + expect(peers.map((action) => action.notice)).toEqual( + notice === `none` ? [] : [notice], + ) + const explicit = history.filter( + (action) => + action.type === `rowSet` || + (action.type === `reinsert` && action.metadata), + ) + expect(explicit.length > 0).toBe(metadata) + const reinserts = history.filter((action) => action.type === `reinsert`) + expect(reinserts.length > 0).toBe(write !== `update`) + const prefix = history.slice( + 0, + history.findIndex((action) => action.type === `commit`), + ) + const depth = prefix.filter((action) => action.type === `begin`).length + expect(depth).toBe(write === `inner-reinsert` ? 2 : 1) + for (const lane of [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, + ]) + expect((await modelCheckpoints(lane, history)).size).toBe( + history.length, + ) + } + } } -}) +}, 120_000) it.each([2, 100, 10_000])( `counts implicit keeps in an unchanged %i-row refresh`, diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 88b6be3096..ec02c36817 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -289,6 +289,7 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-seed-reinsert-reload`, `sync-metadata-session.sqlite-seed-reinsert-reload`, `sync-metadata-session.persisted-kept-metadata-durability`, + `sync-metadata-session.persisted-queued-metadata-repair`, `sync-metadata-session.sqlite-kept-metadata-durability`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index fd67b6b445..7aa107040f 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -729,7 +729,7 @@ const ownerGroups: ReadonlyArray< [ `db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts`, `sync-metadata-session`, - `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand persisted-seed-reinsert-reload persisted-kept-metadata-durability`, + `persisted-flat persisted-nested persisted-seeds persisted-conflicts persisted-cached persisted-nested-seeds persisted-reads persisted-reads-seeds persisted-peers persisted-peers-full persisted-peers-on-demand persisted-seed-reinsert-reload persisted-kept-metadata-durability persisted-queued-metadata-repair`, ], [ `db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts`, diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 6fa39ec4fe..a3f1a27cc0 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -88,13 +88,15 @@ import { declareOracle } from './oracle-declaration.js' * source through its notices; no other lane has a peer, and a peer's write * reaches this tab only at a point where the wrapper has no durable step or * storage read in flight. Only the peer lanes model storage, so only they - * compare the stored rows, and only after settlement: a restart reads them, - * but this oracle's claim ends with one sync run. A + * compare stored rows after settlement. Durable metadata is also compared + * after each action whose storage I/O is not held, and after settlement. + * A restart reads storage, but this oracle's claim ends with one sync run. A * source update of an absent row is not a legal history (the change-message * protocol), so it is not generated; it arises only when an inner * transaction's update commits ahead of the enclosing write it relied on. - * The native receiver also records actual successful pull responses without - * replay details. It completes each source commit's correlated notice and + * The native receiver records successful pull responses without replay details + * only after their hydration task completes a storage read and application. + * It completes each source commit's correlated notice and * recovery before the next authored action, then compares the same model's * D15 storage observation. Arbitrary response/application interleavings, * recovery during cleanup, and the necessity of gap detection are outside @@ -666,50 +668,90 @@ export const seedReinsertReloadReplay: Array = [ * reload expose differing durable metadata. Cross updates and equal re-inserts * with implicit keeps and explicit writes before that later reload can erase * the distinction. Every generated batch contains each conditional premise; - * the seed shape, timing, repeats, and final notice still vary. + * the seed shape, timing and repeats still vary. Every batch includes no + * notice, a named notice and a full reload for each write/seed/metadata case, + * so a random campaign cannot omit the unknown-author observation cut. */ export const keptMetadataDurabilityHistoriesArbitrary = fc .record({ narrow: fc.boolean(), seedInside: fc.boolean(), repeats: fc.integer({ min: 1, max: 3 }), - notice: fc.constantFrom(`none`, `row`, `reload`), }) .map((input) => ([`update`, `reinsert`, `inner-reinsert`] as const).flatMap((write) => [false, true].flatMap((bare) => - [false, true].map((metadata) => { - const seed: SessionAction = { - type: `seed`, - key: 1, - bare, - narrow: input.narrow, - } - const history: Array = [ - ...(input.seedInside - ? [{ type: `begin` as const }, seed] - : [seed, { type: `begin` as const }]), - { - type: write === `inner-reinsert` ? `insertOverSeed` : `upsert`, + [false, true].flatMap((metadata) => + ([`none`, `row`, `reload`] as const).map((notice) => { + const seed: SessionAction = { + type: `seed`, key: 1, - metadata: false, - }, - ] - if (write !== `update`) { - if (write === `reinsert`) history.push({ type: `commit` }) - history.push({ type: `begin` }) - for (let index = 0; index < input.repeats; index++) - history.push({ type: `reinsert`, key: 1, metadata }) - } else if (metadata) history.push({ type: `rowSet`, key: 1 }) - history.push({ type: `commit` }) - if (input.notice !== `none`) - history.push({ type: `peer`, key: 2, notice: input.notice }) - return { write, bare, metadata, history } - }), + bare, + narrow: input.narrow, + } + const history: Array = [ + ...(input.seedInside + ? [{ type: `begin` as const }, seed] + : [seed, { type: `begin` as const }]), + { + type: write === `inner-reinsert` ? `insertOverSeed` : `upsert`, + key: 1, + metadata: false, + }, + ] + if (write !== `update`) { + if (write === `reinsert`) history.push({ type: `commit` }) + history.push({ type: `begin` }) + for (let index = 0; index < input.repeats; index++) + history.push({ type: `reinsert`, key: 1, metadata }) + } else if (metadata) history.push({ type: `rowSet`, key: 1 }) + history.push({ type: `commit` }) + if (notice !== `none`) + history.push({ type: `peer`, key: 2, notice }) + return { write, bare, metadata, notice, history } + }), + ), ), ), ) +/** + * A later explicit write can repair a wrong implicit durable keep. Cross both + * seed metadata forms with immediate writes and writes queued behind a store + * hold. Releasing that hold may complete both commits before the next action + * cut, so the controlled driver records each completed durable transaction. + * All histories replay the same model and comparison, without fixed answers. + */ +export const queuedMetadataRepairHistoriesArbitrary = fc + .record({ + narrow: fc.boolean(), + repeats: fc.integer({ min: 1, max: 3 }), + }) + .map(({ narrow, repeats }) => + [false, true].flatMap((bare) => + [false, true].map((held) => { + const history: Array = [ + { type: `seed`, key: 1, bare, narrow }, + ...(held ? [{ type: `hold` as const, store: true }] : []), + { type: `begin` }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + ...Array.from({ length: repeats }, (): SessionAction => ({ + type: `reinsert`, + key: 1, + metadata: false, + })), + { type: `commit` }, + { type: `begin` }, + { type: `rowSet`, key: 1 }, + { type: `commit` }, + ...(held ? [{ type: `release` as const }] : []), + ] + return { bare, held, history } + }), + ), + ) + /** * This classifier records a completed implicit re-insert across a full reload * while the enclosing write remains unaccepted. "Kept" names that implicit @@ -1969,6 +2011,8 @@ export type SessionDriver = { storedRows?: () => Record /** Durable metadata, distinct from the source's tab-local seed reads. */ storedRowMetadata?: () => Record + /** Raw completed storage snapshots, including commits drained by one release. */ + storedMetadataTrace?: () => ReadonlyArray> /** Actual host recovery inputs since the previous completed comparison. */ takeRecoveryInputs?: () => Array /** Lets receipt continuations run before an observation. */ @@ -2046,6 +2090,10 @@ export async function runSessionHistory( let steps = 0 let primary: unknown const recoveryInputs: Array = [] + const durableTrace = [model.storedRowMetadata()] + const recordDurableMetadata = () => { + durableTrace.push(model.storedRowMetadata()) + } const applyRecoveryInputs = () => { for (const input of driver.takeRecoveryInputs?.() ?? []) { recoveryInputs.push(input) @@ -2053,6 +2101,10 @@ export async function runSessionHistory( } } try { + // Setup can complete a storage observation before any authored action. + // Consume it against the initial model, never against a later seed/write. + applyRecoveryInputs() + recordDurableMetadata() for (const [step, action] of history.entries()) { if (!isLegal(lane, model, action)) continue steps++ @@ -2077,6 +2129,7 @@ export async function runSessionHistory( } await driver.flush() applyRecoveryInputs() + recordDurableMetadata() mismatch = compare( step, action, @@ -2084,6 +2137,17 @@ export async function runSessionHistory( driver.observe(), driver.consumerCheckpoint === `final`, ) + // A later explicit write must not hide an earlier wrong durable keep. + // Store/read holds deliberately postpone this cut until their release. + if (!mismatch && !model.blocksPeerNotices) + mismatch = compareStoredMetadata( + step, + action, + lane, + model, + driver, + durableTrace, + ) if (mismatch) break } if (!mismatch) { @@ -2091,6 +2155,7 @@ export async function runSessionHistory( if (model.isHolding) model.release() while (model.depth > 0) model.cancel() applyRecoveryInputs() + recordDurableMetadata() if (rejections.length > 0) mismatch = { step: history.length, @@ -2118,18 +2183,15 @@ export async function runSessionHistory( actual, } } - if (!mismatch && lane.peers === true && driver.storedRowMetadata) { - const expected = model.storedRowMetadata() - const actual = driver.storedRowMetadata() - if (canonical(expected) !== canonical(actual)) - mismatch = { - step: history.length, - action: `settle`, - law: `keeping metadata does not write it to durable storage (decisions 9 and 18)`, - expected, - actual, - } - } + if (!mismatch) + mismatch = compareStoredMetadata( + history.length, + `settle`, + lane, + model, + driver, + durableTrace, + ) } } catch (error) { primary = error @@ -2160,6 +2222,40 @@ export async function runSessionHistory( return { mismatch, steps, recoveryInputs } } +/** Compare completed durable metadata independently of this tab's source read. */ +function compareStoredMetadata( + step: number, + action: SessionAction | `settle`, + lane: SessionLane, + model: SessionModel, + driver: SessionDriver, + durableTrace: ReadonlyArray>, +): SessionMismatch | undefined { + if (lane.peers !== true || !driver.storedRowMetadata) return undefined + // Empty transactions may stutter. Preserve every change of durable metadata, + // including a wrong write that a later queued commit repairs during release. + const changes = (trace: ReadonlyArray>) => + trace.filter( + (snapshot, index) => + index === 0 || canonical(snapshot) !== canonical(trace[index - 1]), + ) + const expected = driver.storedMetadataTrace + ? changes(durableTrace) + : model.storedRowMetadata() + const actual = driver.storedMetadataTrace + ? changes(driver.storedMetadataTrace()) + : driver.storedRowMetadata() + return canonical(expected) === canonical(actual) + ? undefined + : { + step, + action, + law: `keeping metadata does not write it to durable storage (decisions 9 and 18)`, + expected, + actual, + } +} + /** * A second formulation of hold invisibility, independent of the model. Two * production runs execute the same history, one with every `hold` and @@ -2775,5 +2871,6 @@ declareOracle({ seedReinsertReloadHistoryArbitrary, completedSeedReinsertReloadRegime, keptMetadataDurabilityHistoriesArbitrary, + queuedMetadataRepairHistoriesArbitrary, ], }) diff --git a/packages/db/tests/sync-metadata-session.test.ts b/packages/db/tests/sync-metadata-session.test.ts index d8bf30bc81..d793683ce6 100644 --- a/packages/db/tests/sync-metadata-session.test.ts +++ b/packages/db/tests/sync-metadata-session.test.ts @@ -1,6 +1,12 @@ import { describe, expect, it } from 'vitest' import { DbClient, collectionOptions } from '../src/client.js' import { createCollection } from '../src/collection/index.js' +import { + createSessionModel, + persistedPeerLane, + runSessionHistory, +} from './sync-metadata-session-oracle.js' +import type { SessionDriver } from './sync-metadata-session-oracle.js' import type { SyncConfig } from '../src/types.js' // Fixed boundaries of the sync metadata session. The session law itself is @@ -9,6 +15,55 @@ import type { SyncConfig } from '../src/types.js' type Row = { id: number; v: unknown } type SyncActions = Parameters[`sync`]>[0] +// This tests runner ordering, rather than introducing another product model. +it(`consumes completed setup inputs before authored actions`, async () => { + const order: Array = [] + let setup = true + const observation = createSessionModel(persistedPeerLane).observe() + const driver: SessionDriver = { + begin: () => { + order.push(`action`) + }, + commit: () => {}, + cancel: () => {}, + rowSet: () => {}, + rowDelete: () => {}, + insert: () => {}, + update: () => {}, + delete: () => {}, + truncate: () => {}, + collectionSet: () => {}, + collectionDelete: () => {}, + hold: () => {}, + release: () => Promise.resolve(), + flush: () => Promise.resolve(), + settle: () => Promise.resolve([]), + cleanup: () => Promise.resolve(), + observe: () => observation, + takeRecoveryInputs: () => { + order.push(`recovery`) + if (!setup) return [] + setup = false + return [ + { + type: `unknown-storage-observation`, + fromRowVersion: 0, + latestRowVersion: 1, + }, + ] + }, + } + const result = await runSessionHistory( + persistedPeerLane, + [{ type: `begin` }], + () => Promise.resolve(driver), + ) + expect(result.mismatch).toBeUndefined() + expect(order[0]).toBe(`recovery`) + expect(order[1]).toBe(`action`) + expect(result.recoveryInputs).toHaveLength(1) +}) + /** * Boundary witnesses for P11. A sync run that cleanup has ended reads nothing * and ignores writes; a metadata write with no open transaction throws. From ef5c66ed2d19841823b0a47de37ba047f2c16827 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 12:35:37 -0700 Subject: [PATCH 07/11] Clarify metadata fixture evidence wording --- .../2026-11-02-p6-metadata-closeout.md | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md index 0077f9d9d9..bfca37d6df 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -332,7 +332,7 @@ An old model expectation or open coverage note cannot supersede that decision. | --- | --- | --- | | An implicit keep does not write metadata; an explicit write does (D9/D18; 1, 4, 6). | The supplied nested bare-seed trace passed on both hosts in all three lanes, restoring cached M0 under D15 as independently predicted. A materialization mutant made an incorrect first durable write, then an explicit write repaired it; the old settlement-only runner passed. Seed 46 also produced eight old batches without a reload; reload-only corruption survived all 96 histories. | The authority accusation is false, but action-level durable comparison and guaranteed reload reach were missing. Both receiving paths now compare completed durable metadata before a later action can hide it. The controlled path also compares a transaction trace across held releases. Every constructive batch includes all three notice variants. | | Only completed storage observations advance the model, at their actual order (D15; 2, 3). | The old runner consumed inputs after its first authored action. The six actual factories recorded zero setup inputs, so no existing setup metadata failure was demonstrated. A separate runner-order witness supplies a setup input. | The recorder now withholds qualifying raw responses until hydration completes a storage read and application. Undrained responses fail a guard. Setup observations are consumed before authored actions; expected metadata still comes only from independent model storage. | -| Constructive histories execute their declared actions; fixture transactions preserve scope (5, 7, 8). | All 3,000 sampled sibling lane histories had no skipped actions. The caught-inner-failure SQL probe committed row 2 incorrectly; a root query remained queued behind its own callback. Empty housekeeping commits under two open authored transactions preserved source metadata, consumer rows, receipts and durable metadata. | The reach count mismatch requires illegal actions the constructive grammar excludes; a new guard makes that premise explicit. SQL nested scopes now use savepoints and async-context root reentry, while unrelated callers retain serialized admission. Empty quiescence commits remain a stuttering abstraction; maxDepth bounds authored histories, not the runtime API. | +| Constructive histories execute their declared actions; fixture transactions preserve scope (5, 7, 8). | All 3,000 sampled sibling lane histories had no skipped actions. The caught-inner-failure SQL probe committed row 2 incorrectly; a root query remained queued behind its own callback. Empty housekeeping commits preserved source metadata, consumer rows, receipts and durable metadata while two authored transactions remained open. | The reach count mismatch requires illegal actions the constructive grammar excludes; a new guard makes that premise explicit. SQL nested scopes now use savepoints and async-context root reentry, while unrelated callers retain serialized admission. Empty quiescence commits remain a stuttering abstraction; maxDepth bounds authored histories, not the runtime API. | The primary law, model and comparison remain in `packages/db/tests/sync-metadata-session-oracle.ts`. The controlled persisted @@ -509,3 +509,27 @@ semantic test/instrument repairs. The new pushed head needs its own checks and review coverage before a readiness verdict. This evaluation's accounting and authorized repairs are complete within the stated receiving boundary. It does not close #2281, #2272 or #2317/P8, or establish whole-library correctness. + + +## Current-head CodeRabbit wording follow-up + +Review `5742229429` covers `adf6a07f97b26116933ad68e3d5e1b2f6fe5093f`. +It contains one additional minor wording item and no inline comments. +`META-CR-5742229429-001` is **fixed-now**: the fixture evidence now says +“while two authored transactions remained open.” This uses the reviewer's +first suggestion and preserves the tested premise. The proposed hyphenated +term would imply a new transaction category and was not suitable. + +This is a clarity improvement, not a correctness defect. Direct comparison +of the sentence and its replacement, formatting, diff checks and unchanged +TypeScript hashes validate it. No product law, model, history or observation +changes, so no new RED test or mutant applies. The 253-test result and controls +above still describe the executable tree. The raw review was reread after the +edit: **1 = 1 fixed-now**, with no omitted suggestion or open item. + +Hire this reviewer for assisted documentation cleanup. The suggestion is +precise and low priority, but this sample supplies no new behavioral evidence. +The mutable summary now covers `adf6a07f9`, reports 100% docstring coverage +and no blocking issue, and notes that it skipped the coverage map as similar +to earlier changes. Final publication-head CI remains pending. This wording +follow-up adds no unreviewed executable change. From 60ace06b044e76fc9e57f2a6942a3c73593a2388 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 13:24:33 -0700 Subject: [PATCH 08/11] Calibrate metadata oracles and controlled recovery schedules Law: preserve D9/D15/D18 and synchronous cleanup invalidation; fixture SQL isolation remains a premise. Reach: all 18 recovery cells reach eight times per fixed/random campaign; existing 108-state and four-regime guards remain. RED: the new detached-context SQL witness fails both transaction entry points before fixture correction. GREEN: 279 local tests pass; scoped types, formatting, lint and unchanged-budget checks pass. Mutant: seven isolated wrong decisions fail their intended assertions; captured recovery seed/path replays RED and restored GREEN. Limits: bounded one-process native SQLite schedules; OPFS, Electron, restart, expiry, generation rotation and arbitrary overlap remain outside evidence. See the versioned instrument report. --- docs/contributing/oracle-coverage.md | 49 +- .../2026-11-02-p6-instrument-calibration.md | 253 ++++++++ .../2026-11-02-p6-metadata-closeout.md | 9 + .../tests/sqlite-fixture-conformance.ts | 166 ++++++ .../tests/sync-metadata-calibration.test.ts | 240 ++++++++ .../sync-metadata-campaign-result.test.ts | 31 + .../tests/sync-metadata-campaign-result.ts | 8 + .../sync-metadata-recovery-recorder.test.ts | 80 +++ .../tests/sync-metadata-recovery-recorder.ts | 42 ++ .../tests/sync-metadata-recovery.test.ts | 70 +++ .../tests/sync-metadata-session-campaign.ts | 7 +- .../sync-metadata-session-persisted-driver.ts | 541 ++++++++++++++++++ .../sync-metadata-session-persisted.test.ts | 497 +--------------- .../sync-metadata-session-sqlite-driver.ts | 540 +++++++++++++++++ .../sync-metadata-session-sqlite.test.ts | 455 +-------------- .../tests/sync-metadata-sqlite-driver.test.ts | 88 +-- .../tests/sync-metadata-sqlite-driver.ts | 18 +- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 5 + .../db/tests/sync-metadata-recovery-oracle.ts | 263 +++++++++ .../db/tests/sync-metadata-session-oracle.ts | 57 +- 21 files changed, 2367 insertions(+), 1053 deletions(-) create mode 100644 docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md create mode 100644 packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts create mode 100644 packages/db/tests/sync-metadata-recovery-oracle.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index f23d71ebbb..d71a7f2554 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4765,8 +4765,9 @@ reload after a named notice keeps a seed's metadata (decision 18). | Owner | Production paths | Observations and checkpoint | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [sync metadata session oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/sync-metadata-session-oracle.ts) (`flat`, `nested`, `seeds`, `nested-seeds`, `seeds-full`, `nested-conflicts` lanes) | `DbClient` Collection, core hold, `DbClient.hydrate` seeds | `metadata.row.get`, `metadata.collection.get`, `list()`, `list(prefix)`, `collection.base`, each commit receipt's state, thrown errors; after every action and after settlement | -| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, stored rows after settlement and stored row metadata at every completed durable action cut and after settlement | -| [native SQLite/coordinator receiver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts) | two real Collections and coordinators, node:sqlite, BroadcastChannel and Web Locks; partial/full/on-demand lanes | shared model and source reads after correlated commit/notice/recovery cuts; rows, metadata and applied receipts after settlement; direct durable metadata after each correlated action and durable rows after settlement | +| [persisted driver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts) and its [campaigns](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts) (`persisted-flat`, `persisted-nested`, `persisted-seeds`, `persisted-cached`, `persisted-conflicts`, `persisted-nested-seeds`, `persisted-reads`, `persisted-reads-seeds`, `persisted-peers`, `persisted-peers-full`, `persisted-peers-on-demand`) | `persistedCollectionOptions` over an in-memory adapter, store hold, core hold, and a storage read held in flight; seeds through a `DbClient`, alone and crossed with nesting and a row cached in storage at startup; a peer's notices, eager, under full row updates, and on demand | the same source reads after every action; `collection.base` and receipts after settlement; in the peer lanes, stored rows after settlement and stored row metadata at every completed durable action cut and after settlement | +| [native SQLite/coordinator receiver](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts) and its [campaigns](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts) | two real Collections and coordinators, node:sqlite, BroadcastChannel and Web Locks; partial/full/on-demand lanes | shared model and source reads after correlated commit/notice/recovery cuts; rows, metadata and applied receipts after settlement; direct durable metadata after each correlated action and durable rows after settlement | +| [recovery schedules](https://github.com/TanStack/db/blob/main/packages/db/tests/sync-metadata-recovery-oracle.ts) using the same session model and native receiver | response delivery, storage reading and captured-row application held separately; bare/non-bare seeds, a second tab's source write, or cleanup during one recovery on demand | source metadata before application, after the interruption and after completion; stale source reads at cleanup start and after late work; direct durable metadata after completion | | paired properties and boundary witnesses in the core driver file | production against itself: holds removed, row 2's actions removed, or seeds performed first; stale run; write without a transaction; cleanup with a held seed | source reads (and row 1's view) at each shared checkpoint, or every observation after settlement for seed order; thrown errors; unhandled rejections | | [row metadata composition oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/collection-row-metadata-composition-oracle.test.ts) held-window and nested lanes | Collection, core hold, a transaction begun inside an open one, `DbClient.hydrate` | implicit row-metadata rules during a hold, and for an outer transaction whose inserts are reclassified after an inner write applies first, with and without a hold | | [hydration authority oracle](https://github.com/TanStack/db/blob/main/packages/db/tests/db-client-hydration-authority-oracle.test.ts) | `DbClient` initialData, hydration chunks, and an adapter sync transaction (inserts, updates, deletes, truncates, and metadata-only writes) in full and partial row modes | public rows, the synced store, and row metadata through `client.dehydrate()` after each action; D5 per object | @@ -4847,14 +4848,43 @@ snapshots from empty commits. A constructive immediate/store-held × bare/metadata-bearing repair campaign declares all four regimes, eight times each. Native commits are individually awaited, so its action cuts already separate these writes. No claim covers intermediate SQL statements. -This is one legal schedule; arbitrary response/application interleavings, expiry, -cleanup during recovery and the necessity of gap detection are not established. +The ordinary campaigns retain this serialized schedule. A separate fixed/random +recovery campaign constructs all 18 response/read/application × seed/peer/cleanup +× bare/non-bare states per batch. Every state reaches its held source comparison +eight times in each normal campaign. The application hold captures the rows +before returning them to the wrapper; it does not delay only an already-applied +promise. Peer writes can complete before a held response is delivered; while a +hydration scope owns SQLite admission they wait until that scope ends. Cleanup +invalidates the sync run before the held work is released. The recorder discards +its inputs, including a hydration task that finishes after invalidation. The +shared model still derives every metadata expectation from its own storage. +Expiry, persisted cache generation rotation, arbitrary overlapping recoveries, +restart and the necessity of gap detection remain outside these schedules. + +Permanent [calibration controls](https://github.com/TanStack/db/blob/main/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts) +feed wrong intermediate durable snapshots, reload-only wrong reads, temporary +wrong reads at every held recovery cut and stale reads after cleanup through +the same comparisons. Recorder controls cover failed tasks, missing reads, +late responses and invalidation. A real fast-check obsolete-path control +demonstrates zero-history GREEN and verifies the shared execution guard rejects +it. These are checker and fixture evidence, not new product oracles or a +production mutation score. The +[instrument audit](oracle-reviews/2026-11-02-p6-instrument-calibration.md) +connects each claim to authority, history, receiving path, checkpoint and control. The receiver serializes every root SQL call with transaction admission on its one physical connection, while scoped transaction calls stay inside that admission. Nested scopes use savepoints; same-transaction async root reentry uses that scope. Nested rollback/root-reentry and cross-adapter isolation -witnesses reject missing savepoints and unqueued root calls. The driver waits for actual correlated notices, remote ensure replies +witnesses reject missing savepoints and unqueued root calls. The reusable SQL +fixture conformance suite checks both transaction entry points, caught inner +rollback, successful inner scopes followed by outer rollback, deeper scopes, +root reentry, unrelated-caller isolation, detached callbacks from completed +transactions, and reuse after failure. A detached callback loses root reentry +when its transaction ends, so it cannot read or join a later unrelated scope. +The new witness failed before this fixture correction. The suite certifies +this native fixture, without lending that result to every shipped SQL driver. +The driver waits for actual correlated notices, remote ensure replies and accepted empty source commits through the wrapper's apply lock, rather than a quiet-time polling interval. It covers two real Browser coordinators over Node BroadcastChannel/Web Locks and native SQLite in one process. Chromium @@ -4874,6 +4904,15 @@ remains separate and open under this owner, with P8 investigating: property `36:2:4:6:10:14:16:14:14:13:13:13`. P6's durability evidence does not close that controlled-notice witness or the broader #2281 persistence reports. +Remaining observation work under this owner: the read-hold and abandonment +lanes retain source-facing comparisons but do not provide the peer lanes' +direct durable transaction trace. A receiving witness must stage a wrong +durable write followed by repair while a read hold buffers a commit, including +abandonment, and reject it at the appropriate durable boundary. These additions +do not retire that obligation. Cross-host recovery schedules still need actual +Chromium OPFS and Electron IPC receiving witnesses; their owner remains this +session oracle with the broader receiving work tracked by #2272. + `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end (decision 21); a seed that arrived after it stays judged against it diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md b/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md new file mode 100644 index 0000000000..f6491ac56a --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md @@ -0,0 +1,253 @@ +# P6 instrument calibration and controlled recovery schedules + +This follow-up implements the four improvements requested after the ten-item +revision review. Its starting revision is +`ef5c66ed2d19841823b0a47de37ba047f2c16827`. The earlier +[closeout record](2026-11-02-p6-metadata-closeout.md) retains the original +review, seven repairs, three refutations and the replay sibling's disposition. +This record does not replace that history or reopen its withdrawn production +diagnosis. No product law or production source changes here. + +## Authority and independent judgment + +D9 and D18 distinguish keeping this tab's metadata from writing it durably. +D15 governs a storage observation without transaction authorship. Those dated +maintainer decisions remain in +[the authority record](2026-10-22-sync-metadata-session.md). The glossary's +cleanup rule invalidates the sync run synchronously; callbacks from the ended +run cannot supply new source reads. SQL rollback and admission are premises of +the native fixture, rather than additional metadata policy. + +The session model still supplies every expected metadata value. Its existing +peer action is split into a durable write and notice delivery for schedules +that observe storage between them. The ordinary peer action retains the same +order and semantics. No new model state or production recovery classifier is +introduced. The new recovery grammar imports the session model, action planner +and source comparison. Raw driver observations never compute expected metadata. + +## The coverage chain + +| Claim and authority | Legal history and reach | Actual receiving path and observation cut | Distinguishing control | +| --- | --- | --- | --- | +| A keep does not become durable, D9/D18 | Existing immediate/store-held × bare/non-bare paired writes; all four regimes retained | Controlled persisted driver; the ordered durable transaction trace, before a later explicit repair can hide the wrong write | Permanent control inserts an incorrect intermediate snapshot while preserving the correct final state; the shared comparison rejects it before settlement | +| Reload must be observed to distinguish a local keep, D9/D15/D18 | Existing constructive grammar retains all 36 cases and 108 case/lane/notice states per host | Controlled and native SQLite receivers; source reads after actual full-reload delivery | Permanent reload-only corruption reaches the actual reload on both hosts and fails the shared source comparison | +| Response delivery alone is insufficient, D15 | One recovery; all three held phases crossed with seed, peer or cleanup and both seed metadata forms: 18 states per batch | Two real Collections/coordinators, native SQLite, Node BroadcastChannel/Web Locks, on demand; source reads before application, after interruption and after completion | Temporary wrong reads are supplied only while each hold is active. Their final state is correct; every held-cut control is rejected | +| Cleanup invalidates an ended sync run, glossary | Cleanup at response, storage read or captured-row application, for both seed forms | Same native receiver; stale source reads at cleanup start and after late work; direct durable metadata after completion | Permanent stale-read controls fail at cleanup start. Recorder controls reject credit for a hydration task that completes after invalidation | +| The fixture preserves nested rollback and unrelated callers | Both transaction entry points, scoped/root reentry, inner failure caught by outer, outer failure after successful deeper scopes, detached callbacks from ended scopes, and subsequent reuse | The same one-connection fixture used by the metadata receiver; SQL reads after rollback and a read queued while another callback owns admission | Missing-savepoint, unqueued-root and stale-context controls are tested in an isolated copy | +| A replay must execute a history, ORC-007/015 | Actual fast-check seed 2220031, valid path `0` and obsolete path `0:0:0` against the durability grammar | The shared campaign execution guard; observed property invocation count | The obsolete path returns fast-check GREEN with zero invocations. The guard rejects it; the valid path executes once | + +The normal recovery campaigns run the same property eight times with fixed +seed `2220035` and eight times without a seed. Every batch constructs all 18 +states, so every declared state reaches its comparison eight times per campaign. +The grammar's independent calibration retains the literal required phase and +interruption set; removing an axis cannot silently shrink its declared reach. +The real holds report the exact response/read/application prefix. A five-second +premise deadline classifies failure to reach the cut separately from a metadata +mismatch. The final durable comparison uses model-owned storage, including when +cleanup discards a successful transport response. + +A peer commit can finish before a held response is released. During a hydration +scope it instead waits for native transaction admission. The runner advances +model storage in that corresponding order. The application hold retains the +captured rows before the wrapper receives them. It does not merely delay a +promise after rows have already applied. Cleanup can legitimately abandon work; +only an `AbortError` in that cleanup schedule is accepted. Other errors remain +failures, and cleanup diagnostics preserve the original law and checkpoint. + +## Permanent machinery and tests + +The controlled and native production drivers now live in ordinary support +modules. Their test files retain registration, replay selection and campaign +budgets. Calibration tests import the drivers without registering an unrelated +campaign. The reusable SQL fixture suite runs through `transaction` and +`transactionWithDriver`; the three original fixture cases remain covered. +Recorder tests cover transport-only responses, missing storage reads, failed +application, responses delivered after a task starts, and invalidation during a +task. A shared nonzero-execution guard is used by the constructive campaigns. + +The wrong observations in permanent calibration tests are intentionally hostile +inputs to the same comparison. They are not new product models. Passing them +means the comparison rejected those specific corruptions at a reached cut. +It does not establish a production mutation score or universal correctness. + +## Remaining limits and review status + +The new schedules cover one recovery in the on-demand native receiver. They do +not establish arbitrary overlapping recoveries, cache claim expiry, persisted +cache generation rotation, leadership turnover, transport loss, restart, +Chromium OPFS or Electron IPC. The broader receiving work remains with #2272. +These schedules compare source metadata and final durable metadata, rather than +claiming a complete publication/event trace during recovery. + +Read-hold and abandonment lanes still have source-facing coverage without the +peer lanes' direct durable transaction trace. The coverage map names this owner +and the needed wrong-write/repair/abandonment witness explicitly. This is remaining +observation work, not a retired law. The open #2317/P8 controlled-notice witness +and the broader #2281 reports remain separate from this bounded addition. + +This is an author self-review. Earlier CodeRabbit coverage of `adf6a07f9` and +`ef5c66ed2` does not review these semantic instrument changes. CI success and +third-party review coverage must be reported separately for the new publication. + +## Rubric audit for the changed instrument + +| Requirement | Outcome | +| --- | --- | +| ORC-001 | D9/D15/D18 and synchronous sync-run invalidation are the authority; the history/path/observation limits above bound every claim. | +| ORC-002 | The same independent session model predicts metadata. The recorder returns version facts only; it never supplies an expected metadata value. | +| ORC-003 | Contract, grammar, shared model, native driver boundaries and comparison cuts remain visible in the owner and directly named companion modules. | +| ORC-004 | The initial nested keep reconstructs the known seed/re-insert/recovery witness. Phase, interruption and seed form contribute all 18 cells. Payloads range from -100 through 100. Independent literal grammar calibration catches axis removal; an unsupported boundary is rejected by the same driver/runner instead of credited. Existing constructive action legality and completed-history guards remain. | +| ORC-005 | The native boundary holds response return, actual storage-read entry, or captured rows before application. Prefix witnesses and completed input guards prove those cuts ran. Peer admission follows the host scope. Final agreement is not credited for the held source observations. | +| ORC-006 | Permanent wrong-observation controls and isolated deliberate mutations challenge the exact comparisons, recorder, scope and replay guard. Outcome categories and concrete failures are recorded below. | +| ORC-007 | Fixed and random campaigns share grammar, driver, comparison and eight-run budgets. The property is registered with its owner and direct replay interface. Captured failure replay and restored execution are checked below. | +| ORC-008 | No model state is added or removed. Splitting a peer's durable write from its notice is distinguishable by a recovery read between them. The ordinary peer action retains both events in their prior order. | +| ORC-009 | Response, storage reading and application are separate driver boundaries. `stagePeerWrite` is explicitly the model projection of durable input before notice delivery. Cleanup retains the glossary meaning. | +| ORC-010 | Failure capture includes the concrete schedule and named cut. Pending work drains before database close; an AggregateError preserves the primary failure and secondary cleanup diagnostics. Only expected cleanup abandonment accepts AbortError. | +| ORC-011 | No new semantic policy or named shared semantic-fault hypothesis requires a second formulation here. Existing hold/key/seed relational properties are retained. The new schedule reuses the primary model rather than duplicating its policy. | +| ORC-012 | This versioned record audits the changed chains and limits. Earlier broader session evidence remains in the linked closeout record. Executable revisions, commands and results follow below. | +| ORC-013 | Both seed metadata forms, named versus unknown reloads, transient versus final corruption, and before-versus-after application distinguish neighboring wrong rules. Scope is finite; no universal recovery closure is claimed. | +| ORC-014 | Actual node:sqlite/coordinator traffic supplies the new receiving premise. The controlled durable trace does not establish arbitrary native interleavings. Browser OPFS and Electron receiving gaps remain explicitly owned. | +| ORC-015 | All 18 recovery states must reach at least once; normal fixed/random campaigns reach each eight times. Existing 108-state durability, four-regime held-write and sibling guards and budgets remain. | +| ORC-016 | The new oracle module owns grammar and shared judgment only. Ordinary drivers, SQL fixture controls, recorder units and calibration tests stay in ordinary modules. No fixed production metadata answer is restated outside the primary model. | + + +## Additional fixture finding: META-INST-S001 + +The reusable conformance pass found a test-fixture isolation defect. The old +`AsyncLocalStorage` remained true in a detached callback after its +transaction ended. Root SQL from that callback could bypass admission while +another transaction was open. The old tests covered an unrelated caller and +active root reentry, but omitted an inherited context after its scope ended. + +The new witness runs through both transaction entry points. Transaction A +commits row 1 and leaves a detached callback waiting. Transaction B inserts row +2 and waits before rollback. The detached callback then submits a root read +and insert of row 3. Its read must wait for B and see only row 1; its write must +survive B's rollback. Before the fixture edit, both tests failed: the read +returned rows 1 and 2. This is RED at the intended isolation assertion, not a +timeout or a production metadata finding. + +The repair gives each outer transaction an active scope and invalidates it in +`finally`. Active root reentry remains supported; callbacks from ended scopes +queue. The ten-case reusable fixture suite and the complete native receiver +suite pass after the repair. Removing that invalidation in the isolated copy +reproduces the same read failure. This finding is **fixed-now**. It is separate +from the earlier ten raw review items and does not reopen the withdrawn +production diagnosis or close #2281, #2272 or #2317/P8. + +## Executed evidence + +Environment: Node 24.19.0, pnpm 11.1.0 and Vitest 4.1.12. Commands below run +with `--maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +--disableConsoleIntercept`. No oracle budget was lowered or removed. + +| Run, from the named package | Result | +| --- | --- | +| `db`: `pnpm exec vitest run tests/sync-metadata-session-oracle.property.test.ts tests/sync-metadata-session.test.ts tests/collection-row-metadata-composition-oracle.test.ts tests/oracle-config.test.ts tests/oracle-replay.test.ts` | 163 tests passed in four executed files, 151.73 seconds | +| `db-sqlite-persistence-core`: `pnpm exec vitest run tests/sync-metadata-session-persisted.test.ts` | 76 tests passed, 406.40 seconds | +| `db-sqlite-persistence-core`: `pnpm exec vitest run tests/sync-metadata-session-sqlite.test.ts tests/sync-metadata-recovery.test.ts tests/sync-metadata-calibration.test.ts tests/sync-metadata-recovery-recorder.test.ts tests/sync-metadata-campaign-result.test.ts tests/sync-metadata-sqlite-driver.test.ts tests/sync-metadata-write-work.test.ts` | 40 tests passed in seven files; includes both recovery campaigns, 11 calibration tests, five recorder tests and ten SQL fixture cases | + +These are 279 local tests. The changed scope also passes a source-mapped, +no-emit TypeScript check, formatting, general ESLint and `pnpm lint:changed`. +The scoped general ESLint run reports zero errors and eight warnings. It is +not a whole-repository lint claim. No install workaround was needed. + +Each normal recovery campaign reports all 18 states at count 8. Existing +constructive durability campaigns retain all 108 states per host, each at +count 8. The controlled queued-repair property retains all four regimes, each +at count 8. Their fixed/random budgets and the older 40-run budgets remain. + +### Deliberate mutations + +An isolated archive of the starting revision, overlaid with the final test +sources, supplies the baseline. It shares dependency directories, not source +files, with the working tree. The unmodified focused baseline passes 29 tests +in five files. Each mutation changes one decision and is restored afterward. +These are self-generated challenges informed by the known gaps, not held-out +independent review or an estimated production mutation score. + +| ID and exact decision changed | Target and observed rejection | +| --- | --- | +| `drop-response`: return false on an OK pull inside `recoverFromSeqGapUnsafe` | Fixed recovery campaign; source metadata differs at `completed response` | +| `no-savepoint`: replace the scoped transaction body with `work(scoped)` | Both transaction entry points fail the caught-inner-rollback value assertion | +| `unqueued-root-read`: replace the root query's queued branch with a direct scoped query | Both entry points expose an unrelated transaction's uncommitted row | +| `stale-transaction-context`: remove `scope.active = false` | Both entry points fail the detached-context read assertion described above | +| `ignore-durable-trace`: make both `driver.storedMetadataTrace` selections false in the session comparison | Permanent intermediate-write calibration fails because the comparison accepts the corruption | +| `credit-invalidated-task`: remove the recorder's `started !== generation` early return | Recorder's task-after-cleanup value assertion fails | +| `accept-zero-replay`: replace the shared executed-history guard with an empty function | The actual obsolete-path calibration fails because zero-invocation GREEN is accepted | + +All seven outcomes are **assertion failures at the intended comparison or +calibration**. None is credited from setup failure, runtime crash or timeout. +No selected mutant survives this bounded experiment. This says nothing about +unselected wrong designs. + +### Direct replay + +The dropped-response campaign shrinks to seed `2220035`, path `0:0`. With that +mutation active, the direct replay selects only the requested property and +reproduces the same completed-response metadata mismatch. Restoring production +passes the same replay and executes six batches; every recovery state reaches +six times. The initial replay verifier incorrectly required exactly one batch. +Its failed bookkeeping result is retained in scratch and superseded by this +checked nonzero-execution result; the oracle itself passed. No run budget or +shared replay configuration was changed to satisfy that mistaken assumption. + +```sh +TANSTACK_DB_ORACLE_PROPERTY=sync-metadata-session.sqlite-recovery-schedules \ +TANSTACK_DB_ORACLE_SEED=2220035 TANSTACK_DB_ORACLE_PATH=0:0 \ +pnpm exec vitest run tests/sync-metadata-recovery.test.ts \ + --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false \ + --disableConsoleIntercept +``` + +The obsolete-path permanent control independently checks seed `2220031`, path +`0:0:0`, against the durability grammar. Fast-check reports GREEN with no +property invocation, and the shared guard rejects it. Valid path `0` executes +one invocation. These two experiments test different replay obligations. + + +### Tested source identity + +The following SHA-256 values bind all 18 changed TypeScript modules to the +executed instrument. Unchanged production is from the starting revision +`ef5c66ed2d19841823b0a47de37ba047f2c16827`. Core and controlled suites ran +before a later comment-only owner edit; executable semantics are unchanged. +The native suite, isolated baseline, mutations and replay use these final +source contents. Documentation revisions do not extend third-party review. + +| Repository path | SHA-256 | +| --- | --- | +| `packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts` | `8c4c6d9d28f42d9e8b461233fbbe3d48d84c0582d9decc9abb935d2c85883ba9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts` | `dc960530b4286b860e6632979df8c1f2c1ce5c86df76a0d084dc59191759e431` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts` | `f75952c705f60519b783cf86887ada61c41c2eca86db98daa773fabc494775b9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts` | `66f2447c1116f3d10bdcee8d6a97192f81bd5e855afc3a69fbee163a75cca7cb` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts` | `a318b173252956b8a7424817d371c1aa2ad5ce3152ad1c40d43fc1bc27259971` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts` | `591206dbde1605b306bcd62b65ff2f5a9d1a001befc648d076ca052138e11a69` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts` | `880e40bd7f6aaf452dd2f3512a4ed0eb027feadaac221e1846dd9cfe0354dbfb` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts` | `b1de1ba24d79db954a4d85d6d7da18422ac29480d7175893d90b1ab07b24f1ae` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts` | `908ae60996e225757c7f6db8ab8408a5c44d7efd6ae1000188ced6588a133240` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts` | `3296a963a8737f84c72b6897817c44fd5c9b57d3fb723307115d78b3df03ded9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts` | `42c6e6257de7c437005e4e0d7a297514178a95998000acf3931032140dc16e07` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts` | `7d4d547c70cd370dab328ef289689bb85c7e4c223907ba508f71158672af181d` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts` | `7582a61d10a944dd9bd794d5deb76bca3c6996158f3c9ef229493b98cc1ca2e9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts` | `e5ef179a2a50d5e42f8b0ee51872885141056e2eef029a620f4ccd53a5512e53` | +| `packages/db/tests/oracle-config.ts` | `36eb19aafe2e36728a6f6798056b8f413cbbd1e3cfba01fdcd95807b2ad9b1c9` | +| `packages/db/tests/oracle-replay-manifest.ts` | `66164d47358978cad61dadbcba4c8d619983added3bf825f0ebda9d65104e496` | +| `packages/db/tests/sync-metadata-recovery-oracle.ts` | `ed05738b4d44411e617b82eccdd62ca51d4e5749ad4f698be9fd86f582976e13` | +| `packages/db/tests/sync-metadata-session-oracle.ts` | `8ad9a7dd1ab3002677461c5117e001cd7993eb45fb42b0bc0fdad268db3430b6` | + + +### Integration and final accounting + +Fetched main is `9be51e037f6c092bc9625f9b5b4b7e23eb4fa5fe`. +The PR's three-dot diff has no production, manifest, lockfile or changeset +delta. Main has since added held query-refetch coverage and negative-limit +admission; this branch has not incorporated those commits. The local results +above describe this branch and its existing dependencies. Publication-head CI +must supply integration evidence; earlier green CI is not transferred. + +The four requested improvements are implemented. The additional finding is +`1 = 1 fixed-now` (META-INST-S001), with no open finding or product decision +from this pass. The known receiving and observation limits above remain open; +they are not removed from the contract by this accounting. The earlier ten-item +review retains its independent `10 = 7 fixed-now + 3 refuted` accounting. diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md index bfca37d6df..fd6faabc9d 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-metadata-closeout.md @@ -533,3 +533,12 @@ The mutable summary now covers `adf6a07f9`, reports 100% docstring coverage and no blocking issue, and notes that it skipped the coverage map as similar to earlier changes. Final publication-head CI remains pending. This wording follow-up adds no unreviewed executable change. + +## Subsequent instrument improvements + +The maintainer requested permanent calibration, controlled recovery schedules, +reusable SQL fixture conformance and a coverage-chain audit after this closeout. +[The instrument calibration record](2026-11-02-p6-instrument-calibration.md) +records that follow-up against this report's publication head `ef5c66ed2`. +Earlier test results and review coverage above remain tied to their recorded +revisions; they do not establish CI or third-party review for the additions. diff --git a/packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts b/packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts new file mode 100644 index 0000000000..b681a670ec --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts @@ -0,0 +1,166 @@ +import { expect, it } from 'vitest' +import type { SQLiteDriver } from '../src' + +/** Reusable fixture checks. The factory owns its database and resource cleanup. + * These supplement product oracles: their expected values describe SQL scope, + * rollback and admission, not metadata policy. + */ +export function registerSQLiteFixtureConformance( + createFixture: () => Promise<{ + driver: SQLiteDriver + cleanup: () => Promise + }>, +) { + it(`queues a detached caller after its original transaction has ended`, async () => { + const { driver, cleanup } = await createFixture() + let releaseLate!: () => void + const lateGate = new Promise((resolve) => { + releaseLate = resolve + }) + let submitted!: () => void + const lateSubmitted = new Promise((resolve) => { + submitted = resolve + }) + let late: Promise | undefined + let releaseCurrent!: () => void + const currentGate = new Promise((resolve) => { + releaseCurrent = resolve + }) + let entered!: () => void + const currentEntered = new Promise((resolve) => { + entered = resolve + }) + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + await driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (1)`) + // This callback inherits the first transaction's async context, but + // invokes root SQL only after that transaction has completed. + late = lateGate.then(() => { + const read = driver.query(`SELECT id FROM rows ORDER BY id`) + const write = driver.run(`INSERT INTO rows VALUES (3)`) + submitted() + return Promise.all([read, write]) + }) + }) + const current = driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (2)`) + entered() + await currentGate + throw new Error(`rollback current scope`) + }) + const rejected = expect(current).rejects.toThrow(`rollback current scope`) + await currentEntered + releaseLate() + await lateSubmitted + releaseCurrent() + await rejected + expect(await late).toEqual([[{ id: 1 }], undefined]) + expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ + { id: 1 }, + { id: 3 }, + ]) + } finally { + releaseLate() + releaseCurrent() + await late?.catch(() => undefined) + await cleanup() + } + }) + it.each([false, true])( + `rolls back only a failed nested scope (root reentry: %s)`, + async (rootReentry) => { + const { driver, cleanup } = await createFixture() + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + await driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (1)`) + const nested = rootReentry ? driver : scoped + await expect( + nested.transaction(async (inner) => { + await inner.run(`INSERT INTO rows VALUES (2)`) + throw new Error(`rollback the inner scope`) + }), + ).rejects.toThrow(`rollback the inner scope`) + await nested.transaction(async (inner) => { + await inner.run(`INSERT INTO rows VALUES (3)`) + }) + // Root calls in this async transaction must use its scope, rather than + // queue behind the callback that is awaiting them. + await (rootReentry ? driver : scoped).run( + `INSERT INTO rows VALUES (4)`, + ) + }) + expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ + { id: 1 }, + { id: 3 }, + { id: 4 }, + ]) + } finally { + await cleanup() + } + }, + ) + + it(`keeps another adapter's root reads and writes outside a rolled-back transaction`, async () => { + const { driver, cleanup } = await createFixture() + let entered!: () => void + const started = new Promise((resolve) => { + entered = resolve + }) + let release!: () => void + const gate = new Promise((resolve) => { + release = resolve + }) + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + const failed = driver.transaction(async (scoped) => { + await scoped.run(`INSERT INTO rows VALUES (1)`) + entered() + await gate + throw new Error(`rollback this transaction only`) + }) + const rejected = expect(failed).rejects.toThrow( + `rollback this transaction only`, + ) + await started + const read = driver.query(`SELECT id FROM rows ORDER BY id`) + const write = driver.run(`INSERT INTO rows VALUES (2)`) + release() + await rejected + expect(await read).toEqual([]) + await write + expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ + { id: 2 }, + ]) + } finally { + release() + await cleanup() + } + }) + + it(`rolls back a successful nested scope when the outer scope fails`, async () => { + const { driver, cleanup } = await createFixture() + const error = new Error(`outer rollback`) + try { + await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) + await expect( + driver.transaction(async (outer) => { + await outer.run(`INSERT INTO rows VALUES (1)`) + await outer.transaction(async (inner) => { + await inner.run(`INSERT INTO rows VALUES (2)`) + await inner.transaction(async (deepest) => { + await deepest.run(`INSERT INTO rows VALUES (3)`) + }) + }) + throw error + }), + ).rejects.toBe(error) + expect(await driver.query(`SELECT id FROM rows`)).toEqual([]) + await driver.run(`INSERT INTO rows VALUES (4)`) + expect(await driver.query(`SELECT id FROM rows`)).toEqual([{ id: 4 }]) + } finally { + await cleanup() + } + }) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts new file mode 100644 index 0000000000..de240fd9ff --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts @@ -0,0 +1,240 @@ +// @vitest-environment node +import { fc } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { + persistedPeerLane, + persistedPeerOnDemandLane, + queuedMetadataRepairHistoriesArbitrary, + runSessionHistory, + seedReinsertReloadReplay, +} from '../../db/tests/sync-metadata-session-oracle' +import { + recoveryPhases, + recoverySchedulesArbitrary, + runRecoverySchedule, +} from '../../db/tests/sync-metadata-recovery-oracle' +import { createPersistedDriver } from './sync-metadata-session-persisted-driver' +import { createSQLiteDriver } from './sync-metadata-session-sqlite-driver' +import type { SessionDriver } from '../../db/tests/sync-metadata-session-oracle' +import type { RecoveryPhase } from '../../db/tests/sync-metadata-recovery-oracle' + +it(`rejects a recovery schedule whose boundary the driver cannot hold`, async () => { + await expect( + runRecoverySchedule( + { + phase: `after-application` as RecoveryPhase, + interruption: `seed`, + bare: false, + value: 0, + }, + () => createSQLiteDriver(persistedPeerOnDemandLane), + ), + ).rejects.toThrow(`commit completed without reaching after-application`) +}) + +it(`retains every recovery boundary and interruption in each generated batch`, () => { + const required = new Set( + [`response`, `read`, `application`].flatMap((phase) => + [`seed`, `peer`, `cleanup`].flatMap((interruption) => + [false, true].map((bare) => `${phase}/${interruption}/${bare}`), + ), + ), + ) + for (const batch of fc.sample(recoverySchedulesArbitrary, { + seed: 2220035, + numRuns: 8, + })) { + expect(batch).toHaveLength(18) + expect( + new Set( + batch.map( + ({ phase, interruption, bare }) => `${phase}/${interruption}/${bare}`, + ), + ), + ).toEqual(required) + } +}) + +// Deliberately wrong observations calibrate the real shared comparison. These +// are sensitivity tests, not additional product models or production mutants. +it(`rejects a wrong durable write even when a later queued write repairs it`, async () => { + const cases = fc.sample(queuedMetadataRepairHistoriesArbitrary, { + seed: 2220031, + numRuns: 1, + })[0]! + for (const { history, bare, held } of cases) { + const baseline = await runSessionHistory(persistedPeerLane, history, () => + createPersistedDriver(true, true, false, true), + ) + expect(baseline.mismatch).toBeUndefined() + let injected = false + const hostile = await runSessionHistory( + persistedPeerLane, + history, + async () => { + const driver = await createPersistedDriver(true, true, false, true) + const trace = driver.storedMetadataTrace! + const commit = driver.commit + let commits = 0 + driver.commit = async () => { + await commit() + commits++ + } + driver.storedMetadataTrace = () => { + const snapshots = trace() + if (commits < 2 || snapshots.length < 2) return snapshots + injected = true + // The final snapshot remains correct. Preserve a wrong intermediate + // value at the durable transaction cut where a later repair hid it. + return [ + snapshots[0]!, + { ...snapshots[0], 1: { corrupt: `kept seed became durable` } }, + ...snapshots.slice(1), + ] + } + return driver + }, + ) + expect(injected, `${bare}/${held}`).toBe(true) + expect(hostile.mismatch?.law).toContain(`keeping metadata does not write`) + expect(hostile.mismatch?.action).not.toBe(`settle`) + } +}, 30_000) + +const hosts: Array<{ name: string; create: () => Promise }> = [ + { + name: `controlled`, + create: () => createPersistedDriver(true, true, false, true), + }, + { + name: `SQLite`, + create: () => createSQLiteDriver(persistedPeerOnDemandLane), + }, +] +it.each(hosts)( + `rejects wrong source metadata that appears only after reload ($name)`, + async ({ create, name }) => { + const lane = + name === `SQLite` ? persistedPeerOnDemandLane : persistedPeerLane + expect( + (await runSessionHistory(lane, seedReinsertReloadReplay, create)) + .mismatch, + ).toBeUndefined() + let reached = false + const result = await runSessionHistory( + lane, + seedReinsertReloadReplay, + async () => { + const driver = await create() + const peer = driver.peer! + const observe = driver.observe + driver.peer = async (...args) => { + await peer(...args) + if (args[1] === `reload`) reached = true + } + driver.observe = () => { + const actual = observe() + return reached + ? { + ...actual, + rowMetadata: { + ...actual.rowMetadata, + 1: { corrupt: `reload` }, + }, + } + : actual + } + return driver + }, + ) + expect(reached).toBe(true) + expect(result.mismatch?.law).toBe( + `source reads its row metadata (metadata.row.get)`, + ) + expect(result.mismatch?.action).toMatchObject({ + type: `peer`, + notice: `reload`, + }) + }, +) + +it.each(recoveryPhases)( + `compares source metadata while recovery is held at %s`, + async (phase) => { + let injected = false + await expect( + runRecoverySchedule( + { phase, interruption: `seed`, bare: false, value: 7 }, + async () => { + const driver = await createSQLiteDriver(persistedPeerOnDemandLane) + const hold = driver.holdRecovery + let holding = false + driver.holdRecovery = (selected) => { + const gate = hold(selected) + return { + reached: gate.reached.then(() => { + holding = true + }), + release: () => { + holding = false + gate.release() + }, + } + } + const observe = driver.observe + driver.observe = () => { + const actual = observe() + if (!holding) return actual + injected = true + return { + ...actual, + rowMetadata: { + ...actual.rowMetadata, + 1: { corrupt: `before application` }, + }, + } + } + return driver + }, + ), + ).rejects.toThrow(`held ${phase}`) + expect(injected).toBe(true) + }, +) + +it.each(recoveryPhases)( + `compares stale source reads when cleanup interrupts %s`, + async (phase) => { + let injected = false + await expect( + runRecoverySchedule( + { phase, interruption: `cleanup`, bare: true, value: 7 }, + async () => { + const driver = await createSQLiteDriver(persistedPeerOnDemandLane) + const stop = driver.stopSource + const observe = driver.observe + let ended = false + driver.stopSource = () => { + const pending = stop() + ended = true + return pending + } + driver.observe = () => { + const actual = observe() + if (!ended) return actual + injected = true + return { + ...actual, + rowMetadata: { + ...actual.rowMetadata, + 1: { corrupt: `stale recovery` }, + }, + } + } + return driver + }, + ), + ).rejects.toThrow(`cleanup started`) + expect(injected).toBe(true) + }, +) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts new file mode 100644 index 0000000000..d08f5cd40c --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts @@ -0,0 +1,31 @@ +import { fc } from '@fast-check/vitest' +import { expect, it } from 'vitest' +import { keptMetadataDurabilityHistoriesArbitrary } from '../../db/tests/sync-metadata-session-oracle' +import { requireExecutedHistory } from './sync-metadata-campaign-result' + +it(`rejects the obsolete shrink path that fast-check accepts without executing a history`, async () => { + for (const path of [`0`, `0:0:0`]) { + let calls = 0 + const result = await fc.check( + fc.asyncProperty(keptMetadataDurabilityHistoriesArbitrary, () => { + calls++ + return Promise.resolve() + }), + { + seed: 2220031, + path, + numRuns: 1, + }, + ) + expect(result.failed).toBe(false) + if (path === `0`) { + expect(calls).toBe(1) + expect(() => requireExecutedHistory(`calibration`, result)).not.toThrow() + } else { + expect(calls).toBe(0) + expect(() => requireExecutedHistory(`calibration`, result)).toThrow( + `executed no history`, + ) + } + } +}) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts new file mode 100644 index 0000000000..a9df4adc9a --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts @@ -0,0 +1,8 @@ +/** A syntactically valid shrink path can finish without invoking a property. */ +export function requireExecutedHistory( + property: string, + result: { numRuns: number }, +) { + if (result.numRuns === 0) + throw new Error(`oracle replay executed no history: ${property}`) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts new file mode 100644 index 0000000000..1ddcb64dfa --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts @@ -0,0 +1,80 @@ +import { expect, it } from 'vitest' +import { createRecoveryRecorder } from './sync-metadata-recovery-recorder' + +const input = () => ({ + type: `unknown-storage-observation` as const, + fromRowVersion: 0, + latestRowVersion: 2, +}) + +it(`withholds a response until storage reading and application both finish`, async () => { + const recorder = createRecoveryRecorder() + const response = input() + recorder.delivered(response) + expect(() => recorder.take()).toThrow(`was not applied`) + let release!: () => void + const gate = new Promise((resolve) => { + release = resolve + }) + const hydration = recorder.hydrate(async (read) => { + read() + await gate + }) + expect(() => recorder.take()).toThrow(`was not applied`) + release() + await hydration + expect(recorder.take()).toEqual([response]) + expect(recorder.take()).toEqual([]) +}) + +it(`rejects a completed task that never read storage`, async () => { + const recorder = createRecoveryRecorder() + recorder.delivered(input()) + await expect(recorder.hydrate(() => Promise.resolve())).rejects.toThrow( + `no storage observation`, + ) + expect(() => recorder.take()).toThrow(`was not applied`) +}) + +it(`preserves failed and later responses instead of crediting an unrelated task`, async () => { + const recorder = createRecoveryRecorder() + recorder.delivered(input()) + await expect( + recorder.hydrate((read) => { + read() + return Promise.reject(new Error(`application failed`)) + }), + ).rejects.toThrow(`application failed`) + expect(() => recorder.take()).toThrow(`was not applied`) + recorder.discard() + const later = input() + await recorder.hydrate((read) => { + read() + recorder.delivered(later) + return Promise.resolve() + }) + expect(() => recorder.take()).toThrow(`was not applied`) + await recorder.hydrate((read) => { + read() + return Promise.resolve() + }) + expect(recorder.take()).toEqual([later]) +}) + +it(`discards an invalidated sync run's response without a model input`, () => { + const recorder = createRecoveryRecorder() + recorder.delivered(input()) + recorder.discard() + expect(recorder.take()).toEqual([]) +}) + +it(`does not credit a hydration that finishes after cleanup`, async () => { + const recorder = createRecoveryRecorder() + recorder.delivered(input()) + await recorder.hydrate((read) => { + read() + recorder.discard() + return Promise.resolve() + }) + expect(recorder.take()).toEqual([]) +}) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts new file mode 100644 index 0000000000..95131c97e2 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts @@ -0,0 +1,42 @@ +import type { SessionRecoveryInput } from '../../db/tests/sync-metadata-session-oracle' + +/** + * Response delivery is transport evidence. Only a successful hydration task + * that read storage supplies a completed model input. Keep outstanding inputs + * visible so a skipped task cannot silently produce a green comparison. + */ +export function createRecoveryRecorder() { + const pending = new Set() + const completed: Array = [] + let generation = 0 + return { + pendingCount: () => pending.size, + delivered: (input: SessionRecoveryInput) => pending.add(input), + hydrate: async (task: (read: () => void) => Promise) => { + const inputs = [...pending] + const started = generation + let reads = 0 + const result = await task(() => reads++) + if (started !== generation) return result + if (inputs.length && reads === 0) + throw new Error(`recovery response completed no storage observation`) + for (const input of inputs) { + pending.delete(input) + completed.push(input) + } + return result + }, + take: () => { + if (pending.size) + throw new Error(`recovery response was not applied before comparison`) + return completed.splice(0) + }, + // Cleanup invalidates the sync run. Its late response supplies no model + // transition, even when the transport itself completed successfully. + discard: () => { + generation++ + pending.clear() + completed.length = 0 + }, + } +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts new file mode 100644 index 0000000000..5640bd461b --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts @@ -0,0 +1,70 @@ +// @vitest-environment node +import { fc } from '@fast-check/vitest' +import { it } from 'vitest' +import { + recoveryInterruptions, + recoveryPhases, + recoverySchedulesArbitrary, + runRecoverySchedule, +} from '../../db/tests/sync-metadata-recovery-oracle' +import { persistedPeerOnDemandLane } from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleRuns, + readOracleRunConfig, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { requireExecutedHistory } from './sync-metadata-campaign-result' +import { createSQLiteDriver } from './sync-metadata-session-sqlite-driver' + +const property = `sync-metadata-session.sqlite-recovery-schedules` +const requested = readOracleRunConfig().replayProperty +for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== property || campaign === `fixed`) + ) + continue + it(`preserves metadata across held recovery boundaries (${campaign})`, async () => { + const states = recoveryPhases.flatMap((phase) => + recoveryInterruptions.flatMap((interruption) => + [false, true].map((bare) => `${phase}/${interruption}/${bare}`), + ), + ) + const reach = createOracleReach(property, states) + const result = await fc.check( + fc.asyncProperty(recoverySchedulesArbitrary, async (batch) => { + for (const schedule of batch) { + reach.beginHistory() + const observation = await runRecoverySchedule(schedule, () => + createSQLiteDriver(persistedPeerOnDemandLane), + ) + if ( + !observation.events.includes(schedule.phase) || + observation.checkpoints < 7 + ) + throw new Error( + `recovery schedule did not compare its declared cut`, + ) + reach.reach( + `${schedule.phase}/${schedule.interruption}/${schedule.bare}`, + ) + } + }), + campaign === `fixed` + ? { numRuns: oracleRuns(8), seed: 2220035 } + : oraclePropertyOptions(8, property), + ) + recordOracleReplay(property, result) + requireExecutedHistory(property, result) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + ) + }, 120_000) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts index 51ad1817ea..7a87a0d09a 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts @@ -14,6 +14,7 @@ import { recordOracleReplay, } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' +import { requireExecutedHistory } from './sync-metadata-campaign-result' import type { SessionAction, SessionLane, @@ -86,8 +87,7 @@ export async function runSeedReinsertReloadCampaign({ : oraclePropertyOptions(40, property), ) recordOracleReplay(property, result) - if (result.numRuns === 0) - throw new Error(`oracle replay executed no history: ${property}`) + requireExecutedHistory(property, result) console.info( `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, shrinks: result.numShrinks, completedBranchGuards: campaign === `fixed` ? `enforced` : `measured`, counts: Object.fromEntries([...reach.counts(), ...completedReach.counts()]) })}`, ) @@ -183,8 +183,7 @@ export async function runKeptMetadataDurabilityCampaign({ : oraclePropertyOptions(8, property), ) recordOracleReplay(property, result) - if (result.numRuns === 0) - throw new Error(`oracle replay executed no history: ${property}`) + requireExecutedHistory(property, result) console.info( `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, ) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts new file mode 100644 index 0000000000..fff5c1ec87 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts @@ -0,0 +1,541 @@ +import { + DbClient, + collectionOptions, + createCollection, + whenSyncAccepted, +} from '@tanstack/db' +import { + absentMarker, + cachedMetadata, + cachedRow, + sessionCollectionKeys, +} from '../../db/tests/sync-metadata-session-oracle' +import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' +import type { CollectionConfig, SyncConfig } from '@tanstack/db' +import type { + PeerNotice, + ReceiptState, + SessionDriver, +} from '../../db/tests/sync-metadata-session-oracle' +import type { PersistedTx, PersistenceAdapter, ProtocolEnvelope } from '../src' + +/** + * The persisted wrapper is a second production path for the sync metadata + * session oracle. Its laws, model, grammar, and comparison are the oracle's + * own (`packages/db/tests/sync-metadata-session-oracle.ts`). The source and + * the wrapper are one session: a source read through the wrapper must give + * what the same read gives on a bare Collection. + * + * Production path: `persistedCollectionOptions` over an in-memory adapter. + * The source sync function receives the wrapper's sync API. A `hold` holds + * one kind of delay. A store hold makes the adapter's next durable write + * wait, so transactions committed after it stay deferred in the wrapper. A + * core hold is a persisting optimistic insert of a fresh fixture key, so + * transactions the wrapper has handed to core stay held there. `release` + * ends either. + * + * Checkpoints: source reads after every action; `collection.base` only at the + * final checkpoint, after every hold ends and the wrapper's continuations + * drain. The wrapper hands rows to core after its durable step, so consumers + * see them a turn or more later even with no hold. In the peer lanes the + * holding adapter's stored rows are compared at the final checkpoint too. + * + * Lanes: `persisted-flat`, `persisted-nested`, `persisted-seeds`, + * `persisted-cached` (row 1 is in storage when the run starts), + * `persisted-conflicts` (inner writes that land ahead of enclosing ones), and + * `persisted-nested-seeds`, which crosses seeds with nesting and the cached + * row, `persisted-reads`, whose read hold keeps an on-demand storage + * read in flight, `persisted-reads-seeds`, which adds seeds and the cached + * row and may abort a buffered commit's signal, and `persisted-peers`, where another tab's source writes + * storage and sends its notices through the persistence coordinator, with + * `persisted-peers-full` under full row updates and + * `persisted-peers-on-demand`, which loads on demand with a demand for every + * row active from the start. Seeds + * use a `DbClient` that owns the persisted Collection. Restart and resume + * reads are outside these lanes. + * + * A peer is a second writer the driver plays. It writes the stored row and + * metadata directly, takes the next stream position after this tab's + * durable commits, and emits the `tx:committed` notice that the Browser and + * Electron coordinators send: changed rows with their metadata, metadata + * alone, a deleted key, or a notice that requires a full reload. A reset + * empties storage, starts the next reset epoch, and sends + * `collection:reset`. The model + * makes a peer action legal only when no store hold or read hold is active, + * so every accepted transaction of this tab is durable and the notice is the + * next one in the stream. + */ + +type Row = { id: number; v: unknown } +type SyncActions = Parameters[`sync`]>[0] + +/** + * An in-memory adapter that can hold its next durable step. With `cached`, + * its reads supply row 1 from a previous run's storage until this run's + * first durable commit; the oracle observes one run. With `stores`, it keeps + * every durable row and returns them to reads, so a storage read during the + * run hands this tab's own rows back to the wrapper. + */ +function createHoldingAdapter(cached = false, stores = false) { + let holdNext = false + let releaseStore: (() => void) | undefined + let latest = { term: 0, seq: 0, rowVersion: 0 } + let resetEpoch = 0 + let cachedRows = cached + ? [{ key: 1, value: cachedRow, metadata: cachedMetadata }] + : [] + const stored = new Map( + stores && cached + ? [[1, { value: cachedRow as Row, metadata: cachedMetadata }]] + : [], + ) + const storedRows = () => + stores ? Array.from(stored, ([key, row]) => ({ key, ...row })) : cachedRows + // Capture every completed durable transaction before another queued write + // can replace it. The shared comparison permits identical empty commits. + const metadataSnapshots: Array> = [] + const recordMetadata = () => { + metadataSnapshots.push( + structuredClone( + Object.fromEntries( + [1, 2].map((key) => { + const value = stored.get(key)?.metadata + return [key, value === undefined ? absentMarker : value] + }), + ), + ), + ) + } + recordMetadata() + // A reload restores collection metadata from storage, so storage keeps it. + const storedCollectionMetadata = new Map() + const collectionMetadata = () => + Array.from(storedCollectionMetadata, ([key, value]) => ({ key, value })) + const adapter: PersistenceAdapter = { + loadSubset: () => Promise.resolve(storedRows()), + loadResumeSnapshot: () => + Promise.resolve({ + rows: storedRows(), + keySet: { status: `consistent` }, + collectionMetadata: collectionMetadata(), + latestTerm: latest.term, + latestSeq: latest.seq, + latestRowVersion: latest.rowVersion, + resetEpoch, + }), + loadCollectionMetadata: () => Promise.resolve(collectionMetadata()), + applyCommittedTx: async (_target, tx: PersistedTx) => { + cachedRows = [] + if (holdNext) { + holdNext = false + await new Promise((resolve) => { + releaseStore = resolve + }) + } + latest = { term: tx.term, seq: tx.seq, rowVersion: tx.rowVersion } + if (!stores) return + if (tx.truncate) stored.clear() + for (const mutation of tx.mutations) { + const key = mutation.key as number + if (mutation.type === `delete`) stored.delete(key) + else + stored.set(key, { + value: (mutation.type === `update` + ? { ...stored.get(key)?.value, ...mutation.value } + : mutation.value) as Row, + metadata: stored.get(key)?.metadata, + }) + } + for (const mutation of tx.rowMetadataMutations ?? []) { + const row = stored.get(mutation.key as number) + if (row) + row.metadata = mutation.type === `delete` ? undefined : mutation.value + } + for (const mutation of tx.collectionMetadataMutations ?? []) + if (mutation.type === `delete`) + storedCollectionMetadata.delete(mutation.key) + else storedCollectionMetadata.set(mutation.key, mutation.value) + recordMetadata() + }, + ensureIndex: () => Promise.resolve(), + } + return { + adapter, + /** A peer's durable write at the next stream position. */ + peerWrite: ( + key: number, + notice: PeerNotice, + row: Row, + metadata: unknown, + rewrites = false, + ): typeof latest => { + if (notice === `reset`) { + // A resetting tab that writes its rows back leaves storage as it was. + if (!rewrites) { + stored.clear() + storedCollectionMetadata.clear() + } + resetEpoch++ + latest = { term: latest.term + 1, seq: 0, rowVersion: 0 } + recordMetadata() + return latest + } + if (notice === `delete`) stored.delete(key) + else if (notice === `metadata`) stored.get(key)!.metadata = metadata + else stored.set(key, { value: row, metadata }) + latest = { + term: Math.max(latest.term, 1), + seq: latest.seq + 1, + rowVersion: latest.rowVersion + 1, + } + recordMetadata() + return latest + }, + resetEpoch: () => resetEpoch, + /** The row storage holds for a key, if any. */ + storedRow: (key: number): Row | undefined => stored.get(key)?.value, + storedRowMetadata: (key: number): unknown => stored.get(key)?.metadata, + storedMetadataTrace: () => metadataSnapshots, + holdNextStore: () => { + holdNext = true + }, + releaseStore: () => { + holdNext = false + releaseStore?.() + releaseStore = undefined + }, + } +} + +const drainTurns = async () => { + for (let turn = 0; turn < 20; turn++) + await new Promise((resolve) => setTimeout(resolve, 0)) +} + +let run = 0 + +/** + * Drives one persisted Collection through the shared session contract. Lane + * options select hydration seeds, cached rows, storage holds and peer notices; + * source reads and durable observations remain separate comparison inputs. + */ +export async function createPersistedDriver( + withClient = false, + cached = false, + reads = false, + peers = false, + fullRows = false, + onDemand = false, +): Promise { + const store = createHoldingAdapter(cached, reads || peers) + // The local coordinator persists this tab's commits through the adapter; + // `deliver` hands it a peer's notice. + let deliver: ((message: ProtocolEnvelope) => void) | undefined + const coordinator = peers + ? Object.assign(new SingleProcessCoordinator(), { + subscribe: ( + _collectionId: string, + onMessage: (message: ProtocolEnvelope) => void, + ) => { + deliver = onMessage + return () => { + deliver = undefined + } + }, + }) + : undefined + const id = `sync-metadata-session-persisted-${run++}` + let sync!: SyncActions + let resolveMutation: (() => void) | undefined + const client = withClient ? new DbClient() : undefined + const options = persistedCollectionOptions({ + id, + getKey: (row) => row.id, + // A read hold needs an on-demand load to keep in flight. + ...(reads || onDemand ? { syncMode: `on-demand` as const } : {}), + sync: { + ...(fullRows ? { rowUpdateMode: `full` as const } : {}), + sync: (actions) => { + sync = actions + actions.begin() + actions.commit() + actions.markReady() + return reads || onDemand ? { loadSubset: () => true } : undefined + }, + }, + onInsert: () => + new Promise((resolve) => { + resolveMutation = resolve + }), + persistence: { adapter: store.adapter, coordinator }, + } as CollectionConfig & { + persistence: { adapter: PersistenceAdapter } + }) + const collection = client + ? client.collection( + collectionOptions(options as CollectionConfig), + ) + : createCollection(options) + await collection.preload() + // Every row is demanded, so a reload or reset reads them all. + if (onDemand) await collection._sync.loadSubset({}) + await drainTurns() + + let depth = 0 + let holdCount = 0 + // Signals of the outermost transactions committed during a read hold. + let readCommits: Array = [] + let blocker: { when: (state: 'settled') => Promise } | undefined + const receipts: Array> = [] + const rejections: Array = [] + const receiptStates: Array = [] + const track = (receipt: true | Promise, expectAbort: boolean) => { + if (receipt === true) return + receipts.push( + receipt.then( + () => { + if (expectAbort) rejections.push(`cancelled transaction applied`) + }, + (error: unknown) => { + if ( + !expectAbort || + !(error instanceof Error) || + error.name !== `AbortError` + ) + rejections.push( + error instanceof Error + ? `${error.name}: ${error.message}` + : String(error), + ) + }, + ), + ) + } + const abort = () => { + const controller = new AbortController() + controller.abort() + track(sync.commit(controller.signal), true) + depth-- + } + const metadata = () => sync.metadata! + const read = (value: unknown) => (value === undefined ? absentMarker : value) + let readGate: { resolve: () => void } | undefined + let reading: Promise | undefined + let storageHeld = false + const accepted: Array> = [] + const release = async () => { + readGate?.resolve() + readGate = undefined + readCommits = [] + await reading + reading = undefined + store.releaseStore() + storageHeld = false + const pending = blocker + blocker = undefined + resolveMutation?.() + resolveMutation = undefined + await pending?.when('settled') + } + const asMetadata = (value: unknown) => + value === undefined ? {} : { metadata: value as Record } + + return { + consumerCheckpoint: `final`, + storedMetadataTrace: peers ? store.storedMetadataTrace : undefined, + begin: () => { + sync.begin() + depth++ + }, + commit: () => { + const index = receiptStates.length + const controller = new AbortController() + if (readGate && depth === 1) readCommits.push(controller) + const receipt = sync.commit(controller.signal) + accepted.push( + Promise.resolve(whenSyncAccepted(receipt)).catch(() => undefined), + ) + depth-- + if (receipt === true) { + receiptStates.push(`applied`) + return + } + receiptStates.push(`pending`) + receipts.push( + receipt.then( + () => { + receiptStates[index] = `applied` + }, + () => { + receiptStates[index] = `rejected` + }, + ), + ) + }, + cancel: abort, + abandon: () => readCommits.pop()!.abort(), + rowSet: (key, value) => metadata().row.set(key, value), + rowDelete: (key) => metadata().row.delete(key), + insert: (_key, value, rowMetadata) => + sync.write({ + type: `insert`, + value: value as Row, + ...asMetadata(rowMetadata), + }), + update: (_key, value, rowMetadata) => + sync.write({ + type: `update`, + value: value as Row, + ...asMetadata(rowMetadata), + }), + delete: (key) => sync.write({ type: `delete`, key }), + truncate: () => sync.truncate(), + collectionSet: (key, value) => metadata().collection.set(key, value), + collectionDelete: (key) => metadata().collection.delete(key), + hold: (holdStore, holdRead) => { + storageHeld = holdStore || holdRead === true + if (holdRead) { + let open!: () => void + const gate = new Promise((resolve) => { + open = resolve + }) + readGate = { resolve: open } + const loadSubset = store.adapter.loadSubset + store.adapter.loadSubset = async (...args) => { + await gate + store.adapter.loadSubset = loadSubset + return loadSubset(...args) + } + reading = Promise.resolve( + collection._sync.loadSubset({ limit: 100 + holdCount++ }), + ).catch(() => undefined) + } else if (holdStore) store.holdNextStore() + else blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) + }, + release, + seed: (key, row, rowMetadata) => + client!.hydrate({ + collections: [ + { + collectionId: id, + rows: [ + { + key, + value: row as Row, + metadata: rowMetadata as Record, + }, + ], + }, + ], + }), + peer: async (key, notice, row, metadata, rewrites) => { + const position = store.peerWrite( + key, + notice, + row as Row, + metadata, + rewrites, + ) + const named = + notice === `row` + ? { + changedRows: [{ key, value: row as Record }], + deletedKeys: [], + rowMetadataMutations: [ + { type: `set` as const, key, value: metadata }, + ], + } + : notice === `metadata` + ? { + changedRows: [], + deletedKeys: [], + rowMetadataMutations: [ + { type: `set` as const, key, value: metadata }, + ], + } + : { changedRows: [], deletedKeys: [key] } + deliver?.({ + v: 1, + dbName: `sync-metadata-session`, + collectionId: id, + senderId: `peer-tab`, + ts: Date.now(), + payload: + notice === `reset` + ? { + type: `collection:reset`, + schemaVersion: 2, + resetEpoch: store.resetEpoch(), + } + : { + type: `tx:committed`, + term: position.term, + seq: position.seq, + txId: `peer-${position.seq}`, + latestRowVersion: position.rowVersion, + ...(notice === `reload` + ? { requiresFullReload: true } + : { requiresFullReload: false, ...named }), + }, + }) + // The wrapper handles a notice under its apply lock and may reload + // storage first. A notice it never handled leaves the peer's write + // invisible, which the comparison reports, because every peer action + // the model allows changes a read except a full reload that a held + // seed keeps (decision 13). + await drainTurns() + }, + storedRows: () => ({ + 1: read(store.storedRow(1)), + 2: read(store.storedRow(2)), + }), + storedRowMetadata: () => ({ + 1: read(store.storedRowMetadata(1)), + 2: read(store.storedRowMetadata(2)), + }), + observe: () => ({ + rowMetadata: { + 1: read(metadata().row.get(1)), + 2: read(metadata().row.get(2)), + }, + collection: { + ...Object.fromEntries( + sessionCollectionKeys.map((key) => [ + key, + read(metadata().collection.get(key)), + ]), + ), + }, + list: sorted(metadata().collection.list()), + prefixList: sorted(metadata().collection.list(`gc:`)), + base: { + 1: read(collection.base.get(1)), + 2: read(collection.base.get(2)), + }, + receipts: [...receiptStates], + }), + flush: async () => { + // Acceptance follows durability even when a core hold delays visibility. + // A deliberately held store/read has no completed durable cut yet. + if (!storageHeld) await Promise.all(accepted) + await new Promise((resolve) => setTimeout(resolve, 0)) + }, + settle: async () => { + await release() + while (depth > 0) abort() + await drainTurns() + await Promise.all(receipts) + await drainTurns() + return rejections + }, + cleanup: () => collection.cleanup(), + } +} + +function sorted( + entries: ReadonlyArray<{ key: string; value: unknown }>, +): Array<[string, unknown]> { + return [...entries] + .map(({ key, value }): [string, unknown] => [key, value]) + .sort(([a], [b]) => a.localeCompare(b)) +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts index 9d440ac721..251b541146 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts @@ -1,15 +1,6 @@ import { fc, test as fcTest } from '@fast-check/vitest' import { describe, expect, it } from 'vitest' import { - DbClient, - collectionOptions, - createCollection, - whenSyncAccepted, -} from '@tanstack/db' -import { - absentMarker, - cachedMetadata, - cachedRow, describeMismatch, historyRegimes, persistedCachedLane, @@ -26,7 +17,6 @@ import { queuedMetadataRepairHistoriesArbitrary, runSessionHistory, seedReinsertReloadReplay, - sessionCollectionKeys, sessionHistoryArbitrary, } from '../../db/tests/sync-metadata-session-oracle' import { @@ -37,20 +27,16 @@ import { recordOracleReplay, } from '../../db/tests/oracle-config' import { createOracleReach } from '../../db/tests/oracle-reach' -import { SingleProcessCoordinator, persistedCollectionOptions } from '../src' +import { requireExecutedHistory } from './sync-metadata-campaign-result' +import { createPersistedDriver } from './sync-metadata-session-persisted-driver' import { runKeptMetadataDurabilityCampaign, runSeedReinsertReloadCampaign, } from './sync-metadata-session-campaign' -import type { CollectionConfig, SyncConfig } from '@tanstack/db' import type { - PeerNotice, - ReceiptState, SessionAction, - SessionDriver, SessionLane, } from '../../db/tests/sync-metadata-session-oracle' -import type { PersistedTx, PersistenceAdapter, ProtocolEnvelope } from '../src' /** * The persisted wrapper is a second production path for the sync metadata @@ -99,480 +85,6 @@ import type { PersistedTx, PersistenceAdapter, ProtocolEnvelope } from '../src' * next one in the stream. */ -type Row = { id: number; v: unknown } -type SyncActions = Parameters[`sync`]>[0] - -/** - * An in-memory adapter that can hold its next durable step. With `cached`, - * its reads supply row 1 from a previous run's storage until this run's - * first durable commit; the oracle observes one run. With `stores`, it keeps - * every durable row and returns them to reads, so a storage read during the - * run hands this tab's own rows back to the wrapper. - */ -function createHoldingAdapter(cached = false, stores = false) { - let holdNext = false - let releaseStore: (() => void) | undefined - let latest = { term: 0, seq: 0, rowVersion: 0 } - let resetEpoch = 0 - let cachedRows = cached - ? [{ key: 1, value: cachedRow, metadata: cachedMetadata }] - : [] - const stored = new Map( - stores && cached - ? [[1, { value: cachedRow as Row, metadata: cachedMetadata }]] - : [], - ) - const storedRows = () => - stores ? Array.from(stored, ([key, row]) => ({ key, ...row })) : cachedRows - // Capture every completed durable transaction before another queued write - // can replace it. The shared comparison permits identical empty commits. - const metadataSnapshots: Array> = [] - const recordMetadata = () => { - metadataSnapshots.push( - structuredClone( - Object.fromEntries( - [1, 2].map((key) => { - const value = stored.get(key)?.metadata - return [key, value === undefined ? absentMarker : value] - }), - ), - ), - ) - } - recordMetadata() - // A reload restores collection metadata from storage, so storage keeps it. - const storedCollectionMetadata = new Map() - const collectionMetadata = () => - Array.from(storedCollectionMetadata, ([key, value]) => ({ key, value })) - const adapter: PersistenceAdapter = { - loadSubset: () => Promise.resolve(storedRows()), - loadResumeSnapshot: () => - Promise.resolve({ - rows: storedRows(), - keySet: { status: `consistent` }, - collectionMetadata: collectionMetadata(), - latestTerm: latest.term, - latestSeq: latest.seq, - latestRowVersion: latest.rowVersion, - resetEpoch, - }), - loadCollectionMetadata: () => Promise.resolve(collectionMetadata()), - applyCommittedTx: async (_target, tx: PersistedTx) => { - cachedRows = [] - if (holdNext) { - holdNext = false - await new Promise((resolve) => { - releaseStore = resolve - }) - } - latest = { term: tx.term, seq: tx.seq, rowVersion: tx.rowVersion } - if (!stores) return - if (tx.truncate) stored.clear() - for (const mutation of tx.mutations) { - const key = mutation.key as number - if (mutation.type === `delete`) stored.delete(key) - else - stored.set(key, { - value: (mutation.type === `update` - ? { ...stored.get(key)?.value, ...mutation.value } - : mutation.value) as Row, - metadata: stored.get(key)?.metadata, - }) - } - for (const mutation of tx.rowMetadataMutations ?? []) { - const row = stored.get(mutation.key as number) - if (row) - row.metadata = mutation.type === `delete` ? undefined : mutation.value - } - for (const mutation of tx.collectionMetadataMutations ?? []) - if (mutation.type === `delete`) - storedCollectionMetadata.delete(mutation.key) - else storedCollectionMetadata.set(mutation.key, mutation.value) - recordMetadata() - }, - ensureIndex: () => Promise.resolve(), - } - return { - adapter, - /** A peer's durable write at the next stream position. */ - peerWrite: ( - key: number, - notice: PeerNotice, - row: Row, - metadata: unknown, - rewrites = false, - ): typeof latest => { - if (notice === `reset`) { - // A resetting tab that writes its rows back leaves storage as it was. - if (!rewrites) { - stored.clear() - storedCollectionMetadata.clear() - } - resetEpoch++ - latest = { term: latest.term + 1, seq: 0, rowVersion: 0 } - recordMetadata() - return latest - } - if (notice === `delete`) stored.delete(key) - else if (notice === `metadata`) stored.get(key)!.metadata = metadata - else stored.set(key, { value: row, metadata }) - latest = { - term: Math.max(latest.term, 1), - seq: latest.seq + 1, - rowVersion: latest.rowVersion + 1, - } - recordMetadata() - return latest - }, - resetEpoch: () => resetEpoch, - /** The row storage holds for a key, if any. */ - storedRow: (key: number): Row | undefined => stored.get(key)?.value, - storedRowMetadata: (key: number): unknown => stored.get(key)?.metadata, - storedMetadataTrace: () => metadataSnapshots, - holdNextStore: () => { - holdNext = true - }, - releaseStore: () => { - holdNext = false - releaseStore?.() - releaseStore = undefined - }, - } -} - -const drainTurns = async () => { - for (let turn = 0; turn < 20; turn++) - await new Promise((resolve) => setTimeout(resolve, 0)) -} - -let run = 0 - -/** - * Drives one persisted Collection through the shared session contract. Lane - * options select hydration seeds, cached rows, storage holds and peer notices; - * source reads and durable observations remain separate comparison inputs. - */ -async function createPersistedDriver( - withClient = false, - cached = false, - reads = false, - peers = false, - fullRows = false, - onDemand = false, -): Promise { - const store = createHoldingAdapter(cached, reads || peers) - // The local coordinator persists this tab's commits through the adapter; - // `deliver` hands it a peer's notice. - let deliver: ((message: ProtocolEnvelope) => void) | undefined - const coordinator = peers - ? Object.assign(new SingleProcessCoordinator(), { - subscribe: ( - _collectionId: string, - onMessage: (message: ProtocolEnvelope) => void, - ) => { - deliver = onMessage - return () => { - deliver = undefined - } - }, - }) - : undefined - const id = `sync-metadata-session-persisted-${run++}` - let sync!: SyncActions - let resolveMutation: (() => void) | undefined - const client = withClient ? new DbClient() : undefined - const options = persistedCollectionOptions({ - id, - getKey: (row) => row.id, - // A read hold needs an on-demand load to keep in flight. - ...(reads || onDemand ? { syncMode: `on-demand` as const } : {}), - sync: { - ...(fullRows ? { rowUpdateMode: `full` as const } : {}), - sync: (actions) => { - sync = actions - actions.begin() - actions.commit() - actions.markReady() - return reads || onDemand ? { loadSubset: () => true } : undefined - }, - }, - onInsert: () => - new Promise((resolve) => { - resolveMutation = resolve - }), - persistence: { adapter: store.adapter, coordinator }, - } as CollectionConfig & { - persistence: { adapter: PersistenceAdapter } - }) - const collection = client - ? client.collection( - collectionOptions(options as CollectionConfig), - ) - : createCollection(options) - await collection.preload() - // Every row is demanded, so a reload or reset reads them all. - if (onDemand) await collection._sync.loadSubset({}) - await drainTurns() - - let depth = 0 - let holdCount = 0 - // Signals of the outermost transactions committed during a read hold. - let readCommits: Array = [] - let blocker: { when: (state: 'settled') => Promise } | undefined - const receipts: Array> = [] - const rejections: Array = [] - const receiptStates: Array = [] - const track = (receipt: true | Promise, expectAbort: boolean) => { - if (receipt === true) return - receipts.push( - receipt.then( - () => { - if (expectAbort) rejections.push(`cancelled transaction applied`) - }, - (error: unknown) => { - if ( - !expectAbort || - !(error instanceof Error) || - error.name !== `AbortError` - ) - rejections.push( - error instanceof Error - ? `${error.name}: ${error.message}` - : String(error), - ) - }, - ), - ) - } - const abort = () => { - const controller = new AbortController() - controller.abort() - track(sync.commit(controller.signal), true) - depth-- - } - const metadata = () => sync.metadata! - const read = (value: unknown) => (value === undefined ? absentMarker : value) - let readGate: { resolve: () => void } | undefined - let reading: Promise | undefined - let storageHeld = false - const accepted: Array> = [] - const release = async () => { - readGate?.resolve() - readGate = undefined - readCommits = [] - await reading - reading = undefined - store.releaseStore() - storageHeld = false - const pending = blocker - blocker = undefined - resolveMutation?.() - resolveMutation = undefined - await pending?.when('settled') - } - const asMetadata = (value: unknown) => - value === undefined ? {} : { metadata: value as Record } - - return { - consumerCheckpoint: `final`, - storedMetadataTrace: peers ? store.storedMetadataTrace : undefined, - begin: () => { - sync.begin() - depth++ - }, - commit: () => { - const index = receiptStates.length - const controller = new AbortController() - if (readGate && depth === 1) readCommits.push(controller) - const receipt = sync.commit(controller.signal) - accepted.push( - Promise.resolve(whenSyncAccepted(receipt)).catch(() => undefined), - ) - depth-- - if (receipt === true) { - receiptStates.push(`applied`) - return - } - receiptStates.push(`pending`) - receipts.push( - receipt.then( - () => { - receiptStates[index] = `applied` - }, - () => { - receiptStates[index] = `rejected` - }, - ), - ) - }, - cancel: abort, - abandon: () => readCommits.pop()!.abort(), - rowSet: (key, value) => metadata().row.set(key, value), - rowDelete: (key) => metadata().row.delete(key), - insert: (_key, value, rowMetadata) => - sync.write({ - type: `insert`, - value: value as Row, - ...asMetadata(rowMetadata), - }), - update: (_key, value, rowMetadata) => - sync.write({ - type: `update`, - value: value as Row, - ...asMetadata(rowMetadata), - }), - delete: (key) => sync.write({ type: `delete`, key }), - truncate: () => sync.truncate(), - collectionSet: (key, value) => metadata().collection.set(key, value), - collectionDelete: (key) => metadata().collection.delete(key), - hold: (holdStore, holdRead) => { - storageHeld = holdStore || holdRead === true - if (holdRead) { - let open!: () => void - const gate = new Promise((resolve) => { - open = resolve - }) - readGate = { resolve: open } - const loadSubset = store.adapter.loadSubset - store.adapter.loadSubset = async (...args) => { - await gate - store.adapter.loadSubset = loadSubset - return loadSubset(...args) - } - reading = Promise.resolve( - collection._sync.loadSubset({ limit: 100 + holdCount++ }), - ).catch(() => undefined) - } else if (holdStore) store.holdNextStore() - else blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) - }, - release, - seed: (key, row, rowMetadata) => - client!.hydrate({ - collections: [ - { - collectionId: id, - rows: [ - { - key, - value: row as Row, - metadata: rowMetadata as Record, - }, - ], - }, - ], - }), - peer: async (key, notice, row, metadata, rewrites) => { - const position = store.peerWrite( - key, - notice, - row as Row, - metadata, - rewrites, - ) - const named = - notice === `row` - ? { - changedRows: [{ key, value: row as Record }], - deletedKeys: [], - rowMetadataMutations: [ - { type: `set` as const, key, value: metadata }, - ], - } - : notice === `metadata` - ? { - changedRows: [], - deletedKeys: [], - rowMetadataMutations: [ - { type: `set` as const, key, value: metadata }, - ], - } - : { changedRows: [], deletedKeys: [key] } - deliver?.({ - v: 1, - dbName: `sync-metadata-session`, - collectionId: id, - senderId: `peer-tab`, - ts: Date.now(), - payload: - notice === `reset` - ? { - type: `collection:reset`, - schemaVersion: 2, - resetEpoch: store.resetEpoch(), - } - : { - type: `tx:committed`, - term: position.term, - seq: position.seq, - txId: `peer-${position.seq}`, - latestRowVersion: position.rowVersion, - ...(notice === `reload` - ? { requiresFullReload: true } - : { requiresFullReload: false, ...named }), - }, - }) - // The wrapper handles a notice under its apply lock and may reload - // storage first. A notice it never handled leaves the peer's write - // invisible, which the comparison reports, because every peer action - // the model allows changes a read except a full reload that a held - // seed keeps (decision 13). - await drainTurns() - }, - storedRows: () => ({ - 1: read(store.storedRow(1)), - 2: read(store.storedRow(2)), - }), - storedRowMetadata: () => ({ - 1: read(store.storedRowMetadata(1)), - 2: read(store.storedRowMetadata(2)), - }), - observe: () => ({ - rowMetadata: { - 1: read(metadata().row.get(1)), - 2: read(metadata().row.get(2)), - }, - collection: { - ...Object.fromEntries( - sessionCollectionKeys.map((key) => [ - key, - read(metadata().collection.get(key)), - ]), - ), - }, - list: sorted(metadata().collection.list()), - prefixList: sorted(metadata().collection.list(`gc:`)), - base: { - 1: read(collection.base.get(1)), - 2: read(collection.base.get(2)), - }, - receipts: [...receiptStates], - }), - flush: async () => { - // Acceptance follows durability even when a core hold delays visibility. - // A deliberately held store/read has no completed durable cut yet. - if (!storageHeld) await Promise.all(accepted) - await new Promise((resolve) => setTimeout(resolve, 0)) - }, - settle: async () => { - await release() - while (depth > 0) abort() - await drainTurns() - await Promise.all(receipts) - await drainTurns() - return rejections - }, - cleanup: () => collection.cleanup(), - } -} - -function sorted( - entries: ReadonlyArray<{ key: string; value: unknown }>, -): Array<[string, unknown]> { - return [...entries] - .map(({ key, value }): [string, unknown] => [key, value]) - .sort(([a], [b]) => a.localeCompare(b)) -} - async function assertPersistedHistory( history: ReadonlyArray, lane: SessionLane = persistedFlatLane, @@ -1437,10 +949,7 @@ for (const campaign of [`fixed`, `random`] as const) { console.info( `ORACLE_REACH ${JSON.stringify({ property: queuedRepairProperty, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, ) - if (result.numRuns === 0) - throw new Error( - `oracle replay executed no history: ${queuedRepairProperty}`, - ) + requireExecutedHistory(queuedRepairProperty, result) if (result.failed) throw new Error( `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts new file mode 100644 index 0000000000..85ece2e9ab --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts @@ -0,0 +1,540 @@ +// @vitest-environment node +/** + * The sync metadata session oracle's seed/re-insert/reload law crosses real + * SQLite and coordinator boundaries here. The law, model, legal histories, + * and comparison are imported from sync-metadata-session-oracle.ts. This + * driver supplies two persisted Collections, two production coordinators over + * Node's BroadcastChannel/Web Locks, and one node:sqlite database. Each + * Collection has its own adapter; their shared physical database supplies the + * shared scheduler premise. + * + * A peer's named notice comes from its real source commit. For a full reload, + * that source takes a durable snapshot, changes the requested row, and commits + * a snapshot transaction. This preserves every other durable row and causes + * production to announce requiresFullReload. The receiver records the actual + * notice and waits for SQL/message quiescence before the shared comparison. + * Neither this driver nor the peer invents an expected metadata value. + * + * Limits: one process, one SQLite database, two Collections of one ID, no + * transport loss or leadership turnover, and core holds only. This is a real + * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. + */ +import { DatabaseSync } from 'node:sqlite' +import { DbClient, collectionOptions, whenSyncAccepted } from '@tanstack/db' +import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' +import { + absentMarker, + cachedMetadata, + cachedRow, + sessionCollectionKeys, +} from '../../db/tests/sync-metadata-session-oracle' +import { + SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, + SQLiteCorePersistenceAdapter, + eagerStorageTarget, + managedStorageTarget, + persistedCollectionOptions, +} from '../src' +import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' +import { createRecoveryRecorder } from './sync-metadata-recovery-recorder' +import type { + RecoveryDriver, + RecoveryPhase, +} from '../../db/tests/sync-metadata-recovery-oracle' +import type { SyncConfig } from '@tanstack/db' +import type { + ReceiptState, + SessionLane, +} from '../../db/tests/sync-metadata-session-oracle' + +type Row = { id: number; v: unknown } +type Actions = Parameters[`sync`]>[0] +let run = 0 + +/** + * Supplies the shared model with two SQLite-backed Collections and real + * coordinator traffic. Each authored action drains its correlated recovery + * before comparison; returned recovery inputs contain no expected metadata. + */ +export async function createSQLiteDriver( + lane: SessionLane, +): Promise { + const database = new DatabaseSync(`:memory:`) + const { driver, drained } = createMetadataSQLiteDriver(database) + Object.defineProperty(driver, SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, { + value: {}, + }) + const adapters = [ + new SQLiteCorePersistenceAdapter({ driver }), + new SQLiteCorePersistenceAdapter({ driver }), + ] + const id = `metadata-items` + const seedClaim = lane.onDemand + ? await adapters[0]!.claimCacheGeneration(id) + : undefined + const target = seedClaim + ? managedStorageTarget(seedClaim.storageCollectionId, seedClaim.claimId) + : eagerStorageTarget(id) + const storageId = seedClaim?.storageCollectionId ?? id + await adapters[0]!.applyCommittedTx(target, { + txId: `cached`, + term: 1, + seq: 1, + rowVersion: 1, + cacheGenerationClaimId: seedClaim?.claimId, + mutations: [{ type: `insert`, key: 1, value: cachedRow }], + rowMetadataMutations: [{ type: `set`, key: 1, value: cachedMetadata }], + }) + const dbName = `sync-metadata-receiving-${process.pid}-${run++}` + const coordinators = adapters.map( + (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), + ) + // Observe the transport's replay information, not the wrapper's decision or a + // metadata mismatch. Without replay details this response cannot distinguish + // earlier cached writes from this run's writes; D15 governs that information + // gap. The model derives its effect solely from its own durable storage. + const recorder = createRecoveryRecorder() + let heldRecovery: + | { + phase: RecoveryPhase + entered: () => void + gate: Promise + release: () => void + } + | undefined + const recoveryEvents: Array = [] + const boundary = async (phase: RecoveryPhase) => { + recoveryEvents.push(phase) + if (heldRecovery?.phase === phase) { + heldRecovery.entered() + await heldRecovery.gate + } + } + const pullSince = coordinators[0]!.pullSince!.bind(coordinators[0]) + coordinators[0]!.pullSince = async (...args) => { + const response = await pullSince(...args) + if (response.ok && (response.requiresFullReload || !response.deltas)) { + recorder.delivered({ + type: `unknown-storage-observation`, + fromRowVersion: args[1], + latestRowVersion: response.latestRowVersion, + }) + await boundary(`response`) + } + return response + } + const hydrate = adapters[0]!.runInHydrationScope.bind(adapters[0]) + adapters[0]!.runInHydrationScope = (task) => + hydrate(async (adapter) => + recorder.hydrate(async (read) => { + const recovery = recorder.pendingCount() > 0 + return task({ + ...adapter, + loadSubset: async (...args) => { + if (recovery) await boundary(`read`) + const rows = await adapter.loadSubset(...args) + read() + // Hold the captured rows before the wrapper receives and applies + // them, while the SQLite hydration scope still owns admission. + if (recovery) await boundary(`application`) + return rows + }, + }) + }), + ) + const pendingEnsures = new Set>() + for (const coordinator of coordinators) { + const request = coordinator.requestEnsureRemoteSubset.bind(coordinator) + coordinator.requestEnsureRemoteSubset = (...args) => { + const pending = request(...args) + pendingEnsures.add(pending) + const done = () => pendingEnsures.delete(pending) + void pending.then(done, done) + return pending + } + } + let lastPeerTxId: string | undefined + const sourceTxIds: Array = [] + const sourceApply = coordinators[0]!.requestApplyCommittedTx.bind( + coordinators[0], + ) + coordinators[0]!.requestApplyCommittedTx = (...args) => { + sourceTxIds.push(args[1].txId) + return sourceApply(...args) + } + let peerSubmitted: (() => void) | undefined + const peerApply = coordinators[1]!.requestApplyCommittedTx.bind( + coordinators[1], + ) + coordinators[1]!.requestApplyCommittedTx = (...args) => { + lastPeerTxId = args[1].txId + peerSubmitted?.() + peerSubmitted = undefined + return peerApply(...args) + } + const notices: Array<{ reload: boolean; txId: string }> = [] + const noticeWaiters = new Set<() => void>() + const actions: Array = [] + let durableRows: ReadonlyArray<{ + key: string | number + value: unknown + metadata?: unknown + }> = [] + const stopNotices = coordinators[0]!.subscribe(storageId, (message) => { + const payload = message.payload as { + type: string + txId: string + requiresFullReload?: boolean + } + if (payload.type === `tx:committed`) { + notices.push({ + reload: payload.requiresFullReload === true, + txId: payload.txId, + }) + for (const wake of noticeWaiters) wake() + } + }) + let sourceStopped = false + const quiesce = async () => { + // Each empty source commit enters the wrapper after already delivered + // notices. Its acceptance drains that wrapper's apply lock without + // waiting for visibility behind a core hold. No write or notice results. + // These empty commits stutter in the model, including under two open + // authored transactions; maxDepth bounds authored histories only. + do { + // An on-demand cached load may settle before its upstream ensure. Wait + // for those actual coordinator replies before draining their receiver. + await Promise.all([...pendingEnsures]) + for (const [index, source] of actions.entries()) { + if (sourceStopped && index === 0) continue + source.begin() + try { + await whenSyncAccepted(source.commit()) + } catch (error) { + if (!( + sourceStopped && + index === 0 && + error instanceof Error && + error.name === `AbortError` + )) + throw error + } + } + await adapters[0]!.runInRegularScope.afterQueuedWork() + await drained() + } while (pendingEnsures.size > 0) + durableRows = await adapters[0]!.loadSubset(target, {}) + } + const awaitNotice = (after: number, reload: boolean, txId: string) => + new Promise((resolve, reject) => { + const timer = setTimeout(() => { + noticeWaiters.delete(check) + reject( + new Error(`the real coordinator did not deliver the expected notice`), + ) + }, 5_000) + const check = () => { + if ( + !notices + .slice(after) + .some((entry) => entry.reload === reload && entry.txId === txId) + ) + return + clearTimeout(timer) + noticeWaiters.delete(check) + resolve() + } + noticeWaiters.add(check) + check() + }) + let resolveMutation: (() => void) | undefined + const clients = [new DbClient(), new DbClient()] + const collections = clients.map((client, index) => + client.collection( + collectionOptions( + persistedCollectionOptions({ + id, + getKey: (row) => row.id, + ...(lane.onDemand ? { syncMode: `on-demand` as const } : {}), + sync: { + rowUpdateMode: lane.fullRows ? `full` : `partial`, + sync: (sync) => { + actions[index] = sync + sync.begin() + sync.commit() + sync.markReady() + return lane.onDemand ? { loadSubset: () => true } : undefined + }, + }, + onInsert: () => + new Promise((resolve) => { + resolveMutation = resolve + }), + persistence: { + adapter: adapters[index]!, + coordinator: coordinators[index]!, + }, + }), + ), + ), + ) + // These direct source histories have no live query to start on-demand sync. + // Start both sources before a follower can route its first demand to its peer. + for (const current of collections) current.startSyncImmediate() + if (!lane.onDemand) + await Promise.all(collections.map((collection) => collection.preload())) + const demands = collections.map(() => ({})) + if (lane.onDemand) + await Promise.all( + collections.map((collection, index) => + collection._sync.loadSubset(demands[index]!), + ), + ) + await quiesce() + const sync = actions[0]! + const metadata = sync.metadata! + const collection = collections[0]! + const receipts: Array> = [] + const receiptStates: Array = [] + const rejections: Array = [] + let depth = 0 + let holdCount = 0 + let blocker: { when: (state: 'settled') => Promise } | undefined + const release = async () => { + resolveMutation?.() + resolveMutation = undefined + await blocker?.when(`settled`) + blocker = undefined + } + const abort = () => { + const controller = new AbortController() + controller.abort() + const receipt = sync.commit(controller.signal) + depth-- + if (receipt !== true) + receipts.push( + receipt.catch((error: unknown) => { + if (!(error instanceof Error) || error.name !== `AbortError`) + rejections.push(String(error)) + }), + ) + } + const read = (value: unknown) => (value === undefined ? absentMarker : value) + const sorted = (entries: ReadonlyArray<{ key: string; value: unknown }>) => + entries + .map(({ key, value }): [string, unknown] => [key, value]) + .sort(([a], [b]) => a.localeCompare(b)) + const rowMetadata = (value: unknown) => + value === undefined ? {} : { metadata: value as Record } + return { + holdRecovery: (phase) => { + if (heldRecovery) throw new Error(`a recovery hold is already armed`) + recoveryEvents.length = 0 + let entered!: () => void + const reached = new Promise((resolve) => { + entered = resolve + }) + let releaseRecovery!: () => void + const gate = new Promise((resolve) => { + releaseRecovery = resolve + }) + heldRecovery = { phase, entered, gate, release: releaseRecovery } + return { + reached, + release: () => { + releaseRecovery() + heldRecovery = undefined + }, + } + }, + recoveryEvents: () => [...recoveryEvents], + startPeerWrite: (value) => { + const submitted = new Promise((resolve) => { + peerSubmitted = resolve + }) + const peer = actions[1]! + peer.begin() + peer.write({ + type: `insert`, + value: { id: 2, v: value }, + metadata: { peer: value }, + }) + return { + submitted, + applied: Promise.resolve(whenSyncAccepted(peer.commit())).then( + () => {}, + ), + } + }, + stopSource: async () => { + while (depth) abort() + sourceStopped = true + const stopped = collection.cleanup() + recorder.discard() + await stopped + }, + consumerCheckpoint: `final`, + begin: () => { + sync.begin() + depth++ + }, + commit: async () => { + const noticeStart = notices.length + const transactionStart = sourceTxIds.length + const index = receiptStates.length + const receipt = sync.commit() + depth-- + receiptStates.push(receipt === true ? `applied` : `pending`) + if (receipt !== true) + receipts.push( + receipt.then( + () => { + receiptStates[index] = `applied` + }, + () => { + receiptStates[index] = `rejected` + }, + ), + ) + // This bounded legal schedule completes the source's correlated notice + // and any resulting recovery before the next authored action. In + // particular, a peer write cannot enter the model ahead of a recovery + // that actually read the pre-peer durable state. Core holds may keep the + // applied receipt pending, so wait for acceptance rather than visibility. + await whenSyncAccepted(receipt) + for (const txId of sourceTxIds.slice(transactionStart)) + await awaitNotice(noticeStart, false, txId) + await quiesce() + }, + cancel: abort, + rowSet: (key, value) => metadata.row.set(key, value), + rowDelete: (key) => metadata.row.delete(key), + insert: (_key, value, meta) => + sync.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }), + update: (_key, value, meta) => + sync.write({ type: `update`, value: value as Row, ...rowMetadata(meta) }), + delete: (key) => sync.write({ type: `delete`, key }), + truncate: () => sync.truncate(), + collectionSet: (key, value) => metadata.collection.set(key, value), + collectionDelete: (key) => metadata.collection.delete(key), + hold: (store, readHold) => { + if (store || readHold) + throw new Error(`the SQLite seed/re-insert grammar uses core holds`) + blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) + }, + release, + seed: (key, value, meta) => + clients[0]!.hydrate({ + collections: [ + { + collectionId: id, + rows: [ + { + key, + value: value as Row, + metadata: meta as Record, + }, + ], + }, + ], + }), + peer: async (key, notice, value, meta) => { + await quiesce() + const before = notices.length + const peer = actions[1]! + if (notice === `reload`) { + const rows = await adapters[1]!.loadSubset(target, {}) + peer.begin({ snapshot: true }) + for (const row of rows) + if (row.key !== key) + peer.write({ + type: `insert`, + value: row.value as Row, + ...rowMetadata(row.metadata), + }) + } else if (notice === `row`) peer.begin() + else + throw new Error( + `unexpected peer notice in the seed/re-insert grammar: ${notice}`, + ) + peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) + await peer.commit() + if (!lastPeerTxId) + throw new Error(`the peer issued no durable transaction`) + await awaitNotice(before, notice === `reload`, lastPeerTxId) + await quiesce() + if ( + !notices + .slice(before) + .some((entry) => entry.reload === (notice === `reload`)) + ) + throw new Error( + `the real coordinator did not deliver the ${notice} notice`, + ) + console.info( + `SQLITE_METADATA_NOTICE ${JSON.stringify({ lane: lane.name, notice, received: notices.length - before })}`, + ) + }, + observe: () => ({ + rowMetadata: { + 1: read(metadata.row.get(1)), + 2: read(metadata.row.get(2)), + }, + collection: Object.fromEntries( + sessionCollectionKeys.map((key) => [ + key, + read(metadata.collection.get(key)), + ]), + ), + list: sorted(metadata.collection.list()), + prefixList: sorted(metadata.collection.list(`gc:`)), + base: { + 1: read(collection.base.get(1)), + 2: read(collection.base.get(2)), + }, + receipts: [...receiptStates], + }), + storedRows: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.value), + ]), + ), + storedRowMetadata: () => + Object.fromEntries( + [1, 2].map((key) => [ + key, + read(durableRows.find((row) => row.key === key)?.metadata), + ]), + ), + takeRecoveryInputs: recorder.take, + flush: quiesce, + settle: async () => { + await release() + while (depth) abort() + await quiesce() + await Promise.all(receipts) + await quiesce() + return rejections + }, + cleanup: async () => { + heldRecovery?.release() + heldRecovery = undefined + await release() + while (depth) abort() + await Promise.all(receipts) + if (lane.onDemand) + for (const [index, current] of collections.entries()) + current._sync.unloadSubset(demands[index]!) + await quiesce() + await Promise.all(collections.map((current) => current.cleanup())) + stopNotices() + for (const coordinator of coordinators) coordinator.dispose() + if (seedClaim) + await adapters[0]!.releaseCacheGenerationClaim(seedClaim.claimId) + await drained() + database.close() + }, + } +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts index e9c7ed6e7c..9cc53b0c75 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts @@ -1,480 +1,27 @@ // @vitest-environment node -/** - * The sync metadata session oracle's seed/re-insert/reload law crosses real - * SQLite and coordinator boundaries here. The law, model, legal histories, - * and comparison are imported from sync-metadata-session-oracle.ts. This - * driver supplies two persisted Collections, two production coordinators over - * Node's BroadcastChannel/Web Locks, and one node:sqlite database. Each - * Collection has its own adapter; their shared physical database supplies the - * shared scheduler premise. - * - * A peer's named notice comes from its real source commit. For a full reload, - * that source takes a durable snapshot, changes the requested row, and commits - * a snapshot transaction. This preserves every other durable row and causes - * production to announce requiresFullReload. The receiver records the actual - * notice and waits for SQL/message quiescence before the shared comparison. - * Neither this driver nor the peer invents an expected metadata value. - * - * Limits: one process, one SQLite database, two Collections of one ID, no - * transport loss or leadership turnover, and core holds only. This is a real - * SQLite/coordinator receiving witness, not a Chromium OPFS or Electron claim. - */ -import { DatabaseSync } from 'node:sqlite' import { it } from 'vitest' -import { DbClient, collectionOptions, whenSyncAccepted } from '@tanstack/db' -import { BrowserCollectionCoordinator } from '../../browser-db-sqlite-persistence/src/browser-coordinator' import { - absentMarker, - cachedMetadata, - cachedRow, describeMismatch, persistedPeerFullLane, persistedPeerLane, persistedPeerOnDemandLane, runSessionHistory, seedReinsertReloadReplay, - sessionCollectionKeys, } from '../../db/tests/sync-metadata-session-oracle' import { oracleReplayTest, readOracleRunConfig, } from '../../db/tests/oracle-config' -import { - SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, - SQLiteCorePersistenceAdapter, - eagerStorageTarget, - managedStorageTarget, - persistedCollectionOptions, -} from '../src' -import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' import { runKeptMetadataDurabilityCampaign, runSeedReinsertReloadCampaign, } from './sync-metadata-session-campaign' -import type { SyncConfig } from '@tanstack/db' +import { createSQLiteDriver } from './sync-metadata-session-sqlite-driver' import type { - ReceiptState, SessionAction, - SessionDriver, SessionLane, - SessionRecoveryInput, } from '../../db/tests/sync-metadata-session-oracle' -type Row = { id: number; v: unknown } -type Actions = Parameters[`sync`]>[0] -let run = 0 - -/** - * Supplies the shared model with two SQLite-backed Collections and real - * coordinator traffic. Each authored action drains its correlated recovery - * before comparison; returned recovery inputs contain no expected metadata. - */ -async function createSQLiteDriver(lane: SessionLane): Promise { - const database = new DatabaseSync(`:memory:`) - const { driver, drained } = createMetadataSQLiteDriver(database) - Object.defineProperty(driver, SQLITE_DRIVER_SHARED_LOGICAL_SCHEDULING_KEY, { - value: {}, - }) - const adapters = [ - new SQLiteCorePersistenceAdapter({ driver }), - new SQLiteCorePersistenceAdapter({ driver }), - ] - const id = `metadata-items` - const seedClaim = lane.onDemand - ? await adapters[0]!.claimCacheGeneration(id) - : undefined - const target = seedClaim - ? managedStorageTarget(seedClaim.storageCollectionId, seedClaim.claimId) - : eagerStorageTarget(id) - const storageId = seedClaim?.storageCollectionId ?? id - await adapters[0]!.applyCommittedTx(target, { - txId: `cached`, - term: 1, - seq: 1, - rowVersion: 1, - cacheGenerationClaimId: seedClaim?.claimId, - mutations: [{ type: `insert`, key: 1, value: cachedRow }], - rowMetadataMutations: [{ type: `set`, key: 1, value: cachedMetadata }], - }) - const dbName = `sync-metadata-receiving-${process.pid}-${run++}` - const coordinators = adapters.map( - (adapter) => new BrowserCollectionCoordinator({ dbName, adapter }), - ) - // Observe the transport's replay information, not the wrapper's decision or a - // metadata mismatch. Without replay details this response cannot distinguish - // earlier cached writes from this run's writes; D15 governs that information - // gap. The model derives its effect solely from its own durable storage. - const recoveryInputs: Array = [] - const pendingRecoveryInputs: Array = [] - const pullSince = coordinators[0]!.pullSince!.bind(coordinators[0]) - coordinators[0]!.pullSince = async (...args) => { - const response = await pullSince(...args) - if (response.ok && (response.requiresFullReload || !response.deltas)) - pendingRecoveryInputs.push({ - type: `unknown-storage-observation`, - fromRowVersion: args[1], - latestRowVersion: response.latestRowVersion, - }) - return response - } - const hydrate = adapters[0]!.runInHydrationScope.bind(adapters[0]) - adapters[0]!.runInHydrationScope = (task) => - hydrate(async (adapter) => { - const inputs = pendingRecoveryInputs.splice(0) - const observation = { reads: 0 } - const result = await task({ - ...adapter, - loadSubset: async (...args) => { - const rows = await adapter.loadSubset(...args) - observation.reads++ - return rows - }, - }) - if (inputs.length && observation.reads === 0) - throw new Error(`recovery response completed no storage observation`) - // The hydration task includes application. Merely receiving a response - // that cleanup or claim recovery discards must not advance the model. - recoveryInputs.push(...inputs) - return result - }) - const pendingEnsures = new Set>() - for (const coordinator of coordinators) { - const request = coordinator.requestEnsureRemoteSubset.bind(coordinator) - coordinator.requestEnsureRemoteSubset = (...args) => { - const pending = request(...args) - pendingEnsures.add(pending) - const done = () => pendingEnsures.delete(pending) - void pending.then(done, done) - return pending - } - } - let lastPeerTxId: string | undefined - const sourceTxIds: Array = [] - const sourceApply = coordinators[0]!.requestApplyCommittedTx.bind( - coordinators[0], - ) - coordinators[0]!.requestApplyCommittedTx = (...args) => { - sourceTxIds.push(args[1].txId) - return sourceApply(...args) - } - const peerApply = coordinators[1]!.requestApplyCommittedTx.bind( - coordinators[1], - ) - coordinators[1]!.requestApplyCommittedTx = (...args) => { - lastPeerTxId = args[1].txId - return peerApply(...args) - } - const notices: Array<{ reload: boolean; txId: string }> = [] - const noticeWaiters = new Set<() => void>() - const actions: Array = [] - let durableRows: ReadonlyArray<{ - key: string | number - value: unknown - metadata?: unknown - }> = [] - const stopNotices = coordinators[0]!.subscribe(storageId, (message) => { - const payload = message.payload as { - type: string - txId: string - requiresFullReload?: boolean - } - if (payload.type === `tx:committed`) { - notices.push({ - reload: payload.requiresFullReload === true, - txId: payload.txId, - }) - for (const wake of noticeWaiters) wake() - } - }) - const quiesce = async () => { - // Each empty source commit enters the wrapper after already delivered - // notices. Its acceptance drains that wrapper's apply lock without - // waiting for visibility behind a core hold. No write or notice results. - // These empty commits stutter in the model, including under two open - // authored transactions; maxDepth bounds authored histories only. - do { - // An on-demand cached load may settle before its upstream ensure. Wait - // for those actual coordinator replies before draining their receiver. - await Promise.all([...pendingEnsures]) - for (const source of actions) { - source.begin() - await whenSyncAccepted(source.commit()) - } - await adapters[0]!.runInRegularScope.afterQueuedWork() - await drained() - } while (pendingEnsures.size > 0) - durableRows = await adapters[0]!.loadSubset(target, {}) - } - const awaitNotice = (after: number, reload: boolean, txId: string) => - new Promise((resolve, reject) => { - const timer = setTimeout(() => { - noticeWaiters.delete(check) - reject( - new Error(`the real coordinator did not deliver the expected notice`), - ) - }, 5_000) - const check = () => { - if ( - !notices - .slice(after) - .some((entry) => entry.reload === reload && entry.txId === txId) - ) - return - clearTimeout(timer) - noticeWaiters.delete(check) - resolve() - } - noticeWaiters.add(check) - check() - }) - let resolveMutation: (() => void) | undefined - const clients = [new DbClient(), new DbClient()] - const collections = clients.map((client, index) => - client.collection( - collectionOptions( - persistedCollectionOptions({ - id, - getKey: (row) => row.id, - ...(lane.onDemand ? { syncMode: `on-demand` as const } : {}), - sync: { - rowUpdateMode: lane.fullRows ? `full` : `partial`, - sync: (sync) => { - actions[index] = sync - sync.begin() - sync.commit() - sync.markReady() - return lane.onDemand ? { loadSubset: () => true } : undefined - }, - }, - onInsert: () => - new Promise((resolve) => { - resolveMutation = resolve - }), - persistence: { - adapter: adapters[index]!, - coordinator: coordinators[index]!, - }, - }), - ), - ), - ) - // These direct source histories have no live query to start on-demand sync. - // Start both sources before a follower can route its first demand to its peer. - for (const current of collections) current.startSyncImmediate() - if (!lane.onDemand) - await Promise.all(collections.map((collection) => collection.preload())) - const demands = collections.map(() => ({})) - if (lane.onDemand) - await Promise.all( - collections.map((collection, index) => - collection._sync.loadSubset(demands[index]!), - ), - ) - await quiesce() - const sync = actions[0]! - const metadata = sync.metadata! - const collection = collections[0]! - const receipts: Array> = [] - const receiptStates: Array = [] - const rejections: Array = [] - let depth = 0 - let holdCount = 0 - let blocker: { when: (state: 'settled') => Promise } | undefined - const release = async () => { - resolveMutation?.() - resolveMutation = undefined - await blocker?.when(`settled`) - blocker = undefined - } - const abort = () => { - const controller = new AbortController() - controller.abort() - const receipt = sync.commit(controller.signal) - depth-- - if (receipt !== true) - receipts.push( - receipt.catch((error: unknown) => { - if (!(error instanceof Error) || error.name !== `AbortError`) - rejections.push(String(error)) - }), - ) - } - const read = (value: unknown) => (value === undefined ? absentMarker : value) - const sorted = (entries: ReadonlyArray<{ key: string; value: unknown }>) => - entries - .map(({ key, value }): [string, unknown] => [key, value]) - .sort(([a], [b]) => a.localeCompare(b)) - const rowMetadata = (value: unknown) => - value === undefined ? {} : { metadata: value as Record } - return { - consumerCheckpoint: `final`, - begin: () => { - sync.begin() - depth++ - }, - commit: async () => { - const noticeStart = notices.length - const transactionStart = sourceTxIds.length - const index = receiptStates.length - const receipt = sync.commit() - depth-- - receiptStates.push(receipt === true ? `applied` : `pending`) - if (receipt !== true) - receipts.push( - receipt.then( - () => { - receiptStates[index] = `applied` - }, - () => { - receiptStates[index] = `rejected` - }, - ), - ) - // This bounded legal schedule completes the source's correlated notice - // and any resulting recovery before the next authored action. In - // particular, a peer write cannot enter the model ahead of a recovery - // that actually read the pre-peer durable state. Core holds may keep the - // applied receipt pending, so wait for acceptance rather than visibility. - await whenSyncAccepted(receipt) - for (const txId of sourceTxIds.slice(transactionStart)) - await awaitNotice(noticeStart, false, txId) - await quiesce() - }, - cancel: abort, - rowSet: (key, value) => metadata.row.set(key, value), - rowDelete: (key) => metadata.row.delete(key), - insert: (_key, value, meta) => - sync.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }), - update: (_key, value, meta) => - sync.write({ type: `update`, value: value as Row, ...rowMetadata(meta) }), - delete: (key) => sync.write({ type: `delete`, key }), - truncate: () => sync.truncate(), - collectionSet: (key, value) => metadata.collection.set(key, value), - collectionDelete: (key) => metadata.collection.delete(key), - hold: (store, readHold) => { - if (store || readHold) - throw new Error(`the SQLite seed/re-insert grammar uses core holds`) - blocker = collection.insert({ id: 1000 + holdCount++, v: 0 }) - }, - release, - seed: (key, value, meta) => - clients[0]!.hydrate({ - collections: [ - { - collectionId: id, - rows: [ - { - key, - value: value as Row, - metadata: meta as Record, - }, - ], - }, - ], - }), - peer: async (key, notice, value, meta) => { - await quiesce() - const before = notices.length - const peer = actions[1]! - if (notice === `reload`) { - const rows = await adapters[1]!.loadSubset(target, {}) - peer.begin({ snapshot: true }) - for (const row of rows) - if (row.key !== key) - peer.write({ - type: `insert`, - value: row.value as Row, - ...rowMetadata(row.metadata), - }) - } else if (notice === `row`) peer.begin() - else - throw new Error( - `unexpected peer notice in the seed/re-insert grammar: ${notice}`, - ) - peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) - await peer.commit() - if (!lastPeerTxId) - throw new Error(`the peer issued no durable transaction`) - await awaitNotice(before, notice === `reload`, lastPeerTxId) - await quiesce() - if ( - !notices - .slice(before) - .some((entry) => entry.reload === (notice === `reload`)) - ) - throw new Error( - `the real coordinator did not deliver the ${notice} notice`, - ) - console.info( - `SQLITE_METADATA_NOTICE ${JSON.stringify({ lane: lane.name, notice, received: notices.length - before })}`, - ) - }, - observe: () => ({ - rowMetadata: { - 1: read(metadata.row.get(1)), - 2: read(metadata.row.get(2)), - }, - collection: Object.fromEntries( - sessionCollectionKeys.map((key) => [ - key, - read(metadata.collection.get(key)), - ]), - ), - list: sorted(metadata.collection.list()), - prefixList: sorted(metadata.collection.list(`gc:`)), - base: { - 1: read(collection.base.get(1)), - 2: read(collection.base.get(2)), - }, - receipts: [...receiptStates], - }), - storedRows: () => - Object.fromEntries( - [1, 2].map((key) => [ - key, - read(durableRows.find((row) => row.key === key)?.value), - ]), - ), - storedRowMetadata: () => - Object.fromEntries( - [1, 2].map((key) => [ - key, - read(durableRows.find((row) => row.key === key)?.metadata), - ]), - ), - takeRecoveryInputs: () => { - if (pendingRecoveryInputs.length) - throw new Error(`recovery response was not applied before comparison`) - return recoveryInputs.splice(0) - }, - flush: quiesce, - settle: async () => { - await release() - while (depth) abort() - await quiesce() - await Promise.all(receipts) - await quiesce() - return rejections - }, - cleanup: async () => { - await release() - while (depth) abort() - await Promise.all(receipts) - if (lane.onDemand) - for (const [index, current] of collections.entries()) - current._sync.unloadSubset(demands[index]!) - await quiesce() - await Promise.all(collections.map((current) => current.cleanup())) - stopNotices() - for (const coordinator of coordinators) coordinator.dispose() - if (seedClaim) - await adapters[0]!.releaseCacheGenerationClaim(seedClaim.claimId) - await drained() - database.close() - }, - } -} - const lanes = [ persistedPeerLane, persistedPeerFullLane, diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts index 4d8013cf20..c1da16082c 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts @@ -1,78 +1,24 @@ // @vitest-environment node import { DatabaseSync } from 'node:sqlite' -import { expect, it } from 'vitest' +import { describe } from 'vitest' +import { registerSQLiteFixtureConformance } from './sqlite-fixture-conformance' import { createMetadataSQLiteDriver } from './sync-metadata-sqlite-driver' -it.each([false, true])( - `rolls back only a failed nested scope (root reentry: %s)`, - async (rootReentry) => { - const database = new DatabaseSync(`:memory:`) - const { driver, drained } = createMetadataSQLiteDriver(database) - try { - await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) - await driver.transaction(async (scoped) => { - await scoped.run(`INSERT INTO rows VALUES (1)`) - const nested = rootReentry ? driver : scoped - await expect( - nested.transaction(async (inner) => { - await inner.run(`INSERT INTO rows VALUES (2)`) - throw new Error(`rollback the inner scope`) - }), - ).rejects.toThrow(`rollback the inner scope`) - await nested.transaction(async (inner) => { - await inner.run(`INSERT INTO rows VALUES (3)`) - }) - // Root calls in this async transaction must use its scope, rather than - // queue behind the callback that is awaiting them. - await (rootReentry ? driver : scoped).run(`INSERT INTO rows VALUES (4)`) +describe.each([`transaction`, `transactionWithDriver`] as const)( + `SQLite fixture admission through %s`, + (entry) => { + registerSQLiteFixtureConformance(() => { + const database = new DatabaseSync(`:memory:`) + const { driver, drained } = createMetadataSQLiteDriver(database) + if (entry === `transactionWithDriver`) + driver.transaction = driver.transactionWithDriver! + return Promise.resolve({ + driver, + cleanup: async () => { + await drained() + database.close() + }, }) - expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ - { id: 1 }, - { id: 3 }, - { id: 4 }, - ]) - } finally { - await drained() - database.close() - } + }) }, ) - -it(`keeps another adapter's root reads and writes outside a rolled-back transaction`, async () => { - const database = new DatabaseSync(`:memory:`) - const { driver, drained } = createMetadataSQLiteDriver(database) - let entered!: () => void - const started = new Promise((resolve) => { - entered = resolve - }) - let release!: () => void - const gate = new Promise((resolve) => { - release = resolve - }) - try { - await driver.exec(`CREATE TABLE rows (id INTEGER PRIMARY KEY)`) - const failed = driver.transaction(async (scoped) => { - await scoped.run(`INSERT INTO rows VALUES (1)`) - entered() - await gate - throw new Error(`rollback this transaction only`) - }) - const rejected = expect(failed).rejects.toThrow( - `rollback this transaction only`, - ) - await started - const read = driver.query(`SELECT id FROM rows ORDER BY id`) - const write = driver.run(`INSERT INTO rows VALUES (2)`) - release() - await rejected - expect(await read).toEqual([]) - await write - expect(await driver.query(`SELECT id FROM rows ORDER BY id`)).toEqual([ - { id: 2 }, - ]) - } finally { - release() - await drained() - database.close() - } -}) diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts index db178aa732..dc769398da 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts @@ -8,10 +8,11 @@ import type { SQLiteDriver } from '../src' * cannot observe it before commit or become part of its rollback. * Nested scopes use savepoints. Root calls in the transaction's async context * reenter its scope, rather than queue behind the callback that awaits them. + * A detached callback loses that privilege when its owning transaction ends. */ export function createMetadataSQLiteDriver(database: DatabaseSync) { let queue: Promise = Promise.resolve() - const context = new AsyncLocalStorage() + const context = new AsyncLocalStorage<{ active: boolean }>() let nextSavepoint = 0 const enqueue = (work: () => Promise): Promise => { const next = queue.then(work) @@ -42,28 +43,33 @@ export function createMetadataSQLiteDriver(database: DatabaseSync) { }, } const transaction: SQLiteDriver[`transaction`] = (work) => - context.getStore() + context.getStore()?.active ? scoped.transaction(work) : enqueue(async () => { database.exec(`BEGIN IMMEDIATE`) + const scope = { active: true } try { - const result = await context.run(true, () => work(scoped)) + const result = await context.run(scope, () => work(scoped)) database.exec(`COMMIT`) return result } catch (error) { database.exec(`ROLLBACK`) throw error + } finally { + scope.active = false } }) const driver: SQLiteDriver = { exec: (sql) => - context.getStore() ? scoped.exec(sql) : enqueue(() => scoped.exec(sql)), + context.getStore()?.active + ? scoped.exec(sql) + : enqueue(() => scoped.exec(sql)), query: (sql: string, params?: ReadonlyArray) => - context.getStore() + context.getStore()?.active ? scoped.query(sql, params) : enqueue(() => scoped.query(sql, params)), run: (sql, params) => - context.getStore() + context.getStore()?.active ? scoped.run(sql, params) : enqueue(() => scoped.run(sql, params)), transaction, diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index ec02c36817..e3ff127211 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -291,6 +291,7 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-kept-metadata-durability`, `sync-metadata-session.persisted-queued-metadata-repair`, `sync-metadata-session.sqlite-kept-metadata-durability`, + `sync-metadata-session.sqlite-recovery-schedules`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, `d2-source.exact-retractions`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 7aa107040f..05d7ba06ee 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -736,6 +736,11 @@ const ownerGroups: ReadonlyArray< `sync-metadata-session`, `sqlite-seed-reinsert-reload sqlite-kept-metadata-durability`, ], + [ + `db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts`, + `sync-metadata-session`, + `sqlite-recovery-schedules`, + ], ] const owners = new Map() diff --git a/packages/db/tests/sync-metadata-recovery-oracle.ts b/packages/db/tests/sync-metadata-recovery-oracle.ts new file mode 100644 index 0000000000..6995c0feb3 --- /dev/null +++ b/packages/db/tests/sync-metadata-recovery-oracle.ts @@ -0,0 +1,263 @@ +import { fc } from '@fast-check/vitest' +import { declareOracle } from './oracle-declaration.js' +import { + compareSource, + createSessionModel, + persistedPeerOnDemandLane, + planAction, +} from './sync-metadata-session-oracle.js' +import type { + SessionAction, + SessionDriver, + SessionMismatch, +} from './sync-metadata-session-oracle.js' + +/** + * Recovery has three separate boundaries: a replay response is delivered, + * storage is read, and the captured rows are supplied for application. + * D15 governs completed storage observations; D9/D18 keep an implicit keep + * distinct from a durable write. Cleanup invalidates the sync run (glossary). + * + * This grammar adds bounded schedules to the session oracle, using its same + * independent model, action planning and source comparison. It introduces no + * second metadata policy. A peer write can enter storage before the response + * is released. During a SQLite hydration scope it instead waits for that + * scope; the fixture's admission conformance supplies this premise. + * + * Limits: one recovery, two rows, on-demand SQLite, a seed, peer write or + * cleanup at one held boundary. No expiry, generation rotation, restart, + * leadership turnover, Chromium OPFS or Electron IPC claim follows. + */ +export const recoveryPhases = [`response`, `read`, `application`] as const +export type RecoveryPhase = (typeof recoveryPhases)[number] +export const recoveryInterruptions = [`seed`, `peer`, `cleanup`] as const +export type RecoverySchedule = { + phase: RecoveryPhase + interruption: (typeof recoveryInterruptions)[number] + bare: boolean + value: number +} + +// Each generated batch includes all 18 schedules. Payloads and the initial +// seed's metadata vary without making the timing premise probabilistic. +export const recoverySchedulesArbitrary = fc + .integer({ min: -100, max: 100 }) + .map((value): Array => + [false, true].flatMap((bare) => + recoveryPhases.flatMap((phase) => + recoveryInterruptions.map((interruption) => ({ + phase, + interruption, + bare, + value, + })), + ), + ), + ) + +export type RecoveryDriver = SessionDriver & { + holdRecovery: (phase: RecoveryPhase) => { + reached: Promise + release: () => void + } + recoveryEvents: () => ReadonlyArray + startPeerWrite: (value: unknown) => { + submitted: Promise + applied: Promise + } + stopSource: () => Promise +} + +/** + * The held cuts compare source reads before application and after the + * interruption. The completed cut uses D15 against model-owned storage. + * Cleanup compares stale source reads against an empty sync run. Durable + * metadata is checked independently, including when cleanup discards a pull. + */ +export async function runRecoverySchedule( + schedule: RecoverySchedule, + createDriver: () => Promise, +) { + const lane = persistedPeerOnDemandLane + const model = createSessionModel(lane) + const driver = await createDriver() + let checkpoints = 0 + const checkSource = (cut: string, ended = false) => { + const expected = ended + ? createSessionModel({ ...lane, cached: false }).observe() + : model.observe() + const mismatch = compareSource( + checkpoints++, + `settle`, + expected, + driver.observe(), + ) + if (mismatch) throw new Error(`${cut}: ${JSON.stringify(mismatch)}`) + } + const history: Array = [ + { type: `seed`, key: 1, bare: schedule.bare }, + { type: `begin` }, + { type: `insertOverSeed`, key: 1, metadata: false }, + { type: `begin` }, + { type: `reinsert`, key: 1, metadata: false }, + ] + let release: (() => void) | undefined + let commit: Promise | undefined + let peer: ReturnType | undefined + let stopped: Promise | undefined + let primary: unknown + let outcome: + | { + checkpoints: number + events: ReadonlyArray + inputs: number + } + | undefined + const expectedAbandonment = (error: unknown) => + schedule.interruption === `cleanup` && + error instanceof Error && + error.name === `AbortError` + try { + if (driver.takeRecoveryInputs?.().length) + throw new Error(`recovery schedule unexpectedly started during setup`) + for (const [step, action] of history.entries()) { + await planAction(model, action, { s: step })(driver) + await driver.flush() + checkSource(`setup action ${step}`) + } + const held = driver.holdRecovery(schedule.phase) + release = held.release + // Model the authored commit once, independently of how long delivery takes. + commit = Promise.resolve( + planAction(model, { type: `commit` }, { s: 5 })(driver), + ) + let timeout!: ReturnType + try { + await Promise.race([ + held.reached, + commit.then(() => { + throw new Error(`commit completed without reaching ${schedule.phase}`) + }), + new Promise((_resolve, reject) => { + timeout = setTimeout( + () => + reject( + new Error( + `recovery never reached ${schedule.phase} within 5 seconds`, + ), + ), + 5_000, + ) + }), + ]) + } finally { + clearTimeout(timeout) + } + const expectedEvents = recoveryPhases.slice( + 0, + recoveryPhases.indexOf(schedule.phase) + 1, + ) + if ( + JSON.stringify(driver.recoveryEvents()) !== JSON.stringify(expectedEvents) + ) + throw new Error( + `the recovery hold did not preserve its response/read/application prefix`, + ) + checkSource(`held ${schedule.phase}`) + if (schedule.interruption === `seed`) { + await planAction( + model, + { type: `seed`, key: 1, bare: !schedule.bare }, + { interleaved: schedule.value }, + )(driver) + checkSource(`seed during ${schedule.phase}`) + } else if (schedule.interruption === `peer`) { + peer = driver.startPeerWrite(schedule.value) + await peer.submitted + if (schedule.phase === `response`) { + await peer.applied + model.stagePeerWrite(2, `row`, schedule.value) + } + checkSource(`peer submitted during ${schedule.phase}`) + } else { + stopped = driver.stopSource() + checkSource(`cleanup started during ${schedule.phase}`, true) + } + release() + release = undefined + try { + await commit + } catch (error) { + if (!expectedAbandonment(error)) throw error + } + if (stopped) await stopped + if (schedule.interruption !== `cleanup`) { + model.observeUnknownStorage() + if (peer) { + await peer.applied + model.peer(2, `row`, schedule.value) + } + await driver.flush() + } + checkSource( + `completed ${schedule.phase}`, + schedule.interruption === `cleanup`, + ) + const inputs = driver.takeRecoveryInputs?.() ?? [] + if (schedule.interruption !== `cleanup` && inputs.length === 0) + throw new Error(`completed recovery supplied no storage input`) + if (schedule.interruption === `cleanup` && inputs.length !== 0) + throw new Error(`an invalidated sync run supplied a recovery input`) + const expected = model.storedRowMetadata() + const actual = driver.storedRowMetadata?.() + if (JSON.stringify(expected) !== JSON.stringify(actual)) { + const mismatch: SessionMismatch = { + step: 6, + action: `settle`, + law: `recovery and cleanup do not write metadata`, + expected, + actual, + } + throw new Error(JSON.stringify(mismatch)) + } + outcome = { + checkpoints, + events: driver.recoveryEvents(), + inputs: inputs.length, + } + } catch (error) { + primary = new Error(`${JSON.stringify(schedule)}: ${String(error)}`, { + cause: error, + }) + } + // Drain before closing the shared connection. Keep secondary diagnostics + // distinguishable from the first violated law and observation cut. + const failures: Array = [] + try { + release?.() + } catch (error) { + failures.push(error) + } + const pending = await Promise.allSettled([commit, peer?.applied, stopped]) + for (const result of pending) + if (result.status === `rejected` && !expectedAbandonment(result.reason)) + failures.push(result.reason) + try { + await driver.cleanup() + } catch (error) { + failures.push(error) + } + if (failures.length) + throw new AggregateError( + primary === undefined ? failures : [primary, ...failures], + `recovery schedule cleanup failed: ${[primary, ...failures].map(String).join(`; `)}`, + { cause: primary }, + ) + if (primary !== undefined) throw primary + return outcome! +} + +declareOracle({ + grammar: [recoverySchedulesArbitrary], + check: [runRecoverySchedule], +}) diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index a3f1a27cc0..16f44b59bb 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -98,9 +98,12 @@ import { declareOracle } from './oracle-declaration.js' * only after their hydration task completes a storage read and application. * It completes each source commit's correlated notice and * recovery before the next authored action, then compares the same model's - * D15 storage observation. Arbitrary response/application interleavings, - * recovery during cleanup, and the necessity of gap detection are outside - * this bounded schedule. + * D15 storage observation. The companion `sync-metadata-recovery-oracle.ts` + * uses this same model and source comparison for held response, storage-read + * and captured-row application cuts, crossed with seeds, peer writes and + * cleanup. Arbitrary overlapping recoveries, cache claim expiry, persisted + * cache generation rotation and the necessity of gap detection remain outside + * these bounded schedules. */ // --------------------------------------------------------------------------- @@ -1421,19 +1424,7 @@ export class SessionModel { this.reset(rewrites) return } - // The peer's write reaches storage before its notice. - const previous = this.storage.get(key) ?? { - row: absent, - metadata: absent, - } - this.storage.set( - key, - notice === `metadata` - ? { row: previous.row, metadata } - : notice === `delete` - ? { row: absent, metadata: absent } - : { row, metadata }, - ) + this.stagePeerWrite(key, notice, value, rowValue) // After a reset, a notice in the reset's term may predate it, so this // tab reloads storage instead (the reset fence); the peer keeps one term. if (notice !== `reload` && !this.fenced) { @@ -1455,6 +1446,34 @@ export class SessionModel { this.observeUnknownStorage() } + /** + * Model projection of a peer's durable write before its notice is delivered. + * Separating these events permits a recovery read between them. The usual + * peer action above still performs both in one authored step. + */ + stagePeerWrite( + key: SessionRowKey, + notice: Exclude, + value: unknown, + rowValue: unknown = value, + ): void { + const row: ModelValue = { present: true, value: { id: key, v: rowValue } } + const metadata: ModelValue = { present: true, value: { peer: value } } + // The peer's write reaches storage before its notice. + const previous = this.storage.get(key) ?? { + row: absent, + metadata: absent, + } + this.storage.set( + key, + notice === `metadata` + ? { row: previous.row, metadata } + : notice === `delete` + ? { row: absent, metadata: absent } + : { row, metadata }, + ) + } + /** * D15: a storage observation without transaction authorship can establish * differing durable objects as source writes. A recovery response with no @@ -2570,7 +2589,7 @@ type DriverCall = (driver: SessionDriver) => void | Promise * Advances the model by one legal action and returns the matching driver * call. The model decides once; the call can be replayed on several drivers. */ -function planAction( +export function planAction( model: SessionModel, action: SessionAction, value: unknown, @@ -2793,9 +2812,9 @@ function canonical(value: unknown): string { ) } -function compareSource( +export function compareSource( step: number, - action: SessionAction, + action: SessionAction | `settle`, expected: SessionObservation, actual: SessionObservation, ): SessionMismatch | undefined { From 88641f0847975ed920fc44eb48e48f5a602c5206 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 13:25:17 -0700 Subject: [PATCH 09/11] Bind metadata calibration evidence to tested commit --- .../2026-11-02-p6-instrument-calibration.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md b/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md index f6491ac56a..2cbddc4e64 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-instrument-calibration.md @@ -251,3 +251,12 @@ The four requested improvements are implemented. The additional finding is from this pass. The known receiving and observation limits above remain open; they are not removed from the contract by this accounting. The earlier ten-item review retains its independent `10 = 7 fixed-now + 3 refuted` accounting. + + +### Commit binding + +Executable instrument revision: `60ace06b044e76fc9e57f2a6942a3c73593a2388`. +All 18 TypeScript hashes above match after commit hooks. This binding appendix +changes documentation only. The 279-test result, seven intended mutation +rejections and direct replay result describe that executable revision. +Final publication-head CI and third-party review remain separate obligations. From 12558a9d380461eca6fc6b824788a3cf19e404bc Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 14:14:01 -0700 Subject: [PATCH 10/11] Preserve run-only metadata across named peer reloads --- .changeset/retain-run-only-row-metadata.md | 5 + docs/contributing/oracle-coverage.md | 22 ++ .../2026-11-02-p6-ci-missing-row.md | 236 ++++++++++++++++++ .../src/persisted.ts | 39 +-- .../tests/sync-metadata-calibration.test.ts | 25 ++ .../tests/sync-metadata-missing-row.test.ts | 109 ++++++++ .../sync-metadata-session-sqlite-driver.ts | 31 ++- packages/db/tests/oracle-config.ts | 2 + packages/db/tests/oracle-replay-manifest.ts | 5 + .../db/tests/sync-metadata-session-oracle.ts | 115 ++++++++- 10 files changed, 562 insertions(+), 27 deletions(-) create mode 100644 .changeset/retain-run-only-row-metadata.md create mode 100644 docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md create mode 100644 packages/db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts diff --git a/.changeset/retain-run-only-row-metadata.md b/.changeset/retain-run-only-row-metadata.md new file mode 100644 index 0000000000..312cb1c84b --- /dev/null +++ b/.changeset/retain-run-only-row-metadata.md @@ -0,0 +1,5 @@ +--- +'@tanstack/db-sqlite-persistence-core': patch +--- + +Retain run-only row metadata when an unrelated peer notice reloads a row recreated by an enclosing sync transaction. Keep that metadata nondurable unless the source explicitly writes it, while preserving peer replacement and delete behavior. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d71a7f2554..6a1b79bfeb 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -4913,6 +4913,28 @@ do not retire that obligation. Cross-host recovery schedules still need actual Chromium OPFS and Electron IPC receiving witnesses; their owner remains this session oracle with the broader receiving work tracked by #2272. +Repaired counterexample, **META-CI-051-S001**: an enclosing update reads a +cached row, an inner transaction deletes that row and sets its metadata while +missing, then both commit. A named peer notice for another key drops the +retained metadata in the on-demand receiver, with and without a core hold. +Both the controlled adapter and native SQLite reproduce this source-read +failure under D16 before the production repair. The model correction separates +this run-only metadata from durable metadata when the outer update recreates the +row; the old model's false durable expectation masks the later source failure. +Owner: this session oracle. The generated +`sync-metadata-session.{persisted,sqlite}-missing-row-metadata` campaigns +cover twelve cases in all three peer lanes: implicit/explicit metadata, +immediate/core-held application, a prior source truncate, same-key peer row or +metadata replacement, unknown-write reloads, peer/source delete and source +truncate. All 36 case/lane states reach eight times in each fixed/random campaign +on both hosts. The wrapper distinguishes local metadata from a named peer +replacement, preserving D16 without defeating D11 or D15. Three wrong retention +policies fail the same source-read comparisons. The +[CI investigation](oracle-reviews/2026-11-02-p6-ci-missing-row.md) +records the pre-edit receiving RED, repaired GREEN, controls and bounded class +claim. This does not determine that the separate #2317/P8 witness has the same +cause or close its broader scope. + `persisted-reads-seeds` crosses read holds with seeds and the cached row and aborts a buffered commit's signal, which abandons it at the read's end (decision 21); a seed that arrived after it stays judged against it diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md b/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md new file mode 100644 index 0000000000..4f5112102b --- /dev/null +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md @@ -0,0 +1,236 @@ +# P6 publication CI: missing-row metadata + +This repair preserves source metadata that exists only in one sync run when a +nested commit deletes its row and an enclosing keeping update recreates it. +An unrelated named peer notice must not replace that value with absent stored +metadata. Same-key peer replacement, unknown-write observations, source deletes +and source truncates retain their accepted meanings. + +Parent publication: `88641f0847975ed920fc44eb48e48f5a602c5206`. +The source identities below bind the tested repair. This is a self-review, +not independent review. The user authorized the production repair after the +preliminary investigation, and retained the merge-when-green instruction. + +## Findings and repair locations + +**META-CI-051 — fixed reference model.** SQLite core CI job `125530540619` +failed `sync-metadata-session.persisted-peers` at seed `1581505626`, path +`18:3:0:3:9:9:12:11:11:7:7:8:8:8:8:8:8`. The original replay fails its durable +metadata comparison at commit step 13. Source metadata still agrees. + +The model retained metadata for a missing row in its storage entry, then +incorrectly made it durable when a later keeping update recreated the row. +`SessionModel.store` now preserves prior metadata only if its prior durable +row existed. Its session events still retain the source-only value. Under D16, +source read M and durable absence are both required. A later explicit metadata +write instead makes M durable. These predictions come from the accepted law, +not production output. The original replay passes after this correction. + +**META-CI-051-S001 — fixed production retention within the boundary below.** +The corrected model exposes a nearby source-read defect on controlled and +native receivers. Starting with cached row 1: + +```text +begin + update row 1 without metadata + begin + delete row 1 + set row 1 metadata M + commit +commit +peer writes row 2 and delivers its named notice +``` + +The inner delete precedes the metadata set. The inner commit has no durable +row 1, so M remains nondurable. The outer update recreates only the row. +This sync run must still read M, and storage must still lack metadata. +An on-demand reload instead replaced M with the absent durable value. + +The wrapper previously tracked only whether metadata had a source writer. +It now tracks the latest local or named peer writer. A known storage read +retains local metadata, including nondurable values. A named same-key peer +write replaces it. Unknown peer writes still use D15 and per-key knowledge. +A local source truncate does not turn later local metadata writes into unknown +peer writes. No public API, protocol, law or stored format changes. + +## Authority and class boundary + +Authority: decisions 9, 11, 15, 16 and 18 in the +[accepted session record](2026-10-22-sync-metadata-session.md). +D16 keeps missing-row metadata in the run until a source deletes the row. +D9/D18 distinguish keeping from writing. D11 admits named current-run peer +replacements. D15 judges differing storage observations after unknown writes. + +The declared class is D16/D9/D18 retention after independent nested commit +order, across partial/full/on-demand persisted receiving paths, observed at +completed source commits, core-held application cuts and completed peer +notices. Source reads, consumer rows, receipts, durable metadata and final +stored rows use the shared session model and comparisons. This bounded claim +includes the adjacent replacements and clears below. It is not universal proof +for every metadata history or transport. + +## Grammar, receiving paths and pre-edit gate + +`missingRowMetadataHistoriesArbitrary` constructs twelve cases per batch: + +- Immediate and core-held application, each with keeping and explicit writes. +- The keeping history after a source truncate, immediate and core-held. +- Same-key peer row and metadata replacement. +- An unknown-write reload. +- Peer delete, source delete and source truncate. + +Each case executes across `persisted-peers`, `persisted-peers-full` and +`persisted-peers-on-demand`: 36 states per host. Both controlled storage and +native SQLite use the same model, histories, recorder contract and comparison. +Each batch varies one to three missing-row metadata writes. Every authored +action must execute. A skipped action fails the campaign. + +The native receiver supplies actual SQLite rows with absent metadata, two real +Browser coordinators, and Node BroadcastChannel/Web Locks in one process. +Metadata-only and delete peer actions now use real source commits. Source +truncate commits correlate their actual reload notices. The pre-edit inventory +required this receiver as well as the controlled adapter. + +`repair-pre-edit-native-red.log` records the final gate on unchanged production: +all four fixed/random campaigns fail at the intended source-read comparison. +The four on-demand implicit states fail, including the source-truncate prefix. +The other 32 states pass in the reached batches. No setup, timeout or +legal-action error supplies this RED. The ledger records the satisfied gate +before the production edit. + +Earlier attempts remain in scratch evidence without RED credit. One had an +unregistered replay name, one accidentally passed a non-arbitrary argument, +and one exposed unsupported native metadata/delete/truncate driver paths. +The final pre-edit run supersedes those setup failures. The old model also +fails earlier at the intended durable comparison; it cannot explain the later +source-read defect away. + +## Executed results + +Commands use Node 24.19.0, pnpm 11.1.0, Vitest 4.1.12, and +`--maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +--disableConsoleIntercept`. Logs live under +`/private/tmp/tanstack-db-p6-2026-11-02/`. + +| Evidence | Result | +| --- | --- | +| `ci-51-replay-red.log` | Original CI seed/path fails durable metadata on the parent model | +| `repair-pre-edit-native-red.log` | Unchanged production fails all four receiving campaigns at the intended source read | +| `repair-missing-row-green.log` | Four repaired campaigns pass; every one of 36 states reaches eight times per host/campaign | +| `repair-ci-replay-green.log` | Guarded original CI replay passes 39 tests, exact seed/path, one nonzero selected execution | +| `repair-new-replay-green.log` | Guarded native seed 2220037/path 0 passes directly; one selected execution, all 36 states reached | +| `repair-core-green.log` + `repair-registry-green.log` | 162 core/registry tests pass initially; literal-owner registration guard then passes after correcting the wrapper's property spelling | +| `repair-helper-green.log` | 20 recorder, replay guard, native SQL conformance and deterministic work tests pass | +| `repair-primary-green.log` | 1,256 passed, 49 existing expected failures, one todo across the five requested owner/receiving/control files | +| `repair-types.log`, `repair-eslint-final.log`, `repair-format.log`, `repair-lint-changed.log`, `repair-budgets.log` | Source-mapped types, scoped ESLint (seven changed TypeScript files, zero errors/warnings), formatting, changed-file oracle lint and unchanged run budgets pass | + +The new campaigns use identical fixed/random budgets of eight, with fixed seed +2220037. The normal random seeds were -861738368 (controlled) and -836725301 (native). +No existing budget, guard, comparison or test is removed or weakened. + +## Hostile controls and replay + +An isolated tree starts from the parent and overlays the candidate sources. +Its unmodified fixed baseline passes both hosts with eight batches each. +Three production mutations fail assertions at the shared source-read comparison: + +| Wrong design | Distinguishing case | +| --- | --- | +| Restore the original written-key retention rule | On-demand immediate/core-held keeping, including after local truncate | +| Treat a named peer writer as local | Same-key peer row replacement | +| Always retain local metadata despite unknown writes | Unknown-write reload | + +All three fail on both receiving hosts. These are assertion failures, not +setup errors, runtime exceptions or timeouts. The original-policy mutant +replays native seed 2220037/path 0 with the same source-read mismatch; restoring +the repair makes that replay GREEN. Scratch `repair-mutants/results.json` +records exact commands, exit codes and reach. These are author-generated +sensitivity challenges, not independent held-out validation or a mutation score. + +Reproduce the original CI replay from `packages/db-sqlite-persistence-core`: + +```sh +TANSTACK_DB_ORACLE_PROPERTY=sync-metadata-session.persisted-peers \ +TANSTACK_DB_ORACLE_SEED=1581505626 \ +TANSTACK_DB_ORACLE_PATH=18:3:0:3:9:9:12:11:11:7:7:8:8:8:8:8:8 \ +node --import tsx ../db/tests/oracle-replay.ts \ + tests/sync-metadata-session-persisted.test.ts \ + --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +``` + +For the native sibling replay, select +`sync-metadata-session.sqlite-missing-row-metadata`, seed `2220037`, path `0`, +and `tests/sync-metadata-missing-row.test.ts` with the same wrapper and flags. +The `node --import tsx` form avoids the tsx CLI's sandbox-blocked IPC listener. + +## ORC audit of this repair + +| Requirement | Evidence or applicability | +| --- | --- | +| ORC-001 | D9/D11/D15/D16/D18 above authorize each expectation and bound the receiving claim | +| ORC-002 | The event-fold model imports no production metadata classifier or authority map | +| ORC-003 | Owner opening prose, the storage rule comment, new grammar prose and receiver opening comment explain the five responsibilities | +| ORC-004 | The original captured witness remains directly replayable. The sibling template reconstructs all twelve cases. Removing explicit writes loses the durable distinction, holds lose applied separation, the truncate prefix loses local/unknown discrimination, and replacement/clear tails lose the named boundaries. Repetition spans 1–3, including both margins. The grammar starts peer metadata/delete only after a durable row exists. Any illegal/skipped authored action fails. Permanent calibration independently requires all twelve case names exactly once per batch | +| ORC-005 | Both hosts invoke the persisted source API and compare source/durable metadata at each completed cut. Native notices correlate real source transaction IDs. No missing-input hypothesis remains for this boundary: named notices identify the changed key, and storage supplies its row and metadata by the comparison | +| ORC-006 | Three classified assertion-rejected production mutants and an unmodified isolated baseline | +| ORC-007 | Same fixed/random property and budget eight. Both captured seed/path replays execute the requested random campaign directly with nonzero witnesses | +| ORC-008 | Durable row absence and presence can share source metadata M, yet a keeping recreation must distinguish their durable outcome. The model now retains that distinction rather than inferring durability from source reads | +| ORC-009 | Session events abstract accepted/open source writes. Storage entries separately model durable rows and metadata. Local/peer writer bookkeeping belongs only to production, not the reference model | +| ORC-010 | Mismatches retain case/lane, history step, expected/actual observation and replay coordinates. Shared runner cleanup preserves the primary mismatch. The same source-read violation reproduces after shrinking and direct replay | +| ORC-011 | No remaining named common semantic fault requires a second interpreter. Independent authority reasoning requires source M/durable absence for a keep and durable M for an explicit write. Both hosts deliberately share one judgment and do not count as independent formulations | +| ORC-012 | This versioned self-review names the class boundary, adjacent witnesses, controls and remaining scope. Source identities below bind the evidence | +| ORC-013 | Keeping versus explicit metadata, known unrelated versus same-key peer notices, and unknown-write reloads distinguish wrong consequences at the same checkpoints | +| ORC-014 | Controlled absent durable metadata and named notices have native SQLite/coordinator receiving witnesses before the production edit. OPFS/Electron handoffs remain unresolved under the existing owner | +| ORC-015 | All 36 declared case/lane states reach eight times in each normal fixed/random campaign on each host, enforced by createOracleReach | +| ORC-016 | New registration runs the shared model/driver/comparison. Ordinary grammar calibration remains in an ordinary test file. No per-example metadata expectation enters an oracle | + +## Review, merge gate and remaining limits + +Existing formal reviews cover `9eef6525d` and `adf6a07f9`; mutable summary +6507648970 last covers `ef5c66ed2` and automatic review remains paused. +At poll 53 all formal reviews, inline comments and conversation comments are +unchanged. They do not cover semantic instrument commit `60ace06b` or this +production repair. The user authorized merge with that known review limit. +Previous review dispositions remain in the earlier versioned reports and the +append-only ledger. CI on a prior head does not certify this repair. + +This evidence covers one sync run, serialized authored actions, controlled +storage and one-process native SQLite/Node messaging. It does not cover +arbitrary overlapping recovery, cache claim expiry/generation rotation, +leadership/transport loss, restart, Chromium OPFS or Electron IPC. +Read-hold/abandonment still lacks the peer/store-held lanes' direct durable +transaction trace; the coverage map names its owner and needed witness. + +The withdrawn P6 production diagnosis stays withdrawn. This newly reproduced +retention defect does not establish a common cause with #2281, #2272, +#2317/P8 or #2298. Their broader obligations remain open. There is no known +counterexample left inside the bounded class above. Final publication-head CI +and fresh review enumeration remain required before the authorized ordinary +merge. No branch protection bypass is authorized. + +## Tested source identities + +| Source | SHA-256 | +| --- | --- | +| `packages/db-sqlite-persistence-core/src/persisted.ts` | `96d8ed9ff3ef38e9c8862ff576e8bc16c8977daef492778e1a102eaddd542f04` | +| `packages/db-sqlite-persistence-core/tests/sqlite-fixture-conformance.ts` | `8c4c6d9d28f42d9e8b461233fbbe3d48d84c0582d9decc9abb935d2c85883ba9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts` | `f90536a86c3efe345faa812c62ed3877cb1c786e2aa0185d6e440e96f7966805` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.test.ts` | `f75952c705f60519b783cf86887ada61c41c2eca86db98daa773fabc494775b9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-campaign-result.ts` | `66f2447c1116f3d10bdcee8d6a97192f81bd5e855afc3a69fbee163a75cca7cb` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts` | `1b473feeed413ff86fa37bcf3a8f4c4551c6f8202a015627f20f68683aa5f23e` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.test.ts` | `a318b173252956b8a7424817d371c1aa2ad5ce3152ad1c40d43fc1bc27259971` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery-recorder.ts` | `591206dbde1605b306bcd62b65ff2f5a9d1a001befc648d076ca052138e11a69` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-recovery.test.ts` | `880e40bd7f6aaf452dd2f3512a4ed0eb027feadaac221e1846dd9cfe0354dbfb` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-campaign.ts` | `b1de1ba24d79db954a4d85d6d7da18422ac29480d7175893d90b1ab07b24f1ae` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted-driver.ts` | `908ae60996e225757c7f6db8ab8408a5c44d7efd6ae1000188ced6588a133240` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-persisted.test.ts` | `3296a963a8737f84c72b6897817c44fd5c9b57d3fb723307115d78b3df03ded9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts` | `62aeef7a47df6b17a8e17485bd129a895846f879862c1dd30f9da2cbf1392583` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite.test.ts` | `7d4d547c70cd370dab328ef289689bb85c7e4c223907ba508f71158672af181d` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.test.ts` | `7582a61d10a944dd9bd794d5deb76bca3c6996158f3c9ef229493b98cc1ca2e9` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-sqlite-driver.ts` | `e5ef179a2a50d5e42f8b0ee51872885141056e2eef029a620f4ccd53a5512e53` | +| `packages/db-sqlite-persistence-core/tests/sync-metadata-write-work.test.ts` | `4dfa36304b65ffc17a1876eb780d23ea567e2b8c35e98bc3c588f0e926da980e` | +| `packages/db/tests/oracle-config.ts` | `f71356276ba2b1558ed6cb2af273fb57a792e76574eb88e2f8135c74e0926e1d` | +| `packages/db/tests/oracle-replay-manifest.ts` | `4023c661149ac6e3fb889db3e5b2bb35dcdfa498a8118ec1f340e90c84899009` | +| `packages/db/tests/sync-metadata-recovery-oracle.ts` | `ed05738b4d44411e617b82eccdd62ca51d4e5749ad4f698be9fd86f582976e13` | +| `packages/db/tests/sync-metadata-session-oracle.ts` | `0a60a92f234f138c7bfcf6a37b8ec3b9aeb473246c0b5ae751734f9c9a2ecb55` | +| `packages/db/tests/sync-metadata-session.test.ts` | `48778b83127e6c2e3c846fcae535d0273f3dcb709da716726c1a36bebd5ddd6f` | diff --git a/packages/db-sqlite-persistence-core/src/persisted.ts b/packages/db-sqlite-persistence-core/src/persisted.ts index 666513c54a..14c902414e 100644 --- a/packages/db-sqlite-persistence-core/src/persisted.ts +++ b/packages/db-sqlite-persistence-core/src/persisted.ts @@ -1687,9 +1687,10 @@ class PersistedCollectionRuntime< // until this tab's source writes the key; a storage read records each row // it finds changed as a source's write. private runWrittenKeys = new Set() - // Row metadata is a separate object (D5): keys whose metadata a source - // wrote in this run, a delete included. - private runWrittenMetadataKeys = new Set() + // Row metadata is a separate object (D5). Its latest writer distinguishes + // local metadata, which can exist only in this run for a missing row (D16), + // from a peer's named replacement that a storage read must receive (D11). + private runMetadataWriters = new Map() private runWritesUnknown = false private peerWritesUnknown = false private keysKnownSincePeerWrites = new Set() @@ -2330,7 +2331,7 @@ class PersistedCollectionRuntime< const previousControls = this.syncControls this.setSyncControls(syncControls) this.runWrittenKeys.clear() - this.runWrittenMetadataKeys.clear() + this.runMetadataWriters.clear() this.runWritesUnknown = false this.peerWritesUnknown = false this.keysKnownSincePeerWrites.clear() @@ -3665,7 +3666,7 @@ class PersistedCollectionRuntime< this.keysKnownSincePeerWrites.add(key) } for (const key of metadataKeys) { - this.runWrittenMetadataKeys.add(key) + this.runMetadataWriters.set(key, `local`) this.keysKnownSincePeerWrites.add(key) } } @@ -3708,7 +3709,7 @@ class PersistedCollectionRuntime< state.getAcceptedSyncedRow(row.key), row.value, ) && - (this.keepsSeedMetadata(row.key) || + (this.keepsMetadata(row.key) || equalPersistedSnapshotValues( this.syncControls.metadata?.row.get(row.key), row.metadata, @@ -3750,29 +3751,33 @@ class PersistedCollectionRuntime< /** * Writes a stored row over the row this tab holds. A stored row is whole, * so it replaces that row. Its metadata replaces this tab's unless this - * tab held the row and keeps its metadata as a seed's (decision 18). + * tab held the row and retains seed or local source metadata (D18/D16). */ private writeStoredRow( row: { key: TKey; value: T; metadata?: unknown }, - keepsSeedMetadata: boolean, + keepsMetadata: boolean, ): void { this.syncControls.write?.({ type: `update`, value: row.value, [syncWriteNamesWholeRow]: true, } as { type: `update`; value: T }) - if (!keepsSeedMetadata || !this.keepsSeedMetadata(row.key)) + if (!keepsMetadata || !this.keepsMetadata(row.key)) this.writeStoredMetadata(row.key, row.metadata) } /** - * Whether a reload keeps this tab's metadata for the key: no source wrote - * it in this run, so it is a seed's (decision 18). When this run's writes - * are unknown, stored metadata that differs is a source write instead - * (decision 15). + * A known storage read keeps seed metadata (D18) and local source metadata + * (D16), including a value that was never durable. A named peer replacement + * supplies its own metadata (D11); unknown peer writes are judged by D15. + * A local truncate removes old seeds, but does not make later local writes + * unknown. Local writes after an unknown notice establish their key again. */ - private keepsSeedMetadata(key: TKey): boolean { - return !this.writesUnknown(key) && !this.runWrittenMetadataKeys.has(key) + private keepsMetadata(key: TKey): boolean { + const writer = this.runMetadataWriters.get(key) + if (writer === `local`) + return !this.peerWritesUnknown || this.keysKnownSincePeerWrites.has(key) + return writer === undefined && !this.writesUnknown(key) } private writeStoredMetadata(key: TKey, metadata: unknown): void { @@ -4194,7 +4199,7 @@ class PersistedCollectionRuntime< if (!durable) this.syncControls.write?.({ type: `delete`, key }) else if (!equalPersistedSnapshotValues(base?.get(key), durable.value)) this.writeStoredRow(durable, true) - else if (!this.keepsSeedMetadata(key)) + else if (!this.keepsMetadata(key)) this.writeStoredMetadata(key, durable.metadata) } } @@ -5621,7 +5626,7 @@ class PersistedCollectionRuntime< (mutation) => mutation.key, ), ]) - this.runWrittenMetadataKeys.add(key as TKey) + this.runMetadataWriters.set(key as TKey, `peer`) } if (this.scopedRecovery) { if (txCommitted.requiresFullReload) this.runWritesUnknown = true diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts index de240fd9ff..f172d5432d 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-calibration.test.ts @@ -2,6 +2,7 @@ import { fc } from '@fast-check/vitest' import { expect, it } from 'vitest' import { + missingRowMetadataHistoriesArbitrary, persistedPeerLane, persistedPeerOnDemandLane, queuedMetadataRepairHistoriesArbitrary, @@ -18,6 +19,30 @@ import { createSQLiteDriver } from './sync-metadata-session-sqlite-driver' import type { SessionDriver } from '../../db/tests/sync-metadata-session-oracle' import type { RecoveryPhase } from '../../db/tests/sync-metadata-recovery-oracle' +it(`retains missing-row metadata boundaries in every generated batch`, () => { + const required = new Set([ + `immediate: kept`, + `immediate: explicit`, + `core held: kept`, + `core held: explicit`, + `after source truncate: immediate`, + `after source truncate: core held`, + `peer row replaces metadata`, + `peer metadata replaces metadata`, + `unknown writes replace metadata`, + `peer delete clears metadata`, + `source delete clears metadata`, + `source truncate clears metadata`, + ]) + for (const batch of fc.sample(missingRowMetadataHistoriesArbitrary, { + seed: 2220037, + numRuns: 8, + })) { + expect(batch).toHaveLength(required.size) + expect(new Set(batch.map(({ name }) => name))).toEqual(required) + } +}) + it(`rejects a recovery schedule whose boundary the driver cannot hold`, async () => { await expect( runRecoverySchedule( diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts new file mode 100644 index 0000000000..92de039069 --- /dev/null +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts @@ -0,0 +1,109 @@ +// @vitest-environment node +import { fc } from '@fast-check/vitest' +import { it } from 'vitest' +import { + describeMismatch, + missingRowMetadataCases, + missingRowMetadataHistoriesArbitrary, + persistedPeerFullLane, + persistedPeerLane, + persistedPeerOnDemandLane, + runSessionHistory, +} from '../../db/tests/sync-metadata-session-oracle' +import { + oraclePropertyOptions, + oracleRuns, + readOracleRunConfig, + recordOracleReplay, +} from '../../db/tests/oracle-config' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { requireExecutedHistory } from './sync-metadata-campaign-result' +import { createPersistedDriver } from './sync-metadata-session-persisted-driver' +import { createSQLiteDriver } from './sync-metadata-session-sqlite-driver' + +/** + * Both receivers use the session oracle's D16/D9/D18 model and comparison. + * A nested delete leaves source metadata for a missing durable row. The outer + * update recreates only its row unless it explicitly writes metadata. Reads + * and durable metadata are compared at each completed commit, then after a + * named peer notice. Core holds vary application without delaying durability. + * This is one sync run; restart and intermediate SQL statements are outside it. + */ +const lanes = [ + persistedPeerLane, + persistedPeerFullLane, + persistedPeerOnDemandLane, +] +const requested = readOracleRunConfig().replayProperty +for (const [host, property] of [ + [`persisted`, `sync-metadata-session.persisted-missing-row-metadata`], + [`sqlite`, `sync-metadata-session.sqlite-missing-row-metadata`], +] as const) { + for (const campaign of [`fixed`, `random`] as const) { + if ( + requested !== undefined && + (requested !== property || campaign === `fixed`) + ) + continue + it(`keeps missing-row metadata local unless explicitly written (${host}, ${campaign})`, async () => { + const state = (lane: string, name: string) => `${lane}: ${name}` + const reach = createOracleReach( + property, + lanes.flatMap((lane) => + missingRowMetadataCases.map(({ name }) => state(lane.name, name)), + ), + ) + const result = await fc.check( + fc.asyncProperty( + missingRowMetadataHistoriesArbitrary, + async (batch) => { + const failures: Array = [] + for (const lane of lanes) { + for (const { name, history } of batch) { + reach.beginHistory() + const historyResult = await runSessionHistory( + lane, + history, + () => + host === `sqlite` + ? createSQLiteDriver(lane) + : createPersistedDriver( + true, + true, + false, + true, + lane.fullRows, + lane.onDemand, + ), + ) + if (historyResult.mismatch) { + failures.push( + `${state(lane.name, name)}\n${describeMismatch(history, historyResult.mismatch)}`, + ) + } else if (historyResult.steps !== history.length) + throw new Error( + `constructive history skipped an authored action`, + ) + else reach.reach(state(lane.name, name)) + } + } + if (failures.length) throw new Error(failures.join(`\n\n`)) + }, + ), + campaign === `fixed` + ? { numRuns: oracleRuns(8), seed: 2220037 } + : oraclePropertyOptions(8, property), + ) + recordOracleReplay(property, result) + requireExecutedHistory(property, result) + console.info( + `ORACLE_REACH ${JSON.stringify({ property, campaign, seed: result.seed, runs: result.numRuns, counts: Object.fromEntries(reach.counts()) })}`, + ) + if (result.failed) + throw new Error( + `${result.error}\nseed=${result.seed} path=${result.counterexamplePath}`, + ) + reach.assertReached() + }, 240_000) + } +} diff --git a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts index 85ece2e9ab..8880680f46 100644 --- a/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts +++ b/packages/db-sqlite-persistence-core/tests/sync-metadata-session-sqlite-driver.ts @@ -154,12 +154,15 @@ export async function createSQLiteDriver( } } let lastPeerTxId: string | undefined - const sourceTxIds: Array = [] + const sourceTransactions: Array<{ txId: string; reload: boolean }> = [] const sourceApply = coordinators[0]!.requestApplyCommittedTx.bind( coordinators[0], ) coordinators[0]!.requestApplyCommittedTx = (...args) => { - sourceTxIds.push(args[1].txId) + sourceTransactions.push({ + txId: args[1].txId, + reload: args[1].truncate === true, + }) return sourceApply(...args) } let peerSubmitted: (() => void) | undefined @@ -380,7 +383,7 @@ export async function createSQLiteDriver( }, commit: async () => { const noticeStart = notices.length - const transactionStart = sourceTxIds.length + const transactionStart = sourceTransactions.length const index = receiptStates.length const receipt = sync.commit() depth-- @@ -402,8 +405,8 @@ export async function createSQLiteDriver( // that actually read the pre-peer durable state. Core holds may keep the // applied receipt pending, so wait for acceptance rather than visibility. await whenSyncAccepted(receipt) - for (const txId of sourceTxIds.slice(transactionStart)) - await awaitNotice(noticeStart, false, txId) + for (const { txId, reload } of sourceTransactions.slice(transactionStart)) + await awaitNotice(noticeStart, reload, txId) await quiesce() }, cancel: abort, @@ -452,12 +455,24 @@ export async function createSQLiteDriver( value: row.value as Row, ...rowMetadata(row.metadata), }) - } else if (notice === `row`) peer.begin() + } else if ( + notice === `row` || + notice === `metadata` || + notice === `delete` + ) + peer.begin() else throw new Error( - `unexpected peer notice in the seed/re-insert grammar: ${notice}`, + `unsupported peer notice in the native metadata driver: ${notice}`, ) - peer.write({ type: `insert`, value: value as Row, ...rowMetadata(meta) }) + if (notice === `metadata`) peer.metadata!.row.set(key, meta) + else if (notice === `delete`) peer.write({ type: `delete`, key }) + else + peer.write({ + type: `insert`, + value: value as Row, + ...rowMetadata(meta), + }) await peer.commit() if (!lastPeerTxId) throw new Error(`the peer issued no durable transaction`) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index e3ff127211..c1fdf9c3c1 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -292,6 +292,8 @@ const staticOracleProperties = [ `sync-metadata-session.persisted-queued-metadata-repair`, `sync-metadata-session.sqlite-kept-metadata-durability`, `sync-metadata-session.sqlite-recovery-schedules`, + `sync-metadata-session.persisted-missing-row-metadata`, + `sync-metadata-session.sqlite-missing-row-metadata`, `sync-metadata-session.nested-conflicts.hold-invariance`, `sync-metadata-session.nested-conflicts.key-independence`, `d2-source.exact-retractions`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 05d7ba06ee..4bed53bada 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -741,6 +741,11 @@ const ownerGroups: ReadonlyArray< `sync-metadata-session`, `sqlite-recovery-schedules`, ], + [ + `db-sqlite-persistence-core/tests/sync-metadata-missing-row.test.ts`, + `sync-metadata-session`, + `persisted-missing-row-metadata sqlite-missing-row-metadata`, + ], ] const owners = new Map() diff --git a/packages/db/tests/sync-metadata-session-oracle.ts b/packages/db/tests/sync-metadata-session-oracle.ts index 16f44b59bb..eacb7199fb 100644 --- a/packages/db/tests/sync-metadata-session-oracle.ts +++ b/packages/db/tests/sync-metadata-session-oracle.ts @@ -67,6 +67,9 @@ import { declareOracle } from './oracle-declaration.js' * re-inserts without explicit metadata leave durable metadata alone. A * named current-run write is source-written (D11); under unknown writes, * differing stored metadata may become source-written (D15 and D18). + * - Metadata written for a missing row stays in this sync run until a source + * deletes the row (D16). It is not durable. Recreating the row without a + * metadata write cannot make that run-only value durable (D9/D18). * * Authority: maintainer decisions of 2026-10-22 to borrow standard theory and * to place seeds beneath source writes (D4, D5, and its clarification "b"); @@ -75,7 +78,7 @@ import { declareOracle } from './oracle-declaration.js' * share the sync API with the source; the glossary * entries for accepted sync transactions and the exposed authoritative base; * and the row-metadata write rules of 2026-10-05 for implicit metadata. - * Decisions 9, 11, 14, 15 and 18 are recorded in + * Decisions 9, 11, 14, 15, 16 and 18 are recorded in * `docs/contributing/oracle-reviews/2026-10-22-sync-metadata-session.md`. * * ## Limits @@ -755,6 +758,108 @@ export const queuedMetadataRepairHistoriesArbitrary = fc ), ) +/** + * D16 keeps metadata written while a row is missing in this run, without + * storing it. An enclosing update can recreate the row after an inner delete + * commits first. Keeping that metadata still does not write it (D9/D18). + * Named writes of the same key replace it (D11), unknown writes use D15, and + * source deletes and truncates clear it. Each batch includes these boundaries + * and immediate/core-held implicit/explicit writes. Repetition varies the + * missing-row value. The source-truncate prefix removes the prior seed and + * creates a source row before reconstructing the same nested history. + */ +export const missingRowMetadataCases: ReadonlyArray<{ + name: string + held?: boolean + metadata?: boolean + priorTruncate?: boolean + tail: ReadonlyArray +}> = [ + ...[false, true].flatMap((held) => + [false, true].map((metadata) => ({ + name: `${held ? `core held` : `immediate`}: ${metadata ? `explicit` : `kept`}`, + held, + metadata, + tail: [{ type: `peer`, key: 2, notice: `row` }] as Array, + })), + ), + ...[false, true].map((held) => ({ + name: `after source truncate: ${held ? `core held` : `immediate`}`, + held, + priorTruncate: true, + tail: [{ type: `peer`, key: 2, notice: `row` }] as Array, + })), + { + name: `peer row replaces metadata`, + tail: [{ type: `peer`, key: 1, notice: `row` }], + }, + { + name: `peer metadata replaces metadata`, + tail: [{ type: `peer`, key: 1, notice: `metadata` }], + }, + { + name: `unknown writes replace metadata`, + tail: [{ type: `peer`, key: 2, notice: `reload` }], + }, + { + name: `peer delete clears metadata`, + tail: [{ type: `peer`, key: 1, notice: `delete` }], + }, + { + name: `source delete clears metadata`, + tail: [ + { type: `begin` }, + { type: `remove`, key: 1 }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `row` }, + ], + }, + { + name: `source truncate clears metadata`, + tail: [ + { type: `begin` }, + { type: `truncate` }, + { type: `commit` }, + { type: `peer`, key: 2, notice: `row` }, + ], + }, +] + +export const missingRowMetadataHistoriesArbitrary = fc + .integer({ min: 1, max: 3 }) + .map((repeats) => + missingRowMetadataCases.map( + ({ name, held, metadata, priorTruncate, tail }) => { + const history: Array = [ + ...(priorTruncate + ? [ + { type: `begin` as const }, + { type: `truncate` as const }, + { type: `commit` as const }, + { type: `begin` as const }, + { type: `upsert` as const, key: 1 as const, metadata: false }, + { type: `commit` as const }, + ] + : []), + ...(held ? [{ type: `hold` as const }] : []), + { type: `begin` }, + { type: `upsert`, key: 1, metadata: metadata === true }, + { type: `begin` }, + { type: `remove`, key: 1 }, + ...Array.from({ length: repeats }, (): SessionAction => ({ + type: `rowSet`, + key: 1, + })), + { type: `commit` }, + { type: `commit` }, + ...(held ? [{ type: `release` as const }] : []), + ...tail, + ] + return { name, history } + }, + ), + ) + /** * This classifier records a completed implicit re-insert across a full reload * while the enclosing write remains unaccepted. "Kept" names that implicit @@ -1304,11 +1409,16 @@ export class SessionModel { row: absent, metadata: absent, } + // D16's metadata for a missing row exists only in this run. Recreating + // the durable row cannot revive it without a metadata write (D9/D18). + // The session events still retain it for source reads. this.storage.set(key, { row: writesRow ? this.fold(`row`, key, entries) : previous.row, metadata: writesMetadata ? this.fold(`rowMetadata`, key, entries) - : previous.metadata, + : previous.row.present + ? previous.metadata + : absent, }) } } @@ -2891,5 +3001,6 @@ declareOracle({ completedSeedReinsertReloadRegime, keptMetadataDurabilityHistoriesArbitrary, queuedMetadataRepairHistoriesArbitrary, + missingRowMetadataHistoriesArbitrary, ], }) From 6d799479a9ec9a26d1c53b5a4ab56e2003fc5417 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 14:14:47 -0700 Subject: [PATCH 11/11] Bind metadata retention evidence to tested repair --- .../2026-11-02-p6-ci-missing-row.md | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md b/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md index 4f5112102b..1d80d96195 100644 --- a/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md +++ b/docs/contributing/oracle-reviews/2026-11-02-p6-ci-missing-row.md @@ -7,6 +7,9 @@ metadata. Same-key peer replacement, unknown-write observations, source deletes and source truncates retain their accepted meanings. Parent publication: `88641f0847975ed920fc44eb48e48f5a602c5206`. +Semantic repair and test commit: `12558a9d380461eca6fc6b824788a3cf19e404bc`. +All 22 source hashes below match after commit hooks. The following publication +commit adds only this evidence binding and command detail. The source identities below bind the tested repair. This is a self-review, not independent review. The user authorized the production repair after the preliminary investigation, and retained the merge-when-green instruction. @@ -128,6 +131,31 @@ The new campaigns use identical fixed/random budgets of eight, with fixed seed 2220037. The normal random seeds were -861738368 (controlled) and -836725301 (native). No existing budget, guard, comparison or test is removed or weakened. +The scoped commands, from `packages/db-sqlite-persistence-core`, were: + +```sh +pnpm exec vitest run tests/sync-metadata-missing-row.test.ts \ + --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +pnpm exec vitest run tests/sync-metadata-session-persisted.test.ts \ + tests/sync-metadata-session-sqlite.test.ts tests/sync-metadata-recovery.test.ts \ + tests/sync-metadata-calibration.test.ts tests/persisted-oracle.test.ts \ + --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +pnpm exec vitest run tests/sync-metadata-recovery-recorder.test.ts \ + tests/sync-metadata-campaign-result.test.ts tests/sync-metadata-sqlite-driver.test.ts \ + tests/sync-metadata-write-work.test.ts \ + --maxWorkers=1 --coverage.enabled=false --typecheck.enabled=false +``` + +From `packages/db`, the core command selected +`tests/sync-metadata-session-oracle.property.test.ts`, +`tests/sync-metadata-session.test.ts`, +`tests/collection-row-metadata-composition-oracle.test.ts`, +`tests/oracle-config.test.ts` and `tests/oracle-replay.test.ts` with the same +flags. Its literal-owner guard then passed separately with +`-t 'maps literal owners'`. These commands establish 1,443 passing tests +across the scopes, with 49 existing expected failures and one todo in the +broad persisted owner. The full workspace command was not run locally. + ## Hostile controls and replay An isolated tree starts from the parent and overlays the candidate sources.