From 139549048b803cc042c15c13542f3048a203c400 Mon Sep 17 00:00:00 2001 From: Test Date: Mon, 2 Nov 2026 19:50:11 -0700 Subject: [PATCH 01/28] docs(db-adapter-conformance): plan the mutations and confirmation family Co-authored-by: Isaac --- packages/db-adapter-conformance/MIGRATION.md | 37 +++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 403ab601d7..189786234c 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -46,9 +46,44 @@ reviewed change. 1. **Push adapters on the existing family:** RxDB (done), TrailBase (done), PowerSync (done), Electric (eager mode, done), and Convex (eager mode, done). Each needs only a controlled-provider driver. A RED result is a finding against that adapter. 2. **Retire duplicated adapter tests (separate PR, maintainer decision 2026-11-02).** For each adapter on the eager family, classify its existing eager-mode tests and oracles against what the shared oracle generates and observes: remove a test only when the shared family checks the same law, history, path, and observation; narrow one it covers in part; keep the rest. Known overlap: TrailBase's lifecycle oracle and the shared family both own eager startup failure and stream-end health. 3. **The Neon update, run against its driver.** This shows an adapter moving from RED to GREEN. Publishing the package for drivers outside this repository (a public package, `@tanstack/db` as a peer dependency, an exported reach counter) is deferred (maintainer decision 2026-11-02). -4. **The mutations family.** It applies to every adapter except the storage hosts. +4. **The mutations family (in progress).** It applies to every adapter except the storage hosts. 5. **The other families,** one at a time. Each family's new driver work goes to the adapters that declare it. +## Next slice: mutations and confirmation (started 2026-11-02) + +The family checks what an adapter owes when an application writes, through +one shared model and one driver per adapter, as the eager family does. Core +keeps its own owners for laws that no adapter can change: joining a +transaction (MUT-4) and holding synced commits behind pending optimistic +state (MUT-5) stay with `packages/db/tests/optimistic-history-oracle.ts`. + +- **Laws, first cut:** MUT-1 (the write is visible until it settles), MUT-2 + (a rejected write rolls back and `isPersisted` rejects with the error), + MUT-13 and MUT-3 (one handler call per write, with the authored values), + MUT-10 (writes reach the backend in order per Collection), FID-5 and MUT-8 + (after settlement the backend holds the fold of accepted writes; a rejected + write leaves no backend effect), and MUT-7 (under confirmation through sync, + the write stays persisting until its own write arrives in a committed + batch). Authority: the law set's section 9 and FID-5. +- **Capabilities:** `adapter-persistence` (the adapter writes to the backend + itself: RxDB, TrailBase, PowerSync) and `confirm-through-sync` (the + application's handler writes and returns evidence: Electric, Convex). +- **Model:** the synced rows, the backend rows, and the pending writes in + author order with their outcome. Visible rows are the synced rows with the + pending writes folded on; a settled write leaves the fold. +- **Grammar:** inserts, updates, and deletes through the Collection, + single or in one manual transaction; the backend accepts or rejects a held + write; the backend's own changes arrive through sync (confirming a pending + write or not); an external change to another row or the same key. +- **Observations:** visible rows and change events, each transaction's state + and settlement, the backend rows and the order of backend operations, + handler calls. +- **Calibration:** a reference adapter for each capability and hostile + mutants per law. +- **Order:** the family and its reference adapter, then RxDB, TrailBase, + PowerSync (adapter persistence), then Electric and Convex (confirmation + through sync). + ## Migration log - **2026-10-29, TrailBase, eager live source.** Both campaigns are GREEN. From da23ed3245d893d818eece8ef891ddbd992ea65d Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 07:55:08 -0700 Subject: [PATCH 02/28] test(db): generate field removal in the optimistic-history oracle The core owner's rows had three required fields and its edits only assigned, so no history removed a field, and its row comparisons used toEqual, which cannot tell an own undefined from an absent field. Rows now carry an optional `d` that only edits set; an edit can remove it with `delete draft.d`, always after an edit that sets it on that key. The model represents the removal as an own `undefined`, in the row and in `changes`, the documented rule of proxy-detachment-contract-oracle (maintainer decision 2026-11-03). Reads, replicas, downstream rows, and the same-tick optimistic check now compare strictly, and a `removedFieldEdits` count must be above zero in the fixed campaign's histories. - No production change: the dimension encodes the existing contract. A core change that left the removed field absent failed proxy-detachment-contract-oracle, which led to the decision. - GREEN: @tanstack/db 30887 passed; the persisted optimistic-history lane in db-sqlite-persistence-core passes. Co-authored-by: Isaac --- ...on-state-retention-oracle.property.test.ts | 13 ++ .../db/tests/optimistic-history-oracle.ts | 132 +++++++++++++++--- 2 files changed, 124 insertions(+), 21 deletions(-) diff --git a/packages/db/tests/collection-state-retention-oracle.property.test.ts b/packages/db/tests/collection-state-retention-oracle.property.test.ts index 2d94e82a4f..3cb4cd8f34 100644 --- a/packages/db/tests/collection-state-retention-oracle.property.test.ts +++ b/packages/db/tests/collection-state-retention-oracle.property.test.ts @@ -809,6 +809,19 @@ it(`writes zero-sign edits over in-flight writes in the fixed campaign`, async ( expect(zeroSignEdits).toBeGreaterThan(0) }) +// Reach: an edit that removes a field the row holds. The update API keeps +// the removed field as an own `undefined`, which only strict row comparisons +// distinguish from an absent one. +it(`removes a held field in the fixed campaign`, async () => { + const histories = fc.sample(optimisticHistory, { seed: 86103, numRuns: 40 }) + let removedFieldEdits = 0 + for (const { initial, steps } of histories) { + const counts = await runOptimisticHistory(initial, steps) + removedFieldEdits += counts.removedFieldEdits + } + expect(removedFieldEdits).toBeGreaterThan(0) +}) + // Pinned replays for a source delete of a key the source never held. The // backend accepted an optimistic insert, then deleted the row before the // source streamed it. The model's drain applies the delete to the applied diff --git a/packages/db/tests/optimistic-history-oracle.ts b/packages/db/tests/optimistic-history-oracle.ts index 90e21f885e..2eba16da1d 100644 --- a/packages/db/tests/optimistic-history-oracle.ts +++ b/packages/db/tests/optimistic-history-oracle.ts @@ -140,7 +140,17 @@ import type { CollectionConfig, SyncConfig } from '../src/types.js' * fields. */ -export type HistoryRow = { id: number; a: number; b: number; c: number } +// `d` is optional: only an application's edit sets it, and an edit can +// remove it again. The update API represents a removed field as an own +// `undefined` field in the row and in `changes` (the documented rule in +// proxy-detachment-contract-oracle.test.ts; maintainer decision 2026-11-03). +export type HistoryRow = { + id: number + a: number + b: number + c: number + d?: number +} type Fields = Partial> type SourceOperation = { type: `row`; row: HistoryRow } | { type: `delete`; key: number } @@ -183,6 +193,8 @@ export type OptimisticStep = // Replace `fields` with the visible row's zero fields, signs flipped. // A missing row, or one with no zero, keeps `fields`. flipZeros?: boolean + // The draft also deletes `d` (`delete draft.d`). + removesD?: boolean } | { type: `delete` @@ -332,16 +344,20 @@ class HistoryModel { row && Object.entries(step.fields).every(([key, value]) => Object.is(row[key as keyof Fields], value), - ) + ) && + !(step.removesD && Object.hasOwn(row, `d`)) ) return + const authored: HistoryRow = { + ...(row ?? { id: step.key, a: 0, b: 0, c: this.insertDefault }), + ...(step.type === `edit` ? step.fields : {}), + } + if (step.type === `edit` && step.removesD && Object.hasOwn(authored, `d`)) + authored.d = undefined this.transactions.push({ key: step.key, kind: step.type === `delete` ? `delete` : row ? `update` : `insert`, - row: { - ...(row ?? { id: step.key, a: 0, b: 0, c: this.insertDefault }), - ...(step.type === `edit` ? step.fields : {}), - }, + row: authored, optimistic: step.optimistic, state: step.type === `edit` && step.pending ? `pending` : `persisting`, held: false, @@ -517,7 +533,14 @@ class HistoryModel { } } -const plain = ({ id, a, b, c }: HistoryRow): HistoryRow => ({ id, a, b, c }) +const plain = (row: HistoryRow): HistoryRow => ({ + id: row.id, + a: row.a, + b: row.b, + c: row.c, + // An own `d`, even `undefined`, stays; an absent one stays absent. + ...(Object.hasOwn(row, `d`) ? { d: row.d } : {}), +}) const observed = ( row: HistoryRow & { $origin: `local` | `remote` @@ -729,7 +752,8 @@ export const optimisticStep: fc.Arbitrary = fc.oneof( key: fc.integer({ min: 1, max: 3 }), fields: fc .record( - { a: fieldValue, b: fieldValue, c: fieldValue }, + // `d` too, so a later edit often finds a field to remove. + { a: fieldValue, b: fieldValue, c: fieldValue, d: fieldValue }, { requiredKeys: [] }, ) .filter((fields) => Object.keys(fields).length > 0), @@ -737,6 +761,29 @@ export const optimisticStep: fc.Arbitrary = fc.oneof( inHandler: handlerSourceBatch, }), }, + { + // An edit that sets the optional `d`, or removes it with `delete` + // (`removesD`). Removal needs a row that holds `d`, so both arms share + // a key range small enough to revisit. + weight: 2, + arbitrary: fc.oneof( + fc.record({ + type: fc.constant(`edit` as const), + key: fc.integer({ min: 1, max: 3 }), + fields: fieldValue.map((d): Fields => ({ d })), + optimistic: fc.boolean(), + inHandler: fc.constant(undefined), + }), + fc.record({ + type: fc.constant(`edit` as const), + key: fc.integer({ min: 1, max: 3 }), + fields: fc.constant({}), + optimistic: fc.boolean(), + inHandler: fc.constant(undefined), + removesD: fc.constant(true), + }), + ), + }, { // An edit that changes only the sign of the row's zeros (-0 and 0 are // distinct row values). Random fields rarely do it by chance. It @@ -785,7 +832,26 @@ export const optimisticHistory = fc.record({ selector: (row) => row.id, maxLength: 3, }), - steps: fc.array(optimisticStep, { minLength: 2, maxLength: 24 }), + steps: fc + .array(optimisticStep, { minLength: 2, maxLength: 24 }) + // A removal of `d` follows an edit that sets it on the same key, so the + // removal finds a field to remove; random steps rarely line them up. + .map((steps) => + steps.flatMap((step): Array => + step.type === `edit` && step.removesD + ? [ + { + type: `edit`, + key: step.key, + fields: { d: 1 }, + optimistic: step.optimistic, + inHandler: undefined, + }, + step, + ] + : [step], + ), + ), }) // The core grammar plus store steps. While a hold is outstanding, an // in-handler batch does not await acceptance. @@ -997,6 +1063,7 @@ export async function runOptimisticHistory( a: z.number(), b: z.number(), c: z.number().default(options.insertDefault), + d: z.number().optional(), }), }) const collection = @@ -1049,6 +1116,8 @@ export async function runOptimisticHistory( // visible row changed only by the sign of a zero: no pending-writes flag // changes, so only value equality can publish the change. zeroSignEditsOverPending: 0, + // Accepted edits that remove `d` from a row holding it. + removedFieldEdits: 0, // Drains that kept a settled row held for a deferred transaction. deferredHolds: 0, abortedDeferred: 0, @@ -1394,18 +1463,19 @@ export async function runOptimisticHistory( expect( actual, `${label}: reads ${JSON.stringify(actual)} expected ${JSON.stringify(expected)}`, - ).toEqual(expected) - expect(sorted(replica.values()), `${label}: event replica`).toEqual( - expected, - ) + ).toStrictEqual(expected) + expect( + sorted(replica.values()), + `${label}: event replica`, + ).toStrictEqual(expected) expect( sorted(rawReplica.values()), `${label}: raw event replica`, - ).toEqual(expected) + ).toStrictEqual(expected) expect( sorted([...downstream.values()].map(plain)), `${label}: downstream`, - ).toEqual(expected.map(plain)) + ).toStrictEqual(expected.map(plain)) // Every per-mutation pass walks the tracked transactions, so the // Collection tracks exactly its unsettled transactions. A held row // outlives its completed transaction in the held-row layer, not here. @@ -1451,6 +1521,13 @@ export async function runOptimisticHistory( if (index === undefined) continue const intent = model.transactions[index]! const was = before.find((row) => row.id === step.key) + if ( + step.type === `edit` && + step.removesD && + was !== undefined && + Object.hasOwn(was, `d`) + ) + counts.removedFieldEdits++ const now = model.visible().get(step.key) if ( inFlight && @@ -1499,7 +1576,10 @@ export async function runOptimisticHistory( : collection.update( step.key, { optimistic: step.optimistic }, - (draft) => Object.assign(draft, step.fields), + (draft) => { + Object.assign(draft, step.fields) + if (step.removesD) delete draft.d + }, ) const manual = step.type === `edit` && step.pending === true const tx = manual @@ -1517,7 +1597,9 @@ export async function runOptimisticHistory( expect( live && plain(live), `${position}: optimistic mutation is live in the same tick`, - ).toEqual(step.type === `delete` ? undefined : plain(intent.row)) + ).toStrictEqual( + step.type === `delete` ? undefined : plain(intent.row), + ) } const operation: (typeof operations)[number] = { tx, @@ -1544,9 +1626,10 @@ export async function runOptimisticHistory( : plain(intent.row) : // An authored field is a change unless it is the same // value; -0 and 0 are distinct row values (maintainer - // decision, 2026-10-29). - Object.fromEntries( - Object.entries(step.fields).filter( + // decision, 2026-10-29). A removed field is a change + // marked `undefined`. + Object.fromEntries([ + ...Object.entries(step.fields).filter( ([key, value]) => !Object.is( before.find((row) => row.id === intent.key)?.[ @@ -1555,7 +1638,14 @@ export async function runOptimisticHistory( value, ), ), - ), + ...(step.removesD && + before.some( + (row) => + row.id === intent.key && Object.hasOwn(row, `d`), + ) + ? [[`d`, undefined]] + : []), + ]), } operations.push(operation) expect( From 30c3bfca7007e9d448b6b08d6faacbd7615cbe4c Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 07:55:14 -0700 Subject: [PATCH 03/28] feat(db-adapter-conformance): the mutations family, with a reference adapter A shared oracle for adapters that persist an application's writes: MUT-1, MUT-2, MUT-8, MUT-10, MUT-13, and FID-5, over auto-transaction inserts, updates, and deletes on a ready Collection. The model holds the provider's rows and the pending mutations in author order; MUT-10 lets a write reach the provider only after the earlier one is decided. A controlled provider answers each request by a plan the check hands it first (accept or reject, at once or held until a release), records every request, and commits changes from other clients. At each cut the check compares the provider's request log, the public rows, and every when('settled') outcome; at a quiescent cut it requires the public rows, collection.base, and the provider's rows to be identical. Publications compare row values, so core's $hasPendingWrites update is not a change. The reference adapter persists each write one at a time per Collection, and its sync run applies the provider's change feed, so an accepted write's copy commits before the mutation function settles. - Law: the law set's section 9 and FID-5 (issue #2118). - Reach: nine declared states, at least five histories each, in the fixed (seed 2118, 300 histories) and random campaigns. - Mutants: a copy applied after settling (MUT-1), a swallowed and a wrapped rejection (MUT-2), a duplicated insert and writes that do not wait (MUT-10/MUT-13), a skipped delete and changes-only updates (MUT-8), each rejected with its law. - Limits: one pending mutation per key; commits from other clients only while nothing is pending; no manual transactions, MUT-7, schema validation, or no-op updates; no published adapter driver yet. ConformanceViolation's history is now `unknown`, so both families use it. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 38 + packages/db-adapter-conformance/MIGRATION.md | 10 + .../src/eager-live-source-oracle.ts | 3 +- packages/db-adapter-conformance/src/index.ts | 23 + .../src/mutation-oracle.ts | 959 ++++++++++++++++++ .../tests/reference-mutation-adapter.ts | 264 +++++ ...reference-mutation-oracle.property.test.ts | 99 ++ packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 9 + 9 files changed, 1405 insertions(+), 1 deletion(-) create mode 100644 packages/db-adapter-conformance/src/mutation-oracle.ts create mode 100644 packages/db-adapter-conformance/tests/reference-mutation-adapter.ts create mode 100644 packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index da2b6fbbbe..e2989ff1fc 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7212,3 +7212,41 @@ the replay can still be taken for it (open limit). A cached valid result served client-cache behavior and is not generated, because the model starts a restarted run empty. A fixed-seed campaign runs the React client (`watchQuery`). + +## Adapter conformance: mutations (#2118) + +Owner: `packages/db-adapter-conformance/src/mutation-oracle.ts`, run against +the reference adapter in +`packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts` +(`adapter-conformance.mutations.reference`). It checks auto-transaction +writes on a ready Collection whose adapter persists them itself: MUT-1 (the +public rows show the optimistic state while the mutation function runs, and +no publication shows the earlier synced value when it fulfills), MUT-2 (a +provider rejection rejects `when('settled')` with the provider's error and +publishes exactly the inverse), MUT-8, MUT-10, and MUT-13 (each write is one +provider request of its type with the authored row, in author order, issued +only after the earlier write is decided), and FID-5 (at a quiescent cut the +public rows, `collection.base`, and the provider's rows are identical and +equal the fold of accepted writes). The provider answers each request by a +plan the check hands it before the write: accept or reject, at once or held +until a `release`. Publications compare row values; a publication of the same +row values, such as core's `$hasPendingWrites` update when a write is +confirmed, counts as no change here (the virtual-row-field owner judges it). +Reach: each of insert, update, and delete accepted, a rejected write, a held +write, a write queued behind a held one, a held write rejected, a commit by +another client, and a quiescent cut after writes, at least five histories +each per campaign. Calibration: seven reference mutants each fail with their +law (an applied copy after the mutation function settles, a swallowed or +wrapped rejection, a duplicated insert request, writes that do not wait for +the earlier one, a skipped delete, an update that sends only changed fields). +A removed field is an own `undefined` in the authored row (maintainer +decision 2026-11-03, keeping the documented rule in +`packages/db/tests/proxy-detachment-contract-oracle.test.ts`); the core owner +`optimistic-history-oracle` now generates that removal (an optional `d`, set +and then removed with `delete draft.d`), compares rows by own fields, and +counts its reach (`removedFieldEdits`). Limits: one pending mutation per +key; commits by another client only while no mutation is pending (a commit +held behind a pending mutation is core's MUT-5); no manual transactions, +confirmation through sync (MUT-7), schema validation, or an update that +changes no field; no published adapter driver yet. + diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 189786234c..7730f49c23 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -534,3 +534,13 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the replay, or skip the check on the React path, are rejected. A fixed-seed campaign now runs the React client. - **Size:** convex-db-collection +26 gzip bytes for LIFE-10. +- **2026-11-03, mutations family, reference adapter.** The family's oracle + and reference adapter are GREEN on both campaigns, with every declared + reach state met, and seven mutants each rejected with their law. + - **Oracle changes this run exposed:** a publication of unchanged row values + (core's `$hasPendingWrites` update) is not a change of row values; a + subscription forwards changes only for keys it sent, so the recorder + starts from the initial state. + - **Maintainer decision:** a removed field stays an own `undefined`, the + documented rule; the core owner `optimistic-history-oracle` now generates + field removal. diff --git a/packages/db-adapter-conformance/src/eager-live-source-oracle.ts b/packages/db-adapter-conformance/src/eager-live-source-oracle.ts index ff802aeed4..87607dff3e 100644 --- a/packages/db-adapter-conformance/src/eager-live-source-oracle.ts +++ b/packages/db-adapter-conformance/src/eager-live-source-oracle.ts @@ -1870,7 +1870,8 @@ export class ConformanceViolation extends Error { readonly step: number, readonly expected: unknown, readonly actual: unknown, - readonly history: EagerLiveSourceHistory, + // The failing history, of whichever family's grammar. + readonly history: unknown, ) { super( `${law} violated at step ${step} of ${show(history)}: ` + diff --git a/packages/db-adapter-conformance/src/index.ts b/packages/db-adapter-conformance/src/index.ts index 1d1b6a2c74..61b7445ebe 100644 --- a/packages/db-adapter-conformance/src/index.ts +++ b/packages/db-adapter-conformance/src/index.ts @@ -34,3 +34,26 @@ export { toSqlRow, } from './observed-view' export type { AdapterCollection } from './observed-view' +export { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStates, + mutationReachStatesFor, +} from './mutation-oracle' +export type { + MutationCapability, + MutationCheckOptions, + MutationDriver, + MutationFixture, + MutationHistory, + MutationProvider, + MutationReachCounter, + MutationReachState, + MutationReceipt, + MutationStep, + MutationWriter, + WritePlan, + WriteRequest, +} from './mutation-oracle' diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts new file mode 100644 index 0000000000..2d8e3b68d6 --- /dev/null +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -0,0 +1,959 @@ +/** + * # Does a sync adapter persist an application's mutations faithfully? + * + * An application writes to a Collection with `insert`, `update`, and + * `delete`. Each write is an optimistic transaction: core overlays its + * optimistic state on the applied synced rows at once, and the adapter's + * mutation function persists it to the provider. This oracle checks the laws + * that every adapter which persists its own writes owes, whatever its + * provider: + * + * - **MUT-1.** While a mutation function runs, the public rows show its + * optimistic state. When it fulfills, no publication shows the earlier + * synced value: the optimistic row drops in the same publication that + * applies the sync transaction carrying the provider's copy of the write. + * A one-row write publishes exactly one change of row values. Core also + * publishes an update when only a virtual row field changes, such as + * `$hasPendingWrites` when the write is confirmed; the check compares row + * values only, so such a publication counts as no change here, and the + * virtual-row-field owner (`packages/db/tests/query/virtual-row-fields-oracle.test.ts`) + * judges it. + * - **MUT-2.** When the provider rejects a write, `when('settled')` rejects + * with the provider's error (or, under `matchesFailure`, the value the driver's + * relation names), the optimistic state drops in one publication that is + * exactly its inverse, and other pending mutations stay. + * - **MUT-13, MUT-3, MUT-8.** Each write reaches the provider as exactly one + * request of its type for its key, carrying the authored row (an update + * sends the complete authored row). + * - **MUT-10.** Writes are ordered per Collection: a write submitted while an + * earlier one is pending reaches the provider only after the earlier one + * is decided, accepted or rejected. + * - **FID-5.** At a quiescent cut the public rows, the exposed authoritative + * base (`collection.base`), and the provider's rows are identical, and all + * three equal the model: the provider's starting rows with every accepted + * write folded in author order, plus commits from other clients. A + * rejected write leaves no effect in the provider's rows. + * - **HEALTH-5.** No history causes an unhandled rejection or an undeclared + * error- or warning-level console diagnostic. + * + * Authority: the adapter conformance law set (issue #2118), section 9 and + * FID-5; the optimistic state, held optimistic row, and accepted sync + * transaction terms in `docs/contributing/glossary.md`. Core owns joining a + * transaction (MUT-4) and holding synced commits behind pending optimistic + * state (MUT-5) in `packages/db/tests/optimistic-history-oracle.ts`. + * + * Model terms. A **quiescent cut** is model-only: no mutation function is + * pending, the provider has delivered every change, and every committed sync + * transaction has applied. A **write request** is one call the adapter makes + * to the provider for one mutation. A **plan** is the provider's answer to the + * next write request, which the check hands the provider before the write: it + * accepts or rejects, at once or after a `release`. + * + * Boundary. The oracle checks auto-transaction writes (`collection.insert`, + * `update`, `delete`) on a ready Collection whose adapter persists them + * itself. It does not check manual transactions, confirmation through sync + * (MUT-7, a later capability), schema validation, or a mutation on a key that + * already has a pending mutation. Commits from other clients arrive only + * while no mutation is pending; a commit held behind a pending mutation is + * core's MUT-5. + */ +import fc from 'fast-check' +import { ConformanceViolation } from './eager-live-source-oracle' +import type { + ConformanceRow, + ObservedCollection, + ProviderChange, +} from './eager-live-source-oracle' + +// --------------------------------------------------------------------------- +// Driver contract +// --------------------------------------------------------------------------- + +/** Optional capabilities a driver declares; the grammar uses only these. */ +export type MutationCapability = + /** The provider can reject a write request (MUT-2). */ + | `write-rejection` + /** The provider can hold a write request until `release` (MUT-1, MUT-10). */ + | `held-writes` + /** The provider can commit a change made by another client. */ + | `foreign-commits` + +/** The optimistic transaction a write call returns. */ +export interface MutationReceipt { + readonly state: string + /** Settles when the mutation function settles, with its outcome. */ + when: (state: `settled`) => Promise +} + +/** The driver's writes, through the adapter's Collection. */ +export interface MutationWriter { + insert: (row: ConformanceRow) => MutationReceipt + /** Replace the held row's fields with `row`'s: a complete authored row. */ + update: (row: ConformanceRow) => MutationReceipt + delete: (id: number) => MutationReceipt +} + +/** One write request the adapter made to the provider. */ +export interface WriteRequest { + readonly type: `insert` | `update` | `delete` + readonly id: number + /** The row the request carries; absent for a delete. */ + readonly row?: ConformanceRow +} + +/** The provider's answer to the next write request. */ +export interface WritePlan { + readonly outcome: `accept` | `reject` + /** Answer only at `release`, so the write stays pending until then. */ + readonly held: boolean + /** The error a rejection answers with. */ + readonly error: Error +} + +/** + * The provider a driver controls. It answers write requests by plan, in the + * order they arrive, and delivers every accepted write and every commit from + * another client through the adapter's sync subscription, as a provider's + * change feed does. It records every request; it must not merge, reorder, or + * drop them. + */ +export interface MutationProvider { + /** Queue the answer to the next write request that arrives. */ + plan: (plan: WritePlan) => void + /** Answer the oldest held request by its plan. */ + release: () => void | Promise + /** Commit a change made by another client (`foreign-commits`). */ + foreignCommit?: (change: ProviderChange) => void | Promise + /** The provider's committed rows. */ + rows: () => + ReadonlyArray | Promise> + /** Every write request so far, in arrival order. */ + requests: () => ReadonlyArray +} + +export interface MutationFixture { + /** The adapter's Collection, read as conformance rows. */ + collection: ObservedCollection + /** The exposed authoritative base (`collection.base`) as conformance rows. */ + base: () => Iterable + writer: MutationWriter + provider: MutationProvider + /** The adapter's propagation step, awaited before every comparison. */ + settle?: () => Promise + dispose?: () => void | Promise +} + +export interface MutationDriver { + readonly adapter: string + readonly capabilities: ReadonlySet + /** + * When the adapter reports its own error for a provider rejection, the + * relation that names the value owed (MUT-2's named error). Without it, + * `when('settled')` must reject with the provider's error by identity. + */ + readonly matchesFailure?: (reported: unknown, failure: Error) => boolean + /** Console diagnostics the adapter declares (HEALTH-5). */ + readonly declaredDiagnostic?: ( + level: `error` | `warn`, + args: ReadonlyArray, + ) => boolean + /** A ready-to-preload fixture whose provider starts with `seed`. */ + create: ( + seed: ReadonlyArray, + ) => MutationFixture | Promise +} + +// --------------------------------------------------------------------------- +// History grammar +// --------------------------------------------------------------------------- + +/** + * A history starts with the provider's seed rows, preloads the Collection, + * and applies these steps. The model decides which steps are legal now; an + * illegal step is skipped and counted, so a list stays meaningful after + * shrinking. + * + * - `write`: an insert of a key the public rows lack, or an update or delete + * of a key they hold, with no other pending mutation on that key. An update + * that changes no field is not generated. An update without `note` of a + * row that holds one removes the field. Its plan rejects only with + * `write-rejection` and is held only with `held-writes`. + * - `release`: the provider answers the oldest held request (`held-writes`). + * - `foreign`: another client commits an upsert or a delete while no + * mutation is pending (`foreign-commits`). + * + * After the last step the check releases every held request, reaching a + * quiescent cut. Keys range over 1..4 and titles over three values, so writes + * revisit keys; a row carries `note` half the time, so an update can add or + * remove a field. + */ +export type MutationStep = + | { + readonly kind: `write` + readonly type: `insert` | `update` | `delete` + readonly id: number + readonly title: string + readonly note?: string + readonly outcome: `accept` | `reject` + readonly held: boolean + } + | { readonly kind: `release` } + | { readonly kind: `foreign`; readonly change: ProviderChange } + +export interface MutationHistory { + readonly seed: ReadonlyArray + readonly steps: ReadonlyArray +} + +const rowArb: fc.Arbitrary = fc.record( + { + id: fc.integer({ min: 1, max: 4 }), + title: fc.constantFrom(`a`, `b`, `c`), + note: fc.constantFrom(`x`, `y`), + }, + { requiredKeys: [`id`, `title`] }, +) + +/** Generates histories whose steps use only the driver's capabilities. */ +export function mutationHistoryArb( + domain: Pick, +): fc.Arbitrary { + const { capabilities } = domain + const write = fc + .record({ + type: fc.constantFrom( + `insert` as const, + `update` as const, + `delete` as const, + ), + row: rowArb, + outcome: capabilities.has(`write-rejection`) + ? fc.constantFrom( + `accept` as const, + `accept` as const, + `reject` as const, + ) + : fc.constant(`accept` as const), + held: capabilities.has(`held-writes`) ? fc.boolean() : fc.constant(false), + }) + .map(({ type, row, outcome, held }): MutationStep => ({ + kind: `write`, + type, + id: row.id, + title: row.title, + ...(row.note === undefined ? {} : { note: row.note }), + outcome, + held, + })) + const steps: Array<{ + weight: number + arbitrary: fc.Arbitrary + }> = [{ weight: 6, arbitrary: write }] + if (capabilities.has(`held-writes`)) { + steps.push({ + weight: 3, + arbitrary: fc.constant({ kind: `release` as const }), + }) + } + if (capabilities.has(`foreign-commits`)) { + steps.push({ + weight: 2, + arbitrary: fc.oneof( + rowArb.map((row): MutationStep => ({ + kind: `foreign`, + change: { type: `upsert`, row }, + })), + fc.integer({ min: 1, max: 4 }).map((id): MutationStep => ({ + kind: `foreign`, + change: { type: `delete`, id }, + })), + ), + }) + } + return fc.record({ + seed: fc.uniqueArray(rowArb, { selector: (row) => row.id, maxLength: 3 }), + steps: fc.array(fc.oneof(...steps), { minLength: 1, maxLength: 10 }), + }) +} + +// --------------------------------------------------------------------------- +// Independent model +// --------------------------------------------------------------------------- + +/** + * A row as the model compares it: its title, and its note or the note's + * absence. Fields compare by own property, so an own `undefined` note differs + * from an absent one (FID-1). + */ +function rowValue(row: ConformanceRow): string { + return JSON.stringify([ + row.title, + Object.hasOwn(row, `note`) ? [row.note ?? null] : [], + ]) +} + +type RowState = ReadonlyArray<[number, string]> + +function rowState(rows: ReadonlyMap): RowState { + return [...rows].sort(([a], [b]) => a - b) +} + +function rowsOf(rows: Iterable): RowState { + return rowState(new Map([...rows].map((row) => [row.id, rowValue(row)]))) +} + +function sameRows(a: RowState, b: RowState): boolean { + return ( + a.length === b.length && + a.every( + ([id, value], index) => b[index]![0] === id && b[index]![1] === value, + ) + ) +} + +/** A mutation whose plan has not decided it yet. */ +interface PendingWrite { + readonly index: number + readonly type: `insert` | `update` | `delete` + readonly id: number + /** The authored row's model value; absent for a delete. */ + readonly value?: string + readonly plan: WritePlan + requested: boolean +} + +type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } + +/** + * The model holds only what a legal next step can observe: + * + * - `provider` is the provider's committed rows: the seed, every accepted + * write, and every commit from another client, in commit order. + * - `pending` lists the mutations not yet decided, in author order. MUT-10 + * orders their requests: the first is requested at once, and each later + * one only after every earlier one is decided. + * - `requests` is every write request MUT-8 and MUT-10 allow so far. + * - `settlements` is each write's `when('settled')` settlement. + * + * The public rows are `provider` with `pending` folded on in author order: + * core shows a mutation's optimistic state until its mutation function + * settles, and a fulfilled write's provider copy replaces it in the same + * publication. Nothing else is state: a decided write is either in + * `provider` or gone. + */ +interface MutationModel { + provider: Map + pending: Array + requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> + settlements: Array +} + +function initialModel(seed: ReadonlyArray): MutationModel { + return { + provider: new Map(seed.map((row) => [row.id, rowValue(row)])), + pending: [], + requests: [], + settlements: [], + } +} + +/** + * The row a write authors. An update that drops `note` from a row holding one + * removes the field, which the update API represents as an own `undefined` + * field (the documented rule in packages/db/tests/proxy-detachment-contract-oracle.test.ts; + * maintainer decision 2026-11-03). + */ +function authoredValue( + step: Extract, + held: string | undefined, +): string { + const removesNote = + step.type === `update` && + step.note === undefined && + held !== undefined && + (JSON.parse(held) as [string, Array])[1].length > 0 + return rowValue({ + id: step.id, + title: step.title, + ...(step.note === undefined + ? removesNote + ? { note: undefined } + : {} + : { note: step.note }), + }) +} + +function fold( + rows: ReadonlyMap, + writes: ReadonlyArray, +): Map { + const folded = new Map(rows) + for (const write of writes) { + if (write.type === `delete`) folded.delete(write.id) + else folded.set(write.id, write.value!) + } + return folded +} + +/** The public rows the model predicts. */ +function visible(state: MutationModel): RowState { + return rowState(fold(state.provider, state.pending)) +} + +/** Decide a write by its plan: an accepted one enters the provider's rows. */ +function decide(state: MutationModel, write: PendingWrite): void { + if (write.plan.outcome === `accept`) { + if (write.type === `delete`) state.provider.delete(write.id) + else state.provider.set(write.id, write.value!) + state.settlements[write.index] = `fulfilled` + } else { + state.settlements[write.index] = { rejected: write.plan.error } + } + state.pending.splice(state.pending.indexOf(write), 1) +} + +/** + * MUT-10: request the first pending mutation if it has not been; a request + * answered at once is decided at once, which admits the next. Returns the + * public rows after each decision, for the publication check. + */ +function advance(state: MutationModel): Array { + const states: Array = [] + for (;;) { + const first = state.pending[0] + if (!first) return states + if (!first.requested) { + first.requested = true + state.requests.push({ + type: first.type, + id: first.id, + ...(first.value === undefined ? {} : { value: first.value }), + }) + } + if (first.plan.held) return states + decide(state, first) + states.push(visible(state)) + } +} + +/** What a step did, for reach and law names. */ +type StepDone = + | { + kind: `write` + type: PendingWrite[`type`] + outcome: WritePlan[`outcome`] + held: boolean + queued: boolean + } + | { kind: `release`; outcome: WritePlan[`outcome`] } + | { kind: `foreign` } + +/** + * Applies one step to the model. Returns undefined when the step is not + * legal now, without moving the model, and otherwise what it did and the + * public rows after each change it publishes, in order. + */ +function stepModel( + state: MutationModel, + step: MutationStep, + plan: (step: Extract) => WritePlan, +): { done: StepDone; published: Array } | undefined { + switch (step.kind) { + case `write`: { + const held = new Map(visible(state)) + const busy = state.pending.some((write) => write.id === step.id) + const legal = + step.type === `insert` ? !held.has(step.id) : held.has(step.id) + const value = + step.type === `delete` + ? undefined + : authoredValue(step, held.get(step.id)) + // An update that changes no field records no mutation in core; no law + // fixes its persistence, so the grammar leaves it out. + const noop = step.type === `update` && held.get(step.id) === value + if (busy || !legal || noop) return undefined + const queued = state.pending.length > 0 + const write: PendingWrite = { + index: state.settlements.length, + type: step.type, + id: step.id, + ...(value === undefined ? {} : { value }), + plan: plan(step), + requested: false, + } + state.settlements.push(`pending`) + state.pending.push(write) + // The optimistic state publishes when the write call returns. + const published = [visible(state), ...advance(state)] + return { + done: { + kind: `write`, + type: step.type, + outcome: step.outcome, + held: step.held, + queued, + }, + published, + } + } + case `release`: { + const first = state.pending[0] + if (!first?.requested || !first.plan.held) return undefined + decide(state, first) + const published = [visible(state), ...advance(state)] + return { + done: { kind: `release`, outcome: first.plan.outcome }, + published, + } + } + case `foreign`: { + if (state.pending.length > 0) return undefined + if (step.change.type === `upsert`) + state.provider.set(step.change.row.id, rowValue(step.change.row)) + else state.provider.delete(step.change.id) + return { done: { kind: `foreign` }, published: [visible(state)] } + } + } +} + +/** + * The publications a step owes: each change of the public rows, in order. A + * change that leaves the rows as they were publishes nothing, as an accepted + * write whose provider copy equals its optimistic state does. + */ +function owedPublications( + before: RowState, + states: ReadonlyArray, +): Array { + const owed: Array = [] + let last = before + for (const next of states) { + if (!sameRows(next, last)) owed.push(next) + last = next + } + return owed +} + +// --------------------------------------------------------------------------- +// Refinement check +// --------------------------------------------------------------------------- + +/** Declared states whose laws need generated histories to reach them. */ +export const mutationReachStates = [ + `insert-accepted`, + `update-accepted`, + `delete-accepted`, + `write-rejected`, + `held-write-pending`, + `write-queued-behind-held`, + `held-write-rejected`, + `foreign-commit`, + `quiescent-after-writes`, +] as const + +export type MutationReachState = (typeof mutationReachStates)[number] + +const capabilityOfState: Partial< + Record +> = { + 'write-rejected': `write-rejection`, + 'held-write-pending': `held-writes`, + 'write-queued-behind-held': `held-writes`, + 'held-write-rejected': `held-writes`, + 'foreign-commit': `foreign-commits`, +} + +/** The declared states a driver's capabilities let histories reach. */ +export function mutationReachStatesFor( + domain: Pick, +): Array { + return mutationReachStates.filter((state) => { + const capability = capabilityOfState[state] + if (state === `held-write-rejected`) + return ( + domain.capabilities.has(`held-writes`) && + domain.capabilities.has(`write-rejection`) + ) + return capability === undefined || domain.capabilities.has(capability) + }) +} + +type Settlement = `pending` | `fulfilled` | { rejected: unknown } + +function recordSettlement(promise: Promise): { current: Settlement } { + const record: { current: Settlement } = { current: `pending` } + promise.then( + () => { + record.current = `fulfilled` + }, + (error: unknown) => { + record.current = { rejected: error } + }, + ) + return record +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +/** + * The checkpoint premise: an adapter answers its provider and applies the + * provider's change within a few event-loop turns, or within the + * propagation step its fixture declares. A slower adapter shows as a + * mismatch at this cut, not as a pass. + */ +async function settle(fixture: MutationFixture): Promise { + for (let turn = 0; turn < 3; turn++) await sleep(0) + await fixture.settle?.() +} + +/** + * Records the public rows after each change publication (MUT-1, MUT-2). A + * subscription forwards changes only for keys it has sent, so it starts with + * the initial state, which is the baseline and not a publication. + */ +function recordPublications(collection: ObservedCollection) { + const rows = new Map() + let inconsistent = false + let initial = true + let publications: Array = [] + const subscription = collection.subscribeChanges( + (changes) => { + if (initial) { + initial = false + for (const change of changes) + rows.set(change.key, rowValue(change.value)) + return + } + if (changes.length === 0) return + for (const change of changes) { + if (change.type === `delete`) { + if (!rows.delete(change.key)) inconsistent = true + } else { + if ((change.type === `insert`) === rows.has(change.key)) + inconsistent = true + rows.set(change.key, rowValue(change.value)) + } + } + publications.push( + inconsistent ? [[-1, `inconsistent change events`]] : rowState(rows), + ) + }, + { includeInitialState: true }, + ) + // An empty Collection's initial state is no callback at all. + initial = false + return { + take: () => { + const taken = publications + publications = [] + return taken + }, + stop: () => subscription.unsubscribe(), + } +} + +function watchDetachedFailures(driver: MutationDriver) { + const found: Array = [] + const onRejection = (reason: unknown) => { + found.push(`unhandled rejection: ${String(reason)}`) + } + const hasProcess = typeof process !== `undefined` + if (hasProcess) process.on(`unhandledRejection`, onRejection) + const original = { error: console.error, warn: console.warn } + for (const level of [`error`, `warn`] as const) { + console[level] = (...args: Array) => { + if (!driver.declaredDiagnostic?.(level, args)) { + found.push(`console.${level}: ${args.map(String).join(` `)}`) + } + } + } + return { + found: () => found, + stop: () => { + if (hasProcess) process.off(`unhandledRejection`, onRejection) + console.error = original.error + console.warn = original.warn + }, + } +} + +export interface MutationCheckOptions { + /** Called when the history enters a declared state (ORC-015). */ + reach?: (state: MutationReachState) => void +} + +/** + * Runs one history through a driver and compares every checkpoint with the + * model. Throws a `ConformanceViolation` naming the first violated law. The + * fixture is always cleaned up; a cleanup failure after a violation is kept + * as a secondary error (ORC-010). + */ +export async function checkMutationHistory( + driver: MutationDriver, + history: MutationHistory, + options: MutationCheckOptions = {}, +): Promise<{ accepted: number; skipped: number }> { + const reach = options.reach ?? (() => {}) + const detached = watchDetachedFailures(driver) + let created: MutationFixture | undefined + try { + created = await driver.create(structuredClone(history.seed)) + } catch (error) { + detached.stop() + throw error + } + const fixture = created + const { collection, provider, writer } = fixture + const state = initialModel(history.seed) + const receipts: Array<{ current: Settlement }> = [] + const failures: Array = [] + let accepted = 0 + let skipped = 0 + let wrote = 0 + let primary: unknown + let publications: ReturnType | undefined + + const steps = history.steps + const violation = (law: string, step: number, want: unknown, got: unknown) => + new ConformanceViolation(law, step, want, got, history) + const matches = (reported: unknown, failure: Error) => + driver.matchesFailure + ? driver.matchesFailure(reported, failure) + : reported === failure + + /** MUT-1, MUT-2, MUT-8, MUT-10, MUT-13, FID-5, HEALTH-5 at one cut. */ + const compare = async (stepIndex: number) => { + await settle(fixture) + const found = detached.found() + if (found.length > 0) + throw violation(`HEALTH-5`, stepIndex, `no detached failure`, found) + // The provider's request log first: it observes MUT-8, MUT-10, and + // MUT-13 directly, where the rows show their effect only later. + const requests = provider.requests().map((request) => ({ + type: request.type, + id: request.id, + ...(request.row === undefined ? {} : { value: rowValue(request.row) }), + })) + if (JSON.stringify(requests) !== JSON.stringify(state.requests)) + throw violation( + requests.length > state.requests.length + ? `MUT-10/MUT-13 (provider requests)` + : `MUT-8 (provider requests)`, + stepIndex, + state.requests, + requests, + ) + const quiescent = state.pending.length === 0 + const rows = rowsOf(collection.values()) + if (!sameRows(rows, visible(state))) + throw violation( + quiescent ? `FID-5 (public rows)` : `MUT-1 (public rows)`, + stepIndex, + visible(state), + rows, + ) + state.settlements.forEach((expected, index) => { + const got = receipts[index]!.current + const ok = + expected === `pending` || expected === `fulfilled` + ? got === expected + : typeof got === `object` && matches(got.rejected, expected.rejected) + if (!ok) + throw violation( + typeof expected === `object` + ? `MUT-2 (settlement)` + : `MUT-13 (settlement)`, + stepIndex, + typeof expected === `object` ? expected.rejected.message : expected, + typeof got === `object` ? String(got.rejected) : got, + ) + }) + if (!quiescent) return + // FID-5 at a quiescent cut: public rows, base, and provider are one. + const base = rowsOf(fixture.base()) + if (!sameRows(base, rowState(state.provider))) + throw violation(`FID-5 (base)`, stepIndex, rowState(state.provider), base) + const providerRows = rowsOf(await provider.rows()) + if (!sameRows(providerRows, rowState(state.provider))) + throw violation( + `FID-5 (provider rows)`, + stepIndex, + rowState(state.provider), + providerRows, + ) + if (wrote > 0) reach(`quiescent-after-writes`) + } + + let published = visible(state) + const checkPublications = ( + stepIndex: number, + owed: ReadonlyArray, + law: string, + ) => { + // Each change of row values, in order; a publication of the same row + // values (a virtual row field change) is not one. + const got: Array = [] + for (const rows of publications!.take()) { + if (!sameRows(rows, got.at(-1) ?? published)) got.push(rows) + } + published = got.at(-1) ?? published + const ok = + got.length === owed.length && + got.every((rows, index) => sameRows(rows, owed[index]!)) + if (!ok) throw violation(`${law} (publication)`, stepIndex, owed, got) + } + + // The plan the model decided the last write by; the provider gets it too. + let lastPlan: WritePlan | undefined + const planFor = ( + step: Extract, + ): WritePlan => { + const error = new Error(`conformance write rejected ${failures.length}`) + failures.push(error) + lastPlan = { outcome: step.outcome, held: step.held, error } + return lastPlan + } + + try { + await collection.preload() + publications = recordPublications(collection) + await compare(-1) + for (const [index, step] of steps.entries()) { + const before = visible(state) + const result = stepModel(state, step, planFor) + if (result === undefined) { + skipped++ + continue + } + accepted++ + const { done } = result + let law = `MUT-1` + switch (done.kind) { + case `write`: { + const write = step as Extract + provider.plan(lastPlan!) + const row: ConformanceRow = { + id: write.id, + title: write.title, + ...(write.note === undefined ? {} : { note: write.note }), + } + const receipt = + write.type === `insert` + ? writer.insert(row) + : write.type === `update` + ? writer.update(row) + : writer.delete(write.id) + receipts.push(recordSettlement(receipt.when(`settled`))) + wrote++ + if (write.held) reach(`held-write-pending`) + if (done.queued) reach(`write-queued-behind-held`) + if (write.outcome === `reject`) reach(`write-rejected`) + else if (!write.held) reach(`${write.type}-accepted` as const) + if (write.outcome === `reject` && !write.held) law = `MUT-2` + break + } + case `release`: + await provider.release() + if (done.outcome === `reject`) { + reach(`held-write-rejected`) + law = `MUT-2` + } + break + case `foreign`: + await provider.foreignCommit!( + (step as Extract).change, + ) + reach(`foreign-commit`) + break + } + await compare(index) + checkPublications(index, owedPublications(before, result.published), law) + } + // Answer every held request, reaching a quiescent cut (FID-5). + while (state.pending.length > 0) { + const before = visible(state) + const result = stepModel(state, { kind: `release` }, planFor) + if (result === undefined) break + await provider.release() + await compare(steps.length) + checkPublications( + steps.length, + owedPublications(before, result.published), + `MUT-1`, + ) + } + } catch (error) { + primary = error + } + + const cleanupErrors: Array = [] + try { + publications?.stop() + await collection.cleanup() + await settle(fixture) + if (primary === undefined && detached.found().length > 0) { + primary = violation( + `HEALTH-5`, + steps.length, + `no detached failure`, + detached.found(), + ) + } + } catch (error) { + cleanupErrors.push(error) + } finally { + detached.stop() + } + try { + await fixture.dispose?.() + } catch (error) { + cleanupErrors.push(error) + } + if (primary !== undefined) { + if (cleanupErrors.length === 0) throw primary + throw new AggregateError(cleanupErrors, `conformance violation`, { + cause: primary, + }) + } + if (cleanupErrors.length > 0) throw cleanupErrors[0] + return { accepted, skipped } +} + +/** A reach counter, such as the one `createOracleReach` returns. */ +export interface MutationReachCounter { + beginHistory: () => void + reach: (state: MutationReachState) => void +} + +/** + * The generated property every driver runs. Shrinking keeps the first + * violated law (ORC-010): a smaller history that fails a different law + * counts as passing. + */ +export function mutationProperty( + driver: MutationDriver, + reach?: MutationReachCounter, +) { + let target: string | undefined + return fc.asyncProperty(mutationHistoryArb(driver), async (history) => { + reach?.beginHistory() + try { + await checkMutationHistory(driver, history, { reach: reach?.reach }) + } catch (error) { + const law = + error instanceof ConformanceViolation + ? error.law + : `failure outside the comparison` + target ??= law + if (law === target) throw error + } + }) +} + +// Exported for calibration tests: the model's prediction for a history. +export const mutationModel = { + initial: initialModel, + step: stepModel, + visible, + owedPublications, +} diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts new file mode 100644 index 0000000000..1a53fcd0d5 --- /dev/null +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -0,0 +1,264 @@ +/** + * A reference adapter that persists its own writes, written to conform to the + * mutation laws, and its production driver. It calibrates the check: the + * adapter must pass, and each hostile mutant must be rejected with the law it + * breaks (ORC-006). + * + * The adapter's sync run loads the provider's rows, marks ready, and applies + * the provider's change feed. Its mutation functions send each mutation to + * the provider one write at a time per Collection (MUT-10) and return once + * the provider has answered. An accepted write reaches the change feed before + * the provider answers, so its sync transaction is committed before the + * mutation function settles, and core drops the optimistic row in the same + * publication that applies it (MUT-1). + */ +import { createCollection } from '@tanstack/db' +import { observedView } from '../src' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../src' + +/** Wrong designs the calibration tests inject, one decision each. */ +export type ReferenceMutationMutant = + /** MUT-1: the provider's copy applies after the mutation function settles. */ + | `applies-copy-late` + /** MUT-2: a rejected write resolves, so its optimistic state stays. */ + | `swallows-rejection` + /** MUT-2: a rejection reports a different error value. */ + | `wraps-rejection` + /** MUT-13: an insert reaches the provider twice. */ + | `writes-insert-twice` + /** MUT-10: a write does not wait for the earlier one to be decided. */ + | `writes-concurrently` + /** FID-5: a delete never reaches the provider. */ + | `skips-delete` + /** MUT-8: an update sends only the changed fields. */ + | `sends-changes-only` + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + error: new Error(`unplanned write`), +} + +/** A provider that answers writes by plan and feeds every commit back. */ +export class ControlledMutationProvider implements MutationProvider { + private readonly committed = new Map() + private readonly plans: Array = [] + private readonly held: Array<() => void> = [] + private readonly log: Array = [] + private listener: ((change: ProviderChange) => void) | undefined + + constructor(seed: ReadonlyArray) { + for (const row of seed) this.committed.set(row.id, structuredClone(row)) + } + + // The adapter's side: its change feed, initial read, and writes. + + subscribe(listener: (change: ProviderChange) => void): () => void { + this.listener = listener + return () => { + if (this.listener === listener) this.listener = undefined + } + } + + snapshot(): Array { + return [...this.committed.values()].map((row) => structuredClone(row)) + } + + write(request: WriteRequest): Promise { + this.log.push(structuredClone(request)) + const plan = this.plans.shift() ?? acceptAtOnce + return new Promise((resolve, reject) => { + const answer = () => { + if (plan.outcome === `reject`) { + reject(plan.error) + return + } + this.commit( + request.type === `delete` + ? { type: `delete`, id: request.id } + : { type: `upsert`, row: request.row! }, + ) + resolve() + } + if (plan.held) this.held.push(answer) + else answer() + }) + } + + /** A write replaces the whole row, as a complete-row API does. */ + private commit(change: ProviderChange): void { + if (change.type === `delete`) this.committed.delete(change.id) + else this.committed.set(change.row.id, structuredClone(change.row)) + this.listener?.(structuredClone(change)) + } + + // The oracle's side. + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + this.held.shift()?.() + } + + foreignCommit(change: ProviderChange): void { + this.commit(change) + } + + rows(): ReadonlyArray { + return this.snapshot() + } + + requests(): ReadonlyArray { + return this.log + } +} + +let sequence = 0 + +/** Builds the reference adapter's Collection over `provider`. */ +function referenceCollection( + provider: ControlledMutationProvider, + mutant: ReferenceMutationMutant | undefined, +) { + // MUT-10: each write waits for the earlier one to be decided. + let turn: Promise = Promise.resolve() + const persist = (send: () => Promise): Promise => { + if (mutant === `writes-concurrently`) return send() + const run = turn.then(send) + turn = run.catch(() => undefined) + return run + } + const answer = async (sent: Promise): Promise => { + try { + await sent + } catch (error) { + if (mutant === `swallows-rejection`) return + if (mutant === `wraps-rejection`) + throw new Error(`wrapped`, { cause: error }) + throw error + } + } + const collection = createCollection({ + id: `reference-mutations-${++sequence}`, + getKey: (row) => row.id, + sync: { + rowUpdateMode: `full`, + sync: ({ begin, write, commit, markReady }) => { + const keys = new Set() + const apply = (change: ProviderChange) => { + begin() + if (change.type === `delete`) { + if (keys.delete(change.id)) + write({ type: `delete`, key: change.id }) + } else { + write({ + type: keys.has(change.row.id) ? `update` : `insert`, + value: change.row, + }) + keys.add(change.row.id) + } + void commit() + } + begin() + for (const row of provider.snapshot()) { + write({ type: `insert`, value: row }) + keys.add(row.id) + } + void commit() + markReady() + return provider.subscribe((change) => { + if (mutant === `applies-copy-late`) setTimeout(() => apply(change), 0) + else apply(change) + }) + }, + }, + onInsert: ({ transaction }) => + answer( + persist(async () => { + for (const mutation of transaction.mutations) { + const request = { + type: `insert` as const, + id: mutation.key, + row: structuredClone(mutation.modified), + } + await provider.write(request) + if (mutant === `writes-insert-twice`) await provider.write(request) + } + }), + ), + onUpdate: ({ transaction }) => + answer( + persist(async () => { + for (const mutation of transaction.mutations) { + await provider.write({ + type: `update`, + id: mutation.key, + row: + mutant === `sends-changes-only` + ? ({ + id: mutation.key, + ...mutation.changes, + } as ConformanceRow) + : structuredClone(mutation.modified), + }) + } + }), + ), + onDelete: ({ transaction }) => + answer( + persist(async () => { + if (mutant === `skips-delete`) return + for (const mutation of transaction.mutations) { + await provider.write({ type: `delete`, id: mutation.key }) + } + }), + ), + }) + return collection +} + +/** A production driver for the reference adapter, optionally mutated. */ +export function referenceMutationDriver( + mutant?: ReferenceMutationMutant, +): MutationDriver { + return { + adapter: `reference mutations${mutant ? ` (${mutant})` : ``}`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + ]), + create(seed): MutationFixture { + const provider = new ControlledMutationProvider(seed) + const collection = referenceCollection(provider, mutant) + return { + collection: observedView(collection, { + row: (row) => row, + key: (key) => key as number, + }), + base: () => collection.base.values(), + writer: { + insert: (row) => collection.insert(row), + update: (row) => + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note + else delete draft.note + }), + delete: (id) => collection.delete(id), + }, + provider, + } + }, + } +} diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts new file mode 100644 index 0000000000..4ce1b36846 --- /dev/null +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -0,0 +1,99 @@ +import { fc } from '@fast-check/vitest' +import { describe, expect, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + ConformanceViolation, + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../src' +import { referenceMutationDriver } from './reference-mutation-adapter' +import type { ReferenceMutationMutant } from './reference-mutation-adapter' + +/** + * # The mutation oracle on a conforming adapter + * + * The shared oracle in `src/mutation-oracle.ts` states the laws, the model, + * the history grammar, and the check. This file supplies one production + * driver: the reference adapter in `reference-mutation-adapter.ts`, which + * persists its own writes and declares every mutation capability. A green + * run here shows that the check accepts a conforming adapter; it says nothing + * about a published adapter, which needs its own driver. + * + * - **Campaigns.** The same property, grammar, and run budget run with a + * fixed seed and without one (ORC-007). Each fails when fewer than five + * histories reach a declared state (ORC-015). + * - **Calibration.** Each hostile mutant of the reference adapter breaks one + * law, and the fixed campaign must reject it with that law (ORC-006). + */ + +const property = `adapter-conformance.mutations.reference` +const driver = referenceMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +const campaignTimeoutMs = 120_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: reference adapter`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + const mutants: Array<[ReferenceMutationMutant, string | RegExp]> = [ + [`applies-copy-late`, `MUT-1`], + [`swallows-rejection`, `MUT-2`], + [`wraps-rejection`, `MUT-2`], + [`writes-insert-twice`, `MUT-10/MUT-13`], + [`writes-concurrently`, `MUT-10/MUT-13`], + [`skips-delete`, /MUT-8|FID-5/], + [`sends-changes-only`, `MUT-8`], + ] + oracleReplayTest(it).each(mutants)( + `rejects the %s mutant with %s`, + async (mutant, law) => { + const result = await fc.check( + mutationProperty(referenceMutationDriver(mutant)), + { seed: 2118, numRuns: runs }, + ) + expect(result.failed).toBe(true) + expect(result.errorInstance).toBeInstanceOf(ConformanceViolation) + expect((result.errorInstance as ConformanceViolation).law).toMatch(law) + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 5238c61180..5a79cf9bd1 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -205,6 +205,7 @@ const staticOracleProperties = [ `sqlite-persistence.request-local-failure-buffered-source`, `sqlite-persistence.optimistic-history`, `adapter-conformance.eager-live-source.reference`, + `adapter-conformance.mutations.reference`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index e1d6db09ec..fb5abda15d 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -787,6 +787,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.eager-live-source`, `convex`, ], + [ + `db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `reference`, + ], ] const owners = new Map() @@ -888,6 +893,10 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), + [ + `adapter-conformance.mutations.reference`, + `conforms on a random campaign or a replayed seed`, + ], [ `query-db.refetch-held-page-end`, `generated same-key page shifts.*replay campaign`, From 8977b7b6eb3d73a0608a540dfa47b33c41b04662 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 08:24:45 -0700 Subject: [PATCH 04/28] test(db-adapter-conformance): model held sync copies and a provider's value domain A JSON provider cannot store an own `undefined`, so a field an update removes is absent from its request and its copy, while core's optimistic row holds it as an own `undefined`. A driver now declares this with `undefinedField` (`own` or `absent`), from FID-1's value domain. With copies that can differ from the authored row, the model must also follow the glossary's accepted sync transaction: an accepted write's copy is held while any mutation is persisting, and the write's held optimistic row stays until it applies. The model now keeps the applied synced rows and the held rows apart from the provider's rows. - Law: MUT-1 and MUT-8 under FID-1's value domain; glossary terms accepted sync transaction and held optimistic row. - Reach: new states `field-removed` and `held-optimistic-row`, at least five histories each, in the reference campaigns and a new JSON-provider campaign. - RED: before the held-row rule, the JSON-provider fixed campaign failed MUT-1 (public rows) on a fulfilled removal behind a pending write. - GREEN: db-adapter-conformance 97 passed. - Mutant: `echoes-authored-row` (the change feed carries the authored row, not the stored copy) is rejected with FID-5. - Limits: the reference provider is the only `absent` calibration. Co-authored-by: Isaac --- .../src/mutation-oracle.ts | 115 ++++++++++++++---- .../tests/reference-mutation-adapter.ts | 43 +++++-- ...reference-mutation-oracle.property.test.ts | 28 ++++- 3 files changed, 154 insertions(+), 32 deletions(-) diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 2d8e3b68d6..5308ebdc9e 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -12,7 +12,11 @@ * optimistic state. When it fulfills, no publication shows the earlier * synced value: the optimistic row drops in the same publication that * applies the sync transaction carrying the provider's copy of the write. - * A one-row write publishes exactly one change of row values. Core also + * While another mutation is persisting, that sync transaction is held and + * the write's optimistic row stays as a held optimistic row until it + * applies (glossary). A one-row write publishes exactly one change of row + * values, and one more when its provider copy differs from the authored + * row, as a JSON provider's does for a removed field. Core also * publishes an update when only a virtual row field changes, such as * `$hasPendingWrites` when the write is confirmed; the check compares row * values only, so such a publication counts as no change here, and the @@ -24,7 +28,10 @@ * exactly its inverse, and other pending mutations stay. * - **MUT-13, MUT-3, MUT-8.** Each write reaches the provider as exactly one * request of its type for its key, carrying the authored row (an update - * sends the complete authored row). + * sends the complete authored row) in the provider's value domain (FID-1): + * a JSON provider has no `undefined`, so a field the update removes is + * absent from its request and its copy, though the optimistic row holds it + * as an own `undefined`. A driver declares this with `undefinedField`. * - **MUT-10.** Writes are ordered per Collection: a write submitted while an * earlier one is pending reaches the provider only after the earlier one * is decided, accepted or rejected. @@ -152,6 +159,14 @@ export interface MutationDriver { * `when('settled')` must reject with the provider's error by identity. */ readonly matchesFailure?: (reported: unknown, failure: Error) => boolean + /** + * How the provider stores an authored field whose value is `undefined`, + * as an update that removes a field authors it. FID-1 parameterizes row + * values by the adapter's value domain: `own` keeps the field as an own + * `undefined`, as a structured clone does; `absent` drops it, as a JSON + * document must, since JSON has no `undefined`. Default `own`. + */ + readonly undefinedField?: `own` | `absent` /** Console diagnostics the adapter declares (HEALTH-5). */ readonly declaredDiagnostic?: ( level: `error` | `warn`, @@ -318,6 +333,8 @@ interface PendingWrite { readonly id: number /** The authored row's model value; absent for a delete. */ readonly value?: string + /** The provider's copy of the authored row; absent for a delete. */ + readonly stored?: string readonly plan: WritePlan requested: boolean } @@ -328,29 +345,51 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } * The model holds only what a legal next step can observe: * * - `provider` is the provider's committed rows: the seed, every accepted - * write, and every commit from another client, in commit order. + * write's stored copy, and every commit from another client, in commit + * order. A write's stored copy is its authored row under the driver's + * `undefinedField` rule. * - `pending` lists the mutations not yet decided, in author order. MUT-10 * orders their requests: the first is requested at once, and each later * one only after every earlier one is decided. * - `requests` is every write request MUT-8 and MUT-10 allow so far. * - `settlements` is each write's `when('settled')` settlement. * - * The public rows are `provider` with `pending` folded on in author order: - * core shows a mutation's optimistic state until its mutation function - * settles, and a fulfilled write's provider copy replaces it in the same - * publication. Nothing else is state: a decided write is either in - * `provider` or gone. + * - `applied` is the synced rows core has applied. An accepted write's + * provider copy reaches core as an accepted sync transaction, which is + * held while any optimistic transaction is persisting, so `applied` takes + * `provider` only when no mutation is pending. + * - `heldRows` lists the accepted writes whose sync transaction is held, in + * author order: each keeps its held optimistic row until that transaction + * applies (glossary: accepted sync transaction, held optimistic row). + * + * The public rows are `applied` with `heldRows` and then `pending` folded on + * in author order: core shows a mutation's optimistic state until its + * mutation function settles, and a fulfilled write's provider copy replaces + * its held optimistic row in the publication that applies it. With a + * structured-clone provider the copy equals the authored row, so only a + * provider whose copy differs, such as a JSON one, shows the hold. */ interface MutationModel { + /** The driver's `undefinedField` declaration. */ + readonly undefinedField: `own` | `absent` provider: Map + applied: Map + heldRows: Array pending: Array requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> settlements: Array } -function initialModel(seed: ReadonlyArray): MutationModel { +function initialModel( + seed: ReadonlyArray, + undefinedField: MutationModel[`undefinedField`] = `own`, +): MutationModel { + const provider = new Map(seed.map((row) => [row.id, rowValue(row)])) return { - provider: new Map(seed.map((row) => [row.id, rowValue(row)])), + undefinedField, + provider, + applied: new Map(provider), + heldRows: [], pending: [], requests: [], settlements: [], @@ -363,16 +402,16 @@ function initialModel(seed: ReadonlyArray): MutationModel { * field (the documented rule in packages/db/tests/proxy-detachment-contract-oracle.test.ts; * maintainer decision 2026-11-03). */ -function authoredValue( +function authoredRow( step: Extract, held: string | undefined, -): string { +): ConformanceRow { const removesNote = step.type === `update` && step.note === undefined && held !== undefined && (JSON.parse(held) as [string, Array])[1].length > 0 - return rowValue({ + return { id: step.id, title: step.title, ...(step.note === undefined @@ -380,7 +419,16 @@ function authoredValue( ? { note: undefined } : {} : { note: step.note }), - }) + } +} + +/** The provider's copy of an authored row under its `undefinedField` rule. */ +function storedRow( + row: ConformanceRow, + undefinedField: MutationModel[`undefinedField`], +): ConformanceRow { + if (undefinedField === `own` || row.note !== undefined) return row + return { id: row.id, title: row.title } } function fold( @@ -397,19 +445,26 @@ function fold( /** The public rows the model predicts. */ function visible(state: MutationModel): RowState { - return rowState(fold(state.provider, state.pending)) + return rowState(fold(fold(state.applied, state.heldRows), state.pending)) } /** Decide a write by its plan: an accepted one enters the provider's rows. */ function decide(state: MutationModel, write: PendingWrite): void { if (write.plan.outcome === `accept`) { if (write.type === `delete`) state.provider.delete(write.id) - else state.provider.set(write.id, write.value!) + else state.provider.set(write.id, write.stored!) + state.heldRows.push(write) state.settlements[write.index] = `fulfilled` } else { state.settlements[write.index] = { rejected: write.plan.error } } state.pending.splice(state.pending.indexOf(write), 1) + // The last persisting transaction settled: every held sync transaction + // applies in its publication. + if (state.pending.length === 0) { + state.applied = new Map(state.provider) + state.heldRows = [] + } } /** @@ -427,7 +482,7 @@ function advance(state: MutationModel): Array { state.requests.push({ type: first.type, id: first.id, - ...(first.value === undefined ? {} : { value: first.value }), + ...(first.stored === undefined ? {} : { value: first.stored }), }) } if (first.plan.held) return states @@ -444,6 +499,8 @@ type StepDone = outcome: WritePlan[`outcome`] held: boolean queued: boolean + /** An update that removes `note`, authored as an own `undefined`. */ + removesField: boolean } | { kind: `release`; outcome: WritePlan[`outcome`] } | { kind: `foreign` } @@ -464,10 +521,11 @@ function stepModel( const busy = state.pending.some((write) => write.id === step.id) const legal = step.type === `insert` ? !held.has(step.id) : held.has(step.id) - const value = + const row = step.type === `delete` ? undefined - : authoredValue(step, held.get(step.id)) + : authoredRow(step, held.get(step.id)) + const value = row && rowValue(row) // An update that changes no field records no mutation in core; no law // fixes its persistence, so the grammar leaves it out. const noop = step.type === `update` && held.get(step.id) === value @@ -477,7 +535,10 @@ function stepModel( index: state.settlements.length, type: step.type, id: step.id, - ...(value === undefined ? {} : { value }), + ...(row && { + value: rowValue(row), + stored: rowValue(storedRow(row, state.undefinedField)), + }), plan: plan(step), requested: false, } @@ -492,6 +553,10 @@ function stepModel( outcome: step.outcome, held: step.held, queued, + removesField: + row !== undefined && + Object.hasOwn(row, `note`) && + row.note === undefined, }, published, } @@ -511,6 +576,7 @@ function stepModel( if (step.change.type === `upsert`) state.provider.set(step.change.row.id, rowValue(step.change.row)) else state.provider.delete(step.change.id) + state.applied = new Map(state.provider) return { done: { kind: `foreign` }, published: [visible(state)] } } } @@ -548,6 +614,8 @@ export const mutationReachStates = [ `write-queued-behind-held`, `held-write-rejected`, `foreign-commit`, + `field-removed`, + `held-optimistic-row`, `quiescent-after-writes`, ] as const @@ -561,6 +629,7 @@ const capabilityOfState: Partial< 'write-queued-behind-held': `held-writes`, 'held-write-rejected': `held-writes`, 'foreign-commit': `foreign-commits`, + 'held-optimistic-row': `held-writes`, } /** The declared states a driver's capabilities let histories reach. */ @@ -704,7 +773,7 @@ export async function checkMutationHistory( } const fixture = created const { collection, provider, writer } = fixture - const state = initialModel(history.seed) + const state = initialModel(history.seed, driver.undefinedField) const receipts: Array<{ current: Settlement }> = [] const failures: Array = [] let accepted = 0 @@ -849,6 +918,8 @@ export async function checkMutationHistory( if (done.queued) reach(`write-queued-behind-held`) if (write.outcome === `reject`) reach(`write-rejected`) else if (!write.held) reach(`${write.type}-accepted` as const) + if (done.removesField && write.outcome === `accept`) + reach(`field-removed`) if (write.outcome === `reject` && !write.held) law = `MUT-2` break } @@ -866,6 +937,7 @@ export async function checkMutationHistory( reach(`foreign-commit`) break } + if (state.heldRows.length > 0) reach(`held-optimistic-row`) await compare(index) checkPublications(index, owedPublications(before, result.published), law) } @@ -874,6 +946,7 @@ export async function checkMutationHistory( const before = visible(state) const result = stepModel(state, { kind: `release` }, planFor) if (result === undefined) break + if (state.heldRows.length > 0) reach(`held-optimistic-row`) await provider.release() await compare(steps.length) checkPublications( diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index 1a53fcd0d5..6f7a9957ca 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -40,6 +40,11 @@ export type ReferenceMutationMutant = | `skips-delete` /** MUT-8: an update sends only the changed fields. */ | `sends-changes-only` + /** + * FID-5: the change feed carries the authored row instead of the stored + * copy, so a JSON provider's removed field stays an own `undefined`. + */ + | `echoes-authored-row` const acceptAtOnce: WritePlan = { outcome: `accept`, @@ -47,7 +52,11 @@ const acceptAtOnce: WritePlan = { error: new Error(`unplanned write`), } -/** A provider that answers writes by plan and feeds every commit back. */ +/** + * A provider that answers writes by plan and feeds every commit back. A JSON + * provider stores and feeds each row as JSON, so a field whose value is + * `undefined` is absent; otherwise rows are structured clones. + */ export class ControlledMutationProvider implements MutationProvider { private readonly committed = new Map() private readonly plans: Array = [] @@ -55,8 +64,18 @@ export class ControlledMutationProvider implements MutationProvider { private readonly log: Array = [] private listener: ((change: ProviderChange) => void) | undefined - constructor(seed: ReadonlyArray) { - for (const row of seed) this.committed.set(row.id, structuredClone(row)) + constructor( + seed: ReadonlyArray, + private readonly json = false, + private readonly echoesAuthored = false, + ) { + for (const row of seed) this.committed.set(row.id, this.stored(row)) + } + + private stored(value: T): T { + return this.json + ? (JSON.parse(JSON.stringify(value)) as T) + : structuredClone(value) } // The adapter's side: its change feed, initial read, and writes. @@ -73,7 +92,7 @@ export class ControlledMutationProvider implements MutationProvider { } write(request: WriteRequest): Promise { - this.log.push(structuredClone(request)) + this.log.push(this.stored(request)) const plan = this.plans.shift() ?? acceptAtOnce return new Promise((resolve, reject) => { const answer = () => { @@ -96,8 +115,10 @@ export class ControlledMutationProvider implements MutationProvider { /** A write replaces the whole row, as a complete-row API does. */ private commit(change: ProviderChange): void { if (change.type === `delete`) this.committed.delete(change.id) - else this.committed.set(change.row.id, structuredClone(change.row)) - this.listener?.(structuredClone(change)) + else this.committed.set(change.row.id, this.stored(change.row)) + this.listener?.( + this.echoesAuthored ? structuredClone(change) : this.stored(change), + ) } // The oracle's side. @@ -230,16 +251,22 @@ function referenceCollection( /** A production driver for the reference adapter, optionally mutated. */ export function referenceMutationDriver( mutant?: ReferenceMutationMutant, + { json = mutant === `echoes-authored-row` }: { json?: boolean } = {}, ): MutationDriver { return { - adapter: `reference mutations${mutant ? ` (${mutant})` : ``}`, + adapter: `reference mutations${json ? ` (JSON provider)` : ``}${mutant ? ` (${mutant})` : ``}`, + ...(json && { undefinedField: `absent` as const }), capabilities: new Set([ `write-rejection`, `held-writes`, `foreign-commits`, ]), create(seed): MutationFixture { - const provider = new ControlledMutationProvider(seed) + const provider = new ControlledMutationProvider( + seed, + json, + mutant === `echoes-authored-row`, + ) const collection = referenceCollection(provider, mutant) return { collection: observedView(collection, { diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts index 4ce1b36846..f252bfb62e 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -16,6 +16,7 @@ import { mutationReachStatesFor, } from '../src' import { referenceMutationDriver } from './reference-mutation-adapter' +import type { MutationDriver } from '../src' import type { ReferenceMutationMutant } from './reference-mutation-adapter' /** @@ -31,6 +32,9 @@ import type { ReferenceMutationMutant } from './reference-mutation-adapter' * - **Campaigns.** The same property, grammar, and run budget run with a * fixed seed and without one (ORC-007). Each fails when fewer than five * histories reach a declared state (ORC-015). + * - **Value domain.** The same campaigns run with a JSON provider, which + * declares `undefinedField: 'absent'`: a field an update removes is absent + * from its request and the provider's copy (FID-1's value domain). * - **Calibration.** Each hostile mutant of the reference adapter breaks one * law, and the fixed campaign must reject it with that law (ORC-006). */ @@ -41,11 +45,16 @@ const historyArb = mutationHistoryArb(driver) const runs = 300 const campaignTimeoutMs = 120_000 -async function campaign(options: fc.Parameters<[unknown]>) { - const reach = createOracleReach(property, mutationReachStatesFor(driver), { +const jsonDriver = referenceMutationDriver(undefined, { json: true }) + +async function campaign( + options: fc.Parameters<[unknown]>, + under: MutationDriver = driver, +) { + const reach = createOracleReach(property, mutationReachStatesFor(under), { minimum: 5, }) - await fc.assert(mutationProperty(driver, reach), options) + await fc.assert(mutationProperty(under, reach), options) return reach } @@ -68,6 +77,18 @@ describe(`mutation conformance: reference adapter`, () => { campaignTimeoutMs, ) + oracleReplayTest(it)( + `conforms with a JSON provider on a fixed-seed campaign`, + async () => { + const reach = await campaign( + { seed: 2118, numRuns: oracleRuns(runs) }, + jsonDriver, + ) + reach.assertReached() + }, + campaignTimeoutMs, + ) + const mutants: Array<[ReferenceMutationMutant, string | RegExp]> = [ [`applies-copy-late`, `MUT-1`], [`swallows-rejection`, `MUT-2`], @@ -76,6 +97,7 @@ describe(`mutation conformance: reference adapter`, () => { [`writes-concurrently`, `MUT-10/MUT-13`], [`skips-delete`, /MUT-8|FID-5/], [`sends-changes-only`, `MUT-8`], + [`echoes-authored-row`, /FID-5/], ] oracleReplayTest(it).each(mutants)( `rejects the %s mutant with %s`, From 9443fb24f6cb70b7bb395312b37147766e88ba58 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 08:24:49 -0700 Subject: [PATCH 05/28] fix(rxdb-db-collection): persist writes in author order A receiving driver runs the RxDB adapter on the shared mutation oracle, over a real RxDB database on memory storage. Its provider wraps the storage's `bulkWrite`: it stores JSON documents, as RxDB's persistent storages do, answers each request by plan, and rejects with a per-row write error, as schema validation does. The oracle found two defects. A write reached RxDB while an earlier one was still pending, so writes could land out of author order. A rejected delete rejected with an array of write errors instead of an Error. Writes now run one after another per collection, and a failed delete rejects with RxDB's RxError, as a failed insert does. - Law: MUT-10 (law set v18: a write submitted while an earlier one is pending reaches the backend only after the earlier one decides) and MUT-2 (a rejection rejects with the provider's error, here the RxError that wraps the storage's write error). - Reach: every declared mutation state, at least five histories each, in the fixed (seed 2118, 300 histories) and random campaigns. - RED: on the parent rxdb.ts, 38 of the fixed campaign's 300 histories failed MUT-10 and 24 failed MUT-2; the campaign shrinks to an insert, a held update, and a second insert that reached RxDB early. - GREEN: both campaigns pass; rxdb-db-collection 25 passed. - Mutant: the parent is the hostile mutant for each half of the fix. - Limits: a `bulkWrite` that throws is not generated. RxDB 17.1.0's incremental write queue does not catch it, so `incrementalPatch` never settles; avoiding it needs a maintainer decision (coverage map). No conflicts (409), and no update of a document another client deleted while the update was pending. - Size: rxdb-db-collection +43 gzip bytes. Insert and delete now share one write-error check; the ordering chain is the remaining cost. Co-authored-by: Isaac --- .changeset/rxdb-ordered-writes.md | 5 + docs/contributing/oracle-coverage.md | 54 ++- packages/db-adapter-conformance/MIGRATION.md | 49 ++- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 16 +- packages/rxdb-db-collection/src/rxdb.ts | 84 +++-- .../tests/mutation-conformance-driver.ts | 353 ++++++++++++++++++ ...tation-conformance-oracle.property.test.ts | 72 ++++ 8 files changed, 572 insertions(+), 62 deletions(-) create mode 100644 .changeset/rxdb-ordered-writes.md create mode 100644 packages/rxdb-db-collection/tests/mutation-conformance-driver.ts create mode 100644 packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts diff --git a/.changeset/rxdb-ordered-writes.md b/.changeset/rxdb-ordered-writes.md new file mode 100644 index 0000000000..b2bb74ebb3 --- /dev/null +++ b/.changeset/rxdb-ordered-writes.md @@ -0,0 +1,5 @@ +--- +'@tanstack/rxdb-db-collection': patch +--- + +Persist writes to RxDB in the order the application made them. A write now starts only after the earlier write on the same collection fulfills or rejects. Before, a later write could reach RxDB while an earlier one was still pending. A rejected delete now rejects with RxDB's `RxError` for the write error, as a rejected insert does, instead of with an array of write errors. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index e2989ff1fc..dd6505f32a 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7218,7 +7218,10 @@ restarted run empty. A fixed-seed campaign runs the React client Owner: `packages/db-adapter-conformance/src/mutation-oracle.ts`, run against the reference adapter in `packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts` -(`adapter-conformance.mutations.reference`). It checks auto-transaction +(`adapter-conformance.mutations.reference`, with a structured-clone and a JSON +provider) and against RxDB in +`packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts` +(`adapter-conformance.mutations.rxdb`). It checks auto-transaction writes on a ready Collection whose adapter persists them itself: MUT-1 (the public rows show the optimistic state while the mutation function runs, and no publication shows the earlier synced value when it fulfills), MUT-2 (a @@ -7232,13 +7235,46 @@ plan the check hands it before the write: accept or reject, at once or held until a `release`. Publications compare row values; a publication of the same row values, such as core's `$hasPendingWrites` update when a write is confirmed, counts as no change here (the virtual-row-field owner judges it). -Reach: each of insert, update, and delete accepted, a rejected write, a held -write, a write queued behind a held one, a held write rejected, a commit by -another client, and a quiescent cut after writes, at least five histories -each per campaign. Calibration: seven reference mutants each fail with their -law (an applied copy after the mutation function settles, a swallowed or -wrapped rejection, a duplicated insert request, writes that do not wait for -the earlier one, a skipped delete, an update that sends only changed fields). +The model follows the glossary's accepted sync transaction: an accepted +write's provider copy is held while any mutation is persisting, and the +write's held optimistic row stays until it applies. A driver declares how its +provider stores an authored `undefined` field (`undefinedField`, from FID-1's +value domain): `own`, as a structured clone keeps it, or `absent`, as a JSON +document must. Under `absent`, a removed field is absent from the request and +the provider's copy, and the copy publishes one more change of row values when +it replaces the held optimistic row. Reach: each of insert, update, and delete +accepted, a rejected write, a held write, a write queued behind a held one, a +held write rejected, a commit by another client, an accepted field removal, a +held optimistic row, and a quiescent cut after writes, at least five +histories each per campaign. Calibration: eight reference mutants each fail +with their law (an applied copy after the mutation function settles, a +swallowed or wrapped rejection, a duplicated insert request, writes that do +not wait for the earlier one, a skipped delete, an update that sends only +changed fields, and, under a JSON provider, a change feed that carries the +authored row instead of the stored copy). Before the held-row rule, the +JSON-provider campaign failed MUT-1 on a fulfilled removal behind a pending +write. + +RxDB (`undefinedField: 'absent'`): the driver runs a real RxDB database on +memory storage. The provider wraps that storage's `bulkWrite`, stores +documents as JSON as RxDB's persistent storages do, rejects a request with a +per-row write error (status 422, as schema validation reports), and reads its +rows from the storage instance rather than RxDB's document cache. MUT-2's +relation is the `RxError` whose `parameters.writeError` is the storage's error +object. On the parent of the fix, 38 of the fixed campaign's 300 histories +failed MUT-10 (a write reached RxDB while an earlier one was pending) and 24 +failed MUT-2 (a rejected delete rejected with an array of write errors). The +adapter now runs each write after the earlier one is decided and rejects a +failed delete with RxDB's `RxError`. RxDB limits: a `bulkWrite` that throws is +not generated. RxDB 17.1.0's incremental write queue does not catch it: the +throw is an unhandled rejection, `incrementalPatch` never settles, and every +later incremental write on the collection waits forever (trace: seed row 4, +an update of row 4 whose storage write throws). Fixing it in the adapter means +an update through `patch` instead of `incrementalPatch`, which gives up +RxDB's conflict retry when another tab writes; that needs a maintainer +decision. A conflict (status 409) is not generated, and an update of a +document that another client deleted while the update was pending (the +adapter skips it and fulfills) is outside the grammar. A removed field is an own `undefined` in the authored row (maintainer decision 2026-11-03, keeping the documented rule in `packages/db/tests/proxy-detachment-contract-oracle.test.ts`); the core owner @@ -7248,5 +7284,5 @@ counts its reach (`removedFieldEdits`). Limits: one pending mutation per key; commits by another client only while no mutation is pending (a commit held behind a pending mutation is core's MUT-5); no manual transactions, confirmation through sync (MUT-7), schema validation, or an update that -changes no field; no published adapter driver yet. +changes no field; only the RxDB adapter driver so far. diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 7730f49c23..9012f4cef8 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -17,7 +17,7 @@ reviewed change. | Family | Status | Laws | | ------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | | Eager live source | Built (`src/eager-live-source-oracle.ts`) | READY-2/3/4/8/10, HEALTH-1/2/5, PUB-1/2/3/5/7/9, FID-1/4, LIFE-2/5/10, EVID-1/2/3, RETRY-1 (eager), RECON-1, RENEW-1 | -| Mutations and confirmation | Needed | MUT-1/2/4/5/7/10/13, FID-5 | +| Mutations and confirmation | Built (`src/mutation-oracle.ts`) | MUT-1/2/4/5/7/10/13, FID-5 | | On-demand loading | Needed | DEM-1 to DEM-23, MAP-1 to MAP-4 | | Reload and retry | Needed | RELOAD-1 to RELOAD-5; RETRY-1 and HEALTH-2 beyond the eager source's retried failures | | Client cache and refetch | Needed | CACHE-1 to CACHE-8, REF-1 to REF-7, DIR-1 to DIR-5 | @@ -27,19 +27,19 @@ reviewed change. ## Adapters -| Adapter | Families that apply (section A) | Driver status | -| ----------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | -| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations, on-demand: not started | -| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: not started | -| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations, on-demand: not started | -| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | -| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | -| local-only | local and shared stores, mutations | Not started | -| localStorage | local and shared stores, mutations | Not started | -| Persisted wrapper | persistence wrapper | Not started | -| Storage hosts (8) | storage hosts | Not started | +| Adapter | Families that apply (section A) | Driver status | +| ----------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | +| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations, on-demand: not started | +| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | +| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations, on-demand: not started | +| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | +| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | +| local-only | local and shared stores, mutations | Not started | +| localStorage | local and shared stores, mutations | Not started | +| Persisted wrapper | persistence wrapper | Not started | +| Storage hosts (8) | storage hosts | Not started | ## Suggested order @@ -544,3 +544,24 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the - **Maintainer decision:** a removed field stays an own `undefined`, the documented rule; the core owner `optimistic-history-oracle` now generates field removal. +- **2026-11-03, RxDB, mutations.** Both campaigns are GREEN after an + adapter fix, with every declared reach state met. + - **Driver:** a real RxDB database on memory storage; the provider wraps + the storage's `bulkWrite`, stores JSON documents, and rejects with a + per-row write error. An insert the storage refuses as a conflict, which + RxDB's `bulkUpsert` sends before it updates, is not a request. + - **RED on the parent:** 38 of 300 fixed-campaign histories failed MUT-10 + (writes did not wait for the earlier one) and 24 failed MUT-2 (a rejected + delete rejected with an array). Fix: writes run in author order, and a + failed delete rejects with RxDB's `RxError`. The parent is the hostile + mutant for both. + - **Oracle changes this migration exposed:** a driver declares how its + provider stores an authored `undefined` field (`undefinedField`, FID-1's + value domain), and the model holds an accepted write's sync transaction + while another mutation persists (the glossary's held optimistic row). A + JSON-provider reference campaign and an eighth mutant calibrate both. + - **Open, needs a maintainer decision:** a storage `bulkWrite` that throws + wedges RxDB 17.1.0's incremental write queue, so `incrementalPatch` + never settles. The adapter could update through `patch`, which gives up + RxDB's conflict retry. Not generated until decided. + - **Size:** rxdb-db-collection +43 gzip bytes for MUT-10. diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 5a79cf9bd1..178d547afa 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -206,6 +206,7 @@ const staticOracleProperties = [ `sqlite-persistence.optimistic-history`, `adapter-conformance.eager-live-source.reference`, `adapter-conformance.mutations.reference`, + `adapter-conformance.mutations.rxdb`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index fb5abda15d..b5126c78e4 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -792,6 +792,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.mutations`, `reference`, ], + [ + `rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `rxdb`, + ], ] const owners = new Map() @@ -893,10 +898,13 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), - [ - `adapter-conformance.mutations.reference`, - `conforms on a random campaign or a replayed seed`, - ], + ...[`reference`, `rxdb`].map( + (driver) => + [ + `adapter-conformance.mutations.${driver}`, + `conforms on a random campaign or a replayed seed`, + ] as const, + ), [ `query-db.refetch-held-page-end`, `generated same-key page shifts.*replay campaign`, diff --git a/packages/rxdb-db-collection/src/rxdb.ts b/packages/rxdb-db-collection/src/rxdb.ts index e9d36b1fe1..cf4bbbc584 100644 --- a/packages/rxdb-db-collection/src/rxdb.ts +++ b/packages/rxdb-db-collection/src/rxdb.ts @@ -13,6 +13,7 @@ import type { FilledMangoQuery, RxCollection, RxDocumentData, + RxStorageWriteError, } from 'rxdb/plugins/core' import type { Subscription } from 'rxjs' @@ -300,47 +301,60 @@ export function rxdbCollectionOptions( rowUpdateMode: `full`, } + // Writes reach RxDB in author order: each starts after the earlier one is + // decided, whether it fulfilled or rejected. + let turn: Promise = Promise.resolve() + const inTurn = (write: () => Promise): Promise => { + const run = turn.then(write) + turn = run.catch(() => {}) + return run + } + const throwWriteError = (result: { + error: Array> + }) => { + if (result.error.length > 0) { + throw rxStorageWriteErrorToRxError(ensureNotFalsy(result.error[0])) + } + } + const collectionConfig: CollectionConfig = { ...restConfig, getKey, sync, - onInsert: async (params) => { - debug(`insert`, params) - const newItems = params.transaction.mutations.map((m) => m.modified) - const result = await rxCollection.bulkUpsert(newItems) - if (result.error.length > 0) { - throw rxStorageWriteErrorToRxError(ensureNotFalsy(result.error[0])) - } - }, - onUpdate: async (params) => { - debug(`update`, params) - const mutations = params.transaction.mutations.filter( - // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition - (m) => m.type === `update`, - ) + onInsert: (params) => + inTurn(async () => { + debug(`insert`, params) + const newItems = params.transaction.mutations.map((m) => m.modified) + throwWriteError(await rxCollection.bulkUpsert(newItems)) + }), + onUpdate: (params) => + inTurn(async () => { + debug(`update`, params) + const mutations = params.transaction.mutations.filter( + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + (m) => m.type === `update`, + ) - for (const mutation of mutations) { - const newValue = stripRxdbFields(mutation.modified) - const id = getKey(newValue) - const doc = await rxCollection.findOne(id).exec() - if (!doc) { - continue + for (const mutation of mutations) { + const newValue = stripRxdbFields(mutation.modified) + const id = getKey(newValue) + const doc = await rxCollection.findOne(id).exec() + if (!doc) { + continue + } + await doc.incrementalPatch(newValue) } - await doc.incrementalPatch(newValue) - } - }, - onDelete: async (params) => { - debug(`delete`, params) - const mutations = params.transaction.mutations.filter( - // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition - (m) => m.type === `delete`, - ) - const ids = mutations.map((mutation) => getKey(mutation.original)) - const result = await rxCollection.bulkRemove(ids) - if (result.error.length > 0) { - throw result.error - } - }, + }), + onDelete: (params) => + inTurn(async () => { + debug(`delete`, params) + const mutations = params.transaction.mutations.filter( + // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition + (m) => m.type === `delete`, + ) + const ids = mutations.map((mutation) => getKey(mutation.original)) + throwWriteError(await rxCollection.bulkRemove(ids)) + }), } return withCollectionConfigFactory(collectionConfig, () => rxdbCollectionOptions(config), diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts new file mode 100644 index 0000000000..b5c30a05db --- /dev/null +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -0,0 +1,353 @@ +/** + * Production driver for the shared mutation oracle + * (`packages/db-adapter-conformance/src/mutation-oracle.ts`). It runs the real + * RxDB adapter on a real RxDB database over RxDB's memory storage. The + * provider is a wrapper around that storage's `bulkWrite`, the one call + * through which every RxDB write reaches storage, so the adapter's own + * `bulkUpsert`, `findOne`, `incrementalPatch`, and `bulkRemove` calls run as + * an application's would. + * + * The oracle's steps map onto RxDB like this: + * + * - the provider stores each document as JSON, as RxDB's persistent + * storages and replication do, so a field whose value is `undefined` is + * absent from it (`undefinedField: 'absent'`); + * - a write request is one row of a `bulkWrite` call: an insert when the row + * has no live previous document, a delete when its document is + * `_deleted`, and an update otherwise. The request carries the document's + * fields without RxDB's reserved fields. A row the storage refuses as a + * conflict, an insert of a document it holds, is not a request: it has no + * effect, and RxDB's `bulkUpsert` sends it and answers it itself before + * it writes the update; + * - a plan answers the next request. A rejection answers with a per-row + * write error in the `bulkWrite` response and writes nothing, as RxDB's + * schema validation refuses a document; RxDB reports it as an `RxError` + * whose `parameters.writeError` is that error object (`matchesFailure`). + * A held request waits in the wrapper until `release`. An accepted + * request runs the memory storage's own `bulkWrite`; + * - a commit from another client is a write through the same RxCollection + * (`upsert`, or `remove` of the found document), as a second tab or a + * replication writes into the database. It bypasses the plans and the + * request log; + * - the provider's rows are the storage's non-deleted documents, read from + * the storage instance rather than RxDB's document cache. + * + * RxDB primary keys are strings. The driver stores row `n` as document id + * `"n"`, and `observedView` maps each key and row id back with + * `conformanceKey`. A JSON document omits an absent `note`. + * + * Limits: a rejection is a per-row write error. A `bulkWrite` that throws, as + * a failing storage does, is not generated: RxDB's incremental write queue + * (17.1.0) does not catch it, so `incrementalPatch` never settles and later + * incremental writes on the collection wait forever (recorded in the + * coverage map). A conflict (status 409) is not generated either. + */ +import { createCollection } from '@tanstack/db' +import { createRxDatabase, prepareQuery } from 'rxdb/plugins/core' +import { getRxStorageMemory } from 'rxdb/plugins/storage-memory' +import { rxdbCollectionOptions } from '../src/rxdb' +import { conformanceKey, observedView } from '../../db-adapter-conformance/src' +import type { + BulkWriteRow, + RxCollection, + RxDatabase, + RxDocumentData, + RxStorageBulkWriteResponse, + RxStorageInstance, + RxStorageWriteError, +} from 'rxdb/plugins/core' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../../db-adapter-conformance/src' + +type Doc = { id: string; title: string; note?: string } + +const schema = { + version: 0, + type: `object`, + primaryKey: `id`, + properties: { + id: { type: `string`, maxLength: 10 }, + title: { type: `string` }, + note: { type: `string` }, + }, + required: [`id`, `title`], +} as const + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + error: new Error(`unplanned write`), +} + +/** + * An RxDB document as a conformance row: key `"n"` is row id `n`. Other + * fields pass through, so an optimistic row's own `undefined` stays, and + * RxDB's reserved fields go. + */ +function conformanceRow(doc: Doc): ConformanceRow { + const { _rev, _meta, _deleted, _attachments, ...fields } = + doc as RxDocumentData + return { ...fields, id: conformanceKey(doc.id) } +} + +function document(row: ConformanceRow): Doc { + return { + id: String(row.id), + title: row.title, + ...(row.note === undefined ? {} : { note: row.note }), + } +} + +function writeRequest(row: BulkWriteRow): WriteRequest { + const id = conformanceKey(row.document.id) + if (row.document._deleted) return { type: `delete`, id } + return { + type: row.previous && !row.previous._deleted ? `update` : `insert`, + id, + row: conformanceRow(row.document), + } +} + +/** + * A write row as a serializing storage stores it. RxDB documents are JSON, + * and its persistent storages and replication serialize them, so the + * provider stores JSON: a field whose value is `undefined` is absent. The + * memory storage alone would keep the object as given. + */ +function jsonRow(row: BulkWriteRow): BulkWriteRow { + return JSON.parse(JSON.stringify(row)) as BulkWriteRow +} + +/** The storage's error object for each rejected row, to its plan's error. */ +const planErrors = new WeakMap() + +/** + * A per-row write error, as RxDB's schema validation reports a document it + * refuses: the row is not written and the response names it. + */ +function writeError( + row: BulkWriteRow, + error: Error, + context: string, +): RxStorageWriteError { + const rejected: RxStorageWriteError = { + status: 422, + isError: true, + context, + documentId: row.document.id, + writeRow: row, + validationErrors: [{ field: `title`, message: error.message }], + schema: schema as never, + } + planErrors.set(rejected, error) + return rejected +} + +/** + * MUT-2's relation for RxDB: RxDB reports a write error as an `RxError` whose + * `parameters.writeError` is the storage's error object for that row. + */ +function matchesFailure(reported: unknown, failure: Error): boolean { + const rowError = ( + reported as { parameters?: { writeError?: object } } | undefined + )?.parameters?.writeError + return ( + reported instanceof Error && + rowError !== undefined && + planErrors.get(rowError) === failure + ) +} + +type BulkWrite = ( + rows: Array>, + context: string, +) => Promise> + +class RxdbMutationProvider implements MutationProvider { + private readonly plans: Array = [] + private readonly held: Array<() => void> = [] + private readonly log: Array = [] + private foreign = false + private instance: RxStorageInstance | undefined + rxCollection: RxCollection | undefined + + /** Memory storage whose `conformance` collection writes go through `write`. */ + readonly storage: ReturnType + + constructor() { + const memory = getRxStorageMemory() + this.storage = { + ...memory, + createStorageInstance: async (params) => { + const instance = await memory.createStorageInstance(params) + if (params.collectionName !== `conformance`) return instance + this.instance = instance as unknown as RxStorageInstance< + Doc, + unknown, + unknown + > + const bulkWrite = instance.bulkWrite.bind( + instance, + ) as unknown as BulkWrite + const write: BulkWrite = (rows, context) => + this.write(rows, context, bulkWrite) + instance.bulkWrite = write as unknown as typeof instance.bulkWrite + return instance + }, + } + } + + private async write( + rows: Array>, + context: string, + bulkWrite: BulkWrite, + ): Promise> { + if (this.foreign) return bulkWrite(rows.map(jsonRow), context) + const response: RxStorageBulkWriteResponse = { error: [] } + for (const row of rows.map(jsonRow)) { + // An insert of a document the storage holds, even deleted, is refused + // as a conflict and has no effect. RxDB's `bulkUpsert` sends one before + // it updates, and answers the conflict itself, so it is not a request. + if (row.previous === undefined && (await this.holds(row.document.id))) { + response.error.push(...(await bulkWrite([row], context)).error) + continue + } + this.log.push(writeRequest(row)) + const plan = this.plans.shift() ?? acceptAtOnce + if (plan.held) { + await new Promise((resolve) => this.held.push(resolve)) + } + if (plan.outcome === `reject`) { + response.error.push(writeError(row, plan.error, context)) + continue + } + response.error.push(...(await bulkWrite([row], context)).error) + } + return response + } + + private async holds(id: string): Promise { + const found = await this.instance!.findDocumentsById([id], true) + return found.length > 0 + } + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + this.held.shift()?.() + } + + /** Answers every held request, so the database can close. */ + releaseAll(): void { + for (const answer of this.held.splice(0)) answer() + } + + async foreignCommit(change: ProviderChange): Promise { + const rxCollection = this.rxCollection! + this.foreign = true + try { + if (change.type === `upsert`) { + await rxCollection.upsert(document(change.row)) + } else { + await rxCollection.findOne(String(change.id)).remove() + } + } finally { + this.foreign = false + } + } + + /** The provider's starting rows, written before the adapter syncs. */ + async seed(rows: ReadonlyArray): Promise { + this.foreign = true + try { + await this.rxCollection!.bulkInsert(rows.map(document)) + } finally { + this.foreign = false + } + } + + /** The storage's documents, not RxDB's document cache. */ + async rows(): Promise> { + const instance = this.instance! + const { documents } = await instance.query( + prepareQuery(instance.schema, { + selector: { _deleted: false }, + sort: [{ id: `asc` }], + skip: 0, + }), + ) + return documents.map(conformanceRow) + } + + requests(): ReadonlyArray { + return this.log + } +} + +let sequence = 0 + +export function rxdbMutationDriver(): MutationDriver { + return { + adapter: `@tanstack/rxdb-db-collection`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + ]), + // A JSON document has no `undefined`: a removed field is absent. + undefinedField: `absent`, + matchesFailure, + async create(seed): Promise { + const provider = new RxdbMutationProvider() + const db: RxDatabase = await createRxDatabase({ + name: `conformance-mutations-${++sequence}`, + storage: provider.storage, + multiInstance: false, + }) + const { conformance } = await db.addCollections({ + conformance: { schema }, + }) + const rxCollection = conformance as RxCollection + provider.rxCollection = rxCollection + await provider.seed(seed) + const collection = createCollection( + rxdbCollectionOptions({ rxCollection, syncMode: `eager` }), + ) + return { + collection: observedView(collection, { + row: conformanceRow, + key: conformanceKey, + }), + base: () => + [...collection.base.values()].map((doc) => conformanceRow(doc)), + writer: { + insert: (row) => collection.insert(document(row)), + update: (row) => + collection.update(String(row.id), (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note + else delete draft.note + }), + delete: (id) => collection.delete(String(id)), + }, + provider, + dispose: async () => { + // Removing the database waits for its writes, so answer any write + // a failed history left held. + provider.releaseAll() + await collection.cleanup() + await db.remove() + }, + } + }, + } +} diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts new file mode 100644 index 0000000000..2794839626 --- /dev/null +++ b/packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -0,0 +1,72 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../../db-adapter-conformance/src' +import { rxdbMutationDriver } from './mutation-conformance-driver' + +/** + * # RxDB on the shared mutation oracle + * + * A receiving driver for `packages/db-adapter-conformance`. The shared + * module states the laws (MUT-1, MUT-2, MUT-8, MUT-10, MUT-13, FID-5, + * HEALTH-5), the model, the history grammar, and the check. This file + * supplies only the RxDB production driver in `mutation-conformance-driver.ts` + * and the campaigns; it states no expected results. + * + * The driver runs a real RxDB database on memory storage, and its provider + * wraps that storage's `bulkWrite`, so it declares every mutation + * capability: the storage can reject or hold a write, and another client can + * write into the database. + */ + +const property = `adapter-conformance.mutations.rxdb` +const driver = rxdbMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +const campaignTimeoutMs = 120_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: RxDB`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) From af5df8933564363ed0337ac7e2b19b02649febfb Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 09:00:11 -0700 Subject: [PATCH 06/28] test(db-adapter-conformance): generate provider copies that differ from the write Providers often store something other than what the client wrote: server defaults, triggers, normalization, a dropped field. A new capability, `server-rewrites`, lets an accepted insert's or update's plan rewrite the stored row (upper-case the title, drop `note`, or set `note`). The request must still carry the authored row (MUT-8); the provider's copy, not the authored row, is the row's value once it applies (MUT-1, FID-5). A grammar arm now generates updates without `note`, so field removals are common. RxDB declares the capability: the driver rewrites through RxDB's `preInsert` and `preSave` hooks, and its request log keeps the document as the adapter sent it. RxDB 17.1.0's incremental-write-queue wedge is pubkey/rxdb#9025, fixed in 17.6.0; the adapter keeps `incrementalPatch` (maintainer decision 2026-11-03). - Law: FID-5 and MUT-1 (law set v18): the Collection's state equals the provider's at a quiescent cut, and the optimistic row gives way to the provider's copy; MUT-8 for the request. - Reach: new state `server-rewrote-copy`, 78 to 146 histories of 400 per seed for the reference adapter; `field-removed` rose from 5 to 20 to 49 to 71 per seed with the new arm. Both reference campaigns and both RxDB campaigns meet every declared state. - RED: the reference mutant `writes-back-authored` (the adapter writes the authored row into its synced rows after the provider accepts) survives the fixed campaign without the capability. - GREEN: db-adapter-conformance 98 passed; rxdb-db-collection 25 passed. - Mutant: `writes-back-authored` fails FID-5 with the capability; RxDB with no-op hooks fails FID-5 on a `set-note` insert. - Limits: four rewrites over `title` and `note`; no rewrite of a delete, no server-generated key, and no rewrite on a commit by another client. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 40 ++++-- packages/db-adapter-conformance/MIGRATION.md | 21 ++- packages/db-adapter-conformance/src/index.ts | 2 + .../src/mutation-oracle.ts | 131 ++++++++++++++++-- .../tests/reference-mutation-adapter.ts | 20 ++- ...reference-mutation-oracle.property.test.ts | 1 + .../tests/mutation-conformance-driver.ts | 50 ++++++- 7 files changed, 226 insertions(+), 39 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index dd6505f32a..f166e85d42 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7242,21 +7242,32 @@ provider stores an authored `undefined` field (`undefinedField`, from FID-1's value domain): `own`, as a structured clone keeps it, or `absent`, as a JSON document must. Under `absent`, a removed field is absent from the request and the provider's copy, and the copy publishes one more change of row values when -it replaces the held optimistic row. Reach: each of insert, update, and delete +it replaces the held optimistic row. With `server-rewrites`, an accepted +insert's or update's plan can also rewrite the stored row (normalize the +title, drop `note`, or set `note` as a server default): the request still +carries the authored row (MUT-8), and the provider's copy, not the authored +row, is the row's value once it applies (MUT-1, FID-5). Reach: each of insert, update, and delete accepted, a rejected write, a held write, a write queued behind a held one, a held write rejected, a commit by another client, an accepted field removal, a -held optimistic row, and a quiescent cut after writes, at least five -histories each per campaign. Calibration: eight reference mutants each fail +held optimistic row, a copy the provider rewrote, and a quiescent cut after +writes, at least five histories each per campaign; a grammar arm generates +updates without `note`, so removals are common. Calibration: nine reference +mutants each fail with their law (an applied copy after the mutation function settles, a swallowed or wrapped rejection, a duplicated insert request, writes that do not wait for the earlier one, a skipped delete, an update that sends only -changed fields, and, under a JSON provider, a change feed that carries the -authored row instead of the stored copy). Before the held-row rule, the +changed fields, under a JSON provider a change feed that carries the +authored row instead of the stored copy, and an adapter that writes the +authored row into its synced rows after the provider accepts). The last +survives the fixed campaign without `server-rewrites` and fails FID-5 with +it. Before the held-row rule, the JSON-provider campaign failed MUT-1 on a fulfilled removal behind a pending write. -RxDB (`undefinedField: 'absent'`): the driver runs a real RxDB database on -memory storage. The provider wraps that storage's `bulkWrite`, stores +RxDB (`undefinedField: 'absent'`, `server-rewrites`): the driver runs a real +RxDB database on memory storage, and a rewrite is RxDB's `preInsert` and +`preSave` hooks; the request keeps the document as the adapter sent it. With +the hooks made no-ops, the fixed campaign fails FID-5. The provider wraps that storage's `bulkWrite`, stores documents as JSON as RxDB's persistent storages do, rejects a request with a per-row write error (status 422, as schema validation reports), and reads its rows from the storage instance rather than RxDB's document cache. MUT-2's @@ -7266,13 +7277,14 @@ failed MUT-10 (a write reached RxDB while an earlier one was pending) and 24 failed MUT-2 (a rejected delete rejected with an array of write errors). The adapter now runs each write after the earlier one is decided and rejects a failed delete with RxDB's `RxError`. RxDB limits: a `bulkWrite` that throws is -not generated. RxDB 17.1.0's incremental write queue does not catch it: the -throw is an unhandled rejection, `incrementalPatch` never settles, and every -later incremental write on the collection waits forever (trace: seed row 4, -an update of row 4 whose storage write throws). Fixing it in the adapter means -an update through `patch` instead of `incrementalPatch`, which gives up -RxDB's conflict retry when another tab writes; that needs a maintainer -decision. A conflict (status 409) is not generated, and an update of a +not generated. RxDB 17.1.0 to 17.5.0's incremental write queue does not catch +it: the throw is an unhandled rejection, `incrementalPatch` never settles, and +every later incremental write on the collection waits forever (trace: seed row +4, an update of row 4 whose storage write throws). This is RxDB issue +pubkey/rxdb#9025, fixed in RxDB 17.6.0 (pubkey/rxdb#9058). The adapter keeps +`incrementalPatch` and its conflict retry (maintainer decision 2026-11-03); +generating thrown storage errors waits for the repository's RxDB dev +dependency to reach 17.6.0. A conflict (status 409) is not generated, and an update of a document that another client deleted while the update was pending (the adapter skips it and fulfills) is outside the grammar. A removed field is an own `undefined` in the authored row (maintainer diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 9012f4cef8..b810eba4d9 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -560,8 +560,21 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the value domain), and the model holds an accepted write's sync transaction while another mutation persists (the glossary's held optimistic row). A JSON-provider reference campaign and an eighth mutant calibrate both. - - **Open, needs a maintainer decision:** a storage `bulkWrite` that throws - wedges RxDB 17.1.0's incremental write queue, so `incrementalPatch` - never settles. The adapter could update through `patch`, which gives up - RxDB's conflict retry. Not generated until decided. + - **Decided 2026-11-03:** a storage `bulkWrite` that throws wedges the + incremental write queue of RxDB 17.1.0 to 17.5.0, so `incrementalPatch` + never settles (pubkey/rxdb#9025, fixed in 17.6.0). The adapter keeps + `incrementalPatch`. Thrown storage errors are generated once the RxDB + dev dependency reaches 17.6.0. - **Size:** rxdb-db-collection +43 gzip bytes for MUT-10. +- **2026-11-03, mutations family, server rewrites.** A new capability, + `server-rewrites`: an accepted insert's or update's plan can rewrite the + stored row (normalize the title, drop `note`, set `note`), as server + defaults, triggers, and normalization do. The request still carries the + authored row; the provider's copy is the row's value. New reach state + `server-rewrote-copy`, and a grammar arm for updates without `note`. + - **Calibration:** the reference mutant `writes-back-authored` survives + without the capability and fails FID-5 with it. RxDB rewrites through + its `preInsert` and `preSave` hooks; with no-op hooks the fixed campaign + fails FID-5. + - **Result:** the reference adapter and RxDB are GREEN, with no + production change. diff --git a/packages/db-adapter-conformance/src/index.ts b/packages/db-adapter-conformance/src/index.ts index 61b7445ebe..ef83c22e90 100644 --- a/packages/db-adapter-conformance/src/index.ts +++ b/packages/db-adapter-conformance/src/index.ts @@ -41,6 +41,7 @@ export { mutationProperty, mutationReachStates, mutationReachStatesFor, + rewriteRow, } from './mutation-oracle' export type { MutationCapability, @@ -54,6 +55,7 @@ export type { MutationReceipt, MutationStep, MutationWriter, + ServerRewrite, WritePlan, WriteRequest, } from './mutation-oracle' diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 5308ebdc9e..a74b9bdc26 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -38,8 +38,11 @@ * - **FID-5.** At a quiescent cut the public rows, the exposed authoritative * base (`collection.base`), and the provider's rows are identical, and all * three equal the model: the provider's starting rows with every accepted - * write folded in author order, plus commits from other clients. A - * rejected write leaves no effect in the provider's rows. + * write's copy folded in author order, plus commits from other clients. A + * rejected write leaves no effect in the provider's rows. A provider may + * store a row that differs from the authored one (`server-rewrites`: a + * server default, trigger, or normalization); its copy is then the row's + * value, and the optimistic row gives way to it when the copy applies. * - **HEALTH-5.** No history causes an unhandled rejection or an undeclared * error- or warning-level console diagnostic. * @@ -84,6 +87,11 @@ export type MutationCapability = | `held-writes` /** The provider can commit a change made by another client. */ | `foreign-commits` + /** + * The provider can store an accepted write's row with its own changes, as + * server defaults, triggers, and normalization do (`ServerRewrite`). + */ + | `server-rewrites` /** The optimistic transaction a write call returns. */ export interface MutationReceipt { @@ -108,9 +116,44 @@ export interface WriteRequest { readonly row?: ConformanceRow } +/** + * A change the provider makes to an accepted insert's or update's row before + * it stores it. Its copy, not the authored row, is then the row's value: + * + * - `none`: the provider stores the row as sent; + * - `retitle`: it normalizes the title to upper case; + * - `drop-note`: it does not store `note`; + * - `set-note`: it sets `note` to `server`, as a server default does. + */ +export type ServerRewrite = `none` | `retitle` | `drop-note` | `set-note` + +/** The row a provider stores for `row` under `rewrite`. */ +export function rewriteRow( + row: ConformanceRow, + rewrite: ServerRewrite, +): ConformanceRow { + switch (rewrite) { + case `none`: + return row + case `retitle`: + return { ...row, title: row.title.toUpperCase() } + case `drop-note`: { + const { note: _dropped, ...rest } = row + return rest + } + case `set-note`: + return { ...row, note: `server` } + } +} + /** The provider's answer to the next write request. */ export interface WritePlan { readonly outcome: `accept` | `reject` + /** + * What an accepted insert or update stores (`server-rewrites`); `none` + * otherwise. The request still carries the authored row. + */ + readonly rewrite: ServerRewrite /** Answer only at `release`, so the write stays pending until then. */ readonly held: boolean /** The error a rejection answers with. */ @@ -192,7 +235,8 @@ export interface MutationDriver { * of a key they hold, with no other pending mutation on that key. An update * that changes no field is not generated. An update without `note` of a * row that holds one removes the field. Its plan rejects only with - * `write-rejection` and is held only with `held-writes`. + * `write-rejection`, is held only with `held-writes`, and rewrites an + * accepted insert's or update's row only with `server-rewrites`. * - `release`: the provider answers the oldest held request (`held-writes`). * - `foreign`: another client commits an upsert or a delete while no * mutation is pending (`foreign-commits`). @@ -200,7 +244,9 @@ export interface MutationDriver { * After the last step the check releases every held request, reaching a * quiescent cut. Keys range over 1..4 and titles over three values, so writes * revisit keys; a row carries `note` half the time, so an update can add or - * remove a field. + * remove a field, and a separate arm generates updates without `note` so + * that removals are common. With `server-rewrites`, half of the plans + * rewrite an accepted row. */ export type MutationStep = | { @@ -211,6 +257,7 @@ export type MutationStep = readonly note?: string readonly outcome: `accept` | `reject` readonly held: boolean + readonly rewrite: ServerRewrite } | { readonly kind: `release` } | { readonly kind: `foreign`; readonly change: ProviderChange } @@ -250,8 +297,18 @@ export function mutationHistoryArb( ) : fc.constant(`accept` as const), held: capabilities.has(`held-writes`) ? fc.boolean() : fc.constant(false), + rewrite: capabilities.has(`server-rewrites`) + ? fc.constantFrom( + `none`, + `none`, + `none`, + `retitle`, + `drop-note`, + `set-note`, + ) + : fc.constant(`none`), }) - .map(({ type, row, outcome, held }): MutationStep => ({ + .map(({ type, row, outcome, held, rewrite }): MutationStep => ({ kind: `write`, type, id: row.id, @@ -259,11 +316,32 @@ export function mutationHistoryArb( ...(row.note === undefined ? {} : { note: row.note }), outcome, held, + rewrite: type === `delete` ? `none` : rewrite, })) const steps: Array<{ weight: number arbitrary: fc.Arbitrary - }> = [{ weight: 6, arbitrary: write }] + }> = [ + { weight: 6, arbitrary: write }, + // An update without `note`: it removes the field when the row holds one. + { + weight: 2, + arbitrary: write.map( + (step): MutationStep => + step.kind === `write` + ? { + kind: `write`, + type: `update`, + id: step.id, + title: step.title, + outcome: step.outcome, + held: step.held, + rewrite: step.rewrite, + } + : step, + ), + }, + ] if (capabilities.has(`held-writes`)) { steps.push({ weight: 3, @@ -333,8 +411,10 @@ interface PendingWrite { readonly id: number /** The authored row's model value; absent for a delete. */ readonly value?: string - /** The provider's copy of the authored row; absent for a delete. */ - readonly stored?: string + /** The row its request carries: the authored row in the provider's value domain. */ + readonly sent?: string + /** The row the provider stores if it accepts: `sent` under the plan's rewrite. */ + readonly copy?: string readonly plan: WritePlan requested: boolean } @@ -345,9 +425,12 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } * The model holds only what a legal next step can observe: * * - `provider` is the provider's committed rows: the seed, every accepted - * write's stored copy, and every commit from another client, in commit - * order. A write's stored copy is its authored row under the driver's - * `undefinedField` rule. + * write's copy, and every commit from another client, in commit order. A + * write's copy is its authored row under the driver's `undefinedField` + * rule and then its plan's `ServerRewrite`: after a rewrite the + * provider's copy, not the authored row, is the row's value (FID-5). + * - `rewritten` counts accepted writes whose copy differs from their + * authored row, for reach. * - `pending` lists the mutations not yet decided, in author order. MUT-10 * orders their requests: the first is requested at once, and each later * one only after every earlier one is decided. @@ -376,6 +459,7 @@ interface MutationModel { applied: Map heldRows: Array pending: Array + rewritten: number requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> settlements: Array } @@ -391,6 +475,7 @@ function initialModel( applied: new Map(provider), heldRows: [], pending: [], + rewritten: 0, requests: [], settlements: [], } @@ -452,7 +537,8 @@ function visible(state: MutationModel): RowState { function decide(state: MutationModel, write: PendingWrite): void { if (write.plan.outcome === `accept`) { if (write.type === `delete`) state.provider.delete(write.id) - else state.provider.set(write.id, write.stored!) + else state.provider.set(write.id, write.copy!) + if (write.copy !== write.value) state.rewritten++ state.heldRows.push(write) state.settlements[write.index] = `fulfilled` } else { @@ -482,7 +568,7 @@ function advance(state: MutationModel): Array { state.requests.push({ type: first.type, id: first.id, - ...(first.stored === undefined ? {} : { value: first.stored }), + ...(first.sent === undefined ? {} : { value: first.sent }), }) } if (first.plan.held) return states @@ -537,7 +623,10 @@ function stepModel( id: step.id, ...(row && { value: rowValue(row), - stored: rowValue(storedRow(row, state.undefinedField)), + sent: rowValue(storedRow(row, state.undefinedField)), + copy: rowValue( + rewriteRow(storedRow(row, state.undefinedField), step.rewrite), + ), }), plan: plan(step), requested: false, @@ -616,6 +705,7 @@ export const mutationReachStates = [ `foreign-commit`, `field-removed`, `held-optimistic-row`, + `server-rewrote-copy`, `quiescent-after-writes`, ] as const @@ -630,6 +720,7 @@ const capabilityOfState: Partial< 'held-write-rejected': `held-writes`, 'foreign-commit': `foreign-commits`, 'held-optimistic-row': `held-writes`, + 'server-rewrote-copy': `server-rewrites`, } /** The declared states a driver's capabilities let histories reach. */ @@ -872,6 +963,7 @@ export async function checkMutationHistory( if (!ok) throw violation(`${law} (publication)`, stepIndex, owed, got) } + let rewritten = 0 // The plan the model decided the last write by; the provider gets it too. let lastPlan: WritePlan | undefined const planFor = ( @@ -879,7 +971,12 @@ export async function checkMutationHistory( ): WritePlan => { const error = new Error(`conformance write rejected ${failures.length}`) failures.push(error) - lastPlan = { outcome: step.outcome, held: step.held, error } + lastPlan = { + outcome: step.outcome, + held: step.held, + rewrite: step.rewrite, + error, + } return lastPlan } @@ -938,6 +1035,8 @@ export async function checkMutationHistory( break } if (state.heldRows.length > 0) reach(`held-optimistic-row`) + if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + rewritten = state.rewritten await compare(index) checkPublications(index, owedPublications(before, result.published), law) } @@ -947,6 +1046,8 @@ export async function checkMutationHistory( const result = stepModel(state, { kind: `release` }, planFor) if (result === undefined) break if (state.heldRows.length > 0) reach(`held-optimistic-row`) + if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + rewritten = state.rewritten await provider.release() await compare(steps.length) checkPublications( diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index 6f7a9957ca..8c59d37c18 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -13,7 +13,7 @@ * publication that applies it (MUT-1). */ import { createCollection } from '@tanstack/db' -import { observedView } from '../src' +import { observedView, rewriteRow } from '../src' import type { ConformanceRow, MutationDriver, @@ -45,10 +45,16 @@ export type ReferenceMutationMutant = * copy, so a JSON provider's removed field stays an own `undefined`. */ | `echoes-authored-row` + /** + * FID-5: after the provider accepts, the adapter writes the authored row + * into its synced rows, over the provider's rewritten copy. + */ + | `writes-back-authored` const acceptAtOnce: WritePlan = { outcome: `accept`, held: false, + rewrite: `none`, error: new Error(`unplanned write`), } @@ -103,7 +109,7 @@ export class ControlledMutationProvider implements MutationProvider { this.commit( request.type === `delete` ? { type: `delete`, id: request.id } - : { type: `upsert`, row: request.row! }, + : { type: `upsert`, row: rewriteRow(request.row!, plan.rewrite) }, ) resolve() } @@ -169,6 +175,12 @@ function referenceCollection( throw error } } + // The `writes-back-authored` mutant writes through the sync run's apply. + let applyChange: ((change: ProviderChange) => void) | undefined + const writeBack = (row: ConformanceRow) => { + if (mutant === `writes-back-authored`) + applyChange?.({ type: `upsert`, row: structuredClone(row) }) + } const collection = createCollection({ id: `reference-mutations-${++sequence}`, getKey: (row) => row.id, @@ -197,6 +209,7 @@ function referenceCollection( } void commit() markReady() + applyChange = apply return provider.subscribe((change) => { if (mutant === `applies-copy-late`) setTimeout(() => apply(change), 0) else apply(change) @@ -214,6 +227,7 @@ function referenceCollection( } await provider.write(request) if (mutant === `writes-insert-twice`) await provider.write(request) + writeBack(mutation.modified) } }), ), @@ -232,6 +246,7 @@ function referenceCollection( } as ConformanceRow) : structuredClone(mutation.modified), }) + writeBack(mutation.modified) } }), ), @@ -260,6 +275,7 @@ export function referenceMutationDriver( `write-rejection`, `held-writes`, `foreign-commits`, + `server-rewrites`, ]), create(seed): MutationFixture { const provider = new ControlledMutationProvider( diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts index f252bfb62e..69e142ad05 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -98,6 +98,7 @@ describe(`mutation conformance: reference adapter`, () => { [`skips-delete`, /MUT-8|FID-5/], [`sends-changes-only`, `MUT-8`], [`echoes-authored-row`, /FID-5/], + [`writes-back-authored`, /FID-5/], ] oracleReplayTest(it).each(mutants)( `rejects the %s mutant with %s`, diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index b5c30a05db..81e7a515db 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -29,6 +29,10 @@ * (`upsert`, or `remove` of the found document), as a second tab or a * replication writes into the database. It bypasses the plans and the * request log; + * - a rewrite (`server-rewrites`) is RxDB's `preInsert` and `preSave` hooks, + * which change a document before RxDB writes it, as an application's + * hooks for defaults or normalization do. The request keeps the document + * as the adapter sent it; * - the provider's rows are the storage's non-deleted documents, read from * the storage instance rather than RxDB's document cache. * @@ -46,7 +50,11 @@ import { createCollection } from '@tanstack/db' import { createRxDatabase, prepareQuery } from 'rxdb/plugins/core' import { getRxStorageMemory } from 'rxdb/plugins/storage-memory' import { rxdbCollectionOptions } from '../src/rxdb' -import { conformanceKey, observedView } from '../../db-adapter-conformance/src' +import { + conformanceKey, + observedView, + rewriteRow, +} from '../../db-adapter-conformance/src' import type { BulkWriteRow, RxCollection, @@ -83,6 +91,7 @@ const schema = { const acceptAtOnce: WritePlan = { outcome: `accept`, held: false, + rewrite: `none`, error: new Error(`unplanned write`), } @@ -105,13 +114,19 @@ function document(row: ConformanceRow): Doc { } } -function writeRequest(row: BulkWriteRow): WriteRequest { +/** + * The request a write row makes. Its row is the document as the adapter sent + * it, before RxDB's hooks rewrote it (`sent`), stored as JSON. + */ +function writeRequest(row: BulkWriteRow, sent?: Doc): WriteRequest { const id = conformanceKey(row.document.id) if (row.document._deleted) return { type: `delete`, id } return { type: row.previous && !row.previous._deleted ? `update` : `insert`, id, - row: conformanceRow(row.document), + row: conformanceRow( + sent ? (JSON.parse(JSON.stringify(sent)) as Doc) : row.document, + ), } } @@ -175,6 +190,8 @@ class RxdbMutationProvider implements MutationProvider { private readonly held: Array<() => void> = [] private readonly log: Array = [] private foreign = false + /** Each pending write's document as the adapter sent it, before hooks. */ + private readonly sent = new Map() private instance: RxStorageInstance | undefined rxCollection: RxCollection | undefined @@ -219,7 +236,8 @@ class RxdbMutationProvider implements MutationProvider { response.error.push(...(await bulkWrite([row], context)).error) continue } - this.log.push(writeRequest(row)) + this.log.push(writeRequest(row, this.sent.get(row.document.id))) + this.sent.delete(row.document.id) const plan = this.plans.shift() ?? acceptAtOnce if (plan.held) { await new Promise((resolve) => this.held.push(resolve)) @@ -238,6 +256,28 @@ class RxdbMutationProvider implements MutationProvider { return found.length > 0 } + /** + * RxDB's `preInsert` and `preSave` hooks, which change a document before + * RxDB writes it: the server-side rewrite (`server-rewrites`). Each keeps + * the document as sent for the request, then applies the next plan's + * rewrite. A conflicting insert that RxDB retries as an update runs both + * hooks; the first keeps the document as sent, and rewrites are + * idempotent. + */ + installHooks(rxCollection: RxCollection): void { + const hook = (data: Doc) => { + if (this.foreign || (data as { _deleted?: boolean })._deleted) return + if (!this.sent.has(data.id)) this.sent.set(data.id, { ...data }) + const rewrite = this.plans[0]?.rewrite ?? `none` + const row = rewriteRow(conformanceRow(data), rewrite) + data.title = row.title + if (row.note === undefined) delete data.note + else data.note = row.note + } + rxCollection.preInsert(hook, false) + rxCollection.preSave(hook, false) + } + plan(plan: WritePlan): void { this.plans.push(plan) } @@ -302,6 +342,7 @@ export function rxdbMutationDriver(): MutationDriver { `write-rejection`, `held-writes`, `foreign-commits`, + `server-rewrites`, ]), // A JSON document has no `undefined`: a removed field is absent. undefinedField: `absent`, @@ -318,6 +359,7 @@ export function rxdbMutationDriver(): MutationDriver { }) const rxCollection = conformance as RxCollection provider.rxCollection = rxCollection + provider.installHooks(rxCollection) await provider.seed(seed) const collection = createCollection( rxdbCollectionOptions({ rxCollection, syncMode: `eager` }), From 1cf77addf88f851ccb2359ddf9aa364de68c8cff Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 09:10:33 -0700 Subject: [PATCH 07/28] fix(rxdb-db-collection): require RxDB 17.6.0 In RxDB 17.1.0 to 17.5.0, a storage write that throws stops the incremental write queue (pubkey/rxdb#9025): the throw is an unhandled rejection, `incrementalPatch` never settles, and every later update on the collection waits forever. RxDB 17.6.0 fixes it (pubkey/rxdb#9058). The adapter keeps `incrementalPatch` and its conflict retry, and now requires RxDB 17.6.0 as its peer and dev dependency (maintainer decision 2026-11-03). The lockfile changes come from RxDB 17.6.0's own dependencies (graphql 17, graphql-ws 6, ws, @babel/runtime 8). The mutation oracle can now generate several ways a provider rejects: a driver declares `rejectionKinds`, the grammar picks one per rejection, and each kind has a reach state. RxDB declares a per-row write error and a thrown storage error. - Law: MUT-2 (a rejection rejects `when('settled')` with the provider's error) and HEALTH-5 (no unhandled rejection), for every rejection kind. - Reach: `rejected-as-write-error` and `rejected-as-thrown`, at least five histories each, with every other declared state, in both RxDB campaigns. - RED: on RxDB 17.1.0 both campaigns failed HEALTH-5 at an update whose storage write throws (an unhandled rejection). - GREEN: on RxDB 17.6.0 both campaigns pass; rxdb-db-collection 25 passed, db-adapter-conformance 98 passed. - Mutant: RxDB 17.1.0 is the hostile mutant. - Limits: no conflict (409) rejection; the reference adapter declares no rejection kinds. Co-authored-by: Isaac --- .changeset/rxdb-require-17-6.md | 5 + docs/contributing/oracle-coverage.md | 19 +- packages/db-adapter-conformance/MIGRATION.md | 6 +- .../src/mutation-oracle.ts | 112 ++++--- packages/rxdb-db-collection/package.json | 4 +- .../tests/mutation-conformance-driver.ts | 23 +- pnpm-lock.yaml | 293 +++++++++++------- 7 files changed, 291 insertions(+), 171 deletions(-) create mode 100644 .changeset/rxdb-require-17-6.md diff --git a/.changeset/rxdb-require-17-6.md b/.changeset/rxdb-require-17-6.md new file mode 100644 index 0000000000..92d702b8c1 --- /dev/null +++ b/.changeset/rxdb-require-17-6.md @@ -0,0 +1,5 @@ +--- +'@tanstack/rxdb-db-collection': patch +--- + +Require RxDB 17.6.0 or later. In RxDB 17.1.0 to 17.5.0, a storage write that throws stops RxDB's incremental write queue (pubkey/rxdb#9025), so the adapter's update never settles and every later update on the collection waits forever. RxDB 17.6.0 fixes this, and the adapter's update now rejects with the storage error. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index f166e85d42..090cad3234 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7276,15 +7276,16 @@ object. On the parent of the fix, 38 of the fixed campaign's 300 histories failed MUT-10 (a write reached RxDB while an earlier one was pending) and 24 failed MUT-2 (a rejected delete rejected with an array of write errors). The adapter now runs each write after the earlier one is decided and rejects a -failed delete with RxDB's `RxError`. RxDB limits: a `bulkWrite` that throws is -not generated. RxDB 17.1.0 to 17.5.0's incremental write queue does not catch -it: the throw is an unhandled rejection, `incrementalPatch` never settles, and -every later incremental write on the collection waits forever (trace: seed row -4, an update of row 4 whose storage write throws). This is RxDB issue -pubkey/rxdb#9025, fixed in RxDB 17.6.0 (pubkey/rxdb#9058). The adapter keeps -`incrementalPatch` and its conflict retry (maintainer decision 2026-11-03); -generating thrown storage errors waits for the repository's RxDB dev -dependency to reach 17.6.0. A conflict (status 409) is not generated, and an update of a +failed delete with RxDB's `RxError`. RxDB rejection kinds (`rejectionKinds`): a per-row write error and a +`bulkWrite` that throws, as a failing storage does; each must be reached. On +RxDB 17.1.0 to 17.5.0 a thrown storage error wedges the incremental write +queue: the throw is an unhandled rejection, `incrementalPatch` never settles, +and every later incremental write on the collection waits forever +(pubkey/rxdb#9025, fixed in RxDB 17.6.0 by pubkey/rxdb#9058). On 17.1.0 both +campaigns failed HEALTH-5 at an update whose storage write throws. The adapter +keeps `incrementalPatch` and its conflict retry, and its RxDB dev dependency +and peer range are now 17.6.0 and `>=17.6.0` (maintainer decision +2026-11-03); both campaigns pass. A conflict (status 409) is not generated, and an update of a document that another client deleted while the update was pending (the adapter skips it and fulfills) is outside the grammar. A removed field is an own `undefined` in the authored row (maintainer diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index b810eba4d9..2eaa285432 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -563,8 +563,10 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the - **Decided 2026-11-03:** a storage `bulkWrite` that throws wedges the incremental write queue of RxDB 17.1.0 to 17.5.0, so `incrementalPatch` never settles (pubkey/rxdb#9025, fixed in 17.6.0). The adapter keeps - `incrementalPatch`. Thrown storage errors are generated once the RxDB - dev dependency reaches 17.6.0. + `incrementalPatch`; its RxDB dev dependency is now 17.6.0 and its peer + range `>=17.6.0`. A driver can declare `rejectionKinds`; RxDB declares + a per-row write error and a thrown storage error. RED on 17.1.0 (both + campaigns failed HEALTH-5 at a thrown update), GREEN on 17.6.0. - **Size:** rxdb-db-collection +43 gzip bytes for MUT-10. - **2026-11-03, mutations family, server rewrites.** A new capability, `server-rewrites`: an accepted insert's or update's plan can rewrite the diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index a74b9bdc26..53b9fd6603 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -156,6 +156,11 @@ export interface WritePlan { readonly rewrite: ServerRewrite /** Answer only at `release`, so the write stays pending until then. */ readonly held: boolean + /** + * How the provider rejects, one of the driver's `rejectionKinds`; absent + * when the driver declares none. + */ + readonly rejectionKind?: string /** The error a rejection answers with. */ readonly error: Error } @@ -202,6 +207,13 @@ export interface MutationDriver { * `when('settled')` must reject with the provider's error by identity. */ readonly matchesFailure?: (reported: unknown, failure: Error) => boolean + /** + * The ways the provider can reject a write request, such as an error in + * its response and a thrown exception (`write-rejection`). The grammar + * picks one for each rejection, and each must be reached; MUT-2 owes the + * same outcome for every kind. + */ + readonly rejectionKinds?: ReadonlyArray /** * How the provider stores an authored field whose value is `undefined`, * as an update that removes a field authors it. FID-1 parameterizes row @@ -235,7 +247,8 @@ export interface MutationDriver { * of a key they hold, with no other pending mutation on that key. An update * that changes no field is not generated. An update without `note` of a * row that holds one removes the field. Its plan rejects only with - * `write-rejection`, is held only with `held-writes`, and rewrites an + * `write-rejection`, in one of the driver's `rejectionKinds` when it + * declares several, is held only with `held-writes`, and rewrites an * accepted insert's or update's row only with `server-rewrites`. * - `release`: the provider answers the oldest held request (`held-writes`). * - `foreign`: another client commits an upsert or a delete while no @@ -258,6 +271,8 @@ export type MutationStep = readonly outcome: `accept` | `reject` readonly held: boolean readonly rewrite: ServerRewrite + /** One of the driver's `rejectionKinds`, for a rejection. */ + readonly rejectionKind?: string } | { readonly kind: `release` } | { readonly kind: `foreign`; readonly change: ProviderChange } @@ -278,9 +293,10 @@ const rowArb: fc.Arbitrary = fc.record( /** Generates histories whose steps use only the driver's capabilities. */ export function mutationHistoryArb( - domain: Pick, + domain: Pick, ): fc.Arbitrary { const { capabilities } = domain + const kinds = domain.rejectionKinds ?? [] const write = fc .record({ type: fc.constantFrom( @@ -307,17 +323,26 @@ export function mutationHistoryArb( `set-note`, ) : fc.constant(`none`), + rejectionKind: + kinds.length > 0 + ? fc.constantFrom(...kinds) + : fc.constant(undefined), }) - .map(({ type, row, outcome, held, rewrite }): MutationStep => ({ - kind: `write`, - type, - id: row.id, - title: row.title, - ...(row.note === undefined ? {} : { note: row.note }), - outcome, - held, - rewrite: type === `delete` ? `none` : rewrite, - })) + .map( + ({ type, row, outcome, held, rewrite, rejectionKind }): MutationStep => ({ + kind: `write`, + type, + id: row.id, + title: row.title, + ...(row.note === undefined ? {} : { note: row.note }), + outcome, + held, + rewrite: type === `delete` ? `none` : rewrite, + ...(outcome === `reject` && rejectionKind !== undefined + ? { rejectionKind } + : {}), + }), + ) const steps: Array<{ weight: number arbitrary: fc.Arbitrary @@ -326,19 +351,21 @@ export function mutationHistoryArb( // An update without `note`: it removes the field when the row holds one. { weight: 2, - arbitrary: write.map( - (step): MutationStep => - step.kind === `write` - ? { - kind: `write`, - type: `update`, - id: step.id, - title: step.title, - outcome: step.outcome, - held: step.held, - rewrite: step.rewrite, - } - : step, + arbitrary: write.map((step): MutationStep => + step.kind === `write` + ? { + kind: `write`, + type: `update`, + id: step.id, + title: step.title, + outcome: step.outcome, + held: step.held, + rewrite: step.rewrite, + ...(step.rejectionKind === undefined + ? {} + : { rejectionKind: step.rejectionKind }), + } + : step, ), }, ] @@ -709,7 +736,9 @@ export const mutationReachStates = [ `quiescent-after-writes`, ] as const -export type MutationReachState = (typeof mutationReachStates)[number] +/** A rejection of one of the driver's `rejectionKinds`, per kind. */ +export type MutationReachState = + (typeof mutationReachStates)[number] | `rejected-as-${string}` const capabilityOfState: Partial< Record @@ -725,17 +754,25 @@ const capabilityOfState: Partial< /** The declared states a driver's capabilities let histories reach. */ export function mutationReachStatesFor( - domain: Pick, + domain: Pick, ): Array { - return mutationReachStates.filter((state) => { - const capability = capabilityOfState[state] - if (state === `held-write-rejected`) - return ( - domain.capabilities.has(`held-writes`) && - domain.capabilities.has(`write-rejection`) + const kinds = domain.capabilities.has(`write-rejection`) + ? (domain.rejectionKinds ?? []).map( + (kind): MutationReachState => `rejected-as-${kind}`, ) - return capability === undefined || domain.capabilities.has(capability) - }) + : [] + return [ + ...mutationReachStates.filter((state) => { + const capability = capabilityOfState[state] + if (state === `held-write-rejected`) + return ( + domain.capabilities.has(`held-writes`) && + domain.capabilities.has(`write-rejection`) + ) + return capability === undefined || domain.capabilities.has(capability) + }), + ...kinds, + ] } type Settlement = `pending` | `fulfilled` | { rejected: unknown } @@ -975,6 +1012,9 @@ export async function checkMutationHistory( outcome: step.outcome, held: step.held, rewrite: step.rewrite, + ...(step.rejectionKind === undefined + ? {} + : { rejectionKind: step.rejectionKind }), error, } return lastPlan @@ -1014,6 +1054,8 @@ export async function checkMutationHistory( if (write.held) reach(`held-write-pending`) if (done.queued) reach(`write-queued-behind-held`) if (write.outcome === `reject`) reach(`write-rejected`) + if (write.rejectionKind !== undefined) + reach(`rejected-as-${write.rejectionKind}`) else if (!write.held) reach(`${write.type}-accepted` as const) if (done.removesField && write.outcome === `accept`) reach(`field-removed`) diff --git a/packages/rxdb-db-collection/package.json b/packages/rxdb-db-collection/package.json index b28d3cdded..80f43c62d0 100644 --- a/packages/rxdb-db-collection/package.json +++ b/packages/rxdb-db-collection/package.json @@ -66,10 +66,10 @@ "@tanstack/db": "workspace:*", "@tanstack/store": "^0.9.2", "debug": "^4.4.3", - "rxdb": "17.1.0" + "rxdb": "17.6.0" }, "peerDependencies": { - "rxdb": ">=17.0.0", + "rxdb": ">=17.6.0", "rxjs": ">=7.8.2", "typescript": ">=4.7" }, diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index 81e7a515db..8103e2ce77 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -19,10 +19,13 @@ * conflict, an insert of a document it holds, is not a request: it has no * effect, and RxDB's `bulkUpsert` sends it and answers it itself before * it writes the update; - * - a plan answers the next request. A rejection answers with a per-row - * write error in the `bulkWrite` response and writes nothing, as RxDB's - * schema validation refuses a document; RxDB reports it as an `RxError` - * whose `parameters.writeError` is that error object (`matchesFailure`). + * - a plan answers the next request. A rejection writes nothing, in one of + * two kinds: `write-error` answers with a per-row write error in the + * `bulkWrite` response, as RxDB's schema validation refuses a document, + * which RxDB reports as an `RxError` whose `parameters.writeError` is that + * error object; `thrown` throws the plan's error from `bulkWrite`, as a + * failing storage does, which reaches the caller as itself + * (`matchesFailure`). * A held request waits in the wrapper until `release`. An accepted * request runs the memory storage's own `bulkWrite`; * - a commit from another client is a write through the same RxCollection @@ -40,11 +43,7 @@ * `"n"`, and `observedView` maps each key and row id back with * `conformanceKey`. A JSON document omits an absent `note`. * - * Limits: a rejection is a per-row write error. A `bulkWrite` that throws, as - * a failing storage does, is not generated: RxDB's incremental write queue - * (17.1.0) does not catch it, so `incrementalPatch` never settles and later - * incremental writes on the collection wait forever (recorded in the - * coverage map). A conflict (status 409) is not generated either. + * Limits: a conflict (status 409) is not generated. */ import { createCollection } from '@tanstack/db' import { createRxDatabase, prepareQuery } from 'rxdb/plugins/core' @@ -166,10 +165,12 @@ function writeError( } /** - * MUT-2's relation for RxDB: RxDB reports a write error as an `RxError` whose + * MUT-2's relation for RxDB: a thrown storage error reaches the caller as + * itself, and RxDB reports a write error as an `RxError` whose * `parameters.writeError` is the storage's error object for that row. */ function matchesFailure(reported: unknown, failure: Error): boolean { + if (reported === failure) return true const rowError = ( reported as { parameters?: { writeError?: object } } | undefined )?.parameters?.writeError @@ -243,6 +244,7 @@ class RxdbMutationProvider implements MutationProvider { await new Promise((resolve) => this.held.push(resolve)) } if (plan.outcome === `reject`) { + if (plan.rejectionKind === `thrown`) throw plan.error response.error.push(writeError(row, plan.error, context)) continue } @@ -347,6 +349,7 @@ export function rxdbMutationDriver(): MutationDriver { // A JSON document has no `undefined`: a removed field is absent. undefinedField: `absent`, matchesFailure, + rejectionKinds: [`write-error`, `thrown`], async create(seed): Promise { const provider = new RxdbMutationProvider() const db: RxDatabase = await createRxDatabase({ diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 01f3da78cf..86d1eccb54 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -318,7 +318,7 @@ importers: version: 5.90.20(react@19.0.0) expo: specifier: ~53.0.26 - version: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + version: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) expo-constants: specifier: ~17.1.0 version: 17.1.8(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)) @@ -415,7 +415,7 @@ importers: version: 5.90.20(react@19.0.0) expo: specifier: ~53.0.26 - version: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + version: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) expo-constants: specifier: ~17.1.0 version: 17.1.8(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)) @@ -1332,7 +1332,7 @@ importers: version: 0.4.1(vitest@4.1.12) '@neon/realtime': specifier: 0.1.2 - version: 0.1.2 + version: 0.1.2(ws@8.22.0) '@standard-schema/spec': specifier: ^1.1.0 version: 1.1.0 @@ -1714,8 +1714,8 @@ importers: specifier: ^4.4.3 version: 4.4.3 rxdb: - specifier: 17.1.0 - version: 17.1.0(@angular/core@20.3.34(@angular/compiler@20.3.34)(rxjs@7.8.2)(zone.js@0.15.1))(react@19.2.4)(rxjs@7.8.2)(vue@3.5.28(typescript@5.9.3)) + specifier: 17.6.0 + version: 17.6.0(@angular/core@20.3.34(@angular/compiler@20.3.34)(rxjs@7.8.2)(zone.js@0.15.1))(react@19.2.4)(rxjs@7.8.2)(typescript@5.9.3)(vue@3.5.28(typescript@5.9.3)) rxjs: specifier: '>=7.8.2' version: 7.8.2 @@ -2623,14 +2623,17 @@ packages: peerDependencies: '@babel/core': ^7.0.0-0 - '@babel/runtime@7.28.6': - resolution: {integrity: sha512-05WQkdpL9COIMz4LjTxGpPNCdlpyimKppYNoJ5Di5EUObifl8t4tuLuUBBZEpoLYOmfvIWrsp9fCl0HoPRVTdA==} - engines: {node: '>=6.9.0'} - '@babel/runtime@7.29.2': resolution: {integrity: sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==} engines: {node: '>=6.9.0'} + '@babel/runtime@7.29.7': + resolution: {integrity: sha512-Nq8OhGWiZIZGV6hLHoyAKLLcJihP/xFeBMGJoUrxTX2psI8dCifzLhZISFb+VWS3wFMRDmCGw5R+dOySCqPLhw==} + engines: {node: '>=6.9.0'} + + '@babel/runtime@8.0.5': + resolution: {integrity: sha512-7NK+Lz3spQ52XsUGTxIEVU4jYN2/dIaX8sTxRFAUtYmttYZnVh3aehiihv+/Gb7+duMNHl2OrFcBQRyOHScxpg==} + '@babel/template@7.29.7': resolution: {integrity: sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg==} engines: {node: '>=6.9.0'} @@ -5027,10 +5030,22 @@ packages: cpu: [x64] os: [win32] + '@noble/ciphers@2.1.1': + resolution: {integrity: sha512-bysYuiVfhxNJuldNXlFEitTVdNnYUc+XNJZd7Qm2a5j1vZHgY+fazadNFWFaMK/2vye0JVlxV3gHmC0WDfAOQw==} + engines: {node: '>= 20.19.0'} + '@noble/ciphers@2.4.0': resolution: {integrity: sha512-AnjFn0Jv92laAkvMrghlFZq4qQCIN/4DxFV/eooqtC2YTjB7kBeLMS2T9KJX4Dn+ZVXLOwK0lSgqDtx9gvxtiw==} engines: {node: '>= 20.19.0'} + '@noble/curves@2.0.1': + resolution: {integrity: sha512-vs1Az2OOTBiP4q0pwjW5aF0xp9n4MxVrmkFBxc6EKZc6ddYx5gaZiAsZoq0uRRXWbi3AT/sBqn05eRPtn1JCPw==} + engines: {node: '>= 20.19.0'} + + '@noble/hashes@2.0.1': + resolution: {integrity: sha512-XlOlEbQcE9fmuXxrVTXCTlG2nlRXa9Rj3rr5Ue/+tX+nmkgbX720YHh0VR3hBF9xDvwnb8D2shVGOwNx+ulArw==} + engines: {node: '>= 20.19.0'} + '@noble/hashes@2.4.0': resolution: {integrity: sha512-X5XaVWZIBCT7HHZGm5I7ZQXDwLG+bGXuSrMQAW+7Zvl87h1kmc1ZB1VSRJcpUfoUrGQp4Fkoxm5kZ+Ms+aW+eA==} engines: {node: '>= 20.19.0'} @@ -6055,6 +6070,15 @@ packages: resolution: {integrity: sha512-jU12K5YJUpK7bsPhWUgxoZf0R1fkt6z+AiOu+qHybQgWRmV8VuwZhPsp4N6lwMUkRrNmPOgk2143xNmTXEBjcQ==} engines: {node: ^20.19.0 || ^22.12.0 || >=24.0.0, npm: ^6.11.0 || ^7.5.6 || >=8.0.0, yarn: '>= 1.13.0'} + '@scure/base@2.0.0': + resolution: {integrity: sha512-3E1kpuZginKkek01ovG8krQ0Z44E3DHPjc5S2rjJw9lZn3KSQOs8S7wqikF/AH7iRanHypj85uGyxk0XAyC37w==} + + '@scure/bip32@2.0.1': + resolution: {integrity: sha512-4Md1NI5BzoVP+bhyJaY3K6yMesEFzNS1sE/cP+9nuvE7p/b0kx9XbpDHHFl8dHtufcbdHRUUQdRqLIPHN/s7yA==} + + '@scure/bip39@2.0.1': + resolution: {integrity: sha512-PsxdFj/d2AcJcZDX1FXN3dDgitDDTmwf78rKZq1a6c1P1Nan1X/Sxc7667zU3U+AN60g7SxxP0YCVw2H/hBycg==} + '@shikijs/engine-oniguruma@3.13.0': resolution: {integrity: sha512-O42rBGr4UDSlhT2ZFMxqM7QzIU+IcpoTMzb3W7AlziI1ZF7R8eS2M0yt5Ry35nnnTX/LTLXFPUjRFCIW+Operg==} @@ -6693,9 +6717,6 @@ packages: '@types/chai@5.2.2': resolution: {integrity: sha512-8kB30R7Hwqf40JPiKhVzodJs2Qc1ZJ5zuT3uzw5Hq/dhNCl3G3l83jfpdI1e20BP348+fV7VIL/+FxaXkqBmWg==} - '@types/clone@2.1.4': - resolution: {integrity: sha512-NKRWaEGaVGVLnGLB2GazvDaZnyweW9FJLLFL5LhywGJB3aqGMT9R/EUoJoSRP4nzofYnZysuDmrEJtJdAqUOtQ==} - '@types/common-tags@1.8.1': resolution: {integrity: sha512-20R/mDpKSPWdJs5TOpz3e7zqbeCNuMCPhV7Yndk9KU2Rbij2r5W4RzwDPkzC+2lzUqXYu9rFzTktCBnDjHuNQg==} @@ -7720,8 +7741,8 @@ packages: resolution: {integrity: sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==} engines: {node: '>=8'} - broadcast-channel@7.3.0: - resolution: {integrity: sha512-UHPhLBQKfQ8OmMFMpmPfO5dRakyA1vsfiDGWTYNvChYol65tbuhivPEGgZZiuetorvExdvxaWiBy/ym1Ty08yA==} + broadcast-channel@7.4.0: + resolution: {integrity: sha512-Oe0EdXTN+BjH0QatATL1tnK0Qs9aKMSML59nYKOJ8Jk6IMT9Y2O+bJGU6MEA9tD7onvj5HnrwQf8fCzXEBcbDQ==} browser-fs-access@0.35.0: resolution: {integrity: sha512-sLoadumpRfsjprP8XzVjpQc0jK8yqHBx0PtUTGYj2fftT+P/t+uyDAQdMgGAPKD011in/O+YYGh7fIs0oG/viw==} @@ -7942,9 +7963,9 @@ packages: resolution: {integrity: sha512-/rFeCpNJQbhSZjGVwO9RFV3xPqbnERS8MmIQzCtD/zl6gpJuV/bMLuN92oG3F7d8oDEHHRrujSXNUr8fpjntKw==} engines: {node: '>=18'} - commander@14.0.3: - resolution: {integrity: sha512-H+y0Jo/T1RZ9qPP4Eh1pkcQcLRglraJaSLoyOtHxu6AapkjWVCy2Sit1QQ4x3Dng8qDlSsZEet7g5Pq06MvTgw==} - engines: {node: '>=20'} + commander@15.0.0: + resolution: {integrity: sha512-z67u4ZhzCL/Tydu1lJARtEZYWbWaN7oYLHbsuzocr6y4N6WZAagG3RQ4FW61V1/0+jImpj293XfrcYnd1qxtPg==} + engines: {node: '>=22.12.0'} commander@2.20.3: resolution: {integrity: sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==} @@ -8249,8 +8270,8 @@ packages: devalue@5.9.4: resolution: {integrity: sha512-sPAT4pztbu6586/hrhOnMKS17IJrvg12mXiSPSS3W5qDeN2RGgvZ0diZCm31dBbnevfVmujNO3IM2wrS4Y2Rhg==} - dexie@4.0.10: - resolution: {integrity: sha512-eM2RzuR3i+M046r2Q0Optl3pS31qTWf8aFuA7H9wnsHTwl8EPvroVLwvQene/6paAs39Tbk6fWZcn2aZaHkc/w==} + dexie@4.4.2: + resolution: {integrity: sha512-zMtV8q79EFE5U8FKZvt0Y/77PCU/Hr/RDxv1EDeo228L+m/HTbeN2AjoQm674rhQCX8n3ljK87lajt7UQuZfvw==} diff@8.0.4: resolution: {integrity: sha512-DPi0FmjiSU5EvQV0++GFDOJ9ASQUVFh5kD+OzOnYdi7n3Wpm9hWWGfB/O2blfHcMVTL5WkQXSnRiK9makhrcnw==} @@ -9327,15 +9348,25 @@ packages: graphemer@1.4.0: resolution: {integrity: sha512-EtKwoO6kxCL9WO5xipiHTZlSzBm7WLT627TqC/uVRd0HKmq8NXyebnNYxDoBi7wt8eTWrUrKXCOVaFq9x1kgag==} - graphql-ws@5.16.2: - resolution: {integrity: sha512-E1uccsZxt/96jH/OwmLPuXMACILs76pKF2i3W861LpKBCYtGIyPQGtWLuBLkND4ox1KHns70e83PS4te50nvPQ==} - engines: {node: '>=10'} + graphql-ws@6.3.0: + resolution: {integrity: sha512-qBAXPtFjWHF3QypDyNcBp8Rycd+rmofxUSJsu3bGI9QMMq7pImKBSLSg9hZ2ojIMbJ54T/LPjKBIeOUoJbNBnQ==} + engines: {node: '>=20'} peerDependencies: - graphql: '>=0.11 <=16' + '@fastify/websocket': ^10 || ^11 + crossws: ~0.3 || ~0.4 + graphql: ^15.10.1 || ^16 || ^17 + ws: ^8 + peerDependenciesMeta: + '@fastify/websocket': + optional: true + crossws: + optional: true + ws: + optional: true - graphql@16.13.2: - resolution: {integrity: sha512-5bJ+nf/UCpAjHM8i06fl7eLyVC9iuNAjm9qzkiu2ZGhM0VscSvS6WDPfAwkdkBuoXGM9FJSbKl6wylMwP9Ktig==} - engines: {node: ^12.22.0 || ^14.16.0 || ^16.0.0 || >=17.0.0} + graphql@17.0.2: + resolution: {integrity: sha512-FRWbddMxfkjiB7z+aQDWIR+E34xo9I8c9mtK2RPv8PmMzKRvrdsreHL/Ui/TmwHJfhHChEtsFPyMHKI+xuarQQ==} + engines: {node: ^22.0.0 || ^24.0.0 || ^25.0.0 || >=26.0.0} h3@2.0.1-rc.20: resolution: {integrity: sha512-28ljodXuUp0fZovdiSRq4G9OgrxCztrJe5VdYzXAB7ueRvI7pIUqLU14Xi3XqdYJ/khXjfpUOOD2EQa6CmBgsg==} @@ -9579,10 +9610,6 @@ packages: resolution: {integrity: sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==} engines: {node: '>= 0.10'} - is-arguments@1.2.0: - resolution: {integrity: sha512-7bVbi0huj/wrIAOzb8U1aszg9kdi3KN/CyU19CTI7tAoZYEZoL9yCDXpbXN+uPsuWnP02cyug1gleqq+TU+YCA==} - engines: {node: '>= 0.4'} - is-array-buffer@3.0.5: resolution: {integrity: sha512-DDfANUiiG2wC1qawP66qlTugJeL5HyzMpfr8lLK+jMQirGzNod0B12cFB/9q838Ru27sBwfw78/rdoU7RERz6A==} engines: {node: '>= 0.4'} @@ -9905,8 +9932,8 @@ packages: jose@6.2.13: resolution: {integrity: sha512-7TIg10ZE1x82YediN+lcrPNvGMKge9yFHQD+Qho/b8y2adfugU3AsvvHfxV0fzyFllT6l9xCjf4RWY9g/Z9PJQ==} - js-base64@3.7.8: - resolution: {integrity: sha512-hNngCeKxIUQiEUN3GPJOkz4wF/YvdUdbNL9hsBcMQTkKzboD7T/q3OYOuuPZLUE6dBxSGpwhk5mwuDud7JVAow==} + js-base64@3.9.4: + resolution: {integrity: sha512-PtOMXpEGuP0RRiRXsjzHzl44dMHxSu2CPvAhinupR1tBa88me+1DPqsobl7eupn5UukjLkln1ZnAOAOXOrYG0Q==} js-tokens@4.0.0: resolution: {integrity: sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==} @@ -10557,8 +10584,8 @@ packages: resolution: {integrity: sha512-I9jwMn07Sy/IwOj3zVkVik2JTvgpaykDZEigL6Rx6N9LbMywwUSMtxET+7lVoDLLd3O3IXwJwvuuns8UB/HeAg==} engines: {node: '>=4'} - mingo@7.2.0: - resolution: {integrity: sha512-UeX942qZpofn5L97h295SkS7j/ADf7Qac8gdRCMBPxi0/1m70aeB2owLFvWbyuMj1dowonlivlVRQVDx+6h+7Q==} + mingo@7.2.4: + resolution: {integrity: sha512-KrxHKzMMguL+G1lBC6YbU5UTXdw2viwVHNTPzNYVfEQfSAoetC3M1a7rlveOksn/96oNy+QwBlIXvx/q6rWyxQ==} miniflare@4.20260317.1: resolution: {integrity: sha512-A3csI1HXEIfqe3oscgpoRMHdYlkReQKPH/g5JE53vFSjoM6YIAOGAzyDNeYffwd9oQkPWDj9xER8+vpxei8klA==} @@ -10844,6 +10871,17 @@ packages: resolution: {integrity: sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA==} engines: {node: '>=0.10.0'} + nostr-tools@2.25.2: + resolution: {integrity: sha512-7JAx+brEPmqGz1yzaK9MOxvis/Nl1B38j9hVsBSRDMRmxk5XSc0+5LlPEYbzeiUyk41RURBDcfuMzmaT754HAg==} + peerDependencies: + typescript: '>=5.0.0' + peerDependenciesMeta: + typescript: + optional: true + + nostr-wasm@0.1.0: + resolution: {integrity: sha512-78BTryCLcLYv96ONU8Ws3Q1JzjlAt+43pWQhIl86xZmWeegYCNLPml7yQ+gG3vR6V5h4XGj+TxO+SS5dsThQIA==} + npm-bundled@5.0.0: resolution: {integrity: sha512-JLSpbzh6UUXIEoqPsYBvVNVmyrjVZ1fzEFbqxKkTJQkWBO3xFzFT+KDnSKQWwOQNbuWRwt5LSD6HOTLGIWzfrw==} engines: {node: ^20.17.0 || >=22.9.0} @@ -11564,9 +11602,6 @@ packages: resolution: {integrity: sha512-9u/XQ1pvrQtYyMpZe7DXKv2p5CNvyVwzUB6uhLAnQwHMSgKMBR62lc7AHljaeteeHXn11XTAaLLUVZYVZyuRBQ==} engines: {node: '>= 20.19.0'} - reconnecting-websocket@4.4.0: - resolution: {integrity: sha512-D2E33ceRPga0NvTDhJmphEgJ7FUYF0v4lr1ki0csq06OdlxKfugGzN0dSkxM/NfqCxYELK4KcaTOUOjTV6Dcng==} - redent@3.0.0: resolution: {integrity: sha512-6tDA8g98We0zd0GvVeMT9arEOnTw9qM03L9cJXaCjrip1OO764RDBLBfrB4cwzNGDj5OA5ioymC9GkizgWJDUg==} engines: {node: '>=8'} @@ -11716,9 +11751,9 @@ packages: run-parallel@1.2.0: resolution: {integrity: sha512-5l4VyZR86LZ/lDxZTR6jqL8AFE2S0IFLMP26AbjsLVADxHdhB/c0GUsH+y39UfCi3dzz8OlQuPmnaJOMoDHQBA==} - rxdb@17.1.0: - resolution: {integrity: sha512-XmIGECuc5qzvdkszWkHEE5Ukes1AP9wKUwTxvdfviQ88tSYNiIu9+Zf4Ohhd9oEr/ChUcmfvh2Zs9dwOecbY3A==} - engines: {node: '>=18'} + rxdb@17.6.0: + resolution: {integrity: sha512-VmnCk/IXQ5KoUXw1ZsnsFt/DJI6DCCepE7SkHk44kybyMXczBawPuEAaShLtVQj+xrc9hVEyHOxzgPgIrnjgqQ==} + engines: {node: '>=20'} peerDependencies: '@angular/core': '*' '@preact/signals-core': '*' @@ -12773,9 +12808,6 @@ packages: util-deprecate@1.0.2: resolution: {integrity: sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==} - util@0.12.5: - resolution: {integrity: sha512-kZf/K6hEIrWHI6XqOFUiiMa+79wE/D8Q+NCNAWclkyg3b4d2k7s0QGepNjiABc+aR3N1PAyHL7p6UcLY6LmrnA==} - utils-merge@1.0.1: resolution: {integrity: sha512-pMZTvIkT1d+TFGvDOqodOclx0QWkkgi6Tdoa8gC8ffGAAqz9pzPTZWAybbsHHoED/ztMtkv/VoYTYyShUn81hA==} engines: {node: '>= 0.4.0'} @@ -13220,6 +13252,18 @@ packages: utf-8-validate: optional: true + ws@8.22.0: + resolution: {integrity: sha512-Ydggc987+RO0AnWtZ/7Wq9FtNvcrL1b/RO0ud9mWjUPgDrsAAwQSF51sm2hm1XofbU/4jkpGEsLFsZZxU+1DOg==} + engines: {node: '>=10.0.0'} + peerDependencies: + bufferutil: ^4.0.1 + utf-8-validate: '>=5.0.2' + peerDependenciesMeta: + bufferutil: + optional: true + utf-8-validate: + optional: true + xcode@3.0.1: resolution: {integrity: sha512-kCz5k7J7XbJtjABOvkc5lJmkiDh8VhjVCGNiqdKCscmVpdVUpEAyXv1xmCLkQJ5dsHqx3IPO4XW+NTDhU/fatA==} engines: {node: '>=10.0.0'} @@ -13311,8 +13355,8 @@ packages: youch@4.1.0-beta.10: resolution: {integrity: sha512-rLfVLB4FgQneDr0dv1oddCVZmKjcJ6yX6mS4pU82Mq/Dt9a3cLZQ62pDBL4AUO+uVrCvtWz3ZFUL2HFAFJ/BXQ==} - z-schema@12.1.0: - resolution: {integrity: sha512-+wLKQ0M58ATDhGxV1rrFHAvOJ8l1p+3VX43OFPpZ1tpTL67m4auvzHKrRtDpnaZnEiCkhInUoRPxewV6moj0uw==} + z-schema@12.4.6: + resolution: {integrity: sha512-jFh7aWwpGeyBAHFcwVPQ1GtS4d1AwhnTmSBtc/m/WtlA5XqCLlzAXu7rIZhKFXQ/iDfg8vEDQxmfWPh0OEJLAw==} engines: {node: '>=22.0.0'} hasBin: true @@ -13338,9 +13382,7 @@ packages: snapshots: - '@0no-co/graphql.web@1.2.0(graphql@16.13.2)': - optionalDependencies: - graphql: 16.13.2 + '@0no-co/graphql.web@1.2.0': {} '@acemir/cssom@0.9.30': {} @@ -14227,10 +14269,12 @@ snapshots: transitivePeerDependencies: - supports-color - '@babel/runtime@7.28.6': {} - '@babel/runtime@7.29.2': {} + '@babel/runtime@7.29.7': {} + + '@babel/runtime@8.0.5': {} + '@babel/template@7.29.7': dependencies: '@babel/code-frame': 7.29.7 @@ -15144,9 +15188,9 @@ snapshots: '@exodus/bytes@1.7.0': {} - '@expo/cli@0.24.24(graphql@16.13.2)': + '@expo/cli@0.24.24': dependencies: - '@0no-co/graphql.web': 1.2.0(graphql@16.13.2) + '@0no-co/graphql.web': 1.2.0 '@babel/runtime': 7.29.2 '@expo/code-signing-certificates': 0.0.6 '@expo/config': 11.0.13 @@ -15165,8 +15209,8 @@ snapshots: '@expo/ws-tunnel': 1.0.6 '@expo/xcpretty': 4.4.0 '@react-native/dev-middleware': 0.79.6 - '@urql/core': 5.2.0(graphql@16.13.2) - '@urql/exchange-retry': 1.3.2(@urql/core@5.2.0(graphql@16.13.2)) + '@urql/core': 5.2.0 + '@urql/exchange-retry': 1.3.2(@urql/core@5.2.0) accepts: 1.3.8 arg: 5.0.2 better-opn: 3.0.2 @@ -15469,7 +15513,7 @@ snapshots: '@expo/dom-webview@55.0.3(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0)': dependencies: - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) react: 19.0.0 react-native: 0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0) optional: true @@ -16611,10 +16655,12 @@ snapshots: '@tybys/wasm-util': 0.10.1 optional: true - '@neon/realtime@0.1.2': + '@neon/realtime@0.1.2(ws@8.22.0)': dependencies: base64-js: 1.5.1 postgres-array: 3.0.4 + optionalDependencies: + ws: 8.22.0 '@next/env@16.4.1': {} @@ -16642,8 +16688,16 @@ snapshots: '@next/swc-win32-x64-msvc@16.4.1': optional: true + '@noble/ciphers@2.1.1': {} + '@noble/ciphers@2.4.0': {} + '@noble/curves@2.0.1': + dependencies: + '@noble/hashes': 2.0.1 + + '@noble/hashes@2.0.1': {} + '@noble/hashes@2.4.0': {} '@nodelib/fs.scandir@2.1.5': @@ -17726,6 +17780,19 @@ snapshots: transitivePeerDependencies: - chokidar + '@scure/base@2.0.0': {} + + '@scure/bip32@2.0.1': + dependencies: + '@noble/curves': 2.0.1 + '@noble/hashes': 2.0.1 + '@scure/base': 2.0.0 + + '@scure/bip39@2.0.1': + dependencies: + '@noble/hashes': 2.0.1 + '@scure/base': 2.0.0 + '@shikijs/engine-oniguruma@3.13.0': dependencies: '@shikijs/types': 3.13.0 @@ -18667,8 +18734,6 @@ snapshots: dependencies: '@types/deep-eql': 4.0.2 - '@types/clone@2.1.4': {} - '@types/common-tags@1.8.1': {} '@types/connect@3.4.38': @@ -19049,16 +19114,16 @@ snapshots: '@unrs/resolver-binding-win32-x64-msvc@1.11.1': optional: true - '@urql/core@5.2.0(graphql@16.13.2)': + '@urql/core@5.2.0': dependencies: - '@0no-co/graphql.web': 1.2.0(graphql@16.13.2) + '@0no-co/graphql.web': 1.2.0 wonka: 6.3.5 transitivePeerDependencies: - graphql - '@urql/exchange-retry@1.3.2(@urql/core@5.2.0(graphql@16.13.2))': + '@urql/exchange-retry@1.3.2(@urql/core@5.2.0)': dependencies: - '@urql/core': 5.2.0(graphql@16.13.2) + '@urql/core': 5.2.0 wonka: 6.3.5 '@vitejs/plugin-basic-ssl@2.1.0(vite@7.3.6(@types/node@25.2.2)(jiti@2.7.0)(lightningcss@1.33.0)(sass@1.90.0)(terser@5.51.2)(tsx@4.21.0)(yaml@2.8.1))': @@ -19931,9 +19996,9 @@ snapshots: dependencies: fill-range: 7.1.1 - broadcast-channel@7.3.0: + broadcast-channel@7.4.0: dependencies: - '@babel/runtime': 7.28.6 + '@babel/runtime': 7.29.7 oblivious-set: 2.0.0 p-queue: 6.6.2 unload: 2.4.1 @@ -20187,7 +20252,7 @@ snapshots: commander@13.1.0: {} - commander@14.0.3: + commander@15.0.0: optional: true commander@2.20.3: {} @@ -20453,7 +20518,7 @@ snapshots: devalue@5.9.4: {} - dexie@4.0.10: {} + dexie@4.4.2: {} diff@8.0.4: {} @@ -21155,7 +21220,7 @@ snapshots: expo-asset@11.1.7(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0): dependencies: '@expo/image-utils': 0.7.6 - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) expo-constants: 17.1.8(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)) react: 19.0.0 react-native: 0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0) @@ -21188,7 +21253,7 @@ snapshots: dependencies: '@expo/config': 11.0.13 '@expo/env': 1.0.7 - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) react-native: 0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0) transitivePeerDependencies: - supports-color @@ -21235,7 +21300,7 @@ snapshots: expo-file-system@18.1.11(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)): dependencies: - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) react-native: 0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0) expo-file-system@55.0.11(expo@55.0.8)(react-native@0.83.2(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.2.0)): @@ -21250,7 +21315,7 @@ snapshots: expo-font@13.3.2(expo@53.0.26)(react@19.0.0): dependencies: - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) fontfaceobserver: 2.3.0 react: 19.0.0 @@ -21270,7 +21335,7 @@ snapshots: expo-keep-awake@14.1.4(expo@53.0.26)(react@19.0.0): dependencies: - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) react: 19.0.0 expo-keep-awake@55.0.4(expo@55.0.8)(react@19.2.0): @@ -21361,7 +21426,7 @@ snapshots: '@react-navigation/native': 7.1.27(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) '@react-navigation/native-stack': 7.9.1(@react-navigation/native@7.1.27(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0))(react-native-safe-area-context@5.4.0(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0))(react-native-screens@4.11.1(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) client-only: 0.0.1 - expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) + expo: 53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) expo-constants: 17.1.8(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)) expo-linking: 7.1.7(expo@53.0.26)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) invariant: 2.2.4 @@ -21467,10 +21532,10 @@ snapshots: react-native-edge-to-edge: 1.6.0(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) react-native-is-edge-to-edge: 1.2.1(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0) - expo@53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(graphql@16.13.2)(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0): + expo@53.0.26(@babel/core@7.29.7)(@expo/dom-webview@55.0.3)(@expo/metro-runtime@5.0.5(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0)))(react-native@0.79.6(@babel/core@7.29.7)(@types/react@19.2.13)(react@19.0.0))(react@19.0.0): dependencies: '@babel/runtime': 7.29.2 - '@expo/cli': 0.24.24(graphql@16.13.2) + '@expo/cli': 0.24.24 '@expo/config': 11.0.13 '@expo/config-plugins': 10.1.2 '@expo/fingerprint': 0.13.4 @@ -21998,11 +22063,13 @@ snapshots: graphemer@1.4.0: {} - graphql-ws@5.16.2(graphql@16.13.2): + graphql-ws@6.3.0(graphql@17.0.2)(ws@8.22.0): dependencies: - graphql: 16.13.2 + graphql: 17.0.2 + optionalDependencies: + ws: 8.22.0 - graphql@16.13.2: {} + graphql@17.0.2: {} h3@2.0.1-rc.20: dependencies: @@ -22215,11 +22282,6 @@ snapshots: ipaddr.js@1.9.1: {} - is-arguments@1.2.0: - dependencies: - call-bound: 1.0.4 - has-tostringtag: 1.0.2 - is-array-buffer@3.0.5: dependencies: call-bind: 1.0.8 @@ -22575,7 +22637,7 @@ snapshots: jose@6.2.13: {} - js-base64@3.7.8: {} + js-base64@3.9.4: {} js-tokens@4.0.0: {} @@ -23298,7 +23360,7 @@ snapshots: min-indent@1.0.1: {} - mingo@7.2.0: {} + mingo@7.2.4: {} miniflare@4.20260317.1: dependencies: @@ -23575,6 +23637,20 @@ snapshots: normalize-path@3.0.0: {} + nostr-tools@2.25.2(typescript@5.9.3): + dependencies: + '@noble/ciphers': 2.1.1 + '@noble/curves': 2.0.1 + '@noble/hashes': 2.0.1 + '@scure/base': 2.0.0 + '@scure/bip32': 2.0.1 + '@scure/bip39': 2.0.1 + nostr-wasm: 0.1.0 + optionalDependencies: + typescript: 5.9.3 + + nostr-wasm@0.1.0: {} + npm-bundled@5.0.0: dependencies: npm-normalize-package-bin: 5.0.0 @@ -24555,8 +24631,6 @@ snapshots: readdirp@5.0.0: {} - reconnecting-websocket@4.4.0: {} - redent@3.0.0: dependencies: indent-string: 4.0.0 @@ -24795,44 +24869,43 @@ snapshots: dependencies: queue-microtask: 1.2.3 - rxdb@17.1.0(@angular/core@20.3.34(@angular/compiler@20.3.34)(rxjs@7.8.2)(zone.js@0.15.1))(react@19.2.4)(rxjs@7.8.2)(vue@3.5.28(typescript@5.9.3)): + rxdb@17.6.0(@angular/core@20.3.34(@angular/compiler@20.3.34)(rxjs@7.8.2)(zone.js@0.15.1))(react@19.2.4)(rxjs@7.8.2)(typescript@5.9.3)(vue@3.5.28(typescript@5.9.3)): dependencies: - '@babel/runtime': 7.29.2 - '@types/clone': 2.1.4 - '@types/cors': 2.8.19 - '@types/express': 5.0.6 + '@babel/runtime': 8.0.5 '@types/simple-peer': 9.11.9 '@types/ws': 8.18.1 - ajv: 8.18.0 - ajv-formats: 3.0.1(ajv@8.18.0) + ajv: 8.20.0 + ajv-formats: 3.0.1(ajv@8.20.0) array-push-at-sort-position: 5.0.0 as-typed: 1.3.2 - broadcast-channel: 7.3.0 + broadcast-channel: 7.4.0 crypto-js: 4.2.0 custom-idle-queue: 5.0.2 - dexie: 4.0.10 + dexie: 4.4.2 event-reduce-js: 6.0.0 get-graphql-from-jsonschema: 8.1.0 - graphql: 16.13.2 - graphql-ws: 5.16.2(graphql@16.13.2) + graphql: 17.0.2 + graphql-ws: 6.3.0(graphql@17.0.2)(ws@8.22.0) is-my-json-valid: 2.20.6 - js-base64: 3.7.8 + js-base64: 3.9.4 jsonschema-key-compression: 1.7.0 - mingo: 7.2.0 + mingo: 7.2.4 + nostr-tools: 2.25.2(typescript@5.9.3) oblivious-set: 2.0.0 - reconnecting-websocket: 4.4.0 rxjs: 7.8.2 simple-peer: 9.11.1 - util: 0.12.5 - ws: 8.20.0 - z-schema: 12.1.0 + ws: 8.22.0 + z-schema: 12.4.6 optionalDependencies: '@angular/core': 20.3.34(@angular/compiler@20.3.34)(rxjs@7.8.2)(zone.js@0.15.1) react: 19.2.4 vue: 3.5.28(typescript@5.9.3) transitivePeerDependencies: + - '@fastify/websocket' - bufferutil + - crossws - supports-color + - typescript - utf-8-validate rxjs@7.8.2: @@ -26038,14 +26111,6 @@ snapshots: util-deprecate@1.0.2: {} - util@0.12.5: - dependencies: - inherits: 2.0.4 - is-arguments: 1.2.0 - is-generator-function: 1.1.0 - is-typed-array: 1.1.15 - which-typed-array: 1.1.19 - utils-merge@1.0.1: {} uuid@13.0.2: {} @@ -26487,6 +26552,8 @@ snapshots: ws@8.21.0: {} + ws@8.22.0: {} + xcode@3.0.1: dependencies: simple-plist: 1.3.1 @@ -26583,13 +26650,13 @@ snapshots: cookie: 1.1.1 youch-core: 0.3.3 - z-schema@12.1.0: + z-schema@12.4.6: dependencies: punycode: 2.3.1 safe-regex2: 5.1.0 validator: 13.15.35 optionalDependencies: - commander: 14.0.3 + commander: 15.0.0 zimmerframe@1.1.5: {} From c66042840d7fecff0f0ec74d057183bae588ec2c Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 09:48:41 -0700 Subject: [PATCH 08/28] test(db-adapter-conformance): field updates and confirmation through sync TrailBase and PowerSync send updates as changed fields to a SQL service and confirm each write through their change stream. The mutation oracle now models both: - `field-updates` (the law set's MUT-8 clause): an update sends only its changed fields, as core's `changes` holds them, and the provider merges them into its row as it is when the write is accepted. - `undefinedField: 'null'`: a SQL row. A removed field is sent as `null` and stored as `NULL`, which the provider reports as absent; a complete row sends every column. - `held-echoes` and a `deliver` step: the provider can answer an accepted write before its change feed delivers the copy. Under MUT-7 the mutation stays pending, holding later writes (MUT-10), until it does. Requests now compare by the fields they carry. A reach count that credited a rejected write as `*-accepted` when the driver declares no rejection kinds is fixed. - Law: MUT-7, MUT-8 under `field-updates`, MUT-10 (law set v18). - Reach: new state `echo-held`; a SQL-provider reference campaign meets every declared state. - RED/mutant: under the SQL provider, three reference mutants fail: an update that sends the complete row (MUT-8), one that sends a removal as JSON as it is so it never arrives (MUT-8), and a mutation function that settles when the provider answers (MUT-1). - GREEN: db-adapter-conformance 102 passed; RxDB mutation campaigns pass. - Limits: no multi-mutation transaction, so no mutation order within one; no newer backend value in a disjoint field; no server-generated key. Co-authored-by: Isaac --- .../src/mutation-oracle.ts | 293 ++++++++++++++---- .../tests/reference-mutation-adapter.ts | 240 ++++++++++---- ...reference-mutation-oracle.property.test.ts | 25 +- .../tests/mutation-conformance-driver.ts | 1 + 4 files changed, 447 insertions(+), 112 deletions(-) diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 53b9fd6603..828e764d03 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -92,6 +92,17 @@ export type MutationCapability = * server defaults, triggers, and normalization do (`ServerRewrite`). */ | `server-rewrites` + /** + * Updates send only the changed fields, and the provider merges them into + * its row (the law set's `field-updates`, MUT-8). + */ + | `field-updates` + /** + * The provider can answer an accepted write before its change feed + * delivers the copy, which it then delivers at `deliver`. Declare it only + * for an adapter that confirms writes through sync (MUT-7). + */ + | `held-echoes` /** The optimistic transaction a write call returns. */ export interface MutationReceipt { @@ -112,8 +123,11 @@ export interface MutationWriter { export interface WriteRequest { readonly type: `insert` | `update` | `delete` readonly id: number - /** The row the request carries; absent for a delete. */ - readonly row?: ConformanceRow + /** + * The row the request carries; absent for a delete. Under + * `field-updates`, an update carries only its changed fields. + */ + readonly row?: Readonly> } /** @@ -156,6 +170,11 @@ export interface WritePlan { readonly rewrite: ServerRewrite /** Answer only at `release`, so the write stays pending until then. */ readonly held: boolean + /** + * After accepting, deliver the copy through the change feed only at + * `deliver` (`held-echoes`). + */ + readonly echoHeld: boolean /** * How the provider rejects, one of the driver's `rejectionKinds`; absent * when the driver declares none. @@ -177,6 +196,8 @@ export interface MutationProvider { plan: (plan: WritePlan) => void /** Answer the oldest held request by its plan. */ release: () => void | Promise + /** Deliver the oldest withheld copy through the change feed (`held-echoes`). */ + deliver?: () => void | Promise /** Commit a change made by another client (`foreign-commits`). */ foreignCommit?: (change: ProviderChange) => void | Promise /** The provider's committed rows. */ @@ -219,9 +240,11 @@ export interface MutationDriver { * as an update that removes a field authors it. FID-1 parameterizes row * values by the adapter's value domain: `own` keeps the field as an own * `undefined`, as a structured clone does; `absent` drops it, as a JSON - * document must, since JSON has no `undefined`. Default `own`. + * document must, since JSON has no `undefined`; `null` sends it as `null` + * and stores `NULL`, which the provider's rows report as an absent field, + * as a SQL row does. Default `own`. */ - readonly undefinedField?: `own` | `absent` + readonly undefinedField?: `own` | `absent` | `null` /** Console diagnostics the adapter declares (HEALTH-5). */ readonly declaredDiagnostic?: ( level: `error` | `warn`, @@ -251,6 +274,8 @@ export interface MutationDriver { * declares several, is held only with `held-writes`, and rewrites an * accepted insert's or update's row only with `server-rewrites`. * - `release`: the provider answers the oldest held request (`held-writes`). + * - `deliver`: the provider delivers the oldest copy it answered but + * withheld from its change feed (`held-echoes`). * - `foreign`: another client commits an upsert or a delete while no * mutation is pending (`foreign-commits`). * @@ -270,11 +295,13 @@ export type MutationStep = readonly note?: string readonly outcome: `accept` | `reject` readonly held: boolean + readonly echoHeld: boolean readonly rewrite: ServerRewrite /** One of the driver's `rejectionKinds`, for a rejection. */ readonly rejectionKind?: string } | { readonly kind: `release` } + | { readonly kind: `deliver` } | { readonly kind: `foreign`; readonly change: ProviderChange } export interface MutationHistory { @@ -313,6 +340,9 @@ export function mutationHistoryArb( ) : fc.constant(`accept` as const), held: capabilities.has(`held-writes`) ? fc.boolean() : fc.constant(false), + echoHeld: capabilities.has(`held-echoes`) + ? fc.boolean() + : fc.constant(false), rewrite: capabilities.has(`server-rewrites`) ? fc.constantFrom( `none`, @@ -329,7 +359,15 @@ export function mutationHistoryArb( : fc.constant(undefined), }) .map( - ({ type, row, outcome, held, rewrite, rejectionKind }): MutationStep => ({ + ({ + type, + row, + outcome, + held, + echoHeld, + rewrite, + rejectionKind, + }): MutationStep => ({ kind: `write`, type, id: row.id, @@ -337,6 +375,7 @@ export function mutationHistoryArb( ...(row.note === undefined ? {} : { note: row.note }), outcome, held, + echoHeld: outcome === `accept` && echoHeld, rewrite: type === `delete` ? `none` : rewrite, ...(outcome === `reject` && rejectionKind !== undefined ? { rejectionKind } @@ -360,6 +399,7 @@ export function mutationHistoryArb( title: step.title, outcome: step.outcome, held: step.held, + echoHeld: step.echoHeld, rewrite: step.rewrite, ...(step.rejectionKind === undefined ? {} @@ -375,6 +415,12 @@ export function mutationHistoryArb( arbitrary: fc.constant({ kind: `release` as const }), }) } + if (capabilities.has(`held-echoes`)) { + steps.push({ + weight: 3, + arbitrary: fc.constant({ kind: `deliver` as const }), + }) + } if (capabilities.has(`foreign-commits`)) { steps.push({ weight: 2, @@ -431,6 +477,30 @@ function sameRows(a: RowState, b: RowState): boolean { ) } +/** + * A request's row as the model compares it: each field it carries other than + * the key, in name order, with `undefined` as `null`, since a request is + * compared by the fields it sends. + */ +function requestValue(fields: Readonly>): string { + return JSON.stringify( + Object.keys(fields) + .filter((field) => field !== `id`) + .sort() + .map((field) => [field, fields[field as keyof ConformanceRow] ?? null]), + ) +} + +/** A model row value back as a row. */ +function rowOf(id: number, value: string): ConformanceRow { + const [title, note] = JSON.parse(value) as [string, Array] + return { + id, + title, + ...(note.length === 0 ? {} : { note: note[0] ?? undefined }), + } +} + /** A mutation whose plan has not decided it yet. */ interface PendingWrite { readonly index: number @@ -438,12 +508,16 @@ interface PendingWrite { readonly id: number /** The authored row's model value; absent for a delete. */ readonly value?: string - /** The row its request carries: the authored row in the provider's value domain. */ - readonly sent?: string - /** The row the provider stores if it accepts: `sent` under the plan's rewrite. */ - readonly copy?: string + /** + * The fields its request carries, in the provider's value domain: the + * authored row, or under `field-updates` an update's changed fields. + * Absent for a delete. + */ + readonly sent?: Readonly> readonly plan: WritePlan requested: boolean + /** Accepted, with its copy withheld from the change feed (`held-echoes`). */ + answered: boolean } type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } @@ -453,17 +527,20 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } * * - `provider` is the provider's committed rows: the seed, every accepted * write's copy, and every commit from another client, in commit order. A - * write's copy is its authored row under the driver's `undefinedField` - * rule and then its plan's `ServerRewrite`: after a rewrite the - * provider's copy, not the authored row, is the row's value (FID-5). + * write's copy is the row it sends (an update under `field-updates` + * merges its fields into the provider's row as it is when the write is + * accepted), stored under the driver's `undefinedField` rule and then the + * plan's `ServerRewrite`: after a rewrite the provider's copy, not the + * authored row, is the row's value (FID-5). * - `rewritten` counts accepted writes whose copy differs from their * authored row, for reach. - * - `pending` lists the mutations not yet decided, in author order. MUT-10 + * - `pending` lists the mutations not yet settled, in author order. MUT-10 * orders their requests: the first is requested at once, and each later - * one only after every earlier one is decided. + * one only after every earlier one is decided. An accepted write whose + * copy the provider withholds is `answered`: it stays pending, and so + * holds later writes, until the copy arrives (MUT-7). * - `requests` is every write request MUT-8 and MUT-10 allow so far. * - `settlements` is each write's `when('settled')` settlement. - * * - `applied` is the synced rows core has applied. An accepted write's * provider copy reaches core as an accepted sync transaction, which is * held while any optimistic transaction is persisting, so `applied` takes @@ -481,7 +558,9 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } */ interface MutationModel { /** The driver's `undefinedField` declaration. */ - readonly undefinedField: `own` | `absent` + readonly undefinedField: `own` | `absent` | `null` + /** The driver declares `field-updates`. */ + readonly fieldUpdates: boolean provider: Map applied: Map heldRows: Array @@ -494,10 +573,12 @@ interface MutationModel { function initialModel( seed: ReadonlyArray, undefinedField: MutationModel[`undefinedField`] = `own`, + fieldUpdates = false, ): MutationModel { const provider = new Map(seed.map((row) => [row.id, rowValue(row)])) return { undefinedField, + fieldUpdates, provider, applied: new Map(provider), heldRows: [], @@ -534,13 +615,60 @@ function authoredRow( } } -/** The provider's copy of an authored row under its `undefinedField` rule. */ -function storedRow( - row: ConformanceRow, +/** + * The fields a request sends for `fields` under the `undefinedField` rule: + * `own` keeps an own `undefined`, `absent` drops it, and `null` sends it as + * `null`. Under `null` a complete row is a SQL row, so it sends every + * column, an absent `note` as `null`; changed fields send only themselves. + */ +function sentFields( + fields: Readonly>, undefinedField: MutationModel[`undefinedField`], -): ConformanceRow { - if (undefinedField === `own` || row.note !== undefined) return row - return { id: row.id, title: row.title } + complete: boolean, +): Partial { + if (undefinedField === `own`) return { ...fields } + if (fields.note !== undefined) return { ...fields } + if (!complete && !Object.hasOwn(fields, `note`)) return { ...fields } + const { note: _removed, ...rest } = fields + return undefinedField === `null` + ? { ...rest, note: null as unknown as undefined } + : rest +} + +/** + * The fields an update changes, as core's `changes` holds them: each authored + * field that differs from the row it was authored over, a removed field as + * an own `undefined`. + */ +function changedFields( + authored: ConformanceRow, + over: ConformanceRow, +): Partial { + return { + id: authored.id, + ...(authored.title === over.title ? {} : { title: authored.title }), + ...(Object.hasOwn(authored, `note`) && + (authored.note !== over.note || !Object.hasOwn(over, `note`)) + ? { note: authored.note } + : {}), + } +} + +/** The provider's copy of an accepted insert or update. */ +function copyOf(state: MutationModel, write: PendingWrite): ConformanceRow { + const sent = write.sent! + const base = + write.type === `update` && state.fieldUpdates + ? rowOf(write.id, state.provider.get(write.id)!) + : { id: write.id } + const merged = { ...base, ...sent } as ConformanceRow + const { note, ...rest } = merged + // A stored `undefined` or `NULL` is an absent field, except to a + // structured-clone provider, which keeps an own `undefined`. + const kept = + (note ?? undefined) !== undefined || + (state.undefinedField === `own` && Object.hasOwn(merged, `note`)) + return rewriteRow(kept ? merged : rest, write.plan.rewrite) } function fold( @@ -560,17 +688,35 @@ function visible(state: MutationModel): RowState { return rowState(fold(fold(state.applied, state.heldRows), state.pending)) } -/** Decide a write by its plan: an accepted one enters the provider's rows. */ +/** + * Decide a write by its plan. An accepted one enters the provider's rows; it + * settles at once unless the provider withholds its copy, and then stays + * pending, answered, until `confirm` (MUT-7). A rejected one settles. + */ function decide(state: MutationModel, write: PendingWrite): void { - if (write.plan.outcome === `accept`) { - if (write.type === `delete`) state.provider.delete(write.id) - else state.provider.set(write.id, write.copy!) - if (write.copy !== write.value) state.rewritten++ - state.heldRows.push(write) - state.settlements[write.index] = `fulfilled` - } else { + if (write.plan.outcome === `reject`) { state.settlements[write.index] = { rejected: write.plan.error } + retire(state, write) + return } + if (write.type === `delete`) state.provider.delete(write.id) + else { + const copy = rowValue(copyOf(state, write)) + state.provider.set(write.id, copy) + if (copy !== write.value) state.rewritten++ + } + if (write.plan.echoHeld) write.answered = true + else confirm(state, write) +} + +/** An accepted write's copy arrives: it fulfills. */ +function confirm(state: MutationModel, write: PendingWrite): void { + state.heldRows.push(write) + state.settlements[write.index] = `fulfilled` + retire(state, write) +} + +function retire(state: MutationModel, write: PendingWrite): void { state.pending.splice(state.pending.indexOf(write), 1) // The last persisting transaction settled: every held sync transaction // applies in its publication. @@ -582,20 +728,23 @@ function decide(state: MutationModel, write: PendingWrite): void { /** * MUT-10: request the first pending mutation if it has not been; a request - * answered at once is decided at once, which admits the next. Returns the - * public rows after each decision, for the publication check. + * answered at once is decided at once, which admits the next unless its + * copy is withheld. Returns the public rows after each decision, for the + * publication check. */ function advance(state: MutationModel): Array { const states: Array = [] for (;;) { const first = state.pending[0] - if (!first) return states + if (!first || first.answered) return states if (!first.requested) { first.requested = true state.requests.push({ type: first.type, id: first.id, - ...(first.sent === undefined ? {} : { value: first.sent }), + ...(first.sent === undefined + ? {} + : { value: requestValue(first.sent) }), }) } if (first.plan.held) return states @@ -616,6 +765,7 @@ type StepDone = removesField: boolean } | { kind: `release`; outcome: WritePlan[`outcome`] } + | { kind: `deliver` } | { kind: `foreign` } /** @@ -634,29 +784,29 @@ function stepModel( const busy = state.pending.some((write) => write.id === step.id) const legal = step.type === `insert` ? !held.has(step.id) : held.has(step.id) - const row = - step.type === `delete` - ? undefined - : authoredRow(step, held.get(step.id)) + const over = held.get(step.id) + const row = step.type === `delete` ? undefined : authoredRow(step, over) const value = row && rowValue(row) // An update that changes no field records no mutation in core; no law // fixes its persistence, so the grammar leaves it out. - const noop = step.type === `update` && held.get(step.id) === value + const noop = step.type === `update` && over === value if (busy || !legal || noop) return undefined const queued = state.pending.length > 0 + const fields = + row && step.type === `update` && state.fieldUpdates + ? changedFields(row, rowOf(step.id, over!)) + : row const write: PendingWrite = { index: state.settlements.length, type: step.type, id: step.id, ...(row && { value: rowValue(row), - sent: rowValue(storedRow(row, state.undefinedField)), - copy: rowValue( - rewriteRow(storedRow(row, state.undefinedField), step.rewrite), - ), + sent: sentFields(fields!, state.undefinedField, fields === row), }), plan: plan(step), requested: false, + answered: false, } state.settlements.push(`pending`) state.pending.push(write) @@ -679,7 +829,8 @@ function stepModel( } case `release`: { const first = state.pending[0] - if (!first?.requested || !first.plan.held) return undefined + if (!first?.requested || first.answered || !first.plan.held) + return undefined decide(state, first) const published = [visible(state), ...advance(state)] return { @@ -687,6 +838,13 @@ function stepModel( published, } } + case `deliver`: { + const first = state.pending[0] + if (!first?.answered) return undefined + confirm(state, first) + const published = [visible(state), ...advance(state)] + return { done: { kind: `deliver` }, published } + } case `foreign`: { if (state.pending.length > 0) return undefined if (step.change.type === `upsert`) @@ -733,6 +891,7 @@ export const mutationReachStates = [ `field-removed`, `held-optimistic-row`, `server-rewrote-copy`, + `echo-held`, `quiescent-after-writes`, ] as const @@ -750,6 +909,7 @@ const capabilityOfState: Partial< 'foreign-commit': `foreign-commits`, 'held-optimistic-row': `held-writes`, 'server-rewrote-copy': `server-rewrites`, + 'echo-held': `held-echoes`, } /** The declared states a driver's capabilities let histories reach. */ @@ -901,7 +1061,11 @@ export async function checkMutationHistory( } const fixture = created const { collection, provider, writer } = fixture - const state = initialModel(history.seed, driver.undefinedField) + const state = initialModel( + history.seed, + driver.undefinedField, + driver.capabilities.has(`field-updates`), + ) const receipts: Array<{ current: Settlement }> = [] const failures: Array = [] let accepted = 0 @@ -929,7 +1093,9 @@ export async function checkMutationHistory( const requests = provider.requests().map((request) => ({ type: request.type, id: request.id, - ...(request.row === undefined ? {} : { value: rowValue(request.row) }), + ...(request.row === undefined + ? {} + : { value: requestValue(request.row) }), })) if (JSON.stringify(requests) !== JSON.stringify(state.requests)) throw violation( @@ -949,6 +1115,11 @@ export async function checkMutationHistory( visible(state), rows, ) + const answered = new Set( + state.pending + .filter((write) => write.answered) + .map((write) => write.index), + ) state.settlements.forEach((expected, index) => { const got = receipts[index]!.current const ok = @@ -959,7 +1130,9 @@ export async function checkMutationHistory( throw violation( typeof expected === `object` ? `MUT-2 (settlement)` - : `MUT-13 (settlement)`, + : answered.has(index) + ? `MUT-7 (settlement)` + : `MUT-13 (settlement)`, stepIndex, typeof expected === `object` ? expected.rejected.message : expected, typeof got === `object` ? String(got.rejected) : got, @@ -1011,6 +1184,7 @@ export async function checkMutationHistory( lastPlan = { outcome: step.outcome, held: step.held, + echoHeld: step.echoHeld, rewrite: step.rewrite, ...(step.rejectionKind === undefined ? {} @@ -1053,10 +1227,11 @@ export async function checkMutationHistory( wrote++ if (write.held) reach(`held-write-pending`) if (done.queued) reach(`write-queued-behind-held`) - if (write.outcome === `reject`) reach(`write-rejected`) - if (write.rejectionKind !== undefined) - reach(`rejected-as-${write.rejectionKind}`) - else if (!write.held) reach(`${write.type}-accepted` as const) + if (write.outcome === `reject`) { + reach(`write-rejected`) + if (write.rejectionKind !== undefined) + reach(`rejected-as-${write.rejectionKind}`) + } else if (!write.held) reach(`${write.type}-accepted` as const) if (done.removesField && write.outcome === `accept`) reach(`field-removed`) if (write.outcome === `reject` && !write.held) law = `MUT-2` @@ -1069,6 +1244,9 @@ export async function checkMutationHistory( law = `MUT-2` } break + case `deliver`: + await provider.deliver!() + break case `foreign`: await provider.foreignCommit!( (step as Extract).change, @@ -1078,19 +1256,26 @@ export async function checkMutationHistory( } if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + if (state.pending.some((write) => write.answered)) reach(`echo-held`) rewritten = state.rewritten await compare(index) checkPublications(index, owedPublications(before, result.published), law) } - // Answer every held request, reaching a quiescent cut (FID-5). + // Answer every held request and deliver every withheld copy, reaching a + // quiescent cut (FID-5). while (state.pending.length > 0) { const before = visible(state) - const result = stepModel(state, { kind: `release` }, planFor) + const step: MutationStep = state.pending[0]!.answered + ? { kind: `deliver` } + : { kind: `release` } + const result = stepModel(state, step, planFor) if (result === undefined) break if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + if (state.pending.some((write) => write.answered)) reach(`echo-held`) rewritten = state.rewritten - await provider.release() + if (step.kind === `deliver`) await provider.deliver!() + else await provider.release() await compare(steps.length) checkPublications( steps.length, diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index 8c59d37c18..81e2392817 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -6,16 +6,32 @@ * * The adapter's sync run loads the provider's rows, marks ready, and applies * the provider's change feed. Its mutation functions send each mutation to - * the provider one write at a time per Collection (MUT-10) and return once - * the provider has answered. An accepted write reaches the change feed before - * the provider answers, so its sync transaction is committed before the - * mutation function settles, and core drops the optimistic row in the same - * publication that applies it (MUT-1). + * the provider one write at a time per Collection (MUT-10). + * + * The provider has three forms: + * + * - by default it stores structured clones and replaces a row on every + * write. An accepted write reaches the change feed before the provider + * answers, so the mutation function returns once the provider answers: + * its sync transaction is committed before the mutation function settles, + * and core drops the optimistic row in the same publication that applies + * it (MUT-1); + * - a JSON provider (`json`) stores and feeds JSON, so a field whose value is + * `undefined` is absent (`undefinedField: 'absent'`); + * - a SQL provider (`sql`), like TrailBase's or PowerSync's service, stores + * rows with every column and an absent field as `NULL`, which its rows and + * change feed report as absent (`undefinedField: 'null'`). An update sends + * only its changed fields, a removed one as `null`, and the provider merges + * them into its row (`field-updates`). The provider can answer before its + * change feed delivers the copy (`held-echoes`), so the adapter confirms + * each write through sync: its mutation function returns when its own + * sync run has applied that write's copy (MUT-7). */ import { createCollection } from '@tanstack/db' import { observedView, rewriteRow } from '../src' import type { ConformanceRow, + MutationCapability, MutationDriver, MutationFixture, MutationProvider, @@ -50,43 +66,72 @@ export type ReferenceMutationMutant = * into its synced rows, over the provider's rewritten copy. */ | `writes-back-authored` + /** MUT-8 (SQL): an update sends the complete row, not its changed fields. */ + | `sends-complete-row` + /** + * MUT-8 (SQL): an update sends its changes as JSON as they are, so a + * removed field, an own `undefined`, never reaches the provider. + */ + | `drops-removal` + /** + * MUT-7 (SQL): the mutation function returns when the provider answers, + * before its copy arrives through sync. + */ + | `settles-on-answer` + +export interface ReferenceMutationOptions { + /** A JSON provider. */ + json?: boolean + /** A SQL provider, with field updates and confirmation through sync. */ + sql?: boolean +} const acceptAtOnce: WritePlan = { outcome: `accept`, held: false, + echoHeld: false, rewrite: `none`, error: new Error(`unplanned write`), } -/** - * A provider that answers writes by plan and feeds every commit back. A JSON - * provider stores and feeds each row as JSON, so a field whose value is - * `undefined` is absent; otherwise rows are structured clones. - */ +const jsonClone = (value: T): T => JSON.parse(JSON.stringify(value)) as T + +/** A provider that answers writes by plan and feeds every commit back. */ export class ControlledMutationProvider implements MutationProvider { private readonly committed = new Map() private readonly plans: Array = [] private readonly held: Array<() => void> = [] + private readonly withheld: Array<() => void> = [] private readonly log: Array = [] - private listener: ((change: ProviderChange) => void) | undefined + private listener: + ((change: ProviderChange, sequence: number) => void) | undefined + private commits = 0 constructor( seed: ReadonlyArray, - private readonly json = false, + private readonly options: ReferenceMutationOptions, private readonly echoesAuthored = false, ) { for (const row of seed) this.committed.set(row.id, this.stored(row)) } - private stored(value: T): T { - return this.json - ? (JSON.parse(JSON.stringify(value)) as T) - : structuredClone(value) + /** A row as this provider stores it. */ + private stored(row: Partial): ConformanceRow { + if (!this.options.json && !this.options.sql) + return structuredClone(row) as ConformanceRow + const stored = jsonClone(row) as Omit & { + note?: string | null + } + // A SQL column holds `NULL`, which the provider reports as absent. + if (stored.note === null) delete stored.note + return stored as ConformanceRow } // The adapter's side: its change feed, initial read, and writes. - subscribe(listener: (change: ProviderChange) => void): () => void { + subscribe( + listener: (change: ProviderChange, sequence: number) => void, + ): () => void { this.listener = listener return () => { if (this.listener === listener) this.listener = undefined @@ -97,34 +142,63 @@ export class ControlledMutationProvider implements MutationProvider { return [...this.committed.values()].map((row) => structuredClone(row)) } - write(request: WriteRequest): Promise { - this.log.push(this.stored(request)) + /** + * Answers with the commit's sequence number, which its change feed + * carries, so an adapter can confirm the write through sync. + */ + write(request: WriteRequest): Promise { + const { json, sql } = this.options + this.log.push(json || sql ? jsonClone(request) : structuredClone(request)) const plan = this.plans.shift() ?? acceptAtOnce - return new Promise((resolve, reject) => { + return new Promise((resolve, reject) => { const answer = () => { if (plan.outcome === `reject`) { reject(plan.error) return } - this.commit( - request.type === `delete` - ? { type: `delete`, id: request.id } - : { type: `upsert`, row: rewriteRow(request.row!, plan.rewrite) }, + if (request.type === `delete`) { + resolve( + this.commit({ type: `delete`, id: request.id }, plan.echoHeld), + ) + return + } + // A SQL update merges the fields it sends into the stored row. + const row = + request.type === `update` && sql + ? { ...this.committed.get(request.id), ...jsonClone(request.row) } + : request.row + resolve( + this.commit( + { + type: `upsert`, + row: rewriteRow(row as ConformanceRow, plan.rewrite), + }, + plan.echoHeld, + ), ) - resolve() } if (plan.held) this.held.push(answer) else answer() }) } - /** A write replaces the whole row, as a complete-row API does. */ - private commit(change: ProviderChange): void { + /** + * Commits a change and feeds it back, at once or at `deliver`. Without + * field updates a write replaces the whole row, as a complete-row API does. + */ + private commit(change: ProviderChange, withhold = false): number { if (change.type === `delete`) this.committed.delete(change.id) else this.committed.set(change.row.id, this.stored(change.row)) - this.listener?.( - this.echoesAuthored ? structuredClone(change) : this.stored(change), - ) + const sequence = ++this.commits + const fed = this.echoesAuthored + ? structuredClone(change) + : change.type === `delete` + ? change + : { ...change, row: this.stored(change.row) } + const feed = () => this.listener?.(fed, sequence) + if (withhold) this.withheld.push(feed) + else feed() + return sequence } // The oracle's side. @@ -137,6 +211,10 @@ export class ControlledMutationProvider implements MutationProvider { this.held.shift()?.() } + deliver(): void { + this.withheld.shift()?.() + } + foreignCommit(change: ProviderChange): void { this.commit(change) } @@ -156,6 +234,7 @@ let sequence = 0 function referenceCollection( provider: ControlledMutationProvider, mutant: ReferenceMutationMutant | undefined, + { sql = false }: ReferenceMutationOptions, ) { // MUT-10: each write waits for the earlier one to be decided. let turn: Promise = Promise.resolve() @@ -175,12 +254,46 @@ function referenceCollection( throw error } } + // MUT-7 (SQL): a write is confirmed when this sync run has applied the + // commit its answer names. + let applied = 0 + const confirmations: Array<{ sequence: number; resolve: () => void }> = [] + const confirmed = (commit: number): Promise => { + if (!sql || mutant === `settles-on-answer` || applied >= commit) + return Promise.resolve() + return new Promise((resolve) => + confirmations.push({ sequence: commit, resolve }), + ) + } + const send = async (request: WriteRequest) => + confirmed(await provider.write(request)) // The `writes-back-authored` mutant writes through the sync run's apply. let applyChange: ((change: ProviderChange) => void) | undefined const writeBack = (row: ConformanceRow) => { if (mutant === `writes-back-authored`) applyChange?.({ type: `upsert`, row: structuredClone(row) }) } + /** The row an update request carries. */ + const updateRow = (mutation: { + key: number + modified: ConformanceRow + changes: Partial + }): Partial => { + if (sql && mutant !== `sends-complete-row`) { + const changes: Record = { + id: mutation.key, + ...mutation.changes, + } + // A removed field is `NULL` in its column. + if (mutant !== `drops-removal`) + for (const [field, value] of Object.entries(changes)) + if (value === undefined) changes[field] = null + return changes as Partial + } + return mutant === `sends-changes-only` + ? { id: mutation.key, ...mutation.changes } + : structuredClone(mutation.modified) + } const collection = createCollection({ id: `reference-mutations-${++sequence}`, getKey: (row) => row.id, @@ -210,9 +323,17 @@ function referenceCollection( void commit() markReady() applyChange = apply - return provider.subscribe((change) => { - if (mutant === `applies-copy-late`) setTimeout(() => apply(change), 0) - else apply(change) + const receive = (change: ProviderChange, commitSequence: number) => { + apply(change) + applied = commitSequence + for (const waiting of confirmations.splice(0)) + if (waiting.sequence <= applied) waiting.resolve() + else confirmations.push(waiting) + } + return provider.subscribe((change, commitSequence) => { + if (mutant === `applies-copy-late`) + setTimeout(() => receive(change, commitSequence), 0) + else receive(change, commitSequence) }) }, }, @@ -223,10 +344,13 @@ function referenceCollection( const request = { type: `insert` as const, id: mutation.key, - row: structuredClone(mutation.modified), + // A SQL row has every column: an absent field is `NULL`. + row: sql + ? { note: null as unknown as undefined, ...mutation.modified } + : structuredClone(mutation.modified), } - await provider.write(request) - if (mutant === `writes-insert-twice`) await provider.write(request) + await send(request) + if (mutant === `writes-insert-twice`) await send(request) writeBack(mutation.modified) } }), @@ -235,16 +359,10 @@ function referenceCollection( answer( persist(async () => { for (const mutation of transaction.mutations) { - await provider.write({ + await send({ type: `update`, id: mutation.key, - row: - mutant === `sends-changes-only` - ? ({ - id: mutation.key, - ...mutation.changes, - } as ConformanceRow) - : structuredClone(mutation.modified), + row: updateRow(mutation), }) writeBack(mutation.modified) } @@ -255,7 +373,7 @@ function referenceCollection( persist(async () => { if (mutant === `skips-delete`) return for (const mutation of transaction.mutations) { - await provider.write({ type: `delete`, id: mutation.key }) + await send({ type: `delete`, id: mutation.key }) } }), ), @@ -263,27 +381,39 @@ function referenceCollection( return collection } +const sqlMutants = new Set([ + `sends-complete-row`, + `drops-removal`, + `settles-on-answer`, +]) + /** A production driver for the reference adapter, optionally mutated. */ export function referenceMutationDriver( mutant?: ReferenceMutationMutant, - { json = mutant === `echoes-authored-row` }: { json?: boolean } = {}, + { + json = mutant === `echoes-authored-row`, + sql = sqlMutants.has(mutant), + }: ReferenceMutationOptions = {}, ): MutationDriver { + const capabilities: Array = [ + `write-rejection`, + `held-writes`, + `foreign-commits`, + `server-rewrites`, + ] + if (sql) capabilities.push(`field-updates`, `held-echoes`) return { - adapter: `reference mutations${json ? ` (JSON provider)` : ``}${mutant ? ` (${mutant})` : ``}`, + adapter: `reference mutations${json ? ` (JSON provider)` : ``}${sql ? ` (SQL provider)` : ``}${mutant ? ` (${mutant})` : ``}`, ...(json && { undefinedField: `absent` as const }), - capabilities: new Set([ - `write-rejection`, - `held-writes`, - `foreign-commits`, - `server-rewrites`, - ]), + ...(sql && { undefinedField: `null` as const }), + capabilities: new Set(capabilities), create(seed): MutationFixture { const provider = new ControlledMutationProvider( seed, - json, + { json, sql }, mutant === `echoes-authored-row`, ) - const collection = referenceCollection(provider, mutant) + const collection = referenceCollection(provider, mutant, { sql }) return { collection: observedView(collection, { row: (row) => row, diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts index 69e142ad05..0f8d94a178 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -32,9 +32,12 @@ import type { ReferenceMutationMutant } from './reference-mutation-adapter' * - **Campaigns.** The same property, grammar, and run budget run with a * fixed seed and without one (ORC-007). Each fails when fewer than five * histories reach a declared state (ORC-015). - * - **Value domain.** The same campaigns run with a JSON provider, which - * declares `undefinedField: 'absent'`: a field an update removes is absent - * from its request and the provider's copy (FID-1's value domain). + * - **Value domain.** The fixed campaign also runs with a JSON provider, + * which declares `undefinedField: 'absent'`: a field an update removes is + * absent from its request and the provider's copy (FID-1's value domain). + * - **SQL provider.** The fixed campaign also runs with a provider like + * TrailBase's or PowerSync's: `undefinedField: 'null'`, `field-updates`, + * and `held-echoes`, so the adapter confirms each write through sync. * - **Calibration.** Each hostile mutant of the reference adapter breaks one * law, and the fixed campaign must reject it with that law (ORC-006). */ @@ -46,6 +49,7 @@ const runs = 300 const campaignTimeoutMs = 120_000 const jsonDriver = referenceMutationDriver(undefined, { json: true }) +const sqlDriver = referenceMutationDriver(undefined, { sql: true }) async function campaign( options: fc.Parameters<[unknown]>, @@ -89,6 +93,18 @@ describe(`mutation conformance: reference adapter`, () => { campaignTimeoutMs, ) + oracleReplayTest(it)( + `conforms with a SQL provider on a fixed-seed campaign`, + async () => { + const reach = await campaign( + { seed: 2118, numRuns: oracleRuns(runs) }, + sqlDriver, + ) + reach.assertReached() + }, + campaignTimeoutMs, + ) + const mutants: Array<[ReferenceMutationMutant, string | RegExp]> = [ [`applies-copy-late`, `MUT-1`], [`swallows-rejection`, `MUT-2`], @@ -99,6 +115,9 @@ describe(`mutation conformance: reference adapter`, () => { [`sends-changes-only`, `MUT-8`], [`echoes-authored-row`, /FID-5/], [`writes-back-authored`, /FID-5/], + [`sends-complete-row`, `MUT-8 (provider requests)`], + [`drops-removal`, `MUT-8 (provider requests)`], + [`settles-on-answer`, /MUT-1|MUT-7/], ] oracleReplayTest(it).each(mutants)( `rejects the %s mutant with %s`, diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index 8103e2ce77..9d30731942 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -90,6 +90,7 @@ const schema = { const acceptAtOnce: WritePlan = { outcome: `accept`, held: false, + echoHeld: false, rewrite: `none`, error: new Error(`unplanned write`), } From 56dc1338b51bf2e9184fb25cb4324b417d025be2 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 09:48:57 -0700 Subject: [PATCH 09/28] fix(trailbase-db-collection): send removals, order writes, confirm by fresh events A receiving driver runs the TrailBase adapter on the shared mutation oracle. Its RecordApi acts as a TrailBase service with SQL rows: it records each request as JSON, as the SDK sends it, merges updates field by field, can rewrite stored rows, and sends change events at once or after it answers. The oracle found three defects: - A removed field (`draft.note = undefined`) was sent as an empty patch, since JSON drops `undefined`, so the service kept the old value. It is now sent as `null`, the column's NULL. - Writes did not wait for the earlier one. They now run in author order; a write starts at once when no other is admitted. - `awaitIds` accepted any event for the key within the last 300 seconds, so an earlier event confirmed a later write and its optimistic row dropped, showing the old value, before its own event arrived. A write now waits for an event after it began. Events are recorded only while a write is admitted and cleared after, which removes the periodic cleanup timer. - Law: MUT-8 (field updates send the authored fields), MUT-10, MUT-1 and MUT-7 (law set v18). - Reach: every declared mutation state, at least five histories each, in both campaigns. - RED: on the parent trailbase.ts, of the fixed campaign's 300 histories, 37 failed MUT-8, 27 failed MUT-10, and 5 failed MUT-1. - GREEN: both campaigns pass; trailbase-db-collection 154 passed. - Mutant: restoring the old confirmation fails MUT-1; the parent fails MUT-8 and MUT-10. - Test change: three `trailbase.test.ts` cases asserted that the removed cleanup interval was installed and cleared; those assertions are gone, and the no-unhandled-rejection and release checks stay. - Limits: removal by assigning `null` is not generated; the record API bypasses the SDK's fetch and SSE decoding; multi-mutation transactions still persist their mutations concurrently. - Size: trailbase-db-collection -46 gzip bytes. Co-authored-by: Isaac --- .../trailbase-ordered-confirmed-writes.md | 9 + docs/contributing/oracle-coverage.md | 54 +++- packages/db-adapter-conformance/MIGRATION.md | 47 +++- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../trailbase-db-collection/src/trailbase.ts | 180 ++++++------ .../tests/mutation-conformance-driver.ts | 261 ++++++++++++++++++ ...tation-conformance-oracle.property.test.ts | 75 +++++ .../tests/trailbase.test.ts | 7 - 9 files changed, 507 insertions(+), 134 deletions(-) create mode 100644 .changeset/trailbase-ordered-confirmed-writes.md create mode 100644 packages/trailbase-db-collection/tests/mutation-conformance-driver.ts create mode 100644 packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts diff --git a/.changeset/trailbase-ordered-confirmed-writes.md b/.changeset/trailbase-ordered-confirmed-writes.md new file mode 100644 index 0000000000..403bde458b --- /dev/null +++ b/.changeset/trailbase-ordered-confirmed-writes.md @@ -0,0 +1,9 @@ +--- +'@tanstack/trailbase-db-collection': patch +--- + +Fix three write bugs. + +- Removing a field in an update (for example `draft.note = undefined`) now sends `null`, so TrailBase clears the column. Before, the update omitted the field, and TrailBase kept the old value. +- Writes now reach TrailBase in the order the application made them. A write starts only after the earlier write on the same collection is confirmed or rejected. +- A write is now confirmed only by a change event that arrives after the write began. Before, an earlier event for the same record could confirm a later write, so its optimistic value dropped and the old value showed until the new event arrived. This change also removes the adapter's periodic cleanup timer. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 090cad3234..18d330fd0b 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7218,10 +7218,12 @@ restarted run empty. A fixed-seed campaign runs the React client Owner: `packages/db-adapter-conformance/src/mutation-oracle.ts`, run against the reference adapter in `packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts` -(`adapter-conformance.mutations.reference`, with a structured-clone and a JSON -provider) and against RxDB in +(`adapter-conformance.mutations.reference`, with a structured-clone, a JSON, +and a SQL provider), against RxDB in `packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.rxdb`). It checks auto-transaction +(`adapter-conformance.mutations.rxdb`), and against TrailBase in +`packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts` +(`adapter-conformance.mutations.trailbase`). It checks auto-transaction writes on a ready Collection whose adapter persists them itself: MUT-1 (the public rows show the optimistic state while the mutation function runs, and no publication shows the earlier synced value when it fulfills), MUT-2 (a @@ -7246,21 +7248,32 @@ it replaces the held optimistic row. With `server-rewrites`, an accepted insert's or update's plan can also rewrite the stored row (normalize the title, drop `note`, or set `note` as a server default): the request still carries the authored row (MUT-8), and the provider's copy, not the authored -row, is the row's value once it applies (MUT-1, FID-5). Reach: each of insert, update, and delete +row, is the row's value once it applies (MUT-1, FID-5). With `field-updates` +(the law set's capability), an update sends only its changed fields and the +provider merges them into its row as it is when the write is accepted; a +removed field is sent in the provider's representation, `null` for a SQL +provider (`undefinedField: 'null'`, which also sends a complete row with +every column). With `held-echoes`, the provider can answer an accepted write +before its change feed delivers the copy, at a `deliver` step: under MUT-7 the +mutation stays pending, and holds later writes (MUT-10), until its copy +arrives. A driver can declare several `rejectionKinds`, each reached. Reach: each of insert, update, and delete accepted, a rejected write, a held write, a write queued behind a held one, a held write rejected, a commit by another client, an accepted field removal, a -held optimistic row, a copy the provider rewrote, and a quiescent cut after -writes, at least five histories each per campaign; a grammar arm generates -updates without `note`, so removals are common. Calibration: nine reference +held optimistic row, a copy the provider rewrote, an answered write whose copy +is withheld, each declared rejection kind, and a quiescent cut after writes, +at least five histories each per campaign; a grammar arm generates +updates without `note`, so removals are common. Calibration: twelve reference mutants each fail with their law (an applied copy after the mutation function settles, a swallowed or wrapped rejection, a duplicated insert request, writes that do not wait for the earlier one, a skipped delete, an update that sends only changed fields, under a JSON provider a change feed that carries the authored row instead of the stored copy, and an adapter that writes the -authored row into its synced rows after the provider accepts). The last -survives the fixed campaign without `server-rewrites` and fails FID-5 with -it. Before the held-row rule, the +authored row into its synced rows after the provider accepts, and under a SQL +provider an update that sends the complete row, one that drops a removed +field, and a mutation function that settles when the provider answers instead +of when its copy arrives). The write-back mutant survives the fixed campaign +without `server-rewrites` and fails FID-5 with it. Before the held-row rule, the JSON-provider campaign failed MUT-1 on a fulfilled removal behind a pending write. @@ -7297,5 +7310,24 @@ counts its reach (`removedFieldEdits`). Limits: one pending mutation per key; commits by another client only while no mutation is pending (a commit held behind a pending mutation is core's MUT-5); no manual transactions, confirmation through sync (MUT-7), schema validation, or an update that -changes no field; only the RxDB adapter driver so far. +changes no field; multi-mutation transactions, so MUT-8's mutation order +within one transaction; a newer backend value in a disjoint field (commits +by another client arrive only while nothing is pending); server-generated +keys. + +TrailBase (`undefinedField: 'null'`, `field-updates`, `held-echoes`, and every +other mutation capability): the driver's RecordApi acts as a TrailBase +service with SQL rows, records each `createBulk` record, `update`, and +`delete` as JSON, as the SDK sends it, and sends change events at once or at +`deliver`. The application writes an absent `note` as `null` and removes a +held one by assigning `undefined`; removal by assigning `null` is not +generated. On the parent of the fix, 37 of the fixed campaign's 300 histories +failed MUT-8 (a removed field was sent as an empty patch, so the service kept +it), 27 failed MUT-10 (writes did not wait for the earlier one), and 5 failed +MUT-1 (a write was confirmed by an earlier event on the same key, so its +optimistic row dropped before its own event arrived). The adapter now sends +a removed field as `null`, runs each write after the earlier one settles, +and confirms a write only by an event after it began; a mutant that keeps the +old confirmation fails MUT-1. The record API is a test double that bypasses +the SDK's fetch and SSE decoding. diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 2eaa285432..6ebc942181 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -27,19 +27,19 @@ reviewed change. ## Adapters -| Adapter | Families that apply (section A) | Driver status | -| ----------------- | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | -| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations, on-demand: not started | -| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | -| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations, on-demand: not started | -| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | -| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | -| local-only | local and shared stores, mutations | Not started | -| localStorage | local and shared stores, mutations | Not started | -| Persisted wrapper | persistence wrapper | Not started | -| Storage hosts (8) | storage hosts | Not started | +| Adapter | Families that apply (section A) | Driver status | +| ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | +| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations, on-demand: not started | +| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | +| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | +| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | +| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | +| local-only | local and shared stores, mutations | Not started | +| localStorage | local and shared stores, mutations | Not started | +| Persisted wrapper | persistence wrapper | Not started | +| Storage hosts (8) | storage hosts | Not started | ## Suggested order @@ -580,3 +580,24 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the fails FID-5. - **Result:** the reference adapter and RxDB are GREEN, with no production change. +- **2026-11-03, mutations family, SQL providers and confirmation through + sync.** New capabilities `field-updates` (an update sends only its changed + fields and the provider merges them, the law set's MUT-8 clause) and + `held-echoes` (the provider answers before its change feed delivers the + copy; the mutation stays pending until it arrives, MUT-7), a `deliver` + step, and `undefinedField: 'null'` (a SQL row: a removed field is sent as + `null`, and a complete row carries every column). A SQL-provider reference + campaign and three mutants calibrate them (a complete-row update, a dropped + removal, settling on the answer). Requests now compare by the fields they + carry. A reach count that credited a rejected write as accepted is fixed. +- **2026-11-03, TrailBase, mutations.** Both campaigns are GREEN after an + adapter fix, with every declared reach state met. + - **RED on the parent:** of 300 fixed-campaign histories, 37 failed MUT-8 + (a removed field was sent as an empty patch), 27 failed MUT-10 (writes + did not wait), and 5 failed MUT-1 (a write was confirmed by an earlier + event on the same key). + - **Fix:** a removed field is sent as `null`; writes run in author order; + a write is confirmed only by an event after it began, recorded only + while a write is admitted, which removes the periodic cleanup timer. + A mutant with the old confirmation fails MUT-1. + - **Size:** trailbase-db-collection −46 gzip bytes. diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 178d547afa..04e5f505ae 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -207,6 +207,7 @@ const staticOracleProperties = [ `adapter-conformance.eager-live-source.reference`, `adapter-conformance.mutations.reference`, `adapter-conformance.mutations.rxdb`, + `adapter-conformance.mutations.trailbase`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index b5126c78e4..dd738212a9 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -797,6 +797,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.mutations`, `rxdb`, ], + [ + `trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `trailbase`, + ], ] const owners = new Map() @@ -898,7 +903,7 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), - ...[`reference`, `rxdb`].map( + ...[`reference`, `rxdb`, `trailbase`].map( (driver) => [ `adapter-conformance.mutations.${driver}`, diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index c4956328fe..5811d8675f 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -83,6 +83,9 @@ function convertPartial< return Object.fromEntries( Object.keys(input).map((k: string) => { const value = input[k] + // A removed field is an own `undefined`, which JSON would drop; it is + // the column's `NULL`. + if (value === undefined) return [k, null] return [k, c[k]?.(value as any) ?? value] }), ) as OutputType @@ -140,17 +143,24 @@ export function trailBaseCollectionOptions< const serialIns = (item: TItem) => convert(config.serialize, item) + // Each key's latest change event, by its sequence number. It is kept only + // while a write waits for its confirmation, which must be an event after + // the write began, not an earlier one on the same key (MUT-7). const seenIds = new Store(new Map()) + let eventSequence = 0 + // Writes admitted and not yet settled. + let writes = 0 const internalSyncMode = config.syncMode ?? `eager` let fullSyncCompleted = false const awaitIds = ( ids: Array, + after: number, timeout: number = 120 * 1000, ): Promise => { const completed = (value: Map) => - ids.every((id) => value.has(id)) + ids.every((id) => (value.get(id) ?? 0) > after) if (completed(seenIds.state)) { return Promise.resolve() } @@ -171,6 +181,23 @@ export function trailBaseCollectionOptions< }) } + // Writes reach TrailBase in author order: a write starts at once when no + // other is admitted, and otherwise after the earlier one is confirmed or + // rejected (MUT-10). A write returns the ids it wrote and settles once the + // change stream carries each of them (MUT-7). + let turn: Promise = Promise.resolve() + const persist = (write: () => Promise>): Promise => { + const start = async () => { + const after = eventSequence + await awaitIds(await write(), after) + } + const run = (writes++ === 0 ? start() : turn.then(start)).finally(() => { + if (--writes === 0) seenIds.setState(() => new Map()) + }) + turn = run.catch(() => {}) + return run + } + let eventReader: ReadableStreamDefaultReader | undefined const cancelEventReader = () => { if (eventReader) { @@ -187,15 +214,10 @@ export function trailBaseCollectionOptions< sync: (params: SyncParams) => { const { begin, write, commit, markReady, fail, collection } = params let cancelled = false - let periodicCleanupTask: ReturnType | undefined const cleanup = () => { cancelled = true settleListener() cancelEventReader() - if (periodicCleanupTask !== undefined) { - clearInterval(periodicCleanupTask) - periodicCleanupTask = undefined - } } // TrailBase cursors are opaque. Cache a cursor by its last fetched key @@ -205,9 +227,8 @@ export function trailBaseCollectionOptions< const cursors = new Map() let listening = false - // Orders this run's change events against its list calls. Only loads - // in flight compare against it, so it is kept only while one runs. - let eventSequence = 0 + // Orders change events against this run's list calls. Only loads in + // flight compare against it, so it is kept only while one runs. const eventSequenceByKey = new Map() let loadsInFlight = 0 // Settles when the listener runs, cannot start (with the failure), or @@ -404,14 +425,13 @@ export function trailBaseCollectionOptions< void commit() if (value) { - ++eventSequence + const sequence = ++eventSequence if (loadsInFlight > 0) - eventSequenceByKey.set(getKey(value), eventSequence) - seenIds.setState((curr: Map) => { - const newIds = new Map(curr) - newIds.set(String(getKey(value)), Date.now()) - return newIds - }) + eventSequenceByKey.set(getKey(value), sequence) + if (writes > 0) + seenIds.setState((curr: Map) => + new Map(curr).set(String(getKey(value)), sequence), + ) } } } @@ -431,7 +451,6 @@ export function trailBaseCollectionOptions< } async function start() { - let reader: ReadableStreamDefaultReader | undefined try { const eventStream = await config.recordApi.subscribe(`*`) if (cancelled) { @@ -439,7 +458,7 @@ export function trailBaseCollectionOptions< return } const subscribedReader = eventStream.getReader() - reader = eventReader = subscribedReader + eventReader = subscribedReader // Start listening for subscriptions first. Otherwise, we'd risk a gap // between the initial fetch and starting to listen. @@ -482,37 +501,6 @@ export function trailBaseCollectionOptions< reportStreamEnd(error, true) return } - - // Lastly, start a periodic cleanup task that will be removed when the - // reader closes. - if (cancelled || !reader) return - - periodicCleanupTask = setInterval(() => { - seenIds.setState((curr) => { - const now = Date.now() - let anyExpired = false - - const notExpired = Array.from(curr.entries()).filter(([_, v]) => { - const expired = now - v > 300 * 1000 - anyExpired = anyExpired || expired - return !expired - }) - - if (anyExpired) { - return new Map(notExpired) - } - return curr - }) - }, 120 * 1000) - - const clearCleanupTask = () => { - if (periodicCleanupTask !== undefined) { - clearInterval(periodicCleanupTask) - periodicCleanupTask = undefined - } - } - // listen() reports read errors. Observe this separate promise too. - void reader.closed.then(clearCleanupTask, clearCleanupTask) } void start() @@ -550,59 +538,47 @@ export function trailBaseCollectionOptions< ...collectionConfig, sync, getKey, - onInsert: async (params: InsertMutationFnParams) => { - const ids = await config.recordApi.createBulk( - params.transaction.mutations.map((tx) => { - const { type, modified } = tx - if (type !== `insert`) { - throw new ExpectedInsertTypeError(type) - } - return serialIns(modified) - }), - ) - - // The optimistic mutation overlay is removed on return, so at this point - // we have to ensure that the new record was properly added to the local - // DB by the subscription. - await awaitIds(ids.map((id) => String(id))) - }, - onUpdate: async (params: UpdateMutationFnParams) => { - const ids: Array = await Promise.all( - params.transaction.mutations.map(async (tx) => { - const { type, changes, key } = tx - if (type !== `update`) { - throw new ExpectedUpdateTypeError(type) - } - - await config.recordApi.update(key, serialUpd(changes)) - - return String(key) - }), - ) - - // The optimistic mutation overlay is removed on return, so at this point - // we have to ensure that the new record was properly updated in the local - // DB by the subscription. - await awaitIds(ids) - }, - onDelete: async (params: DeleteMutationFnParams) => { - const ids: Array = await Promise.all( - params.transaction.mutations.map(async (tx) => { - const { type, key } = tx - if (type !== `delete`) { - throw new ExpectedDeleteTypeError(type) - } - - await config.recordApi.delete(key) - return String(key) - }), - ) - - // The optimistic mutation overlay is removed on return, so at this point - // we have to ensure that the new record was properly updated in the local - // DB by the subscription. - await awaitIds(ids) - }, + // The optimistic mutation overlay is removed when a handler returns, so + // each one waits until the subscription has applied what it wrote. + onInsert: (params: InsertMutationFnParams) => + persist(async () => { + const ids = await config.recordApi.createBulk( + params.transaction.mutations.map((tx) => { + const { type, modified } = tx + if (type !== `insert`) { + throw new ExpectedInsertTypeError(type) + } + return serialIns(modified) + }), + ) + return ids.map((id) => String(id)) + }), + onUpdate: (params: UpdateMutationFnParams) => + persist(() => + Promise.all( + params.transaction.mutations.map(async (tx) => { + const { type, changes, key } = tx + if (type !== `update`) { + throw new ExpectedUpdateTypeError(type) + } + await config.recordApi.update(key, serialUpd(changes)) + return String(key) + }), + ), + ), + onDelete: (params: DeleteMutationFnParams) => + persist(() => + Promise.all( + params.transaction.mutations.map(async (tx) => { + const { type, key } = tx + if (type !== `delete`) { + throw new ExpectedDeleteTypeError(type) + } + await config.recordApi.delete(key) + return String(key) + }), + ), + ), utils: { cancel: cancelEventReader, }, diff --git a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts new file mode 100644 index 0000000000..d6729d685f --- /dev/null +++ b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts @@ -0,0 +1,261 @@ +/** + * Production driver for the shared mutation oracle + * (`packages/db-adapter-conformance/src/mutation-oracle.ts`). It runs the real + * TrailBase adapter in eager mode against a controlled RecordApi that acts as + * a TrailBase service: it holds the table's rows, answers `list`, `createBulk`, + * `update`, and `delete` by plan, and sends the change stream's events. + * + * The oracle's steps map onto TrailBase like this: + * + * - a record is a SQL row with every column, so an absent `note` is `NULL` + * (`undefinedField: 'null'`); the observed view maps `NULL` back to absent + * (`fromSqlRow`). The driver inserts an absent `note` as `null`, as a + * TrailBase application must (its record type has every column), and + * removes a held `note` by assigning `undefined`, the update API's + * removed field. Removing it by assigning `null` is not generated; + * - a write request is one record of a `createBulk` call, one `update` call, + * which carries only the changed fields and which the service merges into + * its row (`field-updates`), or one `delete` call. Each request is + * recorded as JSON, as the SDK sends it; + * - a plan answers the next request: a rejection rejects the call with the + * plan's error, and a held request waits until `release`. An accepted + * write commits and sends its change event at once, or, with a held echo, + * only at `deliver`, as an event can trail the HTTP response + * (`held-echoes`); TrailBase confirms each write through its change + * stream (the law set's `confirm-through-sync`); + * - a rewrite (`server-rewrites`) is the service changing the stored row, as + * a column default, trigger, or check does; its event carries the stored + * row; + * - a commit by another client changes the service's rows and sends its + * event at once. + * + * Limits: the record API is a test double that bypasses the SDK's fetch and + * SSE decoding, as `conformance-driver.ts` does; the key is the client's, + * never server-generated. + */ +import { createCollection } from '@tanstack/db' +import { trailBaseCollectionOptions } from '../src/trailbase' +import { + fromSqlRow, + observedView, + rewriteRow, + toSqlRow, +} from '../../db-adapter-conformance/src' +import { MockRecordApi } from './mock-record-api' +import type { Event } from 'trailbase' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../../db-adapter-conformance/src' + +/** + * A TrailBase record: a SQL row, so it has every column. The application + * writes an absent `note` as `null`, and removes one by assigning + * `undefined`, the update API's removed field (an own `undefined`). + */ +type ConformanceRecord = { + id: number + title: string + note: string | null | undefined +} + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + echoHeld: false, + rewrite: `none`, + error: new Error(`unplanned write`), +} + +/** A value as the SDK sends it: JSON, so an `undefined` field is absent. */ +const asSent = (value: T): T => JSON.parse(JSON.stringify(value)) as T + +/** A request's record as the request log holds it; `null` is kept. */ +const sentRow = (record: Partial) => + asSent(record) as Partial + +class TrailBaseMutationProvider implements MutationProvider { + readonly api = new MockRecordApi() + /** The service's table: SQL rows. */ + private readonly table = new Map() + private readonly plans: Array = [] + private readonly held: Array<() => void> = [] + private readonly withheld: Array<() => void> = [] + private readonly log: Array = [] + private controller: ReadableStreamDefaultController | undefined + + constructor(seed: ReadonlyArray) { + for (const row of seed) this.table.set(row.id, toSqlRow(row)) + this.api.subscribe.mockImplementation(() => + Promise.resolve( + new ReadableStream({ + start: (controller) => { + this.controller = controller + }, + // Cleanup cancels the stream; later events have no reader. + cancel: () => { + this.controller = undefined + }, + }), + ), + ) + this.api.list.mockImplementation(() => + Promise.resolve({ records: [...this.table.values()].map(asSent) }), + ) + this.api.createBulk.mockImplementation((records) => + Promise.all( + records.map((record) => + this.request({ type: `insert`, id: record.id, row: sentRow(record) }), + ), + ).then(() => records.map((record) => record.id)), + ) + this.api.update.mockImplementation((id, record) => + this.request({ + type: `update`, + id: Number(id), + row: { id: Number(id), ...sentRow(record) }, + }), + ) + this.api.delete.mockImplementation((id) => + this.request({ type: `delete`, id: Number(id) }), + ) + } + + /** One write request: recorded, then answered by the next plan. */ + private request(request: WriteRequest): Promise { + this.log.push(request) + const plan = this.plans.shift() ?? acceptAtOnce + return new Promise((resolve, reject) => { + const answer = () => { + if (plan.outcome === `reject`) { + reject(plan.error) + return + } + this.commit(this.change(request, plan), plan.echoHeld) + resolve() + } + if (plan.held) this.held.push(answer) + else answer() + }) + } + + /** The change an accepted request makes to the service's table. */ + private change(request: WriteRequest, plan: WritePlan): ProviderChange { + if (request.type === `delete`) return { type: `delete`, id: request.id } + // An update merges the fields it sends into the stored row. + const merged = { + ...(request.type === `update` ? this.table.get(request.id) : {}), + ...request.row, + } as Partial + return { + type: `upsert`, + row: rewriteRow(fromSqlRow({ note: null, ...merged }), plan.rewrite), + } + } + + /** Commits a change and sends its event, at once or at `deliver`. */ + private commit(change: ProviderChange, withhold = false): void { + let event: Event + if (change.type === `delete`) { + const record = this.table.get(change.id) + // Deleting a row the table lacks changes nothing and sends no event. + if (!record) return + this.table.delete(change.id) + event = { Delete: asSent(record) } + } else { + const record = toSqlRow(change.row) + const held = this.table.has(change.row.id) + this.table.set(change.row.id, record) + event = held ? { Update: asSent(record) } : { Insert: asSent(record) } + } + const send = () => this.controller?.enqueue(event) + if (withhold) this.withheld.push(send) + else send() + } + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + this.held.shift()?.() + } + + /** Answers every held request and sends every withheld event. */ + drain(): void { + for (const answer of this.held.splice(0)) answer() + for (const send of this.withheld.splice(0)) send() + } + + deliver(): void { + this.withheld.shift()?.() + } + + foreignCommit(change: ProviderChange): void { + this.commit(change) + } + + rows(): ReadonlyArray { + return [...this.table.values()].map(fromSqlRow) + } + + requests(): ReadonlyArray { + return this.log + } +} + +export function trailBaseMutationDriver(): MutationDriver { + return { + adapter: `@tanstack/trailbase-db-collection (eager)`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + `server-rewrites`, + `field-updates`, + `held-echoes`, + ]), + undefinedField: `null`, + create(seed): MutationFixture { + const provider = new TrailBaseMutationProvider(seed) + const collection = createCollection( + trailBaseCollectionOptions({ + recordApi: provider.api, + getKey: (row: ConformanceRecord) => row.id, + syncMode: `eager`, + parse: {}, + serialize: {}, + }), + ) + return { + collection: observedView(collection, { + row: fromSqlRow, + key: (key) => key as number, + }), + base: () => [...collection.base.values()].map(fromSqlRow), + writer: { + insert: (row) => collection.insert(toSqlRow(row)), + update: (row) => + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note + // A `NULL` note is already absent; only a held one is removed. + else if (typeof draft.note === `string`) draft.note = undefined + }), + delete: (id) => collection.delete(id), + }, + provider, + dispose: async () => { + // A write waiting for its event would otherwise outlive the history. + provider.drain() + await collection.cleanup() + }, + } + }, + } +} diff --git a/packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts new file mode 100644 index 0000000000..42cfa3dd6c --- /dev/null +++ b/packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -0,0 +1,75 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../../db-adapter-conformance/src' +import { trailBaseMutationDriver } from './mutation-conformance-driver' + +/** + * # TrailBase on the shared mutation oracle + * + * A receiving driver for `packages/db-adapter-conformance`. The shared + * module states the laws (MUT-1, MUT-2, MUT-7, MUT-8, MUT-10, MUT-13, FID-5, + * HEALTH-5), the model, the history grammar, and the check. This file + * supplies only the TrailBase production driver in + * `mutation-conformance-driver.ts` and the campaigns; it states no expected + * results. + * + * The driver's RecordApi acts as a TrailBase service with SQL rows. It + * declares every mutation capability: the service can reject or hold a + * write, rewrite a stored row, and send a write's change event after it + * answers; another client can write; and updates send only their changed + * fields (the law set's `field-updates`). TrailBase confirms each write + * through its change stream (`confirm-through-sync`, MUT-7). + */ + +const property = `adapter-conformance.mutations.trailbase` +const driver = trailBaseMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +const campaignTimeoutMs = 120_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: TrailBase`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) diff --git a/packages/trailbase-db-collection/tests/trailbase.test.ts b/packages/trailbase-db-collection/tests/trailbase.test.ts index 4eda28a43e..16ef9b7021 100644 --- a/packages/trailbase-db-collection/tests/trailbase.test.ts +++ b/packages/trailbase-db-collection/tests/trailbase.test.ts @@ -300,15 +300,11 @@ describe(`TrailBase Integration`, () => { const errors: Array = [] const recordUnhandled = (error: unknown) => errors.push(error) const reported = vi.spyOn(console, `error`).mockImplementation(() => {}) - const intervals = vi.spyOn(globalThis, `setInterval`) - const clear = vi.spyOn(globalThis, `clearInterval`) process.on(`unhandledRejection`, recordUnhandled) const collection = createCollection(setUp(recordApi)) try { await collection.preload() - const timer = intervals.mock.results.at(-1)?.value - expect(timer).toBeDefined() const expectedRows = new Map([[1, row]]) if (ending === `buffered-close`) { for (const id of [2, 3]) { @@ -322,7 +318,6 @@ describe(`TrailBase Integration`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) expect(errors).toEqual([]) - expect(clear).toHaveBeenCalledWith(timer) expect(stream.locked).toBe(false) expect(collection.status).toBe(`ready`) expect(stripState(collection.state)).toEqual(expectedRows) @@ -343,8 +338,6 @@ describe(`TrailBase Integration`, () => { await new Promise((resolve) => setTimeout(resolve, 0)) process.off(`unhandledRejection`, recordUnhandled) reported.mockRestore() - intervals.mockRestore() - clear.mockRestore() } }, ) From 9ebd847f9f1b857d5d36eab9c4fda65a458921b1 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 11:19:02 -0700 Subject: [PATCH 10/28] test(db-adapter-conformance): count only plan rewrites as server-rewrote-copy The model counted any accepted copy that differs from its authored row as `server-rewrote-copy`. Under `undefinedField: 'null'` a removal's copy differs by the value domain alone (an own `undefined` authored, an absent field stored), so a driver that declares no `server-rewrites` reached a state it does not declare, and its campaign failed reach. The count now needs the plan's rewrite. - Reach: the reference campaigns still meet `server-rewrote-copy`. - RED: the PowerSync driver's campaigns failed with "undeclared reach state server-rewrote-copy". - GREEN: db-adapter-conformance 102 passed; PowerSync, TrailBase, and RxDB mutation campaigns pass. Co-authored-by: Isaac --- packages/db-adapter-conformance/src/mutation-oracle.ts | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 828e764d03..b79b6eccb2 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -532,8 +532,9 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } * accepted), stored under the driver's `undefinedField` rule and then the * plan's `ServerRewrite`: after a rewrite the provider's copy, not the * authored row, is the row's value (FID-5). - * - `rewritten` counts accepted writes whose copy differs from their - * authored row, for reach. + * - `rewritten` counts accepted writes whose plan's rewrite made their copy + * differ from their authored row, for reach. A copy can also differ by + * the value domain alone, as a removed field does under `null`. * - `pending` lists the mutations not yet settled, in author order. MUT-10 * orders their requests: the first is requested at once, and each later * one only after every earlier one is decided. An accepted write whose @@ -703,7 +704,7 @@ function decide(state: MutationModel, write: PendingWrite): void { else { const copy = rowValue(copyOf(state, write)) state.provider.set(write.id, copy) - if (copy !== write.value) state.rewritten++ + if (write.plan.rewrite !== `none` && copy !== write.value) state.rewritten++ } if (write.plan.echoHeld) write.answered = true else confirm(state, write) From 85e978e069dd3089c107759743cdfcca3891d823 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 11:19:05 -0700 Subject: [PATCH 11/28] fix(powersync-db-collection): persist writes in author order A receiving driver runs the PowerSync adapter on the shared mutation oracle over a real local PowerSync database, the adapter's provider. Each statement the adapter executes on the table is a request; a plan can reject it (the statement throws and the write transaction rolls back) or hold it inside the write lock. A held echo holds the change watcher's next callback, so the adapter's confirmation through its diff trigger (MUT-7) arrives late. The oracle found that a write started as soon as the earlier one committed locally, before the collection had observed it. The law set's MUT-10 orders a write after the earlier one decides, and counts a synchronously failed handler as decided before its receipt settles as the exception, so "decides" is settlement. Each write now starts after the earlier one on the collection settles. - Law: MUT-10 (law set v18). - Reach: every declared mutation state, at least five histories each, in the fixed (seed 2118, 300 histories) and random campaigns. - RED: on the parent powersync.ts both campaigns failed MUT-10, shrunk to an update whose echo is held and an insert that ran before it settled. - GREEN: both campaigns pass (about 47 s each); powersync-db-collection 198 passed. - Mutant: the parent is the hostile mutant. - Limits: the local database rewrites nothing, so no `server-rewrites`; removal by assigning `null`, multi-mutation transactions, and the upload queue to a PowerSync service are outside these histories. - Size: powersync-db-collection +26 gzip bytes, the ordering chain. Co-authored-by: Isaac --- .changeset/powersync-ordered-writes.md | 5 + docs/contributing/oracle-coverage.md | 24 +- packages/db-adapter-conformance/MIGRATION.md | 18 +- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../powersync-db-collection/src/powersync.ts | 27 +- .../tests/mutation-conformance-driver.ts | 422 ++++++++++++++++++ ...tation-conformance-oracle.property.test.ts | 77 ++++ 8 files changed, 565 insertions(+), 16 deletions(-) create mode 100644 .changeset/powersync-ordered-writes.md create mode 100644 packages/powersync-db-collection/tests/mutation-conformance-driver.ts create mode 100644 packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts diff --git a/.changeset/powersync-ordered-writes.md b/.changeset/powersync-ordered-writes.md new file mode 100644 index 0000000000..fa56f84386 --- /dev/null +++ b/.changeset/powersync-ordered-writes.md @@ -0,0 +1,5 @@ +--- +'@tanstack/powersync-db-collection': patch +--- + +Persist writes to PowerSync in the order the application made them. A write now starts only after the earlier write on the same collection is observed by the collection or rejected. Before, a later write could start as soon as the earlier one was written to SQLite, before the collection had observed it. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 18d330fd0b..066466ec16 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7221,9 +7221,11 @@ the reference adapter in (`adapter-conformance.mutations.reference`, with a structured-clone, a JSON, and a SQL provider), against RxDB in `packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.rxdb`), and against TrailBase in +(`adapter-conformance.mutations.rxdb`), against TrailBase in `packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.trailbase`). It checks auto-transaction +(`adapter-conformance.mutations.trailbase`), and against PowerSync in +`packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts` +(`adapter-conformance.mutations.powersync`). It checks auto-transaction writes on a ready Collection whose adapter persists them itself: MUT-1 (the public rows show the optimistic state while the mutation function runs, and no publication shows the earlier synced value when it fulfills), MUT-2 (a @@ -7331,3 +7333,21 @@ and confirms a write only by an event after it began; a mutant that keeps the old confirmation fails MUT-1. The record API is a test double that bypasses the SDK's fetch and SSE decoding. +PowerSync (`undefinedField: 'null'`, `field-updates`, `held-echoes`, +`write-rejection`, `held-writes`, `foreign-commits`): the driver runs a real +local PowerSync database, which is the adapter's provider; uploading to a +PowerSync service is the SDK's upload queue and is outside this owner. A +request is one `INSERT`, `UPDATE`, or `DELETE` the adapter executes on the +table inside its write transaction; a rejection throws from that statement, +and a held request holds the write lock. A held echo holds the change +watcher's next callback until `deliver`. The propagation step waits until +every write's receipt settles, or, while a request or echo is held, until the +request log, the watcher, and the receipts are quiet. The local database does +not rewrite rows, so `server-rewrites` is not declared. On the parent of the +fix, both campaigns failed MUT-10: a write started once the earlier one +committed locally, before its diff trigger observed it. MUT-10's "decides" +is settlement (the law set counts a synchronously failed handler as decided +before its receipt settles, as the exception). The adapter now runs each +write after the earlier one settles. Removal binds an own `undefined` as +`NULL`, so MUT-8 holds. Limits: removal by assigning `null` and +multi-mutation transactions are not generated. diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 6ebc942181..9ec4808096 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -31,7 +31,7 @@ reviewed change. | ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | | Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations, on-demand: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | | RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | | TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | | Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | @@ -601,3 +601,19 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the while a write is admitted, which removes the periodic cleanup timer. A mutant with the old confirmation fails MUT-1. - **Size:** trailbase-db-collection −46 gzip bytes. +- **2026-11-03, PowerSync, mutations.** Both campaigns are GREEN after an + adapter fix, with every declared reach state met. + - **Driver:** a real local PowerSync database is the provider; each + adapter statement on the table is a request, a held echo holds the + change watcher's next callback, and the propagation step waits for + every write's receipt or, while something is held, for quiet. + - **RED on the parent:** both campaigns failed MUT-10: a write started + once the earlier one committed locally, before its diff trigger + observed it. The law set's "decides" is settlement. Fix: each write + runs after the earlier one settles. Removals already reach SQLite as + `NULL`. + - **Oracle change this migration exposed:** `server-rewrote-copy` counted + a copy that differs only by the value domain (a removal under `null`), + so a driver without `server-rewrites` reached an undeclared state. It + now counts only a plan's rewrite. + - **Size:** powersync-db-collection +26 gzip bytes for MUT-10. diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index 04e5f505ae..fab4b73052 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -208,6 +208,7 @@ const staticOracleProperties = [ `adapter-conformance.mutations.reference`, `adapter-conformance.mutations.rxdb`, `adapter-conformance.mutations.trailbase`, + `adapter-conformance.mutations.powersync`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index dd738212a9..e6ae675760 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -802,6 +802,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.mutations`, `trailbase`, ], + [ + `powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `powersync`, + ], ] const owners = new Map() @@ -903,7 +908,7 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), - ...[`reference`, `rxdb`, `trailbase`].map( + ...[`reference`, `rxdb`, `trailbase`, `powersync`].map( (driver) => [ `adapter-conformance.mutations.${driver}`, diff --git a/packages/powersync-db-collection/src/powersync.ts b/packages/powersync-db-collection/src/powersync.ts index dfb52c15f4..e673924678 100644 --- a/packages/powersync-db-collection/src/powersync.ts +++ b/packages/powersync-db-collection/src/powersync.ts @@ -1199,6 +1199,17 @@ function createPowerSyncCollectionConfig< ...restConfig, } + // Writes reach the database in author order: each starts after the + // earlier one has been observed or rejected (MUT-10). + let turn: Promise = Promise.resolve() + const persist = ( + transaction: Parameters[0], + ): Promise => { + const run = turn.then(() => transactor.applyTransaction(transaction)) + turn = run.catch(() => {}) + return run + } + const outputConfig: EnhancedPowerSyncCollectionConfig< TTable, OutputType, @@ -1211,18 +1222,10 @@ function createPowerSyncCollectionConfig< startSync: true, syncMode, sync, - onInsert: async (params) => { - // The transaction here should only ever contain a single insert mutation - return await transactor.applyTransaction(params.transaction) - }, - onUpdate: async (params) => { - // The transaction here should only ever contain a single update mutation - return await transactor.applyTransaction(params.transaction) - }, - onDelete: async (params) => { - // The transaction here should only ever contain a single delete mutation - return await transactor.applyTransaction(params.transaction) - }, + // Each handler's transaction holds a single mutation of its type. + onInsert: (params) => persist(params.transaction), + onUpdate: (params) => persist(params.transaction), + onDelete: (params) => persist(params.transaction), utils: { getMeta: () => ({ tableName: viewName, diff --git a/packages/powersync-db-collection/tests/mutation-conformance-driver.ts b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts new file mode 100644 index 0000000000..a5735638de --- /dev/null +++ b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts @@ -0,0 +1,422 @@ +/** + * Production driver for the shared mutation oracle + * (`packages/db-adapter-conformance/src/mutation-oracle.ts`). It runs the real + * PowerSync adapter in eager mode over a real local PowerSync database + * (SQLite through `@powersync/node`), as `conformance-driver.ts` does. The + * adapter persists a write to that database and settles once its own diff + * trigger has observed it, so the provider is the local database; uploading + * to a PowerSync service is the SDK's upload queue, outside this oracle. + * + * The oracle's steps map onto PowerSync like this: + * + * - a write request is one `INSERT`, `UPDATE`, or `DELETE` statement the + * adapter executes on the table inside its write transaction. An insert + * sends every column; an update sets only its changed columns + * (`field-updates`); the request carries the bound values; + * - a plan answers the next request inside the adapter's transaction: a + * rejection throws the plan's error from the statement, as SQLite does for + * a failed write, which rolls the transaction back; a held request waits + * until `release`, holding the write lock as a slow disk does; + * - the adapter applies the database's changes through its change watcher + * (`onChangeWithCallback`). With a held echo, the watcher's next callback + * waits until `deliver` (`held-echoes`), as a throttled or busy watcher + * delays it; the adapter must not settle the write before (MUT-7); + * - a commit by another client is a write to the table that is not the + * adapter's, as the SDK's download of server changes is; + * - the provider's rows are the table's rows. `note` is a nullable text + * column, so an absent `note` is SQL `NULL` (`undefinedField: 'null'`, + * `toSqlRow`, `fromSqlRow`). + * + * PowerSync keys are strings, so row `n` is id `"n"`. The application + * inserts an absent `note` as `null` and removes a held one by assigning + * `undefined`, the update API's removed field; removal by assigning `null` + * is not generated. The local database does not rewrite rows, so the driver + * does not declare `server-rewrites`. + */ +import { randomUUID } from 'node:crypto' +import { rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { PowerSyncDatabase, Schema, Table, column } from '@powersync/node' +import { createCollection } from '@tanstack/db' +import { powerSyncCollectionOptions } from '../src' +import { + conformanceKey, + fromSqlRow, + observedView, + toSqlRow, +} from '../../db-adapter-conformance/src' +import { TEST_DATABASE_IMPLEMENTATION } from './test-db-implementation' +import type { LockContext } from '@powersync/node' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../../db-adapter-conformance/src' + +const SCHEMA = new Schema({ + rows: new Table({ + title: column.text, + note: column.text, + }), +}) + +type Record = { id: string; title: string | null; note: string | null } + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + echoHeld: false, + rewrite: `none`, + error: new Error(`unplanned write`), +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) +const drainTimeoutMs = 5_000 +// How long nothing must change before a cut with a held request or echo. +const quietMs = 25 + +/** Field names in a statement, without the quoting `sanitizeSQL` adds. */ +const unquote = (name: string) => name.trim().replace(/^['"]|['"]$/g, ``) +const names = (list: string) => list.split(`,`).map(unquote) + +/** + * The request an adapter statement makes, or undefined for a statement that + * is not a write to the table. + */ +function writeRequest( + sql: string, + params: ReadonlyArray, +): WriteRequest | undefined { + const insert = /^\s*INSERT\s+into\s+['"]?rows['"]?\s*\(([^)]*)\)/i.exec(sql) + if (insert) { + const row = Object.fromEntries( + names(insert[1]!).map((name, index) => [name, params[index]]), + ) as Record + return { + type: `insert`, + id: conformanceKey(row.id), + row: { ...row, id: conformanceKey(row.id) } as Partial, + } + } + const update = + /^\s*UPDATE\s+['"]?rows['"]?\s+SET\s+([\s\S]*?)\s+WHERE\s+id\s*=\s*\?/i.exec( + sql, + ) + if (update) { + const fields = names(update[1]!).map((set) => set.split(`=`)[0]!.trim()) + const id = conformanceKey(String(params.at(-1))) + return { + type: `update`, + id, + row: { + id, + ...Object.fromEntries( + fields.map((field, index) => [unquote(field), params[index]]), + ), + } as Partial, + } + } + const remove = + /^\s*DELETE\s+FROM\s+['"]?rows['"]?\s+WHERE\s+id\s*=\s*\?/i.exec(sql) + if (remove) return { type: `delete`, id: conformanceKey(String(params[0])) } + return undefined +} + +class PowerSyncMutationProvider implements MutationProvider { + readonly database: PowerSyncDatabase + readonly file: string + private readonly plans: Array = [] + private readonly held: Array<() => void> = [] + private readonly log: Array = [] + /** Opens the watcher's next callback when an echo is withheld. */ + private echoes: Array<() => void> = [] + private gate: Promise | undefined + private running = 0 + private drained = 0 + private drainTarget: number | undefined + private readonly rawWriteTransaction: PowerSyncDatabase[`writeTransaction`] + + constructor() { + if (!TEST_DATABASE_IMPLEMENTATION) { + throw new Error(`no SQLite implementation is available`) + } + this.file = `mutations-${randomUUID()}.sqlite` + this.database = new PowerSyncDatabase({ + database: { + dbFilename: this.file, + dbLocation: tmpdir(), + implementation: TEST_DATABASE_IMPLEMENTATION, + }, + schema: SCHEMA, + }) + this.rawWriteTransaction = this.database.writeTransaction.bind( + this.database, + ) + // The adapter's write transactions: each table write is a request. + this.database.writeTransaction = (async ( + callback: (tx: LockContext) => Promise, + lockTimeout?: number, + ) => { + // Set by the statement wrapper inside the transaction callback. + let wrote = false as boolean + const result = await this.rawWriteTransaction(async (tx) => { + const execute = tx.execute.bind(tx) + const guarded = Object.create(tx) as LockContext + guarded.execute = async (sql: string, params: Array = []) => { + const request = writeRequest(sql, params) + if (!request) return execute(sql, params) + this.log.push(request) + const plan = this.plans.shift() ?? acceptAtOnce + if (plan.held) + await new Promise((resolve) => this.held.push(resolve)) + if (plan.outcome === `reject`) throw plan.error + if (plan.echoHeld) this.withholdEcho() + wrote = true + return execute(sql, params) + } + return callback(guarded) + }, lockTimeout) + if (wrote) this.expectDrain() + return result + }) as PowerSyncDatabase[`writeTransaction`] + // The adapter's change watcher: count its drains, and hold its next + // callback while an echo is withheld. + const register = this.database.onChangeWithCallback.bind(this.database) + this.database.onChangeWithCallback = (handler, options) => + register( + { + ...handler, + onChange: async (event) => { + this.running++ + try { + await this.gate + await handler?.onChange(event) + } finally { + this.running-- + this.drained++ + } + }, + }, + options, + ) + } + + /** + * A commit is applied by a watcher callback that starts after it: one + * already running read the tracking table before the commit. + */ + private expectDrain(): void { + this.drainTarget = this.drained + this.running + 1 + } + + private withholdEcho(): void { + this.gate = new Promise((resolve) => this.echoes.push(resolve)) + } + + /** + * Writes that are not the adapter's: the seed and other clients. A table + * is a view, so an upsert is an update or an insert. A delete of a row the + * table lacks changes nothing, so the watcher has nothing to drain. + */ + private async write(change: ProviderChange): Promise { + const changed = await this.rawWriteTransaction(async (tx) => { + // A view's `INSTEAD OF` trigger reports no affected rows, so the row's + // presence decides whether the write changes anything. + if (change.type === `delete`) { + const id = String(change.id) + const held = await tx.getOptional(`SELECT id FROM rows WHERE id = ?`, [ + id, + ]) + if (held) await tx.execute(`DELETE FROM rows WHERE id = ?`, [id]) + return held !== null + } + const { id, title, note } = toSqlRow(change.row) + const held = await tx.getOptional(`SELECT id FROM rows WHERE id = ?`, [ + String(id), + ]) + if (held) + await tx.execute(`UPDATE rows SET title = ?, note = ? WHERE id = ?`, [ + title, + note, + String(id), + ]) + else + await tx.execute( + `INSERT INTO rows (id, title, note) VALUES (?, ?, ?)`, + [String(id), title, note], + ) + return true + }) + if (changed) this.expectDrain() + } + + /** Writes made through the Collection whose receipts have not settled. */ + private unsettled = 0 + + /** Counts a write's receipt until it settles. */ + track Promise }>( + receipt: T, + ): T { + this.unsettled++ + receipt.when(`settled`).then( + () => this.unsettled--, + () => this.unsettled--, + ) + return receipt + } + + async seed(rows: ReadonlyArray): Promise { + for (const row of rows) await this.write({ type: `upsert`, row }) + this.drainTarget = undefined + } + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + this.held.shift()?.() + } + + deliver(): void { + this.echoes.shift()?.() + if (this.echoes.length === 0) this.gate = undefined + } + + /** Answers every held request and opens every withheld echo. */ + drain(): void { + for (const answer of this.held.splice(0)) answer() + for (const open of this.echoes.splice(0)) open() + this.gate = undefined + } + + foreignCommit(change: ProviderChange): Promise { + return this.write(change) + } + + async rows(): Promise> { + const records = await this.database.getAll( + `SELECT id, title, note FROM rows ORDER BY id`, + ) + return records.map(conformanceRow) + } + + requests(): ReadonlyArray { + return this.log + } + + /** + * The propagation step (`propagationStep`: the diff-trigger watch). The + * adapter reaches its write transaction a few milliseconds after a write, + * so the step waits until every write's receipt has settled and the + * watcher has drained every commit. While a request or an echo is held, + * some writes legitimately wait, so it waits instead until the request + * log, the watcher, and the receipts are quiet. + */ + settle = async () => { + const deadline = Date.now() + drainTimeoutMs + const state = () => + `${this.log.length} ${this.drained} ${this.running} ${this.unsettled}` + let last = state() + let quietSince = Date.now() + while (Date.now() < deadline) { + const drainedAll = + (this.drainTarget === undefined || this.drained >= this.drainTarget) && + this.running === 0 + if (this.unsettled === 0 && drainedAll) { + this.drainTarget = undefined + return + } + const blocked = this.held.length > 0 || this.gate !== undefined + if (blocked && Date.now() - quietSince >= quietMs) return + await sleep(5) + const now = state() + if (now !== last) { + last = now + quietSince = Date.now() + } + } + } + + dispose = async () => { + await this.database.disconnectAndClear() + await this.database.close() + await rm(join(tmpdir(), this.file), { force: true }) + } +} + +/** A table row as a conformance row: key `"n"` is row id `n`. */ +function conformanceRow(record: Partial): ConformanceRow { + return fromSqlRow({ ...record, id: conformanceKey(String(record.id)) }) +} + +export function powerSyncMutationDriver(): MutationDriver { + return { + adapter: `@tanstack/powersync-db-collection (eager)`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + `field-updates`, + `held-echoes`, + ]), + undefinedField: `null`, + async create(seed): Promise { + const provider = new PowerSyncMutationProvider() + await provider.seed(seed) + const collection = createCollection( + powerSyncCollectionOptions({ + database: provider.database, + table: SCHEMA.props.rows, + }), + ) + const view = observedView(collection, { + row: conformanceRow, + key: conformanceKey, + }) + // A held request holds the write lock, which cleanup waits for; a + // history that ends early with one held must answer it first. + const cleanup = view.cleanup.bind(view) + view.cleanup = () => { + provider.drain() + return cleanup() + } + return { + collection: view, + base: () => [...collection.base.values()].map(conformanceRow), + writer: { + insert: (row) => { + const { id, title, note } = toSqlRow(row) + return provider.track( + collection.insert({ id: String(id), title, note }), + ) + }, + update: (row) => + provider.track( + collection.update(String(row.id), (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note ?? null + // A `NULL` note is already absent; only a held one is removed. + else if (typeof draft.note === `string`) + (draft as { note?: string | null }).note = undefined + }), + ), + delete: (id) => provider.track(collection.delete(String(id))), + }, + provider, + settle: provider.settle, + dispose: async () => { + // A write waiting for its echo would otherwise outlive the history. + provider.drain() + await collection.cleanup() + await provider.dispose() + }, + } + }, + } +} diff --git a/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts new file mode 100644 index 0000000000..d695c3e6f9 --- /dev/null +++ b/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -0,0 +1,77 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../../db-adapter-conformance/src' +import { powerSyncMutationDriver } from './mutation-conformance-driver' + +/** + * # PowerSync on the shared mutation oracle + * + * A receiving driver for `packages/db-adapter-conformance`. The shared + * module states the laws (MUT-1, MUT-2, MUT-7, MUT-8, MUT-10, MUT-13, FID-5, + * HEALTH-5), the model, the history grammar, and the check. This file + * supplies only the PowerSync production driver in + * `mutation-conformance-driver.ts` and the campaigns; it states no expected + * results. + * + * The driver runs a real local PowerSync database, the adapter's provider: + * the adapter persists each write to it and settles once its diff trigger + * has observed the write (`confirm-through-sync`, MUT-7). The database can + * reject or hold a write, another client can write, the watcher can deliver + * a write's change late, and updates set only their changed columns (the law + * set's `field-updates`). + */ + +const property = `adapter-conformance.mutations.powersync` +const driver = powerSyncMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +// Each history opens, clears, and closes its own SQLite database, so a +// campaign takes about 47 s locally; the budget leaves room for CI. +const campaignTimeoutMs = 240_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: PowerSync`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) From 2cbae92f04c82ce5a00f27556f9862bf89b63901 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 11:48:23 -0700 Subject: [PATCH 12/28] test(db-adapter-conformance): a write is decided when the provider answers MUT-10 orders a write after the earlier one decides. The model read that as the earlier write's settlement, so under confirmation through sync a write had to wait for the earlier one's copy to arrive. The maintainer decided (2026-11-03) that "decides" is the provider's answer: core does not chain independent transactions, and an application that needs that chains them itself. Waiting for confirmation had made a burst of 50 PowerSync inserts take 1.6 s instead of 31 ms. The model now treats an answered write as decided, so the next write's request is owed at once; the answered write stays pending until its copy arrives (MUT-7). `deliver` delivers the oldest withheld copy, and `held-echoes` is declared only by a driver that can withhold each write's copy on its own. The reference SQL adapter now starts the next write at the provider's answer and confirms each write by its own commit. Its earlier confirmation by the highest applied commit confirmed a withheld copy as soon as a later one applied; that design is now a mutant. - Law: MUT-10 as decided; MUT-7. - RED: under the new model the reference SQL adapter failed MUT-1 (a withheld insert confirmed by a later insert's copy), and the committed TrailBase, which waits for each event, fails the fixed campaign. - GREEN: db-adapter-conformance 104 passed. - Mutants: `chains-on-confirmation` (waits for the earlier copy) fails MUT-8, a request owed after the answer is missing; `confirms-by-high-water` fails MUT-1. Co-authored-by: Isaac --- .changeset/powersync-ordered-writes.md | 5 - .../src/mutation-oracle.ts | 71 +++++++------ .../tests/reference-mutation-adapter.ts | 99 +++++++++++++------ ...reference-mutation-oracle.property.test.ts | 2 + 4 files changed, 113 insertions(+), 64 deletions(-) delete mode 100644 .changeset/powersync-ordered-writes.md diff --git a/.changeset/powersync-ordered-writes.md b/.changeset/powersync-ordered-writes.md deleted file mode 100644 index fa56f84386..0000000000 --- a/.changeset/powersync-ordered-writes.md +++ /dev/null @@ -1,5 +0,0 @@ ---- -'@tanstack/powersync-db-collection': patch ---- - -Persist writes to PowerSync in the order the application made them. A write now starts only after the earlier write on the same collection is observed by the collection or rejected. Before, a later write could start as soon as the earlier one was written to SQLite, before the collection had observed it. diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index b79b6eccb2..11954f737c 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -34,7 +34,10 @@ * as an own `undefined`. A driver declares this with `undefinedField`. * - **MUT-10.** Writes are ordered per Collection: a write submitted while an * earlier one is pending reaches the provider only after the earlier one - * is decided, accepted or rejected. + * is decided, accepted or rejected by the provider's answer. A write does + * not wait for an earlier one's confirmation through sync: core does not + * chain independent transactions, and an application that needs that + * chains them itself (maintainer decision 2026-11-03). * - **FID-5.** At a quiescent cut the public rows, the exposed authoritative * base (`collection.base`), and the provider's rows are identical, and all * three equal the model: the provider's starting rows with every accepted @@ -100,7 +103,9 @@ export type MutationCapability = /** * The provider can answer an accepted write before its change feed * delivers the copy, which it then delivers at `deliver`. Declare it only - * for an adapter that confirms writes through sync (MUT-7). + * for an adapter that confirms writes through sync (MUT-7), and only when + * the provider can withhold each write's copy on its own: later writes + * proceed while a copy is withheld, and `deliver` delivers the oldest. */ | `held-echoes` @@ -537,9 +542,10 @@ type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } * the value domain alone, as a removed field does under `null`. * - `pending` lists the mutations not yet settled, in author order. MUT-10 * orders their requests: the first is requested at once, and each later - * one only after every earlier one is decided. An accepted write whose - * copy the provider withholds is `answered`: it stays pending, and so - * holds later writes, until the copy arrives (MUT-7). + * one only after every earlier one is decided. A write is decided when + * the provider answers it. An accepted write whose copy the provider + * withholds is `answered`: it is decided, so the next write is + * requested, but it stays pending until its copy arrives (MUT-7). * - `requests` is every write request MUT-8 and MUT-10 allow so far. * - `settlements` is each write's `when('settled')` settlement. * - `applied` is the synced rows core has applied. An accepted write's @@ -728,32 +734,36 @@ function retire(state: MutationModel, write: PendingWrite): void { } /** - * MUT-10: request the first pending mutation if it has not been; a request - * answered at once is decided at once, which admits the next unless its - * copy is withheld. Returns the public rows after each decision, for the - * publication check. + * MUT-10: request the oldest undecided mutation if it has not been; a request + * answered at once is decided at once, which admits the next. A write is + * decided when the provider answers it, also when its copy is withheld + * (maintainer decision 2026-11-03). Returns the public rows after each + * decision, for the publication check. */ function advance(state: MutationModel): Array { const states: Array = [] for (;;) { - const first = state.pending[0] - if (!first || first.answered) return states - if (!first.requested) { - first.requested = true + const next = undecided(state) + if (!next) return states + if (!next.requested) { + next.requested = true state.requests.push({ - type: first.type, - id: first.id, - ...(first.sent === undefined - ? {} - : { value: requestValue(first.sent) }), + type: next.type, + id: next.id, + ...(next.sent === undefined ? {} : { value: requestValue(next.sent) }), }) } - if (first.plan.held) return states - decide(state, first) + if (next.plan.held) return states + decide(state, next) states.push(visible(state)) } } +/** The oldest pending write the provider has not answered. */ +function undecided(state: MutationModel): PendingWrite | undefined { + return state.pending.find((write) => !write.answered) +} + /** What a step did, for reach and law names. */ type StepDone = | { @@ -829,20 +839,19 @@ function stepModel( } } case `release`: { - const first = state.pending[0] - if (!first?.requested || first.answered || !first.plan.held) - return undefined - decide(state, first) + const next = undecided(state) + if (!next?.requested || !next.plan.held) return undefined + decide(state, next) const published = [visible(state), ...advance(state)] return { - done: { kind: `release`, outcome: first.plan.outcome }, + done: { kind: `release`, outcome: next.plan.outcome }, published, } } case `deliver`: { - const first = state.pending[0] - if (!first?.answered) return undefined - confirm(state, first) + const oldest = state.pending.find((write) => write.answered) + if (!oldest) return undefined + confirm(state, oldest) const published = [visible(state), ...advance(state)] return { done: { kind: `deliver` }, published } } @@ -1266,9 +1275,9 @@ export async function checkMutationHistory( // quiescent cut (FID-5). while (state.pending.length > 0) { const before = visible(state) - const step: MutationStep = state.pending[0]!.answered - ? { kind: `deliver` } - : { kind: `release` } + const step: MutationStep = undecided(state) + ? { kind: `release` } + : { kind: `deliver` } const result = stepModel(state, step, planFor) if (result === undefined) break if (state.heldRows.length > 0) reach(`held-optimistic-row`) diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index 81e2392817..c08edc1332 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -25,7 +25,9 @@ * them into its row (`field-updates`). The provider can answer before its * change feed delivers the copy (`held-echoes`), so the adapter confirms * each write through sync: its mutation function returns when its own - * sync run has applied that write's copy (MUT-7). + * sync run has applied that write's copy (MUT-7). The next write starts + * when the provider answers, without waiting for that confirmation + * (MUT-10). */ import { createCollection } from '@tanstack/db' import { observedView, rewriteRow } from '../src' @@ -78,6 +80,17 @@ export type ReferenceMutationMutant = * before its copy arrives through sync. */ | `settles-on-answer` + /** + * MUT-10 (SQL): a write waits for the earlier one's confirmation through + * sync, not only for the provider's answer, so it reaches the provider + * late. + */ + | `chains-on-confirmation` + /** + * MUT-7 (SQL): a write counts as confirmed once any later commit's copy + * has applied, so a withheld copy is skipped. + */ + | `confirms-by-high-water` export interface ReferenceMutationOptions { /** A JSON provider. */ @@ -236,13 +249,22 @@ function referenceCollection( mutant: ReferenceMutationMutant | undefined, { sql = false }: ReferenceMutationOptions, ) { - // MUT-10: each write waits for the earlier one to be decided. + // MUT-10: each write waits for the earlier one to be decided, by the + // provider's answer. A write's turn covers sending it; its confirmation + // (SQL) follows after the turn. let turn: Promise = Promise.resolve() - const persist = (send: () => Promise): Promise => { - if (mutant === `writes-concurrently`) return send() - const run = turn.then(send) - turn = run.catch(() => undefined) - return run + const persist = async ( + sendAll: () => Promise>, + ): Promise => { + if (mutant === `writes-concurrently`) { + await confirmAll(await sendAll()) + return + } + const sent = turn.then(sendAll) + const chained = + mutant === `chains-on-confirmation` ? sent.then(confirmAll) : sent + turn = chained.catch(() => undefined) + await confirmAll(await sent) } const answer = async (sent: Promise): Promise => { try { @@ -256,17 +278,22 @@ function referenceCollection( } // MUT-7 (SQL): a write is confirmed when this sync run has applied the // commit its answer names. - let applied = 0 - const confirmations: Array<{ sequence: number; resolve: () => void }> = [] + // Copies can arrive out of commit order, since a withheld one trails later + // ones, so each write waits for its own commit, not for the highest seen. + const applied = new Set() + let highest = 0 + const confirmations = new Map void>() + const isApplied = (commit: number) => + mutant === `confirms-by-high-water` + ? highest >= commit + : applied.has(commit) const confirmed = (commit: number): Promise => { - if (!sql || mutant === `settles-on-answer` || applied >= commit) + if (!sql || mutant === `settles-on-answer` || isApplied(commit)) return Promise.resolve() - return new Promise((resolve) => - confirmations.push({ sequence: commit, resolve }), - ) + return new Promise((resolve) => confirmations.set(commit, resolve)) } - const send = async (request: WriteRequest) => - confirmed(await provider.write(request)) + const confirmAll = (commits: Array) => + Promise.all(commits.map(confirmed)).then(() => undefined) // The `writes-back-authored` mutant writes through the sync run's apply. let applyChange: ((change: ProviderChange) => void) | undefined const writeBack = (row: ConformanceRow) => { @@ -325,10 +352,13 @@ function referenceCollection( applyChange = apply const receive = (change: ProviderChange, commitSequence: number) => { apply(change) - applied = commitSequence - for (const waiting of confirmations.splice(0)) - if (waiting.sequence <= applied) waiting.resolve() - else confirmations.push(waiting) + applied.add(commitSequence) + highest = Math.max(highest, commitSequence) + for (const [commit, resolve] of confirmations) + if (isApplied(commit)) { + confirmations.delete(commit) + resolve() + } } return provider.subscribe((change, commitSequence) => { if (mutant === `applies-copy-late`) @@ -340,6 +370,7 @@ function referenceCollection( onInsert: ({ transaction }) => answer( persist(async () => { + const commits: Array = [] for (const mutation of transaction.mutations) { const request = { type: `insert` as const, @@ -349,32 +380,42 @@ function referenceCollection( ? { note: null as unknown as undefined, ...mutation.modified } : structuredClone(mutation.modified), } - await send(request) - if (mutant === `writes-insert-twice`) await send(request) + commits.push(await provider.write(request)) + if (mutant === `writes-insert-twice`) + commits.push(await provider.write(request)) writeBack(mutation.modified) } + return commits }), ), onUpdate: ({ transaction }) => answer( persist(async () => { + const commits: Array = [] for (const mutation of transaction.mutations) { - await send({ - type: `update`, - id: mutation.key, - row: updateRow(mutation), - }) + commits.push( + await provider.write({ + type: `update`, + id: mutation.key, + row: updateRow(mutation), + }), + ) writeBack(mutation.modified) } + return commits }), ), onDelete: ({ transaction }) => answer( persist(async () => { - if (mutant === `skips-delete`) return + if (mutant === `skips-delete`) return [] + const commits: Array = [] for (const mutation of transaction.mutations) { - await send({ type: `delete`, id: mutation.key }) + commits.push( + await provider.write({ type: `delete`, id: mutation.key }), + ) } + return commits }), ), }) @@ -385,6 +426,8 @@ const sqlMutants = new Set([ `sends-complete-row`, `drops-removal`, `settles-on-answer`, + `chains-on-confirmation`, + `confirms-by-high-water`, ]) /** A production driver for the reference adapter, optionally mutated. */ diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts index 0f8d94a178..618f567c2d 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -118,6 +118,8 @@ describe(`mutation conformance: reference adapter`, () => { [`sends-complete-row`, `MUT-8 (provider requests)`], [`drops-removal`, `MUT-8 (provider requests)`], [`settles-on-answer`, /MUT-1|MUT-7/], + [`chains-on-confirmation`, /MUT-8|MUT-10/], + [`confirms-by-high-water`, /MUT-1|MUT-7/], ] oracleReplayTest(it).each(mutants)( `rejects the %s mutant with %s`, From 6bc495bb229e793fdb1544db32660811f4da9b01 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 11:48:26 -0700 Subject: [PATCH 13/28] fix(trailbase-db-collection): start the next write at the earlier response Under MUT-10 as decided (a write is decided when the provider answers), a write waits for the earlier one's HTTP response, not for its change event. Each write still settles only once the change stream carries its own ids after it began. - Law: MUT-10 as decided 2026-11-03; MUT-7. - RED: the parent, which started a write only after the earlier one's event, fails the fixed campaign: the next request is missing at the cut after the answer. - GREEN: both TrailBase mutation campaigns pass; trailbase-db-collection 153 passed, including the synchronous first request. - Size: trailbase-db-collection -8 gzip bytes against main. Co-authored-by: Isaac --- .../trailbase-ordered-confirmed-writes.md | 2 +- .../trailbase-db-collection/src/trailbase.ts | 32 ++++++++++++------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/.changeset/trailbase-ordered-confirmed-writes.md b/.changeset/trailbase-ordered-confirmed-writes.md index 403bde458b..c0f2d0af44 100644 --- a/.changeset/trailbase-ordered-confirmed-writes.md +++ b/.changeset/trailbase-ordered-confirmed-writes.md @@ -5,5 +5,5 @@ Fix three write bugs. - Removing a field in an update (for example `draft.note = undefined`) now sends `null`, so TrailBase clears the column. Before, the update omitted the field, and TrailBase kept the old value. -- Writes now reach TrailBase in the order the application made them. A write starts only after the earlier write on the same collection is confirmed or rejected. +- Writes now reach TrailBase in the order the application made them. A write starts only after TrailBase has answered the earlier write on the same collection. It does not wait for the earlier write's change event. - A write is now confirmed only by a change event that arrives after the write began. Before, an earlier event for the same record could confirm a later write, so its optimistic value dropped and the old value showed until the new event arrived. This change also removes the adapter's periodic cleanup timer. diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index 5811d8675f..280d2922cd 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -182,20 +182,28 @@ export function trailBaseCollectionOptions< } // Writes reach TrailBase in author order: a write starts at once when no - // other is admitted, and otherwise after the earlier one is confirmed or - // rejected (MUT-10). A write returns the ids it wrote and settles once the - // change stream carries each of them (MUT-7). + // other is being sent, and otherwise after the earlier one's response + // (MUT-10). It does not wait for the earlier one's change event. A write + // returns the ids it wrote and settles once the change stream carries each + // of them (MUT-7). + let sending = 0 let turn: Promise = Promise.resolve() - const persist = (write: () => Promise>): Promise => { - const start = async () => { - const after = eventSequence - await awaitIds(await write(), after) - } - const run = (writes++ === 0 ? start() : turn.then(start)).finally(() => { + const persist = async (write: () => Promise>) => { + writes++ + try { + const send = async () => { + const after = eventSequence + return { ids: await write(), after } + } + const sent = (sending++ === 0 ? send() : turn.then(send)).finally(() => { + sending-- + }) + turn = sent.catch(() => {}) + const { ids, after } = await sent + await awaitIds(ids, after) + } finally { if (--writes === 0) seenIds.setState(() => new Map()) - }) - turn = run.catch(() => {}) - return run + } } let eventReader: ReadableStreamDefaultReader | undefined From 24bf6f0e504ccc6ba4976aedffd123aaf116a005 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 11:48:53 -0700 Subject: [PATCH 14/28] revert(powersync-db-collection): keep writes pipelined behind the write lock The previous fix made each PowerSync write wait until the earlier one's diff trigger had observed it. Under MUT-10 as decided (2026-11-03), a write is decided when the database answers it, and PowerSync's write lock already orders its writes, so that wait only slowed bursts (50 inserts: 31 ms before, 1.6 s with it). The adapter is restored as it was; its changeset was removed in 2cbae92f0. The driver no longer declares `held-writes` or `held-echoes`. A held statement holds the database's one write lock, which the adapter's diff flush also needs, so it delayed the confirmation of earlier writes, a coupling the model does not represent; and one watcher callback applies every committed write, so the driver cannot withhold one write's copy while a later write proceeds. The propagation step is simpler for it. - Law: MUT-10 as decided; MUT-7; MUT-8 under `field-updates`. - GREEN: both PowerSync campaigns pass with no adapter change; powersync-db-collection 198 passed. - Mutant: an `applyTransaction` that does not wait for its diff trigger to observe the write fails MUT-1 at the first insert. - Limits: no held write or held echo for PowerSync (coverage map). - Size: powersync-db-collection unchanged against main. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 48 +++++++----- packages/db-adapter-conformance/MIGRATION.md | 23 +++++- .../powersync-db-collection/src/powersync.ts | 27 +++---- .../tests/mutation-conformance-driver.ts | 77 ++++--------------- ...tation-conformance-oracle.property.test.ts | 5 +- 5 files changed, 78 insertions(+), 102 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 066466ec16..a728a24693 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7257,8 +7257,11 @@ removed field is sent in the provider's representation, `null` for a SQL provider (`undefinedField: 'null'`, which also sends a complete row with every column). With `held-echoes`, the provider can answer an accepted write before its change feed delivers the copy, at a `deliver` step: under MUT-7 the -mutation stays pending, and holds later writes (MUT-10), until its copy -arrives. A driver can declare several `rejectionKinds`, each reached. Reach: each of insert, update, and delete +mutation stays pending until its copy arrives. MUT-10's "the earlier one +decides" is the provider's answer, so a later write's request is owed once +the earlier write is answered, without waiting for its copy (maintainer +decision 2026-11-03: core does not chain independent transactions; an +application that needs that chains them itself). A driver can declare several `rejectionKinds`, each reached. Reach: each of insert, update, and delete accepted, a rejected write, a held write, a write queued behind a held one, a held write rejected, a commit by another client, an accepted field removal, a held optimistic row, a copy the provider rewrote, an answered write whose copy @@ -7330,24 +7333,27 @@ MUT-1 (a write was confirmed by an earlier event on the same key, so its optimistic row dropped before its own event arrived). The adapter now sends a removed field as `null`, runs each write after the earlier one settles, and confirms a write only by an event after it began; a mutant that keeps the -old confirmation fails MUT-1. The record API is a test double that bypasses +old confirmation fails MUT-1. A write starts after the earlier one's HTTP +response, not after its change event; the version that waited for the event +fails the fixed campaign (a request owed after the answer is missing). The record API is a test double that bypasses the SDK's fetch and SSE decoding. -PowerSync (`undefinedField: 'null'`, `field-updates`, `held-echoes`, -`write-rejection`, `held-writes`, `foreign-commits`): the driver runs a real -local PowerSync database, which is the adapter's provider; uploading to a -PowerSync service is the SDK's upload queue and is outside this owner. A -request is one `INSERT`, `UPDATE`, or `DELETE` the adapter executes on the -table inside its write transaction; a rejection throws from that statement, -and a held request holds the write lock. A held echo holds the change -watcher's next callback until `deliver`. The propagation step waits until -every write's receipt settles, or, while a request or echo is held, until the -request log, the watcher, and the receipts are quiet. The local database does -not rewrite rows, so `server-rewrites` is not declared. On the parent of the -fix, both campaigns failed MUT-10: a write started once the earlier one -committed locally, before its diff trigger observed it. MUT-10's "decides" -is settlement (the law set counts a synchronously failed handler as decided -before its receipt settles, as the exception). The adapter now runs each -write after the earlier one settles. Removal binds an own `undefined` as -`NULL`, so MUT-8 holds. Limits: removal by assigning `null` and -multi-mutation transactions are not generated. +PowerSync (`undefinedField: 'null'`, `field-updates`, `write-rejection`, +`foreign-commits`): the driver runs a real local PowerSync database, which is +the adapter's provider; uploading to a PowerSync service is the SDK's upload +queue and is outside this owner. A request is one `INSERT`, `UPDATE`, or +`DELETE` the adapter executes on the table inside its write transaction; a +rejection throws from that statement and rolls the transaction back. The +database's write lock orders the adapter's writes, which meets MUT-10 under +the decided-on-answer rule, and each write settles once the adapter's diff +trigger has observed it (MUT-7); the watcher's 30 ms throttle makes that +copy arrive late, and a mutant that settles without waiting for it fails +MUT-1. The adapter needed no change. The propagation step waits until every +write's receipt settles and the watcher has drained. Limits: `held-writes` is +not declared, because a held statement holds the one write lock that the +adapter's diff flush also needs, so it would delay earlier writes' +confirmation, a coupling the model does not represent; `held-echoes` is not +declared, because one watcher callback applies every committed write; +`server-rewrites` is not declared, because the local database rewrites +nothing; removal by assigning `null` and multi-mutation transactions are not +generated. diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 9ec4808096..e21781d6cb 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -31,7 +31,7 @@ reviewed change. | ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | | Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03), no adapter change, `undefinedField: 'null'`, `field-updates`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | | RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | | TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | | Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | @@ -617,3 +617,24 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the so a driver without `server-rewrites` reached an undeclared state. It now counts only a plan's rewrite. - **Size:** powersync-db-collection +26 gzip bytes for MUT-10. +- **Decided 2026-11-03, MUT-10 under confirmation through sync.** "The + earlier one decides" is the provider's answer, not the write's + confirmation through sync: core does not chain independent transactions, + and an application that needs that chains them itself. Waiting for + confirmation made a burst of 50 PowerSync inserts take 1.6 s instead of + 31 ms. + - **Oracle:** an answered write is decided, so the next write's request is + owed; it stays pending until its copy arrives. The reference SQL adapter + now starts the next write at the answer and confirms each write by its + own commit; new mutants `chains-on-confirmation` (MUT-8) and + `confirms-by-high-water`, which confirms a withheld copy once a later + one applies (MUT-1). + - **PowerSync:** the ordering change is reverted; the write lock already + orders its writes. Its driver no longer declares `held-writes` or + `held-echoes` (a held statement holds the lock the diff flush needs, and + one watcher callback applies every write). GREEN with no adapter change; + a mutant that settles before its diff trigger observes the write fails + MUT-1. + - **TrailBase:** a write starts after the earlier one's HTTP response, not + after its change event. The committed version, which waited for the + event, fails the fixed campaign. diff --git a/packages/powersync-db-collection/src/powersync.ts b/packages/powersync-db-collection/src/powersync.ts index e673924678..dfb52c15f4 100644 --- a/packages/powersync-db-collection/src/powersync.ts +++ b/packages/powersync-db-collection/src/powersync.ts @@ -1199,17 +1199,6 @@ function createPowerSyncCollectionConfig< ...restConfig, } - // Writes reach the database in author order: each starts after the - // earlier one has been observed or rejected (MUT-10). - let turn: Promise = Promise.resolve() - const persist = ( - transaction: Parameters[0], - ): Promise => { - const run = turn.then(() => transactor.applyTransaction(transaction)) - turn = run.catch(() => {}) - return run - } - const outputConfig: EnhancedPowerSyncCollectionConfig< TTable, OutputType, @@ -1222,10 +1211,18 @@ function createPowerSyncCollectionConfig< startSync: true, syncMode, sync, - // Each handler's transaction holds a single mutation of its type. - onInsert: (params) => persist(params.transaction), - onUpdate: (params) => persist(params.transaction), - onDelete: (params) => persist(params.transaction), + onInsert: async (params) => { + // The transaction here should only ever contain a single insert mutation + return await transactor.applyTransaction(params.transaction) + }, + onUpdate: async (params) => { + // The transaction here should only ever contain a single update mutation + return await transactor.applyTransaction(params.transaction) + }, + onDelete: async (params) => { + // The transaction here should only ever contain a single delete mutation + return await transactor.applyTransaction(params.transaction) + }, utils: { getMeta: () => ({ tableName: viewName, diff --git a/packages/powersync-db-collection/tests/mutation-conformance-driver.ts b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts index a5735638de..946902bc5f 100644 --- a/packages/powersync-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts @@ -15,12 +15,17 @@ * (`field-updates`); the request carries the bound values; * - a plan answers the next request inside the adapter's transaction: a * rejection throws the plan's error from the statement, as SQLite does for - * a failed write, which rolls the transaction back; a held request waits - * until `release`, holding the write lock as a slow disk does; + * a failed write, which rolls the transaction back. The driver does not + * declare `held-writes`: a held statement holds the database's one write + * lock, which the adapter's diff flush also needs, so it would delay the + * confirmation of earlier writes, a coupling the model does not represent; * - the adapter applies the database's changes through its change watcher - * (`onChangeWithCallback`). With a held echo, the watcher's next callback - * waits until `deliver` (`held-echoes`), as a throttled or busy watcher - * delays it; the adapter must not settle the write before (MUT-7); + * (`onChangeWithCallback`), which runs at most every 30 ms, so each write's + * copy arrives after the database answers; the adapter must not settle + * the write before it applies (MUT-7), which the publication check sees. + * The driver does not declare `held-echoes`: one watcher callback applies + * every committed write, so it cannot withhold one write's copy while a + * later write proceeds; * - a commit by another client is a write to the table that is not the * adapter's, as the SDK's download of server changes is; * - the provider's rows are the table's rows. `note` is a nullable text @@ -77,8 +82,6 @@ const acceptAtOnce: WritePlan = { const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) const drainTimeoutMs = 5_000 -// How long nothing must change before a cut with a held request or echo. -const quietMs = 25 /** Field names in a statement, without the quoting `sanitizeSQL` adds. */ const unquote = (name: string) => name.trim().replace(/^['"]|['"]$/g, ``) @@ -131,11 +134,7 @@ class PowerSyncMutationProvider implements MutationProvider { readonly database: PowerSyncDatabase readonly file: string private readonly plans: Array = [] - private readonly held: Array<() => void> = [] private readonly log: Array = [] - /** Opens the watcher's next callback when an echo is withheld. */ - private echoes: Array<() => void> = [] - private gate: Promise | undefined private running = 0 private drained = 0 private drainTarget: number | undefined @@ -172,10 +171,7 @@ class PowerSyncMutationProvider implements MutationProvider { if (!request) return execute(sql, params) this.log.push(request) const plan = this.plans.shift() ?? acceptAtOnce - if (plan.held) - await new Promise((resolve) => this.held.push(resolve)) if (plan.outcome === `reject`) throw plan.error - if (plan.echoHeld) this.withholdEcho() wrote = true return execute(sql, params) } @@ -184,8 +180,7 @@ class PowerSyncMutationProvider implements MutationProvider { if (wrote) this.expectDrain() return result }) as PowerSyncDatabase[`writeTransaction`] - // The adapter's change watcher: count its drains, and hold its next - // callback while an echo is withheld. + // The adapter's change watcher: count its drains. const register = this.database.onChangeWithCallback.bind(this.database) this.database.onChangeWithCallback = (handler, options) => register( @@ -194,7 +189,6 @@ class PowerSyncMutationProvider implements MutationProvider { onChange: async (event) => { this.running++ try { - await this.gate await handler?.onChange(event) } finally { this.running-- @@ -214,10 +208,6 @@ class PowerSyncMutationProvider implements MutationProvider { this.drainTarget = this.drained + this.running + 1 } - private withholdEcho(): void { - this.gate = new Promise((resolve) => this.echoes.push(resolve)) - } - /** * Writes that are not the adapter's: the seed and other clients. A table * is a view, so an upsert is an update or an insert. A delete of a row the @@ -279,21 +269,8 @@ class PowerSyncMutationProvider implements MutationProvider { this.plans.push(plan) } - release(): void { - this.held.shift()?.() - } - - deliver(): void { - this.echoes.shift()?.() - if (this.echoes.length === 0) this.gate = undefined - } - - /** Answers every held request and opens every withheld echo. */ - drain(): void { - for (const answer of this.held.splice(0)) answer() - for (const open of this.echoes.splice(0)) open() - this.gate = undefined - } + /** No request is held: the driver does not declare `held-writes`. */ + release(): void {} foreignCommit(change: ProviderChange): Promise { return this.write(change) @@ -313,17 +290,11 @@ class PowerSyncMutationProvider implements MutationProvider { /** * The propagation step (`propagationStep`: the diff-trigger watch). The * adapter reaches its write transaction a few milliseconds after a write, - * so the step waits until every write's receipt has settled and the - * watcher has drained every commit. While a request or an echo is held, - * some writes legitimately wait, so it waits instead until the request - * log, the watcher, and the receipts are quiet. + * and the watcher runs at most every 30 ms, so the step waits until every + * write's receipt has settled and the watcher has drained every commit. */ settle = async () => { const deadline = Date.now() + drainTimeoutMs - const state = () => - `${this.log.length} ${this.drained} ${this.running} ${this.unsettled}` - let last = state() - let quietSince = Date.now() while (Date.now() < deadline) { const drainedAll = (this.drainTarget === undefined || this.drained >= this.drainTarget) && @@ -332,14 +303,7 @@ class PowerSyncMutationProvider implements MutationProvider { this.drainTarget = undefined return } - const blocked = this.held.length > 0 || this.gate !== undefined - if (blocked && Date.now() - quietSince >= quietMs) return await sleep(5) - const now = state() - if (now !== last) { - last = now - quietSince = Date.now() - } } } @@ -360,10 +324,8 @@ export function powerSyncMutationDriver(): MutationDriver { adapter: `@tanstack/powersync-db-collection (eager)`, capabilities: new Set([ `write-rejection`, - `held-writes`, `foreign-commits`, `field-updates`, - `held-echoes`, ]), undefinedField: `null`, async create(seed): Promise { @@ -379,13 +341,6 @@ export function powerSyncMutationDriver(): MutationDriver { row: conformanceRow, key: conformanceKey, }) - // A held request holds the write lock, which cleanup waits for; a - // history that ends early with one held must answer it first. - const cleanup = view.cleanup.bind(view) - view.cleanup = () => { - provider.drain() - return cleanup() - } return { collection: view, base: () => [...collection.base.values()].map(conformanceRow), @@ -411,8 +366,6 @@ export function powerSyncMutationDriver(): MutationDriver { provider, settle: provider.settle, dispose: async () => { - // A write waiting for its echo would otherwise outlive the history. - provider.drain() await collection.cleanup() await provider.dispose() }, diff --git a/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts index d695c3e6f9..7ba874d6cb 100644 --- a/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts +++ b/packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -29,9 +29,8 @@ import { powerSyncMutationDriver } from './mutation-conformance-driver' * The driver runs a real local PowerSync database, the adapter's provider: * the adapter persists each write to it and settles once its diff trigger * has observed the write (`confirm-through-sync`, MUT-7). The database can - * reject or hold a write, another client can write, the watcher can deliver - * a write's change late, and updates set only their changed columns (the law - * set's `field-updates`). + * reject or hold a write, another client can write, and updates set only + * their changed columns (the law set's `field-updates`). */ const property = `adapter-conformance.mutations.powersync` From 226b42963abc38d763c979b96aa9e6e16c0e99ab Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 12:33:14 -0700 Subject: [PATCH 15/28] test(db-adapter-conformance): application handlers, log-ordered echoes, in-order answers Electric and Convex do not persist writes: the application's handlers do, through its own backend, and the adapter confirms them through sync. The mutation oracle now models those providers: - `application-handlers`: each write's request goes out when it is made. The adapter cannot order the application's handlers, and core does not chain independent transactions, so MUT-10 is the application's (maintainer decision 2026-11-03). - `ordered-answers`: the provider answers requests in arrival order, as a Convex server runs one client's mutations in order. - `log-ordered-echoes`: the change feed delivers copies in commit order, as Electric's shape log and Convex's transitions do, so a withheld copy holds back every later one; `deliver` releases it with the copies queued behind it. Writes that settle at one step settle in an order no law fixes, so the publication check then compares each row's own sequence of values, which still rejects an earlier synced value or an extra change on any row. A step that settles one write keeps the whole-sequence comparison. The reference adapter separates confirmation through sync from SQL rows. - Law: MUT-1, MUT-7, MUT-10 as decided. - Reach: new state `echo-queued-behind-withheld`; Electric-like (SQL, log, application handlers) and Convex-like (JSON, log, application handlers, in-order answers) reference campaigns meet every declared state. - RED: a provider whose copies overtake a withheld one fails MUT-7 under `log-ordered-echoes`; an adapter that orders its writes fails MUT-8 under `application-handlers`; a provider that answers out of order fails MUT-13 under `ordered-answers`. Before the per-row rule, the Convex-like campaign failed MUT-1 (publication) where a rejection and an earlier acceptance settled together in the other order. - GREEN: db-adapter-conformance 106 passed; the RxDB, TrailBase, and PowerSync mutation campaigns pass. Co-authored-by: Isaac --- .../src/mutation-oracle.ts | 151 +++++++++++++++--- .../tests/reference-mutation-adapter.ts | 76 +++++++-- ...reference-mutation-oracle.property.test.ts | 42 +++++ 3 files changed, 235 insertions(+), 34 deletions(-) diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 11954f737c..92d13a3457 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -108,6 +108,27 @@ export type MutationCapability = * proceed while a copy is withheld, and `deliver` delivers the oldest. */ | `held-echoes` + /** + * The provider's change feed delivers copies in commit order, as a log + * does: a withheld copy also holds back every later commit's copy, and + * `deliver` releases it with the copies queued behind it. + */ + | `log-ordered-echoes` + /** + * The application's own handlers persist each write; the adapter supplies + * only confirmation utilities. The adapter cannot order the handlers, so + * each write's request goes out when it is made: MUT-10 is the + * application's, since core does not chain independent transactions + * (maintainer decision 2026-11-03). + */ + | `application-handlers` + /** + * The provider answers requests in the order they arrive, as a server + * that runs one client's mutations in order does: a request behind a held + * one waits for it. Meaningful with `application-handlers`, whose requests + * go out at once. + */ + | `ordered-answers` /** The optimistic transaction a write call returns. */ export interface MutationReceipt { @@ -521,8 +542,10 @@ interface PendingWrite { readonly sent?: Readonly> readonly plan: WritePlan requested: boolean - /** Accepted, with its copy withheld from the change feed (`held-echoes`). */ + /** Accepted, with its copy not yet delivered by the change feed. */ answered: boolean + /** Its own plan withheld its copy (`held-echoes`). */ + withheld: boolean } type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } @@ -568,10 +591,18 @@ interface MutationModel { readonly undefinedField: `own` | `absent` | `null` /** The driver declares `field-updates`. */ readonly fieldUpdates: boolean + /** The adapter orders writes (MUT-10): no `application-handlers`. */ + readonly ordered: boolean + /** The driver declares `log-ordered-echoes`. */ + readonly logOrdered: boolean + /** The driver declares `ordered-answers`. */ + readonly answersInOrder: boolean provider: Map applied: Map heldRows: Array pending: Array + /** Answered writes whose copies are not delivered, in commit order. */ + echoes: Array rewritten: number requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> settlements: Array @@ -579,17 +610,20 @@ interface MutationModel { function initialModel( seed: ReadonlyArray, - undefinedField: MutationModel[`undefinedField`] = `own`, - fieldUpdates = false, + domain: Pick, ): MutationModel { const provider = new Map(seed.map((row) => [row.id, rowValue(row)])) return { - undefinedField, - fieldUpdates, + undefinedField: domain.undefinedField ?? `own`, + fieldUpdates: domain.capabilities.has(`field-updates`), + ordered: !domain.capabilities.has(`application-handlers`), + logOrdered: domain.capabilities.has(`log-ordered-echoes`), + answersInOrder: domain.capabilities.has(`ordered-answers`), provider, applied: new Map(provider), heldRows: [], pending: [], + echoes: [], rewritten: 0, requests: [], settlements: [], @@ -712,8 +746,22 @@ function decide(state: MutationModel, write: PendingWrite): void { state.provider.set(write.id, copy) if (write.plan.rewrite !== `none` && copy !== write.value) state.rewritten++ } - if (write.plan.echoHeld) write.answered = true - else confirm(state, write) + // A copy is withheld by its own plan or, from a log, by an earlier one. + if (write.plan.echoHeld || (state.logOrdered && state.echoes.length > 0)) { + write.answered = true + write.withheld = write.plan.echoHeld + state.echoes.push(write) + } else confirm(state, write) +} + +/** + * The oldest withheld copy arrives, and from a log every later copy queued + * behind it, up to the next one its own plan withheld. + */ +function deliverEchoes(state: MutationModel): void { + confirm(state, state.echoes.shift()!) + while (state.logOrdered && state.echoes[0] && !state.echoes[0].withheld) + confirm(state, state.echoes.shift()!) } /** An accepted write's copy arrives: it fulfills. */ @@ -743,7 +791,21 @@ function retire(state: MutationModel, write: PendingWrite): void { function advance(state: MutationModel): Array { const states: Array = [] for (;;) { - const next = undecided(state) + // Ordered, the oldest undecided write goes next; with application + // handlers, every write goes out when it is made, and the provider + // answers each one not held, in arrival order with `ordered-answers`. + const next = state.ordered + ? undecided(state) + : (state.pending.find((write) => !write.requested) ?? + state.pending.find( + (write, index) => + !write.answered && + !write.plan.held && + (!state.answersInOrder || + state.pending + .slice(0, index) + .every((earlier) => earlier.answered)), + )) if (!next) return states if (!next.requested) { next.requested = true @@ -752,6 +814,7 @@ function advance(state: MutationModel): Array { id: next.id, ...(next.sent === undefined ? {} : { value: requestValue(next.sent) }), }) + if (!state.ordered) continue } if (next.plan.held) return states decide(state, next) @@ -818,6 +881,7 @@ function stepModel( plan: plan(step), requested: false, answered: false, + withheld: false, } state.settlements.push(`pending`) state.pending.push(write) @@ -839,8 +903,11 @@ function stepModel( } } case `release`: { - const next = undecided(state) - if (!next?.requested || !next.plan.held) return undefined + // The provider answers its oldest held request. + const next = state.pending.find( + (write) => !write.answered && write.requested && write.plan.held, + ) + if (!next) return undefined decide(state, next) const published = [visible(state), ...advance(state)] return { @@ -849,9 +916,8 @@ function stepModel( } } case `deliver`: { - const oldest = state.pending.find((write) => write.answered) - if (!oldest) return undefined - confirm(state, oldest) + if (state.echoes.length === 0) return undefined + deliverEchoes(state) const published = [visible(state), ...advance(state)] return { done: { kind: `deliver` }, published } } @@ -866,6 +932,28 @@ function stepModel( } } +/** + * Each row's sequence of values from `start` through `states`, a value that + * repeats counted once; an absent row is `null`. + */ +function rowPaths(start: RowState, states: ReadonlyArray): string { + const valueOf = (rows: RowState, id: number) => + rows.find(([key]) => key === id)?.[1] ?? null + const ids = [ + ...new Set([start, ...states].flatMap((rows) => rows.map(([id]) => id))), + ].sort((a, b) => a - b) + return JSON.stringify( + ids.map((id) => { + const path = [valueOf(start, id)] + for (const rows of states) { + const value = valueOf(rows, id) + if (value !== path.at(-1)) path.push(value) + } + return [id, path] + }), + ) +} + /** * The publications a step owes: each change of the public rows, in order. A * change that leaves the rows as they were publishes nothing, as an accepted @@ -902,6 +990,7 @@ export const mutationReachStates = [ `held-optimistic-row`, `server-rewrote-copy`, `echo-held`, + `echo-queued-behind-withheld`, `quiescent-after-writes`, ] as const @@ -920,6 +1009,7 @@ const capabilityOfState: Partial< 'held-optimistic-row': `held-writes`, 'server-rewrote-copy': `server-rewrites`, 'echo-held': `held-echoes`, + 'echo-queued-behind-withheld': `log-ordered-echoes`, } /** The declared states a driver's capabilities let histories reach. */ @@ -1071,11 +1161,7 @@ export async function checkMutationHistory( } const fixture = created const { collection, provider, writer } = fixture - const state = initialModel( - history.seed, - driver.undefinedField, - driver.capabilities.has(`field-updates`), - ) + const state = initialModel(history.seed, driver) const receipts: Array<{ current: Settlement }> = [] const failures: Array = [] let accepted = 0 @@ -1169,19 +1255,28 @@ export async function checkMutationHistory( stepIndex: number, owed: ReadonlyArray, law: string, + settledSeveral: boolean, ) => { // Each change of row values, in order; a publication of the same row // values (a virtual row field change) is not one. + const start = published const got: Array = [] for (const rows of publications!.take()) { if (!sameRows(rows, got.at(-1) ?? published)) got.push(rows) } published = got.at(-1) ?? published - const ok = - got.length === owed.length && - got.every((rows, index) => sameRows(rows, owed[index]!)) + // Writes that settle at one step settle in an order no law fixes, so + // then each row's own sequence of values is compared: it does not depend + // on that order, and it still shows an earlier synced value or an extra + // change. + const ok = settledSeveral + ? rowPaths(start, got) === rowPaths(start, owed) + : got.length === owed.length && + got.every((rows, index) => sameRows(rows, owed[index]!)) if (!ok) throw violation(`${law} (publication)`, stepIndex, owed, got) } + const settledCount = () => + state.settlements.filter((settlement) => settlement !== `pending`).length let rewritten = 0 // The plan the model decided the last write by; the provider gets it too. @@ -1210,6 +1305,7 @@ export async function checkMutationHistory( await compare(-1) for (const [index, step] of steps.entries()) { const before = visible(state) + const settledBefore = settledCount() const result = stepModel(state, step, planFor) if (result === undefined) { skipped++ @@ -1267,14 +1363,22 @@ export async function checkMutationHistory( if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) if (state.pending.some((write) => write.answered)) reach(`echo-held`) + if (state.echoes.some((write) => !write.withheld)) + reach(`echo-queued-behind-withheld`) rewritten = state.rewritten await compare(index) - checkPublications(index, owedPublications(before, result.published), law) + checkPublications( + index, + owedPublications(before, result.published), + law, + settledCount() - settledBefore > 1, + ) } // Answer every held request and deliver every withheld copy, reaching a // quiescent cut (FID-5). while (state.pending.length > 0) { const before = visible(state) + const settledBefore = settledCount() const step: MutationStep = undecided(state) ? { kind: `release` } : { kind: `deliver` } @@ -1283,6 +1387,8 @@ export async function checkMutationHistory( if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) if (state.pending.some((write) => write.answered)) reach(`echo-held`) + if (state.echoes.some((write) => !write.withheld)) + reach(`echo-queued-behind-withheld`) rewritten = state.rewritten if (step.kind === `deliver`) await provider.deliver!() else await provider.release() @@ -1291,6 +1397,7 @@ export async function checkMutationHistory( steps.length, owedPublications(before, result.published), `MUT-1`, + settledCount() - settledBefore > 1, ) } } catch (error) { diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index c08edc1332..c4a7067bc3 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -97,6 +97,18 @@ export interface ReferenceMutationOptions { json?: boolean /** A SQL provider, with field updates and confirmation through sync. */ sql?: boolean + /** + * The provider can answer before its change feed delivers the copy, so + * the adapter confirms each write through sync (`held-echoes`); a SQL + * provider implies it. + */ + confirms?: boolean + /** The change feed is a log: a withheld copy holds back later ones. */ + log?: boolean + /** The application's handlers persist writes: the adapter orders none. */ + application?: boolean + /** The provider answers requests in arrival order. */ + inOrder?: boolean } const acceptAtOnce: WritePlan = { @@ -114,7 +126,8 @@ export class ControlledMutationProvider implements MutationProvider { private readonly committed = new Map() private readonly plans: Array = [] private readonly held: Array<() => void> = [] - private readonly withheld: Array<() => void> = [] + /** Copies not yet fed, in commit order; `own` marks a plan's withholding. */ + private readonly withheld: Array<{ feed: () => void; own: boolean }> = [] private readonly log: Array = [] private listener: ((change: ProviderChange, sequence: number) => void) | undefined @@ -190,11 +203,30 @@ export class ControlledMutationProvider implements MutationProvider { ), ) } - if (plan.held) this.held.push(answer) - else answer() + // In order, an answer waits behind every earlier unanswered request. + const entry = { answer, held: plan.held } + this.unanswered.push(entry) + if (plan.held) this.held.push(() => this.answerEntry(entry)) + else if (!this.options.inOrder || this.unanswered[0] === entry) + this.answerEntry(entry) }) } + /** Requests not yet answered, in arrival order. */ + private readonly unanswered: Array<{ answer: () => void; held: boolean }> = [] + + private answerEntry(entry: { answer: () => void; held: boolean }): void { + this.unanswered.splice(this.unanswered.indexOf(entry), 1) + entry.answer() + // In order, the requests behind it that are not held follow. + while ( + this.options.inOrder && + this.unanswered[0] && + !this.unanswered[0].held + ) + this.answerEntry(this.unanswered[0]) + } + /** * Commits a change and feeds it back, at once or at `deliver`. Without * field updates a write replaces the whole row, as a complete-row API does. @@ -209,7 +241,9 @@ export class ControlledMutationProvider implements MutationProvider { ? change : { ...change, row: this.stored(change.row) } const feed = () => this.listener?.(fed, sequence) - if (withhold) this.withheld.push(feed) + // A log feeds in commit order, so a copy waits behind a withheld one. + if (withhold || (this.options.log && this.withheld.length > 0)) + this.withheld.push({ feed, own: withhold }) else feed() return sequence } @@ -225,7 +259,9 @@ export class ControlledMutationProvider implements MutationProvider { } deliver(): void { - this.withheld.shift()?.() + this.withheld.shift()?.feed() + while (this.options.log && this.withheld[0] && !this.withheld[0].own) + this.withheld.shift()!.feed() } foreignCommit(change: ProviderChange): void { @@ -247,7 +283,11 @@ let sequence = 0 function referenceCollection( provider: ControlledMutationProvider, mutant: ReferenceMutationMutant | undefined, - { sql = false }: ReferenceMutationOptions, + { + sql = false, + confirms = sql, + application = false, + }: ReferenceMutationOptions, ) { // MUT-10: each write waits for the earlier one to be decided, by the // provider's answer. A write's turn covers sending it; its confirmation @@ -256,7 +296,7 @@ function referenceCollection( const persist = async ( sendAll: () => Promise>, ): Promise => { - if (mutant === `writes-concurrently`) { + if (mutant === `writes-concurrently` || application) { await confirmAll(await sendAll()) return } @@ -288,7 +328,7 @@ function referenceCollection( ? highest >= commit : applied.has(commit) const confirmed = (commit: number): Promise => { - if (!sql || mutant === `settles-on-answer` || isApplied(commit)) + if (!confirms || mutant === `settles-on-answer` || isApplied(commit)) return Promise.resolve() return new Promise((resolve) => confirmations.set(commit, resolve)) } @@ -436,6 +476,10 @@ export function referenceMutationDriver( { json = mutant === `echoes-authored-row`, sql = sqlMutants.has(mutant), + log = false, + application = false, + inOrder = false, + confirms = sql, }: ReferenceMutationOptions = {}, ): MutationDriver { const capabilities: Array = [ @@ -444,19 +488,27 @@ export function referenceMutationDriver( `foreign-commits`, `server-rewrites`, ] - if (sql) capabilities.push(`field-updates`, `held-echoes`) + if (sql) capabilities.push(`field-updates`) + if (confirms) capabilities.push(`held-echoes`) + if (log) capabilities.push(`log-ordered-echoes`) + if (application) capabilities.push(`application-handlers`) + if (inOrder) capabilities.push(`ordered-answers`) return { - adapter: `reference mutations${json ? ` (JSON provider)` : ``}${sql ? ` (SQL provider)` : ``}${mutant ? ` (${mutant})` : ``}`, + adapter: `reference mutations${json ? ` (JSON provider)` : ``}${sql ? ` (SQL provider)` : ``}${log ? ` (log)` : ``}${application ? ` (application handlers)` : ``}${inOrder ? ` (in order)` : ``}${mutant ? ` (${mutant})` : ``}`, ...(json && { undefinedField: `absent` as const }), ...(sql && { undefinedField: `null` as const }), capabilities: new Set(capabilities), create(seed): MutationFixture { const provider = new ControlledMutationProvider( seed, - { json, sql }, + { json, sql, log, inOrder }, mutant === `echoes-authored-row`, ) - const collection = referenceCollection(provider, mutant, { sql }) + const collection = referenceCollection(provider, mutant, { + sql, + confirms, + application, + }) return { collection: observedView(collection, { row: (row) => row, diff --git a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts index 618f567c2d..7ae5496e9f 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-oracle.property.test.ts @@ -38,6 +38,12 @@ import type { ReferenceMutationMutant } from './reference-mutation-adapter' * - **SQL provider.** The fixed campaign also runs with a provider like * TrailBase's or PowerSync's: `undefinedField: 'null'`, `field-updates`, * and `held-echoes`, so the adapter confirms each write through sync. + * - **Log provider, application handlers.** The fixed campaign also runs + * like Electric: a SQL provider whose change feed is a log + * (`log-ordered-echoes`) and handlers the application writes, which the + * adapter does not order (`application-handlers`). Another runs like + * Convex: a JSON provider that answers requests in arrival order + * (`ordered-answers`), with a log feed and application handlers. * - **Calibration.** Each hostile mutant of the reference adapter breaks one * law, and the fixed campaign must reject it with that law (ORC-006). */ @@ -50,6 +56,18 @@ const campaignTimeoutMs = 120_000 const jsonDriver = referenceMutationDriver(undefined, { json: true }) const sqlDriver = referenceMutationDriver(undefined, { sql: true }) +const logDriver = referenceMutationDriver(undefined, { + sql: true, + log: true, + application: true, +}) +const inOrderDriver = referenceMutationDriver(undefined, { + json: true, + log: true, + application: true, + inOrder: true, + confirms: true, +}) async function campaign( options: fc.Parameters<[unknown]>, @@ -105,6 +123,30 @@ describe(`mutation conformance: reference adapter`, () => { campaignTimeoutMs, ) + oracleReplayTest(it)( + `conforms with a SQL log provider and application handlers on a fixed-seed campaign`, + async () => { + const reach = await campaign( + { seed: 2118, numRuns: oracleRuns(runs) }, + logDriver, + ) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it)( + `conforms with an in-order JSON log provider and application handlers on a fixed-seed campaign`, + async () => { + const reach = await campaign( + { seed: 2118, numRuns: oracleRuns(runs) }, + inOrderDriver, + ) + reach.assertReached() + }, + campaignTimeoutMs, + ) + const mutants: Array<[ReferenceMutationMutant, string | RegExp]> = [ [`applies-copy-late`, `MUT-1`], [`swallows-rejection`, `MUT-2`], From 76a00a995abfdd56fa754f1e7613d5367f842c73 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 12:33:17 -0700 Subject: [PATCH 16/28] test(electric-db-collection): run the mutation oracle over Electric's shape stream A receiving driver runs the Electric adapter on the shared mutation oracle. Electric does not persist writes, so the driver supplies the application: handlers that write the complete SQL row through a backend API, which answers by plan with the commit's txid, then await the adapter's `awaitTxId`. The real `ShapeStream` reads a shape log over a controlled `fetchClient`: each commit is one batch with its txid, in commit order, and a held echo stalls the log until `deliver`. - Law: MUT-7 (a handler's confirmation waits for its own transaction in the shape stream), MUT-1, MUT-2, FID-5; MUT-8 and MUT-10 are the application's. - Reach: every declared mutation state, at least five histories each, in the fixed (seed 2118, 300 histories) and random campaigns, about 15 s each. - GREEN: both campaigns pass with no adapter change; electric-db-collection 1245 passed. - Mutant: a txid evidence check that accepts any txid fails MUT-1 at the first insert. - Limits: one sync run, so no reconnect or must-refetch while a write waits; `awaitMatch` is not exercised. The documented guidance handlers stay with `tests/documented-mutations-oracle.test.ts`. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 33 +- packages/db-adapter-conformance/MIGRATION.md | 41 +- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 7 +- .../tests/mutation-conformance-driver.ts | 426 ++++++++++++++++++ ...tation-conformance-oracle.property.test.ts | 77 ++++ 6 files changed, 568 insertions(+), 17 deletions(-) create mode 100644 packages/electric-db-collection/tests/mutation-conformance-driver.ts create mode 100644 packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index a728a24693..4a615a3c3a 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7223,9 +7223,11 @@ and a SQL provider), against RxDB in `packages/rxdb-db-collection/tests/mutation-conformance-oracle.property.test.ts` (`adapter-conformance.mutations.rxdb`), against TrailBase in `packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.trailbase`), and against PowerSync in +(`adapter-conformance.mutations.trailbase`), against PowerSync in `packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.powersync`). It checks auto-transaction +(`adapter-conformance.mutations.powersync`), and against Electric in +`packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts` +(`adapter-conformance.mutations.electric`). It checks auto-transaction writes on a ready Collection whose adapter persists them itself: MUT-1 (the public rows show the optimistic state while the mutation function runs, and no publication shows the earlier synced value when it fulfills), MUT-2 (a @@ -7261,7 +7263,17 @@ mutation stays pending until its copy arrives. MUT-10's "the earlier one decides" is the provider's answer, so a later write's request is owed once the earlier write is answered, without waiting for its copy (maintainer decision 2026-11-03: core does not chain independent transactions; an -application that needs that chains them itself). A driver can declare several `rejectionKinds`, each reached. Reach: each of insert, update, and delete +application that needs that chains them itself). With `application-handlers` +the application's own handlers persist writes and the adapter only confirms +them, so each write's request goes out when it is made and MUT-10 is the +application's; with `ordered-answers` the provider answers requests in arrival +order. With `log-ordered-echoes` the change feed delivers copies in commit +order, so a withheld copy holds back every later one. When one step settles +several writes, they settle in an order no law fixes, so the publication check +compares each row's own sequence of values instead of the whole publication +sequence (by the same maintainer decision): it still rejects an earlier synced +value or an extra change on any row. A step that settles one write keeps the +whole-sequence comparison. A driver can declare several `rejectionKinds`, each reached. Reach: each of insert, update, and delete accepted, a rejected write, a held write, a write queued behind a held one, a held write rejected, a commit by another client, an accepted field removal, a held optimistic row, a copy the provider rewrote, an answered write whose copy @@ -7357,3 +7369,18 @@ declared, because one watcher callback applies every committed write; `server-rewrites` is not declared, because the local database rewrites nothing; removal by assigning `null` and multi-mutation transactions are not generated. + +Electric (`application-handlers`, `log-ordered-echoes`, `held-echoes`, +`undefinedField: 'null'`, `write-rejection`, `held-writes`, `foreign-commits`, +`server-rewrites`): Electric does not persist writes. The driver's application +handlers write the complete SQL row through a backend API that answers by plan +with the commit's txid, then await the adapter's `awaitTxId`; the shape stream +is the real `@electric-sql/client` `ShapeStream` over a controlled +`fetchClient`, each commit one batch with its txid, in commit order, stalled +at a held echo until `deliver`. This checks what Electric owns, confirmation +through its shape stream (MUT-7), with the handlers' requests and the +backend's rows as the application's. GREEN with no adapter change; a mutant +whose txid evidence check accepts any txid fails MUT-1 at the first insert. +The existing `tests/documented-mutations-oracle.test.ts` owner keeps the +documented guidance handlers. Limits: one sync run (no reconnect or +must-refetch while a write waits); `awaitMatch` is not exercised. diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index e21781d6cb..d981b8c303 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -27,19 +27,19 @@ reviewed change. ## Adapters -| Adapter | Families that apply (section A) | Driver status | -| ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | -| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03), no adapter change, `undefinedField: 'null'`, `field-updates`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | -| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | -| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | -| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | -| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | -| local-only | local and shared stores, mutations | Not started | -| localStorage | local and shared stores, mutations | Not started | -| Persisted wrapper | persistence wrapper | Not started | -| Storage hosts (8) | storage hosts | Not started | +| Adapter | Families that apply (section A) | Driver status | +| ----------------- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | +| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Mutations: GREEN (2026-11-03), no adapter change, `application-handlers`, `log-ordered-echoes`, `held-echoes`. Driver `electric-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.electric`. Other families: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03), no adapter change, `undefinedField: 'null'`, `field-updates`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | +| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | +| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | +| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | +| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | +| local-only | local and shared stores, mutations | Not started | +| localStorage | local and shared stores, mutations | Not started | +| Persisted wrapper | persistence wrapper | Not started | +| Storage hosts (8) | storage hosts | Not started | ## Suggested order @@ -638,3 +638,18 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the - **TrailBase:** a write starts after the earlier one's HTTP response, not after its change event. The committed version, which waited for the event, fails the fixed campaign. +- **2026-11-03, Electric, mutations.** Both campaigns are GREEN with no + adapter change, with every declared reach state met. + - **Driver:** Electric does not persist writes, so the driver supplies + application handlers that write through a backend API and await + `awaitTxId`; the real `ShapeStream` reads a shape log that carries each + commit's txid in commit order. + - **Oracle changes this migration exposed:** `application-handlers` (the + application's handlers persist writes, so requests go out when made and + MUT-10 is the application's), `log-ordered-echoes` (a withheld copy holds + back later ones), and `ordered-answers` (the provider answers in arrival + order). Writes that settle at one step settle in an order no law fixes, + so the check then compares each row's own value sequence. Reference + campaigns calibrate each: Electric-like and Convex-like providers, and a + provider that breaks each declaration fails (MUT-7, MUT-8, MUT-13). + - **Mutant:** a txid evidence check that accepts any txid fails MUT-1. diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index fab4b73052..ee252c7bd6 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -209,6 +209,7 @@ const staticOracleProperties = [ `adapter-conformance.mutations.rxdb`, `adapter-conformance.mutations.trailbase`, `adapter-conformance.mutations.powersync`, + `adapter-conformance.mutations.electric`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index e6ae675760..97af9f5a43 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -807,6 +807,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.mutations`, `powersync`, ], + [ + `electric-db-collection/tests/mutation-conformance-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `electric`, + ], ] const owners = new Map() @@ -908,7 +913,7 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), - ...[`reference`, `rxdb`, `trailbase`, `powersync`].map( + ...[`reference`, `rxdb`, `trailbase`, `powersync`, `electric`].map( (driver) => [ `adapter-conformance.mutations.${driver}`, diff --git a/packages/electric-db-collection/tests/mutation-conformance-driver.ts b/packages/electric-db-collection/tests/mutation-conformance-driver.ts new file mode 100644 index 0000000000..eb06c56bbb --- /dev/null +++ b/packages/electric-db-collection/tests/mutation-conformance-driver.ts @@ -0,0 +1,426 @@ +/** + * Production driver for the shared mutation oracle + * (`packages/db-adapter-conformance/src/mutation-oracle.ts`). It runs the real + * Electric adapter in eager mode over the installed `@electric-sql/client` + * `ShapeStream`, which fetches through a controlled `fetchClient` that + * answers the Electric HTTP protocol, as `conformance-driver.ts` does. + * + * Electric does not persist writes: the application's handlers do, through + * its own backend API, and confirm each through the shape stream with the + * adapter's `awaitTxId`. The driver supplies such an application: + * + * - the provider is the backend and its Postgres table. Its API takes a + * write request, answers by plan, and on acceptance commits it in a new + * transaction and answers with that txid; + * - each handler sends the complete row as a SQL row (an absent `note` is + * `NULL`, `undefinedField: 'null'`), then awaits `awaitTxId(txid)`. The + * adapter does not order the handlers (`application-handlers`), so each + * write's request goes out when it is made; + * - each commit reaches the shape log as one batch: its change messages, + * carrying its txid, and `up-to-date`. An update carries the primary key + * and the changed columns, as in Electric's default replica mode. The log + * is in commit order (`log-ordered-echoes`); a held echo stalls it at that + * commit until `deliver` (`held-echoes`); + * - a rewrite (`server-rewrites`) is the backend storing a changed row, as a + * trigger or column default does; its message carries the stored row; + * - a commit by another client is a backend transaction that is not the + * application's. + * + * The application inserts an absent `note` as `null` and removes a held one + * by assigning `undefined`, the update API's removed field. Limits: one sync + * run; no reconnect or must-refetch while a write waits; `awaitMatch` is not + * exercised. + */ +import { createCollection } from '@tanstack/db' +import { electricCollectionOptions } from '../src/electric' +import { + fromSqlRow, + observedView, + rewriteRow, + toSqlRow, +} from '../../db-adapter-conformance/src' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../../db-adapter-conformance/src' + +const SCHEMA = JSON.stringify({ + id: { type: `int4`, pk_index: 0 }, + title: { type: `text` }, + note: { type: `text` }, +}) + +type ElectricRow = { id: number; title: string | null; note: string | null } +type Wire = Record +type Batch = Array> + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + echoHeld: false, + rewrite: `none`, + error: new Error(`unplanned write`), +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) +const clientTimeoutMs = 2_000 +// How long nothing must change before a cut with a held request or echo. +const quietMs = 25 + +/** A row as Electric sends it: every column as text, `NULL` as null. */ +function wire(row: ConformanceRow): Wire { + const sql = toSqlRow(row) + return { id: String(sql.id), title: sql.title, note: sql.note } +} + +type Request = { + url: URL + respond: (response: Response) => void +} + +class ElectricMutationProvider implements MutationProvider { + /** The backend's table. */ + private readonly table = new Map() + private readonly plans: Array = [] + private readonly held: Array<() => void> = [] + private readonly log: Array = [] + /** Committed batches the shape log has not sent, in commit order. */ + private readonly outbox: Array = [] + /** Batches the log holds at a held echo; `own` marks the holding one. */ + private readonly stalled: Array<{ batch: Batch; own: boolean }> = [] + private waiting: Request | undefined + private txid = 100 + private offset = 0 + private unsettled = 0 + /** Cleanup aborted the client's request: nothing more will arrive. */ + private closed = false + + constructor(seed: ReadonlyArray) { + for (const row of seed) this.table.set(row.id, row) + } + + // The shape stream's side. + + readonly fetchClient: typeof fetch = (input, init) => + new Promise((resolve, reject) => { + const url = new URL(String(input)) + const signal = init?.signal ?? undefined + const request: Request = { url, respond: resolve } + signal?.addEventListener( + `abort`, + () => { + if (this.waiting === request) this.waiting = undefined + this.closed = true + reject( + signal.reason ?? new DOMException(`request aborted`, `AbortError`), + ) + }, + { once: true }, + ) + if (url.searchParams.get(`offset`) === `-1`) { + this.respond(request, [ + ...[...this.table.values()].map((row) => ({ + key: String(row.id), + value: wire(row), + headers: { operation: `insert` }, + })), + ]) + return + } + this.waiting = request + this.flush() + }) + + private respond(request: Request, messages: Batch): void { + this.offset++ + request.respond( + new Response( + JSON.stringify([ + ...messages, + { + headers: { + control: `up-to-date`, + global_last_seen_lsn: String(this.offset), + }, + }, + ]), + { + headers: { + 'electric-handle': `mutations`, + 'electric-offset': `${this.offset}_0`, + 'electric-cursor': String(this.offset), + 'electric-schema': SCHEMA, + 'electric-up-to-date': `true`, + }, + }, + ), + ) + } + + /** Sends the next unsent batch to the client's waiting live request. */ + private flush(): void { + const request = this.waiting + const batch = this.outbox.shift() + if (!request || !batch) { + if (batch) this.outbox.unshift(batch) + return + } + this.waiting = undefined + this.respond(request, batch) + } + + /** Commits a change in a new transaction and queues its batch. */ + private commit(change: ProviderChange, holdEcho: boolean): number { + const txid = ++this.txid + let message: Record | undefined + if (change.type === `delete`) { + if (this.table.delete(change.id)) + message = { + key: String(change.id), + value: { id: String(change.id) }, + headers: { operation: `delete`, txids: [txid] }, + } + } else { + const held = this.table.get(change.row.id) + this.table.set(change.row.id, change.row) + const after = wire(change.row) + const before = held && wire(held) + message = { + key: String(change.row.id), + // Default replica mode: the key and the changed columns. + value: before + ? Object.fromEntries( + Object.entries(after).filter( + ([name, value]) => name === `id` || before[name] !== value, + ), + ) + : after, + headers: { operation: before ? `update` : `insert`, txids: [txid] }, + } + } + const batch = message ? [message] : [] + // The log is in commit order: a batch waits behind a held echo. + if (holdEcho || this.stalled.length > 0) + this.stalled.push({ batch, own: holdEcho }) + else this.outbox.push(batch) + this.flush() + return txid + } + + // The application's side: its backend API. + + /** One write request; answers with the commit's txid. */ + api(request: WriteRequest): Promise<{ txid: number }> { + this.log.push(JSON.parse(JSON.stringify(request)) as WriteRequest) + const plan = this.plans.shift() ?? acceptAtOnce + return new Promise((resolve, reject) => { + const answer = () => { + if (plan.outcome === `reject`) { + reject(plan.error) + return + } + const change: ProviderChange = + request.type === `delete` + ? { type: `delete`, id: request.id } + : { + type: `upsert`, + row: rewriteRow(fromSqlRow({ ...request.row }), plan.rewrite), + } + resolve({ txid: this.commit(change, plan.echoHeld) }) + } + if (plan.held) this.held.push(answer) + else answer() + }) + } + + /** Counts a write's receipt until it settles. */ + track Promise }>( + receipt: T, + ): T { + this.unsettled++ + const settled = () => this.unsettled-- + receipt.when(`settled`).then(settled, settled) + return receipt + } + + // The oracle's side. + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + this.held.shift()?.() + } + + /** The log resumes at the oldest held echo, up to the next one. */ + deliver(): void { + const first = this.stalled.shift() + if (!first) return + this.outbox.push(first.batch) + while (this.stalled[0] && !this.stalled[0].own) + this.outbox.push(this.stalled.shift()!.batch) + this.flush() + } + + /** Answers every held request and resumes the log. */ + drain(): void { + for (const answer of this.held.splice(0)) answer() + while (this.stalled.length > 0) this.deliver() + } + + foreignCommit(change: ProviderChange): void { + this.commit(change, false) + } + + rows(): ReadonlyArray { + return [...this.table.values()] + } + + requests(): ReadonlyArray { + return this.log + } + + /** + * The propagation step: every batch sent and the client asking again, + * and every write's receipt settled; while a request or an echo is held, + * until the request log, the sent batches, and the receipts are quiet. + */ + settle = async () => { + const deadline = Date.now() + clientTimeoutMs + const state = () => + `${this.log.length} ${this.offset} ${this.unsettled} ${this.outbox.length}` + let last = state() + let quietSince = Date.now() + while (Date.now() < deadline && !this.closed) { + const delivered = this.outbox.length === 0 && this.waiting !== undefined + if (delivered && this.unsettled === 0) break + const blocked = this.held.length > 0 || this.stalled.length > 0 + if (blocked && delivered && Date.now() - quietSince >= quietMs) break + await sleep(2) + const now = state() + if (now !== last) { + last = now + quietSince = Date.now() + } + } + // The client applies a batch just before it asks again. + for (let turn = 0; turn < 3; turn++) await sleep(0) + } +} + +let sequence = 0 + +export function electricMutationDriver(): MutationDriver { + return { + adapter: `@tanstack/electric-db-collection (eager)`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + `server-rewrites`, + `held-echoes`, + `log-ordered-echoes`, + `application-handlers`, + ]), + undefinedField: `null`, + create(seed): MutationFixture { + const provider = new ElectricMutationProvider(seed) + // The application's handlers: write through its API, then wait for the + // write's transaction in the shape stream. + const persist = async ( + collection: { + utils: { awaitTxId: (txid: number) => Promise } + }, + requests: Array, + ) => { + for (const request of requests) { + const { txid } = await provider.api(request) + await collection.utils.awaitTxId(txid) + } + } + const collection = createCollection( + electricCollectionOptions({ + id: `electric-mutations-${++sequence}`, + shapeOptions: { + url: `http://conformance/mutations-${sequence}`, + params: { table: `rows` }, + fetchClient: provider.fetchClient, + }, + getKey: (row) => row.id, + onInsert: ({ transaction, collection: target }) => + persist( + target, + transaction.mutations.map((mutation) => ({ + type: `insert`, + id: mutation.modified.id, + // A SQL row has every column: an absent `note` is `NULL`. + row: toSqlRow( + fromSqlRow(mutation.modified), + ) as Partial, + })), + ), + onUpdate: ({ transaction, collection: target }) => + persist( + target, + transaction.mutations.map((mutation) => ({ + type: `update`, + id: mutation.modified.id, + // The complete row: a removed field is `NULL`. + row: { + id: mutation.modified.id, + title: mutation.modified.title, + note: mutation.modified.note ?? null, + } as Partial, + })), + ), + onDelete: ({ transaction, collection: target }) => + persist( + target, + transaction.mutations.map((mutation) => ({ + type: `delete`, + id: mutation.original.id, + })), + ), + }), + ) + return { + collection: observedView(collection, { + row: fromSqlRow, + key: (key) => key as number, + }), + base: () => [...collection.base.values()].map(fromSqlRow), + writer: { + insert: (row) => { + const { id, title, note } = toSqlRow(row) + return provider.track(collection.insert({ id, title, note })) + }, + update: (row) => + provider.track( + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note ?? null + // A `NULL` note is already absent; only a held one is removed. + else if (typeof draft.note === `string`) + (draft as { note?: string | null }).note = undefined + }), + ), + delete: (id) => provider.track(collection.delete(id)), + }, + provider, + settle: provider.settle, + dispose: async () => { + // A handler waiting on a held request or echo would outlive the + // history. + provider.drain() + await collection.cleanup() + }, + } + }, + } +} diff --git a/packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts new file mode 100644 index 0000000000..0c69936188 --- /dev/null +++ b/packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -0,0 +1,77 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../../db-adapter-conformance/src' +import { electricMutationDriver } from './mutation-conformance-driver' + +/** + * # Electric on the shared mutation oracle + * + * A receiving driver for `packages/db-adapter-conformance`. The shared + * module states the laws (MUT-1, MUT-2, MUT-7, MUT-8, MUT-10, MUT-13, FID-5, + * HEALTH-5), the model, the history grammar, and the check. This file + * supplies only the Electric production driver in + * `mutation-conformance-driver.ts` and the campaigns; it states no expected + * results. + * + * Electric does not persist writes: the driver's application handlers write + * through a backend API and confirm each with the adapter's `awaitTxId`, so + * this checks what Electric owns, the confirmation through its shape stream + * (MUT-7), with the handlers' requests and the backend's rows as the + * application's. The backend can reject or hold a write, rewrite a stored + * row, and stall its shape log at a commit (`held-echoes`, + * `log-ordered-echoes`); another client can write; and the adapter does not + * order the application's handlers (`application-handlers`). + */ + +const property = `adapter-conformance.mutations.electric` +const driver = electricMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +const campaignTimeoutMs = 120_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: Electric`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) From 6dc0a3f4d5a178a4d92c7b08f7d6f65102f342cc Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 12:58:12 -0700 Subject: [PATCH 17/28] test(convex-db-collection): run the mutation oracle over the Convex client A receiving driver runs the Convex adapter on the shared mutation oracle over the real Convex JavaScript client, whose WebSocket it controls; the driver plays the server. Convex does not persist writes, so the driver supplies the application: handlers that call one Convex mutation per write with the complete row as Convex values. The server runs one client's mutations in arrival order, answers each by plan, and commits an acceptance at a new timestamp; its `Transition` carries the query's complete result after the `MutationResponse`. The client resolves a mutation only at a transition that reaches its timestamp, so the transitions are the copies, in commit order, and a held echo stalls them. - Law: MUT-7 (the wrapped handler settles only once the subscription holds the result after the mutation), MUT-1, MUT-2, FID-5; MUT-8 and MUT-10 are the application's. - Reach: every declared mutation state, at least five histories each, in the fixed (seed 2118, 300 histories) and random campaigns, about 16 s each. - GREEN: both campaigns pass with no adapter change; convex-db-collection 118 passed. - Mutant: an adapter that does not await the application's handler fails MUT-1. - Limits: an adapter that skips its wait for the source after the handler survives these histories, which keep one subscription whose result the client applies before it resolves the mutation; `tests/collection.test.ts` and the ownership oracle own that cached-listener path and reject the mutant. JavaScript client only; no reconnect while a mutation waits. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 24 +- .../tests/mutation-conformance-driver.ts | 462 ++++++++++++++++++ ...tation-conformance-oracle.property.test.ts | 76 +++ packages/db-adapter-conformance/MIGRATION.md | 36 +- packages/db/tests/oracle-config.ts | 1 + packages/db/tests/oracle-replay-manifest.ts | 7 +- 6 files changed, 590 insertions(+), 16 deletions(-) create mode 100644 packages/convex-db-collection/tests/mutation-conformance-driver.ts create mode 100644 packages/convex-db-collection/tests/mutation-conformance-oracle.property.test.ts diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 4a615a3c3a..87a4048171 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7225,9 +7225,11 @@ and a SQL provider), against RxDB in `packages/trailbase-db-collection/tests/mutation-conformance-oracle.property.test.ts` (`adapter-conformance.mutations.trailbase`), against PowerSync in `packages/powersync-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.powersync`), and against Electric in +(`adapter-conformance.mutations.powersync`), against Electric in `packages/electric-db-collection/tests/mutation-conformance-oracle.property.test.ts` -(`adapter-conformance.mutations.electric`). It checks auto-transaction +(`adapter-conformance.mutations.electric`), and against Convex in +`packages/convex-db-collection/tests/mutation-conformance-oracle.property.test.ts` +(`adapter-conformance.mutations.convex`). It checks auto-transaction writes on a ready Collection whose adapter persists them itself: MUT-1 (the public rows show the optimistic state while the mutation function runs, and no publication shows the earlier synced value when it fulfills), MUT-2 (a @@ -7384,3 +7386,21 @@ whose txid evidence check accepts any txid fails MUT-1 at the first insert. The existing `tests/documented-mutations-oracle.test.ts` owner keeps the documented guidance handlers. Limits: one sync run (no reconnect or must-refetch while a write waits); `awaitMatch` is not exercised. + +Convex (`application-handlers`, `ordered-answers`, `log-ordered-echoes`, +`held-echoes`, `undefinedField: 'absent'`, `write-rejection`, `held-writes`, +`foreign-commits`, `server-rewrites`): the driver runs the real Convex +JavaScript client over a controlled WebSocket and plays the server. The +application's handlers call one Convex mutation per write with the complete +row as Convex values; the server runs one client's mutations in arrival order, +answers each by plan (a rejection is a failed `MutationResponse`), and commits +an acceptance at a new timestamp, whose `Transition` carries the query's +complete result after the response. The client resolves a mutation only at a +transition that reaches its timestamp, so transitions are the copies; a held +echo stalls them. GREEN with no adapter change; an adapter that does not +await the application's handler fails MUT-1. Limits: an adapter that skips +its wait for the source after the handler survives both campaigns, because +the histories keep one subscription whose result the client applies before it +resolves the mutation; the cached-listener path that wait protects is owned by +`tests/collection.test.ts` and the ownership oracle, which reject that mutant. +The JavaScript client only; no reconnect while a mutation waits. diff --git a/packages/convex-db-collection/tests/mutation-conformance-driver.ts b/packages/convex-db-collection/tests/mutation-conformance-driver.ts new file mode 100644 index 0000000000..857d135dc0 --- /dev/null +++ b/packages/convex-db-collection/tests/mutation-conformance-driver.ts @@ -0,0 +1,462 @@ +/** + * Production driver for the shared mutation oracle + * (`packages/db-adapter-conformance/src/mutation-oracle.ts`). It runs the real + * Convex adapter in eager mode over the real Convex JavaScript client, whose + * WebSocket the driver controls, as `conformance-driver.ts` does: the + * client's query set, mutation queue, timestamps, and subscriptions are real, + * and the driver plays the server. + * + * Convex does not persist writes: the application's handlers call a Convex + * mutation, and the adapter wraps each handler so it settles only once its + * subscription holds the result after that mutation. The driver supplies + * such an application: + * + * - the provider is the server's table. A write request is one `Mutation` + * message; its args carry the complete row as Convex values, so a field + * whose value is `undefined` is absent (`undefinedField: 'absent'`); + * - the server runs one client's mutations in arrival order + * (`ordered-answers`) and answers each by plan. A rejection is a failed + * `MutationResponse` carrying the plan's message; an acceptance commits at + * a new timestamp and answers with it; + * - each commit reaches the query as one `Transition` to its timestamp, + * carrying the complete result. The client resolves a mutation only once a + * transition reaches its timestamp, so transitions are the copies, in + * commit order (`log-ordered-echoes`); a held echo stalls them at that + * commit until `deliver` (`held-echoes`); + * - a rewrite (`server-rewrites`) is the mutation storing a changed row; a + * commit by another client is a server commit that is not the client's; + * - the adapter cannot order the application's handlers + * (`application-handlers`), so each write's request goes out when it is + * made. + * + * Rows are Convex values: an absent `note` is a missing field. Limits: the + * JavaScript client only; no reconnect while a mutation waits. + */ +import { createCollection } from '@tanstack/db' +import { ConvexClient } from 'convex/browser' +import { makeFunctionReference } from 'convex/server' +import { convexToJson, jsonToConvex } from 'convex/values' +import { convexCollectionOptions } from '../src/index' +import { observedView, rewriteRow } from '../../db-adapter-conformance/src' +import type { JSONValue, Value } from 'convex/values' +import type { + ConformanceRow, + MutationDriver, + MutationFixture, + MutationProvider, + ProviderChange, + WritePlan, + WriteRequest, +} from '../../db-adapter-conformance/src' + +const rowsQuery = makeFunctionReference< + `query`, + Record, + Array +>(`rows:list`) + +const writeMutation = makeFunctionReference< + `mutation`, + { request: WriteRequest }, + null +>(`rows:write`) + +const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + echoHeld: false, + rewrite: `none`, + error: new Error(`unplanned write`), +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) +const clientTimeoutMs = 2_000 +// How long nothing must change before a cut with a held request or echo. +const quietMs = 25 + +function timestamp(value: number): string { + const bytes = Buffer.alloc(8) + bytes.writeBigUInt64LE(BigInt(value)) + return bytes.toString(`base64`) +} + +/** The WebSocket the Convex client opens; the driver plays the server. */ +class Socket { + static provider: ConvexMutationProvider | undefined + onopen: (() => void) | null = null + onmessage: ((event: { data: string }) => void) | null = null + onclose: ((event: { code: number; reason: string }) => void) | null = null + onerror: (() => void) | null = null + readonly sent: Array> = [] + private readonly provider: ConvexMutationProvider + constructor(_url: string) { + this.provider = Socket.provider! + this.provider.socket = this + queueMicrotask(() => this.onopen?.()) + } + send(data: string) { + const message = JSON.parse(data) as Record + this.sent.push(message) + // A server answers after the message leaves; the client registers a + // mutation as in flight only after it sends it. + queueMicrotask(() => this.provider.receive(message)) + } + close() { + queueMicrotask(() => this.onclose?.({ code: 1000, reason: `` })) + } + deliver(message: unknown) { + this.onmessage?.({ data: JSON.stringify(message) }) + } +} + +type Answer = { run: () => void; held: boolean } + +class ConvexMutationProvider implements MutationProvider { + readonly client: ConvexClient + socket: Socket | undefined + private readonly table = new Map() + private readonly plans: Array = [] + private readonly held: Array = [] + /** Requests not yet answered, in arrival order. */ + private readonly unanswered: Array = [] + private readonly log: Array = [] + /** Commits whose transitions are stalled; `own` marks a held echo. */ + private readonly stalled: Array<{ + rows: Array + ts: number + own: boolean + }> = [] + private time = 0 + /** The timestamp of the last transition sent. */ + private sentTs = 0 + /** The query-set version the client last saw, and the one it asked for. */ + private querySet = 0 + private requestedQuerySet = 0 + private queryId: number | undefined + private unsettled = 0 + private activity = 0 + + constructor(seed: ReadonlyArray) { + for (const row of seed) this.table.set(row.id, row) + Socket.provider = this + this.client = new ConvexClient(`https://mutations.convex.cloud`, { + webSocketConstructor: Socket as unknown as typeof WebSocket, + unsavedChangesWarning: false, + logger: false, + }) + } + + private result(): Array { + return [...this.table.values()].sort((a, b) => a.id - b.id) + } + + /** A transition from the last one sent to `ts`, with the query's result. */ + private transition( + ts: number, + rows?: Array, + extra: Array = [], + ): void { + this.socket!.deliver({ + type: `Transition`, + startVersion: { + querySet: this.querySet, + ts: timestamp(this.sentTs), + identity: 0, + }, + endVersion: { + querySet: this.requestedQuerySet, + ts: timestamp(ts), + identity: 0, + }, + modifications: [ + ...extra, + ...(rows && this.queryId !== undefined + ? [ + { + type: `QueryUpdated`, + queryId: this.queryId, + value: convexToJson(rows as unknown as Value), + logLines: [], + journal: null, + }, + ] + : []), + ], + }) + this.sentTs = ts + this.querySet = this.requestedQuerySet + this.activity++ + } + + /** A message from the client. */ + receive(message: Record): void { + if (message.type === `ModifyQuerySet`) { + this.requestedQuerySet = message.newVersion as number + const modifications = message.modifications as Array<{ + type: string + queryId: number + }> + for (const modification of modifications) { + if (modification.type === `Add`) this.queryId = modification.queryId + } + const added = modifications.some((m) => m.type === `Add`) + const removed = modifications + .filter((m) => m.type === `Remove`) + .map(({ queryId }) => ({ type: `QueryRemoved`, queryId })) + // The server acknowledges a query-set change in a new transition. + queueMicrotask(() => + this.transition( + ++this.time, + added && this.stalled.length === 0 ? this.result() : undefined, + removed, + ), + ) + return + } + if (message.type !== `Mutation`) return + const requestId = message.requestId as number + const { request } = jsonToConvex( + (message.args as Array)[0]!, + ) as unknown as { request: WriteRequest } + this.log.push(request) + this.activity++ + const plan = this.plans.shift() ?? acceptAtOnce + const answer: Answer = { + held: plan.held, + run: () => { + if (plan.outcome === `reject`) { + this.socket!.deliver({ + type: `MutationResponse`, + requestId, + success: false, + result: plan.error.message, + logLines: [], + }) + return + } + // A server answers a mutation before the transition that carries + // it; the client resolves the mutation at that later transition. + const { ts, publish } = this.commit( + request.type === `delete` + ? { type: `delete`, id: request.id } + : { + type: `upsert`, + row: rewriteRow(request.row as ConformanceRow, plan.rewrite), + }, + plan.echoHeld, + ) + this.socket!.deliver({ + type: `MutationResponse`, + requestId, + success: true, + result: null, + ts: timestamp(ts), + logLines: [], + }) + publish() + }, + } + this.unanswered.push(answer) + if (plan.held) this.held.push(answer) + else if (this.unanswered[0] === answer) this.answer(answer) + } + + /** Answers a request, then each one behind it that is not held. */ + private answer(answer: Answer): void { + this.unanswered.splice(this.unanswered.indexOf(answer), 1) + answer.run() + this.activity++ + while (this.unanswered[0] && !this.unanswered[0].held) + this.answer(this.unanswered[0]) + } + + /** + * Commits at a new timestamp. `publish` sends its transition, or stalls + * it behind a held echo. + */ + private commit( + change: ProviderChange, + holdEcho: boolean, + ): { ts: number; publish: () => void } { + if (change.type === `delete`) this.table.delete(change.id) + else this.table.set(change.row.id, change.row) + const ts = ++this.time + const rows = this.result() + return { + ts, + publish: () => { + if (holdEcho || this.stalled.length > 0) + this.stalled.push({ rows, ts, own: holdEcho }) + else this.transition(ts, rows) + }, + } + } + + /** Counts a write's receipt until it settles. */ + track Promise }>( + receipt: T, + ): T { + this.unsettled++ + const settled = () => { + this.unsettled-- + this.activity++ + } + receipt.when(`settled`).then(settled, settled) + return receipt + } + + // The oracle's side. + + plan(plan: WritePlan): void { + this.plans.push(plan) + } + + release(): void { + const answer = this.held.shift() + if (answer) this.answer(answer) + } + + /** Transitions resume at the oldest held echo, up to the next one. */ + deliver(): void { + const first = this.stalled.shift() + if (!first) return + this.transition(first.ts, first.rows) + while (this.stalled[0] && !this.stalled[0].own) { + const next = this.stalled.shift()! + this.transition(next.ts, next.rows) + } + } + + /** Answers every held request and resumes the transitions. */ + drain(): void { + while (this.held.length > 0) this.release() + while (this.stalled.length > 0) this.deliver() + } + + foreignCommit(change: ProviderChange): void { + this.commit(change, false).publish() + } + + rows(): ReadonlyArray { + return this.result() + } + + requests(): ReadonlyArray { + return this.log + } + + /** + * The propagation step: every write's receipt settled; while a request or + * an echo is held, until the requests, transitions, and receipts are + * quiet. The client applies a transition within a few turns. + */ + settle = async () => { + const deadline = Date.now() + clientTimeoutMs + let last = this.activity + let quietSince = Date.now() + while (Date.now() < deadline) { + if (this.unsettled === 0) break + const blocked = this.held.length > 0 || this.stalled.length > 0 + if (blocked && Date.now() - quietSince >= quietMs) break + await sleep(2) + if (this.activity !== last) { + last = this.activity + quietSince = Date.now() + } + } + for (let turn = 0; turn < 3; turn++) await sleep(0) + } + + dispose = () => this.client.close() +} + +let sequence = 0 + +export function convexMutationDriver(): MutationDriver { + return { + adapter: `@tanstack/convex-db-collection (eager, javascript client)`, + capabilities: new Set([ + `write-rejection`, + `held-writes`, + `foreign-commits`, + `server-rewrites`, + `held-echoes`, + `log-ordered-echoes`, + `application-handlers`, + `ordered-answers`, + ]), + undefinedField: `absent`, + // The client creates a failed mutation's error from the server's + // message, which ends with the injected failure's index. + matchesFailure: (reported, failure) => + reported instanceof Error && + new RegExp(`${failure.message}(?!\\d)`).test(reported.message), + create(seed): MutationFixture { + const provider = new ConvexMutationProvider(seed) + // The application's handlers: one Convex mutation per write. + const persist = async (requests: Array) => { + for (const request of requests) + await provider.client.mutation(writeMutation, { request }) + } + const collection = createCollection( + convexCollectionOptions({ + id: `convex-mutations-${++sequence}`, + client: provider.client, + query: rowsQuery, + queryArgs: {}, + getKey: (row) => row.id, + onInsert: ({ transaction }) => + persist( + transaction.mutations.map((mutation) => ({ + type: `insert`, + id: mutation.modified.id, + row: mutation.modified, + })), + ), + onUpdate: ({ transaction }) => + persist( + transaction.mutations.map((mutation) => ({ + type: `update`, + id: mutation.modified.id, + // The complete row; Convex values drop an `undefined` field. + row: mutation.modified, + })), + ), + onDelete: ({ transaction }) => + persist( + transaction.mutations.map((mutation) => ({ + type: `delete`, + id: mutation.original.id, + })), + ), + }), + ) + return { + collection: observedView(collection, { + row: (row) => row, + key: (key) => key as number, + }), + base: () => collection.base.values(), + writer: { + insert: (row) => provider.track(collection.insert(row)), + update: (row) => + provider.track( + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note + else delete draft.note + }), + ), + delete: (id) => provider.track(collection.delete(id)), + }, + provider, + settle: provider.settle, + dispose: async () => { + provider.drain() + await collection.cleanup() + await provider.dispose() + }, + } + }, + } +} diff --git a/packages/convex-db-collection/tests/mutation-conformance-oracle.property.test.ts b/packages/convex-db-collection/tests/mutation-conformance-oracle.property.test.ts new file mode 100644 index 0000000000..482f64bae2 --- /dev/null +++ b/packages/convex-db-collection/tests/mutation-conformance-oracle.property.test.ts @@ -0,0 +1,76 @@ +import { fc } from '@fast-check/vitest' +import { describe, it } from 'vitest' +import { + oraclePropertyOptions, + oracleReplayTest, + oracleRuns, +} from '../../db/tests/oracle-config' +import { declareOracle } from '../../db/tests/oracle-declaration' +import { createOracleReach } from '../../db/tests/oracle-reach' +import { + checkMutationHistory, + mutationHistoryArb, + mutationModel, + mutationProperty, + mutationReachStatesFor, +} from '../../db-adapter-conformance/src' +import { convexMutationDriver } from './mutation-conformance-driver' + +/** + * # Convex on the shared mutation oracle + * + * A receiving driver for `packages/db-adapter-conformance`. The shared + * module states the laws (MUT-1, MUT-2, MUT-7, MUT-8, MUT-10, MUT-13, FID-5, + * HEALTH-5), the model, the history grammar, and the check. This file + * supplies only the Convex production driver in + * `mutation-conformance-driver.ts` and the campaigns; it states no expected + * results. + * + * Convex does not persist writes: the driver's application handlers call a + * Convex mutation, and the adapter's wrapped handler settles only once its + * subscription holds the result after that mutation (MUT-7). The server runs + * one client's mutations in order (`ordered-answers`), can reject or hold + * one, rewrite a stored row, and stall its transitions at a commit + * (`held-echoes`, `log-ordered-echoes`); another client can write; and the + * adapter does not order the application's handlers (`application-handlers`). + */ + +const property = `adapter-conformance.mutations.convex` +const driver = convexMutationDriver() +const historyArb = mutationHistoryArb(driver) +const runs = 300 +const campaignTimeoutMs = 120_000 + +async function campaign(options: fc.Parameters<[unknown]>) { + const reach = createOracleReach(property, mutationReachStatesFor(driver), { + minimum: 5, + }) + await fc.assert(mutationProperty(driver, reach), options) + return reach +} + +describe(`mutation conformance: Convex`, () => { + oracleReplayTest(it)( + `conforms on a fixed-seed campaign`, + async () => { + const reach = await campaign({ seed: 2118, numRuns: oracleRuns(runs) }) + reach.assertReached() + }, + campaignTimeoutMs, + ) + + oracleReplayTest(it, property)( + `conforms on a random campaign or a replayed seed`, + async () => { + const reach = await campaign(oraclePropertyOptions(runs, property)) + reach.assertReached() + }, + campaignTimeoutMs, + ) +}) + +declareOracle({ + check: [checkMutationHistory, mutationProperty], + model: [mutationModel.step, mutationModel.visible], + grammar: [historyArb], +}) diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index d981b8c303..6f9612eef5 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -27,19 +27,19 @@ reviewed change. ## Adapters -| Adapter | Families that apply (section A) | Driver status | -| ----------------- | ------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | -| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Mutations: GREEN (2026-11-03), no adapter change, `application-handlers`, `log-ordered-echoes`, `held-echoes`. Driver `electric-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.electric`. Other families: not started | -| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03), no adapter change, `undefinedField: 'null'`, `field-updates`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | -| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | -| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | -| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations, on-demand, source shapes: not started | -| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | -| local-only | local and shared stores, mutations | Not started | -| localStorage | local and shared stores, mutations | Not started | -| Persisted wrapper | persistence wrapper | Not started | -| Storage hosts (8) | storage hosts | Not started | +| Adapter | Families that apply (section A) | Driver status | +| ----------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Neon Realtime | eager live source, mutations | Eager: RED (READY-4, HEALTH-1, and LIFE-10 behind them); adapter update pending | +| Electric | eager live source, mutations, on-demand, source shapes, reload and retry, persistence | Eager: GREEN (2026-10-30) after three fixes, per-batch commits, typed Postgres columns, over the installed client. Driver `electric-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.electric`. Mutations: GREEN (2026-11-03), no adapter change, `application-handlers`, `log-ordered-echoes`, `held-echoes`. Driver `electric-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.electric`. Other families: not started | +| PowerSync | eager live source, mutations, on-demand | Eager: GREEN (2026-10-30) after a fix, per-batch commits, over real local SQLite. Driver `powersync-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.powersync`. Mutations: GREEN (2026-11-03), no adapter change, `undefinedField: 'null'`, `field-updates`. Driver `powersync-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.powersync`. On-demand: not started | +| RxDB | eager live source, mutations | Eager: GREEN (2026-10-29) after a fix, per-change commits, `listen-before-fetch`. Driver `rxdb-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.rxdb`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'absent'`. Driver `rxdb-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.rxdb` | +| TrailBase | eager live source, mutations, on-demand | Eager: GREEN (2026-10-29), per-change commits. Driver `trailbase-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.trailbase`; declares `duplicate-key-delivery` and `source-failure-after-ready`. Mutations: GREEN (2026-11-03) after a fix, `undefinedField: 'null'`, `field-updates`, `held-echoes`. Driver `trailbase-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.trailbase`. On-demand: not started | +| Convex | eager live source, mutations, on-demand, source shapes | Eager: GREEN (2026-11-02) after a fix, per-batch commits, complete results, over the real Convex JavaScript client and a controlled WebSocket. Driver `convex-db-collection/tests/conformance-driver.ts`, property `adapter-conformance.eager-live-source.convex`. Mutations: GREEN (2026-11-03), no adapter change, `application-handlers`, `ordered-answers`, `log-ordered-echoes`, `held-echoes`. Driver `convex-db-collection/tests/mutation-conformance-driver.ts`, property `adapter-conformance.mutations.convex`. On-demand, source shapes: not started | +| query-db | client cache and refetch, on-demand, mutations, reload and retry, persistence | Not started; not a push source, so the eager family does not apply | +| local-only | local and shared stores, mutations | Not started | +| localStorage | local and shared stores, mutations | Not started | +| Persisted wrapper | persistence wrapper | Not started | +| Storage hosts (8) | storage hosts | Not started | ## Suggested order @@ -653,3 +653,13 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the campaigns calibrate each: Electric-like and Convex-like providers, and a provider that breaks each declaration fails (MUT-7, MUT-8, MUT-13). - **Mutant:** a txid evidence check that accepts any txid fails MUT-1. +- **2026-11-03, Convex, mutations.** Both campaigns are GREEN with no + adapter change, with every declared reach state met. + - **Driver lessons:** a server answers a `Mutation` after the client + registers it, and sends the `MutationResponse` before the transition + that carries the commit: the client completes a mutation only at a later + transition, and drops a response for a request it is not tracking yet. + - **Mutants:** an adapter that does not await the handler fails MUT-1. One + that skips its wait for the source survives these histories; the + cached-listener path it protects is owned by `tests/collection.test.ts` + and the ownership oracle, which reject it. diff --git a/packages/db/tests/oracle-config.ts b/packages/db/tests/oracle-config.ts index ee252c7bd6..f871492c80 100644 --- a/packages/db/tests/oracle-config.ts +++ b/packages/db/tests/oracle-config.ts @@ -210,6 +210,7 @@ const staticOracleProperties = [ `adapter-conformance.mutations.trailbase`, `adapter-conformance.mutations.powersync`, `adapter-conformance.mutations.electric`, + `adapter-conformance.mutations.convex`, `adapter-conformance.eager-live-source.neon-realtime`, `adapter-conformance.eager-live-source.trailbase`, `adapter-conformance.eager-live-source.rxdb`, diff --git a/packages/db/tests/oracle-replay-manifest.ts b/packages/db/tests/oracle-replay-manifest.ts index 97af9f5a43..f4db1ff153 100644 --- a/packages/db/tests/oracle-replay-manifest.ts +++ b/packages/db/tests/oracle-replay-manifest.ts @@ -812,6 +812,11 @@ const ownerGroups: ReadonlyArray< `adapter-conformance.mutations`, `electric`, ], + [ + `convex-db-collection/tests/mutation-conformance-oracle.property.test.ts`, + `adapter-conformance.mutations`, + `convex`, + ], ] const owners = new Map() @@ -913,7 +918,7 @@ const replayTestNames = new Map([ `conforms on a random campaign or a replayed seed`, ] as const, ), - ...[`reference`, `rxdb`, `trailbase`, `powersync`, `electric`].map( + ...[`reference`, `rxdb`, `trailbase`, `powersync`, `electric`, `convex`].map( (driver) => [ `adapter-conformance.mutations.${driver}`, From 29681ad585e9c649c01f76485e6e56083e4c11be Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 16:28:01 -0700 Subject: [PATCH 18/28] test(db-adapter-conformance): generate same-key writes while one is pending Review found that the grammar never let a write follow a pending one on its key, so a TrailBase write confirmed by an earlier write's late event on that key was unreachable (R1, R2). The grammar now allows it, with a `same-key-pending` reach state and an arm that writes one key twice in a row, behind writes whose plans accept them; for an adapter that does not order writes, only behind answered ones. After a rejected insert or delete, the next write on the key depends on the law set's per-adapter `unknownKeyUpdate`, which no driver declares, so those histories are not generated. Same-key histories exposed model gaps, each fixed against its authority: - every provider keeps one key's copies in commit order, so a copy waits behind an undelivered one for its key; TrailBase's change stream is in commit order, so its driver declares `log-ordered-echoes`; - writes that settle at one step may settle in any order, also on one key, so the publication check accepts each row value that some subset of them settled shows, with at most one change per settled write; - `deliver` records the public rows after each copy it confirms; - an update removes `note` only when the held row holds a note value (a held own `undefined` stays), in the model and every writer. Also from the review: `write-queued-behind-held` counts only a write an ordering adapter holds back behind a held request (R5), and a fixture's `drain` runs before cleanup, since cleanup closes the stream a later drain would use (R7). - RED: with same-key histories the TrailBase campaigns fail MUT-1 and MUT-7 on unchanged production (an update or a delete confirmed by an earlier write's event on its key). - GREEN: db-adapter-conformance 106 passed; RxDB, PowerSync, Electric, and Convex campaigns pass with same-key histories. Co-authored-by: Isaac --- .../tests/mutation-conformance-driver.ts | 5 +- .../src/mutation-oracle.ts | 270 ++++++++++++++---- .../tests/reference-mutation-adapter.ts | 33 ++- .../tests/mutation-conformance-driver.ts | 8 +- .../tests/mutation-conformance-driver.ts | 3 +- .../tests/mutation-conformance-driver.ts | 27 +- 6 files changed, 263 insertions(+), 83 deletions(-) diff --git a/packages/convex-db-collection/tests/mutation-conformance-driver.ts b/packages/convex-db-collection/tests/mutation-conformance-driver.ts index 857d135dc0..c7ac45de04 100644 --- a/packages/convex-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/convex-db-collection/tests/mutation-conformance-driver.ts @@ -444,15 +444,16 @@ export function convexMutationDriver(): MutationDriver { collection.update(row.id, (draft) => { draft.title = row.title if (Object.hasOwn(row, `note`)) draft.note = row.note - else delete draft.note + // Only a held note value is removed. + else if (typeof draft.note === `string`) delete draft.note }), ), delete: (id) => provider.track(collection.delete(id)), }, provider, settle: provider.settle, + drain: () => provider.drain(), dispose: async () => { - provider.drain() await collection.cleanup() await provider.dispose() }, diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index 92d13a3457..daa7999043 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -62,13 +62,15 @@ * next write request, which the check hands the provider before the write: it * accepts or rejects, at once or after a `release`. * - * Boundary. The oracle checks auto-transaction writes (`collection.insert`, - * `update`, `delete`) on a ready Collection whose adapter persists them - * itself. It does not check manual transactions, confirmation through sync - * (MUT-7, a later capability), schema validation, or a mutation on a key that - * already has a pending mutation. Commits from other clients arrive only - * while no mutation is pending; a commit held behind a pending mutation is - * core's MUT-5. + * Boundary. The oracle checks auto-transaction writes of one row + * (`collection.insert`, `update`, `delete`) on a ready Collection whose + * adapter persists them itself or, with `application-handlers`, confirms the + * writes the application's handlers persist. A write may follow a pending + * one on the same key. With `held-echoes` it checks confirmation through + * sync (MUT-7). It does not check manual or multi-mutation transactions, + * schema validation, or `awaitMatch`. Commits from other clients arrive only + * while no mutation is pending, so MUT-11 is not checked; a commit held + * behind a pending mutation is core's MUT-5. */ import fc from 'fast-check' import { ConformanceViolation } from './eager-live-source-oracle' @@ -140,7 +142,11 @@ export interface MutationReceipt { /** The driver's writes, through the adapter's Collection. */ export interface MutationWriter { insert: (row: ConformanceRow) => MutationReceipt - /** Replace the held row's fields with `row`'s: a complete authored row. */ + /** + * Replace the held row's fields with `row`'s: a complete authored row. A + * `row` without `note` removes a note value the held row holds; a held + * row without one is left as it is. + */ update: (row: ConformanceRow) => MutationReceipt delete: (id: number) => MutationReceipt } @@ -240,6 +246,11 @@ export interface MutationFixture { base: () => Iterable writer: MutationWriter provider: MutationProvider + /** + * Answers every held request and delivers every withheld copy. The check + * calls it before cleanup, so no write outlives the history waiting on one. + */ + drain?: () => void | Promise /** The adapter's propagation step, awaited before every comparison. */ settle?: () => Promise dispose?: () => void | Promise @@ -293,7 +304,9 @@ export interface MutationDriver { * shrinking. * * - `write`: an insert of a key the public rows lack, or an update or delete - * of a key they hold, with no other pending mutation on that key. An update + * of a key they hold, also while earlier mutations on that key are + * pending, as long as their plans accept them; an arm generates two writes + * on one key in a row. An update * that changes no field is not generated. An update without `note` of a * row that holds one removes the field. Its plan rejects only with * `write-rejection`, in one of the driver's `rejectionKinds` when it @@ -462,9 +475,29 @@ export function mutationHistoryArb( ), }) } + // Two writes on one key in a row, so a write often follows a pending one + // on its key. + const sameKeyPair = fc + .tuple(write, write) + .map(([first, second]): Array => + first.kind === `write` && second.kind === `write` + ? [first, { ...second, id: first.id }] + : [first, second], + ) return fc.record({ seed: fc.uniqueArray(rowArb, { selector: (row) => row.id, maxLength: 3 }), - steps: fc.array(fc.oneof(...steps), { minLength: 1, maxLength: 10 }), + steps: fc + .array( + fc.oneof( + ...steps.map(({ weight, arbitrary }) => ({ + weight, + arbitrary: arbitrary.map((step) => [step]), + })), + { weight: 2, arbitrary: sameKeyPair }, + ), + { minLength: 1, maxLength: 10 }, + ) + .map((groups) => groups.flat()), }) } @@ -603,6 +636,8 @@ interface MutationModel { pending: Array /** Answered writes whose copies are not delivered, in commit order. */ echoes: Array + /** Every write, by index. */ + writes: Array rewritten: number requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> settlements: Array @@ -624,6 +659,7 @@ function initialModel( heldRows: [], pending: [], echoes: [], + writes: [], rewritten: 0, requests: [], settlements: [], @@ -640,6 +676,8 @@ function authoredRow( step: Extract, held: string | undefined, ): ConformanceRow { + // A held `note` stays an own `undefined`: removed when it held a value, + // kept when it already was one. const removesNote = step.type === `update` && step.note === undefined && @@ -746,8 +784,13 @@ function decide(state: MutationModel, write: PendingWrite): void { state.provider.set(write.id, copy) if (write.plan.rewrite !== `none` && copy !== write.value) state.rewritten++ } - // A copy is withheld by its own plan or, from a log, by an earlier one. - if (write.plan.echoHeld || (state.logOrdered && state.echoes.length > 0)) { + // A copy is withheld by its own plan or waits behind an undelivered one: + // any earlier one from a log, an earlier one for its key otherwise, since + // every provider keeps one key's copies in commit order. + if ( + write.plan.echoHeld || + state.echoes.some((earlier) => blocks(state, earlier, write)) + ) { write.answered = true write.withheld = write.plan.echoHeld state.echoes.push(write) @@ -758,10 +801,31 @@ function decide(state: MutationModel, write: PendingWrite): void { * The oldest withheld copy arrives, and from a log every later copy queued * behind it, up to the next one its own plan withheld. */ -function deliverEchoes(state: MutationModel): void { +function deliverEchoes(state: MutationModel): Array { confirm(state, state.echoes.shift()!) - while (state.logOrdered && state.echoes[0] && !state.echoes[0].withheld) - confirm(state, state.echoes.shift()!) + const states = [visible(state)] + for (;;) { + const next = state.echoes.find( + (write, index) => + !write.withheld && + state.echoes + .slice(0, index) + .every((earlier) => !blocks(state, earlier, write)), + ) + if (!next) return states + state.echoes.splice(state.echoes.indexOf(next), 1) + confirm(state, next) + states.push(visible(state)) + } +} + +/** Whether an undelivered copy holds back a later write's copy. */ +function blocks( + state: MutationModel, + earlier: PendingWrite, + later: PendingWrite, +): boolean { + return state.logOrdered || earlier.id === later.id } /** An accepted write's copy arrives: it fulfills. */ @@ -835,6 +899,8 @@ type StepDone = outcome: WritePlan[`outcome`] held: boolean queued: boolean + /** An earlier mutation on the same key was still pending. */ + sameKeyPending: boolean /** An update that removes `note`, authored as an own `undefined`. */ removesField: boolean } @@ -855,7 +921,24 @@ function stepModel( switch (step.kind) { case `write`: { const held = new Map(visible(state)) - const busy = state.pending.some((write) => write.id === step.id) + // A write may follow a pending one on the same key: core layers their + // optimistic states in author order. + const sameKey = state.pending.filter((write) => write.id === step.id) + const sameKeyPending = sameKey.length > 0 + // Only behind writes the provider will accept: after a rejected insert + // or delete, what a provider does with the next write on that key is + // the law set's per-adapter `unknownKeyUpdate`, which no driver here + // declares. + if (sameKey.some((write) => write.plan.outcome === `reject`)) + return undefined + // Requests that go out at once can reach the provider out of author + // order, so a write follows one on its key only once that is answered. + if ( + !state.ordered && + !state.answersInOrder && + sameKey.some((write) => !write.answered) + ) + return undefined const legal = step.type === `insert` ? !held.has(step.id) : held.has(step.id) const over = held.get(step.id) @@ -864,8 +947,12 @@ function stepModel( // An update that changes no field records no mutation in core; no law // fixes its persistence, so the grammar leaves it out. const noop = step.type === `update` && over === value - if (busy || !legal || noop) return undefined - const queued = state.pending.length > 0 + if (!legal || noop) return undefined + // MUT-10 holds this write back: the adapter orders writes and an earlier + // one waits for its held answer. + const queued = + state.ordered && + state.pending.some((write) => !write.answered && write.plan.held) const fields = row && step.type === `update` && state.fieldUpdates ? changedFields(row, rowOf(step.id, over!)) @@ -885,6 +972,7 @@ function stepModel( } state.settlements.push(`pending`) state.pending.push(write) + state.writes.push(write) // The optimistic state publishes when the write call returns. const published = [visible(state), ...advance(state)] return { @@ -894,10 +982,14 @@ function stepModel( outcome: step.outcome, held: step.held, queued, + sameKeyPending, + // Only a held note value is removed. removesField: - row !== undefined && - Object.hasOwn(row, `note`) && - row.note === undefined, + step.type === `update` && + step.note === undefined && + over !== undefined && + typeof (JSON.parse(over) as [string, Array])[1][0] === + `string`, }, published, } @@ -917,8 +1009,7 @@ function stepModel( } case `deliver`: { if (state.echoes.length === 0) return undefined - deliverEchoes(state) - const published = [visible(state), ...advance(state)] + const published = [...deliverEchoes(state), ...advance(state)] return { done: { kind: `deliver` }, published } } case `foreign`: { @@ -932,26 +1023,63 @@ function stepModel( } } +/** The model state a step starts from, for the settlement check. */ +interface StepStart { + readonly pending: ReadonlyArray + readonly heldRows: ReadonlyArray + readonly applied: ReadonlyMap + readonly writes: number +} + +function stepStart(state: MutationModel): StepStart { + return { + pending: [...state.pending], + heldRows: [...state.heldRows], + applied: new Map(state.applied), + writes: state.writes.length, + } +} + /** - * Each row's sequence of values from `start` through `states`, a value that - * repeats counted once; an absent row is `null`. + * The writes a step settled, and each row's values while they settle in any + * order: the public rows with any subset of them settled. Settlements at one + * step are in no order a law fixes, so a publication may show any of these. */ -function rowPaths(start: RowState, states: ReadonlyArray): string { - const valueOf = (rows: RowState, id: number) => - rows.find(([key]) => key === id)?.[1] ?? null - const ids = [ - ...new Set([start, ...states].flatMap((rows) => rows.map(([id]) => id))), - ].sort((a, b) => a - b) - return JSON.stringify( - ids.map((id) => { - const path = [valueOf(start, id)] - for (const rows of states) { - const value = valueOf(rows, id) - if (value !== path.at(-1)) path.push(value) - } - return [id, path] - }), +function settlementValues( + state: MutationModel, + start: StepStart, +): { settled: number; values: Map> } { + const pending = [...start.pending, ...state.writes.slice(start.writes)] + const settled = pending.filter( + (write) => state.settlements[write.index] !== `pending`, ) + const values = new Map>() + for (let mask = 0; mask < 1 << settled.length; mask++) { + const done = new Set(settled.filter((_, bit) => mask & (1 << bit))) + const still = pending.filter((write) => !done.has(write)) + // Held rows and the applied rows change only when nothing is pending. + const rows = + still.length === 0 + ? state.applied + : fold( + fold(start.applied, [ + ...start.heldRows, + ...pending.filter( + (write) => + done.has(write) && + state.settlements[write.index] === `fulfilled`, + ), + ]), + still, + ) + const ids = new Set([...rows.keys(), ...start.applied.keys()]) + for (const id of [...ids, ...pending.map((write) => write.id)]) { + const set = values.get(id) ?? new Set() + set.add(rows.get(id) ?? null) + values.set(id, set) + } + } + return { settled: settled.length, values } } /** @@ -987,6 +1115,7 @@ export const mutationReachStates = [ `held-write-rejected`, `foreign-commit`, `field-removed`, + `same-key-pending`, `held-optimistic-row`, `server-rewrote-copy`, `echo-held`, @@ -1009,7 +1138,7 @@ const capabilityOfState: Partial< 'held-optimistic-row': `held-writes`, 'server-rewrote-copy': `server-rewrites`, 'echo-held': `held-echoes`, - 'echo-queued-behind-withheld': `log-ordered-echoes`, + 'echo-queued-behind-withheld': `held-echoes`, } /** The declared states a driver's capabilities let histories reach. */ @@ -1029,6 +1158,18 @@ export function mutationReachStatesFor( domain.capabilities.has(`held-writes`) && domain.capabilities.has(`write-rejection`) ) + // Only an adapter that orders writes holds one back (MUT-10). + if (state === `write-queued-behind-held`) + return ( + domain.capabilities.has(`held-writes`) && + !domain.capabilities.has(`application-handlers`) + ) + // A write stays pending past its step only when held or confirmed late. + if (state === `same-key-pending`) + return ( + domain.capabilities.has(`held-writes`) || + domain.capabilities.has(`held-echoes`) + ) return capability === undefined || domain.capabilities.has(capability) }), ...kinds, @@ -1255,7 +1396,7 @@ export async function checkMutationHistory( stepIndex: number, owed: ReadonlyArray, law: string, - settledSeveral: boolean, + from: StepStart, ) => { // Each change of row values, in order; a publication of the same row // values (a virtual row field change) is not one. @@ -1265,18 +1406,33 @@ export async function checkMutationHistory( if (!sameRows(rows, got.at(-1) ?? published)) got.push(rows) } published = got.at(-1) ?? published - // Writes that settle at one step settle in an order no law fixes, so - // then each row's own sequence of values is compared: it does not depend - // on that order, and it still shows an earlier synced value or an extra - // change. - const ok = settledSeveral - ? rowPaths(start, got) === rowPaths(start, owed) - : got.length === owed.length && - got.every((rows, index) => sameRows(rows, owed[index]!)) + // Writes that settle at one step settle in an order no law fixes. Then + // each row may show only a value it has with some of them settled, and + // changes at most once per settled write and once for the step's own + // optimistic state; an earlier synced value or an extra change still + // shows. + const { settled, values } = settlementValues(state, from) + const ok = + settled > 1 + ? [...values].every(([id, allowed]) => { + const path = [start, ...got].map( + (rows) => rows.find(([key]) => key === id)?.[1] ?? null, + ) + const changes = path.filter( + (value, index) => index > 0 && value !== path[index - 1], + ).length + return ( + path.every((value) => allowed.has(value)) && + changes <= settled + 1 + ) + }) && + [...got.flatMap((rows) => rows.map(([id]) => id))].every((id) => + values.has(id), + ) + : got.length === owed.length && + got.every((rows, index) => sameRows(rows, owed[index]!)) if (!ok) throw violation(`${law} (publication)`, stepIndex, owed, got) } - const settledCount = () => - state.settlements.filter((settlement) => settlement !== `pending`).length let rewritten = 0 // The plan the model decided the last write by; the provider gets it too. @@ -1305,7 +1461,7 @@ export async function checkMutationHistory( await compare(-1) for (const [index, step] of steps.entries()) { const before = visible(state) - const settledBefore = settledCount() + const from = stepStart(state) const result = stepModel(state, step, planFor) if (result === undefined) { skipped++ @@ -1340,6 +1496,7 @@ export async function checkMutationHistory( } else if (!write.held) reach(`${write.type}-accepted` as const) if (done.removesField && write.outcome === `accept`) reach(`field-removed`) + if (done.sameKeyPending) reach(`same-key-pending`) if (write.outcome === `reject` && !write.held) law = `MUT-2` break } @@ -1371,14 +1528,14 @@ export async function checkMutationHistory( index, owedPublications(before, result.published), law, - settledCount() - settledBefore > 1, + from, ) } // Answer every held request and deliver every withheld copy, reaching a // quiescent cut (FID-5). while (state.pending.length > 0) { const before = visible(state) - const settledBefore = settledCount() + const from = stepStart(state) const step: MutationStep = undecided(state) ? { kind: `release` } : { kind: `deliver` } @@ -1397,7 +1554,7 @@ export async function checkMutationHistory( steps.length, owedPublications(before, result.published), `MUT-1`, - settledCount() - settledBefore > 1, + from, ) } } catch (error) { @@ -1407,6 +1564,7 @@ export async function checkMutationHistory( const cleanupErrors: Array = [] try { publications?.stop() + await fixture.drain?.() await collection.cleanup() await settle(fixture) if (primary === undefined && detached.found().length > 0) { diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index c4a7067bc3..158aa3b3e1 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -127,7 +127,11 @@ export class ControlledMutationProvider implements MutationProvider { private readonly plans: Array = [] private readonly held: Array<() => void> = [] /** Copies not yet fed, in commit order; `own` marks a plan's withholding. */ - private readonly withheld: Array<{ feed: () => void; own: boolean }> = [] + private readonly withheld: Array<{ + feed: () => void + own: boolean + id: number + }> = [] private readonly log: Array = [] private listener: ((change: ProviderChange, sequence: number) => void) | undefined @@ -241,9 +245,14 @@ export class ControlledMutationProvider implements MutationProvider { ? change : { ...change, row: this.stored(change.row) } const feed = () => this.listener?.(fed, sequence) - // A log feeds in commit order, so a copy waits behind a withheld one. - if (withhold || (this.options.log && this.withheld.length > 0)) - this.withheld.push({ feed, own: withhold }) + // A copy waits behind a withheld one: any, from a log, or one for its + // key, since a provider keeps one key's copies in commit order. + const id = change.type === `delete` ? change.id : change.row.id + if ( + withhold || + this.withheld.some((entry) => this.options.log || entry.id === id) + ) + this.withheld.push({ feed, own: withhold, id }) else feed() return sequence } @@ -260,8 +269,17 @@ export class ControlledMutationProvider implements MutationProvider { deliver(): void { this.withheld.shift()?.feed() - while (this.options.log && this.withheld[0] && !this.withheld[0].own) - this.withheld.shift()!.feed() + for (;;) { + const index = this.withheld.findIndex( + (entry, at) => + !entry.own && + this.withheld + .slice(0, at) + .every((earlier) => !this.options.log && earlier.id !== entry.id), + ) + if (index < 0) return + this.withheld.splice(index, 1)[0]!.feed() + } } foreignCommit(change: ProviderChange): void { @@ -521,7 +539,8 @@ export function referenceMutationDriver( collection.update(row.id, (draft) => { draft.title = row.title if (Object.hasOwn(row, `note`)) draft.note = row.note - else delete draft.note + // Only a held note value is removed. + else if (typeof draft.note === `string`) delete draft.note }), delete: (id) => collection.delete(id), }, diff --git a/packages/electric-db-collection/tests/mutation-conformance-driver.ts b/packages/electric-db-collection/tests/mutation-conformance-driver.ts index eb06c56bbb..1b791aec1c 100644 --- a/packages/electric-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/electric-db-collection/tests/mutation-conformance-driver.ts @@ -414,12 +414,8 @@ export function electricMutationDriver(): MutationDriver { }, provider, settle: provider.settle, - dispose: async () => { - // A handler waiting on a held request or echo would outlive the - // history. - provider.drain() - await collection.cleanup() - }, + drain: () => provider.drain(), + dispose: () => collection.cleanup(), } }, } diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index 9d30731942..49cec95478 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -381,7 +381,8 @@ export function rxdbMutationDriver(): MutationDriver { collection.update(String(row.id), (draft) => { draft.title = row.title if (Object.hasOwn(row, `note`)) draft.note = row.note - else delete draft.note + // Only a held note value is removed. + else if (typeof draft.note === `string`) delete draft.note }), delete: (id) => collection.delete(String(id)), }, diff --git a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts index d6729d685f..5c2855239b 100644 --- a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts @@ -21,8 +21,10 @@ * plan's error, and a held request waits until `release`. An accepted * write commits and sends its change event at once, or, with a held echo, * only at `deliver`, as an event can trail the HTTP response - * (`held-echoes`); TrailBase confirms each write through its change - * stream (the law set's `confirm-through-sync`); + * (`held-echoes`). The change stream is in commit order, so a held event + * stalls it and later events wait behind it (`log-ordered-echoes`); + * TrailBase confirms each write through its change stream (the law set's + * `confirm-through-sync`); * - a rewrite (`server-rewrites`) is the service changing the stored row, as * a column default, trigger, or check does; its event carries the stored * row; @@ -85,7 +87,8 @@ class TrailBaseMutationProvider implements MutationProvider { private readonly table = new Map() private readonly plans: Array = [] private readonly held: Array<() => void> = [] - private readonly withheld: Array<() => void> = [] + /** Events the stream holds at a held echo; `own` marks the holding one. */ + private readonly withheld: Array<{ send: () => void; own: boolean }> = [] private readonly log: Array = [] private controller: ReadableStreamDefaultController | undefined @@ -174,7 +177,9 @@ class TrailBaseMutationProvider implements MutationProvider { event = held ? { Update: asSent(record) } : { Insert: asSent(record) } } const send = () => this.controller?.enqueue(event) - if (withhold) this.withheld.push(send) + // The change stream is in commit order: an event waits behind a held one. + if (withhold || this.withheld.length > 0) + this.withheld.push({ send, own: withhold }) else send() } @@ -189,11 +194,13 @@ class TrailBaseMutationProvider implements MutationProvider { /** Answers every held request and sends every withheld event. */ drain(): void { for (const answer of this.held.splice(0)) answer() - for (const send of this.withheld.splice(0)) send() + for (const { send } of this.withheld.splice(0)) send() } deliver(): void { - this.withheld.shift()?.() + this.withheld.shift()?.send() + while (this.withheld[0] && !this.withheld[0].own) + this.withheld.shift()!.send() } foreignCommit(change: ProviderChange): void { @@ -219,6 +226,7 @@ export function trailBaseMutationDriver(): MutationDriver { `server-rewrites`, `field-updates`, `held-echoes`, + `log-ordered-echoes`, ]), undefinedField: `null`, create(seed): MutationFixture { @@ -250,11 +258,8 @@ export function trailBaseMutationDriver(): MutationDriver { delete: (id) => collection.delete(id), }, provider, - dispose: async () => { - // A write waiting for its event would otherwise outlive the history. - provider.drain() - await collection.cleanup() - }, + drain: () => provider.drain(), + dispose: () => collection.cleanup(), } }, } From 91c55a6f24e66a9c0d80c385c62f514c4d4699c8 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 16:28:07 -0700 Subject: [PATCH 19/28] fix(trailbase-db-collection): confirm each write by its own change event A write waited for any change event on its key after it began. Since a write now starts once the earlier one is answered, the earlier write's event can still be in flight, so it confirmed the later write and the later write's optimistic row dropped, showing the earlier value, before its own event arrived. Each answered write on a key produces one event there, in order, so the adapter now numbers this client's answered writes per key and confirms the n-th by the n-th event on that key. - Law: MUT-1 and MUT-7 (law set v18). - RED: with same-key histories, both TrailBase mutation campaigns failed MUT-1 (an update's optimistic row dropped at the earlier insert's event) and MUT-7 (a delete settled at the earlier insert's event). - GREEN: both campaigns pass; trailbase-db-collection 153 passed. - Mutant: the parent, confirming by any event after the write began, is the hostile mutant. - Limits: an event from another client on the key while a write is pending would still count (MUT-11, outside these histories). - Size: +19 gzip bytes over the parent for the per-key numbering; trailbase-db-collection is +11 against main. Coverage map: corrects prose that said MUT-7 was unchecked and that TrailBase waited for settlement, and records the same-key rules. Co-authored-by: Isaac --- .../trailbase-ordered-confirmed-writes.md | 2 +- docs/contributing/oracle-coverage.md | 37 ++++++++++------ .../trailbase-db-collection/src/trailbase.ts | 42 +++++++++++-------- 3 files changed, 50 insertions(+), 31 deletions(-) diff --git a/.changeset/trailbase-ordered-confirmed-writes.md b/.changeset/trailbase-ordered-confirmed-writes.md index c0f2d0af44..d0da3778ac 100644 --- a/.changeset/trailbase-ordered-confirmed-writes.md +++ b/.changeset/trailbase-ordered-confirmed-writes.md @@ -6,4 +6,4 @@ Fix three write bugs. - Removing a field in an update (for example `draft.note = undefined`) now sends `null`, so TrailBase clears the column. Before, the update omitted the field, and TrailBase kept the old value. - Writes now reach TrailBase in the order the application made them. A write starts only after TrailBase has answered the earlier write on the same collection. It does not wait for the earlier write's change event. -- A write is now confirmed only by a change event that arrives after the write began. Before, an earlier event for the same record could confirm a later write, so its optimistic value dropped and the old value showed until the new event arrived. This change also removes the adapter's periodic cleanup timer. +- A write is now confirmed only by its own change event. Before, an earlier event for the same record could confirm a later write, so its optimistic value dropped and the earlier value showed until the later event arrived. This change also removes the adapter's periodic cleanup timer. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 87a4048171..7e1885a4f7 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7325,14 +7325,20 @@ decision 2026-11-03, keeping the documented rule in `packages/db/tests/proxy-detachment-contract-oracle.test.ts`); the core owner `optimistic-history-oracle` now generates that removal (an optional `d`, set and then removed with `delete draft.d`), compares rows by own fields, and -counts its reach (`removedFieldEdits`). Limits: one pending mutation per -key; commits by another client only while no mutation is pending (a commit -held behind a pending mutation is core's MUT-5); no manual transactions, -confirmation through sync (MUT-7), schema validation, or an update that -changes no field; multi-mutation transactions, so MUT-8's mutation order -within one transaction; a newer backend value in a disjoint field (commits -by another client arrive only while nothing is pending); server-generated -keys. +counts its reach (`removedFieldEdits`). A write may follow pending ones on +its key (reach `same-key-pending`, and a grammar arm that writes one key twice +in a row) when their plans accept them; for an adapter that does not order +writes, only once they are answered. After a rejected insert or delete, what a +provider does with the next write on that key is the law set's per-adapter +`unknownKeyUpdate` (MUT-10), which no driver declares, so those histories are +not generated. Every provider keeps one key's copies in commit order, so a +copy waits behind an undelivered one for its key. An update removes `note` +only when the held row holds a note value, in the model and every writer. +Limits: commits by another client only while no mutation is pending, so MUT-11 +is not checked (a commit held behind a pending mutation is core's MUT-5); no +manual or multi-mutation transactions, so MUT-8's mutation order within one +transaction is not checked; no schema validation or update that changes no +field; a newer backend value in a disjoint field; server-generated keys. TrailBase (`undefinedField: 'null'`, `field-updates`, `held-echoes`, and every other mutation capability): the driver's RecordApi acts as a TrailBase @@ -7345,12 +7351,17 @@ failed MUT-8 (a removed field was sent as an empty patch, so the service kept it), 27 failed MUT-10 (writes did not wait for the earlier one), and 5 failed MUT-1 (a write was confirmed by an earlier event on the same key, so its optimistic row dropped before its own event arrived). The adapter now sends -a removed field as `null`, runs each write after the earlier one settles, -and confirms a write only by an event after it began; a mutant that keeps the -old confirmation fails MUT-1. A write starts after the earlier one's HTTP +a removed field as `null` and starts each write after the earlier one's HTTP response, not after its change event; the version that waited for the event -fails the fixed campaign (a request owed after the answer is missing). The record API is a test double that bypasses -the SDK's fetch and SSE decoding. +fails the fixed campaign (a request owed after the answer is missing). Its +change stream is in commit order (`log-ordered-echoes`). The adapter confirms +the n-th answered write on a key by the n-th change event on that key, so an +earlier write's late event cannot confirm a later write on the key; with same-key +histories, the version that confirmed by any event after the write began +failed MUT-1 and MUT-7, and a mutant keeping it is rejected. Events from +another client while a write is pending are outside these histories (MUT-11). +The record API is a test double that bypasses the SDK's fetch and SSE +decoding. PowerSync (`undefinedField: 'null'`, `field-updates`, `write-rejection`, `foreign-commits`): the driver runs a real local PowerSync database, which is diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index 280d2922cd..5655db144f 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -143,10 +143,12 @@ export function trailBaseCollectionOptions< const serialIns = (item: TItem) => convert(config.serialize, item) - // Each key's latest change event, by its sequence number. It is kept only - // while a write waits for its confirmation, which must be an event after - // the write began, not an earlier one on the same key (MUT-7). + // Change events per key, and this client's answered writes per key, + // counted while a write is admitted. Each answered write on a key produces + // one event there, in order, so a write is confirmed by the event its + // answer is numbered for, not by an earlier write's late one (MUT-7). const seenIds = new Store(new Map()) + const answeredIds = new Map() let eventSequence = 0 // Writes admitted and not yet settled. let writes = 0 @@ -156,11 +158,15 @@ export function trailBaseCollectionOptions< const awaitIds = ( ids: Array, - after: number, timeout: number = 120 * 1000, ): Promise => { + const tickets = ids.map((id) => { + const ticket = (answeredIds.get(id) ?? 0) + 1 + answeredIds.set(id, ticket) + return [id, ticket] as const + }) const completed = (value: Map) => - ids.every((id) => (value.get(id) ?? 0) > after) + tickets.every(([id, ticket]) => (value.get(id) ?? 0) >= ticket) if (completed(seenIds.state)) { return Promise.resolve() } @@ -191,18 +197,18 @@ export function trailBaseCollectionOptions< const persist = async (write: () => Promise>) => { writes++ try { - const send = async () => { - const after = eventSequence - return { ids: await write(), after } - } - const sent = (sending++ === 0 ? send() : turn.then(send)).finally(() => { - sending-- - }) + const sent = (sending++ === 0 ? write() : turn.then(write)).finally( + () => { + sending-- + }, + ) turn = sent.catch(() => {}) - const { ids, after } = await sent - await awaitIds(ids, after) + await awaitIds(await sent) } finally { - if (--writes === 0) seenIds.setState(() => new Map()) + if (--writes === 0) { + seenIds.setState(() => new Map()) + answeredIds.clear() + } } } @@ -436,10 +442,12 @@ export function trailBaseCollectionOptions< const sequence = ++eventSequence if (loadsInFlight > 0) eventSequenceByKey.set(getKey(value), sequence) - if (writes > 0) + if (writes > 0) { + const id = String(getKey(value)) seenIds.setState((curr: Map) => - new Map(curr).set(String(getKey(value)), sequence), + new Map(curr).set(id, (curr.get(id) ?? 0) + 1), ) + } } } } From 2b64a4abb5d35f43037f1a234ae1a176c6b15d75 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 16:35:02 -0700 Subject: [PATCH 20/28] fix(trailbase-db-collection): send a transaction's requests in order onUpdate and onDelete sent every request of a transaction at once with Promise.all. When one was rejected, the transaction's send failed at once, so the next write started while a sibling request was still unanswered, against MUT-10, and the requests were unordered, against MUT-8. They now go out one at a time in mutation order, and a rejection ends the transaction there. - Law: MUT-8 (mutation order within a transaction) and MUT-10 (law set v18). - RED: the new `trailbase.test.ts` witness failed on the parent: the next write's request started before row 2's request was answered. - GREEN: trailbase-db-collection 154 passed. - Limits: the mutation oracle generates no multi-mutation transactions, so this is a supplemental witness; generating them, and the law for a rejection partway through a transaction, are a recorded follow-up (maintainer decision 2026-11-03). - Size: +12 gzip bytes; trailbase-db-collection is +23 against main. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 8 +++ .../trailbase-db-collection/src/trailbase.ts | 45 ++++++++--------- .../tests/trailbase.test.ts | 49 +++++++++++++++++++ 3 files changed, 78 insertions(+), 24 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 7e1885a4f7..1d46c55193 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7360,6 +7360,14 @@ earlier write's late event cannot confirm a later write on the key; with same-ke histories, the version that confirmed by any event after the write began failed MUT-1 and MUT-7, and a mutant keeping it is rejected. Events from another client while a write is pending are outside these histories (MUT-11). +A transaction's update and delete requests go out one at a time in mutation +order, so the next write starts only after each is answered, and a rejection +ends the transaction there; `trailbase.test.ts` holds a supplemental witness +(RED on the parent: the next write started before a sibling request was +answered). Multi-mutation transactions are not generated, so this and the +law for a rejection partway through a transaction (whether later requests +still go out, and what partial acceptance leaves) remain open: follow-up for +this owner, decision 2026-11-03. The record API is a test double that bypasses the SDK's fetch and SSE decoding. diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index 5655db144f..112618062f 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -569,32 +569,29 @@ export function trailBaseCollectionOptions< ) return ids.map((id) => String(id)) }), + // A transaction's requests go out one at a time in mutation order, so the + // next write starts only after each of them is answered (MUT-8, MUT-10); + // a rejection ends the transaction there. onUpdate: (params: UpdateMutationFnParams) => - persist(() => - Promise.all( - params.transaction.mutations.map(async (tx) => { - const { type, changes, key } = tx - if (type !== `update`) { - throw new ExpectedUpdateTypeError(type) - } - await config.recordApi.update(key, serialUpd(changes)) - return String(key) - }), - ), - ), + persist(async () => { + const ids: Array = [] + for (const { type, changes, key } of params.transaction.mutations) { + if (type !== `update`) throw new ExpectedUpdateTypeError(type) + await config.recordApi.update(key, serialUpd(changes)) + ids.push(String(key)) + } + return ids + }), onDelete: (params: DeleteMutationFnParams) => - persist(() => - Promise.all( - params.transaction.mutations.map(async (tx) => { - const { type, key } = tx - if (type !== `delete`) { - throw new ExpectedDeleteTypeError(type) - } - await config.recordApi.delete(key) - return String(key) - }), - ), - ), + persist(async () => { + const ids: Array = [] + for (const { type, key } of params.transaction.mutations) { + if (type !== `delete`) throw new ExpectedDeleteTypeError(type) + await config.recordApi.delete(key) + ids.push(String(key)) + } + return ids + }), utils: { cancel: cancelEventReader, }, diff --git a/packages/trailbase-db-collection/tests/trailbase.test.ts b/packages/trailbase-db-collection/tests/trailbase.test.ts index 16ef9b7021..26283ec3c6 100644 --- a/packages/trailbase-db-collection/tests/trailbase.test.ts +++ b/packages/trailbase-db-collection/tests/trailbase.test.ts @@ -845,6 +845,55 @@ describe(`TrailBase Integration`, () => { } }) + it(`sends a transaction's updates in order and starts the next write only after they are answered`, async () => { + const recordApi = new MockRecordApi() + const rows = [1, 2, 3].map((id) => ({ id, updated: 0, data: `row-${id}` })) + recordApi.list.mockResolvedValue({ records: rows }) + const stream = new TransformStream() + recordApi.subscribe.mockResolvedValue(stream.readable) + const log: Array = [] + const second = createDeferred() + recordApi.update.mockImplementation((id) => { + log.push(`start ${id}`) + const answer = + id === 1 + ? Promise.reject(new Error(`rejected`)) + : id === 2 + ? second.promise + : new Promise(() => {}) + return answer.finally(() => log.push(`answer ${id}`)) + }) + const collection = createCollection(setUp(recordApi)) + try { + await collection.preload() + // One transaction updates rows 1 and 2; TrailBase rejects row 1. + const batch = collection.update([1, 2], (drafts) => { + for (const draft of drafts) draft.updated = 1 + }) + const batchSettled = batch.when(`settled`).catch(() => {}) + const next = collection.update(3, (draft) => { + draft.updated = 1 + }) + void next.when(`settled`).catch(() => {}) + await batchSettled + for (let turn = 0; turn < 5; turn++) await Promise.resolve() + second.resolve() + await new Promise((resolve) => setTimeout(resolve, 0)) + // Requests go out in mutation order, and the next write's request + // starts only once every request of the earlier transaction is + // answered (MUT-8, MUT-10). + const startsOf = (id: number) => log.indexOf(`start ${id}`) + expect(startsOf(1)).toBe(0) + expect(startsOf(3)).toBeGreaterThan(0) + for (const id of [1, 2]) { + if (startsOf(id) >= 0) + expect(log.indexOf(`answer ${id}`)).toBeLessThan(startsOf(3)) + } + } finally { + await collection.cleanup() + } + }) + it(`local inserts, updates and deletes`, async () => { // Prepare mock API. const recordApi = new MockRecordApi() From 1f8b749b9c5e6da935b448820ca926b15f314290 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 16:42:48 -0700 Subject: [PATCH 21/28] fix(rxdb-db-collection): reject an update whose document is gone onUpdate skipped a mutation whose document `findOne` no longer found and fulfilled, so an update whose document another client deleted while the update waited was silently lost and reported as persisted. It now rejects with core's `UpdateKeyNotFoundError`, so the optimistic change rolls back (maintainer decision 2026-11-03: reject rather than recreate the row). - Law: MUT-2 and MUT-13 (a handler's settlement reports whether it persisted); MUT-11 composes the other client's delete with it. - RED: the new `rxdb.test.ts` witness failed on the parent: the receipt resolved instead of rejecting. - GREEN: rxdb-db-collection 26 passed (the RxDB random mutation campaign can fail on a separate, pre-existing finding; see the coverage map). - Limits: commits by another client while a write is pending are not generated by the mutation oracle (MUT-11), so this is a supplemental witness. - Size: rxdb-db-collection +23 gzip bytes over the parent (the error import and throw). Co-authored-by: Isaac --- .changeset/rxdb-reject-missing-update.md | 5 +++ docs/contributing/oracle-coverage.md | 9 ++++-- packages/rxdb-db-collection/src/rxdb.ts | 11 ++++--- .../rxdb-db-collection/tests/rxdb.test.ts | 31 ++++++++++++++++++- 4 files changed, 48 insertions(+), 8 deletions(-) create mode 100644 .changeset/rxdb-reject-missing-update.md diff --git a/.changeset/rxdb-reject-missing-update.md b/.changeset/rxdb-reject-missing-update.md new file mode 100644 index 0000000000..c9e5138ffb --- /dev/null +++ b/.changeset/rxdb-reject-missing-update.md @@ -0,0 +1,5 @@ +--- +'@tanstack/rxdb-db-collection': patch +--- + +Reject an update whose document no longer exists in RxDB, for example because another tab deleted it while the update waited. The update now fails with `UpdateKeyNotFoundError` and its optimistic change rolls back. Before, the adapter skipped it and reported success, so the write was silently lost. diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index 1d46c55193..d9ba2ac0ac 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7317,9 +7317,12 @@ and every later incremental write on the collection waits forever campaigns failed HEALTH-5 at an update whose storage write throws. The adapter keeps `incrementalPatch` and its conflict retry, and its RxDB dev dependency and peer range are now 17.6.0 and `>=17.6.0` (maintainer decision -2026-11-03); both campaigns pass. A conflict (status 409) is not generated, and an update of a -document that another client deleted while the update was pending (the -adapter skips it and fulfills) is outside the grammar. +2026-11-03); both campaigns pass. A conflict (status 409) is not generated. An update of a +document that another client deleted while the update was pending now +rejects with `UpdateKeyNotFoundError` (maintainer decision 2026-11-03; it +used to fulfill without persisting); commits by another client while a write +is pending are outside the grammar (MUT-11), so `rxdb.test.ts` holds the +supplemental witness, RED on the parent. A removed field is an own `undefined` in the authored row (maintainer decision 2026-11-03, keeping the documented rule in `packages/db/tests/proxy-detachment-contract-oracle.test.ts`); the core owner diff --git a/packages/rxdb-db-collection/src/rxdb.ts b/packages/rxdb-db-collection/src/rxdb.ts index cf4bbbc584..84c16ca641 100644 --- a/packages/rxdb-db-collection/src/rxdb.ts +++ b/packages/rxdb-db-collection/src/rxdb.ts @@ -7,7 +7,10 @@ import { rxStorageWriteErrorToRxError, } from 'rxdb/plugins/core' import DebugModule from 'debug' -import { withCollectionConfigFactory } from '@tanstack/db' +import { + UpdateKeyNotFoundError, + withCollectionConfigFactory, +} from '@tanstack/db' import { stripRxdbFields } from './helper' import type { FilledMangoQuery, @@ -339,9 +342,9 @@ export function rxdbCollectionOptions( const newValue = stripRxdbFields(mutation.modified) const id = getKey(newValue) const doc = await rxCollection.findOne(id).exec() - if (!doc) { - continue - } + // Another client deleted it while the update waited: the update + // cannot persist, so it fails rather than report success. + if (!doc) throw new UpdateKeyNotFoundError(id) await doc.incrementalPatch(newValue) } }), diff --git a/packages/rxdb-db-collection/tests/rxdb.test.ts b/packages/rxdb-db-collection/tests/rxdb.test.ts index 53314f4542..6e22088c7c 100644 --- a/packages/rxdb-db-collection/tests/rxdb.test.ts +++ b/packages/rxdb-db-collection/tests/rxdb.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it, vi } from 'vitest' -import { createCollection, createTransaction } from '@tanstack/db' +import { + UpdateKeyNotFoundError, + createCollection, + createTransaction, +} from '@tanstack/db' import { addRxPlugin, createRxDatabase, @@ -743,6 +747,31 @@ describe(`RxDB Integration`, () => { }) describe(`error handling`, () => { + it(`rejects an update whose document another client deleted`, async () => { + const { collection, rxCollection, db } = await createTestState([ + { id: `1`, name: `first` }, + ]) + try { + // Another tab deleted the document before the update's handler ran. + const findOne = rxCollection.findOne.bind(rxCollection) + vi.spyOn(rxCollection, `findOne`).mockImplementationOnce((id) => { + const query = findOne(id as string) + return Object.assign(query, { + exec: () => Promise.resolve(null), + }) as unknown as ReturnType + }) + const tx = collection.update(`1`, (draft) => { + draft.name = `second` + }) + // The write is not persisted, so it must not report success. + await expect(tx.when(`settled`)).rejects.toBeInstanceOf( + UpdateKeyNotFoundError, + ) + } finally { + await db.remove() + } + }) + function expectAdditionalPropertyError(error: unknown, id: string) { expect(error).toBeInstanceOf(Error) expect(error).toHaveProperty( From a63ea9479df715346405847e06fe2aa84882d9d3 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 17:04:12 -0700 Subject: [PATCH 22/28] test(db-adapter-conformance): model merged updates for RxDB Same-key histories found that RxDB's `incrementalPatch` keeps a stored field an update does not name: a held insert stored with a server-set `note`, then an update authored over the optimistic row without one, left the note in place (random seed 371775937, MUT-8). The maintainer kept that behavior (2026-11-03), revising the law set's complete-row MUT-8 clause for RxDB. A new capability, `merged-updates`: an update's stored row, and its write as the provider stores it, is the authored row merged into the provider's row. A grammar arm stores a field the client has not seen, from a held write's rewrite, before an update that does not name it, which also checks that complete-row adapters drop it. - Law: MUT-8 as revised for RxDB; FID-5. - Reach: new state `merge-kept-field`, 7 to 13 of 300 histories per seed for RxDB; all other declared states still met. - RED: the seed failed MUT-8 before the mode. - GREEN: the seed replays clean; RxDB, TrailBase, PowerSync, Electric, Convex, and the reference campaigns pass. - Mutant: an RxDB update that replaces the row through `incrementalModify` fails MUT-8. Co-authored-by: Isaac --- docs/contributing/oracle-coverage.md | 10 +- packages/db-adapter-conformance/MIGRATION.md | 13 +++ .../src/mutation-oracle.ts | 101 ++++++++++++++++-- .../tests/mutation-conformance-driver.ts | 2 + 4 files changed, 117 insertions(+), 9 deletions(-) diff --git a/docs/contributing/oracle-coverage.md b/docs/contributing/oracle-coverage.md index d9ba2ac0ac..b2cb018784 100644 --- a/docs/contributing/oracle-coverage.md +++ b/docs/contributing/oracle-coverage.md @@ -7296,7 +7296,15 @@ without `server-rewrites` and fails FID-5 with it. Before the held-row rule, the JSON-provider campaign failed MUT-1 on a fulfilled removal behind a pending write. -RxDB (`undefinedField: 'absent'`, `server-rewrites`): the driver runs a real +RxDB (`undefinedField: 'absent'`, `server-rewrites`, `merged-updates`): an +update writes the fields its authored row names and keeps the others, as +`incrementalPatch` does (maintainer decision 2026-11-03, revising the law +set's complete-row MUT-8 clause for RxDB); a grammar arm stores a field the +client has not seen, from a held write's rewrite, before an update that does +not name it (reach `merge-kept-field`, 7 to 13 of 300 histories per seed), +and an RxDB that replaces the row through `incrementalModify` fails MUT-8. +Before the mode, seed 371775937 of the random campaign failed MUT-8 there. +The driver runs a real RxDB database on memory storage, and a rewrite is RxDB's `preInsert` and `preSave` hooks; the request keeps the document as the adapter sent it. With the hooks made no-ops, the fixed campaign fails FID-5. The provider wraps that storage's `bulkWrite`, stores diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 6f9612eef5..1c1efac9f6 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -663,3 +663,16 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the that skips its wait for the source survives these histories; the cached-listener path it protects is owned by `tests/collection.test.ts` and the ownership oracle, which reject it. +- **Decided 2026-11-03, RxDB updates merge named fields.** Same-key + histories found that RxDB's `incrementalPatch` keeps a stored field an + update does not name (seed 371775937). RxDB keeps that behavior: the + driver declares `merged-updates`, under which an update's stored row is + its authored row merged into the provider's row. This revises the law + set's complete-row MUT-8 clause for RxDB. An RxDB that replaces the row + fails MUT-8. +- **Decided 2026-11-03, RxDB update of a deleted document.** It rejects + with `UpdateKeyNotFoundError` instead of fulfilling without persisting. +- **Decided 2026-11-03, TrailBase multi-mutation transactions.** Requests + go out one at a time in mutation order and a rejection ends the + transaction; generating multi-mutation transactions, and the law for a + rejection partway through one, are a follow-up for the mutations owner. diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index daa7999043..d02f80bda1 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -131,6 +131,14 @@ export type MutationCapability = * go out at once. */ | `ordered-answers` + /** + * An update writes the fields its authored row names into the provider's + * row and keeps the others, as RxDB's `incrementalPatch` does; a removed + * field is named, so it is cleared. Its write, as the provider stores it, + * is that merge (maintainer decision 2026-11-03, revising the law set's + * complete-row MUT-8 clause for RxDB). + */ + | `merged-updates` /** The optimistic transaction a write call returns. */ export interface MutationReceipt { @@ -484,6 +492,39 @@ export function mutationHistoryArb( ? [first, { ...second, id: first.id }] : [first, second], ) + // A held write the provider stores with a field the client has not seen, + // then an update of that key that does not name it: a merged update keeps + // the field, a complete-row one drops it. + const unseenFieldPair = + capabilities.has(`server-rewrites`) && capabilities.has(`held-writes`) + ? [ + { + weight: 2, + arbitrary: fc + .tuple(write, write) + .map(([first, second]): Array => { + if (first.kind !== `write` || second.kind !== `write`) + return [first, second] + const { note: _note, ...update } = second + return [ + { + ...first, + type: first.type === `delete` ? `insert` : first.type, + outcome: `accept`, + held: true, + rewrite: `set-note`, + }, + { + ...update, + type: `update`, + id: first.id, + outcome: `accept`, + }, + ] + }), + }, + ] + : [] return fc.record({ seed: fc.uniqueArray(rowArb, { selector: (row) => row.id, maxLength: 3 }), steps: fc @@ -494,6 +535,7 @@ export function mutationHistoryArb( arbitrary: arbitrary.map((step) => [step]), })), { weight: 2, arbitrary: sameKeyPair }, + ...unseenFieldPair, ), { minLength: 1, maxLength: 10 }, ) @@ -630,6 +672,8 @@ interface MutationModel { readonly logOrdered: boolean /** The driver declares `ordered-answers`. */ readonly answersInOrder: boolean + /** The driver declares `merged-updates`. */ + readonly mergedUpdates: boolean provider: Map applied: Map heldRows: Array @@ -639,6 +683,8 @@ interface MutationModel { /** Every write, by index. */ writes: Array rewritten: number + /** Merged updates whose stored row kept a field they did not name. */ + mergeKept: number requests: Array<{ type: WriteRequest[`type`]; id: number; value?: string }> settlements: Array } @@ -654,6 +700,7 @@ function initialModel( ordered: !domain.capabilities.has(`application-handlers`), logOrdered: domain.capabilities.has(`log-ordered-echoes`), answersInOrder: domain.capabilities.has(`ordered-answers`), + mergedUpdates: domain.capabilities.has(`merged-updates`), provider, applied: new Map(provider), heldRows: [], @@ -661,6 +708,7 @@ function initialModel( echoes: [], writes: [], rewritten: 0, + mergeKept: 0, requests: [], settlements: [], } @@ -735,19 +783,35 @@ function changedFields( /** The provider's copy of an accepted insert or update. */ function copyOf(state: MutationModel, write: PendingWrite): ConformanceRow { - const sent = write.sent! - const base = - write.type === `update` && state.fieldUpdates - ? rowOf(write.id, state.provider.get(write.id)!) - : { id: write.id } - const merged = { ...base, ...sent } as ConformanceRow + return rewriteRow(storedFields(state, write), write.plan.rewrite) +} + +/** + * The row an accepted insert or update stores before the provider's own + * rewrite. An update under `field-updates` merges the fields it sends, and + * under `merged-updates` the fields its authored row names, into the + * provider's row; otherwise the sent row replaces it. + */ +function storedFields( + state: MutationModel, + write: PendingWrite, +): ConformanceRow { + const merges = + write.type === `update` && (state.fieldUpdates || state.mergedUpdates) + const base = merges + ? rowOf(write.id, state.provider.get(write.id)!) + : { id: write.id } + const named = state.mergedUpdates + ? rowOf(write.id, write.value!) + : write.sent! + const merged = { ...base, ...named } as ConformanceRow const { note, ...rest } = merged // A stored `undefined` or `NULL` is an absent field, except to a // structured-clone provider, which keeps an own `undefined`. const kept = (note ?? undefined) !== undefined || (state.undefinedField === `own` && Object.hasOwn(merged, `note`)) - return rewriteRow(kept ? merged : rest, write.plan.rewrite) + return kept ? merged : rest } function fold( @@ -783,6 +847,13 @@ function decide(state: MutationModel, write: PendingWrite): void { const copy = rowValue(copyOf(state, write)) state.provider.set(write.id, copy) if (write.plan.rewrite !== `none` && copy !== write.value) state.rewritten++ + if ( + state.mergedUpdates && + write.type === `update` && + !Object.hasOwn(rowOf(write.id, write.value!), `note`) && + Object.hasOwn(storedFields(state, write), `note`) + ) + state.mergeKept++ } // A copy is withheld by its own plan or waits behind an undelivered one: // any earlier one from a log, an earlier one for its key otherwise, since @@ -876,7 +947,17 @@ function advance(state: MutationModel): Array { state.requests.push({ type: next.type, id: next.id, - ...(next.sent === undefined ? {} : { value: requestValue(next.sent) }), + ...(next.sent === undefined + ? {} + : { + // A merged update's write is the merge, as the provider + // stores it; nothing commits on its key before it is decided. + value: requestValue( + state.mergedUpdates && next.type === `update` + ? storedFields(state, next) + : next.sent, + ), + }), }) if (!state.ordered) continue } @@ -1118,6 +1199,7 @@ export const mutationReachStates = [ `same-key-pending`, `held-optimistic-row`, `server-rewrote-copy`, + `merge-kept-field`, `echo-held`, `echo-queued-behind-withheld`, `quiescent-after-writes`, @@ -1138,6 +1220,7 @@ const capabilityOfState: Partial< 'held-optimistic-row': `held-writes`, 'server-rewrote-copy': `server-rewrites`, 'echo-held': `held-echoes`, + 'merge-kept-field': `merged-updates`, 'echo-queued-behind-withheld': `held-echoes`, } @@ -1519,6 +1602,7 @@ export async function checkMutationHistory( } if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + if (state.mergeKept > 0) reach(`merge-kept-field`) if (state.pending.some((write) => write.answered)) reach(`echo-held`) if (state.echoes.some((write) => !write.withheld)) reach(`echo-queued-behind-withheld`) @@ -1543,6 +1627,7 @@ export async function checkMutationHistory( if (result === undefined) break if (state.heldRows.length > 0) reach(`held-optimistic-row`) if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + if (state.mergeKept > 0) reach(`merge-kept-field`) if (state.pending.some((write) => write.answered)) reach(`echo-held`) if (state.echoes.some((write) => !write.withheld)) reach(`echo-queued-behind-withheld`) diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index 49cec95478..d3b9741392 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -346,6 +346,8 @@ export function rxdbMutationDriver(): MutationDriver { `held-writes`, `foreign-commits`, `server-rewrites`, + // `incrementalPatch` keeps the fields an update does not name. + `merged-updates`, ]), // A JSON document has no `undefined`: a removed field is absent. undefinedField: `absent`, From cba2ca67ab4ba3875d321c4fdb014046d14649b6 Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 17:10:46 -0700 Subject: [PATCH 23/28] docs(trailbase-db-collection): a removed field is NULL, not serialized An update that removes a field sends `null`, which clears the column, without calling that field's `serialize` conversion; a NOT NULL column rejects it (maintainer decision 2026-11-03). Documented in the collection guide and on the `serialize` option. Co-authored-by: Isaac --- docs/collections/trailbase-collection.md | 2 +- packages/trailbase-db-collection/src/trailbase.ts | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/collections/trailbase-collection.md b/docs/collections/trailbase-collection.md index ea1c5ead36..946c3ed190 100644 --- a/docs/collections/trailbase-collection.md +++ b/docs/collections/trailbase-collection.md @@ -52,7 +52,7 @@ The `trailBaseCollectionOptions` function accepts the following options: - `schema`: [Standard Schema](https://standardschema.dev) compatible schema (e.g., Zod, Effect) for client-side validation - `parse`: Object mapping field names to parsing functions that transform data coming from TrailBase -- `serialize`: Object mapping field names to serialization functions that transform data going to TrailBase +- `serialize`: Object mapping field names to serialization functions that transform data going to TrailBase. An update that removes a field (for example `draft.tags = undefined`) sends `null`, which clears the column, without calling that field's function. A `NOT NULL` column rejects such an update. - `onInsert`: Handler function called when items are inserted - `onUpdate`: Handler function called when items are updated - `onDelete`: Handler function called when items are deleted diff --git a/packages/trailbase-db-collection/src/trailbase.ts b/packages/trailbase-db-collection/src/trailbase.ts index 112618062f..01145e1be7 100644 --- a/packages/trailbase-db-collection/src/trailbase.ts +++ b/packages/trailbase-db-collection/src/trailbase.ts @@ -116,6 +116,11 @@ export interface TrailBaseCollectionConfig< syncMode?: TrailBaseSyncMode parse: Conversions + /** + * Per-field conversions for values sent to TrailBase. An update that + * removes a field sends `null`, which clears the column, without calling + * that field's conversion. + */ serialize: Conversions } From 921888d4e18cef1579417f9c4c40d42ba2ec343c Mon Sep 17 00:00:00 2001 From: Test Date: Tue, 3 Nov 2026 18:26:57 -0700 Subject: [PATCH 24/28] test(db-adapter-conformance): apply code simplification from review Behavior-preserving cleanups the prep review proposed and the maintainer accepted: - the model reads a plan's `echoHeld` instead of a copied `withheld` field (S1); - the check's reach updates live in one helper, and its write index and write count come from the receipts (S2, S3); - a `WriteStep` alias replaces four `Extract`s, and grammar arms lose a branch that never ran (S4); - reach states declare the capabilities they need in one table (S5); - `stepModel` names the public rows `rows`, not `held`, a glossary term (S6); the core optimistic-history oracle computes its removal condition once (S7); - one exported `acceptAtOnce`, an optional `release` tied to `held-writes`, the check's unsettled-receipt count passed to a fixture's `settle`, and a shared `settleWhenQuiet` helper replace the drivers' copies and their `track` wrappers (S8); a dead `?? null` and Electric's shift-then-unshift go (S9, S10). - GREEN: db-adapter-conformance 106, rxdb 26, trailbase 154, powersync 198, electric 1245, convex 118 passed. Co-authored-by: Isaac --- .../tests/mutation-conformance-driver.ts | 75 ++----- .../src/driver-support.ts | 51 +++++ packages/db-adapter-conformance/src/index.ts | 4 + .../src/mutation-oracle.ts | 211 +++++++++--------- .../tests/reference-mutation-adapter.ts | 13 +- .../db/tests/optimistic-history-oracle.ts | 17 +- .../tests/mutation-conformance-driver.ts | 86 +++---- .../tests/mutation-conformance-driver.ts | 54 ++--- .../tests/mutation-conformance-driver.ts | 8 +- .../tests/mutation-conformance-driver.ts | 8 +- 10 files changed, 232 insertions(+), 295 deletions(-) create mode 100644 packages/db-adapter-conformance/src/driver-support.ts diff --git a/packages/convex-db-collection/tests/mutation-conformance-driver.ts b/packages/convex-db-collection/tests/mutation-conformance-driver.ts index c7ac45de04..0c26366ba2 100644 --- a/packages/convex-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/convex-db-collection/tests/mutation-conformance-driver.ts @@ -37,7 +37,12 @@ import { ConvexClient } from 'convex/browser' import { makeFunctionReference } from 'convex/server' import { convexToJson, jsonToConvex } from 'convex/values' import { convexCollectionOptions } from '../src/index' -import { observedView, rewriteRow } from '../../db-adapter-conformance/src' +import { + acceptAtOnce, + observedView, + rewriteRow, + settleWhenQuiet, +} from '../../db-adapter-conformance/src' import type { JSONValue, Value } from 'convex/values' import type { ConformanceRow, @@ -45,6 +50,7 @@ import type { MutationFixture, MutationProvider, ProviderChange, + SettleCut, WritePlan, WriteRequest, } from '../../db-adapter-conformance/src' @@ -61,19 +67,6 @@ const writeMutation = makeFunctionReference< null >(`rows:write`) -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} - -const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) -const clientTimeoutMs = 2_000 -// How long nothing must change before a cut with a held request or echo. -const quietMs = 25 - function timestamp(value: number): string { const bytes = Buffer.alloc(8) bytes.writeBigUInt64LE(BigInt(value)) @@ -133,7 +126,6 @@ class ConvexMutationProvider implements MutationProvider { private querySet = 0 private requestedQuerySet = 0 private queryId: number | undefined - private unsettled = 0 private activity = 0 constructor(seed: ReadonlyArray) { @@ -292,19 +284,6 @@ class ConvexMutationProvider implements MutationProvider { } } - /** Counts a write's receipt until it settles. */ - track Promise }>( - receipt: T, - ): T { - this.unsettled++ - const settled = () => { - this.unsettled-- - this.activity++ - } - receipt.when(`settled`).then(settled, settled) - return receipt - } - // The oracle's side. plan(plan: WritePlan): void { @@ -350,22 +329,12 @@ class ConvexMutationProvider implements MutationProvider { * an echo is held, until the requests, transitions, and receipts are * quiet. The client applies a transition within a few turns. */ - settle = async () => { - const deadline = Date.now() + clientTimeoutMs - let last = this.activity - let quietSince = Date.now() - while (Date.now() < deadline) { - if (this.unsettled === 0) break - const blocked = this.held.length > 0 || this.stalled.length > 0 - if (blocked && Date.now() - quietSince >= quietMs) break - await sleep(2) - if (this.activity !== last) { - last = this.activity - quietSince = Date.now() - } - } - for (let turn = 0; turn < 3; turn++) await sleep(0) - } + settle = ({ unsettled }: SettleCut) => + settleWhenQuiet({ + done: () => unsettled() === 0, + blocked: () => this.held.length > 0 || this.stalled.length > 0, + activity: () => `${this.activity} ${unsettled()}`, + }) dispose = () => this.client.close() } @@ -438,17 +407,15 @@ export function convexMutationDriver(): MutationDriver { }), base: () => collection.base.values(), writer: { - insert: (row) => provider.track(collection.insert(row)), + insert: (row) => collection.insert(row), update: (row) => - provider.track( - collection.update(row.id, (draft) => { - draft.title = row.title - if (Object.hasOwn(row, `note`)) draft.note = row.note - // Only a held note value is removed. - else if (typeof draft.note === `string`) delete draft.note - }), - ), - delete: (id) => provider.track(collection.delete(id)), + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note + // Only a held note value is removed. + else if (typeof draft.note === `string`) delete draft.note + }), + delete: (id) => collection.delete(id), }, provider, settle: provider.settle, diff --git a/packages/db-adapter-conformance/src/driver-support.ts b/packages/db-adapter-conformance/src/driver-support.ts new file mode 100644 index 0000000000..0aba765e22 --- /dev/null +++ b/packages/db-adapter-conformance/src/driver-support.ts @@ -0,0 +1,51 @@ +/** + * Support for receiving drivers: code a driver may share, outside the + * oracle's model and check. + */ + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) + +export interface QuietSettleOptions { + /** The cut is ready: everything the adapter owes has happened. */ + done: () => boolean + /** + * Something is held, so some writes legitimately wait; the cut is ready + * once `activity` has not changed for `quietMs`. + */ + blocked: () => boolean + /** A value that changes whenever the adapter or the provider acts. */ + activity: () => unknown + /** The adapter has stopped; nothing more will happen. */ + stopped?: () => boolean + timeoutMs?: number + quietMs?: number +} + +/** + * A propagation step for a provider that can hold requests or copies: wait + * until `done`, or, while something is held, until nothing changes for a + * while, then give the client a few turns to apply what it received. + */ +export async function settleWhenQuiet({ + done, + blocked, + activity, + stopped = () => false, + timeoutMs = 2_000, + quietMs = 25, +}: QuietSettleOptions): Promise { + const deadline = Date.now() + timeoutMs + let last = activity() + let quietSince = Date.now() + while (Date.now() < deadline && !stopped()) { + if (done()) break + if (blocked() && Date.now() - quietSince >= quietMs) break + await sleep(2) + const now = activity() + if (now !== last) { + last = now + quietSince = Date.now() + } + } + for (let turn = 0; turn < 3; turn++) await sleep(0) +} diff --git a/packages/db-adapter-conformance/src/index.ts b/packages/db-adapter-conformance/src/index.ts index ef83c22e90..113e53c791 100644 --- a/packages/db-adapter-conformance/src/index.ts +++ b/packages/db-adapter-conformance/src/index.ts @@ -42,7 +42,10 @@ export { mutationReachStates, mutationReachStatesFor, rewriteRow, + acceptAtOnce, } from './mutation-oracle' +export { settleWhenQuiet } from './driver-support' +export type { QuietSettleOptions } from './driver-support' export type { MutationCapability, MutationCheckOptions, @@ -56,6 +59,7 @@ export type { MutationStep, MutationWriter, ServerRewrite, + SettleCut, WritePlan, WriteRequest, } from './mutation-oracle' diff --git a/packages/db-adapter-conformance/src/mutation-oracle.ts b/packages/db-adapter-conformance/src/mutation-oracle.ts index d02f80bda1..f8ab2b9153 100644 --- a/packages/db-adapter-conformance/src/mutation-oracle.ts +++ b/packages/db-adapter-conformance/src/mutation-oracle.ts @@ -201,6 +201,15 @@ export function rewriteRow( } /** The provider's answer to the next write request. */ +/** The plan for a request the check did not plan: accept it at once. */ +export const acceptAtOnce: WritePlan = { + outcome: `accept`, + held: false, + echoHeld: false, + rewrite: `none`, + error: new Error(`unplanned write`), +} + export interface WritePlan { readonly outcome: `accept` | `reject` /** @@ -234,8 +243,8 @@ export interface WritePlan { export interface MutationProvider { /** Queue the answer to the next write request that arrives. */ plan: (plan: WritePlan) => void - /** Answer the oldest held request by its plan. */ - release: () => void | Promise + /** Answer the oldest held request by its plan (`held-writes`). */ + release?: () => void | Promise /** Deliver the oldest withheld copy through the change feed (`held-echoes`). */ deliver?: () => void | Promise /** Commit a change made by another client (`foreign-commits`). */ @@ -247,6 +256,12 @@ export interface MutationProvider { requests: () => ReadonlyArray } +/** What the check tells a fixture's propagation step. */ +export interface SettleCut { + /** How many writes' `when('settled')` receipts have not settled yet. */ + unsettled: () => number +} + export interface MutationFixture { /** The adapter's Collection, read as conformance rows. */ collection: ObservedCollection @@ -260,7 +275,7 @@ export interface MutationFixture { */ drain?: () => void | Promise /** The adapter's propagation step, awaited before every comparison. */ - settle?: () => Promise + settle?: (cut: SettleCut) => Promise dispose?: () => void | Promise } @@ -351,6 +366,9 @@ export type MutationStep = | { readonly kind: `deliver` } | { readonly kind: `foreign`; readonly change: ProviderChange } +/** A write step. */ +export type WriteStep = Extract + export interface MutationHistory { readonly seed: ReadonlyArray readonly steps: ReadonlyArray @@ -414,7 +432,7 @@ export function mutationHistoryArb( echoHeld, rewrite, rejectionKind, - }): MutationStep => ({ + }): WriteStep => ({ kind: `write`, type, id: row.id, @@ -437,23 +455,10 @@ export function mutationHistoryArb( // An update without `note`: it removes the field when the row holds one. { weight: 2, - arbitrary: write.map((step): MutationStep => - step.kind === `write` - ? { - kind: `write`, - type: `update`, - id: step.id, - title: step.title, - outcome: step.outcome, - held: step.held, - echoHeld: step.echoHeld, - rewrite: step.rewrite, - ...(step.rejectionKind === undefined - ? {} - : { rejectionKind: step.rejectionKind }), - } - : step, - ), + arbitrary: write.map(({ note: _note, ...step }): WriteStep => ({ + ...step, + type: `update`, + })), }, ] if (capabilities.has(`held-writes`)) { @@ -487,11 +492,10 @@ export function mutationHistoryArb( // on its key. const sameKeyPair = fc .tuple(write, write) - .map(([first, second]): Array => - first.kind === `write` && second.kind === `write` - ? [first, { ...second, id: first.id }] - : [first, second], - ) + .map(([first, second]): Array => [ + first, + { ...second, id: first.id }, + ]) // A held write the provider stores with a field the client has not seen, // then an update of that key that does not name it: a merged update keeps // the field, a complete-row one drops it. @@ -503,8 +507,6 @@ export function mutationHistoryArb( arbitrary: fc .tuple(write, write) .map(([first, second]): Array => { - if (first.kind !== `write` || second.kind !== `write`) - return [first, second] const { note: _note, ...update } = second return [ { @@ -619,8 +621,6 @@ interface PendingWrite { requested: boolean /** Accepted, with its copy not yet delivered by the change feed. */ answered: boolean - /** Its own plan withheld its copy (`held-echoes`). */ - withheld: boolean } type ExpectedSettlement = `pending` | `fulfilled` | { rejected: Error } @@ -721,16 +721,16 @@ function initialModel( * maintainer decision 2026-11-03). */ function authoredRow( - step: Extract, - held: string | undefined, + step: WriteStep, + over: string | undefined, ): ConformanceRow { - // A held `note` stays an own `undefined`: removed when it held a value, - // kept when it already was one. + // The row's `note` stays an own `undefined`: removed when it held a + // value, kept when it already was one. const removesNote = step.type === `update` && step.note === undefined && - held !== undefined && - (JSON.parse(held) as [string, Array])[1].length > 0 + over !== undefined && + (JSON.parse(over) as [string, Array])[1].length > 0 return { id: step.id, title: step.title, @@ -863,7 +863,6 @@ function decide(state: MutationModel, write: PendingWrite): void { state.echoes.some((earlier) => blocks(state, earlier, write)) ) { write.answered = true - write.withheld = write.plan.echoHeld state.echoes.push(write) } else confirm(state, write) } @@ -878,7 +877,7 @@ function deliverEchoes(state: MutationModel): Array { for (;;) { const next = state.echoes.find( (write, index) => - !write.withheld && + !write.plan.echoHeld && state.echoes .slice(0, index) .every((earlier) => !blocks(state, earlier, write)), @@ -997,11 +996,12 @@ type StepDone = function stepModel( state: MutationModel, step: MutationStep, - plan: (step: Extract) => WritePlan, + plan: (step: WriteStep) => WritePlan, ): { done: StepDone; published: Array } | undefined { switch (step.kind) { case `write`: { - const held = new Map(visible(state)) + // The public rows the write is authored over. + const rows = new Map(visible(state)) // A write may follow a pending one on the same key: core layers their // optimistic states in author order. const sameKey = state.pending.filter((write) => write.id === step.id) @@ -1021,8 +1021,8 @@ function stepModel( ) return undefined const legal = - step.type === `insert` ? !held.has(step.id) : held.has(step.id) - const over = held.get(step.id) + step.type === `insert` ? !rows.has(step.id) : rows.has(step.id) + const over = rows.get(step.id) const row = step.type === `delete` ? undefined : authoredRow(step, over) const value = row && rowValue(row) // An update that changes no field records no mutation in core; no law @@ -1043,13 +1043,12 @@ function stepModel( type: step.type, id: step.id, ...(row && { - value: rowValue(row), + value, sent: sentFields(fields!, state.undefinedField, fields === row), }), plan: plan(step), requested: false, answered: false, - withheld: false, } state.settlements.push(`pending`) state.pending.push(write) @@ -1209,51 +1208,53 @@ export const mutationReachStates = [ export type MutationReachState = (typeof mutationReachStates)[number] | `rejected-as-${string}` -const capabilityOfState: Partial< - Record +/** + * The capabilities a declared state needs: every one in `all`, at least one + * in `any`, and none in `none`. A state with no entry needs none. + */ +const stateNeeds: Partial< + Record< + MutationReachState, + { + all?: ReadonlyArray + any?: ReadonlyArray + none?: ReadonlyArray + } + > > = { - 'write-rejected': `write-rejection`, - 'held-write-pending': `held-writes`, - 'write-queued-behind-held': `held-writes`, - 'held-write-rejected': `held-writes`, - 'foreign-commit': `foreign-commits`, - 'held-optimistic-row': `held-writes`, - 'server-rewrote-copy': `server-rewrites`, - 'echo-held': `held-echoes`, - 'merge-kept-field': `merged-updates`, - 'echo-queued-behind-withheld': `held-echoes`, + 'write-rejected': { all: [`write-rejection`] }, + 'held-write-pending': { all: [`held-writes`] }, + // Only an adapter that orders writes holds one back (MUT-10). + 'write-queued-behind-held': { + all: [`held-writes`], + none: [`application-handlers`], + }, + 'held-write-rejected': { all: [`held-writes`, `write-rejection`] }, + 'foreign-commit': { all: [`foreign-commits`] }, + 'held-optimistic-row': { all: [`held-writes`] }, + 'server-rewrote-copy': { all: [`server-rewrites`] }, + 'echo-held': { all: [`held-echoes`] }, + 'merge-kept-field': { all: [`merged-updates`] }, + 'echo-queued-behind-withheld': { all: [`held-echoes`] }, + // A write stays pending past its step only when held or confirmed late. + 'same-key-pending': { any: [`held-writes`, `held-echoes`] }, } /** The declared states a driver's capabilities let histories reach. */ export function mutationReachStatesFor( domain: Pick, ): Array { - const kinds = domain.capabilities.has(`write-rejection`) + const has = (capability: MutationCapability) => + domain.capabilities.has(capability) + const kinds = has(`write-rejection`) ? (domain.rejectionKinds ?? []).map( (kind): MutationReachState => `rejected-as-${kind}`, ) : [] return [ ...mutationReachStates.filter((state) => { - const capability = capabilityOfState[state] - if (state === `held-write-rejected`) - return ( - domain.capabilities.has(`held-writes`) && - domain.capabilities.has(`write-rejection`) - ) - // Only an adapter that orders writes holds one back (MUT-10). - if (state === `write-queued-behind-held`) - return ( - domain.capabilities.has(`held-writes`) && - !domain.capabilities.has(`application-handlers`) - ) - // A write stays pending past its step only when held or confirmed late. - if (state === `same-key-pending`) - return ( - domain.capabilities.has(`held-writes`) || - domain.capabilities.has(`held-echoes`) - ) - return capability === undefined || domain.capabilities.has(capability) + const { all = [], any, none = [] } = stateNeeds[state] ?? {} + return all.every(has) && (any?.some(has) ?? true) && !none.some(has) }), ...kinds, ] @@ -1282,9 +1283,9 @@ const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) * propagation step its fixture declares. A slower adapter shows as a * mismatch at this cut, not as a pass. */ -async function settle(fixture: MutationFixture): Promise { +async function settle(fixture: MutationFixture, cut: SettleCut): Promise { for (let turn = 0; turn < 3; turn++) await sleep(0) - await fixture.settle?.() + await fixture.settle?.(cut) } /** @@ -1387,10 +1388,12 @@ export async function checkMutationHistory( const { collection, provider, writer } = fixture const state = initialModel(history.seed, driver) const receipts: Array<{ current: Settlement }> = [] - const failures: Array = [] + const cut: SettleCut = { + unsettled: () => + receipts.filter((receipt) => receipt.current === `pending`).length, + } let accepted = 0 let skipped = 0 - let wrote = 0 let primary: unknown let publications: ReturnType | undefined @@ -1404,7 +1407,7 @@ export async function checkMutationHistory( /** MUT-1, MUT-2, MUT-8, MUT-10, MUT-13, FID-5, HEALTH-5 at one cut. */ const compare = async (stepIndex: number) => { - await settle(fixture) + await settle(fixture, cut) const found = detached.found() if (found.length > 0) throw violation(`HEALTH-5`, stepIndex, `no detached failure`, found) @@ -1471,7 +1474,7 @@ export async function checkMutationHistory( rowState(state.provider), providerRows, ) - if (wrote > 0) reach(`quiescent-after-writes`) + if (receipts.length > 0) reach(`quiescent-after-writes`) } let published = visible(state) @@ -1518,13 +1521,22 @@ export async function checkMutationHistory( } let rewritten = 0 + /** The declared states the model's state shows after a step. */ + const reachModelStates = () => { + if (state.heldRows.length > 0) reach(`held-optimistic-row`) + if (state.rewritten > rewritten) reach(`server-rewrote-copy`) + if (state.mergeKept > 0) reach(`merge-kept-field`) + if (state.pending.some((write) => write.answered)) reach(`echo-held`) + if (state.echoes.some((write) => !write.plan.echoHeld)) + reach(`echo-queued-behind-withheld`) + rewritten = state.rewritten + } // The plan the model decided the last write by; the provider gets it too. let lastPlan: WritePlan | undefined - const planFor = ( - step: Extract, - ): WritePlan => { - const error = new Error(`conformance write rejected ${failures.length}`) - failures.push(error) + const planFor = (step: WriteStep): WritePlan => { + // Indexed by write, so a driver's `matchesFailure` can name it; the write + // records its receipt after its plan. + const error = new Error(`conformance write rejected ${receipts.length}`) lastPlan = { outcome: step.outcome, held: step.held, @@ -1555,7 +1567,7 @@ export async function checkMutationHistory( let law = `MUT-1` switch (done.kind) { case `write`: { - const write = step as Extract + const write = step as WriteStep provider.plan(lastPlan!) const row: ConformanceRow = { id: write.id, @@ -1569,7 +1581,6 @@ export async function checkMutationHistory( ? writer.update(row) : writer.delete(write.id) receipts.push(recordSettlement(receipt.when(`settled`))) - wrote++ if (write.held) reach(`held-write-pending`) if (done.queued) reach(`write-queued-behind-held`) if (write.outcome === `reject`) { @@ -1584,7 +1595,7 @@ export async function checkMutationHistory( break } case `release`: - await provider.release() + await provider.release!() if (done.outcome === `reject`) { reach(`held-write-rejected`) law = `MUT-2` @@ -1600,13 +1611,7 @@ export async function checkMutationHistory( reach(`foreign-commit`) break } - if (state.heldRows.length > 0) reach(`held-optimistic-row`) - if (state.rewritten > rewritten) reach(`server-rewrote-copy`) - if (state.mergeKept > 0) reach(`merge-kept-field`) - if (state.pending.some((write) => write.answered)) reach(`echo-held`) - if (state.echoes.some((write) => !write.withheld)) - reach(`echo-queued-behind-withheld`) - rewritten = state.rewritten + reachModelStates() await compare(index) checkPublications( index, @@ -1625,15 +1630,9 @@ export async function checkMutationHistory( : { kind: `deliver` } const result = stepModel(state, step, planFor) if (result === undefined) break - if (state.heldRows.length > 0) reach(`held-optimistic-row`) - if (state.rewritten > rewritten) reach(`server-rewrote-copy`) - if (state.mergeKept > 0) reach(`merge-kept-field`) - if (state.pending.some((write) => write.answered)) reach(`echo-held`) - if (state.echoes.some((write) => !write.withheld)) - reach(`echo-queued-behind-withheld`) - rewritten = state.rewritten + reachModelStates() if (step.kind === `deliver`) await provider.deliver!() - else await provider.release() + else await provider.release!() await compare(steps.length) checkPublications( steps.length, @@ -1651,7 +1650,7 @@ export async function checkMutationHistory( publications?.stop() await fixture.drain?.() await collection.cleanup() - await settle(fixture) + await settle(fixture, cut) if (primary === undefined && detached.found().length > 0) { primary = violation( `HEALTH-5`, diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index 158aa3b3e1..ff202e12b9 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -30,7 +30,11 @@ * (MUT-10). */ import { createCollection } from '@tanstack/db' -import { observedView, rewriteRow } from '../src' +import { + acceptAtOnce, + observedView, + rewriteRow, +} from '../src' import type { ConformanceRow, MutationCapability, @@ -111,13 +115,6 @@ export interface ReferenceMutationOptions { inOrder?: boolean } -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} const jsonClone = (value: T): T => JSON.parse(JSON.stringify(value)) as T diff --git a/packages/db/tests/optimistic-history-oracle.ts b/packages/db/tests/optimistic-history-oracle.ts index 2eba16da1d..07b41016c7 100644 --- a/packages/db/tests/optimistic-history-oracle.ts +++ b/packages/db/tests/optimistic-history-oracle.ts @@ -1521,13 +1521,14 @@ export async function runOptimisticHistory( if (index === undefined) continue const intent = model.transactions[index]! const was = before.find((row) => row.id === step.key) - if ( + // The edit removes a `d` the row holds (the documented own + // `undefined` in `changes`). + const removesHeldD = step.type === `edit` && - step.removesD && + step.removesD === true && was !== undefined && Object.hasOwn(was, `d`) - ) - counts.removedFieldEdits++ + if (removesHeldD) counts.removedFieldEdits++ const now = model.visible().get(step.key) if ( inFlight && @@ -1638,13 +1639,7 @@ export async function runOptimisticHistory( value, ), ), - ...(step.removesD && - before.some( - (row) => - row.id === intent.key && Object.hasOwn(row, `d`), - ) - ? [[`d`, undefined]] - : []), + ...(removesHeldD ? [[`d`, undefined]] : []), ]), } operations.push(operation) diff --git a/packages/electric-db-collection/tests/mutation-conformance-driver.ts b/packages/electric-db-collection/tests/mutation-conformance-driver.ts index 1b791aec1c..9f46558bb4 100644 --- a/packages/electric-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/electric-db-collection/tests/mutation-conformance-driver.ts @@ -34,9 +34,11 @@ import { createCollection } from '@tanstack/db' import { electricCollectionOptions } from '../src/electric' import { + acceptAtOnce, fromSqlRow, observedView, rewriteRow, + settleWhenQuiet, toSqlRow, } from '../../db-adapter-conformance/src' import type { @@ -45,6 +47,7 @@ import type { MutationFixture, MutationProvider, ProviderChange, + SettleCut, WritePlan, WriteRequest, } from '../../db-adapter-conformance/src' @@ -59,19 +62,6 @@ type ElectricRow = { id: number; title: string | null; note: string | null } type Wire = Record type Batch = Array> -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} - -const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) -const clientTimeoutMs = 2_000 -// How long nothing must change before a cut with a held request or echo. -const quietMs = 25 - /** A row as Electric sends it: every column as text, `NULL` as null. */ function wire(row: ConformanceRow): Wire { const sql = toSqlRow(row) @@ -96,7 +86,6 @@ class ElectricMutationProvider implements MutationProvider { private waiting: Request | undefined private txid = 100 private offset = 0 - private unsettled = 0 /** Cleanup aborted the client's request: nothing more will arrive. */ private closed = false @@ -165,13 +154,9 @@ class ElectricMutationProvider implements MutationProvider { /** Sends the next unsent batch to the client's waiting live request. */ private flush(): void { const request = this.waiting - const batch = this.outbox.shift() - if (!request || !batch) { - if (batch) this.outbox.unshift(batch) - return - } + if (!request || this.outbox.length === 0) return this.waiting = undefined - this.respond(request, batch) + this.respond(request, this.outbox.shift()!) } /** Commits a change in a new transaction and queues its batch. */ @@ -238,16 +223,6 @@ class ElectricMutationProvider implements MutationProvider { }) } - /** Counts a write's receipt until it settles. */ - track Promise }>( - receipt: T, - ): T { - this.unsettled++ - const settled = () => this.unsettled-- - receipt.when(`settled`).then(settled, settled) - return receipt - } - // The oracle's side. plan(plan: WritePlan): void { @@ -291,26 +266,17 @@ class ElectricMutationProvider implements MutationProvider { * and every write's receipt settled; while a request or an echo is held, * until the request log, the sent batches, and the receipts are quiet. */ - settle = async () => { - const deadline = Date.now() + clientTimeoutMs - const state = () => - `${this.log.length} ${this.offset} ${this.unsettled} ${this.outbox.length}` - let last = state() - let quietSince = Date.now() - while (Date.now() < deadline && !this.closed) { - const delivered = this.outbox.length === 0 && this.waiting !== undefined - if (delivered && this.unsettled === 0) break - const blocked = this.held.length > 0 || this.stalled.length > 0 - if (blocked && delivered && Date.now() - quietSince >= quietMs) break - await sleep(2) - const now = state() - if (now !== last) { - last = now - quietSince = Date.now() - } - } - // The client applies a batch just before it asks again. - for (let turn = 0; turn < 3; turn++) await sleep(0) + settle = ({ unsettled }: SettleCut) => { + const delivered = () => + this.outbox.length === 0 && this.waiting !== undefined + return settleWhenQuiet({ + done: () => delivered() && unsettled() === 0, + blocked: () => + delivered() && (this.held.length > 0 || this.stalled.length > 0), + activity: () => + `${this.log.length} ${this.offset} ${unsettled()} ${this.outbox.length}`, + stopped: () => this.closed, + }) } } @@ -398,19 +364,17 @@ export function electricMutationDriver(): MutationDriver { writer: { insert: (row) => { const { id, title, note } = toSqlRow(row) - return provider.track(collection.insert({ id, title, note })) + return collection.insert({ id, title, note }) }, update: (row) => - provider.track( - collection.update(row.id, (draft) => { - draft.title = row.title - if (Object.hasOwn(row, `note`)) draft.note = row.note ?? null - // A `NULL` note is already absent; only a held one is removed. - else if (typeof draft.note === `string`) - (draft as { note?: string | null }).note = undefined - }), - ), - delete: (id) => provider.track(collection.delete(id)), + collection.update(row.id, (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note! + // A `NULL` note is already absent; only a held one is removed. + else if (typeof draft.note === `string`) + (draft as { note?: string | null }).note = undefined + }), + delete: (id) => collection.delete(id), }, provider, settle: provider.settle, diff --git a/packages/powersync-db-collection/tests/mutation-conformance-driver.ts b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts index 946902bc5f..67fb60a199 100644 --- a/packages/powersync-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/powersync-db-collection/tests/mutation-conformance-driver.ts @@ -46,6 +46,7 @@ import { PowerSyncDatabase, Schema, Table, column } from '@powersync/node' import { createCollection } from '@tanstack/db' import { powerSyncCollectionOptions } from '../src' import { + acceptAtOnce, conformanceKey, fromSqlRow, observedView, @@ -59,6 +60,7 @@ import type { MutationFixture, MutationProvider, ProviderChange, + SettleCut, WritePlan, WriteRequest, } from '../../db-adapter-conformance/src' @@ -72,14 +74,6 @@ const SCHEMA = new Schema({ type Record = { id: string; title: string | null; note: string | null } -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} - const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)) const drainTimeoutMs = 5_000 @@ -245,21 +239,6 @@ class PowerSyncMutationProvider implements MutationProvider { if (changed) this.expectDrain() } - /** Writes made through the Collection whose receipts have not settled. */ - private unsettled = 0 - - /** Counts a write's receipt until it settles. */ - track Promise }>( - receipt: T, - ): T { - this.unsettled++ - receipt.when(`settled`).then( - () => this.unsettled--, - () => this.unsettled--, - ) - return receipt - } - async seed(rows: ReadonlyArray): Promise { for (const row of rows) await this.write({ type: `upsert`, row }) this.drainTarget = undefined @@ -269,9 +248,6 @@ class PowerSyncMutationProvider implements MutationProvider { this.plans.push(plan) } - /** No request is held: the driver does not declare `held-writes`. */ - release(): void {} - foreignCommit(change: ProviderChange): Promise { return this.write(change) } @@ -293,13 +269,13 @@ class PowerSyncMutationProvider implements MutationProvider { * and the watcher runs at most every 30 ms, so the step waits until every * write's receipt has settled and the watcher has drained every commit. */ - settle = async () => { + settle = async ({ unsettled }: SettleCut) => { const deadline = Date.now() + drainTimeoutMs while (Date.now() < deadline) { const drainedAll = (this.drainTarget === undefined || this.drained >= this.drainTarget) && this.running === 0 - if (this.unsettled === 0 && drainedAll) { + if (unsettled() === 0 && drainedAll) { this.drainTarget = undefined return } @@ -347,21 +323,17 @@ export function powerSyncMutationDriver(): MutationDriver { writer: { insert: (row) => { const { id, title, note } = toSqlRow(row) - return provider.track( - collection.insert({ id: String(id), title, note }), - ) + return collection.insert({ id: String(id), title, note }) }, update: (row) => - provider.track( - collection.update(String(row.id), (draft) => { - draft.title = row.title - if (Object.hasOwn(row, `note`)) draft.note = row.note ?? null - // A `NULL` note is already absent; only a held one is removed. - else if (typeof draft.note === `string`) - (draft as { note?: string | null }).note = undefined - }), - ), - delete: (id) => provider.track(collection.delete(String(id))), + collection.update(String(row.id), (draft) => { + draft.title = row.title + if (Object.hasOwn(row, `note`)) draft.note = row.note! + // A `NULL` note is already absent; only a held one is removed. + else if (typeof draft.note === `string`) + (draft as { note?: string | null }).note = undefined + }), + delete: (id) => collection.delete(String(id)), }, provider, settle: provider.settle, diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index d3b9741392..afa59c366e 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -50,6 +50,7 @@ import { createRxDatabase, prepareQuery } from 'rxdb/plugins/core' import { getRxStorageMemory } from 'rxdb/plugins/storage-memory' import { rxdbCollectionOptions } from '../src/rxdb' import { + acceptAtOnce, conformanceKey, observedView, rewriteRow, @@ -87,13 +88,6 @@ const schema = { required: [`id`, `title`], } as const -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} /** * An RxDB document as a conformance row: key `"n"` is row id `n`. Other diff --git a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts index 5c2855239b..2f22fb1737 100644 --- a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts @@ -38,6 +38,7 @@ import { createCollection } from '@tanstack/db' import { trailBaseCollectionOptions } from '../src/trailbase' import { + acceptAtOnce, fromSqlRow, observedView, rewriteRow, @@ -66,13 +67,6 @@ type ConformanceRecord = { note: string | null | undefined } -const acceptAtOnce: WritePlan = { - outcome: `accept`, - held: false, - echoHeld: false, - rewrite: `none`, - error: new Error(`unplanned write`), -} /** A value as the SDK sends it: JSON, so an `undefined` field is absent. */ const asSent = (value: T): T => JSON.parse(JSON.stringify(value)) as T From 43cb8c9fee334a113dc7d1f7d9da3be87cb1b955 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 4 Nov 2026 08:24:50 -0700 Subject: [PATCH 25/28] chore(changeset): describe TrailBase's final write behavior The TrailBase changeset now covers the in-order transaction requests and that a removal does not call the field's serializer. Co-authored-by: Isaac --- .changeset/trailbase-ordered-confirmed-writes.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.changeset/trailbase-ordered-confirmed-writes.md b/.changeset/trailbase-ordered-confirmed-writes.md index d0da3778ac..2dbbdee04b 100644 --- a/.changeset/trailbase-ordered-confirmed-writes.md +++ b/.changeset/trailbase-ordered-confirmed-writes.md @@ -2,8 +2,9 @@ '@tanstack/trailbase-db-collection': patch --- -Fix three write bugs. +Fix four write bugs. -- Removing a field in an update (for example `draft.note = undefined`) now sends `null`, so TrailBase clears the column. Before, the update omitted the field, and TrailBase kept the old value. +- Removing a field in an update (for example `draft.note = undefined`) now sends `null`, so TrailBase clears the column. Before, the update omitted the field, and TrailBase kept the old value. The field's `serialize` function is not called for a removal. - Writes now reach TrailBase in the order the application made them. A write starts only after TrailBase has answered the earlier write on the same collection. It does not wait for the earlier write's change event. +- A transaction's update and delete requests now go out one at a time, in mutation order. Before, they went out together, so after one was rejected the next write could start while another request was still unanswered. A rejection now ends the transaction, and its later requests are not sent. - A write is now confirmed only by its own change event. Before, an earlier event for the same record could confirm a later write, so its optimistic value dropped and the earlier value showed until the later event arrived. This change also removes the adapter's periodic cleanup timer. From 78e995b17e88f8ddcd7c076e24e674158a2e1d23 Mon Sep 17 00:00:00 2001 From: "autofix-ci[bot]" <114827586+autofix-ci[bot]@users.noreply.github.com> Date: Wed, 4 Nov 2026 15:41:54 +0000 Subject: [PATCH 26/28] ci: apply automated fixes --- .../tests/reference-mutation-adapter.ts | 7 +------ .../tests/mutation-conformance-driver.ts | 1 - .../tests/mutation-conformance-driver.ts | 1 - 3 files changed, 1 insertion(+), 8 deletions(-) diff --git a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts index ff202e12b9..d94484d3f7 100644 --- a/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts +++ b/packages/db-adapter-conformance/tests/reference-mutation-adapter.ts @@ -30,11 +30,7 @@ * (MUT-10). */ import { createCollection } from '@tanstack/db' -import { - acceptAtOnce, - observedView, - rewriteRow, -} from '../src' +import { acceptAtOnce, observedView, rewriteRow } from '../src' import type { ConformanceRow, MutationCapability, @@ -115,7 +111,6 @@ export interface ReferenceMutationOptions { inOrder?: boolean } - const jsonClone = (value: T): T => JSON.parse(JSON.stringify(value)) as T /** A provider that answers writes by plan and feeds every commit back. */ diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index afa59c366e..84a00df581 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -88,7 +88,6 @@ const schema = { required: [`id`, `title`], } as const - /** * An RxDB document as a conformance row: key `"n"` is row id `n`. Other * fields pass through, so an optimistic row's own `undefined` stays, and diff --git a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts index 2f22fb1737..1007c55b3d 100644 --- a/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/trailbase-db-collection/tests/mutation-conformance-driver.ts @@ -67,7 +67,6 @@ type ConformanceRecord = { note: string | null | undefined } - /** A value as the SDK sends it: JSON, so an `undefined` field is absent. */ const asSent = (value: T): T => JSON.parse(JSON.stringify(value)) as T From a4bbb873de1d95ab8c5e39383c1f86a0bda0a7f6 Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 4 Nov 2026 09:36:41 -0700 Subject: [PATCH 27/28] test(rxdb-db-collection): rename the driver's storage-instance field The mangle check rejects `instance` outside packages/db, because core's minified build mangles a private member with that name. The RxDB mutation driver's field is now `storageInstance`. Co-authored-by: Isaac --- .../tests/mutation-conformance-driver.ts | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts index afa59c366e..9555ccf698 100644 --- a/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts +++ b/packages/rxdb-db-collection/tests/mutation-conformance-driver.ts @@ -88,7 +88,6 @@ const schema = { required: [`id`, `title`], } as const - /** * An RxDB document as a conformance row: key `"n"` is row id `n`. Other * fields pass through, so an optimistic row's own `undefined` stays, and @@ -188,7 +187,7 @@ class RxdbMutationProvider implements MutationProvider { private foreign = false /** Each pending write's document as the adapter sent it, before hooks. */ private readonly sent = new Map() - private instance: RxStorageInstance | undefined + private storageInstance: RxStorageInstance | undefined rxCollection: RxCollection | undefined /** Memory storage whose `conformance` collection writes go through `write`. */ @@ -201,7 +200,7 @@ class RxdbMutationProvider implements MutationProvider { createStorageInstance: async (params) => { const instance = await memory.createStorageInstance(params) if (params.collectionName !== `conformance`) return instance - this.instance = instance as unknown as RxStorageInstance< + this.storageInstance = instance as unknown as RxStorageInstance< Doc, unknown, unknown @@ -249,7 +248,7 @@ class RxdbMutationProvider implements MutationProvider { } private async holds(id: string): Promise { - const found = await this.instance!.findDocumentsById([id], true) + const found = await this.storageInstance!.findDocumentsById([id], true) return found.length > 0 } @@ -314,7 +313,7 @@ class RxdbMutationProvider implements MutationProvider { /** The storage's documents, not RxDB's document cache. */ async rows(): Promise> { - const instance = this.instance! + const instance = this.storageInstance! const { documents } = await instance.query( prepareQuery(instance.schema, { selector: { _deleted: false }, From 08035f0fc04e2b05a22230489f654e2d3ac5739d Mon Sep 17 00:00:00 2001 From: Test Date: Wed, 4 Nov 2026 09:41:36 -0700 Subject: [PATCH 28/28] test(db): measure removed-field reach in each optimistic-history campaign CodeRabbit review of 78e995b17: - The removed-field reach came from a 40-history sample of the fixed seed, so the random campaign's reach was never measured (ORC-015). Both optimistic-history campaigns now declare `removed-held-field` with a minimum of five histories and assert it after the campaign; a path replay skips the assertion. The sampled test is gone. - MIGRATION.md's plan said the grammar generates manual transactions and same-key commits by another client. It now says those are planned extensions. The TrailBase log entry now records the per-key numbered confirmation that replaced "any event after the write began". - GREEN: collection-state-retention-oracle and oracle-replay, 189 passed; removal reach met in three runs of each campaign. Co-authored-by: Isaac --- packages/db-adapter-conformance/MIGRATION.md | 17 ++++-- ...on-state-retention-oracle.property.test.ts | 58 ++++++++++--------- 2 files changed, 42 insertions(+), 33 deletions(-) diff --git a/packages/db-adapter-conformance/MIGRATION.md b/packages/db-adapter-conformance/MIGRATION.md index 1c1efac9f6..f0327710b3 100644 --- a/packages/db-adapter-conformance/MIGRATION.md +++ b/packages/db-adapter-conformance/MIGRATION.md @@ -71,10 +71,12 @@ state (MUT-5) stay with `packages/db/tests/optimistic-history-oracle.ts`. - **Model:** the synced rows, the backend rows, and the pending writes in author order with their outcome. Visible rows are the synced rows with the pending writes folded on; a settled write leaves the fold. -- **Grammar:** inserts, updates, and deletes through the Collection, - single or in one manual transaction; the backend accepts or rejects a held - write; the backend's own changes arrive through sync (confirming a pending - write or not); an external change to another row or the same key. +- **Grammar:** single-row inserts, updates, and deletes through the + Collection; the backend accepts or rejects a write, at once or held; the + backend's copy arrives through sync, at once or withheld; a commit by + another client while no write is pending. Manual and multi-mutation + transactions, and commits by another client while a write is pending + (MUT-11), are planned grammar extensions, not generated yet. - **Observations:** visible rows and change events, each transaction's state and settlement, the backend rows and the order of backend operations, handler calls. @@ -599,7 +601,12 @@ error/sync-give-up, accept]`). Fix: remember that give-up and fail the - **Fix:** a removed field is sent as `null`; writes run in author order; a write is confirmed only by an event after it began, recorded only while a write is admitted, which removes the periodic cleanup timer. - A mutant with the old confirmation fails MUT-1. + A mutant with the old confirmation fails MUT-1. Refined on 2026-11-04: + same-key histories showed that an earlier write's late event could + still confirm a later write on the key, so the adapter now counts + events per key and confirms the n-th answered write on a key by the + n-th event on it; a late event from an earlier write does not confirm + a later one. - **Size:** trailbase-db-collection −46 gzip bytes. - **2026-11-03, PowerSync, mutations.** Both campaigns are GREEN after an adapter fix, with every declared reach state met. diff --git a/packages/db/tests/collection-state-retention-oracle.property.test.ts b/packages/db/tests/collection-state-retention-oracle.property.test.ts index 3cb4cd8f34..3da4555702 100644 --- a/packages/db/tests/collection-state-retention-oracle.property.test.ts +++ b/packages/db/tests/collection-state-retention-oracle.property.test.ts @@ -809,19 +809,6 @@ it(`writes zero-sign edits over in-flight writes in the fixed campaign`, async ( expect(zeroSignEdits).toBeGreaterThan(0) }) -// Reach: an edit that removes a field the row holds. The update API keeps -// the removed field as an own `undefined`, which only strict row comparisons -// distinguish from an absent one. -it(`removes a held field in the fixed campaign`, async () => { - const histories = fc.sample(optimisticHistory, { seed: 86103, numRuns: 40 }) - let removedFieldEdits = 0 - for (const { initial, steps } of histories) { - const counts = await runOptimisticHistory(initial, steps) - removedFieldEdits += counts.removedFieldEdits - } - expect(removedFieldEdits).toBeGreaterThan(0) -}) - // Pinned replays for a source delete of a key the source never held. The // backend accepted an optimistic insert, then deleted the row before the // source streamed it. The model's drain applies the delete to the applied @@ -1215,21 +1202,36 @@ it.each( }) }, ) -fcTest.prop([optimisticHistory], { numRuns: oracleRuns(100), seed: 86103 })( - `matches optimistic ownership and publication histories with a fixed seed`, - async ({ initial, steps }) => { - await runOptimisticHistory(initial, steps) - }, -) -fcTest.prop( - [optimisticHistory], - oraclePropertyOptions(100, `collection-state.optimistic-history`), -)( - `matches optimistic ownership and publication histories with a random or replayed seed`, - async ({ initial, steps }) => { - await runOptimisticHistory(initial, steps) - }, -) +// Reach, per campaign (ORC-015): an edit that removes a field the row holds. +// The update API keeps the removed field as an own `undefined`, which only +// strict row comparisons distinguish from an absent one. +async function optimisticCampaign( + campaign: string, + options: Parameters[1], +): Promise { + const reach = createOracleReach( + `collection-state.optimistic-history (${campaign})`, + [`removed-held-field`], + { minimum: 5 }, + ) + await fc.assert( + fc.asyncProperty(optimisticHistory, async ({ initial, steps }) => { + reach.beginHistory() + const counts = await runOptimisticHistory(initial, steps) + if (counts.removedFieldEdits > 0) reach.reach(`removed-held-field`) + }), + options, + ) + reach.assertReached() +} + +it(`matches optimistic ownership and publication histories with a fixed seed`, () => + optimisticCampaign(`fixed`, { numRuns: oracleRuns(100), seed: 86103 })) +it(`matches optimistic ownership and publication histories with a random or replayed seed`, () => + optimisticCampaign( + `random`, + oraclePropertyOptions(100, `collection-state.optimistic-history`), + )) // An aborted open batch never applies, so the source's own rows must return // to their state before it. A later partial update then omits `c` again, and // the Collection keeps the held `c`. Shrunk from seed -526998278.