Repository navigation
Add driver-backed claims for TinyMemory queues - #34
Merged
Merged
Conversation
Introduce an opt-in `storage` feature that pulls in TinyStorageDrivers' core crate and exposes a new storage module. The module builds driver-backed, per-item durable queues on the public ports, leaving driver selection to the host so this crate stays backend-agnostic. Auto-committed-on: dragonfly
Queue stores now expose enqueue, claim, and settle operations so each job is inserted and settled individually instead of rewriting the whole queue state. The default implementations keep older local stores working, while durable multi-consumer stores can override them with atomic claims and visibility leases. Auto-committed-on: dragonfly
Adds an import_items method that copies decoded legacy queue entries into the document store without touching the source, using stable ids and negative sequences so imported rows sort ahead of native appends. The deletion and pending-write queues now append, claim, acknowledge and retry through the list store instead of rewriting the whole list, so a drain no longer races with concurrent enqueues. Auto-committed-on: dragonfly
Add a remove_matching default to ListStore so pending writes can drop stale entries in one pass instead of deleting them one by one. The driver queue store gains import_legacy, which loads a legacy whole-state snapshot into the job and history families while reporting identical and conflicting rows, and history reads are now capped at the queue's maximum size. Auto-committed-on: dragonfly
Follow-up jobs were enqueued in separate writes after a claim was settled, so a crash between the two steps could lose them. Settlement now takes the follow-ups and inserts them in the same atomic batch as the run record, skipping any that already exist. Auto-committed-on: dragonfly
The default list store now records the claimed item's index so acknowledge and nack update the correct entry instead of always operating on the first one. Claim selection also honors the excluded id list via a caller-supplied id function, and queue claims populate the new position field. Auto-committed-on: dragonfly
The deletion queue and pending writes now supply an exclusion predicate when claiming the oldest item, so already-processed entries are skipped during a drain loop instead of being re-claimed. Auto-committed-on: dragonfly
Moved the driver-backed list and queue store implementations out of the storage module into dedicated list and queue_store submodules, re-exporting the public types so callers are unaffected. This keeps the module focused on schema and shared helpers instead of holding the full adapter code. Auto-committed-on: dragonfly
Moved the error helper and Stored type into the storage module so the queue store can reuse them, and added a QueueImportReport that tracks pending job and history row outcomes separately when importing a legacy snapshot. Auto-committed-on: dragonfly
The remove_matching helper now uses the `?` operator on load so storage failures surface as StoreError instead of being silently ignored. The unused document_version helper and its Version import were dropped since nothing referenced them anymore. Auto-committed-on: dragonfly
The pending-writes and job-queue tests that raced two consumers against a shared store have been dropped, and the storage module now wires up its own test submodule behind the storage feature. The removed tests asserted single-claim behaviour that the storage layer no longer needs to prove. Auto-committed-on: dragonfly
Introduce a test module covering the storage layer so its behaviour is verified automatically. The tests exercise the core storage operations and guard against regressions in future changes. Auto-committed-on: dragonfly
Drops the unused MemoryEngine and Entry imports along with the dead _entry_shape_proof helper, which only existed to keep the Entry import alive. Auto-committed-on: dragonfly
The candidate query is only a hint, so another consumer may have acknowledged, removed, or retried an item before its lease was acquired. Claiming now refreshes eligibility and observed state under ownership via a fenced read and a stricter claim filter, and a hook exposes the candidate-read and lease-acquired points for instrumentation. Auto-committed-on: dragonfly
…licting Import now re-reads a row when the absent-precondition insert loses a race, counting it as identical when the stored item matches and as a conflict when it differs. The queue store's import path likewise only treats conflict errors as races and propagates other storage errors instead of swallowing them. Auto-committed-on: dragonfly
Expanded the import_items doc comment to explain that legacy snapshots must be supplied as one ordered batch and that ids must remain stable across retries. Also added a test covering exclusive deletion consumption across two drivers. Auto-committed-on: dragonfly
Expanded the doc comments on both storage traits to describe them as synchronous host ports, noting that local providers may use the default snapshot-based read-modify-write methods while multi-consumer providers override the item-level operations with atomic variants. Also added a pending_store test helper and switched the two-consumer pending write test to use it. Auto-committed-on: dragonfly
Add tests asserting that a retried driver claim stays hidden until its visibility deadline and reappears with the updated payload, and that importing a legacy queue state is idempotent while preserving newer native jobs and conflicting target rows. Document the durable queue adapters and their lease semantics in the crate README. Auto-committed-on: dragonfly
The two-driver queue tests relied on tokio::join! racing two drains, which could pass without ever exercising overlap. The executors now pause on a Notify until the second consumer has started, so the tests assert that a healthy item is executed exactly once under genuine concurrency. Auto-committed-on: dragonfly
The doc comment on the queue's mutex now states that it only serialises operations through a single handle, and that independent durable handles coordinate claims through their storage provider. This corrects a misleading claim that the lock prevented two runs from taking the same job. Auto-committed-on: dragonfly
The list and queue snapshot importers now delete documents missing from the incoming snapshot and write entries in a single blocking task, so a save reflects the snapshot exactly instead of appending on top of existing state. Queue history is rewritten in reverse order after clearing prior runs to keep sequence numbering consistent. Auto-committed-on: dragonfly
Extend the storage driver tests to verify that saving a fresh set of items or queue state fully replaces the previously persisted contents rather than merging with them. This guards against regressions where stale entries survive a subsequent save. Auto-committed-on: dragonfly
The save methods on the list and queue driver stores now return an error instead of replacing snapshots, since live processing goes through append, claim/ack/retry, and import paths. The previous replacement logic could delete or rewrite documents that concurrent queue operations still relied on. Auto-committed-on: dragonfly
Driver-backed queue stores now return an error from save instead of silently accepting a snapshot, and the trait docs were updated to match. The tests were adjusted to assert that save fails and leaves the stored state unchanged. Auto-committed-on: dragonfly
The list store no longer caches a holder id at construction time; each lease acquisition now generates a fresh unique holder. This prevents concurrent claims through the same store handle from treating a single lease as belonging to both workers. Auto-committed-on: dragonfly
Auto-committed-on: dragonfly
The test asserting that concurrent claims through one store handle use distinct lease holders has been dropped, leaving the exclusive fenced claim coverage to the driver-level test that follows. Auto-committed-on: dragonfly
Add a regression test asserting that a single store handle cannot claim a live item twice, and that the original claim can still be acknowledged afterwards. This guards the single-owner invariant against future changes to the claim path. Auto-committed-on: dragonfly
The queue store now imports QueueImportReport directly from its parent module instead of relying on a public re-export in storage.rs. This keeps the public surface of the storage module limited to DriverQueueStore. Auto-committed-on: dragonfly
The enqueue path now clones the job id before moving the job into the append call, avoiding a borrow conflict without changing behaviour. An identity mapping in the list storage path was dropped as dead code. Auto-committed-on: dragonfly
Adds the tinystoragedrivers-core git dependency to the lockfile along with its transitive crates, including data-encoding, ulid, zstd, and a second rand_core version. The duplicate rand and r-efi entries are pinned to keep the existing dependency graph intact. Auto-committed-on: dragonfly
The Stored type is now imported from the list module in queue_store and list, and the re-export from storage.rs was dropped. The deletion module gained a HashMap import to support its work. Auto-committed-on: dragonfly
Added explicit error type annotations to fallible results in the list and pending stores so type inference resolves without ambiguity. Also removed redundant default write options from queue store write operations and restructured the import path to return its report directly. Auto-committed-on: dragonfly
Added explicit bool and StorageError type annotations to the return values in the queue store's claim path so the inferred types are unambiguous. No behaviour change. Auto-committed-on: dragonfly
Dropped the lease_key and leases helpers from the list store since nothing calls them anymore. Updated the storage tests to match the current module paths and gate construction. Auto-committed-on: dragonfly
Auto-committed-on: dragonfly
Added explicit generic type annotations to the DriverListStore constructors so the intended item type is clear at each call site, and imported DocumentStoreExt to bring the extension trait into scope. The import hook closures in the concurrent import test were reformatted into block bodies so the barrier wait is unambiguous. Auto-committed-on: dragonfly
The claim hook now returns a boxed future so callers can await it, and both invocation points in the claim loop await the result and propagate storage errors. This lets hooks perform asynchronous work during claiming instead of being limited to synchronous callbacks. Auto-committed-on: dragonfly
Claim hooks now return a boxed future instead of running through a blocking adapter, so tests can await storage operations directly. The takeover test acquires the expired lease through the leases API rather than a second store instance. Auto-committed-on: dragonfly
Upgraded calamine to 0.37, quick-xml to 0.42 and zip to 9 in the integrations crate so the document and office readers stay on the versions calamine links. The zip bump pulls in nix as a transitive dependency. Also allowed the tinystoragedrivers git source in deny.toml. Auto-committed-on: dragonfly
…orrectly The OOXML text extractor now skips unreadable archive entries instead of dropping the whole slide list, and compares tag names without allocating a byte slice per event. Text events use xml10_content so character references are decoded consistently, and predefined entities are resolved directly from the reference name rather than through a redundant decode step. Auto-committed-on: dragonfly
Replace the CORTEXDB_VERSION variable with CORTEXDB_IMAGE so the live harness selects a server image directly. The default is now the official v0.8.21 image pinned by its multi-platform manifest digest, which keeps the integration tests reproducible while still allowing an override to check another release. Auto-committed-on: dragonfly
Added comments explaining that the image is the official v0.8.21 release and that the digest pins the multi-platform OCI index so CI and local runs stay consistent. Auto-committed-on: dragonfly
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background jobs and tool-write/deletion FIFOs stored as whole snapshots cannot coordinate independent consumers. The new
storagefeature providesDriverQueueStoreandDriverListStore: one document per item, visibility leases, atomic claims, and fenced acknowledgement/retry.Job settlement records history, prunes retained history, acknowledges the job, and adds follow-ups atomically. It retries history contention under the same claim, stops after five conflicts, deduplicates identical follow-up IDs, and rejects divergent payloads sharing an ID. A public
ItemClaimconstructor lets external store implementations use the claim port.Decoded legacy imports are idempotent, preserve target rows, and report conflicts. Host leaf senamakel/openhuman#293 owns reading and retaining desktop source files. Driver snapshot replacement fails closed; legacy snapshot providers remain local and exclusive. Healthy-consumer tests use separate driver handles sharing a real memory backend. The earlier RacingStore RED fixtures demonstrate the old snapshot limitation; they are not a claim that arbitrary legacy stores gained atomicity.
Delivery is at least once after crashes or visibility expiry, so external effects may repeat. System-clock leases require reasonably synchronized hosts.
Refs #30
Part of tinyhumansai/openhuman#7419
Local validation at
086381786f4c5b611891c863938025e85f9e4452:cargo test -p tinymemory-tools --features storage: 284 tests passed (256 unit, 2 contract, 11 roundtrip, 7 scoping, 8 doctests).cargo check -p tinymemory-tools --features storageandcargo check -p tinymemory-tools --no-default-featurespassed.cargo clippy -p tinymemory-tools --features storage --all-targets -- -D warningsandcargo fmt --all -- --checkpassed.Regression evidence includes failed-then-passing full-history settlement, bounded-conflict, and duplicate-follow-up cases against unfixed production code. The retry/dedupe RED checks used saved working-tree source variants; their logged HEAD values identify the baseline, not separate test-only commits. Logs are retained under
target/debug-logs/memory-claims/.Hosted CI must be fully green before this native fork PR merges.
Supply-chain repair and dependency compatibility at
0fe305864eb2052ea49aa27de607cbc89f6977a1:deny.toml; unknown git sources remain denied.cargo test -p tinymemory-integrations --all-features -- --nocapturepassed (748 library tests, integration targets, and 8 doctests); strict all-target/all-feature Clippy,cargo fmt --all -- --check,git diff --check, andcargo deny check allpassed.cortexdb/cortexdb:v0.10.4image tag is absent; the live assertions have not been verified by that job.Full logs are retained in
target/debug-logs/mem30-dependency-repair/.The live CortexDB harness defaults to the official published
cortexdb/cortexdbv0.8.21 image, pinned by multi-platform OCI manifest digestsha256:f38e6f4573d1585aa37d87ef76d70eb3bf6681dbf7e30c2ebf2f6d91176f6be5; image metadata identifieshttps://github.com/cortexdbai/cortexdb-releases. The nonexistent v0.10.4 tag was replaced andCORTEXDB_IMAGEremains available for compatibility testing. Current hosted CI runs the live tests, with 13/14 passing; the remaining test exposes an upstream server defect, so this PR must not merge until a fixed official artifact is available. A same-scope HTTP probe against the pinned image wrote and recalled eventevt_01a153f75c077962ac576f0ef695807f, then/v1/forgetacknowledged deletion (deleted.events=1,matched=1). Answering with the old pack still returned 200 and cited that deleted event; a fresh recall returned an empty context and its fresh pack had no citations. The complete probe log istarget/debug-logs/mem30-dependency-repair/cortexdb-forget-http-probe.log. As of 2026-10-19, the official releases and Docker Hub registry list v0.8.21 as the newest published image; no newer fixed artifact is available.