Repository navigation
Views: v3 storage for every index and worker publication through v3 - #998
Open
aft-alfonso[bot] wants to merge 12 commits into
Open
aft-alfonso[bot] wants to merge 12 commits into
aft-alfonso[bot] wants to merge 12 commits into
Conversation
…robe Adds the base_refs, base_builds, strict_rebuild, base_sweeps and sweep_state tables as registry format/schema 2/2 (reserved for slice 4 in docs/design/views-v3-format.md), installed only by an exact-predecessor upgrade from 1/1 after the registry reader floor is raised to 2. Registries created by the 3a copy path stay 1/1. views_v3_registry is now advertised as readable at 2, so reader_floor's advertisement test and the v3 newer-format refusal test (which used 2 as 'newer') move to 2 and 3 respectively. Also: barrier-only registry opens (no covering measurement), a plane-row sweep over the family's v3 plane stores, LedgerReader member/read helpers, and GetDiskFreeSpaceExW for the non-unix free-space probe, which 3a left unsupported. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Base identity: tagged, length-prefixed BLAKE3 digests of a base manifest's membership and content (readiness excluded; head_tree and git_oid refused, slice 4 bases are membership-keyed) and of its effective identity (family, producers, ignore fingerprint, platform family, membership). The base scope name derives from the identity digest. resolve_base pins the base's current generation with a read marker, re-verifies pointer and member row, and recomputes both digests from the pinned manifest; only that AuthenticatedBase validates a BaseHeader (bound_generation stays diagnostic). Worker side: publish_worker runs the insertion barrier (BaseRefMarker with a per-acquisition nonce plus an intent row in one IMMEDIATE transaction under the reservation lock), authenticates the base after the barrier, rebinds the delta to the authenticated header, verify_held and a base re-check right before the pointer CAS, then ReferrerPublished. Conflict/EquivalentWinner and failed verifies delete only their own intent row under the barrier before the marker drops; a CAS that may have landed is published, never cleaned up. open_foreign_v3 (and read::open_foreign_v3_generation) does the dual pin-then-verify with three attempts and owns both markers. read::open_foreign_generation keeps refusing a base header, now because a v2 view can never hold an authenticatable base reference (rule 6); its test asserts the new reason. Sweep: marking of current, read-marked and pinned generations under the reservation lock, exact edges, dangling edges counted and recorded in strict_rebuild without aborting, bounded reconciliation (256 rows / 25 ms, updated_epoch < S only, wrap-around position), repair of valid edges, Hold::Referenced via eviction::v3_base_hold, the two-sweep reclaim, and the plane-row sweep. The retention collector runs it per v3 family visit and deletes a deregistered worker's rows with its member row. DurabilityStep gains the seven base-path boundaries (BASE_PATH); the v2 kill test's step list excludes them. Tests use injected EIO/EXDEV/ENOSPC at each boundary through contracts::inject_fault_at. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
base_builds claim with the owner-token CAS: insert when absent, Busy while the owner lives (running, stopped or uncertain), takeover of a dead owner with attempt+1 and a fresh token, after which the dead attempt's staging/a<attempt>-<token>-* directories are removed. Staging and derived names carry (attempt, owner_token). The live-base cap (64 per family, counting claimed-but-unregistered bases) refuses new claims and counts the hit in sweep_state.live_base_cap_hits. plan_binder: only the claim owner builds; others wait (ACK + direct scan) and go strict only on a failed claim, the cap, or the ruled 600 s wait. initial_reservation implements the ruled formula with the 1 GiB cap. DiskBudget::admit_v3/extend_v3 admit v3 reservations against the family hard limit (refused as storage_gap:v3_family_hard_limit, no legacy fallback) and flag usage over the 2 GiB target; RegistryWriter::extend grows a live reservation under the floor and namespace checks. No limit or budget constant changes. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
build_base reads the walker's membership with stability-checked reads (equal size, mtime and device/inode before and after, three attempts), builds a membership-keyed manifest (head_tree None, no git_oid), claims the base, admits the ruled initial reservation through DiskBudget::admit_v3 (a refusal releases the claim and returns the storage gap for the strict path), registers the base, writes the lexical plane into the v3 trigram store (trigram::materialize_v3, keys protected by a v3 live pin first) and first-publishes with graph and semantic planes pending. publish_successor lets only the live claim owner publish monotone readiness successors; a model outage is recorded as Failed while lexical serving continues. v3 generations carry no trigram segment: v3 has no segment rows or segment collection yet, so readers build from payloads. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
- G7: two child binders (one SIGSTOPped with its intent row across two sweeps, then killed) plus in-process binds with one aborting; the base is never counted, the stopped owner's intent is kept, then orphaned, then retired, and published references keep the base. - Kill test: a child parks at each DurabilityStep::BASE_PATH boundary (the child refuses unknown or non-base step names; the parent checks the ready file names the step), is SIGKILLed, and a fresh process sweeps: dead reservations are released, every published reference resolves, a dead BaseClaimed claim is taken over with attempt 2, and nothing appears under <storage>/views/v3. - Delete main, kill main's daemon, run GC in a standalone process, restart the worker: it composes over its base with correct contents. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
… format doc Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…m the republication row test The republication test now runs its sweep with a 60 s reconciliation time budget: under a loaded test runner the ruled 25 ms budget could end the pass before its final empty batch, which marks the pass incomplete without changing the rows the test asserts. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…callgraph planes Storage route: a generation opened or published in the v3 namespace carries a V3Route (storage, family, member scope); ViewAccess::route(generation) returns StorageRoute::V2 or ::V3, and ViewRegistration::v3_route names a checkout's v3 member. Routing is per generation rather than per access because one checkout reads both v2 and v3 generations during the switch. resolve_base, publish_worker and open_foreign_v3 set the route. - v3 store API: PlaneReader::contains, Publication::put_or_touch and Publication::touch (missing keys reported), PayloadSource for PlaneReader. - Trigram: a v3 generation has no segment file. The adapter assembles the base generation's segment in memory from the base's stored payloads (cached per base generation) and overlays the worker's changed entries from the v3 store. No new on-disk rows. materialize_v3_observed writes a worker delta's payloads from attachments or verified reads. - Semantic: SemanticStore (v2 FamilyStore or v3 route) carries put_or_touch/touch/get/reader for fills; v3 writes go through v3::publish, fills of a v3 generation protect keys in a v3 live pin (LivePin::create_v3), and verified-run markers are skipped on v3. The arena reads runs through a RunSource trait (v2 reader or v3 PlaneReader). SemanticPlane::materialize_v3 applies the existing readiness rules with v3 presence checks and touches. - Callgraph: CompositePlane::materialize_v3/finish_generation_v3. The bridge puts delta blobs into v3 and builds the composed generation's derived DB in the worker's v3 member directory once the projection is complete; CallgraphPlane::open_generation reads a v3 generation's derived DB from its member directory. Base successors build derived-<gen>-a<attempt>-<token>.sqlite named by derived-<gen>.ref. - publish_worker_with runs a step with the final generation name before the CAS (for derived renames). Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit 50757b3b99fc5881775c636377d92a316b14266b)
…hrough v3 The base-header branch of build_own_generation no longer refuses. It authenticates the named base (pin-then-verify, digests recomputed; a header that names no registered base is refused before any write), then folds the cut into a sparse delta against it. A path whose bytes equal the base's inherits the base entry, with its readiness patch as an override; a deleted base path is a tombstone; only changed paths are entries. The plane delta keys are materialized into v3 under a v3 live pin, and a private staging name is pinned in the checkout's v3 member directory. Publication goes through base_refs::publish_worker_with, whose pre-CAS step runs each plane's finish_generation_v3. Wold is the installed v3 generation or the durable v3 pointer. Published opens the generation's readers for installation; Lost (Conflict/EquivalentWinner) and failed or possibly landed publications set the retry flag and return CHECKOUT_CHANGED_BEFORE_INSTALL, the existing revalidate path. No v1/v2 row, manifest or pointer is written. A plane without v3 storage now leaves its delta entries pending, instead of refusing. unwired_worker_publication_refuses_before_any_legacy_write is replaced: a_base_referencing_worker_publishes_a_sparse_v3_generation_readable_after_restart checks that the generation is sparse (entries = changed paths, tombstone for the deleted base path, nothing inherited copied), names the authenticated base, leaves the v2 pointer untouched, and composes correctly for a fresh reader after the driver is dropped. Its refusal half survives as a_worker_header_naming_an_unknown_base_is_refused_before_any_write. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit 6eb0522ec0ca0e74b79af075eaa17f5ddfbb89cb)
…segments Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit c26f1c23821947845961d0f6b686efa836893f0e)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
ualtinok
force-pushed
the
land/views-c-base-references
branch
from
October 27, 2026 06:01
142f103 to
8fd657c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #997 (and #993); retarget to main as they merge. Views stay off by default; no routing change.
Known gaps, by design for this step: the v3 live pin is not trimmed after a fold (over-protects), each v3 semantic write is its own publication, and there is no end-to-end semantic fill test over a v3 generation yet.