Conversation
This was referenced Sep 18, 2026
grumbach
force-pushed
the
feat/pointers-immutable-owner
branch
from
September 24, 2026 02:48
aba2f7c to
a052ca7
Compare
Pointers (ADR-0015) are mutable, owner-signed references addressed at
BLAKE3(domain || owner_key). The record and its wire messages live in
ant-protocol; this is the node's half.
Storage. The chunk store answers "already have it" and stops, which is
right for an immutable record and drops every update for a mutable one,
and it requires BLAKE3(content) == address, which no pointer satisfies.
Pointers get their own store: merge-on-put under one lock, the rename as
the commit point, an exclusive directory lock, and an index dropped for
any address whose file stops validating so a repair is accepted rather
than answered "unchanged".
Payment. A quote is paid against the record's state_id and must be issued
by the close group around its address — two different values with two
different jobs, carried as PaymentTarget { routing, content }. Chunks pass
the same address for both, so their behaviour and cache entries are
unchanged. The paid cache is keyed by a typed Chunk vs PointerState: a raw
32-byte key would let a client store a chunk crafted to sit on a pointer's
entry and buy its update at chunk price.
One payment buys one increment. A create is counter 0 and a client update
is exactly one past what the node holds, so an owner cannot pay once and
jump the counter. Replication keeps the counter order instead, so a
replica behind a gap can still catch up.
Admission runs before the signature check — capacity, cross-kind refusal,
and responsibility for the pointer's address — so a forged record for
someone else's address buys no ML-DSA verification.
Audit. The commitment leaf gains a record kind, bound by hashing pointer
leaves under their own domain, so a chunk leaf cannot be relabelled to
escape the bytes_hash == key guard. Chunk-only roots are bit-identical.
Pointer leaves are refused at round 1 until round 2 serves a whole record:
a peer signs its own commitment, so it could otherwise name any key with
the hash of cheap bytes it holds.
as_bytes became to_bytes upstream: the record no longer caches its encoding, because a fixed-width layout has exactly one, so it re-encodes from the fields instead. The ADR now shows the record as the five fields it is, and says plainly why the key is carried — ML-DSA has no key recovery and a 1,952-byte key cannot be a 32-byte address.
Three simplifications, all deletions. The kind-tagged audit leaf goes. Nothing in production ever built a pointer leaf — commitment rotation reads the chunk store only — so LeafKind, pointer_leaf_hash, build_of_kinds and the per-leaf wire tag were machinery for a case that cannot arise. Removing them also puts the subtree audit family back to v1: the tag changed a positionally-encoded leaf, which would have paused mixed-version audits through a rollout for no benefit. The audit format is now untouched by this work. Prepared and PreparedPut go. PreparedPut was a one-field wrapper around Pointer re-exposing accessors Pointer already had, and Prepared mirrored Inspected arm for arm. Inspected gains a Verified arm and commit takes a Pointer directly, so there is one enum where there were two plus a wrapper. PaymentTarget and PaidKey merge into one enum that is both the routing decision and the cache key. Being an enum is the load-bearing part: a raw 32-byte cache key would let a client store a chunk crafted to sit on a pointer's entry and buy its update at chunk price. The ADR drops to Proposed while the PR is open, and its defence table now says which rows are local guarantees and which still need replication — fork convergence across the network is the client's doing until nodes forward pointers to each other.
bytes_hash, all_keys, all_states, missing_states and holds_state described commitments and version-aware sync that this PR does not build, and the conditional GET they served has gone from the wire with them. Nothing called any of it outside its own tests. The ADR listed bytes_hash as a third identity binding 'this node's commitment', which was false in the same breath as the section saying pointers take no part in commitments. Two identities now: one that routes, one that authorizes payment.
The index is a claim about a file, and both of inspect's early answers -- "unchanged" and "stale" -- assert the node already holds something at least as good as what arrived. Answered from the claim alone, a lost or corrupted file made the client count an acknowledgement for a record nobody could serve. The file is now read back before either answer, and a failed read disowns the entry so the submission lands as the repair it is. Also: - A pointer quote no longer consults the chunk store. A chunk sitting at a pointer's state address could set already_stored, and a majority of nodes saying "no payment needed" would strand the write as unpaid. - IndexEntry carries the held PointerState rather than three fields copied out of it, so merge order and the paid increment are the protocol's own rules applied to the held state -- is_successor_of now has one definition and one caller instead of two implementations. - Inspected::Verified, Inspected::state() and prepare() are gone; the tests that used them drive the same three steps production does. - tests/pointer_convergence.rs lacked the test-module lint allow every other integration test carries, so clippy --all-targets -D warnings failed on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It cancelled the commit with a one-nanosecond timeout and then asserted the cancellation had happened. On a fast runner the commit finishes inside any timeout, so the assertion fired and the CI unit-test job went red while the same test passed locally. It now polls the future exactly once -- that poll hands the write to a blocking task and returns Pending, always -- and drops it. Also: - Inspected loses its nested outcome: Unchanged(state) / Stale(state) / Candidate. The handler reads the address and state off the variant instead of re-parsing the record, which deletes state_of, both "parsed then failed to re-parse" internal errors, and the arm for a no-op outcome that inspect could never return. - The store module doc described a two-step put through prepare(), which no longer exists -- rustdoc's broken-link check would have failed CI on it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Disowning the index entry for a record whose file had gone left the address looking untouched -- and an address nothing is known about admits only a counter 0 record. So the repair added in the previous commit worked at creation and nowhere else: a pointer that had ever been updated could never be restored. An entry now records that it is no longer on disk rather than disappearing. The node stops serving it and stops counting it, but it still knows what it had, and admits anything at least as good as the lost state -- the merge rule replication would use to catch a replica up. A repair skips no payment: the state it carries was paid for and the rest of the group already serves it. Also: - The read-back compares the whole state, not just the address. The index and the disk are written under one lock, so a valid record for the same address that is not the one the index names means the answer would describe something this node cannot serve. - inspect runs on a blocking thread. It reads a record back off the disk, and a flood of arrivals must not put a blocking read on a runtime worker each. - The ADR no longer claims write and read use one group "by the same call" -- they are the same definition, but each does its own lookup -- and now says plainly that a majority of dishonest peers is not defended against. - PaymentTarget::is_single_address, which only its own test called, is gone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A directory cannot be opened as a file on Windows without backup semantics, so the post-rename directory sync failed on every write there. It was the Windows unit-test job's only failure on this branch, and the warning it logged was not a durability warning -- it was the wrong question asked of the wrong filesystem. The sync is now Unix-only, where a directory entry is a thing you can flush; NTFS orders the rename's own metadata. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The comments still described BLAKE3 over a domain prefix and the preimage that construction allowed; the identities are derive-key outputs now, and reaching one from a chunk takes a collision across two BLAKE3 modes rather than content anyone can write down. The typed paid-cache key does not rest on that either way, which is the point of it being typed. The ADR now also states two things it was quiet about: a read returns a state only when two answering peers name it, because one peer serving an owner-signed state nobody paid for would otherwise be believed by every reader; and a node that loses a record can repair it while running but not across a restart, where a missing file leaves nothing to remember. That, and a node joining a close group after a pointer exists, are the same missing mechanism -- replication. Also removes the empty test left behind when is_single_address was deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…y claims The ADR contradicted itself about the hash spaces on consecutive lines: it said neither identity was reachable from any chunk content, then said nothing proved the two ranges disjoint. Only the second is true. What derive-key buys is cost, not impossibility. It also understated where the defence ends. A read needs two peers to name a state, so one peer cannot decide what a pointer says -- but two colluding ones can, and since only the owner can sign, what that buys is the owner's own updates unpaid. Raising the bar only raises the number of nodes to grind: the pointer's address follows a key the owner chooses and node ids are choosable too, so an owner determined to sit beside their own pointer reaches any fixed threshold. Replication and audits are what actually answer it, and neither is built. The table says so now. And a read no longer necessarily ends at the answer quorum -- it keeps asking while the states it has seen are uncorroborated -- so the consequence that said it did is corrected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…can show Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… the handler The paid gate required exactly counter + 1, so a node holding one state refused another at the same counter even though both were separately paid and the merge rule says which wins. Give two nodes those states in opposite orders and each keeps a different record for ever -- the fork the merge rule exists to prevent, put back by the gate standing in front of it. The convergence tests never saw it: they drive the store directly, below the gate. The store now asks is_paid_update_of, which is "wins the merge and does not skip", and two new service tests drive the real request handler: two nodes given the same two states in opposite orders agree, and a counter jump is still refused after a tie-break has been taken. Also corrects two docs that claimed more than the code checks. get serves the file rather than the index, so a file replaced by a different validly signed record for the same address is served -- the reader verifies it and the read quorum decides between replicas that disagree. And the held-state check parses the body, so corruption inside the signature passes it and is caught on the next read; verifying there would put ML-DSA in front of the payment gate, which is the one place it must not be. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two corrections, both mine. The ADR and the admission comment still said every update is exactly counter + 1 after the rule changed to admit a separately paid tie-break winner at the counter already held. What holds now is that one payment buys one state and at most one increment: a tie-break moves the pointer without advancing the counter, but it must strictly descend in target bytes and each step is bought on its own state_id, so it buys nothing an ordinary update would not. And PaymentTarget had been inserted between VerificationContext's doc comment and its enum, so the long explanation of admission paths documented the wrong type and VerificationContext was left with a one-line stand-in. PaymentTarget now sits above that block with its own doc. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ADR-0015 was taken by direct browser clients over WebRTC-direct while this branch was open, and the governance check refuses two ADRs wearing one number. Renames the file, its title, and every reference to it across the node.
The rebase carried this branch's old Cargo.lock forward, which quietly downgraded dozens of transitive dependencies main had moved on from. It is now main's lock with one entry repinned: ant-protocol, for the pointer record and its wire messages.
Repinning ant-protocol with cargo update also re-resolved an unrelated Windows edge, putting winapi-util back on windows-sys 0.48.0 where main had moved it to 0.61.2. The lock is now main's with the one ant-protocol entry edited by hand, so nothing else moves. Also adds ADR-0016 to the index, and records there that browser clients cannot reach a pointer yet and what opening that path would take.
Pointer requests and responses were counted as Other, the bucket for non-request and unknown future variants. That is what every other request kind avoids, and it makes paid pointer traffic indistinguishable from messages the table does not understand. Both request kinds and all eight response outcomes now have their own keys, on a third summary line because group 2 is already near tracing's field cap. Unchanged and Stale are itemised apart from Success, since a re-submission and a lost race are exactly what you want to tell apart when counting paid writes. The classifiers are exhaustive -- the pointer response enums are not non_exhaustive -- so a new variant is a compile error here rather than silently becoming Other.
A pointer PUT asked check_capacity(), which asks whether a write of *zero* bytes would fit and charges nothing, and then wrote 5,303 bytes. The file store names that race itself: dozens of handlers can each pass against one cached measurement before any of them has written a byte, and collectively cross the configured reserve. Chunks avoid it by reserving; pointers did not. Pointer writes now take a reservation for the size a record actually is and hold it until the write lands, settled by net growth -- a create commits the charge, a replacement or a no-op releases it, because the disk only grows in the first case. The early admission check asks about the same size rather than about nothing, so a client still learns a full disk before paying. Three tests: a full disk refuses and stores nothing; eight concurrent creations each take and settle a charge; and sixty-four writes that change nothing give their charges back, which is the drop path -- a charge stranded there would be permanent, and enough of them would make an empty disk look full until the process restarted. The narrow race itself, where the reserve is crossed between check and write, cannot be driven deterministically without a test hook into free-space measurement; these cover the paths, not the interleaving.
Two holes in the reservation, both found by review. The charge was settled in handle_put while commit moved the write into a blocking task. Drop that future after it spawns and the charge is released while the detached closure goes on to publish the file -- exactly the case the Reservation type says it exists for. It now moves into commit and is settled inside the transaction that owns the write. And settling by observed net growth could under-count, not only over-count as I claimed. The check read the index before the rename; a file the index claims can be gone, so what looked like a replacement grows the disk after all, and the charge was released for bytes that landed. Every write that lands now commits its charge. Over-counting is the safe direction and the next measurement corrects it. Staged bytes that cannot be removed keep their charge rather than releasing it, since they are still on the disk. The test that claimed to cover the release path could not reach it -- Unchanged and Stale return from inspect, before any reservation is taken -- so it is replaced by one at the store level that loses a commit race, which is the reachable way to hold a charge and write nothing. inspect, verify and commit are pub(crate) now: they are the request path's internal steps, put_bytes is the public one, and a public method cannot take a crate-private reservation anyway.
Third site of the same hole. A failed write or fsync leaves a partial temporary file; if removing it also fails, the bytes are still on the disk while the charge for them was being released. stage now reports whether it left anything behind, and the caller settles the charge when it did. The reservation tests now assert on the capacity guard's own counters rather than on a following reservation succeeding -- which it would, stranded charge or not, on a disk with room. A losing commit race must leave written_since and in_flight exactly where it found them; a write that lands must move its bytes from one to the other; and eight concurrent creations must each be charged with none left hanging. That needs a test-only view of the two counters, which FileStore and ChunkStore now expose under cfg(test). The full-disk test's comment claimed to cover the reservation. It does not -- it refuses at the admission check, before payment -- and now says so.
A record rounds up to the allocation unit, so a charge is 12,288 bytes for 5,303 bytes of pointer. The eight-write assertion compared against eight raw record sizes -- which four charges already exceed -- so it would have passed with half the writes uncharged. The single-write and replacement checks were inequalities for the same reason. All three now assert exact deltas against the charge itself, exposed under cfg(test) beside the counters. The stage doc also said every failure removes the temporary file, one line above the field that reports when it could not.
main released 0.20.0 and moved off the git patches: ant-protocol 3.0.0, saorsa-core 0.28.0, saorsa-pqc 0.5.2, evmlib 0.10.0 and saorsa-transport 0.37.0 all come from crates.io now, and [patch.crates-io] is gone. The pointer work needs exactly one thing back: ant-protocol carries the Pointer record and its wire messages, and the published 3.0.0 does not have them. So the patch section returns with that single entry, pinned by rev to the pointer protocol branch at 3.1.0 -- which satisfies the 3.0.0 requirement main declares, being additive on top of it. Everything else stays on the release. No source change was needed: the node compiles against saorsa-core 0.28 and the record still signs and verifies under saorsa-pqc 0.5.2.
grumbach
force-pushed
the
feat/pointers-immutable-owner
branch
from
September 24, 2026 05:39
be63fed to
9de2bf2
Compare
A node now takes any paid record the merge rule prefers to what it holds, whatever its counter, including a first record above 0 at an address it knows nothing about. The +1 gate is gone. It was the one thing that stopped a node catching up. A write ends once five of the seven close-group peers answer and the rest are cancelled, so a peer can miss an update; a peer that joins the group later holds nothing. Under +1 each refused every later update for good, and three such peers left no write able to reach its quorum -- after the client had paid. One guard stays: a record this node lost is still remembered, so it takes that state or anything newer back, and refuses anything older. A replay cannot roll a node back just because its file went missing. A record that loses is now answered Stale rather than PaymentRequired, which is what it is. Pins ant-protocol to 4412b6a, which drops is_paid_update_of and is_genesis. ADR-0016 is updated to match: the counter orders states rather than metering them, and a joining or lagging node is brought level by the next update.
A pointer's copies used to be exactly the ones the client wrote: a member the write missed, a node that joined the group later, or one that lost the file stayed without the current state until the owner's next update. Now pointers replicate through the same engine as chunks -- the same close groups, sync rounds, churn triggers, quorum, pruning and possession rules -- but by state rather than by key, since a pointer's address is stable while its state changes and replicas may hold different signatures of one state. - Fresh: the node that accepts a paid state from a client offers the record with its payment proof to the rest of the close group; each receiver checks signature, responsibility and payment itself before storing. - Repair: every sync round pushes hints (the states the receiver should hold) both ways. A receiver lacking a state, or holding an older one, asks the group which state each holds, adopts the best one a quorum hold exactly -- counted over the whole group, so a silent peer is never a vote -- and fetches it from a holder. - Pruning: after the hysteresis a record out of range is deleted, at once if the node is far outside the group, otherwise only when all but one of the current group return a valid record at that state or newer. - Possession: minutes after an offer, each member must produce the record or is penalised, as a chunk holder is. Six appended replication variants carry this. Requests only go to peers that have sent a pointer message, and hints go out every round even when empty, so an older peer is never asked what it cannot decode and never penalised for silence. The store gains what this needs: sorted enumeration, delete with the disk charge credited back, stats, 256 shards by last byte as the chunk store keeps, a structure-only startup scan (reads verify anyway), and the chunk store's Windows rename retry. Tests: nine multi-node E2E cases over real QUIC (fresh reach, update replacement, repair of a lagging node, a late joiner via the engine's own loops, a lone state refused though its hints arrived, an unpaid offer refused, possession penalising only the dropper, pruning with and without proofs), the verdict function, and the store additions. The existing chunk replication E2E suite passes unchanged (46/46).
A node now commits to the pointers it is responsible for in the same signed storage commitment as its chunks, so they count toward its quoted price and are spot-checked by the subtree audit. - A pointer leaf is (address, pointer_leaf_hash(address)): the root binds which pointers are held, not their state, so an update never moves it. - Round 1 reports a pointer leaf at the fixed record length; a node that lost one refuses, a confirmed failure. The auditor accepts that leaf shape only at exactly a pointer's length. - Round 2 proves a pointer with its whole signed record, which the auditor verifies and checks belongs at the committed address. - The subtree audit protocol id moves to v2 for the new slice item. - Retention persists which leaves are pointers (format 2), written only when some slot holds one, so a rollback still reloads format 1. - Pruning never deletes a pointer a retained commitment still holds.
Browser clients (ADR-0015) can now read and write pointers: - The WebRTC-direct listener admits a pointer GET and a paid pointer PUT through chunk_protocol, each bounded by the small response size; a full record fits well inside it. - Every pointer reply passes the sanitizer; errors and refused payments are redacted as for chunks. - HELLO advertises pointer_protocol, so a browser never asks a node that would refuse. Telemetry parity with chunks: pointer_put_rpc and pointer_get_rpc latency events on the rpc_latency target, the disk pre-check refusal on the disk_precheck target, and the pointer count on commitment rotation.
From an adversarial review of the pointer audit and browser work: - A pointer audit could be passed by a node holding nothing, fetching the few sampled records once round 2 named them. Round 1 now binds a nonced root over each pointer record's bytes, as a chunk leaf does, and round 2 must reproduce it; another replica's copy of the same state does not. The store keeps the record each update replaced for five minutes (bounded), and round 2 serves it beside the current one, so an owner updating mid-audit does not fail an honest holder. - Retention wrote a new format whenever a slot held a pointer, which a rolled-back node could not read at all. The retention file is back to the pre-pointer format; pointer leaves go in a sidecar keyed by commitment hash, so an older release keeps every chunk-only commitment. - HELLO advertised pointer_protocol even on a node with no pointer store.
A node already retries its own port, which clears a socket still being released. It cannot clear a port the host will not hand out at all, as happens on Windows runners, where "Failed to create transport" for node 0 has failed runs on main and on this branch in whichever test drew that range. More tests starting networks means more draws, so the harness now moves a network that cannot create a node to a fresh random range, up to three times, and a test holds a port to prove it does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear issue
Closes V2-1277
Risk tier
A new stored record type with its own payment identity, replicated, committed, audited and priced like a chunk.
Compatibility
ChunkMessageBody. Six pointer replication messages are appended to the replication enum, so every earlier discriminant keeps its value; an older peer cannot decode them, so a node only ever asks peers that have sent it a pointer message, and never penalises silence.v2, because round 2 carries a new slice item for pointers. As with ADR-0009, old and new releases do not audit each other across the upgrade.{root}/pointers/<shard>/. The chunk store is untouched, and a chunk-only node's commitment root does not move. The persisted commitment retention keeps its format; which leaves are pointers is written to a sidecar beside it.chunk_protocol, and HELLO advertisespointer_protocolwhen the node has a pointer store.Semver impact
Test evidence
cargo test --lib --all-features: 1227 passed.cargo test --test pointer_convergence --test poc_audit_handler_live --test poc_commitment_audit_attacks: 15, 16 and 19 passed.--test e2e):pointer_replication12 passed;replication49 passed;subtree_auditandfirst_auditpassed.cargo clippy --all-targets --all-features -- -D warningsandcargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used: clean.cargo fmt --all -- --check,RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-depsandscripts/adr-governance.py: all clean.ant-devnetbuilt from this branch, with on-chain payment on Anvil (the ant-client browser suite): a pointer is created, updated, pointed at a second pointer, read back by a client that wrote nothing, and resolved to its chunk.Adversarial coverage:
u64::MAX; anything older is refused as stale.Known pre-existing failure, unrelated and not caused by this branch:
poc_price_floor_live::enforced_floor_rejects_cheapest_of_k_real_settlementfails identically on an untouched checkout ofmain.New dependency
None.
ADR
https://github.com/WithAutonomi/ant-node/blob/feat/pointers-immutable-owner/docs/adr/ADR-0016-pointers-immutable-owner.md —
docs/adr/ADR-0016-pointers-immutable-owner.md, added by this PR.Mitigation / rollback
Pointers are opt-in at construction: a node built without a
PointerServicerefuses pointer messages cleanly and does not advertise them to browsers. A rollback keeps every chunk-only commitment answerable: the retention file keeps its old format, and only commitments that include pointers, which the older release could not answer anyway, are dropped. The subtree audit id change means old and new releases stop auditing each other until the upgrade is through; that pause is the only coordinated part of the rollout.