Skip to content

fix: a list or watch pinned to one name agrees with Get before the node publishes - #46

Merged
CMGS merged 5 commits into
masterfrom
fix/name-pinned-reads
Sep 25, 2026
Merged

CMGS merged 5 commits into
masterfrom
fix/name-pinned-reads

Conversation

@CMGS

@CMGS CMGS commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem

kubectl wait --for=delete and kubectl delete read the sandbox with Get, then open a watch-list with fieldSelector=metadata.name=<name>. Get asks the nodes for a claim the NodeInventory does not hold yet. List and Watch read only the inventories, which nodes republish every 30 s. So for a sandbox claimed less than one publish ago, the Get found it, the watch's initial sync was empty, and kubectl's delete precondition reported it deleted at once.

The 2026-09-24 hardware round hit this: kubectl wait --for=delete on a fresh L3 sandbox returned rc 0 in 177 ms, about 1.8 s before the case issued the delete. Before #45 the same wait hung instead, because the watch-list never got its initial-events-end bookmark.

Change

  • A List pinned to one name in a namespace resolves through the same lookup as Get, then applies the label and field selectors.
  • A Watch with that pin takes its initial events from that list.
  • On each poll, before a pinned watch reports a known entry deleted, it asks the entry's node. It keeps the entry while the node still holds the claim and the claim still matches the selectors.
  • An absent name polls the inventories only, so the pin adds no node traffic.
  • Fleet lists and watches are unchanged and stay eventually consistent.

A node that does not answer within 500 ms counts as a miss, the rule Get already follows. A pinned watch therefore agrees with Get in that case too.

Cost

  • The claim path is unchanged.
  • A pinned list costs one Get.
  • A pinned watch asks at most one node per tick, and only while a known entry is absent from the published inventory. That window runs from the claim to the first publish, plus the tick that sees a delete.

Tests

pkg/scale/sandboxstore_live_test.go:

  • A pinned list finds an unpublished claim. It honours a label selector, and the fleet view and a name across every namespace stay inventory-only.
  • A pinned watch's initial events carry an unpublished claim. It keeps the claim while the node holds it and reports it deleted once the node releases it or it leaves the selector.
  • A pinned watch on an absent name makes no node request over ten ticks.

The first two fail on master. The selector case fails when the selector check on the live answer is removed.

Gates, all with GOWORK=off:

  • make fmt-check
  • make lint: 6 × 0 issues.
  • make test
  • asl -forwarder=false ./... on darwin and linux

Docs

usage.md, lifecycle.md, index.md and scaling-design.md now state that a list or watch pinned to one name in a namespace reads like Get when it opens. A claim made after the watch opened still appears at the next publish.

Commits

  • 578feef the fix.
  • 54fab65 review: interface-implementing store methods drop the godoc the SandboxStore and ClaimIDResolver interfaces already carry.
  • 284cc0a review: synthAnnotations, lifecycle.md, usage.md and the lifecycle example no longer say a name read waits for a publish, and scaling-design.md adds that a pinned list or watch for a name no node holds asks every node once when it opens.
  • 7285d60 review: sandboxstore.go and sandboxstore_impl.go meet the comment budget, one fact per line, dropping from 82 and 146 comment lines to 32 and 59.
  • e88c7df review: objKey goes through namespacedName, and TestASilentNodeBoundsAMiss runs under synctest.

Hardware verification

Verified on 2026-09-25 at 54fab65 on two bare-metal hosts, as an A/B pair against the same live stack. SBL-08 fails on master and passes on this branch; the corrected SBL-06b passes on both. Results: #46 (comment)
The later commits change comments, docs and tests; objKey returns the same string as before.

Not in this change

A pinned watch still re-derives the whole fleet's inventories on every tick to follow one name. #47, stacked on this PR, makes that tick read only the entry's node.

…de publishes

kubectl wait --for=delete and kubectl delete read the sandbox with Get,
then open a watch-list with fieldSelector=metadata.name=<name>. Get asks
the nodes for a claim the NodeInventory does not hold yet; List and Watch
read only the inventories. So a sandbox claimed before its node published
(up to 30 s) was found by the Get, missing from the watch's initial sync,
and reported deleted at once. Before the watch-list bookmark fix the same
wait hung instead.

A list pinned to one name in a namespace now resolves through Get. A
watch with that pin starts from it, and before it reports a known entry
deleted it asks the entry's node, keeping the entry while the node holds
it and it still matches the selectors. An absent name polls the
inventories only, so the pin adds no node traffic. Fleet lists and
watches are unchanged.
The SandboxStore and ClaimIDResolver interfaces, WithClaimRouting and WithWatchPollInterval already state every fact these five godocs repeated; the lifecycle verbs in the same package carry none (Comment Style: interface-implementing methods omit godoc).
@CMGS

CMGS commented Sep 25, 2026

Copy link
Copy Markdown
Contributor Author

Hardware verification on 2026-09-25, on two bare-metal hosts with the two-host E2E kit. Every kit binary, the firmware and the running operator, webhook and vk-cocoon processes were checked by sha256 before and after each lane.

Both arms ran against the same live stack. Only sandbox-apiserver changed between them: master d14d801 (sha256 ff7cc749), then this PR at 54fab65 (sha256 c77d74c6), then master restored. Each arm restarted the apiserver and checked the sha256 of its /proc/<pid>/exe before the cases ran. vk-sandbox published its NodeInventory every 5 s, so each case allows the delete plus 20 s.

case master d14d801 this PR 54fab65
SBL-08: kubectl wait --for=delete opened about 70 ms after the create, before the node published the sandbox FAIL. The wait returned 179 ms after it opened with condition met, while the sandbox was live. kubectl logged Exiting watch because received the bookmark that marks the end of initial events stream first. PASS. The wait held while the sandbox was live, and returned 0.94 s after the delete.
SBL-06b: the same wait, opened after kubectl get sandboxes lists the sandbox PASS. Returned 3.08 s after the delete. PASS. Returned 3.09 s after the delete.

After the PR arm, master was reinstalled and the version gate passed again.

Report and evidence: cocoonstack/cocoon-specs tests/2026-09-25-two-host-rerun.md at fb74e17, with one evidence file per case under tests/2026-09-25-two-host-rerun-evidence/.

…de publishes

A read by name now takes the claim id, deadline and claim time from the node's own row, so synthAnnotations, lifecycle.md, usage.md and the lifecycle example no longer say these wait for a publish, and the example limits the lag to fleet List and Watch. scaling-design.md adds that a list or watch pinned to a name no node holds asks every node once when it opens.
Multi-line godoc and const comments become one fact per line. Comments that restated a name, a signature, the interface godoc or scaling-design.md are gone: warmCandidate, scatterGatherStore, its claim-routing fields, matchOnNode, parseSelectors, the embedded SandboxLifecycle note, and the watch cost figures the design doc already measures. The NodeInventoryGVK reason moves onto its var entry. The files drop from 82 and 146 comment lines to 32 and 59.
…ilent-node test runs under synctest

heldByNode passes objKey as the claim ref that Claim and lookupName spell with namespacedName, so objKey now calls it. TestASilentNodeBoundsAMiss waited out two real 500 ms timeouts; under synctest they elapse on the fake clock, and dropping the timeout still fails it as a deadlock. unpublishedStore moves below the countingSource type it builds.
@CMGS
CMGS merged commit 34fd93c into master Sep 25, 2026
2 checks passed
@CMGS
CMGS deleted the fix/name-pinned-reads branch September 25, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant