Skip to content

#891: route binds read config off the frame loop and acknowledge on their own lane - #984

Merged
ualtinok merged 13 commits into
mainfrom
land/frame-loop-a0
Oct 27, 2026
Merged

ualtinok merged 13 commits into
mainfrom
land/frame-loop-a0

Conversation

@aft-alfonso

@aft-alfonso aft-alfonso Bot commented Oct 26, 2026

Copy link
Copy Markdown

Part of #891 (the daemon frame loop stalling on disk). Route binds now read project config once at preflight and reuse it, bind acknowledgements go out on a dedicated control lane before any route traffic, and requests that arrive before a bind is acknowledged wait for it instead of failing with route_not_bound. Too many waiting binds refuse with busy ("too many requests are waiting for a route bind; retry shortly"). This does not close #891: moving maintenance submission and the remaining blocking lookups off the loop is later work.

ualtinok added a commit that referenced this pull request Oct 26, 2026
#984)

Reproduced the merged checkpoint on Linux: conformance24 and scope7 pass; bridge130 pass,5 fail,2 ignored. Preserve bounded latest status while an ACK is pending so a newly published sibling gets the coalesced snapshot exactly once. Construct the large loop future in a non-inlined heap-allocation boundary: Windows CI logs show successful bind/status followed by native main stack overflow.

Restore generic lifecycle observers' prior event stream with explicit ACK-queue observation for the queued-ACK fixture. Run real configure in the S1 guard fixture, use a supported page harness, and hold configure's actual database in the confinement test. Observe flood pressure before releasing configure rather than requiring the raw ACK lane to lose a scheduling race. No immediate-route_not_bound contract assertion was reversed in this follow-up; the flood pressure and ordering assertions remain.

Committed before the requested single filtered Linux2c verification job. No mutation checks or full-suite gate. The final result will record that job's exact outcomes, including small-native-stack coverage and the exact storm regression.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 806ed7a68d91b77cb665467b5efb4bf256be4d08)
ualtinok added a commit that referenced this pull request Oct 26, 2026
…984)

The filtered Linux verification passed conformance24 and scope7 under a1MiB native stack, the storm regression, and133 bridge tests. Two fixtures still raced setup: taking the real DB before scope-bind settled blocked unrelated registration, and observing flood pressure after arming a turn hold could deadlock before the flood was drained.

Fence the scope-bind mutating lane after a control round-trip, then hold the configured DB. Observe the first flood frame before arming the hold and use an explicit push to reach the held turn. Keep the2s health bound, pressure assertion, exact flood count and ACK/push ordering assertions. Re-run only these two remaining failures; no mutation checks or full-suite gate.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit f59ae6824b93dc3fe9d26bba7cf3e169e12565a7)
@ualtinok
ualtinok added this pull request to the merge queue Oct 26, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 26, 2026
ualtinok added a commit that referenced this pull request Oct 26, 2026
…984)

Keep both assertions unchanged, but print the first control frame's type/body when health is not correlation40 and enable the existing durable-ledger warning for the exact codemode regression. These diagnostics distinguish an early bind Error from an ACK and a failed ledger write from a missing/mismatched terminal row.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 17287f215abfdefd1dd8eb00831542e7dfbe1a2d)
ualtinok added a commit that referenced this pull request Oct 26, 2026
…storage (#984)

The requested20-run Linux comparisons were main20/20 for both tests, main+#984 bind19/20 and ledger20/20. Unchanged-assertion diagnostics then captured7/20 bind refusals: correlation20 carried config_divergence / invalid_config_document, not a missing acknowledgement. Adjacent binds rewrote the same live JSON file while prepared config read it. Use sibling temporary files and atomic replacement in both bind publishers.

Per the parent decision, limit storage changes to the serialized fixture. Select its private directory via AFT_STORAGE_DIR unless env_setup explicitly controls that variable, and assert both actor configs and SQLite's main file remain there. Production actor storage selection and recovery semantics are unchanged; the pre-existing module/actor storage propagation mismatch is documented separately as a migration question. All original health, acknowledgement and ledger-shell assertions remain.

The fixture-isolation diagnostic does not claim that the reported rare ledger settlement error was reproduced: it passed20/20 on main and two20-run merged loops. Capture the existing ledger warning if it recurs. No mutation checks or full-suite gate; repeated exact tests follow this committed fix.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit be748fd6ff85ff3d6885f66f41048be9d7118a12)
@ualtinok
ualtinok added this pull request to the merge queue Oct 26, 2026
@ualtinok
ualtinok removed this pull request from the merge queue due to a manual request Oct 27, 2026
@ualtinok
ualtinok added this pull request to the merge queue Oct 27, 2026
@ualtinok
ualtinok removed this pull request from the merge queue due to a manual request Oct 27, 2026
@ualtinok
ualtinok added this pull request to the merge queue Oct 27, 2026
ualtinok and others added 4 commits October 27, 2026 03:44
Capture user/project observations in the selected harness's order, rejecting unknown or oversized reads instead of selecting a lower-priority file. Migration rereads use the same byte limit. Pass the immutable local snapshot through a scoped internal binding on every repeatable configure attempt; wire JSON cannot supply that scope.

Prepared configure reuses content, selected source paths, migration outcome and legacy notice without file reselection. Normal configure and cache recovery remain separate. Correct the comments that conflated AFT's retained call records with SUBC preserving a client route handle.

The focused two-core compiler check, two prepared-bind tests and thirteen local-reader tests pass. Independent restored-baseline/source/notice controls are pending remote admission and will be reported with this review chunk. Config observation is still at the existing bind admission site until the later ticket/preflight mechanism; this dependency commit is not a disk-free-loop claim.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Keep control and management replies off the native call-ledger proxy. Split configure-ready from ACK-written publication: route maps, push replay and candidate dispatch remain unavailable until the actual writer finishes the ACK. Reserve completion capacity before exposing bytes so a fast peer cannot race an asynchronously scheduled publication event.

Queue the ACK synchronously when writer capacity is available to preserve pre-turn bind priority. Bound outstanding publication waiters, use the existing actor_not_ready retry code on saturation, retract only ACKs not yet writing, and cancel queued publication at loop exit. Native tool replies and teardown still use their ledger path.

Two-core focused checks pass: native compiler and fmt, six egress unit tests, the strengthened ACK-order test, scope-workdir and bind-log fixtures, plus management, reliable-flood priority, pre-ACK refusal, Goodbye-cancel and duplicate-bind integration tests. Early-receipt and post-proxy-sender deliberate breaks each redden only their named egress regression. Three independent prepared-config baseline/source/notice controls also redden only the snapshot regression; source was restored between controls and all positive filters passed.

This is the second dependency mechanism, not completed A0: owners, intake tickets/deadlines, incumbent retirement and narrowed unknown-root guards follow after this chunk is reviewed. Existing inline lifecycle/finalizer/config-observation waits are not claimed eliminated.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Carry the same private prepared-config Arc through provisional identity replay and record the snapshot actually applied by configure before route publication. Keep the config read bounded at the I/O boundary, including locked migration rereads, and test nested scope restoration plus real captured-source watch attachment. Configure fixtures now execute configure rather than inventing a successful response with no applied witness.

Linux RLIMIT_NOFILE=4096: cargo fmt --check, cargo check --tests --bins, prepared_bind_ (6), subc_config:: (16), config_fix:: (25), config_live::tests:: (46) passed. Independent controls reddened only their named test: discard replay snapshot; remove production scope; remove publication witness; restore None rather than nested scope; use read-all migration; reselect watched source; remove read-time bound. Each job restored source bytes before pristine green filters. See the carried investigation for exact output and earlier content/source/notice controls.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
#891)

Reserve bind receipt capacity before configure side effects. Replace retracted queued ACKs in place with one existing cancelled, bind_superseded or module_reloading control error, and hold candidate requests until the bind ACK completes. Bound held frames and wire bytes, preserve failure-order replies, and never execute a cancelled held call. Receipt slots now cover waiter lifetime; configure-ready messages cannot deadlock against ACK-reserved completion slots.

The approved protocol change intentionally replaces the old pre-ACK route_not_bound assertion with hold-then-serve, and zero-terminal cancelled-bind expectations with one error. Already closed stale route generations remain silent. ACK write-start wins success and may finish after its deadline; queued refusals name their actual wait stage. The separate busy text names route binds rather than storage.

This is a preserved review step, not final delivery: 42 recorded controls include 41 sole-test reds plus one initially surviving live receipt control, now defended by direct consumption tests. Linux and Windows GNU compile checks passed before the latest test-only RESERVED deadline extension. Compatibility follow-up remains: the broad tool-result fixture hit a loaded-host watchdog, and the existing long bind-deadline integration timed out. Fixture index startup and deadline diagnostics are pending build-server capacity; see the carried investigation and exact proof artifacts. Do not read this intermediate commit as all final gates passing.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
@ualtinok
ualtinok removed this pull request from the merge queue due to a manual request Oct 27, 2026
ualtinok and others added 9 commits October 27, 2026 03:45
The snapshot-witness path now runs real configure in transport tests. Give those fixtures an explicit user config with indexes and views disabled rather than an absent file resolving to enabled index defaults. Retain the AST actor and assert the applied switches stay off; no production reply or index behavior changes.

Committed at the parent's explicit checkpoint request. These small test-only changes and the RESERVED deadline-unit extension have not been compiled or executed: six Linux2/4-core submissions were refused for more than73 minutes, including the focused finish job. Earlier Linux and Windows compile checks passed, but the existing bind-deadline recovery integration timed out and a tool-result run watchdog-exited on a starved host. Those are recorded as failures, not passes. The PR's three-platform CI will compile and test this checkpoint; no new chunks, mutation checks or full-suite worker gates were started after the process change.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
#984)

Reproduced the merged checkpoint on Linux: conformance24 and scope7 pass; bridge130 pass,5 fail,2 ignored. Preserve bounded latest status while an ACK is pending so a newly published sibling gets the coalesced snapshot exactly once. Construct the large loop future in a non-inlined heap-allocation boundary: Windows CI logs show successful bind/status followed by native main stack overflow.

Restore generic lifecycle observers' prior event stream with explicit ACK-queue observation for the queued-ACK fixture. Run real configure in the S1 guard fixture, use a supported page harness, and hold configure's actual database in the confinement test. Observe flood pressure before releasing configure rather than requiring the raw ACK lane to lose a scheduling race. No immediate-route_not_bound contract assertion was reversed in this follow-up; the flood pressure and ordering assertions remain.

Committed before the requested single filtered Linux2c verification job. No mutation checks or full-suite gate. The final result will record that job's exact outcomes, including small-native-stack coverage and the exact storm regression.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…984)

The filtered Linux verification passed conformance24 and scope7 under a1MiB native stack, the storm regression, and133 bridge tests. Two fixtures still raced setup: taking the real DB before scope-bind settled blocked unrelated registration, and observing flood pressure after arming a turn hold could deadlock before the flood was drained.

Fence the scope-bind mutating lane after a control round-trip, then hold the configured DB. Observe the first flood frame before arming the hold and use an explicit push to reach the held turn. Keep the2s health bound, pressure assertion, exact flood count and ACK/push ordering assertions. Re-run only these two remaining failures; no mutation checks or full-suite gate.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…984)

Keep both assertions unchanged, but print the first control frame's type/body when health is not correlation40 and enable the existing durable-ledger warning for the exact codemode regression. These diagnostics distinguish an early bind Error from an ACK and a failed ledger write from a missing/mismatched terminal row.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…storage (#984)

The requested20-run Linux comparisons were main20/20 for both tests, main+#984 bind19/20 and ledger20/20. Unchanged-assertion diagnostics then captured7/20 bind refusals: correlation20 carried config_divergence / invalid_config_document, not a missing acknowledgement. Adjacent binds rewrote the same live JSON file while prepared config read it. Use sibling temporary files and atomic replacement in both bind publishers.

Per the parent decision, limit storage changes to the serialized fixture. Select its private directory via AFT_STORAGE_DIR unless env_setup explicitly controls that variable, and assert both actor configs and SQLite's main file remain there. Production actor storage selection and recovery semantics are unchanged; the pre-existing module/actor storage propagation mismatch is documented separately as a migration question. All original health, acknowledgement and ledger-shell assertions remain.

The fixture-isolation diagnostic does not claim that the reported rare ledger settlement error was reproduced: it passed20/20 on main and two20-run merged loops. Capture the existing ledger warning if it recurs. No mutation checks or full-suite gate; repeated exact tests follow this committed fix.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
The failing drain-close artifact contains the loop's exit0 classification but ends before the binary's final shutdown record. Durable-file flushing does not acknowledge the separate best-effort stderr queue. Add a process-exit-only acknowledged fence, sharing the error-exit budget and using the remaining graceful-exit budget. Ordinary logger writes/flushes remain nonblocking; no stderr I/O lock is taken by the producer.

The strict descriptor census reproduced on main1a655 plus the rebased PR: ast_t6_route retains eventfd/eventpoll transiently. Add failure-only thread names without changing the census predicate or exempting descriptors so the owning teardown can be identified. The A0 receipt lane spawns tasks on the existing module runtime, not one runtime per route; fixture/product classification remains under diagnosis. No mutation checks or full-suite gate.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
…own (#984)

Real configure introduced deferred project-watcher startup into fixtures which only test result transport. notify8.2 drops its public watcher by signalling a detached mio event loop, so returning from the fixture or joining AFT's filter is not a barrier for its eventfd/eventpoll closure. The strict census reproduced the temporary pair and later returned to baseline.

Use a scoped watcher-disabled setting under the existing reentrant process-env lock for the AST and capability fixtures, and assert AST never starts a project watcher. The setting is restored before unlocking. No census predicate, allowance, anon_inode exemption or timeout changed; module runtimes and reader/writer tasks retain their normal joined teardown. A0 owns no runtime per route. The exact census will be verified again with stronger repetitions; no mutation or full-suite gate.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
@ualtinok
ualtinok added this pull request to the merge queue Oct 27, 2026
@ualtinok
ualtinok removed this pull request from the merge queue due to a manual request Oct 27, 2026
@ualtinok
ualtinok added this pull request to the merge queue Oct 27, 2026
Merged via the queue into main with commit 65047cb Oct 27, 2026
91 checks passed
@ualtinok
ualtinok deleted the land/frame-loop-a0 branch October 27, 2026 06:03
ualtinok added a commit that referenced this pull request Oct 27, 2026
…ed by #984

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
jacobocasado pushed a commit to starredjaco/aft that referenced this pull request Oct 27, 2026
Merge-queue commits carry committer dates out of order. git rev-list
base..head walks by date and stopped early, so a queue group counted main's
own cortexkit#984 merge commit as a commit of cortexkit#1034 and refused it for a missing
co-author trailer. Commit ranges are now computed from full ancestry sets, and
the queue's first-parent walk stops exactly at the group base.

Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Disk read load stalls the subc frame loop for 30 s; the watchdog exits and every client loses its calls (two incidents on v0.62.0)

1 participant