Repository navigation
Views step G: sealed base donors and call-graph routing through views - #1026
Open
aft-alfonso[bot] wants to merge 6 commits into
Open
aft-alfonso[bot] wants to merge 6 commits into
aft-alfonso[bot] wants to merge 6 commits into
Conversation
A sealed donor is a private, fully built derived graph switched to journal_mode=DELETE, checkpointed and synced, with no -wal/-shm/-journal sibling, plus an atomically published <db>-seal record (fingerprint, size, journal mode). Readers open sealed files with immutable=1; IdentityConnection refuses write opens while the seal record exists. clone_sealed verifies the seal and copies into a fresh, never-reused name with clonefile (APFS), FICLONE (Linux) or a logged plain copy; a live (unsealed) database is refused. The generation sweep reclaims seal records with their database. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit 4da050bd3d2a68effe2d0e2aa90ff7bbede9b7d5)
…one sealed donors for workers Fresh v3 bootstrap now accepts planes that complete their own base entries (CompositePlane::completes_fresh_base / complete_v3_base). CallgraphBridge completes a fresh base from the bootstrap's strict-reconciliation attachments under the base build claim (taking over a dead builder's claim, refusing a live one), publishes a readiness successor whose derived graph is sealed as the base's donor, and the claim is released after the worker publishes. Semantic producers and other nonlexical planes stay refused. Base entries now carry resolution_input before the base identity is computed (base_builder::base_entry, used by build_base and the bootstrap's expected identity), and with a registered graph producer only call-graph inputs start pending, so a completed base projects. A v3 worker graph is a fresh clone of the base's sealed donor patched with the manifest diff (apply_manifest_diff_from_blob_reader); a cold build happens only for a reason from donor::ColdReason and is recorded. Fix: deleting a file a worker added (absent from the base) produced a tombstone for a non-inherited path, which ComposedGeneration::compose refuses; such paths are now dropped from the delta without a tombstone. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit 27584ea4496e88dbacc4d889b98e8413f468c9b8)
With views on (and the call graph enabled, a heavy-work root and an available cache), callgraph_store_for_ops and its no-wait variant acquire a reader from a CallgraphView: a v3-only CheckoutDriver with the lexical and call-graph planes under its own member scope (<scope>.callgraph). Each acquisition verified-binds, publishes pending edits as a new worker generation before its single install, refuses uncertain snapshots (pending intent, unhealthy watcher, unpublished changes) and never falls back to the legacy store or a v1 view. The reader is pinned once per pinned request. Status, zoom, inspect's graph-backed categories and the borrowed-graph disclosure follow the route; warming never binds on the transport loop. An explicitly installed CheckoutQueryRuntime keeps precedence. Views off is unchanged. Fix (views-on): view-materialized graphs store the call-graph blob key as a file's content hash, so source-line freshness checks marked every view answer stale (approximate, no call_expression). The adapter now also accepts the current bytes' derived blob key. Parity tests compare callers/impact/call_tree/trace_to output with a cold legacy store over the same files on a plain clone and a linked worktree, across edits, a deletion, an overflow and a restart; a refused acquisition never answers from a resident legacy store. Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com> (cherry picked from commit 57a9c36ab4d1a796f1146c43c1af6c7c23607ac8)
With views on, every call-graph read now goes through the checkout's v3
view, so tests that pinned a v1 view and asserted the old route's answers
are ported to the same contracts on the new route, or retired when the old
route itself was their subject:
Ported (contract kept, assertions as strict):
- views_keyed_generation_{answers_building,zoom_reports_building,
inspect_reports_pending}_while_derived_store_is_busy: a SQLite lock on the
view database opening is reported as busy/building/pending, never as an
absent graph. The view route now carries a typed busy flag
(CallgraphViewError) and inspect observes it from the request's access.
- views_pending_head_callers_... -> views_changed_head_callers_answer_the_
current_tree_not_the_stale_legacy_store: never serve stale rows.
- views_publication_skips_a_non_utf8_callgraph_source_and_completes: now
also passes on the view route because call-graph attachment gives a
non-UTF-8 source an empty graph, as checkout assembly does (fix).
- publication_build_does_not_delay_same_root_bind_and_read: the read still
completes while a publication is held, and answers the current tree.
- callgraph_worktree_published_owner_assembles_its_own_changed_source and
callgraph_unindexed_path_differs_from_missing_symbol: a worktree answers
from its own source at once; a new file is indexed, not 'not_in_generation'.
- scoped_inspect_views_worktree_reports_checkout_only_dead_function: an edit
after binding is reflected rather than reported unavailable.
- views_removed_callgraph_blob_store_and_view_rebuild_on_restart: the v1
checkpoint-contention section is dropped (the view route no longer reads
the v1 database); rebuild, parity and edit checks remain.
- blob_removal_stop waits (bounded) for v3 session-end/checkpoint threads.
Added: views_callgraph_index_off_answers_off_without_binding_a_view.
Retired (subject was the v1 route):
- views_keyless_generation_answers_disabled_while_derived_store_is_busy and
per_checkout_9::callgraph_ops_report_a_keyless_view_generation_as_disabled:
a v1 generation published with the call graph off; v3 keyless refusal
stays covered by per_checkout_callgraph keyless tests.
- generation_sweep_waits_for_the_last_query_handle: v1 QueryPin vs v1 sweep;
the v3 equivalent (a held reader keeps its generation) is asserted in
views::callgraph::tests.
- inspect_view_pending_reply_names_publication_not_changed_sources: wording
of the v1 'view pending' reply.
- inspect_views_owner_persists_and_reuses_tier2_contributions: the
writer-backed reuse pipeline read the v1 published view; v3-routed dead
code is projected per request from the pinned view (no contribution reuse).
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit 7773c24d17f3c4e26c12a9a038a524a0f9371857)
…out view route
The on-demand worktree tests asserted the v1 route: answering from the
main checkout's published graph, reporting building until a lazy v1 build
finished. On the view route a worktree always answers at once from its own
v3 view (the shared base's sealed donor avoids rebuilding it).
Ported: linked_worktree_{with_its_own_commit_is_never_answered_from_main_graph,
stops_answering_from_main_graph_after_main_moves, with_a_graph_from_an_earlier_
process_defers_its_next_publication, defers_and_builds_on_first_query_when_main_
is_not_bound} now assert an immediate answer from the worktree's own view
equal to a cold build (lazy_query_builds_own_graph); edit_reports_building_then_
answers_from_its_own_graph -> linked_worktree_edit_answers_from_its_own_view_at_
once; concurrent_first_queries_start_one_call_graph_build asserts both answer
and one sealed base build; main_checkout_still_builds_its_call_graph_at_bind
and busy_readiness_defers_revalidation_without_a_cold_rebuild keep their v1
publication assertions and read the view route; views_restart_dirty_source_
reports_working_tree_refresh_until_publication -> views_restart_dirty_source_
answers_the_working_tree_at_once.
Retired (the old route itself): linked_worktree_answers_from_main_graph_
without_building_its_own and linked_worktree_waits_for_a_locked_main_graph_
instead_of_building, with their unused helpers (lazy_lock_derived,
lazy_wait_for_own_graph, inspect current_view_projection_store_for_test).
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
(cherry picked from commit dbdd5480bdc6af8a67e95042ee8f88b4ad8cf605)
Co-authored-by: alfonso-aft <289616620+alfonso-aft@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Views step G (slices 5-2 and 7b), stacked on step F (#1017). Views stay off by default; views-off behaviour is unchanged.
What it does. With views on, every call-graph read goes through the checkout's v3 view: a v3-only checkout driver per plane, verified bind, pending edits published as a new worker generation before the read, uncertain snapshots refused, and no fallback to the legacy store or a v1 view. A fresh checkout builds its base graph once through the base builder and seals it as a donor; every other checkout of the same bytes clones the sealed donor and patches only its own changes, so a second checkout parses nothing and builds no graph cold.
Bugs found and fixed on the way. A worker-added file deleted again produced a tombstone for a path the base never had, and compose refused it. View-materialized graphs stored the blob key as the file's content hash, so source-line freshness checks marked every views answer stale (this affected the old v1 views route too). Call-graph attachment failed a whole reconcile on a non-UTF-8 source; it now gives an empty graph, as assembly does.
Tests. Main test: sealed_base_donor_serves_checkouts_without_reextraction_or_cold_builds (extraction counts, donor events, the donor's fingerprint unchanged by clones, a held reader keeping its generation, restart reopening without parsing, no blobs/v2). Views-on tests that pinned the old route are ported to the same contracts on the new route; tests whose subject was the old route itself are retired with reasons in the commit messages (7773c24d17, dbdd5480bd).
Deferred. Stopping the old call-graph publication on routed roots moves to step H (dispatched). Measuring read-path cost, tier-2 dead-code reuse on views-on, crash tests at each donor phase, and edit/branch-switch parity go to step I.
Verified on the build server: warnings-as-errors compile of all targets; views::callgraph, views::donor and views::first_load (59 passed); integration callgraph_worktree and per_checkout_callgraph (14 passed).