[DJ Semantic Fingerprint 2] Evaluate fingerprints across deployment graphs

[philipfweiss]

Fingerprint construction primitives can compose parent hashes, but deployment code has no server-owned evaluator for the full semantic parent graph. Cyclic dimension links also have no topological starting node, and recursive graph traversal can exceed Python's recursion limit on deep chains.

Depends on #2482.

This PR:

• adds SemanticFingerprintGraph for lazy, memoized evaluation within one graph snapshot
• resolves semantic parents from SQL, dimension links, required dimensions, cube members, and cube filters
• loads external ancestors and computes current and proposed Merkle fingerprints
• requires every concrete node spec type to register its parent resolver
• normalizes source columns and required-dimension identity across authored and persisted specs
• propagates unavailable fingerprints to dependent nodes without blocking deployment
• evaluates cycles with deterministic strongly connected component hashes

Callers construct one graph for each current or proposed snapshot, then use fingerprint(name) or fingerprints(names). Missing parents return unknown, so an incomplete snapshot cannot produce a plausible graph-bound hash.

Why cyclic graphs need SCC hashing

Direct Merkle evaluation cannot compute A, B, or C here because every object hash requires another unfinished object hash:

flowchart LR
    A["dimension A<br/>h_A = H(f_A, h_B)"]
    B["dimension B<br/>h_B = H(f_B, h_C)"]
    C["dimension C<br/>h_C = H(f_C, h_A)"]

    A -->|requires h_B| B
    B -->|requires h_C| C
    C -->|requires h_A| A

    classDef cycle fill:#fee2e2,stroke:#dc2626,color:#450a0a
    class A,B,C cycle

Loading

The evaluator uses an iterative two-pass SCC traversal. Its graph walk is O(V + E), where V is the node count and E is the semantic parent edge count. Deterministic ordering adds sorting overhead. Explicit stacks avoid Python recursion limits on deep dependency chains. Parent discovery, SQL parsing, and database loading occur outside this graph-walk bound.

Each cycle becomes one atomic component in a condensation graph, which is always a DAG:

h_scc = H(
  sorted(member name + intrinsic fingerprint),
  sorted(internal child + parent edges),
  sorted(external child + parent + parent hash)
)

h_A = H(f_A, h_scc)
h_B = H(f_B, h_scc)
h_C = H(f_C, h_scc)

flowchart LR
    P["external parent P<br/>object hash: h_P"]
    SCC["SCC {A, B, C}<br/>component hash: h_scc"]
    A["dimension A<br/>h_A = H(f_A, h_scc)"]
    B["dimension B<br/>h_B = H(f_B, h_scc)"]
    C["dimension C<br/>h_C = H(f_C, h_scc)"]
    D["downstream node D<br/>object hash changes"]

    P -->|external parent hash| SCC
    SCC --> A
    SCC --> B
    SCC --> C
    A --> D
    B --> D
    C --> D

    classDef parent fill:#fff1d6,stroke:#c2410c,color:#431407
    classDef component fill:#ede9fe,stroke:#7c3aed,color:#2e1065
    classDef member fill:#ecfdf5,stroke:#059669,color:#064e3b
    classDef downstream fill:#dbeafe,stroke:#2563eb,color:#172554
    class P parent
    class SCC component
    class A,B,C member
    class D downstream

Loading

Changing a member definition, internal edge, or external parent hash changes h_scc, every member hash, and downstream hashes. Acyclic singleton components keep the ordinary Merkle path. An unavailable member or external parent makes the component and its dependents unavailable.

---

Verification:

1. Checked out the exact PR head and built it in a fresh Docker slot:

git rev-parse HEAD
printf '2\n' > .dev-port
./dev.sh up -d
curl http://localhost:8200/health/

Observed:

de5ed41cd22d6d3ea3cfc18ba94e30157b0a8c85
[{"name":"database","status":"ok"}]

2. Logged in as dj and deployed graph_verification.external_source so the graph evaluator could load a parent from committed database state:

POST /basic/login/                              -> 200
POST /deployments                              -> 200
GET /deployments/{uuid}                        -> status=success

3. Ran SemanticFingerprintGraph inside the backend container with:

docker exec -i dj-s2 /code/.venv/bin/python -

The script exercised these inputs directly against the code at /code/datajunction_server:

• source -> transform -> metric, then changed only the source table
• repeated lookup from one graph snapshot to check memoization
• first -> second -> third -> first plus a downstream node, with reversed input order and a changed cycle member
• a 1,100-node dimension-link chain
• invalid SQL and unresolved-parent nodes with dependents
• authored and persisted forms of the same required dimension
• a proposed transform whose source parent existed only in Postgres

Observed:

{
  "acyclic_propagation": {
    "all_descendants_changed": true,
    "changed": [
      "verify.metric",
      "verify.source",
      "verify.transform"
    ],
    "memoized": true
  },
  "cycle_evaluation": {
    "input_order_stable": true,
    "member_change_propagates": true,
    "members": [
      "cycle_verify.downstream",
      "cycle_verify.first",
      "cycle_verify.second",
      "cycle_verify.third"
    ]
  },
  "deep_chain": {
    "nodes": 1100,
    "target": "deep_verify.node_1099",
    "target_known": true
  },
  "external_ancestor": {
    "loaded_from_database": true,
    "matches_expected_merkle_hash": true,
    "fingerprint": {
      "version": 1,
      "digest": "ecce67b36517bb7ad6ce126902d0116909759c96838b08d5ce1b372cd96a4aac"
    }
  },
  "failure_propagation": {
    "verify.invalid": "unknown",
    "verify.invalid_dependent": "unknown",
    "verify.unresolved": "unknown",
    "verify.unresolved_dependent": "unknown"
  },
  "required_dimension_identity_stable": true
}

---

Semantic fingerprint stack

This is PR 2 of 4:

1. #2482: Add semantic node fingerprints
2. #2488: Evaluate fingerprints across deployment graphs (this PR)
3. #2483: Add semantic fingerprints to deployment impact
4. #2490: Bulk endpoint for viewing fingerprints and testing fixtures

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	a54e789
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6a9b08311ae45200074c4e4a

[shangyian]

Thanks for addressing comments, looks good!