[DJ Semantic Fingerprint 1] Add semantic node fingerprints

[philipfweiss]

DJ already classifies semantic and cosmetic node changes, but that decision is not exposed as a reusable node identity. Consumers otherwise have to duplicate field selection and SQL normalization, which can drift from the server.

This PR adds:

• version-dispatched local fingerprint and explicit parent-composition primitives derived from ChangeTier field definitions
• isolated semantic normalization and frozen builders under semantic_fingerprints
• an explicit version 1 field projection that new model fields cannot change implicitly
• structural SQL AST serialization under sql/parsing, independent of render settings
• versioned Merkle composition from sorted parent fingerprints

local_node_fingerprint() hashes one node definition. The lower-level compose_node_fingerprint() requires explicit parent fingerprints. These are construction primitives, and NodeSpec has no fingerprint method. #2488 adds SemanticFingerprintGraph, which owns graph snapshots, parent resolution, and graph-bound evaluation.

Fingerprint composition example

For each node, f_node is its intrinsic semantic fingerprint and h_node is its Merkle object hash:

h_node = H(f_node, sorted parent h values)

flowchart LR
    S["source: orders<br/>fingerprint: f_orders<br/>object hash: h_orders = H(f_orders)"]
    D["dimension: customer<br/>fingerprint: f_customer<br/>object hash: h_customer = H(f_customer)"]
    T["transform: clean_orders<br/>fingerprint: f_clean_orders<br/>object hash: h_clean_orders = H(f_clean_orders, sort(h_orders, h_customer))"]
    M["metric: order_count<br/>fingerprint: f_order_count<br/>object hash: h_order_count = H(f_order_count, h_clean_orders)"]
    C["cube: orders_cube<br/>fingerprint: f_orders_cube<br/>object hash: h_orders_cube = H(f_orders_cube, sort(h_order_count, h_customer))"]

    S -->|SQL parent| T
    D -.->|dimension link| T
    T -->|SQL parent| M
    M -->|cube metric| C
    D -->|cube dimension| C

    classDef source fill:#fff1d6,stroke:#c2410c,color:#431407
    classDef dimension fill:#ecfdf5,stroke:#059669,color:#064e3b
    classDef derived fill:#dbeafe,stroke:#2563eb,color:#172554
    class S source
    class D dimension
    class T,M,C derived

Loading

---

Verification:

from datajunction_server.api.main import app
from datajunction_server.models.deployment import SourceSpec, TransformSpec
from datajunction_server.semantic_fingerprints.engine import (
    compose_node_fingerprint,
    local_node_fingerprint,
)

source = SourceSpec(
    name="source",
    catalog="warehouse",
    schema="sales",
    table="orders",
)
changed = source.model_copy(update={"table": "orders_v2"})
compact = TransformSpec(name="orders", query="SELECT id FROM source")
formatted = TransformSpec(name="orders", query=" SELECT  id\nFROM source ")

source_hash = local_node_fingerprint(source)
original = compose_node_fingerprint(
    compact,
    parent_fingerprints=[source_hash],
)
format_only = compose_node_fingerprint(
    formatted,
    parent_fingerprints=[source_hash],
)
downstream = compose_node_fingerprint(
    compact,
    parent_fingerprints=[local_node_fingerprint(changed)],
)

print(f"version={original.version}")
print(f"format_stable={original == format_only}")
print(f"parent_change_propagates={original != downstream}")

Observed:

version=1
format_stable=True
parent_change_propagates=True

---

Semantic fingerprint stack

This is PR 1 of 4:

1. #2482: Add semantic node fingerprints (this PR)
2. #2488: Evaluate fingerprints across deployment graphs
3. #2483: Add semantic fingerprints to deployment impact
4. #2490: Bulk endpoint for viewing fingerprints and testing fixtures

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	3411fcf
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6a987c07a67dd700080f880d

[shangyian]

This looks great -- liking the shift to having the logic on NodeSpec with ChangeTier.

[betodealmeida]

In the example:

source_hash = source.semantic_fingerprint()
original = compact.semantic_fingerprint(parent_fingerprints=[source_hash])
format_only = formatted.semantic_fingerprint(parent_fingerprints=[source_hash])

How hard would it be to not require parent_fingerprints, and instead just traverse the node upstream (with memoization) to compute it? So that we can simply do:

source_hash = source.semantic_fingerprint()
original = compact.semantic_fingerprint()
format_only = formatted.semantic_fingerprint()

This way we don't have the risk of missing a fingerprint and generating an incorrect fingerprint.

(Unless this is supposed to be an internal API that the machinery from the next PRs is going to use.)

[philipfweiss]

In the example:

source_hash = source.semantic_fingerprint()
original = compact.semantic_fingerprint(parent_fingerprints=[source_hash])
format_only = formatted.semantic_fingerprint(parent_fingerprints=[source_hash])

How hard would it be to not require parent_fingerprints, and instead just traverse the node upstream (with memoization) to compute it? So that we can simply do:

source_hash = source.semantic_fingerprint()
original = compact.semantic_fingerprint()
format_only = formatted.semantic_fingerprint()

This way we don't have the risk of missing a fingerprint and generating an incorrect fingerprint.

Good question! The argless implementation is actually what I have done in the past, but I think unfortunately due to DJ's data model it doesn't work. I ran into the following failure case:

orders_v1 = SourceSpec(
    namespace="sales",
    name="orders",
    catalog="warehouse",
    schema_="raw",
    table="orders_v1",
)
orders_v2 = orders_v1.model_copy(update={"table": "orders_v2"})
revenue = TransformSpec(
    namespace="sales",
    name="revenue",
    query="SELECT amount FROM ${prefix}orders",
)
current = {"sales.orders": orders_v1, "sales.revenue": revenue}
proposed = {"sales.orders": orders_v2, "sales.revenue": revenue}
assert (
    current["sales.revenue"].semantic_fingerprint()
    == proposed["sales.revenue"].semantic_fingerprint()
)  # Incorrect: the parent changed

The argless call never receives current or proposed, so it doesn't know which orders definition to traverse.

[philipfweiss]

Hmm- ok I've thought about the issue some more. I think a better API might be:

current_graph["revenue"].semantic_fingerprint()
proposed_graph["revenue"].semantic_fingerprint()

or something instead like

graph = SemanticFingerprintGraph(...)
graph.fingerprint("revenue")

The key insight here is that the hash depends on the graph existing already, making it a property of an already computed (and frozen) graph. When it's on the node, you essentially have to supply the whole graph context anyways. If you're wrong, you can footgun and get the wrong hashes.

Requiring the full graph first prevents omitted-parent bugs, memoizes traversal, and handles cycles/ missing parents consistently.