[DJ Semantic Fingerprint 3] Add semantic fingerprints to deployment impact

[philipfweiss]

/deployments/impact reports operations and changed fields without a server-owned semantic identity. Metadata consumers otherwise have to repeat DJ's classification, SQL handling, and graph traversal.

Depends on #2488.

This PR:

• wires deployment graph fingerprints into create, update, no-op, delete, and downstream impact results
• discovers external downstream nodes across SQL, dimension-link, required-dimension, and cube-filter edges
• preserves current fingerprint state when a requested deletion is blocked
• returns additive nullable change_tier and semantic_fingerprint fields
• preserves deploy_type and parses old and new response shapes in the Python client

Impact propagation example

flowchart LR
    X{"orders table changes"}
    S["source: orders<br/>object hash: h_orders to h_orders_next<br/>operation: update<br/>direct tier: major"]
    T["transform: clean_orders<br/>object hash: h_clean_orders to h_clean_orders_next<br/>operation: noop<br/>direct tier: none"]
    M["metric: order_count<br/>object hash: h_order_count to h_order_count_next<br/>operation: noop<br/>direct tier: none"]
    C["cube: orders_cube<br/>object hash: h_orders_cube to h_orders_cube_next<br/>operation: noop<br/>direct tier: none"]

    X --> S
    S ==>|h_orders changed| T
    T ==>|h_clean_orders changed| M
    M ==>|h_order_count changed| C

    classDef direct fill:#fff1d6,stroke:#c2410c,color:#431407
    classDef propagated fill:#dbeafe,stroke:#2563eb,color:#172554
    class S direct
    class T,M,C propagated

Loading

The metadata rows become source::orders::h_orders_next, transform::clean_orders::h_clean_orders_next, metric::order_count::h_order_count_next, and cube::orders_cube::h_orders_cube_next. All four CODEOWNERS are therefore included in review.

---

Verification:

Started this branch on development slot 1:

printf '1\n' > .dev-port
./dev.sh up -d

Then deployed a source, transform, and metric before changing only the source table and calling POST /deployments/impact:

import copy
import json
import time

import requests

base = "http://127.0.0.1:8100"
namespace = "fingerprint_verification"
session = requests.Session()
session.post(
    f"{base}/basic/login/",
    data={"username": "dj", "password": "dj"},
).raise_for_status()

initial = {
    "namespace": namespace,
    "nodes": [
        {
            "name": "source",
            "node_type": "source",
            "catalog": "default",
            "schema": "test",
            "table": "orders",
            "columns": [{"name": "id", "type": "bigint"}],
        },
        {
            "name": "transform",
            "node_type": "transform",
            "query": "SELECT id FROM ${prefix}source",
        },
        {
            "name": "metric",
            "node_type": "metric",
            "query": "SELECT COUNT(*) FROM ${prefix}transform",
        },
    ],
}

def deploy(spec):
    deployment = session.post(f"{base}/deployments", json=spec).json()
    while True:
        result = session.get(f"{base}/deployments/{deployment['uuid']}").json()
        if result["status"] in {"success", "failed"}:
            return result
        time.sleep(0.1)

before = deploy(initial)
before_hashes = {
    result["name"]: result["semantic_fingerprint"]["digest"]
    for result in before["results"]
    if result["deploy_type"] == "node"
}
proposed = copy.deepcopy(initial)
proposed["nodes"][0]["table"] = "orders_v2"
response = session.post(f"{base}/deployments/impact", json=proposed)
response.raise_for_status()
results = [
    {
        "name": result["name"],
        "operation": result["operation"],
        "change_tier": result["change_tier"],
        "semantic_fingerprint": result["semantic_fingerprint"],
        "hash_changed": (
            result["semantic_fingerprint"]["digest"]
            != before_hashes[result["name"]]
        ),
    }
    for result in response.json()["results"]
    if result["deploy_type"] == "node"
]
print(json.dumps({"status": response.status_code, "results": results}, indent=2))

Observed:

{
  "status": 200,
  "results": [
    {
      "name": "fingerprint_verification.transform",
      "operation": "noop",
      "change_tier": "none",
      "semantic_fingerprint": {
        "version": 1,
        "digest": "396e92df092afb466e34ef038b4bd6b2987b6ec09b1f39700f5d1cbf649e3c71"
      },
      "hash_changed": true
    },
    {
      "name": "fingerprint_verification.metric",
      "operation": "noop",
      "change_tier": "none",
      "semantic_fingerprint": {
        "version": 1,
        "digest": "3f6230e34838ea84ca8cd10ce6b4049b45b3ff79cf2e9840a419ade1731d19d2"
      },
      "hash_changed": true
    },
    {
      "name": "fingerprint_verification.source",
      "operation": "update",
      "change_tier": "major",
      "semantic_fingerprint": {
        "version": 1,
        "digest": "dd0e10eeead298cdb31af85f3ccd3dac98760564c4d1f97473eb25bc5d024e98"
      },
      "hash_changed": true
    }
  ]
}

---

Semantic fingerprint stack

This is PR 3 of 4:

1. #2482: Add semantic node fingerprints
2. #2488: Evaluate fingerprints across deployment graphs
3. #2483: Add semantic fingerprints to deployment impact (this PR)
4. #2490: Bulk endpoint for viewing fingerprints and testing fixtures

[netlify]

✅ Deploy Preview for thriving-cassata-78ae72 canceled.

Name	Link
🔨 Latest commit	e137874
🔍 Latest deploy log	https://app.netlify.com/projects/thriving-cassata-78ae72/deploys/6aa30574711892000827ff37

[shangyian]

metric_revision is pinned to Node.current_version below, but parent_revision is not, so a column belonging to a stale parent revision can still pull its metric into the frontier. The dimension-link query just above pins, so should this one match it in behavior?

[philipfweiss]

I think the current behavior is intentional. A metric can still depend on an older revision of a dimension column. If we only consider the dimension’s current revision, we could miss that dependency and fail to mark the metric as affected. I’ll add a test to make this behavior clear.

[shangyian]

I might be missing something, but it looks like the rows query will end up loading every non-deactivated cube's filters? As in, it's not filtering to whether a cube is in a set of ids (e.g., a cube is either within or downstream of the deployment). This query could be skipped if there are no downstream cubes, but also limited to the ones that are affected by the deployment.

[philipfweiss]

You’re right that this scans all active cubes. We can’t limit it to cubes already identified as downstream because this query is what discovers cubes affected only through filters...

That being said, it shouldn't be too bad of a cost, roughly proportional to the total number of cube filters. It is one query per deployment with changes, so I expect the cost to be pretty modest.