Support for Databricks deletion vectors

[ianton-ru]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support for Databricks deletion vectors

Documentation entry for user-facing changes

Databricks put deletion vector into .bin file instead of puffin files.

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [d8e94ed]

[ianton-ru]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d8e94ed1bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject unknown wrappers before selecting the slice-only path

When a Puffin header is damaged—or a .bin object uses an unknown version—this accepts any valid envelope found at the manifest offset as SliceOnly. That skips bindDeletionVectorBlob, including the footer checks for the referenced data file, fields, and cardinality, so malformed input can silently filter rows instead of being rejected. Require either the known Delta 0x01 container layout or an exact bare-envelope layout before taking this path.

AGENTS.md reference: AGENTS.md:L153-L153

Useful? React with 👍 / 👎.

[ianton-ru]

I did not find strong description what can and can't be in non-puffin DV files. So prefer to accept all, until report about specific files that must be declined.

[ianton-ru]

@blau-ai

[blau-ai]

CI triage — @ianton-ru

Verdict: 0 of the failures are caused by this PR. They break down as 3 flaky/infra stateless tests (all in non-default storage configs), 2 integration shards killed by the 3-hour xdist session-timeout (with only pre-existing flaky tests failing inside them), and 1 base-image CVE scan. The PR's own feature is healthy: all 36 test_deletion_vectors.py cases PASSED (incl. every test_deletion_vectors_delta_bin[...]), and other Iceberg-read tests (test_minmax_pruning, test_expire_snapshots, …) passed on the same build.

Head SHA analyzed: d8e94ed.

Per-failure breakdown

Check	Failing test	Cause	PR-caused?
Stateless (amd_debug, distributed plan, s3)	03572_export_merge_tree_part_limits_and_table_functions	Flaky — CI's own diagnosis: "All reruns passed (6/6), not reproducible". S3/hive export, not Iceberg.	❌ No
Stateless (amd_binary, cas s3)	02265_column_ttl	Flaky (7/10 reruns failed) — NETWORK_ERROR on manifest relink in the experimental cas s3 replication path. Column-TTL test, not Iceberg.	❌ No
Stateless (amd_asan_ubsan, cas s3)	03578_merges_with_and_without_ttl	Infra — S3Exception … Code 111, Connection refused writing to the cas_s3 bucket. TTL merges, not Iceberg.	❌ No
Integration 4/8 (asan_ubsan, db disk, old analyzer)	test_schema_inference ×12	Shard hit session-timeout: 10800s exceeded. The 12 test_schema_inference cases failed en-masse in 40s with instant ATTEMPT_TO_READ_AFTER_EOF — the module's server instance came up unhealthy (log shows Can't connect to Minio … Connection refused during its cluster startup) on an overloaded runner. Isolated to that instance; the other worker kept passing Iceberg tests.	❌ No
Integration 5/8 (asan_ubsan, db disk, old analyzer)	test_rabbitmq_select[0/1], test_rabbitmq_csv_with_delimiter	Same session-timeout: 10800s exceeded. RabbitMQ tests failed with Failed: Time… (known-flaky/slow RabbitMQ), no Iceberg involvement.	❌ No
GrypeScanServer (-alpine)	image CVE scan	7 high/critical CVEs in the alpine base image (apk/musl packages). The non-alpine clickhouse-server and clickhouse-keeper images scanned 0 vulns. Nothing in this C++ change touches it.	❌ No

Why the two integration shards are the systemic issue (not the PR)

Both amd_asan_ubsan, db disk, old analyzer shards run a huge per-shard test list across only 2 workers with a 3-hour cap (-n 2 --dist=loadfile --session-timeout=10800) and both blew that budget:

4/8: !!!!!! xdist.dsession.Interrupted: session-timeout: 10800.0 sec exceeded !!!!!!!
5/8: !!!!!! xdist.dsession.Interrupted: session-timeout: 10800.0 sec exceeded !!!!!!!

Shard 5/8 contains none of this PR's new tests and still timed out — so the timeout is a capacity/sharding problem, not something the added tests introduced. The PR's Iceberg tests that did run were quick (test_deletion_vectors.py finished 10:30→10:38, ~8 min) and all green.

Suggested next steps

• No code change is needed for CI here — nothing points at the diff.
• Re-run the failed jobs (Integration tests 4/8 & 5/8, and the 3 stateless suites). The stateless ones are flaky/infra and the integration ones just need to fit under the 3h window; a fresh run on a less-loaded runner should go green. The Grype-alpine finding is orthogonal to this PR (base-image CVEs) and shouldn't gate it.
• ⚠️ One coverage gap to note: because shard 4/8 timed out, your new test_deletion_vectors_databricks_uniform.py was scheduled but never actually executed (it appears in the shard's collection list but produced no result). test_deletion_vectors.py covered the .bin path via test_deletion_vectors_delta_bin[...] and passed, but the dedicated Databricks-UniForm test still needs a green run to be sure it's covered. Worth confirming it runs on the re-triggered job.

_{Automated triage from CI reports (praktika result_pr.json + shard job.logs). I can't build/run ClickHouse in this environment, so correctness is asserted from CI evidence, not a local run.}

[Selfeer]

This is from the audit review. I was also wandering about this, but I don't see any unexpected behavior while testing this specific path. For now we agree fully that we will not be filtering out the deletion vector files as valid or invalid?

PR #2271 Audit Findings

• Medium — Unknown/non-Puffin wrappers are accepted as valid deletion-vector containers
  • Why this is an issue: In detectIcebergDeletionVectorContainer (in src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp), any object that does not start with PFA1 is accepted as SliceOnly if bytes at content_offset look like a deletion-vector-v1 envelope. This creates a fallback path where corrupted or non-standard containers are treated as valid instead of being rejected.
  • Impact: This bypasses Puffin-footer identity checks (bindDeletionVectorBlob) that validate constraints like the referenced data file binding and footer-level DV metadata. As a result, malformed/corrupted objects can be accepted and their row masks applied, which risks silent incorrect filtering rather than a fail-closed exception.

[ianton-ru]

This is from the audit review. I was also wandering about this, but I don't see any unexpected behavior while testing this specific path. For now we agree fully that we will not be filtering out the deletion vector files as valid or invalid?

PR #2271 Audit Findings

• Medium — Unknown/non-Puffin wrappers are accepted as valid deletion-vector containers

  • Why this is an issue: In detectIcebergDeletionVectorContainer (in src/Storages/ObjectStorage/DataLakes/PuffinDeletionVectorReader.cpp), any object that does not start with PFA1 is accepted as SliceOnly if bytes at content_offset look like a deletion-vector-v1 envelope. This creates a fallback path where corrupted or non-standard containers are treated as valid instead of being rejected.
  • Impact: This bypasses Puffin-footer identity checks (bindDeletionVectorBlob) that validate constraints like the referenced data file binding and footer-level DV metadata. As a result, malformed/corrupted objects can be accepted and their row masks applied, which risks silent incorrect filtering rather than a fail-closed exception.

The same as codex find above.

I did not find strong description what can and can't be in non-puffin DV files. So prefer to accept all, until report about specific files that must be declined.

[Selfeer]

Iceberg v3 deletion vectors in Databricks Delta .bin files

Coverage for Altinity/ClickHouse#2271: ClickHouse read support for Iceberg v3 deletion vectors stored in a Delta-style deletion_vector_*.bin instead of a Puffin file.

Databricks UniForm writes the same deletion-vector-v1 roaring blob a Puffin would embed, wrapped in one format-version byte, and still declares file_format = PUFFIN in the delete manifest. ClickHouse identifies the container from the object’s bytes (PFA1 → Puffin, otherwise an envelope at content_offset). Almost everything below that decision is the existing read path.

Run just this pass:

./regression.py ... --only "/iceberg/delta deletion vectors/*" -l test.log

---

What we test

A separate TestFlows feature (delta_feature.py, path /iceberg/delta deletion vectors) — not a switch inside the Puffin suite.

Reused as-is (same scenarios, vectors rewritten to .bin):

• access forms (icebergS3, engine, catalog, icebergLocal)
• vector shapes (empty, all-rows-deleted, boundaries, roaring, shared container)
• coexistence with other delete formats
• count paths
• distributed / cluster reads
• sanity (read, read-only, mutations rejected, writer DML)
• error handling that is not footer-specific (malformed blob, bounds, manifest consistency, resource limits, non-Parquet data files)
• Avro damage to the delete manifest and manifest list

New, .bin-only (container_formats.py):

Kind	What it pins
Layout	A .bin is 0x01 + the byte-identical Iceberg blob; content_offset = 1
Fixtures	Our builder matches ClickHouse’s own gtest blobs ({2,5}, {7,9}, two-slice container)
Spark blob + our wrapper	Crafted container around a real writer blob adds only the version byte
Converted writer table	In-place Puffin → .bin does not change visible rows
Installed .bin	Object overwrite, and a sibling deletion_vector_*.bin the manifest points at
Permissive wrappers	Bare envelope at offset 0; unexpected version byte; unknown 4-byte prefix — all accepted
Manifest vs bytes	file_format = PARQUET still classified as a DV from content_offset / content_size_in_bytes
Fail-closed	Misdeclared offset 5; declared size ≠ envelope length; object that is neither container
Golden table	Customer’s own Databricks UniForm table (see below)

---

The three ways we get .bin files

We cannot run Databricks. Everything the reader can see reduces to: version-byte container, file_format = PUFFIN on a non-Puffin object, content_offset = 1, and (for the customer table) UniForm’s _iceberg/ metadata. Three complementary routes:

1. Craft the container

build_delta_bin / install_delta_bin: one version byte in front of the blob builder we already use for Puffin.

Used for hostile and edge cases a real writer will never produce — wrong offset, wrong size, no version byte, unknown prefix, truncated object, file_format mismatch. Also the path behind replace_vector_with_positions when a scenario installs a crafted bitmap.

2. Convert a Spark-written Puffin in place

Default for every reused scenario. Spark commits a normal v3 MoR table (real Puffin, real blob). The harness extracts each deletion-vector-v1 blob, writes 0x01 ‖ blob… over the same key, and patches content_offset / content_size_in_bytes / file_size_in_bytes. The blob bytes stay the writer’s; only the wrapper is ours.

This is what makes the existing suite a .bin suite without copying it. After conversion we assert the objects really are .bin — a no-op conversion would leave a Puffin table and the whole feature would pass while proving nothing.

3. Golden Databricks fixture

The customer table from the PR, stored under fixtures/databricks_uniform_v3/. We did not write the .bin, the Parquet, or the Avro. Install only rewrites S3 paths and flattens _iceberg/metadata → metadata/.

This is the only input that is not an expression of our own layout assumption. Databricks also run-length-encoded the roaring bitmap (cookie 12347, 251 bytes for 1000 positions); our writer emits array containers for the same positions, so this is a decode path crafting does not cover.

---

What we do for the Databricks / UniForm case

The customer shape, in one line: deletion_vector_<uuid>.bin beside the data files, manifest file_format = PUFFIN, content_offset = 1, content_size_in_bytes = 251, gzip-compressed Iceberg metadata, no version-hint.text.

1. Install the frozen table and assert that layout (size 252, leading 0x01, not PFA1, offsets, cardinality 1000, run-encoded cookie) so a mangled fixture cannot produce a green read.
2. Read it through icebergS3: 9000 live rows, ids 1001..10000, the deleted Id <= 1000 band gone, trivial count() agrees.
3. Replay the same file layout on crafted tables: sibling .bin + retargeted file_path (how Databricks leaves a DELETE), and file_format = PUFFIN on a non-Puffin object (how Databricks labels it).
4. Fail closed on the layout people actually misread: content_offset = 5 (blob starts at the magic instead of the length prefix) — the error must name offset 5, not “out of bounds.”
5. Run the reused suite on converted Spark tables so access forms, counts, cluster reads, malformed blobs, and writer DML all see .bin containers, not only the one golden table.

The golden table is the fidelity anchor (one schema, one snapshot, customer bytes). Conversion plus crafting is the coverage. Together they are the Databricks .bin test.