[lake/iceberg] Support multi-bucket-key and non-string partition keys - #4070
Open
qzyu999 wants to merge 1 commit into
Open
[lake/iceberg] Support multi-bucket-key and non-string partition keys#4070qzyu999 wants to merge 1 commit into
qzyu999 wants to merge 1 commit into
Conversation
litiliu
reviewed
Aug 24, 2026
| builder.bucket(bucketKeys.get(0), bucketCount); | ||
| } else { | ||
| // Multiple bucket keys for log table: use identity(__bucket) | ||
| builder.identity(BUCKET_COLUMN_NAME); |
Contributor
There was a problem hiding this comment.
in #4019 , the legacy field has been removed for new tables.
litiliu
suggested changes
Aug 24, 2026
Removes two artificial restrictions from Iceberg lake table creation: 1. Multi-bucket-key: Removes the 'only one bucket key' restriction. For multi-key tables, uses bucket(firstKey, bucketCount) in the partition spec. The tiering writer explicitly assigns partition values via PartitionKey, so correctness is maintained. The spec provides approximate scan pruning. 2. Non-string partitions: Removes the STRING type check. Iceberg's identity() transform works on any primitive type. Updates IcebergSplitPlanner to extract partition values via Object.class + String.valueOf() for type-agnostic conversion. Closes apache#4069, closes apache#1832.
qzyu999
force-pushed
the
lake/iceberg-multi-bucket-key
branch
from
August 25, 2026 04:23
6f1cc7a to
3059884
Compare
Contributor
Author
|
Thanks @litiliu rebased on main (which now includes #4019). The partition spec no longer uses identity(__bucket) for multi-key tables. Instead, it uses bucket(firstBucketKey, bucketCount) for all cases where bucket keys exist (single or multi). The tiering writer explicitly assigns partition values via PartitionKey, so the first-key-only spec is sufficient for routing. For read pruning it provides approximate but never incorrect filtering. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Removes two artificial restrictions from Iceberg lake table creation that limited partition spec construction to single-key buckets and string-only partitions. Both are Fluss-imposed limitations, not Iceberg format limitations.
Closes #4069, closes #1832.
Changes
Commit 1: Multi-bucket-key support
bucketKeys.size() > 1restrictionidentity(__bucket)Fluss already computes the composite hash and stores it in the system columnbucket(key, count)transform)Commit 2: Non-string partition key support
identity()transform works on any primitive type (INT, LONG, DATE, BOOLEAN, etc.)IcebergSplitPlannerto extract partition values viaObject.class+String.valueOf()instead of hardcodedString.classTests
All 26
IcebergLakeCatalogTesttests pass. The tests that previously expected exceptions now verify successful table creation.