[python] Introduce hll_sketch aggregator function - #9274
Open
zhuxiangyi wants to merge 1 commit into
Open
Conversation
Add the Python FieldHllSketchAgg implementation and align the HLL aggregation behaviour with Java's FieldHllSketchAgg / HllSketchUtil: the union is seeded from the input sketch and the result is emitted as a compact HLL_4 sketch, so the serialized bytes stay interchangeable with HllSketch.toCompactByteArray(). datasketches stays an optional dependency: the import is lazy and raises a clear pypaimon[hll-sketch] hint, following the theta_sketch precedent. Register the identifier in merge_engine_support so the guard stops rejecting it, and narrow the "unsupported aggregators" message to rbm64. rbm64 remains deferred because Java serializes it with Roaring64Bitmap's private ART format, which has no portable counterpart in pyroaring; the module docstring now records that. Also extend RegistrationTest.EXPECTED, which was still pinned to the 10 aggregators from the initial scope, to all 20, and assert the registry and the merge-engine guard list stay in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR introduces the Python
FieldHllSketchAggimplementation and aligns thePython HyperLogLog aggregation behavior with the Java implementation.
The changes include:
hll_sketchfield aggregation support, mirroring Java'sHllSketchUtil.union(the union is seeded from the input sketch, and theresult is emitted as a compact
HLL_4sketch).VARBINARYtype validation, matchingFieldHllSketchAggFactory.datasketchesan optional dependency: the import is lazy and raises aclear
pypaimon[hll-sketch]hint, following thetheta_sketchprecedent.hll_sketchinmerge_engine_support._AGGREGATION_SUPPORTED_AGG_FUNCSand narrow the "unsupported aggregators" message to
rbm64.HllSketchUtil(datasketches-java 4.2.0, lgK=12), verified in both directions.RegistrationTest.EXPECTEDlist, which was still pinned to the 10aggregators from the initial scope in [python] Implement aggregation merge engine in pypaimon #7952, to all 20, and add a test
asserting the registry and the merge-engine guard list stay in sync.
rbm64remains deferred: Java serializes it withRoaring64Bitmap's private ARTformat, which has no portable counterpart in
pyroaring, so the bytes are notinterchangeable in either direction. The module docstring now records that reason.
Tests
python -m pytest pypaimon/tests/test_field_aggregators.py -qpython -m pytest pypaimon/tests/test_aggregation_e2e.py -qpython -m pytest pypaimon/tests/test_aggregation_merge_function.py -qpython -m pytest pypaimon/tests/test_field_aggregator_registry.py -qpython -m pytest pypaimon/tests/test_merge_engine_dispatch.py -qpython -m pytest pypaimon/tests/test_optional_datasketches_dependency.py -qflake8 --config=dev/cfg.ini