Skip to content

[fix](fe) Keep MAP key/value decimal slots independent in default deimal v3 precision promotion - #67154

Draft
starocean999 wants to merge 1 commit into
apache:masterfrom
starocean999:master_0527
Draft

[fix](fe) Keep MAP key/value decimal slots independent in default deimal v3 precision promotion#67154
starocean999 wants to merge 1 commit into
apache:masterfrom
starocean999:master_0527

Conversation

@starocean999

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Nereids' default DECIMALV3 precision promotion
(ComputeSignatureHelper.defaultDecimalV3PrecisionPromotion) recursively collects every
DECIMALV3 found in the argument types, computes a single global "wider" type, and then
replaces ALL decimal slots with that one type via
TypeCoercionUtils.replaceDecimalV3WithTarget.

For MAP<K, V> where K and V are independent type variables with different
precision/scale, merging them into one wider type breaks correctness in two ways
(they share the same type-derivation path):

  1. MAP_KEYS (and other map consumers) silently turns legal keys into NULL.
    Repro: a MAP<DECIMAL(76,0), DECIMAL(76,18)> built from a 76-digit integral key
    and 0.125000000000000000. Widening the key slot to scale 18 overflows the big
    integral key, so MAP_KEYS returns [null] instead of the key.
  2. UNNEST(MAP) (rewritten to EXPLODE_MAP) fails type analysis. The promoted MAP
    argument type no longer matches the original input MAP type.

Fix: promote each decimal slot independently instead of merging them. Each slot
(MAP key/value, ARRAY item, scalar) keeps its own precision/scale derived from its
own argument type; the wider type is only used for NULL-typed slots and the
placeholder return type. Vararg slots keep the previous behavior to avoid changing
vararg decimal functions (e.g. field).

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@starocean999

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fixed-argument MAP path now preserves independent DecimalV3 leaves, and the added unit/regression expectations are deterministic and internally consistent. However, the shared promotion helper still introduces three blocking correctness failures: linked Any/Follow slots can diverge across Decimal storage widths; field can retain different representations for its fixed operand and repeated search operands; and complex vararg signatures still collapse independent nested leaves into one global type. The two inline comments cover those three root causes.

Checkpoint summary: the four-file scope is focused, but the generic helper affects many built-ins and extension functions. Parallel Follow consumers and scalar/complex vararg paths were traced through FE cast insertion and BE dispatch. NULL/empty inputs and all Decimal32/64/128/256 boundaries were checked. There are no new concurrency, lifecycle, static-initialization, configuration-definition, transaction, persistence, data-write, wire/protocol, rolling-upgrade, observability, or status-propagation surfaces; the bounded signature-tree traversal has no actionable performance concern. Test coverage is useful but omits all three blocking triggers. No build or test command was run because this runner's review instructions prohibit them. No additional reviewer focus was supplied. The three-round review converged with no unresolved or undeduplicated candidates.

List<DataType> newArgTypes = Lists.newArrayListWithCapacity(signature.argumentsTypes.size());
for (int i = 0; i < signature.argumentsTypes.size(); i++) {
DataType sigType = signature.argumentsTypes.get(i);
if (signature.hasVarArgs && i == signature.argumentsTypes.size() - 1) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Model logical Decimal groups inside vararg signatures

This blanket last-slot replacement breaks both relationships that the new promotion needs to preserve:

  1. Field declares .varArgs(DecimalV3Type.WILDCARD, DecimalV3Type.WILDCARD), so its fixed first operand and repeated tail are one comparison type. With a nonconstant DECIMAL(3,2) first operand equal to a DECIMAL(4,3) search operand, they instead remain split; BE dispatches from argument 0 and compares raw values 120/1200, returning 0 (and cross-width cases hit an incompatible column cast).
  2. Variadic user functions can declare a repeated MAP type and use this default precision chain. For an exactly typed MAP<DECIMAL(76,0), DECIMAL(76,76)> argument, the scan computes DECIMAL(76,6) globally and this line rewrites both independent leaves to it before execution, overflowing the key and truncating the value.

Please preserve common scalar families such as field while aggregating each corresponding nested leaf independently across repeated complex arguments, with regressions for both shapes.

// keep the previous behavior for the vararg slot
newArgTypes.add(TypeCoercionUtils.replaceDecimalV3WithTarget(sigType, widerType));
} else {
newArgTypes.add(promoteDecimalV3Slot(sigType, arguments.get(i).getDataType(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep linked Decimal slots on one promoted type

This now promotes each declared argument separately after Any/Follow resolution has already tied logical slots together. For ElementAt's MAP<Any(0), Any(1)>, Follow(0) signature, a nonliteral MAP<DECIMAL(9,2), ...> column and an equal DECIMAL(10,3) lookup are first resolved to the map-key type, but this call leaves the key as Decimal32 while independently widening the lookup to Decimal64. expectedInputTypes() preserves that split, and BE's map lookup calls ColumnDecimal32::compare_at on the Decimal64 RHS even though it requires the same concrete column class. The NullType fallback has the same identity loss because it can replace a resolved Follow slot with widerType from an unrelated MAP leaf. Please promote by logical Any/Follow type-variable group, and cover storage-width boundaries plus NULL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants