Skip to content

[feature](lance) Support OSS storage provider - #67157

Draft
FANNG1 wants to merge 1 commit into
apache:branch-4.1from
FANNG1:lance-oss-67131
Draft

[feature](lance) Support OSS storage provider#67157
FANNG1 wants to merge 1 commit into
apache:branch-4.1from
FANNG1:lance-oss-67131

Conversation

@FANNG1

@FANNG1 FANNG1 commented Aug 26, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #67131

Related PR: #66805

Problem Summary:

Lance catalogs already forward provider-native storage options, but Doris did not translate typed OSS properties and filesystem catalogs rejected oss:// warehouses.

This change:

  • maps Doris OSS properties to Lance public OSS options;
  • normalizes native Namespace-vended aliases before merging so vended values deterministically override static values;
  • rejects conflicting aliases while preserving unknown vended options;
  • accepts OSS filesystem warehouses; and
  • redacts OSS credentials from metadata errors.

It reuses the FE-to-BE storage-option transport introduced by #66805 and does not change Thrift or BE code.

Release note

Support native Alibaba Cloud OSS storage options for Lance catalogs.

Check List (For Author)

  • Test:
    • Unit Test: Official Doris build image run-fe-ut.sh, 31 tests passed
    • Manual test: PyLance real OSS write/read and Doris FE metadata access through filesystem and REST catalogs
    • Build: Full FE build and Checkstyle passed; ASAN BE compiled and linked successfully
    • Validation boundary: An end-to-end BE scan could not run because the x86_64 ASAN binary crashes before service initialization under ARM emulation
  • Behavior changed: Yes. Lance catalogs can use oss:// with Doris OSS properties and OSS Namespace-vended options.
  • Does this need documentation: No

Scope

Not included: role-based credential vending, STS credential refresh during a scan, a new Thrift envelope, or static support for other object-store providers.

### What problem does this PR solve?

Issue Number: close apache#67131

Related PR: apache#66805

Problem Summary: Lance catalogs already forward provider-native storage options, but Doris did not translate typed OSS properties and filesystem catalogs rejected oss:// warehouses. Add an OSS provider that maps Doris OSS properties to Lance public OSS options, normalizes native Namespace-vended aliases before deterministic merging, rejects conflicting aliases, preserves unknown vended options, accepts OSS filesystem warehouses, and redacts OSS credentials from metadata errors.

### Release note

Support native Alibaba Cloud OSS storage options for Lance catalogs.

### Check List (For Author)

- Test:
    - Unit Test: Official Doris build image run-fe-ut.sh, 31 tests passed
    - Manual test: PyLance real OSS write/read and Doris FE metadata access through filesystem and REST catalogs; BE compiled successfully, but an end-to-end BE scan could not run because the x86_64 ASAN binary crashes before service initialization under ARM emulation
- Behavior changed: Yes. Lance catalogs can use oss:// with Doris OSS properties and OSS Namespace-vended options.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@FANNG1

FANNG1 commented Aug 26, 2026

Copy link
Copy Markdown
Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 88.00% (44/50) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants