Skip to content

ARSN-648: replace archived level/level-sublevel with classic-level - #2713

Open
francoisferrand wants to merge 2 commits into
development/8.6from
improvement/ARSN-648-level-dependency-chain
Open

francoisferrand wants to merge 2 commits into
development/8.6from
improvement/ARSN-648-level-dependency-chain

Conversation

@francoisferrand

@francoisferrand francoisferrand commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

level@~5.0.1 and level-sublevel@~6.6.5 are archived, and drag in an equally old levelup/abstract-leveldown/bytewise stack carrying three advisories. Moving the file metadata backend onto maintained classic-level takes that to zero, and drops 47 transitive deps.

The risky part: on-disk compatibility

Deployed stores have to keep working, so the key layout must stay byte-identical. level-sublevel's encoding isn't what native abstract-level sublevels produce by nesting, so this reproduces it explicitly rather than relying on the new default.

Verified against a real legacy store: wrote one with level@5 + level-sublevel@6.6.5, then opened it with this code — root, bucket and nested recordLog sublevels, stream ordering, and new writes landing in the same namespace as the legacy ones. tests/unit/storage/metadata/file/levelUtils.spec.js pins the raw key bytes so this can't drift silently.

Scope

MetadataFileServer, RecordLog and level-net move onto the abstract-level API directly, with no compatibility facade left behind. What remains are helpers for our own contracts rather than the library's: the # flattened sublevel paths, the sublevel lookup cache, batch operations carrying a sublevel path across RPC rather than a handle, and the iterator to Readable bridge the RPC layer needs to pipe listings back.

lib/db.ts is left alone — IndexTransaction.commit() takes a caller-supplied handle, so changing it would be a public API break outside this ticket.

Also fixes a shadowed err in metadataAPI.batch that was silently swallowing createLogRecordOps failures.

Validation

Worth flagging: this backend's own coverage is thin (MetadataFileServer.initMetadataService at 6.5% function coverage, BucketFileInterface at 5%). Pre-existing and not made worse here, but it does mean the real net under this change is CloudServer's S3BACKEND=file functional suite rather than Arsenal's. Probably deserves a follow-up ticket.

Issue: ARSN-648

@bert-e

bert-e commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@codecov

codecov Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.90291% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.87%. Comparing base (ca7b099) to head (12aabeb).

Files with missing lines Patch % Lines
lib/storage/metadata/file/MetadataFileServer.js 6.89% 27 Missing ⚠️
lib/storage/metadata/file/levelUtils.ts 91.66% 4 Missing ⚠️
Additional details and impacted files
@@                 Coverage Diff                 @@
##           development/8.6    #2713      +/-   ##
===================================================
+ Coverage            74.80%   74.87%   +0.06%     
===================================================
  Files                  229      230       +1     
  Lines                18686    18729      +43     
  Branches              3909     3916       +7     
===================================================
+ Hits                 13978    14023      +45     
+ Misses                4703     4701       -2     
  Partials                 5        5              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread tests/unit/network/rpc/level-net.spec.js
Comment thread tests/unit/storage/metadata/file/RecordLog.spec.js Outdated
@francoisferrand
francoisferrand force-pushed the improvement/ARSN-648-level-dependency-chain branch 2 times, most recently from 611139e to 70e2b1a Compare September 25, 2026 12:39
@scality scality deleted a comment from bert-e Sep 25, 2026
@scality scality deleted a comment from bert-e Sep 25, 2026
@bert-e

bert-e commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

level@5 and level-sublevel are archived and drag in an old levelup /
abstract-leveldown / leveldown chain, along with audit debt (bl memory
exposure, semver ReDoS). classic-level is the maintained successor and
embeds the same leveldown storage engine, so existing databases keep
working as-is.

The API moved on though: abstract-level is promise-first, dropped the
NotFoundError that callers detect through err.notFound, and nests
sublevels as '!a!!b!key' where level-sublevel flattened the whole path
into '!a#b!key'. The file backend moves onto that API directly, keeping
only what our own contracts need: the '#' flattened sublevel paths that
keep databases written by level-sublevel readable, the sublevel lookup
cache, batch operations that carry a sublevel path across RPC rather
than a handle, and the iterator to Readable bridge the RPC layer needs
to pipe listings back.

IndexTransaction keeps its callback based batch: it commits to the
handle its caller provides, which over RPC is the metadata client.

Issue: ARSN-648
@francoisferrand
francoisferrand force-pushed the improvement/ARSN-648-level-dependency-chain branch from 70e2b1a to 8f15a5a Compare September 25, 2026 12:52
@francoisferrand
francoisferrand marked this pull request as ready for review September 25, 2026 12:55
Comment thread lib/network/rpc/level-net.ts Outdated

@delthas delthas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but would be easier to review with format changes moved into their own commit

return undefined;
}
for (let i = 0; i < nbKeys; ++i) {
client.withRequestLogger(reqLogger)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I feel this commit mixes preexisting prettier file changes with actual changes. Would it be possible to move preexisting prettify into its own first commit?

Comment on lines +3 to +6
// level-sublevel stored a nested path as a single '!a#b!key' section, where
// abstract-level nests them as '!a!!b!key'. Flattening the path into one
// sublevel name reproduces the historical layout, so databases written by
// previous versions stay readable.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is hard to understand, feels like its written from the pov of doing a migration (so having the context of level-sublevel) instead of focusing on what it is doing

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I guess its easier to understand when knowing about leveldb

@SylvainSenechal SylvainSenechal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment from claude might be relevant.

I think it could be nice to have a review from hopo team too

The default lookup nested sublevels per path component, which used to
match the level-sublevel layout but now yields '!a!!b!key' on
classic-level, and it dropped the root encodings on the way. Rather than
keep a fallback that silently writes keys nobody can read back, callers
now have to pass the lookup they want.

Issue: ARSN-648

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants