Skip to content

vfs: integrate with CJS and ESM module loaders - #63653

Closed
mcollina wants to merge 50 commits into
nodejs:mainfrom
mcollina:vfs-module-loader-integration
Closed

vfs: integrate with CJS and ESM module loaders#63653
mcollina wants to merge 50 commits into
nodejs:mainfrom
mcollina:vfs-module-loader-integration

Conversation

@mcollina

@mcollina mcollina commented May 30, 2026

Copy link
Copy Markdown
Member

Makes require() and import resolve files served by node:vfs. Before this PR, mounted VFS files were only visible through fs.*; the loaders went straight to the real filesystem.

Design

vfs.mount() takes no arguments and returns the reserved absolute mount point of the instance: ${os.devNull}/vfs/<layerId> (for example /dev/null/vfs/0). os.devNull is a character device on POSIX and a device-namespace path on Windows; neither can have child filesystem entries, so no real path can ever exist under this root.

Everything about ownership is decidable from the path alone:

  • Dispatch: one path normalization, one prefix comparison against the namespace root, one map lookup on the layer id. O(1) in the number of mounted layers — the benchmark (benchmark/vfs/bench-fs-dispatch.js) reports flat per-call latency across 1..10 mounted layers.
  • Real-fs paths short-circuit with a single prefix check.
  • Unmount purges loader caches by prefix-scanning the mount point — no per-VFS ownership tracking is needed, and no other-VFS or real-fs entries are ever touched.
  • Symlink correctness: realpathSync of a VFS entry always resolves to another path under the same mount point, so cache entries hidden behind symlinks are captured by the prefix scan.
  • Dynamic-import identity: resolved URLs are plain file: URLs under the mount point; import(import.meta.resolve(x)) re-hits the same module job.

Two instances mounting simultaneously never collide (each gets its own layer-<id> segment), so there is no overlap validation and no ordering hazard between mounts.

Loader integration

Toggleable wrappers in the loaders. Null fast-path when no VFS is mounted; otherwise the VFS answers stat / readFile / realpath / legacyMainResolve / getFormatOfExtensionlessFile and the four package.json C++-binding calls.

Module identity follows the path: __filename, module.filename, and import.meta.url are the plain absolute path (or file: URL) of the module under the mount point — no synthetic decorations.

Review guide

Suggested reading order:

# File Role
1 lib/internal/vfs/router.js Reserved-namespace helpers: getVfsRoot, getLayerRoot, getLayerIdFromPath.
2 lib/internal/vfs/file_system.js mount() returns the layer's reserved mount point.
3 lib/internal/vfs/setup.js Heart of the PR. findVFS (O(1) lookup), fs handler, loader overrides with parity to src/node_modules.cc / src/node_file.cc, prefix-scan cache purge.
4 lib/internal/modules/helpers.js Hook surface: loader* wrappers, setLoaderFsOverrides / setLoaderPackageOverrides, purgeRealpathCacheForPrefix.
5 lib/internal/modules/cjs/loader.js stat() + TS read routed through wrappers; purgeModuleCachesForPrefix for unmount.
6 lib/internal/modules/esm/resolve.js legacyMainResolve + internalModuleStat + toRealPath routed. No URL decoration.
7 lib/internal/modules/esm/load.js getSourceSync reads via the wrapper.
8 lib/internal/modules/esm/get_format.js Extensionless format detection routed.
9 lib/internal/modules/package_json_reader.js 4 C++ binding calls routed; purgePackageJSONCacheForPrefix.
10 lib/fs.js statSync / lstatSync honour throwIfNoEntry:false on ENOENT from the VFS handler.
11 doc/api/vfs.md Mount semantics + Module loader integration section.

Tests (all gated by --experimental-vfs): test-vfs-mount, test-vfs-mount-errors, test-vfs-multi-mount, test-vfs-require, test-vfs-import, test-vfs-module-hooks, test-vfs-module-hooks-cleanup, test-vfs-package-json, test-vfs-package-json-cache, test-vfs-invalid-package-json, test-vfs-scoped-cache-purge, test-vfs-layer-id, test-vfs-layer-tag-prefix.

Refs

The reserved-namespace design follows the "no interference with valid paths in the file system" requirement from the SEA VFS requirements doc.

Out of scope

SEA + VFS, overlay/stacking of multiple VFS layers under one prefix, migrating the C++ package_configs_ cache, broader permission-model integration.

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/loaders

@nodejs-github-bot nodejs-github-bot added lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run. labels May 30, 2026
@codecov

codecov Bot commented May 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.32532% with 67 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.13%. Comparing base (4bd9821) to head (7f564e4).
⚠️ Report is 215 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/vfs/setup.js 88.59% 48 Missing ⚠️
src/node_modules.cc 80.70% 1 Missing and 10 partials ⚠️
lib/internal/modules/cjs/loader.js 90.00% 4 Missing ⚠️
lib/internal/vfs/file_system.js 97.18% 2 Missing ⚠️
lib/internal/vfs/router.js 96.42% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #63653      +/-   ##
==========================================
+ Coverage   90.10%   90.13%   +0.02%     
==========================================
  Files         752      752              
  Lines      252197   252868     +671     
  Branches    47440    47580     +140     
==========================================
+ Hits       227245   227925     +680     
+ Misses      16251    16246       -5     
+ Partials     8701     8697       -4     
Files with missing lines Coverage Δ
lib/fs.js 98.39% <100.00%> (+<0.01%) ⬆️
lib/internal/modules/esm/get_format.js 95.20% <100.00%> (ø)
lib/internal/modules/esm/load.js 91.40% <100.00%> (ø)
lib/internal/modules/esm/resolve.js 99.02% <100.00%> (-0.04%) ⬇️
lib/internal/modules/helpers.js 98.74% <100.00%> (-0.19%) ⬇️
lib/internal/modules/package_json_reader.js 99.28% <100.00%> (+0.04%) ⬆️
lib/internal/modules/run_main.js 97.64% <100.00%> (ø)
src/node_modules.h 100.00% <ø> (ø)
lib/internal/vfs/file_system.js 99.69% <97.18%> (+0.31%) ⬆️
lib/internal/vfs/router.js 97.46% <96.42%> (-0.36%) ⬇️
... and 3 more

... and 63 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollina

Copy link
Copy Markdown
Member Author

@joyeecheung take a look, should be easier to review.

Comment thread lib/internal/modules/esm/load.js Outdated
Comment thread lib/internal/modules/esm/resolve.js Outdated
@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 6321e08 to 51b033a Compare June 1, 2026 15:54
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jun 3, 2026
@github-actions github-actions Bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jun 3, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@joyeecheung joyeecheung left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A design question recently occurred to me: have we explored the versioning of the mounting?

Comment thread lib/internal/vfs/setup.js Outdated
@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

A design question recently occurred to me: have we explored the versioning of the mounting?

what do you mean? You mean multiple vfs layers on top of each other?

@joyeecheung

joyeecheung commented Jun 3, 2026

Copy link
Copy Markdown
Member

what do you mean? You mean multiple vfs layers on top of each other?

For the stacks to have some kind of version number/ID to identify the current status?

BTW I just noticed that there's no mention of unmount() and mount() in the VFS docs..

@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

BTW I just noticed that there's no mention of unmount() and mount() in the VFS docs..

I did purge them when doing the splitting; I forgot to bring them back. I'll add them to this PR.

@mcollina

mcollina commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

For the stacks to have some kind of version number/ID to identify the current status?

No but we totally should.

@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 51b033a to 294a19c Compare June 4, 2026 07:23
@mcollina

mcollina commented Jun 5, 2026

Copy link
Copy Markdown
Member Author

@joyeecheung PTAL

@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from 75bb5c7 to 99a5a5c Compare June 15, 2026 09:00
@mcollina
mcollina force-pushed the vfs-module-loader-integration branch from db320eb to 7f564e4 Compare August 21, 2026 06:18
@pipobscure

Copy link
Copy Markdown
Contributor

Can we get full CI on this? I‘d really like to avoid having this pick up more conflicts with main. Thanks!

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 24, 2026
@aduh95 aduh95 removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 26, 2026
@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@nodejs-github-bot

This comment was marked as outdated.

@mcollina

mcollina commented Aug 28, 2026

Copy link
Copy Markdown
Member Author

@joyeecheung PTAL, I plan to land this on Monday/when I can get a clean CI. If there are more things to fix, could you please open an issue?

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

@nodejs-github-bot

This comment was marked as duplicate.

@mcollina mcollina added the commit-queue PRs queued for automated landing through the Commit Queue. label Aug 31, 2026
@nodejs-github-bot nodejs-github-bot added commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. and removed commit-queue PRs queued for automated landing through the Commit Queue. labels Aug 31, 2026
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Commit Queue failed

✘ GitHub CI is still running

The pull request was removed from the Commit Queue and labeled commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. . After resolving the failure, remove that label and add commit-queue PRs queued for automated landing through the Commit Queue. to retry.

Full Commit Queue output
- Loading data for nodejs/node/pull/63653
✔  Done loading data for nodejs/node/pull/63653
----------------------------------- PR info ------------------------------------
Title      vfs: integrate with CJS and ESM module loaders (#63653)
Author     Matteo Collina <matteo.collina@gmail.com> (@mcollina)
Branch     mcollina:vfs-module-loader-integration -> nodejs:main
Labels     semver-minor, lib / src, needs-ci, commit-queue, vfs
Commits    50
 - vfs: integrate with CJS and ESM module loaders
 - vfs: fix module loader paths on Windows
 - vfs: scope-purge loader caches via per-VFS owned-keys sets
 - vfs: always allocate a fresh stat cache in clearStatCache
 - vfs: anchor vfs-layer URL tag match in urlBelongsToLayer
 - vfs: drop cache-busting/HMR reference from vfs-layer tag comment
 - benchmark: add VFS fs-dispatch overhead bench
 - vfs: hoist path normalization out of dispatch loop
 - vfs: mount inside a reserved namespace
 - test: fix VFS module tests on Windows
 - doc: clarify VFS ESM imports on Windows
 - doc: remove em dashes from VFS docs
 - vfs: drop mount() prefix argument and layer- path segment
 - Update lib/internal/vfs/setup.js
 - Update lib/internal/vfs/setup.js
 - vfs: drop unused shouldHandle / router exports
 - vfs: use internal EXTENSIONLESS_FORMAT_* constants
 - vfs: raise ERR_INVALID_PACKAGE_CONFIG for CJS too
 - vfs: default all read errors in getFormatOfExtensionlessFile to JS
 - vfs: return normalized path from findVFS
 - vfs: inline isUnderMountPoint and use shouldHandleNormalized
 - vfs,esm: share legacyMainResolveExtensions arrays
 - vfs: add cleanForVfsPrefix helper for cache purges
 - vfs: use indexed for loops in cleanForVfsPrefix
 - Revert "vfs: use indexed for loops in cleanForVfsPrefix"
 - vfs: fold loader wrappers into wrapLoaderMethod factory
 - vfs: fix lint on wrapLoaderMethod curly braces
 - vfs: throw ERR_INVALID_PACKAGE_CONFIG on malformed ancestor pjson
 - src: split GetPackageJSON and expose parsePackageJSON binding
 - vfs: use native parsePackageJSON binding, drop serializePackageJSON
 - vfs: read pjson as Buffer, skip UTF-8 decode
 - vfs: strip verbose comments across the PR
 - vfs: unexpose layerId property
 - Update doc/api/vfs.md
 - benchmark: remove vfs fs-dispatch microbenchmark
 - benchmark: add vfs module-graph cold-load benchmark
 - doc: remove unused os.devNull link definition
 - doc: specify vfs precedence in module resolution
 - Update doc/api/vfs.md
 - vfs: restore jsdoc removed in comment cleanup
 - vfs: name loader hooks after the methods they wrap
 - vfs: add mountPointURL property
 - vfs: return string from mountPointURL
 - vfs: key loader overrides by method name
 - vfs: stop node_modules lookup at the mount point
 - doc: make vfs module lookup list exact
 - test: adapt vfs lchown test to reserved mounts
 - vfs: fix package lookup on Windows
 - vfs: fix lint in loader override helpers
 - vfs: purge directory package.json cache on unmount
Committers 1
 - Matteo Collina <hello@matteocollina.com>
PR-URL: https://github.com/nodejs/node/pull/63653
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/63653
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Sat, 30 May 2026 05:41:48 GMT
   ✔  Approvals: 2
   ✔  - Paolo Insogna (@ShogunPanda) (TSC): https://github.com/nodejs/node/pull/63653#pullrequestreview-5037277613
   ✔  - James M Snell (@jasnell) (TSC): https://github.com/nodejs/node/pull/63653#pullrequestreview-5036169455
   ✘  GitHub CI is still running
   ℹ  Last Full PR CI on 2026-08-28T19:10:29Z: https://ci.nodejs.org/job/node-test-pull-request/76655/
- Querying data for job/node-test-pull-request/76655/
✔  Build data downloaded
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

View workflow run

@trivikr

trivikr commented Aug 31, 2026

Copy link
Copy Markdown
Member

@mcollina Looks like this needs to land manually since node-core-utils isn't aware of CI being successful.

@mcollina mcollina added the commit-queue-squash PRs the Commit Queue should land as one squashed commit. label Aug 31, 2026
mcollina added a commit that referenced this pull request Aug 31, 2026
Co-authored-by: James M Snell <jasnell@gmail.com>
Signed-off-by: Matteo Collina <hello@matteocollina.com>
PR-URL: #63653
Reviewed-By: Paolo Insogna <paolo@cowtech.it>
Reviewed-By: James M Snell <jasnell@gmail.com>
@mcollina

Copy link
Copy Markdown
Member Author

Landed in 4d9cb71

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commit-queue-failed PRs whose Commit Queue landing failed and need manual intervention before retrying. commit-queue-squash PRs the Commit Queue should land as one squashed commit. lib / src Issues and PRs involving general changes in the lib/ or src/ directories. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version. vfs Issues and PRs related to the virtual filesystem subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants