refactor(cli): replace deprecated vm2 with native node:vm runner

[mbiernacik]

Summary

This PR replaces the deprecated and unmaintained vm2 library with native node:vm (VmRunner) across Dataform CLI, Core testing, and build targets.

Migrating to native node:vm removes an unmaintained third-party dependency, addresses deprecation warnings and known CVEs associated with vm2, and delivers substantial performance gains across compilation workflows.

---

Key Changes Across the Codebase

1. Native VmRunner Implementation (common/vm/vm_runner.ts & common/vm/BUILD):

   • Implemented a standalone, reusable VmRunner under //common/vm (preventing circular layering with //cli and //testing).
   • Uses node:vm.createContext and node:vm.compileFunction with scoped CommonJS require simulation.
   • Enforces projectDir boundary containment (isPathContained) with an opt-in allowedExternalPaths configuration.
   • Evicts failed module entries from moduleCache on throw so that errors re-throw consistently on subsequent require() calls.
   • Supports configurable environment isolation via env and envAllowlist options.
   • Exposes host Uint8Array and ArrayBuffer in sandbox globals to ensure cross-realm instanceof Uint8Array checks pass in protobufjs / JIT compilation.
   • Memoizes resolution lookups via resolveCache and captures suppressed resolution errors as err.cause on MODULE_NOT_FOUND.
   • Comprehensive unit test suite (common/vm/vm_runner_test.ts via //common/vm:tests).

2. CLI Compilation & JIT Worker Migration (cli/vm/compile.ts, cli/vm/jit_worker.ts):

   • Migrated standard project compilation in cli/vm/compile.ts from NodeVM to VmRunner.
   • Maintained compilation support for Dataform notebook actions (.ipynb, .md) via sourceExtensions.
   • Migrated JIT worker compilation in cli/vm/jit_worker.ts to VmRunner.

3. Core Test Harness & Property Graphs (testing/run_core.ts, testing/BUILD, core/main_property_graphs_test.ts):

   • Migrated testing/run_core.ts (runMainInVm) to execute tests through VmRunner.
   • Cleaned up workaround logic in core/main_property_graphs_test.ts by removing the artificial vm2 empty CallSite stack workaround and unifying error assertions via asPlainGraph.

4. Packaging & Dependency Cleanup (package.json, yarn.lock, packages/@dataform/cli/BUILD, packages/rollup.config.js):

   • Removed vm2 from package.json and yarn.lock.
   • Removed vm2 from externals in packages/@dataform/cli/BUILD.
   • Added vm and module to knownNodeBuiltins in packages/rollup.config.js.

5. Performance Benchmark (common/vm/vm_runner_benchmark.ts):

   • Added a reproducible benchmark runnable via bazel run //common/vm:benchmark (~10,300 module requires/sec).

[apilaskowski]

Thanks for taking this on — removing vm2 is clearly the right call, and I think VmRunner is a good shape for the replacement.

Most of my inline comments are suggestions rather than objections. The one I'd genuinely flag is the module cache entry being retained when a module throws (cli/vm/vm_runner.ts:148), since that can turn a load failure into a silently incomplete compiledGraph rather than a clear error.

A few things that didn't fit on a specific line:

• Security framing. The description says this "eliminates security vulnerabilities" and describes a "hermetic" runner with a "secure module resolver". node:vm doesn't really provide isolation — Node's docs are fairly explicit that it isn't a security mechanism, and vm2 existed because of that gap. If the CLI only ever compiles the invoking user's own repo on their own machine, then vm2's CVEs probably weren't reachable here either, so the security angle may not be doing much work in either direction. "Removes an unmaintained dependency and is meaningfully faster" seems like ample justification on its own. Would you consider rewording that part?
• Feature list. I couldn't find implementations for three things the description mentions: execution timeouts, node:vm.Script (only vm.compileFunction is used), and script caching (moduleCache holds module exports rather than compiled scripts). Possibly planned and dropped, or I'm looking in the wrong place — either way it'd be good to bring the description back in line.
• Benchmarks. Could the benchmark script land under cli/vm/? The numbers are a nice result and it'd be good to be able to re-run them. I did notice the vm2 baselines all end in .00 while the node:vm figures carry two decimals, which made me wonder whether the "before" column was measured or estimated.
• Leftover vm2 references. core/session.ts:72 and :634, core/main_session_test.ts:164 and cli/index_compile_test.ts:102 still explain live workarounds in terms of "vm2's sandbox stack stripping". Since compile.ts:65-68 now says the opposite holds, is the __df_enter/__dataform_current_file file-stack machinery still needed for @dataform/core >= 3.0.57?

And lets make sure we are properly rebased on top of main branch now :)

[apilaskowski]

ipynb and md look unrelated to the vm2 removal

Adding these two isn't mentioned in the description and doesn't have a test, and it has a couple of knock-on effects: notebook and markdown files now go through the SQLX compiler, and both extensions join allExtensions, so they participate in extension-less resolution and directory-index lookup.

Is this by accident?

[mbiernacik]

Required for Dataform notebook actions (actions.yaml), which load .ipynb/.md via nativeRequire().asJson. Without registering these extensions, VmRunner tries to evaluate them as JS and throws a syntax error on "cells": [...]. Added a test.

[apilaskowski]

After my review I would also like @Ikolina or someone who is longer in Dataform team to also take a look here.

[apilaskowski]

Thanks for addressing all the previous feedback so thoroughly — moving VmRunner to //common/vm and the test coverage look great!

Just one small thing I noticed on customResolve / isPathContained (left inline), and a quick heads-up that the PR description on GitHub might not have saved when you edited it (it still lists cli/vm/vm_runner.ts and mentions timeouts/vm.Script). Otherwise this looks ready to go!