diff --git a/CHANGELOG.md b/CHANGELOG.md index cb64aa372e..35ca4b897d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -99,7 +99,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes -- **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries. +- **Indexing a project with a bundled minified JavaScript file is faster.** For each reference in a one-line file such as `jquery.min.js`, the resolver counted characters from the start of the line to find the reference's column, many times per reference. It now skips straight to the column when the text before it is ASCII. Alamofire, whose docs folder carries jQuery, resolves its references in a median 0.93 s instead of 1.25 s (profiling runs on a busy host, 5,614 nodes and 13,673 edges either way). Separately, the resolver now looks nodes up by query only when there are fewer than 15,000 pending references and fewer than 0.3 per node; at either limit it loads the node table. A full index of a small project such as gin (about 4.4 references per node) therefore loads the table, saving about 60 ms of its one-second index, while a one-file sync still queries. In the 2026-10-12 benchmark on Node, a full index of Alamofire takes 1.81 s against upstream's 1.73 s and gin 1.15 s against 1.13 s ([results](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md)). - **Code written without a space before a name resolves like spaced code.** Where one token ends exactly where the next begins, the resolver read the syntax at that point as the earlier token: in `x=1;local function helper()` it saw the `;`, so the Lua local read as global and calls from other files linked to it, and in `if ($x instanceof Foo)$x->bar();` it saw the `)`, so the PHP guard did not type `$x`. It now reads the token that starts there, the same way the TypeScript side already did. On json, jq and CPython the graph is unchanged. - **A Kotlin infix call on a parenthesized receiver now reaches the right class.** `(b) put 1`, `(b)put(1)` and `((b)) put (1)` linked to whichever class had a `put`, so with `b: A` and another class `B` that also declared `put`, the call landed on `B::put`. They now follow what `b` holds, as `b put 1` already did. An extension on a project class called through a typed receiver is still not linked, as before; the parenthesized form no longer guesses it by name. - **A Vite plugin that spreads another plugin no longer turns off a React Router app's file routes.** Any `...` in a root `vite.config`, `react-router.config` or `remix.config` was taken for custom route configuration, so React Router's own RSC playground, which merges `...mdx({ … })` into a plugin entry, got no file routes. A spread of a call inside a `plugins` list is now ignored, since it only builds a plugin; a spread of anything else (`...shared`, `...makeConfig()` at the top level, `[...plugins]`, or inside a plugin call's arguments) still turns file routes off. The playground goes from 0 to 12 file routes. diff --git a/README.md b/README.md index f3f4c9cff8..3abfc93398 100644 --- a/README.md +++ b/README.md @@ -186,35 +186,37 @@ Call resolution the fork adds: ### Measured results -Upstream `main` at `6560052` (v1.6.2) against the fork at `34cc55e`, each run on Node.js 24.21.0 and on Bun 1.4.2. Measured 2026-10-03 on a 16-vCPU WSL2 host over seven corpora, with the arms in mirrored order; index figures are the median of two runs and sync figures the median of four. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer. With `CODEGRAPH_ALLOW_UNSAFE_NODE=1` it runs; those numbers, the method and the graph sizes are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md), which also compares this run with the 2026-09-28 one. The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). +Upstream `main` at `6560052` (v1.6.2) against the fork at `ce6120be`, each run on Node.js 24.21.0 and on Bun 1.4.3. Measured 2026-10-12 on a 16-vCPU WSL2 host over seven corpora, with the arms in mirrored order; index figures are the median of two runs and sync figures the median of four. Upstream refuses to start on Bun, which reports itself as Node 26, because upstream blocks Node 25 and newer. With `CODEGRAPH_ALLOW_UNSAFE_NODE=1` it runs; those numbers, the method and the graph sizes are in [`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md`](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md), which also compares this run with the 2026-10-03 one. The scripts that produced them are in [`scripts/benchmarks/runtime/`](scripts/benchmarks/runtime/). + +![Fork against upstream CodeGraph on n8n and the command line: the fork indexes n8n 2.3 times faster, syncs an edited file 2.9 times faster, answers a warm question 3.2 times faster and uses 34% less memory while indexing. The fork is better in 68 of 76 comparisons, about the same in 5 and worse in 3.](assets/fork-vs-upstream-performance.svg?v=1) + +The three comparisons where the fork is worse are listed in the chart: indexing Alamofire on Node (+5%, with overlapping runs) and the warm question on gin (+8% on Node, +4% on Bun, 1–2 ms). [Every result](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md#every-result-at-a-glance) is in one heatmap in the benchmark doc. **Full index** (`codegraph init`), time and peak memory: | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin (Go, 119 files) | 1.03 s, 549 MiB | 1.17 s, 479 MiB | 1.11 s, 383 MiB | -| Alamofire (Swift, 129 files) | 1.62 s, 838 MiB | 2.05 s, 664 MiB | 2.08 s, 486 MiB | -| pretix (Python + JS, 1,472 files) | 21.1 s, 3.27 GiB | 10.8 s, 3.70 GiB | 9.41 s, 3.23 GiB | -| CPython (C + Python, 3,708 files) | 94.7 s, 8.34 GiB | 45.5 s, 6.41 GiB | 45.1 s, 5.73 GiB | -| discourse (Ruby + JS, 20,277 files) | 23.7 s, 3.86 GiB | 25.1 s, 2.88 GiB | 22.1 s, 2.44 GiB | -| supabase (React + Next.js + TS, 10,716 files) | 26.7 s, 4.26 GiB | 23.1 s, 3.95 GiB | 23.4 s, 3.65 GiB | -| n8n (Vue + TS, 24,434 files) | 168.9 s, 9.45 GiB | 78.4 s, 6.33 GiB | 79.1 s, 5.76 GiB | - -Upstream at `6560052` takes 2.3 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 21.1 s, CPython 35.6 s to 94.7 s). For pretix, [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) shows the cause is upstream's code, not the host. +| gin (Go, 119 files) | 1.13 s, 547 MiB | 1.15 s, 403 MiB | 1.34 s, 302 MiB | +| Alamofire (Swift, 129 files) | 1.73 s, 837 MiB | 1.81 s, 702 MiB | 1.69 s, 475 MiB | +| pretix (Python + JS, 1,472 files) | 18.8 s, 3.38 GiB | 11.0 s, 3.25 GiB | 11.2 s, 2.55 GiB | +| CPython (C + Python, 3,708 files) | 95.3 s, 8.70 GiB | 41.6 s, 4.13 GiB | 39.1 s, 3.56 GiB | +| discourse (Ruby + JS, 20,277 files) | 26.6 s, 3.74 GiB | 22.7 s, 2.90 GiB | 20.8 s, 2.23 GiB | +| supabase (React + Next.js + TS, 10,716 files) | 26.0 s, 4.46 GiB | 20.9 s, 3.76 GiB | 21.2 s, 3.65 GiB | +| n8n (Vue + TS, 24,434 files) | 163.6 s, 9.66 GiB | 71.8 s, 6.38 GiB | 72.9 s, 6.10 GiB | -The table predates three fork fixes, each measured on the same host on 2026-10-04 (Node, two runs each). A fix to Python name checks cut CPython's full index from 48.8 s to 39.5 s and its peak memory from 6.45 GiB to 5.90 GiB; pretix stayed at 10.0 s. A bound on the resolver's per-thread cache of name patterns then cut peak memory from 5.87 GiB to 4.25–4.35 GiB on CPython and from 3.54 GiB to 3.0 GiB on pretix, with the same index times and graphs. Bounding each resolver worker's cache of per-file bindings then cut peak memory from 4.23–4.29 GiB to 3.73–3.77 GiB on CPython and from 5.94–5.97 GiB to 5.29–5.48 GiB on n8n, again with the same graphs. The table itself has not been re-run. +Upstream at `6560052` takes 2.0 to 2.7 times as long to index Python as at `290e03f` (pretix 9.2 s to 18.8 s, CPython 35.6 s to 95.3 s). For pretix, [a same-host check](docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md#since-2026-09-28) shows the cause is upstream's code, not the host. **One-file sync** (edit one file, `codegraph sync`): | Corpus | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| gin | 0.47 s | 0.37 s | 0.32 s | -| Alamofire | 0.69 s | 0.51 s | 0.46 s | -| pretix | 3.01 s | 1.90 s | 1.67 s | -| CPython | 8.30 s | 4.67 s | 4.97 s | -| discourse | 6.03 s | 2.79 s | 2.64 s | -| supabase | 6.30 s | 2.94 s | 2.77 s | -| n8n | 13.2 s | 4.38 s | 4.06 s | +| gin | 0.51 s | 0.48 s | 0.41 s | +| Alamofire | 0.73 s | 0.55 s | 0.48 s | +| pretix | 3.15 s | 2.38 s | 2.38 s | +| CPython | 8.54 s | 7.87 s | 7.97 s | +| discourse | 5.83 s | 3.01 s | 2.71 s | +| supabase | 6.12 s | 3.17 s | 2.60 s | +| n8n | 12.8 s | 4.39 s | 4.12 s | Both builds rebuild the links inferred from events, callbacks and function pointers after a sync, so a sync ends with the graph a full index would build ([colbymchenry/codegraph#1988](https://github.com/colbymchenry/codegraph/issues/1988)). Upstream reruns every inference pass inside each sync that touches an inferred link; the fork reruns the same passes a full index runs once edits pause. `CODEGRAPH_SYNC_RESYNTHESIS=0` turns the fork's rebuild off, trading it for stale inferred links. @@ -222,23 +224,23 @@ Both builds rebuild the links inferred from events, callbacks and function point | | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| Start to first explore answer | 4.23 s | 2.89 s | 2.66 s | -| Explore, warm median | 707 ms | 227 ms | 223 ms | -| 8 explores at once | 3.62 s | 2.25 s | 2.51 s | -| Edited file re-indexed by the watcher | 1.40 s | 0.87 s | 0.70 s | -| Memory while busy | 4.29 GiB | 3.84 GiB | 3.14 GiB | -| Memory at idle | 4.77 GiB | 2.07 GiB | 1.22 GiB | -| CPU at idle, share of one core | 0.47% | 0.32% | 0.67% | +| Start to first explore answer | 4.07 s | 3.00 s | 2.76 s | +| Explore, warm median | 693 ms | 214 ms | 211 ms | +| 8 explores at once | 3.89 s | 2.35 s | 2.43 s | +| Memory while busy | 4.41 GiB | 3.89 GiB | 3.01 GiB | +| Edited file re-indexed by the watcher | 1.52 s | 872 ms | 683 ms | +| Memory at idle | 4.77 GiB | 2.09 GiB | 1.21 GiB | +| CPU at idle, share of one core | 0.47% | 0.33% | 0.48% | **CLI startup** on gin (`hyperfine`, mean of 20 runs): | Command | Upstream, Node | Fork, Node | Fork, Bun | |---|---|---|---| -| `codegraph --version` | 63 ms | 31 ms | 23 ms | -| `codegraph status` | 173 ms | 117 ms | 86 ms | -| `codegraph explore ""` | 224 ms | 121 ms | 99 ms | +| `codegraph --version` | 68 ms | 39 ms | 30 ms | +| `codegraph status` | 183 ms | 144 ms | 97 ms | +| `codegraph explore ""` | 243 ms | 130 ms | 101 ms | -**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun, peak index memory is 8–27% lower than on Node, the idle MCP server holds 37–41% less memory, and commands start faster. Index and sync times range from 6% slower to 15% faster than Node's. Bun has two costs. Eight concurrent explores on n8n take 12% longer; contention between worker threads reading through Bun's SQLite is the suspected cause ([oven-sh/bun#44084](https://github.com/oven-sh/bun/issues/44084), [#44187](https://github.com/oven-sh/bun/issues/44187)). Idle CPU is two to four times Node's, though still under 1% of one core. +**Node or Bun.** The fork produces the same node and edge counts on both, and the full test suite passed on Node.js 24.21.0, Node.js 26.10.0 and Bun 1.4.2 at `48903f5` (`npm run test:bun`; one test is skipped under Bun for [oven-sh/bun#42891](https://github.com/oven-sh/bun/issues/42891)). Since then the suite has grown, and the Bun run skips 14 more tests that need what Bun does not provide: patching a builtin so the library sees it (`syncBuiltinESMExports` is a no-op, [oven-sh/bun#42616](https://github.com/oven-sh/bun/pull/42616)) and an emptied `PATH` blocking a spawn ([oven-sh/bun#44948](https://github.com/oven-sh/bun/issues/44948)). Each skip is measured at run time, so the test returns when Bun supports it. On Bun, peak index memory is 3–32% lower than on Node (n8n 4%, supabase 3%), the idle MCP server holds 40–42% less memory, and commands start faster. Index times range from 17% slower (gin, 1.34 s against 1.15 s) to 8% faster than Node's, and sync times from 1% slower to 18% faster. Bun has two costs. Eight concurrent explores take 32% longer on gin (227 ms against 172 ms) and 3% longer on n8n, which is within the 3% band; contention between worker threads reading through Bun's SQLite is the suspected cause ([oven-sh/bun#44084](https://github.com/oven-sh/bun/issues/44084), [#44187](https://github.com/oven-sh/bun/issues/44187)). Idle CPU is 1.4 to 2.7 times Node's, though still under 1% of one core. Measured separately: @@ -253,10 +255,10 @@ The [benchmark](#benchmark-results) and [speed](#built-for-speed--the-rust-kerne ### What it costs -- **Larger database:** 11–21% bigger on six of the seven corpora above, and 37% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. -- **Fewer edges on some projects:** 3–5% fewer on pretix and CPython, because the fork declines links it cannot confirm. Some of those were correct links. -- **Slower full index on small projects:** at `34cc55e` (the measured build), on Node, gin and Alamofire took 0.14–0.43 s longer to index than on upstream. Two causes were found and fixed afterwards: the resolver walked a minified one-line JavaScript file's text from its start for every reference in it (Alamofire's bundled jQuery was 40% of its resolution time), and a project under about 15,000 references looked nodes up by query where loading the table is faster. In a profiling run, not the benchmark, Alamofire's resolution step fell from a median of 1.25 s to 0.93 s with the same graph. The benchmark has not been re-run. -- **On Bun:** slower concurrent explores and higher idle CPU, as above. +- **Larger database:** 12–22% bigger on the other six corpora above, and 38% on supabase, which has 1,978 Markdown files. It holds Markdown, binding rows and more nodes. +- **Fewer edges on some projects:** 8% fewer on pretix and 3% fewer on CPython, because the fork declines links it cannot confirm. Some of those were correct links. +- **A little slower on the smallest projects, on Node:** Alamofire took 1.81 s to index against upstream's 1.73 s (+5%, and the two runs per arm overlap), and gin 1.15 s against 1.13 s (within 3%). On Bun the fork is faster on both. A warm question on gin takes 26 ms against 24 ms on Node and 25 ms against 24 ms on Bun. +- **On Bun:** slower concurrent explores on small projects and higher idle CPU, as above. --- diff --git a/assets/fork-vs-upstream-performance.svg b/assets/fork-vs-upstream-performance.svg new file mode 100644 index 0000000000..10bd4d36f1 --- /dev/null +++ b/assets/fork-vs-upstream-performance.svg @@ -0,0 +1,6 @@ + +Fork vs upstream CodeGraph: headline numbers +On n8n the fork indexes in 71.8 s against 163.6 s, syncs one edited file in 4.39 s against 12.8 s, and answers a warm question in 214 ms against 693 ms. Of 76 comparisons it is better in 68, about the same in 5 and worse in 3. + +n8n and CPython: the fork indexes 2.3× faster and uses 34–53% less memoryNumbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.Index n8n from scratch2.3× faster163.6 s → 71.8 son Bun: 157.8 s → 72.9 sUpdate after editing one file2.9× faster12.8 s → 4.39 son Bun: 10.6 s → 4.12 sAnswer a question (warm)3.2× faster693 ms → 214 mson Bun: 685 ms → 211 msMemory while indexing n8n34% less9.66 GiB → 6.38 GiBon Bun: 7.28 GiB → 6.10 GiBMCP server memory when idle56% less4.77 GiB → 2.09 GiBon Bun: 3.07 GiB → 1.21 GiBStart the command line1.7× faster68 ms → 39 mson Bun: 46 ms → 30 msWhere the fork is not betterIndexing Alamofire: 1.73 s → 1.81 s on Node (+5%)Warm ask on gin (MCP server): 24 ms → 26 ms on Node (+8%), 24 ms → 25 ms on Bun (+4%)About the same (within 3%): indexing gin on Node; memory indexing pretix on Bun; memory indexing supabase on Bun;syncing one file in gin on Bun; file watcher on gin (MCP server) on Bun.Over all 76 comparisons on Node and Bun: better in 68, about the same in 5, worse in 3.Measured 2026-10-12 · upstream 6560052 vs fork ce6120be · Node 24.21.0, Bun 1.4.3 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted + diff --git a/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-12.svg b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-12.svg new file mode 100644 index 0000000000..54bca306e8 --- /dev/null +++ b/docs/benchmarks/fork-vs-upstream-heatmap-2026-10-12.svg @@ -0,0 +1,6 @@ + +Fork vs upstream CodeGraph: every result +Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of 76 comparisons it is better in 68, about the same in 5 and worse in 3. + +Every result at a glanceEach cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better by over 3%, amber = about the same, red = worse.Full index: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node+2%+5%−42%−56%−14%−19%−56%Fork on Bun−9%−19%−32%−64%−41%−28%−54%Full index: peak memoryginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−26%−16%−4%−53%−23%−16%−34%Fork on Bun−42%−47%+2%−62%−37%−3%−16%One-file sync: timeginAlamofirepretixCPythondiscoursesupabasen8nFork on Node−5%−24%−24%−8%−48%−48%−66%Fork on Bun+1%−18%−7%−7%−47%−54%−61%MCP server on n8nStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−26%−69%−40%−43%−12%−56%−30%Fork on Bun−19%−69%−31%−43%−19%−61%−33%MCP server on ginStart→answerWarm ask8 asks at onceFile watcherMemory busyMemory idleCPU idleFork on Node−28%+8%−35%−11%−37%−34%−44%Fork on Bun−20%+4%−43%−1%−55%−50%−16%Command-line startup on gin--versionstatusexploreFork on Node−43%−21%−47%Fork on Bun−34%−22%−60%Measured 2026-10-12 · upstream 6560052 vs fork ce6120be · Node 24.21.0, Bun 1.4.3 · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted + diff --git a/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md new file mode 100644 index 0000000000..70741cccbd --- /dev/null +++ b/docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md @@ -0,0 +1,239 @@ +# Fork vs upstream on Node and Bun (2026-10-12) + +The measurements behind the README's "Measured results" and "What it costs" +sections. Each build runs on Node.js 24 and on Bun, so one table covers both +the fork-versus-upstream comparison and the runtime comparison. This +supersedes [`fork-vs-upstream-node-bun-2026-10-03.md`](fork-vs-upstream-node-bun-2026-10-03.md), +which measured the same upstream build (`6560052a`) against fork `34cc55e2` +with the same arms, corpora and measured steps, under a looser host-quiet rule +than the [one below](#quiet-host-rule). [Since 2026-10-03](#since-2026-10-03) +compares the two runs. + +## Method + +The scripts are in [`scripts/benchmarks/runtime/`](../../scripts/benchmarks/runtime/): `run-index-sync.sh`, `run-mcp.sh` and `run-cli.sh`, with the four arms below. `render-charts.mjs` turns the raw output into the README chart, the heatmap and the tables on this page. + +- **Builds.** Upstream `main` at `6560052a` (release 1.6.2), the same build as + on 2026-10-03, and fork `fork/consolidated` at `ce6120be`. Each worktree + ran `npm ci`, built its own native kernel from source, and ran + `npm run build`. +- **Arms.** + + | Arm | Command | + |---|---| + | upstream, Node | `node` 24.21.0 running upstream's `dist/bin/codegraph.js` | + | fork, Node | `node` 24.21.0 running the fork's `dist/bin/codegraph.js` | + | fork, Bun | `bun` 1.4.3 running the fork's `dist/bin/codegraph.js` | + | upstream, Bun | `bun` 1.4.3 running upstream's build, with `CODEGRAPH_ALLOW_UNSAFE_NODE=1` | + + Bun reports a Node.js compatibility version of 26 or newer, and upstream + refuses Node 25 and newer, so without the override upstream exits with + `Unsupported Node.js version`. The fork needs no override. The previous run + used Bun 1.4.2. +- **Host.** WSL2 Ubuntu, 16 vCPUs, 62 GiB RAM, ext4. Each run waited until + the host was quiet (below), and the benchmark held the host lock while it + ran. +- **Environment.** `CODEGRAPH_DIR=.cg-bench`, `CODEGRAPH_TELEMETRY=0`, and + `CODEGRAPH_NO_DAEMON=1` for the CLI runs. Every timed command ran at normal + priority under `~/.local/bin/run-capped 16G` (a 16 GiB memory cap) and + `/usr/bin/time -v`. Peak memory is MaxRSS. Database size is `codegraph.db` + after a WAL checkpoint. +- **Order.** Per corpus the arms ran in the order upstream Node, fork Node, + fork Bun, upstream Bun, then the reverse. Each run did a full index + (`codegraph init --yes .` into an empty directory) and then two one-file + syncs (each appends a comment line and a one-line symbol to the same file, + then runs `codegraph sync .`). Index figures are the median of two runs; sync + figures are the median of four. +- **Corpora**, pinned to the same commits as on 2026-09-28 and 2026-10-03, + shallow, no dependencies installed: + - gin (Go) `1bd0ecf` + - Alamofire (Swift) `bda9ed5` + - pretix (Python and JavaScript) `aa14505` + - CPython (C and Python) `1e8ff18` + - discourse (Ruby and JavaScript) `ae5ba2c4` + - supabase (React, Next.js and TypeScript) `36371de1` + - n8n (Vue and TypeScript) `7cc0450a` + +Both runtimes of a build produced the same node and edge counts on every +corpus. + +### Quiet-host rule + +A run started only when swap in use was at most 64 MiB, the 1-minute load +average was under 2, no `capped-*` scope was running, and Windows CPU, read from +the Windows side, was under 20% for three samples 20 s apart. An attempt was +rejected as suspect when the command exited non-zero or a Windows CPU sample +before or after it exceeded 30%. Each corpus had one retry; the first accepted +attempt is the one reported. The two runs per arm are the spread this page +offers; no wider variation criterion was set before the run, so a difference +near the 3% reporting band is labelled no clear change. + +- **n8n index and sync:** attempt 1 was rejected (Windows CPU read 32% before + it); attempt 2 is reported. The two attempts agreed within about 4% on index time. +- **gin, Alamofire, pretix and CPython** ran under an earlier form of the rule + that required swap in use to be zero and also rejected an attempt when the + 1-minute load after it exceeded 4. That load figure includes the benchmark's + own CPU, so the rule was dropped after it rejected both pretix attempts and + one CPython attempt with Windows CPU readings of 4–17% on both sides. Those + attempts were re-evaluated under the current rule from the same readings and + accepted. Pretix's first attempt is reported. +- **Limit.** Other work on the host during a run was observed only by the + load before the run and by the Windows CPU samples before and after it, not + continuously. + +## Full index + +Time and peak memory, median of two. + +| Corpus | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---| +| gin | 1.13 s, 547 MiB | 1.15 s, 403 MiB | 1.34 s, 302 MiB | 1.48 s, 522 MiB | +| Alamofire | 1.73 s, 837 MiB | 1.81 s, 702 MiB | 1.69 s, 475 MiB | 2.09 s, 894 MiB | +| pretix | 18.8 s, 3.38 GiB | 11.0 s, 3.25 GiB | 11.2 s, 2.55 GiB | 16.4 s, 2.51 GiB | +| CPython | 95.3 s, 8.70 GiB | 41.6 s, 4.13 GiB | 39.1 s, 3.56 GiB | 107.6 s, 9.33 GiB | +| discourse | 26.6 s, 3.74 GiB | 22.7 s, 2.90 GiB | 20.8 s, 2.23 GiB | 35.5 s, 3.52 GiB | +| supabase | 26.0 s, 4.46 GiB | 20.9 s, 3.76 GiB | 21.2 s, 3.65 GiB | 29.4 s, 3.75 GiB | +| n8n | 163.6 s, 9.66 GiB | 71.8 s, 6.38 GiB | 72.9 s, 6.10 GiB | 157.8 s, 7.28 GiB | + +Graph size. Node and edge counts are identical across runtimes; database +sizes are from the Node runs: + +| Corpus | Upstream nodes / edges / database | Fork nodes / edges / database | +|---|---|---| +| gin | 2,582 / 7,802 / 8.3 MiB | 3,089 / 8,329 / 10.1 MiB | +| Alamofire | 4,600 / 12,507 / 17.1 MiB | 5,614 / 13,673 / 19.2 MiB | +| pretix | 32,322 / 80,949 / 122 MiB | 32,797 / 74,462 / 148 MiB | +| CPython | 167,253 / 620,214 / 494 MiB | 167,929 / 600,467 / 574 MiB | +| discourse | 162,890 / 299,213 / 384 MiB | 167,866 / 299,071 / 429 MiB | +| supabase | 113,457 / 284,976 / 338 MiB | 151,607 / 319,674 / 465 MiB | +| n8n | 310,034 / 1,179,729 / 1.43 GiB | 324,762 / 1,213,274 / 1.68 GiB | + +The edge counts differ because the builds resolve differently: the fork adds +Markdown, route and dispatch links, and declines call links it cannot confirm. +The timings compare the cost of building each build's own graph, not the same +graph. + +## One-file sync + +Median of four syncs. + +| Corpus | File | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---|---| +| gin | `gin.go` | 0.51 s | 0.48 s | 0.41 s | 0.41 s | +| Alamofire | `Source/Core/Session.swift` | 0.73 s | 0.55 s | 0.48 s | 0.59 s | +| pretix | `src/pretix/base/models/orders.py` | 3.15 s | 2.38 s | 2.38 s | 2.56 s | +| CPython | `Lib/os.py` | 8.54 s | 7.87 s | 7.97 s | 8.56 s | +| discourse | `app/models/user.rb` | 5.83 s | 3.01 s | 2.71 s | 5.09 s | +| supabase | `apps/studio/data/projects/clone-mutation.ts` | 6.12 s | 3.17 s | 2.60 s | 5.61 s | +| n8n | `packages/cli/src/active-executions.ts` | 12.8 s | 4.39 s | 4.12 s | 10.6 s | + +## MCP server + +The method matches the [2026-09-28 run](fork-vs-upstream-node-bun-2026-09-28.md#mcp-server): +`mcp-bench.mjs` speaks JSON-RPC to `codegraph serve --mcp --path ` over +stdio, with each build's index prebuilt by its own Node arm, two runs per arm +in mirrored order. Each run times 12 warm explores and records the 7th fastest +as its median and the 11th as its p90; the tables show the median of the two +runs. Memory and CPU are summed over the stdio server, the detached daemon it +starts, and their children. Idle CPU is the percentage of one core. + +**n8n** (edited `packages/workflow/src/workflow.ts`): + +| Measure | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---| +| `initialize` answered | 119 ms | 67 ms | 59 ms | 86 ms | +| Start to first explore answer | 4.07 s | 3.00 s | 2.76 s | 3.41 s | +| Warm explore, median | 693 ms | 214 ms | 211 ms | 685 ms | +| Warm explore, p90 | 836 ms | 295 ms | 294 ms | 799 ms | +| 8 concurrent explores | 3.89 s | 2.35 s | 2.43 s | 3.54 s | +| Memory while busy | 4.41 GiB | 3.89 GiB | 3.01 GiB | 3.73 GiB | +| Edited file re-indexed | 1.52 s | 872 ms | 683 ms | 1.19 s | +| Memory at idle | 4.77 GiB | 2.09 GiB | 1.21 GiB | 3.07 GiB | +| CPU at idle | 0.47% | 0.33% | 0.48% | 0.73% | + +**gin** (edited `gin.go`): + +| Measure | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---| +| `initialize` answered | 114 ms | 66 ms | 56 ms | 82 ms | +| Start to first explore answer | 519 ms | 375 ms | 290 ms | 361 ms | +| Warm explore, median | 24 ms | 26 ms | 25 ms | 24 ms | +| Warm explore, p90 | 34 ms | 32 ms | 29 ms | 29 ms | +| 8 concurrent explores | 265 ms | 172 ms | 227 ms | 398 ms | +| Memory while busy | 1.10 GiB | 708 MiB | 426 MiB | 948 MiB | +| Edited file re-indexed | 436 ms | 386 ms | 385 ms | 387 ms | +| Memory at idle | 1.10 GiB | 743 MiB | 449 MiB | 899 MiB | +| CPU at idle | 0.28% | 0.16% | 0.43% | 0.51% | + +On gin, a warm explore takes 26 ms on the fork against 24 ms on upstream on +Node, and 25 ms against 24 ms on Bun. Each figure is the median of two runs of +12 explores, so a 1–2 ms difference is close to the timer's noise. It is the +only place a warm explore is slower. + +Bun's concurrent explores on gin are slower than the fork on Node (227 ms +against 172 ms) and 3% slower on n8n (2.43 s against 2.35 s), which is inside +the 3% band. Contention between worker threads reading through Bun's SQLite is +the suspected cause, not confirmed +([oven-sh/bun#44187](https://github.com/oven-sh/bun/issues/44187)). Bun's idle +CPU is 1.4 to 2.7 times Node's, still under 1% of one core. + +## CLI startup + +`hyperfine -N --warmup 3 --runs 20` on gin with a prebuilt index and +`CODEGRAPH_NO_DAEMON=1`, mean ± standard deviation: + +| Command | Upstream, Node | Fork, Node | Fork, Bun | Upstream, Bun | +|---|---|---|---|---| +| `codegraph --version` | 68 ± 4 ms | 39 ± 2 ms | 30 ± 1 ms | 46 ± 2 ms | +| `codegraph status .` | 183 ± 10 ms | 144 ± 10 ms | 97 ± 5 ms | 125 ± 6 ms | +| `codegraph explore ''` | 243 ± 14 ms | 130 ± 6 ms | 101 ± 3 ms | 256 ± 37 ms | + +## Every result at a glance + +![Heatmap of the fork's change against upstream for index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. The fork is better in 68 of 76 comparisons, about the same in 5 and worse in 3.](fork-vs-upstream-heatmap-2026-10-12.svg) + +Each cell is the fork's change against upstream on the same runtime (Node +against Node, Bun against Bun). Green is better by more than 3%, amber is +within 3%, red is worse. Of the 76 comparisons, the fork is better in 68, +about the same in 5, and worse in 3: indexing Alamofire on Node (+5%, with +overlapping runs) and the warm explore on gin on Node (+8%) and on Bun (+4%). + +## Since 2026-10-03 + +Full index time on Node, upstream `6560052a` in both runs: + +| Corpus | Upstream, 2026-10-03 | Upstream, 2026-10-12 | Fork `34cc55e2` | Fork `ce6120be` | Fork as a share of upstream, then and now | +|---|---|---|---|---|---| +| gin | 1.03 s | 1.13 s | 1.17 s | 1.15 s | 114% → 102% | +| Alamofire | 1.62 s | 1.73 s | 2.05 s | 1.81 s | 127% → 105% | +| pretix | 21.1 s | 18.8 s | 10.8 s | 11.0 s | 51% → 58% | +| CPython | 94.7 s | 95.3 s | 45.5 s | 41.6 s | 48% → 44% | +| discourse | 23.7 s | 26.6 s | 25.1 s | 22.7 s | 106% → 86% | +| supabase | 26.7 s | 26.0 s | 23.1 s | 20.9 s | 87% → 81% | +| n8n | 168.9 s | 163.6 s | 78.4 s | 71.8 s | 46% → 44% | + +Upstream is the same build in both runs, so its change between runs (from 11% +faster to 12% slower) is the spread of the host and the method, and the +fork's own change since 2026-10-03 should be read against that spread. The +last column is the steadier comparison. It shows the earlier losses on gin, +Alamofire and discourse gone or inside the band. pretix moved the other way, +from 51% to 58% of upstream's time (the fork 10.8 s to 11.0 s, upstream 11% +faster this time); that difference is inside the spread above. + +Between `34cc55e2` and `ce6120be` the fork gained 245 commits, among them +upstream merges and #477, which changed how a small project's references are +resolved (a project under about 15,000 references looks nodes up by query, and +a minified one-line file's text is no longer walked from its start for every +reference in it). This run did not measure each change on its own. + +The fork's pretix graph has 74,462 edges against 77,554 on 2026-10-03; its +CPython graph has 600,467 against 599,991. The cause of the pretix change was +not investigated. + +## Test suite + +The full suite on the #477 branch, which became `ce6120be`, passed on +Node.js 24.21.0: 696 test files, 8,905 tests passed and 39 skipped. It was not +re-run on Node.js 26 or Bun for this measurement; the last passes on +Node.js 26.10.0 and Bun 1.4.2 were at `48903f51`, in the +[2026-09-28 run](fork-vs-upstream-node-bun-2026-09-28.md#test-suite). diff --git a/scripts/benchmarks/runtime/README.md b/scripts/benchmarks/runtime/README.md index b9c6a0b2ce..36f902e828 100644 --- a/scripts/benchmarks/runtime/README.md +++ b/scripts/benchmarks/runtime/README.md @@ -1,7 +1,7 @@ # Runtime benchmark These scripts produced the README's "Measured results" and the tables in -[`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md`](../../../docs/benchmarks/fork-vs-upstream-node-bun-2026-10-03.md). +[`docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md`](../../../docs/benchmarks/fork-vs-upstream-node-bun-2026-10-12.md). Each arm is a runtime (a `node` or `bun` executable) running a built checkout. The arms can differ in build, in runtime, or in both. @@ -11,6 +11,7 @@ The arms can differ in build, in runtime, or in both. | `run-mcp.sh ` | MCP server: start to first explore answer, warm explore median and p90, 8 explores at once, watcher sync, idle memory and CPU | `mcp.jsonl` | | `run-cli.sh ` | CLI start: `--version`, `status`, one `explore` (hyperfine, 20 runs) | `cli-*.json`, `cli.txt` | | `summarize.py $BENCH_OUT` | Medians per corpus and arm from `index-sync.tsv`, and from `mcp.jsonl` when present | stdout | +| `render-charts.mjs $BENCH_OUT --date D --upstream SHA --fork SHA --node V --bun V` | The README chart (`--cards`), the benchmark doc's heatmap (`--heatmap`) and the current run's result tables as Markdown (`--tables`; the benchmark doc's comparison with an earlier run is written by hand), from the accepted attempts in `$BENCH_OUT` (directories `index--aN`, `mcp--aN` and `cli-gin-aN` whose `status` file reads `accepted`) | SVG and Markdown files | Every run needs a quiet host. Each runner waits before a corpus until no `capped-*` scope runs and the 1-minute load is under 2, and each arm runs twice @@ -23,9 +24,12 @@ The 2026-10-03 run behind the README took 1 h 47 min (17:09 to 18:56): `run-mcp.sh` on gin and n8n and `run-cli.sh` on gin took 48 min, with four arms (upstream and fork, each on Node and Bun). That includes the idle-host waits. The n8n MCP runs took about 4 min each. Building the two worktrees took 4 min -before it. The 2026-09-28 run of the same steps took 1 h 26 min. Start a re-run -estimate from these numbers, and update them here after a run that takes much -longer or shorter. +before it. The 2026-09-28 run of the same steps took 1 h 26 min. The +2026-10-12 run took 3 h 2 min (00:16 to 03:18), most of it waiting for the host +to go quiet and re-running attempts it rejected. Its measuring time per corpus +was about 5 min for discourse and supabase, 18 min for n8n index and sync, and +26 min for the n8n MCP run. Start a re-run estimate from these numbers, and +update them here after a run that takes much longer or shorter. ## Arms diff --git a/scripts/benchmarks/runtime/render-charts.mjs b/scripts/benchmarks/runtime/render-charts.mjs new file mode 100644 index 0000000000..7fda31ca36 --- /dev/null +++ b/scripts/benchmarks/runtime/render-charts.mjs @@ -0,0 +1,257 @@ +#!/usr/bin/env node +// Turns a benchmark run's raw output into the README chart, the benchmark doc's +// heatmap and the Markdown tables both pages quote, so no figure is typed by hand. +// +// node scripts/benchmarks/runtime/render-charts.mjs $BENCH_OUT \ +// --date 2026-10-12 --upstream 6560052 --fork ce6120be --node 24.21.0 --bun 1.4.3 \ +// [--cards assets/fork-vs-upstream-performance.svg] \ +// [--heatmap docs/benchmarks/fork-vs-upstream-heatmap-2026-10-12.svg] [--tables tables.md] +// +// Per corpus the accepted attempt is the first `index--aN` directory whose +// `status` file reads `accepted`; the same goes for `mcp--aN` and `cli-gin-aN`. +// Medians are taken over the runs in that attempt (two per arm, mirrored order). +// A change counts as better or worse only outside the 3% reporting band. +import { readFileSync, readdirSync, existsSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +const argv = process.argv.slice(2); +const root = argv.find((a, i) => !a.startsWith('--') && !(argv[i - 1] ?? '').startsWith('--')); +const opt = (k, d) => { const i = argv.indexOf(`--${k}`); return i >= 0 ? argv[i + 1] : d; }; +if (!root) { console.error('usage: render-charts.mjs --date D --upstream SHA --fork SHA --node V --bun V [--cards F] [--heatmap F] [--tables F]'); process.exit(2); } +const META = { date: opt('date'), upstream: opt('upstream'), fork: opt('fork'), node: opt('node'), bun: opt('bun') }; +for (const [k, v] of Object.entries(META)) if (!v) { console.error(`missing --${k}`); process.exit(2); } + +const ARMS = ['upnode', 'forknode', 'forkbun', 'upbun']; +const CORPORA = [['gin', 'gin'], ['Alamofire', 'Alamofire'], ['pretix', 'pretix'], ['cpython', 'CPython'], ['discourse', 'discourse'], ['supabase', 'supabase'], ['n8n', 'n8n']]; +const BAND = 0.03; +const median = (a) => { const s = [...a].sort((x, y) => x - y), m = s.length >> 1; return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2; }; +const seconds = (t) => { const p = t.split(':'); return parseFloat(p.at(-1)) + 60 * parseFloat(p.at(-2)); }; +const accepted = (prefix) => { + const attempt = (d) => +d.slice(d.lastIndexOf('-a') + 2); + const dir = readdirSync(root).filter((d) => d.startsWith(prefix + '-a') && Number.isInteger(attempt(d))).sort((x, y) => attempt(x) - attempt(y)) + .find((d) => existsSync(join(root, d, 'status')) && readFileSync(join(root, d, 'status'), 'utf8').trim() === 'accepted'); + if (!dir) throw new Error(`no accepted attempt for ${prefix}`); + return join(root, dir); +}; + +// ---- load +const index = {}; +for (const [key] of CORPORA) { + const rows = readFileSync(join(accepted(`index-${key}`), 'index-sync.tsv'), 'utf8').trim().split('\n').map((l) => l.split('\t')); + index[key] = {}; + for (const arm of ARMS) { + const i = rows.filter((r) => r[1] === arm && r[2] === 'index'); + const s = rows.filter((r) => r[1] === arm && r[2] !== 'index'); + if (!i.length || !s.length) throw new Error(`${key} ${arm}: missing rows`); + index[key][arm] = { + indexS: median(i.map((r) => seconds(r[4]))), peakMiB: median(i.map((r) => +r[5])) / 1024, + dbMiB: median(i.map((r) => +r[7])) / 2 ** 20, nodes: +i[0][8], edges: +i[0][9], + syncS: median(s.map((r) => seconds(r[4]))), runs: i.length, syncRuns: s.length, + }; + } +} +const mcp = {}; +for (const key of ['gin', 'n8n']) { + const runs = readFileSync(join(accepted(`mcp-${key}`), 'mcp.jsonl'), 'utf8').trim().split('\n').map((l) => JSON.parse(l)); + mcp[key] = {}; + for (const arm of ARMS) { + const x = runs.filter((r) => r.arm === arm); + const g = (k) => { + // A timed-out watcher sync is recorded as null; publishing the other run alone would mislead. + if (x.length === 0 || x.some((r) => r[k] == null)) throw new Error(`mcp ${key} ${arm}: ${k} is missing in ${x.length} run(s)`); + return median(x.map((r) => r[k])); + }; + mcp[key][arm] = { + initMs: g('initMs'), firstMs: g('coldToFirstAnswerMs'), warmMs: g('warmMedianMs'), p90Ms: g('warmP90Ms'), + wave8Ms: g('wave8Ms'), watchMs: g('watchSyncMs'), busyMiB: g('busy_totalMB'), idleMiB: g('idle_totalMB'), idleCpu: g('idleCpuPct'), + }; + } +} +const cli = {}; +{ + const dir = accepted('cli-gin'); + for (const [n, label] of [['version', '--version'], ['status', 'status'], ['explore', 'explore']]) { + const res = JSON.parse(readFileSync(join(dir, `cli-${n}.json`), 'utf8')).results; + cli[n] = { label }; + for (const arm of ARMS) { + const w = res.find((r) => r.name === arm).summary.time_wall_clock; + cli[n][arm] = { mean: w.mean * 1000, sd: w.stddev * 1000 }; + } + } +} + +// ---- formatting +const fSec = (v) => `${v < 10 ? v.toFixed(2) : v.toFixed(1)} s`; +const fMem = (miB) => (miB >= 1024 ? `${(miB / 1024).toFixed(2)} GiB` : miB < 100 ? `${miB.toFixed(1)} MiB` : `${Math.round(miB)} MiB`); +const fMs = (v) => (v >= 1000 ? fSec(v / 1000) : `${Math.round(v)} ms`); +const fPct = (v) => `${v.toFixed(2)}%`; +const pctChange = (r) => { if (!Number.isFinite(r)) return 'n/a'; const d = Math.round((r - 1) * 100); return (d > 0 ? '+' : d < 0 ? '−' : '') + Math.abs(d) + '%'; }; +const esc = (s) => String(s).replace(/&/g, '&').replace(/ [a, cli[n][a].mean])), fMs]); +function pick(o, k) { return Object.fromEntries(ARMS.map((a) => [a, o[a][k]])); } + +// Two zero measurements are equal; a zero baseline with a positive fork value is Infinity. +const ratio = (m, rt) => { + const [up, fk] = rt === 'Node' ? [m[2].upnode, m[2].forknode] : [m[2].upbun, m[2].forkbun]; + return up === 0 ? (fk === 0 ? 1 : Infinity) : fk / up; +}; +const cmp = []; +for (const m of metrics) for (const rt of ['Node', 'Bun']) cmp.push({ m, rt, r: ratio(m, rt) }); +const nBetter = cmp.filter((c) => c.r < 1 - BAND).length; +const nSame = cmp.filter((c) => c.r >= 1 - BAND && c.r <= 1 + BAND).length; +const nWorse = cmp.filter((c) => c.r > 1 + BAND).length; + +// ---- SVG pieces +const C = { bg: '#0d1117', panel: '#161b22', text: '#e6edf3', muted: '#8b949e', good: '#2ea043', same: '#d29922', bad: '#f85149' }; +const col = (r) => (r < 1 - BAND ? C.good : r > 1 + BAND ? C.bad : C.same); +const FONT = "-apple-system, BlinkMacSystemFont, 'Segoe UI', Helvetica, Arial, sans-serif"; +const T = (x, y, s, o = {}) => `${esc(s)}`; +const wrap = (W, H, body, title, desc) => `\n${esc(title)}\n${esc(desc)}\n\n${body}\n\n`; +const FOOT = `Measured ${META.date} · upstream ${META.upstream} vs fork ${META.fork} · Node ${META.node}, Bun ${META.bun} · 16-vCPU WSL2 · two runs per arm, median · fork on Node unless noted`; +const find = (group, label) => metrics.find((m) => m[0] === group && m[1] === label); + +function cards() { + const W = 880; + const idx = (name) => find('Full index: time', name), mem = (name) => find('Full index: peak memory', name); + const speedups = ['n8n', 'CPython'].map((n) => 1 / ratio(idx(n), 'Node')); + const savings = ['n8n', 'CPython'].map((n) => (1 - ratio(mem(n), 'Node')) * 100); + const lo = (a) => Math.min(...a), hi = (a) => Math.max(...a); + const sp = lo(speedups).toFixed(1) === hi(speedups).toFixed(1) ? `${lo(speedups).toFixed(1)}×` : `${lo(speedups).toFixed(1)}–${hi(speedups).toFixed(1)}×`; + const title = `n8n and CPython: the fork indexes ${sp} faster and uses ${Math.round(lo(savings))}–${Math.round(hi(savings))}% less memory`; + const card = [ + ['Index n8n from scratch', idx('n8n'), 'time'], + ['Update after editing one file', find('One-file sync: time', 'n8n'), 'time'], + ['Answer a question (warm)', find('MCP server on n8n', 'Warm ask'), 'time'], + ['Memory while indexing n8n', mem('n8n'), 'mem'], + ['MCP server memory when idle', find('MCP server on n8n', 'Memory idle'), 'mem'], + ['Start the command line', find('Command-line startup on gin', '--version'), 'time'], + ]; + let b = T(24, 38, title, { size: 19, w: 700 }); + b += T(24, 60, 'Numbers are for n8n (the largest test repo) and the command line. Lower is better. Upstream → fork.', { size: 12, fill: C.muted }); + card.forEach(([t, m, kind], i) => { + const x = 24 + (i % 3) * 284, y = 82 + Math.floor(i / 3) * 150, r = ratio(m, 'Node'); + const badge = kind === 'mem' ? `${Math.round((1 - r) * 100)}% less` : `${(1 / r).toFixed(1)}× faster`; + const f = m[3], v = m[2]; + b += ``; + b += T(x + 16, y + 26, t, { size: 13, fill: C.muted }); + b += T(x + 16, y + 72, badge, { size: 30, w: 700, fill: col(r) }); + b += T(x + 16, y + 100, `${f(v.upnode)} → ${f(v.forknode)}`, { size: 15 }); + b += T(x + 16, y + 120, `on Bun: ${f(v.upbun)} → ${f(v.forkbun)}`, { size: 12, fill: C.muted }); + }); + + // every comparison that is not clearly better, grouped by what was measured + const phrase = { 'Full index: time': (l) => `Indexing ${l}`, 'Full index: peak memory': (l) => `Memory indexing ${l}`, 'One-file sync: time': (l) => `Syncing one file in ${l}` }; + const name = (m) => (phrase[m[0]] ? phrase[m[0]](m[1]) : m[0].startsWith('MCP server on ') ? `${m[1]} on ${m[0].slice(14)} (MCP server)` : `codegraph ${m[1]}`); + const byMetric = new Map(); + for (const c of cmp) if (c.r > 1 + BAND) { const k = c.m; (byMetric.get(k) ?? byMetric.set(k, []).get(k)).push(c); } + const worse = [...byMetric].map(([m, cs]) => `${name(m)}: ` + cs.map((c) => { const v = c.rt === 'Node' ? [m[2].upnode, m[2].forknode] : [m[2].upbun, m[2].forkbun]; return `${m[3](v[0])} → ${m[3](v[1])} on ${c.rt} (${pctChange(c.r)})`; }).join(', ')); + const lower = (s) => s[0].toLowerCase() + s.slice(1); + const same = cmp.filter((c) => c.r >= 1 - BAND && c.r <= 1 + BAND).map((c) => `${lower(name(c.m))} on ${c.rt}`); + const wrapText = (s, n) => { const out = []; let cur = ''; for (const w of s.split(' ')) { if ((cur + ' ' + w).trim().length > n) { out.push(cur); cur = w; } else cur = (cur + ' ' + w).trim(); } if (cur) out.push(cur); return out; }; + const lines = [...worse, `About the same (within 3%): ${same.join('; ')}.`].flatMap((l) => wrapText(l, 118)); + const boxH = 44 + lines.length * 22; + let y = 392; + b += ``; + b += T(40, y + 26, 'Where the fork is not better', { size: 14, w: 700, fill: C.bad }); + lines.forEach((s, i) => { b += T(40, y + 54 + i * 22, s, { size: 12.5 }); }); + y += boxH + 26; + b += T(24, y, `Over all ${cmp.length} comparisons on Node and Bun: better in ${nBetter}, about the same in ${nSame}, worse in ${nWorse}.`, { size: 12, fill: C.muted }); + y += 20; + b += T(24, y, FOOT, { size: 11, fill: C.muted }); + const n8 = index.n8n; + const desc = `On n8n the fork indexes in ${fSec(n8.forknode.indexS)} against ${fSec(n8.upnode.indexS)}, syncs one edited file in ${fSec(n8.forknode.syncS)} against ${fSec(n8.upnode.syncS)}, and answers a warm question in ${fMs(mcp.n8n.forknode.warmMs)} against ${fMs(mcp.n8n.upnode.warmMs)}. Of ${cmp.length} comparisons it is better in ${nBetter}, about the same in ${nSame} and worse in ${nWorse}.`; + return { svg: wrap(W, y + 16, b, 'Fork vs upstream CodeGraph: headline numbers', desc), lines }; +} + +function heatmap() { + const W = 880; + let y = 36, b = T(24, y, 'Every result at a glance', { size: 20, w: 700 }); + y += 22; + b += T(24, y, 'Each cell is the fork’s change versus upstream. Negative is better (less time or memory). Green = better by over 3%, amber = about the same, red = worse.', { size: 12, fill: C.muted }); + y += 26; + const groups = [...new Set(metrics.map((m) => m[0]))]; + for (const g of groups) { + const cols = metrics.filter((m) => m[0] === g), LX = 24, LW = 70, CW = Math.min(112, (W - LX - LW - 24) / cols.length), CH = 34; + b += T(24, y, g, { size: 14, w: 700 }); + y += 14; + cols.forEach((c, i) => { b += T(LX + LW + i * CW + CW / 2, y + 10, c[1], { size: 11, fill: C.muted, a: 'middle' }); }); + y += 18; + for (const rt of ['Node', 'Bun']) { + b += T(LX, y + CH / 2 + 4, `Fork on ${rt}`, { size: 11, fill: C.muted }); + cols.forEach((c, i) => { + const r = ratio(c, rt), op = r > 1 + BAND ? 0.85 : r < 1 - BAND ? 0.25 + Math.min(0.6, 1 - r) : 0.55; + b += ``; + b += T(LX + LW + i * CW + CW / 2, y + CH / 2 + 3, pctChange(r), { size: 13, w: 700, a: 'middle' }); + }); + y += CH; + } + y += 18; + } + b += T(24, y + 4, FOOT, { size: 11, fill: C.muted }); + const desc = `Heatmap of the fork's change against upstream for full index time, peak memory and one-file sync on seven repositories, the MCP server on n8n and gin, and command-line startup. Of ${cmp.length} comparisons it is better in ${nBetter}, about the same in ${nSame} and worse in ${nWorse}.`; + return wrap(W, y + 24, b, 'Fork vs upstream CodeGraph: every result', desc); +} + +// ---- Markdown tables +function tables() { + const L = []; + const row = (cells) => `| ${cells.join(' | ')} |`; + const cell = (key, arm, k, f) => f(index[key][arm][k]); + L.push('## Full index, time and peak memory (README: Upstream Node, Fork Node, Fork Bun)', ''); + L.push(row(['Corpus', 'Upstream, Node', 'Fork, Node', 'Fork, Bun']), row(['---', '---', '---', '---'])); + for (const [key, name] of CORPORA) L.push(row([name, ...['upnode', 'forknode', 'forkbun'].map((a) => `${fSec(index[key][a].indexS)}, ${fMem(index[key][a].peakMiB)}`)])); + L.push('', '## Full index (doc: all four arms)', '', row(['Corpus', 'Upstream, Node', 'Fork, Node', 'Fork, Bun', 'Upstream, Bun']), row(['---', '---', '---', '---', '---'])); + for (const [key, name] of CORPORA) L.push(row([name, ...ARMS.map((a) => `${fSec(index[key][a].indexS)}, ${fMem(index[key][a].peakMiB)}`)])); + L.push('', '## Graph size', '', row(['Corpus', 'Upstream nodes / edges / database', 'Fork nodes / edges / database']), row(['---', '---', '---'])); + const num = (n) => n.toLocaleString('en-US'); + for (const [key, name] of CORPORA) L.push(row([name, ...['upnode', 'forknode'].map((a) => `${num(index[key][a].nodes)} / ${num(index[key][a].edges)} / ${fMem(index[key][a].dbMiB)}`)])); + L.push('', '## One-file sync (README)', '', row(['Corpus', 'Upstream, Node', 'Fork, Node', 'Fork, Bun']), row(['---', '---', '---', '---'])); + for (const [key, name] of CORPORA) L.push(row([name, ...['upnode', 'forknode', 'forkbun'].map((a) => fSec(index[key][a].syncS))])); + L.push('', '## One-file sync (doc)', '', row(['Corpus', 'Upstream, Node', 'Fork, Node', 'Fork, Bun', 'Upstream, Bun']), row(['---', '---', '---', '---', '---'])); + for (const [key, name] of CORPORA) L.push(row([name, ...ARMS.map((a) => fSec(index[key][a].syncS))])); + for (const key of ['n8n', 'gin']) { + L.push('', `## MCP server on ${key}`, '', row(['Measure', 'Upstream, Node', 'Fork, Node', 'Fork, Bun', 'Upstream, Bun']), row(['---', '---', '---', '---', '---'])); + for (const [label, k, f] of [['`initialize` answered', 'initMs', fMs], ['Start to first explore answer', 'firstMs', fMs], ['Warm explore, median', 'warmMs', fMs], ['Warm explore, p90', 'p90Ms', fMs], + ['8 concurrent explores', 'wave8Ms', fMs], ['Memory while busy', 'busyMiB', fMem], ['Edited file re-indexed', 'watchMs', fMs], ['Memory at idle', 'idleMiB', fMem], ['CPU at idle', 'idleCpu', fPct]]) + L.push(row([label, ...ARMS.map((a) => f(mcp[key][a][k]))])); + } + L.push('', '## CLI startup on gin (mean ± sd)', '', row(['Command', 'Upstream, Node', 'Fork, Node', 'Fork, Bun', 'Upstream, Bun']), row(['---', '---', '---', '---', '---'])); + for (const n of ['version', 'status', 'explore']) L.push(row([cli[n].label, ...ARMS.map((a) => `${Math.round(cli[n][a].mean)} ± ${Math.round(cli[n][a].sd)} ms`)])); + // derived statements the README prose quotes + const lines = []; + const range = (f, keys) => { const v = keys.map(f); return [Math.min(...v), Math.max(...v)]; }; + const ks = CORPORA.map((c) => c[0]); + const pct = (a, b) => (1 - a / b) * 100; + const fr = (lo, hi) => `${Math.round(lo)}–${Math.round(hi)}%`; + L.push('', '## Derived (README prose)', ''); + const bunPeak = range((k) => pct(index[k].forkbun.peakMiB, index[k].forknode.peakMiB), ks); + L.push(`- Bun peak index memory vs Node (fork), % lower: ${fr(...bunPeak)} (${ks.map((k) => `${k} ${pct(index[k].forkbun.peakMiB, index[k].forknode.peakMiB).toFixed(0)}`).join(', ')})`); + L.push(`- Bun idle MCP memory vs Node (fork), % lower: ${['n8n', 'gin'].map((k) => `${k} ${pct(mcp[k].forkbun.idleMiB, mcp[k].forknode.idleMiB).toFixed(0)}`).join(', ')}`); + L.push(`- Bun index time vs Node (fork), % change (negative faster): ${ks.map((k) => `${k} ${((index[k].forkbun.indexS / index[k].forknode.indexS - 1) * 100).toFixed(0)}`).join(', ')}`); + L.push(`- Bun sync time vs Node (fork), % change: ${ks.map((k) => `${k} ${((index[k].forkbun.syncS / index[k].forknode.syncS - 1) * 100).toFixed(0)}`).join(', ')}`); + L.push(`- Bun 8 concurrent explores vs Node (fork), % change: ${['n8n', 'gin'].map((k) => `${k} ${((mcp[k].forkbun.wave8Ms / mcp[k].forknode.wave8Ms - 1) * 100).toFixed(0)}`).join(', ')}`); + L.push(`- Idle CPU, fork Bun / fork Node: ${['n8n', 'gin'].map((k) => `${k} ${(mcp[k].forkbun.idleCpu / mcp[k].forknode.idleCpu).toFixed(1)}x (${fPct(mcp[k].forkbun.idleCpu)} vs ${fPct(mcp[k].forknode.idleCpu)})`).join(', ')}`); + L.push(`- Database size, fork vs upstream, % bigger: ${ks.map((k) => `${k} ${((index[k].forknode.dbMiB / index[k].upnode.dbMiB - 1) * 100).toFixed(0)}`).join(', ')}`); + L.push(`- Edges, fork vs upstream, % change: ${ks.map((k) => `${k} ${((index[k].forknode.edges / index[k].upnode.edges - 1) * 100).toFixed(1)}`).join(', ')}`); + L.push(`- Node/edge counts identical across runtimes: ${ks.every((k) => index[k].upnode.nodes === index[k].upbun.nodes && index[k].forknode.nodes === index[k].forkbun.nodes && index[k].upnode.edges === index[k].upbun.edges && index[k].forknode.edges === index[k].forkbun.edges)}`); + L.push(`- Comparisons: ${cmp.length}; better ${nBetter}, about the same ${nSame}, worse ${nWorse}`); + return L.join('\n') + '\n'; +} + +const cardOut = cards(); +const cardsPath = opt('cards'), heatPath = opt('heatmap'), tablesPath = opt('tables'); +if (cardsPath) writeFileSync(cardsPath, cardOut.svg); +if (heatPath) writeFileSync(heatPath, heatmap()); +const md = tables(); +if (tablesPath) writeFileSync(tablesPath, md); else process.stdout.write(md); +console.error(`comparisons ${cmp.length}: better ${nBetter}, same ${nSame}, worse ${nWorse}`); +for (const l of cardOut.lines) console.error(' not-better line: ' + l);