From c13bd0890f8b3d968b5635e5518249326f17ad44 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 05:22:18 +0000 Subject: [PATCH 001/363] fix(wsl): keep WSL off Windows' index on /mnt drives, and explain the I/O error (#995) (#2061) When Windows-native and WSL CodeGraph both open the same .codegraph/codegraph.db under /mnt//, SQLite's locking and -shm shared memory don't hold across the 9p/DrvFs bridge and WSL fails with a bare "disk I/O error". - A SQLite I/O error on the default index of a WSL /mnt/ project is rewritten (in the SQLite adapter, so every open and query is covered) into the CODEGRAPH_DIR=.codegraph-wsl instruction. MCP answers it success-shaped, and a failed default-project open is reported instead of "no project loaded". - On WSL + /mnt/ with CODEGRAPH_DIR unset, the data dir is resolved per root: an existing .codegraph-wsl/ wins, then an existing .codegraph/ index (kept, no silent re-index), else a fresh index goes in .codegraph-wsl/. `codegraph init` says where it went. Every other host is unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + README.md | 2 +- __tests__/wsl-shared-index.test.ts | 362 +++++++++++++++++++++++ site/src/content/docs/troubleshooting.md | 2 +- src/bin/codegraph.ts | 11 +- src/db/sqlite-adapter.ts | 56 +++- src/db/wsl-shared-index.ts | 98 ++++++ src/directory.ts | 39 ++- src/installer/index.ts | 7 +- src/mcp/engine.ts | 6 +- src/mcp/tools.ts | 39 +++ src/sync/watch-policy.ts | 20 +- 12 files changed, 627 insertions(+), 17 deletions(-) create mode 100644 __tests__/wsl-shared-index.test.ts create mode 100644 src/db/wsl-shared-index.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index da9629214d..45bafb8d2c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -217,6 +217,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Edits inside a symlinked directory now auto-sync on macOS, Windows and Linux, including links added, removed or retargeted while watching. Thanks @ztibeike for the report and @dengzhongyuan365-dev. (#770) - A project queried through `projectPath` now catches up on first access and stays in sync like the default project, coordinating with any daemon that already owns it. Thanks @nakisen for the report and @danusha2345. (#1835) - Two spellings of one repository (a symlink, or different casing on a case-insensitive drive such as WSL `/mnt/c`) now share one index connection instead of opening it twice. Thanks @yjtdkj for the report and @inth3shadows. (#1057) +- In WSL, a project on a Windows drive (`/mnt/c/…`) indexed for the first time now gets its own `.codegraph-wsl/` index, so it never shares one with CodeGraph on Windows; an index you already have in `.codegraph/` stays where it is, and `CODEGRAPH_DIR` still picks the name when set. Thanks @Jek-Mena for the report and @inth3shadows for narrowing it down. (#995) +- In WSL, a project on a Windows drive whose `.codegraph/` index CodeGraph on Windows also uses no longer fails with a bare "disk I/O error": the CLI, and your agent through MCP, now explain that Windows and WSL can't share one index there and that setting `CODEGRAPH_DIR=.codegraph-wsl` in WSL gives it its own. (#995) - Direct (non-daemon) MCP sessions now serve concurrent calls from a small off-thread query pool (2 workers by default; `CODEGRAPH_QUERY_POOL_SIZE` changes it). Thanks @blabla-yy. (#1465) - `impact --depth N`, and callers/callees at a depth, now find every dependent within N hops even when a longer path reaches one of them first. Thanks @inth3shadows for the report and @danusha2345. (#1974) - Path finding no longer re-queues the same node many times through a dense fan-in hub. Thanks @inth3shadows. (#1359) diff --git a/README.md b/README.md index 693fcd487a..17efcad3d5 100644 --- a/README.md +++ b/README.md @@ -892,7 +892,7 @@ Framework routing is validated the same way, on a canonical app per framework: E **Missing symbols** — The MCP server auto-syncs on save (wait a couple seconds). Run `codegraph sync` manually if needed. Check that the file's language is supported and isn't inside a `.gitignore`d or default-excluded directory (e.g. `node_modules`, `dist`). -**Sharing one checkout between Windows and WSL** — Don't point both at the same `.codegraph/`: the background-server lock and the SQLite index are tied to the OS that wrote them, and SQLite locking across the WSL2/Windows filesystem boundary is unreliable. Give each side its own index in the same tree by setting `CODEGRAPH_DIR` to a distinct name on one of them — e.g. `CODEGRAPH_DIR=.codegraph-win` on Windows, leaving WSL on the default `.codegraph`. CodeGraph skips any sibling `.codegraph-*` directory when indexing and watching, so the two never trip over each other. +**Sharing one checkout between Windows and WSL** — Don't point both at the same `.codegraph/`: the background-server lock and the SQLite index are tied to the OS that wrote them, and SQLite locking across the WSL2/Windows filesystem boundary is unreliable (WSL reports it as a `disk I/O error`). For a project on a Windows drive (a `/mnt/c/…` path), WSL keeps its own index automatically: an index first built from WSL goes in `.codegraph-wsl/`, leaving `.codegraph/` to Windows. An index already in `.codegraph/` stays where it is, so if Windows built that one, give WSL its own by setting `CODEGRAPH_DIR=.codegraph-wsl` in WSL and running `codegraph init` there. `CODEGRAPH_DIR` always picks the name when set, on either side. CodeGraph skips any sibling `.codegraph-*` directory when indexing and watching, so the two never trip over each other. **Very large repositories (hundreds of thousands of files), or a large `.codegraph/codegraph.db-wal` file** — The `-wal` file is SQLite's write-ahead log: writes waiting to be folded into `codegraph.db`. While a big index is being built, CodeGraph lets it grow in proportion to the index (soft threshold = the larger of 256 MB and a quarter of the index size, up to 2 GB) before folding it back, because folding too often is what made large indexes slow on ordinary disks. At rest it is trimmed to 64 MB, and a leftover from a killed session is folded and trimmed the next time the project opens — the index itself has no size limit. Two environment variables tune this: `CODEGRAPH_WAL_VALVE_MB` (the soft threshold during indexing) and `CODEGRAPH_WAL_HEAL_MB` (the resting size and the trim threshold). `CODEGRAPH_WAL_VALVE_DEBUG=1` prints every decision to stderr. diff --git a/__tests__/wsl-shared-index.test.ts b/__tests__/wsl-shared-index.test.ts new file mode 100644 index 0000000000..a23a1a6859 --- /dev/null +++ b/__tests__/wsl-shared-index.test.ts @@ -0,0 +1,362 @@ +/** + * Windows + WSL sharing one index on a Windows drive (issue #995). + * + * When Windows-native CodeGraph and WSL CodeGraph both open the same + * `.codegraph/codegraph.db` under `/mnt//`, SQLite's locking and `-shm` + * shared memory don't hold across the 9p/DrvFs bridge and WSL fails with a bare + * "disk I/O error". These tests pin both halves of the fix: + * + * - a fresh WSL index there gets its own `.codegraph-wsl`, while an index + * already in `.codegraph` stays where it is (no silent re-index), and + * - such an error is rewritten into the `CODEGRAPH_DIR=.codegraph-wsl` + * instruction — only on WSL, only under `/mnt/`, only for the + * default-named index — which MCP answers SUCCESS-shaped (never `isError`, + * see AGENTS.md "Errors teach abandonment"). + * + * WSL detection is injected, the way the watch-policy tests inject it, so the + * decision runs on any host. The end-to-end cases raise a REAL node:sqlite I/O + * error by planting a FIFO where SQLite expects the `-shm` file: its lock + * calls fail exactly like a cross-OS lock does. + */ + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; +import { DatabaseConnection } from '../src/db'; +import { WslSharedIndexError, isSqliteIoError, toWslSharedIndexError } from '../src/db/wsl-shared-index'; +import { codeGraphDirNameFor, findNearestCodeGraphRoot, getCodeGraphDir, isInitialized } from '../src/directory'; +import { __setWslWindowsDriveForTests, isWslWindowsDrive } from '../src/sync/watch-policy'; +import { ToolHandler } from '../src/mcp/tools'; + +// Engine default-project opens use its lazy CommonJS loader, so the engine +// cases drive the built modules (and set the built module's override). +const { MCPEngine: BuiltMCPEngine } = require('../dist/mcp/engine') as typeof import('../src/mcp/engine'); +const builtWatchPolicy = require('../dist/sync/watch-policy') as typeof import('../src/sync/watch-policy'); + +/** Every path counts as a Windows drive seen from WSL. */ +const onWslDrive = (): boolean => true; + +const SHARED_DB = '/mnt/c/src/app/.codegraph/codegraph.db'; + +/** A node:sqlite-shaped I/O error (SQLITE_IOERR_LOCK). */ +function ioError(): Error { + return Object.assign(new Error('disk I/O error'), { + code: 'ERR_SQLITE_ERROR', + errcode: 3850, + errstr: 'disk I/O error', + }); +} + +function sharedIndexError(): WslSharedIndexError { + return toWslSharedIndexError(ioError(), SHARED_DB, { isWsl: true })!; +} + +describe('isSqliteIoError', () => { + it('matches the whole SQLITE_IOERR family by its extended code', () => { + for (const errcode of [10, 522, 3850, 4618, 5130]) { + expect(isSqliteIoError(Object.assign(new Error('x'), { errcode }))).toBe(true); + } + }); + + it('does not match other SQLite failures', () => { + // BUSY, CORRUPT, CANTOPEN, UNIQUE constraint + for (const errcode of [5, 11, 14, 2067]) { + expect(isSqliteIoError(Object.assign(new Error('disk I/O error'), { errcode }))).toBe(false); + } + }); + + it('falls back to the message when no code survived', () => { + expect(isSqliteIoError(new Error('disk I/O error'))).toBe(true); + expect(isSqliteIoError(new Error('database is locked'))).toBe(false); + expect(isSqliteIoError('disk I/O error')).toBe(false); + expect(isSqliteIoError(null)).toBe(false); + }); +}); + +describe('toWslSharedIndexError', () => { + it('rewrites an I/O error on the default index of a WSL /mnt/ project', () => { + const original = ioError(); + const err = toWslSharedIndexError(original, SHARED_DB, { isWsl: true }); + expect(err).toBeInstanceOf(WslSharedIndexError); + expect(err!.name).toBe('WslSharedIndexError'); + expect(err!.message).toContain('disk I/O error on the CodeGraph index at /mnt/c/src/app/.codegraph\n'); + expect(err!.message).toContain("Windows and WSL can't share one index on a Windows drive"); + expect(err!.message).toContain('CODEGRAPH_DIR=.codegraph-wsl'); + expect(err!.message).toContain('codegraph init'); + // The original stays reachable, and its SQLite codes still read the same. + expect(err!.cause).toBe(original); + expect(err!.errcode).toBe(3850); + expect(err!.code).toBe('ERR_SQLITE_ERROR'); + }); + + it('is idempotent', () => { + const err = sharedIndexError(); + expect(toWslSharedIndexError(err, SHARED_DB, { isWsl: true })).toBe(err); + }); + + it('leaves the error alone off WSL', () => { + expect(toWslSharedIndexError(ioError(), SHARED_DB, { isWsl: false })).toBeNull(); + }); + + it('leaves the error alone on a native WSL path', () => { + expect(toWslSharedIndexError(ioError(), '/home/me/app/.codegraph/codegraph.db', { isWsl: true })).toBeNull(); + }); + + it('does not treat /mnt/wsl (a Linux mount) as a Windows drive', () => { + expect(toWslSharedIndexError(ioError(), '/mnt/wsl/app/.codegraph/codegraph.db', { isWsl: true })).toBeNull(); + }); + + it('leaves the error alone when WSL already has its own index directory', () => { + // The advice would be wrong: CODEGRAPH_DIR is already split. + expect(toWslSharedIndexError(ioError(), '/mnt/c/src/app/.codegraph-wsl/codegraph.db', { isWsl: true })).toBeNull(); + }); + + it('leaves other SQLite errors alone', () => { + const busy = Object.assign(new Error('database is locked'), { errcode: 5 }); + expect(toWslSharedIndexError(busy, SHARED_DB, { isWsl: true })).toBeNull(); + }); + + // Real detection: WSL is a Linux kernel, so on any other host the rewrite + // can never fire, whatever the path looks like. + it.runIf(process.platform !== 'linux')('never fires off Linux with real detection', () => { + expect(toWslSharedIndexError(ioError(), SHARED_DB)).toBeNull(); + }); + +}); + +async function makeProject(dir: string): Promise { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'sample.ts'), 'export function parseToken() { return 1; }\n'); + (await CodeGraph.init(dir, { index: true })).close(); +} + +describe('the data directory on a Windows drive under WSL', () => { + let root: string; + const prevDir = process.env.CODEGRAPH_DIR; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-wsl-dir-')); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src', 'sample.ts'), 'export function parseToken() { return 1; }\n'); + delete process.env.CODEGRAPH_DIR; + }); + + afterEach(() => { + __setWslWindowsDriveForTests(null); + if (prevDir === undefined) delete process.env.CODEGRAPH_DIR; + else process.env.CODEGRAPH_DIR = prevDir; + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('gives a fresh project its own .codegraph-wsl', async () => { + __setWslWindowsDriveForTests(onWslDrive); + expect(codeGraphDirNameFor(root)).toBe('.codegraph-wsl'); + const cg = await CodeGraph.init(root, { index: true }); + cg.close(); + expect(fs.existsSync(path.join(root, '.codegraph-wsl', 'codegraph.db'))).toBe(true); + expect(fs.existsSync(path.join(root, '.codegraph'))).toBe(false); + // It keeps itself out of git like .codegraph does. + expect(fs.readFileSync(path.join(root, '.codegraph-wsl', '.gitignore'), 'utf8')).toMatch(/^\*$/m); + expect(findNearestCodeGraphRoot(path.join(root, 'src'))).toBe(root); + }); + + it('keeps an index already built in .codegraph, without a re-index', async () => { + const built = await CodeGraph.init(root, { index: true }); + const nodes = built.getStats().nodeCount; + built.close(); + __setWslWindowsDriveForTests(onWslDrive); + expect(getCodeGraphDir(root)).toBe(path.join(root, '.codegraph')); + expect(isInitialized(root)).toBe(true); + const reopened = CodeGraph.openSync(root); + expect(reopened.getStats().nodeCount).toBe(nodes); + reopened.close(); + expect(fs.existsSync(path.join(root, '.codegraph-wsl'))).toBe(false); + }); + + it('stays on .codegraph-wsl after Windows builds a .codegraph beside it', async () => { + __setWslWindowsDriveForTests(onWslDrive); + (await CodeGraph.init(root, { index: true })).close(); + // Windows-native CodeGraph indexes the same tree afterwards. + __setWslWindowsDriveForTests(() => false); + (await CodeGraph.init(root, { index: true })).close(); + expect(fs.existsSync(path.join(root, '.codegraph', 'codegraph.db'))).toBe(true); + __setWslWindowsDriveForTests(onWslDrive); + expect(codeGraphDirNameFor(root)).toBe('.codegraph-wsl'); + }); + + it('does not count a .codegraph with no database as an index', () => { + __setWslWindowsDriveForTests(onWslDrive); + fs.mkdirSync(path.join(root, '.codegraph')); + expect(codeGraphDirNameFor(root)).toBe('.codegraph-wsl'); + }); + + it('lets CODEGRAPH_DIR decide', () => { + __setWslWindowsDriveForTests(onWslDrive); + process.env.CODEGRAPH_DIR = '.codegraph-custom'; + expect(codeGraphDirNameFor(root)).toBe('.codegraph-custom'); + process.env.CODEGRAPH_DIR = '.codegraph'; + expect(codeGraphDirNameFor(root)).toBe('.codegraph'); + }); + + it('keeps .codegraph off a Windows drive', () => { + __setWslWindowsDriveForTests(() => false); + expect(codeGraphDirNameFor(root)).toBe('.codegraph'); + }); + + // Real detection: only a Linux kernel can be WSL. + it.runIf(process.platform !== 'linux')('keeps .codegraph off Linux with real detection', () => { + expect(isWslWindowsDrive('/mnt/c/src/app')).toBe(false); + expect(codeGraphDirNameFor(root)).toBe('.codegraph'); + }); +}); + +/** Replace the index's `-shm` with a FIFO: SQLite's first read then fails with a real SQLITE_IOERR. */ +function breakSharedMemory(dbPath: string): void { + fs.rmSync(`${dbPath}-shm`, { force: true }); + execFileSync('mkfifo', [`${dbPath}-shm`]); +} + +// mkfifo is POSIX-only. +describe.runIf(process.platform !== 'win32')('a real SQLite I/O error on open', () => { + let root: string; + let dbPath: string; + + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-wsl-shared-')); + await makeProject(root); + dbPath = path.join(root, '.codegraph', 'codegraph.db'); + breakSharedMemory(dbPath); + }); + + afterEach(() => { + __setWslWindowsDriveForTests(null); + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('surfaces as the actionable message on WSL + /mnt/', () => { + __setWslWindowsDriveForTests(onWslDrive); + let thrown: unknown; + try { + DatabaseConnection.open(dbPath).close(); + } catch (err) { + thrown = err; + } + expect(thrown).toBeInstanceOf(WslSharedIndexError); + // Pins node:sqlite's error shape: the extended code sits on `errcode`. + const cause = (thrown as WslSharedIndexError).cause as { errcode?: number; message?: string }; + expect(cause.errcode! & 0xff).toBe(10); + expect(cause.message).toBe('disk I/O error'); + expect((thrown as Error).message).toContain('CODEGRAPH_DIR=.codegraph-wsl'); + }); + + it('stays the raw SQLite error everywhere else', () => { + __setWslWindowsDriveForTests(() => false); + let thrown: unknown; + try { + DatabaseConnection.open(dbPath).close(); + } catch (err) { + thrown = err; + } + expect(thrown).toBeInstanceOf(Error); + expect(thrown).not.toBeInstanceOf(WslSharedIndexError); + expect((thrown as Error).message).toBe('disk I/O error'); + }); + + describe('through the MCP engine', () => { + let engine: import('../src/mcp/engine').MCPEngine | undefined; + + beforeEach(() => { + builtWatchPolicy.__setWslWindowsDriveForTests(onWslDrive); + }); + + afterEach(async () => { + builtWatchPolicy.__setWslWindowsDriveForTests(null); + await engine?.stop(); + engine = undefined; + }); + + const expectGuidance = (result: { isError?: boolean; content: Array<{ text: string }> }) => { + expect(result.isError).not.toBe(true); + expect(result.content[0]!.text).toContain('CODEGRAPH_DIR=.codegraph-wsl'); + expect(result.content[0]!.text).toContain('If you are an AI agent'); + expect(result.content[0]!.text).not.toContain('No CodeGraph project is loaded'); + }; + + it('answers a default-project call with the fix, not "no project loaded"', async () => { + engine = new BuiltMCPEngine({ watch: false }); + await engine.ensureInitialized(root); + expectGuidance(await engine.getToolHandler().execute('codegraph_explore', { query: 'parseToken' })); + }); + + it('keeps the fix on the per-call retry path', async () => { + engine = new BuiltMCPEngine({ watch: false }); + engine.retryInitializeSync(root); + expectGuidance(await engine.getToolHandler().execute('codegraph_search', { query: 'parseToken' })); + }); + + it('answers an explicit projectPath with the fix', async () => { + engine = new BuiltMCPEngine({ watch: false }); + expectGuidance(await engine.getToolHandler().execute('codegraph_search', { query: 'parseToken', projectPath: root })); + }); + + it('serves normally once the index opens again', async () => { + engine = new BuiltMCPEngine({ watch: false }); + engine.retryInitializeSync(root); + fs.rmSync(`${dbPath}-shm`, { force: true }); + engine.retryInitializeSync(root); + const result = await engine.getToolHandler().execute('codegraph_search', { query: 'parseToken' }); + expect(result.isError).not.toBe(true); + expect(result.content[0]!.text).toContain('parseToken'); + }); + }); +}); + +describe('ToolHandler answers the shared-index error success-shaped', () => { + let root: string; + let cg: CodeGraph; + + beforeEach(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-wsl-tools-')); + await makeProject(root); + cg = CodeGraph.openSync(root); + }); + + afterEach(() => { + cg.close(); + fs.rmSync(root, { recursive: true, force: true }); + }); + + it('when a query fails mid-session', async () => { + cg.searchNodes = () => { throw sharedIndexError(); }; + const result = await new ToolHandler(cg).execute('codegraph_search', { query: 'parseToken' }); + expect(result.isError).not.toBe(true); + expect(result.content[0]!.text).toContain('CODEGRAPH_DIR=.codegraph-wsl'); + }); + + it('on the worker dispatch path', async () => { + cg.searchNodes = () => { throw sharedIndexError(); }; + const result = await new ToolHandler(cg).executeReadTool('codegraph_search', { query: 'parseToken' }); + expect(result.isError).not.toBe(true); + expect(result.content[0]!.text).toContain('CODEGRAPH_DIR=.codegraph-wsl'); + }); + + it('keeps a genuine malfunction an error', async () => { + cg.searchNodes = () => { throw new Error('disk I/O error'); }; + const result = await new ToolHandler(cg).execute('codegraph_search', { query: 'parseToken' }); + expect(result.isError).toBe(true); + }); + + it('forgets a recorded open failure once a default project loads', async () => { + const handler = new ToolHandler(null); + handler.setDefaultOpenFailure(sharedIndexError()); + expect((await handler.execute('codegraph_search', { query: 'parseToken' })).content[0]!.text) + .toContain('CODEGRAPH_DIR=.codegraph-wsl'); + handler.setDefaultCodeGraph(cg); + const result = await handler.execute('codegraph_search', { query: 'parseToken' }); + expect(result.content[0]!.text).not.toContain('CODEGRAPH_DIR'); + expect(result.content[0]!.text).toContain('parseToken'); + }); +}); diff --git a/site/src/content/docs/troubleshooting.md b/site/src/content/docs/troubleshooting.md index 26999428a5..da194c1f4b 100644 --- a/site/src/content/docs/troubleshooting.md +++ b/site/src/content/docs/troubleshooting.md @@ -28,4 +28,4 @@ The MCP server auto-syncs on save (wait a couple of seconds). Run `codegraph syn ## Sharing one checkout between Windows and WSL -Don't point both at the same `.codegraph/`: the background-server lock and the SQLite index are tied to the OS that wrote them, and SQLite locking across the WSL2/Windows filesystem boundary is unreliable. Give each side its own index in the same tree by setting `CODEGRAPH_DIR` to a distinct name on one of them — e.g. `CODEGRAPH_DIR=.codegraph-win` on Windows, leaving WSL on the default `.codegraph`. CodeGraph skips any sibling `.codegraph-*` directory when indexing and watching, so the two never trip over each other. +Don't point both at the same `.codegraph/`: the background-server lock and the SQLite index are tied to the OS that wrote them, and SQLite locking across the WSL2/Windows filesystem boundary is unreliable (WSL reports it as a `disk I/O error`). For a project on a Windows drive (a `/mnt/c/…` path), WSL keeps its own index automatically: an index first built from WSL goes in `.codegraph-wsl/`, leaving `.codegraph/` to Windows. An index already in `.codegraph/` stays where it is, so if Windows built that one, give WSL its own by setting `CODEGRAPH_DIR=.codegraph-wsl` in WSL and running `codegraph init` there. `CODEGRAPH_DIR` always picks the name when set, on either side. CodeGraph skips any sibling `.codegraph-*` directory when indexing and watching, so the two never trip over each other. diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 6664d1c53b..e9da3a6e6f 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -41,7 +41,7 @@ try { import { Command } from 'commander'; import * as path from 'path'; import * as fs from 'fs'; -import { getCodeGraphDir, isInitialized, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection } from '../directory'; +import { getCodeGraphDir, isInitialized, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; import { extractProseCandidates } from '../search/identifier-segments'; import { detectWorktreeIndexMismatch, worktreeMismatchWarning } from '../sync/worktree'; import { createShimmerProgress } from '../ui/shimmer-progress'; @@ -700,6 +700,15 @@ async function runInit( const { default: CodeGraph, getDatabasePath } = await loadCodeGraph(); const cg = await CodeGraph.init(projectPath, { index: false }); clack.log.success(`Initialized in ${projectPath}`); + // A fresh index on a Windows drive under WSL gets its own directory (#995). + // It isn't the documented name, so say where it went and why. + const dataDir = path.basename(getCodeGraphDir(projectPath)); + if (dataDir !== codeGraphDirName()) { + clack.log.info( + `The index is in ${dataDir}/: this project is on a Windows drive, so WSL keeps its own index ` + + `rather than share ${DEFAULT_CODEGRAPH_DIR}/ with CodeGraph on Windows. Set CODEGRAPH_DIR to choose the name yourself.` + ); + } // Indexing runs by default now. The legacy -i/--index flag is still // accepted (so existing muscle memory and scripts don't break) but is a diff --git a/src/db/sqlite-adapter.ts b/src/db/sqlite-adapter.ts index 2ca02e5bee..e94b587b95 100644 --- a/src/db/sqlite-adapter.ts +++ b/src/db/sqlite-adapter.ts @@ -10,6 +10,8 @@ * wasm fallback. When run from source instead, it requires Node >= 22.5. */ +import { toWslSharedIndexError } from './wsl-shared-index'; + export interface SqliteStatement { run(...params: any[]): { changes: number; lastInsertRowid: number | bigint }; get(...params: any[]): any; @@ -50,13 +52,27 @@ export type SqliteBackend = 'node-sqlite'; class NodeSqliteAdapter implements SqliteDatabase { private _db: any; private _txDepth = 0; + private readonly _dbPath: string; constructor(dbPath: string, opts?: { readOnly?: boolean }) { // eslint-disable-next-line @typescript-eslint/no-require-imports const { DatabaseSync } = require('node:sqlite'); + this._dbPath = dbPath; this._db = opts?.readOnly ? new DatabaseSync(dbPath, { readOnly: true }) : new DatabaseSync(dbPath); } + /** + * What a failed call throws: the error itself, or — for a "disk I/O error" + * on a WSL index that Windows CodeGraph shares — the actionable rewrite + * (#995). Every open runs its PRAGMAs and first reads through the methods + * below, and so does every later query. `iterate()` is left raw: a + * row-by-row wrapper would tax the unbounded scans it exists for, and a + * session reads through `get`/`all` long before it reaches one. + */ + private failure(err: unknown): unknown { + return toWslSharedIndexError(err, this._dbPath) ?? err; + } + get open(): boolean { return this._db.isOpen; } @@ -65,20 +81,39 @@ class NodeSqliteAdapter implements SqliteDatabase { // node:sqlite matches better-sqlite3's calling convention (variadic // positional args, or a single object for @named params), so params forward // through unchanged. - const stmt = this._db.prepare(sql); + let stmt: any; + try { + stmt = this._db.prepare(sql); + } catch (err) { + throw this.failure(err); + } + const failure = (err: unknown): unknown => this.failure(err); return { run(...params: any[]) { - const r = stmt.run(...params); + let r: any; + try { + r = stmt.run(...params); + } catch (err) { + throw failure(err); + } return { changes: Number(r?.changes ?? 0), lastInsertRowid: r?.lastInsertRowid ?? 0, }; }, get(...params: any[]) { - return stmt.get(...params); + try { + return stmt.get(...params); + } catch (err) { + throw failure(err); + } }, all(...params: any[]) { - return stmt.all(...params); + try { + return stmt.all(...params); + } catch (err) { + throw failure(err); + } }, iterate(...params: any[]) { return stmt.iterate(...params); @@ -87,7 +122,11 @@ class NodeSqliteAdapter implements SqliteDatabase { } exec(sql: string): void { - this._db.exec(sql); + try { + this._db.exec(sql); + } catch (err) { + throw this.failure(err); + } } pragma(str: string, options?: { simple?: boolean }): any { @@ -95,12 +134,12 @@ class NodeSqliteAdapter implements SqliteDatabase { // Write pragma ("key = value"): node:sqlite is real SQLite, so every pragma // (WAL, mmap, synchronous, …) applies as-is. if (trimmed.includes('=')) { - this._db.exec(`PRAGMA ${trimmed}`); + this.exec(`PRAGMA ${trimmed}`); return; } // Read pragma. Default: the row object (e.g. { journal_mode: 'wal' }). // `{ simple: true }` returns just the single column value, like better-sqlite3. - const row = this._db.prepare(`PRAGMA ${trimmed}`).get(); + const row = this.prepare(`PRAGMA ${trimmed}`).get(); if (options?.simple) { return row && typeof row === 'object' ? Object.values(row)[0] : row; } @@ -155,6 +194,9 @@ export function createDatabase(dbPath: string, opts?: { readOnly?: boolean }): { try { return { db: new NodeSqliteAdapter(dbPath, opts), backend: 'node-sqlite' }; } catch (error) { + // node:sqlite loaded and SQLite itself failed — not a missing-module case. + const shared = toWslSharedIndexError(error, dbPath); + if (shared) throw shared; const msg = error instanceof Error ? error.message : String(error); throw new Error( 'Failed to open SQLite via the built-in node:sqlite module.\n' + diff --git a/src/db/wsl-shared-index.ts b/src/db/wsl-shared-index.ts new file mode 100644 index 0000000000..41f8205f0f --- /dev/null +++ b/src/db/wsl-shared-index.ts @@ -0,0 +1,98 @@ +/** + * Windows + WSL sharing one index on a Windows drive (issue #995). + * + * A repo under `/mnt//` can be used from both sides: Windows-native + * CodeGraph and WSL CodeGraph then open the same `.codegraph/codegraph.db` + * across the 9p/DrvFs bridge, where SQLite's file locking and its `-shm` + * shared memory don't hold between the two OSes. WSL's connection fails with a + * bare "disk I/O error" (SQLITE_IOERR). A private WSL index on the same drive + * works fine, so a fresh WSL project there already gets its own + * `.codegraph-wsl` (`codeGraphDirNameFor`). An index already in `.codegraph` — + * built by Windows first, or by WSL before that default — is kept as it is, + * and when it fails this way the SQLite adapter rewrites the error into the + * `CODEGRAPH_DIR=.codegraph-wsl` instruction instead of letting it reach the + * user as an unexplained I/O failure. + */ + +import * as path from 'path'; +import { DEFAULT_CODEGRAPH_DIR, WSL_CODEGRAPH_DIR } from '../directory'; +import { isWindowsDriveMount, isWslWindowsDrive } from '../sync/watch-policy'; + +/** SQLite's primary result code for an I/O failure; extended codes keep it in the low byte. */ +const SQLITE_IOERR = 10; + +/** + * Is `err` a SQLite I/O error? node:sqlite reports the EXTENDED result code on + * `errcode` (SQLITE_IOERR_LOCK = 3850, SQLITE_IOERR_SHMMAP = 5130, ...), whose + * low byte is the primary code. An error without one (re-wrapped somewhere + * along the way) is matched by SQLite's message for the whole family. + */ +export function isSqliteIoError(err: unknown): boolean { + if (!err || typeof err !== 'object') return false; + const { errcode, message } = err as { errcode?: unknown; message?: unknown }; + if (typeof errcode === 'number') return (errcode & 0xff) === SQLITE_IOERR; + return typeof message === 'string' && /disk I\/O error/i.test(message); +} + +/** + * A SQLite I/O error on an index that Windows CodeGraph most likely shares. + * Keeps the original as `cause` and carries its SQLite codes, so a caller that + * branches on them sees the same values as before the rewrite. + */ +export class WslSharedIndexError extends Error { + readonly code: unknown; + readonly errcode: unknown; + readonly errstr: unknown; + + constructor(original: Error, dataDir: string) { + super( + `${original.message} on the CodeGraph index at ${dataDir}\n` + + "This project is on a Windows drive, where SQLite's file locking doesn't work across " + + 'the Windows/WSL boundary, so this usually means CodeGraph on Windows is using the same ' + + "index. Windows and WSL can't share one index on a Windows drive; give WSL its own:\n" + + ` 1. Set CODEGRAPH_DIR=${WSL_CODEGRAPH_DIR} in the WSL environment (your shell profile, ` + + 'or the env of the MCP server your agent starts).\n' + + ` 2. Run "codegraph init" in WSL to build that index. Windows keeps using ${DEFAULT_CODEGRAPH_DIR}.`, + { cause: original } + ); + this.name = 'WslSharedIndexError'; + const codes = original as Error & { code?: unknown; errcode?: unknown; errstr?: unknown }; + this.code = codes.code; + this.errcode = codes.errcode; + this.errstr = codes.errstr; + } +} + +/** + * Inputs that can be overridden in tests so the decision is deterministic + * without touching `/proc/version` (see `WatchProbe`). + */ +export interface WslSharedIndexProbe { + /** Defaults to `detectWsl()` (through `isWslWindowsDrive`). */ + isWsl?: boolean; +} + +/** + * The error to throw in place of `err`: a {@link WslSharedIndexError} when a + * SQLite I/O error hit the default-named index of a project on a Windows drive + * under WSL, else `null` (throw `err` unchanged). + * + * Only the default `.codegraph` counts: it is the directory Windows CodeGraph + * opens too. A WSL that already has its own directory (`.codegraph-wsl`, or + * any `CODEGRAPH_DIR`) hit some other I/O failure, and pointing it at + * `CODEGRAPH_DIR` would mislead. + */ +export function toWslSharedIndexError( + err: unknown, + dbPath: string, + probe: WslSharedIndexProbe = {} +): WslSharedIndexError | null { + if (err instanceof WslSharedIndexError) return err; + if (!(err instanceof Error) || !isSqliteIoError(err)) return null; + const dataDir = path.dirname(dbPath); + if (path.basename(dataDir) !== DEFAULT_CODEGRAPH_DIR) return null; + const onWslWindowsDrive = probe.isWsl === undefined + ? isWslWindowsDrive(dbPath) + : probe.isWsl && isWindowsDriveMount(dbPath); + return onWslWindowsDrive ? new WslSharedIndexError(err, dataDir) : null; +} diff --git a/src/directory.ts b/src/directory.ts index cbc147f724..1d758dbe03 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -7,9 +7,17 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; +import { isWslWindowsDrive } from './sync/watch-policy'; /** The default per-project data directory name. */ -const DEFAULT_CODEGRAPH_DIR = '.codegraph'; +export const DEFAULT_CODEGRAPH_DIR = '.codegraph'; + +/** + * The data directory name WSL gives a fresh project on a Windows drive, so it + * never shares one index with CodeGraph on Windows (issue #995). Indexing and + * watching skip every `.codegraph-*` sibling on both sides (#636). + */ +export const WSL_CODEGRAPH_DIR = '.codegraph-wsl'; let warnedBadDirName = false; @@ -79,11 +87,38 @@ export function isCodeGraphDataDir(name: string): boolean { ); } +/** + * The data directory name for one project: {@link codeGraphDirName}, except + * for a project on a Windows drive under WSL (`/mnt/c/...`) with no + * `CODEGRAPH_DIR` set. Windows-native CodeGraph opens `.codegraph` in that + * same tree, and SQLite's locking doesn't hold across the 9p/DrvFs bridge, so + * the two sharing one index fails with "disk I/O error" (issue #995). There: + * + * 1. `.codegraph-wsl/` exists → it. Once WSL has its own index it keeps it, + * even after Windows builds a `.codegraph` beside it. + * 2. `.codegraph/codegraph.db` exists → `.codegraph`. An index built before + * this default is kept rather than silently rebuilt somewhere else. + * 3. neither → `.codegraph-wsl`, so a fresh WSL index never shares. + * + * Every other host keeps the plain name without a stat: the WSL check is + * cached per process. + */ +export function codeGraphDirNameFor(projectRoot: string): string { + if (process.env.CODEGRAPH_DIR?.trim() || !isWslWindowsDrive(projectRoot)) return codeGraphDirName(); + try { + if (fs.statSync(path.join(projectRoot, WSL_CODEGRAPH_DIR)).isDirectory()) return WSL_CODEGRAPH_DIR; + } catch { + // absent — fall through + } + if (fs.existsSync(path.join(projectRoot, DEFAULT_CODEGRAPH_DIR, 'codegraph.db'))) return DEFAULT_CODEGRAPH_DIR; + return WSL_CODEGRAPH_DIR; +} + /** * Get the .codegraph directory path for a project */ export function getCodeGraphDir(projectRoot: string): string { - return path.join(projectRoot, codeGraphDirName()); + return path.join(projectRoot, codeGraphDirNameFor(projectRoot)); } /** diff --git a/src/installer/index.ts b/src/installer/index.ts index 199a6de75e..c1fe1c4091 100644 --- a/src/installer/index.ts +++ b/src/installer/index.ts @@ -28,7 +28,7 @@ import type { AgentTarget, Location, TargetId } from './targets/types'; // installer must stay importable even when native modules can't load). import { watchDisabledReason } from '../sync/watch-policy'; import { isGitRepo, isSyncHookInstalled, installGitSyncHook } from '../sync/git-hooks'; -import { getCodeGraphDir, codeGraphDirName } from '../directory'; +import { getCodeGraphDir } from '../directory'; import { getTelemetry, TELEMETRY_DOCS } from '../telemetry'; import { maybeOfferBetaSignup } from './beta-signup'; @@ -514,8 +514,9 @@ export async function runUninstaller(opts: RunUninstallerOptions): Promise // Step 4: for local uninstall, the index dir is separate — point at // `uninit` so the user knows it's still there (and how to remove it). - if (location === 'local' && fs.existsSync(getCodeGraphDir(process.cwd()))) { - clack.log.info(`The ${codeGraphDirName()}/ index for this project is still here. Run \`codegraph uninit\` to delete it.`); + const indexDir = location === 'local' ? getCodeGraphDir(process.cwd()) : null; + if (indexDir && fs.existsSync(indexDir)) { + clack.log.info(`The ${path.basename(indexDir)}/ index for this project is still here. Run \`codegraph uninit\` to delete it.`); } // Step 4b: the CLI binary itself (global uninstall only — a project-scoped diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index cb6fa4c1e7..0276bb4772 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -15,6 +15,7 @@ import * as path from 'path'; import type CodeGraph from '../index'; import { resolveServerRoot } from '../directory'; import { ToolHandler } from './tools'; +import { WslSharedIndexError } from '../db/wsl-shared-index'; import { assertNoRebuild, releaseWriterLock, tryAcquireWriterLock, writerLockHeldMessage } from './writer-lock'; import { QueryPool, resolvePoolSize } from './query-pool'; import { acquireProject, ProjectLease } from './project-lifecycle'; @@ -236,8 +237,9 @@ export class MCPEngine { this.startWatching(); this.catchUpSync(); this.maybeStartPool(resolvedRoot); - } catch { + } catch (err) { // Still failing — caller will try again on the next tool call. + this.toolHandler.setDefaultOpenFailure(err instanceof WslSharedIndexError ? err : null); } } @@ -364,6 +366,8 @@ export class MCPEngine { } catch (err) { const msg = err instanceof Error ? err.message : String(err); process.stderr.write(`[CodeGraph MCP] Failed to open project at ${resolvedRoot}: ${msg}\n`); + // The agent otherwise hears only "no project loaded" (#995). + this.toolHandler.setDefaultOpenFailure(err instanceof WslSharedIndexError ? err : null); } } diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index c6fb38f45a..8e9c8138a0 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -7,6 +7,7 @@ import type CodeGraph from '../index'; import type { QueryPool } from './query-pool'; import { findNearestCodeGraphRoot, isSameIndexRoot } from '../directory'; +import { WslSharedIndexError } from '../db/wsl-shared-index'; // Lazy-load the heavy CodeGraph chain off the MCP startup path — see the same // helper in engine.ts. ToolHandler must load to answer tools/list (static // schemas), but it must NOT drag in sqlite/query layers before the daemon binds; @@ -87,6 +88,22 @@ import { */ export class NotIndexedError extends Error {} +/** + * The Windows/WSL shared-index failure (#995) phrased for the agent. It is an + * expected condition the USER fixes in their environment, so like + * {@link NotIndexedError} it answers SUCCESS-shaped — never `isError`, which + * would teach the agent to abandon codegraph for projects that work fine. + */ +function wslSharedIndexGuidance(err: WslSharedIndexError): string { + return ( + `${err.message}\n\n` + + "If you are an AI agent: codegraph can't read this project's index from WSL until the user " + + 'makes that change. Use your built-in tools (Read/Grep/Glob) for this task and pass the message ' + + "above on to the user — setting CODEGRAPH_DIR and building the index are the user's decisions, " + + "so don't do either yourself." + ); +} + /** * A security refusal (sensitive system path). Stays `isError: true` WITHOUT * retry guidance — abandoning this path is the desired agent reaction. @@ -1593,6 +1610,10 @@ export class ToolHandler { // retry) — tool calls themselves never scan. private knownSubprojects: string[] = []; private knownSubprojectsBase: string | null = null; + // Why the default project failed to open, when that is worth telling the + // agent instead of "no project loaded" — today only the Windows/WSL + // shared-index error (#995). Engine-maintained; cleared by a successful open. + private defaultOpenFailure: WslSharedIndexError | null = null; // Per-start-path cache of the git worktree/index mismatch (issue #155). The // mismatch is a fixed property of (where the request came from → which // .codegraph/ it resolves to), so the up-to-two `git rev-parse` spawns run @@ -1637,6 +1658,16 @@ export class ToolHandler { */ setDefaultCodeGraph(cg: CodeGraph): void { this.cg = cg; + this.defaultOpenFailure = null; + } + + /** + * Engine-only: record why the default project failed to open (#995), so a + * call that needs it answers with that fix rather than "no project loaded". + * `null` clears it. + */ + setDefaultOpenFailure(err: WslSharedIndexError | null): void { + this.defaultOpenFailure = err; } /** @@ -1839,6 +1870,7 @@ export class ToolHandler { private getCodeGraph(projectPath?: string): CodeGraph { if (!projectPath) { if (!this.cg) { + if (this.defaultOpenFailure) throw this.defaultOpenFailure; const searched = this.defaultProjectHint ?? process.cwd(); throw new NotIndexedError( 'No CodeGraph project is loaded for this session.\n' + @@ -2428,6 +2460,10 @@ export class ToolHandler { if (err instanceof NotIndexedError || (err as Error | null)?.name === 'RebuildInProgressError') { return this.textResult((err as Error).message); } + // Windows and WSL sharing one index (#995): the user's fix, not a malfunction. + if (err instanceof WslSharedIndexError) { + return this.textResult(wslSharedIndexGuidance(err)); + } // Security refusal: a clean error, no retry encouragement. if (err instanceof PathRefusalError) { return this.errorResult(err.message); @@ -2514,6 +2550,9 @@ export class ToolHandler { if (err instanceof NotIndexedError) { return this.textResult(err.message); } + if (err instanceof WslSharedIndexError) { + return this.textResult(wslSharedIndexGuidance(err)); + } if (err instanceof PathRefusalError) { return this.errorResult(err.message); } diff --git a/src/sync/watch-policy.ts b/src/sync/watch-policy.ts index 426a8869d9..3e5c4ef66b 100644 --- a/src/sync/watch-policy.ts +++ b/src/sync/watch-policy.ts @@ -15,6 +15,7 @@ */ import * as fs from 'fs'; +import * as path from 'path'; import { normalizePath } from '../utils'; let wslChecked = false; @@ -53,10 +54,22 @@ export function detectWsl(): boolean { * Deliberately matches only single-letter drive mounts, so genuinely fast * Linux mounts such as `/mnt/wsl/...` are not flagged. */ -function isWindowsDriveMount(projectRoot: string): boolean { +export function isWindowsDriveMount(projectRoot: string): boolean { return /^\/mnt\/[a-z](\/|$)/i.test(normalizePath(projectRoot)); } +let wslWindowsDriveOverride: ((p: string) => boolean) | null = null; + +/** + * Is `p` on a Windows drive as WSL sees it (`/mnt/c/...`)? There the same + * tree is reachable from Windows-native CodeGraph too, and the two must not + * share one index (#995). Always false off WSL, without touching the disk. + */ +export function isWslWindowsDrive(p: string): boolean { + if (wslWindowsDriveOverride) return wslWindowsDriveOverride(p); + return detectWsl() && isWindowsDriveMount(path.resolve(p)); +} + /** * Inputs that can be overridden in tests so the decision is deterministic * without touching real env vars or `/proc/version`. @@ -102,3 +115,8 @@ export function __resetWslCacheForTests(): void { wslChecked = false; wslValue = false; } + +/** Test-only: decide {@link isWslWindowsDrive} with `fn` (`null` restores detection). */ +export function __setWslWindowsDriveForTests(fn: ((p: string) => boolean) | null): void { + wslWindowsDriveOverride = fn; +} From adad8407ff10e7880e9dd79a92fdc15160ed1808 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 06:04:10 +0000 Subject: [PATCH 002/363] fix(explore): pay named members before incidental ones across clusters (#2062) Clusters of equal max importance rank by density, and density counts every member, so a named function with unrelated helpers merged around it outranked a cluster holding a named function alone. Taken whole, the higher clusters spent the file's reservation on the helpers and the isolated named function rendered nothing (lib/response.ts repro: sendBody 0 of 36 lines). Reordering alone only moves the victim. Two rules, both scoped to incidental members (not named, not an entry point, not on the call path): - a cluster's incidental members may only use what is left once every lower-ranked cluster's protected members are paid; its own protected members are never held back, so named-vs-named stays rank's call; - in a cluster holding a protected member, incidental members are never kept past the render's ceiling in rendered size, so the source-order window back to the ceiling no longer cuts a named body's tail (gin's handleHTTPRequest) on selection or on any re-render. 47 deterministic queries over the 7 README repos plus express vs main: named bodies complete 198 -> 234 (17 responses up, 0 down), named lines +10%, responses over the 25K cap 5 -> 1. One new final-cut truncation (django-bag) comes from owedPayableBelow's MIN_CHARS threshold, recorded in the design doc. On main + the named-concentration branch: 235 -> 279. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../explore-isolated-named-cluster.test.ts | 267 ++++++++++++++++++ docs/design/explore-budget-allocation.md | 170 +++++++++++ src/mcp/tools.ts | 134 +++++++-- 4 files changed, 552 insertions(+), 20 deletions(-) create mode 100644 __tests__/explore-isolated-named-cluster.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 45bafb8d2c..cef29ae955 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -164,6 +164,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph callers`, `codegraph callees` and `codegraph impact` (CLI and MCP) now report missing names with did-you-mean suggestions instead of another symbol's results, and exact matches with no callers stay empty; thanks @uvmplus. (#1473, #1481) - When `codegraph_explore` trims a large file, the gaps in its source now name the symbols that were left out, using only room the file's budget has left, so the names never push out the code itself. (#1711) - `codegraph_explore` no longer reports "runtime dispatch" through a base class that doesn't declare the method, so a common name shared by many unrelated subclasses no longer opens an answer with a list of thousands of implementations. +- When you name several functions in one file and unrelated code sits between them, `codegraph_explore` now returns the functions you named before the code around them, instead of spending the file's room on that code and leaving one of the functions you asked for with no source at all. #### MCP / indexing diff --git a/__tests__/explore-isolated-named-cluster.test.ts b/__tests__/explore-isolated-named-cluster.test.ts new file mode 100644 index 0000000000..6981bfe004 --- /dev/null +++ b/__tests__/explore-isolated-named-cluster.test.ts @@ -0,0 +1,267 @@ +/** + * Regression gate: a cluster holding only a named function must not lose its + * file's budget to the incidental members of a cluster ranked above it. + * + * `handleExplore` ranks a file's clusters spine first, then by max importance, + * then by DENSITY — summed member importance over span. Density counts every + * member, so a named function (importance 10) with unrelated helpers merged + * around it scores the helpers' definitions (1) and their edge lines (2) too, + * and outranks a cluster holding the same kind of named function alone. Taken + * whole, the higher clusters then spent the reservation on the helpers, and the + * isolated named function below them rendered nothing: the query named five + * functions and got two to four of them back, depending on the build. + * + * The fixture is that shape and nothing else: one file, five independent + * exported functions the query names, three unrelated `headerSlotN` helpers + * between each pair, and ~45 more helpers after them. Three controls pin that the + * gate measures the ordering and not the budget: + * + * - the same five functions CONTIGUOUS all render whole under the same + * reservation, so the budget holds the named bodies; + * - their rendered cost is well under the reservation, and the file is far + * over it, so something has to be dropped and it need not be a named body; + * - a query naming only `renderView` still renders the helpers merged around + * it: with nothing named below a cluster, its incidental context stays. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; +import type { ExploreDiagnosticReport } from '../src/mcp/explore-diagnostics'; + +const TARGET = 'lib/response.ts'; +const NAMED = ['sendBody', 'sendJson', 'renderView', 'redirectTo', 'sendFileStream']; +const QUERY = NAMED.join(' '); +/** Statements per named body: 26 lines with the signature and closing brace. */ +const BODY_STATEMENTS = 24; +const HELPERS_BETWEEN = 3; +const HELPERS_AFTER = 45; + +/** + * `lib/response.ts`. The helpers are the filler: never named, never called by a + * named function, and wide enough that the named functions plus the helpers + * merged around them overrun the file's reservation both at the 9,100-char + * `MAX_SHARE` valve and at the ~12,800 a file the query named gets when the + * valve is lifted for it — so the gate stays red without the fix either way. + */ +function responseSource(interleave: boolean): string { + const out: string[] = [ + 'export interface Res {', + ' body: string[];', + ' setHeader(name: string, value: string): void;', + '}', + '', + ]; + let slot = 0; + const helper = (): void => { + slot++; + out.push(`export function headerSlot${slot}(res: Res, value: string): void {`); + for (let k = 0; k < 6; k++) { + out.push(` res.setHeader('x-slot-${slot}-${k}', \`max-age=31536000; includeSubDomains; preload; ` + + `report-uri=https://reports.example.test/csp/slot-${slot}/${k}?format=json&v=\${value}\`);`); + } + out.push('}', ''); + }; + NAMED.forEach((name, i) => { + out.push(`export function ${name}(res: Res, payload: string): void {`); + for (let k = 0; k < BODY_STATEMENTS; k++) out.push(` res.body.push('${name}:${k}:' + payload);`); + out.push('}', ''); + if (interleave && i < NAMED.length - 1) for (let h = 0; h < HELPERS_BETWEEN; h++) helper(); + }); + for (let h = 0; h < HELPERS_AFTER; h++) helper(); + return out.join('\n'); +} + +interface Run { + dir: string; + cg: CodeGraph; + response: string; + report: ExploreDiagnosticReport; +} + +async function runExplore(interleave: boolean, query: string): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-isolated-named-')); + fs.mkdirSync(path.join(dir, 'lib')); + fs.writeFileSync(path.join(dir, TARGET), responseSource(interleave)); + fs.writeFileSync(path.join(dir, 'package.json'), '{"name":"isolated-named","version":"1.0.0"}\n'); + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + + const sidecar = path.join(dir, 'explore-diag.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + let response: string; + try { + response = (await new ToolHandler(cg).execute('codegraph_explore', { query })) + .content?.[0]?.text ?? ''; + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + const written = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + return { dir, cg, response, report: JSON.parse(written[written.length - 1]!) }; +} + +function teardown(run: Run | undefined): void { + if (!run) return; + run.cg.destroy(); + if (fs.existsSync(run.dir)) fs.rmSync(run.dir, { recursive: true, force: true }); +} + +/** Every `\t` line number the response sent. The fixture has one file. */ +function renderedLines(response: string): Set { + const out = new Set(); + for (const m of response.matchAll(/^(\d+)\t/gm)) out.add(Number(m[1])); + return out; +} + +function defOf(run: Run, name: string) { + const node = run.cg.getNodesInFile(TARGET).find((n) => n.name === name && n.kind === 'function'); + expect(node, `${name} is not indexed in ${TARGET}`).toBeDefined(); + return node!; +} + +/** Lines of `name`'s body the response carries, out of its full span. */ +function coverage(run: Run, name: string): { got: number; of: number } { + const node = defOf(run, name); + const lines = renderedLines(run.response); + let got = 0; + for (let ln = node.startLine; ln <= node.endLine; ln++) if (lines.has(ln)) got++; + return { got, of: node.endLine - node.startLine + 1 }; +} + +function targetFile(run: Run): ExploreDiagnosticReport['files'][number] { + const file = run.report.files.find((f) => f.path === TARGET); + expect(file, `${TARGET} is not among the ranked candidates`).toBeDefined(); + return file!; +} + +describe('an isolated named cluster keeps its share of the file budget', () => { + let run: Run; + let contiguous: Run; + + beforeAll(async () => { + run = await runExplore(true, QUERY); + contiguous = await runExplore(false, QUERY); + }, 120_000); + + afterAll(() => { + teardown(run); + teardown(contiguous); + }); + + describe('fixture shape — if this rots, the gate below means nothing', () => { + it('puts unrelated helpers between every pair of named functions', () => { + const helpers = run.cg.getNodesInFile(TARGET) + .filter((n) => n.kind === 'function' && n.name.startsWith('headerSlot')); + for (let i = 0; i + 1 < NAMED.length; i++) { + const above = defOf(run, NAMED[i]!); + const below = defOf(run, NAMED[i + 1]!); + const between = helpers.filter((h) => h.startLine > above.endLine && h.endLine < below.startLine); + expect(between.length, `${NAMED[i]} → ${NAMED[i + 1]}`).toBe(HELPERS_BETWEEN); + } + }); + + it('names functions that do not call each other, so no call path ranks one first', () => { + const ids = new Set(NAMED.map((name) => defOf(run, name).id)); + for (const name of NAMED) { + const reaches = run.cg.getCallees(defOf(run, name).id).filter(({ node }) => ids.has(node.id)); + expect(reaches, name).toHaveLength(0); + } + }); + + it('clusters a file far bigger than its reservation', () => { + const file = targetFile(run); + expect(file.render).toBe('clusters'); + expect(file.clipped).toBe(true); + const size = fs.readFileSync(path.join(run.dir, TARGET), 'utf-8').length; + expect(size).toBeGreaterThan(3 * file.allowance!); + }); + + it('holds the named bodies well inside that reservation', () => { + // What the five bodies cost rendered: their lines, the three lines of + // context padding on each side, and the `\t` line-number prefix. + const lines = fs.readFileSync(path.join(run.dir, TARGET), 'utf-8').split('\n'); + let cost = 0; + for (const name of NAMED) { + const node = defOf(run, name); + for (let ln = node.startLine - 3; ln <= node.endLine + 3; ln++) { + cost += (lines[ln - 1] ?? '').length + String(ln).length + 2; + } + } + expect(cost).toBeLessThan(0.9 * targetFile(run).allowance!); + }); + + it('renders every named body whole when the same functions sit together', () => { + expect(targetFile(contiguous).allowance).toBe(targetFile(run).allowance); + for (const name of NAMED) { + const { got, of } = coverage(contiguous, name); + expect(got, `${name} (contiguous)`).toBe(of); + } + }); + }); + + describe('the gate', () => { + it('renders every named body whole with helpers between them', () => { + for (const name of NAMED) { + const { got, of } = coverage(run, name); + expect(got, `${name}: ${got}/${of} lines`).toBe(of); + } + }); + + it('keeps the response inside the hard ceiling', () => { + expect(run.report.envelope.chars).toBeLessThanOrEqual(run.report.budget.hardCeiling); + }); + }); +}); + +/** + * The same rule inside one cluster. A cluster taken after the first is bounded + * by its room exactly, and its member shrink estimated on raw source, which runs + * well under the rendered size: it kept the helpers merged above `renderView`, + * overran the room, and the window back to it cut in source order — through the + * last lines of `renderView`, the body the query named. + */ +describe('a named body in a later cluster is not trimmed for the helpers above it', () => { + let run: Run; + + beforeAll(async () => { + run = await runExplore(true, 'renderView redirectTo'); + }, 120_000); + + afterAll(() => teardown(run)); + + it('renders both named bodies whole', () => { + for (const name of ['renderView', 'redirectTo']) { + const { got, of } = coverage(run, name); + expect(got, `${name}: ${got}/${of} lines`).toBe(of); + } + }); + + it('keeps the response inside the hard ceiling', () => { + expect(run.report.envelope.chars).toBeLessThanOrEqual(run.report.budget.hardCeiling); + }); +}); + +describe('a named cluster with nothing named below it keeps its incidental context', () => { + let run: Run; + + beforeAll(async () => { + run = await runExplore(true, 'renderView'); + }, 120_000); + + afterAll(() => teardown(run)); + + it('renders the helper merged in above renderView, not just renderView', () => { + const view = coverage(run, 'renderView'); + expect(view.got).toBe(view.of); + const target = defOf(run, 'renderView'); + const above = run.cg.getNodesInFile(TARGET) + .filter((n) => n.kind === 'function' && n.name.startsWith('headerSlot') && n.endLine < target.startLine) + .sort((a, b) => b.endLine - a.endLine)[0]!; + const { got, of } = coverage(run, above.name); + expect(got, `${above.name}: ${got}/${of} lines`).toBe(of); + }); +}); diff --git a/docs/design/explore-budget-allocation.md b/docs/design/explore-budget-allocation.md index ee28f1da36..d77d6dd87a 100644 --- a/docs/design/explore-budget-allocation.md +++ b/docs/design/explore-budget-allocation.md @@ -664,3 +664,173 @@ Full record: [`../benchmarks/explore-allocation-ab-1500.md`](../benchmarks/explo CG-15 session and this one — the same magnitude as the gap. Agent wall-clock on this repo is noise-dominated at this sample size, which is the known shape (host-model thinking dominates, not tool latency). + +## Named members before incidental ones (2026-09-28) + +A query naming five independent functions in one file got two to four of them back. Repro, small +tier: `lib/response.ts` holds `sendBody`, `sendJson`, `renderView`, `redirectTo` and +`sendFileStream`, ~36 lines each, with three unrelated `headerSlotN(res, value)` helpers between +each pair and ~45 more after. Query: the five names. On `main`, `sendBody` rendered 0 of 36 lines +and `sendFileStream` 4 while helper bodies rendered in full. The same five functions written +contiguously all rendered. + +### Two places the helpers were paid first + +**Between clusters.** `rankedClusters` orders clusters of equal max importance by density, summed +member importance over span. Density counts every member, so the helpers merged around a named +function (a definition at 1, each `setHeader` edge line at 2) raised the density of its cluster, +and a cluster holding a named function alone ranked below it: 0.39 against 0.95 and 0.84 in the +repro. Taken whole, the two filler-heavy clusters spent the file's 9,100-char reservation and the +isolated `sendBody` got nothing. Reordering does not fix this. Rank `sendBody` first and the +victim moves to whichever named cluster is now last, because each cluster above it still pays for +its helpers before the one below gets a turn. + +**Inside one cluster, at its ceiling.** Every render is bounded by a ceiling, and for a cluster +taken after the first, and for every re-render the ceiling fit makes, that ceiling is its room +exactly. The member shrink estimates on raw source, which runs well under the rendered size +(padding and line numbers). Incidental members were kept on that estimate, the render overran the +ceiling, and the window back to it cut in source order, through the tail of a named body. That is +gin's `handleHTTPRequest` at 61 of 71 lines once the first fix had given it room, and, with the +named-concentration branch merged, vscode's `_remoteCall` at 23 of 36 when that branch's regrow +re-rendered `rpcProtocol.ts`. When the 12-line minimum +windows overran the room instead, the loop dropped the whole cluster, named members and all, and +the room went to clusters with nothing named in them. + +### The rule + +A member is **protected** if the query named it or it is an entry point (importance >= 9), or it +sits on the rendered call path. Everything else in a cluster is incidental. + +- Before selection, `protectedCoreCost` prices each cluster's protected members rendered on their + own, after the session history. `owedFrom` sums that cost over every cluster ranked below each + position. +- A cluster's incidental members may only use what is left of its room once the clusters below it + are paid their protected cost. Its own protected members are chosen first, against the same cap + as before, and are never held back. So when named bodies themselves do not all fit, the + trade-off between them is still the one rank makes. +- In any cluster holding a protected member, incidental members are also never kept past the + render's ceiling, in rendered size. That is `renderCluster`'s own rule, so it covers selection + and every re-render alike: the render fits without being windowed, and nothing named is cut to + make space for its neighbours. For the first cluster the ceiling is the 1.5× spine bound, so + the rule only bites near the point where the render would be windowed anyway (a little before + it, since each incidental member's padded cost is an over-estimate). +- A cluster with nothing protected in it renders byte-for-byte as before, and so does one that + fits its cap with room left for everything owed below it. A lone named function keeps the + helpers merged around it (the test's control). + +The incidental cap is measured against `fileBudget` even for a spine cluster: the clusters below +it that are not on the spine are held to `fileBudget`, so room a spine cluster's incidental +members take past it is room they never get. + +**Rejected: ranking by named density.** Dividing only protected members' importance by span, in +the named tier, stops filler from raising a cluster's rank. Measured on top of the hold-back +(before the ceiling guard was added), it changed 4 of 47 responses, left complete named +bodies where they were (218), and put tokio-pin over the 25K inline cap (25,007). Rank is left +alone. + +### Measured effect + +Deterministic, 47 fixed queries over the 7 README repos plus express: the README questions, the +suite's prose questions, precise symbol bags, pinned-file flows and named-concentration stress +queries (the set the unmerged named-concentration work was measured on). Baseline `main` +`e63fe2ec`, same index for both builds. Named coverage means every definition of every queried +name in every rendered file. + +| | `main` | this change | +|---|---|---| +| named bodies complete | 198 | **234** (17 responses up, **0 down**) | +| named lines delivered | 6,912 | **7,631** (+10%) | +| byte-identical responses | — | 17 of 47 | +| responses over the 25K inline cap | 5 | **1** | +| responses truncated by the final cut | 2 | 3 (see below) | + +No definition that rendered whole on `main` renders partially, except one 5-line overload: tokio's +`park.rs` `park` (5 → 2), in a response that completes three other named bodies while the 46-line +`park` beside it grows (11 → 20). Otherwise what gets smaller is the un-named part of a whole-class container +too big to keep as one member, whose body had been arriving as incidental members: Alamofire's +`Session` (133 → 103 of 1,301 lines), vscode's `RPCProtocol` (194 → 129) beside a now-complete +`MessageIO` class, `ChannelClient` (109 → 63) where the file's incidental lines are now chosen by +importance rather than by source order. No named method inside those classes loses a line. + +| query | complete named bodies | named lines | +|---|---|---| +| vscode `ipc.net.ts` pinned flow | 3 → 8 | 139 → 331 | +| alamofire `Request.swift` pinned | 2 → 6 | 18 → 90 | +| tokio `pb-worker2` | 7 → 10 | 104 → 168 | +| tokio `worker.rs` pinned | 6 → 9 | 182 → 278 | +| gin `gin.go ServeHTTP handleHTTPRequest addRoute` | 2 → 4 | 138 → 223 | +| django symbol bag (`execute_sql` ×2) | 5 → 7 | 201 → 261 | +| excalidraw symbol bag (`_renderStaticScene`) | 3 → 3 | 32 → 79 | + +**The one new truncation is django-bag, and it is not this rule's.** The named file now spends +157 chars less than its reservation, the carry-forward moves that down the rank order, and the +rank-3 file's funding rises from 963 to 1,777 because `owedPayableBelow` stops holding a partial +share for the last file once that share falls under `MIN_CHARS` + overhead. A 32-char shift +upstream crossed that threshold. The last file, a 151-char skeleton, is dropped, and the pointer +list after it is cut by the final trim because the epilogue is never paid for in the render loop. +The named-concentration branch reserves the epilogue; with both, the same query is not truncated +and renders as many files as that branch does alone (below). + +### On the named-concentration branch + +`claude/inspiring-grothendieck-8a7b3e` (`fd8eab23`, unmerged) was cut before #2057, and its own +tip is the wrong base to measure against: there `sectionText` still measures gaps with their names +while the window to a ceiling measures them bare, so a bound render that "fits" overruns its room +and the cluster is dropped. Merged onto `main` (one conflict: keep the branch's `buildSection` and +`main`'s bare-gap `sectionText`), then this change on top (four hunks in `shrinkCluster`, +`renderCluster` and the selection loop, all mechanical), the same 47 queries: + +| | `main` + named concentration | + this change | +|---|---|---| +| named bodies complete | 235 | **279** (19 responses up, 1 down) | +| named lines delivered | 9,330 | **9,965** (+7%) | +| responses over the 25K inline cap | 0 | 0 | +| responses truncated by the final cut | 0 | 0 | + +The response that goes down is tokio's pinned stress query, the same 5-line `park` overload as on +`main`. That branch's own tests (`explore-merged-spine-cluster`, `explore-named-file-valve`) pass +with this change merged in. The two hold-backs apply together to a merged spine cluster: that +branch's limits the whole first cluster to leave the other clusters' top members room, and this +one limits only incidental members, of any cluster. On vscode's `rpcProtocol.ts` that moves room +from `RPCProtocol`'s un-named methods to the named `MessageBuffer` class (66 → 187 of 187 lines, +`RPCProtocol` 383 → 220), with every named method of `RPCProtocol` still whole. + +On that branch a lone named file's reservation rises to 12,800, which absorbs the filler of the +original repro, so the hermetic fixture uses wider helper lines. It is red there without this +change (`sendJson` 0 of 26 lines). + +### Coverage + +`__tests__/explore-isolated-named-cluster.test.ts`, one generated file in the repro's shape +(26-line bodies, three wide helpers between each pair). Its shape block checks that helpers sit +between every pair, that no named function calls another (so no call path ranks one), that the +file is more than three times its reservation, that the named bodies cost under 90% of it, and +that the same functions written contiguously all render whole under the same reservation. + +| Mutation | Red | +|---|---| +| `main`'s `tools.ts` | five-name gate: `sendBody` 0/26 | +| `main` + named concentration, without this change | five-name gate: `sendJson` 0/26 | +| No hold-back for clusters below (ceiling guard kept) | five-name gate: `sendBody` 21/26 | +| No ceiling guard on incidental members (hold-back kept) | `renderView redirectTo`: `renderView` 22/26 | +| Incidental members always dropped | control: the helper above a lone `renderView` | +| Protected members not sorted first in the guarded shrink | none — and all 47 sweep responses are byte-identical | + +The last lever is kept on reasoning alone. Named members already sort first by importance, so the +sort only matters for a call-path member below importance 9 that shares a guarded cluster with a +higher-importance incidental one. There it keeps the incidental member from claiming the cap +first, and it keeps the size the incidental test starts from exact: that is the rendered cost of +the protected set only when every protected member has been weighed before the first incidental +one. + +With the repro's exact geometry (36-line bodies), five padded bodies come to more than the 9,100 +reservation. This change renders four whole and `sendBody` at 29 of 36, the lowest-ranked named +cluster windowed to what is left. That is the budget, not the defect: with the named-concentration +branch as well, all five render whole. + +### Found, not fixed + +- **`owedPayableBelow`'s partial hold has a cliff at `MIN_CHARS` + overhead.** Just above it, the + last admitted file is held its partial share. Just below it, it is held nothing, and the file + above takes ~800 chars the last file needed. Any upstream shift of a few dozen chars can flip it + (django-bag above). diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 8e9c8138a0..850e1644b6 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -5648,6 +5648,48 @@ export class ToolHandler { }]; }; + /** Rendered cost of one source line, line numbering included. */ + const lineCost = (ln: number): number => + (fileLines[ln - 1] ?? '').length + 1 + (withLineNumbers ? String(ln).length + 1 : 0); + /** Rendered cost of one member on its own, context padding included. */ + const paddedCost = (r: ExploreRange): number => { + let chars = 0; + const hi = Math.min(fileLines.length, r.end + contextPadding); + for (let ln = Math.max(1, r.start - contextPadding); ln <= hi; ln++) chars += lineCost(ln); + return chars; + }; + /** + * A member the query asked for: one it named or that is an entry point + * (importance >= 9), or one on the rendered call path. The rest of a + * cluster — glue, connected nodes, edge lines, peripheral declarations — + * is incidental context that happens to sit within `gapThreshold` of it. + */ + const isProtectedMember = (r: ExploreRange): boolean => r.importance >= 9 || r.spine; + /** Whole member ranges, re-merged in source order and rendered. */ + const partsOfMembers = (members: ReadonlyArray): SectionPart[] => { + // Re-merge in source order so adjacent survivors read as one block + // rather than a stutter of one-symbol fragments. + const sorted = [...members].sort((a, b) => a.start - b.start); + const merged: Array<{ start: number; end: number }> = []; + for (const r of sorted) { + const last = merged[merged.length - 1]; + if (last && r.start <= last.end + gapThreshold) last.end = Math.max(last.end, r.end); + else merged.push({ start: r.start, end: r.end }); + } + return merged.flatMap((m) => buildSection(m)); + }; + /** + * What this cluster's protected members cost to render on their own, after + * the session history — the room a cluster ranked ABOVE it must leave for + * them. 0 for a cluster with nothing the query asked for. + */ + const protectedCoreCost = (c: ExploreCluster): number => { + const core = c.members.filter(isProtectedMember); + if (core.length === 0) return 0; + const parts = core.length === c.members.length ? buildSection(c) : partsOfMembers(core); + return sectionText(dedupeSpans(parts).parts).length + GAP_MARKER.length; + }; + /** * Shrink an oversize cluster to the highest-importance symbols inside it * that fit `cap`, rendered in source order with gap markers (CG-12). @@ -5676,14 +5718,30 @@ export class ToolHandler { * decide WHICH members survive: that is the ceiling trim's job, and CG-38 is * why that trim now protects the named spans instead of cutting in source * order. See `docs/benchmarks/explore-tail-render-cg38.md`. + * + * `incidentalCap` is set for a cluster holding a member the query asked for + * (see `renderCluster` and `protectedCoreCost`). Protected members are then + * chosen first, exactly as above; every other member must also fit + * `incidentalCap`, which is measured on the rendered text (padding, line + * numbers) because it is a bound — the ceiling past which the render is + * windowed, or bytes owed to a cluster below — not an estimate. */ - const shrinkCluster = (c: ExploreCluster, cap: number): SectionPart[] | null => { + const shrinkCluster = ( + c: ExploreCluster, cap: number, incidentalCap?: number, + ): SectionPart[] | null => { if (c.members.length < 2) return null; + const guarded = incidentalCap !== undefined; const byImportance = [...c.members].sort((a, b) => - b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start); + (guarded ? Number(isProtectedMember(b)) - Number(isProtectedMember(a)) : 0) + || b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start); const sizeOf = (r: ExploreRange) => fileLines.slice(r.start - 1, r.end).join('\n').length; const keep: ExploreRange[] = []; let kept = 0; + // Rendered size of what is kept so far, once the first incidental member + // is weighed. Every protected member sorts ahead of it, so this is the + // exact cost of the protected set; each incidental member then adds its + // own padded span, an over-estimate that errs toward the cluster below. + let rendered: number | undefined; for (const r of byImportance) { const sz = sizeOf(r) + GAP_MARKER.length; // Always keep the most important range, even if it alone is oversize — @@ -5691,20 +5749,17 @@ export class ToolHandler { // far it may overshoot is bounded by the caller's ceiling (CG-30), which // windows a runaway member instead of dropping it. if (keep.length > 0 && kept + sz > cap) continue; + if (guarded && keep.length > 0 && !isProtectedMember(r)) { + rendered ??= sectionText(dedupeSpans(partsOfMembers(keep)).parts).length; + const cost = paddedCost(r) + GAP_MARKER.length; + if (rendered + cost > incidentalCap) continue; + rendered += cost; + } keep.push(r); kept += sz; } if (keep.length === c.members.length) return null; - // Re-merge the kept ranges in source order so adjacent survivors read as - // one block rather than a stutter of one-symbol fragments. - keep.sort((a, b) => a.start - b.start); - const merged: Array<{ start: number; end: number }> = []; - for (const r of keep) { - const last = merged[merged.length - 1]; - if (last && r.start <= last.end + gapThreshold) last.end = Math.max(last.end, r.end); - else merged.push({ start: r.start, end: r.end }); - } - return merged.flatMap((m) => buildSection(m)); + return partsOfMembers(keep); }; /** @@ -5721,9 +5776,6 @@ export class ToolHandler { * the call path IS the answer. */ const MIN_WINDOW_LINES = 12; - /** Rendered cost of one source line, line numbering included. */ - const lineCost = (ln: number): number => - (fileLines[ln - 1] ?? '').length + 1 + (withLineNumbers ? String(ln).length + 1 : 0); /** * Longest prefix of `r` that fits `room`. `minLines` is the never-empty * floor — it may overrun `room`, so it is only ever asked for when nothing @@ -5891,8 +5943,25 @@ export class ToolHandler { * never touched. */ ceiling: number = Infinity, + /** + * What this cluster's INCIDENTAL members may render into, when a cluster + * ranked below it holds a member the query asked for — see the selection + * loop. + */ + incidentalCap?: number, ): { parts: SectionPart[]; covered: ExploreLineRange[]; shrunk: boolean } => { const base = dedupeSpans(buildSection(c)); + // In a cluster holding a member the query asked for, incidental members + // are never kept past `ceiling` either. The member shrink estimates on raw + // source, which runs well under the rendered size, so it kept helpers the + // render then had no room for, and the window back to the ceiling cut in + // source order — through the tail of a named body (gin's + // `handleHTTPRequest` lost 10 of 71 lines to the helpers above it), or, + // when its minimum windows overran a later cluster's room, costing that + // cluster its place. Every re-render into an exact room is the same case. + const guard = Number.isFinite(ceiling) && c.members.some(isProtectedMember) + ? Math.min(incidentalCap ?? Infinity, ceiling) + : incidentalCap; const bound = ( r: { parts: SectionPart[]; covered: ExploreLineRange[]; shrunk: boolean }, ) => { @@ -5902,10 +5971,10 @@ export class ToolHandler { const parts = windowToCeiling(r.parts, ceiling, focusLinesOf(c)); return { parts, covered: r.covered, shrunk: true }; }; - if (sectionText(base.parts).length <= cap) { + if (sectionText(base.parts).length <= Math.min(cap, guard ?? Infinity)) { return { parts: base.parts, covered: base.covered, shrunk: false }; } - const shrunk = shrinkCluster(c, cap); + const shrunk = shrinkCluster(c, cap, guard); if (shrunk === null) { return bound({ parts: base.parts, covered: base.covered, shrunk: false }); } @@ -5969,7 +6038,32 @@ export class ToolHandler { const renderedClusters = new Map>(); let anyClusterShrunk = false; let projectedChars = 0; - for (const rc of rankedClusters) { + // What the protected members of every cluster ranked BELOW position p cost + // to render. Rank orders clusters, not members: among clusters of equal + // max importance it prefers density, and density counts every member, so + // a named function with unrelated helpers merged around it outranks a + // cluster holding that same kind of function alone — the helpers' edge + // lines raise its density. Taken whole, the first cluster then spent the + // file's budget on the helpers and the isolated named function rendered + // nothing. So a cluster's incidental members may only use what is left + // once the named members below it are paid for. Its own protected members + // are never held back, so the trade-off between two named symbols is the + // one rank already made. + const owedFrom = new Array(rankedClusters.length + 1).fill(0); + for (let p = rankedClusters.length - 1; p >= 0; p--) { + owedFrom[p] = owedFrom[p + 1]! + protectedCoreCost(rankedClusters[p]!.c); + } + // Measured against `fileBudget` even for a spine cluster: the clusters + // below it that are not on the spine are held to `fileBudget`, so room a + // spine cluster's incidental members take past it is room they never get. + const incidentalCapFor = (p: number, room: number): number | undefined => { + const owedBelow = owedFrom[p + 1]!; + if (owedBelow === 0) return undefined; + const spent = projectedChars + (chosenIndices.size > 0 ? GAP_MARKER.length : 0); + return Math.min(room, fileBudget - spent) - owedBelow; + }; + for (let p = 0; p < rankedClusters.length; p++) { + const rc = rankedClusters[p]!; // The top-ranked cluster is always taken — an empty file section sends the // agent to Read, negating the savings. But "always taken" is not "taken at // any size": when it overruns the reservation it is SHRUNK to the @@ -5987,7 +6081,7 @@ export class ToolHandler { // this holds it to it rather than letting the member rule walk past it. const ceiling = Math.max(cap, SPINE_CEILING); if (first) { - const section = renderCluster(rc.c, cap, ceiling); + const section = renderCluster(rc.c, cap, ceiling, incidentalCapFor(p, cap)); renderedClusters.set(rc.idx, section); anyClusterShrunk = anyClusterShrunk || section.shrunk; chosenIndices.add(rc.idx); @@ -6012,7 +6106,7 @@ export class ToolHandler { // fragments the next call's dedup then has to shred around. const room = cap - projectedChars - GAP_MARKER.length; if (room < EXPLORE_ALLOCATION.MIN_CHARS) continue; - const section = renderCluster(rc.c, room, room); + const section = renderCluster(rc.c, room, room, incidentalCapFor(p, room)); const text = sectionText(section.parts); if (text.length === 0) continue; // The never-empty floors inside the windowing may overrun `room` (a From 47840a6ed47ada8b10bf28d67b574ccc290cb349 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 06:20:54 +0000 Subject: [PATCH 003/363] fix(explore): return a qualified or line-anchored method's body instead of its signature (#2063) `SQLCompiler.as_sql pre_sql_setup get_select` on django returned get_select and pre_sql_setup in full and SQLCompiler.as_sql -- qualified, step 1 of the Flow, 226 lines -- as one signature line. compiler.py renders as a per-symbol focused view, which picked bodies in source order within a tier, so the unnamed bridge get_qualify_sql and get_select (both above as_sql in the file) took the cap. The follow-ups agents issue (`compiler.py:776`, `compiler.py lines 900-1003`) pinned the file and dropped the line numbers. Every agent run ended in a Read of compiler.py. - query-paths: keep `:776` / `:12-40` / `#L88-L120` suffixes as line anchors and bind prose ranges (`lines 900-1003`, `L900-L1003`, `lines 900 to 1003`) to the nearest single-file path. - EXACT targets: a qualified name resolving to <=3 callables, or the callable enclosing a single-line anchor. Multi-line anchors render as that span. - Focused view: tiers exact > named spine step > unnamed bridge > unique named > family co-named, spine in call order. A tier 0-1 body that does not fit is windowed (head + calls into the question's other symbols) with the explore range that returns each hole, never reduced to a signature. Priced in rendered chars net of the signature lines, so the section keeps its promise. - Cluster path: exact clusters rank first with the spine's bounded overshoot, shrink by exact rendered size, and hold back what lower-ranked exact clusters still owe; files holding one are funded like spine files. - Blast radius leads with exact targets. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/explore-exact-target.test.ts | 306 +++++++++++ __tests__/query-paths.test.ts | 60 ++- docs/design/adaptive-explore-sizing.md | 58 ++ src/mcp/tools.ts | 699 +++++++++++++++++-------- src/search/query-paths.ts | 128 ++++- 6 files changed, 1030 insertions(+), 223 deletions(-) create mode 100644 __tests__/explore-exact-target.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cef29ae955..4bcd635d1a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -165,6 +165,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - When `codegraph_explore` trims a large file, the gaps in its source now name the symbols that were left out, using only room the file's budget has left, so the names never push out the code itself. (#1711) - `codegraph_explore` no longer reports "runtime dispatch" through a base class that doesn't declare the method, so a common name shared by many unrelated subclasses no longer opens an answer with a list of thousands of implementations. - When you name several functions in one file and unrelated code sits between them, `codegraph_explore` now returns the functions you named before the code around them, instead of spending the file's room on that code and leaving one of the functions you asked for with no source at all. +- `codegraph_explore` now returns the body of a method you name together with its class, such as `SQLCompiler.as_sql`, even in a file full of same-named overrides. It used to shrink the method you asked for to its signature line while smaller methods around it came back in full. A method too long for the answer now comes back as its opening lines plus the places it calls the other symbols you named, with the exact follow-up query that returns the rest. +- `codegraph_explore` now honours line numbers in a query: `compiler.py:776` returns the method containing that line, and `compiler.py lines 900-1003` (or `:900-1003`, `#L900-L1003`, `L900-L1003`) returns exactly those lines. It used to pin the file and ignore the numbers. #### MCP / indexing diff --git a/__tests__/explore-exact-target.test.ts b/__tests__/explore-exact-target.test.ts new file mode 100644 index 0000000000..d54d36798f --- /dev/null +++ b/__tests__/explore-exact-target.test.ts @@ -0,0 +1,306 @@ +/** + * EXACT targets in `codegraph_explore`: a qualified name (`SQLCompiler.as_sql`) + * or a line anchor (`compiler.py:776`, `compiler.py lines 900-1003`). + * + * The originating gap (django, measured 2026-09-27 on main and on every branch + * since): `SQLCompiler.as_sql pre_sql_setup get_select` returned `get_select` + * and `pre_sql_setup` in full and `SQLCompiler.as_sql` — the qualified name, + * step 1 of the rendered Flow, 226 lines — as ONE signature line. compiler.py + * is a base+subclasses "family" file, so it renders as a per-symbol focused + * view, and that view chose bodies in SOURCE order within a tier: the unnamed + * bridge `get_qualify_sql` and `get_select` sit above `as_sql` in the file, took + * the cap, and the method the agent asked for collapsed. Its follow-ups + * (`SQLCompiler.as_sql full body compiler.py:776`, `compiler.py lines 900-1003`) + * pinned the file but dropped the line numbers, so they returned other methods' + * clusters — and every agent run ended in a Read of compiler.py. + * + * The fixture is that file in miniature (Python, like django): `SQLCompiler` + * with `get_select` and `get_qualify_sql` above a larger `as_sql`, three + * subclasses overriding `as_sql`, and a second family file with a base method + * too big for any budget. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { ToolHandler } from '../src/mcp/tools'; +import CodeGraph from '../src/index'; + +/** Line numbers rendered for `file` in an explore response. */ +function renderedLines(text: string, file: string): Set { + const out = new Set(); + let current: string | null = null; + let inFence = false; + for (const line of text.split('\n')) { + const header = /^\*\*`([^`]+)`\*\*/.exec(line); + if (header && !inFence) { current = header[1]!; continue; } + if (line.startsWith('```')) { inFence = !inFence; continue; } + if (inFence && current === file) { + const m = /^(\d+)\t/.exec(line); + if (m) out.add(Number(m[1])); + } + } + return out; +} + +/** The `**``**` section, header through the line before the next file header. */ +function sectionFor(text: string, file: string): string { + const lines = text.split('\n'); + const start = lines.findIndex((l) => l.startsWith(`**\`${file}\`**`)); + if (start < 0) return ''; + let end = lines.length; + for (let i = start + 1; i < lines.length; i++) { + if (lines[i]!.startsWith('**`')) { end = i; break; } + } + return lines.slice(start, end).join('\n'); +} + +/** `n` filler statements, each a distinct line so nothing dedups or folds. */ +function filler(tag: string, n: number, indent = ' '): string { + return Array.from({ length: n }, (_, i) => + `${indent}${tag}_${i} = self.query.alias_refcount.get("${tag}_${i}", 0) + len(self.query.select)`, + ).join('\n'); +} + +/** + * Unrelated methods, so compiler.py is a real family file's size (django's is + * 2,291 lines): small enough to ship WHOLE, the file would answer every + * question by accident and hide the gap. + */ +function helpers(n: number): string { + return Array.from({ length: n }, (_, i) => ` + def helper_${i}(self, value): + """Unrelated helper ${i}.""" + first = self.query.alias_map.get(value) + second = self.query.alias_refcount.get(value, 0) + third = self.query.external_aliases.get(value, False) + return first, second, third`).join('\n'); +} + +const COMPILER = 'compiler.py'; +const PLANNER = 'planner.py'; + +describe('codegraph_explore — exact targets (qualified names, line anchors)', () => { + let testDir: string; + let cg: CodeGraph; + let handler: ToolHandler; + const lineOf: Record = {}; + + beforeAll(async () => { + testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-explore-exact-')); + + // compiler.py — get_select and get_qualify_sql ABOVE as_sql, as in django. + const compiler = ` +class SQLCompiler: + def pre_sql_setup(self, with_col_aliases=False): + # PRE_SQL_SETUP_BODY + self.setup_query(with_col_aliases=with_col_aliases) + order_by = self.get_order_by() + return order_by + + def setup_query(self, with_col_aliases=False): + self.select = self.get_select(with_col_aliases=with_col_aliases) + return self.select + + def get_order_by(self): + return [] + + def get_select(self, with_col_aliases=False): + # GET_SELECT_BODY +${filler('sel', 34)} + return [] + + def get_qualify_sql(self): + # GET_QUALIFY_BODY +${filler('qual', 34)} + inner = self.get_select() + return inner + + def as_sql(self, with_limits=True, with_col_aliases=False): + # AS_SQL_HEAD + order_by = self.pre_sql_setup(with_col_aliases=with_col_aliases) +${filler('head', 30)} + result = self.get_qualify_sql() +${filler('tail', 30)} + return result # AS_SQL_TAIL_MARKER + + def execute_sql(self): + sql = self.as_sql() + return sql + + +${helpers(40)} + +def render_sql(compiler): + return SQLCompiler.as_sql(compiler) + + +class SQLInsertCompiler(SQLCompiler): + def as_sql(self): + # INSERT_AS_SQL_BODY + return super().as_sql() + + def execute_sql(self): + sql = self.as_sql() + return sql + + +class SQLUpdateCompiler(SQLCompiler): + def as_sql(self): + # UPDATE_AS_SQL_BODY + return super().as_sql() + + def pre_sql_setup(self): + # UPDATE_PRE_SQL_SETUP_BODY + return super().pre_sql_setup() + + +class SQLDeleteCompiler(SQLCompiler): + def as_sql(self): + # DELETE_AS_SQL_BODY + return super().as_sql() +`.trimStart(); + fs.writeFileSync(path.join(testDir, COMPILER), compiler); + const compilerLines = compiler.split('\n'); + const find = (lines: string[], needle: string) => lines.findIndex((l) => l.includes(needle)) + 1; + lineOf.asSqlDef = find(compilerLines, 'def as_sql(self, with_limits'); + lineOf.asSqlTail = find(compilerLines, 'AS_SQL_TAIL_MARKER'); + lineOf.asSqlTailStart = find(compilerLines, 'result = self.get_qualify_sql()') + 1; + + // planner.py — a family whose base method is too big for ANY budget. It + // calls `stage()` (the spine's next hop) near its END, so a head-only cut + // would lose the very line the flow runs through. + const planner = ` +class Planner: + def plan(self, query): + # PLAN_HEAD +${filler('plan', 420)} + staged = self.stage(query) # PLAN_CALLS_STAGE +${filler('post', 6)} + return staged + + def stage(self, query): + return self.finalize(query) + + def finalize(self, query): + # FINALIZE_BODY + return query + + def explain_plan(self, query): + # EXPLAIN_PLAN_BODY + return str(query) + + +class HashPlanner(Planner): + def plan(self, query): + return super().plan(query) + + +class MergePlanner(Planner): + def plan(self, query): + return super().plan(query) + + +class NestedPlanner(Planner): + def plan(self, query): + return super().plan(query) +`.trimStart(); + fs.writeFileSync(path.join(testDir, PLANNER), planner); + lineOf.planCallsStage = find(planner.split('\n'), 'PLAN_CALLS_STAGE'); + + // Other `as_sql`s, so the bare name is a family (as django's 110 are). + fs.writeFileSync(path.join(testDir, 'lookups.py'), ` +class Exact: + def as_sql(self, compiler, connection): + return "%s = %s" + + +class IExact: + def as_sql(self, compiler, connection): + return "UPPER(%s) = UPPER(%s)" +`.trimStart()); + + cg = CodeGraph.initSync(testDir, { config: { include: ['**/*.py'], exclude: [] } }); + await cg.indexAll(); + handler = new ToolHandler(cg); + }); + + afterAll(() => { + if (cg) cg.destroy(); + if (testDir && fs.existsSync(testDir)) fs.rmSync(testDir, { recursive: true, force: true }); + }); + + const explore = async (query: string): Promise => { + const result = await handler.execute('codegraph_explore', { query }); + return result.content?.[0]?.text ?? ''; + }; + + it('fixture sanity: the flow runs as_sql → get_qualify_sql → get_select and compiler.py goes focused', async () => { + const text = await explore('SQLCompiler.as_sql pre_sql_setup get_select'); + expect(text).toContain('**Flow (call path among the symbols you queried)'); + expect(text).toMatch(/1\. as_sql \(compiler\.py:\d+\)[\s\S]*2\. get_qualify_sql[\s\S]*3\. get_select/); + expect(sectionFor(text, COMPILER), 'the family file renders as a per-symbol view').toContain('· focused'); + }); + + it('returns the qualified method\'s WHOLE body, not its signature line', async () => { + const text = await explore('SQLCompiler.as_sql pre_sql_setup get_select'); + const section = sectionFor(text, COMPILER); + const lines = renderedLines(text, COMPILER); + for (let ln = lineOf.asSqlDef!; ln <= lineOf.asSqlTail!; ln++) { + expect(lines.has(ln), `as_sql line ${ln} rendered`).toBe(true); + } + expect(section).toContain('AS_SQL_TAIL_MARKER'); + // The other named step still gets its body too. + expect(section).toContain('GET_SELECT_BODY'); + }); + + it('keeps the family skeleton for the subclasses: their as_sql overrides stay signatures', async () => { + const text = await explore('SQLCompiler.as_sql pre_sql_setup get_select'); + const section = sectionFor(text, COMPILER); + expect(section).not.toContain('INSERT_AS_SQL_BODY'); + expect(section).not.toContain('DELETE_AS_SQL_BODY'); + }); + + it('leads the blast radius with the qualified method, not a same-named override', async () => { + const text = await explore('SQLCompiler.as_sql pre_sql_setup get_select'); + const blast = text.slice(text.indexOf('**Blast radius'), text.indexOf('**Relationships**')); + const first = blast.split('\n').find((l) => l.startsWith('- `')); + expect(first).toContain(`\`as_sql\` (${COMPILER}:${lineOf.asSqlDef})`); + }); + + it('a `file:line` anchor returns the method enclosing that line, whole', async () => { + // No symbol named — the line alone has to say which method. + const text = await explore(`${COMPILER}:${lineOf.asSqlDef! + 40} full body`); + const lines = renderedLines(text, COMPILER); + for (let ln = lineOf.asSqlDef!; ln <= lineOf.asSqlTail!; ln++) { + expect(lines.has(ln), `as_sql line ${ln} rendered`).toBe(true); + } + }); + + it('a `lines A-B` range returns exactly that span of the pinned file', async () => { + // The tail of `as_sql` — the span a windowed render elides and the agent + // then asks for by number (django's `lines 900-1003 as_sql tail`). + const a = lineOf.asSqlTailStart!; + const b = lineOf.asSqlTail!; + const text = await explore(`${COMPILER} lines ${a}-${b} tail`); + const lines = renderedLines(text, COMPILER); + for (let ln = a; ln <= b; ln++) expect(lines.has(ln), `line ${ln} rendered`).toBe(true); + }); + + it('windows an exact body too big for the budget — head AND the call the flow runs through — and names the rest by range', async () => { + const text = await explore('Planner.plan finalize explain_plan'); + expect(text).toContain('**Flow (call path among the symbols you queried)'); + const section = sectionFor(text, PLANNER); + const lines = renderedLines(text, PLANNER); + expect(section).toContain('PLAN_HEAD'); + // The spine's next-hop call sits ~420 lines into the body: a head-only cut + // would drop it. + expect(lines.has(lineOf.planCallsStage!), 'the call into stage() is rendered').toBe(true); + // Windowed, not dumped: most of the 430-line body is elided... + expect(lines.size).toBeLessThan(300); + // ...and each hole is named with the explore query that returns it. + expect(section).toMatch(/lines \d+-\d+ of `plan` elided — codegraph_explore `planner\.py:\d+-\d+` returns them/); + // Steered to explore, never to Read (the tag's own "do NOT Read" aside). + expect(section.replace(/do NOT Read/g, '')).not.toMatch(/\bRead\b/); + }); +}); diff --git a/__tests__/query-paths.test.ts b/__tests__/query-paths.test.ts index 06ea29a5ea..956c33c855 100644 --- a/__tests__/query-paths.test.ts +++ b/__tests__/query-paths.test.ts @@ -151,7 +151,7 @@ describe('extractQueryPaths — resolution and stripping', () => { it('passes through untouched when nothing resolves', () => { const q = 'plain prose question about scrolling'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); }); }); @@ -191,13 +191,13 @@ describe('extractQueryPaths — extension-less kebab basenames', () => { it('leaves kebab prose that names no indexed file untouched — and unreported', () => { const q = 'how does cross-call dedup make explore non-blocking'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); }); it('leaves a stem shared by too many files alone — one hot name must not pin half the repo', () => { const q = 'refactor the user-profile rendering'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); }); it('pins all files sharing a stem when within the ambiguity budget', () => { @@ -317,3 +317,57 @@ describe('extractQueryPaths — dotless slashed spans, decided on disk', () => { expect(p.asked).not.toContain('scripts/pre-commit'); }); }); + +describe('extractQueryPaths — line anchors', () => { + // The django follow-ups that went unanswered: an agent handed a signature for + // a 226-line method asks for the body by line, and explore used to pin the + // file but drop the lines — answering "the file's most relevant clusters" + // instead of the span it named. + const CHAT = 'src/lib/chat-manager.ts'; + + it('keeps a :line / :start-end / #L suffix as an anchor on the pinned file', () => { + expect(extractQueryPaths(`body of ${CHAT}:776`, INDEX).lineAnchors) + .toEqual([{ file: CHAT, start: 776, end: 776 }]); + expect(extractQueryPaths(`see ${CHAT}:12-40`, INDEX).lineAnchors) + .toEqual([{ file: CHAT, start: 12, end: 40 }]); + expect(extractQueryPaths('regression at src/lib/task-runner-manager.ts#L88-L120', INDEX).lineAnchors) + .toEqual([{ file: 'src/lib/task-runner-manager.ts', start: 88, end: 120 }]); + }); + + it('binds a prose line range to the path it sits next to and removes it from the query', () => { + const out = extractQueryPaths(`${CHAT} lines 900-1003 flushQueue tail`, INDEX); + expect(out.lineAnchors).toEqual([{ file: CHAT, start: 900, end: 1003 }]); + // `lines` would feed FTS a word every file holds; the numbers match nothing. + expect(out.strippedQuery).toBe('flushQueue tail'); + }); + + it('accepts the other spellings agents write', () => { + const anchor = (q: string) => extractQueryPaths(q, INDEX).lineAnchors; + expect(anchor(`lines 900 to 1003 of ${CHAT}`)).toEqual([{ file: CHAT, start: 900, end: 1003 }]); + expect(anchor(`L900-L1003 in ${CHAT}`)).toEqual([{ file: CHAT, start: 900, end: 1003 }]); + expect(anchor(`${CHAT} line 42`)).toEqual([{ file: CHAT, start: 42, end: 42 }]); + expect(anchor(`${CHAT} 900-1003`)).toEqual([{ file: CHAT, start: 900, end: 1003 }]); + expect(anchor(`${CHAT} (lines 1003-900)`)).toEqual([{ file: CHAT, start: 900, end: 1003 }]); + }); + + it('binds each range to its NEAREST path when the query names two files', () => { + const out = extractQueryPaths( + `${CHAT} lines 10-20 and src/lib/task-runner-manager.ts lines 30-40`, INDEX, + ); + expect(out.lineAnchors).toEqual([ + { file: CHAT, start: 10, end: 20 }, + { file: 'src/lib/task-runner-manager.ts', start: 30, end: 40 }, + ]); + }); + + it('leaves line numbers alone when there is no path, or the path is ambiguous', () => { + const noPath = extractQueryPaths('flushQueue lines 900-1003', INDEX); + expect(noPath.lineAnchors).toEqual([]); + // A bare number with no line context is not a line number either. + expect(extractQueryPaths(`${CHAT} retries 3 times`, INDEX).lineAnchors).toEqual([]); + // `generic-modal` pins two files; a line number means nothing across both. + const twoFiles = extractQueryPaths('generic-modal.tsx:40', INDEX); + expect(twoFiles.pinnedFiles).toHaveLength(2); + expect(twoFiles.lineAnchors).toEqual([]); + }); +}); diff --git a/docs/design/adaptive-explore-sizing.md b/docs/design/adaptive-explore-sizing.md index 5c72ef199c..9e8135da2a 100644 --- a/docs/design/adaptive-explore-sizing.md +++ b/docs/design/adaptive-explore-sizing.md @@ -56,6 +56,57 @@ source. > first, so the budget cap never binds it). OkHttp's interceptors stay a pure > signature skeleton (no named callable in them, don't define a supertype). +> **Refinement (2026-09-28) — exact targets.** The per-symbol view above still +> lost the one method the agent asked for by its qualified name. On a fresh django +> index `SQLCompiler.as_sql pre_sql_setup get_select` returned `get_select` and +> `pre_sql_setup` in full and `SQLCompiler.as_sql` — step 1 of the rendered Flow, +> 226 lines — as its signature line, on main and on every branch since. Bodies +> were chosen in SOURCE order within a tier, so the unnamed bridge +> `get_qualify_sql` and `get_select` (both above `as_sql` in the file) took the +> cap. The agents' follow-ups (`SQLCompiler.as_sql full body compiler.py:776`, +> `compiler.py lines 900-1003 as_sql tail`) pinned the file but dropped the line +> numbers, returning `SQLInsertCompiler.as_sql` and friends instead; every run +> Read compiler.py. The fix: +> 1. **Exact targets** — a qualified name that resolves to ≤3 callables +> (`SQLCompiler.as_sql`: 1 of 110), or the callable enclosing a line anchor +> (`compiler.py:776`). A bare `as_sql` is NOT exact, so the family rules above +> still keep the 110 overrides as signatures. +> 2. **Tiers in the focused view:** exact → spine step the agent named → unnamed +> spine bridge → uniquely-named off-spine → family co-named; the spine in CALL +> order within its tier. +> 3. **Window, never signature, for tiers 0–1.** A body too big for what is left +> is cut to its head plus the lines where it calls the question's other +> symbols, and each hole names the explore query that returns it +> (`codegraph_explore \`compiler.py:871-1001\``). +> 4. **Line anchors** (`query-paths.ts`): `file:line`, `file:a-b`, `#La-Lb`, and +> prose `lines a-b` / `L a-L b` next to a path. A range renders as exactly that +> span (cluster path, ranked first). +> 5. **Priced in rendered chars.** The view counted raw source while it rendered +> line numbers and ~20 signature lines; once windows fill the cap the +> undercount became a 19% overshoot that zeroed four excalidraw files below it. +> +> Deterministic replay (54 queries × 8 repos): 43 byte-identical; `as_sql` +> complete in all 5 distinct django queries that single it out (was 1, 0, 175, +> 0 and 226 of 226 lines), the `lines 900-1003` span 104/104 (was 1); named-body +> lines 8,706 → 9,443. The trade: where the file's budget cannot hold everything, a large named +> step now wins over several small off-spine bodies (`pb-compiler`: `as_sql` whole, +> `execute_sql`/`results_iter` to signatures). +> +> Agent A/B (`ab-new-vs-baseline.sh`, django, README question pair, Sonnet / +> `--effort high`, RUNS=3, baseline `e63fe2ec`): **Read 0 in 3/3 new runs vs a +> Read of compiler.py in 2/3 baseline runs** — each right after the agent's own +> `SQLCompiler.as_sql … compiler.py:776` explore, the exact follow-up the report +> described. Tool calls 3 → 4 median, retrieval residual 19.5K vs 22.9K tokens, +> median cost $0.195 vs $0.229 and 30s vs 35s (n=3, the last two within noise). +> +> Rebased onto #2062 (protected members paid before incidental ones across +> clusters), which leaves this gap open — `as_sql` 1/226 on that main. The two +> compose: #2062's protected/incidental rule is the cluster path's base, and +> exact targets sit on top of it (ranked first, every member priced in rendered +> chars, held back for lower exact clusters). Re-measured against that main, 59 +> queries: 43 byte-identical, named-body lines 9,804 → 10,731, every target span +> above complete. + --- ## TL;DR @@ -260,8 +311,15 @@ the agent's real query — that mismatch is what this refinement corrects.) `definesPolymorphicSupertype()` (a node HAS ≥3 incoming `implements`/`extends` — i.e. the file is the family base). The skeleton branch: `off-spine && isPolymorphicSibling && !(namedInFile && !definesSupertype)`. + `exactNodeIds` / `anchorSpans` (the exact targets) and `bodyFocusLines()` + (what a window of an oversize body must reach) are shared by the focused + view and the cluster path. +- `src/search/query-paths.ts` — `lineAnchors`: `file:line` / range parsing. - `__tests__/adaptive-explore-sizing.test.ts` — 7 cases incl. the named-callable spare (RealCall) and the supertype-family override (compiler.py). +- `__tests__/explore-exact-target.test.ts` — the django shape in miniature: + qualified body whole, subclass overrides still signatures, blast-radius lead, + `file:line` and `lines a-b` anchors, and a windowed body keeping its tail call. ## Frontier / future work diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 850e1644b6..21a4bbdf02 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -34,7 +34,7 @@ import type { PendingFile } from '../sync'; import type { Node, Edge, SearchResult, Subgraph, NodeKind } from '../types'; import { isTestFile, normalizeNameToken } from '../search/query-utils'; import { groupDefinitions, lastQualifierPart, matchesSymbol } from '../graph/symbol-lookup'; -import { extractQueryPaths, queryMightContainPaths } from '../search/query-paths'; +import { extractQueryPaths, queryMightContainPaths, type QueryLineAnchor } from '../search/query-paths'; import { existsSync, readFileSync, @@ -679,7 +679,10 @@ export interface ExploreAllocationCandidate { * self-referential generated file scores well on both. */ worth: number; - /** Carries a symbol on the rendered flow spine. */ + /** + * Carries a symbol on the rendered flow spine — or an exact target (a + * qualified name, a line anchor), which is the answer by the same argument. + */ spine: boolean; /** * The query named this file by PATH (see query-paths.ts). Pinned files are @@ -902,6 +905,23 @@ const POINTER_MAX_FILES = 10; */ const ELIDED_SYMBOL_CAP = 6; +/** + * Kinds a single-line anchor (`compiler.py:776`) resolves to: the innermost + * one containing the line is the symbol the agent is pointing at. A class + * enclosing the line is deliberately NOT a target — it spans most of its file, + * and "the whole class" is not what a line number asks for. + */ +const ANCHOR_CALLABLE_KINDS = new Set(['method', 'function', 'constructor', 'component']); +/** Lines either side of a single-line anchor that no callable encloses. */ +const ANCHOR_LINE_CONTEXT = 15; + +/** + * Most windows one oversize exact / named-spine body is cut into in the + * focused view: the head plus the call sites into the other symbols the + * question named. More turns a method into confetti. + */ +const MAX_BODY_FOCUS_LINES = 6; + type ElidedSymbolRef = { name: string; kind: string; startLine: number }; /** @@ -3441,7 +3461,17 @@ export class ToolHandler { * that have no dependents (nothing to warn about), and returns '' when none * qualify so a leaf-only exploration stays clean. */ - private buildBlastRadiusSection(cg: CodeGraph, subgraph: Subgraph): string { + private buildBlastRadiusSection( + cg: CodeGraph, + subgraph: Subgraph, + /** + * Exact targets (a qualified name, a line anchor) lead the list. The search + * roots are whatever FTS ranked first for the bare name, so without this a + * query for `SQLCompiler.as_sql` headlined `SQLInsertCompiler.as_sql` — and + * the agent took that as the tool having found the wrong method. + */ + leadingIds: Iterable = [], + ): string { const ROOT_CAP = 5; // only the symbols the query actually targeted const FILE_CAP = 4; // caller files listed per symbol before "+N more" const MEANINGFUL = new Set([ @@ -3450,7 +3480,7 @@ export class ToolHandler { ]); const rel = (p: string) => p.replace(/\\/g, '/'); - const roots = subgraph.roots + const roots = [...new Set([...leadingIds, ...subgraph.roots])] .map((id) => subgraph.nodes.get(id)) .filter((n): n is Node => !!n && MEANINGFUL.has(n.kind)) .slice(0, ROOT_CAP); @@ -3656,6 +3686,7 @@ export class ToolHandler { // `+page.svelte`), starving the very files the agent asked for. let pinnedFiles: string[] = []; let unresolvedPathSpans: string[] = []; + let lineAnchors: QueryLineAnchor[] = []; let matchQuery = query; if (queryMightContainPaths(rawQuery)) { try { @@ -3667,6 +3698,7 @@ export class ToolHandler { if (extraction.pinnedFiles.length > 0 || extraction.unresolvedPathSpans.length > 0) { pinnedFiles = extraction.pinnedFiles; unresolvedPathSpans = extraction.unresolvedPathSpans; + lineAnchors = extraction.lineAnchors; matchQuery = normalizeQuerySpelling(extraction.strippedQuery); } } catch { /* path pinning must never fail an explore call */ } @@ -3752,6 +3784,42 @@ export class ToolHandler { .forEach((n) => { if (!subgraph.nodes.has(n.id)) subgraph.nodes.set(n.id, n); }); } + // EXACT targets: callables the agent pointed at with no ambiguity left — a + // qualified name that resolves to at most a handful of defs + // (`SQLCompiler.as_sql`, filled in by the seeding loop below) or the + // callable enclosing a line anchor (`compiler.py:776`). Every render path + // puts these first and never reduces one to a signature line: the agent + // has already said which of the 110 `as_sql`s it means, and an answer that + // spends the file on the named method's neighbours instead is the one it + // Reads the file to get past. + const exactNodeIds = new Set(); + // A multi-line anchor (`lines 900-1003`, `foo.ts:12-40`) names a SPAN, not + // a symbol — often the tail of a body a previous response windowed — and + // is rendered as exactly that span. A single-line anchor with no enclosing + // callable (a module-level line) becomes a small span around the line. + const anchorSpans = new Map(); + for (const anchor of lineAnchors) { + let fileNodes: Node[] = []; + try { fileNodes = cg.getNodesInFile(anchor.file); } catch { continue; } + if (anchor.start === anchor.end) { + const enclosing = fileNodes + .filter((n) => ANCHOR_CALLABLE_KINDS.has(n.kind) + && n.startLine <= anchor.start && n.endLine >= anchor.start) + .sort((a, b) => (a.endLine - a.startLine) - (b.endLine - b.startLine))[0]; + if (enclosing) { + exactNodeIds.add(enclosing.id); + if (!subgraph.nodes.has(enclosing.id)) subgraph.nodes.set(enclosing.id, enclosing); + continue; + } + } + const span = anchor.start === anchor.end + ? { start: Math.max(1, anchor.start - ANCHOR_LINE_CONTEXT), end: anchor.start + ANCHOR_LINE_CONTEXT } + : { start: anchor.start, end: anchor.end }; + const spans = anchorSpans.get(anchor.file) ?? []; + spans.push(span); + anchorSpans.set(anchor.file, spans); + } + if (subgraph.nodes.size === 0) { diag?.finishEmpty('no relevant code found — empty subgraph'); const missNote = unresolvedPathSpans.length > 0 @@ -4040,6 +4108,11 @@ export class ToolHandler { // named-file sort below. (Previously only NEW injections were marked, // so a named symbol FTS already gathered never sorted to the top.) namedSeedIds.add(n.id); + // A QUALIFIED name that resolved to a handful of defs is the agent + // choosing one overload out of a family — `SQLCompiler.as_sql`, not + // the 109 other `as_sql`s. The bare name keeps the family rules; the + // qualified one is an exact target (see `exactNodeIds`). + if (isQual && cands.length <= 3 && CALLABLE.has(n.kind)) exactNodeIds.add(n.id); } // An interface's `method_signature` seeds (so RWR and the flow ranking // still see it, and a query that names it still reaches its file) but @@ -4578,7 +4651,7 @@ export class ToolHandler { // Blast radius (always-on, compact): for the entry symbols, who depends on // them + which tests cover them — locations only, no source — so the agent // knows what to update/verify before editing without a separate call. - const blastRadius = this.buildBlastRadiusSection(cg, subgraph); + const blastRadius = this.buildBlastRadiusSection(cg, subgraph, exactNodeIds); if (blastRadius) lines.push(blastRadius); // Relationship map — show how symbols connect @@ -4625,6 +4698,33 @@ export class ToolHandler { await warmBranchGuardGrammars(); const flow = this.buildFlowFromNamedSymbols(cg, matchQuery); + // The symbols the question is about: exact targets, the named ones, the spine. + const questionIds = new Set([...exactNodeIds, ...flow.pathNodeIds, ...flow.namedNodeIds]); + /** + * The lines a WINDOW of an oversize body must reach: an anchored line + * inside it, the spine's next-hop call, and every call it makes into + * another symbol the question is about — "the parts the question is + * about", in the graph's own terms rather than by matching words. Most + * important first, capped so a window stays a window. + */ + const bodyFocusLines = (n: Node): number[] => { + const focus: number[] = []; + for (const a of lineAnchors) { + if (a.file === n.filePath && a.start > n.startLine && a.start <= n.endLine) focus.push(a.start); + } + const call = flow.spineCallSites.get(n.id); + if (call) focus.push(call); + const calls: number[] = []; + try { + for (const e of cg.getOutgoingEdges(n.id)) { + if (e.target === n.id || !questionIds.has(e.target) || !e.line) continue; + if (e.line > n.startLine && e.line <= n.endLine) calls.push(e.line); + } + } catch { /* focus is a nicety; the head window still renders */ } + focus.push(...calls.sort((a, b) => a - b)); + return [...new Set(focus)].slice(0, MAX_BODY_FOCUS_LINES); + }; + // Snapshot every ranked candidate's scoring inputs, in final sort order, so // the diagnostic can show what each file's share of the envelope was BOUGHT // with (score, graph mass, term hits, flags) — not just what it cost. @@ -4668,7 +4768,11 @@ export class ToolHandler { // A pinned file's bytes are worth full price by definition — the agent // asked for the file itself, generated/test or not. worth: pinnedSet.has(fp) ? 1 : rankPenalty(fp), - spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id)), + // A file holding an EXACT target is funded like a spine file: it is the + // answer to the question as asked, and clipping it is what sends the + // agent to Read — the exact reason the spine gets this treatment. + spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id) || exactNodeIds.has(n.id)) + || anchorSpans.has(fp), pinned: pinnedSet.has(fp), })), budget, @@ -5146,6 +5250,171 @@ export class ToolHandler { } }; + // Window helpers — shared by the focused per-symbol view and the cluster + // path below, so an oversize body is cut by ONE set of rules wherever it + // renders. Parts are SPAN-KEYED: every part carries the exact line range + // its text was sliced from (see `buildSection`). + const SPINE_WINDOW = 28; // lines each side of a focus line (the next-hop call site) + type SectionPart = { range: ExploreLineRange; text: string }; + /** + * Bounded overshoot for one cluster's render (CG-30). + * + * `shrinkCluster` keeps the highest-importance member whole even when that + * member alone is oversize — an empty file section sends the agent to Read, + * which is exactly what explore exists to prevent. But "never empty" is not + * "any size": with nothing bounding it, one 22K member rendered against a + * 9K reservation (2.4x), which collapses the headroom every file ranked + * below it draws from. Past the ceiling the member is WINDOWED rather than + * dropped — a leading window (signature + head of the body), plus a window + * on the spine's call site when the head misses it, since on a flow cluster + * the call path IS the answer. + */ + const MIN_WINDOW_LINES = 12; + /** Rendered cost of one source line, line numbering included. */ + const lineCost = (ln: number): number => + (fileLines[ln - 1] ?? '').length + 1 + (withLineNumbers ? String(ln).length + 1 : 0); + /** `lineCostPrefix[n]` = rendered chars of lines 1..n, newlines included. */ + let lineCostPrefix: number[] | null = null; + /** Rendered chars of lines `start..end` (line numbers included), in O(1). */ + const renderedSpanCost = (start: number, end: number): number => { + if (!lineCostPrefix) { + lineCostPrefix = [0]; + for (let ln = 1; ln <= fileLines.length; ln++) lineCostPrefix.push(lineCostPrefix[ln - 1]! + lineCost(ln)); + } + const s = Math.max(1, start); + const e = Math.min(fileLines.length, end); + return e < s ? 0 : Math.max(0, lineCostPrefix[e]! - lineCostPrefix[s - 1]! - 1); + }; + /** + * Longest prefix of `r` that fits `room`. `minLines` is the never-empty + * floor — it may overrun `room`, so it is only ever asked for when nothing + * else has been emitted and the alternative is an empty section. + */ + const headWindowOf = ( + r: ExploreLineRange, room: number, minLines = 0, + ): ExploreLineRange | null => { + let end = r.start - 1; + let chars = 0; + for (let ln = r.start; ln <= r.end; ln++) { + const cost = lineCost(ln); + if (chars + cost > room && end - r.start + 1 >= minLines) break; + chars += cost; + end = ln; + } + return end >= r.start ? { start: r.start, end } : null; + }; + /** Widest window around `line` inside [lo, hi] that fits `room`. */ + const centeredWindowOf = ( + line: number, lo: number, hi: number, room: number, + ): ExploreLineRange | null => { + if (line < lo || line > hi) return null; + let start = line, end = line, chars = lineCost(line); + for (let grown = true; grown;) { + grown = false; + if (end + 1 <= hi && chars + lineCost(end + 1) <= room) { end += 1; chars += lineCost(end); grown = true; } + if (start - 1 >= lo && chars + lineCost(start - 1) <= room) { start -= 1; chars += lineCost(start); grown = true; } + } + return { start, end }; + }; + /** + * Reduce rendered parts to fit `ceiling`, never to nothing. Whole parts are + * kept while they fit; the first part that overruns is cut to a leading + * window on whole lines (a body is never cut mid-line), and everything past + * it is dropped. The GAP_MARKER between surviving parts — and the line-number + * jump — is what tells the agent the cut happened. + * + * A partial window shorter than MIN_WINDOW_LINES is not worth emitting, and + * emitting one is actively harmful: the session record then claims a 4-line + * sliver, and the NEXT call's dedup has to either shred a whole block around + * it or re-send it. Below that floor the part is simply dropped — unless + * nothing has been emitted at all, where the floor wins over the ceiling + * because an empty section is the one outcome worse than an oversize one. + * + * `focusLines` are the lines this trim must not lose: the spine's next-hop + * call site (CG-30) and every definition the agent NAMED inside the cluster + * (CG-38). The head fill is source-ordered, so a named def in the TAIL of a + * large file is otherwise always the first thing an over-ceiling render + * drops — the one span the agent asked for by name, cut in favour of + * head-of-file filler it did not ask for. The full-ceiling fill is tried + * FIRST and the 60% hold-back applies only when a focus line is actually + * left uncovered, so a cluster whose head already reaches its focus keeps + * the whole ceiling for source. + */ + const windowToCeiling = ( + parts: ReadonlyArray, + ceiling: number, + focusLines: ReadonlyArray = [], + ): SectionPart[] => { + const inParts = (line: number) => + parts.some((p) => line >= p.range.start && line <= p.range.end); + const focus = [...new Set(focusLines)] + .filter((l) => typeof l === 'number' && l > 0 && inParts(l)) + .sort((a, b) => a - b); + /** Source-ordered fill of whole parts, the overrunning one cut to a head window. */ + const fill = (room: number): { emit: ExploreLineRange[]; used: number } => { + const emit: ExploreLineRange[] = []; + let used = 0; + for (const p of parts) { + const join = emit.length > 0 ? GAP_MARKER.length : 0; + if (used + join + p.text.length <= room) { + emit.push(p.range); + used += join + p.text.length; + continue; + } + const first = emit.length === 0; + const win = headWindowOf( + p.range, Math.max(0, room - used - join), first ? MIN_WINDOW_LINES : 0); + if (win && (first || win.end - win.start + 1 >= MIN_WINDOW_LINES)) { + emit.push(win); + used += join + renderSpan(win).length; + } + break; + } + return { emit, used }; + }; + let { emit, used } = fill(ceiling); + const reached = () => (emit.length ? emit[emit.length - 1]!.end : 0); + if (focus.some((l) => l > reached())) { + // Hold room back for the focus windows so the head can't eat all of it. + ({ emit, used } = fill(Math.floor(ceiling * 0.6))); + } + // What is left is SPLIT between the uncovered focus lines rather than + // handed to them in order. Greedy-in-source-order reproduces the very bug + // this guards: on a prose query resolving four focus lines, the two + // earliest took the whole reserve and `flushQueuedMessages` at L1102 — + // named in the question — was dropped again. A skipped or undersized + // window returns its share to the pool for the ones after it. + let covered = reached(); + let room = Math.max(0, ceiling - used); + const pending = focus.filter((l) => l > covered); + for (let i = 0; i < pending.length; i++) { + const line = pending[i]!; + if (line <= covered) continue; // an earlier window already reached it + const share = Math.floor(room / (pending.length - i)) - GAP_MARKER.length; + if (share <= 0) continue; + const host = parts.find((p) => line >= p.range.start && line <= p.range.end)!; + const lo = Math.max(host.range.start, line - SPINE_WINDOW, covered + 1); + const hi = Math.min(host.range.end, line + SPINE_WINDOW); + const win = centeredWindowOf(line, lo, hi, share); + // Same sliver floor as the head window — a two-line peek at the call + // site teaches the next call's dedup to shred the block around it. + if (!win || win.end - win.start + 1 < MIN_WINDOW_LINES) continue; + emit.push(win); + const cost = GAP_MARKER.length + renderSpan(win).length; + used += cost; + room -= cost; + covered = win.end; + } + // Never empty: a section with no source sends the agent to Read. + if (emit.length === 0 && parts.length > 0) { + const first = headWindowOf(parts[0]!.range, ceiling, MIN_WINDOW_LINES); + if (first) emit.push(first); + } + return emit + .sort((a, b) => a.start - b.start) + .map((r) => ({ range: r, text: renderSpan(r) })); + }; + // Disk-drift gate (#1474): every render branch below except whole-file // slices fileContent (CURRENT bytes) at INDEXED line ranges. Content is // already in hand, so the check costs one stat (hash only on mismatch). @@ -5189,7 +5458,11 @@ export class ToolHandler { const onSpineGodFile = hasSpineNode && namedBodyChars > allowance && group.nodes.some(n => CALLABLE_BODY.has(n.kind) && flow.uniqueNamedNodeIds.has(n.id) && !flow.pathNodeIds.has(n.id)); + // A line RANGE the agent asked for (`lines 900-1003`) is not a symbol, so + // the per-symbol view has no way to show it — such a file takes the + // cluster path, which renders anchor spans first. if (!fileStale && adaptiveExploreEnabled() && flow.pathNodeIds.size > 0 + && !anchorSpans.has(filePath) && (onSpineGodFile || (!hasSpineNode && isPolymorphicSibling(group.nodes) && !spared))) { const syms = group.nodes .filter(n => n.kind !== 'import' && n.kind !== 'export' && n.startLine > 0) @@ -5197,33 +5470,92 @@ export class ToolHandler { // Pass 1: choose which symbols get a FULL body, by priority, greedily within // a per-file body cap — so one huge family file can't body every named method // and crowd out the other flow files (Django's query.py). A symbol earns a - // body if it's on-spine, or UNIQUELY named (`SQLCompiler.execute_sql`), or a - // co-named method WHEN this file DEFINES the family supertype (so the base - // `SQLCompiler.as_sql` body shows, but the 110 leaf `as_sql` overrides — and - // OkHttp's 5 `intercept`s if the agent names `intercept` — stay signatures). + // body if it's EXACT, on-spine, UNIQUELY named (`SQLCompiler.execute_sql`), + // or a co-named method WHEN this file DEFINES the family supertype (so the + // base `SQLCompiler.as_sql` body shows, but the 110 leaf `as_sql` overrides — + // and OkHttp's 5 `intercept`s if the agent names `intercept` — stay + // signatures). Tiers, most-wanted first: + // 0 EXACT — a qualified name or a line anchor (see `exactNodeIds`), + // 1 a spine step the agent NAMED, + // 2 a spine step it did not (the one unnamed bridge between two named), + // 3 a UNIQUELY named off-spine method, + // 4 a co-named method in the family's base file. + // Within a tier the spine keeps its CALL order, then source order. Source + // order alone let the smaller steps that happen to sit higher in the file + // take the cap: Django's `SQLCompiler.as_sql` — named, qualified, step 1 of + // the flow, 226 lines — lost to the unnamed bridge `get_qualify_sql` and to + // `get_select` and came back as its signature line, every time. + const chainPos = new Map([...flow.pathNodeIds].map((id, i) => [id, i])); const prio = (n: Node) => !CALLABLE_BODY.has(n.kind) ? 99 - : flow.pathNodeIds.has(n.id) ? 0 - : flow.uniqueNamedNodeIds.has(n.id) ? 1 - : (fileDefinesSuper && flow.namedNodeIds.has(n.id)) ? 2 : 99; + : exactNodeIds.has(n.id) ? 0 + : flow.pathNodeIds.has(n.id) ? (flow.namedNodeIds.has(n.id) ? 1 : 2) + : flow.uniqueNamedNodeIds.has(n.id) ? 3 + : (fileDefinesSuper && flow.namedNodeIds.has(n.id)) ? 4 : 99; + // Tiers 0-1 are never reduced to a signature: a body too big for what is + // left is WINDOWED instead — its head plus the lines where it calls the + // other symbols the question named. A signature is the one render of a + // method the agent explicitly asked for that is guaranteed to send it to + // Read. + const WINDOWED_TIER = 1; // One WINDOW per file, sized by this file's RESERVATION. syms are taken by - // priority (spine first, then uniquely-named, then family-base), and the cap - // applies to ALL of them — including the spine — so a big-spine god-file - // (tokio's worker.rs: run→run_task→next_task→steal_work) can't eat the whole - // response and starve the co-flow file (harness.rs's poll). The native agent - // windows such a file too (~190 lines at a time), so this mimics, not - // truncates. Always emit ≥1 (never an empty section). + // priority, and the cap applies to ALL of them — including the spine — so a + // big-spine god-file (tokio's worker.rs: run→run_task→next_task→steal_work) + // can't eat the whole response and starve the co-flow file (harness.rs's + // poll). The native agent windows such a file too (~190 lines at a time), + // so this mimics, not truncates. Always emit ≥1 (never an empty section). // // Held to `fundedHeadroom` as well (CG-31) so this path cannot spend a // reservation still owed below it either. It never exceeds `allowance` // today, so the bound only bites once the ceiling is genuinely tight — // but "every render path" has to mean every one, or the guard is just a // detour the next god-file takes. - const bodyCap = Math.min(allowance, fundedHeadroom); + // + // Priced in RENDERED chars — line numbers included — and net of the + // signature lines the view also prints. Raw source length undercounts + // both, and once an oversize body is windowed to the room that is left + // the undercount stops being slack: the section ran 19% past what it + // was funded (excalidraw's App.tsx) and the four files below it + // rendered nothing. + const SIG_MAX = Math.max(12, budget.maxSymbolsInFileHeader * 2); + const sigReserve = syms.slice(0, SIG_MAX).reduce((sum, n) => sum + lineCost(n.startLine), 0); + const bodyCap = Math.max(0, Math.min(allowance, fundedHeadroom) - sigReserve); const bodyIds = new Set(); + /** Tier 0-1 bodies too big to fit whole, as the windows they render as. */ + const bodyWindows = new Map(); let bodyChars = 0; - for (const n of syms.filter(n => prio(n) < 99 && n.endLine >= n.startLine).sort((a, b) => prio(a) - prio(b))) { - const sz = fileLines.slice(n.startLine - 1, n.endLine).join('\n').length; - if (bodyChars + sz > bodyCap && bodyIds.size > 0) continue; + const ranked = syms + .filter(n => prio(n) < 99 && n.endLine >= n.startLine) + .sort((a, b) => prio(a) - prio(b) + || (chainPos.get(a.id) ?? Number.MAX_SAFE_INTEGER) - (chainPos.get(b.id) ?? Number.MAX_SAFE_INTEGER) + || a.startLine - b.startLine); + const deferred: Node[] = []; + for (const n of ranked) { + if (prio(n) > WINDOWED_TIER) continue; + const sz = renderedSpanCost(n.startLine, n.endLine); + if (bodyChars + sz <= bodyCap) { + bodyIds.add(n.id); + bodyChars += sz; + } else { + deferred.push(n); + } + } + for (const n of deferred) { + const room = bodyCap - bodyChars; + // Below MIN_CHARS a window cannot hold a readable block; with nothing + // emitted yet the never-empty floor inside `windowToCeiling` wins. + if (room < EXPLORE_ALLOCATION.MIN_CHARS && bodyIds.size + bodyWindows.size > 0) continue; + const range = { start: n.startLine, end: n.endLine }; + const windows = windowToCeiling( + [{ range, text: renderSpan(range) }], Math.max(0, room), bodyFocusLines(n), + ).map((p) => p.range); + if (windows.length === 0) continue; + bodyWindows.set(n.id, windows); + bodyChars += windows.reduce((sum, w) => sum + renderedSpanCost(w.start, w.end), 0); + } + for (const n of ranked) { + if (prio(n) <= WINDOWED_TIER) continue; + const sz = renderedSpanCost(n.startLine, n.endLine); + if (bodyChars + sz > bodyCap && bodyIds.size + bodyWindows.size > 0) continue; bodyIds.add(n.id); bodyChars += sz; } @@ -5233,7 +5565,6 @@ export class ToolHandler { const skel: Array<{ range: ExploreLineRange; text: string }> = []; let coveredUntil = 0; // skip symbols already inside an emitted body let sigCount = 0, sigDropped = 0; - const SIG_MAX = Math.max(12, budget.maxSymbolsInFileHeader * 2); for (const n of syms) { if (n.startLine <= coveredUntil) continue; if (bodyIds.has(n.id)) { @@ -5244,6 +5575,19 @@ export class ToolHandler { text: withLineNumbers ? numberSourceLines(body, n.startLine) : body, }); coveredUntil = end; + } else if (bodyWindows.has(n.id)) { + // Each hole in a windowed body is named with the explore query that + // returns it — a line range, which explore now honours — so the + // follow-up is one call, not a Read of the file. + const windows = bodyWindows.get(n.id)!; + windows.forEach((w, k) => { + const holeEnd = (windows[k + 1]?.start ?? n.endLine + 1) - 1; + const hole = holeEnd > w.end + ? `\n… lines ${w.end + 1}-${holeEnd} of \`${n.name}\` elided — codegraph_explore \`${filePath}:${w.end + 1}-${holeEnd}\` returns them` + : ''; + skel.push({ range: w, text: renderSpan(w) + hole }); + }); + coveredUntil = n.endLine; } else { // Elide the body, emit the signature. node.startLine can point at a // decorator/annotation, so scan forward for the line that names the symbol. @@ -5273,7 +5617,7 @@ export class ToolHandler { // (Session.swift, DataRequest.swift) that fired an over-investigation // spiral (the agent Read the skeletonized file, then kept digging). // CLAUDE.md: explore output must never tell the agent to Read. - const tag = bodyIds.size > 0 + const tag = bodyIds.size + bodyWindows.size > 0 ? 'focused (the methods you named in full, the rest as signatures — codegraph_explore a signature by name for its body; do NOT Read)' : 'skeleton (signatures only — codegraph_explore a name for its full body; do NOT Read)'; // Dedup runs on the per-symbol parts, so a body the agent already has @@ -5289,7 +5633,7 @@ export class ToolHandler { ranges: dd.parts.map((p) => p.range), covered: dd.covered, overhead: 120, - mode: bodyIds.size > 0 ? 'focused' : 'skeleton', + mode: bodyIds.size + bodyWindows.size > 0 ? 'focused' : 'skeleton', // Always "clipped": the per-symbol view elides bodies by construction. clipped: true, fullBody: withTail(skel), @@ -5485,17 +5829,24 @@ export class ToolHandler { // budget — the agent explicitly asked for these symbols. const rangeNodes = new Map(); for (const n of group.nodes) if (n.startLine > 0 && n.endLine > 0) rangeNodes.set(n.id, n); - for (const id of flow.namedNodeIds) { + for (const id of [...flow.namedNodeIds, ...exactNodeIds]) { if (rangeNodes.has(id)) continue; const n = cg.getNode(id); if (n && n.filePath === filePath && n.startLine > 0 && n.endLine > 0) rangeNodes.set(id, n); } + // EXACT targets outrank even the query's entry points: an entry is what the + // search matched, an exact target is the def the agent singled out. Their + // clusters rank first and get the spine's bounded overshoot (below), so a + // qualified `SQLCompiler.as_sql` is no longer outranked by a denser cluster + // around `SQLInsertCompiler.as_sql` that merely matched the bare name. + const EXACT_IMPORTANCE = 11; const ranges: Array<{ start: number; end: number; name: string; kind: string; importance: number; spine: boolean; spineCallLine?: number }> = [...rangeNodes.values()] // Drop whole-file envelope nodes (containers covering >50% of the file). .filter(n => !(ENVELOPE_KINDS.has(n.kind) && (n.endLine - n.startLine + 1) > fileLines.length * 0.5)) .map(n => { let importance = 1; - if (entryNodeIds.has(n.id)) importance = 10; + if (exactNodeIds.has(n.id)) importance = EXACT_IMPORTANCE; + else if (entryNodeIds.has(n.id)) importance = 10; else if (flow.namedNodeIds.has(n.id)) importance = 9; // agent named it → keep its cluster else if (glueNodeIds.has(n.id)) importance = 6; // bridging caller/callee of an entry else if (connectedToEntry.has(n.id)) importance = 3; @@ -5526,6 +5877,15 @@ export class ToolHandler { } } + // Line spans the agent anchored in this file (`lines 900-1003`) are exact + // targets too — rendered as exactly the span asked for, not as the + // symbols around it. + for (const span of anchorSpans.get(filePath) ?? []) { + if (span.start > fileLines.length) continue; + const end = Math.min(span.end, fileLines.length); + ranges.push({ start: span.start, end, name: `lines ${span.start}-${end}`, kind: 'range', importance: EXACT_IMPORTANCE, spine: false }); + } + ranges.sort((a, b) => a.start - b.start); if (ranges.length === 0) { @@ -5595,7 +5955,6 @@ export class ToolHandler { // the exact gap this closes. Bounded, so a god-method can't blow the budget yet // the spine's call still appears in context. const OVERSIZE_SPINE_LINES = 200; - const SPINE_WINDOW = 28; // lines each side of the next-hop call site // Returns the rendered text as SPAN-KEYED PARTS. Every part carries the // exact line range its text was sliced from, which two things depend on: // the session record (CG-17) — a record claiming lines it never sent would @@ -5604,7 +5963,6 @@ export class ToolHandler { // agent already holds the rest. Both read the spans from the function that // does the slicing; a second function mirroring these window/padding rules // would drift. - type SectionPart = { range: ExploreLineRange; text: string }; // Every fit decision below measures parts joined by BARE gap markers. // Naming what a gap skipped (#1711) is added once, at assembly, from what // the file's budget has left: measured with the names, a shrunk cluster @@ -5614,9 +5972,13 @@ export class ToolHandler { const sectionText = (parts: ReadonlyArray): string => joinPartsWithNamedGaps(filePath, parts, fileIndexNodes, 0); const buildSection = ( - c: { start: number; end: number; hasSpine?: boolean; spineCallLine?: number }, + c: { start: number; end: number; hasSpine?: boolean; spineCallLine?: number; maxImportance?: number }, ): SectionPart[] => { - if (c.hasSpine && c.spineCallLine && (c.end - c.start + 1) > OVERSIZE_SPINE_LINES) { + // Not for a cluster holding an exact target: its body is what was asked + // for, so it is kept whole up to the ceiling and only then windowed — + // by `windowToCeiling`, which keeps the call site AND the named defs. + if (c.hasSpine && c.spineCallLine && (c.end - c.start + 1) > OVERSIZE_SPINE_LINES + && (c.maxImportance ?? 0) < EXACT_IMPORTANCE) { const call = c.spineCallLine; const winStart = Math.max(c.start, call - SPINE_WINDOW); const winEnd = Math.min(c.end, call + SPINE_WINDOW); @@ -5648,9 +6010,6 @@ export class ToolHandler { }]; }; - /** Rendered cost of one source line, line numbering included. */ - const lineCost = (ln: number): number => - (fileLines[ln - 1] ?? '').length + 1 + (withLineNumbers ? String(ln).length + 1 : 0); /** Rendered cost of one member on its own, context padding included. */ const paddedCost = (r: ExploreRange): number => { let chars = 0; @@ -5665,19 +6024,23 @@ export class ToolHandler { * is incidental context that happens to sit within `gapThreshold` of it. */ const isProtectedMember = (r: ExploreRange): boolean => r.importance >= 9 || r.spine; - /** Whole member ranges, re-merged in source order and rendered. */ - const partsOfMembers = (members: ReadonlyArray): SectionPart[] => { - // Re-merge in source order so adjacent survivors read as one block - // rather than a stutter of one-symbol fragments. - const sorted = [...members].sort((a, b) => a.start - b.start); + /** + * Kept ranges re-merged in source order, the way a cluster merges — so + * adjacent survivors read as one block rather than a stutter of + * one-symbol fragments. + */ + const mergeKept = (keep: ReadonlyArray): Array<{ start: number; end: number }> => { const merged: Array<{ start: number; end: number }> = []; - for (const r of sorted) { + for (const r of [...keep].sort((a, b) => a.start - b.start)) { const last = merged[merged.length - 1]; if (last && r.start <= last.end + gapThreshold) last.end = Math.max(last.end, r.end); else merged.push({ start: r.start, end: r.end }); } - return merged.flatMap((m) => buildSection(m)); + return merged; }; + /** Whole member ranges, re-merged in source order and rendered. */ + const partsOfMembers = (members: ReadonlyArray): SectionPart[] => + mergeKept(members).flatMap((m) => buildSection(m)); /** * What this cluster's protected members cost to render on their own, after * the session history — the room a cluster ranked ABOVE it must leave for @@ -5726,6 +6089,14 @@ export class ToolHandler { * numbers) because it is a bound — the ceiling past which the render is * windowed, or bytes owed to a cluster below — not an estimate. */ + /** + * Exact rendered size of `keep` once merged, padded and joined — what + * `sectionText` will measure — via a prefix sum, so pricing a candidate + * member costs O(members), not a re-render. + */ + const renderedSizeOfKept = (keep: readonly ExploreRange[]): number => + mergeKept(keep).reduce((sum, m, i) => + sum + renderedSpanCost(m.start - contextPadding, m.end + contextPadding) + (i > 0 ? GAP_MARKER.length : 0), 0); const shrinkCluster = ( c: ExploreCluster, cap: number, incidentalCap?: number, ): SectionPart[] | null => { @@ -5734,176 +6105,50 @@ export class ToolHandler { const byImportance = [...c.members].sort((a, b) => (guarded ? Number(isProtectedMember(b)) - Number(isProtectedMember(a)) : 0) || b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start); - const sizeOf = (r: ExploreRange) => fileLines.slice(r.start - 1, r.end).join('\n').length; const keep: ExploreRange[] = []; - let kept = 0; - // Rendered size of what is kept so far, once the first incidental member - // is weighed. Every protected member sorts ahead of it, so this is the - // exact cost of the protected set; each incidental member then adds its - // own padded span, an over-estimate that errs toward the cluster below. - let rendered: number | undefined; - for (const r of byImportance) { - const sz = sizeOf(r) + GAP_MARKER.length; - // Always keep the most important range, even if it alone is oversize — - // an empty section sends the agent to Read, which costs far more. How - // far it may overshoot is bounded by the caller's ceiling (CG-30), which - // windows a runaway member instead of dropping it. - if (keep.length > 0 && kept + sz > cap) continue; - if (guarded && keep.length > 0 && !isProtectedMember(r)) { - rendered ??= sectionText(dedupeSpans(partsOfMembers(keep)).parts).length; - const cost = paddedCost(r) + GAP_MARKER.length; - if (rendered + cost > incidentalCap) continue; - rendered += cost; + if (c.maxImportance >= EXACT_IMPORTANCE) { + // An EXACT target prices EVERY member in rendered chars, protected + // ones included. The raw estimate below over-keeps on purpose and + // leaves the ceiling trim to cut — but that trim fills in SOURCE + // order, so any member above the exact body in the file (an entry + // point, a named neighbour) was paid for first and the body the agent + // asked for lost its tail (Django's `SQLCompiler.as_sql` stopped at + // L944 of 1001 behind a 15-line neighbour). Priced exactly, the kept + // set fits and the trim has nothing left to cut unless the exact body + // ALONE overruns the ceiling — where it is windowed on its focus + // lines. Incidental members also fit `incidentalCap`, as below. + for (const r of byImportance) { + const limit = guarded && !isProtectedMember(r) ? Math.min(cap, incidentalCap!) : cap; + if (keep.length > 0 && renderedSizeOfKept([...keep, r]) > limit) continue; + keep.push(r); } - keep.push(r); - kept += sz; - } - if (keep.length === c.members.length) return null; - return partsOfMembers(keep); - }; - - /** - * Bounded overshoot for one cluster's render (CG-30). - * - * `shrinkCluster` keeps the highest-importance member whole even when that - * member alone is oversize — an empty file section sends the agent to Read, - * which is exactly what explore exists to prevent. But "never empty" is not - * "any size": with nothing bounding it, one 22K member rendered against a - * 9K reservation (2.4x), which collapses the headroom every file ranked - * below it draws from. Past the ceiling the member is WINDOWED rather than - * dropped — a leading window (signature + head of the body), plus a window - * on the spine's call site when the head misses it, since on a flow cluster - * the call path IS the answer. - */ - const MIN_WINDOW_LINES = 12; - /** - * Longest prefix of `r` that fits `room`. `minLines` is the never-empty - * floor — it may overrun `room`, so it is only ever asked for when nothing - * else has been emitted and the alternative is an empty section. - */ - const headWindowOf = ( - r: ExploreLineRange, room: number, minLines = 0, - ): ExploreLineRange | null => { - let end = r.start - 1; - let chars = 0; - for (let ln = r.start; ln <= r.end; ln++) { - const cost = lineCost(ln); - if (chars + cost > room && end - r.start + 1 >= minLines) break; - chars += cost; - end = ln; - } - return end >= r.start ? { start: r.start, end } : null; - }; - /** Widest window around `line` inside [lo, hi] that fits `room`. */ - const centeredWindowOf = ( - line: number, lo: number, hi: number, room: number, - ): ExploreLineRange | null => { - if (line < lo || line > hi) return null; - let start = line, end = line, chars = lineCost(line); - for (let grown = true; grown;) { - grown = false; - if (end + 1 <= hi && chars + lineCost(end + 1) <= room) { end += 1; chars += lineCost(end); grown = true; } - if (start - 1 >= lo && chars + lineCost(start - 1) <= room) { start -= 1; chars += lineCost(start); grown = true; } - } - return { start, end }; - }; - /** - * Reduce rendered parts to fit `ceiling`, never to nothing. Whole parts are - * kept while they fit; the first part that overruns is cut to a leading - * window on whole lines (a body is never cut mid-line), and everything past - * it is dropped. The GAP_MARKER between surviving parts — and the line-number - * jump — is what tells the agent the cut happened. - * - * A partial window shorter than MIN_WINDOW_LINES is not worth emitting, and - * emitting one is actively harmful: the session record then claims a 4-line - * sliver, and the NEXT call's dedup has to either shred a whole block around - * it or re-send it. Below that floor the part is simply dropped — unless - * nothing has been emitted at all, where the floor wins over the ceiling - * because an empty section is the one outcome worse than an oversize one. - * - * `focusLines` are the lines this trim must not lose: the spine's next-hop - * call site (CG-30) and every definition the agent NAMED inside the cluster - * (CG-38). The head fill is source-ordered, so a named def in the TAIL of a - * large file is otherwise always the first thing an over-ceiling render - * drops — the one span the agent asked for by name, cut in favour of - * head-of-file filler it did not ask for. The full-ceiling fill is tried - * FIRST and the 60% hold-back applies only when a focus line is actually - * left uncovered, so a cluster whose head already reaches its focus keeps - * the whole ceiling for source. - */ - const windowToCeiling = ( - parts: ReadonlyArray, - ceiling: number, - focusLines: ReadonlyArray = [], - ): SectionPart[] => { - const inParts = (line: number) => - parts.some((p) => line >= p.range.start && line <= p.range.end); - const focus = [...new Set(focusLines)] - .filter((l) => typeof l === 'number' && l > 0 && inParts(l)) - .sort((a, b) => a - b); - /** Source-ordered fill of whole parts, the overrunning one cut to a head window. */ - const fill = (room: number): { emit: ExploreLineRange[]; used: number } => { - const emit: ExploreLineRange[] = []; - let used = 0; - for (const p of parts) { - const join = emit.length > 0 ? GAP_MARKER.length : 0; - if (used + join + p.text.length <= room) { - emit.push(p.range); - used += join + p.text.length; - continue; - } - const first = emit.length === 0; - const win = headWindowOf( - p.range, Math.max(0, room - used - join), first ? MIN_WINDOW_LINES : 0); - if (win && (first || win.end - win.start + 1 >= MIN_WINDOW_LINES)) { - emit.push(win); - used += join + renderSpan(win).length; + } else { + const sizeOf = (r: ExploreRange) => fileLines.slice(r.start - 1, r.end).join('\n').length; + let kept = 0; + // Rendered size of what is kept so far, once the first incidental member + // is weighed. Every protected member sorts ahead of it, so this is the + // exact cost of the protected set; each incidental member then adds its + // own padded span, an over-estimate that errs toward the cluster below. + let rendered: number | undefined; + for (const r of byImportance) { + const sz = sizeOf(r) + GAP_MARKER.length; + // Always keep the most important range, even if it alone is oversize — + // an empty section sends the agent to Read, which costs far more. How + // far it may overshoot is bounded by the caller's ceiling (CG-30), which + // windows a runaway member instead of dropping it. + if (keep.length > 0 && kept + sz > cap) continue; + if (guarded && keep.length > 0 && !isProtectedMember(r)) { + rendered ??= sectionText(dedupeSpans(partsOfMembers(keep)).parts).length; + const cost = paddedCost(r) + GAP_MARKER.length; + if (rendered + cost > incidentalCap!) continue; + rendered += cost; } - break; + keep.push(r); + kept += sz; } - return { emit, used }; - }; - let { emit, used } = fill(ceiling); - const reached = () => (emit.length ? emit[emit.length - 1]!.end : 0); - if (focus.some((l) => l > reached())) { - // Hold room back for the focus windows so the head can't eat all of it. - ({ emit, used } = fill(Math.floor(ceiling * 0.6))); } - // What is left is SPLIT between the uncovered focus lines rather than - // handed to them in order. Greedy-in-source-order reproduces the very bug - // this guards: on a prose query resolving four focus lines, the two - // earliest took the whole reserve and `flushQueuedMessages` at L1102 — - // named in the question — was dropped again. A skipped or undersized - // window returns its share to the pool for the ones after it. - let covered = reached(); - let room = Math.max(0, ceiling - used); - const pending = focus.filter((l) => l > covered); - for (let i = 0; i < pending.length; i++) { - const line = pending[i]!; - if (line <= covered) continue; // an earlier window already reached it - const share = Math.floor(room / (pending.length - i)) - GAP_MARKER.length; - if (share <= 0) continue; - const host = parts.find((p) => line >= p.range.start && line <= p.range.end)!; - const lo = Math.max(host.range.start, line - SPINE_WINDOW, covered + 1); - const hi = Math.min(host.range.end, line + SPINE_WINDOW); - const win = centeredWindowOf(line, lo, hi, share); - // Same sliver floor as the head window — a two-line peek at the call - // site teaches the next call's dedup to shred the block around it. - if (!win || win.end - win.start + 1 < MIN_WINDOW_LINES) continue; - emit.push(win); - const cost = GAP_MARKER.length + renderSpan(win).length; - used += cost; - room -= cost; - covered = win.end; - } - // Never empty: a section with no source sends the agent to Read. - if (emit.length === 0 && parts.length > 0) { - const first = headWindowOf(parts[0]!.range, ceiling, MIN_WINDOW_LINES); - if (first) emit.push(first); - } - return emit - .sort((a, b) => a.start - b.start) - .map((r) => ({ range: r, text: renderSpan(r) })); + if (keep.length === c.members.length) return null; + return partsOfMembers(keep); }; /** @@ -5921,7 +6166,14 @@ export class ToolHandler { .sort((a, b) => b.importance - a.importance || a.start - b.start) .slice(0, MAX_FOCUS_LINES) .map((m) => m.start); - return c.spineCallLine ? [c.spineCallLine, ...named] : named; + // An exact body cut by the ceiling keeps its calls into the question's + // other symbols too, not only its def line — the same focus the + // per-symbol view windows by. + const exactFocus = c.maxImportance < EXACT_IMPORTANCE ? [] : [...exactNodeIds] + .map((id) => subgraph.nodes.get(id)) + .filter((n): n is Node => !!n && n.filePath === filePath && n.startLine >= c.start && n.endLine <= c.end) + .flatMap((n) => bodyFocusLines(n)); + return [...exactFocus, ...(c.spineCallLine ? [c.spineCallLine] : []), ...named]; }; /** @@ -5995,10 +6247,14 @@ export class ToolHandler { const rankedClusters = clusters .map((c, i) => ({ idx: i, span: c.end - c.start + 1, c })) .sort((a, b) => { - // Spine clusters first — the rendered call path IS the flow answer, so it + // Exact-target clusters first — the def (or span) the agent singled out. + // Then spine clusters — the rendered call path IS the flow answer, so it // outranks any denser block of peripheral declarations (a low-density entry // method must not lose the budget to them). Within spine / within non-spine, // the existing importance → density → score → span order holds. + const aExact = a.c.maxImportance >= EXACT_IMPORTANCE; + const bExact = b.c.maxImportance >= EXACT_IMPORTANCE; + if (aExact !== bExact) return (bExact ? 1 : 0) - (aExact ? 1 : 0); if (a.c.hasSpine !== b.c.hasSpine) return (b.c.hasSpine ? 1 : 0) - (a.c.hasSpine ? 1 : 0); if (b.c.maxImportance !== a.c.maxImportance) return b.c.maxImportance - a.c.maxImportance; const densityA = a.c.score / a.span; @@ -6062,6 +6318,9 @@ export class ToolHandler { const spent = projectedChars + (chosenIndices.size > 0 ? GAP_MARKER.length : 0); return Math.min(room, fileBudget - spent) - owedBelow; }; + /** Rendered cost of a cluster's EXACT members on their own. */ + const exactCostOf = (c: ExploreCluster): number => + renderedSizeOfKept(c.members.filter((m) => m.importance >= EXACT_IMPORTANCE)); for (let p = 0; p < rankedClusters.length; p++) { const rc = rankedClusters[p]!; // The top-ranked cluster is always taken — an empty file section sends the @@ -6073,15 +6332,31 @@ export class ToolHandler { // A spine cluster (the rendered call path) is the flow answer — it may run // past the per-file budget up to the spine ceiling; non-spine clusters obey // the normal per-file budget. - const cap = rc.c.hasSpine ? SPINE_CEILING : fileBudget; + const cap = rc.c.hasSpine || rc.c.maxImportance >= EXACT_IMPORTANCE ? SPINE_CEILING : fileBudget; // CG-30: shrinking keeps the top member whole however big it is, so bound // how far that member may overshoot — the same 1.5x-of-reservation bound // SPINE_CEILING already draws, never below `cap` (a cluster that fits its // cap is never windowed). A spine cluster's cap already IS that bound, so // this holds it to it rather than letting the member rule walk past it. const ceiling = Math.max(cap, SPINE_CEILING); + // Several exact targets can land in different clusters of one big file — + // a pinned 13K-line App.tsx merges into a handful of giant clusters, and + // `App.triggerRender` and `App.handleCanvasPointerDown` sat in two. The + // one ranked first spent the whole cap on its own neighbours and the + // other target rendered nothing. So an exact cluster holds back what the + // exact clusters ranked BELOW it still owe — never cutting into its own + // targets — and each target renders, windowed if it must. + const owedExact = rc.c.maxImportance < EXACT_IMPORTANCE ? 0 + : rankedClusters.slice(p + 1).reduce((n, other) => (other.c.maxImportance < EXACT_IMPORTANCE + ? n : n + exactCostOf(other.c) + GAP_MARKER.length), 0); + const holdBack = (room: number): number => (owedExact > 0 + ? Math.min(room, Math.max(exactCostOf(rc.c), room - owedExact)) + : room); if (first) { - const section = renderCluster(rc.c, cap, ceiling, incidentalCapFor(p, cap)); + const firstCap = holdBack(cap); + const section = renderCluster( + rc.c, firstCap, owedExact > 0 ? firstCap : ceiling, incidentalCapFor(p, firstCap), + ); renderedClusters.set(rc.idx, section); anyClusterShrunk = anyClusterShrunk || section.shrunk; chosenIndices.add(rc.idx); @@ -6104,7 +6379,7 @@ export class ToolHandler { // nothing") applied between CLUSTERS. Below `MIN_CHARS` the remainder can't // hold one readable block, so it stays a drop rather than a stutter of // fragments the next call's dedup then has to shred around. - const room = cap - projectedChars - GAP_MARKER.length; + const room = holdBack(cap - projectedChars - GAP_MARKER.length); if (room < EXPLORE_ALLOCATION.MIN_CHARS) continue; const section = renderCluster(rc.c, room, room, incidentalCapFor(p, room)); const text = sectionText(section.parts); diff --git a/src/search/query-paths.ts b/src/search/query-paths.ts index 9855c8b7d3..302c442a3b 100644 --- a/src/search/query-paths.ts +++ b/src/search/query-paths.ts @@ -43,6 +43,23 @@ export interface QueryPathExtraction { * surfaced to the agent so the miss is visible instead of silent. */ unresolvedPathSpans: string[]; + /** + * Line spans the query anchored to a pinned file, 1-based and inclusive: + * `compiler.py:776`, `foo.ts:12-40`, `foo.ts#L88-L120`, or prose next to a + * path (`compiler.py lines 900-1003`, `L900-L1003 of compiler.py`). An agent + * writes these when it wants THOSE lines — typically the body a previous + * response elided — so a pin that drops them answers a different question + * (the file's most relevant clusters) than the one asked. Only a span whose + * path resolved to exactly one file is kept: a line number means nothing + * across two candidate files. + */ + lineAnchors: QueryLineAnchor[]; +} + +export interface QueryLineAnchor { + file: string; + start: number; + end: number; } /** @@ -106,15 +123,48 @@ function buildBasenameStems(indexedPaths: readonly string[]): Map MAX_LINE_NUMBER || end > MAX_LINE_NUMBER) return null; + return start <= end ? { start, end } : { start: end, end: start }; +} + +/** Prose punctuation a line-number token can carry: "lines 900-1003," / "(L88)". */ +function stripLineTokenPunctuation(token: string): string { + return token.replace(/^[('"`[]+/, '').replace(/[)'"`\].,;:!?]+$/, ''); +} + /** * Strip prose punctuation wrapped around a token without eating punctuation * that is PART of the path: quotes/backticks always strip; a trailing `)`/`]` * strips only when the token has no matching opener (so `(protected)` and * `[id]` segments survive, while "…(see src/foo.ts)" loses its parenthesis); * a leading `(`/`[` mirrors that. Trailing sentence punctuation strips last, - * so "src/foo.ts." resolves. + * so "src/foo.ts." resolves. A trailing line reference is split off and + * returned separately — the file is what resolves, the lines are what the + * agent wants from it. */ -function stripWrapping(token: string): string { +function stripWrapping(token: string): { path: string; lines: { start: number; end: number } | null } { let s = token; for (;;) { const first = s[0]; @@ -134,10 +184,12 @@ function stripWrapping(token: string): string { if (last === '}' && !s.includes('{')) { s = s.slice(0, -1); continue; } break; } - // Line references ride along in agent-written paths: `foo.ts:123`, - // `foo.ts:12-40`, `foo.ts#L88`. The file is what gets pinned. - s = s.replace(/(?::\d+(?:-\d+)?|#L\d+(?:-L?\d+)?)$/, ''); - return s; + const ref = LINE_REF_SUFFIX.exec(s); + if (!ref) return { path: s, lines: null }; + return { + path: s.slice(0, ref.index), + lines: ref[1] !== undefined ? lineSpan(ref[1], ref[2]) : lineSpan(ref[3], ref[4]), + }; } /** Normalize a span into the repo-relative shape the files table stores. */ @@ -213,6 +265,7 @@ export function extractQueryPaths( strippedQuery: query, pinnedFiles: [], unresolvedPathSpans: [], + lineAnchors: [], }; if (!query.trim() || indexedPaths.length === 0) return passthrough; @@ -226,12 +279,15 @@ export function extractQueryPaths( const pinned: string[] = []; const pinnedSeen = new Set(); const unresolved: string[] = []; + const anchors: QueryLineAnchor[] = []; + /** Token index → the ONE file it pinned, for binding prose line ranges. */ + const singleFileAt = new Map(); let candidatesExamined = 0; for (let i = 0; i < tokens.length; i++) { if (pinned.length >= maxPins) break; if (candidatesExamined >= MAX_CANDIDATE_SPANS) break; - const stripped = stripWrapping(tokens[i]!); + const { path: stripped, lines } = stripWrapping(tokens[i]!); if (stripped.length < 4) continue; const hasSlash = /[/\\]/.test(stripped); if (!hasSlash && !DOTTED_BASENAME.test(stripped)) continue; @@ -250,6 +306,10 @@ export function extractQueryPaths( pinnedSeen.add(m); pinned.push(m); } + if (matches.length === 1 && pinnedSeen.has(matches[0]!)) { + singleFileAt.set(i, matches[0]!); + if (lines) anchors.push({ file: matches[0]!, ...lines }); + } } else if ( ambiguous || isClearlyPathShaped(normalized) @@ -289,7 +349,7 @@ export function extractQueryPaths( let basenameStems: Map | null = null; for (let i = 0; i < tokens.length && pinned.length < maxPins; i++) { if (consumed.has(i)) continue; - const stripped = stripWrapping(tokens[i]!); + const { path: stripped, lines } = stripWrapping(tokens[i]!); if (stripped.length < 4 || !KEBAB_BASENAME.test(stripped)) continue; basenameStems ??= buildBasenameStems(indexedPaths); const matches = basenameStems.get(stripped.toLowerCase()); @@ -300,12 +360,64 @@ export function extractQueryPaths( pinnedSeen.add(m); pinned.push(m); } + if (matches.length === 1 && pinnedSeen.has(matches[0]!)) { + singleFileAt.set(i, matches[0]!); + if (lines) anchors.push({ file: matches[0]!, ...lines }); + } + } + + // Third pass — prose line ranges (`lines 900-1003`, `line 42`, `L88-L120`, + // `lines 900 to 1003`) bound to the NEAREST single-file path token. These + // are what an agent writes when a previous response elided a body and it + // asks for the rest; left in the query, the numbers match nothing and + // `lines` feeds FTS a word every file contains. A range with no path to + // bind to says nothing about which file, so it is left alone. + if (singleFileAt.size > 0) { + const pathIdx = [...singleFileAt.keys()]; + const nearestFile = (i: number): string => { + let best = pathIdx[0]!; + for (const p of pathIdx) if (Math.abs(p - i) < Math.abs(best - i)) best = p; + return singleFileAt.get(best)!; + }; + for (let i = 0; i < tokens.length; i++) { + if (consumed.has(i)) continue; + const tok = stripLineTokenPunctuation(tokens[i]!); + const afterWord = i > 0 && !consumed.has(i - 1) && LINE_WORD.test(stripLineTokenPunctuation(tokens[i - 1]!)); + const afterPath = singleFileAt.has(i - 1); + const m = LINE_NUMBER_TOKEN.exec(tok); + if (!m) continue; + // A bare number is a line number only in a line context; `L900` is one + // on its own. + if (!m[1] && !afterWord && !afterPath) continue; + let span = lineSpan(m[2], m[3]); + const used = [i]; + // `lines 900 to 1003` + if (span && m[3] === undefined && i + 2 < tokens.length + && RANGE_CONNECTIVE.test(stripLineTokenPunctuation(tokens[i + 1]!))) { + const tail = LINE_NUMBER_TOKEN.exec(stripLineTokenPunctuation(tokens[i + 2]!)); + if (tail && tail[3] === undefined) { + span = lineSpan(m[2], tail[2]); + used.push(i + 1, i + 2); + } + } + if (!span) continue; + anchors.push({ file: nearestFile(i), ...span }); + for (const u of used) consumed.add(u); + if (afterWord) consumed.add(i - 1); + } } if (consumed.size === 0) return passthrough; + const seenAnchor = new Set(); return { strippedQuery: tokens.filter((_, i) => !consumed.has(i)).join(' '), pinnedFiles: pinned, unresolvedPathSpans: unresolved, + lineAnchors: anchors.filter((a) => { + const key = `${a.file}:${a.start}-${a.end}`; + if (seenAnchor.has(key)) return false; + seenAnchor.add(key); + return true; + }), }; } From 7506dbff692ff1dc310011bd7aa50e3d2ae42226 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 06:34:36 +0000 Subject: [PATCH 004/363] test(explore): guard named symbols in a pinned file past the node cap (#2064) A file named by path is pinned: its first 300 symbols by start line enter the gather. Past that cap the file splits into a head cluster of pinned filler and, one cluster down, the named symbols the cap cut off, which used to render nothing (`protocol.ts serializeAck`: 0 of 3 lines; the same query unpinned: all of them). #2062's cross-cluster hold-back fixes this shape without knowing about pinning; its own fixture is an unpinned file. This fixture pins the pinned entry path to it: red on tools.ts before #2062 (2 of 6), green on main. Co-authored-by: Claude Opus 5.5 --- __tests__/explore-pinned-cap-named.test.ts | 252 +++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 __tests__/explore-pinned-cap-named.test.ts diff --git a/__tests__/explore-pinned-cap-named.test.ts b/__tests__/explore-pinned-cap-named.test.ts new file mode 100644 index 0000000000..cc10a26907 --- /dev/null +++ b/__tests__/explore-pinned-cap-named.test.ts @@ -0,0 +1,252 @@ +/** + * A symbol the query names survives a pinned file's node cap. + * + * A file named by path is PINNED: its symbols enter the gather unconditionally, + * the first `PINNED_FILE_NODE_CAP` (300) of them by start line. Under the cap a + * dense file merges into one cluster, and the cluster shrinks by member in + * importance order, so what the query named is kept first. Past the cap the file + * splits in two: the head (one named method merged with ~300 pinned neighbours) + * and, one cluster down, the named symbols the cap cut off. Those were gathered + * and ranked importance 10 — the named-def injection was never the problem — + * but the head cluster ranked first on density and shrank against its OWN + * members only, so pin filler spent the file's budget and the named cluster was + * left under MIN_CHARS and dropped. `protocol.ts receiveOne serializeAck` + * rendered `serializeAck` at 0 of 3 lines; `receiveOne serializeAck`, without + * the path, rendered it whole. + * + * Fixed by the cross-cluster hold-back of #2062 (a cluster's incidental members + * wait until lower-ranked clusters' named members are paid), which reaches this + * shape without knowing about pinning. #2062's own fixture is an unpinned file, + * so this one guards the pinned entry path: the pin is what floods the head + * cluster with filler, and a hold-back re-scoped away from it would reopen the + * bug here and nowhere else. + * + * The fixture is that shape, generated so the sizes that decide it are visible + * here: `receiveOne` → `dispatchRequest` / `MessageCodec.serializeAck` at the + * top, 300 unrelated `trackMetric*` methods, then the `MessageKind` enum and a + * `MessageCodec` whose five serializers are followed by 150 `encodeField*` + * members. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; +import type { ExploreDiagnosticReport } from '../src/mcp/explore-diagnostics'; + +const FILE = 'src/rpc/protocol.ts'; +/** `PINNED_FILE_NODE_CAP` in `handleExplore`. */ +const PIN_CAP = 300; + +function protocolSource(): string { + const L: string[] = [ + 'export class Protocol {', + ' private handlers = new Map unknown>();', + ' private outbox: Uint8Array[] = [];', + '', + ' receiveOne(raw: Uint8Array): void {', + ' const kind = raw[0] as MessageKind;', + ' if (kind === MessageKind.Request) {', + ' this.dispatchRequest(raw);', + ' this.outbox.push(MessageCodec.serializeAck(raw[1] ?? 0));', + ' } else if (kind === MessageKind.Cancel) {', + ' this.outbox.push(MessageCodec.serializeCancel(raw[1] ?? 0));', + ' }', + ' }', + '', + ' private dispatchRequest(raw: Uint8Array): void {', + ' const id = raw[1] ?? 0;', + " const method = String(raw[2] ?? '');", + ' const handler = this.handlers.get(method);', + ' if (!handler) {', + ' this.outbox.push(MessageCodec.serializeReplyErr(id, new Error(`no handler: ${method}`)));', + ' return;', + ' }', + ' const started = Date.now();', + ' let result: unknown;', + ' try {', + ' result = handler(raw.subarray(3));', + ' } catch (err) {', + ' this.outbox.push(MessageCodec.serializeReplyErr(id, err as Error));', + ' return;', + ' }', + ' if (Date.now() - started > 1000) {', + ' this.outbox.push(MessageCodec.serializeAck(id));', + ' }', + ' if (this.outbox.length > 64) {', + ' this.outbox.splice(0, this.outbox.length - 64);', + ' }', + ' this.outbox.push(MessageCodec.serializeReply(id, result ?? null));', + ' }', + ]; + for (let i = 0; i < 300; i++) { + L.push('', ` trackMetric${i}(value: number): number {`, ` return value * ${i + 1} + this.outbox.length;`, ' }'); + } + L.push( + '}', + '', + 'export const enum MessageKind {', + ' Request = 1,', + ' Reply = 2,', + ' ReplyErr = 3,', + ' Cancel = 4,', + ' Ack = 5,', + '}', + '', + 'export class MessageCodec {', + ' static serializeRequest(id: number, method: string): Uint8Array {', + ' const body = new TextEncoder().encode(method);', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.Request;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + '', + ' static serializeAck(id: number): Uint8Array {', + ' return Uint8Array.of(MessageKind.Ack, id & 0xff);', + ' }', + '', + ' static serializeCancel(id: number): Uint8Array {', + ' return Uint8Array.of(MessageKind.Cancel, id & 0xff);', + ' }', + '', + ' static serializeReply(id: number, value: unknown): Uint8Array {', + ' const body = new TextEncoder().encode(JSON.stringify(value ?? null));', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.Reply;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + '', + ' static serializeReplyErr(id: number, err: Error): Uint8Array {', + ' const body = new TextEncoder().encode(err.message);', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.ReplyErr;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + ); + for (let i = 0; i < 150; i++) { + L.push('', ` static encodeField${i}(id: number): Uint8Array {`, ` return Uint8Array.of(${i % 250}, id & 0xff, ${(i * 7) % 250});`, ' }'); + } + L.push('}'); + return L.join('\n') + '\n'; +} + +let dir: string; +let cg: CodeGraph; + +/** One explore call plus the CG-4 diagnostic for it. */ +async function explore(query: string): Promise<{ text: string; report: ExploreDiagnosticReport }> { + const sidecar = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-pin-cap-diag-')), 'r.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + let text = ''; + try { + const res = await new ToolHandler(cg).execute('codegraph_explore', { query }); + text = res.content?.[0]?.text ?? ''; + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + const lines = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + fs.rmSync(path.dirname(sidecar), { recursive: true, force: true }); + return { text, report: JSON.parse(lines[lines.length - 1]!) as ExploreDiagnosticReport }; +} + +/** Every `\t` line number the response sent for `file`. */ +function renderedLines(response: string, file: string): Set { + const out = new Set(); + let current: string | null = null; + let inFence = false; + for (const line of response.split('\n')) { + const header = !inFence ? line.match(/^\*\*`([^`]+)`\*\*/) : null; + if (header) { current = header[1]!; continue; } + if (line.startsWith('```')) { inFence = !inFence; continue; } + const m = inFence && current === file ? line.match(/^(\d+)\t/) : null; + if (m) out.add(Number(m[1])); + } + return out; +} + +/** The pinned file's indexable symbols in the order the pin gathers them. */ +function pinOrder() { + return cg.getNodesInFile(FILE) + .filter((n) => n.kind !== 'file' && n.kind !== 'import' && n.kind !== 'export') + .sort((a, b) => a.startLine - b.startLine); +} + +/** Names whose whole definition did NOT reach the response. */ +function incompleteBodies(response: string, names: string[]): string[] { + const sent = renderedLines(response, FILE); + return names.filter((name) => { + const node = cg.getNodesInFile(FILE).find((n) => n.name === name && n.kind !== 'import'); + expect(node, `${name} is not indexed in ${FILE}`).toBeDefined(); + for (let l = node!.startLine; l <= node!.endLine; l++) if (!sent.has(l)) return true; + return false; + }); +} + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-pin-cap-')); + fs.mkdirSync(path.join(dir, path.dirname(FILE)), { recursive: true }); + fs.writeFileSync(path.join(dir, FILE), protocolSource()); + cg = CodeGraph.initSync(dir); + await cg.indexAll(); +}, 180_000); + +afterAll(() => { + cg?.destroy(); + if (dir && fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('fixture shape — if this rots, the tests below mean nothing', () => { + it('holds more symbols than a pinned file contributes to the gather', () => { + expect(pinOrder().length).toBeGreaterThan(PIN_CAP); + }); + + it('puts the named serializers past the cap and receiveOne inside it', () => { + const order = pinOrder().map((n) => n.name); + expect(order.indexOf('receiveOne')).toBeLessThan(PIN_CAP); + for (const name of ['serializeAck', 'serializeCancel', 'serializeReply']) { + expect(order.indexOf(name), name).toBeGreaterThanOrEqual(PIN_CAP); + } + }); + + it('renders the named symbols whole when the file is NOT pinned', async () => { + // The parity the pinned query is held to below. + const { text } = await explore('receiveOne serializeAck'); + expect(incompleteBodies(text, ['receiveOne', 'serializeAck'])).toEqual([]); + }); +}); + +describe('a pinned file past the node cap', () => { + it('returns a named symbol past the cap, and the named one before it', async () => { + // Pre-fix: 0 of serializeAck's 3 lines — the head cluster's filler took the budget. + const { text, report } = await explore('protocol.ts receiveOne serializeAck'); + expect(report.files.find((f) => f.path === FILE)?.pinned).toBe(true); + expect(incompleteBodies(text, ['receiveOne', 'serializeAck'])).toEqual([]); + }); + + it('returns a lone named symbol past the cap', async () => { + // No flow at all (one token), so no named-def injection either: the symbol + // is gathered as a search entry, and the render alone decided its fate. + const { text } = await explore('protocol.ts serializeAck'); + expect(incompleteBodies(text, ['serializeAck'])).toEqual([]); + }); + + it('keeps the file inside the room it had — the fix redistributes, never raises', async () => { + // Before the fix the head cluster alone fit inside the reservation (8,700 of + // 9,100), so paying the named symbols must come out of that room, not on + // top of it. Letting the file's clusters share the head cluster's CEILING + // instead was measured at ~3K more filler per pinned response. + const { report } = await explore('protocol.ts receiveOne serializeAck'); + const rec = report.files.find((f) => f.path === FILE)!; + expect(rec.render).toBe('clusters'); + expect(rec.emittedChars).toBeLessThanOrEqual(rec.allowance); + }); +}); From 724b5dee0cdc83d5aeaa373858080bdb112f13db Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 07:21:01 +0000 Subject: [PATCH 005/363] fix(explore): return named bodies from merged spine clusters; let named files use spare budget (#2068) * fix(explore): return named bodies from merged spine clusters; let named files use spare budget Re-applied onto main after #2062 and #2063 landed in the same render path (originally e31cc9fc on claude/inspiring-grothendieck-8a7b3e): - the merged-spine hold-back and its top-up are dropped: #2062's hold-back (every cluster leaves room for the protected members ranked below it) subsumes them, and the past-the-pin-cap test passes on it; - #2063's "don't window a cluster holding an exact target" becomes the member rule's "don't window a spine method that is or holds one"; - the shrink orders exact targets ahead of spine members, and the exact branch prices member windows (sectionRangesOf) rather than whole bodies; - the final ceiling fit shortens the file header back to its estimate before it cuts source. With an exact target's overshoot (#2063) spending a named file to its funded line, a lower file's header running past its estimate cost it 109 chars of source (922 of 1,031; 995 on main). The valve test now asserts funding in full and delivery to the line, since #2062's exact rendering stops at whole lines. Co-Authored-By: Claude Opus 5.5 * docs(explore): record the agent A/B for the named-concentration fix Co-Authored-By: Claude Opus 5.5 * docs(explore): record landing named concentration on main after #2062/#2063 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + .../explore-merged-spine-cluster.test.ts | 382 +++++++++++++++++ __tests__/explore-named-file-valve.test.ts | 245 +++++++++++ .../explore-proportional-allocation.test.ts | 69 +++ docs/design/explore-budget-allocation.md | 246 +++++++++++ src/mcp/tools.ts | 392 ++++++++++++++---- 6 files changed, 1258 insertions(+), 78 deletions(-) create mode 100644 __tests__/explore-merged-spine-cluster.test.ts create mode 100644 __tests__/explore-named-file-valve.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bcd635d1a..3d562a5ea6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -212,6 +212,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - The stale-file warning no longer names a pending file just because its path is a prefix or suffix of a path the answer shows (`src/app.ts` vs `src/app.tsx`). Thanks @inth3shadows. (#1968) - When `codegraph_explore` finds nothing, it now says that matching is lexical, lists the query words the index doesn't contain, and suggests real symbol names to retry with. Thanks @iacore. (#1904) - `codegraph_explore` now finds a file when the query is its exact Chinese filename, with or without the extension. Thanks @Syh1906. (#1372) +- `codegraph_explore` now returns the methods and types you name from a large file instead of a few lines around one call. When a file's symbols sat close together, and whenever you named the file by path, the answer could show only its first lines and a window on one call site, leave out the bodies you asked for, and still call the file complete, so your agent read it back. A genuinely long function on the call path is still shortened to its signature and the call that matters, and the answer now says when it did. +- When your question names symbols concentrated in one file and the answer has room to spare, `codegraph_explore` now uses that room for the file you named instead of stopping at a fixed share of the response, so every body you asked for fits. Other files in the answer keep the space they were given. - Java, Kotlin and Scala packages named `build` under a source root are indexed again, while real build-output directories stay excluded. Thanks @CmmVoid for the report and @danusha2345. (#1642) - Files over the 1 MB source limit, such as a package archive an import points at, are now rejected before they are read during resolution, removing a multi-gigabyte memory spike at startup. Thanks @hcg1023 for the report and @danusha2345. (#1553) - `codegraph sync` now reports that the index is busy and exits non-zero when another process holds the write lock, instead of printing "Already up to date". Thanks @inth3shadows. (#1361) diff --git a/__tests__/explore-merged-spine-cluster.test.ts b/__tests__/explore-merged-spine-cluster.test.ts new file mode 100644 index 0000000000..6a944439ad --- /dev/null +++ b/__tests__/explore-merged-spine-cluster.test.ts @@ -0,0 +1,382 @@ +/** + * An oversize SPINE METHOD is windowed; a cluster that is long only because it + * merged is not. + * + * `buildSection` cuts a god-method on the flow path (n8n's 962-line + * `processRunExecutionData`) to its signature head plus a window on its + * next-hop call site. It used to decide that on the CLUSTER's span, and a + * cluster merges every symbol within `gapThreshold` lines of the next — so a + * dense file, and every PINNED file (whose whole symbol list enters the + * gather), became one cluster spanning hundreds of lines around a short spine + * method, and the window threw away everything else in it. Measured on + * microsoft/vscode: `rpcProtocol.ts`, pinned by the query "MessageType enum + * rpcProtocol.ts _receiveOneMessage MessageIO serializeRequest …", merged into + * one cluster over lines 20–968 around a 77-line `_receiveOneMessage`. The + * response carried a 5-line file head and ±28 lines of one call site — 2,910 + * chars of a 16,800 reservation, reported unclipped, and not one of the five + * `MessageIO.serialize*` bodies or the `MessageType` enum the query named. The + * agent Read the file back five times. + * + * The fixtures below are that shape, generated so the sizes that decide it are + * visible here. Each was checked against the pre-fix build: + * + * protocol (pinned, small tier) pre-fix: lines 1–37 only, 0 of 5 serializers + * workflow (a real god-method) pre-fix: `executeNode` absent, its call site + * absent, the file reported unclipped + * protocol past the pin cap render-by-member without the hold-back: the + * five serializers, one cluster down, absent + * protocol, medium tier without the last-resort trim: the file + * skipped whole (0 chars) + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; +import type { ExploreDiagnosticReport } from '../src/mcp/explore-diagnostics'; + +// ── Fixture generators ─────────────────────────────────────────────────────── + +/** + * vscode's `rpcProtocol.ts`, reduced: a `receiveOne` → `dispatchRequest` → + * `MessageCodec.serializeReply` flow at the top, the `MessageKind` enum, then a + * named `MessageCodec` whose five serializers are followed by `fields` small + * members (importance 3 — connected to the named class through `contains`) and + * optionally `helpers` unrelated methods inside `Protocol` itself. + */ +function protocolSource(fields: number, helpers = 0): string { + const L: string[] = [ + 'export class Protocol {', + ' private handlers = new Map unknown>();', + ' private outbox: Uint8Array[] = [];', + '', + ' receiveOne(raw: Uint8Array): void {', + ' const kind = raw[0] as MessageKind;', + ' switch (kind) {', + ' case MessageKind.Request:', + ' this.dispatchRequest(raw);', + ' break;', + ' case MessageKind.Cancel:', + ' this.outbox.push(MessageCodec.serializeCancel(raw[1] ?? 0));', + ' break;', + ' default:', + ' this.outbox.push(MessageCodec.serializeAck(raw[1] ?? 0));', + ' }', + ' }', + '', + ' dispatchRequest(raw: Uint8Array): void {', + ' const id = raw[1] ?? 0;', + " const method = String(raw[2] ?? '');", + ' const handler = this.handlers.get(method);', + ' if (!handler) {', + ' this.outbox.push(MessageCodec.serializeReplyErr(id, new Error(`no handler: ${method}`)));', + ' return;', + ' }', + ' const started = Date.now();', + ' let result: unknown;', + ' try {', + ' result = handler(raw.subarray(3));', + ' } catch (err) {', + ' this.outbox.push(MessageCodec.serializeReplyErr(id, err as Error));', + ' return;', + ' }', + ' const elapsed = Date.now() - started;', + ' if (elapsed > 1000) {', + ' this.outbox.push(MessageCodec.serializeAck(id));', + ' }', + ' if (result === undefined) {', + ' result = null;', + ' }', + ' if (this.outbox.length > 64) {', + ' this.outbox.splice(0, this.outbox.length - 64);', + ' }', + ' this.outbox.push(MessageCodec.serializeReply(id, result));', + ' }', + ]; + for (let i = 0; i < helpers; i++) { + L.push('', ` trackMetric${i}(value: number): number {`, ` return value * ${i + 1} + this.outbox.length;`, ' }'); + } + L.push( + '}', + '', + 'export const enum MessageKind {', + ' Request = 1,', + ' Reply = 2,', + ' ReplyErr = 3,', + ' Cancel = 4,', + ' Ack = 5,', + '}', + '', + 'export class MessageCodec {', + ' static serializeRequest(id: number, method: string): Uint8Array {', + ' const body = new TextEncoder().encode(method);', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.Request;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + '', + ' static serializeAck(id: number): Uint8Array {', + ' return Uint8Array.of(MessageKind.Ack, id & 0xff);', + ' }', + '', + ' static serializeCancel(id: number): Uint8Array {', + ' return Uint8Array.of(MessageKind.Cancel, id & 0xff);', + ' }', + '', + ' static serializeReply(id: number, value: unknown): Uint8Array {', + ' const body = new TextEncoder().encode(JSON.stringify(value ?? null));', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.Reply;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + '', + ' static serializeReplyErr(id: number, err: Error): Uint8Array {', + ' const body = new TextEncoder().encode(err.message);', + ' const out = new Uint8Array(body.length + 2);', + ' out[0] = MessageKind.ReplyErr;', + ' out[1] = id & 0xff;', + ' out.set(body, 2);', + ' return out;', + ' }', + ); + for (let i = 0; i < fields; i++) { + L.push('', ` static encodeField${i}(id: number): Uint8Array {`, ` return Uint8Array.of(${i % 250}, id & 0xff, ${(i * 7) % 250});`, ' }'); + } + L.push('}'); + return L.join('\n') + '\n'; +} + +/** + * A real god-method on the flow: `prepareRun` → `runExecution` (240 lines, its + * call to `executeNode` in the middle) → `executeNode`, adjacent, so all three + * merge into one cluster. + */ +function workflowSource(): string { + const L = ['export class WorkflowRunner {', ' private log: string[] = [];', '', ' prepareRun(input: number[]): number {']; + for (let i = 0; i < 9; i++) L.push(` this.log.push('prepare-${i}:' + input.length);`); + L.push(' return this.runExecution(input, input.length);', ' }', ''); + L.push(' runExecution(input: number[], seed: number): number {', ' let acc = seed;'); + for (let i = 0; i < 117; i++) L.push(` acc = acc * 3 + ${i};`); + L.push(' acc = this.executeNode(acc);'); + for (let i = 0; i < 118; i++) L.push(` acc = acc - ${i};`); + L.push(' return acc;', ' }', '', ' executeNode(value: number): number {'); + for (let i = 0; i < 7; i++) L.push(` value = value ^ ${i + 11};`); + L.push(" this.log.push('node:' + value);", ' return value;', ' }', ''); + for (let i = 0; i < 12; i++) L.push(` auditTrail${i}(): string {`, ` return this.log.slice(${i}).join(',');`, ' }', ''); + L.push('}'); + return L.join('\n') + '\n'; +} + +/** Enough one-line files to put a project in the >=500-file (24K) tier. */ +function tierPadding(count: number): Record { + const out: Record = {}; + for (let i = 0; i < count; i++) out[`src/pad/pad${i}.ts`] = `export const pad${i} = ${i};\n`; + return out; +} + +// ── Harness ───────────────────────────────────────────────────────────────── + +interface Project { dir: string; cg: CodeGraph } + +async function buildProject(files: Record): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-merged-spine-')); + for (const [rel, body] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + } + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + return { dir, cg }; +} + +function destroyProject(p?: Project): void { + if (!p) return; + p.cg.destroy(); + if (fs.existsSync(p.dir)) fs.rmSync(p.dir, { recursive: true, force: true }); +} + +/** One explore call plus the CG-4 diagnostic for it. */ +async function explore(p: Project, query: string) { + const sidecar = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-merged-spine-diag-')), 'r.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + let text = ''; + try { + const res = await new ToolHandler(p.cg).execute('codegraph_explore', { query }); + text = res.content?.[0]?.text ?? ''; + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + const lines = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + fs.rmSync(path.dirname(sidecar), { recursive: true, force: true }); + const report = JSON.parse(lines[lines.length - 1]!) as ExploreDiagnosticReport; + return { text, report }; +} + +/** Every `\t` line number the response sent for `file`. */ +function renderedLines(response: string, file: string): Set { + const out = new Set(); + let current: string | null = null; + let inFence = false; + for (const line of response.split('\n')) { + const header = !inFence ? line.match(/^\*\*`([^`]+)`\*\*/) : null; + if (header) { current = header[1]!; continue; } + if (line.startsWith('```')) { inFence = !inFence; continue; } + const m = inFence && current === file ? line.match(/^(\d+)\t/) : null; + if (m) out.add(Number(m[1])); + } + return out; +} + +/** Names whose whole definition in `file` did NOT reach the response. */ +function incompleteBodies(p: Project, response: string, file: string, names: string[]): string[] { + const sent = renderedLines(response, file); + return names.filter((name) => { + const node = p.cg.getNodesInFile(file).find((n) => n.name === name && n.kind !== 'import'); + expect(node, `${name} is not indexed in ${file}`).toBeDefined(); + for (let l = node!.startLine; l <= node!.endLine; l++) if (!sent.has(l)) return true; + return false; + }); +} + +const PROTOCOL = 'src/rpc/protocol.ts'; +/** The vscode query, transposed: the file by path plus the symbols it asks about. */ +const PROTOCOL_QUERY = + 'protocol.ts MessageKind MessageCodec receiveOne serializeRequest serializeReply serializeReplyErr serializeCancel serializeAck'; +/** Every body the answer needs: the named symbols plus the flow's own bridge. */ +const PROTOCOL_ANSWER = [ + 'receiveOne', 'dispatchRequest', 'MessageKind', + 'serializeRequest', 'serializeAck', 'serializeCancel', 'serializeReply', 'serializeReplyErr', +]; + +// ── The reported shape ────────────────────────────────────────────────────── + +describe('a pinned file whose spine cluster is long only because it merged', () => { + let p: Project; + let text: string; + let report: ExploreDiagnosticReport; + + beforeAll(async () => { + p = await buildProject({ [PROTOCOL]: protocolSource(150) }); + ({ text, report } = await explore(p, PROTOCOL_QUERY)); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('fixture shape: one merged cluster far past the god-method line, around a short spine method', () => { + const lines = fs.readFileSync(path.join(p.dir, PROTOCOL), 'utf-8').split('\n').length; + expect(lines).toBeGreaterThan(280); // past the whole-file window, so it clusters + const spine = p.cg.getNodesInFile(PROTOCOL).find((n) => n.name === 'receiveOne')!; + expect(spine.endLine - spine.startLine + 1).toBeLessThan(200); // not a god-method + expect(text).toContain('Flow (call path among the symbols you queried)'); + expect(report.files.find((f) => f.path === PROTOCOL)?.render).toBe('clusters'); + }); + + it('returns every body the query named, and the flow bridge between them', () => { + // Pre-fix: a 5-line file head plus ±28 lines of `receiveOne`'s call site. + expect(incompleteBodies(p, text, PROTOCOL, PROTOCOL_ANSWER)).toEqual([]); + }); +}); + +describe('a real god-method on the flow is still windowed — as a member, not as its cluster', () => { + const FILE = 'src/engine/workflow.ts'; + let p: Project; + let text: string; + let report: ExploreDiagnosticReport; + + beforeAll(async () => { + p = await buildProject({ [FILE]: workflowSource() }); + ({ text, report } = await explore(p, 'prepareRun runExecution executeNode')); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('renders the named neighbours merged into its cluster in full', () => { + // Pre-fix: the WHOLE cluster was windowed, around `prepareRun`'s call — so + // `executeNode`, below the god-method, never appeared. + expect(incompleteBodies(p, text, FILE, ['prepareRun', 'executeNode'])).toEqual([]); + }); + + it('windows the god-method itself to its signature and its own next-hop call site', () => { + const god = p.cg.getNodesInFile(FILE).find((n) => n.name === 'runExecution')!; + expect(god.endLine - god.startLine + 1).toBeGreaterThan(200); + const sent = renderedLines(text, FILE); + const source = fs.readFileSync(path.join(p.dir, FILE), 'utf-8').split('\n'); + const callLine = source.findIndex((l) => l.includes('this.executeNode(acc)')) + 1; + expect(sent.has(god.startLine), 'signature').toBe(true); + expect(sent.has(callLine), 'the call to executeNode').toBe(true); + let bodySent = 0; + for (let l = god.startLine; l <= god.endLine; l++) if (sent.has(l)) bodySent++; + expect(bodySent).toBeLessThan((god.endLine - god.startLine + 1) / 2); + }); + + it('reports the windowed file as clipped', () => { + // Pre-fix the window elided ~200 lines and the diagnostic called the + // section complete, which is how the rpcProtocol render hid. + expect(report.files.find((f) => f.path === FILE)?.clipped).toBe(true); + }); +}); + +// ── Two consequences of rendering such a cluster by member ───────────────── + +describe('a merged spine cluster leaves room for the file\'s other clusters', () => { + // Past the 300-node pin cap the gather stops, so the file splits: a spine + // cluster of `receiveOne` plus 300 unrelated `trackMetric*` methods, and the + // codec with the named serializers one cluster down. The spine cluster ranks + // first; shrunk by its own members only, it filled the cap with filler. + let p: Project; + let text: string; + + beforeAll(async () => { + p = await buildProject({ [PROTOCOL]: protocolSource(150, 300) }); + ({ text } = await explore(p, PROTOCOL_QUERY)); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('fixture shape: more symbols than a pinned file contributes to the gather', () => { + expect(p.cg.getNodesInFile(PROTOCOL).length).toBeGreaterThan(300); + }); + + it('still returns every named body, and the flow bridge', () => { + expect(incompleteBodies(p, text, PROTOCOL, PROTOCOL_ANSWER)).toEqual([]); + }); +}); + +describe('a shrunk cluster that still overruns is trimmed to fit, never skipped', () => { + // At the 24K tier the cap is large enough to keep hundreds of scattered small + // members, and every gap between them is NAMED in the response (#1711) while + // every budget upstream charges a bare gap marker. The re-render into the + // room left still overran, and the file — every named body inside it — was + // skipped whole. + let p: Project; + let text: string; + let report: ExploreDiagnosticReport; + + beforeAll(async () => { + p = await buildProject({ [PROTOCOL]: protocolSource(300), ...tierPadding(520) }); + ({ text, report } = await explore(p, PROTOCOL_QUERY)); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('fixture shape: the 24K tier', () => { + expect(report.budget.maxOutputChars).toBe(24000); + }); + + it('renders the file rather than skipping it', () => { + const rec = report.files.find((f) => f.path === PROTOCOL); + expect(rec?.skipped ?? null).toBeNull(); + expect(rec?.emittedChars ?? 0).toBeGreaterThan(0); + }); + + it('keeps every named body through the trim', () => { + expect(incompleteBodies(p, text, PROTOCOL, PROTOCOL_ANSWER)).toEqual([]); + }); + + it('stays inside the inline cap', () => { + expect(text.length).toBeLessThanOrEqual(25000); + }); +}); diff --git a/__tests__/explore-named-file-valve.test.ts b/__tests__/explore-named-file-valve.test.ts new file mode 100644 index 0000000000..c56e44a728 --- /dev/null +++ b/__tests__/explore-named-file-valve.test.ts @@ -0,0 +1,245 @@ +/** + * A file the query NAMED may draw on the call's unspent budget past the + * `MAX_SHARE` valve — and only on budget no other file is owed. + * + * The valve caps any one file at 70% of the envelope, as a hedge against a + * mis-ranked dominant file. It never redistributed: a clamped file's excess was + * left unreserved, and the render loop's carry-forward only moves + * reservations, so nothing could spend it. On express, "response.js res.send + * res.json res.render res.redirect res.sendFile res.format" admits one file, + * clamps it at 9,100 with 3,700 of the pool unreserved, and cut the named + * `send` body at 55 of 97 lines in a 10.9K response to a 13K budget. + * + * Two halves, each with a fixture that goes red when that half is removed: + * + * 1. **Far from spent.** One file, five named functions whose bodies together + * overrun the valve but not the budget. Pre-change (and with either the + * allocator exemption or the render-loop extension removed): 3 of 5 bodies + * complete. With both: 5 of 5. + * + * 2. **Saturated.** The 24K tier, a dominant named file and a lower admitted + * file, the response at the inline cap. The extension must come out of + * SPARE budget only. Bounded by `fundedHeadroom` instead, the lower file + * was dropped whole; bounded by the render ceiling instead of the budget, + * it was cut to 74% of its reservation by the named file's render + * overshoot. Both variants were measured, and are what the assertions + * below exclude. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler, EXPLORE_ALLOCATION } from '../src/mcp/tools'; +import type { ExploreDiagnosticReport } from '../src/mcp/explore-diagnostics'; + +interface Project { dir: string; cg: CodeGraph } + +async function buildProject(files: Record): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-named-valve-')); + for (const [rel, body] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), body); + } + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + return { dir, cg }; +} + +function destroyProject(p?: Project): void { + if (!p) return; + p.cg.destroy(); + if (fs.existsSync(p.dir)) fs.rmSync(p.dir, { recursive: true, force: true }); +} + +async function explore(p: Project, query: string) { + const sidecar = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-named-valve-diag-')), 'r.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + let text = ''; + try { + const res = await new ToolHandler(p.cg).execute('codegraph_explore', { query }); + text = res.content?.[0]?.text ?? ''; + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + const lines = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + fs.rmSync(path.dirname(sidecar), { recursive: true, force: true }); + const report = JSON.parse(lines[lines.length - 1]!) as ExploreDiagnosticReport; + return { text, report, file: (fp: string) => report.files.find((f) => f.path === fp) }; +} + +/** Names whose whole definition in `file` did NOT reach the response. */ +function incompleteBodies(p: Project, response: string, file: string, names: string[]): string[] { + const sent = new Set(); + let current: string | null = null; + let inFence = false; + for (const line of response.split('\n')) { + const header = !inFence ? line.match(/^\*\*`([^`]+)`\*\*/) : null; + if (header) { current = header[1]!; continue; } + if (line.startsWith('```')) { inFence = !inFence; continue; } + const m = inFence && current === file ? line.match(/^(\d+)\t/) : null; + if (m) sent.add(Number(m[1])); + } + return names.filter((name) => { + const node = p.cg.getNodesInFile(file).find((n) => n.name === name); + expect(node, `${name} is not indexed in ${file}`).toBeDefined(); + for (let l = node!.startLine; l <= node!.endLine; l++) if (!sent.has(l)) return true; + return false; + }); +} + +// ── 1. Far from spent ─────────────────────────────────────────────────────── + +describe('a named file far from its budget draws past the valve', () => { + const FILE = 'lib/response.ts'; + const NAMED = ['sendBody', 'sendJson', 'renderView', 'redirectTo', 'sendFileStream']; + + /** Five independent named functions, then unrelated header helpers. */ + function responseSource(): string { + const L = ['export interface Res { headers: Record; body: string[]; status: number }', '']; + for (const n of NAMED) { + L.push(`export function ${n}(res: Res, payload: unknown): Res {`); + for (let i = 0; i < 33; i++) L.push(` res.body.push(String(payload ?? '').slice(${i}, ${i + 7}) + '${n}:${i}');`); + L.push(' return res;', '}', ''); + } + for (let i = 0; i < 60; i++) { + L.push(`export function headerSlot${i}(res: Res, value: string): Res {`); + for (let k = 0; k < 6; k++) L.push(` res.headers['x-slot-${i}-${k}'] = value.slice(${k}, ${k + 3});`); + L.push(' return res;', '}', ''); + } + return L.join('\n'); + } + + let p: Project; + let r: Awaited>; + + beforeAll(async () => { + p = await buildProject({ [FILE]: responseSource() }); + r = await explore(p, NAMED.join(' ')); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('fixture shape: the named bodies overrun the valve, not the budget', () => { + const valve = Math.round(r.report.budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE); + const source = fs.readFileSync(path.join(p.dir, FILE), 'utf-8').split('\n'); + const bodies = NAMED.map((name) => p.cg.getNodesInFile(FILE).find((n) => n.name === name)!) + .reduce((sum, n) => sum + source.slice(n.startLine - 1, n.endLine).join('\n').length, 0); + expect(r.report.budget.maxOutputChars).toBe(13000); + expect(bodies).toBeGreaterThan(valve); + expect(bodies).toBeLessThan(r.report.budget.maxOutputChars); + // No flow among them: the spine's own 1.5x overshoot is not what pays here. + expect(r.file(FILE)?.spine).toBe(false); + }); + + it('may spend past the valve', () => { + const valve = Math.round(r.report.budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE); + expect(r.file(FILE)?.spendable ?? 0).toBeGreaterThan(valve); + }); + + it('returns every named body', () => { + expect(incompleteBodies(p, r.text, FILE, NAMED)).toEqual([]); + }); + + it('stays inside the response ceiling', () => { + expect(r.report.envelope.chars).toBeLessThanOrEqual(r.report.budget.hardCeiling); + }); +}); + +// ── 2. Saturated ──────────────────────────────────────────────────────────── + +describe('a named file on a saturated response takes nothing a lower file is owed', () => { + const NAMED_FILE = 'src/store/query.ts'; + const LOWER = 'src/admin/listFilter.ts'; + const METHODS = ['filter', 'exclude', 'annotate', 'orderBy', 'distinct', 'values', 'only', 'defer']; + + /** A QuerySet whose named methods are interleaved with unrelated cache slots. */ + function querySource(): string { + const L = [ + 'export class QuerySet {', + ' steps: Array<{ op: string; at: number; arg: unknown }> = [];', + '', + ' chain(op: string): QuerySet {', + ' const next = new QuerySet();', + ' next.steps = [...this.steps];', + ' return next;', + ' }', + ]; + let slot = 0; + const cacheSlot = () => { + const i = slot++; + L.push('', ` cacheSlot${i}(key: string): string {`); + for (let k = 0; k < 8; k++) L.push(` key = key.padStart(${k + (i % 7)}, '${String.fromCharCode(97 + ((i + k) % 26))}');`); + L.push(' return key;', ' }'); + }; + for (const name of METHODS) { + L.push('', ` ${name}(...args: unknown[]): QuerySet {`, ` const clone = this.chain('${name}');`); + for (let i = 0; i < 40; i++) L.push(` clone.steps.push({ op: '${name}', at: ${i}, arg: args[${i % 3}] ?? null });`); + L.push(' return clone;', ' }'); + cacheSlot(); + cacheSlot(); + } + while (slot < 70) cacheSlot(); + L.push('}'); + return L.join('\n') + '\n'; + } + + function listFilterSource(): string { + return [ + 'import { QuerySet } from "../store/query";', + '', + 'export class ListFilter {', + ' constructor(private readonly field: string) {}', + '', + ' queryset(qs: QuerySet, params: Record): QuerySet {', + ...Array.from({ length: 30 }, (_, i) => ` if (params['p${i}']) qs = qs.filter(this.field, params['p${i}'], ${i});`), + ' return qs.distinct();', + ' }', + '}', + '', + ].join('\n'); + } + + let p: Project; + let r: Awaited>; + + beforeAll(async () => { + const files: Record = { [NAMED_FILE]: querySource(), [LOWER]: listFilterSource() }; + // Into the >=500-file tier: at 13K the ceiling sits 50% over the budget + // and a lower file always has room; at 24K it sits 4% over, as on django. + for (let i = 0; i < 520; i++) files[`src/pad/pad${i}.ts`] = `export const pad${i} = ${i};\n`; + p = await buildProject(files); + r = await explore(p, METHODS.map((m) => `QuerySet.${m}`).join(' ')); + }, 120_000); + afterAll(() => destroyProject(p)); + + it('fixture shape: the 24K tier, a lower file with a reservation', () => { + expect(r.report.budget.maxOutputChars).toBe(24000); + expect(r.file(LOWER)?.allowance ?? 0).toBeGreaterThan(EXPLORE_ALLOCATION.MIN_CHARS); + }); + + it('still extends the named file past the valve, into a saturated response — the guard is not vacuous', () => { + const valve = Math.round(r.report.budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE); + expect(r.file(NAMED_FILE)?.spendable ?? 0).toBeGreaterThan(valve); + expect(r.report.envelope.chars).toBeGreaterThan(r.report.budget.maxOutputChars); + }); + + it('delivers the lower file at least its reservation', () => { + const lower = r.file(LOWER); + expect(lower?.skipped ?? null).toBeNull(); + // Funded in full: the named file above took nothing the lower file is owed. + expect(lower?.funded ?? 0).toBeGreaterThanOrEqual(lower?.allowance ?? Infinity); + // And delivered in full, to the line: a render stops at the last WHOLE line + // that fits (#2062), so it may come in under its reservation by less than + // one line, never by more. More is the final ceiling fit taking source to + // pay for a header that ran past its estimate — 922 of 1,031 here, once an + // exact target's overshoot (#2063) spent the named file to its funded line. + const oneLine = Math.max(...listFilterSource().split('\n').map((l, i) => l.length + String(i + 1).length + 2)); + expect(lower?.emittedChars ?? 0).toBeGreaterThan((lower?.allowance ?? Infinity) - oneLine); + }); + + it('stays inside the inline cap', () => { + expect(r.text.length).toBeLessThanOrEqual(25000); + }); +}); diff --git a/__tests__/explore-proportional-allocation.test.ts b/__tests__/explore-proportional-allocation.test.ts index 2c11ac0417..dfa4bbf82b 100644 --- a/__tests__/explore-proportional-allocation.test.ts +++ b/__tests__/explore-proportional-allocation.test.ts @@ -550,3 +550,72 @@ describe('allocateExploreBudget — the diffuse-query control', () => { expect(precise).toBeGreaterThan(0.45); }); }); + +// ── A file the query named is exempt from the MAX_SHARE valve ───────────────── + +describe('allocateExploreBudget — a named file keeps its share past the valve', () => { + // The valve hedges against a mis-ranked dominant file, and it never + // redistributed: a clamped file's excess was left unreserved, where no file + // could spend it. A file the query asked for by name — a symbol it defines, or + // its path — is not the file the hedge is for. Whether the render loop may + // actually SPEND that share is a separate question, answered against spare + // room in `explore-named-file-valve.test.ts`. + const budget = getExploreOutputBudget(1000); + const valve = Math.round(budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE); + + it('lets a lone named file keep the whole pool rather than the valve', () => { + const { allowances, pool } = allocateExploreBudget([cand('named.ts', 42, { named: true })], budget, 8); + expect(allowances.get('named.ts')).toBe(pool); + expect(allowances.get('named.ts')!).toBeGreaterThan(valve); + }); + + it('treats a file named by PATH the same way', () => { + const { allowances, pool } = allocateExploreBudget([cand('pinned.ts', 42, { pinned: true })], budget, 8); + expect(allowances.get('pinned.ts')).toBe(pool); + }); + + it('takes the excess from the valve, never from the other files', () => { + const peers = [cand('p1.ts', 40), cand('p2.ts', 30)]; + const plain = allocateExploreBudget([cand('top.ts', 900), ...peers], budget, 8).allowances; + const named = allocateExploreBudget([cand('top.ts', 900, { named: true }), ...peers], budget, 8).allowances; + expect(plain.get('top.ts')).toBe(valve); + expect(named.get('top.ts')!).toBeGreaterThan(valve); + for (const peer of ['p1.ts', 'p2.ts']) expect(named.get(peer), peer).toBe(plain.get(peer)); + }); + + it('still clamps an unnamed dominant file when a lower file is the named one', () => { + const { allowances } = allocateExploreBudget( + [cand('dominant.ts', 900), cand('asked.ts', 40, { named: true })], + budget, + 8, + ); + expect(allowances.get('dominant.ts')).toBe(valve); + }); + + it('still fits the pool exactly, at every tier', () => { + for (const fileCount of TIER_FILE_COUNTS) { + const tier = getExploreOutputBudget(fileCount); + const a = allocateExploreBudget( + [cand('top.ts', 900, { named: true }), cand('b.ts', 40), cand('c.ts', 20, { named: true })], + tier, + 8, + ); + expect(reservedTotal(a), `tier ${fileCount}`).toBeLessThanOrEqual(a.pool); + expect(a.pool).toBeLessThanOrEqual(tier.maxOutputChars); + } + }); + + it('never gives a named file less at a larger tier', () => { + const files = [cand('top.ts', 900, { named: true }), cand('b.ts', 40)]; + let previous: Map | null = null; + for (const fileCount of TIER_FILE_COUNTS) { + const { allowances } = allocateExploreBudget(files, getExploreOutputBudget(fileCount), 8); + if (previous) { + for (const [path, chars] of allowances) { + expect(chars, `${path} shrank at ${fileCount} files`).toBeGreaterThanOrEqual(previous.get(path)!); + } + } + previous = allowances; + } + }); +}); diff --git a/docs/design/explore-budget-allocation.md b/docs/design/explore-budget-allocation.md index d77d6dd87a..e8fe5d2556 100644 --- a/docs/design/explore-budget-allocation.md +++ b/docs/design/explore-budget-allocation.md @@ -834,3 +834,249 @@ branch as well, all five render whole. last admitted file is held its partial share. Just below it, it is held nothing, and the file above takes ~800 chars the last file needed. Any upstream shift of a few dozen chars can flip it (django-bag above). + +## Named concentration — merged spine clusters and the valve (2026-09-27) + +Found in a pre-release agent A/B on microsoft/vscode (14,698 files, the 24K tier), identical on +v1.6.0 and `main`. The query + +``` +MessageType enum rpcProtocol.ts _receiveOneMessage MessageIO serializeRequest serializeReplyOK serializeReplyErr serializeCancel serializeAcknowledged +``` + +came back at 8.6K. `rpcProtocol.ts` rendered lines 20–24 and 269–325: none of the five +`MessageIO.serialize*` bodies, not the `MessageType` enum, and not the Flow section's own step 2. +The agent Read the file five times. + +### It was not the per-file cap + +The report's hypothesis was that the named file hit its per-file cap while the call budget sat +unspent. The diagnostic said otherwise: `rpcProtocol.ts` was **reserved 16,800** (the `MAX_SHARE` +valve), **funded 18,948, and spent 2,910**, with the render reported as `clusters`, *unclipped*. +The bytes were there. The render path threw them away. + +`buildSection` windows an oversize spine method (n8n's 962-line `processRunExecutionData`) to its +signature head plus ±28 lines of its next-hop call. It decided that on the **cluster's** span. +A cluster merges every symbol within `gapThreshold` lines of the next, so a pinned file (whose +whole symbol list enters the gather), or any dense gather, becomes one cluster spanning the file +around a short spine method. `rpcProtocol.ts` was one cluster over lines 20–968 around a 77-line +`_receiveOneMessage`: head 20–24, call site 297 ± 28. The window fit the cap, so nothing reported +it as clipped, and the large tiers' completeness line told the agent the file was complete. + +### The window is a member rule + +The size test is now on the spine member itself. A cluster that is long only because it merged +renders plainly and, if it overruns, shrinks by member like any other cluster. A god-method +inside a merged cluster is windowed where it sits, with its neighbours rendered around it. Six +consequences surfaced while validating that change; each is fixed: + +| # | Consequence | Fix | +|---|---|---| +| 1 | The window elided source and reported the section unclipped | A windowed spine member marks the cluster shrunk (`clipped` in the diagnostic, the trimmed note on small tiers) | +| 2 | Shrink ranked members by importance alone, so an unnamed spine bridge lost to smaller incidental members (vscode's `_receiveRequest`: 4/36 lines) | Spine members first, as `rankedClusters` already ranks spine clusters first → 36/36 | +| 3 | Past the 300-node pin cap a pinned file splits. Its spine cluster (receiveOne plus 300 unrelated methods) ranks first and, shrunk against its own members only, filled the cap with filler while the named serializers one cluster down rendered **nothing** | For a merged spine cluster only, room for the other clusters' top members (spine, importance ≥ 9, never a container's whole span, never under `MIN_CHARS`) is held back; what they don't spend returns to it | +| 4 | Every budget upstream charges a gap as a bare `GAP_MARKER`, but gaps are named (#1711). A cluster shrunk to scattered members overran even the last-resort re-render, and the file was **skipped whole** | The last resort drops whole parts (non-focus first, never the final part) until the real text fits | +| 5 | The adaptive (focused/skeleton) path had no ceiling fit test. With the named file now spending its reservation, tail skeletons ran past the ceiling and the final cut took them and the pointer list | Same fit test as the other paths: trim trailing signatures, then trailing bodies; skip only if nothing fits | +| 6 | The summary line ("Found N symbols across M files. …") sits in `lines` as an empty placeholder and is filled in after the final cut, so nothing paid for it: responses ran up to 70 chars past the 25K inline cap, or squeezed out the lost-pointer note | Reserved at its worst case (the gather's counts bound the shown ones) in the epilogue floor and in the epilogue fit | + +Two alternatives for #3 were measured and rejected. **An unscoped hold-back** (every first +cluster) re-orders a trade-off that predates this change, and measured net-negative: tokio's +`run_task`, a spine method, fell from 60 lines to 2. **Charging containers** (a named 1,022-line +class) for their whole span over-reserved, and vscode's `RPCProtocol` lost 44 lines to room +nobody used. + +### The valve, for a file the query named + +This is the lever the report asked for. It turned out to matter less than the render bug, but it +is real: on express, "response.js res.send res.json res.render …" admits one file, clamps it at +9,100 with 3,700 of the pool unreserved, and cuts the named `send` at 55 of 97 lines in a 10.9K +response to a 13K budget. `MAX_SHARE` hedges against a mis-ranked dominant file, and it **never +redistributed**: the clamped excess was left unreserved, and the carry-forward only moves +reservations. + +- **Allocation:** a file the query named (the named-first tier, `namedSeedFiles`) or pinned is + exempt from the valve; its proportional share stands. Reservations still fit the pool exactly. +- **Render:** the part of that share above the valve is a *soft* share. The loop *guarantees* + `min(reserved, valve)` (the `fundedHeadroom` floor, and what `owedPayableBelow` holds for it). + The file may spend past the old rule only into **spare budget**: the call's `maxOutputChars` + left after every file below is paid its **full** reservation, payable or not. The spine's 1.5× + overshoot applies to the old rule's allowance, not to the extension. A soft share enters the + carry-forward ledger once the loop has moved past the file, and only as far as it was spent. + +Each part of the guard was the fix for a measured failure: + +| Variant | What happened | +|---|---| +| Guarantee the whole uncapped reservation | django (saturated 24K): two admitted files **dropped** so the named `query.py` could ship 3K more | +| Bound the extension by `fundedHeadroom` | It holds back only the *payable* prefix of what is owed below (CG-31), so on vscode the room it released still belonged to a partially-payable rank-4 file, which was **dropped** | +| Measure spare against the render ceiling, not the budget | The ceiling exists to absorb the render's own overshoot (named gaps, never-empty windows). django's named file overran its bound by 561 and a lower file was cut to **53%** of its reservation | +| Book the soft share in the ledger up front | What the file couldn't spend flowed down as slack that existed only in the ledger | + +### Measured effect + +Deterministic, 47 queries over the 7 README repos plus express. They are the README questions, +the suite's prose questions, precise symbol bags, pinned-file flows, and named-concentration +stress queries. Baseline is `main` `1cbac0af`. Named coverage means every definition of every +queried name in every rendered file. + +| | baseline | this change | +|---|---|---| +| named bodies complete | 140 | **183** (14 responses up, **0 down**) | +| named lines delivered | 5,397 | **7,876** (+46%; 2 responses −3 and −9 lines, the summary-line reserve) | +| responses truncated by the final cut | 3 | **0** | +| responses over the 25K inline cap | 5 (max 25,032) | **0** (max 24,999) | +| byte-identical responses | — | 15 of 47 | + +The reported query and the shapes it stands for: + +| query | complete named bodies | named lines | response | +|---|---|---|---| +| vscode, the report | 0 → **8 of 8** (+ the Flow's `_receiveRequest`) | 46 → 356 | 8.6K → 24.9K | +| same, without the path | 0 → 8 of 8 | 46 → 356 | 11.0K → 24.1K | +| django `query.py` pinned flow | 1 → 6 | 87 → 317 | 10.9K → 19.3K | +| okhttp `RealCall` named concentration | 4 → 9 | 187 → 732 | 13.8K → 24.7K | +| okhttp `RealInterceptorChain.kt` pinned | 2 → 3 | 107 → 356 | 24,058 → 24,978 | +| express `response.js` (the valve) | 5 → 6 | 192 → 234 | 10.9K → 13.8K | + +Files fall in four responses: django-pin 6 → 5, vscode-bag 5 → 4, okhttp-pb-chain 6 → 4, +excalidraw-pin 6 → 5 (that one was over the 25K cap on baseline). In each, the rank-1 named file +used to spend a fraction of its own reservation because of the windowing bug, and the +under-spend was passing down to incidental files. Of the five files that drop out, four are +still named in the pointer list. The fifth (okhttp's `ExchangeCodec.kt`) is counted in its +"… and 24 more files" line, which is CG-26's elastic list running out of room. + +### Coverage + +| File | Guards | +|---|---| +| `__tests__/explore-merged-spine-cluster.test.ts` | the member rule (pinned rpcProtocol shape), a real god-method still windowed with its named neighbours whole and the file clipped, the merged-spine hold-back past the pin cap, the last-resort trim at the 24K tier | +| `__tests__/explore-named-file-valve.test.ts` | the valve for a named file far from its budget, and the spare-budget guard on a saturated 24K response | +| `__tests__/explore-proportional-allocation.test.ts` | the allocator exemption: peers unchanged, pool fit at every tier, tier monotonicity | + +Mutation-tested, same method as CG-14: + +| Mutation | Red | +|---|---| +| `main`'s `tools.ts` | 9 of the 19 new e2e tests (6 of 11 render, 3 of 8 valve) | +| Windowing not reported as clipped | god-method `clipped` | +| No hold-back for other clusters | past-the-pin-cap named bodies | +| No last-resort trim | 24K file skipped (0 chars) | +| Allocator exemption removed, or render extension removed | far-from-spent: 3 of 5 named bodies | +| Extension bounded by `fundedHeadroom` | saturated: lower file dropped | +| Spare measured against the render ceiling | saturated: lower file cut below its reservation | + +Four levers are validated only by the deterministic sweep. No hermetic fixture reproduced them +without over-fitting: + +- **spine-first member order:** vscode's `_receiveRequest`, 4/36 → 36/36 +- **top-up of unspent hold-back:** alamofire's `Session.swift`, 876 chars and a named method short without it +- **adaptive-path fit test and summary-line reserve:** 3 truncated and 5 over-cap responses → 0 + +### Agent A/B + +`scripts/agent-eval/ab-new-vs-baseline.sh`, baseline pinned to `1cbac0af` by SHA, Sonnet +`--effort high`, `RUNS=2`, prompts of the form `Use codegraph to answer: `. Raw logs are +not committed. The three questions: vscode's rpcProtocol serialization and dispatch (the reported +shape); django's `SQLCompiler.as_sql` / `pre_sql_setup` / `get_select`; and tokio's worker `run` / +`run_task` / `next_task` / `steal_work` / `park`. + +| repo | Read (new / base) | explore calls | duration median | do the builds differ on these queries? | +|---|---|---|---|---| +| vscode | **0, 0 / 1, 0** | 3, 3 / 2, 3 | 69s / 48s | yes: +82 to +320 source lines on the `rpcProtocol.ts` / `MessageIO` queries | +| django | 0, 2 / 1, 1 | 3, 2 / 2, 2 | 43s / 30s | no: 4 of 9 byte-identical, the rest ≤1 line apart except one tail-file swap | +| tokio | 0, 2 / 0, 1 | 3, 3 / 2, 2 | 177s [49–306] / 31s | no: 0–1 lines apart on 7 of 10, 22–46 on 3 | + +Reads tie overall (4 / 4). On vscode, the one repo where the builds actually return different +bytes, the baseline's Read was the reported symptom itself (`rpcProtocol.ts lines 299 to 355` +after an explore). The new arm's answers covered the named serializers in both runs (11 of 13 +key symbols vs 9 and 11). + +The new arm was slower in every repo. Attribution, per the CG-22 rule (never quote a wall-clock +gap without the explore-latency and response-size probes): + +- **vscode:** model time was equal or lower in the new arm (30.7s / 23.4s vs 31.6s / 30.2s). The + gap sat inside explore calls (39s per run vs 12–17s). Replaying the new arm's six queries, CPU + per call was within 1–4% across new/base build × new/base index, while the machine's load + average was 7–10 from other sessions. Load, not the change. +- **django, tokio:** the two builds return near-identical responses to these agents' own queries, + so a difference between the arms there cannot be the change. tokio's 306s run is a model-side + stall (302s of API time, 1.2s in tools). +- The new arm made 3 explore calls in all 6 runs vs 2.3 per run for baseline. On django and tokio + that is sampling for the same reason. A larger n would settle whether vscode's richer responses + prompt a deeper follow-up. + +Every Read in the django runs, on both builds, is a separate pre-existing gap: +`SQLCompiler.as_sql`'s 226-line body is never returned, not even to a query that names it and +its line number. + +### Found, not fixed + +- **`SQLCompiler.as_sql` is never returned** (above). It is the cause of every django Read in the + A/B, on both builds. +- **Density ranking starves an isolated named cluster.** Among clusters of equal max importance, + `rankedClusters` orders by summed member importance ÷ span. Edge-line members inflate the + density of a named-function-plus-filler cluster, so a cluster holding only a named function + loses the room. Repro: five named functions interleaved with helpers, where two render 0 lines + on both builds. *Fixed by #2062 ("Named members before incidental ones", above).* +- **A pinned file past the 300-node pin cap drops named symbols beyond the cap.** Identical on + baseline. The named symbols were in fact gathered (a search entry at importance 10) and + injected; the render dropped them, by the same density starvation as the item above. *Fixed by + #2062; `__tests__/explore-pinned-cap-named.test.ts` (#2064) guards the pinned entry path.* +- **The large tiers' completeness line** ("Complete source for N files is included above") is + unconditional, so it overclaims whenever a section was trimmed. *Still open.* + +### Landing on `main` (2026-09-28) + +This work sat unmerged while #2062 and #2063 landed in the same render path, so it was +re-applied onto `main` rather than merged as written: + +- **The merged-spine hold-back (#3 above) and its top-up were dropped.** #2062 holds back, for + every cluster, room for the protected members (named, entry point, spine) of the clusters + ranked below it. That is #3 without the scoping. The scoping existed because an unscoped + hold-back of *top-member room* cost tokio's `run_task` its body; #2062's form never holds back a + cluster's own protected members, only its incidental ones, which is why it can apply everywhere. + The past-the-pin-cap test above (`a merged spine cluster leaves room for the file's other + clusters`) passes on #2062's hold-back. +- **#2063's exact-target rule moves to the member.** `main` skipped the spine window for any + cluster holding an exact target (a qualified name, a line anchor). With the window now a + member rule, a spine method is not windowed if it is, or contains, an exact target, so an + anchored `lines 900-1003` inside it is never elided with the rest of the body. +- **The shrink orders exact targets ahead of spine members.** #2063's exact branch keeps members + in sort order until the cap, so a long spine method sorted first would price out the body the + agent singled out. Order: protected (when a hold-back applies), exact, spine, importance. +- **Exact pricing sees member windows.** `renderedSizeOfKept` prices the spans `buildSection` + renders (`sectionRangesOf`), so an oversize spine member costs its window. +- **The final fit shortens the file header before it cuts source.** Every file is funded for + source against an *estimated* header (`sectionOverhead`, the file's own symbol names). The + real header also lists edge-line targets (`filter(calls)`) and can run past the estimate. + Once #2063 gave exact targets the spine's cap, a named file with qualified targets spent right + up to its funded line, and the saturated valve fixture's lower file lost 109 chars of source + to its own header (922 of 1,031; 995 on `main`). Where a section no longer fits, the header now + lists fewer names until it is back within its estimate, and only then is source cut. The + valve test asserts funding in full and delivery to the line: #2062's exact rendering stops at + the last whole line that fits, so a lower file may come in under its reservation by less than + one line. + +Measured on the same 47-query replay, `main` `7506dbff` vs this re-application, one index: + +| | `main` | `main` + named concentration | +|---|---|---| +| named bodies complete | 230 | **272** (13 responses up, 1 down) | +| named lines delivered | 7,986 | **10,282** (+29%) | +| responses over the 25K inline cap | 2 | **0** | +| byte-identical responses | — | 16 of 47 | + +The one response down is okhttp's pinned `RealCall.kt`, which drops the mock server's +`QueueDispatcher.kt`, holding an `enqueue` namesake, as the named file spends its reservation. +The named definitions that lose lines all sit in responses that gain more: vscode's +`MessageBuffer` class (187 → 65, with three more named bodies complete and +258 named lines), +alamofire's `Session.request` (17 → 12, +48), and one swap: vscode's stress query renders +`_doActivateExtension` whole (0 → 33) where `main` rendered `_startExtensionHost` (19 → 0), four +complete bodies either way. Files also fall where the record above says they do (okhttp's +`RealInterceptorChain.kt` pin, 6 → 4). + +A second replay: 17 queries that pin a real file past the 300-node pin cap and name symbols beyond +it (vscode's `editorOptions.ts`, `chatService.ts`, `languages.ts`, `extHostTypes.ts`, +`extHost.protocol.ts`; excalidraw's `App.tsx`; okhttp's `URLConnectionTest.kt`), most with an +unpinned twin: 48 → 49 complete named bodies, 0 responses down, 753 → 760 named lines, and 3 → 0 +over the 25K cap (the summary-line reserve). diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 21a4bbdf02..0455afbed8 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -594,6 +594,15 @@ export const EXPLORE_ALLOCATION = { * Safety valve, as a fraction of the envelope. Not the primary guard any more — * the proportional split is — so this only has to stop a pathological * single-file response. + * + * It does not apply to a file the query NAMED — one that defines a symbol the + * query spelled, or that the query named by path. The valve hedges against a + * mis-ranked dominant file, and a file the agent asked for by name is not + * one. It also never redistributed: a clamped file's excess was simply left + * unreserved, and the carry-forward only moves reservations, so nobody could + * spend it. On express, "response.js res.send res.json res.render …" admits + * one file, clamps it at 9,100 with 3,700 of the pool unreserved, and cuts + * the named `send` body at 55 of 97 lines in a 10.9K response to a 13K budget. */ MAX_SHARE: 0.7, /** @@ -691,6 +700,12 @@ export interface ExploreAllocationCandidate { * graph scores (which a pure-path query doesn't produce) defeats the ask. */ pinned?: boolean; + /** + * The query named a symbol this file defines (the named-first sort tier). Like + * a pinned file, it is exempt from the `MAX_SHARE` valve: its proportional + * share stands, bounded only by the pool. + */ + named?: boolean; } export interface ExploreAllocation { @@ -704,6 +719,11 @@ export interface ExploreAllocation { pool: number; } +/** A file the query asked for by name — a symbol it defines, or its path. */ +function isNamedCandidate(c: Pick): boolean { + return c.named === true || c.pinned === true; +} + /** * Split `budget.maxOutputChars` across ranked candidates in proportion to * relevance, with a hard relative cliff. @@ -795,10 +815,12 @@ export function allocateExploreBudget( // exactly — the render loop spends them, so an over-allocation is an over-long // response the hard ceiling then has to truncate. Flooring costs at most one // char per file. + // The valve spares a file the query named (see MAX_SHARE): its share is already + // a slice of `pool`, so the reservations still fit the pool exactly. for (const c of admitted) { const share = Math.floor(floors / admitted.length) + Math.floor((remainder * (weights.get(c.path) ?? 0)) / total); - allowances.set(c.path, Math.min(share, ceiling)); + allowances.set(c.path, isNamedCandidate(c) ? share : Math.min(share, ceiling)); } return { allowances, cliffed, cliffAt, pool }; } @@ -4774,6 +4796,7 @@ export class ToolHandler { spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id) || exactNodeIds.has(n.id)) || anchorSpans.has(fp), pinned: pinnedSet.has(fp), + named: namedSeedFiles.has(fp), })), budget, maxFiles, @@ -4871,7 +4894,21 @@ export class ToolHandler { const g = fileGroups.get(fp); return g ? n + pointerLineFor(fp, g.nodes).length + 1 : n; }, cliffedFiles.size > 0 ? POINTER_HEADER.length + 2 : 0); - const epilogueFloor = EPILOGUE_LOST_NOTE.length + 2 + cliffPointerFloor; + // The summary line ("Found N symbols across M files. …") is written only + // after the final cut, and sits in `lines` as an empty placeholder until + // then — so the loop never paid for it, and a response filled to the + // ceiling ended up past it by the line's length once it was filled in: five + // of 47 measured responses over the 25K inline cap (up to 25,032), or the + // lost-pointer note above squeezed out. + // Reserved at its worst case: the gather's symbol and file counts bound the + // shown ones from above. + const plural = (n: number, word: string) => `${n} ${word}${n === 1 ? '' : 's'}`; + const summaryReserve = `Found ${plural(subgraph.nodes.size, 'symbol')} across ${plural(fileGroups.size, 'file')}.`.length + + (pinnedFiles.length > 0 ? ` ${plural(pinnedFiles.length, 'file')} pinned from the query.`.length : 0) + + (unresolvedPathSpans.length > 0 + ? ` No indexed file uniquely matches ${unresolvedPathSpans.map((sp) => `\`${sp}\``).join(', ')}.`.length + : 0); + const epilogueFloor = EPILOGUE_LOST_NOTE.length + 2 + cliffPointerFloor + summaryReserve; // Absolute stop for the render loop. Reservations already fit the envelope, so // this only catches their bounded overshoot (the whole-file grace, an oversize // first cluster) — and catches it HERE, where a file can be skipped cleanly and @@ -4936,16 +4973,42 @@ export class ToolHandler { // `sourceSpent` outrun `reservedSoFar`, which suppresses slack until a later // under-spend covers the debt. So the pool is conserved in both directions, // and no file is ever cut BELOW the reservation it was promised. + // + // Both totals count what a file is GUARANTEED (below), which is its whole + // reservation for every file but a named one. A named file's share above + // the `MAX_SHARE` valve is spare-room money, not a promise: it is entered + // only once the loop has moved past the file, and only as far as the file + // actually spent it (`unsettledSoft`). Booked in full up front, whatever the + // file could not spend would flow down as slack that exists only in the + // ledger, and count against the buy rule's funding line as a debt nobody + // owes. let reservedSoFar = 0; let sourceSpent = 0; + let unsettledSoft = 0; + const valveCap = Math.round(budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE); + const isNamedFile = (path: string): boolean => + isNamedCandidate({ named: namedSeedFiles.has(path), pinned: pinnedSet.has(path) }); + /** + * What the render loop GUARANTEES a file, whatever that costs the files + * below it: its reservation, except a named file's share above the + * `MAX_SHARE` valve. The allocator lifts the valve for a named file to hand + * it budget no other file would spend, not to take room a lower file is + * owed — so that part is funded only from spare room, and is never held back + * for it either. Guaranteeing the whole reservation dropped two admitted + * files from a saturated django response so the named `query.py` could ship + * 3K more of itself. + */ + const guaranteedOf = (path: string, reserved: number): number => + (isNamedFile(path) ? Math.min(reserved, valveCap) : reserved); // Funding line for the whole-file BUY rule: the response's SOURCE may reach // everything the allocator promised plus one bounded overshoot, and no more. // Measured against the promise rather than `renderCeiling` on purpose — the // ceiling sits 50% above the envelope and says nothing about who is owed // what, so funding a buy from it just moves the shortfall to whichever file - // the loop reaches last. See WHOLE_FILE_BUY_OVERSHOOT_FRACTION. - const reservedTotal = [...allocation.allowances.values()].reduce((sum, n) => sum + n, 0); - const sourceCeiling = reservedTotal + Math.round( + // the loop reaches last. See WHOLE_FILE_BUY_OVERSHOOT_FRACTION. The promise + // grows as a named file's soft share is settled (see `unsettledSoft`). + let reservedTotal = [...allocation.allowances].reduce((sum, [path, n]) => sum + guaranteedOf(path, n), 0); + const buyOvershoot = Math.round( budget.maxOutputChars * EXPLORE_ALLOCATION.WHOLE_FILE_BUY_OVERSHOOT_FRACTION, ); /** @@ -4994,12 +5057,29 @@ export class ToolHandler { * whole reservation, and when they do not, the carry-forward hands the * difference to whoever comes next anyway. */ + /** + * Everything still owed below `fileIndex`, in render-space chars, whether the + * ceiling can pay it or not — the bar a named file's extension has to clear + * (see the render loop). What is left above it is room no admitted file will + * ever claim. + */ + const owedInFullBelow = (fileIndex: number): number => { + let owed = 0; + for (let j = fileIndex + 1; j < sortedFiles.length; j++) { + const [path, g] = sortedFiles[j]!; + const r = allocation.allowances.get(path); + if (r !== undefined) owed += guaranteedOf(path, r) + sectionOverhead(path, g.nodes); + } + return owed; + }; const owedPayableBelow = (fileIndex: number, budgetLeft: number): number => { let held = 0; for (let j = fileIndex + 1; j < sortedFiles.length; j++) { const path = sortedFiles[j]![0]; - const r = allocation.allowances.get(path); - if (r === undefined) continue; + const reservedBelow = allocation.allowances.get(path); + if (reservedBelow === undefined) continue; + // Only what that file is guaranteed is owed to it (see `guaranteedOf`). + const r = guaranteedOf(path, reservedBelow); const overhead = sectionOverhead(path, sortedFiles[j]![1].nodes); const need = r + overhead; if (held + need > budgetLeft) { @@ -5021,6 +5101,16 @@ export class ToolHandler { }; for (let fileIndex = 0; fileIndex < sortedFiles.length; fileIndex++) { + // Settle the soft share of the file the loop just left: what it spent past + // its guarantee becomes a promise it kept; the rest was never available. + // Here rather than after its render, because every exit path below — + // however it `continue`s — comes back through this line. + if (unsettledSoft > 0) { + const settled = Math.min(unsettledSoft, Math.max(0, sourceSpent - reservedSoFar)); + reservedSoFar += settled; + reservedTotal += settled; + unsettledSoft = 0; + } const [filePath, group] = sortedFiles[fileIndex]!; if (filesIncluded >= maxFiles) { if (diag) for (const [fp] of sortedFiles) diag.recordSkip(fp, 'max-files'); @@ -5049,13 +5139,14 @@ export class ToolHandler { // order because that is the only file a single-pass loop can still pay; // `MAX_SHARE` keeps that from turning a weak tail file into the response, // so the allocator's share ceiling holds end-to-end and not just at - // reservation time. - const allowance = Math.min( - reserved + Math.max(0, reservedSoFar - sourceSpent), - Math.max(reserved, Math.round(budget.maxOutputChars * EXPLORE_ALLOCATION.MAX_SHARE)), - ); - reservedSoFar += reserved; - diag?.recordSpendable(filePath, allowance); + // reservation time. (A named file may go past it — but only into spare + // room; see below the displacement guard.) + const slack = Math.max(0, reservedSoFar - sourceSpent); + const guaranteed = guaranteedOf(filePath, reserved); + const ruleAllowance = Math.min(guaranteed + slack, Math.max(guaranteed, valveCap)); + let allowance = ruleAllowance; + reservedSoFar += guaranteed; + unsettledSoft = reserved - guaranteed; // DISPLACEMENT GUARD, in render space (CG-31). `allowance` says what this // file MAY spend; it does not say the bytes are still there to spend. The // hard ceiling is shared with every file the loop has not reached yet, and @@ -5072,7 +5163,8 @@ export class ToolHandler { // // Floored at this file's OWN reservation, never below: a kept promise is // not a displacement, and cutting a file under what it earned is the - // failure this whole allocation layer exists to prevent. + // failure this whole allocation layer exists to prevent. (For a named + // file, the part of the reservation it is `guaranteed` — see above.) // // Slack still reaches the file: a file above that under-spends leaves // `totalChars` lower, which raises `headroom` one-for-one, so the @@ -5080,9 +5172,30 @@ export class ToolHandler { // this bound funds. const headroom = Math.max(0, renderCeiling - totalChars - sectionOverhead(filePath, group.nodes)); const fundedHeadroom = Math.max( - Math.min(reserved, headroom), - headroom - owedPayableBelow(fileIndex, Math.max(0, headroom - reserved)), + Math.min(guaranteed, headroom), + headroom - owedPayableBelow(fileIndex, Math.max(0, headroom - guaranteed)), ); + // A file the query NAMED may spend past the rule above — its share over the + // `MAX_SHARE` valve, and slack past the valve — but only into SPARE room: + // the call's BUDGET left once every file below is paid its full + // reservation, payable or not. Two bars that look right and are not: + // - `fundedHeadroom` holds back only the payable prefix of what is owed + // below (CG-31), so on a saturated response the room it releases still + // belongs to a partially-payable file: reaching it cost vscode an + // admitted rank-4 file its whole section. + // - the render CEILING sits above the budget to absorb the render's own + // bounded overshoot (named gap markers, never-empty windows). Spend up + // to it and that overshoot lands on the files below: django's named + // `query.py` ran 561 chars past its bound and cut a lower file to 53% + // of its reservation. + // Where nothing is spare the extension is zero and the file renders + // exactly as it did before the valve was lifted for it. + if (isNamedFile(filePath)) { + const spare = budget.maxOutputChars - totalChars - sectionOverhead(filePath, group.nodes) + - owedInFullBelow(fileIndex); + allowance = Math.max(allowance, Math.min(reserved + slack, spare)); + } + diag?.recordSpendable(filePath, allowance); diag?.recordFunded(filePath, fundedHeadroom); const absPath = validatePathWithinRoot(projectRoot, filePath); if (!absPath || !existsSync(absPath)) { @@ -5607,7 +5720,6 @@ export class ToolHandler { } } } - const sigTail = sigDropped > 0 ? `… +${sigDropped} more (signatures elided)` : ''; if (skel.length > 0) { const names = [...new Set(group.nodes.filter(n => n.kind !== 'import' && n.kind !== 'export').map(n => n.name))] .slice(0, budget.maxSymbolsInFileHeader).join(', '); @@ -5625,12 +5737,39 @@ export class ToolHandler { // (a one-line signature is far under MIN_COVERED_LINES and is never // withheld — the structure map is what makes this render legible). const dd = dedupeSpans(skel); - const withTail = (parts: ReadonlyArray<{ text: string }>) => - [...parts.map((p) => p.text), ...(sigTail ? [sigTail] : [])].join('\n'); + const withTail = (parts: ReadonlyArray<{ text: string }>, extraDropped = 0) => { + const dropped = sigDropped + extraDropped; + const tail = dropped > 0 ? `… +${dropped} more (signatures elided)` : ''; + return [...parts.map((p) => p.text), ...(tail ? [tail] : [])].join('\n'); + }; + const header = fileSectionHeader(filePath, `${names} · ${tag}`); + // The ceiling fit test every other render path has (CG-26), which this + // one lacked: it emitted whatever it built, so a skeleton at the tail + // of a full response ran past the ceiling and the final cut threw away + // it, the section before it, and the pointer list behind them. Trim + // from the end instead — trailing signatures first, then trailing + // bodies — and skip the file only if not one part fits. + let parts = dd.parts; + let trimmedSigs = 0; + const sectionCost = (ps: ReadonlyArray<{ text: string }>) => + header.length + 2 + (ps.length > 0 ? withTail(ps, trimmedSigs).length + lang.length + 11 : 0); + while (parts.length > 1 && totalChars + sectionCost(parts) > renderCeiling) { + let drop = -1; + for (let k = parts.length - 1; k >= 0; k--) { + if (parts[k]!.range.start === parts[k]!.range.end) { drop = k; break; } + } + if (drop < 0) drop = parts.length - 1; + else trimmedSigs++; + parts = parts.filter((_, k) => k !== drop); + } + if (dd.parts.length > 0 && totalChars + sectionCost(parts) > renderCeiling) { + diag?.recordSkip(filePath, 'budget-clusters'); + continue; + } emitFileSection({ - header: fileSectionHeader(filePath, `${names} · ${tag}`), - body: dd.parts.length > 0 ? withTail(dd.parts) : '', - ranges: dd.parts.map((p) => p.range), + header, + body: parts.length > 0 ? withTail(parts, trimmedSigs) : '', + ranges: parts.map((p) => p.range), covered: dd.covered, overhead: 120, mode: bodyIds.size + bodyWindows.size > 0 ? 'focused' : 'skeleton', @@ -5719,7 +5858,7 @@ export class ToolHandler { // the emission site below, so it cannot displace either. const buysWhole = fileContent.length <= graceBound || (reserved >= fileContent.length * EXPLORE_ALLOCATION.WHOLE_FILE_BUY_FRACTION - && sourceSpent + fileContent.length + owedBelow <= sourceCeiling + && sourceSpent + fileContent.length + owedBelow <= reservedTotal + buyOvershoot && fileContent.length <= fundedHeadroom); // Set by the whole-file arm when it actually emits. A whole render that // does not FIT no longer ends the file's turn (CG-26) — it falls through @@ -5950,11 +6089,54 @@ export class ToolHandler { const contextPadding = 3; // An oversize spine method (the call path runs THROUGH a god-method — n8n's // processRunExecutionData is 962 lines) is windowed to its next-hop CALL site - // plus the signature head, NOT dumped whole. Without this the cluster is too big + // plus the signature head, NOT dumped whole. Without this the method is too big // for any per-file cap and gets dropped, so the agent Reads the method back — // the exact gap this closes. Bounded, so a god-method can't blow the budget yet // the spine's call still appears in context. + // + // The size test is on the spine METHOD, never on the cluster holding it. A + // cluster merges every symbol within `gapThreshold` lines of the next, so on a + // dense file — and on every PINNED file, whose whole symbol list enters the + // gather — it spans hundreds of lines around a short spine method. Windowing + // that merge threw away every other symbol in it: vscode's pinned + // `rpcProtocol.ts` merged into one cluster over lines 20–968 around a 77-line + // `_receiveOneMessage`, and the five `MessageIO.serialize*` bodies and the + // `MessageType` enum the query named were cut to a 5-line file head plus ±28 + // lines of one call site — 2,910 chars spent of a 16,800 reservation, and + // reported unclipped. A cluster that is long only because it merged goes + // through the shrink-by-member path like any other oversize cluster. const OVERSIZE_SPINE_LINES = 200; + /** + * Spine methods too long to render whole, in source order. Not one that is, + * or holds, an EXACT target (a qualified name, an anchored line span): its + * body is what was asked for, so it is kept whole up to the ceiling and only + * then windowed — by `windowToCeiling`, which keeps the call site AND the + * named defs. Windowed here, an anchored `lines 900-1003` inside it would go + * with the rest of the body. + */ + const exactRanges = ranges.filter((r) => r.importance >= EXACT_IMPORTANCE); + const oversizeSpine = ranges + .filter((r) => r.spine && r.spineCallLine !== undefined + && r.end - r.start + 1 > OVERSIZE_SPINE_LINES + && r.spineCallLine >= r.start && r.spineCallLine <= r.end + && !exactRanges.some((x) => x.start <= r.end && x.end >= r.start)) + .sort((a, b) => a.start - b.start); + const oversizeSpineSet = new Set(oversizeSpine); + /** What an oversize spine method renders as: its signature head plus its call site. */ + const spineWindowOf = (g: { start: number; end: number; spineCallLine?: number }): ExploreLineRange[] => { + const call = g.spineCallLine!; + const winStart = Math.max(g.start, call - SPINE_WINDOW); + const winEnd = Math.min(g.end, call + SPINE_WINDOW); + // Signature head, only when it sits clearly above the window (else the + // window already covers the method opening). + const headEnd = Math.min(g.start + 4, winStart - 2); + return headEnd >= g.start + ? [{ start: g.start, end: headEnd }, { start: winStart, end: winEnd }] + : [{ start: winStart, end: winEnd }]; + }; + /** Whether rendering [start, end] windows an oversize spine method inside it. */ + const windowsSpineIn = (start: number, end: number): boolean => + oversizeSpine.some((g) => g.start >= start && g.end <= end); // Returns the rendered text as SPAN-KEYED PARTS. Every part carries the // exact line range its text was sliced from, which two things depend on: // the session record (CG-17) — a record claiming lines it never sent would @@ -5971,44 +6153,41 @@ export class ToolHandler { // 3-line stub where ~5,200 chars of its body had fit. const sectionText = (parts: ReadonlyArray): string => joinPartsWithNamedGaps(filePath, parts, fileIndexNodes, 0); - const buildSection = ( - c: { start: number; end: number; hasSpine?: boolean; spineCallLine?: number; maxImportance?: number }, - ): SectionPart[] => { - // Not for a cluster holding an exact target: its body is what was asked - // for, so it is kept whole up to the ceiling and only then windowed — - // by `windowToCeiling`, which keeps the call site AND the named defs. - if (c.hasSpine && c.spineCallLine && (c.end - c.start + 1) > OVERSIZE_SPINE_LINES - && (c.maxImportance ?? 0) < EXACT_IMPORTANCE) { - const call = c.spineCallLine; - const winStart = Math.max(c.start, call - SPINE_WINDOW); - const winEnd = Math.min(c.end, call + SPINE_WINDOW); - const parts: SectionPart[] = []; - // Signature head, only when it sits clearly above the window (else the - // window already covers the method opening). - const headEnd = Math.min(c.start + 4, winStart - 2); - if (headEnd >= c.start) { - const head = fileLines.slice(c.start - 1, headEnd).join('\n'); - parts.push({ - range: { start: c.start, end: headEnd }, - text: withLineNumbers ? numberSourceLines(head, c.start) : head, - }); - } - const win = fileLines.slice(winStart - 1, winEnd).join('\n'); - parts.push({ - range: { start: winStart, end: winEnd }, - text: withLineNumbers ? numberSourceLines(win, winStart) : win, - }); - return parts; + /** + * The line spans `buildSection` renders for [start, end] — split out so the + * exact pricing below (`renderedSizeOfKept`) prices a member window as the + * window it renders, not as the body it elides. + */ + const sectionRangesOf = (c: { start: number; end: number }): ExploreLineRange[] => { + // Plain source for [start, end], except each oversize spine method inside + // it, which renders as its window. Context padding goes on the outer edges + // of PLAIN source only: padding past a windowed method would add lines + // after a tail that was elided. + const spans: Array<{ range: ExploreLineRange; plain: boolean }> = []; + let cursor = c.start; + for (const g of oversizeSpine) { + if (g.start < cursor || g.start < c.start || g.end > c.end) continue; + if (g.start > cursor) spans.push({ range: { start: cursor, end: g.start - 1 }, plain: true }); + for (const w of spineWindowOf(g)) spans.push({ range: w, plain: false }); + cursor = g.end + 1; + } + if (cursor <= c.end) spans.push({ range: { start: cursor, end: c.end }, plain: true }); + const first = spans[0]; + const last = spans[spans.length - 1]; + if (first?.plain) first.range.start = Math.max(1, first.range.start - contextPadding); + if (last?.plain) last.range.end = Math.min(fileLines.length, last.range.end + contextPadding); + // Spans that touch in the file are one block, not two parts with a gap + // marker between adjacent lines. + const merged: ExploreLineRange[] = []; + for (const { range } of spans) { + const prev = merged[merged.length - 1]; + if (prev && range.start <= prev.end + 1) prev.end = Math.max(prev.end, range.end); + else merged.push({ ...range }); } - const startIdx = Math.max(0, c.start - 1 - contextPadding); - const endIdx = Math.min(fileLines.length, c.end + contextPadding); - const slice = fileLines.slice(startIdx, endIdx).join('\n'); - // startIdx is 0-based, so the slice's first line is line startIdx + 1. - return [{ - range: { start: startIdx + 1, end: endIdx }, - text: withLineNumbers ? numberSourceLines(slice, startIdx + 1) : slice, - }]; + return merged; }; + const buildSection = (c: { start: number; end: number }): SectionPart[] => + sectionRangesOf(c).map((r) => ({ range: r, text: renderSpan(r) })); /** Rendered cost of one member on its own, context padding included. */ const paddedCost = (r: ExploreRange): number => { @@ -6092,18 +6271,29 @@ export class ToolHandler { /** * Exact rendered size of `keep` once merged, padded and joined — what * `sectionText` will measure — via a prefix sum, so pricing a candidate - * member costs O(members), not a re-render. + * member costs O(members), not a re-render. Priced on the spans + * `buildSection` renders, so an oversize spine member costs its window. */ const renderedSizeOfKept = (keep: readonly ExploreRange[]): number => - mergeKept(keep).reduce((sum, m, i) => - sum + renderedSpanCost(m.start - contextPadding, m.end + contextPadding) + (i > 0 ? GAP_MARKER.length : 0), 0); + mergeKept(keep).flatMap((m) => sectionRangesOf(m)).reduce((sum, r, i) => + sum + renderedSpanCost(r.start, r.end) + (i > 0 ? GAP_MARKER.length : 0), 0); const shrinkCluster = ( c: ExploreCluster, cap: number, incidentalCap?: number, ): SectionPart[] | null => { if (c.members.length < 2) return null; const guarded = incidentalCap !== undefined; + // Protected members first when a hold-back applies (see `renderCluster`); + // then an EXACT target, then the spine, as `rankedClusters` ranks spine + // clusters first: the call path IS the answer. An unnamed bridge on it + // (vscode's `_receiveRequest`, the Flow section's own step 2) otherwise + // ranks with incidental helpers and is the member the cap drops. Exact + // ahead of spine because the exact branch below keeps members in this + // order until the cap: a long spine method first would price out the + // body the agent singled out. const byImportance = [...c.members].sort((a, b) => (guarded ? Number(isProtectedMember(b)) - Number(isProtectedMember(a)) : 0) + || Number(b.importance >= EXACT_IMPORTANCE) - Number(a.importance >= EXACT_IMPORTANCE) + || (b.spine ? 1 : 0) - (a.spine ? 1 : 0) || b.importance - a.importance || (a.end - a.start) - (b.end - b.start) || a.start - b.start); const keep: ExploreRange[] = []; if (c.maxImportance >= EXACT_IMPORTANCE) { @@ -6123,7 +6313,11 @@ export class ToolHandler { keep.push(r); } } else { - const sizeOf = (r: ExploreRange) => fileLines.slice(r.start - 1, r.end).join('\n').length; + // An oversize spine method costs its window, not its body — `buildSection` + // windows it wherever it lands. Charging the full body kept it as the + // most important member and then had no room left for anything beside it. + const sizeOf = (r: ExploreRange) => (oversizeSpineSet.has(r) ? spineWindowOf(r) : [r]) + .reduce((n, w) => n + fileLines.slice(w.start - 1, w.end).join('\n').length, 0); let kept = 0; // Rendered size of what is kept so far, once the first incidental member // is weighed. Every protected member sorts ahead of it, so this is the @@ -6214,6 +6408,10 @@ export class ToolHandler { const guard = Number.isFinite(ceiling) && c.members.some(isProtectedMember) ? Math.min(incidentalCap ?? Infinity, ceiling) : incidentalCap; + // A windowed spine method elides source even when nothing else is cut, so + // it counts as shrunk: reported unclipped, the one-file-head-plus-a-call- + // site render hid in the diagnostic as a complete section. + const windowed = windowsSpineIn(c.start, c.end); const bound = ( r: { parts: SectionPart[]; covered: ExploreLineRange[]; shrunk: boolean }, ) => { @@ -6224,11 +6422,11 @@ export class ToolHandler { return { parts, covered: r.covered, shrunk: true }; }; if (sectionText(base.parts).length <= Math.min(cap, guard ?? Infinity)) { - return { parts: base.parts, covered: base.covered, shrunk: false }; + return { parts: base.parts, covered: base.covered, shrunk: windowed }; } const shrunk = shrinkCluster(c, cap, guard); if (shrunk === null) { - return bound({ parts: base.parts, covered: base.covered, shrunk: false }); + return bound({ parts: base.parts, covered: base.covered, shrunk: windowed }); } const dd = dedupeSpans(shrunk); return bound({ parts: dd.parts, covered: dd.covered, shrunk: true }); @@ -6286,8 +6484,11 @@ export class ToolHandler { // spine can't run away or starve co-flow files entirely. The 1.5x is drawn // from the shared envelope, so it is exactly the overshoot the displacement // guard has to fund: past `fundedHeadroom` the extra half-reservation is - // another file's, not spare room. - const SPINE_CEILING = Math.min(Math.round(allowance * 1.5), fundedHeadroom); + // another file's, not spare room. The half is of `ruleAllowance`: a named + // file's spare-room extension is already the budget's last spare byte, and + // half again of it ran a 13K-budget response to 19.3K of its 19.5K ceiling. + // + const SPINE_CEILING = Math.min(Math.max(Math.round(ruleAllowance * 1.5), allowance), fundedHeadroom); const chosenIndices = new Set(); // Final renders (deduped, shrunk where oversize) by cluster index. Computed // during selection and reused at emission so the two never disagree. @@ -6395,7 +6596,6 @@ export class ToolHandler { chosenIndices.add(rc.idx); projectedChars += text.length + GAP_MARKER.length; } - // Gap names (#1711) are paid from what selection left of this file's // budget, never from source: selection measured every gap as bare. const namedGapBudget = Math.max(fileBudget, projectedChars); @@ -6459,13 +6659,16 @@ export class ToolHandler { const headerFor = ( symbols: readonly string[], elided: ReadonlyArray = [], + /** Longest the header may run; fewer names are listed to meet it (never under one). */ + maxLength = Infinity, ): string => { - const { shown, omitted } = biasHeaderSymbols( - symbols, elided, budget.maxSymbolsInFileHeader, - ); - return fileSectionHeader(filePath, omitted > 0 - ? `${shown.join(', ')}, +${omitted} more` - : shown.join(', ')); + for (let cap = budget.maxSymbolsInFileHeader; ; cap--) { + const { shown, omitted } = biasHeaderSymbols(symbols, elided, cap); + const header = fileSectionHeader(filePath, omitted > 0 + ? `${shown.join(', ')}, +${omitted} more` + : shown.join(', ')); + if (header.length <= maxLength || cap <= 1) return header; + } }; // Last stop before the hard ceiling. The reservation already bounded cluster @@ -6488,6 +6691,20 @@ export class ToolHandler { let chosenNow = chosenIndices; const costOfSection = (header: string, body: string) => header.length + 2 + (body.length > 0 ? body.length + lang.length + 11 : 0); + // Every file was funded for SOURCE against an ESTIMATED header + // (`sectionOverhead`: the file's symbol names), and the real one also lists + // the edge lines' targets (`filter(calls)`, `QuerySet(imports)`), so it can + // run past the estimate. Where the section no longer fits, the header pays + // that back first by listing fewer names — a name it drops is still in the + // gap markers and the pointer list; a source line the trim drops is not. + // Without this, a file above spending exactly its funded line (an exact + // target's overshoot, #2063) left the one below its estimate, and the trim + // took the difference out of source: 922 of the 1,031 it was owed. + let headerLimit = Infinity; + if (totalChars + costOfSection(fileHeader, assembled.text) > renderCeiling) { + headerLimit = sectionOverhead(filePath, group.nodes) - 2 - lang.length - 11; + fileHeader = headerFor(assembled.symbols, assembled.elided, headerLimit); + } // The weakest cluster is SHRUNK into the room that is left before it is // dropped (CG-36). Dropping it whole makes this loop as all-or-nothing as // the selection above it was, and at the same cost: on excalidraw's @@ -6530,7 +6747,7 @@ export class ToolHandler { chosenNow = trimmed; } assembled = assembleSection(chosenNow); - fileHeader = headerFor(assembled.symbols, assembled.elided); + fileHeader = headerFor(assembled.symbols, assembled.elided, headerLimit); anyFileTrimmed = true; } // One cluster left and still over — by the header estimate's error, at @@ -6546,6 +6763,23 @@ export class ToolHandler { - (fileHeader.length + 2 + lang.length + 11); if (room > 0) { const reshrunk = renderCluster(clusters[idx]!, room, room); + // Every budget upstream charges a gap as a bare GAP_MARKER, but the + // joined text NAMES what each gap elides (#1711). A cluster shrunk to + // scattered members has dozens of gaps, and their names can outweigh + // the source: re-rendered into the room, it still overran, and a + // pinned 1,300-line file with every named body inside its render was + // skipped whole. Drop whole parts until the REAL text fits instead — + // the last one that holds no focus line first, never the final part. + const focus = focusLinesOf(clusters[idx]!); + while (reshrunk.parts.length > 1 && sectionText(reshrunk.parts).length > room) { + const holdsFocus = (p: SectionPart) => focus.some((l) => l >= p.range.start && l <= p.range.end); + let drop = reshrunk.parts.length - 1; + for (let k = reshrunk.parts.length - 1; k >= 0; k--) { + if (!holdsFocus(reshrunk.parts[k]!)) { drop = k; break; } + } + reshrunk.parts.splice(drop, 1); + reshrunk.shrunk = true; + } renderedClusters.set(idx, reshrunk); anyClusterShrunk = anyClusterShrunk || reshrunk.shrunk; assembled = assembleSection(chosenNow); @@ -6730,7 +6964,9 @@ export class ToolHandler { // document order. const roomFor = (block: readonly string[]): number => block.reduce((n, s) => n + s.length + 1, 0); - let room = hardCeiling - (flow.text.length + lines.join('\n').length); + // Less what the summary line will grow by when its sentinel is filled in. + let room = hardCeiling - (flow.text.length + lines.join('\n').length) + - Math.max(0, summaryReserve - SUMMARY_SENTINEL.length); const keepCompleteness = completenessBlock.length > 0 && roomFor(completenessBlock) <= room; From 290e03f7bc582b8726a104428f39bc178bac30bd Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 08:03:11 +0000 Subject: [PATCH 006/363] fix(explore): pin only the same-named file that defines the symbols a query names (#2071) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A bare basename several directories share (`editorOptions.ts`, `worker.rs`, `compiler.py`) pinned every match, and the matches split the pinned reservation: on vscode `editorOptions.ts clampedInt cursorStyleToString cursorStyleFromString` gave the file holding all three 8,626 chars, against 15,181 without the path, and spent the rest on a workbench helper defining none of them. Path extraction now takes an injected `symbolFiles` lookup (the module stays DB-free, like `existsOnDisk`). When a span matches several files, the matches defining a precise symbol token the query names (camelCase, PascalCase, snake_case, qualified — the seeder's NL-stopword shape test) are pinned and the rest set aside; the summary line names them. When no match defines one, or every match does, the span resolves exactly as before. The same test resolves a basename shared past the ambiguity budget (django's `models.py`) when it narrows into budget, and a line anchor binds once the span narrows to one file. Replay of 196 queries over 8 indexed repos, main vs this branch: every unambiguous pin is byte-identical (38 unique basenames, 13 dir/basename, 8 file:line, 38 earlier-sweep path queries), as are 26 path-free queries, 13 where every match defines a named symbol and 8 naming no symbol. All 52 changed responses are shared basenames — named FULL 207 -> 231 (11 up, 0 down), named lines 6,045 -> 7,577, none over 25K. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/explore-same-basename-pin.test.ts | 290 ++++++++++++++++++++ __tests__/query-paths.test.ts | 137 ++++++++- src/mcp/tools.ts | 52 +++- src/search/query-paths.ts | 127 ++++++++- 5 files changed, 595 insertions(+), 12 deletions(-) create mode 100644 __tests__/explore-same-basename-pin.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d562a5ea6..6990630451 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -167,6 +167,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - When you name several functions in one file and unrelated code sits between them, `codegraph_explore` now returns the functions you named before the code around them, instead of spending the file's room on that code and leaving one of the functions you asked for with no source at all. - `codegraph_explore` now returns the body of a method you name together with its class, such as `SQLCompiler.as_sql`, even in a file full of same-named overrides. It used to shrink the method you asked for to its signature line while smaller methods around it came back in full. A method too long for the answer now comes back as its opening lines plus the places it calls the other symbols you named, with the exact follow-up query that returns the rest. - `codegraph_explore` now honours line numbers in a query: `compiler.py:776` returns the method containing that line, and `compiler.py lines 900-1003` (or `:900-1003`, `#L900-L1003`, `L900-L1003`) returns exactly those lines. It used to pin the file and ignore the numbers. +- When a file name in a `codegraph_explore` query matches files in more than one folder, only the ones defining a function or class the query also names are pinned, and the summary line lists the files left out. Every match used to be pinned, so the file you meant got half the room it gets when you leave its name out; a name shared by many files, like `models.py`, now resolves the same way instead of being reported as ambiguous. #### MCP / indexing diff --git a/__tests__/explore-same-basename-pin.test.ts b/__tests__/explore-same-basename-pin.test.ts new file mode 100644 index 0000000000..3c4f255793 --- /dev/null +++ b/__tests__/explore-same-basename-pin.test.ts @@ -0,0 +1,290 @@ +/** + * A bare basename two directories share pins only the file defining what the + * query names. + * + * vscode has two `editorOptions.ts`: the editor's option registry (which holds + * `clampedInt`, `cursorStyleToString`, `cursorStyleFromString`) and a small + * workbench helper holding none of them. `editorOptions.ts clampedInt + * cursorStyleToString cursorStyleFromString` pinned BOTH, and the two pins split + * the pinned reservation — 8,626 chars each, against 15,181 for the named file + * when the same query left the path out. So writing the file's name made the + * answer worse: the named file got half the room, and the unrelated one spent + * the rest. + * + * Path extraction now asks which of a span's matches define a symbol the query + * names, and pins those; the rest are set aside and named in the summary line. + * When no match defines one, or every match does, the symbols pick nothing out + * and every match pins as before. + * + * The fixture is that shape, generated so the sizes that decide it are visible + * here: the registry is ~60 option classes with the three named functions in + * the middle, the helper is a handful of unrelated functions. + */ +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ToolHandler } from '../src/mcp/tools'; +import type { ExploreDiagnosticReport } from '../src/mcp/explore-diagnostics'; + +const REGISTRY = 'src/editor/common/config/editorOptions.ts'; +const HELPER = 'src/workbench/common/editor/editorOptions.ts'; +const NAMED = ['clampedInt', 'cursorStyleToString', 'cursorStyleFromString']; + +function registrySource(): string { + const L: string[] = [ + 'export interface IConfigurationPropertySchema {', + ' type?: string;', + ' default?: unknown;', + ' minimum?: number;', + ' maximum?: number;', + '}', + '', + 'export abstract class BaseEditorOption {', + ' constructor(public readonly id: K, public readonly name: string, public readonly defaultValue: V) {}', + ' public abstract validate(input: unknown): V;', + '}', + ]; + const optionClass = (i: number) => { + L.push( + '', + `export class EditorOption${i} extends BaseEditorOption {`, + ` private readonly allowed${i} = ['alpha${i}', 'beta${i}', 'gamma${i}'];`, + '', + ` constructor() {`, + ` super(${i}, 'option${i}', 'alpha${i}');`, + ' }', + '', + ' public validate(input: unknown): string {', + " if (typeof input !== 'string') {", + ' return this.defaultValue;', + ' }', + ` return this.allowed${i}.includes(input) ? input : this.defaultValue;`, + ' }', + '}', + ); + }; + for (let i = 0; i < 30; i++) optionClass(i); + L.push( + '', + 'export function clampedInt(value: unknown, defaultValue: T, minimum: number, maximum: number): number | T {', + " if (typeof value === 'undefined') {", + ' return defaultValue;', + ' }', + ' let r = parseInt(String(value), 10);', + ' if (isNaN(r)) {', + ' return defaultValue;', + ' }', + ' r = Math.max(minimum, r);', + ' r = Math.min(maximum, r);', + ' return r | 0;', + '}', + '', + 'export class EditorIntOption extends BaseEditorOption {', + ' public static clampedInt(value: unknown, defaultValue: T, minimum: number, maximum: number): number | T {', + ' return clampedInt(value, defaultValue, minimum, maximum);', + ' }', + '', + ' constructor(id: K, name: string, defaultValue: number, public readonly minimum: number, public readonly maximum: number, schema?: IConfigurationPropertySchema) {', + " if (typeof schema !== 'undefined') {", + " schema.type = 'integer';", + ' schema.default = defaultValue;', + ' schema.minimum = minimum;', + ' schema.maximum = maximum;', + ' }', + ' super(id, name, defaultValue);', + ' }', + '', + ' public validate(input: unknown): number {', + ' return EditorIntOption.clampedInt(input, this.defaultValue, this.minimum, this.maximum);', + ' }', + '}', + '', + 'export const enum TextEditorCursorStyle {', + ' Line = 1,', + ' Block = 2,', + ' Underline = 3,', + ' LineThin = 4,', + ' BlockOutline = 5,', + ' UnderlineThin = 6,', + '}', + '', + 'export function cursorStyleToString(cursorStyle: TextEditorCursorStyle): string {', + ' switch (cursorStyle) {', + " case TextEditorCursorStyle.Line: return 'line';", + " case TextEditorCursorStyle.Block: return 'block';", + " case TextEditorCursorStyle.Underline: return 'underline';", + " case TextEditorCursorStyle.LineThin: return 'line-thin';", + " case TextEditorCursorStyle.BlockOutline: return 'block-outline';", + " case TextEditorCursorStyle.UnderlineThin: return 'underline-thin';", + ' }', + '}', + '', + 'export function cursorStyleFromString(cursorStyle: string): TextEditorCursorStyle {', + ' switch (cursorStyle) {', + " case 'line': return TextEditorCursorStyle.Line;", + " case 'block': return TextEditorCursorStyle.Block;", + " case 'underline': return TextEditorCursorStyle.Underline;", + " case 'line-thin': return TextEditorCursorStyle.LineThin;", + " case 'block-outline': return TextEditorCursorStyle.BlockOutline;", + " case 'underline-thin': return TextEditorCursorStyle.UnderlineThin;", + ' }', + ' return TextEditorCursorStyle.Line;', + '}', + ); + for (let i = 30; i < 60; i++) optionClass(i); + return L.join('\n') + '\n'; +} + +function helperSource(): string { + const L: string[] = [ + 'export interface ITextEditorViewState { scrollTop: number; cursor: number }', + '', + 'export function applyTextEditorOptions(options: { selection?: number; viewState?: ITextEditorViewState }, editor: { reveal(line: number): void; restore(s: ITextEditorViewState): void }): boolean {', + ' if (options.viewState) {', + ' editor.restore(massageEditorViewState(options.viewState));', + ' return true;', + ' }', + ' if (typeof options.selection === "number") {', + ' editor.reveal(options.selection);', + ' return true;', + ' }', + ' return false;', + '}', + '', + 'function massageEditorViewState(state: ITextEditorViewState): ITextEditorViewState {', + ' return { scrollTop: Math.max(0, state.scrollTop), cursor: Math.max(0, state.cursor) };', + '}', + ]; + for (let i = 0; i < 12; i++) { + L.push( + '', + `export function restoreEditorGroup${i}(state: ITextEditorViewState): number {`, + ` const offset = state.scrollTop * ${i + 2} + state.cursor;`, + ` return offset > ${100 * (i + 1)} ? offset - ${i} : offset + ${i};`, + '}', + ); + } + return L.join('\n') + '\n'; +} + +let dir: string; +let cg: CodeGraph; + +/** One explore call plus the CG-4 diagnostic for it. */ +async function explore(query: string): Promise<{ text: string; report: ExploreDiagnosticReport }> { + const sidecar = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-same-basename-diag-')), 'r.jsonl'); + const previous = process.env.CODEGRAPH_EXPLORE_DEBUG; + process.env.CODEGRAPH_EXPLORE_DEBUG = sidecar; + let text = ''; + try { + const res = await new ToolHandler(cg).execute('codegraph_explore', { query }); + text = res.content?.[0]?.text ?? ''; + } finally { + if (previous === undefined) delete process.env.CODEGRAPH_EXPLORE_DEBUG; + else process.env.CODEGRAPH_EXPLORE_DEBUG = previous; + } + const lines = fs.readFileSync(sidecar, 'utf-8').trim().split('\n').filter(Boolean); + fs.rmSync(path.dirname(sidecar), { recursive: true, force: true }); + return { text, report: JSON.parse(lines[lines.length - 1]!) as ExploreDiagnosticReport }; +} + +const fileRecord = (report: ExploreDiagnosticReport, file: string) => + report.files.find((f) => f.path === file); + +/** The response renders a source section for `file`. */ +const hasSection = (response: string, file: string): boolean => + response.includes('**`' + file + '`'); + +/** Every `\t` line number the response sent for `file`. */ +function renderedLines(response: string, file: string): Set { + const out = new Set(); + let current: string | null = null; + let inFence = false; + for (const line of response.split('\n')) { + const header = !inFence ? line.match(/^\*\*`([^`]+)`\*\*/) : null; + if (header) { current = header[1]!; continue; } + if (line.startsWith('```')) { inFence = !inFence; continue; } + const m = inFence && current === file ? line.match(/^(\d+)\t/) : null; + if (m) out.add(Number(m[1])); + } + return out; +} + +/** How many of the registry's named definitions the response sent whole. */ +function completeNamedBodies(response: string): number { + const sent = renderedLines(response, REGISTRY); + return cg.getNodesInFile(REGISTRY) + .filter((n) => NAMED.includes(n.name) && n.kind !== 'import') + .filter((n) => { + for (let l = n.startLine; l <= n.endLine; l++) if (!sent.has(l)) return false; + return true; + }).length; +} + +beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-same-basename-')); + for (const [file, source] of [[REGISTRY, registrySource()], [HELPER, helperSource()]] as const) { + fs.mkdirSync(path.join(dir, path.dirname(file)), { recursive: true }); + fs.writeFileSync(path.join(dir, file), source); + } + cg = CodeGraph.initSync(dir); + await cg.indexAll(); +}, 180_000); + +afterAll(() => { + cg?.destroy(); + if (dir && fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('fixture shape — if this rots, the gates below mean nothing', () => { + it('the named symbols live in the registry and nowhere in the helper', () => { + const registry = cg.getNodesInFile(REGISTRY).map((n) => n.name); + const helper = cg.getNodesInFile(HELPER).map((n) => n.name); + for (const name of NAMED) { + expect(registry).toContain(name); + expect(helper).not.toContain(name); + } + // Two `clampedInt`s — the function and the static method — so four bodies. + expect(cg.getNodesInFile(REGISTRY).filter((n) => NAMED.includes(n.name))).toHaveLength(4); + }); +}); + +describe('a basename two directories share', () => { + it('pins only the file defining the named symbols, and names the one it set aside', async () => { + const { text, report } = await explore(`editorOptions.ts ${NAMED.join(' ')}`); + expect(fileRecord(report, REGISTRY)?.pinned).toBe(true); + expect(fileRecord(report, HELPER)?.pinned ?? false).toBe(false); + expect(hasSection(text, HELPER)).toBe(false); + expect(text).toContain('1 file pinned from the query.'); + expect(text).toContain(`Not pinned: \`${HELPER}\`, which defines none of the named symbols.`); + }); + + it('gives the named file the room it gets when the query leaves the path out', async () => { + const pinned = await explore(`editorOptions.ts ${NAMED.join(' ')}`); + const unpinned = await explore(NAMED.join(' ')); + const pinnedAllowance = fileRecord(pinned.report, REGISTRY)?.allowance ?? 0; + const unpinnedAllowance = fileRecord(unpinned.report, REGISTRY)?.allowance ?? 0; + expect(unpinnedAllowance).toBeGreaterThan(0); + // Pre-fix the two pins split the reservation: half the room, or less. + expect(pinnedAllowance).toBeGreaterThanOrEqual(unpinnedAllowance); + expect(completeNamedBodies(pinned.text)).toBeGreaterThanOrEqual(completeNamedBodies(unpinned.text)); + expect(completeNamedBodies(pinned.text)).toBe(4); + }); + + it('pins both when the query names no symbol to choose between them', async () => { + const { text, report } = await explore('editorOptions.ts'); + expect(fileRecord(report, REGISTRY)?.pinned).toBe(true); + expect(fileRecord(report, HELPER)?.pinned).toBe(true); + expect(text).toContain('2 files pinned from the query.'); + expect(text).not.toContain('Not pinned:'); + }); + + it('a path with its directory pins that file alone, with no set-aside note', async () => { + const { text, report } = await explore(`${HELPER} ${NAMED.join(' ')}`); + expect(fileRecord(report, HELPER)?.pinned).toBe(true); + expect(fileRecord(report, REGISTRY)?.pinned ?? false).toBe(false); + expect(text).not.toContain('Not pinned:'); + }); +}); diff --git a/__tests__/query-paths.test.ts b/__tests__/query-paths.test.ts index 956c33c855..7c23a5dcce 100644 --- a/__tests__/query-paths.test.ts +++ b/__tests__/query-paths.test.ts @@ -151,7 +151,7 @@ describe('extractQueryPaths — resolution and stripping', () => { it('passes through untouched when nothing resolves', () => { const q = 'plain prose question about scrolling'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [], setAsideMatches: [] }); }); }); @@ -191,13 +191,13 @@ describe('extractQueryPaths — extension-less kebab basenames', () => { it('leaves kebab prose that names no indexed file untouched — and unreported', () => { const q = 'how does cross-call dedup make explore non-blocking'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [], setAsideMatches: [] }); }); it('leaves a stem shared by too many files alone — one hot name must not pin half the repo', () => { const q = 'refactor the user-profile rendering'; const out = extractQueryPaths(q, INDEX); - expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [] }); + expect(out).toEqual({ strippedQuery: q, pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [], setAsideMatches: [] }); }); it('pins all files sharing a stem when within the ambiguity budget', () => { @@ -371,3 +371,134 @@ describe('extractQueryPaths — line anchors', () => { expect(twoFiles.lineAnchors).toEqual([]); }); }); + +/** + * A span that matches several files, narrowed by the symbols the query names. + * + * vscode has two `editorOptions.ts`; `editorOptions.ts clampedInt + * cursorStyleToString` pinned both, and the pins split the reservation, so the + * file holding the named functions got half the room it got WITHOUT the path. + * The caller injects `symbolFiles` (the index lookup; this module stays + * DB-free) and the matches defining a named symbol are the ones pinned. + */ +describe('extractQueryPaths — same-named files, narrowed by named symbols', () => { + const REGISTRY = 'src/editor/config/editorOptions.ts'; + const HELPER = 'src/workbench/editor/editorOptions.ts'; + const SHARED = [...INDEX, REGISTRY, HELPER]; + /** Records every symbol the lookup is asked about. */ + const lookup = (defs: Record) => { + const asked: string[] = []; + return { + asked, + symbolFiles: (symbol: string) => { asked.push(symbol); return defs[symbol] ?? []; }, + }; + }; + + it('pins only the match defining a named symbol and reports the one set aside', () => { + const l = lookup({ clampedInt: [REGISTRY], cursorStyleToString: [REGISTRY] }); + const out = extractQueryPaths( + 'editorOptions.ts clampedInt cursorStyleToString', SHARED, { symbolFiles: l.symbolFiles }, + ); + expect(out.pinnedFiles).toEqual([REGISTRY]); + expect(out.setAsideMatches).toEqual([{ span: 'editorOptions.ts', files: [HELPER] }]); + expect(out.strippedQuery).toBe('clampedInt cursorStyleToString'); + // The span itself is a file name, not a symbol to look up. + expect(l.asked).toEqual(['clampedInt', 'cursorStyleToString']); + }); + + it('narrows a shared kebab stem the same way', () => { + const l = lookup({ GenericModalFooter: ['src/y/generic-modal.tsx'] }); + const out = extractQueryPaths('generic-modal GenericModalFooter', SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual(['src/y/generic-modal.tsx']); + expect(out.setAsideMatches).toEqual([{ span: 'generic-modal', files: ['src/x/generic-modal.tsx'] }]); + }); + + it('pins every match when none defines a named symbol, or every one does', () => { + const none = lookup({ clampedInt: ['src/lib/chat-manager.ts'] }); + const noneOut = extractQueryPaths('editorOptions.ts clampedInt', SHARED, { symbolFiles: none.symbolFiles }); + expect(noneOut.pinnedFiles).toEqual([REGISTRY, HELPER]); + expect(noneOut.setAsideMatches).toEqual([]); + // Consulted and found nothing — not skipped. + expect(none.asked).toEqual(['clampedInt']); + + const both = lookup({ EditorOptions: [HELPER, REGISTRY] }); + const bothOut = extractQueryPaths('editorOptions.ts EditorOptions', SHARED, { symbolFiles: both.symbolFiles }); + expect(bothOut.pinnedFiles).toEqual([REGISTRY, HELPER]); + expect(bothOut.setAsideMatches).toEqual([]); + }); + + it('does not let a bare English word pick a file', () => { + // `options` and `close` are words as much as names: a file defining one + // says nothing about which same-named file the agent meant. + const l = lookup({ options: [HELPER], close: ['src/x/generic-modal.tsx'] }); + expect(extractQueryPaths('editorOptions.ts options', SHARED, { symbolFiles: l.symbolFiles }).pinnedFiles) + .toEqual([REGISTRY, HELPER]); + expect(extractQueryPaths('generic-modal close behavior', SHARED, { symbolFiles: l.symbolFiles }).pinnedFiles) + .toEqual(['src/x/generic-modal.tsx', 'src/y/generic-modal.tsx']); + expect(l.asked).toEqual([]); + }); + + it('looks a qualified token up as written, and reads `name()` as the name', () => { + const l = lookup({ 'EditorIntOption.clampedInt': [REGISTRY], cursorStyleFromString: [REGISTRY] }); + const out = extractQueryPaths( + 'editorOptions.ts EditorIntOption.clampedInt `cursorStyleFromString()`', SHARED, + { symbolFiles: l.symbolFiles }, + ); + expect(out.pinnedFiles).toEqual([REGISTRY]); + expect(l.asked).toEqual(['EditorIntOption.clampedInt', 'cursorStyleFromString']); + }); + + it('resolves a basename shared past the ambiguity budget when one match defines the named symbol', () => { + const l = lookup({ UserProfileCard: ['src/c/user-profile.tsx'] }); + const out = extractQueryPaths('user-profile UserProfileCard', SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual(['src/c/user-profile.tsx']); + expect(out.setAsideMatches).toEqual([{ + span: 'user-profile', + files: ['src/a/user-profile.tsx', 'src/b/user-profile.tsx', 'src/d/user-profile.tsx'], + }]); + + const dotted = extractQueryPaths('+page.svelte ChatScroller', SHARED, { + symbolFiles: lookup({ ChatScroller: ['src/routes/(protected)/chat-window/+page.svelte'] }).symbolFiles, + }); + expect(dotted.pinnedFiles).toEqual(['src/routes/(protected)/chat-window/+page.svelte']); + expect(dotted.unresolvedPathSpans).toEqual([]); + }); + + it('keeps an over-budget span ambiguous when the symbols do not narrow it into budget', () => { + const l = lookup({ PageHeader: [ + 'src/routes/m/projects/[id]/runs/[runId]/+page.svelte', + 'src/routes/m/projects/[id]/chat/[scope]/+page.svelte', + 'src/routes/m/projects/[id]/+page.svelte', + 'src/routes/(protected)/chat-window/+page.svelte', + ] }); + const out = extractQueryPaths('why do all +page.svelte PageHeader files flash', SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual([]); + expect(out.unresolvedPathSpans).toEqual(['+page.svelte']); + expect(out.setAsideMatches).toEqual([]); + + const kebab = extractQueryPaths('refactor the user-profile rendering', SHARED, { symbolFiles: l.symbolFiles }); + expect(kebab.pinnedFiles).toEqual([]); + expect(kebab.strippedQuery).toBe('refactor the user-profile rendering'); + }); + + it('binds a line anchor once the span narrows to one file', () => { + const l = lookup({ clampedInt: [REGISTRY] }); + const out = extractQueryPaths('editorOptions.ts:1291 clampedInt', SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual([REGISTRY]); + expect(out.lineAnchors).toEqual([{ file: REGISTRY, start: 1291, end: 1291 }]); + }); + + it('does not report a file the query also named by its full path', () => { + const l = lookup({ clampedInt: [REGISTRY] }); + const out = extractQueryPaths(`editorOptions.ts clampedInt ${HELPER}`, SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual([REGISTRY, HELPER]); + expect(out.setAsideMatches).toEqual([]); + }); + + it('never looks anything up for a span that matches one file', () => { + const l = lookup({ ChatManager: ['src/lib/chat-manager.ts'] }); + const out = extractQueryPaths('chat-manager.ts ChatManager', SHARED, { symbolFiles: l.symbolFiles }); + expect(out.pinnedFiles).toEqual(['src/lib/chat-manager.ts']); + expect(l.asked).toEqual([]); + }); +}); diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 0455afbed8..f2d3980256 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -33,8 +33,13 @@ import { import type { PendingFile } from '../sync'; import type { Node, Edge, SearchResult, Subgraph, NodeKind } from '../types'; import { isTestFile, normalizeNameToken } from '../search/query-utils'; -import { groupDefinitions, lastQualifierPart, matchesSymbol } from '../graph/symbol-lookup'; -import { extractQueryPaths, queryMightContainPaths, type QueryLineAnchor } from '../search/query-paths'; +import { groupDefinitions, isQualifiedSymbol, lastQualifierPart, matchesSymbol } from '../graph/symbol-lookup'; +import { + extractQueryPaths, + queryMightContainPaths, + type QueryLineAnchor, + type QuerySetAsideMatch, +} from '../search/query-paths'; import { existsSync, readFileSync, @@ -196,6 +201,29 @@ function pathIsProjectFile(projectRoot: string, relPath: string): boolean { } } +/** Kinds that name a thing without defining it in the file that holds them. */ +const NOT_A_DEFINITION = new Set(['file', 'import', 'export', 'parameter']); + +/** + * Which indexed files define a symbol spelled like this query token? + * + * The `symbolFiles` lookup `extractQueryPaths` takes, split out for the same + * reason as `pathIsProjectFile`: that module stays DB-free. Exact names only — + * and the shared matcher for a qualified token (`SQLCompiler.as_sql`) — so it + * agrees with what explore's named-symbol seeding resolves. Lookup errors + * answer "none", which leaves the span's pins as they were. + */ +function filesDefiningSymbol(cg: CodeGraph, symbol: string): string[] { + try { + const nodes = isQualifiedSymbol(symbol) + ? cg.getNodesByName(lastQualifierPart(symbol)).filter((n) => matchesSymbol(n, symbol)) + : cg.getNodesByName(symbol); + return nodes.filter((n) => !NOT_A_DEFINITION.has(n.kind)).map((n) => n.filePath); + } catch { + return []; + } +} + /** * Calculate the recommended number of codegraph_explore calls based on project size. * Larger codebases need more exploration calls to cover their surface area, @@ -3709,22 +3737,36 @@ export class ToolHandler { let pinnedFiles: string[] = []; let unresolvedPathSpans: string[] = []; let lineAnchors: QueryLineAnchor[] = []; + let setAsideMatches: QuerySetAsideMatch[] = []; let matchQuery = query; if (queryMightContainPaths(rawQuery)) { try { const extraction = extractQueryPaths( rawQuery, cg.getFiles().map((f) => f.path), - { maxPins: maxFiles, existsOnDisk: (rel) => pathIsProjectFile(projectRoot, rel) }, + { + maxPins: maxFiles, + existsOnDisk: (rel) => pathIsProjectFile(projectRoot, rel), + symbolFiles: (symbol) => filesDefiningSymbol(cg, symbol), + }, ); if (extraction.pinnedFiles.length > 0 || extraction.unresolvedPathSpans.length > 0) { pinnedFiles = extraction.pinnedFiles; unresolvedPathSpans = extraction.unresolvedPathSpans; lineAnchors = extraction.lineAnchors; + setAsideMatches = extraction.setAsideMatches; matchQuery = normalizeQuerySpelling(extraction.strippedQuery); } } catch { /* path pinning must never fail an explore call */ } } + // A same-named file the span did not pin is named in the summary line, so + // an agent that did mean it sees where it went instead of a silent drop. + const setAsideNote = setAsideMatches.map((s) => { + const which = s.files.length <= 2 + ? s.files.map((f) => `\`${f}\``).join(', ') + : `${s.files.length} other \`${s.span}\` files`; + return ` Not pinned: ${which}, which define${s.files.length === 1 ? 's' : ''} none of the named symbols.`; + }).join(''); const pinnedSet = new Set(pinnedFiles); const pinnedOrder = new Map(pinnedFiles.map((p, i) => [p, i])); @@ -4907,7 +4949,8 @@ export class ToolHandler { + (pinnedFiles.length > 0 ? ` ${plural(pinnedFiles.length, 'file')} pinned from the query.`.length : 0) + (unresolvedPathSpans.length > 0 ? ` No indexed file uniquely matches ${unresolvedPathSpans.map((sp) => `\`${sp}\``).join(', ')}.`.length - : 0); + : 0) + + setAsideNote.length; const epilogueFloor = EPILOGUE_LOST_NOTE.length + 2 + cliffPointerFloor + summaryReserve; // Absolute stop for the render loop. Reservations already fit the envelope, so // this only catches their bounded overshoot (the whole-file grace, an oversize @@ -7071,6 +7114,7 @@ export class ToolHandler { if (unresolvedPathSpans.length > 0) { summaryLine += ` No indexed file uniquely matches ${unresolvedPathSpans.map((s) => `\`${s}\``).join(', ')}.`; } + summaryLine += setAsideNote; finalText = finalText.replace(SUMMARY_SENTINEL, summaryLine); // Emit the allocation diagnostic from the FINAL text, so per-file bytes and diff --git a/src/search/query-paths.ts b/src/search/query-paths.ts index 302c442a3b..072e133cb2 100644 --- a/src/search/query-paths.ts +++ b/src/search/query-paths.ts @@ -54,6 +54,18 @@ export interface QueryPathExtraction { * across two candidate files. */ lineAnchors: QueryLineAnchor[]; + /** + * Files a span matched but did NOT pin: the span named several files, some + * of which define a symbol the query also names, and these define none. + * Surfaced so a set-aside file is visible, not silently dropped. + */ + setAsideMatches: QuerySetAsideMatch[]; +} + +export interface QuerySetAsideMatch { + /** The span as the query wrote it, normalized (`editorOptions.ts`). */ + span: string; + files: string[]; } export interface QueryLineAnchor { @@ -100,6 +112,43 @@ const KEBAB_BASENAME = /^[A-Za-z0-9]+(?:-[A-Za-z0-9]+)+$/; /** A basename's last dot-extension, same shape DOTTED_BASENAME accepts. */ const LAST_EXTENSION = /\.[A-Za-z][A-Za-z0-9]{0,7}$/; +/** + * A query token that names a symbol, in the shape explore's named-symbol + * seeder reads: `clampedInt`, `SQLCompiler.as_sql`, `Engine::ServeHTTP`. + */ +const SYMBOL_TOKEN = /^[A-Za-z_$][\w$]*(?:(?:::|\.)[\w$]+)*$/; +/** Symbol tokens consulted per query — the seeder's cap. */ +const MAX_SYMBOL_TOKENS = 16; + +/** + * The seeder's NL-stopword test: camelCase, PascalCase, snake_case, `$` and + * qualified tokens are unmistakably code. A bare lowercase word (`options`, + * `render`) is also English, and a file defining one proves nothing about + * which same-named file the agent meant. + */ +function isPreciseSymbolToken(t: string): boolean { + return /[._$]|::/.test(t) || /[a-z][A-Z]/.test(t) || /^[A-Z]/.test(t); +} + +/** + * The precise symbol tokens a query names, excluding the files it names + * (`editorOptions.ts` is identifier-shaped too). Split on the same brackets + * the seeder splits on, so `clampedInt()` still counts. + */ +function querySymbolTokens(tokens: readonly string[], indexedBasenames: ReadonlySet): string[] { + const out = new Set(); + for (const raw of tokens) { + for (const part of raw.split(/[,()[\]{}]+/)) { + const t = part.replace(/^['"`<]+/, '').replace(/['"`>.,;:!?]+$/, ''); + if (t.length < 3 || !SYMBOL_TOKEN.test(t) || !isPreciseSymbolToken(t)) continue; + if (indexedBasenames.has(t.toLowerCase())) continue; + out.add(t); + if (out.size >= MAX_SYMBOL_TOKENS) return [...out]; + } + } + return [...out]; +} + /** * Lowercased basename stems of the hyphen-named indexed files, stem → paths. * A stem drops only the LAST extension (`a-b.module.scss` → `a-b.module`), so @@ -256,6 +305,16 @@ export function extractQueryPaths( * Must not throw — the caller absorbs fs errors and returns false. */ existsOnDisk?: (relPath: string) => boolean; + /** + * Which indexed files define a symbol spelled like this query token + * (`clampedInt`, `SQLCompiler.as_sql`)? Optional, injected by the caller + * for the same reason as `existsOnDisk`: the answer lives in the index, + * and this module stays DB-free. Consulted only when a span matches + * several files (a basename two directories share): the matches that + * define a symbol the query also names are pinned, the rest set aside. + * Must not throw — the caller absorbs lookup errors and returns nothing. + */ + symbolFiles?: (symbol: string) => Iterable; } = {}, ): QueryPathExtraction { const maxPins = Math.max(1, opts.maxPins ?? 8); @@ -266,6 +325,7 @@ export function extractQueryPaths( pinnedFiles: [], unresolvedPathSpans: [], lineAnchors: [], + setAsideMatches: [], }; if (!query.trim() || indexedPaths.length === 0) return passthrough; @@ -280,10 +340,60 @@ export function extractQueryPaths( const pinnedSeen = new Set(); const unresolved: string[] = []; const anchors: QueryLineAnchor[] = []; + const setAside: QuerySetAsideMatch[] = []; /** Token index → the ONE file it pinned, for binding prose line ranges. */ const singleFileAt = new Map(); let candidatesExamined = 0; + // A bare basename two directories share (`editorOptions.ts` is both vscode's + // editor option registry and a workbench helper) used to pin EVERY match, + // and the matches split the pinned reservation — so a query that also named + // three functions in one of them got half the room for them it would have + // got without the path. When the query names symbols, the file it means is + // the one defining them: keep the matches that define at least one, set the + // rest aside. When no match defines one, or every match does, the symbols + // pick nothing out and the span resolves exactly as before. + // + // The same test lets a basename shared by more files than the ambiguity + // budget (django's 40 `models.py`) resolve after all — to the one defining + // the named class — so a span is collected past the budget only while + // narrowing is possible, and still reports as ambiguous if it doesn't + // narrow into the budget. + let symbolTokens: string[] | null = null; + const definersOf = new Map>(); + const narrowBySymbols = (matches: readonly string[]): string[] | null => { + if (!opts.symbolFiles || matches.length < 2) return null; + if (symbolTokens === null) { + const basenames = new Set(); + for (const p of indexedPaths) { + basenames.add(p.slice(Math.max(p.lastIndexOf('/'), p.lastIndexOf('\\')) + 1).toLowerCase()); + } + symbolTokens = querySymbolTokens(tokens, basenames); + } + const candidates = new Set(matches); + const definers = new Set(); + for (const t of symbolTokens) { + let files = definersOf.get(t); + if (!files) { + files = new Set(opts.symbolFiles(t)); + definersOf.set(t, files); + } + for (const f of files) if (candidates.has(f)) definers.add(f); + } + if (definers.size === 0 || definers.size === candidates.size) return null; + return matches.filter((m) => definers.has(m)); + }; + /** Narrow a span's matches, recording what was set aside; null = over budget. */ + const pinnableMatches = (span: string, matches: string[]): string[] | null => { + const narrowed = matches.length > 1 ? narrowBySymbols(matches) : null; + if (narrowed && narrowed.length <= maxMatchesPerSpan) { + setAside.push({ span, files: matches.filter((m) => !narrowed.includes(m)) }); + return narrowed; + } + return matches.length > maxMatchesPerSpan ? null : matches; + }; + const collectLimit = opts.symbolFiles ? Number.POSITIVE_INFINITY : maxMatchesPerSpan; + for (let i = 0; i < tokens.length; i++) { if (pinned.length >= maxPins) break; if (candidatesExamined >= MAX_CANDIDATE_SPANS) break; @@ -296,9 +406,10 @@ export function extractQueryPaths( if (!normalized) continue; candidatesExamined++; - const { matches, ambiguous } = resolveSpan( - normalized.toLowerCase(), lowerToOriginal, maxMatchesPerSpan, - ); + const resolved = resolveSpan(normalized.toLowerCase(), lowerToOriginal, collectLimit); + const pinnable = resolved.matches.length > 0 ? pinnableMatches(normalized, resolved.matches) : null; + const matches = pinnable ?? []; + const ambiguous = resolved.ambiguous || (resolved.matches.length > 0 && pinnable === null); if (matches.length > 0) { consumed.add(i); for (const m of matches) { @@ -352,8 +463,9 @@ export function extractQueryPaths( const { path: stripped, lines } = stripWrapping(tokens[i]!); if (stripped.length < 4 || !KEBAB_BASENAME.test(stripped)) continue; basenameStems ??= buildBasenameStems(indexedPaths); - const matches = basenameStems.get(stripped.toLowerCase()); - if (!matches || matches.length > maxMatchesPerSpan) continue; + const stemMatches = basenameStems.get(stripped.toLowerCase()); + const matches = stemMatches ? pinnableMatches(stripped, stemMatches) : null; + if (!matches) continue; consumed.add(i); for (const m of matches) { if (pinnedSeen.has(m) || pinned.length >= maxPins) continue; @@ -419,5 +531,10 @@ export function extractQueryPaths( seenAnchor.add(key); return true; }), + // A file another span pinned outright (the agent also wrote its full path) + // was not set aside after all. + setAsideMatches: setAside + .map((s) => ({ span: s.span, files: s.files.filter((f) => !pinnedSeen.has(f)) })) + .filter((s) => s.files.length > 0), }; } From 003305f1e7edbb2f3b966f1631585aab02cef7bf Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 15:54:00 +0000 Subject: [PATCH 007/363] fix(explore): claim complete source only for sections that are complete (#2077) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(explore): claim complete source only for sections that are complete On the tiers with a completeness signal (>= 500 indexed files) every codegraph_explore response ended with "Complete source for N files is included above — do NOT re-read them", whatever the render had cut. A windowed spine method, a shrunk or dropped cluster and the per-symbol view all went out under that line; the oversize-spine window never even set the trim flag the small tiers key off, which is how a 62-line slice of vscode's 968-line rpcProtocol.ts was called complete. The line also said "Reserve Read for a single specific line range", which explore output must never say. Completeness is now measured at emission from the ranges actually sent (plus back-referenced spans) against the symbols each section set out to deliver. A trimmed response keeps the verbatim / already-Read guarantee, names the trimmed files and the most relevant elided symbols, and steers to another codegraph_explore. The note is offered from most to least specific and fitted to the leftover room (CG-26); a trimmed note yields the pointer list's first entry. The lost-pointer, cut and truncation notes drop "complete" the same way, at no greater length, so the epilogue floor is unchanged. The small tiers' trimmed note now also fires for the cuts the flag missed. Co-Authored-By: Claude Opus 5.5 * fix(explore): offer an elided method as Owner.member in the completeness note The note offered `as_sql` for django's elided SQLCompiler.as_sql — one of 110 definitions — so agents appended a path and line to disambiguate it, and that query shape returns none of the body. `SQLCompiler.as_sql` alone returns most of it. Methods are now offered as `Owner.member` from the indexed qualified name; everything else keeps its bare name. Co-Authored-By: Claude Opus 5.5 * refactor(explore): fit the completeness note and pointer list in one tested function A trimmed note's optional detail (the files, the elided names) never costs a pointer entry the least specific note would have left. The fit moves out of handleExplore into fitExploreEpilogue so both rules are pinned by tests; the second-try branch it had could never fire and is gone. Co-Authored-By: Claude Opus 5.5 * test(explore): completeness note wording after #2068 flags the spine window Co-Authored-By: Claude Opus 5.5 * docs(explore): record the completeness-note replay and agent A/B Co-Authored-By: Claude Opus 5.5 * fix(explore): address review of the completeness note - Judge completeness AFTER the dedup fold: a remainder too small to fence is sent in neither call, so a section with one is trimmed, not complete. The whole-file render now declares its symbols too (clamped to the lines it prints; the file node is not one), so a fold there registers. - Label trimmed files unique among every path the response can name, the pointer list included. - "1 file", and no "0 files" when every section was held from an earlier call. - The lost-pointer, cut and truncation notes are one exported table, EXPLORE_FALLBACK_NOTES, with the length rule tested; the lost note picks its wording from trimmedShown instead of re-joining the response. - The design doc's follow-up list marks the overclaim fixed. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/explore-completeness-note.test.ts | 408 +++++++++++++++++++ docs/benchmarks/explore-completeness-note.md | 132 ++++++ docs/design/explore-budget-allocation.md | 4 +- src/mcp/tools.ts | 361 +++++++++++++--- 5 files changed, 858 insertions(+), 48 deletions(-) create mode 100644 __tests__/explore-completeness-note.test.ts create mode 100644 docs/benchmarks/explore-completeness-note.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 6990630451..19ed81c182 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -215,6 +215,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph_explore` now finds a file when the query is its exact Chinese filename, with or without the extension. Thanks @Syh1906. (#1372) - `codegraph_explore` now returns the methods and types you name from a large file instead of a few lines around one call. When a file's symbols sat close together, and whenever you named the file by path, the answer could show only its first lines and a window on one call site, leave out the bodies you asked for, and still call the file complete, so your agent read it back. A genuinely long function on the call path is still shortened to its signature and the call that matters, and the answer now says when it did. - When your question names symbols concentrated in one file and the answer has room to spare, `codegraph_explore` now uses that room for the file you named instead of stopping at a fixed share of the response, so every body you asked for fits. Other files in the answer keep the space they were given. +- `codegraph_explore` no longer tells your agent an answer's source is complete when part of it was shortened to fit. On larger projects it now names the files it trimmed and the functions it left out, so your agent asks for those by name instead of reading the file, and it no longer suggests reading a file as a fallback. - Java, Kotlin and Scala packages named `build` under a source root are indexed again, while real build-output directories stay excluded. Thanks @CmmVoid for the report and @danusha2345. (#1642) - Files over the 1 MB source limit, such as a package archive an import points at, are now rejected before they are read during resolution, removing a multi-gigabyte memory spike at startup. Thanks @hcg1023 for the report and @danusha2345. (#1553) - `codegraph sync` now reports that the index is busy and exits non-zero when another process holds the write lock, instead of printing "Already up to date". Thanks @inth3shadows. (#1361) diff --git a/__tests__/explore-completeness-note.test.ts b/__tests__/explore-completeness-note.test.ts new file mode 100644 index 0000000000..b3537064d7 --- /dev/null +++ b/__tests__/explore-completeness-note.test.ts @@ -0,0 +1,408 @@ +/** + * The completeness note at the end of a codegraph_explore response claims + * "complete" only for sections that are. + * + * On the tiers with `includeCompletenessSignal` (>= 500 indexed files) every + * response used to end with "Complete source for N files is included above — + * do NOT re-read them", whatever the render had cut. That is how a 62-line + * slice of vscode's 968-line `rpcProtocol.ts` was presented as complete: the + * oversize-spine window elided most of the file, and until #2068 it did not + * even set the trim flag the small tiers' note keys off. The same line also + * said "Reserve Read for a single specific line range", and explore output must + * never tell the agent to Read (AGENTS.md). + * + * The fixture reproduces that shape: a flow whose spine runs through one long + * method, which the render windows to its head plus the next-hop call site. + * The large-tier test fails without the measured check; the small-tier one + * pins that the two tiers agree. + */ +import { describe, it, expect, vi, beforeAll, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src/index'; +import { ExploreSessionState } from '../src/mcp/explore-session-state'; +import { + EXPLORE_FALLBACK_NOTES, + ToolHandler, + elidedWantedSpans, + exploreCompletenessNotes, + fitExploreEpilogue, + shortestUniqueSuffixes, + type ExploreWantedSpan, +} from '../src/mcp/tools'; + +const span = ( + name: string, start: number, end: number, importance = 1, spine = false, kind = 'method', +): ExploreWantedSpan => ({ name, kind, start, end, importance, spine }); + +/** The note this change replaced, for the size bound. */ +const OLD_NOTE = '> **Complete source for 8 files is included above — do NOT re-read them.** If your question also needs files/symbols listed under "Not shown above" (or any area this call didn\'t cover), make ANOTHER codegraph_explore targeting those names — it returns the same source with line numbers and is cheaper and more complete than reading. Reserve Read for a single specific line range explore can\'t surface.'; + +/** + * Anything that offers Read as a way forward. "treat it as already Read" is the + * guarantee and "do not Read a file shown here" a prohibition — neither is an offer. + */ +const OFFERS_READ = /Reserve Read|use Read|Read for |fall back to Read/; +/** What follows the last source fence: the epilogue this change rewrote. */ +const epilogueOf = (text: string): string => text.slice(text.lastIndexOf('```') + 3); + +describe('elidedWantedSpans — judged on what was sent', () => { + it('a span inside what was delivered is complete; one past its edge is not', () => { + const elided = elidedWantedSpans( + [span('inside', 10, 20), span('straddles', 25, 60), span('absent', 90, 95)], + [{ start: 1, end: 40 }], + ); + expect(elided.map((e) => e.name)).toEqual(['straddles', 'absent']); + }); + + it('counts a back-referenced span, and joins adjacent ranges', () => { + // Lines 1–30 sent now, 31–50 held from an earlier call: one continuous copy. + const elided = elidedWantedSpans([span('whole', 5, 45)], [{ start: 1, end: 30 }, { start: 31, end: 50 }]); + expect(elided).toEqual([]); + }); + + it('orders the spine first, then importance, then source order', () => { + const elided = elidedWantedSpans( + [span('late', 80, 81, 9), span('peripheral', 5, 6, 1), span('flow', 50, 60, 3, true), span('early', 40, 41, 9)], + [], + ); + expect(elided.map((e) => e.name)).toEqual(['flow', 'early', 'late', 'peripheral']); + }); + + it('skips spans with no usable line range', () => { + expect(elidedWantedSpans([span('zero', 0, 0), span('inverted', 9, 3)], [])).toEqual([]); + }); +}); + +describe('shortestUniqueSuffixes', () => { + it('uses the basename unless another path ends with it', () => { + const labels = shortestUniqueSuffixes([ + 'src/vs/workbench/api/common/extHostExtensionService.ts', + 'src/vs/workbench/api/node/extHostExtensionService.ts', + 'src/vs/workbench/services/extensions/common/rpcProtocol.ts', + ]); + expect(labels.get('src/vs/workbench/api/common/extHostExtensionService.ts')).toBe('common/extHostExtensionService.ts'); + expect(labels.get('src/vs/workbench/api/node/extHostExtensionService.ts')).toBe('node/extHostExtensionService.ts'); + expect(labels.get('src/vs/workbench/services/extensions/common/rpcProtocol.ts')).toBe('rpcProtocol.ts'); + }); + + it('falls back to the whole path when one path is a suffix of another', () => { + const labels = shortestUniqueSuffixes(['a/b.ts', 'x/a/b.ts']); + expect(labels.get('a/b.ts')).toBe('a/b.ts'); + expect(labels.get('x/a/b.ts')).toBe('x/a/b.ts'); + }); +}); + +describe('exploreCompletenessNotes', () => { + it('claims complete source only when nothing was trimmed, and never offers Read', () => { + const notes = exploreCompletenessNotes(4, [], ['a.ts', 'b.ts']); + expect(notes).toHaveLength(1); + expect(notes[0]).toContain('Complete source for 4 files'); + expect(notes[0]).not.toMatch(OFFERS_READ); + }); + + it('a trimmed response keeps the already-Read guarantee, names the files and the elided symbols', () => { + const trimmed = [ + { + filePath: 'src/vs/workbench/services/extensions/common/rpcProtocol.ts', + elided: [ + span('_receiveOneMessage', 280, 357, 9, true), + span('RPCProtocol', 200, 900, 9, false, 'class'), // a container is no follow-up target + span('serializeRequest', 700, 720, 9), + span('helperNobodyAskedFor', 10, 12, 1), + ], + }, + ]; + const notes = exploreCompletenessNotes(3, trimmed, ['src/vs/workbench/services/extensions/common/rpcProtocol.ts', 'src/a.ts']); + for (const note of notes) { + expect(note).not.toContain('Complete source'); + expect(note).toContain('Verbatim source for 3 files'); + expect(note).toContain('treat it as already Read'); + expect(note).toContain('codegraph_explore'); + expect(note).not.toMatch(OFFERS_READ); + } + expect(notes[0]).toContain('`rpcProtocol.ts`'); + expect(notes[0]).toContain('`_receiveOneMessage`, `serializeRequest`'); + expect(notes[0]).not.toContain('`RPCProtocol`'); + expect(notes[0]).not.toContain('helperNobodyAskedFor'); + }); + + it('counts files in words that read right for one and for none', () => { + const t = [{ filePath: 'a.ts', elided: [span('f', 1, 9, 9)] }]; + expect(exploreCompletenessNotes(1, [], ['a.ts'])[0]).toContain('Complete source for 1 file is included'); + expect(exploreCompletenessNotes(1, t, ['a.ts'])[0]).toContain('Verbatim source for 1 file is included'); + // Every section held from an earlier call: no "0 files" beside a note about what was shown. + for (const note of exploreCompletenessNotes(0, t, ['a.ts'])) { + expect(note).not.toMatch(/\b0 files?\b/); + expect(note).toContain('Verbatim source for these files'); + } + }); + + it('labels a trimmed file apart from a same-named file the pointer list names', () => { + const trimmed = [{ filePath: 'src/common/rpcProtocol.ts', elided: [span('f', 1, 9, 9)] }]; + const [note] = exploreCompletenessNotes(1, trimmed, ['src/common/rpcProtocol.ts', 'src/node/rpcProtocol.ts']); + expect(note).toContain('`common/rpcProtocol.ts`'); + }); + + it('offers an elided method as Owner.member, so an overloaded name reaches the one that was cut', () => { + const q = (s: ExploreWantedSpan, qualifiedName: string): ExploreWantedSpan => ({ ...s, qualifiedName }); + const trimmed = [{ + filePath: 'django/db/models/sql/compiler.py', + elided: [ + q(span('as_sql', 776, 1003, 9, true), 'SQLCompiler::as_sql'), + q(span('PLUGIN_ID', 5, 5, 9, false, 'method'), 'org.lamport.tla::HelpActivator::PLUGIN_ID'), + q(span('inner', 40, 44, 9, false, 'function'), 'Outer::run::inner'), // a local function keeps its bare name + q(span('odd', 50, 52, 9), 'src/a.py::odd'), // a path is no owner + ], + }]; + const [note] = exploreCompletenessNotes(1, trimmed, ['django/db/models/sql/compiler.py']); + expect(note).toContain('(e.g. `SQLCompiler.as_sql`, `HelpActivator.PLUGIN_ID`, `inner`, `odd`)'); + }); + + it('offers candidates from most to least specific, the last no longer than the note it replaced', () => { + const trimmed = ['a', 'b', 'c', 'd', 'e'].map((n) => ({ + filePath: `packages/${n}/src/deeply/nested/${n}Service.ts`, + elided: [span(`${n}Handler`, 10, 90, 9)], + })); + const notes = exploreCompletenessNotes(8, trimmed, trimmed.map((t) => t.filePath)); + expect(notes).toHaveLength(3); + for (let i = 1; i < notes.length; i++) expect(notes[i]!.length).toBeLessThan(notes[i - 1]!.length); + // Three files named, the rest counted. + expect(notes[1]).toContain('`aService.ts`, `bService.ts`, `cService.ts` +2 more'); + // The last resort names nothing, so wherever the old note fit, it fits. + expect(notes[2]).not.toContain('Service.ts'); + expect(notes[2]!.length).toBeLessThan(OLD_NOTE.length); + }); +}); + +describe('EXPLORE_FALLBACK_NOTES', () => { + it.each(Object.entries(EXPLORE_FALLBACK_NOTES))('%s: the trimmed wording drops "complete" and is no longer', (_kind, note) => { + // The epilogue floor and the cut note's fit test are sized before the render + // knows which wording it needs. + expect(note.trimmed.length).toBeLessThanOrEqual(note.complete.length); + expect(note.complete).toContain('complete and verbatim'); + expect(note.trimmed).not.toMatch(/\bcomplete\b/); + expect(note.trimmed).toContain('treat it as already Read'); + for (const text of [note.complete, note.trimmed]) { + expect(text).toContain('codegraph_explore'); + expect(text).not.toMatch(OFFERS_READ); + } + }); +}); + +describe('fitExploreEpilogue — the note and the pointer list share what is left', () => { + const cost = (b: readonly string[]) => b.reduce((n, l) => n + l.length + 1, 0); + const entries = ['a', 'b', 'c', 'd', 'e'].map((n) => `- src/${n}/${n.repeat(40)}.ts: ${n}Handler:10, ${n}Helper:40`); + const note = (len: number) => ['', '---', `> ${'x'.repeat(len)}`]; + const names = note(470); + const files = note(390); + const generic = note(330); + const fit = (room: number, noteCandidates: string[][], noteYields: boolean) => + fitExploreEpilogue({ room, noteCandidates, noteYields, pointerEntries: entries, pointerOmitted: 20 }); + const entriesIn = (block: readonly string[]) => block.filter((l) => l.startsWith('- src/')).length; + /** Room for exactly `k` pointer entries (header, entries and tail) beside `withNote`. */ + const roomFor = (k: number, withNote: readonly string[] = []) => { + const full = fit(1e6, [], false).pointers; + const header = full.slice(0, 2); + return cost(withNote) + cost([...header, ...entries.slice(0, k), `- ... and ${entries.length - k + 20} more files`]); + }; + + it('a complete-source note keeps its precedence over the pointer list', () => { + const complete = note(330); + const r = fit(cost(complete) + 20, [complete], false); + expect(r.note).toEqual(complete); + expect(r.pointers).toEqual([]); + }); + + it('a trimmed note leaves the pointer list its first entry', () => { + // The generic note fits alone, but not beside the list's first entry. + const room = cost(generic) + 20; + expect(room).toBeLessThan(roomFor(1, generic)); + const r = fit(room, [names, files, generic], true); + expect(r.note).toEqual([]); + expect(entriesIn(r.pointers)).toBeGreaterThanOrEqual(1); + }); + + it("a trimmed note's detail never costs a pointer entry", () => { + // Beside the generic note three entries fit; beside the named one, fewer. + const room = roomFor(3, generic); + const r = fit(room, [names, files, generic], true); + expect(r.note).toEqual(generic); + expect(entriesIn(r.pointers)).toBe(3); + }); + + it('takes the most specific note when it costs no entry', () => { + const r = fit(roomFor(5, names), [names, files, generic], true); + expect(r.note).toEqual(names); + expect(entriesIn(r.pointers)).toBe(5); + }); + + it('with no pointer list, takes the most specific note that fits', () => { + const r = fitExploreEpilogue({ + room: cost(files) + 10, noteCandidates: [names, files, generic], noteYields: true, pointerEntries: [], pointerOmitted: 0, + }); + expect(r.note).toEqual(files); + expect(r.pointers).toEqual([]); + }); +}); + +describe('codegraph_explore — the note follows what the render cut', () => { + let dir: string; + let cg: CodeGraph; + let handler: ToolHandler; + + // Long enough that no tier ships the file whole, and past the 200-line + // oversize-spine threshold, so the render windows `runPipeline` to its head + // plus the `stepNext` call site. + const FILLER = 360; + const CALL_AT = 180; + + beforeAll(async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-completeness-')); + fs.writeFileSync(path.join(dir, 'package.json'), '{"name":"cg-completeness","version":"1.0.0"}\n'); + const src = path.join(dir, 'src'); + fs.mkdirSync(src); + const body: string[] = [ + "import { stepNext } from './step';", + '', + 'export function runPipeline(input: number): number {', + ' let acc = input;', + ]; + for (let i = 0; i < FILLER; i++) { + if (i === CALL_AT) body.push(' acc = stepNext(acc);'); + body.push(` acc = acc + ${i}; // pipeline stage ${i}`); + } + body.push(' const PIPELINE_TAIL_MARKER = acc;', ' return PIPELINE_TAIL_MARKER;', '}', ''); + fs.writeFileSync(path.join(src, 'pipeline.ts'), body.join('\n')); + // Three hops, because the flow only has a spine (and so a call site to + // window to) for a chain of three or more. + fs.writeFileSync( + path.join(src, 'step.ts'), + "import { finalizeStep } from './finalize';\n\nexport function stepNext(v: number): number {\n return finalizeStep(v * 2);\n}\n", + ); + fs.writeFileSync(path.join(src, 'finalize.ts'), 'export function finalizeStep(v: number): number {\n return v + 1;\n}\n'); + // A second, small flow that renders whole: nothing to trim. + fs.writeFileSync( + path.join(src, 'format.ts'), + "import { padValue } from './pad';\n\nexport function formatValue(v: number): string {\n return padValue(String(v));\n}\n", + ); + fs.writeFileSync(path.join(src, 'pad.ts'), "export function padValue(s: string): string {\n return s.padStart(8, ' ');\n}\n"); + + cg = CodeGraph.initSync(dir, { config: { include: ['**/*.ts'], exclude: [] } }); + await cg.indexAll(); + handler = new ToolHandler(cg); + }, 120_000); + + afterAll(() => { + cg?.destroy(); + if (dir && fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + const explore = async (query: string, fileCount?: number): Promise => { + const spy = fileCount === undefined + ? null + : vi.spyOn(cg, 'getStats').mockReturnValue({ fileCount, nodeCount: 50 } as ReturnType); + try { + const result = await handler.execute('codegraph_explore', { query }); + return result.content?.[0]?.text ?? ''; + } finally { + spy?.mockRestore(); + } + }; + + it('large tier: a windowed spine method is reported trimmed, not complete', async () => { + const text = await explore('runPipeline stepNext finalizeStep', 1000); + // The fixture does what it is for: the method is windowed, so its tail is not in the response. + expect(text).toContain('src/pipeline.ts'); + expect(text).toContain('stepNext(acc)'); + expect(text).not.toContain('PIPELINE_TAIL_MARKER'); + + expect(text).not.toContain('Complete source for'); + expect(text).toContain('Verbatim source for'); + expect(text).toContain('treat it as already Read'); + expect(text).toMatch(/Trimmed for size: `pipeline\.ts`|Some sections were trimmed for size/); + expect(epilogueOf(text)).not.toMatch(OFFERS_READ); + }); + + it('small tier: the same cut gets the trimmed note', async () => { + const text = await explore('runPipeline stepNext finalizeStep'); + expect(text).not.toContain('PIPELINE_TAIL_MARKER'); + expect(text).toContain('Some file sections were trimmed for size'); + }); + + it('large tier: complete sections are still called complete, without the Read escape hatch', async () => { + const text = await explore('formatValue padValue', 1000); + expect(text).toContain('src/format.ts'); + expect(text).toContain('src/pad.ts'); + expect(text).toMatch(/Complete source for \d+ files is included above/); + expect(text).not.toContain('Verbatim source for'); + expect(epilogueOf(text)).not.toMatch(OFFERS_READ); + }); + + it('small tier: complete sections get no trimmed note', async () => { + const text = await explore('formatValue padValue'); + expect(text).not.toContain('trimmed for size'); + }); +}); + +describe('codegraph_explore — a dedup remainder folded into the back-reference is not delivered', () => { + let dir: string; + let cg: CodeGraph; + let handler: ToolHandler; + let previousDedup: string | undefined; + + beforeAll(async () => { + previousDedup = process.env.CODEGRAPH_EXPLORE_DEDUP; + process.env.CODEGRAPH_EXPLORE_DEDUP = '1'; + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-completeness-fold-')); + fs.writeFileSync(path.join(dir, 'package.json'), '{"name":"cg-completeness-fold","version":"1.0.0"}\n'); + const src = path.join(dir, 'src'); + fs.mkdirSync(src); + // `loadLedger`, then a tiny unrelated `tinyTail` just below it, then filler far + // away so no tier ships the file whole. The first call sends `loadLedger` and + // its padding; the second adds `tinyTail`, whose few lines are the remainder + // dedup folds into the "Already sent" pointer instead of fencing them. + const body: string[] = ['export function loadLedger(rows: number[]): number {', ' let total = 0;']; + for (let i = 0; i < 30; i++) body.push(` total += rows[${i}] ?? ${i};`); + // Past the first call's 3 lines of padding, inside the cluster gap. + body.push(' return total;', '}', '', '//', '//', '//', 'export const tinyTail = (): number => 7;', ''); + for (let i = 0; i < 40; i++) { + body.push(`export function farFiller${i}(x: number): number {`); + for (let j = 0; j < 5; j++) body.push(` x = x * ${j + 2} + ${i};`); + body.push(' return x;', '}', ''); + } + fs.writeFileSync(path.join(src, 'ledger.ts'), body.join('\n')); + cg = CodeGraph.initSync(dir, { config: { include: ['**/*.ts'], exclude: [] } }); + await cg.indexAll(); + handler = new ToolHandler(cg); + }, 120_000); + + afterAll(() => { + if (previousDedup === undefined) delete process.env.CODEGRAPH_EXPLORE_DEDUP; + else process.env.CODEGRAPH_EXPLORE_DEDUP = previousDedup; + cg?.destroy(); + if (dir && fs.existsSync(dir)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('does not call the file complete when the folded lines were never sent', async () => { + const spy = vi.spyOn(cg, 'getStats').mockReturnValue({ fileCount: 1000, nodeCount: 50 } as ReturnType); + try { + const session = new ExploreSessionState(); + const run = (query: string) => + handler.execute('codegraph_explore', { query }, session).then((r) => r.content?.[0]?.text ?? ''); + const first = await run('loadLedger'); + expect(first).toContain('loadLedger'); + expect(first).not.toContain('tinyTail = ()'); + const second = await run('loadLedger tinyTail'); + // The fixture reaches the fold: the section is a pointer, and `tinyTail`'s + // source is in neither response. + expect(second).toContain('Already sent earlier in this conversation'); + expect(second).not.toContain('tinyTail = ()'); + expect(second).not.toContain('Complete source for'); + expect(second).toContain('Trimmed for size: `ledger.ts`'); + } finally { + spy.mockRestore(); + } + }); +}); diff --git a/docs/benchmarks/explore-completeness-note.md b/docs/benchmarks/explore-completeness-note.md new file mode 100644 index 0000000000..b8f75d39c5 --- /dev/null +++ b/docs/benchmarks/explore-completeness-note.md @@ -0,0 +1,132 @@ +# The completeness note claims only what was sent + +**Date:** 2026-09-28 · **New:** `claude/nostalgic-mccarthy-31c89d` (measured at `024d88b9`, then +rebased onto #2071) · **Baseline:** `main` @ `724b5dee` (#2068) · **Harness:** `scripts/agent-eval/ab-new-vs-baseline.sh`, `--model +sonnet --effort high`, both arms codegraph-on, CLI blocked, `CODEGRAPH_NO_PROMPT_HOOK=1`, every +index rebuilt per arm. + +## The defect + +On the tiers with `includeCompletenessSignal` (>= 500 indexed files), every `codegraph_explore` +response ended with the same line: + +> **Complete source for N files is included above — do NOT re-read them.** … Reserve Read for a +> single specific line range explore can't surface. + +It was printed whatever the render had cut. The small tiers switch to a "trimmed for size" note on +`anyFileTrimmed`, but that flag is set at individual trim sites, and before #2068 one of them, the +oversize-spine window, set nothing. That is how a 62-line slice of vscode's 968-line +`rpcProtocol.ts` went out as complete. Even with every site flagged, the large tiers never read the +flag. The last sentence also offered Read, which explore output must never do. + +## The change + +- **Completeness is measured, not flagged.** Each file section passes the symbols it set out to + deliver (the cluster members, or the per-symbol view's symbols) to `emitFileSection`. + `elidedWantedSpans` checks them against the ranges actually sent, plus back-referenced ones. A + span not fully covered marks the file trimmed. Any path that elides source is caught this way, + including paths added later. +- **The note says what is true.** When no section was trimmed it is the old line, without the + Read sentence. Otherwise it keeps the guarantee that still holds and names what was cut: + + > **Verbatim source for 2 files is included above — treat it as already Read.** Trimmed for + > size: `rpcProtocol.ts`; gap markers and file headers name what was elided (e.g. + > `RPCProtocol._receiveOneMessage`, `MessageIO.serializeReplyOK`, …). For those, or anything + > under "Not shown above", make ANOTHER codegraph_explore with those exact names instead of + > reading the files — it returns their source with line numbers. + + Elided methods are offered as `Owner.member` (django has 110 `as_sql`s). Containers are never + offered, because a class too big for one section comes back trimmed on a follow-up too. +- **Fitted like the rest of the epilogue (CG-26), in `fitExploreEpilogue`.** The note comes in + three sizes: with names, with files, and generic. The generic one (≈340 chars) is shorter than + the line it replaced (401). A complete-source note keeps its old precedence. A trimmed note + yields to the pointer list, which names files the response does not show at all, while a + trimmed section already names its elisions in its own gap markers. It leaves the list's header + and first entry, and its optional detail never costs an entry the generic note would have left. +- **The fallback notes follow.** The lost-pointer note, the epilogue-cut note and the truncation + note drop "complete" when a section they vouch for was trimmed. Each trimmed variant is no longer + than the original, so the epilogue floor is unchanged. + +## Deterministic replay + +39 agent-shaped queries (the vscode and django queries from the named-concentration A/B, the +README questions, and pinned / stress queries), replayed through both builds on one index with a +fresh `ToolHandler` per query. + +| | `main` @ `724b5dee` | new | +|---|---|---| +| source sections byte-identical | — | **39 / 39** (only the epilogue changed) | +| responses claiming "Complete source" | 24 | 1 | +| … of which a section was actually trimmed | **23** | 0 | +| responses with the trimmed note | 0 | 16 (1 naming elided symbols) | +| responses with a pointer list | 23 | 33 | +| pointer entries, total | 71 | 102 (no list shorter than baseline) | +| responses offering Read | 24 | 0 | +| responses over the 25K inline cap | 0 | 0 | + +The named variant is rare because the fit spends leftover room on pointer entries first. In +saturated responses the names cost an entry, and the gap markers already carry them. + +Re-run after rebasing onto `290e03f7` (#2071, same-named file pinning), against that build: source +identical 39 / 39, "Complete source" 23 → 1, trimmed note 16, pointer entries 67 → 93 (none +shorter), Read offered 23 → 0, nothing over the cap. + +## Agent A/B + +Four questions, three runs per arm, against `724b5dee`. + +| question | Read runs (new / base) | Grep/Glob | explore calls (new / base) | duration median (s) | +|---|---|---|---|---| +| vscode: rpcProtocol serialize + dispatch | 0 / 0 | 0 / 0 | 7 / 6 | 45 / 36 | +| vscode: extension host ↔ main process | 0 / 0 | 0 / 0 | 12 / 12 | 61 / 58 | +| django: `SQLCompiler.as_sql` / `pre_sql_setup` / `get_select` | 0 / 0 | 0 / 0 | 6 / 6 | 31 / 32 | +| django: QuerySet → SQL | 0 / 0 | 0 / 0 | 4 / 5 | 22 / 23 | +| **pooled, 12 runs** | **0 / 0** | **0 / 0** | **29 / 29** | 36 / 36 | + +The change was exercised. Of the 29 explore responses the new arm's agents received, 20 carried +the trimmed note (4 naming elided methods). The baseline agents received "Complete source … Reserve +Read" on 26 of 29, and on this build all but one of those are false. One pre-explore Bash call in +the new arm happens before any codegraph output and is noise. Load average was 30–95 from other +work on the machine, so durations are indicative only. + +### Earlier round, against `e63fe2ec` + +Before #2062 / #2063 / #2068 landed, the same four questions (the django pair run twice, 18 runs +per arm) showed Read in 8 new runs vs 7 baseline runs, Grep/Glob 0 / 0, explore 42 / 49. Every +django Read targeted `SQLCompiler.as_sql`, which no build returned for the query shape the agents +wrote (`SQLCompiler.as_sql … compiler.py:776`). Naming the file pinned it and dropped the method: +175 of 228 lines for the bare name, 0 with the path. #2063 fixed that, and on current `main` no run +in either arm reads. + +## Coverage + +`__tests__/explore-completeness-note.test.ts`: + +- `elidedWantedSpans`: partial and absent spans, back-referenced and adjacent ranges joining, + ordering, and invalid spans. +- `shortestUniqueSuffixes`. +- `exploreCompletenessNotes`: the complete case never offers Read; the trimmed case keeps the + already-Read guarantee, names files and `Owner.member` names, and skips containers; candidates + shrink in order, and the last is shorter than the old line; "1 file", never "0 files"; a trimmed + file's label stays distinct from a same-named file in the pointer list. +- `EXPLORE_FALLBACK_NOTES`: each trimmed wording drops "complete", keeps the already-Read + guarantee, and is no longer than the wording the floor and the cut test were sized on. +- `fitExploreEpilogue`: + - a complete note keeps precedence; + - a trimmed note leaves the first pointer entry; + - detail never costs an entry; + - with no pointer list, the most specific note that fits wins. +- End to end on a three-hop flow whose spine method the render windows: the large tier reports it + trimmed, the small tier agrees, and complete flows are still called complete. +- End to end with dedup on: a second call whose new lines are too few to fence, and are folded + into the "Already sent" pointer, is reported trimmed rather than complete. + +Mutation-checked: + +| Mutation | Red | +|---|---| +| Derived detection removed | the large-tier windowed-spine test | +| Old Read sentence restored | both Read checks | +| The entry rule dropped from the fit | "detail never costs a pointer entry" | +| The first-entry reservation dropped | "a trimmed note leaves the pointer list its first entry" | +| Completeness judged before the dedup fold | the folded-remainder test | diff --git a/docs/design/explore-budget-allocation.md b/docs/design/explore-budget-allocation.md index e8fe5d2556..83b6b1a523 100644 --- a/docs/design/explore-budget-allocation.md +++ b/docs/design/explore-budget-allocation.md @@ -1023,7 +1023,9 @@ its line number. injected; the render dropped them, by the same density starvation as the item above. *Fixed by #2062; `__tests__/explore-pinned-cap-named.test.ts` (#2064) guards the pinned entry path.* - **The large tiers' completeness line** ("Complete source for N files is included above") is - unconditional, so it overclaims whenever a section was trimmed. *Still open.* + unconditional, so it overclaims whenever a section was trimmed. *Fixed by #2077: completeness is + measured from the ranges each section actually sent (`elidedWantedSpans`), and a trimmed + response says which files were trimmed; see `docs/benchmarks/explore-completeness-note.md`.* ### Landing on `main` (2026-09-28) diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index f2d3980256..5c99378c85 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -1144,6 +1144,236 @@ function pointerLineFor(filePath: string, nodes: readonly Node[]): string { * and that another explore — not a Read — is how to reach it. */ const EPILOGUE_LOST_NOTE = '> (Trailing pointer list omitted for size. The source above is complete and verbatim — treat it as already Read. For anything this call did not cover, run another codegraph_explore with the specific names rather than reading those files.)'; +/** + * The notes that stand in for the epilogue, or close a truncated response, in + * two wordings each. `complete` says the source above is complete; `trimmed` is + * used when a section it vouches for was trimmed (see `elidedWantedSpans`) and + * drops that claim, keeping the guarantee that is still true. Each trimmed + * wording is no longer than its complete one: the epilogue floor (`lost`) and + * the cut note's fit test (`cut`) are sized before the render knows which of the + * two it will need. + */ +export const EXPLORE_FALLBACK_NOTES = { + lost: { + complete: EPILOGUE_LOST_NOTE, + trimmed: '> (Trailing pointer list omitted for size. The source above is verbatim — treat it as already Read. For symbols its gap markers name, and anything else not covered, run another codegraph_explore with those names rather than reading.)', + }, + cut: { + complete: '\n\n> (Trailing notes omitted for size. The source above is complete and verbatim — treat it as already Read. For anything this call did not cover, run another codegraph_explore with the specific names rather than reading those files.)', + trimmed: '\n\n> (Trailing notes omitted for size. The source above is verbatim — treat it as already Read. For symbols its gap markers name, and anything else not covered, run another codegraph_explore with those names rather than reading.)', + }, + truncated: { + complete: '\n\n... (output truncated to budget; the source above is complete and verbatim — treat it as already Read. For any area not covered, run another codegraph_explore with the specific names — do NOT Read these files.)', + trimmed: '\n\n... (output truncated to budget; the source above is verbatim — treat it as already Read. For names its gap markers list, or any area not covered, run another codegraph_explore — do NOT Read these files.)', + }, +} as const; + +/** + * One symbol a file section set out to deliver: a cluster member, or a symbol + * of the per-symbol (focused/skeleton) view. Completeness is judged against + * these, not against the file — explore never promises whole files, only the + * symbols it selected for each one. + */ +export type ExploreWantedSpan = { + name: string; + kind: string; + start: number; + end: number; + importance: number; + spine: boolean; + /** The indexed qualified name (`SQLCompiler::as_sql`), when the span is a node. */ + qualifiedName?: string; +}; + +/** + * The wanted spans a section did NOT deliver in full: some line of the span is + * in neither what this call sent nor what an earlier call already sent (a + * back-referenced span counts — the agent holds that copy). + * + * Derived from the emitted ranges rather than from a flag each trim site has to + * remember to set. The oversize-spine window set none until #2068, which is how + * a 62-line slice of vscode's 968-line `rpcProtocol.ts` went out under "Complete + * source … do NOT re-read them". Whatever elides source — a member shrink, a + * ceiling window, a dropped cluster, the per-symbol view, a path added later — + * shows up here. + * + * Most relevant first (spine, then importance), then source order. + */ +export function elidedWantedSpans( + wanted: ReadonlyArray, + delivered: ReadonlyArray, +): ExploreWantedSpan[] { + const merged = mergeRanges(delivered); + const out = wanted.filter((w) => w.start > 0 && w.end >= w.start + && !merged.some((r) => r.start <= w.start && w.end <= r.end)); + return out.sort((a, b) => + Number(b.spine) - Number(a.spine) || b.importance - a.importance || a.start - b.start); +} + +/** A rendered file whose section is missing some of what it set out to deliver. */ +export type ExplorePartialFile = { filePath: string; elided: ReadonlyArray }; + +/** Trimmed files the completeness note names one by one; the rest are a count. */ +const TRIMMED_FILES_NAMED = 3; +/** Elided symbols the completeness note names, from the spine or named by the agent. */ +const TRIMMED_SYMBOLS_NAMED = 4; +/** + * Kinds the note never offers as a follow-up target. A container elided by a + * trim is too big for one section by construction, so exploring it by name + * comes back trimmed too; its members are the useful names. + */ +const TRIMMED_NAME_SKIP_KINDS = new Set([ + 'file', 'module', 'namespace', 'class', 'struct', 'union', 'interface', 'protocol', 'trait', +]); + +/** + * The name the completeness note offers for an elided symbol: `Owner.member` + * for a method, so an overloaded name resolves to the definition that was cut + * (django has 110 `as_sql`s; the note used to offer the bare one and agents + * then added a path and line to disambiguate it). The bare name otherwise. + */ +function followUpName(e: ExploreWantedSpan): string { + if (e.kind !== 'method' || !e.qualifiedName) return e.name; + const segs = e.qualifiedName.split('::'); + const owner = segs.length >= 2 ? segs[segs.length - 2]! : ''; + return /^[A-Za-z_$][\w$]*$/.test(owner) ? `${owner}.${e.name}` : e.name; +} + +/** + * The shortest trailing slice of each path that no other path in `paths` + * ends with: `extHostExtensionService.ts` alone when it is the only one, + * `node/extHostExtensionService.ts` beside `common/extHostExtensionService.ts`. + */ +export function shortestUniqueSuffixes(paths: ReadonlyArray): Map { + const all = [...new Set(paths)]; + const out = new Map(); + for (const p of all) { + const segs = p.split('/'); + let n = 1; + for (; n < segs.length; n++) { + const suffix = segs.slice(-n).join('/'); + if (!all.some((o) => o !== p && (o === suffix || o.endsWith(`/${suffix}`)))) break; + } + out.set(p, segs.slice(-n).join('/')); + } + return out; +} + +/** + * The large tiers' completeness note (`includeCompletenessSignal`), as + * candidates from most to least specific. The epilogue fit keeps the first + * one that fits the room left (CG-26). + * + * "Complete" is claimed only when no section elided anything it set out to + * deliver. Otherwise the note keeps the guarantee that is still true (every + * block shown is verbatim; treat it as already Read), names the trimmed files + * and the most relevant elided symbols as room allows, and sends the agent to + * another codegraph_explore for them. It never offers Read: explore output must + * not tell the agent to Read (AGENTS.md). + */ +export function exploreCompletenessNotes( + filesIncluded: number, + trimmed: ReadonlyArray, + /** Every path the response can name (sections and pointer list); labels are unique among them. */ + knownPaths: ReadonlyArray, +): string[] { + // No count when every section is held from an earlier call: "0 files" reads + // as nothing shown, beside a note about what was shown. + const files = filesIncluded === 0 ? 'these files' + : filesIncluded === 1 ? '1 file' : `${filesIncluded} files`; + if (trimmed.length === 0) { + return [`> **Complete source for ${files} is included above — do NOT re-read them.** If your question also needs files/symbols listed under "Not shown above" (or any area this call didn't cover), make ANOTHER codegraph_explore targeting those names — it returns the same source with line numbers and is cheaper and more complete than reading.`]; + } + const label = shortestUniqueSuffixes([...knownPaths, ...trimmed.map((t) => t.filePath)]); + const shownFiles = trimmed.slice(0, TRIMMED_FILES_NAMED).map((t) => `\`${label.get(t.filePath)}\``); + const moreFiles = trimmed.length - shownFiles.length; + const trimmedList = shownFiles.join(', ') + (moreFiles > 0 ? ` +${moreFiles} more` : ''); + const names: string[] = []; + for (const t of trimmed) { + for (const e of t.elided) { + if (names.length >= TRIMMED_SYMBOLS_NAMED) break; + if (!(e.spine || e.importance >= 9) || TRIMMED_NAME_SKIP_KINDS.has(e.kind)) continue; + const name = followUpName(e); + if (!names.includes(name)) names.push(name); + } + } + const head = `> **Verbatim source for ${files} is included above — treat it as already Read.**`; + const what = 'gap markers and file headers name what was elided'; + const tail = 'For those, or anything under "Not shown above", make ANOTHER codegraph_explore with those exact names instead of reading the files — it returns their source with line numbers.'; + const withFiles = `${head} Trimmed for size: ${trimmedList}; ${what}`; + const candidates = names.length > 0 + ? [`${withFiles} (e.g. ${names.map((n) => `\`${n}\``).join(', ')}). ${tail}`] + : []; + candidates.push(`${withFiles}. ${tail}`, `${head} Some sections were trimmed for size; ${what}. ${tail}`); + return candidates; +} + +/** Chars a block of lines costs once joined into the response. */ +const roomForLines = (block: readonly string[]): number => + block.reduce((n, s) => n + s.length + 1, 0); + +/** + * Fit the completeness note and the pointer list into the room the response has + * left (CG-26). Returns the note (a candidate block, or none), the pointer + * block, and the room that remains. + * + * The note is one of `noteCandidates`, most specific first. A complete-source + * note keeps the precedence it always had and goes first. A note that YIELDS (a + * trimmed one) gives way to the pointer list, which names files the response + * does not show at all, while what a trimmed section elided is already named in + * its own gap markers and header. It leaves the list's header and first entry + * when that much could fit, and its optional detail never costs an entry the + * least specific candidate would have left. + */ +export function fitExploreEpilogue(opts: { + room: number; + noteCandidates: ReadonlyArray; + noteYields: boolean; + pointerEntries: readonly string[]; + pointerOmitted: number; +}): { note: string[]; pointers: string[]; room: number } { + const { noteCandidates, noteYields, pointerEntries, pointerOmitted } = opts; + let room = opts.room; + // The pointer list as it fits `space`: entries in rank order, and a tail + // line confessing every entry left out. + const fitPointers = (space: number): { block: string[]; taken: number } => { + if (pointerEntries.length === 0) return { block: [], taken: 0 }; + const head = [POINTER_HEADER, '']; + let left = space - roomForLines(head); + if (left < 0) return { block: [], taken: 0 }; + let taken = 0; + for (const entry of pointerEntries) { + // Every entry we do NOT take has to be confessed by the tail line, so + // the tail's cost is part of taking one less than all of them. + const dropped = pointerEntries.length - taken - 1 + pointerOmitted; + const tail = dropped > 0 ? roomForLines([`- ... and ${dropped} more files`]) : 0; + if (entry.length + 1 + tail > left) break; + left -= entry.length + 1; + taken++; + } + if (taken === 0) return { block: [], taken: 0 }; + const block = [...head, ...pointerEntries.slice(0, taken)]; + const dropped = pointerEntries.length - taken + pointerOmitted; + if (dropped > 0) block.push(`- ... and ${dropped} more files`); + return { block, taken }; + }; + + const pointerNeed = pointerEntries.length > 0 + ? roomForLines([POINTER_HEADER, '', pointerEntries[0]!, + `- ... and ${pointerEntries.length - 1 + pointerOmitted} more files`]) + : 0; + const pointerMin = noteYields && pointerNeed <= room ? pointerNeed : 0; + const leastSpecific = noteCandidates[noteCandidates.length - 1]; + const entriesBeside = (b: readonly string[]) => fitPointers(room - roomForLines(b)).taken; + const entriesFloor = noteYields && leastSpecific ? entriesBeside(leastSpecific) : 0; + const note = [...(noteCandidates.find((b) => roomForLines(b) + pointerMin <= room + && (!noteYields || entriesBeside(b) >= entriesFloor)) ?? [])]; + room -= roomForLines(note); + + const pointers = fitPointers(room).block; + room -= roomForLines(pointers); + return { note, pointers, room }; +} /** * Match response delimiters rather than ASCII "path characters": filenames @@ -4951,7 +5181,9 @@ export class ToolHandler { ? ` No indexed file uniquely matches ${unresolvedPathSpans.map((sp) => `\`${sp}\``).join(', ')}.`.length : 0) + setAsideNote.length; - const epilogueFloor = EPILOGUE_LOST_NOTE.length + 2 + cliffPointerFloor + summaryReserve; + const epilogueFloor = Math.max( + EXPLORE_FALLBACK_NOTES.lost.complete.length, EXPLORE_FALLBACK_NOTES.lost.trimmed.length, + ) + 2 + cliffPointerFloor + summaryReserve; // Absolute stop for the render loop. Reservations already fit the envelope, so // this only catches their bounded overshoot (the whole-file grace, an oversize // first cluster) — and catches it HERE, where a file can be skipped cleanly and @@ -4972,6 +5204,11 @@ export class ToolHandler { // (#1046) — it must reflect what we show, not the raw candidate gather. const renderedFilePaths: string[] = []; let anyFileTrimmed = false; + // Rendered files whose section is missing some of what it set out to + // deliver, by path — measured at emission from the ranges actually sent + // (`elidedWantedSpans`), not from the trim sites. Drives the completeness + // note: "complete" is claimed only for sections that are. + const trimmedFiles = new Map(); // Files that changed on disk after their last index sync (#1474). Their // indexed line ranges are untrustworthy, so sliced renders (adaptive / // skeleton / clusters) are OFF for them: a small drifted file still ships @@ -5335,6 +5572,12 @@ export class ToolHandler { overhead: number; mode: 'whole' | 'clusters' | 'focused' | 'skeleton'; clipped: boolean; + /** + * The symbols this section set out to deliver. Any not covered by what + * is sent (`ranges`, after the fold) plus `covered` marks the file + * trimmed for the completeness note. + */ + wanted: ReadonlyArray; /** The undeduped render, kept for the no-new-source fallback. */ fullBody: string; fullRanges: ExploreLineRange[]; @@ -5345,6 +5588,10 @@ export class ToolHandler { const folded = opts.covered.length > 0 && opts.body.length < EXPLORE_DEDUP.MIN_DELTA_CHARS; const body = folded ? '' : opts.body; const ranges = folded ? [] : opts.ranges; + // Judged on what this call sends plus what the agent already holds — + // AFTER the fold, whose remainder is in neither. + const elided = elidedWantedSpans(opts.wanted, [...ranges, ...opts.covered]); + if (elided.length > 0) trimmedFiles.set(filePath, elided); const at = lines.length; lines.push(opts.header, ''); // Charge what the section ACTUALLY costs, not a flat 200 (CG-26). A @@ -5594,6 +5841,22 @@ export class ToolHandler { // doesn't Read the file back for it — Django's SQLCompiler.execute_sql/as_sql); // every other symbol is just its signature. So the base mechanism survives while // the file's other ~80 symbols + the redundant subclasses collapse to one line each. + // The symbols a section of this file sets out to deliver, for the + // completeness check (`elidedWantedSpans`), ranked the way the cluster + // path ranks its members. + // The file node is not one of them: no section delivers "the file" as a + // symbol, and it spans trailing lines no render prints. + const wantedFrom = (nodes: readonly Node[]): ExploreWantedSpan[] => nodes + .filter((n) => n.kind !== 'import' && n.kind !== 'export' && n.kind !== 'file' && n.startLine > 0) + .map((n) => ({ + name: n.name, + kind: n.kind, + start: n.startLine, + end: n.endLine, + importance: entryNodeIds.has(n.id) ? 10 : flow.namedNodeIds.has(n.id) ? 9 : 1, + spine: flow.pathNodeIds.has(n.id), + qualifiedName: n.qualifiedName, + })); const spareNamed = group.nodes.some(n => flow.uniqueNamedNodeIds.has(n.id)); const fileDefinesSuper = definesPolymorphicSupertype(group.nodes); const spared = spareNamed && !fileDefinesSuper; @@ -5818,6 +6081,8 @@ export class ToolHandler { mode: bodyIds.size + bodyWindows.size > 0 ? 'focused' : 'skeleton', // Always "clipped": the per-symbol view elides bodies by construction. clipped: true, + // Every symbol whose body this view could have shown. + wanted: wantedFrom(syms), fullBody: withTail(skel), fullRanges: skel.map((p) => p.range), }); @@ -5955,6 +6220,11 @@ export class ToolHandler { overhead: 200, mode: 'whole', clipped: false, + // Sent whole, so nothing is elided — unless dedup folds a remainder + // the agent does not hold (`emitFileSection`). Clamped to the lines + // the render carries, which stop before trailing blank lines. + wanted: wantedFrom(group.nodes) + .map((w) => ({ ...w, end: Math.min(w.end, wholeRange.end) })), fullBody: fullSection, fullRanges: [wholeRange], }); @@ -6022,7 +6292,7 @@ export class ToolHandler { // qualified `SQLCompiler.as_sql` is no longer outranked by a denser cluster // around `SQLInsertCompiler.as_sql` that merely matched the bare name. const EXACT_IMPORTANCE = 11; - const ranges: Array<{ start: number; end: number; name: string; kind: string; importance: number; spine: boolean; spineCallLine?: number }> = [...rangeNodes.values()] + const ranges: Array<{ start: number; end: number; name: string; kind: string; importance: number; spine: boolean; spineCallLine?: number; qualifiedName?: string }> = [...rangeNodes.values()] // Drop whole-file envelope nodes (containers covering >50% of the file). .filter(n => !(ENVELOPE_KINDS.has(n.kind) && (n.endLine - n.startLine + 1) > fileLines.length * 0.5)) .map(n => { @@ -6037,7 +6307,7 @@ export class ToolHandler { // processRunExecutionData, the named flow ENTRY at L1562, is a large // low-density method that lost the budget to denser blocks and got cut, so // the agent Read it back — the very thing explore exists to prevent). - return { start: n.startLine, end: n.endLine, name: n.name, kind: n.kind, importance, spine: flow.pathNodeIds.has(n.id), spineCallLine: flow.spineCallSites.get(n.id) }; + return { start: n.startLine, end: n.endLine, name: n.name, kind: n.kind, importance, spine: flow.pathNodeIds.has(n.id), spineCallLine: flow.spineCallSites.get(n.id), qualifiedName: n.qualifiedName }; }); // Add edge source locations in this file — captures template references @@ -6855,6 +7125,9 @@ export class ToolHandler { // Windowing an oversize member elides source too — reporting it as // unclipped would hide exactly the cut the diagnostic exists to show. clipped: chosenNow.size < clusters.length || anyClusterShrunk, + // Every member of every cluster, chosen or not: a dropped cluster, a + // shrunk one and a windowed spine all leave a member short. + wanted: ranges, fullBody: sectionText(fullClusterParts), fullRanges: fullClusterParts.map((p) => p.range), }); @@ -6968,15 +7241,23 @@ export class ToolHandler { pointerOmitted = Math.max(0, remainingFiles.length - pointerEntries.length); } - // Completeness signal so agents know they don't need to re-read these files. - // On small projects the budget gates this off — but if we actually had to - // trim or drop clusters, surface a brief note so the agent knows it can - // still Read for more detail. - const completenessBlock: string[] = budget.includeCompletenessSignal - ? ['', '---', `> **Complete source for ${filesIncluded} files is included above — do NOT re-read them.** If your question also needs files/symbols listed under "Not shown above" (or any area this call didn't cover), make ANOTHER codegraph_explore targeting those names — it returns the same source with line numbers and is cheaper and more complete than reading. Reserve Read for a single specific line range explore can't surface.`] - : anyFileTrimmed - ? ['', `> Some file sections were trimmed for size. Elided symbols are named inside gap markers as \`name (file:line)\` and preferred in the file header — run another \`codegraph_explore\` (or \`codegraph_node\`) with those exact names for their source.`] + // Completeness signal so agents know they don't need to re-read these files + // — claimed only for sections that ARE complete. A trimmed section is + // measured from what was sent (`trimmedFiles`), in render order. On small + // projects the budget gates the signal off, but a trim still gets a brief + // note pointing at the names the gap markers carry. + const trimmedShown: ExplorePartialFile[] = renderedFilePaths + .filter((fp) => trimmedFiles.has(fp)) + .map((fp) => ({ filePath: fp, elided: trimmedFiles.get(fp)! })); + const completenessCandidates: string[][] = budget.includeCompletenessSignal + ? exploreCompletenessNotes(filesIncluded, trimmedShown, [...renderedFilePaths, ...fileGroups.keys()]) + .map((note) => ['', '---', note]) + : anyFileTrimmed || trimmedShown.length > 0 + ? [['', `> Some file sections were trimmed for size. Elided symbols are named inside gap markers as \`name (file:line)\` and preferred in the file header — run another \`codegraph_explore\` (or \`codegraph_node\`) with those exact names for their source.`]] : []; + /** Whether a trimmed section survives in `text` — picks a fallback note's wording after a cut. */ + const trimmedIn = (text: string): boolean => + trimmedShown.some((t) => text.includes(`${FILE_SECTION_PREFIX}${t.filePath}\``)); // Advisory exploration-guidance note based on project size. Deliberately // phrased as guidance, NOT a quota: agents read "budget / remaining calls / @@ -7002,42 +7283,25 @@ export class ToolHandler { // one thing: a fixed floor the loop reserves for (the cut note, plus a // pointer for every file whose bytes were deliberately WITHHELD — CG-12 // makes those names load-bearing) and an elastic tail that takes what is - // left. Assembled in priority order — the do-not-re-read reminder first, - // then pointers in rank order, then the budget note — and emitted in - // document order. - const roomFor = (block: readonly string[]): number => - block.reduce((n, s) => n + s.length + 1, 0); + // left. Assembled in priority order — the completeness note and the + // pointers by `fitExploreEpilogue`'s rules, then the budget note — and + // emitted in document order. + const roomFor = roomForLines; // Less what the summary line will grow by when its sentinel is filled in. let room = hardCeiling - (flow.text.length + lines.join('\n').length) - Math.max(0, summaryReserve - SUMMARY_SENTINEL.length); - const keepCompleteness = completenessBlock.length > 0 - && roomFor(completenessBlock) <= room; - if (keepCompleteness) room -= roomFor(completenessBlock); - - const pointerBlock: string[] = []; - if (pointerEntries.length > 0) { - const head = [POINTER_HEADER, '']; - let left = room - roomFor(head); - if (left >= 0) { - let taken = 0; - for (const entry of pointerEntries) { - // Every entry we do NOT take has to be confessed by the tail line, so - // the tail's cost is part of taking one less than all of them. - const dropped = pointerEntries.length - taken - 1 + pointerOmitted; - const tail = dropped > 0 ? roomFor([`- ... and ${dropped} more files`]) : 0; - if (entry.length + 1 + tail > left) break; - left -= entry.length + 1; - taken++; - } - if (taken > 0) { - pointerBlock.push(...head, ...pointerEntries.slice(0, taken)); - const dropped = pointerEntries.length - taken + pointerOmitted; - if (dropped > 0) pointerBlock.push(`- ... and ${dropped} more files`); - room -= roomFor(pointerBlock); - } - } - } + const fitted = fitExploreEpilogue({ + room, + noteCandidates: completenessCandidates, + noteYields: trimmedShown.length > 0, + pointerEntries, + pointerOmitted, + }); + const completenessBlock = fitted.note; + const pointerBlock = fitted.pointers; + room = fitted.room; + const keepCompleteness = completenessBlock.length > 0; // Nothing of the pointer list survived, but there WAS one — say so, in the // one line that carries its instruction forward. const pointersLost = pointerEntries.length > 0 && pointerBlock.length === 0; @@ -7048,8 +7312,10 @@ export class ToolHandler { lines.push(...pointerBlock); if (keepCompleteness) lines.push(...completenessBlock); if (keepBudgetNote) lines.push(...budgetBlock); - if (pointersLost && roomFor([EPILOGUE_LOST_NOTE, '']) <= room) { - lines.push('', EPILOGUE_LOST_NOTE); + // Nothing has been cut yet, so every trimmed section is still in the response. + const lostNote = EXPLORE_FALLBACK_NOTES.lost[trimmedShown.length > 0 ? 'trimmed' : 'complete']; + if (pointersLost && roomFor([lostNote, '']) <= room) { + lines.push('', lostNote); } const output = flow.text + lines.join('\n'); @@ -7063,7 +7329,8 @@ export class ToolHandler { const epilogueOnlyCut = epilogueStart < lines.length ? flow.text + lines.slice(0, epilogueStart).join('\n') : null; - const EPILOGUE_CUT_NOTE = '\n\n> (Trailing notes omitted for size. The source above is complete and verbatim — treat it as already Read. For anything this call did not cover, run another codegraph_explore with the specific names rather than reading those files.)'; + const EPILOGUE_CUT_NOTE = EXPLORE_FALLBACK_NOTES.cut[ + epilogueOnlyCut !== null && trimmedIn(epilogueOnlyCut) ? 'trimmed' : 'complete']; if (output.length > hardCeiling && epilogueOnlyCut !== null @@ -7080,7 +7347,7 @@ export class ToolHandler { const lastSection = cut.lastIndexOf('\n' + FILE_SECTION_PREFIX); const boundary = lastSection > hardCeiling * 0.5 ? lastSection : cut.lastIndexOf('\n'); const safe = boundary > 0 ? cut.slice(0, boundary) : cut; - finalText = safe + '\n\n... (output truncated to budget; the source above is complete and verbatim — treat it as already Read. For any area not covered, run another codegraph_explore with the specific names — do NOT Read these files.)'; + finalText = safe + EXPLORE_FALLBACK_NOTES.truncated[trimmedIn(safe) ? 'trimmed' : 'complete']; } else { finalText = output; } From c405ac5e6c340ca52c33d956a3c580a597d19c64 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 18:13:35 +0000 Subject: [PATCH 008/363] fix(windows): make load-sensitive tests wait on real signals; longer pid-file retry (#1773) (#2078) * test: make Windows load-sensitive tests wait on the tree, not the clock (#1773) Each failure from the loaded Windows run was traced to its mechanism and reproduced deterministically before changing it. No assertion is weakened. - daemon-lock-refresh: the PowerShell holder slept 200ms against a 375ms retry budget. Windows refuses a replace-rename while any handle to the target is open, even a Node read handle, so hold one in-process and close it inside the rename mock after the first real failure. Also asserts the first rename really failed with a retryable code. - mcp-projectpath-lifecycle: the owner daemon armed a 500ms idle timer at start and exited (code 0) before a starved engine reached it. It now starts with idle exit off and is armed over IPC once the engines hold a session. The file waits for catch-up instead of the gate's 3s serve-anyway deadline, and its index-heavy hooks and eviction case get explicit timeouts. - mcp-daemon / mcp-writer-lock teardown: racing launchers spawn detached daemon candidates whose cwd is the fixture; a loser still starting at teardown pinned it (EBUSY) or took over after the winner was stopped. A --require preload records candidate pids, and teardown waits for the losers before stopping the winner. - mcp-writer-lock: writer.pid appears before the daemon binds and rewrites daemon.pid, so cleanup could probe a daemon mid-start ('unverified'); wait for both proxies to attach. Removal uses fs.promises.rm, because fs.rmSync gives up on the first Windows access-denied error despite maxRetries. - mcp-daemon #1963: connect only after the proxy attached (lock before bind). - sync-rebuild-convergence / multi-repo-workspace: explicit timeouts scoped to the cases that build real indexes; the global timeout is unchanged. Co-Authored-By: Claude Opus 5.5 * fix(daemon): give the Windows pid-file replace a longer retry budget (#1773) Windows refuses to replace a file while any handle to it is open, including a plain Node read handle, so an antivirus scan, an indexer, or another session reading daemon.pid can each block the daemon's startup refresh for a moment, and for longer on a loaded machine. The retry waited 375ms in all; it now backs off 25, 50, 100, 200, then 400ms three times, about 1.6s over eight attempts, still well inside a launcher's ~6s connect window. Permanent failures and non-Windows errors still abort at once, and an ownership change still stops the retry. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-candidates.ts | 63 +++++++++++++++++++ __tests__/daemon-lock-refresh.test.ts | 44 +++++++------ __tests__/fixtures/record-detached-spawns.cjs | 22 +++++++ __tests__/mcp-daemon.test.ts | 17 ++++- __tests__/mcp-projectpath-lifecycle.test.ts | 60 +++++++++++++++--- __tests__/mcp-writer-lock.test.ts | 50 ++++++++++----- __tests__/multi-repo-workspace.test.ts | 4 +- __tests__/sync-rebuild-convergence.test.ts | 15 ++++- src/mcp/daemon.ts | 18 +++++- 10 files changed, 243 insertions(+), 51 deletions(-) create mode 100644 __tests__/daemon-candidates.ts create mode 100644 __tests__/fixtures/record-detached-spawns.cjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 19ed81c182..047ac1cf8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -171,6 +171,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). #### MCP / indexing +- On Windows, the shared MCP daemon now waits longer for another program — an antivirus scan, an indexer, or another session reading its lock file — to let go of that file, so it starts instead of leaving the session to fall back to a slower in-process server. (#1773) - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. - Incremental sync now keeps edge rebinding crash-safe: replacing a resolved edge with its recovery reference commits atomically, so an interruption cannot permanently remove the relationship. diff --git a/__tests__/daemon-candidates.ts b/__tests__/daemon-candidates.ts new file mode 100644 index 0000000000..d7f31e777a --- /dev/null +++ b/__tests__/daemon-candidates.ts @@ -0,0 +1,63 @@ +/** + * Teardown for tests whose `serve --mcp` launchers race to spawn the shared + * daemon (#1773). Each launcher that finds no daemon spawns a detached + * candidate; one wins the lock and the rest exit once they see it. Stopping the + * winner while a loser is still starting lets that loser take over the fixture + * being removed, and on Windows a live process whose working directory is the + * fixture makes the removal fail with EBUSY or EPERM. So the teardown waits for + * the losers, by pid, before it stops the winner. + */ +import * as fs from 'fs'; +import * as path from 'path'; + +const SPAWN_RECORDER = path.resolve(__dirname, 'fixtures/record-detached-spawns.cjs'); + +/** Where launchers started for `fixture` record their candidates: beside it, never inside. */ +export function spawnLogFor(fixture: string): string { + return path.join(path.dirname(fixture), `${path.basename(fixture)}.daemon-spawns`); +} + +/** Node arguments and environment that make a launcher record its candidates. */ +export function recordSpawns(fixture: string): { args: string[]; env: NodeJS.ProcessEnv } { + return { args: ['--require', SPAWN_RECORDER], env: { CG_TEST_SPAWN_LOG: spawnLogFor(fixture) } }; +} + +function recordedPids(fixture: string): number[] { + let raw = ''; + try { raw = fs.readFileSync(spawnLogFor(fixture), 'utf8'); } catch { return []; } + return raw.split('\n').map(Number).filter((pid) => Number.isSafeInteger(pid) && pid > 0); +} + +function isAlive(pid: number): boolean { + try { process.kill(pid, 0); return true; } catch (err) { + return (err as NodeJS.ErrnoException).code === 'EPERM'; + } +} + +/** + * Resolve once every recorded candidate except the current lock holder has + * exited. Call it after the launchers have exited, so no new candidate can + * appear. The losers leave on their own; this never signals one, so a loser + * that stays alive fails the teardown instead of being hidden. + */ +export async function settleLosingCandidates( + fixture: string, + holder: () => number | null | undefined, + timeoutMs = 30_000, +): Promise { + const deadline = Date.now() + timeoutMs; + for (;;) { + const current = holder(); + const pending = recordedPids(fixture).filter((pid) => pid !== current && isAlive(pid)); + if (pending.length === 0) return; + if (Date.now() > deadline) { + throw new Error(`Daemon candidates ${pending.join(', ')} were still running with ${current ?? 'no process'} holding the lock`); + } + await new Promise((resolve) => setTimeout(resolve, 25)); + } +} + +/** Remove the spawn record once the fixture's processes are gone. */ +export function removeSpawnLog(fixture: string): void { + fs.rmSync(spawnLogFor(fixture), { force: true }); +} diff --git a/__tests__/daemon-lock-refresh.test.ts b/__tests__/daemon-lock-refresh.test.ts index 215c2c22c2..12be3ddb1a 100644 --- a/__tests__/daemon-lock-refresh.test.ts +++ b/__tests__/daemon-lock-refresh.test.ts @@ -2,8 +2,6 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; -import { spawn } from 'child_process'; -import { once } from 'events'; import { refreshDaemonLock, type DaemonLockInfo } from '../src/mcp/daemon'; vi.mock('fs', async (importOriginal) => { @@ -51,7 +49,7 @@ describe('daemon ownership refresh under Windows sharing violations', () => { const error = denied('EPERM'); vi.mocked(fs.renameSync).mockImplementation(() => { throw error; }); expect(() => refreshDaemonLock(pidPath, initial, lock, 'win32')).toThrow(error); - expect(fs.renameSync).toHaveBeenCalledTimes(6); + expect(fs.renameSync).toHaveBeenCalledTimes(8); expect(fs.readFileSync(pidPath, 'utf8')).toBe(initial); noTemporaryFile(); }); @@ -79,25 +77,33 @@ describe('daemon ownership refresh under Windows sharing violations', () => { }, ); - it.runIf(process.platform === 'win32')('waits for a real Windows handle denying delete sharing', async () => { - // Node opens files with delete sharing; a .NET handle lets us reproduce - // the real rename failure deterministically without mocking filesystem I/O. - const script = `$h = [IO.File]::Open('${pidPath.replace(/'/g, "''")}', 'Open', 'Read', 'Read'); ` - + "[Console]::WriteLine('locked'); Start-Sleep -Milliseconds 200; $h.Dispose()"; - const child = spawn('powershell.exe', ['-NoProfile', '-NonInteractive', '-Command', script], { - stdio: ['ignore', 'pipe', 'pipe'], + it.runIf(process.platform === 'win32')('waits for a real Windows handle on the pid file to close', () => { + // Windows will not replace a file while any handle to it is open, even + // one that shares delete access, which is how a scanner, an indexer or + // another process reading the lock trips this path. Hold a real handle, + // let the first real rename fail, and close it only then, so the retry + // follows the release by order rather than racing a timer against the + // backoff (#1773). + const handle = fs.openSync(pidPath, 'r'); + const held: NodeJS.ErrnoException[] = []; + vi.mocked(fs.renameSync).mockImplementationOnce((from, to) => { + try { + realRename(from, to); + } catch (error) { + held.push(error as NodeJS.ErrnoException); + fs.closeSync(handle); + throw error; + } }); - const closed = once(child, 'close'); try { - const [ready] = await once(child.stdout, 'data'); - expect(String(ready)).toContain('locked'); refreshDaemonLock(pidPath, initial, lock); - expect(vi.mocked(fs.renameSync).mock.calls.length).toBeGreaterThan(1); - expect(JSON.parse(fs.readFileSync(pidPath, 'utf8'))).toEqual(lock); - noTemporaryFile(); } finally { - if (child.exitCode === null && child.signalCode === null) child.kill(); - await closed; + if (held.length === 0) fs.closeSync(handle); } - }, 10_000); + expect(held).toHaveLength(1); + expect(['EPERM', 'EACCES', 'EBUSY']).toContain(held[0]!.code); + expect(vi.mocked(fs.renameSync).mock.calls.length).toBeGreaterThan(1); + expect(JSON.parse(fs.readFileSync(pidPath, 'utf8'))).toEqual(lock); + noTemporaryFile(); + }); }); diff --git a/__tests__/fixtures/record-detached-spawns.cjs b/__tests__/fixtures/record-detached-spawns.cjs new file mode 100644 index 0000000000..50ef59f074 --- /dev/null +++ b/__tests__/fixtures/record-detached-spawns.cjs @@ -0,0 +1,22 @@ +/** + * Preloaded with `--require` into the `serve --mcp` launchers a test starts, so + * the test learns the pid of every detached daemon candidate they spawn + * (#1773). Racing launchers may each spawn one; the losers exit on their own + * once they see the winner's lock, but on a loaded machine one can still be + * starting when the test tears down. Written synchronously at spawn time, so + * the pid is on disk before the launcher can be stopped. Candidates inherit + * the preload through `process.execArgv`; they spawn nothing detached. + */ +const childProcess = require('child_process'); +const fs = require('fs'); + +const log = process.env.CG_TEST_SPAWN_LOG; +if (log) { + const spawn = childProcess.spawn; + childProcess.spawn = function recordDetached(...args) { + const child = spawn.apply(this, args); + const options = args.find((arg) => arg !== null && typeof arg === 'object' && !Array.isArray(arg)); + if (options && options.detached && child.pid) fs.appendFileSync(log, `${child.pid}\n`); + return child; + }; +} diff --git a/__tests__/mcp-daemon.test.ts b/__tests__/mcp-daemon.test.ts index eac53516f7..71cc7b71ff 100644 --- a/__tests__/mcp-daemon.test.ts +++ b/__tests__/mcp-daemon.test.ts @@ -43,6 +43,7 @@ import { getDaemonSocketPath } from '../src/mcp/daemon-paths'; import { CodeGraphPackageVersion } from '../src/mcp/version'; import { once } from 'events'; import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; +import { recordSpawns, removeSpawnLog, settleLosingCandidates } from './daemon-candidates'; const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); @@ -53,13 +54,15 @@ interface SpawnedServer { } function spawnServer(cwd: string, env: NodeJS.ProcessEnv = {}, args: string[] = []): SpawnedServer { - const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, BIN, 'serve', '--mcp', ...args], { + // Record the daemon candidates this launcher spawns, for the teardown. + const recorder = recordSpawns(cwd); + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, ...recorder.args, BIN, 'serve', '--mcp', ...args], { cwd, stdio: ['pipe', 'pipe', 'pipe'], // #618: the daemon-attach log line is now off by default; opt the test // harness into it (CODEGRAPH_MCP_LOG_ATTACH=1) so the attach assertions // below can still observe a successful attach. A per-test env still wins. - env: { CODEGRAPH_MCP_LOG_ATTACH: '1', ...process.env, ...env }, + env: { CODEGRAPH_MCP_LOG_ATTACH: '1', ...process.env, ...recorder.env, ...env }, }) as ChildProcessWithoutNullStreams; // Swallow spawn/EPIPE errors so killing a child mid-write can't surface as an // unhandled error that crashes the vitest worker. @@ -205,6 +208,10 @@ describe('Shared MCP daemon (issue #411)', () => { child.kill('SIGKILL'); await exited; })); + // Racing launchers may each have spawned a daemon candidate, and a loser + // can still be starting on a loaded machine. Stopping the winner first + // would let it take over the fixture being removed (#1773). + await settleLosingCandidates(tempDir, () => readLockPid(realRoot)); // The daemon is detached (not a tracked child) — reap it explicitly via the // pid it recorded, so a test can't leak a background daemon. Guard against // our own pid: the version-mismatch test plants `pid: process.pid` in the @@ -216,14 +223,18 @@ describe('Shared MCP daemon (issue #411)', () => { } await new Promise((r) => setTimeout(r, 50)); servers.length = 0; + removeSpawnLog(tempDir); await fs.promises.rm(tempDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); - }); + }, 45_000); it.runIf(process.platform !== 'win32')('stops despite a socket still waiting for its client hello (#1963)', async () => { const server = spawnServer(tempDir); servers.push(server); sendInitialize(server.child, `file://${tempDir}`, 1); await waitFor(() => findResponse(server.stdout, 1), 10000); + // The lock is written before the socket is bound; an attached proxy proves + // the daemon is listening (#1773). + await waitFor(() => server.stderr.some((l) => l.includes('Attached to shared daemon')), 10000); const pid = await waitFor(() => readLockPid(realRoot), 10000); const raw = net.connect(getDaemonSocketPath(realRoot)); try { diff --git a/__tests__/mcp-projectpath-lifecycle.test.ts b/__tests__/mcp-projectpath-lifecycle.test.ts index a8b5d07a45..23a1a36a95 100644 --- a/__tests__/mcp-projectpath-lifecycle.test.ts +++ b/__tests__/mcp-projectpath-lifecycle.test.ts @@ -50,7 +50,10 @@ async function waitFor(check: () => Promise, timeoutMs: number): Promis return check(); } -describe('MCP explicit projectPath lifecycle (#1835)', () => { +// Every case drives real catch-up syncs and waits for them (see the gate note +// below), so each gets room for a loaded Windows VM; cases that do more set +// their own (#1773). These bounds only catch a hang. +describe('MCP explicit projectPath lifecycle (#1835)', { timeout: 30_000 }, () => { let workspace: string; let serviceA: string; let serviceB: string; @@ -58,7 +61,10 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { const engines: MCPEngine[] = []; const children: ChildProcess[] = []; const prevDebounce = process.env.CODEGRAPH_WATCH_DEBOUNCE_MS; + const prevGate = process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS; + // Both hooks build indexes or wait out a sync; a loaded Windows VM exceeds + // the default 10s hook timeout, and has taken over 30s (#1773). beforeEach(async () => { workspace = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-1835-'))); serviceA = path.join(workspace, 'service-a'); @@ -66,6 +72,11 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { await makeProject(serviceA, 'alphaOriginal'); await makeProject(serviceB, 'betaOriginal'); process.env.CODEGRAPH_WATCH_DEBOUNCE_MS = '100'; + // These cases assert what a call left behind, so wait for its catch-up + // rather than the gate's 3s serve-anyway deadline, which a loaded machine + // misses (#1773). That deadline has its own coverage in + // mcp-catchup-gate.test.ts; the eviction case sets its own values. + process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS = '0'; opened.length = 0; onOpen = undefined; __setLoadCodeGraphForTests(RecordingCodeGraph as unknown as typeof CodeGraph); @@ -73,7 +84,7 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { engines.push(engine); // Two indexed children, none at the root: no default project (#1607). await engine.ensureInitialized(workspace); - }); + }, 60_000); afterEach(async () => { await Promise.all(engines.splice(0).map((e) => e.stop())); @@ -88,8 +99,10 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { __setLoadCodeGraphForTests(null); if (prevDebounce === undefined) delete process.env.CODEGRAPH_WATCH_DEBOUNCE_MS; else process.env.CODEGRAPH_WATCH_DEBOUNCE_MS = prevDebounce; + if (prevGate === undefined) delete process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS; + else process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS = prevGate; fs.rmSync(workspace, { recursive: true, force: true }); - }); + }, 60_000); function names(root: string): string[] { const reader = CodeGraph.openSync(root); @@ -97,13 +110,25 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { finally { reader.close(); } } + /** + * A daemon owner starts with idle exit OFF. Armed at startup, its 500ms + * timer raced this process's first connection and lost on a loaded machine, + * so the daemon exited before any engine reached it (#1773). A test that + * needs the idle exit enables it with {@link armIdleExit} once it holds a + * session; the daemon reads the value each time its last client leaves. + */ async function startOwner(mode: 'direct' | 'daemon', slowCatchUp = false): Promise { const modulePath = path.resolve(__dirname, '../dist/mcp'); const ownerScript = mode === 'daemon' ? ` const { Daemon, tryAcquireDaemonLock } = require(process.argv[1] + '/daemon'); const root = process.argv[2]; tryAcquireDaemonLock(root); - const daemon = new Daemon(root, { idleTimeoutMs: 500 }); + const daemon = new Daemon(root, { idleTimeoutMs: 0 }); + process.on('message', (message) => { + if (message !== 'arm-idle') return; + daemon.idleTimeoutMs = 500; + process.send('idle-armed'); + }); daemon.start().then(() => { if (!process.env.CG_TEST_HOLD_CATCHUP) process.send('ready'); }); ` : ` const { MCPEngine } = require(process.argv[1] + '/engine'); @@ -142,6 +167,21 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { return child; } + async function armIdleExit(owner: ChildProcess): Promise { + await new Promise((resolve, reject) => { + const onMessage = (message: unknown): void => { + if (message !== 'idle-armed') return; + owner.off('exit', onExit); + owner.off('message', onMessage); + resolve(); + }; + const onExit = (): void => { owner.off('message', onMessage); reject(new Error('Owner exited before arming idle exit')); }; + owner.on('message', onMessage); + owner.once('exit', onExit); + owner.send('arm-idle'); + }); + } + async function search(projectPath: string, symbol: string): Promise { const res = await engine.getToolHandler().execute('codegraph_search', { query: symbol, projectPath }); expect(res.isError).toBeFalsy(); @@ -357,7 +397,8 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { } }); - // Builds and reconciles nine real indexes; a Windows VM exceeds the default 5s. + // Builds and reconciles nine real indexes; a Windows VM exceeds the default + // 5s, and a loaded one exceeded 15s (#1773). it('defers eviction and shutdown until an active catch-up finishes', async () => { const roots = [serviceA, serviceB]; for (let i = roots.length; i <= MAX_CACHED_PROJECTS; i++) { @@ -398,7 +439,7 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { if (prev === undefined) delete process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS; else process.env.CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS = prev; } - }, 15_000); + }, 60_000); it('drains a tool operation before closing its cached graph', async () => { @@ -438,7 +479,7 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { fs.writeFileSync(path.join(serviceA, 'src/sample.ts'), 'export function realOwnerExit() {}\n'); expect(await waitFor(async () => names(serviceA).includes('realOwnerExit'), 10000)).toBe(true); expect(opened[0]!.isWatching()).toBe(true); - }, 20000); + }, 60_000); it('retains a daemon while either accessing engine remains connected', async () => { const owner = await startOwner('daemon'); @@ -446,6 +487,9 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { const second = new MCPEngine(); engines.push(second); await second.getToolHandler().execute('codegraph_search', { projectPath: serviceA, query: 'alphaOriginal' }); + // Both engines now share one daemon session; from here on the daemon may + // idle out, and only losing that session should make it do so. + await armIdleExit(owner); await engine.stop(); await new Promise((resolve) => setTimeout(resolve, 1000)); expect(owner.exitCode).toBeNull(); @@ -456,6 +500,6 @@ describe('MCP explicit projectPath lifecycle (#1835)', () => { await second.stop(); await exited; expect(owner.exitCode).toBe(0); - }, 20000); + }, 60_000); }); diff --git a/__tests__/mcp-writer-lock.test.ts b/__tests__/mcp-writer-lock.test.ts index b8a1b18b3c..353268b7a3 100644 --- a/__tests__/mcp-writer-lock.test.ts +++ b/__tests__/mcp-writer-lock.test.ts @@ -14,6 +14,7 @@ import type { MCPEngine } from '../src/mcp/engine'; import { createDatabase } from '../src/db/sqlite-adapter'; import { getWriterPidPath } from '../src/mcp/writer-lock'; import { isProcessAlive, stopDaemonAt } from '../src/mcp/daemon-registry'; +import { recordSpawns, removeSpawnLog, settleLosingCandidates } from './daemon-candidates'; // Exercise the shipped lazy CommonJS loader as well as the engine lifecycle. const { MCPEngine: BuiltMCPEngine } = require('../dist/mcp/engine') as typeof import('../src/mcp/engine'); @@ -28,10 +29,12 @@ function spawnMcp( cwd: string, env: NodeJS.ProcessEnv, ): { child: ChildProcessWithoutNullStreams; getStderr: () => string } { - const child = spawn(process.execPath, [BIN, 'serve', '--mcp'], { + // Record the daemon candidates this launcher spawns, for the teardown. + const recorder = recordSpawns(cwd); + const child = spawn(process.execPath, [...recorder.args, BIN, 'serve', '--mcp'], { cwd, stdio: ['pipe', 'pipe', 'pipe'], - env: { ...process.env, ...env }, + env: { ...process.env, ...recorder.env, ...env }, }) as ChildProcessWithoutNullStreams; child.on('error', () => {}); child.stdin.on('error', () => {}); @@ -82,15 +85,20 @@ describe('issue #1740 — direct-mode writer lock', () => { // daemon stop helper verifies its socket identity before signaling it. const childPids = new Set(children.map((c) => c.pid)); const writerPid = readWriterPid(realRoot); - const results = await Promise.allSettled([ - ...children.map(stopChild), - ...(writerPid !== undefined && !childPids.has(writerPid) - ? [stopDaemonAt(realRoot).then((result) => { - expect(result.pid).toBe(writerPid); - expect(result.outcome).not.toBe('unverified'); - })] - : []), - ]); + // With the launchers gone no new daemon candidate can appear. A loser can + // still be starting on a loaded machine; stopping the winner first would + // let it take over the fixture being removed (#1773). + const stopWriter = async (): Promise => { + await settleLosingCandidates(realRoot, () => readWriterPid(realRoot)); + if (writerPid === undefined || childPids.has(writerPid)) return; + const result = await stopDaemonAt(realRoot); + expect(result.pid).toBe(writerPid); + expect(result.outcome).not.toBe('unverified'); + }; + const results = [ + ...await Promise.allSettled(children.map(stopChild)), + ...await Promise.allSettled([stopWriter()]), + ]; for (const result of results) { if (result.status === 'rejected') throw result.reason; } @@ -98,11 +106,16 @@ describe('issue #1740 — direct-mode writer lock', () => { expect(isProcessAlive(writerPid), `Writer ${writerPid} did not exit`).toBe(false); } children.length = 0; - fs.rmSync(tempDir, { recursive: true, force: true, maxRetries: 10 }); + removeSpawnLog(realRoot); + // A stopped daemon's files can stay held for a moment (handle rundown, an + // antivirus scan on close) with nothing to wait on. fs.rmSync gives up on + // the first Windows access-denied error despite maxRetries (Node 24), so + // use the asynchronous removal, which does back off and retry (#1773). + await fs.promises.rm(tempDir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); expect(fs.existsSync(tempDir)).toBe(false); } - afterEach(cleanup, 15000); + afterEach(cleanup, 45_000); it('second CODEGRAPH_NO_DAEMON serve --mcp exits with writer-lock error', async () => { const env = { @@ -163,6 +176,15 @@ describe('issue #1740 — direct-mode writer lock', () => { await sleep(50); } expect(fs.existsSync(lockPath)).toBe(true); + // writer.pid appears before the daemon binds and rewrites daemon.pid. + // Attaching proves both happened, so cleanup never probes a daemon that is + // still starting (#1773). + const attached = () => [a, b].every((p) => p.getStderr().includes('Attached to shared daemon')); + const attachDeadline = Date.now() + 30000; + while (Date.now() < attachDeadline && !attached()) { + await sleep(50); + } + expect(attached()).toBe(true); await sleep(1000); expect(a.child.exitCode).toBeNull(); expect(b.child.exitCode).toBeNull(); @@ -179,7 +201,7 @@ describe('issue #1740 — direct-mode writer lock', () => { expect(isProcessAlive(a.child.pid!)).toBe(false); expect(isProcessAlive(b.child.pid!)).toBe(false); expect(fs.existsSync(tempDir)).toBe(false); - }, 25000); + }, 90_000); }); diff --git a/__tests__/multi-repo-workspace.test.ts b/__tests__/multi-repo-workspace.test.ts index 282834038a..8c9802c016 100644 --- a/__tests__/multi-repo-workspace.test.ts +++ b/__tests__/multi-repo-workspace.test.ts @@ -212,6 +212,8 @@ describe('multi-repo workspaces (#514) + .gitignore-respect default (#970, #976) expect(scope.ignores('src/app.ts')).toBe(false); }); + // Indexes, then syncs, through git in two repositories; on a loaded Windows + // VM that exceeds the default 5s without anything being wrong (#1773). it('sync picks up a change inside an opted-in gitignored embedded repo', async () => { write(path.join(ws, 'packages/proj-a/src/auth.ts'), 'export function login() { return 1; }\n'); makeRepo(path.join(ws, 'packages/proj-a')); @@ -233,7 +235,7 @@ describe('multi-repo workspaces (#514) + .gitignore-respect default (#970, #976) } finally { cg.destroy(); } - }); + }, 60_000); }); describe('discovery/classifier machinery (exercised under opt-in)', () => { diff --git a/__tests__/sync-rebuild-convergence.test.ts b/__tests__/sync-rebuild-convergence.test.ts index 36967d6b86..63cd57fa4f 100644 --- a/__tests__/sync-rebuild-convergence.test.ts +++ b/__tests__/sync-rebuild-convergence.test.ts @@ -44,7 +44,16 @@ import { createDatabase } from '../src/db/sqlite-adapter'; import { QueryBuilder } from '../src/db/queries'; import { ToolHandler } from '../src/mcp/tools'; -describe('Incremental sync converges to a full rebuild (CG-33)', () => { +/** + * Every case here builds a real index, and most rebuild it from scratch to + * compare against, three to nine index passes each. On a Windows VM with + * on-access scanning and a busy host even a single index-and-sync case took + * over 5s and the heaviest 35s, so the default timeout measured the machine + * rather than the tree (#1773). This bound only catches a hang. + */ +const INDEXING_TIMEOUT = { timeout: 60_000 }; + +describe('Incremental sync converges to a full rebuild (CG-33)', INDEXING_TIMEOUT, () => { let testDir: string; let cg: CodeGraph; @@ -491,7 +500,7 @@ describe('Incremental sync converges to a full rebuild (CG-33)', () => { * cannot fix: without it, re-resolving a reference against the very same graph * can still pick a different winner than a rebuild does. */ -describe('Same-name candidate order is content-derived, not insertion-derived (CG-33)', () => { +describe('Same-name candidate order is content-derived, not insertion-derived (CG-33)', INDEXING_TIMEOUT, () => { let testDir: string; let cg: CodeGraph; @@ -522,7 +531,7 @@ describe('Same-name candidate order is content-derived, not insertion-derived (C }); /** Synthesis owns a whole-graph result, including registrations in third files. */ -describe('Synthesized edges converge after sync (#1988)', () => { +describe('Synthesized edges converge after sync (#1988)', INDEXING_TIMEOUT, () => { let dir: string; let cg: CodeGraph; const write = (file: string, content: string) => fs.writeFileSync(path.join(dir, file), content); diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index 706e3e9703..5f44ff6e85 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -546,11 +546,22 @@ export class Daemon { } } +/** + * Waits between retries of the pid-file replace on Windows, doubling to a cap: + * eight attempts wait about 1.6s in all. Windows refuses to replace a file + * while any handle to it is open, so an antivirus scan, an indexer, or another + * session reading the lock can each hold it for a moment, and for longer on a + * loaded machine (#1773). The total stays well inside a launcher's ~6s connect + * window. + */ +const LOCK_REFRESH_RETRY_DELAYS_MS = [25, 50, 100, 200, 400, 400, 400]; + /** * Publish the bound socket without abandoning a live daemon on a transient * Windows sharing violation. Keep this startup step synchronous: accepting a * client before ownership is refreshed could initialize an engine too early. - * Six attempts wait at most 375ms; permanent failures still abort startup. + * Retries wait per {@link LOCK_REFRESH_RETRY_DELAYS_MS}; permanent failures + * still abort startup. */ export function refreshDaemonLock( pidPath: string, @@ -571,10 +582,11 @@ export function refreshDaemonLock( return; } catch (error) { const code = (error as NodeJS.ErrnoException).code; - if (platform !== 'win32' || !['EPERM', 'EACCES', 'EBUSY'].includes(code ?? '') || attempt >= 5) { + const delay = LOCK_REFRESH_RETRY_DELAYS_MS[attempt]; + if (platform !== 'win32' || !['EPERM', 'EACCES', 'EBUSY'].includes(code ?? '') || delay === undefined) { throw error; } - Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 25 * (attempt + 1)); + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, delay); } } } finally { From c3b1348065800f616c8ee6920054d943a7f572a3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 18:14:10 +0000 Subject: [PATCH 009/363] fix(indexing): reject indexAll after a failed store instead of spinning (#1773) (#2079) The in-order commit window waits for the commit cursor to advance before dispatching more parses. After a store fails, flushOrdered returns at once and the cursor never moves, so the backpressure loop spun on microtasks forever: indexAll never settled, a core stayed pinned, and no timer in the process could fire. It reproduces whenever a store error (a closed database, disk full, an I/O error) arrives with more files left than the window holds. In the test suite, which runs from source with a window of one, a test that timed out mid-index and closed its graph in afterEach wedged the whole worker, so a loaded Windows run never produced a report. Stop waiting once a store has failed or the run was aborted, and destroy the parse pool on that error path so its worker threads do not outlive the index. The regression test runs the scenario in a child process under a hard timeout. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/index-store-failure.test.ts | 57 +++++++++++++++++++++++++++ src/extraction/index.ts | 9 ++++- 3 files changed, 65 insertions(+), 2 deletions(-) create mode 100644 __tests__/index-store-failure.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 047ac1cf8d..e7559656f7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -241,6 +241,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, simply reading a file no longer shows up as a pending edit: watcher events whose file still has the indexed size and modification time (NTFS access-time notifications) are ignored. Thanks @radeilic for the report and @JJordan0C. (#1451) - The Windows bundle now includes an extensionless `codegraph` launcher, so Git Bash — and the Claude Code prompt hook that runs through it — can find `codegraph` instead of failing with exit code 127. Thanks @Dj-Khalle. (#1278) - Running `install.sh` from Git Bash, MSYS or Cygwin on Windows now prints the PowerShell install command instead of failing with "unsupported OS". Thanks @Dnllns. (#1294) +- Indexing no longer hangs at full CPU when a database write fails partway through; it now stops and reports the error. (#1773) #### Screens, links and navigation diff --git a/__tests__/index-store-failure.test.ts b/__tests__/index-store-failure.test.ts new file mode 100644 index 0000000000..5d3dd2e9d6 --- /dev/null +++ b/__tests__/index-store-failure.test.ts @@ -0,0 +1,57 @@ +/** + * A store failure during indexAll must reject it, not hang the process (#1773). + * + * The in-order commit window waited for the commit cursor to advance, but after + * a failed store the cursor never moves: flushOrdered returns at once, so the + * backpressure loop spun on microtasks forever. The process pinned a core and + * no timer could fire, so a test that closed its graph mid-index (after a + * timeout) wedged the whole vitest worker. Runs in a child process under a hard + * timeout: a regression would otherwise hang this worker the same way. + */ +import { describe, it, expect } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as path from 'path'; +import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; + +const DIST = path.resolve(__dirname, '../dist'); + +const script = ` +const fs = require('fs'); const os = require('os'); const path = require('path'); +const { default: CodeGraph } = require(path.join(process.argv[1], 'index.js')); +const { ExtractionOrchestrator } = require(path.join(process.argv[1], 'extraction/index.js')); +const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-store-failure-')); +// More files than the commit window holds, so the failed cursor is waited on. +for (let i = 0; i < 12; i++) fs.writeFileSync(path.join(dir, 'f' + i + '.ts'), 'export function f' + i + '() { return ' + i + '; }\\n'); +const store = ExtractionOrchestrator.prototype.storeExtractionResult; +let calls = 0; +ExtractionOrchestrator.prototype.storeExtractionResult = function (...args) { + if (calls++ === 0) throw new Error('injected store failure'); + return store.apply(this, args); +}; +const cg = CodeGraph.initSync(dir, { config: { include: ['**/*.ts'], exclude: [] } }); +cg.indexAll().then( + () => process.stdout.write('resolved\\n'), + (err) => process.stdout.write('rejected: ' + err.message + '\\n'), +).finally(() => { + try { cg.close(); } catch {} + fs.rmSync(dir, { recursive: true, force: true }); +}); +`; + +describe('indexAll after a failed store (#1773)', () => { + it('rejects with the store error instead of spinning', () => { + const child = spawnSync(process.execPath, [...WASM_RUNTIME_FLAGS, '-e', script, DIST], { + encoding: 'utf8', + timeout: 60_000, + env: { + ...process.env, + // The main-thread store path, with the smallest pooled commit window. + CODEGRAPH_NO_STORE_WORKER: '1', + CODEGRAPH_PARSE_WORKERS: '1', + CODEGRAPH_TELEMETRY: '0', + }, + }); + expect(child.signal, `indexAll did not settle: ${child.stderr}`).toBeNull(); + expect(child.stdout.trim()).toBe('rejected: injected store failure'); + }, 90_000); +}); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 9dbc7ef039..399f333acf 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -2235,8 +2235,11 @@ export class ExtractionOrchestrator { // window has room. When nothing is in flight but the window is still full, // the async commit chain is what's behind — await it so the cursor // advances (buffered items hold whole file contents, so this bound is - // load-bearing for memory). - while (nextSeq - nextToStore >= windowSize) { + // load-bearing for memory). Once a store has failed the cursor never + // moves again — flushOrdered returns at once — so stop waiting and let + // the drain below rethrow; waiting would spin on microtasks forever, + // pinning a core and starving every timer in the process. + while (nextSeq - nextToStore >= windowSize && !flushError && !aborted) { if (inFlight.size > 0) await Promise.race(inFlight); else await flushOrdered(); } @@ -2323,6 +2326,8 @@ export class ExtractionOrchestrator { await flushOrdered(); if (flushError) { if (storeWriter) await storeWriter.close(); + // Its worker threads would otherwise outlive the failed index. + if (pool) await pool.destroy(); throw flushError; } // All bundles are posted; wait for the writer to apply them, then close From fef37765c1da96d90b10de4eae4dd9ab3ef0962b Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 28 Sep 2026 23:57:18 +0000 Subject: [PATCH 010/363] =?UTF-8?q?perf(resolution):=20remove=20the=20inde?= =?UTF-8?q?xing=20regressions=20since=20v1.6.0=20(vscode=20stall,=20per-re?= =?UTF-8?q?f=20rescans)=20=E2=80=94=20graph=20byte-identical=20(#2072)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * perf(resolution): read JS/TS parameter lists without exponential backtracking isLocallyBoundJsName (#1759, cd4e65b5) scans a file for a parameter that binds a bare-called name. Its per-parameter pattern `(type)?(default)?\s*` let one `name: T = value` text split several ways, so when the name was absent the search backtracked through every combination. On vscode's markersModel.test.ts, whose helper takes nine typed, defaulted parameters, each `suite`/`test` scan took 30-40s — one pool worker held a whole resolution batch while the rest idled, a 70-400s stall per run. Each parameter now has exactly one parse: its first non-space character after the identifier picks the type, default or bare alternative. The alternatives match exactly the strings the old optional groups did, so the result is identical for every input (4M generated cases and 5.6k real vscode file/name pairs agree); the scan is 0.1-0.3ms on that file. Co-Authored-By: Claude Opus 5.5 * perf(resolution): memoize Python module-file lookups per resolution pass #2034 (5eaa6fed) sent every Python named import whose path the import resolver cannot place — stdlib and third-party modules such as `from unittest import mock` — through findPythonModuleFile, which scans every `__init__.py` file node for a path suffix. It ran once per ref: on django that scan alone went from 0.6s to 1.4s of resolution CPU. The candidates for a module depend only on the module path, so they are now collected once per resolution context, in the same name-lookup order; the importing file is still excluded per call, so the first survivor is the node the scan returned. The memo drops with the other import-resolver memos. Resolution output on django is hash-identical; single-threaded resolution CPU there goes from 6.5-6.8s to 4.9s (v1.6.0: 4.5s). Co-Authored-By: Claude Opus 5.5 * perf(resolution): test sealed-module candidates cheapest-first isSealedModule (#1719, bffd50e4) masks a candidate file's comments and strings before asking whether it is a module that exports nothing. Import refs call it for every same-named candidate across the tree, so on vscode it masked nearly every TypeScript file once per resolver: 3.3s of masking in a single-threaded pass, paid again in each pool worker. The verdict is a pure conjunction, so it now asks the cheap questions first: the source contains `import`, no node in the file is exported, and there is no CommonJS export. Masking runs only for files that pass all three, which in practice are the few that can be sealed. The masker blanks text and never deletes it, so a file without `import` in its source has none in its masked code either. Resolution output is hash-identical on vscode and django. Co-Authored-By: Claude Opus 5.5 * perf(resolution): scan a class for a field's declared type once per field matchTsThisFieldCall (#1496, cece0720) resolves `this..()` by finding the field's declaration in the enclosing class. It re-split the whole file, stripped comments line by line and compiled three patterns for every such call, although the answer depends only on the class and the field: 16.5s of a 148s single-threaded resolution pass on vscode. The first declaring line — `: typeof X`, `: X` or `= new X`, in that order per line — is now memoized per (class node, field), and the last few classes' stripped lines are kept, since calls arrive file by file. Owners are still tried in the same order and everything after the declaration lookup is unchanged. Resolution output is hash-identical on vscode and django. Co-Authored-By: Claude Opus 5.5 * perf(synthesis): start cross-tier receiver chains only at a word's start The cross-tier pass (b1f40c57) scans every non-test JS/TS file whose text has a `.add(`, `.get(` or `.on(` — nearly all of vscode — with receiver- chain patterns such as `a.b.c.add('job')`. A chain could begin at any letter of an identifier, so each letter re-read the dotted chain behind it; on vscode the pass ran 35s on one pool worker and set the synthesis phase's wall. A match that starts mid-word implies one from the word's first letter, and every match of these patterns ends on a non-word character, so a scan never resumes mid-word: a lookbehind that rejects those starts skips only attempts bound to fail. Over all 14,135 vscode JS/TS files the five patterns return identical matches (positions and groups) before and after, in 8.8s instead of 32.8s; the pass itself drops from ~9.8s to ~5.6s CPU standalone. Co-Authored-By: Claude Opus 5.5 * perf(resolution): filter same-named candidates in one pass, cheap checks first Interface members (#1638) and nested const-bound functions (#1669) made common names — `editor`, `model`, `dispose` — carry thousands of nodes. matchByExactName ran ten chained filters over that list for every reference, copying it once per rule before the source-reading checks ran, and the method-call strategies copied it again to put the call site's file first before looking at only the class-like or same-file entries. Each rule judges one candidate on its own, so the exact-name rules now run as a single pass with the kind and language checks ahead of the lexical, sealed-module and local-binding checks; the class strategies narrow to owner kinds before ordering by file (the same nodes in the same order), and the object-literal strategy, which keeps same-file holders only, skips the reordering altogether. Resolution output is hash-identical on vscode and django. Co-Authored-By: Claude Opus 5.5 * perf(synthesis): scope language-specific node scans in SQL Eleven synthesis passes stream every node of a kind and keep only their language's (Go, Java/Kotlin, XML, native, ArkTS, Nix, Erlang). On a TypeScript monorepo the Go passes materialized all 113k vscode methods to find two Go ones — goMethodContains alone went from 0.5s to ~2.9s on the main thread after interface members (#1638) grew the method count and the canonical ORDER BY (#1988) landed. iterateNodesByKindIn adds the language predicate to the same query. Its ORDER BY (file_path, start_line, id) is a total order, so it yields exactly the nodes the callers kept, in the same sequence. Co-Authored-By: Claude Opus 5.5 * perf(synthesis): run the synthesis-site index and source-gate scan beside the passes Two serial main-thread steps landed with the incremental synthesis refresh (#1988, #2038). The bulk-edge window rebuilt idx_edges_synthesis_site before synthesis — its partial predicate runs json_valid over the metadata of every edge (~1.5M rows, 2.7s on vscode) — and after the merged insert synthesis re-read every file in the project to record its source gates (~1.9s). Neither result is read until the index is finished: only sync consults the index, and the gates depend only on the files. The window's end now leaves that one index for later, and the resolver builds it while the pool computes the passes, when the main thread was only waiting; the source-gate scan runs in the same window. Both still happen before resolution returns, whatever path synthesis takes, and a crash in between is healed on the next open as before. On vscode edge-index-recreate drops from 8.3s to 3.6s of serial time; the graph, synthesis_inputs and the final index set are identical. Co-Authored-By: Claude Opus 5.5 * perf(resolution): stop re-reading resolved targets and re-probing missing files Every resolved reference now re-reads its target node several times — the kind gate (#1796), the cross-family language gate (#1986), alias following (#1482) and edge creation — through the query layer's 1,000-entry node cache, while targets recur ~5x (1.46M vscode edges onto 293k nodes). The language gate alone spent ~6s of single-threaded resolution on vscode fetching whole nodes to read one field, and fileExists re-probed the same missing candidate paths (every extension of a `.js` specifier) from every importing file, now under a path.resolve containment check (#1631). The resolver keeps its own id → node cache and a memo of filesystem probes, both in the stable window of its other caches and dropped by clearCaches; the language gate returns early when the reference's language has no code family (no target can cross one) and otherwise memoizes each target's language. Resolution output is hash-identical on vscode and django. Co-Authored-By: Claude Opus 5.5 * perf(resolution): cheaper sealed-module and receiver-less call checks isSealedModule read and decoded a candidate file — its source and every node — to learn whether anything is exported, though nearly every module does. It now asks one indexed EXISTS probe first and reads the source only for files that export nothing (the conjunction is unchanged). isReceiverLessCall (#1759, #1857) compiled a pattern per call and end-anchored two more over the line's prefix, which V8 scans from the start. It now matches the name literally with a sticky `\s*[(<]`, and reads the prefix backwards: the last non-space character decides whether a receiver precedes the call, and the word run ending there decides the keyword exception — exactly what `[.\w$\])]\s*$` and `\b(?:return|...)\s*$` accepted. Resolution output is hash-identical on vscode and django. Co-Authored-By: Claude Opus 5.5 * perf(synthesis): pull passes onto workers heaviest-first; cheaper registry and tier scans The pooled fan-out handed every pass out at once, so each worker held a count-based share and a heavy pass finished only with its worker's whole share (vscode's registry pass ended ~16s in). Workers now pull one pass at a time, heaviest first; edges still merge by registry index, so the output is unchanged. vscode synthesis: 14.5s vs 19.6-24.4s. The registry pass retried its greedy name at every character of every identifier; both scans now start a name only at an identifier's first character or a `this.` (a mid-identifier resume after a `.method` dispatch falls back to the unguarded pattern), and dispatch lines come from a newline-offset lookup instead of splitting the prefix per match. Checked match-for-match against the old patterns on every vscode JS/TS file plus 300k fuzzed strings. The tier pass reads a file's nodes only when a site needs them. Standalone on vscode: registry 5.6s -> 4.5s, tier 5.1s -> 4.4s. Co-Authored-By: Claude Opus 5.5 * perf(resolution): lean export indexes; no per-name regex compiles in field and local-binding checks Three per-lookup costs the resolver pool paid once per worker: - A file's export index decoded every node in the file and read its source, for each file an import reached, in every worker. It now reads only the exported rows (same `start_line, id` order); the default-export binding and names a local `export { a as b }` clause introduces are read on first use, from name-targeted rows. findExportedSymbol across vscode's workers: 12.5s -> 3.7s. - The `this.` declaration scan compiled three patterns per field of every class and walked every class line. Each pattern needs the field as a whole `[\w$#]` token, so a class's lines are indexed by token once and three sticky patterns, compiled once, are tried at the field's token occurrences. Identical on 721k vscode (file, field) pairs; 56.7s -> 1.1s standalone. - isLocallyBoundJsName searched the whole file with four patterns per name. Each can only start at a `const`/`let`/`var` or `function`/`class` keyword, an `=>`, or the `(` opening a list before the name, so the patterns (compiled once per name, sticky) are tried there. Identical on 198k vscode (file, name) pairs; 68.1s -> 10.7s standalone. Replaying every vscode (2.19M) and django (270k) reference single-threaded gives the same result hash before and after. Co-Authored-By: Claude Opus 5.5 * perf(resolution): prepare kind-filtered edge reads once; memoize the inherited this-member walk - getOutgoingEdges / getIncomingEdges with a kind filter prepared a fresh statement on every call, a third of the read's cost; they are issued per node by supertype walks and the member-lookup synthesis passes. Each SQL shape is now prepared once per connection (dropped on rebind). - The post-loop `this.` pass walked the same supertypes for every reference, reading each one's contains edges and decoding members one by one until a name matched. Nothing is written until the pass ends, so each type's supertype edges and each supertype's callable members (by name, in edge order) are read once. vscode: 4.1s -> 1.0s. - Express's mount pass decoded every JS/TS file to look for `.use(`; it now checks the raw bytes first (an ASCII needle survives UTF-8 decoding unchanged) and decodes only files that have it. A full vscode init dumps the identical graph before and after. Co-Authored-By: Claude Opus 5.5 * perf(resolution): fan the next batch out before inserting edges resolution never reads The batch loop inserted every batch's edges before fanning out the next one, so the pool sat idle through each insert. That ordering exists for one reason: later batches walk supertype chains over the `extends` / `implements` edges earlier batches resolved. Those are the only edges resolution reads, and the prerequisite phase drains every such ref first, so after it no batch makes one. A batch without them now fans the next batch out first and inserts while it resolves: the same writes in the same order, and nothing the workers read differs. The kinds live in one constant, shared with the supertype walk. vscode resolution, three interleaved rounds against the parent commit: 50.3-51.0s vs 56.8-59.1s wall (the pool's wait per batch halves; the overlapped inserts cost ~7s more CPU). Full vscode and django inits dump identical graphs before and after, and identical to main. Co-Authored-By: Claude Opus 5.5 * perf(db): prepare the edge-endpoint existence check once Every edge insert checks its endpoints with a 500-id IN query, a chunk at a time, and prepared that statement afresh for each chunk. The batch loop now runs these inserts on the main thread while the pool resolves, so the full-size statement is prepared once; only a final partial chunk prepares ad hoc (as deleteReferencesByRowIds already does). Same SQL, same result. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- __tests__/bare-call-no-method.test.ts | 39 ++ src/db/index.ts | 19 +- src/db/queries.ts | 89 +++- src/index.ts | 3 +- src/resolution/callback-synthesizer.ts | 188 ++++--- src/resolution/frameworks/express.ts | 2 + src/resolution/import-resolver.ts | 128 +++-- src/resolution/index.ts | 220 +++++++-- src/resolution/name-matcher.ts | 658 +++++++++++++++++++------ src/resolution/resolver-pool.ts | 5 + src/resolution/tier-synthesizer.ts | 59 ++- src/resolution/types.ts | 8 + 12 files changed, 1078 insertions(+), 340 deletions(-) diff --git a/__tests__/bare-call-no-method.test.ts b/__tests__/bare-call-no-method.test.ts index ba5e1d3f8e..144ed33868 100644 --- a/__tests__/bare-call-no-method.test.ts +++ b/__tests__/bare-call-no-method.test.ts @@ -130,6 +130,45 @@ describe('a receiver-less JS/TS call never binds to a method (#1714)', () => { expect(names).toContain('test'); }); + it('reads typed, defaulted parameter lists in linear time', async () => { + // A helper with many `name: T = value` parameters made the parameter scan + // backtrack through every way to split each type from its default — tens + // of seconds per called name on vscode's markersModel.test.ts, which + // stalled a whole resolution batch. + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-1714-')); + fs.writeFileSync( + path.join(tempDir, 'harness.ts'), + 'export function suite(name: string, fn: () => void) { fn(); }\nexport function resolve(p: string) { return p; }\n' + ); + const params = Array.from({ length: 14 }, (_, i) => `p${i}: number = ${i} + ${i}`).join(', '); + fs.writeFileSync( + path.join(tempDir, 'markers.test.ts'), + [ + `function aMarker(${params}, related?: string[]): number {`, + ' return p0;', + '}', + 'function settle(value: number = 1, resolve: Handler = noop) {', + ' resolve(value);', + '}', + 'suite("markers", () => {', + ' aMarker(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14);', + ' settle();', + '});', + '', + ].join('\n') + ); + cg = await CodeGraph.init(tempDir, { index: true }); + cg.resolveReferences(); + const harness = cg.getNodesByKind('function').filter((n) => n.filePath === 'harness.ts'); + const suiteFn = harness.find((n) => n.name === 'suite')!; + const resolveFn = harness.find((n) => n.name === 'resolve')!; + const settle = cg.getNodesByKind('function').find((n) => n.name === 'settle')!; + // `suite` is bound nowhere in the test file, so it still resolves across files. + expect(cg.getIncomingEdges(suiteFn.id).some((e) => e.kind === 'calls')).toBe(true); + // `resolve` is a defaulted parameter of `settle` — a local binding. + expect(cg.getOutgoingEdges(settle.id).filter((e) => e.kind === 'calls').map((e) => e.target)).not.toContain(resolveFn.id); + }, 30_000); + it('keeps `other.serialize()` — a call through a receiver', async () => { const callees = await callsFromMethod( [ diff --git a/src/db/index.ts b/src/db/index.ts index 29ae0887f5..cfe7c10a72 100644 --- a/src/db/index.ts +++ b/src/db/index.ts @@ -394,10 +394,11 @@ export class DatabaseConnection { * One yield per statement keeps every stall to a single index build, which * stays inside the window. */ - async endBulkEdgeLoad(): Promise { + async endBulkEdgeLoad(options: { deferSynthesisSite?: boolean } = {}): Promise { const schemaPath = path.join(__dirname, 'schema.sql'); const schema = fs.readFileSync(schemaPath, 'utf-8'); for (const idx of DatabaseConnection.BULK_EDGE_INDEX_NAMES) { + if (options.deferSynthesisSite && idx === DatabaseConnection.SYNTHESIS_SITE_INDEX) continue; const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`)); if (!m) throw new Error(`schema.sql: edge index ${idx} not found for bulk-load recreation`); this.db.exec(m[0]); @@ -405,6 +406,22 @@ export class DatabaseConnection { } } + /** + * The sync-only synthesis-site index (#1988), which endBulkEdgeLoad can + * leave for later. Its partial predicate runs json_valid over the metadata of + * every edge — ~1.5M rows, several seconds on vscode — while nothing before + * the end of an index reads it, so the resolver builds it while the pool is + * busy with synthesis rather than on the critical path. Idempotent. + */ + createSynthesisSiteIndex(): void { + const schema = fs.readFileSync(path.join(__dirname, 'schema.sql'), 'utf-8'); + const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${DatabaseConnection.SYNTHESIS_SITE_INDEX}\\b[^;]*;`)); + if (!m) throw new Error(`schema.sql: edge index ${DatabaseConnection.SYNTHESIS_SITE_INDEX} not found`); + this.db.exec(m[0]); + } + + private static readonly SYNTHESIS_SITE_INDEX = 'idx_edges_synthesis_site'; + /** Recreate the FTS triggers + rebuild if a bulk-load window never closed. */ private healBulkNodeLoad(): void { if (!this.fts5Available) return; diff --git a/src/db/queries.ts b/src/db/queries.ts index 14a6643412..9666a256f5 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -278,6 +278,10 @@ export class QueryBuilder { getUnresolvedByName?: SqliteStatement; getNodesByName?: SqliteStatement; getNodesByNamePrefix?: SqliteStatement; + fileHasExportedNode?: SqliteStatement; + existingNodeIdsFull?: SqliteStatement; + getExportedNodesByFile?: SqliteStatement; + getNodesByFileAndName?: SqliteStatement; getFileNodesByNamePrefix?: SqliteStatement; getNodesByQualifiedNameExact?: SqliteStatement; getNodesByLowerName?: SqliteStatement; @@ -310,6 +314,11 @@ export class QueryBuilder { // (and therefore resolution's insertion-order disambiguation) is identical // to the one-row-per-run path. private batchStmts: Map = new Map(); + // Kind-filtered edge reads build their SQL per call (a variable IN list), + // but from a handful of kind sets: prepare each shape once. Supertype walks + // and the member-lookup passes issue them per node, and preparing cost + // about a third of the read. + private edgeKindStmts: Map = new Map(); private static readonly BATCH_SIZES: readonly number[] = [128, 32, 8, 1]; /** @@ -369,6 +378,17 @@ export class QueryBuilder { this.db = db; this.stmts = {}; this.batchStmts.clear(); + this.edgeKindStmts.clear(); + } + + private edgeKindStmt(sql: string): SqliteStatement { + let stmt = this.edgeKindStmts.get(sql); + if (!stmt) { + if (this.edgeKindStmts.size >= 64) this.edgeKindStmts.delete(this.edgeKindStmts.keys().next().value!); + stmt = this.db.prepare(sql); + this.edgeKindStmts.set(sql, stmt); + } + return stmt; } /** Set the normalized project-name tokens used to down-weight non-discriminative @@ -914,10 +934,17 @@ export class QueryBuilder { const uniqueIds = [...new Set(ids)]; for (let i = 0; i < uniqueIds.length; i += SQLITE_PARAM_CHUNK_SIZE) { const chunk = uniqueIds.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); - const placeholders = chunk.map(() => '?').join(','); - const rows = this.db - .prepare(`SELECT id FROM nodes WHERE id IN (${placeholders})`) - .all(...chunk) as { id: string }[]; + // Every edge insert checks its endpoints here, a chunk at a time: the + // full-size statement is prepared once, the final partial chunk ad hoc. + let stmt: SqliteStatement; + if (chunk.length === SQLITE_PARAM_CHUNK_SIZE) { + stmt = this.stmts.existingNodeIdsFull ??= this.db.prepare( + `SELECT id FROM nodes WHERE id IN (${new Array(SQLITE_PARAM_CHUNK_SIZE).fill('?').join(',')})` + ); + } else { + stmt = this.db.prepare(`SELECT id FROM nodes WHERE id IN (${chunk.map(() => '?').join(',')})`); + } + const rows = stmt.all(...chunk) as { id: string }[]; for (const row of rows) { out.add(row.id); } @@ -947,6 +974,39 @@ export class QueryBuilder { this.nodeCache.clear(); } + /** + * Whether any node in `filePath` is exported — `getNodesByFile(f).some((n) => + * n.isExported)` as one indexed probe, without decoding the file's nodes. + */ + fileHasExportedNode(filePath: string): boolean { + if (!this.stmts.fileHasExportedNode) { + this.stmts.fileHasExportedNode = this.db.prepare( + 'SELECT 1 FROM nodes WHERE file_path = ? AND is_exported = 1 LIMIT 1' + ); + } + return this.stmts.fileHasExportedNode.get(filePath) !== undefined; + } + + /** The exported nodes of a file, in {@link getNodesByFile} order, decoding only those rows. */ + getExportedNodesByFile(filePath: string): Node[] { + if (!this.stmts.getExportedNodesByFile) { + this.stmts.getExportedNodesByFile = this.db.prepare( + 'SELECT * FROM nodes WHERE file_path = ? AND is_exported = 1 ORDER BY start_line, id' + ); + } + return (this.stmts.getExportedNodesByFile.all(filePath) as NodeRow[]).map(rowToNode); + } + + /** The nodes of a file with one name, in {@link getNodesByFile} order, decoding only those rows. */ + getNodesByFileAndName(filePath: string, name: string): Node[] { + if (!this.stmts.getNodesByFileAndName) { + this.stmts.getNodesByFileAndName = this.db.prepare( + 'SELECT * FROM nodes WHERE file_path = ? AND name = ? ORDER BY start_line, id' + ); + } + return (this.stmts.getNodesByFileAndName.all(filePath, name) as NodeRow[]).map(rowToNode); + } + /** * Get all nodes in a file */ @@ -1180,6 +1240,23 @@ export class QueryBuilder { } } + /** + * iterateNodesByKind narrowed to some languages, in the same canonical + * order — the ORDER BY is total (`id` is unique), so this yields exactly the + * nodes a caller filtering iterateNodesByKind by language would keep, in the + * same sequence. A Go pass on a TypeScript monorepo otherwise materialized + * every method in the project to find a couple of Go ones. + */ + *iterateNodesByKindIn(kind: NodeKind, languages: readonly string[]): IterableIterator { + if (languages.length === 0) return; + const stmt = this.db.prepare( + `SELECT * FROM nodes WHERE kind = ? AND language IN (${languages.map(() => '?').join(', ')}) ORDER BY file_path, start_line, id` + ); + for (const row of stmt.iterate(kind, ...languages)) { + yield rowToNode(row as NodeRow); + } + } + /** * Get all nodes in the database */ @@ -1992,7 +2069,7 @@ export class QueryBuilder { } sql += ' ORDER BY kind, target, line, col'; - const rows = this.db.prepare(sql).all(...params) as EdgeRow[]; + const rows = this.edgeKindStmt(sql).all(...params) as EdgeRow[]; return rows.map(rowToEdge); } @@ -2012,7 +2089,7 @@ export class QueryBuilder { getIncomingEdges(targetId: string, kinds?: EdgeKind[]): Edge[] { if (kinds && kinds.length > 0) { const sql = `SELECT * FROM edges WHERE target = ? AND kind IN (${kinds.map(() => '?').join(',')}) ORDER BY kind, source, line, col`; - const rows = this.db.prepare(sql).all(targetId, ...kinds) as EdgeRow[]; + const rows = this.edgeKindStmt(sql).all(targetId, ...kinds) as EdgeRow[]; return rows.map(rowToEdge); } diff --git a/src/index.ts b/src/index.ts index df79743519..f61e161ffd 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1368,7 +1368,8 @@ export class CodeGraph { // just degrade to scans until the recreate. bulkEdgeLoad: { begin: () => this.db.beginBulkEdgeLoad(), - end: () => this.db.endBulkEdgeLoad(), + end: () => this.db.endBulkEdgeLoad({ deferSynthesisSite: true }), + deferred: () => this.db.createSynthesisSiteIndex(), }, refIndexLoad: { begin: () => this.db.beginBulkRefLoad(), diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 23ede99210..0dc21edf92 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -495,9 +495,8 @@ async function arkuiStateBuildEdges(queries: QueryBuilder, ctx: ResolutionContex let scanned255 = 0; const edges: Edge[] = []; const seen = new Set(); - for (const struct of queries.iterateNodesByKind('struct')) { + for (const struct of queries.iterateNodesByKindIn('struct', ['arkts'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (struct.language !== 'arkts') continue; const children = queries.getOutgoingEdges(struct.id, ['contains']) .map((e) => queries.getNodeById(e.target)) .filter((n): n is Node => !!n); @@ -832,18 +831,15 @@ async function goImplementsEdges(queries: QueryBuilder, onYield: MaybeYield): Pr // never the whole struct kind — that array is O(nodes) on struct-heavy // repos like the Linux kernel (#1212). const goStructs: Node[] = []; - for (const s of queries.iterateNodesByKind('struct')) { + for (const s of queries.iterateNodesByKindIn('struct', ['go'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (s.language === 'go') goStructs.push(s); + goStructs.push(s); } const structMethods = new Map>(); for (const s of goStructs) structMethods.set(s.id, methodNameSet(s.id)); - for (const iface of queries.iterateNodesByKind('interface')) { - if ((++scanned255 & 63) === 0) await onYield(); - + for (const iface of queries.iterateNodesByKindIn('interface', ['go'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (iface.language !== 'go') continue; const want = methodNameSet(iface.id); if (want.size === 0) continue; // empty interface (`any`) — would match everything let added = 0; @@ -905,11 +901,8 @@ async function goCrossFileMethodContainsEdges(queries: QueryBuilder, onYield: Ma return i >= 0 ? p.slice(0, i) : ''; }; - for (const method of queries.iterateNodesByKind('method')) { + for (const method of queries.iterateNodesByKindIn('method', ['go'])) { if ((++scanned255 & 63) === 0) await onYield(); - - if ((++scanned255 & 63) === 0) await onYield(); - if (method.language !== 'go') continue; // The receiver type is encoded in the method's qualifiedName as `Recv::name` // (extraction sets `${receiverType}::${name}` for receiver methods). const qn = method.qualifiedName; @@ -1128,9 +1121,8 @@ async function goGrpcStubImplEdges(queries: QueryBuilder, onYield: MaybeYield): const methodNamesByStruct = new Map>(); const methodNodesByStruct = new Map(); const goStructs: Node[] = []; - for (const s of queries.iterateNodesByKind('struct')) { + for (const s of queries.iterateNodesByKindIn('struct', ['go'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (s.language !== 'go') continue; goStructs.push(s); const ms = queries .getOutgoingEdges(s.id, ['contains']) @@ -1762,9 +1754,8 @@ async function rnCrossPlatformEdges(queries: QueryBuilder, onYield: MaybeYield): // impls in ≥2 native languages can pair, so the per-method JS-caller check // below only runs for genuine cross-platform candidates. const byName = new Map(); - for (const m of queries.iterateNodesByKind('method')) { + for (const m of queries.iterateNodesByKindIn('method', [...NATIVE])) { if ((++scanned255 & 63) === 0) await onYield(); - if (!NATIVE.has(m.language)) continue; const key = norm(m.name); const arr = byName.get(key); if (arr) arr.push(m); @@ -1886,17 +1877,16 @@ async function mybatisJavaXmlEdges(queries: QueryBuilder, onYield: MaybeYield): // stream below. Same rowid stream order as matching inline, so the edge // output is byte-identical when mappers do exist. const xmlMethods: Node[] = []; - for (const m of queries.iterateNodesByKind('method')) { + for (const m of queries.iterateNodesByKindIn('method', ['xml'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (m.language === 'xml') xmlMethods.push(m); + xmlMethods.push(m); } if (xmlMethods.length === 0) return edges; // Index Java methods by `::` for O(1) lookup. const javaIndex = new Map(); - for (const m of queries.iterateNodesByKind('method')) { + for (const m of queries.iterateNodesByKindIn('method', ['java', 'kotlin'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (m.language !== 'java' && m.language !== 'kotlin') continue; const parts = m.qualifiedName.split('::'); const last = parts[parts.length - 1]; const cls = parts[parts.length - 2]; @@ -2005,9 +1995,8 @@ async function ginMiddlewareChainEdges(queries: QueryBuilder, ctx: ResolutionCon let scannedFiles = 0; // 1. Find the chain dispatcher(s): a Go method that invokes a `handlers` slice by index. const dispatchers: Node[] = []; - for (const n of queries.iterateNodesByKind('method')) { + for (const n of queries.iterateNodesByKindIn('method', ['go'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (n.language !== 'go') continue; const content = ctx.readFile(n.filePath); const src = content && sliceLines(content, n.startLine, n.endLine); if (src && GIN_DISPATCH_RE.test(src)) dispatchers.push(n); @@ -2233,13 +2222,42 @@ async function reduxThunkEdges(queries: QueryBuilder, ctx: ResolutionContext, on // the SAME file (the cross-file barrel-namespace variant, e.g. trezor's getMethod, is // deferred). Gated on a real object literal with ≥2 entries that RESOLVE to callables (a // `{ width: 5 }` literal resolves to nothing → no edges); fan-out capped. -const REGISTRY_ASSIGN_RE = /(?:(?:const|let|var)\s+([A-Za-z_$][\w$]*)|((?:this\.)?[A-Za-z_$][\w$]*))\s*=\s*\{/g; -const REGISTRY_DISPATCH_RE = /(?:\bnew\s+)?((?:this\.)?[A-Za-z_$][\w$]*)\s*\[\s*([A-Za-z_$][\w$.]*)\s*\]\s*(?:\(|\.[A-Za-z_$])/g; +// Both scans only START a name at an identifier's first character (or at a `this.`): the +// engine otherwise retries the greedy name at every later character of every identifier in +// the file. A match found from inside an identifier always has one from the identifier's +// start (same name tail, same continuation), so the result is unchanged — except where the +// scan itself resumes mid-identifier after a dispatch ending in `.method`, which +// `nextRegistryDispatch` covers with the unguarded pattern. +const REGISTRY_NAME_START = String.raw`(?:(? 0 && at < src.length && IDENT_RUN_CHAR.test(src[at - 1]!) && IDENT_RUN_CHAR.test(src[at]!)) { + for (; at < src.length && IDENT_RUN_CHAR.test(src[at]!); at++) { + REGISTRY_DISPATCH_AT.lastIndex = at; + const m = REGISTRY_DISPATCH_AT.exec(src); + if (m) return m; + } + } + REGISTRY_DISPATCH_RE.lastIndex = at; + return REGISTRY_DISPATCH_RE.exec(src); +} + /** From the index of an opening `{`, return the brace-balanced body up to its matching `}`. */ function braceBody(src: string, openIdx: number): string | null { let depth = 0; @@ -2319,13 +2337,12 @@ async function objectRegistryEdges(ctx: ResolutionContext, onYield: MaybeYield): // 1. Dispatch sites: `(new )?[]` followed by a call or a chained method. // A quoted-string key (`['save']`) does NOT match — that's a static access, not dispatch. - REGISTRY_DISPATCH_RE.lastIndex = 0; + const lineOf = makeLineAt(safe, 1); const dispatches: Array<{ ref: string; line: number; chained: string | null }> = []; - let dm: RegExpExecArray | null; - while ((dm = REGISTRY_DISPATCH_RE.exec(safe))) { + for (let dm = nextRegistryDispatch(safe, 0); dm; dm = nextRegistryDispatch(safe, dm.index + dm[0].length)) { const win = safe.slice(dm.index, dm.index + 160); const cm = /\]\s*\([^)]*\)\s*\.\s*([A-Za-z_$][\w$]*)/.exec(win) || /\]\s*\.\s*([A-Za-z_$][\w$]*)/.exec(win); - dispatches.push({ ref: dm[1]!, line: safe.slice(0, dm.index).split('\n').length, chained: cm ? cm[1]! : null }); + dispatches.push({ ref: dm[1]!, line: lineOf(dm.index), chained: cm ? cm[1]! : null }); } if (!dispatches.length) continue; // Normalize a leading `this.` so a class FIELD-INITIALIZER registry (`commands = {…}`) @@ -2344,7 +2361,7 @@ async function objectRegistryEdges(ctx: ResolutionContext, onYield: MaybeYield): if (!body) continue; const names = registryEntryNames(body); // depth-0 `key: Identifier` entries only if (names.length >= REGISTRY_MIN_ENTRIES) { - registries.set(lhs, { names, line: safe.slice(0, am.index).split('\n').length }); + registries.set(lhs, { names, line: lineOf(am.index) }); } } if (!registries.size) continue; @@ -3185,10 +3202,9 @@ async function nixOptionPathEdges(queries: QueryBuilder, onYield: MaybeYield): P const byFile = new Map(); let scanned = 0; for (const kind of ['variable', 'function'] as NodeKind[]) { - for (const node of queries.iterateNodesByKind(kind)) { + for (const node of queries.iterateNodesByKindIn(kind, ['nix'])) { if ((++scanned255 & 63) === 0) await onYield(); if ((++scanned & 0x3fff) === 0 && onYield) await onYield(); - if (node.language !== 'nix') continue; const segs = nixLeadingPlainSegments(node.name); if (segs.length === 0) continue; const rec: Rec = { @@ -3298,9 +3314,9 @@ async function erlangBehaviourDispatchEdges(queries: QueryBuilder, ctx: Resoluti // Cheap language gate: no Erlang modules → no cost beyond one streamed // kind scan (never a materialized array of every namespace — #1212). const erlangModules: Node[] = []; - for (const n of queries.iterateNodesByKind('namespace')) { + for (const n of queries.iterateNodesByKindIn('namespace', ['erlang'])) { if ((++scanned255 & 63) === 0) await onYield(); - if (n.language === 'erlang') erlangModules.push(n); + erlangModules.push(n); } if (erlangModules.length === 0) return []; @@ -3723,6 +3739,21 @@ export const SYNTH_PASSES: SynthPassDef[] = [ { name: 'nixOptionEdges', gate: (has) => has('nix'), run: (q, _c, y) => nixOptionPathEdges(q, y) }, ]; +/** + * Rough relative cost of the passes that run longest on large repos. Only + * the pooled dispatch ORDER reads it — heaviest first, so the longest passes + * start before the short ones fill the workers; unlisted passes keep registry + * order after these. A wrong hint costs wall time, never edges. + */ +const SYNTH_PASS_COST_HINT: Readonly> = { + cFnPtrEdges: 100, registryEdges: 60, tierEdges: 55, jsxEdges: 25, rnEventEdgesList: 24, + ifaceEdges: 18, flutterEdges: 16, cppEdges: 15, emitterEdges: 12, fieldEdges: 11, + mybatisEdges: 10, vuexEdges: 8, closureCollEdges: 6, piniaEdges: 5, renderEdges: 4, +}; +function synthPassCostHint(name: string): number { + return SYNTH_PASS_COST_HINT[name] ?? 0; +} + /** Fixed non-registry steps: goMethodContains, goImplements, dedupe-merge, insertMergedEdges. */ const FIXED_SYNTH_STEPS = 4; export const SYNTH_PROGRESS_STEPS = SYNTH_PASSES.length + FIXED_SYNTH_STEPS; @@ -3733,11 +3764,14 @@ export async function synthesizeCallbackEdges( // A live resolver pool to fan the independent passes across (structural type // so this file never imports the pool — resolver-worker imports THIS file). // Null/omitted → the sequential path, byte-identical to the pool path. - pool?: { runSynthPass(name: string): Promise<{ edges: Edge[]; ms: number }> } | null, + pool?: { runSynthPass(name: string): Promise<{ edges: Edge[]; ms: number }>; readonly size?: number } | null, // WAL-valve writer backstop (WalCheckpointValve.backpressure), called at // pool-idle points in the edge-insert loops below — the passes themselves // only read; every write in this function happens with the pool idle. - backpressure?: () => Promise | null + backpressure?: () => Promise | null, + // Main-thread work to run while the pool computes the passes (the main + // thread otherwise only waits there) — e.g. building a deferred index. + whilePoolBusy?: () => void ): Promise { // Each sub-pass below is a whole-graph scan, and there are ~30 of them, all // running synchronously on the indexer's main thread. Their AGGREGATE can run @@ -3868,30 +3902,62 @@ export async function synthesizeCallbackEdges( const MAIN_RETRY_MAX_NODES = 1_500_000; const graphNodes = queries.getNodeAndEdgeCount().nodes; + // Files whose text could feed a pass (inputs that produce NO edges included, + // e.g. an over-cap channel: deleting one may make the full pass viable). It + // depends only on the files, so with a pool the main thread reads them while + // the workers run the passes instead of after. + const collectInputs = async (): Promise => { + const found: string[] = []; + for (const file of ctx.getAllFiles()) { + const content = ctx.readFile(file); + if (content !== null && hasSynthesisPattern(file, content)) found.push(file); + await yieldToLoop(); + } + return found; + }; + let inputs: string[] | null = null; + if (pool && gatedIn.length > 1) { - await Promise.all( - gatedIn.map(async (i) => { - const pass = SYNTH_PASSES[i]!; - try { - const out = await pool.runSynthPass(pass.name); - passEdges[i] = out.edges; - markPass(pass.name, out.ms); - } catch (err) { - if (graphNodes > MAIN_RETRY_MAX_NODES) { - // Worker died at a scale where the main-thread retry is a process - // OOM risk: skip the pass, keep the index alive, and say so. - console.error( - `[synthesis] pass '${pass.name}' failed on a worker at ${graphNodes} nodes — skipped (edges from this pass are absent): ${err instanceof Error ? err.message : String(err)}` - ); - markPass(`${pass.name} (skipped at scale)`, 0); - return; - } - // Worker-side failure (crash, OOM, unknown pass after a version - // mismatch): retry this one pass on the main thread. - await runPassOnMain(i); + const runPooled = async (i: number): Promise => { + const pass = SYNTH_PASSES[i]!; + try { + const out = await pool.runSynthPass(pass.name); + passEdges[i] = out.edges; + markPass(pass.name, out.ms); + } catch (err) { + if (graphNodes > MAIN_RETRY_MAX_NODES) { + // Worker died at a scale where the main-thread retry is a process + // OOM risk: skip the pass, keep the index alive, and say so. + console.error( + `[synthesis] pass '${pass.name}' failed on a worker at ${graphNodes} nodes — skipped (edges from this pass are absent): ${err instanceof Error ? err.message : String(err)}` + ); + markPass(`${pass.name} (skipped at scale)`, 0); + return; } - }) + // Worker-side failure (crash, OOM, unknown pass after a version + // mismatch): retry this one pass on the main thread. + await runPassOnMain(i); + } + }; + // One pass per worker at a time, heaviest first. Handing every pass out + // at once split them by count, and a worker interleaves what it holds, so + // a heavy pass finished only with its worker's whole share: on vscode the + // 6s registry pass ended ~16s in. Pulling keeps workers busy until the + // queue drains. Edges merge by registry index below, so order is free. + const lanes = Math.min(pool.size ?? gatedIn.length, gatedIn.length); + const queue = [...gatedIn].sort( + (a, b) => synthPassCostHint(SYNTH_PASSES[b]!.name) - synthPassCostHint(SYNTH_PASSES[a]!.name) || a - b ); + const lane = async (): Promise => { + for (let i = queue.shift(); i !== undefined; i = queue.shift()) await runPooled(i); + }; + const fanOut = Promise.all(Array.from({ length: lanes }, lane)); + // Observed now, awaited below: a rejection while the scan runs must not + // surface as an unhandled one. + fanOut.catch(() => undefined); + whilePoolBusy?.(); + inputs = await collectInputs(); + await fanOut; } else { for (const i of gatedIn) { await runPassOnMain(i); @@ -3915,15 +3981,7 @@ export async function synthesizeCallbackEdges( await yieldToLoop(); await foldIfOver(); } - // Remember source gates, including inputs that currently produce NO edges - // (e.g. an over-cap channel). Deleting one may make the full pass viable. - const inputs: string[] = []; - for (const file of ctx.getAllFiles()) { - const content = ctx.readFile(file); - if (content !== null && hasSynthesisPattern(file, content)) inputs.push(file); - await yieldToLoop(); - } - queries.replaceSynthesisInputs(inputs); + queries.replaceSynthesisInputs(inputs ?? await collectInputs()); __mark('insertMergedEdges'); return merged.length + goImpl.length + goMethodContains.length; } diff --git a/src/resolution/frameworks/express.ts b/src/resolution/frameworks/express.ts index 8980d29e29..7aaa8af7f0 100644 --- a/src/resolution/frameworks/express.ts +++ b/src/resolution/frameworks/express.ts @@ -319,6 +319,8 @@ export const expressResolver: FrameworkResolver = { const files = context.getAllFiles().filter((f) => /\.(m?js|tsx?|cjs)$/.test(f)); const mounts = new Map>(); for (const file of files) { + // Most files mount nothing: look for `.use(` before decoding the file. + if (context.fileContains && !context.fileContains(file, '.use(')) continue; const content = context.readFile(file); if (!content || !content.includes('.use(')) continue; const lang = detectLanguage(file); diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index e94d56a34f..70e018f57a 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -76,8 +76,15 @@ const exportedSymbolMemos = new WeakMap; defaultComponent: Node | undefined; defaultFnClass: Node | undefined; @@ -87,9 +94,16 @@ interface FileExportIndex { * guess. `const Home = () => …; export default Home` and the namespace * object `const UploadApi = { uploadARCapture }; export default UploadApi` * are both invisible to the `isExported` index above: neither declaration - * has an `export_statement` ancestor. + * has an `export_statement` ancestor. `undefined` until first read through + * {@link defaultExportBindingNode}; `null` when there is none. + */ + defaultBinding?: Node | null; + /** + * Names a local export clause (`export { impl as alias }`) binds to a + * declaration the extractor never flagged isExported, for names not in + * `byName`. `undefined` until first read through {@link exportedByName}. */ - defaultBinding: Node | undefined; + clauseAliases?: Map; } const DEFAULT_BINDING_KINDS = new Set(['function', 'class', 'component', 'constant', 'variable']); @@ -113,38 +127,57 @@ function getFileExportIndex(filePath: string, context: ResolutionContext): FileE } let idx = perFile.get(filePath); if (!idx) { - idx = { byName: new Map(), defaultComponent: undefined, defaultFnClass: undefined, defaultBinding: undefined }; - const nodesInFile = context.getNodesInFile(filePath); - // Every declaration, exported or not: a local `export { impl as alias }` - // clause exports a declaration the extractor never flagged isExported. - const declared = new Map(); - for (const n of nodesInFile) { - if (!declared.has(n.name)) declared.set(n.name, n); - if (!n.isExported) continue; + idx = { byName: new Map(), defaultComponent: undefined, defaultFnClass: undefined }; + const exported = context.getExportedNodesInFile?.(filePath) ?? context.getNodesInFile(filePath).filter((n) => n.isExported); + for (const n of exported) { if (!idx.byName.has(n.name)) idx.byName.set(n.name, n); if (idx.defaultComponent === undefined && n.kind === 'component') idx.defaultComponent = n; if (idx.defaultFnClass === undefined && (n.kind === 'function' || n.kind === 'class')) idx.defaultFnClass = n; } + perFile.set(filePath, idx); + } + return idx; +} + +/** The file's nodes named `name`, in `getNodesInFile` order. */ +function nodesInFileNamed(filePath: string, name: string, context: ResolutionContext): Node[] { + return context.getNodesInFileNamed?.(filePath, name) ?? context.getNodesInFile(filePath).filter((n) => n.name === name); +} + +/** The declaration `export default NAME` names in this file (see FileExportIndex.defaultBinding). */ +function defaultExportBindingNode(filePath: string, idx: FileExportIndex, context: ResolutionContext): Node | undefined { + if (idx.defaultBinding === undefined) { const bound = defaultExportBinding(filePath, context); - if (bound !== null) { - idx.defaultBinding = nodesInFile - .filter((n) => n.name === bound && DEFAULT_BINDING_KINDS.has(n.kind)) - .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn)[0]; - } + idx.defaultBinding = + bound === null + ? null + : (nodesInFileNamed(filePath, bound, context) + .filter((n) => DEFAULT_BINDING_KINDS.has(n.kind)) + .sort((a, b) => a.startLine - b.startLine || a.startColumn - b.startColumn)[0] ?? null); + } + return idx.defaultBinding ?? undefined; +} + +/** What this file exports as `name`: an exported declaration, else a local export clause's binding. */ +function exportedByName(filePath: string, idx: FileExportIndex, name: string, context: ResolutionContext): Node | undefined { + const direct = idx.byName.get(name); + if (direct) return direct; + if (idx.clauseAliases === undefined) { // Bind names introduced by a local export clause to their declarations, so // an importer asking for the renamed name gets the real symbol instead of // falling through to the name-matcher (which cannot cross the rename). + // The declaration is the file's first node of that name, exported or not. + idx.clauseAliases = new Map(); const content = context.readFile(filePath); if (content && content.includes('export')) { for (const { exportedName, localName } of extractLocalExportAliases(content)) { - if (idx.byName.has(exportedName)) continue; - const decl = declared.get(localName); - if (decl) idx.byName.set(exportedName, decl); + if (idx.byName.has(exportedName) || idx.clauseAliases.has(exportedName)) continue; + const decl = nodesInFileNamed(filePath, localName, context)[0]; + if (decl) idx.clauseAliases.set(exportedName, decl); } } - perFile.set(filePath, idx); } - return idx; + return idx.clauseAliases.get(name); } /** Drop the per-context memo tables (see ReferenceResolver.clearCaches). */ @@ -154,6 +187,7 @@ export function clearImportResolverMemos(context: ResolutionContext): void { fileExportIndexes.delete(context); luaFileBasenameIndexes.delete(context); cobolCopybookIndexes.delete(context); + pythonModuleFileMemos.delete(context); } export function resolveImportPath( @@ -1977,23 +2011,48 @@ function resolveModuleImportToFile( * no edge. Shared by absolute `import a.b.c` and absolute `from a.b import c` * (where `c` is a submodule) resolution. */ +/** + * Per-context memo for findPythonModuleFile: module path → the `.py` and + * `/__init__.py` file nodes whose path ends with it, in name-lookup + * order. Only the importing file's own path is excluded per call, so taking + * the first survivor returns the node the unmemoized scan found. Without it, + * every ref naming a module outside the project (`from unittest import mock`) + * rescanned every `__init__.py` in the tree. Same stable window as the name + * caches; dropped by clearImportResolverMemos. + */ +const pythonModuleFileMemos = new WeakMap>(); + function findPythonModuleFile( mod: string, context: ResolutionContext, excludeFilePath: string ): Node | null { if (!mod || mod.startsWith('.')) return null; // relative imports handled elsewhere - const rel = mod.replace(/\./g, '/'); - const lastSeg = mod.split('.').pop()!; - const endsWith = (p: string, want: string): boolean => p === want || p.endsWith('/' + want); - const moduleFile = context - .getNodesByName(`${lastSeg}.py`) - .find((n) => n.kind === 'file' && n.filePath !== excludeFilePath && endsWith(n.filePath, `${rel}.py`)); - if (moduleFile) return moduleFile; - const pkgFile = context - .getNodesByName('__init__.py') - .find((n) => n.kind === 'file' && n.filePath !== excludeFilePath && endsWith(n.filePath, `${rel}/__init__.py`)); - return pkgFile ?? null; + let memo = pythonModuleFileMemos.get(context); + if (!memo) { + memo = new Map(); + pythonModuleFileMemos.set(context, memo); + } + let files = memo.get(mod); + if (!files) { + const rel = mod.replace(/\./g, '/'); + const lastSeg = mod.split('.').pop()!; + const endsWith = (p: string, want: string): boolean => p === want || p.endsWith('/' + want); + files = { + module: context + .getNodesByName(`${lastSeg}.py`) + .filter((n) => n.kind === 'file' && endsWith(n.filePath, `${rel}.py`)), + pkg: context + .getNodesByName('__init__.py') + .filter((n) => n.kind === 'file' && endsWith(n.filePath, `${rel}/__init__.py`)), + }; + memo.set(mod, files); + } + return ( + files.module.find((n) => n.filePath !== excludeFilePath) ?? + files.pkg.find((n) => n.filePath !== excludeFilePath) ?? + null + ); } /** @@ -2353,13 +2412,14 @@ function findExportedSymbolWalk( // resolves and the component shows a false 0 callers (#629). // A component file IS its default export; otherwise the statement that // names the binding beats the first-exported-function guess. - const direct = exportIndex.defaultComponent ?? exportIndex.defaultBinding ?? exportIndex.defaultFnClass; + const direct = + exportIndex.defaultComponent ?? defaultExportBindingNode(filePath, exportIndex, context) ?? exportIndex.defaultFnClass; if (direct) return direct; } else if (want.isNamespace && want.memberName) { - const direct = exportIndex.byName.get(want.memberName); + const direct = exportedByName(filePath, exportIndex, want.memberName, context); if (direct) return direct; } else { - const direct = exportIndex.byName.get(want.exportedName); + const direct = exportedByName(filePath, exportIndex, want.exportedName, context); if (direct) return direct; } diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 98e2473923..17f1300b91 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -45,6 +45,14 @@ const SUPERTYPE_BEARING_KINDS = new Set([ 'class', 'struct', 'interface', 'trait', 'protocol', 'enum', ]); +/** + * The only edges resolving a reference reads (getSupertypes' walk). The batch + * loop commits a batch that made any of these before the next batch fans out; + * its other edges land while that batch resolves. A resolution strategy that + * reads another edge kind must be added here. + */ +const RESOLUTION_READ_EDGE_KINDS: Edge['kind'][] = ['implements', 'extends']; + // SUPERTYPE_TARGET_KINDS (the kinds an extends/implements edge may TARGET) // lives in ./types — the name-matcher needs the same set to restrict its // candidate pool before ranking. It is deliberately wider than @@ -242,6 +250,12 @@ export class ReferenceResolver { private lowerNameCache: LRUCache; // lower(name) → nodes cache private qualifiedNameCache: LRUCache; // qualified_name → nodes cache private fileLinesCache: LRUCache; // file → split lines cache + // id → node for the resolver's own point reads. Every resolved reference + // re-reads its target a few times (kind gate, language gate, alias + // following, edge creation) and targets recur ~5× on large repos; the query + // layer's cache is too small for that working set. Same stable window as the + // name caches above; absent ids are not cached. + private nodeByIdCache: LRUCache; private methodMatchCache: LRUCache; // lang\0Type::method → matching method nodes // Per-(language, methodName) owner index for getMethodMatches: buckets a // method name's candidates by their qualifiedName's last two segments so a @@ -278,6 +292,8 @@ export class ReferenceResolver { // resolution pass (same lifetime assumption as nameCache); clearCaches() resets // it between passes. Callers must treat the returned array as read-only. private nodesByKindCache = new Map(); + // Filesystem existence probes behind context.fileExists (paths not in knownFiles). + private fileExistsMemo = new Map(); private knownNames: Set | null = null; // all known symbol names for fast pre-filtering private knownFiles: Set | null = null; private cachesWarmed = false; @@ -308,6 +324,7 @@ export class ReferenceResolver { // Split-lines arrays are heavier than content strings; refs arrive // file-ordered, so a small cache still hits nearly always. this.fileLinesCache = new LRUCache(contentLimit); + this.nodeByIdCache = new LRUCache(Math.max(limit * 4, 20_000)); this.methodMatchCache = new LRUCache(limit); this.context = this.createContext(); @@ -392,6 +409,15 @@ export class ReferenceResolver { this.cachesWarmed = true; } + /** `queries.getNodeById` through the resolver's own id → node cache. */ + private nodeById(id: string): Node | null { + const cached = this.nodeByIdCache.get(id); + if (cached !== undefined) return cached; + const node = this.queries.getNodeById(id); + if (node) this.nodeByIdCache.set(id, node); + return node; + } + /** * Clear internal caches */ @@ -404,11 +430,13 @@ export class ReferenceResolver { this.lowerNameCache.clear(); this.qualifiedNameCache.clear(); this.fileLinesCache.clear(); + this.nodeByIdCache.clear(); this.methodMatchCache.clear(); this.methodOwnerIndexCache.clear(); this.supertypeMemo.clear(); this.supertypeGen++; this.nodesByKindCache.clear(); + this.fileExistsMemo.clear(); this.knownNames = null; this.knownFiles = null; this.cachesWarmed = false; @@ -446,6 +474,23 @@ export class ReferenceResolver { } } + /** + * `readFileCached(filePath)?.includes(needle)` for an ASCII needle, searched + * in the raw bytes: an ASCII byte sequence survives UTF-8 decoding + * unchanged, and decoding is most of reading a file nobody keeps. + */ + private fileContains(filePath: string, needle: string): boolean { + if (this.fileCache.has(filePath)) return this.fileCache.get(filePath)?.includes(needle) ?? false; + const fullPath = path.join(this.projectRoot, filePath); + try { + const stats = fs.statSync(fullPath); + if (!stats.isFile() || stats.size > MAX_SOURCE_FILE_SIZE_BYTES) return false; + return fs.readFileSync(fullPath).includes(needle); + } catch { + return false; + } + } + /** * Create the resolution context */ @@ -459,6 +504,21 @@ export class ReferenceResolver { return this.nodeCache.get(filePath)!; }, + fileHasExportedNode: (filePath: string) => { + const cached = this.nodeCache.get(filePath); + return cached !== undefined ? cached.some((n) => n.isExported) : this.queries.fileHasExportedNode(filePath); + }, + + getExportedNodesInFile: (filePath: string) => { + const cached = this.nodeCache.get(filePath); + return cached !== undefined ? cached.filter((n) => n.isExported) : this.queries.getExportedNodesByFile(filePath); + }, + + getNodesInFileNamed: (filePath: string, name: string) => { + const cached = this.nodeCache.get(filePath); + return cached !== undefined ? cached.filter((n) => n.name === name) : this.queries.getNodesByFileAndName(filePath, name); + }, + getNodesByName: (name: string) => { const cached = this.nameCache.get(name); if (cached !== undefined) return cached; @@ -568,18 +628,30 @@ export class ReferenceResolver { // calls per probe (~70x slower here). It would also be wrong to apply // — indexing deliberately follows in-root symlinks whose targets live // outside the root (#935), so only the `../` escape is refused. + // Memoized: import resolution probes the same candidate paths (every + // extension of a specifier) from every file that imports it, and the + // tree does not change within a resolution pass (same window as the + // file-content cache; dropped by clearCaches). + const probed = this.fileExistsMemo.get(filePath); + if (probed !== undefined) return probed; const fullPath = lexicalPathWithinRoot(this.projectRoot, filePath); - if (fullPath === null) return false; - try { - return fs.existsSync(fullPath); - } catch (error) { - logDebug('Error checking file existence', { filePath, error: String(error) }); - return false; + let exists = false; + if (fullPath !== null) { + try { + exists = fs.existsSync(fullPath); + } catch (error) { + logDebug('Error checking file existence', { filePath, error: String(error) }); + } } + if (this.fileExistsMemo.size >= 200_000) this.fileExistsMemo.clear(); + this.fileExistsMemo.set(filePath, exists); + return exists; }, readFile: (filePath: string) => this.readFileCached(filePath), + fileContains: (filePath: string, needle: string) => this.fileContains(filePath, needle), + getFileLines: (filePath: string) => { const cached = this.fileLinesCache.get(filePath); if (cached !== undefined) return cached; @@ -622,7 +694,7 @@ export class ReferenceResolver { }, getNodeById: (id: string) => { - return this.queries.getNodeById(id); + return this.nodeById(id); }, getSupertypes: (typeName: string, language) => { @@ -648,8 +720,8 @@ export class ReferenceResolver { } else { const supertypes = new Set(); for (const tn of typeNodes) { - for (const edge of this.queries.getOutgoingEdges(tn.id, ['implements', 'extends'])) { - const target = this.queries.getNodeById(edge.target); + for (const edge of this.queries.getOutgoingEdges(tn.id, RESOLUTION_READ_EDGE_KINDS)) { + const target = this.nodeById(edge.target); if (target?.name && target.name !== typeName) supertypes.add(target.name); } } @@ -903,7 +975,7 @@ export class ReferenceResolver { : this.gateLanguage(candidate, ref); if (!resolved || ref.referenceKind !== 'calls') return resolved; - const target = this.queries.getNodeById(resolved.targetNodeId); + const target = this.nodeById(resolved.targetNodeId); if (!target) return resolved; const dot = ref.referenceName.lastIndexOf('.'); @@ -990,7 +1062,7 @@ export class ReferenceResolver { } const viaImport = this.gateLanguage(resolveViaImport(ref, this.context), ref); if (viaImport) { - const target = this.queries.getNodeById(viaImport.targetNodeId); + const target = this.nodeById(viaImport.targetNodeId); if ( target && (target.kind === 'function' || @@ -1104,7 +1176,7 @@ export class ReferenceResolver { // matcher would link each `mkOption` call to whichever file's inherit // binding it happened to pick. Same-file matches only. if (nameResult) { - const target = this.queries.getNodeById(nameResult.targetNodeId); + const target = this.nodeById(nameResult.targetNodeId); // A definition its language makes file-local — a C `static`, a Kotlin // `private fun`, a Go unexported name in another package, a Rust // non-`pub` item outside its module subtree — cannot be what a name in @@ -1172,9 +1244,9 @@ export class ReferenceResolver { // Promote "extends" to "implements" when a class/struct targets an interface if (kind === 'extends') { - const targetNode = this.queries.getNodeById(ref.targetNodeId); + const targetNode = this.nodeById(ref.targetNodeId); if (targetNode && (targetNode.kind === 'interface' || targetNode.kind === 'protocol')) { - const sourceNode = this.queries.getNodeById(ref.original.fromNodeId); + const sourceNode = this.nodeById(ref.original.fromNodeId); if (sourceNode && sourceNode.kind !== 'interface' && sourceNode.kind !== 'protocol') { kind = 'implements'; } @@ -1187,7 +1259,7 @@ export class ReferenceResolver { // apart from a function call without symbol info, but resolution // can: if `Foo` resolves to a class, the call IS an instantiation. if (kind === 'calls') { - const targetNode = this.queries.getNodeById(ref.targetNodeId); + const targetNode = this.nodeById(ref.targetNodeId); if ( targetNode && (targetNode.kind === 'class' || targetNode.kind === 'struct' || targetNode.kind === 'union') @@ -1693,7 +1765,12 @@ export class ReferenceResolver { // a 22GB WAL on a 4.6GB DB (migration plan §7a.1). parallel?: { dbPath: string; - bulkEdgeLoad?: { begin: () => void; end: () => void | Promise }; + bulkEdgeLoad?: { + begin: () => void; + end: () => void | Promise; + /** Indexes `end` leaves for later; built while the pool runs synthesis. */ + deferred?: () => void; + }; /** unresolved_refs index window for the batched loop — the loop only * reads the status index + PK; dropping the sync-path ref indexes cuts * each per-batch DELETE's B-tree work (DatabaseConnection.beginBulkRefLoad). */ @@ -1981,16 +2058,23 @@ export class ReferenceResolver { const PERSIST_CHUNK = 1000; const tPersist = Date.now(); - // Persist edges BEFORE fanning out the next batch: later batches read - // this batch's edges — resolveMethodOnType walks supertype chains over - // `extends`/`implements` edges that earlier batches resolved, so a - // receiver typed as a subclass only reaches a method declared on its - // base class if those edges are visible. (Validated on dubbo: fanning - // out first downgraded exactly those supertype-method resolutions from - // the 0.9 typed-receiver path to the 0.65 word-overlap fallback.) + // A batch that made `extends`/`implements` edges persists them BEFORE + // fanning out the next batch: later batches read them — + // resolveMethodOnType walks supertype chains over edges that earlier + // batches resolved, so a receiver typed as a subclass only reaches a + // method declared on its base class if those edges are visible. + // (Validated on dubbo: fanning out first downgraded exactly those + // supertype-method resolutions from the 0.9 typed-receiver path to the + // 0.65 word-overlap fallback.) They are the only edges resolution reads + // (RESOLUTION_READ_EDGE_KINDS), so a batch that made none — every batch + // after the prerequisite phase readNextBatch drains first — fans the + // next batch out now and inserts while it resolves: the same writes in + // the same order, and nothing the workers read differs. tLp = Date.now(); const edges = this.createEdges(result.resolved); lp('createEdges', tLp); + const readEdge = edges.some((e) => RESOLUTION_READ_EDGE_KINDS.includes(e.kind)); + let nextInFlight = !readEdge && nextBatch.length > 0 ? beginBatch(nextBatch) : null; tLp = Date.now(); for (let i = 0; i < edges.length; i += PERSIST_CHUNK) { this.queries.insertEdges(edges.slice(i, i + PERSIST_CHUNK)); @@ -1998,11 +2082,12 @@ export class ReferenceResolver { } lp('insertEdges', tLp); - // NOW fan the next batch out — workers see exactly the edge state the - // sequential baseline would (every batch ≤ this one committed), while - // the main thread spends the REST of the persist (ref deletes + failed - // parking below) overlapped with their resolution — the double-buffer. - const nextInFlight = nextBatch.length > 0 ? beginBatch(nextBatch) : null; + // NOW fan the next batch out (if not already) — workers see exactly the + // edge state the sequential baseline would (every batch ≤ this one + // committed), while the main thread spends the REST of the persist (ref + // deletes + failed parking below) overlapped with their resolution — + // the double-buffer. + if (!nextInFlight && nextBatch.length > 0) nextInFlight = beginBatch(nextBatch); // Clean up resolved refs so they don't appear in the next batch — // by row id, so a same-key sibling ref in a LATER batch (same caller @@ -2113,6 +2198,19 @@ export class ReferenceResolver { } } + // Indexes the bulk-edge window's end deferred: built once, while the pool + // is busy with synthesis when it is, and in any case before this returns. + let deferredBuilt = !bulkEdgesActive || !parallel?.bulkEdgeLoad?.deferred; + const buildDeferredIndexes = (): void => { + if (deferredBuilt) return; + deferredBuilt = true; + const tDeferred = Date.now(); + try { + parallel!.bulkEdgeLoad!.deferred!(); + } catch { /* healed by schema.sql on the next open, like a crash in the window */ } + if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] deferred-edge-index: ${Date.now() - tDeferred}ms`); + }; + // Dynamic-edge synthesis: now that all base `calls` edges are persisted, // synthesize observer/callback dispatch edges (dispatcher → registered // callbacks) that static parsing leaves out. Best-effort — never fail the @@ -2127,11 +2225,13 @@ export class ReferenceResolver { this.context, onSynthesisProgress, pool, - parallel?.backpressure + parallel?.backpressure, + buildDeferredIndexes ); } catch { // synthesis is additive and optional; ignore failures } + buildDeferredIndexes(); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] callback-synthesis: ${Date.now() - tSynth}ms`); } finally { if (pool) await pool.destroy().catch(() => undefined); @@ -2349,7 +2449,7 @@ export class ReferenceResolver { * Get file path from node ID */ private getFilePathFromNodeId(nodeId: string): string { - const node = this.queries.getNodeById(nodeId); + const node = this.nodeById(nodeId); return node?.filePath || ''; } @@ -2357,7 +2457,7 @@ export class ReferenceResolver { * Get language from node ID */ private getLanguageFromNodeId(nodeId: string): UnresolvedRef['language'] { - const node = this.queries.getNodeById(nodeId); + const node = this.nodeById(nodeId); return node?.language || 'unknown'; } @@ -2515,7 +2615,7 @@ export class ReferenceResolver { private resolveThisMemberFnRef(ref: UnresolvedRef): ResolvedRef | null { const member = ref.referenceName.slice('this.'.length); if (!member) return null; - const fromNode = this.queries.getNodeById(ref.fromNodeId); + const fromNode = this.nodeById(ref.fromNodeId); if (!fromNode) return null; // A hook declared at class-body level (Ruby `before_action :authenticate`) // attributes to the CLASS node itself — its qualified name IS the scope. @@ -2572,10 +2672,38 @@ export class ReferenceResolver { // periodically so the #850 liveness watchdog heartbeat can fire (#1091). const maybeYield = createYielder(); const resolved: ResolvedRef[] = []; + // Refs share supertypes (every `this.dispose` walks the same chain), and + // nothing is written until the loop ends: read each type's supertype + // edges, and each supertype's callable members by name, once. + const supertypeEdges = new Map(); + const callableMembers = new Map>(); + const supertypesOf = (id: string): Edge[] => { + let edges = supertypeEdges.get(id); + if (!edges) { + edges = this.queries.getOutgoingEdges(id, ['implements', 'extends']); + supertypeEdges.set(id, edges); + } + return edges; + }; + const callableMembersOf = (id: string): Map => { + let byName = callableMembers.get(id); + if (!byName) { + byName = new Map(); + for (const c of this.queries.getOutgoingEdges(id, ['contains'])) { + const m = this.nodeById(c.target); + if (!m || (m.kind !== 'function' && m.kind !== 'method')) continue; + const list = byName.get(m.name); + if (list) list.push(m); + else byName.set(m.name, [m]); + } + callableMembers.set(id, byName); + } + return byName; + }; for (const ref of deferred) { await maybeYield(); const member = ref.referenceName.slice('this.'.length); - const fromNode = this.queries.getNodeById(ref.fromNodeId); + const fromNode = this.nodeById(ref.fromNodeId); if (!fromNode || !member) continue; // Class-body-level hooks (Ruby) attribute to the CLASS node itself. let className: string; @@ -2619,24 +2747,16 @@ export class ReferenceResolver { for (let depth = 0; depth < 5 && frontierNodes.length > 0 && !target; depth++) { const next: Node[] = []; for (const typeNode of frontierNodes) { - for (const edge of this.queries.getOutgoingEdges(typeNode.id, ['implements', 'extends'])) { - const superNode = this.queries.getNodeById(edge.target); + for (const edge of supertypesOf(typeNode.id)) { + const superNode = this.nodeById(edge.target); if (!superNode || seenNodes.has(superNode.id)) continue; seenNodes.add(superNode.id); if (!SUPERTYPE_BEARING_KINDS.has(superNode.kind)) continue; // Member lookup anchored on the supertype's contains edges. - for (const c of this.queries.getOutgoingEdges(superNode.id, ['contains'])) { - const m = this.queries.getNodeById(c.target); - if ( - m && - m.name === member && - (m.kind === 'function' || m.kind === 'method') && - sameLanguageFamily(m.language, ref.language) - ) { - target = m; - break; - } - } + target = + callableMembersOf(superNode.id) + .get(member) + ?.find((m) => sameLanguageFamily(m.language, ref.language)) ?? null; if (target) break; next.push(superNode); } @@ -2694,19 +2814,19 @@ export class ReferenceResolver { // A `#define` is a value, never a callee (#1838): a macro defined only in // an unrelated file is not what `NAME(x)` here expands to either. if (ref.referenceKind === 'calls') { - const target = this.queries.getNodeById(result.targetNodeId); + const target = this.nodeById(result.targetNodeId); if (target?.kind === 'constant' && CPP_DEFINE_SIGNATURE.test(target.signature ?? '')) return null; } // An `imports` reference names something importable — never a member that // only exists inside a type. if (ref.referenceKind === 'imports') { - const target = this.queries.getNodeById(result.targetNodeId); + const target = this.nodeById(result.targetNodeId); return target && !isImportableKind(target.kind) ? null : result; } if (!isInheritanceRef(ref)) return result; - const target = this.queries.getNodeById(result.targetNodeId); + const target = this.nodeById(result.targetNodeId); if (target && !isSupertypeTarget(target)) { const type = this.sameNamedTypeOfValue(target); if (!type) return null; diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index ac2470b17e..d93d54fd27 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -211,6 +211,15 @@ function hasBridgeEvidence(candidate: Node, ref: UnresolvedRef, context: Resolut return false; } +/** + * Per-context memo: node id → its language, for gateLanguageMatch. Matches + * land on ~5 refs per target on vscode, and each check otherwise fetched the + * whole node (a point read + row decode past the query layer's small cache) + * only to read one field. Nodes are fixed within a resolution pass; the memo + * drops with clearNameMatcherMemos. + */ +const TARGET_LANGUAGE = new WeakMap>(); + /** Reject the chosen result without shrinking a pool or trying a replacement. */ export function gateLanguageMatch( result: ResolvedRef | null, @@ -218,6 +227,29 @@ export function gateLanguageMatch( context: ResolutionContext ): ResolvedRef | null { if (!result) return result; + // No code family on the reference's side: no target can cross a boundary. + if (CODE_FAMILY[ref.language] === undefined) return result; + if (context.getNodeById) { + let languages = TARGET_LANGUAGE.get(context); + if (!languages) { + languages = new Map(); + TARGET_LANGUAGE.set(context, languages); + } + let language = languages.get(result.targetNodeId); + if (language === undefined) { + const node = context.getNodeById(result.targetNodeId); + if (node) { + language = node.language as string; + if (languages.size >= 400_000) languages.clear(); + languages.set(result.targetNodeId, language); + } + } + if (language !== undefined) { + if (!crossesCodeBoundary(ref.language, language)) return result; + const target = context.getNodeById(result.targetNodeId); + return target && !hasBridgeEvidence(target, ref, context) ? null : result; + } + } const target = context.getNodeById?.(result.targetNodeId) ?? context.getNodesByName(ref.referenceName).find((n) => n.id === result.targetNodeId); if (target && crossesCodeBoundary(ref.language, target.language) && @@ -677,14 +709,23 @@ function isSealedModule(filePath: string, context: ResolutionContext): boolean { } const hit = memo.get(filePath); if (hit !== undefined) return hit; - const source = context.readFile(filePath); - const code = source === null ? '' : blankStringContents(stripCommentsForRegex(source, 'typescript')); // CommonJS assignments can execute inside template interpolations, which the // masker blanks. Keep the conservative raw-source exemption for those forms. + // Cheapest disqualifiers first: nearly every module exports a node (asked + // without reading or decoding the file), and masking the source is only + // needed to rule out the ones that don't. (The masker only blanks text, so + // no `import` in the source means none in code.) + const exportsNode = context.fileHasExportedNode + ? context.fileHasExportedNode(filePath) + : context.getNodesInFile(filePath).some((n) => n.isExported); + const source = exportsNode ? null : context.readFile(filePath); const sealed = - source !== null && HAS_IMPORT_STATEMENT.test(code) && - !context.getNodesInFile(filePath).some((n) => n.isExported) && - !HAS_ESM_EXPORT.test(code) && !HAS_CJS_EXPORT.test(source); + !exportsNode && source !== null && source.includes('import') && + !HAS_CJS_EXPORT.test(source) && + (() => { + const code = blankStringContents(stripCommentsForRegex(source, 'typescript')); + return HAS_IMPORT_STATEMENT.test(code) && !HAS_ESM_EXPORT.test(code); + })(); memo.set(filePath, sealed); return sealed; } @@ -898,16 +939,106 @@ function isReceiverLessCall(ref: UnresolvedRef, context: ResolutionContext): boo const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; if (line === undefined) return false; - const at = line.slice(ref.column); - const nameEsc = ref.referenceName.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - if (!new RegExp('^' + nameEsc + '\\s*[(<]').test(at)) return false; - // Nothing but whitespace, an operator or an opener may precede a bare call. - return !/[.\w$\]\)]\s*$/.test(line.slice(0, ref.column)) || /\b(?:return|await|yield|typeof|void|new|else|case|throw|in|of|instanceof|go|defer)\s*$/.test(line.slice(0, ref.column)); + // `^\s*[(<]` at the call's column, without compiling a pattern per call. + if (!line.startsWith(ref.referenceName, ref.column)) return false; + CALL_OPENER.lastIndex = ref.column + ref.referenceName.length; + if (!CALL_OPENER.test(line)) return false; + // Nothing but whitespace, an operator or an opener may precede a bare call: + // read the text before the column backwards, past trailing whitespace, + // instead of end-anchoring a pattern that scans the prefix from its start. + let end = Math.min(ref.column, line.length); + while (end > 0 && WHITESPACE.test(line[end - 1]!)) end--; + if (end === 0 || !RECEIVER_TAIL_CHAR.test(line[end - 1]!)) return true; + let start = end; + while (start > 0 && WORD_CHAR.test(line[start - 1]!)) start--; + return BARE_CALL_KEYWORDS.has(line.slice(start, end)); } +/** `\s*[(<]` from a given index (sticky). */ +const CALL_OPENER = /\s*[(<]/y; +const WHITESPACE = /\s/; +const WORD_CHAR = /\w/; +/** A character that ends a receiver: `.`, a word character, `$`, `]` or `)`. */ +const RECEIVER_TAIL_CHAR = /[.\w$\])]/; +/** Keywords after which a name starts an expression, so the call has no receiver. */ +const BARE_CALL_KEYWORDS: ReadonlySet = new Set([ + 'return', 'await', 'yield', 'typeof', 'void', 'new', 'else', 'case', 'throw', 'in', 'of', 'instanceof', 'go', 'defer', +]); + /** Per-context memo: `file\0name` → "the file binds this name locally". */ const LOCAL_BINDING_MEMO = new WeakMap>(); +/** + * Where a file's local bindings can start, for isLocallyBoundJsName: every + * `const`/`let`/`var` and `function`/`class` keyword and every `=>`. Each + * binding pattern can only match from one of these (or, for a parameter, + * from the `(` before an occurrence of the name), so a lookup tries its + * patterns at those offsets instead of searching the file once per pattern + * per name. Kept for the last few files — calls arrive file by file. + */ +interface LocalBindingSites { + varDecls: number[]; + fnDecls: number[]; + arrows: number[]; +} +const LOCAL_BINDING_SITES = new WeakMap>(); +const LOCAL_BINDING_SITES_KEEP = 16; +const VAR_DECL_SITE = /\b(?:const|let|var)\s/g; +const FN_DECL_SITE = /\b(?:function|class)\s/g; + +function localBindingSites(filePath: string, source: string, context: ResolutionContext): LocalBindingSites { + let cache = LOCAL_BINDING_SITES.get(context); + if (!cache) { + cache = new Map(); + LOCAL_BINDING_SITES.set(context, cache); + } + let sites = cache.get(filePath); + if (!sites) { + const offsets = (re: RegExp): number[] => Array.from(source.matchAll(re), (m) => m.index!); + const arrows: number[] = []; + for (let a = source.indexOf('=>'); a !== -1; a = source.indexOf('=>', a + 2)) arrows.push(a); + sites = { varDecls: offsets(VAR_DECL_SITE), fnDecls: offsets(FN_DECL_SITE), arrows }; + if (cache.size >= LOCAL_BINDING_SITES_KEEP) cache.delete(cache.keys().next().value!); + cache.set(filePath, sites); + } + return sites; +} + +type LocalBindingPatterns = { decl: RegExp; fn: RegExp; param: RegExp }; +/** Sticky binding patterns by name — the same names recur file after file. */ +const LOCAL_BINDING_PATTERNS = new Map(); +const LOCAL_BINDING_PATTERNS_CAP = 4096; +const JS_BINDING_NAME = /^[\w$]+$/; +const ARROW_HEAD_CHAR = /[\w$.]/; +const IMPORT_BINDING_VALUE = /^\s*(?:await\s+)?(?:require|import)\s*\(/; + +function localBindingPatterns(name: string, flags: string): LocalBindingPatterns { + const n = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + return { + // `const { name } = require('./m')` / `= await import('./m')` binds an IMPORT, + // not a shadow: the symbol lives in the other file and the call means it. + decl: new RegExp( + '\\b(?:const|let|var)\\s+(?:' + n + '\\b|[{\\[][^;=]*?\\b' + n + '\\b[^;=]*?[}\\]])\\s*(?:=\\s*([^;\\n]*))?', + flags + ), + fn: new RegExp('\\b(?:function|class)\\s+' + n + '\\b', flags), + // a parameter: every token before the name in the list is itself a + // parameter (identifier, optional type, optional default) — so a string + // argument containing the word cannot match. + // Each earlier parameter has exactly one parse: its first non-space + // character after the identifier picks the type (`?`/`:`), default (`=`) + // or bare alternative. Written as `(type)?(default)?\s*`, the same + // strings split several ways per parameter, and a failing search + // backtracked through every combination — 30-40s per name on a vscode + // test file whose helper takes nine `name: T = value` parameters. + param: new RegExp( + '\\(\\s*(?:(?:\\.\\.\\.)?[\\w$]+(?:\\s*(?:\\?\\s*)?:[^,()]+|\\s*=[^,()]+|\\s*),\\s*)*' + + n + '\\b(?:\\s*\\??\\s*:[^,()]*)?(?:\\s*=[^,()]*)?(?:\\s*,\\s*[^()]*)?\\)\\s*(?::[^=;{]*)?(?:=>|\\{)', + flags + ), + }; +} + /** * Whether a JS/TS file binds `name` itself — as a `const`/`let`/`var`/ * `function`/`class` declaration (destructuring included) or as a parameter @@ -931,33 +1062,73 @@ function isLocallyBoundJsName(name: string, filePath: string, context: Resolutio const hit = memo.get(key); if (hit !== undefined) return hit; const source = context.readFile(filePath) ?? ''; - const n = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - // `const { name } = require('./m')` / `= await import('./m')` binds an IMPORT, - // not a shadow: the symbol lives in the other file and the call means it. - const declRe = new RegExp( - '\\b(?:const|let|var)\\s+(?:' + n + '\\b|[{\\[][^;=]*?\\b' + n + '\\b[^;=]*?[}\\]])\\s*(?:=\\s*([^;\\n]*))?', - 'g' - ); - let bound = false; - for (const m of source.matchAll(declRe)) { - if (!/^\s*(?:await\s+)?(?:require|import)\s*\(/.test(m[1] ?? '')) { bound = true; break; } - } - if (!bound) { - bound = - new RegExp('\\b(?:function|class)\\s+' + n + '\\b').test(source) || - // a parameter: every token before the name in the list is itself a - // parameter (identifier, optional type, optional default) — so a string - // argument containing the word cannot match. - new RegExp( - '\\(\\s*(?:(?:\\.\\.\\.)?[\\w$]+(?:\\s*\\??\\s*:\\s*[^,()]+)?(?:\\s*=\\s*[^,()]+)?\\s*,\\s*)*' + - n + '\\b(?:\\s*\\??\\s*:[^,()]*)?(?:\\s*=[^,()]*)?(?:\\s*,\\s*[^()]*)?\\)\\s*(?::[^=;{]*)?(?:=>|\\{)' - ).test(source) || - new RegExp('(?:^|[^\\w$.])' + n + '\\s*=>').test(source); - } + const bound = JS_BINDING_NAME.test(name) + ? bindsAtSites(source, name, localBindingSites(filePath, source, context)) + : bindsAnywhere(source, name); memo.set(key, bound); return bound; } +/** isLocallyBoundJsName's patterns, searched through the whole source. */ +function bindsAnywhere(source: string, name: string): boolean { + const { decl, fn, param } = localBindingPatterns(name, 'g'); + for (const m of source.matchAll(decl)) { + if (!IMPORT_BINDING_VALUE.test(m[1] ?? '')) return true; + } + const n = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + return fn.test(source) || param.test(source) || new RegExp('(?:^|[^\\w$.])' + n + '\\s*=>').test(source); +} + +/** + * bindsAnywhere for a plain identifier, tried only where a match can start: + * a declaration at its keyword (in order, resuming past each match exactly as + * the global search does), a parameter at the `(` its list opens with — the + * last `(` before an occurrence of the name, with no `)` between, since no + * earlier parameter can hold a parenthesis — and `name =>` at each arrow. + */ +function bindsAtSites(source: string, name: string, sites: LocalBindingSites): boolean { + let patterns = LOCAL_BINDING_PATTERNS.get(name); + if (!patterns) { + patterns = localBindingPatterns(name, 'y'); + if (LOCAL_BINDING_PATTERNS.size >= LOCAL_BINDING_PATTERNS_CAP) { + LOCAL_BINDING_PATTERNS.delete(LOCAL_BINDING_PATTERNS.keys().next().value!); + } + LOCAL_BINDING_PATTERNS.set(name, patterns); + } + const { decl, fn, param } = patterns; + let from = 0; + for (const at of sites.varDecls) { + if (at < from) continue; + decl.lastIndex = at; + const m = decl.exec(source); + if (!m) continue; + from = at + m[0].length; + if (!IMPORT_BINDING_VALUE.test(m[1] ?? '')) return true; + } + for (const at of sites.fnDecls) { + fn.lastIndex = at; + if (fn.test(source)) return true; + } + let tried = -1; + for (let at = source.indexOf(name); at !== -1; at = source.indexOf(name, at + 1)) { + const open = at > 0 ? source.lastIndexOf('(', at - 1) : -1; + if (open < 0 || open === tried || source.lastIndexOf(')', at - 1) > open) continue; + tried = open; + param.lastIndex = open; + if (param.test(source)) return true; + } + // `name =>`: the name ends where the whitespace before the arrow starts. + for (const arrow of sites.arrows) { + let end = arrow; + while (end > 0 && WHITESPACE.test(source[end - 1]!)) end--; + const start = end - name.length; + if (start >= 0 && source.startsWith(name, start) && (start === 0 || !ARROW_HEAD_CHAR.test(source[start - 1]!))) { + return true; + } + } + return false; +} + /** * Try to resolve a reference by exact name match */ @@ -980,27 +1151,24 @@ export function matchByExactName( const storeAction = matchJsStoreBindingCall(ref, context); if (storeAction) return storeAction; } - const candidates = context.getNodesByName(ref.referenceName) + // Every rule below judges one candidate on its own, so they run as ONE pass, + // the kind/language checks before the ones that read source: a common name + // has thousands of same-named nodes, and a chain of filters copied that + // list once per rule for every reference. + const cMacroCall = ref.referenceKind === 'calls' && (ref.language === 'c' || ref.language === 'cpp'); + const valueRef = ref.referenceKind === 'references' || ref.referenceKind === 'function_ref'; + const importRef = ref.referenceKind === 'imports'; + const inheritanceRef = isInheritanceRef(ref); + const candidates = context.getNodesByName(ref.referenceName).filter((n) => // Macro constants are not callees and must not consume the same-name // ceiling (#1839). Keep upstream's language gate on the chosen result. - .filter((n) => !(ref.referenceKind === 'calls' && - (ref.language === 'c' || ref.language === 'cpp') && - n.kind === 'constant' && CPP_DEFINE_SIGNATURE.test(n.signature ?? ''))) + !(cMacroCall && n.kind === 'constant' && CPP_DEFINE_SIGNATURE.test(n.signature ?? '')) && // Type/value references retain same-family eligibility: a native namesake // must not hide the actual web type. Calls still gate only the winner. - .filter((n) => (ref.referenceKind !== 'references' && ref.referenceKind !== 'function_ref') || - sameLanguageFamily(n.language, ref.language)) - .filter((n) => n.kind !== 'import') - // Nested locals are only reachable from inside their container (#1230). - .filter((n) => isLexicallyReachable(n, ref, context)) - // Preserve import ranking; calls reject the winner without promoting another. - .filter((n) => ref.referenceKind !== 'imports' || n.filePath === ref.filePath || - !ESM_FAMILY.has(n.language) || !isSealedModule(n.filePath, context)) + (!valueRef || sameLanguageFamily(n.language, ref.language)) && + n.kind !== 'import' && // A receiver-less JS/TS or Go call cannot reach a method (#1714, #1857). - .filter((n) => !((bareJs || bareGo) && n.kind === 'method')) - // A name the file binds itself (a parameter, a const) shadows every other - // file's symbol of that name, so a bare call has no cross-file candidate. - .filter((n) => !(bareJs && n.filePath !== ref.filePath && isLocallyBoundJsName(ref.referenceName, ref.filePath, context))) + !((bareJs || bareGo) && n.kind === 'method') && // An `extends`/`implements` ref names a supertype, so anything that can't // BE one is not a candidate at all. This is eligibility, not // ranking: kind is only a scoring bonus below (and none is awarded for @@ -1008,11 +1176,20 @@ export function matchByExactName( // the real `trait`, and as the sole candidate was adopted outright by the // single-match shortcut. Restricting the pool BEFORE ranking lets the // legitimate supertype win instead of merely dropping the false edge. - .filter((n) => !isInheritanceRef(ref) || isSupertypeTarget(n)) + (!inheritanceRef || isSupertypeTarget(n)) && // Likewise for `imports`: a member that only exists inside a type is not // importable, so it is not a candidate. Without this a `path`/`id`/`url` // import resolved to some interface's same-named property. - .filter((n) => ref.referenceKind !== 'imports' || isImportableKind(n.kind)); + (!importRef || isImportableKind(n.kind)) && + // Nested locals are only reachable from inside their container (#1230). + isLexicallyReachable(n, ref, context) && + // Preserve import ranking; calls reject the winner without promoting another. + (!importRef || n.filePath === ref.filePath || + !ESM_FAMILY.has(n.language) || !isSealedModule(n.filePath, context)) && + // A name the file binds itself (a parameter, a const) shadows every other + // file's symbol of that name, so a bare call has no cross-file candidate. + !(bareJs && n.filePath !== ref.filePath && isLocallyBoundJsName(ref.referenceName, ref.filePath, context)) + ); if (candidates.length === 0) { return null; @@ -1159,6 +1336,12 @@ export function matchByQualifiedName( return null; } +/** A node a `Receiver.method()` call can name as the method's owning type. */ +function isMethodOwnerKind(n: Node): boolean { + return n.kind === 'class' || n.kind === 'struct' || n.kind === 'union' || n.kind === 'interface' || + (n.language === 'scala' && n.kind === 'module'); +} + /** * When a symbol name is ambiguous across files, prefer the candidate(s) declared * in the call site's own file, keeping the rest in their original order (#1079). @@ -2069,8 +2252,12 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { RUST_TRAIT_IMPL_MEMO.delete(context); SEALED_MODULES.delete(context); LOCAL_BINDING_MEMO.delete(context); + LOCAL_BINDING_SITES.delete(context); SELECTOR_NAMES.delete(context); GET_STATE_FILES.delete(context); + TS_FIELD_DECL_MEMO.delete(context); + TS_CLASS_LINES.delete(context); + TARGET_LANGUAGE.delete(context); } function memoPatterns(key: string, build: () => RegExp[]): RegExp[] { @@ -2892,7 +3079,8 @@ export function matchMethodCall( // cross-file use reaches the same helper through the import path. if (dotMatch && !objectOrClass!.includes('.') && OBJECT_LITERAL_LANGUAGES.has(ref.language)) { const literalMatch = nmTimedT('mc-literal', ref, (): ResolvedRef | null => { - const holders = preferCallSiteFile(context.getNodesByName(objectOrClass!), ref.filePath).filter( + // Same-file holders only, so the call-site-first ordering is moot. + const holders = context.getNodesByName(objectOrClass!).filter( (n) => (n.kind === 'constant' || n.kind === 'variable') && n.filePath === ref.filePath ); for (const holder of holders) { @@ -2913,31 +3101,29 @@ export function matchMethodCall( // resolves to `a/`'s method (#1079). const strat1 = nmTimedT('mc-class', ref, (): ResolvedRef | null => { const classCandidates = preferCallSiteFile( - context.getNodesByName(objectOrClass!), + context.getNodesByName(objectOrClass!).filter(isMethodOwnerKind), ref.filePath, ); for (const classNode of classCandidates) { - if (classNode.kind === 'class' || classNode.kind === 'struct' || classNode.kind === 'union' || classNode.kind === 'interface' || (classNode.language === 'scala' && classNode.kind === 'module')) { - // Skip cross-language class matches - if (classNode.language !== ref.language) continue; + // Skip cross-language class matches + if (classNode.language !== ref.language) continue; - const nodesInFile = context.getNodesInFile(classNode.filePath); - const methodNode = nodesInFile.find( - (n) => - n.kind === 'method' && - n.name === methodName && - n.qualifiedName.includes(classNode.name) - ); + const nodesInFile = context.getNodesInFile(classNode.filePath); + const methodNode = nodesInFile.find( + (n) => + n.kind === 'method' && + n.name === methodName && + n.qualifiedName.includes(classNode.name) + ); - if (methodNode) { - return { - original: ref, - targetNodeId: methodNode.id, - confidence: 0.85, - resolvedBy: 'qualified-name', - }; - } + if (methodNode) { + return { + original: ref, + targetNodeId: methodNode.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + }; } } return null; @@ -2958,30 +3144,28 @@ export function matchMethodCall( if (capitalizedReceiver !== objectOrClass) { const strat2 = nmTimedT('mc-capital', ref, (): ResolvedRef | null => { const fuzzyClassCandidates = preferCallSiteFile( - context.getNodesByName(capitalizedReceiver), + context.getNodesByName(capitalizedReceiver).filter(isMethodOwnerKind), ref.filePath, ); for (const classNode of fuzzyClassCandidates) { - if (classNode.kind === 'class' || classNode.kind === 'struct' || classNode.kind === 'union' || classNode.kind === 'interface' || (classNode.language === 'scala' && classNode.kind === 'module')) { - // Skip cross-language class matches - if (classNode.language !== ref.language) continue; - - const nodesInFile = context.getNodesInFile(classNode.filePath); - const methodNode = nodesInFile.find( - (n) => - n.kind === 'method' && - n.name === methodName && - n.qualifiedName.includes(classNode.name) - ); + // Skip cross-language class matches + if (classNode.language !== ref.language) continue; - if (methodNode) { - return { - original: ref, - targetNodeId: methodNode.id, - confidence: 0.8, - resolvedBy: 'instance-method', - }; - } + const nodesInFile = context.getNodesInFile(classNode.filePath); + const methodNode = nodesInFile.find( + (n) => + n.kind === 'method' && + n.name === methodName && + n.qualifiedName.includes(classNode.name) + ); + + if (methodNode) { + return { + original: ref, + targetNodeId: methodNode.id, + confidence: 0.8, + resolvedBy: 'instance-method', + }; } } return null; @@ -3302,37 +3486,115 @@ function matchRustSelfFieldCall( } /** - * Resolve a TS/JS `this..()` call (#1496) through the field's - * declared type, read off the ENCLOSING class's own declaration lines: - * a field or constructor-parameter property (`private mailer: Mailer`, - * `mailer?: Mailer`, `readonly mailer: Mailer`) or an initializer - * (`mailer = new Mailer()`, `this.mailer = new Mailer()`). The method is then - * VALIDATED on that type by resolveMethodOnType. Null — never a bare-name - * fallback — when the field is not declared there or its type is external, - * a builtin (`this.items.push()`) or not spelled out. + * Per-context memo for matchTsThisFieldCall's declaration scan: `classId\0field` + * → the first line of the class that declares the field (as `: typeof X`, + * `: X`, or `= new X`, tried in that order per line) with its captured type, + * or null. The scan depends only on the class and the field, but ran for every + * `this..()` call — re-splitting the file and compiling three + * patterns each time, a third of all method-call matching on vscode. Derived + * from file source: drops with clearNameMatcherMemos. */ -function matchTsThisFieldCall( - field: string, - methodName: string, - ref: UnresolvedRef, - context: ResolutionContext, -): ResolvedRef | null { - if (!field || field.includes('.')) return null; - const caller = context.getNodeById?.(ref.fromNodeId); - if (!caller) return null; - const sep = caller.qualifiedName.lastIndexOf('::'); - if (sep <= 0) return null; // not inside a class - const owner = caller.qualifiedName.slice(0, sep).split('::').pop(); - if (!owner) return null; +const TS_FIELD_DECL_MEMO = new WeakMap>(); +/** A class's comment-stripped lines, and which of them hold each `[\w$#]` token. */ +interface TsClassDecl { + lines: string[]; + /** Field lookups so far; the token index is built on the second. */ + lookups: number; + linesByToken: Map | null; +} +/** The last few classes' declarations — calls arrive file by file. */ +const TS_CLASS_LINES = new WeakMap>(); +const TS_CLASS_LINES_KEEP = 32; + +function tsClassDecl(cls: Node, context: ResolutionContext): TsClassDecl | null { + let cache = TS_CLASS_LINES.get(context); + if (!cache) { + cache = new Map(); + TS_CLASS_LINES.set(context, cache); + } + const hit = cache.get(cls.id); + if (hit !== undefined) return hit; + const source = context.readFile(cls.filePath); + const decl = source + ? { + lines: source + .split('\n') + .slice(Math.max(0, cls.startLine - 1), cls.endLine) + .map((rawLine) => rawLine.replace(/\/\/.*$/, '').replace(/\/\*.*?\*\//g, '')), + lookups: 0, + linesByToken: null, + } + : null; + if (cache.size >= TS_CLASS_LINES_KEEP) cache.delete(cache.keys().next().value!); + cache.set(cls.id, decl); + return decl; +} - const owners = preferCallSiteFile(context.getNodesByName(owner), ref.filePath).filter( - (n) => (n.kind === 'class' || n.kind === 'component') && sameLanguageFamily(n.language, ref.language) - ); +const TS_FIELD_TOKEN = /^[\w$#]+$/; +const TS_LINE_TOKEN = /[\w$#]+/g; +const TS_TOKEN_CHAR = /[\w$#]/; + +/** + * tsFieldPatterns with the field spelled `[\w$#]+`, tried sticky at a whole- + * token occurrence of the field. There the run consumes exactly the field (a + * shorter prefix cannot continue: every pattern needs `\s`, `?`, `!`, `:` or + * `=` next), so each matches exactly where the field's own pattern would — + * without compiling three patterns for every field of every class. + */ +const TS_FIELD_PATTERNS_AT: readonly TsFieldPattern[] = [ + { re: /(? 0 && TS_TOKEN_CHAR.test(line[i - 1]!)) || (end < line.length && TS_TOKEN_CHAR.test(line[end]!))) continue; + (at ??= []).push(i); + } + if (!at) return null; + for (const { re, valueType } of TS_FIELD_PATTERNS_AT) { + for (const i of at) { + re.lastIndex = i; + const m = re.exec(line); + if (m && m[1]) return { valueType, type: m[1] }; + } + } + return null; +} + +/** Indices of the class lines holding `token` as a whole `[\w$#]` run, ascending. */ +function tsClassLinesWithToken(decl: TsClassDecl, token: string): readonly number[] { + if (!decl.linesByToken) { + const index = new Map(); + for (let i = 0; i < decl.lines.length; i++) { + for (const m of decl.lines[i]!.matchAll(TS_LINE_TOKEN)) { + const rows = index.get(m[0]); + if (!rows) index.set(m[0], [i]); + else if (rows[rows.length - 1] !== i) rows.push(i); + } + } + decl.linesByToken = index; + } + return decl.linesByToken.get(token) ?? []; +} + +type TsFieldPattern = { re: RegExp; valueType: boolean }; +/** Compiled declaration patterns by field name — fields recur across classes. */ +const TS_FIELD_PATTERNS = new Map(); +const TS_FIELD_PATTERNS_CAP = 4096; + +function tsFieldPatterns(field: string): TsFieldPattern[] { + const hit = TS_FIELD_PATTERNS.get(field); + if (hit) return hit; const fieldEsc = field.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // A word boundary cannot open a private name; it also lets a public // `items` match `#items`. Keep the two field namespaces distinct (#1987). const fieldStart = '(? = [ + const patterns: TsFieldPattern[] = [ // `storage: typeof DraftHubStorage` — the type OF a value: an object // literal used as a namespace. Its members are bare-named functions inside // the constant's extent (#1573), so they are found by containment, not by @@ -3352,60 +3614,128 @@ function matchTsThisFieldCall( // `mailer = new Mailer()` / `this.mailer = new Mailer()` { re: new RegExp(`${fieldStart}${fieldEsc}\\b\\s*=\\s*new\\s+([A-Za-z_$][\\w.$]*)`), valueType: false }, ]; - for (const cls of owners) { - const source = context.readFile(cls.filePath); - if (!source) continue; - const declLines = source.split('\n').slice(Math.max(0, cls.startLine - 1), cls.endLine); - for (const rawLine of declLines) { - const line = rawLine.replace(/\/\/.*$/, '').replace(/\/\*.*?\*\//g, ''); + if (TS_FIELD_PATTERNS.size >= TS_FIELD_PATTERNS_CAP) TS_FIELD_PATTERNS.delete(TS_FIELD_PATTERNS.keys().next().value!); + TS_FIELD_PATTERNS.set(field, patterns); + return patterns; +} + +function tsFieldDeclaration( + cls: Node, + field: string, + context: ResolutionContext +): { valueType: boolean; type: string } | null { + let memo = TS_FIELD_DECL_MEMO.get(context); + if (!memo) { + memo = new Map(); + TS_FIELD_DECL_MEMO.set(context, memo); + } + const key = cls.id + '\0' + field; + const hit = memo.get(key); + if (hit !== undefined) return hit; + let found: { valueType: boolean; type: string } | null = null; + const decl = tsClassDecl(cls, context); + if (decl && TS_FIELD_TOKEN.test(field)) { + // Every pattern needs the field as a whole `[\w$#]` run — the lookbehind + // bars one before it, and nothing after it but `\s`, `?`, `!`, `:` or `=` + // can continue a match — so only lines holding that token can match. A + // class asked about several fields is indexed by token once. + const rows = ++decl.lookups > 1 ? tsClassLinesWithToken(decl, field) : null; + const count = rows ? rows.length : decl.lines.length; + for (let k = 0; k < count && !found; k++) { + const line = decl.lines[rows ? rows[k]! : k]!; + // Every pattern spells the field literally, so only a line that contains + // it can match — and most of a class's lines never mention a given field. + if (line.includes(field)) found = tsFieldOnLine(line, field); + } + } else if (decl) { + const patterns = tsFieldPatterns(field); + scan: for (const line of decl.lines) { + if (!line.includes(field)) continue; for (const { re, valueType } of patterns) { const m = line.match(re); if (!m || !m[1]) continue; - if (valueType) { - // The value's declaration may live in another file (it is imported); - // the call site's file is preferred when several share the name. - const holderName = m[1].split('.').pop()!; - const holders = preferCallSiteFile(context.getNodesByName(holderName), ref.filePath).filter( - (n) => (n.kind === 'constant' || n.kind === 'variable') && sameLanguageFamily(n.language, ref.language) - ); - for (const holder of holders) { - const hit = resolveObjectLiteralMember(holder, methodName, ref, context, 0.85, 'instance-method'); - if (hit) return hit; - } - return null; - } - // `ns.Mailer` → `Mailer`; a primitive or builtin names no project type. - const typeName = m[1].split('.').pop()!; - if (!/^[A-Z]/.test(typeName)) return null; - // Two apps in one repo may each declare a `UserService`. The bare-name - // path this replaces broke that tie by directory proximity, so keep the - // same signal: among the type's declarations of the method, prefer the - // one closest to the call site's directory (its own app), never index - // order. resolveMethodOnType still answers the single-declaration and - // supertype cases. - const declared = context - .getNodesByName(methodName) - .filter( - (n) => - n.kind === 'method' && - sameLanguageFamily(n.language, ref.language) && - (n.qualifiedName === `${typeName}::${methodName}` || n.qualifiedName.endsWith(`::${typeName}::${methodName}`)) - ); - if (declared.length > 1) { - const callDirs = ref.filePath.split('/').slice(0, -1); - const shared = (fp: string) => { - const dirs = fp.split('/').slice(0, -1); - let i = 0; - while (i < dirs.length && i < callDirs.length && dirs[i] === callDirs[i]) i++; - return i; - }; - const nearest = [...declared].sort((a, b) => shared(b.filePath) - shared(a.filePath) || a.filePath.localeCompare(b.filePath))[0]!; - return { original: ref, targetNodeId: nearest.id, confidence: 0.85, resolvedBy: 'instance-method' }; - } - return resolveMethodOnType(typeName, methodName, ref, context, 0.85, 'instance-method'); + found = { valueType, type: m[1] }; + break scan; } } } + memo.set(key, found); + return found; +} + +/** + * Resolve a TS/JS `this..()` call (#1496) through the field's + * declared type, read off the ENCLOSING class's own declaration lines: + * a field or constructor-parameter property (`private mailer: Mailer`, + * `mailer?: Mailer`, `readonly mailer: Mailer`) or an initializer + * (`mailer = new Mailer()`, `this.mailer = new Mailer()`). The method is then + * VALIDATED on that type by resolveMethodOnType. Null — never a bare-name + * fallback — when the field is not declared there or its type is external, + * a builtin (`this.items.push()`) or not spelled out. + */ +function matchTsThisFieldCall( + field: string, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, +): ResolvedRef | null { + if (!field || field.includes('.')) return null; + const caller = context.getNodeById?.(ref.fromNodeId); + if (!caller) return null; + const sep = caller.qualifiedName.lastIndexOf('::'); + if (sep <= 0) return null; // not inside a class + const owner = caller.qualifiedName.slice(0, sep).split('::').pop(); + if (!owner) return null; + + const owners = preferCallSiteFile(context.getNodesByName(owner), ref.filePath).filter( + (n) => (n.kind === 'class' || n.kind === 'component') && sameLanguageFamily(n.language, ref.language) + ); + for (const cls of owners) { + const decl = tsFieldDeclaration(cls, field, context); + if (!decl) continue; + if (decl.valueType) { + // The value's declaration may live in another file (it is imported); + // the call site's file is preferred when several share the name. + const holderName = decl.type.split('.').pop()!; + const holders = preferCallSiteFile(context.getNodesByName(holderName), ref.filePath).filter( + (n) => (n.kind === 'constant' || n.kind === 'variable') && sameLanguageFamily(n.language, ref.language) + ); + for (const holder of holders) { + const hit = resolveObjectLiteralMember(holder, methodName, ref, context, 0.85, 'instance-method'); + if (hit) return hit; + } + return null; + } + // `ns.Mailer` → `Mailer`; a primitive or builtin names no project type. + const typeName = decl.type.split('.').pop()!; + if (!/^[A-Z]/.test(typeName)) return null; + // Two apps in one repo may each declare a `UserService`. The bare-name + // path this replaces broke that tie by directory proximity, so keep the + // same signal: among the type's declarations of the method, prefer the + // one closest to the call site's directory (its own app), never index + // order. resolveMethodOnType still answers the single-declaration and + // supertype cases. + const declared = context + .getNodesByName(methodName) + .filter( + (n) => + n.kind === 'method' && + sameLanguageFamily(n.language, ref.language) && + (n.qualifiedName === `${typeName}::${methodName}` || n.qualifiedName.endsWith(`::${typeName}::${methodName}`)) + ); + if (declared.length > 1) { + const callDirs = ref.filePath.split('/').slice(0, -1); + const shared = (fp: string) => { + const dirs = fp.split('/').slice(0, -1); + let i = 0; + while (i < dirs.length && i < callDirs.length && dirs[i] === callDirs[i]) i++; + return i; + }; + const nearest = [...declared].sort((a, b) => shared(b.filePath) - shared(a.filePath) || a.filePath.localeCompare(b.filePath))[0]!; + return { original: ref, targetNodeId: nearest.id, confidence: 0.85, resolvedBy: 'instance-method' }; + } + return resolveMethodOnType(typeName, methodName, ref, context, 0.85, 'instance-method'); + } return null; } diff --git a/src/resolution/resolver-pool.ts b/src/resolution/resolver-pool.ts index eac808ba48..56b9785a66 100644 --- a/src/resolution/resolver-pool.ts +++ b/src/resolution/resolver-pool.ts @@ -222,6 +222,11 @@ export class ResolverPool { this.recycleWaiters.clear(); } + /** Number of workers — how many passes the synthesis fan-out keeps in flight. */ + get size(): number { + return this.workers.length; + } + /** Whether this batch is worth fanning out. */ static worthParallel(batchLength: number): boolean { return batchLength >= MIN_PARALLEL_BATCH; diff --git a/src/resolution/tier-synthesizer.ts b/src/resolution/tier-synthesizer.ts index b146fdce09..be119f0a7f 100644 --- a/src/resolution/tier-synthesizer.ts +++ b/src/resolution/tier-synthesizer.ts @@ -248,12 +248,13 @@ function decorators(safe: string, name: string): Array<{ args: string; index: nu interface FileFacts { file: string; safe: string; - nodes: Node[]; + /** Read on first use: most files that pass a gate hold no site, and never need them. */ + readonly nodes: Node[]; lineOf: (idx: number) => number; /** The 0-based column of an index on its line — where the site reader looks for the call. */ columnOf: (idx: number) => number; /** Lines a framework resolver made a route node on — registrations, never client calls. */ - routeLines: Set; + readonly routeLines: Set; /** Local names bound to an HTTP client instance, with their literal base URL when written. */ clients: Map; /** The module's default export is a client instance. */ @@ -284,9 +285,8 @@ function readFacts(ctx: ResolutionContext, file: string): FileFacts | null { const content = ctx.readFile(file); if (!content) return null; const safe = stripCommentsForRegex(content, 'typescript'); - const nodes = ctx.getNodesInFile(file); - const routeLines = new Set(); - for (const n of nodes) if (n.kind === 'route') routeLines.add(n.startLine); + let nodes: Node[] | null = null; + let routeLines: Set | null = null; const clients = new Map(); CLIENT_FACTORY.lastIndex = 0; let m: RegExpExecArray | null; @@ -303,10 +303,18 @@ function readFacts(ctx: ResolutionContext, file: string): FileFacts | null { return { file, safe, - nodes, + get nodes() { + return (nodes ??= ctx.getNodesInFile(file)); + }, lineOf: makeLineAt(safe, 1), columnOf: (idx: number) => idx - (safe.lastIndexOf('\n', idx - 1) + 1), - routeLines, + get routeLines() { + if (!routeLines) { + routeLines = new Set(); + for (const n of this.nodes) if (n.kind === 'route') routeLines.add(n.startLine); + } + return routeLines; + }, clients, defaultClient, queues, @@ -347,9 +355,20 @@ const CLIENT_NAMES = const SERVER_NAMES = /^(?:app|router|route|routes|express|fastify|koa|hono|elysia|apiRouter|v1|v2|r)$/; /** A type argument between the callee and its `(` — `useSWR('/api/team')`, `ky.get('/x')`. */ const GENERIC = String.raw`(?:<[^()<>]*(?:<[^()<>]*>[^()<>]*)*>)?`; +/** + * Where a receiver chain (`this.api.client`) may begin: not after an earlier + * identifier-start character of the same word (the second lookbehind is the + * whole rule; the first is its one-character fast path). A match from the + * middle of a word implies one from its first letter — the chain regexes read + * the rest of the word either way — and each of their matches ends on a + * non-word character, so a scan never resumes mid-word: this only skips starts + * bound to fail. Without it every letter of every identifier re-read the + * dotted chain behind it, which on vscode made this the slowest pass. + */ +const CHAIN_START = String.raw`(?): void { - const { safe, nodes, lineOf } = facts; + const { safe, lineOf } = facts; const add = (index: number, open: number, verb: string | null, baseURL: string | null): void => { const line = lineOf(index); const callee = safe.slice(index, open).replace(/\s+/g, '').replace(/<.*>$/, ''); if (facts.routeLines.has(line)) return; // a registration the resolver already read - const fn = enclosingFn(nodes, line); + const fn = enclosingFn(facts.nodes, line); if (!fn) return; const args = argumentsAt(safe, open); if (!args || !args[0]) return; @@ -556,10 +575,10 @@ function collectHttpSites(ctx: ResolutionContext, facts: FileFacts, sites: HttpS // 2. Queue job → consumer // ============================================================================= -const QUEUE_ADD = /((?:this\s*\.\s*)?[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*\.\s*add\s*\(\s*(['"`])([^'"`]+)\2/g; +const QUEUE_ADD = new RegExp(CHAIN_START + /((?:this\s*\.\s*)?[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*\.\s*add\s*\(\s*(['"`])([^'"`]+)\2/.source, 'g'); const QUEUE_SHAPED = /queue|jobs?$|worker|bull|flow|producer/i; const NEW_WORKER = /\bnew\s+Worker\s*(?:<[^>]*>)?\s*\(\s*(['"`])([^'"`]+)\1\s*,\s*/g; -const QUEUE_PROCESS = /((?:this\s*\.\s*)?[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*\.\s*process\s*\(\s*(?:(['"`])([^'"`]+)\2\s*,\s*)?(?:\d+\s*,\s*)?/g; +const QUEUE_PROCESS = new RegExp(CHAIN_START + /((?:this\s*\.\s*)?[A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)\s*\.\s*process\s*\(\s*(?:(['"`])([^'"`]+)\2\s*,\s*)?(?:\d+\s*,\s*)?/.source, 'g'); /** A handler argument: a named function (group 1), or an inline function. */ const HANDLER_ARG = /^(?:(?:async\s+)?([A-Za-z_$][\w$.]*)\s*(?:[,)]|$)|(?:async\s*)?(?:\(|function\b|[A-Za-z_$][\w$]*\s*=>))/; @@ -644,7 +663,7 @@ function decoratedMethod(facts: FileFacts, end: number, cls: Node | null): Node } function collectQueue(ctx: ResolutionContext, facts: FileFacts, producers: QueueProducer[], consumers: QueueConsumer[], cache: Map): void { - const { safe, nodes, lineOf } = facts; + const { safe, lineOf } = facts; let m: RegExpExecArray | null; QUEUE_ADD.lastIndex = 0; while ((m = QUEUE_ADD.exec(safe)) !== null) { @@ -653,7 +672,7 @@ function collectQueue(ctx: ResolutionContext, facts: FileFacts, producers: Queue const last = receiver.replace(/\s+/g, '').split('.').pop()!; if (queue === null && !QUEUE_SHAPED.test(last)) continue; const line = lineOf(m.index); - const fn = enclosingFn(nodes, line); + const fn = enclosingFn(facts.nodes, line); if (!fn) continue; producers.push({ fn, file: facts.file, line, column: facts.columnOf(m.index), callee: `${receiver.replace(/\s+/g, '')}.add`, queue, job: m[3]! }); } @@ -674,7 +693,7 @@ function collectQueue(ctx: ResolutionContext, facts: FileFacts, producers: Queue consumers.push({ node: method, file: facts.file, line, queue, job: firstLiteral(job.args) }); } if (!any) { - const process = nodes.find((n) => n.kind === 'method' && n.name === 'process' && n.startLine >= cls.startLine && n.endLine <= cls.endLine); + const process = facts.nodes.find((n) => n.kind === 'method' && n.name === 'process' && n.startLine >= cls.startLine && n.endLine <= cls.endLine); if (process) consumers.push({ node: process, file: facts.file, line: process.startLine, queue, job: null }); } } @@ -745,8 +764,10 @@ function pairQueue(producers: readonly QueueProducer[], consumers: readonly Queu // 3. Events: a bus, and sockets both ways // ============================================================================= -const EMIT = /((?:[\w$]+(?:\([^()]*\))?\s*\.\s*)*[\w$]+)\s*\.\s*(emit|emitAsync)\s*\(\s*(['"`])([^'"`\n]+)\3/g; -const SOCKET_ON = /((?:[\w$]+(?:\([^()]*\))?\s*\.\s*)*[\w$]+)\s*\.\s*(?:on|once)\s*\(\s*(['"`])([^'"`\n]+)\2\s*,\s*/g; +// A word character opens these chains, so a word's first character is the +// only start that can match (same argument as CHAIN_START). +const EMIT = /(?): void { - const { safe, nodes, lineOf } = facts; + const { safe, lineOf } = facts; const side: 'server' | 'client' = facts.socketServer ? 'server' : 'client'; let m: RegExpExecArray | null; EMIT.lastIndex = 0; @@ -790,7 +811,7 @@ function collectEvents(ctx: ResolutionContext, facts: FileFacts, dispatches: Dis const shape = shapeOf(m[1]!); if (!shape || GENERIC_EVENT.test(m[4]!)) continue; const line = lineOf(m.index); - const fn = enclosingFn(nodes, line); + const fn = enclosingFn(facts.nodes, line); if (!fn) continue; dispatches.push({ fn, file: facts.file, line, column: facts.columnOf(m.index), callee: `${m[1]!.replace(/\s+/g, '')}.${m[2]!}`, event: m[4]!, shape, side }); } diff --git a/src/resolution/types.ts b/src/resolution/types.ts index 75d7ab5ebb..f3ff14a075 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -90,6 +90,12 @@ export interface ResolutionResult { export interface ResolutionContext { /** Get all nodes in a file */ getNodesInFile(filePath: string): Node[]; + /** Whether any node in the file is exported (`getNodesInFile(f).some(n => n.isExported)`), as one indexed probe. */ + fileHasExportedNode?(filePath: string): boolean; + /** `getNodesInFile(f).filter(n => n.isExported)`, without decoding the rest of the file. */ + getExportedNodesInFile?(filePath: string): Node[]; + /** `getNodesInFile(f).filter(n => n.name === name)`, without decoding the rest of the file. */ + getNodesInFileNamed?(filePath: string, name: string): Node[]; /** Get all nodes by name */ getNodesByName(name: string): Node[]; /** Get all nodes by qualified name */ @@ -109,6 +115,8 @@ export interface ResolutionContext { fileExists(filePath: string): boolean; /** Read file content */ readFile(filePath: string): string | null; + /** `readFile(filePath)?.includes(needle) ?? false` for an ASCII `needle`, without decoding a file that lacks it. */ + fileContains?(filePath: string, needle: string): boolean; /** * `readFile(filePath)` split into lines, LRU-cached per file. Receiver-type * inference scans source lines for EVERY `receiver.method()` ref; splitting From 15a5d1d0cd8103dcf919cd8d20ce8bb9f8c2d1c5 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 00:46:41 +0000 Subject: [PATCH 011/363] fix(c): keep an #undef under an undecidable #if unknown; a macro call never binds to a type (#2069, #2070) (#2081) Two follow-ups to the unreleased macro-visibility work (#1838/#1839): - #2069: `#undef FLAG` under an `#if` nothing in the source decides made a never-seen FLAG certainly undefined, so `#ifdef FLAG` was read as false and the functions it guards lost their callers (CPython's `Py_GIL_DISABLED`). A name no directive has touched is now unknown; only an #undef on a certain line clears it. - #2070: dropping the macro constant from a call's candidates let the next same-named node win, so expat's `PREFIX(scanRef)(...)` instantiated an unrelated `struct PREFIX`. When the name is a function-like macro in the project, only a function or method is an eligible callee. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 +- .../cpp-macro-and-constructor-calls.test.ts | 114 ++++++++++++++++++ src/resolution/cpp-macro-visibility.ts | 5 +- src/resolution/name-matcher.ts | 15 ++- 4 files changed, 129 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7559656f7..6b8642f246 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -387,7 +387,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A Scala `object` is now indexed as a module, so `extends`/`with` resolves to the trait or class rather than its companion object. Thanks @htarnacki. (#1824) - C/C++ functions defined through a single-argument macro are now indexed under their real name when the macro's own `#define` proves it. Thanks @Dshuishui. (#1373) - MSVC COM `interface` declarations in C++ headers are now indexed as types with their methods. Thanks @timxx. (#1519) -- A function-like C/C++ macro invocation no longer binds to a same-named function in another file, and a local C++ object initialization now records a call to its constructor. Thanks @netbrah for the reports and @danusha2345. (#1838, #1839) +- A function-like C/C++ macro invocation no longer binds to a same-named function or type in another file, a flag the build can still set on the command line keeps both sides of its `#ifdef` reachable, and a local C++ object initialization now records a call to its constructor. Thanks @netbrah and @bompus for the reports and @danusha2345. (#1838, #1839, #2069, #2070) - A Delphi DFM component's range now runs to its closing `end`, so its properties and event bindings belong to it. Thanks @inth3shadows. (#1350) - Two same-named symbols on one line (a one-line getter/setter pair) no longer collide and drop one; existing symbol IDs are unchanged. Thanks @inth3shadows. (#1349) - A callback registered with `subscribe(this.handler)` now links to that exact handler, not the first same-named method in the project. Thanks @inth3shadows. (#1355) diff --git a/__tests__/cpp-macro-and-constructor-calls.test.ts b/__tests__/cpp-macro-and-constructor-calls.test.ts index 803b7c24d2..242bfe3047 100644 --- a/__tests__/cpp-macro-and-constructor-calls.test.ts +++ b/__tests__/cpp-macro-and-constructor-calls.test.ts @@ -503,3 +503,117 @@ describe('#1839 — local object initialization calls the constructor, not the t ]); }); }); + +/** Every outgoing edge of a function, as `edgeKind kind qualifiedName (file)`. */ +function edges(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(fn(cg, caller).id) + .map((r) => `${r.edge.kind} ${r.node.kind} ${r.node.qualifiedName} (${r.node.filePath})`) + .sort(); +} + +describe('#2069 — an #undef under an undecidable #if leaves a never-seen flag unknown', () => { + it.each(['c', 'cpp'] as const)('reproduction: the #ifdef branch after it stays possible (%s)', async (language) => { + const cg = await indexed({ + 'config.h': '#if FREE_THREADED == 0\n#undef FREE_THREADED\n#endif\n', + [`unit.${language}`]: [ + '#include "config.h"', + '#ifdef FREE_THREADED', + 'static int world_stopped(void) { return 0; }', + '#else', + '#define world_stopped() 1', + '#endif', + 'int check(void) { return world_stopped(); }', + '', + ].join('\n'), + }); + try { + expect(calls(cg, 'check')).toEqual([`function world_stopped (unit.${language})`]); + } finally { + cg.close(); + } + }); + + it.each(['c', 'cpp'] as const)('control: an #undef on a certain line still makes the flag certainly undefined (%s)', async (language) => { + const cg = await indexed({ + 'config.h': '#undef TOP_LEVEL\n#if 1\n#undef TRUE_BRANCH\n#endif\n', + [`unit.${language}`]: [ + '#include "config.h"', + '#ifdef TOP_LEVEL', + '#else', + '#define top_hook() 1', + '#endif', + '#ifdef TRUE_BRANCH', + '#else', + '#define branch_hook() 1', + '#endif', + 'int use_top(void) { return top_hook(); }', + 'int use_branch(void) { return branch_hook(); }', + '', + ].join('\n'), + [`decoy.${language}`]: 'int top_hook(void) { return 0; }\nint branch_hook(void) { return 0; }\n', + }); + try { + expect(calls(cg, 'use_top')).toEqual([]); + expect(calls(cg, 'use_branch')).toEqual([]); + } finally { + cg.close(); + } + }); +}); + +describe('#2070 — a call through a function-like macro never binds to a same-named type', () => { + it.each(['c', 'cpp'] as const)('reproduction: PREFIX(scan)(x) reaches no struct PREFIX, in another file or an included header (%s)', async (language) => { + const cg = await indexed({ + [`unit.${language}`]: [ + '#ifdef TOK_IMPL', + '#define PREFIX(ident) ident', + '#endif', + 'int tok(int x) { return PREFIX(scan)(x); }', + '', + ].join('\n'), + [`parser.${language}`]: 'struct PREFIX { int x; };\n', + 'types.h': 'struct WRAP { int x; };\n', + [`wrapped.${language}`]: [ + '#include "types.h"', + '#ifdef WRAP_IMPL', + '#define WRAP(ident) ident', + '#endif', + 'int wrapped(int x) { return WRAP(scan)(x); }', + '', + ].join('\n'), + }); + try { + expect(edges(cg, 'tok').filter((e) => e.includes('struct PREFIX'))).toEqual([]); + expect(edges(cg, 'wrapped').filter((e) => e.includes('struct WRAP'))).toEqual([]); + } finally { + cg.close(); + } + }); + + it.each(['c', 'cpp'] as const)('control: a same-named function is still the callee when the macro may be off (%s)', async (language) => { + const cg = await indexed({ + [`unit.${language}`]: '#ifdef FAST\n#define helper(x) (x)\n#endif\nint run(int x) { return helper(x); }\n', + [`lib.${language}`]: 'int helper(int x) { return x; }\n', + [`types.${language}`]: 'struct helper { int x; };\n', + }); + try { + expect(edges(cg, 'run')).toEqual([`calls function helper (lib.${language})`]); + } finally { + cg.close(); + } + }); + + it('control: a C++ type whose name is no macro is still constructed', async () => { + const cg = await indexed({ + 'widget.hpp': 'struct Widget { int x; };\n', + 'use.cpp': '#include "widget.hpp"\nvoid make() { auto w = Widget(1); }\n', + 'other.cpp': '#define OTHER(x) (x)\n', + }); + try { + expect(edges(cg, 'make')).toEqual(['instantiates struct Widget (widget.hpp)']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/cpp-macro-visibility.ts b/src/resolution/cpp-macro-visibility.ts index f9946c1ccd..e5c76e2f11 100644 --- a/src/resolution/cpp-macro-visibility.ts +++ b/src/resolution/cpp-macro-visibility.ts @@ -306,8 +306,11 @@ function walkTranslationUnit( const defining = ev.kind === 'define'; const macro = defining && ev.functionLike && !ev.wrapsItself; const now = active === true ? macro : prior?.macro === macro ? macro : undefined; + // A name no directive has touched is unknown, not undefined: the build + // can set it on the command line. So an `#undef` under an undecidable + // `#if` leaves it unknown (#2069); only a certain one clears it. definitions.set(ev.name, { - defined: defining ? or(prior?.defined ?? false, active) : and(prior?.defined ?? false, not(active)), + defined: defining ? or(prior?.defined, active) : and(prior?.defined, not(active)), value: defining && active === true ? condition(ev.value) : undefined, macro: now, }); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d93d54fd27..1f1c29da1f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1155,14 +1155,19 @@ export function matchByExactName( // the kind/language checks before the ones that read source: a common name // has thousands of same-named nodes, and a chain of filters copied that // list once per rule for every reference. - const cMacroCall = ref.referenceKind === 'calls' && (ref.language === 'c' || ref.language === 'cpp'); const valueRef = ref.referenceKind === 'references' || ref.referenceKind === 'function_ref'; const importRef = ref.referenceKind === 'imports'; const inheritanceRef = isInheritanceRef(ref); - const candidates = context.getNodesByName(ref.referenceName).filter((n) => - // Macro constants are not callees and must not consume the same-name - // ceiling (#1839). Keep upstream's language gate on the chosen result. - !(cMacroCall && n.kind === 'constant' && CPP_DEFINE_SIGNATURE.test(n.signature ?? '')) && + const sameName = context.getNodesByName(ref.referenceName); + // `NAME(...)` where NAME is a function-like macro somewhere in the project is + // an expansion or a call to a same-named function — never the macro itself + // (#1839), and never a type that happens to share the name (#2070: expat's + // `PREFIX(scanRef)(…)` bound to an unrelated `struct PREFIX`). Keep + // upstream's language gate on the chosen result. + const cMacroCall = ref.referenceKind === 'calls' && (ref.language === 'c' || ref.language === 'cpp') && + sameName.some((n) => n.kind === 'constant' && CPP_DEFINE_SIGNATURE.test(n.signature ?? '')); + const candidates = sameName.filter((n) => + !(cMacroCall && n.kind !== 'function' && n.kind !== 'method') && // Type/value references retain same-family eligibility: a native namesake // must not hide the actual web type. Calls still gate only the winner. (!valueRef || sameLanguageFamily(n.language, ref.language)) && From e5a3d85b0cd4826d0a48b0db9fc09150c2ea53a5 Mon Sep 17 00:00:00 2001 From: Daniil Date: Tue, 29 Sep 2026 06:01:38 +0300 Subject: [PATCH 012/363] fix(watcher): follow a rebuilt index instead of syncing into the replaced file (#1902) (#1917) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(watcher): follow a rebuilt index instead of syncing into the replaced file (#1902) `codegraph index` rebuilds through `CodeGraph.recreate()`, which unlinks `.codegraph/codegraph.db` and creates a new file at the same path. A running MCP daemon kept its handle on the unlinked inode and its watcher kept "auto-syncing" into it, so no edit reached the on-disk index. The #925 self-heal only ran on the tool-call path and ran no catch-up after reopening. - `sync()` now checks, once it holds the index mutex and the write lock, whether the database was replaced on disk (one stat, the same inode check `reopenIfReplaced` uses). If so it reopens the live file in place and widens a scoped sync to a full reconcile. A failed reopen (rebuild mid-way) returns the lock-busy shape so the watcher keeps its pending files and retries. - `reopenIfReplaced()` refuses while an index/sync holds the mutex, so the tool-call path never closes the handle an in-flight sync writes through. - When a tool call's `freshen()` does reopen, the engine runs its existing catch-up sync (serialized on the same mutex), but only for its own watched instance, so an engine that is not the project's writer never writes. Co-Authored-By: Claude Opus 5.5 * fix(sync): step aside for a rebuild that has recreated the file but not locked it yet `codegraph index` recreates the database, then takes the write lock inside indexAll. With the watcher now following a replaced file, a sync landing in that gap reopened the empty file and started a full reconcile, holding the lock the rebuild was about to request — `codegraph index` could then fail with "another process may be indexing". Review flagged the gap; this closes it. After following a replaced file, a sync that finds no index_state yet (the rebuild has not started indexing) and a file written within the last two minutes returns the lock-busy shape so the watcher retries; the first sync after that reconciles the whole tree once, then syncs go back to scoped. The time bound keeps a rebuild that died before indexing from parking the watcher forever. New test: fails without the guard, passes with it; 136 related tests pass. Co-Authored-By: Claude Opus 5.5 * fix(sync): keep the full catch-up until a full sync completes (#1902) Two gaps in following a replaced database: - Only a reopen done by sync() itself set pendingFullReconcile. The tool-call self-heal (reopenIfReplaced) swapped the handle without it, so the next scoped watcher sync skipped whatever the old handle had absorbed. The flag is now set wherever the database is reopened. - The flag was cleared before the sync ran, so a sync that threw lost the catch-up. It is now cleared only after a full sync completes. Ported from the maintainer's hardening of this fix in #1919. Co-Authored-By: Claude Opus 5.5 * fix(sync): report a rebuild in progress as lock contention, not an empty sync (#1902) Since #2014 the watcher tells lock contention apart from success only by LockUnavailableError; it no longer recognises the all-zero result. Following a rebuilt index still returned that shape when the reopen failed or when the recreated file had not been indexed yet, so the watcher counted the step-aside as a clean sync and dropped the pending edits. Both paths now throw LockUnavailableError. A watcher test proves the edit stays pending through the rebuild gap and is reconciled once the rebuild finishes. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Opus 5.5 Co-authored-by: danusha2345 --- CHANGELOG.md | 1 + __tests__/watcher-replaced-db.test.ts | 317 ++++++++++++++++++++++++++ src/index.ts | 67 ++++++ src/mcp/engine.ts | 15 +- src/mcp/tools.ts | 13 ++ 5 files changed, 411 insertions(+), 2 deletions(-) create mode 100644 __tests__/watcher-replaced-db.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6b8642f246..c30c17b3c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -173,6 +173,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, the shared MCP daemon now waits longer for another program — an antivirus scan, an indexer, or another session reading its lock file — to let go of that file, so it starts instead of leaving the session to fall back to a slower in-process server. (#1773) - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) +- Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. (#1902) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. - Incremental sync now keeps edge rebinding crash-safe: replacing a resolved edge with its recovery reference commits atomically, so an interruption cannot permanently remove the relationship. - Status now detects committed but unindexed changes and restored edits without scanning every source file; thanks @inth3shadows. (#1829) diff --git a/__tests__/watcher-replaced-db.test.ts b/__tests__/watcher-replaced-db.test.ts new file mode 100644 index 0000000000..a81220f3d9 --- /dev/null +++ b/__tests__/watcher-replaced-db.test.ts @@ -0,0 +1,317 @@ +/** + * Live sync follows a rebuilt index (issue #1902). + * + * `codegraph index` rebuilds through `CodeGraph.recreate()`, which unlinks + * `.codegraph/codegraph.db` and creates a new file (a new inode) at the same + * path. A long-lived instance — the MCP daemon — keeps its handle on the old, + * unlinked inode. Before the fix its watcher kept "auto-syncing" into that dead + * inode (nothing it wrote was visible to any other process), and the #925 + * self-heal on the tool-call path reopened the live file without a catch-up, + * so every edit absorbed in between was lost. + * + * POSIX-only: an open file can't be unlinked on Windows, and st_ino is + * unreliable there, so the replaced-inode hazard does not arise. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph, { LockUnavailableError } from '../src/index'; +import type { MCPEngine } from '../src/mcp/engine'; + +const posixOnly = it.runIf(process.platform !== 'win32'); + +/** Rebuild the index in a separate instance, the way `codegraph index` does. */ +async function rebuild(root: string): Promise { + const cg = await CodeGraph.recreate(root); + await cg.indexAll(); + cg.close(); +} + +/** Open the file at the path fresh (as a new CLI process would) and look up a name. */ +async function onDiskHas(root: string, name: string): Promise { + const cg = await CodeGraph.open(root); + try { + return cg.searchNodes(name).some((r) => r.node.name === name); + } finally { + cg.close(); + } +} + +async function waitFor(cond: () => boolean | Promise, timeoutMs = 10000): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + if (await cond()) return; + await new Promise((r) => setTimeout(r, 50)); + } + throw new Error('timed out waiting for condition'); +} + +describe('live sync after the index is rebuilt by another process (#1902)', () => { + let root: string; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-1902-')); + fs.mkdirSync(path.join(root, 'src')); + fs.writeFileSync(path.join(root, 'src', 'a.ts'), 'export function alpha() { return 1; }\n'); + fs.writeFileSync(path.join(root, 'src', 'b.ts'), 'export function bravo() { return 2; }\n'); + }); + + afterEach(() => { + fs.rmSync(root, { recursive: true, force: true }); + }); + + posixOnly('a watcher-picked-up edit lands in the NEW database file, not the unlinked one', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + let synced = 0; + expect(server.watch({ debounceMs: 100, onSyncComplete: () => { synced++; } })).toBe(true); + try { + await rebuild(root); + + fs.appendFileSync(path.join(root, 'src', 'a.ts'), 'export function beta() { return alpha(); }\n'); + await waitFor(() => synced > 0); + + // A fresh open of the path sees the edit — the watcher wrote into the live file. + expect(await onDiskHas(root, 'beta')).toBe(true); + // And the server itself now holds the live file. + expect(server.searchNodes('beta').some((r) => r.node.name === 'beta')).toBe(true); + expect(server.reopenIfReplaced()).toBe(false); + } finally { + server.unwatch(); + server.close(); + } + }); + + posixOnly('a sync in the gap between recreate and indexAll steps aside, then reconciles in full', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + try { + // `codegraph index`: the file is recreated first, the write lock is taken later by indexAll. + const rebuilder = await CodeGraph.recreate(root); + fs.appendFileSync(path.join(root, 'src', 'a.ts'), 'export function gamma() { return 3; }\n'); + // The server's sync lands in the gap: it must not claim the lock for a + // full reconcile of the empty file, and reports contention instead of success. + await expect(server.sync({ paths: ['src/a.ts'] })).rejects.toBeInstanceOf(LockUnavailableError); + // So the rebuild still gets its lock and completes. + const built = await rebuilder.indexAll(); + expect(built.success).toBe(true); + rebuilder.close(); + // An edit the rebuild did not see, made after it finished. + fs.appendFileSync(path.join(root, 'src', 'b.ts'), 'export function delta() { return 4; }\n'); + // The next (scoped, unrelated) sync is widened to the whole tree once. + fs.writeFileSync(path.join(root, 'src', 'c.ts'), 'export function echo() { return 5; }\n'); + const after = await server.sync({ paths: ['src/c.ts'] }); + expect(after.filesChecked).toBeGreaterThan(1); + expect(await onDiskHas(root, 'gamma')).toBe(true); + expect(await onDiskHas(root, 'delta')).toBe(true); + expect(await onDiskHas(root, 'echo')).toBe(true); + // And only once: the following scoped sync stays scoped. + fs.appendFileSync(path.join(root, 'src', 'c.ts'), 'export function foxtrot() { return 6; }\n'); + const scoped = await server.sync({ paths: ['src/c.ts'] }); + expect(scoped.filesChecked).toBe(1); + } finally { + server.close(); + } + }); + + posixOnly('a watcher sync that lands in the rebuild gap keeps its pending files until the rebuild is reconciled', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + // Record every sync the watcher starts, so the test can wait for the one in the gap. + const attempts: Array> = []; + const realSync = server.sync.bind(server); + (server as any).sync = (options?: Parameters[0]) => { + const run = realSync(options); + attempts.push(run); + return run; + }; + let synced = 0; + // The lock-contention backoff (debounce · 2^n, degrading after 5 retries) + // must outlast the rebuild below; 200 ms leaves several seconds. + expect(server.watch({ debounceMs: 200, onSyncComplete: () => { synced++; } })).toBe(true); + try { + // `codegraph index` has recreated the file but not taken the write lock yet. + const rebuilder = await CodeGraph.recreate(root); + fs.appendFileSync(path.join(root, 'src', 'a.ts'), 'export function gamma() { return 3; }\n'); + await waitFor(() => attempts.length > 0); + await expect(attempts[0]).rejects.toBeInstanceOf(LockUnavailableError); + // Stepping aside is not a sync: the edit stays pending, nothing is reported as synced. + expect(synced).toBe(0); + expect(server.getPendingFiles().map((f) => f.path)).toContain('src/a.ts'); + + const built = await rebuilder.indexAll(); + expect(built.success).toBe(true); + rebuilder.close(); + + // The watcher's retry reconciles the rebuilt file and only then clears the edit. + await waitFor(() => synced > 0); + expect(await onDiskHas(root, 'gamma')).toBe(true); + expect(server.getPendingFiles()).toEqual([]); + } finally { + server.unwatch(); + server.close(); + } + }); + + posixOnly('a scoped sync that finds the database replaced reconciles the whole tree', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + try { + await rebuild(root); + + // Two edits after the rebuild, but the sync is told about only one of + // them — the other stands for everything the dead handle absorbed. + fs.appendFileSync(path.join(root, 'src', 'a.ts'), 'export function beta() { return 3; }\n'); + fs.appendFileSync(path.join(root, 'src', 'b.ts'), 'export function charlie() { return 4; }\n'); + await server.sync({ paths: ['src/a.ts'] }); + + expect(await onDiskHas(root, 'beta')).toBe(true); + expect(await onDiskHas(root, 'charlie')).toBe(true); + + // Once the handle is live again, a scoped sync stays scoped. + fs.appendFileSync(path.join(root, 'src', 'b.ts'), 'export function delta() { return 5; }\n'); + await server.sync({ paths: ['src/a.ts'] }); + expect(await onDiskHas(root, 'delta')).toBe(false); + } finally { + server.close(); + } + }); + + posixOnly('reopenIfReplaced does not swap the connection under an in-flight sync', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + try { + await rebuild(root); + const inFlight = server.sync(); + expect(server.isIndexing()).toBe(true); + expect(server.reopenIfReplaced()).toBe(false); + await inFlight; + // The sync itself followed the path. + expect(server.reopenIfReplaced()).toBe(false); + fs.appendFileSync(path.join(root, 'src', 'a.ts'), 'export function echo() { return 6; }\n'); + await server.sync({ paths: ['src/a.ts'] }); + expect(await onDiskHas(root, 'echo')).toBe(true); + } finally { + server.close(); + } + }); + + // The engine lazily `require`s the CodeGraph module, which only resolves in + // the built output, so this block drives the built engine (as the + // spawned-CLI suites do); `npm run build` first. + describe('tool-call reopen through the MCP engine', () => { + const ENGINE = path.resolve(__dirname, '../dist/mcp/engine.js'); + let prevDebounce: string | undefined; + + beforeEach(() => { + // Keep the watcher from syncing the edit on its own, so only the + // tool-call path's catch-up can put it in the rebuilt file. + prevDebounce = process.env.CODEGRAPH_WATCH_DEBOUNCE_MS; + process.env.CODEGRAPH_WATCH_DEBOUNCE_MS = '60000'; + }); + + afterEach(() => { + if (prevDebounce === undefined) delete process.env.CODEGRAPH_WATCH_DEBOUNCE_MS; + else process.env.CODEGRAPH_WATCH_DEBOUNCE_MS = prevDebounce; + }); + + async function openEngine(watch: boolean): Promise { + const seed = CodeGraph.initSync(root); + await seed.indexAll(); + seed.close(); + if (!fs.existsSync(ENGINE)) throw new Error(`${ENGINE} missing — run \`npm run build\` first`); + const { MCPEngine: Engine } = require(ENGINE) as { MCPEngine: typeof MCPEngine }; + const engine = new Engine({ watch, writerLockRoot: watch ? root : undefined }); + await engine.ensureInitialized(root); + // Drain the post-open catch-up gate (and prove the project is loaded). + const first = await engine.getToolHandler().execute('codegraph_search', { query: 'alpha' }); + expect(first.content[0].text).toMatch(/alpha/); + return engine; + } + + posixOnly('reopening a replaced database runs a catch-up sync into the new file', async () => { + const engine = await openEngine(true); + try { + await rebuild(root); + fs.appendFileSync(path.join(root, 'src', 'b.ts'), 'export function foxtrot() { return 7; }\n'); + expect(await onDiskHas(root, 'foxtrot')).toBe(false); + + const handler = engine.getToolHandler(); + // This call reopens the replaced database and starts the catch-up; + // the next call awaits its gate. + await handler.execute('codegraph_search', { query: 'alpha' }); + const res = await handler.execute('codegraph_search', { query: 'foxtrot' }); + expect(res.isError).toBeFalsy(); + expect(res.content[0].text).toMatch(/foxtrot/); + expect(await onDiskHas(root, 'foxtrot')).toBe(true); + } finally { + engine.stop(); + } + }); + + posixOnly('an engine that is not watching reopens but does not write', async () => { + const engine = await openEngine(false); + try { + await rebuild(root); + fs.appendFileSync(path.join(root, 'src', 'b.ts'), 'export function golf() { return 8; }\n'); + + const handler = engine.getToolHandler(); + await handler.execute('codegraph_search', { query: 'alpha' }); + await handler.execute('codegraph_search', { query: 'golf' }); + expect(await onDiskHas(root, 'golf')).toBe(false); + } finally { + engine.stop(); + } + }); + }); + + posixOnly('a reopen by a tool call widens the next scoped sync to the whole tree', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + try { + await rebuild(root); + // The self-heal on the tool-call path, not a sync, follows the path. + expect(server.reopenIfReplaced()).toBe(true); + const live = (server as any).orchestrator; + const calls: Array = []; + const sync = live.sync.bind(live); + live.sync = (onProgress: unknown, paths: string[] | undefined, ...rest: unknown[]) => { + calls.push(paths); + return sync(onProgress, paths, ...rest); + }; + await server.sync({ paths: ['src/a.ts'] }); + await server.sync({ paths: ['src/a.ts'] }); + // The first sync after the reopen reconciled everything; the next stays scoped. + expect(calls).toEqual([undefined, ['src/a.ts']]); + } finally { + server.close(); + } + }); + + posixOnly('a catch-up sync that fails keeps the full reconcile for the next one', async () => { + const server = CodeGraph.initSync(root); + await server.indexAll(); + try { + await rebuild(root); + expect(server.reopenIfReplaced()).toBe(true); + const live = (server as any).orchestrator; + const calls: Array = []; + const sync = live.sync.bind(live); + let failNext = true; + live.sync = (onProgress: unknown, paths: string[] | undefined, ...rest: unknown[]) => { + calls.push(paths); + if (failNext) { + failNext = false; + return Promise.reject(new Error('injected sync failure')); + } + return sync(onProgress, paths, ...rest); + }; + await expect(server.sync({ paths: ['src/a.ts'] })).rejects.toThrow('injected sync failure'); + await server.sync({ paths: ['src/a.ts'] }); + expect(calls).toEqual([undefined, undefined]); + } finally { + server.close(); + } + }); +}); diff --git a/src/index.ts b/src/index.ts index f61e161ffd..99303b6fd6 100644 --- a/src/index.ts +++ b/src/index.ts @@ -235,6 +235,25 @@ export class CodeGraph { ); } + /** + * Set when this instance followed a database replaced on disk (#1902): the + * next sync that can run reconciles the whole tree, because whatever the old + * handle absorbed since the rebuild never reached the new file. + */ + private pendingFullReconcile = false; + + /** How long a recreated, not-yet-indexed database is treated as a rebuild in progress. */ + private static readonly RECREATE_GRACE_MS = 120_000; + + /** The database file at the path was written within the recreate grace window. */ + private isFreshlyRecreated(): boolean { + try { + return Date.now() - fs.statSync(getDatabasePath(this.projectRoot)).mtimeMs < CodeGraph.RECREATE_GRACE_MS; + } catch { + return false; + } + } + /** * Heal a stale database handle in place. If `.codegraph/` was removed and * recreated at the SAME path while this instance held the DB open — a git @@ -248,8 +267,19 @@ export class CodeGraph { * * POSIX-only in practice: `isReplacedOnDisk` never fires on Windows (an open * file can't be unlinked there, and st_ino is unreliable). + * + * Refuses (returns false) while an index/sync holds the index mutex: closing + * the handle that run is writing through would break it mid-flight. `sync()` + * performs the same check itself once it holds the mutex (#1902), so the + * replaced file is still picked up — by that sync, or by the caller's retry. */ reopenIfReplaced(): boolean { + if (this.indexMutex.isLocked()) return false; + return this.reopenReplacedDatabase(); + } + + /** The body of {@link reopenIfReplaced}, without the in-flight-sync guard. */ + private reopenReplacedDatabase(): boolean { if (!this.db.isReplacedOnDisk()) return false; const dbPath = this.db.getPath(); // Open the live file FIRST — if that throws (e.g. mid-recreate), the old @@ -260,6 +290,9 @@ export class CodeGraph { this.db = fresh; this.queries = new QueryBuilder(fresh.getDb()); this.wireLayers(); + // Whoever reopened — a sync, or a tool call's self-heal — the next sync + // must reconcile the whole tree (#1902). + this.pendingFullReconcile = true; // Releasing the dead handle also frees the leaked db/-wal/-shm fds that were // pinning the unlinked inode (#925). try { stale.close(); } catch { /* the old inode is gone; closing just frees fds */ } @@ -798,6 +831,39 @@ export class CodeGraph { `Sync could not acquire the file lock; retry when the index is available. ${err instanceof Error ? err.message : String(err)}` ); } + // A full rebuild in another process (`codegraph index` → recreate) + // unlinks the database and creates a new file at the same path. A + // long-lived instance — the MCP daemon's watcher — would otherwise keep + // "syncing" into the dead inode, and nothing it wrote there is visible + // to anyone (#1902). Follow the path before writing (one stat), and + // widen a scoped sync to a full one: whatever the old handle absorbed + // since the rebuild is gone, so the new file has to be reconciled whole. + // If the reopen fails (the rebuild is mid-way), report lock contention + // so the watcher keeps its pending files and retries. + try { + this.reopenReplacedDatabase(); + } catch (err) { + this.fileLock.release(); + throw new LockUnavailableError( + `Sync could not open the rebuilt index yet; retry when the rebuild finishes. ${err instanceof Error ? err.message : String(err)}` + ); + } + if (this.pendingFullReconcile) { + // `codegraph index` recreates the file, THEN takes the write lock in + // indexAll. A sync landing in that gap would otherwise run a full + // reconcile of the empty file and hold the lock the rebuild is about + // to ask for. A fresh file with no index_state yet is that rebuild: + // step aside (lock contention, the watcher retries) and reconcile in + // full once it is done. Bounded, so a rebuild that died before + // indexing does not park the watcher forever. + if (this.getIndexState() === null && this.isFreshlyRecreated()) { + this.fileLock.release(); + throw new LockUnavailableError('A rebuild of this index is in progress; retry when it finishes.'); + } + // Cleared only once this run completes (below): a sync that throws + // must leave the full catch-up for the next one. + options = { ...options, paths: undefined }; + } // Defer WAL auto-checkpointing for the whole incremental run, exactly // as indexAll does for the bulk path (#1231): sync's store loop and its // resolution passes churn the same FTS + secondary-index hot pages, and @@ -1076,6 +1142,7 @@ export class CodeGraph { this.orchestrator.finishGitIndexState(gitState, fullReconcile, result.failedFilePaths); + if (fullReconcile && result.filesChecked > 0) this.pendingFullReconcile = false; return result; } finally { // Mirror indexAll's teardown: stop the valve, then restore the diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 0276bb4772..2898f203b1 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -109,6 +109,15 @@ export class MCPEngine { activate: (cg) => this.explicitProjects.get(cg)?.ready() ?? Promise.resolve(), release: (cg) => this.releaseExplicitProject(cg), }); + // A tool call found the default project's database replaced on disk (a + // `codegraph index` rebuild) and reopened it (#1902). Reconcile the new + // file with the usual catch-up — `sync()` serializes on the index mutex, + // so it never overlaps an in-flight watcher sync. Only when this engine is + // watching, i.e. it is the project's writer: a read-only engine (writer + // lock held elsewhere, watching disabled) must not start writing. + this.toolHandler.setOnDatabaseReopened((cg) => { + if (cg === this.cg && cg.isWatching()) this.catchUpSync(true); + }); if (opts.writerLockRoot && !this.opts.readOnly) { assertNoRebuild(opts.writerLockRoot); const writer = tryAcquireWriterLock(opts.writerLockRoot, 'fallback'); @@ -407,10 +416,12 @@ export class MCPEngine { * and the per-file staleness banner can't help because `getPendingFiles()` * is populated by the watcher, not by catch-up). */ - private catchUpSync(): void { + private catchUpSync(afterReopen = false): void { const cg = this.cg; if (!cg || this.opts.readOnly) return; - if (this.defaultLease) { + // The lease's gate is the startup reconcile and stays settled once caught + // up; a database reopened after a rebuild (#1902) needs a sync of its own. + if (this.defaultLease && !afterReopen) { this.toolHandler.setCatchUpGate(this.defaultLease.ready()); return; } diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 5c99378c85..05b6039115 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -1929,6 +1929,9 @@ export class ToolHandler { // (see {@link resolveCatchUpGateTimeoutMs}) so a minutes-long reconcile on a // huge repo can't hang a call (#905); cleared when the reconcile settles. private catchUpGate: Promise | null = null; + // Engine hook fired when `freshen` reopened a replaced database (#1902), so + // the engine can reconcile the new file with a catch-up sync. + private onDatabaseReopened: ((cg: CodeGraph) => void) | null = null; // Optional worker-thread pool for off-loop read-tool dispatch. When ready + // healthy, heavy reads leave the main loop free for the MCP transport. private queryPool: QueryPool | null = null; @@ -1986,6 +1989,15 @@ export class ToolHandler { }); } + /** + * Engine-only: called after a tool call's {@link freshen} reopened a database + * that was replaced on disk (#1902). The engine decides whether a catch-up + * sync is its to run (only for the instance it watches and writes). + */ + setOnDatabaseReopened(fn: ((cg: CodeGraph) => void) | null): void { + this.onDatabaseReopened = fn; + } + /** * Await the catch-up gate, but no longer than the configured timeout (#905). * If the reconcile settles first, we got the fully-reconciled answer. If the @@ -2319,6 +2331,7 @@ export class ToolHandler { '[CodeGraph MCP] The index was replaced on disk (e.g. a git worktree ' + 'recreated at the same path); reopened the live database in place.\n' ); + this.onDatabaseReopened?.(cg); } } catch { // Best-effort self-heal — a failed reopen must never break the tool call; From 04ec5f554d579ffaaa5c10f51573adaf004d2235 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 03:01:46 +0000 Subject: [PATCH 013/363] fix(extraction): never read an oversize file; an MPEG-TS video named .ts is not TypeScript (#1910) (#2082) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(extraction): an MPEG-TS video named .ts is not TypeScript (#1910) `.ts` mapped to TypeScript by extension alone, so MPEG transport-stream fixtures (`testdata/*.ts`) were parsed by tree-sitter: ~28 s of CPU for a 900 KB clip, minutes for a directory of them, for zero symbols. Recognise the stream from the head of the file — the 0x47 sync byte at offsets 0, 188, 376 and 564 plus a NUL byte, which every stream carries in its first packets and UTF-8 source never does — and drop it at discovery: not indexed, not parsed, not counted, not tallied as an unsupported language. The check reads under 1 KB and only for `.ts` files; the batch reader sniffs the bytes it already read, and the single-file path (sync, watcher) does the same head read, so a clip handed in by name is skipped too. The skip happens before parse dispatch, so no kernel mirror is needed. Co-Authored-By: Claude Fable 5.1 * fix(extraction): a video .ts is never pending on the git path or a scoped sync Review found two places that decide "is this a source file" without the MPEG-TS check the scan applies: the git fast path's candidate loop in getChangedFiles and the scoped-sync path filter (watcher events). An untracked clip in a git repo was reported as `added`, skipped by sync without a record, and reported as `added` again on every status; a tracked .ts that became a clip stayed `modified` with its stale nodes. Both now treat the clip as non-source: removed when tracked, ignored otherwise. Regression tests fail without this change and pass with it, on both arms. Co-Authored-By: Claude Opus 5.5 * fix(extraction): require a binary head before calling a .ts file video (#1910) Review on #1915: the sniff took four aligned 0x47 sync bytes plus any NUL as proof of a transport stream, so TypeScript with a `G` at four 188-byte strides and one raw NUL in a comment was dropped unindexed. The sniff now needs the sync byte on sixteen consecutive packets (3 KB of head) and at least 1/64 of the head to be control bytes. Compressed payload carries 8-20% of them (checked on ffmpeg-made H.264/AAC, MPEG-2 and MP2 streams); source text carries none. A shorter clip is cheap to parse anyway. Also drops the benchmark figures from the changelog entry. Co-Authored-By: Claude Opus 5.5 * fix(extraction): a file over the size limit is never read (#1910) The size gate ran after `readFile`, so a committed video or blob fixture was decoded in full — 3.4 GB of RSS for a 400 MB file, `Invalid string length` past ~512 MB — only to be stored as skipped. It was then read again by every pass that scans files by name: the framework detectors' `readFile` and the resolver's cached reader, three full passes on the reporter's fixtures. Stat first, everywhere a source file is read by path: the batch reader and `extractFile` store an oversize file with a size stamp in place of its content, change detection hashes the same stamp (so a same-size rewrite of a file nothing is indexed from is not a change, while crossing the limit in either direction is), and both by-name readers return null above the limit. The limit moves to `src/file-limits.ts` so the three sites share one number. strace on a sparse 400 MB `.ts`: 153 603 reads before, 0 after; indexing it next to a real file costs no measurable RSS. Stacked on #1914, which shares the batch-reader hunk. Co-Authored-By: Claude Fable 5.1 * refactor(extraction): one-purpose size-stamp reader, comments in operation order `readSourceOrStamp` took a `stats` argument no caller passed and returned stats no caller read; it now returns the text or stamp change detection hashes. The batch reader's two comments are back in the order its code runs: the size gate, then the MPEG-TS head check. No behaviour change. Co-Authored-By: Claude Opus 5.5 * fix(extraction): bound the read itself, not only the stat before it (#1910) A file can grow between the stat that gates the size limit and the read that follows (a log, a download, a build output being written). Every source read now goes through readBoundedSource/readBoundedSourceSync: the open descriptor is re-checked, and the read stops one byte past the limit, so an oversize file is still stamped rather than decoded. The extraction batch reader, single-file extraction, framework detectors and the resolver's file cache all use it. Ported from the maintainer's hardening of this fix in #1919. Co-Authored-By: Claude Opus 5.5 * fix(extraction): bound the parse-retry reads too (#1910) The retry passes for files whose worker crashed or timed out re-read the source with an unbounded fsp.readFile. The file can have grown since the first, bounded read; an oversize file is now skipped by the retry instead of being decoded. Co-Authored-By: Claude Opus 5.5 * fix(extraction): check an oversize file against its size stamp everywhere (#1910) Review on #1915: - The viewer (readFileShape, hasDriftedOnDisk, the source endpoint) and MCP's drift gate hashed a file's real bytes, but the index stores the size stamp for a file over the limit, so every unchanged 1-8 MB file read as "changed on disk". oversizeStamp moves to file-limits.ts with an indexedHashInput helper, and all four checks hash what the index stored. An oversize file is now compared without being read. - git-index-currency's "keeps a committed path pending when sync cannot read it" injected its failure only into readFileSync, which the bounded reader no longer calls, so it failed. The failure now reaches openSync too. - The changelog entry drops its benchmark figures. Co-Authored-By: Claude Opus 5.5 * fix(mcp): check an oversize answer file against its size stamp (#1910) #2043's answer-freshness check hashes each contributing file whole and compares it with the stored contentHash. For a file over the index's size limit the stored hash is the size stamp, so an unchanged oversize file would read as stale. Compare it by its stamp, without reading it, like the viewer and the MCP drift gate do. Co-Authored-By: Claude Opus 5.5 * perf(extraction): judge a .ts clip from the bytes it is read for, not at discovery (#1910) Discovery sniffed the head of every `.ts` file on every full scan — an open + 3 KB read + close per file, before anything asked for its content. On a 10k-file TypeScript tree that took a warm-cache scan from ~105 ms to ~243 ms; on a slow disk or a network share it is a random read per file per scan, the access pattern #1231 and the SMB reports (#1014, #448) are about. The bytes are now judged where they are read anyway, so the check costs no I/O and runs only for a file that is new or changed: the batch reader (as before), `indexFile`, the sync reconcile (an untracked clip is skipped, a tracked file that became one is removed through the same removal path as a deletion), and both `getChangedFiles` paths. The scan lists `.ts` files by name again; a test pins that discovery opens none of them on the walk or the git path. Scan time is back to main's (~98 ms), and a full index is unchanged within noise. Also collapses the four #1910 changelog bullets (#1914 and #1915 each added both) to two, without the benchmark figures. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Fable 5.1 Co-authored-by: danusha2345 --- CHANGELOG.md | 2 + __tests__/bounded-source.test.ts | 96 +++++++++ __tests__/git-index-currency.test.ts | 14 +- __tests__/mpeg-ts-not-typescript.test.ts | 263 +++++++++++++++++++++++ __tests__/oversize-file-not-read.test.ts | 117 ++++++++++ src/extraction/grammars.ts | 61 ++++++ src/extraction/index.ts | 155 +++++++++---- src/file-limits.ts | 104 +++++++++ src/mcp/answer-freshness.ts | 10 + src/mcp/tools.ts | 5 +- src/ui-server/api/source.ts | 14 +- 11 files changed, 792 insertions(+), 49 deletions(-) create mode 100644 __tests__/bounded-source.test.ts create mode 100644 __tests__/mpeg-ts-not-typescript.test.ts create mode 100644 __tests__/oversize-file-not-read.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index c30c17b3c4..a95367e000 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -172,6 +172,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). #### MCP / indexing - On Windows, the shared MCP daemon now waits longer for another program — an antivirus scan, an indexer, or another session reading its lock file — to let go of that file, so it starts instead of leaving the session to fall back to a slower in-process server. (#1773) +- A file over the size limit is no longer read before it is skipped. Large committed video and blob fixtures used to be loaded into memory in full, once per indexing pass, only to be discarded; indexing, change detection and the viewer now go by the file's size instead. Thanks @danusha2345. (#1910) +- An MPEG transport stream video that happens to be named `.ts` (golden fixtures under `testdata/`, e2e clips) is now recognised from its first bytes and skipped instead of being fed to the TypeScript parser, which spent a long time on each clip for no symbols. Real TypeScript files are still indexed. Thanks @danusha2345. (#1910) - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) - Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. (#1902) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. diff --git a/__tests__/bounded-source.test.ts b/__tests__/bounded-source.test.ts new file mode 100644 index 0000000000..e531b02a2a --- /dev/null +++ b/__tests__/bounded-source.test.ts @@ -0,0 +1,96 @@ +/** + * Source reads are bounded by the read itself, not only by a stat taken + * before it (#1910). A file can grow between the stat and the read; the + * reader must still never hold more than the limit plus one byte. + */ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as fsp from 'fs/promises'; +import * as os from 'os'; +import * as path from 'path'; +import { readBoundedSource, readBoundedSourceSync, MAX_SOURCE_FILE_SIZE_BYTES as LIMIT } from '../src/file-limits'; + +// Pass-through wrappers, so a test can inject a file growing mid-read into the +// exact calls file-limits.ts makes. +vi.mock('fs', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, readSync: vi.fn(actual.readSync) }; +}); +vi.mock('fs/promises', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, open: vi.fn(actual.open) }; +}); + +const dirs: string[] = []; +function sourceFile(bytes: Buffer): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-bounded-')); + dirs.push(dir); + const file = path.join(dir, 'source.ts'); + fs.writeFileSync(file, bytes); + return file; +} + +afterEach(() => { + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); +}); + +describe('bounded source reads (#1910)', () => { + for (const size of [0, 100, LIMIT, LIMIT + 1]) { + it(`returns the bytes up to the limit and null past it (${size} bytes)`, async () => { + const file = sourceFile(Buffer.alloc(size, 0x61)); + for (const result of [readBoundedSourceSync(file), await readBoundedSource(file)]) { + expect(result.stats.size).toBe(size); + if (size > LIMIT) expect(result.bytes).toBeNull(); + else expect(result.bytes?.length).toBe(size); + } + }); + } + + it('stops at the limit when the file grows during a synchronous read', () => { + const file = sourceFile(Buffer.from('hello')); + const readSync = vi.mocked(fs.readSync); + const original = readSync.getMockImplementation()!; + let requested = 0; + let grown = false; + readSync.mockImplementation(((fd: number, buf: Buffer, off: number, len: number, pos: number) => { + requested += len; + if (!grown) { + grown = true; + fs.writeFileSync(file, Buffer.alloc(LIMIT + 10, 0x61)); + } + return original(fd, buf, off, len, pos); + }) as typeof fs.readSync); + let result; + try { result = readBoundedSourceSync(file); } finally { readSync.mockImplementation(original); } + expect(grown).toBe(true); + expect(requested).toBeLessThanOrEqual(LIMIT + 1); + expect(result.bytes).toBeNull(); + }); + + it('rechecks the open descriptor when the file grows between stat and open', async () => { + const file = sourceFile(Buffer.from('hello')); + const open = vi.mocked(fsp.open); + const original = open.getMockImplementation()!; + let grown = false; + open.mockImplementationOnce((async (...args: Parameters) => { + grown = true; + fs.writeFileSync(file, Buffer.alloc(LIMIT + 1)); + return original(...args); + }) as typeof fsp.open); + expect((await readBoundedSource(file)).bytes).toBeNull(); + expect(grown).toBe(true); + }); + + it('returns multibyte source byte-for-byte', async () => { + const text = 'export const greeting = "こんにちは 🌿";'; + const file = sourceFile(Buffer.from(text)); + expect((await readBoundedSource(file)).bytes?.toString('utf8')).toBe(text); + expect(readBoundedSourceSync(file).bytes?.toString('utf8')).toBe(text); + }); + + it('refuses a path that is not a regular file', () => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-bounded-')); + dirs.push(dir); + expect(() => readBoundedSourceSync(dir)).toThrow(/not a regular file/); + }); +}); diff --git a/__tests__/git-index-currency.test.ts b/__tests__/git-index-currency.test.ts index fbe79fb722..a5ec39195f 100644 --- a/__tests__/git-index-currency.test.ts +++ b/__tests__/git-index-currency.test.ts @@ -132,12 +132,18 @@ describe('git index currency across commits and restores (#1829)', () => { it('keeps a committed path pending when sync cannot read it', async () => { write('new.ts', 'newSymbol'); commit(); - const real = fs.readFileSync; + // Sync reads a source file through a bounded reader that opens a + // descriptor (#1910), so the failure is injected at openSync as well as + // readFileSync: it must reach whichever one the read goes through. + const realRead = fs.readFileSync; + const realOpen = fs.openSync; let injected = 0; - vi.spyOn(fs, 'readFileSync').mockImplementation(((file: any, ...args: any[]) => { + const failNewTs = (real: (...args: any[]) => unknown) => (file: any, ...args: any[]) => { if (String(file) === path.join(root, 'new.ts')) { injected++; throw new Error('Injected transient read error'); } - return (real as any)(file, ...args); - }) as typeof fs.readFileSync); + return real(file, ...args); + }; + vi.spyOn(fs, 'readFileSync').mockImplementation(failNewTs(realRead as any) as typeof fs.readFileSync); + vi.spyOn(fs, 'openSync').mockImplementation(failNewTs(realOpen as any) as typeof fs.openSync); await cg.sync(); expect(injected).toBeGreaterThan(0); expect(symbols('newSymbol')).not.toContain('newSymbol'); diff --git a/__tests__/mpeg-ts-not-typescript.test.ts b/__tests__/mpeg-ts-not-typescript.test.ts new file mode 100644 index 0000000000..5d44003d2e --- /dev/null +++ b/__tests__/mpeg-ts-not-typescript.test.ts @@ -0,0 +1,263 @@ +/** + * An MPEG transport stream named `.ts` is not TypeScript (#1910). + * + * Golden video fixtures (`testdata/*.ts`) share TypeScript's extension; fed to + * the tree-sitter TypeScript parser a 900 KB clip costs ~28 s of CPU for zero + * symbols. The fix recognises the stream from the bytes it reads anyway (0x47 + * sync byte at each 188-byte packet boundary, and a binary head) and never + * indexes it — not parsed, not stored, not reported as an unsupported + * language, and never pending on a later sync. + */ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import { execFileSync } from 'child_process'; +import { CodeGraph } from '../src'; +import { scanDirectoryAsync, type ScanSkipStats } from '../src/extraction'; +import { detectLanguage, isMpegTransportStream, MPEG_TS_SNIFF_BYTES } from '../src/extraction/grammars'; + +// A configurable `fs`, so a test can watch the opens discovery makes. +vi.mock('fs', async (importOriginal) => ({ ...await importOriginal() })); + +const PACKET = 188; + +/** A synthetic transport stream: `packets` × 188 bytes, 0x47 then pseudo-random payload. */ +function makeMpegTs(packets: number, seed = 1): Buffer { + const buf = Buffer.alloc(packets * PACKET); + let x = seed >>> 0; + for (let i = 0; i < buf.length; i++) { + x = (x * 1664525 + 1013904223) >>> 0; + buf[i] = i % PACKET === 0 ? 0x47 : x >>> 24; + } + // Every real stream opens with PSI tables whose pointer field is 0x00. + buf[4] = 0; + return buf; +} + +/** + * Real TypeScript engineered to put the letter `G` (0x47) at the start of each + * of its first `packets` 188-byte strides — the sync-byte pattern alone. With + * `nul`, one raw NUL sits inside a comment (the review's counterexample on + * #1915). It must stay TypeScript either way. + */ +function makeGammaSource(packets = 4, nul = false): string { + let text = ''; + for (let i = 0; i < packets; i++) { + const line = `Gamma${i}();` + (nul && i === 1 ? ' // \u0000' : ''); + const pad = PACKET - line.length - 1; + text += line + ' '.repeat(pad) + '\n'; + } + for (let i = 0; i < packets; i++) text += `export function Gamma${i}() { return ${i}; }\n`; + text += 'export function realFn() { return 1; }\n'; + for (let off = 0; off < packets * PACKET; off += PACKET) { + if (text.charCodeAt(off) !== 0x47) throw new Error(`fixture: expected G at ${off}`); + } + return text; +} + +/** A stream that opens the usual way: PAT and PMT packets stuffed with 0xFF, then payload. */ +function makePsiLedMpegTs(packets: number): Buffer { + const buf = makeMpegTs(packets, 7); + for (const start of [0, PACKET]) { + buf.fill(0xff, start + 4, start + PACKET); + buf[start + 1] = 0x40; + buf[start + 2] = start === 0 ? 0x00 : 0x10; + buf[start + 3] = 0x10; + } + return buf; +} + +const tempDirs: string[] = []; +function createProject(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-mpegts-')); + tempDirs.push(dir); + return dir; +} + +afterEach(() => { + for (const d of tempDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }); +}); + +describe('isMpegTransportStream', () => { + it('recognises a transport stream from its head', () => { + const ts = makeMpegTs(40); + expect(isMpegTransportStream(ts.subarray(0, MPEG_TS_SNIFF_BYTES))).toBe(true); + expect(isMpegTransportStream(ts)).toBe(true); + }); + + it('needs sixteen aligned sync bytes — a head too short, or one packet off, is not video', () => { + const ts = makeMpegTs(40); + expect(isMpegTransportStream(ts.subarray(0, 15 * PACKET))).toBe(false); + for (const packet of [2, 15]) { + const broken = Buffer.from(ts); + broken[packet * PACKET] = 0x48; + expect(isMpegTransportStream(broken)).toBe(false); + } + expect(isMpegTransportStream(Buffer.alloc(0))).toBe(false); + }); + + it('recognises a stream that opens with 0xFF-stuffed PAT and PMT packets', () => { + expect(isMpegTransportStream(makePsiLedMpegTs(40).subarray(0, MPEG_TS_SNIFF_BYTES))).toBe(true); + }); + + it('does not take source text with G at every 188th byte for video', () => { + const bytes = Buffer.from(makeGammaSource(), 'utf-8'); + expect(bytes[0]).toBe(0x47); + expect(bytes[3 * PACKET]).toBe(0x47); + expect(isMpegTransportStream(bytes)).toBe(false); + expect(detectLanguage('gamma.ts', makeGammaSource())).toBe('typescript'); + }); + + it('does not take source with G at every stride and a NUL in a comment for video (#1915 review)', () => { + for (const packets of [4, 16, 20]) { + const bytes = Buffer.from(makeGammaSource(packets, true), 'utf-8'); + expect(bytes.includes(0)).toBe(true); + expect(isMpegTransportStream(bytes.subarray(0, MPEG_TS_SNIFF_BYTES))).toBe(false); + } + }); +}); + +describe('MPEG-TS video named .ts is skipped, real TypeScript is indexed (#1910)', () => { + it('never indexes the clip: no file record, no nodes, no unsupported-language report', async () => { + const dir = createProject(); + fs.mkdirSync(path.join(dir, 'testdata')); + fs.writeFileSync(path.join(dir, 'testdata', 'clip.ts'), makeMpegTs(40)); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export function greet(n: string) { return `hi ${n}`; }\n'); + fs.writeFileSync(path.join(dir, 'gamma.ts'), makeGammaSource()); + fs.writeFileSync(path.join(dir, 'gamma-nul.ts'), makeGammaSource(16, true)); + + // Discovery lists every `.ts` by name and reads none of them: sniffing each + // one's head there would be a random read per file per scan. The bytes are + // judged where they are read anyway. + const stats: ScanSkipStats = { unsupportedByExtension: new Map() }; + const scanned = await scanDirectoryAsync(dir, undefined, stats); + expect(scanned.sort()).toEqual(['app.ts', 'gamma-nul.ts', 'gamma.ts', 'testdata/clip.ts']); + expect(stats.unsupportedByExtension.size).toBe(0); + + const cg = await CodeGraph.init(dir, { index: true }); + try { + const files = cg.getFiles().map((f) => f.path).sort(); + expect(files).toEqual(['app.ts', 'gamma-nul.ts', 'gamma.ts']); + expect(cg.searchNodes('greet').some((r) => r.node.name === 'greet')).toBe(true); + expect(cg.searchNodes('Gamma2').some((r) => r.node.name === 'Gamma2')).toBe(true); + // The review's counterexample: G at every stride and a NUL in a comment. + expect(cg.searchNodes('realFn').filter((r) => r.node.filePath === 'gamma-nul.ts')).toHaveLength(1); + + // A named re-sync (the watcher / `sync` path hands files in by name) must + // not let the clip back in either. + await cg.sync({ paths: ['testdata/clip.ts', 'app.ts'] }); + expect(cg.getFiles().map((f) => f.path).sort()).toEqual(['app.ts', 'gamma-nul.ts', 'gamma.ts']); + } finally { + await cg.close(); + } + }); + + it('discovery opens no file, on the directory walk or the git path', async () => { + for (const withGit of [false, true]) { + const dir = createProject(); + fs.writeFileSync(path.join(dir, 'clip.ts'), makeMpegTs(40)); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export const a = 1;\n'); + if (withGit) { + const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe' }); + git('init', '-q'); git('config', 'user.email', 't@t'); git('config', 'user.name', 't'); + git('add', '.'); git('commit', '-qm', 'init'); + } + const opened: string[] = []; + const real = fs.openSync; + const spy = vi.spyOn(fs, 'openSync').mockImplementation(((file: fs.PathLike, ...rest: unknown[]) => { + opened.push(String(file)); + return (real as (...args: unknown[]) => number)(file, ...rest); + }) as typeof fs.openSync); + let scanned: string[]; + try { + scanned = await scanDirectoryAsync(dir); + } finally { + spy.mockRestore(); + } + expect(scanned.sort()).toEqual(['app.ts', 'clip.ts']); + expect(opened.filter((f) => f.endsWith('.ts'))).toEqual([]); + } + }); + + it('reports nothing skipped for a project of only source and video', async () => { + const dir = createProject(); + fs.writeFileSync(path.join(dir, 'clip.ts'), makeMpegTs(40)); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export const a = 1;\n'); + const cg = await CodeGraph.init(dir); + try { + const result = await cg.indexAll(); + expect(result.filesSkippedUnsupported).toBeUndefined(); + expect(result.topUnsupportedExtensions).toBeUndefined(); + expect(result.errors.filter((e) => e.filePath === 'clip.ts')).toEqual([]); + expect(cg.getFiles().map((f) => f.path)).toEqual(['app.ts']); + } finally { + await cg.close(); + } + }); + + it('indexes past a 900 KB clip in well under two seconds', async () => { + const dir = createProject(); + fs.mkdirSync(path.join(dir, 'testdata')); + fs.writeFileSync(path.join(dir, 'testdata', 'golden.ts'), makeMpegTs(Math.ceil((900 * 1024) / PACKET))); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export function greet(n: string) { return `hi ${n}`; }\n'); + const cg = await CodeGraph.init(dir); + try { + const t0 = Date.now(); + const result = await cg.indexAll(); + const elapsed = Date.now() - t0; + expect(result.filesIndexed).toBe(1); + expect(cg.getFiles().map((f) => f.path)).toEqual(['app.ts']); + expect(elapsed).toBeLessThan(2000); + } finally { + await cg.close(); + } + }); +}); + +describe('a video .ts never stays pending (#1910)', () => { + const dirs: string[] = []; + afterEach(() => { for (const d of dirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }); }); + const gitProject = (): string => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-mpegts-git-')); + dirs.push(dir); + const git = (...a: string[]) => execFileSync('git', a, { cwd: dir, stdio: 'pipe' }); + git('init', '-q'); git('config', 'user.email', 't@t'); git('config', 'user.name', 't'); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export const a = 1;\n'); + git('add', '.'); git('commit', '-qm', 'init'); + return dir; + }; + + it('an untracked clip is not reported as added, before or after sync', async () => { + const dir = gitProject(); + const cg = await CodeGraph.init(dir, { index: true }); + try { + fs.writeFileSync(path.join(dir, 'clip.ts'), makeMpegTs(40)); + expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] }); + await cg.sync(); + expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] }); + expect(cg.getFiles().map((f) => f.path)).toEqual(['app.ts']); + } finally { + cg.close(); + } + }); + + it('a tracked TypeScript file that becomes a clip is removed, on the git path and on a scoped sync', async () => { + for (const scoped of [false, true]) { + const dir = gitProject(); + fs.writeFileSync(path.join(dir, 'clip.ts'), 'export const clip = 1;\n'); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(cg.getNodesInFile('clip.ts').some((n) => n.name === 'clip')).toBe(true); + fs.writeFileSync(path.join(dir, 'clip.ts'), makeMpegTs(40)); + if (!scoped) expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: ['clip.ts'] }); + await (scoped ? cg.sync({ paths: ['clip.ts'] }) : cg.sync()); + expect(cg.getFiles().map((f) => f.path)).toEqual(['app.ts']); + expect(cg.getNodesInFile('clip.ts')).toEqual([]); + expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] }); + } finally { + cg.close(); + } + } + }); +}); diff --git a/__tests__/oversize-file-not-read.test.ts b/__tests__/oversize-file-not-read.test.ts new file mode 100644 index 0000000000..f32267f73f --- /dev/null +++ b/__tests__/oversize-file-not-read.test.ts @@ -0,0 +1,117 @@ +import { afterEach, describe, expect, it } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { oversizeStamp, hashContent } from '../src/extraction'; +import { hasDriftedOnDisk, readFileShape } from '../src/ui-server/api/source'; +import { ToolHandler } from '../src/mcp/tools'; +import { validateAnswerFiles } from '../src/mcp/answer-freshness'; + +/** + * A file over the size limit is stored as skipped without ever being read + * (#1910): committed video/blob fixtures used to be decoded in full — and + * hashed — only to be discarded, costing their size in RSS per file. + */ +describe('oversize files are stat-gated, never read (#1910)', () => { + let dir: string; + afterEach(() => { if (dir) fs.rmSync(dir, { recursive: true, force: true }); }); + + const big = (bytes: number, fill = 0x41) => Buffer.alloc(bytes, fill); + + it('indexes the neighbours, records the oversize file as skipped with a size-stamp hash, and does not decode it', async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-')); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export function alpha() { return beta(); }\nexport function beta() { return 1; }\n'); + // 1 MB + 1: over the limit; invalid UTF-8 (0xFF) so any decode would be visible as replacement chars. + fs.writeFileSync(path.join(dir, 'blob.ts'), big(1024 * 1024 + 1, 0xff)); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(cg.getNodesByKind('function').map(n => n.name).sort()).toEqual(['alpha', 'beta']); + const rec = cg.getFiles().find(f => f.path === 'blob.ts'); + expect(rec).toBeDefined(); + expect(rec!.contentHash).toBe(hashContent(oversizeStamp(1024 * 1024 + 1))); + // Nothing from the blob reached the graph. + expect(cg.getNodesInFile('blob.ts').filter(n => n.kind !== 'file')).toEqual([]); + // Change detection agrees with what was stored: nothing pending. + expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] }); + // A same-size rewrite is not a change (nothing about it is indexed)... + fs.writeFileSync(path.join(dir, 'blob.ts'), big(1024 * 1024 + 1, 0xfe)); + expect(cg.getChangedFiles().modified).toEqual([]); + // ...crossing the limit is: the file becomes ordinary source. + fs.writeFileSync(path.join(dir, 'blob.ts'), 'export const gamma = 3;\n'); + expect(cg.getChangedFiles().modified).toEqual(['blob.ts']); + await cg.sync(); + expect(cg.getNodesInFile('blob.ts').some(n => n.name === 'gamma')).toBe(true); + // ...and growing back over it is a change too, stored as the stamp again. + fs.writeFileSync(path.join(dir, 'blob.ts'), big(2 * 1024 * 1024, 0xff)); + expect(cg.getChangedFiles().modified).toEqual(['blob.ts']); + await cg.sync(); + expect(cg.getFiles().find(f => f.path === 'blob.ts')!.contentHash).toBe(hashContent(oversizeStamp(2 * 1024 * 1024))); + expect(cg.getNodesInFile('blob.ts').some(n => n.name === 'gamma')).toBe(false); + } finally { + cg.close(); + } + }); + + it('a 400 MB sparse fixture indexes in well under a second and without growing the heap by its size', async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-big-')); + fs.writeFileSync(path.join(dir, 'ok.ts'), 'export const one = 1;\n'); + // Sparse: occupies no disk, but stat() reports 400 MB — a read would decode all of it. + const fd = fs.openSync(path.join(dir, 'huge.ts'), 'w'); + fs.ftruncateSync(fd, 400 * 1024 * 1024); + fs.closeSync(fd); + // Warm the engine on a sibling project first, so the grammar and worker + // start-up cost is not mistaken for the file being read. + const warm = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-warm-')); + fs.writeFileSync(path.join(warm, 'w.ts'), 'export const w = 1;\n'); + (await CodeGraph.init(warm, { index: true })).close(); + fs.rmSync(warm, { recursive: true, force: true }); + const before = process.memoryUsage().rss; + const t0 = Date.now(); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(Date.now() - t0).toBeLessThan(5000); + // Reading 400 MB would show as at least that much RSS; the stamp shows as none. + expect(process.memoryUsage().rss - before).toBeLessThan(100 * 1024 * 1024); + expect(cg.getFiles().map(f => f.path).sort()).toEqual(['huge.ts', 'ok.ts']); + expect(cg.getChangedFiles()).toEqual({ added: [], modified: [], removed: [] }); + } finally { + cg.close(); + } + }); +}); + +describe('an unchanged file over the size limit is not reported as drifted (#1910, #1915 review)', () => { + let dir: string; + afterEach(() => { if (dir) fs.rmSync(dir, { recursive: true, force: true }); }); + + it('reads as current in the viewer and in MCP until its size changes', async () => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-drift-')); + fs.writeFileSync(path.join(dir, 'app.ts'), 'export function alpha() { return 1; }\n'); + // 1.4 MB of ordinary text: over the index limit, under the viewer's 8 MB read cap. + const line = 'const x = 1;\n'; + fs.writeFileSync(path.join(dir, 'big.js'), line.repeat(Math.ceil((1.4 * 1024 * 1024) / line.length))); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const record = cg.getFiles().find((f) => f.path === 'big.js')!; + expect(record).toBeDefined(); + // Touch it: the stat fast path no longer answers, so the hash has to. + const later = new Date(Date.now() + 5000); + fs.utimesSync(path.join(dir, 'big.js'), later, later); + + expect(readFileShape(dir, 'big.js', record).drift).toBe(false); + expect(hasDriftedOnDisk(dir, 'big.js', record)).toBe(false); + expect((new ToolHandler(cg) as any).isFileStaleOnDisk(cg, 'big.js')).toBe(false); + const answer = [{ path: 'big.js', contentHash: record.contentHash }]; + expect((await validateAnswerFiles(dir, answer)).stale).toEqual([]); + + // A different size is a different stamp, and that is drift. + fs.appendFileSync(path.join(dir, 'big.js'), line); + expect(readFileShape(dir, 'big.js', record).drift).toBe(true); + expect(hasDriftedOnDisk(dir, 'big.js', record)).toBe(true); + expect((await validateAnswerFiles(dir, answer)).stale).toEqual(['big.js']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index c7710f2007..84596d81c5 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -172,6 +172,67 @@ export const EXTENSION_MAP: Record = { '.tofu': 'terraform', }; +/** MPEG transport stream: fixed 188-byte packets, each opening with 0x47. */ +const MPEG_TS_PACKET_SIZE = 188; +const MPEG_TS_SYNC_BYTE = 0x47; +/** + * Consecutive packets whose sync byte must line up before a file counts as + * video — 3 KB of head. A stream shorter than that is cheap to parse anyway; + * the cost #1910 is about comes from clips hundreds of KB long. + */ +const MPEG_TS_MIN_PACKETS = 16; +/** + * Share of the head that must be control bytes (below 0x20, other than the + * whitespace ones) for it to count as binary. Compressed audio and video put + * about one byte in eight there; source text puts none. + */ +const MPEG_TS_MIN_CONTROL_SHARE = 1 / 64; +/** + * How many bytes of a file's head `isMpegTransportStream` needs — enough to + * see `MPEG_TS_MIN_PACKETS` sync bytes plus the packets between them. + */ +export const MPEG_TS_SNIFF_BYTES = MPEG_TS_PACKET_SIZE * MPEG_TS_MIN_PACKETS; + +/** + * Whether these leading bytes are an MPEG transport stream — the OTHER thing a + * `.ts` file can be. Golden video fixtures (`testdata/*.ts`, e2e clips) share + * TypeScript's extension, and tree-sitter takes ~28 s to chew through a 900 KB + * clip for zero symbols (#1910), so the decision has to be made from the head + * of the file, before any parse. + * + * Two conditions, both required: + * 1. the sync byte 0x47 sits at every 188-byte packet boundary of the first + * `MPEG_TS_MIN_PACKETS` packets — every packet of a transport stream + * opens with it, and nothing else pads to 188; + * 2. the head is binary: at least `MPEG_TS_MIN_CONTROL_SHARE` of it is + * control bytes, as any compressed payload is. + * 0x47 is the letter `G`, so (1) alone could match source whose lines happen + * to put a `G` at every 188-byte stride. Checking for a single NUL was not + * enough to close that: one NUL in a comment is still TypeScript. (2) asks for + * dozens of control bytes, which no source file carries. + * + * `head` is the first `MPEG_TS_SNIFF_BYTES` (or fewer) bytes of the file. + */ +export function isMpegTransportStream(head: Uint8Array): boolean { + const lastSync = MPEG_TS_PACKET_SIZE * (MPEG_TS_MIN_PACKETS - 1); + if (head.length <= lastSync) return false; + for (let off = 0; off <= lastSync; off += MPEG_TS_PACKET_SIZE) { + if (head[off] !== MPEG_TS_SYNC_BYTE) return false; + } + let control = 0; + for (let i = 0; i < head.length; i++) { + const b = head[i]!; + // Tab, newline, vertical tab, form feed and carriage return are text. + if (b < 0x20 && (b < 0x09 || b > 0x0d)) control++; + } + return control >= head.length * MPEG_TS_MIN_CONTROL_SHARE; +} + +/** Whether `filePath` carries the one extension MPEG-TS shares with a language. */ +export function hasMpegTsExtension(filePath: string): boolean { + return filePath.length > 3 && filePath.slice(-3).toLowerCase() === '.ts'; +} + /** * Whether a file is one CodeGraph can parse, based purely on its extension. * This is the single source of truth for "should we index this file" — derived diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 399f333acf..a4fec2ac3c 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -26,7 +26,7 @@ import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './ import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer'; import { materializeKernelResult } from './kernel'; import { detectGeneratedFile } from './generated-detection'; -import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars'; +import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES } from './grammars'; import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config'; import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; @@ -35,7 +35,8 @@ import ignore, { Ignore } from 'ignore'; import { detectFrameworks } from '../resolution/frameworks'; import type { ResolutionContext } from '../resolution/types'; import { createYielder, type MaybeYield } from '../resolution/cooperative-yield'; -import { MAX_SOURCE_FILE_SIZE_BYTES } from '../file-limits'; +import { MAX_SOURCE_FILE_SIZE_BYTES, oversizeStamp, readBoundedSource, readBoundedSourceSync } from '../file-limits'; +export { oversizeStamp }; /** * Number of files to read in parallel during indexing. @@ -158,6 +159,28 @@ export function hashContent(content: string): string { return crypto.createHash('sha256').update(content).digest('hex'); } + +/** + * What change detection hashes for a file: its text when it is under the size + * limit, the size stamp when it is over — an oversize file is never decoded. + * `null` when the bytes show a `.ts` file is an MPEG transport stream, not + * TypeScript (#1910): decided from the bytes already read, so it costs no I/O + * and runs only for a file that is new or changed — never once per `.ts` file + * at discovery, which on a slow disk is a random read per file per scan. + */ +function readSourceOrStamp(fullPath: string): string | null { + const { stats, bytes } = readBoundedSourceSync(fullPath); + if (bytes === null) return oversizeStamp(stats.size); + return isMpegTsBytes(fullPath, bytes) ? null : bytes.toString('utf8'); +} + +/** Whether these bytes, read from `filePath`, are a `.ts` MPEG transport stream (#1910). */ +function isMpegTsBytes(filePath: string, bytes: Buffer): boolean { + if (!hasMpegTsExtension(filePath) || !isMpegTransportStream(bytes.subarray(0, MPEG_TS_SNIFF_BYTES))) return false; + logDebug('Skipping MPEG transport stream named .ts — not TypeScript', { filePath }); + return true; +} + /** * Directory names that are dependency, build, cache, or tooling output across the * languages/frameworks CodeGraph supports — curated from the canonical @@ -1520,6 +1543,7 @@ export function scanDirectory( * Async variant of scanDirectory that yields to the event loop periodically, * allowing worker threads to receive and render progress messages. */ + /** * What a scan saw but could not index, tallied by extension. * @@ -1846,7 +1870,9 @@ export class ExtractionOrchestrator { const full = validatePathWithinRoot(rootDir, relativePath); if (!full) return null; try { - return fs.readFileSync(full, 'utf-8'); + // Framework detectors scan source by name; a file over the size + // limit was never indexed and must not be decoded here either (#1910). + return readBoundedSourceSync(full).bytes?.toString('utf8') ?? null; } catch { return null; } @@ -2263,8 +2289,20 @@ export class ExtractionOrchestrator { logWarn('Path traversal blocked in batch reader', { filePath: fp }); return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: new Error('Path traversal blocked') }; } - const content = await fsp.readFile(fullPath, 'utf-8'); - const stats = await fsp.stat(fullPath); + // Stat first: a file over the size limit is stored as skipped + // without ever being read or decoded (#1910), so ten oversize + // fixtures in one I/O batch no longer cost their size in RSS. + const { stats, bytes } = await readBoundedSource(fullPath); + if (bytes === null) { + return { filePath: fp, content: oversizeStamp(stats.size), stats, error: null as Error | null }; + } + // Read bytes, not text: a `.ts` that is really an MPEG transport + // stream (#1910) is recognised from its head here, at no extra I/O, + // and never decoded or parsed. + if (isMpegTsBytes(fp, bytes)) { + return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: null as Error | null, skipped: true }; + } + const content = bytes.toString('utf-8'); return { filePath: fp, content, stats, error: null as Error | null }; } catch (err) { return { filePath: fp, content: null as string | null, stats: null as fs.Stats | null, error: err as Error }; @@ -2274,9 +2312,16 @@ export class ExtractionOrchestrator { // Dispatch each readable file into the bounded parse window; the window // stores results on the main thread as they arrive. - for (const { filePath, content, stats, error } of fileContents) { + for (const { filePath, content, stats, error, skipped } of fileContents) { if (signal?.aborted) { aborted = true; break; } + if (skipped) { + // Not a source file after all — counted as done, stored as nothing. + processed++; + onProgress?.({ phase: 'parsing', current: processed, total }); + continue; + } + if (error || content === null || stats === null) { processed++; filesErrored++; @@ -2404,7 +2449,10 @@ export class ExtractionOrchestrator { try { const fullPath = validatePathWithinRoot(this.rootDir, filePath); if (!fullPath) continue; - content = await fsp.readFile(fullPath, 'utf-8'); + // Bounded like the first read: the file may have grown since (#1910). + const bytes = (await readBoundedSource(fullPath)).bytes; + if (bytes === null) continue; + content = bytes.toString('utf8'); } catch { continue; } @@ -2456,7 +2504,9 @@ export class ExtractionOrchestrator { try { const fullPath = validatePathWithinRoot(this.rootDir, filePath); if (!fullPath) continue; - fullContent = await fsp.readFile(fullPath, 'utf-8'); + const bytes = (await readBoundedSource(fullPath)).bytes; + if (bytes === null) continue; + fullContent = bytes.toString('utf8'); } catch { continue; } @@ -2586,8 +2636,14 @@ export class ExtractionOrchestrator { let content: string; let stats: fs.Stats; try { - stats = await fsp.stat(fullPath); - content = await fsp.readFile(fullPath, 'utf-8'); + // An oversize file is stored as skipped; its bytes are never needed (#1910). + const read = await readBoundedSource(fullPath); + stats = read.stats; + // An MPEG transport stream named `.ts` is not TypeScript (#1910). + if (read.bytes !== null && isMpegTsBytes(relativePath, read.bytes)) { + return { nodes: [], edges: [], unresolvedReferences: [], errors: [], durationMs: 0 }; + } + content = read.bytes === null ? oversizeStamp(stats.size) : read.bytes.toString('utf8'); } catch (error) { return { nodes: [], @@ -3139,30 +3195,33 @@ export class ExtractionOrchestrator { // `reconcileChecks` drives the cooperative yield shared with the adds/mods loop // below (see SYNC_RECONCILE_YIELD_INTERVAL / issue #905). let reconcileChecks = 0; + const removeTracked = (tracked: FileRecord): void => { + // Before the cascade deletes them, resurrect incoming cross-file + // resolution edges as their original refs (#1240 removal case): the + // callers live in files this sync will NOT revisit, so this is their + // only chance to rebind to an alternative definition — or to park as + // failed until the symbol reappears somewhere. (A deleted file whose + // CALLERS are also being deleted is fine: their nodes cascade later + // in this loop and take the resurrected rows with them.) + // Every name this file defined is about to stop existing here, which + // narrows the candidate set for that name repo-wide (CG-33). + for (const pair of this.queries.getNodeNamePairsByFiles([tracked.path])) pairsBefore.add(pair); + const incoming = this.queries.getCrossFileIncomingEdgesWithTarget(tracked.path); + if (incoming.length > 0) { + const resurrected = incoming + .map((e) => resurrectRefFromDroppedEdge(e)) + .filter((r): r is UnresolvedReference => r !== null); + if (resurrected.length > 0) { + this.queries.insertUnresolvedRefsBatch(resurrected); + } + } + onFileChange?.(tracked.path); + this.queries.deleteFile(tracked.path); + filesRemoved++; + }; for (const tracked of trackedFiles) { if (!currentSet.has(tracked.path) || !fs.existsSync(path.join(this.rootDir, tracked.path))) { - // Before the cascade deletes them, resurrect incoming cross-file - // resolution edges as their original refs (#1240 removal case): the - // callers live in files this sync will NOT revisit, so this is their - // only chance to rebind to an alternative definition — or to park as - // failed until the symbol reappears somewhere. (A deleted file whose - // CALLERS are also being deleted is fine: their nodes cascade later - // in this loop and take the resurrected rows with them.) - // Every name this file defined is about to stop existing here, which - // narrows the candidate set for that name repo-wide (CG-33). - for (const pair of this.queries.getNodeNamePairsByFiles([tracked.path])) pairsBefore.add(pair); - const incoming = this.queries.getCrossFileIncomingEdgesWithTarget(tracked.path); - if (incoming.length > 0) { - const resurrected = incoming - .map((e) => resurrectRefFromDroppedEdge(e)) - .filter((r): r is UnresolvedReference => r !== null); - if (resurrected.length > 0) { - this.queries.insertUnresolvedRefsBatch(resurrected); - } - } - onFileChange?.(tracked.path); - this.queries.deleteFile(tracked.path); - filesRemoved++; + removeTracked(tracked); } if (++reconcileChecks % SYNC_RECONCILE_YIELD_INTERVAL === 0) { await new Promise((resolve) => setImmediate(resolve)); @@ -3199,14 +3258,21 @@ export class ExtractionOrchestrator { } // New, or size/mtime changed — read + hash to confirm a real content change. - let content: string; + // (An oversize file hashes as its size stamp, unread — #1910.) + let content: string | null; try { - content = fs.readFileSync(fullPath, 'utf-8'); + content = readSourceOrStamp(fullPath); } catch (error) { logDebug('Skipping unreadable file during sync', { filePath, error: String(error) }); failedFilePaths.push(filePath); continue; } + // Not source after all — an MPEG transport stream named `.ts` (#1910): + // a new one is ignored, a tracked file that became one is removed. + if (content === null) { + if (tracked) removeTracked(tracked); + continue; + } const contentHash = hashContent(content); if (!tracked) { @@ -3363,12 +3429,18 @@ export class ExtractionOrchestrator { if (tracked) removed.push(filePath); continue; } - let content: string; - try { content = fs.readFileSync(fullPath, 'utf-8'); } + let content: string | null; + try { content = readSourceOrStamp(fullPath); } catch (error) { logDebug('Skipping unreadable file while detecting changes', { filePath, error: String(error) }); continue; } + // A `.ts` that is an MPEG transport stream is not source (#1910), so an + // untracked clip is never pending and a tracked file that became one is gone. + if (content === null) { + if (tracked) removed.push(filePath); + continue; + } if (!tracked) added.push(filePath); else if (tracked.contentHash !== hashContent(content)) modified.push(filePath); } @@ -3400,16 +3472,21 @@ export class ExtractionOrchestrator { // Find added and modified files for (const filePath of currentFiles) { const fullPath = path.join(this.rootDir, filePath); - let content: string; + let content: string | null; try { - content = fs.readFileSync(fullPath, 'utf-8'); + content = readSourceOrStamp(fullPath); } catch (error) { logDebug('Skipping unreadable file while detecting changes', { filePath, error: String(error) }); continue; } - const contentHash = hashContent(content); const tracked = trackedMap.get(filePath); + // An MPEG transport stream named `.ts` (#1910) is not source. + if (content === null) { + if (tracked) removed.push(filePath); + continue; + } + const contentHash = hashContent(content); if (!tracked) { added.push(filePath); diff --git a/src/file-limits.ts b/src/file-limits.ts index 4bb4d6da1a..276d7f2021 100644 --- a/src/file-limits.ts +++ b/src/file-limits.ts @@ -1,6 +1,110 @@ +import * as fs from 'fs'; +import * as fsp from 'fs/promises'; + /** * Largest source file CodeGraph will parse or read during resolution. Generated * bundles, minified sources, and dependency archives above this limit provide no * useful symbols; 1 MB covers essentially all hand-written source. */ export const MAX_SOURCE_FILE_SIZE_BYTES = 1024 * 1024; + +/** + * What stands in for the content of a file over MAX_SOURCE_FILE_SIZE_BYTES. Such a file is + * never parsed, so its bytes are never needed — reading them only to hash and + * discard cost multi-GB RSS spikes on committed video/blob fixtures and could + * fail outright with `Invalid string length` (#1910). The stamp is a function + * of size alone: change detection compares it to the stored hash, so a + * same-size rewrite of an oversize file is not a change (nothing about it is + * indexed), while crossing the limit in either direction is. + */ +export function oversizeStamp(size: number): string { + return `codegraph:oversize:${size}`; +} + +/** + * The text whose hash the index stores for a file of `size` bytes: its content + * within the limit, the size stamp over it. Anything that checks a file on disk + * against its stored `contentHash` has to hash this, or every unchanged file + * over the limit reads as drifted. `content` is only read within the limit. + */ +export function indexedHashInput(size: number, content: () => string): string { + return size > MAX_SOURCE_FILE_SIZE_BYTES ? oversizeStamp(size) : content(); +} + +/** A source file's stats, and its bytes when they are within the limit (null = oversize). */ +export interface BoundedSource { + stats: fs.Stats; + bytes: Buffer | null; +} + +const READ_CHUNK_BYTES = 64 * 1024; + +function assertRegularFile(stats: fs.Stats): void { + if (!stats.isFile()) throw new Error('Source path is not a regular file'); +} + +/** + * Read a source file without ever holding more than the limit plus one byte. + * A stat before the read is not enough: the file can grow between the stat + * and the read (a log, a download, a build output being written), so the + * descriptor is re-checked after opening and the read itself stops one byte + * past the limit. Returns `bytes: null` for an oversize file. + */ +export async function readBoundedSource(file: string): Promise { + const initial = await fsp.stat(file); + assertRegularFile(initial); + if (initial.size > MAX_SOURCE_FILE_SIZE_BYTES) return { stats: initial, bytes: null }; + const handle = await fsp.open(file, 'r'); + try { + let stats = await handle.stat(); + assertRegularFile(stats); + if (stats.size > MAX_SOURCE_FILE_SIZE_BYTES) return { stats, bytes: null }; + const chunks: Buffer[] = []; + let size = 0; + while (size <= MAX_SOURCE_FILE_SIZE_BYTES) { + const chunk = Buffer.allocUnsafe(Math.min(READ_CHUNK_BYTES, MAX_SOURCE_FILE_SIZE_BYTES + 1 - size)); + const { bytesRead } = await handle.read(chunk, 0, chunk.length, size); + if (!bytesRead) break; + chunks.push(chunk.subarray(0, bytesRead)); + size += bytesRead; + } + stats = await handle.stat(); + if (size > MAX_SOURCE_FILE_SIZE_BYTES || stats.size > MAX_SOURCE_FILE_SIZE_BYTES) { + stats.size = Math.max(size, stats.size); + return { stats, bytes: null }; + } + return { stats, bytes: Buffer.concat(chunks, size) }; + } finally { + await handle.close(); + } +} + +/** Synchronous {@link readBoundedSource}. */ +export function readBoundedSourceSync(file: string): BoundedSource { + const initial = fs.statSync(file); + assertRegularFile(initial); + if (initial.size > MAX_SOURCE_FILE_SIZE_BYTES) return { stats: initial, bytes: null }; + const fd = fs.openSync(file, 'r'); + try { + let stats = fs.fstatSync(fd); + assertRegularFile(stats); + if (stats.size > MAX_SOURCE_FILE_SIZE_BYTES) return { stats, bytes: null }; + const chunks: Buffer[] = []; + let size = 0; + while (size <= MAX_SOURCE_FILE_SIZE_BYTES) { + const chunk = Buffer.allocUnsafe(Math.min(READ_CHUNK_BYTES, MAX_SOURCE_FILE_SIZE_BYTES + 1 - size)); + const bytesRead = fs.readSync(fd, chunk, 0, chunk.length, size); + if (!bytesRead) break; + chunks.push(chunk.subarray(0, bytesRead)); + size += bytesRead; + } + stats = fs.fstatSync(fd); + if (size > MAX_SOURCE_FILE_SIZE_BYTES || stats.size > MAX_SOURCE_FILE_SIZE_BYTES) { + stats.size = Math.max(size, stats.size); + return { stats, bytes: null }; + } + return { stats, bytes: Buffer.concat(chunks, size) }; + } finally { + fs.closeSync(fd); + } +} diff --git a/src/mcp/answer-freshness.ts b/src/mcp/answer-freshness.ts index 297eab4728..5f5e224f59 100644 --- a/src/mcp/answer-freshness.ts +++ b/src/mcp/answer-freshness.ts @@ -1,5 +1,7 @@ import { createHash } from 'crypto'; import { createReadStream } from 'fs'; +import { stat } from 'fs/promises'; +import { MAX_SOURCE_FILE_SIZE_BYTES, oversizeStamp } from '../file-limits'; import { validatePathWithinRoot } from '../utils'; export interface AnswerFile { @@ -35,6 +37,14 @@ export async function validateAnswerFiles(root: string, files: AnswerFile[]): Pr const hash = createHash('sha256'); const absolute = validatePathWithinRoot(root, file.path); if (!absolute) { stale.push(file.path); continue; } + // A file over the index's size limit is stored as its size stamp + // (#1910): when the stamp still matches, the file is current without + // being read. Anything else falls through to the bounded read below. + const { size } = await stat(absolute); + if (size > MAX_SOURCE_FILE_SIZE_BYTES && + createHash('sha256').update(oversizeStamp(size)).digest('hex') === file.contentHash) { + continue; + } const stream = createReadStream(absolute, { encoding: 'utf8', highWaterMark: 64 * 1024, signal, }); diff --git a/src/mcp/tools.ts b/src/mcp/tools.ts index 05b6039115..7553ca5a65 100644 --- a/src/mcp/tools.ts +++ b/src/mcp/tools.ts @@ -119,6 +119,7 @@ function wslSharedIndexGuidance(err: WslSharedIndexError): string { */ export { PathRefusalError } from '../errors'; import { PathRefusalError } from '../errors'; +import { indexedHashInput } from '../file-limits'; import { resolve as resolvePath, relative as relativePath } from 'path'; /** Maximum output length to prevent context bloat (characters) */ @@ -2513,7 +2514,9 @@ export class ToolHandler { // Same freshness test as the sync fast path (extraction/index.ts): // equal size + equal floored mtime ⇒ unchanged, no read needed. if (st.size !== rec.size || Math.floor(st.mtimeMs) !== Math.floor(rec.modifiedAt)) { - const data = content ?? readFileSync(absPath, 'utf-8'); + // A file over the index's size limit is stored as its size stamp + // (#1910), so it is compared as one, without reading it. + const data = indexedHashInput(st.size, () => content ?? readFileSync(absPath, 'utf-8')); // Must stay byte-identical to extraction's `hashContent` (sha256 over // the utf-8 string) — the identical-rewrite test in // mcp-stale-slice.test.ts pins the parity. Inlined (not imported) diff --git a/src/ui-server/api/source.ts b/src/ui-server/api/source.ts index d2f32069bd..a81ce031e2 100644 --- a/src/ui-server/api/source.ts +++ b/src/ui-server/api/source.ts @@ -39,6 +39,7 @@ import * as path from 'path'; import type { FileRecord } from '../../types'; import type { CodeGraph } from '../../index'; import { resolveProjectFile } from '../security'; +import { indexedHashInput } from '../../file-limits'; import { highlightLines, type HighlightResult } from '../highlight'; import { ApiError, badRequest, intParam, notFound, textParam } from './respond'; @@ -209,9 +210,10 @@ export function hasDriftedOnDisk( if (stats.size === record.size && Math.floor(stats.mtimeMs) === Math.floor(record.modifiedAt)) { return false; } - if (stats.size > MAX_SOURCE_BYTES) return true; - const content = fs.readFileSync(absolute, 'utf-8'); - return createHash('sha256').update(content).digest('hex') !== record.contentHash; + // A file over the index's size limit is stored as its size stamp (#1910), + // so it is compared as one — without reading it. + const hashed = indexedHashInput(stats.size, () => fs.readFileSync(absolute, 'utf-8')); + return createHash('sha256').update(hashed).digest('hex') !== record.contentHash; } catch { return false; } @@ -250,7 +252,8 @@ export function readFileShape( return { drift: false, totalLines: null, reason: 'The file is too large to read here.' }; } const content = fs.readFileSync(absolute, 'utf-8'); - const drift = createHash('sha256').update(content).digest('hex') !== record.contentHash; + const hashed = indexedHashInput(stats.size, () => content); + const drift = createHash('sha256').update(hashed).digest('hex') !== record.contentHash; return { drift, totalLines: splitLines(content).length, @@ -394,7 +397,8 @@ export async function buildSource( // Byte-identical to extraction's `hashContent` (sha256 over the utf-8 // string). A touch or a checkout that rewrote the same bytes must not count // as drift, which is exactly what hashing content rather than mtime buys. - const hash = createHash('sha256').update(content).digest('hex'); + // A file over the index's size limit is stored as its size stamp (#1910). + const hash = createHash('sha256').update(indexedHashInput(stats.size, () => content)).digest('hex'); const drift = hash !== record.contentHash; if (drift && onDrift === 'omit') { return { From ee97a57a9bea47099fe235f1dc7341ad015e1ae8 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 03:01:50 +0000 Subject: [PATCH 014/363] fix(directory): a schema-less codegraph.db does not make a project initialized (#1895) (#2083) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(directory): a schema-less codegraph.db does not make a project initialized (#1895) isInitialized() accepted any existing .codegraph/codegraph.db, so an empty or table-less file in an ancestor directory (an interrupted init, a stray touch, a never-populated ~/.codegraph/) captured the upward walk of findNearestCodeGraphRoot / resolveProjectPath / resolveServerRoot for every project beneath it: the real index of a sub-project became unreachable, the sub-project down-scan never ran, and commands either operated on the broken file or failed opening it instead of showing the not-initialized guidance. An initialized project is now one whose db carries the schema: the probe is gated by file size (>= the 100-byte SQLite header), then the header magic, then a read-only node:sqlite open with a single sqlite_master lookup for the `nodes` table, memoized per path + mtime + size so hot callers (the prompt hook, MCP root resolution on every call) never reopen an unchanged db. The handle is closed immediately so the probe never holds the db on Windows; a locked/busy db counts as live. createDirectory() uses the same predicate, so `codegraph init` in the directory with the broken file repairs it in place (the schema is applied with CREATE ... IF NOT EXISTS) instead of refusing with "Already initialized"; the CLI says what it found. Nothing is deleted automatically. Co-Authored-By: Claude Fable 5.1 * fix(directory): a database we cannot open read-only still counts as initialized The schema probe failed CLOSED: a real WAL codegraph.db in a directory the process cannot write (a read-only checkout, a mount, another user's tree) made the read-only node:sqlite open throw — SQLite must create `-shm` to open a WAL database — and isInitialized() reported the project as not initialized, where main's file-exists check had been right. The probe now fails OPEN. Once the size gate and the `SQLite format 3\0` header have passed, the file is a SQLite database; only a SUCCESSFUL sqlite_master query proving the `nodes` table absent may return false. Any open/prepare error (locked, busy, cantopen, readonly, disk I/O) returns true, the pre-existing behaviour for a database we cannot inspect. No error strings are matched. The per path + mtime + size memo is unchanged, and schema-less/empty files still return false, so `codegraph init` still repairs them and never touches a real db. Test: a WAL db copied into a `.codegraph/` made read-only with chmod stays initialized — skipped on win32 and when running as root (root ignores mode bits). Co-Authored-By: Claude Fable 5.1 * fix(init): refuse a codegraph.db that is not SQLite instead of promising a rebuild Review found that a `.codegraph/codegraph.db` holding bytes that are not SQLite at all counted as "schema-less", so `init` printed "rebuilding it" and then failed with SQLite's "file is not a database". Only an empty file or a SQLite database without the codegraph tables can be rebuilt in place; `hasSchemalessDb` now means exactly that, and the new `hasForeignDbFile` names the other case. `init` stops on it with exit 1, says which file it is and to move or delete it, and never touches it. Co-Authored-By: Claude Opus 5.5 * fix(directory): ask SQLite what codegraph.db is instead of reading its header ourselves (#1895) The header check opened and closed a descriptor of our own on `codegraph.db`. Closing any descriptor on a database file drops every POSIX lock the process holds on it, including those of a connection it already has open (sqlite.org/howtocorrupt.html §2.2.1) — and the MCP server resolves projects through isInitialized on every call while it holds the index as its writer. mcp-projectpath-lifecycle's takeover tests caught it: after a real owner exits, the taking-over engine's next read failed with "disk I/O error" (2 of 21 fail with the header read; 21/21 on main and with this change). One read-only SQLite probe now answers all three questions: the schema is there, SQLite opened it and the schema is not (repairable), or SQLite refused it as not a database (SQLITE_NOTADB). Anything else — locked, busy, no `-shm` possible — still fails open as initialized. SQLite's own connections share one lock table per file, so opening and closing a second one leaves the first one's locks alone. A test pins that isInitialized and both init helpers never open the file themselves while this process holds it. Co-Authored-By: Claude Opus 5.5 * docs(changelog): keep the #1895 entry clear of #1917's, so the carries merge in any order Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Fable 5.1 --- CHANGELOG.md | 1 + __tests__/frontload-hook.test.ts | 9 +- .../schemaless-db-not-initialized.test.ts | 237 ++++++++++++++++++ src/bin/codegraph.ts | 13 +- src/directory.ts | 108 +++++++- 5 files changed, 357 insertions(+), 11 deletions(-) create mode 100644 __tests__/schemaless-db-not-initialized.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a95367e000..60b244fe04 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,6 +177,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) - Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. (#1902) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. +- An empty or table-less `codegraph.db` left behind in a parent directory (by an interrupted `codegraph init`, or a never-populated `~/.codegraph/`) no longer counts as an initialized project, so it can no longer hide the real index of every project beneath it. Only a database that actually carries the codegraph schema is treated as initialized, and `codegraph init` in the directory with the broken file now repairs it instead of refusing with "Already initialized". Thanks @danusha2345. (#1895) - Incremental sync now keeps edge rebinding crash-safe: replacing a resolved edge with its recovery reference commits atomically, so an interruption cannot permanently remove the relationship. - Status now detects committed but unindexed changes and restored edits without scanning every source file; thanks @inth3shadows. (#1829) diff --git a/__tests__/frontload-hook.test.ts b/__tests__/frontload-hook.test.ts index 6868e4c754..3e81e6090e 100644 --- a/__tests__/frontload-hook.test.ts +++ b/__tests__/frontload-hook.test.ts @@ -19,10 +19,13 @@ import { planFrontload, isTaskNotification, findIndexedSubprojectRoots, unsafeIn // fixture without changing the process environment or the user's home files. vi.mock('os', async (importOriginal) => ({ ...await importOriginal() })); -/** Make `dir` look indexed (isInitialized needs `.codegraph/codegraph.db`). */ +/** + * Make `dir` indexed. isInitialized needs `.codegraph/codegraph.db` WITH the + * codegraph schema — an empty file no longer counts (#1895). + */ function mkIndexed(dir: string): string { - fs.mkdirSync(path.join(dir, '.codegraph'), { recursive: true }); - fs.writeFileSync(path.join(dir, '.codegraph', 'codegraph.db'), ''); + fs.mkdirSync(dir, { recursive: true }); + CodeGraph.initSync(dir).close(); return dir; } /** A workspace-root manifest so the down-scan gate (looksLikeProjectRoot) passes. */ diff --git a/__tests__/schemaless-db-not-initialized.test.ts b/__tests__/schemaless-db-not-initialized.test.ts new file mode 100644 index 0000000000..10ae7b7689 --- /dev/null +++ b/__tests__/schemaless-db-not-initialized.test.ts @@ -0,0 +1,237 @@ +/** + * A schema-less `codegraph.db` does not make a project initialized (#1895). + * + * `isInitialized()` used to accept any existing `.codegraph/codegraph.db`, so + * an empty or table-less file in an ANCESTOR directory (an interrupted `init`, + * a stray `touch`, a never-populated `$HOME/.codegraph/`) captured the upward + * resolution of every project beneath it: the CLI and the MCP server operated + * on the broken file instead of the project's own index, or instead of the + * "not initialized" guidance. An initialized project is now one whose db + * carries the codegraph schema. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +// eslint-disable-next-line @typescript-eslint/no-require-imports +const { DatabaseSync } = require('node:sqlite'); +import { CodeGraph } from '../src'; +import { isInitialized, findNearestCodeGraphRoot, resolveServerRoot, planFrontload, hasSchemalessDb, hasForeignDbFile } from '../src/directory'; +import { ToolHandler } from '../src/mcp/tools'; + +// A configurable `fs`, so a test can watch which files the probe opens. +vi.mock('fs', async (importOriginal) => ({ ...await importOriginal() })); + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function run(cwd: string, args: string[]) { + const r = spawnSync(process.execPath, [BIN, ...args], { + cwd, + encoding: 'utf-8', + env: { ...process.env, CODEGRAPH_NO_DAEMON: '1', CODEGRAPH_WASM_RELAUNCHED: '1', NO_COLOR: '1' }, + }); + return { status: r.status, out: (r.stdout ?? '') + (r.stderr ?? '') }; +} + +/** A `.codegraph/codegraph.db` that exists but is not a codegraph index. */ +function plantBrokenDb(dir: string, kind: 'empty' | 'no-tables' | 'garbage'): string { + const cgDir = path.join(dir, '.codegraph'); + fs.mkdirSync(cgDir, { recursive: true }); + const dbPath = path.join(cgDir, 'codegraph.db'); + if (kind === 'empty') { + fs.writeFileSync(dbPath, ''); + } else if (kind === 'garbage') { + fs.writeFileSync(dbPath, 'x'.repeat(4096)); + } else { + // A valid SQLite file with zero tables — what an interrupted init leaves. + const db = new DatabaseSync(dbPath); + db.exec('CREATE TABLE t(x); DROP TABLE t;'); + db.close(); + } + return dbPath; +} + +async function indexProject(dir: string): Promise { + fs.mkdirSync(path.join(dir, 'src'), { recursive: true }); + fs.writeFileSync(path.join(dir, 'src', 'a.py'), 'def alpha(x):\n return beta(x) + 1\n\ndef beta(x):\n return x * 2\n'); + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + cg.close(); +} + +describe('a schema-less codegraph.db is not an initialized project (#1895)', () => { + let tmp: string; + let parent: string; + let child: string; + + beforeEach(() => { + tmp = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'cg-1895-'))); + parent = path.join(tmp, 'parent'); + child = path.join(parent, 'child'); + fs.mkdirSync(child, { recursive: true }); + }); + afterEach(() => { + fs.rmSync(tmp, { recursive: true, force: true }); + }); + + it.each(['empty', 'no-tables', 'garbage'] as const)('isInitialized() rejects a %s codegraph.db', (kind) => { + plantBrokenDb(parent, kind); + expect(isInitialized(parent)).toBe(false); + expect(findNearestCodeGraphRoot(child)).toBeNull(); + expect(resolveServerRoot(child).root).toBeNull(); + expect(planFrontload(child, 'how does alpha call beta').exploreRoot).toBeNull(); + }); + + // Root ignores mode bits and Windows has no chmod-based directory permissions. + const canDenyWrite = process.platform !== 'win32' && process.getuid?.() !== 0; + it.runIf(canDenyWrite)('fails OPEN: a WAL db in a directory we cannot open read-only is still initialized', async () => { + // SQLite needs to create `-shm` to open a WAL database; with the directory + // non-writable the read-only open throws. That is a database we cannot + // inspect, not a schema-less one — it must keep resolving (#1913 review). + await indexProject(parent); + const src = path.join(parent, '.codegraph', 'codegraph.db'); + const other = path.join(tmp, 'other'); + fs.mkdirSync(path.join(other, '.codegraph'), { recursive: true }); + fs.copyFileSync(src, path.join(other, '.codegraph', 'codegraph.db')); + const dir = path.join(other, '.codegraph'); + fs.chmodSync(dir, 0o555); + try { + expect(isInitialized(other)).toBe(true); + expect(findNearestCodeGraphRoot(path.join(other, 'sub'))).toBe(other); + } finally { + fs.chmodSync(dir, 0o755); + } + }); + + it('never opens the database through a descriptor of its own while this process holds it', async () => { + // Closing any descriptor on a database file drops every POSIX lock the + // process holds on it, including an open connection's (the MCP server + // resolves projects through isInitialized on every call, as the writer). + await indexProject(parent); + const live = await CodeGraph.open(parent); + const opened: string[] = []; + const real = fs.openSync; + const spy = vi.spyOn(fs, 'openSync').mockImplementation(((file: fs.PathLike, ...rest: unknown[]) => { + opened.push(String(file)); + return (real as (...args: unknown[]) => number)(file, ...rest); + }) as typeof fs.openSync); + try { + expect(isInitialized(parent)).toBe(true); + expect(hasSchemalessDb(parent)).toBe(false); + expect(hasForeignDbFile(parent)).toBe(false); + expect(live.getNodesByKind('function').map((n) => n.name).sort()).toEqual(['alpha', 'beta']); + } finally { + spy.mockRestore(); + live.close(); + } + expect(opened.filter((f) => path.basename(f).startsWith('codegraph.db'))).toEqual([]); + }); + + it('a real index still resolves upward from a subdirectory', async () => { + await indexProject(parent); + expect(isInitialized(parent)).toBe(true); + expect(findNearestCodeGraphRoot(child)).toBe(parent); + const r = run(child, ['status']); + expect(r.status).toBe(0); + expect(r.out).toContain(parent); + }); + + it('the cache follows the file: a db that gains its schema flips to initialized', async () => { + plantBrokenDb(parent, 'no-tables'); + expect(isInitialized(parent)).toBe(false); + await indexProject(parent); // repairs the file in place + expect(isInitialized(parent)).toBe(true); + }); + + it('(a) an empty db in the parent does not hijack the initialized child', async () => { + plantBrokenDb(parent, 'empty'); + await indexProject(child); + expect(findNearestCodeGraphRoot(child)).toBe(child); + expect(resolveServerRoot(child).root).toBe(child); + + const status = run(child, ['status']); + expect(status.status).toBe(0); + expect(status.out).toContain(child); + expect(status.out).not.toMatch(/SQLITE_|not a database|no such table/i); + + const explore = run(child, ['explore', 'alpha']); + expect(explore.status).toBe(0); + expect(explore.out).toContain('alpha'); + }); + + it('(a) a workspace container adopts its indexed sub-project even under a schema-less ancestor', async () => { + // The issue's exact reproduction: ws/ has .git and no index, ws/sub is + // indexed, and an ancestor of ws carries a zero-table db. + plantBrokenDb(tmp, 'no-tables'); + const ws = path.join(tmp, 'ws'); + const sub = path.join(ws, 'sub'); + fs.mkdirSync(path.join(ws, '.git'), { recursive: true }); + await indexProject(sub); + fs.writeFileSync(path.join(sub, 'pyproject.toml'), '[project]\nname = "sub"\nversion = "0.1"\n'); + const res = resolveServerRoot(ws); + expect(res.root).toBe(sub); + expect(res.viaSubScan).toBe(true); + }); + + it('(b) CLI: parent with a schema-less db, child not initialized → not-initialized guidance, never a SQLite error', () => { + plantBrokenDb(parent, 'no-tables'); + const r = run(child, ['status']); + expect(r.out).toMatch(/not initialized/i); + expect(r.out).toContain('codegraph init'); + expect(r.out).not.toMatch(/SQLITE_|not a database|no such table/i); + }); + + it('(b) MCP: the handler returns success-shaped not-indexed guidance for the child', async () => { + plantBrokenDb(parent, 'no-tables'); + const handler = new ToolHandler(null); + const res = await handler.execute('codegraph_explore', { query: 'alpha', projectPath: child }); + expect(res.isError).toBeUndefined(); + const text = res.content.map((c: any) => c.text ?? '').join('\n'); + expect(text).toMatch(/isn't indexed|codegraph init/i); + expect(text).not.toMatch(/SQLITE_|not a database|no such table/i); + }); + + it('(d) codegraph init in the directory with the broken file repairs it', () => { + const dbPath = plantBrokenDb(parent, 'no-tables'); + fs.writeFileSync(path.join(parent, 'a.py'), 'def alpha():\n return 1\n'); + const r = run(parent, ['init', '--yes']); + expect(r.status).toBe(0); + expect(r.out).toContain('without the codegraph schema'); + expect(r.out).not.toContain('Already initialized'); + expect(isInitialized(parent)).toBe(true); + const db = new DatabaseSync(dbPath, { readOnly: true }); + const row = db.prepare("SELECT count(*) AS c FROM sqlite_master WHERE type = 'table' AND name = 'nodes'").get() as { c: number }; + db.close(); + expect(row.c).toBe(1); + }); + + it('(d) an empty file is repaired too; a file that is not SQLite is refused with guidance and left alone', () => { + plantBrokenDb(parent, 'empty'); + fs.writeFileSync(path.join(parent, 'a.py'), 'def alpha():\n return 1\n'); + const empty = run(parent, ['init', '--yes']); + expect(empty.status).toBe(0); + expect(empty.out).toContain('without the codegraph schema'); + expect(isInitialized(parent)).toBe(true); + + for (const bytes of ['x'.repeat(4096), 'hello']) { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-1895-foreign-')); + try { + const cgDir = path.join(dir, '.codegraph'); + fs.mkdirSync(cgDir); + const dbPath = path.join(cgDir, 'codegraph.db'); + fs.writeFileSync(dbPath, bytes); + fs.writeFileSync(path.join(dir, 'a.py'), 'def alpha():\n return 1\n'); + const r = run(dir, ['init', '--yes']); + expect(r.status).toBe(1); + expect(r.out).toContain('is not a SQLite database'); + expect(r.out).not.toContain('rebuilding'); + expect(r.out).not.toContain('file is not a database'); + // Nothing is deleted or rewritten behind the user's back. + expect(fs.readFileSync(dbPath, 'utf-8')).toBe(bytes); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } + } + }); +}); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index e9da3a6e6f..cfe1cecfd0 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -41,7 +41,7 @@ try { import { Command } from 'commander'; import * as path from 'path'; import * as fs from 'fs'; -import { getCodeGraphDir, isInitialized, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; +import { getCodeGraphDir, isInitialized, hasSchemalessDb, hasForeignDbFile, unsafeIndexRootReason, findNearestCodeGraphRoot, planFrontload, isTaskNotification, hasStructuralKeyword, extractCodeTokens, capPromptHookInjection, codeGraphDirName, DEFAULT_CODEGRAPH_DIR } from '../directory'; import { extractProseCandidates } from '../search/identifier-segments'; import { detectWorktreeIndexMismatch, worktreeMismatchWarning } from '../sync/worktree'; import { createShimmerProgress } from '../ui/shimmer-progress'; @@ -697,6 +697,17 @@ async function runInit( return; } + if (hasForeignDbFile(projectPath)) { + const dbFile = path.join(getCodeGraphDir(projectPath), 'codegraph.db'); + clack.log.error(`${dbFile} is not a SQLite database, so it cannot be rebuilt in place.`); + clack.log.info('Move or delete that file, then run "codegraph init" again.'); + clack.outro(''); + process.exitCode = 1; + return; + } + if (hasSchemalessDb(projectPath)) { + clack.log.warn(`Found a codegraph.db without the codegraph schema in ${getCodeGraphDir(projectPath)} (left by an interrupted init?) — rebuilding it.`); + } const { default: CodeGraph, getDatabasePath } = await loadCodeGraph(); const cg = await CodeGraph.init(projectPath, { index: false }); clack.log.success(`Initialized in ${projectPath}`); diff --git a/src/directory.ts b/src/directory.ts index 1d758dbe03..0a08df9521 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -122,8 +122,21 @@ export function getCodeGraphDir(projectRoot: string): string { } /** - * Check if a project has been initialized with CodeGraph - * Requires both .codegraph/ directory AND codegraph.db to exist + * Check if a project has been initialized with CodeGraph. + * + * Requires `.codegraph/codegraph.db` to exist AND to carry the codegraph + * schema. A file that merely exists — empty, or a SQLite database with no + * tables, as an interrupted `init` or a stray `touch` leaves behind — used to + * count as initialized, so one such file in an ANCESTOR directory (worst + * case: `$HOME`) captured the upward resolution of every project beneath it + * and made their real indexes unreachable (#1895). + * + * The probe is cheap and gated so hot callers (the prompt hook, MCP root + * resolution on every call) pay one `stat`: file size, then a read-only + * SQLite open with a single `sqlite_master` lookup — memoized per path + + * mtime + size so an unchanged db is never reopened. A database that cannot + * be inspected counts as initialized (see probeSchema) — only a proven-absent + * schema, or a file SQLite refuses as not a database, says no. */ export function isInitialized(projectRoot: string): boolean { const codegraphDir = getCodeGraphDir(projectRoot); @@ -132,7 +145,88 @@ export function isInitialized(projectRoot: string): boolean { } // Must have codegraph.db, not just .codegraph folder const dbPath = path.join(codegraphDir, 'codegraph.db'); - return fs.existsSync(dbPath); + let st: fs.Stats; + try { + st = fs.statSync(dbPath); + } catch { + return false; + } + return hasCodeGraphSchema(dbPath, st); +} + +/** + * `codegraph.db` exists at `projectRoot` but does not carry the schema, and + * `init` can add it in place: an empty file, or a SQLite database without the + * codegraph tables (#1895). A file that is not SQLite at all is NOT this case — + * see {@link hasForeignDbFile}. + */ +export function hasSchemalessDb(projectRoot: string): boolean { + const dbPath = path.join(getCodeGraphDir(projectRoot), 'codegraph.db'); + let st: fs.Stats; + try { st = fs.statSync(dbPath); } catch { return false; } + if (!st.isFile() || isInitialized(projectRoot)) return false; + return st.size === 0 || probeSchema(dbPath) === 'no-schema'; +} + +/** + * `codegraph.db` exists at `projectRoot` and is not a SQLite database (no + * header magic): SQLite refuses to open it, so `init` cannot rebuild it in + * place. The caller must say so rather than promise a repair; nothing here + * deletes the file. + */ +export function hasForeignDbFile(projectRoot: string): boolean { + const dbPath = path.join(getCodeGraphDir(projectRoot), 'codegraph.db'); + let st: fs.Stats; + try { st = fs.statSync(dbPath); } catch { return false; } + return st.isFile() && st.size > 0 && probeSchema(dbPath) === 'not-sqlite'; +} + +/** A SQLite file header is 100 bytes; anything shorter cannot hold a schema. */ +const SQLITE_HEADER_SIZE = 100; +/** SQLITE_NOTADB: SQLite read the file and it is not a database. */ +const SQLITE_NOTADB = 26; +const schemaProbeCache = new Map(); + +function hasCodeGraphSchema(dbPath: string, st: fs.Stats): boolean { + if (!st.isFile() || st.size < SQLITE_HEADER_SIZE) return false; + const cached = schemaProbeCache.get(dbPath); + if (cached && cached.mtimeMs === st.mtimeMs && cached.size === st.size) return cached.ok; + const probe = probeSchema(dbPath); + const ok = probe === 'schema' || probe === 'unknown'; + schemaProbeCache.set(dbPath, { mtimeMs: st.mtimeMs, size: st.size, ok }); + return ok; +} + +/** + * What `codegraph.db` holds, asked of SQLite itself through a read-only + * connection: the codegraph schema, a database without it, not a database at + * all (SQLITE_NOTADB), or `unknown` — locked, busy, a WAL db in a directory we + * cannot create `-shm` in (read-only checkout, mount, another user's tree), + * disk I/O. Callers treat `unknown` as initialized: the pre-existing behaviour + * for a database we cannot inspect. + * + * The file is never read through a descriptor of our own, not even for its + * 16-byte header. Closing ANY descriptor on a database file drops every POSIX + * lock this process holds on it, including those of a connection it already + * has open (sqlite.org/howtocorrupt.html §2.2.1) — and the MCP server resolves + * projects through isInitialized on every call while it holds the index as + * its writer. SQLite's own connections share one lock table per file, so a + * second connection opened and closed here leaves the first one's locks alone. + */ +function probeSchema(dbPath: string): 'schema' | 'no-schema' | 'not-sqlite' | 'unknown' { + // eslint-disable-next-line @typescript-eslint/no-require-imports + const { DatabaseSync } = require('node:sqlite'); + let db: any = null; + try { + db = new DatabaseSync(dbPath, { readOnly: true }); + const row = db.prepare("SELECT 1 AS ok FROM sqlite_master WHERE type = 'table' AND name = 'nodes'").get(); + return row !== undefined ? 'schema' : 'no-schema'; + } catch (error) { + return (error as { errcode?: number })?.errcode === SQLITE_NOTADB ? 'not-sqlite' : 'unknown'; + } finally { + // Never hold the handle: Windows file locking would block the owner. + try { db?.close(); } catch { /* already closed */ } + } } /** @@ -806,11 +900,11 @@ function ensureGitignore(gitignorePath: string): boolean { */ export function createDirectory(projectRoot: string): void { const codegraphDir = getCodeGraphDir(projectRoot); - const dbPath = path.join(codegraphDir, 'codegraph.db'); - // Only throw if CodeGraph is actually initialized (db exists) - // .codegraph/ folder alone is fine - if (fs.existsSync(dbPath)) { + // Only throw if CodeGraph is actually initialized (db with a schema). + // .codegraph/ folder alone — or a schema-less codegraph.db left by an + // interrupted init (#1895) — is fine: initialize() adds the schema to it. + if (isInitialized(projectRoot)) { throw new Error(`CodeGraph already initialized in ${projectRoot}`); } From b44550e328258da4bd98c5bd902765cbea822b71 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 03:05:02 +0000 Subject: [PATCH 015/363] docs(changelog): credit @danusha2345 for the #1902 fix (#2085) Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 60b244fe04..b20e237c84 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -175,7 +175,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file over the size limit is no longer read before it is skipped. Large committed video and blob fixtures used to be loaded into memory in full, once per indexing pass, only to be discarded; indexing, change detection and the viewer now go by the file's size instead. Thanks @danusha2345. (#1910) - An MPEG transport stream video that happens to be named `.ts` (golden fixtures under `testdata/`, e2e clips) is now recognised from its first bytes and skipped instead of being fed to the TypeScript parser, which spent a long time on each clip for no symbols. Real TypeScript files are still indexed. Thanks @danusha2345. (#1910) - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) -- Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. (#1902) +- Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. Thanks @danusha2345. (#1902) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. - An empty or table-less `codegraph.db` left behind in a parent directory (by an interrupted `codegraph init`, or a never-populated `~/.codegraph/`) no longer counts as an initialized project, so it can no longer hide the real index of every project beneath it. Only a database that actually carries the codegraph schema is treated as initialized, and `codegraph init` in the directory with the broken file now repairs it instead of refusing with "Already initialized". Thanks @danusha2345. (#1895) - Incremental sync now keeps edge rebinding crash-safe: replacing a resolved edge with its recovery reference commits atomically, so an interruption cannot permanently remove the relationship. From 09b9646b7e340b750a8e997bcf1ed755dd0343fb Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 04:37:41 +0000 Subject: [PATCH 016/363] perf(context): compute the dominant file once per index state (#1864) (#2086) * perf(context): compute the dominant file once per index state (#1864) getDominantFile() backs context ranking's core-directory boost. Its answer depends only on the graph, but the whole-graph aggregation behind it (edges joined to both endpoints' nodes, grouped by file) ran on every generic codegraph_explore - seconds per call on a large index. Memoize it in QueryBuilder against a database change stamp: total_changes() (rows this connection wrote) plus PRAGMA data_version (commits by any other connection or process). Both are O(1), so no write path has to remember to invalidate, and a sync made by another process is seen on the next call. Any write forces a recompute; the heuristic's result is unchanged. Co-Authored-By: Claude Opus 5.5 * fix(context): never keep a dominant file read inside a transaction (#1864) total_changes() counts writes that are later rolled back, so a result computed inside a transaction could survive the ROLLBACK under an unchanged stamp and describe edges that no longer exist. getDominantFile now skips the memo while a transaction is open (and on a runtime without isTransaction). Ported from the maintainer's hardening of this fix in #1919. Co-Authored-By: Claude Opus 5.5 * fix(db): drop the dominant-file memo when a QueryBuilder is rebound (#1864) The memo's change stamp (total_changes + data_version) is per connection, and fresh read-only connections to two different databases report the same one ("0:2"). A pool worker rebound onto a rebuilt index kept serving the old database's dominant file until something wrote through the new connection. A test rebinds onto a second project and fails without the reset. Co-Authored-By: Claude Opus 5.5 * docs(changelog): credit @danusha2345 for #1864 Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/dominant-file-cache.test.ts | 140 ++++++++++++++++++++++++++ src/db/queries.ts | 53 ++++++++++ src/db/sqlite-adapter.ts | 6 ++ 4 files changed, 200 insertions(+) create mode 100644 __tests__/dominant-file-cache.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b20e237c84..c7eedc92c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,6 +174,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, the shared MCP daemon now waits longer for another program — an antivirus scan, an indexer, or another session reading its lock file — to let go of that file, so it starts instead of leaving the session to fall back to a slower in-process server. (#1773) - A file over the size limit is no longer read before it is skipped. Large committed video and blob fixtures used to be loaded into memory in full, once per indexing pass, only to be discarded; indexing, change detection and the viewer now go by the file's size instead. Thanks @danusha2345. (#1910) - An MPEG transport stream video that happens to be named `.ts` (golden fixtures under `testdata/`, e2e clips) is now recognised from its first bytes and skipped instead of being fed to the TypeScript parser, which spent a long time on each clip for no symbols. Real TypeScript files are still indexed. Thanks @danusha2345. (#1910) +- Repeated `codegraph_explore` calls from a running MCP server no longer re-rank the whole graph each time to find the project's core file; the answer is reused until the index changes. Thanks @danusha2345. (#1864) - File watching no longer drops the full re-scan a removed directory asks for when that sync fails, so the deleted files leave the index instead of lingering. (#1964) - Running `codegraph index` while the MCP server is running no longer leaves live auto-sync writing into the old, replaced index: the server now switches to the rebuilt index before its next update and catches up on everything it would otherwise have missed. Thanks @danusha2345. (#1902) - Daemon startup and cleanup now preserve live legacy PID-only locks while still reclaiming dead or identity-disproved records, preventing two writers from serving the same project. diff --git a/__tests__/dominant-file-cache.test.ts b/__tests__/dominant-file-cache.test.ts new file mode 100644 index 0000000000..f9ba57eef5 --- /dev/null +++ b/__tests__/dominant-file-cache.test.ts @@ -0,0 +1,140 @@ +/** + * getDominantFile() is computed once per database state (#1864). + * + * The dominant-file heuristic (context ranking's core-directory boost) is a + * whole-graph aggregation whose answer does not depend on the query, yet it + * ran on every generic explore — seconds per call on a large index. It is now + * memoized against a database change stamp. These tests pin both halves: + * repeated explores reuse the answer, and any write — by this connection or + * by another process's sync — makes the next call see the new graph. + */ + +import { describe, it, expect, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as os from 'os'; +import CodeGraph from '../src'; +import { QueryBuilder } from '../src/db/queries'; +import { DatabaseConnection, getDatabasePath } from '../src/db'; + +/** A file whose functions call each other in a chain: `n` in-file call edges. */ +function chain(prefix: string, n: number): string { + const lines: string[] = []; + for (let i = 0; i < n; i++) { + const next = i + 1 < n ? `${prefix}${i + 1}();` : ''; + lines.push(`export function ${prefix}${i}(): void { ${next} }`); + } + return lines.join('\n') + '\n'; +} + +function queriesOf(cg: CodeGraph): QueryBuilder { + return (cg as unknown as { queries: QueryBuilder }).queries; +} + +function spyCompute(cg: CodeGraph) { + return vi.spyOn(queriesOf(cg) as unknown as { computeDominantFile: () => unknown }, 'computeDominantFile'); +} + +describe('dominant file — computed once per index state (#1864)', () => { + let dir: string; + const open: CodeGraph[] = []; + + afterEach(() => { + for (const cg of open.splice(0)) cg.close(); + vi.restoreAllMocks(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function setup(): Promise { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dominant-')); + fs.mkdirSync(path.join(dir, 'core')); + fs.mkdirSync(path.join(dir, 'ext')); + fs.writeFileSync(path.join(dir, 'core', 'engine.ts'), chain('engineStep', 40)); + fs.writeFileSync(path.join(dir, 'ext', 'plugin.ts'), chain('pluginStep', 5)); + const cg = await CodeGraph.init(dir, { index: true }); + open.push(cg); + return cg; + } + + it('reuses the answer across explores while the database is unchanged', async () => { + const cg = await setup(); + const compute = spyCompute(cg); + + for (const q of ['engine step', 'plugin step', 'how does the engine run']) { + await cg.findRelevantContext(q); + } + expect(compute).toHaveBeenCalledTimes(1); + expect(queriesOf(cg).getDominantFile()?.filePath).toBe('core/engine.ts'); + expect(compute).toHaveBeenCalledTimes(1); + }); + + it('sees a new dominant file after a sync changes the graph', async () => { + const cg = await setup(); + expect(queriesOf(cg).getDominantFile()?.filePath).toBe('core/engine.ts'); + + fs.writeFileSync(path.join(dir, 'ext', 'plugin.ts'), chain('pluginStep', 120)); + await cg.sync(); + + const compute = spyCompute(cg); + expect(queriesOf(cg).getDominantFile()?.filePath).toBe('ext/plugin.ts'); + expect(compute).toHaveBeenCalledTimes(1); + }); + + it('sees a sync made through another connection (another process)', async () => { + const writer = await setup(); + const reader = await CodeGraph.open(dir); + open.push(reader); + expect(queriesOf(reader).getDominantFile()?.filePath).toBe('core/engine.ts'); + + fs.writeFileSync(path.join(dir, 'ext', 'plugin.ts'), chain('pluginStep', 120)); + await writer.sync(); + + expect(queriesOf(reader).getDominantFile()?.filePath).toBe('ext/plugin.ts'); + }); + + it('does not keep a result read inside a transaction that is rolled back', async () => { + const cg = await setup(); + const q = queriesOf(cg); + const before = q.getDominantFile(); + const db = (cg as any).db.getDb(); + db.exec('BEGIN'); + try { + db.exec('DELETE FROM edges'); + expect(q.getDominantFile()).toBeNull(); + } finally { + db.exec('ROLLBACK'); + } + expect(q.getDominantFile()).toEqual(before); + }); + + it('forgets the answer when rebound to another connection (a pool worker following a rebuilt index)', async () => { + await setup(); + const other = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-dominant-other-')); + const conns: DatabaseConnection[] = []; + try { + fs.mkdirSync(path.join(other, 'ext')); + fs.writeFileSync(path.join(other, 'ext', 'plugin.ts'), chain('pluginStep', 120)); + (await CodeGraph.init(other, { index: true })).close(); + conns.push(DatabaseConnection.open(getDatabasePath(dir)), DatabaseConnection.open(getDatabasePath(other))); + const q = new QueryBuilder(conns[0]!.getDb()); + expect(q.getDominantFile()?.filePath).toBe('core/engine.ts'); + // Two fresh connections to different databases can report the same + // change stamp, so only dropping the memo on rebind keeps this honest. + q.rebind(conns[1]!.getDb()); + expect(q.getDominantFile()?.filePath).toBe('ext/plugin.ts'); + } finally { + for (const c of conns) c.close(); + fs.rmSync(other, { recursive: true, force: true }); + } + }); + + it.runIf(process.platform !== 'win32')('sees a database rebuilt and reopened under it', async () => { + const reader = await setup(); + expect(queriesOf(reader).getDominantFile()?.filePath).toBe('core/engine.ts'); + fs.writeFileSync(path.join(dir, 'ext', 'plugin.ts'), chain('pluginStep', 120)); + const rebuilt = await CodeGraph.recreate(dir); + try { await rebuilt.indexAll(); } finally { rebuilt.close(); } + expect(reader.reopenIfReplaced()).toBe(true); + expect(queriesOf(reader).getDominantFile()?.filePath).toBe('ext/plugin.ts'); + }); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index 9666a256f5..86215d061b 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -251,6 +251,13 @@ export class QueryBuilder { private nodeCache: Map = new Map(); private readonly maxCacheSize = 1000; + // getDominantFile()'s answer, tagged with the database change stamp it was + // computed under (see getChangeStamp). Query-independent, so one value + // serves every explore until the database changes (#1864). + private dominantFileMemo: + | { stamp: string; value: { filePath: string; edgeCount: number; nextEdgeCount: number } | null } + | undefined; + // Prepared statements (lazily initialized) private stmts: { insertNode?: SqliteStatement; @@ -294,6 +301,7 @@ export class QueryBuilder { getAllFilePaths?: SqliteStatement; getAllNodeNames?: SqliteStatement; getDominantFile?: SqliteStatement; + getChangeStamp?: SqliteStatement; getTopRouteFile?: SqliteStatement; getRoutingManifest?: SqliteStatement; insertNameSegment?: SqliteStatement; @@ -379,6 +387,10 @@ export class QueryBuilder { this.stmts = {}; this.batchStmts.clear(); this.edgeKindStmts.clear(); + // The change stamp is per connection, and fresh connections to two + // different databases report the same one — the memo goes with the old + // connection, or a worker following a rebuilt index keeps its answer (#1864). + this.dominantFileMemo = undefined; } private edgeKindStmt(sql: string): SqliteStatement { @@ -1055,8 +1067,49 @@ export class QueryBuilder { * Excludes test/spec files from candidacy via path-pattern. The agent's * typical question is "how does X work", not "how is X tested", so * boosting a test file's directory would be a misfire. + * + * The answer depends only on the graph, never on the query, and the + * aggregation behind it scans every edge — seconds on a large index, paid + * by every generic explore (#1864). So it is memoized per database change + * stamp: recomputed only after something wrote to the database. */ getDominantFile(): { filePath: string; edgeCount: number; nextEdgeCount: number } | null { + // total_changes() counts writes that are later rolled back, so a result + // read inside a transaction could outlive a ROLLBACK under an unchanged + // stamp. Never keep one; `undefined` (a runtime without the getter) is + // treated the same way. + if (this.db.inTransaction !== false) { + this.dominantFileMemo = undefined; + return this.computeDominantFile(); + } + const stamp = this.getChangeStamp(); + if (this.dominantFileMemo?.stamp === stamp) return this.dominantFileMemo.value; + const value = this.computeDominantFile(); + this.dominantFileMemo = { stamp, value }; + return value; + } + + /** + * A value that differs whenever the database content may have changed + * since the last call, whoever changed it: `total_changes()` counts the + * rows this connection inserted, updated or deleted, and + * `PRAGMA data_version` moves when any OTHER connection — another process's + * sync, a CLI `codegraph index` beside a running MCP server — commits. Both + * are O(1), so no write path has to remember to invalidate anything. + * Coarse on purpose: any write, not just one to nodes/edges, forces a + * recompute, which only costs time, never a stale answer. + */ + private getChangeStamp(): string { + if (!this.stmts.getChangeStamp) { + this.stmts.getChangeStamp = this.db.prepare( + 'SELECT total_changes() AS changes, (SELECT data_version FROM pragma_data_version) AS version' + ); + } + const row = this.stmts.getChangeStamp.get() as { changes: number; version: number }; + return `${row.changes}:${row.version}`; + } + + private computeDominantFile(): { filePath: string; edgeCount: number; nextEdgeCount: number } | null { if (!this.stmts.getDominantFile) { // Pull top 20 candidates; we then filter out test/generated files // in code (regex-grade matching that SQL LIKE can't express). The diff --git a/src/db/sqlite-adapter.ts b/src/db/sqlite-adapter.ts index e94b587b95..39b2d8ba0f 100644 --- a/src/db/sqlite-adapter.ts +++ b/src/db/sqlite-adapter.ts @@ -32,6 +32,8 @@ export interface SqliteDatabase { transaction(fn: (...args: any[]) => T): (...args: any[]) => T; close(): void; readonly open: boolean; + /** Undefined on a runtime without `isTransaction`; callers must then not memoize. */ + readonly inTransaction?: boolean; } /** @@ -73,6 +75,10 @@ class NodeSqliteAdapter implements SqliteDatabase { return toWslSharedIndexError(err, this._dbPath) ?? err; } + get inTransaction(): boolean | undefined { + return this._db.isTransaction; + } + get open(): boolean { return this._db.isOpen; } From 2c4d4eb9e4c86a0cf7050136ba55e73aaa277503 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Tue, 29 Sep 2026 05:01:34 +0000 Subject: [PATCH 017/363] release: withhold codegraph ui until the viewer launches; hold its changelog entries (#2089) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(cli): withhold `codegraph ui` until the viewer launches (CODEGRAPH_UI=1 opts in) The viewer has been exercised on Expo apps only, not on API, Laravel, Spring, Angular or Swift projects, so it stays out of this release. `ui` and its alias `web` — also as `help ui` or `ui --help` — are refused before any startup work, and the command is hidden from `--help`, unless CODEGRAPH_UI=1 is set, which keeps it usable for the people testing it. The viewer assets still ship, so the opt-in works on a released build. Reverting this commit is the launch. Co-Authored-By: Claude Opus 5.5 * docs(site): say the viewer is not in a release yet The viewer pages have been live on the docs site since they landed, and 1.6.0 answers `codegraph ui` with "unknown command" (#1816, #1900). Until the viewer launches, the guide, the CLI reference and the next-steps link say so. Reverting this commit is part of the launch. Co-Authored-By: Claude Opus 5.5 * docs(changelog): hold the viewer's entries for its launch; new Highlights for the fixes release 53 entries that describe `codegraph ui` and its screens move, verbatim, to docs/viewer-launch-changelog.md along with the launch Highlights. 12 entries whose graph and `codegraph_explore` half ships now (the router navigation edges, the cross-tier hops, explore's `when` conditions, SvelteKit layouts, wrapped Express handlers, …) are reworded around the graph; their original wording is kept in the holding file for the launch. A few kept entries lose a viewer clause. The Highlights are rewritten for this release. prepare-release.mjs promotes the section and extract-release-notes.mjs reads it back with no mention of the viewer; the released sections are untouched. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- AGENTS.md | 1 + CHANGELOG.md | 213 +++--------------- __tests__/cli-ui-command.test.ts | 2 + __tests__/viewer-gate.test.ts | 72 ++++++ docs/viewer-launch-changelog.md | 204 +++++++++++++++++ .../docs/getting-started/next-steps.md | 2 +- site/src/content/docs/guides/viewer.md | 4 + site/src/content/docs/reference/cli.md | 6 +- src/bin/codegraph.ts | 16 +- src/bin/viewer-gate.ts | 29 +++ 10 files changed, 363 insertions(+), 186 deletions(-) create mode 100644 __tests__/viewer-gate.test.ts create mode 100644 docs/viewer-launch-changelog.md create mode 100644 src/bin/viewer-gate.ts diff --git a/AGENTS.md b/AGENTS.md index 144734c71a..0f5e26c68c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -89,6 +89,7 @@ The public API surface is `src/index.ts` — the `CodeGraph` class wires all the - `src/bin/codegraph.ts` — CLI (commander). Subcommands: `install`, `init`, `uninit`, `index`, `sync`, `status`, `query`, `files`, `context`, `affected`, `serve --mcp`. - `src/ui/` — terminal UI (shimmer progress, worker). - `src/ui-server/` -- read-only JSON API for the `codegraph ui` browser viewer (`api/`: `node`, `flow`, `map`, `screens`, `steps`, `deadcode`, `trails`, `program`, ...) plus static server; Svelte viewer lives in `ui/` (see `docs/design/codegraph-ui-design-spec.md`). `screens`/`steps`/`program` share one fold (`via`/`when` via `graph/branch-guards.ts`); `api/effects.ts` curates calls that leave the index; `api/route-roots.ts` names where a route's code starts. Derivations rendered by more than one surface belong in `src/graph/`, not `ToolHandler`. + **Not released yet:** `codegraph ui` / `web` are refused (and hidden from `--help`) unless `CODEGRAPH_UI=1` is set — `src/bin/viewer-gate.ts`. Viewer changelog entries wait in `docs/viewer-launch-changelog.md`, not under `[Unreleased]`. At launch: delete the gate, move those entries back, drop the "not released yet" notes from `site/`. ### NodeKind / EdgeKind diff --git a/CHANGELOG.md b/CHANGELOG.md index c7eedc92c8..ecc2fec329 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,134 +14,33 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Highlights -- **`codegraph ui` — your graph in a browser.** A local, read-only viewer for the project you already indexed: your code with its callers and callees in the margin, a map of the whole repository, and a strip that shows how one symbol reaches another. -- **See your app the way its users meet it.** A Screens tab draws every screen and the navigation between them, for Expo Router, React Router, Next.js, TanStack Router, Vue Router / Nuxt and SvelteKit apps. -- **See what happens from a screen or an endpoint.** A Steps tab draws what one action sets in motion — the handlers it fires, the state it writes, the calls that leave your code, and every way it can answer — with the condition on each arrow. -- **APIs too, and across tiers.** Endpoints in Express, NestJS, Fastify, Koa, Hono, FastAPI, Flask, Django, Spring, ASP.NET, Vapor and Gin, with a page's `fetch` following through to the route that serves it, a queued job to its consumer, an event to its handler. -- **Read a handler in the order its code runs.** The same picture laid out by when things happen rather than by distance, so a reply sits below the token it carries. Where the code chooses, the condition is said once and each arrow answers it. -- **Find what nothing reaches, and where everything starts.** Dead code and entry points as screens of their own, a type hierarchy showing what is built on what, and trails you can name, save and come back to after the code has changed. -- **Conditions and arguments for more languages.** What a call passes, and the `if` / `switch` / `try` it runs under, are now read for Python, Java, Kotlin, C#, Go and C as well as JavaScript, TypeScript and Swift. -- **Upgrading:** re-index your projects after this release — several of the new readings rest on edges that are written while indexing. +- **Sharper answers from `codegraph_explore`.** It returns the functions and methods you name — even from a very large file, or one full of same-named overrides — honours line numbers in a query, and says when it shortened an answer instead of calling it complete. +- **Your app's navigation and requests are in the graph.** Expo Router, Next.js, React Router, TanStack Router, Vue Router / Nuxt and SvelteKit pages and the links between them are routes and `navigates` edges, and a page's `fetch` reaches the route that serves it, so "where does this go" and "what calls this endpoint" are one question to `codegraph_explore`. +- **Explore says when a call happens.** Each hop in `codegraph_explore`'s Flow section carries the condition it runs under, for JavaScript, TypeScript, Swift, Python, Java, Kotlin, C#, Go and C. +- **Fewer wrong links.** Many fixes to how calls, imports and inheritance resolve in TypeScript, JavaScript, Python, Java, Kotlin, Go, Rust, Scala, PHP and C/C++, so a call stops landing on an unrelated symbol that happens to share its name. +- **Live sync you can rely on.** The MCP server follows a rebuilt index after `codegraph index`, a sync interrupted halfway recovers what a clean index would have, symlinked folders stay in sync, and a WSL project on a Windows drive gets an index of its own. +- **Lighter on big repositories.** Repeated explores from the MCP server no longer rescan the whole graph, and oversize files and `.ts` video fixtures are no longer read or parsed while indexing. +- **Upgrading:** re-index your projects after this release — several fixes and the new navigation links are written while indexing. ### New Features - **Codex and Astra read project guidance from `AGENTS.md`.** The canonical agent guide now lives in `AGENTS.md` (with a nested `docs/AGENTS.md` for long validation notes); `CLAUDE.md` is a thin `@AGENTS.md` wrapper for Claude Code. Codex/Astra no longer miss the old CLAUDE-only instructions. -- **A big screen's picture stops wrapping into a column.** How wide a screen's lines run before they wrap was worked out with a formula, and the formula was wrong for the way these pictures are actually drawn: a part of a screen spends lines on its own structure — a step that fires things gets a line to itself, and what it fires starts another — so estimating the lines from the boxes alone badly undercounted them, and one screen's 98 boxes wrapped into a 4,356px column. Laying a picture out is cheap and exact, so the widths are now simply tried and the one that comes out closest to the shape of a window is kept. Across one app's 51 screens the tallest picture went from 4,356px to 3,796px, total height fell 8%, and — because a shorter picture is also a picture whose lines have less far to go — lines running over other boxes fell by a third and lines crossing each other went from 13 to 5. +- **Next.js pages and their navigation are in the graph.** App Router pages (`app/(group)/blog/[slug]/page.tsx` → `/blog/:slug`) and Pages Router pages are routes bound to the component they export, and ``, an internal ``, `router.push` / `router.replace` (`next/navigation` and `next/router`), `redirect()` / `permanentRedirect()` and the middleware's `NextResponse.redirect(…)` are `navigates` edges between them. `app/api/**/route.ts` exports (`GET`, `POST`, …) are endpoints bound to their functions, and `pages/api/*` handlers are `ANY /api/…`. Re-index after upgrading. -- **A screen's picture stops being mostly empty.** The parts of a screen were tiled a row at a time, each row as tall as its tallest member — so one short region beside a tall one left the whole rest of that row blank. Measured across one app: the median screen's canvas was only 55% picture and 45% nothing, and its busiest screen was 44% — 4,860px tall to hold about 2,160px of content, all of which a reader has to scroll through. Each part now goes as high as it can and then as far left as it can, so a short one tucks under another short one instead of waiting for the tall one beside it. Reading order is unchanged: parts are still placed in the screen's own source order, and an earlier one is never pushed below a later one. That screen is now 3,584px instead of 4,860px, and close to square rather than a ribbon. +- **TanStack Router routes and navigation are in the graph.** Routes are read both ways a TanStack app declares them — file-based, where `createFileRoute('/posts/$postId')` carries the whole path, and code-based, where each `createRoute({ path, getParentRoute })` is composed through its parent — and `navigate({ to })`, a thrown `redirect({ to })` and `` / `` are `navigates` edges to the route they name. A pathless `_auth` layout, a `(group)` folder and `__root` are not pages, and a computed destination is left out rather than guessed. Re-index after upgrading. -- **The screen's own line into each part of itself stops sweeping across the picture.** A screen's picture holds back the screen's whole fan-out and draws one line into each region instead — but on a screen with ten regions those regions tile into bands, so the line into a region two bands down had to travel the height of the whole picture to get there. Measured across one app's 51 screens: those lines were 17% of everything drawn and caused **79% of every remaining crossing**. Two fixes. The line now lands on the box nearest the region's top-left that the screen actually leads to, rather than on whichever box the walk happened to meet first — which, once a screen is drawn in clusters, could be lines down inside the region. And when the region is still too far for a line to be followed, the link is said in words like any other, with no exception for the screen itself. Across those 51 screens, lines crossing each other went from 47 to **10**, with no screen above 10; the busiest screen went from 19 to 2. +- **Vue Router, Nuxt and SvelteKit navigation is in the graph.** **Vue:** routes are read from `createRouter({ routes: [...] })`, including a lazy `component: () => import('@/views/Login')`, and `router.push` / `router.replace` / `$router.push`, Nuxt's `navigateTo` and `` / `` / `` are `navigates` edges, resolved by route name as well as by path. **SvelteKit:** `goto('/login')`, `redirect(303, …)` from a load or a form action, and a plain `` are `navigates` edges, and a page is joined to the page file that serves it and the `+page.server.js` beside it. A computed destination or a path nothing declares is left out rather than guessed. Re-index after upgrading. -- **A busy screen's picture no longer draws itself as a web.** On a screen whose whole body is one component, the region grouping had nothing to divide — 98 of 100 boxes landed in a single column — and the picture came back as a 5,600px ribbon whose 113 lines crossed each other 652 times, each one running over five other boxes' names. Three things are fixed. A screen is now laid out in **clusters**: a step, then the steps it sets in motion on the line under it, stepped in — so a box and the thing it fires are one line apart instead of seven, which is what the crossings were made of. A region's lines run to a width its size earns rather than a fixed one, so a big screen comes out about as wide as it is tall instead of a narrow ribbon. And a link whose two boxes end up too far apart to follow is now **said in words at both ends** — `→ resumeInference` under the box that leads there, `← CaptureView` under the box it arrives at — rather than drawn as a line across the whole picture; select either box and every one of its real lines draws, exactly as before. On the screen that prompted this, the lines drawn at rest cross each other **once**, down from 652, with half of them still drawn as real lines. Steps that fire nothing still share a line, so a flat screen is unchanged. Nothing needs a re-index. +- **React Router navigation is in the graph.** `history.push` / `history.replace`, `navigate(…)` from `useNavigate`, `redirect()` in a loader or an action, and `` / `` / `` / react-router-bootstrap's `` are now `navigates` edges to the route they open, and a route with an optional parameter (`/cart/:id?`) is reached by both `/cart` and `/cart/5`. A computed destination or a path no route serves is left out rather than guessed, and an ordinary `paths.push('/x')` on an array is never mistaken for navigation. Re-index after upgrading. -- **A region that holds a cycle no longer stretches the picture.** Rows inside a region were settled by relaxation, which never converges on a graph that has a loop in it: one real screen's 65 boxes were pushed to rows 294-301 while the rest sat at rows 0-2, and the order they were drawn in had nothing to do with what leads to what. Cycle-closing links are now set aside before the rows are settled, as the in-order reading already did — the loop is still drawn, it just cannot stretch the picture. +- **`codegraph_explore` follows a web app across its tiers.** A page's `fetch('/api/users', { method: 'POST' })` — or an `axios` / `ky` / `got` / `$fetch` call, including one through an `axios.create({ baseURL })` instance — now reaches the route that serves it in the same index. A job put on a BullMQ / Bull queue reaches the `@Process` method, `Worker` or `queue.process` handler that consumes it, a NestJS `EventEmitter2` event its `@OnEvent` listeners, a socket message the gateway's `@SubscribeMessage` handler and back, and a Next.js server action called from a client file is a crossing to the server. Each is a synthesized hop that the Flow section names, with where it was wired up. Only a literal path or event name pairs. Re-index to pick the new edges up. -- **A busy screen's picture is laid out by the parts of the screen.** A screen is a set of handlers with no order between them, so on a hub screen the old rows-by-distance collapsed into one enormous row — the main screen of one app put 89 boxes side by side on a canvas over 28,000px wide, every line a near-horizontal sweep across all of it. The Steps tab now groups a screen's picture by region — the component that owns each handler, named in a small caption over its boxes — with each region a column where a step sits above what it sets in motion, tiled in the screen's own source order. At rest the picture hides only two things: the screen's own fan-out — one line into each region stands in for it — and lines that point back up; every other line draws where it leads, between two regions included, and selecting a step brings out its whole story in the side panel, link by link. A box nothing points at is the screen's own doing — run on render or mount, or from a binding written inline — the key says so, and selecting it lights its line from the screen with what fires it. The same app's widest screen now lays out under 3,500px with every line local, and the whole picture fits on screen when it opens. Endpoints, handlers and the in-order reading are untouched, and nothing needs a re-index: the regions come from the same walk that draws the steps. +- **`codegraph_explore` says when each call happens.** Every hop in the Flow section carries the condition its call site sits under — `↓ calls (when isCollected)` — read from the `if` / `else`, `switch` / `when` / `match`, ternary, `try` and `&&` / `and` branches around the call and the early returns before it (`if (busy) return` reads as `!busy`, Go's `if err != nil { return }` as `err == nil`). It works for JavaScript, TypeScript, Swift, Python, Java, Kotlin, C#, Go and C, is read from the source as it stands, and a language without rules says nothing rather than guess. -- **Where the code chooses, the picture says so once.** A helper that ends `return (await hasSeenWelcome(id)) ? '/home/' : '/welcome/'` sends the app to one of two screens, but the Steps picture drew that as two separate arrows, each carrying the whole condition with one of them negated and both cut off at the same forty characters — and before you clicked anything, neither arrow was labelled at all, so nothing said it was a choice. Now sibling arrows out of one box that are the arms of one `if`, `switch` or ternary are drawn as the choice they are: the condition is written once under the box that decides it, and each arrow out says only which way it is — `yes`, `no`, or a case's own value. They are the only arrows labelled before you select anything, so the picture reads at a glance without becoming a wall of text. A one-sided guard — an early exit, an `if` with only one side drawn — still carries its condition on the arrow, and an arrow that is reached whether or not the condition holds never claims a side. Nothing needs a re-index: the decision is read from the source at request time. +- **React Native apps: Swift native modules and their events connect end to end.** A JS call like `captureView.finalizeCaptureSession()` — where `captureView` is bound to `NativeModules.CaptureView` and the module is a Swift class exposed through an `RCT_EXTERN_MODULE` shim — now resolves to the Swift method itself instead of stopping at the constant, so `codegraph_explore` follows the code into native. Native → JS events now also land on listeners written inline (`addListener('onZipComplete', (data) => { … })`), attributed to the component that registers them. Re-index after upgrading to pick the new edges up. -- **A dialog's buttons fire what they run.** `Alert.prompt('Add Folder', …, [{ onPress: (name) => createBackgroundFolder(name) }])` is two facts: the prompt is a call that leaves the index, and its button fires the handler. The handler's line now arrives from the dialog's own box — with the condition on it — instead of from the screen, so the confirm-then-act chains a mobile app is full of read as chains: the delete alert leads to the delete, which leads to the request it sends. The same holds for anything bound inside the arguments of a call that leaves the index. Nothing needs a re-index: it is read from the source at request time. - -- **The Steps tab draws a handler in the order its code runs.** The picture of what an endpoint sets in motion put the lookup, the token signing, the 200 and the 401 side by side, because each is one step from the anchor — true, and not how the code reads. Now a handler, an endpoint or any function opens as the same picture laid out by *when* things happen: a line means **and then**, so the 200 sits below the token signing it is built from and the 401 branches off the check that chose it. Where the code forks both ways, a small box asks the condition once and each line out of it answers — `yes`, `no`, a case's value — so the two arms of one `if` read as one decision instead of two lines that each carry the whole predicate, one of them negated; selecting a step next to the decision lights the whole fork through it. A lone guard — an early exit, an `if` with one drawn side — keeps its condition on the line, and an arm that answers the request, returns or throws simply has nothing leaving it. A call written inside another call's arguments happens first, so the token is signed before the reply that carries it. A helper is drawn where it is called (`via generateToken`), a body that repeats says so (`for each item of items`), and work registered to run later (`later · then`) or started at once (`together · Promise.all`) says that rather than pretending to be a sequence. A screen still opens as before — its handlers fire on events and have no order between them — and either reading is one click, or one `&view=order` / `&view=tree` in the link, away. Nothing to re-index: it is read from the source at request time, and where the conditions cannot be read the picture is a plain sequence rather than an invented structure. - -- **A Next.js app lands on the Screens tab like a mobile app.** App Router pages (`app/(group)/blog/[slug]/page.tsx` → `/blog/:slug`) and Pages Router pages are screens bound to the component they export; ``, an internal ``, `router.push` / `router.replace` (`next/navigation` and `next/router`), `redirect()` / `permanentRedirect()` in a server action or a page, and the middleware's `NextResponse.redirect(new URL('/login', req.url))` are the transitions between them — each attributed back to the page it starts on with the plumbing folded and the condition on the arrow, a link written in markup drawn dashed as an inferred hop. `app/api/**/route.ts` exports (`GET`, `POST`, …) are endpoints bound to their functions, `pages/api/*` handlers are `ANY /api/…`, and a page's Steps picture fires from its load (`FIRES FROM page load · /users`), draws the data it reads, the handlers it wires, the server actions it crosses to and the pages it leads to as boundaries. A response's status written as `{ status: 201 }` is read too. Re-index after upgrading. - -- **TanStack Router apps land on the Screens tab too.** Routes are read both ways a TanStack app declares them: file-based, where `createFileRoute('/posts/$postId')` carries the whole path as a literal, and code-based, where each `createRoute({ path, getParentRoute })` names a fragment that is composed through its parent into `/posts/$postId`. `navigate({ to })` from `useNavigate`, a thrown `redirect({ to })` from a loader or `beforeLoad`, and `` / `` are the transitions between them. TanStack is the one router here whose destination is the route PATTERN rather than a filled address — `` names the route and passes the values beside it — so a destination is read as a pattern and matched against the route it names. Addresses that are not pages are left off the map: a `_auth` segment is a pathless layout and never appears in the URL, a `(group)` folder is invisible, a `__root` route wraps everything without being a page, and a file that renders an `` is the layout around an address while the index route beside it is the page at it. A computed `to`, a pattern no route serves, and a `navigate({ search })` that only changes the query are left out rather than guessed. Re-index after upgrading. - -- **Vue Router and SvelteKit apps land on the Screens tab too.** Both drew their screens as islands with no transitions, so the tab stayed hidden; now the navigation between pages is read for each. **Vue:** the routes are read out of `createRouter({ routes: [...] })` — path, name, and the view each entry names, including a lazy `component: () => import('@/views/Login')` — and `router.push` / `router.replace` / `$router.push`, Nuxt's `navigateTo`, and `` / `` / `` are the transitions. Vue apps usually navigate by route NAME rather than by path, so `router.push({ name: 'profile' })` and `:to="{ name: 'profile' }"` resolve by name, and `router.push({ path: '/', query })` by path. **SvelteKit:** `goto('/login')`, `redirect(303, '/article/' + slug)` from a load or a form action — whose destination is its *second* argument, after the status — and the plain `` that is a link in a SvelteKit app. A SvelteKit page also opens with a body now — it is joined to the page file that serves it and to the `+page.server.js` beside it — so its Steps picture draws its loader's work, its form actions, and the auth guard the loader performs (`redirect(302, '/login')` under `if (!locals.user)`) as a transition to the sign-in page, with the condition on the arrow. A computed destination, a path or name nothing declares, a relative path in a nested route, and a conditional whose two arms go to different pages are left out rather than guessed. Re-index after upgrading. - -- **A React Router app lands on the Screens tab too.** `` (v5), `}>` (v6) and `createBrowserRouter([{ path, element }])` already named a project's screens; now the navigation between them is drawn as well. `history.push('/placeorder')` and `history.replace`, `navigate('/placeorder')` from `useNavigate`, `redirect()` in a loader or an action, and `` / `` / `` / react-router-bootstrap's `` each become a transition — attributed back to the screen it starts on, with the plumbing folded and the condition on the arrow, a link written in markup drawn dashed as an inferred hop. A route with an optional parameter (`/cart/:id?`) is reached by both `/cart` and `/cart/5`. Until now a React Router project's screens were drawn as islands with no transitions at all, and a screen's Steps picture left out every page it sends you to — a checkout step showed its saved payment method but not that it goes on to place the order. A computed destination (`history.push(redirect)`), a path no route serves, and a relative path inside a nested route are left out rather than guessed, and an ordinary `paths.push('/x')` on an array is never mistaken for navigation. Re-index after upgrading. - -- **Double-click a box to go there.** On the Steps tab a double-click on any step starts the picture from it — the same as the panel's *Start here* — so an endpoint the page calls, or another screen drawn as a boundary, opens as its own chapter in one gesture; on the Screens tab a double-click on a screen opens what happens from it. A boundary's panel now says it is not entered rather than that nothing leaves it. - -- **The Steps tab follows a web app across its tiers.** A page's `fetch('/api/users', { method: 'POST' })` (or an `axios` / `ky` / `got` / `$fetch` call, including one through a project instance made with `axios.create({ baseURL })`) now reaches the route that serves it in the same index — drawn as a crossing to the server (`⇢ POST /api/users`) with the handler named on the box and the registration site in the panel, a boundary by default and entered with *Continue through*, so the picture reads page → handler → the endpoint → its database write → its response. A job put on a BullMQ / Bull queue lands on the `@Process` method, `Worker` or `queue.process` handler that consumes it; a NestJS `EventEmitter2` event lands on its `@OnEvent` listeners (globs included); a socket message crosses from a client's `socket.emit` to the gateway's `@SubscribeMessage` and back from the server's `emit` to the component that registered `socket.on`; and a Next.js server action called from a client file is a crossing to the server by its `'use server'` directive. Each of these is a synthesized hop — dashed, with where it was wired up — and `codegraph_explore`'s Flow section names them too. Only a literal path or event name pairs: a variable url, a path no route serves, or one two routes serve alike produce nothing. Re-index to pick the new edges up. - -- **The Steps tab now draws an API as well as an app.** Anchor on an endpoint — `POST /users` in Express, NestJS, Fastify, FastAPI, Flask, Django, Spring (Java or Kotlin), ASP.NET, Vapor or Gin — and the viewer starts at the handler the route runs (or at the route itself when the handler is an inline arrow), says what fires it (`FIRES FROM POST /users · after authenticate, validate(…)` — the middleware arguments at the registration, or the guard decorators on the method and its class, or a FastAPI `dependencies=[…]`), and draws what the request sets in motion: the database calls with the model and whether they read or write (`prisma.user.create({ data })` · `database · user · write`), jobs put on a queue, emails, payments, cache reads, token checks, calls to other services, files and processes — and the **responses**, one box per status a handler can send (`201`, `404`) whose panel rows are the endpoint's contract as the code has it: `WHEN NOT user → 404 · NotFoundException('no such user')`, `always → 201 · res.status(201).json(user)`. A queue consumer or a scheduled job anchored by name says the decorator that fires it (`@Process('email')`). The legend, the panel and the chooser use the project's own words — endpoint, data call, another tier — and the bare Steps tab lists an API's endpoints by router file when there are no screens. Re-index is not needed: everything new is read from the source at request time. - -- **Calls are read as written, so the database is the database.** The index keeps only the last segment of a deep member call (`create` for `prisma.user.create`), and a bare name matches by name alone — often to the wrong `create`. The Steps walk now reads each call from the source as written, classifies `prisma.user.create`, `this.usersRepository.save`, `session.commit`, `owners.save` and `_context.TodoItems.Add` by the whole chain and by the receiver's declared type (`OwnerRepository owners`, `private readonly usersService: UsersService`, `val owners: OwnerRepository`, read from the class body), and follows `this.usersService.findByEmail(…)` into the class the type names instead of the name-only guess. A hop resolved this way says so in the panel. - -- **Conditions and arguments for Python, Java, Kotlin, C#, Go and C.** The `when` on a call — `if` / `elif` / `else`, `switch` / `when` / `match`, the ternary and Kotlin's `if` expression, `try` / `except` / `catch`, `and` / `or`, and the early exits before it (`if err != nil { return }` reads as `err == nil`, `if not item.title: raise` as `item.title`) — and what each call passes (`HTTPException(status_code=422, detail="bad price")`, `c.JSON(http.StatusCreated, gin.H{…})`) are now read for those languages too, in `codegraph ui`'s rails, Flow strip and Steps tab and in `codegraph_explore`'s Flow section. As before: read from the source as it stands, never stored, and a language without rules yields nothing rather than a wrong label. - -- **A Steps tab in `codegraph ui` — what happens from here.** Pick a screen (or search any symbol and choose *What happens from here*) and the viewer draws everything it sets in motion as typed steps: the handlers wired to its taps and listeners, the calls that cross into native code, the native events that come back, the store actions it writes, and the calls that leave the app into the network, storage, the device or telemetry — one box per step, an arrow for every way one leads to the next, and on each arrow the condition under which it happens. The plumbing between two steps (hooks, helpers, the components in between) is folded into the arrow and listed in the side panel, exactly as the Screens tab folds a tap's chain into one transition — and every call the panel lists says what it passes, read from the source as written (`SecureStore.setItemAsync('userEmail', values.email)`, `axios.post('/auth/login', { email, password })`), so a step is not just *that* something was stored or sent but *what*. And each handler says what fires it — the JSX prop and its element (`onPress · +`, + // Deno: the import map says `@std/assert` comes from JSR. + 'deno/deno.json': JSON.stringify({ imports: { '@std/assert': 'jsr:@std/assert@^1', 'app/': '../src/' } }), + 'deno/test/assert.ts': `export function assertEquals(a: unknown, b: unknown) { return a === b; } +`, + 'deno/app.test.ts': `import { assertEquals } from '@std/assert'; +assertEquals(1, 1); +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const callsFrom = (file: string): string[] => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids, ['calls']) + .map((e) => cg.getNode(e.target)?.qualifiedName ?? '?') + .sort(); + }; + const fromVue = callsFrom('frontend/pages/labels.vue'); + expect(fromVue).not.toContain('ProvidersContext::t'); + expect(fromVue).not.toContain('dayjs'); + expect(fromVue).toContain('useLabelStore'); + // A `workspace:*` dependency is in the repository. + expect(fromVue).toContain('formatUrl'); + expect(callsFrom('frontend/composables/use-labels.ts')).toContain('useLabelStore'); + const fromSvelte = callsFrom('docs/src/routes/+page.svelte'); + expect(fromSvelte).toContain('useDesignSystem'); + expect(fromSvelte).not.toContain('goto'); + expect(fromSvelte).not.toContain('fetch'); + expect(callsFrom('deno/app.test.ts')).not.toContain('assertEquals'); + // A built-in's name imported from the project is the import. + const svelteIds = cg.getNodesInFile('docs/src/routes/+page.svelte').map((n) => n.id); + const imported = cg.getOutgoingEdgesFrom(svelteIds, ['imports']).map((e) => cg.getNode(e.target)!.filePath); + expect(imported).toContain('docs/src/lib/Map.svelte'); + expect(callsFrom('docs/src/lib/use.ts')).toContain('useDesignSystem'); + } finally { + cg.close(); + } + }); }); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index a5a90d3296..e5fa4f332d 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -41,6 +41,10 @@ import { lexicalPathWithinRoot } from '../utils'; import type { ReExport } from './types'; import { LRUCache } from './lru-cache'; import { JS_BUILT_INS } from './js-builtins'; +import { builtinModules } from 'module'; +import { parse as parseJsonc } from 'jsonc-parser'; + +const NODE_BUILTINS = new Set(builtinModules); /** Node kinds that can declare supertypes (extends/implements). */ const SUPERTYPE_BEARING_KINDS = new Set([ @@ -439,6 +443,7 @@ export class ReferenceResolver { this.supertypeGen++; this.nodesByKindCache.clear(); this.fileExistsMemo.clear(); + this.manifestScopes.clear(); this.knownNames = null; this.knownFiles = null; this.cachesWarmed = false; @@ -503,8 +508,8 @@ export class ReferenceResolver { resolveImport: (ref) => resolveViaImport(ref, this.context), isOutOfRepoImport: (source, fromFile, language) => isExternalImport(source, language, this.context) && - !this.isOwnPackage(source) && - resolveImportPath(source, fromFile, language, this.context) === null, + resolveImportPath(source, fromFile, language, this.context) === null && + this.isDeclaredOutsidePackage(source, fromFile), getNodesInFile: (filePath: string) => { if (!this.nodeCache.has(filePath)) { this.nodeCache.set(filePath, this.queries.getNodesByFile(filePath)); @@ -2343,10 +2348,13 @@ export class ReferenceResolver { private isBuiltInOrExternal(ref: UnresolvedRef): boolean { const name = ref.referenceName; const isJsTs = ref.language === 'typescript' || ref.language === 'javascript' - || ref.language === 'tsx' || ref.language === 'jsx' || ref.language === 'arkts'; + || ref.language === 'tsx' || ref.language === 'jsx' || ref.language === 'arkts' + || ref.language === 'vue' || ref.language === 'svelte' || ref.language === 'astro'; - // JavaScript/TypeScript built-ins - if (isJsTs && JS_BUILT_INS.has(name)) { + // JavaScript/TypeScript built-ins — unless the file imports its own + // binding of that name (`import Map from './Map.svelte'`). + if (isJsTs && JS_BUILT_INS.has(name) && + !this.context.getImportMappings(ref.filePath, ref.language).some((m) => m.localName === name)) { return true; } @@ -2873,20 +2881,67 @@ export class ReferenceResolver { } /** The repository's own package name, from its root package.json; null without one. */ - private ownPackageName: string | null | undefined; + /** Per directory: the package names its package.json and every enclosing one own and depend on. */ + private manifestScopes = new Map; deps: Set }>(); - /** `axios` (or `axios/unsafe/…`) inside the axios repository: a package importing itself. */ - private isOwnPackage(source: string): boolean { - if (this.ownPackageName === undefined) { - let name: string | null = null; - try { - const json = JSON.parse(this.context.readFile('package.json') ?? 'null') as { name?: unknown } | null; - if (json && typeof json.name === 'string' && json.name.length > 0) name = json.name; - } catch { /* no or unreadable package.json */ } - this.ownPackageName = name; + /** + * Is `source` a package from outside the repository? Only when the importing + * file's package.json (or an enclosing one) declares it, or it is a Node + * built-in or a runtime's virtual module. A specifier nothing declares is + * an alias this resolver does not know — SvelteKit's `$lib/…`, Nuxt's + * `~/…`, a nested app's own `@/…` — and stays the project's. + */ + private isDeclaredOutsidePackage(source: string, fromFile: string): boolean { + // Deno's standard library is `@std/…` from JSR. + if (/^(?:node|bun|jsr|npm|https?):/.test(source) || source.startsWith('@std/') || NODE_BUILTINS.has(source)) return true; + const name = source.startsWith('@') ? source.split('/').slice(0, 2).join('/') : source.split('/')[0]!; + const normalized = fromFile.replace(/\\/g, '/'); + const scope = this.manifestScope(normalized.includes('/') ? normalized.slice(0, normalized.lastIndexOf('/')) : ''); + if (scope.own.has(name)) return false; + if (scope.deps.has(name)) return true; + // SvelteKit's `$app/…` and Astro's `astro:…` belong to the framework — + // unless this repository is that framework. + const provider = source.startsWith('astro:') ? 'astro' : /^\$(?:app|env|service-worker)(?:\/|$)/.test(source) ? '@sveltejs/kit' : null; + return provider !== null && !scope.own.has(provider) && !this.context.getWorkspacePackages?.()?.byName.has(provider); + } + + private manifestScope(dir: string): { own: Set; deps: Set } { + const memo = this.manifestScopes.get(dir); + if (memo) return memo; + const parent = dir === '' ? null : this.manifestScope(dir.includes('/') ? dir.slice(0, dir.lastIndexOf('/')) : ''); + let scope = parent ?? { own: new Set(), deps: new Set() }; + try { + const json = JSON.parse(this.context.readFile(dir ? `${dir}/package.json` : 'package.json') ?? 'null') as Record | null; + if (json && typeof json === 'object') { + scope = { own: new Set(scope.own), deps: new Set(scope.deps) }; + if (typeof json.name === 'string' && json.name.length > 0) scope.own.add(json.name); + for (const field of ['dependencies', 'devDependencies', 'peerDependencies', 'optionalDependencies']) { + const deps = json[field]; + if (!deps || typeof deps !== 'object') continue; + for (const [dep, version] of Object.entries(deps)) { + // `workspace:*`, `file:../shared`, `link:…`: a package in this repository. + if (typeof version === 'string' && /^(?:workspace|file|link|portal):/.test(version)) scope.own.add(dep); + else scope.deps.add(dep); + } + } + } + } catch { /* no or unreadable package.json */ } + // A Deno import map names packages the same way — an entry that maps to a + // registry or a URL, not one that maps to a path in the repository. + for (const file of ['deno.json', 'deno.jsonc']) { + const raw = this.context.readFile(dir ? `${dir}/${file}` : file); + if (!raw) continue; + const json = parseJsonc(raw) as { imports?: Record } | undefined; + const imports = json && typeof json === 'object' ? json.imports : undefined; + if (!imports || typeof imports !== 'object') continue; + for (const [key, target] of Object.entries(imports)) { + if (typeof target !== 'string' || !/^(?:jsr|npm|https?):/.test(target)) continue; + if (scope === parent) scope = { own: new Set(scope.own), deps: new Set(scope.deps) }; + scope.deps.add(key.replace(/\/$/, '')); + } } - const own = this.ownPackageName; - return own !== null && (source === own || source.startsWith(`${own}/`)); + this.manifestScopes.set(dir, scope); + return scope; } /** The one supertype-kind node a TypeScript value shares its name and file with. */ diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index a290d30dab..ea09c6c01b 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -935,7 +935,13 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex return isCrossFileReachable(candidate, ref, context); } -const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +/** + * Languages whose calls are JS/TS calls — Vue, Svelte and Astro components' + * scripts and template expressions included: a bare `t('key')` in a `.vue` + * file resolves lexically exactly as in a `.ts` one. + */ +const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'vue', 'svelte', 'astro']); +const JS_TS = new Set(['typescript', 'tsx', 'javascript', 'jsx']); /** * Whether a JS/TS `calls` ref is a RECEIVER-LESS call — `serialize(x)`, not @@ -4283,9 +4289,11 @@ export function matchJsStoreBindingCall(ref: UnresolvedRef, context: ResolutionC } /** A qualified untyped chain is useful source evidence, not permission to - * infer a property type. Framework resolution runs before this guard. */ + * infer a property type. Framework resolution runs before this guard. Vue, + * Svelte and Astro files keep resolving them: there `api.groupReports.getAll()` + * reaches its API client class far more often than a wrong namesake. */ export function isUnresolvedJsMemberCall(ref: UnresolvedRef): boolean { - return ref.referenceKind === 'calls' && JS_FAMILY.has(ref.language) && + return ref.referenceKind === 'calls' && JS_TS.has(ref.language) && !/^(?:this|window)\./.test(ref.referenceName) && /^[A-Za-z_$][\w$]*(?:\.[A-Za-z_$][\w$]*){2,}$/.test(ref.referenceName); } From 92e14eda40e98dbdf43ea1c69f089afaa99e783b Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:32:33 +0000 Subject: [PATCH 045/363] fix(extraction): a file-based router reads only its own app in a monorepo (#2129) create-t3-turbo, tamagui's starter and react-native-reusables keep an Expo app beside a Next.js app. Expo Router read the Next app's `app/` folder as its own, so `layout.tsx` became a `/layout` screen, `page.tsx` a `/page` one, `_components/posts.tsx` and `api/auth/[...all]/route.ts` screens too, and a shared `packages/app/` library's files became screens as well. A resolver can now name `appDependencies`; its extractor runs on a file only when that file's package.json or an enclosing one declares one of them. When no manifest in the project declares any, detection found the framework by other evidence and it runs everywhere, as before. Expo Router names `expo-router`, Next.js `next`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/monorepo-app-frameworks.test.ts | 88 +++++++++++++++++++++++ src/extraction/index.ts | 73 +++++++++++++++++-- src/resolution/frameworks/expo-router.ts | 1 + src/resolution/frameworks/nextjs.ts | 1 + src/resolution/types.ts | 10 +++ 6 files changed, 170 insertions(+), 4 deletions(-) create mode 100644 __tests__/monorepo-app-frameworks.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f85c403290..c549985a80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. - A bare Java call like `verify(mock)`, `assertThat(x)` or `hashCode()` now links only to a method of the class it's in, of that class's parents, or one the file imports statically. It used to link to any class's method of that name: in halo, Mockito's `verify` and `eq` gave one service's `verify` over a thousand made-up callers, and in retrofit Truth's `assertThat` did the same to a test helper. Calls to inherited methods and static imports from the project now link to the right one, like jsoup's `attr(…)` through `LeafNode` or halo's `and(…)` / `equal(…)` through `import static …Queries.*`. Re-index Java projects after upgrading. - Python calls like `User.objects.get(…)`, `self.client.login(…)` or `request.POST.get(…)` no longer link to whichever class in the project has a method of that name. Such a call now links only to a member of what its receiver names, like `helpers.slugify()` to `slugify` in `helpers.py`, and a bare `get(1)` never links to a method. A name the file imports from a package outside the project, like `from django.shortcuts import render`, no longer links to a same-named project function. In Django projects these made-up links had given single test or serializer methods thousands of callers, for example netbox's `.all()` calls on one `UserConfig.all`. Re-index Python projects after upgrading. diff --git a/__tests__/monorepo-app-frameworks.test.ts b/__tests__/monorepo-app-frameworks.test.ts new file mode 100644 index 0000000000..f7efbf8caa --- /dev/null +++ b/__tests__/monorepo-app-frameworks.test.ts @@ -0,0 +1,88 @@ +/** + * In a monorepo, a file-based router reads only its own app's files. + * + * create-t3-turbo, tamagui's starter and react-native-reusables keep an Expo + * app beside a Next.js app. Expo Router read the Next app's `app/` folder as + * its own: `app/layout.tsx` became a `/layout` screen, `app/page.tsx` a + * `/page` one, a `_components/posts.tsx` a `/_components/posts` screen, and + * `app/api/auth/[...all]/route.ts` a screen named after the file. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-app-frameworks-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +const routesIn = (cg: CodeGraph, dir: string): string[] => + cg + .getNodesByKind('route') + .filter((n) => n.filePath.startsWith(dir)) + .map((n) => n.name) + .sort(); + +describe('file-based routers in a monorepo', () => { + it('Expo Router reads the Expo app, Next.js the Next app', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'mono', private: true, workspaces: ['apps/*'] }), + 'apps/mobile/package.json': JSON.stringify({ name: 'mobile', dependencies: { expo: '*', 'expo-router': '*', react: '*' } }), + 'apps/mobile/app/_layout.tsx': `export default function Layout() { return null; } +`, + 'apps/mobile/app/index.tsx': `export default function Home() { return null; } +`, + 'apps/mobile/app/user/[id].tsx': `export default function User() { return null; } +`, + 'apps/web/package.json': JSON.stringify({ name: 'web', dependencies: { next: '*', react: '*' } }), + 'apps/web/app/layout.tsx': `export default function RootLayout({ children }: { children: unknown }) { return children; } +`, + 'apps/web/app/page.tsx': `export default function Page() { return null; } +`, + 'apps/web/app/user/[id]/page.tsx': `export default function UserPage() { return null; } +`, + 'apps/web/app/_components/posts.tsx': `export default function Posts() { return null; } +`, + 'apps/web/app/api/auth/[...all]/route.ts': `export async function GET() { return new Response('ok'); } +`, + }); + try { + expect(routesIn(cg, 'apps/mobile/')).toEqual(['/', '/user/[id]']); + const web = routesIn(cg, 'apps/web/'); + expect(web).toContain('/'); + expect(web).toContain('/user/:id'); + for (const bogus of ['/layout', '/page', '/_components/posts', '/api/auth/[...all]/route', '/user/[id]/page']) { + expect(web).not.toContain(bogus); + } + } finally { + cg.close(); + } + }); + + it('a single Expo app at the repository root keeps its routes', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'demo', dependencies: { expo: '*', 'expo-router': '*' } }), + 'app/_layout.tsx': `export default function Layout() { return null; } +`, + 'app/settings.tsx': `export default function Settings() { return null; } +`, + }); + try { + expect(routesIn(cg, 'app/')).toEqual(['/settings']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index a4fec2ac3c..706bc0bd12 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -32,7 +32,8 @@ import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; import { validatePathWithinRoot, normalizePath } from '../utils'; import ignore, { Ignore } from 'ignore'; -import { detectFrameworks } from '../resolution/frameworks'; +import { detectFrameworks, getFrameworkResolver } from '../resolution/frameworks'; +import { declaredDependencies } from '../resolution/frameworks/package-deps'; import type { ResolutionContext } from '../resolution/types'; import { createYielder, type MaybeYield } from '../resolution/cooperative-yield'; import { MAX_SOURCE_FILE_SIZE_BYTES, oversizeStamp, readBoundedSource, readBoundedSourceSync } from '../file-limits'; @@ -1908,9 +1909,72 @@ export class ExtractionOrchestrator { const fileList = files ?? scanDirectory(this.rootDir); const context = this.buildDetectionContext(fileList); this.detectedFrameworkNames = detectFrameworks(context).map((r) => r.name); + const declared = declaredDependencies(context); + this.gatedFrameworks = new Map(); + for (const name of this.detectedFrameworkNames) { + const deps = getFrameworkResolver(name)?.appDependencies; + // Gated only when some manifest names the framework: otherwise detection + // found it by other evidence and every file is its app's. + if (deps && deps.some((d) => declared.has(d))) this.gatedFrameworks.set(name, deps); + } + this.appFrameworkMemo.clear(); + this.manifestDependencies.clear(); return this.detectedFrameworkNames; } + /** Detected frameworks whose extractors run only inside their own apps, with the packages that mark one. */ + private gatedFrameworks = new Map(); + /** `|` → does the package.json at or above `dir` declare the framework. */ + private appFrameworkMemo = new Map(); + /** Directory → the dependency names its package.json declares, null without one. */ + private manifestDependencies = new Map | null>(); + + /** + * The detected frameworks whose extractors apply to `filePath`: all of them, + * except one with `appDependencies` when neither the file's package.json nor + * an enclosing one declares any of them. + */ + private frameworksForFile(filePath: string, names: string[]): string[] { + if (this.gatedFrameworks.size === 0) return names; + const slash = filePath.lastIndexOf('/'); + const dir = slash < 0 ? '' : filePath.slice(0, slash); + const kept = names.filter((name) => this.frameworkAppliesIn(dir, name)); + return kept.length === names.length ? names : kept; + } + + private frameworkAppliesIn(dir: string, name: string): boolean { + const deps = this.gatedFrameworks.get(name); + if (!deps) return true; + const key = `${dir}|${name}`; + const memo = this.appFrameworkMemo.get(key); + if (memo !== undefined) return memo; + let applies = false; + for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { + const declared = this.dependenciesDeclaredIn(d); + if (declared && deps.some((dep) => declared.has(dep))) { + applies = true; + break; + } + } + this.appFrameworkMemo.set(key, applies); + return applies; + } + + private dependenciesDeclaredIn(dir: string): Set | null { + if (this.manifestDependencies.has(dir)) return this.manifestDependencies.get(dir)!; + let declared: Set | null = null; + try { + const pkg = JSON.parse(fs.readFileSync(path.join(this.rootDir, dir, 'package.json'), 'utf8')) as Record; + declared = new Set(); + for (const field of ['dependencies', 'devDependencies', 'peerDependencies']) { + const group = pkg[field]; + if (group && typeof group === 'object') for (const name of Object.keys(group)) declared.add(name); + } + } catch { /* no or unreadable manifest */ } + this.manifestDependencies.set(dir, declared); + return declared; + } + /** * Index all files in the project */ @@ -2099,8 +2163,9 @@ export class ExtractionOrchestrator { */ const parseFile = (filePath: string, content: string): Promise => { const language = detectLanguage(filePath, content, overrides); - if (!pool) return Promise.resolve(extractFromSource(filePath, content, language, frameworkNames)); - return pool.requestParse({ filePath, content, language, frameworkNames }); + const names = this.frameworksForFile(filePath, frameworkNames); + if (!pool) return Promise.resolve(extractFromSource(filePath, content, language, names)); + return pool.requestParse({ filePath, content, language, frameworkNames: names }); }; // --- Bounded rolling-window dispatch, ordered commit --- @@ -2722,7 +2787,7 @@ export class ExtractionOrchestrator { // Extract from source. Use cached framework names if indexAll has run, // otherwise detect on the spot so single-file re-index paths still emit // route nodes / middleware / etc. - const frameworkNames = this.ensureDetectedFrameworks(); + const frameworkNames = this.frameworksForFile(relativePath, this.ensureDetectedFrameworks()); const result = extractFromSource(relativePath, content, language, frameworkNames); // Store in database diff --git a/src/resolution/frameworks/expo-router.ts b/src/resolution/frameworks/expo-router.ts index 2f513e3fad..5d82b94e26 100644 --- a/src/resolution/frameworks/expo-router.ts +++ b/src/resolution/frameworks/expo-router.ts @@ -713,6 +713,7 @@ function scoreMatch(href: string[], route: string[]): number | null { export const expoRouterResolver: FrameworkResolver = { name: 'expo-router', languages: [...ROUTE_LANGUAGES], + appDependencies: ['expo-router'], detect(context: ResolutionContext): boolean { if (dependsOn(context, 'expo-router')) return true; diff --git a/src/resolution/frameworks/nextjs.ts b/src/resolution/frameworks/nextjs.ts index 7eb543dee9..22b85f7466 100644 --- a/src/resolution/frameworks/nextjs.ts +++ b/src/resolution/frameworks/nextjs.ts @@ -222,6 +222,7 @@ export function nextNavVerb(name: string): string | null { export const nextjsResolver: FrameworkResolver = { name: 'nextjs', languages: [...ROUTE_LANGUAGES], + appDependencies: ['next'], detect(context: ResolutionContext): boolean { if (dependsOn(context, 'next')) return true; diff --git a/src/resolution/types.ts b/src/resolution/types.ts index 7b76bb324e..65fd98cafe 100644 --- a/src/resolution/types.ts +++ b/src/resolution/types.ts @@ -239,6 +239,16 @@ export interface FrameworkResolver { name: string; /** Languages this framework applies to. If omitted, applies to all languages. */ languages?: Language[]; + /** + * Packages an app declares when it is built on this framework. When set, + * `extract()` runs only on files of an app whose package.json — the file's + * own or an enclosing one — declares one of them: in a monorepo with an Expo + * app beside a Next.js app, Expo Router must not read the Next app's + * `app/layout.tsx` as a `/layout` screen. When no manifest in the project + * declares any, detection found the framework by other evidence and the + * extractor runs on every file, as before. + */ + appDependencies?: readonly string[]; /** Detect if project uses this framework (project-level, called once at startup) */ detect(context: ResolutionContext): boolean; /** Resolve a reference using framework-specific patterns */ From 54a1698caa219563d136453ddcf4b573f9c170b0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:46:26 +0000 Subject: [PATCH 046/363] fix(vue-router): admin-template route tables, children and layouts; Nuxt routes only in Nuxt (#2130) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit vue-element-admin, vue-admin-template and vben had no routes: their tables are named arrays (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`) or per-module objects (`const tableRouter = {…}`) handed to `new Router(...)`, and most of their screens are `children`. The Vue Router reader now reads those tables in any file that builds a router, imports vue-router or lives in a router/ directory; joins children onto their parent's path (a parent is a screen only when no child claims its address and it doesn't redirect); binds a lazy view by the FILE it imports (all of vue-element-admin's are `…/index.vue`), through an alias the resolver can't follow by the one file in the app with that path; links each screen to its parents' components as layouts; and takes `this.$router.push`. Nuxt's file routes move into their own `nuxt` resolver, detected only in a Nuxt app: halo's plain-Vue console got 30 made-up screens from its `pages/` folders. A Nuxt app at the repository root now gets its routes, and a top-level `pages/index.vue` is `/`, not `/index`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/vue-router.test.ts | 164 ++++++++++++++++- docs/design/framework-coverage.md | 6 +- src/resolution/frameworks/index.ts | 6 +- src/resolution/frameworks/vue-router.ts | 229 +++++++++++++++++++----- src/resolution/frameworks/vue.ts | 38 +++- src/ui-server/api/screens.ts | 2 +- 8 files changed, 386 insertions(+), 62 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c549985a80..a7dd3a4bfd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. - A bare Java call like `verify(mock)`, `assertThat(x)` or `hashCode()` now links only to a method of the class it's in, of that class's parents, or one the file imports statically. It used to link to any class's method of that name: in halo, Mockito's `verify` and `eq` gave one service's `verify` over a thousand made-up callers, and in retrofit Truth's `assertThat` did the same to a test helper. Calls to inherited methods and static imports from the project now link to the right one, like jsoup's `attr(…)` through `LeafNode` or halo's `and(…)` / `equal(…)` through `import static …Queries.*`. Re-index Java projects after upgrading. diff --git a/README.md b/README.md index e0aae08860..3d02481dd8 100644 --- a/README.md +++ b/README.md @@ -344,7 +344,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | -| **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` with the view each entry names, plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | +| **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | | **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | | **Angular** | `Routes` arrays (`RouterModule.forRoot` / `forChild`, `provideRouter`, a routes file's default export) with `component` or a lazy `loadComponent`; `children` and lazy `loadChildren` (an NgModule's through its routing module) joined into full paths; paths written as route constants or `$localize` strings; a route with children is a layout around the screens inside it | `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array, a route constant, or a component property holding one — `routerLink` / `[routerLink]` in the component's template, and `redirectTo`. Each template's child components (``) are linked to the component that renders them | diff --git a/__tests__/vue-router.test.ts b/__tests__/vue-router.test.ts index abe0871866..de94f2989e 100644 --- a/__tests__/vue-router.test.ts +++ b/__tests__/vue-router.test.ts @@ -65,15 +65,18 @@ describe('vue-router: parseVueRoutes', () => { ['home', '/'], ['login', '/login'], ['settings', '/settings'], + [null, '/profile/:username/favorites'], ['profile', '/profile/:username'], ]); }); it('reads the component from a lazy import and from an identifier', () => { - expect(entries.map((e) => e.component)).toEqual(['Home', 'Login', 'Settings', 'Profile']); + expect(entries.map((e) => e.component)).toEqual(['Home', 'Login', 'Settings', 'Favorites', 'Profile']); }); - it('skips a child route, whose path is relative to a parent this does not compose', () => { + it('joins a child route onto its parent, which is the layout around it', () => { + const child = entries.find((e) => e.path === '/profile/:username/favorites')!; + expect(child.layouts.map((l) => l.component)).toEqual(['Profile']); expect(entries.some((e) => e.path === 'favorites')).toBe(false); }); @@ -299,3 +302,160 @@ describe('vue-router: a routed app end to end', () => { expect(screens.dropped).toBe(0); }); }); + +// ============================================================================= +// Admin-template route tables, and Nuxt's own file routes +// ============================================================================= + +describe('vue-router: route tables beyond an inline createRouter', () => { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vue-tables-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); + } + + const routeNames = (cg: CodeGraph): string[] => cg.getNodesByKind('route').map((n) => n.name).sort(); + const edgesFrom = (cg: CodeGraph, routeName: string, kind: 'calls' | 'references') => { + const r = cg.getNodesByKind('route').find((n) => n.name === routeName)!; + return cg.getOutgoingEdges(r.id).filter((e) => e.kind === kind).map((e) => ({ edge: e, target: cg.getNode(e.target)! })); + }; + + it('vue-element-admin: named tables, `new Router`, module files, children and layouts', async () => { + const view = (name: string) => `\n\n`; + const cg = await project({ + 'package.json': JSON.stringify({ name: 'admin', dependencies: { vue: '^2.6.0', 'vue-router': '^3.0.0' } }), + 'src/router/index.js': `import Vue from 'vue' +import Router from 'vue-router' +import Layout from '@/layout' +import tableRouter from './modules/table' +Vue.use(Router) + +export const constantRoutes = [ + { path: '/login', component: () => import('@/views/login/index'), hidden: true }, + { + path: '/', + component: Layout, + redirect: '/dashboard', + children: [ + { path: 'dashboard', component: () => import('@/views/dashboard/index'), name: 'Dashboard' } + ] + } +] + +export const asyncRoutes = [tableRouter, { path: '*', redirect: '/404', hidden: true }] + +const createRouter = () => new Router({ routes: constantRoutes }) +export default createRouter() +`, + 'src/router/modules/table.js': `import Layout from '@/layout' + +const tableRouter = { + path: '/table', + component: Layout, + redirect: '/table/complex-table', + name: 'Table', + children: [ + { path: 'complex-table', component: () => import('@/views/table/complex-table'), name: 'ComplexTable' } + ] +} +export default tableRouter +`, + 'src/layout/index.vue': view('Layout'), + 'src/views/login/index.vue': ` + +`, + 'src/views/dashboard/index.vue': view('Dashboard'), + 'src/views/table/complex-table.vue': view('ComplexTable'), + }); + try { + // `/` and `/table` redirect into their children: frames, not pages. + expect(routeNames(cg)).toEqual(['/dashboard', '/login', '/table/complex-table']); + // Each lazy view binds to ITS file's component, though every one is `index`. + expect(edgesFrom(cg, '/dashboard', 'calls').map((e) => e.target.filePath)).toEqual(['src/views/dashboard/index.vue']); + expect(edgesFrom(cg, '/login', 'calls').map((e) => e.target.filePath)).toEqual(['src/views/login/index.vue']); + // The parent's component is the layout around it. + const layouts = edgesFrom(cg, '/table/complex-table', 'references'); + expect(layouts.map((e) => e.target.filePath)).toEqual(['src/layout/index.vue']); + expect(layouts[0]!.edge.metadata).toMatchObject({ layout: true }); + // Navigation by name reaches the composed child. + const nav = cg + .getOutgoingEdgesFrom(cg.getNodesInFile('src/views/login/index.vue').map((n) => n.id), ['navigates' as never]) + .filter((e) => e.kind === 'navigates'); + expect(nav.map((e) => cg.getNode(e.target)!.name)).toEqual(['/dashboard']); + } finally { + cg.close(); + } + }); + + it('vben: a typed routes module with an alias the resolver cannot follow', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'mono', private: true }), + 'apps/web/package.json': JSON.stringify({ name: 'web', imports: { '#/*': './src/*' }, dependencies: { vue: '*', 'vue-router': '*' } }), + 'apps/web/src/router/routes/modules/dashboard.ts': `import type { RouteRecordRaw } from 'vue-router'; + +const routes: RouteRecordRaw[] = [ + { + name: 'Dashboard', + path: '/dashboard', + children: [ + { name: 'Analytics', path: 'analytics', component: () => import('#/views/dashboard/analytics/index.vue') }, + ], + }, +]; + +export default routes; +`, + 'apps/web/src/views/dashboard/analytics/index.vue': ` + +`, + }); + try { + expect(routeNames(cg)).toEqual(['/dashboard/analytics']); + expect(edgesFrom(cg, '/dashboard/analytics', 'calls').map((e) => e.target.filePath)).toEqual([ + 'apps/web/src/views/dashboard/analytics/index.vue', + ]); + } finally { + cg.close(); + } + }); + + it('Nuxt pages at the repository root are routes; a plain Vue app’s pages/ folder is not', async () => { + const nuxt = await project({ + 'package.json': JSON.stringify({ name: 'site', dependencies: { nuxt: '*', vue: '*' } }), + 'pages/index.vue': '\n', + 'pages/users/[id].vue': '\n', + }); + try { + expect(routeNames(nuxt)).toEqual(['/', '/users/:id']); + } finally { + nuxt.close(); + } + const plain = await project({ + 'ui/package.json': JSON.stringify({ name: 'console', dependencies: { vue: '*', 'vue-router': '*' } }), + 'ui/src/modules/contents/pages/SinglePageList.vue': '\n', + }); + try { + expect(routeNames(plain)).toEqual([]); + } finally { + plain.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index febc60a301..b149ccb7d7 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -40,7 +40,7 @@ guessed. | Next.js | `frameworks/nextjs.ts` | `next-router-synthesizer.ts` | `nextjs.test.ts` | next-saas-starter | | React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | -| Vue Router / Nuxt | `frameworks/vue-router.ts` | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges) | +| Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts` | sveltekit-realworld (31 edges) | | Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | @@ -191,6 +191,10 @@ Each of these cost real debugging time; they are not hypothetical. `name` is written above its `path`, so a text window handed every entry its predecessor's name — silently, for every route in the file. Use `frameworks/object-literal.ts`. + A lazy view binds by the FILE it imports, never by the import's last + segment: vue-element-admin's views are all `…/index.vue`. + Nuxt's `pages/` convention belongs to a Nuxt app only — a plain Vue app's + `pages/` folder (halo's console) holds components a router config names. 5. **A receiver is required for a generic verb.** `push` and `replace` are two of the most common method names in JavaScript; claiming a bare one puts every `paths.push('/tmp/x')` one string-match away from a route. diff --git a/src/resolution/frameworks/index.ts b/src/resolution/frameworks/index.ts index 97af4183fb..88221fdfa6 100644 --- a/src/resolution/frameworks/index.ts +++ b/src/resolution/frameworks/index.ts @@ -18,7 +18,7 @@ import { vueRouterResolver } from './vue-router'; import { angularRouterResolver } from './angular-router'; import { svelteKitRouterResolver } from './sveltekit-router'; import { svelteResolver } from './svelte'; -import { vueResolver } from './vue'; +import { vueResolver, nuxtResolver } from './vue'; import { astroResolver } from './astro'; import { djangoResolver, flaskResolver, fastapiResolver } from './python'; import { railsResolver } from './ruby'; @@ -58,6 +58,8 @@ const FRAMEWORK_RESOLVERS: FrameworkResolver[] = [ // SvelteKit — `src/routes/**/+page.svelte` routes are `svelteResolver`'s; `goto('/x')` / `redirect(303, '/x')` → navigates edges svelteKitRouterResolver, vueResolver, + // Nuxt — `pages/**` screens, `server/api/**` endpoints and `middleware/`, in a Nuxt app only + nuxtResolver, // Vue Router — `createRouter({ routes })` → route nodes; `router.push({ name })` / `router.push('/x')` → navigates edges vueRouterResolver, angularRouterResolver, @@ -162,7 +164,7 @@ export { vueRouterResolver } from './vue-router'; export { angularRouterResolver } from './angular-router'; export { svelteKitRouterResolver } from './sveltekit-router'; export { svelteResolver } from './svelte'; -export { vueResolver } from './vue'; +export { vueResolver, nuxtResolver } from './vue'; export { astroResolver } from './astro'; export { djangoResolver, flaskResolver, fastapiResolver } from './python'; export { railsResolver } from './ruby'; diff --git a/src/resolution/frameworks/vue-router.ts b/src/resolution/frameworks/vue-router.ts index 1a9884e8ba..84a1a9beb5 100644 --- a/src/resolution/frameworks/vue-router.ts +++ b/src/resolution/frameworks/vue-router.ts @@ -64,6 +64,7 @@ import { type RouteTable, } from './expo-router'; import { destinationsForHref } from './nextjs'; +import { resolveImportPath } from '../import-resolver'; const ROUTE_LANGUAGES: readonly Language[] = ['typescript', 'javascript', 'vue']; @@ -72,75 +73,137 @@ const ROUTE_LANGUAGES: readonly Language[] = ['typescript', 'javascript', 'vue'] // ============================================================================= export interface VueRouteEntry { - /** `/profile/:username` — the path, in the form every other framework's routes use. */ + /** `/profile/:username` — the full path, a child's joined onto its parents'. */ path: string; /** `profile` — what `router.push({ name })` names, when the entry has one. */ name: string | null; /** The component the entry names, by identifier or by the tail of its lazy import. */ component: string | null; + /** `@/views/dashboard/index` — the file a lazy `() => import(…)` component names. */ + spec: string | null; + /** The components of the parent routes it renders inside, outermost first. */ + layouts: VueLayout[]; line: number; } +export interface VueLayout { + component: string; + spec: string | null; +} + /** A file that builds a router — the cheap gate before parsing anything. */ -const ROUTER_FACTORY = /\b(?:createRouter|createWebHistory|createWebHashHistory|createMemoryHistory)\s*\(|\bnew\s+VueRouter\s*\(/; +const ROUTER_FACTORY = /\b(?:createRouter|createWebHistory|createWebHashHistory|createMemoryHistory)\s*\(|\bnew\s+(?:VueRouter|Router)\s*\(/; /** `routes: [` / `routes = [` — the array itself, for a file that only holds the table. */ const ROUTES_ARRAY = /\broutes\s*[:=]\s*\[/; +/** A file that imports from `vue-router`. */ +const VUE_ROUTER_IMPORT = /\bfrom\s*['"]vue-router['"]/; + +/** A file in a `router/` or `routes/` directory, or named `router.js` / `routes.ts`. */ +const ROUTE_FILE_PATH = /(?:^|\/)(?:router|routes)(?:\/|\.[cm]?[jt]s$)/; + +/** + * Where a routes table starts: a `routes: [` field, or a declaration named + * for what it holds — `export const constantRoutes = [`, `const routes: + * RouteRecordRaw[] = [`, vue-element-admin's per-module `const tableRouter = {`. + */ +const TABLE_OPENERS = /\broutes\s*[:=]\s*\[|\b(?:const|let|var)\s+(?:[A-Za-z_$][\w$]*)?(?:[Rr]outes?|[Rr]outers?)\s*(?::[^=;\n]*)?=\s*([[{])/g; + /** - * Every top-level entry of a `routes: [...]` array. + * Every route a file's routes tables declare. * - * The array is walked, not pattern-matched: a `name` is written ABOVE the + * The tables are walked, not pattern-matched: a `name` is written ABOVE the * `path` it belongs to, so reading fields out of a window around each `path` * hands an entry its PREDECESSOR's name — vue-realworld's `login` came out as - * `/register`, silently, for every route in the file. So each top-level `{…}` - * is matched as a unit and only its own depth-1 fields are read; a nested - * `children:` array, a `meta: {…}` and a lazy `component: () => import(…)` - * are stepped over rather than searched. + * `/register`, silently, for every route in the file. So each `{…}` is + * matched as a unit and only its own depth-1 fields are read; a `meta: {…}` + * and a lazy `component: () => import(…)` are stepped over rather than + * searched, and `children` are walked in turn with the parent's path in front + * of theirs. + * + * A route with children is a layout: its component renders the outlet they + * fill. It is a screen of its own only when no child claims its address and + * it does not `redirect` elsewhere — vue-element-admin's `{ path: '/table', + * component: Layout, redirect: '/table/complex-table', children }` is the + * frame around four tables, not a fifth page. * - * An entry whose path does not start with `/` is a child route, relative to a - * parent this does not compose, and is not a destination on its own. + * A top-level entry whose path does not start with `/` (`*`, a catch-all) is + * not an address. A table is read from a file that builds a router, imports + * `vue-router`, or lives in a `router/` directory. */ -export function parseVueRoutes(content: string): VueRouteEntry[] { - if (!ROUTER_FACTORY.test(content) && !ROUTES_ARRAY.test(content)) return []; +export function parseVueRoutes(content: string, filePath = ''): VueRouteEntry[] { + const routeFile = ROUTER_FACTORY.test(content) || ROUTES_ARRAY.test(content) || VUE_ROUTER_IMPORT.test(content) || ROUTE_FILE_PATH.test(filePath); + if (!routeFile) return []; const safe = stripCommentsForRegex(content, 'typescript'); const out: VueRouteEntry[] = []; const seen = new Set(); - const arrays = /\broutes\s*[:=]\s*\[/g; - let a: RegExpExecArray | null; - while ((a = arrays.exec(safe)) !== null) { - const open = a.index + a[0].length - 1; - const close = matchBracket(safe, open); - if (close < 0) continue; - for (const obj of topLevelObjects(safe, open + 1, close)) { - const fields = readFields(safe, obj.start, obj.end); - const pathField = fields.get('path'); - if (!pathField) continue; - const path = readStringAt(pathField.text.trimStart(), 0); - if (path === null || !path.startsWith('/')) continue; - const componentField = fields.get('component') ?? fields.get('components'); - if (!componentField) continue; // no component in the entry → not a route object - const component = componentName(componentField.text); - if (!component) continue; - const nameField = fields.get('name'); - const name = nameField ? readStringAt(nameField.text.trimStart(), 0) : null; - const line = safe.slice(0, pathField.at).split('\n').length; - const key = `${path} ${name ?? ''}`; - if (seen.has(key)) continue; - seen.add(key); - out.push({ path, name, component, line }); + const walked = new Set(); + const lineOf = (at: number) => safe.slice(0, at).split('\n').length; + + const visit = (obj: { start: number; end: number }, prefix: string | null, layouts: VueLayout[], depth: number): void => { + const fields = readFields(safe, obj.start, obj.end); + const pathField = fields.get('path'); + if (!pathField) return; + const own = readStringAt(pathField.text.trimStart(), 0); + if (own === null) return; + let path: string; + if (own.startsWith('/')) path = own; + else if (prefix === null) return; // a top-level `*` or relative path is no address + else path = own === '' ? prefix : `${prefix === '/' ? '' : prefix}/${own}`; + if (path.length > 1) path = path.replace(/\/+$/, ''); + + const componentField = fields.get('component') ?? fields.get('components'); + const component = componentField ? componentOf(componentField.text) : null; + + const children = fields.get('children'); + let childClaimsAddress = false; + if (children && depth < 12) { + const open = safe.indexOf('[', children.at); + const close = open < 0 ? -1 : matchBracket(safe, open); + if (open >= 0 && close > open) { + const inner = component ? [...layouts, component] : layouts; + for (const child of topLevelObjects(safe, open + 1, close)) { + const childPath = readFields(safe, child.start, child.end).get('path'); + const childOwn = childPath ? readStringAt(childPath.text.trimStart(), 0) : null; + if (childOwn === '' || childOwn === path) childClaimsAddress = true; + visit(child, path, inner, depth + 1); + } + } } - arrays.lastIndex = close; + if (!component) return; + if (children && (childClaimsAddress || fields.has('redirect'))) return; + const nameField = fields.get('name'); + const name = nameField ? readStringAt(nameField.text.trimStart(), 0) : null; + const key = `${path} ${name ?? ''}`; + if (seen.has(key)) return; + seen.add(key); + out.push({ path, name, component: component.component, spec: component.spec, layouts: [...layouts], line: lineOf(pathField.at) }); + }; + + TABLE_OPENERS.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = TABLE_OPENERS.exec(safe)) !== null) { + const open = m.index + m[0].length - 1; + const close = matchBracket(safe, open); + if (close < 0 || walked.has(open)) continue; + walked.add(open); + if (safe[open] === '{') visit({ start: open, end: close }, null, [], 0); + else for (const obj of topLevelObjects(safe, open + 1, close)) visit(obj, null, [], 0); + TABLE_OPENERS.lastIndex = close; } return out; } /** The component an entry names: an identifier, or the file a lazy import names. */ -function componentName(value: string): string | null { +function componentOf(value: string): VueLayout | null { const lazy = /\bimport\s*\(\s*['"`]([^'"`]+)['"`]/.exec(value); - if (lazy) return (lazy[1]!.split('/').pop() ?? '').replace(/\.\w+$/, '') || null; + if (lazy) { + const tail = (lazy[1]!.split('/').pop() ?? '').replace(/\.\w+$/, ''); + return tail ? { component: tail, spec: lazy[1]! } : null; + } const ident = /^\s*([A-Z][A-Za-z0-9_]*)\s*$/.exec(value); - return ident?.[1] ?? null; + return ident ? { component: ident[1]!, spec: null } : null; } function languageForFile(filePath: string): Language { @@ -175,7 +238,7 @@ function isVueConfigRoute(node: Node): boolean { function isNuxtPage(node: Node): boolean { return ( node.language === 'vue' && - node.filePath.includes('/pages/') && + `/${node.filePath}`.includes('/pages/') && node.id === `route:${node.filePath}:${node.name}:1` ); } @@ -212,7 +275,7 @@ export function vueRouteTable(context: ResolutionContext): VueRouteTable { if (!content) continue; const byName = tableAt(group.root).byName; const byPath = new Map(group.nodes.map((n) => [n.name, n])); - for (const entry of parseVueRoutes(content)) { + for (const entry of parseVueRoutes(content, filePath)) { if (!entry.name) continue; const node = byPath.get(entry.path); if (node && !byName.has(entry.name)) byName.set(entry.name, node); @@ -228,13 +291,13 @@ export function vueRouteTable(context: ResolutionContext): VueRouteTable { // ============================================================================= /** - * `router.push` / `.replace` (the Composition API), `$router.push` / - * `.replace` (the Options API and templates), and Nuxt's `navigateTo`. + * `router.push` / `.replace` (the Composition API), `this.$router.push` / + * `$router.push` (the Options API and templates), and Nuxt's `navigateTo`. * * As everywhere else, `push` and `replace` need a receiver that names a * router: an unqualified `push` is an array's. */ -const NAV_CALL = /^\$?router\.(?:push|replace)$|^navigateTo$/; +const NAV_CALL = /^(?:this\.)?\$?router\.(?:push|replace)$|^navigateTo$/; /** The verb a navigation call name stands for, or null. */ export function vueNavVerb(name: string): string | null { @@ -282,6 +345,56 @@ function vueComponentNamed(name: string, fromFile: string, context: ResolutionCo return near.length === 1 ? near[0]! : null; } +/** `import:@/views/Login#Login` for a lazy component, the bare name for an identifier. */ +function componentRefName(component: string, spec: string | null): string { + return spec === null ? component : `import:${spec}#${component}`; +} + +/** + * The component a route's reference names. A lazy import names a FILE, and + * the component is that file's: vue-element-admin's views are all + * `…/index.vue`, so the name alone (`index`) fits dozens. The import is + * resolved as the router file would resolve it, or — through an alias the + * resolver doesn't know, like vben's `#/views/…` — by the one file in the + * same app whose path ends the way the specifier does. + */ +function routeComponent(encoded: string, fromFile: string, context: ResolutionContext): Node | null { + const lazy = /^import:(.+)#([^#]+)$/.exec(encoded); + // `component: Layout` — the file's own import of it says which file that is + // (`import Layout from '@/layout'`, a `layout/index.vue`). + const spec = lazy ? lazy[1]! : (context.getImportMappings(fromFile, languageForFile(fromFile)).find((m) => m.localName === encoded)?.source ?? null); + const name = lazy ? lazy[2]! : encoded; + if (spec !== null) { + const file = resolveImportPath(spec, fromFile, 'vue', context) ?? fileBySuffix(spec, fromFile, context); + const component = file ? componentInFile(file, context) : null; + if (component) return component; + } + return name === 'index' ? null : vueComponentNamed(name, fromFile, context); +} + +function componentInFile(file: string, context: ResolutionContext): Node | null { + const nodes = context.getNodesInFile(file); + return nodes.find((n) => n.kind === 'component') ?? nodes.find((n) => n.kind === 'function' && /^[A-Z]/.test(n.name)) ?? null; +} + +const COMPONENT_EXTENSIONS = ['', '.vue', '.tsx', '.jsx', '.ts', '.js', '/index.vue', '/index.tsx', '/index.ts', '/index.js']; + +/** The one file under `fromFile`'s app whose path ends like `spec` minus its alias (`#/`, `@/`, `~/`). */ +function fileBySuffix(spec: string, fromFile: string, context: ResolutionContext): string | null { + const rest = spec.replace(/^(?:[@#~$][\w-]*\/|\.{1,2}\/)+/, ''); + if (rest === spec || rest.length === 0) return null; + const root = appRootFor(fromFile); + const hits = new Set(); + for (const file of context.getAllFiles()) { + if (!file.startsWith(root)) continue; + for (const ext of COMPONENT_EXTENSIONS) { + const tail = rest + ext; + if (file === tail || file.endsWith('/' + tail)) hits.add(file); + } + } + return hits.size === 1 ? [...hits][0]! : null; +} + // ============================================================================= // The resolver // ============================================================================= @@ -289,17 +402,18 @@ function vueComponentNamed(name: string, fromFile: string, context: ResolutionCo export const vueRouterResolver: FrameworkResolver = { name: 'vue-router', languages: [...ROUTE_LANGUAGES], + appDependencies: ['vue-router', 'nuxt', 'nuxt3'], detect(context: ResolutionContext): boolean { return dependsOn(context, 'vue-router', 'nuxt', 'nuxt3'); }, claimsReference(name: string): boolean { - return NAV_CALL.test(name); + return NAV_CALL.test(name) || name.startsWith('import:') || name.startsWith('layout:'); }, extract(filePath: string, content: string): FrameworkExtractionResult { - const entries = parseVueRoutes(content); + const entries = parseVueRoutes(content, filePath); if (entries.length === 0) return { nodes: [], references: [] }; const language = languageForFile(filePath); const now = Date.now(); @@ -329,7 +443,7 @@ export const vueRouterResolver: FrameworkResolver = { // a `calls` edge to a component as the page a screen renders. references.push({ fromNodeId: node.id, - referenceName: entry.component, + referenceName: componentRefName(entry.component, entry.spec), referenceKind: 'calls', line: entry.line, column: 0, @@ -338,11 +452,30 @@ export const vueRouterResolver: FrameworkResolver = { candidates: [entry.component], }); } + // The layouts around it: what they render — a sidebar, a navbar — is on + // this screen too. + for (const layout of entry.layouts) { + references.push({ + fromNodeId: node.id, + referenceName: `layout:${componentRefName(layout.component, layout.spec)}`, + referenceKind: 'references', + line: entry.line, + column: 0, + filePath, + language, + }); + } } return { nodes, references }; }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind === 'references' && isVueRouteRef(ref) && ref.referenceName.startsWith('layout:')) { + const layout = routeComponent(ref.referenceName.slice('layout:'.length), ref.filePath, context); + return layout + ? { original: ref, targetNodeId: layout.id, confidence: 0.95, resolvedBy: 'framework', metadata: { layout: true } } + : null; + } if (ref.referenceKind !== 'calls') return null; // A route naming the component it renders — this resolver's own reference, @@ -350,7 +483,7 @@ export const vueRouterResolver: FrameworkResolver = { // `Login` view AND a `login` action in a store, and only one of them is // the screen. if (isVueRouteRef(ref)) { - const component = vueComponentNamed(ref.referenceName, ref.filePath, context); + const component = routeComponent(ref.referenceName, ref.filePath, context); return component ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework' } : null; diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index c830885be3..03e50509ac 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -1,12 +1,13 @@ /** * Vue / Nuxt Framework Resolver * - * Handles Vue component references, compiler macros (defineProps, etc.), - * Nuxt auto-imports, and Nuxt file-based routing patterns. + * Handles Vue component references, compiler macros (defineProps, etc.) and + * Nuxt auto-imports; `nuxtResolver` reads Nuxt's file-based routes. */ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; +import { dependsOn } from './package-deps'; /** * Vue 3 compiler macros — compiler-provided, not user code @@ -186,13 +187,34 @@ export const vueResolver: FrameworkResolver = { return null; }, +}; + +/** + * Nuxt's file-based routes: `pages/` screens, `server/api/` endpoints and + * `middleware/`. Its own resolver, detected only in a Nuxt app: a plain Vue + * app keeps its views in a `pages/` folder just as often (halo's console + * does), and those are components a router config names, not addresses. + */ +export const nuxtResolver: FrameworkResolver = { + name: 'nuxt', + appDependencies: ['nuxt', 'nuxt3', '@nuxt/kit'], + + detect(context: ResolutionContext): boolean { + if (dependsOn(context, 'nuxt', 'nuxt3', '@nuxt/kit')) return true; + return context.getAllFiles().some((f) => /(?:^|\/)nuxt\.config\.(?:[cm]?[jt]s)$/.test(f)); + }, + + resolve(): ResolvedRef | null { + return null; + }, extract(filePath: string, _content: string) { const nodes: Node[] = []; const now = Date.now(); - // Normalize to forward slashes - const normalized = filePath.replace(/\\/g, '/'); + // Forward slashes, and a leading `/` so an app at the repository root + // (`pages/index.vue`) is found by the same `/pages/` search as a nested one. + const normalized = '/' + filePath.replace(/\\/g, '/'); // Detect Nuxt page routes (pages/ directory) const pagesIndex = normalized.indexOf('/pages/'); @@ -221,8 +243,10 @@ export const vueResolver: FrameworkResolver = { const afterApi = normalized.substring(apiIndex + '/server/api/'.length); const routeName = afterApi .replace(/\.[^/.]+$/, '') // Remove extension - .replace(/\/index$/, ''); // index -> parent path - const apiRoute = '/api/' + routeName; + .replace(/(?:^|\/)index$/, '') // index -> parent path + .replace(/\[\.\.\.([^\]]+)\]/g, '*$1') // [...slug] -> *slug + .replace(/\[([^\]]+)\]/g, ':$1'); // [id] -> :id, as a page's params are + const apiRoute = routeName === '' ? '/api' : '/api/' + routeName; nodes.push({ id: `route:${filePath}:${apiRoute}:1`, @@ -317,7 +341,7 @@ function filePathToNuxtRoute(normalized: string, afterPagesStart: number): strin const withoutExt = afterPages.replace(/\.vue$/, ''); // Remove /index suffix (index.vue -> parent route) - const withoutIndex = withoutExt.replace(/\/index$/, ''); + const withoutIndex = withoutExt.replace(/(?:^|\/)index$/, ''); // Convert Nuxt param syntax [param] to :param let route = '/' + withoutIndex diff --git a/src/ui-server/api/screens.ts b/src/ui-server/api/screens.ts index 94b70761a9..8a79ff2d82 100644 --- a/src/ui-server/api/screens.ts +++ b/src/ui-server/api/screens.ts @@ -218,7 +218,7 @@ function writtenHere(edge: Edge, holder: Node): boolean { * Entry points, which is the list of what a request or a user can arrive at. */ function isScreenRoute(route: Node): boolean { - return route.name.startsWith('/') && !route.filePath.includes('/server/api/'); + return route.name.startsWith('/') && !`/${route.filePath}`.includes('/server/api/'); } export async function buildScreens(cg: CodeGraph, projectRoot: string): Promise { From b62da73550fea1e153d94116c902ffe7ee7d3af2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:50:41 +0000 Subject: [PATCH 047/363] fix(sveltekit): route names drop (group) folders and parameter matchers (#2131) A `(group)` directory shares a layout and is never part of the URL, and a parameter's `=matcher` checks its value. Kept in the name, shadcn-svelte's `src/routes/(app)/(layout)/blocks/+page.svelte` was `/(app)/(layout)/blocks`, so no `goto('/blocks')` or `` ever reached it and its Screens showed no navigation at all. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/sveltekit-route-names.test.ts | 61 +++++++++++++++++++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/frameworks/svelte.ts | 13 ++++-- 5 files changed, 72 insertions(+), 7 deletions(-) create mode 100644 __tests__/sveltekit-route-names.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index a7dd3a4bfd..7fb6d96024 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. - Calls in Vue, Svelte and Astro components now follow the same rules as JavaScript and TypeScript. A `t('…')` from `useI18n()` or a `ref(…)` from `vue` used to link to an interface property or a local variable of the same name elsewhere in the project, so on halo two unrelated symbols had over a thousand made-up callers each. A call imported from a package is also no longer mistaken for a project symbol when the package is declared in a nested app's `package.json` or a Deno import map. An import through an alias CodeGraph can't follow, like SvelteKit's `$lib/…` or a nested Nuxt app's `~/…`, is still treated as project code. Re-index Vue, Svelte, Astro and JavaScript/TypeScript projects after upgrading. diff --git a/README.md b/README.md index 3d02481dd8..a02bfc030b 100644 --- a/README.md +++ b/README.md @@ -345,7 +345,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | | **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | -| **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | +| **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`, `[id=matcher]` → `:id`, `(group)` folders stripped), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | | **Angular** | `Routes` arrays (`RouterModule.forRoot` / `forChild`, `provideRouter`, a routes file's default export) with `component` or a lazy `loadComponent`; `children` and lazy `loadChildren` (an NgModule's through its routing module) joined into full paths; paths written as route constants or `$localize` strings; a route with children is a layout around the screens inside it | `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` / `parseUrl` — a command array, a route constant, or a component property holding one — `routerLink` / `[routerLink]` in the component's template, and `redirectTo`. Each template's child components (``) are linked to the component that renders them | In a repository holding several apps, each app's routes are matched only against navigation written inside that app. diff --git a/__tests__/sveltekit-route-names.test.ts b/__tests__/sveltekit-route-names.test.ts new file mode 100644 index 0000000000..1f7ab5e465 --- /dev/null +++ b/__tests__/sveltekit-route-names.test.ts @@ -0,0 +1,61 @@ +/** + * SvelteKit route names are the URL a browser asks for. + * + * - A `(group)` directory organizes layouts and never appears in the URL: + * shadcn-svelte's `src/routes/(app)/(layout)/blocks/+page.svelte` is `/blocks`. + * - `[param=matcher]` is a parameter whose value a matcher checks: `/view/:view`. + * + * Named with the groups, no `goto('/blocks')` or `` ever + * matched its page. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +describe('SvelteKit route names', () => { + it('drop (group) directories and parameter matchers', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sveltekit-groups-')); + roots.push(root); + const files: Record = { + 'package.json': JSON.stringify({ name: 'site', devDependencies: { '@sveltejs/kit': '*', svelte: '*' } }), + 'src/routes/(app)/(layout)/blocks/+page.svelte': `

Blocks

+`, + 'src/routes/(app)/(layout)/+page.svelte': ` + +Card +`, + 'src/routes/(view)/view/[view=view]/+page.svelte': `

view

+`, + 'src/routes/docs/[...slug=doc]/+page.svelte': `

doc

+`, + 'src/params/view.ts': `export function match(p: string) { return p.length > 0; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const routes = cg.getNodesByKind('route').map((n) => n.name).sort(); + expect(routes).toEqual(['/', '/blocks', '/docs/*slug', '/view/:view']); + const home = cg.getNodesByKind('route').find((n) => n.name === '/')!; + const navigates = cg + .getOutgoingEdgesFrom(cg.getNodesInFile(home.filePath).map((n) => n.id), ['navigates' as never]) + .map((e) => cg.getNode(e.target)!.name) + .sort(); + expect(navigates).toEqual(['/blocks', '/view/:view']); + } finally { + cg.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index b149ccb7d7..e0be9a92bc 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -41,7 +41,7 @@ guessed. | React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | -| SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts` | sveltekit-realworld (31 edges) | +| SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | | Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | Shared machinery all seven use, in `frameworks/expo-router.ts`: `RouteTable` / diff --git a/src/resolution/frameworks/svelte.ts b/src/resolution/frameworks/svelte.ts index d3271a6fc8..9598f98107 100644 --- a/src/resolution/frameworks/svelte.ts +++ b/src/resolution/frameworks/svelte.ts @@ -276,11 +276,14 @@ function filePathToSvelteKitRoute(filePath: string): string | null { const lastSlash = afterRoutes.lastIndexOf('/'); const dirPath = lastSlash === -1 ? '' : afterRoutes.substring(0, lastSlash); - // Convert SvelteKit param syntax [param] to :param - let route = '/' + dirPath - .replace(/\[\.\.\.([^\]]+)\]/g, '*$1') // [...rest] -> *rest - .replace(/\[{2}([^\]]+)\]{2}/g, ':$1?') // [[optional]] -> :optional? - .replace(/\[([^\]]+)\]/g, ':$1'); // [param] -> :param + // A `(group)` directory shares a layout and never appears in the URL: + // `(app)/(layout)/blocks` is `/blocks`. A parameter's `=matcher` checks + // its value and is no part of its name. + const segments = dirPath.split('/').filter((seg) => seg.length > 0 && !/^\(.*\)$/.test(seg)); + let route = '/' + segments.join('/') + .replace(/\[\.\.\.([^\]=]+)(?:=[^\]]*)?\]/g, '*$1') // [...rest] / [...rest=m] -> *rest + .replace(/\[{2}([^\]=]+)(?:=[^\]]*)?\]{2}/g, ':$1?') // [[optional]] -> :optional? + .replace(/\[([^\]=]+)(?:=[^\]]*)?\]/g, ':$1'); // [param] / [param=m] -> :param if (route === '/') return '/'; // Remove trailing slash From 4423891e76df05f2a2dcd6257992731e6bb64768 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 06:55:41 +0000 Subject: [PATCH 048/363] fix(react-router): v5 and styled(Link) wrappers navigate (#2132) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit react-boilerplate's header links are `export default styled(Link)`…``, used as ``; takenote's guards render v5's ``. Neither tag was read, so both apps had routes and no links between their screens. The link synthesizer now reads `` (navMethod `redirect`) and any styled-components / emotion wrapper of `Link` / `NavLink` — declared in the file, or imported from a file that declares or default-exports one. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/react-router.test.ts | 98 ++++++++++++++++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/react-router-synthesizer.ts | 85 +++++++++++++++++-- 5 files changed, 177 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7fb6d96024..4c4bb38e8e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. - In a monorepo with an Expo app next to a Next.js app, like create-t3-turbo or tamagui's starter, each router now reads only its own app. Expo Router used to read the Next.js app's files too, so `layout.tsx`, `page.tsx`, a `_components` folder and an API route showed up as screens named `/layout`, `/page` or `/_components/posts`, and a shared package's files became screens as well. Re-index such monorepos after upgrading. diff --git a/README.md b/README.md index a02bfc030b..23a046f54c 100644 --- a/README.md +++ b/README.md @@ -342,7 +342,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send |---|---|---| | **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | -| **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / react-router-bootstrap's `` | +| **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / v5's `` / react-router-bootstrap's ``, and a `styled(Link)` wrapper | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | | **Vue Router** / **Nuxt** | `createRouter({ routes: [...] })` / `new Router(...)` and the route tables it's given (`export const constantRoutes = [...]`, per-module route files), with the view each entry names — a lazy `() => import(…)` bound to its file — and `children` joined onto their parent's path, the parent being the layout around them; plus Nuxt `pages/` file-based routes, `server/api/` endpoints and route middleware | `router.push` / `replace`, `this.$router.push`, Nuxt's `navigateTo`, `` / `` / `` — **by route name** (`push({ name: 'profile' })`) as well as by path | | **SvelteKit** | `src/routes/**/+page.svelte` (`[slug]` → `:slug`, `[[opt]]` → `:opt?`, `[id=matcher]` → `:id`, `(group)` folders stripped), joined to the `+page.server.js` beside it so a loader's guard belongs to its page | `goto('/x')`, `redirect(status, '/x')` from a load or form action, and the plain `` that is a link in a SvelteKit app | diff --git a/__tests__/react-router.test.ts b/__tests__/react-router.test.ts index 70ea8266ce..fd4c9d7e90 100644 --- a/__tests__/react-router.test.ts +++ b/__tests__/react-router.test.ts @@ -593,3 +593,101 @@ describe('react-router: route declaration boundaries (#1348)', () => { expect(result).toEqual({ paths: ['/child', '/shell'], bindings: ['/child->Child', '/shell->Shell'] }); }); }); + +describe('react-router: v5 redirects and styled link wrappers', () => { + let root: string; + let cg: CodeGraph; + beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rr-wrappers-')); + const files: Record = { + 'package.json': JSON.stringify({ name: 'app', dependencies: { react: '*', 'react-router-dom': '^5.0.0', 'styled-components': '*' } }), + // react-boilerplate: the header's links are a styled Link, default-exported. + 'app/components/Header/HeaderLink.js': `import { Link } from 'react-router-dom'; +import styled from 'styled-components'; + +export default styled(Link)\` + color: #41addd; +\`; +`, + 'app/components/Header/index.js': `import React from 'react'; +import HeaderLink from './HeaderLink'; + +export default function Header() { + return ( + + ); +} +`, + 'app/components/Nav.js': `import React from 'react'; +import { NavLink } from 'react-router-dom'; +import styled from 'styled-components'; + +const MenuLink = styled(NavLink)\` + padding: 4px; +\`; + +export default function Nav() { + return Features; +} +`, + // takenote: a guard renders v5's . + 'app/router/PrivateRoute.js': `import React from 'react'; +import { Route, Redirect } from 'react-router-dom'; + +export default function PrivateRoute({ component: Component, ...rest }) { + return (rest.isAuthenticated ? : )} />; +} +`, + 'app/containers/App.js': `import React from 'react'; +import { Switch, Route } from 'react-router-dom'; +import HomePage from './HomePage'; +import FeaturePage from './FeaturePage'; + +export default function App() { + return ( + + + + + ); +} +`, + 'app/containers/HomePage.js': `import React from 'react'; +export default function HomePage() { return

Home

; } +`, + 'app/containers/FeaturePage.js': `import React from 'react'; +export default function FeaturePage() { return

Features

; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); + }); + afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); + }); + + const navsFrom = (name: string) => { + const from = cg.getNodesByName(name).find((n) => n.kind === 'function' || n.kind === 'component')!; + return cg + .getOutgoingEdges(from.id) + .filter((e) => e.kind === 'navigates') + .map((e) => `${(e.metadata as Record).navMethod} ${cg.getNode(e.target)!.name}`) + .sort(); + }; + + it('a styled(Link) wrapper, imported or local, is a link', () => { + expect(navsFrom('Header')).toEqual(['link /', 'link /features']); + expect(navsFrom('Nav')).toEqual(['link /features']); + }); + + it('v5’s navigates', () => { + expect(navsFrom('PrivateRoute')).toEqual(['redirect /']); + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index e0be9a92bc..fcb12619ce 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -38,7 +38,7 @@ guessed. |---|---|---|---|---| | Expo Router | `frameworks/expo-router.ts` | `expo-router-synthesizer.ts` | `expo-router.test.ts` | — | | Next.js | `frameworks/nextjs.ts` | `next-router-synthesizer.ts` | `nextjs.test.ts` | next-saas-starter | -| React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges) | +| React Router | `frameworks/react-router.ts` | `react-router-synthesizer.ts` | `react-router.test.ts` | proshop (44 edges), proshop-v2 (28), react-redux-realworld (22), react-boilerplate (`styled(Link)`), takenote (v5 ``); bulletproof-react's `paths.x.path` constants and `lazy` routes are not read yet | | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | diff --git a/src/resolution/react-router-synthesizer.ts b/src/resolution/react-router-synthesizer.ts index a0706219e7..3787b625a7 100644 --- a/src/resolution/react-router-synthesizer.ts +++ b/src/resolution/react-router-synthesizer.ts @@ -6,6 +6,8 @@ * * … // react-router-bootstrap * … // v5's object form + * // v5 + * // HeaderLink = styled(Link) * * A JSX attribute is not a call, so the extractor records no reference for it * and the resolver in `frameworks/react-router.ts` — which binds @@ -27,7 +29,7 @@ * attribute (`to`, not `href`) and the other table. */ -import type { Edge } from '../types'; +import type { Edge, Language } from '../types'; import type { ResolutionContext } from './types'; import type { MaybeYield } from './cooperative-yield'; import { stripCommentsForRegex } from './strip-comments'; @@ -37,14 +39,75 @@ import { matchBracket } from './frameworks/object-literal'; import { destinationsForHref } from './frameworks/nextjs'; import { reactRouterTable } from './frameworks/react-router'; import { enclosingFn, makeLineAt } from './synth-utils'; +import { resolveImportPath } from './import-resolver'; const JSX_FILE = /\.(?:[cm]?[jt]sx?|mdx)$/; -/** The tags that carry a route as a `to` attribute, the attribute anywhere in the tag. */ -const LINK_TAG = /<(Link|NavLink|Navigate|LinkContainer|IndexLinkContainer)\b([^>]*?)\bto\s*=\s*(?:"([^"]*)"|'([^']*)'|(?=\{))/g; +/** The tags that carry a route as a `to` attribute. */ +const LINK_TAGS = ['Link', 'NavLink', 'Navigate', 'Redirect', 'LinkContainer', 'IndexLinkContainer']; -/** A tag this pass could possibly match — the cheap prefilter before stripping comments. */ -const HAS_LINK_TAG = /<(?:Link|NavLink|Navigate|LinkContainer|IndexLinkContainer)\b/; +/** `` for any of `tags`, the attribute anywhere in the tag. */ +function linkTagPattern(tags: readonly string[]): RegExp { + return new RegExp(`<(${tags.map((t) => t.replace(/[$]/g, '\\$&')).join('|')})\\b([^>]*?)\\bto\\s*=\\s*(?:"([^"]*)"|'([^']*)'|(?=\\{))`, 'g'); +} + +const LINK_TAG = linkTagPattern(LINK_TAGS); + +/** A file that imports React Router's own components. */ +const ROUTER_IMPORT = /\bfrom\s*['"]react-router(?:-dom)?['"]/; + +/** `const HeaderLink = styled(Link)` — a styled wrapper is the link it wraps. */ +const STYLED_LINK_CONST = /\b(?:export\s+)?(?:const|let)\s+([A-Z][\w$]*)\s*=\s*styled\s*\(\s*(?:Link|NavLink)\s*\)/g; +const STYLED_LINK_DEFAULT = /\bexport\s+default\s+styled\s*\(\s*(?:Link|NavLink)\s*\)/; + +interface LinkWrappers { + /** File → the wrapper components it declares by name. */ + named: Map>; + /** Files whose default export is a wrapper. */ + defaults: Set; +} + +/** + * Every styled-components / emotion wrapper of `Link` or `NavLink` in the + * project — react-boilerplate's header links are `export default + * styled(Link)`…``, used as `` from another file. + */ +function linkWrappers(ctx: ResolutionContext): LinkWrappers { + const named = new Map>(); + const defaults = new Set(); + for (const file of ctx.getAllFiles()) { + if (!JSX_FILE.test(file) || isTestPath(file)) continue; + if (ctx.fileContains && !ctx.fileContains(file, 'styled')) continue; + const source = ctx.readFile(file); + if (!source || !/\bstyled\s*\(\s*(?:Link|NavLink)\s*\)/.test(source) || !ROUTER_IMPORT.test(source)) continue; + if (STYLED_LINK_DEFAULT.test(source)) defaults.add(file); + STYLED_LINK_CONST.lastIndex = 0; + let m: RegExpExecArray | null; + while ((m = STYLED_LINK_CONST.exec(source)) !== null) { + let set = named.get(file); + if (!set) named.set(file, (set = new Set())); + set.add(m[1]!); + } + } + return { named, defaults }; +} + +/** The link tags `file` can write: React Router's own and the wrappers in scope there. */ +function linkTagsFor(file: string, wrappers: LinkWrappers, ctx: ResolutionContext): string[] { + const tags = [...LINK_TAGS, ...(wrappers.named.get(file) ?? [])]; + if (wrappers.named.size > 0 || wrappers.defaults.size > 0) { + for (const m of ctx.getImportMappings(file, languageOf(file))) { + const target = resolveImportPath(m.source, file, languageOf(file), ctx); + if (!target) continue; + if ((m.isDefault && wrappers.defaults.has(target)) || wrappers.named.get(target)?.has(m.exportedName)) tags.push(m.localName); + } + } + return tags; +} + +function languageOf(file: string): Language { + return /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : /\.jsx$/.test(file) ? 'jsx' : 'javascript'; +} /** Links a single component may carry before it is a navigation menu, not a decision. */ const MAX_LINKS_PER_COMPONENT = 24; @@ -55,6 +118,7 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb const edges: Edge[] = []; const seen = new Set(); const perComponent = new Map(); + const wrappers = linkWrappers(ctx); let scanned = 0; for (const file of ctx.getAllFiles()) { if (!JSX_FILE.test(file) || isTestPath(file)) continue; @@ -62,13 +126,16 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb if (!routes || routes.exact.size === 0) continue; if ((++scanned & 63) === 0) await onYield(); const source = ctx.readFile(file); - if (!source || !HAS_LINK_TAG.test(source)) continue; + if (!source || !/\bto\s*=/.test(source)) continue; + const tags = linkTagsFor(file, wrappers, ctx); + const pattern = tags.length === LINK_TAGS.length ? LINK_TAG : linkTagPattern(tags); + if (!new RegExp(`<(?:${tags.join('|')})\\b`).test(source)) continue; const safe = stripCommentsForRegex(source, 'typescript'); const nodes = ctx.getNodesInFile(file); const lineOf = makeLineAt(safe, 1); - LINK_TAG.lastIndex = 0; + pattern.lastIndex = 0; let m: RegExpExecArray | null; - while ((m = LINK_TAG.exec(safe)) !== null) { + while ((m = pattern.exec(safe)) !== null) { const tag = m[1]!; const quoted: string | null = m[3] ?? m[4] ?? null; let href: HrefLiteral | null; @@ -109,7 +176,7 @@ export async function reactRouterLinkEdges(ctx: ResolutionContext, onYield: Mayb metadata: { synthesizedBy: 'react-router-link', href: arm.display, - navMethod: tag === 'Navigate' ? 'navigate' : 'link', + navMethod: tag === 'Navigate' ? 'navigate' : tag === 'Redirect' ? 'redirect' : 'link', registeredAt: `${file}:${line}`, }, }); From cec041d241321060f17a612f034e1bcac36e0530 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:06:23 +0000 Subject: [PATCH 049/363] fix(angular): Nx workspaces, barrels, class-constant paths and root-relative navigate (#2133) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From the router sweep: - angular-spotify had 12 routes all named `/` and no navigation. Its lazy routes are `async () => (await import('@ws/home')).HomeModule` through an Nx library's `src/index.ts` barrel, which the mount pass never looked through; and every lib is its own `src/`, so each routes file got its own route table and no template's `routerLink` found the screen it names. - jira-clone had 3 routes and no navigation: its issue route is `issue/:${ProjectConst.IssueId}` (a class `static readonly`), its root redirect lives in `app.routes.ts`, which only mounts, and `navigate(['project', 'issue', id])` has no leading `/`. Now: lazy mounts follow `export … from` barrels (and an NgModule's routing imports); `(await import(x)).M` is read; route constants may be class statics (strings or objects), enum members, or live behind a barrel, and a template-literal path takes constant holes; a routes file that only mounts contributes its redirects at its mount prefix; the route table is keyed by the `angular.json` / `nx.json` workspace; and `router.navigate` / `createUrlTree` without `relativeTo` resolve a bare first command from the root, as Angular does (template `routerLink` stays relative). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/angular-router.test.ts | 140 +++++++++ docs/design/framework-coverage.md | 2 +- src/resolution/frameworks/angular-router.ts | 313 ++++++++++++++------ 4 files changed, 370 insertions(+), 86 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4c4bb38e8e..037f61f834 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. - React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. - Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. diff --git a/__tests__/angular-router.test.ts b/__tests__/angular-router.test.ts index a5309abd60..52c13ca47d 100644 --- a/__tests__/angular-router.test.ts +++ b/__tests__/angular-router.test.ts @@ -317,3 +317,143 @@ export class AdminRoutingModule {} ]); }); }); + +describe('an Angular workspace split across libraries', () => { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-angular-ws-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); + } + const routeNames = (cg: CodeGraph) => cg.getNodesByKind('route').map((n) => n.name).sort(); + const navs = (cg: CodeGraph) => + cg + .getNodesByKind('route') + .flatMap((r) => cg.getIncomingEdges(r.id).filter((e) => e.kind === 'navigates').map((e) => `${cg.getNode(e.source)!.name} -> ${r.name}`)) + .sort(); + const component = (name: string, selector: string, template: string) => `import { Component } from '@angular/core'; +@Component({ selector: '${selector}', template: \`${template}\` }) +export class ${name} {} +`; + + it('angular-spotify: Nx libraries behind barrels, `(await import(…)).M`, a class-static path constant', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'ws', dependencies: { '@angular/core': '*', '@angular/router': '*' } }), + 'nx.json': '{}', + 'tsconfig.base.json': JSON.stringify({ + compilerOptions: { + paths: { + '@ws/home': ['libs/home/src/index.ts'], + '@ws/lyrics': ['libs/lyrics/src/index.ts'], + '@ws/utils': ['libs/utils/src/index.ts'], + }, + }, + }), + 'libs/utils/src/index.ts': `export * from './lib/router-util';\n`, + 'libs/utils/src/lib/router-util.ts': `export class RouterUtil { + static Configuration = { Lyrics: 'lyrics' }; +} +`, + 'libs/shell/src/lib/shell.routes.ts': `import { Route } from '@angular/router'; +import { RouterUtil } from '@ws/utils'; + +export const shellRoutes: Route[] = [ + { path: '', loadChildren: async () => (await import('@ws/home')).HomeModule }, + { path: RouterUtil.Configuration.Lyrics, loadChildren: async () => (await import('@ws/lyrics')).LyricsModule }, +]; +`, + 'libs/home/src/index.ts': `export * from './lib/home.module';\n`, + 'libs/home/src/lib/home.module.ts': `import { NgModule } from '@angular/core'; +import { RouterModule } from '@angular/router'; +import { HomeComponent } from './home.component'; +@NgModule({ imports: [RouterModule.forChild([{ path: '', component: HomeComponent }])] }) +export class HomeModule {} +`, + 'libs/home/src/lib/home.component.ts': component('HomeComponent', 'as-home', 'Lyrics'), + 'libs/lyrics/src/index.ts': `export * from './lib/lyrics.module';\n`, + 'libs/lyrics/src/lib/lyrics.module.ts': `import { NgModule } from '@angular/core'; +import { RouterModule } from '@angular/router'; +import { LyricsComponent } from './lyrics.component'; +@NgModule({ imports: [RouterModule.forChild([{ path: '', component: LyricsComponent }])] }) +export class LyricsModule {} +`, + 'libs/lyrics/src/lib/lyrics.component.ts': component('LyricsComponent', 'as-lyrics', 'Home'), + }); + try { + expect(routeNames(cg)).toEqual(['/', '/lyrics']); + // A template in one library links to a screen another library declares. + expect(navs(cg)).toEqual(['HomeComponent -> /lyrics', 'LyricsComponent -> /']); + } finally { + cg.close(); + } + }); + + it('jira-clone: a template-literal path, a redirect in a file that only mounts, navigate() from the root', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'jira', dependencies: { '@angular/core': '*', '@angular/router': '*' } }), + 'angular.json': '{}', + 'src/app/app.routes.ts': `import { Routes } from '@angular/router'; +export const appRoutes: Routes = [ + { path: 'project', loadChildren: () => import('./project/project.routes').then((m) => m.PROJECT_ROUTES) }, + { path: '', redirectTo: 'project', pathMatch: 'full' }, +]; +`, + 'src/app/project/config/const.ts': `export class ProjectConst { + static readonly IssueId = 'issueId'; +} +`, + 'src/app/project/project.routes.ts': `import { Routes } from '@angular/router'; +import { ProjectComponent } from './project.component'; +import { BoardComponent } from './board.component'; +import { IssueComponent } from './issue.component'; +import { ProjectConst } from './config/const'; +export const PROJECT_ROUTES: Routes = [ + { + path: '', + component: ProjectComponent, + children: [ + { path: 'board', component: BoardComponent }, + { path: \`issue/:\${ProjectConst.IssueId}\`, component: IssueComponent }, + { path: '', redirectTo: 'board', pathMatch: 'full' }, + ], + }, +]; +`, + 'src/app/project/project.component.ts': component('ProjectComponent', 'j-project', ''), + 'src/app/project/board.component.ts': `import { Component } from '@angular/core'; +import { Router } from '@angular/router'; +@Component({ selector: 'j-board', template: '
' }) +export class BoardComponent { + constructor(private _router: Router) {} + openIssuePage(issueId: string) { + this._router.navigate(['project', 'issue', issueId]); + } +} +`, + 'src/app/project/issue.component.ts': `import { Component } from '@angular/core'; +import { Router } from '@angular/router'; +@Component({ selector: 'j-issue', template: '
' }) +export class IssueComponent { + constructor(private _router: Router) {} + backHome() { + this._router.navigate(['/']); + } +} +`, + }); + try { + expect(routeNames(cg)).toEqual(['/project/board', '/project/issue/:issueId']); + expect(navs(cg)).toEqual(['backHome -> /project/board', 'openIssuePage -> /project/issue/:issueId']); + } finally { + cg.close(); + } + }); +}); diff --git a/docs/design/framework-coverage.md b/docs/design/framework-coverage.md index fcb12619ce..6e123c2f51 100644 --- a/docs/design/framework-coverage.md +++ b/docs/design/framework-coverage.md @@ -42,7 +42,7 @@ guessed. | TanStack Router | `frameworks/tanstack-router.ts` | `tanstack-router-synthesizer.ts` | `tanstack-router.test.ts` | TanStack examples, fastapi-template frontend | | Vue Router / Nuxt | `frameworks/vue-router.ts` (Nuxt file routes: `nuxtResolver` in `frameworks/vue.ts`) | `vue-router-synthesizer.ts` | `vue-router.test.ts` | vue-realworld (23 edges); vue-element-admin (62 routes), vue-admin-template (14), vben (192), halo console (34) — named tables, module files, `children` + layouts; Nuxt: mealie, elk, nuxt/movies | | SvelteKit | `frameworks/sveltekit-router.ts` | `sveltekit-synthesizer.ts` | `sveltekit-router.test.ts`, `sveltekit-route-names.test.ts` | sveltekit-realworld (31 edges); shadcn-svelte and skeleton (`(group)` layouts: 13 and 23 edges), svelte.dev (74), kit's test apps (47) | -| Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config) | +| Angular | `frameworks/angular-router.ts` | `angular-template-synthesizer.ts` | `angular-router.test.ts` | angular-realworld (31 edges, 18 renders), Ghostfolio (189 edges, 170 renders), ngx-admin (routes and renders; its menus are config), angular-spotify (Nx libs behind barrels: 14 routes), jira-clone (class-constant paths, mount-only redirects), jhipster (60), ionic-conference (18), Angular-JumpStart (18) | Shared machinery all seven use, in `frameworks/expo-router.ts`: `RouteTable` / `RootedRouteTable`, `routesForFile`, `addRouteTo`, `matchRoute`, `appRootFor`, diff --git a/src/resolution/frameworks/angular-router.ts b/src/resolution/frameworks/angular-router.ts index 02f7dc9d81..0980c4a5c5 100644 --- a/src/resolution/frameworks/angular-router.ts +++ b/src/resolution/frameworks/angular-router.ts @@ -172,6 +172,16 @@ function pathSegments(text: string): string[] | null { return value.split('/').filter((s) => s.length > 0); } if (CONSTANT_PATH.test(trimmed)) return [`{${trimmed.replace(/\s+/g, '')}}`]; + // `` `issue/:${ProjectConst.IssueId}` ``: a template whose holes are constants. + if (trimmed.startsWith('`') && skipString(trimmed, 0) === trimmed.length - 1) { + let ok = true; + const body = trimmed.slice(1, -1).replace(/\$\{([^}]*)\}/g, (_all, hole: string) => { + if (!CONSTANT_PATH.test(hole.trim())) ok = false; + return `{${hole.replace(/\s+/g, '')}}`; + }); + if (ok && !body.includes('**')) return body.split('/').filter((seg) => seg.length > 0); + return null; + } // `internalRoutes.account.subRoutes.access.path + '/:id'`: each part a // string or a constant, the constant a whole segment of its own. if (trimmed.includes('+')) { @@ -225,8 +235,11 @@ function componentRef(field: 'component' | 'loadComponent', text: string): Angul function lazyImport(text: string): { spec: string; member: string | null } | null { const imp = /\bimport\s*\(\s*(['"`])([^'"`]+)\1\s*\)/.exec(text); if (!imp) return null; - const then = /\.then\s*\(\s*\(?\s*([A-Za-z_$][\w$]*)\s*\)?\s*=>\s*\(?\s*\1\s*\.\s*([A-Za-z_$][\w$]*)/.exec(text.slice(imp.index + imp[0].length)); - return { spec: imp[2]!, member: then ? then[2]! : null }; + const after = text.slice(imp.index + imp[0].length); + const then = /^\s*\.then\s*\(\s*\(?\s*([A-Za-z_$][\w$]*)\s*\)?\s*=>\s*\(?\s*\1\s*\.\s*([A-Za-z_$][\w$]*)/.exec(after); + // `async () => (await import('./x')).HomeModule` + const awaited = /^\s*\)\s*\.\s*([A-Za-z_$][\w$]*)/.exec(after); + return { spec: imp[2]!, member: then ? then[2]! : awaited ? awaited[1]! : null }; } function joinPath(segs: readonly string[]): string { @@ -395,9 +408,12 @@ function resolvedSegments(pathText: string, file: string, context: ResolutionCon .split('/') .filter((seg) => seg.length > 0) .flatMap((seg) => { - if (!seg.startsWith('{') || !seg.endsWith('}')) return [seg]; - const value = constantValue(seg.slice(1, -1), file, context); - return value === null ? [seg] : value.split('/').filter((v) => v.length > 0); + if (seg.startsWith('{') && seg.endsWith('}') && seg.indexOf('{', 1) < 0) { + const value = constantValue(seg.slice(1, -1), file, context); + return value === null ? [seg] : value.split('/').filter((v) => v.length > 0); + } + // `:{ProjectConst.IssueId}` — a constant inside a segment. + return [seg.replace(/\{([^{}]+)\}/g, (all, expr: string) => constantValue(expr, file, context) ?? all)]; }); } @@ -413,21 +429,48 @@ function routeComponent(encoded: string, fromFile: string, context: ResolutionCo // The cross-file pass: mounts and constant paths // ============================================================================= -/** The file a routes import names, and — for an NgModule — the routing modules it imports. */ +/** + * The file a routes import names, and — for an NgModule — the routing modules + * it imports. A barrel is looked through first: an Nx library is imported by + * its alias (`@angular-spotify/web/home/feature`), which names the library's + * `src/index.ts`, whose `export * from './lib/home.module'` names the module. + */ function routeFilesLoadedBy(spec: string, fromFile: string, context: ResolutionContext, routeFiles: ReadonlySet, mountFiles: ReadonlySet): string[] { const target = resolveImportPath(spec, fromFile, 'typescript', context); - if (!target) return []; - if (routeFiles.has(target) || mountFiles.has(target)) return [target]; - // `loadChildren: () => import('./layout/layout.module').then(m => m.LayoutModule)`: - // the module holds no routes; the routing module it imports does. - const content = context.readFile(target); - if (!content) return []; + const seen = new Set(); const out: string[] = []; - const imports = /\bimport\s+[^'"]*?from\s+(['"])([^'"]+)\1/g; - let m: RegExpExecArray | null; - while ((m = imports.exec(content)) !== null) { - const file = resolveImportPath(m[2]!, target, 'typescript', context); - if (file && (routeFiles.has(file) || mountFiles.has(file))) out.push(file); + // A barrel can re-export another barrel; a few hops settle it. + let barrels = target ? [target] : []; + for (let hop = 0; hop < 4 && barrels.length > 0; hop++) { + const next: string[] = []; + for (const file of barrels) { + if (seen.has(file)) continue; + seen.add(file); + if (routeFiles.has(file) || mountFiles.has(file)) { + out.push(file); + continue; + } + const content = context.readFile(file); + if (!content) continue; + // `loadChildren: () => import('./layout/layout.module').then(m => m.LayoutModule)`: + // the module holds no routes; the routing module it imports does. + if (/@NgModule\s*\(/.test(content)) { + const imports = /\bimport\s+[^'"]*?from\s+(['"])([^'"]+)\1/g; + let m: RegExpExecArray | null; + while ((m = imports.exec(content)) !== null) { + const imported = resolveImportPath(m[2]!, file, 'typescript', context); + if (imported && (routeFiles.has(imported) || mountFiles.has(imported)) && !out.includes(imported)) out.push(imported); + } + continue; + } + const reexports = /\bexport\s+(?:\*|\{[^}]*\})\s*(?:as\s+[\w$]+\s*)?from\s+(['"])([^'"]+)\1/g; + let m: RegExpExecArray | null; + while ((m = reexports.exec(content)) !== null) { + const reexported = resolveImportPath(m[2]!, file, 'typescript', context); + if (reexported && !seen.has(reexported)) next.push(reexported); + } + } + barrels = next; } return out; } @@ -458,29 +501,68 @@ export function constantText(expr: string, fromFile: string, context: Resolution const [root, ...chain] = expr.split('.').map((part) => part.trim()); let value: string | null = null; const file = root ? declaringFile(root, fromFile, context) : null; - const content = file ? context.readFile(file) : null; - if (content && chain.length > 0) { + const content = file && root ? declarationSource(root, file, context, 0) : null; + if (content && root && chain.length > 0) { const safe = stripCommentsForRegex(content, 'typescript'); - const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); - if (decl) { - let start = decl.index + decl[0].length - 1; + const readChain = (open: number, links: readonly string[]): string | null => { + let start = open; let end = matchBracket(safe, start); - for (let i = 0; i < chain.length && end > start; i++) { - const field = readFields(safe, start, end).get(chain[i]!); - if (!field) break; - if (i === chain.length - 1) { - value = field.text.trim(); - break; - } + for (let i = 0; i < links.length && end > start; i++) { + const field = readFields(safe, start, end).get(links[i]!); + if (!field) return null; + if (i === links.length - 1) return field.text.trim(); start = safe.indexOf('{', field.at); end = start < 0 ? -1 : matchBracket(safe, start); } + return null; + }; + const decl = new RegExp(String.raw`\b(?:const|let)\s+${root}\s*(?::[^=]+)?=\s*\{`).exec(safe); + const enumDecl = decl ? null : new RegExp(String.raw`\benum\s+${root}\s*\{`).exec(safe); + const classDecl = decl || enumDecl ? null : new RegExp(String.raw`\bclass\s+${root}\b[^{]*\{`).exec(safe); + if (decl) value = readChain(decl.index + decl[0].length - 1, chain); + else if (enumDecl && chain.length === 1) { + // `enum Paths { Board = 'board' }` + const open = enumDecl.index + enumDecl[0].length - 1; + const body = safe.slice(open, Math.max(open, matchBracket(safe, open))); + value = new RegExp(String.raw`\b${chain[0]}\s*=\s*(['"\`][^'"\`]*['"\`])`).exec(body)?.[1] ?? null; + } else if (classDecl) { + // `class ProjectConst { static readonly IssueId = 'issueId' }`, and + // `class RouterUtil { static Configuration = { Visualizer: 'visualizer' } }`. + const open = classDecl.index + classDecl[0].length - 1; + const close = matchBracket(safe, open); + const body = close > open ? safe.slice(open, close) : ''; + const field = new RegExp(String.raw`\bstatic\s+(?:readonly\s+)?${chain[0]}\s*(?::[^=;]+)?=\s*`).exec(body); + if (field) { + const at = open + field.index + field[0].length; + if (safe[at] === '{') value = chain.length > 1 ? readChain(at, chain.slice(1)) : null; + else if (chain.length === 1) value = /^(['"\`])[^'"\`]*\1/.exec(safe.slice(at))?.[0] ?? null; + } } } memo.set(key, value); return value; } +/** + * The source that declares `root`: `file` itself, or — when `file` is a + * barrel (an Nx library's `src/index.ts`) — the module it re-exports it from. + */ +function declarationSource(root: string, file: string, context: ResolutionContext, hops: number): string | null { + const content = context.readFile(file); + if (!content) return null; + if (new RegExp(String.raw`\b(?:const|let|class|enum)\s+${root}\b`).test(content)) return content; + if (hops >= 3) return null; + const reexports = /\bexport\s+(\*|\{[^}]*\})\s*from\s+(['"])([^'"]+)\2/g; + let m: RegExpExecArray | null; + while ((m = reexports.exec(content)) !== null) { + if (m[1] !== '*' && !new RegExp(String.raw`\b${root}\b`).test(m[1]!)) continue; + const target = resolveImportPath(m[3]!, file, 'typescript', context); + const found = target ? declarationSource(root, target, context, hops + 1) : null; + if (found) return found; + } + return null; +} + /** `create.path` as the chain its root was destructured from in `fromFile`, or null when the root is no local alias. */ function localAlias(expr: string, fromFile: string, context: ResolutionContext): string | null { const content = context.readFile(fromFile); @@ -522,6 +604,93 @@ function declaringFile(name: string, fromFile: string, context: ResolutionContex // Route table // ============================================================================= +interface AngularMounts { + /** Every file that lazy-loads routes, whether or not it declares a screen of its own. */ + mountFiles: ReadonlySet; + /** The path segments a file's routes sit under: its parent's prefix plus the mount's. */ + prefixOf(file: string): string[]; +} + +const mountMemo = new WeakMap(); + +/** + * Where every routes file is mounted — the `loadChildren` chain from the app + * down, settled from the top. Shared by the cross-file pass, which names the + * routes, and the route table, which reads the redirects in files that only + * mount others (jira-clone's `app.routes.ts` holds nothing but two mounts and + * `'' → 'project'`). + */ +function angularMounts(context: ResolutionContext, routes: readonly Node[]): AngularMounts { + const source = context.getNodesByKind('route'); + const cached = mountMemo.get(context); + if (cached && cached.source === source) return cached.mounts; + const routeFiles = new Set(routes.map((r) => r.filePath)); + const mountsByFile = new Map(); + for (const file of context.getAllFiles()) { + if (!/\.[cm]?ts$/.test(file) || !(context.fileContains?.(file, 'loadChildren') ?? context.readFile(file)?.includes('loadChildren'))) continue; + const content = context.readFile(file); + if (!content) continue; + const { mounts } = parseAngularRoutes(content); + if (mounts.length > 0) mountsByFile.set(file, mounts); + } + const mountFiles = new Set(mountsByFile.keys()); + const loadedBy = new Map(); + for (const [file, mounts] of mountsByFile) { + for (const mount of mounts) { + for (const target of routeFilesLoadedBy(mount.spec, file, context, routeFiles, mountFiles)) { + // A file mounted from two places keeps its first mount. + if (target !== file && !loadedBy.has(target)) loadedBy.set(target, { parent: file, prefix: mount.prefix }); + } + } + } + const memo = new Map(); + const prefixOf = (file: string, seen: Set = new Set()): string[] => { + const hit = memo.get(file); + if (hit !== undefined) return hit; + const mount = loadedBy.get(file); + let prefix: string[] = []; + if (mount && !seen.has(file)) { + seen.add(file); + prefix = [...prefixOf(mount.parent, seen), ...resolvedSegments(mount.prefix, mount.parent, context)]; + } + memo.set(file, prefix); + return prefix; + }; + const mounts: AngularMounts = { mountFiles, prefixOf: (file) => prefixOf(file) }; + mountMemo.set(context, { source, mounts }); + return mounts; +} + +const workspaceRoots = new WeakMap>(); + +/** + * The app a routes file belongs to. An Angular workspace — the directory an + * `angular.json` or an Nx `nx.json` sits in — is one app however its routes + * are split: angular-spotify declares its screens across a dozen `libs/*` + * packages, each with a `src/` of its own, and keyed by those every + * template's `routerLink` looked for its routes in the wrong table. Outside a + * workspace, the conventional app root. + */ +function angularAppRoot(filePath: string, context: ResolutionContext): string { + let memo = workspaceRoots.get(context); + if (!memo) workspaceRoots.set(context, (memo = new Map())); + const slash = filePath.lastIndexOf('/'); + const dir = slash < 0 ? '' : filePath.slice(0, slash); + let found = memo.get(dir); + if (found === undefined) { + found = null; + for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { + const at = d === '' ? '' : `${d}/`; + if (context.fileExists(`${at}angular.json`) || context.fileExists(`${at}nx.json`)) { + found = at; + break; + } + } + memo.set(dir, found); + } + return found ?? appRootFor(filePath); +} + export type AngularRouteTable = RootedRouteTable; const tables = new WeakMap(); @@ -534,7 +703,7 @@ export function angularRouteTable(context: ResolutionContext): AngularRouteTable const byFile = new Map(); for (const node of all) { if (!isAngularRoute(node) || !node.name.startsWith('/')) continue; - const root = appRootFor(node.filePath); + const root = angularAppRoot(node.filePath, context); let t = byRoot.get(root); if (!t) byRoot.set(root, (t = { source: all, exact: new Map(), dynamic: [] })); addRouteTo(t, node.name, node); @@ -545,13 +714,21 @@ export function angularRouteTable(context: ResolutionContext): AngularRouteTable // in-file paths. A chain (`/` → `/pages` → `/pages/dashboard`) settles in // a few passes. const aliases: Array<{ table: RouteTable; from: string; to: string }> = []; + const mounts = byFile.size > 0 ? angularMounts(context, all.filter(isAngularRoute)) : null; + const redirectFiles = new Map(); for (const [file, sample] of byFile) { - const content = context.readFile(file); - if (!content || !content.includes('redirectTo')) continue; const own = resolvedSegments(inFilePath(sample), file, context); const full = sample.name.split('/').filter((seg) => seg.length > 0); - const prefix = full.slice(0, Math.max(0, full.length - own.length)); - const table = byRoot.get(appRootFor(file)); + redirectFiles.set(file, full.slice(0, Math.max(0, full.length - own.length))); + } + // A file that only mounts others can still redirect: its prefix is where it is mounted. + for (const file of mounts?.mountFiles ?? []) { + if (!redirectFiles.has(file)) redirectFiles.set(file, mounts!.prefixOf(file)); + } + for (const [file, prefix] of redirectFiles) { + const content = context.readFile(file); + if (!content || !content.includes('redirectTo')) continue; + const table = byRoot.get(angularAppRoot(file, context)) ?? (byRoot.size === 1 ? [...byRoot.values()][0]! : undefined); if (!table) continue; for (const r of parseAngularRoutes(content).redirects) { const from = joinPath([...prefix, ...resolvedSegments(r.from, file, context)]); @@ -601,7 +778,7 @@ export function angularRoutesFor(table: AngularRouteTable, filePath: string): Ro const NAV_CALL = /^(?:this\.)?_?[rR]outer\.(navigate|navigateByUrl|createUrlTree|parseUrl)$/; /** The destination a command array names: `['/article', slug]` is `/article/${…}`; null when it is relative or not a literal array. */ -export function commandsHref(text: string): HrefLiteral | null { +export function commandsHref(text: string, fromRoot = false): HrefLiteral | null { const arr = text.trim(); if (arr[0] !== '[') return null; const close = matchBracket(arr, 0); @@ -644,11 +821,13 @@ export function commandsHref(text: string): HrefLiteral | null { i++; } flush(); - // Only an absolute destination: `['/login']`. `['../', id]` and a bare - // `['edit']` are relative to wherever the component is. + // Only an absolute destination: `['/login']`. In a template, `['../', id]` + // and a bare `['edit']` are relative to wherever the component is; a + // `router.navigate(['project', 'issue', id])` with no `relativeTo` starts + // from the root. const firstElement = arr.slice(1, close).split(',')[0] ?? ''; const first = staticString(firstElement); - if (first === null || !first.startsWith('/')) return null; + if (first === null || (!first.startsWith('/') && !(fromRoot && !first.startsWith('.')))) return null; const pathText = '/' + parts.filter((p) => p.length > 0).join('/'); return namesSomewhere({ path: pathText, display: pathText.split(HOLE).join('${…}') }); } @@ -725,14 +904,14 @@ function arrowReturn(fn: string): string | null { * * `owner` is the class the expression is written in, for `this.` properties. */ -export function angularDestination(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth = 0): HrefLiteral | null { - return namesSomewhere(destinationOf(expr, file, owner, context, depth)); +export function angularDestination(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth = 0, fromRoot = false): HrefLiteral | null { + return namesSomewhere(destinationOf(expr, file, owner, context, depth, fromRoot)); } -function destinationOf(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth: number): HrefLiteral | null { +function destinationOf(expr: string, file: string, owner: Node | null, context: ResolutionContext, depth: number, fromRoot: boolean): HrefLiteral | null { const text = expr.trim(); if (text.length === 0 || depth > 4) return null; - if (text[0] === '[') return commandsHref(text); + if (text[0] === '[') return commandsHref(text, fromRoot); const literal = staticString(text); if (literal !== null) return toHref(literal); // `'/editor/' + article.slug`: the literal head, a hole for the rest. @@ -743,7 +922,7 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: // `this.routerLinkAdminControlUsers.concat(userId)`: one segment more. const concat = /^(.*?)\.concat\s*\((.*)\)$/s.exec(text); if (concat) { - const base = angularDestination(concat[1]!, file, owner, context, depth + 1); + const base = angularDestination(concat[1]!, file, owner, context, depth + 1, fromRoot); const added = concat[2]!.split(',').filter((a) => a.trim().length > 0).length; return base && added > 0 ? toHref(base.path.replace(/\/$/, '') + ('/' + HOLE).repeat(added)) : base; } @@ -753,7 +932,7 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: if (call) { const fn = constantText(call[1]!.replace(/\s+/g, '').replace(/^this\./, ''), file, context); const returned = fn ? arrowReturn(fn) : null; - return returned ? commandsHref(returned) : null; + return returned ? commandsHref(returned, fromRoot) : null; } const chain = /^(?:this\s*\.\s*)?([A-Za-z_$][\w$]*(?:\s*\.\s*[A-Za-z_$][\w$]*)*)$/.exec(text); if (!chain) return parseHrefExpression(text); @@ -762,12 +941,12 @@ function destinationOf(expr: string, file: string, owner: Node | null, context: if (content && owner && (text.startsWith('this') || parts.length === 1)) { const init = propertyInitializer(owner, parts[0]!, content); if (init !== null) { - if (parts.length === 1) return angularDestination(init, file, owner, context, depth + 1); - return angularDestination(`${init}.${parts.slice(1).join('.')}`, file, owner, context, depth + 1); + if (parts.length === 1) return angularDestination(init, file, owner, context, depth + 1, fromRoot); + return angularDestination(`${init}.${parts.slice(1).join('.')}`, file, owner, context, depth + 1, fromRoot); } } const constant = constantText(parts.join('.'), file, context); - return constant ? angularDestination(constant, file, owner, context, depth + 1) : null; + return constant ? angularDestination(constant, file, owner, context, depth + 1, fromRoot) : null; } // ============================================================================= @@ -837,44 +1016,8 @@ export const angularRouterResolver: FrameworkResolver = { postExtract(context: ResolutionContext): Node[] { const routes = context.getNodesByKind('route').filter(isAngularRoute); if (routes.length === 0) return []; - const routeFiles = new Set(routes.map((r) => r.filePath)); - // Every file that lazy-loads routes, whether or not it declares a screen of its own. - const mountsByFile = new Map(); - for (const file of context.getAllFiles()) { - if (!/\.[cm]?ts$/.test(file) || !(context.fileContains?.(file, 'loadChildren') ?? context.readFile(file)?.includes('loadChildren'))) continue; - const content = context.readFile(file); - if (!content) continue; - const { mounts } = parseAngularRoutes(content); - if (mounts.length > 0) mountsByFile.set(file, mounts); - } - const mountFiles = new Set(mountsByFile.keys()); - // file → the prefix it is mounted under (the parent file's own prefix - // plus the mount's), settled from the top down. - const loadedBy = new Map(); - for (const [file, mounts] of mountsByFile) { - for (const mount of mounts) { - for (const target of routeFilesLoadedBy(mount.spec, file, context, routeFiles, mountFiles)) { - // A file mounted from two places keeps its first mount. - if (target !== file && !loadedBy.has(target)) loadedBy.set(target, { parent: file, prefix: mount.prefix }); - } - } - } + const { prefixOf } = angularMounts(context, routes); const resolved = (pathText: string, file: string): string[] => resolvedSegments(pathText, file, context); - - const memo = new Map(); - const prefixOf = (file: string, seen: Set = new Set()): string[] => { - const hit = memo.get(file); - if (hit !== undefined) return hit; - const mount = loadedBy.get(file); - let prefix: string[] = []; - if (mount && !seen.has(file)) { - seen.add(file); - prefix = [...prefixOf(mount.parent, seen), ...resolved(mount.prefix, mount.parent)]; - } - memo.set(file, prefix); - return prefix; - }; - const changed: Node[] = []; for (const route of routes) { const name = joinPath([...prefixOf(route.filePath), ...resolved(inFilePath(route), route.filePath)]); @@ -915,7 +1058,7 @@ export const angularRouterResolver: FrameworkResolver = { .getNodesInFile(ref.filePath) .filter((n) => n.kind === 'class' && n.startLine <= ref.line && n.endLine >= ref.line) .reduce((inner, n) => (!inner || n.startLine >= inner.startLine ? n : inner), null); - const href = angularDestination(arg, ref.filePath, owner, context); + const href = angularDestination(arg, ref.filePath, owner, context, 0, true); if (!href || !href.path.startsWith('/')) return null; const targets = destinationsForHref(href, routes); const target = targets[0]; From 67744468d2bdf473a70559e61e4cd20dcff9fd5f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:28:28 +0000 Subject: [PATCH 050/363] fix(resolution): Expo module bindings, imported receivers, TS type-only imports (#2134) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Expo Modules: a JS call on a `requireNativeModule('N')` binding — typed (``) or not, local or imported (expo-camera's `import CameraManager from './ExpoCameraManager'`) — resolves to module N's declared function, Swift first and Kotlin beside it, else to the member on the binding's declared type. It went to the same-named static method making the call. - A JS/TS member call on an import binding never picks a method declared in the calling file, and ruling those out never manufactures a unique guess. - `import type { X }` was parsed as a default import named `type` (every type-only import bound `type`), and `{ a, type B }` bound `type B`. - A name the file binds from an out-of-repo package names nothing in the project, for every reference kind, and JS/TS fuzzy matching is exact-case: trpc's 402 `Record<…>` references went to a `record` property, element-plus's 121 `mount` imports from @vue/test-utils to a test helper. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 3 + README.md | 2 +- __tests__/expo-modules.test.ts | 77 ++++++++++++++++ __tests__/import-type-modifier.test.ts | 75 ++++++++++++++++ src/resolution/frameworks/expo-modules.ts | 105 ++++++++++++++++++++++ src/resolution/frameworks/index.ts | 6 +- src/resolution/import-resolver.ts | 10 ++- src/resolution/name-matcher.ts | 34 ++++++- 8 files changed, 303 insertions(+), 9 deletions(-) create mode 100644 __tests__/import-type-modifier.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 037f61f834..c0c959e44b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. +- A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. +- TypeScript's `import type { X }` no longer reads as an import of something named `type`, and `import { a, type B }` binds `B`, not `type B`. So a type-only import now links to the module it names, where it used to link to whatever project symbol had that name. A type imported from a package, like `RsbuildConfig` from `@rsbuild/core`, no longer links to a project symbol, and JavaScript and TypeScript names no longer match a symbol whose name differs only in case (`Mock` is not `mock`). - Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. - React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. - SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. diff --git a/README.md b/README.md index 23a046f54c..284027063a 100644 --- a/README.md +++ b/README.md @@ -363,7 +363,7 @@ Real iOS and React Native codebases live across multiple languages — a Swift c | **React Native legacy bridge** | JS `NativeModules.X.fn(...)` | ObjC `RCT_EXPORT_METHOD` / `RCT_REMAP_METHOD` · Java/Kotlin `@ReactMethod` | Parses macro/annotation declarations to build a JS-name → native-method map | | **React Native TurboModules** | JS `import M from './NativeM'; M.fn(...)` | Native impl matching the Codegen spec | Treats the `Native.ts` spec interface as ground truth | | **RN native → JS events** | JS `new NativeEventEmitter(...).addListener('e', cb)` | ObjC `[self sendEventWithName:@"e" body:...]` · Swift `sendEvent(withName: "e", ...)` · Java/Kotlin `.emit("e", ...)` | Synthesized cross-language event channel keyed by literal event name | -| **Expo Modules** | JS `requireNativeModule('X').fn(...)` | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals; synthetic method nodes resolve via existing name-match | +| **Expo Modules** | JS `requireNativeModule('X').fn(...)`, directly or through a binding (`export default requireNativeModule('X')`) | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals into method nodes; a call on a binding resolves to module `X`'s `fn` on both platforms, else to the method on the binding's declared type | | **Fabric view components** | JSX `` | TS Codegen spec + native impl class | Spec → `component` node; convention-based name+suffix lookup (`View`/`ComponentView`/`Manager`/`ViewManager`) bridges to native | | **Legacy Paper view managers** | JSX `` | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — Paper-era declarations also produce `component` + `property` nodes | diff --git a/__tests__/expo-modules.test.ts b/__tests__/expo-modules.test.ts index eab16c1c33..4df758a2ea 100644 --- a/__tests__/expo-modules.test.ts +++ b/__tests__/expo-modules.test.ts @@ -205,3 +205,80 @@ class BatteryModule : Module() { expect(pair.c).toBeGreaterThanOrEqual(2); // swift->kotlin AND kotlin->swift }); }); + +describe('Expo Modules — a JS binding named for its role, not its module', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'expo-modules-binding-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('expo-camera: `CameraManager.fn()` on `export default requireNativeModule("ExpoCamera")` reaches the module’s functions', async () => { + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + write('package.json', '{"dependencies":{"expo-modules-core":"^1.0.0","expo":"*"}}'); + write('ios/CameraViewModule.swift', `import ExpoModulesCore +public final class CameraViewModule: Module { + public func definition() -> ModuleDefinition { + Name("ExpoCamera") + AsyncFunction("launchScanner") { (options: ScannerOptions) in } + AsyncFunction("dismissScanner") { } + } +} +`); + write('android/src/main/java/expo/modules/camera/CameraViewModule.kt', `package expo.modules.camera +class CameraViewModule : Module() { + override fun definition() = ModuleDefinition { + Name("ExpoCamera") + AsyncFunction("launchScanner") { options: ScannerOptions -> } + AsyncFunction("dismissScanner") { } + } +} +`); + write('src/Camera.types.ts', `import { NativeModule } from 'expo'; +export declare class CameraNativeModule extends NativeModule { + readonly isAvailableAsync: () => Promise; +} +`); + write('src/ExpoCameraManager.ts', `import { requireNativeModule } from 'expo'; +import type { CameraNativeModule } from './Camera.types'; + +export default requireNativeModule('ExpoCamera'); +`); + write('src/CameraView.tsx', `import CameraManager from './ExpoCameraManager'; + +export default class CameraView { + static async isAvailableAsync(): Promise { + return CameraManager.isAvailableAsync(); + } + static async launchScanner(options: object): Promise { + await CameraManager.launchScanner(options); + } +} +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const view = cg.getNodesInFile('src/CameraView.tsx'); + const from = (name: string) => view.find((n) => n.name === name && n.kind === 'method')!; + const callees = (name: string) => + cg + .getOutgoingEdges(from(name).id) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.language}:${n.qualifiedName.split('::').pop()}`) + .sort(); + // Both platforms' `launchScanner`, not CameraView's own. + expect(callees('launchScanner')).toEqual(['kotlin:ExpoCamera.launchScanner', 'swift:ExpoCamera.launchScanner']); + // Declared natively nowhere here: the type the binding is given, never the caller itself. + expect(callees('isAvailableAsync')).toEqual(['typescript:isAvailableAsync']); + const target = cg.getOutgoingEdges(from('isAvailableAsync').id).find((e) => e.kind === 'calls')!; + expect(cg.getNode(target.target)!.filePath).toBe('src/Camera.types.ts'); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/import-type-modifier.test.ts b/__tests__/import-type-modifier.test.ts new file mode 100644 index 0000000000..a74b430703 --- /dev/null +++ b/__tests__/import-type-modifier.test.ts @@ -0,0 +1,75 @@ +/** + * TypeScript's `type` modifiers are not bindings. + * + * `import type { X } from './x'` used to read as a default import named + * `type` beside `X` — every type-only import line bound `type` — so zod's + * `type.innerType()` (a parameter named `type`) was taken for a call on an + * import. An inline `{ util, type objectUtil }` named its binding + * `type objectUtil`. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { extractImportMappings } from '../src/resolution/import-resolver'; + +const bindings = (source: string) => + extractImportMappings('src/a.ts', source, 'typescript').map((m) => `${m.localName}<${m.exportedName}${m.isNamespace ? ' ns' : ''}`); + +describe('import mappings and TypeScript type modifiers', () => { + it('a type-only import binds its names, never `type`', () => { + expect(bindings(`import type { enumUtil } from './helpers/enumUtil.js';\n`)).toEqual(['enumUtil { + expect(bindings(`import { util, type objectUtil } from './helpers/util.js';\n`)).toEqual(['util { + expect(bindings(`import type from './type';\n`)).toEqual(['type { + const roots: string[] = []; + afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); + }); + + it('halo: `type RsbuildConfig` and `type Command` are not a local `rsbuildConfig` or `command`', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-import-type-')); + roots.push(root); + const files: Record = { + 'package.json': JSON.stringify({ name: 'console', dependencies: { '@rsbuild/core': '*', '@tiptap/core': '*' } }), + 'src/rsbuild.ts': `import { defineConfig, type RsbuildConfig } from '@rsbuild/core'; +export function rsbuildConfig(): RsbuildConfig { return defineConfig({}); } +`, + 'src/menu.ts': `export function command() { return 1; } +`, + 'src/gap.ts': `import type { Command } from '@tiptap/core'; +export function gapCursor(): Command { return () => true; } +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const targets = (file: string) => + cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id), ['references', 'imports', 'type_of', 'returns']) + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.kind !== 'file' && n.kind !== 'import') + .map((n) => n.name); + expect(targets('src/rsbuild.ts')).not.toContain('rsbuildConfig'); + expect(targets('src/gap.ts')).not.toContain('command'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/frameworks/expo-modules.ts b/src/resolution/frameworks/expo-modules.ts index a9f60174ad..c70d1d5acd 100644 --- a/src/resolution/frameworks/expo-modules.ts +++ b/src/resolution/frameworks/expo-modules.ts @@ -42,7 +42,12 @@ import type { Node } from '../../types'; import { FrameworkExtractionResult, FrameworkResolver, + ResolutionContext, + ResolvedRef, + UnresolvedRef, } from '../types'; +import { resolveImportPath } from '../import-resolver'; +import { stripCommentsForRegex } from '../strip-comments'; /** * Match `Function("name")`, `AsyncFunction("name")`, or `Property("name")` @@ -196,3 +201,103 @@ export const expoModulesResolver: FrameworkResolver = { return null; }, }; + +// ============================================================================= +// The JS side: `ExpoCamera.scanFromURLAsync(…)` → the module's function +// ============================================================================= + +/** `requireNativeModule('ExpoCamera')`, typed or not, optional or not. */ +const REQUIRE_NATIVE = /\brequire(?:Optional)?NativeModule\s*(?:<\s*([A-Za-z_$][\w$]*)[^>]*>)?\s*\(\s*['"]([A-Za-z_$][\w$]*)['"]\s*\)/; + +interface NativeBinding { + /** The `Name("…")` the module declares. */ + module: string; + /** The declared JS type, from `requireNativeModule(…)`. */ + type: string | null; + /** The file the binding (and its type) is written in. */ + file: string; +} + +const bindingMemo = new WeakMap>(); + +/** + * What `name` is bound to in `file`: `const X = requireNativeModule('N')` + * here, or the import of a module whose default export (or a named export) + * is one — expo-camera's `import CameraManager from './ExpoCameraManager'`, + * whose whole body is `export default requireNativeModule('ExpoCamera')`. + */ +function nativeBinding(name: string, file: string, context: ResolutionContext): NativeBinding | null { + let memo = bindingMemo.get(context); + if (!memo) bindingMemo.set(context, (memo = new Map())); + const key = `${file}\0${name}`; + if (memo.has(key)) return memo.get(key)!; + let found: NativeBinding | null = null; + const own = context.readFile(file); + if (own && own.includes('NativeModule')) { + const local = new RegExp(String.raw`\b(?:const|let|var)\s+${name}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source).exec(stripCommentsForRegex(own, 'typescript')); + if (local) found = { type: local[1] ?? null, module: local[2]!, file }; + } + if (!found) { + const language = /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : /\.jsx$/.test(file) ? 'jsx' : 'javascript'; + const binding = context.getImportMappings(file, language).find((m) => m.localName === name); + const target = binding && binding.source.startsWith('.') ? resolveImportPath(binding.source, file, language, context) : null; + const content = target ? context.readFile(target) : null; + if (target && content && content.includes('NativeModule')) { + const safe = stripCommentsForRegex(content, 'typescript'); + const pattern = binding!.isDefault + ? new RegExp(String.raw`\bexport\s+default\s+` + REQUIRE_NATIVE.source) + : new RegExp(String.raw`\bexport\s+(?:const|let)\s+${binding!.exportedName}\s*(?::[^=]+)?=\s*` + REQUIRE_NATIVE.source); + const m = pattern.exec(safe); + if (m) found = { type: m[1] ?? null, module: m[2]!, file: target }; + } + } + memo.set(key, found); + return found; +} + +/** + * JS calls into an Expo module, bound by the module's NAME, not by the + * method name alone: expo-camera calls its module `CameraManager`, so + * `CameraManager.isAvailableAsync()` went to `CameraView`'s own static + * `isAvailableAsync` in the same file. The native functions the module + * declares are the targets — Swift first, Kotlin beside it — and when it + * declares none of that name, the method on the type the binding is given + * (`requireNativeModule`). + */ +export const expoModulesJsResolver: FrameworkResolver = { + name: 'expo-modules-js', + languages: ['typescript', 'tsx', 'javascript', 'jsx'], + + detect(context) { + return expoModulesResolver.detect(context); + }, + + resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind !== 'calls') return null; + const m = /^([A-Za-z_$][\w$]*)\.([A-Za-z_$][\w$]*)$/.exec(ref.referenceName); + if (!m) return null; + const binding = nativeBinding(m[1]!, ref.filePath, context); + if (!binding) return null; + const fn = m[2]!; + const declared = context + .getNodesByName(fn) + .filter((n) => n.id.startsWith('expo-module:') && n.qualifiedName.endsWith(`::${binding.module}.${fn}`)) + .sort((a, b) => (a.language === 'swift' ? 0 : 1) - (b.language === 'swift' ? 0 : 1)); + if (declared.length > 0) { + return { + original: ref, + targetNodeId: declared[0]!.id, + ...(declared.length > 1 ? { alsoTargets: declared.slice(1).map((n) => ({ targetNodeId: n.id })) } : {}), + confidence: 0.95, + resolvedBy: 'framework', + }; + } + if (binding.type) { + const typed = context + .getNodesByName(fn) + .find((n) => (n.kind === 'method' || n.kind === 'property' || n.kind === 'field') && n.qualifiedName.endsWith(`${binding.type}::${fn}`)); + if (typed) return { original: ref, targetNodeId: typed.id, confidence: 0.9, resolvedBy: 'framework' }; + } + return null; + }, +}; diff --git a/src/resolution/frameworks/index.ts b/src/resolution/frameworks/index.ts index 88221fdfa6..da76403c3f 100644 --- a/src/resolution/frameworks/index.ts +++ b/src/resolution/frameworks/index.ts @@ -31,7 +31,7 @@ import { aspnetResolver } from './csharp'; import { swiftUIResolver, uikitResolver, vaporResolver } from './swift'; import { swiftObjcBridgeResolver } from './swift-objc'; import { reactNativeBridgeResolver } from './react-native'; -import { expoModulesResolver } from './expo-modules'; +import { expoModulesResolver, expoModulesJsResolver } from './expo-modules'; import { expoRouterResolver } from './expo-router'; import { fabricViewResolver } from './fabric'; import { cicsResolver } from './cics'; @@ -90,6 +90,8 @@ const FRAMEWORK_RESOLVERS: FrameworkResolver[] = [ reactNativeBridgeResolver, // Expo Modules — Function/AsyncFunction/Property DSL on Swift/Kotlin expoModulesResolver, + // Expo Modules, JS side — `M.fn()` on a `requireNativeModule('N')` binding → module N's `fn` + expoModulesJsResolver, // Expo Router — `app/` screen files → route nodes; `router.push('/x')` → navigates edges expoRouterResolver, // React Native Fabric / Codegen view components — TS spec → component nodes @@ -177,6 +179,6 @@ export { aspnetResolver } from './csharp'; export { swiftUIResolver, uikitResolver, vaporResolver } from './swift'; export { swiftObjcBridgeResolver } from './swift-objc'; export { reactNativeBridgeResolver } from './react-native'; -export { expoModulesResolver } from './expo-modules'; +export { expoModulesResolver, expoModulesJsResolver } from './expo-modules'; export { expoRouterResolver } from './expo-router'; export { fabricViewResolver } from './fabric'; diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index 1d2e716301..01f3dc2449 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -933,8 +933,11 @@ export function extractImportMappings( function extractJSImports(content: string): ImportMapping[] { const mappings: ImportMapping[] = []; - // ES6 imports - const importRegex = /import\s+(?:(\w+)\s*,?\s*)?(?:\{([^}]+)\})?\s*(?:(\*)\s+as\s+(\w+))?\s*from\s*['"]([^'"]+)['"]/g; + // ES6 imports. `import type { X }` / `import type * as ns` is TypeScript's + // type-only form, not a default import named `type` — which every such + // line used to add, making `type.innerType()` a call on an import. + // (`import type from './x'` still binds `type`: backtracking gives it back.) + const importRegex = /import\s+(?:type\s+(?=[{*]|(?!from\b)\w))?(?:(\w+)\s*,?\s*)?(?:\{([^}]+)\})?\s*(?:(\*)\s+as\s+(\w+))?\s*from\s*['"]([^'"]+)['"]/g; let match; while ((match = importRegex.exec(content)) !== null) { @@ -953,7 +956,8 @@ function extractJSImports(content: string): ImportMapping[] { // Named imports if (namedImports) { - const names = namedImports.split(',').map((s) => s.trim()); + // `{ util, type objectUtil }`: an inline `type` modifier is not part of the name. + const names = namedImports.split(',').map((s) => s.trim().replace(/^type\s+(?=\w)/, '')); for (const name of names) { const aliasMatch = name.match(/(\w+)\s+as\s+(\w+)/); if (aliasMatch) { diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index ea09c6c01b..a35fcf7534 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1509,6 +1509,12 @@ export function matchByExactName( const bareJs = isBareJsCall(ref, context); const bareGo = isBareGoCall(ref, context); const barePhp = isBarePhpCall(ref, context); + // A type, a value or an import the file binds from a package outside the + // repository names nothing in it, whatever kind of reference it is. + if (!bareJs && JS_FAMILY.has(ref.language) && ref.referenceKind !== 'calls' && + /^[A-Za-z_$][\w$]*$/.test(ref.referenceName) && isOutOfRepoBinding(ref.referenceName, ref, context)) { + return null; + } if (bareJs) { const storeAction = matchJsStoreBindingCall(ref, context); if (storeAction) return storeAction; @@ -3598,10 +3604,22 @@ export function matchMethodCall( // Filter to same-language candidates first const sameLanguageMethods = methods.filter(m => m.language === ref.language); - const targetMethods = sameLanguageMethods.length > 0 ? sameLanguageMethods : methods; + let targetMethods = sameLanguageMethods.length > 0 ? sameLanguageMethods : methods; + // A receiver the file imports is another module's value: never a method + // declared in the calling file. expo-camera's `CameraManager.isAvailableAsync()` + // (`import CameraManager from './ExpoCameraManager'`) went to `CameraView`'s + // own static `isAvailableAsync` — the method making the call. + // Ruling the caller's file out may reject a guess; it must never + // manufacture one — the one method left is then no likelier than before. + let narrowed = false; + if (JS_FAMILY.has(ref.language) && isImportBinding(objectOrClass!, ref, context)) { + const kept = targetMethods.filter((m) => m.filePath !== ref.filePath); + narrowed = kept.length !== targetMethods.length; + targetMethods = kept; + } // If only one same-language method with this name exists, use it - if (targetMethods.length === 1 && targetMethods[0]!.language === ref.language) { + if (targetMethods.length === 1 && !narrowed && targetMethods[0]!.language === ref.language) { return { original: ref, targetNodeId: targetMethods[0]!.id, @@ -3660,6 +3678,12 @@ function isImportedModuleReceiver(receiver: string, ref: UnresolvedRef, context: return binding.isNamespace || context.isOutOfRepoImport?.(binding.source, ref.filePath, ref.language) === true; } +/** Is the root of a member call's receiver (`CameraManager` in `CameraManager.x`) one of the file's imports? */ +function isImportBinding(receiver: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + const root = receiver.split('.')[0]!; + return context.getImportMappings?.(ref.filePath, ref.language)?.some((m) => m.localName === root) === true; +} + /** Does the file bind `name` by importing it from outside the repository? */ function isOutOfRepoBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const binding = context.getImportMappings?.(ref.filePath, ref.language)?.find((m) => m.localName === name); @@ -4558,6 +4582,9 @@ export function matchFuzzy( const callableCandidates = candidates.filter((n) => callableKinds.has(n.kind) && !(typeRef && !canNameInTypePosition(n)) && !(rustBare && (n.name !== ref.referenceName || !isRustNameInScope(n, ref, context))) && !(ref.language === 'python' && n.name !== ref.referenceName) && + // JS and TS too: halo's `type RsbuildConfig` (imported from @rsbuild/core) + // is not its local `rsbuildConfig`, kit's vitest `Mock` not a `mock`. + !(JS_FAMILY.has(ref.language) && n.name !== ref.referenceName) && !(pythonShape && !fitsPythonCallShape(n, pythonShape, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context))) .filter((n) => (ref.referenceKind !== 'references' && ref.referenceKind !== 'function_ref') || @@ -4587,8 +4614,9 @@ export function matchFuzzy( isVisibleAcrossFiles(finalCandidates[0]!, ref, context) && isCrossFileReachable(finalCandidates[0]!, ref, context) && !(isBareJsCall(ref, context) && - (TYPE_MEMBER_KINDS.has(finalCandidates[0]!.kind) || isOutOfRepoBinding(ref.referenceName, ref, context) || + (TYPE_MEMBER_KINDS.has(finalCandidates[0]!.kind) || (finalCandidates[0]!.filePath !== ref.filePath && isLocallyBoundJsName(ref.referenceName, ref.filePath, context)))) && + !(JS_FAMILY.has(ref.language) && isOutOfRepoBinding(ref.referenceName, ref, context)) && !(finalCandidates[0]!.kind === 'method' && isBareGoCall(ref, context)) && // A bare PHP call is a function call (case-insensitive, so fuzzy may find // one) — never the class `View` for `view(…)`, never a method. From 72a9346b122be1f95a342c524fb6546105e57191 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:34:14 +0000 Subject: [PATCH 051/363] fix(expo-router): +api files are endpoints; the nearest router manifest decides (#2135) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `app/blog/og-image/[post]+api.ts` was a screen named `/blog/og-image/[post]+api` (evanbacon.dev). An Expo API route is now one endpoint per exported HTTP method (`GET /blog/og-image/:post`), bound to its handler, the way Next.js `route.ts` files are. - The per-app framework gate (#2129) took ANY enclosing manifest's declaration: react-native-true-sheet's root declares expo-router for its example app, so its `docs/` Next.js app got 16 Expo screens (`/layout`, `/page`, `/api/search/route`, …). Walking up from a file, the first manifest that names any gated framework now decides. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/monorepo-app-frameworks.test.ts | 40 ++++++++++++++++ src/extraction/index.ts | 8 +++- src/resolution/frameworks/expo-router.ts | 56 +++++++++++++++++++++++ 5 files changed, 105 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c0c959e44b..25ff68733c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. - A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. - TypeScript's `import type { X }` no longer reads as an import of something named `type`, and `import { a, type B }` binds `B`, not `type B`. So a type-only import now links to the module it names, where it used to link to whatever project symbol had that name. A type imported from a package, like `RsbuildConfig` from `@rsbuild/core`, no longer links to a project symbol, and JavaScript and TypeScript names no longer match a symbol whose name differs only in case (`Mock` is not `mock`). diff --git a/README.md b/README.md index 284027063a..6e91a52f63 100644 --- a/README.md +++ b/README.md @@ -340,7 +340,7 @@ These frameworks additionally emit **`navigates`** edges: the function that send | Router | Routes from | Navigation from | |---|---|---| -| **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | +| **Expo Router** | Every screen file under `app/` (`app/item/[id].tsx` → `/item/[id]`, groups stripped), bound to its default-export component; `+api` files are endpoints (`GET /hello`) bound to their handlers | `router.push` / `replace` / `navigate`, template hrefs, `{ pathname }` objects, and a helper's returned href | | **Next.js** | App Router `app/**/page.tsx` and Pages Router pages (`(group)` stripped, `[slug]` → `:slug`); `app/api/**/route.ts` exports and `pages/api/*` are endpoints, not screens | `router.push` / `replace` / `prefetch`, `redirect()` / `permanentRedirect()` in a server action or page, `NextResponse.redirect(new URL(…))` in middleware, `` and internal `` | | **React Router** | `` (v5 and v6) and `createBrowserRouter([{ path, element }])` | `history.push` / `replace`, `useNavigate`'s `navigate`, a loader's `redirect`, `` / `` / `` / v5's `` / react-router-bootstrap's ``, and a `styled(Link)` wrapper | | **TanStack Router** | `createFileRoute('/posts/$postId')` (file-based) and `createRoute({ path, getParentRoute })` composed up its parent chain (code-based); `_pathless` segments, `(group)` folders, `__root` and `` layouts are not addresses | `navigate({ to })`, a thrown `redirect({ to })`, `` / `` — where `to` is the route PATTERN and the values ride beside it in `params` | diff --git a/__tests__/monorepo-app-frameworks.test.ts b/__tests__/monorepo-app-frameworks.test.ts index f7efbf8caa..c5a4664fc8 100644 --- a/__tests__/monorepo-app-frameworks.test.ts +++ b/__tests__/monorepo-app-frameworks.test.ts @@ -85,4 +85,44 @@ describe('file-based routers in a monorepo', () => { cg.close(); } }); + + it('the nearest manifest that names a router decides: a Next docs app under an Expo root', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'true-sheet', devDependencies: { expo: '*', 'expo-router': '*', react: '*' } }), + 'docs/package.json': JSON.stringify({ name: 'docs', dependencies: { next: '*', react: '*' } }), + 'docs/app/layout.tsx': `export default function RootLayout({ children }: { children: unknown }) { return children; } +`, + 'docs/app/(docs)/[...slug]/page.tsx': `export default function DocPage() { return null; } +`, + 'example/app/_layout.tsx': `export default function Layout() { return null; } +`, + 'example/app/sheet.tsx': `export default function Sheet() { return null; } +`, + }); + try { + expect(routesIn(cg, 'docs/')).toEqual(['/:slug*']); + expect(routesIn(cg, 'example/')).toEqual(['/sheet']); + } finally { + cg.close(); + } + }); + + it('an Expo API route is an endpoint per exported method, not a screen', async () => { + const cg = await project({ + 'package.json': JSON.stringify({ name: 'site', dependencies: { expo: '*', 'expo-router': '*' } }), + 'app/index.tsx': `export default function Home() { return null; } +`, + 'app/blog/og-image/[post]+api.ts': `export async function GET(request: Request) { return new Response('png'); } +export const POST = async () => new Response('ok'); +`, + }); + try { + expect(routesIn(cg, 'app/')).toEqual(['/', 'GET /blog/og-image/:post', 'POST /blog/og-image/:post']); + const get = cg.getNodesByKind('route').find((n) => n.name.startsWith('GET '))!; + const handler = cg.getOutgoingEdges(get.id).map((e) => cg.getNode(e.target)!).find((n) => n.name === 'GET'); + expect(handler?.kind).toBe('function'); + } finally { + cg.close(); + } + }); }); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 706bc0bd12..bb5b9c0fab 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -1948,13 +1948,19 @@ export class ExtractionOrchestrator { const key = `${dir}|${name}`; const memo = this.appFrameworkMemo.get(key); if (memo !== undefined) return memo; + // The nearest manifest that names ANY gated framework decides: true-sheet's + // root package.json declares expo-router for its example app, and its + // `docs/` Next.js app — whose own package.json declares `next` — is not + // an Expo app for it. let applies = false; for (let d: string | null = dir; d !== null; d = d === '' ? null : d.includes('/') ? d.slice(0, d.lastIndexOf('/')) : '') { const declared = this.dependenciesDeclaredIn(d); - if (declared && deps.some((dep) => declared.has(dep))) { + if (!declared) continue; + if (deps.some((dep) => declared.has(dep))) { applies = true; break; } + if ([...this.gatedFrameworks].some(([other, otherDeps]) => other !== name && otherDeps.some((dep) => declared.has(dep)))) break; } this.appFrameworkMemo.set(key, applies); return applies; diff --git a/src/resolution/frameworks/expo-router.ts b/src/resolution/frameworks/expo-router.ts index 5d82b94e26..ef6f44d81c 100644 --- a/src/resolution/frameworks/expo-router.ts +++ b/src/resolution/frameworks/expo-router.ts @@ -70,6 +70,8 @@ export function routePathForFile(filePath: string): string | null { const segs = bare.split('/'); if (segs.includes('__tests__') || segs.includes('__mocks__')) return null; const base = segs[segs.length - 1]!; + // `hello+api.ts` is an endpoint (`apiRoutePathForFile`), not a screen. + if (base.endsWith('+api')) return null; // `_layout` (and any other `_`-prefixed file) is not navigable. `+not-found` // is a real screen; the other `+` files (`+html`, `+native-intent`) are not. if (base.startsWith('_')) return null; @@ -79,6 +81,26 @@ export function routePathForFile(filePath: string): string | null { return '/' + kept.join('/'); } +/** + * An API route's path — `app/blog/og-image/[post]+api.ts` is + * `/blog/og-image/:post`, written the way every server route is so a + * client's `fetch('/blog/og-image/…')` can find it. Null for any other file. + */ +export function apiRoutePathForFile(filePath: string): string | null { + const dir = APP_DIR.exec(filePath); + if (!dir) return null; + const rel = filePath.slice(dir.index + dir[0].length); + const ext = /\.(tsx|ts|jsx|js|mjs|cjs)$/.exec(rel); + if (!ext) return null; + const bare = rel.slice(0, ext.index); + if (!bare.endsWith('+api')) return null; + const segs = bare.slice(0, -'+api'.length).split('/').filter((seg) => seg.length > 0 && !(seg.startsWith('(') && seg.endsWith(')'))); + if (segs[segs.length - 1] === 'index') segs.pop(); + return '/' + segs.map((seg) => seg.replace(/^\[\.\.\.(.+)\]$/, ':$1*').replace(/^\[(.+)\]$/, ':$1')).join('/'); +} + +const API_METHODS = 'GET|POST|PUT|PATCH|DELETE|HEAD|OPTIONS'; + function languageForFile(filePath: string): Language { const ext = ROUTE_EXT.exec(filePath)?.[1]; switch (ext) { @@ -728,6 +750,40 @@ export const expoRouterResolver: FrameworkResolver = { }, extract(filePath: string, content: string) { + const apiPath = apiRoutePathForFile(filePath); + if (apiPath !== null) { + // `export async function GET(request) {…}` / `export const POST = …` — one endpoint per method. + const language = languageForFile(filePath); + const stripped = stripCommentsForRegex(content, 'typescript'); + const nodes: Node[] = []; + const references: UnresolvedRef[] = []; + const seen = new Set(); + const decl = new RegExp(`\\bexport\\s+(?:async\\s+)?function\\s+(${API_METHODS})\\b|\\bexport\\s+(?:const|let)\\s+(${API_METHODS})\\s*=`, 'g'); + let m: RegExpExecArray | null; + while ((m = decl.exec(stripped)) !== null) { + const method = (m[1] ?? m[2])!; + if (seen.has(method)) continue; + seen.add(method); + const line = stripped.slice(0, m.index).split('\n').length; + const node: Node = { + id: `route:${filePath}:${line}:${method}:${apiPath}`, + kind: 'route', + name: `${method} ${apiPath}`, + qualifiedName: `${filePath}::${method}:${apiPath}`, + filePath, + startLine: line, + endLine: line, + startColumn: 0, + endColumn: m[0].length, + language, + isExported: true, + updatedAt: Date.now(), + }; + nodes.push(node); + references.push({ fromNodeId: node.id, referenceName: method, referenceKind: 'references', line, column: 0, filePath, language, candidates: [method] }); + } + return { nodes, references }; + } const routePath = routePathForFile(filePath); if (routePath === null) return { nodes: [], references: [] }; const language = languageForFile(filePath); From a4f982023329371fa24b72d036edd5e597bfe97d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:48:48 +0000 Subject: [PATCH 052/363] fix(resolution): a super call in an override is not a self-call (#2136) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extraction keeps `super.didMoveToWindow()` under the bare method name, so every strategy resolved it to the enclosing override itself. Across the bridge sweep that was 139 self-edges on Charts, 142 on commons-lang, 101 on BookStack, 92 on realm-swift, 82 on react-native-svg — lifecycle overrides and delegates drawn as recursion. `gateSuperSelfCall` declines a `calls` result that targets the calling node when the call site is written through `super` / `base` (C#) / `[super …]` (Objective-C) / `parent::` (PHP) / `super().` / `super(C, self).` (Python). Plain recursion keeps its edge. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/super-call-self-edge.test.ts | 96 ++++++++++++++++++++++++++ src/resolution/index.ts | 31 ++++++++- 3 files changed, 125 insertions(+), 3 deletions(-) create mode 100644 __tests__/super-call-self-edge.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 25ff68733c..dde38f8da8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. - A JavaScript or TypeScript call on something the file imports, like `fireEvent.click(…)` or `util.omit(…)`, no longer links to a same-named method declared in the calling file itself. diff --git a/__tests__/super-call-self-edge.test.ts b/__tests__/super-call-self-edge.test.ts new file mode 100644 index 0000000000..6067a54919 --- /dev/null +++ b/__tests__/super-call-self-edge.test.ts @@ -0,0 +1,96 @@ +/** + * A `super` call inside an override calls the parent's implementation, never + * the method making the call. + * + * Extraction keeps `super.didMoveToWindow()` under the bare method name, so + * every strategy resolved it to the enclosing override itself — a self-edge + * that made a view's lifecycle methods look recursive (eleven of one + * expo-camera view's overrides). Real recursion keeps its edge. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-super-self-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +/** `Owner.method` names of every method whose calls include itself. */ +function selfCalling(cg: CodeGraph): string[] { + const out: string[] = []; + for (const kind of ['method', 'function'] as const) { + for (const n of cg.getNodesByKind(kind)) { + if (cg.getOutgoingEdges(n.id).some((e) => e.kind === 'calls' && e.target === n.id)) out.push(n.name); + } + } + return out.sort(); +} + +describe('a super call is not a self-call', () => { + it('Swift, Kotlin, Java, Python, TypeScript and C#', async () => { + const cg = await project({ + 'ios/CameraView.swift': `import UIKit +class CameraView: UIView { + override func didMoveToWindow() { + super.didMoveToWindow() + setup() + } + func setup() {} + func countdown(_ n: Int) { + if n > 0 { countdown(n - 1) } + } +} +`, + 'android/CameraView.kt': `package app +class CameraView(context: Context) : FrameLayout(context) { + override fun onAttachedToWindow() { + super.onAttachedToWindow() + } +} +`, + 'java/Service.java': `package app; +public class Service extends Base { + @Override + public void start() { + super.start(); + } +} +`, + 'py/views.py': `class ProfileView(BaseView): + def dispatch(self, request): + return super().dispatch(request) +`, + 'ts/list.ts': `export class List extends Base { + render(): string { + return super.render() + '!'; + } +} +`, + 'cs/Handler.cs': `public class Handler : BaseHandler { + public override void Handle() { + base.Handle(); + } +} +`, + }); + try { + // Only the real recursion is left. + expect(selfCalling(cg)).toEqual(['countdown']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index e5fa4f332d..1a9906d819 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -1000,9 +1000,10 @@ export class ReferenceResolver { this.context, ); const scoped = this.gateRustScope(candidate, ref); - const resolved = scoped?.resolvedBy === 'framework' - ? this.gateFrameworkLanguage(scoped, ref) - : this.gateLanguage(scoped, ref); + const resolved = this.gateSuperSelfCall( + scoped?.resolvedBy === 'framework' ? this.gateFrameworkLanguage(scoped, ref) : this.gateLanguage(scoped, ref), + ref, + ); if (!resolved || ref.referenceKind !== 'calls') return resolved; const target = this.nodeById(resolved.targetNodeId); @@ -2869,6 +2870,30 @@ export class ReferenceResolver { return result; } + /** + * `super.didMoveToWindow()` inside an override of `didMoveToWindow` calls + * the PARENT's implementation, never the method making the call. Extraction + * keeps a `super` call under the bare method name, so every strategy found + * the enclosing method itself: one expo-camera view had eleven of its + * overrides "calling" themselves. The parent's method is usually a + * framework's (UIKit, Android, React); a self-edge is never it. Real + * recursion keeps its edge — only a call written through `super` / `base` + * (C#) / `[super …]` (Objective-C) / `parent::` (PHP) / `super().` (Python) + * is declined. + */ + private gateSuperSelfCall(result: ResolvedRef | null, ref: UnresolvedRef): ResolvedRef | null { + if (!result || ref.referenceKind !== 'calls' || result.targetNodeId !== ref.fromNodeId) return result; + const name = ref.referenceName.slice(Math.max(ref.referenceName.lastIndexOf('.'), ref.referenceName.lastIndexOf(':')) + 1); + if (!/^[A-Za-z_$][\w$]*$/.test(name)) return result; + const line = (this.context.getFileLines?.(ref.filePath) ?? this.context.readFile(ref.filePath)?.split(/\r?\n/))?.[ref.line - 1]; + if (!line) return result; + const escaped = name.replace(/\$/g, '\\$'); + const viaSuper = new RegExp( + String.raw`(?:\b(?:super|base)\s*(?:\(\s*(?:[\w.]+\s*,\s*\w+)?\s*\))?\s*\??\.\s*|\[\s*super\s+|\bparent\s*::\s*)` + escaped + String.raw`\b`, + ); + return viaSuper.test(line) ? null : result; + } + /** * A bare Rust name reaches only what is in scope — every strategy's result, * a framework resolver's `Ok(x)` → `struct Ok` construction included (see From 6f798dce6b241c291d06c8cb1039b2f1b9a31c74 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:49:17 +0000 Subject: [PATCH 053/363] fix(react-native): Paper requireNativeComponent views reach native; JSX via default imports (#2137) - `requireNativeComponent('X')` (Paper) now emits the same JS `component` node a Codegen spec does, so the Fabric pass links it to the native view / manager classes; the extractor also runs on `.js` / `.jsx` spec modules (segmented-control's spec is Flow `.js`). react-native-maps' `AIRMap` gains its iOS implementation. - A JSX tag that names no node is the file's DEFAULT import of a module's one component: ``, and element-plus's test `` for `autocomplete.vue` (whose component node is `autocomplete`). A named import never takes a barrel's one component. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + README.md | 2 +- __tests__/fabric-view.test.ts | 52 ++++++++++++++++++++++++++ src/resolution/callback-synthesizer.ts | 25 ++++++++++++- src/resolution/frameworks/fabric.ts | 24 ++++++------ 5 files changed, 91 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dde38f8da8..3336627c07 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. - Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. diff --git a/README.md b/README.md index 6e91a52f63..cd6c7bb442 100644 --- a/README.md +++ b/README.md @@ -365,7 +365,7 @@ Real iOS and React Native codebases live across multiple languages — a Swift c | **RN native → JS events** | JS `new NativeEventEmitter(...).addListener('e', cb)` | ObjC `[self sendEventWithName:@"e" body:...]` · Swift `sendEvent(withName: "e", ...)` · Java/Kotlin `.emit("e", ...)` | Synthesized cross-language event channel keyed by literal event name | | **Expo Modules** | JS `requireNativeModule('X').fn(...)`, directly or through a binding (`export default requireNativeModule('X')`) | Swift / Kotlin `Module { Name("X"); AsyncFunction("fn") { ... } }` | Parses the Expo DSL literals into method nodes; a call on a binding resolves to module `X`'s `fn` on both platforms, else to the method on the binding's declared type | | **Fabric view components** | JSX `` | TS Codegen spec + native impl class | Spec → `component` node; convention-based name+suffix lookup (`View`/`ComponentView`/`Manager`/`ViewManager`) bridges to native | -| **Legacy Paper view managers** | JSX `` | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — Paper-era declarations also produce `component` + `property` nodes | +| **Legacy Paper view managers** | JSX ``, through a `requireNativeComponent('X')` module | ObjC `RCT_EXPORT_VIEW_PROPERTY` · Java/Kotlin `@ReactProp` | Same as Fabric — `requireNativeComponent('X')` is a JS `component` node, and Paper-era declarations also produce `component` + `property` nodes | **Validated on real codebases** (small + medium + large for each bridge): diff --git a/__tests__/fabric-view.test.ts b/__tests__/fabric-view.test.ts index 84e6529404..857167d265 100644 --- a/__tests__/fabric-view.test.ts +++ b/__tests__/fabric-view.test.ts @@ -142,3 +142,55 @@ export function App() { // The full flow: App (TSX) → MyView (fabric-component) → MyViewView (ObjC native class) }); }); + +describe('Paper end-to-end: a requireNativeComponent module rendered under another name', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'paper-fixture-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('segmented-control: → requireNativeComponent("RNCSegmentedControl") → the native view', async () => { + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + write('package.json', '{"dependencies":{"react-native":"^0.73"}}'); + write('js/RNCSegmentedControlNativeComponent.js', `import { requireNativeComponent } from 'react-native'; +module.exports = requireNativeComponent('RNCSegmentedControl'); +`); + write('js/SegmentedControl.js', `import * as React from 'react'; +import RNCSegmentedControlNativeComponent from './RNCSegmentedControlNativeComponent'; +export default function SegmentedControl(props) { + return ; +} +`); + write('ios/RNCSegmentedControl.m', `@implementation RNCSegmentedControl +- (void)setValues:(NSArray *)values { } +@end +`); + write('ios/RNCSegmentedControlManager.m', `@implementation RNCSegmentedControlManager +RCT_EXPORT_MODULE() +RCT_EXPORT_VIEW_PROPERTY(values, NSArray) +@end +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + const control = cg.getNodesByName('SegmentedControl').find((n) => n.filePath === 'js/SegmentedControl.js')!; + const rendered = cg.getOutgoingEdges(control.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!); + const component = rendered.find((n) => n.kind === 'component' && n.name === 'RNCSegmentedControl'); + expect(component?.filePath).toBe('js/RNCSegmentedControlNativeComponent.js'); + const native = cg + .getOutgoingEdges(component!.id) + .filter((e) => (e.metadata as Record | undefined)?.synthesizedBy === 'fabric-native-impl') + .map((e) => cg.getNode(e.target)!) + .filter((n) => n.language === 'objc') + .map((n) => n.name); + expect(native).toContain('RNCSegmentedControl'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index fbd34b3303..806978e842 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -1222,6 +1222,15 @@ const JSX_CHILD_KINDS = new Set(['component', 'function', 'class']); */ const JSX_CHILD_LANGUAGES = [...JS_FAMILY, 'vue', 'svelte']; +function languageForJsxFile(file: string): Language { + if (file.endsWith('.tsx')) return 'tsx'; + if (/\.[cm]?ts$/.test(file)) return 'typescript'; + if (file.endsWith('.jsx')) return 'jsx'; + if (file.endsWith('.vue')) return 'vue'; + if (file.endsWith('.svelte')) return 'svelte'; + return 'javascript'; +} + /** `localName` → the project file it is imported from, for one file's imports. */ function importedFrom(ctx: ResolutionContext, file: string, language: Language): Map { const out = new Map(); @@ -1259,7 +1268,21 @@ function jsxChild( importsOf: () => Map ): Node | undefined { const candidates = ctx.getNodesByName(name).filter((n) => JSX_CHILD_KINDS.has(n.kind)); - if (candidates.length <= 1) return candidates[0]; + if (candidates.length === 0) { + // A name nothing declares is the file's DEFAULT import of a module's one + // component under another name: segmented-control renders + // ``, the default export of a module + // that is `requireNativeComponent('RNCSegmentedControl')`; element-plus's + // tests render `` from `autocomplete.vue`. A named import + // names an export of its own, which a barrel's one component is not. + const isDefault = ctx + .getImportMappings(file, languageForJsxFile(file)) + .some((m) => m.localName === name && m.isDefault); + const from = isDefault ? importsOf().get(name) : undefined; + const components = from ? ctx.getNodesInFile(from).filter((n) => n.kind === 'component') : []; + return components.length === 1 ? components[0] : undefined; + } + if (candidates.length === 1) return candidates[0]; const local = candidates.find((n) => n.filePath === file); if (local) return local; const from = importsOf().get(name); diff --git a/src/resolution/frameworks/fabric.ts b/src/resolution/frameworks/fabric.ts index 308feca686..e5c15e835c 100644 --- a/src/resolution/frameworks/fabric.ts +++ b/src/resolution/frameworks/fabric.ts @@ -50,7 +50,7 @@ import { } from '../types'; const CODEGEN_DECL_RE = - /codegenNativeComponent\s*(?:<[^>]+>)?\s*\(\s*['"]([A-Za-z_][A-Za-z0-9_]*)['"]/g; + /\b(codegenNativeComponent|requireNativeComponent)\s*(?:<[^>]+>)?\s*\(\s*['"]([A-Za-z_][A-Za-z0-9_]*)['"]/g; /** * Legacy Paper view manager macros — older RN libs (still very common, @@ -95,7 +95,9 @@ function deriveComponentNameFromManager(className: string): string { * spec signal. */ function isFabricSpec(source: string): boolean { - return source.includes('codegenNativeComponent'); + // A Paper-era module names its native view the same way: + // segmented-control's `module.exports = requireNativeComponent('RNCSegmentedControl')`. + return source.includes('codegenNativeComponent') || source.includes('requireNativeComponent'); } /** @@ -299,7 +301,8 @@ function extractFabricNodes(filePath: string, source: string): Node[] { CODEGEN_DECL_RE.lastIndex = 0; let m: RegExpExecArray | null; while ((m = CODEGEN_DECL_RE.exec(source)) !== null) { - const componentName = m[1]!; + const declaredWith = m[1]!; + const componentName = m[2]!; const before = source.slice(0, m.index); const startLine = before.split('\n').length; const startColumn = before.length - before.lastIndexOf('\n') - 1; @@ -314,15 +317,14 @@ function extractFabricNodes(filePath: string, source: string): Node[] { name: componentName, qualifiedName: `${filePath}::${componentName}`, filePath, - // The spec file is .ts or .tsx; use the file's apparent language - // by extension. Trim to a known Language value. - language: filePath.endsWith('.tsx') ? 'tsx' : 'typescript', + // The spec file's language by extension (a Paper module is often Flow `.js`). + language: filePath.endsWith('.tsx') ? 'tsx' : /\.[cm]?ts$/.test(filePath) ? 'typescript' : filePath.endsWith('.jsx') ? 'jsx' : 'javascript', startLine, endLine: startLine, startColumn, - endColumn: startColumn + 'codegenNativeComponent'.length, - docstring: `Fabric/Codegen native component '${componentName}'`, - signature: `codegenNativeComponent('${componentName}')`, + endColumn: startColumn + declaredWith.length, + docstring: declaredWith === 'codegenNativeComponent' ? `Fabric/Codegen native component '${componentName}'` : `Native component '${componentName}' (requireNativeComponent)`, + signature: declaredWith === 'codegenNativeComponent' ? `codegenNativeComponent('${componentName}')` : `requireNativeComponent('${componentName}')`, isExported: true, updatedAt: now, }); @@ -363,7 +365,7 @@ function extractFabricNodes(filePath: string, source: string): Node[] { export const fabricViewResolver: FrameworkResolver = { name: 'fabric-view', - languages: ['typescript', 'tsx', 'objc', 'java', 'kotlin'], + languages: ['typescript', 'tsx', 'javascript', 'jsx', 'objc', 'java', 'kotlin'], detect(context) { // Root package.json is the common case. The indexer only tracks @@ -392,7 +394,7 @@ export const fabricViewResolver: FrameworkResolver = { // Pick the right extractor by file language. The framework registry // already filters by `languages` so we only see relevant files. let nodes: Node[] = []; - if (filePath.endsWith('.ts') || filePath.endsWith('.tsx')) { + if (/\.(?:[cm]?[jt]s|[jt]sx)$/.test(filePath)) { nodes = extractFabricNodes(filePath, source); } else if (filePath.endsWith('.m') || filePath.endsWith('.mm')) { nodes = extractLegacyViewManagerNodes(filePath, source); From 6a02dcaa76da7bc2b05132302a2d33f9d7c2fc05 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 07:57:43 +0000 Subject: [PATCH 054/363] =?UTF-8?q?fix(react-native):=20event=20names=20he?= =?UTF-8?q?ld=20in=20constants=20join=20the=20native=E2=86=94JS=20channel?= =?UTF-8?q?=20(#2138)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rn-event-channel pairs a native emit with a JS listener on the event's LITERAL name. Most libraries name events through constants — NetInfo listens with `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)`, many emit with `.emit(EVENT_NAME, …)` or `sendEventWithName:kEvent` — so their channels were empty. Identifier / member-chain event names are now read to the literal their declaration holds (JS/TS const & enum, Java `static final`, Kotlin `const val`, ObjC `NSString *const` / `#define`, Swift `let` / enum case), scoped as the language scopes them: a bare name in the same file or behind an import, `Owner.NAME` inside Owner or a namespace import's file. The handler of a constant-named listener is a function in the same file, else the enclosing one. A first cut that read constants by name alone turned a `.emit(eventName, …)` parameter into some test's `eventName = "pong"`. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rn-event-channel.test.ts | 83 +++++++++++++++++++ src/resolution/callback-synthesizer.ts | 105 +++++++++++++++++++++++++ 3 files changed, 189 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3336627c07..ba364048bd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. - React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. - Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. diff --git a/__tests__/rn-event-channel.test.ts b/__tests__/rn-event-channel.test.ts index ccf6e751e3..a5b01e58ed 100644 --- a/__tests__/rn-event-channel.test.ts +++ b/__tests__/rn-event-channel.test.ts @@ -226,3 +226,86 @@ function tick() {} expect(rows[1].registered_at).toBe('App.tsx:4'); }); }); + +describe('RN event channel — event names held in constants', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'rn-event-const-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + const write = (rel: string, content: string) => { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + }; + const channel = (cg: CodeGraph) => + ((cg as any).db.db + .prepare( + `SELECT s.name s, t.name t, json_extract(e.metadata,'$.event') event FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'rn-event-channel' ORDER BY s, t` + ) + .all() as Array<{ s: string; t: string; event: string }>).map((r) => `${r.s} -> ${r.t} (${r.event})`); + + it('NetInfo: a JS listener on `PrivateTypes.DEVICE_CONNECTIVITY_EVENT`, native sends the literal', async () => { + write('package.json', '{"name":"netinfo","dependencies":{"react-native":"^0.73"}}'); + write('ios/RNCNetInfo.m', `@implementation RNCNetInfo +- (void)connectionChanged { + [self sendEventWithName:@"netInfo.networkStatusDidChange" body:@{}]; +} +@end +`); + write('src/internal/privateTypes.ts', `export const DEVICE_CONNECTIVITY_EVENT = 'netInfo.networkStatusDidChange'; +`); + write('src/internal/state.ts', `import * as PrivateTypes from './privateTypes'; +export default class State { + start(emitter: any) { + emitter.addListener( + PrivateTypes.DEVICE_CONNECTIVITY_EVENT, + this._handleNativeStateUpdate, + ); + } + _handleNativeStateUpdate(state: unknown) { return state; } +} +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(channel(cg)).toEqual(['connectionChanged -> _handleNativeStateUpdate (netInfo.networkStatusDidChange)']); + } finally { + cg.close(); + } + }); + + it('a Java constant emit and a Kotlin const val meet a JS literal listener', async () => { + write('package.json', '{"name":"lib","dependencies":{"react-native":"^0.73"}}'); + write('android/src/main/java/app/Tracker.java', `package app; +public class Tracker { + private static final String PROGRESS_EVENT = "downloadProgress"; + void report(ReactContext ctx) { + ctx.getJSModule(RCTDeviceEventEmitter.class).emit(PROGRESS_EVENT, null); + } +} +`); + write('android/src/main/java/app/Finisher.kt', `package app +const val DONE_EVENT = "downloadDone" +class Finisher { + fun finish(ctx: ReactContext) { + sendEvent(ctx, DONE_EVENT, null) + } +} +`); + write('src/index.js', `export function onProgress(e) { return e; } +export function onDone(e) { return e; } +emitter.addListener('downloadProgress', onProgress); +emitter.addListener('downloadDone', onDone); +`); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(channel(cg)).toEqual(['finish -> onDone (downloadDone)', 'report -> onProgress (downloadProgress)']); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/callback-synthesizer.ts b/src/resolution/callback-synthesizer.ts index 806978e842..d86ad94e1a 100644 --- a/src/resolution/callback-synthesizer.ts +++ b/src/resolution/callback-synthesizer.ts @@ -1513,8 +1513,79 @@ const RN_JVM_EMIT_RE = /\.emit\s*\(\s*"([^"]+)"\s*,/g; // is followed by `… ) {`) never matches. Multi-line tolerant. (java/kotlin/swift) const RN_NATIVE_SENDEVENT_RE = /\bsendEvent\s*\([^;{}]*?"([^"]+)"/g; +// The same calls with the event named by a CONSTANT — NetInfo listens with +// `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)`, many libraries emit +// with `.emit(EVENT_NAME, …)` or `sendEventWithName:kLocationEvent`. +const RN_OBJC_SEND_CONST_RE = /\bsendEventWithName\s*:\s*([A-Za-z_]\w*)\b/g; +const RN_SWIFT_SEND_CONST_RE = /\bsendEvent\s*\(\s*withName\s*:\s*([A-Za-z_][\w.]*)/g; +const RN_JVM_EMIT_CONST_RE = /\.emit\s*\(\s*([A-Za-z_][\w.]*)\s*,/g; +const RN_NATIVE_SENDEVENT_CONST_RE = /\bsendEvent\s*\(\s*[A-Za-z_][\w.]*\s*,\s*([A-Za-z_][\w.]*)\s*,/g; +const RN_JS_LISTEN_CONST_RE = /\.(?:on|once|addListener)\(\s*((?:[A-Za-z_$][\w$]*\.)*[A-Za-z_$][\w$]*)\s*,\s*([A-Za-z_$][\w$.]*|(?:async\s*)?(?:\([^)]*\)\s*=>|[A-Za-z_$][\w$]*\s*=>|function\s*\())/g; + +const CONSTANT_KINDS = new Set(['constant', 'variable', 'field', 'property', 'enum_member']); + +/** + * The string an event-name constant holds, read off its declaration line: + * `export const DEVICE_CONNECTIVITY_EVENT = 'netInfo.networkStatusDidChange'`, + * `static final String EVENT = "x"`, `const val EVENT = "x"`, + * `NSString *const kEvent = @"x"`, `#define kEvent @"x"`, `static let event = "x"`, + * an enum case `Changed = 'changed'`. + * + * Scoped as the language scopes the name, never by the name alone — a + * parameter `eventName` is not some test's `eventName = "pong"`: + * - a bare `NAME` is a declaration in the same file, or (JS) the one an + * import of that name points at; + * - `Owner.NAME` is a declaration inside `Owner`, or (JS) `NAME` in the file + * a namespace import `Owner` points at. + * Null unless those declarations agree on one literal. + */ +function constantEventName(expr: string, file: string, ctx: ResolutionContext, memo: Map): string | null { + const key = `${file}\0${expr}`; + if (memo.has(key)) return memo.get(key)!; + const parts = expr.replace(/\.rawValue$/, '').split('.'); + const name = parts[parts.length - 1]!; + const owner = parts.length > 1 ? parts[parts.length - 2]! : null; + let value: string | null = null; + if (/^[A-Za-z_$][\w$]*$/.test(name) && name !== 'this' && (parts.length <= 2 || parts[0] !== 'this')) { + const read = (n: Node): string | null => { + const line = (ctx.getFileLines?.(n.filePath) ?? ctx.readFile(n.filePath)?.split(/\r?\n/))?.[n.startLine - 1] ?? ''; + const at = line.indexOf(name); + if (at < 0) return null; + const rest = line.slice(at + name.length); + const m = /^\s*(?::[^=;]*)?[=:]\s*@?(["'`])([^"'`]+)\1/.exec(rest) ?? /^\s+@?(")([^"]+)"/.exec(rest); + return m ? m[2]! : null; + }; + const declaredIn = (f: string) => ctx.getNodesInFile(f).filter((n) => n.name === name && CONSTANT_KINDS.has(n.kind)); + const js = /\.(?:[cm]?[jt]sx?)$/.test(file); + const language: Language = /\.tsx$/.test(file) ? 'tsx' : /\.[cm]?ts$/.test(file) ? 'typescript' : file.endsWith('.jsx') ? 'jsx' : 'javascript'; + const importedFile = (local: string): string | null => { + if (!js) return null; + const binding = ctx.getImportMappings(file, language).find((m) => m.localName === local); + return binding ? resolveImportPath(binding.source, file, language, ctx) : null; + }; + let decls: Node[] = []; + if (owner === null) { + decls = declaredIn(file); + if (decls.length === 0) { + const from = importedFile(name); + if (from) decls = declaredIn(from); + } + } else { + const from = importedFile(owner); + decls = from + ? declaredIn(from) + : ctx.getNodesByName(name).filter((n) => CONSTANT_KINDS.has(n.kind) && new RegExp(`(?:^|[.:])${owner}(?:[.:]|$)`).test(n.qualifiedName.slice(0, n.qualifiedName.lastIndexOf(name)))); + } + const values = new Set(decls.map(read).filter((v): v is string => v !== null)); + value = values.size === 1 ? [...values][0]! : null; + } + memo.set(key, value); + return value; +} + async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise { let scannedFiles = 0; + const constants = new Map(); // Native dispatchers (source = the native method whose body sends the // event) and JS handlers (target = the function/method registered as // the listener) keyed by event name. @@ -1544,6 +1615,11 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_OBJC_SEND_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + RN_OBJC_SEND_CONST_RE.lastIndex = 0; + while ((m = RN_OBJC_SEND_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } } // Swift side: same RCTEventEmitter method, parens/named-args syntax. @@ -1557,6 +1633,11 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_NATIVE_SENDEVENT_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + RN_SWIFT_SEND_CONST_RE.lastIndex = 0; + while ((m = RN_SWIFT_SEND_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } } // JVM side: `.emit("X", …)` in Java/Kotlin, plus the common @@ -1573,6 +1654,13 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis while ((m = RN_NATIVE_SENDEVENT_RE.exec(content))) { if (m[1]) addDispatcher(m[1], lineOf(m.index)); } + for (const re of [RN_JVM_EMIT_CONST_RE, RN_NATIVE_SENDEVENT_CONST_RE]) { + re.lastIndex = 0; + while ((m = re.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (event) addDispatcher(event, lineOf(m.index)); + } + } } // JS subscribers (.addListener("X", handler)). Restrict to JS-family @@ -1658,6 +1746,23 @@ async function rnEventEdges(ctx: ResolutionContext, onYield: MaybeYield): Promis if (!map.has(enclosing.id)) map.set(enclosing.id, `${file}:${lineOf(m.index)}`); jsHandlersByEvent.set(event, map); } + // A constant event name: the listener lands where a literal one would — + // the named handler when it is a node, else the enclosing function. + RN_JS_LISTEN_CONST_RE.lastIndex = 0; + while ((m = RN_JS_LISTEN_CONST_RE.exec(content))) { + const event = constantEventName(m[1]!, file, ctx, constants); + if (!event) continue; + const arg = m[2]!; + const bare = /^[A-Za-z_$][\w$.]*$/.test(arg) ? arg.slice(arg.lastIndexOf('.') + 1) : null; + const line = lineOf(m.index); + // The handler this file names — never a same-named function elsewhere. + const named = bare ? nodesInFile.find((n) => (n.kind === 'function' || n.kind === 'method') && n.name === bare) : undefined; + const target = named ?? enclosingFn(nodesInFile, line); + if (!target) continue; + const map = jsHandlersByEvent.get(event) ?? new Map(); + if (!map.has(target.id)) map.set(target.id, `${file}:${line}`); + jsHandlersByEvent.set(event, map); + } } } From 66d5dc36418a8b8becf4a0a7572e527dedf4a5a5 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 08:10:26 +0000 Subject: [PATCH 055/363] fix(resolution): fuzzy matching is exact-case outside case-insensitive languages (#2139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The last-resort fuzzy match looks names up in a lowercase index — right for PHP, Pascal/Delphi, CFML, COBOL and VB.NET, whose identifiers ignore case, and wrong for every other language. jsoup's 1,844 `@Test` annotations decorated a `CharPredicate.test` method; commons-lang's `new BitSet()` instantiated a `bitSet` field accessor 64 times; mall's `new Info()` an `info()` endpoint; gson's `Method` type a test's `method()`; fmt's `#include ` gtest's `Optional`. This generalizes the per-language exact-case rules (Rust, Python, JS) into one: outside CASE_INSENSITIVE_LANGUAGES a fuzzy candidate must match the reference's exact name. The cross-language fuzzy-gate test now builds its winner from an exact-case name. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cross-language-resolution.test.ts | 7 ++- __tests__/fuzzy-case-sensitivity.test.ts | 69 +++++++++++++++++++++ src/resolution/name-matcher.ts | 18 +++--- 4 files changed, 86 insertions(+), 9 deletions(-) create mode 100644 __tests__/fuzzy-case-sensitivity.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ba364048bd..f35a343785 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Java, Kotlin, Swift, C#, C and C++, Go, Scala, Ruby, Dart and the other case-sensitive languages, a name no longer links to a symbol whose name differs only in case. On jsoup, JUnit's `@Test` had linked every test to a `test` method; `new CookieManager()` had linked to a `cookieManager()` getter, and a `Method` type to a `method()`. PHP, Pascal, CFML, COBOL and VB.NET, whose names really do ignore case, are unchanged. - React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. - React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. - A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. diff --git a/__tests__/cross-language-resolution.test.ts b/__tests__/cross-language-resolution.test.ts index bfd40dbbde..9fd5925a46 100644 --- a/__tests__/cross-language-resolution.test.ts +++ b/__tests__/cross-language-resolution.test.ts @@ -132,8 +132,11 @@ describe('cross-language name resolution (#1986)', () => { expect(matchByQualifiedName(ref(method.qualifiedName), context)?.targetNodeId).toBe(method.id); expect(matchReference(ref(method.qualifiedName), context)).toBeNull(); expect(matchReference(ref('Foreign.act'), context)).toBeNull(); - expect(matchFuzzy(ref('FUZZYNAME'), context)).not.toBeNull(); - expect(matchReference(ref('FUZZYNAME'), context)).toBeNull(); + // Fuzzy is case-sensitive outside PHP/Pascal/CFML/COBOL/VB.NET, so the + // fuzzy winner here shares the exact name; the language gate still rejects it. + expect(matchFuzzy(ref('fuzzyName'), context)).not.toBeNull(); + expect(matchReference(ref('fuzzyName'), context)).toBeNull(); + expect(matchFuzzy(ref('FUZZYNAME'), context)).toBeNull(); // Exact qualified match selects Python even though a Swift type has the // same bare name. Rejection must not fall through to that Swift type. const foreign = graph.getNodesByName('Winner').find((n) => n.language === 'python')!; diff --git a/__tests__/fuzzy-case-sensitivity.test.ts b/__tests__/fuzzy-case-sensitivity.test.ts new file mode 100644 index 0000000000..fcfd301d4c --- /dev/null +++ b/__tests__/fuzzy-case-sensitivity.test.ts @@ -0,0 +1,69 @@ +/** + * The last-resort fuzzy match compares names without regard to case. That + * is right for PHP, Pascal, CFML, COBOL and VB.NET, whose identifiers + * resolve that way, and wrong everywhere else: jsoup's 1,844 `@Test` + * annotations decorated a `CharPredicate.test` method, `new CookieManager()` + * instantiated a `cookieManager()` getter, gson's `Method` type referenced a + * test's `method()`. + */ +import { describe, it, expect, afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const roots: string[] = []; +afterAll(() => { + for (const r of roots.splice(0)) fs.rmSync(r, { recursive: true, force: true }); +}); + +async function project(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fuzzy-case-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +const targetsFrom = (cg: CodeGraph, file: string): string[] => + cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id)) + .filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind}:${n.name}`) + .sort(); + +describe('fuzzy matching and case', () => { + it('Java: an annotation or a type is not a method whose name differs only in case', async () => { + const cg = await project({ + 'src/main/java/app/CharPredicate.java': `package app; +public interface CharPredicate { + boolean test(char c); +} +`, + 'src/main/java/app/Request.java': `package app; +public class Request { + public Object cookieManager() { return null; } +} +`, + 'src/test/java/app/ParserTest.java': `package app; +import org.junit.jupiter.api.Test; +import java.net.CookieManager; +public class ParserTest { + @Test void parses() { + Object manager = new CookieManager(); + } +} +`, + }); + try { + const targets = targetsFrom(cg, 'src/test/java/app/ParserTest.java'); + expect(targets).not.toContain('method:test'); + expect(targets).not.toContain('method:cookieManager'); + } finally { + cg.close(); + } + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index a35fcf7534..aa1db3deb3 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -943,6 +943,9 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex const JS_FAMILY = new Set(['typescript', 'tsx', 'javascript', 'jsx', 'vue', 'svelte', 'astro']); const JS_TS = new Set(['typescript', 'tsx', 'javascript', 'jsx']); +/** Languages whose identifiers resolve regardless of case. */ +const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfml', 'cfscript', 'cfquery', 'cobol', 'vbnet']); + /** * Whether a JS/TS `calls` ref is a RECEIVER-LESS call — `serialize(x)`, not * `this.serialize(x)` / `obj.serialize(x)`. The extractor emits `this.m()` @@ -4577,14 +4580,15 @@ export function matchFuzzy( const rustBare = ref.language === 'rust' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const pythonShape = pythonCallShape(ref, context); const javaBare = ref.language === 'java' && ref.referenceKind === 'calls' && /^[A-Za-z_$][\w$]*$/.test(ref.referenceName); - // Rust and Python names are case-sensitive: `Bytes` is not the method `bytes`, - // Python's builtin `dir(…)` not a class `Dir`. + // Names are case-sensitive in every language but a handful: Rust's + // `Bytes` is not the method `bytes`, Python's builtin `dir(…)` not a class + // `Dir`, halo's `type RsbuildConfig` not its local `rsbuildConfig`, a Java + // `Node` not a `node()`. Only PHP, Pascal/Delphi, CFML, COBOL and VB.NET + // resolve a name without regard to case, which is what this fallback's + // lowercase index is for. const callableCandidates = candidates.filter((n) => callableKinds.has(n.kind) && !(typeRef && !canNameInTypePosition(n)) && - !(rustBare && (n.name !== ref.referenceName || !isRustNameInScope(n, ref, context))) && - !(ref.language === 'python' && n.name !== ref.referenceName) && - // JS and TS too: halo's `type RsbuildConfig` (imported from @rsbuild/core) - // is not its local `rsbuildConfig`, kit's vitest `Mock` not a `mock`. - !(JS_FAMILY.has(ref.language) && n.name !== ref.referenceName) && + !(rustBare && !isRustNameInScope(n, ref, context)) && + !(!CASE_INSENSITIVE_LANGUAGES.has(ref.language) && n.name !== ref.referenceName) && !(pythonShape && !fitsPythonCallShape(n, pythonShape, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context))) .filter((n) => (ref.referenceKind !== 'references' && ref.referenceKind !== 'function_ref') || From a3d675ef4d7c0d014ee928e598bcacb8a632470d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 08:22:09 +0000 Subject: [PATCH 056/363] fix(vue): Options API methods, computed, watchers and hooks are symbols (#2140) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A ` +` + ); + const cg = await CodeGraph.init(root, { index: true }); + try { + const nodes = cg.getNodesInFile('src/views/login/index.vue'); + const method = (name: string) => nodes.find((n) => n.kind === 'method' && n.name === name)!; + expect(nodes.filter((n) => n.kind === 'method').map((n) => n.name).sort()).toEqual(['data', 'handleLogin', 'validate']); + expect(method('handleLogin').qualifiedName).toBe('index::handleLogin'); + // The call inside handleLogin is handleLogin's, not the file's. + const calls = cg.getOutgoingEdges(method('handleLogin').id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!.name); + expect(calls).toContain('validate'); + // `@click="handleLogin"` in the template runs the method. + const component = nodes.find((n) => n.kind === 'component')!; + const handlers = cg.getOutgoingEdges(component.id).filter((e) => e.target === method('handleLogin').id); + expect(handlers.length).toBeGreaterThan(0); + } finally { + cg.close(); + } + }); +}); diff --git a/src/extraction/vue-extractor.ts b/src/extraction/vue-extractor.ts index 862f17c793..5429a90dea 100644 --- a/src/extraction/vue-extractor.ts +++ b/src/extraction/vue-extractor.ts @@ -2,6 +2,7 @@ import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference, Lan import { generateNodeId } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { isLanguageSupported } from './grammars'; +import { vueOptionsMembers } from './vue-options-api'; /** * Vue built-in components — skipped so a `` / `` in the @@ -114,6 +115,67 @@ export class VueExtractor { return node; } + /** + * Method nodes for an Options API component's members (see + * ./vue-options-api), and the references and edges written inside each — + * which the TS extractor attributed to the file — re-attributed to it. + * Lines are block-relative here; the caller offsets them with the rest. + */ + private addOptionsMembers( + block: { content: string; startLine: number }, + result: ExtractionResult, + componentNodeId: string + ): void { + const members = vueOptionsMembers(block.content); + if (members.length === 0) return; + const component = this.nodes.find((n) => n.id === componentNodeId); + const owner = component?.name ?? 'component'; + const lineAt = (offset: number) => block.content.slice(0, offset).split('\n').length; + const colAt = (offset: number) => offset - block.content.lastIndexOf('\n', offset - 1) - 1; + const now = Date.now(); + const created: Node[] = []; + for (const m of members) { + const startLine = lineAt(m.start); + const endLine = lineAt(m.end); + created.push({ + id: generateNodeId(this.filePath, 'method', `${owner}.${m.name}`, startLine + block.startLine), + kind: 'method', + name: m.name, + qualifiedName: `${owner}::${m.name}`, + filePath: this.filePath, + language: 'vue', + startLine, + endLine, + startColumn: colAt(m.start), + endColumn: colAt(m.end), + updatedAt: now, + }); + } + // Innermost member for a line: `computed: { x: { get() {…} } }` is one member. + const memberAt = (line: number): Node | undefined => { + let best: Node | undefined; + for (const n of created) { + if (n.startLine <= line && n.endLine >= line && (!best || n.startLine >= best.startLine)) best = n; + } + return best; + }; + // What the TS extractor attributed to the file (or to nothing narrower). + const fileNode = result.nodes.find((n) => n.kind === 'file'); + const narrower = new Set(result.nodes.filter((n) => n.kind !== 'file').map((n) => n.id)); + const isFileLevel = (id: string) => (fileNode ? id === fileNode.id : !narrower.has(id)); + for (const ref of result.unresolvedReferences) { + if (!isFileLevel(ref.fromNodeId)) continue; + const member = memberAt(ref.line); + if (member) ref.fromNodeId = member.id; + } + for (const edge of result.edges) { + if (edge.kind === 'contains' || !edge.line || !isFileLevel(edge.source)) continue; + const member = memberAt(edge.line); + if (member) edge.source = member.id; + } + result.nodes.push(...created); + } + /** * Extract +`, + 'backend/query.py': `class QueryFilterBuilder: + def __init__(self, raw): + self.raw = raw +`, + 'backend/test_query.py': `from mealie.services.query_filter.builder import QueryFilterBuilder + + +def test_builder(): + return QueryFilterBuilder("x") +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('the Vue resolver', () => { + it('leaves a Python call to its own class alone', () => { + const ids = cg.getNodesInFile('backend/test_query.py').map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.filePath); + expect(targets).not.toContain('frontend/components/QueryFilterBuilder.vue'); + }); +}); diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index 03e50509ac..e894e9b468 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -9,6 +9,9 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { dependsOn } from './package-deps'; +/** The languages a Vue app's scripts are written in. */ +const VUE_SCRIPT_LANGUAGES: ReadonlySet = new Set(['vue', 'javascript', 'typescript', 'tsx', 'jsx']); + /** * Vue 3 compiler macros — compiler-provided, not user code */ @@ -102,6 +105,11 @@ export const vueResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // Vue's macros, auto-imports and components are a script's, never a + // backend's: mealie's Python `QueryFilterBuilder(...)` is not the + // `QueryFilterBuilder.vue` component. + if (!VUE_SCRIPT_LANGUAGES.has(ref.language)) return null; + // Pattern 1: Vue compiler macros (defineProps, defineEmits, etc.) if (VUE_COMPILER_MACROS.has(ref.referenceName)) { return { From 5c01f017cf9dcf8372684cfb42808786b2e516e2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:10:14 +0000 Subject: [PATCH 122/363] fix(react): the React resolver only resolves a script's references (#2211) The React resolver's `languages` gates its route extraction, not its resolve(): framework resolvers see every language's references. Its component rule was already JSX-only, but its hook (`use...`) and context (`...Context` / `...Provider`) rules were not: in halo - a Spring app whose UI package makes React detected - 209 Java import edges (`SecurityContext`, `ApplicationContext`, `ObjectProvider`, `DirtiesContext`...) were resolved by it onto import nodes. It now answers only JavaScript / TypeScript / TSX / JSX references. halo -209 +3 (a Java test's `responseContext()` helper now resolves to itself); trpc, excalidraw, react-redux-realworld, bulletproof-react, react-native-reusables unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/react-resolver-language.test.ts | 65 +++++++++++++++++++++++ src/resolution/frameworks/react.ts | 6 +++ 3 files changed, 72 insertions(+) create mode 100644 __tests__/react-resolver-language.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 77cd625e10..58a270a00c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. - Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. - A Dart call like `ext.endsWith(".avi")` or `map.putIfAbsent(…)` on a value whose type isn't known now counts as Dart's own String, List or Map method. It no longer lands on a project method of the same name. A Dart extension is also matched by the type it extends, not by its own name, so getx's `ext.endsWith` on a String stopped reaching `extension RxStringExt on Rx`. diff --git a/__tests__/react-resolver-language.test.ts b/__tests__/react-resolver-language.test.ts new file mode 100644 index 0000000000..50bbf3bf36 --- /dev/null +++ b/__tests__/react-resolver-language.test.ts @@ -0,0 +1,65 @@ +/** + * The React resolver's hook (`use…`) and context (`…Context` / `…Provider`) + * rules are a script's, never another language's: in halo — a Spring app + * with a React-detected UI package — Java imports of `SecurityContext` or + * `ObjectProvider` were taken by the context rule, and a Java test's own + * `responseContext()` helper went unresolved. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-react-lang-')); + const files: Record = { + 'package.json': JSON.stringify({ name: 'app', dependencies: { react: '^18' } }), + 'ui/src/ThemeContext.tsx': `import { createContext } from 'react'; + +export const responseContext = createContext(null); +`, + 'src/main/java/app/Filter.java': `package app; + +import org.springframework.security.core.context.SecurityContext; + +public class Filter { + Object responseContext() { + return null; + } + + void filter() { + responseContext(); + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('the React resolver', () => { + it('does not take a Java import of a …Context class', () => { + const ids = cg.getNodesInFile('src/main/java/app/Filter.java').map((n) => n.id); + const byReact = cg.getOutgoingEdgesFrom(ids).filter((e) => (e.metadata as { framework?: string } | undefined)?.framework === 'react'); + expect(byReact).toEqual([]); + }); + + it('leaves a Java call named like a context to Java', () => { + const filter = cg.getNodesInFile('src/main/java/app/Filter.java').find((n) => n.name === 'filter')!; + const targets = cg.getOutgoingEdges(filter.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!); + expect(targets.map((t) => t.filePath)).not.toContain('ui/src/ThemeContext.tsx'); + expect(targets.map((t) => t.qualifiedName)).toContain('app::Filter::responseContext'); + }); +}); diff --git a/src/resolution/frameworks/react.ts b/src/resolution/frameworks/react.ts index f9180fa0be..c4fa6946c1 100644 --- a/src/resolution/frameworks/react.ts +++ b/src/resolution/frameworks/react.ts @@ -10,6 +10,9 @@ import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from import { dependsOn } from './package-deps'; import { resolveImportPath } from '../import-resolver'; +/** The languages React components, hooks and contexts are written and used in. */ +const REACT_SCRIPT_LANGUAGES: ReadonlySet = new Set(['typescript', 'javascript', 'tsx', 'jsx']); + export const reactResolver: FrameworkResolver = { name: 'react', // Includes 'tsx'/'jsx' so route extraction runs on JSX files (where @@ -34,6 +37,9 @@ export const reactResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // Components, hooks and contexts are a script's: halo's Java + // `import org.springframework…SecurityContext` is no React context. + if (!REACT_SCRIPT_LANGUAGES.has(ref.language)) return null; if (ref.referenceName.startsWith(LAZY_ROUTE_PREFIX)) { const target = lazyRouteComponent(ref.referenceName.slice(LAZY_ROUTE_PREFIX.length), ref.filePath, context); return target ? { original: ref, targetNodeId: target, confidence: 0.9, resolvedBy: 'framework' } : null; From c20283e2ce8b07f5842bb6cd7c8dcfbc320536ab Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:21:17 +0000 Subject: [PATCH 123/363] fix(resolution): a call to an overload of the calling method reaches it (#2212) A method that delegates to another overload of itself - `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`, `DeserializeXNode(value)` returning `DeserializeXNode(value, null)` - bound to itself, since every overload shares the name: the fuller overload never listed its convenience overloads among its callers, and its impact missed them. When a name match lands on the calling method itself, in a language that overloads by arity (C#, Java, Kotlin, Swift, C++, Scala, Dart, VB.NET), the call's argument count is read and compared with the method's parameters (defaults and varargs / `params` / `...` counted); if they cannot take it, the one same-owner overload they fit is the target. Real recursion (`depth(n - 1)`) keeps its self-edge. Self-loops -> overload edges: commons-lang 637 -> 457, Newtonsoft 303 -> 196, halo 173 -> 116, okhttp 248 -> 208, Alamofire 147 -> 120, gson 74 -> 55, AutoMapper 71 -> 57 (edge counts unchanged; overloads share a qualified name, so only the target moves). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-overload-calls.test.ts | 82 +++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 39 ++++++++++++- 3 files changed, 121 insertions(+), 1 deletion(-) create mode 100644 __tests__/self-overload-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 58a270a00c..b0370b308e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. - Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. diff --git a/__tests__/self-overload-calls.test.ts b/__tests__/self-overload-calls.test.ts new file mode 100644 index 0000000000..c0b121c5b9 --- /dev/null +++ b/__tests__/self-overload-calls.test.ts @@ -0,0 +1,82 @@ +/** + * A method calling its own name with arguments its own parameters cannot + * take calls another overload — `toInstant(instant)` delegating to + * `toInstant(instant, Instant.EPOCH)` — not itself: the self-edge hid the + * convenience overloads from the full overload's callers (commons-lang had + * 180 such, Newtonsoft 107). + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-self-overload-')); + const files: Record = { + 'src/main/java/app/Instants.java': `package app; + +public class Instants { + public static Object toInstant(final Object instant) { + return toInstant(instant, null); + } + + public static Object toInstant(final Object instant, final Object defaultInstant) { + return instant != null ? instant : defaultInstant; + } + + public static int depth(final int n) { + return n <= 0 ? 0 : depth(n - 1); + } +} +`, + 'src/Json/Convert.cs': `namespace App +{ + public static class Convert + { + public static string DeserializeXNode(string value) + { + return DeserializeXNode(value, null); + } + + public static string DeserializeXNode(string value, string root) + { + return value + root; + } + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `callerLine -> targetLine` of the calls between same-named methods of a file. */ +const selfNamed = (file: string, name: string) => { + const nodes = cg.getNodesInFile(file).filter((n) => n.name === name); + const ids = new Set(nodes.map((n) => n.id)); + return cg.getOutgoingEdgesFrom([...ids]).filter((e) => e.kind === 'calls' && ids.has(e.target)) + .map((e) => `${cg.getNode(e.source)!.startLine} -> ${cg.getNode(e.target)!.startLine}`); +}; + +describe('a method calling its own name', () => { + it('Java: reaches the overload the arguments fit; real recursion stays', () => { + expect(selfNamed('src/main/java/app/Instants.java', 'toInstant')).toEqual(['4 -> 8']); + expect(selfNamed('src/main/java/app/Instants.java', 'depth')).toEqual(['12 -> 12']); + }); + + it('C#: reaches the overload the arguments fit', () => { + expect(selfNamed('src/Json/Convert.cs', 'DeserializeXNode')).toEqual(['5 -> 10']); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 866cdb41d1..e91a20ec40 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7962,7 +7962,44 @@ export function matchReference( ref: UnresolvedRef, context: ResolutionContext ): ResolvedRef | null { - return gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + return result ? retargetSelfOverload(result, ref, context) : result; +} + +/** Languages whose methods overload by arity. */ +const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'kotlin', 'swift', 'cpp', 'scala', 'dart', 'vbnet']); + +/** + * A call a method makes to its own name, with arguments its own parameters + * cannot take, is to another overload of it: Newtonsoft's + * `DeserializeXNode(value)` body `return DeserializeXNode(value, null);` + * bound to itself, so the two-argument overload never saw the one-argument + * one among its callers. The same-owner overload the argument count fits. + */ +function retargetSelfOverload(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef { + if (result.targetNodeId !== ref.fromNodeId || ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; + const self = context.getNodeById?.(ref.fromNodeId); + if (!self || (self.kind !== 'method' && self.kind !== 'function')) return result; + const name = self.name; + const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), name, context); + if (args === null) return result; + const argc = args.trim() === '' ? 0 : splitCppTopLevel(args).length; + const arity = (n: Node): { min: number; max: number } | null => { + const list = cppParenListAfter(n.filePath, n.startLine, 0, name, context); + if (list === null) return null; + const params = splitCppTopLevel(list).filter((p) => p !== '' && p !== 'void'); + const variadic = params.some((p) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))); + const min = params.filter((p) => !/=/.test(p) && !/\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))).length; + return { min, max: variadic ? Infinity : params.length }; + }; + const own = arity(self); + if (!own || (argc >= own.min && argc <= own.max)) return result; + const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); + const fits = context.getNodesInFile(self.filePath).filter((n) => + n.id !== self.id && n.name === name && (n.kind === 'method' || n.kind === 'function') && + n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner && + ((a) => a !== null && argc >= a.min && argc <= a.max)(arity(n))); + return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; } function matchReferenceInner( From 5f1a63571417799b9e57c05492f889e7a21dbc65 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:32:43 +0000 Subject: [PATCH 124/363] fix(resolution): any call reaches the overload its arguments fit (#2213) #2212 moved a method's call to its own name onto the sibling overload the arguments fit. The same holds for every caller: a match landing on an overload whose parameters cannot take the call's arguments moves to the one same-owner overload they fit - commons-lang's `HashCodeBuilder.reflectionHashCode(this)` from the three-parameter overload indexed first to the `(Object, String...)` one, Newtonsoft's `SetToken(token, value)` from `SetToken(token)`, halo's four-argument `menuItem(...)` from the one-argument `menuItem`. Overload sets and arities are read once per target node, and a target with no same-owner sibling costs one lookup, so indexing time is unchanged (commons-lang 4.67s vs 4.70s, Newtonsoft 4.04s vs 4.05s, wasm both arms). C++ keeps only the self-call case: its overload sets (templates, SFINAE priority tags, `data()` on any container) move targets unreliably. Targets moved between same-named overloads: commons-lang 2,334, Newtonsoft 697, halo 286, gson 232, okhttp 188, Alamofire 50; nlohmann json and fmt unchanged. Edge counts unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-overload-calls.test.ts | 37 +++++++++++++++--- src/resolution/name-matcher.ts | 56 +++++++++++++++++++-------- 3 files changed, 72 insertions(+), 22 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b0370b308e..13176aaf50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. - In a project with a Vue front end, the backend's calls no longer land on Vue components. mealie's Python `QueryFilterBuilder(…)` had been linking to the `QueryFilterBuilder.vue` component. The Vue rules (compiler macros, Nuxt auto-imports, components by name) now apply only to the app's scripts. diff --git a/__tests__/self-overload-calls.test.ts b/__tests__/self-overload-calls.test.ts index c0b121c5b9..d18b7c66e7 100644 --- a/__tests__/self-overload-calls.test.ts +++ b/__tests__/self-overload-calls.test.ts @@ -1,9 +1,9 @@ /** - * A method calling its own name with arguments its own parameters cannot - * take calls another overload — `toInstant(instant)` delegating to - * `toInstant(instant, Instant.EPOCH)` — not itself: the self-edge hid the - * convenience overloads from the full overload's callers (commons-lang had - * 180 such, Newtonsoft 107). + * A call that lands on an overload its arguments cannot fit is to the one + * sibling overload they do: `toInstant(instant)` delegating to + * `toInstant(instant, Instant.EPOCH)` bound to itself (commons-lang had 180 + * such self-edges, Newtonsoft 107), and `HashCodeBuilder.reflectionHashCode(this)` + * to the first overload indexed, a three-parameter one. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -32,6 +32,26 @@ public class Instants { return n <= 0 ? 0 : depth(n - 1); } } +`, + 'src/main/java/app/HashCodeBuilder.java': `package app; + +public class HashCodeBuilder { + public static int reflectionHashCode(final int initial, final int multiplier, final Object object) { + return initial * multiplier; + } + + public static int reflectionHashCode(final Object object, final String... excludeFields) { + return 1; + } +} +`, + 'src/main/java/app/Point.java': `package app; + +public class Point { + public int hash() { + return HashCodeBuilder.reflectionHashCode(this); + } +} `, 'src/Json/Convert.cs': `namespace App { @@ -76,6 +96,13 @@ describe('a method calling its own name', () => { expect(selfNamed('src/main/java/app/Instants.java', 'depth')).toEqual(['12 -> 12']); }); + it('from another class: reaches the overload the arguments fit, not the first indexed', () => { + const hash = cg.getNodesInFile('src/main/java/app/Point.java').find((n) => n.name === 'hash')!; + const lines = cg.getOutgoingEdges(hash.id).filter((e) => e.kind === 'calls').map((e) => cg.getNode(e.target)!) + .filter((t) => t.name === 'reflectionHashCode').map((t) => t.startLine); + expect(lines).toEqual([8]); + }); + it('C#: reaches the overload the arguments fit', () => { expect(selfNamed('src/Json/Convert.cs', 'DeserializeXNode')).toEqual(['5 -> 10']); }); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e91a20ec40..ac62250ce8 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -5257,6 +5257,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JVM_PACKAGES.delete(context); MINIFIED_SCRIPTS.delete(context); PY_LOCAL_BINDS.delete(context); + OVERLOAD_SETS.delete(context); PHP_FILE_SCOPES.delete(context); JAVA_STATIC_IMPORTS.delete(context); PY_IMPORTS.delete(context); @@ -7976,30 +7977,51 @@ const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'k * bound to itself, so the two-argument overload never saw the one-argument * one among its callers. The same-owner overload the argument count fits. */ +type Arity = { min: number; max: number } | null; +/** Per context: node id → its same-owner overloads with their arities (null: none). */ +const OVERLOAD_SETS = new WeakMap } | null>>(); + function retargetSelfOverload(result: ResolvedRef, ref: UnresolvedRef, context: ResolutionContext): ResolvedRef { - if (result.targetNodeId !== ref.fromNodeId || ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; - const self = context.getNodeById?.(ref.fromNodeId); - if (!self || (self.kind !== 'method' && self.kind !== 'function')) return result; - const name = self.name; - const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), name, context); + if (ref.referenceKind !== 'calls' || !OVERLOADING_LANGUAGES.has(ref.language)) return result; + // C++ overload sets (templates, SFINAE tags, a `data()` on any container) are + // only trusted for a method's call to itself. + if (ref.language === 'cpp' && result.targetNodeId !== ref.fromNodeId) return result; + let memo = OVERLOAD_SETS.get(context); + if (!memo) OVERLOAD_SETS.set(context, (memo = new Map())); + let set = memo.get(result.targetNodeId); + if (set === undefined) { + set = overloadSetOf(result.targetNodeId, context); + memo.set(result.targetNodeId, set); + } + // Only an overload set has a sibling to move to. + if (!set || set.own === null) return result; + const args = cppParenListAfter(ref.filePath, ref.line, Math.max(0, ref.column), set.name, context); if (args === null) return result; const argc = args.trim() === '' ? 0 : splitCppTopLevel(args).length; - const arity = (n: Node): { min: number; max: number } | null => { + if (argc >= set.own.min && argc <= set.own.max) return result; + const fits = set.siblings.filter((s) => s.arity !== null && argc >= s.arity.min && argc <= s.arity.max); + return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; +} + +/** A method's same-owner overloads and every one's arity, read from its declaration. */ +function overloadSetOf(id: string, context: ResolutionContext): { name: string; own: Arity; siblings: Array<{ id: string; arity: Arity }> } | null { + const self = context.getNodeById?.(id); + if (!self || (self.kind !== 'method' && self.kind !== 'function')) return null; + const name = self.name; + const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); + const siblings = (context.getNodesInFileNamed?.(self.filePath, name) ?? context.getNodesInFile(self.filePath).filter((n) => n.name === name)) + .filter((n) => n.id !== self.id && (n.kind === 'method' || n.kind === 'function') && + n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner); + if (siblings.length === 0) return null; + const arity = (n: Node): Arity => { const list = cppParenListAfter(n.filePath, n.startLine, 0, name, context); if (list === null) return null; const params = splitCppTopLevel(list).filter((p) => p !== '' && p !== 'void'); - const variadic = params.some((p) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))); - const min = params.filter((p) => !/=/.test(p) && !/\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, ''))).length; - return { min, max: variadic ? Infinity : params.length }; + const pack = (p: string) => /\.\.\.|\bparams\s|\bvararg\s/.test(p.replace(/<[^<>]*>/g, '')); + const min = params.filter((p) => !/=/.test(p) && !pack(p)).length; + return { min, max: params.some(pack) ? Infinity : params.length }; }; - const own = arity(self); - if (!own || (argc >= own.min && argc <= own.max)) return result; - const owner = self.qualifiedName.slice(0, Math.max(0, self.qualifiedName.lastIndexOf('::'))); - const fits = context.getNodesInFile(self.filePath).filter((n) => - n.id !== self.id && n.name === name && (n.kind === 'method' || n.kind === 'function') && - n.qualifiedName.slice(0, Math.max(0, n.qualifiedName.lastIndexOf('::'))) === owner && - ((a) => a !== null && argc >= a.min && argc <= a.max)(arity(n))); - return fits.length === 1 ? { ...result, targetNodeId: fits[0]!.id } : result; + return { name, own: arity(self), siblings: siblings.map((n) => ({ id: n.id, arity: arity(n) })) }; } function matchReferenceInner( From fa8d7284c5efe85ee86ede063967f95cd651ffb4 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 22:54:08 +0000 Subject: [PATCH 125/363] fix(rust): a bare name is in scope only through its module or a use (#2214) The Rust scope rule (#2124) only checked prelude names; every other bare name could mean any same-named project item, and a name after any path (`io::Result`) could mean any at all. The Rust framework resolver's PascalCase / handler / service patterns bound names with no scope check at 0.7-0.8. tokio's 1,221 `io::Result<...>` went to a private `runtime::task::Result` alias, 642 `Context<'_>` (from `use std::task::{Context, Poll}`) to `runtime::context::Context`. - A type or function from another file is in scope only through a `use` that binds it (its leaf, `as` alias, or `{self}`) or a glob over its module; methods and fields (reached through a value), import refs and module / file targets are unaffected. - A name a file imports from outside the project - `std` / `core` / `alloc`, or a crate its Cargo manifests declare as a dependency - is that outside item. (An inline `mod support { pub mod panic; }` is the project's.) - A path names its module: `seg::Name` reaches an item of module `seg`; a path rooted at std (`std::io::Error`) or at an outside import (`io::` under `use std::io`) reaches none; `crate::` / `self::` / `super::` and a project crate's name (`clap::Command`, re-exported) look it up as before. The path is read anywhere on the line, not only at the reference's column. - The framework resolver's patterns apply the same check. tokio -3008 +610, serde -792 +650, ripgrep -602 +247, axum -453 +186, clap -342 +32, bat -38 +9. Sampled losses: std / core / futures / syn / http / anyhow imports and paths, associated types (`S::Ok`, `S::Error`, `V::Value`, `Fut::Output`); gains are same-module items the old first match hid. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rust-type-scope.test.ts | 90 +++++++++++++++++++++++ src/resolution/frameworks/rust.ts | 17 ++++- src/resolution/name-matcher.ts | 116 ++++++++++++++++++++++++++++-- 4 files changed, 217 insertions(+), 7 deletions(-) create mode 100644 __tests__/rust-type-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 13176aaf50..219e0257b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. - In a project with a React front end, the React rules for hooks (`use…`) and contexts (`…Context`, `…Provider`) no longer apply to backend code. In halo, a Spring app with a React UI, Java imports such as `SecurityContext` and `ObjectProvider` had been handled as React contexts. diff --git a/__tests__/rust-type-scope.test.ts b/__tests__/rust-type-scope.test.ts new file mode 100644 index 0000000000..766252fc65 --- /dev/null +++ b/__tests__/rust-type-scope.test.ts @@ -0,0 +1,90 @@ +/** + * A bare Rust type or function from another file is in scope only through a + * `use` that binds it (or a glob over its module), and one the file imports + * from outside the project — `use std::task::{Context, Poll}` — is that + * outside item: tokio's 1,221 `io::Result<…>` bound to a private + * `runtime::task::Result` alias, its 642 `Context<'_>` to + * `runtime::context::Context`. A path names its module (`io::Result` is no + * `task` alias; `std::io::Error` is std's), a project crate's name + * (`clap::Command`) re-exports as `crate::` does, and an inline test module + * (`mod support { pub mod panic; }`) is the project's own. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rust-scope-')); + const files: Record = { + 'Cargo.toml': `[package]\nname = "app"\nversion = "0.1.0"\n\n[dependencies]\nfutures = "0.3"\n`, + 'src/lib.rs': `pub mod runtime;\npub mod io;\npub mod net;\n`, + 'src/runtime/mod.rs': `pub mod context;\npub mod task;\npub mod driver;\n`, + 'src/runtime/context.rs': `pub struct Context {\n pub depth: u8,\n}\n`, + 'src/runtime/task/mod.rs': `pub type Result = std::result::Result;\n`, + 'src/io/mod.rs': `pub mod copy;\n`, + 'src/io/copy.rs': `pub struct Error;\n`, + 'src/net.rs': `use std::io; +use std::task::{Context, Poll}; + +pub fn poll_ready(cx: &mut Context<'_>) -> Poll> { + let _ = cx; + Poll::Ready(Ok(())) +} + +pub fn fail() -> std::io::Error { + std::io::Error::other("x") +} +`, + 'src/runtime/driver.rs': `use crate::runtime::context::Context; + +pub fn enter(ctx: &Context) -> u8 { + ctx.depth +} +`, + 'tests/support/panic.rs': `pub fn test_panic() {}\n`, + 'tests/io_panic.rs': `mod support { + pub mod panic; +} +use support::panic::test_panic; + +#[test] +fn panics() { + test_panic(); +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.filePath); +}; + +describe('Rust names in scope', () => { + it('an outside import, or a path through one, is never a project item', () => { + const targets = targetsFrom('src/net.rs'); + expect(targets).not.toContain('src/runtime/context.rs'); + expect(targets).not.toContain('src/runtime/task/mod.rs'); + expect(targets).not.toContain('src/io/copy.rs'); + }); + + it('a crate-local use, and an inline test module, still reach the project item', () => { + expect(targetsFrom('src/runtime/driver.rs')).toContain('src/runtime/context.rs'); + expect(targetsFrom('tests/io_panic.rs')).toContain('tests/support/panic.rs'); + }); +}); diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index f4828a77ae..283c453521 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,6 +8,17 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; +import { isRustNameInScope } from '../name-matcher'; + +/** + * Whether the item a name heuristic found is one the reference can name: + * `Context<'_>` under `use std::task::{Context, Poll}` is std's, not tokio's + * `runtime::context::Context` (642 of them). + */ +function inRustScope(id: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + const node = context.getNodeById?.(id); + return !node || isRustNameInScope(node, ref, context); +} const cargoWorkspaceMapCache = new WeakMap>(); @@ -32,7 +43,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 1: Handler references if (ref.referenceName.endsWith('_handler') || ref.referenceName.startsWith('handle_')) { const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, HANDLER_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, @@ -45,7 +56,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 2: Service/Repository trait implementations if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository')) { const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, @@ -58,7 +69,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 3: Struct references (PascalCase) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { const result = resolveByNameAndKind(ref.referenceName, STRUCT_KINDS, MODEL_DIRS, context); - if (result) { + if (result && inRustScope(result, ref, context)) { return { original: ref, targetNodeId: result, diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index ac62250ce8..18473aeda7 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -13,6 +13,7 @@ import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-built import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; import { isTestPath } from '../search/query-utils'; import { isMinifiedContent } from '../extraction/generated-detection'; +import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; /** * Ceiling on how many same-named definitions a FUZZY name-match strategy will * score. A name defined more times than this is "ubiquitous" — a method/symbol @@ -3775,6 +3776,59 @@ interface RustUses { names: Set; /** `X` of each `use …::X::*` (`super` for `use super::*`). */ globs: Set; + /** Names the file imports from outside the project — `use std::task::{Context, Poll}`, `use futures::Stream`. */ + external: Set; + /** The items the file's project `use`s bind — their leaves, not the paths they walk. */ + bound: Set; +} + +const RUST_CRATES = new WeakMap>(); +const RUST_DEPENDENCIES = new WeakMap>(); + +/** + * The crates the project's manifests depend on (`[dependencies]`, + * `[dev-dependencies]`, `[build-dependencies]`, per-target ones), by the name + * code writes them (`futures_util`), less the project's own. + */ +function rustDependencyCrates(context: ResolutionContext): Set { + const hit = RUST_DEPENDENCIES.get(context); + if (hit) return hit; + const deps = new Set(); + const manifests = ['Cargo.toml', ...[...getCargoWorkspaceCrateMap(context).values()].map((dir) => `${dir}/Cargo.toml`)]; + for (const manifest of new Set(manifests)) { + const text = context.readFile(manifest) ?? ''; + let inDeps = false; + for (const raw of text.split(/\r?\n/)) { + const line = raw.replace(/#.*$/, '').trim(); + const header = /^\[([^\]]+)\]$/.exec(line); + if (header) { + const table = header[1]!.trim(); + // `[dependencies.tokio]` names one dependency in its header. + const named = /(?:^|\.)(?:dev-|build-)?dependencies\.([A-Za-z0-9_-]+)$/.exec(table); + if (named) deps.add(named[1]!.replace(/-/g, '_')); + inDeps = /(?:^|\.)(?:dev-|build-)?dependencies$/.test(table); + continue; + } + const key = inDeps ? /^([A-Za-z0-9_-]+)\s*=/.exec(line)?.[1] : undefined; + if (key) deps.add(key.replace(/-/g, '_')); + } + } + for (const own of rustProjectCrates(context)) deps.delete(own); + RUST_DEPENDENCIES.set(context, deps); + return deps; +} + +/** The project's own crate names (`tokio`, `tokio_util`), from its Cargo.toml files. */ +function rustProjectCrates(context: ResolutionContext): Set { + const hit = RUST_CRATES.get(context); + if (hit) return hit; + const crates = new Set(); + // The manifests are not indexed files: the root's package, and the workspace's members. + const root = /\[package\][^[]*?\bname\s*=\s*"([^"]+)"/.exec(context.readFile('Cargo.toml') ?? '')?.[1]; + if (root) crates.add(root.replace(/-/g, '_')); + for (const name of getCargoWorkspaceCrateMap(context).keys()) crates.add(name.replace(/-/g, '_')); + RUST_CRATES.set(context, crates); + return crates; } const RUST_USES = new WeakMap>(); @@ -3786,12 +3840,28 @@ function rustUsesOf(filePath: string, context: ResolutionContext): RustUses { } const hit = memo.get(filePath); if (hit) return hit; - const uses: RustUses = { names: new Set(), globs: new Set() }; + const uses: RustUses = { names: new Set(), globs: new Set(), external: new Set(), bound: new Set() }; + const leaves = (tree: string): string[] => [ + ...[...tree.matchAll(/([A-Za-z_]\w*)\s*(?=[,}]|$|\s+as\b)|\bas\s+([A-Za-z_]\w*)/g)] + .map((leaf) => leaf[2] ?? leaf[1]!).filter((id) => id !== 'self' && id !== 'as'), + // `use std::io::{self, Read}` binds `io` too. + ...[...tree.matchAll(/([A-Za-z_]\w*)\s*::\s*\{[^{}]*\bself\b/g)].map((m) => m[1]!), + ]; // Comments first: a doc comment's prose ("…use the Option…") is not a `use`. const text = stripCommentsForRegex(context.readFile(filePath) ?? '', 'rust'); + const dependencies = rustDependencyCrates(context); for (const m of text.matchAll(/(?:^|[;{}\s])use\s+([^;]{1,2000});/g)) { const tree = m[1]!; - if (/^\s*(?:::)?(?:std|core|alloc)\b/.test(tree)) continue; + const root = /^\s*(?:::)?([A-Za-z_]\w*)/.exec(tree)?.[1] ?? ''; + // Outside: the standard library or a crate the manifests depend on — not a + // module of the project's (`mod support { … }` inline in a test). + const outside = root === 'std' || root === 'core' || root === 'alloc' || (root !== '' && dependencies.has(root)); + // The items it binds: each leaf (`as` aliases by their alias), never the path it walks. + if (outside) { + for (const id of leaves(tree)) uses.external.add(id); + continue; + } + for (const id of leaves(tree)) uses.bound.add(id); for (const id of tree.matchAll(/[A-Za-z_]\w*/g)) uses.names.add(id[0]); for (const g of tree.matchAll(/(\w+)\s*::\s*(?:\{[^}]*)?\*/g)) uses.globs.add(g[1]!); } @@ -3829,7 +3899,27 @@ export function isRustNameInScope(candidate: Node, ref: UnresolvedRef, context: // Bare in the SOURCE: the index keeps `crate::error::Result` by its last // segment, and a path is not a prelude lookup. const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; - if (line !== undefined && line.startsWith(name, ref.column) && /::\s*$/.test(line.slice(0, ref.column))) return true; + // Written through a path on its line (`jsont::SubMatch { … }`, `io::Result<…>`), + // wherever the reference's column points. + const pathed = line === undefined ? null + : (line.startsWith(name, ref.column) && /::\s*$/.test(line.slice(0, ref.column)) ? /((?:[A-Za-z_]\w*\s*::\s*)*)([A-Za-z_]\w*)?\s*::\s*$/.exec(line.slice(0, ref.column)) + : !new RegExp(`(?= 0 ? candidate.qualifiedName.slice(0, cut).split('::').pop()! : ''; return (owner !== '' && uses.globs.has(owner)) || (uses.names.has(name) && uses.names.has(owner)); } - if (!RUST_PRELUDE.has(name) || candidate.filePath === ref.filePath) return true; + if (candidate.filePath === ref.filePath) return true; const uses = rustUsesOf(ref.filePath, context); + // `use std::task::{Context, Poll}`: the file's `Context` is std's, not tokio's + // `runtime::context::Context`. (A method call `.env(…)` is no imported name.) + if (uses.external.has(name) && !uses.bound.has(name) && line !== undefined) { + // Not on its line at all: a later link of a chain written across lines (`Arg::new(…)\n.env(…)`). + const at = new RegExp(`(?` is + // not `runtime::task::trace`'s `Context` unless the file brings that one + // in. A method is reached through a value, never a `use`. + if (TYPE_MEMBER_KINDS.has(candidate.kind) || ref.referenceKind === 'imports' || + candidate.kind === 'file' || candidate.kind === 'module' || candidate.kind === 'namespace') return true; + return uses.bound.has(name) || rustGlobCovers(uses, candidate, ref); + } return uses.names.has(name) || rustGlobCovers(uses, candidate, ref); } @@ -5217,6 +5323,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { C_STATIC_MEMO.delete(context); RUST_TRAIT_IMPL_MEMO.delete(context); RUST_USES.delete(context); + RUST_CRATES.delete(context); + RUST_DEPENDENCIES.delete(context); LEXICAL_SCOPE_MEMO.delete(context); KOTLIN_LAMBDA_RECEIVERS.delete(context); SCALA_IMPORTED_SUPERS.delete(context); From a8b578f7d8d4bd24502a9c10706a2657a21ba7d5 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:02:05 +0000 Subject: [PATCH 126/363] fix(csharp): a bare type name is one its namespaces can see (#2216) A bare C# type name could mean any same-named project type. Newtonsoft's `async Task` tests (`using System.Threading.Tasks;`) bound `Task` to a test class `Task` in `Newtonsoft.Json.Tests.Schema` 433 times; AutoMapper's DTOs in `OmmitedDTOModel3` extended `OmmitedDatabaseModel3`'s `BaseEntity`; Newtonsoft's `BindingFlags` went to its portable-build polyfill. A type in namespace N is now in reach of a file whose code runs in N or a namespace inside it, or that imports N: its `using N;`, any file's `global using N;`, the project files' ``, the SDK's implicit usings when `` is on (serilog's own `System.TimeProvider` polyfill is seen through them), or an alias naming the type (`using License = AutoMapper.Licensing.License;`). A type in the global namespace is seen everywhere; qualified names are not judged. Newtonsoft -529, AutoMapper -72, eShop -4; serilog, CleanArchitecture unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/csharp-namespace-visibility.test.ts | 86 +++++++++++++++++ src/resolution/name-matcher.ts | 96 +++++++++++++++++++ 3 files changed, 183 insertions(+) create mode 100644 __tests__/csharp-namespace-visibility.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 219e0257b4..f4dc07df9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. - A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. diff --git a/__tests__/csharp-namespace-visibility.test.ts b/__tests__/csharp-namespace-visibility.test.ts new file mode 100644 index 0000000000..8ffac2ff4d --- /dev/null +++ b/__tests__/csharp-namespace-visibility.test.ts @@ -0,0 +1,86 @@ +/** + * A bare C# type name means a type its file can see: one in its own + * namespace or an enclosing one, or in a namespace it imports (`using`, + * `global using`, the project's `` / implicit usings), or the one an + * alias names. Newtonsoft's `async Task` tests (`using + * System.Threading.Tasks;`) bound `Task` to a test class of that name in + * `Newtonsoft.Json.Tests.Schema` 433 times. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-ns-')); + const files: Record = { + 'src/App.Tests/Schema/Generator.cs': `namespace App.Tests.Schema +{ + public class Task + { + } +} +`, + 'src/App.Tests/Reader/ReadTests.cs': `using System.Threading.Tasks; + +namespace App.Tests.Reader +{ + public class ReadTests + { + public async Task Read() + { + await System.Threading.Tasks.Task.Yield(); + } + } +} +`, + 'src/App.Tests/Schema/Planner.cs': `namespace App.Tests.Schema +{ + public class Planner + { + public Task Next() { return new Task(); } + } +} +`, + 'src/App.Tests/Other/Aliased.cs': `using Task = App.Tests.Schema.Task; + +namespace App.Tests.Other +{ + public class Aliased + { + public Task Make() { return new Task(); } + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const reachesTask = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).some((e) => cg.getNode(e.target)!.qualifiedName === 'App.Tests.Schema::Task'); +}; + +describe('C# type names and namespaces', () => { + it('a type another namespace does not import is not in reach', () => { + expect(reachesTask('src/App.Tests/Reader/ReadTests.cs')).toBe(false); + }); + + it('the same namespace, and an alias naming the type, reach it', () => { + expect(reachesTask('src/App.Tests/Schema/Planner.cs')).toBe(true); + expect(reachesTask('src/App.Tests/Other/Aliased.cs')).toBe(true); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 18473aeda7..d9b6fb8148 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1317,6 +1317,8 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex // A test suite is not linked into the program: typeorm's `Record` is // not a test entity `Record`, tokio's `Output` not a `runtime/tests` type. if (isTestSuitePath(candidate.filePath) && !isTestPath(ref.filePath)) return false; + if (candidate.language === 'csharp' && ref.language === 'csharp' && CSHARP_TYPE_KINDS.has(candidate.kind) && + /^[A-Za-z_]\w*$/.test(ref.referenceName) && !isCsharpTypeVisible(candidate, ref, context)) return false; const lang = candidate.language as string; if (lang === 'c' || lang === 'cpp') { return ( @@ -3209,6 +3211,98 @@ function csharpProjectStaticUsings(dir: string, context: ResolutionContext, memo return owners; } +const CSHARP_NAMESPACE_SCOPES = new WeakMap; aliases: Map }>>(); +const CSHARP_PROJECT_USINGS = new WeakMap>>(); + +/** + * The namespaces a C# file's code runs in and the ones it imports: its + * `namespace` declarations, its `using X;`, every file's `global using X;` + * and the `` of the project files above it. + */ +function csharpNamespaceScope(file: string, context: ResolutionContext): { namespaces: string[]; usings: Set; aliases: Map } { + let memo = CSHARP_NAMESPACE_SCOPES.get(context); + if (!memo) CSHARP_NAMESPACE_SCOPES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const text = stripCommentsForRegex(context.readFile(file) ?? '', 'java'); + const namespaces = [...text.matchAll(/^\s*namespace\s+([\w.]+)/gm)].map((m) => m[1]!); + let projectMemo = CSHARP_PROJECT_USINGS.get(context); + if (!projectMemo) CSHARP_PROJECT_USINGS.set(context, (projectMemo = new Map())); + const usings = new Set(csharpProjectUsings(path.posix.dirname(file), context, projectMemo)); + const aliases = new Map(); + for (const m of text.matchAll(/^\s*(?:global\s+)?using\s+(?!static\b)(?:([A-Za-z_]\w*)\s*=\s*)?([\w.]+)\s*;/gm)) { + if (m[1]) aliases.set(m[1], m[2]!); + else usings.add(m[2]!); + } + const scope = { namespaces, usings, aliases }; + memo.set(file, scope); + return scope; +} + +/** `global using X;` from any file, and `` of the project files from `dir` up. */ +function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map>): Set { + const key = dir; + const hit = memo.get(key); + if (hit) return hit; + let usings: Set; + if (dir === '.' || dir === '' || dir === '/') { + usings = new Set(); + for (const f of context.getAllFiles()) { + if (!f.endsWith('.cs') || (context.fileContains && !context.fileContains(f, 'global using'))) continue; + for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); + } + } else { + usings = new Set(csharpProjectUsings(path.posix.dirname(dir), context, memo)); + } + let entries: string[] = []; + try { + entries = fs.readdirSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir)); + } catch { + entries = []; + } + for (const entry of entries) { + if (!/\.(?:csproj|props)$/i.test(entry)) continue; + let project = ''; + try { + project = fs.readFileSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir, entry), 'utf8'); + } catch { + continue; + } + for (const m of project.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); + // The SDK's implicit usings (serilog's own `System.TimeProvider` polyfill is seen through them). + if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(project)) { + for (const ns of CSHARP_IMPLICIT_USINGS) usings.add(ns); + } + } + memo.set(key, usings); + return usings; +} + +/** The namespaces `enable` imports into every file (Microsoft.NET.Sdk). */ +const CSHARP_IMPLICIT_USINGS: readonly string[] = [ + 'System', 'System.Collections.Generic', 'System.IO', 'System.Linq', 'System.Net.Http', 'System.Threading', 'System.Threading.Tasks', +]; + +/** + * Whether a bare C# type name can mean `n` — a type in namespace `N` is seen + * from `N` and the namespaces inside it, and through a `using N;` — + * Newtonsoft's `async Task` tests (`using System.Threading.Tasks;`) bound + * `Task` to a test class of that name in `Newtonsoft.Json.Tests.Schema`, 433 + * times. A type in the global namespace is seen everywhere. + */ +function isCsharpTypeVisible(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const cut = n.qualifiedName.indexOf('::'); + if (cut < 0) return true; + const ns = n.qualifiedName.slice(0, cut); + // A nested type (`Outer::Inner`) is judged by its outermost type's namespace. + const scope = csharpNamespaceScope(ref.filePath, context); + // `using License = AutoMapper.Licensing.License;` names that type, whatever the file's usings. + const aliased = scope.aliases.get(ref.referenceName); + if (aliased !== undefined) return aliased === `${ns}.${n.qualifiedName.slice(cut + 2).replace(/::/g, '.')}`; + if (scope.namespaces.some((own) => own === ns || own.startsWith(ns + '.'))) return true; + return scope.usings.has(ns); +} + const OBJC_SUPERS = new WeakMap>(); const OBJC_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field']); @@ -5357,6 +5451,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { OBJC_SUPERS.delete(context); CSHARP_SUPERS.delete(context); CSHARP_STATIC_USINGS.delete(context); + CSHARP_NAMESPACE_SCOPES.delete(context); + CSHARP_PROJECT_USINGS.delete(context); SCALA_SUPERS.delete(context); SCALA_IMPORTS.delete(context); ESM_EXPORT_LISTS.delete(context); From fa43e259b2bb19cca8f2469130ffe8c8e406acdc Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:07:39 +0000 Subject: [PATCH 127/363] fix(scala): a symbolic name in a type is a type, never an operator (#2217) A kind-projector placeholder (`Align[Either[A, *]]`, `Functor[Map[K, ?]]`) reached the resolver as a type reference named `*`, and exact or fuzzy matching bound it to the one `*` in the project - algebra's `Signed.Sign.*` operator - 567 times in cats. A symbolic name in a Scala type now resolves only to a type or type alias of that name (cats' `F ~> G`), and otherwise to nothing, whatever strategy runs. cats -567; sttp unchanged. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/scala-kind-projector.test.ts | 53 ++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 12 +++++- 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 __tests__/scala-kind-projector.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f4dc07df9b..02b0bd5273 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. - In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. diff --git a/__tests__/scala-kind-projector.test.ts b/__tests__/scala-kind-projector.test.ts new file mode 100644 index 0000000000..54d195712d --- /dev/null +++ b/__tests__/scala-kind-projector.test.ts @@ -0,0 +1,53 @@ +/** + * A Scala kind-projector placeholder in a type — `Align[Either[A, *]]`, + * `Functor[Map[K, ?]]` — names no method: cats' 567 `*` type arguments went + * to an algebra `Sign`'s `*` operator. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-scala-kp-')); + const files: Record = { + 'src/main/scala/algebra/Signed.scala': `package algebra + +object Signed { + sealed abstract class Sign(val toInt: Int) { + def *(that: Sign): Sign = this + } +} +`, + 'src/main/scala/cats/Align.scala': `package cats + +trait Align[F[_]] + +object Align { + implicit def catsAlignForEither[A]: Align[Either[A, *]] = null +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('Scala kind-projector placeholders', () => { + it('are not references to a `*` method', () => { + const ids = cg.getNodesInFile('src/main/scala/cats/Align.scala').map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references').map((e) => cg.getNode(e.target)!.name); + expect(targets).not.toContain('*'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index d9b6fb8148..5777451ec6 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -4308,7 +4308,9 @@ export function matchByExactName( // A Scala type position (`Arbitrary[B]`) never names a method: an // `implicit def A: Order[A]` shares its name with half of cats' type // parameters. Scala's value references only read a file's own vals. - !(ref.language === 'scala' && ref.referenceKind === 'references' && /^[A-Z]/.test(ref.referenceName) && + // Nor does a kind-projector placeholder (`F[*]`, `G[?]`): cats' 567 `*` + // type arguments went to an algebra `Sign`'s `*` method. + !(ref.language === 'scala' && ref.referenceKind === 'references' && /^(?:[A-Z]|[^\w\s]+$)/.test(ref.referenceName) && (n.kind === 'method' || n.kind === 'function')) && // Type/value references retain same-family eligibility: a native namesake // must not hide the actual web type. Calls still gate only the winner. @@ -8278,6 +8280,14 @@ function matchReferenceInner( return null; } + // A symbolic name in a Scala type is a type (`F ~> G`) or a kind-projector + // placeholder (`Either[A, *]`) — never an operator method, by any strategy. + if (ref.language === 'scala' && ref.referenceKind === 'references' && /^[^\w\s]+$/.test(ref.referenceName)) { + const types = context.getNodesByName(ref.referenceName).filter((n) => n.language === 'scala' && (SCALA_TYPE_KINDS.has(n.kind) || n.kind === 'type_alias')); + const chosen = types.length > 1 ? preferCallSiteFile(types, ref.filePath) : types; + return chosen.length === 1 ? { original: ref, targetNodeId: chosen[0]!.id, confidence: 0.8, resolvedBy: 'exact-match' } : null; + } + // A bare Lua call through a `local` alias reaches what the alias names. if ((ref.language === 'lua' || ref.language === 'luau') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName)) { const aliased = luaAliasTarget(ref, context); From e62032e95a5d0e025c022b587f090c4057810a3f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:18:21 +0000 Subject: [PATCH 128/363] fix(python): a member value through an untyped receiver is a method only its owner names (#2218) A lone project method now stands for an untyped receiver's member value only when the receiver is named after the method's class: netbox's `device=self.parent.device` model-field reads no longer reach a GraphQL filter's `device` method. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/member-function-ref-owner.test.ts | 61 +++++++++++++++++++++ src/resolution/name-matcher.ts | 10 +++- 3 files changed, 70 insertions(+), 2 deletions(-) create mode 100644 __tests__/member-function-ref-owner.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 02b0bd5273..ad3be10988 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. - Rust type and function names now follow Rust's scoping. A name from another file counts only when a `use` brings it in (or a `*` glob over its module). One imported from outside the project, like `use std::task::{Context, Poll}` or `use futures::Stream`, means that outside item. A path names its module: `io::Result` isn't an unrelated `Result` alias elsewhere, and `std::io::Error` is std's. On tokio, over 1,200 `io::Result` and 600 `Context` references had been landing on private project types. diff --git a/__tests__/member-function-ref-owner.test.ts b/__tests__/member-function-ref-owner.test.ts new file mode 100644 index 0000000000..697057da83 --- /dev/null +++ b/__tests__/member-function-ref-owner.test.ts @@ -0,0 +1,61 @@ +/** + * A member passed as a value through a receiver nothing types — + * `device=self.parent.device`, `self.check.last_ping` — is only the + * project's one method of that name when the receiver is named after its + * owner: netbox's model-field reads bound to a GraphQL filter's `device` + * method, healthchecks' `check.last_ping` to a transport's. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-member-fnref-')); + const files: Record = { + 'app/filters.py': `class FHRPGroupAssignmentFilter: + def device(self, queryset): + return queryset +`, + 'app/components.py': `def describe(parent): + return "{interface}".format(interface=parent, device=parent.device) +`, + 'app/recorder.py': `class HookRecorder: + def finish_recording(self): + return None + + +def register(request, hook_recorder): + request.addfinalizer(hook_recorder.finish_recording) +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const refsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references').map((e) => cg.getNode(e.target)!.qualifiedName); +}; + +describe('member values through an untyped receiver', () => { + it('are not a method whose owner the receiver is not named after', () => { + expect(refsFrom('app/components.py')).not.toContain('FHRPGroupAssignmentFilter::device'); + }); + + it('are a method whose owner the receiver is named after', () => { + expect(refsFrom('app/recorder.py')).toContain('HookRecorder::finish_recording'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 5777451ec6..888ac17ddb 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -325,8 +325,14 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): } } // Unknown receivers retain the old unique-or-drop discipline, across ALL - // files. Tests and abstract-looking bodies are candidates too. - return result(context.getNodesByName(member), 0.8); + // files. Tests and abstract-looking bodies are candidates too. A lone method + // stands only when the receiver is named after its owner: netbox's + // `device=self.parent.device` is a model field, not the project's one + // `device` method (a GraphQL filter's). A veto, never a filter — filtering + // first would promote some other lone match into a new guess. + const unique = result(context.getNodesByName(member), 0.8); + const target = unique ? context.getNodeById?.(unique.targetNodeId) : null; + return target && target.kind === 'method' && !sharesReceiverWord(receiverLink(receiver), target) ? null : unique; } function pythonRefClass(name: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { From 03123335b0e3ecffb44e627dd6a2205862d66271 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:33:28 +0000 Subject: [PATCH 129/363] fix(resolution): a name passed as a value is what is in scope where it is written (#2219) A function nested in another function is reachable only from inside it; a Python name the enclosing function binds is that local's value; a pytest fixture is a test's parameter only in its own module or under its conftest. Unreachable same-named functions still count against a lone cross-file guess. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/python-function-value-scope.test.ts | 121 ++++++++++++++++++ src/resolution/name-matcher.ts | 61 ++++++++- 3 files changed, 176 insertions(+), 7 deletions(-) create mode 100644 __tests__/python-function-value-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index ad3be10988..1b5dd83c83 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. - C# type names now follow namespaces. A type is in reach from its own namespace or one nested inside it, or through a `using` of its namespace. Also counted: a project-wide `global using`, the project's `` entries, the SDK's implicit usings, or an alias that names it. Before, Newtonsoft's `async Task` tests linked `Task` to a test class of that name in another namespace, and AutoMapper's DTOs linked to another namespace's `BaseEntity`. diff --git a/__tests__/python-function-value-scope.test.ts b/__tests__/python-function-value-scope.test.ts new file mode 100644 index 0000000000..d6152c78a0 --- /dev/null +++ b/__tests__/python-function-value-scope.test.ts @@ -0,0 +1,121 @@ +/** + * A name passed as a value means what is in scope where it is written. A + * function nested in another function is in scope only in there — httpx's + * `self._build_auth(auth)` passes its parameter, not the `auth` a test + * defines inside `test_custom_auth`. A Python name the function around it + * binds is that local's — `auth_flow(self, request)` handing `request` on is + * not the package's `request()`. And a pytest fixture is a test's parameter + * only in its own module or under its `conftest.py`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-fnvalue-')); + const files: Record = { + 'httpx/_client.py': `class Client: + @property + def auth(self): + return self._auth + + def _build_auth(self, auth): + return auth + + def set_auth(self, auth): + self._auth = self._build_auth(auth) +`, + 'httpx/_api.py': `def request(method, url): + return method, url +`, + 'httpx/_auth.py': `class Auth: + def auth_flow(self, request): + yield request + + def sync_auth_flow(self, request): + flow = self.auth_flow(request) + return flow + + +class Recorder: + def request(self): + return None +`, + 'tests/test_auth.py': `def test_custom_auth(): + def auth(request): + return request + + return register(auth) + + +def test_basic_auth(): + auth = ("user", "pass") + return register(auth) +`, + 'docs/example/test_order.py': `import pytest + + +@pytest.fixture +def func(): + return 1 +`, + 'tests/test_coverage.py': `import pytest + + +def _run_both(func): + return func() + + +@pytest.fixture( + scope="module", +) +def recipe(): + return {} + + +def test_recipe(recipe): + return use(recipe) +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string, qualifiedName?: string) => { + const ids = cg.getNodesInFile(file).filter((n) => !qualifiedName || n.qualifiedName === qualifiedName).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'references' || e.kind === 'calls') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('Python names passed as values', () => { + it('never reach a function nested in another function', () => { + expect(targetsFrom('httpx/_client.py')).not.toContain('tests/test_auth.py:test_custom_auth::auth'); + expect(targetsFrom('tests/test_auth.py', 'test_basic_auth')).not.toContain('tests/test_auth.py:test_custom_auth::auth'); + }); + + it('reach a nested function from inside its container', () => { + expect(targetsFrom('tests/test_auth.py', 'test_custom_auth')).toContain('tests/test_auth.py:test_custom_auth::auth'); + }); + + it('are the local a parameter binds', () => { + expect(targetsFrom('httpx/_auth.py')).not.toContain('httpx/_api.py:request'); + }); + + it('reach a fixture only from its own module', () => { + expect(targetsFrom('tests/test_coverage.py', '_run_both')).not.toContain('docs/example/test_order.py:func'); + expect(targetsFrom('tests/test_coverage.py', 'test_recipe')).toContain('tests/test_coverage.py:recipe'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 888ac17ddb..e4f45f48a7 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -523,7 +523,7 @@ export function matchFunctionRef( }; } - let candidates = context + const named = context .getNodesByName(ref.referenceName) .filter( (n) => @@ -533,7 +533,19 @@ export function matchFunctionRef( sameLanguageFamily(n.language, ref.language) && n.id !== ref.fromNodeId // a function registering itself is not a dependency edge ); + // A function declared inside another is in scope only in there: httpx's + // `self._build_auth(auth)` passes its own parameter, not the `auth` a test + // defines inside `test_custom_auth`. Those still count against a lone + // cross-file guess below — a name several functions use for themselves is + // as likely a local's. + let candidates = named.filter((n) => isLexicallyReachable(n, ref, context)); if (candidates.length === 0) return null; + // A Python name the function around it binds — a parameter, an assignment — + // is that local's value: httpx's `auth_flow(self, request)` handing `request` + // on is not the package's `request()` function. A pytest fixture is what a + // test's parameter of its name receives. + if (ref.language === 'python' && !candidates.some((n) => isFixtureInReach(n, ref.filePath, context)) && + isPythonLocallyBound(ref.referenceName, ref, context)) return null; // Swift implicit-self: a bare identifier can name a METHOD only of the // ENCLOSING type (`Button(action: handleTap)` written inside that type) — @@ -593,8 +605,11 @@ export function matchFunctionRef( } // Cross-file (imported names the import resolver didn't already claim): - // only an unambiguous match resolves. - if (candidates.length === 1) { + // only an unambiguous match resolves — or, in Python, the one in reach of + // a name the file imports (netbox's `sender=CustomField` beside a test's + // own nested `CustomField`). + if (candidates.length === 1 && (named.length === 1 || + (ref.language === 'python' && pythonFromImports(ref.filePath, context).has(ref.referenceName)))) { return { original: ref, targetNodeId: candidates[0]!.id, @@ -1557,7 +1572,7 @@ function fitsPythonCallShape(n: Node, shape: PythonCallShape, ref: UnresolvedRef if (isPythonNameImportedFromOutside(ref.referenceName, ref, context)) return false; // `view = UserView.as_view()` … `view(request)`: the file's own value — // unless it is a pytest fixture, which a test takes as a parameter of that name. - return isPytestFixture(n) || !isPythonLocallyBound(ref.referenceName, ref, context); + return isFixtureInReach(n, ref.filePath, context) || !isPythonLocallyBound(ref.referenceName, ref, context); } // A member of what the chain names: a method of a class of that name, or a // function / class in a module of that name (`helpers.slugify()`). @@ -1573,9 +1588,41 @@ function fitsPythonCallShape(n: Node, shape: PythonCallShape, ref: UnresolvedRef return stem === shape.owner || (stem === '__init__' && parts[parts.length - 2] === shape.owner); } -/** A pytest fixture: `@pytest.fixture` / `@fixture`, or anything a `conftest.py` defines. */ -function isPytestFixture(n: Node): boolean { - return /(?:^|\/)conftest\.py$/.test(n.filePath) || (n.decorators ?? []).some((d) => /(?:^|\.)fixture\b/.test(d)); +/** + * A pytest fixture: `@pytest.fixture` / `@fixture`, or anything a `conftest.py` + * defines. Python decorators are not kept on the node, so they are read from + * the lines above its `def` (a decorator's arguments may span lines). + */ +function isPytestFixture(n: Node, context: ResolutionContext): boolean { + if (/(?:^|\/)conftest\.py$/.test(n.filePath) || (n.decorators ?? []).some((d) => /(?:^|\.)fixture\b/.test(d))) return true; + return isDecoratedFixture(n, context); +} + +/** + * A fixture a test at `filePath` can take by name: one its own module defines, + * or one a `conftest.py` of its directory or a parent does. A test module's + * fixture is that module's alone — pytest's `_run_both(func)` is handing on its + * parameter, not a doc example's `func` fixture. + */ +function isFixtureInReach(n: Node, filePath: string, context: ResolutionContext): boolean { + if (n.filePath === filePath) return isPytestFixture(n, context); + const conftest = /^(.*?)(?:^|\/)conftest\.py$/.exec(n.filePath); + return conftest !== null && (conftest[1] === '' || filePath.startsWith(`${conftest[1]}/`)); +} + +function isDecoratedFixture(n: Node, context: ResolutionContext): boolean { + if (n.language !== 'python' || n.kind !== 'function') return false; + const lines = context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []; + // Upward through the decorator lines: each one starts with `@`, or sits inside one's parentheses. + let open = 0; + for (let i = n.startLine - 2; i >= 0 && i >= n.startLine - 16; i--) { + const text = lines[i]?.trim() ?? ''; + open += (text.match(/\)/g)?.length ?? 0) - (text.match(/\(/g)?.length ?? 0); + if (open > 0) continue; + if (!text.startsWith('@')) return false; + if (/^@(?:\w+\.)*fixture\b/.test(text)) return true; + } + return false; } const PY_LOCAL_BINDS = new WeakMap>(); From ff75a6ca5f6a3ccadae43ed55b1edc5304e80259 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Wed, 30 Sep 2026 23:56:34 +0000 Subject: [PATCH 130/363] fix(csharp): namespaces, usings and nested types follow the language's scopes (#2220) A block namespace qualifies only its own body (a nested one is Outer.Inner), mirrored in the kernel; a using names the project's namespace, never another file's using; a global using is its own project's; a nested type is reached by bare name only from inside its owner or a type deriving from it, through any partial part; visible C# types are filtered before ranking. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/csharp-namespace-scope.test.ts | 172 ++++++++++++++++++++ __tests__/fixtures/kernel-parity/Torture.cs | 5 + codegraph-kernel/src/csharp.rs | 75 ++++++--- src/extraction/languages/csharp.ts | 8 +- src/extraction/tree-sitter.ts | 26 +++ src/resolution/name-matcher.ts | 155 +++++++++++++++--- 7 files changed, 391 insertions(+), 51 deletions(-) create mode 100644 __tests__/csharp-namespace-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 1b5dd83c83..fbb2017ead 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. - In Scala, a kind-projector placeholder in a type, like `Align[Either[A, *]]`, is no longer linked to an operator method named `*`. A symbolic name in a type, like cats' `F ~> G`, now resolves only to a type. diff --git a/__tests__/csharp-namespace-scope.test.ts b/__tests__/csharp-namespace-scope.test.ts new file mode 100644 index 0000000000..b185d3f33d --- /dev/null +++ b/__tests__/csharp-namespace-scope.test.ts @@ -0,0 +1,172 @@ +/** + * C# names follow the scopes the language gives them: + * - a block `namespace X { … }` holds only its body — serilog's Guard.cs + * declares `static class Guard` after a `namespace JetBrains.Annotations` + * block, and a file's second namespace (or one nested inside another, + * `Outer.Inner`) is its own; + * - `using X;` names the project's namespace X, never another file's using; + * - a `global using` is its own project's — serilog's two test projects each + * `global using` their own `Support` namespace, and both define `Some`; + * - a nested type is reached by bare name only from inside its owner or a + * type deriving from it, through any partial part. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-csharp-ns-')); + const files: Record = { + 'src/Lib/Lib.csproj': '\n', + 'src/Lib/Guard.cs': `using JetBrains.Annotations; + +namespace JetBrains.Annotations +{ + sealed class NoEnumerationAttribute : System.Attribute { } +} + +static class Guard +{ + public static T AgainstNull(T value) where T : class => value; +} +`, + 'src/Lib/Multi.cs': `namespace Alpha +{ + class X { } +} + +namespace Beta +{ + class Y { } + + namespace Inner + { + class Z { } + } +} +`, + 'src/Lib/Uses.cs': `using Beta; + +namespace Gamma +{ + class Consumer { } +} +`, + 'src/Lib/Reader.cs': `namespace Lib +{ + public abstract class Reader + { + internal enum State { Start, Done } + } + + public partial class TextReader : Reader + { + } +} +`, + 'src/Lib/TextReader.Async.cs': `namespace Lib +{ + public partial class TextReader + { + object Peek() => State.Start; + } +} +`, + 'src/Lib/Writer.cs': `namespace Lib +{ + public abstract class Writer + { + internal enum State { Start, Done } + + object Current() => State.Done; + } +} +`, + 'test/Unit/Unit.csproj': '\n', + 'test/Unit/GlobalUsings.cs': 'global using Unit.Support;\n', + 'test/Unit/Support/Some.cs': `namespace Unit.Support; + +static class Some +{ + public static int InformationEvent() => 1; +} +`, + 'test/Unit/LogTests.cs': `namespace Unit.Context; + +class LogTests +{ + int Run() => Some.InformationEvent(); +} +`, + 'test/Unit/MappingTests.cs': `namespace Unit.Mapping; + +class FirstCase +{ + class Source { } + object Make() => new Source(); +} + +class SecondCase +{ + class Source { } + object Make() => new Source(); +} +`, + 'test/Perf/Perf.csproj': '\n', + 'test/Perf/GlobalUsings.cs': 'global using Perf.Support;\n', + 'test/Perf/Support/Some.cs': `namespace Perf.Support; + +static class Some +{ + public static int InformationEvent() => 2; +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const qualifiedNames = (file: string) => cg.getNodesInFile(file).filter((n) => n.kind !== 'file').map((n) => `${n.kind} ${n.qualifiedName}`); +const targetsAt = (file: string, line: number, kind?: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.line === line && (!kind || e.kind === kind)) + .map((e) => cg.getNode(e.target)!).map((t) => `${t.kind} ${t.filePath}:${t.qualifiedName}`); +}; + +describe('C# namespaces, usings and nested types', () => { + it('a block namespace qualifies only its own body', () => { + expect(qualifiedNames('src/Lib/Guard.cs')).toEqual(expect.arrayContaining([ + 'class JetBrains.Annotations::NoEnumerationAttribute', 'class Guard', 'method Guard::AgainstNull', + ])); + expect(qualifiedNames('src/Lib/Multi.cs')).toEqual(expect.arrayContaining([ + 'class Alpha::X', 'class Beta::Y', 'class Beta.Inner::Z', + ])); + }); + + it('a using names the project’s namespace', () => { + expect(targetsAt('src/Lib/Uses.cs', 1, 'imports')).toEqual(['namespace src/Lib/Multi.cs:Beta']); + }); + + it('a global using is its own project’s', () => { + expect(targetsAt('test/Unit/LogTests.cs', 5, 'calls')).toEqual(['method test/Unit/Support/Some.cs:Unit.Support::Some::InformationEvent']); + }); + + it('a nested type is its owner’s, or a derived type’s through any partial part', () => { + expect(targetsAt('test/Unit/MappingTests.cs', 12, 'instantiates')).toEqual(['class test/Unit/MappingTests.cs:Unit.Mapping::SecondCase::Source']); + expect(targetsAt('src/Lib/TextReader.Async.cs', 5)).toContain('enum src/Lib/Reader.cs:Lib::Reader::State'); + expect(targetsAt('src/Lib/Writer.cs', 7)).toContain('enum src/Lib/Writer.cs:Lib::Writer::State'); + }); +}); diff --git a/__tests__/fixtures/kernel-parity/Torture.cs b/__tests__/fixtures/kernel-parity/Torture.cs index 520b1d4e32..39e7743dbe 100644 --- a/__tests__/fixtures/kernel-parity/Torture.cs +++ b/__tests__/fixtures/kernel-parity/Torture.cs @@ -179,3 +179,8 @@ namespace Torture.Beta { public class Other { } } + +static class TopLevelGuard +{ + public static T AgainstNull(T value) where T : class => value; +} diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index aec1656f82..7d481662a7 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -205,34 +205,21 @@ pub fn extract(file_path: &str, source: &str) -> Result { w.node_ids.push(ids::file_node_id(file_path)); w.stack.push(Scope { row: 0, kind: "file", name: base_name.to_string() }); - // extractFilePackage: the FIRST top-level namespace declaration mints ONE - // `namespace` node that stays pushed for the ENTIRE file — a second - // top-level namespace's types nest under the first's node/QN, nested - // namespaces leave no trace, and every import ref in a namespaced file - // hangs off this node (checklist §namespace). + // extractFilePackage: a top-level file-scoped `namespace X;` mints ONE + // `namespace` node that stays pushed for the ENTIRE file, so every import + // ref in the file hangs off it (checklist §namespace). A block namespace + // scopes only its own body — visit_node. let root = tree.root_node(); let mut pkg_pushed = false; for i in 0..root.named_child_count() { let Some(child) = root.named_child(i) else { continue }; - if child.kind() != "namespace_declaration" - && child.kind() != "file_scoped_namespace_declaration" - { + if child.kind() != "file_scoped_namespace_declaration" { continue; } - // csharpExtractor.extractPackage: `name` field ?? first - // qualified_name/identifier named child. No trim. - let name_node = child.child_by_field_name("name").or_else(|| { - (0..child.named_child_count()) - .filter_map(|j| child.named_child(j)) - .find(|c| matches!(c.kind(), "qualified_name" | "identifier")) - }); - if let Some(name_node) = name_node { - let pkg = w.text(name_node).to_string(); - if !pkg.is_empty() { - if let Some(row) = w.create_node("namespace", &pkg, child, Extra::default()) { - w.stack.push(Scope { row, kind: "namespace", name: pkg }); - pkg_pushed = true; - } + if let Some(pkg) = namespace_name(&w, child) { + if let Some(row) = w.create_node("namespace", &pkg, child, Extra::default()) { + w.stack.push(Scope { row, kind: "namespace", name: pkg }); + pkg_pushed = true; } } break; @@ -265,6 +252,18 @@ fn record_is_struct(node: Node) -> bool { .any(|c| c.kind() == "struct") } +/// csharpExtractor.extractPackage: `name` field ?? first qualified_name / +/// identifier named child. No trim; None when empty. +fn namespace_name(w: &Walker, node: Node) -> Option { + let name_node = node.child_by_field_name("name").or_else(|| { + (0..node.named_child_count()) + .filter_map(|j| node.named_child(j)) + .find(|c| matches!(c.kind(), "qualified_name" | "identifier")) + })?; + let name = w.text(name_node); + (!name.is_empty()).then(|| name.to_string()) +} + impl<'t> Walker<'t> { fn text(&self, node: Node) -> &'t str { &self.src[node.byte_range()] @@ -518,6 +517,38 @@ impl<'t> Walker<'t> { let kind = node.kind(); let mut skip_children = false; + // A block namespace scopes only its body (tree-sitter.ts visitNode): + // one written inside another is `Outer.Inner`, taking the outer's + // place on the stack while its body is walked. + if kind == "namespace_declaration" { + if let Some(ns_name) = namespace_name(self, node) { + let outer = match self.stack.last() { + Some(top) if top.kind == "namespace" => self.stack.pop(), + _ => None, + }; + let full = match &outer { + Some(o) => format!("{}.{}", o.name, ns_name), + None => ns_name, + }; + let row = self.create_node("namespace", &full, node, Extra::default()); + if let Some(row) = row { + self.stack.push(Scope { row, kind: "namespace", name: full }); + } + for i in 0..node.named_child_count() { + if let Some(c) = node.named_child(i) { + self.visit_node(c); + } + } + if row.is_some() { + self.stack.pop(); + } + if let Some(o) = outer { + self.stack.push(o); + } + return; + } + } + self.maybe_capture_fn_refs(node); if kind == "class_declaration" || kind == "record_declaration" { diff --git a/src/extraction/languages/csharp.ts b/src/extraction/languages/csharp.ts index a43539a7fe..a88b01f817 100644 --- a/src/extraction/languages/csharp.ts +++ b/src/extraction/languages/csharp.ts @@ -82,10 +82,10 @@ export const csharpExtractor: LanguageExtractor = { typeAliasTypes: [], // Namespaces qualify type names so same-named types in different namespaces are // distinguishable (e.g. `ApplicationCore.Entities.CatalogBrand` vs - // `BlazorShared.Models.CatalogBrand`). Both block (`namespace Foo { … }`, which - // nests its types) and file-scoped (`namespace Foo;`) forms — extractFilePackage - // pushes the namespace onto the scope so nested/top-level types pick it up. - packageTypes: ['namespace_declaration', 'file_scoped_namespace_declaration'], + // `BlazorShared.Models.CatalogBrand`). A file-scoped `namespace Foo;` covers the + // whole file (extractFilePackage); a block `namespace Foo { … }` only its body + // (the visitor scopes it). + packageTypes: ['file_scoped_namespace_declaration'], extractPackage: (node: SyntaxNode, source: string) => { const name = node.childForFieldName('name') ?? diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 51086c3bfd..7c9baa69d0 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -1140,6 +1140,32 @@ export class TreeSitterExtractor { } } + // C# block namespaces scope only their own body: serilog's Guard.cs opens + // `namespace JetBrains.Annotations { … }` and then declares `static class + // Guard` at the top level, and a file's second namespace is its own. A + // namespace written inside another is `Outer.Inner` — the dotted name a + // type's qualifiedName leads with — so it takes the outer's place on the + // scope while its body is walked. (A file-scoped `namespace X;` covers + // the whole file: extractFilePackage.) Mirrored in the kernel (csharp.rs). + if (this.language === 'csharp' && nodeType === 'namespace_declaration') { + const nsName = this.extractor.extractPackage?.(node, this.source); + if (nsName) { + const topId = this.nodeStack[this.nodeStack.length - 1]; + const top = this.nodes.find((n) => n.id === topId); + const outer = top?.kind === 'namespace' ? top : null; + if (outer) this.nodeStack.pop(); + const ns = this.createNode('namespace', outer ? `${outer.name}.${nsName}` : nsName, node); + if (ns) this.nodeStack.push(ns.id); + for (let i = 0; i < node.namedChildCount; i++) { + const child = node.namedChild(i); + if (child) this.visitNode(child); + } + if (ns) this.nodeStack.pop(); + if (outer) this.nodeStack.push(outer.id); + return; + } + } + // Function-as-value capture (#756) — independent of the dispatch ladder // below (the captured container types have no other handler there), so it // can never shadow or be shadowed by an extraction branch. diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e4f45f48a7..83dbda2453 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1339,7 +1339,8 @@ export function isVisibleAcrossFiles(candidate: Node, ref: UnresolvedRef, contex // not a test entity `Record`, tokio's `Output` not a `runtime/tests` type. if (isTestSuitePath(candidate.filePath) && !isTestPath(ref.filePath)) return false; if (candidate.language === 'csharp' && ref.language === 'csharp' && CSHARP_TYPE_KINDS.has(candidate.kind) && - /^[A-Za-z_]\w*$/.test(ref.referenceName) && !isCsharpTypeVisible(candidate, ref, context)) return false; + /^[A-Za-z_]\w*$/.test(ref.referenceName) && + (!isCsharpTypeVisible(candidate, ref, context) || !isCsharpNestedTypeInScope(candidate, ref, context))) return false; const lang = candidate.language as string; if (lang === 'c' || lang === 'cpp') { return ( @@ -3265,12 +3266,13 @@ function csharpProjectStaticUsings(dir: string, context: ResolutionContext, memo } const CSHARP_NAMESPACE_SCOPES = new WeakMap; aliases: Map }>>(); -const CSHARP_PROJECT_USINGS = new WeakMap>>(); +const CSHARP_PROJECT_USINGS = new WeakMap; project: boolean }>>(); /** * The namespaces a C# file's code runs in and the ones it imports: its - * `namespace` declarations, its `using X;`, every file's `global using X;` - * and the `` of the project files above it. + * `namespace` declarations, its `using X;`, the `global using X;` of its + * project's files (every file's, outside any project) and the + * `` of the project files above it. */ function csharpNamespaceScope(file: string, context: ResolutionContext): { namespaces: string[]; usings: Set; aliases: Map } { let memo = CSHARP_NAMESPACE_SCOPES.get(context); @@ -3281,7 +3283,9 @@ function csharpNamespaceScope(file: string, context: ResolutionContext): { names const namespaces = [...text.matchAll(/^\s*namespace\s+([\w.]+)/gm)].map((m) => m[1]!); let projectMemo = CSHARP_PROJECT_USINGS.get(context); if (!projectMemo) CSHARP_PROJECT_USINGS.set(context, (projectMemo = new Map())); - const usings = new Set(csharpProjectUsings(path.posix.dirname(file), context, projectMemo)); + const project = csharpProjectUsings(path.posix.dirname(file), context, projectMemo); + const usings = new Set(project.usings); + if (!project.project) for (const u of csharpGlobalUsings('.', context)) usings.add(u); const aliases = new Map(); for (const m of text.matchAll(/^\s*(?:global\s+)?using\s+(?!static\b)(?:([A-Za-z_]\w*)\s*=\s*)?([\w.]+)\s*;/gm)) { if (m[1]) aliases.set(m[1], m[2]!); @@ -3292,42 +3296,65 @@ function csharpNamespaceScope(file: string, context: ResolutionContext): { names return scope; } -/** `global using X;` from any file, and `` of the project files from `dir` up. */ -function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map>): Set { +/** + * The `` of the project files from `dir` up, and the + * `global using X;` of each project's own files — a global using is its + * project's alone: serilog's Serilog.Tests and Serilog.PerformanceTests each + * `global using` their own `Support` namespace, and both define `Some`. + * `project` says whether a `.csproj` sits at `dir` or above it. + */ +function csharpProjectUsings(dir: string, context: ResolutionContext, memo: Map; project: boolean }>): { usings: Set; project: boolean } { const key = dir; const hit = memo.get(key); if (hit) return hit; - let usings: Set; - if (dir === '.' || dir === '' || dir === '/') { - usings = new Set(); - for (const f of context.getAllFiles()) { - if (!f.endsWith('.cs') || (context.fileContains && !context.fileContains(f, 'global using'))) continue; - for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); - } - } else { - usings = new Set(csharpProjectUsings(path.posix.dirname(dir), context, memo)); - } + const root = dir === '.' || dir === '' || dir === '/'; + const parent = root ? null : csharpProjectUsings(path.posix.dirname(dir), context, memo); + const usings = new Set(parent?.usings ?? []); + let project = parent?.project ?? false; let entries: string[] = []; try { - entries = fs.readdirSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir)); + entries = fs.readdirSync(path.join(context.getProjectRoot(), root ? '' : dir)); } catch { entries = []; } for (const entry of entries) { if (!/\.(?:csproj|props)$/i.test(entry)) continue; - let project = ''; + let text = ''; try { - project = fs.readFileSync(path.join(context.getProjectRoot(), dir === '.' ? '' : dir, entry), 'utf8'); + text = fs.readFileSync(path.join(context.getProjectRoot(), root ? '' : dir, entry), 'utf8'); } catch { continue; } - for (const m of project.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); + if (/\.csproj$/i.test(entry) && !project) { + project = true; + for (const u of csharpGlobalUsings(root ? '.' : dir, context)) usings.add(u); + } + for (const m of text.matchAll(/]*\bStatic\s*=\s*"true")[^>]*>/gi)) usings.add(m[1]!); // The SDK's implicit usings (serilog's own `System.TimeProvider` polyfill is seen through them). - if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(project)) { + if (/\s*(?:enable|true)\s*<\/ImplicitUsings>/i.test(text)) { for (const ns of CSHARP_IMPLICIT_USINGS) usings.add(ns); } } - memo.set(key, usings); + const result = { usings, project }; + memo.set(key, result); + return result; +} + +const CSHARP_GLOBAL_USINGS = new WeakMap>>(); + +/** The `global using X;` of the `.cs` files under `dir` (`.` = the whole repository). */ +function csharpGlobalUsings(dir: string, context: ResolutionContext): Set { + let memo = CSHARP_GLOBAL_USINGS.get(context); + if (!memo) CSHARP_GLOBAL_USINGS.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + const usings = new Set(); + const prefix = dir === '.' ? '' : `${dir}/`; + for (const f of context.getAllFiles()) { + if (!f.endsWith('.cs') || !f.startsWith(prefix) || (context.fileContains && !context.fileContains(f, 'global using'))) continue; + for (const m of (context.readFile(f) ?? '').matchAll(/^\s*global\s+using\s+(?!static\b)([\w.]+)\s*;/gm)) usings.add(m[1]!); + } + memo.set(dir, usings); return usings; } @@ -3356,6 +3383,60 @@ function isCsharpTypeVisible(n: Node, ref: UnresolvedRef, context: ResolutionCon return scope.usings.has(ns); } +/** + * Whether a bare C# name can reach `n` as a nested type: only from inside the + * type that declares it (any partial part, any depth) or a class deriving from + * it. AutoMapper's tests each declare their own nested `Source`, and a + * same-file `new Source()` went to whichever test class came first. + */ +function isCsharpNestedTypeInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return true; + const owner = n.qualifiedName.slice(0, cut); + const ownerType = context.getNodesInFile(n.filePath).find((p) => CSHARP_TYPE_KINDS.has(p.kind) && p.qualifiedName === owner); + // Declared in a namespace, not a type. + if (!ownerType) return true; + const enclosing = context.getNodesInFile(ref.filePath) + .filter((p) => CSHARP_TYPE_KINDS.has(p.kind) && p.startLine <= ref.line && p.endLine >= ref.line); + if (enclosing.some((p) => p.qualifiedName === owner || p.qualifiedName.startsWith(`${owner}::`))) return true; + // Inherited: a type around the ref derives from the owner (`class SourceA : + // Source`), through any partial part — Newtonsoft's JsonTextReader.Async.cs + // is `partial class JsonTextReader` with no base list, reading JsonReader's `State`. + return enclosing.some((p) => csharpAncestorNames(p.qualifiedName, context).has(ownerType.name)); +} + +const CSHARP_ANCESTORS = new WeakMap>>(); + +/** The simple names of the C# types `qn` derives from, through every partial part and base, a few levels up. */ +function csharpAncestorNames(qn: string, context: ResolutionContext, depth = 0): Set { + let memo = CSHARP_ANCESTORS.get(context); + if (!memo) CSHARP_ANCESTORS.set(context, (memo = new Map())); + const hit = memo.get(qn); + if (hit) return hit; + const names = new Set(); + memo.set(qn, names); // a cycle reads what is gathered so far + for (const decl of context.getNodesByQualifiedName(qn)) { + if (decl.language !== 'csharp' || !CSHARP_TYPE_KINDS.has(decl.kind)) continue; + const lines = context.getFileLines?.(decl.filePath) ?? context.readFile(decl.filePath)?.split(/\r?\n/) ?? []; + let header = ''; + for (let i = decl.startLine - 1; i < Math.min(lines.length, decl.startLine + 6) && !header.includes('{'); i++) header += `${lines[i] ?? ''} `; + const list = /:\s*([^{;]*)/.exec(header.split('{')[0]!.replace(/\bwhere\b[\s\S]*$/, ''))?.[1] ?? ''; + for (const base of splitCppTopLevel(list)) { + const name = /([A-Za-z_]\w*)\s*(?:<.*)?$/.exec(base.trim())?.[1]; + if (name) names.add(name); + } + } + if (depth < 4) { + for (const base of [...names]) { + for (const t of context.getNodesByName(base)) { + if (t.language !== 'csharp' || !CSHARP_TYPE_KINDS.has(t.kind) || t.qualifiedName === qn) continue; + for (const up of csharpAncestorNames(t.qualifiedName, context, depth + 1)) names.add(up); + } + } + } + return names; +} + const OBJC_SUPERS = new WeakMap>(); const OBJC_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field']); @@ -4395,6 +4476,13 @@ export function matchByExactName( (!importRef || isImportableKind(n.kind)) && // Nested locals are only reachable from inside their container (#1230). isLexicallyReachable(n, ref, context) && + // A C# type name is a type its namespaces can see — ahead of the ranking, + // so a visible namesake wins where the veto after it would drop the + // ref: eShop's `WebhookType.OrderPaid` under `using Webhooks.API.Model;`. + // A nested type, only from inside its owner: AutoMapper's same-file `new + // Source()` in one test class is not the previous test class's `Source`. + !(ref.language === 'csharp' && n.language === 'csharp' && CSHARP_TYPE_KINDS.has(n.kind) && /^[A-Za-z_]\w*$/.test(ref.referenceName) && + (!isCsharpNestedTypeInScope(n, ref, context) || (n.filePath !== ref.filePath && !isCsharpTypeVisible(n, ref, context)))) && // Preserve import ranking; calls reject the winner without promoting another. (!importRef || n.filePath === ref.filePath || !ESM_FAMILY.has(n.language) || !isSealedModule(n.filePath, context)) && @@ -4474,7 +4562,14 @@ export function matchByQualifiedName( ) : nodes; - const candidates = keepForRef(context.getNodesByQualifiedName(ref.referenceName)); + let candidates = keepForRef(context.getNodesByQualifiedName(ref.referenceName)); + // A C# `using X.Y;` names a namespace: one the project declares, else it is + // the file's own (external) using — never another file's using of that name. + if (ref.language === 'csharp' && ref.referenceKind === 'imports') { + const namespaces = candidates.filter((n) => n.kind === 'namespace'); + candidates = namespaces.length > 0 ? preferCallSiteFile(namespaces, ref.filePath).slice(0, 1) + : candidates.filter((n) => n.kind !== 'import' || n.filePath === ref.filePath); + } if (candidates.length === 1) { return { @@ -5508,6 +5603,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { CSHARP_STATIC_USINGS.delete(context); CSHARP_NAMESPACE_SCOPES.delete(context); CSHARP_PROJECT_USINGS.delete(context); + CSHARP_GLOBAL_USINGS.delete(context); + CSHARP_ANCESTORS.delete(context); SCALA_SUPERS.delete(context); SCALA_IMPORTS.delete(context); ESM_EXPORT_LISTS.delete(context); @@ -6756,10 +6853,18 @@ export function matchMethodCall( // own file first — otherwise the first-indexed class wins and a call in `b/` // resolves to `a/`'s method (#1079). const strat1 = nmTimedT('mc-class', ref, (): ResolvedRef | null => { - const classCandidates = preferCallSiteFile( + let classCandidates = preferCallSiteFile( context.getNodesByName(objectOrClass!).filter(isMethodOwnerKind), ref.filePath, ); + // A C# class the call's namespaces can see before one they can't: + // serilog's `Some.InformationEvent()` in Serilog.Tests is its own + // Support namespace's `Some`, not the performance tests'. + if (ref.language === 'csharp' && classCandidates.length > 1) { + const typeRef = { ...ref, referenceName: objectOrClass! }; + const visible = classCandidates.filter((c) => c.language !== 'csharp' || isCsharpTypeVisible(c, typeRef, context)); + classCandidates = [...visible, ...classCandidates.filter((c) => !visible.includes(c))]; + } for (const classNode of classCandidates) { // Skip cross-language class matches From ff190b30d8112f4aa6a5848138831468495cee0a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:09:37 +0000 Subject: [PATCH 131/363] fix(resolution): a method handing its call on to another object is not calling itself (#2221) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A TS/JS call written through this.… or window.… reaches the resolver by its bare name; it resolves to the calling method only when the field is declared as the caller's own class. A name-scored receiver guess never lands on the calling method. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-call-delegation.test.ts | 99 ++++++++++++++++++++++++++ src/resolution/name-matcher.ts | 56 ++++++++++++++- 3 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 __tests__/self-call-delegation.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index fbb2017ead..e2b990520e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. - In Python, an attribute passed as a value through an object nothing types, like netbox's `device=self.parent.device`, is no longer linked to the project's only method of that name when the object isn't named after the method's class. Before, netbox's model-field reads went to a GraphQL filter's `device` method, and healthchecks' `check.last_ping` went to a notification transport's. diff --git a/__tests__/self-call-delegation.test.ts b/__tests__/self-call-delegation.test.ts new file mode 100644 index 0000000000..d23bcae87d --- /dev/null +++ b/__tests__/self-call-delegation.test.ts @@ -0,0 +1,99 @@ +/** + * A method that hands its call on to another object — a wrapper — is not + * calling itself. TS/JS calls through `this.…` or `window.…` reach + * the resolver by their bare name, so BookStack's `toggle()` doing + * `this.container.classList.toggle('open')` and `listen()` doing + * `window.$events.listen(…)` bound to themselves; a guess from a receiver's + * name alone (`FileStorage::delete` doing `$storage->delete($path)`) did too. + * A recursion through a field the class declares as its own type stays. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-self-call-')); + const files: Record = { + 'resources/js/editor-toolbox.ts': `export class EditorToolbox { + container: HTMLElement; + + toggle(): void { + this.container.classList.toggle('open'); + this.toggle(); + } +} +`, + 'resources/js/tree.ts': `export class TreeNode { + left: TreeNode | null = null; + + insert(v: number): void { + this.left.insert(v); + } +} +`, + 'resources/js/common-events.js': `export function listen(editor) { + window.$events.listen('editor::replace', () => editor.reset()); +} +`, + 'app/Uploads/FileStorage.php': `getStorageDisk(); + $storage->delete($path); + } + + protected function getStorageDisk() + { + return null; + } +} +`, + 'app/Uploads/ImageStorage.php': ` { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const selfCallLines = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'calls' && e.source === e.target).map((e) => e.line); +}; + +describe('a call a method hands on', () => { + it('through a field of another type is not the method itself', () => { + expect(selfCallLines('resources/js/editor-toolbox.ts')).toEqual([6]); + expect(selfCallLines('resources/js/common-events.js')).toEqual([]); + }); + + it('through a field of the class’s own type is a recursion', () => { + expect(selfCallLines('resources/js/tree.ts')).toEqual([5]); + }); + + it('through a receiver named like the caller’s class is not the method itself', () => { + expect(selfCallLines('app/Uploads/FileStorage.php')).toEqual([]); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 83dbda2453..e12f46169f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1402,6 +1402,52 @@ export const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfm * enclosing method itself least of all, which the same-file proximity term * used to pick over the module-scope function the call actually means. */ +/** + * The links between `this` / `super` / `window` and the method of a TS/JS + * call the extractor recorded by its bare name — `this.container.classList + * .toggle()` reaches the resolver as `toggle`, with links `container`, + * `classList`. Null for a call not written that way, `this.toggle()` included. + */ +function collapsedJsChain(ref: UnresolvedRef, context: ResolutionContext): { root: string; links: string[] } | null { + if (!JS_FAMILY.has(ref.language) || ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); + if (!lines) return null; + const text = lines.slice(ref.line - 1, ref.line + 3).join('\n').slice(ref.column); + const name = ref.referenceName.replace(/\$/g, '\\$'); + const m = new RegExp(`^(this|super|window)((?:\\s*\\??\\.\\s*#?[\\w$]+(?:\\([^()]*\\))?)+?)\\s*\\??\\.\\s*${name}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text); + if (!m) return null; + return { root: m[1]!, links: m[2]!.split('.').map((l) => l.replace(/[\s?]/g, '')).filter((l) => l !== '') }; +} + +/** + * Whether a collapsed `this..m()` inside `m` can be `m` itself: only + * when the class declares the field as its own type — a tree node's + * `this.left.insert(v)` — never through a field of another type + * (`this.editor.input.focus()`, `this.container.classList.toggle()`). + */ +function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { + const chain = collapsedJsChain(ref, context); + return chain !== null && !isCollapsedSelfRecursion(chain, ref, context); +} + +function isCollapsedSelfRecursion(chain: { root: string; links: string[] }, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (chain.root !== 'this' || chain.links.length !== 1 || chain.links[0]!.includes('(')) return false; + const field = chain.links[0]!.replace(/^#/, ''); + const caller = context.getNodeById?.(ref.fromNodeId); + const cut = caller ? caller.qualifiedName.lastIndexOf('::') : -1; + if (!caller || cut <= 0) return false; + const owner = caller.qualifiedName.slice(0, cut).split('::').pop()!; + const cls = context.getNodesInFile(ref.filePath).find((n) => + n.kind === 'class' && n.name === owner && n.startLine <= ref.line && n.endLine >= ref.line); + if (!cls) return false; + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; + const body = lines.slice(cls.startLine - 1, cls.endLine).join('\n'); + const f = field.replace(/\$/g, '\\$'); + const declared = new RegExp(`(?:^|[\\s(,])#?${f}\\s*[?!]?\\s*:\\s*([A-Za-z_$][\\w$]*)`, 'm').exec(body)?.[1] ?? + new RegExp(`\\bthis\\.${f}\\s*=\\s*new\\s+([A-Za-z_$][\\w$]*)`).exec(body)?.[1]; + return declared === owner; +} + function isBareJsCall(ref: UnresolvedRef, context: ResolutionContext): boolean { return JS_FAMILY.has(ref.language) && isReceiverLessCall(ref, context); } @@ -7101,7 +7147,11 @@ export function matchMethodCall( } } - if (bestMatch && bestScore >= 2) { + // A wrapper handing its call on — BookStack's `FileStorage::delete` doing + // `$storage->delete($path)`, `CommentRepo::delete` doing + // `$comment->delete()` — names the caller's own class only by a shared + // word. The guess is the caller itself, so there is no guess. + if (bestMatch && bestScore >= 2 && bestMatch.id !== ref.fromNodeId) { return { original: ref, targetNodeId: bestMatch.id, @@ -8328,6 +8378,10 @@ export function matchReference( context: ResolutionContext ): ResolvedRef | null { const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); + // `this.container.classList.toggle()` inside `toggle()`, `window.$events + // .listen()` inside `listen()`: a member of what the chain reaches, which is + // the calling method only through a field of the caller's own type. + if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; return result ? retargetSelfOverload(result, ref, context) : result; } From d6f68648cb3c83712a66206a07118b975b9bf8fe Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:16:23 +0000 Subject: [PATCH 132/363] fix(spring): name heuristics start from the reference's own file and package (#2222) The Spring resolver's entity/service/controller name heuristics took the first same-named class under a conventional folder; they now prefer the ref's own file, then its package, and never reach another file's nested class by its bare name. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/spring-model-name-scope.test.ts | 64 +++++++++++++++++++++++ src/resolution/frameworks/java.ts | 36 ++++++++++--- 3 files changed, 93 insertions(+), 8 deletions(-) create mode 100644 __tests__/spring-model-name-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index e2b990520e..c6e97b3384 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. - A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. diff --git a/__tests__/spring-model-name-scope.test.ts b/__tests__/spring-model-name-scope.test.ts new file mode 100644 index 0000000000..d8d60c7c29 --- /dev/null +++ b/__tests__/spring-model-name-scope.test.ts @@ -0,0 +1,64 @@ +/** + * The Spring resolver's name heuristics (an entity under `/model/`, a + * `…Service`, a `…Controller`) start from what the reference's own scope + * declares — its file, then its package — and never reach a class nested in + * another file's class by its bare name: every MyBatis `XExample` declares its + * own nested `Criteria`, and mall's `new Criteria()` all went to the first. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const example = (name: string) => `package com.macro.mall.model; + +public class ${name} { + public Criteria createCriteria() { + Criteria criteria = new Criteria(); + return criteria; + } + + public static class Criteria { + } +} +`; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-spring-model-')); + const files: Record = { + 'pom.xml': 'spring-boot-starter\n', + 'mall-mbg/src/main/java/com/macro/mall/model/CmsHelpCategoryExample.java': example('CmsHelpCategoryExample'), + 'mall-mbg/src/main/java/com/macro/mall/model/OmsOrderExample.java': example('OmsOrderExample'), + 'mall-admin/src/main/java/com/macro/mall/service/OrderService.java': `package com.macro.mall.service; + +@Service +public class OrderService { +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +describe('Spring name heuristics', () => { + it('resolve a name to its own file’s nested class, not another file’s', () => { + const file = 'mall-mbg/src/main/java/com/macro/mall/model/OmsOrderExample.java'; + const ids = cg.getNodesInFile(file).map((n) => n.id); + const targets = cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'instantiates' || e.kind === 'references') + .map((e) => cg.getNode(e.target)!).filter((t) => t.name === 'Criteria').map((t) => t.filePath); + expect(targets.length).toBeGreaterThan(0); + expect(new Set(targets)).toEqual(new Set([file])); + }); +}); diff --git a/src/resolution/frameworks/java.ts b/src/resolution/frameworks/java.ts index bd36b1e4ed..3e1b7d6053 100644 --- a/src/resolution/frameworks/java.ts +++ b/src/resolution/frameworks/java.ts @@ -137,7 +137,7 @@ export const springResolver: FrameworkResolver = { // Pattern 1: Service references (dependency injection) if (ref.referenceName.endsWith('Service')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result) { return { original: ref, @@ -150,7 +150,7 @@ export const springResolver: FrameworkResolver = { // Pattern 2: Repository references if (ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, REPO_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, REPO_DIRS, context); if (result) { return { original: ref, @@ -163,7 +163,7 @@ export const springResolver: FrameworkResolver = { // Pattern 3: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -176,7 +176,7 @@ export const springResolver: FrameworkResolver = { // Pattern 4: Entity/Model references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, ENTITY_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, ENTITY_DIRS, context); if (result) { return { original: ref, @@ -189,7 +189,7 @@ export const springResolver: FrameworkResolver = { // Pattern 5: Component references if (ref.referenceName.endsWith('Component') || ref.referenceName.endsWith('Config')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, COMPONENT_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, COMPONENT_DIRS, context); if (result) { return { original: ref, @@ -555,19 +555,30 @@ function joinPath(prefix: string, sub: string): string { /** * Resolve a symbol by name using indexed queries instead of scanning all files. + * What the reference's own scope declares comes first — its file, then its + * package (directory) — and a class nested in another file's class is out of + * reach by its bare name: every MyBatis `XExample` declares its own nested + * `Criteria`, and mall's 8,747 `new Criteria()` went to the first one. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); + const candidates = context.getNodesByName(ref.referenceName); if (candidates.length === 0) return null; - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); + const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && + (n.filePath === ref.filePath || !isNestedType(n, context))); if (kindFiltered.length === 0) return null; + const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); + const samePackage = kindFiltered.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); + if (samePackage) return samePackage.id; + // Prefer candidates in framework-conventional directories const preferred = kindFiltered.filter((n) => preferredDirPatterns.some((d) => n.filePath.includes(d)) @@ -578,3 +589,12 @@ function resolveByNameAndKind( // Fall back to any match return kindFiltered[0]!.id; } + +/** Whether a JVM type is declared inside another type of its file. */ +function isNestedType(n: Node, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return false; + const owner = n.qualifiedName.slice(0, cut); + return context.getNodesInFile(n.filePath).some((p) => + p.qualifiedName === owner && (p.kind === 'class' || p.kind === 'interface' || p.kind === 'enum' || p.kind === 'struct' || p.kind === 'trait')); +} From ca1eba98bd52de3fdb321660f10006efd74f513c Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:27:15 +0000 Subject: [PATCH 133/363] fix(resolution): a bare-recorded call on another expression's value is not the caller (#2223) Generalizes the collapsed-chain self veto beyond TS/JS this./window. roots: a call the extractor recorded by its bare name but the source writes as a member call on anything other than this/self/super is never the calling method; and a value's initializer never calls the value itself. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/self-call-delegation.test.ts | 43 +++++++++++++++++--- src/resolution/name-matcher.ts | 54 ++++++++++++++++---------- 3 files changed, 71 insertions(+), 27 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c6e97b3384..7108327cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. - C# names now follow the language's scopes. A block `namespace X { … }` qualifies only the types inside it, so a type declared after the block, or in a file's second namespace, is no longer filed under the first. A namespace written inside another is `Outer.Inner`. A `using` links to the project's namespace instead of another file's `using` of the same name. A `global using` applies only within its own project. A nested type is reachable by bare name only from inside its owner or a type deriving from it, including through another partial part. Before, serilog's `Guard` was filed under `JetBrains.Annotations`, its tests' `Some.InformationEvent()` went to the performance tests' `Some`, and AutoMapper's same-file `new Source()` could reach another test class's nested `Source`. diff --git a/__tests__/self-call-delegation.test.ts b/__tests__/self-call-delegation.test.ts index d23bcae87d..40e2a617aa 100644 --- a/__tests__/self-call-delegation.test.ts +++ b/__tests__/self-call-delegation.test.ts @@ -1,11 +1,13 @@ /** * A method that hands its call on to another object — a wrapper — is not - * calling itself. TS/JS calls through `this.…` or `window.…` reach - * the resolver by their bare name, so BookStack's `toggle()` doing - * `this.container.classList.toggle('open')` and `listen()` doing - * `window.$events.listen(…)` bound to themselves; a guess from a receiver's - * name alone (`FileStorage::delete` doing `$storage->delete($path)`) did too. - * A recursion through a field the class declares as its own type stays. + * calling itself. Calls through `this.…` / `window.…` (TS/JS) or + * through an expression's value (Scala `requestToArmeria(request).execute()`, + * Rust `self.0.into_route(state)`) reach the resolver by their bare name, so + * BookStack's `toggle()` doing `this.container.classList.toggle('open')` + * bound to itself; a guess from a receiver's name alone (`FileStorage::delete` + * doing `$storage->delete($path)`) did too, and a value's initializer chain + * (`val response = basicRequest.get(…).response(…)`) to the value. A recursion + * through a field the class declares as its own type stays. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -64,6 +66,30 @@ class ImageStorage { } } +`, + 'core/src/main/scala/sttp/ArmeriaBackend.scala': `package sttp + +class ArmeriaBackend { + def requestToArmeria(request: String): Client = new Client + + def execute(request: String): Unit = { + val armeriaRes = requestToArmeria(request).execute() + } +} + +class CurlTest { + val response = basicRequest + .get("http://example.com") + .response(asString) +} +`, + 'src/routing/route.rs': `pub struct BoxedIntoRoute(Box); + +impl BoxedIntoRoute { + pub fn into_route(self, state: u32) -> u32 { + self.0.into_route(state) + } +} `, }; for (const [rel, content] of Object.entries(files)) { @@ -93,6 +119,11 @@ describe('a call a method hands on', () => { expect(selfCallLines('resources/js/tree.ts')).toEqual([5]); }); + it('through another expression’s value is not the method itself', () => { + expect(selfCallLines('core/src/main/scala/sttp/ArmeriaBackend.scala')).toEqual([]); + expect(selfCallLines('src/routing/route.rs')).toEqual([]); + }); + it('through a receiver named like the caller’s class is not the method itself', () => { expect(selfCallLines('app/Uploads/FileStorage.php')).toEqual([]); }); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e12f46169f..640a2beeb0 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1403,31 +1403,35 @@ export const CASE_INSENSITIVE_LANGUAGES = new Set(['php', 'pascal', 'cfm * used to pick over the module-scope function the call actually means. */ /** - * The links between `this` / `super` / `window` and the method of a TS/JS - * call the extractor recorded by its bare name — `this.container.classList - * .toggle()` reaches the resolver as `toggle`, with links `container`, - * `classList`. Null for a call not written that way, `this.toggle()` included. - */ -function collapsedJsChain(ref: UnresolvedRef, context: ResolutionContext): { root: string; links: string[] } | null { - if (!JS_FAMILY.has(ref.language) || ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; + * The receiver a call the extractor recorded by its bare name is written on, + * read from the source: TS/JS keeps `this.container.classList.toggle()` and + * `window.$events.listen()` bare, Scala `requestToArmeria(request).execute()` + * and `_.get.whenRequestMatchesPartial(…)`. `'self'` for `this.m()` / + * `self.m()` / `super.m()` / `super().m()`; null for a call written bare (or + * not found). `links` are the member names between `this` and the method. + */ +function bareCallReceiver(ref: UnresolvedRef, context: ResolutionContext): { receiver: string; links: string[] } | null { + if (ref.referenceKind !== 'calls' || !/^[A-Za-z_$][\w$]*$/.test(ref.referenceName)) return null; const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/); if (!lines) return null; - const text = lines.slice(ref.line - 1, ref.line + 3).join('\n').slice(ref.column); + const text = lines.slice(ref.line - 1, ref.line + 7).join('\n').slice(Math.max(0, ref.column)); const name = ref.referenceName.replace(/\$/g, '\\$'); - const m = new RegExp(`^(this|super|window)((?:\\s*\\??\\.\\s*#?[\\w$]+(?:\\([^()]*\\))?)+?)\\s*\\??\\.\\s*${name}\\s*(?:<[^<>()]*>)?\\s*\\(`).exec(text); - if (!m) return null; - return { root: m[1]!, links: m[2]!.split('.').map((l) => l.replace(/[\s?]/g, '')).filter((l) => l !== '') }; + const at = new RegExp(`(?()]*>|\\[(?:[^\\[\\]]|\\[[^\\[\\]]*\\])*\\])?\\s*[({]`).exec(text); + if (!at) return null; + const before = text.slice(0, at.index).replace(/\s+$/, ''); + if (!/\??\.$/.test(before)) return null; + const head = before.replace(/\??\.$/, '').replace(/\s+$/, ''); + if (/(?:^|[^\w$.])(?:this|self|super|Self)$/.test(head) || /(?:^|[^\w$.])super\s*\([^()]*\)$/.test(head)) return { receiver: 'self', links: [] }; + const chain = /(?:^|[^\w$.#])((?:this|super)(?:\s*\??\.\s*#?[\w$]+)+)$/.exec(head); + const links = chain ? chain[1]!.split('.').slice(1).map((l) => l.replace(/[\s?]/g, '')) : []; + return { receiver: head.slice(-40), links }; } -/** - * Whether a collapsed `this..m()` inside `m` can be `m` itself: only - * when the class declares the field as its own type — a tree node's - * `this.left.insert(v)` — never through a field of another type - * (`this.editor.input.focus()`, `this.container.classList.toggle()`). - */ +/** Whether a call recorded by its bare name is written on something other than the caller's own object. */ function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { - const chain = collapsedJsChain(ref, context); - return chain !== null && !isCollapsedSelfRecursion(chain, ref, context); + const written = bareCallReceiver(ref, context); + if (!written || written.receiver === 'self') return false; + return !(JS_FAMILY.has(ref.language) && isCollapsedSelfRecursion({ root: 'this', links: written.links }, ref, context)); } function isCollapsedSelfRecursion(chain: { root: string; links: string[] }, ref: UnresolvedRef, context: ResolutionContext): boolean { @@ -8379,12 +8383,20 @@ export function matchReference( ): ResolvedRef | null { const result = gateLanguageMatch(matchReferenceInner(ref, context), ref, context); // `this.container.classList.toggle()` inside `toggle()`, `window.$events - // .listen()` inside `listen()`: a member of what the chain reaches, which is - // the calling method only through a field of the caller's own type. + // .listen()` inside `listen()`, Scala's `requestToArmeria(request).execute()` + // inside `execute()`: a member of what the receiver is, which is the calling + // method only through a TS/JS field of the caller's own type. if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; + // Nor does a value's initializer call the value: sttp's `val response = + // basicRequest.get(…).response(asStringAlways)` is a request's `response`. + if (result && result.targetNodeId === ref.fromNodeId && ref.referenceKind === 'calls' && + VALUE_KINDS.has(context.getNodeById?.(ref.fromNodeId)?.kind ?? '')) return null; return result ? retargetSelfOverload(result, ref, context) : result; } +/** Node kinds that hold a value rather than run code. */ +const VALUE_KINDS: ReadonlySet = new Set(['variable', 'constant', 'field', 'property']); + /** Languages whose methods overload by arity. */ const OVERLOADING_LANGUAGES: ReadonlySet = new Set(['csharp', 'java', 'kotlin', 'swift', 'cpp', 'scala', 'dart', 'vbnet']); From 7365427d4f199ccfbfa25711449ca4378ce0570f Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:34:57 +0000 Subject: [PATCH 134/363] fix(rust): framework name heuristics never reach a function-local item (#2224) The Rust resolver's handler/service/struct heuristics took the first same-named item and then checked scope; they now drop items declared inside a function body and out-of-scope names before choosing, and take the file's own item first. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/rust-framework-name-scope.test.ts | 83 +++++++++++++++++++++ src/resolution/frameworks/rust.ts | 27 +++++-- src/resolution/name-matcher.ts | 2 +- 4 files changed, 105 insertions(+), 8 deletions(-) create mode 100644 __tests__/rust-framework-name-scope.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 7108327cb0..1d1ee60c32 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. - A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. diff --git a/__tests__/rust-framework-name-scope.test.ts b/__tests__/rust-framework-name-scope.test.ts new file mode 100644 index 0000000000..614834f249 --- /dev/null +++ b/__tests__/rust-framework-name-scope.test.ts @@ -0,0 +1,83 @@ +/** + * The Rust resolver's struct/handler/service name heuristics never reach an + * item declared inside a function body — axum's examples' `Uri` (imported + * from `http`) went to a `struct Uri` a routing test declares inside itself — + * and a file's own item comes first: ripgrep's `Ok(Glob { … })` in + * globset's glob.rs is its own `Glob`, not the CLI flags' `Glob`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rust-fw-')); + const files: Record = { + 'Cargo.toml': '[workspace]\nmembers = ["crates/core", "crates/globset", "crates/web", "examples/tls"]\n', + 'crates/core/Cargo.toml': '[package]\nname = "rg-core"\n', + 'crates/web/Cargo.toml': '[package]\nname = "web"\n', + 'crates/web/src/lib.rs': 'pub mod http;\n', + 'crates/web/src/http.rs': 'pub use ::http::Uri;\n', + 'crates/web/src/routing/tests.rs': `fn outer_middleware_still_see_whole_url() { + struct Uri; + let _ = Uri; +} +`, + 'examples/tls/Cargo.toml': '[package]\nname = "example-tls"\n\n[dependencies]\nweb = { path = "../../crates/web" }\n', + 'examples/tls/src/main.rs': `use web::http::Uri; + +fn make_https(uri: Uri, port: u16) -> Uri { + uri +} +`, + 'crates/core/src/flags/defs.rs': `pub struct Glob; + +pub fn glob_flag() -> Glob { + Glob +} +`, + 'crates/globset/Cargo.toml': '[package]\nname = "globset"\n', + 'crates/globset/src/glob.rs': `pub struct Glob { + glob: String, +} + +impl Glob { + pub fn new(glob: &str) -> Result { + Ok(Glob { glob: glob.to_string() }) + } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('Rust framework name heuristics', () => { + it('never reach an item declared inside a function body', () => { + expect(targetsFrom('examples/tls/src/main.rs').filter((t) => t.includes('outer_middleware'))).toEqual([]); + }); + + it('take the file’s own item first', () => { + const globs = targetsFrom('crates/globset/src/glob.rs').filter((t) => t.endsWith(':Glob')); + expect(globs.length).toBeGreaterThan(0); + expect(new Set(globs)).toEqual(new Set(['crates/globset/src/glob.rs:Glob'])); + }); +}); diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index 283c453521..f7e79927a0 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,7 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; -import { isRustNameInScope } from '../name-matcher'; +import { isRustNameInScope, isLexicallyReachable } from '../name-matcher'; /** * Whether the item a name heuristic found is one the reference can name: @@ -42,7 +42,7 @@ export const rustResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Handler references if (ref.referenceName.endsWith('_handler') || ref.referenceName.startsWith('handle_')) { - const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, HANDLER_DIRS, context); + const result = resolveByNameAndKind(ref, FUNCTION_KINDS, HANDLER_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -55,7 +55,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 2: Service/Repository trait implementations if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -68,7 +68,7 @@ export const rustResolver: FrameworkResolver = { // Pattern 3: Struct references (PascalCase) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, STRUCT_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, STRUCT_KINDS, MODEL_DIRS, context); if (result && inRustScope(result, ref, context)) { return { original: ref, @@ -291,18 +291,31 @@ function findMatchingParen(s: string, openIdx: number): number { /** * Resolve a symbol by name using indexed queries instead of scanning all files. */ +/** + * The same-named item of these kinds that the reference can see — in scope + * by its `use`s and module paths, and not declared inside some function body + * (axum's examples' `Uri`, imported from `http`, went to a `struct Uri` a + * routing test declares inside itself) — its own file's first, then a + * framework-conventional directory's. + */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); + const candidates = context.getNodesByName(ref.referenceName); if (candidates.length === 0) return null; - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); + const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && + isLexicallyReachable(n, ref, context) && isRustNameInScope(n, ref, context)); if (kindFiltered.length === 0) return null; + // The file is a module: its own items are in scope. A sibling file is + // another module, reached only through a `use` — no nearer than any other. + const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + // Prefer candidates in framework-conventional directories const preferred = kindFiltered.filter((n) => preferredDirPatterns.some((d) => n.filePath.includes(d)) diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 640a2beeb0..9cb0459b6b 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -636,7 +636,7 @@ const LOCAL_TYPE_KINDS = new Set(['class', 'struct', 'enum', 'interface' * unaffected (their parent resolves to a class-like node), as are top-level * symbols and C++ namespace-prefixed names (the prefix has no node). */ -function isLexicallyReachable( +export function isLexicallyReachable( candidate: Node, ref: UnresolvedRef, context: ResolutionContext From 179939055c6d0e0bb094028825fcd8797daeaf30 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 00:45:56 +0000 Subject: [PATCH 135/363] fix(frameworks): name heuristics pick only what the reference can see, own scope first (#2225) One shared pick behind the Django/FastAPI/Flask, ASP.NET, Gin, SwiftUI/Vapor, Spring and Rust name heuristics: never a function-local declaration, a type nested in another file's type, or a node the language keeps out of the reference's file; the reference's own file first, then its directory (not Rust), then a conventional folder. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + .../framework-name-heuristic-scope.test.ts | 68 ++++++++++++++++++ __tests__/frameworks.test.ts | 6 +- src/resolution/frameworks/csharp.ts | 33 +++------ src/resolution/frameworks/go.ts | 42 ++++------- src/resolution/frameworks/java.ts | 40 +---------- src/resolution/frameworks/name-heuristic.ts | 70 +++++++++++++++++++ src/resolution/frameworks/python.ts | 36 +++------- src/resolution/frameworks/rust.ts | 40 +++-------- src/resolution/frameworks/swift.ts | 44 ++++-------- 10 files changed, 198 insertions(+), 182 deletions(-) create mode 100644 __tests__/framework-name-heuristic-scope.test.ts create mode 100644 src/resolution/frameworks/name-heuristic.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 1d1ee60c32..0374fc2041 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. - In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. - In Spring projects, a class name now resolves to the one in the reference's own file or package before any guess by folder convention. A class nested inside another file's class is never reached by its bare name. Before, every MyBatis `Example` class's `new Criteria()` in mall linked to the first `Example`'s nested `Criteria`, and halo's Lombok `@Builder` linked to a nested `Builder` class elsewhere. diff --git a/__tests__/framework-name-heuristic-scope.test.ts b/__tests__/framework-name-heuristic-scope.test.ts new file mode 100644 index 0000000000..7457c2ea71 --- /dev/null +++ b/__tests__/framework-name-heuristic-scope.test.ts @@ -0,0 +1,68 @@ +/** + * The framework resolvers' name heuristics ("a `…View` is a view under + * `/views/`", "a `…Form` is a form") pick only what the reference can see, + * starting from its own scope: never a class a test declares inside one of + * its functions (netbox's `TestForm`, DRF's `MockView`), its own file's + * first, then its package's, then a conventional folder's. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fw-name-')); + const files: Record = { + 'manage.py': '', + 'utilities/tests/test_forms.py': `class GetFieldValueTestCase: + def setUpTestData(self): + class TestForm: + pass + return TestForm() +`, + 'utilities/tests/test_templatetags.py': `def test_any_required(): + return TestForm() +`, + 'tests/browsable_api/views.py': `class MockView: + pass +`, + 'tests/authentication/test_authentication.py': `class MockView: + pass + + +def test_auth(): + return MockView() +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind === 'instantiates' || e.kind === 'calls' || e.kind === 'references') + .map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.qualifiedName}`); +}; + +describe('framework name heuristics', () => { + it('never pick a class declared inside another file’s function', () => { + expect(targetsFrom('utilities/tests/test_templatetags.py').filter((t) => t.includes('TestForm'))).toEqual([]); + }); + + it('take the reference’s own file first', () => { + expect(targetsFrom('tests/authentication/test_authentication.py').filter((t) => t.includes('MockView'))) + .toEqual(['tests/authentication/test_authentication.py:MockView']); + }); +}); diff --git a/__tests__/frameworks.test.ts b/__tests__/frameworks.test.ts index cc03c7702b..940c5a0066 100644 --- a/__tests__/frameworks.test.ts +++ b/__tests__/frameworks.test.ts @@ -998,11 +998,11 @@ describe('springResolver.resolve — DI heuristics are gated to Java/Kotlin non- // hijack a Scala `extends X` to a same-named class found via directory // heuristics — inheritance must resolve through imports/name matching. const decoyNode: Node = { - id: 'class:src/test/model/ExtCustomer.java:ExtCustomer:3', + id: 'class:src/main/model/ExtCustomer.java:ExtCustomer:3', kind: 'class', name: 'ExtCustomer', - qualifiedName: 'src/test/model/ExtCustomer.java::ExtCustomer', - filePath: 'src/test/model/ExtCustomer.java', + qualifiedName: 'src/main/model/ExtCustomer.java::ExtCustomer', + filePath: 'src/main/model/ExtCustomer.java', language: 'java', startLine: 3, endLine: 10, diff --git a/src/resolution/frameworks/csharp.ts b/src/resolution/frameworks/csharp.ts index 51040c20b4..a70daa4a99 100644 --- a/src/resolution/frameworks/csharp.ts +++ b/src/resolution/frameworks/csharp.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const aspnetResolver: FrameworkResolver = { name: 'aspnet', @@ -67,7 +68,7 @@ export const aspnetResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -80,7 +81,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 2: Service references (dependency injection) if (ref.referenceName.endsWith('Service') || ref.referenceName.startsWith('I') && ref.referenceName.length > 1) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, SERVICE_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, SERVICE_DIRS, context); if (result) { return { original: ref, @@ -93,7 +94,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 3: Repository references if (ref.referenceName.endsWith('Repository')) { - const result = resolveByNameAndKind(ref.referenceName, SERVICE_KINDS, REPO_DIRS, context); + const result = resolveByNameAndKind(ref, SERVICE_KINDS, REPO_DIRS, context); if (result) { return { original: ref, @@ -106,7 +107,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 4: Model/Entity references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, MODEL_DIRS, context); if (result) { return { original: ref, @@ -119,7 +120,7 @@ export const aspnetResolver: FrameworkResolver = { // Pattern 5: ViewModel references if (ref.referenceName.endsWith('ViewModel') || ref.referenceName.endsWith('Dto')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VIEWMODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VIEWMODEL_DIRS, context); if (result) { return { original: ref, @@ -410,28 +411,12 @@ const VIEWMODEL_DIRS = ['/ViewModels/', '/ViewModel/', '/DTOs/', '/Dto/']; const CLASS_KINDS = new Set(['class']); const SERVICE_KINDS = new Set(['class', 'interface']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } diff --git a/src/resolution/frameworks/go.ts b/src/resolution/frameworks/go.ts index 168d4b22c3..5e819060eb 100644 --- a/src/resolution/frameworks/go.ts +++ b/src/resolution/frameworks/go.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const goResolver: FrameworkResolver = { name: 'go', @@ -27,7 +28,7 @@ export const goResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: Handler references if (ref.referenceName.endsWith('Handler') || ref.referenceName.startsWith('Handle')) { - const result = resolveByNameAndKind(ref.referenceName, 'function', HANDLER_DIRS, context); + const result = resolveByNameAndKind(ref, 'function', HANDLER_DIRS, context); if (result) { return { original: ref, @@ -40,7 +41,7 @@ export const goResolver: FrameworkResolver = { // Pattern 2: Service/Repository references if (ref.referenceName.endsWith('Service') || ref.referenceName.endsWith('Repository') || ref.referenceName.endsWith('Store')) { - const result = resolveByNameAndKind(ref.referenceName, null, SERVICE_DIRS, context, SERVICE_KINDS); + const result = resolveByNameAndKind(ref, null, SERVICE_DIRS, context, SERVICE_KINDS); if (result) { return { original: ref, @@ -53,7 +54,7 @@ export const goResolver: FrameworkResolver = { // Pattern 3: Middleware references if (ref.referenceName.endsWith('Middleware') || ref.referenceName.startsWith('Auth') || ref.referenceName.startsWith('Log')) { - const result = resolveByNameAndKind(ref.referenceName, 'function', MIDDLEWARE_DIRS, context); + const result = resolveByNameAndKind(ref, 'function', MIDDLEWARE_DIRS, context); if (result) { return { original: ref, @@ -66,7 +67,7 @@ export const goResolver: FrameworkResolver = { // Pattern 4: Model/Entity references (typically PascalCase structs) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, 'struct', MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, 'struct', MODEL_DIRS, context); if (result) { return { original: ref, @@ -172,36 +173,17 @@ const MIDDLEWARE_DIRS = ['middleware', 'middlewares']; const MODEL_DIRS = ['model', 'models', 'entity', 'entities', 'domain', 'pkg']; const SERVICE_KINDS = new Set(['struct', 'interface']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - * Uses getNodesByName (O(log n) indexed lookup) instead of iterating every file. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kind: string | null, preferredDirs: string[], context: ResolutionContext, kinds?: Set ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - // Filter by kind - const kindFiltered = candidates.filter((n) => { - if (kinds) return kinds.has(n.kind); - if (kind) return n.kind === kind; - return true; - }); - - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirs.some((d) => n.filePath.includes(`/${d}/`)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + const allowed: ReadonlySet = kinds ?? (kind ? new Set([kind]) : GO_NAMED_KINDS); + return pickByNameAndKind(ref, allowed, (f) => preferredDirs.some((d) => f.includes(`/${d}/`)), context); } + +/** Any declaration a name can be (the heuristic with no kind). */ +const GO_NAMED_KINDS: ReadonlySet = new Set(['function', 'method', 'struct', 'interface', 'type_alias', 'variable', 'constant']); diff --git a/src/resolution/frameworks/java.ts b/src/resolution/frameworks/java.ts index 3e1b7d6053..1759b4ea59 100644 --- a/src/resolution/frameworks/java.ts +++ b/src/resolution/frameworks/java.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; export const springResolver: FrameworkResolver = { name: 'spring', @@ -553,48 +554,13 @@ function joinPath(prefix: string, sub: string): string { return '/' + parts.join('/'); } -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - * What the reference's own scope declares comes first — its file, then its - * package (directory) — and a class nested in another file's class is out of - * reach by its bare name: every MyBatis `XExample` declares its own nested - * `Criteria`, and mall's 8,747 `new Criteria()` went to the first one. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(ref.referenceName); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && - (n.filePath === ref.filePath || !isNestedType(n, context))); - if (kindFiltered.length === 0) return null; - - const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); - if (sameFile) return sameFile.id; - const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); - const samePackage = kindFiltered.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); - if (samePackage) return samePackage.id; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } -/** Whether a JVM type is declared inside another type of its file. */ -function isNestedType(n: Node, context: ResolutionContext): boolean { - const cut = n.qualifiedName.lastIndexOf('::'); - if (cut < 0) return false; - const owner = n.qualifiedName.slice(0, cut); - return context.getNodesInFile(n.filePath).some((p) => - p.qualifiedName === owner && (p.kind === 'class' || p.kind === 'interface' || p.kind === 'enum' || p.kind === 'struct' || p.kind === 'trait')); -} diff --git a/src/resolution/frameworks/name-heuristic.ts b/src/resolution/frameworks/name-heuristic.ts new file mode 100644 index 0000000000..9b06843a21 --- /dev/null +++ b/src/resolution/frameworks/name-heuristic.ts @@ -0,0 +1,70 @@ +/** + * The shared pick behind the framework resolvers' name heuristics — "a + * `…Service` is a class under `/services/`", "a PascalCase name is a model + * under `/models/`", "`router` is a FastAPI router under `/routers/`". + * + * Each resolver used to take the FIRST same-named node of the right kind, + * preferring a conventional folder, and that heuristic outranks name matching + * on a tie. So every MyBatis `XExample`'s `new Criteria()` in mall went to the + * first `Example`'s nested `Criteria`, and axum's examples' `Uri` to a struct a + * routing test declares inside itself. The pick now considers only what the + * reference can see, and starts from its own scope. + */ +import { Node } from '../../types'; +import { UnresolvedRef, ResolutionContext } from '../types'; +import { isLexicallyReachable, isVisibleAcrossFiles } from '../name-matcher'; + +/** Node kinds a nested type can be declared in. */ +const TYPE_OWNER_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'trait', 'protocol', 'record']); + +export interface NamePickOptions { + /** + * Whether a file in the reference's directory comes before the conventional + * folders — its package (JVM, Go, C#, Python). Not for Rust, where a sibling + * file is another module, reached only through a `use`. + */ + sameDirectory?: boolean; + /** A language's own scope rule over the candidates (Rust's `use` / module paths). */ + accept?: (n: Node) => boolean; +} + +/** + * The id of the node named `ref.referenceName`, of one of `kinds`, that the + * reference can see: never a declaration inside some function body, never a + * type nested in another file's type (a nested type is reached by its bare + * name only from inside its owner), never one the language keeps out of the + * reference's file (a `private` member, an unexported Go name, a C# namespace + * the file does not use, a test suite from production code). Its own file's + * first, then (by default) its directory's, then a conventional folder's. + */ +export function pickByNameAndKind( + ref: UnresolvedRef, + kinds: ReadonlySet, + inPreferredDir: (filePath: string) => boolean, + context: ResolutionContext, + options: NamePickOptions = {}, +): string | null { + const candidates = context.getNodesByName(ref.referenceName).filter((n) => + kinds.has(n.kind) && + isLexicallyReachable(n, ref, context) && + (n.filePath === ref.filePath || (!isNestedType(n, context) && isVisibleAcrossFiles(n, ref, context))) && + (!options.accept || options.accept(n))); + if (candidates.length === 0) return null; + + const sameFile = candidates.find((n) => n.filePath === ref.filePath); + if (sameFile) return sameFile.id; + if (options.sameDirectory !== false) { + const dir = ref.filePath.slice(0, ref.filePath.lastIndexOf('/') + 1); + const sameDir = candidates.find((n) => n.filePath.startsWith(dir) && !n.filePath.slice(dir.length).includes('/')); + if (sameDir) return sameDir.id; + } + return (candidates.find((n) => inPreferredDir(n.filePath)) ?? candidates[0]!).id; +} + +/** Whether a node is declared inside a type of its own file. */ +function isNestedType(n: Node, context: ResolutionContext): boolean { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return false; + const owner = n.qualifiedName.slice(0, cut); + return context.getNodesInFile(n.filePath).some((p) => p.qualifiedName === owner && TYPE_OWNER_KINDS.has(p.kind)); +} diff --git a/src/resolution/frameworks/python.ts b/src/resolution/frameworks/python.ts index 7ef69f0dfb..777016f2ac 100644 --- a/src/resolution/frameworks/python.ts +++ b/src/resolution/frameworks/python.ts @@ -8,6 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolutionContext, FrameworkExtractionResult } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { resolveImportPath } from '../import-resolver'; +import { pickByNameAndKind } from './name-heuristic'; export const djangoResolver: FrameworkResolver = { name: 'django', @@ -25,15 +26,15 @@ export const djangoResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('Model') || /^[A-Z][a-z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, MODEL_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.endsWith('View') || ref.referenceName.endsWith('ViewSet')) { - const result = resolveByNameAndKind(ref.referenceName, VIEW_KINDS, VIEW_DIRS, context); + const result = resolveByNameAndKind(ref, VIEW_KINDS, VIEW_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.endsWith('Form')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, FORM_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, FORM_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } // ORM dynamic dispatch: QuerySet._fetch_all (and siblings) call @@ -202,7 +203,7 @@ export const flaskResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('_bp') || ref.referenceName.endsWith('_blueprint')) { - const result = resolveByNameAndKind(ref.referenceName, VARIABLE_KINDS, [], context); + const result = resolveByNameAndKind(ref, VARIABLE_KINDS, [], context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } return null; @@ -265,11 +266,11 @@ export const fastapiResolver: FrameworkResolver = { resolve(ref, context) { if (ref.referenceName.endsWith('_router') || ref.referenceName === 'router') { - const result = resolveByNameAndKind(ref.referenceName, VARIABLE_KINDS, ROUTER_DIRS, context); + const result = resolveByNameAndKind(ref, VARIABLE_KINDS, ROUTER_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.8, resolvedBy: 'framework' }; } if (ref.referenceName.startsWith('get_') || ref.referenceName.startsWith('Depends')) { - const result = resolveByNameAndKind(ref.referenceName, FUNCTION_KINDS, DEP_DIRS, context); + const result = resolveByNameAndKind(ref, FUNCTION_KINDS, DEP_DIRS, context); if (result) return { original: ref, targetNodeId: result, confidence: 0.75, resolvedBy: 'framework' }; } return null; @@ -640,29 +641,12 @@ const VIEW_KINDS = new Set(['class', 'function']); const VARIABLE_KINDS = new Set(['variable']); const FUNCTION_KINDS = new Set(['function']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - if (preferredDirPatterns.length > 0) { - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - if (preferred.length > 0) return preferred[0]!.id; - } - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } diff --git a/src/resolution/frameworks/rust.ts b/src/resolution/frameworks/rust.ts index f7e79927a0..1fa4f79d19 100644 --- a/src/resolution/frameworks/rust.ts +++ b/src/resolution/frameworks/rust.ts @@ -8,7 +8,8 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; import { getCargoWorkspaceCrateMap } from './cargo-workspace'; -import { isRustNameInScope, isLexicallyReachable } from '../name-matcher'; +import { isRustNameInScope } from '../name-matcher'; +import { pickByNameAndKind } from './name-heuristic'; /** * Whether the item a name heuristic found is one the reference can name: @@ -288,43 +289,18 @@ function findMatchingParen(s: string, openIdx: number): number { return -1; } -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ -/** - * The same-named item of these kinds that the reference can see — in scope - * by its `use`s and module paths, and not declared inside some function body - * (axum's examples' `Uri`, imported from `http`, went to a `struct Uri` a - * routing test declares inside itself) — its own file's first, then a - * framework-conventional directory's. - */ +/** A framework name heuristic's pick (see name-heuristic.ts): in scope by its `use`s and module paths. */ function resolveByNameAndKind( ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(ref.referenceName); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind) && - isLexicallyReachable(n, ref, context) && isRustNameInScope(n, ref, context)); - if (kindFiltered.length === 0) return null; - - // The file is a module: its own items are in scope. A sibling file is - // another module, reached only through a `use` — no nearer than any other. - const sameFile = kindFiltered.find((n) => n.filePath === ref.filePath); - if (sameFile) return sameFile.id; - - // Prefer candidates in framework-conventional directories - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - - if (preferred.length > 0) return preferred[0]!.id; - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context, { + // The file is a module: a sibling file is another one, reached only through a `use`. + sameDirectory: false, + accept: (n) => isRustNameInScope(n, ref, context), + }); } interface ModuleResolution { diff --git a/src/resolution/frameworks/swift.ts b/src/resolution/frameworks/swift.ts index 8c91bf95dc..5702aef4a0 100644 --- a/src/resolution/frameworks/swift.ts +++ b/src/resolution/frameworks/swift.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { stripCommentsForRegex } from '../strip-comments'; +import { pickByNameAndKind } from './name-heuristic'; // No extract(): a SwiftUI view is its own struct node, and a UIKit controller // its class. A one-line `component`/`class` twin per `struct X: View` (and per @@ -41,7 +42,7 @@ export const swiftUIResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: View references (SwiftUI views are PascalCase ending in View) if (ref.referenceName.endsWith('View') && /^[A-Z]/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, VIEW_KINDS, VIEW_DIRS, context); + const result = resolveByNameAndKind(ref, VIEW_KINDS, VIEW_DIRS, context); if (result) { return { original: ref, @@ -54,7 +55,7 @@ export const swiftUIResolver: FrameworkResolver = { // Pattern 2: ViewModel/ObservableObject references if (ref.referenceName.endsWith('ViewModel') || ref.referenceName.endsWith('Store') || ref.referenceName.endsWith('Manager')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VIEWMODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VIEWMODEL_DIRS, context); if (result) { return { original: ref, @@ -67,7 +68,7 @@ export const swiftUIResolver: FrameworkResolver = { // Pattern 3: Model references if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, MODEL_KINDS, MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, MODEL_KINDS, MODEL_DIRS, context); if (result) { return { original: ref, @@ -107,7 +108,7 @@ export const uikitResolver: FrameworkResolver = { resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { // Pattern 1: ViewController references if (ref.referenceName.endsWith('ViewController')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, VC_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, VC_DIRS, context); if (result) { return { original: ref, @@ -120,7 +121,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 2: UIView subclass references if (ref.referenceName.endsWith('View') && !ref.referenceName.endsWith('ViewController')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, UIVIEW_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, UIVIEW_DIRS, context); if (result) { return { original: ref, @@ -133,7 +134,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 3: Cell references if (ref.referenceName.endsWith('Cell')) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, CELL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, CELL_DIRS, context); if (result) { return { original: ref, @@ -146,7 +147,7 @@ export const uikitResolver: FrameworkResolver = { // Pattern 4: Delegate/DataSource references if (ref.referenceName.endsWith('Delegate') || ref.referenceName.endsWith('DataSource')) { - const result = resolveByNameAndKind(ref.referenceName, PROTOCOL_KINDS, [], context); + const result = resolveByNameAndKind(ref, PROTOCOL_KINDS, [], context); if (result) { return { original: ref, @@ -206,7 +207,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 1: Controller references if (ref.referenceName.endsWith('Controller')) { - const result = resolveByNameAndKind(ref.referenceName, VAPOR_CONTROLLER_KINDS, VAPOR_CONTROLLER_DIRS, context); + const result = resolveByNameAndKind(ref, VAPOR_CONTROLLER_KINDS, VAPOR_CONTROLLER_DIRS, context); if (result) { return { original: ref, @@ -219,7 +220,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 2: Model references (Fluent) if (/^[A-Z][a-zA-Z]+$/.test(ref.referenceName)) { - const result = resolveByNameAndKind(ref.referenceName, CLASS_KINDS, FLUENT_MODEL_DIRS, context); + const result = resolveByNameAndKind(ref, CLASS_KINDS, FLUENT_MODEL_DIRS, context); if (result) { return { original: ref, @@ -232,7 +233,7 @@ export const vaporResolver: FrameworkResolver = { // Pattern 3: Middleware references if (ref.referenceName.endsWith('Middleware')) { - const result = resolveByNameAndKind(ref.referenceName, VAPOR_CONTROLLER_KINDS, VAPOR_MIDDLEWARE_DIRS, context); + const result = resolveByNameAndKind(ref, VAPOR_CONTROLLER_KINDS, VAPOR_MIDDLEWARE_DIRS, context); if (result) { return { original: ref, @@ -495,29 +496,12 @@ const MODEL_KINDS = new Set(['struct', 'class']); const PROTOCOL_KINDS = new Set(['protocol']); const VAPOR_CONTROLLER_KINDS = new Set(['class', 'struct']); -/** - * Resolve a symbol by name using indexed queries instead of scanning all files. - */ +/** A framework name heuristic's pick (see name-heuristic.ts), preferring these folders. */ function resolveByNameAndKind( - name: string, + ref: UnresolvedRef, kinds: Set, preferredDirPatterns: string[], context: ResolutionContext, ): string | null { - const candidates = context.getNodesByName(name); - if (candidates.length === 0) return null; - - const kindFiltered = candidates.filter((n) => kinds.has(n.kind)); - if (kindFiltered.length === 0) return null; - - // Prefer candidates in framework-conventional directories - if (preferredDirPatterns.length > 0) { - const preferred = kindFiltered.filter((n) => - preferredDirPatterns.some((d) => n.filePath.includes(d)) - ); - if (preferred.length > 0) return preferred[0]!.id; - } - - // Fall back to any match - return kindFiltered[0]!.id; + return pickByNameAndKind(ref, kinds, (f) => preferredDirPatterns.some((d) => f.includes(d)), context); } From 83dac15cffad14e16c402efcd555ad9b84a7a912 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 01:18:09 +0000 Subject: [PATCH 136/363] fix(js): a name the calling function binds is its local (#2226) A JS/TS parameter or plain var/let/const above the reference shadows a same-named function declared elsewhere in the file, whichever strategy found it; destructured call results, member calls and followed aliases are left alone. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/js-function-local-binding.test.ts | 81 +++++++++++++++++++++ src/resolution/name-matcher.ts | 63 ++++++++++++++++ 3 files changed, 145 insertions(+) create mode 100644 __tests__/js-function-local-binding.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 0374fc2041..fe4a555058 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In JavaScript and TypeScript, a name the calling function binds itself, as a parameter or a plain `var`/`let`/`const` above the reference, now means that local instead of a same-named function elsewhere in the file. Before, every lodash helper's `object` parameter linked to a `function object() {}` declared inside `runInContext`, and zod's `new Class(…)` linked to an unrelated `Class` when `Class` was the function's parameter. Destructured hook results like `const { t } = useI18n()` still link to the function they come from. - Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. - In Rust projects, a struct, handler or service name no longer resolves to an item declared inside some function body, and a file's own item comes before any other crate's. Before, axum's examples' `Uri` linked to a `struct Uri` declared inside a routing test, tokio's runtime `Handle` linked to `tokio-test`'s, and ripgrep's `Glob { … }` in globset linked to the CLI flags' `Glob`. - A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. diff --git a/__tests__/js-function-local-binding.test.ts b/__tests__/js-function-local-binding.test.ts new file mode 100644 index 0000000000..a0360929bd --- /dev/null +++ b/__tests__/js-function-local-binding.test.ts @@ -0,0 +1,81 @@ +/** + * A JS/TS name the calling function binds itself — a parameter, or a `var` / + * `let` / `const` above the reference — is that local, never a same-named + * function declared elsewhere in the file. Every lodash helper lives inside + * `runInContext`, so `baseHas(object, key)`'s `object` and `mixin`'s + * `object(this.__wrapped__)` reached a `function object() {}` an IIFE declares + * there. A function that calls itself through its own `const` still does, and + * `if (handler) handler()` is not a parameter list. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-js-local-')); + const files: Record = { + 'lodash.js': `function runInContext(context) { + var baseCreate = (function() { + function object() {} + return function(proto) { + object.prototype = proto; + return new object; + }; + }()); + + function baseHas(object, key) { + return object != null && hasOwnProperty.call(object, key); + } + + function mixin(object, source) { + var result = object(this.__wrapped__); + return result; + } + + function handler() {} + + function run() { + if (handler) { + handler(); + } + const walk = (node) => (node ? walk(node.next) : null); + return walk(context); + } + + return { baseCreate, baseHas, mixin, run }; +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (qualifiedName: string) => { + const ids = cg.getNodesInFile('lodash.js').filter((n) => n.qualifiedName === qualifiedName).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!.qualifiedName); +}; + +describe('JS names the calling function binds', () => { + it('are its parameters, not a same-named function elsewhere in the file', () => { + expect(targetsFrom('runInContext::baseHas')).not.toContain('runInContext::object'); + expect(targetsFrom('runInContext::mixin')).not.toContain('runInContext::object'); + }); + + it('leave unbound names and a const’s own recursion alone', () => { + expect(targetsFrom('runInContext::run')).toContain('runInContext::handler'); + expect(targetsFrom('runInContext::run::walk')).toContain('runInContext::run::walk'); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 9cb0459b6b..482690b328 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -546,6 +546,12 @@ export function matchFunctionRef( // test's parameter of its name receives. if (ref.language === 'python' && !candidates.some((n) => isFixtureInReach(n, ref.filePath, context)) && isPythonLocallyBound(ref.referenceName, ref, context)) return null; + // Likewise a JS/TS parameter or local: lodash's `baseHas(object, key)` passes its own `object`. + const jsLocal = jsFunctionLocalScope(ref.referenceName, ref, context); + if (jsLocal) { + candidates = candidates.filter((n) => n.filePath === ref.filePath && n.startLine >= jsLocal.start && n.startLine <= jsLocal.end); + if (candidates.length === 0) return null; + } // Swift implicit-self: a bare identifier can name a METHOD only of the // ENCLOSING type (`Button(action: handleTap)` written inside that type) — @@ -4230,6 +4236,51 @@ export function isRustNameInScope(candidate: Node, ref: UnresolvedRef, context: const TYPE_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field', 'enum_member']); /** Per-context memo: `file\0name` → "the file binds this name locally". */ +/** Whether `n` lies outside the function that binds the reference's name itself (see jsFunctionLocalScope). */ +function isOutsideJsLocal(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + // `const indexName = this.dataSource.namingStrategy.indexName(…)`: a member, whatever the local's name. + if (ref.referenceKind === 'calls' && bareCallReceiver(ref, context) !== null) return false; + const scope = jsFunctionLocalScope(ref.referenceName, ref, context); + return scope !== null && !(n.filePath === ref.filePath && n.startLine >= scope.start && n.startLine <= scope.end); +} + +const JS_FN_LOCAL_MEMO = new WeakMap>(); + +/** + * The lines of the JS/TS function a reference sits in when that function binds + * the name itself — a parameter, or a `var`/`let`/`const` above the reference. + * Such a name is the local, never a same-named function declared elsewhere: + * every lodash helper lives inside `runInContext`, so `baseHas(object, key)`'s + * `object` and `mixin`'s `object(this.__wrapped__)` reached a `function + * object() {}` an IIFE declares there. Null when the function does not bind it. + */ +function jsFunctionLocalScope(name: string, ref: UnresolvedRef, context: ResolutionContext): { start: number; end: number } | null { + if (!JS_FAMILY.has(ref.language) || !/^[A-Za-z_$][\w$]*$/.test(name)) return null; + let memo = JS_FN_LOCAL_MEMO.get(context); + if (!memo) JS_FN_LOCAL_MEMO.set(context, (memo = new Map())); + const key = `${ref.fromNodeId}\0${name}\0${ref.line}`; + const hit = memo.get(key); + if (hit !== undefined) return hit; + let scope: { start: number; end: number } | null = null; + const fn = context.getNodeById?.(ref.fromNodeId); + if (fn && (fn.kind === 'function' || fn.kind === 'method') && fn.startLine <= ref.line && fn.endLine >= ref.line) { + const lines = context.getFileLines?.(ref.filePath) ?? context.readFile(ref.filePath)?.split(/\r?\n/) ?? []; + const text = stripCommentsForRegex(lines.slice(fn.startLine - 1, ref.line).join('\n'), 'javascript'); + const { param } = localBindingPatterns(name, 'g'); + const n = name.replace(/\$/g, '\\$'); + // A plain declaration. Destructuring re-binds what a call returns under + // the same name — `const { t } = useI18n()`, `const { getLabel } = + // useProps(props)` — which is the same-named function more often than not. + const declared = new RegExp(`\\b(?:const|let|var)\\s+${n}\\b(?!\\s*[,\\]}])`).test(text); + // A parameter list — never a control-flow head (`if (openMarkerClose) {`). + // A return type stays on its line, never a ternary's `: data.slice()` below `filter(canRowExpand)`. + const parameter = new RegExp(`(?>(); /** @@ -8387,6 +8438,15 @@ export function matchReference( // inside `execute()`: a member of what the receiver is, which is the calling // method only through a TS/JS field of the caller's own type. if (result && result.targetNodeId === ref.fromNodeId && isCollapsedNonRecursion(ref, context)) return null; + // A name the calling JS/TS function binds itself shadows the file's own: + // lodash's `mixin(object, …)` calling `object(this.__wrapped__)` is its + // parameter, whichever strategy (fuzzy included) found a `function object`. + if (result && JS_LOCAL_REF_KINDS.has(ref.referenceKind)) { + const target = context.getNodeById?.(result.targetNodeId); + // (A target of another name is what the local was followed to: `const + // selected = useStore(s => s.reset); selected()` is the store's `reset`.) + if (target && target.name === ref.referenceName && isOutsideJsLocal(target, ref, context)) return null; + } // Nor does a value's initializer call the value: sttp's `val response = // basicRequest.get(…).response(asStringAlways)` is a request's `response`. if (result && result.targetNodeId === ref.fromNodeId && ref.referenceKind === 'calls' && @@ -8394,6 +8454,9 @@ export function matchReference( return result ? retargetSelfOverload(result, ref, context) : result; } +/** Reference kinds a bare JS/TS local can be: a call, a value, a construction. */ +const JS_LOCAL_REF_KINDS: ReadonlySet = new Set(['calls', 'references', 'function_ref', 'instantiates']); + /** Node kinds that hold a value rather than run code. */ const VALUE_KINDS: ReadonlySet = new Set(['variable', 'constant', 'field', 'property']); From 3e6252f38bfbabaab31c48388361bb28517efae3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 01:25:35 +0000 Subject: [PATCH 137/363] fix(svelte,vue): a component's instance script declarations are its own (#2227) isVisibleAcrossFiles rejects a declaration in a .svelte instance + +
{items.length}
+`, + 'src/lib/examples/item-demo.svelte': ` + +{label} +`, + 'src/components/TabItem.vue': ` + + +`, + 'src/extensions/code-block.ts': `import { isActive } from "@/tiptap/core"; + +export function active(state: unknown) { + return isActive(state); +} +`, + 'src/components/CrudTable.vue': ` + + +`, + 'src/pages/GroupDataPage.vue': ` + + +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const targetsFrom = (file: string) => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg.getOutgoingEdgesFrom(ids).filter((e) => e.kind !== 'contains').map((e) => cg.getNode(e.target)!).map((t) => `${t.filePath}:${t.name}`); +}; + +describe('single-file component declarations', () => { + it('are private to the component', () => { + expect(targetsFrom('src/lib/examples/item-demo.svelte')).not.toContain('src/lib/examples/attachment-group.svelte:Item'); + expect(targetsFrom('src/extensions/code-block.ts')).not.toContain('src/components/TabItem.vue:isActive'); + }); + + it('except a type a Vue +`); + const file = r.nodes.find((n) => n.kind === 'file')!; + const component = componentOf(r); + expect(file).toMatchObject({ id: 'file:pages/index.vue', startLine: 1, endLine: 9 }); + const p = parents(r); + expect(p.get(component.id)).toEqual([file.id]); + expect(p.get(file.id)).toBeUndefined(); + const load = r.nodes.find((n) => n.name === 'load')!; + expect(p.get(load.id)).toEqual([component.id]); + for (const [, list] of p) expect(list).toHaveLength(1); + }); + + it('gives a +`); + expect(from(r, 'useFetch').filter((id) => id.startsWith('component:'))).toHaveLength(1); + expect(r.unresolvedReferences.filter((ref) => ref.referenceKind === 'imports').every((ref) => ref.fromNodeId === 'file:pages/index.vue')).toBe(true); + }); + + it('leaves a plain +`); + expect(from(r, 'registerWidget')).toEqual(['file:src/Widget.vue']); + }); + + it('gives a Svelte instance script to the component, and a module script to the file', () => { + const r = extractFromSource('src/routes/+page.svelte', ` + + +

items

+`); + const component = componentOf(r).id; + expect(from(r, 'onMount')).toEqual([component]); + expect(from(r, 'preload')).toEqual(['file:src/routes/+page.svelte']); + expect(from(r, 'register')).toEqual(['file:src/routes/+page.svelte']); + }); + + it('gives Astro frontmatter to the component, and makes a file node for a file without one', () => { + const r = extractFromSource('src/pages/blog.astro', `--- +const posts = await getCollection('blog'); +--- +
    {posts.map((p) =>
  • {p.title}
    • )}
+`); + expect(from(r, 'getCollection')).toEqual([componentOf(r).id]); + const bare = extractFromSource('src/pages/about.astro', `

about

\n`); + expect(bare.nodes.map((n) => n.kind).sort()).toEqual(['component', 'file']); + }); +}); diff --git a/src/extraction/astro-extractor.ts b/src/extraction/astro-extractor.ts index e38989375f..efdc33f795 100644 --- a/src/extraction/astro-extractor.ts +++ b/src/extraction/astro-extractor.ts @@ -2,6 +2,7 @@ import { Node, Edge, ExtractionResult, ExtractionError, UnresolvedReference } fr import { generateNodeId } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { isLanguageSupported } from './grammars'; +import { foldScriptResult, sfcFileNode } from './sfc-script'; /** * Astro built-in components — compiler-provided (``) or shipped by @@ -45,8 +46,10 @@ export class AstroExtractor { const startTime = Date.now(); try { - // Create component node for the .astro file itself + // The file, holding the component the .astro file is + this.nodes.push(sfcFileNode(this.filePath, this.source, 'astro')); const componentNode = this.createComponentNode(); + this.edges.push({ source: `file:${this.filePath}`, target: componentNode.id, kind: 'contains' }); // Extract and process the frontmatter block (--- fenced, TypeScript) const frontmatter = this.extractFrontmatter(); @@ -199,45 +202,13 @@ export class AstroExtractor { const extractor = new TreeSitterExtractor(this.filePath, block.content, 'typescript'); const result = extractor.extract(); - // Offset line numbers from the block back to .astro file positions - for (const node of result.nodes) { - node.startLine += block.startLine; - node.endLine += block.startLine; - node.language = 'astro'; // Mark as astro, not TS - - this.nodes.push(node); - - // Add containment edge from component to this node - this.edges.push({ - source: componentNodeId, - target: node.id, - kind: 'contains', - }); - } - - // Offset edges (they reference line numbers) - for (const edge of result.edges) { - if (edge.line) { - edge.line += block.startLine; - } - this.edges.push(edge); - } - - // Offset unresolved references - for (const ref of result.unresolvedReferences) { - ref.line += block.startLine; - ref.filePath = this.filePath; - ref.language = 'astro'; - this.unresolvedReferences.push(ref); - } - - // Carry over errors - for (const error of result.errors) { - if (error.line) { - error.line += block.startLine; - } - this.errors.push(error); - } + // Frontmatter runs on every render of the component, and a +`, + 'pages/admin.vue': ` +`, + 'pages/orders/index.vue': ` +`, + 'components/index.vue': ` +`, + }, + astro: { + 'package.json': JSON.stringify({ name: 'site', private: true, dependencies: { astro: '^4.0.0' } }), + 'src/pages/index.astro': `--- +const title = 'Home'; +--- +

{title}

+`, + 'src/pages/about.astro': `

about

+`, + 'src/pages/blog/index.astro': `

blog

+`, + 'src/pages/api/hello.ts': `export const GET = async () => new Response('hi'); +export async function POST() { return new Response('ok'); } +`, + }, +}; + +const graphs: Record = {}; + +beforeAll(async () => { + for (const [name, files] of Object.entries(projects)) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), `cg-page-${name}-`)); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + graphs[name] = { root, cg: await CodeGraph.init(root, { index: true }) }; + } +}); + +afterAll(() => { + for (const { root, cg } of Object.values(graphs)) { + cg.close(); + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +/** route name → the files of what it calls or references. */ +function served(cg: CodeGraph): Record { + const out: Record = {}; + for (const route of cg.getNodesByKind('route')) { + out[route.name] = cg.getOutgoingEdgesFrom([route.id], ['calls', 'references']) + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind} ${n.filePath}`); + } + return out; +} + +describe('a Nuxt page', () => { + it('is served by its own file’s component', () => { + const map = served(graphs.nuxt!.cg); + expect(map['/']).toEqual(['component pages/index.vue']); + expect(map['/admin']).toEqual(['component pages/admin.vue']); + expect(map['/orders']).toEqual(['component pages/orders/index.vue']); + }); + + it('is listed with the component that serves it', () => { + const { entries } = buildRoutes(graphs.nuxt!.cg, new URLSearchParams()); + expect(entries.map((e) => `${e.url} ${e.handler} ${e.file}`).sort()).toEqual([ + '/ index pages/index.vue', + '/admin admin pages/admin.vue', + '/orders index pages/orders/index.vue', + ]); + }); +}); + +describe('an Astro page', () => { + it('is served by its own file’s component, and an endpoint by the verbs it exports', () => { + const map = served(graphs.astro!.cg); + expect(map['/']).toEqual(['component src/pages/index.astro']); + expect(map['/about']).toEqual(['component src/pages/about.astro']); + expect(map['/blog']).toEqual(['component src/pages/blog/index.astro']); + expect(map['/api/hello']).toEqual(['function src/pages/api/hello.ts', 'function src/pages/api/hello.ts']); + }); +}); diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 7376e5517c..71b2a2e10d 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -125,6 +125,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel ## Fixes — Screens, links and navigation +- **Routes served by a component are listed.** Entry points and the Steps picker listed a route only when a function, method or class served it. A Vue Router, Nuxt, Svelte or Astro screen is served by a component, so those apps' routes weren't listed at all. They are now, each with the component that serves it. + - **Where the app goes after login is a fork, not two always-es.** A navigation whose destination comes back from a helper — `router.replace(await resolvePostLoginRoute())` over `return (await hasSeenWelcome(…)) ? '/home/' : '/welcome/'` — drew both screens with no condition, reading as if the welcome screen always shows. The two arms share a line, and only a column can tell them apart; each synthesized edge now carries its literal's own position, so the guard reader says which arm it is: `WHEN await hasSeenWelcome(…)` → home, and its negation → welcome. And the scan starts at the helper's body, so a literal-union return type — `Promise<'/welcome/' | '/home/'>`, whose routes are string literals too, written first — no longer stands in for the navigation itself. Re-index after upgrading to pick the positions up. - **A screen that talks to native code keeps its own navigations.** In a React Native or Expo app, a `router.push` written inside a listener for a native event was credited to whichever screen had *started* that round trip, not to the screen the push is written on. In one app that moved seven transitions off the capture screen and onto the review screen it opens — leaving the review screen looking as though nothing in the app could reach it, stranded in the "no transition reaches this" band at the bottom of the Screens tab, and printing Swift conditions like `Thread.isMainThread` on a JavaScript navigation. A navigation now belongs to the screen whose file it is written in; an event arriving from native code, from an HTTP call or off a queue is no longer read backwards as if it were a caller. diff --git a/src/db/queries.ts b/src/db/queries.ts index 443b851372..655f98a981 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -1204,7 +1204,8 @@ export class QueryBuilder { if (!this.stmts.getRoutingManifest) { // Edge kind varies across framework resolvers: Spring/Rails/ // Laravel/Drupal emit `references`, Express emits `calls`. Accept - // both — the semantic is the same (route → its handler). + // both — the semantic is the same (route → its handler). A screen in + // a Vue / Svelte / Astro app is served by a `component`. this.stmts.getRoutingManifest = this.db.prepare(` SELECT r.name AS url, @@ -1220,7 +1221,7 @@ export class QueryBuilder { JOIN nodes h ON e.target = h.id WHERE r.kind = 'route' AND e.kind IN ('references', 'calls') - AND h.kind IN ('function', 'method', 'class', 'constant', 'variable') + AND h.kind IN ('function', 'method', 'class', 'constant', 'variable', 'component') ORDER BY r.file_path, r.start_line LIMIT ? `); diff --git a/src/resolution/frameworks/astro.ts b/src/resolution/frameworks/astro.ts index 4645335bad..635d19e971 100644 --- a/src/resolution/frameworks/astro.ts +++ b/src/resolution/frameworks/astro.ts @@ -7,6 +7,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; +import { pageComponentRef, resolvePageComponent } from './page-component'; /** * Astro virtual module prefixes — framework-provided, not user code @@ -47,6 +48,10 @@ export const astroResolver: FrameworkResolver = { }, resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // A page route names the component its file is. + const page = resolvePageComponent(ref, context); + if (page) return page; + // Pattern 1: the `Astro` global (Astro.props, Astro.url, Astro.params, …) // — runtime-provided in every component's frontmatter. Resolving it as // framework-provided keeps it from name-matching a user symbol named Astro. @@ -95,8 +100,9 @@ export const astroResolver: FrameworkResolver = { return null; }, - extract(filePath: string, _content: string) { + extract(filePath: string, content: string) { const nodes: Node[] = []; + const references: UnresolvedRef[] = []; const now = Date.now(); // Normalize to forward slashes @@ -118,8 +124,9 @@ export const astroResolver: FrameworkResolver = { !/\.config\.[a-z]+$/.test(base) ) { const routePath = filePathToAstroRoute(afterPages); - - nodes.push({ + const isPage = normalized.endsWith('.astro'); + const language = isPage ? 'astro' : 'typescript'; + const route: Node = { id: `route:${filePath}:${routePath}:1`, kind: 'route', name: routePath, @@ -129,16 +136,31 @@ export const astroResolver: FrameworkResolver = { endLine: 1, startColumn: 0, endColumn: 0, - language: normalized.endsWith('.astro') ? 'astro' : 'typescript', + language, updatedAt: now, - }); + }; + nodes.push(route); + if (isPage) { + references.push(pageComponentRef(route, '.astro', 'astro')); + } else { + // An endpoint is served by the verbs it exports: `export const GET: + // APIRoute = …`, `export async function POST(…)`. + for (const m of content.matchAll(ENDPOINT_EXPORT)) { + const verb = m[1] ?? m[2]!; + const line = content.slice(0, m.index).split('\n').length; + references.push({ fromNodeId: route.id, referenceName: verb, referenceKind: 'references', line, column: 0, filePath, language, candidates: [verb] }); + } + } } } - return { nodes, references: [] }; + return { nodes, references }; }, }; +/** An Astro endpoint's exported handler: `export const GET`, `export async function POST`. */ +const ENDPOINT_EXPORT = /^\s*export\s+(?:(?:async\s+)?function\s+(GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|ALL)\b|const\s+(GET|POST|PUT|PATCH|DELETE|OPTIONS|HEAD|ALL)\b)/gm; + /** * Check if string is PascalCase */ diff --git a/src/resolution/frameworks/page-component.ts b/src/resolution/frameworks/page-component.ts new file mode 100644 index 0000000000..38d311b113 --- /dev/null +++ b/src/resolution/frameworks/page-component.ts @@ -0,0 +1,44 @@ +/** + * A file-routed page — Nuxt's `pages/admin.vue`, Astro's `src/pages/about.astro` + * — is served by the component its file IS: the single-file-component + * extractor's one component per file, named after the file. Without the link a + * page route stood alone, so the viewer's Steps tab drew nothing for a page and + * the routes list never named what serves it. + * + * The route says so with a `calls` reference by that name, the way a Next.js + * page names its default export, and {@link resolvePageComponent} lands it on + * the SAME file's component — never on a same-named one elsewhere, since every + * `index.vue` is a component named `index`. + */ + +import type { Language, Node } from '../../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from '../types'; + +/** The component name a single-file-component extractor gives a file: its basename, extension off. */ +export function pageComponentName(filePath: string, extension: string): string { + const base = filePath.split(/[/\\]/).pop() ?? filePath; + return base.endsWith(extension) ? base.slice(0, -extension.length) : base; +} + +export function pageComponentRef(route: Node, extension: string, language: Language): UnresolvedRef { + const name = pageComponentName(route.filePath, extension); + return { + fromNodeId: route.id, + referenceName: name, + referenceKind: 'calls', + line: 1, + column: 0, + filePath: route.filePath, + language, + candidates: [name], + }; +} + +/** A page route's reference to its file's own component, or null when this is not one. */ +export function resolvePageComponent(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + if (ref.referenceKind !== 'calls' || !ref.fromNodeId.startsWith(`route:${ref.filePath}:`)) return null; + const component = context + .getNodesInFile(ref.filePath) + .find((n) => n.kind === 'component' && n.name === ref.referenceName); + return component ? { original: ref, targetNodeId: component.id, confidence: 0.95, resolvedBy: 'framework' } : null; +} diff --git a/src/resolution/frameworks/vue.ts b/src/resolution/frameworks/vue.ts index e894e9b468..6a5d9027c4 100644 --- a/src/resolution/frameworks/vue.ts +++ b/src/resolution/frameworks/vue.ts @@ -8,6 +8,7 @@ import { Node } from '../../types'; import { FrameworkResolver, UnresolvedRef, ResolvedRef, ResolutionContext } from '../types'; import { dependsOn } from './package-deps'; +import { pageComponentRef, resolvePageComponent } from './page-component'; /** The languages a Vue app's scripts are written in. */ const VUE_SCRIPT_LANGUAGES: ReadonlySet = new Set(['vue', 'javascript', 'typescript', 'tsx', 'jsx']); @@ -212,12 +213,14 @@ export const nuxtResolver: FrameworkResolver = { return context.getAllFiles().some((f) => /(?:^|\/)nuxt\.config\.(?:[cm]?[jt]s)$/.test(f)); }, - resolve(): ResolvedRef | null { - return null; + resolve(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + // A page route names the component its file is. + return resolvePageComponent(ref, context); }, extract(filePath: string, _content: string) { const nodes: Node[] = []; + const references: UnresolvedRef[] = []; const now = Date.now(); // Forward slashes, and a leading `/` so an app at the repository root @@ -229,7 +232,7 @@ export const nuxtResolver: FrameworkResolver = { if (pagesIndex !== -1 && normalized.endsWith('.vue')) { const routePath = filePathToNuxtRoute(normalized, pagesIndex + '/pages/'.length); if (routePath !== null) { - nodes.push({ + const route: Node = { id: `route:${filePath}:${routePath}:1`, kind: 'route', name: routePath, @@ -241,7 +244,9 @@ export const nuxtResolver: FrameworkResolver = { endColumn: 0, language: 'vue', updatedAt: now, - }); + }; + nodes.push(route); + references.push(pageComponentRef(route, '.vue', 'vue')); } } @@ -292,7 +297,7 @@ export const nuxtResolver: FrameworkResolver = { }); } - return { nodes, references: [] }; + return { nodes, references }; }, }; From 104b8de3e6325315456c861c6cb6648916b81b46 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Thu, 1 Oct 2026 21:40:38 +0000 Subject: [PATCH 168/363] fix(laravel): a string controller resolves without a Controller suffix (#2271) Only `XController@action` was claimed and resolved, and the written namespace was stripped, so akaunting's 'Common\\Uploads@inline' (and a Route::resource with a string controller plus options) bound 304 of 313 routes to nothing. Any `Class@action` is claimed; the namespace path under App\\Http\\Controllers is kept and picks between same-named controllers; a resource reads its first argument and resolves its controller class in the Laravel resolver (no `use` brings a string-named class into scope). Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/laravel-controller-strings.test.ts | 74 +++++++++++++ src/resolution/frameworks/laravel.ts | 104 ++++++++++++++----- 3 files changed, 152 insertions(+), 27 deletions(-) create mode 100644 __tests__/laravel-controller-strings.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index d71cb51f33..e165649c7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- In Laravel apps, a route whose controller is named by string now links to its handler, without needing a `Controller` suffix and keeping the namespace it's written with: `'Common\Uploads@inline'`, and `Route::resource('companies', 'Common\Companies', [...])` with an options array. Two same-named controllers in different namespaces are told apart by that namespace. Before, akaunting linked 304 of its 313 routes to nothing. - In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. - In Rails apps, a `resources` line's `only:` and `except:` are now read in every form Rails accepts: `%i[new create index]`, `%w(index show)`, a quoted name, and the older `:only => [...]`. Before, only a bracketed list was read, so maybe's `resources :family_exports, only: %i[new create index]` became all seven routes, four of them to actions the controller doesn't have. - In Vue, Svelte and Astro files, a component now owns its script, and what a `\n{$title}\n'), + ).toBe('php'); + // Prose that happens to put Pascal keywords on their own lines. + expect(detectLanguage('f.inc', '
  • \n Implementation\n
    • \n
  • \n Begin\n
    • \n')).toBe('php'); + }); + + it('keeps the path-only answer (no source) on PHP', () => { + expect(detectLanguage('defs.inc')).toBe('php'); + expect(detectLanguage('defs.inc', '')).toBe('php'); + }); + + it('lets an explicit codegraph.json mapping for .inc win', () => { + expect(detectLanguage('defs.inc', DIRECTIVES, { '.inc': 'php' })).toBe('php'); + expect(detectLanguage('defs.inc', '#define X 1\n', { '.inc': 'cpp' })).toBe('cpp'); + expect(detectLanguage('page.inc', ' { + it('loads the Pascal grammar when an .inc file is in the set', () => { + expect(preloadLanguagesForFiles(['defs.inc'])).toEqual(expect.arrayContaining(['php', 'pascal'])); + }); + + it('adds nothing for a PHP project without .inc files, or when .inc is mapped explicitly', () => { + expect(preloadLanguagesForFiles(['index.php', 'a.module'])).not.toContain('pascal'); + expect(preloadLanguagesForFiles(['defs.inc'], { '.inc': 'php' })).not.toContain('pascal'); + }); + }); + + describe('index and sync', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-pascal-inc-')); + clearProjectConfigCache(); + }); + afterEach(() => { + clearProjectConfigCache(); + fs.rmSync(dir, { recursive: true, force: true }); + }); + const write = (rel: string, body: string) => { + const p = path.join(dir, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, body); + }; + const languageOf = (cg: CodeGraph, rel: string) => cg.getFiles().find((f) => f.path === rel)?.language; + const symbolsIn = (cg: CodeGraph, rel: string) => + cg.getNodesInFile(rel).filter((n) => n.kind !== 'file').map((n) => `${n.kind}:${n.name}:${n.language}`).sort(); + + // Runs first, before anything in this file has loaded the Pascal grammar + // through a `.pas` file: an include-only tree must still get a parser. + it('indexes a tree of Pascal includes with no .pas file', async () => { + write('inc/defs.inc', DIRECTIVES); + write('inc/routines.inc', ROUTINES); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(languageOf(cg, 'inc/defs.inc')).toBe('pascal'); + expect(languageOf(cg, 'inc/routines.inc')).toBe('pascal'); + expect(symbolsIn(cg, 'inc/routines.inc')).toEqual(['function:Bar:pascal', 'function:Foo:pascal']); + expect(cg.getFiles().find((f) => f.path === 'inc/routines.inc')?.errors ?? []).toEqual([]); + } finally { + cg.close(); + } + }); + + it('classifies each include by content, and sync keeps the language', async () => { + write('U.pas', 'unit U;\n\ninterface\n\n{$I defs.inc}\n\nprocedure Foo;\n\nimplementation\n\n{$I routines.inc}\n\nend.\n'); + write('defs.inc', DIRECTIVES); + write('routines.inc', ROUTINES); + write('drupal/mymodule.inc', ' f.path === 'defs.inc')?.errors ?? []).toEqual([]); + expect(symbolsIn(cg, 'routines.inc')).toEqual(['function:Bar:pascal', 'function:Foo:pascal']); + expect(symbolsIn(cg, 'drupal/mymodule.inc')).toEqual(['function:mymodule_helper:php']); + + write('defs.inc', DIRECTIVES + '{$DEFINE USE_FAST_MM}\n'); + write('routines.inc', ROUTINES + 'procedure Baz;\nbegin\nend;\n'); + await cg.sync(); + + expect(languageOf(cg, 'defs.inc')).toBe('pascal'); + expect(languageOf(cg, 'routines.inc')).toBe('pascal'); + expect(symbolsIn(cg, 'routines.inc')).toEqual(['function:Bar:pascal', 'function:Baz:pascal', 'function:Foo:pascal']); + expect(languageOf(cg, 'drupal/mymodule.inc')).toBe('php'); + } finally { + cg.close(); + } + }); + + it('honors a codegraph.json .inc mapping through index and sync', async () => { + write('codegraph.json', JSON.stringify({ extensions: { '.inc': 'php' } })); + write('defs.inc', DIRECTIVES); + const cg = await CodeGraph.init(dir, { index: true }); + try { + expect(languageOf(cg, 'defs.inc')).toBe('php'); + write('defs.inc', DIRECTIVES + '{$DEFINE MORE}\n'); + await cg.sync(); + expect(languageOf(cg, 'defs.inc')).toBe('php'); + } finally { + cg.close(); + } + }); + }); +}); diff --git a/scripts/kernel-parity.mjs b/scripts/kernel-parity.mjs index d02ab6d6c6..5209369d8c 100644 --- a/scripts/kernel-parity.mjs +++ b/scripts/kernel-parity.mjs @@ -198,8 +198,12 @@ process.env.CODEGRAPH_KERNEL_LANGS = 'all'; for (const { file, lang: extLang } of files) { const source = fs.readFileSync(file, 'utf8'); const rel = path.relative(ROOT, file); - // `.h` resolves C vs C++ by content — the same call the indexer makes. - const lang = extLang === 'detect' || extLang === 'javascript' || extLang === 'jsx' ? detectLanguage(rel, source) : extLang; + // `.h` resolves C vs C++ by content — the same call the indexer makes; so + // does `.inc` (PHP vs Pascal, #2279 — a Pascal include is not a kernel file). + const lang = + extLang === 'detect' || extLang === 'javascript' || extLang === 'jsx' || path.extname(file).toLowerCase() === '.inc' + ? detectLanguage(rel, source) + : extLang; if (!KERNEL_LANGS.has(lang)) continue; if (langFilter && !langFilter.has(lang)) continue; processed++; diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index 5913349f3b..6a05b96c0d 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -561,9 +561,101 @@ export function detectLanguage(filePath: string, source?: string, overrides?: Re if (looksLikeObjc(source)) return 'objc'; } + // `.inc` is PHP's include extension (Drupal) and Pascal/Delphi's too + // (`{$I defs.inc}`: directive blocks, declaration fragments), so it is + // decided per file by content (#2279). An explicit codegraph.json mapping + // for `.inc` is the user's answer and is not second-guessed. + if (lang === 'php' && ext === '.inc' && source && !(overrides && overrides[ext]) && looksLikePascalInclude(source)) { + return 'pascal'; + } + return lang; } +/** A PHP open tag: `\n]*>)?`; +const PAS_QNAME = String.raw`[a-z_][\w.]*(?:<[^>\n]*>)?`; +const PAS_COMMENT = String.raw`(?:\/\/[^\n]*|\{[^$}\n][^}\n]*\}[ \t]*)?`; +const PAS_SECTION_BREAK = String.raw`[ \t]*${PAS_COMMENT}(?:\r?\n[ \t]*${PAS_COMMENT})+`; +const pascalLine = (shape: string): RegExp => new RegExp(PAS_LINE + shape, 'im'); + +/** + * Line shapes only Pascal writes, any one of which makes an untagged `.inc` + * Pascal. Each leans on Pascal's own punctuation, so the dialects that share a + * keyword with it stay out: JavaScript `const x = 1;` / `function f() {`, + * SourcePawn `function void (int client);`, C++ `const T X::Y = …`, VBScript + * `Const X = 1` / `Function F(a)`, Smarty `{$var}`, Makefile `X := y`, prose + * with `Begin` on a line of its own. Every pattern stays linear on a long + * whitespace run: no run can be split two ways between neighbouring + * quantifiers. + */ +const PASCAL_INCLUDE_SIGNALS: readonly RegExp[] = [ + // A compiler directive: a name and an argument (`{$IFDEF X}`, `{$DEFINE X}`, + // `{$I file.inc}`, `{$WARN X OFF}`), a bare `{$ELSE}` / `{$ENDIF}` / + // `{$IFEND}`, or a switch (`{$R-}`, `{$A+,B-}`) — or the `{%MainUnit x.pp}` + // line Lazarus opens its include files with. + pascalLine(String.raw`\{(?:\$(?:[a-z]\w*[ \t]+[^\s}]|(?:else|endif|ifend)[ \t]*\}|[a-z][+-][,}])|%MainUnit\b)`), + // A routine header, closed by `;`: `procedure Foo;`, `procedure TForm1.Click(Sender: TObject);`, + // `class constructor Create;` — and a function's result type after its + // parameters (`function Bar(A: Integer): string;`; bare `function Bar;` is + // the implementation-section short form). The parameter list stops at any + // parenthesis, so a file of unclosed `procedure X(` lines stays linear. + pascalLine( + String.raw`(?:class[ \t]+)?(?:(?:procedure|constructor|destructor)[ \t]+${PAS_QNAME}[ \t]*(?:\([^()]*\)[ \t]*)?` + + String.raw`|function[ \t]+${PAS_QNAME}[ \t]*(?:(?:\([^()]*\)[ \t]*)?:[ \t]*[\w.]+(?:<[^>\n]*>)?[ \t]*)?);` + ), + // `unit Foo;` and a `uses A, B;` clause. + pascalLine(String.raw`unit[ \t]+[a-z_][\w.]*[ \t]*;`), + pascalLine(String.raw`uses\s+[a-z_][\w.]*(?:\s*,\s*[a-z_][\w.]*)*\s*;`), + // A `const` section, then `X = …` / `X: T = …`; or a typed constant on one + // line (`const Max: Integer = 10;`). The type never holds a `:`. + pascalLine( + String.raw`(?:const|resourcestring)(?:${PAS_SECTION_BREAK}[a-z_]\w*[ \t]*(?::[^=;:\n]+)?=|[ \t]+[a-z_]\w*[ \t]*:[^=;:\n]+=)` + ), + // A `var` section: `G, H: Integer;`, on the keyword's line or below it. + pascalLine( + String.raw`(?:var|threadvar)(?:${PAS_SECTION_BREAK}|[ \t]+)[a-z_]\w*(?:[ \t]*,[ \t]*[a-z_]\w*)*[ \t]*:(?!:)[^;\n]*;` + ), + // A `type` section, then `TFoo =`; or `type TFoo = class…` (record / + // interface / set of / array / procedure type) on one line. + pascalLine( + String.raw`type(?:${PAS_SECTION_BREAK}${PAS_NAME}[ \t]*=|[ \t]+${PAS_NAME}[ \t]*=[ \t]*(?:packed[ \t]+)?` + + String.raw`(?:class|record|object|interface|dispinterface|set[ \t]+of|array|reference[ \t]+to|procedure|function)\b)` + ), +]; + +/** A `begin` … `end;` block: both halves needed, so neither alone flips a file. */ +const PASCAL_BEGIN_RE = pascalLine(String.raw`begin\b`); +const PASCAL_END_RE = pascalLine(String.raw`end[ \t]*[;.][ \t]*$`); + +/** + * Whether an `.inc` file is a Pascal include rather than a PHP one (#2279). + * + * A PHP include always opens a PHP tag somewhere, so a tag anywhere keeps the + * file PHP. Without one, a Pascal-only line shape (`PASCAL_INCLUDE_SIGNALS`, + * or a `begin` … `end;` pair) makes it Pascal. Anything else keeps the PHP + * mapping — untagged text is inline HTML to PHP, so nothing is extracted — + * rather than handing a C / assembly / POV-Ray / ASP `.inc` to the Pascal + * grammar's error recovery. + * + * Deliberately per file, not "does this project have `.pas` files": the + * answer depends only on the file's own bytes, so a full index, a sync of one + * edited include, and a fresh re-index always agree — a project-level gate + * would flip an untouched include whenever the last `.pas` file came or went. + */ +function looksLikePascalInclude(source: string): boolean { + if (PHP_OPEN_TAG_RE.test(source)) return false; + if (PASCAL_INCLUDE_SIGNALS.some((re) => re.test(source))) return true; + return PASCAL_BEGIN_RE.test(source) && PASCAL_END_RE.test(source); +} + /** Whether a JavaScript file's leading comments carry Flow's `@flow` pragma (and not `@noflow`). */ export function hasFlowPragma(source: string): boolean { const head = source.slice(0, 4096).replace(/^#![^\n]*\n/, ''); diff --git a/src/extraction/index.ts b/src/extraction/index.ts index f87eeacd4c..0ca3145be2 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -808,6 +808,15 @@ export function preloadLanguagesForFiles( if (!languages.includes(ambiguous)) languages.push(ambiguous); } } + // An `.inc` path-detects as PHP but may read as Pascal (#2279) — unless + // codegraph.json maps `.inc` explicitly, which detectLanguage never overrides. + if ( + !languages.includes('pascal') && + !(overrides && overrides['.inc']) && + files.some((f) => f.toLowerCase().endsWith('.inc')) + ) { + languages.push('pascal'); + } return languages; } From a4bd890ec634f00c8e5e418ec8e4c0fe6babcaad Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 04:42:23 +0000 Subject: [PATCH 181/363] fix(python): a call on self is never resolved through a same-named import (#2290) The Python extractor records `self.get_ip(request)` by its bare name, so the import strategy matched it to a function the file imports as `get_ip` and the call never reached the class's own (or inherited) method. A bare-named Python call written on `self` now skips import resolution; the method strategies then find it. A plain `get_ip(request)` still binds to the import. Across allauth, DRF, netbox and mealie this retargets 16 calls, all from an imported helper to the class's own method; no edge is added or lost. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/python-self-call-not-import.test.ts | 94 +++++++++++++++++++ src/resolution/index.ts | 9 +- src/resolution/name-matcher.ts | 9 ++ 4 files changed, 111 insertions(+), 2 deletions(-) create mode 100644 __tests__/python-self-call-not-import.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a5b87f38b..cff9f381b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -166,6 +166,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - When an agent asks CodeGraph about a separate git repository nested inside an indexed project, one the project's index leaves out (for example because the parent's `.gitignore` excludes it), it now gets the usual "isn't indexed" guidance instead of answers from the parent project's code. Nested repositories the parent does index, such as submodules, work as before. Thanks @wstczyw for the report. (#2110) - Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) - Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ` { + const dirs: string[] = []; + let cg: CodeGraph | undefined; + + afterEach(() => { + cg?.close(); + cg = undefined; + for (const dir of dirs.splice(0)) fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function callsIn(files: Record): Promise { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-py-self-')); + dirs.push(dir); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); + fs.writeFileSync(path.join(dir, rel), content); + } + cg = await CodeGraph.init(dir, { index: true }); + const rows = (cg as any).db.db + .prepare( + `SELECT s.qualified_name s, t.qualified_name t, t.file_path f FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target WHERE e.kind = 'calls'`, + ) + .all() as { s: string; t: string; f: string }[]; + return rows.map((r) => `${r.s} -> ${r.t} (${r.f})`).sort(); + } + + const httpkit = 'def get_ip(request):\n return "1.2.3.4"\n'; + + it('a method calling its own same-named method through self', async () => { + expect( + await callsIn({ + 'pkg/__init__.py': '', + 'pkg/httpkit.py': httpkit, + 'pkg/adapter.py': [ + 'from pkg.httpkit import get_ip', + '', + '', + 'class Adapter:', + ' def get_ip(self, request):', + ' return get_ip(request)', + '', + ' def notify(self, request):', + ' return self.get_ip(request)', + '', + ].join('\n'), + }), + ).toEqual([ + // The bare call is still the imported function. + 'Adapter::get_ip -> get_ip (pkg/httpkit.py)', + 'Adapter::notify -> Adapter::get_ip (pkg/adapter.py)', + ]); + }); + + it('a method a base class in another file declares', async () => { + expect( + await callsIn({ + 'pkg/__init__.py': '', + 'pkg/httpkit.py': httpkit, + 'pkg/base.py': 'class Base:\n def get_ip(self, request):\n return "base"\n', + 'pkg/adapter.py': [ + 'from pkg.base import Base', + 'from pkg.httpkit import get_ip', + '', + '', + 'class Adapter(Base):', + ' def notify(self, request):', + ' return self.get_ip(request)', + '', + ' def raw(self, request):', + ' return get_ip(request)', + '', + ].join('\n'), + }), + ).toEqual([ + 'Adapter::notify -> Base::get_ip (pkg/base.py)', + 'Adapter::raw -> get_ip (pkg/httpkit.py)', + ]); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index b0c80e01ac..f4bb91a237 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { matchJsStoreBindingCall, isUnresolvedJsMemberCall, isVisibleAcrossFiles, matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, CASE_INSENSITIVE_LANGUAGES } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, isVisibleAcrossFiles, matchReference, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, CASE_INSENSITIVE_LANGUAGES } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; @@ -1240,7 +1240,12 @@ export class ReferenceResolver { } const tImp = this.profileStages ? process.hrtime.bigint() : 0n; - const importResult = this.gateLanguage(resolveViaImport(ref, this.context), ref); + // `self.get_ip()` is a method call on the instance even when the file + // also imports a function named `get_ip`: the import never names it. + const selfCall = ref.language === 'python' && ref.referenceKind === 'calls' && + this.context.getImportMappings(ref.filePath, ref.language).some((m) => m.localName === ref.referenceName) && + isPythonSelfCall(ref, this.context); + const importResult = selfCall ? null : this.gateLanguage(resolveViaImport(ref, this.context), ref); if (this.profileStages) this.stageAdd('viaImport', ref, !!importResult, tImp); if (importResult) { if (importResult.confidence >= 0.9) return importResult; diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 16609dd72f..1f5d6f8a2f 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -1714,6 +1714,15 @@ function bareCallReceiver(ref: UnresolvedRef, context: ResolutionContext): { rec return { receiver: head.slice(-40), links }; } +/** + * Whether a Python call recorded by its bare name was written on the instance, + * `self.get_ip(request)`. The extractor drops the `self.`, so a same-named name + * the file imports would otherwise claim the method call (#2074 follow-up). + */ +export function isPythonSelfCall(ref: UnresolvedRef, context: ResolutionContext): boolean { + return ref.language === 'python' && bareCallReceiver(ref, context)?.receiver === 'self'; +} + /** Whether a call recorded by its bare name is written on something other than the caller's own object. */ function isCollapsedNonRecursion(ref: UnresolvedRef, context: ResolutionContext): boolean { const written = bareCallReceiver(ref, context); From f838c5a02ab05cbd938039ae89aef9542604eb57 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 04:47:36 +0000 Subject: [PATCH 182/363] fix(resolution): resolve method values on untyped Python module globals (#2074) (#2291) * fix(resolution): resolve method values on untyped Python module globals (#1820) A method value whose receiver is a module global without a static type (`thread_pool_exec(settings.conn.fetch, ...)`, where `conn = None` is rebound by `global conn; conn = Backend()`) resolved to a `variable` node and produced no edge. The global's type is now the set of classes its module assigns to it, at module scope or in a function that declares it `global`. One class resolves to its own method; several bind to the nearest declaration every candidate inherits, as a base-typed receiver does. No edge (a wrong callback edge is worse than none) when: - any binding of the global in its module is not a whole constructor call: a factory, a conditional, a tuple target, `for`/`with`/import, a star import, or an annotation the constructor contradicts; - the caller or an enclosing function binds the name itself: parameter, assignment (any target position), loop, comprehension, `as`, `case` pattern, lambda parameter; - the receiver is a deeper chain (`settings.conn.pool.fetch`); - the importing file rebinds the name, or imports it from two sources. Statements are read with their continuation lines, and only statement lines (bracket depth 0) can assign. Not seen: rebinding the global from another module (`settings.conn = X`) or through `globals()`. `pkg.mod.Cls` now resolves through `import pkg.mod`, unless two namespace imports share the last segment. Python import mappings also read parenthesized from-imports (`from x import (\n A,\n B,\n)`), which previously yielded the single name `(` and hid every class imported that way from receiver and inheritance resolution. Comments and docstrings are stripped first. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 0aab32a18b59cb42d895e0fbc54d885f0b4ee38b) * fix(resolution): account for writes to a Python module global from other files (#2074) The module-global receiver type was read from the global's own module only, so `settings.conn = Decoy()` in another module, or `globals()["conn"] = X`, left a stale type and a wrong edge. - A production write `. = Cls(...)` from another file joins the type set (resolved in the writing file); any other production write (another value, a tuple target, `setattr`/`patch.object`) makes the type unknown. The module is matched through each file's imports, including aliases and relative imports. - Test files install doubles (`settings.conn = MagicMock()`, `monkeypatch.setattr(settings, "conn", ...)`): they do not change the production type, but a ref inside such a test resolves nothing. - `globals()[...] = ...` with the global's name or a computed key, and `globals().update(...)`, in the global's module make the type unknown. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 205571122653ea42b8d00f68fbb4082585a1922c) * fix(resolution): match a module global's writers by file, not dotted suffix Review follow-up to the cross-module write scan. - A write now lands on the file its import names: relative imports resolve exactly, absolute ones by path suffix; when a spelling could name another file sharing the tail (`x/settings.py` and `y/settings.py` for `import settings`), a write through it makes the type unknown instead of feeding or clearing the wrong module. - `import a.b` binds `a`: the module is spelled `a.b`; the mapping's last-segment name is not treated as an alias of it. - Test doubles are recognised by the narrow test-suite set plus any `conftest.py`; examples, fixtures and benchmarks are production writers. - Writes inside `if __name__ == "__main__":` are script code, not module state, in the global's module and elsewhere. - Namespace-dict writes: `globals()`, `vars()` and `sys.modules[__name__]` are read per statement (continuations joined, strings ignored); any use other than a literal-key read or `.get`, or a literal-key write of the name, makes the type unknown. `.__dict__[...]` / `.update(` from another file does too. - `settings.conn: Store = Store()` and `settings.conn = (Store())` are plain constructor writes. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 7aab29269a7fabf3a9b1b9dafaa544ada76f4a7a) * fix(resolution): explicit dotted aliases, one-line __main__ guards, scoped vars() Review follow-up. - `import pkg.settings as settings` binds `settings` even though it equals the last segment; the import mapping cannot tell it from a plain `import pkg.settings`, so the source line decides. - `if __name__ == "__main__": stmt` on one line is script code, and a `globals()` write inside a `__main__` block no longer counts. - `vars()` is the module dict only at module scope; inside a function it is the locals, so `return vars()` there no longer makes the type unknown. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit 0a5fb697970ab21edbf11f3baddfa072bbeae593) * fix(resolution): explicit-alias check matches the exact module, outside strings Review follow-up: `import other.pkg.settings as settings` or a string `"import pkg.settings as settings"` no longer marks a plain `import pkg.settings` as explicitly aliased, which attached a write to the wrong module. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_018CQubRS1bXvaoApgPyLf66 (cherry picked from commit a5569a07ea2c09bdb73e1262007c8d501fccd8ec) * test(resolution): give the Python module-global adversarial table its own timeout It indexes one project per case, which takes about nine seconds on macOS, past vitest's five-second default. Co-Authored-By: Claude Opus 5.5 * docs(changelog): Python module globals and multi-line imports (#2074) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Josh66 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/function-ref.test.ts | 305 ++++++++++++++++++ __tests__/resolution.test.ts | 22 ++ src/resolution/import-resolver.ts | 10 +- src/resolution/name-matcher.ts | 499 ++++++++++++++++++++++++++++++ 5 files changed, 834 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cff9f381b9..cdb9a6a6f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -167,6 +167,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) - Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ` { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-')); + fs.writeFileSync(path.join(tmpDir, 'store.py'), 'class Store:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'decoy.py'), 'class Decoy:\n def fetch(self, ids):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), `from decoy import Decoy +conn = None + +def init(): + global conn + from store import Store + conn = Store() + +def shadow(): + conn = Decoy() + return conn + +def same_file(pool): + pool.submit(conn.fetch, []) + +def param_shadow(pool, conn): + pool.submit(conn.fetch, []) + +def loop_shadow(pool, items): + for conn in items: + pool.submit(conn.fetch, []) +`); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), `import settings +from settings import conn + +def via_module(pool): + pool.submit(settings.conn.fetch, []) + +def via_name(pool): + pool.submit(conn.fetch, []) + +def import_shadow(pool, settings): + pool.submit(settings.conn.fetch, []) + +def local_import(pool): + from settings import conn + pool.submit(conn.fetch, []) +`); + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + const store = cg.getNodesByName('fetch').find(n => n.qualifiedName.startsWith('Store::'))!; + const into = cg.getIncomingEdges(store.id).filter(e => e.kind === 'references' && e.metadata?.fnRef === true); + // A parameter or loop variable of the same name is not the global; a local import of it is. + expect(sourceNames(cg, into)).toEqual(['local_import', 'same_file', 'via_module', 'via_name']); + const decoy = cg.getNodesByName('fetch').find(n => n.qualifiedName.startsWith('Decoy::'))!; + expect(cg.getIncomingEdges(decoy.id).filter(e => e.metadata?.fnRef === true)).toHaveLength(0); + } finally { cg.close(); } + }); + + it('#1820: a module global with several backends binds to the declaration they share', async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-many-')); + fs.mkdirSync(path.join(tmpDir, 'backends')); + fs.writeFileSync(path.join(tmpDir, 'backends', '__init__.py'), ''); + fs.writeFileSync(path.join(tmpDir, 'backends', 'base.py'), + 'class Base:\n def fetch(self, ids):\n raise NotImplementedError("backend")\n' + + ' def search(self):\n raise NotImplementedError("backend")\n'); + fs.writeFileSync(path.join(tmpDir, 'backends', 'es.py'), + 'from backends.base import Base\nclass ES(Base):\n def search(self):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'backends', 'vec.py'), + 'from backends.base import (\n Base,\n)\nclass Vec(Base):\n def fetch(self, ids):\n return ids\n' + + ' def search(self):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'other.py'), 'class Other:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), `import backends.es +conn = None + +def init(engine): + global conn + if engine == "es": + conn = backends.es.ES() + else: + from backends import vec as vec_module + conn = vec_module.Vec() +`); + fs.writeFileSync(path.join(tmpDir, 'unknown.py'), `from other import Other +conn = None +mixed = None + +def init(flag): + global conn, mixed + conn = make_conn() + mixed = Other() if flag else None +`); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), `import settings +import unknown + +def gc(pool): + pool.submit(settings.conn.fetch, []) + +def find(pool): + pool.submit(settings.conn.search) + +def opaque(pool): + pool.submit(unknown.conn.fetch, []) + pool.submit(unknown.mixed.fetch, []) +`); + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + const byOwner = (owner: string, name = 'fetch') => cg.getNodesByName(name).find(n => n.qualifiedName.startsWith(`${owner}::`))!; + const fnRefs = (owner: string, name = 'fetch') => cg.getIncomingEdges(byOwner(owner, name).id) + .filter(e => e.kind === 'references' && e.metadata?.fnRef === true); + // ES inherits Base.fetch and Vec overrides it: the shared declaration is Base.fetch. + expect(sourceNames(cg, fnRefs('Base'))).toEqual(['gc']); + expect(fnRefs('Vec')).toHaveLength(0); + // Both backends override search: the declaration they inherit is still Base.search. + expect(sourceNames(cg, fnRefs('Base', 'search'))).toEqual(['find']); + expect([...fnRefs('ES', 'search'), ...fnRefs('Vec', 'search')]).toHaveLength(0); + // An opaque initializer (`make_conn()`, a conditional) leaves the global's type unknown. + expect(fnRefs('Other')).toHaveLength(0); + } finally { cg.close(); } + }); + + // One index per adversarial case: several seconds on a loaded machine. + it('#1820: a module global never binds through a shadow, an unknown value or a deeper chain', async () => { + const header = `from store import Store +from decoy import Decoy +conn = None + +def init(): + global conn + conn = Store() +`; + // Each case: its files, and the fnRef edges into any `fetch` it must produce. + const cases: Record, string[]]> = { + control: [{ 'm.py': header + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + tuple_first: [{ 'm.py': header + 'def cb(pool):\n conn, _ = (Decoy(), None)\n pool.submit(conn.fetch)\n' }, []], + loop_and_nested_global: [{ 'm.py': header + + 'def cb(pool, items):\n for conn in items:\n pass\n def nested():\n global conn\n pool.submit(conn.fetch)\n' }, []], + multiline_conditional: [{ 'm.py': header.replace('conn = Store()', 'conn = Store(\n x=1,\n ) if flag else Decoy()') + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + multiline_constructor: [{ 'm.py': header.replace('conn = Store()', 'conn = Store(\n x=1,\n )') + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + closure_param: [{ 'm.py': header + 'def outer(conn):\n def cb(pool):\n pool.submit(conn.fetch)\n return cb\n' }, []], + closure_local: [{ 'm.py': header + 'def outer(pool):\n conn = Decoy()\n def cb():\n pool.submit(conn.fetch)\n return cb\n' }, []], + lambda_param: [{ 'm.py': header + 'def cb(pool):\n pool.submit(lambda conn: pool.map(conn.fetch))\n' }, []], + comprehension: [{ 'm.py': header + 'def cb(pool, xs):\n [pool.submit(conn.fetch) for conn in xs]\n' }, []], + except_as: [{ 'm.py': header + 'def cb(pool):\n try:\n pass\n except Exception as conn:\n pool.submit(conn.fetch)\n' }, []], + match_as: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case Decoy() as conn:\n pool.submit(conn.fetch)\n' }, []], + global_in_nested_def: [{ 'm.py': 'from decoy import Decoy\nconn = None\n\ndef init():\n def helper():\n global conn\n conn = Decoy()\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + keyword_in_multiline_call: [{ 'm.py': 'from decoy import Decoy\nconn = None\napp = dict(\n conn=Decoy()\n)\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + tuple_rebind: [{ 'm.py': header + 'def reset():\n global conn\n conn, other = Decoy(), 1\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + with_rebind: [{ 'm.py': header + 'def reset():\n global conn\n with open_decoy() as conn:\n pass\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + docstring: [{ 'm.py': 'from store import Store\nconn = None\n"""\nconn = Store()\n"""\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + deeper_chain: [{ 'settings.py': header, 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.pool.fetch)\n' }, []], + importer_rebinds: [{ 'settings.py': header, + 'c.py': 'from settings import conn\nfrom decoy import Decoy\n\ndef reset():\n global conn\n conn = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + local_import_of_other_module: [{ 'settings.py': header, 'other_settings.py': 'from decoy import Decoy\nconn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n import other_settings as settings\n pool.submit(settings.conn.fetch)\n' }, []], + local_from_import_of_other_module: [{ 'settings.py': header, 'other_settings.py': 'from decoy import Decoy\nconn = Decoy()\n', + 'c.py': 'from settings import conn\n\ndef cb(pool):\n from other_settings import conn\n pool.submit(conn.fetch)\n' }, []], + comment_in_parenthesized_import: [{ 'settings.py': header + 'other = 1\n', + 'c.py': 'from settings import (\n other, # see fetch()\n conn,\n)\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, ['cb -> Store::fetch']], + star_import_override: [{ 'local.py': 'from decoy import Decoy\nconn = Decoy()\n', 'settings.py': header + 'from local import *\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + match_sequence_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case [conn]:\n pool.submit(conn.fetch)\n' }, []], + match_keyword_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case Decoy(inner=conn):\n pool.submit(conn.fetch)\n' }, []], + match_bare_capture: [{ 'm.py': header + 'def cb(pool, x):\n match x:\n case conn:\n pool.submit(conn.fetch)\n' }, []], + multiline_loop_target: [{ 'm.py': header + 'def cb(pool, xs):\n for (a,\n conn) in xs:\n pool.submit(conn.fetch)\n' }, []], + multiline_tuple_local: [{ 'm.py': header + 'def cb(pool):\n (a,\n conn) = 1, Decoy()\n pool.submit(conn.fetch)\n' }, []], + multiline_tuple_rebind: [{ 'm.py': header + 'def reset():\n global conn\n (a,\n conn) = 1, Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + annotation_contradicts_value: [{ 'm.py': 'from store import Store\nfrom decoy import Decoy\nconn: Store = Decoy()\n\n' + + 'def cb(pool):\n pool.submit(conn.fetch)\n' }, []], + annotation_over_factory: [{ 'm.py': 'from store import Store\nconn: Store = make_store()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + external_production_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_aliased_write: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'reset.py': 'import pkg.settings as cfg\n\ndef reset():\n cfg.conn = make_conn()\n', + 'c.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_relative_write: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'pkg/reset.py': 'from . import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'c.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_setattr: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n setattr(settings, "conn", Decoy())\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_subclass_write: [{ 'settings.py': header, + 'sub.py': 'from store import Store\nclass SubStore(Store):\n def fetch(self, ids):\n return ids\n', + 'reset.py': 'import settings\nfrom sub import SubStore\n\ndef reset():\n settings.conn = SubStore()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + test_double_write: [{ 'settings.py': header, + 'tests/test_c.py': 'import settings\nfrom unittest.mock import MagicMock\n\ndef test_cb(pool):\n settings.conn = MagicMock()\n pool.submit(settings.conn.fetch)\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + test_monkeypatch: [{ 'settings.py': header, + 'tests/test_c.py': 'import settings\n\ndef test_cb(monkeypatch, pool):\n monkeypatch.setattr(settings, "conn", object())\n pool.submit(settings.conn.fetch)\n' }, []], + globals_literal_write: [{ 'm.py': header + 'def reset():\n globals()["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_dynamic_write: [{ 'm.py': header + 'for name, obj in [("conn", Decoy())]:\n globals()[name] = obj\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + unrelated_attribute_write: [{ 'settings.py': header, + 'other.py': 'from decoy import Decoy\n\nclass Holder:\n def __init__(self):\n self.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + // A write lands on the module its import names, never on another module sharing its tail. + write_path_is_exact: [{ 'x/__init__.py': '', 'y/__init__.py': '', 'x/settings.py': header, 'y/settings.py': header, + 'reset.py': 'from x import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'cx.py': 'from x import settings\n\ndef cb_x(pool):\n pool.submit(settings.conn.fetch)\n', + 'cy.py': 'from y import settings\n\ndef cb_y(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb_y -> Store::fetch']], + ambiguous_write_target: [{ 'x/__init__.py': '', 'y/__init__.py': '', 'x/settings.py': header, 'y/settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'cx.py': 'from x import settings\n\ndef cb_x(pool):\n pool.submit(settings.conn.fetch)\n', + 'cy.py': 'from y import settings\n\ndef cb_y(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + relative_write_is_exact: [{ 'pkg/__init__.py': '', 'pkg/sub/__init__.py': '', 'pkg/settings.py': header, + 'other/__init__.py': '', 'other/pkg/__init__.py': '', 'other/pkg/settings.py': header, + 'pkg/sub/reset.py': 'from .. import settings\n\ndef reset():\n settings.conn = make_conn()\n', + 'c2.py': 'from other.pkg import settings\n\ndef cb2(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb2 -> Store::fetch']], + namespace_import_binds_root: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'c.py': 'import pkg.settings\nfrom decoy import Decoy\n\nsettings = Bag()\nsettings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + // `other.pkg.settings as settings` is not an explicit alias of `pkg.settings`: the + // write goes to the other module, so pkg's global keeps its type (the consumer's + // relative import names pkg/settings.py exactly). + alias_of_longer_module_is_not_explicit: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'other/__init__.py': '', 'other/pkg/__init__.py': '', 'other/pkg/settings.py': 'conn = None\n', + 'reset.py': 'import pkg.settings\nimport other.pkg.settings as settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'pkg/d.py': 'from . import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + alias_in_string_is_not_explicit: [{ 'pkg/__init__.py': '', 'pkg/settings.py': 'conn = None\n', + 'reset.py': 'import pkg.settings\nfrom decoy import Decoy\nnote = "import pkg.settings as settings"\n\ndef reset():\n settings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + explicit_alias_equal_to_leaf: [{ 'pkg/__init__.py': '', 'pkg/settings.py': header, + 'reset.py': 'import pkg.settings as settings\nfrom decoy import Decoy\n\ndef reset():\n settings.conn = Decoy()\n', + 'd.py': 'from pkg import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + one_line_main_write: [{ 'settings.py': header, + 'run.py': 'import settings\nfrom decoy import Decoy\n\nif __name__ == "__main__": settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + globals_in_main_block: [{ 'm.py': header + 'if __name__ == "__main__":\n globals()["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + vars_in_function_is_locals: [{ 'm.py': header + 'def dump():\n return vars()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + example_write_is_production: [{ 'settings.py': header, + 'examples/wire.py': 'import settings\nfrom decoy import Decoy\n\ndef wire():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + conftest_write_is_test: [{ 'settings.py': header, + 'pkg/conftest.py': 'import settings\nfrom decoy import Decoy\n\ndef fixture():\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + main_block_write: [{ 'settings.py': header, + 'run.py': 'import settings\nfrom decoy import Decoy\n\nif __name__ == "__main__":\n settings.conn = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + own_main_block_write: [{ 'm.py': header + 'if __name__ == "__main__":\n conn = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + globals_continued_key: [{ 'm.py': header + 'def reset():\n globals()[\n "conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_alias: [{ 'm.py': header + 'def reset():\n g = globals()\n g["conn"] = Decoy()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + globals_in_string: [{ 'm.py': header + 'NOTE = "globals().update(mapping)"\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, + ['cb -> Store::fetch']], + external_dict_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom decoy import Decoy\n\ndef reset():\n settings.__dict__["conn"] = Decoy()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, []], + external_annotated_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom store import Store\n\ndef reset():\n settings.conn: Store = Store()\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + external_parenthesized_write: [{ 'settings.py': header, + 'reset.py': 'import settings\nfrom store import Store\n\ndef reset():\n settings.conn = (Store())\n', + 'c.py': 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n' }, ['cb -> Store::fetch']], + dotted_import_collision: [{ 'alpha/__init__.py': '', 'beta/__init__.py': '', + 'alpha/foo.py': 'class Client:\n def fetch(self):\n return 1\n', 'beta/foo.py': 'class Client:\n def fetch(self):\n return 2\n', + 'm.py': 'import beta.foo\nimport alpha.foo\nconn = None\n\ndef init():\n global conn\n conn = alpha.foo.Client()\n\ndef cb(pool):\n pool.submit(conn.fetch)\n' }, []], + dotted_base_collision: [{ 'alpha/__init__.py': '', 'beta/__init__.py': '', + 'alpha/foo.py': 'class Client:\n def fetch(self):\n return 1\n', 'beta/foo.py': 'class Client:\n def fetch(self):\n return 2\n', + 'm.py': 'import beta.foo\nimport alpha.foo\n\nclass Sub(alpha.foo.Client):\n pass\n\ndef go(pool, s: Sub):\n pool.submit(s.fetch)\n' }, []], + }; + const got: Record = {}; + for (const [name, [files, _]] of Object.entries(cases)) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), `cg-fnref-global-${name}-`)); + const all = { 'store.py': 'class Store:\n def fetch(self, ids):\n return ids\n', + 'decoy.py': 'class Decoy:\n def fetch(self, ids):\n return []\n', ...files }; + for (const [file, content] of Object.entries(all)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, file)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, file), content); + } + const cg = CodeGraph.initSync(tmpDir); + try { + await cg.indexAll(); + got[name] = cg.getNodesByName('fetch').flatMap(t => cg.getIncomingEdges(t.id) + .filter(e => e.kind === 'references' && e.metadata?.fnRef === true) + .map(e => `${cg.getNode(e.source)?.name} -> ${t.qualifiedName}`)).sort(); + } finally { cg.close(); fs.rmSync(tmpDir, { recursive: true, force: true }); tmpDir = undefined; } + } + expect(got).toEqual(Object.fromEntries(Object.entries(cases).map(([name, [, want]]) => [name, want]))); + }, 60_000); + + it('#1820: a module global re-resolves after its module changes (sync)', async () => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-global-sync-')); + const settings = 'from store import Store\nfrom decoy import Decoy\nconn = None\n\ndef init():\n global conn\n conn = Store()\n'; + const consumer = 'import settings\n\ndef cb(pool):\n pool.submit(settings.conn.fetch)\n'; + fs.writeFileSync(path.join(tmpDir, 'store.py'), 'class Store:\n def fetch(self, ids):\n return ids\n'); + fs.writeFileSync(path.join(tmpDir, 'decoy.py'), 'class Decoy:\n def fetch(self, ids):\n return []\n'); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), settings); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), consumer); + const cg = CodeGraph.initSync(tmpDir); + const edges = () => cg.getNodesByName('fetch').flatMap(t => cg.getIncomingEdges(t.id) + .filter(e => e.metadata?.fnRef === true).map(e => `${cg.getNode(e.source)?.name} -> ${t.qualifiedName}`)); + try { + await cg.indexAll(); + expect(edges()).toEqual(['cb -> Store::fetch']); + fs.writeFileSync(path.join(tmpDir, 'settings.py'), settings.replace('conn = Store()', 'conn = Decoy()')); + fs.writeFileSync(path.join(tmpDir, 'consumer.py'), consumer + '\n# touched\n'); + await cg.sync(); + expect(edges()).toEqual(['cb -> Decoy::fetch']); + } finally { cg.close(); } + }); + it('#1820: Go receiver types disambiguate method values and reject external fields', async () => { tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-fnref-go-scope-')); fs.writeFileSync(path.join(tmpDir, 'main.go'), `package demo diff --git a/__tests__/resolution.test.ts b/__tests__/resolution.test.ts index 9bdf768cd0..14b5198f47 100644 --- a/__tests__/resolution.test.ts +++ b/__tests__/resolution.test.ts @@ -648,6 +648,28 @@ from ..services import auth_service expect(mappings.some((m) => m.localName === 'helper')).toBe(true); expect(mappings.some((m) => m.localName === 'User')).toBe(true); }); + + it('should extract parenthesized Python from-imports', () => { + const content = ` +from store.base import ( + Base, + helper as h, # a comment, with a comma and a call() +) +""" +from docstring import NotAnImport +""" +from store.rows import (Row, Col) +`; + + const mappings = extractImportMappings('src/main.py', content, 'python'); + + expect(mappings.map((m) => [m.localName, m.exportedName, m.source])).toEqual([ + ['Base', 'Base', 'store.base'], + ['h', 'helper', 'store.base'], + ['Row', 'Row', 'store.rows'], + ['Col', 'Col', 'store.rows'], + ]); + }); }); describe('JVM FQN Import Resolution', () => { diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index dc3b583959..7fa9912e23 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -11,6 +11,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, ReExport import { applyAliases } from './path-aliases'; import { extractLocalExportAliases } from './alias-binding'; import { resolveWorkspaceImport } from './workspace-packages'; +import { stripCommentsForRegex } from './strip-comments'; import { resolveMethodOnType, resolveObjectLiteralMember, @@ -937,7 +938,7 @@ export function extractImportMappings( // whole SFC (markup + styles included) is safe. mappings.push(...extractJSImports(content)); } else if (language === 'python') { - mappings.push(...extractPythonImports(content)); + mappings.push(...extractPythonImports(stripCommentsForRegex(content, 'python'))); } else if (language === 'go') { mappings.push(...extractGoImports(content)); } else if (language === 'java' || language === 'kotlin') { @@ -1066,13 +1067,14 @@ function extractJSImports(content: string): ImportMapping[] { function extractPythonImports(content: string): ImportMapping[] { const mappings: ImportMapping[] = []; - // from X import Y - const fromImportRegex = /from\s+([\w.]+)\s+import\s+([^#\n]+)/g; + // from X import Y, and the parenthesized form `from X import (\n Y,\n Z,\n)` + const fromImportRegex = /from\s+([\w.]+)\s+import\s+(\([^)]*\)|[^#\n]+)/g; let match; while ((match = fromImportRegex.exec(content)) !== null) { const [, source, imports] = match; - const names = imports!.split(',').map((s) => s.trim()); + const names = imports!.trim().replace(/^\(|\)$/g, '') + .split(',').map((s) => s.trim()); for (const name of names) { const aliasMatch = name.match(/(\w+)\s+as\s+(\w+)/); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 1f5d6f8a2f..701c0ea03c 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -281,6 +281,17 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): if (ref.language === 'python') { const cls = pythonRefClass(receiver, ref, context); if (cls) return result(pythonMembers(cls, member, ref, context)); + // `mod.global` / an imported `global` itself — never a deeper chain (`mod.global.field`). + const segments = receiver.split('.'); + const hit = segments.length <= 2 ? context.resolveImport?.({ ...ref, referenceName: receiver, referenceKind: 'references' }) : null; + const global = hit && context.getNodeById?.(hit.targetNodeId); + // A local import of the root binds what the file imports, unless the file imports it from two places. + if (global && global.name === segments[segments.length - 1] && isPythonModuleGlobal(global, context) && + pythonImportKeys(segments[0]!, ref.filePath, context).size === 1 && + !pythonGlobalBindings(segments[0]!, ref.filePath, context).some(b => b.kind !== 'import') && + !pythonBindsLocally(segments[0]!, ref, context, false)) { + return result(pythonGlobalMembers(global, member, ref, context)); + } } const imported = context.resolveImport?.(ref); const node = imported && context.getNodeById?.(imported.targetNodeId); @@ -305,6 +316,11 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): type = pythonFieldType(receiver, owner, ref, context); } else { type = pythonLocalType(receiver, ref, context); + const global = type === null + ? context.getNodesInFile(ref.filePath).find(n => n.name === receiver && isPythonModuleGlobal(n, context)) : undefined; + if (global && !pythonBindsLocally(receiver, ref, context, true)) { + return result(pythonGlobalMembers(global, member, ref, context)); + } } // A type name used directly (`Store.fetch`) is scoped just like an annotation. if (!type && /^[A-Z]\w*$/.test(receiver)) type = receiver; @@ -337,6 +353,14 @@ function matchMemberFunctionRef(ref: UnresolvedRef, context: ResolutionContext): function pythonRefClass(name: string, ref: UnresolvedRef, context: ResolutionContext): Node | null { const imports = context.getImportMappings(ref.filePath, 'python'); + // `import pkg.mod` then `pkg.mod.Cls`: the mapping keys the module by its last segment. + const module = imports.find(i => i.isNamespace && name.startsWith(`${i.source}.`) && + /^\w+$/.test(name.slice(i.source.length + 1))); + if (module) { + // Two modules with the same last segment (`import a.foo`, `import b.foo`) share the key: refuse. + if (imports.filter(i => i.localName === module.localName).length !== 1) return null; + name = `${module.localName}.${name.slice(module.source.length + 1)}`; + } if (imports.some(i => i.localName === name.split('.')[0])) { const hit = context.resolveImport?.({ ...ref, referenceName: name, referenceKind: 'references' }); const node = hit && context.getNodeById?.(hit.targetNodeId); @@ -411,6 +435,478 @@ function pythonLocalType(receiver: string, ref: UnresolvedRef, context: Resoluti return caller?.signature?.match(new RegExp(`\\b${receiver}\\s*:\\s*["']?([\\w.]+)`))?.[1] ?? null; } +/** A module-scope Python variable (not a class attribute or a function local). */ +function isPythonModuleGlobal(node: Node, context: ResolutionContext): boolean { + return node.language === 'python' && (node.kind === 'variable' || node.kind === 'constant') && + !context.getNodesInFile(node.filePath).some(n => + (n.kind === 'class' || n.kind === 'function' || n.kind === 'method') && + n.startLine <= node.startLine && n.endLine >= node.startLine); +} + +/** How one line binds a name: `global`, a plain `name = value` / `name: T`, an import, or any other binding. */ +type PythonBinding = + | { kind: 'global' } + | { kind: 'assign'; type: string | null; value: string; line: number } + | { kind: 'import'; key: string } + | { kind: 'other' }; + +const PYTHON_STATEMENT_STARTS = new WeakMap>(); +/** Per line: does it start a statement (bracket depth 0, no `\` continuation)? String contents are skipped. */ +function pythonStatementStarts(filePath: string, context: ResolutionContext): boolean[] { + let files = PYTHON_STATEMENT_STARTS.get(context); + if (!files) { files = new Map(); PYTHON_STATEMENT_STARTS.set(context, files); } + let starts = files.get(filePath); + if (starts) return starts; + starts = []; + let depth = 0; + let continued = false; + for (const line of pythonMemberLines(filePath, context)) { + starts.push(depth === 0 && !continued); + let quote = ''; + for (let i = 0; i < line.length; i++) { + const c = line[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") quote = c; + else if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth = Math.max(0, depth - 1); + } + continued = /\\\s*$/.test(line); + } + files.set(filePath, starts); + return starts; +} + +/** Line indexes that belong to `scope` itself (null: the module), not to a def or class nested in it. */ +function pythonOwnLines(scope: Node | null, filePath: string, context: ResolutionContext): number[] { + const count = pythonMemberLines(filePath, context).length; + const from = scope ? scope.startLine : 1; + const to = scope ? Math.min(scope.endLine, count) : count; + const nested = new Uint8Array(to - from + 1); + for (const n of context.getNodesInFile(filePath)) { + if ((n.kind !== 'function' && n.kind !== 'method' && n.kind !== 'class') || n.id === scope?.id) continue; + if (n.startLine < from || n.endLine > to || (scope && n.startLine <= scope.startLine)) continue; + nested.fill(1, n.startLine - from, n.endLine - from + 1); + } + const own: number[] = []; + for (let l = from; l <= to; l++) if (!nested[l - from]) own.push(l - 1); + return own; +} + +/** Split `a = b = value` at its top-level assignment operators; null when the line assigns nothing. */ +function pythonAssignment(line: string): { targets: string[]; value: string; augmented: boolean } | null { + const targets: string[] = []; + let depth = 0; + let quote = ''; + let start = 0; + let augmented = false; + for (let i = 0; i < line.length; i++) { + const c = line[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") { quote = c; continue; } + if (c === '(' || c === '[' || c === '{') { depth++; continue; } + if (c === ')' || c === ']' || c === '}') { depth--; continue; } + if (c !== '=' || depth !== 0) continue; + const prev = line[i - 1] ?? ''; + if (line[i + 1] === '=') { i++; continue; } // == + if (prev === '!' || prev === ':') continue; // != and the walrus + if ((prev === '<' || prev === '>') && line[i - 2] !== prev) continue; // <= >= + const op = line.slice(start, i).match(/(?:\/\/|\*\*|>>|<<|[-+*/%&|^@])$/)?.[0]; + if (op) augmented = true; + targets.push(line.slice(start, i - (op?.length ?? 0))); + start = i + 1; + } + return targets.length ? { targets, value: line.slice(start), augmented } : null; +} + +/** + * Every way the statement starting at line `index` binds `name`, read across + * its continuation lines. Only statement lines can assign or import; any line + * can bind through `as`, a loop, a lambda or the walrus. + */ +function pythonLineBindings(lines: string[], index: number, statements: boolean[], name: string): PythonBinding[] { + let line = lines[index]!; + if (statements[index]) for (let j = index + 1; j < lines.length && !statements[j]; j++) line += '\n' + lines[j]; + const out: PythonBinding[] = []; + // An import statement can name `name` on a continuation line: `from x import (\n name,\n)`. + const imported = statements[index] ? line.match(/^\s*(?:from\s+([\w.]+)\s+)?import\s+([\s\S]*)$/) : null; + if (imported) { + const names = imported[2]!; + // `from x import *` can bind any name. + if (imported[1] && names.trim() === '*') return [{ kind: 'other' }]; + if (!names.includes(name)) return out; + for (const part of names.replace(/[()\\]/g, ' ').split(',')) { + const m = part.trim().match(/^([\w.]+)(?:\s+as\s+(\w+))?$/); + const local = m && (m[2] ?? (imported[1] ? m[1]! : m[1]!.split('.')[0]!)); + if (local !== name) continue; + out.push({ kind: 'import', key: imported[1] ? `${imported[1]}:${m![1]}` : `${m![2] ? m![1] : local}:*` }); + } + return out; + } + if (!line.includes(name)) return out; + const word = new RegExp(`(? s.trim() === name) + ? [declared[1] === 'global' ? { kind: 'global' } : { kind: 'other' }] : []; + } + if (statements[index]) { + // A `case` pattern binds its capture names (`case [name]:`, `case Cls(k=name):`, `case name:`). + if (/^\s*case\b/.test(line)) return [{ kind: 'other' }]; + const assignment = pythonAssignment(line); + if (assignment) { + const target = assignment.targets.length === 1 && !assignment.augmented ? assignment.targets[0]!.trim() : ''; + const annotated = target.match(new RegExp(`^${name}\\s*:\\s*["']?([\\w.]+)["']?$`)); + if (target === name || annotated) { + out.push({ kind: 'assign', type: annotated?.[1] ?? null, value: assignment.value.trim(), line: index }); + } else if (assignment.targets.some(t => word.test(t))) { + out.push({ kind: 'other' }); + } + } else { + const annotated = line.match(new RegExp(`^\\s*${name}\\s*:\\s*["']?([\\w.]+)["']?\\s*$`)); + if (annotated) out.push({ kind: 'assign', type: annotated[1]!, value: '', line: index }); + else if (new RegExp(`^\\s*del\\b`).test(line)) out.push({ kind: 'other' }); + } + } + if (new RegExp(`\\b${name}[ \\t]*:=|\\bas[ \\t]+${name}\\b`).test(line)) out.push({ kind: 'other' }); + for (const loop of line.matchAll(/\bfor\s+([^:]+?)\s+in\b/g)) if (word.test(loop[1]!)) out.push({ kind: 'other' }); + for (const lambda of line.matchAll(/\blambda\b([^:]*):/g)) if (word.test(lambda[1]!)) out.push({ kind: 'other' }); + return out; +} + +/** + * Whether `name`, read at the ref, is bound by the calling function or one that + * encloses it (parameter, assignment, loop, `as`, lambda, import) rather than + * being the module global. With `importsBind` false, an import of the name is + * not a shadow: it binds the same module the file imports. + */ +function pythonBindsLocally(name: string, ref: UnresolvedRef, context: ResolutionContext, importsBind: boolean): boolean { + const lines = pythonMemberLines(ref.filePath, context); + const statements = pythonStatementStarts(ref.filePath, context); + const param = new RegExp(`[(,]\\s*\\*{0,2}${name}\\s*[:=,)]`); + const scopes = context.getNodesInFile(ref.filePath).filter(n => + (n.kind === 'function' || n.kind === 'method') && n.startLine <= ref.line && n.endLine >= ref.line) + .sort((a, b) => b.startLine - a.startLine); + for (const scope of scopes) { + const own = pythonOwnLines(scope, ref.filePath, context); + const bindings = own.flatMap(i => pythonLineBindings(lines, i, statements, name)); + if (bindings.some(b => b.kind === 'global')) return false; + const def = own.find(i => /^\s*(?:async\s+)?def\b/.test(lines[i]!)); + let header = scope.signature ?? ''; + for (let j = def ?? lines.length; j < lines.length && (j === def || !statements[j]); j++) header += lines[j]; + if (param.test(header)) return true; + if (bindings.some(b => b.kind !== 'global' && (b.kind !== 'import' || importsBind))) return true; + } + return false; +} + +/** Distinct sources a file imports `name` from, at any scope (`from a import x` → `a:x`). */ +function pythonImportKeys(name: string, filePath: string, context: ResolutionContext): Set { + return pythonNameScan(context, `imports\0${filePath}\0${name}`, () => scanPythonImportKeys(name, filePath, context)); +} + +const PYTHON_NAME_SCANS = new WeakMap>(); +/** Per-file, per-name scans are shared by every ref in the file; cleared with the other memos on sync. */ +function pythonNameScan(context: ResolutionContext, key: string, scan: () => T): T { + let memo = PYTHON_NAME_SCANS.get(context); + if (!memo) { memo = new Map(); PYTHON_NAME_SCANS.set(context, memo); } + if (!memo.has(key)) memo.set(key, scan()); + return memo.get(key) as T; +} + +function scanPythonImportKeys(name: string, filePath: string, context: ResolutionContext): Set { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const keys = new Set(); + for (let i = 0; i < lines.length; i++) { + if (!statements[i] || !/\bimport\b/.test(lines[i]!)) continue; + for (const b of pythonLineBindings(lines, i, statements, name)) if (b.kind === 'import') keys.add(b.key); + } + return keys; +} + +/** Every binding of module global `name` in `filePath`: at module scope, and in each function that declares it `global`. */ +function pythonGlobalBindings(name: string, filePath: string, context: ResolutionContext): PythonBinding[] { + return pythonNameScan(context, `globals\0${filePath}\0${name}`, () => scanPythonGlobalBindings(name, filePath, context)); +} + +function scanPythonGlobalBindings(name: string, filePath: string, context: ResolutionContext): PythonBinding[] { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const scopes = context.getNodesInFile(filePath).filter(n => n.kind === 'class' || n.kind === 'function' || n.kind === 'method'); + const regions = [pythonOwnLines(null, filePath, context)]; + for (let i = 0; i < lines.length; i++) { + if (!/^\s*global\b/.test(lines[i]!) || !pythonLineBindings(lines, i, statements, name).length) continue; + const scope = scopes.filter(n => n.startLine <= i + 1 && n.endLine >= i + 1).sort((a, b) => b.startLine - a.startLine)[0]; + if (scope && scope.kind !== 'class') regions.push(pythonOwnLines(scope, filePath, context)); + } + const script = pythonMainBlockLines(filePath, context); + return regions.flatMap(region => region.filter(i => !script.has(i)).flatMap(i => pythonLineBindings(lines, i, statements, name))) + .filter(b => b.kind !== 'global'); +} + +const PYTHON_GLOBAL_CLASSES = new WeakMap>(); +/** + * The classes a module global can hold. It has no static type (`conn = None`, + * rebound by `global conn; conn = Backend()`), so its type is the set of + * classes its module assigns to it: at module scope, or in a function that + * declares it `global`. Any other binding of it there (an opaque value, a + * tuple target, `for`/`with`/import, a star import, a `globals()` write) + * makes the type unknown (null). Production writes from other modules + * (`settings.conn = X`) join the set; see pythonExternalWrites. + */ +function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + let memo = PYTHON_GLOBAL_CLASSES.get(context); + if (!memo) { memo = new Map(); PYTHON_GLOBAL_CLASSES.set(context, memo); } + if (memo.has(global.id)) return memo.get(global.id)!; + const file = global.filePath; + const external = pythonExternalWrites(global, context); + // Each type is resolved in the file that wrote it (its imports name the class). + const writes: Array<{ type: string; file: string }> = [...external.writes]; + let classes: Node[] | null = external.unknown || pythonDynamicGlobalWrite(global.name, file, context) ? null : []; + for (const b of classes ? pythonGlobalBindings(global.name, file, context) : []) { + if (b.kind !== 'assign') { classes = null; break; } + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (b.type) { + // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) { classes = null; break; } + writes.push({ type: b.type, file }); + continue; + } + if (b.value === 'None') continue; + if (!constructor) { classes = null; break; } + writes.push({ type: constructor, file }); + } + const seen = new Set(); + for (const write of classes ? writes : []) { + const cls = pythonRefClass(write.type, { ...ref, filePath: write.file }, context); + if (!cls) { classes = null; break; } + if (!seen.has(cls.id)) { seen.add(cls.id); classes!.push(cls); } + } + memo.set(global.id, classes); + return classes; +} + +/** + * The repo files a Python module path can name from `fromFile`. Relative + * paths (`..settings`) resolve exactly; absolute ones match a file path + * suffix, so a source root (`src/`) still resolves — and two files sharing + * the tail (`x/settings.py`, `y/settings.py`) both come back. + */ +function pythonModuleFiles(dotted: string, fromFile: string, context: ResolutionContext): string[] { + const dots = dotted.match(/^\.+/)?.[0].length ?? 0; + const parts = dotted.slice(dots).split('.').filter(Boolean); + if (!parts.length) return []; + let dir = ''; + if (dots) { + dir = path.posix.dirname(fromFile.replace(/\\/g, '/')); + for (let i = 1; i < dots; i++) dir = path.posix.dirname(dir); + if (dir === '.') dir = ''; + } + const rel = [dir, ...parts].filter(Boolean).join('/'); + const matches = (file: string, want: string) => dots ? file === want : file === want || file.endsWith(`/${want}`); + const last = parts[parts.length - 1]!; + return [ + ...context.getNodesByName(`${last}.py`), ...context.getNodesByName(`${last}.pyi`), ...context.getNodesByName('__init__.py'), + ].filter(n => n.kind === 'file' && (matches(n.filePath, `${rel}.py`) || matches(n.filePath, `${rel}.pyi`) || matches(n.filePath, `${rel}/__init__.py`))) + .map(n => n.filePath); +} + +/** + * How `filePath` spells the module `moduleFile`: `aliases` import exactly that + * file; `ambiguous` could also be another file sharing its dotted tail. + * `import a.b` binds `a`, so that module is spelled `a.b` (the mapping's + * last-segment `localName` is not a binding); `import a.b as c` binds `c`. + */ +function pythonModuleAliases(filePath: string, moduleFile: string, context: ResolutionContext): { aliases: string[]; ambiguous: string[] } { + const aliases = new Set(); + const ambiguous = new Set(); + for (const m of context.getImportMappings(filePath, 'python')) { + const dotted = m.isNamespace ? m.source + : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; + const files = pythonModuleFiles(dotted, filePath, context); + if (!files.includes(moduleFile)) continue; + // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. + // Exactly this module (not `other.a.b`), outside string literals. + const explicit = new RegExp(`\\bimport\\s[^\\n]*(? explicit.test(blankPythonStrings(line))); + (files.length === 1 ? aliases : ambiguous).add(plainDotted ? m.source : m.localName); + } + return { aliases: [...aliases], ambiguous: [...ambiguous] }; +} + +const PYTHON_MAIN_GUARD = /^if\s+(?:__name__\s*==\s*(['"])__main__\1|(['"])__main__\2\s*==\s*__name__)\s*:/; +/** Line indexes inside a top-level `if __name__ == "__main__":` block — script code, not module state. */ +function pythonMainBlockLines(filePath: string, context: ResolutionContext): Set { + return pythonNameScan(context, `main\0${filePath}`, () => { + const lines = pythonMemberLines(filePath, context); + const inside = new Set(); + for (let i = 0; i < lines.length; i++) { + const guard = lines[i]!.match(PYTHON_MAIN_GUARD); + if (!guard) continue; + if (lines[i]!.slice(guard[0].length).trim()) inside.add(i); // `if __name__ == "__main__": stmt` + for (let j = i + 1; j < lines.length && !/^\S/.test(lines[j]!); j++) inside.add(j); + } + return inside; + }); +} + +/** Blank single-line string contents (triple-quoted strings are already blanked by the comment stripper). */ +function blankPythonStrings(text: string): string { + return text.replace(/(['"])(?:\\.|(?!\1)[^\\\n])*\1/g, m => m[0] + ' '.repeat(m.length - 2) + m[0]); +} + +/** Test code installs doubles: the narrow test-suite set, plus pytest's `conftest.py` wherever it sits. */ +function isPythonTestFile(filePath: string): boolean { + return isTestPath(filePath) || /(?:^|\/)conftest\.py$/.test(filePath.replace(/\\/g, '/')); +} + +/** + * Writes to module global `global` from OTHER files. A production write + * `. = Cls(...)` adds a type; any other production write + * (another value, a tuple target, `setattr`) makes the type unknown. Test + * files install doubles (`settings.conn = MagicMock()`, `monkeypatch.setattr`) + * that do not define the production type; they are recorded as `writers`, and + * a ref inside a writer resolves nothing through the global. + */ +function pythonExternalWrites(global: Node, context: ResolutionContext): { writes: Array<{ type: string; file: string }>; unknown: boolean; writers: Set } { + return pythonNameScan(context, `external\0${global.id}`, () => { + const out = { writes: [] as Array<{ type: string; file: string }>, unknown: false, writers: new Set() }; + const name = global.name; + const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + for (const file of context.getAllFiles()) { + if (file === global.filePath || !/\.pyi?$/.test(file) || !context.readFile(file)?.includes(name)) continue; + const test = isPythonTestFile(file); + const { aliases, ambiguous } = pythonModuleAliases(file, global.filePath, context); + if (!test && !aliases.length && !ambiguous.length) continue; + const spell = (names: string[]) => `(?:${names.map(escape).join('|')})`; + // Test files: any receiver (the double may sit on any spelling of the module). + const receiver = test ? '[\\w.]+' : spell([...aliases, ...ambiguous]); + const target = new RegExp(`(? target.test(t.trim())) ?? false; + if (!assigned && !dynamic.test(statement)) continue; + if (test) { out.writers.add(file); continue; } + // A write through an ambiguous spelling may land on another module: unknown. + const single = assignment && assignment.targets.length === 1 && !assignment.augmented && exact.test(assignment.targets[0]!.trim()); + const value = single ? assignment!.value.trim() : ''; + if (value === 'None') continue; + const constructor = value ? pythonConstructorCall(value) : null; + if (!constructor) { out.unknown = true; return out; } + out.writes.push({ type: constructor, file }); + } + } + return out; + }); +} + +/** + * Whether the global's own module can write it through its namespace dict: + * `globals()` / `vars()` / `sys.modules[__name__]` used as anything but a + * literal-key read or `.get`, or a literal-key write of this name. Read per + * statement (continuations joined), with string contents ignored. + */ +function pythonDynamicGlobalWrite(name: string, filePath: string, context: ResolutionContext): boolean { + const lines = pythonMemberLines(filePath, context); + const statements = pythonStatementStarts(filePath, context); + const script = pythonMainBlockLines(filePath, context); + // `vars()` is the module dict only at module scope; inside a function it is the locals. + const moduleLines = new Set(pythonOwnLines(null, filePath, context)); + for (let i = 0; i < lines.length; i++) { + if (!statements[i] || script.has(i)) continue; + let statement = lines[i]!; + for (let j = i + 1; j < lines.length && !statements[j]; j++) statement += '\n' + lines[j]; + const code = blankPythonStrings(statement); + const namespace = moduleLines.has(i) + ? /\bglobals\(\s*\)|\bvars\(\s*\)|\bsys\.modules\s*\[\s*__name__\s*\]/g + : /\bglobals\(\s*\)|\bsys\.modules\s*\[\s*__name__\s*\]/g; + const uses = code.match(namespace)?.length ?? 0; + if (!uses) continue; + const targets = pythonAssignment(statement)?.targets ?? []; + let safe = 0; + for (const m of statement.matchAll(/\bglobals\(\s*\)\s*(?:\[\s*(['"])(\w+)\1\s*\]|\.\s*get\s*\()/g)) { + const key = m[2]; + const written = key !== undefined && targets.some(t => t.includes(m[0])); + if (written && key === name) return true; + safe++; + } + if (uses > safe) return true; + } + return false; +} + +/** + * The method a module global's value can dispatch to: one class resolves to + * its own method; several bind to the nearest declaration they all inherit, + * as a base-typed receiver does. Otherwise, no edge. + */ +function pythonGlobalMembers(global: Node, member: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + // A test that installs its own double sees the double, not the production type. + if (pythonExternalWrites(global, context).writers.has(ref.filePath)) return []; + const classes = pythonGlobalClasses(global, ref, context); + if (!classes) return []; + const targets = [...new Map(classes.flatMap(cls => pythonMembers(cls, member, ref, context)).map(n => [n.id, n])).values()]; + if (targets.length <= 1) return targets; + // Several targets: the nearest declaration every candidate class inherits, + // whether or not a candidate overrides it. + const owner = (n: Node) => context.getNodesInFile(n.filePath).find(c => + c.kind === 'class' && n.qualifiedName === `${c.qualifiedName}::${member}`); + const declarations = new Map(); + const queue = [...targets]; + while (queue.length && declarations.size < 32) { + const decl = queue.shift()!; + const cls = decl && owner(decl); + if (!cls || declarations.has(decl.id)) continue; + declarations.set(decl.id, { decl, cls }); + queue.push(...pythonBases(cls, ref, context).flatMap(base => pythonMembers(base, member, ref, context))); + } + const inherits = (cls: Node, base: Node) => cls.id === base.id || pythonDerivesFrom(cls, base, ref, context); + const shared = [...declarations.values()].filter(d => classes.every(c => inherits(c, d.cls))); + const nearest = shared.filter(d => shared.every(o => inherits(d.cls, o.cls))); + return nearest.length === 1 ? [nearest[0]!.decl] : targets; +} + +/** `Cls(...)` / `pkg.mod.Cls(...)` as the WHOLE (possibly multi-line) value; else null (`Cls() if x else y`). */ +function pythonConstructorCall(text: string): string | null { + // `(Cls())` is the same value; peel parentheses that wrap the whole expression. + for (let wrapped = text.trim(); wrapped.startsWith('('); ) { + let depth = 0; + let close = -1; + for (let i = 0; i < wrapped.length && close < 0; i++) { + if (wrapped[i] === '(') depth++; + else if (wrapped[i] === ')' && --depth === 0) close = i; + } + if (close !== wrapped.length - 1) break; + text = wrapped = wrapped.slice(1, -1).trim(); + } + const callee = text.match(/^((?:[A-Za-z_]\w*\.)*[A-Z]\w*)\s*\(/); + if (!callee) return null; + let depth = 0; + let quote = ''; + for (let i = callee[0].length - 1; i < text.length; i++) { + const c = text[i]!; + if (quote) { if (c === '\\') i++; else if (c === quote) quote = ''; continue; } + if (c === '"' || c === "'") quote = c; + else if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') { + if (--depth === 0) return /^[\s\\]*$/.test(text.slice(i + 1)) ? callee[1]! : null; + } + } + return null; +} + /** Read a field's own annotation/initializer, or a constructor parameter assigned to it. */ function pythonFieldType(receiver: string, owner: Node, ref: UnresolvedRef, context: ResolutionContext): string | null { const lines = pythonMemberLines(ref.filePath, context); @@ -6094,6 +6590,9 @@ function getInferScanStates(context: ResolutionContext): Map Date: Fri, 2 Oct 2026 04:49:00 +0000 Subject: [PATCH 183/363] fix(cpp): bound the macro-visibility include walk (#2127) (#2292) * fix(cpp): bound the macro-visibility include walk (#2127) isVisibleCppMacro walks the call site's translation unit to learn whether a function-like macro is visible there (#1838). A header whose guard the walk cannot decide -- `#define X_H 1` (the OpenSceneGraph / osgEarth idiom, which guardsItself does not read as a guard), or any header reached under an unknown `#if`, where neither its guard nor `#pragma once` settles -- was re-scanned on every inclusion path, and every re-scan pushed its define events onto the cached timeline again. On a layered include graph that is 2^depth scans and 2^depth timeline entries: "Resolving refs" died with "Ineffective mark-compacts near heap limit" at the first ref whose file reached such a tree, at the same ref whatever the heap size. The walk now keeps a version that every observable change bumps (definitions, `#pragma once`, the macro-name set). A re-entry with the same inherited truth, while the version still equals that of an earlier visit which itself changed nothing, is skipped: it would start from the same state, take the same branches and change nothing again, and the events it would push repeat each macro's current state, already its last event. Include cycles cut by the recursion stack are recorded per visit and must still be cut for a skip. A hard budget of 1M directive events per walk bounds what is left; past it the timeline is unknown and nothing at or after the cut line is suppressed, so the budget can only fall back to ordinary resolution, never invent a macro. Answers are unchanged: 20,000 random include graphs (guards bare / valued / pragma once / under flags, #if/#elif/#else, #undef, cycles) give identical isVisibleCppMacro results to the previous walker on 1.25M queries, with about one re-entry in eight skipped. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 73db32f81ab9ffe1b02b4443914e9fd74cc2c4e1) * docs(changelog): credit the C/C++ macro walk bound Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: danusha2345 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cpp-macro-include-walk.test.ts | 125 +++++++++++++++++++++++ src/resolution/cpp-macro-visibility.ts | 68 ++++++++++-- 3 files changed, 185 insertions(+), 9 deletions(-) create mode 100644 __tests__/cpp-macro-include-walk.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index cdb9a6a6f8..b83668890e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -169,6 +169,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a call written on the instance, like `self.get_breadcrumbs(...)`, now links to the class's own method even when the file also imports a function of the same name. Before, the import claimed it: Django REST Framework's renderers, netbox's change logging and django-allauth's account adapter all linked their own method calls to an imported helper. - In Python, a method passed as a value through a module-level variable that a setup function assigns, like `settings.docStoreConn.delete_payload_fields` after `global docStoreConn; docStoreConn = QdrantConnection()`, now links to that class's method. When several backend classes can be assigned, it links to the declaration they all inherit. Thanks @JosefAschauer. (#2074) - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. +- Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/cpp-macro-include-walk.test.ts b/__tests__/cpp-macro-include-walk.test.ts new file mode 100644 index 0000000000..acaf9c0d0f --- /dev/null +++ b/__tests__/cpp-macro-include-walk.test.ts @@ -0,0 +1,125 @@ +/** + * #2127 — the C/C++ macro-visibility walk must stay bounded on deep include + * graphs. + * + * A header whose guard the walk cannot decide — `#define X_H 1` (the OSG / + * osgEarth idiom, which the guard reader does not take for a guard), or any + * header reached under an unknown `#if` — used to be re-scanned on every + * inclusion path: a layered include graph where each header includes the two + * of the next layer cost 2^depth header scans and grew the timeline with + * every one of them, which is how "Resolving refs" ran out of heap on + * osgEarth. The answers must not change: a macro definitely visible at the + * call site is still a macro, and an unknown one still is not. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { isVisibleCppMacro } from '../src/resolution/cpp-macro-visibility'; +import type { ResolutionContext, UnresolvedRef } from '../src/resolution/types'; + +type Guard = 'value' | 'bare' | 'unknown-flag'; + +/** `depth` layers of two headers; each includes both headers of the next layer. */ +function layeredHeaders(depth: number, guard: Guard): Record { + const files: Record = {}; + for (let d = 0; d < depth; d++) { + for (const side of ['a', 'b']) { + const g = `H_${d}_${side}`; + const open = + guard === 'value' ? `#ifndef ${g}\n#define ${g} 1\n` : + guard === 'bare' ? `#ifndef ${g}\n#define ${g}\n` : + `#ifdef USE_${d}\n`; + const next = d + 1 < depth ? `#include "h${d + 1}_a.h"\n#include "h${d + 1}_b.h"\n` : ''; + files[`h${d}_${side}.h`] = `${open}${next}#define M_${d}_${side}(x) (x)\nint f_${d}_${side}(int);\n#endif\n`; + } + } + return files; +} + +/** A context over in-memory files where every name is both a macro and a function. */ +function contextOver(files: Record): ResolutionContext { + return { + getNodesByName: (name: string) => [ + { kind: 'constant', signature: `#define ${name}(v) (v)` }, + { kind: 'function' }, + ], + readFile: (f: string) => files[f] ?? null, + fileExists: (f: string) => f in files, + getAllFiles: () => Object.keys(files), + } as unknown as ResolutionContext; +} + +const call = (name: string, line: number): UnresolvedRef => + ({ language: 'cpp', referenceKind: 'calls', referenceName: name, filePath: 'unit.cpp', line }) as UnresolvedRef; + +describe('#2127 — the macro-visibility include walk is bounded', () => { + it.each(['value', 'bare', 'unknown-flag'] as const)( + 'a 40-layer include graph (2^40 inclusion paths) is walked in well under a second (%s guards)', + (guard) => { + const files = { + ...layeredHeaders(40, guard), + 'trace.h': '#define TRACE(v) ((void)(v))\n', + 'unit.cpp': '#include "h0_a.h"\n#include "h0_b.h"\n#include "trace.h"\nvoid unit() { TRACE(1); }\n', + }; + const context = contextOver(files); + const started = performance.now(); + // Before the fix this walk never finished (2^40 header scans). + expect(isVisibleCppMacro(call('TRACE', 4), context)).toBe(true); + expect(performance.now() - started).toBeLessThan(1000); + // A header macro under an undecidable guard is not definitely visible, + // and a call above the include that defines TRACE is still a call. + expect(isVisibleCppMacro(call('M_39_a', 4), context)).toBe(guard === 'bare'); + expect(isVisibleCppMacro(call('TRACE', 2), context)).toBe(false); + }, + ); + + it('a walk that cannot be shortened stops at its budget and suppresses nothing past it', () => { + // No guards and a known flag toggled on every inclusion: each visit + // changes the state, so no re-entry repeats an earlier one and the real + // preprocessor would expand all 2^22 paths too. The walk gives up instead: + // before the exploding include the macro is still known, after it nothing is. + const files: Record = { + 'toggle.h': '#ifdef T\n#undef T\n#else\n#define T\n#endif\n', + 'unit.cpp': '#define T\n#define TRACE(v) ((void)(v))\nvoid before() { TRACE(1); }\n#include "h0.h"\nvoid after() { TRACE(2); }\n', + }; + for (let d = 0; d < 22; d++) { + files[`h${d}.h`] = d + 1 < 22 ? `#include "toggle.h"\n#include "h${d + 1}.h"\n#include "h${d + 1}.h"\n` : '#include "toggle.h"\n'; + } + const context = contextOver(files); + const started = performance.now(); + expect(isVisibleCppMacro(call('TRACE', 3), context)).toBe(true); + expect(isVisibleCppMacro(call('TRACE', 5), context)).toBe(false); + expect(performance.now() - started).toBeLessThan(10_000); + }); +}); + +describe('#2127 — indexing a deep include graph keeps #1838 macro suppression', () => { + const roots: string[] = []; + afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); + }); + + it('the macro reached through value-guarded headers still does not bind to the decoy function', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-walk-')); + roots.push(root); + const files = { + ...layeredHeaders(24, 'value'), + 'trace.h': '#define TRACE(v) ((void)(v))\n', + 'unit.cpp': '#include "h0_a.h"\n#include "h0_b.h"\n#include "trace.h"\nvoid unit() { TRACE(1); }\n', + 'decoy.cpp': 'void TRACE(int v) {}\nvoid caller() { TRACE(2); }\n', + }; + for (const [rel, content] of Object.entries(files)) fs.writeFileSync(path.join(root, rel), content); + const cg = await CodeGraph.init(root, { index: true }); + try { + const fn = (name: string) => cg.getNodesByKind('function').find((n) => n.name === name)!; + const callees = (name: string) => + cg.getCallees(fn(name).id).filter((r) => r.edge.kind === 'calls').map((r) => r.node.filePath); + expect(callees('unit')).toEqual([]); + expect(callees('caller')).toEqual(['decoy.cpp']); + } finally { + cg.close(); + } + }, 60_000); +}); diff --git a/src/resolution/cpp-macro-visibility.ts b/src/resolution/cpp-macro-visibility.ts index e5c76e2f11..b0b3ed18a8 100644 --- a/src/resolution/cpp-macro-visibility.ts +++ b/src/resolution/cpp-macro-visibility.ts @@ -15,6 +15,12 @@ * flags remain possible, so only definite macro visibility suppresses a call. * Object-like definitions participate in conditions but never enter the cached * call-site timelines (vendor headers can contain tens of thousands of them). + * + * A guard the walk cannot decide (`#define X_H 1`, or any header reached under + * an unknown `#if`) re-enters its header on every inclusion path, which is + * exponential in the include graph's depth (#2127). A re-entry that provably + * repeats an earlier visit which changed nothing is skipped, and a hard budget + * bounds whatever is left: past it, nothing is known and nothing is suppressed. */ import * as path from 'path'; import { maskCppRawStrings } from '../extraction/languages/c-cpp'; @@ -29,17 +35,22 @@ type FileEvent = | { kind: 'branch'; op: string; expression: string; guard: boolean; line: number } | { kind: 'once'; line: number }; type Event = { line: number; defined: Truth }; +/** Macro name → events in root-file line order; from `cutoff` on, nothing is known. */ +type Timeline = { events: Map; cutoff: number }; type Cache = { summaries: Map; includes: Map; /** Indexed files by basename, for `#include "dir/name.h"` that no include root explains. */ byBasename: Map | null; /** Per root file: macro name → define/undef events in root-file line order. */ - roots: Map>; + roots: Map; }; const memo = new WeakMap(); const ROOT_TIMELINE_CAP = 32; +/** Directive events one translation-unit walk may evaluate (#2127). */ +const WALK_EVENT_BUDGET = 1_000_000; +const NO_CUTS: string[] = []; const and = (a: Truth, b: Truth): Truth => a === false || b === false ? false : a === true && b === true ? true : undefined; @@ -79,7 +90,8 @@ export function isVisibleCppMacro(ref: UnresolvedRef, context: ResolutionContext if (cache.roots.size >= ROOT_TIMELINE_CAP) cache.roots.delete(cache.roots.keys().next().value!); cache.roots.set(rootKey, timeline); } - const before = (timeline.get(ref.referenceName) ?? []).filter((e) => e.line <= ref.line); + if (ref.line >= timeline.cutoff) return false; + const before = (timeline.events.get(ref.referenceName) ?? []).filter((e) => e.line <= ref.line); return before.length > 0 && before[before.length - 1]!.defined === true; } @@ -250,12 +262,24 @@ function walkTranslationUnit( language: UnresolvedRef['language'], context: ResolutionContext, cache: Cache -): Map { +): Timeline { const timeline = new Map(); const definitions = new Map(); const scanning = new Set(); const macroNames = new Set(); const once = new Map(); + // Every change a later directive could observe (definitions, `#pragma once`, + // the macro-name set) bumps `version`. A visit that left it unchanged, entered + // again with the same inherited truth while it is still unchanged, starts from + // the same state, so it would take the same branches and change nothing again; + // the events it would push repeat each name's current state, which is already + // its last event. Include cycles cut by the recursion stack (`cuts`) must still + // be cut for the repeat to hold. + let version = 0; + const cuts: string[] = []; + const visits = new Map(); + let budget = WALK_EVENT_BUDGET; + let cutoff = Infinity; const condition = (expression: string): Truth => { const text = expression.trim(); if (/^(?:0x[\da-f]+|\d+)[uUlL]*$/i.test(text)) return Number(text.replace(/[uUlL]+$/, '')) !== 0; @@ -267,12 +291,28 @@ function walkTranslationUnit( return /^\w+$/.test(text) ? definitions.get(text)?.value : undefined; }; const scan = (file: string, inherited: Truth, includeLine?: number): void => { - if (inherited === false || scanning.has(file) || once.get(file) === true) return; + if (inherited === false || once.get(file) === true || cutoff !== Infinity) return; + if (scanning.has(file)) { + cuts.push(file); + return; + } + const seen = visits.get(file); + if ( + seen && seen.inherited === inherited && seen.start === seen.end && seen.end === version && + seen.cuts.every((c) => scanning.has(c)) + ) { + cuts.push(...seen.cuts); + return; + } + const start = version; + const firstCut = cuts.length; scanning.add(file); let active: Truth = inherited; const frames: Array<{ parent: Truth; taken: Truth }> = []; for (const ev of summarize(file, context, cache)) { const line = includeLine ?? ev.line; + if (cutoff === Infinity && --budget < 0) cutoff = line; + if (cutoff !== Infinity) break; if (ev.kind === 'branch') { if (ev.op === 'if' || ev.op === 'ifdef' || ev.op === 'ifndef') { const known = definitions.get(ev.expression.trim())?.defined; @@ -294,7 +334,9 @@ function walkTranslationUnit( } if (active === false) continue; if (ev.kind === 'once') { - once.set(file, or(once.get(file) ?? false, active)); + const next = or(once.get(file) ?? false, active); + if (next !== (once.get(file) ?? false)) version++; + once.set(file, next); continue; } if (ev.kind === 'include') { @@ -309,12 +351,17 @@ function walkTranslationUnit( // A name no directive has touched is unknown, not undefined: the build // can set it on the command line. So an `#undef` under an undecidable // `#if` leaves it unknown (#2069); only a certain one clears it. - definitions.set(ev.name, { + const next = { defined: defining ? or(prior?.defined, active) : and(prior?.defined, not(active)), value: defining && active === true ? condition(ev.value) : undefined, macro: now, - }); - if (ev.functionLike) macroNames.add(ev.name); + }; + if (next.defined !== prior?.defined || next.value !== prior?.value || next.macro !== prior?.macro) version++; + definitions.set(ev.name, next); + if (ev.functionLike && !macroNames.has(ev.name)) { + macroNames.add(ev.name); + version++; + } if (macroNames.has(ev.name)) { const events = timeline.get(ev.name) ?? []; events.push({ line, defined: now }); @@ -322,8 +369,11 @@ function walkTranslationUnit( } } scanning.delete(file); + const own = cuts.length > firstCut ? [...new Set(cuts.splice(firstCut))] : NO_CUTS; + cuts.push(...own); + visits.set(file, { inherited, start, end: version, cuts: own }); }; scan(rootFile, true); - return timeline; + return { events: timeline, cutoff }; } From 7c01910e545165cae56353044014ca03d479e646 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 05:00:47 +0000 Subject: [PATCH 184/363] fix(cfml): tag-based calls, wrapped functions and are indexed (#2091) (#2296) * fix(cfml): tag-based calls, wrapped functions and are indexed (#2091) Tag-based CFML lost almost all of its call edges. The tag walk sent only and bodies through call extraction, although the cfml grammar parses the expressions in , /, and #hash# (output, strings, attributes) with the same rules as cfscript. The walk now records each such expression with the scope that owns it (the enclosing , else the component, else the file) and reads them all through the existing cfscript extraction in one parse per file: the expressions are copied out on their original lines, each ended with a `;`, and every reference is handed back to its expression's scope with its column restored. is read the same way. A nested in a generic container tag at component or template scope ( in Application.cfc, ) was skipped because only direct children were examined; it is now extracted (never through an ERROR node, where a lost would misfile methods as top-level functions). , a generic tag to the grammar, now yields an interface node with its functions as methods and an `extends` ref per comma-separated parent, so `implements` resolves to it. The tag-based parse tree is now freed after the walk. Resolution: the CFML/Ruby/C# bare-call scope filters evaluated the receiver-on-line check once per same-named candidate though only the ref decides it. With tag calls extracted, that made Mach-II's resolution ~8x slower; the check is now memoized per ref (edges byte-identical). Co-Authored-By: Claude Opus 5.5 * docs(changelog): tag-based CFML calls (#2091) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/cfml-tag-calls.test.ts | 85 ++++++++ __tests__/extraction.test.ts | 156 +++++++++++++ src/extraction/cfml-extractor.ts | 363 ++++++++++++++++++++++++++----- src/resolution/name-matcher.ts | 14 ++ 5 files changed, 560 insertions(+), 59 deletions(-) create mode 100644 __tests__/cfml-tag-calls.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b83668890e..01a871b167 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -170,6 +170,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, a method passed as a value through a module-level variable that a setup function assigns, like `settings.docStoreConn.delete_payload_fields` after `global docStoreConn; docStoreConn = QdrantConnection()`, now links to that class's method. When several backend classes can be assigned, it links to the declaration they all inherit. Thanks @JosefAschauer. (#2074) - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) +- In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/cfml-tag-calls.test.ts b/__tests__/cfml-tag-calls.test.ts new file mode 100644 index 0000000000..f2741ed2d7 --- /dev/null +++ b/__tests__/cfml-tag-calls.test.ts @@ -0,0 +1,85 @@ +/** + * Tag-based CFML end to end (#2091). Most calls in a tag-based component sit + * in ``/``/`` and `#hash#` expressions, not in + * ``; on a 6,000-file tag-based codebase only 6.4% of methods had a + * caller in the graph. Functions wrapped in `` (the + * Application.cfc shape) and `` components were missing as well, + * so their callers and implementers had nothing to land on. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cfml-tag-calls-')); + const files: Record = { + 'model/Svc.cfc': ` +\t +\t\t +\t\t +\t\t +\t +\t +\t +\t + +`, + 'Application.cfc': ` +\t +\t\t +\t\t\t +\t\t +\t\t +\t + +`, + 'model/ISearchable.cfc': ` +\t + +`, + 'model/Catalog.cfc': ` +\t + +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +const edgesFrom = (file: string, kind: string): string[] => { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === kind) + .map((e) => `${cg.getNode(e.source)!.qualifiedName} -> ${cg.getNode(e.target)!.qualifiedName}`) + .sort(); +}; + +describe('tag-based CFML (#2091)', () => { + it('links calls written in , and to the methods they call', () => { + expect(edgesFrom('model/Svc.cfc', 'calls')).toEqual(['Svc::a -> Svc::b', 'Svc::a -> Svc::c', 'Svc::a -> Svc::d']); + }); + + it('indexes a method wrapped in and the calls it makes', () => { + expect(edgesFrom('Application.cfc', 'calls')).toEqual(['Application::onRequestStart -> Application::loadConfig']); + }); + + it('resolves to a component', () => { + const iface = cg.getNodesInFile('model/ISearchable.cfc').find((n) => n.kind === 'interface'); + expect(iface?.name).toBe('ISearchable'); + expect(edgesFrom('model/Catalog.cfc', 'implements')).toEqual(['model/Catalog.cfc::Catalog -> model/ISearchable.cfc::ISearchable']); + }); +}); diff --git a/__tests__/extraction.test.ts b/__tests__/extraction.test.ts index 7fad827adb..c6a522decf 100644 --- a/__tests__/extraction.test.ts +++ b/__tests__/extraction.test.ts @@ -11099,6 +11099,162 @@ import foo.cfm; expect(result.nodes.find((n) => n.name === 'innerHelper')?.kind).toBe('function'); }); }); + + describe('Calls in tag expressions outside / (#2091)', () => { + const callsFrom = (result: ReturnType, fromId: string | undefined) => + result.unresolvedReferences + .filter((r) => r.fromNodeId === fromId && (r.referenceKind === 'calls' || r.referenceKind === 'instantiates')) + .map((r) => `${r.referenceKind === 'instantiates' ? 'new ' : ''}${r.referenceName}@${r.line}`) + .sort(); + + it('should extract calls in , , and , attributed to the enclosing method', () => { + const code = ` +\t +\t\t +\t\t +\t\t +\t\t +\t\t +\t +\t + +`; + const result = extractFromSource('Svc.cfc', code); + const a = result.nodes.find((n) => n.kind === 'method' && n.name === 'a'); + expect(a).toBeDefined(); + expect(callsFrom(result, a?.id)).toEqual(['b@3', 'c@4', 'd@7', 'e@5']); + }); + + it('should extract #hash# expressions in output, strings and tag attributes — once each, alongside bodies', () => { + const code = ` + + + + #fmt(y)# and #variables.mailer.send()# + + SELECT #col()# FROM t + + + +`; + const result = extractFromSource('View.cfc', code); + const render = result.nodes.find((n) => n.kind === 'method' && n.name === 'render'); + expect(callsFrom(result, render?.id)).toEqual([ + 'col@7', + 'dsn@7', + 'fmt@5', + 'getItems@6', + 'new Widget@8', + 'svc.load@3', + 'userName@4', + 'variables.mailer.send@5', + ]); + // `` is a function local — no variable node for it. + expect(result.nodes.find((n) => n.name === 'y')).toBeUndefined(); + }); + + it('should attribute component-scope and template-scope calls to the component and the file', () => { + const component = `\n\n\n\n`; + const cfc = extractFromSource('Pseudo.cfc', component); + const cls = cfc.nodes.find((n) => n.kind === 'class'); + expect(callsFrom(cfc, cls?.id)).toEqual(['setup@2']); + + const template = `\n#renderList(items)#\n`; + const cfm = extractFromSource('index.cfm', template); + const file = cfm.nodes.find((n) => n.kind === 'file'); + expect(callsFrom(cfm, file?.id)).toEqual(['loadItems@1', 'renderList@2']); + }); + + it("should keep each call's own line and column (several per line, and across lines)", () => { + const code = ` + + #last()# + + +`; + const lines = code.split('\n'); + const result = extractFromSource('Pos.cfc', code); + const calls = result.unresolvedReferences.filter((r) => r.referenceKind === 'calls'); + expect(calls.map((r) => r.referenceName).sort()).toEqual(['a', 'b.c', 'count', 'd', 'e', 'last', 'other.fetch', 'sum']); + for (const r of calls) { + expect(lines[r.line - 1]!.slice(r.column).startsWith(r.referenceName)).toBe(true); + } + }); + + it('should read a attribute as an expression', () => { + const code = `\n\n\n\n\n\n`; + const result = extractFromSource('Queue.cfc', code); + const drain = result.nodes.find((n) => n.kind === 'method' && n.name === 'drain'); + expect(callsFrom(result, drain?.id)).toEqual(['hasNext@3']); + }); + + it('should not read HTML \n

    also(not)

    \n    \n\n`; + const result = extractFromSource('Page.cfc', code); + expect(result.unresolvedReferences.filter((r) => r.referenceKind === 'calls')).toEqual([]); + }); + }); + + describe(' inside a generic container tag (#2091)', () => { + it('should extract a method nested in (the Application.cfc shape)', () => { + const code = ` +\t +\t\t +\t\t\t +\t\t +\t + +`; + const result = extractFromSource('Application.cfc', code); + const cls = result.nodes.find((n) => n.kind === 'class'); + const method = result.nodes.find((n) => n.kind === 'method' && n.name === 'onRequestStart'); + expect(method).toBeDefined(); + expect(method?.qualifiedName).toBe('Application::onRequestStart'); + expect(result.edges.some((e) => e.kind === 'contains' && e.source === cls?.id && e.target === method?.id)).toBe(true); + const call = result.unresolvedReferences.find((r) => r.referenceName === 'loadConfig'); + expect(call?.fromNodeId).toBe(method?.id); + }); + + it('should extract a top-level function nested in in a template', () => { + const code = `\n\n\n`; + const result = extractFromSource('lib.cfm', code); + expect(result.nodes.find((n) => n.name === 'helper')?.kind).toBe('function'); + }); + }); + + describe(' (#2091)', () => { + const code = ` +\t +\t\t +\t +\t + +`; + + it('should extract an interface node named from the file, with its functions as methods', () => { + const result = extractFromSource('ISearchable.cfc', code); + const iface = result.nodes.find((n) => n.kind === 'interface'); + expect(iface?.name).toBe('ISearchable'); + expect(iface?.startLine).toBe(1); + expect(iface?.endLine).toBe(6); + expect(result.nodes.filter((n) => n.kind === 'class')).toHaveLength(0); + const methods = result.nodes.filter((n) => n.kind === 'method'); + expect(methods.map((m) => m.qualifiedName).sort()).toEqual(['ISearchable::count', 'ISearchable::search']); + for (const m of methods) { + expect(result.edges.some((e) => e.kind === 'contains' && e.source === iface?.id && e.target === m.id)).toBe(true); + } + expect(methods.find((m) => m.name === 'search')?.returnType).toBe('any'); + }); + + it('should extract each interface it extends', () => { + const result = extractFromSource('ISearchable.cfc', code); + const iface = result.nodes.find((n) => n.kind === 'interface'); + const ext = result.unresolvedReferences.filter((r) => r.referenceKind === 'extends'); + expect(ext.map((r) => r.referenceName)).toEqual(['IBase', 'IOther']); + expect(ext.every((r) => r.fromNodeId === iface?.id)).toBe(true); + }); + }); }); describe('COBOL Extraction', () => { diff --git a/src/extraction/cfml-extractor.ts b/src/extraction/cfml-extractor.ts index 8395940c10..efd2ea6ea6 100644 --- a/src/extraction/cfml-extractor.ts +++ b/src/extraction/cfml-extractor.ts @@ -4,6 +4,55 @@ import { generateNodeId, NodeIdAllocator } from './tree-sitter-helpers'; import { TreeSitterExtractor } from './tree-sitter'; import { getParser } from './grammars'; +/** Tags whose own children include an expression: ``, ``, ``, ``, and `#…#`. */ +const TAG_EXPRESSION_PARENTS: ReadonlySet = new Set([ + 'cf_set_tag', + 'cf_if_tag', + 'cf_elseif_tag', + 'cf_return_tag', + 'hash_expression', +]); + +/** + * The expression node types (shared with the cfscript grammar) that can + * contain a call. A positive list, so a tag's markup — body content, its + * `var` keyword, an ERROR — is never mistaken for an expression; a bare + * identifier or literal can't call anything and is skipped. + */ +const TAG_EXPRESSION_TYPES: ReadonlySet = new Set([ + 'call_expression', + 'member_expression', + 'subscript_expression', + 'new_expression', + 'assignment_expression', + 'augmented_assignment_expression', + 'binary_expression', + 'unary_expression', + 'update_expression', + 'ternary_expression', + 'elvis_expression', + 'parenthesized_expression', + 'sequence_expression', + 'string', + 'array', + 'object', + 'ordered_struct', + 'function_expression', + 'arrow_function', +]); + +/** An expression in tag markup and the scope (function, component or file node) that owns its calls. */ +interface TagExpression { + startIndex: number; + endIndex: number; + /** Where it starts in the file — the same line in the source extractTagExpressions synthesizes. */ + row: number; + column: number; + /** Its start column in the synthesized source. */ + textColumn: number; + scopeId: string; +} + /** * CfmlExtractor - Extracts code relationships from CFML source (.cfc/.cfm). * @@ -15,7 +64,9 @@ import { getParser } from './grammars'; * raw AST, so this extractor replicates it: a file whose first real token * isn't `<` is delegated wholesale to the cfscript grammar (the dominant * modern style); otherwise the file is walked tag-by-tag with the cfml - * grammar, delegating any `` tag bodies the same way. + * grammar, delegating any `` tag bodies the same way — and the + * expressions written in tags themselves (``, ``, `#hash#`, …) + * through the same cfscript extraction (see extractTagExpressions). */ export class CfmlExtractor { private filePath: string; @@ -26,6 +77,7 @@ export class CfmlExtractor { private edges: Edge[] = []; private unresolvedReferences: UnresolvedReference[] = []; private errors: ExtractionError[] = []; + private tagExpressions: TagExpression[] = []; /** `language` is the file's detected language — `'cfml'` for `.cfc`/`.cfm`, `'cfscript'` for `.cfs`. Both dialect-switch internally; this only controls the language tag stamped onto emitted nodes/refs. */ constructor(filePath: string, source: string, language: Language = 'cfml') { @@ -113,8 +165,14 @@ export class CfmlExtractor { return; } - const fileNode = this.createFileNode(); - this.walkProgram(tree.rootNode, fileNode.id); + try { + const fileNode = this.createFileNode(); + this.walkProgram(tree.rootNode, fileNode.id); + } finally { + // Trees hold wasm heap memory V8's GC never sees — free it per file. + tree.delete(); + } + this.extractTagExpressions(); } /** Build the file's own `kind:'file'` node, spanning the whole source. Tag-based files need this explicitly — unlike `extractBareScript` (which delegates the whole file to `TreeSitterExtractor` and inherits its file node), `extractTagBased` walks the tree itself and has no other source of one. */ @@ -151,17 +209,11 @@ export class CfmlExtractor { if (child.type === 'cf_component_open_tag') { child = this.extractComponent(child, fileNodeId).nextSibling; continue; - } else if (child.type === 'cf_function_tag') { - // A cffunction outside any cfcomponent wrapper (rare, but legal in a - // .cfm template) — extract as a top-level function, contained by the file. - this.extractFunctionTag(child, undefined, fileNodeId); - } else if (child.type === 'cf_script_tag') { - this.delegateScriptTag(child, fileNodeId); - } else if (child.type === 'cf_query_tag') { - this.delegateQueryTag(child, fileNodeId); - } else { - this.delegateNestedTags(child, fileNodeId); } + // Template scope: a cffunction outside any cfcomponent wrapper (rare, + // but legal in a .cfm template) is a top-level function contained by + // the file, and template code's calls are the file's. + this.visitTag(child, root.type, fileNodeId, undefined, true); child = child.nextSibling; } } @@ -208,20 +260,7 @@ export class CfmlExtractor { language: this.language, }); } - const implementsAttr = this.tagAttr(openTag, 'implements'); - if (implementsAttr) { - for (const iface of implementsAttr.split(',').map((s) => s.trim()).filter(Boolean)) { - this.unresolvedReferences.push({ - fromNodeId: classNode.id, - referenceName: iface, - referenceKind: 'implements', - filePath: this.filePath, - line: openTag.startPosition.row + 1, - column: openTag.startPosition.column, - language: this.language, - }); - } - } + this.pushInheritanceRefs(classNode.id, this.tagAttr(openTag, 'implements'), 'implements', openTag); // Walk siblings between the open tag and its close tag. let sibling = openTag.nextSibling; @@ -231,15 +270,9 @@ export class CfmlExtractor { lastNode = sibling; break; } - if (sibling.type === 'cf_function_tag') { - this.extractFunctionTag(sibling, classNode.id, classNode.id, classNode.name); - } else if (sibling.type === 'cf_script_tag') { - this.delegateScriptTag(sibling, classNode.id, classNode.name); - } else if (sibling.type === 'cf_query_tag') { - this.delegateQueryTag(sibling, classNode.id); - } else { - this.delegateNestedTags(sibling, classNode.id, classNode.name); - } + // Component scope: cffunctions are its methods, pseudo-constructor + // code's calls (a component-level ``) are the component's. + this.visitTag(sibling, openTag.parent?.type ?? '', classNode.id, classNode.name, true); lastNode = sibling; sibling = sibling.nextSibling; } @@ -247,6 +280,54 @@ export class CfmlExtractor { return lastNode; } + /** + * `...` (#2091). Unlike + * `` (see the implicit-end-tag note on `extractComponent`) the + * grammar has no dedicated rule for it: it is a generic `cf_tag` whose body + * — the `` signatures — is ordinary nested children. Named like + * a component (the file name, unless a `name` attribute says otherwise), so + * `` resolves to it. An interface may + * extend several interfaces, comma-separated. + */ + private extractInterfaceTag(tag: SyntaxNode, containerId: string): void { + const startTag = this.cfStartTag(tag) ?? tag; + const name = this.tagAttr(startTag, 'name') ?? this.componentNameFromPath(); + const id = this.nodeIds.generate(this.filePath, 'interface', name, tag.startPosition.row + 1, tag.startPosition.column); + this.nodes.push({ + id, + kind: 'interface', + name, + qualifiedName: `${this.filePath}::${name}`, + filePath: this.filePath, + language: this.language, + startLine: tag.startPosition.row + 1, + endLine: tag.endPosition.row + 1, + startColumn: tag.startPosition.column, + endColumn: tag.endPosition.column, + isExported: true, + updatedAt: Date.now(), + }); + this.edges.push({ source: containerId, target: id, kind: 'contains' }); + this.pushInheritanceRefs(id, this.tagAttr(startTag, 'extends'), 'extends', startTag); + this.delegateNestedTags(tag, id, name, true); + } + + /** One unresolved `extends`/`implements` ref per name in a comma-separated tag attribute. */ + private pushInheritanceRefs(fromNodeId: string, list: string | undefined, kind: 'extends' | 'implements', tag: SyntaxNode): void { + if (!list) return; + for (const name of list.split(',').map((s) => s.trim()).filter(Boolean)) { + this.unresolvedReferences.push({ + fromNodeId, + referenceName: name, + referenceKind: kind, + filePath: this.filePath, + line: tag.startPosition.row + 1, + column: tag.startPosition.column, + language: this.language, + }); + } + } + /** * `...`. * `parentClassId` decides `method` vs top-level `function`; `containerId` is @@ -290,36 +371,190 @@ export class CfmlExtractor { this.edges.push({ source: containerId, target: fnNode.id, kind: 'contains' }); } - // Delegate any / bodies nested inside this function, at - // any depth (e.g. inside // control-flow tags). + // Walk the body, at any depth (e.g. inside // + // control-flow tags): its / bodies and tag + // expressions are this function's. this.delegateNestedTags(tag, fnNode.id); } /** - * Recursively delegates any `cf_script_tag`/`cf_query_tag` found within - * `node`'s subtree — e.g. a ``/`` nested inside - * ``/``/`` control-flow tags, which (unlike - * ``'s body — see the implicit-end-tag note on `extractComponent`) - * ARE normal children, just possibly several levels deep, so a direct-children - * check misses them. Does not descend into a nested `cf_function_tag` — that - * has its own scope and is walked separately. `parentClassName` rides along - * so a `` at component scope classifies its functions as methods - * scoped under the component. + * Visit one node of tag markup. `containerId` is the scope the node sits in + * — the function whose body it is, else the component, else the file — and + * owns whatever calls it makes. `declScope` is true outside function bodies + * (component/interface/template scope), where a `` declares a + * function of that scope; `parentClassName` is set at component/interface + * scope, where those functions are methods. + * + * - ``/`` bodies go to their own grammars (see + * delegateScriptTag / delegateQueryTag). + * - A `` is extracted when reached at declaration scope, at any + * depth: `` around Application.cfc's handlers + * (#2091) or `` around a template's helpers doesn't make them + * any less the scope's functions. Never through an ERROR node — at file + * scope that is typically a `` the parser lost (behind an + * unclosed ``), whose methods must not be misfiled as + * top-level functions. Inside a function body a `` would be + * its own scope; CFML rejects that, so it is skipped. + * - An expression in tag position — the value of ``, the condition + * of ``/``, the operand of ``, a `#hash#` + * anywhere in markup, attributes and strings included — is recorded for + * extractTagExpressions. The grammar parses these structurally (the same + * expression rules as cfscript), so `parentType` (the node's parent) is + * enough to tell the expression from the tag around it. + */ + private visitTag(node: SyntaxNode, parentType: string, containerId: string, parentClassName: string | undefined, declScope: boolean): void { + switch (node.type) { + case 'cf_script_tag': + this.delegateScriptTag(node, containerId, parentClassName); + return; + case 'cf_query_tag': + // The SQL body is opaque to this grammar; the walk below only reaches + // the tag's attributes (`datasource="#dsn()#"`), never the body twice. + this.delegateQueryTag(node, containerId); + break; + case 'cf_function_tag': + if (declScope) this.extractFunctionTag(node, parentClassName ? containerId : undefined, containerId, parentClassName); + return; + case 'cf_tag': { + const name = this.cfTagName(node); + if (name === 'interface' && declScope && !parentClassName) { + this.extractInterfaceTag(node, containerId); + return; + } + // `` — the one common tag attribute whose + // plain-text value is a CFML expression rather than a literal. + if (name === 'loop') { + const condition = this.tagAttrValueNode(this.cfStartTag(node) ?? node, 'condition'); + if (condition) this.addTagExpression(condition, containerId); + } + break; + } + default: + if (TAG_EXPRESSION_PARENTS.has(parentType) && TAG_EXPRESSION_TYPES.has(node.type)) { + this.addTagExpression(node, containerId); + return; + } + } + this.delegateNestedTags(node, containerId, parentClassName, declScope && node.type !== 'ERROR'); + } + + /** + * Visit `node`'s children (see visitTag) — e.g. a ``/`` + * nested inside ``/``/`` control-flow tags, which + * (unlike ``'s body — see the implicit-end-tag note on + * `extractComponent`) ARE normal children, just possibly several levels + * deep, so a direct-children check misses them. `parentClassName` rides + * along so a `` at component scope classifies its functions as + * methods scoped under the component. */ - private delegateNestedTags(node: SyntaxNode, containerId: string | undefined, parentClassName?: string): void { + private delegateNestedTags(node: SyntaxNode, containerId: string, parentClassName?: string, declScope = false): void { for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); - if (!child) continue; - if (child.type === 'cf_script_tag') { - this.delegateScriptTag(child, containerId, parentClassName); - } else if (child.type === 'cf_query_tag') { - this.delegateQueryTag(child, containerId); - } else if (child.type === 'cf_function_tag') { - continue; + if (child) this.visitTag(child, node.type, containerId, parentClassName, declScope); + } + } + + /** Record an expression in tag markup, owned by `scopeId`, for extractTagExpressions. */ + private addTagExpression(node: SyntaxNode, scopeId: string): void { + if (node.endIndex <= node.startIndex) return; + this.tagExpressions.push({ + startIndex: node.startIndex, + endIndex: node.endIndex, + row: node.startPosition.row, + column: node.startPosition.column, + textColumn: 0, + scopeId, + }); + } + + /** + * Extract calls from the expressions visitTag recorded, through the same + * cfscript extraction a `` body gets — one parse per file, not + * one per expression: the expressions are copied out in order, each on its + * own line (the line breaks between them are kept, so every line number is + * the file's) and ended with a `;`, and the result is read with the + * cfscript grammar. Each reference is handed back to the scope whose + * expression contains it, its column shifted back to the file's. Only + * references are kept: an expression declares nothing (`` + * is a function local, and its `var` isn't copied), so a closure written in + * one is part of its scope's code. + */ + private extractTagExpressions(): void { + const exprs = this.tagExpressions; + if (exprs.length === 0) return; + exprs.sort((a, b) => a.startIndex - b.startIndex); + + const parts: string[] = []; + let pos = 0; + let column = 0; // column in the synthesized source + for (const expr of exprs) { + if (pos > 0) { + parts.push(';'); + column++; + } + let breaks = 0; + for (let at = this.source.indexOf('\n', pos); at !== -1 && at < expr.startIndex; at = this.source.indexOf('\n', at + 1)) breaks++; + if (breaks > 0) { + parts.push('\n'.repeat(breaks)); + column = 0; + } + expr.textColumn = column; + const text = this.source.slice(expr.startIndex, expr.endIndex); + parts.push(text); + const lastBreak = text.lastIndexOf('\n'); + column = lastBreak === -1 ? column + text.length : text.length - lastBreak - 1; + pos = expr.endIndex; + } + + const result = new TreeSitterExtractor(this.filePath, parts.join(''), 'cfscript').extract(); + for (const ref of result.unresolvedReferences) { + const expr = this.tagExpressionAt(ref.line - 1, ref.column); + // Only an expression's first line moved; its later lines are verbatim. + if (ref.line - 1 === expr.row) ref.column += expr.column - expr.textColumn; + ref.fromNodeId = expr.scopeId; + ref.filePath = this.filePath; + ref.language = this.language; + this.unresolvedReferences.push(ref); + } + // A broken expression is routine in hand-written markup and the file's + // symbols came from the tag walk, so only a genuine failure is reported — + // not the extractor's "no symbols" warning, which an expressions-only + // source always earns. + for (const error of result.errors) { + if (error.severity === 'error') this.errors.push(error); + } + } + + /** The recorded expression containing a synthesized-source position: the last one starting at or before it. */ + private tagExpressionAt(row: number, column: number): TagExpression { + const exprs = this.tagExpressions; + let lo = 0; + let hi = exprs.length - 1; + let found = 0; + while (lo <= hi) { + const mid = (lo + hi) >> 1; + const e = exprs[mid]!; + if (e.row < row || (e.row === row && e.textColumn <= column)) { + found = mid; + lo = mid + 1; } else { - this.delegateNestedTags(child, containerId, parentClassName); + hi = mid - 1; } } + return exprs[found]!; + } + + /** A generic `cf_tag`'s start tag — where its name and attributes live. */ + private cfStartTag(tag: SyntaxNode): SyntaxNode | undefined { + return tag.namedChildren.find( + (c: SyntaxNode) => c.type === 'cf_start_tag' || c.type === 'cf_start_tag_with_selfclose' + ); + } + + /** A generic `cf_tag`'s name, lowercased, without the `cf` prefix (`` → `loop`). */ + private cfTagName(tag: SyntaxNode): string | undefined { + const nameNode = this.cfStartTag(tag)?.namedChildren.find((c: SyntaxNode) => c.type === 'cf_tag_name'); + return nameNode ? this.source.substring(nameNode.startIndex, nameNode.endIndex).toLowerCase() : undefined; } /** @@ -445,6 +680,18 @@ export class CfmlExtractor { /** Read a `cf_attribute`'s value by name from a tag node's direct `cf_attribute`/`cf_tag_attributes` children. */ private tagAttr(tag: SyntaxNode, attrName: string): string | undefined { + const valueNode = this.tagAttrValueNode(tag, attrName); + if (valueNode === undefined) return undefined; + if (!valueNode) return ''; + return this.source.substring(valueNode.startIndex, valueNode.endIndex); + } + + /** + * The `attribute_value` node of a tag attribute — `undefined` when the tag + * has no such attribute, `null` when its value isn't plain text (empty, or + * a `#hash#` expression). + */ + private tagAttrValueNode(tag: SyntaxNode, attrName: string): SyntaxNode | null | undefined { const attrs: SyntaxNode[] = []; for (let i = 0; i < tag.namedChildCount; i++) { const child = tag.namedChild(i); @@ -467,9 +714,7 @@ export class CfmlExtractor { const valueWrapper = attr.namedChildren.find( (c: SyntaxNode) => c.type === 'quoted_cf_attribute_value' || c.type === 'cf_attribute_value' ); - const valueNode = valueWrapper?.namedChildren.find((c: SyntaxNode) => c.type === 'attribute_value'); - if (!valueNode) return ''; - return this.source.substring(valueNode.startIndex, valueNode.endIndex); + return valueWrapper?.namedChildren.find((c: SyntaxNode) => c.type === 'attribute_value') ?? null; } return undefined; } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 701c0ea03c..e0b4ad28b8 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -4644,6 +4644,8 @@ function cfmlComponentFiles(dotted: string, from: string, context: ResolutionCon return files; } +const NO_RECEIVER_LINES = new WeakMap>(); + /** * Whether a bare reference is receiver-less at its call site, name case * aside: the name is not preceded by a `.` on its line (true when the line @@ -4651,6 +4653,17 @@ function cfmlComponentFiles(dotted: string, from: string, context: ResolutionCon * links of a chain (`newFuture(f).then(g)`) over bare. */ function hasNoReceiverOnLine(ref: UnresolvedRef, context: ResolutionContext): boolean { + // Asked once per CANDIDATE by the per-language scope filters, though only + // the ref decides it — a bare `init()` in a CFML codebase has hundreds of + // same-named methods, each re-reading the line (#2091). + let memo = NO_RECEIVER_LINES.get(context); + if (!memo) NO_RECEIVER_LINES.set(context, (memo = new WeakMap())); + let answer = memo.get(ref); + if (answer === undefined) memo.set(ref, (answer = readNoReceiverOnLine(ref, context))); + return answer; +} + +function readNoReceiverOnLine(ref: UnresolvedRef, context: ResolutionContext): boolean { const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split('\n')[ref.line - 1]; if (line === undefined) return true; const lower = line.toLowerCase(); @@ -6629,6 +6642,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { KOTLIN_FILE_SCOPES.delete(context); RUBY_ANCESTRY.delete(context); CFML_CHAINS.delete(context); + NO_RECEIVER_LINES.delete(context); OBJC_SUPERS.delete(context); CSHARP_SUPERS.delete(context); CSHARP_STATIC_USINGS.delete(context); From 45a680df07df2297dd7997a9376501351088eee9 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:16:25 +0000 Subject: [PATCH 185/363] fix(mcp): one daemon per project however its root is cased on Windows (#2278) (#2285) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(mcp): canonicalize the daemon rendezvous root on Windows A project spelled d:\work\codegraph or D:\work\codegraph is one directory, but the daemon socket/pipe name and the registry record name hashed a raw path.resolve, which preserves whichever casing the caller passed. A proxy that computed the other casing probed a socket nobody bound, spawned a redundant daemon, and that daemon exited on the lock the running one held, so the session silently served the graph in-process: no shared watcher, no auto-sync, one extra copy of the index per open window. Both names now come from one shared canonicalProjectRoot (native realpath, lowercased on win32), so every spelling meets the same daemon and codegraph list / stop --all see one record per project. (cherry picked from commit 181178801827351d713e1d552886c0482e6db67b) * fix(mcp): share one in-process project however its root is cased (#2278) The engine's project lifecycle and `codegraph stop`'s registry fallback keyed a project by its realpath, which keeps the caller's casing, so two spellings of one root opened two graphs and two watchers in one process. Both now use the same identity key as the daemon socket. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 0a3239e603d5796b9f2cf636b4c09635239e32dd) * docs(changelog): file the Windows rendezvous fix under Unreleased Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: 养心堂主 Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-rendezvous-case.test.ts | 123 +++++++++++++++++++++++ src/directory.ts | 34 +++++++ src/mcp/daemon-paths.ts | 12 ++- src/mcp/daemon-registry.ts | 11 +- src/mcp/project-lifecycle.ts | 13 ++- 6 files changed, 184 insertions(+), 10 deletions(-) create mode 100644 __tests__/daemon-rendezvous-case.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 01a871b167..62d005a12f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -171,6 +171,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In Python, an import split over several lines in parentheses, `from x import (A, B)` written one name per line, is now read in full. Before, its names were invisible to resolution, so Django REST Framework's `serializers.CharField()` and netbox's model mixins linked to the wrong class or to nothing. Thanks @JosefAschauer. - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) +- On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/daemon-rendezvous-case.test.ts b/__tests__/daemon-rendezvous-case.test.ts new file mode 100644 index 0000000000..6c8a1a1a3e --- /dev/null +++ b/__tests__/daemon-rendezvous-case.test.ts @@ -0,0 +1,123 @@ +/** + * One project must map to ONE daemon rendezvous key, however its root is + * spelled — the regression behind "the MCP server starts but never shares the + * daemon". + * + * The daemon's named pipe (Windows) / tmpdir socket (POSIX) is + * `…codegraph-.slice(0,16)`, and the lockfile is shared + * while the SOCKET NAME is derived independently by each process. So the moment + * two processes hash the same directory differently they stop meeting: the + * proxy's probe finds nothing, it spawns a redundant daemon, and that daemon + * exits on the lock the first one holds + * (`Another daemon (pid N) already holds the lock; exiting.`) — the session then + * serves in-process, without the shared watcher or auto-sync. + * + * That is reachable on Windows because the root arrives two ways: a cwd-derived + * root is the on-disk casing (`D:\work\codegraph`), while a client-supplied + * `rootUri`/`workspaceFolders` path arrives as `file:///d%3A/…` → `d:\…`, and + * `path.resolve` preserves whichever it got (NTFS is case-insensitive, so both + * name one directory). Pinned here at the level that actually broke: the key, + * not the filename. + * + * The real-filesystem assertions are Windows-gated — on POSIX the two spellings + * are genuinely different directories, so only the Windows case has a + * case-insensitive filesystem to converge on. + */ + +import { afterEach, describe, expect, it } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { canonicalProjectRoot } from '../src/directory'; +import { getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { acquireProject } from '../src/mcp/project-lifecycle'; +import type CodeGraph from '../src/index'; + +const tmpDirs: string[] = []; +function makeDir(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-rendezvous-')); + tmpDirs.push(dir); + return dir; +} +afterEach(() => { + while (tmpDirs.length) { + try { + fs.rmSync(tmpDirs.pop()!, { recursive: true, force: true }); + } catch { + /* best-effort */ + } + } +}); + +/** The same drive letter, forced to one case. No-op on a path without a drive. */ +function withDriveCase(p: string, upper: boolean): string { + return p.replace(/^[a-z]:/i, (drive) => (upper ? drive.toUpperCase() : drive.toLowerCase())); +} + +describe('daemon rendezvous key', () => { + it('collapses redundant path segments to one socket name', () => { + const root = makeDir(); + const indirect = path.join(root, 'a', '..', 'b', '.'); + const direct = path.join(root, 'b'); + // `.`, `..` and a trailing separator all name the same directory, so both + // spellings must resolve to the same socket — otherwise `--path` with a + // stray separator is enough to miss a running daemon. + expect(getDaemonSocketPath(indirect)).toBe(getDaemonSocketPath(direct)); + }); + + it.runIf(process.platform === 'win32')( + 'maps both drive-letter cases of one directory onto one pipe', + () => { + const root = makeDir(); + const upper = withDriveCase(root, true); + const lower = withDriveCase(root, false); + + expect(canonicalProjectRoot(upper)).toBe(canonicalProjectRoot(lower)); + expect(getDaemonSocketPath(upper)).toBe(getDaemonSocketPath(lower)); + // …and the converged key is the on-disk casing, lowercased, so it does not + // depend on which of the two the caller happened to hold. + expect(canonicalProjectRoot(upper)).toBe(fs.realpathSync.native(root).toLowerCase()); + }, + ); + + it.runIf(process.platform === 'win32')( + 'converges case variants even when the root cannot be realpath’d', + () => { + // The `.codegraph/` root normally exists, so this is the fallback arm — + // but a divergence there would fail identically, so pin it. + const missing = path.join(os.tmpdir(), 'cg-rendezvous-absent', 'nested'); + expect(canonicalProjectRoot(missing)).toBe(canonicalProjectRoot(missing.toUpperCase())); + }, + ); +}); + +/** True when the temp filesystem ignores case (default macOS APFS, NTFS). */ +function caseInsensitiveTmp(): boolean { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-CaseProbe-')); + try { + return fs.existsSync(dir.replace('cg-CaseProbe-', 'cg-caseprobe-')); + } finally { + fs.rmSync(dir, { recursive: true, force: true }); + } +} + +describe('in-process project sharing (#2278)', () => { + it.runIf(caseInsensitiveTmp())('opens one graph for two casings of one root', async () => { + const root = path.join(makeDir(), 'Repo'); + fs.mkdirSync(root); + let opened = 0; + const open = () => { + opened++; + return { close() {}, isIndexing: () => false } as unknown as CodeGraph; + }; + const a = acquireProject(root, open, {}); + const b = acquireProject(path.join(path.dirname(root), 'REPO'), open, {}); + try { + expect(opened).toBe(1); + expect(b.cg).toBe(a.cg); + } finally { + await a.release(); + await b.release(); + } + }); +}); diff --git a/src/directory.ts b/src/directory.ts index d99860f04e..c458457e4e 100644 --- a/src/directory.ts +++ b/src/directory.ts @@ -303,6 +303,40 @@ export function statInode(p: string): string | null { } } +/** + * Canonicalize a project root for IDENTITY: hashing it into a rendezvous key + * (the daemon's named pipe / tmpdir socket, a registry record). Not for display + * — the caller-visible root keeps its own spelling. + * + * The contract is that two spellings of ONE directory yield ONE string, because + * a client that derives a different key silently fails to meet the daemon that + * is already running: it probes a socket nobody bound, spawns a redundant + * daemon, and that daemon dies on the lock the first one holds, so the session + * degrades to a single-process engine. `path.resolve` alone does NOT satisfy + * that contract — NTFS is case-insensitive, so `d:\work\codegraph` and + * `D:\work\codegraph` name one directory but two strings, and non-native + * `fs.realpathSync` keeps whichever casing the caller passed (only `.native` + * asks the filesystem for the on-disk name — the same reason + * {@link isSameIndexRoot} uses it). Both spellings really do occur: a + * cwd-derived root is the on-disk case, while a client-supplied + * `rootUri`/`workspaceFolders` path arrives as `file:///d%3A/…`. + * + * The Windows lowercase is belt-and-braces on top of the native realpath — NTFS + * can't hold two directories differing only by case, and it also absorbs a + * `\\?\`-prefixed native result diverging from a plain one. + */ +export function canonicalProjectRoot(projectRoot: string): string { + const resolved = path.resolve(projectRoot); + let canonical = resolved; + try { + canonical = fs.realpathSync.native(resolved); + } catch { + // ENOENT/EACCES/ELOOP — the root is normally there (`.codegraph/` lives in + // it), so this is a fallback rather than a path we expect to take. + } + return process.platform === 'win32' ? canonical.toLowerCase() : canonical; +} + /** * Whether two resolved index roots are one index spelled two ways — a symlinked * checkout, or a case-variant on a case-insensitive mount (macOS, NTFS, WSL diff --git a/src/mcp/daemon-paths.ts b/src/mcp/daemon-paths.ts index 34751814f3..8f78d5063e 100644 --- a/src/mcp/daemon-paths.ts +++ b/src/mcp/daemon-paths.ts @@ -32,14 +32,20 @@ import * as crypto from 'crypto'; import * as net from 'net'; import * as os from 'os'; import * as path from 'path'; -import { getCodeGraphDir } from '../directory'; +import { canonicalProjectRoot, getCodeGraphDir } from '../directory'; /** Soft upper bound for in-project socket paths. */ const POSIX_SOCKET_PATH_LIMIT = 100; -/** Short stable identifier for a project root — used in tmpdir/pipe names. */ +/** + * Short stable identifier for a project root — used in tmpdir/pipe names. + * + * Hashed over {@link canonicalProjectRoot}, never a raw `path.resolve`: the key + * is a rendezvous, so every spelling of one directory must land on one name or + * a proxy probes a pipe the running daemon never bound (see that function). + */ function projectHash(projectRoot: string): string { - return crypto.createHash('sha256').update(path.resolve(projectRoot)).digest('hex').slice(0, 16); + return crypto.createHash('sha256').update(canonicalProjectRoot(projectRoot)).digest('hex').slice(0, 16); } /** diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index b4a01c307f..c6c4043b60 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -22,6 +22,7 @@ import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import * as crypto from 'crypto'; +import { canonicalProjectRoot } from '../directory'; import { getDaemonPidPath, getDaemonSocketCandidates, @@ -50,8 +51,14 @@ export function getRegistryDir(): string { return path.join(os.homedir(), '.codegraph', 'daemons'); } +/** + * One record per project, so it is keyed the same way the daemon socket is: + * over {@link canonicalProjectRoot}, not a raw `path.resolve` — otherwise the + * same project spelled with another drive-letter case files two records, and + * `list` over-lists while `stop --all` misses one. + */ function recordPath(root: string): string { - const hash = crypto.createHash('sha256').update(path.resolve(root)).digest('hex').slice(0, 16); + const hash = crypto.createHash('sha256').update(canonicalProjectRoot(root)).digest('hex').slice(0, 16); return path.join(getRegistryDir(), `${hash}.json`); } @@ -242,7 +249,7 @@ export async function stopDaemonAt(root: string, options: { preserveUnverified?: } if (pid == null) { const rec = listDaemons({ prune: false }).find( - (r) => path.resolve(r.root) === path.resolve(root) + (r) => canonicalProjectRoot(r.root) === canonicalProjectRoot(root) ); pid = rec?.pid ?? null; if (rec) identity = rec; diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index ba405f57d1..b7a52c1a5b 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -2,13 +2,15 @@ import { realpathSync } from 'fs'; import type { Socket } from 'net'; import type CodeGraph from '../index'; -import { isInitialized } from '../directory'; +import { canonicalProjectRoot, isInitialized } from '../directory'; import { LockUnavailableError, watchDisabledReason } from '../sync'; import { getDaemonSocketCandidates } from './daemon-paths'; import { connectWithHello } from './proxy'; import { markWriterReady, readWriterLock, releaseWriterLock, tryAcquireWriterLock } from './writer-lock'; interface Project { + /** Identity key: one entry however the root is spelled (#2278). */ + key: string; cg: CodeGraph; refs: number; owner: boolean; @@ -35,13 +37,14 @@ export function acquireProject( options: Parameters[0], ): ProjectLease { root = realpathSync(root); - let project = projects.get(root); + const key = canonicalProjectRoot(root); + let project = projects.get(key); if (!project) { const cg = open(); - project = { cg, refs: 0, owner: false, caughtUp: false, retirement: null, gate: null, socket: null, options, + project = { key, cg, refs: 0, owner: false, caughtUp: false, retirement: null, gate: null, socket: null, options, timer: setInterval(() => { void ready(root, project!); }, 1000) }; project.timer.unref(); - projects.set(root, project); + projects.set(key, project); } const entry = project; entry.refs++; @@ -80,7 +83,7 @@ function retire(root: string, project: Project): Promise { setTimeout(finish, 25); return; } - projects.delete(root); + projects.delete(project.key); project.cg.close(); if (project.owner) releaseWriterLock(root); resolve(); From e7389b22fb18381eae9639feb6e09ae2af5d128d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:18:02 +0000 Subject: [PATCH 186/363] fix(mcp): a session serving in-process hands the project back to the shared daemon (#2277) (#2293) * fix(mcp): a session serving in-process hands the project back to the shared daemon (#2277) When a proxy lost its daemon (or never reached one) it served the rest of the session through an in-process engine. That engine takes the project's writer lock in fallback mode and nothing ever released it or tried the daemon again, so every later daemon start for the project failed on the held lock and every other session ran read-only without auto-sync until that one session ended. Sessions that fell back read-only stayed read-only for life too. The degraded proxy now retries the shared daemon with backoff (5 s, doubling to a 5 min cap; CODEGRAPH_DAEMON_RETRY_MS / _MAX_MS, 0 disables) through the existing probe/spawn/poll path. A writing engine is stopped first, after the calls it is serving have answered, so its lock is free before a daemon is spawned and two writers never overlap; calls arriving meanwhile are buffered and go to the daemon, or are served in-process again if none comes up. A read-only engine keeps serving until a daemon answers and is closed once its calls finish. The client's initialize is replayed to the new daemon with its reply suppressed, as on first attach. Co-Authored-By: Claude Opus 5.5 * docs(changelog): a degraded session hands the project back (#2277) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/mcp-daemon.test.ts | 138 ++++++++++++++++++++++++++++++ src/mcp/engine.ts | 5 ++ src/mcp/proxy.ts | 159 +++++++++++++++++++++++++++++------ 4 files changed, 279 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 62d005a12f..cf27df4914 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -172,6 +172,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) +- An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/mcp-daemon.test.ts b/__tests__/mcp-daemon.test.ts index 71cc7b71ff..5b8b305002 100644 --- a/__tests__/mcp-daemon.test.ts +++ b/__tests__/mcp-daemon.test.ts @@ -817,4 +817,142 @@ describe('Shared MCP daemon (issue #411)', () => { expect(resp.result !== undefined || resp.error !== undefined).toBe(true); expect(isAlive(server.child.pid!)).toBe(true); }, 45000); + + it('a proxy serving in-process hands the writer lock back to a fresh daemon (#2277)', async () => { + // After its daemon dies, the proxy's in-process engine takes writer.pid in + // fallback mode. Every later daemon start used to fail on that lock for the + // rest of the session, so every other session on the project ran read-only + // without auto-sync. The degraded proxy now retries the daemon: it stops + // its engine (releasing the lock), lets a daemon start, and proxies again. + const env = { + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + CODEGRAPH_PPID_POLL_MS: '5000', + // The default, spelled out: long enough to observe the lockout first. + CODEGRAPH_DAEMON_RETRY_MS: '5000', + }; + const a = spawnServer(tempDir, env); + servers.push(a); + const proxyPid = a.child.pid!; + sendInitialize(a.child, `file://${tempDir}`, 1); + await waitFor(() => findResponse(a.stdout, 1), 20000, 25, 'initialize response'); + await waitFor(() => a.stderr.some((l) => l.includes('Attached to shared daemon')), 8000, 25, 'first daemon attach'); + sendMessage(a.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + await waitFor(() => findResponse(a.stdout, 2), 30000, 25, 'warm tools/call via daemon'); + const firstDaemon = readLockPid(realRoot)!; + + process.kill(firstDaemon, 'SIGTERM'); + expect(await waitProcessExit(firstDaemon, 8000)).toBe(true); + await waitFor(() => a.stderr.some((l) => l.includes('serving this session in-process')), 8000, 25, 'in-process failover'); + sendMessage(a.child, { jsonrpc: '2.0', id: 3, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + await waitFor(() => findResponse(a.stdout, 3), 15000, 25, 'in-process tools/call'); + + // The lockout: the in-process engine owns the project's writer slot. + expect(readWriterInfo(realRoot)).toMatchObject({ pid: proxyPid, mode: 'fallback' }); + + // Another session starting now cannot get a daemon while that lock is held. + const b = spawnServer(tempDir, env); + servers.push(b); + sendInitialize(b.child, `file://${tempDir}`, 1); + await waitFor(() => findResponse(b.stdout, 1), 20000, 25, 'second session initialize'); + + // Keep calling through the handover: every request gets exactly one reply. + let nextId = 4; + const send = (): void => { + sendMessage(a.child, { jsonrpc: '2.0', id: nextId++, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + }; + const ticker = setInterval(send, 250); + let daemonWriter: { pid: number; mode: string } | null = null; + try { + daemonWriter = await waitFor(() => { + const w = readWriterInfo(realRoot); + return w && w.mode === 'daemon' && w.pid !== proxyPid && isAlive(w.pid) ? w : null; + }, 30000, 25, 'a daemon to own the writer lock'); + await waitFor( + () => a.stderr.some((l) => l.includes(`Attached to shared daemon`) && l.includes(`(pid ${daemonWriter!.pid},`)), + 15000, 25, 'the proxy to reattach to the new daemon', + ); + } finally { + clearInterval(ticker); + } + send(); + const lastId = nextId - 1; + await waitFor(() => findResponse(a.stdout, lastId), 15000, 25, 'a tools/call through the new daemon'); + for (let id = 2; id <= lastId; id++) { + const replies = a.stdout.filter((line) => { + try { const m = JSON.parse(line); return m.id === id && ('result' in m || 'error' in m); } catch { return false; } + }); + expect(replies, `replies to request ${id}`).toHaveLength(1); + expect(JSON.parse(replies[0]).error, `request ${id}`).toBeUndefined(); + } + expect(readLockPid(realRoot)).toBe(daemonWriter!.pid); + + // The session that started during the lockout ends up on the shared daemon too. + await waitFor( + () => b.stderr.some((l) => l.includes('Attached to shared daemon') && l.includes(`(pid ${daemonWriter!.pid},`)), + 30000, 25, 'the second session to attach', + ); + sendMessage(b.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const bReply = await waitFor(() => findResponse(b.stdout, 2), 15000, 25, 'second session tools/call'); + expect(bReply.error).toBeUndefined(); + }, 90000); + + it('a read-only in-process session moves to the shared daemon once one can start (#2277)', async () => { + // A session that fell back read-only (another daemon held the project) + // used to stay that way for life. Once the blocker is gone it now starts + // and attaches to a daemon of its own version. + const sockPath = getDaemonSocketPath(realRoot); + const pidPath = path.join(realRoot, '.codegraph', 'daemon.pid'); + fs.writeFileSync(pidPath, JSON.stringify({ pid: process.pid, version: '0.0.0-mismatch', socketPath: sockPath, startedAt: Date.now() })); + const miniServer = net.createServer((sock) => { + sock.write(JSON.stringify({ codegraph: '0.0.0-mismatch', pid: process.pid, socketPath: sockPath, protocol: 1 }) + '\n'); + }); + await new Promise((resolve) => miniServer.listen(sockPath, () => resolve())); + let miniServerOpen = true; + try { + const server = spawnServer(tempDir, { + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + CODEGRAPH_DAEMON_RETRY_MS: '300', + CODEGRAPH_DAEMON_RETRY_MAX_MS: '1000', + }); + servers.push(server); + sendInitialize(server.child, `file://${tempDir}`, 1); + await waitFor(() => server.stderr.some((l) => l.includes('serving this session in-process')), 10000, 25, 'in-process fallback'); + sendMessage(server.child, { jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const local = await waitFor(() => findResponse(server.stdout, 2), 10000, 25, 'read-only tools/call'); + expect(local.error).toBeUndefined(); + expect(server.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + // Retries while the other version still answers start no daemon. + await waitFor( + () => server.stderr.filter((l) => l.includes('differs from ours')).length >= 3, + 10000, 25, 'repeated retries against the other version', + ); + expect(countListeningLines(realRoot)).toBe(0); + + // The other-version daemon goes away. + await new Promise((resolve) => miniServer.close(() => resolve())); + miniServerOpen = false; + fs.rmSync(pidPath, { force: true }); + + const attached = await waitFor( + () => server.stderr.find((l) => l.includes('Attached to shared daemon')), + 30000, 25, 'the session to attach to a daemon', + ); + const daemonPid = readLockPid(realRoot)!; + expect(attached).toContain(`(pid ${daemonPid},`); + expect(readWriterInfo(realRoot)).toMatchObject({ pid: daemonPid, mode: 'daemon' }); + sendMessage(server.child, { jsonrpc: '2.0', id: 3, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const viaDaemon = await waitFor(() => findResponse(server.stdout, 3), 15000, 25, 'tools/call through the daemon'); + expect(viaDaemon.error).toBeUndefined(); + expect(JSON.stringify(viaDaemon.result)).toContain('CodeGraph Status'); + } finally { + if (miniServerOpen) await new Promise((resolve) => miniServer.close(() => resolve())); + } + }, 60000); }); + +function readWriterInfo(root: string): { pid: number; mode: string } | null { + try { + const info = JSON.parse(fs.readFileSync(path.join(root, '.codegraph', 'writer.pid'), 'utf8')); + return typeof info.pid === 'number' && typeof info.mode === 'string' ? info : null; + } catch { return null; } +} diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 2898f203b1..5409c36d48 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -167,6 +167,11 @@ export class MCPEngine { this.toolHandler.setDefaultProjectHint(projectPath); } + /** Whether this engine only reads: no watcher, no sync, no writer slot. */ + isReadOnly(): boolean { + return this.opts.readOnly; + } + /** Project root that the engine resolved on first init (null if none). */ getProjectPath(): string | null { return this.projectPath; diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts index 17c5a59cb1..a983c2ebdc 100644 --- a/src/mcp/proxy.ts +++ b/src/mcp/proxy.ts @@ -39,6 +39,15 @@ import type { MCPEngine } from './engine'; /** Default poll cadence for the PPID watchdog (same as the direct server). */ const DEFAULT_PPID_POLL_MS = 5000; +/** + * How long a proxy serving in-process waits before trying the shared daemon + * again, doubling after each miss up to the cap (#2277). Tunable through + * `CODEGRAPH_DAEMON_RETRY_MS` (`0` turns retrying off) and + * `CODEGRAPH_DAEMON_RETRY_MAX_MS`. + */ +const DEFAULT_DAEMON_RETRY_MS = 5_000; +const DEFAULT_DAEMON_RETRY_MAX_MS = 300_000; + /** * Env var that opts INTO the "attached to shared daemon" log line. Off by * default: the line is benign INFO, but MCP hosts render any server stderr at @@ -196,8 +205,9 @@ export interface LocalHandshakeDeps { /** Probe → spawn → retry → hello-verify; resolves a connected daemon socket, * or null when the daemon path is genuinely unavailable (→ in-process fallback). */ getDaemonSocket(): Promise; - /** Lazily create an in-process engine — used ONLY if the daemon never comes up, - * preserving the "a broken daemon never wedges a session" guarantee. */ + /** Lazily create an in-process engine — used only while the daemon is + * unreachable, preserving the "a broken daemon never wedges a session" + * guarantee. Called again whenever the previous engine was retired (#2277). */ makeEngine(): MCPEngine; /** Project root for the fallback engine's lazy init. */ root: string; @@ -215,6 +225,12 @@ export interface LocalHandshakeDeps { * the local one). If the daemon never comes up (version mismatch / spawn fail), * a lazily-created in-process engine serves the calls — so the handshake speedup * never costs the old fall-back-to-direct robustness. + * + * Serving in-process is not for the rest of the session (#2277): that engine + * may own the project's writer lock, and no daemon can start while it does, so + * every other session on the project would be stuck without auto-sync. The + * proxy retries the daemon with backoff, handing the writer lock back first, + * and proxies again once one answers. */ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise { // The proxy is long-lived and can serve fallback tool calls in-process. Match @@ -224,6 +240,7 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< let daemonStatus: 'connecting' | 'ready' | 'failed' = 'connecting'; let daemonSocket: net.Socket | null = null; let clientInitId: unknown = undefined; // suppress the daemon's reply to the forwarded initialize + let clientInitLine: string | undefined; // replayed to a daemon that comes up mid-session (#2277) // Telemetry attribution for the in-process fallback only — calls routed to // the daemon are counted by the daemon's own session (which receives the // forwarded initialize, clientInfo included), never double-counted here. @@ -231,7 +248,14 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< const pending: string[] = []; // client lines buffered until the daemon resolves let engine: MCPEngine | null = null; let engineReady: Promise | null = null; + // Calls being served in-process; the engine is only ever stopped after they finish. + const localCalls = new Set>(); let shuttingDown = false; + const retryBaseMs = parseDelayMs(process.env.CODEGRAPH_DAEMON_RETRY_MS, DEFAULT_DAEMON_RETRY_MS); + const retryMaxMs = Math.max(retryBaseMs, parseDelayMs(process.env.CODEGRAPH_DAEMON_RETRY_MAX_MS, DEFAULT_DAEMON_RETRY_MAX_MS)); + let retryDelayMs = retryBaseMs; + let retryTimer: NodeJS.Timeout | null = null; + let attachedAt = 0; // Requests forwarded to the daemon and not yet answered, keyed by JSON-RPC id. // If the daemon dies mid-session (#662 — e.g. an MCP host SIGTERM's it when a // new session starts), these would otherwise hang forever; we re-serve them @@ -256,24 +280,47 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< const shutdown = (): void => { if (shuttingDown) return; shuttingDown = true; try { livenessWatchdog?.stop(); } catch { /* ignore */ } + if (retryTimer) clearTimeout(retryTimer); try { daemonSocket?.destroy(); } catch { /* ignore */ } try { engine?.stop(); } catch { /* ignore */ } process.exit(0); }; - const ensureEngine = (): Promise => { - if (!engine) engine = deps.makeEngine(); - if (!engineReady) engineReady = engine.ensureInitialized(deps.root).catch(() => { /* degraded */ }); - return engineReady; + // Resolves the engine a call was started on, so a call in flight while the + // engine is being retired still finishes on it. + const ensureEngine = async (): Promise => { + if (!engine) { + engine = deps.makeEngine(); + engineReady = engine.ensureInitialized(deps.root).catch(() => { /* degraded */ }); + } + const current = engine; + await engineReady; + return current; + }; + // Stop the in-process engine once the calls it is serving have answered. + // Stopping it releases the writer lock if it held one. + const retireEngine = async (): Promise => { + const retired = engine; + engine = null; + engineReady = null; + await Promise.allSettled([...localCalls]); + try { await retired?.stop(); } catch { /* best-effort */ } }; // Daemon-unavailable fallback: serve a client message in-process. - const handleLocally = async (line: string): Promise => { + const handleLocally = (line: string): Promise => { + const call = serveLocally(line); + localCalls.add(call); + const done = (): void => { localCalls.delete(call); }; + call.then(done, done); + return call; + }; + const serveLocally = async (line: string): Promise => { let msg: JsonRpc; try { msg = JSON.parse(line) as JsonRpc; } catch { return; } const id = msg.id; if (msg.method === 'tools/call' && id !== undefined) { try { - await ensureEngine(); + const local = await ensureEngine(); const params = (msg.params || {}) as { name: string; arguments?: Record }; - const result = await engine!.getToolHandler().execute(params.name, params.arguments || {}, exploreSession); + const result = await local.getToolHandler().execute(params.name, params.arguments || {}, exploreSession); writeClient({ jsonrpc: '2.0', id, result }); getTelemetry().recordUsage('mcp_tool', params.name, !result.isError, telemetryClient); } catch (err) { @@ -314,6 +361,7 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< let msg: JsonRpc; try { msg = JSON.parse(line) as JsonRpc; } catch { routeToDaemon(line); continue; } if (msg.method === 'initialize') { clientInitId = msg.id; + clientInitLine = line; const initParams = (msg.params ?? {}) as { clientInfo?: { name?: unknown; version?: unknown } }; if (initParams.clientInfo) { telemetryClient = { @@ -364,15 +412,58 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< }); // ---- daemon connection (background) ---- - let socket: net.Socket | null = null; - try { socket = await deps.getDaemonSocket(); } catch { socket = null; } + const connectDaemon = async (): Promise => { + let socket: net.Socket | null = null; + try { socket = await deps.getDaemonSocket(); } catch { socket = null; } + // `socket.destroyed`: the connect-window error guard above can absorb an + // 'error' that already destroyed the socket before we got here (#974) — treat + // a dead socket as "no daemon" so we cleanly fall back to the in-process engine. + if (socket && (socket.destroyed || shuttingDown)) { + try { socket.destroy(); } catch { /* ignore */ } + socket = null; + } + return socket; + }; + // Every buffered call binds to the engine now, so a later retirement waits for all of them. + const serveBufferedLocally = (): void => { + for (const line of pending.splice(0)) void handleLocally(line); + }; + const scheduleDaemonRetry = (): void => { + if (shuttingDown || retryTimer || retryBaseMs <= 0) return; + const delay = retryDelayMs; + retryDelayMs = Math.min(retryDelayMs * 2, retryMaxMs); + retryTimer = setTimeout(() => { retryTimer = null; void retryDaemon(); }, delay); + retryTimer.unref?.(); + }; + const retryDaemon = async (): Promise => { + if (shuttingDown || daemonStatus !== 'failed') return; + // While a writing engine owns writer.pid no daemon can start (#2277). Hand + // the lock back first, once the calls it is serving have answered, and + // buffer new calls until a daemon answers or we resume in-process. One + // writer at a time: the engine is fully stopped before a daemon is spawned. + const handover = engine !== null && !engine.isReadOnly(); + if (handover) { + daemonStatus = 'connecting'; + await retireEngine(); + } + const socket = await connectDaemon(); + if (shuttingDown) return; + if (socket) { + process.stderr.write('[CodeGraph MCP] Shared daemon reachable; proxying this session to it instead of serving in-process.\n'); + attachDaemon(socket, true); + // A read-only engine holds no lock: let its calls finish, then close it. + if (!handover) void retireEngine(); + return; + } + daemonStatus = 'failed'; + serveBufferedLocally(); + scheduleDaemonRetry(); + }; - // `!socket.destroyed`: the connect-window error guard above can absorb an - // 'error' that already destroyed the socket before we got here (#974) — treat - // a dead socket as "no daemon" so we cleanly fall back to the in-process engine. - if (socket && !socket.destroyed && !shuttingDown) { + const attachDaemon = (socket: net.Socket, replayInitialize: boolean): void => { daemonSocket = socket; daemonStatus = 'ready'; + attachedAt = Date.now(); let sockBuf = ''; socket.setEncoding('utf8'); socket.on('data', (chunk: string) => { @@ -397,33 +488,48 @@ export async function runLocalHandshakeProxy(deps: LocalHandshakeDeps): Promise< // The daemon going away does NOT end the session (#662). An MCP host can // SIGTERM the shared daemon when another session starts; if we exited here, // this host would silently lose CodeGraph and any in-flight request would - // hang. Instead, fall back to the in-process engine for the rest of the - // session and re-serve whatever the dead daemon never answered. + // hang. Instead, fall back to the in-process engine until a daemon answers + // again, and re-serve whatever the dead daemon never answered. const onDaemonLost = (): void => { - if (shuttingDown || daemonStatus !== 'ready') return; // host teardown, or already handled + if (shuttingDown || daemonSocket !== socket) return; // host teardown, or already handled daemonStatus = 'failed'; - try { daemonSocket?.destroy(); } catch { /* ignore */ } daemonSocket = null; + try { socket.destroy(); } catch { /* ignore */ } process.stderr.write( `[CodeGraph MCP] Shared daemon connection lost; serving this session in-process (degraded), re-serving ${inflight.size} in-flight request(s).\n` ); const orphaned = [...inflight.values()]; inflight.clear(); for (const line of orphaned) void handleLocally(line); + // A link that stayed up past the longest wait starts the backoff over; one + // that keeps dropping right after it attaches keeps backing off. + if (Date.now() - attachedAt >= retryMaxMs) retryDelayMs = retryBaseMs; + scheduleDaemonRetry(); }; socket.on('close', onDaemonLost); socket.on('error', onDaemonLost); + // A daemon that came up mid-session has never seen this client: prime its + // session with the client's own initialize, whose reply is suppressed + // above like the first one. + if (replayInitialize && clientInitLine !== undefined) { + try { socket.write(clientInitLine + '\n'); } catch { /* close path */ } + } for (const line of pending) { trackInflight(line); if (process.env.CODEGRAPH_MCP_DEBUG) process.stderr.write(`[mcp-debug] proxy-flush ${line.slice(0, 80)}\n`); try { socket.write(line + '\n'); } catch { /* ignore */ } } pending.length = 0; + }; + + const socket = await connectDaemon(); + if (socket) { + attachDaemon(socket, false); } else if (!shuttingDown) { daemonStatus = 'failed'; process.stderr.write('[CodeGraph MCP] Shared daemon unavailable; serving this session in-process (degraded).\n'); - const buffered = pending.splice(0); - for (const line of buffered) await handleLocally(line); + serveBufferedLocally(); + scheduleDaemonRetry(); } await new Promise(() => { /* stdin keeps the loop alive; exit via shutdown() */ }); @@ -588,10 +694,15 @@ function startPpidWatchdog(socket: net.Socket): void { } function parsePollMs(raw: string | undefined): number { - if (raw === undefined || raw === '') return DEFAULT_PPID_POLL_MS; + return parseDelayMs(raw, DEFAULT_PPID_POLL_MS); +} + +/** A non-negative millisecond env value; unset or malformed gives `fallback`. */ +function parseDelayMs(raw: string | undefined, fallback: number): number { + if (raw === undefined || raw === '') return fallback; const parsed = Number(raw); - if (!Number.isFinite(parsed)) return DEFAULT_PPID_POLL_MS; - if (parsed < 0) return DEFAULT_PPID_POLL_MS; + if (!Number.isFinite(parsed)) return fallback; + if (parsed < 0) return fallback; return Math.floor(parsed); } From 7bbc1fb7f6bc4627a6e7930f7229b34879a1cf57 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:18:46 +0000 Subject: [PATCH 187/363] fix(tests): the test suite no longer writes to the developer's real home dir (#2275) (#2295) Running the suite on a machine with Claude Code configured could wire the CodeGraph prompt hook into the developer's own ~/.claude/settings.json: every upgrade test whose fake upgrade succeeded ran the real post-upgrade prompt-hook self-heal, which detected the real global Claude profile and wrote it. Daemon suites likewise left records in the real ~/.codegraph/daemons. On Windows a HOME-only redirect would not have helped, since os.homedir() reads USERPROFILE there. - The prompt-hook writer is now injected through UpgradeDeps (wirePromptHook); the CLI passes the real defaultWirePromptHook and the upgrade tests a recorder. New tests pin that a successful upgrade calls the recorder and that the real writer never touches a configured profile. - A vitest setupFiles entry gives every engine test file a throwaway home: HOME/USERPROFILE (plus HOMEDRIVE/HOMEPATH/APPDATA/LOCALAPPDATA on Windows), XDG_CONFIG_HOME, and GIT_CONFIG_GLOBAL (seeded with a dummy identity) point into it, and CLAUDE_CONFIG_DIR/CODEX_HOME/COPILOT_HOME/ HERMES_HOME are cleared. Spawned CLI/MCP children inherit it. A sandbox test asserts os.homedir(), the Claude profile, the daemon registry, a child process and git's global config all resolve inside it. Co-authored-by: Claude Opus 5.5 --- AGENTS.md | 2 + __tests__/setup-home-sandbox.ts | 66 +++++++++++++++++ __tests__/test-home-sandbox.test.ts | 65 +++++++++++++++++ __tests__/upgrade.test.ts | 109 +++++++++++++++++++++++++++- src/bin/codegraph.ts | 1 + src/upgrade/index.ts | 26 ++++++- vitest.config.mts | 3 + vitest.workspace.mts | 6 +- 8 files changed, 270 insertions(+), 8 deletions(-) create mode 100644 __tests__/setup-home-sandbox.ts create mode 100644 __tests__/test-home-sandbox.test.ts diff --git a/AGENTS.md b/AGENTS.md index 2abe06c4ad..0f5e9a882e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -181,6 +181,8 @@ Tests live in `__tests__/` and mirror the module they cover. Notable ones beyond Tests create temp dirs with `fs.mkdtempSync` and clean up in `afterEach`. They write real files and exercise real SQLite — there is no DB mocking. +Every engine test file runs in a throwaway home dir (`__tests__/setup-home-sandbox.ts`, a `setupFiles` entry): `HOME`/`USERPROFILE` (+ Windows vars), `XDG_CONFIG_HOME` and `GIT_CONFIG_GLOBAL` point into it, and `CLAUDE_CONFIG_DIR`/`CODEX_HOME`/… are cleared; spawned children inherit it. It's a backstop — still inject writes to global state (e.g. `UpgradeDeps.wirePromptHook`, #2275). + ### Windows-gated tests Behavior that differs by platform (path resolution, drive letters, `SENSITIVE_PATHS`, `%APPDATA%` config dirs, CRLF) must be gated, not assumed. Use `it.runIf(process.platform === 'win32')(...)` for Windows-only assertions and `it.runIf(process.platform !== 'win32')(...)` for POSIX-only ones — e.g. `/etc` is sensitive on POSIX but resolves to `C:\etc` (non-existent) on Windows, so an ungated `/etc` assertion fails on Windows. Validate the Windows side for real (see below); don't merge a Windows-gated test you haven't seen run. diff --git a/__tests__/setup-home-sandbox.ts b/__tests__/setup-home-sandbox.ts new file mode 100644 index 0000000000..5acb473944 --- /dev/null +++ b/__tests__/setup-home-sandbox.ts @@ -0,0 +1,66 @@ +/** + * Give every engine test file a throwaway home directory (#2275). + * + * The suite exercises code that writes to the user's GLOBAL state: the Claude + * prompt hook in `~/.claude/settings.json`, the daemon registry under + * `~/.codegraph/daemons`, every agent target's global config. A test that + * forgot to redirect the home dir wrote to the developer's real one — a + * successful fake `codegraph upgrade` wired the prompt hook into their own + * Claude profile, and the daemon suites left thousands of registry records + * behind. CI never noticed: its runners have no Claude profile to edit. + * + * So the home dir is redirected here, before any test module loads, rather + * than file by file. Everything that resolves a home dir is covered: + * `os.homedir()` reads HOME on POSIX but USERPROFILE on Windows; git on + * Windows falls back to HOMEDRIVE+HOMEPATH; the per-agent overrides + * (CLAUDE_CONFIG_DIR, CODEX_HOME, ...) win over either, so they are cleared; + * and git's global config is pinned to a seeded file, so a test's `git + * commit` neither reads the developer's identity, signing or hooks nor + * depends on them. Spawned CLI / MCP processes inherit all of it. + * + * Tests that redirect the home dir themselves (spying on `os.homedir`, or + * setting HOME and restoring it) keep working: they restore to this sandbox. + */ +import { afterAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; + +// Resolved, so a comparison against a realpath'd cwd (macOS `/var` → +// `/private/var`) sees the same home everything else does. +const home = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-test-home-'))); + +process.env.HOME = home; +process.env.USERPROFILE = home; +if (process.platform === 'win32') { + const drive = /^([A-Za-z]:)(\\.*)$/.exec(home); + if (drive) { + process.env.HOMEDRIVE = drive[1]; + process.env.HOMEPATH = drive[2]; + } + process.env.APPDATA = path.join(home, 'AppData', 'Roaming'); + process.env.LOCALAPPDATA = path.join(home, 'AppData', 'Local'); +} else { + // Only ever set on Windows in real life; on POSIX a stray one would point + // the opencode/Copilot legacy-path logic somewhere real. + delete process.env.APPDATA; + delete process.env.LOCALAPPDATA; +} +process.env.XDG_CONFIG_HOME = path.join(home, '.config'); +for (const override of ['CLAUDE_CONFIG_DIR', 'CODEX_HOME', 'COPILOT_HOME', 'HERMES_HOME']) { + delete process.env[override]; +} + +// Both spellings: GIT_CONFIG_GLOBAL for git >= 2.32, and `$HOME/.gitconfig` +// (the same file) for older ones. +const gitConfig = path.join(home, '.gitconfig'); +fs.writeFileSync(gitConfig, '[user]\n\tname = CodeGraph Test\n\temail = test@codegraph.invalid\n'); +process.env.GIT_CONFIG_GLOBAL = gitConfig; + +afterAll(() => { + try { + fs.rmSync(home, { recursive: true, force: true, maxRetries: 3 }); + } catch { + /* a straggling child still holding a file there; it's in the temp dir */ + } +}); diff --git a/__tests__/test-home-sandbox.test.ts b/__tests__/test-home-sandbox.test.ts new file mode 100644 index 0000000000..f5ebb7a718 --- /dev/null +++ b/__tests__/test-home-sandbox.test.ts @@ -0,0 +1,65 @@ +/** + * The suite's home-dir sandbox (`setup-home-sandbox.ts`, #2275) is in force: + * nothing a test runs — in-process or spawned — resolves the developer's real + * home, Claude profile, daemon registry or git identity. + */ +import { describe, it, expect } from 'vitest'; +import { execFileSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { claudeTarget } from '../src/installer/targets/claude'; +import { getRegistryDir } from '../src/mcp/daemon-registry'; + +const tmpRoot = fs.realpathSync(os.tmpdir()); +const under = (p: string, dir: string): boolean => path.resolve(p).startsWith(dir + path.sep); +/** The sandbox home — asserted to be one, so the checks below can't pass on a real home. */ +function sandboxHome(): string { + const home = os.homedir(); + expect(under(home, tmpRoot), home).toBe(true); + return home; +} + +describe('test home sandbox', () => { + it('os.homedir() is a throwaway dir under the temp root, not the real home', () => { + const home = sandboxHome(); + expect(path.basename(home)).toMatch(/^codegraph-test-home-/); + // The account's profile dir, which ignores HOME / USERPROFILE. + expect(home).not.toBe(os.userInfo().homedir); + }); + + it('the global Claude profile and the daemon registry resolve inside it', () => { + const home = sandboxHome(); + expect(process.env.CLAUDE_CONFIG_DIR).toBeUndefined(); + expect(under(claudeTarget.detect('global').configPath, home)).toBe(true); + expect(under(getRegistryDir(), home)).toBe(true); + }); + + it('a spawned child sees the same home', () => { + const childHome = execFileSync(process.execPath, ['-p', 'require("os").homedir()'], { + encoding: 'utf-8', + windowsHide: true, + }).trim(); + expect(childHome).toBe(sandboxHome()); + }); + + it("git's global config is the seeded sandbox file", () => { + const gitConfig = process.env.GIT_CONFIG_GLOBAL ?? ''; + expect(under(gitConfig, sandboxHome())).toBe(true); + const name = execFileSync('git', ['config', '--global', '--get', 'user.name'], { + encoding: 'utf-8', + windowsHide: true, + }).trim(); + expect(name).toBe('CodeGraph Test'); + }); + + // os.homedir() reads USERPROFILE there, and git falls back to + // HOMEDRIVE+HOMEPATH — a HOME-only sandbox would not hold. + it.runIf(process.platform === 'win32')('covers the Windows home variables', () => { + const home = sandboxHome(); + expect(process.env.USERPROFILE).toBe(home); + expect(`${process.env.HOMEDRIVE}${process.env.HOMEPATH}`).toBe(home); + expect(under(process.env.APPDATA ?? '', home)).toBe(true); + expect(under(process.env.LOCALAPPDATA ?? '', home)).toBe(true); + }); +}); diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index 244957af4e..e1b05e4d40 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -14,6 +14,7 @@ import { reindexAdvisory, runUpgrade, verifyResolvedVersion, + defaultWirePromptHook, buildWindowsUpgradeScript, NPM_PACKAGE, type InstallMethod, @@ -230,13 +231,15 @@ interface Calls { captures: Array<{ cmd: string; args: string[] }>; logs: string[]; errors: string[]; + /** How many times the upgrade asked deps to wire the Claude prompt hook. */ + promptHookWires: number; } function makeDeps( overrides: Partial & { method: InstallMethod; currentVersion: string }, runExit = 0 ): { deps: UpgradeDeps; calls: Calls } { - const calls: Calls = { runs: [], captures: [], logs: [], errors: [] }; + const calls: Calls = { runs: [], captures: [], logs: [], errors: [], promptHookWires: 0 }; const deps: UpgradeDeps = { currentVersion: overrides.currentVersion, method: overrides.method, @@ -252,6 +255,12 @@ function makeDeps( return overrides.capture ? overrides.capture(cmd, args) : null; }, hasCommand: overrides.hasCommand ?? ((c) => c === 'curl'), + // A recorder, never the real writer: that one edits the GLOBAL Claude + // profile, which here would be the developer's own (#2275). + wirePromptHook: async () => { + calls.promptHookWires += 1; + return overrides.wirePromptHook ? overrides.wirePromptHook() : false; + }, log: (m) => calls.logs.push(m), warn: (m) => calls.logs.push(m), error: (m) => calls.errors.push(m), @@ -473,6 +482,104 @@ describe('runUpgrade beta signup offer', () => { }); }); +// --------------------------------------------------------------------------- +// Post-upgrade prompt-hook self-heal — only ever through deps (#2275). The +// real writer edits the GLOBAL Claude profile; while the upgrade called it +// directly, every successful fake upgrade in this file could wire the hook +// into the developer's own ~/.claude/settings.json. +// --------------------------------------------------------------------------- + +describe('post-upgrade prompt-hook self-heal', () => { + // A configured global Claude profile the REAL writer would act on, so an + // upgrade that bypasses deps shows up as a settings.json written here. + const KEYS = ['CLAUDE_CONFIG_DIR', 'CODEGRAPH_NO_PROMPT_HOOK', 'CODEGRAPH_PROMPT_HOOK'] as const; + const saved: Partial> = {}; + let profile: string; + + function configureProfile(): void { + fs.writeFileSync( + path.join(profile, '.claude.json'), + JSON.stringify({ mcpServers: { codegraph: { command: 'codegraph', args: ['serve', '--mcp'] } } }), + ); + } + + beforeEach(() => { + for (const k of KEYS) saved[k] = process.env[k]; + delete process.env.CODEGRAPH_NO_PROMPT_HOOK; + delete process.env.CODEGRAPH_PROMPT_HOOK; + profile = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-upgrade-claude-')); + process.env.CLAUDE_CONFIG_DIR = profile; + }); + afterEach(() => { + for (const k of KEYS) { + if (saved[k] === undefined) delete process.env[k]; + else process.env[k] = saved[k]; + } + fs.rmSync(profile, { recursive: true, force: true }); + }); + + it('a successful upgrade wires the hook through deps, never the real writer', async () => { + configureProfile(); + const { deps, calls } = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(0); + expect(fs.existsSync(path.join(profile, 'settings.json')), 'the real writer ran').toBe(false); + expect(calls.promptHookWires).toBe(1); + }); + + it('notes the hook only when the writer changed something', async () => { + const quiet = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, quiet.deps)).toBe(0); + expect(quiet.calls.logs.join('\n')).not.toMatch(/front-load hook/); + + const wired = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + wirePromptHook: async () => true, + }); + expect(await runUpgrade({}, wired.deps)).toBe(0); + expect(wired.calls.logs.join('\n')).toMatch(/Enabled the CodeGraph front-load hook/); + }); + + it('the kill-switch skips the writer entirely', async () => { + process.env.CODEGRAPH_NO_PROMPT_HOOK = '1'; + const { deps, calls } = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({}, deps)).toBe(0); + expect(calls.promptHookWires).toBe(0); + }); + + it('does not wire on --check, when up to date, or when the upgrade fails', async () => { + const check = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }); + expect(await runUpgrade({ check: true }, check.deps)).toBe(0); + const current = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.9' }); + expect(await runUpgrade({}, current.deps)).toBe(0); + const failed = makeDeps({ method: { kind: 'npm', scope: 'global' }, currentVersion: '0.9.8' }, 1); + expect(await runUpgrade({}, failed.deps)).toBe(1); + for (const { calls } of [check, current, failed]) expect(calls.promptHookWires).toBe(0); + }); + + it('a throwing writer never fails the upgrade', async () => { + const { deps } = makeDeps({ + method: { kind: 'npm', scope: 'global' }, + currentVersion: '0.9.8', + wirePromptHook: async () => { throw new Error('EACCES'); }, + }); + expect(await runUpgrade({}, deps)).toBe(0); + }); + + // The production writer itself, pointed at the temp profile above. + it('the real writer wires a configured profile once, and leaves an unconfigured one alone', async () => { + const settings = path.join(profile, 'settings.json'); + expect(await defaultWirePromptHook()).toBe(false); + expect(fs.existsSync(settings)).toBe(false); + + configureProfile(); + expect(await defaultWirePromptHook()).toBe(true); + const written = JSON.parse(fs.readFileSync(settings, 'utf-8')); + expect(JSON.stringify(written.hooks.UserPromptSubmit)).toMatch(/prompt-hook/); + expect(await defaultWirePromptHook()).toBe(false); // idempotent + }); +}); + // --------------------------------------------------------------------------- // Post-upgrade self-heal of installed agent surfaces // --------------------------------------------------------------------------- diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index 64078dfab6..e5f8eacfc6 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -2859,6 +2859,7 @@ program run: up.defaultRun, capture: up.defaultCapture, hasCommand: up.hasCommand, + wirePromptHook: up.defaultWirePromptHook, log: (m: string) => console.log(m), warn: (m: string) => warn(m), error: (m: string) => error(m), diff --git a/src/upgrade/index.ts b/src/upgrade/index.ts index 625e45b831..76354378a8 100644 --- a/src/upgrade/index.ts +++ b/src/upgrade/index.ts @@ -297,6 +297,15 @@ export interface UpgradeDeps { warn: (msg: string) => void; error: (msg: string) => void; platform: NodeJS.Platform; + /** + * Wire Claude Code's front-load prompt hook into the GLOBAL Claude profile + * when that profile already has CodeGraph configured; resolves true when it + * changed the settings file. That file is the user's real + * `~/.claude/settings.json`, so the writer is injected like every other side + * effect here — the CLI passes {@link defaultWirePromptHook}, unit tests a + * recorder (#2275: tests that reached the real one rewrote the developer's). + */ + wirePromptHook: () => Promise; /** * Offer the one-time CodeGraph Pro beta opt-in after a successful update * (see installer/beta-signup — self-gating: TTY only, and silent forever @@ -528,10 +537,7 @@ function selfHealInstalledSurfaces(deps: UpgradeDeps): void { */ async function selfHealPromptHook(deps: UpgradeDeps): Promise { if (process.env.CODEGRAPH_NO_PROMPT_HOOK === '1' || process.env.CODEGRAPH_PROMPT_HOOK === '0') return; - const { claudeTarget, writePromptHookEntry } = await import('../installer/targets/claude'); - if (!claudeTarget.detect('global').alreadyConfigured) return; - const res = writePromptHookEntry('global'); - if (res.action === 'created' || res.action === 'updated') { + if (await deps.wirePromptHook()) { deps.log( c.dim('Enabled the CodeGraph front-load hook for Claude Code (structural prompts). Disable any time: CODEGRAPH_NO_PROMPT_HOOK=1'), ); @@ -714,3 +720,15 @@ export function defaultCapture(cmd: string, args: string[]): { code: number; std if (r.error) return null; return { code: r.status ?? -1, stdout: r.stdout ?? '' }; } + +/** + * The production `UpgradeDeps.wirePromptHook`: writes the hook only when the + * global Claude profile already carries CodeGraph's MCP entry, and leaves the + * file byte-for-byte alone once the hook is there. + */ +export async function defaultWirePromptHook(): Promise { + const { claudeTarget, writePromptHookEntry } = await import('../installer/targets/claude'); + if (!claudeTarget.detect('global').alreadyConfigured) return false; + const res = writePromptHookEntry('global'); + return res.action === 'created' || res.action === 'updated'; +} diff --git a/vitest.config.mts b/vitest.config.mts index ae02ba84c6..96fb2573a6 100644 --- a/vitest.config.mts +++ b/vitest.config.mts @@ -14,6 +14,9 @@ export default defineConfig({ include: ['__tests__/**/*.test.ts'], // Suites that spawn the built CLI need a current dist/ (#1879). globalSetup: ['./__tests__/global-setup-dist.ts'], + // A throwaway home dir (and git global config) per test file, so nothing + // the suite runs can write to the developer's real one (#2275). + setupFiles: ['./__tests__/setup-home-sandbox.ts'], /** * Several MCP integration tests (mcp-daemon, mcp-initialize, mcp-ppid-watchdog, * mcp-roots) spawn `dist/bin/codegraph.js serve --mcp` with `process.execPath` diff --git a/vitest.workspace.mts b/vitest.workspace.mts index c37fae14ba..9abdc721c7 100644 --- a/vitest.workspace.mts +++ b/vitest.workspace.mts @@ -20,9 +20,9 @@ import { defineWorkspace } from 'vitest/config'; * browser builds of `web-tree-sitter` and friends, and the failures that * causes look nothing like their cause. * - * The engine project `extends` the shared base, so the env vars and Node guard - * in `vitest.config.mts` still apply to every engine test. The ui project does - * not — see the note on it. + * The engine project `extends` the shared base, so the env vars, Node guard and + * home-dir sandbox in `vitest.config.mts` still apply to every engine test. The + * ui project does not — see the note on it. */ export default defineWorkspace([ { From 273e5cdc4c2ce60a5e99b134c2d05681a8de6cd7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 06:19:41 +0000 Subject: [PATCH 188/363] fix(upgrade): a Windows upgrade no longer breaks the install while CodeGraph is running (#2185) (#2294) * fix(upgrade): a Windows upgrade no longer breaks the install while CodeGraph is running (#2185) On Windows, `codegraph upgrade` renamed only node.exe aside and then Copy-Item'd the new bundle over current\. Any open agent session runs a CodeGraph MCP server from current\, which keeps the native kernel loaded, so the copy failed on lib\kernel\codegraph-kernel.node halfway through, leaving no node.exe and a mix of versions that no codegraph command could repair. Re-running install.ps1 failed the same way, because it Remove-Items current\. Both now share one PowerShell function (a test pins the two copies equal). The bundle is unpacked next to current\ (same volume), then swapped in file by file: every file being replaced or dropped is first renamed aside (Windows allows renaming a running exe or a loaded DLL, just not overwriting or deleting it), then the staged file is moved in. Any failure or interruption undoes every step. Renamed-aside files, including the node.exe.old-* left by earlier upgrades, are deleted on the next successful run once nothing holds them. The upgrade's exit code now tells whether the install was left unchanged or needs a reinstall, and the message says which. Co-Authored-By: Claude Opus 5.5 * docs(changelog): Windows upgrade with CodeGraph running (#2185) Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/upgrade.test.ts | 154 ++++++++++++++++++++++++++++++++-- install.ps1 | 105 ++++++++++++++++++++--- src/upgrade/index.ts | 172 +++++++++++++++++++++++++++++++------- 4 files changed, 382 insertions(+), 50 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cf27df4914..348a804c87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -173,6 +173,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In tag-based CFML, calls written in tags are now linked: in ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) +- On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) ## [1.6.1] - 2026-09-29 diff --git a/__tests__/upgrade.test.ts b/__tests__/upgrade.test.ts index e1b05e4d40..ad46a17ad4 100644 --- a/__tests__/upgrade.test.ts +++ b/__tests__/upgrade.test.ts @@ -16,6 +16,8 @@ import { verifyResolvedVersion, defaultWirePromptHook, buildWindowsUpgradeScript, + WINDOWS_SWAP_FUNCTION, + WINDOWS_UPGRADE_DAMAGED, NPM_PACKAGE, type InstallMethod, type UpgradeDeps, @@ -211,17 +213,124 @@ describe('version helpers', () => { expect(a).toContain('codegraph index -f'); }); - it('buildWindowsUpgradeScript targets the right asset per arch and renames-not-deletes the exe', () => { + it('buildWindowsUpgradeScript targets the right asset per arch', () => { const arm = buildWindowsUpgradeScript('C:\\cg\\current', 'v1.2.3', 'arm64'); expect(arm).toContain('releases/download/v1.2.3/codegraph-win32-arm64.zip'); expect(arm).toContain("$dest='C:\\cg\\current'"); - expect(arm).toContain('Rename-Item'); // never Remove-Item on the locked exe - expect(arm).not.toMatch(/Remove-Item[^;]*\$dest'?\s*;/); // doesn't delete current\ + expect(arm).toContain("Join-Path $stage 'codegraph-win32-arm64'"); const x64 = buildWindowsUpgradeScript('C:\\cg\\current', 'v1.2.3', 'x64'); expect(x64).toContain('codegraph-win32-x64.zip'); }); }); +// --------------------------------------------------------------------------- +// Windows file swap (#2185) — a running CodeGraph process (an agent session's +// MCP server) keeps node.exe and the native kernel locked: they can be renamed +// but not overwritten or deleted. The upgrade used to rename only node.exe and +// then Copy-Item over the rest, so the locked kernel failed the copy halfway +// and left the install with no node.exe. PowerShell can't run on the macOS +// test host, so these pin the script's shape; the behavior is validated on a +// real Windows machine. +// --------------------------------------------------------------------------- + +describe('windows bundle swap script (#2185)', () => { + const INSTALL_PS1 = path.join(__dirname, '..', 'install.ps1'); + const installPs1 = () => fs.readFileSync(INSTALL_PS1, 'utf-8').replace(/\r\n/g, '\n'); + const script = () => buildWindowsUpgradeScript('C:\\Users\\me\\AppData\\Local\\codegraph\\current', 'v1.6.2', 'x64'); + /** The swap function's own statements, comments dropped. */ + const swapCode = () => WINDOWS_SWAP_FUNCTION.split('\n').filter((l) => !l.trim().startsWith('#')).join('\n'); + + it('install.ps1 carries the same swap function, verbatim', () => { + const m = /# >>> Install-CodeGraphFiles[^\n]*\n([\s\S]*?)# <<< Install-CodeGraphFiles/.exec(installPs1()); + expect(m, 'install.ps1 lost its Install-CodeGraphFiles markers').not.toBeNull(); + expect(m![1]).toBe(WINDOWS_SWAP_FUNCTION); + }); + + it('never copies over or deletes current\\ in place', () => { + for (const text of [script(), installPs1()]) { + expect(text).not.toMatch(/Copy-Item/); + expect(text).not.toMatch(/Remove-Item[^\n]*\$dest\b/); + expect(text).not.toMatch(/Expand-Archive[^\n]*\$dest\b/); + } + }); + + it('unpacks next to current\\ (same volume), not into %TEMP%, then swaps', () => { + const s = script(); + expect(s).toContain(`$stage=Join-Path (Split-Path -Parent $dest) ('.staging-'`); + expect(s).toMatch(/Expand-Archive -Path \$zip -DestinationPath \$stage/); + expect(s.indexOf('Expand-Archive')).toBeLessThan(s.indexOf('Install-CodeGraphFiles $(')); + // install.ps1 stages next to current\ too. + expect(installPs1()).toContain(`$stage = Join-Path $installDir ('.staging-'`); + expect(installPs1()).toMatch(/Install-CodeGraphFiles \$\(.*\) \$dest$/m); + }); + + it('renames every replaced file aside before moving the staged file in', () => { + const code = swapCode(); + // Replaced files and files the new version drops are both renamed aside… + expect(code.match(/Move-Logged \$at "\$at\.old-\$token"/g)).toHaveLength(2); + // …and the aside-rename of a target comes before the staged file moves in. + const aside = code.indexOf('if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" }'); + const moveIn = code.indexOf('Move-Logged ($stageDir + $rel) $at'); + expect(aside).toBeGreaterThan(0); + expect(moveIn).toBeGreaterThan(aside); + // No in-place overwrite or delete of a live file during the swap. + expect(code).not.toMatch(/\[IO\.File\]::(Copy|Replace)\(/); + expect(code).not.toMatch(/Move-Item|Rename-Item/); + }); + + it('rolls back on failure and on interruption, and says whether the install still works', () => { + const code = swapCode(); + expect(code).toMatch(/catch \{[\s\S]*\$lost = Undo-Logged[\s\S]*\} finally \{\s*if \(-not \$done\) \{ \[void\]\(Undo-Logged\) \}/); + // Undo walks the log backwards, moving each file back to where it was. + expect(code).toContain('for ($n = $undo.Count - 1; $n -ge 0; $n--)'); + expect(code).toContain('[IO.File]::Move($u[1], $u[0])'); + expect(code).toContain('Nothing was changed: the existing install still works.'); + expect(code).toContain("$e.Data['codegraphDamaged'] = [bool]$lost"); + // The upgrade script maps that flag to the exit code runUpgrade reads. + expect(script()).toContain(`$code=if($_.Exception.Data['codegraphDamaged']){${WINDOWS_UPGRADE_DAMAGED}}else{1}`); + expect(script().trimEnd().endsWith('exit $code')).toBe(true); + }); + + it('refuses a download that is not a bundle before touching current\\', () => { + const code = swapCode(); + expect(code).toContain("foreach ($need in 'node.exe', 'bin\\codegraph.cmd')"); + expect(code.indexOf("foreach ($need in")).toBeLessThan(code.indexOf('try {')); + }); + + it('cleans up renamed-aside leftovers, including ones from earlier upgrades', () => { + const m = /\$asideName = '([^']+)'/.exec(WINDOWS_SWAP_FUNCTION); + expect(m).not.toBeNull(); + const aside = new RegExp(m![1]!, 'i'); // PowerShell -match is case-insensitive + // The pre-fix upgrade left node.exe.old-<32-hex guid>; this one uses 8 hex. + expect(aside.test('node.exe.old-0123456789abcdef0123456789ABCDEF')).toBe(true); + expect(aside.test('codegraph-kernel.node.old-deadbeef')).toBe(true); + for (const shipped of ['node.exe', 'codegraph-kernel.node', 'old-deadbeef.js', 'x.old-1234567', 'a.old-deadbeef.js']) { + expect(aside.test(shipped), shipped).toBe(false); + } + // No file a bundle actually ships looks like a leftover. + const walk = (dir: string): string[] => + fs.readdirSync(dir, { withFileTypes: true }).flatMap((e) => (e.isDirectory() ? walk(path.join(dir, e.name)) : [e.name])); + const distDir = path.join(__dirname, '..', 'dist'); + if (fs.existsSync(distDir)) expect(walk(distDir).filter((n) => aside.test(n))).toEqual([]); + // Deleted only after a successful swap, and a still-locked one is skipped. + const code = swapCode(); + expect(code.indexOf('[IO.File]::Delete($f.FullName)')).toBeGreaterThan(code.indexOf('$done = $true')); + expect(code).toContain('try { [IO.File]::Delete($f.FullName) } catch {}'); + }); + + it('quotes the install path for PowerShell', () => { + const s = buildWindowsUpgradeScript("C:\\Users\\o'brien\\codegraph\\current", 'v1.6.2', 'x64'); + expect(s).toContain("$dest='C:\\Users\\o''brien\\codegraph\\current'"); + }); + + it('fits a Windows command line even for a long install path', () => { + const root = `C:\\${'very-long-directory-name\\'.repeat(8)}codegraph\\current`; + const encoded = Buffer.from(buildWindowsUpgradeScript(root, 'v10.20.30', 'arm64'), 'utf16le').toString('base64'); + // CreateProcess caps the whole command line at 32,767 characters. + expect(encoded.length).toBeLessThan(24_000); + }); +}); + // --------------------------------------------------------------------------- // runUpgrade orchestration — mocked side-effects // --------------------------------------------------------------------------- @@ -335,11 +444,10 @@ describe('runUpgrade', () => { expect(calls.runs).toHaveLength(1); expect(calls.runs[0].cmd).toBe('powershell.exe'); const decoded = decodeEncodedCommand(calls.runs[0].args); - // Downloads the right asset, renames the locked exe aside, copies over current\. + // Downloads the right asset and swaps it in with the shared rename-aside function. expect(decoded).toContain('releases/download/v0.9.9/codegraph-win32-'); - expect(decoded).toContain('Rename-Item'); - expect(decoded).toContain('node.exe.old-'); - expect(decoded).toContain('Copy-Item'); + expect(decoded).toContain(WINDOWS_SWAP_FUNCTION); + expect(decoded).toMatch(/^\s*Install-CodeGraphFiles .* \$dest$/m); }); it('windows bundle: a non-zero installer exit is a failure', async () => { @@ -353,7 +461,37 @@ describe('runUpgrade', () => { ); const code = await runUpgrade({}, deps); expect(code).toBe(1); - expect(calls.errors.join('\n')).toMatch(/exited with code/i); + // Exit 1 is the script's "failed, install unchanged" code (#2185). + expect(calls.errors.join('\n')).toMatch(/did not complete; your existing install was left as it was/i); + }); + + it('windows bundle: an incomplete rollback points at the reinstall command', async () => { + const { deps, calls } = makeDeps( + { + method: { kind: 'bundle', os: 'windows', bundleRoot: 'C:/x/codegraph/current', installDir: 'C:/x/codegraph' }, + currentVersion: '0.9.8', + platform: 'win32', + }, + WINDOWS_UPGRADE_DAMAGED + ); + const code = await runUpgrade({}, deps); + expect(code).toBe(1); + expect(calls.errors.join('\n')).toMatch(/could not be put back/i); + expect(calls.logs.join('\n')).toContain('install.ps1 | iex'); + expect(calls.runs).toHaveLength(1); // no post-upgrade refresh/probe after a failure + }); + + it('windows bundle: any other exit code is reported as-is', async () => { + const { deps, calls } = makeDeps( + { + method: { kind: 'bundle', os: 'windows', bundleRoot: 'C:/x/codegraph/current', installDir: 'C:/x/codegraph' }, + currentVersion: '0.9.8', + platform: 'win32', + }, + -1 + ); + expect(await runUpgrade({}, deps)).toBe(1); + expect(calls.errors.join('\n')).toMatch(/exited with code -1/i); }); it('npm global: shells out to npm install -g @pkg@latest', async () => { diff --git a/install.ps1 b/install.ps1 index 7c0b523caf..a079205b46 100644 --- a/install.ps1 +++ b/install.ps1 @@ -5,7 +5,8 @@ # # irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex # -# Upgrade with `codegraph upgrade` (or just re-run this). To uninstall: remove +# Upgrade with `codegraph upgrade` (or just re-run this -- safe even while agent +# sessions are running CodeGraph). To uninstall: remove # $env:LOCALAPPDATA\codegraph and drop its \current\bin entry from your user PATH. # # Environment: @@ -13,6 +14,81 @@ # CODEGRAPH_INSTALL_DIR install location (default: %LOCALAPPDATA%\codegraph) $ErrorActionPreference = 'Stop' + +# >>> Install-CodeGraphFiles -- keep identical to WINDOWS_SWAP_FUNCTION in src/upgrade/index.ts +function Install-CodeGraphFiles([string]$Stage, [string]$Dest) { + # Move an unpacked bundle into $Dest. Windows can't overwrite or delete a + # running node.exe or a loaded .node addon, but it can rename one, so every + # file being replaced (or dropped by the new version) is first renamed aside + # to .old-. Any failure puts every file back, so the install is + # never left half-replaced or without its node.exe. + $ErrorActionPreference = 'Stop' + $stageDir = (Resolve-Path -LiteralPath $Stage).ProviderPath.TrimEnd('\') + foreach ($need in 'node.exe', 'bin\codegraph.cmd') { + if (-not (Test-Path -LiteralPath (Join-Path $stageDir $need))) { throw "The CodeGraph download is incomplete (no $need); nothing was changed." } + } + $token = [guid]::NewGuid().ToString('N').Substring(0, 8) + $asideName = '\.old-[0-9a-f]{8,32}$' + $files = @{}; $dirs = @{} + foreach ($i in @(Get-ChildItem -LiteralPath $stageDir -Recurse -Force)) { + $rel = $i.FullName.Substring($stageDir.Length) + if ($i.PSIsContainer) { $dirs[$rel] = $true } else { $files[$rel] = $true } + } + $undo = New-Object System.Collections.ArrayList + function Move-Logged([string]$From, [string]$To) { [IO.File]::Move($From, $To); [void]$undo.Add(@($From, $To)) } + function Undo-Logged { + $lost = 0 + for ($n = $undo.Count - 1; $n -ge 0; $n--) { + $u = $undo[$n] + try { if ($u.Count -eq 2) { [IO.File]::Move($u[1], $u[0]) } else { [IO.Directory]::Delete($u[0]) } } catch { if ($u.Count -eq 2) { $lost++ } } + } + $undo.Clear() + $lost + } + $done = $false; $at = $Dest + try { + if (-not (Test-Path -LiteralPath $Dest)) { [void][IO.Directory]::CreateDirectory($Dest); [void]$undo.Add(@($Dest)) } + $destDir = (Resolve-Path -LiteralPath $Dest).ProviderPath.TrimEnd('\') + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File)) { + if (-not $files.ContainsKey($f.FullName.Substring($destDir.Length)) -and $f.Name -notmatch $asideName) { + $at = $f.FullName; Move-Logged $at "$at.old-$token" + } + } + foreach ($rel in @($dirs.Keys | Sort-Object Length)) { + $at = $destDir + $rel + if (-not [IO.Directory]::Exists($at)) { [void][IO.Directory]::CreateDirectory($at); [void]$undo.Add(@($at)) } + } + foreach ($rel in @($files.Keys)) { + $at = $destDir + $rel + if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" } + Move-Logged ($stageDir + $rel) $at + } + $done = $true + } catch { + $x = $_.Exception; while ($x.InnerException) { $x = $x.InnerException } + $lost = Undo-Logged + $msg = "Could not replace $at ($($x.Message))." + if ($lost) { + $msg += " $lost file(s) could not be put back, so the install may not start. Close your agent sessions and any running codegraph commands, then reinstall: irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex" + } else { + $msg += " Nothing was changed: the existing install still works. If another program has CodeGraph's files open, close your agent sessions (they run the CodeGraph MCP server) and any running codegraph commands, then try again." + } + $e = New-Object System.Exception($msg); $e.Data['codegraphDamaged'] = [bool]$lost; throw $e + } finally { + # Interrupted (Ctrl+C) without reaching catch: still put everything back. + if (-not $done) { [void](Undo-Logged) } + } + # Delete what this run and earlier ones renamed aside. A file a running + # process still holds can't be deleted yet; the next install retries it. + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File -ErrorAction SilentlyContinue)) { + if ($f.Name -match $asideName) { try { [IO.File]::Delete($f.FullName) } catch {} } + } + foreach ($d in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -Directory -ErrorAction SilentlyContinue | Sort-Object { $_.FullName.Length } -Descending)) { + if (-not $dirs.ContainsKey($d.FullName.Substring($destDir.Length))) { try { [IO.Directory]::Delete($d.FullName) } catch {} } + } +} +# <<< Install-CodeGraphFiles + $repo = 'colbymchenry/codegraph' $installDir = if ($env:CODEGRAPH_INSTALL_DIR) { $env:CODEGRAPH_INSTALL_DIR } else { Join-Path $env:LOCALAPPDATA 'codegraph' } @@ -27,25 +103,28 @@ if (-not $version) { } if (-not $version) { throw "codegraph: could not resolve latest version; set CODEGRAPH_VERSION." } -# 3. Download + extract the bundle into a stable 'current' dir (overwritten on upgrade). +# 3. Download the bundle and unpack it next to the stable 'current' dir (same +# volume, so moving it in is renames, not copies), then move it into 'current'. +# 'current' is never deleted wholesale: a CodeGraph process that is still +# running (an agent session's MCP server) holds node.exe and the native kernel, +# which can't be deleted -- Install-CodeGraphFiles renames them aside instead. $url = "https://github.com/$repo/releases/download/$version/codegraph-$target.zip" Write-Host "Installing CodeGraph $version ($target)..." $tmp = Join-Path $env:TEMP ("cg-" + [guid]::NewGuid().ToString()) New-Item -ItemType Directory -Force -Path $tmp | Out-Null $zip = Join-Path $tmp 'cg.zip' -Invoke-WebRequest -Uri $url -OutFile $zip - $dest = Join-Path $installDir 'current' -if (Test-Path $dest) { Remove-Item -Recurse -Force $dest } -New-Item -ItemType Directory -Force -Path $dest | Out-Null -Expand-Archive -Path $zip -DestinationPath $dest -Force -# Archives contain a top-level codegraph-\ dir; flatten it. -$inner = Join-Path $dest "codegraph-$target" -if (Test-Path $inner) { - Get-ChildItem -Force $inner | Move-Item -Destination $dest -Force - Remove-Item -Recurse -Force $inner +$stage = Join-Path $installDir ('.staging-' + [guid]::NewGuid().ToString('N').Substring(0, 8)) +try { + Invoke-WebRequest -Uri $url -OutFile $zip + New-Item -ItemType Directory -Force -Path $stage | Out-Null + Expand-Archive -Path $zip -DestinationPath $stage -Force + # Archives contain a top-level codegraph-\ dir. + $inner = Join-Path $stage "codegraph-$target" + Install-CodeGraphFiles $(if (Test-Path $inner) { $inner } else { $stage }) $dest +} finally { + Remove-Item -LiteralPath $stage, $tmp -Recurse -Force -ErrorAction SilentlyContinue } -Remove-Item -Recurse -Force $tmp # 4. Put the launcher dir on the user's PATH. $binDir = Join-Path $dest 'bin' diff --git a/src/upgrade/index.ts b/src/upgrade/index.ts index 76354378a8..0e088e9982 100644 --- a/src/upgrade/index.ts +++ b/src/upgrade/index.ts @@ -17,11 +17,12 @@ * vendored `node` binary and a `bin/codegraph` launcher next to its `lib/`, so * we can recognize it from the running file's path without a marker file. * - * Windows wrinkle: a running `node.exe` is locked and can't be deleted, so the - * bundle's `current\` dir can't be overwritten in place by the process doing - * the upgrade. We therefore spawn a DETACHED helper that waits for this - * process to exit (releasing the lock), then runs `install.ps1`. This is the - * conventional Windows self-update dance (rustup/nvm-windows do the same). + * Windows wrinkle: a running `node.exe` and a loaded `.node` addon are locked — + * they can't be overwritten or deleted, by this process or by the MCP servers + * of open agent sessions. They CAN be renamed, so the Windows upgrade unpacks + * the new bundle next to `current\` and swaps it in file by file, renaming each + * replaced file aside and rolling every step back on failure (see + * `WINDOWS_SWAP_FUNCTION`, shared verbatim with `install.ps1`). */ import * as fs from 'fs'; @@ -34,6 +35,7 @@ export const REPO = 'colbymchenry/codegraph'; export const NPM_PACKAGE = '@colbymchenry/codegraph'; const RAW_BASE = `https://raw.githubusercontent.com/${REPO}/main`; export const INSTALL_SH_URL = `${RAW_BASE}/install.sh`; +export const INSTALL_PS1_URL = `${RAW_BASE}/install.ps1`; // --------------------------------------------------------------------------- // Install-method detection (pure — fully unit-testable via injected probes) @@ -578,36 +580,140 @@ function upgradeUnixBundle( return 0; } +/** + * The PowerShell function that moves an unpacked Windows bundle into the + * install's `current\` dir. Shared VERBATIM with `install.ps1` (a test pins the + * two copies equal), so a first install, a re-run of the installer, and + * `codegraph upgrade` all replace files the same way. + * + * Why file-by-file renames (#2185): every open agent session runs a CodeGraph + * MCP server from `current\`, which keeps `node.exe` and the native kernel + * (`lib\kernel\codegraph-kernel.node`) locked. Windows refuses to overwrite or + * delete a running exe or a loaded DLL but does let it be renamed. The old + * upgrade renamed only `node.exe` and then `Copy-Item`ed over the rest, so the + * locked kernel failed the copy halfway — leaving no `node.exe` and a mix of + * versions, which no `codegraph` command could repair. Now each file being + * replaced (or dropped by the new version) is renamed aside to + * `.old-` before the staged file is moved in; any failure undoes + * every step, and the renamed-aside files are deleted on the next successful + * run once nothing holds them. + */ +export const WINDOWS_SWAP_FUNCTION = String.raw`function Install-CodeGraphFiles([string]$Stage, [string]$Dest) { + # Move an unpacked bundle into $Dest. Windows can't overwrite or delete a + # running node.exe or a loaded .node addon, but it can rename one, so every + # file being replaced (or dropped by the new version) is first renamed aside + # to .old-. Any failure puts every file back, so the install is + # never left half-replaced or without its node.exe. + $ErrorActionPreference = 'Stop' + $stageDir = (Resolve-Path -LiteralPath $Stage).ProviderPath.TrimEnd('\') + foreach ($need in 'node.exe', 'bin\codegraph.cmd') { + if (-not (Test-Path -LiteralPath (Join-Path $stageDir $need))) { throw "The CodeGraph download is incomplete (no $need); nothing was changed." } + } + $token = [guid]::NewGuid().ToString('N').Substring(0, 8) + $asideName = '\.old-[0-9a-f]{8,32}$' + $files = @{}; $dirs = @{} + foreach ($i in @(Get-ChildItem -LiteralPath $stageDir -Recurse -Force)) { + $rel = $i.FullName.Substring($stageDir.Length) + if ($i.PSIsContainer) { $dirs[$rel] = $true } else { $files[$rel] = $true } + } + $undo = New-Object System.Collections.ArrayList + function Move-Logged([string]$From, [string]$To) { [IO.File]::Move($From, $To); [void]$undo.Add(@($From, $To)) } + function Undo-Logged { + $lost = 0 + for ($n = $undo.Count - 1; $n -ge 0; $n--) { + $u = $undo[$n] + try { if ($u.Count -eq 2) { [IO.File]::Move($u[1], $u[0]) } else { [IO.Directory]::Delete($u[0]) } } catch { if ($u.Count -eq 2) { $lost++ } } + } + $undo.Clear() + $lost + } + $done = $false; $at = $Dest + try { + if (-not (Test-Path -LiteralPath $Dest)) { [void][IO.Directory]::CreateDirectory($Dest); [void]$undo.Add(@($Dest)) } + $destDir = (Resolve-Path -LiteralPath $Dest).ProviderPath.TrimEnd('\') + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File)) { + if (-not $files.ContainsKey($f.FullName.Substring($destDir.Length)) -and $f.Name -notmatch $asideName) { + $at = $f.FullName; Move-Logged $at "$at.old-$token" + } + } + foreach ($rel in @($dirs.Keys | Sort-Object Length)) { + $at = $destDir + $rel + if (-not [IO.Directory]::Exists($at)) { [void][IO.Directory]::CreateDirectory($at); [void]$undo.Add(@($at)) } + } + foreach ($rel in @($files.Keys)) { + $at = $destDir + $rel + if ([IO.File]::Exists($at)) { Move-Logged $at "$at.old-$token" } + Move-Logged ($stageDir + $rel) $at + } + $done = $true + } catch { + $x = $_.Exception; while ($x.InnerException) { $x = $x.InnerException } + $lost = Undo-Logged + $msg = "Could not replace $at ($($x.Message))." + if ($lost) { + $msg += " $lost file(s) could not be put back, so the install may not start. Close your agent sessions and any running codegraph commands, then reinstall: irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex" + } else { + $msg += " Nothing was changed: the existing install still works. If another program has CodeGraph's files open, close your agent sessions (they run the CodeGraph MCP server) and any running codegraph commands, then try again." + } + $e = New-Object System.Exception($msg); $e.Data['codegraphDamaged'] = [bool]$lost; throw $e + } finally { + # Interrupted (Ctrl+C) without reaching catch: still put everything back. + if (-not $done) { [void](Undo-Logged) } + } + # Delete what this run and earlier ones renamed aside. A file a running + # process still holds can't be deleted yet; the next install retries it. + foreach ($f in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -File -ErrorAction SilentlyContinue)) { + if ($f.Name -match $asideName) { try { [IO.File]::Delete($f.FullName) } catch {} } + } + foreach ($d in @(Get-ChildItem -LiteralPath $destDir -Recurse -Force -Directory -ErrorAction SilentlyContinue | Sort-Object { $_.FullName.Length } -Descending)) { + if (-not $dirs.ContainsKey($d.FullName.Substring($destDir.Length))) { try { [IO.Directory]::Delete($d.FullName) } catch {} } + } +} +`; + +/** Single-quote a value for PowerShell (a `'` inside is doubled). */ +function psQuote(value: string): string { + return `'${value.replace(/'/g, "''")}'`; +} + +/** Exit code of the upgrade script when files could not all be put back. */ +export const WINDOWS_UPGRADE_DAMAGED = 2; + /** Build the in-place Windows upgrade script (exported for unit-testing). */ export function buildWindowsUpgradeScript(bundleRoot: string, version: string, arch: string): string { const target = `win32-${arch}`; const url = `https://github.com/${REPO}/releases/download/${version}/codegraph-${target}.zip`; - // Windows can't DELETE a running exe but CAN rename it, so we upgrade IN - // PLACE: download → rename the locked node.exe aside → extract the new bundle - // over current\. Synchronous, no detached helper (which dies under SSH/job - // objects and has worse UX). The running process keeps its renamed node.exe - // mapped; the NEXT `codegraph` invocation uses the new one. We can't reuse - // install.ps1 here — it `Remove-Item`s current\, which fails on the locked exe. + // Synchronous, no detached helper (which dies under SSH/job objects and has + // worse UX). The bundle is unpacked into a sibling of current\ — the same + // volume, so the swap is renames, never a half-finished copy — and nothing + // in current\ changes until it is fully unpacked. The running process keeps + // its renamed node.exe mapped; the NEXT `codegraph` invocation uses the new + // one. Exit codes: 0 installed, 1 failed with the install unchanged, + // WINDOWS_UPGRADE_DAMAGED when the rollback could not restore every file. return [ `$ErrorActionPreference='Stop'`, - `$dest='${bundleRoot}'`, - `$url='${url}'`, - `Write-Host "Downloading $url"`, + WINDOWS_SWAP_FUNCTION, + `$dest=${psQuote(bundleRoot)}`, + `$url=${psQuote(url)}`, `$tmp=Join-Path $env:TEMP ('cg-up-'+[guid]::NewGuid().ToString('N'))`, - `New-Item -ItemType Directory -Force -Path $tmp | Out-Null`, - `$zip=Join-Path $tmp 'cg.zip'`, - `Invoke-WebRequest -Uri $url -OutFile $zip`, - `$stage=Join-Path $tmp 'stage'`, - `Expand-Archive -Path $zip -DestinationPath $stage -Force`, - `$inner=Join-Path $stage 'codegraph-${target}'`, - `$src=if(Test-Path $inner){$inner}else{$stage}`, - `$node=Join-Path $dest 'node.exe'`, - `if(Test-Path $node){Rename-Item -Path $node -NewName ('node.exe.old-'+[guid]::NewGuid().ToString('N')) -Force}`, - `Copy-Item -Path (Join-Path $src '*') -Destination $dest -Recurse -Force`, - `Get-ChildItem -Path $dest -Filter 'node.exe.old-*' -ErrorAction SilentlyContinue | ForEach-Object { try { Remove-Item $_.FullName -Force -ErrorAction Stop } catch {} }`, - `Remove-Item -Recurse -Force $tmp -ErrorAction SilentlyContinue`, - `Write-Host "Installed CodeGraph ${version} to $dest"`, - ].join(';'); + `$stage=Join-Path (Split-Path -Parent $dest) ('.staging-'+[guid]::NewGuid().ToString('N').Substring(0,8))`, + `$code=0`, + `try {`, + ` Write-Host "Downloading $url"`, + ` New-Item -ItemType Directory -Force -Path $tmp | Out-Null`, + ` $zip=Join-Path $tmp 'cg.zip'`, + ` Invoke-WebRequest -Uri $url -OutFile $zip`, + ` Expand-Archive -Path $zip -DestinationPath $stage -Force`, + ` $inner=Join-Path $stage 'codegraph-${target}'`, + ` Install-CodeGraphFiles $(if(Test-Path $inner){$inner}else{$stage}) $dest`, + ` Write-Host "Installed CodeGraph ${version} to $dest"`, + `} catch {`, + ` [Console]::Error.WriteLine($_.Exception.Message)`, + ` $code=if($_.Exception.Data['codegraphDamaged']){${WINDOWS_UPGRADE_DAMAGED}}else{1}`, + `}`, + `Remove-Item -LiteralPath $stage,$tmp -Recurse -Force -ErrorAction SilentlyContinue`, + `exit $code`, + ].join('\n'); } function upgradeWindowsBundle( @@ -624,7 +730,15 @@ function upgradeWindowsBundle( deps.log(c.dim(`Downloading and installing ${latest}…`)); const code = deps.run('powershell.exe', ['-NoProfile', '-ExecutionPolicy', 'Bypass', '-EncodedCommand', encoded]); if (code !== 0) { - deps.error(`Installer exited with code ${code}.`); + // The script has already printed what went wrong and what to do. + if (code === WINDOWS_UPGRADE_DAMAGED) { + deps.error('The upgrade failed and some files could not be put back.'); + deps.log(c.dim(`Close your agent sessions and any running codegraph commands, then reinstall: irm ${INSTALL_PS1_URL} | iex`)); + } else if (code === 1) { + deps.error(`The upgrade to ${latest} did not complete; your existing install was left as it was.`); + } else { + deps.error(`Installer exited with code ${code}.`); + } return 1; } deps.log(''); From 8abebda02cbe893a393d60ecc4f231f18d194f4a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 07:07:34 +0000 Subject: [PATCH 189/363] test(windows): two #1910 tests no longer time out on Windows (#2298) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit oversize-file-not-read's drift test wrote a 1.4 MB `.js` made of one statement repeated 110k times. Windows Defender's script scan of that freshly written file takes about 54 s on its first read (the viewer's read in the test), so the test always timed out on Windows and its cleanup then hit EPERM on the still open file. A varied 2.4 MB `.js` and a real 2 MB bundle scan in 35 ms and 144 ms, so this is the fixture, not something users hit; the fixture now has a different statement on every line. The mpeg-ts git-path cases build a git repository and index it, the last one twice. At ~100 ms per git process on Windows that is 5–6 s, past vitest's 5 s default; the describe now allows 30 s. Co-authored-by: Claude Opus 5.5 --- __tests__/mpeg-ts-not-typescript.test.ts | 5 ++++- __tests__/oversize-file-not-read.test.ts | 13 +++++++++++-- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/__tests__/mpeg-ts-not-typescript.test.ts b/__tests__/mpeg-ts-not-typescript.test.ts index 5d44003d2e..dfc23a1950 100644 --- a/__tests__/mpeg-ts-not-typescript.test.ts +++ b/__tests__/mpeg-ts-not-typescript.test.ts @@ -215,7 +215,10 @@ describe('MPEG-TS video named .ts is skipped, real TypeScript is indexed (#1910) }); }); -describe('a video .ts never stays pending (#1910)', () => { +// Each case builds a git repository and indexes it, the last one twice over. On +// Windows every git process costs ~100 ms, which puts these cases at 5–6 s there, +// past vitest's 5 s default. +describe('a video .ts never stays pending (#1910)', { timeout: 30_000 }, () => { const dirs: string[] = []; afterEach(() => { for (const d of dirs.splice(0)) fs.rmSync(d, { recursive: true, force: true }); }); const gitProject = (): string => { diff --git a/__tests__/oversize-file-not-read.test.ts b/__tests__/oversize-file-not-read.test.ts index f32267f73f..fad014c375 100644 --- a/__tests__/oversize-file-not-read.test.ts +++ b/__tests__/oversize-file-not-read.test.ts @@ -89,8 +89,17 @@ describe('an unchanged file over the size limit is not reported as drifted (#191 dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-oversize-drift-')); fs.writeFileSync(path.join(dir, 'app.ts'), 'export function alpha() { return 1; }\n'); // 1.4 MB of ordinary text: over the index limit, under the viewer's 8 MB read cap. - const line = 'const x = 1;\n'; - fs.writeFileSync(path.join(dir, 'big.js'), line.repeat(Math.ceil((1.4 * 1024 * 1024) / line.length))); + // Each line differs: Windows Defender's script scan of a freshly written `.js` + // made of one statement repeated 110k times takes close to a minute on the + // first read, so the viewer's read below blew the test timeout there. Real + // code of the same size is scanned in milliseconds. + const line = 'const last = 0;\n'; + const lines: string[] = []; + for (let size = 0, i = 0; size < 1.4 * 1024 * 1024; i++) { + lines.push(`const x${i} = ${i};\n`); + size += lines[i]!.length; + } + fs.writeFileSync(path.join(dir, 'big.js'), lines.join('')); const cg = await CodeGraph.init(dir, { index: true }); try { const record = cg.getFiles().find((f) => f.path === 'big.js')!; From 91b9bba1336cb6fe33113b0df25c37ae07b078ca Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 08:48:01 +0000 Subject: [PATCH 190/363] fix(extraction): never terminate a parse worker while it is loading grammars (#2301) On a loaded Windows machine `codegraph init` / `index` sometimes died right after the parse phase with 0xC0000005 (STATUS_ACCESS_VIOLATION), no WER report and no output past the last parsed file. The parse pool's destroy() terminated every worker at once, and a short index often ends while a late-spawned worker is still compiling its grammar WASM; terminating a worker with that compile in flight crashes the process. Reproduced on a Windows 11 ARM64 VM without CodeGraph's pipeline: starting three parse workers with preloaded grammar bytes and terminating them mid-load crashed 5 of 720 child processes, terminating them only after grammars-loaded crashed 0 of 360. Under the real load (24 parallel `codegraph init` runs on 4 cores) main crashed after parsing in 8 of 1,200 runs; with this change 0 of 1,080 did. destroy() now waits for a worker that is still loading to report grammars-loaded or exit before terminating it, capped at 15 s so a wedged load can't hold teardown forever. Recycling only ever touches idle workers, and a hung parse is still killed by the hard timeout. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/parse-pool.test.ts | 79 ++++++++++++++++++++++++++++++++++++ src/extraction/parse-pool.ts | 54 ++++++++++++++++++++++-- 3 files changed, 131 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 348a804c87..f7b195d0d2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,6 +174,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) +- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/parse-pool.test.ts b/__tests__/parse-pool.test.ts index 6211481a44..ac51b984b5 100644 --- a/__tests__/parse-pool.test.ts +++ b/__tests__/parse-pool.test.ts @@ -248,3 +248,82 @@ describe('ParseWorkerPool', () => { await expect(pool.requestParse(task('y.ts'))).rejects.toThrow(/destroyed/); }); }); + +/** + * A fake whose grammar load takes `loadMs` (or never finishes, with `null`), and + * which records when it finished loading and when it was terminated. + */ +class SlowLoadWorker implements ParsePoolWorker { + private msgCb?: (m: unknown) => void; + private exitCb?: (code: number) => void; + loadedAt: number | null = null; + terminatedAt: number | null = null; + constructor(private loadMs: number | null) {} + on(event: string, cb: (...args: any[]) => void): void { + if (event === 'message') this.msgCb = cb; + else if (event === 'exit') this.exitCb = cb; + } + postMessage(msg: unknown): void { + if ((msg as { type: string }).type !== 'load-grammars' || this.loadMs === null) return; + setTimeout(() => { + if (this.terminatedAt !== null) return; + this.loadedAt = performance.now(); + this.msgCb?.({ type: 'grammars-loaded' }); + }, this.loadMs); + } + exit(code: number): void { this.exitCb?.(code); } + terminate(): Promise { this.terminatedAt = performance.now(); return Promise.resolve(0); } +} + +describe('teardown never terminates a worker mid-load', () => { + // Terminating a worker whose grammar WASM is still compiling can crash the + // process with an access violation (0xC0000005) on Windows; a pool torn down + // right after a short index often has a late-spawned worker in that state. + const poolOf = (worker: SlowLoadWorker, loadSettleMs?: number) => + new ParseWorkerPool({ languages: ['typescript'] as Language[], size: 1, createWorker: () => worker, loadSettleMs }); + + it('waits for a loading worker to finish before terminating it', async () => { + const w = new SlowLoadWorker(120); + const pool = poolOf(w); + await pool.destroy(); + expect(w.loadedAt).not.toBeNull(); + expect(w.terminatedAt).not.toBeNull(); + expect(w.terminatedAt!).toBeGreaterThanOrEqual(w.loadedAt!); + }); + + it('terminates a worker that is already loaded right away', async () => { + const w = new SlowLoadWorker(0); + const pool = poolOf(w); + await sleep(20); + expect(w.loadedAt).not.toBeNull(); + const t0 = performance.now(); + await pool.destroy(); + expect(w.terminatedAt! - t0).toBeLessThan(50); + }); + + it('does not wait past the cap on a load that never finishes', async () => { + const w = new SlowLoadWorker(null); + const pool = poolOf(w, 80); + const t0 = performance.now(); + await pool.destroy(); + expect(w.terminatedAt).not.toBeNull(); + expect(w.terminatedAt! - t0).toBeGreaterThanOrEqual(70); + expect(w.terminatedAt! - t0).toBeLessThan(1000); + }); + + it('does not wait on a worker that died while loading', async () => { + const w = new SlowLoadWorker(null); + // The pool respawns a crashed worker; the replacement loads normally. The + // default 15s cap stays, so waiting on the dead one would time the test out. + const workers = [w]; + const pool = new ParseWorkerPool({ + languages: ['typescript'] as Language[], size: 1, + createWorker: () => workers.shift() ?? new SlowLoadWorker(0), + }); + w.exit(1); + await sleep(20); + const t0 = performance.now(); + await pool.destroy(); + expect(performance.now() - t0).toBeLessThan(200); + }); +}); diff --git a/src/extraction/parse-pool.ts b/src/extraction/parse-pool.ts index 9b2b100e4e..62967a3dfb 100644 --- a/src/extraction/parse-pool.ts +++ b/src/extraction/parse-pool.ts @@ -75,6 +75,16 @@ const MAX_SCALED_PARSE_TIMEOUT_MS = 20_000; * only a worker that stays silent the whole window is treated as hung. */ const HARD_KILL_MULTIPLIER = 3; +/** + * How long `destroy()` waits for a worker that is still loading its grammars + * before terminating it anyway. Terminating a worker whose WebAssembly + * compiles are still in flight can take the whole process down with an access + * violation (0xC0000005, seen on Windows): a pool torn down right after a short + * index often has a late-spawned worker in exactly that state. A load finishes + * in well under a second normally and a few seconds under heavy load; the cap + * only bounds a load that is wedged. + */ +const GRAMMAR_LOAD_SETTLE_MS = 15_000; /** * Max workers cold-starting at once. A worker's cold start is heavy (module load * + grammar WASM compile); starting the whole pool simultaneously thrashes CPU. @@ -169,6 +179,8 @@ export interface ParseWorkerPoolOptions { parseTimeoutMs?: number; /** Worker factory (tests inject a fake). Defaults to a real `worker_threads` Worker. */ createWorker?: () => ParsePoolWorker; + /** How long destroy() waits on a worker still loading grammars (tests shorten it). Default 15s. */ + loadSettleMs?: number; /** Optional verbose logger (the orchestrator's `[worker] …` logger). */ log?: (msg: string) => void; /** @@ -190,6 +202,10 @@ export class ParseWorkerPool { // Spawned but not yet 'grammars-loaded'. Growth counts these so a single first // parse doesn't spawn the whole pool before the eager worker reports ready. private pending = new Set(); + // Each worker's grammar load, settled when it reports 'grammars-loaded' or + // goes away — what destroy() waits on before terminating it (see + // GRAMMAR_LOAD_SETTLE_MS). + private loads = new Map; settle: () => void }>(); private parseCounts = new Map(); private nextId = 1; private totalCrashes = 0; @@ -200,6 +216,7 @@ export class ParseWorkerPool { private readonly recycleInterval: number; private readonly parseTimeoutMs: number; private readonly createWorker: () => ParsePoolWorker; + private readonly loadSettleMs: number; private readonly log: (msg: string) => void; private readonly grammarBuffers?: Record; @@ -209,6 +226,7 @@ export class ParseWorkerPool { this.maxSize = Math.max(1, Math.min(opts.size, MAX_PARSE_POOL_SIZE)); this.recycleInterval = opts.recycleInterval ?? DEFAULT_RECYCLE_INTERVAL; this.parseTimeoutMs = opts.parseTimeoutMs ?? DEFAULT_PARSE_TIMEOUT_MS; + this.loadSettleMs = opts.loadSettleMs ?? GRAMMAR_LOAD_SETTLE_MS; this.log = opts.log ?? (() => {}); if (opts.createWorker) { this.createWorker = opts.createWorker; @@ -278,9 +296,12 @@ export class ParseWorkerPool { this.workers.add(w); this.pending.add(w); this.parseCounts.set(w, 0); + let settle!: () => void; + const settled = new Promise((resolve) => { settle = resolve; }); + this.loads.set(w, { settled, settle }); w.on('message', (m) => this.onMessage(w, (m ?? {}) as ParseWorkerMessage)); - w.on('error', (e) => this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`)); - w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); }); + w.on('error', (e) => { this.loadSettled(w); this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`); }); + w.on('exit', (code) => { this.loadSettled(w); if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); }); // Load grammars; the worker replies 'grammars-loaded' and only then is idle. // Pre-read WASM bytes (when the orchestrator provided them) make this a // memory load instead of a per-spawn disk read. @@ -289,6 +310,7 @@ export class ParseWorkerPool { private onMessage(w: ParsePoolWorker, m: ParseWorkerMessage): void { if (m.type === 'grammars-loaded') { + this.loadSettled(w); if (!this.workers.has(w)) return; // recycled/destroyed before ready this.pending.delete(w); this.idle.push(w); @@ -351,6 +373,32 @@ export class ParseWorkerPool { if (this.healthy && !this.destroyed) this.spawnOne(); } + /** The worker's grammar load is over: it reported ready, errored or exited. */ + private loadSettled(w: ParsePoolWorker): void { + const load = this.loads.get(w); + if (!load) return; + this.loads.delete(w); + load.settle(); + } + + /** + * Terminate a worker, but not while it is still loading grammars: wait for + * the load to finish (or the worker to go away), up to `loadSettleMs` + * (GRAMMAR_LOAD_SETTLE_MS — see that constant for the crash this avoids). + */ + private async terminateSettled(w: ParsePoolWorker): Promise { + const load = this.loads.get(w); + if (load) { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + load.settled, + new Promise((resolve) => { cap = setTimeout(resolve, this.loadSettleMs); cap.unref?.(); }), + ]); + clearTimeout(cap); + } + try { await w.terminate(); } catch { /* already gone */ } + } + private removeWorker(w: ParsePoolWorker): void { this.workers.delete(w); this.pending.delete(w); @@ -471,6 +519,6 @@ export class ParseWorkerPool { } this.inflight.clear(); this.queue = []; - await Promise.all(ws.map((w) => Promise.resolve(w.terminate()).catch(() => { /* already gone */ }))); + await Promise.all(ws.map((w) => this.terminateSettled(w))); } } From e0f9c95e2f7e5b8752b5b1bbd1362842c44b1ff3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 10:20:23 +0000 Subject: [PATCH 191/363] fix(resolution): a busy machine doesn't boot the resolver pool for a small project (#2303) The resolve loop boots the worker pool mid-run when the remaining refs' projected sequential settle (observed per-ref rate x refs left) clears 400 ms. Load inflates the observed rate, so on a loaded 4-core Windows VM a ten-file project (19 refs) projected 506 ms for its last 11 and booted three workers in 122 of 240 runs under 24 parallel inits (0 of 80 at 8). The pool only fans out batches of 1,000 refs, so it resolved nothing; synthesis then waited out its boot (dedupe-merge 6.3 s vs a 6.5 s boot) and its three extra database connections pushed the VM's free commit to 11 MB. Adaptive engagement now also needs 20,000 refs left. tokio-class Rust repos (~55k refs at ~36 us) still engage; a smaller remainder never repays a seconds-long boot. With the floor, the same load booted the pool in 0 of 240 runs. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/resolver-pool-sizing.test.ts | 16 +++++++++++++++- src/resolution/index.ts | 4 ++-- src/resolution/resolver-pool.ts | 17 +++++++++++++++++ 4 files changed, 35 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f7b195d0d2..0669709589 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -175,6 +175,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) - On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. +- On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/resolver-pool-sizing.test.ts b/__tests__/resolver-pool-sizing.test.ts index cd949a4f0e..e70b7a6911 100644 --- a/__tests__/resolver-pool-sizing.test.ts +++ b/__tests__/resolver-pool-sizing.test.ts @@ -8,7 +8,7 @@ */ import { describe, it, expect } from 'vitest'; import * as os from 'os'; -import { ResolverPool } from '../src/resolution/resolver-pool'; +import { ResolverPool, ADAPTIVE_ENGAGE_MIN_REFS, shouldEngageAdaptively } from '../src/resolution/resolver-pool'; import { cgroupMemoryAvailable, darwinMemoryAvailable, @@ -106,3 +106,17 @@ describe('memory budget helpers', () => { } ); }); + +describe('adaptive pool engagement needs a real amount of work left', () => { + it('does not boot the pool for a small remainder, however slow the observed rate', () => { + // A busy machine: 11 refs left projected at 506 ms (seen on a loaded 4-core VM). + expect(shouldEngageAdaptively(506, 11, 400)).toBe(false); + expect(shouldEngageAdaptively(60_000, ADAPTIVE_ENGAGE_MIN_REFS - 1, 400)).toBe(false); + }); + + it('boots it for a large remainder whose projected settle clears the bar', () => { + // tokio-class: ~55k Rust refs at ~36 µs each. + expect(shouldEngageAdaptively(1_980, 55_000, 400)).toBe(true); + expect(shouldEngageAdaptively(399, 55_000, 400)).toBe(false); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index f4bb91a237..1eaf14b145 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -27,7 +27,7 @@ import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; -import { ResolverPool, minRefsForPool } from './resolver-pool'; +import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; import { resolveAliasBinding } from './alias-binding'; import { detectFrameworks } from './frameworks'; import { synthesizeCallbackEdges } from './callback-synthesizer'; @@ -2109,7 +2109,7 @@ export class ReferenceResolver { adaptiveSeqRefs += batch.length; const remaining = total - processed - batch.length; const projectedMs = (adaptiveSeqMs / Math.max(1, adaptiveSeqRefs)) * Math.max(0, remaining); - if (projectedMs >= ADAPTIVE_ENGAGE_SETTLE_MS) { + if (shouldEngageAdaptively(projectedMs, remaining, ADAPTIVE_ENGAGE_SETTLE_MS)) { if (process.env.CODEGRAPH_SYNTH_TIMINGS) { console.error(`[pool-timing] adaptive engage: projected ${Math.round(projectedMs)}ms sequential settle over ${remaining} remaining refs`); } diff --git a/src/resolution/resolver-pool.ts b/src/resolution/resolver-pool.ts index 56b9785a66..96f4340d8d 100644 --- a/src/resolution/resolver-pool.ts +++ b/src/resolution/resolver-pool.ts @@ -58,6 +58,23 @@ export function minRefsForPool(): number { return 150_000; } +/** + * Fewest refs still to settle for the batch loop to boot the pool mid-run + * from its observed per-ref rate. Booting takes seconds (each worker opens the + * database and warms a resolver), only batches of MIN_PARALLEL_BATCH refs fan + * out, and synthesis waits for a booting pool. A rate measured on a busy + * machine inflates every projection, so with no floor a ten-file project booted + * the pool in half its runs under heavy load and paid seconds for nothing. + * Repos the adaptive bar is for (Rust-class per-ref cost) have tens of + * thousands of refs and still engage. + */ +export const ADAPTIVE_ENGAGE_MIN_REFS = 20_000; + +/** Whether the resolve loop should boot the pool mid-run (see ADAPTIVE_ENGAGE_MIN_REFS). */ +export function shouldEngageAdaptively(projectedMs: number, remainingRefs: number, barMs: number): boolean { + return remainingRefs >= ADAPTIVE_ENGAGE_MIN_REFS && projectedMs >= barMs; +} + export class ResolverPool { private workers: PoolWorker[] = []; private nextId = 0; From 9d4e805bec36d62332d7fba0a735329ba7621777 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 10:36:42 +0000 Subject: [PATCH 192/363] test(windows): daemon-pid-reuse cleanup retries a still-held temp dir (#2304) afterEach SIGKILLs the servers and the detached daemon, waits 50 ms and removes the temp dir. A killed process releases its handles asynchronously on Windows, and under full-suite load the removal hit EPERM on the VM (the test itself passed; it passes 3/3 alone). Retry the removal like the other process-spawning suites do. Co-authored-by: Claude Opus 5.5 --- __tests__/daemon-pid-reuse.test.ts | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/__tests__/daemon-pid-reuse.test.ts b/__tests__/daemon-pid-reuse.test.ts index 48650ccdf5..8bb116a0ce 100644 --- a/__tests__/daemon-pid-reuse.test.ts +++ b/__tests__/daemon-pid-reuse.test.ts @@ -197,7 +197,9 @@ describe('Shared MCP daemon (issue #411)', () => { } await new Promise((r) => setTimeout(r, 50)); servers.length = 0; - fs.rmSync(tempDir, { recursive: true, force: true }); + // A killed process releases its handles asynchronously on Windows; under + // full-suite load 50 ms is not always enough, so retry EPERM/EBUSY. + fs.rmSync(tempDir, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); }); it('takes over after SIGKILL even when the stale PID has been reused (#1553)', async () => { From 3d86bcc0371079070d25901830fb2102100434c3 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 18:03:28 +0000 Subject: [PATCH 193/363] fix(windows): the query pool never terminates a worker still starting up (#2307) Terminating a worker thread while it is still loading its modules can crash the whole process on Windows (exit 3221225477 / 0xC0000005, no error, no dump). QueryPool.destroy() terminated every worker at once, and the daemon destroys its pool whenever it stops - a session that ends within a few seconds of starting had its eager worker in exactly that state. MCPEngine.stop() didn't even wait for the pool before the daemon called process.exit(). - QueryPool tracks each worker's start (settled on 'ready', error or exit); destroy() answers outstanding calls first, then terminates each worker once it has started, capped at 15 s - the parse pool's #2301 mechanism. - MCPEngine.stop() awaits the pool. - #2301's explanation is corrected: on the Windows VM, terminating workers mid grammar compile never crashed (0 in 2,161), but terminating them while they load CodeGraph's extraction modules did. The parse pool's wait already covers that window, so only its comment and changelog wording change. Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 3 +- __tests__/query-pool.test.ts | 101 +++++++++++++++++++++++++++++++++++ src/extraction/parse-pool.ts | 14 ++--- src/mcp/engine.ts | 10 ++-- src/mcp/query-pool.ts | 62 +++++++++++++++++++-- 5 files changed, 173 insertions(+), 17 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0669709589..86258f27b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -174,8 +174,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) - An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) - On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) -- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still loading its language grammars, which took the whole process down; they now finish loading first. +- On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still starting up, which took the whole process down; they now finish starting first. - On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. +- On Windows, the CodeGraph background server no longer sometimes crashes as it shuts down (exit code 3221225477, an access violation) when it stops shortly after starting. Shutting down could stop one of its background threads while that thread was still starting up; it now lets the thread finish starting first. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/query-pool.test.ts b/__tests__/query-pool.test.ts index 2087e6af47..7c28f48f04 100644 --- a/__tests__/query-pool.test.ts +++ b/__tests__/query-pool.test.ts @@ -182,6 +182,63 @@ describe('QueryPool', () => { expect(pool.healthy).toBe(false); }); + describe('destroy never terminates a worker still starting up', () => { + // Terminating a worker while it is still loading its modules can crash the + // whole process on Windows (0xC0000005), and the daemon destroys its pool + // whenever it stops. FakeWorker(…, null) never posts 'ready' by itself. + it('answers callers at once, then waits for the worker to start', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ + root: '/x', size: 1, softTimeoutMs: 10_000, + createWorker: () => (worker = new FakeWorker(() => ({ hang: true }), null)), + }); + const pending = pool.run('codegraph_explore', { query: 'q' }); + let destroyed = false; + const down = pool.destroy().then(() => { destroyed = true; }); + expect((await pending).isError).toBe(true); // not held behind the start + await sleep(50); + expect(destroyed).toBe(false); + expect(worker.alive).toBe(true); + worker.emitMessage({ type: 'ready', ok: true }); + await down; + expect(worker.alive).toBe(false); + }); + + it('terminates a started worker at once', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ root: '/x', size: 1, startSettleMs: 10_000, createWorker: () => (worker = new FakeWorker(() => ({ result: ok('r') }))) }); + await sleep(5); + const started = Date.now(); + await pool.destroy(); + expect(worker.alive).toBe(false); + expect(Date.now() - started).toBeLessThan(1_000); + }); + + it('gives up waiting on a start that never finishes', async () => { + let worker!: FakeWorker; + const pool = new QueryPool({ root: '/x', size: 1, startSettleMs: 60, createWorker: () => (worker = new FakeWorker(() => ({ hang: true }), null)) }); + const started = Date.now(); + await pool.destroy(); + expect(worker.alive).toBe(false); + expect(Date.now() - started).toBeGreaterThanOrEqual(55); + }); + + it('does not wait on a worker that died while starting', async () => { + const workers: FakeWorker[] = []; + const pool = new QueryPool({ + root: '/x', size: 1, startSettleMs: 10_000, + // The first never starts; its replacement does. + createWorker: () => { const w = new FakeWorker(() => ({ result: ok('r') }), workers.length ? true : null); workers.push(w); return w; }, + }); + workers[0].emitExit(); + await sleep(5); + const started = Date.now(); + await pool.destroy(); + expect(Date.now() - started).toBeLessThan(1_000); + expect(workers.every((w) => !w.alive)).toBe(true); + }); + }); + it('is not `ready` until a worker completes its cold start (#662 first-call stall)', async () => { // A worker cold start is seconds (tens under load); a call queued behind it // waits for the 45s busy backstop with nothing served. The ToolHandler must @@ -408,6 +465,50 @@ describe('MCP query pool with real projects (#1465)', () => { } }, 30000); + it('never terminates a real worker before it has started', async () => { + const alpha = await indexProject('alpha', 'alphaSymbol'); + const startedWhenEnded: boolean[] = []; + pool = new QueryPool({ + root: alpha, size: 1, + createWorker: () => { + const worker = new Worker(path.resolve(__dirname, '../dist/mcp/query-worker.js'), { workerData: { root: alpha } }); + let started = false; + worker.on('message', (m: { type?: string }) => { if (m?.type === 'ready') started = true; }); + const terminate = worker.terminate.bind(worker); + worker.terminate = () => { + startedWhenEnded.push(started); + return terminate(); + }; + return worker; + }, + }); + await pool.destroy(); // its eager worker is still loading + expect(startedWhenEnded).toEqual([true]); + }, 30000); + + it('engine stop waits for the query pool to shut down', async () => { + const alpha = await indexProject('alpha', 'alphaSymbol'); + const activeEngine = await start(alpha); + expect(pool).not.toBeNull(); + const realPool = pool!; + let release!: () => void; + const destroy = vi.spyOn(realPool, 'destroy').mockReturnValue(new Promise((r) => { release = r; })); + try { + let stopped = false; + const stopping = activeEngine.stop().then(() => { stopped = true; }); + // Everything else stop() closes is done well inside this; only the pool is held. + await Promise.race([stopping, sleep(1_500)]); + expect(stopped).toBe(false); + release(); + await stopping; + expect(stopped).toBe(true); + } finally { + release(); + destroy.mockRestore(); + await realPool.destroy(); + } + }, 30000); + it('honors size=0 for projectPath-only sessions', async () => { const alpha = await indexProject('alpha', 'alphaSymbol'); const workspace = path.join(tempDir, 'workspace'); diff --git a/src/extraction/parse-pool.ts b/src/extraction/parse-pool.ts index 62967a3dfb..b406a0916d 100644 --- a/src/extraction/parse-pool.ts +++ b/src/extraction/parse-pool.ts @@ -76,13 +76,13 @@ const MAX_SCALED_PARSE_TIMEOUT_MS = 20_000; */ const HARD_KILL_MULTIPLIER = 3; /** - * How long `destroy()` waits for a worker that is still loading its grammars - * before terminating it anyway. Terminating a worker whose WebAssembly - * compiles are still in flight can take the whole process down with an access - * violation (0xC0000005, seen on Windows): a pool torn down right after a short - * index often has a late-spawned worker in exactly that state. A load finishes - * in well under a second normally and a few seconds under heavy load; the cap - * only bounds a load that is wedged. + * How long `destroy()` waits for a worker that is still starting up — loading + * its modules, then its grammars — before terminating it anyway. Terminating a + * worker while it is still loading its modules can take the whole process down + * with an access violation (0xC0000005, seen on Windows): a pool torn down + * right after a short index often has a late-spawned worker in exactly that + * state. A start finishes in well under a second normally and a few seconds + * under heavy load; the cap only bounds a start that is wedged. */ const GRAMMAR_LOAD_SETTLE_MS = 15_000; /** diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 5409c36d48..62a4c8041c 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -271,13 +271,13 @@ export class MCPEngine { // Detach + terminate the worker pool first so no tool call routes to a // worker mid-teardown; outstanding pool calls resolve with graceful guidance. + // Stopping waits for the workers to end: the daemon exits right after, and + // exiting while a worker is still starting up can crash the process. this.toolHandler.setQueryPool(null); - if (this.queryPool) { - void this.queryPool.destroy(); - this.queryPool = null; - } + const poolDown = this.queryPool ? this.queryPool.destroy() : Promise.resolve(); + this.queryPool = null; const drained = this.toolHandler.closeAll(); - this.stopPromise = drained.then(async () => { + this.stopPromise = Promise.all([drained, poolDown]).then(async () => { if (this.initPromise) await this.initPromise; if (this.defaultLease) { await this.defaultLease.release(); diff --git a/src/mcp/query-pool.ts b/src/mcp/query-pool.ts index b3a0f5322e..6b1f473991 100644 --- a/src/mcp/query-pool.ts +++ b/src/mcp/query-pool.ts @@ -69,6 +69,17 @@ const CRASH_BUDGET = 12; */ const MAX_CONCURRENT_SPAWN = 2; +/** + * How long `destroy()` waits for a worker that is still starting up before + * terminating it anyway. Terminating a worker while it is still loading its + * modules can take the whole process down with an access violation + * (0xC0000005, seen on Windows), and the daemon shuts its pool down whenever + * it stops — a session that ends within a few seconds of starting has a + * worker in exactly that state. A start takes about a second normally and a + * few under heavy load; the cap only bounds a start that is wedged. + */ +const WORKER_START_SETTLE_MS = 15_000; + /** Shape of a message a worker posts back (ready handshake or a tool result). */ interface WorkerMessage { type?: string; @@ -99,6 +110,8 @@ export interface QueryPoolOptions { maxRetries?: number; /** Worker factory (tests inject a fake). Defaults to a real `worker_threads` Worker. */ createWorker?: () => PoolWorker; + /** How long destroy() waits on a worker still starting up (tests shorten it). Default 15s. */ + startSettleMs?: number; } /** @@ -150,6 +163,10 @@ export class QueryPool { // a large DB open) saturate the box and starve the main loop. Grow only when // the queue outstrips idle + pending. private pendingWorkers = new Set(); + // Each worker's start, settled when it posts 'ready' (either way) or goes + // away — what destroy() waits on before terminating it (see + // WORKER_START_SETTLE_MS). + private starts = new Map; settle: () => void }>(); private nextId = 1; private totalCrashes = 0; private destroyed = false; @@ -158,6 +175,7 @@ export class QueryPool { private readonly softTimeoutMs: number; private readonly maxRetries: number; private readonly createWorker: () => PoolWorker; + private readonly startSettleMs: number; constructor(opts: QueryPoolOptions) { this.root = opts.root; @@ -165,6 +183,7 @@ export class QueryPool { this.softTimeoutMs = opts.softTimeoutMs ?? resolveBusyTimeoutMs(); this.maxRetries = opts.maxRetries ?? 1; this.createWorker = opts.createWorker ?? (() => new Worker(WORKER_FILE, { workerData: { root: this.root } })); + this.startSettleMs = opts.startSettleMs ?? WORKER_START_SETTLE_MS; this.spawnOne(); // one eager warm worker, ready for the first call } @@ -211,12 +230,18 @@ export class QueryPool { } this.workers.add(w); this.pendingWorkers.add(w); + let settle!: () => void; + const settled = new Promise((resolve) => { settle = resolve; }); + this.starts.set(w, { settled, settle }); w.on('message', (m) => this.onMessage(w, (m ?? {}) as WorkerMessage)); - w.on('error', () => this.onWorkerGone(w)); - w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w); }); + w.on('error', () => { this.startSettled(w); this.onWorkerGone(w); }); + w.on('exit', (code) => { this.startSettled(w); if (code !== 0) this.onWorkerGone(w); }); } private onMessage(w: PoolWorker, m: WorkerMessage): void { + // Before the retired-worker check: destroy() waits on this for a worker + // it has already let go of. + if (m?.type === 'ready') this.startSettled(w); if (!m || !this.workers.has(w)) return; // ignore late messages from retired workers if (m.type === 'ready') { if (!this.pendingWorkers.delete(w)) return; // already handled this handshake @@ -311,7 +336,36 @@ export class QueryPool { }); } - /** Terminate all workers and answer any outstanding calls gracefully. */ + private startSettled(w: PoolWorker): void { + const start = this.starts.get(w); + if (!start) return; + this.starts.delete(w); + start.settle(); + } + + /** + * Terminate a worker, but not while it is still starting up: wait for it to + * post 'ready' (or go away), up to `startSettleMs` (WORKER_START_SETTLE_MS — + * see that constant for the crash this avoids). + */ + private async terminateStarted(w: PoolWorker): Promise { + const start = this.starts.get(w); + if (start) { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + start.settled, + new Promise((resolve) => { cap = setTimeout(resolve, this.startSettleMs); cap.unref?.(); }), + ]); + clearTimeout(cap); + } + try { await w.terminate(); } catch { /* already gone */ } + } + + /** + * Terminate all workers and answer any outstanding calls gracefully. A worker + * still starting up is terminated once it has started (see + * WORKER_START_SETTLE_MS); outstanding calls are answered first. + */ async destroy(): Promise { if (this.destroyed) return; this.destroyed = true; @@ -324,6 +378,6 @@ export class QueryPool { } this.inflight.clear(); this.queue = []; - await Promise.all(ws.map((w) => Promise.resolve(w.terminate()).catch(() => { /* already gone */ }))); + await Promise.all(ws.map((w) => this.terminateStarted(w))); } } From 70fe0ecc52751c5a3f8a0912388411bd61e5de13 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Fri, 2 Oct 2026 21:58:28 +0000 Subject: [PATCH 194/363] fix(windows): helper threads never end mid-start or with GC marking in flight (#2309) On Windows a worker thread that ends - terminate(), its own process.exit(), or the process exiting - while V8's concurrent marker is marking its heap crashes the whole process (exit 3221225477 / 0xC0000005, no error, no dump). Bisected on the Windows 11 ARM64 VM: --no-concurrent-marking alone stops it (18/480 -> 0/480; no other concurrency flag does), but it costs about a third of indexing speed, so CodeGraph avoids the dangerous moments instead: - src/worker-teardown.ts: owners terminate a worker only once it has started (its first message, posted after its modules load - mid-load is by far the worst moment), and a worker about to exit runs a full collection first, which finishes any marking in flight (gc reached at runtime, never global). - Resolver pool / store writer: the worker collects before exiting on 'close' (the normal path - main 21/480 crashed on the VM, now 2/1200), and the 5 s close fallback never terminates a worker still starting. - codegraph_status change counts: the worker posts 'loaded' after its heavy require and collects before answering; a deadline answers unknown at once but terminates only once loaded (3/720 -> 0/1440), and MCPEngine.stop() waits for measurements in flight. Co-authored-by: Claude Opus 5.5 --- AGENTS.md | 1 + CHANGELOG.md | 1 + __tests__/mcp-freshness-bounds.test.ts | 39 +++++++- __tests__/mcp-status-freshness.test.ts | 29 +++++- __tests__/worker-teardown.test.ts | 131 +++++++++++++++++++++++++ src/extraction/store-worker.ts | 3 + src/extraction/store-writer.ts | 16 ++- src/mcp/engine.ts | 10 +- src/mcp/index-freshness-worker.ts | 8 +- src/mcp/index-freshness.ts | 38 +++++-- src/resolution/resolver-pool.ts | 21 +++- src/resolution/resolver-worker.ts | 3 + src/worker-teardown.ts | 100 +++++++++++++++++++ 13 files changed, 376 insertions(+), 24 deletions(-) create mode 100644 __tests__/worker-teardown.test.ts create mode 100644 src/worker-teardown.ts diff --git a/AGENTS.md b/AGENTS.md index 0f5e9a882e..fd5daee8ef 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -214,6 +214,7 @@ For any Windows-specific PR, bug, or implementation, validate it on the real Win - Guest toolchain (winget): Node LTS, Git, and the **VC++ ARM64 redistributable** (required by `@rollup/rollup-win32-arm64-msvc`, which vitest pulls in). - Fetch a contributor PR head straight from their fork to dodge `pull//head` lag: `git fetch ` then `git checkout -f FETCH_HEAD`. - Windows baseline: as of #2053 the full suite passes on the Windows 11 (ARM64) VM. The only expected exception is `security.test.ts > Session marker symlink resistance > does not follow a pre-planted symlink`, which needs symlink privileges (Developer Mode) — confirm any other failure against `origin/main` before blaming your PR. The former `mcp-initialize.test.ts` / `mcp-roots.test.ts` `EPERM` teardown failures came from tests spawning `serve --mcp` without the runtime flags (its `--liftoff-only` re-exec grandchild kept the cwd / SQLite file open); spawn it with `WASM_RUNTIME_FLAGS` and await the child's exit before removing the temp dir. Windows checkouts may be CRLF — split source lines on `/\r?\n/` in tests. +- Windows worker crashes: a worker thread that ends (`terminate()`, its own `process.exit()`, or the process exiting) while V8's concurrent marker is marking its heap kills the process with exit 3221225477 (0xC0000005) and leaves no error or dump. It's worst while the worker is still loading its modules. Owners must not terminate a worker before its first message (`workerStarted` / `terminateOnceStarted`, or the pool's own handshake), and a worker that exits itself calls `collectBeforeExit()` first — both in `src/worker-teardown.ts`. `--no-concurrent-marking` also stops it, but measured about a third slower indexing on the Windows VM, so it is not used. ## Releases diff --git a/CHANGELOG.md b/CHANGELOG.md index 86258f27b6..9b76be6319 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -177,6 +177,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - On Windows, `codegraph init`, `codegraph index` and the background server's syncs no longer sometimes crash right after parsing on a busy machine (exit code 3221225477, an access violation). Shutting down the parsing threads could stop one while it was still starting up, which took the whole process down; they now finish starting first. - On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. - On Windows, the CodeGraph background server no longer sometimes crashes as it shuts down (exit code 3221225477, an access violation) when it stops shortly after starting. Shutting down could stop one of its background threads while that thread was still starting up; it now lets the thread finish starting first. +- On Windows, indexing, syncing and the CodeGraph background server no longer sometimes crash (exit code 3221225477, an access violation) when they stop one of their helper threads: the ones that resolve references, write the index, and count changes for `codegraph_status`. A thread is no longer stopped while it is still starting up, and each one finishes its memory cleanup before it exits. ## [1.6.1] - 2026-09-29 diff --git a/__tests__/mcp-freshness-bounds.test.ts b/__tests__/mcp-freshness-bounds.test.ts index 5710b4e137..a9dae57aff 100644 --- a/__tests__/mcp-freshness-bounds.test.ts +++ b/__tests__/mcp-freshness-bounds.test.ts @@ -5,7 +5,7 @@ import * as path from 'path'; import { createHash } from 'crypto'; import { Readable } from 'stream'; import { validateAnswerFiles } from '../src/mcp/answer-freshness'; -import { measurePendingChanges } from '../src/mcp/index-freshness'; +import { endFreshnessMeasurements, measurePendingChanges } from '../src/mcp/index-freshness'; vi.mock('fs', async importOriginal => { const actual = await importOriginal(); @@ -46,16 +46,47 @@ describe('bounded freshness validation (#1959)', () => { const pending = measurePendingChanges('/freshness-timeout'); expect(measurePendingChanges('/freshness-timeout')).toBe(pending); expect(workers).toHaveLength(1); + workers[0].emit('message', { type: 'loaded' }); // stuck past loading, in the scan await vi.advanceTimersByTimeAsync(8000); expect(await pending).toBeNull(); expect(workers[0].terminate).toHaveBeenCalledOnce(); }); + it('answers unknown on deadline but never terminates a worker still loading its modules', async () => { + // Terminating a worker while it loads its modules can crash the process + // on Windows (0xC0000005) — see worker-start.ts. + vi.useFakeTimers(); + const pending = measurePendingChanges('/freshness-still-loading'); + await vi.advanceTimersByTimeAsync(8000); + expect(await pending).toBeNull(); + await vi.advanceTimersByTimeAsync(1000); + expect(workers[0].terminate).not.toHaveBeenCalled(); + workers[0].emit('message', { type: 'loaded' }); + await vi.advanceTimersByTimeAsync(0); + expect(workers[0].terminate).toHaveBeenCalledOnce(); + }); + + it('a server shutting down ends measurements once they have loaded', async () => { + vi.useFakeTimers(); + const pending = measurePendingChanges('/freshness-shutdown'); + let ended = false; + const ending = endFreshnessMeasurements().then(() => { ended = true; }); + await vi.advanceTimersByTimeAsync(1000); + expect(ended).toBe(false); + expect(workers[0].terminate).not.toHaveBeenCalled(); + workers[0].emit('message', { type: 'loaded' }); + await ending; + expect(workers[0].terminate).toHaveBeenCalled(); + workers[0].emit('exit', 1); // what a terminated worker does + expect(await pending).toBeNull(); + }); + it('returns unknown on deadline even if worker termination is delayed', async () => { vi.useFakeTimers(); const pending = measurePendingChanges('/freshness-delayed-exit'); let release!: () => void; workers[0].terminate.mockImplementation(() => new Promise(resolve => { release = resolve; })); + workers[0].emit('message', { type: 'loaded' }); await vi.advanceTimersByTimeAsync(8000); expect(await pending).toBeNull(); release(); @@ -81,11 +112,13 @@ describe('bounded freshness validation (#1959)', () => { expect(await measurePendingChanges('/freshness-three')).toBeNull(); expect(workers).toHaveLength(2); workers[0].emit('error', new Error('worker failed')); - workers[1].emit('message', { added: -1, modified: 0, removed: 0 }); + workers[1].emit('message', { type: 'counts', counts: { added: -1, modified: 0, removed: 0 } }); expect(await first).toBeNull(); expect(await second).toBeNull(); + await new Promise((r) => setTimeout(r, 0)); // the slots free once each worker is terminated const retry = measurePendingChanges('/freshness-three'); - workers[2].emit('message', { added: 0, modified: 1, removed: 0 }); + workers[2].emit('message', { type: 'loaded' }); + workers[2].emit('message', { type: 'counts', counts: { added: 0, modified: 1, removed: 0 } }); expect(await retry).toEqual({ added: 0, modified: 1, removed: 0 }); }); }); diff --git a/__tests__/mcp-status-freshness.test.ts b/__tests__/mcp-status-freshness.test.ts index 923f77ae3f..c78844db53 100644 --- a/__tests__/mcp-status-freshness.test.ts +++ b/__tests__/mcp-status-freshness.test.ts @@ -1,8 +1,9 @@ -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import * as fs from 'fs'; import * as os from 'os'; import * as path from 'path'; import { execFileSync } from 'child_process'; +import { Worker } from 'worker_threads'; import CodeGraph from '../src/index'; import { measurePendingChanges } from '../src/mcp/index-freshness'; import { ToolHandler } from '../src/mcp/tools'; @@ -52,6 +53,32 @@ describe('MCP status freshness (#1959)', () => { expect(await measurePendingChanges(path.join(root, 'missing'))).toBeNull(); }); + it('past its deadline, never terminates the worker before it has loaded its modules', async () => { + // Terminating a worker while it loads its modules can crash the whole + // process on Windows (0xC0000005). A 1 ms deadline passes while the real + // worker is still loading: the answer is unknown at once, and the worker + // is terminated only after it has posted 'loaded'. + const posted = new WeakSet(); + const loadedWhenEnded: boolean[] = []; + const emit = Worker.prototype.emit; + vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) { + if (event === 'message') posted.add(this); + return emit.call(this, event, ...args); + }); + const terminate = Worker.prototype.terminate; + vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) { + loadedWhenEnded.push(posted.has(this)); + return terminate.call(this); + }); + try { + expect(await measurePendingChanges(root, 1)).toBeNull(); + await vi.waitFor(() => expect(loadedWhenEnded).toHaveLength(1), { timeout: 20_000 }); + expect(loadedWhenEnded).toEqual([true]); + } finally { + vi.restoreAllMocks(); + } + }, 30_000); + it('counts edits committed after the index even when the working tree is clean', async () => { fs.writeFileSync(path.join(root, 'modify.ts'), 'export const modify = 99;\n'); execFileSync('git', ['add', 'modify.ts'], { cwd: root, stdio: 'pipe' }); diff --git a/__tests__/worker-teardown.test.ts b/__tests__/worker-teardown.test.ts new file mode 100644 index 0000000000..22d63bc830 --- /dev/null +++ b/__tests__/worker-teardown.test.ts @@ -0,0 +1,131 @@ +/** + * On Windows, a worker thread that ends while V8's concurrent marker is + * marking its heap can crash the whole process (0xC0000005, no error, no + * dump) — most of all while it is still loading its modules. Owners wait for a + * worker's first message before terminating it, and workers collect garbage + * before exiting (worker-teardown.ts). These pin both halves, then end real + * resolver and store workers right after starting them and record whether + * each had started when it was terminated. + */ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { EventEmitter } from 'events'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { Worker } from 'worker_threads'; +import CodeGraph from '../src/index'; +import { terminateOnceStarted, workerStarted } from '../src/worker-teardown'; + +const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); + +class FakeWorker extends EventEmitter { + terminate = vi.fn(async () => 0); +} + +describe('terminating a worker once it has started', () => { + it('waits for the first message', async () => { + const w = new FakeWorker(); + const started = workerStarted(w as unknown as Worker); + const ending = terminateOnceStarted(w, started); + await sleep(30); + expect(w.terminate).not.toHaveBeenCalled(); + w.emit('message', { type: 'ready' }); + await ending; + expect(w.terminate).toHaveBeenCalledOnce(); + }); + + it('counts an error or an exit as started', async () => { + for (const event of ['error', 'exit'] as const) { + const w = new FakeWorker(); + w.on('error', () => {}); + const started = workerStarted(w as unknown as Worker); + w.emit(event, event === 'error' ? new Error('boom') : 1); + await terminateOnceStarted(w, started); + expect(w.terminate).toHaveBeenCalledOnce(); + } + }); + + it('stops waiting after the cap, and never rejects', async () => { + const w = new FakeWorker(); + w.terminate.mockRejectedValue(new Error('already gone')); + const t0 = Date.now(); + await terminateOnceStarted(w, new Promise(() => {}), 60); + expect(w.terminate).toHaveBeenCalledOnce(); + expect(Date.now() - t0).toBeGreaterThanOrEqual(55); + }); +}); + +describe('collecting garbage before a worker exits', () => { + it('runs a full collection in a real worker, and leaves no global gc behind', async () => { + const teardown = path.resolve(__dirname, '../dist/worker-teardown.js'); + const run = (code: string) => new Promise((resolve, reject) => { + const w = new Worker(code, { eval: true }); + w.once('message', resolve); + w.once('error', reject); + }); + const ran = await run(` + const { collectBeforeExit } = require(${JSON.stringify(teardown)}); + let junk = Array.from({ length: 200000 }, (_, i) => ({ i, s: 'x'.repeat(32) })); + junk = null; + const before = process.memoryUsage().heapUsed; + const collected = collectBeforeExit(); + const after = process.memoryUsage().heapUsed; + require('worker_threads').parentPort.postMessage({ collected, freed: before - after, gc: typeof globalThis.gc }); + `) as { collected: boolean; freed: number; gc: string }; + expect(ran.collected).toBe(true); + expect(ran.freed).toBeGreaterThan(0); + expect(ran.gc).toBe('undefined'); + // A worker created afterwards doesn't get a global gc either. + expect(await run(`require('worker_threads').parentPort.postMessage(typeof globalThis.gc)`)).toBe('undefined'); + }, 30_000); +}); + +describe('real workers are never terminated before they have started', () => { + let root: string; + let dbPath: string; + const seen = new WeakSet(); + const startedWhenEnded: boolean[] = []; + + beforeEach(async () => { + root = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-worker-start-'))); + fs.writeFileSync(path.join(root, 'app.ts'), 'export function alpha() { return beta(); }\nexport function beta() { return 1; }\n'); + const cg = await CodeGraph.init(root); + try { await cg.indexAll(); } finally { cg.close(); } + dbPath = path.join(root, '.codegraph', 'codegraph.db'); + startedWhenEnded.length = 0; + // A worker re-emits each message it posts; note which have, and what each + // had done by the time it was terminated. + const emit = Worker.prototype.emit; + vi.spyOn(Worker.prototype, 'emit').mockImplementation(function (this: Worker, event: string | symbol, ...args: unknown[]) { + if (event === 'message') seen.add(this); + return emit.call(this, event, ...args); + }); + const terminate = Worker.prototype.terminate; + vi.spyOn(Worker.prototype, 'terminate').mockImplementation(function (this: Worker) { + startedWhenEnded.push(seen.has(this)); + return terminate.call(this); + }); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + fs.rmSync(root, { recursive: true, force: true, maxRetries: 10, retryDelay: 100 }); + }); + + it('the resolver pool, torn down while its workers boot', async () => { + vi.stubEnv('CODEGRAPH_RESOLVE_WORKERS', '2'); + const { ResolverPool } = require('../dist/resolution/resolver-pool') as typeof import('../src/resolution/resolver-pool'); + const pool = ResolverPool.tryCreate(dbPath, root); + expect(pool).not.toBeNull(); + await pool!.destroy(0); // the close fallback fires at once, mid-boot + expect(startedWhenEnded).toEqual([true, true]); + }, 30_000); + + it('the store writer, closed while its worker boots', async () => { + const { StoreWriter } = require('../dist/extraction/store-writer') as typeof import('../src/extraction/store-writer'); + const writer = new StoreWriter(path.resolve(__dirname, '../dist/extraction/store-worker.js'), dbPath, false); + await writer.close(0); + expect(startedWhenEnded.every(Boolean)).toBe(true); + }, 30_000); +}); diff --git a/src/extraction/store-worker.ts b/src/extraction/store-worker.ts index a562f557af..65b1f20972 100644 --- a/src/extraction/store-worker.ts +++ b/src/extraction/store-worker.ts @@ -32,6 +32,7 @@ import { QueryBuilder } from '../db/queries'; import { createDatabase, SqliteDatabase } from '../db/sqlite-adapter'; import { finalizeStoreBundle, type KernelStoreBundle, type StoreBundle } from './store-writer'; import { decodeExtractBuffers } from './kernel/decode'; +import { collectBeforeExit } from '../worker-teardown'; if (!parentPort) { throw new Error('store-worker must be run as a worker thread'); @@ -135,6 +136,8 @@ port.on('message', (msg: InMessage) => { } catch { /* already closed */ } + // No GC marking in flight when the thread ends (worker-teardown.ts). + collectBeforeExit(); process.exit(0); break; } diff --git a/src/extraction/store-writer.ts b/src/extraction/store-writer.ts index 683f09117e..4accc3d3c4 100644 --- a/src/extraction/store-writer.ts +++ b/src/extraction/store-writer.ts @@ -9,6 +9,7 @@ import { Worker } from 'worker_threads'; import { ExtractionResult, Language, Node, Edge, UnresolvedReference, FileRecord } from '../types'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; /** One file's complete store payload (pre-filtered — see storeFileBundle). */ export interface StoreBundle { @@ -66,6 +67,8 @@ export function finalizeStoreBundle( export class StoreWriter { private worker: Worker; + /** Settles on the worker's first message or its end — see worker-teardown.ts. */ + private started: Promise; private readyPromise: Promise; private firstError: Error | null = null; private drainWaiters = new Map void; reject: (e: Error) => void }>(); @@ -77,6 +80,7 @@ export class StoreWriter { constructor(workerScriptPath: string, dbPath: string, fastInit: boolean) { this.worker = new Worker(workerScriptPath); + this.started = workerStarted(this.worker); let readyResolve!: () => void; let readyReject!: (e: Error) => void; this.readyPromise = new Promise((resolve, reject) => { @@ -175,14 +179,18 @@ export class StoreWriter { return p; } - /** Close the worker's DB connection and join the thread. */ - async close(): Promise { + /** + * Close the worker's DB connection and join the thread; the worker collects + * garbage and exits by itself (worker-teardown.ts). One that hasn't by + * `timeoutMs` is terminated — but never while it is still starting up. + */ + async close(timeoutMs = 5000): Promise { if (this.exited) return; this.worker.postMessage({ type: 'close' }); await new Promise((resolve) => { const t = setTimeout(() => { - void this.worker.terminate().then(() => resolve()); - }, 5000); + void terminateOnceStarted(this.worker, this.started).then(() => resolve()); + }, timeoutMs); this.worker.once('exit', () => { clearTimeout(t); resolve(); diff --git a/src/mcp/engine.ts b/src/mcp/engine.ts index 62a4c8041c..cc0b43ba1d 100644 --- a/src/mcp/engine.ts +++ b/src/mcp/engine.ts @@ -18,6 +18,7 @@ import { ToolHandler } from './tools'; import { WslSharedIndexError } from '../db/wsl-shared-index'; import { assertNoRebuild, releaseWriterLock, tryAcquireWriterLock, writerLockHeldMessage } from './writer-lock'; import { QueryPool, resolvePoolSize } from './query-pool'; +import { endFreshnessMeasurements } from './index-freshness'; import { acquireProject, ProjectLease } from './project-lifecycle'; // Lazy-load the heavy CodeGraph chain (sqlite + query/graph/context layers) OFF @@ -271,13 +272,16 @@ export class MCPEngine { // Detach + terminate the worker pool first so no tool call routes to a // worker mid-teardown; outstanding pool calls resolve with graceful guidance. - // Stopping waits for the workers to end: the daemon exits right after, and - // exiting while a worker is still starting up can crash the process. + // Stopping waits for the workers to end — the pool's, and any + // `codegraph_status` change count still measuring: the daemon exits right + // after, and exiting while a worker is still starting up can crash the + // process. this.toolHandler.setQueryPool(null); const poolDown = this.queryPool ? this.queryPool.destroy() : Promise.resolve(); this.queryPool = null; + const measurementsDown = endFreshnessMeasurements(); const drained = this.toolHandler.closeAll(); - this.stopPromise = Promise.all([drained, poolDown]).then(async () => { + this.stopPromise = Promise.all([drained, poolDown, measurementsDown]).then(async () => { if (this.initPromise) await this.initPromise; if (this.defaultLease) { await this.defaultLease.release(); diff --git a/src/mcp/index-freshness-worker.ts b/src/mcp/index-freshness-worker.ts index 06d9787372..b678f1d20a 100644 --- a/src/mcp/index-freshness-worker.ts +++ b/src/mcp/index-freshness-worker.ts @@ -1,5 +1,6 @@ /** Exact CLI-parity change count, isolated from the MCP transport event loop. */ import { parentPort, workerData } from 'worker_threads'; +import { collectBeforeExit } from '../worker-teardown'; if (parentPort) { const port = parentPort; @@ -7,6 +8,9 @@ if (parentPort) { let counts: { added: number; modified: number; removed: number } | null = null; try { const CodeGraph = (require('../index') as typeof import('../index')).default; + // Loaded: from here on the owner may terminate this worker. Ending it while + // it loads those modules can crash the process on Windows (worker-teardown.ts). + port.postMessage({ type: 'loaded' }); cg = CodeGraph.openSync((workerData as { root: string }).root); const changes = cg.getChangedFiles(); counts = { @@ -19,5 +23,7 @@ if (parentPort) { } finally { try { cg?.close(); } catch { /* the worker is exiting */ } } - port.postMessage(counts); + // The owner terminates this worker on the answer: no GC marking in flight then. + collectBeforeExit(); + port.postMessage({ type: 'counts', counts }); } diff --git a/src/mcp/index-freshness.ts b/src/mcp/index-freshness.ts index ed6c66b02f..504216f425 100644 --- a/src/mcp/index-freshness.ts +++ b/src/mcp/index-freshness.ts @@ -1,6 +1,7 @@ import { existsSync } from 'fs'; import * as path from 'path'; import { Worker } from 'worker_threads'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; export interface PendingChangeCounts { added: number; @@ -12,20 +13,31 @@ export interface PendingChangeCounts { const MEASURE_TIMEOUT_MS = 8_000; let liveMeasurements = 0; const active = new Map>(); +/** Every measurement worker not yet terminated, with its load — for {@link endFreshnessMeasurements}. */ +const live = new Map>(); -export function measurePendingChanges(root: string): Promise { +export function measurePendingChanges(root: string, timeoutMs = MEASURE_TIMEOUT_MS): Promise { const key = path.resolve(root); const existing = active.get(key); if (existing) return existing; // Status probes for many projects must not exhaust the shared daemon. if (liveMeasurements >= 2) return Promise.resolve(null); - const pending = runMeasurement(key).finally(() => active.delete(key)); + const pending = runMeasurement(key, timeoutMs).finally(() => active.delete(key)); active.set(key, pending); return pending; } -function runMeasurement(root: string): Promise { +/** + * Terminate every measurement still running, each once it has loaded its + * modules (worker-teardown.ts) — for a server that is about to exit, since + * exiting tears the workers down the same way terminating them does. + */ +export async function endFreshnessMeasurements(): Promise { + await Promise.all([...live].map(([worker, loaded]) => terminateOnceStarted(worker, loaded))); +} + +function runMeasurement(root: string, timeoutMs: number): Promise { // The compiled sibling is beside us in production; Vitest loads src/ but // builds dist/ before the tests, so use that copy for the worker there. const sibling = path.join(__dirname, 'index-freshness-worker.js'); @@ -41,6 +53,10 @@ function runMeasurement(root: string): Promise { return Promise.resolve(null); } + // Its first message is 'loaded', posted once its modules are in. Terminating + // it before then can crash the whole process on Windows (worker-teardown.ts). + const loaded = workerStarted(worker); + live.set(worker, loaded); liveMeasurements++; return new Promise(resolve => { let settled = false; @@ -48,13 +64,19 @@ function runMeasurement(root: string): Promise { if (settled) return; settled = true; clearTimeout(timer); - // A worker inside a synchronous Git call may take time to terminate. - // Return unknown on deadline, but keep its concurrency slot until exit. - void worker.terminate().catch(() => {}).finally(() => { liveMeasurements--; }); + // A worker inside a synchronous Git call may take time to terminate, and + // one still loading its modules must not be terminated yet. Return + // unknown on deadline, but keep its concurrency slot until it is gone. + void terminateOnceStarted(worker, loaded).finally(() => { + live.delete(worker); + liveMeasurements--; + }); resolve(counts); }; - const timer = setTimeout(() => finish(null), MEASURE_TIMEOUT_MS); - worker.once('message', (value: PendingChangeCounts | null) => { + const timer = setTimeout(() => finish(null), timeoutMs); + worker.on('message', (msg: { type?: string; counts?: PendingChangeCounts | null }) => { + if (msg?.type !== 'counts') return; + const value = msg.counts; const valid = value && typeof value === 'object' && ['added', 'modified', 'removed'].every( key => Number.isSafeInteger(value[key as keyof PendingChangeCounts]) && value[key as keyof PendingChangeCounts] >= 0, ); diff --git a/src/resolution/resolver-pool.ts b/src/resolution/resolver-pool.ts index 96f4340d8d..457e1d0c0f 100644 --- a/src/resolution/resolver-pool.ts +++ b/src/resolution/resolver-pool.ts @@ -16,6 +16,7 @@ import * as os from 'os'; import type { Edge, UnresolvedReference } from '../types'; import type { ResolvedRef, UnresolvedRef } from './types'; import { memoryBudgetBytes } from './memory-budget'; +import { terminateOnceStarted, workerStarted } from '../worker-teardown'; /** One synthesis pass's output: its edge list + worker-measured wall clock. */ export interface SynthPassResult { @@ -34,9 +35,14 @@ export interface ChunkResult { interface PoolWorker { worker: Worker; ready: Promise; + /** Settles on the worker's first message or its end — see worker-teardown.ts. */ + started: Promise; busy: number; } +/** How long destroy() waits for a worker to exit on 'close' before terminating it. */ +const CLOSE_TIMEOUT_MS = 5000; + const MIN_PARALLEL_BATCH = 1000; const CHUNK_SIZE = 500; @@ -166,13 +172,14 @@ export class ResolverPool { private constructor(workerScript: string, dbPath: string, projectRoot: string, size: number) { for (let i = 0; i < size; i++) { const worker = new Worker(workerScript); + const started = workerStarted(worker); let readyResolve!: () => void; let readyReject!: (e: Error) => void; const ready = new Promise((resolve, reject) => { readyResolve = resolve; readyReject = reject; }); - const pw: PoolWorker = { worker, ready, busy: 0 }; + const pw: PoolWorker = { worker, ready, started, busy: 0 }; worker.on('message', (msg: { type: string; id?: number; message?: string; edges?: Edge[]; ms?: number } & Partial) => { if (msg.type === 'ready') { readyResolve(); @@ -341,14 +348,20 @@ export class ResolverPool { ); } - async destroy(): Promise { + /** + * Ask every worker to close; each collects garbage and exits by itself (see + * worker-teardown.ts). One that hasn't by `closeTimeoutMs` is terminated — + * but never while it is still starting up: a pool torn down soon after it + * booted, on a busy machine, can have a worker still loading its modules. + */ + async destroy(closeTimeoutMs = CLOSE_TIMEOUT_MS): Promise { await Promise.all( this.workers.map( (pw) => new Promise((resolve) => { const t = setTimeout(() => { - void pw.worker.terminate().then(() => resolve()); - }, 5000); + void terminateOnceStarted(pw.worker, pw.started).then(() => resolve()); + }, closeTimeoutMs); pw.worker.once('exit', () => { clearTimeout(t); resolve(); diff --git a/src/resolution/resolver-worker.ts b/src/resolution/resolver-worker.ts index f1d3020128..6cc3ec71f7 100644 --- a/src/resolution/resolver-worker.ts +++ b/src/resolution/resolver-worker.ts @@ -26,6 +26,7 @@ import { QueryBuilder } from '../db/queries'; import { ReferenceResolver } from './index'; import { SYNTH_PASSES } from './callback-synthesizer'; import { createYielder } from './cooperative-yield'; +import { collectBeforeExit } from '../worker-teardown'; import type { UnresolvedReference } from '../types'; if (!parentPort) { @@ -126,6 +127,8 @@ port.on('message', (msg: InMessage) => { } catch { /* already closed */ } + // No GC marking in flight when the thread ends (worker-teardown.ts). + collectBeforeExit(); process.exit(0); break; } diff --git a/src/worker-teardown.ts b/src/worker-teardown.ts new file mode 100644 index 0000000000..fa5d08e34a --- /dev/null +++ b/src/worker-teardown.ts @@ -0,0 +1,100 @@ +/** + * Ending a worker thread without taking the process down. + * + * On Windows, a worker thread that ends — terminated, by its own + * `process.exit()`, or with the process — while V8's concurrent marker is + * still marking its heap can crash the whole process with an access violation + * (exit 3221225477 / 0xC0000005), with no error and no dump. Bisected on a + * Windows 11 ARM64 VM (Node 24): `--no-concurrent-marking` alone stops it, and + * no other concurrency flag does — but that flag makes indexing about a third + * slower, so CodeGraph avoids the dangerous moments instead. + * + * Measured there, 480 child processes per row, four workers each: + * + * ended while still loading CodeGraph's modules 18 crashed + * loaded and allocating, terminated by the owner 4 crashed + * loaded and allocating, exiting by itself 1 crashed + * loaded and allocating, full collection, then exit 0 crashed + * + * So an owner never ends a worker that is still starting (it waits for the + * worker's first message, which is posted only after its modules load), and a + * worker that is about to exit collects garbage first — a full collection + * finishes any marking in flight. + */ + +import type { Worker } from 'worker_threads'; + +/** + * Longest an owner waits for a worker to finish starting before terminating + * it anyway. A start takes about a second normally and a few under heavy + * load; the cap only bounds a start that is wedged. + */ +export const WORKER_START_SETTLE_MS = 15_000; + +/** + * Settles once `worker` has started — its first message — or has gone away. + * Call it right after creating the worker, before it can post anything. + */ +export function workerStarted(worker: Worker): Promise { + return new Promise((resolve) => { + worker.once('message', () => resolve()); + worker.once('error', () => resolve()); + worker.once('exit', () => resolve()); + }); +} + +/** + * Terminate `worker`, but not while it is still starting: wait for `started` + * (from {@link workerStarted}) first, up to `capMs`. Never rejects. + */ +export async function terminateOnceStarted( + worker: Pick, + started: Promise, + capMs = WORKER_START_SETTLE_MS +): Promise { + let cap: NodeJS.Timeout | undefined; + await Promise.race([ + started, + new Promise((resolve) => { + cap = setTimeout(resolve, capMs); + cap.unref?.(); + }), + ]); + clearTimeout(cap); + try { + await worker.terminate(); + } catch { + // already gone + } +} + +let collect: (() => void) | null | undefined; + +/** + * In a worker that is about to exit: run a full garbage collection, so no + * marking is in flight when the thread ends. Returns whether one ran. The + * collector is reached at runtime (no `--expose-gc` on the command line) and + * is never exposed as a global. + */ +export function collectBeforeExit(): boolean { + if (collect === undefined) { + try { + // eslint-disable-next-line @typescript-eslint/no-require-imports + (require('v8') as typeof import('v8')).setFlagsFromString('--expose-gc'); + // eslint-disable-next-line @typescript-eslint/no-require-imports + collect = (require('vm') as typeof import('vm')).runInNewContext('gc') as () => void; + // Only the context made above needed it; workers created later don't get a global gc. + // eslint-disable-next-line @typescript-eslint/no-require-imports + (require('v8') as typeof import('v8')).setFlagsFromString('--no-expose-gc'); + } catch { + collect = null; + } + } + if (!collect) return false; + try { + collect(); + return true; + } catch { + return false; + } +} From 2965b7a349706fd650013bc0d738741b3539c228 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Sat, 3 Oct 2026 04:46:58 +0000 Subject: [PATCH 195/363] docs(changelog): ready [Unreleased] for 1.6.2; release verify waits up to 5 min (#2314) - Adds the Highlights block and groups the 164 entries the way 1.6.1's notes are: New Features for the newly supported routes and newly indexed code, Fixes under sub-headings (Windows; the background server and CLI; indexing; across languages; then one per language family) so a skimmer finds their area. Duplicates left by merges are folded together (the Windows console window entry twice; two overlapping Vapor closure-route entries; three Windows crash entries into one). No entry's wording is otherwise changed. - The release workflow's "Verify every package is actually on the registry" gave each package 60 s. 1.6.1 published all seven packages fine, but the first took longer than that to appear and the run went red. It now waits up to 5 minutes per package. No viewer entries: those stay in docs/viewer-launch-changelog.md, and `codegraph ui` stays behind CODEGRAPH_UI=1 (checked on a fresh build: absent from --help, refused with the "not in this release yet" message). Co-authored-by: Claude Opus 5.5 --- .github/workflows/release.yml | 4 +- CHANGELOG.md | 300 +++++++++++++++++++--------------- 2 files changed, 174 insertions(+), 130 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d48973f50f..38b097e17a 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -275,10 +275,12 @@ jobs: V="${{ steps.ver.outputs.version }}" # npm publish can print success without persisting; confirm against the # registry (with retries for propagation) so green means really shipped. + # Up to 5 minutes per package: 1.6.1's packages all published fine but + # took longer than the old 60 s to appear, which failed the run. for dir in release/npm/codegraph-* release/npm/main; do name=$(node -p "require('./$dir/package.json').name") ok= - for i in 1 2 3 4 5 6; do + for i in $(seq 1 30); do if npm view "$name@$V" version >/dev/null 2>&1; then ok=1; break; fi echo "waiting for $name@$V to appear ($i)…"; sleep 10 done diff --git a/CHANGELOG.md b/CHANGELOG.md index 9b76be6319..a333b1e2d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,172 +12,214 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Highlights + +- **Far fewer wrong links, in every language.** A call, import or type now resolves the way its language scopes names — through imports, packages, namespaces and the class it is written in — so it stops landing on an unrelated symbol that only shares its name. This covers TypeScript and JavaScript, Python, Java, Kotlin, Scala, C#, VB.NET, Swift, Objective-C, Go, Rust, C and C++, PHP, Ruby, Dart, Lua, R and more, and makes callers, impact and `codegraph_explore` answers more trustworthy. +- **Overloads, `super` and recursion resolve correctly.** A call reaches the overload its arguments fit, a `super` call reaches the parent's method, and a method that calls another overload, or another object's method of the same name, no longer looks like it calls itself. +- **Routes for more frameworks.** Angular (including Nx libraries), Vue Router admin apps, Rails nested routes, Flask's `add_url_rule`, ASP.NET FastEndpoints, Play subprojects, Vapor closure routes and Expo Router API routes are found. NestJS and Laravel routes are named by the path a request actually takes, and a Spring or Laravel test that calls a route by URL counts as covering it. +- **More of your code in the graph.** Vue Options API components, Java and Kotlin enum constants with bodies, Flow-typed JavaScript, Kotlin infix calls, React Native native views and events, Expo modules, tag-based CFML and Delphi `.inc` files are all indexed now. +- **Windows: no flashing windows, no random crashes.** The background server no longer pops up console windows while it works, sporadic access-violation crashes (exit code 3221225477) while indexing or shutting down are fixed, and `codegraph upgrade` works while an agent session is using CodeGraph. +- **A steadier background server.** Newer MCP clients such as Antigravity connect again, Claude Code sees the full text of `codegraph_status`, a session that lost the shared server reconnects by itself, and a project queried through `projectPath` is released after 10 idle minutes. +- **Indexing that finishes.** Large C and C++ projects no longer run out of memory while resolving references, a COBOL file can no longer stall indexing, and `codegraph sync` keeps every caller of a name that is defined more than once. +- **Upgrading:** re-index your projects after this release — most of these fixes are written while indexing. + +### New Features + +- In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. +- React Router links and `navigate(…)` calls written through a route-config object, like bulletproof-react's `paths.app.discussion.getHref(id)`, now reach the route they open. A `` with an arrow-function attribute before its `to`, such as `onMouseEnter={() => …}`, is read too. +- React Router data-router routes are named by their full path: nested `children` paths are joined to their parent's, `path: paths.app.root.path` constants are read, and `lazy: () => import('./routes/x')` routes link to the module's component. A guard wrapper like `` no longer stands in for the page. On bulletproof-react, every route now has its real path and page. +- Vapor routes whose handler is a trailing closure, like `app.get("hello") { req in … }`, WebSocket routes like `app.webSocket("chat") { req, ws in … }`, and `routes.on(.POST, "x", use: handler)` registrations are now found, and a closure route links to what the closure calls, the way an Express inline handler does. Before, only routes with a `use:` handler were read, and a closure route linked to nothing, so callers, impact and flows stopped at it. +- Flow-typed JavaScript files (those with `@flow` in their header comment) are now parsed like TypeScript, so type annotations like `render(): React.Node` no longer cut classes short. This mostly affects React Native code: on React Native's own libraries the graph gained over 8,000 links, and component classes keep their methods. Files without the pragma are unchanged. Re-index after upgrading. +- A link made by a framework or bridge resolver now records which one made it (`metadata.framework`, such as `swift-objc-bridge` or `react-native-bridge`), as the README describes. Before, only synthesized event and view channels were named. +- ASP.NET apps built with FastEndpoints now have their routes: each endpoint class's `Get(…)` / `Post(…)` in `Configure()` becomes a route linked to its own `HandleAsync` / `ExecuteAsync`, including paths kept in a request class's `Route` constant. ardalis/CleanArchitecture went from no routes to 24. Minimal API routes written without a leading slash (`app.MapGet("api/todos", …)`) are now named `/api/todos`. +- Play projects kept in subdirectories are recognized, so each one's `conf/routes` is read. A repository with no Play build at its root, like playframework's samples, went from no routes to over 130. +- Rails routes are now read with their nesting: `namespace` and `scope` add their path and controller module, nested `resources` sit under their parent's `:id`, and `member` / `collection` blocks add their actions. A namespaced route now links to its own module's controller. Routes in a Rails engine's `config/routes.rb` are found too: solidus went from no routes to over 600, and mastodon's route-to-action links nearly tripled. +- Flask routes registered with `add_url_rule(…)` are now found and linked to their view function or class-based view. So are routes registered through a project's own helper that passes a list of paths and a `view_func=`. flaskbb, which registers every view that way, went from no routes to over 100. +- Vue components written with the Options API (`export default { methods: {…}, computed: {…}, watch: {…}, mounted() {…} }`), as every Vue 2 app is, now have their methods, computed properties, watchers and lifecycle hooks in the graph. Each one has its own callers and calls, and a template's `@click="handleLogin"` links to the method it runs. On vue-element-admin that's over 800 new links, where a component's calls used to belong to the whole file. +- React Native events whose name is held in a constant now link native code to the JavaScript that listens for them, like NetInfo's `addListener(PrivateTypes.DEVICE_CONNECTIVITY_EVENT, …)` or a Java `.emit(PROGRESS_EVENT, …)`. The constant is read the way the language scopes it, from the same file, an import or its owning class, so a parameter that happens to share a constant's name is never mistaken for it. +- React Native views declared with `requireNativeComponent('X')`, the older Paper style that react-native-maps and segmented-control use, now link from JavaScript to their native iOS and Android implementations, as Codegen specs already did, including in plain `.js` files. A JSX tag written under a different name than its component, like `` for `import Autocomplete from './autocomplete.vue'`, now links to the component it renders. +- Expo Router API routes (`app/hello+api.ts`) are now endpoints, one per exported method, like `GET /hello`, bound to the function that handles them. They used to show up as a screen named `/hello+api`. In a repository whose root declares Expo Router for an example app, a Next.js app in its own folder, like react-native-true-sheet's `docs/`, no longer gets Expo screens made from its files. +- Calls into an Expo module now reach the module's native functions even when the JavaScript names the module something else, like expo-camera's `CameraManager` for `requireNativeModule('ExpoCamera')`. Both the iOS and the Android implementations are linked. expo-camera's calls used to link back to the same-named method making the call, so a method looked like it called itself. +- Angular apps split across Nx libraries, like angular-spotify, now have their routes and navigation. A lazily loaded route written `async () => (await import('@app/home')).HomeModule` is followed through the library's `index.ts` to the module it re-exports, and a `routerLink` in one library now reaches a screen declared in another. Route paths built from a class constant (`RouterUtil.Configuration.Lyrics`, `` `issue/:${ProjectConst.IssueId}` ``) or an enum are now read. A redirect in a routes file that only lazy-loads others now counts, and `router.navigate(['project', 'issue', id])` without `relativeTo` is read from the root, as Angular does. Re-index Angular projects after upgrading. +- React Router apps now show navigation written as React Router v5's `` and through a styled link, like react-boilerplate's `HeaderLink = styled(Link)` used as ``. Apps built this way had routes but no links between their screens. +- SvelteKit routes are now named by the address a browser asks for. A `(group)` folder like `(app)` or `(marketing)` is no longer part of a route's path, and a parameter with a matcher, like `[id=integer]`, is just `:id`. So `goto('/blocks')` and `` now reach a page that lives in `src/routes/(app)/blocks/`, where before they reached nothing and the app's screens showed no navigation. Re-index SvelteKit projects after upgrading. +- Vue Router apps built like vue-element-admin, vue-admin-template or vben now have their routes and screens. Routes declared in a named table (`export const constantRoutes = [...]`, `const routes: RouteRecordRaw[] = [...]`), in per-module route files, or through `new Router(...)` are now read. Child routes are joined onto their parent's path, and the parent's component is the layout around them. A lazily loaded view (`() => import('@/views/dashboard/index')`) is bound to the file it names, and `this.$router.push(...)` counts as navigation. Nuxt's file-based routes now come only from a Nuxt app, so a plain Vue app's `pages/` folder no longer turns into made-up screens, and a Nuxt app at the root of its repository now gets its routes. Re-index Vue projects after upgrading. +- NestJS routes are now named by the path a request takes: the global prefix from `app.setGlobalPrefix('api')` is applied, along with its `exclude` list, and so is URI versioning from `app.enableVersioning(...)`, with `@Version`, `VERSION_NEUTRAL` and a controller's own `version` taken into account. A route that used to appear as `GET /user` is now `GET /api/v1/user`, so a front end's `this.http.get('/api/v1/user')` or `fetch` call connects to the controller method that serves it. Re-index NestJS projects after upgrading. +- Laravel routes are now named by the path a request takes: a leading `/` is added where the routes file leaves it out, `Route::prefix()` and `Route::group(['prefix' => …])` groups are applied, and routes in `routes/api.php` carry the `/api` prefix Laravel serves them under, read from your `RouteServiceProvider`, `bootstrap/app.php` or any file that mounts a routes file. A front-end `fetch('/api/…')` now connects to the Laravel route that serves it. Re-index Laravel projects after upgrading. +- Controllers that Spring or Laravel tests exercise by URL now count as tested. That covers MockMvc's `perform(post("/owners/new"))`, WebTestClient, TestRestTemplate, RestAssured, Laravel's `$this->postJson('api/me')`, Pest's `get('/about')`, and a project's own request helpers built on them. `codegraph_explore` now names the test suite that reaches such an endpoint, where before every one of them looked untested. Re-index Spring and Laravel projects after upgrading. +- In an Angular app, a method a template calls, like `(click)="toggleFavorite()"` or `(ngSubmit)="submitForm()"`, is now linked from its component. These methods used to show no callers at all, so impact and callers questions stopped at them, and dead-code checks could list them as unused. Re-index Angular projects after upgrading. +- Angular apps now have routes and navigation in the graph. Each screen in a `Routes` array is a route named by its full path, including children, lazily loaded `loadChildren` files and NgModules, paths written as route constants or `$localize` strings, and redirects. Each route is linked to its component. `router.navigate([...])`, `navigateByUrl`, a guard's `createUrlTree` and every `routerLink` in a component's template link to the screen they open, and a template's child components (``) are linked to the component that renders them. Questions like "where does this button go" and "what renders this component" now have answers in `codegraph_explore`. Re-index Angular projects after upgrading. +- Delphi and Free Pascal include files (`.inc`) are now indexed as Pascal. Before, they were read as PHP and came up empty. A `.inc` file with a ``, ``/``, ``, `` and `#…#` expressions. Before, only `` and `` code was read, so callers and impact found almost nothing in tag-based components. Functions wrapped in tags like `` or `` are now indexed too, and a `` is indexed as an interface that `implements` links to. Re-index CFML projects after upgrading. Thanks @HarryMuc for the report. (#2091) +- Java enum constants and Kotlin enum entries with a body of their own now have their methods in the graph. Examples are `PLUS { int apply(…) { … } }` and `NewBuffer { override fun pipe() … }`. Before, those methods were missing and every call inside them was dropped. jsoup's HTML tokenizer and tree builder are written this way, so over 2,000 of their calls were invisible. Re-index to pick this up. +- Kotlin infix calls, like `Users.id eq id1` or koin's `single { … } bind MyApi::class`, are now recorded as calls to the infix function. Before, they had no callers at all. On Exposed, `eq` now shows over a thousand callers. Bit operations on numbers (`x and 0xff`, `h shr 8`) are left to Kotlin's own operators, even where the project defines a same-named extension. Re-index after upgrading. + ### Fixes +#### Windows + - On Windows, terminal windows no longer flash open and closed while CodeGraph runs in the background. Since 1.6.1, a background MCP server popped up a console window (a full Windows Terminal window when that is the default terminal) several times when it started and again every time it re-synced a changed file. All of CodeGraph's git calls now run hidden. Thanks @Suharaz, @23q3, @A-Van-Gestel and @HarryMuc. (#2094, #2096) -- In Vapor apps, a route whose handler is a trailing closure, like `app.get("hello") { req in … }` or `app.webSocket("chat") { req, ws in … }`, now links to what the closure calls, the way an Express inline handler does. Before, the route linked to nothing, so callers, impact and flows stopped at it. +- On Windows, a project opened with a different drive-letter or folder-name case, like `d:\work\app` and `D:\Work\App`, now joins the CodeGraph background server that is already running for it. Before, each spelling started a server of its own that exited at once, so the session quietly served the graph by itself with no shared file watching or auto-sync, and `codegraph list` showed the project twice. Thanks @greatjackzhou. (#2278) +- On Windows, `codegraph upgrade` no longer breaks the install while an agent session is using CodeGraph. It used to stop partway and leave CodeGraph unable to start; it now swaps the new files in with CodeGraph running and puts the previous version back if anything fails, and re-running the PowerShell installer works the same way. If an earlier upgrade already broke your install, re-run `irm https://raw.githubusercontent.com/colbymchenry/codegraph/main/install.ps1 | iex` to repair it. Thanks @tippmar-nr for the report. (#2185) +- On Windows, CodeGraph no longer sometimes crashes with exit code 3221225477 (an access violation) on a busy machine: right after parsing in `codegraph init`, `codegraph index` and background syncs, or as the background server shuts down. Stopping one of its helper threads (the ones that parse files, resolve references, write the index and count changes for `codegraph_status`) while it was still starting up, or before it had finished cleaning up its memory, took the whole process down. Threads now finish starting and clean up before they exit. + +#### The background server, sessions and the CLI + +- The Claude Code prompt hook no longer runs on the messages Claude Code uses to hand a subagent's report back to the main session. Before, such a long report could keep the hook busy past Claude Code's 30-second hook timeout and inject context unrelated to what you asked. Thanks @danusha2345, and @tippmar-nr for the report. (#2184) +- Newer MCP clients such as Antigravity 2.5 connect again: the server now answers their `server/discover` probe right away with "method not found", so they go straight on to the regular handshake instead of waiting on a reply that could take seconds or never come. Thanks @danusha2345, and @samem26 for the report. (#2084) +- A CodeGraph session that queried another project through `projectPath` no longer keeps that project locked for as long as it runs: after 10 minutes without a query it lets the project go, so the project's own session and `codegraph index` can take over again (tune with `CODEGRAPH_PROJECT_IDLE_TIMEOUT_MS`, `0` keeps it open). Thanks @danusha2345, and @bompus for the report. (#2087) +- Claude Code now shows the model the full text of `codegraph_status` and of explore's stale-index refusal. Both results also carried `structuredContent`, and Claude Code shows a result's `structuredContent` in place of its text, so the model saw only a freshness JSON object: no refusal message, no retry guidance and no status report. Both results are now text only. Thanks @bompus. (#2088) +- When an agent asks CodeGraph about a separate git repository nested inside an indexed project, one the project's index leaves out (for example because the parent's `.gitignore` excludes it), it now gets the usual "isn't indexed" guidance instead of answers from the parent project's code. Nested repositories the parent does index, such as submodules, work as before. Thanks @wstczyw for the report. (#2110) +- An agent session that lost its connection to CodeGraph's shared background server, or couldn't reach it at startup, no longer stops that server from running for the rest of the session. It reconnects on its own once the server can start, so other sessions on the same project get auto-sync back instead of running read-only. Thanks @ijbranch for the report. (#2277) + +#### Indexing and sync + - A COBOL file whose last line ends inside the sequence area (columns 1-6), such as a closing period indented only four spaces, no longer stalls indexing for over a minute and then ends up in the index with no symbols. -- In Laravel apps, a route whose controller is named by string now links to its handler, without needing a `Controller` suffix and keeping the namespace it's written with: `'Common\Uploads@inline'`, and `Route::resource('companies', 'Common\Companies', [...])` with an options array. Two same-named controllers in different namespaces are told apart by that namespace. Before, akaunting linked 304 of its 313 routes to nothing. -- In Nuxt and Astro projects, a file-routed page is now linked to the component its file defines. An Astro endpoint is linked to the HTTP-verb handlers it exports, like `export const GET`. Before, a page route like Nuxt's `pages/admin.vue` or Astro's `src/pages/about.astro` linked to nothing, so callers, impact and flows stopped at the route. -- In Rails apps, a `resources` line's `only:` and `except:` are now read in every form Rails accepts: `%i[new create index]`, `%w(index show)`, a quoted name, and the older `:only => [...]`. Before, only a bracketed list was read, so maybe's `resources :family_exports, only: %i[new create index]` became all seven routes, four of them to actions the controller doesn't have. +- Editing a file that defines the same name more than once, like two classes that each have an `execute` method or a method's overloads, no longer moves every caller from other files onto one of them during `codegraph sync`. A sync interrupted partway through a file also no longer loses those callers until a full re-index. Thanks @ijbranch for the report. (#2276) +- Indexing a large C or C++ project no longer runs out of memory partway through "Resolving refs". Headers whose include guard is written as `#define X_H 1`, as in OpenSceneGraph and osgEarth, or that are included under a build flag codegraph cannot know, used to be re-read on every include path, which grows exponentially with the depth of the include tree. Each is now read once per include state, and a hard limit stops the check on any include tree that is still too large. Thanks @danusha2345, and @coolhitmanleon for the report. (#2127) +- On a busy machine, indexing or syncing a small project no longer starts extra background threads for resolving references that it can't use. Under heavy load the slower pace made a ten-file project look big enough to need them, which added seconds to the run and held extra memory. + +#### Across languages + +- A type that extends another type of the same name now links to that other type instead of to itself. When the source spells out the qualifier, that qualifier picks between same-named candidates. Before, cats' `trait BigDecimalInstances extends cats.kernel.instances.BigDecimalInstances` and every `trait AllOps … with Functor.AllOps[…]` pointed back at the declaring trait, and Newtonsoft.Json's `JsonConverter : JsonConverter` extended itself. +- Framework name conventions for Django, FastAPI, Flask, ASP.NET, Gin, SwiftUI and Vapor now only pick a class the reference can see. Examples of these conventions are "a `…View` is a view", "a `…Service` is a service" and "a capitalized name is a model". The pick starts from the reference's own file and then its package, and never takes a class declared inside some function or nested in another file's class. Before, Django REST framework's tests linked each file's own `MockView` and `Serializer` to another file's, and netbox's tests linked `TestForm()` to a form declared inside a different test function. +- A method that calls a same-named method on another expression's result is no longer linked to itself in any language. Examples are Scala's `requestToArmeria(request).execute()` inside `execute()`, Rust's `self.0.into_route(state)` inside `into_route()`, and Kotlin's `this@Buffer.write(…)` from an inner object. A value whose initializer chain calls a same-named method, like sttp's `val response = basicRequest.get(…).response(…)`, no longer links to itself either. +- A method that hands its call on to another object is no longer linked to itself. In TypeScript and JavaScript, a call through `this.` or `window.` counts as recursion only when the field is declared as the method's own class. A guess from a receiver's name alone never lands on the calling method. Before, BookStack's `toggle()` doing `this.container.classList.toggle('open')`, `listen()` doing `window.$events.listen(…)`, and `FileStorage::delete` doing `$storage->delete($path)` each pointed at themselves. +- A function or name passed as a value now resolves to what is in scope where it's written. A function nested inside another function is only reachable from inside it. In Python, a parameter or local of the same name is that local. A pytest fixture is a test's parameter only in its own module or under its `conftest.py`. Before, httpx's `self._build_auth(auth)` linked to an `auth` a test defines inside another function, and `auth_flow(self, request)` handing `request` on linked to the package's `request()` function. +- In Java, C#, Kotlin, Swift, Scala, Dart and VB.NET, a call now reaches the overload whose parameters fit its arguments, not whichever same-named overload was indexed first. For example, `HashCodeBuilder.reflectionHashCode(this)` now reaches the one-argument overload with varargs, not a three-parameter one. Each overload's callers are now its own. +- A method that calls another overload of itself now links to that overload rather than to itself. An example is `toInstant(instant)` returning `toInstant(instant, Instant.EPOCH)`. The fuller overload now lists its convenience overloads among its callers, so its impact includes them. This applies to Java, C#, Kotlin, Swift, C++, Scala, Dart and VB.NET. Real recursion is unchanged. +- Production code no longer links to a same-named symbol inside a test suite, since tests aren't built into the program. Before, typeorm's `Record` types reached a test entity called `Record`, and okhttp's sample `@Override` annotations reached a test's nested `Override` class. Test-support code a project ships, such as a `testing/` folder or a `*-test` module, stays in reach. +- A method call on a variable is no longer guessed to be a test double's method just because the double has the only method of that name. That covered classes named `Mocked…`, `Fake…`, `Stub…` or `Dummy…`: django-allauth's `resp.json()` calls had all gone to its `MockedResponse`. A test that names the double still calls it. +- A call through a class name, like `THorse.Get(…)`, `MyView.as_view()` or `ColorChoices.values()`, now reaches the class method that class inherits. The inheritance is read from the class declarations themselves, in Pascal, Python, Ruby, PHP, TypeScript/JavaScript and the Java family. Before, such a call was resolved by guessing from the name. On Horse, `THorse.Get('/ping', …)` used to land on a route-group interface's `Get` 120 times. Re-index after upgrading. +- A method guessed from a receiver's name no longer lands on a request handler the web framework dispatches to, such as a Django or DRF view's `post` / `get` or a controller's `update` / `destroy`. It also no longer lands on a test double the calling file never mentions. On django-allauth, tests' `client.post(…)` used to land on a `ClientRegistrationView` 425 times. On mealie, `response.json()` used to land on a test's `_FakeHTTPResponse` 424 times. In Python, `site = Site.objects.create(…)` now makes `site` a `Site`, so `site.save()` reaches the model's own `save`. Re-index after upgrading. +- In C#, Java and Kotlin, a call through a field, property or parameter now lands on the method of the type it is declared with. That includes a type inherited from a base class with its type argument filled in, a generic or `for`-each declaration, a C# `using` alias, and a type parameter's bound. A call no longer falls back to a guess at some project class's same-named method, and a type from outside the project gives no link. On Newtonsoft.Json, `_innerWriter.WriteValue(…)` inside a writer wrapper used to land on the wrapper's own `WriteValue`, and a `TextWriter`'s `Write(…)` on a test writer's. On commons-lang, a `DateFormat` field's `parse(…)` used to land on the project's `DateParser` 284 times. +- A call on a variable declared with a type from outside the project (`List list`, `var sb = new StringBuilder()`, `signer = TimestampSigner()`) no longer lands on a project type's method with the same name. On gson, `list.add(…)` used to land on a project list wrapper; on commons-lang, `s.length()` and `buffer.append(…)` on project classes. In Python, a keyword argument inside a call (`prefix=IPNetwork(…)`) is no longer taken for an assignment to that name. +- A call on a type from outside the project, like `Integer.valueOf(…)`, `Arrays.asList(…)`, `Object.assign(…)` or Delphi's `Exception.Create(…)`, no longer lands on the project's own method with that name. On commons-lang, `Integer.valueOf` used to land on `StringUtils.valueOf` over 800 times. Go's exported variables and Pascal's capitalized parameters and locals are not mistaken for type names. +- In PHP, CFML, Pascal, COBOL and VB.NET, a call written in a different case than its definition now resolves, like `formatprice()` for `FormatPrice` or CFML's `ToMatchWithCase()` for `toMatchWithCase`. These languages ignore case in names, but such calls used to be dropped before resolution. +- In Java, Kotlin, Swift, C#, C and C++, Go, Scala, Ruby, Dart and the other case-sensitive languages, a name no longer links to a symbol whose name differs only in case. On jsoup, JUnit's `@Test` had linked every test to a `test` method; `new CookieManager()` had linked to a `cookieManager()` getter, and a `Method` type to a `method()`. PHP, Pascal, CFML, COBOL and VB.NET, whose names really do ignore case, are unchanged. +- A `super` call in an overriding method, like `super.viewDidLoad()`, `[super init]`, `base.Handle()`, `parent::setUp()` or `super().dispatch(…)`, no longer makes the method look like it calls itself. The call goes to the parent's version, so an override stops showing up as recursive and no longer lists itself among its own callers. +- A generic type parameter, like `A` in `def zipWith[A, B](…)`, `T` in ` T max(…)` or `class Foo`, no longer links to a project symbol that happens to share its name. On cats, one `implicit def A` had thousands of made-up dependents from every `A` in the library. The same goes for Rust, Dart, TypeScript, Java, Kotlin, Swift, Go, C# and C++. A class or other type declared inside a function is only linked from inside that function. In Scala, a `def`'s own parameter, like `f` in `(f: A => B)`, no longer links to a same-named field elsewhere. A bare name no longer reaches a type nested inside another type unless the code is inside that type, extends it or imports its members; the type in scope is linked instead, like cats' own `FlatMap` in place of `Eval`'s nested `FlatMap`. Re-index projects in these languages after upgrading. + +#### JavaScript and TypeScript, with Vue, Svelte, Astro, React, Express, NestJS and Hono + - In Vue, Svelte and Astro files, a component now owns its script, and what a ` diff --git a/telemetry-dashboard/public/styles.css b/telemetry-dashboard/public/styles.css index 73ec9751c1..e2dc7f88c7 100644 --- a/telemetry-dashboard/public/styles.css +++ b/telemetry-dashboard/public/styles.css @@ -107,6 +107,28 @@ button:focus-visible { padding: 0 4px; } +/* --- stale-data warning ------------------------------------------------- */ + +.data-warning { + margin-top: 16px; + padding: 12px 16px; + background: var(--surface); + border: 1px solid var(--oxblood); +} + +.data-warning p { + margin: 0; +} + +.data-warning p + p { + margin-top: 6px; +} + +.data-warning strong { + color: var(--oxblood); + font-weight: 600; +} + /* --- buttons ------------------------------------------------------------ */ button { diff --git a/telemetry-dashboard/scripts/fixture.sql b/telemetry-dashboard/scripts/fixture.sql index 108df44365..9e2ddef020 100644 --- a/telemetry-dashboard/scripts/fixture.sql +++ b/telemetry-dashboard/scripts/fixture.sql @@ -117,6 +117,12 @@ SELECT machine_id, min(day) FROM events GROUP BY machine_id; INSERT INTO daily_machines (day, machines, prod_machines) SELECT day, count(*), coalesce(sum(prod), 0) FROM machine_days GROUP BY day; +-- The activation funnel's input: each machine's earliest index day (FIRST_INDEX_DAY). +-- m04 and m06 never index, so theirs stays NULL. +UPDATE machine_first_seen + SET first_index_day = (SELECT min(day) FROM events e + WHERE e.machine_id = machine_first_seen.machine_id AND e.event = 'index'); + INSERT INTO daily_event_counts (day, event, count, machines) SELECT day, event, CASE WHEN event = 'usage_rollup' diff --git a/telemetry-dashboard/scripts/render-check.mjs b/telemetry-dashboard/scripts/render-check.mjs index 9411bec541..fd1a740fbd 100644 --- a/telemetry-dashboard/scripts/render-check.mjs +++ b/telemetry-dashboard/scripts/render-check.mjs @@ -176,8 +176,12 @@ const PROBE = `(() => { tableHidden: section.querySelector('[data-role="table"]').hidden, }; }); + const warning = document.getElementById('data-warning'); return { ready: document.body.dataset.ready === 'true', + warning: warning.hidden ? null : warning.textContent, + customFrom: document.querySelector('[data-role="custom-from"]').value, + customTo: document.querySelector('[data-role="custom-to"]').value, range: document.getElementById('range-summary').textContent, dataThrough: document.getElementById('data-through').textContent, refreshed: document.getElementById('refreshed-at').textContent, @@ -299,17 +303,32 @@ async function main() { // The very same registry the page just rendered from, imported here so the // expectations cannot drift from the panels under test. const { PANELS } = await import(pathToFileURL(join(root, 'public', 'panels.js')).href); + const { shortDay } = await import(pathToFileURL(join(root, 'public', 'theme.js')).href); + const utcDay = (ms) => new Date(ms).toISOString().slice(0, 10); + const today = utcDay(Date.now()); check(`all ${PANELS.length} panels are on the page`, view.panels.length === PANELS.length, `got ${view.panels.length}`); - const broken = view.panels.filter((p) => p.state !== 'ready'); + // The fixture lives in July, so a range ending today has nothing in it: every + // panel should say so plainly, and none should error. + const broken = view.panels.filter((p) => p.state !== 'ready' && p.state !== 'empty'); check( - 'every panel reached its ready state', + 'every panel settled without an error', broken.length === 0, broken.map((p) => `${p.id}: ${p.state} ${p.message}`).join(' | '), ); check('the default range is the 30-day preset', view.selectedPreset === 'Last 30 days', view.selectedPreset); - check('the range is stated in the filter row', /Jun|Jul/.test(view.range), view.range); - check('the data horizon is stated', view.dataThrough.includes('Jul 10'), view.dataThrough); + same('…ending today, not on the last rolled-up day', today, view.customTo); + same('…and starting 29 days before', utcDay(Date.now() - 29 * 864e5), view.customFrom); + check('the range is stated in the filter row', view.range.includes(shortDay(today)), view.range); + check('the rollup horizon is stated', view.dataThrough.includes('Event counts through Jul 10'), view.dataThrough); + check('…with yesterday\'s activity beside it', view.dataThrough.includes('0 machines active yesterday'), view.dataThrough); check('the refresh time is stated', view.refreshed.startsWith('Last refreshed'), view.refreshed); + // Nothing has arrived since the fixture's last day: the page must say so rather + // than present July as current. + check( + 'a stalled ingest is called out above the panels', + view.warning?.includes('No new events since Jul 10.') === true, + String(view.warning), + ); // A CSP violation surfaces here as a `security` log entry, which is the point // of the check: the page must work under `script-src 'self'` with no inline @@ -339,10 +358,10 @@ async function main() { (await cdp.evaluate(sessionId, 'document.body.dataset.ready === "true"')) === true, ); view = await cdp.evaluate(sessionId, PROBE); - const weekly = view.panels.find((p) => p.id === 'daily-production-users'); - check('a daily line now holds 7 points', weekly.chart?.labels.length === 7, `${weekly.chart?.labels.length}`); + same('the range now starts 6 days back', utcDay(Date.now() - 6 * 864e5), view.customFrom); + check('…and still ends today', view.customTo === today, view.customTo); check('the 7-day preset is marked selected', view.selectedPreset === 'Last 7 days', view.selectedPreset); - check('every panel re-rendered cleanly', view.panels.every((p) => p.state === 'ready')); + check('every panel re-rendered cleanly', view.panels.every((p) => p.state === 'ready' || p.state === 'empty')); console.log('\nA custom range works the same way'); await cdp.evaluate( @@ -358,6 +377,12 @@ async function main() { view = await cdp.evaluate(sessionId, PROBE); check('the fixture window is 10 days', view.panels.find((p) => p.id === 'daily-production-users').chart?.labels.length === 10); check('no preset stays highlighted', view.selectedPreset === null, view.selectedPreset); + const notReady = view.panels.filter((p) => p.state !== 'ready'); + check( + 'every panel reached its ready state', + notReady.length === 0, + notReady.map((p) => `${p.id}: ${p.state} ${p.message}`).join(' | '), + ); // -- every panel plots what the API returned ------------------------------ console.log('\nEvery panel plots the API’s own numbers'); @@ -428,6 +453,29 @@ async function main() { same('languages lead with typescript', 'typescript', byId.languages.chart.labels[0]); check('retention starts at 100%', byId.retention.chart.datasets[0].data[0] === 100); + console.log('\nPast the rollup\'s last day, lines stop instead of dropping to zero'); + await cdp.evaluate( + sessionId, + `document.body.dataset.ready = ""; + document.querySelector('[data-role="custom-from"]').value = "2026-07-05"; + document.querySelector('[data-role="custom-to"]').value = "2026-07-14"; + document.querySelector('[data-role="custom-apply"]').click();`, + ); + await waitFor('the over-the-edge render', async () => + (await cdp.evaluate(sessionId, 'document.body.dataset.ready === "true"')) === true, + ); + const edge = Object.fromEntries((await cdp.evaluate(sessionId, PROBE)).panels.map((p) => [p.id, p])); + same( + 'daily production users ends in gaps after Jul 10', + [[2, 3, 2, 1, 1, 1, null, null, null, null]], + edge['daily-production-users'].chart?.datasets.map((d) => d.data), + ); + same( + 'new installs are live, so they stay zero rather than gap', + [[2, 0, 0, 1, 2, 0, 0, 0, 0, 0]], + edge['new-installs'].chart?.datasets.map((d) => d.data), + ); + // Colour, spacing and label collisions are not things an assertion catches. // RENDER_SHOT=/tmp/dash.png npm run smoke:render → look at it. if (process.env.RENDER_SHOT) { diff --git a/telemetry-dashboard/scripts/seed-fixture.sh b/telemetry-dashboard/scripts/seed-fixture.sh index 88c8b1e271..3f4eb66085 100755 --- a/telemetry-dashboard/scripts/seed-fixture.sh +++ b/telemetry-dashboard/scripts/seed-fixture.sh @@ -11,17 +11,20 @@ set -uo pipefail cd "$(dirname "$0")/.." DB=codegraph-telemetry -MIGRATION=../telemetry-worker/migrations/0001_init.sql +MIGRATIONS=../telemetry-worker/migrations -if [[ ! -f "$MIGRATION" ]]; then - echo "seed: cannot find $MIGRATION — run this from a full checkout" >&2 +if [[ ! -f "$MIGRATIONS/0001_init.sql" ]]; then + echo "seed: cannot find $MIGRATIONS — run this from a full checkout" >&2 exit 1 fi -# The migration is plain CREATE TABLE, so a second run fails on "table already -# exists". That is the expected steady state here, hence the swallowed output — -# the fixture load below is the step whose failure actually matters. -npx wrangler d1 execute "$DB" --local --file="$MIGRATION" >/dev/null 2>&1 +# Applied in order, each as a plain file. A second run fails every one of them +# ("table already exists", "duplicate column") — the expected steady state here, +# hence the swallowed output. The fixture load below is the step whose failure +# actually matters, and it fails loudly if a migration is missing. +for migration in "$MIGRATIONS"/*.sql; do + npx wrangler d1 execute "$DB" --local --file="$migration" >/dev/null 2>&1 +done if ! npx wrangler d1 execute "$DB" --local --file=scripts/fixture.sql >/dev/null; then echo "seed: loading scripts/fixture.sql failed" >&2 diff --git a/telemetry-dashboard/scripts/smoke-api.sh b/telemetry-dashboard/scripts/smoke-api.sh index 964c1b6819..67008c47b4 100755 --- a/telemetry-dashboard/scripts/smoke-api.sh +++ b/telemetry-dashboard/scripts/smoke-api.sh @@ -92,12 +92,21 @@ check "health stays uncached" "no-store" \ "$(curl -sD - -o /dev/null -b "$JAR" "$BASE/api/health" | grep -i '^cache-control:' | cut -d' ' -f2- | tr -d '\r')" echo -echo "/api/meta — what the range picker anchors on" +echo "/api/meta — how current each kind of number is" +TODAY="$(node -e 'console.log(new Date().toISOString().slice(0, 10))')" +YESTERDAY="$(node -e 'console.log(new Date(Date.now() - 864e5).toISOString().slice(0, 10))')" META="$(get "/api/meta")" -field "latest day" latest_day 2026-07-10 "$META" -field "earliest day" earliest_day 2026-07-01 "$META" -field "raw events start" earliest_raw_day 2026-07-01 "$META" -field "retention window" retention_days 14 "$META" +field "today, as a UTC day" today "$TODAY" "$META" +field "latest rollup day" latest_rollup_day 2026-07-10 "$META" +field "…kept as latest_day" latest_day 2026-07-10 "$META" +field "earliest day" earliest_day 2026-07-01 "$META" +field "raw events start" earliest_raw_day 2026-07-01 "$META" +field "retention window" retention_days 14 "$META" +# The fixture stops on 07-10, months ago: nothing has arrived since, but the +# rollup did cover every day that saw activity, so it is not behind. +field "ingest stalled (no events since 07-10)" ingest_stalled true "$META" +field "rollup not behind (it covered every active day)" rollup_behind false "$META" +field "nobody active yesterday" machines_yesterday 0 "$META" echo echo "/api/summary — the big numbers (12 machines, one of them CI)" @@ -134,6 +143,7 @@ field "calls per day" datasets.0.data '[0,40,28,0,0,12,0,0,0,5]' "$TS" field "machines per day" datasets.1.data '[0,1,2,0,0,1,0,0,0,1]' "$TS" TS="$(get "/api/timeseries?metric=duration_buckets&$RANGE")" +field "coverage reported" covered_through 2026-07-10 "$TS" field "bucket order is the scale" datasets.0.label '<10s' "$TS" field "…and ends at the longest" datasets.3.label '5m+' "$TS" field "<10s over time" datasets.0.data '[2,0,1,1,0,0,0,1,0,0]' "$TS" @@ -141,6 +151,22 @@ field "10-60s over time" datasets.1.data '[0,2,0,0,0,0,1,0,0,1]' "$TS" field "1-5m over time" datasets.2.data '[0,0,0,0,1,1,0,0,0,0]' "$TS" field "5m+ over time" datasets.3.data '[0,0,1,0,0,0,0,0,1,0]' "$TS" +echo +echo "Past the rollup's last day: not counted yet, so null — never a zero" +# The fixture is rolled up through 07-10; 07-11 and 07-12 have not been. +LATE="from=2026-07-09&to=2026-07-12" +TS="$(get "/api/timeseries?metric=installs_uninstalls&$LATE")" +field "installs stop at the rollup" datasets.0.data '[2,0,null,null]' "$TS" +field "…and say where" covered_through 2026-07-10 "$TS" +field "the table twin keeps the gap" rows.2.Installs null "$TS" +TS="$(get "/api/timeseries?metric=production_users&$LATE")" +field "daily production users too" datasets.0.data '[1,1,null,null]' "$TS" +TS="$(get "/api/timeseries?metric=duration_buckets&$LATE")" +field "run length over time too" datasets.0.data '[0,0,null,null]' "$TS" +TS="$(get "/api/timeseries?metric=new_installs&$LATE")" +field "new installs are live: zeros, not gaps" datasets.0.data '[2,0,0,0]' "$TS" +field "…and claim no rollup coverage" covered_through null "$TS" + echo echo "/api/breakdown — bars and pies" # machine-days, taking the largest per-event count per day so one machine's @@ -209,6 +235,7 @@ field "…so two dropped" dropped 2 "$ACT" field "window" window_days 7 "$ACT" field "daily rate, null where no cohort" datasets.0.data '[75,50,100,null,100,null,null,100,100,null]' "$ACT" field "recent cohorts flagged incomplete" incomplete_from 2026-07-04 "$ACT" +field "cohorts counted through the rollup" covered_through 2026-07-10 "$ACT" field "…and the completed ones are not" rows.2.complete true "$ACT" field "…while the last week is" rows.8.complete false "$ACT" @@ -217,6 +244,14 @@ field "…while the last week is" rows.8.complete false "$ACT" ACT="$(get "/api/activation?window=1&$RANGE")" field "a 1-day window converts fewer" activated 9 "$ACT" +# m11 (indexed 07-10) and m12 (indexed 07-09) arrived on 07-09. 07-11 onward has +# not been rolled up, so those cohorts are gaps and stay out of the totals. +ACT="$(get "/api/activation?from=2026-07-09&to=2026-07-12")" +field "uncounted cohorts stay out of the totals" installs 2 "$ACT" +field "…both counted ones converted" activated 2 "$ACT" +field "…and draw as gaps, not 0%" datasets.0.data '[100,null,null,null]' "$ACT" +field "…with no conversions claimed" rows.2.activated null "$ACT" + echo echo "/api/retention — day 0–14, denominator per day" RET="$(get "/api/retention?$RANGE")" @@ -259,6 +294,19 @@ field "no bars" labels '[]' "$EMPTY" EMPTY="$(get "/api/timeseries?metric=production_users&from=2025-01-01&to=2025-01-03")" field "still a dense axis" datasets.0.data '[0,0,0]' "$EMPTY" +echo +echo "A rollup that stops is called out" +# One machine active yesterday that no rollup has covered — what a failed nightly +# run (or a database refusing writes) leaves behind. Removed again straight after. +STALE_ID=00000000-0000-4000-8000-0000000000ff +npx wrangler d1 execute codegraph-telemetry --local \ + --command "INSERT INTO machine_days (machine_id, day, prod) VALUES ('$STALE_ID', '$YESTERDAY', 1)" >/dev/null 2>&1 +META="$(get "/api/meta")" +field "rollup behind once a day goes un-rolled" rollup_behind true "$META" +field "…and yesterday's machine is counted" machines_yesterday 1 "$META" +npx wrangler d1 execute codegraph-telemetry --local \ + --command "DELETE FROM machine_days WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 + echo printf '%d passed, %d failed\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] diff --git a/telemetry-dashboard/scripts/smoke-auth.sh b/telemetry-dashboard/scripts/smoke-auth.sh index 3fe357baec..73777cadb4 100755 --- a/telemetry-dashboard/scripts/smoke-auth.sh +++ b/telemetry-dashboard/scripts/smoke-auth.sh @@ -64,9 +64,10 @@ lacks() { # lacks fi } -echo "Seeding local D1 from the ingest worker's migration…" -npx wrangler d1 execute codegraph-telemetry --local \ - --file=../telemetry-worker/migrations/0001_init.sql >/dev/null 2>&1 +echo "Seeding local D1 from the ingest worker's migrations…" +for migration in ../telemetry-worker/migrations/*.sql; do + npx wrangler d1 execute codegraph-telemetry --local --file="$migration" >/dev/null 2>&1 +done echo "Starting wrangler dev on :${DASH_PORT}…" npx wrangler dev --port "$DASH_PORT" --ip 127.0.0.1 >"$LOG" 2>&1 & diff --git a/telemetry-dashboard/src/api.ts b/telemetry-dashboard/src/api.ts index 5b5fac3760..63b77556c6 100644 --- a/telemetry-dashboard/src/api.ts +++ b/telemetry-dashboard/src/api.ts @@ -3,10 +3,14 @@ * scoped by the same `?from=&to=` range the picker drives. * * Rules this file keeps: - * - **Rollups first.** Every panel is answered from `daily_*` / `machine_days`, - * which are kept forever. Only the activation funnel touches raw `events`, - * because "did this machine ever run an index" is not a daily aggregate — and - * that is also the only endpoint with a horizon (the retention window). + * - **Rollups only.** Every panel is answered from `daily_*`, `machine_days` and + * `machine_first_seen`, which are kept forever. No panel reads raw `events`: + * D1 runs one query at a time per database, so one slow scan there fails every + * panel queued behind it. (/api/meta reads the table's first and last day, one + * indexed lookup each.) + * - **Today is in range; uncounted days are not zeros.** Rolled-up numbers stop + * at the nightly rollup's last day and come back null after it, so a chart + * ending today draws a gap where the count has not happened yet, not a cliff. * - **Parameterized, always.** No value from the query string is ever * concatenated into SQL. Dimensions and metrics are looked up in the tables * below and rejected with a 400 if they are not there, so even the column @@ -88,9 +92,7 @@ export interface Range { /** * The range every endpoint shares. Absent params default to the last 30 days - * ending today so a bare `curl /api/summary` still answers something sensible; - * the dashboard itself always sends both, anchored on /api/meta's latest day so - * no chart ends on a day the nightly rollup has not written yet. + * ending today, which is also what the dashboard's presets send. */ function parseRange(url: URL): Range | ApiResult { const rawTo = url.searchParams.get('to'); @@ -212,43 +214,92 @@ interface MetaRow { earliest_active_day: string | null; earliest_raw_day: string | null; latest_raw_day: string | null; + machines_yesterday: number | null; } /** - * What the picker anchors on. The dashboard asks for this first and ends every - * default range on `latest_day`, because the nightly cron has not rolled up - * today yet — anchoring on the wall clock would put a phantom zero on the right - * edge of every line chart. + * How fresh the data is, and whether either writer has stopped. + * + * The page's ranges end on today; this is what tells it how much of that range + * each kind of number actually covers. Machine counts (`machine_days`, + * `machine_first_seen`) are written by the ingest worker as events arrive, so they + * run through today. Everything rolled up by the nightly cron runs through + * `latest_rollup_day` — normally yesterday. + * + * Every lookup here is a single min() or max() per subquery, on purpose: SQLite only + * answers those from an index when the aggregate stands alone, and + * `SELECT min(day), max(day) FROM events` is a full scan of the biggest table. + * + * The two flags exist because both writers have failed silently before: from + * 2026-08-11 almost no event was stored and no rollup ran, nothing errored where + * anyone would see it, and the dashboard kept presenting Aug 9 as "the latest day" + * for seven weeks. Each flag is a fact the page states, not a tuned threshold. */ async function meta(env: Env): Promise { + const now = Date.now(); + const today = utcDay(now); + const yesterday = utcDay(now - DAY_MS); const row = await env.DB.prepare( `SELECT (SELECT max(day) FROM daily_event_counts) AS latest_rollup_day, (SELECT min(day) FROM daily_event_counts) AS earliest_rollup_day, (SELECT max(day) FROM machine_days) AS latest_active_day, (SELECT min(day) FROM machine_days) AS earliest_active_day, (SELECT min(day) FROM events) AS earliest_raw_day, - (SELECT max(day) FROM events) AS latest_raw_day`, - ).first(); + (SELECT max(day) FROM events) AS latest_raw_day, + (SELECT count(*) FROM machine_days WHERE day = ?) AS machines_yesterday`, + ) + .bind(yesterday) + .first(); + + const latestRollup = row?.latest_rollup_day ?? null; + const latestRaw = row?.latest_raw_day ?? null; + const latestActive = row?.latest_active_day ?? null; + + // Nothing at all since before yesterday. (Client clocks may run a few minutes + // ahead, so the latest day can be tomorrow — that is fresh, not stale.) + const ingestStalled = latestRaw !== null && latestRaw < yesterday; + // The 00:30 UTC run rolls up yesterday, so the day before that must always be in + // by now. Only "behind" if there was activity after the last rolled-up day — a day + // nobody used codegraph would be a silent rollup, not a missed one. + const rollupBehind = + latestActive !== null && + (latestRollup === null || (latestRollup < addDays(today, -2) && latestActive > latestRollup)); - const latest = row?.latest_rollup_day ?? row?.latest_active_day ?? null; - const earliest = row?.earliest_rollup_day ?? row?.earliest_active_day ?? null; return { body: { - latest_day: latest, - earliest_day: earliest, - latest_rollup_day: row?.latest_rollup_day ?? null, - latest_active_day: row?.latest_active_day ?? null, - /** Below this day the activation funnel is blind — raw events are purged. */ + today, + /** The last day the nightly rollup covers. Kept under its old name for callers. */ + latest_day: latestRollup ?? latestActive, + earliest_day: row?.earliest_rollup_day ?? row?.earliest_active_day ?? null, + latest_rollup_day: latestRollup, + latest_active_day: latestActive, earliest_raw_day: row?.earliest_raw_day ?? null, - latest_raw_day: row?.latest_raw_day ?? null, + latest_raw_day: latestRaw, + machines_yesterday: row?.machines_yesterday ?? 0, + rollup_behind: rollupBehind, + ingest_stalled: ingestStalled, max_range_days: MAX_RANGE_DAYS, retention_days: RETENTION_DAYS, - generated_at: new Date().toISOString(), + generated_at: new Date(now).toISOString(), }, - cacheControl: CACHE_CONTROL, + // Short: this is the staleness check, and it is one indexed lookup per field. + cacheControl: 'private, max-age=60', }; } +/** + * The last day the nightly rollup has written. Rolled-up series stop here and go + * null after it — "not counted yet", which a chart draws as a gap, rather than a zero + * it would draw as a cliff. `daily_event_counts` is keyed (day, event), so this is + * one step down its primary key. + */ +const COVERAGE_SQL = 'SELECT max(day) AS day FROM daily_event_counts'; + +/** Like densify(), but days past `coveredThrough` are null: not counted yet, not zero. */ +function densifyCovered(labels: string[], byDay: Map, coveredThrough: string | null): (number | null)[] { + return labels.map((day) => (coveredThrough === null || day > coveredThrough ? null : (byDay.get(day) ?? 0))); +} + // --------------------------------------------------------------------------- // /api/summary — the big numbers // --------------------------------------------------------------------------- @@ -314,12 +365,19 @@ interface SeriesSpec { labels: [string] | [string, string]; sql: string; binds: (range: Range) => (string | number)[]; + /** + * Written by the ingest worker as events arrive rather than by the nightly + * rollup, so it runs through today instead of stopping at the rollup's last day. + */ + live?: boolean; } /** - * Every metric here reads a rollup table, so a line stays correct for days whose - * raw events are long gone. Each query returns (day, a[, b]) and is densified - * against the full day list, because a day with no rows means zero, not a gap. + * Every metric here reads a rollup table (or, if `live`, a table the ingest path + * keeps), so a line stays correct for days whose raw events are long gone. Each + * query returns (day, a[, b]) and is densified against the full day list, because + * a covered day with no rows means zero, not a gap. Days the rollup has not + * reached yet are the gap. */ const SERIES: Record = { installs_uninstalls: { @@ -341,6 +399,7 @@ const SERIES: Record = { WHERE first_day BETWEEN ? AND ? GROUP BY first_day`, binds: (r) => [r.from, r.to], + live: true, }, production_users: { title: 'Daily production users', @@ -379,9 +438,12 @@ async function timeseries(env: Env, url: URL, range: Range): Promise return fail(`unknown metric — one of: ${[...Object.keys(SERIES), 'duration_buckets'].join(', ')}`); } - const { results } = await env.DB.prepare(spec.sql) - .bind(...spec.binds(range)) - .all(); + const batch = await env.DB.batch([ + env.DB.prepare(spec.sql).bind(...spec.binds(range)), + env.DB.prepare(COVERAGE_SQL), + ]); + const results = rowsOf(batch[0]); + const coveredThrough = spec.live ? null : (firstOf<{ day: string | null }>(batch[1])?.day ?? null); const labels = dayList(range); const a = new Map(); @@ -391,19 +453,23 @@ async function timeseries(env: Env, url: URL, range: Range): Promise b.set(row.day, row.b ?? 0); } - const datasets = [{ label: spec.labels[0], data: densify(labels, a) }]; - if (spec.labels.length === 2) datasets.push({ label: spec.labels[1], data: densify(labels, b) }); + const fill = (byDay: Map): (number | null)[] => + spec.live ? densify(labels, byDay) : densifyCovered(labels, byDay, coveredThrough); + const datasets = [{ label: spec.labels[0], data: fill(a) }]; + if (spec.labels.length === 2) datasets.push({ label: spec.labels[1], data: fill(b) }); return { body: { range, metric, title: spec.title, + /** Last day with real numbers; null for a live series, which runs through today. */ + covered_through: spec.live ? null : coveredThrough, labels, datasets, rows: labels.map((day, i) => ({ day, - ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? 0])), + ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? null])), })), }, cacheControl: CACHE_CONTROL, @@ -412,14 +478,17 @@ async function timeseries(env: Env, url: URL, range: Range): Promise /** "Session run length over time": one series per duration bucket, bucket-ordered. */ async function durationBucketSeries(env: Env, range: Range): Promise { - const { results } = await env.DB.prepare( - `SELECT day, value, sum(count) AS n - FROM daily_dim_counts - WHERE dim = 'duration_bucket' AND event = 'index' AND day BETWEEN ? AND ? - GROUP BY day, value`, - ) - .bind(range.from, range.to) - .all<{ day: string; value: string; n: number }>(); + const batch = await env.DB.batch([ + env.DB.prepare( + `SELECT day, value, sum(count) AS n + FROM daily_dim_counts + WHERE dim = 'duration_bucket' AND event = 'index' AND day BETWEEN ? AND ? + GROUP BY day, value`, + ).bind(range.from, range.to), + env.DB.prepare(COVERAGE_SQL), + ]); + const results = rowsOf<{ day: string; value: string; n: number }>(batch[0]); + const coveredThrough = firstOf<{ day: string | null }>(batch[1])?.day ?? null; const labels = dayList(range); const perBucket = new Map>(); @@ -437,7 +506,7 @@ async function durationBucketSeries(env: Env, range: Range): Promise const datasets = order.map((bucket) => ({ label: bucket, - data: densify(labels, perBucket.get(bucket) ?? new Map()), + data: densifyCovered(labels, perBucket.get(bucket) ?? new Map(), coveredThrough), })); return { @@ -445,11 +514,12 @@ async function durationBucketSeries(env: Env, range: Range): Promise range, metric: 'duration_buckets', title: 'Indexing run length over time', + covered_through: coveredThrough, labels, datasets, rows: labels.map((day, i) => ({ day, - ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? 0])), + ...Object.fromEntries(datasets.map((d) => [d.label, d.data[i] ?? null])), })), }, cacheControl: CACHE_CONTROL, @@ -600,11 +670,17 @@ interface ActivationRow { * reinstalls does not re-enter the funnel, which is what makes this a * conversion rate rather than an install-event ratio. * - * The LEFT JOIN rides events_machine_day (machine_id, day) and `count(DISTINCT)` - * absorbs the fan-out from a machine that indexed many times. This is the one - * endpoint that reads raw `events`, so it is bounded by the retention window — - * `raw_events_from` tells the caller where the data actually starts, and the UI - * says so rather than drawing a cliff and calling it a drop in conversion. + * "Ran an index" is `machine_first_seen.first_index_day`, which the nightly rollup + * keeps at the earliest day each machine indexed. That makes this a range read over + * one small table. It used to be a join against raw `events` — on production volume + * ~55 s per week of cohorts, which held D1's single query lane long enough to fail + * every other panel waiting behind it. A first index day is never before the first + * day (the ingest path keeps first_day at the machine's earliest event), so "within + * the window" is just `first_index_day <= first_day + window`. + * + * Because first_index_day is rolled up, cohorts after the rollup's last day have no + * conversions counted yet. They are left out of the totals and drawn as gaps — + * counting them would show a drop in conversion that is only a lag. */ async function activation(env: Env, url: URL, range: Range): Promise { const rawWindow = url.searchParams.get('window'); @@ -615,41 +691,40 @@ async function activation(env: Env, url: URL, range: Range): Promise const batch = await env.DB.batch([ env.DB.prepare( - `SELECT f.first_day AS day, - count(DISTINCT f.machine_id) AS installs, - count(DISTINCT CASE WHEN e.machine_id IS NOT NULL THEN f.machine_id END) AS activated - FROM machine_first_seen f - LEFT JOIN events e - ON e.machine_id = f.machine_id - AND e.event = 'index' - AND e.day >= f.first_day - AND e.day <= date(f.first_day, ?) - WHERE f.first_day BETWEEN ? AND ? - GROUP BY f.first_day`, + `SELECT first_day AS day, + count(*) AS installs, + count(CASE WHEN first_index_day <= date(first_day, ?) THEN 1 END) AS activated + FROM machine_first_seen + WHERE first_day BETWEEN ? AND ? + GROUP BY first_day`, // A bound modifier string, built from an integer this function validated — // date() takes the modifier as data, so nothing is concatenated into SQL. ).bind(`+${window} days`, range.from, range.to), - env.DB.prepare(`SELECT min(day) AS raw_from, max(day) AS raw_to FROM events`), + env.DB.prepare(COVERAGE_SQL), ]); const rows = rowsOf(batch[0]); const byDay = new Map(rows.map((r) => [r.day, r])); const labels = dayList(range); - const installs = rows.reduce((n, r) => n + (r.installs ?? 0), 0); - const activated = rows.reduce((n, r) => n + (r.activated ?? 0), 0); - // Cohorts younger than the window have not finished converting yet, so their // rate is a floor, not a result. Marked rather than dropped: hiding the last // week of a conversion chart is its own kind of lie. - const boundsRow = firstOf<{ raw_from: string | null; raw_to: string | null }>(batch[1]); - const latestRaw = boundsRow?.raw_to ?? utcDay(Date.now()); - const incompleteFrom = addDays(latestRaw, -(window - 1)); + const coveredThrough = firstOf<{ day: string | null }>(batch[1])?.day ?? null; + const incompleteFrom = addDays(coveredThrough ?? utcDay(Date.now()), -(window - 1)); + const counted = (day: string): boolean => coveredThrough !== null && day <= coveredThrough; + let installs = 0; + let activated = 0; const detail = labels.map((day) => { const row = byDay.get(day); const dayInstalls = row?.installs ?? 0; + if (!counted(day)) { + return { day, installs: dayInstalls, activated: null, rate: null, complete: false }; + } const dayActivated = row?.activated ?? 0; + installs += dayInstalls; + activated += dayActivated; return { day, installs: dayInstalls, @@ -669,8 +744,8 @@ async function activation(env: Env, url: URL, range: Range): Promise rate: installs > 0 ? activated / installs : null, /** Cohorts from this day on have not had the full window to convert. */ incomplete_from: incompleteFrom, - /** Raw events start here; a range reaching further back under-counts. */ - raw_events_from: boundsRow?.raw_from ?? null, + /** The last cohort day in the totals: later cohorts have no conversions counted yet. */ + covered_through: coveredThrough, labels, datasets: [ { diff --git a/telemetry-worker/README.md b/telemetry-worker/README.md index 204074c914..296ec2251d 100644 --- a/telemetry-worker/README.md +++ b/telemetry-worker/README.md @@ -31,15 +31,19 @@ Telemetry is stored in the `codegraph-telemetry` D1 database on the same account events are written in a single `batch()` (one implicit transaction) under `ctx.waitUntil`, so the write is off the response path. It is deliberately **fail-silent**: a D1 error is logged to Workers Logs (counts only, never the payload) and the client still gets its `204`, -because clients never retry — losing a datapoint beats losing availability. Alongside the -raw rows, the worker upserts `machine_days` and `machine_first_seen`; when a batch is emptied -by the allowlist, nothing at all is written, so those tables only ever describe stored events. - -The complete schema is [`migrations/0001_init.sql`](migrations/0001_init.sql) — +because clients never retry — losing a datapoint beats losing availability. Lifecycle events +(`install`, `index`, `uninstall`) are one `events` row each. `usage_rollup` counters are +**added** into `usage_daily`, one row per machine × day × tool: clients upload a counter per +process rather than per day, and storing a row per upload is what filled the database in +August 2026. Alongside those, the worker upserts `machine_days` and `machine_first_seen`; when +a batch is emptied by the allowlist, nothing at all is written, so those tables only ever +describe stored events. + +The complete schema is [`migrations/`](migrations/) (applied in order) — checked in for the same reason this worker's source is public: it is the entire list of what gets kept, with a comment on every column and on which dashboard chart each rollup table -serves. Shape: raw sanitized `events`, `daily_*` rollups recomputed nightly, and -`machine_days` / `machine_first_seen` for retention cohorts. The dashboard reads rollups; raw +serves. Shape: raw sanitized `events`, `usage_daily` counters, `daily_*` rollups recomputed +nightly, and `machine_days` / `machine_first_seen` for retention cohorts. The dashboard reads rollups; raw events exist for drill-down and are purged past the retention window. ```bash @@ -56,29 +60,47 @@ edit to a migration that has been applied. Volume, at ~97k accepted POSTs/day: ≈30M D1 row writes/month against the 50M included on Workers Paid, plus roughly as much again once the purge reaches steady state — a delete bills like an insert, and at steady state every row written is eventually deleted, so budget ≈48M. -D1 bills a row write per index touched on top of the table row, which is why `events` carries -only two indexes; dropping `events_machine_day` is the first lever if that gets tight. Storage -is the other constraint, and it is what sets the window: raw events grow ≈74 MB/day, so 90 days -lands at ≈6.7 GB against D1's 10 GB per-database cap, while 180 days would exceed it. Full +D1 bills a row write per index touched on top of the table row, which is why `events` now +carries one index (`events_machine_day` is dropped by `0004`, once the legacy usage rows are +folded out — on the full table the drop runs past D1's per-query limit). Storage is the other +constraint, and it is what sets the window: the estimate here was ≈74 MB/day of raw events +(≈6.7 GB at 90 days against D1's 10 GB per-database cap), but it assumed one usage row per +machine × day × tool. Clients actually sent one per process — ≈3.8M rows a day by early +August — and the database hit the cap on 2026-08-11. `usage_daily` bounds that by +construction; re-measure with `npx wrangler d1 info codegraph-telemetry` before widening the +window. Full arithmetic and the remaining levers are in the migration's footer comment. +**A full database fails quietly.** D1 caps a database at 10 GB on Workers Paid (500 MB on +Free), and upgrading cannot raise it. Once full, nearly every write fails with +`D1_ERROR: Exceeded maximum DB size`: ingest stops storing events (the client never retries, +so they are lost) and the nightly rollup stops too, while deletes and reads still work, so +nothing looks broken from the outside. That ran from 2026-08-11 to October 2026. +`npx wrangler d1 info codegraph-telemetry` shows the current size. + ## Rollups & retention (nightly cron) `src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things. -**Rolls up** the day that just ended into `daily_machines`, `daily_event_counts` and -`daily_dim_counts`, then re-runs the two days before it — offline clients ship completed-day -rollups late, so a day keeps growing after it ends. The aggregation is one +**Rolls up** the day that just ended into `daily_machines`, `daily_event_counts`, +`daily_dim_counts` and `machine_first_seen.first_index_day`, then re-runs the two days before +it — offline clients ship completed-day rollups late, so a day keeps growing after it ends. +It then **catches up** on any earlier day that saw activity but never got a rollup (a +`machine_days` day with no `daily_machines` row — a night the run failed or the database +refused writes), newest first, up to 31 a night. An outage heals on the first good night +instead of leaving a hole someone has to notice. The aggregation is one `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and no event row crosses the wire. Every write overwrites the recomputed value rather than adding to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful -about — a `usage_rollup` row is a counter the client pre-aggregated, so its `count` prop is -summed rather than the rows counted; and `index.languages` / `install.targets` are unnested -with `json_each`, one row per element. Adding a breakdown is a line in `ROLLUP_STATEMENTS`, +about — usage figures are sums of `usage_daily.count` (calls), not row counts; and +`index.languages` / `install.targets` are unnested with `json_each`, one row per element. +Before rolling up a day it folds any legacy `usage_rollup` rows still in `events` (how usage +was stored before `0003`) into `usage_daily`, 50k rows per transaction, so re-running the +rollup over old days is the whole migration. Adding a breakdown is a line in `ROLLUP_STATEMENTS`, never a migration — that is what the generic `(dim, value)` shape buys. -**Purges** raw `events` older than `RETENTION_DAYS` (90, a var in `wrangler.jsonc`) in bounded -`DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are +**Purges** raw `events` and `usage_daily` rows older than `RETENTION_DAYS` (90, a var in +`wrangler.jsonc`) in bounded `DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are never purged — retention cohorts need the full history and they are two orders of magnitude smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a chart. @@ -98,6 +120,37 @@ retention window, where it would delete rows and then find no events to rebuild the response says which days it refused. Keep manual ranges to a few days at production volume; each day is a full scan of that day's events, and the request has a wall-clock budget. +### Backfilling first_index_day + +`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads, and only +the rollup writes it. Days rolled up before the migration left it NULL, and days stored before +`0003` still hold their usage as one `events` row per upload. Re-running the rollup over every +day that still has raw events fixes both — it folds that day's legacy usage rows into +`usage_daily`, sets `first_index_day`, and recomputes the day's rollups. It is idempotent, so +overlapping or repeating a range is harmless. A day with millions of legacy rows takes several +minutes, so go a day at a time: + +```bash +# One day per call, from the earliest raw event (2026-07-04) through yesterday. +for day in $(node -e 'for (let t = Date.parse("2026-07-04"); t < Date.now() - 864e5; t += 864e5) + console.log(new Date(t).toISOString().slice(0, 10))'); do + curl -sS -X POST -H "x-admin-token: $ADMIN_TOKEN" \ + "https://telemetry.getcodegraph.com/admin/rollup?day=$day"; echo +done +``` + +Move the start to the earliest day `events` still holds (`/api/meta` on the dashboard +reports it as `earliest_raw_day`). A machine whose index events were purged before the backfill stays NULL +and counts as not activated — so run it before the retention window passes those days. + +### When the dashboard says ingest stalled or the rollup is behind + +Both banners mean a writer stopped. In order: `npx wrangler tail codegraph-telemetry` (look +for `d1 write failed` / `rollup day failed` and the error text), then +`npx wrangler d1 info codegraph-telemetry` against the plan's database cap (see Storage +above). Once writes succeed again the next nightly run catches the rollup up on its own; the +events that arrived while writes were failing are gone. + ## Deploy Prereqs: the `getcodegraph.com` zone on the deploying Cloudflare account (the custom diff --git a/telemetry-worker/migrations/0002_first_index_day.sql b/telemetry-worker/migrations/0002_first_index_day.sql new file mode 100644 index 0000000000..37b2910a1f --- /dev/null +++ b/telemetry-worker/migrations/0002_first_index_day.sql @@ -0,0 +1,19 @@ +-- codegraph telemetry — the first day each machine ran an index. +-- +-- Serves the install → first-index activation funnel. Until this column, the +-- dashboard answered that funnel by joining every cohort machine against raw +-- `events`, which on production volume took ~55 s for ONE week of cohorts — long +-- enough to stall D1 (one query at a time per database) and fail every other +-- panel queued behind it. With the first index day stored next to the first-seen +-- day, the funnel is a range read over `machine_first_seen` alone. +-- +-- It is also what lets the funnel outlive the raw-event retention purge, like +-- every other panel: the nightly cron fills it from `events` while they exist. +-- +-- Written by the nightly rollup (telemetry-worker/src/rollup.ts), never by the +-- ingest path, and only ever lowered: NULL means "has not indexed (yet)". +-- Existing rows start NULL — backfill them by re-running the rollup over the +-- retained days (see "Backfilling first_index_day" in telemetry-worker/README.md). +-- +-- ADD COLUMN with no default is a schema-only change in SQLite: no row is rewritten. +ALTER TABLE machine_first_seen ADD COLUMN first_index_day TEXT; diff --git a/telemetry-worker/migrations/0003_usage_daily.sql b/telemetry-worker/migrations/0003_usage_daily.sql new file mode 100644 index 0000000000..3cfaa07357 --- /dev/null +++ b/telemetry-worker/migrations/0003_usage_daily.sql @@ -0,0 +1,35 @@ +-- codegraph telemetry — usage counters, one row per machine × day × tool. +-- +-- `usage_rollup` events were stored one row per event in `events`. The client was +-- meant to send one pre-aggregated counter per machine × day × tool, but it only +-- aggregates within a single process: every short-lived codegraph process (each +-- `serve` launch, CLI command or prompt hook) uploads its own `count: 1` line. By +-- 2026-08-10 that was ~3.8M rows a day, 98% of the database, and the database hit +-- D1's 10 GB cap — from 2026-08-11 nearly every write failed and was lost. +-- +-- This table makes storage independent of how a client batches: the ingest worker +-- ADDS each counter into the row for its key instead of inserting a new row, so a +-- machine that uploads the same counter 80,000 times still owns one row. Same +-- fields as the `usage_rollup` props plus the envelope the rollup breaks down by; +-- nothing new is collected. +-- +-- Every key column is NOT NULL with '' for "absent": a primary key treats NULLs as +-- distinct, which would quietly turn the upsert back into an insert. The envelope +-- columns are in the key so a machine that upgrades mid-day keeps exact per-version +-- counts. Purged at the same retention window as `events`. +CREATE TABLE usage_daily ( + day TEXT NOT NULL, -- UTC YYYY-MM-DD the counts belong to + machine_id TEXT NOT NULL, -- random UUIDv4, client-minted + kind TEXT NOT NULL, -- mcp_tool | cli_command + name TEXT NOT NULL, -- tool or command name + client_name TEXT NOT NULL DEFAULT '', + client_version TEXT NOT NULL DEFAULT '', + codegraph_version TEXT NOT NULL DEFAULT '', + os TEXT NOT NULL DEFAULT '', + arch TEXT NOT NULL DEFAULT '', + node_major TEXT NOT NULL DEFAULT '', + count INTEGER NOT NULL DEFAULT 0, -- calls + error_count INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major) +) WITHOUT ROWID; diff --git a/telemetry-worker/migrations/0004_drop_events_machine_day.sql b/telemetry-worker/migrations/0004_drop_events_machine_day.sql new file mode 100644 index 0000000000..3a637e8523 --- /dev/null +++ b/telemetry-worker/migrations/0004_drop_events_machine_day.sql @@ -0,0 +1,11 @@ +-- codegraph telemetry — drop the per-machine index on `events`. +-- +-- Its only query reader was the dashboard's old activation join, which now reads +-- machine_first_seen.first_index_day. Dropping it saves a row write per stored event, +-- and it was the first storage lever the 0001 footer names. +-- +-- A separate migration, applied AFTER the legacy usage rows are folded out of +-- `events` (see "Backfilling first_index_day" in README.md): with ~30M rows in the +-- table, dropping the index ran past D1's per-query time limit and was rolled back. +-- On the folded table it is quick. +DROP INDEX IF EXISTS events_machine_day; diff --git a/telemetry-worker/scripts/backfill-rollup.sh b/telemetry-worker/scripts/backfill-rollup.sh new file mode 100755 index 0000000000..4f9e24bfbe --- /dev/null +++ b/telemetry-worker/scripts/backfill-rollup.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +# Re-runs the nightly rollup over a range of days against the DEPLOYED ingest worker, +# one day per request. Each day: legacy usage rows still in `events` are folded into +# `usage_daily`, `first_index_day` is set, and the day's rollups are recomputed. All of +# it is idempotent, so a range can be repeated or overlapped safely. +# +# ./scripts/backfill-rollup.sh 2026-07-04 2026-10-02 [token-file] +# +# The admin token comes from the token file when one is given, else from $ADMIN_TOKEN. +# Stops at the first day that does not answer 200, so a rerun can start from there. +set -euo pipefail + +FROM=${1:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]} +TO=${2:?usage: backfill-rollup.sh FROM_DAY TO_DAY [TOKEN_FILE]} +if [ -n "${3:-}" ]; then ADMIN_TOKEN=$(tr -d '\n' < "$3"); fi +: "${ADMIN_TOKEN:?set ADMIN_TOKEN or pass a token file}" +BASE="${ROLLUP_BASE:-https://telemetry.getcodegraph.com}" + +days=$(node -e ' +const [from, to] = process.argv.slice(1); +for (let t = Date.parse(`${from}T00:00:00Z`); t <= Date.parse(`${to}T00:00:00Z`); t += 864e5) + console.log(new Date(t).toISOString().slice(0, 10));' "$FROM" "$TO") + +for day in $days; do + started=$(date +%s) + # A day with millions of legacy rows takes minutes; never let one hang forever. + response=$(curl -sS --max-time 1800 -X POST -H "x-admin-token: $ADMIN_TOKEN" \ + -w $'\n%{http_code}' "$BASE/admin/rollup?day=$day") || { echo "$day request failed"; exit 1; } + code=${response##*$'\n'} + body=${response%$'\n'*} + printf '%s %s %ss %s\n' "$day" "$code" "$(( $(date +%s) - started ))" "$body" + [ "$code" = 200 ] || { echo "stopping at $day"; exit 1; } +done diff --git a/telemetry-worker/scripts/smoke-cutover.sh b/telemetry-worker/scripts/smoke-cutover.sh index 47aed75370..f0dff12443 100755 --- a/telemetry-worker/scripts/smoke-cutover.sh +++ b/telemetry-worker/scripts/smoke-cutover.sh @@ -233,11 +233,13 @@ ts "indexing activity" indexing_activity '[3]' '[3]' ts "tool calls (sums the prop)" tool_calls '[20]' '[2]' MET=$(api "meta") -is "meta anchors on the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" +is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)" -# The funnel is the one panel that reads RAW events rather than a rollup, so it is -# also the one the retention purge can blind — worth pinning that it works today. +# The funnel reads machine_first_seen.first_index_day, which only the nightly +# rollup writes (from raw `index` events). This is the seam that pins it: ingest +# writes the events, the rollup sets the column, the dashboard reads it. If the +# rollup stopped setting it, "activated" here would read 0. # # Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine # that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never @@ -247,7 +249,7 @@ ACT=$(api "activation?$RANGE&window=1") is "funnel counts new machines, not install events" 3 "$(jget "$ACT" installs)" is "all three indexed within the window" 3 "$(jget "$ACT" activated)" is "nobody dropped out" 0 "$(jget "$ACT" dropped)" -is "raw-event floor is reported to the caller" "$DAY" "$(jget "$ACT" raw_events_from)" +is "cohorts are counted through the rolled-up day" "$DAY" "$(jget "$ACT" covered_through)" is "retention endpoint answers" 200 \ "$(curl -s -o /dev/null -w '%{http_code}' -b "$JAR" "$DASH/api/retention?$RANGE")" diff --git a/telemetry-worker/scripts/smoke-ingest.sh b/telemetry-worker/scripts/smoke-ingest.sh index f9dd4795be..458123443a 100755 --- a/telemetry-worker/scripts/smoke-ingest.sh +++ b/telemetry-worker/scripts/smoke-ingest.sh @@ -37,6 +37,9 @@ q() { # --------------------------------------------------------------------------- # Boot # --------------------------------------------------------------------------- +echo "applying migrations to local D1" +npx wrangler d1 migrations apply "$DB" --local >/dev/null 2>&1 + echo "booting wrangler dev on :$PORT" npx wrangler dev --port "$PORT" >/tmp/cg-smoke-ingest.log 2>&1 & DEV_PID=$! @@ -83,26 +86,31 @@ is "oversized body → 413" 413 "$(post "$BIG")" echo echo "ingest" -M_OK=$(uuid); M_DROP=$(uuid); M_CI=$(uuid); M_BACK=$(uuid) +M_OK=$(uuid); M_DROP=$(uuid); M_CI=$(uuid); M_BACK=$(uuid); M_USE=$(uuid) TODAY=$(date -u +%F) +# Client timestamps older than 30 days are rejected at ingest, so the dated events +# here are relative to today — fixed dates rot into "too old" a month after writing. +day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; } +RECENT_DAY=$(day_ago 2) +BACK_DAY=$(day_ago 9) # Three valid events + one unknown event + unknown/malformed props that must be stripped. is "valid batch → 204" 204 "$(post "$(node -e ' -const [m] = process.argv.slice(1); +const [m, day] = process.argv.slice(1); process.stdout.write(JSON.stringify({ machine_id: m, codegraph_version: "1.5.0", os: "darwin", arch: "arm64", node_major: 22, ci: false, schema_version: 1, secret_field: "must not be stored", events: [ - { event: "install", ts: "2026-07-27T10:00:00Z", + { event: "install", ts: `${day}T10:00:00Z`, props: { scope: "local", kind: "fresh", targets: ["claude", "cursor"], nope: "strip me" } }, - { event: "index", ts: "2026-07-27T10:01:00Z", + { event: "index", ts: `${day}T10:01:00Z`, props: { languages: ["typescript"], file_count_bucket: "100-1k", duration_bucket: "bogus-bucket", repo_path: "/Users/someone/secret" } }, { event: "usage_rollup", props: { kind: "mcp_tool", name: "codegraph_explore", count: 12, client_name: "Claude Code" } }, { event: "not_an_event", props: { count: 1 } }, ], -}));' "$M_OK")")" +}));' "$M_OK" "$RECENT_DAY")")" # Nothing survives the allowlist: unknown event + usage_rollup missing required props. is "all-dropped batch → 204" 204 "$(post "$(node -e ' @@ -113,6 +121,26 @@ process.stdout.write(JSON.stringify({ machine_id: m, os: "linux", events: [ { event: "install", props: { scope: "local" } }, ]}));' "$M_DROP")")" +# Usage counters ADD into one row per machine × day × tool — what old clients send +# is the same `count: 1` line once per process, many times over. Two batches: the +# first repeats one counter three times (one of them an error) beside a second tool, +# the second sends the first counter again. +USAGE_A=$(node -e ' +const [m] = process.argv.slice(1); +const explore = (errors) => ({ event: "usage_rollup", + props: { kind: "mcp_tool", name: "codegraph_explore", count: 1, error_count: errors, client_name: "Claude Code" } }); +process.stdout.write(JSON.stringify({ machine_id: m, os: "darwin", events: [ + explore(0), explore(1), explore(0), + { event: "usage_rollup", props: { kind: "cli_command", name: "index", count: 2, error_count: 0 } }, +]}));' "$M_USE") +USAGE_B=$(node -e ' +const [m] = process.argv.slice(1); +process.stdout.write(JSON.stringify({ machine_id: m, os: "darwin", events: [ + { event: "usage_rollup", props: { kind: "mcp_tool", name: "codegraph_explore", count: 1, error_count: 0, client_name: "Claude Code" } }, +]}));' "$M_USE") +is "usage batch → 204" 204 "$(post "$USAGE_A")" +is "the same counter again → 204" 204 "$(post "$USAGE_B")" + # NOTE: build every body into a variable first. Escaped quotes nested inside # "$(post "…\"…\"…")" break out of the quoting context and get brace-expanded. index_batch() { # [ci] [ts] @@ -130,8 +158,8 @@ is "ci batch → 204" 204 "$(post "$CI_ON")" is "same machine, non-ci → 204" 204 "$(post "$CI_OFF")" # A late offline buffer arriving second must move first_day EARLIER, never later. -RECENT=$(index_batch "$M_BACK" "" 2026-07-27T09:00:00Z) -BACKDATED=$(index_batch "$M_BACK" "" 2026-07-20T09:00:00Z) +RECENT=$(index_batch "$M_BACK" "" "${RECENT_DAY}T09:00:00Z") +BACKDATED=$(index_batch "$M_BACK" "" "${BACK_DAY}T09:00:00Z") is "recent batch → 204" 204 "$(post "$RECENT")" is "backdated batch → 204" 204 "$(post "$BACKDATED")" @@ -145,17 +173,22 @@ sleep 1 # and let miniflare release the local sqlite file echo echo "stored rows" -is "3 of 4 events stored (unknown dropped)" 3 "$(q "select count(*) from events where machine_id='$M_OK'")" +is "lifecycle events land in events (usage and the unknown event do not)" 2 \ + "$(q "select count(*) from events where machine_id='$M_OK'")" +is "the usage counter lands in usage_daily" "codegraph_explore|mcp_tool|12|0|Claude Code" \ + "$(q "select name||'|'||kind||'|'||count||'|'||error_count||'|'||client_name from usage_daily where machine_id='$M_OK'")" +is "…carrying the batch's envelope" "1.5.0|darwin|arm64|22" \ + "$(q "select codegraph_version||'|'||os||'|'||arch||'|'||node_major from usage_daily where machine_id='$M_OK'")" is "all-dropped batch stored nothing" 0 "$(q "select count(*) from events where machine_id='$M_DROP'")" is "…and no machine_days row for it" 0 "$(q "select count(*) from machine_days where machine_id='$M_DROP'")" is "envelope columns land in their own columns" "darwin|arm64|22|0|1.5.0" \ "$(q "select os||'|'||arch||'|'||node_major||'|'||ci||'|'||codegraph_version from events where machine_id='$M_OK' limit 1")" -is "day derived from the client ts" "2026-07-27" \ +is "day derived from the client ts" "$RECENT_DAY" \ "$(q "select day from events where machine_id='$M_OK' and event='install'")" is "day falls back to received_at when ts is absent" "$TODAY" \ - "$(q "select day from events where machine_id='$M_OK' and event='usage_rollup'")" + "$(q "select day from usage_daily where machine_id='$M_OK'")" is "ts is NULL when the client sent none" 1 \ - "$(q "select ts is null from events where machine_id='$M_OK' and event='usage_rollup'")" + "$(q "select ts is null from events where machine_id='$M_CI' limit 1")" is "allowlisted props stored" "local|fresh|2" \ "$(q "select json_extract(props,'\$.scope')||'|'||json_extract(props,'\$.kind')||'|'||json_array_length(props,'\$.targets') from events where machine_id='$M_OK' and event='install'")" is "unknown prop stripped" 0 \ @@ -167,7 +200,7 @@ is "path-shaped prop stripped" 0 \ is "unknown envelope field stored nowhere" 0 \ "$(q "select count(*) from events where props like '%must not be stored%'")" -# The valid batch mixes ts-dated events (2026-07-27) with an undated rollup (today), +# The valid batch mixes ts-dated events ($RECENT_DAY) with an undated rollup (today), # so it legitimately spans two days and must produce a machine_days row for each. is "machine_days: one row per distinct day in the batch" 2 \ "$(q "select count(*) from machine_days where machine_id='$M_OK'")" @@ -175,13 +208,24 @@ is "machine_days: non-ci machine is production" 1 \ "$(q "select min(prod) from machine_days where machine_id='$M_OK'")" is "machine_days: a later non-ci batch flips the day to production" 1 \ "$(q "select prod from machine_days where machine_id='$M_CI'")" -is "machine_days: each backdated batch gets its own day" "2026-07-20,2026-07-27" \ +is "machine_days: each backdated batch gets its own day" "$BACK_DAY,$RECENT_DAY" \ "$(q "select group_concat(day) from (select day from machine_days where machine_id='$M_BACK' order by day)")" -is "machine_first_seen recorded" "2026-07-27" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" -is "machine_first_seen only moves earlier" "2026-07-20" \ +is "machine_first_seen recorded" "$RECENT_DAY" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" +is "machine_first_seen only moves earlier" "$BACK_DAY" \ "$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")" +echo +echo "usage counters add up instead of piling up" +is "no usage row ever lands in events" 0 "$(q "select count(*) from events where event='usage_rollup' and machine_id='$M_USE'")" +is "one row per tool, however many uploads" 2 "$(q "select count(*) from usage_daily where machine_id='$M_USE'")" +is "repeats within a batch and across batches are summed" "4/1" \ + "$(q "select count||'/'||error_count from usage_daily where machine_id='$M_USE' and name='codegraph_explore'")" +is "…and a different tool keeps its own count" "2/0" \ + "$(q "select count||'/'||error_count from usage_daily where machine_id='$M_USE' and name='index'")" +is "an absent client name is stored as empty, not NULL (it is part of the key)" "" \ + "$(q "select client_name from usage_daily where machine_id='$M_USE' and name='index'")" + echo echo "$pass passed, $fail failed" [ "$fail" -eq 0 ] diff --git a/telemetry-worker/scripts/smoke-rollup.sh b/telemetry-worker/scripts/smoke-rollup.sh index c0bfbef0b5..294b4378f8 100755 --- a/telemetry-worker/scripts/smoke-rollup.sh +++ b/telemetry-worker/scripts/smoke-rollup.sh @@ -15,6 +15,10 @@ # raw events are already purged # * the purge deletes only rows past the window, and leaves machine_days / # machine_first_seen alone +# * each machine's first_index_day is its earliest index, and survives the purge +# * the cron catches up on a day that saw activity but was never rolled up +# * usage comes from usage_daily; a legacy usage row still in `events` is folded +# into it (counts added, row deleted) before the day is rolled up # * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one # # Re-runnable: it wipes its own synthetic days first, and they are chosen to sit @@ -39,10 +43,13 @@ is() { [ "$2" = "$3" ] && ok "$1" || bad "$1" "$2" "$3"; } day_ago() { node -e 'console.log(new Date(Date.now()-process.argv[1]*864e5).toISOString().slice(0,10))' "$1"; } # Synthetic days. MAIN/RESET sit inside the 90-day retention window but outside the -# cron's 3-day lookback; OLD sits past the window so the purge takes it. +# cron's 3-day lookback; OLD sits past the window so the purge takes it. MISSED is +# never rolled up by hand — the cron has to find it on its own. DAY_MAIN=$(day_ago 40) DAY_RESET=$(day_ago 41) +DAY_MISSED=$(day_ago 30) DAY_OLD=$(day_ago 200) +DAY_ANCIENT=$(day_ago 199) # a usage counter past the window, for the usage purge CUTOFF=$(day_ago 90) # First column of the first row of a query against the LOCAL D1 state. @@ -91,18 +98,18 @@ shutdown() { echo "applying migrations to local D1" npx wrangler d1 migrations apply "$DB" --local >/dev/null 2>&1 -echo "seeding $DAY_MAIN / $DAY_RESET / $DAY_OLD" +echo "seeding $DAY_MAIN / $DAY_RESET / $DAY_MISSED / $DAY_OLD" node -e ' -const [main, reset, old, seedFile] = process.argv.slice(1); +const [main, reset, old, missed, ancient, seedFile] = process.argv.slice(1); const sq = (v) => `'"'"'${String(v).replace(/'"'"'/g, "'"'"''"'"'")}'"'"'`; const M = ["11111111-1111-4111-8111-111111111111", "22222222-2222-4222-8222-222222222222", "33333333-3333-4333-8333-333333333333", "44444444-4444-4444-8444-444444444444", - "99999999-9999-4999-8999-999999999999"]; + "99999999-9999-4999-8999-999999999999", "55555555-5555-4555-8555-555555555555"]; const out = []; // Re-runnable: every table this script touches, scoped to its own synthetic days. -for (const t of ["events", "daily_event_counts", "daily_dim_counts", "daily_machines", "machine_days"]) { - out.push(`DELETE FROM ${t} WHERE day IN (${[main, reset, old].map(sq).join(", ")});`); +for (const t of ["events", "daily_event_counts", "daily_dim_counts", "daily_machines", "machine_days", "usage_daily"]) { + out.push(`DELETE FROM ${t} WHERE day IN (${[main, reset, old, missed, ancient].map(sq).join(", ")});`); } out.push(`DELETE FROM machine_first_seen WHERE machine_id IN (${M.map(sq).join(", ")});`); @@ -110,7 +117,6 @@ out.push(`DELETE FROM machine_first_seen WHERE machine_id IN (${M.map(sq).join(" const rows = [ [main, M[0], "install", "darwin", "arm64", "1.5.0", 22, 0, {targets:["claude","cursor"], scope:"local", kind:"fresh"}], [main, M[0], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["typescript","go"], file_count_bucket:"100-1k", duration_bucket:"10-60s"}], - [main, M[0], "usage_rollup", "darwin", "arm64", "1.5.0", 22, 0, {kind:"mcp_tool", name:"codegraph_explore", count:10, error_count:2, client_name:"Claude Code"}], [main, M[1], "index", "darwin", "x64", "1.5.0", 20, 0, {languages:["typescript"], file_count_bucket:"1k-10k", duration_bucket:"10-60s"}], [main, M[1], "usage_rollup", "darwin", "x64", "1.5.0", 20, 0, {kind:"mcp_tool", name:"codegraph_explore", count:5, error_count:0, client_name:"Cursor"}], [main, M[2], "install", "linux", "x64", "1.4.1", 22, 1, {targets:["claude"], scope:"global", kind:"upgrade"}], @@ -118,6 +124,8 @@ const rows = [ [reset, M[3], "index", "darwin", "arm64", "1.5.0", 22, 0, {languages:["python"], file_count_bucket:"<100", duration_bucket:"<10s"}], [old, M[4], "install", "linux", "x64", "1.0.0", 20, 0, {targets:["codex"], scope:"local", kind:"fresh"}], [old, M[4], "index", "linux", "x64", "1.0.0", 20, 0, {languages:["rust"], file_count_bucket:"<100", duration_bucket:"<10s"}], + [missed, M[5], "install", "win32", "x64", "1.5.0", 22, 0, {targets:["claude"], scope:"local", kind:"fresh"}], + [missed, M[5], "index", "win32", "x64", "1.5.0", 22, 0, {languages:["go"], file_count_bucket:"<100", duration_bucket:"<10s"}], ]; for (const [day, m, event, os, arch, version, node, ci, props] of rows) { out.push(`INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, node_major, ci, schema_version, props) @@ -125,8 +133,19 @@ for (const [day, m, event, os, arch, version, node, ci, props] of rows) { ${sq(version)}, ${sq(os)}, ${sq(arch)}, ${node}, ${ci}, 1, ${sq(JSON.stringify(props))});`); } +// Usage for M[0] arrives the way the ingest path writes it now: added into usage_daily. +// Usage for M[1] (in the rows above) is a LEGACY row in `events`, as stored before +// migrations/0003 — the rollup has to fold it in before counting. +out.push(`INSERT INTO usage_daily (day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count) + VALUES (${[main, M[0], "mcp_tool", "codegraph_explore", "Claude Code", "", + "1.5.0", "darwin", "arm64", "22"].map(sq).join(", ")}, 10, 2);`); +// A usage counter past the retention window: the purge must take it. +out.push(`INSERT INTO usage_daily (day, machine_id, kind, name, count, error_count) + VALUES (${[ancient, M[4], "cli_command", "index"].map(sq).join(", ")}, 3, 0);`); + // What the ingest path would have written alongside those events. -for (const [m, day, prod] of [[M[0], main, 1], [M[1], main, 1], [M[2], main, 0], [M[3], reset, 1], [M[4], old, 1]]) { +for (const [m, day, prod] of [[M[0], main, 1], [M[1], main, 1], [M[2], main, 0], [M[3], reset, 1], [M[4], old, 1], [M[5], missed, 1]]) { out.push(`INSERT INTO machine_days (machine_id, day, prod) VALUES (${sq(m)}, ${sq(day)}, ${prod});`); out.push(`INSERT INTO machine_first_seen (machine_id, first_day) VALUES (${sq(m)}, ${sq(day)}) ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day);`); @@ -137,7 +156,7 @@ out.push(`INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) VALUES (${sq(reset)}, ${sq("index")}, ${sq("obsolete_dim")}, ${sq("stale")}, 99, 99);`); require("fs").writeFileSync(seedFile, out.join("\n")); -' "$DAY_MAIN" "$DAY_RESET" "$DAY_OLD" "$SEED_SQL" +' "$DAY_MAIN" "$DAY_RESET" "$DAY_OLD" "$DAY_MISSED" "$DAY_ANCIENT" "$SEED_SQL" npx wrangler d1 execute "$DB" --local --file "$SEED_SQL" >/dev/null # --------------------------------------------------------------------------- @@ -173,7 +192,8 @@ is "rollup $DAY_MAIN again → 200" 200 "$(roll "day=$DAY_MAIN")" is "rollup $DAY_OLD, whose events are still there → 200" 200 "$(roll "day=$DAY_OLD")" is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")" -# The cron body: rolls up the last three days and purges everything past the window. +# The cron body: rolls up the last three days, catches up $DAY_MISSED (active, never +# rolled up), and purges everything past the window. is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")" sleep 2 @@ -184,6 +204,12 @@ is "…and reports the reset it refused to run" "[\"$DAY_OLD\"]" \ "$(curl -s -X POST -H "x-admin-token: $TOKEN" "$BASE/admin/rollup?day=$DAY_OLD&reset=1" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>console.log(JSON.stringify(JSON.parse(s).reset_ignored)))')" +# first_index_day only ever moves earlier. Give M[1] an index day before $DAY_MAIN, +# then re-roll $DAY_MAIN (where it indexed again): the earlier day must stand. +q "update machine_first_seen set first_index_day='$DAY_RESET' + where machine_id='22222222-2222-4222-8222-222222222222'" >/dev/null +is "re-roll $DAY_MAIN over an earlier first index day → 200" 200 "$(roll "day=$DAY_MAIN")" + shutdown # --------------------------------------------------------------------------- @@ -239,10 +265,19 @@ echo "cross-check against the raw events" is "machines matches count(distinct machine_id)" \ "$(q "select count(distinct machine_id) from events where day='$DAY_MAIN' and event='index'")" \ "$(q "select machines from daily_event_counts where day='$DAY_MAIN' and event='index'")" -is "usage count matches sum(props.count)" \ - "$(q "select sum(json_extract(props,'\$.count')) from events where day='$DAY_MAIN' and event='usage_rollup'")" \ +is "usage count matches sum(usage_daily.count)" \ + "$(q "select sum(count) from usage_daily where day='$DAY_MAIN'")" \ "$(q "select count from daily_event_counts where day='$DAY_MAIN' and event='usage_rollup'")" +echo +echo "legacy usage rows" +is "folded out of events" 0 "$(q "select count(*) from events where day='$DAY_MAIN' and event='usage_rollup'")" +is "…into usage_daily, counts and client intact" "5|0|Cursor|darwin|x64|20" \ + "$(q "select count||'|'||error_count||'|'||client_name||'|'||os||'|'||arch||'|'||node_major from usage_daily + where day='$DAY_MAIN' and machine_id='22222222-2222-4222-8222-222222222222'")" +is "…and re-rolling the day adds nothing (no double count)" 15 \ + "$(q "select sum(count) from usage_daily where day='$DAY_MAIN'")" + echo echo "reset" is "?reset=1 drops a rollup row whose dimension no longer exists" 0 \ @@ -255,12 +290,38 @@ echo echo "retention purge" is "raw events past the window are gone" 0 "$(q "select count(*) from events where day='$DAY_OLD'")" is "nothing older than the cutoff survives" 0 "$(q "select count(*) from events where day<'$CUTOFF'")" -is "events inside the window are untouched" 7 "$(q "select count(*) from events where day='$DAY_MAIN'")" +# 6 seeded, minus the legacy usage row the rollup folded into usage_daily. +is "events inside the window are untouched" 5 "$(q "select count(*) from events where day='$DAY_MAIN'")" +is "usage counters past the window are purged" 0 "$(q "select count(*) from usage_daily where day='$DAY_ANCIENT'")" +is "…and inside it they stay" 2 "$(q "select count(*) from usage_daily where day='$DAY_MAIN'")" is "machine_days is NOT purged (retention cohorts need it)" 1 \ "$(q "select count(*) from machine_days where day='$DAY_OLD'")" is "machine_first_seen is NOT purged" "$DAY_OLD" \ "$(q "select first_day from machine_first_seen where machine_id='99999999-9999-4999-8999-999999999999'")" +echo +echo "first_index_day — the activation funnel's input" +is "machines that indexed get that day" "$DAY_MAIN" \ + "$(q "select first_index_day from machine_first_seen where machine_id='11111111-1111-4111-8111-111111111111'")" +is "a machine that never indexed stays NULL" "null" \ + "$(q "select coalesce(first_index_day, 'null') from machine_first_seen where machine_id='33333333-3333-4333-8333-333333333333'")" +is "set on a reset run too" "$DAY_RESET" \ + "$(q "select first_index_day from machine_first_seen where machine_id='44444444-4444-4444-8444-444444444444'")" +is "outlives the purge of the events it came from" "$DAY_OLD" \ + "$(q "select first_index_day from machine_first_seen where machine_id='99999999-9999-4999-8999-999999999999'")" +is "only ever lowered: re-rolling a later index day leaves an earlier one" "$DAY_RESET" \ + "$(q "select first_index_day from machine_first_seen where machine_id='22222222-2222-4222-8222-222222222222'")" + +echo +echo "the cron catches up on a day nobody rolled up" +is "$DAY_MISSED got its daily_machines row" "1/1" \ + "$(q "select machines||'/'||prod_machines from daily_machines where day='$DAY_MISSED'")" +is "…and its event counts" "1/1" \ + "$(q "select count||'/'||machines from daily_event_counts where day='$DAY_MISSED' and event='index'")" +is "…and its breakdowns" "1/1" "$(dim "$DAY_MISSED" index language go)" +is "…and its first index day" "$DAY_MISSED" \ + "$(q "select first_index_day from machine_first_seen where machine_id='55555555-5555-4555-8555-555555555555'")" + echo echo "rollups outlive the events they came from" is "daily_event_counts survives the purge" "1/1" \ diff --git a/telemetry-worker/src/index.ts b/telemetry-worker/src/index.ts index 801d9a783e..9a6efa3394 100644 --- a/telemetry-worker/src/index.ts +++ b/telemetry-worker/src/index.ts @@ -4,7 +4,7 @@ * This file is public on purpose: it is the exact code that receives codegraph's * anonymous usage telemetry, so anyone can audit what is (and is not) stored. * The schema contract lives in docs/design/telemetry.md; the storage schema — the - * complete list of what is kept — is migrations/0001_init.sql. + * complete list of what is kept — is migrations/. * * Guarantees enforced here: * - strict allowlist: unknown events are dropped, unknown properties are stripped @@ -39,7 +39,7 @@ sent; the client IP is never read or stored; the machine ID is a random UUID the client mints locally and can delete at any time. Accepted events are stored in our own database on Cloudflare (D1) and are never forwarded to any third-party analytics vendor. The stored schema is the complete list of what is kept: -https://github.com/colbymchenry/codegraph/blob/main/telemetry-worker/migrations/0001_init.sql +https://github.com/colbymchenry/codegraph/tree/main/telemetry-worker/migrations Individual events are deleted after ${keepDays} days. What outlives them: anonymous daily totals (counts per day of things like operating system, version and language), @@ -204,10 +204,54 @@ const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VAL const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`; +// usage_rollup counters ADD into one row per machine × day × tool (migrations/0003): +// clients upload the same counter many times over — once per process — so storing +// a row per upload grew without bound. +const UPSERT_USAGE = `INSERT INTO usage_daily ( + day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + ON CONFLICT (day, machine_id, kind, name, client_name, client_version, codegraph_version, os, arch, node_major) + DO UPDATE SET count = usage_daily.count + excluded.count, + error_count = usage_daily.error_count + excluded.error_count`; + +interface UsageCounter { + day: string; + kind: string; + name: string; + clientName: string; + clientVersion: string; + count: number; + errors: number; +} + +/** + * Folds a batch's usage_rollup events into one counter per day × tool × client. + * Old clients send up to 100 copies of the same `count: 1` line per request; this + * turns them into one upsert instead of a hundred. + */ +function foldUsage(events: StoredEvent[], receivedAt: string): UsageCounter[] { + const byKey = new Map(); + for (const e of events) { + const day = (e.ts ?? receivedAt).slice(0, 10); + const kind = String(e.props.kind); + const name = String(e.props.name); + const clientName = typeof e.props.client_name === 'string' ? e.props.client_name : ''; + const clientVersion = typeof e.props.client_version === 'string' ? e.props.client_version : ''; + const key = [day, kind, name, clientName, clientVersion].join('\u0000'); + const counter = byKey.get(key) ?? { day, kind, name, clientName, clientVersion, count: 0, errors: 0 }; + counter.count += typeof e.props.count === 'number' ? e.props.count : 0; + counter.errors += typeof e.props.error_count === 'number' ? e.props.error_count : 0; + byKey.set(key, counter); + } + return [...byKey.values()]; +} + /** - * Persist a sanitized batch: one `events` row per event, plus the machine×day and - * first-seen bookkeeping the dashboard's retention/activation panels need. One D1 - * `batch()` = one implicit transaction = one round trip. + * Persist a sanitized batch: one `events` row per lifecycle event, usage counters + * added into `usage_daily`, plus the machine×day and first-seen bookkeeping the + * dashboard's retention/activation panels need. One D1 `batch()` = one implicit + * transaction = one round trip. * * Fail-silent by design: the client treats every response as final and never retries, * so a failed write loses a datapoint rather than costing availability. The error is @@ -236,7 +280,33 @@ async function writeToD1( // machine_days gets one row per distinct day rather than one per batch. const days = new Set(); + const usage = batch.filter((e) => e.event === 'usage_rollup'); + if (usage.length > 0) { + const upsertUsage = env.DB.prepare(UPSERT_USAGE); + const [version, os, arch, nodeMajor] = envelopeCols; + for (const u of foldUsage(usage, receivedAt)) { + days.add(u.day); + stmts.push( + upsertUsage.bind( + u.day, + machineId, + u.kind, + u.name, + u.clientName, + u.clientVersion, + version ?? '', + os ?? '', + arch ?? '', + nodeMajor === null ? '' : String(nodeMajor), + u.count, + u.errors, + ), + ); + } + } + for (const e of batch) { + if (e.event === 'usage_rollup') continue; const day = (e.ts ?? receivedAt).slice(0, 10); days.add(day); stmts.push( diff --git a/telemetry-worker/src/rollup.ts b/telemetry-worker/src/rollup.ts index abd1671c98..87072a59a9 100644 --- a/telemetry-worker/src/rollup.ts +++ b/telemetry-worker/src/rollup.ts @@ -6,11 +6,13 @@ * * Two jobs, both driven by the cron trigger in wrangler.jsonc (00:30 UTC daily): * - * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts` - * and `daily_dim_counts` — plus the two days before it, because clients buffer - * offline and ship completed-day rollups late, so a day keeps growing after it - * ends. Every write is an upsert that OVERWRITES the recomputed value rather than - * adding to it, so re-running a day is a no-op and never double-counts. + * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`, + * `daily_dim_counts` and `machine_first_seen.first_index_day` — plus the two days + * before it, because clients buffer offline and ship completed-day rollups late, so + * a day keeps growing after it ends, plus any earlier day a failed run missed. Every + * write is an upsert that OVERWRITES the recomputed value rather than adding to it + * (or, for first_index_day, only ever lowers it), so re-running a day is a no-op + * and never double-counts. * * 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are * kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and @@ -28,6 +30,8 @@ export const DEFAULT_RETENTION_DAYS = 90; export const ROLLUP_LOOKBACK_DAYS = 3; /** Widest range one manual /admin/rollup call will attempt. */ export const MAX_MANUAL_DAYS = 31; +/** Most missed days one nightly run catches up on, newest first; the next night takes the rest. */ +export const MAX_CATCHUP_DAYS = 31; /** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */ const PURGE_BATCH_ROWS = 5_000; @@ -66,18 +70,17 @@ export function retentionCutoff(atMs: number, keepDays: number): string { // aggregation happens inside D1, so a day rolls up in one round trip and no event // row ever crosses the wire. Each takes exactly one bound parameter — the day. // +// Lifecycle events (install / index / uninstall) are one `events` row each, so their +// volume is count(*). Usage is NOT in `events`: the ingest worker adds every +// usage_rollup counter into `usage_daily` (one row per machine × day × tool), so the +// usage figures are sums over that table — its `count` is calls, and counting its +// rows would report "machines that used the tool" instead. +// // Adding a breakdown is a line in ROLLUP_STATEMENTS, never a migration — that is // what the generic (dim, value) shape of daily_dim_counts buys. -/** - * A group's event volume. For install/index/uninstall one row is one event, but a - * usage_rollup row is a counter the client pre-aggregated (one per machine × day × - * tool), so its `count` prop is what has to be summed — counting rows there would - * silently report "machines that used the tool" and undercount by an order of magnitude. - */ -const COUNT = `CASE WHEN e.event = 'usage_rollup' - THEN sum(coalesce(json_extract(e.props, '$.count'), 0)) - ELSE count(*) END`; +/** Lifecycle events only; usage comes from `usage_daily`. */ +const LIFECYCLE = ` AND e.event <> 'usage_rollup'`; const DIM_CONFLICT = `ON CONFLICT (day, event, dim, value) DO UPDATE SET count = excluded.count, machines = excluded.machines`; @@ -87,15 +90,29 @@ const quoted = (values: readonly string[]): string => values.map((v) => `'${v}'` const onlyEvents = (...events: readonly string[]): string => ` AND e.event IN (${quoted(events)})`; /** One dimension whose value is a scalar column or a scalar prop. */ -function dimStatement(dim: string, value: string, where = ''): string { +function dimStatement(dim: string, value: string, where = LIFECYCLE): string { return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) - SELECT e.day, e.event, '${dim}', CAST(${value} AS TEXT), ${COUNT}, count(DISTINCT e.machine_id) + SELECT e.day, e.event, '${dim}', CAST(${value} AS TEXT), count(*), count(DISTINCT e.machine_id) FROM events e WHERE e.day = ? AND ${value} IS NOT NULL AND ${value} <> ''${where} GROUP BY e.day, e.event, ${value} ${DIM_CONFLICT}`; } +/** + * One usage dimension, from a `usage_daily` column. Absent values are stored as '' + * (they are part of the primary key), so '' is what gets skipped. `where` narrows + * the rows further — name_error keeps only counters that saw an error. + */ +function usageDimStatement(dim: string, column: string, total = 'count', where = ''): string { + return `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) + SELECT day, 'usage_rollup', '${dim}', ${column}, sum(${total}), count(DISTINCT machine_id) + FROM usage_daily + WHERE day = ? AND ${column} <> ''${where} + GROUP BY day, ${column} + ${DIM_CONFLICT}`; +} + /** * One dimension unnested from a JSON array prop — one row per element, so an index * of a TypeScript+Go repo counts once under each language. `json_each` over a path @@ -125,22 +142,48 @@ const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines ON CONFLICT (day) DO UPDATE SET machines = excluded.machines, prod_machines = excluded.prod_machines`; +/** + * The activation funnel's input: each machine's first index day, lowered to this day + * if the machine indexed on it and has no earlier index on record. The dashboard reads + * it straight off `machine_first_seen`, which keeps the funnel off raw `events` (a + * cohort join there took most of a minute per week of cohorts) and past the purge. + * + * Only rows that actually move are written, so re-running a day is free. `?1` is the + * day — still the one bound parameter, used three times. + */ +const FIRST_INDEX_DAY = `UPDATE machine_first_seen + SET first_index_day = ?1 + WHERE (first_index_day IS NULL OR first_index_day > ?1) + AND machine_id IN (SELECT machine_id FROM events WHERE day = ?1 AND event = 'index')`; + const ROLLUP_STATEMENTS: readonly string[] = [ DAILY_MACHINES, + FIRST_INDEX_DAY, `INSERT INTO daily_event_counts (day, event, count, machines) - SELECT e.day, e.event, ${COUNT}, count(DISTINCT e.machine_id) + SELECT e.day, e.event, count(*), count(DISTINCT e.machine_id) FROM events e - WHERE e.day = ? + WHERE e.day = ?${LIFECYCLE} GROUP BY e.day, e.event ON CONFLICT (day, event) DO UPDATE SET count = excluded.count, machines = excluded.machines`, + `INSERT INTO daily_event_counts (day, event, count, machines) + SELECT day, 'usage_rollup', sum(count), count(DISTINCT machine_id) + FROM usage_daily + WHERE day = ? + GROUP BY day + ON CONFLICT (day, event) DO UPDATE + SET count = excluded.count, machines = excluded.machines`, // Envelope dimensions — every event type carries them. dimStatement('os', 'e.os'), dimStatement('arch', 'e.arch'), dimStatement('codegraph_version', 'e.codegraph_version'), dimStatement('node_major', 'e.node_major'), + usageDimStatement('os', 'os'), + usageDimStatement('arch', 'arch'), + usageDimStatement('codegraph_version', 'codegraph_version'), + usageDimStatement('node_major', 'node_major'), // Event-specific scalar props. dimStatement('file_count_bucket', prop('file_count_bucket'), onlyEvents('index')), @@ -148,9 +191,10 @@ const ROLLUP_STATEMENTS: readonly string[] = [ dimStatement('scope', prop('scope'), onlyEvents('install')), // `kind` is fresh/upgrade/reinstall on install and mcp_tool/cli_command on // usage_rollup; `event` is part of the primary key, so both live here without colliding. - dimStatement('kind', prop('kind'), onlyEvents('install', 'usage_rollup')), - dimStatement('name', prop('name'), onlyEvents('usage_rollup')), - dimStatement('client_name', prop('client_name'), onlyEvents('usage_rollup')), + dimStatement('kind', prop('kind'), onlyEvents('install')), + usageDimStatement('kind', 'kind'), + usageDimStatement('name', 'name'), + usageDimStatement('client_name', 'client_name'), // Array props. arrayDimStatement('language', '$.languages', ['index']), @@ -158,20 +202,13 @@ const ROLLUP_STATEMENTS: readonly string[] = [ // Errors per tool/command. Not in the migration's documented dim list because dims // are a cron concern rather than a schema one, but rolled up because it is the one - // usage number that is gone for good after the purge. Only groups with at least one - // error are stored, so `count` is errors and `machines` is the machines that saw one + // usage number that is gone for good after the purge. Only counters with at least + // one error count, so `count` is errors and `machines` is the machines that saw one // — NOT the machines that ran the tool (that is the `name` dim). - `INSERT INTO daily_dim_counts (day, event, dim, value, count, machines) - SELECT e.day, e.event, 'name_error', CAST(${prop('name')} AS TEXT), - sum(${prop('error_count')}), count(DISTINCT e.machine_id) - FROM events e - WHERE e.day = ? AND e.event = 'usage_rollup' - AND ${prop('name')} IS NOT NULL AND coalesce(${prop('error_count')}, 0) > 0 - GROUP BY e.day, e.event, ${prop('name')} - ${DIM_CONFLICT}`, + usageDimStatement('name_error', 'name', 'error_count', ' AND error_count > 0'), ]; -/** Rollup tables derived from raw `events` — the ones `reset` wipes before recomputing. */ +/** Rollup tables derived from raw `events` / `usage_daily` — the ones `reset` wipes before recomputing. */ const EVENT_DERIVED_TABLES = ['daily_event_counts', 'daily_dim_counts'] as const; // --------------------------------------------------------------------------- @@ -206,6 +243,7 @@ export async function rollupDay( opts: { cutoff: string; reset?: boolean }, ): Promise { const pastRetention = day < opts.cutoff; + await foldLegacyUsage(env, day); const statements: D1PreparedStatement[] = []; if (opts.reset && !pastRetention) { @@ -222,36 +260,147 @@ export async function rollupDay( return { day, rows, pastRetention }; } +// --------------------------------------------------------------------------- +// Legacy usage rows +// --------------------------------------------------------------------------- +// Before migrations/0003, every usage_rollup event was its own `events` row — about +// 3.8M a day by August 2026, because clients upload a counter per process rather +// than per day, and enough to fill the database. These statements fold one day's +// worth of those rows into `usage_daily`, adding counts the same way the ingest path +// does, and delete them in the same transaction, so a count is never in both places +// and never in neither. rollupDay runs it first, which makes re-running the rollup +// over old days the whole migration; once a day has no legacy rows left it costs +// one indexed probe. + +/** Rows folded per transaction — bounded so one GROUP BY + DELETE stays well inside D1's limits. */ +const LEGACY_CHUNK_ROWS = 50_000; +/** Ceiling per day (the worst day on record had ~3.9M rows). */ +const LEGACY_MAX_CHUNKS = 200; + +const LEGACY_FOLD = `INSERT INTO usage_daily ( + day, machine_id, kind, name, client_name, client_version, + codegraph_version, os, arch, node_major, count, error_count) + SELECT day, machine_id, + coalesce(json_extract(props, '$.kind'), ''), + coalesce(json_extract(props, '$.name'), ''), + coalesce(json_extract(props, '$.client_name'), ''), + coalesce(json_extract(props, '$.client_version'), ''), + coalesce(codegraph_version, ''), coalesce(os, ''), coalesce(arch, ''), + coalesce(CAST(node_major AS TEXT), ''), + sum(coalesce(json_extract(props, '$.count'), 0)), + sum(coalesce(json_extract(props, '$.error_count'), 0)) + FROM events + WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2 + GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 + ON CONFLICT (day, machine_id, kind, name, client_name, client_version, codegraph_version, os, arch, node_major) + DO UPDATE SET count = usage_daily.count + excluded.count, + error_count = usage_daily.error_count + excluded.error_count`; + +const LEGACY_DELETE = `DELETE FROM events WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2`; + +/** Folds any legacy usage rows for `day` into usage_daily. Returns how many rows it moved. */ +export async function foldLegacyUsage(env: Env, day: string): Promise { + const any = await env.DB.prepare(`SELECT 1 AS hit FROM events WHERE day = ? AND event = 'usage_rollup' LIMIT 1`) + .bind(day) + .first<{ hit: number }>(); + if (!any) return 0; + + let moved = 0; + for (let chunk = 0; chunk < LEGACY_MAX_CHUNKS; chunk++) { + // Always the oldest remaining rows: everything below the next chunk's first id. + // Rows folded by earlier chunks are gone, so there is no cursor to carry. + const next = await env.DB.prepare( + `SELECT id FROM events WHERE day = ? AND event = 'usage_rollup' ORDER BY id LIMIT 1 OFFSET ${LEGACY_CHUNK_ROWS}`, + ) + .bind(day) + .first<{ id: number }>(); + const below = next?.id ?? Number.MAX_SAFE_INTEGER; + const [, deleted] = await env.DB.batch([ + env.DB.prepare(LEGACY_FOLD).bind(day, below), + env.DB.prepare(LEGACY_DELETE).bind(day, below), + ]); + moved += deleted?.meta?.changes ?? 0; + if (!next) return moved; + } + throw new Error(`legacy usage fold for ${day} did not finish within ${LEGACY_MAX_CHUNKS} chunks`); +} + export interface PurgeResult { /** Everything strictly before this day was deleted. */ cutoff: string; deleted: number; + /** usage_daily rows past the window — per-machine counters, so they expire with the events. */ + usageDeleted: number; batches: number; /** Hit the per-run batch ceiling — more rows are still due, next run takes them. */ capped: boolean; } /** - * Delete raw events older than the window, oldest first, in bounded batches. - * `id` is a rowid alias and the purge only ever removes the oldest rows, so the - * keyset subquery stays a cheap index range scan on (day, event). + * Delete raw events older than the window, oldest first, in bounded batches, then + * the usage counters older than it, a day per statement (`usage_daily` is keyed by + * day first, so each is one primary-key range). `id` is a rowid alias and the purge + * only ever removes the oldest rows, so the keyset subquery stays a cheap index range + * scan on (day, event). */ export async function purgeOldEvents(env: Env, cutoff: string): Promise { const del = env.DB.prepare( `DELETE FROM events WHERE id IN (SELECT id FROM events WHERE day < ? LIMIT ${PURGE_BATCH_ROWS})`, ); let deleted = 0; - for (let batch = 1; batch <= PURGE_MAX_BATCHES; batch++) { + let batches = 0; + let capped = true; + while (batches < PURGE_MAX_BATCHES) { + batches++; const { meta } = await del.bind(cutoff).run(); const removed = meta?.changes ?? 0; deleted += removed; - if (removed < PURGE_BATCH_ROWS) return { cutoff, deleted, batches: batch, capped: false }; + if (removed < PURGE_BATCH_ROWS) { + capped = false; + break; + } } - return { cutoff, deleted, batches: PURGE_MAX_BATCHES, capped: true }; + + const delUsageDay = env.DB.prepare( + `DELETE FROM usage_daily WHERE day = (SELECT min(day) FROM usage_daily WHERE day < ?)`, + ); + let usageDeleted = 0; + for (let day = 0; day < PURGE_MAX_BATCHES; day++) { + const { meta } = await delUsageDay.bind(cutoff).run(); + const removed = meta?.changes ?? 0; + if (removed === 0) break; + usageDeleted += removed; + } + return { cutoff, deleted, usageDeleted, batches, capped }; +} + +/** + * Days that saw activity but never got a rollup: the nights a run failed, or the + * database refused writes. Every active day has `machine_days` rows and every + * rolled-up day has a `daily_machines` row, so the difference is exactly the missed + * days — a day nobody was active on is in neither, and needs nothing. + * + * Without this, the cron only ever looks three days back, so one bad week leaves a + * permanent hole that someone has to notice and backfill by hand. Bounded below by + * the retention cutoff, past which the raw events are gone and there is nothing left + * to roll up. + */ +async function missedDays(env: Env, cutoff: string, before: string): Promise { + const { results } = await env.DB.prepare( + `SELECT DISTINCT day FROM machine_days + WHERE day >= ? AND day < ? + AND day NOT IN (SELECT day FROM daily_machines) + ORDER BY day DESC + LIMIT ${MAX_CATCHUP_DAYS}`, + ) + .bind(cutoff, before) + .all<{ day: string }>(); + return results.map((r) => r.day); } /** - * The cron body: roll up the completed day and the two before it, then purge. + * The cron body: roll up the completed day and the two before it, catch up any + * earlier day a past run missed, then purge. * * Logs one line of counts — never a day's contents, never a machine id. Throws if * anything failed so the invocation is marked failed (and retried) rather than @@ -262,11 +411,23 @@ export async function runNightly(env: Env, atMs: number): Promise { const keepDays = retentionDays(env); const cutoff = retentionCutoff(atMs, keepDays); + const days: string[] = []; + for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) days.push(utcDay(atMs - back * DAY_MS)); + + // A failure to find the missed days must not cost tonight's regular rollup. + let caughtUp = 0; + try { + const missed = await missedDays(env, cutoff, days[days.length - 1] ?? utcDay(atMs)); + days.push(...missed); + caughtUp = missed.length; + } catch (err) { + console.error(JSON.stringify({ msg: 'missed-day scan failed', err: String(err) })); + } + const rolled: string[] = []; const failed: string[] = []; let rows = 0; - for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) { - const day = utcDay(atMs - back * DAY_MS); + for (const day of days) { try { rows += (await rollupDay(env, day, { cutoff })).rows; rolled.push(day); @@ -287,11 +448,13 @@ export async function runNightly(env: Env, atMs: number): Promise { JSON.stringify({ msg: 'nightly rollup', days: rolled, + caught_up: caughtUp, rows, failed: failed.length, retention_days: keepDays, purged_before: cutoff, purged: purge?.deleted ?? null, + usage_purged: purge?.usageDeleted ?? null, purge_batches: purge?.batches ?? null, purge_capped: purge?.capped ?? null, ms: Date.now() - started, From 0f0af7bf22a5083a47e34fcb1e1bf67355174ef8 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Sat, 3 Oct 2026 06:35:17 +0000 Subject: [PATCH 199/363] release: 1.6.2 (#2318) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps package.json and ui/package.json (the component library is versioned with the engine; ui-package.test.ts pins it) to 1.6.2, and the matching version fields in package-lock.json — the top level, packages[""] and the ui workspace, whose entry was still at 1.6.0. Only those three fields change; `npm ci` accepts the lock. Co-authored-by: Claude Opus 5.5 --- package-lock.json | 6 +++--- package.json | 2 +- ui/package.json | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/package-lock.json b/package-lock.json index e6a232af46..9591645893 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "license": "MIT", "workspaces": [ "ui" @@ -3134,7 +3134,7 @@ }, "ui": { "name": "@colbymchenry/codegraph-ui", - "version": "1.6.0", + "version": "1.6.2", "license": "MIT", "dependencies": { "@xyflow/svelte": "^1.6.5" diff --git a/package.json b/package.json index 92d1aa6c7f..ab8ba5cca6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@colbymchenry/codegraph", - "version": "1.6.1", + "version": "1.6.2", "description": "Supercharge AI coding agents with semantic code intelligence — surgical context, fewer tool calls, faster answers. 100% local.", "repository": { "type": "git", diff --git a/ui/package.json b/ui/package.json index 65cd32c89f..4543403516 100644 --- a/ui/package.json +++ b/ui/package.json @@ -1,7 +1,7 @@ { "name": "@colbymchenry/codegraph-ui", "private": true, - "version": "1.6.1", + "version": "1.6.2", "type": "module", "description": "The CodeGraph reader as Svelte components: Symbol view, Flow strip and architecture Map behind one data adapter.", "keywords": [ From 6560052a6f856855d3f71eee838fd66ccfa4285d Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Sat, 3 Oct 2026 06:38:45 +0000 Subject: [PATCH 200/363] docs(changelog): promote [Unreleased] into [1.6.2] [skip ci] Auto-generated by Release workflow. --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index e027687d6b..06ca319d98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] + +## [1.6.2] - 2026-10-03 + ### Highlights - **Far fewer wrong links, in every language.** A call, import or type now resolves the way its language scopes names — through imports, packages, namespaces and the class it is written in — so it stops landing on an unrelated symbol that only shares its name. This covers TypeScript and JavaScript, Python, Java, Kotlin, Scala, C#, VB.NET, Swift, Objective-C, Go, Rust, C and C++, PHP, Ruby, Dart, Lua, R and more, and makes callers, impact and `codegraph_explore` answers more trustworthy. @@ -1313,3 +1316,4 @@ Thanks @andreinknv for the substantive draft this release was based on. [1.5.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.5.0 [1.6.0]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.0 [1.6.1]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.1 +[1.6.2]: https://github.com/colbymchenry/codegraph/releases/tag/v1.6.2 From 489b4749c725cc446df7245fa611d77dab9c301e Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 14:41:59 +0000 Subject: [PATCH 201/363] fix(vbnet,csharp): index every Structure member; walk property bodies and VB initializers (#2345) VB.NET's grammar tags each member of a `Structure` as its own `body` field, so extractAggregate's field lookup saw only the first member. SCrawler's UserMedia indexed none of its members, and a structure that opened with a nested enum indexed only the enum. Aggregates now resolve their body the way classes and enums do (resolveBody, which VB maps to the declaration itself). Objective-C, the only other struct language with that hook, is unchanged. Property bodies were never walked: the propertyTypes branch set skipChildren after extractProperty, so calls, instantiations and reads in a VB `Get`/`Set` block, `= initializer` or `As New`, and in a C# accessor body or `=> expr`, were lost. - propertyBodies() names the parts of a property that run code (VB: the whole declaration, as its methods are walked; C#: each accessor's body and an arrow value), walked with the property on the stack. The candidates-only fn-ref scan skips the walked subtrees by node id, so a function value is captured once, from the property. C# `= initializer`s stay unwalked, as C# field initializers are. - The C# kernel mirrors it (property_bodies, scan skip list). The parity suite and a new micro pass; a serilog sweep is 209/214 byte-identical with 0 diffs (5 deferred for parse errors). - VB field declarators (`= expr`, `As New T`) and Custom Event AddHandler/RemoveHandler/RaiseEvent blocks are walked as their field or event. Before/after (nodes / edges): SCrawler 9,950 -> 10,427 / 14,336 -> 15,092; staxrip 13,440 -> 13,644 / 24,120 -> 28,002, most of it its `Property X As New NumParam With {...}` settings; serilog edges 6,642 -> 6,654. All 23 removed edges are the same call sites re-resolved by VB's name-guessing resolver now that struct members are candidates: 10 wrong edges dropped, 9 correct ones now guessed wrong, 4 wrong either way. Designer -> Add/Size edge counts are unchanged. EXTRACTION_VERSION 27 -> 28. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + __tests__/csharp-property-accessors.test.ts | 91 +++++++++ __tests__/kernel-csharp-parity.test.ts | 20 +- __tests__/vbnet-member-bodies.test.ts | 209 ++++++++++++++++++++ codegraph-kernel/src/csharp.rs | 80 ++++++-- docs/design/csharp-kernel-port-checklist.md | 22 ++- src/extraction/extraction-version.ts | 2 +- src/extraction/languages/vbnet.ts | 9 +- src/extraction/tree-sitter.ts | 62 ++++-- 9 files changed, 459 insertions(+), 42 deletions(-) create mode 100644 __tests__/csharp-property-accessors.test.ts create mode 100644 __tests__/vbnet-member-bodies.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 06ca319d98..44144e1606 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,6 +12,12 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Fixes + +- In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. +- In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. +- In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. + ## [1.6.2] - 2026-10-03 diff --git a/__tests__/csharp-property-accessors.test.ts b/__tests__/csharp-property-accessors.test.ts new file mode 100644 index 0000000000..b77d8549d6 --- /dev/null +++ b/__tests__/csharp-property-accessors.test.ts @@ -0,0 +1,91 @@ +/** + * C# property accessor bodies belong to the property. `get { … }`, + * `set { … }`, `get => …` and an expression-bodied `=> …` property used to + * be skipped by both extractors, so the calls, instantiations and static + * reads written there were lost and a getter's callee looked unused. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const FILES: Record = { + 'Helper.cs': `public static class Helper { + public static int Compute(int n) => n; + public static void Store(int n) {} + public static int Arrow() => 1; + public static int GetA() => 1; + public static void SetA(int v) {} +} +public class Widget {} +public static class Defaults { public const string Name = "x"; } +`, + 'Box.cs': `public class Box { + private int _x; + public int Value { + get { return Helper.Compute(_x); } + set { Helper.Store(value); _x = value; } + } + public int Arrow => Helper.Arrow(); + public int Both { get => Helper.GetA(); set => Helper.SetA(value); } + public Widget Made { get { return new Widget(); } } + public string Label { get { return Defaults.Name; } } + private void Handle(int v) { } + public System.Action Handler { get { return Pick(Handle); } } +} +`, +}; + +describe('C# property accessor bodies', () => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-accessors-')); + for (const [rel, content] of Object.entries(FILES)) fs.writeFileSync(path.join(root, rel), content); + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('are walked as the property (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const member = (qualifiedName: string): Node => { + const node = graph.getNodesInFile('Box.cs').find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; + }; + const targets = (node: Node, kind: string): string[] => + graph + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); + + expect(targets(member('Box::Value'), 'calls')).toEqual(['Helper::Compute', 'Helper::Store']); + expect(targets(member('Box::Arrow'), 'calls')).toEqual(['Helper::Arrow']); + expect(targets(member('Box::Both'), 'calls')).toEqual(['Helper::GetA', 'Helper::SetA']); + expect(targets(member('Box::Made'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::Label'), 'references')).toContain('Defaults'); + // A method passed as a value inside an accessor is the property's + // reference, captured once — not the class's as well. + expect(targets(member('Box::Handler'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box'), 'references')).not.toContain('Box::Handle'); + expect(targets(member('Box'), 'calls')).toEqual([]); + }); +}); diff --git a/__tests__/kernel-csharp-parity.test.ts b/__tests__/kernel-csharp-parity.test.ts index f26f5072df..3d09c19d98 100644 --- a/__tests__/kernel-csharp-parity.test.ts +++ b/__tests__/kernel-csharp-parity.test.ts @@ -8,7 +8,8 @@ * * - Torture.cs — block namespace + nested/second-namespace quirks, * base_list shapes, records, properties (incl. the bare-identifier - * signature loss and never-walked accessor bodies), fields/constants, + * signature loss, accessor bodies walked as the property and + * never-walked initializers), fields/constants, * events/operators/indexer/destructor (no nodes, calls → class), ctor * initializer hole, explicit interface impl, local functions, the call * zoo (raw member-access texts, chained re-encode, `(myDel)(x)` conv, @@ -167,6 +168,23 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { source: 'public record Empty;\n', minNodes: 2, }, + { + // Accessor bodies and `=> expr` are walked as the property (calls, + // instantiations, static reads, fn-ref candidates); an `= initializer` + // is only scanned for candidates, attributed to the class. + name: 'property bodies are walked as the property; initializers only scanned', + source: [ + 'public class C {', + ' void H(int v) { }', + ' public Action P { get { return Make(H); } set { Register(H); } }', + ' public int Q => Run(H);', + ' public Action R { get; } = Wrap(H);', + ' public int S { get => Calc.Max; init => Store(new Widget()); }', + '}', + '', + ].join('\n'), + minNodes: 7, + }, ]; for (const m of MICROS) { diff --git a/__tests__/vbnet-member-bodies.test.ts b/__tests__/vbnet-member-bodies.test.ts new file mode 100644 index 0000000000..8c280a5b55 --- /dev/null +++ b/__tests__/vbnet-member-bodies.test.ts @@ -0,0 +1,209 @@ +/** + * VB.NET member bodies the extractor used to skip: + * + * - A `Structure` lists each member as its own `body` field, so reading the + * field returned only the first member: SCrawler's `UserMedia` indexed its + * nested `States` enum (declared first) and nothing else, and a structure + * that opened with a field indexed no members at all. + * - A property's `Get` / `Set` blocks, its `= initializer` and its + * `As New T` were never walked, nor a `Custom Event`'s accessors. + * - A field's `= initializer` and `As New T` were never walked. + * + * The calls, instantiations and reads in those places belong to the member + * that declares them. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-bodies-')); + const files: Record = { + 'Helper.vb': `Public Class Helper + Public Sub New(ByVal n As Integer) + End Sub + Public Shared Function Compute(ByVal n As Integer) As Integer + Return n + End Function + Public Shared Sub Store(ByVal n As Integer) + End Sub + Public Shared Function Seed() As Integer + Return 1 + End Function + Public Shared Function Format(ByVal o As Object) As String + Return "" + End Function + Public Shared Sub Log(ByVal s As String) + End Sub + Public Shared Sub Attach(ByVal h As EventHandler) + End Sub +End Class +`, + 'Widget.vb': `Public Class Widget +End Class +`, + 'UserMedia.vb': `Public Structure UserMedia + Public State As States + Private Count As Integer + Public Enum States + Unknown + Downloaded + End Enum + Public ReadOnly Property Name As String + Get + Return Helper.Format(State) + End Get + End Property + Public Sub New(ByVal s As States) + State = s + Helper.Log("created") + End Sub + Public Function IsDone() As Boolean + Return State = States.Downloaded + End Function +End Structure +`, + 'Box.vb': `Public Class Box + Private _x As Integer + Private _items As New Widget + Private _helper As Helper = New Helper(1) + Private Shared ReadOnly Def As Integer = Helper.Compute(3), Other As Integer = Helper.Seed() + Public Property Value As Integer + Get + Return Helper.Compute(_x) + End Get + Set(ByVal v As Integer) + Helper.Store(v) + _x = v + End Set + End Property + Public Property Auto As Integer = Helper.Seed() + Public ReadOnly Property Lst As New Widget + Public Custom Event Changed As EventHandler + AddHandler(ByVal value As EventHandler) + Helper.Attach(value) + End AddHandler + RemoveHandler(ByVal value As EventHandler) + End RemoveHandler + RaiseEvent(ByVal sender As Object, ByVal e As EventArgs) + End RaiseEvent + End Event +End Class +`, + // The 1.6.2 receiver gate must hold in the newly walked places too: + // `Me.Panel.Controls.Add(…)` and `New System.Drawing.Size(…)` name no + // project member. + 'Collections/DataColorCollection.vb': `Friend Class DataColorCollection + Friend Sub Add(ByVal Item As Object) + End Sub +End Class +`, + 'Editors/UsersInfoForm.vb': `Friend Class UsersInfoForm + Private Enum EComparers + Name + Size + End Enum +End Class +`, + 'MainForm.vb': `Public Class MainForm + Private ReadOnly DefaultSize As System.Drawing.Size = New System.Drawing.Size(184, 25) + Public ReadOnly Property Ready As Boolean + Get + Me.Panel.Controls.Add(Me.Button1) + Return True + End Get + End Property +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +function member(file: string, qualifiedName: string): Node { + const node = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName); + expect(node, `${qualifiedName} in ${file}`).toBeDefined(); + return node!; +} + +/** `kind` edges leaving `node`, as target qualified names. */ +function targets(node: Node, kind: string): string[] { + return cg + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET member bodies', () => { + it('indexes every member of a Structure, not just the first', () => { + const struct = member('UserMedia.vb', 'UserMedia'); + expect(struct.kind).toBe('struct'); + const contained = cg + .getOutgoingEdgesFrom([struct.id]) + .filter((e) => e.kind === 'contains') + .map((e) => cg.getNode(e.target)!) + .map((n) => `${n.kind} ${n.qualifiedName}`) + .sort(); + expect(contained).toEqual([ + 'enum UserMedia::States', + 'field UserMedia::Count', + 'field UserMedia::State', + 'method UserMedia::IsDone', + 'method UserMedia::New', + 'property UserMedia::Name', + ]); + expect(member('UserMedia.vb', 'UserMedia::States::Downloaded').kind).toBe('enum_member'); + expect(targets(member('UserMedia.vb', 'UserMedia::New'), 'calls')).toEqual(['Helper::Log']); + }); + + it("attributes a property's Get and Set blocks to the property", () => { + expect(targets(member('Box.vb', 'Box::Value'), 'calls')).toEqual(['Helper::Compute', 'Helper::Store']); + expect(targets(member('UserMedia.vb', 'UserMedia::Name'), 'calls')).toEqual(['Helper::Format']); + }); + + it("walks a property's initializer and its As New", () => { + expect(targets(member('Box.vb', 'Box::Auto'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box.vb', 'Box::Lst'), 'instantiates')).toEqual(['Widget']); + }); + + it("walks each field's initializer and its As New, per declarator", () => { + expect(targets(member('Box.vb', 'Box::Def'), 'calls')).toEqual(['Helper::Compute']); + expect(targets(member('Box.vb', 'Box::Other'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box.vb', 'Box::_items'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box.vb', 'Box::_helper'), 'instantiates')).toEqual(['Helper']); + }); + + it("attributes a Custom Event's accessors to the event", () => { + expect(targets(member('Box.vb', 'Box::Changed'), 'calls')).toEqual(['Helper::Attach']); + }); + + it('keeps the class out of what its members do', () => { + expect(targets(member('Box.vb', 'Box'), 'calls')).toEqual([]); + expect(targets(member('Box.vb', 'Box'), 'instantiates')).toEqual([]); + }); + + it('still links nothing through a receiver no name resolves', () => { + const ids = cg.getNodesInFile('MainForm.vb').map((n) => n.id); + const linked = cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind !== 'contains') + .map((e) => cg.getNode(e.target)!.qualifiedName); + expect(linked).not.toContain('DataColorCollection::Add'); + expect(linked).not.toContain('UsersInfoForm::EComparers::Size'); + }); +}); diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 7d481662a7..04e20ca5a5 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -5,7 +5,7 @@ //! path, bug-for-bug, verified by scripts/kernel-parity.mjs and the full-index //! dump-diff gate. The authoritative quirk list is //! docs/design/csharp-kernel-port-checklist.md — including every deliberate -//! emission hole (property/accessor bodies, constructor initializers, +//! emission hole (field/property initializers, constructor initializers, //! delegates/events/operators/indexers, top-level locals) and garbage ref //! (`(repo)` primary-ctor extends, `: byte` enum extends, `nameof` calls) //! this file preserves on purpose. Positions in UTF-16 code units. Files whose @@ -572,21 +572,36 @@ impl<'t> Walker<'t> { self.extract_enum(node); skip_children = true; } else if kind == "property_declaration" && self.inside_class_like() { - // Property accessor/expression bodies are NEVER walked (calls - // inside are lost by design) — candidates-only scan. - self.extract_property(node); - self.scan_fn_ref_subtree(node, 0); + // The code a property runs — its accessor bodies and `=> expr` — + // is walked with the property on the stack (propertyBodies). The + // candidates-only scan covers the rest (an `= initializer`), + // skipping what the body walk captured. + let walked: Vec = match self.extract_property(node) { + Some((row, name)) => { + let bodies = property_bodies(node); + if !bodies.is_empty() { + self.stack.push(Scope { row, kind: "property", name }); + for body in &bodies { + self.visit_function_body(*body); + } + self.stack.pop(); + } + bodies.iter().map(|b| b.id()).collect() + } + None => Vec::new(), + }; + self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "field_declaration" && self.inside_class_like() { self.extract_field(node); - self.scan_fn_ref_subtree(node, 0); + self.scan_fn_ref_subtree(node, 0, &[]); skip_children = true; } else if kind == "local_declaration_statement" && !self.inside_class_like() { // Top-level statements: extractVariable's generic fallback finds no // direct identifier/variable_declarator children (C# nests them in // variable_declaration) → ZERO nodes, zero refs. Candidates only. self.extract_variable(node); - self.scan_fn_ref_subtree(node, 0); + self.scan_fn_ref_subtree(node, 0, &[]); skip_children = true; } else if kind == "using_directive" { self.extract_import(node); @@ -789,9 +804,9 @@ impl<'t> Walker<'t> { } /// extractProperty (1986) — property_declaration only (dispatch-gated to - /// class-like scopes). Accessor bodies and `=>` value clauses are never - /// walked; type refs DO come from the `type` field. - fn extract_property(&mut self, node: Node<'t>) { + /// class-like scopes). Type refs come from the `type` field; the caller + /// walks the bodies (property_bodies) with the returned row on the stack. + fn extract_property(&mut self, node: Node<'t>) -> Option<(u32, String)> { let docstring = preceding_docstring(node, self.src); let visibility = Some(self.visibility_of(node)); let is_static = Some(self.is_static(node)); // ?? false — always concrete @@ -804,10 +819,10 @@ impl<'t> Walker<'t> { .filter_map(|i| node.named_child(i)) .find(|c| c.kind() == "identifier") }); - let Some(name_node) = name_node else { return }; + let name_node = name_node?; let name = self.text(name_node).to_string(); if name.is_empty() { - return; + return None; } // Generic scan (isTsJsField=false): FIRST namedChild that isn't a @@ -842,11 +857,10 @@ impl<'t> Walker<'t> { &name, node, Extra { docstring, signature: Some(signature), visibility, is_static, ..Extra::default() }, - ); - if let Some(row) = row { - // decorators: none for C#; then the csharp type-ref path. - self.extract_csharp_type_refs(node, row); - } + )?; + // decorators: none for C#; then the csharp type-ref path. + self.extract_csharp_type_refs(node, row); + Some((row, name)) } /// extractField (2046) — field_declaration; each declarator becomes a @@ -1468,11 +1482,15 @@ impl<'t> Walker<'t> { }); } - fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32) { + fn scan_fn_ref_subtree(&mut self, node: Node<'t>, depth: u32, walked: &[usize]) { stack_guard!(); if depth > 12 { return; } + // Subtrees the body walker has already been through. + if walked.contains(&node.id()) { + return; + } // functionTypes is EMPTY for C#; the literal halt list applies — // lambda_expression IS C#'s lambda, so initializer lambdas stop the // scan; anonymous_method_expression is NOT listed and scans through. @@ -1487,7 +1505,7 @@ impl<'t> Walker<'t> { self.maybe_capture_fn_refs(node); for i in 0..node.named_child_count() { if let Some(c) = node.named_child(i) { - self.scan_fn_ref_subtree(c, depth + 1); + self.scan_fn_ref_subtree(c, depth + 1, walked); } } } @@ -1624,6 +1642,30 @@ impl<'t> Walker<'t> { } } +/// propertyBodies (tree-sitter.ts) — the parts of a property that run code: +/// each accessor's `body` (a block or `=> expr`) and an expression-bodied +/// property's `=> …` value. An `= initializer` value is not a body. +fn property_bodies(node: Node) -> Vec { + let mut bodies = Vec::new(); + if let Some(accessors) = node.child_by_field_name("accessors") { + for i in 0..accessors.named_child_count() { + let Some(accessor) = accessors.named_child(i) else { continue }; + if accessor.kind() != "accessor_declaration" { + continue; + } + if let Some(body) = accessor.child_by_field_name("body") { + bodies.push(body); + } + } + } + if let Some(value) = node.child_by_field_name("value") { + if value.kind() == "arrow_expression_clause" { + bodies.push(value); + } + } + bodies +} + fn find_anonymous_class_body(node: Node) -> Option { for i in 0..node.named_child_count() { if let Some(child) = node.named_child(i) { diff --git a/docs/design/csharp-kernel-port-checklist.md b/docs/design/csharp-kernel-port-checklist.md index 41894cb067..e817968c95 100644 --- a/docs/design/csharp-kernel-port-checklist.md +++ b/docs/design/csharp-kernel-port-checklist.md @@ -234,7 +234,7 @@ all PRESERVE): | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | body `enum_member_declaration_list` required (bodiless → no node); extractInheritance sees `base_list` → **the underlying type `: byte` emits an `extends` ref named `byte`** (quirk, §inheritance); `enum_member_declaration` children → extractEnumMembers:1958 — `name` field path: ONE `enum_member` node per member, positioned at the member node (attributes included in its span), values/attributes ignored; non-member children (preproc_*, comment) → visitNode (no-op) | | `method_declaration` | methodTypes:1027 → extractMethod:1737 | classifyMethodNode absent → always extractMethod. Gate 1747 passes via class-like (a method_declaration outside a type does not occur in non-erroring C# — top-level `void M(){}` parses as local_function_statement, probed); bodyless interface/partial signatures mint nodes with no body walk; **expression-bodied methods have `body: arrow_expression_clause` (a real body FIELD, probed) → walked** | | `constructor_declaration` | methodTypes → extractMethod | name field = the class-name identifier → **method node named like the class**; returnType undefined; **`constructor_initializer` (`: base(args)` / `: this(args)`) is a sibling of the body field → NEVER walked → calls inside initializer args are LOST** (probed); expression-bodied ctor body = arrow_expression_clause → walked | -| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node + scanFnRefSubtree (capture-only) + skipChildren → **accessor bodies (`get { … }`, `get => …`) and the `=> expr` value clause are NEVER walked — calls inside property getters/setters/expression bodies emit NOTHING** (only fn-ref candidates). §property below | +| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`) and an expression-bodied `=> expr` `value:` — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; changed 2026-10-04, previously never walked), then scanFnRefSubtree (capture-only, attributed to the class) over the rest — **the `= initializer` stays unwalked** — skipping the walked bodies, + skipChildren. §property below | | `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator + scanFnRefSubtree + skipChildren → **field initializers emit no calls/instantiates/static-member refs** (fn-ref candidates only). §field below | | `local_declaration_statement` | variableTypes:1098 (only reachable at top level — global statements; body locals go through visitFunctionBody instead) | not class-like → extractVariable:2538 → **generic fallback (2863-2881) finds no direct `identifier`/`variable_declarator` children (the declarator nests inside `variable_declaration`, probed) → ZERO nodes minted**; isClassScopeConstantAssignment (1508) needs node.type `assignment` → never true. skipChildren=true + scanFnRefSubtree → **a top-level `var builder = WebApplication.CreateBuilder(args);` produces NO node, NO calls ref, NO instantiates** — only fn-ref candidates. PRESERVE | | `using_directive` | importTypes:1209 → extractImport:3170 | hook (§config) → import node + ONE generic `imports` ref {fromNodeId: nodeStack top (namespace node if present, else file), referenceName: moduleName, line/col of the directive}; **no per-binding emitter** (the TS/py/rust/php/ruby ladder at 3197-3234 excludes csharp) | @@ -305,15 +305,22 @@ for C#) or bare `name`. QUIRKS (probed, PRESERVE): - An expression-bodied property (`public int Computed => MaxItems + 1;`) has children [modifier, predefined_type, identifier, **arrow_expression_clause (`value:` field)**] → typeNode = predefined_type → signature `"int Computed"`; - the arrow clause is NEVER walked (calls inside lost). -- `{ get; } = new();` initializers: the `value:` implicit_object_creation and - the accessor_list are both skipped/excluded → no refs, no instantiates. + the arrow clause is a property body (walked as the property, below). +- `{ get; } = new();` initializers: the `value:` implicit_object_creation is + NOT a body → no refs, no instantiates (candidates-only scan, attributed to + the class); the accessor_list is excluded from the type scan. Then extractDecoratorsFor (no-op) and **extractTypeAnnotations (2037) → extractCsharpTypeRefs** — the `type` field IS walked for refs (so `public List Items` emits references `List` + `Foo` even though the signature -kept the raw text). Return value feeds no body walk (the classifyMethodNode -initializer-walk path at 1031-1047 is TS-only). +kept the raw text). The returned node is pushed while propertyBodies +(tree-sitter.ts; csharp.rs `property_bodies`) are walked: every +`accessor_declaration`'s `body` field (block or arrow_expression_clause, in +accessor order), then the property's `value:` when it is an +arrow_expression_clause. The fn-ref scan that follows skips those subtrees +by node id, so a candidate is captured once — from the property when it sits +in a body, from the class when it sits in an initializer. (The +classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) ### extractField (2046) — field_declaration @@ -758,7 +765,8 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. + multi-declarator + instance fields (signatures `Type name`); `protected internal` (→ protected); property shapes: predefined-type, bare-identifier type (signature loses type), generic type, expression-bodied - (`=>` calls LOST), `{ get; } = new();`, accessor bodies with calls (LOST); + (`=>` calls → the property), `{ get; } = new();` (initializer LOST), + accessor bodies with calls (→ the property); event_field_declaration + event_declaration with add/remove bodies (no nodes; accessor calls → class); operator + conversion operator + indexer + destructor (no nodes; body calls → class); constructor with diff --git a/src/extraction/extraction-version.ts b/src/extraction/extraction-version.ts index 3691e95f0a..e41b5f6149 100644 --- a/src/extraction/extraction-version.ts +++ b/src/extraction/extraction-version.ts @@ -21,4 +21,4 @@ * turns the re-index hint into noise — keep it honest (see CLAUDE.md, "Honesty * in the product is load-bearing"). */ -export const EXTRACTION_VERSION = 27; +export const EXTRACTION_VERSION = 28; diff --git a/src/extraction/languages/vbnet.ts b/src/extraction/languages/vbnet.ts index b2ec4aab50..e8502b2e30 100644 --- a/src/extraction/languages/vbnet.ts +++ b/src/extraction/languages/vbnet.ts @@ -116,8 +116,13 @@ export const vbnetExtractor: LanguageExtractor = { // findable declaration (WinForms/WPF code is built around them). if (node.type === 'event_declaration' || node.type === 'custom_event_declaration') { const nameNode = node.childForFieldName('name'); - if (nameNode) { - ctx.createNode('field', getNodeText(nameNode, ctx.source), node); + const event = nameNode ? ctx.createNode('field', getNodeText(nameNode, ctx.source), node) : null; + // A Custom Event's AddHandler / RemoveHandler / RaiseEvent blocks run + // code, which is the event's. + if (event && node.type === 'custom_event_declaration') { + ctx.pushScope(event.id); + ctx.visitFunctionBody(node, event.id); + ctx.popScope(); } return true; } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 5c27523acc..9a10862475 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -733,8 +733,10 @@ export class TreeSitterExtractor { * nested function definitions: their bodies are walked — and their * candidates attributed — by extractFunction's own body walk. */ - private scanFnRefSubtree(node: SyntaxNode, depth: number): void { + private scanFnRefSubtree(node: SyntaxNode, depth: number, walked?: ReadonlySet): void { if (!this.fnRefSpec || depth > 12) return; + // Subtrees the body walker has already been through. + if (walked?.has(node.id)) return; const nodeType = node.type; if (depth > 0 && ( this.extractor?.functionTypes.includes(nodeType) || @@ -749,7 +751,7 @@ export class TreeSitterExtractor { this.maybeCaptureFnRefs(node, nodeType); for (let i = 0; i < node.namedChildCount; i++) { const child = node.namedChild(i); - if (child) this.scanFnRefSubtree(child, depth + 1); + if (child) this.scanFnRefSubtree(child, depth + 1, walked); } } @@ -1279,11 +1281,19 @@ export class TreeSitterExtractor { } // Check for class properties (e.g. C# property_declaration) else if (this.extractor.propertyTypes?.includes(nodeType) && this.isInsideClassLikeNode()) { - this.extractProperty(node); - // Property initializers aren't walked — scan for function-as-value - // candidates (#756): Scala `val table = Seq(targetCb)` in an object, - // Kotlin `val cb = ::handler` class properties. - this.scanFnRefSubtree(node, 0); + const propNode = this.extractProperty(node); + // The code a property runs is the property's: its calls, + // instantiations and reads attribute to it, as a method's do. + const bodies = propNode ? this.propertyBodies(node) : []; + if (propNode && bodies.length > 0) { + this.nodeStack.push(propNode.id); + for (const body of bodies) this.visitFunctionBody(body, propNode.id); + this.nodeStack.pop(); + } + // Whatever the body walk didn't cover (a C# `= initializer`, any other + // language's whole declaration) is scanned for function-as-value + // candidates (#756); the bodies captured their own. + this.scanFnRefSubtree(node, 0, new Set(bodies.map((b) => b.id))); skipChildren = true; } // Check for class fields (e.g. Java field_declaration, C# field_declaration) @@ -2113,7 +2123,12 @@ export class TreeSitterExtractor { // (#1093) because the two defaults differ — a bodiless CLASS is kept // unless a language opts into skipping, a bodiless STRUCT is skipped // unless a language opts into keeping. - const body = getChildByField(node, this.extractor.bodyField); + // + // resolveBody first, as for classes and enums: a VB.NET Structure tags + // every member as its own `body` field, so the field alone yields only + // the first member. + const body = this.extractor.resolveBody?.(node, this.extractor.bodyField) + ?? getChildByField(node, this.extractor.bodyField); if (!body && node.type !== 'record_declaration' && !this.extractor.allowBodilessStruct) return; @@ -2304,6 +2319,28 @@ export class TreeSitterExtractor { return propNode; } + /** + * The parts of a property declaration that run code, for the body walker. + * VB.NET writes its `Get` / `Set` blocks, `= initializer` and `As New T` + * as children of the declaration itself, so the declaration is walked + * whole, the way its methods are (resolveBody). C# runs code in each + * accessor's body (`get { … }`, `set => …`) and in an expression-bodied + * property's `=> …`; its `= initializer`, like a field's, stays unwalked. + * Mirrored in the kernel (csharp.rs property_bodies). + */ + private propertyBodies(node: SyntaxNode): SyntaxNode[] { + if (this.language === 'vbnet') return [node]; + if (this.language !== 'csharp') return []; + const bodies: SyntaxNode[] = []; + for (const accessor of getChildByField(node, 'accessors')?.namedChildren ?? []) { + const body = accessor.type === 'accessor_declaration' ? getChildByField(accessor, 'body') : null; + if (body) bodies.push(body); + } + const value = getChildByField(node, 'value'); + if (value?.type === 'arrow_expression_clause') bodies.push(value); + return bodies; + } + /** * Extract a class field declaration (e.g. Java field_declaration, C# field_declaration). * Extracts each declarator as a 'field' kind node inside the owning class. @@ -2408,10 +2445,11 @@ export class TreeSitterExtractor { // candidates, so a lambda / method reference / anonymous class in // `private final Runnable r = () -> target();` contributed NO call // edge at all and `target` looked callerless. Keyed on the `value` - // FIELD, which only Java's `variable_declarator` carries — C#, - // VB.NET and PHP spell their initializer differently and are - // deliberately untouched here. - const valueNode = getChildByField(decl, 'value'); + // FIELD, which only Java's `variable_declarator` carries. VB.NET + // writes `= expr` (the declarator's `initializer`) or `As New T(…)` + // (inside its as_clause), so its whole declarator is walked. C# + // and PHP spell their initializer differently and are untouched. + const valueNode = this.language === 'vbnet' ? decl : getChildByField(decl, 'value'); if (valueNode) { this.nodeStack.push(fieldNode.id); this.visitFunctionBody(valueNode, fieldNode.id); From 511d86e94dc9f26e74e32741477b6ffaae1695a7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 14:45:10 +0000 Subject: [PATCH 202/363] fix(mcp): replace a daemon left running by an older install (#2335) (#2343) A daemon keeps running the code it started with. The 1.6.1 daemon in #2335 outlived an upgrade, kept loading grammars from the install the upgrade removed and wrote every file it re-indexed as empty, while every session from the new install served itself in-process, read-only, because the old daemon held the writer lock. Daemons from before this change cannot notice an upgrade, so the launcher acts: - A launcher that finds a daemon of an older plain X.Y.Z release stops it the way `codegraph daemon` does (daemon.pid and the socket hello must agree; SIGTERM, SIGKILL only if it still answers) and starts one from its own install. It finds the daemon by its hello on the launcher's own socket, or by the project lock when it listens elsewhere: on Windows, daemons before #2278 named their pipe after the root as typed. A daemon of the same, a newer, a prerelease or an unknown version is never touched, so two installs cannot take turns stopping each other's. - The writer slot never falls free on the way. The launcher swaps writer.pid to itself (mode `handover`) before the signal, clears the old daemon's leftovers while it holds the slot, and the daemon it spawns takes the slot over (CODEGRAPH_DAEMON_HANDOVER). Sessions of the old daemon that fall back in-process find the slot held and serve reads only, instead of claiming it with the removed install's code. A daemon that does not exit gets the slot back, and a successor outwaits a racing candidate that took daemon.pid first. Validated on Windows with real npm upgrades from the published 1.6.1 and 1.6.2 to this build packed as 1.6.3, with the old session calling throughout. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 1 + __tests__/daemon-older-version.test.ts | 716 +++++++++++++++++++++++++ __tests__/fixtures/older-daemon.cjs | 36 ++ src/mcp/daemon-registry.ts | 173 +++++- src/mcp/daemon.ts | 19 +- src/mcp/index.ts | 144 ++++- src/mcp/project-lifecycle.ts | 3 +- src/mcp/proxy.ts | 24 +- src/mcp/version.ts | 25 +- src/mcp/writer-lock.ts | 38 ++ 10 files changed, 1120 insertions(+), 59 deletions(-) create mode 100644 __tests__/daemon-older-version.test.ts create mode 100644 __tests__/fixtures/older-daemon.cjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 44144e1606..32db6b23ab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. +- Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/daemon-older-version.test.ts b/__tests__/daemon-older-version.test.ts new file mode 100644 index 0000000000..188cd6cf82 --- /dev/null +++ b/__tests__/daemon-older-version.test.ts @@ -0,0 +1,716 @@ +/** + * A launcher replaces a daemon from an older install (#2335). + * + * A daemon keeps running the code it started with. The 1.6.1 daemon in #2335 + * outlived an upgrade to 1.6.2, went on loading grammars from the install the + * upgrade removed, and wrote every file its watcher re-indexed as empty. A + * launcher that found it could only serve its own session in-process, read-only: + * the old daemon held the project's writer lock and file watcher, so no daemon + * from the new install could start. Daemons from before the fix cannot notice + * the upgrade themselves, so the launcher acts: a daemon of an OLDER release is + * stopped the way `codegraph daemon` stops one (identity-checked SIGTERM, a + * graceful shutdown on POSIX) and one from the launcher's own install takes its + * place. The launcher hears an older daemon's hello on its own socket, or — + * when it listens elsewhere (on Windows, daemons before #2278 named their pipe + * after the root as typed) — finds it by the project lock. A daemon of the + * same, a newer or an unknown version is never touched, so two installed + * versions cannot fight over the daemon. + * + * The end-to-end cases run real daemons of other versions: this build's dist/ + * linked into a temp "install" whose package.json carries another version. A + * daemon resolves its version, and every grammar it loads, from its own files, + * so that install's daemon advertises that version in its lock and its hello. + */ +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import { ChildProcessWithoutNullStreams, execFileSync, spawn } from 'child_process'; +import { once } from 'events'; +import * as fs from 'fs'; +import * as net from 'net'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import { encodeLockInfo, getDaemonPidPath, getDaemonSocketPath } from '../src/mcp/daemon-paths'; +import { isProcessAlive, stopOlderDaemon } from '../src/mcp/daemon-registry'; +import { connectWithHello } from '../src/mcp/proxy'; +import { CodeGraphPackageVersion, isOlderRelease } from '../src/mcp/version'; +import { getWriterPidPath, readWriterLock, swapWriterLock } from '../src/mcp/writer-lock'; +import { WASM_RUNTIME_FLAGS } from '../src/extraction/wasm-runtime-flags'; +import { recordSpawns, removeSpawnLog, settleLosingCandidates } from './daemon-candidates'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); +const DIST = path.resolve(__dirname, '../dist'); +const NODE_MODULES = path.resolve(__dirname, '../node_modules'); +const OLDER_DAEMON = path.resolve(__dirname, 'fixtures/older-daemon.cjs'); + +/** Plain releases always below / above any version this repo will carry. */ +const OLDER = '0.0.1'; +const NEWER = '999.0.0'; + +describe('isOlderRelease', () => { + it('orders plain releases numerically', () => { + expect(isOlderRelease('1.6.1', '1.6.2')).toBe(true); + expect(isOlderRelease('1.6.9', '1.6.10')).toBe(true); + expect(isOlderRelease('1.9.0', '1.10.0')).toBe(true); + expect(isOlderRelease('0.99.99', '1.0.0')).toBe(true); + expect(isOlderRelease('1.6.2', '1.6.2')).toBe(false); + expect(isOlderRelease('1.6.3', '1.6.2')).toBe(false); + expect(isOlderRelease('2.0.0', '1.99.99')).toBe(false); + }); + + it('never calls a prerelease, a build or an unknown version older, in either position', () => { + for (const odd of ['0.0.0-unknown', '0.0.0-mismatch', '1.6.1-beta.1', '1.6.1+local', 'v1.6.1', 'unknown', 'test', '', '1.6']) { + expect(isOlderRelease(odd, '1.6.2'), odd).toBe(false); + expect(isOlderRelease('1.6.1', odd), odd).toBe(false); + } + }); + + it('lets at most one of two versions replace the other', () => { + const versions = ['0.9.0', '1.6.1', '1.6.2', '1.6.10', '1.7.0', '2.0.0', '1.6.2-rc.1', '0.0.0-unknown']; + for (const a of versions) { + for (const b of versions) { + expect(isOlderRelease(a, b) && isOlderRelease(b, a), `${a} / ${b}`).toBe(false); + } + } + }); +}); + +/** A path a test server can listen on: a named pipe on Windows, a socket file elsewhere. */ +function listenPath(dir: string, name: string): string { + return process.platform === 'win32' + ? `\\\\.\\pipe\\${path.basename(dir)}-${name}` + : path.join(dir, `${name}.sock`); +} + +async function listen(server: net.Server, socketPath: string): Promise { + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(socketPath, resolve); + }); +} + +describe('connectWithHello', () => { + let dir: string; + let server: net.Server | null = null; + + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-hello-')); }); + afterEach(async () => { + if (server) await new Promise((resolve) => server!.close(() => resolve())); + server = null; + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it.each([ + [OLDER, 'older-version', false], + [NEWER, 'version-mismatch', true], + ['0.0.0-unknown', 'version-mismatch', true], + ])('reports a %s daemon as %s', async (version, expected, logged) => { + const socketPath = listenPath(dir, 'daemon'); + server = net.createServer((socket) => { + socket.write(JSON.stringify({ codegraph: version, pid: process.pid, socketPath, protocol: 1 }) + '\n'); + }); + await listen(server, socketPath); + const stderr = vi.spyOn(process.stderr, 'write').mockImplementation(() => true); + try { + expect(await connectWithHello(socketPath, '1.6.2')).toBe(expected); + // An older daemon is the launcher's to report; it is not a mismatch to serve around. + expect(stderr.mock.calls.some(([line]) => String(line).includes('differs from ours'))).toBe(logged); + } finally { + stderr.mockRestore(); + } + }); +}); + +describe('swapWriterLock', () => { + let root: string; + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-writer-swap-')); + fs.mkdirSync(path.join(root, '.codegraph')); + }); + afterEach(() => { fs.rmSync(root, { recursive: true, force: true }); }); + + const record = (pid: number, mode: string) => ({ pid, mode, startedAt: 1, ready: false }); + + it('replaces the record of the given pid, and only that record', () => { + const file = getWriterPidPath(root); + fs.writeFileSync(file, JSON.stringify(record(111, 'daemon')) + '\n'); + expect(swapWriterLock(root, 222, record(333, 'handover'))).toBe(false); + expect(readWriterLock(root)).toMatchObject({ pid: 111, mode: 'daemon' }); + expect(swapWriterLock(root, 111, record(333, 'handover'))).toBe(true); + expect(readWriterLock(root)).toMatchObject({ pid: 333, mode: 'handover' }); + // A slot nobody holds is not handed over: it is acquired the usual way. + fs.rmSync(file); + expect(swapWriterLock(root, 333, record(444, 'daemon'))).toBe(false); + expect(fs.existsSync(file)).toBe(false); + expect(fs.readdirSync(path.join(root, '.codegraph'))).toEqual([]); + }); +}); + +/** A live process to stand in for a daemon; signals reach it, never the test. */ +function startDetachedProcess(): number { + const source = [ + "const { spawn } = require('child_process');", + "const child = spawn(process.execPath, ['-e', 'setInterval(() => {}, 1000)'], { detached: true, stdio: 'ignore' });", + 'child.unref();', + 'console.log(child.pid);', + ].join(' '); + const pid = Number(execFileSync(process.execPath, ['-e', source], { encoding: 'utf8' }).trim()); + if (!Number.isInteger(pid) || pid <= 0) throw new Error('could not start daemon fixture'); + return pid; +} + +async function deadPid(): Promise { + const child = spawn(process.execPath, ['-e', 'process.exit(0)']); + await once(child, 'exit'); + await new Promise((r) => setTimeout(r, 50)); + return child.pid!; +} + +describe('stopOlderDaemon', () => { + let root: string; + let pidPath: string; + let socketPath: string; + let server: net.Server | null; + let pids: number[]; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-older-daemon-')); + fs.mkdirSync(path.join(root, '.codegraph')); + pidPath = getDaemonPidPath(root); + socketPath = getDaemonSocketPath(root); + server = null; + pids = []; + }); + + afterEach(async () => { + vi.restoreAllMocks(); + for (const pid of pids) if (isProcessAlive(pid)) process.kill(pid, 'SIGKILL'); + if (server) await new Promise((resolve) => server!.close(() => resolve())); + fs.rmSync(root, { recursive: true, force: true }); + }); + + /** A daemon of `version`: its lock, and a socket answering the hello as `helloPid`. */ + async function fakeDaemon(version: string, helloPid?: number): Promise<{ pid: number; lock: string }> { + const pid = startDetachedProcess(); + pids.push(pid); + server = net.createServer((socket) => { + socket.end(JSON.stringify({ protocol: 1, codegraph: version, pid: helloPid ?? pid, socketPath }) + '\n'); + }); + await listen(server, socketPath); + const lock = encodeLockInfo({ pid, version, socketPath, startedAt: Date.now() }); + fs.writeFileSync(pidPath, lock); + return { pid, lock }; + } + + /** Signals sent to `pid`, other than the signal-0 liveness probe. */ + function signalsTo(kill: ReturnType, pid: number): unknown[] { + return kill.mock.calls.filter(([target, signal]) => target === pid && signal !== 0).map(([, signal]) => signal); + } + + /** The writer record, decoded. */ + const writerRecord = (): any => { + try { return JSON.parse(fs.readFileSync(getWriterPidPath(root), 'utf8')); } catch { return null; } + }; + + it('stops a verified daemon of an older release and clears its lock', async () => { + const { pid } = await fakeDaemon('1.6.1'); + const result = await stopOlderDaemon(root, '1.6.2'); + expect(result).toMatchObject({ pid, outcome: 'term', version: '1.6.1' }); + expect(isProcessAlive(pid)).toBe(false); + expect(fs.existsSync(pidPath)).toBe(false); + // Kept for the daemon the caller starts next. + expect(writerRecord()).toMatchObject({ pid: process.pid, mode: 'handover' }); + }, 15000); + + it('holds the writer slot from before the signal, so it is never free or stale', async () => { + const { pid } = await fakeDaemon('1.6.1'); + fs.writeFileSync(getWriterPidPath(root), JSON.stringify({ pid, mode: 'daemon', startedAt: 1, ready: true }) + '\n'); + let atSignal: unknown = null; + const originalKill = process.kill.bind(process); + vi.spyOn(process, 'kill').mockImplementation((target, signal) => { + if (target === pid && signal === 'SIGTERM') atSignal = writerRecord(); + return originalKill(target, signal); + }); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ pid, outcome: 'term' }); + // A session of the old daemon serving itself the moment it went found the + // slot held, and so served reads only. + expect(atSignal).toMatchObject({ pid: process.pid, mode: 'handover' }); + expect(writerRecord()).toMatchObject({ pid: process.pid, mode: 'handover' }); + expect(fs.existsSync(pidPath)).toBe(false); + }, 15000); + + it('gives the writer slot back to an older daemon that does not exit', async () => { + const { pid, lock } = await fakeDaemon('1.6.1'); + const record = { pid, mode: 'daemon', startedAt: 1, ready: true }; + fs.writeFileSync(getWriterPidPath(root), JSON.stringify(record) + '\n'); + // The signal reaches nothing: the daemon closes its socket, as its own + // shutdown does first, and then never exits. + const originalKill = process.kill.bind(process); + vi.spyOn(process, 'kill').mockImplementation((target, signal) => { + if (target === pid && signal === 'SIGTERM') { server?.close(); return true; } + return originalKill(target, signal); + }); + expect(await stopOlderDaemon(root, '1.6.2', { shutdownGraceMs: 300 })).toMatchObject({ pid, outcome: 'still-running' }); + expect(writerRecord()).toEqual(record); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + expect(isProcessAlive(pid)).toBe(true); + }, 15000); + + it('leaves the daemon to another launcher already holding the slot to replace it', async () => { + const { pid, lock } = await fakeDaemon('1.6.1'); + const other = startDetachedProcess(); + pids.push(other); + const claim = { pid: other, mode: 'handover', startedAt: 1, ready: false }; + fs.writeFileSync(getWriterPidPath(root), JSON.stringify(claim) + '\n'); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(signalsTo(kill, pid)).toEqual([]); + expect(writerRecord()).toEqual(claim); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it.each(['1.6.2', '1.6.3', '2.0.0', '1.6.1-beta.1', '0.0.0-unknown', 'test'])( + 'never signals a daemon of version %s', + async (version) => { + const { pid, lock } = await fakeDaemon(version); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(signalsTo(kill, pid)).toEqual([]); + expect(isProcessAlive(pid)).toBe(true); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }, + ); + + it('leaves an older lock alone when its socket answers for another process (#1553)', async () => { + const { pid, lock } = await fakeDaemon('1.6.1', process.pid); + const kill = vi.spyOn(process, 'kill'); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ pid, outcome: 'unverified' }); + expect(signalsTo(kill, pid)).toEqual([]); + expect(signalsTo(kill, process.pid)).toEqual([]); + expect(isProcessAlive(pid)).toBe(true); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it('reports an older daemon that already exited, leaving its lock to its successor', async () => { + const lock = encodeLockInfo({ pid: await deadPid(), version: '1.6.1', socketPath, startedAt: Date.now() }); + fs.writeFileSync(pidPath, lock); + expect(await stopOlderDaemon(root, '1.6.2')).toMatchObject({ outcome: 'not-running', version: '1.6.1' }); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(lock); + }); + + it('has nothing to stop without a lock, and leaves a listening socket alone', async () => { + server = net.createServer((socket) => { + socket.end(JSON.stringify({ protocol: 1, codegraph: '1.6.1', pid: process.pid, socketPath }) + '\n'); + }); + await listen(server, socketPath); + expect(await stopOlderDaemon(root, '1.6.2')).toBeNull(); + expect(await connectWithHello(socketPath, '1.6.2')).toBe('older-version'); + }); + + it('does not act on an unreadable lock', async () => { + fs.mkdirSync(pidPath); + expect(await stopOlderDaemon(root, '1.6.2')).toEqual({ root, pid: null, outcome: 'unverified' }); + }); +}); + +/** + * A CodeGraph install of `version`: this build's dist/, hard-linked file by + * file (copied where a link is refused), next to a package.json of that version + * and a link to this checkout's node_modules. + */ +function makeInstall(version: string): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), `cg-install-${version}-`)); + const linkTree = (from: string, to: string): void => { + fs.mkdirSync(to, { recursive: true }); + for (const entry of fs.readdirSync(from, { withFileTypes: true })) { + const source = path.join(from, entry.name); + const target = path.join(to, entry.name); + if (entry.isDirectory()) linkTree(source, target); + else { + try { fs.linkSync(source, target); } catch { fs.copyFileSync(source, target); } + } + } + }; + linkTree(DIST, path.join(dir, 'dist')); + fs.symlinkSync(NODE_MODULES, path.join(dir, 'node_modules'), 'junction'); + fs.writeFileSync(path.join(dir, 'package.json'), JSON.stringify({ name: '@colbymchenry/codegraph', version }) + '\n'); + return dir; +} + +interface Session { + child: ChildProcessWithoutNullStreams; + stdout: string[]; + stderr: string[]; +} + +function lines(stream: NodeJS.ReadableStream, into: string[]): void { + let buffer = ''; + stream.on('data', (chunk: Buffer) => { + buffer += chunk.toString('utf8'); + let idx: number; + while ((idx = buffer.indexOf('\n')) !== -1) { + into.push(buffer.slice(0, idx)); + buffer = buffer.slice(idx + 1); + } + }); +} + +function findResponse(stdout: string[], id: number): any | null { + for (const line of stdout) { + try { + const parsed = JSON.parse(line); + if (parsed && parsed.id === id && (parsed.result !== undefined || parsed.error !== undefined)) return parsed; + } catch { /* not JSON */ } + } + return null; +} + +function waitFor(predicate: () => T | undefined | null | false, timeoutMs: number, label: string): Promise { + return new Promise((resolve, reject) => { + const started = Date.now(); + const tick = (): void => { + let value: T | undefined | null | false; + try { value = predicate(); } catch (e) { reject(e); return; } + if (value) { resolve(value as T); return; } + if (Date.now() - started > timeoutMs) { reject(new Error(`Timed out after ${timeoutMs}ms waiting for: ${label}`)); return; } + setTimeout(tick, 25); + }; + tick(); + }); +} + +function isAlive(pid: number): boolean { + try { process.kill(pid, 0); return true; } catch { return false; } +} + +async function waitProcessExit(pid: number, timeoutMs: number): Promise { + return waitFor(() => !isAlive(pid), timeoutMs, `pid ${pid} to exit`).then(() => true, () => false); +} + +function readJson(file: string): any | null { + try { return JSON.parse(fs.readFileSync(file, 'utf8')); } catch { return null; } +} + +describe('a launcher meeting a daemon of another version (#2335)', () => { + let olderInstall: string; + let newerInstall: string; + let tempDir: string; + let realRoot: string; + const sessions: Session[] = []; + const daemonPids = new Set(); + + beforeAll(() => { + olderInstall = makeInstall(OLDER); + newerInstall = makeInstall(NEWER); + }); + + afterAll(async () => { + for (const dir of [olderInstall, newerInstall]) { + if (!dir) continue; + // The link first: removing an install must never reach this checkout's modules. + try { fs.unlinkSync(path.join(dir, 'node_modules')); } catch { /* not created */ } + await fs.promises.rm(dir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }); + + beforeEach(async () => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-older-version-')); + fs.writeFileSync(path.join(tempDir, 'app.ts'), 'export function appMain() { return 1; }\n'); + const cg = await CodeGraph.init(tempDir); + try { await cg.indexAll(); } finally { cg.close(); } + realRoot = fs.realpathSync(tempDir); + }); + + afterEach(async () => { + // Every launcher first (with the runtime flags there is no relaunch child), + // then the losing daemon candidates, then the daemons themselves — and only + // then the fixture: on Windows a live process inside it blocks the removal. + await Promise.all(sessions.map(async ({ child }) => { + if (child.exitCode !== null || child.signalCode !== null) return; + const exited = once(child, 'exit'); + child.kill('SIGKILL'); + await exited; + })); + sessions.length = 0; + const lockPid = (): number | null => readJson(path.join(realRoot, '.codegraph', 'daemon.pid'))?.pid ?? null; + await settleLosingCandidates(tempDir, lockPid); + const holder = lockPid(); + if (holder) daemonPids.add(holder); + for (const pid of daemonPids) { + if (pid === process.pid || !isAlive(pid)) continue; + try { process.kill(pid, 'SIGKILL'); } catch { /* raced to exit */ } + await waitProcessExit(pid, 5000); + } + daemonPids.clear(); + removeSpawnLog(tempDir); + await fs.promises.rm(tempDir, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + }, 45_000); + + /** An agent session: `install`'s launcher, `serve --mcp` in the project. */ + function startSession(install: string | null, env: NodeJS.ProcessEnv = {}): Session { + const bin = install ? path.join(install, 'dist', 'bin', 'codegraph.js') : BIN; + const recorder = recordSpawns(tempDir); + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, ...recorder.args, bin, 'serve', '--mcp'], { + cwd: tempDir, + stdio: ['pipe', 'pipe', 'pipe'], + env: { + ...process.env, + CODEGRAPH_MCP_LOG_ATTACH: '1', + CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', + ...recorder.env, + ...env, + }, + }) as ChildProcessWithoutNullStreams; + child.on('error', () => { /* ignore */ }); + child.stdin.on('error', () => { /* ignore */ }); + const session: Session = { child, stdout: [], stderr: [] }; + lines(child.stdout, session.stdout); + lines(child.stderr, session.stderr); + sessions.push(session); + send(session, { + jsonrpc: '2.0', id: 1, method: 'initialize', + params: { protocolVersion: '2024-11-05', capabilities: {}, clientInfo: { name: 'test', version: '0.0.0' }, rootUri: `file://${tempDir}` }, + }); + return session; + } + + function send(session: Session, msg: unknown): void { + try { session.child.stdin.write(JSON.stringify(msg) + '\n'); } catch { /* gone */ } + } + + async function status(session: Session, id: number, label: string): Promise { + send(session, { jsonrpc: '2.0', id, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + const reply = await waitFor(() => findResponse(session.stdout, id), 20000, label).catch((err: Error) => { + throw new Error(`${err.message}\n--- stderr ---\n${session.stderr.join('\n')}\n--- daemon.log ---\n${daemonLog()}`); + }); + expect(reply.error, label).toBeUndefined(); + expect(reply.result?.isError, label).not.toBe(true); + expect(JSON.stringify(reply.result), label).toContain('CodeGraph Status'); + return reply; + } + + const attachedLine = (session: Session): string | undefined => + session.stderr.find((l) => l.includes('Attached to shared daemon')); + const daemonLock = (): any => readJson(path.join(realRoot, '.codegraph', 'daemon.pid')); + const writerLock = (): any => readJson(path.join(realRoot, '.codegraph', 'writer.pid')); + const daemonLog = (): string => { + try { return fs.readFileSync(path.join(realRoot, '.codegraph', 'daemon.log'), 'utf8'); } catch { return ''; } + }; + + /** A session of `install` attached to a daemon of its own, which owns the project. */ + async function sessionWithDaemon(install: string, version: string, env: NodeJS.ProcessEnv = {}): Promise<{ session: Session; pid: number }> { + const session = startSession(install, env); + await waitFor(() => attachedLine(session), 20000, `the ${version} session to attach`); + // Attached means listening, not initialized: let the engine open the + // database before anything stops this daemon. + await status(session, 2, `a ${version} tool call`); + const lock = daemonLock(); + expect(lock).toMatchObject({ version }); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + daemonPids.add(lock.pid); + return { session, pid: lock.pid }; + } + + /** A daemon process for the project, started the way a launcher starts one. */ + function startDaemon(env: NodeJS.ProcessEnv): { child: ChildProcessWithoutNullStreams; log: string[] } { + const child = spawn(process.execPath, [...WASM_RUNTIME_FLAGS, BIN, 'serve', '--mcp', '--path', realRoot], { + cwd: tempDir, + stdio: ['pipe', 'pipe', 'pipe'], + env: { ...process.env, CODEGRAPH_DAEMON_INTERNAL: '1', CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS: '30000', ...env }, + }) as ChildProcessWithoutNullStreams; + child.on('error', () => { /* ignore */ }); + child.stdout.resume(); + const log: string[] = []; + lines(child.stderr, log); + daemonPids.add(child.pid!); + return { child, log }; + } + + /** The slot as a launcher that just stopped an older daemon holds it: this test process stands in. */ + function holdWriterSlotForSuccessor(): string { + const claim = JSON.stringify({ pid: process.pid, mode: 'handover', startedAt: Date.now(), ready: false }) + '\n'; + fs.writeFileSync(path.join(realRoot, '.codegraph', 'writer.pid'), claim); + return claim; + } + + it('a daemon takes over the writer slot held for it, and only that one', async () => { + const claim = holdWriterSlotForSuccessor(); + + // A daemon the slot is not held for finds it held, as by any live writer. + const stranger = startDaemon({}); + const [strangerCode] = await once(stranger.child, 'exit'); + expect(strangerCode).toBe(1); + expect(stranger.log.join('\n')).toContain(`writer lock held by PID ${process.pid} (handover mode)`); + expect(fs.readFileSync(path.join(realRoot, '.codegraph', 'writer.pid'), 'utf8')).toBe(claim); + + const successor = startDaemon({ CODEGRAPH_DAEMON_HANDOVER: String(process.pid) }); + await waitFor( + () => daemonLock()?.pid === successor.child.pid && successor.log.some((l) => l.includes('Listening on')), + 20000, 'the successor to listen', + ); + expect(writerLock()).toMatchObject({ pid: successor.child.pid, mode: 'daemon' }); + expect(successor.log.join('\n')).toContain(`Took over the writer lock from launcher pid ${process.pid}`); + }, 60000); + + it('a daemon the slot is held for outwaits a candidate still starting in its way', async () => { + holdWriterSlotForSuccessor(); + // A racing launcher's candidate holds the daemon lock, still starting; it + // cannot get the writer slot, so it will give up. + const candidate = startDetachedProcess(); + daemonPids.add(candidate); + const pidPath = path.join(realRoot, '.codegraph', 'daemon.pid'); + const candidateLock = encodeLockInfo({ + pid: candidate, version: CodeGraphPackageVersion, socketPath: getDaemonSocketPath(realRoot), startedAt: Date.now(), + }); + fs.writeFileSync(pidPath, candidateLock); + + // Any other daemon yields to it at once. + const yielding = startDaemon({}); + const [yieldingCode] = await once(yielding.child, 'exit'); + expect(yieldingCode).toBe(0); + expect(yielding.log.join('\n')).toContain(`Another daemon (pid ${candidate}) already holds the lock`); + + const successor = startDaemon({ CODEGRAPH_DAEMON_HANDOVER: String(process.pid) }); + await waitFor( + () => successor.log.some((l) => l.includes(`Waiting for daemon candidate pid ${candidate}`)), + 20000, 'the successor to wait for the candidate', + ); + expect(successor.child.exitCode).toBeNull(); + expect(fs.readFileSync(pidPath, 'utf8')).toBe(candidateLock); + // The candidate gives up: its lock goes, then so does it. + fs.rmSync(pidPath); + process.kill(candidate, 'SIGKILL'); + await waitFor( + () => daemonLock()?.pid === successor.child.pid && successor.log.some((l) => l.includes('Listening on')), + 20000, 'the successor to listen', + ); + expect(writerLock()).toMatchObject({ pid: successor.child.pid, mode: 'daemon' }); + }, 60000); + + it('keeps the older daemon\'s sessions read-only through the replacement, even mid-call', async () => { + const before = await sessionWithDaemon(olderInstall, OLDER); + // The older session calls every 20ms throughout, so it serves itself + // in-process the instant its daemon goes. Finding the writer slot free + // then, it would claim it — with the code of the install being replaced. + let next = 100; + const calling = setInterval(() => { + send(before.session, { jsonrpc: '2.0', id: next++, method: 'tools/call', params: { name: 'codegraph_status', arguments: {} } }); + }, 20); + let after: Session; + try { + after = startSession(null); + await waitFor(() => attachedLine(after), 30000, 'the new session to attach'); + await new Promise((r) => setTimeout(r, 500)); + } finally { + clearInterval(calling); + } + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + await status(after, 2, 'a tool call through the new daemon'); + + const last = next - 1; + await waitFor(() => findResponse(before.session.stdout, last), 20000, 'the older session to answer every call'); + for (let id = 100; id <= last; id++) expect(findResponse(before.session.stdout, id), `call ${id}`).toBeTruthy(); + const stderr = before.session.stderr.join('\n'); + expect(stderr).toContain('Serving reads in-process without auto-sync'); + expect(stderr).not.toContain('File watcher active'); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + }, 120000); + + it('stops a daemon from an older install and starts one from its own', async () => { + const before = await sessionWithDaemon(olderInstall, OLDER); + + const after = startSession(null); + const stopped = await waitFor( + () => after.stderr.find((l) => l.includes('[CodeGraph MCP] Stopped')), + 30000, 'the older daemon to be stopped', + ); + expect(stopped).toContain(`the CodeGraph ${OLDER} daemon (pid ${before.pid})`); + expect(stopped).toContain(`from this install (${CodeGraphPackageVersion})`); + expect(await waitProcessExit(before.pid, 10000)).toBe(true); + + const attached = await waitFor(() => attachedLine(after), 20000, 'the new session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(lock.pid).not.toBe(before.pid); + expect(attached).toContain(`(pid ${lock.pid}, v${CodeGraphPackageVersion})`); + await status(after, 2, 'a tool call through the new daemon'); + // The replacement owns updates for the project: writer lock and watcher, + // handed to it by the launcher that held the slot through the stop. + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLog()).toContain(`Took over the writer lock from launcher pid ${after.child.pid}`); + if (process.platform !== 'win32') { + // SIGTERM ran the old daemon's own shutdown. (On Windows it is + // TerminateProcess, and the stop clears what is left.) + expect(daemonLog()).toContain(`Shutting down (SIGTERM`); + } + + // The session from before the upgrade keeps answering, read-only, and + // leaves the new daemon its writer lock. + await waitFor( + () => before.session.stderr.some((l) => l.includes('Shared daemon connection lost')), + 10000, 'the older session to lose its daemon', + ); + await status(before.session, 3, 'a tool call in the older session'); + expect(before.session.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLock()).toEqual(lock); + expect(isAlive(lock.pid)).toBe(true); + }, 120000); + + it('finds an older daemon its probe cannot reach by the project lock, and stops it', async () => { + // On Windows a 1.6.1 daemon named its pipe after the root as typed (#2278), + // so a newer launcher's probe never hears its hello; only its lock says + // where it listens. Stand one up on a socket no launcher probes. + const socketDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sock-')); + try { + const socketPath = listenPath(socketDir, 'older'); + const older = spawn(process.execPath, [OLDER_DAEMON, realRoot, OLDER, socketPath], { detached: true, stdio: 'ignore' }); + older.unref(); + daemonPids.add(older.pid!); + await waitFor(() => daemonLock()?.pid === older.pid, 10000, 'the older daemon to hold the project'); + expect(writerLock()).toMatchObject({ pid: older.pid, mode: 'daemon' }); + + const session = startSession(null); + const stopped = await waitFor( + () => session.stderr.find((l) => l.includes('[CodeGraph MCP] Stopped')), + 30000, 'the older daemon to be stopped', + ); + expect(stopped).toContain(`the CodeGraph ${OLDER} daemon (pid ${older.pid})`); + expect(await waitProcessExit(older.pid!, 10000)).toBe(true); + const attached = await waitFor(() => attachedLine(session), 20000, 'the session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(lock.version).toBe(CodeGraphPackageVersion); + expect(attached).toContain(`(pid ${lock.pid}, v${CodeGraphPackageVersion})`); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + expect(daemonLog()).toContain(`Took over the writer lock from launcher pid ${session.child.pid}`); + await status(session, 2, 'a tool call through the new daemon'); + } finally { + fs.rmSync(socketDir, { recursive: true, force: true }); + } + }, 90000); + + it('never stops a daemon from a newer install', async () => { + const retry = { CODEGRAPH_DAEMON_RETRY_MS: '300', CODEGRAPH_DAEMON_RETRY_MAX_MS: '600' }; + const newer = await sessionWithDaemon(newerInstall, NEWER); + const lock = daemonLock(); + + const current = startSession(null, retry); + // The first probe and the retries that follow all find the newer daemon. + await waitFor( + () => current.stderr.filter((l) => l.includes(`version (${NEWER}) differs from ours`)).length >= 3, + 20000, 'repeated probes of the newer daemon', + ); + await status(current, 2, 'a tool call served in-process'); + expect(current.stderr.some((l) => l.includes('Serving reads in-process without auto-sync'))).toBe(true); + expect(current.stderr.some((l) => l.includes('[CodeGraph MCP] Stopped'))).toBe(false); + + expect(isAlive(newer.pid)).toBe(true); + expect(daemonLock()).toEqual(lock); + expect(writerLock()).toMatchObject({ pid: newer.pid, mode: 'daemon' }); + await status(newer.session, 3, 'a tool call through the newer daemon'); + }, 90000); +}); diff --git a/__tests__/fixtures/older-daemon.cjs b/__tests__/fixtures/older-daemon.cjs new file mode 100644 index 0000000000..a6535f4d9b --- /dev/null +++ b/__tests__/fixtures/older-daemon.cjs @@ -0,0 +1,36 @@ +/** + * A daemon of an older release, listening where a current launcher does not + * look (#2335). On Windows, daemons before #2278 named their pipe after the + * project root as typed, so a newer launcher's probe never reaches them and + * only the project lock says where they are. Like a real daemon it holds the + * project's daemon and writer locks and answers the hello; on SIGTERM it lets + * go of whichever of the two locks still names it and exits, the shape of a + * real daemon's shutdown. (On Windows SIGTERM is TerminateProcess, and the + * stop clears what is left.) + * + * node older-daemon.cjs + */ +const fs = require('fs'); +const net = require('net'); +const path = require('path'); + +const [root, version, socketPath] = process.argv.slice(2); +const dir = path.join(root, '.codegraph'); +const writeLock = (name, record) => fs.writeFileSync(path.join(dir, name), JSON.stringify(record) + '\n'); + +const server = net.createServer((socket) => { + socket.end(JSON.stringify({ codegraph: version, pid: process.pid, socketPath, protocol: 1 }) + '\n'); +}); +server.listen(socketPath, () => { + writeLock('writer.pid', { pid: process.pid, mode: 'daemon', startedAt: Date.now(), ready: true }); + writeLock('daemon.pid', { pid: process.pid, version, socketPath, startedAt: Date.now() }); +}); +process.on('SIGTERM', () => { + for (const name of ['writer.pid', 'daemon.pid']) { + const file = path.join(dir, name); + try { + if (JSON.parse(fs.readFileSync(file, 'utf8')).pid === process.pid) fs.unlinkSync(file); + } catch { /* already gone */ } + } + server.close(() => process.exit(0)); +}); diff --git a/src/mcp/daemon-registry.ts b/src/mcp/daemon-registry.ts index aabb79139b..a9e5860116 100644 --- a/src/mcp/daemon-registry.ts +++ b/src/mcp/daemon-registry.ts @@ -31,7 +31,14 @@ import { probeDaemonIdentity, type DaemonLockInfo, } from './daemon-paths'; -import { readWriterLock, releaseWriterLock, tryAcquireWriterLock } from './writer-lock'; +import { + readWriterLock, + releaseWriterLock, + swapWriterLock, + tryAcquireWriterLock, + type WriterLockInfo, +} from './writer-lock'; +import { isOlderRelease } from './version'; import { WORKER_START_SETTLE_MS } from '../worker-teardown'; export interface DaemonRecord { @@ -155,7 +162,6 @@ function cleanupDaemonArtifacts( root: string, expectedLockContents: string | null, ): boolean { - const pidPath = getDaemonPidPath(root); // A daemon owns writer.pid before binding or relocating its socket. Claiming // the writer slot therefore freezes every legitimate daemon artifact writer // while we compare the inspected lock snapshot and clean it up. @@ -164,33 +170,39 @@ function cleanupDaemonArtifacts( if (claim.kind === 'taken') return false; try { - if (expectedLockContents === null) { - if (fs.existsSync(pidPath)) return false; - } else { - try { - if (fs.readFileSync(pidPath, 'utf8') !== expectedLockContents) return false; - } catch { - return false; - } - } - // POSIX sockets are real files; Windows named pipes vanish with the process. - // Sweep every candidate before releasing daemon.pid, so no successor can - // acquire the lock and bind a socket that this cleanup then removes. - if (process.platform !== 'win32') { - for (const candidate of getDaemonSocketCandidates(root)) { - try { fs.unlinkSync(candidate); } catch { /* gone */ } - } - } - deregisterDaemon(root); - try { fs.unlinkSync(pidPath); } catch (err) { - if ((err as NodeJS.ErrnoException).code !== 'ENOENT') return false; - } - return true; + return removeDaemonArtifacts(root, expectedLockContents); } finally { releaseWriterLock(root); } } +/** The removal behind {@link cleanupDaemonArtifacts}, for a caller already holding the writer slot. */ +function removeDaemonArtifacts(root: string, expectedLockContents: string | null): boolean { + const pidPath = getDaemonPidPath(root); + if (expectedLockContents === null) { + if (fs.existsSync(pidPath)) return false; + } else { + try { + if (fs.readFileSync(pidPath, 'utf8') !== expectedLockContents) return false; + } catch { + return false; + } + } + // POSIX sockets are real files; Windows named pipes vanish with the process. + // Sweep every candidate before releasing daemon.pid, so no successor can + // acquire the lock and bind a socket that this cleanup then removes. + if (process.platform !== 'win32') { + for (const candidate of getDaemonSocketCandidates(root)) { + try { fs.unlinkSync(candidate); } catch { /* gone */ } + } + } + deregisterDaemon(root); + try { fs.unlinkSync(pidPath); } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') return false; + } + return true; +} + /** Remove daemon artifacts only when no matching daemon answers the socket hello. */ export async function clearStaleDaemonArtifacts(root: string): Promise { const pidPath = getDaemonPidPath(root); @@ -240,6 +252,8 @@ export interface StopResult { pid: number | null; /** 'term' graceful, 'kill' force, 'still-running' refused, 'not-running' stale, 'no-daemon' absent, 'unverified' preserved. */ outcome: 'term' | 'kill' | 'still-running' | 'not-running' | 'no-daemon' | 'unverified'; + /** The daemon's version as its lock recorded it, when the stop got that far. */ + version?: string; } /** @@ -292,13 +306,27 @@ export async function stopDaemonAt( const removed = cleanupDaemonArtifacts(root, lockContents); return { root, pid, outcome: removed ? 'not-running' : 'unverified' }; } + return stopVerifiedDaemon(root, identity, lockContents, options.shutdownGraceMs); +} +/** + * The signalling half of a stop, for a daemon whose socket hello just proved + * `identity` against the lock read as `lockContents`: SIGTERM, wait, SIGKILL + * only one that still answers, then sweep its artifacts. + */ +async function stopVerifiedDaemon( + root: string, + identity: DaemonLockInfo, + lockContents: string | null, + shutdownGraceMs = DAEMON_SHUTDOWN_GRACE_MS, +): Promise { + const { pid, version } = identity; // Identity probing awaits I/O: never act on a superseded ownership record. const sameLock = (): boolean => { try { return fs.readFileSync(getDaemonPidPath(root), 'utf8') === lockContents; } catch { return lockContents === null; } }; - if (!sameLock()) return { root, pid, outcome: 'unverified' }; + if (!sameLock()) return { root, pid, outcome: 'unverified', version }; // POSIX: SIGTERM runs the daemon's graceful shutdown. Windows: TerminateProcess // (no graceful path), so we always sweep artifacts ourselves below. @@ -309,7 +337,7 @@ export async function stopDaemonAt( if (sameLock() && await probeDaemonIdentity(identity) && sameLock()) { try { process.kill(pid, 'SIGKILL'); } catch { /* raced to exit */ } if (!(await waitForDeath(pid, 2000))) { - return { root, pid, outcome: 'still-running' }; + return { root, pid, outcome: 'still-running', version }; } outcome = 'kill'; } else { @@ -321,14 +349,101 @@ export async function stopDaemonAt( // Its identity can't be re-proven without the socket, so signal nothing // more; just wait, bounded, for the PID to go. One that outlives the // wait is reported, as before. - if (!(await waitForDeath(pid, options.shutdownGraceMs ?? DAEMON_SHUTDOWN_GRACE_MS))) { - return { root, pid, outcome: 'still-running' }; + if (!(await waitForDeath(pid, shutdownGraceMs))) { + return { root, pid, outcome: 'still-running', version }; } } } // Compares the lock with the one we signalled, so a successor's is kept. cleanupDaemonArtifacts(root, lockContents); - return { root, pid, outcome }; + return { root, pid, outcome, version }; +} + +/** + * Stop the daemon serving `root` when it runs an older CodeGraph release than + * `version`, so the caller can start one from its own install (#2335). A + * daemon keeps running the code it started with: one that outlived an upgrade + * goes on loading grammars from files the upgrade removed, while it holds the + * project's writer lock and file watcher. Only a daemon whose lock records an + * older release ({@link isOlderRelease}) and whose socket hello confirms that + * lock is signalled, exactly as {@link stopDaemonAt} would; one of the same, a + * newer or an unknown version is never touched, so two installs cannot take + * turns stopping each other's daemon. + * + * The project's writer slot never falls free on the way: before the signal, + * this process takes it over from the old daemon (mode `handover`), and the + * caller hands it on to the daemon it starts, which takes it over in turn + * ({@link swapWriterLock}). The old daemon's sessions serve themselves + * in-process the moment it goes, and one that found the slot free would claim + * it as their writer — with the code of an install the upgrade removed — and + * keep the new daemon from starting. When the old daemon is not stopped, the + * slot goes back to it. + * + * Resolves null when the lock names no older daemon (there is none, or another + * launcher already replaced it) or the slot could not be taken (another + * launcher holds it to replace the same daemon). Otherwise says what became of + * the daemon: `term` or `kill` when it was stopped, and this process now holds + * the slot for its successor (release it with {@link releaseWriterLock} once + * that one has taken over or failed); `not-running` when it had already exited + * (its successor clears the stale lock); `unverified` or `still-running` when + * it is still there. + */ +export async function stopOlderDaemon( + root: string, + version: string, + options: { shutdownGraceMs?: number } = {}, +): Promise { + let lockContents: string; + try { + lockContents = fs.readFileSync(getDaemonPidPath(root), 'utf8'); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'ENOENT') return null; + return { root, pid: null, outcome: 'unverified' }; + } + const identity = decodeLockInfo(lockContents); + if (!identity) return { root, pid: null, outcome: 'unverified' }; + if (!isOlderRelease(identity.version, version)) return null; + const { pid } = identity; + if (!isProcessAlive(pid)) return { root, pid, outcome: 'not-running', version: identity.version }; + // The hello must name this pid and version (#1553): a reused pid is no daemon. + if (!canProbeDaemonIdentity(identity) || !await probeDaemonIdentity(identity)) { + return { root, pid, outcome: 'unverified', version: identity.version }; + } + const slot = takeWriterSlotFrom(root, pid); + if (!slot) return null; + const result = await stopVerifiedDaemon(root, identity, lockContents, options.shutdownGraceMs); + if (result.outcome === 'term' || result.outcome === 'kill') { + // The stop's own sweep needs the slot this process now holds: clear what + // the old daemon left (all of it on Windows, where the stop is + // TerminateProcess), so its successor starts on a clean lock. + removeDaemonArtifacts(root, lockContents); + } else { + slot.giveBack(); + } + return result; +} + +/** + * Take the project's writer slot from the daemon `pid` this process is about + * to stop (see {@link stopOlderDaemon}), whether the record is that daemon's, + * stale, or absent. Returns how to give it back to that daemon, or null when + * the slot could not be taken: another live process holds it (another + * launcher replacing the same daemon), or the swap lost a race. + */ +function takeWriterSlotFrom(root: string, pid: number): { giveBack(): void } | null { + const previous = readWriterLock(root); + if (previous && previous.pid !== pid && previous.pid !== process.pid && isProcessAlive(previous.pid)) return null; + const claim: WriterLockInfo = { pid: process.pid, mode: 'handover', startedAt: Date.now(), ready: false }; + const held = previous + ? swapWriterLock(root, previous.pid, claim) + : tryAcquireWriterLock(root, 'handover').kind === 'acquired'; + if (!held) return null; + return { + giveBack: () => { + if (previous?.pid === pid) swapWriterLock(root, process.pid, previous); + else releaseWriterLock(root); + }, + }; } /** Stop every registered, live daemon. */ diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index 5f44ff6e85..73a4969598 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -57,9 +57,11 @@ import { import { CodeGraphPackageVersion } from './version'; import { releaseWriterLock, + swapWriterLock, tryAcquireWriterLock, assertNoRebuild, writerLockHeldMessage, + type WriterLockInfo, } from './writer-lock'; import { registerDaemon, deregisterDaemon } from './daemon-registry'; @@ -187,15 +189,18 @@ export class Daemon { private stopping = false; private socketPath: string; private pidPath: string; + /** The launcher holding the writer slot for this daemon, when it replaced an older one (#2335). */ + private handoverFrom: number | null; constructor( private projectRoot: string, - opts: { idleTimeoutMs?: number; maxIdleMs?: number } = {}, + opts: { idleTimeoutMs?: number; maxIdleMs?: number; handoverFrom?: number | null } = {}, ) { this.socketPath = getDaemonSocketPath(projectRoot); this.pidPath = getDaemonPidPath(projectRoot); this.idleTimeoutMs = opts.idleTimeoutMs ?? resolveIdleTimeoutMs(); this.maxIdleMs = opts.maxIdleMs ?? resolveMaxIdleMs(); + this.handoverFrom = opts.handoverFrom ?? null; // Daemon mode serves many concurrent clients on one event loop, so off-load // read-tool dispatch to a worker pool — otherwise concurrent explores // serialize and starve the MCP transport (clients time out). Direct mode @@ -214,7 +219,17 @@ export class Daemon { // #1740: claim the project writer lock before opening/watching so a // concurrent direct-mode serve --mcp cannot start a second watcher. assertNoRebuild(this.projectRoot); - const writer = tryAcquireWriterLock(this.projectRoot, 'daemon'); + let writer = tryAcquireWriterLock(this.projectRoot, 'daemon'); + // The launcher that stopped an older daemon has held the slot for us since + // (#2335); take it over without letting it fall free. + if (writer.kind === 'taken' && this.handoverFrom !== null && + writer.existing?.pid === this.handoverFrom && writer.existing.mode === 'handover') { + const info: WriterLockInfo = { pid: process.pid, mode: 'daemon', startedAt: Date.now(), ready: false }; + if (swapWriterLock(this.projectRoot, this.handoverFrom, info)) { + writer = { kind: 'acquired', pidPath: writer.pidPath, info }; + process.stderr.write(`[CodeGraph daemon] Took over the writer lock from launcher pid ${this.handoverFrom}, which stopped an older daemon.\n`); + } + } if (writer.kind === 'taken') { const msg = writerLockHeldMessage(writer.existing, writer.pidPath); process.stderr.write(`[CodeGraph daemon] ${msg}\n`); diff --git a/src/mcp/index.ts b/src/mcp/index.ts index 824132d629..ac7aa1d0ad 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -47,8 +47,9 @@ import { isProcessAlive, tryAcquireDaemonLock, } from './daemon'; -import { clearStaleDaemonArtifacts } from './daemon-registry'; +import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry'; import { connectWithHello, runLocalHandshakeProxy } from './proxy'; +import { CodeGraphPackageVersion } from './version'; import { readWriterLock, assertNoRebuild, @@ -83,6 +84,13 @@ const DIRECT_QUERY_POOL_MAX = 2; */ const DAEMON_INTERNAL_ENV = 'CODEGRAPH_DAEMON_INTERNAL'; +/** + * Env var naming the launcher that holds the project's writer slot for the + * daemon it spawns, after stopping an older one (#2335). That daemon takes the + * slot over instead of finding it held. + */ +const DAEMON_HANDOVER_ENV = 'CODEGRAPH_DAEMON_HANDOVER'; + /** * Retries for the detached daemon arbitrating the O_EXCL lock against a racing * sibling. Tiny — the lock resolves on the first round in practice; the retries @@ -91,6 +99,13 @@ const DAEMON_INTERNAL_ENV = 'CODEGRAPH_DAEMON_INTERNAL'; const TAKEOVER_MAX_RETRIES = 5; const TAKEOVER_RETRY_DELAY_MS = 100; +/** + * How long a daemon the writer slot is held for (#2335) waits out a racing + * candidate that took the daemon lock first. That candidate cannot get the + * slot, so it leaves; this bounds a wait that never ends. + */ +const HANDOVER_CANDIDATE_WAIT_MS = 5_000; + /** * A fallback that serves reads without a watcher or a writer lock (#1963). * Say so on stderr: otherwise a session that quietly stopped syncing looks @@ -243,9 +258,10 @@ function resolveDaemonRoot(explicitPath: string | null): string | null { * `process.execArgv` (carries `--liftoff-only`, so the daemon never re-execs) * and `process.argv[1]` (this script). The spawned process self-arbitrates the * O_EXCL lock, so racing launchers may each spawn one — losers exit and every - * launcher proxies through the single winner. + * launcher proxies through the single winner. `handover` tells the daemon this + * launcher holds the writer slot for it (see {@link DAEMON_HANDOVER_ENV}). */ -function spawnDetachedDaemon(root: string): void { +function spawnDetachedDaemon(root: string, handover = false): void { const scriptPath = process.argv[1]; if (!scriptPath) { // No resolvable CLI entry point to re-invoke — let the caller fall back to @@ -267,6 +283,8 @@ function spawnDetachedDaemon(root: string): void { // where a long-dead session's host pid would trigger spurious shutdowns. const env: NodeJS.ProcessEnv = { ...process.env, [DAEMON_INTERNAL_ENV]: '1' }; delete env[HOST_PPID_ENV]; + if (handover) env[DAEMON_HANDOVER_ENV] = String(process.pid); + else delete env[DAEMON_HANDOVER_ENV]; const child = spawn( process.execPath, [...process.execArgv, scriptPath, 'serve', '--mcp', '--path', root], @@ -286,6 +304,76 @@ function spawnDetachedDaemon(root: string): void { } } +/** + * How long a launcher replacing an older daemon waits for one that is partway + * through its shutdown, past the stop's own 3s wait. Tool calls are held + * meanwhile, so it is short: a daemon that takes longer leaves the session + * served in-process until the proxy's next retry (#2277) finds it gone. + */ +const OLDER_DAEMON_SHUTDOWN_GRACE_MS = 5_000; + +/** + * Stop the daemon serving `root` from an older install so the caller can start + * one from this install (#2335). A daemon keeps the code it started with: one + * that outlived an upgrade went on loading grammars from the removed install + * and wrote every file it re-indexed as empty, and as the project's writer it + * kept every newer session from syncing. {@link stopOlderDaemon} finds it by + * the project lock, verifies it, and never stops one of the same, a newer or an + * unknown version. `answered` says an older daemon answered this launcher's + * own probe. Resolves `stopped` (this launcher now holds the project's writer + * slot for the daemon it starts next), `none` (no older daemon holds the + * project), or `kept` (one does, unverified or still shutting down). + */ +async function replaceOlderDaemon(root: string, answered: boolean): Promise<'stopped' | 'none' | 'kept'> { + const result = await stopOlderDaemon(root, CodeGraphPackageVersion, { + shutdownGraceMs: OLDER_DAEMON_SHUTDOWN_GRACE_MS, + }); + // No older daemon in the lock (none, or already replaced), or it had exited. + if (!result || result.outcome === 'not-running') return 'none'; + const daemon = result.version + ? `the CodeGraph ${result.version} daemon (pid ${result.pid})` + : 'an older CodeGraph daemon'; + if (result.outcome === 'term' || result.outcome === 'kill') { + process.stderr.write( + `[CodeGraph MCP] Stopped ${daemon} serving this project; starting one from this install (${CodeGraphPackageVersion}).\n` + ); + return 'stopped'; + } + if (result.outcome === 'still-running') { + process.stderr.write(`[CodeGraph MCP] Asked ${daemon} serving this project to stop, but it has not exited yet.\n`); + } else if (answered) { + process.stderr.write( + `[CodeGraph MCP] Found ${daemon} serving this project but could not verify it, so left it running; serving this session in-process.\n` + ); + } + return 'kept'; +} + +/** + * Wait out the daemon candidate `pid` holding the daemon lock while launcher + * `handoverFrom` holds the writer slot for this daemon (#2335): the candidate + * cannot get the slot, so it releases the lock and exits. Resolves true once + * `pid` no longer holds the lock; false at once when no slot is held for this + * daemon, and false when that stops being so or the wait runs out. + */ +async function outwaitCandidate(root: string, handoverFrom: number | null, pid: number): Promise { + const heldForUs = (): boolean => { + const writer = readWriterLock(root); + return handoverFrom !== null && writer?.pid === handoverFrom && writer.mode === 'handover'; + }; + if (!heldForUs()) return false; + process.stderr.write(`[CodeGraph daemon] Waiting for daemon candidate pid ${pid} to give up the lock; the writer lock is held for this daemon.\n`); + const deadline = Date.now() + HANDOVER_CANDIDATE_WAIT_MS; + while (Date.now() < deadline) { + if (!heldForUs()) return false; + let holder: number | undefined; + try { holder = decodeLockInfo(fs.readFileSync(getDaemonPidPath(root), 'utf8'))?.pid; } catch { holder = undefined; } + if (holder !== pid || !isProcessAlive(pid)) return true; + await sleep(25); + } + return false; +} + /** * MCP Server for CodeGraph * @@ -495,11 +583,15 @@ export class MCPServer { // kills/restarts can be placed in time (#1431 — the log was undatable). timestampStderrLines(); const root = resolveDaemonRoot(this.projectPath) ?? this.projectPath ?? process.cwd(); + // Read once and dropped, so nothing this daemon spawns inherits it. + const handoverPid = Number(process.env[DAEMON_HANDOVER_ENV]); + const handoverFrom = Number.isInteger(handoverPid) && handoverPid > 0 ? handoverPid : null; + delete process.env[DAEMON_HANDOVER_ENV]; for (let attempt = 0; attempt < TAKEOVER_MAX_RETRIES; attempt++) { const lock = tryAcquireDaemonLock(root); if (lock.kind === 'acquired') { - const daemon = new Daemon(root); + const daemon = new Daemon(root, { handoverFrom }); await daemon.start(); this.daemon = daemon; this.mode = 'daemon'; @@ -529,6 +621,9 @@ export class MCPServer { stillStarting || await probeDaemonIdentity(existing) ) { + // A daemon the writer slot is held for outwaits a candidate still + // starting in its way, which cannot get the slot and leaves (#2335). + if (stillStarting && await outwaitCandidate(root, handoverFrom, existing.pid)) continue; process.stderr.write( `[CodeGraph daemon] Another daemon (pid ${existing.pid}) already holds the lock; exiting.\n` ); @@ -560,7 +655,8 @@ export class MCPServer { /** * Proxy mode (the common case). Serve the MCP handshake LOCALLY for instant * tool registration, forwarding tool calls to the shared daemon — which is - * connected in the background (probed, then spawned + polled if absent) so the + * connected in the background (probed, then spawned + polled if absent; one of + * an older release is stopped first, #2335) so the * handshake never waits ~600ms on it. Runs until the host disconnects; the * proxy falls back to an in-process engine if the daemon never binds, so this * never wedges a session. @@ -577,9 +673,8 @@ export class MCPServer { for (const candidate of candidates) { const s = await connectWithHello(candidate); // A wrong-version daemon IS up — definitive; propagate so the caller - // serves in-process instead of spawning + polling for 6s. Don't keep - // probing fallbacks past it. - if (s === 'version-mismatch') return s; + // replaces it (an older release) or serves in-process instead of + // spawning + polling for 6s. Don't keep probing fallbacks past it. if (s) return s; } return null; @@ -588,16 +683,31 @@ export class MCPServer { // Fast path: a daemon may already be listening (on either candidate). const probe = await connectAnyCandidate(); if (probe === 'version-mismatch') return null; // definitive — serve in-process, don't poll for 6s - if (probe) return probe; - // None reachable — spawn one (detached) and poll for its bind. - spawnDetachedDaemon(root); - for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { - await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); - const s = await connectAnyCandidate(); - if (s === 'version-mismatch') return null; - if (s) return s; + if (probe && probe !== 'older-version') return probe; + // A daemon of an older release answered — or none did, while the project + // lock may still name one on a socket we don't probe (on Windows, daemons + // before #2278 named their pipe after the root as typed). Either way it + // makes way for one from this install (#2335). + const answered = probe === 'older-version'; + const older = await replaceOlderDaemon(root, answered); + if (older === 'kept' && answered) return null; + // None reachable — spawn one (detached) and poll for its bind. After a + // replacement this launcher holds the writer slot for it meanwhile. + try { + spawnDetachedDaemon(root, older === 'stopped'); + for (let attempt = 0; attempt < DAEMON_CONNECT_MAX_RETRIES; attempt++) { + await sleep(DAEMON_CONNECT_RETRY_DELAY_MS); + const s = await connectAnyCandidate(); + // Another version answering now, older or not, waits for the next + // retry: one replacement per attempt, so this can never loop. + if (s === 'older-version' || s === 'version-mismatch') return null; + if (s) return s; + } + return null; // never bound — the proxy serves this session in-process + } finally { + // The new daemon has taken the slot over, or never came; a no-op unless still ours. + if (older === 'stopped') releaseWriterLock(root); } - return null; // never bound — the proxy serves this session in-process }; await runLocalHandshakeProxy({ getDaemonSocket, makeEngine: () => makeFallbackEngine(root), root }); } diff --git a/src/mcp/project-lifecycle.ts b/src/mcp/project-lifecycle.ts index b7a52c1a5b..1a634815eb 100644 --- a/src/mcp/project-lifecycle.ts +++ b/src/mcp/project-lifecycle.ts @@ -118,7 +118,8 @@ async function activate(root: string, project: Project): Promise { if (writer.existing?.mode === 'daemon') { for (const candidate of getDaemonSocketCandidates(root)) { const socket = await connectWithHello(candidate); - if (!socket || socket === 'version-mismatch') continue; + // A daemon of another version, older or not, is left to the project's own launchers. + if (!socket || typeof socket === 'string') continue; if (project.refs === 0) { socket.destroy(); return; } project.socket = socket; socket.once('close', () => { diff --git a/src/mcp/proxy.ts b/src/mcp/proxy.ts index 52e6b77fd5..c0d9f6cafb 100644 --- a/src/mcp/proxy.ts +++ b/src/mcp/proxy.ts @@ -26,7 +26,7 @@ import { EARLY_PPID } from './early-ppid'; import { supervisionLostReason } from './ppid-watchdog'; import { armStartupHandshakeTimeout } from './startup-handshake'; import { treatStdinFailureAsShutdown } from './stdin-teardown'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageVersion, isOlderRelease } from './version'; import { SERVER_INFO, PROTOCOL_VERSION, initializeInstructions } from './session'; import { SERVER_INSTRUCTIONS } from './server-instructions'; import { getStaticTools } from './tools'; @@ -148,14 +148,17 @@ export async function runProxy( /** * Connect to a daemon at `socketPath` and verify its hello (exact version match). - * Returns the live socket (hello already consumed) or null if unreachable / stale - * / version-mismatched. Unlike {@link runProxy} it does NOT pipe — the caller - * owns the socket. Used by the local-handshake proxy's background connect. + * Returns the live socket (hello already consumed), null if unreachable / stale, + * or — for a daemon of another version — `'older-version'` when it runs an older + * release ({@link isOlderRelease}; the launcher replaces it, #2335) and + * `'version-mismatch'` for any other version, which is left alone. Unlike + * {@link runProxy} it does NOT pipe — the caller owns the socket. Used by the + * local-handshake proxy's background connect. */ export async function connectWithHello( socketPath: string, expectedVersion: string = CodeGraphPackageVersion, -): Promise { +): Promise { if (process.platform !== 'win32' && !fs.existsSync(socketPath)) return null; const socket = net.createConnection(socketPath); socket.setEncoding('utf8'); @@ -176,12 +179,14 @@ export async function connectWithHello( } if (hello.codegraph !== expectedVersion) { // A daemon IS up but it's the wrong version — definitive, not a "not yet". - // Don't poll; the caller serves in-process so we never run stale-vs-new. + socket.destroy(); + // An older release is the launcher's to replace (#2335); it says what it did. + if (isOlderRelease(hello.codegraph, expectedVersion)) return 'older-version'; + // Any other: don't poll; the caller serves in-process so we never run stale-vs-new. process.stderr.write( `[CodeGraph MCP] Found a daemon on ${socketPath} but version (${hello.codegraph}) ` + `differs from ours (${expectedVersion}); serving this session in-process.\n` ); - socket.destroy(); return 'version-mismatch'; } logAttachedDaemon(socketPath, hello); @@ -212,8 +217,9 @@ type JsonRpc = Record; /** Dependencies the local-handshake proxy needs, injected by MCPServer (which * owns the daemon-spawn machinery and the engine factory). */ export interface LocalHandshakeDeps { - /** Probe → spawn → retry → hello-verify; resolves a connected daemon socket, - * or null when the daemon path is genuinely unavailable (→ in-process fallback). */ + /** Probe (replacing a daemon of an older release, #2335) → spawn → retry → + * hello-verify; resolves a connected daemon socket, or null when the daemon + * path is genuinely unavailable (→ in-process fallback). */ getDaemonSocket(): Promise; /** Lazily create an in-process engine — used only while the daemon is * unreachable, preserving the "a broken daemon never wedges a session" diff --git a/src/mcp/version.ts b/src/mcp/version.ts index cef1b7834a..b6c919f0f0 100644 --- a/src/mcp/version.ts +++ b/src/mcp/version.ts @@ -4,7 +4,8 @@ * The version string is the rendezvous datum between cooperating daemon and * proxy processes: the daemon advertises its version in the hello line, and * the proxy refuses to share IPC across a mismatch (falls back to direct - * mode). Keeping the resolution in one place avoids drift between the CLI + * mode), or replaces a daemon of an older release ({@link isOlderRelease}). + * Keeping the resolution in one place avoids drift between the CLI * `--version` output (which reads `package.json` directly) and the daemon * handshake. * @@ -34,3 +35,25 @@ function readPackageVersion(): string { } export const CodeGraphPackageVersion = readPackageVersion(); + +/** + * Whether `version` is a CodeGraph release older than `than` — the test a + * launcher applies before it replaces a running daemon (#2335). Only plain + * `MAJOR.MINOR.PATCH` releases compare: a prerelease, a build suffix or the + * "0.0.0-unknown" sentinel is never older. A daemon is replaced only by a + * strictly newer release, so two installed versions can never take turns + * stopping each other's daemon. + */ +export function isOlderRelease(version: string, than: string): boolean { + const a = parseRelease(version); + const b = parseRelease(than); + if (!a || !b) return false; + if (a[0] !== b[0]) return a[0] < b[0]; + if (a[1] !== b[1]) return a[1] < b[1]; + return a[2] < b[2]; +} + +function parseRelease(version: string): [number, number, number] | null { + const m = /^(\d+)\.(\d+)\.(\d+)$/.exec(version); + return m ? [Number(m[1]), Number(m[2]), Number(m[3])] : null; +} diff --git a/src/mcp/writer-lock.ts b/src/mcp/writer-lock.ts index 9acbdb9a46..7785cde674 100644 --- a/src/mcp/writer-lock.ts +++ b/src/mcp/writer-lock.ts @@ -165,6 +165,44 @@ export function markWriterReady(projectRoot: string): void { } } +/** Waits between retries of a writer-record swap refused by Windows (another process has the file open). */ +const SWAP_RETRY_DELAYS_MS = [25, 50, 100, 200]; + +/** + * Replace the writer record of `fromPid` with `next`, and only that record, so + * the slot passes between two processes without ever falling free (#2335). A + * stopped daemon's sessions serve themselves in-process the moment it goes, and + * one that found the slot free or stale would claim it as their own writer. + * Compare and rename are separate steps, so callers check the result: whether + * the record was replaced. Never throws. + */ +export function swapWriterLock(projectRoot: string, fromPid: number, next: WriterLockInfo): boolean { + const pidPath = getWriterPidPath(projectRoot); + const tmp = `${pidPath}.${process.pid}.swap.tmp`; + try { + if (readWriterLock(projectRoot)?.pid !== fromPid) return false; + fs.writeFileSync(tmp, encode(next), { mode: 0o600 }); + for (let attempt = 0; ; attempt++) { + try { + fs.renameSync(tmp, pidPath); + return true; + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + const delay = SWAP_RETRY_DELAYS_MS[attempt]; + if (process.platform !== 'win32' || !['EPERM', 'EACCES', 'EBUSY'].includes(code ?? '') || delay === undefined) { + return false; + } + Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, delay); + if (readWriterLock(projectRoot)?.pid !== fromPid) return false; + } + } + } catch { + return false; + } finally { + try { fs.unlinkSync(tmp); } catch { /* renamed, or never written */ } + } +} + /** Release if we still own the lock (pid match). */ export function releaseWriterLock(projectRoot: string, lockName: 'writer.pid' | 'rebuild.pid' = 'writer.pid'): void { const pidPath = getWriterPidPath(projectRoot, lockName); From dea076fd1e9fa3e236e93fa915bb0feebe73579d Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 15:08:14 +0000 Subject: [PATCH 203/363] fix(mcp,extraction): exit a daemon whose install changed; never store a grammar-load failure (#2335, #2336) (#2346) * fix(mcp,extraction): a daemon whose install changed exits; grammar-load failures are never stored (#2335, #2336) A daemon that outlived an upgrade loaded grammars lazily from an install that no longer existed, and stored every file its watcher re-indexed as a zero-node row under the file's new content hash: no hash-based sync ever revisited those rows, and `codegraph status` called the index up to date. - The daemon checks its own package.json every 30 s (CODEGRAPH_DAEMON_INSTALL_CHECK_MS, 0 turns it off) and exits once the file is gone or carries another version, so the next session starts a daemon from the current install. - A result whose grammar failed to load (`parser_error`) is never stored: the file keeps its previous data and the next sync or index retries it, and a row an older engine stored that way is re-indexed even though its hash matches. The tag-based CFML path reports a missing grammar with the same code. - A launcher whose own install was deleted no longer crashes on the spawn error of the daemon it tried to start. - `codegraph status` and `status --json` (filesNeedingReindex, filesWithParseErrors) report indexed files whose symbols are missing although their content is current (#2336), and `files --json` carries each file's recorded errors. Co-Authored-By: Claude Opus 5.5 * fix(mcp): a session whose own install changed never claims the writer lock (#2335) A daemon now exits once its install is upgraded or removed, and nobody holds the writer slot for a successor at that point. Its sessions run the same replaced code and serve themselves in-process the moment it goes; one that found the slot free claimed it as its writer even though its engine could not load (a real npm upgrade: "Cannot find module './sqlite-adapter'"), and so kept a daemon from the current install from starting. Such a session now falls back read-only and says to restart it. Also adds the CHANGELOG entries for the install check and grammar-load failures (#2335) and for the status report (#2336). Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/daemon-install-check.test.ts | 83 ++++++++++++ __tests__/daemon-older-version.test.ts | 45 +++++++ __tests__/grammar-load-failure.test.ts | 167 +++++++++++++++++++++++++ __tests__/status-index-health.test.ts | 159 +++++++++++++++++++++++ src/bin/codegraph.ts | 49 +++++++- src/db/queries.ts | 12 ++ src/extraction/cfml-extractor.ts | 4 +- src/extraction/grammars.ts | 17 ++- src/extraction/index.ts | 40 ++++-- src/index.ts | 30 +++++ src/mcp/daemon.ts | 72 ++++++++++- src/mcp/index.ts | 17 ++- src/mcp/version.ts | 6 +- src/types.ts | 18 +++ 15 files changed, 706 insertions(+), 15 deletions(-) create mode 100644 __tests__/daemon-install-check.test.ts create mode 100644 __tests__/grammar-load-failure.test.ts create mode 100644 __tests__/status-index-health.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 32db6b23ab..f280890b4f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) +- A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) +- `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/daemon-install-check.test.ts b/__tests__/daemon-install-check.test.ts new file mode 100644 index 0000000000..614afb6e4a --- /dev/null +++ b/__tests__/daemon-install-check.test.ts @@ -0,0 +1,83 @@ +/** + * A daemon whose install was upgraded or deleted underneath it exits (#2335). + * + * The daemon loads grammars (and the native kernel, and worker scripts) lazily + * from the directory it was started from. An upgrade replaces or deletes that + * directory while the daemon keeps running, so every later load fails; the + * 1.6.1 daemon in #2335 went on re-indexing files as empty for as long as it + * lived. It now checks its own package.json periodically and makes way for a + * daemon started from the current install. The check is pure and the Daemon's + * reaction is tested with it injected; neither starts a real daemon. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { Daemon, installChangedReason } from '../src/mcp/daemon'; + +describe('installChangedReason', () => { + let dir: string; + let pkg: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-install-check-')); + pkg = path.join(dir, 'package.json'); + fs.writeFileSync(pkg, JSON.stringify({ name: '@colbymchenry/codegraph', version: '1.6.2' })); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('is null while the install still carries the running version', () => { + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + }); + + it('reports an install upgraded in place', () => { + fs.writeFileSync(pkg, JSON.stringify({ version: '1.6.3' })); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/replaced by v1\.6\.3/); + }); + + it('reports an install that was deleted, file or whole directory', () => { + fs.rmSync(pkg); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/removed/); + fs.rmSync(dir, { recursive: true, force: true }); + expect(installChangedReason(pkg, '1.6.2')).toMatch(/removed/); + }); + + it('waits for a package.json caught mid-write instead of exiting on it', () => { + fs.writeFileSync(pkg, '{"name": "@colbymchenry/codeg'); + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + fs.writeFileSync(pkg, ''); + expect(installChangedReason(pkg, '1.6.2')).toBeNull(); + }); +}); + +describe('Daemon.checkInstall', () => { + // A Daemon is safe to construct (nothing binds until start()); stop() is + // replaced so a positive check can't tear down the test process. + const makeDaemon = () => { + const d = new Daemon('/tmp/codegraph-install-check-unit-test', { idleTimeoutMs: 0 }) as any; + d.stop = vi.fn(async () => {}); + return d; + }; + + it('stops the daemon when its install changed', () => { + const d = makeDaemon(); + expect(d.checkInstall(() => 'Install removed')).toBe(true); + expect(d.stop).toHaveBeenCalledWith('install changed'); + }); + + it('keeps running while the install is unchanged', () => { + const d = makeDaemon(); + expect(d.checkInstall(() => null)).toBe(false); + expect(d.stop).not.toHaveBeenCalled(); + }); + + it('does nothing once the daemon is already stopping', () => { + const d = makeDaemon(); + d.stopping = true; + expect(d.checkInstall(() => 'Install removed')).toBe(false); + expect(d.stop).not.toHaveBeenCalled(); + }); +}); diff --git a/__tests__/daemon-older-version.test.ts b/__tests__/daemon-older-version.test.ts index 188cd6cf82..b29cbba3c6 100644 --- a/__tests__/daemon-older-version.test.ts +++ b/__tests__/daemon-older-version.test.ts @@ -713,4 +713,49 @@ describe('a launcher meeting a daemon of another version (#2335)', () => { expect(writerLock()).toMatchObject({ pid: newer.pid, mode: 'daemon' }); await status(newer.session, 3, 'a tool call through the newer daemon'); }, 90000); + + it('a session whose own install was upgraded serves reads only once its daemon exits', async () => { + // An upgrade in place: the daemon notices its package.json changed and + // exits (its install check), and its session falls back in-process. As + // the project's writer that session — running code the upgrade replaced — + // would only keep a daemon from the new install from starting. + const install = makeInstall('0.0.2'); + try { + const before = await sessionWithDaemon(install, '0.0.2', { + CODEGRAPH_DAEMON_INSTALL_CHECK_MS: '200', + CODEGRAPH_DAEMON_RETRY_MS: '0', + }); + fs.writeFileSync(path.join(install, 'package.json'), JSON.stringify({ name: '@colbymchenry/codegraph', version: '0.0.3' }) + '\n'); + expect(await waitProcessExit(before.pid, 10000)).toBe(true); + expect(daemonLog()).toContain('Install replaced by v0.0.3'); + await waitFor( + () => before.session.stderr.some((l) => l.includes('Shared daemon connection lost')), + 10000, 'the session to lose its daemon', + ); + + await status(before.session, 3, 'a tool call after the daemon exited'); + const stderr = before.session.stderr.join('\n'); + expect(stderr).toContain("this session's CodeGraph install was upgraded or removed"); + expect(stderr).not.toContain('File watcher active'); + expect(writerLock()).toBeNull(); + + // So a session from the current install gets a daemon straight away. + const after = startSession(null); + await waitFor(() => attachedLine(after), 20000, 'the new session to attach'); + const lock = daemonLock(); + daemonPids.add(lock.pid); + expect(writerLock()).toMatchObject({ pid: lock.pid, mode: 'daemon' }); + } finally { + // On Windows a process still running from the install blocks its removal. + for (const { child } of sessions) { + if (child.exitCode === null && child.signalCode === null) { + const exited = once(child, 'exit'); + child.kill('SIGKILL'); + await exited; + } + } + try { fs.unlinkSync(path.join(install, 'node_modules')); } catch { /* not created */ } + await fs.promises.rm(install, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); + } + }, 90000); }); diff --git a/__tests__/grammar-load-failure.test.ts b/__tests__/grammar-load-failure.test.ts new file mode 100644 index 0000000000..1f285f0fd4 --- /dev/null +++ b/__tests__/grammar-load-failure.test.ts @@ -0,0 +1,167 @@ +/** + * A grammar that fails to load is an environment failure, not a parse result + * (#2335). + * + * A long-running daemon loads tree-sitter grammars lazily from its install + * directory. When an upgrade deletes that directory underneath it, every later + * grammar load fails (ENOENT) and the extractor answers `parser_error` with no + * nodes. That used to be STORED: the file's good symbols were replaced by a + * zero-node row carrying the new content hash, so `sync` never revisited it + * and `status` stayed green. Now nothing is stored for such a result — the + * file keeps its previous index data and the next sync or index retries it — + * and rows an older engine already wrote that way are re-indexed by `sync`. + * + * The failure is simulated at `getParser` (what a failed load leaves behind: + * no parser for the language), with the native kernel switched off so every + * file goes through the grammar path regardless of whether a kernel is built. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { CodeGraph } from '../src'; +import type { Language } from '../src/types'; + +const { failing } = vi.hoisted(() => ({ failing: new Set() })); + +vi.mock('../src/extraction/grammars', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + getParser: (language: Language) => (failing.has(language) ? null : actual.getParser(language)), + }; +}); + +const MATH_TS = + 'export function add(a: number, b: number): number {\n' + + ' return a + b;\n' + + '}\n' + + 'export class Calculator {\n' + + ' total = 0;\n' + + ' plus(n: number): this { this.total = add(this.total, n); return this; }\n' + + '}\n'; + +const VIEW_TS = + "import { Calculator } from './math';\n" + + 'export function total(n: number): number {\n' + + ' return new Calculator().plus(n).total;\n' + + '}\n'; + +const SUB_TS = '\nexport function sub(a: number, b: number): number {\n return a - b;\n}\n'; + +function names(cg: CodeGraph, file: string): string[] { + return cg.getNodesInFile(file).map((n) => n.name); +} + +describe('grammar load failures are never stored (#2335)', () => { + let dir: string; + let cg: CodeGraph; + let kernelEnv: string | undefined; + + beforeEach(() => { + kernelEnv = process.env.CODEGRAPH_KERNEL; + process.env.CODEGRAPH_KERNEL = '0'; + failing.clear(); + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-grammar-load-failure-')); + fs.mkdirSync(path.join(dir, 'src')); + fs.writeFileSync(path.join(dir, 'src', 'math.ts'), MATH_TS); + fs.writeFileSync(path.join(dir, 'src', 'view.ts'), VIEW_TS); + }); + + afterEach(() => { + failing.clear(); + cg?.destroy(); + if (kernelEnv === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernelEnv; + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('sync keeps the previous symbols of a file whose grammar failed to load, then retries it', async () => { + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/math.ts')!; + expect(before.nodeCount).toBeGreaterThan(1); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + + // The upgrade removed the grammar; the watcher then re-indexes an edit. + failing.add('typescript'); + fs.appendFileSync(path.join(dir, 'src', 'math.ts'), SUB_TS); + const failed = await cg.sync(); + + expect(failed.failedFilePaths).toContain('src/math.ts'); + const kept = cg.getFile('src/math.ts')!; + expect(kept.nodeCount).toBe(before.nodeCount); + expect(kept.contentHash).toBe(before.contentHash); + expect(kept.errors ?? []).toEqual([]); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + + // Grammar available again (a fresh process from the new install): the + // next sync sees the file as still changed and indexes the edit. + failing.clear(); + const retried = await cg.sync(); + expect(retried.filesModified).toBe(1); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator', 'sub'])); + expect(cg.getFile('src/math.ts')!.nodeCount).toBeGreaterThan(before.nodeCount); + }); + + it('a full index stores nothing for files whose grammar failed to load, and sync adds them later', async () => { + failing.add('typescript'); + cg = await CodeGraph.init(dir); + const result = await cg.indexAll(); + + expect(result.filesErrored).toBe(2); + expect(result.errors.filter((e) => e.code === 'parser_error')).toHaveLength(2); + expect(cg.getFile('src/math.ts')).toBeNull(); + expect(cg.getFile('src/view.ts')).toBeNull(); + + failing.clear(); + const synced = await cg.sync(); + expect(synced.filesAdded).toBe(2); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + expect(names(cg, 'src/view.ts')).toContain('total'); + }); + + it('sync re-indexes rows an older engine stored while the grammar was missing', async () => { + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/math.ts')!; + + // What v1.6.2 wrote for a file it re-indexed with no grammar: no nodes, + // the parser error, and the CURRENT content hash — so a hash-based + // reconcile considers it up to date forever. + const db = (cg as unknown as { db: { getDb(): { prepare(sql: string): { run(...args: unknown[]): unknown } } } }).db.getDb(); + db.prepare('DELETE FROM nodes WHERE file_path = ?').run('src/math.ts'); + db.prepare('UPDATE files SET node_count = 0, errors = ? WHERE path = ?').run( + JSON.stringify([{ message: 'Failed to get parser for language: typescript', filePath: 'src/math.ts', severity: 'error', code: 'parser_error' }]), + 'src/math.ts' + ); + expect(cg.getFile('src/math.ts')!.nodeCount).toBe(0); + + await cg.sync(); + + const healed = cg.getFile('src/math.ts')!; + expect(healed.nodeCount).toBe(before.nodeCount); + expect(healed.errors ?? []).toEqual([]); + expect(names(cg, 'src/math.ts')).toEqual(expect.arrayContaining(['add', 'Calculator'])); + }); + + it('a component whose script grammar failed keeps its previous symbols too', async () => { + fs.writeFileSync( + path.join(dir, 'src', 'Counter.svelte'), + '\n\n' + ); + cg = await CodeGraph.init(dir); + await cg.indexAll(); + const before = cg.getFile('src/Counter.svelte')!; + expect(names(cg, 'src/Counter.svelte')).toContain('increment'); + + failing.add('typescript'); + fs.appendFileSync(path.join(dir, 'src', 'Counter.svelte'), '

    edited

    \n'); + await cg.sync(); + + // The component node alone would have been stored before: the script's + // symbols gone, with a nonzero node count hiding it. + expect(cg.getFile('src/Counter.svelte')!.nodeCount).toBe(before.nodeCount); + expect(names(cg, 'src/Counter.svelte')).toContain('increment'); + }); +}); diff --git a/__tests__/status-index-health.test.ts b/__tests__/status-index-health.test.ts new file mode 100644 index 0000000000..b94562c8eb --- /dev/null +++ b/__tests__/status-index-health.test.ts @@ -0,0 +1,159 @@ +/** + * `codegraph status` must not call an index "up to date" while indexed files + * are missing their symbols (#2336, #2335). + * + * "Up to date" used to mean only "every file's content hash matches the + * index". Two kinds of file pass that check with nothing usable in the graph: + * + * - a file the parser could not read — `export type * from` is valid + * TypeScript 5.0 the bundled grammar has no rule for, so the file is stored + * with a parse error and no symbols (#2336); + * - a row stored without its symbols — by an engine whose grammar failed to + * load (#2335), or the #1541 wipe — which `codegraph sync` re-indexes. + * + * Both are counted in `status` (human and `--json`), and `files --json` carries + * each file's recorded errors. Exercised against the built CLI so the output + * and the JSON field names are what users and scripts actually see. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { spawnSync } from 'child_process'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +const BIN = path.resolve(__dirname, '../dist/bin/codegraph.js'); + +function spawn(cwd: string, args: string[]) { + return spawnSync(process.execPath, [BIN, ...args], { + cwd, + encoding: 'utf-8', + timeout: 30_000, + env: { + ...process.env, + CODEGRAPH_NO_DAEMON: '1', + CODEGRAPH_TELEMETRY: '0', + CODEGRAPH_NO_UPDATE_CHECK: '1', + NO_COLOR: '1', + }, + }); +} + +function run(cwd: string, args: string[]): { status: number | null; out: string } { + const result = spawn(cwd, args); + return { status: result.status, out: (result.stdout ?? '') + (result.stderr ?? '') }; +} + +function runJson(cwd: string, args: string[]): any { + const result = spawn(cwd, args); + expect(result.status, (result.stdout ?? '') + (result.stderr ?? '')).toBe(0); + return JSON.parse(result.stdout); +} + +describe('status reports files indexed without their symbols (#2336, #2335)', () => { + let dir: string; + + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-status-health-')); + fs.mkdirSync(path.join(dir, 'src', 'constants'), { recursive: true }); + fs.mkdirSync(path.join(dir, 'src', 'types'), { recursive: true }); + fs.writeFileSync( + path.join(dir, 'src', 'constants', 'index.ts'), + 'export const MAX_ITEMS = 10;\nexport function clamp(n: number): number { return Math.min(n, MAX_ITEMS); }\n' + ); + fs.writeFileSync( + path.join(dir, 'src', 'types', 'database.ts'), + 'export interface Row { id: number; name: string }\nexport type RowId = Row["id"];\n' + ); + }); + + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + async function index(): Promise { + const cg = CodeGraph.initSync(dir); + await cg.indexAll(); + cg.destroy(); + } + + it('a clean index is up to date, with zero counts and empty error lists', async () => { + await index(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).toContain('Index is up to date'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesNeedingReindex).toBe(0); + expect(json.index.filesWithParseErrors).toBe(0); + + const files = runJson(dir, ['files', '--json']); + expect(files).toHaveLength(2); + for (const f of files) expect(f.errors).toEqual([]); + }); + + it('a file the parser could not read keeps status from reporting "up to date"', async () => { + fs.writeFileSync( + path.join(dir, 'src', 'index.ts'), + 'export * from "./constants/index.js";\nexport type * from "./types/database.js";\n' + ); + await index(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).not.toContain('Index is up to date'); + expect(human.out).toMatch(/1 file could not be parsed/); + expect(human.out).toContain('src/index.ts'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesWithParseErrors).toBe(1); + expect(json.index.filesNeedingReindex).toBe(0); + + const files = runJson(dir, ['files', '--json']); + const broken = files.find((f: any) => f.path === 'src/index.ts'); + expect(broken.errors).toHaveLength(1); + expect(broken.errors[0]).toMatchObject({ severity: 'warning', code: 'parse_error' }); + expect(broken.errors[0].message).toContain('parse produced no symbols'); + expect(files.find((f: any) => f.path === 'src/constants/index.ts').errors).toEqual([]); + }); + + it('rows stored without their symbols ask for a sync, and the sync repairs them', async () => { + await index(); + + // A row v1.6.2 wrote with its grammar missing (#2335), and a #1541 wipe: + // both carry the current content hash, so the hash check calls them current. + const { DatabaseSync } = require('node:sqlite'); + const db = new DatabaseSync(path.join(dir, '.codegraph', 'codegraph.db')); + db.prepare('DELETE FROM nodes WHERE file_path IN (?, ?)').run('src/constants/index.ts', 'src/types/database.ts'); + db.prepare('UPDATE files SET node_count = 0, errors = ? WHERE path = ?').run( + JSON.stringify([{ message: 'Failed to get parser for language: typescript', severity: 'error', code: 'parser_error' }]), + 'src/constants/index.ts' + ); + db.prepare('UPDATE files SET node_count = 0 WHERE path = ?').run('src/types/database.ts'); + db.close(); + + const human = run(dir, ['status']); + expect(human.status, human.out).toBe(0); + expect(human.out).not.toContain('Index is up to date'); + expect(human.out).toMatch(/2 files are missing their symbols/); + expect(human.out).toContain('codegraph sync'); + + const json = runJson(dir, ['status', '--json']); + expect(json.index.filesNeedingReindex).toBe(2); + expect(json.index.filesWithParseErrors).toBe(0); + expect(json.pendingChanges).toEqual({ added: 0, modified: 0, removed: 0 }); + + const synced = run(dir, ['sync']); + expect(synced.status, synced.out).toBe(0); + + const after = runJson(dir, ['status', '--json']); + expect(after.index.filesNeedingReindex).toBe(0); + expect(run(dir, ['status']).out).toContain('Index is up to date'); + const files = runJson(dir, ['files', '--json']); + for (const f of files) { + expect(f.nodeCount).toBeGreaterThan(1); + expect(f.errors).toEqual([]); + } + }); +}); diff --git a/src/bin/codegraph.ts b/src/bin/codegraph.ts index e5f8eacfc6..de33c1a47e 100644 --- a/src/bin/codegraph.ts +++ b/src/bin/codegraph.ts @@ -1056,6 +1056,33 @@ program } }); +/** + * The `status` warnings for indexed files whose symbols are missing although + * their content is current (#2335, #2336): one line per group, saying what is + * wrong, what to do, and naming up to three of the files. + */ +function describeFilesMissingSymbols(health: { needsReindex: string[]; parseErrors: string[] }): string[] { + const dash = getGlyphs().dash; + const sample = (paths: string[]): string => + paths.slice(0, 3).join(', ') + (paths.length > 3 ? ` (+${formatNumber(paths.length - 3)} more)` : ''); + const lines: string[] = []; + const reindex = health.needsReindex.length; + if (reindex > 0) { + lines.push( + `${formatNumber(reindex)} ${reindex === 1 ? 'file is missing its' : 'files are missing their'} symbols ${dash} ` + + `run "codegraph sync" to re-index ${reindex === 1 ? 'it' : 'them'}: ${sample(health.needsReindex)}` + ); + } + const unparsed = health.parseErrors.length; + if (unparsed > 0) { + lines.push( + `${formatNumber(unparsed)} ${unparsed === 1 ? 'file could not be parsed, so its symbols are' : 'files could not be parsed, so their symbols are'} ` + + `missing: ${sample(health.parseErrors)} ${dash} "codegraph files --json" shows the errors` + ); + } + return lines; +} + /** * codegraph status [path] */ @@ -1103,6 +1130,9 @@ program // Zero on a healthy index; non-zero at rest means a resolution pass was // interrupted, so some files' call edges are missing (#1187). const pendingRefs = cg.getPendingReferenceCount(); + // Files whose content is current but whose symbols are missing — the + // content-hash check behind "up to date" cannot see them (#2336). + const health = cg.getIndexHealth(); // JSON output mode if (options.json) { @@ -1143,6 +1173,12 @@ program // interrupted resolution pass left edges missing; the next // sync sweeps them (#1187). pendingRefs, + // Files stored without their symbols (e.g. while their grammar + // could not load, #2335); the next sync re-indexes them. + filesNeedingReindex: health.needsReindex.length, + // Files with a recorded parse error and no symbols from it + // (#2336); unchanged until the file or the parser changes. + filesWithParseErrors: health.parseErrors.length, }, })); cg.destroy(); @@ -1235,7 +1271,10 @@ program console.log(` Removed: ${changes.removed.length} files`); } info('Run "codegraph sync" to update the index'); - } else { + } + const missingSymbols = describeFilesMissingSymbols(health); + for (const line of missingSymbols) warn(line); + if (totalChanges === 0 && missingSymbols.length === 0) { success('Index is up to date'); } console.log(); @@ -1751,6 +1790,14 @@ program language: f.language, nodeCount: f.nodeCount, size: f.size, + // What extraction recorded for the file — a parse error, a skip + // reason — so a health check needn't read the database (#2336). + errors: (f.errors ?? []).map((e) => ({ + severity: e.severity, + code: e.code, + message: e.message, + line: e.line, + })), })); console.log(JSON.stringify(output, null, 2)); cg.destroy(); diff --git a/src/db/queries.ts b/src/db/queries.ts index a12e7c2643..704cfd5bcd 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3253,6 +3253,18 @@ export class QueryBuilder { return rows.map(rowToFileRecord); } + /** + * Files stored with no nodes or with recorded extraction errors — the only + * rows that can be missing their symbols (`CodeGraph.getIndexHealth`). A + * healthy index returns few or none, so this stays cheap on a large one. + */ + getFilesWithoutNodesOrWithErrors(): FileRecord[] { + const rows = this.db + .prepare('SELECT * FROM files WHERE node_count = 0 OR errors IS NOT NULL ORDER BY path') + .all() as FileRow[]; + return rows.map(rowToFileRecord); + } + /** * Most recent index timestamp (ms since epoch) across all tracked files, or * null when nothing is indexed yet. One indexed aggregate, no per-row scan. (#329) diff --git a/src/extraction/cfml-extractor.ts b/src/extraction/cfml-extractor.ts index efd2ea6ea6..a1a668706a 100644 --- a/src/extraction/cfml-extractor.ts +++ b/src/extraction/cfml-extractor.ts @@ -147,10 +147,12 @@ export class CfmlExtractor { private extractTagBased(): void { const parser = getParser('cfml'); if (!parser) { + // Same code as TreeSitterExtractor's missing parser, so a grammar that + // failed to load is never stored as this file's index data (#2335). this.errors.push({ message: 'cfml grammar not loaded', severity: 'error', - code: 'unsupported_language', + code: 'parser_error', }); return; } diff --git a/src/extraction/grammars.ts b/src/extraction/grammars.ts index 6a05b96c0d..0a12e61d7d 100644 --- a/src/extraction/grammars.ts +++ b/src/extraction/grammars.ts @@ -9,7 +9,7 @@ import * as path from 'path'; import * as fsp from 'fs/promises'; import { Parser, Language as WasmLanguage } from 'web-tree-sitter'; -import { Language } from '../types'; +import { ExtractionError, Language } from '../types'; export type GrammarLanguage = Exclude; @@ -828,6 +828,21 @@ export function getUnavailableGrammarErrors(): Partial> return out; } +/** + * Whether these extraction errors say the file was never parsed because its + * grammar was not available (`parser_error`, recorded by the extractor when + * `getParser` has nothing for the language). + * + * That is a fact about the running process, not about the file: a daemon whose + * install was upgraded or deleted underneath it fails every lazy grammar load + * (#2335). Such a result must never replace a file's index data, and a row an + * older engine stored that way must be re-indexed — not treated as current + * because its content hash still matches. + */ +export function hasGrammarLoadFailure(errors: readonly ExtractionError[] | undefined): boolean { + return !!errors && errors.some((e) => e.code === 'parser_error'); +} + /** * Get language display name */ diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 0ca3145be2..ecb946eafb 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -25,7 +25,7 @@ import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './ import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer'; import { materializeKernelResult } from './kernel'; import { detectGeneratedFile } from './generated-detection'; -import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES } from './grammars'; +import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes, isMpegTransportStream, hasMpegTsExtension, MPEG_TS_SNIFF_BYTES, hasGrammarLoadFailure } from './grammars'; import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config'; import { isCodeGraphDataDir } from '../directory'; import { logDebug, logWarn } from '../errors'; @@ -2289,11 +2289,17 @@ export class ExtractionOrchestrator { const nodeCount = result.kernelCounts?.nodes ?? result.nodes.length; const edgeCount = result.kernelCounts?.edges ?? result.edges.length; + // A file whose grammar failed to load was never parsed (#2335). + const grammarUnavailable = hasGrammarLoadFailure(result.errors); + // Store: on the writer thread when active (fresh DB — bundles applied // in the same file order this chain dispatches them), else on the main // thread (SQLite connections are per-thread). const language = detectLanguage(filePath, content, overrides); - if (storeWriter) { + if (grammarUnavailable) { + // Store nothing: a row from an earlier run keeps its data, and with no + // row (or an older hash) the next sync or index retries the file. + } else if (storeWriter) { if (result.kernelBuffers) { // Buffers go to the writer as-is; the worker decodes + finalizes. // The main thread's only per-file work stays O(1) + the content hash. @@ -2321,7 +2327,9 @@ export class ExtractionOrchestrator { errors.push(...result.errors); } - if (nodeCount > 0) { + if (grammarUnavailable) { + filesErrored++; + } else if (nodeCount > 0) { filesIndexed++; totalNodes += nodeCount; totalEdges += edgeCount; @@ -2735,7 +2743,9 @@ export class ExtractionOrchestrator { errors.push(...result.errors); } - if (result.nodes.length > 0) { + if (hasGrammarLoadFailure(result.errors)) { + filesErrored++; // nothing was stored (#2335) + } else if (result.nodes.length > 0) { filesIndexed++; totalNodes += result.nodes.length; totalEdges += result.edges.length; @@ -2918,6 +2928,12 @@ export class ExtractionOrchestrator { result: ExtractionResult, onYield?: MaybeYield ): Promise { + // The file was never parsed: its grammar failed to load (#2335). Storing + // this would replace the file's symbols with an empty row under the new + // content hash, which no hash-based sync revisits. Keep whatever the index + // has; the stale (or missing) row makes the next sync retry the file. + if (hasGrammarLoadFailure(result.errors)) return; + // A kernel result can arrive as an undecoded buffer transport (empty // node/edge arrays, tables riding in kernelBuffers). Decode it before // storing — persisting the transport as-is records the file as having no @@ -2944,7 +2960,10 @@ export class ExtractionOrchestrator { const existingIsMarker = existingFile.nodeCount === 0 && (existingFile.errors?.length ?? 0) > 0; const incomingHasContent = result.nodes.length > 0; - if (!existingIsMarker || !incomingHasContent) { + // A row an older engine stored while the grammar could not load + // (#2335) records no parse at all: any real result replaces it. + const existingNeverParsed = hasGrammarLoadFailure(existingFile.errors); + if (!existingNeverParsed && (!existingIsMarker || !incomingHasContent)) { return; // No changes } } @@ -3400,13 +3419,18 @@ export class ExtractionOrchestrator { } const fullPath = path.join(this.rootDir, filePath); const tracked = trackedMap.get(filePath); + // A row an older engine stored while the file's grammar could not load + // (#2335) holds no parse of these bytes: re-index it even though its + // size, mtime and hash all match. Rows with a real parse error are + // deterministic and are not retried. + const neverParsed = tracked !== undefined && hasGrammarLoadFailure(tracked.errors); // Cheap pre-filter: an already-indexed file whose size AND mtime both match // the DB is unchanged — skip it without reading or hashing. (A content // change that preserves both exactly is the blind spot every mtime-based // incremental tool accepts; `index --force` is the escape hatch. Git bumps // mtime on every file it writes during checkout/merge, so pulls are caught.) - if (tracked) { + if (tracked && !neverParsed) { try { const stat = fs.statSync(fullPath); if (stat.size === tracked.size && Math.floor(stat.mtimeMs) === Math.floor(tracked.modifiedAt)) { @@ -3442,7 +3466,7 @@ export class ExtractionOrchestrator { filesToIndex.push(filePath); changedFilePaths.push(filePath); filesAdded++; - } else if (tracked.contentHash !== contentHash) { + } else if (tracked.contentHash !== contentHash || neverParsed) { onFileChange?.(filePath, content); filesToIndex.push(filePath); changedFilePaths.push(filePath); @@ -3477,7 +3501,7 @@ export class ExtractionOrchestrator { const result = await this.indexFile(filePath); if (result.errors.some(e => e.severity === 'error')) failedFilePaths.push(filePath); - nodesUpdated += result.nodes.length; + if (!hasGrammarLoadFailure(result.errors)) nodesUpdated += result.nodes.length; // else nothing was stored (#2335) const pause = backpressure?.(); if (pause) await pause; diff --git a/src/index.ts b/src/index.ts index 47330503aa..c1e5d25e58 100644 --- a/src/index.ts +++ b/src/index.ts @@ -25,6 +25,7 @@ import { BuildContextOptions, FindRelevantContextOptions, UnresolvedReference, + IndexHealth, } from './types'; import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; @@ -43,6 +44,7 @@ import { extractFromSource, initGrammars, } from './extraction'; +import { hasGrammarLoadFailure, isFileLevelOnlyLanguage } from './extraction/grammars'; import { ReferenceResolver, createResolver, @@ -2065,6 +2067,34 @@ export class CodeGraph { return this.queries.countGeneratedFiles(); } + /** + * Indexed files whose symbols are missing although their content is current + * — rows a content-hash comparison calls up to date (#2336). Reported by + * `status`; `sync` repairs the `needsReindex` group. + */ + getIndexHealth(): IndexHealth { + const needsReindex: string[] = []; + const parseErrors: string[] = []; + for (const file of this.queries.getFilesWithoutNodesOrWithErrors()) { + const errors = file.errors ?? []; + if (hasGrammarLoadFailure(errors)) { + // Stored without being parsed — its grammar could not load (#2335). + needsReindex.push(file.path); + } else if (file.nodeCount === 0) { + // Every parse stores at least the file node, so zero nodes means a + // wiped row (#1541) or a recorded failure — except file-level-only + // languages and files over the size limit, which are empty on purpose. + if (isFileLevelOnlyLanguage(file.language)) continue; + if (errors.length === 0) needsReindex.push(file.path); + else if (errors.some((e) => e.severity === 'error' || e.code === 'parse_error')) parseErrors.push(file.path); + } else if (errors.some((e) => e.code === 'parse_error')) { + // Parsed, but the tree had errors and no symbols survived. + parseErrors.push(file.path); + } + } + return { needsReindex, parseErrors }; + } + // =========================================================================== // Graph Query Methods // =========================================================================== diff --git a/src/mcp/daemon.ts b/src/mcp/daemon.ts index 73a4969598..db38f97f5e 100644 --- a/src/mcp/daemon.ts +++ b/src/mcp/daemon.ts @@ -54,7 +54,7 @@ import { getDaemonSocketCandidates, getDaemonSocketPath, } from './daemon-paths'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version'; import { releaseWriterLock, swapWriterLock, @@ -120,6 +120,38 @@ export function finalizeDaemonExit( /** How often the daemon sweeps connected clients for a dead peer process (#692). */ const DEFAULT_CLIENT_SWEEP_MS = 30_000; +/** How often the daemon checks that the install it runs from is still in place (#2335). */ +const DEFAULT_INSTALL_CHECK_MS = 30_000; + +/** + * Why the install this daemon was started from no longer matches it, or null + * while it still does. An upgrade replaces or deletes the package directory + * under a running daemon (npm reinstalling the package, the launcher pruning + * an old bundle), and everything the daemon loads lazily from there afterwards + * fails — grammars among them, so every file its watcher re-indexed came out + * empty (#2335). Such a daemon should make way for one started from the + * current install. Any other read failure, or a package.json caught mid-write, + * answers null: the next check looks again. Exported for testing. + */ +export function installChangedReason(packageJsonPath: string, runningVersion: string): string | null { + let raw: string; + try { + raw = fs.readFileSync(packageJsonPath, 'utf8'); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + return code === 'ENOENT' || code === 'ENOTDIR' ? `Install removed (${packageJsonPath} is gone)` : null; + } + let version: unknown; + try { + version = (JSON.parse(raw) as { version?: unknown } | null)?.version; + } catch { + return null; + } + return typeof version === 'string' && version !== runningVersion + ? `Install replaced by v${version} (this daemon is v${runningVersion})` + : null; +} + /** How long the daemon waits for the optional client-hello before proceeding without it. */ const CLIENT_HELLO_TIMEOUT_MS = 3_000; @@ -185,6 +217,7 @@ export class Daemon { private lastActivityAt = Date.now(); private maxIdleTimer: NodeJS.Timeout | null = null; private clientSweepTimer: NodeJS.Timeout | null = null; + private installCheckTimer: NodeJS.Timeout | null = null; private engine: MCPEngine; private stopping = false; private socketPath: string; @@ -372,6 +405,10 @@ export class Daemon { clearInterval(this.clientSweepTimer); this.clientSweepTimer = null; } + if (this.installCheckTimer) { + clearInterval(this.installCheckTimer); + this.installCheckTimer = null; + } process.stderr.write(`[CodeGraph daemon] Shutting down (${reason}; clients=${this.clients.size}).\n`); for (const session of [...this.clients]) { try { session.stop(); } catch { /* best-effort */ } @@ -491,6 +528,31 @@ export class Daemon { this.clientSweepTimer = setInterval(() => this.reapDeadClients(isProcessAlive), sweepMs); this.clientSweepTimer.unref?.(); } + // An install whose version could not be read at startup has nothing to + // compare against; leave that daemon alone. + const installMs = resolveInstallCheckMs(); + if (installMs > 0 && CodeGraphPackageVersion !== '0.0.0-unknown') { + this.installCheckTimer = setInterval(() => { this.checkInstall(); }, installMs); + this.installCheckTimer.unref?.(); + } + } + + /** + * Exit once the install this daemon runs from has been upgraded or removed + * (#2335; see {@link installChangedReason}). Its clients' proxies fall back + * and reconnect, and the next launch starts a daemon from the current + * install. Returns whether the daemon is exiting. The check is injected for + * tests; the timer passes the real one. + */ + checkInstall( + changed: () => string | null = () => installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion), + ): boolean { + if (this.stopping) return false; + const reason = changed(); + if (!reason) return false; + process.stderr.write(`[CodeGraph daemon] ${reason}; exiting so the next session starts a daemon from the current install.\n`); + void this.stop('install changed'); + return true; } /** @@ -865,6 +927,14 @@ function resolveClientSweepMs(): number { return Math.floor(parsed); // 0 disables the sweep } +function resolveInstallCheckMs(): number { + const raw = process.env.CODEGRAPH_DAEMON_INSTALL_CHECK_MS; + if (raw === undefined || raw === '') return DEFAULT_INSTALL_CHECK_MS; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 0) return DEFAULT_INSTALL_CHECK_MS; + return Math.floor(parsed); // 0 disables the check +} + /** * Parse one client-hello line. Returns the peer pids if `line` is a well-formed * client-hello (carries the `codegraph_client` marker), or null otherwise — in diff --git a/src/mcp/index.ts b/src/mcp/index.ts index ac7aa1d0ad..e41bd43e05 100644 --- a/src/mcp/index.ts +++ b/src/mcp/index.ts @@ -44,12 +44,13 @@ import { MCPSession } from './session'; import { Daemon, clearStaleDaemonLock, + installChangedReason, isProcessAlive, tryAcquireDaemonLock, } from './daemon'; import { clearStaleDaemonArtifacts, stopOlderDaemon } from './daemon-registry'; import { connectWithHello, runLocalHandshakeProxy } from './proxy'; -import { CodeGraphPackageVersion } from './version'; +import { CodeGraphPackageJsonPath, CodeGraphPackageVersion } from './version'; import { readWriterLock, assertNoRebuild, @@ -150,6 +151,14 @@ function makeFallbackEngine(root: string): MCPEngine { if (existing && isProcessAlive(existing.pid)) { return readOnlyFallback(`live daemon PID ${existing.pid} holds the project lock`); } + // This session's own install was upgraded or removed underneath it (#2335): + // its code no longer loads whole, so as the project's writer it would only + // keep a daemon from the current install from starting. Its daemon exits + // for the same reason, which is how it got here. + if (CodeGraphPackageVersion !== '0.0.0-unknown' && + installChangedReason(CodeGraphPackageJsonPath, CodeGraphPackageVersion) !== null) { + return readOnlyFallback('this session\'s CodeGraph install was upgraded or removed; restart the session to use the current one'); + } return new MCPEngine({ writerLockRoot: root, queryPool: true, queryPoolDefaultMax: DIRECT_QUERY_POOL_MAX }); } @@ -295,6 +304,12 @@ function spawnDetachedDaemon(root: string, handover = false): void { env, }, ); + // An upgrade can delete the install this launcher runs from — its daemon + // then exits (#2335) and a respawn finds no executable. spawn reports that + // as an asynchronous 'error' event, which would crash this process if + // nobody listened. No daemon binds either way; the caller's poll gives up + // and the session is served in-process. + child.on('error', () => { /* no daemon — see above */ }); child.unref(); } finally { // The child holds its own dup of the log fd now; the launcher doesn't need it. diff --git a/src/mcp/version.ts b/src/mcp/version.ts index b6c919f0f0..e2ee11ea4d 100644 --- a/src/mcp/version.ts +++ b/src/mcp/version.ts @@ -20,10 +20,12 @@ import * as fs from 'fs'; import * as path from 'path'; +/** The `package.json` of the install this process runs from. */ +export const CodeGraphPackageJsonPath = path.join(__dirname, '..', '..', 'package.json'); + function readPackageVersion(): string { try { - const pkgPath = path.join(__dirname, '..', '..', 'package.json'); - const raw = fs.readFileSync(pkgPath, 'utf8'); + const raw = fs.readFileSync(CodeGraphPackageJsonPath, 'utf8'); const parsed = JSON.parse(raw); if (typeof parsed?.version === 'string' && parsed.version.length > 0) { return parsed.version; diff --git a/src/types.ts b/src/types.ts index 44ffaf4e43..f30080002f 100644 --- a/src/types.ts +++ b/src/types.ts @@ -265,6 +265,24 @@ export interface FileRecord { generated?: boolean; } +/** + * Indexed files whose symbols are missing even though their content hash is + * current — what a hash comparison alone calls "up to date" (#2336). Paths are + * project-relative and sorted. + */ +export interface IndexHealth { + /** + * Files `sync` re-indexes: stored while their grammar could not load (#2335) + * or stored with no nodes and no recorded reason (#1541). + */ + needsReindex: string[]; + /** + * Files with a recorded parse error and no symbols from it. Deterministic: + * they stay this way until the file (or the parser) changes. + */ + parseErrors: string[]; +} + // ============================================================================= // Extraction Types // ============================================================================= From 0c7d0f7d422aadbe5b78366e926823afaa5e7bd0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 16:09:30 +0000 Subject: [PATCH 204/363] =?UTF-8?q?perf(resolution):=20remove=20the=20Pyth?= =?UTF-8?q?on=20indexing=20regressions=20since=201.6.2=20=E2=80=94=20graph?= =?UTF-8?q?=20byte-identical=20(#2332)=20(#2344)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Indexing CPython took about three times as long on 1.6.2 as at 290e03f (193 s vs 59 s wall, 8.3 GB vs 3.5 GB peak on a 16-thread Windows box). A CPU profile put ~290 s of the resolver workers' 412 s on two Python checks that first shipped in 1.6.2: - pythonExternalWrites (#2291) read every .py file once per module global its own module types. CPython has 2,375 Python files against the resolver's 1,000-entry content cache, so every read went to disk (184 s). - isPythonLocallyBound (#2198, #2219) re-stripped the calling file for every (function, name), and, asked once per same-named candidate by fitsPythonCallShape, re-derived the enclosing function each time: 3.55M calls for 53K distinct answers (104 s). Now: - A global's own module is read first; when its bindings already leave the type unknown (319 of 379 globals on CPython), no other module is read. - One pass per resolution indexes which Python files spell a name after a dot or a quote, and which write through .__dict__; the write scan visits only those files, in the same path order. Comment stripping only blanks text, so any write the scan accepts is spelled that way in the raw text; a non-ASCII name still reads every file. Index keys are flat copies (a regex capture would pin its whole file). - The repo files an import names are resolved once per mapping, in a WeakMap that lives as long as the resolver's import cache keeps it. - isPythonLocallyBound keeps the current file's stripped lines and per-name module-level answer, and answers a ref once. Verification: nodes, edges and unresolved_refs dumps of CPython and pretix are byte-identical to 6560052a. Medians of 3 interleaved CPython runs on a loaded machine: 290e03f 100 s / 271 s CPU / 3.3 GB, 6560052a 322 s / 801 s / 7.9 GB, this change 123 s / 361 s / 5.5 GB. In the CPU profile pythonExternalWrites drops 184 s -> 16 s and isPythonLocallyBound 104 s -> 13 s. Most of the remaining gap to 290e03f on CPython is JS checks running on its bundled minified d3 (matchDestructuredCallResult, #2334; jsFunctionLocalScope from #2226). __tests__/python-resolution-work.test.ts counts the work instead of timing it: reads of untouched files, strips of the calling file, and node lookups with 16 same-named candidates (4 / 2 / 8 with the fix, 19 / 18 / 128 on main). Each count fails when only its own part of the fix is reverted. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- __tests__/python-resolution-work.test.ts | 102 +++++++++++++++ src/resolution/name-matcher.ts | 155 +++++++++++++++++++---- 3 files changed, 231 insertions(+), 28 deletions(-) create mode 100644 __tests__/python-resolution-work.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f280890b4f..68d7cc062b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - +- Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/python-resolution-work.test.ts b/__tests__/python-resolution-work.test.ts new file mode 100644 index 0000000000..f4faf445b5 --- /dev/null +++ b/__tests__/python-resolution-work.test.ts @@ -0,0 +1,102 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { ReferenceResolver } from '../src/resolution'; +import { stripCommentsForRegex } from '../src/resolution/strip-comments'; + +// Pass-through, so a test can count how often a file's text is stripped. +vi.mock('../src/resolution/strip-comments', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, stripCommentsForRegex: vi.fn(actual.stripCommentsForRegex) }; +}); + +/** + * #2332: resolving Python does a bounded amount of work per file and per call. + * The cross-module write scan for a module global read every Python file once + * per global; the local-binding check stripped the calling file once per + * function, and looked up the function around a call once per same-named + * candidate. On CPython that was minutes of CPU. Counted, never timed. + */ +describe('Python resolution work (#2332)', () => { + let tmpDir: string | undefined; + let cg: CodeGraph | undefined; + + afterEach(() => { + vi.restoreAllMocks(); + vi.mocked(stripCommentsForRegex).mockClear(); + cg?.close(); + cg = undefined; + if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true, maxRetries: 5 }); + tmpDir = undefined; + }); + + /** Index `files`, counting per file what resolution reads, strips and looks up. */ + async function indexCounting(files: Record) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-work-')); + for (const [file, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, file)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, file), content); + } + cg = CodeGraph.initSync(tmpDir); + const context = (cg as unknown as { resolver: ReferenceResolver }).resolver.getResolutionContext(); + const reads = vi.spyOn(context, 'readFile'); + const lookups = vi.spyOn(context, 'getNodesInFile'); + await cg.indexAll(); + return { + graph: cg, + reads: (file: string) => reads.mock.calls.filter(([f]) => f === file).length, + lookups: (file: string) => lookups.mock.calls.filter(([f]) => f === file).length, + strips: (file: string) => vi.mocked(stripCommentsForRegex).mock.calls.filter(([text]) => text === files[file]).length, + }; + } + + it('reads no file once per module global, and strips no file once per function', async () => { + const SCALE = 16; + const conns = Array.from({ length: SCALE }, (_, i) => `conn${i}`); + const files: Record = { + // Globals typed by their own module's writes: each one's type also + // depends on what every other module writes to it. + 'store.py': 'class Store:\n def fetch(self, ids):\n return ids\n', + 'settings.py': `from store import Store\n${conns.map(c => `${c} = None\n`).join('')}\n` + + `def init():\n global ${conns.join(', ')}\n${conns.map(c => ` ${c} = Store()\n`).join('')}`, + 'consumer.py': `import settings\n${conns.map((c, i) => `\ndef cb${i}(pool):\n pool.submit(settings.${c}.fetch)\n`).join('')}`, + // A bare call in every function: each asks whether its function binds the name. + 'helpers.py': 'def helper():\n return 1\n', + 'callers.py': Array.from({ length: SCALE }, (_, i) => `def f${i}():\n return helper()\n\n`).join('') + + 'def shadowed(make):\n helper = make()\n return helper()\n', + }; + // Files no reference touches. + const fillers = Array.from({ length: 20 }, (_, i) => `pkg/filler${i}.py`); + fillers.forEach((file, i) => { files[file] = `def filler${i}(x):\n return x + ${i}\n`; }); + const { graph, reads, strips } = await indexCounting(files); + + // Both paths ran: every global's method value reached Store.fetch, and the + // function that binds `helper` itself calls its own value. + const fetch = graph.getNodesByName('fetch').find(n => n.qualifiedName === 'Store::fetch')!; + expect(graph.getIncomingEdges(fetch.id).filter(e => e.metadata?.fnRef === true) + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(conns.map((_, i) => `cb${i}`).sort()); + const helper = graph.getNodesByName('helper').find(n => n.kind === 'function')!; + expect(graph.getIncomingEdges(helper.id).filter(e => e.kind === 'calls') + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(Array.from({ length: SCALE }, (_, i) => `f${i}`).sort()); + // A few whole-project passes read each file once; the write scan read every + // file once more per global. + expect(Math.max(...fillers.map(reads))).toBeLessThan(SCALE / 2); + // Stripped once for the pass, not once per function. + expect(strips('callers.py')).toBeLessThan(SCALE / 2); + }); + + it('finds the function around a call once, however many functions share its name', async () => { + const CANDIDATES = 16; + const CALLS = 8; + const files: Record = { + 'callers.py': Array.from({ length: CALLS }, (_, i) => `def f${i}():\n return helper()\n\n`).join(''), + }; + for (let k = 0; k < CANDIDATES; k++) files[`lib${k}/helpers.py`] = 'def helper():\n return 1\n'; + const { lookups } = await indexCounting(files); + // Each same-named candidate asks whether the caller binds `helper`; the + // answer is the call's, so the caller's nodes are read per call. + expect(lookups('callers.py')).toBeLessThanOrEqual(2 * CALLS); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e0b4ad28b8..2984a43464 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7,7 +7,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { Language, Node } from '../types'; -import { UnresolvedRef, ResolvedRef, ResolutionContext, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; +import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; @@ -659,26 +659,13 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti let memo = PYTHON_GLOBAL_CLASSES.get(context); if (!memo) { memo = new Map(); PYTHON_GLOBAL_CLASSES.set(context, memo); } if (memo.has(global.id)) return memo.get(global.id)!; - const file = global.filePath; - const external = pythonExternalWrites(global, context); + const own = pythonOwnGlobalWrites(global, context); + const external = own && pythonExternalWrites(global, context); // Each type is resolved in the file that wrote it (its imports name the class). - const writes: Array<{ type: string; file: string }> = [...external.writes]; - let classes: Node[] | null = external.unknown || pythonDynamicGlobalWrite(global.name, file, context) ? null : []; - for (const b of classes ? pythonGlobalBindings(global.name, file, context) : []) { - if (b.kind !== 'assign') { classes = null; break; } - const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; - if (b.type) { - // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. - if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) { classes = null; break; } - writes.push({ type: b.type, file }); - continue; - } - if (b.value === 'None') continue; - if (!constructor) { classes = null; break; } - writes.push({ type: constructor, file }); - } + const writes = external && !external.unknown ? [...external.writes, ...own!] : null; + let classes: Node[] | null = writes && []; const seen = new Set(); - for (const write of classes ? writes : []) { + for (const write of writes ?? []) { const cls = pythonRefClass(write.type, { ...ref, filePath: write.file }, context); if (!cls) { classes = null; break; } if (!seen.has(cls.id)) { seen.add(cls.id); classes!.push(cls); } @@ -687,6 +674,33 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti return classes; } +/** + * The types the global's own module writes to it, or null when a binding + * there leaves its type unknown — whatever other modules write, so they are + * not read (#2332). + */ +function pythonOwnGlobalWrites(global: Node, context: ResolutionContext): Array<{ type: string; file: string }> | null { + return pythonNameScan(context, `own\0${global.id}`, () => { + const file = global.filePath; + if (pythonDynamicGlobalWrite(global.name, file, context)) return null; + const writes: Array<{ type: string; file: string }> = []; + for (const b of pythonGlobalBindings(global.name, file, context)) { + if (b.kind !== 'assign') return null; + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (b.type) { + // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) return null; + writes.push({ type: b.type, file }); + continue; + } + if (b.value === 'None') continue; + if (!constructor) return null; + writes.push({ type: constructor, file }); + } + return writes; + }); +} + /** * The repo files a Python module path can name from `fromFile`. Relative * paths (`..settings`) resolve exactly; absolute ones match a file path @@ -722,9 +736,7 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso const aliases = new Set(); const ambiguous = new Set(); for (const m of context.getImportMappings(filePath, 'python')) { - const dotted = m.isNamespace ? m.source - : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; - const files = pythonModuleFiles(dotted, filePath, context); + const files = pythonImportedFiles(m, filePath, context); if (!files.includes(moduleFile)) continue; // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. // Exactly this module (not `other.a.b`), outside string literals. @@ -736,6 +748,24 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso return { aliases: [...aliases], ambiguous: [...ambiguous] }; } +const PYTHON_IMPORTED_FILES = new WeakMap>(); +/** + * The repo files an import of `filePath` can name. Every global's write scan + * asks again of the same files (#2332); kept for as long as the resolver keeps + * the mapping itself, so the memo never outlives its import cache. + */ +function pythonImportedFiles(m: ImportMapping, filePath: string, context: ResolutionContext): string[] { + let memo = PYTHON_IMPORTED_FILES.get(context); + if (!memo) PYTHON_IMPORTED_FILES.set(context, (memo = new WeakMap())); + let files = memo.get(m); + if (!files) { + const dotted = m.isNamespace ? m.source + : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; + memo.set(m, (files = pythonModuleFiles(dotted, filePath, context))); + } + return files; +} + const PYTHON_MAIN_GUARD = /^if\s+(?:__name__\s*==\s*(['"])__main__\1|(['"])__main__\2\s*==\s*__name__)\s*:/; /** Line indexes inside a top-level `if __name__ == "__main__":` block — script code, not module state. */ function pythonMainBlockLines(filePath: string, context: ResolutionContext): Set { @@ -775,8 +805,8 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write const out = { writes: [] as Array<{ type: string; file: string }>, unknown: false, writers: new Set() }; const name = global.name; const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - for (const file of context.getAllFiles()) { - if (file === global.filePath || !/\.pyi?$/.test(file) || !context.readFile(file)?.includes(name)) continue; + for (const file of pythonWriteCandidates(name, context)) { + if (file === global.filePath) continue; const test = isPythonTestFile(file); const { aliases, ambiguous } = pythonModuleAliases(file, global.filePath, context); if (!test && !aliases.length && !ambiguous.length) continue; @@ -812,6 +842,41 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write }); } +/** + * The Python files that can write a module global named `name`, in path + * order: those that spell it after a dot (`settings.conn = X`) or a quote + * (`setattr(settings, "conn", X)`), and those that write through `.__dict__`, + * whose statement may spell it anywhere. Comment stripping only blanks text, + * so whatever a stripped statement spells, the file's text spells too. The + * files are indexed once per pass, instead of every global reading every + * Python file (#2332). + */ +function pythonWriteCandidates(name: string, context: ResolutionContext): string[] { + const index = pythonNameScan(context, 'write-candidates', () => { + const files = context.getAllFiles().filter(f => /\.pyi?$/.test(f)); + const spelled = new Map(); + const dict: Array<{ at: number; words: string }> = []; + files.forEach((file, at) => { + const source = context.readFile(file) ?? ''; + const names = new Set(); + for (const m of source.matchAll(/[.'"](\w+)/g)) names.add(m[1]!); + for (const n of names) { + const list = spelled.get(n); + // A capture is a sliced view that would pin the file's whole text: key a flat copy. + if (list) list.push(at); else spelled.set(Buffer.from(n).toString(), [at]); + } + // A word-only name is in the text exactly when it is in one of its words. + if (names.has('__dict__')) dict.push({ at, words: [...new Set(source.match(/\w+/g))].join('\n') }); + }); + return { files, spelled, dict }; + }); + // The index holds ASCII words; any other name is looked for in every file's text. + if (!/^\w+$/.test(name)) return index.files.filter(f => context.readFile(f)?.includes(name)); + const hits = new Set(index.spelled.get(name)); + for (const { at, words } of index.dict) if (words.includes(name)) hits.add(at); + return [...hits].sort((a, b) => a - b).map(at => index.files[at]!); +} + /** * Whether the global's own module can write it through its namespace dict: * `globals()` / `vars()` / `sys.modules[__name__]` used as anything but a @@ -853,6 +918,8 @@ function pythonDynamicGlobalWrite(name: string, filePath: string, context: Resol * as a base-typed receiver does. Otherwise, no edge. */ function pythonGlobalMembers(global: Node, member: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + // Unknown from its own module alone: no edge, and no other module to read. + if (!pythonOwnGlobalWrites(global, context)) return []; // A test that installs its own double sees the double, not the production type. if (pythonExternalWrites(global, context).writers.has(ref.filePath)) return []; const classes = pythonGlobalClasses(global, ref, context); @@ -2538,6 +2605,14 @@ function isDecoratedFixture(n: Node, context: ResolutionContext): boolean { } const PY_LOCAL_BINDS = new WeakMap>(); +/** + * The file isPythonLocallyBound last read: its code lines, and per name + * whether its module binds it. Refs arrive grouped by file, so one file per + * context spares re-stripping the file for every function and name (#2332). + */ +const PY_LOCAL_FILE = new WeakMap }>(); +/** The ref isPythonLocallyBound last answered, and the answer. */ +const PY_LOCAL_LAST = new WeakMap(); /** * Whether the function around a Python call — or its module, at top level — @@ -2547,6 +2622,17 @@ const PY_LOCAL_BINDS = new WeakMap>(); * every such call went to one test file's `def view`. */ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + // fitsPythonCallShape asks once per same-named candidate, and finding the + // function around the call reads every node in the file: answer a ref once (#2332). + const last = PY_LOCAL_LAST.get(context); + if (last?.ref === ref && last.name === name) return last.bound; + const bound = pythonLocalBinding(name, ref, context); + PY_LOCAL_LAST.set(context, { ref, name, bound }); + return bound; +} + +/** isPythonLocallyBound's answer, kept per calling function and name. */ +function pythonLocalBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const fn = context.getNodesInFile(ref.filePath) .filter((f) => (f.kind === 'function' || f.kind === 'method') && f.startLine <= ref.line && f.endLine >= ref.line) .sort((a, b) => (a.endLine - a.startLine) - (b.endLine - b.startLine))[0]; @@ -2561,7 +2647,12 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut return false; } // Code only: `{% user_display user as user_display %}` in a docstring binds nothing. - const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + let file = PY_LOCAL_FILE.get(context); + if (file?.filePath !== ref.filePath) { + const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + PY_LOCAL_FILE.set(context, (file = { filePath: ref.filePath, lines, module: new Map() })); + } + const lines = file.lines; const n = name; const assigns = new RegExp(`^\\s*(?:[\\w\\s,*()\\[\\]]*,\\s*)?\\(?\\*?${n}\\)?\\s*(?:,[\\w\\s,*()\\[\\]]*)?(?::[^=]+)?=(?!=)`); const targets = new RegExp(`\\bfor\\s+[\\w\\s,()]*\\b${n}\\b[\\w\\s,()]*\\s+in\\b|\\bas\\s+${n}\\b`); @@ -2582,8 +2673,15 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut } } // A module-level binding (`view = api_view(['GET'])(handler)`). - const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); - for (let line = 0; !bound && line < lines.length; line++) bound = top.test(lines[line] ?? ''); + if (!bound) { + let module = file.module.get(n); + if (module === undefined) { + const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); + module = lines.some(line => top.test(line)); + file.module.set(n, module); + } + bound = module; + } memo.set(key, bound); return bound; } @@ -6606,6 +6704,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { PYTHON_STATEMENT_STARTS.delete(context); PYTHON_GLOBAL_CLASSES.delete(context); PYTHON_NAME_SCANS.delete(context); + PYTHON_IMPORTED_FILES.delete(context); AWAITED_TYPE_MEMO.delete(context); AWAITED_FILES.delete(context); C_STATIC_MEMO.delete(context); @@ -6664,6 +6763,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JVM_PACKAGES.delete(context); MINIFIED_SCRIPTS.delete(context); PY_LOCAL_BINDS.delete(context); + PY_LOCAL_FILE.delete(context); + PY_LOCAL_LAST.delete(context); OVERLOAD_SETS.delete(context); PHP_FILE_SCOPES.delete(context); JAVA_STATIC_IMPORTS.delete(context); From 355afc70b14c22713fee491eafb881b577d34ef1 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:20:15 +0000 Subject: [PATCH 205/363] fix(csharp): walk field and property initializers; a constant never names a type (#2337) (#2348) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(csharp): walk field and property initializers as the member Field declarators and property `= initializer` values were skipped by both extractors, so `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` linked nothing, and a method or class used only from an initializer looked unused. - tree-sitter.ts / csharp.rs: walk each C# field declarator whole with its field on the stack, and every property `value:` (not only `=> expr`) with the property on the stack. Calls, instantiations, static reads, lambda bodies and fn-ref candidates (varinit included) belong to the member; the fn-ref scan skips the walked subtrees, so each candidate is captured once. Attributes stay unwalked. - A target-typed `new()` that is the initializer instantiates the declared type (`List _items = new();` -> List). In bodies it stays invisible, as before. - name-matcher.ts: a .NET type position never names a `constant` (the kind `const` / `static readonly` fields get). Walking initializers exposed `new Version(5, 18)` binding to a `const string Version` and `static readonly Meter Meter = new(...)` instantiating itself. Kernel/wasm parity: 0 diffs on serilog, Newtonsoft.Json and jellyfin, and full-index dumps byte-identical between the two arms. No EXTRACTION_VERSION bump: #2345 already moved it to 28 this release. Co-Authored-By: Claude Opus 5.5 * docs(changelog): credit the report and contributor behind the constant change The type-position change here is what issue #2337 reports (`new Station { … }` unresolved beside a `private const string Station`) and the same one-line change as @drakeo338's #2339. Verified on the issue's own three-file repro: the `instantiates` edge to `Demo.Core::Station`, missing on main, is present with this branch. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/csharp-member-initializers.test.ts | 155 +++++++++++++++++++ __tests__/csharp-type-position-refs.test.ts | 42 +++++ __tests__/kernel-csharp-parity.test.ts | 40 ++++- codegraph-kernel/src/csharp.rs | 94 +++++++++-- docs/design/csharp-kernel-port-checklist.md | 96 ++++++++---- src/extraction/tree-sitter.ts | 103 +++++++++--- src/resolution/name-matcher.ts | 8 +- 8 files changed, 469 insertions(+), 71 deletions(-) create mode 100644 __tests__/csharp-member-initializers.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 68d7cc062b..da460b5b77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. +- In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. +- In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/csharp-member-initializers.test.ts b/__tests__/csharp-member-initializers.test.ts new file mode 100644 index 0000000000..f5125b8321 --- /dev/null +++ b/__tests__/csharp-member-initializers.test.ts @@ -0,0 +1,155 @@ +/** + * C# field and property initializers belong to the member they initialize. + * `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` and + * `public List Items { get; } = new List();` used to be skipped by + * both extractors, so the calls, instantiations and static reads written + * there were lost, and a method passed as a value there was the class's + * reference rather than the member's. + * + * A target-typed `new()` names no type of its own; as an initializer it + * constructs the member's declared type. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const FILES: Record = { + 'Lib.cs': `namespace Lib +{ + public class Crate { } +} +public interface ILogger { } +public static class LogManager +{ + public static ILogger GetLogger(System.Type t) => null; +} +public static class Helper +{ + public static int Compute(int n) => n; + public static int Seed() => 1; + public static void Register(System.Action a) { } + public static System.Action Wrap(System.Action a) => a; +} +public class Widget +{ + public Widget() { } + public Widget(int n) { } + public int Size { get; set; } +} +public class Bag { } +public static class Defaults +{ + public const string Name = "x"; + public static readonly Widget Empty = new Widget(); +} +`, + 'Box.cs': `using Lib; + +public class Box +{ + private readonly ILogger _log = LogManager.GetLogger(typeof(Box)); + private int _a = 1, _b = Helper.Compute(2); + private static readonly int Max = Helper.Seed(); + private readonly Widget _made = new Widget(3); + private readonly Widget _typed = new() { Size = Helper.Seed() }; + private Widget? _maybe = new(); + private readonly Bag _bag = new(); + private readonly Lib.Crate _crate = new(); + private readonly string _label = Defaults.Name; + private readonly System.Func _square = x => Helper.Compute(x * x); + private readonly System.Action _later = () => Helper.Register(Handle); + private readonly System.Action _direct = Handle; + [System.Obsolete("use Helper.Seed()")] private int _flagged; + public Bag Items { get; } = new Bag(); + public Widget Made { get; set; } = new(); + public Widget Copy { get; } = Defaults.Empty; + public System.Action Wrapped { get; } = Helper.Wrap(Handle); + public int Seeded { get; } = Helper.Seed(); + private static void Handle() { } +} +`, +}; + +describe('C# field and property initializers', () => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-initializers-')); + for (const [rel, content] of Object.entries(FILES)) fs.writeFileSync(path.join(root, rel), content); + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('are walked as the member they initialize (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const member = (qualifiedName: string): Node => { + const node = graph.getNodesInFile('Box.cs').find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; + }; + const targets = (node: Node, kind: string): string[] => + graph + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); + + // Calls, per declarator. + expect(targets(member('Box::_log'), 'calls')).toEqual(['LogManager::GetLogger']); + expect(targets(member('Box::_a'), 'calls')).toEqual([]); + expect(targets(member('Box::_b'), 'calls')).toEqual(['Helper::Compute']); + expect(member('Box::Max').kind).toBe('constant'); + expect(targets(member('Box::Max'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::Seeded'), 'calls')).toEqual(['Helper::Seed']); + + // Instantiations, including a target-typed `new()` of the declared type. + expect(targets(member('Box::_made'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::_maybe'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_bag'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::_crate'), 'instantiates')).toEqual(['Lib::Crate']); + expect(targets(member('Box::Items'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::Made'), 'instantiates')).toEqual(['Widget']); + + // Static reads. + expect(targets(member('Box::_label'), 'references')).toContain('Defaults'); + expect(targets(member('Box::Copy'), 'references')).toContain('Defaults'); + + // Lambda bodies belong to the member the lambda initializes. + expect(targets(member('Box::_square'), 'calls')).toEqual(['Helper::Compute']); + expect(targets(member('Box::_later'), 'calls')).toEqual(['Helper::Register']); + + // A method passed as a value is the member's reference, captured once — + // not the class's as well. + expect(targets(member('Box::_later'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::_direct'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::Wrapped'), 'calls')).toEqual(['Helper::Wrap']); + expect(targets(member('Box::Wrapped'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box'), 'references')).not.toContain('Box::Handle'); + + // Attribute arguments are not an initializer, and the class itself + // calls and creates nothing. + expect(targets(member('Box::_flagged'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'instantiates')).toEqual([]); + }); +}); diff --git a/__tests__/csharp-type-position-refs.test.ts b/__tests__/csharp-type-position-refs.test.ts index c000d92105..ccfba566b9 100644 --- a/__tests__/csharp-type-position-refs.test.ts +++ b/__tests__/csharp-type-position-refs.test.ts @@ -69,4 +69,46 @@ describe('C#: a type position names a type', () => { cg.close(); } }); + + it('never a constant that shares the name', async () => { + // `const` and `static readonly` fields are constants: jellyfin's `new + // Version(5, 18)` (System.Version) bound to a `const string Version` + // claim name, and serilog's `static readonly Meter Meter = new(…)` + // instantiated itself. + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-type-refs-')); + roots.push(root); + const files: Record = { + 'src/Claims.cs': `namespace App; +public static class ClaimTypes { + public const string Version = "v"; +} +`, + 'src/Encoder.cs': `namespace App; +using System; +public class ValueFormatter { } +public class Encoder { + private static readonly Version MinVersion = new Version(5, 18); + private static readonly Meter Meter = new("app"); + private static readonly ValueFormatter ValueFormatter = new(); + public void Check() { var v = new Version(1, 0); } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const members = cg.getNodesInFile('src/Encoder.cs').filter((n) => n.kind !== 'file'); + const targets = cg + .getOutgoingEdgesFrom(members.map((n) => n.id), ['references', 'instantiates', 'type_of']) + .map((e) => `${cg.getNode(e.source)!.name} ${e.kind} ${cg.getNode(e.target)!.kind}:${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + expect(targets.filter((t) => t.includes(' constant:'))).toEqual([]); + expect(targets).toContain('ValueFormatter instantiates class:App::ValueFormatter'); + } finally { + cg.close(); + } + }); }); diff --git a/__tests__/kernel-csharp-parity.test.ts b/__tests__/kernel-csharp-parity.test.ts index 3d09c19d98..7a15371891 100644 --- a/__tests__/kernel-csharp-parity.test.ts +++ b/__tests__/kernel-csharp-parity.test.ts @@ -8,12 +8,12 @@ * * - Torture.cs — block namespace + nested/second-namespace quirks, * base_list shapes, records, properties (incl. the bare-identifier - * signature loss, accessor bodies walked as the property and - * never-walked initializers), fields/constants, + * signature loss; accessor bodies and initializers walked as the + * property), fields/constants (initializers walked as the field), * events/operators/indexer/destructor (no nodes, calls → class), ctor * initializer hole, explicit interface impl, local functions, the call * zoo (raw member-access texts, chained re-encode, `(myDel)(x)` conv, - * `nameof`), instantiation shapes (incl. invisible `new()`/`new {}`/ + * `nameof`), instantiation shapes (incl. invisible body `new()`/`new {}`/ * arrays), static value reads, C# type refs, fn-ref candidates * (`+=` subscription, `this.X` bare-name form, initializer lists), * value-ref targets + local shadow prune, preprocessor passthrough. @@ -169,10 +169,9 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { minNodes: 2, }, { - // Accessor bodies and `=> expr` are walked as the property (calls, - // instantiations, static reads, fn-ref candidates); an `= initializer` - // is only scanned for candidates, attributed to the class. - name: 'property bodies are walked as the property; initializers only scanned', + // Accessor bodies, `=> expr` and `= initializer` are walked as the + // property (calls, instantiations, static reads, fn-ref candidates). + name: 'property bodies and initializers are walked as the property', source: [ 'public class C {', ' void H(int v) { }', @@ -185,6 +184,33 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { ].join('\n'), minNodes: 7, }, + { + // Each field declarator and each property `= initializer` is walked + // with its member on the stack — lambdas included — and its fn-ref + // candidates (varinit too) are the member's, captured once. A + // target-typed `new()` there instantiates the declared type; attribute + // arguments stay with the class's candidates-only scan. + name: 'field and property initializers are walked as the member', + source: [ + 'public class C {', + ' static void H() { }', + ' private readonly ILogger _log = LogManager.GetLogger(typeof(C));', + ' int a = 1, b = Calc.Max(2);', + ' private static readonly List Table = new() { H };', + ' private Widget? _w = new(1) { Size = Make() };', + ' private global::Ns.Box _g = new(), _h = (new());', + ' private (int, int) _t = new();', + ' Action g = () => Register(H);', + ' Del d = H;', + ' public Widget P { get; } = new();', + ' public Action R { get; } = Wrap(H);', + ' public string S { get; } = Defaults.Name;', + ' [Attr(Make(H))] public int T { get; set; } = 3;', + '}', + '', + ].join('\n'), + minNodes: 17, + }, ]; for (const m of MICROS) { diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 04e20ca5a5..c10696ec99 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -5,8 +5,8 @@ //! path, bug-for-bug, verified by scripts/kernel-parity.mjs and the full-index //! dump-diff gate. The authoritative quirk list is //! docs/design/csharp-kernel-port-checklist.md — including every deliberate -//! emission hole (field/property initializers, constructor initializers, -//! delegates/events/operators/indexers, top-level locals) and garbage ref +//! emission hole (constructor initializers, delegates/events/operators/ +//! indexers, top-level locals) and garbage ref //! (`(repo)` primary-ctor extends, `: byte` enum extends, `nameof` calls) //! this file preserves on purpose. Positions in UTF-16 code units. Files whose //! parse tree contains ERRORS defer to the wasm extractor. @@ -572,16 +572,18 @@ impl<'t> Walker<'t> { self.extract_enum(node); skip_children = true; } else if kind == "property_declaration" && self.inside_class_like() { - // The code a property runs — its accessor bodies and `=> expr` — - // is walked with the property on the stack (propertyBodies). The - // candidates-only scan covers the rest (an `= initializer`), + // The code a property runs — its accessor bodies, `=> expr` and + // `= initializer` — is walked with the property on the stack + // (propertyBodies). The candidates-only scan covers the rest, // skipping what the body walk captured. let walked: Vec = match self.extract_property(node) { Some((row, name)) => { let bodies = property_bodies(node); if !bodies.is_empty() { self.stack.push(Scope { row, kind: "property", name }); + let declared = node.child_by_field_name("type"); for body in &bodies { + self.extract_target_typed_new(Some(*body), declared); self.visit_function_body(*body); } self.stack.pop(); @@ -593,8 +595,11 @@ impl<'t> Walker<'t> { self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "field_declaration" && self.inside_class_like() { - self.extract_field(node); - self.scan_fn_ref_subtree(node, 0, &[]); + // Each declarator, `= initializer` included, is walked with its + // field on the stack (extract_field); the candidates-only scan + // covers the rest, skipping the declarators that walk captured. + let walked = self.extract_field(node); + self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "local_declaration_statement" && !self.inside_class_like() { // Top-level statements: extractVariable's generic fallback finds no @@ -864,8 +869,11 @@ impl<'t> Walker<'t> { } /// extractField (2046) — field_declaration; each declarator becomes a - /// field/constant node anchored at the DECLARATOR. - fn extract_field(&mut self, node: Node<'t>) { + /// field/constant node anchored at the DECLARATOR, and the declarator — + /// its `= initializer` — is walked with that node on the stack. Returns + /// the walked declarators, for the fn-ref scan to skip. + fn extract_field(&mut self, node: Node<'t>) -> Vec { + let mut walked = Vec::new(); let docstring = preceding_docstring(node, self.src); let visibility = Some(self.visibility_of(node)); let is_static = Some(self.is_static(node)); @@ -932,6 +940,15 @@ impl<'t> Walker<'t> { // multi-declarator fields emit the type refs once PER // declarator, each from its own field node. self.extract_csharp_type_refs(node, row); + // The initializer is the declarator's last, unnamed child: + // its calls, instantiations, static reads and fn-ref + // candidates (varinit included) are the field's. + self.stack.push(Scope { row, kind: field_kind, name }); + let declared = var_decl.and_then(|vd| vd.child_by_field_name("type")); + self.extract_target_typed_new(last_named_child(decl), declared); + self.visit_function_body(decl); + self.stack.pop(); + walked.push(decl.id()); } } } else { @@ -951,6 +968,44 @@ impl<'t> Walker<'t> { ); } } + walked + } + + /// extractTargetTypedNew (tree-sitter.ts) — a target-typed `new()` + /// (implicit_object_creation_expression) names no type, so it is no + /// instantiation kind; as a field's or property's initializer it + /// constructs the declared type (`List _items = new();` → List). + /// Emitted from the stack top (the member) at the `new()`. + fn extract_target_typed_new(&mut self, value: Option>, declared: Option>) { + let Some(value) = value else { return }; + if value.kind() != "implicit_object_creation_expression" || self.stack.is_empty() { + return; + } + let Some(class_name) = declared.and_then(|t| self.class_type_name(t)) else { return }; + let from = self.top_row(); + self.push_ref_at(from, &class_name, edge_kind_index("instantiates").unwrap(), value); + } + + /// csharpClassTypeName (tree-sitter.ts) — the class a declared type + /// names, as `new T()` would name it: `List` → List, `Ns.Foo` / + /// `global::Foo` → Foo, `Foo?` → Foo. Predefined, array, tuple and pointer + /// types name no class. + fn class_type_name(&self, node: Node) -> Option { + match node.kind() { + "identifier" => { + let text = self.text(node); + (!text.is_empty()).then(|| text.to_string()) + } + "generic_name" => { + let ident = (0..node.named_child_count()) + .filter_map(|i| node.named_child(i)) + .find(|c| c.kind() == "identifier")?; + self.class_type_name(ident) + } + "qualified_name" | "alias_qualified_name" => self.class_type_name(node.child_by_field_name("name")?), + "nullable_type" => self.class_type_name(node.child_by_field_name("type")?), + _ => None, + } } /// extractMethod (1737) — method_declaration + constructor_declaration. @@ -1142,7 +1197,8 @@ impl<'t> Walker<'t> { let Some(ctor) = ctor else { return }; // `new List()` → `List`; `new Ns.Foo()` → `Foo`. Target-typed // `new()` / anonymous `new { }` / arrays `new T[n]` never reach here - // (not in INSTANTIATION_KINDS) — invisible by design. + // (not in INSTANTIATION_KINDS) — invisible by design, except a `new()` + // that initializes a field or property (extract_target_typed_new). let class_name = strip_generic_and_qualifier(self.text(ctor)); if !class_name.is_empty() { let from = self.top_row(); @@ -1492,8 +1548,9 @@ impl<'t> Walker<'t> { return; } // functionTypes is EMPTY for C#; the literal halt list applies — - // lambda_expression IS C#'s lambda, so initializer lambdas stop the - // scan; anonymous_method_expression is NOT listed and scans through. + // lambda_expression IS C#'s lambda, so a lambda the scan reaches (a + // top-level statement's) stops it; anonymous_method_expression is NOT + // listed and scans through. Member initializers are walked instead. if depth > 0 && matches!( node.kind(), @@ -1643,8 +1700,9 @@ impl<'t> Walker<'t> { } /// propertyBodies (tree-sitter.ts) — the parts of a property that run code: -/// each accessor's `body` (a block or `=> expr`) and an expression-bodied -/// property's `=> …` value. An `= initializer` value is not a body. +/// each accessor's `body` (a block or `=> expr`), then the property's +/// `value` — an expression body's `=> …` or an `= initializer`. Attributes +/// are not walked. fn property_bodies(node: Node) -> Vec { let mut bodies = Vec::new(); if let Some(accessors) = node.child_by_field_name("accessors") { @@ -1659,13 +1717,15 @@ fn property_bodies(node: Node) -> Vec { } } if let Some(value) = node.child_by_field_name("value") { - if value.kind() == "arrow_expression_clause" { - bodies.push(value); - } + bodies.push(value); } bodies } +fn last_named_child(node: Node) -> Option { + node.named_child(node.named_child_count().checked_sub(1)?) +} + fn find_anonymous_class_body(node: Node) -> Option { for i in 0..node.named_child_count() { if let Some(child) = node.named_child(i) { diff --git a/docs/design/csharp-kernel-port-checklist.md b/docs/design/csharp-kernel-port-checklist.md index e817968c95..919e767687 100644 --- a/docs/design/csharp-kernel-port-checklist.md +++ b/docs/design/csharp-kernel-port-checklist.md @@ -234,8 +234,8 @@ all PRESERVE): | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | body `enum_member_declaration_list` required (bodiless → no node); extractInheritance sees `base_list` → **the underlying type `: byte` emits an `extends` ref named `byte`** (quirk, §inheritance); `enum_member_declaration` children → extractEnumMembers:1958 — `name` field path: ONE `enum_member` node per member, positioned at the member node (attributes included in its span), values/attributes ignored; non-member children (preproc_*, comment) → visitNode (no-op) | | `method_declaration` | methodTypes:1027 → extractMethod:1737 | classifyMethodNode absent → always extractMethod. Gate 1747 passes via class-like (a method_declaration outside a type does not occur in non-erroring C# — top-level `void M(){}` parses as local_function_statement, probed); bodyless interface/partial signatures mint nodes with no body walk; **expression-bodied methods have `body: arrow_expression_clause` (a real body FIELD, probed) → walked** | | `constructor_declaration` | methodTypes → extractMethod | name field = the class-name identifier → **method node named like the class**; returnType undefined; **`constructor_initializer` (`: base(args)` / `: this(args)`) is a sibling of the body field → NEVER walked → calls inside initializer args are LOST** (probed); expression-bodied ctor body = arrow_expression_clause → walked | -| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`) and an expression-bodied `=> expr` `value:` — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; changed 2026-10-04, previously never walked), then scanFnRefSubtree (capture-only, attributed to the class) over the rest — **the `= initializer` stays unwalked** — skipping the walked bodies, + skipChildren. §property below | -| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator + scanFnRefSubtree + skipChildren → **field initializers emit no calls/instantiates/static-member refs** (fn-ref candidates only). §field below | +| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`), then the `value:` (an expression body's `=> expr` or an `= initializer`) — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; accessor/arrow bodies changed 2026-10-04, `= initializer` 2026-10-05 — previously never walked); a target-typed `= new()` value instantiates the declared type (extractTargetTypedNew); then scanFnRefSubtree (capture-only, attributed to the class) over the rest (attributes, type) skipping the walked bodies, + skipChildren. §property below | +| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator, **each declarator walked whole by visitFunctionBody with its field pushed** (its `= initializer`: calls, instantiates, static reads, fn-ref candidates incl. varinit attribute to the field; changed 2026-10-05, previously never walked); a target-typed `= new()` instantiates the declared type; then scanFnRefSubtree (capture-only, attributed to the class) skipping the walked declarators, + skipChildren. §field below | | `local_declaration_statement` | variableTypes:1098 (only reachable at top level — global statements; body locals go through visitFunctionBody instead) | not class-like → extractVariable:2538 → **generic fallback (2863-2881) finds no direct `identifier`/`variable_declarator` children (the declarator nests inside `variable_declaration`, probed) → ZERO nodes minted**; isClassScopeConstantAssignment (1508) needs node.type `assignment` → never true. skipChildren=true + scanFnRefSubtree → **a top-level `var builder = WebApplication.CreateBuilder(args);` produces NO node, NO calls ref, NO instantiates** — only fn-ref candidates. PRESERVE | | `using_directive` | importTypes:1209 → extractImport:3170 | hook (§config) → import node + ONE generic `imports` ref {fromNodeId: nodeStack top (namespace node if present, else file), referenceName: moduleName, line/col of the directive}; **no per-binding emitter** (the TS/py/rust/php/ruby ladder at 3197-3234 excludes csharp) | | `invocation_expression` (top level — global statements) | callTypes:1248 → extractCall:3684 | fires with caller = file/namespace node; children still visited (no skipChildren) so nested invocations recurse | @@ -306,9 +306,10 @@ for C#) or bare `name`. QUIRKS (probed, PRESERVE): children [modifier, predefined_type, identifier, **arrow_expression_clause (`value:` field)**] → typeNode = predefined_type → signature `"int Computed"`; the arrow clause is a property body (walked as the property, below). -- `{ get; } = new();` initializers: the `value:` implicit_object_creation is - NOT a body → no refs, no instantiates (candidates-only scan, attributed to - the class); the accessor_list is excluded from the type scan. +- `{ get; } = new();` initializers: the `value:` is the + implicit_object_creation_expression itself (no type child) → walked as a + body, and extractTargetTypedNew emits `instantiates` named for the + declared `type` (below); the accessor_list is excluded from the type scan. Then extractDecoratorsFor (no-op) and **extractTypeAnnotations (2037) → extractCsharpTypeRefs** — the `type` field IS walked for refs (so `public @@ -316,10 +317,22 @@ List Items` emits references `List` + `Foo` even though the signature kept the raw text). The returned node is pushed while propertyBodies (tree-sitter.ts; csharp.rs `property_bodies`) are walked: every `accessor_declaration`'s `body` field (block or arrow_expression_clause, in -accessor order), then the property's `value:` when it is an -arrow_expression_clause. The fn-ref scan that follows skips those subtrees -by node id, so a candidate is captured once — from the property when it sits -in a body, from the class when it sits in an initializer. (The +accessor order), then the property's `value:` whatever it is (an +arrow_expression_clause or an `= initializer` expression). Each body is +preceded by **extractTargetTypedNew(body, `type` field)** (csharp.rs +`extract_target_typed_new`): when the body IS an +`implicit_object_creation_expression` it emits ONE `instantiates` ref from +the property at the `new()`'s position, named by csharpClassTypeName +(csharp.rs `class_type_name`): `identifier` → its text; `generic_name` → its +identifier (`List` → `List`); `qualified_name` / `alias_qualified_name` +→ recurse into the `name` field (`Ns.Foo` → `Foo`, `global::Foo` → +`Foo`, `Outer.Inner` → `Inner`); `nullable_type` → recurse into `type` +(`Foo?` → `Foo`); anything else (predefined/array/tuple/pointer) → nothing. +Only a `new()` that IS the value counts — `(new())`, `c ? new() : null` and +nested `new()`s emit nothing. The fn-ref scan that follows skips the walked +bodies by node id, so a candidate is captured once — from the property; only +what is left (attribute arguments, e.g. `[Attr(Make(H))]` → `H` from the +class) is the class's. Attributes are never walked for calls. (The classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) ### extractField (2046) — field_declaration @@ -342,6 +355,23 @@ classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) the variable_declaration's `type` field (5905-5909) → **multi-declarator fields (`Foo A, B;`) emit the type refs ONCE PER DECLARATOR**, each from its own field node. +- Then, per created field node (changed 2026-10-05 — initializers used to + emit nothing but fn-ref candidates, attributed to the class): push the + field, **extractTargetTypedNew(the declarator's LAST named child, the + variable_declaration's `type` field)** (§property above — a target-typed + `private readonly List _items = new();` instantiates `List`), then + **visitFunctionBody(the whole declarator)**, pop. The C# declarator has no + `value` field — the initializer is its last, unnamed child (probed: + [name, expression]; a fixed buffer is [name, bracketed_argument_list]) — + so the declarator is walked whole, like VB.NET's: maybeCaptureFnRefs fires + on it first (varinit: `Del d = Handler;` → candidate `Handler` FROM THE + FIELD), then the name (inert) and the initializer — calls, instantiates, + static reads (`Defaults.Name` → `Defaults`) and lambda bodies (no halt: + `Func f = x => Compute(x)` → the field calls `Compute`) attribute + to the field. extractField returns the walked declarator ids and the + dispatcher's scanFnRefSubtree skips them, so each candidate is captured + once, by the field. (Java's walk of its `value` field reports nothing back + — its class-level scan still captures those candidates too, unchanged.) - docstring/visibility/isStatic computed once from the outer declaration, shared by all declarators. The PHP property_element and bare-fallback branches (2078-2154) are unreachable for C#. @@ -425,16 +455,20 @@ statements at top level) AND visitFunctionBody:5145. QUIRKS, PRESERVE: **`implicit_object_creation_expression` (`new()`) and `anonymous_object_creation_expression` (`new { X = 1 }`) and `array_creation_expression` (`new Widget[10]`) are NOT in INSTANTIATION_KINDS -→ no instantiates refs** (target-typed `new()` — everywhere in modern C# — is -invisible); object/collection initializer args and `new[] { Mk() }` contents -still recurse to their own calls. Top-level `var w = new Widget();` emits -nothing at all (§local_declaration_statement). +→ no instantiates refs** (a target-typed `new()` in a body — `Widget c = +new();`, an argument, a return — is invisible); the one exception is a +`new()` that IS a field's or property's initializer, which +extractTargetTypedNew names for the declared type (§property, §field). +Object/collection initializer args and `new[] { Mk() }` contents still +recurse to their own calls. Top-level `var w = new Widget();` emits nothing +at all (§local_declaration_statement). ### extractStaticMemberRef (4750) — csharp ∈ STATIC_MEMBER_LANGS (345) Called for EVERY node in visitFunctionBody (5218) — body walker only (never -visitNode, so class-level field initializers and top-level statements emit no -static refs). Node gate: MEMBER_ACCESS_TYPES (323) contains +visitNode, so top-level statements emit no static refs; field and property +initializers ARE body-walked, so `= Defaults.Name` reads attribute to the +member). Node gate: MEMBER_ACCESS_TYPES (323) contains `member_access_expression`. Skip when the access IS a call's callee (4772-4779: parent ∈ callTypes && callee.startIndex === node.startIndex — so `Console.WriteLine(…)`'s access is skipped but `DoThing(Constants.MAX)`'s @@ -621,14 +655,17 @@ layers: `argument`→null (descend named children). special: nothing (not idTypes/special). - Capture fires from visitNode:990, visitFunctionBody:5137, and scanFnRefSubtree (property/field/variable declarations + top-level - statements, depth ≤12, capture-only). **scanFnRefSubtree's halt list - (tree-sitter.ts:606-612) includes the literal type `lambda_expression` — - which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda in a - field/property initializer (`Action A = () => Register(H);` yields no - candidates from inside the lambda), while `anonymous_method_expression` - (`delegate() { … }`) is NOT in the list and is scanned through. Method-BODY - lambdas are unaffected (visitFunctionBody recursion has no halt — capture - fires per node). + statements, depth ≤12, capture-only — skipping the subtrees the body + walker already went through: property bodies/initializers and field + declarators, whose candidates are the member's). **scanFnRefSubtree's halt + list (tree-sitter.ts:606-612) includes the literal type `lambda_expression` + — which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda it + reaches (a top-level statement's), while `anonymous_method_expression` + (`delegate() { … }`) is NOT in the list and is scanned through. Body-walked + lambdas — methods, accessors, and since 2026-10-05 field/property + initializers (`Action A = () => Register(H);` → candidate `H` from `A`) — + are unaffected (visitFunctionBody recursion has no halt — capture fires per + node). - Flush gate (flushFnRefCandidates:639): generated-file skip; `this.`-prefixed names skip the gate (C# never produces them — its this-forms are bare); otherwise name ∈ definedHere ∪ importedNames. **definedHere = same-file @@ -762,10 +799,14 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `struct Fwd;` (NO node); interface with bodyless method + property + default-impl arrow method; enum with `: byte` (extends `byte` quirk), attributed member, valued members; const + static-readonly (→ `constant`) - + multi-declarator + instance fields (signatures `Type name`); + + multi-declarator + instance fields (signatures `Type name`), field + initializers (→ the field: `= new() { TargetCb }` instantiates the declared + `List` and the list candidate is the field's); `protected internal` (→ protected); property shapes: predefined-type, bare-identifier type (signature loses type), generic type, expression-bodied - (`=>` calls → the property), `{ get; } = new();` (initializer LOST), + (`=>` calls → the property), `{ get; } = new();` (→ the property, + instantiates the declared `Widget`), `{ get; } = () => Register(H)` (the + lambda's call and candidate → the property), accessor bodies with calls (→ the property); event_field_declaration + event_declaration with add/remove bodies (no nodes; accessor calls → class); operator + conversion operator + indexer + @@ -779,7 +820,7 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `Foo.Create(1).Bar()` (re-encode + inner both) + `GetThing().Bar()`, `(myDel)(x)` (conv regex → `myDel`), `nameof(Widget)` (calls ref `nameof`); `new Widget(…) { … }` (instantiates + initializer calls) + `new Ns.Foo()` - (strip both) + `new()` / `new { }` / `new Widget[10]` (all NOTHING); + (strip both) + body `new()` / `new { }` / `new Widget[10]` (all NOTHING); static value reads (`ReadType.ReadAsDouble` → `ReadType`; `Outer.Inner.DEEP` → `Outer`; skip-as-callee; lowercase skip); type refs: params (nullable/array/tuple-element/generic/qualified/`dynamic`), returns @@ -792,7 +833,8 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. function + trailing `partial class Program`; fn-refs: `Register(HandleThing)`, `Register(this.HandleThing)` (bare name), `Register(C.StaticHandler)` (nothing), `Click += OnClick`, initializer_expression list, varinit - (`Action g = () => …` no candidate; `Del d = Handler;` candidate), + (`Action g = () => …` no candidate; `Del d = Handler;` candidate — from + the method for a local, from the field for a field initializer), `this.x = x` param-storage skip; value-refs: const target + reader methods + a `var MaxItems = …` local shadow (prune) + `static readonly` multi-target; preprocessor: `#region`/`#endregion`/`#pragma`/`#nullable`/`#define` diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 9a10862475..222291f14c 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -239,6 +239,29 @@ function scalaBaseTypeName(node: SyntaxNode | null, source: string): string | nu } } +/** + * The class a C# declared type names, as `new T()` would name it: `Foo`, + * `List` → `List`, `Ns.Foo` / `global::Foo` → `Foo`, `Foo?` → `Foo`. + * Predefined, array, tuple and pointer types name no class → null. Mirrored + * in the kernel (csharp.rs class_type_name). + */ +function csharpClassTypeName(node: SyntaxNode | null, source: string): string | null { + if (!node) return null; + switch (node.type) { + case 'identifier': + return getNodeText(node, source) || null; + case 'generic_name': + return csharpClassTypeName(node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier') ?? null, source); + case 'qualified_name': + case 'alias_qualified_name': + return csharpClassTypeName(getChildByField(node, 'name'), source); + case 'nullable_type': + return csharpClassTypeName(getChildByField(node, 'type'), source); + default: + return null; + } +} + /** * Resolve the declared identifier inside a C declarator. A `declaration`'s * `declarator` field nests the name through `init_declarator` (with value), @@ -1287,22 +1310,27 @@ export class TreeSitterExtractor { const bodies = propNode ? this.propertyBodies(node) : []; if (propNode && bodies.length > 0) { this.nodeStack.push(propNode.id); - for (const body of bodies) this.visitFunctionBody(body, propNode.id); + const declaredType = getChildByField(node, 'type'); + for (const body of bodies) { + this.extractTargetTypedNew(body, declaredType); + this.visitFunctionBody(body, propNode.id); + } this.nodeStack.pop(); } - // Whatever the body walk didn't cover (a C# `= initializer`, any other - // language's whole declaration) is scanned for function-as-value + // Whatever the body walk didn't cover (a C# property's attributes, any + // other language's whole declaration) is scanned for function-as-value // candidates (#756); the bodies captured their own. this.scanFnRefSubtree(node, 0, new Set(bodies.map((b) => b.id))); skipChildren = true; } // Check for class fields (e.g. Java field_declaration, C# field_declaration) else if (this.extractor.fieldTypes?.includes(nodeType) && this.isInsideClassLikeNode()) { - this.extractField(node); - // Field initializers aren't walked — scan for function-as-value - // candidates (#756): Java `List table = List.of(Main::cb)`, - // C# `List> table = new() { TargetCb }`. - this.scanFnRefSubtree(node, 0); + const walked = this.extractField(node); + // Scan the declaration for function-as-value candidates (#756): Java + // `List table = List.of(Main::cb)`. A C# declarator + // extractField walked captured its own (`List> table = + // new() { TargetCb }` is the field's), so the scan skips it. + this.scanFnRefSubtree(node, 0, walked); skipChildren = true; } // Check for variable declarations (const, let, var, etc.) @@ -2324,9 +2352,9 @@ export class TreeSitterExtractor { * VB.NET writes its `Get` / `Set` blocks, `= initializer` and `As New T` * as children of the declaration itself, so the declaration is walked * whole, the way its methods are (resolveBody). C# runs code in each - * accessor's body (`get { … }`, `set => …`) and in an expression-bodied - * property's `=> …`; its `= initializer`, like a field's, stays unwalked. - * Mirrored in the kernel (csharp.rs property_bodies). + * accessor's body (`get { … }`, `set => …`) and in the property's `value`: + * an expression body's `=> …` or an `= initializer`. Its attributes are + * not walked. Mirrored in the kernel (csharp.rs property_bodies). */ private propertyBodies(node: SyntaxNode): SyntaxNode[] { if (this.language === 'vbnet') return [node]; @@ -2337,16 +2365,41 @@ export class TreeSitterExtractor { if (body) bodies.push(body); } const value = getChildByField(node, 'value'); - if (value?.type === 'arrow_expression_clause') bodies.push(value); + if (value) bodies.push(value); return bodies; } + /** + * A C# target-typed `new()` names no type, which is why INSTANTIATION_KINDS + * leaves `implicit_object_creation_expression` out. As a field's or + * property's initializer, though, it constructs the declared type: + * `private readonly List _items = new();` instantiates List, as + * `new List()` does. Emitted from the node-stack top (the member). + * Mirrored in the kernel (csharp.rs extract_target_typed_new). + */ + private extractTargetTypedNew(value: SyntaxNode | null, declaredType: SyntaxNode | null): void { + if (this.language !== 'csharp' || value?.type !== 'implicit_object_creation_expression') return; + const className = declaredType ? csharpClassTypeName(declaredType, this.source) : null; + const fromNodeId = this.nodeStack[this.nodeStack.length - 1]; + if (!className || !fromNodeId) return; + this.unresolvedReferences.push({ + fromNodeId, + referenceName: className, + referenceKind: 'instantiates', + line: value.startPosition.row + 1, + column: value.startPosition.column, + }); + } + /** * Extract a class field declaration (e.g. Java field_declaration, C# field_declaration). * Extracts each declarator as a 'field' kind node inside the owning class. + * Returns the C# declarators it walked, for the function-as-value scan to + * skip. */ - private extractField(node: SyntaxNode): void { - if (!this.extractor) return; + private extractField(node: SyntaxNode): Set { + const walked = new Set(); + if (!this.extractor) return walked; const docstring = this.docstringFor(node); const visibility = this.extractor.getVisibility?.(node); @@ -2401,7 +2454,7 @@ export class TreeSitterExtractor { isStatic, }); } - return; + return walked; } } @@ -2447,11 +2500,24 @@ export class TreeSitterExtractor { // edge at all and `target` looked callerless. Keyed on the `value` // FIELD, which only Java's `variable_declarator` carries. VB.NET // writes `= expr` (the declarator's `initializer`) or `As New T(…)` - // (inside its as_clause), so its whole declarator is walked. C# - // and PHP spell their initializer differently and are untouched. - const valueNode = this.language === 'vbnet' ? decl : getChildByField(decl, 'value'); + // (inside its as_clause), and C# writes `= expr` as the + // declarator's last, unnamed child, so both walk the whole + // declarator. PHP spells its initializer differently and is + // untouched. + const wholeDeclarator = this.language === 'vbnet' || this.language === 'csharp'; + const valueNode = wholeDeclarator ? decl : getChildByField(decl, 'value'); if (valueNode) { this.nodeStack.push(fieldNode.id); + if (this.language === 'csharp') { + this.extractTargetTypedNew( + decl.namedChild(decl.namedChildCount - 1), + varDecl ? getChildByField(varDecl, 'type') : null, + ); + // Its function-as-value candidates are the field's, captured + // here once. (VB.NET captures none; Java's scan still takes + // its initializers for the class as well.) + walked.add(decl.id); + } this.visitFunctionBody(valueNode, fieldNode.id); this.nodeStack.pop(); } @@ -2470,6 +2536,7 @@ export class TreeSitterExtractor { }); } } + return walked; } /** diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 2984a43464..f721a2cc43 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2438,10 +2438,14 @@ function isDotNetTypeRef(ref: UnresolvedRef, context: ResolutionContext): boolea * method (a constructor is one) or an enum case shares the type's name, not * its meaning: AutoMapper's `Type sourceType` bound to an attribute's `Type` * property and `TypeMap typeMap` to a `TypeMap` property beside the `TypeMap` - * class. A field never names a type either. + * class. A field never names a type either, nor does a constant — the kind a + * C# `const` / `static readonly` field gets: jellyfin's `new Version(5, 18)` + * bound to a claim-name `const string Version`, and serilog's `static + * readonly Meter Meter = new(…)` to itself. */ function canNameInTypePosition(n: Node): boolean { - return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field'); + return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field' || + n.kind === 'constant'); } /** From c9944147bbd579869a6f3eee3cbc3533e3167f44 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:32:53 +0000 Subject: [PATCH 206/363] fix(vbnet): resolve receiver calls by declared type; guard .NET names, bare calls and ties (#2351) VB.NET's resolver guessed a receiver call's target from the receiver's name whenever no class matched it, without any of the guards C# has. Indexing Structure members (#2345) gave those guesses new targets: 23 SCrawler `xxxFile.Delete()` calls on PersonalUtilities' SFile went to a nested struct's `TempFileConversion::Delete`, and staxrip's main app sent 90 calls to its AutoCrop tool's duplicate `ColorHSL`. - Declared types (new src/resolution/vbnet-receivers.ts): a receiver's `As` type decides the call (local, parameter, field, property, `For Each`, `As New`, ...). The candidates are that type's member, one it inherits, or an extension method written for it. A type from outside the project gets no link. - .NET standard method names (`Add`, `Contains`, `Clear`, `Dispose`, `ToString`, ...) on an untyped receiver need a receiver named after the owner, as in C#. VB.NET matches them case-insensitively. - A bare call, or one on `Me`, reaches the enclosing class, what it inherits, or a Module. A nested type is matched by its bare name only from inside its owner. Types resolve through enclosing namespaces, `Imports` (aliases included) and the caller's project. A tie between equally good guesses goes to the caller's file, then its project, then the nearer directory, or gets no link. - vbReceiverOf reads the name at or after the reference's column, so `Me.Size = New System.Drawing.Size(...)` is read through `System.Drawing`, not `Me`. Before/after on main (nodes unchanged): SCrawler 15,092 -> 14,893 edges, staxrip 28,002 -> 26,647. Of the non-containment edges removed, 855 / 2,802 are the same link with new resolver metadata. The rest: - .NET names on untyped receivers no longer guessed: 196 / 1,378. 731 of staxrip's are StringBuilder `sb.Append` calls that had gone to `LogBuilder::Append`. - Other receiver guesses dropped: 86 / 103, including the SFile.Delete calls. - Bare calls out of scope: 38 / 834. 590 of staxrip's are `New Point(...)` that had gone to a class nested in `ButtonEx`; others are WinForms `Refresh()`, `Focus()` and `Activate()`. - Retargeted: 227 / 532, e.g. 208 bare `Add(...)` calls in encoder classes now reach the inherited `CommandLineParams::Add`. 121 / 958 call sites resolve that did not before, e.g. `cms.Add` -> ContextMenuStripEx and `td.AddButton` -> TaskDialog. Designer -> Size links fall from 5 / 1 to 0; designer -> Add stays 0. Known trade-off, shared with C#'s guard: about 31 staxrip calls like `switches.Join(BR)` that may be the project's own `MiscExtensions` extension methods are no longer linked. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 + __tests__/vbnet-bare-member-scope.test.ts | 100 ++ __tests__/vbnet-member-access.test.ts | 35 + __tests__/vbnet-project-ties.test.ts | 143 +++ __tests__/vbnet-receiver-types.test.ts | 465 +++++++ __tests__/vbnet-std-methods.test.ts | 75 ++ src/resolution/index.ts | 2 + src/resolution/name-matcher.ts | 81 +- src/resolution/vbnet-receivers.ts | 1335 +++++++++++++++++++++ 9 files changed, 2235 insertions(+), 6 deletions(-) create mode 100644 __tests__/vbnet-bare-member-scope.test.ts create mode 100644 __tests__/vbnet-project-ties.test.ts create mode 100644 __tests__/vbnet-receiver-types.test.ts create mode 100644 __tests__/vbnet-std-methods.test.ts create mode 100644 src/resolution/vbnet-receivers.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index da460b5b77..b5476b128f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,11 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. - In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) +- In VB.NET, a call on a variable, parameter, field or property now reaches a method of the type it is declared with — one that type inherits, or an extension method written for it — and nothing when that type comes from outside your project, instead of any project method that merely shares the name. +- In VB.NET, a call to one of .NET's own methods such as `Add`, `Contains`, `Clear` or `Dispose` on a value whose type isn't known is no longer linked to a project method of that name, unless the value is named after that method's class. +- In VB.NET, a type name is looked up the way VB.NET does it — through the namespaces around it, the file's and project's `Imports` (aliases included), and the caller's own project — so a class declared in several namespaces or projects no longer draws every call to whichever copy was indexed first, and two candidates nothing tells apart get no link at all. +- In VB.NET, a call with no receiver, or on `Me`, now reaches only a member of the class it is written in, of a class it inherits, or of a Module, and no longer a nearby class's member of the same name. +- VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/vbnet-bare-member-scope.test.ts b/__tests__/vbnet-bare-member-scope.test.ts new file mode 100644 index 0000000000..a2601671b1 --- /dev/null +++ b/__tests__/vbnet-bare-member-scope.test.ts @@ -0,0 +1,100 @@ +/** + * A VB.NET call with no receiver — or with `Me` / `MyClass` / `MyBase`, + * which the extractor drops — is a member of the type it is written in, of a + * type around it, or of one they inherit; a `Module`'s members are reached + * from anywhere. Never another class's same-named member, however near its + * file: SCrawler's `{ToString()}` in the Instagram `UserData` (which + * `Inherits UserDataBase`) went to a nearby structure's `ToString`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-bare-')); + const files: Record = { + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Public Overrides Function ToString() As String + Return "user" + End Function + Protected Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/MediaItem.vb': `Namespace API.Instagram + Friend Class MediaItem + Public Overrides Function ToString() As String + Return "media" + End Function + Friend Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + Friend Sub SetTagsLimit() + Dim aStr$ = $"Enter the number of posts from user {ToString()}" + Me.Refresh() + Log(aStr) + End Sub + Private Class Counter + Friend Sub Tick() + Report() + End Sub + End Class + Private Shared Sub Report() + End Sub + End Class +End Namespace +`, + 'Tools/Logger.vb': `Public Module Logger + Public Sub Log(ByVal Text As String) + End Sub +End Module +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(file: string, qualifiedName: string): string[] { + const from = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET receiver-less calls', () => { + it('reach the members the class inherits, not a nearer class’s, and a module’s', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::SetTagsLimit')).toEqual([ + 'API.Base::UserDataBase::Refresh', + 'API.Base::UserDataBase::ToString', + 'Logger::Log', + ]); + }); + + it('reach a member of the class around a nested one', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::Counter::Tick')).toEqual([ + 'API.Instagram::UserData::Report', + ]); + }); +}); diff --git a/__tests__/vbnet-member-access.test.ts b/__tests__/vbnet-member-access.test.ts index d8643ecc52..668cc18aac 100644 --- a/__tests__/vbnet-member-access.test.ts +++ b/__tests__/vbnet-member-access.test.ts @@ -7,6 +7,12 @@ * (`GetService(Of Notifier).Notify()`). SCrawler's designer code sent * `New System.Drawing.Size(…)` to a nested enum's `Size` case 713 times and * `Controls.Add(…)` to a collection class's `Add` 547 times. + * + * The name read is the one the reference starts at (or after) — in + * `Me.Size = New System.Drawing.Size(…)` that is `System.Drawing.Size`, not + * `Me.Size` — and a type is one its written qualifier names: staxrip's + * designer `New System.Drawing.Point(…)` went to a nested `Point` class 550 + * times. A type nested in a class is named bare only inside it. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -56,6 +62,24 @@ End Module Private Sub Refresh() End Sub End Class +`, + 'UI/ButtonEx.vb': `Public Class ButtonEx + Public Class SymbolDrawer + Public Class Point + End Class + Public Sub Draw() + Dim p As New Point() + End Sub + End Class +End Class +`, + 'Forms/AppsForm.vb': `Public Class AppsForm + Private Sub InitializeComponent() + Me.Size = New System.Drawing.Size(315, 205) + Me.ToolStrip.Location = New System.Drawing.Point(473, 10) + Me.lDescription.Location = New Point(4, 285) + End Sub +End Class `, }; for (const [rel, content] of Object.entries(files)) { @@ -84,4 +108,15 @@ describe('VB.NET member access', () => { expect(targets).toContain('Notifier::Notify'); expect(targets).toContain('MainForm::Refresh'); }); + + it('reads the name the reference starts at, and a type its qualifier names', () => { + const targetsOf = (file: string) => cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id)) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates') + .map((e) => `${e.kind} ${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + // `Me.Size = New System.Drawing.Size(…)`, `New System.Drawing.Point(…)`, `New Point(…)` outside ButtonEx. + expect(targetsOf('Forms/AppsForm.vb')).toEqual([]); + expect(targetsOf('UI/ButtonEx.vb')).toEqual(['instantiates ButtonEx::SymbolDrawer::Point']); + }); }); diff --git a/__tests__/vbnet-project-ties.test.ts b/__tests__/vbnet-project-ties.test.ts new file mode 100644 index 0000000000..9d8e2fcc0e --- /dev/null +++ b/__tests__/vbnet-project-ties.test.ts @@ -0,0 +1,143 @@ +/** + * A VB.NET solution often carries the same type in two projects: staxrip's + * main app and its separate AutoCrop tool (its own AutoCrop.vbproj) each + * declare `ColorHSL`, `FrameServerFactory` and `DirectFrameServer`. A call + * in the main app means its own project's copy — about 90 calls such as + * `_backColor.AddLuminance(0.025)` went to the AutoCrop copy, whichever was + * indexed first. Between equally good guesses, the caller's project decides, + * then the nearer directory; when neither does, there is no guess. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const VBPROJ = ` + + WinExe + + +`; + +const COLOR_HSL = `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function +End Class +`; + +const FRAME_SERVER_FACTORY = `Public Class FrameServerFactory + Public Shared Function Create(ByVal path As String) As Object + Return Nothing + End Function +End Class +`; + +const HELPER = `Public Class Helper + Public Sub Run() + End Sub +End Class +`; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-ties-')); + const files: Record = { + 'Source/StaxRip.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/AutoCrop.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/Main.vb': COLOR_HSL + FRAME_SERVER_FACTORY, + 'Source/UI/ColorHSL.vb': COLOR_HSL, + 'Source/Video/FrameServer.vb': FRAME_SERVER_FACTORY, + 'Source/General/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply(ByVal palette As Object) + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + For Each backgroundColor In palette + backgroundColor.AddLuminance(-0.1) + Next + Dim server = FrameServerFactory.Create("video.mkv") + End Sub +End Class +`, + 'Source/UI/TipProvider.vb': `Public Class TipProvider + Public Sub SetTip(ByVal text As String) + End Sub + Public Sub SetTip(ByVal text As String, ByVal title As String) + End Sub +End Class +`, + 'Source/General/Tips.vb': `Public Class Tips + Public Sub Apply(ByVal providers As Object) + For Each tipProvider In providers + tipProvider.SetTip("x") + Next + End Sub +End Class +`, + 'Plugins/A/A.vbproj': VBPROJ, + 'Plugins/A/Helper.vb': HELPER, + 'Plugins/B/B.vbproj': VBPROJ, + 'Plugins/B/Helper.vb': HELPER, + 'App/App.vbproj': VBPROJ, + 'App/Main.vb': `Public Class Main + Public Sub Start(ByVal helpers As Object) + For Each runHelper In helpers + runHelper.Run() + Next + End Sub +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member @ file` of every call / instantiation edge out of a file. */ +function targetsFrom(file: string, kind: 'calls' | 'instantiates'): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === kind) + .map((e) => { + const target = cg.getNode(e.target)!; + return `${target.qualifiedName} @ ${target.filePath}`; + }) + .sort(); +} + +describe('VB.NET duplicate types across projects', () => { + it('resolve a call to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'calls')).toEqual([ + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'FrameServerFactory::Create @ Source/Video/FrameServer.vb', + ]); + }); + + it('resolve a construction to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'instantiates')).toEqual([ + 'ColorHSL @ Source/UI/ColorHSL.vb', + ]); + }); + + it('make no guess between copies nothing tells apart', () => { + expect(targetsFrom('App/Main.vb', 'calls')).toEqual([]); + }); + + it('take one type’s overloads as one guess, not a tie', () => { + expect(targetsFrom('Source/General/Tips.vb', 'calls')).toEqual(['TipProvider::SetTip @ Source/UI/TipProvider.vb']); + }); +}); diff --git a/__tests__/vbnet-receiver-types.test.ts b/__tests__/vbnet-receiver-types.test.ts new file mode 100644 index 0000000000..0a81309747 --- /dev/null +++ b/__tests__/vbnet-receiver-types.test.ts @@ -0,0 +1,465 @@ +/** + * A VB.NET call through a local, a parameter, a field or a property is a call + * on the type it is declared with — `Dim x As T`, `ByVal x As T`, + * `x As New T`, `Private File As SFile`, `Property File As SFile`, a type + * character (`Dim name$`), or what `Dim x = obj.GetString(…)` returns — + * never a guess at a same-named method of some project class. Keywords are + * matched without regard to case. A declared type the project does not define + * means nothing of the project's, except an extension method declared for it. + * + * - SCrawler's `ThumbnailFile.Delete(SFO.File, …)` on an `SFile` (from the + * external PersonalUtilities library) went to a nested `TempFileConversion` + * class's `Delete`, `GroupFile.Delete()` to a download group's, + * `UserUpdatedEventHandlers.Add(e)` on a `List(Of …)` to `UserDataBind.Add` + * and `TotalSize.CompareTo(…)` on a `Double` to a plugin's `VSize.CompareTo`; + * - staxrip's `timestampFontColorValue.ToColor(…)`, a String from + * `settings.GetString(…)`, went to a `ColorHSL.ToColor()` instead of the + * String extension `StringExtensions.ToColor`. + * + * The type a name means is the one VB.NET's lookup finds where it is written: + * the namespaces around it (SCrawler declares a `SiteSettings` and an `M3U8` + * per site namespace), a base class's nested types, the file's `Imports` + * aliases, and the type arguments a subclass gives its base. The declared + * type wins over whatever is assigned later, a `For Each` variable is an + * element of what it loops over, and an interface a class implements lends + * it no members to read bare. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-receiver-')); + const files: Record = { + 'Download/Groups/DownloadGroup.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroup + Friend Sub Delete() + End Sub + End Class +End Namespace +`, + 'YouTube/Objects/YouTubeMediaContainerBase.vb': `Namespace API.YouTube.Objects + Public MustInherit Class YouTubeMediaContainerBase + Public ReadOnly Property ThumbnailFile As SFile + Private Class TempFileConversion + Friend Sub Delete(ByVal Mode As Integer) + End Sub + End Class + End Class +End Namespace +`, + 'Download/Groups/DownloadGroupCollection.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroupCollection + Private ReadOnly GroupFile As SFile = "Settings\\Groups.xml" + Friend Sub Update() + GroupFile.Delete() + End Sub + Friend Sub Reset(byval groupFile as SFile) + groupFile.Delete() + End Sub + End Class +End Namespace +`, + 'Hosts/DownloadableMediaHost.vb': `Namespace Hosts + Friend Class DownloadableMediaHost : Inherits API.YouTube.Objects.YouTubeMediaContainerBase + Friend Sub Clean() + ThumbnailFile.Delete(1) + End Sub + End Class +End Namespace +`, + 'API/UserDataBind.vb': `Namespace API + Friend Class UserDataBind + Friend Sub Add(ByVal User As Object) + End Sub + End Class +End Namespace +`, + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Private ReadOnly UserUpdatedEventHandlers As List(Of UserUpdatedEventHandler) + Friend Sub AddUpdateHandler(ByVal e As UserUpdatedEventHandler) + UserUpdatedEventHandlers.Add(e) + End Sub + Friend Sub Download() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + End Class +End Namespace +`, + 'API/Instagram/Downloader.vb': `Namespace API.Instagram + Friend Class Downloader + Friend Sub Run(ByVal u As UserData) + u.Download() + End Sub + End Class +End Namespace +`, + 'Plugin/UserData.vb': `Friend Class VSize + Public Function CompareTo(ByVal Other As VSize) As Integer + Return 0 + End Function +End Class +`, + 'Editors/UsersInfoForm.vb': `Friend Class UsersInfoForm + Private NotInheritable Class UserOpt + Friend Property TotalSize As Double = 0 + Friend Function CompareTo(ByVal Other As UserOpt) As Integer + Return TotalSize.CompareTo(Other.TotalSize) * -1 + End Function + End Class +End Class +`, + 'UI/ColorHSL.vb': `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function + Public Function ToColor() As Integer + Return 0 + End Function +End Class +`, + 'UI/BackColorAdjuster.vb': `Public Class BackColorAdjuster + Public Function AddLuminance(ByVal offset As Single) As Integer + Return 0 + End Function +End Class +`, + 'UI/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply() + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + End Sub +End Class +`, + 'UI/Popup.vb': `Public Class Popup + Public Sub Show() + End Sub +End Class +`, + 'UI/MainForm.vb': `Public Class MainForm + Inherits Form + + Public Sub OpenSettings() + Dim settingsForm As New MainForm + settingsForm.Show() + End Sub +End Class +`, + 'General/Extensions.vb': `Imports System.Runtime.CompilerServices + +Module StringExtensions + + Function ToColor(ByVal str As String, Optional ByVal defaultColor As Integer = 0) As Integer + Return 0 + End Function + + Function Join(ByVal instance As IEnumerable(Of String), ByVal delimiter As String) As String + Return "" + End Function + + Function Sort(Of T)(ByVal instance As IEnumerable(Of T)) As IEnumerable(Of T) + Return instance + End Function +End Module +`, + 'General/ObjectStorage.vb': `Public Class ObjectStorage + Public Function GetString(ByVal key As String, Optional ByVal defaultValue As String = Nothing) As String + Return defaultValue + End Function +End Class +`, + 'General/Thumbnailer.vb': `Public Class Thumbnailer + Public Sub Run(ByVal settings As ObjectStorage) + Dim timestampFontColorValue = settings.GetString("TimestampFontColor", "#fff") + Dim timestampFontColor = timestampFontColorValue.ToColor(1) + Dim colorText As String = "#000" + Dim outline = colorText.ToColor() + Dim name$ = "x" + Dim named = name.ToColor() + End Sub +End Class +`, + 'General/Lists.vb': `Public Class Lists + Public Sub Run() + Dim names As New List(Of String) + names.Sort() + Dim joined = names.Join(", ") + End Sub +End Class +`, + 'Base/DownDetector.vb': `Namespace API.Base + Friend NotInheritable Class DownDetector + Friend MustInherit Class Checker(Of T) + Protected ReadOnly Property Source As T + End Class + End Class +End Namespace +`, + 'Sites/Bluesky/SiteSettings.vb': `Namespace API.Bluesky + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return False + End Function + Friend Function AvailableTrueValue() As Boolean + Return False + End Function + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return True + End Function + Friend Function AvailableTrueValue() As Boolean + Return True + End Function + Private Class MyDownDetector : Inherits API.Base.DownDetector.Checker(Of SiteSettings) + Friend Sub Check() + Source.AvailableTrueValue() + End Sub + End Class + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/UserData.vb': `Namespace API.Reddit + Friend Class UserData + Private ReadOnly Property MySettings As SiteSettings + Friend Sub Check(ByVal url As String) + MySettings.IsMyUser(url) + M3U8.Download(url) + End Sub + End Class +End Namespace +`, + 'Encoding/VideoEncoder.vb': `Public MustInherit Class VideoEncoder + Public Class MenuList + Public Sub Add(ByVal text As String) + End Sub + End Class +End Class +Public Class BatchEncoder + Inherits VideoEncoder + Public Function GetMenu() As Object + Dim ret As New MenuList + ret.Add("Codec Configuration") + Return ret + End Function +End Class +`, + 'YouTube/YouTubeFunctions.vb': `Friend Interface IContainer + Sub Parse() +End Interface +Friend Class Channel : Implements IContainer + Friend Sub Parse() Implements IContainer.Parse + End Sub +End Class +Friend Module YouTubeFunctions + Friend Sub Load() + Dim item As IContainer + item = New Channel + item.Parse() + End Sub +End Module +`, + 'UI/Labels.vb': `Public Class LabelUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function +End Class +Public Class SimpleUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function + Public Function AddLabel(ByVal text As String, ByVal width As Integer) As Object + Return Nothing + End Function +End Class +Public Class SimpleSettingsForm + Friend WithEvents SimpleUI As SimpleUI +End Class +`, + 'UI/AudioForm.vb': `Public Class AudioForm + Public Sub ShowAdvanced() + Using form As New SimpleSettingsForm() + Dim ui = form.SimpleUI + ui.AddLabel("EBU R128") + End Using + End Sub +End Class +`, + 'UI/Theme.vb': `Public Class ButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ToggleButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ThemeApplier + Public Sub Apply(ByVal controls As Object) + For Each control In controls.OfType(Of ToggleButtonLabel) + control.ApplyTheme() + Next + End Sub + Public Sub ApplyAll(ByVal labels As List(Of ToggleButtonLabel)) + For Each label In labels + label.ApplyTheme() + Next + End Sub +End Class +`, + 'Download/TDownloader.vb': `Namespace App.Download + Friend Class TDownloader + Friend Class Job + Friend Sub Start() + End Sub + End Class + End Class + Friend Class Worker + Friend Sub Start() + End Sub + End Class +End Namespace +`, + 'Download/DownloadProgress.vb': `Imports TDJob = App.Download.TDownloader.Job + +Namespace App.Download + Friend Class DownloadProgress + Friend ReadOnly Property Worker As TDJob + Friend Sub Run() + Worker.Start() + End Sub + End Class +End Namespace +`, + 'Plugin/IPluginContentProvider.vb': `Namespace Plugin + Public Interface ISiteSettings + End Interface + Public Interface IPluginContentProvider + Property Settings As ISiteSettings + End Interface +End Namespace +`, + 'MainMod.vb': `Friend Module MainMod + Friend Settings As SettingsCLS +End Module +`, + 'SettingsCLS.vb': `Friend Class SettingsCLS + Friend Sub UpdateUsersList() + End Sub +End Class +`, + 'API/Base/UserDataProvider.vb': `Namespace API.Base + Friend MustInherit Class UserDataProvider : Implements Plugin.IPluginContentProvider + Friend Property MySettings As Plugin.ISiteSettings Implements Plugin.IPluginContentProvider.Settings + Friend Sub Delete() + Settings.UpdateUsersList() + End Sub + End Class +End Namespace +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of a file. */ +function callsFrom(file: string): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET calls through declared receivers', () => { + it('reach nothing of the project on a type it does not define', () => { + expect(callsFrom('Download/Groups/DownloadGroupCollection.vb')).toEqual([]); + expect(callsFrom('Hosts/DownloadableMediaHost.vb')).toEqual([]); + expect(callsFrom('API/Base/UserDataBase.vb')).toEqual([]); + }); + + it('reach nothing of the project on a VB.NET built-in type', () => { + expect(callsFrom('Editors/UsersInfoForm.vb')).toEqual([]); + }); + + it('reach the declared type’s own method, or the one it inherits', () => { + expect(callsFrom('UI/ThemeManager.vb')).toEqual(['ColorHSL::AddLuminance']); + expect(callsFrom('API/Instagram/Downloader.vb')).toEqual(['API.Base::UserDataBase::Download']); + }); + + it('reach nothing of the project for a method the declared type inherits from outside it', () => { + expect(callsFrom('UI/MainForm.vb')).toEqual([]); + }); + + it('reach an extension method declared for the type, through what a call returns too', () => { + expect(callsFrom('General/Thumbnailer.vb')).toEqual([ + 'ObjectStorage::GetString', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + ]); + }); + + it('reach a .NET type’s own method before an extension of that name, and an extension it lacks', () => { + // `List(Of T)` has its own `Sort()`, but no `Join`. + expect(callsFrom('General/Lists.vb')).toEqual(['StringExtensions::Join']); + }); + + it('name the type their declaration’s namespace sees, through a base class’s type argument too', () => { + // SCrawler declares a `SiteSettings` and an `M3U8` in each site's namespace. + expect(callsFrom('Sites/Reddit/UserData.vb')).toEqual([ + 'API.Reddit::M3U8::Download', + 'API.Reddit::SiteSettings::IsMyUser', + ]); + // `Inherits DownDetector.Checker(Of SiteSettings)` makes its `Source As T` a SiteSettings. + expect(callsFrom('Sites/Reddit/SiteSettings.vb')).toEqual(['API.Reddit::SiteSettings::AvailableTrueValue']); + // A type nested in a base class is named bare in a subclass: staxrip's `Dim ret As New MenuList`. + expect(callsFrom('Encoding/VideoEncoder.vb')).toEqual(['VideoEncoder::MenuList::Add']); + }); + + it('take the declared type over what is assigned, and an import alias for what it names', () => { + expect(callsFrom('YouTube/YouTubeFunctions.vb')).toEqual(['IContainer::Parse']); + expect(callsFrom('Download/DownloadProgress.vb')).toEqual(['App.Download::TDownloader::Job::Start']); + }); + + it('read a field through what a call returns, and a loop variable through what it loops over', () => { + expect(callsFrom('UI/AudioForm.vb')).toEqual(['SimpleUI::AddLabel']); + // `For Each control In controls.OfType(Of ToggleButtonLabel)`, `For Each label In labels` over a `List(Of ToggleButtonLabel)`. + expect(callsFrom('UI/Theme.vb')).toEqual(['ToggleButtonLabel::ApplyTheme', 'ToggleButtonLabel::ApplyTheme']); + }); + + it('see a module’s field, not the property of an interface the class implements', () => { + expect(callsFrom('API/Base/UserDataProvider.vb')).toEqual(['SettingsCLS::UpdateUsersList']); + }); +}); diff --git a/__tests__/vbnet-std-methods.test.ts b/__tests__/vbnet-std-methods.test.ts new file mode 100644 index 0000000000..8c29b1e775 --- /dev/null +++ b/__tests__/vbnet-std-methods.test.ts @@ -0,0 +1,75 @@ +/** + * VB.NET runs on the same .NET base library as C#, so a call such as + * `x.Contains(…)`, `x.Add(…)` or `x.Dispose()` on a receiver whose type is not + * known is far more likely the library's method than the one project method + * that happens to share the name. As in C#, such a guess is kept only when the + * receiver is named after the method's owner. staxrip's + * `SupportedInput.Contains(ret)` — a String array — went to `VideoScript`'s + * `Contains`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-std-')); + const files: Record = { + 'Video/VideoScript.vb': `Public Class VideoScript + Public Function Contains(ByVal value As String) As Boolean + Return False + End Function +End Class +`, + 'Audio/AudioProfile.vb': `Public Class AudioProfile + Public Function IsInputSupported(ByVal inputs As Object, ByVal ext As String) As Boolean + For Each supported In inputs + If supported.Contains(ext) Then Return True + Next + Return False + End Function + + Public Function HasScript(ByVal scripts As Object, ByVal name As String) As Boolean + For Each script In scripts + If script.Contains(name) Then Return True + Next + Return False + End Function +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(qualifiedName: string): string[] { + const from = cg.getNodesInFile('Audio/AudioProfile.vb').find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET standard-library method names', () => { + it('are not a project method on a receiver that does not name its owner', () => { + expect(callsOf('AudioProfile::IsInputSupported')).toEqual([]); + }); + + it('still reach the project method when the receiver names its owner', () => { + expect(callsOf('AudioProfile::HasScript')).toEqual(['VideoScript::Contains']); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 1eaf14b145..4c9fb45f2c 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,6 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; +import { clearVbnetReceiverMemos } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -457,6 +458,7 @@ export class ReferenceResolver { clearNameMatcherMemos(this.context); clearCppMacroVisibility(this.context); clearSwiftTypeVisibility(this.context); + clearVbnetReceiverMemos(this.context); clearTypeParameterMemos(this.context); } } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index f721a2cc43..0ba9c946d4 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -11,6 +11,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSuperty import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; +import { breakVbTie, isVbMemberInScope, isVbNestedTypeInScope, isVbTypeQualifiedBy, matchVbTypedCall, preferVbProject, sameVbProject } from './vbnet-receivers'; import { isTestPath } from '../search/query-utils'; import { isMinifiedContent } from '../extraction/generated-detection'; import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; @@ -3112,6 +3113,9 @@ const CSHARP_STD_METHODS: ReadonlySet = new Set([ 'ForEach', 'Sort', 'Reverse', 'Clone', 'Seek', 'SetLength', ]); +/** The same .NET names as VB.NET writes them — in any case. */ +const VBNET_STD_METHODS: ReadonlySet = new Set([...CSHARP_STD_METHODS].map((m) => m.toLowerCase())); + /** * A request handler the web framework dispatches to: a Django / DRF / Flask * view's `get` / `post` / …, a controller's `index` / `store` / `update` / @@ -3156,11 +3160,17 @@ function stdMethodNames(language: string): ReadonlySet | null { case 'rust': return RUST_STD_METHODS; case 'kotlin': return KOTLIN_STD_METHODS; case 'csharp': return CSHARP_STD_METHODS; + case 'vbnet': return VBNET_STD_METHODS; case 'dart': return DART_STD_METHODS; default: return null; } } +/** Whether `name` is one of `language`'s standard-library method names (VB.NET's in any case). */ +function isStdMethodName(language: string, name: string): boolean { + return stdMethodNames(language)?.has(language === 'vbnet' ? name.toLowerCase() : name) ?? false; +} + /** Methods of Dart's String, List, Iterable, Map and Set — names a project type rarely carries itself. */ const DART_STD_METHODS: ReadonlySet = new Set([ 'endsWith', 'startsWith', 'contains', 'split', 'substring', 'trim', 'trimLeft', 'trimRight', 'toLowerCase', @@ -4657,7 +4667,13 @@ function vbReceiverOf(ref: UnresolvedRef, context: ResolutionContext): string | const name = ref.referenceName.toLowerCase(); let start = lower.startsWith(name, ref.column) ? ref.column : -1; if (start < 0) { - const m = new RegExp(`(?>>(); /** @@ -5536,6 +5568,8 @@ export function matchByExactName( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -5557,6 +5591,9 @@ export function matchByExactName( !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context)) && @@ -7817,6 +7854,14 @@ export function matchMethodCall( // shared source-based inferrer. resolveMethodOnType validates the method // exists on the inferred type, so a mis-inference produces no edge. if (inferableReceiver) { + // A VB.NET receiver's declared type decides the call, or that there is no + // project method to call: SCrawler's `ThumbnailFile.Delete(…)` on an + // external `SFile` went to a nested class's `Delete` by a shared word. + if (ref.language === 'vbnet' && dotMatch) { + const typed = nmTimedT('mc-vbtyped', ref, () => + matchVbTypedCall(objectOrClass!, methodName!, ref, context, (name) => isStdMethodName('vbnet', name))); + if (typed !== undefined) return typed; + } let inferredType = nmTimedT('mc-infer', ref, () => ref.language === 'cpp' ? inferCppReceiverType(objectOrClass!, ref, context) @@ -8024,6 +8069,9 @@ export function matchMethodCall( const visible = classCandidates.filter((c) => c.language !== 'csharp' || isCsharpTypeVisible(c, typeRef, context)); classCandidates = [...visible, ...classCandidates.filter((c) => !visible.includes(c))]; } + // A VB.NET type declared in two projects is the caller's own project's: + // staxrip's `FrameServerFactory.Create(…)` went to its AutoCrop tool's copy. + if (ref.language === 'vbnet') classCandidates = preferVbProject(classCandidates, ref, context); for (const classNode of classCandidates) { // Skip cross-language class matches @@ -8213,10 +8261,10 @@ export function matchMethodCall( // `json` of its `MockedResponse`. !isUnnamedTestDouble(targetMethods[0]!, objectOrClass!, ref, context) && !((ref.language === 'lua' || ref.language === 'luau') && isLuaLibraryCall(objectOrClass!, methodName!, ref, targetMethods[0]!)) && - // Rust / Go / Kotlin / C#: a standard-library method name on an - // untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, + // Rust / Go / Kotlin / C# / VB.NET: a standard-library method name on + // an untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, // `reader.Value.ToString()`) is the library type's. - !(stdMethodNames(ref.language)?.has(methodName!) && + !(isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!) && !receiverNamesOwner(receiverLink(objectOrClass!), targetMethods[0]!, context)) && !(UNTYPED_RECEIVER_LANGUAGES.has(ref.language) && !/^(?:self|self\.class|this|super|weak_?self|strong_?self)$/i.test(objectOrClass!) && !sharesReceiverWord(objectOrClass!, targetMethods[0]!) && @@ -8238,11 +8286,12 @@ export function matchMethodCall( const head = receiverWords[receiverWords.length - 1]?.toLowerCase(); let bestMatch: typeof targetMethods[0] | undefined; let bestScore = 0; + let tied: typeof targetMethods = []; // Same-file candidates first, so a score tie (`score > bestScore` keeps // the first seen) resolves to the call site's own file rather than the // first-indexed duplicate (#1079). - const std = stdMethodNames(ref.language)?.has(methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); + const std = isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); for (const method of preferCallSiteFile(targetMethods, ref.filePath)) { if (std && !receiverNamesOwner(receiverLink(objectOrClass!), method, context)) continue; // The owner type's own name — not its namespace (`eShop.ClientApp…` @@ -8265,8 +8314,16 @@ export function matchMethodCall( if (score > bestScore) { bestScore = score; bestMatch = method; + tied = [method]; + } else if (score === bestScore) { + tied.push(method); } } + // VB.NET: between equally good guesses, the caller's own file, then its + // project, then the nearer directory — and no guess when none of them + // decides. staxrip's main app and its AutoCrop tool each declare a + // `ColorHSL`, and the first indexed took about 90 of the app's calls. + if (ref.language === 'vbnet' && tied.length > 1 && bestScore >= 2) bestMatch = breakVbTie(tied, ref, context) ?? undefined; // A wrapper handing its call on — BookStack's `FileStorage::delete` doing // `$storage->delete($path)`, `CommentRepo::delete` doing @@ -9229,7 +9286,7 @@ function computePathProximity(filePath1: string, filePath2: string): number { function findBestMatch( ref: UnresolvedRef, candidates: Node[], - _context: ResolutionContext + context: ResolutionContext ): Node | null { // Prioritization rules: // 1. Same file > different file @@ -9269,6 +9326,13 @@ function findBestMatch( // Directory proximity bonus — strongly prefer same module/package score += pathProximityFromDirs(refDirs, candidate.filePath); + // A VB.NET project compiles its own files: the caller's project weighs as + // much as the nearest a directory can be. staxrip's `New ColorHSL(…)` went + // to its AutoCrop tool's copy. + if (ref.language === 'vbnet' && candidate.language === 'vbnet' && sameVbProject(candidate.filePath, ref.filePath, context)) { + score += 80; + } + // Language matching: strongly prefer same language, penalize cross-language if (candidate.language === ref.language) { score += 50; @@ -9353,6 +9417,8 @@ export function matchFuzzy( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -9388,6 +9454,9 @@ export function matchFuzzy( !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(objcShape === 'c-call' && OBJC_MEMBER_KINDS.has(n.kind)) && !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts new file mode 100644 index 0000000000..07732927e1 --- /dev/null +++ b/src/resolution/vbnet-receivers.ts @@ -0,0 +1,1335 @@ +/** + * VB.NET receivers, projects and receiver-less calls. + * + * The name matcher guesses a `receiver.Method()` call it cannot type by the + * method's name alone (matchMethodCall's Strategy 3: a unique name, or the + * receiver sharing words with the owner), and a receiver-less call by the + * nearest same-named member. VB.NET brought none of the evidence C# does to + * those guesses, so a call landed on whichever project method shared its + * name: SCrawler's `ThumbnailFile.Delete(…)` on an external `SFile` went to a + * nested `TempFileConversion.Delete`, staxrip's main app called its AutoCrop + * tool's copy of `ColorHSL`. What this module reads, from the source: + * + * 1. A receiver's declared type — `Dim x As T`, `ByVal x As T`, `x As New T`, + * a type character (`Dim name$`), `Dim x = New T(…)`, a `For Each` over a + * typed collection, a field or property of the class, of one it inherits + * (with the type arguments the subclass gives) or of a `Module`, or what + * `Dim x = obj.GetString(…)` returns. A typed receiver's call is the + * type's own method or one it inherits, else an extension method declared + * for the type, else nothing of the project's: a type the project does + * not define (`SFile`, `String`, `List(Of T)`) has none of its methods. + * 2. Which type a name means where it is written: the namespaces around it, + * outward, then its file's and project's `Imports` (aliases included), + * then the project a file belongs to — the `.vbproj` above it, whose + * `RootNamespace` its namespaces are inside. Between same-named types, and + * between guesses nothing else tells apart, the caller's own project's + * comes first; a written qualifier (`System.Drawing.Point`) must match. + * 3. A receiver-less call (or one on `Me` / `MyClass` / `MyBase`, which the + * extractor drops) is a member of a type around it, of what those inherit, + * or of a `Module`; a type nested in a class is named bare only inside it. + * + * VB.NET's keywords and names are matched without regard to case. Nothing + * here runs for another language. + */ +import * as fs from 'fs'; +import * as path from 'path'; +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; + +/** A VB.NET declared type: its simple name (no namespace, no type arguments) and whether it is an array. */ +interface VbType { + name: string; + array: boolean; + /** The namespaces or types written before the name (`API.Base` in `API.Base.UserDataBase`), lowercased. */ + qualifier?: string[]; + /** Where the name is written, which decides the namespaces it is looked up in. */ + file?: string; + line?: number; + /** A supertype named in an `Implements` statement: its members are not the type's own. */ + implemented?: boolean; + /** The type arguments written with it: `SiteSettings` in `Checker(Of SiteSettings)`. */ + args?: VbType[]; +} + +/** What a statement says a local is: its type, the call whose result it holds, or a binding that names no type. */ +type VbBinding = + | { kind: 'type'; type: VbType } + | { kind: 'call'; receiver: string | null; member: string } + | { kind: 'each'; collection: string } + | { kind: 'unknown' }; + +/** .NET's collections of one element type: `For Each x In list` over a `List(Of T)` binds a `T`. */ +const VB_ELEMENT_COLLECTIONS = /^(?:List|IList|IEnumerable|ICollection|IReadOnlyList|IReadOnlyCollection|HashSet|SortedSet|Queue|Stack|LinkedList|ObservableCollection|Collection|ReadOnlyCollection|BindingList|ConcurrentBag|ConcurrentQueue|ConcurrentStack|BlockingCollection)$/i; + +/** The type of what a `For Each` over a value of type `t` binds: an array's or a .NET collection's element type. */ +function elementType(t: VbType): VbType | null { + if (t.array) return { ...t, array: false }; + return t.args?.length === 1 && VB_ELEMENT_COLLECTIONS.test(t.name) && t.args[0]!.name !== '?' ? t.args[0]! : null; +} + +/** VB.NET's built-in types, by keyword and by .NET name, keyed to the keyword. */ +const VB_BUILTIN_TYPES: ReadonlyMap = new Map([ + ...['boolean', 'byte', 'char', 'date', 'decimal', 'double', 'integer', 'long', 'object', 'sbyte', 'short', 'single', + 'string', 'uinteger', 'ulong', 'ushort'].map((k): [string, string] => [k, k]), + ['int16', 'short'], ['int32', 'integer'], ['int64', 'long'], ['uint16', 'ushort'], ['uint32', 'uinteger'], + ['uint64', 'ulong'], ['datetime', 'date'], +]); + +/** The type a type character declares: `Dim aStr$`, `For i% = 0 …`. */ +const VB_TYPE_CHARS: Readonly> = { + $: 'String', '%': 'Integer', '&': 'Long', '!': 'Single', '#': 'Double', '@': 'Decimal', +}; + +/** VB.NET's conversion functions and the built-in type each returns. */ +const VB_CONVERSIONS: Readonly> = { + cbool: 'Boolean', cbyte: 'Byte', cchar: 'Char', cdate: 'Date', cdbl: 'Double', cdec: 'Decimal', cint: 'Integer', + clng: 'Long', cobj: 'Object', csbyte: 'SByte', cshort: 'Short', csng: 'Single', cstr: 'String', cuint: 'UInteger', + culng: 'ULong', cushort: 'UShort', +}; + +/** VB.NET type nodes a member can belong to (a `Module` is indexed as a class). */ +const VB_TYPE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface']); +const VB_VALUE_KINDS: ReadonlySet = new Set(['field', 'property', 'constant', 'variable']); +const VB_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field', 'enum_member', 'constant', 'variable']); + +/** A declaration keyword right before a name: the name is a member being declared, not a variable. */ +const VB_MEMBER_HEAD = /\b(?:Function|Sub|Property|Event|Operator|Declare|Delegate|Class|Structure|Module|Interface|Enum|Namespace)\s+$/i; + +function escapeRegex(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** A type's key for comparing two of them: the built-in keyword or the lowercased name, `[]` for an array. */ +function typeKey(t: VbType): string { + const lower = t.name.toLowerCase(); + return `${VB_BUILTIN_TYPES.get(lower) ?? lower}${t.array ? '[]' : ''}`; +} + +function isBuiltin(t: VbType): boolean { + return !t.array && VB_BUILTIN_TYPES.has(t.name.toLowerCase()); +} + +/** The index of the `)` closing the `(` at `open`, or -1. */ +function closeParen(text: string, open: number): number { + let depth = 0; + for (let i = open; i < text.length; i++) { + if (text[i] === '(') depth++; + else if (text[i] === ')' && --depth === 0) return i; + } + return -1; +} + +/** The top-level items of the parenthesized list opening at `open`, or null when it does not close. */ +function splitArgs(text: string, open: number): string[] | null { + const close = closeParen(text, open); + if (close < 0) return null; + const args: string[] = []; + let depth = 0; + let cur = ''; + for (const ch of text.slice(open + 1, close)) { + if (ch === '(' || ch === '{') depth++; + else if (ch === ')' || ch === '}') depth--; + if (ch === ',' && depth === 0) { + args.push(cur); + cur = ''; + } else cur += ch; + } + args.push(cur); + return args.map((a) => a.trim()); +} + +/** + * The type written at `text[at]` — `List(Of Foo)`, `String()`, `Integer?`, + * `Global.A.B` — by its last name; null when no type starts there. + */ +function readType(text: string, at: number): VbType | null { + const head = /^\s*([A-Za-z_][\w.]*)/.exec(text.slice(at)); + if (!head) return null; + let i = at + head[0].length; + let args: VbType[] | undefined; + if (/^\s*\(\s*Of\b/i.test(text.slice(i))) { + const open = text.indexOf('(', i); + const close = closeParen(text, open); + if (close < 0) return null; + args = (splitArgs(text, open) ?? []).map((a) => readType(a.replace(/^Of\s+/i, ''), 0) ?? { name: '?', array: false }); + i = close + 1; + } + const segments = head[1]!.split('.'); + const name = segments.pop()!; + if (!/^[A-Za-z_]\w*$/.test(name) || /^(?:New|Of|As|In|Out|From|With)$/i.test(name)) return null; + const qualifier = segments.map((s) => s.toLowerCase()).filter((s, i) => !(i === 0 && s === 'global')); + return { + name, + array: /^\s*\??\s*\(\s*,*\s*\)/.test(text.slice(i)), + ...(qualifier.length > 0 ? { qualifier } : {}), + ...(args ? { args } : {}), + }; +} + +/** `t` — and the type arguments written inside it — written at `file:line`. */ +function sited(t: VbType | null, file: string, line: number): VbType | null { + return t ? { ...t, file, line, ...(t.args ? { args: t.args.map((a) => sited(a, file, line)!) } : {}) } : null; +} + +/** A VB.NET line's code: its comment dropped and every string literal emptied to `""`. */ +function vbCode(line: string): string { + // Most lines have neither: the line itself, without a copy. + if (!/["'‘’\r]/.test(line)) return line; + let out = ''; + for (let i = 0; i < line.length; i++) { + const ch = line[i]!; + if (ch === '"') { + let j = i + 1; + for (; j < line.length; j++) { + if (line[j] !== '"') continue; + if (line[j + 1] === '"') j++; + else break; + } + out += '""'; + i = j; + continue; + } + if (ch === "'" || ch === '\u2018' || ch === '\u2019' || ch === '\r') break; + out += ch; + } + return out; +} + +const VB_CODE_LINES = new WeakMap>(); + +/** A file's lines as code (see vbCode), read once per file. */ +function codeLines(file: string, context: ResolutionContext): string[] { + let memo = VB_CODE_LINES.get(context); + if (!memo) VB_CODE_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const lines = (context.getFileLines?.(file) ?? context.readFile(file)?.split(/\r?\n/) ?? []).map(vbCode); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, lines); + return lines; +} + +interface VbPatterns { + /** The name standing where a binding names it — not as `name.Member` nor `name(args)`, which only use it. */ + declared: RegExp; + /** `name As T`, `name() As T`, `name As New T`: a variable, parameter, loop or catch variable. */ + asClause: RegExp; + /** `Dim name, other As T`: a declarator sharing the next one's type. */ + listed: RegExp; + /** `Dim name$`, `ByVal name%`, `For i% = …`. */ + typeChar: RegExp; + /** `Dim name = …`: the value's type. */ + inferred: RegExp; + /** `For Each name In …`: the collection's element type, when it says it. */ + forEach: RegExp; + /** A binding that names no type: `For name = …`, `Function(name)`, `From name In`, `Catch name`. */ + loose: RegExp; +} + +const VB_PATTERNS = new Map(); + +function patternsFor(name: string): VbPatterns { + const key = name.toLowerCase(); + const hit = VB_PATTERNS.get(key); + if (hit) return hit; + const r = escapeRegex(key); + const patterns: VbPatterns = { + declared: new RegExp(`(?= 4096) VB_PATTERNS.delete(VB_PATTERNS.keys().next().value!); + VB_PATTERNS.set(key, patterns); + return patterns; +} + +/** + * What every binding of a name has on its line: a declaring keyword, or a + * type character. Most lines naming a variable just use it (`sb.Append(…)`). + */ +const VB_MAY_BIND = /\b(?:As|Dim|Static|Const|For|Each|Function|Sub|Catch|Using|From|Let|Aggregate)\b|\w[$%&!#@]/i; + +/** What one line of code says `name` is, or undefined when it doesn't bind the name. */ +function bindingOn(code: string, name: string): VbBinding | undefined { + if (!VB_MAY_BIND.test(code)) return undefined; + const p = patternsFor(name); + if (!p.declared.test(code)) return undefined; + p.asClause.lastIndex = 0; + for (let m = p.asClause.exec(code); m; m = p.asClause.exec(code)) { + if (VB_MEMBER_HEAD.test(code.slice(0, m.index))) continue; + const type = readType(code, m.index + m[0].length); + if (!type) return { kind: 'unknown' }; + // `As New T(…)`: the parentheses are the constructor's. + return { kind: 'type', type: m[2] ? { ...type, array: false } : { ...type, array: type.array || !!m[1] } }; + } + const listed = p.listed.exec(code); + if (listed) { + const type = readType(code, listed.index + listed[0].length); + return type ? { kind: 'type', type } : { kind: 'unknown' }; + } + const typeChar = p.typeChar.exec(code); + if (typeChar) return { kind: 'type', type: { name: VB_TYPE_CHARS[typeChar[1]!]!, array: !!typeChar[2] } }; + // An assignment (`x = New Channel`) says nothing: the declared type + // (`Dim x As IYouTubeMediaContainer`) is what the call binds to. + const inferred = p.inferred.exec(code); + if (inferred) return valueBinding(code.slice(inferred.index + inferred[0].length)); + // `For Each c In controls.OfType(Of ButtonLabel)`: the loop's elements are + // that type; `For Each user In users`: the elements of what `users` is. + const loop = p.forEach.exec(code); + if (loop) { + const collection = loop[1]!.trim(); + const elements = /\.\s*(?:OfType|Cast)\s*\(\s*Of\s+/i.exec(collection); + const type = elements && /^[^()]*\)\s*(?:\(\s*\))?\s*$/.test(collection.slice(elements.index + elements[0].length)) + ? readType(collection, elements.index + elements[0].length) : null; + if (type) return { kind: 'type', type }; + return /^[A-Za-z_]\w*$/.test(collection) ? { kind: 'each', collection } : { kind: 'unknown' }; + } + return p.loose.test(code) ? { kind: 'unknown' } : undefined; +} + +/** What a local initialized with `expr` (`Dim x = expr`) is. */ +function valueBinding(expr: string): VbBinding { + const e = expr.split(/\s:(?!=)/)[0]!.trim(); + const unknown: VbBinding = { kind: 'unknown' }; + const created = /^New\s+/i.exec(e); + if (created) { + const type = readType(e, created[0].length); + return type ? { kind: 'type', type: { ...type, array: false } } : unknown; + } + const cast = /^(?:DirectCast|TryCast|CType)\s*\(/i.exec(e); + if (cast) { + const args = splitArgs(e, cast[0].length - 1); + const type = args?.length === 2 ? readType(args[1]!, 0) : null; + return type ? { kind: 'type', type } : unknown; + } + if (/^\$?""/.test(e)) return { kind: 'type', type: { name: 'String', array: false } }; + const conversion = /^(C[A-Za-z]+)\s*\(/.exec(e); + const converted = conversion ? VB_CONVERSIONS[conversion[1]!.toLowerCase()] : undefined; + if (converted) return { kind: 'type', type: { name: converted, array: false } }; + // `obj.Member(…)`, `Member(…)`, `Me.Member`: a call or a read as the whole value. + const call = /^(?:([A-Za-z_]\w*)\s*\.\s*)?([A-Za-z_]\w*)\s*/.exec(e); + if (call) { + let rest = e.slice(call[0].length); + if (rest.startsWith('(')) { + const close = closeParen(rest, 0); + rest = close < 0 ? 'unclosed' : rest.slice(close + 1).trim(); + } + if (rest === '') return { kind: 'call', receiver: call[1] ?? null, member: call[2]! }; + } + return unknown; +} + +/** The 0-based line a call's own declarations start on: its member's first line, which holds a method's parameters. */ +function scopeStartLine(ref: UnresolvedRef, context: ResolutionContext): number { + const from = context.getNodeById?.(ref.fromNodeId); + if (from && from.filePath === ref.filePath && from.startLine <= ref.line && from.endLine >= ref.line && + (from.kind === 'method' || from.kind === 'function' || from.kind === 'property' || from.kind === 'field')) { + return from.startLine - 1; + } + let start = -1; + for (const n of context.getNodesInFile(ref.filePath)) { + if (n.kind !== 'method' && n.kind !== 'function' && n.kind !== 'property') continue; + if (n.startLine <= ref.line && n.endLine >= ref.line && n.startLine - 1 > start) start = n.startLine - 1; + } + return start < 0 ? ref.line - 1 : start; +} + +const VB_WORD_LINES = new WeakMap>>(); + +/** + * The 0-based lines of a file's code each lowercased word appears on, in + * order: a call's receiver is looked for only on the lines that name it, + * not on every line back to its method's start (a long method's calls each + * rescanned it). + */ +function wordLines(file: string, context: ResolutionContext): Map { + let memo = VB_WORD_LINES.get(context); + if (!memo) VB_WORD_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const index = new Map(); + const words = /[A-Za-z_]\w*/g; + const lines = codeLines(file, context); + for (let i = 0; i < lines.length; i++) { + const code = lines[i]!.toLowerCase(); + words.lastIndex = 0; + for (let m = words.exec(code); m; m = words.exec(code)) { + let at = index.get(m[0]); + if (!at) index.set(m[0], (at = [])); + if (at[at.length - 1] !== i) at.push(i); + } + } + if (memo.size >= 256) memo.delete(memo.keys().next().value!); + memo.set(file, index); + return index; +} + +const VB_BINDINGS = new WeakMap>>(); + +/** + * Every statement of a file that binds `name`, by 0-based line — read once: + * staxrip's `GetArgs` methods call `sb.Append(…)` on hundreds of lines, each + * of which would otherwise walk back over all the others to the `Dim sb`. + */ +function bindingsOf(name: string, file: string, context: ResolutionContext): Array<{ line: number; binding: VbBinding }> { + let memo = VB_BINDINGS.get(context); + if (!memo) VB_BINDINGS.set(context, (memo = new Map())); + const key = `${file}|${name.toLowerCase()}`; + const hit = memo.get(key); + if (hit) return hit; + const lines = codeLines(file, context); + const found: Array<{ line: number; binding: VbBinding }> = []; + for (const i of wordLines(file, context).get(name.toLowerCase()) ?? []) { + const binding = bindingOn(lines[i]!, name); + if (binding) found.push({ line: i, binding }); + } + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(key, found); + return found; +} + +/** The nearest statement above a call (in its member) that binds `name`, and its 1-based line. */ +function localBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): { binding: VbBinding; line: number } | undefined { + const found = bindingsOf(name, ref.filePath, context); + // The last binding at or above the call's line. + let lo = 0; + let hi = found.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (found[mid]!.line <= ref.line - 1) lo = mid + 1; + else hi = mid; + } + const nearest = found[lo - 1]; + return nearest && nearest.line >= scopeStartLine(ref, context) ? { binding: nearest.binding, line: nearest.line + 1 } : undefined; +} + +const VB_FILE_TYPES = new WeakMap>(); + +/** The VB.NET types a line is written inside, innermost first. */ +function typesAround(file: string, line: number, context: ResolutionContext): Node[] { + let memo = VB_FILE_TYPES.get(context); + if (!memo) VB_FILE_TYPES.set(context, (memo = new Map())); + let types = memo.get(file); + if (!types) { + types = context.getNodesInFile(file) + .filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)) + .sort((a, b) => b.startLine - a.startLine); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, types); + } + return types.filter((n) => n.startLine <= line && n.endLine >= line); +} + +/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +} + +const VB_MEMBERS = new WeakMap>(); + +/** Members named `name` declared directly in `type` (any partial part of it), case aside. */ +function membersNamed(type: Node, name: string, context: ResolutionContext): Node[] { + let memo = VB_MEMBERS.get(context); + if (!memo) VB_MEMBERS.set(context, (memo = new Map())); + const qn = `${type.qualifiedName}::${name}`.toLowerCase(); + const hit = memo.get(qn); + if (hit) return hit; + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && n.qualifiedName.toLowerCase() === qn); + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(qn, members); + return members; +} + +const VB_MODULES = new WeakMap>(); + +/** Whether a VB.NET owner (by qualified name) is a `Module`, whose members are reached without a qualifier. */ +function isModule(ownerQn: string, context: ResolutionContext): boolean { + let memo = VB_MODULES.get(context); + if (!memo) VB_MODULES.set(context, (memo = new Map())); + const hit = memo.get(ownerQn); + if (hit !== undefined) return hit; + // Its own head line or the two after it (attributes may come first); the raw + // lines, as the whole file need not be read as code for this. + const module = context.getNodesByQualifiedName(ownerQn).some((n) => n.language === 'vbnet' && n.kind === 'class' && + (context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []).slice(n.startLine - 1, n.startLine + 2) + .some((l) => /^\s*(?:<[^>]*>\s*)*(?:(?:Public|Friend|Private|Partial)\s+)*Module\s/i.test(l))); + memo.set(ownerQn, module); + return module; +} + +/** The members named `name` of the project's Modules, which code reaches without a qualifier — the caller's project's first. */ +function moduleMembersNamed(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => { + const cut = n.qualifiedName.lastIndexOf('::'); + return n.language === 'vbnet' && cut > 0 && isModule(n.qualifiedName.slice(0, cut), context); + }); + return preferVbProject(members, ref, context); +} + +const VB_SUPERS = new WeakMap>(); + +/** + * The types a VB.NET type's declarations (every `Partial` part) inherit or + * implement: the `Inherits` / `Implements` statements that open its body, + * including the `Class X : Inherits Y` form. + */ +function supertypesOf(type: Node, context: ResolutionContext): VbType[] { + let memo = VB_SUPERS.get(context); + if (!memo) VB_SUPERS.set(context, (memo = new Map())); + const key = type.qualifiedName.toLowerCase(); + const hit = memo.get(key); + if (hit) return hit; + const supers: VbType[] = []; + const parts = context.getNodesByQualifiedName(type.qualifiedName).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); + for (const part of parts.length > 0 ? parts : [type]) { + const lines = codeLines(part.filePath, context); + scan: for (let i = part.startLine - 1; i < Math.min(lines.length, part.startLine + 12, part.endLine); i++) { + for (const statement of lines[i]!.split(/:(?!=)/)) { + const s = statement.trim(); + // Blank, an attribute, a directive, or the declaration's own head. + if (s === '' || /^<.*>$/.test(s) || s.startsWith('#') || /\b(?:Class|Structure|Interface|Module)\s+[A-Za-z_]/i.test(s) && !/^(?:Inherits|Implements)\b/i.test(s)) continue; + const clause = /^(Inherits|Implements)\s+(.+)$/i.exec(s); + if (!clause) break scan; + for (const sup of splitArgs(`(${clause[2]})`, 0) ?? []) { + const t = sited(readType(sup, 0), part.filePath, part.startLine); + if (t) supers.push(/^Implements$/i.test(clause[1]!) ? { ...t, implemented: true } : t); + } + } + } + } + memo.set(key, supers); + return supers; +} + +/** A type a VB.NET type inherits, with what its type parameters stand for there (`T` → `SiteSettings`). */ +interface VbAncestor { + node: Node; + args: Map; +} + +const VB_ANCESTRIES = new WeakMap>(); + +/** `t` with a type parameter replaced by what `args` says it stands for. */ +function substitute(t: VbType, args: ReadonlyMap): VbType { + return (!t.qualifier && !t.array && args.get(t.name.toLowerCase())) || t; +} + +/** + * A VB.NET type and the project types it inherits, nearest first: a class's + * base classes, an interface's base interfaces — whose members are the + * type's own. Each carries the type arguments `Inherits Checker(Of + * SiteSettings)` gives its parameters. An interface a class implements lends + * it no members: they are reached through the class's own `… Implements + * IFoo.Bar` ones. + */ +function ancestry(type: Node, context: ResolutionContext): VbAncestor[] { + let memo = VB_ANCESTRIES.get(context); + if (!memo) VB_ANCESTRIES.set(context, (memo = new Map())); + const hit = memo.get(type.id); + if (hit) return hit; + const out: VbAncestor[] = []; + const seen = new Set(); + const queue: VbAncestor[] = [{ node: type, args: new Map() }]; + while (queue.length > 0 && out.length < 16) { + const current = queue.shift()!; + const key = current.node.qualifiedName.toLowerCase(); + if (seen.has(key)) continue; + seen.add(key); + out.push(current); + for (const sup of supertypesOf(current.node, context)) { + if (sup.implemented) continue; + const found = typesNamedAt(sup, context); + if (found.ambiguous) continue; + const given = (sup.args ?? []).map((a) => substitute(a, current.args)); + for (const base of found.owners) { + const params = [...typeParametersOf(base, context).keys()]; + queue.push({ node: base, args: new Map(params.flatMap((p, i): Array<[string, VbType]> => (given[i] ? [[p, given[i]!]] : []))) }); + } + } + } + memo.set(type.id, out); + return out; +} + +/** A VB.NET type and the project types it inherits, nearest first (see ancestry). */ +function hierarchy(type: Node, context: ResolutionContext): Node[] { + return ancestry(type, context).map((a) => a.node); +} + +const VB_PROJECTS = new WeakMap>(); + +/** The directory of the nearest `.vbproj` at or above `dir` (`''` for the root), or null outside any project. */ +function projectOfDir(dir: string, context: ResolutionContext, memo: Map): string | null { + const hit = memo.get(dir); + if (hit !== undefined) return hit; + let found: string | null = null; + try { + if (fs.readdirSync(path.join(context.getProjectRoot(), dir)).some((e) => /\.vbproj$/i.test(e))) found = dir; + } catch { + // Unreadable: no project file here. + } + if (found === null && dir !== '') { + const cut = dir.lastIndexOf('/'); + found = projectOfDir(cut < 0 ? '' : dir.slice(0, cut), context, memo); + } + memo.set(dir, found); + return found; +} + +/** The project (the directory of the nearest `.vbproj` above it) a file belongs to, or null. */ +function projectOf(file: string, context: ResolutionContext): string | null { + let memo = VB_PROJECTS.get(context); + if (!memo) VB_PROJECTS.set(context, (memo = new Map())); + const cut = file.lastIndexOf('/'); + return projectOfDir(cut < 0 ? '' : file.slice(0, cut), context, memo); +} + +/** Whether two files are in the same VB.NET project — false when either is in none. */ +export function sameVbProject(a: string, b: string, context: ResolutionContext): boolean { + const project = projectOf(a, context); + return project !== null && project === projectOf(b, context); +} + +interface VbProjectInfo { + /** The root namespace every file of the project declares its namespaces inside, as segments. */ + root: string[]; + /** The namespaces the project imports into every file (``). */ + imports: string[]; +} + +const VB_PROJECT_INFO = new WeakMap>(); + +/** A file's project's root namespace (``, else the project's name) and project-wide imports, lowercased. */ +function projectInfo(file: string, context: ResolutionContext): VbProjectInfo { + const dir = projectOf(file, context); + if (dir === null) return { root: [], imports: [] }; + let memo = VB_PROJECT_INFO.get(context); + if (!memo) VB_PROJECT_INFO.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + const info: VbProjectInfo = { root: [], imports: [] }; + try { + const abs = path.join(context.getProjectRoot(), dir); + const project = fs.readdirSync(abs).find((e) => /\.vbproj$/i.test(e)); + if (project) { + const text = fs.readFileSync(path.join(abs, project), 'utf8'); + const root = /\s*([\w.]*)\s*<\/RootNamespace>/i.exec(text)?.[1] ?? project.replace(/\.vbproj$/i, ''); + info.root = root.toLowerCase().split('.').filter((s) => s !== ''); + for (const m of text.matchAll(/ s.split('.'))]; +} + +/** A file's lines before its first declaration: where its `Option` and `Imports` statements are. */ +function headerLines(file: string, context: ResolutionContext): string[] { + const lines = codeLines(file, context); + const end = lines.findIndex((l) => + /^\s*(?:Namespace|Module|Class|Structure|Interface|Enum|Delegate|Public|Friend|Private|Protected|Partial|NotInheritable|MustInherit)\b/i.test(l)); + return end < 0 ? lines : lines.slice(0, end); +} + +const VB_FILE_IMPORTS = new WeakMap>(); + +/** The namespaces a file imports — its own `Imports` and its project's — lowercased. */ +function importedNamespaces(file: string, context: ResolutionContext): string[] { + let memo = VB_FILE_IMPORTS.get(context); + if (!memo) VB_FILE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const imports = [...projectInfo(file, context).imports]; + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) imports.push(m[1]!.toLowerCase().replace(/^global\./, '')); + } + memo.set(file, imports); + return imports; +} + +const VB_ALIASES = new WeakMap>>(); + +/** A file's import aliases: `Imports TDJob = SCrawler.DownloadObjects.TDownloader.Job`, by lowercased alias. */ +function importAliases(file: string, context: ResolutionContext): Map { + let memo = VB_ALIASES.get(context); + if (!memo) VB_ALIASES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const aliases = new Map(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([A-Za-z_]\w*)\s*=\s*(.+)$/i.exec(line); + const target = m ? readType(m[2]!, 0) : null; + if (target) aliases.set(m![1]!.toLowerCase(), target); + } + memo.set(file, aliases); + return aliases; +} + +/** `t` with an import alias it is written with — as itself or as its first qualifier — spelled out. */ +function unalias(t: VbType, file: string, context: ResolutionContext): VbType { + const aliases = importAliases(file, context); + if (aliases.size === 0) return t; + if (!t.qualifier) { + const target = aliases.get(t.name.toLowerCase()); + return target ? { ...t, name: target.name, ...(target.qualifier ? { qualifier: target.qualifier } : {}) } : t; + } + const target = aliases.get(t.qualifier[0]!); + return target ? { ...t, qualifier: [...(target.qualifier ?? []), target.name.toLowerCase(), ...t.qualifier.slice(1)] } : t; +} + +/** Whether `tail` is the end of `full`. */ +function endsWith(full: string[], tail: string[]): boolean { + return tail.length <= full.length && tail.every((s, i) => full[full.length - tail.length + i] === s); +} + +const VB_TYPE_LIKE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'type_alias']); + +/** The class, structure or interface (not a `Module`) a VB.NET type is nested in, or null. */ +function enclosingType(n: Node, context: ResolutionContext): Node | null { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return null; + const parentQn = n.qualifiedName.slice(0, cut); + const parent = context.getNodesByQualifiedName(parentQn).find((p) => p.language === 'vbnet' && VB_TYPE_KINDS.has(p.kind)); + return parent && !isModule(parentQn, context) ? parent : null; +} + +function isNestedInType(n: Node, context: ResolutionContext): boolean { + return enclosingType(n, context) !== null; +} + +const VB_BASE_NAMES = new WeakMap>>(); + +/** + * The lowercased names of the types around a line and of the classes they + * inherit, walked by name (a few levels): the types whose nested types code + * there names bare — `Dim ret As New MenuList` in a `VideoEncoder` subclass. + */ +function baseTypeNamesAround(file: string, line: number, context: ResolutionContext): Set { + const around = typesAround(file, line, context); + let memo = VB_BASE_NAMES.get(context); + if (!memo) VB_BASE_NAMES.set(context, (memo = new Map())); + const key = around.map((t) => t.id).join('|'); + const hit = memo.get(key); + if (hit) return hit; + const names = new Set(); + let frontier = around; + for (let depth = 0; depth < 5 && frontier.length > 0; depth++) { + const next: Node[] = []; + for (const t of frontier) { + if (names.has(t.name.toLowerCase())) continue; + names.add(t.name.toLowerCase()); + for (const sup of supertypesOf(t, context)) if (!sup.implemented) next.push(...projectTypesNamed(sup.name, context)); + } + frontier = next; + } + memo.set(key, names); + return names; +} + +/** + * Whether a type is one a name written with this qualifier can mean: its + * namespaces and outer types end with what is written before the name. + * Designer code's `New System.Drawing.Point(3, 3)` is not staxrip's nested + * `ButtonEx.SymbolDrawer.Point`; `New API.Base.UserDataBase(…)` is that one. + */ +export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const written = readType(`${qualifier}.${n.name}`, 0); + if (!written?.qualifier) return true; + const t = unalias(written, file, context); + return !t.qualifier || endsWith(fullSegments(n, context).slice(0, -1), t.qualifier); +} + +/** + * The project types a type name written at a site means, as VB.NET looks it + * up: one its written qualifier names, nested in or declared in the namespace + * of a type around the site — the nearest first — else nested in a class + * those inherit, else in a namespace the file or project imports, else in + * the site's own project (a class's nested type only where it is in scope). + * SCrawler declares + * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, + * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is + * Pinterest's. `ambiguous` when what is left are different types. + */ +function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { + const t = written.file ? unalias(written, written.file, context) : written; + let candidates = projectTypesNamed(t.name, context); + // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. + if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); + if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; + const file = t.file; + const site = typesAround(file, t.line ?? 0, context)[0]; + const sitePath = site ? fullSegments(site, context) : projectInfo(file, context).root; + let tier: Node[] = []; + let best = -1; + for (const c of candidates) { + const container = fullSegments(c, context).slice(0, -1); + if (container.length > sitePath.length || container.some((s, i) => s !== sitePath[i])) continue; + if (container.length > best) { + best = container.length; + tier = [c]; + } else if (container.length === best) tier.push(c); + } + // A type nested in a class the site's types inherit. + if (tier.length === 0 && !t.qualifier) { + const bases = baseTypeNamesAround(file, t.line ?? 0, context); + tier = candidates.filter((c) => bases.has(enclosingType(c, context)?.name.toLowerCase() ?? '')); + } + if (tier.length === 0) { + const imports = importedNamespaces(file, context); + tier = candidates.filter((c) => imports.includes(fullSegments(c, context).slice(0, -1).join('.'))); + } + // Else any the project declares in a namespace (the root namespaces and + // project imports this cannot see) — but not a class's nested type, which + // is named bare only inside it or a class deriving from it. + if (tier.length === 0) tier = t.qualifier ? candidates : candidates.filter((c) => !isNestedInType(c, context)); + // A project's own declaration over another project's of the same name + // (staxrip's main app and its AutoCrop tool each declare a `ColorHSL`). + const own = tier.filter((c) => sameVbProject(c.filePath, file, context)); + if (own.length > 0) tier = own; + // One type's partial parts are one type. + const distinct = new Set(tier.map((c) => `${projectOf(c.filePath, context)}|${c.qualifiedName.toLowerCase()}`)); + return { owners: [...tier.filter((c) => c.filePath === file), ...tier.filter((c) => c.filePath !== file)], ambiguous: distinct.size > 1 }; +} + +/** The call site's own file first, then its project's, the rest after, each in its given order. */ +export function preferVbProject(nodes: Node[], ref: UnresolvedRef, context: ResolutionContext): Node[] { + if (nodes.length < 2) return nodes; + const file: Node[] = []; + const project: Node[] = []; + const rest: Node[] = []; + for (const n of nodes) { + if (n.filePath === ref.filePath) file.push(n); + else if (sameVbProject(n.filePath, ref.filePath, context)) project.push(n); + else rest.push(n); + } + return [...file, ...project, ...rest]; +} + +/** How many leading directories two files share. */ +function sharedDirs(a: string, b: string): number { + const da = a.split('/').slice(0, -1); + const db = b.split('/').slice(0, -1); + let i = 0; + while (i < da.length && i < db.length && da[i] === db[i]) i++; + return i; +} + +/** + * The one of several equally good guesses a call means: the one in its own + * file, else in its own project, else in the nearest directory — or null + * when none of these tells them apart. + */ +export function breakVbTie(tied: Node[], ref: UnresolvedRef, context: ResolutionContext): Node | null { + // One type's overloads (and partial parts) are one guess, not a tie: the first stands for them. + const seen = new Set(); + const distinct = tied.filter((n) => { + const key = `${projectOf(n.filePath, context)}|${n.qualifiedName.toLowerCase()}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); + if (distinct.length === 1) return distinct[0]!; + let pool = distinct.filter((n) => n.filePath === ref.filePath); + if (pool.length === 0) { + const own = distinct.filter((n) => sameVbProject(n.filePath, ref.filePath, context)); + pool = own.length > 0 ? own : distinct; + } + if (pool.length === 1) return pool[0]!; + let best = -1; + let winners: Node[] = []; + for (const n of pool) { + const shared = sharedDirs(ref.filePath, n.filePath); + if (shared > best) { + best = shared; + winners = [n]; + } else if (shared === best) winners.push(n); + } + return winners.length === 1 ? winners[0]! : null; +} + +/** The type a field or property declares itself with, read from its declaration. */ +function memberDeclaredType(member: Node, context: ResolutionContext): VbType | null { + const code = codeLines(member.filePath, context).slice(member.startLine - 1, member.startLine + 2).join(' '); + const r = escapeRegex(member.name); + const property = new RegExp(`\\bProperty\\s+${r}\\b\\s*`, 'i').exec(code); + if (property) { + let at = property.index + property[0].length; + if (code[at] === '(') { + const close = closeParen(code, at); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+(New\s+)?/i.exec(code.slice(at)); + const type = as ? readType(code, at + as[0].length) : null; + return type && as?.[1] ? { ...type, array: false } : type; + } + const binding = bindingOn(code, member.name); + if (binding?.kind === 'type') return binding.type; + // `Private x = New Foo()`: a field's initializer is what it holds. + const init = new RegExp(`(? = new Map()): VbType | null { + const type = sited(member.kind === 'method' ? declaredReturnType(member, context) : memberDeclaredType(member, context), + member.filePath, member.startLine); + const given = type ? substitute(type, args) : null; + return given !== type ? given : resolveTypeParameter(type, member, context); +} + +/** + * The type of what a receiver-less `name` reads where a call is: a member of + * a class around the call or of one it inherits, else of a `Module`. Null when + * it declares no type the source shows, undefined when nothing has the name. + */ +function memberType(name: string, ref: UnresolvedRef, context: ResolutionContext): VbType | null | undefined { + const fits = (n: Node) => VB_VALUE_KINDS.has(n.kind) || n.kind === 'method'; + for (const around of typesAround(ref.filePath, ref.line, context)) { + for (const { node, args } of ancestry(around, context)) { + const member = membersNamed(node, name, context).find(fits); + if (member) return valueMemberType(member, context, args); + } + } + const global = moduleMembersNamed(name, ref, context).find(fits); + return global ? valueMemberType(global, context) : undefined; +} + +/** The type a method, function or property returns, read from its declaration; null for a `Sub` or none written. */ +function declaredReturnType(n: Node, context: ResolutionContext): VbType | null { + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 6).join(' '); + const head = new RegExp(`\\b(Function|Property|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!head || /^Sub$/i.test(head[1]!)) return null; + let at = head.index + head[0].length; + // `(Of T)`, then the parameter list. + for (let group = 0; group < 2; group++) { + const open = /^\s*\(/.exec(code.slice(at)); + if (!open) break; + const close = closeParen(code, at + open[0].length - 1); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+/i.exec(code.slice(at)); + return as ? readType(code, at + as[0].length) : null; +} + +const VB_TYPE_PARAMS = new WeakMap>>(); + +/** The type parameters a declaration's head names (`Class Repo(Of T As Entity)`), each with its constraint or null. */ +function typeParametersOf(n: Node, context: ResolutionContext): Map { + let memo = VB_TYPE_PARAMS.get(context); + if (!memo) VB_TYPE_PARAMS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit) return hit; + const params = readTypeParameters(n, context); + memo.set(n.id, params); + return params; +} + +function readTypeParameters(n: Node, context: ResolutionContext): Map { + const params = new Map(); + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 2).join(' '); + const head = new RegExp(`\\b(?:Class|Structure|Interface|Module|Function|Sub)\\s+${escapeRegex(n.name)}\\s*(?=\\(\\s*Of\\b)`, 'i').exec(code); + if (!head) return params; + for (const p of splitArgs(code, head.index + head[0].length) ?? []) { + const m = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)(?:\s+As\s+(.+))?$/i.exec(p); + if (!m) continue; + const constraints = (m[2] ?? '').replace(/^\{|\}$/g, '').split(',').map((c) => c.trim()) + .filter((c) => c !== '' && !/^(?:New|Class|Structure)$/i.test(c)); + params.set(m[1]!.toLowerCase(), constraints.length > 0 ? sited(readType(constraints[0]!, 0), n.filePath, n.startLine) : null); + } + return params; +} + +/** + * What a type means where it is written: a type parameter of the method + * (`owner`) or of a type around it stands for its constraint — null when it + * has none, as its value's type is then unknown — and any other name for itself. + */ +function resolveTypeParameter(t: VbType | null, owner: Node | null, context: ResolutionContext): VbType | null { + if (!t || t.array || isBuiltin(t) || t.qualifier || !t.file) return t; + const key = t.name.toLowerCase(); + for (const decl of [...(owner && owner.kind === 'method' ? [owner] : []), ...typesAround(t.file, t.line ?? 0, context)]) { + const params = typeParametersOf(decl, context); + if (params.has(key)) return params.get(key) ?? null; + } + return t; +} + +/** + * The project types `t` names where it is written, null when the name leaves + * different types (a guess between them would be no better than none), or + * none for an array or a built-in type. + */ +function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { + if (t.array || isBuiltin(t)) return []; + const found = typesNamedAt(t, context); + return found.ambiguous ? null : found.owners; +} + +/** + * A method — or, `withValues`, a field or property — named `name` on one of + * `owners` (the type `typed` names) or a project type they inherit, with what + * the type parameters of the type declaring it stand for. + */ +function memberOn( + owners: Node[], + name: string, + ref: UnresolvedRef, + context: ResolutionContext, + withValues: boolean, + typed?: VbType, +): { node: Node; args: Map } | null { + const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); + for (const level of [0, 1]) { + for (const owner of preferVbProject(owners, ref, context)) { + // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). + const given = new Map([...typeParametersOf(owner, context).keys()].flatMap((p, i): Array<[string, VbType]> => + (typed?.args?.[i] ? [[p, typed.args[i]!]] : []))); + const ancestors = ancestry(owner, context); + for (const { node, args } of level === 0 ? ancestors.slice(0, 1) : ancestors.slice(1)) { + const found = preferVbProject(membersNamed(node, name, context).filter(fits), ref, context)[0]; + if (found) return { node: found, args: level === 0 ? given : new Map([...args].map(([p, t]) => [p, substitute(t, given)])) }; + } + } + } + return null; +} + +/** The type of the value `Member(…)` / `obj.Member(…)` gives, from the member's declaration. */ +function callResultType( + call: { receiver: string | null; member: string }, + ref: UnresolvedRef, + context: ResolutionContext, + depth: number, +): VbType | null { + let member: { node: Node; args: Map } | null = null; + if (call.receiver === null || /^(?:Me|MyClass|MyBase)$/i.test(call.receiver)) { + const owners = typesAround(ref.filePath, ref.line, context).slice(0, 1); + member = owners.length > 0 ? memberOn(owners, call.member, ref, context, true) : null; + if (!member && call.receiver === null) { + const global = moduleMembersNamed(call.member, ref, context).find((n) => n.kind === 'method' || VB_VALUE_KINDS.has(n.kind)); + member = global ? { node: global, args: new Map() } : null; + } + } else { + const type = receiverType(call.receiver, ref, context, depth + 1); + // Not a variable: a type or module named for a shared call (`ObjectStorage.Load()`). + const owners = type === undefined ? ownersOf({ name: call.receiver, array: false, file: ref.filePath, line: ref.line }, context) + : type ? ownersOf(type, context) : null; + member = owners && owners.length > 0 ? memberOn(owners, call.member, ref, context, true, type ?? undefined) : null; + } + return member ? valueMemberType(member.node, context, member.args) : null; +} + +/** + * What a receiver named `name` is declared as where the call is: a local or + * parameter of its member, else a field or property of its class, of one the + * class inherits, or of a `Module`. Null when the receiver is bound but its + * type isn't known; undefined when nothing binds the name (a type or module). + */ +function receiverType(name: string, ref: UnresolvedRef, context: ResolutionContext, depth: number): VbType | null | undefined { + if (depth > 2) return null; + const local = localBinding(name, ref, context); + if (local) { + const { binding, line } = local; + const owner = context.getNodeById?.(ref.fromNodeId) ?? null; + if (binding.kind === 'type') return resolveTypeParameter(sited(binding.type, ref.filePath, line), owner, context); + if (binding.kind === 'call') return callResultType(binding, { ...ref, line }, context, depth); + if (binding.kind === 'each') { + const collection = receiverType(binding.collection, { ...ref, line }, context, depth + 1); + return collection ? elementType(collection) : null; + } + return null; + } + return memberType(name, ref, context); +} + +interface VbExtension { + node: Node; + /** The type it extends; null when that is one of its own type parameters (it extends anything). */ + param: VbType | null; +} + +const VB_EXTENSIONS = new WeakMap>(); + +/** A method's extension-method declaration (` Function F(s As String, …)`), or null when it isn't one. */ +function extensionOf(n: Node, context: ResolutionContext): VbExtension | null { + if (n.kind !== 'method' || n.language !== 'vbnet') return null; + let memo = VB_EXTENSIONS.get(context); + if (!memo) VB_EXTENSIONS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit !== undefined) return hit; + const extension = readExtension(n, context); + memo.set(n.id, extension); + return extension; +} + +function readExtension(n: Node, context: ResolutionContext): VbExtension | null { + const code = codeLines(n.filePath, context).slice(Math.max(0, n.startLine - 2), n.startLine + 6).join(' '); + const decl = new RegExp(`\\b(?:Function|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!decl) return null; + const attributes = /((?:<[^<>]*>\s*)+)(?:(?:Public|Friend|Private|Protected|Shared|Overloads|Async|Iterator)\s+)*$/i + .exec(code.slice(0, decl.index))?.[1]; + if (!attributes || !/\bExtension(?:Attribute)?\b/i.test(attributes)) return null; + let at = decl.index + decl[0].length; + const typeParams = new Set(); + if (/^\s*\(\s*Of\b/i.test(code.slice(at))) { + const open = code.indexOf('(', at); + for (const p of splitArgs(code, open) ?? []) { + const tp = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)/i.exec(p); + if (tp) typeParams.add(tp[1]!.toLowerCase()); + } + const close = closeParen(code, open); + if (close < 0) return null; + at = close + 1; + } + const open = /^\s*\(/.exec(code.slice(at)); + const first = open ? splitArgs(code, at + open[0].length - 1)?.[0]?.replace(/<[^<>]*>/g, '') : undefined; + const as = first ? /\bAs\s+/i.exec(first) : null; + const type = as ? readType(first!, as.index + as[0].length) : null; + if (!type) return null; + return { node: n, param: typeParams.has(type.name.toLowerCase()) && !type.array ? null : type }; +} + +/** `Object`'s instance methods, which every type has. */ +const OBJECT_METHODS = ['tostring', 'equals', 'gethashcode', 'gettype']; + +/** + * The instance methods of .NET's everyday types: a call one of them answers + * is the type's own, whatever extension of that name the project declares + * (`list.Sort()` on a `List(Of T)`), and a name one of them lacks is no + * instance method there (`list.Join(", ")` is an extension's). + */ +const BCL_INSTANCE_METHODS: ReadonlyMap> = new Map(Object.entries({ + string: ['clone', 'compareto', 'contains', 'copyto', 'endswith', 'getenumerator', 'gettypecode', 'indexof', 'indexofany', + 'insert', 'isnormalized', 'lastindexof', 'lastindexofany', 'normalize', 'padleft', 'padright', 'remove', 'replace', 'split', + 'startswith', 'substring', 'tochararray', 'tolower', 'tolowerinvariant', 'toupper', 'toupperinvariant', 'trim', 'trimend', + 'trimstart'], + stringbuilder: ['append', 'appendformat', 'appendjoin', 'appendline', 'clear', 'copyto', 'ensurecapacity', 'getchunks', 'insert', + 'remove', 'replace'], + list: ['add', 'addrange', 'asreadonly', 'binarysearch', 'clear', 'contains', 'convertall', 'copyto', 'exists', 'find', 'findall', + 'findindex', 'findlast', 'findlastindex', 'foreach', 'getenumerator', 'getrange', 'indexof', 'insert', 'insertrange', + 'lastindexof', 'remove', 'removeall', 'removeat', 'removerange', 'reverse', 'sort', 'toarray', 'trimexcess', 'trueforall'], + dictionary: ['add', 'clear', 'containskey', 'containsvalue', 'ensurecapacity', 'getenumerator', 'remove', 'trimexcess', 'tryadd', + 'trygetvalue'], + hashset: ['add', 'clear', 'contains', 'copyto', 'exceptwith', 'getenumerator', 'intersectwith', 'ispropersubsetof', + 'ispropersupersetof', 'issubsetof', 'issupersetof', 'overlaps', 'remove', 'removewhere', 'setequals', 'symmetricexceptwith', + 'trimexcess', 'trygetvalue', 'unionwith'], + '[]': ['clone', 'copyto', 'getenumerator', 'getlength', 'getlonglength', 'getlowerbound', 'getupperbound', 'getvalue', + 'initialize', 'setvalue'], +}).map(([type, methods]) => [type, new Set([...methods, ...OBJECT_METHODS])])); + +/** The instance methods of a .NET type the table above knows; undefined for any other. */ +function bclInstanceMethods(t: VbType): ReadonlySet | undefined { + return BCL_INSTANCE_METHODS.get(t.array ? '[]' : typeKey(t)); +} + +/** + * Whether a value of type `t` might reach an extension declared for `param` + * through a conversion nothing here can check: `Object`, an array's + * collection interfaces, an interface a built-in type implements, or any + * type an outside base type may lead to. + */ +function mayExtend(t: VbType, open: boolean, param: VbType): boolean { + if (typeKey(param) === 'object') return true; + if (param.array !== t.array) { + return t.array && /^(?:IEnumerable|IList|ICollection|IReadOnlyList|IReadOnlyCollection|Array)$/i.test(param.name); + } + if (isBuiltin(param)) return false; + if (isBuiltin(t)) return /^I[A-Z]/.test(param.name); + return open; +} + +/** + * The extension method a call on a value of type `t` reaches: one declared + * for `t` or for a type it inherits — else, when exactly one other extension + * of that name could apply and the name is no instance method of `t`'s + * (one of .NET's own names, for a type nothing here describes), that one. + */ +function extensionFor( + t: VbType, + owners: Node[], + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null { + const instanceMethods = owners.length === 0 ? bclInstanceMethods(t) : undefined; + if (instanceMethods?.has(method.toLowerCase())) return null; + const extensions = context.getNodesByLowerName(method.toLowerCase()) + .map((n) => extensionOf(n, context)) + .filter((e): e is VbExtension => e !== null); + if (extensions.length === 0) return null; + // Every name the type goes by — its own, its base types', the interfaces + // they implement — and whether one of them is an outside type, whose own + // ancestry nothing here shows. + const names = new Set([typeKey({ name: t.name, array: false })]); + let open = owners.length === 0 && !isBuiltin(t) && !t.array; + for (const owner of owners) { + for (const type of hierarchy(owner, context)) { + names.add(type.name.toLowerCase()); + for (const sup of supertypesOf(type, context)) { + names.add(typeKey(sup)); + const found = typesNamedAt(sup, context).owners; + if (found.length === 0) open = true; + else if (sup.implemented) for (const i of found) for (const h of hierarchy(i, context)) names.add(h.name.toLowerCase()); + } + } + } + const exact = extensions.filter((e) => e.param !== null && + (typeKey(e.param) === typeKey(t) || (!t.array && !e.param.array && names.has(typeKey(e.param))))); + if (exact.length > 0) { + const target = preferVbProject(exact.map((e) => e.node), ref, context)[0]!; + return { original: ref, targetNodeId: target.id, confidence: 0.85, resolvedBy: 'instance-method' }; + } + const loose = extensions.filter((e) => e.param === null || mayExtend(t, open, e.param)); + if (loose.length === 1 && (instanceMethods !== undefined || !isStdMethod(method))) { + return { original: ref, targetNodeId: loose[0]!.node.id, confidence: 0.7, resolvedBy: 'instance-method' }; + } + return null; +} + +/** + * Resolve `receiver.method()` through the receiver's declared type: the + * type's own method or one it inherits, else an extension method for it, + * else null — a typed receiver is never a guess. A receiver naming one of the + * project's types or modules is a shared call on it. Undefined when the type + * isn't known, or is `Object` (a late-bound call), for the name strategies. + */ +export function matchVbTypedCall( + receiver: string, + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null | undefined { + if (!/^[A-Za-z_]\w*$/.test(receiver)) return undefined; + const type = receiverType(receiver, ref, context, 0); + if (type === undefined) { + // A type or module named for a shared call — `M3U8.Download(…)` in SCrawler's + // `API.Reddit.UserData` is `API.Reddit.M3U8`'s, not another site's: its own + // member or one it inherits. + const named = typesNamedAt({ name: receiver, array: false, file: ref.filePath, line: ref.line }, context); + if (named.owners.length === 0) return undefined; + if (named.ambiguous) return null; + const shared = memberOn(named.owners, method, ref, context, false); + return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + } + if (!type || typeKey(type) === 'object') return undefined; + const owners = ownersOf(type, context); + // A type name that leaves two of the project's types: no guess between them. + if (owners === null) return null; + if (owners.length > 0) { + const own = memberOn(owners, method, ref, context, false, type); + if (own) return { original: ref, targetNodeId: own.node.id, confidence: 0.9, resolvedBy: 'instance-method' }; + } + return extensionFor(type, owners, method, ref, context, isStdMethod); +} + +const VB_TYPE_IMPORTS = new WeakMap>>(); + +/** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ +function importedNames(file: string, context: ResolutionContext): Set { + let memo = VB_TYPE_IMPORTS.get(context); + if (!memo) VB_TYPE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const names = new Set(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) names.add(m[1]!.split('.').pop()!.toLowerCase()); + } + memo.set(file, names); + return names; +} + +const VB_SCOPES = new WeakMap | null>>(); + +/** The lowercased names of the types around a call and of the project types they inherit; null outside any type. */ +function scopeOwners(ref: UnresolvedRef, context: ResolutionContext): Set | null { + let memo = VB_SCOPES.get(context); + if (!memo) VB_SCOPES.set(context, (memo = new WeakMap())); + const hit = memo.get(ref); + if (hit !== undefined) return hit; + const around = typesAround(ref.filePath, ref.line, context); + const owners = around.length === 0 ? null + : new Set(around.flatMap((t) => hierarchy(t, context)).map((t) => t.name.toLowerCase())); + memo.set(ref, owners); + return owners; +} + +/** + * Whether an unqualified VB.NET type name can mean `n`: a type nested in a + * class, structure or interface is named bare only inside it or a type + * deriving from it — designer code's `New Point(4, 285)` is not staxrip's + * nested `ButtonEx.SymbolDrawer.Point`. A `Module`'s types belong to its + * namespace; a file can import the outer type, or alias the nested one + * (`Imports UserMediaD = SCrawler.DownloadObjects.TDownloader.UserMediaD`). + * Not judged when no type is around the name. + */ +export function isVbNestedTypeInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const parent = enclosingType(n, context); + if (!parent) return true; + const owners = scopeOwners(ref, context); + if (owners === null || owners.has(parent.name.toLowerCase())) return true; + const alias = importAliases(ref.filePath, context).get(ref.referenceName.toLowerCase()); + if (alias && alias.name.toLowerCase() === n.name.toLowerCase() && + (!alias.qualifier || endsWith(fullSegments(n, context).slice(0, -1), alias.qualifier))) return true; + return importedNames(ref.filePath, context).has(parent.name.toLowerCase()); +} + +/** + * Whether a receiver-less VB.NET call (or one through `Me` / `MyClass` / + * `MyBase`) can mean member `n`: one of a type around the call or of a type + * those inherit, of a `Module`, or of a type the file imports. Not judged + * when no type is around the call, or `n` is not a member of a type. + */ +export function isVbMemberInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_MEMBER_KINDS.has(n.kind)) return true; + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return true; + const owners = scopeOwners(ref, context); + if (owners === null) return true; + const ownerQn = n.qualifiedName.slice(0, cut); + const owner = ownerQn.split(/::|\./).pop()!.toLowerCase(); + return owners.has(owner) || isModule(ownerQn, context) || importedNames(ref.filePath, context).has(owner); +} + +/** + * Drop this module's per-context memos — file lines and what was read from + * them — with the resolver's own caches (ReferenceResolver.clearCaches), so a + * sync never reads a changed file's old declarations. + */ +export function clearVbnetReceiverMemos(context: ResolutionContext): void { + for (const memo of [VB_CODE_LINES, VB_WORD_LINES, VB_BINDINGS, VB_FILE_TYPES, VB_MEMBERS, VB_MODULES, VB_SUPERS, VB_ANCESTRIES, + VB_PROJECTS, VB_PROJECT_INFO, VB_FILE_IMPORTS, VB_ALIASES, VB_BASE_NAMES, VB_TYPE_PARAMS, VB_EXTENSIONS, VB_TYPE_IMPORTS, + VB_SCOPES] as Array>) { + memo.delete(context); + } +} From 26e8488aedd632ff8102e2626808d71435a602a2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:37:39 +0000 Subject: [PATCH 207/363] fix(windows): the kernel-parity sweep loads the engine as file:// URLs (#2352) scripts/kernel-parity.mjs handed import() a path from path.join(). On Windows that is `C:\...`, which Node's ESM loader rejects with ERR_UNSUPPORTED_ESM_URL_SCHEME, so the sweep could not run there at all and contributors kept patched scratch copies. The three engine imports now go through pathToFileURL(...).href, as the agent-eval probe scripts already do. On macOS and Linux an ordinary path resolves to the same file:// URL as before. Checked on Windows 11 after npm ci, npm run build and scripts/build-kernel.sh: the unmodified script threw at its first import; with the fix, the C# fixtures (3/3), src/search (4/4), all of src/ as TypeScript (261/263, 2 deferred to wasm) and every fixture language (42/42) run to the summary line and exit 0. Co-authored-by: Claude Opus 5.5 --- scripts/kernel-parity.mjs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/kernel-parity.mjs b/scripts/kernel-parity.mjs index 5209369d8c..98c347eefd 100644 --- a/scripts/kernel-parity.mjs +++ b/scripts/kernel-parity.mjs @@ -25,7 +25,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); const dist = (p) => path.join(ROOT, 'dist', p); @@ -106,9 +106,11 @@ if (files.length === 0) { } // --- load the built engine --------------------------------------------------- -const { extractFromSource } = await import(dist('extraction/tree-sitter.js')); -const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(dist('extraction/grammars.js')); -const kernel = await import(dist('extraction/kernel/index.js')); +// As file:// URLs: import() rejects a bare `C:\...` path on Windows +// (ERR_UNSUPPORTED_ESM_URL_SCHEME). +const { extractFromSource } = await import(pathToFileURL(dist('extraction/tree-sitter.js')).href); +const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(pathToFileURL(dist('extraction/grammars.js')).href); +const kernel = await import(pathToFileURL(dist('extraction/kernel/index.js')).href); await initGrammars(); await loadGrammarsForLanguages([...KERNEL_LANGS]); From 69ee3b9a8597940b2a12ea5fe26c48ab0c9c0650 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 19:03:03 +0000 Subject: [PATCH 208/363] fix(ui): fall back past a port Windows refuses with EACCES, and say why (#2299) (#2353) On Windows, a port another program holds exclusively (SO_EXCLUSIVEADDRUSE) or one inside an excluded port range fails listen() with EACCES rather than EADDRINUSE. The `codegraph ui` port fallback only moved on after EADDRINUSE, so it stopped at the first such port. The error then blamed the POSIX privileged-port rule ("Ports below 1024 usually need elevated privileges") for ports like 4747 or 49912. Confirmed on Windows 11: a .NET listener on 0.0.0.0:P with ExclusiveAddressUse makes Node's listen(P, '127.0.0.1') fail with EACCES. Fix: - shouldTryNextPort moves past EADDRINUSE on every platform. It moves past EACCES at any port on Windows, which has no privileged ports, and from 1024 up elsewhere. - Below 1024 off Windows, EACCES is the privileged-port rule, and the next port falls under it too. The walk stops there with the message it always gave, rather than trying 20 privileged ports. - describeBindFailure takes the platform: - Windows names the exclusive hold or reserved range and how to list the ranges (netsh int ipv4 show excludedportrange protocol=tcp). - POSIX keeps the privileged-port text below 1024. Above it, it says the system refused permission. - A walk that ran out says "in use or reserved" (Windows) or "in use or not allowed" when any port was refused rather than in use. - Both helpers take the platform as a parameter, like browserOpenCommand, so the tests check every platform's answer on any host. This builds on PR #2319 by @sx4im, which advanced on EACCES, used the Windows wording and added the issue's listen-spy test. That PR advanced on EACCES on every platform and printed "Windows refused port N ... netsh" on Linux and macOS too. On POSIX it would also walk 20 privileged ports and end with "Ports 80-99 are all in use or reserved". This change restricts both to where they are true. The changelog entry goes in the viewer-launch file, because the viewer is not released yet. Verification: - __tests__/ui-server.test.ts gains three kinds of test: - The issue's end-to-end test: port N is taken and N+1 is refused through a listen spy, so the server must land past N+1 and serve 200. - A pinned-port message test. - Unit tests of the decision and the messages for win32, linux and darwin. - Results on __tests__/ui-server.test.ts: - Main source: 9 failed / 40 passed. The walk fails with "Not allowed to listen on port 57423. Ports below 1024 usually need elevated privileges". - With the fix: 49 passed / 2 skipped (existing POSIX-gated tests). - Built CLI on Windows, with 0.0.0.0: held exclusively: - `codegraph ui` while 4747 is held: main exits 1 with the privileged-port message. With the fix it serves on 4748 (GET / 200). - `codegraph ui --port `: prints the Windows message. - A 3-port walk over held ports reports "in use or reserved". - Re-listening on the same server after a real EACCES works on Node 22.22 and 24.16. - tsc --noEmit is clean and npm run build succeeds. The 20 viewer test files pass (439 passed, 5 skipped). - Full suite: 5732 passed, 10 failed, 319 skipped. The 10 failures are timeouts and EBUSY teardown under load in function-ref, git-index-currency, index-daemon-rebuild and rust-self-owner. All four files pass when run alone. Issue #2299 reported by @ijbranch. Co-authored-by: Saim Shafique Co-authored-by: Claude Opus 5.5 --- __tests__/ui-server.test.ts | 160 +++++++++++++++++++++++++++++++- docs/viewer-launch-changelog.md | 2 + src/ui-server/index.ts | 92 +++++++++++++++--- 3 files changed, 238 insertions(+), 16 deletions(-) diff --git a/__tests__/ui-server.test.ts b/__tests__/ui-server.test.ts index df25858a84..4b4e63fc8c 100644 --- a/__tests__/ui-server.test.ts +++ b/__tests__/ui-server.test.ts @@ -12,7 +12,7 @@ * name in undici, and forging it is the whole point of half these cases. */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; import * as http from 'http'; import * as fs from 'fs'; import * as os from 'os'; @@ -21,12 +21,14 @@ import { browserOpenCommand, cacheControlFor, contentTypeFor, + describeBindFailure, isAllowedHost, isAllowedOrigin, isSafeRequestPath, PathRefusalError, resolveProjectFile, resolveStaticAsset, + shouldTryNextPort, startUiServer, type UiServerHandle, } from '../src/ui-server'; @@ -69,6 +71,31 @@ function request( }); } +/** A failed `listen()` as node reports it. */ +function bindError(code: string, port: number): NodeJS.ErrnoException { + return Object.assign(new Error(`listen ${code}: 127.0.0.1:${port}`), { code, syscall: 'listen' }); +} + +/** + * Make `listen()` on `refused` fail with EACCES, the way Windows refuses a port + * another program holds exclusively or one inside a reserved range, without + * needing either on the machine running the test (#2299). Every other port is + * a real bind. + */ +function refuseListenOn(...refused: number[]): { mockRestore(): void } { + const realListen = http.Server.prototype.listen; + return vi + .spyOn(http.Server.prototype, 'listen') + .mockImplementation(function (this: http.Server, ...args: unknown[]) { + const port = args[0]; + if (typeof port === 'number' && refused.includes(port)) { + process.nextTick(() => this.emit('error', bindError('EACCES', port))); + return this; + } + return (realListen as (...a: unknown[]) => http.Server).apply(this, args); + }); +} + describe('codegraph ui server', () => { let tempDir: string; let viewerDir: string; @@ -198,6 +225,50 @@ describe('codegraph ui server', () => { await new Promise((resolve) => blocker.close(() => resolve())); } }); + + /** + * #2299: on Windows a port another program holds exclusively, or one in a + * reserved range, fails with EACCES rather than EADDRINUSE, and the walk + * used to stop there. Port N is really taken, N+1 is refused, so the server + * has to land beyond both. N+1 is an ephemeral port, above 1023, so this + * holds on every platform. + */ + it('falls back past a port refused with EACCES', async () => { + const blocker = http.createServer(() => {}); + await new Promise((resolve) => blocker.listen(0, '127.0.0.1', resolve)); + const taken = (blocker.address() as { port: number }).port; + const refusal = refuseListenOn(taken + 1); + + try { + const second = await startUiServer({ projectRoot, viewerDir, port: taken }); + try { + expect(second.port).toBeGreaterThan(taken + 1); + // …and it actually works on the port it landed on. + const res = await request(second.port, '/'); + expect(res.status).toBe(200); + } finally { + await second.close(); + } + } finally { + refusal.mockRestore(); + await new Promise((resolve) => blocker.close(() => resolve())); + } + }); + + it('blames a refused pinned port on what refused it, not on privileged ports', async () => { + // The port from the report: nowhere near 1024. + const refusal = refuseListenOn(49912); + try { + const failure = startUiServer({ projectRoot, viewerDir, port: 49912, portFallback: false }); + await expect(failure).rejects.toThrow(/49912/); + const message = await failure.catch((err: Error) => err.message); + expect(message).not.toMatch(/below 1024|elevated privileges/); + expect(message).not.toMatch(/already in use/); + expect(message).toMatch(/refused/); + } finally { + refusal.mockRestore(); + } + }); }); describe('Host allowlist (DNS rebinding)', () => { @@ -538,6 +609,93 @@ describe('security helpers', () => { }); }); +/** + * What a bind failure means depends on the platform (#2299): Windows has no + * privileged ports and refuses a taken or reserved one with EACCES, while + * elsewhere EACCES below 1024 is the privileged-port rule. The platform is a + * parameter, so every host checks every platform's answer. + */ +describe('port fallback on each platform', () => { + const LIST_RESERVED = 'netsh int ipv4 show excludedportrange protocol=tcp'; + + it('moves past a port in use everywhere, and past EACCES on Windows at any port', () => { + for (const platform of ['win32', 'linux', 'darwin'] as const) { + expect(shouldTryNextPort('EADDRINUSE', 80, platform), platform).toBe(true); + expect(shouldTryNextPort('EADDRINUSE', 4747, platform), platform).toBe(true); + } + expect(shouldTryNextPort('EACCES', 49912, 'win32')).toBe(true); + // Windows has no privileged-port rule: a low port is held, not forbidden. + expect(shouldTryNextPort('EACCES', 80, 'win32')).toBe(true); + }); + + it('stops at EACCES below 1024 elsewhere, where the next port needs the same privilege', () => { + expect(shouldTryNextPort('EACCES', 80, 'linux')).toBe(false); + expect(shouldTryNextPort('EACCES', 1023, 'darwin')).toBe(false); + // From 1024 up the refusal is about that port, and the next may be allowed. + expect(shouldTryNextPort('EACCES', 1024, 'linux')).toBe(true); + expect(shouldTryNextPort('EACCES', 4747, 'darwin')).toBe(true); + }); + + it('never moves past any other failure', () => { + for (const platform of ['win32', 'linux'] as const) { + expect(shouldTryNextPort('EADDRNOTAVAIL', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort('EPERM', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort(undefined, 4747, platform), platform).toBe(false); + } + }); + + it('says why Windows refused a pinned port, without the POSIX privileged-port story', () => { + for (const port of [49912, 80]) { + const message = describeBindFailure(bindError('EACCES', port), port, { port, fallback: false }, 'win32').message; + expect(message).toMatch( + new RegExp(`^Windows refused port ${port}: another program holds it, or it is in a reserved range`) + ); + expect(message).toContain(LIST_RESERVED); + expect(message).toContain('omit --port'); + expect(message).not.toMatch(/below 1024|elevated privileges/); + } + }); + + it('keeps the privileged-port explanation off Windows, and only below 1024', () => { + for (const fallback of [false, true]) { + const message = describeBindFailure(bindError('EACCES', 80), 80, { port: 80, fallback }, 'linux').message; + expect(message).toBe( + 'Not allowed to listen on port 80. Ports below 1024 usually need elevated privileges — pick a higher one with --port.' + ); + } + const high = describeBindFailure(bindError('EACCES', 4747), 4747, { port: 4747, fallback: false }, 'darwin').message; + expect(high).toMatch(/^Not allowed to listen on port 4747: the system refused permission\./); + expect(high).not.toMatch(/below 1024|elevated privileges|Windows|netsh/); + }); + + it('says a walk that ran out met refusals, not only ports in use', () => { + const walk = { port: 4747, fallback: true }; + const inUse = bindError('EADDRINUSE', 4766); + const refused = bindError('EACCES', 4766); + + // Nothing refused: the message it always gave. + expect(describeBindFailure(inUse, 4766, walk, 'win32').message).toBe( + 'Ports 4747–4766 are all in use. Free one, or pick another with --port.' + ); + const windows = describeBindFailure(refused, 4766, walk, 'win32').message; + expect(windows).toMatch(/^Ports 4747–4766 are all in use or reserved/); + expect(windows).toContain(LIST_RESERVED); + // An earlier port was refused even though the last one was only in use. + expect(describeBindFailure(inUse, 4766, walk, 'win32', true).message).toBe(windows); + const posix = describeBindFailure(refused, 4766, walk, 'linux').message; + expect(posix).toMatch(/^Ports 4747–4766 are all in use or not allowed\./); + expect(posix).not.toMatch(/reserved|netsh/); + }); + + it('leaves the pinned in-use message and other failures as they were', () => { + expect( + describeBindFailure(bindError('EADDRINUSE', 8080), 8080, { port: 8080, fallback: false }, 'win32').message + ).toBe('Port 8080 is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.'); + const other = bindError('EADDRNOTAVAIL', 4747); + expect(describeBindFailure(other, 4747, { port: 4747, fallback: true }, 'linux')).toBe(other); + }); +}); + describe('browserOpenCommand', () => { it('uses the platform opener', () => { expect(browserOpenCommand('http://x', 'darwin')).toEqual({ command: 'open', args: ['http://x'] }); diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 593ba632c0..84ed815051 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -205,6 +205,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel - A viewer trail now holds up to 64 hops everywhere — the trail bar, saved trails and "Read as flow". Thanks @inth3shadows for the report and @danusha2345. (#1976) +- **`codegraph ui` on Windows moves on from a port another program holds.** Windows refuses a port that another program keeps for itself, or one inside a range it reserves, with a different error than a port that is simply in use, so the viewer gave up at the first such port instead of trying the next one, and blamed ports below 1024 needing elevated privileges — a rule Windows doesn't have. It now moves on to the next free port, and when none is left, or the port you pinned with `--port` is refused, it says what Windows did and how to list the reserved ranges. Thanks @ijbranch for the report and @sx4im. (#2299) + ## Entries whose graph and `codegraph_explore` half already shipped The release notes carry a reworded version of each of these. Keep only the viewer half at launch. diff --git a/src/ui-server/index.ts b/src/ui-server/index.ts index 40c170f703..e5d0cced8a 100644 --- a/src/ui-server/index.ts +++ b/src/ui-server/index.ts @@ -397,9 +397,10 @@ function safeDecode(value: string): string { /** * Bind the first free port at or after `port`, on loopback only. * - * Only `EADDRINUSE` advances to the next port — a permission failure or a bad - * address will not get better one port over, and retrying twenty times would - * only bury the real error. + * Only a port that is taken advances to the next one (see + * {@link shouldTryNextPort}) — a permission failure every port shares, or a + * bad address, will not get better one port over, and retrying twenty times + * would only bury the real error. */ async function listenWithFallback( server: http.Server, @@ -407,6 +408,9 @@ async function listenWithFallback( ): Promise { // Port 0 means "any free port", so there is nothing to fall back from. const attempts = opts.port === 0 || !opts.fallback ? 1 : Math.max(1, opts.attempts); + const platform = process.platform; + // Whether any port was refused rather than in use — the message says so. + let refused = false; for (let i = 0; i < attempts; i++) { const candidate = opts.port === 0 ? 0 : opts.port + i; @@ -419,8 +423,9 @@ async function listenWithFallback( return address.port; } catch (err) { const code = (err as NodeJS.ErrnoException).code; - if (code !== 'EADDRINUSE' || i === attempts - 1) { - throw describeBindFailure(err, candidate, opts); + if (code === 'EACCES') refused = true; + if (!shouldTryNextPort(code, candidate, platform) || i === attempts - 1) { + throw describeBindFailure(err, candidate, opts, platform, refused); } } } @@ -434,7 +439,8 @@ async function listenWithFallback( * The same `http.Server` is reused across attempts: a `listen()` that failed * with EADDRINUSE never took a handle, so it can be listened on again directly * (verified on Node 20 and 22 — `server.listening` is still `false` afterwards, - * and `close()` on a never-listening server would itself throw). + * and `close()` on a never-listening server would itself throw). The same goes + * for a port Windows refused with EACCES (checked on Node 22 and 24). */ function listenOnce(server: http.Server, port: number): Promise { return new Promise((resolve, reject) => { @@ -452,22 +458,78 @@ function listenOnce(server: http.Server, port: number): Promise { }); } -/** Turn a bind failure into something a user can act on. */ -function describeBindFailure( +/** + * Whether a `listen()` on `port` that failed with `code` leaves the next port + * worth trying. + * + * `EADDRINUSE` does, and so does `EACCES` on Windows: a port another program + * holds exclusively (`SO_EXCLUSIVEADDRUSE`), or one inside a range the system + * reserves (Hyper-V and WinNAT reserve whole blocks of them), fails there with + * EACCES rather than EADDRINUSE. Elsewhere, EACCES below 1024 is the + * privileged-port rule, which the next port is under too, so walking on would + * only bury it. From 1024 up it is something refusing that one port — a + * security policy, say — and the next one may be allowed. + */ +export function shouldTryNextPort( + code: string | undefined, + port: number, + platform: NodeJS.Platform = process.platform +): boolean { + if (code === 'EADDRINUSE') return true; + return code === 'EACCES' && !isPrivilegedPort(port, platform); +} + +/** + * Whether `port` is one only an administrator may bind: below 1024, on every + * platform but Windows, which has no such rule. + */ +function isPrivilegedPort(port: number, platform: NodeJS.Platform): boolean { + return platform !== 'win32' && port > 0 && port < 1024; +} + +/** The command that lists the port ranges Windows reserves. */ +const LIST_RESERVED_PORTS = '`netsh int ipv4 show excludedportrange protocol=tcp`'; + +/** + * Turn a bind failure into something a user can act on, and that is true where + * it is read: Windows refuses ports for reasons POSIX doesn't have, and has no + * privileged ports. + * + * `refused` says whether any port a fallback walk tried was refused (EACCES) + * rather than in use; it only changes the message for a walk that ran out. + */ +export function describeBindFailure( err: unknown, port: number, - opts: { port: number; fallback: boolean; attempts: number } + opts: { port: number; fallback: boolean }, + platform: NodeJS.Platform = process.platform, + refused = false ): Error { const code = (err as NodeJS.ErrnoException).code; + const windows = platform === 'win32'; + if (opts.fallback && shouldTryNextPort(code, port, platform)) { + // A failure that moves on only ends the walk when there is nowhere left. + const taken = + !refused && code === 'EADDRINUSE' + ? 'in use' + : windows + ? `in use or reserved (${LIST_RESERVED_PORTS} lists the reserved ranges)` + : 'in use or not allowed'; + return new Error(`Ports ${opts.port}–${port} are all ${taken}. Free one, or pick another with --port.`); + } if (code === 'EADDRINUSE') { - return opts.fallback - ? new Error( - `Ports ${opts.port}–${port} are all in use. Free one, or pick another with --port.` - ) - : new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); + return new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); } if (code === 'EACCES') { - return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + if (isPrivilegedPort(port, platform)) { + return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + } + return new Error( + (windows + ? `Windows refused port ${port}: another program holds it, or it is in a reserved range (${LIST_RESERVED_PORTS} lists them).` + : `Not allowed to listen on port ${port}: the system refused permission.`) + + ' Pick another with --port, or omit --port to let CodeGraph find a free one.' + ); } return err instanceof Error ? err : new Error(String(err)); } From 6c61dd74ca672dd8aa4adc86718406641967e109 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:05:18 +0000 Subject: [PATCH 209/363] fix(vbnet): link Shared fields and properties read through their class (#2305) (#2355) VB.NET emitted a reference for calls only, so a value read or write through a type name -- `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level` -- linked nothing, and `codegraph callers` on the field, the property and the class came back empty. The reporter's central session class showed 6 callers against ~460 referencing files. - Extraction (TS only; VB.NET is not kernel-routed): the static-member pass now covers VB.NET. A value read through a Capitalized name is sent as one `references` ref, `Name.Member`, the receiver kept as a call's is. Namespace roots (`System`, `Microsoft`, `My`, `Global`) and the built-in type keywords (`String.Empty`) are not sent. - Resolution (vbnet-receivers.ts matchVbMemberRead, ahead of the framework, import and name strategies): the receiver is looked up with #2351's VB.NET scoping, case-insensitively. A local, parameter, field, property or Module member of that name holds a value and links nothing; one typed as the type of its own name ("Color Color") reads that type. Otherwise the class, Module, Structure, Interface or Enum the namespaces, Imports and aliases around the read see is the type: the read links the member it declares or inherits (field, property, constant, event, Enum value, nested type; Shared before a same-named instance member, never the member the read is in) and, from outside the type, the type itself. A method named this way is a call made without parentheses (`calls`) unless AddressOf / NameOf only names it. - `AppSession.Items(0)`, an index into a Shared field that VB.NET writes as a call, now reads the field and the type instead of linking nothing. Contributor PR #2321 (@ChrisPrapas) had the right idea -- enable the static-member pass for VB.NET with a member-level ref -- and its test scenarios are adopted. Its resolution is replaced: it name-matched the receiver's bare name and checked a member only against the receiver's spelling. Ported onto main it passes its own tests, but binds a parameter `appSession As OtherSession` to AppSession's members, misses import aliases, and adds 8,652 / 13,554 edges on SCrawler / staxrip, about 2,000 per repo binding a receiver to another class's same-named member (a plugin form's local `Dim CONTAINER_MAIN` went to another form's control field). Verification: new vbnet-shared-member-refs.test.ts (7 tests) fails 6/7 on main and passes; the other VB.NET suites pass; tsc and build clean; the issue's repro lists Consumer.Run and Consumer2.Describe for SessionId, CurrentUser and AppSession. Validation, main vs this branch: nodes unchanged, 0 edges removed, no self-loops. SCrawler 14,893 -> 17,437 edges, staxrip 26,647 -> 32,568: Enum values 803 / 1,448, Shared fields and properties 363 / 1,459, the type read through 1,374 / 2,879, Shared functions called without parentheses 3 / 126, AddressOf / NameOf 1 / 9. Spot-checked against source: per-site SiteSettings, import aliases, escaped `[Date]` / `[New]`, nested enums, "Color Color" fields; designer `Point.Empty` and staxrip's AutoCrop duplicates stay unlinked. Index time within noise; value reads that link nothing are kept for re-resolution like failed calls (db 22.6 -> 28.9 MB, 28.3 -> 35.8 MB). Not covered: reads through Me / MyBase, unqualified reads, instance reads through a typed variable, Module variables as receivers, namespace-qualified receivers and With blocks. Co-authored-by: Claude Opus 5.5 Co-authored-by: ChrisPrapas <129202926+ChrisPrapas@users.noreply.github.com> --- CHANGELOG.md | 1 + __tests__/vbnet-shared-member-refs.test.ts | 218 +++++++++++++++++++++ src/extraction/tree-sitter.ts | 43 +++- src/resolution/index.ts | 8 +- src/resolution/vbnet-receivers.ts | 140 +++++++++++-- 5 files changed, 396 insertions(+), 14 deletions(-) create mode 100644 __tests__/vbnet-shared-member-refs.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b5476b128f..9c5f79e4c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) +- In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/vbnet-shared-member-refs.test.ts b/__tests__/vbnet-shared-member-refs.test.ts new file mode 100644 index 0000000000..bbcfa77ffe --- /dev/null +++ b/__tests__/vbnet-shared-member-refs.test.ts @@ -0,0 +1,218 @@ +/** + * VB.NET: a value read or written through a type or module name — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — is a use of the member and of the type (#2305). Before, only + * calls linked, so `codegraph callers` on a Shared field or property, and on + * the class itself, came back empty. + * + * The name is looked up as VB.NET does, without regard to case: a local, a + * parameter or a field of that name holds a value, and links nothing here (a + * member typed as its own name's type, `Property Settings As Settings`, reads + * that type's members either way); a type is the one the namespaces around + * the read, its `Imports` and aliases see — SCrawler declares a + * `SiteSettings` in every site's namespace. A method named without + * parentheses is called, unless `AddressOf` only names it, and an index into + * a Shared field (`AppSession.Items(0)`), which VB.NET writes as a call, reads + * the field. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const files: Record = { + // The issue's three files. + 'AppSession.vb': `Public Class AppSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Shared Property CurrentUser As String + Public Shared Items As New List(Of String) + Public Shared Function GetGreeting() As String + Return "Hello " & CurrentUser + End Function + Public Shared Sub OnTick(ByVal sender As Object, ByVal e As EventArgs) + End Sub + Public Shared Function Describe() As String + Return AppSession.CurrentUser + End Function +End Class +`, + 'Consumer.vb': `Public Class Consumer + Public Sub Run() + Dim id As Guid = AppSession.SessionId + AppSession.CurrentUser = "demo" + Console.WriteLine(AppSession.GetGreeting()) + End Sub +End Class +`, + 'Consumer2.vb': `Public Class Consumer2 + Public Function Describe() As String + Return AppSession.CurrentUser & AppSession.SessionId.ToString() + End Function +End Class +`, + 'OtherSession.vb': `Public Class OtherSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Property Size As Integer +End Class +`, + 'Size.vb': `Public Class Size +End Class +`, + 'Logger.vb': `Public Module Logger + Public Level As Integer +End Module +`, + 'Mode.vb': `Public Enum Mode + Fast + Slow +End Enum +`, + 'Form1.vb': `Public Class Form1 + Private Panel1 As Panel + Public Sub Setup() + Logger.Level = 3 + Dim a = Panel1.Size + Me.Panel1.Size = New System.Drawing.Size(1, 2) + Dim speed = Mode.Fast + Dim greeting = AppSession.GetGreeting + AddHandler Timer1.Tick, AddressOf AppSession.OnTick + Dim first = AppSession.Items(0) + Dim empty = String.Empty + End Sub + Public Sub Shadowed(ByVal appSession As OtherSession) + Dim id = AppSession.SessionId + End Sub +End Class +`, + // A type declared in each site's namespace, and one read through an import alias. + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Const Header As String = "r" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Twitter/SiteSettings.vb': `Namespace API.Twitter + Friend Class SiteSettings + Friend Const Header As String = "t" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Facebook/UserData.vb': `Imports RS = API.Reddit.SiteSettings +Namespace API.Facebook + Friend Class UserData + Friend Function Key() As String + Return RS.Header + End Function + End Class +End Namespace +`, + // A property typed as the type of its own name; a property initializer. + 'Settings.vb': `Public Class Settings + Public Property Theme As String +End Class +`, + 'Window.vb': `Public Class Window + Public Property Settings As Settings + Public Property Title As String = AppSession.CurrentUser + Public Function CurrentTheme() As String + Return Settings.Theme + End Function +End Class +`, + // CRLF line endings and multibyte text before the read. + 'Weird.vb': [ + 'Public Class Weird', + ' Public Sub Go()', + ' Dim s = "SessionId éééé 😀" & AppSession.SessionId', + ' End Sub', + 'End Class', + '', + ].join('\r\n'), +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-shared-')); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +function nodeId(qualifiedName: string): string { + const found = cg.getNodesInFiles(Object.keys(files)).filter((n) => n.qualifiedName === qualifiedName); + if (found.length !== 1) throw new Error(`${found.length} nodes named ${qualifiedName}`); + return found[0]!.id; +} + +/** The qualified names of what links to `qualifiedName` with an edge of `kind`. */ +function linkedFrom(qualifiedName: string, kind: 'references' | 'calls'): string[] { + return cg + .getIncomingEdgesTo([nodeId(qualifiedName)], [kind]) + .map((e) => cg.getNode(e.source)!.qualifiedName) + .sort(); +} + +describe('VB.NET Shared members read through their type (#2305)', () => { + it('link a Shared field and property to the methods that read or write them', () => { + expect(linkedFrom('AppSession::SessionId', 'references')).toEqual(['Consumer2::Describe', 'Consumer::Run', 'Weird::Go']); + expect(linkedFrom('AppSession::CurrentUser', 'references')) + .toEqual(['AppSession::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Window::Title']); + }); + + it('link the class to the code outside it that reads through it', () => { + expect(linkedFrom('AppSession', 'references')) + .toEqual(['Consumer2::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + expect(cg.getCallers(nodeId('AppSession')).map((c) => c.node.qualifiedName).sort()) + .toEqual(['Consumer2::Describe', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + }); + + it('keep the call to a Shared method, and call one named without parentheses', () => { + expect(linkedFrom('AppSession::GetGreeting', 'calls')).toEqual(['Consumer::Run', 'Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'calls')).toEqual([]); + }); + + it('read a Shared field indexed like a call, a Module variable and an Enum value', () => { + expect(linkedFrom('AppSession::Items', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Logger::Level', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode::Fast', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode', 'references')).toEqual(['Form1::Setup']); + }); + + it('link nothing through a local, parameter or field, whatever its case', () => { + expect(linkedFrom('OtherSession::SessionId', 'references')).toEqual([]); + expect(linkedFrom('OtherSession::Size', 'references')).toEqual([]); + expect(linkedFrom('Size', 'references')).toEqual([]); + expect(linkedFrom('OtherSession', 'references')).toEqual([]); + }); + + it('read the type the namespaces and imports around the read see', () => { + expect(linkedFrom('API.Reddit::SiteSettings::Header', 'references')) + .toEqual(['API.Facebook::UserData::Key', 'API.Reddit::UserData::Key']); + expect(linkedFrom('API.Twitter::SiteSettings::Header', 'references')).toEqual(['API.Twitter::UserData::Key']); + }); + + it('read through a member typed as the type of its own name', () => { + expect(linkedFrom('Settings::Theme', 'references')).toEqual(['Window::CurrentTheme']); + }); +}); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 222291f14c..17b0b02caf 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -361,7 +361,7 @@ const PHP_TYPE_NODES: ReadonlySet = new Set([ */ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ 'field_access', // java (`Foo.BAR`) - 'member_access_expression', // c# (`Foo.Bar`) + 'member_access_expression', // c# / vb.net (`Foo.Bar`) 'navigation_expression', // kotlin / swift (`Foo.bar`) 'field_expression', // scala (`Foo.bar`) 'class_constant_access_expression', // php (`Foo::CONST`, `Foo::class`) @@ -380,11 +380,21 @@ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ * static read is pure duplication) — while adding real graph noise (+1813 edges / * +2448 `references` on excalidraw, the retrieval-perf benchmark, all pointing at * already-covered types). Don't re-add `member_expression`/`attribute` here. + * VB.NET (#2305) sends the member with its receiver instead, and its resolver + * decides whether the receiver is a type (see extractVbMemberRead). */ const STATIC_MEMBER_LANGS: ReadonlySet = new Set([ - 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', + 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', 'vbnet', ]); +/** + * VB.NET receivers no project type can be named: the namespace roots + * (`System.IO.Path`, `My.Settings`, `Global.X`) and the built-in type keywords + * (`String.Empty`, `Integer.MaxValue`). A read through one is never sent. + */ +const VB_NON_TYPE_RECEIVERS = + /^(?:Global|System|Microsoft|My|Boolean|Byte|Char|Date|Decimal|Double|Integer|Long|Object|SByte|Short|Single|String|UInteger|ULong|UShort)$/i; + /** * Tree-sitter node kinds that represent constructor invocations * (`new Foo()` and friends). Used by extractInstantiation to emit @@ -5549,6 +5559,10 @@ export class TreeSitterExtractor { getChildByField(node, 'scope') ?? node.namedChild(0); if (!recv) return; + if (this.language === 'vbnet') { + this.extractVbMemberRead(node, recv, ownerId); + return; + } const t = recv.type; if ( t === 'identifier' || t === 'type_identifier' || t === 'simple_identifier' || @@ -5569,6 +5583,31 @@ export class TreeSitterExtractor { }); } + /** + * VB.NET: a value read or write through a name — `AppSession.SessionId`, + * `AppSession.CurrentUser = "demo"`, `Logger.Level`, `Mode.Fast` — is a use + * of the member as well as of what the name names (#2305). One `references` + * ref carries both, as `Name.Member` (the receiver kept, as a call's is); + * the resolver links the member and the type when the name means a project + * type or module there, and nothing when it holds a value (a local, a + * parameter, a field), which only it can tell in case-insensitive VB.NET + * (see vbnet-receivers' matchVbMemberRead). Its types are Capitalized all + * the same, so a lowercase receiver — a local, nearly always — is skipped. + */ + private extractVbMemberRead(node: SyntaxNode, recv: SyntaxNode, ownerId: string): void { + const member = getChildByField(node, 'member'); + if (recv.type !== 'identifier' || member?.type !== 'identifier') return; + const name = getNodeText(recv, this.source); + if (!/^[A-Z]\w*$/.test(name) || VB_NON_TYPE_RECEIVERS.test(name)) return; + this.unresolvedReferences.push({ + fromNodeId: ownerId, + referenceName: `${name}.${getNodeText(member, this.source)}`, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + /** * Find a `class_body` child of an `object_creation_expression` — the * marker for an anonymous class (`new T() { ... }`). Returns the body diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 4c9fb45f2c..fcd47b7720 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,7 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; -import { clearVbnetReceiverMemos } from './vbnet-receivers'; +import { clearVbnetReceiverMemos, isVbMemberRead, matchVbMemberRead } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -1134,6 +1134,12 @@ export class ReferenceResolver { return this.gateLanguage(matchJsStoreBindingCall(ref, this.context), ref); } + // A VB.NET value read through a name (`AppSession.SessionId`, #2305) means + // what VB.NET's scoping says the name is — a project type, whose member + // and the type itself it links, or a value, which links nothing here — and + // no framework, import or name strategy guesses past that. + if (isVbMemberRead(ref)) return this.gateLanguage(matchVbMemberRead(ref, this.context), ref); + // Function-as-value refs (#756) get a dedicated, strictly-gated path: // import-based resolution first (an imported callback resolves through its // import, the most precise cross-file signal), then matchFunctionRef diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts index 07732927e1..f252dfc979 100644 --- a/src/resolution/vbnet-receivers.ts +++ b/src/resolution/vbnet-receivers.ts @@ -432,9 +432,9 @@ function typesAround(file: string, line: number, context: ResolutionContext): No return types.filter((n) => n.startLine <= line && n.endLine >= line); } -/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ -function projectTypesNamed(name: string, context: ResolutionContext): Node[] { - return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +/** The project's VB.NET types (classes, modules, structures, interfaces — or `kinds`) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext, kinds: ReadonlySet = VB_TYPE_KINDS): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && kinds.has(n.kind)); } const VB_MEMBERS = new WeakMap>(); @@ -768,11 +768,16 @@ export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, co * SCrawler declares * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is - * Pinterest's. `ambiguous` when what is left are different types. + * Pinterest's. `ambiguous` when what is left are different types. `kinds` + * widens the types looked for (an `Enum` a value is read through). */ -function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { +function typesNamedAt( + written: VbType, + context: ResolutionContext, + kinds: ReadonlySet = VB_TYPE_KINDS, +): { owners: Node[]; ambiguous: boolean } { const t = written.file ? unalias(written, written.file, context) : written; - let candidates = projectTypesNamed(t.name, context); + let candidates = projectTypesNamed(t.name, context, kinds); // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; @@ -993,9 +998,9 @@ function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { } /** - * A method — or, `withValues`, a field or property — named `name` on one of - * `owners` (the type `typed` names) or a project type they inherit, with what - * the type parameters of the type declaring it stand for. + * A method — or, `withValues`, a field or property; or what `fits` — named + * `name` on one of `owners` (the type `typed` names) or a project type they + * inherit, with what the type parameters of the type declaring it stand for. */ function memberOn( owners: Node[], @@ -1004,8 +1009,8 @@ function memberOn( context: ResolutionContext, withValues: boolean, typed?: VbType, + fits: (n: Node) => boolean = (n) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)), ): { node: Node; args: Map } | null { - const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); for (const level of [0, 1]) { for (const owner of preferVbProject(owners, ref, context)) { // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). @@ -1238,7 +1243,11 @@ export function matchVbTypedCall( if (named.owners.length === 0) return undefined; if (named.ambiguous) return null; const shared = memberOn(named.owners, method, ref, context, false); - return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + if (shared) return { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' }; + // `AppSession.Items(0)`: an index into a shared field or property, which + // VB.NET writes as a call — a read of the member and of its type (#2305). + const indexed = memberOn(named.owners, method, ref, context, true); + return indexed ? readThrough(indexed.node, named.owners, ref, context, { edgeKind: 'references' }) : null; } if (!type || typeKey(type) === 'object') return undefined; const owners = ownersOf(type, context); @@ -1251,6 +1260,115 @@ export function matchVbTypedCall( return extensionFor(type, owners, method, ref, context, isStdMethod); } +/** The types a value is read through: those a shared call is made on, and an `Enum`. */ +const VB_READ_TYPE_KINDS: ReadonlySet = new Set([...VB_TYPE_KINDS, 'enum']); + +/** + * What a read through a type names: a value it holds or gives (a field, a + * property, a constant, an `Enum` case, an event), a method — which VB.NET + * runs when it is named without parentheses — or a type nested in it. + */ +const VB_READ_KINDS: ReadonlySet = new Set([...VB_VALUE_KINDS, 'enum_member', 'method', ...VB_TYPE_LIKE_KINDS]); + +/** `Name.Member`, as the extractor sends a VB.NET value read through a name (extractVbMemberRead). */ +const VB_MEMBER_READ = /^([A-Za-z_]\w*)\.(\[?[A-Za-z_]\w*\]?)$/; + +/** Whether a reference is a VB.NET value read through a name, which matchVbMemberRead alone resolves. */ +export function isVbMemberRead(ref: UnresolvedRef): boolean { + return ref.language === 'vbnet' && ref.referenceKind === 'references' && VB_MEMBER_READ.test(ref.referenceName); +} + +/** + * The project types `name` means where a value is read through it + * (`AppSession` in `AppSession.SessionId`): a class, module, structure, + * interface or enum. Null when the name holds a value there — a local, a + * parameter, a member of a type around the read or of a module — or names + * no project type, or two. A member typed as the type of its own name + * (`Public Property Settings As Settings`) reaches that type's members + * either way, as VB.NET's "Color Color" rule has it. + */ +function typesReadThrough(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + const bound = receiverType(name, ref, context, 0); + const written: VbType | null = bound === undefined ? { name, array: false, file: ref.filePath, line: ref.line } + : bound && !bound.array && bound.name.toLowerCase() === name.toLowerCase() ? bound : null; + if (!written) return null; + const found = typesNamedAt(written, context, VB_READ_TYPE_KINDS); + return found.ambiguous || found.owners.length === 0 ? null : found.owners; +} + +/** + * A read of `member` through one of `owners` (the parts of the type its + * receiver names): the member, and the type as a second target — unless the + * read is written inside that type, which doesn't depend on itself. + */ +function readThrough( + member: Node, + owners: Node[], + ref: UnresolvedRef, + context: ResolutionContext, + extra: Partial = {}, +): ResolvedRef | null { + if (member.id === ref.fromNodeId) return null; + const owner = owners.find((o) => o.filePath === member.filePath) ?? owners[0]!; + const from = context.getNodeById?.(ref.fromNodeId)?.qualifiedName.toLowerCase(); + const own = owner.qualifiedName.toLowerCase(); + const inside = owner.id === ref.fromNodeId || (from !== undefined && (from === own || from.startsWith(`${own}::`))); + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...extra, + ...(inside ? {} : { alsoTargets: [{ targetNodeId: owner.id }] }), + }; +} + +/** Whether a method is named at a reference without being run: `AddressOf Type.Method`, `NameOf(Type.Method)`. */ +function namesWithoutRunning(ref: UnresolvedRef, context: ResolutionContext): boolean { + const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; + return /\b(?:AddressOf\s+|NameOf\s*\(\s*)$/i.test(line.slice(0, Math.max(0, ref.column))); +} + +/** + * Resolve a VB.NET value read or write through a name (#2305) — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — to the member that the type the name means declares or + * inherits, and to the type. A method named this way is called (no type is + * linked, as for a call written with parentheses), unless `AddressOf` or + * `NameOf` only names it. Null when the name holds a value there (an + * instance's members are not read through its type), names no project type, + * or names one without that member: `Color.Red` is not a project `Color`'s. + */ +export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const m = VB_MEMBER_READ.exec(ref.referenceName); + if (!m) return null; + const owners = typesReadThrough(m[1]!, ref, context); + if (!owners) return null; + // Not the member the read is written in: staxrip's `Overrides ReadOnly + // Property Package` returns the class's `Shared ReadOnly Property Package`. + const fits = (n: Node) => VB_READ_KINDS.has(n.kind) && n.id !== ref.fromNodeId; + // `MySettings.Default` reads the property declared `[Default]`, and back. + const name = m[2]!.replace(/^\[(.*)\]$/, '$1'); + let member = (memberOn(owners, name, ref, context, true, undefined, fits) ?? + memberOn(owners, `[${name}]`, ref, context, true, undefined, fits))?.node; + if (!member) return null; + // Of a `Shared` member and an instance one of the same name, a type's name reads the `Shared` one. + if (!member.isStatic) { + const found = member; + member = context.getNodesByQualifiedName(found.qualifiedName).find((n) => n.isStatic && n.filePath === found.filePath && fits(n)) ?? found; + } + if (member.kind === 'method') { + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...(namesWithoutRunning(ref, context) ? {} : { edgeKind: 'calls' as const }), + }; + } + return readThrough(member, owners, ref, context, { confidence: 0.9 }); +} + const VB_TYPE_IMPORTS = new WeakMap>>(); /** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ From a192df40f96cd3fee5bc86100e5555eeac9316a7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:06:42 +0000 Subject: [PATCH 210/363] fix(telemetry): count late index uploads, treat usage as live ingest, purge before catch-up (#2333) (#2356) Three bugs in the self-hosted telemetry services (telemetry-worker/ and telemetry-dashboard/), reported against 6560052a and all still on main. 1. An index run that uploads late never counted toward activation. The dashboard's funnel reads machine_first_seen.first_index_day, which only the nightly rollup wrote, and the rollup re-rolls just the last three days plus days with no daily_machines row. Ingest accepts timestamps up to 30 days old and the client keeps an event's original timestamp when it re-queues it, so an index event 4+ days late landed on a day nothing revisited and first_index_day stayed NULL. The ingest upsert of machine_first_seen now lowers first_index_day the way it lowers first_day: coalesce(min(old, new), old, new), because SQLite's min() is NULL if either side is. It is the same statement and row, with no new index and no migration (the column exists since 0002). The rollup still re-derives it from raw events, which covers events stored before this change. 2. ingest_stalled read only max(day) FROM events, but usage counters have lived in usage_daily since 0003, so a day with usage and no lifecycle events read as "The ingest worker ... is not storing anything". /api/meta now also reads max(day) FROM usage_daily (a primary-key lookup) and returns latest_usage_day and latest_ingest_day. Ingest counts as stalled only when there is no lifecycle event from yesterday or later and no usage counter from the day before yesterday or later, because clients upload a day's counters only after that day ends. The banner names latest_ingest_day. 3. runNightly rolled up the 3 regular days, then up to 31 missed days, and ran purgeOldEvents last. Each day first folds its legacy usage_rollup rows, which takes minutes on a heavy day, so a backlog could run past Cloudflare's 15-minute Cron Trigger limit and skip the purge and the summary line night after night. The purge now runs first: it is bounded and touches only days past the window, which no rollup reads. After that, rollups stop starting new days or fold chunks 10 minutes in (NIGHTLY_BUDGET_MS). A day stopped mid-fold gets no rollup at all, so it stays a missed day and the next night continues from its last committed chunk. The summary line counts those days as `deferred`, and `caught_up` now counts missed days actually rolled up. Verification: __tests__/telemetry-services.test.ts (new; the root vitest config picks it up) runs the worker's fetch/cron code and the dashboard API against the checked-in migrations in in-memory node:sqlite through a small D1-shaped adapter. With the source fix reversed, all 5 tests fail (activation 0, first_index_day null, ingest_stalled true, and "nightly rollup incomplete: 4 day(s) failed, purge failed" once statements pass the 15-minute mark). With the fix, all 5 pass. `npm run check` is clean in both packages. Against wrangler dev, run locally only: smoke:rollup 71/71, smoke:cutover 62/62, smoke:api 122/122 and render-check 87/87. smoke (ingest) is 46/47; the one failure is the 70 KB oversized-body curl argument hitting the Windows command-line limit, and main fails it the same way. The new smoke assertions fail on main. Takes effect only once both workers are redeployed; no migration. Co-authored-by: Claude Opus 5.5 --- __tests__/telemetry-services.test.ts | 438 ++++++++++++++++++++++ docs/design/telemetry.md | 6 +- telemetry-dashboard/README.md | 9 +- telemetry-dashboard/public/app.js | 3 +- telemetry-dashboard/scripts/fixture.sql | 1 + telemetry-dashboard/scripts/smoke-api.sh | 14 + telemetry-dashboard/src/api.ts | 51 ++- telemetry-worker/README.md | 32 +- telemetry-worker/scripts/smoke-cutover.sh | 11 +- telemetry-worker/scripts/smoke-ingest.sh | 8 + telemetry-worker/scripts/smoke-rollup.sh | 9 +- telemetry-worker/src/index.ts | 20 +- telemetry-worker/src/rollup.ts | 127 +++++-- 13 files changed, 662 insertions(+), 67 deletions(-) create mode 100644 __tests__/telemetry-services.test.ts diff --git a/__tests__/telemetry-services.test.ts b/__tests__/telemetry-services.test.ts new file mode 100644 index 0000000000..657681cc78 --- /dev/null +++ b/__tests__/telemetry-services.test.ts @@ -0,0 +1,438 @@ +/** + * The self-hosted telemetry services — `telemetry-worker/` (ingest + nightly rollup) + * and `telemetry-dashboard/` (the read API) — run against the checked-in D1 + * migrations in an in-memory node:sqlite database (#2333). + * + * D1 is SQLite, so every statement here is the SQL production runs. The one stand-in + * is a thin adapter giving node:sqlite D1's prepare/bind/first/all/run/batch shape, + * with `batch()` as one transaction the way D1 runs it. No wrangler, no workerd, no + * network: the end-to-end versions of these checks are the packages' own smoke + * suites (`npm run smoke`, `smoke:rollup`, `smoke:cutover`, `smoke:api`), which boot + * `wrangler dev`. + * + * npx vitest run __tests__/telemetry-services.test.ts + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import worker from '../telemetry-worker/src/index'; +import { runNightly } from '../telemetry-worker/src/rollup'; +import { handleApi } from '../telemetry-dashboard/src/api'; + +let nodeSqlite: typeof import('node:sqlite') | null = null; +try { + nodeSqlite = require('node:sqlite') as typeof import('node:sqlite'); +} catch { + /* Node < 22.5 — skipped below */ +} + +const MIGRATIONS = path.join(__dirname, '..', 'telemetry-worker', 'migrations'); +const MINUTE = 60_000; +/** Cloudflare ends a Cron Trigger invocation after 15 minutes of wall-clock time. */ +const CRON_LIMIT_MS = 15 * MINUTE; + +type Row = Record; + +// --------------------------------------------------------------------------- +// D1 over node:sqlite +// --------------------------------------------------------------------------- + +/** Binds integral numbers as INTEGER, as D1 does (node:sqlite would bind them as REAL). */ +const bindable = (v: unknown): unknown => (typeof v === 'number' && Number.isInteger(v) ? BigInt(v) : v); + +class SqliteD1Statement { + constructor( + private readonly d1: SqliteD1, + readonly sql: string, + private readonly params: unknown[] = [], + ) {} + + bind(...params: unknown[]): SqliteD1Statement { + return new SqliteD1Statement(this.d1, this.sql, params); + } + + execute(): { success: true; results: Row[]; meta: { changes: number } } { + this.d1.onStatement?.(this.sql); + const stmt = this.d1.db.prepare(this.sql); + const params = this.params.map(bindable) as never[]; + if (/^\s*(SELECT|WITH)\b/i.test(this.sql)) { + return { success: true, results: stmt.all(...params) as Row[], meta: { changes: 0 } }; + } + const info = stmt.run(...params); + return { success: true, results: [], meta: { changes: Number(info.changes) } }; + } + + async first(): Promise { + return (this.execute().results[0] as T | undefined) ?? null; + } + + async all(): Promise> { + return this.execute(); + } + + async run(): Promise> { + return this.execute(); + } +} + +class SqliteD1 { + /** Runs before every statement — the cron tests use it to model elapsed wall-clock time. */ + onStatement: ((sql: string) => void) | null = null; + + constructor(readonly db: import('node:sqlite').DatabaseSync) {} + + prepare(sql: string): SqliteD1Statement { + return new SqliteD1Statement(this, sql); + } + + async batch(statements: SqliteD1Statement[]): Promise[]> { + this.db.exec('BEGIN'); + try { + const results = statements.map((s) => s.execute()); + this.db.exec('COMMIT'); + return results; + } catch (err) { + this.db.exec('ROLLBACK'); + throw err; + } + } +} + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +describe.skipIf(!nodeSqlite)('telemetry services against the D1 schema (#2333)', () => { + let db: import('node:sqlite').DatabaseSync; + let d1: SqliteD1; + let env: never; + let logged: Row[]; + + const at = (iso: string): void => { + vi.setSystemTime(new Date(iso)); + }; + + const one = (sql: string, ...params: unknown[]): Row | undefined => + db.prepare(sql).get(...(params.map(bindable) as never[])) as Row | undefined; + + /** POST /v1/events the way a client does, then wait for the off-response-path write. */ + async function post(machineId: string, events: Row[]): Promise { + const body = JSON.stringify({ + machine_id: machineId, + codegraph_version: '1.6.2', + os: 'linux', + arch: 'x64', + node_major: 22, + ci: false, + schema_version: 2, + events, + }); + const pending: Promise[] = []; + const request = new Request('https://telemetry.test/v1/events', { + method: 'POST', + headers: { 'content-type': 'application/json', 'content-length': String(Buffer.byteLength(body)) }, + body, + }); + const ctx = { waitUntil: (p: Promise) => void pending.push(p), passThroughOnException: () => {} }; + const response = await worker.fetch(request as never, env, ctx as never); + await Promise.all(pending); + return response.status; + } + + /** GET one dashboard endpoint, as the signed-in page does. */ + async function api(pathAndQuery: string): Promise { + const result = await handleApi(env, new URL(`https://stats.test${pathAndQuery}`)); + expect(result.status ?? 200).toBe(200); + return result.body as Row; + } + + /** + * One cron invocation at `iso`. `chunkMs` is the wall-clock time each legacy-usage + * fold chunk is charged; a statement that starts after Cloudflare's 15-minute limit + * throws, the way the runtime ends the invocation there. + */ + async function nightly(iso: string, chunkMs = 0): Promise<{ latestStartMs: number; summary: Row | undefined }> { + at(iso); + const started = Date.now(); + let latestStartMs = 0; + d1.onStatement = (sql) => { + const elapsed = Date.now() - started; + if (elapsed > CRON_LIMIT_MS) { + throw new Error(`statement started ${elapsed / MINUTE} min into the cron run — past the 15-minute limit`); + } + latestStartMs = Math.max(latestStartMs, elapsed); + if (/INSERT INTO usage_daily/.test(sql) && /FROM events/.test(sql)) vi.setSystemTime(Date.now() + chunkMs); + }; + const before = logged.length; + try { + await runNightly(env, Date.parse(iso)); + } finally { + d1.onStatement = null; + } + const summary = logged.slice(before).find((line) => line.msg === 'nightly rollup'); + return { latestStartMs, summary }; + } + + beforeEach(() => { + vi.useFakeTimers({ toFake: ['Date'] }); + db = new nodeSqlite!.DatabaseSync(':memory:'); + for (const file of fs.readdirSync(MIGRATIONS).filter((f) => f.endsWith('.sql')).sort()) { + db.exec(fs.readFileSync(path.join(MIGRATIONS, file), 'utf8')); + } + d1 = new SqliteD1(db); + env = { + DB: d1, + RETENTION_DAYS: 90, + MACHINE_RATE_LIMITER: { limit: async () => ({ success: true }) }, + ADMIN_RATE_LIMITER: { limit: async () => ({ success: true }) }, + } as never; + logged = []; + const capture = (line: unknown): void => { + try { + logged.push(JSON.parse(String(line)) as Row); + } catch { + /* not one of the worker's JSON lines */ + } + }; + vi.spyOn(console, 'log').mockImplementation(capture); + vi.spyOn(console, 'error').mockImplementation(capture); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.useRealTimers(); + db.close(); + }); + + const M1 = '00000000-0000-4000-8000-000000000001'; + const M2 = '00000000-0000-4000-8000-000000000002'; + const M3 = '00000000-0000-4000-8000-000000000003'; + const install = (ts: string): Row => ({ event: 'install', ts, props: { scope: 'local', kind: 'fresh' } }); + const index = (ts: string): Row => ({ event: 'index', ts, props: { languages: ['typescript'] } }); + const usage = (day: string, count: number): Row => ({ + event: 'usage_rollup', + ts: `${day}T12:00:00.000Z`, + props: { kind: 'mcp_tool', name: 'codegraph_explore', count, client_name: 'Claude Code' }, + }); + + // ------------------------------------------------------------------------- + // 1. Activation counts an index run that uploads late + // ------------------------------------------------------------------------- + + describe('a late index upload', () => { + it('counts toward activation even when it arrives after the nightly run stopped revisiting its day', async () => { + // Sep 25: the machine installs. Its index run fails to upload and stays in the + // client's queue, keeping its original timestamp. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + await nightly('2026-09-26T00:30:00Z'); + + // Oct 3: the queued index run finally goes out — eight days late, inside the + // 30 days ingest accepts, but long past the three days each nightly run re-rolls. + at('2026-10-03T10:00:00Z'); + expect(await post(M1, [index('2026-09-25T14:05:00Z')])).toBe(204); + // Someone else's activity, so the rollup's coverage moves past the cohort's window. + expect(await post(M2, [install('2026-10-03T09:00:00Z')])).toBe(204); + await nightly('2026-10-04T00:30:00Z'); + + at('2026-10-04T08:00:00Z'); + const funnel = await api('/api/activation?from=2026-09-25&to=2026-09-25&window=7'); + expect(funnel).toMatchObject({ installs: 1, activated: 1, dropped: 0, covered_through: '2026-10-03' }); + expect(one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', M1)).toEqual({ + first_day: '2026-09-25', + first_index_day: '2026-09-25', + }); + }); + + it('is applied as it is stored, and only ever moves the first index day earlier', async () => { + const firstSeen = (machineId: string): Row | undefined => + one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', machineId); + + at('2026-10-03T10:00:00Z'); + // A batch without an index run says nothing about indexing. + expect(await post(M1, [install('2026-10-02T10:00:00Z')])).toBe(204); + expect(await post(M1, [usage('2026-10-01', 4)])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: null }); + + // The earliest index run in a batch wins, with no rollup involved. + expect(await post(M1, [index('2026-10-03T09:00:00Z'), index('2026-10-02T11:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + + // A later index run leaves it alone; a backdated one lowers it, and first_day with it. + expect(await post(M1, [index('2026-10-03T09:30:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + expect(await post(M1, [index('2026-09-20T08:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + + // A machine whose very first batch carries an index run. + expect(await post(M2, [install('2026-10-03T08:00:00Z'), index('2026-10-03T08:01:00Z')])).toBe(204); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + + // The nightly rollup agrees with what ingest already wrote. + await nightly('2026-10-04T00:30:00Z'); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + }); + }); + + // ------------------------------------------------------------------------- + // 2. Usage-only days are not a stalled ingest + // ------------------------------------------------------------------------- + + describe('the stalled-ingest flag', () => { + it('counts usage counters as ingest that is still storing', async () => { + // The last lifecycle event is Sep 25; after that only usage counters arrive. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + // Clients upload a day's counters only once it is over: Oct 3's arrive on Oct 4. + at('2026-10-04T06:00:00Z'); + expect(await post(M2, [usage('2026-10-03', 12)])).toBe(204); + + at('2026-10-04T08:00:00Z'); + const meta = await api('/api/meta'); + expect(meta).toMatchObject({ latest_raw_day: '2026-09-25', latest_ingest_day: '2026-10-03', ingest_stalled: false }); + + // Just past midnight nobody has sent Oct 4's counters yet; that is not a stall. + at('2026-10-05T00:10:00Z'); + expect(await api('/api/meta')).toMatchObject({ ingest_stalled: false }); + + // A whole day later with nothing stored at all, it is — and the banner date is + // the last day anything was stored, not the last lifecycle event. + at('2026-10-06T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: true }); + }); + + it('still reports a stall when nothing at all has arrived, and not on an empty database', async () => { + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: null, ingest_stalled: false }); + + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-09-25', ingest_stalled: true }); + + // A lifecycle event from yesterday is fresh, as before. + expect(await post(M1, [index('2026-10-03T22:00:00Z')])).toBe(204); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: false }); + }); + }); + + // ------------------------------------------------------------------------- + // 3. A long catch-up stays inside the cron's wall-clock limit, and the purge runs + // ------------------------------------------------------------------------- + + describe('the nightly run under a catch-up backlog', () => { + /** The rollup folds legacy usage rows 50,000 per transaction (rollup.ts, LEGACY_CHUNK_ROWS). */ + const CHUNK_ROWS = 50_000; + /** + * Each fold chunk is charged 4 minutes, so a backlog of legacy-heavy days costs + * several minutes apiece, as it did in production. The six missed days below hold + * seven chunks: 28 minutes of folding, against a 15-minute limit. + */ + const CHUNK_MS = 4 * MINUTE; + const NIGHT_1 = '2026-10-05T00:30:00Z'; + /** Missed days, newest first — the order catch-up takes them in. Sep 21 needs two chunks. */ + const MISSED = ['2026-09-23', '2026-09-22', '2026-09-21', '2026-09-20', '2026-09-19', '2026-09-18']; + const LEGACY_ROWS: Record = Object.fromEntries( + MISSED.map((day) => [day, day === '2026-09-21' ? CHUNK_ROWS + 3 : 3]), + ); + const PAST_WINDOW = '2026-06-01'; + + /** Legacy rows as they were stored before migrations/0003: one `events` row per upload, count 1. */ + function seedLegacyDay(day: string, rows: number): void { + db.prepare( + `WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ?1) + INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, + node_major, ci, schema_version, props) + SELECT ?2 || 'T13:00:00.000Z', ?2 || 'T12:00:00.000Z', ?2, 'usage_rollup', + '00000000-0000-4000-8000-00000000010' || ((i - 1) % 5), '1.5.0', 'linux', 'x64', 22, 0, 1, + '{"kind":"mcp_tool","name":"codegraph_explore","count":1,"error_count":0,"client_name":"Claude Code"}' + FROM n`, + ).run(BigInt(rows), day); + for (let m = 0; m < Math.min(rows, 5); m++) { + const machine = `00000000-0000-4000-8000-00000000010${m}`; + db.prepare('INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, 1)').run(machine, day); + db.prepare( + `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) + ON CONFLICT (machine_id) DO UPDATE SET first_day = min(first_day, excluded.first_day)`, + ).run(machine, day); + } + } + + /** Where each seeded usage count sits right now, and whether the day has been rolled up. */ + function ledger(day: string) { + const folded = Number(one('SELECT coalesce(sum(count), 0) AS n FROM usage_daily WHERE day = ?', day)?.n); + const unfolded = Number( + one( + `SELECT coalesce(sum(json_extract(props, '$.count')), 0) AS n FROM events + WHERE day = ? AND event = 'usage_rollup'`, + day, + )?.n, + ); + const rolledUsage = one(`SELECT count FROM daily_event_counts WHERE day = ? AND event = 'usage_rollup'`, day); + const rolledUp = one('SELECT machines FROM daily_machines WHERE day = ?', day) !== undefined; + return { folded, unfolded, rolledUsage: rolledUsage ? Number(rolledUsage.count) : null, rolledUp }; + } + + function expectConsistent(): void { + for (const day of MISSED) { + const l = ledger(day); + // Nothing lost and nothing counted twice, however far the fold got. + expect(l.folded + l.unfolded, day).toBe(LEGACY_ROWS[day]); + if (l.rolledUp) { + expect(l, day).toMatchObject({ unfolded: 0, rolledUsage: LEGACY_ROWS[day] }); + } else { + // A day whose fold did not finish gets no rollup at all, so it stays "missed". + expect(l.rolledUsage, day).toBeNull(); + } + } + } + + beforeEach(() => { + for (const day of MISSED) seedLegacyDay(day, LEGACY_ROWS[day]!); + // Past the 90-day window: the purge must take these. + db.prepare( + `INSERT INTO events (received_at, ts, day, event, machine_id, props) + VALUES ('2026-06-01T10:00:00.000Z', NULL, ?1, 'install', ?2, '{}')`, + ).run(PAST_WINDOW, M3); + db.prepare( + `INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES (?, ?, 'cli_command', 'index', 2)`, + ).run(PAST_WINDOW, M3); + }); + + it('purges on schedule, stops before the limit, and resumes the backlog the next nights', async () => { + // The three days the run re-rolls every night have ordinary, current traffic. + at('2026-10-04T12:00:00Z'); + expect(await post(M1, [install('2026-10-04T11:00:00Z'), usage('2026-10-03', 5)])).toBe(204); + expect(await post(M2, [index('2026-10-02T11:00:00Z')])).toBe(204); + + const first = await nightly(NIGHT_1, CHUNK_MS); + expect(first.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + // The purge ran, and the summary line was written. + expect(one('SELECT count(*) AS n FROM events WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(one('SELECT count(*) AS n FROM usage_daily WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(first.summary).toMatchObject({ purged: 1, usage_purged: 1, failed: 0 }); + // Tonight's regular days were rolled up first. + for (const day of ['2026-10-04', '2026-10-03', '2026-10-02']) { + expect(one('SELECT machines FROM daily_machines WHERE day = ?', day), day).toEqual({ machines: 1 }); + } + // The backlog did not fit: Sep 21 was left part-folded and un-rolled, the rest untouched. + expect(ledger('2026-09-21')).toMatchObject({ folded: CHUNK_ROWS, unfolded: 3, rolledUp: false }); + expect(MISSED.filter((day) => !ledger(day).rolledUp)).toEqual(MISSED.slice(2)); + expect(first.summary).toMatchObject({ caught_up: 2, deferred: 4 }); + expectConsistent(); + + // The next nights pick up exactly where this one stopped. + let nights = 1; + for (let day = 6; MISSED.some((d) => !ledger(d).rolledUp) && nights < 5; day++, nights++) { + const night = await nightly(`2026-10-0${day}T00:30:00Z`, CHUNK_MS); + expect(night.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + expect(night.summary).toMatchObject({ failed: 0 }); + expectConsistent(); + } + expect(nights).toBe(3); + for (const day of MISSED) expect(ledger(day), day).toMatchObject({ unfolded: 0, rolledUp: true }); + }); + }); +}); diff --git a/docs/design/telemetry.md b/docs/design/telemetry.md index 530600d93c..bf5deed9cc 100644 --- a/docs/design/telemetry.md +++ b/docs/design/telemetry.md @@ -232,9 +232,9 @@ Full documentation is [`telemetry-dashboard/README.md`](../../telemetry-dashboar `telemetry-worker/scripts/smoke-cutover.sh` exists to cover — a mismatch there is silent, showing up as a panel that reads zero forever rather than as an error. - **Reads rollups, not raw events**, so a chart stays correct for days whose raw rows have - been purged. `/api/activation` is the one exception — "did this machine ever run an index" - is not a daily aggregate — so it reads raw `events` and is bounded by the retention window, - which it reports as `raw_events_from`. + been purged. The activation funnel's "did this machine ever run an index" is not a daily + aggregate, so it reads `machine_first_seen.first_index_day` instead, which the ingest Worker + lowers as each index event is stored and the nightly cron re-derives from raw `events`. - **Auth is a shared password and a signed cookie**, sized for exactly two people: `ADMIN_PASSWORD` + `SESSION_SECRET` as Worker secrets, constant-time compare, HMAC-signed cookie with no session store, everything except `/login` and `robots.txt` gated. Rotating diff --git a/telemetry-dashboard/README.md b/telemetry-dashboard/README.md index 26feaac2df..10c1fb3fc2 100644 --- a/telemetry-dashboard/README.md +++ b/telemetry-dashboard/README.md @@ -40,7 +40,7 @@ renders. Bad input is a `400` with a message, never a guess. Chart data carries | Endpoint | Answers | |---|---| -| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day an event arrived, machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | +| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day ingest stored anything (`latest_ingest_day` — a lifecycle event or a usage counter), machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | | `/api/summary` | Big numbers: production users, active machines, new machines, installs, uninstalls, indexing runs, tool calls. | | `/api/timeseries?metric=` | `installs_uninstalls`, `new_installs`, `production_users`, `indexing_activity`, `tool_calls`, `duration_buckets`. One dense point per day — a day with nothing is a zero, not a gap. Days after the rollup's last day are `null` ("not counted yet") and `covered_through` says where that is; `new_installs` is written live and runs through today. | | `/api/breakdown?dim=` | `os`, `arch`, `codegraph_version`, `node_major`, `language`, `file_count_bucket`, `duration_bucket`, `target`, `scope`, `kind`, `name`, `client_name`, `name_error`. Optional `&event=`, `&metric=count\|machines`, `&limit=`. | @@ -53,7 +53,8 @@ kept forever, so a chart stays correct for days whose raw events have been purge panel reads raw `events`.** D1 runs one query at a time per database: the activation funnel used to join every cohort machine against `events`, which took ~55 s for one week of cohorts in production and failed every panel queued behind it. It now reads -`machine_first_seen.first_index_day`, which the nightly rollup maintains. +`machine_first_seen.first_index_day`, which the ingest worker lowers as each index event is +stored and the nightly rollup re-derives from raw events. The presets end on **today** (UTC — every event and rollup is keyed on the UTC day). Live numbers — production users, new machines, retention — run through today; rolled-up ones @@ -138,6 +139,10 @@ npm run smoke:render # the panels, in a browser (79 assertions) Each suite starts its own throwaway `wrangler dev` on its own port and cleans up after itself, so they can be run in any order (`DASH_PORT` overrides the port). +The repo's own test suite also runs `src/api.ts`, with the ingest worker feeding it, against +the writer's migrations in an in-memory SQLite database — no wrangler needed. From the repo +root (it is part of `npm test` there): `npx vitest run __tests__/telemetry-services.test.ts`. + **`smoke-auth.sh`** is the regression net for the gate: unauthenticated requests reach nothing (pages, API *and* static assets), the cookie is persistent and correctly flagged, flipped/truncated/forged cookies are all rejected, brute force is capped, and rotating the diff --git a/telemetry-dashboard/public/app.js b/telemetry-dashboard/public/app.js index b29506aada..abb2f46efa 100644 --- a/telemetry-dashboard/public/app.js +++ b/telemetry-dashboard/public/app.js @@ -408,7 +408,8 @@ function drawDataWarning() { if (meta?.ingest_stalled) { lines.push([ - `No new events since ${shortDay(meta.latest_raw_day)}.`, + // The last day anything was stored, lifecycle event or usage counter. + `No new events since ${shortDay(meta.latest_ingest_day ?? meta.latest_raw_day)}.`, ' The ingest worker at telemetry.getcodegraph.com is not storing anything; its logs and the D1 database are where to look.', ]); } diff --git a/telemetry-dashboard/scripts/fixture.sql b/telemetry-dashboard/scripts/fixture.sql index 9e2ddef020..e9ca7c151d 100644 --- a/telemetry-dashboard/scripts/fixture.sql +++ b/telemetry-dashboard/scripts/fixture.sql @@ -36,6 +36,7 @@ DELETE FROM daily_machines; DELETE FROM machine_days; DELETE FROM machine_first_seen; DELETE FROM events; +DELETE FROM usage_daily; -- --------------------------------------------------------------------------- -- install — 12, one per machine on its first day diff --git a/telemetry-dashboard/scripts/smoke-api.sh b/telemetry-dashboard/scripts/smoke-api.sh index 67008c47b4..8dda68403b 100755 --- a/telemetry-dashboard/scripts/smoke-api.sh +++ b/telemetry-dashboard/scripts/smoke-api.sh @@ -105,6 +105,7 @@ field "retention window" retention_days 14 "$META" # The fixture stops on 07-10, months ago: nothing has arrived since, but the # rollup did cover every day that saw activity, so it is not behind. field "ingest stalled (no events since 07-10)" ingest_stalled true "$META" +field "…named by the last day anything was stored" latest_ingest_day 2026-07-10 "$META" field "rollup not behind (it covered every active day)" rollup_behind false "$META" field "nobody active yesterday" machines_yesterday 0 "$META" @@ -307,6 +308,19 @@ field "…and yesterday's machine is counted" machines_yesterday 1 "$META" npx wrangler d1 execute codegraph-telemetry --local \ --command "DELETE FROM machine_days WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 +echo +echo "Usage counters alone are not a stalled ingest" +# Usage lands in usage_daily, not events, so a day of tool calls with no install or +# index run must not read as "not storing anything". Removed again straight after. +npx wrangler d1 execute codegraph-telemetry --local \ + --command "INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES ('$YESTERDAY', '$STALE_ID', 'mcp_tool', 'codegraph_explore', 3)" >/dev/null 2>&1 +META="$(get "/api/meta")" +field "not stalled while usage arrives" ingest_stalled false "$META" +field "…the usage day is the last day stored" latest_ingest_day "$YESTERDAY" "$META" +field "…though the last lifecycle event is 07-10" latest_raw_day 2026-07-10 "$META" +npx wrangler d1 execute codegraph-telemetry --local \ + --command "DELETE FROM usage_daily WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 + echo printf '%d passed, %d failed\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] diff --git a/telemetry-dashboard/src/api.ts b/telemetry-dashboard/src/api.ts index 63b77556c6..24deb33522 100644 --- a/telemetry-dashboard/src/api.ts +++ b/telemetry-dashboard/src/api.ts @@ -6,8 +6,8 @@ * - **Rollups only.** Every panel is answered from `daily_*`, `machine_days` and * `machine_first_seen`, which are kept forever. No panel reads raw `events`: * D1 runs one query at a time per database, so one slow scan there fails every - * panel queued behind it. (/api/meta reads the table's first and last day, one - * indexed lookup each.) + * panel queued behind it. (/api/meta reads the table's first and last day, and + * `usage_daily`'s last, one indexed lookup each.) * - **Today is in range; uncounted days are not zeros.** Rolled-up numbers stop * at the nightly rollup's last day and come back null after it, so a chart * ending today draws a gap where the count has not happened yet, not a cliff. @@ -214,6 +214,7 @@ interface MetaRow { earliest_active_day: string | null; earliest_raw_day: string | null; latest_raw_day: string | null; + latest_usage_day: string | null; machines_yesterday: number | null; } @@ -246,6 +247,7 @@ async function meta(env: Env): Promise { (SELECT min(day) FROM machine_days) AS earliest_active_day, (SELECT min(day) FROM events) AS earliest_raw_day, (SELECT max(day) FROM events) AS latest_raw_day, + (SELECT max(day) FROM usage_daily) AS latest_usage_day, (SELECT count(*) FROM machine_days WHERE day = ?) AS machines_yesterday`, ) .bind(yesterday) @@ -253,11 +255,22 @@ async function meta(env: Env): Promise { const latestRollup = row?.latest_rollup_day ?? null; const latestRaw = row?.latest_raw_day ?? null; + const latestUsage = row?.latest_usage_day ?? null; const latestActive = row?.latest_active_day ?? null; - - // Nothing at all since before yesterday. (Client clocks may run a few minutes - // ahead, so the latest day can be tomorrow — that is fresh, not stale.) - const ingestStalled = latestRaw !== null && latestRaw < yesterday; + // Ingest stores lifecycle events in `events` and usage counters in `usage_daily`, so + // either one arriving means it is working — a day of usage and no installs or index + // runs is quiet, not stalled. + const latestIngest = + latestUsage !== null && (latestRaw === null || latestUsage > latestRaw) ? latestUsage : latestRaw; + + // Stalled: nothing stored since before yesterday. (Client clocks may run a few + // minutes ahead, so the latest day can be tomorrow — that is fresh, not stale.) Usage + // counters get a day more, because a client uploads a day's counters only once that + // day is over: just after midnight UTC the newest one can be the day before yesterday. + const ingestStalled = + latestIngest !== null && + (latestRaw === null || latestRaw < yesterday) && + (latestUsage === null || latestUsage < addDays(today, -2)); // The 00:30 UTC run rolls up yesterday, so the day before that must always be in // by now. Only "behind" if there was activity after the last rolled-up day — a day // nobody used codegraph would be a silent rollup, not a missed one. @@ -274,7 +287,12 @@ async function meta(env: Env): Promise { latest_rollup_day: latestRollup, latest_active_day: latestActive, earliest_raw_day: row?.earliest_raw_day ?? null, + /** The last day of a stored lifecycle event (install, index, uninstall). */ latest_raw_day: latestRaw, + /** The last day of a stored usage counter. */ + latest_usage_day: latestUsage, + /** The later of the two: the last day ingest stored anything, which the stall banner names. */ + latest_ingest_day: latestIngest, machines_yesterday: row?.machines_yesterday ?? 0, rollup_behind: rollupBehind, ingest_stalled: ingestStalled, @@ -670,17 +688,18 @@ interface ActivationRow { * reinstalls does not re-enter the funnel, which is what makes this a * conversion rate rather than an install-event ratio. * - * "Ran an index" is `machine_first_seen.first_index_day`, which the nightly rollup - * keeps at the earliest day each machine indexed. That makes this a range read over - * one small table. It used to be a join against raw `events` — on production volume - * ~55 s per week of cohorts, which held D1's single query lane long enough to fail - * every other panel waiting behind it. A first index day is never before the first - * day (the ingest path keeps first_day at the machine's earliest event), so "within - * the window" is just `first_index_day <= first_day + window`. + * "Ran an index" is `machine_first_seen.first_index_day`, the earliest day each + * machine indexed. The ingest worker lowers it as each index event is stored — so a + * run that uploads days late still counts — and the nightly rollup re-derives it from + * raw events. That makes this a range read over one small table. It used to be a + * join against raw `events` — on production volume ~55 s per week of cohorts, which + * held D1's single query lane long enough to fail every other panel waiting behind + * it. A first index day is never before the first day (the ingest path keeps + * first_day at the machine's earliest event), so "within the window" is just + * `first_index_day <= first_day + window`. * - * Because first_index_day is rolled up, cohorts after the rollup's last day have no - * conversions counted yet. They are left out of the totals and drawn as gaps — - * counting them would show a drop in conversion that is only a lag. + * Cohorts after the rollup's last day are left out of the totals and drawn as gaps, + * like every other rolled-up number on the page. */ async function activation(env: Env, url: URL, range: Range): Promise { const rawWindow = url.searchParams.get('window'); diff --git a/telemetry-worker/README.md b/telemetry-worker/README.md index 296ec2251d..077d0812f0 100644 --- a/telemetry-worker/README.md +++ b/telemetry-worker/README.md @@ -80,7 +80,8 @@ nothing looks broken from the outside. That ran from 2026-08-11 to October 2026. ## Rollups & retention (nightly cron) -`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things. +`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things, the purge +first. **Rolls up** the day that just ended into `daily_machines`, `daily_event_counts`, `daily_dim_counts` and `machine_first_seen.first_index_day`, then re-runs the two days before @@ -88,7 +89,12 @@ it — offline clients ship completed-day rollups late, so a day keeps growing a It then **catches up** on any earlier day that saw activity but never got a rollup (a `machine_days` day with no `daily_machines` row — a night the run failed or the database refused writes), newest first, up to 31 a night. An outage heals on the first good night -instead of leaving a hole someone has to notice. The aggregation is one +instead of leaving a hole someone has to notice. Cloudflare ends a Cron Trigger after 15 +minutes of wall-clock time, and a day that still holds millions of legacy usage rows (below) +takes minutes, so the run starts no new day and no new fold chunk once 10 minutes have +passed (`NIGHTLY_BUDGET_MS`). A day it stops partway gets no rollup at all, so it is still a +missed day the next night, which carries on from the last folded chunk; the summary line +counts those days as `deferred`. The aggregation is one `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and no event row crosses the wire. Every write overwrites the recomputed value rather than adding to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful @@ -100,7 +106,9 @@ rollup over old days is the whole migration. Adding a breakdown is a line in `RO never a migration — that is what the generic `(dim, value)` shape buys. **Purges** raw `events` and `usage_daily` rows older than `RETENTION_DAYS` (90, a var in -`wrangler.jsonc`) in bounded `DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are +`wrangler.jsonc`) in bounded `DELETE` batches, before any rollup: behind a long catch-up it +would be the part the 15-minute limit cuts off, night after night, while the database +grows. The run ends with one line of counts. `machine_days` and `machine_first_seen` are never purged — retention cohorts need the full history and they are two orders of magnitude smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a chart. @@ -122,10 +130,13 @@ each day is a full scan of that day's events, and the request has a wall-clock b ### Backfilling first_index_day -`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads, and only -the rollup writes it. Days rolled up before the migration left it NULL, and days stored before +`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads. The +ingest worker lowers it as each index event is stored, so an index run that uploads days late +still counts, and the rollup re-derives it from the raw events of every day it rolls up. Days +rolled up before the migration left it NULL, an index event stored before the ingest worker +began setting it (#2333) counts only once its day is rolled up again, and days stored before `0003` still hold their usage as one `events` row per upload. Re-running the rollup over every -day that still has raw events fixes both — it folds that day's legacy usage rows into +day that still has raw events fixes all three — it folds that day's legacy usage rows into `usage_daily`, sets `first_index_day`, and recomputes the day's rollups. It is idempotent, so overlapping or repeating a range is harmless. A day with millions of legacy rows takes several minutes, so go a day at a time: @@ -300,6 +311,15 @@ To drive the cron body by hand, run `wrangler dev --test-scheduled` and hit `.dev.vars.example` to `.dev.vars` — without an `ADMIN_TOKEN` the route 404s, exactly as a deploy that never set the secret does. +The repo's own test suite also covers this worker and the dashboard API without wrangler: +it runs their source against these migrations in an in-memory SQLite database, including +late uploads and a nightly run whose catch-up outlasts the cron's time limit. From the repo +root (it is part of `npm test` there): + +```bash +npx vitest run __tests__/telemetry-services.test.ts +``` + ## Changing the schema The allowlist in `src/index.ts` mirrors `docs/design/telemetry.md` (and the user-facing diff --git a/telemetry-worker/scripts/smoke-cutover.sh b/telemetry-worker/scripts/smoke-cutover.sh index f0dff12443..6d55f8c79a 100755 --- a/telemetry-worker/scripts/smoke-cutover.sh +++ b/telemetry-worker/scripts/smoke-cutover.sh @@ -235,11 +235,14 @@ ts "tool calls (sums the prop)" tool_calls '[20]' '[2]' MET=$(api "meta") is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)" +# Usage lands in usage_daily rather than events; the stalled-ingest check reads both. +is "meta reports the last day ingest stored" "$DAY" "$(jget "$MET" latest_ingest_day)" -# The funnel reads machine_first_seen.first_index_day, which only the nightly -# rollup writes (from raw `index` events). This is the seam that pins it: ingest -# writes the events, the rollup sets the column, the dashboard reads it. If the -# rollup stopped setting it, "activated" here would read 0. +# The funnel reads machine_first_seen.first_index_day. The ingest worker sets it as +# each index event is stored, and the nightly rollup re-derives it from raw `index` +# events (smoke-rollup.sh pins that half, on events it seeds past the ingest path). +# This is the seam: ingest writes the column, the dashboard reads it. If neither +# writer set it, "activated" here would read 0. # # Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine # that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never diff --git a/telemetry-worker/scripts/smoke-ingest.sh b/telemetry-worker/scripts/smoke-ingest.sh index 458123443a..9fd2b87fe4 100755 --- a/telemetry-worker/scripts/smoke-ingest.sh +++ b/telemetry-worker/scripts/smoke-ingest.sh @@ -214,6 +214,14 @@ is "machine_days: each backdated batch gets its own day" "$BACK_DAY,$RECENT_DAY" is "machine_first_seen recorded" "$RECENT_DAY" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" is "machine_first_seen only moves earlier" "$BACK_DAY" \ "$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")" +# The activation funnel's input is kept here too, not only by the nightly rollup: an +# index run that uploads after the cron stopped re-rolling its day must still count. +is "first_index_day recorded as the index run is stored" "$RECENT_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_OK'")" +is "first_index_day only moves earlier, like first_day" "$BACK_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_BACK'")" +is "a machine that never sent an index run has none" "null" \ + "$(q "select coalesce(first_index_day, 'null') from machine_first_seen where machine_id='$M_USE'")" echo echo "usage counters add up instead of piling up" diff --git a/telemetry-worker/scripts/smoke-rollup.sh b/telemetry-worker/scripts/smoke-rollup.sh index 294b4378f8..2befab4eb8 100755 --- a/telemetry-worker/scripts/smoke-rollup.sh +++ b/telemetry-worker/scripts/smoke-rollup.sh @@ -16,7 +16,8 @@ # * the purge deletes only rows past the window, and leaves machine_days / # machine_first_seen alone # * each machine's first_index_day is its earliest index, and survives the purge -# * the cron catches up on a day that saw activity but was never rolled up +# * the cron catches up on a day that saw activity but was never rolled up, and +# logs its one summary line # * usage comes from usage_daily; a legacy usage row still in `events` is folded # into it (counts added, row deleted) before the day is rolled up # * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one @@ -196,6 +197,12 @@ is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")" # rolled up), and purges everything past the window. is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")" sleep 2 +# Its one summary line: written once the purge and every rollup are done, and with +# nothing this small left for the next night by the time budget. +is "the cron logs its summary, nothing failed or deferred" "0/0" \ + "$(grep -ao '{"msg":"nightly rollup".*}' "$LOG" | tail -1 | + node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{ + const r=JSON.parse(s||"{}");console.log(`${r.failed}/${r.deferred}`);})')" # Rolling a purged day with reset=1 must NOT blank the rollups it already has: past # the window the reset is ignored, so the delete-then-rebuild can't find zero events. diff --git a/telemetry-worker/src/index.ts b/telemetry-worker/src/index.ts index 9a6efa3394..3c4ac7cde3 100644 --- a/telemetry-worker/src/index.ts +++ b/telemetry-worker/src/index.ts @@ -200,9 +200,18 @@ const INSERT_EVENT = `INSERT INTO events ( const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, ?) ON CONFLICT (machine_id, day) DO UPDATE SET prod = max(machine_days.prod, excluded.prod)`; -// A late-arriving offline buffer can move a machine's first day earlier, never later. -const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) - ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`; +// A late-arriving offline buffer can move a machine's first day earlier, never later — +// and its first index day (the activation funnel's input) the same way. That one is set +// here as well as by the nightly rollup because the rollup only re-rolls the last three +// days, while this path accepts events up to 30 days old: an index run that uploads a +// week late would otherwise never count. SQLite's multi-argument min() is NULL when +// either side is, hence the coalesce: a batch with no index run leaves the stored day +// alone, and the first index run on record sets it. +const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day, first_index_day) VALUES (?, ?, ?) + ON CONFLICT (machine_id) DO UPDATE SET + first_day = min(machine_first_seen.first_day, excluded.first_day), + first_index_day = coalesce(min(machine_first_seen.first_index_day, excluded.first_index_day), + machine_first_seen.first_index_day, excluded.first_index_day)`; // usage_rollup counters ADD into one row per machine × day × tool (migrations/0003): // clients upload the same counter many times over — once per process — so storing @@ -305,10 +314,13 @@ async function writeToD1( } } + // The earliest day this batch indexed on, if it did — see UPSERT_FIRST_SEEN. + let firstIndexDay: string | null = null; for (const e of batch) { if (e.event === 'usage_rollup') continue; const day = (e.ts ?? receivedAt).slice(0, 10); days.add(day); + if (e.event === 'index' && (firstIndexDay === null || day < firstIndexDay)) firstIndexDay = day; stmts.push( insertEvent.bind( receivedAt, @@ -328,7 +340,7 @@ async function writeToD1( const firstDay = [...days].sort()[0]; if (firstDay !== undefined) { - stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay)); + stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay, firstIndexDay)); } await env.DB.batch(stmts); diff --git a/telemetry-worker/src/rollup.ts b/telemetry-worker/src/rollup.ts index 87072a59a9..9502210984 100644 --- a/telemetry-worker/src/rollup.ts +++ b/telemetry-worker/src/rollup.ts @@ -9,12 +9,13 @@ * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`, * `daily_dim_counts` and `machine_first_seen.first_index_day` — plus the two days * before it, because clients buffer offline and ship completed-day rollups late, so - * a day keeps growing after it ends, plus any earlier day a failed run missed. Every - * write is an upsert that OVERWRITES the recomputed value rather than adding to it - * (or, for first_index_day, only ever lowers it), so re-running a day is a no-op - * and never double-counts. + * a day keeps growing after it ends, plus any earlier day a failed run missed, as + * many as fit in the run's time budget. Every write is an upsert that OVERWRITES + * the recomputed value rather than adding to it (or, for first_index_day, only ever + * lowers it), so re-running a day is a no-op and never double-counts. * - * 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are + * 2. PURGE raw `events` past the retention window, in bounded batches — first, so a + * long catch-up can never push it past the cron's wall-clock limit. Rollups are * kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and * `machine_first_seen` are never purged, because retention cohorts need the full * history and they are two orders of magnitude smaller than the raw rows. @@ -32,6 +33,15 @@ export const ROLLUP_LOOKBACK_DAYS = 3; export const MAX_MANUAL_DAYS = 31; /** Most missed days one nightly run catches up on, newest first; the next night takes the rest. */ export const MAX_CATCHUP_DAYS = 31; +/** + * How long into a nightly run it still starts rollup work. Cloudflare ends a Cron + * Trigger after 15 minutes of wall-clock time, and catching up a day that still holds + * millions of legacy usage rows takes minutes, so a backlog can run straight into that + * limit. Past this point the run starts no new day and no new fold chunk; the 5 + * minutes left cover the one statement already in flight and the summary line. What + * did not fit stays a missed day, so the next night carries on from there. + */ +export const NIGHTLY_BUDGET_MS = 10 * 60_000; /** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */ const PURGE_BATCH_ROWS = 5_000; @@ -148,6 +158,10 @@ const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines * it straight off `machine_first_seen`, which keeps the funnel off raw `events` (a * cohort join there took most of a minute per week of cohorts) and past the purge. * + * The ingest path lowers it the same way as each index event is stored (src/index.ts), + * because a run that uploads late lands on a day this cron has stopped re-rolling. This + * statement is what fills it for events stored before that, and what a backfill re-runs. + * * Only rows that actually move are written, so re-running a day is free. `?1` is the * day — still the one bound parameter, used three times. */ @@ -221,6 +235,11 @@ export interface DayResult { rows: number; /** Day is past the retention window — a `reset` on it is ignored (see below). */ pastRetention: boolean; + /** + * The deadline passed before the day's legacy usage rows were all folded, so nothing + * was rolled up: the next run finishes the fold and rolls the day up then. + */ + deferred: boolean; } /** @@ -236,14 +255,20 @@ export interface DayResult { * exists would otherwise linger. It is IGNORED past the retention window, where it * would delete rows and then find no events to rebuild them from: silently blanking a * real day is the one irreversible thing this file could do. + * + * `deadline` (epoch ms) bounds the legacy fold that runs first. If it stops the fold + * partway, the day is not rolled up at all — usage counted from a half-folded day would + * be short, and its `daily_machines` row would stop the nightly run from ever coming + * back to it. The chunks already folded are committed, so the next call carries on. */ export async function rollupDay( env: Env, day: string, - opts: { cutoff: string; reset?: boolean }, + opts: { cutoff: string; reset?: boolean; deadline?: number }, ): Promise { const pastRetention = day < opts.cutoff; - await foldLegacyUsage(env, day); + const fold = await foldLegacyUsage(env, day, opts.deadline); + if (!fold.complete) return { day, rows: 0, pastRetention, deferred: true }; const statements: D1PreparedStatement[] = []; if (opts.reset && !pastRetention) { @@ -257,7 +282,7 @@ export async function rollupDay( const results = await env.DB.batch(statements); const rows = results.reduce((total, r) => total + (r.meta?.changes ?? 0), 0); - return { day, rows, pastRetention }; + return { day, rows, pastRetention, deferred: false }; } // --------------------------------------------------------------------------- @@ -298,15 +323,32 @@ const LEGACY_FOLD = `INSERT INTO usage_daily ( const LEGACY_DELETE = `DELETE FROM events WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2`; -/** Folds any legacy usage rows for `day` into usage_daily. Returns how many rows it moved. */ -export async function foldLegacyUsage(env: Env, day: string): Promise { +export interface FoldResult { + /** Legacy rows moved into usage_daily by this call. */ + moved: number; + /** False when the deadline stopped it with rows still to fold; the next call carries on. */ + complete: boolean; +} + +/** + * Folds any legacy usage rows for `day` into usage_daily, a chunk per transaction. + * With a `deadline` (epoch ms) it starts no chunk once that has passed: each chunk + * commits on its own and always takes the oldest rows left, so stopping between two + * loses nothing and the next call simply continues. + */ +export async function foldLegacyUsage( + env: Env, + day: string, + deadline = Number.POSITIVE_INFINITY, +): Promise { const any = await env.DB.prepare(`SELECT 1 AS hit FROM events WHERE day = ? AND event = 'usage_rollup' LIMIT 1`) .bind(day) .first<{ hit: number }>(); - if (!any) return 0; + if (!any) return { moved: 0, complete: true }; let moved = 0; for (let chunk = 0; chunk < LEGACY_MAX_CHUNKS; chunk++) { + if (Date.now() >= deadline) return { moved, complete: false }; // Always the oldest remaining rows: everything below the next chunk's first id. // Rows folded by earlier chunks are gone, so there is no cursor to carry. const next = await env.DB.prepare( @@ -320,7 +362,7 @@ export async function foldLegacyUsage(env: Env, day: string): Promise { env.DB.prepare(LEGACY_DELETE).bind(day, below), ]); moved += deleted?.meta?.changes ?? 0; - if (!next) return moved; + if (!next) return { moved, complete: true }; } throw new Error(`legacy usage fold for ${day} did not finish within ${LEGACY_MAX_CHUNKS} chunks`); } @@ -399,56 +441,81 @@ async function missedDays(env: Env, cutoff: string, before: string): Promise { const started = Date.now(); + const deadline = started + NIGHTLY_BUDGET_MS; const keepDays = retentionDays(env); const cutoff = retentionCutoff(atMs, keepDays); - const days: string[] = []; - for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) days.push(utcDay(atMs - back * DAY_MS)); + let purge: PurgeResult | null = null; + try { + purge = await purgeOldEvents(env, cutoff); + } catch (err) { + console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); + } + + const regular: string[] = []; + for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) regular.push(utcDay(atMs - back * DAY_MS)); // A failure to find the missed days must not cost tonight's regular rollup. - let caughtUp = 0; + let missed: string[] = []; try { - const missed = await missedDays(env, cutoff, days[days.length - 1] ?? utcDay(atMs)); - days.push(...missed); - caughtUp = missed.length; + missed = await missedDays(env, cutoff, regular[regular.length - 1] ?? utcDay(atMs)); } catch (err) { console.error(JSON.stringify({ msg: 'missed-day scan failed', err: String(err) })); } const rolled: string[] = []; const failed: string[] = []; + const deferred: string[] = []; + let caughtUp = 0; let rows = 0; - for (const day of days) { + for (const day of [...regular, ...missed]) { + if (Date.now() >= deadline) { + deferred.push(day); + continue; + } try { - rows += (await rollupDay(env, day, { cutoff })).rows; + const result = await rollupDay(env, day, { cutoff, deadline }); + if (result.deferred) { + deferred.push(day); + continue; + } + rows += result.rows; rolled.push(day); + if (!regular.includes(day)) caughtUp++; } catch (err) { failed.push(day); console.error(JSON.stringify({ msg: 'rollup day failed', day, err: String(err) })); } } - let purge: PurgeResult | null = null; - try { - purge = await purgeOldEvents(env, cutoff); - } catch (err) { - console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); - } - console.log( JSON.stringify({ msg: 'nightly rollup', days: rolled, + /** Missed days rolled up tonight. */ caught_up: caughtUp, + /** Days the time budget left for the next run. */ + deferred: deferred.length, rows, failed: failed.length, retention_days: keepDays, From 8998697f1d87fc670771817a11943868d4821e1a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:17:51 +0000 Subject: [PATCH 211/363] fix(vue,ts): link calls in Vue templates and in top-level destructuring (#2340) (#2357) `codegraph callers ` missed most of the components using it in a Nuxt app (Issue #2340), through two extraction gaps: - A destructuring declaration at module or ` +`; + const code = ending === 'CRLF' ? lf.replace(/\n/g, '\r\n') : lf; + const result = extractFromSource('Card.vue', code); + const component = result.nodes.find((n) => n.kind === 'component')!; + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => `${r.fromNodeId === component.id ? 'component' : r.fromNodeId}:${r.referenceName}@${r.line}`) + .sort(); + expect(calls).toEqual([ + 'component:label@3', + 'component:useBar@2', + 'component:useBar@2', + 'component:useFoo@7', + 'component:useFoo@8', + ]); + }); + it('should extract calls from Vue Options API object methods', () => { const code = `