diff --git a/CHANGELOG.md b/CHANGELOG.md index 88a1a65189..dd5a43e916 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -564,7 +564,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In C++, a call on a range-based `for` loop's variable, like `suite->SetVerbose(verbose)` inside `for (ConformanceTestSuite *suite : suites)`, now links to the method of the type the loop declares. Before, the loop's declaration was never read, so the call was matched by the variable's name alone: in RocksDB, `log->file()` on a `log::Writer` went to `BlobLogWriter::file`, and a loop variable named like a member of its class was taken for that member. The variable counts only inside its loop. Re-index C++ projects after upgrading. -- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, the call no longer links to a project method. Re-index C++ projects after upgrading. +- In C++, a call made with `->` on a variable, parameter or member declared as a `std::unique_ptr`, `std::shared_ptr` or `std::optional`, like `iter->Valid()` on a `std::unique_ptr iter`, now links to that method of the type it holds, or of a class that type inherits from, with the type looked up the way C++ does it, so a class of the same name nested elsewhere is not mistaken for it. Before, the call went to whichever project method shared its name: in facebook/rocksdb, `iter->Valid()` reached `ArenaWrappedDBIter::Valid` and `fs->GetFileSize(…)` on a `std::shared_ptr` reached a test's stand-in file system, and calls like `it->Next()` were often not linked at all. When what it holds is a template parameter or a standard-library type, like `std::unique_ptr>`, or a class that has no method of that name and neither do the classes it derives from, the call no longer links to a project method; a class derived from something that can't be followed, like a `decltype(…)`, may still have it, so that call is left unlinked. Re-index C++ projects after upgrading. - `codegraph sync` and the file watcher now move a React Router route to its page when the router file loads that page lazily into a value of its own, like `const Docs = lazy(() => import('./pages/Docs'))` rendered by `} />`, and the page's file is added after the router was indexed, comes back after being deleted, or gets or changes its default export in a later edit. Before, the route kept pointing at the `Docs` declaration, or at the page's old component, until the router file itself changed or the project was re-indexed, unless the page's component happened to share the declaration's name. The same now holds for a `lazy: () => import('./pages/Docs')` route whose page switches its default export to another component. Re-index React projects after upgrading to correct routes an earlier version left behind. @@ -575,6 +575,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - `codegraph sync` and the file watcher now update the files that import through a module that re-exports what it imports, like a package's `__init__.py` or an `index.ts` barrel, when that module is deleted or added, and in Python also the files that reach it through another package re-exporting it. Before, deleting `shop/__init__.py` left every `from shop import Order` linked to the class it re-exported, and deleting `components/index.ts` left every `import { Button } from './components'` linked, until each importing file changed or the project was re-indexed. When such a module came back, the imports stayed linked the way they were while it was missing, and in a large Python project an import of a module whose name is as common as `models` or `test`, like `from django.test import TestCase`, stayed unlinked. Python, JavaScript, TypeScript, Vue, Svelte and Astro. - In JavaScript and TypeScript, a file that re-exports a module, like an `index.ts` with `export * from './button'` or `export { Button } from './button'`, now counts as importing that module, the way an `import` statement does. `codegraph affected` and a module's dependents now include the barrels that re-export it; before, a barrel made only of `export *` lines showed no dependency at all. Re-index JavaScript and TypeScript projects after upgrading. - `codegraph sync` and the file watcher now also update the files that import through an `index.ts` re-exporting another barrel, like `export * from './dropdown-menu'`, when that inner barrel is deleted or added. Before, their imports stayed linked through the deleted barrel, or stayed linked the way they were while it was missing, until each importing file changed or the project was re-indexed. +- An import in a Vue, Svelte or Astro component now links the component to the file it imports, as an import in a script does, and so does an import of a scoped package that a `tsconfig` path or a workspace package maps to a project file, like `import { debounce } from '@excalidraw/common'`. Before, only the names such an import binds were linked, so a barrel or a module imported for its side effects showed no dependency from those files in `codegraph affected`, and `codegraph sync` could not update them when a barrel they import through was deleted or added. `codegraph sync` and the file watcher also link such an import again, and one written as `.` or `..`, when the file it names comes back. Re-index JavaScript, TypeScript, Vue, Svelte and Astro projects after upgrading. +- `codegraph sync` and the file watcher now move an import written without an extension, like `import { sizeOf } from './button'`, to a file added beside the one it found before when the new file comes first, such as `button.ts` next to `button.vue`. Before, the import and the names it binds stayed linked to `button.vue` until the importing file changed or the project was re-indexed. JavaScript, TypeScript, Vue, Svelte and Astro. - In Kotlin, a call on a typed receiver now links to an extension function declared in a package, like `value.shout()` or the infix `a join b` with `value: Foo` and `fun Foo.shout()` under `package demo`, including one declared in another file of the package. Before, it linked only when the file had no `package` line, so the usual Kotlin project got no edge. On JetBrains Exposed, 1,020 such calls now link (`Table.selectAll()` to `FieldSet.selectAll`, `transaction.throwUnsupportedException(…)`), and none that linked before is lost. A member of the receiver's class still wins over an extension of the same name. diff --git a/README.md b/README.md index 616662ff58..0db7f309b7 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Follow [@getcodegraph](https://x.com/getcodegraph) on X for updates. ## About this fork -This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`041db101`](https://github.com/colbymchenry/codegraph/commit/041db101) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. +This is **bompus/codegraph**, a fork of [colbymchenry/codegraph](https://github.com/colbymchenry/codegraph). Its default branch, `fork/consolidated`, contains upstream `main` through [`c19f0284`](https://github.com/colbymchenry/codegraph/commit/c19f0284) (after v1.6.2) plus the fork's own work, and it takes upstream changes as they land. Changes that suit upstream are also offered there as pull requests. The fork publishes no releases. The install scripts, npm package, badges and `codegraph upgrade` further down this page install **upstream's** releases. To run the fork, build it from source (below). diff --git a/__tests__/cpp-smart-pointer-receiver.test.ts b/__tests__/cpp-smart-pointer-receiver.test.ts index fddbf8370e..df1816397a 100644 --- a/__tests__/cpp-smart-pointer-receiver.test.ts +++ b/__tests__/cpp-smart-pointer-receiver.test.ts @@ -17,9 +17,9 @@ * `Iterator` the call means inherits `Valid` from `IteratorBase`. A template * parameter, a library type and an alias of a type outside the project hold * no project class, so those calls get no edge; neither does a call the held - * class has no method for (unless a class template's specialization may - * declare it). A held type the project doesn't declare — a generated message - * — leaves the call as it was. + * class has no method for (unless a class template's specialization, or a + * base the index doesn't link, may declare it). A held type the project + * doesn't declare — a generated message — leaves the call as it was. */ import { describe, it, expect, afterEach } from 'vitest'; import * as fs from 'fs'; @@ -454,4 +454,39 @@ describe('what keeps the call as it was', () => { cg.close(); } }); + + it('a held class with a base the index does not link, which may declare the method', async () => { + // protobuf's FlatAllocator derives from a `decltype(…)` that names FlatAllocatorImpl. + const cg = await indexed({ + 'src/google/protobuf/descriptor.cc': lines( + '#include ', + 'namespace google {', + 'namespace protobuf {', + 'namespace internal {', + 'template ', + 'class FlatAllocatorImpl {', + ' public:', + ' void PlanArray(int array_size) {}', + '};', + 'template ', + 'FlatAllocatorImpl ApplyTypeList(int);', + 'class FlatAllocator : public decltype(ApplyTypeList(0)) {};', + '} // namespace internal', + 'void PlanAllocationSize(std::unique_ptr& alloc) {', + ' alloc->PlanArray(1);', + '}', + '} // namespace protobuf', + '} // namespace google', + ), + }); + try { + // The held-type lookup leaves this call alone (the base is unlinked), and the + // fork's fallback guesses no method by its name for a receiver whose class it + // knows but which lacks the method, so the call stays unresolved (as for a + // reference in cpp-declared-type-receiver.test.ts). + expect(calls(cg, 'google::protobuf::PlanAllocationSize')).toEqual([]); + } finally { + cg.close(); + } + }); }); diff --git a/__tests__/cpp-string-literal-declarations.test.ts b/__tests__/cpp-string-literal-declarations.test.ts new file mode 100644 index 0000000000..c0076eb547 --- /dev/null +++ b/__tests__/cpp-string-literal-declarations.test.ts @@ -0,0 +1,182 @@ +/** + * A declaration written inside a C++ string literal is text, not code, so + * receiver inference doesn't take it for the receiver's declaration. + * + * protocolbuffers/protobuf's upb generator emits C code from raw string + * templates: `GenerateRepeatedSetters(upb::FieldDefPtr field, …)` writes + * `upb_MiniTableField field = $field_init$;` inside an `R"cc( … )cc"`, then + * calls `field.ctype()` on its own parameter. Reading back from the call + * found the template's line first, took `field` for a `upb_MiniTableField` — + * a C struct with no methods — and the call got no edge (before #2493, a + * guess by the method's name). + * + * A raw string can span many lines and hold quotes and comment markers, so + * its lines are found by reading the file once, as the compiler does. An + * ordinary string's contents are blanked on the line being read. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const roots: string[] = []; +afterEach(() => { + for (const root of roots.splice(0)) fs.rmSync(root, { recursive: true, force: true }); +}); + +async function indexed(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-cpp-strdecl-')); + roots.push(root); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + return CodeGraph.init(root, { index: true }); +} + +function callable(cg: CodeGraph, qualifiedName: string): Node { + const node = [...cg.getNodesByKind('method'), ...cg.getNodesByKind('function')].find((n) => n.qualifiedName === qualifiedName); + if (!node) throw new Error(`no function or method ${qualifiedName}`); + return node; +} + +/** `calls` callees of a function or method, as `qualifiedName`. */ +function calls(cg: CodeGraph, caller: string): string[] { + return cg + .getCallees(callable(cg, caller).id) + .filter((r) => r.edge.kind === 'calls') + .map((r) => r.node.qualifiedName) + .sort(); +} + +const lines = (...l: string[]): string => [...l, ''].join('\n'); + +/** upb's field wrapper, and the C struct a generated template declares. */ +const UPB = { + 'upb/reflection/def.hpp': lines( + 'namespace upb {', + 'class FieldDefPtr {', + ' public:', + ' int ctype() const { return 0; }', + ' const char* name() const { return ""; }', + '};', + '} // namespace upb', + ), + 'upb/mini_table/field.h': lines( + 'typedef struct upb_MiniTableField {', + ' int number;', + '} upb_MiniTableField;', + ), +}; + +describe('a declaration inside a string literal is not the receiver\'s', () => { + it('reproduction: upb\'s generator declares `field` in the C code a raw string emits', async () => { + const cg = await indexed({ + ...UPB, + 'upb_generator/c/generator.cc': lines( + '#include "upb/reflection/def.hpp"', + '#include "upb/mini_table/field.h"', + 'namespace upb {', + 'namespace generator {', + 'void Emit(const char* text) {}', + 'void GenerateRepeatedSetters(upb::FieldDefPtr field) {', + ' Emit(R"cc(', + ' UPB_INLINE void $0_resize(void* msg, size_t size) {', + ' upb_MiniTableField field = $1;', + ' // "a quote" and ) in the template', + ' upb_Message_Resize(msg, &field, size);', + ' }', + ' )cc");', + ' if (field.ctype() == 1) Emit(field.name());', + '}', + '} // namespace generator', + '} // namespace upb', + ), + }); + try { + expect([...new Set(calls(cg, 'upb::generator::GenerateRepeatedSetters'))]).toEqual([ + 'upb::FieldDefPtr::ctype', + 'upb::FieldDefPtr::name', + 'upb::generator::Emit', + ]); + } finally { + cg.close(); + } + }); + + it('an ordinary string that reads like a declaration', async () => { + const cg = await indexed({ + 'src/runner.h': lines( + 'namespace demo {', + 'class Runner {', + ' public:', + ' void Run() {}', + '};', + 'class Stub {', + ' public:', + ' void Run() {}', + '};', + 'class Printer {', + ' public:', + ' void Print(const char* text) {}', + '};', + '} // namespace demo', + ), + 'src/generator.cc': lines( + '#include "src/runner.h"', + 'namespace demo {', + 'void Generate(Printer* printer, Runner* runner) {', + ' printer->Print("Stub* runner = nullptr;");', + ' runner->Run();', + '}', + '} // namespace demo', + ), + }); + try { + expect(calls(cg, 'demo::Generate')).toEqual(['demo::Printer::Print', 'demo::Runner::Run']); + } finally { + cg.close(); + } + }); +}); + +describe('what a literal on the line does not hide', () => { + it('a declaration after a character literal holding a quote, or after a comment holding `R"(`', async () => { + const cg = await indexed({ + 'src/runner.h': lines( + 'namespace demo {', + 'class Runner {', + ' public:', + ' void Run() {}', + '};', + 'class Stub {', + ' public:', + ' void Run() {}', + '};', + 'inline Runner* MakeRunner() { return nullptr; }', + '} // namespace demo', + ), + 'src/quotes.cc': lines( + '#include "src/runner.h"', + 'namespace demo {', + 'void Quote(Stub* runner) {', + ' char q = \'"\'; int n = 1\'000; Runner* runner = MakeRunner();', + ' runner->Run();', + '}', + 'void Comment(Stub* runner) {', + ' /* R"( not a raw string */ Runner* runner = MakeRunner();', + ' runner->Run();', + '}', + '} // namespace demo', + ), + }); + try { + expect(calls(cg, 'demo::Quote')).toEqual(['demo::MakeRunner', 'demo::Runner::Run']); + expect(calls(cg, 'demo::Comment')).toEqual(['demo::MakeRunner', 'demo::Runner::Run']); + } finally { + cg.close(); + } + }); +}); diff --git a/__tests__/sfc-alias-module-imports.test.ts b/__tests__/sfc-alias-module-imports.test.ts new file mode 100644 index 0000000000..600c0252e6 --- /dev/null +++ b/__tests__/sfc-alias-module-imports.test.ts @@ -0,0 +1,78 @@ +/** + * An import statement's module reference links the importing file to the file + * it names: from a Vue, Svelte or Astro component as from a script, and for a + * scoped package a `tsconfig` path maps to a project file. Before, a + * component's `import { x } from '../lib/util'` and a script's `import { x } + * from '@acme/common'` bound `x` through that file while the statement itself + * linked nothing, so the module never counted the importer as a dependent. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import CodeGraph from '../src/index'; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +/** The files a file's import statements link it to (its module edges, not its calls). */ +function moduleImports(graph: CodeGraph, file: string): string[] { + const ids = graph.getNodesInFile(file).map((n) => n.id); + return [...new Set(graph.getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === 'imports' && e.target.startsWith('file:')) + .map((e) => e.target.slice('file:'.length)))].sort(); +} + +async function index(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-sfc-alias-')); + roots.push(root); + for (const [file, text] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, file)), { recursive: true }); + fs.writeFileSync(path.join(root, file), text); + } + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +describe('an import links the importing file to the module file', () => { + it('from a Vue, Svelte or Astro component, by relative path', async () => { + const graph = await index({ + 'package.json': '{"name":"app"}', + 'src/lib/util.ts': 'export function format(s: string) { return s; }\n', + 'src/components/Card.vue': + '\n\n', + 'src/components/Badge.svelte': + '\n{label}\n', + 'src/pages/index.astro': + '---\nimport { format } from \'../lib/util\';\nconst heading = format(\'z\');\n---\n

{heading}

\n', + }); + for (const component of ['src/components/Card.vue', 'src/components/Badge.svelte', 'src/pages/index.astro']) { + expect(moduleImports(graph, component), component).toEqual(['src/lib/util.ts']); + } + }, 120_000); + + it('for a scoped package a tsconfig path maps to a project file', async () => { + const graph = await index({ + 'package.json': '{"name":"app"}', + 'tsconfig.json': JSON.stringify({ + compilerOptions: { baseUrl: '.', paths: { '@acme/common': ['./packages/common/src/index.ts'] } }, + }), + 'packages/common/src/index.ts': "export * from './debounce';\n", + 'packages/common/src/debounce.ts': 'export function debounce(fn: () => void) { return fn; }\n', + // `useTesting` keeps the outside package's import in the file, unused. + 'apps/web/src/search.ts': + "import { debounce } from '@acme/common';\nimport { render } from '@testing-library/react';\nexport const useTesting = render;\nexport const run = debounce(() => {});\n", + }); + // The outside package names no project file: the file links to nothing for it. + expect(moduleImports(graph, 'apps/web/src/search.ts')).toEqual(['packages/common/src/index.ts']); + expect(graph.getFileDependents('packages/common/src/index.ts')).toEqual(['apps/web/src/search.ts']); + }, 120_000); +}); diff --git a/__tests__/sync-module-importers.test.ts b/__tests__/sync-module-importers.test.ts index b03f6f37d9..44148768a7 100644 --- a/__tests__/sync-module-importers.test.ts +++ b/__tests__/sync-module-importers.test.ts @@ -57,10 +57,18 @@ const PYTHON_BARRELS = ['app/shop/__init__.py', 'lib/api.py', 'app/orm/__init__. * ES modules: an `index.ts` re-exporting under the same name and under * another, and modules re-exporting it in turn — an outer `index.ts`, all of * it and one name renamed, and a module passing on one name, read through a - * namespace import — imported in its stead. + * namespace import — imported in its stead. It is imported from Vue + * components too (one beside it, as `.`), and through a scoped package a + * `tsconfig` path maps to it. */ const TS: Files = { 'package.json': '{"name":"app"}', + 'tsconfig.json': JSON.stringify({ compilerOptions: { baseUrl: '.', paths: { '@app/widgets': ['./src/components/index.ts'] } } }), + 'src/View.vue': + "\n\n", + 'src/aliased.ts': "import { Button } from '@app/widgets';\nexport function aliased() { return Button(); }\n", + 'src/components/Panel.vue': + "\n\n", 'src/components/Button.ts': 'export function Button() { return 1; }\n', 'src/components/Icon.ts': 'export function BaseIcon() { return 2; }\n', 'src/components/index.ts': "export { Button } from './Button';\nexport { BaseIcon as Icon } from './Icon';\n", @@ -139,10 +147,30 @@ const without = (files: Files, gone: string[]): Files => const only = (files: Files, kept: string[]): Files => Object.fromEntries(Object.entries(files).filter(([file]) => kept.includes(file))); +/** + * A file beside the one an extensionless import resolved to, which outranks + * it: a component's `./button` names `button.ts` when it is there, and + * `button.vue` when it is not — `sizeOf`, which both export, from the one or + * the other, and `onlyVue`, which only `button.vue` exports, through it or by + * its name. + */ +const OUTRANKING: Files = { + 'package.json': '{"name":"app"}', + 'src/widgets/button.vue': + '\n\n', + 'src/widgets/button.ts': + "export default function button() { return 'b'; }\nexport function buttonClass() { return 'c'; }\nexport function sizeOf() { return 2; }\n", + 'src/widgets/group.vue': + "\n\n", + 'src/widgets/panel.ts': "import button from './button';\nexport function panel() { return button(); }\n", +}; +const OUTRANKING_FILES = ['src/widgets/button.ts']; + describe.each([ ['Python package and module', PYTHON, PYTHON_BARRELS], ['ES module barrel', TS, TS_BARRELS], ['ES module reached by two routes', TS_TWO_ROUTES, ['src/core/index.ts']], + ['file outranking the one an import resolved to', OUTRANKING, OUTRANKING_FILES], ])('sync re-resolves the imports through a re-exporting file: %s', (_label, files, barrels) => { it('when it is deleted', async () => { const root = tempRoot(); diff --git a/codegraph-kernel/src/resolve/cpp_receivers.rs b/codegraph-kernel/src/resolve/cpp_receivers.rs index f19303dac3..0ce4bba2b5 100644 --- a/codegraph-kernel/src/resolve/cpp_receivers.rs +++ b/codegraph-kernel/src/resolve/cpp_receivers.rs @@ -109,9 +109,11 @@ impl KernelResolver { // Preserve the existing guarded alias and specialization fallback on a miss. let member = self.match_bound_type_member(&owner.qualified_name, method, r)?; // A class that lacks the method, and whose bases lack it too, is called nothing (but a - // class template's specialization may declare it, and so may a base the index doesn't link). + // class template's specialization may declare it, and so may a base the index doesn't link, + // held in a smart pointer or not). if member.is_none() && self.supertypes_complete && !self.cpp_alias_template_owner(&owner, r)? - && (held || (self.cpp_methods_named(&owner, method, r)?.is_empty() && self.cpp_bases_known(&owner, 0, &mut HashSet::new())?)) { return Ok(Some(None)); } + && (held || self.cpp_methods_named(&owner, method, r)?.is_empty()) + && self.cpp_bases_known(&owner, 0, &mut HashSet::new())? { return Ok(Some(None)); } Ok(member.map(Some)) } diff --git a/src/codegraph.ts b/src/codegraph.ts index d4dd581f3e..de6a02c73a 100644 --- a/src/codegraph.ts +++ b/src/codegraph.ts @@ -29,6 +29,7 @@ import { FindRelevantContextOptions, UnresolvedReference, IndexHealth, + Language, } from './types'; import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db'; import { WalCheckpointValve, resolveWalValveMb } from './db/wal-valve'; @@ -1069,12 +1070,31 @@ export class CodeGraph { ...this.queries.getNodeNamesByFiles(result.changedFilePaths), ...result.changedFilePaths.flatMap(moduleReferenceKeys), ])]); + // A scoped package that a `tsconfig` path or a workspace package maps + // to an added file (`@excalidraw/common` to `packages/common/src/ + // index.ts`) waits under its last segment, which that file's keys + // never give: look its failed imports up by the specifier itself. + // An import of the importer's own folder or the one above (`.`, + // `..`) waits under no key at all, so each is tried where written. + const addedFiles = new Set(result.addedFilePaths ?? []); + const resolvesToAdded = (specifier: string, fromFile: string, language: Language): boolean => { + const file = this.resolver.resolveModuleFile(specifier, fromFile, language); + return file !== null && addedFiles.has(file); + }; + const resolvedModules = addedFiles.size === 0 ? [] : this.queries.getFailedScopedImports() + .filter((s) => resolvesToAdded(s.name, s.filePath, s.language)) + .map((s) => s.name); + const folderImports = addedFiles.size === 0 ? [] : this.queries.getFailedFolderImports() + .filter((ref) => ref.filePath !== undefined && ref.language !== undefined && + resolvesToAdded(ref.referenceName, ref.filePath, ref.language)); const retryRows = new Set(retryable.map(ref => ref.rowId)); const importRetry = this.queries.getRetryableFailedImports( (result.addedFilePaths ?? []).flatMap(importPathKeys), - this.queries.getNodeNamesByFiles(result.changedFilePaths, ['namespace', 'module']) + this.queries.getNodeNamesByFiles(result.changedFilePaths, ['namespace', 'module']), + undefined, + resolvedModules ); - for (const ref of importRetry) { + for (const ref of [...importRetry, ...folderImports]) { if (!retryRows.has(ref.rowId)) { retryable.push(ref); retryRows.add(ref.rowId); } } retryable.sort((a, b) => (a.rowId ?? 0) - (b.rowId ?? 0)); @@ -1104,7 +1124,10 @@ export class CodeGraph { const file = this.queries.getFileByPath(filePath); return file ? [{ path: filePath, language: file.language }] : []; }); - const reopened = reopenReferencesThroughAddedFiles(this.queries, createImportReader(this.projectRoot, this.queries), added, edgesBeforeResolution); + const reopened = reopenReferencesThroughAddedFiles( + this.queries, createImportReader(this.projectRoot, this.queries), added, edgesBeforeResolution, + (specifier, fromFile, language) => this.resolver.resolveModuleFile(specifier, fromFile, language) + ); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-added-module-importers: ${Date.now() - tAdded}ms (${reopened} refs re-opened)`); } } else { diff --git a/src/db/queries.ts b/src/db/queries.ts index c8ac08f25e..c72d4b67fd 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3364,6 +3364,48 @@ export class QueryBuilder { })); } + /** + * The module edges (`imports` edges into a file node) into the files an + * import of `dir/name` can find instead of a file added there: `dir/name.*` + * under another extension, and the folder's own entry, `dir/name/index.*` + * or `dir/name/__init__.py`. With the source file and language a + * resurrection needs. + */ + getModuleImportsNamed( + dir: string, + name: string + ): Array { + const base = dir ? `${dir}/${name}` : name; + // `x.` up to `x/` and `x/index.` up to `x/index/` ('/' is the byte after '.'). + const rows = this.db + .prepare( + `SELECT e.*, tgt.file_path AS target_file_path, src.file_path AS source_file_path, src.language AS source_language + FROM nodes tgt + JOIN edges e ON e.target = tgt.id AND e.kind = 'imports' + JOIN nodes src ON src.id = e.source + WHERE tgt.kind = 'file' + AND ((tgt.file_path >= ? AND tgt.file_path < ?) OR (tgt.file_path >= ? AND tgt.file_path < ?) OR tgt.file_path = ?)` + ) + .all(`${base}.`, `${base}/`, `${base}/index.`, `${base}/index/`, `${base}/__init__.py`) as Array< + EdgeRow & { target_file_path: string; source_file_path: string; source_language: Language } + >; + return rows + // Not a deeper path that only shares the prefix (`dir/name.d/x.ts`). + .filter((row) => { + const file = row.target_file_path; + if (file === `${base}/__init__.py`) return true; + const index = `${base}/index.`; + return !(file.startsWith(index) ? file.slice(index.length) : file.slice(base.length + 1)).includes('/'); + }) + .map((row) => ({ + ...rowToEdge(row), + edgeId: row.id, + targetFilePath: row.target_file_path, + sourceFilePath: row.source_file_path, + sourceLanguage: row.source_language, + })); + } + /** The files with a resolution edge into one of `targetIds`. */ getFilesWithEdgesInto(targetIds: string[]): string[] { const files = new Set(); @@ -4254,26 +4296,40 @@ export class QueryBuilder { /** * Failed `imports` refs a sync should retry once files appear. An import * names a file, a folder or a namespace, which the symbol lookup above - * cannot match, so it is found two ways: + * cannot match, so it is found three ways: * - by `pathKeys` (`importPathKeys` of each added file), as its tail or its * whole name: `package:app/b.dart` waits for a file named `b.dart`, a * bare `#include "b.h"` or `require 'db.php'` is the file's whole name; * - by `names` (the changed files' namespaces and modules), as its whole * name when its tail is something else: a C# `using Foo.Bar` waits for * the namespace node `Foo.Bar`. An import whose tail is the name is the - * symbol lookup's, under that lookup's ceiling. - * Same per-name ceiling as {@link getRetryableFailedReferences}, counted - * per key. - */ - getRetryableFailedImports(pathKeys: string[], names: string[] = [], perNameCeiling: number = 500): UnresolvedReference[] { + * symbol lookup's, under that lookup's ceiling; + * - by `resolvedModules`, module specifiers that now resolve to an added + * file, as its whole name: `@excalidraw/common`, which a `tsconfig` path + * maps to `packages/common/src/index.ts`, waits under the tail `common`, + * which the file's own keys (`src`, `index`) never give. + * The other lookups share the per-name ceiling of + * {@link getRetryableFailedReferences}, counted per key; this one has the + * module ceiling, since a specifier that resolves to the file is no noise. + */ + getRetryableFailedImports( + pathKeys: string[], + names: string[] = [], + perNameCeiling: number = 500, + resolvedModules: string[] = [], + perModuleCeiling: number = 5000, + ): UnresolvedReference[] { const failedImports = "status = 'failed' AND reference_kind = 'imports'"; const lookups = [ - { keys: pathKeys, column: 'name_tail', where: failedImports }, - { keys: pathKeys, column: 'reference_name', where: failedImports }, - { keys: names, column: 'reference_name', where: `${failedImports} AND name_tail != reference_name` }, + { keys: pathKeys, column: 'name_tail', where: failedImports, ceiling: perNameCeiling }, + { keys: pathKeys, column: 'reference_name', where: failedImports, ceiling: perNameCeiling }, + { keys: names, column: 'reference_name', where: `${failedImports} AND name_tail != reference_name`, ceiling: perNameCeiling }, + // A module specifier that now resolves to an added file: no noise, so + // the module ceiling, as for a module key. + { keys: resolvedModules, column: 'reference_name', where: failedImports, ceiling: perModuleCeiling }, ]; const rows = new Map(); - for (const { keys, column, where } of lookups) { + for (const { keys, column, where, ceiling } of lookups) { const unique = [...new Set(keys)].filter((key) => key.length > 0); // Pass 1: per-key counts, chunked under the SQLite parameter limit. const retryKeys: string[] = []; @@ -4284,7 +4340,7 @@ export class QueryBuilder { .prepare(`SELECT ${column} AS key, COUNT(*) AS count FROM unresolved_refs WHERE ${where} AND ${column} IN (${placeholders}) GROUP BY ${column}`) .all(...chunk) as Array<{ key: string; count: number }>; for (const row of counts) { - if (row.count <= perNameCeiling) retryKeys.push(row.key); + if (row.count <= ceiling) retryKeys.push(row.key); } } // Pass 2: load the surviving rows; a row two lookups find is kept once. @@ -4312,6 +4368,49 @@ export class QueryBuilder { })); } + /** + * The scoped package specifiers (`@excalidraw/common`) of failed imports, + * each with one file that wrote it — what a `tsconfig` path or a workspace + * package can map to a project file that a sync adds. + */ + getFailedScopedImports(): Array<{ name: string; filePath: string; language: Language }> { + return (this.db + .prepare( + `SELECT reference_name AS name, MIN(file_path) AS file_path, language + FROM unresolved_refs + WHERE status = 'failed' AND reference_kind = 'imports' + AND reference_name GLOB '@?*/?*' AND reference_name NOT GLOB '@*/*/*' + GROUP BY reference_name` + ) + .all() as Array<{ name: string; file_path: string; language: Language }>) + .map((row) => ({ name: row.name, filePath: row.file_path, language: row.language })); + } + + /** + * Failed imports of the folder the importing file is in or under (`.`, + * `..`). Their tail is empty, since they name a folder only through where + * they are written, so no added file's keys find them. + */ + getFailedFolderImports(): UnresolvedReference[] { + return (this.db + .prepare( + `SELECT * FROM unresolved_refs + WHERE status = 'failed' AND reference_kind = 'imports' AND reference_name IN ('.', './', '..', '../')` + ) + .all() as UnresolvedRefRow[]) + .map((row) => ({ + fromNodeId: row.from_node_id, + referenceName: row.reference_name, + referenceKind: row.reference_kind as EdgeKind, + line: row.line, + column: row.col, + candidates: row.candidates ? safeJsonParse(row.candidates, undefined) : undefined, + filePath: row.file_path, + language: row.language as Language, + rowId: row.id, + })); + } + /** * Failed `calls` refs whose name tail is one of `tails` — the navigation * calls (`history.push`, `navigate`) a sync looks through after a route diff --git a/src/resolution/index.ts b/src/resolution/index.ts index ae130adb44..84655ca67e 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -2478,6 +2478,11 @@ export class ReferenceResolver { return this.frameworks.map((f) => f.name); } + /** The project file a module specifier written in `fromFile` names, as its import resolves it; null when none. */ + resolveModuleFile(specifier: string, fromFile: string, language: Language): string | null { + return resolveImportPath(specifier, fromFile, language, this.context); + } + /** True when a detected router binds navigation calls to its routes (see {@link reopenNavigationsFor}). */ hasNavigationRouters(): boolean { return this.frameworks.some((f) => f.navigation !== undefined); diff --git a/src/resolution/module-importers.ts b/src/resolution/module-importers.ts index f87f984521..018d9f304f 100644 --- a/src/resolution/module-importers.ts +++ b/src/resolution/module-importers.ts @@ -20,7 +20,7 @@ import * as fs from 'fs'; import * as path from 'path'; import type { QueryBuilder } from '../db/queries'; -import type { Binding, Language, UnresolvedReference } from '../types'; +import type { Binding, Edge, Language, UnresolvedReference } from '../types'; import type { ImportMapping, ReExport } from './types'; import { importMappingsFromBindings, pythonModulePath } from './import-resolver'; import { resurrectRefFromDroppedEdge } from './resurrect-ref'; @@ -158,6 +158,9 @@ export function reopenReferencesThroughRemovedFiles( return reopenEdges(queries, bindings, (e) => !gone.has(e.targetFilePath)); } +/** Resolves a module specifier written in `fromFile` as its import does: the project file it names, or null. */ +export type ModuleResolver = (specifier: string, fromFile: string, language: Language) => string | null; + /** * The other half: after a sync's retries, re-open the references the files * importing any of `added` make through those imports — the edges a name @@ -165,28 +168,96 @@ export function reopenReferencesThroughRemovedFiles( * `resolvedUpToEdgeId`: later ones were just resolved with the modules there) * and the references that failed, the import of the module itself among * them. (The failed-import retry looks those up by the module's name, and - * skips a name as common as `models` or `test`.) Returns the number re-opened. + * skips a name as common as `models` or `test`.) With `resolveModule`, also an + * import that resolved to a file beside an added one that now outranks it + * (`./button` to `button.vue` while `button.ts` was missing), and what that + * statement binds. Returns the number re-opened. */ export function reopenReferencesThroughAddedFiles( queries: QueryBuilder, reader: ImportReader, added: ReadonlyArray<{ path: string; language: Language }>, resolvedUpToEdgeId: number, + resolveModule?: ModuleResolver, ): number { - const bindings = bindingsOf(importersOf(queries, added, reader), reader); + const outranked = resolveModule ? outrankedImports(queries, added, resolveModule) : []; + const bindings = bindingsOf(importersOf(queries, added, reader, outranked), reader); const failed = [...bindings].flatMap(([file, { modules, names }]) => queries.getFailedReferencesThrough(file, modules, names)); - return reopenEdges(queries, bindings, (e) => e.edgeId <= resolvedUpToEdgeId) + queries.reopenFailedReferences(failed); + return reopenModuleEdges(queries, outranked) + + reopenEdges(queries, bindings, (e) => e.edgeId <= resolvedUpToEdgeId) + queries.reopenFailedReferences(failed); +} + +/** An import's module edge into a file that an added file now outranks. */ +interface OutrankedImport { + edge: Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }; + specifier: string; + added: string; +} + +/** + * The module edges an import resolved into a file beside one of `added` — + * another extension of its name, or its folder's own entry — when the + * import's specifier now names the added file: `./button` resolved to + * `button.vue` while `button.ts` was missing, and `button.ts` comes first. + */ +function outrankedImports( + queries: QueryBuilder, + added: ReadonlyArray<{ path: string; language: Language }>, + resolveModule: ModuleResolver, +): OutrankedImport[] { + const out: OutrankedImport[] = []; + for (const file of added) { + if (!BINDING_LANGUAGES.has(file.language)) continue; + const { dir, name } = importedName(file.path); + for (const edge of queries.getModuleImportsNamed(dir, name)) { + const specifier = edge.metadata?.refName; + if (edge.targetFilePath === file.path || typeof specifier !== 'string') continue; + if (resolveModule(specifier, edge.sourceFilePath, edge.sourceLanguage) === file.path) { + out.push({ edge, specifier, added: file.path }); + } + } + } + return out; +} + +/** The folder and name an import finds a file by: `a/b.ts` → `a`, `b`; a folder's own entry `a/b/index.ts` → `a`, `b`. */ +function importedName(filePath: string): { dir: string; name: string } { + const slash = filePath.lastIndexOf('/'); + const base = filePath.slice(slash + 1); + const dir = slash < 0 ? '' : filePath.slice(0, slash); + if (base.startsWith('index.') || base === '__init__.py') { + const up = dir.lastIndexOf('/'); + return { dir: up < 0 ? '' : dir.slice(0, up), name: dir.slice(up + 1) }; + } + const dot = base.indexOf('.', 1); + return { dir, name: dot < 0 ? base : base.slice(0, dot) }; +} + +/** Swap the outranked module edges for their references, for the sweep. Returns how many. */ +function reopenModuleEdges(queries: QueryBuilder, outranked: readonly OutrankedImport[]): number { + const edgeIds: number[] = []; + const refs: UnresolvedReference[] = []; + for (const { edge } of outranked) { + const ref = resurrectRefFromDroppedEdge(edge); + if (!ref || edgeIds.includes(edge.edgeId)) continue; + edgeIds.push(edge.edgeId); + refs.push(ref); + } + if (refs.length > 0) queries.replaceResolutionEdgesWithUnresolvedRefs(edgeIds, refs); + return refs.length; } /** * The files with an import statement naming one of `files`, or one of the * modules re-exporting from them, other than `files` themselves (going away, - * or just resolved with the others in place). + * or just resolved with the others in place); `outranked` imports name the + * added file they now resolve to. */ function importersOf( queries: QueryBuilder, files: ReadonlyArray<{ path: string; language: Language }>, reader: ImportReader, + outranked: readonly OutrankedImport[] = [], ): Importers { const importers: Importers = new Map(); const changed = new Set(files.map((file) => file.path)); @@ -216,6 +287,10 @@ function importersOf( for (const i of queries.getImportsInto(`file:${module.path}`, module.path)) { if (i.refName !== null) add(i.filePath, i.language, i.refName, module); } + if (depth > 1) continue; + for (const o of outranked) { + if (o.added === module.path) add(o.edge.sourceFilePath, o.edge.sourceLanguage, o.specifier, module); + } } const python = level.filter((module) => module.language === 'python'); if (python.length > 0) addPythonImporters(queries, python, reader, add);