fix(explore): names the query spells that the index never saw are said, not glossed over

CHANGELOG.md

@@ -223,6 +223,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - `codegraph_explore`, `codegraph_node`, `codegraph_callers`, `codegraph_callees`, `codegraph_impact` and the `codegraph callers`, `callees` and `impact` commands now find a symbol written with its package, the way Go code names it: `refresh.Discovery.refresh`, `storage.Queryable.Querier`, or `gin.Context.Next` in gin's root package. Before, such a name found nothing, so a question written that way lost the symbol and explore's Flow had nothing to connect. A name the search ranked below other results, like gin's `binding.Default`, is found as well, and a Python or Rust module followed by a class and method works the same way.
 - When `codegraph_explore` finds more dynamic-dispatch links than its list shows, the symbols you named now take turns, so each one's links make the list: asking about prometheus's `refresh.Discovery.refresh hetzner.refresher.refresh hcloudDiscovery.refresh` now lists hetzner's `robotDiscovery` beside the links into other discovery packages that used to fill the list.
 - `codegraph_explore` now shows a function you name even when it is too long for its share of the answer: its definition and the start of its body, with a note where the body is cut that says which lines were left out and which `codegraph_explore` query returns them. Before, a file made of one long function and a line that calls it could come back as just its last few lines, without the function, and asking for the file by path did the same. A trimmed answer also no longer suggests `codegraph_node`, which isn't offered by default. Thanks @Huy3ko for the report and @danusha2345 for the first fix. (#2486)
+- When a `codegraph_explore` query names symbols the index has never seen, the answer now says so first: it lists those names and points to `codegraph status`, and to how a folder skipped by default, like a root `build/` folder, is brought into the index. Before, the answer showed only what the query's other words matched, such as an unrelated `Main` and a C++ `Parser`, with nothing to say the names asked about were missing. Thanks @tippmar-nr for the report. (#2462)
 - In C and C++, a struct, class, union or enum defined in a variable declaration, like `struct Foo { … } foo;`, is now indexed with its methods and enum values, next to the variable. Before, at most the variable was, so the type was missing from search and `codegraph_explore`, a class deriving from it had no base, and calls to its methods linked to nothing: in redis, the Sentinel's `sentinelState` and `redis-cli`'s `config` struct were missing. A struct, union or enum with no name of its own now takes the name of the variable it declares, so redis's `static struct { … } SPT;` is found as `SPT`, and one declared inside a function is named after its variable instead of `<anonymous>`. The comment above the declaration becomes the type's description. Re-index C and C++ projects after upgrading.
 - In Objective-C, a struct, union or enum defined in a variable declaration, like `static struct { Class isa; } placeholder;` or `enum { stateType, stateParamName } state;`, is now indexed with its enum values, next to the variable, as in C and C++. Before, only the variable was, so the type was missing from search and `codegraph_explore`. The same went for a `typedef enum { … } Name;` or `typedef struct … Name;` written after a line holding only a macro, like `OF_ASSUME_NONNULL_BEGIN` or `RCT_EXTERN_C_BEGIN`: ObjFW's `OFZIPArchiveEntryCompressionMethod` and every value in it were missing, and so was React Native's `RCTMethodInfo`. A struct, union or enum with no name of its own takes the name of the variable it declares, so ObjFW's class-cluster placeholders are found as `placeholder`, one declared inside a function or method is named after its variable instead of `<anonymous>`, and the comment above the declaration becomes the type's description. As in C, a type that a `.m` file defines links only from that file, so a call elsewhere that just shares its name, like ObjFW's calls of AmigaOS's `Lock(path, mode)`, doesn't link to the runtime's private `struct Lock`. Re-index Objective-C projects after upgrading.
 - In Objective-C, a macro written alone on its line, like `NS_ASSUME_NONNULL_BEGIN`, `OF_SUBCLASSING_RESTRICTED` or React Native's `RCT_EXTERN_C_BEGIN`, no longer hides the declaration below it. Before, the parser read such a macro as the start of a declaration, so an `@interface` under it could come out as a function and a `typedef` under it as two variables: many of ObjFW's classes, with their properties and methods and the classes they extend and protocols they adopt, were missing, and so were classes and protocols in React Native and KSCrash headers. A message to `super` from a class whose `@interface` follows an attribute macro like `NS_SWIFT_NAME(…)` still reaches the superclass. Re-index Objective-C projects after upgrading.

__tests__/explore-unindexed-names.test.ts

@@ -0,0 +1,196 @@
+/**
+ * Names the query spells that the index has never seen are said plainly,
+ * instead of answering with code that only matched the query's other words
+ * (#2462).
+ *
+ * A .NET repo kept real tool projects in a root `build/` folder, which the
+ * default skip list leaves unindexed. `codegraph_explore "ReleaseNotesBuilder
+ * Program Main ChangelogParser PersistentData Entry AddFeatures"` came back as
+ * an unrelated `Program.Main` and a C++ `Parser`, with nothing to say that four
+ * of the names matched nothing. The agent read the wrong code.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import * as fs from 'fs';
+import * as path from 'path';
+import * as os from 'os';
+import CodeGraph from '../src/index';
+import { ToolHandler, isIdentifierShapedToken } from '../src/mcp/tools';
+
+const QUERY = 'ReleaseNotesBuilder Program Main ChangelogParser PersistentData Entry AddFeatures';
+const MISSING = ['ReleaseNotesBuilder', 'ChangelogParser', 'PersistentData', 'AddFeatures'];
+
+/** The report's layout: the product under src/, its release tooling under a root build/. */
+const FILES: Record<string, string> = {
+  '.gitignore': 'bin/\nobj/\n',
+  'src/Agent/Program.cs': `namespace Agent
+{
+    public static class Program
+    {
+        public static int Main(string[] args)
+        {
+            var host = new AgentHost();
+            return host.Run(args);
+        }
+    }
+
+    public class AgentHost
+    {
+        public int Run(string[] args) { return args.Length; }
+    }
+}
+`,
+  'src/native/parser.cpp': `#include <string>
+namespace native {
+class Parser {
+public:
+  explicit Parser(std::string text) : text_(std::move(text)) {}
+  int parse() { return static_cast<int>(text_.size()); }
+private:
+  std::string text_;
+};
+}  // namespace native
+`,
+  // `ChangelogWriter` and `auditTrail` are seen only through a docstring and a
+  // folder name, never as a symbol.
+  'src/auditTrail/trail.ts': `/** Hands the audit entries to the ChangelogWriter. */
+export function recordTrail(): number { return 1; }
+`,
+  // Indexed as `handle`, with no `app.` on it, as express's own is.
+  'src/server/app.js': `const app = {};
+app.handle = function handle(req, res) { return res; };
+module.exports = app;
+`,
+  'build/ReleaseNotesBuilder/Program.cs': `namespace ReleaseNotesBuilder
+{
+    public static class Program
+    {
+        public static int Main(string[] args)
+        {
+            var parser = new ChangelogParser(args[0]);
+            var data = PersistentData.Load(args[1]);
+            foreach (var entry in parser.Entries()) entry.AddFeatures(data);
+            return 0;
+        }
+    }
+}
+`,
+  'build/ReleaseNotesBuilder/ChangelogParser.cs': `using System.Collections.Generic;
+namespace ReleaseNotesBuilder
+{
+    public class ChangelogParser
+    {
+        private readonly string _path;
+        public ChangelogParser(string path) { _path = path; }
+        public IEnumerable<Entry> Entries() { yield return new Entry(_path); }
+    }
+}
+`,
+  'build/ReleaseNotesBuilder/Entry.cs': `using System.Collections.Generic;
+namespace ReleaseNotesBuilder
+{
+    public class Entry
+    {
+        private readonly List<string> _features = new List<string>();
+        public Entry(string title) { }
+        public void AddFeatures(PersistentData data) { _features.AddRange(data.Features); }
+    }
+}
+`,
+  'build/ReleaseNotesBuilder/PersistentData.cs': `using System.Collections.Generic;
+namespace ReleaseNotesBuilder
+{
+    public class PersistentData
+    {
+        public List<string> Features { get; } = new List<string>();
+        public static PersistentData Load(string path) { return new PersistentData(); }
+    }
+}
+`,
+};
+
+describe('#2462 — explore says which named symbols are not in the index', () => {
+  let dir: string;
+  let cg: CodeGraph;
+  const explore = async (query: string) => {
+    const result = await new ToolHandler(cg).execute('codegraph_explore', { query });
+    return { text: result.content?.[0]?.text ?? '', isError: result.isError === true };
+  };
+
+  beforeAll(async () => {
+    dir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-2462-'));
+    for (const [rel, text] of Object.entries(FILES)) {
+      fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true });
+      fs.writeFileSync(path.join(dir, rel), text);
+    }
+    cg = CodeGraph.initSync(dir);
+    await cg.indexAll();
+  }, 120_000);
+
+  afterAll(() => {
+    cg?.destroy();
+    if (dir) fs.rmSync(dir, { recursive: true, force: true });
+  });
+
+  it('fixture shape: the root build/ folder is skipped by default, so its names are not indexed', () => {
+    expect(cg.getFiles().map((f) => f.path).filter((p) => p.startsWith('build/'))).toEqual([]);
+    expect(cg.getNodesByName('Main').map((n) => n.filePath)).toEqual(['src/Agent/Program.cs']);
+  });
+
+  it('names the missing symbols above the unrelated matches, success-shaped', async () => {
+    const { text, isError } = await explore(QUERY);
+    expect(isError).toBe(false);
+    expect(text).toContain(`**Not in the index:** ${MISSING.map((n) => `\`${n}\``).join(', ')}.`);
+    expect(text).toContain("so the code below matches the query's other words, not them");
+    // Above the source, where it is read first.
+    expect(text.indexOf('**Not in the index:**')).toBeLessThan(text.indexOf('**`src/Agent/Program.cs`**'));
+    // Where to look, never "Read"; including a skipped folder is the user's call.
+    expect(text).toContain('`codegraph status` shows what is.');
+    expect(text).toContain('the user can include that folder by adding a re-include such as `!/build/` to `.gitignore`.');
+    expect(text).not.toMatch(/\bRead (?:the|these|those|it)\b/);
+  });
+
+  it('never names a single word, which may be English, nor a name the index has', async () => {
+    const { text } = await explore(QUERY);
+    const note = text.split('\n').find((l) => l.startsWith('> **Not in the index:**'))!;
+    for (const word of ['Program', 'Main', 'Entry']) expect(note).not.toContain(`\`${word}\``);
+    expect((await explore('Program Main AgentHost Run')).text).not.toContain('Not in the index');
+    expect((await explore('How does the agent host run its main program?')).text).not.toContain('Not in the index');
+  });
+
+  it('a name seen only in a docstring or a folder name is in the index', () => {
+    expect(cg.getUnseenNames(['ChangelogWriter', 'auditTrail', 'agenthost', 'Agent.Program', ...MISSING]))
+      .toEqual(MISSING);
+  });
+
+  it('a qualified name is in the index when its member is, whatever the receiver', () => {
+    // `app.handle` is indexed as `handle`; only a member never seen makes it missing.
+    expect(cg.getUnseenNames(['app.handle', 'Zorblax.frobnicate', 'pkg::recordTrail'])).toEqual(['Zorblax.frobnicate']);
+  });
+
+  it('says so too when nothing matched at all', async () => {
+    const { text, isError } = await explore('ReleaseNotesBuilder PersistentData');
+    expect(isError).toBe(false);
+    expect(text).toContain('No relevant code found');
+    expect(text).toContain('**Not in the index:** `ReleaseNotesBuilder`, `PersistentData`.');
+    expect(text).not.toContain('the code below');
+  });
+
+  it('goes away once the folder is re-included and synced', async () => {
+    fs.writeFileSync(path.join(dir, '.gitignore'), 'bin/\nobj/\n!/build/\n');
+    await cg.sync();
+    const { text } = await explore(QUERY);
+    expect(text).not.toContain('Not in the index');
+    expect(text).toContain('build/ReleaseNotesBuilder/ChangelogParser.cs');
+  }, 120_000);
+});
+
+describe('isIdentifierShapedToken', () => {
+  it('takes names written like code and leaves single words alone', () => {
+    for (const t of ['ChangelogParser', 'addFeatures', 'build_report', 'MAX_SIZE', 'Session.request', 'pkg::run', '$store.items']) {
+      expect(isIdentifierShapedToken(t), t).toBe(true);
+    }
+    for (const t of ['Program', 'Main', 'How', 'parser', 'SQL', 'i18n', 'v1.2']) {
+      expect(isIdentifierShapedToken(t), t).toBe(false);
+    }
+  });
+});

src/db/queries.ts

@@ -359,6 +359,7 @@ export class QueryBuilder {
     getDominantFile?: SqliteStatement;
     getChangeStamp?: SqliteStatement;
     getTopRouteFile?: SqliteStatement;
+    unseenName?: SqliteStatement;
     getRoutingManifest?: SqliteStatement;
     insertNameSegment?: SqliteStatement;
   } = {};
@@ -1881,6 +1882,42 @@ export class QueryBuilder {
     };
   }
 
+  /**
+   * Which of `names` the index has never seen: no node is called that, in any
+   * case, and no indexed name, qualified name, signature or docstring has a
+   * word starting with it. A file's path is its qualified name, so a folder or
+   * file named that counts as seen too. Explore uses this to say plainly that
+   * a name the query spelled is not in the index (#2462).
+   *
+   * A qualified name (`app.handle`, `pkg::run`) is asked about by its last
+   * segment: the index does not always record the receiver it was written on
+   * — express's `app.handle = function handle(…)` is indexed as `handle` — so
+   * only a member it has never seen makes the whole name unseen.
+   *
+   * Bounded like `getExploreMissDiagnostics`: two index probes per name (the
+   * lower(name) index and FTS postings), never a source scan, at most 16 names.
+   */
+  getUnseenNames(names: readonly string[]): string[] {
+    const checked = [...new Set(names)].filter((n) => n.length > 0 && n.length <= 128).slice(0, 16);
+    if (checked.length === 0) return [];
+    if (!this.stmts.unseenName) {
+      this.stmts.unseenName = this.db.prepare(`
+        SELECT EXISTS (SELECT 1 FROM nodes WHERE lower(name) = lower(?))
+          OR EXISTS (SELECT 1 FROM nodes_fts WHERE nodes_fts MATCH ?) AS seen
+      `);
+    }
+    const stmt = this.stmts.unseenName;
+    return checked.filter((name) => {
+      const member = name.split(/::|\./).pop()!;
+      // FTS splits the phrase on everything but letters and digits, so a
+      // snake_case name is matched as the words it is made of.
+      const phrase = member.replace(/"/g, '');
+      if (phrase.length === 0) return false;
+      const row = stmt.get(member, `{name qualified_name signature docstring} : "${phrase}"*`) as { seen: number };
+      return !row.seen;
+    });
+  }
+
   /**
    * FTS5 search with prefix matching
    */

src/index.ts

@@ -2119,6 +2119,14 @@ export class CodeGraph {
     return this.queries.getExploreMissDiagnostics(query);
   }
 
+  /**
+   * Which of `names` the index has never seen, by name, path, signature or
+   * docstring — the names explore reports as not in the index (#2462).
+   */
+  getUnseenNames(names: readonly string[]): string[] {
+    return this.queries.getUnseenNames(names);
+  }
+
   /**
    * The COBOL copybooks a query names (`CVACT01Y`, `MYCOPYBOOK`): each
    * member's indexed copybook file(s) and every `COPY` / `EXEC SQL INCLUDE`