From e16a0ddea605bd435605c3a80eca1439f383704b Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 00:47:16 -0600 Subject: [PATCH 1/3] docs: record the call-graph reach check for the missed JS/TS questions --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 85dba88350..b9edd4113b 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -178,7 +178,7 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances. -What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names has not been tested at all. +What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names was checked and does not help: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget. A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement. From 9ae5461d17f75fab3f5257a76355ebf1c3af540d Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 00:53:44 -0600 Subject: [PATCH 2/3] docs: qualify the reach conclusion --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index b9edd4113b..226a6435b4 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -178,7 +178,7 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances. -What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names was checked and does not help: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget. +What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names was checked and is not promising as a starting point from the symbols a report names: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget, so whether reach adds anything beyond what `explore` already does is not settled. A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement. From c5c2565e4f090e80216b085353916447ef702c54 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 00:56:01 -0600 Subject: [PATCH 3/3] docs: say the reach probe starts from symbols, not files --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 226a6435b4..8e8abf7fd0 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -178,7 +178,7 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances. -What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names was checked and is not promising as a starting point from the symbols a report names: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget, so whether reach adds anything beyond what `explore` already does is not settled. +What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the symbols named in reports was checked and is not promising as a starting point from the symbols a report names: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget, so whether reach adds anything beyond what `explore` already does is not settled. A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement.