From 40edafa1a1b291849e3a10b29594a495a8ad4c7e Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 00:13:28 -0600 Subject: [PATCH 1/2] docs: record what explore returns for the missed JS/TS questions and the docs-penalty result --- docs/design/competitive-landscape-adoption.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 6de69930d4..75a82314a5 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -178,6 +178,8 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances. +What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, moved the 19 questions from 1 to 3 needed files. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal, such as embeddings or call-graph reach from the files a report names; neither has been tested. + A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement. | Instance | Needed files | Octocode | 1.6.2 | Fork | From 070966481791045ca3beffff7d81a783b0e22378 Mon Sep 17 00:00:00 2001 From: Aaron Queen Date: Sun, 11 Oct 2026 00:17:02 -0600 Subject: [PATCH 2/2] docs: say which build each baseline comes from and what was tried --- docs/design/competitive-landscape-adoption.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/competitive-landscape-adoption.md b/docs/design/competitive-landscape-adoption.md index 75a82314a5..85dba88350 100644 --- a/docs/design/competitive-landscape-adoption.md +++ b/docs/design/competitive-landscape-adoption.md @@ -178,7 +178,7 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances. -What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, moved the 19 questions from 1 to 3 needed files. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal, such as embeddings or call-graph reach from the files a report names; neither has been tested. +What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names has not been tested at all. A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement.