Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/competitive-landscape-adoption.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ Where the JavaScript and TypeScript misses sit (fork, 133 needed files across th

Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances.

What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the files a report names has not been tested at all.
What the fork returns for those 19 questions (500-character query, 208 files in all): 59% are docs or site pages, tests or `.d.ts` files, although all 58 needed files are source. Eight of the 19 get no source file back at all, and eight get a file from the same directory as a needed one, but none gets a needed file in that run. Treating docs, site and markdown files as low value, with the same 0.5 penalty tests already get, was compared on a newer build than the run that picked these 19 (a build with the getter fix, over a fresh index), where the baseline already found 1 of the 58; with the penalty it found 3. On the full ContextBench sets, with the same index for both builds, it changed nothing worth keeping: the 39 JavaScript and TypeScript questions went from 0.2810 to 0.2831 file coverage (1 better, 1 worse), and the 42 Python and Go questions stayed at 0.6290 (none changed). Demoting non-source files frees result slots but does not bring the compiler files into view, so this change was dropped. A fix for JavaScript and TypeScript retrieval needs a different signal. Embeddings helped once in the Octocode probe below (one of six needed files on two chosen instances), but nothing like it has been tried inside CodeGraph, and call-graph reach from the symbols named in reports was checked and is not promising as a starting point from the symbols a report names: only 5 of the 19 reports name any symbol in the index, and walking two hops out from those symbols reaches 1 of the 58 needed files. Treating every word of a report that matches a symbol name as a starting point reaches 18 of 58 within two hops, but it returns about 54 files per question against about 11 today, and `explore` already scores files by name matches and graph weight; that variant was not compared with it at the same file budget, so whether reach adds anything beyond what `explore` already does is not settled.

A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement.

Expand Down