Skip to content

docs: record the getter fix result and why JS/TS retrieval misses #451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bompus merged 2 commits into from
Oct 11, 2026
Merged

docs: record the getter fix result and why JS/TS retrieval misses #451

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
56b331a
docs: record the getter fix result and why JS/TS retrieval misses
bompus Oct 11, 2026
ae39a9e
docs: say what the indexing figures support
bompus Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/design/competitive-landscape-adoption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,8 @@ The first row follows from repowise's own rule: its TypeScript write-up says att

The 112 real gaps split into 54 named function or method calls, 42 calls into the JSX runtime (`useContext`, `useCallback` and similar) and 16 anonymous callbacks. The largest single pattern is a chained receiver: `c.req.header(...)` appears 16 times, where `c` is a typed parameter and `req` a property whose type the fork would need to follow. This analysis does not show why those calls get no edge.

A first fix for the largest real pattern (PR #450) types a property written as a getter by its return type, which is how hono declares `Context.req`, and stops a plain constructor parameter named like a field from standing in for it. On hono it adds 74 call edges and removes none, and zod's edges are unchanged. It matches only 3 more oracle edges (469 to 472 of 706, recall 0.664 to 0.669), so the earlier ceiling of about 3 points from counting `c.req.header()` call sites was too high. Of the 74 new edges, 63 are in test files the oracle does not judge, 8 are in callers it did not reach, and 3 match on another overload line: the fork links `header` to the first overload signature (line 185) while the oracle keys the implementation (line 188). Treating overloads as one declaration would move recall further than the receiver fix did; I have not measured that.

### 6.2 Natural-language file retrieval, 2026-10-10

ContextBench asks a natural-language bug description and scores the share of the gold files the tool returns. The first table is repowise's sealed 42 (30 django and 12 cli/cli instances), graded the way ContextBench's own grader does it: only predicted files that exist at the instance's base commit count. The harness reproduces repowise's published 1.5.0 figure exactly (0.6095). The question is the instance's problem statement passed to `codegraph explore`, with the files taken in order of first mention, capped at 50.
Expand All @@ -174,6 +176,8 @@ On these instances the fork wins 4, loses 7 and ties 28 against 1.6.2, and 19 of

Where the JavaScript and TypeScript misses sit (fork, 133 needed files across the 39 instances): the fork returned 31, and of the other 102, 22 had a different file from the same directory in its results, 16 had one from the parent directory, and 64 (48% of all needed files) had nothing nearby. The needed files are source files (121 of 133) and the median instance needs 3 of them, against 1 on the Python and Go set. About 29% of the files the fork returned were tests, docs or the svelte website. This analysis uses the saved results only and does not show why the ranking misses.

Why the JavaScript and TypeScript questions miss (the 19 instances where the fork found none of the needed files, 58 needed files in all): 57 of the 58 files exist at the base commit and 54 are in the fork's index, and the four unindexed ones are `package.json` files and a repro file. Those four cannot be returned by any query, but the other 54 are indexed and still missed, so indexing is not the main cause. A query made of up to 10 identifiers taken from each report found 4 of the 58, so wording is not the main cause either. One of the 19 reports quotes text that appears word for word in a needed file (svelte's "Attributes need to be unique", in `parse/state/tag.ts`); `codegraph explore` on that sentence alone does not return the file, because the index covers names, docstrings and signatures and only short strings such as module paths and storage keys. That would fix at most 1 of 19. The reports mostly describe symptoms in user code and the needed files sit in the compiler, with no shared words; matching that takes more than name overlap. The Octocode probe below is the only evidence on whether embeddings do it, and it covers two instances.

A last probe asked whether Octocode's embeddings recover instances the CodeGraph builds miss. I picked the first two vue core instances where both 1.6.2 and the fork found none of the needed files, and ran Octocode 0.26.6 (hybrid search over local embeddings) on them. Octocode rejects queries over 500 characters, so all three tools were rerun with the first 500 characters of the problem statement.

| Instance | Needed files | Octocode | 1.6.2 | Fork |
Expand Down