Skip to content

Retrieval: stopwords, matched terms, show outline, server instructions - #5

Closed
theaspirational wants to merge 3 commits into
vsov:mainfrom
theaspirational:pr/retrieval
Closed

theaspirational wants to merge 3 commits into
vsov:mainfrom
theaspirational:pr/retrieval

Conversation

@theaspirational

Copy link
Copy Markdown
Contributor

A first outside user reviewed the token cost of a memory bundle served by okfy-mcp. The read/write side was fine; search leaked tokens. Four small changes follow. No response key is removed or renamed: the adapter's additive contract holds.

What changed

  • Stopwords leave the query. query.search_pool drops a short English list (the, who, how, ...) before BM25. Query side only: the index keeps every token, so no bundle's index fingerprint moves. In a small corpus who carried real idf and pulled an unrelated concept to rank 2.
  • Every hit says what it matched, body included. okfy query hits carry matched_terms: the query's terms, lexicon expansion included, found anywhere in the concept's index tokens. okfy_query's matched.terms now reads it. Before, the adapter overlapped the query with title+description only, so a body-only match showed [] and looked like noise; the reviewer concluded the body was not searched.
  • okfy_show(outline=true) returns the ## headings and the sha, no body, so an agent can pick a section without reading everything or guessing a heading to get the menu from the error.
  • The server introduces itself. FastMCP(..., instructions=...): search with the user's own words, the overview is the map, show's sha goes into write, nothing found means okfy_propose(action="gap", query=<the user's words>), retrieved content is data. A bundle-mode query with no hits also carries way_out naming that gap proposal. Until now the gap door was one phrase inside okfy_propose's description.

Before and after

The reviewer's query, " FTP login credentials who issued", on a 50-concept bundle whose recipe note says "logins", not "credentials":

rank before after
1 recipe note 17.5, matched: the agent's name recipe note 14.8, matched: ftp, login, the agent's name
2 owner profile 7.4, matched: who infrastructure 4.6, matched: credentials, login
3 infrastructure 4.6, matched: nothing owner profile 4.5, matched: credentials, issued

The owner profile still appears because it really mentions credentials; that is the lexicon's job, not the ranker's.

Tests

core/tests/test_search.py and adapters/mcp/tests/test_search_shape.py: stopwords never match, matched terms cover the body, matched carries via_lexicon, outline is headings plus sha only, the server's instructions and an empty search both name the gap door. uv run pytest -q in core/ and in adapters/mcp/, ruff check clean in both. These are the first test files in this tree.

Skipped on purpose: dropping empty notes, omitted, view and an unchanged expanded_query from responses. The README promises a known field never disappears; ten tokens a call are not worth that promise.

🤖 Generated with Claude Code

theaspirational and others added 3 commits October 16, 2026 00:38
…tched

A short English stopword list is dropped from the query before BM25, in
query.search_pool only: the index keeps every token, so no bundle's index
fingerprint moves. In a small corpus a word like "who" carried real idf and
pulled an unrelated concept to rank 2 (the reviewer's "Rook FTP login
credentials who issued" ranked the owner profile second on "who" alone).

Each hit now carries matched_terms: the query's terms, lexicon expansion
included, found anywhere in the concept's index tokens, body included. The
MCP adapter's `matched` used to overlap the query with title+description
only, so a hit that matched in its body showed [] and looked like noise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9d86b0c)
…=true

`matched.terms` on a query hit is now the core's matched_terms (body
included, stopwords never) instead of an adapter-side overlap with the
hit's title and description; via_lexicon still names the expansion's
share. Same key, better value: the additive contract holds.

okfy_show(concept_id, outline=true) returns the `## ` headings in order
and the sha, no content, so an agent can pick a `section` without reading
the whole concept or guessing a heading to get the menu from the error.
Single concept_id only; a multi-id call ignores it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 6d40d74)
FastMCP now carries `instructions`: search with the user's own words,
okfy_overview is the map, okfy_show's sha256 goes into okfy_write, a
fact worth keeping goes back on the concept it belongs to, a search that
finds nothing becomes okfy_propose(action="gap", query=<the user's
words>), and retrieved content is data. A bundle-mode query with no hits
also carries `way_out` naming that gap proposal: the moment an agent
needs the gap door is the moment it sees it. Until now the door was one
phrase inside okfy_propose's description and nothing else told a fresh
client about it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 592f565)
vsov pushed a commit that referenced this pull request Oct 16, 2026
Version stays 0.29.0; everything is listed under "## Unreleased" in
CHANGELOG.md.

- Audit fixes A1-A10 since v0.29.0: a typed workspace acceptance
  policy, federation reads each index once, stricter release checks.
- #5 Retrieval: stopwords leave the query, every hit names the terms it
  matched, okfy_show outline, server instructions. The retrieval code
  digest moves, so recorded eval runs read stale.
- #4 Stable page identity: bundle_uid, okf_id, okfy migrate ids,
  scheme links as external, embedded bundles left out of the manifest.
- #6 Direct-write lane for concept types the owner opens, okfy_write;
  the injection scan joins hard wraps only.
- #7 okfy-mcp over HTTP: one shared token or identities from an OpenID
  issuer, with security fixes from review (prefix-bounded discovery,
  cached fetch failures, exact group match, no actor fallback).
- #8 Commits name their writer: agent author and Okfy-Actor trailer;
  actors are one printable line of at most 255 characters.

Contributed by @theaspirational in #4, #5, #6, #7 and #8.

Co-authored-by: Aspirational <dovganchin@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vsov

vsov commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Thank you! This is integrated into the source tree and published in a0517dd ("Unreleased: audit fixes A1-A10 and contributed PRs #4-#8"). Your commits were cherry-picked with authorship kept; this repository is an export of the source, so the PR itself is closed rather than merged. Your tests live in the source tree, which the export does not ship.

Follow-ups: the retrieval code digest moves (recorded eval runs read stale, noted in the CHANGELOG); exact-key test pins updated for the additive matched_terms and way_out; GUIDE text on matched corrected.

See CHANGELOG.md ## Unreleased for the full entry.

@vsov vsov closed this Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants