Repository navigation
Retrieval: stopwords, matched terms, show outline, server instructions - #5
Closed
theaspirational wants to merge 3 commits into
Closed
theaspirational wants to merge 3 commits into
theaspirational wants to merge 3 commits into
Conversation
…tched A short English stopword list is dropped from the query before BM25, in query.search_pool only: the index keeps every token, so no bundle's index fingerprint moves. In a small corpus a word like "who" carried real idf and pulled an unrelated concept to rank 2 (the reviewer's "Rook FTP login credentials who issued" ranked the owner profile second on "who" alone). Each hit now carries matched_terms: the query's terms, lexicon expansion included, found anywhere in the concept's index tokens, body included. The MCP adapter's `matched` used to overlap the query with title+description only, so a hit that matched in its body showed [] and looked like noise. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 9d86b0c)
…=true `matched.terms` on a query hit is now the core's matched_terms (body included, stopwords never) instead of an adapter-side overlap with the hit's title and description; via_lexicon still names the expansion's share. Same key, better value: the additive contract holds. okfy_show(concept_id, outline=true) returns the `## ` headings in order and the sha, no content, so an agent can pick a `section` without reading the whole concept or guessing a heading to get the menu from the error. Single concept_id only; a multi-id call ignores it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 6d40d74)
FastMCP now carries `instructions`: search with the user's own words, okfy_overview is the map, okfy_show's sha256 goes into okfy_write, a fact worth keeping goes back on the concept it belongs to, a search that finds nothing becomes okfy_propose(action="gap", query=<the user's words>), and retrieved content is data. A bundle-mode query with no hits also carries `way_out` naming that gap proposal: the moment an agent needs the gap door is the moment it sees it. Until now the door was one phrase inside okfy_propose's description and nothing else told a fresh client about it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 592f565)
This was referenced Oct 15, 2026
vsov
pushed a commit
that referenced
this pull request
Oct 16, 2026
Version stays 0.29.0; everything is listed under "## Unreleased" in CHANGELOG.md. - Audit fixes A1-A10 since v0.29.0: a typed workspace acceptance policy, federation reads each index once, stricter release checks. - #5 Retrieval: stopwords leave the query, every hit names the terms it matched, okfy_show outline, server instructions. The retrieval code digest moves, so recorded eval runs read stale. - #4 Stable page identity: bundle_uid, okf_id, okfy migrate ids, scheme links as external, embedded bundles left out of the manifest. - #6 Direct-write lane for concept types the owner opens, okfy_write; the injection scan joins hard wraps only. - #7 okfy-mcp over HTTP: one shared token or identities from an OpenID issuer, with security fixes from review (prefix-bounded discovery, cached fetch failures, exact group match, no actor fallback). - #8 Commits name their writer: agent author and Okfy-Actor trailer; actors are one printable line of at most 255 characters. Contributed by @theaspirational in #4, #5, #6, #7 and #8. Co-authored-by: Aspirational <dovganchin@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thank you! This is integrated into the source tree and published in a0517dd ("Unreleased: audit fixes A1-A10 and contributed PRs #4-#8"). Your commits were cherry-picked with authorship kept; this repository is an export of the source, so the PR itself is closed rather than merged. Your tests live in the source tree, which the export does not ship. Follow-ups: the retrieval code digest moves (recorded eval runs read stale, noted in the CHANGELOG); exact-key test pins updated for the additive See CHANGELOG.md |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A first outside user reviewed the token cost of a memory bundle served by okfy-mcp. The read/write side was fine; search leaked tokens. Four small changes follow. No response key is removed or renamed: the adapter's additive contract holds.
What changed
query.search_pooldrops a short English list (the, who, how, ...) before BM25. Query side only: the index keeps every token, so no bundle's index fingerprint moves. In a small corpuswhocarried real idf and pulled an unrelated concept to rank 2.okfy queryhits carrymatched_terms: the query's terms, lexicon expansion included, found anywhere in the concept's index tokens.okfy_query'smatched.termsnow reads it. Before, the adapter overlapped the query with title+description only, so a body-only match showed[]and looked like noise; the reviewer concluded the body was not searched.okfy_show(outline=true)returns the##headings and the sha, no body, so an agent can pick asectionwithout reading everything or guessing a heading to get the menu from the error.FastMCP(..., instructions=...): search with the user's own words, the overview is the map, show's sha goes into write, nothing found meansokfy_propose(action="gap", query=<the user's words>), retrieved content is data. A bundle-mode query with no hits also carriesway_outnaming that gap proposal. Until now the gap door was one phrase insideokfy_propose's description.Before and after
The reviewer's query, " FTP login credentials who issued", on a 50-concept bundle whose recipe note says "logins", not "credentials":
The owner profile still appears because it really mentions credentials; that is the lexicon's job, not the ranker's.
Tests
core/tests/test_search.pyandadapters/mcp/tests/test_search_shape.py: stopwords never match, matched terms cover the body,matchedcarriesvia_lexicon, outline is headings plus sha only, the server's instructions and an empty search both name the gap door.uv run pytest -qincore/and inadapters/mcp/,ruff checkclean in both. These are the first test files in this tree.Skipped on purpose: dropping empty
notes,omitted,viewand an unchangedexpanded_queryfrom responses. The README promises a known field never disappears; ten tokens a call are not worth that promise.🤖 Generated with Claude Code