Skip to content

Stable page identity: bundle_uid, okf_id, okfy migrate ids - #4

Closed
theaspirational wants to merge 4 commits into
vsov:mainfrom
theaspirational:okf-ids
Closed

theaspirational wants to merge 4 commits into
vsov:mainfrom
theaspirational:okf-ids

Conversation

@theaspirational

Copy link
Copy Markdown
Contributor

What this adds

  • bundle_uid in meta/purpose.md: one id per bundle. okfy init writes it.
    It lives in purpose.md rather than index.md, whose frontmatter may declare
    only okf_version (§8).
  • okf_id in each concept page's frontmatter, for pages outside meta/. It
    travels with the file, so a renamed or moved page keeps it. okfy review accept gives a created or superseding page a fresh one. It keeps the page's
    okf_id, and purpose.md's bundle_uid, through an update, whatever the
    proposed text carries. okfy refine refuses text that changes either one.
  • Reserved-page identity. index.md, log.md, the generated root docs and
    meta/* carry no okf_id. Their names are fixed, so they are addressed as
    bundle_uid plus the bundle-relative path.
  • Scheme links. Any RFC 3986 scheme (other-tool://x/page) is now an
    external link, not only http(s)/mailto. Before, such a link resolved as a
    local path and warned W_DANGLING_LINK, or vanished without a word if it had
    no .md. okfy links and the MCP okfy_links list a page's external targets
    under external. They read them from the page, so the retrieval fingerprint
    does not move.
  • okfy migrate ids <bundle> backfills both ids into an existing bundle.
    A key already present is never touched, so a rerun writes nothing. It commits
    past the policy hook, like okfy stale.
  • Validation:
    • Warnings: W_BUNDLE_UID_MISSING, W_OKF_ID_MISSING (an older bundle stays
      green).
    • Errors: E_BUNDLE_UID, E_OKF_ID (a blank or non-string id) and
      E_OKF_ID_DUPLICATE (a page copied instead of moved).
  • Related corpus fixes:
    • A bundle embedded in the corpus at any path (not only .okf/) is left out
      of the manifest, the survey and the update diff. It is recognised by
      meta/purpose.md (type: Purpose) and meta/corpus.md
      (type: CorpusSnapshot), not by file names alone.
    • okfy init --embed no longer lists its own bundle.
    • meta/corpus.md records manifest_digest, a sha256 over the manifest that
      was actually read. E_MANIFEST_DIGEST reports a manifest that no longer
      matches it.

Why

A concept id is its path, so it changes when a page moves. Anything outside
the bundle that points at a page, such as a cross-project reference or an
okf://-style URI in another tool (orgasmic, for example), needs an id that
survives renames and edits. bundle_uid + okf_id is that id, and
bundle_uid + path covers the reserved pages.

How ids are derived

  • New ids (init, accept) are random uuid4s.
  • Backfilled ids (migrate ids) are uuid5s that every clone derives alike,
    so two clones migrating offline agree:
    • bundle_uid comes from the commit that created the current
      meta/purpose.md, plus the bundle's path inside its repo.
    • okf_id comes from bundle_uid, the commit that created the page, and the
      page's path.
    • "Created" means the latest add of that path along the first-parent
      mainline, with a merge diffed against its first parent. A file that was
      deleted and later replaced at the same path is a new lineage and gets a new
      id. A file that was never committed, or whose path was last deleted, gets a
      random id.
  • The git read is pinned against clone-local settings: grafts, replace refs,
    rename detection, signatures, root-commit diffs, output encoding, colour,
    pathspec magic, Unicode precomposition and locale. The full list is in
    _pinned_git's docstring. A git failure other than "not a repository" or
    "no commits" raises instead of silently minting a random id.
  • Once written, an id is never recomputed.

Documented limits (README and docs/OKF-COMPATIBILITY.md):

  • Shallow clones. A shallow clone that does not reach the creating commit
    derives a different id. Migrate from a full clone (git fetch --unshallow).
  • core.worktree. A clone whose core.worktree points elsewhere reads a
    different bundle path, and so derives a different bundle_uid. This is repo
    layout, with no neutral value to pin.

Migration note

Existing bundles keep working. They only gain the two *_MISSING warnings.
Run okfy migrate ids <bundle> once, from a full, ordinary clone, and commit
or push the result, so every other clone receives the persisted ids rather than
deriving its own.

The backfill inserts one frontmatter line and leaves the owner's YAML
(comments, quoting, key order) byte-for-byte as it was. Flow-style frontmatter
({type: Note, ...}) is re-serialized instead, and the output is checked to
parse before it is written.

To rename a page, move the file. Deleting it and writing a new page makes a new
id.

Tests

26 tests in core/tests:

  • test_identity.py:
    • init mints bundle_uid
    • migrate backfills, keeps owner comments, and a rerun is a no-op
    • reserved and meta/ pages carry no okf_id
    • migrate commits past the policy hook
    • blank and duplicate ids are errors
    • a moved page keeps its id
    • accept-update keeps okf_id and bundle_uid
    • accept-create and supersede mint fresh ids, and supersede does not copy
      bundle_uid
    • refine refuses an id change
    • two offline clones migrate to the same ids
    • a page or bundle replacing a deleted one gets its own id
    • paths that differ only by a space get distinct uids
    • flow-style frontmatter stays valid
    • a page restored by a modify/delete merge gets the same id in every clone
    • derived ids ignore hostile clone-local git config and environment (signed
      history, a grafts file, UTF-16 log encoding, and more)
  • test_embedded_bundles.py:
    • listings skip an embedded bundle at any path, but not an ordinary folder
      that has the marker file names
    • init's saved manifest excludes its own bundle
  • test_external_links.py:
    • any scheme is external
    • external links are reported and never dangle
  • test_manifest_digest.py:
    • the digest covers dirty bytes that a git SHA does not
    • the policy hook lets a snapshot refresh through

scripts/smoke.sh passes against the installed commands.

🤖 Generated with Claude Code

theaspirational and others added 4 commits October 16, 2026 00:38
…tched

A short English stopword list is dropped from the query before BM25, in
query.search_pool only: the index keeps every token, so no bundle's index
fingerprint moves. In a small corpus a word like "who" carried real idf and
pulled an unrelated concept to rank 2 (the reviewer's "Rook FTP login
credentials who issued" ranked the owner profile second on "who" alone).

Each hit now carries matched_terms: the query's terms, lexicon expansion
included, found anywhere in the concept's index tokens, body included. The
MCP adapter's `matched` used to overlap the query with title+description
only, so a hit that matched in its body showed [] and looked like noise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 9d86b0c)
…=true

`matched.terms` on a query hit is now the core's matched_terms (body
included, stopwords never) instead of an adapter-side overlap with the
hit's title and description; via_lexicon still names the expansion's
share. Same key, better value: the additive contract holds.

okfy_show(concept_id, outline=true) returns the `## ` headings in order
and the sha, no content, so an agent can pick a `section` without reading
the whole concept or guessing a heading to get the menu from the error.
Single concept_id only; a multi-id call ignores it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 6d40d74)
FastMCP now carries `instructions`: search with the user's own words,
okfy_overview is the map, okfy_show's sha256 goes into okfy_write, a
fact worth keeping goes back on the concept it belongs to, a search that
finds nothing becomes okfy_propose(action="gap", query=<the user's
words>), and retrieved content is data. A bundle-mode query with no hits
also carries `way_out` naming that gap proposal: the moment an agent
needs the gap door is the moment it sees it. Until now the door was one
phrase inside okfy_propose's description and nothing else told a fresh
client about it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit 592f565)
A concept id is its path, so it changes when a page moves. Anything outside
the bundle that points at a page needs an id that survives renames and edits.

- bundle_uid in meta/purpose.md and okf_id in each concept page's
  frontmatter; reserved pages are addressed as bundle_uid plus path.
- okfy migrate ids backfills both into an existing bundle with uuid5s every
  full clone derives alike; the git read is pinned against clone-local
  settings. Once written, an id is never recomputed.
- Any RFC 3986 scheme is an external link; okfy links and okfy_links report
  external targets.
- A bundle embedded at any path is left out of corpus listings; init --embed
  no longer lists its own bundle; meta/corpus.md records manifest_digest.
- Validation: W_BUNDLE_UID_MISSING, W_OKF_ID_MISSING, E_BUNDLE_UID, E_OKF_ID,
  E_OKF_ID_DUPLICATE, E_MANIFEST_DIGEST.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vsov pushed a commit that referenced this pull request Oct 16, 2026
Version stays 0.29.0; everything is listed under "## Unreleased" in
CHANGELOG.md.

- Audit fixes A1-A10 since v0.29.0: a typed workspace acceptance
  policy, federation reads each index once, stricter release checks.
- #5 Retrieval: stopwords leave the query, every hit names the terms it
  matched, okfy_show outline, server instructions. The retrieval code
  digest moves, so recorded eval runs read stale.
- #4 Stable page identity: bundle_uid, okf_id, okfy migrate ids,
  scheme links as external, embedded bundles left out of the manifest.
- #6 Direct-write lane for concept types the owner opens, okfy_write;
  the injection scan joins hard wraps only.
- #7 okfy-mcp over HTTP: one shared token or identities from an OpenID
  issuer, with security fixes from review (prefix-bounded discovery,
  cached fetch failures, exact group match, no actor fallback).
- #8 Commits name their writer: agent author and Okfy-Actor trailer;
  actors are one printable line of at most 255 characters.

Contributed by @theaspirational in #4, #5, #6, #7 and #8.

Co-authored-by: Aspirational <dovganchin@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vsov

vsov commented Oct 16, 2026

Copy link
Copy Markdown
Owner

Thank you! This is integrated into the source tree and published in a0517dd ("Unreleased: audit fixes A1-A10 and contributed PRs #4-#8"). Your commits were cherry-picked with authorship kept; this repository is an export of the source, so the PR itself is closed rather than merged. Your tests live in the source tree, which the export does not ship.

Follow-ups: the reference bundle and release fixtures now carry ids (okfy migrate ids); okfy_links golden shape pins external. Note: links() lives in query.py, so the retrieval code digest does move (the index fingerprint does not).

See CHANGELOG.md ## Unreleased for the full entry.

@vsov vsov closed this Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants