Skip to content

Retain valid model results across collection updates - #1096

Draft
dlukeomalley wants to merge 2 commits into
tobi:mainfrom
dlukeomalley:fix/model-cache-lifecycle
Draft

dlukeomalley wants to merge 2 commits into
tobi:mainfrom
dlukeomalley:fix/model-cache-lifecycle

Conversation

@dlukeomalley

Copy link
Copy Markdown

Collection updates currently erase the entire model-result cache, even when nothing changed. Both CLI indexing paths and the public SDK update do this. Repeated searches therefore lose reusable query expansions and rerank scores.

Keep those entries through indexing and qualify the rerank cache at its owner:

  • Key scores by the effective intent-prefixed query, chunk content, resolved model and actual context token bound. Read only the new versioned keys; old entries omit facts needed to establish reuse.
  • Persist a score only when the returned model matches the selected model and is not the explicit fallback sentinel. Keep current-call fallback results, and keep genuine model scores of 0.5 cacheable.
  • Read context size from the same LlamaCpp getter used by ranking-context allocation and document truncation.
  • Remove the now-unused duplicate clearCache API. Explicit Maintenance.clearLLMCache and cleanup still use deleteLLMCache; pruning and empty-expansion eviction remain.

This retains model results, not retrieved documents or final rankings. Index scans, fresh retrieval and fusion still execute. Replacing model weights under an unchanged URI or changing future prompt/scoring implementations still requires a cache version change or explicit cleanup. A successful local model literally named fallback conservatively receives no persistent reuse because that name collides with the existing degraded-result marker.

Regression witnesses failed before correction for SDK/CLI invalidation, fallback persistence, legacy intent and context-size changes. Tests use actual stores and LlamaCpp instances with public model-method spies; the context boundary runs in two processes. No model-quality or latency improvement is inferred from these deterministic controls.

Validation at 4a6d1b0:

  • Full TypeScript check including source, tests and helper workers; Oxlint.
  • Project test script: Node 1,544 passed / 80 skipped; Bun 1,540 passed / 88 skipped.
  • Package build, grammar checks and compiled CLI/wrapper smoke under Node and Bun.
  • Independent implementation and installed language reviews; duplicate API finding removed.

Model-dependent integration cases remain skipped in CI mode. The CLI update cache control uses an empty collection registry; populated update/reuse is covered through the SDK, and collection-add indexes real fixtures. These checks do not establish live inference correctness or broad retrieval quality.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant