Skip to content

Bug: namespace store and tree sign the same embedder with different provider labels (managed vs cloud), so every vector written under the other alias is skipped by recall #139

Closed as not planned

Description

@YellowSnnowmann

Version or commit

tinymemory v1.14.1 (52836fd), running as the prebuilt module in openhuman main f5c5ba892. tinycortex vendored at cb1c163.

Rust toolchain

rustc 1.96.1 (aarch64-apple-darwin)

Reproduction

  1. Run openhuman with the managed cloud embedder ([memory] embedding_provider = "managed", the desktop default) on a store that already holds vectors signed with the cloud alias. Such rows come from the host's hard-coded managed embedder, which names itself cloud regardless of the configured provider (Memory sync ignores the configured memory embedding provider and 401s against the cloud embedder on local installs openhuman#5730), or from a config that carried embedding_provider = "cloud".
  2. Sync a connector or write any document, then inspect memory.db:
SELECT model_signature, COUNT(*), SUM(embedding IS NULL) FROM vector_chunks GROUP BY 1;
  1. Compare with the tree sidecar in memory_tree/chunks.db:
SELECT model_signature, COUNT(*) FROM mem_tree_chunk_embeddings GROUP BY 1;

Observed on a live desktop store (2026-09-04):

store signature rows
vector_chunks (namespace store), written after the slug changed to managed (2026-09-01 17:13) provider=managed;model=embedding-v1;dims=1024 8,365
vector_chunks, written before that (label cloud) provider=cloud;model=embedding-v1;dims=1024 12,672
vector_chunks, embed failed, never repaired NULL 30,358
mem_tree_chunk_embeddings (tree), written the same day provider=cloud;model=embedding-v1;dims=1024 1,078

The flip is sharp and matches a settings change, not a release. At 17:10:20 the host logged [embeddings::rpc] update_settings applied provider="managed" model="embedding-v1" dimensions=1024 sig_changed=true requeued=0 (the user switched the Embeddings tab from Ollama to Managed), the app restarted at 17:12:24, the last provider=cloud row is from 17:11:55 and the first provider=managed row landed at 17:13:46. Same endpoint, same model, same dimensions, new label. Two tree sidecar rows written at 17:10:41, between the switch and the restart, are signed provider=cloud;model=nomic-embed-text:latest;dims=768 while holding 1024-wide vectors, which shows the tree signature is assembled from config fields rather than from the embedder that actually ran.

(An earlier draft of this issue blamed the module-driver cut-over. The driver was bound at every boot that day, so that was wrong.)

Expected behavior

One embedder produces one signature everywhere it writes, and a label-only change never hides vectors. Rows whose model and dims match the active embedder stay retrievable (or are re-signed once), and the tree and namespace store agree about what a stored vector is.

Actual behavior

Two code paths spell the provider component differently for the same physical embedder:

  • Tree: crates/tinymemory-core/src/engine/config.rs:60 sets EmbeddingConfig.provider = effective_embedder_slug(config), which is "cloud" for EmbedderChoice::Cloud (tree/score/embed/factory.rs:330); build_cloud_embedder also hard-codes ProviderEmbedder::new(provider, "cloud") (factory.rs:377). tree_active_signature (tinycortex src/memory/chunks/embeddings.rs) formats that → provider=cloud;….
  • Namespace store: crates/tinymemory-core/src/store/factories.rs:264 takes memory.embedding_provider from the host config ("managed") and calls require_embedding_host().create_embedding_provider_with_credentials("managed", …); the module's BusEmbeddingProvider keeps that literal as name() (crates/tinymemory-module/src/embedding.rs:46-49), so the default trait signature() (tinymemory-api/src/host/embeddings.rs:82) yields provider=managed;…. The older rows carry cloud because the host's default_embedding_provider_with_config() hard-codes OpenHumanCloudEmbedding (inner tinyinference CloudEmbeddingModel::name() is "cloud") regardless of the configured provider (Memory sync ignores the configured memory embedding provider and 401s against the cloud embedder on local installs openhuman#5730), and because openhuman's config migrations write both spellings (unify_ai_provider_settings falls back to "cloud", migrate_legacy_embedding_provider writes "managed"). Both aliases exist in the wild, and the tree resolves either of them to EmbedderChoice::Cloud whenever a cloud session exists (factory.rs:184).

The namespace reader (store/namespace_store/query.rs:630-637) does if sig != active_signature { continue } and then if embedding.len() != query_embedding.len() { continue }, so every pre-cutover row is skipped silently. Nothing re-signs or re-embeds vector_chunks (the only writers are the per-document DELETE … WHERE document_id on rewrite and DELETE … WHERE namespace on forget; the reembed_backfill job covers only the tree sidecars). On the store above the global namespace, the one per-turn recall reads, has 0 rows at the active signature (124 null + 24 cloud).

tinycortex#132 unified the format ({model}@{dims} vs provider=…;model=…;dims=…); the label still comes from two sources.

Additional context

Suggested fix, in order:

  1. One label source. Derive the namespace store's provider component from the same resolution the tree uses (effective_embedder_slug), or map the managed/cloud aliases to one canonical slug before format_embedding_signature. "managed" | "cloud" | "openhuman" are already treated as aliases in tree/score/embed/openai_compat.rs:97.
  2. Re-sign, don't re-embed. A one-time migration: rows whose model and dim match the active signature and whose provider is an alias of it get model_signature rewritten. No provider calls, no budget.
  3. Repair the null rows (the openhuman#5300 item 1 sweep): a durable pass over vector_chunks WHERE embedding IS NULL, re-embedding in provider-sized batches and writing only width-matching vectors. Separate PR; listed here because the same store shows 30k of them.
  4. Doctor/health should report a signature split between the two stores; today nothing surfaces it.

Related: tinyhumansai/openhuman#5300 (parent, failure 3), tinyhumansai/openhuman#6025, tinyhumansai/tinycortex#132, #138.

Activity

  1. added
    bugSomething isn't working
    rustPull requests that update rust code
    priority: p1Next. Wrong behaviour a user will hit, or a security weakness behind a condition.
    on Sep 4, 2026
  2. changed the title [-]Bug: namespace store and tree sign the same embedder with different provider labels (managed vs cloud), so every vector written before the module cut-over is skipped by recall[/-] [+]Bug: namespace store and tree sign the same embedder with different provider labels (managed vs cloud), so every vector written under the other alias is skipped by recall[/+] on Sep 4, 2026
  3. YellowSnnowmann commented on Sep 4, 2026

    @YellowSnnowmann
    MemberAuthor

    Corrected the trigger in the body. The label flip on the reference store followed an Embeddings-tab switch from Ollama to Managed (update_settings applied provider="managed" … sig_changed=true at 17:10:20, restart at 17:12:24), not the module-driver cut-over; the driver had been bound at every boot that day. The pre-switch rows were cloud-labelled cloud vectors because of the hard-coded managed embedder in tinyhumansai/openhuman#5730. Root cause, numbers and proposed fix are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: p1Next. Wrong behaviour a user will hit, or a security weakness behind a condition.rustPull requests that update rust code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions