Body
Hi SK team — I've built on the SK ecosystem before ([one line: name the project/contribution]), so this comes from a friendly place: I just finished a construct-validity audit of embedding-cosine gates across agent frameworks, and SK's memory-recall path has the same test blind spot I found in four other suites. Preprint: https://arxiv.org/abs/2608.10216.
The pattern: SemanticTextMemory.search gates results on min_relevance_score (default 0.0 — everything passes) and TextMemoryPlugin.recall gates on DEFAULT_RELEVANCE = 0.75. Both implement the same decision — "is this stored text relevant to (i.e. does it agree with / answer) this query?" — by thresholding embedding similarity. That score measures shared wording, not shared meaning, and in the pairs a recall gate most needs to rank correctly the two run opposite ways: reversing an instruction is a one-token edit that keeps similarity near-maximal, while a faithful restatement in fresh words drops it substantially. Neither the memory tests nor the plugin tests contain a case where two texts share surface form and mean opposite things — so a recall that surfaces "administer the drug" for a query about "withhold the drug" is invisible to CI. To be precise about framing: nothing here is a bug in SK's code; the gate does exactly what it says. The blind spot is in what the test suite asserts the gate means.
Real specimen from my audit corpus (cosine under nomic-embed-text-v1.5, MRL-256; frozen results in the paper's artifact; the mutation class scored 0.83–0.9997 across all nine encoders tested, so this is not one model's quirk):
- Stored memory: "Withhold the study drug from any participant who reports chest tightness."
- Query: "Administer the study drug to any participant who reports chest tightness."
- Cosine 0.9608 — sails past 0.75, and past any plausible relevance floor. At the 0.0 default it isn't even a speed bump; at 0.75 it ranks as a strong recall.
- Mirror case: "Retry the request at most three times." vs. the equivalent "Give the call up to three attempts, then stop." averages ≈ 0.76 — right at the 0.75 plugin cut, so faithful restatements sit on the knife edge while reversals clear it comfortably.
Concrete proposal, sized to be PR-able (I'm happy to write it):
- Add polarity test cases to the memory/search test suites — pairs sharing wording with opposite decisions (negation,
must→may, quantity changes, scope inversion) asserted at the shipped defaults, and pairs sharing the decision with no shared wording. Section 11.3 of the paper gives the two case families; the corpus pairs are released and were authored blind to any encoder.
- Carry the same fixtures into the successor
collection.as_text_search / vector-store search path, since the deprecation migration otherwise ports the blind spot forward.
- Optionally: a one-line docstring note on
min_relevance_score/relevance that the score is surface-form similarity, so "relevant" should not be read as "in agreement."
The audit found the identical gap in LlamaIndex, LangChain, and GPTCache-style caches — SK is in good company, which is rather the point: the failure mode ships because no framework's tests can see it. Glad to discuss and to send the PR.
\
Body
Hi SK team — I've built on the SK ecosystem before ([one line: name the project/contribution]), so this comes from a friendly place: I just finished a construct-validity audit of embedding-cosine gates across agent frameworks, and SK's memory-recall path has the same test blind spot I found in four other suites. Preprint: https://arxiv.org/abs/2608.10216.
The pattern:
SemanticTextMemory.searchgates results onmin_relevance_score(default 0.0 — everything passes) andTextMemoryPlugin.recallgates onDEFAULT_RELEVANCE = 0.75. Both implement the same decision — "is this stored text relevant to (i.e. does it agree with / answer) this query?" — by thresholding embedding similarity. That score measures shared wording, not shared meaning, and in the pairs a recall gate most needs to rank correctly the two run opposite ways: reversing an instruction is a one-token edit that keeps similarity near-maximal, while a faithful restatement in fresh words drops it substantially. Neither the memory tests nor the plugin tests contain a case where two texts share surface form and mean opposite things — so a recall that surfaces "administer the drug" for a query about "withhold the drug" is invisible to CI. To be precise about framing: nothing here is a bug in SK's code; the gate does exactly what it says. The blind spot is in what the test suite asserts the gate means.Real specimen from my audit corpus (cosine under
nomic-embed-text-v1.5, MRL-256; frozen results in the paper's artifact; the mutation class scored 0.83–0.9997 across all nine encoders tested, so this is not one model's quirk):Concrete proposal, sized to be PR-able (I'm happy to write it):
must→may, quantity changes, scope inversion) asserted at the shipped defaults, and pairs sharing the decision with no shared wording. Section 11.3 of the paper gives the two case families; the corpus pairs are released and were authored blind to any encoder.collection.as_text_search/ vector-store search path, since the deprecation migration otherwise ports the blind spot forward.min_relevance_score/relevancethat the score is surface-form similarity, so "relevant" should not be read as "in agreement."The audit found the identical gap in LlamaIndex, LangChain, and GPTCache-style caches — SK is in good company, which is rather the point: the failure mode ships because no framework's tests can see it. Glad to discuss and to send the PR.
\