Best approach for per-query vector-search tool selection to reduce LLM token usage + user-feedback loop for embedding quality

[user1500177]

I'm building an agent using agno that connects to an MCP gateway (with help of the consumer tag i already filter it down to 20 or 30 tools) . Sending all tool schemas to the LLM on every request wastes tokens, confuses routing, and inflates latency - so only against the user query i want to actually send the tools to LLM

I have an internal embeddings service that, given a user query, returns the top_k most relevant tool names:

POST https://embeddings.example.internal/api/v1/embeddings/tools/match
{
"user_query": "get names of the employee",
"top_k": 5
}

→ {
"matches": ["get_emplyee_list", "get_employee_details"],
"total": 2,
"user_query": "lget names of the employee"
}

Now another endpoint :
POST https://embeddings.example.internal/api/v1/embeddings/tools/register
{
"tool_name": "get_weather",
"example_queries": ["what is the whetehr today", "is it rainy day"]
}

can i make the follwing :

Per-run tool injection:
On each run, call the embeddings endpoint with the user query, get top_k matching tool names, and give the LLM only those tools — not the full 100+. [Prehook , Or use Both prehook and dynamic tools for this purpose ??, and make sure each time ony the filtered tool list form the gateway is passed via that ? IS such a thing possible so taht at runtime its configurable ; POINT To not i have already enabled the refresh_connection etc ]

Empty-match safety: If embeddings return nothing (cold start, endpoint down), fall back to full tool set so the agent still works.

User satisfaction feedback loop: After a successful tool call, ask the user "Was this the right tool?" and if confirmed, register that (query, tool_name) pair to the embeddings store to improve future matching. (For THIS how to handle the best way is it Supposed to raise a custom even with the tool name and the query and make the UI call the endpoint ; or is htere a better concept in Agno for this specific use case - IS POST HOOK a place to do this - but still i need to get the user feed back and be able to know whether to call the embedding register endpoint OR not RIGHT ; so what is the best thing i did not find that the post hook could be used for event emmission thoug - its in docs like telling only output validation etc ; Is there any other way of making sure (My Agno agnet is stateless and NOT connected to DB))

I use the agui endpoint in the UI side. so what is the best strategy for this ? requirement //

[user1500177]

Hi @kausmeows , @dirkbrnd

is there a way to do this in the best possible manner ?

Would it be better if i enabled a in memory db ?

Currenlty i am trying to kind of store the top k tool names in the session state (in a prehook i always hit the enpoint and do it)
After that i use that tool name in the dynamic tools to filter out the and keep those tools that come from my endpoint (as fallback all tools enabled if endpoint of searching in the vector DB fails);
So then i will get like the correct tool against the user query ; which might actually reduce some token which are sent to the LLM and also like get a better understanding of the right tool to call maybe right?

And now about the passing feedback so that i can update the vecotr db which might be the best possible way to do this (i thought about raising the data relate to query and the tools called as a customevent via Agui event , and then let UI call an endpoint for updating to the DB and has almost done this) [but this would take some more time right is there a better way of doing the same in Agno or any other possibilities or is this one a good approach] - will enabling a in memory storage be of use ?

[Ghraven]

I am not a maintainer, but the shape you described sounds reasonable. I would just be careful about where the source of truth lives.

The pattern I would try is:

1. Keep the vector search as a router step, not as the final authority. Return a small candidate set of tool names plus scores/reasons.
2. Store that candidate set in session state for observability and reuse during the same turn/run.
3. Use dynamic tools to expose only the matched tools, with a conservative fallback to all tools when the router fails or the scores are weak.
4. Keep the actual tool metadata in one registry/catalog, then index a derived representation of that metadata. That way the vector DB can be rebuilt instead of becoming a second, drifting source of truth.

I would not use an in-memory DB as the main solution unless this is only for one process/session. It can help as a short-lived cache for the current run, but if you want feedback to improve future routing, I would persist feedback separately as training/eval data: query, candidate tools, selected tool, actual called tool, success/failure, and maybe a short user/developer label.

For the feedback loop, your AG-UI custom event idea seems like a good boundary. I would probably make it asynchronous/offline rather than updating embeddings immediately in the request path. Immediate updates can make behavior harder to debug because the router changes while you are still evaluating it. A batch job that reviews feedback and refreshes the tool index is easier to reason about.

The main guardrail I would add: if the top score is below a threshold or the top two tools are close, either expose a slightly larger candidate set or fall back. Reducing tokens is useful, but hiding the right tool is usually more expensive than sending a few extra schemas.