Skip to content

[OMEGA-501] fix(rag): bound cloud embedding request batches - #382

Merged
paul-v-snet merged 3 commits into
singnet:mainfrom
Milbaxter:fix/bound-cloud-embedding-batches
Oct 16, 2026
Merged

paul-v-snet merged 3 commits into
singnet:mainfrom
Milbaxter:fix/bound-cloud-embedding-batches

Conversation

@Milbaxter

Copy link
Copy Markdown

Description

Fixes #366.

Runtime knowledge loading sends all chunks of one source file to the embedding API in one request. A large file can exceed the request limit and stop the knowledge load before later files are indexed.

cloud_embed_batch now sends at most 32 texts per request and joins the returned vectors in input order. The new embeddingBatchSize setting lets operators tune the request size for their provider. It accepts positive integers from YAML or command-line/environment strings. Empty input makes no request; a failed batch raises the existing error without returning a partial result.

The OpenAI embedding API reference sets an 8192-token limit per input and a 300,000-token limit per request. With individually valid inputs, the default caps a request at 262,144 tokens and 32 inputs. Custom sizes must fit the provider's limits. Individual overlong texts are still the chunker's responsibility; this change does not truncate them.

How Has This Been Tested?

  • Reproduced the original failure with a mock endpoint that enforces the 300,000-token request limit for simulated 8192-token inputs. The 70-input call failed before the change; it now sends batches of 32, 32, and 6 and returns all vectors in order.
  • python -m pytest -q tests: 128 passed, 5 skipped on macOS/Python 3.12.12. The skips are four Linux-only cases and the optional import-kb model comparison (dependency not installed).
  • python -m pytest -q --noconftest Autotests/test_openai_runtime_embeddings.py: 6 passed. These retain provider/model routing and single-memory embedding behavior. --noconftest omits the live-agent cleanup fixture.
  • The new tests also cover ASICloud routing with custom batch sizes, CLI string values, invalid settings, empty input, and failure in a later batch.
  • git diff --check: passed.

No live embedding API requests were made. The Docker daemon is unavailable on this host, so the mandatory container scenario suite and MeTTa tests were not run locally. This PR is a draft pending that CI validation.

Checklist

  • PR contains autogenerated code
  • Self-review completed
  • Test scenarios above are passed with the version of the code from PR

@paul-v-snet paul-v-snet changed the title fix(rag): bound cloud embedding request batches [OMEGA-382] fix(rag): bound cloud embedding request batches Oct 14, 2026
@paul-v-snet
paul-v-snet marked this pull request as ready for review October 14, 2026 10:45
@aquashade aquashade changed the title [OMEGA-382] fix(rag): bound cloud embedding request batches [OMEGA-501] fix(rag): bound cloud embedding request batches Oct 14, 2026
@paul-v-snet

Copy link
Copy Markdown
Collaborator

@Milbaxter,

We've reviewed your PR and approved it.
Now it has been moved to QA. The QA team will report the results here.

Thank you for your contribution.

@MorozovMisha

Copy link
Copy Markdown

Tested: PR #382 at commit 48c39a3

What I checked

  • Request batching: cloud_embed_batch caps request slices at 32 texts by default, staying safely below the 300,000-token API limit.
  • Result integrity: embeddings across chunked batches are concatenated in the exact input order.
  • Configuration: embeddingBatchSize is documented in reference-configuration.md and strictly validates positive integers.
  • Automated tests: all 5 unit tests in tests/test_rag_embeddings.py pass cleanly.
    Verdict: PASS

@paul-v-snet
paul-v-snet merged commit 1a7624e into singnet:main Oct 16, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OMEGA-501] cloud_embed_batch sends one request per source file, which exceeds the embeddings API's 300,000-token per-request limit

4 participants