Repository navigation
Conversation
Contributor
Author
|
@rzo1 I needed to make this to test the embeddings. I know before you were talking about expanding something like this/unifying it.. so I'm just keeping as a draft now. I'll add more details to the ticket a lil later |
krickert
force-pushed
the
feat/embeddings-timings-tests
branch
from
October 11, 2026 12:20
b26898c to
5bcfe05
Compare
krickert
force-pushed
the
feat/GH-1391-embed-batch-cap
branch
from
October 11, 2026 22:39
dc68fd7 to
f2a2206
Compare
krickert
force-pushed
the
feat/embeddings-timings-tests
branch
from
October 11, 2026 22:40
5bcfe05 to
926e192
Compare
krickert
force-pushed
the
feat/GH-1391-embed-batch-cap
branch
from
October 12, 2026 01:20
f2a2206 to
e540ab2
Compare
…rs off the heap SentenceVectorsDLBenchmark times embed per text and embedAll for calls of 1 to 128 texts of mixed length, on the CPU or CUDA device 0. The opennlp-dl module gets the same jmh profile as opennlp-runtime, and BENCHMARKS.md documents how to run it and the allocation numbers. Each inference copied the inputs from long arrays into heap buffers, which ONNX Runtime copied again into direct memory, and fetched the output through getValue(), which builds one float array per token. The inputs are now written into direct buffers, and ONNX Runtime writes the output into a direct buffer that pooling uses in place. Only the requested output is computed. SentenceVectorsDLBenchmark with -prof gc on all-MiniLM-L6-v2: heap allocation per call is 58 to 69 percent lower on CPU and GPU, for example embedAll of 128 texts on the GPU drops from 7904 KB to 2424 KB. Time per call on the GPU is unchanged. Signed-off-by: Kristian Rickert <kristian@apache.org> Co-authored-by: Claude Code 2.1.296 (Opus 5.5)
krickert
force-pushed
the
feat/embeddings-timings-tests
branch
from
October 12, 2026 01:24
926e192 to
7ecbba4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1471
Allocation per call (
gc.alloc.rate.norm, all-MiniLM-L6-v2, JDK 21, RTX 4080 SUPER, 1 fork x 5 iterations), before and after the inputs and outputs moved to direct buffers:embedAllembedAllembedAllembedAllembedEachembedEachembedEachembedEachAverage time per call on the GPU did not change (it is bound by inference).
embedAllat callSize 128 takes 23.8 ms against 67.4 ms forembedEach. CPU times vary by more than 50 percent between iterations with ONNX Runtime's default thread settings, so they are not listed.