Skip to content

GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix the first CUDA request in a fresh JVM - #1462

Open
mawiesne wants to merge 1 commit into
mainfrom
OPENNLP-1982-sentence-vectors-inference-options
Open

GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix the first CUDA request in a fresh JVM#1462
mawiesne wants to merge 1 commit into
mainfrom
OPENNLP-1982-sentence-vectors-inference-options

Conversation

@mawiesne

@mawiesne mawiesne commented Oct 9, 2026 •
edited by krickert
Loading

Copy link
Copy Markdown
Contributor

SentenceVectorsDL always created default session options, so it ran on the CPU even with opennlp-dl-gpu. A new constructor takes InferenceOptions, like NameFinderDL and DocumentCategorizerDL, and uses the GPU settings and the lower case setting from them. A lower case setting in the options takes precedence over the lowerCase argument. The existing constructors keep their behavior.

AbstractDL.sessionOptions added the CUDA provider before any OrtEnvironment existed. Loading the provider needs ONNX Runtime's default logger, which only exists once an environment does, so the first GPU request in a fresh JVM failed with "Attempt to use DefaultLogger but none has been registered". The environment is now created first. This affects NameFinderDL and DocumentCategorizerDL as well.

A test runs the first CUDA request in a child JVM. Other tests request a device id no machine has and check that ONNX Runtime rejects it, and compare vectors on CUDA device 0 with the CPU session (skipped without a CUDA provider or device). The opennlp-dl-gpu test classpath no longer has the CPU onnxruntime next to onnxruntime_gpu. The machine learning chapter documents the GPU setup. ONNX Runtime does not fall back to the CPU when CUDA or the device is unavailable; the constructor throws.

Tested in opennlp-dl-gpu on an RTX 4080 SUPER inside nvidia/cuda:12.8.1-cudnn with --gpus all: all 1214 tests pass and none are skipped. A host with CUDA 12.4 cannot load the ONNX Runtime 1.30 GPU build.

Fixes #1389
Fixes #1390

@mawiesne mawiesne added this to the 3.0.0 milestone Oct 9, 2026
@mawiesne
mawiesne requested a review from rzo1 October 9, 2026 09:30
@mawiesne mawiesne added bug Something does not work as documented (Jira: Bug) component: ml Machine learning (maxent, perceptron, naive bayes, libsvm) labels Oct 9, 2026
@rzo1
rzo1 marked this pull request as draft October 9, 2026 09:31
@rzo1
rzo1 force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from 3d7395d to fb06aef Compare October 9, 2026 09:33
mawiesne
mawiesne commented Oct 9, 2026
View reviewed changes
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/ColdStartCudaProbe.java Outdated
@mawiesne

mawiesne commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

This needs backporting for OpenNLP 2.x branch. Pls, @rzo1 or @krickert provide such PR once this one is fine and lands on main.

@mawiesne mawiesne changed the title OPENNLP-1982: Allow for sentence vectors inference options OPENNLP-1982: Fix SentenceVectorsDL cannot run on a GPU Oct 9, 2026
@krickert

krickert commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

I'll get this out of draft tonight. Thanks for pushing it. It's a cool feature

@krickert

krickert commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

Working through this now.

@krickert
krickert force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from fb06aef to 576206d Compare October 11, 2026 01:55
@krickert krickert changed the title OPENNLP-1982: Fix SentenceVectorsDL cannot run on a GPU GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix the first CUDA request in a fresh JVM Oct 11, 2026
@krickert
krickert marked this pull request as ready for review October 11, 2026 02:06
@krickert krickert mentioned this pull request Oct 11, 2026
Draft
@krickert
krickert force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from ea2c0a8 to bfaf52b Compare October 11, 2026 10:03
rzo1
rzo1 requested changes Oct 11, 2026
View reviewed changes

@rzo1 rzo1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @krickert, thanks for the PR. The change to AbstractDL and the new constructor look good to me. Some remarks inline.

No eval build is needed, since no model and no default behaviour changes.

Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java Outdated
Comment thread ...ore/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/OnnxTextEmbedderProviderTest.java Outdated
Comment thread ...lp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLInferenceOptionsTest.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml
Comment thread opennlp-docs/src/docbkx/machine-learning.xml
@krickert
krickert force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from bfaf52b to 880b364 Compare October 11, 2026 22:22
@krickert krickert mentioned this pull request Oct 11, 2026
Open
@krickert
krickert force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from 880b364 to 528974d Compare October 12, 2026 01:16
rzo1
rzo1 requested changes Oct 13, 2026
View reviewed changes

@rzo1 rzo1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @krickert - thanks, the first round is addressed. The remaining points are inline: a rebase onto 1.31.0, the test-only overload in AbstractDL, the unchecked device id, and some nits.

WDYT?

Comment thread opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
Comment thread opennlp-docs/src/docbkx/machine-learning.xml Outdated
Comment thread opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java Outdated
Comment thread ...re/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
@krickert @rzo1 @mawiesne
GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix the firs…
aeac043 
…t CUDA request in a fresh JVM

SentenceVectorsDL always created default session options, so it ran on
the CPU even with opennlp-dl-gpu. A new constructor takes
InferenceOptions, like NameFinderDL and DocumentCategorizerDL, and uses
the GPU settings and the lower case setting from them; a lower case
setting in the options takes precedence over the lowerCase argument. The
existing constructors keep their behavior.

AbstractDL.sessionOptions added the CUDA provider before any
OrtEnvironment existed. Loading the provider needs ONNX Runtime's default
logger, which only exists once an environment does, so the first GPU
request in a fresh JVM failed with "Attempt to use DefaultLogger but none
has been registered". The environment is now created first. This affects
NameFinderDL and DocumentCategorizerDL as well. When the provider cannot
be added, the session options created for the request are closed before
the exception is thrown.

A test makes the first CUDA request of a child JVM and requires that the
provider is added or rejected for a reason other than the missing
default logger; it runs in opennlp-dl-gpu and fails with the AbstractDL
of main. Other tests request a device id no machine has and check that ONNX
Runtime rejects it for SentenceVectorsDL, NameFinderDL and
DocumentCategorizerDL, and compare vectors on CUDA device 0 with the CPU
session (skipped without a CUDA provider or device). The tiny model test
fixtures are shared in TinyVectorModels.

The opennlp-dl-gpu test classpath no longer has the CPU onnxruntime next
to onnxruntime_gpu. Since onnxruntime_gpu has native libraries for
linux-x64 and win-x64 only, the opennlp-dl-gpu tests are skipped on
macOS and aarch64. The machine learning chapter documents the GPU
setup and the supported platforms.

Fixes #1389
Fixes #1390

Co-authored-by: Richard Zowalla <rzo1@apache.org>
Co-authored-by: Martin Wiesner <mawiesne@apache.org>
Signed-off-by: Kristian Rickert <kristian@apache.org>
Co-authored-by: Claude Code 2.1.296 (Opus 5.5)
@krickert
krickert force-pushed the OPENNLP-1982-sentence-vectors-inference-options branch from 528974d to aeac043 Compare October 13, 2026 19:22
@krickert

krickert commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Pushed the second round. opennlp-dl-gpu passes 1205 tests with none skipped on an RTX 4080 SUPER with CUDA 13 and cuDNN 9, including the cold start and CPU/GPU parity tests.

@krickert
krickert requested a review from rzo1 October 14, 2026 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@krickert krickert krickert left review comments
@rzo1 rzo1 Awaiting requested review from rzo1

Requested changes must be addressed to merge this pull request.

Assignees

@krickert krickert

Labels

bug Something does not work as documented (Jira: Bug) component: ml Machine learning (maxent, perceptron, naive bayes, libsvm)

Projects

None yet

Milestone

3.0.0

Development

Successfully merging this pull request may close these issues.

OPENNLP-1982: SentenceVectorsDL cannot run on a GPU GPU inference fails on the first CUDA request in a fresh JVM

3 participants

@mawiesne @krickert @rzo1