Repository navigation
README: serving on the network, and any vLLM or llama.cpp model - #2
Merged
Merged
Conversation
A reader of the NVIDIA forum thread had to discover --host 0.0.0.0 and detaching on their own, and read the single vLLM recipe as "only supports DeepSeek". The quickstart now says both: any model behind vLLM or llama.cpp works by copying the nearest recipe and probing, and serve binds to localhost unless told otherwise, with the api key to set when it is not. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two things user Zambonilli of the NVIDIA forum thread had to work out on their own:
servebinds to127.0.0.1; reaching it from another machine needs--host 0.0.0.0. The quickstart now shows that, withJEVIFY_API_KEYset first so the network-facing endpoint requires a bearer key, and a detached launch that keeps its log.endpoint.base_urlandmodel.name, and running the probe, which re-verifies the answer token ids and writes its findings into the recipe.Docs only; no runtime change. Re-read against
jevify/cli.py(--host,--port,--api-key-envdefaulting toJEVIFY_API_KEY) andjevify/api/app.py(/health).🤖 Generated with Claude Code