Repository navigation
endpoint.extra_body: top-level request fields, so Ollama can turn thinking off - #3
Merged
Merged
Conversation
…nking off The generic dialect sends no template arguments, and Ollama's /v1 route turns thinking off only through a top-level field (reasoning_effort: "none"; think: false is ignored there). A thinking model on Ollama therefore reads its first thinking token as the answer. extra_body is merged into every request at the client, the one path all requests take, and refuses a key the readout already sets. It is omitted from the recipe dump when empty, so existing recipes and their scorecards keep their hashes (pinned by a test).
Owner
|
Thank you for your contribution! Just merged it. |
Contributor
Author
|
No worries, thanks for the project .. works great over Qwen3.8 on my local machine to allow my personal assistant to triage messages I send while it's mid-task. It reads each one as redirect / refine / queue / none off a single token's logprobs, in about half a second, and nothing leaves the Server. I used probe, eval and calibrate to choose the model and test it. Local Qwen3.8 27B got 22/23 on held-out cases, and its confidence was good enough without calibration. The extra_body PR came out of this setup: Ollama's thinking models need reasoning_effort: "none" on /v1, or the one-token readout picks up the first token of the reasoning instead of the answer. Really nice tool to build on. |
haohao3k
pushed a commit
to haohao3k/jevify
that referenced
this pull request
Oct 6, 2026
Conflicts with endpoint.extra_body (fidecastro#3, fidecastro#4): - SKILL.md: keep both recipe notes. - tests/test_recipes.py: keep the extra_body tests and the frozen-hash canary. - tests/fakes/fake_openai_server.py: keep main's template_requests log and drop the branch's tokenize_requests; test_probe reads /tokenize bodies from template_requests. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The
genericdialect sends no template arguments. That is correct for a plainOpenAI surface, but it leaves no way to turn thinking off on a server that
takes the switch as a top-level request field. Ollama is the common case: its
/v1/chat/completionsroute turns thinking off only withreasoning_effort: "none".think: falseis ignored on that route, andchat_template_kwargsis never sent undergeneric.So a thinking model behind Ollama answers every question with its first
thinking token. Measured against Ollama 0.30.10 serving
qwen3.6:27b, oneChoice question,
max_tokens: 1,top_logprobs: 5:Here-0.76,Thinking-1.06,The-1.74think: falseHere-0.76,Thinking-1.06,The-1.74reasoning_effort: "none"C-0.64,A-0.82,D-3.79The change
endpoint.extra_body: dictin the recipe schema, defaulting to empty.OpenAICompatibleClientmerges it into every request body. The client is theone path every request takes (readout, warm, probe, rerank, embeddings), so
every caller gets the field and the guard with no extra wiring.
max_tokens,logprobs,logit_bias, ...) is refused with aBackendErrornaming thekey, before anything is sent. It is never silently overridden.
(
exclude_if), so every existing recipe keeps its hash and every committedscorecard still resolves. A test pins two pre-change hashes (the
fake-vllmfixture and the shippedgemma-4-e4b-it.llamacpprecipe) asliterals recorded on the parent commit.
SKILL.md(the endpoint field list, plus the Ollama recipe) and theADR-0004 D2 field table.
Tests
The fake server gains
thinking_off_field, a top-level request field thatturns thinking off the way Ollama's
/v1route does. DefaultNone, so noexisting behaviour changes.
test_generic_dialect_cannot_turn_thinking_off_without_extra_body: reproducesthe failure (
MissingLabelsError, because the answer labels are crowded outby the thinking opener).
test_extra_body_reaches_every_request_on_the_generic_dialect: with the field,the readout gets the answer, every request carries it, and
chat_template_kwargsis still absent.test_extra_body_cannot_override_a_readout_field:max_tokensis refused andnothing reaches the server.
test_extra_body_is_part_of_the_recipe_and_its_hashtest_a_recipe_without_extra_body_keeps_the_hash_it_had_before_the_field_existedBefore the implementation, 3 of these fail. The other 2 describe existing
behaviour and pass either way.
Evidence
uv run pytest -m "not live and not encoder": 136 passed (131 before).ruff checkandruff format --checkare clean.mypy jevify: 24 errors, the same 24 as onmain. This change adds none.jevify evalfrom this branch against Ollama 0.30.10 /qwen3.6:27b, usingendpoint.extra_body: {reasoning_effort: none}on a23-case choice suite: accuracy 1.0, median ~600 ms per question over a LAN.
That suite is private, so no scorecard is committed here.
Not exercised
probepath withextra_bodyset was run live only indirectly (theprobe builds its client through the same
build_backend). Its thinking checkstill reports "thinking state unknown" on
generic, because it has noserver-side template to inspect. Detecting a thinking opener in the readout
on
genericwould be a separate change.rerankandembeddingendpoints receive the field through the same client.No live test was run against either.
back one. Happy to add one if you want it.