Skip to content

endpoint.extra_body: top-level request fields, so Ollama can turn thinking off - #3

Merged
fidecastro merged 1 commit into
fidecastro:mainfrom
Carltk:endpoint-extra-body
Sep 28, 2026
Merged

fidecastro merged 1 commit into
fidecastro:mainfrom
Carltk:endpoint-extra-body

Conversation

@Carltk

@Carltk Carltk commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

The problem

The generic dialect sends no template arguments. That is correct for a plain
OpenAI surface, but it leaves no way to turn thinking off on a server that
takes the switch as a top-level request field. Ollama is the common case: its
/v1/chat/completions route turns thinking off only with
reasoning_effort: "none". think: false is ignored on that route, and
chat_template_kwargs is never sent under generic.

So a thinking model behind Ollama answers every question with its first
thinking token. Measured against Ollama 0.30.10 serving qwen3.6:27b, one
Choice question, max_tokens: 1, top_logprobs: 5:

request latency top tokens
no extra field 22.7 s Here -0.76, Thinking -1.06, The -1.74
think: false 0.28 s Here -0.76, Thinking -1.06, The -1.74
reasoning_effort: "none" 0.34 s C -0.64, A -0.82, D -3.79

The change

  • endpoint.extra_body: dict in the recipe schema, defaulting to empty.
  • OpenAICompatibleClient merges it into every request body. The client is the
    one path every request takes (readout, warm, probe, rerank, embeddings), so
    every caller gets the field and the guard with no extra wiring.
  • Guard at the chokepoint: a key the readout already set (max_tokens,
    logprobs, logit_bias, ...) is refused with a BackendError naming the
    key, before anything is sent. It is never silently overridden.
  • Hash stability: the field is left out of the dump when empty
    (exclude_if), so every existing recipe keeps its hash and every committed
    scorecard still resolves. A test pins two pre-change hashes (the
    fake-vllm fixture and the shipped gemma-4-e4b-it.llamacpp recipe) as
    literals recorded on the parent commit.
  • Docs: SKILL.md (the endpoint field list, plus the Ollama recipe) and the
    ADR-0004 D2 field table.

Tests

The fake server gains thinking_off_field, a top-level request field that
turns thinking off the way Ollama's /v1 route does. Default None, so no
existing behaviour changes.

  • test_generic_dialect_cannot_turn_thinking_off_without_extra_body: reproduces
    the failure (MissingLabelsError, because the answer labels are crowded out
    by the thinking opener).
  • test_extra_body_reaches_every_request_on_the_generic_dialect: with the field,
    the readout gets the answer, every request carries it, and
    chat_template_kwargs is still absent.
  • test_extra_body_cannot_override_a_readout_field: max_tokens is refused and
    nothing reaches the server.
  • test_extra_body_is_part_of_the_recipe_and_its_hash
  • test_a_recipe_without_extra_body_keeps_the_hash_it_had_before_the_field_existed

Before the implementation, 3 of these fail. The other 2 describe existing
behaviour and pass either way.

Evidence

  • uv run pytest -m "not live and not encoder": 136 passed (131 before).
  • ruff check and ruff format --check are clean.
  • mypy jevify: 24 errors, the same 24 as on main. This change adds none.
  • Live: jevify eval from this branch against Ollama 0.30.10 /
    qwen3.6:27b, using endpoint.extra_body: {reasoning_effort: none} on a
    23-case choice suite: accuracy 1.0, median ~600 ms per question over a LAN.
    That suite is private, so no scorecard is committed here.

Not exercised

  • The probe path with extra_body set was run live only indirectly (the
    probe builds its client through the same build_backend). Its thinking check
    still reports "thinking state unknown" on generic, because it has no
    server-side template to inspect. Detecting a thinking opener in the readout
    on generic would be a separate change.
  • rerank and embedding endpoints receive the field through the same client.
    No live test was run against either.
  • No shipped Ollama recipe is added, since there's no committed scorecard to
    back one. Happy to add one if you want it.

…nking off

The generic dialect sends no template arguments, and Ollama's /v1 route turns
thinking off only through a top-level field (reasoning_effort: "none"; think:
false is ignored there). A thinking model on Ollama therefore reads its first
thinking token as the answer.

extra_body is merged into every request at the client, the one path all
requests take, and refuses a key the readout already sets. It is omitted from
the recipe dump when empty, so existing recipes and their scorecards keep
their hashes (pinned by a test).
@fidecastro
fidecastro merged commit 13a6e41 into fidecastro:main Sep 28, 2026
@fidecastro

Copy link
Copy Markdown
Owner

Thank you for your contribution! Just merged it.

@Carltk

Carltk commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

No worries, thanks for the project .. works great over Qwen3.8 on my local machine to allow my personal assistant to triage messages I send while it's mid-task. It reads each one as redirect / refine / queue / none off a single token's logprobs, in about half a second, and nothing leaves the Server.

I used probe, eval and calibrate to choose the model and test it. Local Qwen3.8 27B got 22/23 on held-out cases, and its confidence was good enough without calibration.

The extra_body PR came out of this setup: Ollama's thinking models need reasoning_effort: "none" on /v1, or the one-token readout picks up the first token of the reasoning instead of the answer.

Really nice tool to build on.

haohao3k pushed a commit to haohao3k/jevify that referenced this pull request Oct 6, 2026
Conflicts with endpoint.extra_body (fidecastro#3, fidecastro#4):
- SKILL.md: keep both recipe notes.
- tests/test_recipes.py: keep the extra_body tests and the frozen-hash canary.
- tests/fakes/fake_openai_server.py: keep main's template_requests log and
  drop the branch's tokenize_requests; test_probe reads /tokenize bodies from
  template_requests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants