Skip to content

[Engine] The Mac (Metal) engine ignores a request's sampling (temperature, top_p, top_k, min_p, seed, penalties) and drops logit_bias #3006

Open
Open
[Engine] The Mac (Metal) engine ignores a request's sampling (temperature, top_p, top_k, min_p, seed, penalties) and drops logit_bias#3006

Description

@oceanio
oceanio
opened on Oct 18, 2026

Area

Engine

Desktop app, Terminal or source?

Source (./setup.sh on a Mac builds the Metal engine)

What happened

On the Mac engine every request decodes greedily, whatever it asks for. temperature, top_p, top_k, min_p,
seed, repetition_penalty, frequency_penalty and presence_penalty reach the engine (they are on its GEN line)
but change nothing. Different seeds at temperature 1.0 give the same text, and it is the temperature-0 text.
logit_bias is dropped too: a token banned with -100 is still picked.

Three consequences:

  1. Sampling settings do nothing on a Mac.
  2. The model card's defaults from 0.1.45 (Strata serves Qwen3.8-Flash-Next with strange behavior - really bad results #2685) do not take effect on a Mac. The server now sends
    temperature 0.7, top_p 0.8, top_k 20 and presence_penalty 1.5 (instruct) when the request and the config give
    none. The Mac engine answers those requests greedily, so Mac users still get the greedy decoding that Strata serves Qwen3.8-Flash-Next with strange behavior - really bad results #2685 was
    about.
  3. logit_bias is dropped on Metal. Verifier::set_logit_bias (src/core/verify.cpp) returns before it stores
    the bias when le_ < g_->n_layers. The Metal init returns before le_ is set to the layer count, so le_ stays
    -1 and every bias is treated as belonging to a later stage. The request gets no error.

Expected: the same behaviour as the PC engine. The settings act in the CUDA sampler's order, a fixed seed gives the
same text again, another seed gives another text, temperature 0 (or none, without the card's defaults) is greedy, and
a banned token is never picked.

The cause is that the Mac decode window always takes its own argmax (argmax_part / argmax_final in the window's
head), and Verifier::run's Metal branch returns that token. The host-side sampling that the CUDA path runs after
its window is not ported. Nothing in the log or the response says that the settings were ignored.

Reproduction

Re-checked on 0.2.0-beta (main 3fa1b52, built from source); first found on main 477ac13 plus #2895 (the 0.1.45 Mac build fix).
I used the official two-file Qwen3.8-Flash-Next IQ3_S (SC117's abliterated build), resident, as written by
./setup.sh. Start the server on a free port with the model's config, then send the same chat request with
different sampling:

python3 serve/server.py --engine strata --config strata-<model>.json --port 8197 &

ask() {  # the same prompt, thinking off, 40 tokens; $1 = extra JSON fields
  curl -s http://127.0.0.1:8197/v1/chat/completions -H 'Content-Type: application/json' -d '{
    "model": "strata", "max_tokens": 40, "chat_template_kwargs": {"enable_thinking": false},
    "messages": [{"role": "user", "content": "Write one sentence about the sea."}] '"$1"' }' |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
}
ask ', "temperature": 0, "presence_penalty": 0'      # greedy
ask ', "temperature": 0'                              # the card adds presence_penalty 1.5 (#2940)
ask ', "temperature": 1.0, "seed": 1'
ask ', "temperature": 1.0, "seed": 2'
ask ', "temperature": 1.0, "seed": 3'
ask ', "temperature": 1.5, "top_k": 50, "top_p": 0.95, "min_p": 0.01, "seed": 7, "presence_penalty": 0'
ask ''                                                # no sampling fields: the model card's defaults
ask ', "temperature": 0, "presence_penalty": 0, "repetition_penalty": 1.5, "frequency_penalty": 1.0'
ask ', "temperature": 0, "presence_penalty": 0, "logit_bias": {"760": -100}'   # 760 = "The", the greedy first token

(760 is the first token of the greedy answer with this model's tokenizer; use the first token of your greedy answer.)

Every line prints the same sentence.

Measured

M5 Max (128 GB), macOS 26.7.1, Qwen3.8-Flash-Next GSQ-RCO IQ3_S, resident, --spec 2 with MTP,
--max-context 262144, 2026-10-18. These are the answers to the requests above, all in one server run:

Request (besides the prompt) Answer
temperature 0, presence_penalty 0 (greedy) The vast, endless sea stretches to the horizon, its rhythmic waves whispering ancient secrets to the shore.
temperature 0 (the card adds presence_penalty 1.5, #2940) the same
temperature 1.0, seed 1 the same
temperature 1.0, seed 2 the same
temperature 1.0, seed 3 the same
temperature 1.5, top_k 50, top_p 0.95, min_p 0.01, seed 7 the same
no sampling fields (the model card's defaults: temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5) the same
temperature 0, repetition_penalty 1.5, frequency_penalty 1.0 the same
temperature 0, logit_bias {"760": -100} (760 = "The", the greedy answer's first token) the same, starting with "The"

The engine reported 0.1.45-metal in /v1/status. An earlier run that day, with nearly the same requests, gave
the same: every answer equal to the greedy one.

With a port of the CUDA sampler's chain to the Mac engine (on the host, over the window's logits in unified
memory; I will open it as a PR once #2895 is merged), the same requests give a different answer for every row but
the first. Seed 1 gives the same text each
time it is sent, seeds 2 and 3 give other texts, and the banned token is not picked ("Beneath the shimmering
surface, ...").

At the engine protocol, a test I wrote for that port (tools/metal_sampling_test.py, run against my Metal branch
without the port) fails 12 of 18 checks. Seeds 1 and 2 give the greedy text. 100 one-token draws at temperature 1.0, top_k 40,
top_p 0.95 all give the greedy token, which has probability 0.81 on those logits; 15 of the 100 picks differ from what
a reference sampler written from the CUDA kernels draws with the same seeds. Greedy with penalty_repeat 1.5 gives the same tokens as
without it. A token banned by logit_bias is still picked first.

Not tested

  • The PC engines (CUDA, HIP, SYCL): not built here. The reference behaviour is read from their code
    (src/kernels/cuda/sampler.cu, Verifier::run) and from the test's reference sampler written from it, not run.
  • Other Macs and chips, models that stream their experts from the SSD, other packs (Q2_0, IQ3_XXS, Unsloth's),
    vision requests. The cause is in code that all of them share, so I expect the same result.
  • Thinking on. The requests above use thinking off; the card's thinking defaults (temperature 1.0, top_p 0.95,
    top_k 20) should be ignored the same way.

Strata version

0.2.0-beta (main 3fa1b52, built from source). First seen on 0.1.45 (main 477ac13 + #2895).

Graphics card, memory and system

Apple M5 Max, 128 GB unified memory (Metal) · macOS 26.7.1

Reproduction status

Reproduced locally on 0.2.0-beta (main 3fa1b52): the script below prints the same sentence for 9 of 9 requests. A fix is in #3301.

Engine log

Not attached. Nothing in it mentions sampling: the engine prints no line about the settings it ignored.

Activity

  1. Niko1221 commented on Oct 19, 2026

    @Niko1221
    Niko1221
    on Oct 19, 2026
    Owner

    Thanks for taking the time to report this.

    We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to read properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected.

    To reopen it takes about two minutes:

    1. Re-test on 0.2.0-beta. It may already be fixed. Write down what you see.
    2. Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other].
    3. Fill in the parts of the issue form: what happened, the Strata version, your graphics card, memory and system.
    4. Comment /reopen. A check runs and either reopens the issue or lists what is still missing.

    The rules are in CONTRIBUTING.md.

  2. Niko1221
    closed this as not plannedon Oct 19, 2026

  3. oinani0721 commented on Oct 19, 2026

    @oinani0721
    oinani0721
    on Oct 19, 2026

    Two of the "Not tested" rows, on another M5 Max (128 GB, macOS 27.0.1): Unsloth's pack with its experts streamed from the SSD, and thinking on. Same result: every answer is the greedy one. The build is older than yours, main 296b59d (0.1.44.1) from source, with #2503, #2631, #2819 and two local cache/memory patches in the engine; the 0.2.0-beta part at the end is reading only.

    Sampling sent, thinking on. Unsloth's Qwen3.8-Flash-Next UD-Q6_K_XL, STRATA_METAL_STREAM=1, --mmap-experts, --spec 3 with MTP. One prompt ("用一句话描述一下秋天。", one sentence about autumn), three times per endpoint, non-streaming, temperature 2.0, top_p 1.0, top_k 64:

    • /v1/responses with reasoning.effort low, max_output_tokens 300: the same reasoning and answer 3 of 3 times, byte for byte.
    • /v1/chat/completions with reasoning_effort low, max_tokens 300: the same 3 of 3 times, and the same text as on /v1/responses.

    Agent requests. #2731's end reproduces on this server with a Codex-shaped /v1/responses request (one function tool, effort xhigh; request and numbers in #2731 (comment)). The reply is the same reasoning-only stub, 80 output tokens with the same ending, in 4 of 4 runs without sampling fields and in 10 of 10 with the card's thinking sampling sent (temperature 1.0, top_p 0.95, top_k 20). On a Mac, resending a request that went wrong gets the same reply back.

    v0.2.0-beta (3d1fb48), reading only: Verifier::run still returns the Metal window's own pick (src/core/verify.cpp:2691-2695) before its host-side sampling (2974), the window still picks with argmax_part / argmax_final (metal/src/engine/window.cpp:1097-1103), and strata::kernels::sample_tokens is still a stub (metal/src/engine/stubs.cpp:317). So I expect the same on the beta, but have not run it.

    @oceanio, in case you want it reopened: the bot takes /reopen only from you, and wants [Engine] at the start of the title and a 0.2.0-beta retest under "Strata version".

  4. oceanio
    changed the title [-][Bug]: The Mac (Metal) engine ignores a request's sampling: temperature, top_p, top_k, min_p, seed and the penalties do nothing, logit_bias is dropped, and every answer is the greedy one[/-] [+][Engine] The Mac (Metal) engine ignores a request's sampling (temperature, top_p, top_k, min_p, seed, penalties) and drops logit_bias[/+] on Oct 20, 2026

  5. oceanio commented on Oct 20, 2026

    @oceanio
    oceanio
    on Oct 20, 2026
    Author

    Re-checked on 0.2.0-beta (main 3fa1b52, M5 Max 128 GB, macOS 26.7.1, Qwen3.8-Flash-Next IQ3_S resident): unchanged.

    • The curl script above, through serve/server.py: 9 of 9 requests print the same sentence ("The vast, endless sea
      stretches to the horizon, ..."), including seeds 1-3 at temperature 1.0, top_k/top_p/min_p, the model card's
      defaults, penalties and logit_bias {"760": -100}.
    • An engine-level check (tools/metal_sampling_test.py in the fix): 11 of 13 checks fail on 0.2.0-beta.

    Reproduction status: reproduced locally. A fix is in #3301 (draft until a CUDA/HIP/SYCL build of its one shared line).

    /reopen

  6. oceanio commented on Oct 20, 2026

    @oceanio
    oceanio
    on Oct 20, 2026
    Author

    /reopen

  7. github-actions
    reopened this on Oct 20, 2026

  8. github-actions commented on Oct 20, 2026

    @github-actions
    github-actionsBot
    on Oct 20, 2026 – with GitHub Actions

    Reopened, thank you. A maintainer will pick it up from here.

  9. joseedson18jc commented on Oct 21, 2026

    @joseedson18jc
    joseedson18jc
    on Oct 21, 2026

    Same on an M1 Max (64 GB, macOS 27.0), on 0.2.1.1 (9ae93f7b, engine 0.1.46-metal) and on 0.2.0-beta (3fa1b524, 0.1.45.3-metal). Qwen3.8-Flash-Next Q2_0 (SC117's abliterated build) resident, --mmap-experts --expert-cache 64 --vision, through serve/server.py.

    It does not depend on speculative decoding. With --spec, --mtp and --lookup-chain removed and --suffix-draft 0, 12 of 12 requests gave the same answer:

    • 4x temperature 1.0, top_p 0.95, top_k 20
    • 4x temperature 1.5, top_k 64
    • 4x no sampling fields (the model card's defaults)

    With the Mac setup's flags (--spec 2 + MTP + --lookup-chain 8) it is the same: 9 of 9, including presence_penalty 1.5.

    ask() {  # thinking off, 20 tokens; $1 = extra JSON fields
  curl -s http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
    '"$1"' "messages": [{"role": "user", "content": "Invente um nome criativo para um gato. Responda só o nome."}],
    "max_tokens": 20, "reasoning_effort": "none"}' |
  python3 -c 'import json,sys; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'
}
for i in 1 2 3 4; do ask '"temperature": 1.5, "top_k": 64,'; done   # 4x the same name

    Two more consequences, besides the three in the report:

    1. The reasoning-loop recovery cannot help on a Mac. serve/server.py retries a looping reply with temperature raised to at least 1.0 and presence_penalty to at least 1.5, but the engine decodes greedily either way. On a 40-question hard set (medium effort, 3000-token budget), one answer recomputed 2^100 by hand until max_tokens (12,000), the same with presence_penalty 0 and 1.5. 38 of 40 answers in both runs had exactly the token count of a temperature-0 run.
    2. /props reports sampling that is not applied. It shows temperature 1.0, top_p 0.95, top_k 20 while every reply is the greedy one, so a client cannot tell.

    No has no Metal implementation yet line appears in the engine log: the window's argmax (metal/kernels/sample.metal) is used and sample_tokens (stub in metal/src/engine/stubs.cpp) is never reached.

    I can test #3301 on the M1 Max if that helps.

  10. oceanio commented on Oct 21, 2026

    @oceanio
    oceanio
    on Oct 21, 2026
    Author

    Thanks, this is very useful: a second chip (M1 Max) and a second model (Qwen Q2_0) with the same result, and the two extra consequences are right. The loop retry and /props both assume the engine applies sampling, and on Metal it doesn't.

    Yes, please test #3301 on the M1 Max. It merges cleanly onto current main (0.2.1.4, 9de04cd4):

    git fetch https://github.com/Niko1221/Strata.git pull/3301/head:pr-3301
git checkout pr-3301
python3 metal/setup_mac.py        # rebuilds the Metal engine; the model and pack are kept

    With #3301 the CPU picks each token from the window's logits (unified memory), with the same chain and seeds as the CUDA engine. Your ask loop should then give different names at temperature 1.5, top_k 64, and the same name every time at temperature 0, or with the same seed. A logit_bias of -100 on the name's first token should also stop it from appearing. With no sampling fields the server's model-card defaults now apply, so /props matches what you get, and the loop retry's higher temperature and presence penalty actually take effect.

    It would help to know:

    • whether 4x temperature 1.5, top_k 64 now gives different answers, and 4x seed: 1 gives the same one;
    • tokens/s with and without sampling fields (on an M5 Max sampled decode was a few to ~15% slower than greedy over 3 runs, mostly from fewer accepted drafts; the sampler itself is ~10-45 us per token);
    • whether your 40-question set still has the 2^100 loop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions