Skip to content

models: the 12288 output cap is applied to every provider, not to the DashScope endpoint whose 400 it encodes #3198

Closed

Description

@HugoXK

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Claimed-by: FlyBot Lead
Claim-Id: ddb5ff3edda7443b9be2b7b952699d10
Branch: dev/5-outbound-cap-per-provider
Base: 0f7e821
At: 2026-11-19T01:20:38Z

Implementation: FlyBot (ours) — PR follows within 24h. Do not pick up without a pickup-ok comment on this issue.

Written-by: FlyBot Lead
Bot: FlyBot Dev-5 (implementation) — found by FlyBot Product, audit of the outbound output cap
Tier: T2
Branch: dev/5-outbound-cap-per-provider

What is wrong

core/src/host/models.js:15 holds one ceiling, OUTBOUND_MAX_TOKENS = 12_288, and the llm/stream listener (:52-62) applies it to every provider: a call that asks for more is restarted with 12288, and a model row whose defaultMaxTokens declares more is sent 12288 too. The comment above the constant says exactly whose limit this is — "above this DashScope refuses the whole request with a 400 (max_completion_tokens … expected a value <= 12288) … (#3132)" — but the code has no provider or endpoint test, so a provider that declares 32000 gets 12288 as well.

Consequence: on any provider whose output limit is higher, the answer is silently cut to 12288 tokens when the caller asks for more (a long document, a big code review), and no error is raised — the cap is applied to the request, not to a provider that refused it.

Reproduction (red-first on live main 0f7e82121832d4cea8e26f1e9681355100432df7)

Using the fakeContext/install harness from core/test/llm-output-cap.test.js (a probe, not in the tree): provider openai, model gpt-x, resolveModelInfo → { defaultMaxTokens: 32000 }.

  • caller asks maxTokens: 20000 → sent.maxTokens = 12288 (expected 20000);
  • caller asks nothing → sent.maxTokens = 12288 (expected: no injected cap, DSH fills the declared default);
  • the same calls against the DashScope route still must be 12288.

Command:
cd core && TZ=UTC TMPDIR=/tmp node --import ./test/harness-resolve.mjs --test test/llm-output-cap.test.js

Expected

  • the 12288 clamp applies only where the endpoint actually refuses more — the DashScope route whose 400 #3132 recorded; every other provider gets the caller's request, or the declared default, unchanged;
  • the DashScope behaviour stays pinned exactly as today (a model row declaring more is asked for 12288, a caller asking for more is sent 12288, the caller's object is never written to, and the second ctx.llm.stream pass still carries the cap);
  • no new cap is invented and no existing cap is raised: this is about which provider the clamp belongs to, not about its value;
  • the predicate is named in the code and carries its evidence (the DSH route/provider id it tests); a guess-based allowlist of providers is not acceptable — if the provider cannot be identified for a route, the safe answer is to keep today's behaviour for that route and say so in the PR.

Scope

  • core/src/host/models.js — the predicate around OUTBOUND_MAX_TOKENS
  • core/test/llm-output-cap.test.js — the two provider-split cases plus the red-first ones
  • patches/ds-bot/0001-core-fixes.patch, patches/ds-bot/0007-version-and-tooling.patch — node scripts/ds-bot-patches.mjs generate (never hand-edit), then check and canonical
  • changelog.d/<this issue>.fixed.md
  • docs/ page that describes the model cap, one sentence

Tier: core/src/host/** is T2 — a Verdict PASS at the exact tip is required. No UI surface (host request shaping only).

Not in scope: the model catalog's own display, the client's token report, retry/backoff, and any change to what a provider's 400 does today.

Duplicate check

Searched issues and PRs for OUTBOUND_MAX_TOKENS, 12288 + provider, maxTokens + dashscope: only closed #3132 (which introduced the clamp for DashScope) and its PR. Nothing open covers narrowing it by provider. No duplicate.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions