Skip to content

fix: fallback model never triggers on a malformed provider response - #117

Closed
fabceolin wants to merge 2 commits into
FSoft-AI4Code:mainfrom
fabceolin:fix/fallback-missing-unexpected-model-behavior
Closed

fabceolin wants to merge 2 commits into
FSoft-AI4Code:mainfrom
fabceolin:fix/fallback-missing-unexpected-model-behavior

Conversation

@fabceolin

Copy link
Copy Markdown
Contributor

Problem

FallbackModel's default fallback_on=(ModelAPIError,) does not cover UnexpectedModelBehavior, which pydantic-ai raises for a 200 response whose body doesn't match the expected schema. Seen against OpenRouter on a real run (voll-intelligence):

pydantic_ai.exceptions.UnexpectedModelBehavior: Invalid response from openai chat completions endpoint: 3 validation errors for ChatCompletion
choices
  Input should be a valid list [type=list_type, input_value=None, ...]
model
  Input should be a valid string [type=string_type, input_value=None, ...]
object
  Input should be 'chat.completion' [type=literal_error, input_value=None, ...]

Likely a transient provider hiccup, not something tied to the specific input.

Because UnexpectedModelBehavior is a sibling of ModelAPIError under AgentRunError (not a subclass), it silently bypasses the configured fallback-model entirely and kills the module outright — even though a fallback-model was explicitly configured for exactly this kind of situation.

Scope

Observed once in 376 generated pages on the run in question — low frequency, but the failure mode is a genuine gap: the fallback exists specifically to catch this kind of thing, and doesn't.

Fix

Pass fallback_on=(ModelAPIError, UnexpectedModelBehavior) explicitly in create_fallback_models.

🤖 Generated with Claude Code

FallbackModel's default fallback_on=(ModelAPIError,) does not cover
UnexpectedModelBehavior, which pydantic-ai raises for a 200 response whose
body doesn't match the expected schema (seen against OpenRouter: a
ChatCompletion with choices/model/object all None — likely a transient
provider hiccup, not something tied to the specific input).

Because UnexpectedModelBehavior is a sibling of ModelAPIError under
AgentRunError (not a subclass), it silently bypasses the configured
fallback-model entirely and kills the module outright, even though a
fallback-model was explicitly configured. Observed once in 376 generated
pages on a real run (voll-intelligence) — low frequency, but the failure
mode is a genuine gap: the fallback exists specifically to catch this kind
of thing, and doesn't.

Fix: pass fallback_on=(ModelAPIError, UnexpectedModelBehavior) explicitly.
…tic-ai 1.0.6)

requirements.txt pins pydantic-ai==1.0.6, and that version's
pydantic_ai.exceptions has no ModelAPIError (it was added in a later
release as a broader base class over ModelHTTPError) — only ModelHTTPError
and UnexpectedModelBehavior exist. Importing ModelAPIError broke collection
for every test module that imports llm_services, transitively or directly.

FallbackModel's own default in 1.0.6 is fallback_on=(ModelHTTPError,), so
using ModelHTTPError here instead of ModelAPIError keeps the exact same
fix intent (widen the default to also cover UnexpectedModelBehavior) while
working with the pinned dependency version.

CI: ImportError: cannot import name 'ModelAPIError' from
'pydantic_ai.exceptions' (9 collection errors).
@anhnh2002

Copy link
Copy Markdown
Collaborator

Thanks a lot for the clear reports and the real-run evidence, it made these easy to confirm. All three issues are real. I've fixed them together in #121, so I'm closing this PR in favour of that one.
A few differences from your version:

Thanks again, and please keep the reports coming.

@anhnh2002 anhnh2002 closed this Sep 29, 2026
pull Bot pushed a commit to soitun/CodeWiki that referenced this pull request Sep 29, 2026
…ons, ship updater

- Package: add codewiki.src.be.updater to [tool.setuptools] packages; a
  non-editable install had no updater, so every --update failed on import.
  New test checks every package directory is listed. (FSoft-AI4Code#119)
- Agent limits: pass UsageLimits(request_limit=...) to every agent run and
  retries=... to every Agent. pydantic-ai defaults are 50 requests per run and
  1 retry per failing tool call, which complex modules and
  generate_sub_module_documentation hit. New settings request_limit (default
  100) and agent_retries (default 3), in `codewiki config set`, `config show`
  and as per-run overrides on `codewiki generate`. (FSoft-AI4Code#115, FSoft-AI4Code#118)
- Fallback: FallbackModel now also falls back on UnexpectedModelBehavior (a 200
  response whose body does not parse), keeping ModelAPIError. (FSoft-AI4Code#117)
- Overview pages: MODULE_OVERVIEW_PROMPT / REPO_OVERVIEW_PROMPT had no slot for
  the user's instructions. complete() takes an optional system_prompt (OpenAI-
  compatible, litellm, Azure, and caw via CawAgent(system_prompt=...)), and
  parent/repo overviews send the instructions as a system message. (FSoft-AI4Code#116)
- Sub-module agents built their system prompt with a raw .format(), so with no
  instructions the prompt ended in the literal text "None"; they now use
  format_system_prompt / format_leaf_system_prompt like the top-level agents.

Reported in FSoft-AI4Code#115, FSoft-AI4Code#116, FSoft-AI4Code#117, FSoft-AI4Code#118, FSoft-AI4Code#119.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants