Repository navigation
fix(aikit): keep images from error tool results - #103
Merged
Merged
Conversation
Tool calls now carry the input modalities of the model they answer (`ToolCallContext.modelInput`). On a model without image input, `read` adds a note that the image will be omitted, next to the placeholder the transform puts in its place. A session test reads an image on a text-only model, switches to a vision model and snapshots both requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Error tool results dropped their images for every provider. Anthropic-wire models now get a text-only error result with a placeholder, followed by a caption and the images after the turn's tool results. Other providers keep the images in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live test now records the request body each provider SDK sends and snapshots the part after the tool call, so it catches a wire that drops or misplaces the image. A unit case covers two failed calls in one turn. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cannot carry them Chat Completions adapters serialize a tool result's images as JSON text, and the Google adapter's legacy shape (before Gemini 3) drops the function response of an image-only result. On those wires a tool result with images now goes out as text, with "(see attached image)" when it has none, and its images follow the turn's tool results in a user message, as Anthropic error results already did. The wire follows the method the request resolves to, so Copilot and OpenAI method overrides are honoured. The live image tests now cover OpenAI over Chat Completions and Gemini 2.5, and the text-and-image prompt asks for what the model learned and the shape's color. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # packages/harness/test/__artifacts__/tools.read.vision.json # packages/harness/test/tools.read.vision.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #102.
Error tool results dropped their image blocks for every provider.
toolResultOutputreturnederror-textfor anyisErrorresult, so the image never reached the model.How each wire gets the images
anthropic,googleVertexAnthropic, Copilotmessages), error resultserror-textwith(image attached after the tool results), then ausermessage withImages from the error result of tool call <id> (<name>):and the images.@ai-sdk/anthropicmerges both into one user message, tool_result blocks first.openai-compatible, Copilot chat, OpenAI withmethod: chat), every result with images(see attached image)when it has no text), then oneusermessage:Attached image(s) from tool result:and the turn's images. These adapters would otherwise serialize the images as JSON text.usermessage withTool result image:and the images for each result. The legacy shape would otherwise drop the function response of an image-only result.content: text, then the images in place. The AI SDKcontentoutput has no error flag, so an error result with images loses it here (Google'serrorkey).The wire follows the method the request resolves to, so Copilot and OpenAI method overrides are honoured. Text-only models still get
(tool image omitted: model does not support images)and nothing moves.Tests
test/transform.test.ts: Anthropic wire tests against the fake Anthropic server. Images move after the tool results, and error results without images are unchanged. Also checks that images stay in place for OpenAI and that text-only models get no split.test/transform.test.ts: Chat Completions, Gemini 2.x/Vertex and Copilot method overrides, multiple tool rounds followed by a user message, and two failed calls in one turn on Anthropic.test/e2e/image.tool.error.e2e.test.ts: a live tool call, then a failed result carryingred-circle.png; the model must describe the image. It records the request body each provider SDK sends and snapshots the part after the tool call to__artifacts__/error-tool-image.<wire>.json. Passes on Anthropic, OpenAI (Responses and Chat Completions), Codex, OpenRouter, Gemini 2.5 and Gemini 3.5.test/e2e/image.tool.result.e2e.test.ts: adds OpenAI Chat Completions, Gemini 2.5 and Gemini 3.5 cases for successful image results.vp run typecheckand the aikit unit tests pass.vp checkstill reports formatting issues only in the committed skill files under.agents/and.claude/.🤖 Generated with Claude Code