Skip to content

fix(decopilot): show MCP image results to the model instead of base64 text - #7870

Open
yuriassuncx wants to merge 1 commit into
mainfrom
fix/decopilot-mcp-image-tool-results
Open

yuriassuncx wants to merge 1 commit into
mainfrom
fix/decopilot-mcp-image-tool-results

Conversation

@yuriassuncx

@yuriassuncx yuriassuncx commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

What is this contribution about?

When an MCP tool returns image content, decopilot sends it to the model as text. toModelOutput in mcp-tools.ts runs JSON.stringify on every non-text block, so the model gets a JSON string holding the base64 bytes.

Measured with the new Figma MCP (decocms/mcps#558). One get_design_context call on a 450×768 frame returned 100,341 bytes, about 25k tokens, almost all of it the PNG as base64. The model cannot read that. On larger frames the total goes over MAX_RESULT_TOKENS (32k), and truncateForModel then replaces the whole result, including the useful text outline, with a read_tool_output preview.

Studio already has a way to show tool images to the model. take_screenshot and the VM Read tool put images on pendingImages, and prepareStep attaches them as a user message. The comment in built-in-tools/index.ts explains why: images inside tool-result messages are not supported by every provider, but images in user content are. This PR routes MCP images through the same queue:

  • attachToolResultImages swaps each image block for a short placeholder, [image/png image, 98 KB, attached below], and queues the bytes on pendingImages.
  • If there is no queue, or the image is over 5 MB (Anthropic's limit), the image is only described and nothing is attached.
  • The truncation check now runs on the result without the base64, so the text is no longer dropped because of the image.
  • Each toolCallId queues its images only once, even if toModelOutput runs again.
  • Only the model-bound copy changes. The tool result streamed to the UI is untouched.
  • tools.ts passes extras.pendingImages, the queue the built-in tools already share, into toolsFromMCP.

Behavior change on non-image results: none. They take the same code path as before.

Why not put the image inside the tool result?

AI SDK v7 does support { type: "content", value: [..., { type: "file", mediaType, data }] }. I checked how each provider converts it:

Provider Image in a tool result
Anthropic native image block
OpenAI Responses input_image
Google function-response parts
OpenRouter image part
OpenAI Chat Completions JSON.stringify: back to base64 text

The pendingImages path works the same way for all of them, and it is the convention Studio already uses.

Related PRs

How did you verify your code works?

  • New mcp-tools.test.ts (3 tests):
    • an image is queued with a data URL and label, and the result keeps a placeholder;
    • with no queue, or above the size cap, the image is only described and nothing is queued;
    • end to end through toolsFromMCP: a result of about 400 KB of base64 plus text gives the model "[image/png image, 293 KB, attached below]\nFRAME Card 320x200" instead of a truncation preview. The image is queued once across two toModelOutput calls, and the original result keeps its image block.
  • bun test src/harnesses/lib/decopilot: 188 pass, 0 fail.
  • tsc --noEmit in apps/api: clean.
  • biome format and oxlint on the changed files: clean. The kebab-case filename warning already existed.

How to Test

  1. Connect an MCP that returns images, for example the Figma MCP from feat(figma): add Figma MCP for design-to-code context mcps#558, or any MCP with a screenshot tool.
  2. In a decopilot chat, ask it to call the tool. With Figma: get_design_context on a frame link.
  3. Expected: the model describes what is in the image, the text part of the result is intact, and no base64 shows up in the model input. The tool card in the UI looks the same as before.

Review Checklist

  • PR title is clear and descriptive
  • Changes are tested and working
  • Documentation is updated (if needed)
  • No breaking changes

🤖 Generated with Claude Code


Summary by cubic

Shows MCP tool image results to the model as attached images instead of unreadable base64 text.

  • MCP image content previously reached the model as a stringified JSON blob of base64 bytes, costing roughly 1 token per 4 bytes and often pushing results past the 32k-token cap so even the text was replaced by a read_tool_output preview.
  • Routes MCP images through the existing pendingImages queue used by take_screenshot and the VM Read tool, leaving a short text placeholder in the model-bound tool result.
  • Images over 5 MB, or when no queue is available, are only described and not attached.
  • Only the model-bound copy changes; the tool result streamed to the UI keeps its original image block.
  • Non-image results take the same code path as before.

Written for commit f87ff0a. Summary will update on new commits.

View guided diff Turn on auto-fix

… text

toModelOutput stringified every non-text MCP content block, so an image
returned by an MCP tool reached the model as a JSON string holding its
base64 bytes: unreadable, about 1 token per 4 bytes, and large enough to
push the whole result past the 32k-token cap so even the text was
replaced by a read_tool_output preview.

Queue those images on pendingImages, the path take_screenshot and the
VM Read tool already use (prepareStep attaches them as a user message,
which every provider accepts), and leave a short text placeholder in
the tool result. Without a queue, or above 5 MB, the image is only
described. The tool result the UI renders is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the claude PR authored by a coding agent label Oct 10, 2026
@yuriassuncx

Copy link
Copy Markdown
Contributor Author

@pedrofrxncx could you take a look at this one? I'd like your read on whether the approach makes sense before it goes in.

Context: I'm adding a Figma MCP in decocms/mcps#558, so task-board agents get a real reference to the design (screenshot, layout as CSS, comments) instead of guessing. When I tested it through a Studio connection, the decopilot chat sent the screenshot to the model as base64 text. That was about 25k tokens for a 450×768 frame, and the model couldn't see the image. On larger frames the result went over the 32k cap, and the whole result, text outline included, was replaced by a read_tool_output preview.

What I'd like you to check:

  1. Reuse of pendingImages. MCP images go through pendingImages, the same path take_screenshot and the VM Read tool use, instead of returning { type: "content" } with a file part. I checked the providers: OpenAI Chat Completions would stringify that file part back to text, and the user-message path works for all of them. Does reusing this queue for MCP results fit how you see it?
  2. History. On later turns, history is loaded from the DB, where fix(storage): stop persisting base64 image bytes on thread parts #7021 already stores { type: "image", data } as [image omitted]. So the image is attached once, in the step right after the tool call, and isn't re-injected. Is there any path where toModelOutput runs on an unredacted stored output that I missed?
  3. Subagents. Not wired here: assemble-agent-tools doesn't pass a queue, so subagent MCP images are only described, never dumped as base64. Is that worth a follow-up?

Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude PR authored by a coding agent

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant