Repository navigation
An error's remedy is in its text, where an agent reads it (CHALLENGES 321) - #212
Merged
Merged
Conversation
… 321) Claude Code hands an agent a failed call's text alone, with no code and no remedy, and the text was the message only: every remedy not also written into its message was seen by no agent, and the CLI's error line left it out too. An error's text is now the message, then "next: " and the remedy when the message does not already say it; the structured payload keeps them apart. Found by auditing the 193 errors agents met in the evaluation's stored runs, which also found two with no next step at all: app_keyboard's read-back failure now says focus moved and to read what landed before typing again, and "not installed" names app_list_apps. ROADMAP: a ledger of what a session changed, listed by app_session; and a remedy on every recoverable error as a build rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In Claude Code an agent is handed a failed call's text alone — no code, no remedy (the stored evaluation transcripts show only
Error: <message>) — and Mobium's text was the message only.ErrorResultput the remedy in the structured half and nowhere else, and the CLI'serror:line printed the message. So a remedy not also written into its message reached no agent and no person; none of the 109 that set one is repeated word for word in its message. CHALLENGES 318's own refusal namedapp_swipeonly in its remedy.Fixed: an error's text — MCP content and the CLI line — is the message, then
next:and the remedy when the message does not already say it (mobiumerr.Text). The structured payload keeps the two apart.Found by auditing every tool error agents met in the evaluation's stored runs (193 errors, 417 trials, 59 kinds;
mobium-evals/probes/error_remedies.py), which also found two errors with no next step at all, now given one:app_keyboard's read-back failure (focus moved; read what landed before typing again, which could type it twice) and "… is not installed" (namesapp_list_apps).ROADMAP gains two entries: a ledger of what a session changed, listed by
app_session, and a remedy on every recoverable error as a build rule.