Repository navigation
feat: REPEAT_ERROR detector — alert on consecutive failed bash commands - #1
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d81b4a0f5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| an int subclass in Python). Used both by detect_repeat_error() and by the | ||
| live watch loop, where the streak must persist across poll ticks. | ||
| """ | ||
| if ev.get("tool") == "bash": |
There was a problem hiding this comment.
For events produced by the advertised agent-specific parsers, tool names are preserved verbatim: the existing tests use Bash for Claude, shell for Codex, and run_shell_command for Gemini. This strict lowercase comparison resets the streak for all of them even when an event carries a nonzero exit code, so REPEAT_ERROR only works with the generic schema spelling its tool exactly bash; normalize or alias the supported shell tool names here.
Useful? React with 👍 / 👎.
| bash_streak = update_repeat_streak(bash_streak, ev) | ||
| if bash_streak >= repeat_n and not incident_reported: | ||
| alert("REPEAT_ERROR", f"bash failed {bash_streak} times consecutively") |
There was a problem hiding this comment.
In live watch mode, when consecutive failing commands have distinct arguments—as in the new fixture—the third failure raises REPEAT_ERROR, but each subsequent new fingerprint clears incident_reported in the generic reset above while bash_streak remains above the threshold. This condition therefore emits another alert for the fourth failure, the fifth, and so on, potentially spamming stdout and Telegram; REPEAT_ERROR needs its own latch that resets only when update_repeat_streak breaks the failure streak.
Useful? React with 👍 / 👎.
| # REPEAT_ERROR: incremental streak, survives across poll ticks | ||
| bash_streak = update_repeat_streak(bash_streak, ev) | ||
| if bash_streak >= repeat_n and not incident_reported: |
There was a problem hiding this comment.
When the same failing command is retried with the default thresholds, the third event is claimed by detect_loop above this block: --once returns immediately, while live mode sets incident_reported, so this REPEAT_ERROR branch cannot emit. Consequently the exact “same bash command fails N times consecutively” case documented in the detector table is always reported as LOOP instead; run the more specific failure detector first or otherwise allow it to supersede LOOP.
Useful? React with 👍 / 👎.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| if not isinstance(obj, dict): | ||
| return generic | ||
| ts = obj.get("timestamp") or obj.get("ts") | ||
| exit_code = obj.get("exit_code", obj.get("return_code")) |
There was a problem hiding this comment.
🟡 Claude failures never form streaks
Claude records each Bash call and result separately, so parse_events never combines the tool name with its exit status. Each event resets bash_streak, preventing every Claude REPEAT_ERROR alert.
Learn more
Claude transcripts place a tool_use block in an assistant message and its outcome in a later tool_result block. The invocation normally has no exit status, while the result does not repeat the tool name. parse_events handles the invocation but returns generic activity for the result, so update_repeat_streak sees neither a complete failed Bash event nor consecutive failures.
Example: A Bash tool use with ID call-1 is followed by a tool_result for call-1 containing exit code 1. The parser emits a Bash event with exit_code=None, then an event with tool=None; the streak remains zero instead of becoming one.
Recommended fix: Track tool-use IDs and correlate result events with their original tool names, then normalize a structured or parsed process exit status into the completed event. Update the streak only when that completed result arrives.
Was this helpful? React with 👍 or 👎 to provide feedback.
| # REPEAT_ERROR: incremental streak, survives across poll ticks | ||
| bash_streak = update_repeat_streak(bash_streak, ev) | ||
| if bash_streak >= repeat_n and not incident_reported: |
There was a problem hiding this comment.
🟡 Repeated failures become LOOP alerts
With default thresholds, three failures of one bash fingerprint emit LOOP, never REPEAT_ERROR. detect_loop claims the shared incident before the repeat-error update runs.
Learn more
The watch loop evaluates LOOP before it updates the failed-command streak. On the third identical call, LOOP reaches its default threshold first. In --once mode it returns immediately; in live mode it sets the shared incident_reported flag, which suppresses REPEAT_ERROR.
Example: Three events for bash false, each with exit code 1, produce LOOP: tool 'bash' repeated 3+ times. The documented same-command REPEAT_ERROR pattern never produces a REPEAT_ERROR verdict.
Recommended fix: Update and evaluate REPEAT_ERROR before the generic loop detectors, or define explicit detector precedence that selects REPEAT_ERROR when both become true on the same event.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
What
New 4th detector REPEAT_ERROR: fires when a
bashcommand exits non-zero--repeat-ntimes in a row (default 3, sticky — reset by a success or a non-bash event, not a sliding window).Catches the classic burn pattern: agent retries a failing build/test command over and over instead of changing approach.
Why this shape
--once(caught by code review; initial implementation lost the streak between poll ticks)exit_code/return_codenormalised at parse time;boolexcluded (exit_code: trueis not a failure)--repeat-n NCLI flag, documented in README detectors tableTests
58 tests (48 pre-existing + 10 new), green on the full CI matrix (ubuntu/macos/windows × 3.10/3.13):
repeat_error.jsonl→ exit 1,repeat_error_reset.jsonl(2 fails + 1 success) → exit 0select()on pipes unsupported there)Verified locally with a live repro: 3 failed bash events appended 2.5s apart →
REPEAT_ERROR: bash failed 3 times consecutively.