Skip to content

feat: single bounded retry layer for agent LLM calls - #167

Merged
SkyeAv merged 1 commit into
net/error-codefrom
net/llm-retry
Sep 15, 2026
Merged

SkyeAv merged 1 commit into
net/error-codefrom
net/llm-retry

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Sep 15, 2026

Copy link
Copy Markdown
Owner

Replaces three stacked, literal-"429"-matching retry layers with exactly one bounded, jittered retry layer shared by the inner agent, reflexion, and judge — with credentials redacted at the retry boundary. The old stack burned ~16 min per failing call and retried permanent errors.

Retry layer

  • agent.make_retrying_model(...) wraps any smolagents Model.generate through net.retry_transient with injectable sleep/rng/secrets: 4 attempts, sleeps summing to ≤ 45.0 s per call (LLM_MAX_TOTAL_BACKOFF).
  • Provider retryers disabled: build_model returns bare providers with retry=False (OpenAI additionally client_kwargs={"max_retries": 0}); LiteLLM's backend-internal retry is not exposed by LiteLLMModel.__init__ and stays outside this control. The CLI composes retry INNER of distilling at all three model roles, passing secrets=(resolved_key,).
  • Worst case: 29 logical calls per article (20 agent + 3 reflexion + 6 judge) × 45 s = 1,305 s ≈ 21.8 min, inside the 90-minute budget.

Coded failures and redaction

  • LlmTransientError (llm-transient) is raised on exhaustion and persisted via the fix: signal coded supervisor skips #166 error_code field; errors.error_code_of's cause-walk preserves the code through smolagents AgentGenerationError wrappers.
  • errors.redact_secrets(text, secrets=()) (named-group regex with quote backreference; exact configured secrets first) is applied at the retry log kwargs and both transient-error constructors — a quoted-JSON credential cannot leak into logs or state.
  • Metadata preserved: RetryingModel explicitly copies model_id/flatten_messages_as_text/custom_role_conversions/client (the Model base defines them, so __getattr__ cannot delegate) and overrides supports_stop_parameter.

Testing

  • uv run pytest tests/test_agent_assembly.py -q --no-cov -n 021 passed; tests/test_errors.py11 passed (redaction: exact secrets, quoted JSON, no over-redaction of benign text)
  • CI-equivalent base env (no [agent] extra): smolagents-touching tests guard with pytest.importorskip
  • Full gate on the stack tip: uv run pytest -q1565 passed, 15 skipped; ruff/format/pyright clean; Tier-2 passed after three rounds (metadata collisions, credential leakage, AgentGenerationError cause-walk all repaired)

@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 1fe243c2-2d5e-4f48-abb7-dc69dd8badeb


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@SkyeAv
SkyeAv added this pull request to stack #170 September 15, 2026 17:40
@SkyeAv SkyeAv changed the title feat: [US-005] single bounded retry layer for agent LLM calls feat: single bounded retry layer for agent LLM calls Sep 15, 2026
@SkyeAv
SkyeAv force-pushed the net/llm-retry branch 2 times, most recently from 4220147 to c06a8fc Compare September 15, 2026 17:53
@SkyeAv
SkyeAv marked this pull request as ready for review September 15, 2026 18:07
The smolagents retryer sleeps 120-240 s and 240-720 s on its two rate-limit retries and only matches 429-style strings, so bare 502/503/504, DNS, and read-timeout failures were never retried while 429 storms burned minutes per exhausted call. Add LlmTransientError and make_retrying_model so all three model roles retry through net.retry_transient with jittered backoff, Retry-After honoring, and a 45-second per-call budget (29 logical calls ≈ 21.8 min worst case), disabling smolagents' and OpenAI's own retryers at build_model while keeping bare provider types for existing tests. The CLI composes the wrapper inner of distillation so recordings stay one-per-logical-call, error_code_of walks wrapper exceptions so llm-transient is still persisted, and configured API keys are redacted from retry logs and coded error messages.
@SkyeAv
SkyeAv merged commit 96d0208 into main Sep 15, 2026
5 of 9 checks passed
SkyeAv added a commit that referenced this pull request Sep 15, 2026
Cut 19.0.0 and bump the package version in pyproject.toml, CITATION.cff,
and uv.lock.

Major: five breaking changes ship. The `tablassert` console command now
requires the optional `[cli]` extra, since `cyclopts` and `rich` left the
base install (#187). Level-one normalization redefines fullmap keys as
cleaned Unicode-lowercase, Porter2-normalized, byte-ordered token sets and
moves the database to schema v6, so every schema-v5 fullmap is rejected and
must be rebuilt (#171, #172, #174, #175, #176). `build-fullmap --aria2c`
is gone -- aria2c is used automatically whenever the `[aria2]` extra is
installed (#178) -- and `--taxon-allowlist` is gone because the built-in
top-100 experimental-taxon allowlist now applies to every build, guarding
both reuse paths by `META.taxon_allowlist` identity (#153). Logging now
requires the `[log]` extra and is fully disabled without it (#152).

Features: `tablassert.net` is a new stdlib-only transient/permanent
classification and retry seam shared by the agent and the BABEL downloader
(#163, #164), with one bounded jittered retry layer across the inner agent,
reflexion, and judge (#167) and machine-readable `error_code` values on
skipped checkpoint records (#166). `build-kg` gained an automatic ephemeral
shared-prefix TCode cache keyed by content-addressed XXH64 op digests
(#155-#159, #161). The `--distill` corpus became schema-uniform v2 with a
sibling `outcomes.ndjson`, a deterministic `RewardConfig`-tunable reward,
and the new zero-dependency `tablassert distill-weigh` command (#180-#183).
`CLASS_FIELD_OVERRIDES` grants the DAKP sparse qualifier stack on the pinned
association classes (#188).

Fixes: `distill-export` partitions its corpus by content and unions the
schema across record files, so a mixed v1/v2 directory no longer CastErrors
or silently stringifies a column (#184); BABEL retry warnings survive the
loguru sink; PMC downloads are idempotent, atomic, and bounded-parallel
(#165); reflexion and judge prompts are hard-bounded (#168).

Changelog:
- versioned the Unreleased section as 19.0.0 - 2026-09-15
- merged three duplicate `### Added` blocks into one
- moved the two `BREAKING:` entries from `### Changed` into
  `### Breaking Changes` and added a `**Migration:**` note to all five
- added PR links to all 17 entries, which carried none
- added the missing entries: the distill v2 corpus/reward/distill-weigh
  stack (#180-#183), the distill-export schema-drift fix (#184), the
  TCode run-cache detail (#155-#161), and the README badge removal (#154)

Docs: none needed here. Every shipped doc change landed with its own PR,
and the docs source-of-truth gate passes against the bumped tree.

Testing:
- make check -> exit 0
- uv run ruff check . -> All checks passed!
- uv run pyright -> 0 errors, 0 warnings, 0 informations
- uv run pytest -> 1622 passed, 52 skipped
- cargo test --manifest-path rust/Cargo.toml -> 156 passed, 0 failed
- cargo clippy --all-targets -- -D warnings -> clean
- uv lock --check -> resolved 169 packages, lock current
- uv run mkdocs build --strict -> exit 0
- docs SSOT + CLI coverage after the changelog edit -> 204 passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant