Skip to content

convergence-review refactor - #8

Draft
kalantar wants to merge 3 commits into
inference-sim:mainfrom
kalantar:convergence-review
Draft

kalantar wants to merge 3 commits into
inference-sim:mainfrom
kalantar:convergence-review

Conversation

@kalantar

Copy link
Copy Markdown
Collaborator

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>
@kalantar
kalantar marked this pull request as draft March 10, 2026 14:10
@kalantar

Copy link
Copy Markdown
Collaborator Author

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>
@jgchn

jgchn commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 10, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

1 similar comment
@claude

claude Bot commented Mar 10, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 10, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 10, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 11, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 11, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 11, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 11, 2026

Copy link
Copy Markdown

Claude Code is working…

I'll analyze this and get back to you.

View job run

@jgchn

jgchn commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

@claude summarize this PR

@claude

claude Bot commented Mar 12, 2026 •

Copy link
Copy Markdown

Claude finished @jgchn's task in 1m 0s —— View job


PR Summary

  • Gather context and read key files
  • Write summary

This PR adds a convergence review system — an automated, multi-round code/document review tool that dispatches parallel reviewer perspectives, consolidates findings, and iteratively fixes issues until convergence (zero CRITICAL + zero IMPORTANT findings).

What it adds

Core skill (.claude/skills/convergence-review/):

  • SKILL.md — Skill definition supporting 12 gate types: design docs, macro plans, PR plans, PR code, cross-system variants, and hypothesis experiment artifacts
  • orchestrator.py (799 lines) — Main round loop: REVIEW → ASSESS → FIX, with stall detection, cross-round ledger tracking, and persistent theme escalation
  • reviewer.py — Parallel async reviewer dispatch via OpenAI-compatible API (LiteLLM). Includes JSON + markdown response parsing with LLM fallback extraction
  • assessor.py — Deduplication, validation, re-prioritization of raw findings. Implements cross-round status tracking (novel, regression, recurring-fixed, recurring-skipped, recurring-escalate)
  • fixer.py — Fix phase using claude -p subprocess with structured fix summary extraction and LLM fallback
  • summary.py — Human-readable summary file writer with progress tables and persistent themes
  • progress.py — JSON event emitter for stdout (machine-readable progress)
  • prompts.py — Dynamic perspective discovery from markdown prompt files
  • models.py — Data classes (Finding, Assessment, FixResult, RoundResult, etc.)

Prompt libraries:

  • design-prompts.md (900 lines) — 8 perspectives each for design docs, macro plans, and cross-system variants
  • pr-prompts.md (536 lines) — 10 perspectives for PR plans/code, 4-5 for cross-system variants

Hypothesis experiment skill (.claude/skills/hypothesis-experiment/):

  • SKILL.md — Defines a structured hypothesis testing workflow
  • review-prompts.md — Review perspectives for hypothesis designs, code, and findings

Tests (7 test files, ~1,000+ lines):

  • Unit tests covering assessment parsing, convergence logic, fix phase, round lifecycle, round protocol, setup validation, and parse fallback

Architecture highlights

  • Three-phase rounds: Review (parallel, cheap) → Assess (single strong model, dedup/validate) → Fix (claude -p, sequential edits)
  • Cross-round intelligence: Assessor receives a ledger of prior findings + fix dispositions to avoid re-raising addressed issues
  • Stall detection: Sliding window monitors finding counts; exits early if not converging and no fixes being applied
  • Persistent themes: Findings flagged in 3+ rounds are escalated as design trade-offs, surfaced for human review but don't block convergence
  • Configurable models: Reviewer model (default: Haiku, fast/cheap) and fixer model (default: Opus, strong) are independently configurable

@jgchn

jgchn commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

@claude what are some highlights of this PR? Be extremely concise. Five sentences max.

@claude

claude Bot commented Mar 12, 2026 •

Copy link
Copy Markdown

Claude finished @jgchn's task in 18s —— View job


PR Highlights

This PR introduces a convergence review system — an automated multi-round review tool that dispatches parallel reviewer perspectives, deduplicates findings, and iteratively fixes issues until zero CRITICAL/IMPORTANT findings remain. It supports 12 gate types across design docs, macro plans, PR code, cross-system variants, and hypothesis experiments. The architecture uses a three-phase round loop (Review → Assess → Fix) with cross-round intelligence so the assessor won't re-raise already-addressed issues. Stall detection and persistent theme escalation prevent infinite loops while surfacing recurring design trade-offs for human review. The PR includes ~4,500 lines of Python, ~1,850 lines of review prompt libraries, and ~1,000 lines of unit tests across 7 test files.


@kalantar

Copy link
Copy Markdown
Collaborator Author

@claude review this pr

1 similar comment
@jgchn

jgchn commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

@claude review this pr

@claude

claude Bot commented Mar 12, 2026 •

Copy link
Copy Markdown

Claude finished @jgchn's task in 8m 32s —— View job


Code review

Found 1 issue:

  1. Field name mismatch in build_prior_round_context: line 257 of orchestrator.py reads f.get("rationale", "") but the fixer schema (fixer.py lines 40, 141) and assessor (assessor.py line 160) use "reason" as the field name. This means skipped-fix rationales will always resolve to empty strings in the prior round context shown to reviewers in subsequent rounds. Fix this →

for f in fix_result.fixes_skipped:
desc = f.get("description", f.get("id", ""))[:150]
rationale = f.get("rationale", "")[:100]
if desc:
skipped_descriptions.append((desc, rationale))

Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.


@kalantar

Copy link
Copy Markdown
Collaborator Author

@claude review this PR.

Signed-off-by: Michael Kalantar <kalantar@us.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants