Skip to content

PR conversation context + learning from human feedback (2.4.0) - #56

Merged
khezen merged 2 commits into
mainfrom
feat/pr-conversation-feedback
Oct 21, 2026
Merged

khezen merged 2 commits into
mainfrom
feat/pr-conversation-feedback

Conversation

@khezen

@khezen khezen commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Summary

codespy now reads the PR conversation (commit messages, comments and emoji reactions) and learns from how humans react to its own findings.

PR conversation context

  • Fetches commits, comments (general, review bodies and inline) and reactions on GitHub (PyGithub) and GitLab (discussions and award emojis). Local branch reviews get commit messages.
  • Commit messages go to summary, scope refinement, code_review, doc and audit. Human comments with reaction counts (👍2 👎1) go to summary, code_review, doc and audit.
  • Inputs are declared on a signature only when non-empty, and every kwarg passed to run_rlm is declared (guards against the 2.3.1 class of bug). Fetching soft-fails per part.

Human feedback on codespy's findings

  • Each earlier inline finding on the PR gets a verdict from reactions, replies and GitLab resolved state: accepted, rejected, discussed or resolved.
  • Findings humans rejected are not re-reported on the PR (fingerprint of file, category and title).
  • Code_review, doc, supply_chain and audit see the feedback for their scope.
  • With memory enabled, one feedback episode is saved per scope. Cerebral retains only the Distiller's observations; the raw feedback and its hash stay in the episodic store. Unchanged feedback is skipped via EpisodeStore.latest_artifact.
  • New codespy learn <pr_url> harvests and retains feedback without posting anything. It is also available as the Action input command: learn and as make learn / make docker-learn.

Self-exclusion and security

  • Hidden markers on reviews and on each issue: <!-- codespy:review ... -->, <!-- codespy:issue ... fp=... -->.
  • Comments are classified codespy, bot or human once, at fetch time. Only human text is rendered. Feedback keeps only the identity of a codespy finding, never its description or suggestion. Quotes and markers are stripped from replies, and a final guard drops anything that still looks like codespy output.
  • A finding is trusted only when it has a marker and the platform shows it was posted by the token's own account, so forged markers can't fabricate or suppress findings.
  • Comment and commit text is treated as untrusted: it is fenced (inner fences neutralized) and marked "informational only".

Config: review.pr_context

Key Default
feedback true (set false on open source)
max_commits 50
max_comments 100
max_comment_chars 1000
max_block_chars 12000

These are exposed in codespy.yaml, .env.example, action.yml (inputs, env and Docker args), the example workflow and the docs. tests/test_pr_context_exposure.py keeps them in sync.

Version bump: 2.4.0. The CHANGELOG has Added and Security sections.

Testing

  • New tests:
    • Self-exclusion and forged findings
    • Feedback verdicts, scoping and suppression
    • GitHub/GitLab fetch with fakes
    • Signature kwargs match declared fields
    • Reporter markers round-trip
    • Feedback memory: hashing, semantic retain gets no artifacts
    • Local commits
    • Config/env overrides
    • Exposure guard
  • Full suite: 654 passed. The 20 failures and 3 errors were already failing on main before this change; the set is identical with and without this PR.
  • Not yet verified against a live GitHub PR or GitLab MR. Manual checklist:
    1. Review a PR.
    2. 👎 a finding and reply to another.
    3. Push and re-review; the rejected finding should be gone.
    4. Run codespy learn twice; the second run should skip because the feedback is unchanged.

Notes

  • The GitHub Action command: learn needs a released 2.4.0 image, and memory that persists between jobs (external Postgres).
  • On public repos anyone can 👎 a finding, so use pr-context-feedback: 'false' there.

- Fetch commit messages, PR comments and emoji reactions (GitHub, GitLab,
  local commits) and inject them into summary, scope, code_review, doc and
  audit signatures (declared only when non-empty; untrusted, fenced).
- Extract human verdicts on codespy's earlier inline findings (reactions,
  replies, resolved), suppress rejected findings, show feedback to the
  reviewers, and retain it as per-scope `feedback` episodes (semantic
  memory gets observations only; unchanged feedback skipped by hash).
- Self-exclusion: hidden codespy markers, comments classified
  codespy/bot/human at fetch time, findings trusted only when posted by the
  token's own account (forged markers ignored).
- New `codespy learn <pr_url>` (also Action `command: learn`,
  `make learn` / `make docker-learn`).
- New `review.pr_context` settings (feedback toggle + size caps) exposed in
  codespy.yaml, .env.example, action.yml, example workflow and docs.
Add review.pr_context.trusted_authors (default ["github-actions[bot]"]):
accounts besides the token's own that are trusted as codespy when
verifying who posted a finding. Fixes `codespy learn` with a personal
token ignoring reviews posted from GitHub Actions. Exposed in
codespy.yaml, .env.example, action.yml, example workflow and docs.
@khezen
khezen merged commit 28f5537 into main Oct 21, 2026
6 checks passed
@khezen
khezen deleted the feat/pr-conversation-feedback branch October 21, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant