Skip to content

[2/5] feat(llm): Estimate LLM editability - #149

Open
thedrow wants to merge 4 commits into
iepathos:masterfrom
thedrow:feat/llm-editability
Open

thedrow wants to merge 4 commits into
iepathos:masterfrom
thedrow:feat/llm-editability

Conversation

@thedrow

@thedrow thedrow commented Oct 12, 2026 •

Copy link
Copy Markdown

Stacked on #148. Review only the last commit (feat(llm): Estimate LLM editability).

Summary

  • Estimate how reliably each model tier (small to frontier) can edit a function, from complexity, nesting, length, coverage, context tokens and ripple (callers).
  • Emit per-model scores, a verdict, and ranked improvement actions in JSON and markdown output.
  • Opt in with [llm_editability] enabled = true. Model profiles and per-language settings are configurable, and debtmap llm-models lists them.

This is PR 2 of 3. Next: recent LLM churn (#150).

Test plan

  • Unit and property tests for llm::
  • cargo test --bin debtmap, output_validation_test

Not yet done

  • Documentation: the book and README do not yet cover the [llm_editability] section (models, profiles, languages, thresholds, weights), the llm-models command, and the JSON/markdown fields.

How to stack these

These pull requests target master. From a clone of iepathos/debtmap, link them bottom to top, then merge the stack:

gh stack link 148 149 150 151 152
gh stack merge

gh stack link records the stack on GitHub. gh stack merge merges that stack in one step. Pass a pull request number to stop at that pull request. Order: CRAP scores (#148), editability (#149), churn (#150), topics (#151), content reverts (#152).

CRAP (Change Risk Anti-Patterns) combines cyclomatic complexity with test
coverage: cc^2 * (1 - cov)^3 + cc. It is computed at output time from the
existing complexity and coverage data, so no analysis types change.

- JSON: metrics.crap is a number with coverage data and null without it,
  so consumers can parse the key unconditionally.
- Terminal and LLM markdown output show it next to coverage.
- [crap].coverage_source selects direct (default) or transitive coverage.
Score each function's editability per model tier (small to frontier)
from complexity, nesting, length, coverage, context tokens and ripple,
with ranked improvement actions. Opt-in via [llm_editability].
@thedrow
thedrow force-pushed the feat/llm-editability branch from e2f8cfa to 60f4040 Compare October 13, 2026 02:18
Match custom profile ids ignoring case, and let llm-models run
when a configured model name is invalid. Clamp callee advice to
the function's own size, reject non-finite weights, and keep
editability fields out of v3 JSON.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant