Skip to content

rule-design.md

Latest commit

 

History

History
380 lines (288 loc) · 49 KB

rule-design.md

File metadata and controls

380 lines (288 loc) · 49 KB
Raw

Rule Design Reference

How + why rules are written as they are. For rule authors and contributors — not loaded at runtime.

Sources

Official

Academic

Vendor Prompting Guidance (2026-07 survey)

Industry

Model-Specific Failure Reports (2026-07 sweep)

Model-Specific Reports (2026-08 sweep)

Model-Specific Failure Reports (pre-2026-07)

Constraint Framing

No published study quantifies per-framing success rates; treat framing as design heuristics grounded in adjacent evidence:

Heuristic Grounding
Positive action first ("Only modify lines required by the task"), prohibition as reinforcement ("Never weaken a test to pass") House style, consistent with prompt-engineering guides; every rule answers "What to DO?" before "What NOT to do?"
Explicit fallback ("If uncertain, state 'not verified' and ask") Literal Interpretation below — models don't invent unstated recovery paths
Unstated constraints fail — a constraint not written is a constraint not applied Anthropic literal-instruction guidance; Instruction Hierarchy (arXiv 2404.13208)
Front-load critical constraints; mid-context placement degrades Context rot, 18-model study (Chroma 2025)
Prompt rules shape behavior measurably, but mechanical gates dominate Guardrails Beat Guidance (arXiv 2604.11088); claude-code #50235: hook-enforced ≈100% vs behavioral moderate-to-low
Outcome-first: state the outcome, constraints, evidence, completion bar — not micro-steps GPT-5.6 guide (removing repeated instructions: ~10-15% eval gain, 41-66% fewer tokens); Moonshot: step micromanagement "may interfere with autonomous decision-making"
Plain Markdown is the safest cross-model format; heavy XML/JSON structure degrades weaker models Format effects model-dependent, dwarfed by capability (arXiv 2411.10541, 2605.29676); no study isolates table-vs-prose for rules specifically (open question)
Pair critical prohibitions with a declarative restatement where cheap ("Never X — X: not permitted") Imperative-interference finding (arXiv 2603.25015) — single study, applied sparingly, not as a rewrite mandate; framing-direction effects are task-dependent (2605.05391, 2602.04306)
User-facing labels self-describing for zero-technical readers (e.g. "Small/Large task", never "Tier 1/2") Maintainer requirement (2026-07): rules serve the user's process management, not only model reliability
Adaptive binding: name capabilities with defaults, not mandates — host/project-native equivalent preferred, named default as fallback, capability never optional 7/8 harnesses ship native plan/todo mechanisms (2026-07 survey) that a hardcoded artifact name would conflict with or duplicate; vendor conventions are short-lived (Windsurf→Devin Desktop rebrand). Constraint: most native todo tools are session-ephemeral, so the persistence requirement stays hard — only the binding adapts
Literalism hardening: condition concrete names at the rule site ("GitHub: gh run watch; other platforms: their equivalent"), not only in a global preamble — and keep the concrete example, don't fully abstract Agents follow file instructions literally even when counterproductive (ETH 2602.11988); a single global disclaimer fades over long contexts (context rot / Gate Recall logic) while the local rule text stays salient; but full abstraction hurts weaker models — examples reduce misinterpretation (Example Density above)
Host-layer deduplication: before adding a rule, read the target host's system prompt — already stated there → portable supplement, never the always-on core Anthropic removed >80% of Claude Code's system prompt for Claude 5 generation models with no measurable eval loss (2026-07-24 blog). The named failure is conflicting guidance, e.g. "leave documentation as appropriate" in one layer against "DO NOT add comments" in another: the model must resolve the clash before acting. Two layers asserting the same thing is not redundant safety
Hard thresholds and absolute negatives need a documented case where judgment alone failed; otherwise state the outcome and let context decide Same source: the removed rule "default to writing no comments. Never write multi-paragraph docstrings" was replaced by "Write code that reads like the surrounding code: match its comment density, naming, and idiom" — the hard form was wrong for a real subset of prompts, and newer models handle the decision unaided
Prefer interface design over usage examples: expressive parameters, enumerated states, and explicit IO contracts guide better than sample invocations Same source: examples constrain the model to the exploration space they demonstrate. A Todo tool's pending/in_progress/completed enumeration conveys intended use more cheaply than an example call
Rich references beat prose specs: a test suite, a working function to port, an HTML mockup, or a rubric carries higher-fidelity intent than a description of the same thing Same source: "prefer files that are in code as it provides clear, high-fidelity instructions to Claude in a language it knows very well"; rubrics let verifier agents check taste (e.g. what good API design looks like) rather than restating it as rules
Process-management rules: session-resume re-anchor, mid-task additions → artifact, escalation always carries 2-3 options + recommendation Maintainer requirement (2026-07, first-hand: forgets tasks across long/parallel sessions); mid-task-addition pattern proven in the 2026-07 overhaul session itself; options-bearing escalation cuts user decision latency

Example Density

Optimal example count per rule:

Examples Effect
0 Rule abstract — frequently misinterpreted or ignored
1-2 Minimum viable — covers happy path
2-3 Optimal — happy path + edge case + error recovery
> 3 Diminishing returns — increases token cost without proportional benefit

Quality over quantity: 3.5% well-selected training data outperforms the full-data baseline by 0.71% on average (arXiv 2602.13773). Same principle applies to rule examples — few precise, representative examples > many generic ones.

Ordering matters: Place most relevant or complex example LAST. AI models exhibit recency bias — final example has strongest influence on behavior.

Pattern per rule: 1) Happy-path case, 2) Edge case showing constraint in action, 3) Error/recovery case.

Token Efficiency

Format efficiency rankings for same information:

Format Token Efficiency Best For
Tables Best Structured comparisons, option lists
Numbered lists Good Sequential procedures, checklists
Bullet lists Good Unordered sets, short items
Prose paragraphs Worst Avoid for rules — use only for context/rationale

Savings strategies:

  • Remove redundant preamble ("In this section we will discuss..."): ~10% savings
  • Use shorthand for repeated concepts ("If X -> Y" not "In the case where X occurs, the appropriate action is Y"): ~15% savings
  • Compress examples to outcome, not full trace: ~20% savings

Budget (measured 2026-07 post process-quality additions, chars/4): main rules file always loaded (~5,200 tokens, 184 lines). The evidence-based prune (toolchain-detection table deleted; log-level/commit/complexity/LSP tables compressed) was spent on higher-value rules (CI Ownership, both-ways Test Integrity, three-field Outcome Report, recurring-issue→mechanical-guard) — net tokens ≈ unchanged, signal density up, instruction count down. Reference files add ~1,400 (operations) to ~3,400 (safety) tokens when conditionally loaded. Total worst case: ~9,000 tokens. Leaves maximum context for actual codebase.

Behavioral Anchoring

Instruction-following degrades once a single prompt carries more than ~150-200 discrete instructions (arXiv:2507.11538); in long sessions, context filling causes earlier instructions to be forgotten (Claude Code best practices). Prevention strategies:

  1. External artifacts: Write progress + decisions to files, not conversation memory
  2. Phase-boundary repetition: Repeat core constraints when transitioning between major work phases
  3. Structured progress: Use explicit "completed / current / next" tracking
  4. Re-read before modify: After context gap (compression, long pause) → re-read source files — conversation memory unreliable

Adaptive Thinking

Forced CoT adds only 2.9-3.1% accuracy improvement for reasoning models while costing 20-80% more time (Wharton GenAI Labs 2025). Implications for rule design:

  • Do not force step-by-step reasoning for tasks model can handle directly
  • Reserve explicit reasoning prompts for genuinely ambiguous or multi-step decisions
  • Use gates, not chains: "Verify X before Y" cheaper + more effective than "Think through X step by step, then do Y"
  • Focused CoT (arXiv 2025): when reasoning needed, constrain to specific decision point, not entire task

The Operating Loop & Mechanical Verification

rules.md opens with an explicit Target → Assess → Gap → Verify-each → Reconcile loop. Rationale:

The gap between a frontier model and a cheaper one concentrates in three places — long-horizon plan coherence, reliable self-verification, and instruction-following under load. All three are externalizable, and the loop mechanizes them:

  • Target / Gap convert open-ended "vibe coding" into a spec-bounded diff — kills scope creep (W3) for any model, and forces the problem-specific check a weak reasoner skips (DeepSeek V4 applies familiar templates without verifying the problem's preconditions).
  • Assess grounds claims in read reality, not memory — the direct mitigation for confident-prose fabrication (Opus 4.7 #50235: emits real-looking but fabricated commit hashes / file paths) and for hallucination (W1) generally.
  • Verify-each + "done = external signal" is the highest-leverage move: it replaces the model's self-judgment — unreliable enough that 19.71% of agent-"solved" tasks fail adversarial re-verification (SWE-ABS, W10) — with a machine-checkable signal a passing test does not fake.

Why mechanical, not behavioral — and the limit of a prompt file. Opus 4.7 field data (#50235) found rules enforced via tool-call hooks reach near-100% compliance, while the same rules enforced purely as in-context behavioral guidance reach moderate-to-low compliance — and fabrications "bucket-bypass" into whatever output form (tags, labels, fields) a prose-level rule failed to name. Two design consequences:

  1. A prompt file cannot be a mechanical gate; its leverage is to make the model defer to mechanical signals it can run (tests, builds, linters, exit codes, diffs). True enforcement (hooks, SAST, sandbox, tool_choice) lives in the harness/CI — see CLAUDE.md "Out of Scope".
  2. Anti-bypass framing: a rule that constrains one output form must name all forms (see Grounded Specifics — "prose, labels, tags, fields"), or the failure migrates to the unnamed one.

The model difference narrows to the degree behavior is mechanized; it closes only where a real gate runs.

Automation Ladder

Encode domain knowledge at the most mechanical level that can express it — each step down is cheaper, later-triggered, and less binding:

type system > test > lint/CI step > harness hook > on-demand skill > always-on prompt rule

A prompt rule is the last resort for what cannot be mechanized. Grounding: hook-enforced rules reach ≈100% compliance vs moderate-to-low for the same rules as in-context guidance (claude-code #50235); "your agent could fix an issue every time it sees it, but that uses tokens and might miss cases — a lint rule, CI step, or routine automates that class forever" (Boris Cherny, x.com/bcherny status 2077460395279692197, 2026); independent practitioner analysis reaches the same detection-vs-enforcement split (dev.to analysis).

Consequences:

  1. rules.md › Process Framework carries the behavioral shadow: recurring issue class (3×) → propose a mechanical guard.
  2. A rule retires from rules.md when promoted to a mechanical gate (see Rule Lifecycle).
  3. Local-first gating (e.g. dev-skills ds-quality stop-hook loop) beats CI-first: CI detects but does not fix; a local gate fires while the agent can still fix immediately. CI remains the safety net, owned via rules.md › CI Ownership.

Rule Lifecycle

Intake: ≥2 documented real-world failure cases (CLAUDE.md pre-checks). Single first-hand observations enter as candidates only when the failure class has independent multi-model corroboration.

Pruning heuristic (apply to every line, every revision): "Would removing this cause the model to make mistakes? If not, cut it" (Anthropic best practices). Exclude anything inferable from the code/config itself, standard language conventions, and self-evident practices — restating what models already know adds cost without lift (Anthropic exclude-table; Windsurf/Devin docs: "no need to add generic rules... already baked into training data"; ETH 2602.11988: limit human-written instructions to non-inferable details).

Sunset check (each major revision): a rule is demoted or removed when (a) its failure mode is no longer reproducible in current target models, (b) it has been promoted to a mechanical gate in the deployment (Automation Ladder), or (c) the target host's own system prompt now asserts it, making a second copy a conflict rather than reinforcement. Record the retirement + reason here; never silently drop.

Profile addition record — 2026-08-12 (floor profile)

floor.md (six rules, ~600 tokens) added as the third profile after a measured A/B program (issue #3, rounds 1-4a, 32+ runs, both-direction-proven scorers): on Haiku-class models the full rules.md costs +17-24% tokens/run while its entire measured value concentrates in evidence discipline (the vacuous-verifier trap) and the guardrail floor; the six-rule floor reproduced 100% of that measured value (9/9 runs, all criteria, real-test evidence 3/3) at +1.0% tokens. Frontier models showed no outcome-level rule effect on single-to-multi-module tasks in any round — consistent with the lean-profile thesis. Full data: dev-rules issue #3 + knowledge repo repos/dev-rules/devrules-eval-round1-2026-08-11.

Retirement record — 2026-07-25 (Claude 5 context-engineering revision)

rules.md went from 187 to 118 lines. Nothing was deleted outright: every clause below moved to exactly one new home, verified present there before the removal landed.

Rule Moved to Reason
Scope Boundary · Over-engineering (YAGNI) · File Creation · External Content Injection · destructive-action confirmation · Code Standards · Error Messages · Error Handling · Format Preservation · Context hygiene · Code-Intelligence-First Navigation · Subagent Capability Routing · Tool Prerequisites references/portable-supplement.md (c) — the Claude Code system prompt asserts each of these for the Claude 5 generation; kept for weaker hosts as an explicit profile, not as an always-on default
The comments clause of File Creation ("never explain WHAT") rewritten in the supplement as "match the surrounding code's comment density" (a) + conflict — the hard form contradicted the host system prompt verbatim, reproducing the article's own documented failure case
The harness enumeration inside Tool-Call Result Verification (empty tool_calls after a tool-call finish, format drift, plain-text tool calls) references/portable-supplement.md › Tool-Call Failure Modes (b) — a harness owns these; CLAUDE.md § Out of Scope already excluded them, so the rule was self-inconsistent. The model-actionable core (verify by observed effect) stayed
Observability · Production Defaults · Database Changes · Idempotency references/operations.md Progressive disclosure — relevant in a minority of sessions, paid for in all of them. Idempotency and the timeout/rate-limit and no-environment-literals clauses had no prior home there and were written in as part of this move
Complexity Limits dev-skills ds-fix › Phase 4 Lint, framed linter-first (b) — Automation Ladder; a linter enforces thresholds, model recall does not
Severity Levels · Skip Patterns · Fix Quality dev-skills ds-review Already present there verbatim; the rules.md copy was cross-repo duplication
Commit History (WIP collapse) dev-skills ds-commit (--fixup + non-interactive autosquash) Already present there
Concurrency Safety · Security detail · i18n/a11y detail references/safety.md pointer · merged into Non-Functional Accountability Pointer pattern, not duplication
Migration Sweep · Refactor Pinning · Meaningful Test Data · Gate Recall · Session Resume · Mid-task Additions · Scope Expansion Stop absorbed inline into the gate they belonged to Not retired — merged to cut heading overhead without losing a clause

Retained deliberately despite the "cut it" pressure: Operating Loop, done-as-external-signal, Verify-Echo, Spec Artifact, CI Ownership, three-field Outcome Report, Decision Framing. These encode the maintainer's process requirements, not a model capability gap, so the shrinking-model-gap argument does not apply to them.

Effectiveness Evidence (what a rules file can and cannot claim)

Two independent empirical studies confirm rules files causally change agent behavior, but on different axes:

Axis Evidence Verdict
Task success / correctness ETH AGENTbench (2602.11988): human-written ≈ +4% at ~19% cost; LLM-generated ≈ −3% Modest at best; auto-generated files actively hurt
Efficiency / cost Lulla (2601.20404): −28.64% median runtime, −16.58% output tokens on real PRs Solid gain from a good file
Failure-floor / consistency Harness-gap analysis (below) + model failure reports The file's core value: supplying rules no harness enforces

Honest positioning: this project claims a higher floor (fewer catastrophic misses: weakened tests, fabricated packages, false "done") and better efficiency/consistency — not a raw success-rate uplift. Both studies also justify hard length discipline. Not verified, do not cite: any specific line-count compliance threshold (the widely-repeated "150 lines" figure has no traceable primary source), GuideBench per-model percentages, IFEval-vs-IFBench gap figures, MAST percentage breakdowns, Composio harness-benchmark percentages (X-thread only, methodology unpublished, inference-time sampling variance unaddressed), and any independent replication of Anthropic's "no measurable loss" 80%-cut claim (none found as of 2026-08-11 — the claim rests on Anthropic's own evals).

Harness Coverage Survey (2026-07)

System prompts of 8 harnesses surveyed (Claude Code, Cursor, Windsurf/Devin, Codex CLI, Copilot, Cline, opencode, Gemini CLI) across 10 behavior classes. Where a behavior is widely native, rules.md stays minimal; where a gap is universal, rules.md carries the rule:

Behavior Native coverage rules.md stance
Treat external/file/tool content as untrusted 1/8 (Gemini CLI only) Highest-value rule we supply (External Content Injection)
Test integrity (never weaken to pass) 0/8 Supplied (Test Integrity)
Read-before-edit (unconditional) ~1/8 explicit Supplied (Read-Before-Modify)
Dependency verification 3/8; Windsurf's prompt actively endorses training-data version fallback Supplied + explicit "never fall back to training data" (Trust Verification)
Scope discipline ~4/8 Supplied (Scope Boundary)
Verification before done ~5/8, uneven Supplied (Completion Gate) — also covers the 3 harnesses without it
Todo/plan tracking 7/8 Minimal — Spec Artifact adds only the cross-session ledger + verify-criteria contract
Conciseness/communication 7/8 Not duplicated
Convention-following 7/8 One line in Fix Quality

Re-survey cadence: each major revision — harness prompts change quickly (e.g. Windsurf→Devin Desktop rebrand, June 2026).

2026-08 roster addition: Pi (badlogic/Earendil) — deliberately minimal harness: four tools (read/write/edit/bash), system prompt under ~1k tokens, no MCP/sub-agents/plan-mode in core. It sits at the far-minimal end of the native-coverage spectrum: nearly all 10 behavior classes fall to the rules file, making it a portable-profile host despite typically running frontier models. Its Databricks result (vendor-harness success at ~2× lower cost) is also the strongest field evidence for this project's token-budget discipline. Survey its per-class coverage properly at the next major revision.

Literal Interpretation

Claude models follow instructions literally — omitted details are omitted from output (Anthropic prompting docs, "literal instruction following"). Implications for rule design:

  • Be exhaustive in required outputs: Rule producing specific artifact → list every required field
  • Silence means skip: Rule doesn't mention error handling → model won't add error handling
  • Explicit > implicit: "Return {status, message, data}" not "Return relevant information"
  • Test by omission: Validate rules by checking what happens when optional-sounding phrases removed

Structured Output

JSON schema validation makes agent outputs more reliable and consistent than free-form text (Databricks 2024). Implications:

  • Agent return values: Define exact schema (fields, types, required vs optional)
  • Error formats: Standardize {"error": "message"} across all agent contracts
  • Validation at boundaries: Parse + validate structured output before passing downstream
  • Prefer tables over prose for any data model must act on programmatically

AI Weakness Taxonomy

Systematic weaknesses in AI coding assistants. Rules address via specific mitigation strategies:

ID Weakness What Happens Rule Mitigation
W1 Hallucination Fabricates APIs, packages, file paths, commit hashes, prices Trust Verification + Grounded Specifics — every emitted specific must trace to an observation, in any output form; Finding Triage — confirm each reported finding against current code before planning a fix
W2 Tunnel Vision Edits file A, breaks file B Cross-file Consistency + Migration Sweep
W3 Scope Creep Reformats untouched code, adds unrequested features Scope Boundary + Over-engineering Prevention + File Creation no-residue rule — working artifacts (scratch files, debug scripts, one-off helpers) deleted at completion
W4 Memory Decay Relies on stale conversation context Artifact-First Recovery — re-read before modifying; Spec Artifact live ledger — mark [x] the moment a check passes, update the plan artifact on approach change (the artifact is the progress ledger, not conversation memory)
W5 Confidence Bias Assigns higher severity than evidence warrants Severity levels — when uncertain, choose lower; Finding Triage — unconfirmed findings are false positives, excluded from the plan
W6 Skip Tendency / Premature Completion Declares done before all steps executed — GPT-5.4 claimed uncollected subagent results collected (codex #14341); MiniMax verbalized 75-step plan, zero tool calls (arXiv 2602.19594); Kimi CLI shipped fixes for "ending goals prematurely"; first-hand DeepSeek V4 Flash case (2026-07) Completion Gate — explicit done checklist incl. no-residue; CI Ownership — pushed work not done until checks green; Spec Artifact — live ledger keeps unexecuted steps visible
W7 Redundancy Blindness Reports same issue multiple times; re-raises resolved concerns (Opus 4.7 loops) Deduplication in Fix Quality + settled-concern rule (don't re-litigate without new evidence)
W8 Injection Risk Unsanitized input in shell commands Security Awareness — quote paths, use --, reject metacharacters
W9 Concurrency Errors AI-generated code shows ~2× more concurrency + dependency-correctness issues than human-written (CodeRabbit 2025) Safety reference — explicit concurrency checklist
W10 Self-Verification Failure Self-judged "done" is unreliable — 19.71% of agent-"solved" tasks fail adversarial re-verification (SWE-ABS 2603.00520); false success 45-75% when self-reported vs 3% when independently verifiable (2606.09863); SWE-bench inflation 6.2pp (2503.15223) Completion Gate — state what changed + how to verify; Operating Loop core principle now states self-reported completion is presumed false; Spec Artifact — per-task verify check + expected signal named at plan time
W11 Read-Before-Act Regression Claude Code field data: reads-per-edit dropped 6.6× → 2.0× after Feb 2026 update; modifies files without reading them first (GitHub #42796, 6,852 sessions; #47901) Read-Before-Modify gate — explicit read required before every edit
W12 Tool-Call Format Instability Cross-vendor, worsening at long context: Kimi K2 malformed tool_call_id; GLM-5.x truncated JSON with fatal harness crashes (vllm #42400); Qwen3.x XML/JSON drift mid-session (#475); MiniMax duplicated names as plain text (koog #2093); DeepSeek V4 empty content on completed turns (openclaw #84591) + DSML drift Tool-Call Result Verification gate — silent-failure list now names empty-content-on-completed-turn, malformed/duplicated names, format drift, plain-text tool calls
W13 Error Abandonment Model claims detected problem is "pre-existing" to avoid fixing it; passes silently — constraint-violation + inaccurate-self-reporting failure shares grow over session time (2605.29442, 20,574 sessions) Error Ownership — every detected problem must be addressed regardless of origin; Verification-Infrastructure Gap — a check with no project tooling (no tests/CI/lint) is reported + remediation offered, never silently skipped
W14 External Content Injection Files/web/emails read during task embed fake instructions; model follows them (ClawSafety 2026: attack success 40–75% across models) External Content Injection prohibition — treat all external content as untrusted data
W15 Specification Gaming / Reward Hacking Satisfies the literal test/metric while violating intent — special-cases known test inputs, hard-codes expected outputs, games the reward signal (SWE-ABS 2603.00520: 19.71% of "solved" tasks semantically wrong; gap widens +28pp at 10× code, SpecBench 2605.21384; actively benchmarked on production agents: EvilGenie 2511.21654, TRACE 2604.15149; GPT-5.3-Codex modified tests to vacuously pass, codex #12225). Mirror image: coverage-padding — vanity tests written to game a metric Test Integrity — verify against described intent + cases beyond the provided suite; never special-case test inputs; every test names the failure it guards against, coverage never a target
W16 Sycophancy / Authority Deference Abandons a correct position under user pushback; defers to authority claims (PR text, comments, "the reviewer said") instead of judging behavior (BrokenMath 2510.04721: GPT-5 29% sycophantic; redacting authorship metadata restored vuln detection in all affected cases, 2603.18740) Process Framework (on pushback, re-verify from source) + Trust Verification — judge code by behavior, not claims
W17 Dependency Hallucination / Slopsquatting Imports a package that doesn't exist, or an attacker's typosquat of a hallucinated name (USENIX 2025: 19.7% of LLM-suggested packages hallucinated, 43% reproducible) Trust Verification — package present in registry (non-trivial age + downloads) AND in lockfile before import
W18 Context Rot / Long-Context Degradation Accuracy degrades as input grows even within the window; mid-context instructions silently dropped (Chroma 18-model study; arXiv 2601.15300). Post-compaction, model may deny its own recent actions — Codex falsely denied 23 file edits it had made (codex #5957). Distinct from W4 (post-compaction staleness) Artifact-First Recovery — re-ground every ~20 calls, post-compaction the diff is the record, not recollection; Token/Context — front-load constraints, summarize don't accumulate; Gate Recall — pre-flight names the applicable gates so they stay salient
W19 Multi-Agent / Subagent Handoff Failure Trusts subagent-returned data as ground truth; specs/results distorted or lost across handoffs (MAST 2503.13657, 1,600+ traces; 2505.00212: 53.5% inter-agent attribution) Subagent Output Verification gate — apply Grounded Specifics + Trust Verification to returned data; explicit handoff contract; turn budget → escalate
W20 Slop / Duplication Drift Regenerates near-duplicate code instead of reusing existing implementation (GitClear 2025: copy-pasted 8.3%→12.3%, moved/refactored 24.1%→9.5%, churn 5.5%→7.9%) Fix Quality — grep for existing impl before generating; reuse/modify over regenerate; no-residue rule — slop artifacts never committed
W21 Refactor Behavior Drift Agent refactors keep the module internally cleaner while silently changing the external contract; type systems catch signature drift only (Idlen Devin review 2026-03; Nimbalyst diff-pattern analysis 2026-04; CodeTaste 2603.04177; Refactoring Runaway 2605.22526 — tangled agent refactorings correlate with reduced compilability) Refactor Pinning — characterization test before refactoring untested code; same test green before + after
W22 Destructive Ops Over Unrecoverable State Agent runs destructive git/DB ops with uncommitted work or no rollback path (Replit prod-DB deletion 2025-07; Dicklesworthstone git checkout -- data-loss incident 2025-12, recovered via git fsck; claude-code#40968 worktree CWD wedge; Koukyosyumei worktree filesystem breach 2026-06) Checkpoint gate — clean committed/stashed state before broad/destructive units; Process Framework destructive-action row
W23 Unmeasured Optimization Agent performance changes skip measurement and regress (Anthropic 2026-04 postmortem — latency optimization degraded model behavior past all review layers; 2510.15494 — DeepSeek Netty fix at 0.34× baseline vs human 1.21×; Purdue 2512.21757 — agent perf PRs validate 45.7% vs human 63.6%, benchmarks 25% vs 49%) Measure-Before-Optimize — baseline + same measurement after; no measured improvement → revert
W24 Docs Drift Code changes leave README/API docs/examples stale; agents then treat stale docs as ground truth (Columbia DAPLab 2026-03; OpenAI community deep-init report 2026-02 — "the model is following instructions — they're just stale"; Mintlify-cited 2025 survey: 67% of API docs stale within 30 days) Completion Gate — behavior/interface changed → affected docs and examples updated in the same task

Each rule in rules.md addresses one or more weaknesses. New rules: explicitly identify weaknesses mitigated.

Exception — process-management rules (customer 2, CLAUDE.md › Two Customers) serve the user's follow-up cost rather than a W-ID: Decision Framing, Question Batching, the Assumed: list in Outcome Report, and the human-owned-blocker row (added 2026-07, maintainer first-hand evidence: repeated bare "X or Y?" choices with no meaning/consequence/recommendation; serial question drip; silent defaults; blockers surfacing only at task end).

Overlap Design Decision

Rules (always loaded) + skills (on-demand) intentionally overlap. Both must be self-contained — skills can't depend on rules being loaded, vice versa. Benign reinforcement costs slightly more tokens but prevents gaps in protection.

Rule Evaluation Rubric

Score each rule 0-3 on criteria:

Criterion 3 (Excellent) 2 (Good) 1 (Needs Work) 0 (Missing)
Clarity Action unambiguous, gate condition explicit Mostly clear, minor ambiguity Vague action or missing gate Unclear what to do
Universality Applies to any language/framework/tool Applies to most with minor exceptions Language-specific Single-tool only
Positive Framing Primary instruction is "do X", negative only as reinforcement Mix of positive + negative Primarily negative Only "don't"
Example Coverage 2-3 examples: happy + edge + error 1 example: happy path No examples but clear rule Abstract, no examples
Token Efficiency Table or single-line rule Short paragraph Multiple paragraphs Excessive prose
Adaptive Thinking No forced CoT; uses gates for verification Minimal forced reasoning Requires unnecessary step-by-step Forces full chain-of-thought

Target: >= 15/18 for production rules. >= 10/18 for draft rules.

Adding New Rules

The pre-check list (evidence, framing, budget, overlap, weakness mapping, rubric score) lives in CLAUDE.md › "Adding or Modifying Rules" — single source of truth. This file supplies the research behind those checks and the rubric above.