diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..a8ecd3c --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,28 @@ +name: tests + +on: + push: + pull_request: + +jobs: + unit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install package + run: python -m pip install --upgrade pip && python -m pip install -e . + - name: Unit tests + run: python -m unittest discover -s tests -v + env: + PYTHONPATH: src + - name: Fixture corpus + run: python scripts/run_eval.py + - name: Scripted orchestrator + run: python scripts/run_orchestration.py + - name: Stub MCP golden + run: python scripts/run_harness.py --backend stub --profile golden --fast-ms 1 --strong-ms 1 + - name: Stub MCP + apply gate + run: python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1 diff --git a/README.md b/README.md index 7c8660f..27d301e 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,8 @@ does bounded, mechanical generation on a private GPU host running scoring (transport / format / structure / behavior) and the committed fixture corpus. Cloud Agents can run the fixtures; live Ollama stays on the workstation. +- **[docs/cloud-orchestrator-results-2026-09-07.md](docs/cloud-orchestrator-results-2026-09-07.md)** + — dated Cloud Agent run of the stub corpus and apply gate (no GPU). - **[docs/security-scan-results-2026-09-06.md](docs/security-scan-results-2026-09-06.md)** — dated Gitleaks, GitHub alert, tracked-tree, and deployment-safety results. - **[examples/](examples/)** — public-safe client config templates. Copy them @@ -77,8 +79,12 @@ connected through SSH local forwarding. .venv/bin/python scripts/run_eval.py .venv/bin/python scripts/run_harness.py --backend stub --profile observed --out eval-runs/observed .venv/bin/python scripts/run_orchestration.py + .venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate ``` + GitHub Actions (`.github/workflows/tests.yml`) runs the same no-GPU path + on push and pull request. + 6. Deployment safety checks (no GPU required; inspects this host only): ```bash diff --git a/docs/cloud-orchestrator-results-2026-09-07.md b/docs/cloud-orchestrator-results-2026-09-07.md new file mode 100644 index 0000000..9d9a908 --- /dev/null +++ b/docs/cloud-orchestrator-results-2026-09-07.md @@ -0,0 +1,69 @@ +# Cloud Agent orchestrator results — 2026-09-07 + +Dated note from a Cursor Cloud Agent. This VM has **no GPU and no Ollama**. +The local worker is the loopback stub. The premium role is this agent +inspecting layer results and the apply-gate outcomes. It is **not** a +desktop Cursor session against `qwen3.5:9b` / `devstral-small-2`. + +Commands (all passed): + +```bash +PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v +.venv/bin/python scripts/run_eval.py +.venv/bin/python scripts/run_orchestration.py +.venv/bin/python scripts/run_harness.py --backend stub --profile golden --fast-ms 1 --strong-ms 1 +.venv/bin/python scripts/run_harness.py --backend stub --profile observed --fast-ms 1 --strong-ms 1 +.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1 +``` + +## Unit suite + +**83 tests passed.** Includes fixture scoring, A6 behavior (shape-only +`def test` fails), local fast→strong policy, and MCP + apply-gate +integration. + +## Fixture corpus + +24 fixtures: every golden passes all four layers; known-fail fixtures +stop at the intended layer (format / structure / behavior / transport). + +New executable cases exercised here: `move_function_imports`, +`extract_shared_parser`, `split_pipeline`, `implement_clamp`, +`explain_clamp`, `review_login`. + +## Stub MCP harness (not GPU time) + +Golden, 10 jobs, 10 attempts: **pass@1 = 1.00**, escalated = 0.00. +`mcp_ms` p50 ≈ 5 ms (loopback stub). + +Observed, 10 jobs, 21 attempts: **pass@1 = 0.00**, **pass@end = 1.00**, +escalated = 0.30. First failures: format 4, structure 5, behavior 2. +Repair success is pass@2 (or pass@3 for `whitespace_extract_vague`). + +## Premium review (this agent) + +I treated the scripted apply-gate outcomes as the verdicts I would give +on these **stub** patches: + +| Job | Route | My verdict | Why | +| --- | --- | --- | --- | +| keep_incident / architecture / live_tools / ambiguous | keep | keep on premium | Spec §8 do-not-delegate | +| mcp_extract_accept | mechanical | **accept** | Layers passed; helper is module-level | +| mcp_move_accept | mechanical | **accept** | `clamp` moved; `report.py` imports updated; oracles match | +| mcp_parser_accept | mechanical | **accept** | Shared `parse_fields`; email/qty oracles match | +| mcp_pipeline_accept | mechanical | **accept** | `run(" 1, 2, 3 ") == "6"` | +| mcp_tests_accept | mechanical | **accept** | Generated tests actually execute (A6) | +| mcp_explain_accept | mechanical | **accept** | Names clamp / lo / hi and clipping | +| mcp_code_rewrite | mechanical | **rewrite** | Local clamp is correct; I would still swap inverted bounds | +| mcp_review_notes_only | mechanical + security | **reject apply** | `local_review` flagged None/auth; that is notes, not a patch | +| mcp_reject_security | mechanical + security | **reject** | Do not apply until a human/premium pass on bounds | + +Closed loop: **13 jobs, 9 delegated, 11 applied, 2 held.** Keep jobs did +not call `local_*`. Rejected jobs had passing local layers and still did +not apply. + +## What this note is not + +- Live tok/s, pass@1 on the workstation tags, or GPU placement +- Proof that desktop Cursor's tool picker chose `local_refactor` +- A claim that stub `mcp_ms` is model latency diff --git a/docs/evaluation-protocol.md b/docs/evaluation-protocol.md index 0fcde04..c3d3add 100644 --- a/docs/evaluation-protocol.md +++ b/docs/evaluation-protocol.md @@ -57,6 +57,12 @@ The same scorer runs against: | `whitespace_extract_vague` | `local_refactor` | **Same checker**, vaguer prompt. Measures prompt-contract, not a looser oracle | | `multi_file_rename` | `local_refactor` | Two files must be returned; `add` → `plus`; `total([1,2,3]) == 6` | | `test_add_execute` | `local_generate_tests` | Generated tests are imported with `add.py` and the `test_*` functions are called | +| `move_function_imports` | `local_refactor` | Move `clamp` into `bounds.py` and update `report.py` imports | +| `extract_shared_parser` | `local_refactor` | Extract `parse_fields` into `csv_parse.py`; both callers import it | +| `split_pipeline` | `local_refactor` | Split `run` into `load.py` / `transform.py` / `pipeline.py` | +| `implement_clamp` | `local_code` | Implement `clamp` from a spec, no starter file | +| `explain_clamp` | `local_explain` | Prose: names the function and bounds; mentions clipping | +| `review_login` | `local_review` | Prose first-pass: flags None and missing auth. **Not** an apply | Known-fail fixtures are part of the corpus. They prove the scorer can tell layers apart: @@ -136,9 +142,15 @@ CI uses a scripted reviewer. These tests do **not** call Cursor, GPT, or Claude, and they do not prove that a live IDE agent followed the rule file. They prove the state machine the agent is supposed to follow. +The same gate runs after **real stdio MCP** calls when you pass +`--orchestrate`. Stub Ollama still supplies the worker text. Keep jobs +never call `local_*`. Accept / rewrite / reject then run on the scored +candidate. + ```bash -PYTHONPATH=src .venv/bin/python -m unittest tests.test_eval_orchestrate -v +PYTHONPATH=src .venv/bin/python -m unittest tests.test_eval_orchestrate tests.test_eval_mcp_orchestrate -v .venv/bin/python scripts/run_orchestration.py +.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate ``` ## Commands @@ -149,8 +161,11 @@ Fixture protocol (no GPU; this is what Cloud Agents can run): PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v .venv/bin/python scripts/run_eval.py .venv/bin/python scripts/run_orchestration.py +.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1 ``` +GitHub Actions (`.github/workflows/tests.yml`) runs the same no-GPU path on every push. + Live protocol (workstation with Ollama): ```bash @@ -174,7 +189,9 @@ After repeated live runs, a paper may claim: - That a vaguer prompt raises structure failures on the same oracle (`whitespace_extract` vs `whitespace_extract_vague`). - That shape-only test generation overstates success relative to - executed tests (`test_add_execute`). + executed tests (`test_add_execute`; A6 now uses this checker). +- That keep-vs-delegate and accept/rewrite/reject are enforceable as a + state machine on stub workers plus real stdio MCP. It still may not claim: diff --git a/docs/phase3-log.md b/docs/phase3-log.md index 085aa9c..d088f92 100644 --- a/docs/phase3-log.md +++ b/docs/phase3-log.md @@ -6,13 +6,16 @@ IPs, or secrets here. Reproducible commands, complete 2026-09-06 observations, and limitations: [local-acceptance-results-2026-09-06.md](local-acceptance-results-2026-09-06.md). Scoring method: [evaluation-protocol.md](evaluation-protocol.md). +Cloud-safe stub + apply-gate run: +[cloud-orchestrator-results-2026-09-07.md](cloud-orchestrator-results-2026-09-07.md). | Date | Tool | Model | Task | Result | Notes | | --- | --- | --- | --- | --- | --- | | 2026-09-04 | local_generate_tests | fast | unittest for `merge_dotenv` | accepted with trim | Dropped one truncated test; quoted-value case kept; applied as `tests/test_envfile.py` | -| 2026-09-06 | local_generate_tests | fast | live A6 tests for synthetic `add()` | accepted | Real Ollama via stdio MCP; output-shape acceptance passed | +| 2026-09-06 | local_generate_tests | fast | live A6 tests for synthetic `add()` | accepted | Real Ollama via stdio MCP; **shape-only** A6 at that date | | 2026-09-06 | local_refactor | fast | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 6.7s | | 2026-09-06 | local_refactor | strong | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 25.6s; model reported 46% CPU / 54% GPU at 16K context | +| 2026-09-07 | all local_* eval tools | stub | 10-case corpus + apply gate | accepted / rewritten / rejected per job | Cloud Agent; no GPU; A6 now executes tests | Columns: diff --git a/docs/roadmap.md b/docs/roadmap.md index fabbef9..f9cd403 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -10,7 +10,7 @@ LAN addresses, or SKUs here. | --- | --- | --- | | 1 — inference host | Done on the workstation lab | Ollama + starter tags on a private GPU | | 2 — MCP bridge | Done | `local-coding-slm` stdio tools; Cursor / Copilot / Claude adapters | -| 3 — measure | Protocol + fixture corpus + orchestrator contract; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet | +| 3 — measure | Protocol + expanded corpus + MCP apply gate + CI; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet | | T12 second NVIDIA host | Blocked on host power | WSL GPU via SSH; see [examples/downstairs-wsl-gpu.md](../examples/downstairs-wsl-gpu.md) | Finish Phase 3 measurement (layered live rates on the committed corpus) diff --git a/scripts/prove_acceptance.py b/scripts/prove_acceptance.py index 665c6e2..4e034ee 100644 --- a/scripts/prove_acceptance.py +++ b/scripts/prove_acceptance.py @@ -1,5 +1,4 @@ -#!/usr/bin/env python3 -"""Run A5/A6 against the local stdio MCP server. Loads repo-root .env.""" +"""A5/A6 against stdio MCP. A6 executes generated tests via the eval scorer.""" from __future__ import annotations @@ -13,6 +12,7 @@ if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) +from local_coding_slm.eval.acceptance import score_a6 # noqa: E402 from local_coding_slm.server import _load_dotenv # noqa: E402 _load_dotenv() @@ -20,8 +20,6 @@ from mcp import ClientSession, StdioServerParameters # noqa: E402 from mcp.client.stdio import stdio_client # noqa: E402 -TINY_FN = "def add(a: int, b: int) -> int:\n return a + b\n" - async def _run() -> int: python = sys.executable @@ -62,18 +60,18 @@ async def _run() -> int: print("FAIL A5 local_status") return 1 + from local_coding_slm.eval.cases import CASES_BY_ID + + case = CASES_BY_ID["test_add_execute"] tests = await session.call_tool( "local_generate_tests", { - "task": ( - "Write pytest unit tests for add(). Cover two positives " - "and one negative. Return a single fenced file." - ), - "files": [{"path": "add.py", "content": TINY_FN}], - "language": "python", - "style": "pytest", + "task": case.task, + "files": list(case.files), + "language": case.language, + "style": case.style, "model": "fast", - "max_tokens": 400, + "max_tokens": case.max_tokens, }, ) tests_text = "".join( @@ -81,9 +79,13 @@ async def _run() -> int: ) print("A6") print(tests_text) - lowered = tests_text.lower() - if tests_text.startswith("ERROR:") or "def test" not in lowered: - print("FAIL A6 local_generate_tests") + result = score_a6(tests_text) + for layer in result.layers: + print(f" {layer.status:4} {layer.name}: {layer.message}") + if not result.passed: + first = result.first_failure + extra = f" stop={first.name}" if first else "" + print(f"FAIL A6 local_generate_tests{extra}") return 1 print("PASS A5 A6") return 0 diff --git a/scripts/run_harness.py b/scripts/run_harness.py index 426b2f2..bd6d8a5 100644 --- a/scripts/run_harness.py +++ b/scripts/run_harness.py @@ -14,7 +14,12 @@ if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) -from local_coding_slm.eval.harness import format_summary, run_campaign # noqa: E402 +from local_coding_slm.eval.harness import ( # noqa: E402 + format_orchestrated, + format_summary, + run_campaign, + run_orchestrated_campaign, +) from local_coding_slm.eval.record import summarize, write_jsonl # noqa: E402 @@ -23,15 +28,54 @@ def main() -> None: parser.add_argument("--backend", choices=("stub", "live"), default="stub") parser.add_argument("--profile", choices=("golden", "observed"), default="golden") parser.add_argument("--case", action="append", dest="case_ids", default=None) + parser.add_argument("--job", action="append", dest="job_ids", default=None) parser.add_argument("--repeat", type=int, default=1) parser.add_argument("--fast-ms", type=float, default=8.0) parser.add_argument("--strong-ms", type=float, default=25.0) + parser.add_argument( + "--orchestrate", + action="store_true", + help="Route + MCP local loop + premium apply gate (scripted reviewer)", + ) parser.add_argument( "--out", default="", - help="Directory for attempts.jsonl and summary.json (gitignored eval-runs/)", + help="Directory for JSONL/JSON summaries (gitignored eval-runs/)", ) args = parser.parse_args() + if args.orchestrate: + results = asyncio.run( + run_orchestrated_campaign( + backend=args.backend, + profile=args.profile, + job_ids=args.job_ids, + fast_ms=args.fast_ms, + strong_ms=args.strong_ms, + ) + ) + print(format_orchestrated(results)) + if args.out: + dest = Path(args.out) + dest.mkdir(parents=True, exist_ok=True) + payload = [ + { + "job": item.job, + "delegated": item.delegated, + "route": item.route_reason, + "outcome": item.outcome, + "applied": item.applied, + "source": item.apply_source, + "models": list(item.local_models), + "review": item.review_decision, + } + for item in results + ] + (dest / "orchestrated.json").write_text( + json.dumps(payload, indent=2) + "\n", + encoding="utf-8", + ) + print(f"wrote {dest / 'orchestrated.json'}") + raise SystemExit(0 if results else 1) rows = asyncio.run( run_campaign( backend=args.backend, diff --git a/spec.md b/spec.md index b5de0c2..ed3fbbc 100644 --- a/spec.md +++ b/spec.md @@ -721,9 +721,10 @@ retry as pass-at-one. The local harness (`scripts/run_harness.py`) measures fast→strong failover after a task is already delegated. Routing, keep-vs-delegate, and the premium accept/rewrite/reject apply gate are the orchestrator tests -(`tests/test_eval_orchestrate.py`, `scripts/run_orchestration.py`). Those -use a scripted stand-in for the main LLM; they do not call Cursor or -another hosted API. +(`tests/test_eval_orchestrate.py`, `scripts/run_orchestration.py`). +`--orchestrate` runs that gate after real stdio MCP (stub or live worker). +CI uses a scripted stand-in for the main LLM; it does not call Cursor or +another hosted API. GitHub Actions runs the no-GPU path on every push. Only after that, consider automatic task classification. @@ -760,7 +761,7 @@ Run from the workstation with `OLLAMA_BASE_URL` set. | A3 | Chat prompt to the strong model | Completes; GPU or GPU+RAM is acceptable | | A4 | Workstation reaches Ollama through SSH; an unauthorized LAN client cannot reach 11434 | No direct LAN exposure; skip if same-machine | | A5 | `local_status` MCP tool | Reports both models and the configured base URL host *without* requiring that URL in git | -| A6 | `local_generate_tests` with one small function | Returns a test file / diff the premium agent can apply | +| A6 | `local_generate_tests` with `test_add_execute` | Four eval layers pass, including executed `test_*` functions. A `def test` substring is not enough | | A7 | Cursor Agent | Premium model calls a `local_*` tool on a mechanical prompt | | A8 | Copilot Agent (VS Code) | Same tool appears and runs | | A9 | Claude Code local | `claude mcp list` shows `local-coding-slm` connected | diff --git a/src/local_coding_slm/eval/acceptance.py b/src/local_coding_slm/eval/acceptance.py new file mode 100644 index 0000000..4a2ff23 --- /dev/null +++ b/src/local_coding_slm/eval/acceptance.py @@ -0,0 +1,8 @@ +"""A6 acceptance uses the behavior-layer scorer, not a `def test` substring.""" + +from local_coding_slm.eval.cases import CASES_BY_ID +from local_coding_slm.eval.score import EvalResult, score_candidate + + +def score_a6(text: str) -> EvalResult: + return score_candidate(text, CASES_BY_ID["test_add_execute"]) diff --git a/src/local_coding_slm/eval/cases.py b/src/local_coding_slm/eval/cases.py index 7f27841..6510a82 100644 --- a/src/local_coding_slm/eval/cases.py +++ b/src/local_coding_slm/eval/cases.py @@ -9,6 +9,17 @@ import ast from dataclasses import dataclass +from local_coding_slm.eval.cases_extended import ( + EXTENDED_CASES, + EXTENDED_GOLDEN, + EXTRACT_PARSER_NESTED, + EXTRACT_PARSER_NO_STRIP, + IMPLEMENT_CLAMP_NO_HI, + MOVE_FUNCTION_PARTIAL, + SPLIT_PIPELINE_MONOLITH, + EXPLAIN_CLAMP_VAGUE, + REVIEW_LOGIN_LGTM, +) from local_coding_slm.eval.score import BehaviorCheck, EvalCase WHITESPACE_SOURCE = """\ @@ -307,6 +318,8 @@ def _run_generated_tests(merged: dict[str, str]) -> None: ), ) +SEED_CASE_IDS: tuple[str, ...] = tuple(case.id for case in CASES) +CASES = CASES + EXTENDED_CASES CASES_BY_ID = {case.id: case for case in CASES} GOLDEN_FOR_CASE = { @@ -314,6 +327,7 @@ def _run_generated_tests(merged: dict[str, str]) -> None: "whitespace_extract_vague": WHITESPACE_GOLDEN, "multi_file_rename": MULTI_FILE_GOLDEN, "test_add_execute": TEST_ADD_GOLDEN, + **EXTENDED_GOLDEN, } @@ -385,4 +399,59 @@ class Fixture: False, "behavior", ), + Fixture("move_function_golden", "move_function_imports", GOLDEN_FOR_CASE["move_function_imports"], True), + Fixture( + "move_function_partial", + "move_function_imports", + MOVE_FUNCTION_PARTIAL, + False, + "format", + ), + Fixture("extract_parser_golden", "extract_shared_parser", GOLDEN_FOR_CASE["extract_shared_parser"], True), + Fixture( + "extract_parser_nested", + "extract_shared_parser", + EXTRACT_PARSER_NESTED, + False, + "structure", + ), + Fixture( + "extract_parser_no_strip", + "extract_shared_parser", + EXTRACT_PARSER_NO_STRIP, + False, + "behavior", + ), + Fixture("split_pipeline_golden", "split_pipeline", GOLDEN_FOR_CASE["split_pipeline"], True), + Fixture( + "split_pipeline_monolith", + "split_pipeline", + SPLIT_PIPELINE_MONOLITH, + False, + "format", + ), + Fixture("implement_clamp_golden", "implement_clamp", GOLDEN_FOR_CASE["implement_clamp"], True), + Fixture( + "implement_clamp_no_hi", + "implement_clamp", + IMPLEMENT_CLAMP_NO_HI, + False, + "behavior", + ), + Fixture("explain_clamp_golden", "explain_clamp", GOLDEN_FOR_CASE["explain_clamp"], True), + Fixture( + "explain_clamp_vague", + "explain_clamp", + EXPLAIN_CLAMP_VAGUE, + False, + "structure", + ), + Fixture("review_login_golden", "review_login", GOLDEN_FOR_CASE["review_login"], True), + Fixture( + "review_login_lgtm", + "review_login", + REVIEW_LOGIN_LGTM, + False, + "structure", + ), ) diff --git a/src/local_coding_slm/eval/cases_extended.py b/src/local_coding_slm/eval/cases_extended.py new file mode 100644 index 0000000..37390a4 --- /dev/null +++ b/src/local_coding_slm/eval/cases_extended.py @@ -0,0 +1,544 @@ +"""Non-seed corpus: multi-file refactors plus local_code / explain / review. + +These cases stay bounded. They are not whole-repo rewrites. Golden strings are +fixtures for the scorer and stub Ollama, not live model dumps. +""" + +from __future__ import annotations + +import ast + +from local_coding_slm.eval.score import BehaviorCheck, EvalCase + +# --- move clamp into bounds.py and fix imports --- + +MATH_OPS_SOURCE = """\ +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + if value > hi: + return hi + return value + + +def mean(xs: list[int]) -> float: + return sum(xs) / len(xs) if xs else 0.0 +""" + +REPORT_SOURCE = """\ +from math_ops import clamp, mean + + +def summarize(xs: list[int], lo: int, hi: int) -> float: + capped = [clamp(x, lo, hi) for x in xs] + return mean(capped) +""" + +MOVE_FUNCTION_TASK = ( + "Move clamp from math_ops.py into a new module bounds.py. " + "Keep mean in math_ops.py. Update report.py so it imports clamp from " + "bounds and mean from math_ops. Preserve summarize() behavior. " + "Return three fenced Python files with path comments bounds.py, " + "math_ops.py, and report.py, no prose." +) + +MOVE_FUNCTION_GOLDEN = '''\ +```python +# bounds.py +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + if value > hi: + return hi + return value +``` + +```python +# math_ops.py +def mean(xs: list[int]) -> float: + return sum(xs) / len(xs) if xs else 0.0 +``` + +```python +# report.py +from bounds import clamp +from math_ops import mean + + +def summarize(xs: list[int], lo: int, hi: int) -> float: + capped = [clamp(x, lo, hi) for x in xs] + return mean(capped) +``` +''' + +MOVE_FUNCTION_PARTIAL = '''\ +```python +# bounds.py +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + if value > hi: + return hi + return value +``` + +```python +# math_ops.py +def mean(xs: list[int]) -> float: + return sum(xs) / len(xs) if xs else 0.0 +``` +''' + + +def _top_level_names(tree: ast.AST) -> set[str]: + return ( + {node.name for node in tree.body if isinstance(node, ast.FunctionDef)} + if isinstance(tree, ast.Module) + else set() + ) + + +def _clamp_must_move(by_path: dict[str, str]) -> str | None: + try: + bounds = ast.parse(by_path.get("bounds.py", "")) + math_ops = ast.parse(by_path.get("math_ops.py", "")) + except SyntaxError as exc: + return f"unparseable Python: {exc.msg}" + if "clamp" not in _top_level_names(bounds): + return "bounds.py is missing module-level clamp" + if "clamp" in _top_level_names(math_ops): + return "clamp is still defined in math_ops.py" + if "mean" not in _top_level_names(math_ops): + return "math_ops.py is missing mean" + report = by_path.get("report.py", "") + if "from bounds import clamp" not in report: + return "report.py must import clamp from bounds" + if "from math_ops import clamp" in report: + return "report.py still imports clamp from math_ops" + return None + + +# --- extract shared CSV field parser --- + +CSV_USERS_SOURCE = """\ +def parse_user(line: str) -> tuple[str, str]: + parts = [p.strip() for p in line.split(",")] + return parts[0], parts[1] + + +def user_email(line: str) -> str: + _, email = parse_user(line) + return email.lower() +""" + +CSV_ORDERS_SOURCE = """\ +def parse_order(line: str) -> tuple[str, int]: + parts = [p.strip() for p in line.split(",")] + return parts[0], int(parts[1]) + + +def order_qty(line: str) -> int: + _, qty = parse_order(line) + return qty +""" + +EXTRACT_PARSER_TASK = ( + "Extract the repeated comma-split-and-strip into a module-level function " + "parse_fields(line) -> list[str] in a new csv_parse.py. " + "csv_users.py and csv_orders.py must import parse_fields and keep " + "parse_user, user_email, parse_order, and order_qty. Preserve behavior. " + "Return three fenced Python files csv_parse.py, csv_users.py, csv_orders.py." +) + +EXTRACT_PARSER_GOLDEN = '''\ +```python +# csv_parse.py +def parse_fields(line: str) -> list[str]: + return [p.strip() for p in line.split(",")] +``` + +```python +# csv_users.py +from csv_parse import parse_fields + + +def parse_user(line: str) -> tuple[str, str]: + parts = parse_fields(line) + return parts[0], parts[1] + + +def user_email(line: str) -> str: + _, email = parse_user(line) + return email.lower() +``` + +```python +# csv_orders.py +from csv_parse import parse_fields + + +def parse_order(line: str) -> tuple[str, int]: + parts = parse_fields(line) + return parts[0], int(parts[1]) + + +def order_qty(line: str) -> int: + _, qty = parse_order(line) + return qty +``` +''' + +EXTRACT_PARSER_NESTED = '''\ +```python +# csv_parse.py +def unused() -> None: + return None +``` + +```python +# csv_users.py +def parse_user(line: str) -> tuple[str, str]: + def parse_fields(line: str) -> list[str]: + return [p.strip() for p in line.split(",")] + + parts = parse_fields(line) + return parts[0], parts[1] + + +def user_email(line: str) -> str: + _, email = parse_user(line) + return email.lower() +``` + +```python +# csv_orders.py +from csv_parse import parse_fields + + +def parse_order(line: str) -> tuple[str, int]: + parts = parse_fields(line) + return parts[0], int(parts[1]) + + +def order_qty(line: str) -> int: + _, qty = parse_order(line) + return qty +``` +''' + +EXTRACT_PARSER_NO_STRIP = '''\ +```python +# csv_parse.py +def parse_fields(line: str) -> list[str]: + return line.split(",") +``` + +```python +# csv_users.py +from csv_parse import parse_fields + + +def parse_user(line: str) -> tuple[str, str]: + parts = parse_fields(line) + return parts[0], parts[1] + + +def user_email(line: str) -> str: + _, email = parse_user(line) + return email.lower() +``` + +```python +# csv_orders.py +from csv_parse import parse_fields + + +def parse_order(line: str) -> tuple[str, int]: + parts = parse_fields(line) + return parts[0], int(parts[1]) + + +def order_qty(line: str) -> int: + _, qty = parse_order(line) + return qty +``` +''' + + +def _parser_is_module_level(by_path: dict[str, str]) -> str | None: + try: + tree = ast.parse(by_path.get("csv_parse.py", "")) + except SyntaxError as exc: + return f"csv_parse.py is not parseable: {exc.msg}" + if "parse_fields" not in _top_level_names(tree): + return "csv_parse.py is missing module-level parse_fields" + users = by_path.get("csv_users.py", "") + orders = by_path.get("csv_orders.py", "") + if "from csv_parse import parse_fields" not in users: + return "csv_users.py must import parse_fields from csv_parse" + if "from csv_parse import parse_fields" not in orders: + return "csv_orders.py must import parse_fields from csv_parse" + return None + + +# --- split a one-file pipeline --- + +PIPELINE_SOURCE = """\ +def run(raw: str) -> str: + parts = [p.strip() for p in raw.split(",") if p.strip()] + nums = [int(p) for p in parts] + return str(sum(nums)) +""" + +SPLIT_PIPELINE_TASK = ( + "Split pipeline.py into three modules: load.py with parse_ints(raw) -> " + "list[int], transform.py with total(nums) -> int, and pipeline.py with " + "run(raw) -> str that calls both. Preserve run(' 1, 2, 3 ') == '6'. " + "Return fenced files load.py, transform.py, and pipeline.py, no prose." +) + +SPLIT_PIPELINE_GOLDEN = '''\ +```python +# load.py +def parse_ints(raw: str) -> list[int]: + parts = [p.strip() for p in raw.split(",") if p.strip()] + return [int(p) for p in parts] +``` + +```python +# transform.py +def total(nums: list[int]) -> int: + return sum(nums) +``` + +```python +# pipeline.py +from load import parse_ints +from transform import total + + +def run(raw: str) -> str: + return str(total(parse_ints(raw))) +``` +''' + +SPLIT_PIPELINE_MONOLITH = '''\ +```python +# pipeline.py +def run(raw: str) -> str: + parts = [p.strip() for p in raw.split(",") if p.strip()] + nums = [int(p) for p in parts] + return str(sum(nums)) +``` +''' + + +def _pipeline_split(by_path: dict[str, str]) -> str | None: + missing = [path for path in ("load.py", "transform.py", "pipeline.py") if path not in by_path] + if missing: + return "missing " + ", ".join(missing) + try: + load = ast.parse(by_path["load.py"]) + transform = ast.parse(by_path["transform.py"]) + pipe = ast.parse(by_path["pipeline.py"]) + except SyntaxError as exc: + return f"unparseable Python: {exc.msg}" + if "parse_ints" not in _top_level_names(load): + return "load.py is missing parse_ints" + if "total" not in _top_level_names(transform): + return "transform.py is missing total" + if "run" not in _top_level_names(pipe): + return "pipeline.py is missing run" + return None + + +# --- local_code: implement clamp from a spec --- + +IMPLEMENT_CLAMP_TASK = ( + "Write a Python function clamp(value, lo, hi) in clamp.py. " + "It must return lo when value is below lo, hi when value is above hi, " + "and value otherwise. Return one fenced file with a path comment." +) + +IMPLEMENT_CLAMP_GOLDEN = '''\ +```python +# clamp.py +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + if value > hi: + return hi + return value +``` +''' + +IMPLEMENT_CLAMP_NO_HI = '''\ +```python +# clamp.py +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + return value +``` +''' + + +# --- local_explain (prose) --- + +EXPLAIN_CLAMP_SOURCE = """\ +def clamp(value: int, lo: int, hi: int) -> int: + if value < lo: + return lo + if value > hi: + return hi + return value +""" + +EXPLAIN_CLAMP_TASK = ( + "Explain clamp() in one short paragraph. Name the function, the lo and hi " + "bounds, and that values outside the range are clipped. Do not rewrite the code." +) + +EXPLAIN_CLAMP_GOLDEN = ( + "clamp(value, lo, hi) returns lo when value is below the lower bound, " + "hi when value is above the upper bound, and value when it already lies " + "inside the range. Out-of-range inputs are clipped." +) + +EXPLAIN_CLAMP_VAGUE = "This helper is useful in several places." + + +def _explain_mentions_clip(merged: dict[str, str]) -> None: + text = merged["_prose"].lower() + if "clip" not in text and "bound" not in text and "range" not in text: + raise AssertionError("explanation never says values are clipped or bounded") + + +# --- local_review (prose, first pass only) --- + +REVIEW_LOGIN_SOURCE = """\ +def login(user, password): + return user.name + password +""" + +REVIEW_LOGIN_TASK = ( + "First-pass review of login(). Flag obvious null/None use of user and the " + "missing password/auth check. Do not rewrite the function." +) + +REVIEW_LOGIN_GOLDEN = ( + "login() dereferences user.name with no None check, so a missing user " + "raises. password is concatenated rather than verified, so there is no " + "auth check. Add a null guard and a real password test before returning." +) + +REVIEW_LOGIN_LGTM = "Looks good to me. No issues." + + +def _review_flags_auth(merged: dict[str, str]) -> None: + text = merged["_prose"].lower() + if "password" not in text and "auth" not in text: + raise AssertionError("review did not flag password/auth") + if "none" not in text and "null" not in text: + raise AssertionError("review did not flag None/null use of user") + + +EXTENDED_CASES: tuple[EvalCase, ...] = ( + EvalCase( + id="move_function_imports", + tool="local_refactor", + task=MOVE_FUNCTION_TASK, + files=( + {"path": "math_ops.py", "content": MATH_OPS_SOURCE}, + {"path": "report.py", "content": REPORT_SOURCE}, + ), + required_paths=("bounds.py", "math_ops.py", "report.py"), + required_top_level=("clamp", "mean", "summarize"), + extra_structure=_clamp_must_move, + behavior=( + BehaviorCheck("report", "summarize", ([1, 8, 3], 0, 5), 3.0), + BehaviorCheck("report", "summarize", ([], 0, 5), 0.0), + ), + max_tokens=1200, + ), + EvalCase( + id="extract_shared_parser", + tool="local_refactor", + task=EXTRACT_PARSER_TASK, + files=( + {"path": "csv_users.py", "content": CSV_USERS_SOURCE}, + {"path": "csv_orders.py", "content": CSV_ORDERS_SOURCE}, + ), + required_paths=("csv_parse.py", "csv_users.py", "csv_orders.py"), + required_top_level=("parse_fields", "parse_user", "user_email", "parse_order", "order_qty"), + extra_structure=_parser_is_module_level, + behavior=( + BehaviorCheck("csv_users", "user_email", (" Ada , ADA@EX.com ",), "ada@ex.com"), + BehaviorCheck("csv_orders", "order_qty", (" widget , 3 ",), 3), + ), + max_tokens=1200, + ), + EvalCase( + id="split_pipeline", + tool="local_refactor", + task=SPLIT_PIPELINE_TASK, + files=({"path": "pipeline.py", "content": PIPELINE_SOURCE},), + required_paths=("load.py", "transform.py", "pipeline.py"), + required_top_level=("parse_ints", "total", "run"), + extra_structure=_pipeline_split, + behavior=( + BehaviorCheck("pipeline", "run", (" 1, 2, 3 ",), "6"), + BehaviorCheck("pipeline", "run", ("",), "0"), + ), + max_tokens=1200, + ), + EvalCase( + id="implement_clamp", + tool="local_code", + task=IMPLEMENT_CLAMP_TASK, + files=(), + required_paths=("clamp.py",), + required_top_level=("clamp",), + behavior=( + BehaviorCheck("clamp", "clamp", (3, 0, 5), 3), + BehaviorCheck("clamp", "clamp", (-2, 0, 5), 0), + BehaviorCheck("clamp", "clamp", (9, 0, 5), 5), + ), + ), + EvalCase( + id="explain_clamp", + tool="local_explain", + task=EXPLAIN_CLAMP_TASK, + files=({"path": "clamp.py", "content": EXPLAIN_CLAMP_SOURCE},), + expect_fences=False, + required_phrases=("clamp", "lo", "hi"), + behavior_fn=_explain_mentions_clip, + ), + EvalCase( + id="review_login", + tool="local_review", + task=REVIEW_LOGIN_TASK, + files=({"path": "auth.py", "content": REVIEW_LOGIN_SOURCE},), + expect_fences=False, + required_phrases=("none",), + behavior_fn=_review_flags_auth, + ), +) + +EXTENDED_GOLDEN = { + "move_function_imports": MOVE_FUNCTION_GOLDEN, + "extract_shared_parser": EXTRACT_PARSER_GOLDEN, + "split_pipeline": SPLIT_PIPELINE_GOLDEN, + "implement_clamp": IMPLEMENT_CLAMP_GOLDEN, + "explain_clamp": EXPLAIN_CLAMP_GOLDEN, + "review_login": REVIEW_LOGIN_GOLDEN, +} + +OBSERVED_FIRST = { + "move_function_imports": MOVE_FUNCTION_PARTIAL, + "extract_shared_parser": EXTRACT_PARSER_NESTED, + "split_pipeline": SPLIT_PIPELINE_MONOLITH, + "implement_clamp": IMPLEMENT_CLAMP_NO_HI, + "explain_clamp": EXPLAIN_CLAMP_VAGUE, + "review_login": REVIEW_LOGIN_LGTM, +} diff --git a/src/local_coding_slm/eval/harness.py b/src/local_coding_slm/eval/harness.py index 0fc9417..21072bf 100644 --- a/src/local_coding_slm/eval/harness.py +++ b/src/local_coding_slm/eval/harness.py @@ -1,4 +1,4 @@ -"""Closed-loop measurement harness: MCP tool call → score → retry/escalate.""" +"""Closed-loop measurement: MCP tool call → score → retry/escalate → apply gate.""" from __future__ import annotations @@ -8,6 +8,14 @@ from pathlib import Path from local_coding_slm.eval.cases import CASES, EvalCase +from local_coding_slm.eval.jobs import MCP_JOBS +from local_coding_slm.eval.orchestrate import ( + JobResult, + LocalAttempt, + OrchestrationJob, + finish_delegated_job, + run_job, +) from local_coding_slm.eval.policy import AttemptPlan, next_plan from local_coding_slm.eval.record import AttemptRecord, summarize from local_coding_slm.eval.score import score_candidate @@ -26,22 +34,98 @@ async def run_campaign( base_url: str | None = None, ) -> list[AttemptRecord]: """Run the corpus through stdio MCP. ``backend=stub`` starts loopback Ollama.""" - from mcp import ClientSession, StdioServerParameters - from mcp.client.stdio import stdio_client + rows: list[AttemptRecord] = [] + + async def _collect(session: object, backend_name: str, profile_name: str) -> None: + selected = [ + case + for case in CASES + if case_ids is None or case.id in case_ids + ] + if not selected: + raise ValueError("no cases selected") + if repeat < 1: + raise ValueError("repeat must be >= 1") + for repeat_i in range(repeat): + for case in selected: + job = f"{case.id}#{repeat_i + 1}" + attempts = await _run_local_mcp( + session, + case, + job=job, + backend=backend_name, + profile=profile_name, + ) + rows.extend(item.record for item in attempts) + await _with_session( + backend=backend, + profile=profile, + fast_ms=fast_ms, + strong_ms=strong_ms, + base_url=base_url, + body=_collect, + ) + return rows + + +async def run_orchestrated_campaign( + *, + backend: str, + profile: str, + job_ids: list[str] | None = None, + fast_ms: float = 8.0, + strong_ms: float = 25.0, + base_url: str | None = None, +) -> list[JobResult]: + """Route → MCP local loop → premium apply gate. Stub reviewer, real MCP.""" + results: list[JobResult] = [] selected = [ - case - for case in CASES - if case_ids is None or case.id in case_ids + job + for job in MCP_JOBS + if job_ids is None or job.id in job_ids ] if not selected: - raise ValueError("no cases selected") - if repeat < 1: - raise ValueError("repeat must be >= 1") + raise ValueError("no orchestration jobs selected") + + async def _collect(session: object, backend_name: str, profile_name: str) -> None: + for job in selected: + results.append( + await _run_orchestrated_job( + session, + job, + backend=backend_name, + profile=profile_name, + ) + ) + + await _with_session( + backend=backend, + profile=profile, + fast_ms=fast_ms, + strong_ms=strong_ms, + base_url=base_url, + body=_collect, + ) + return results + + +async def _with_session( + *, + backend: str, + profile: str, + fast_ms: float, + strong_ms: float, + base_url: str | None, + body, +) -> None: + from mcp import ClientSession, StdioServerParameters + from mcp.client.stdio import stdio_client stub_cm = None env = os.environ.copy() env["PYTHONPATH"] = str(ROOT / "src") + os.pathsep + env.get("PYTHONPATH", "") + profile_name = profile if backend == "stub": from local_coding_slm.eval.stub_ollama import StubOllama @@ -51,7 +135,7 @@ async def run_campaign( elif backend == "live": if base_url: env["OLLAMA_BASE_URL"] = base_url - profile = "live" + profile_name = "live" else: raise ValueError("backend must be 'stub' or 'live'") @@ -62,46 +146,59 @@ async def run_campaign( env=env, cwd=str(ROOT), ) - rows: list[AttemptRecord] = [] try: async with stdio_client(params) as (read, write): async with ClientSession(read, write) as session: await session.initialize() - for repeat_i in range(repeat): - for case in selected: - job = f"{case.id}#{repeat_i + 1}" - rows.extend( - await _run_case( - session, - case, - job=job, - backend=backend, - profile=profile, - ) - ) + await body(session, backend, profile_name) finally: if stub_cm is not None: stub_cm.__exit__(None, None, None) - return rows -async def _run_case( +async def _run_orchestrated_job( + session: object, + job: OrchestrationJob, + *, + backend: str, + profile: str, +) -> JobResult: + from local_coding_slm.eval.routing import route + + decision = route(job.signals) + if decision.action == "keep": + return run_job(job) + if job.eval_case is None: + raise ValueError(f"{job.id}: delegated jobs need an eval_case") + attempts = await _run_local_mcp( + session, + job.eval_case, + job=job.id, + backend=backend, + profile=profile, + ) + return finish_delegated_job(job, attempts, job.review) + + +async def _run_local_mcp( session: object, case: EvalCase, *, job: str, backend: str, profile: str, -) -> list[AttemptRecord]: - rows: list[AttemptRecord] = [] +) -> list[LocalAttempt]: + attempts: list[LocalAttempt] = [] + records: list[AttemptRecord] = [] while True: - plan = next_plan(case, rows) + plan = next_plan(case, records) if plan is None: - return rows - record = await _one_attempt( - session, case, plan, backend, profile, job, len(rows) + 1 + return attempts + item = await _one_attempt( + session, case, plan, backend, profile, job, len(records) + 1 ) - rows.append(record) + attempts.append(item) + records.append(item.record) async def _one_attempt( @@ -112,14 +209,14 @@ async def _one_attempt( profile: str, job: str, attempt: int, -) -> AttemptRecord: +) -> LocalAttempt: task = case.task if not plan.suffix else f"{case.task}\n\n{plan.suffix}" payload = { "task": task, "files": list(case.files), "language": case.language, "model": plan.model, - "max_tokens": 700, + "max_tokens": case.max_tokens, } if case.style: payload["style"] = case.style @@ -133,7 +230,7 @@ async def _one_attempt( scored = score_candidate(text, case) score_ms = (time.perf_counter() - t1) * 1000.0 fail = scored.first_failure - return AttemptRecord( + record = AttemptRecord( job=job, case_id=case.id, attempt=attempt, @@ -149,6 +246,7 @@ async def _one_attempt( profile=profile, layers={item.name: item.status for item in scored.layers}, ) + return LocalAttempt(record=record, text=text, scored=scored) def format_summary(rows: list[AttemptRecord]) -> str: @@ -170,3 +268,18 @@ def format_summary(rows: list[AttemptRecord]) -> str: f"ms={item['elapsed_ms']:.1f}" ) return "\n".join(lines) + + +def format_orchestrated(results: list[JobResult]) -> str: + lines = [ + f"jobs={len(results)} applied={sum(1 for item in results if item.applied)} " + f"delegated={sum(1 for item in results if item.delegated)}" + ] + for item in results: + models = list(item.local_models) or "-" + lines.append( + f" {item.job}: delegated={item.delegated} route={item.route_reason} " + f"outcome={item.outcome} source={item.apply_source or '-'} " + f"models={models}" + ) + return "\n".join(lines) diff --git a/src/local_coding_slm/eval/jobs.py b/src/local_coding_slm/eval/jobs.py new file mode 100644 index 0000000..5a29c77 --- /dev/null +++ b/src/local_coding_slm/eval/jobs.py @@ -0,0 +1,106 @@ +"""Scripted premium jobs for the MCP + apply-gate loop.""" + +from __future__ import annotations + +from local_coding_slm.eval.cases import CASES_BY_ID +from local_coding_slm.eval.orchestrate import OrchestrationJob +from local_coding_slm.eval.review import accept, reject, rewrite +from local_coding_slm.eval.routing import RouteSignals, mechanical_signals + +REWRITE_CLAMP = '''\ +```python +# clamp.py +def clamp(value: int, lo: int, hi: int) -> int: + if lo > hi: + lo, hi = hi, lo + if value < lo: + return lo + if value > hi: + return hi + return value +``` +''' + +MCP_JOBS: tuple[OrchestrationJob, ...] = ( + OrchestrationJob( + id="keep_incident", + signals=mechanical_signals(incident_debug=True), + premium_keep_text="premium incident patch", + ), + OrchestrationJob( + id="keep_architecture", + signals=mechanical_signals(architectural=True), + premium_keep_text="premium architecture patch", + ), + OrchestrationJob( + id="keep_live_tools", + signals=mechanical_signals(needs_live_tools=True), + premium_keep_text="premium uses repo search", + ), + OrchestrationJob( + id="keep_ambiguous", + signals=RouteSignals(), + premium_keep_text="premium clarifying questions", + ), + OrchestrationJob( + id="mcp_extract_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["whitespace_extract"], + review=accept(notes="layers passed; apply local extract"), + ), + OrchestrationJob( + id="mcp_move_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["move_function_imports"], + review=accept(notes="imports updated"), + ), + OrchestrationJob( + id="mcp_parser_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["extract_shared_parser"], + review=accept(), + ), + OrchestrationJob( + id="mcp_pipeline_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["split_pipeline"], + review=accept(), + ), + OrchestrationJob( + id="mcp_tests_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["test_add_execute"], + review=accept(), + ), + OrchestrationJob( + id="mcp_code_rewrite", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["implement_clamp"], + review=rewrite(REWRITE_CLAMP, notes="also swap lo/hi if inverted"), + ), + OrchestrationJob( + id="mcp_explain_accept", + signals=mechanical_signals(), + eval_case=CASES_BY_ID["explain_clamp"], + review=accept(notes="prose is accurate"), + ), + OrchestrationJob( + id="mcp_review_notes_only", + signals=mechanical_signals(security_sensitive=True), + eval_case=CASES_BY_ID["review_login"], + review=reject(notes="local_review notes are not an apply"), + ), + OrchestrationJob( + id="mcp_reject_security", + signals=mechanical_signals(security_sensitive=True), + eval_case=CASES_BY_ID["implement_clamp"], + review=reject(notes="do not apply until bounds are reviewed"), + ), +) + + +def job_by_id(job_id: str) -> OrchestrationJob: + for job in MCP_JOBS: + if job.id == job_id: + return job + raise KeyError(job_id) diff --git a/src/local_coding_slm/eval/orchestrate.py b/src/local_coding_slm/eval/orchestrate.py index 7aa4b7e..d623863 100644 --- a/src/local_coding_slm/eval/orchestrate.py +++ b/src/local_coding_slm/eval/orchestrate.py @@ -192,6 +192,20 @@ def run_local_loop( attempts.append(LocalAttempt(record=record, text=text, scored=scored)) +def finish_delegated_job( + job: OrchestrationJob, + attempts: list[LocalAttempt], + verdict: ReviewVerdict | None, +) -> JobResult: + """Apply gate after local attempts (scripted or MCP) already ran.""" + decision = route(job.signals) + if decision.action != "delegate": + raise ValueError(f"{job.id}: finish_delegated_job requires a delegated route") + last = attempts[-1] if attempts else None + apply = decide_apply(delegated=True, last=last, verdict=verdict) + return _result(job, decision, apply, attempts, verdict) + + def run_job( job: OrchestrationJob, *, diff --git a/src/local_coding_slm/eval/score.py b/src/local_coding_slm/eval/score.py index 1e8fbc3..ea064f7 100644 --- a/src/local_coding_slm/eval/score.py +++ b/src/local_coding_slm/eval/score.py @@ -11,6 +11,7 @@ import ast import importlib.util +import re import sys import tempfile from collections.abc import Callable, Sequence @@ -80,11 +81,27 @@ class EvalCase: behavior: tuple[BehaviorCheck, ...] = () extra_structure: Callable[[dict[str, str]], str | None] | None = None behavior_fn: Callable[[dict[str, str]], None] | None = None + expect_fences: bool = True + required_phrases: tuple[str, ...] = () + max_tokens: int = 700 def score_candidate(text: str, case: EvalCase) -> EvalResult: layers: list[LayerResult] = [] files: tuple[ExtractedFile, ...] = () + raw = (text or "").strip() + if raw.startswith("ERROR:"): + layers.extend( + [ + LayerResult("transport", "fail", raw.splitlines()[0][:240]), + LayerResult("format", "skip", "skipped after transport failure"), + LayerResult("structure", "skip", "skipped after transport failure"), + LayerResult("behavior", "skip", "skipped after transport failure"), + ] + ) + return EvalResult(case.id, tuple(layers), files) + if not case.expect_fences: + return _score_prose(raw, case) try: extracted = extract_files(text) @@ -196,6 +213,70 @@ def score_candidate(text: str, case: EvalCase) -> EvalResult: return EvalResult(case.id, tuple(layers), files) +def _score_prose(raw: str, case: EvalCase) -> EvalResult: + """Score local_explain / local_review style output. No fenced files required.""" + layers: list[LayerResult] = [ + LayerResult("transport", "pass", "candidate is not an ERROR payload"), + ] + if not raw: + layers.extend( + [ + LayerResult("format", "fail", "empty tool response"), + LayerResult("structure", "skip", "skipped after format failure"), + LayerResult("behavior", "skip", "skipped after format failure"), + ] + ) + return EvalResult(case.id, tuple(layers)) + layers.append(LayerResult("format", "pass", f"prose candidate ({len(raw)} chars)")) + missing = [phrase for phrase in case.required_phrases if not _phrase_present(raw, phrase)] + if missing: + layers.append( + LayerResult( + "structure", + "fail", + "missing required phrase(s): " + ", ".join(missing), + ) + ) + layers.append(LayerResult("behavior", "skip", "skipped after structure failure")) + return EvalResult(case.id, tuple(layers)) + if case.extra_structure is not None: + extra = case.extra_structure({"_prose": raw}) + if extra: + layers.append(LayerResult("structure", "fail", extra)) + layers.append(LayerResult("behavior", "skip", "skipped after structure failure")) + return EvalResult(case.id, tuple(layers)) + phrase_note = ( + "required phrases present" + if case.required_phrases + else "no required phrases" + ) + layers.append(LayerResult("structure", "pass", phrase_note)) + blob = {"_prose": raw} + if case.behavior_fn is None and not case.behavior: + layers.append(LayerResult("behavior", "pass", "no executable checks on this case")) + return EvalResult(case.id, tuple(layers)) + try: + if case.behavior_fn is not None: + case.behavior_fn(blob) + detail = "custom prose check passed" + else: + raise AssertionError("prose cases cannot run module BehaviorCheck oracles") + except Exception as exc: + layers.append(LayerResult("behavior", "fail", f"{type(exc).__name__}: {exc}")) + return EvalResult(case.id, tuple(layers)) + layers.append(LayerResult("behavior", "pass", detail)) + return EvalResult(case.id, tuple(layers)) + + +def _phrase_present(haystack: str, phrase: str) -> bool: + """Whole-token match for short words so ``this`` does not satisfy ``hi``.""" + token = phrase.lower() + text = haystack.lower() + if " " in token: + return token in text + return re.search(rf"(? dict[str, str]: """If the model omitted a path comment on a single-file case, use the source path.""" source_paths = [item["path"] for item in case.files if item.get("path")] diff --git a/src/local_coding_slm/eval/stub_ollama.py b/src/local_coding_slm/eval/stub_ollama.py index fab951c..ed2424c 100644 --- a/src/local_coding_slm/eval/stub_ollama.py +++ b/src/local_coding_slm/eval/stub_ollama.py @@ -17,14 +17,15 @@ WHITESPACE_NESTED, WHITESPACE_NO_FENCE, ) +from local_coding_slm.eval.cases_extended import OBSERVED_FIRST from local_coding_slm.ollama_client import DEFAULT_FAST_MODEL, DEFAULT_STRONG_MODEL def infer_case_id(user: str) -> str | None: - for case in CASES: - if case.task and case.task in user: - return case.id - return None + matches = [case for case in CASES if case.task and case.task in user] + if not matches: + return None + return max(matches, key=lambda case: len(case.task)).id def scripted_content(case_id: str, model_choice: str, visit: int, profile: str) -> str: @@ -34,14 +35,18 @@ def scripted_content(case_id: str, model_choice: str, visit: int, profile: str) return golden if profile != "observed": raise ValueError(f"unknown stub profile {profile!r}") - if case_id == "whitespace_extract_vague" and model_choice == "fast": + # These stay wrong for every fast call so the policy must escalate to strong. + if model_choice == "fast" and case_id == "whitespace_extract_vague": return WHITESPACE_NESTED - if case_id == "whitespace_extract" and model_choice == "fast" and visit == 1: - return WHITESPACE_NO_FENCE - if case_id == "multi_file_rename" and model_choice == "fast" and visit == 1: - return MULTI_FILE_PARTIAL - if case_id == "test_add_execute" and model_choice == "fast": + if model_choice == "fast" and case_id == "test_add_execute": return TEST_ADD_SHAPE_ONLY + first_fail = { + "whitespace_extract": WHITESPACE_NO_FENCE, + "multi_file_rename": MULTI_FILE_PARTIAL, + **OBSERVED_FIRST, + } + if model_choice == "fast" and visit == 1 and case_id in first_fail: + return first_fail[case_id] return golden diff --git a/tests/test_acceptance_a6.py b/tests/test_acceptance_a6.py new file mode 100644 index 0000000..0525562 --- /dev/null +++ b/tests/test_acceptance_a6.py @@ -0,0 +1,26 @@ +"""A6 must execute generated tests. Shape-only `def test` is not a pass.""" + +from __future__ import annotations + +import unittest + +from local_coding_slm.eval.acceptance import score_a6 +from local_coding_slm.eval.cases import TEST_ADD_GOLDEN, TEST_ADD_SHAPE_ONLY + + +class A6BehaviorTests(unittest.TestCase): + def test_golden_executes(self) -> None: + result = score_a6(TEST_ADD_GOLDEN) + self.assertTrue(result.passed) + self.assertEqual(result.layer("behavior").status, "pass") + + def test_shape_only_is_not_a6(self) -> None: + result = score_a6(TEST_ADD_SHAPE_ONLY) + self.assertFalse(result.passed) + self.assertEqual(result.layer("structure").status, "pass") + self.assertEqual(result.layer("behavior").status, "fail") + self.assertIn("def test", TEST_ADD_SHAPE_ONLY) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_eval_harness.py b/tests/test_eval_harness.py index d59778d..68cf8ba 100644 --- a/tests/test_eval_harness.py +++ b/tests/test_eval_harness.py @@ -6,7 +6,7 @@ import unittest from urllib.request import urlopen -from local_coding_slm.eval.cases import CASES_BY_ID, WHITESPACE_TASK +from local_coding_slm.eval.cases import CASES_BY_ID, SEED_CASE_IDS, WHITESPACE_TASK from local_coding_slm.eval.harness import run_campaign from local_coding_slm.eval.policy import next_plan from local_coding_slm.eval.record import AttemptRecord, summarize @@ -118,6 +118,7 @@ async def test_observed_campaign_records_layers(self) -> None: rows = await run_campaign( backend="stub", profile="observed", + case_ids=list(SEED_CASE_IDS), fast_ms=1, strong_ms=1, ) diff --git a/tests/test_eval_mcp_orchestrate.py b/tests/test_eval_mcp_orchestrate.py new file mode 100644 index 0000000..c069005 --- /dev/null +++ b/tests/test_eval_mcp_orchestrate.py @@ -0,0 +1,100 @@ +"""MCP + apply-gate closed loop. Stub Ollama, no GPU.""" + +from __future__ import annotations + +import unittest + +from local_coding_slm.eval.harness import run_campaign, run_orchestrated_campaign +from local_coding_slm.eval.jobs import MCP_JOBS +from local_coding_slm.eval.record import summarize + + +class McpOrchestratorTests(unittest.IsolatedAsyncioTestCase): + async def test_golden_corpus_pass_at_one(self) -> None: + rows = await run_campaign( + backend="stub", + profile="golden", + fast_ms=1, + strong_ms=1, + ) + stats = summarize(rows) + self.assertGreaterEqual(stats["cases"], 10) + self.assertEqual(stats["pass_at_1"], 1.0) + self.assertEqual(stats["pass_end"], 1.0) + + async def test_keep_jobs_never_call_local(self) -> None: + results = await run_orchestrated_campaign( + backend="stub", + profile="golden", + job_ids=["keep_incident", "keep_architecture", "keep_ambiguous"], + fast_ms=1, + strong_ms=1, + ) + self.assertEqual(len(results), 3) + for item in results: + self.assertFalse(item.delegated) + self.assertEqual(item.outcome, "kept_on_premium") + self.assertEqual(item.local_attempts, 0) + self.assertIsNone(item.review_decision) + + async def test_mcp_accept_rewrite_reject(self) -> None: + results = await run_orchestrated_campaign( + backend="stub", + profile="golden", + job_ids=[ + "mcp_extract_accept", + "mcp_move_accept", + "mcp_code_rewrite", + "mcp_reject_security", + "mcp_review_notes_only", + ], + fast_ms=1, + strong_ms=1, + ) + by_id = {item.job: item for item in results} + extract = by_id["mcp_extract_accept"] + self.assertTrue(extract.delegated) + self.assertEqual(extract.outcome, "applied_local") + self.assertEqual(extract.apply_source, "local") + self.assertTrue(extract.local_passed) + self.assertGreater(extract.local_attempts, 0) + + moved = by_id["mcp_move_accept"] + self.assertEqual(moved.outcome, "applied_local") + self.assertIn("_normalize_whitespace", extract.applied_text or "") + + rewritten = by_id["mcp_code_rewrite"] + self.assertEqual(rewritten.outcome, "applied_rewrite") + self.assertEqual(rewritten.apply_source, "premium") + self.assertIn("lo, hi = hi, lo", rewritten.applied_text or "") + + rejected = by_id["mcp_reject_security"] + self.assertTrue(rejected.local_passed) + self.assertEqual(rejected.outcome, "rejected") + self.assertFalse(rejected.applied) + self.assertIsNone(rejected.applied_text) + + notes = by_id["mcp_review_notes_only"] + self.assertEqual(notes.case_id, "review_login") + self.assertEqual(notes.outcome, "rejected") + self.assertFalse(notes.applied) + + async def test_observed_move_repairs_then_accepts(self) -> None: + results = await run_orchestrated_campaign( + backend="stub", + profile="observed", + job_ids=["mcp_move_accept"], + fast_ms=1, + strong_ms=1, + ) + item = results[0] + self.assertEqual(item.local_models, ("fast", "fast")) + self.assertEqual(item.outcome, "applied_local") + + def test_job_table_covers_keep_and_delegate(self) -> None: + self.assertTrue(any(job.eval_case is None for job in MCP_JOBS)) + self.assertTrue(any(job.eval_case is not None for job in MCP_JOBS)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_eval_score.py b/tests/test_eval_score.py index 8407815..dbce060 100644 --- a/tests/test_eval_score.py +++ b/tests/test_eval_score.py @@ -4,13 +4,14 @@ import unittest -from local_coding_slm.eval.cases import CASES_BY_ID, FIXTURES +from local_coding_slm.eval.cases import CASES_BY_ID, FIXTURES, SEED_CASE_IDS from local_coding_slm.eval.score import score_candidate class FixtureCorpusTests(unittest.TestCase): def test_every_fixture_matches_expected_layer(self) -> None: - self.assertGreaterEqual(len(FIXTURES), 8) + self.assertGreaterEqual(len(FIXTURES), 16) + self.assertGreaterEqual(len(SEED_CASE_IDS), 4) for fixture in FIXTURES: with self.subTest(fixture=fixture.name): case = CASES_BY_ID[fixture.case_id] @@ -49,6 +50,36 @@ def test_vague_and_precise_share_the_same_checker(self) -> None: self.assertEqual(precise.behavior, vague.behavior) self.assertNotEqual(precise.task, vague.task) + def test_corpus_covers_all_generation_tools(self) -> None: + from local_coding_slm.eval.cases import CASES + + tools = {case.tool for case in CASES} + self.assertTrue( + { + "local_code", + "local_explain", + "local_generate_tests", + "local_refactor", + "local_review", + }.issubset(tools) + ) + + def test_prose_explain_missing_phrase_is_structure(self) -> None: + case = CASES_BY_ID["explain_clamp"] + vague = next(item for item in FIXTURES if item.name == "explain_clamp_vague") + result = score_candidate(vague.text, case) + self.assertEqual(result.layer("format").status, "pass") + self.assertEqual(result.layer("structure").status, "fail") + self.assertEqual(result.layer("behavior").status, "skip") + self.assertIn("hi", result.layer("structure").message) + + def test_move_partial_is_format_not_behavior(self) -> None: + case = CASES_BY_ID["move_function_imports"] + partial = next(item for item in FIXTURES if item.name == "move_function_partial") + result = score_candidate(partial.text, case) + self.assertEqual(result.layer("format").status, "fail") + self.assertEqual(result.layer("behavior").status, "skip") + if __name__ == "__main__": unittest.main()