Expand eval corpus, MCP apply gate, A6 behavior, and CI

.github/workflows/tests.yml

@@ -0,0 +1,28 @@
+name: tests
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  unit:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install package
+        run: python -m pip install --upgrade pip && python -m pip install -e .
+      - name: Unit tests
+        run: python -m unittest discover -s tests -v
+        env:
+          PYTHONPATH: src
+      - name: Fixture corpus
+        run: python scripts/run_eval.py
+      - name: Scripted orchestrator
+        run: python scripts/run_orchestration.py
+      - name: Stub MCP golden
+        run: python scripts/run_harness.py --backend stub --profile golden --fast-ms 1 --strong-ms 1
+      - name: Stub MCP + apply gate
+        run: python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1

README.md

@@ -19,6 +19,8 @@ does bounded, mechanical generation on a private GPU host running
   scoring (transport / format / structure / behavior) and the committed
   fixture corpus. Cloud Agents can run the fixtures; live Ollama stays on
   the workstation.
+- **[docs/cloud-orchestrator-results-2026-09-07.md](docs/cloud-orchestrator-results-2026-09-07.md)**
+  — dated Cloud Agent run of the stub corpus and apply gate (no GPU).
 - **[docs/security-scan-results-2026-09-06.md](docs/security-scan-results-2026-09-06.md)**
   — dated Gitleaks, GitHub alert, tracked-tree, and deployment-safety results.
 - **[examples/](examples/)** — public-safe client config templates. Copy them
@@ -77,8 +79,12 @@ connected through SSH local forwarding.
    .venv/bin/python scripts/run_eval.py
    .venv/bin/python scripts/run_harness.py --backend stub --profile observed --out eval-runs/observed
    .venv/bin/python scripts/run_orchestration.py
+   .venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate
    ```
 
+   GitHub Actions (`.github/workflows/tests.yml`) runs the same no-GPU path
+   on push and pull request.
+
 6. Deployment safety checks (no GPU required; inspects this host only):
 
    ```bash

docs/cloud-orchestrator-results-2026-09-07.md

@@ -0,0 +1,69 @@
+# Cloud Agent orchestrator results — 2026-09-07
+
+Dated note from a Cursor Cloud Agent. This VM has **no GPU and no Ollama**.
+The local worker is the loopback stub. The premium role is this agent
+inspecting layer results and the apply-gate outcomes. It is **not** a
+desktop Cursor session against `qwen3.5:9b` / `devstral-small-2`.
+
+Commands (all passed):
+
+```bash
+PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
+.venv/bin/python scripts/run_eval.py
+.venv/bin/python scripts/run_orchestration.py
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --fast-ms 1 --strong-ms 1
+.venv/bin/python scripts/run_harness.py --backend stub --profile observed --fast-ms 1 --strong-ms 1
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1
+```
+
+## Unit suite
+
+**83 tests passed.** Includes fixture scoring, A6 behavior (shape-only
+`def test` fails), local fast→strong policy, and MCP + apply-gate
+integration.
+
+## Fixture corpus
+
+24 fixtures: every golden passes all four layers; known-fail fixtures
+stop at the intended layer (format / structure / behavior / transport).
+
+New executable cases exercised here: `move_function_imports`,
+`extract_shared_parser`, `split_pipeline`, `implement_clamp`,
+`explain_clamp`, `review_login`.
+
+## Stub MCP harness (not GPU time)
+
+Golden, 10 jobs, 10 attempts: **pass@1 = 1.00**, escalated = 0.00.
+`mcp_ms` p50 ≈ 5 ms (loopback stub).
+
+Observed, 10 jobs, 21 attempts: **pass@1 = 0.00**, **pass@end = 1.00**,
+escalated = 0.30. First failures: format 4, structure 5, behavior 2.
+Repair success is pass@2 (or pass@3 for `whitespace_extract_vague`).
+
+## Premium review (this agent)
+
+I treated the scripted apply-gate outcomes as the verdicts I would give
+on these **stub** patches:
+
+| Job | Route | My verdict | Why |
+| --- | --- | --- | --- |
+| keep_incident / architecture / live_tools / ambiguous | keep | keep on premium | Spec §8 do-not-delegate |
+| mcp_extract_accept | mechanical | **accept** | Layers passed; helper is module-level |
+| mcp_move_accept | mechanical | **accept** | `clamp` moved; `report.py` imports updated; oracles match |
+| mcp_parser_accept | mechanical | **accept** | Shared `parse_fields`; email/qty oracles match |
+| mcp_pipeline_accept | mechanical | **accept** | `run(" 1, 2, 3 ") == "6"` |
+| mcp_tests_accept | mechanical | **accept** | Generated tests actually execute (A6) |
+| mcp_explain_accept | mechanical | **accept** | Names clamp / lo / hi and clipping |
+| mcp_code_rewrite | mechanical | **rewrite** | Local clamp is correct; I would still swap inverted bounds |
+| mcp_review_notes_only | mechanical + security | **reject apply** | `local_review` flagged None/auth; that is notes, not a patch |
+| mcp_reject_security | mechanical + security | **reject** | Do not apply until a human/premium pass on bounds |
+
+Closed loop: **13 jobs, 9 delegated, 11 applied, 2 held.** Keep jobs did
+not call `local_*`. Rejected jobs had passing local layers and still did
+not apply.
+
+## What this note is not
+
+- Live tok/s, pass@1 on the workstation tags, or GPU placement
+- Proof that desktop Cursor's tool picker chose `local_refactor`
+- A claim that stub `mcp_ms` is model latency

docs/evaluation-protocol.md

@@ -57,6 +57,12 @@ The same scorer runs against:
 | `whitespace_extract_vague` | `local_refactor` | **Same checker**, vaguer prompt. Measures prompt-contract, not a looser oracle |
 | `multi_file_rename` | `local_refactor` | Two files must be returned; `add` → `plus`; `total([1,2,3]) == 6` |
 | `test_add_execute` | `local_generate_tests` | Generated tests are imported with `add.py` and the `test_*` functions are called |
+| `move_function_imports` | `local_refactor` | Move `clamp` into `bounds.py` and update `report.py` imports |
+| `extract_shared_parser` | `local_refactor` | Extract `parse_fields` into `csv_parse.py`; both callers import it |
+| `split_pipeline` | `local_refactor` | Split `run` into `load.py` / `transform.py` / `pipeline.py` |
+| `implement_clamp` | `local_code` | Implement `clamp` from a spec, no starter file |
+| `explain_clamp` | `local_explain` | Prose: names the function and bounds; mentions clipping |
+| `review_login` | `local_review` | Prose first-pass: flags None and missing auth. **Not** an apply |
 
 Known-fail fixtures are part of the corpus. They prove the scorer can
 tell layers apart:
@@ -136,9 +142,15 @@ CI uses a scripted reviewer. These tests do **not** call Cursor, GPT, or
 Claude, and they do not prove that a live IDE agent followed the rule
 file. They prove the state machine the agent is supposed to follow.
 
+The same gate runs after **real stdio MCP** calls when you pass
+`--orchestrate`. Stub Ollama still supplies the worker text. Keep jobs
+never call `local_*`. Accept / rewrite / reject then run on the scored
+candidate.
+
 ```bash
-PYTHONPATH=src .venv/bin/python -m unittest tests.test_eval_orchestrate -v
+PYTHONPATH=src .venv/bin/python -m unittest tests.test_eval_orchestrate tests.test_eval_mcp_orchestrate -v
 .venv/bin/python scripts/run_orchestration.py
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate
 ```
 
 ## Commands
@@ -149,8 +161,11 @@ Fixture protocol (no GPU; this is what Cloud Agents can run):
 PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v
 .venv/bin/python scripts/run_eval.py
 .venv/bin/python scripts/run_orchestration.py
+.venv/bin/python scripts/run_harness.py --backend stub --profile golden --orchestrate --fast-ms 1 --strong-ms 1
 ```
 
+GitHub Actions (`.github/workflows/tests.yml`) runs the same no-GPU path on every push.
+
 Live protocol (workstation with Ollama):
 
 ```bash
@@ -174,7 +189,9 @@ After repeated live runs, a paper may claim:
 - That a vaguer prompt raises structure failures on the same oracle
   (`whitespace_extract` vs `whitespace_extract_vague`).
 - That shape-only test generation overstates success relative to
-  executed tests (`test_add_execute`).
+  executed tests (`test_add_execute`; A6 now uses this checker).
+- That keep-vs-delegate and accept/rewrite/reject are enforceable as a
+  state machine on stub workers plus real stdio MCP.
 
 It still may not claim:
 

docs/phase3-log.md

@@ -6,13 +6,16 @@ IPs, or secrets here.
 Reproducible commands, complete 2026-09-06 observations, and limitations:
 [local-acceptance-results-2026-09-06.md](local-acceptance-results-2026-09-06.md).
 Scoring method: [evaluation-protocol.md](evaluation-protocol.md).
+Cloud-safe stub + apply-gate run:
+[cloud-orchestrator-results-2026-09-07.md](cloud-orchestrator-results-2026-09-07.md).
 
 | Date | Tool | Model | Task | Result | Notes |
 | --- | --- | --- | --- | --- | --- |
 | 2026-09-04 | local_generate_tests | fast | unittest for `merge_dotenv` | accepted with trim | Dropped one truncated test; quoted-value case kept; applied as `tests/test_envfile.py` |
-| 2026-09-06 | local_generate_tests | fast | live A6 tests for synthetic `add()` | accepted | Real Ollama via stdio MCP; output-shape acceptance passed |
+| 2026-09-06 | local_generate_tests | fast | live A6 tests for synthetic `add()` | accepted | Real Ollama via stdio MCP; **shape-only** A6 at that date |
 | 2026-09-06 | local_refactor | fast | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 6.7s |
 | 2026-09-06 | local_refactor | strong | extract module-level whitespace helper | accepted | Real Ollama; generated module parsed and preserved behavior across 3 executed cases; warm run 25.6s; model reported 46% CPU / 54% GPU at 16K context |
+| 2026-09-07 | all local_* eval tools | stub | 10-case corpus + apply gate | accepted / rewritten / rejected per job | Cloud Agent; no GPU; A6 now executes tests |
 
 Columns:
 

docs/roadmap.md

@@ -10,7 +10,7 @@ LAN addresses, or SKUs here.
 | --- | --- | --- |
 | 1 — inference host | Done on the workstation lab | Ollama + starter tags on a private GPU |
 | 2 — MCP bridge | Done | `local-coding-slm` stdio tools; Cursor / Copilot / Claude adapters |
-| 3 — measure | Protocol + fixture corpus + orchestrator contract; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet |
+| 3 — measure | Protocol + expanded corpus + MCP apply gate + CI; live rates still informal | Layered scoring; scripted premium routing/review; no auto-classifier yet |
 | T12 second NVIDIA host | Blocked on host power | WSL GPU via SSH; see [examples/downstairs-wsl-gpu.md](../examples/downstairs-wsl-gpu.md) |
 
 Finish Phase 3 measurement (layered live rates on the committed corpus)

scripts/prove_acceptance.py

@@ -1,5 +1,4 @@
-#!/usr/bin/env python3
-"""Run A5/A6 against the local stdio MCP server. Loads repo-root .env."""
+"""A5/A6 against stdio MCP. A6 executes generated tests via the eval scorer."""
 
 from __future__ import annotations
 
@@ -13,15 +12,14 @@
 if str(SRC) not in sys.path:
     sys.path.insert(0, str(SRC))
 
+from local_coding_slm.eval.acceptance import score_a6  # noqa: E402
 from local_coding_slm.server import _load_dotenv  # noqa: E402
 
 _load_dotenv()
 
 from mcp import ClientSession, StdioServerParameters  # noqa: E402
 from mcp.client.stdio import stdio_client  # noqa: E402
 
-TINY_FN = "def add(a: int, b: int) -> int:\n    return a + b\n"
-
 
 async def _run() -> int:
     python = sys.executable
@@ -62,28 +60,32 @@ async def _run() -> int:
                 print("FAIL A5 local_status")
                 return 1
 
+            from local_coding_slm.eval.cases import CASES_BY_ID
+
+            case = CASES_BY_ID["test_add_execute"]
             tests = await session.call_tool(
                 "local_generate_tests",
                 {
-                    "task": (
-                        "Write pytest unit tests for add(). Cover two positives "
-                        "and one negative. Return a single fenced file."
-                    ),
-                    "files": [{"path": "add.py", "content": TINY_FN}],
-                    "language": "python",
-                    "style": "pytest",
+                    "task": case.task,
+                    "files": list(case.files),
+                    "language": case.language,
+                    "style": case.style,
                     "model": "fast",
-                    "max_tokens": 400,
+                    "max_tokens": case.max_tokens,
                 },
             )
             tests_text = "".join(
                 block.text for block in tests.content if getattr(block, "text", None)
             )
             print("A6")
             print(tests_text)
-            lowered = tests_text.lower()
-            if tests_text.startswith("ERROR:") or "def test" not in lowered:
-                print("FAIL A6 local_generate_tests")
+            result = score_a6(tests_text)
+            for layer in result.layers:
+                print(f"  {layer.status:4} {layer.name}: {layer.message}")
+            if not result.passed:
+                first = result.first_failure
+                extra = f" stop={first.name}" if first else ""
+                print(f"FAIL A6 local_generate_tests{extra}")
                 return 1
     print("PASS A5 A6")
     return 0

scripts/run_harness.py

@@ -14,7 +14,12 @@
 if str(SRC) not in sys.path:
     sys.path.insert(0, str(SRC))
 
-from local_coding_slm.eval.harness import format_summary, run_campaign  # noqa: E402
+from local_coding_slm.eval.harness import (  # noqa: E402
+    format_orchestrated,
+    format_summary,
+    run_campaign,
+    run_orchestrated_campaign,
+)
 from local_coding_slm.eval.record import summarize, write_jsonl  # noqa: E402
 
 
@@ -23,15 +28,54 @@ def main() -> None:
     parser.add_argument("--backend", choices=("stub", "live"), default="stub")
     parser.add_argument("--profile", choices=("golden", "observed"), default="golden")
     parser.add_argument("--case", action="append", dest="case_ids", default=None)
+    parser.add_argument("--job", action="append", dest="job_ids", default=None)
     parser.add_argument("--repeat", type=int, default=1)
     parser.add_argument("--fast-ms", type=float, default=8.0)
     parser.add_argument("--strong-ms", type=float, default=25.0)
+    parser.add_argument(
+        "--orchestrate",
+        action="store_true",
+        help="Route + MCP local loop + premium apply gate (scripted reviewer)",
+    )
     parser.add_argument(
         "--out",
         default="",
-        help="Directory for attempts.jsonl and summary.json (gitignored eval-runs/)",
+        help="Directory for JSONL/JSON summaries (gitignored eval-runs/)",
     )
     args = parser.parse_args()
+    if args.orchestrate:
+        results = asyncio.run(
+            run_orchestrated_campaign(
+                backend=args.backend,
+                profile=args.profile,
+                job_ids=args.job_ids,
+                fast_ms=args.fast_ms,
+                strong_ms=args.strong_ms,
+            )
+        )
+        print(format_orchestrated(results))
+        if args.out:
+            dest = Path(args.out)
+            dest.mkdir(parents=True, exist_ok=True)
+            payload = [
+                {
+                    "job": item.job,
+                    "delegated": item.delegated,
+                    "route": item.route_reason,
+                    "outcome": item.outcome,
+                    "applied": item.applied,
+                    "source": item.apply_source,
+                    "models": list(item.local_models),
+                    "review": item.review_decision,
+                }
+                for item in results
+            ]
+            (dest / "orchestrated.json").write_text(
+                json.dumps(payload, indent=2) + "\n",
+                encoding="utf-8",
+            )
+            print(f"wrote {dest / 'orchestrated.json'}")
+        raise SystemExit(0 if results else 1)
     rows = asyncio.run(
         run_campaign(
             backend=args.backend,

spec.md

@@ -721,9 +721,10 @@ retry as pass-at-one.
 The local harness (`scripts/run_harness.py`) measures fast→strong failover
 after a task is already delegated. Routing, keep-vs-delegate, and the
 premium accept/rewrite/reject apply gate are the orchestrator tests
-(`tests/test_eval_orchestrate.py`, `scripts/run_orchestration.py`). Those
-use a scripted stand-in for the main LLM; they do not call Cursor or
-another hosted API.
+(`tests/test_eval_orchestrate.py`, `scripts/run_orchestration.py`).
+`--orchestrate` runs that gate after real stdio MCP (stub or live worker).
+CI uses a scripted stand-in for the main LLM; it does not call Cursor or
+another hosted API. GitHub Actions runs the no-GPU path on every push.
 
 Only after that, consider automatic task classification.
 
@@ -760,7 +761,7 @@ Run from the workstation with `OLLAMA_BASE_URL` set.
 | A3 | Chat prompt to the strong model | Completes; GPU or GPU+RAM is acceptable |
 | A4 | Workstation reaches Ollama through SSH; an unauthorized LAN client cannot reach 11434 | No direct LAN exposure; skip if same-machine |
 | A5 | `local_status` MCP tool | Reports both models and the configured base URL host *without* requiring that URL in git |
-| A6 | `local_generate_tests` with one small function | Returns a test file / diff the premium agent can apply |
+| A6 | `local_generate_tests` with `test_add_execute` | Four eval layers pass, including executed `test_*` functions. A `def test` substring is not enough |
 | A7 | Cursor Agent | Premium model calls a `local_*` tool on a mechanical prompt |
 | A8 | Copilot Agent (VS Code) | Same tool appears and runs |
 | A9 | Claude Code local | `claude mcp list` shows `local-coding-slm` connected |

src/local_coding_slm/eval/acceptance.py

@@ -0,0 +1,8 @@
+"""A6 acceptance uses the behavior-layer scorer, not a `def test` substring."""
+
+from local_coding_slm.eval.cases import CASES_BY_ID
+from local_coding_slm.eval.score import EvalResult, score_candidate
+
+
+def score_a6(text: str) -> EvalResult:
+    return score_candidate(text, CASES_BY_ID["test_add_execute"])