diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 03005439..04e188c5 100755 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -17,7 +17,11 @@ jobs: - run: ruff check . frontend: - runs-on: ubuntu-latest + # Pinned: ubuntu-latest now resolves to Ubuntu 26.04 on some runners, where + # `playwright install --with-deps chromium` (Playwright 1.55.0) has no + # supported system packages and exits 1. Return to ubuntu-latest once + # @playwright/test is upgraded to a release that supports 26.04. + runs-on: ubuntu-24.04 defaults: run: working-directory: evalrx/reporting/web diff --git a/CHANGELOG.md b/CHANGELOG.md index 2de147c3..80b3c9ae 100755 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,77 @@ Versions follow [Semantic Versioning](https://semver.org/). ## [Unreleased] +### Added — validation report for the published weight edit on Qwen3.5-0.8B + +`docs/validation/astra_p50c_20261115.md` records the structure read-out, the +teacher-forced audit, the generation-level repair through the L4 fix path, and +the attempt to derive the edit from a loop contrast, with their limits. The +README gains sections on probe tools and weight edits. + +### Added — generation-level check of a weight edit + +`tools/generate_astra_edit_qwen35.py` runs the published p50c edit through the +L4 fix path on GSM8K: baseline, edit and random-direction control are each +generated (thinking on, the original sampling settings, batched and cached per +weight state), validated with the paired test, and summarised by whether the +model left its thinking block, stopped on its own, ended in a loop, and was +correct. `--timing-only` checks that padded batching is exact on the +architecture and prints a time estimate. + +### Added — deriving and comparing weight edits along a generation + +`continuation_contrast_edit` derives a rank-one edit from the contrast between +positions inside a repeated stretch and positions outside one, along recorded +generations; `collect_continuation_stats` and `rank_columns` expose the group +means and per-neuron signals. `fit_rank_one` re-expresses any edit as the +rank-one rule that reproduces it (so it can be scaled), and `edit_agreement` +compares two edits by direction and neuron overlap. `contrast_edit` and the new +function share one core, `edit_from_stats`. `tools/recover_astra_edit_qwen35.py` +runs the comparison against the published p50c edit. + +### Added — weight-edit audit along a generation + +`weight_edit_continuation_audit` follows a weight edit along recorded +generations (`case.observed`), teacher-forced, and reports the change in +next-token NLL and the KL from the unedited model separately for positions +inside a repeated stretch and outside one, beside the random-direction, +random-neuron and steering controls. `tools/check_astra_edit_qwen35.py` and +`tools/audit_astra_edit_qwen35.py` run the structure check and this audit for +the published p50c edit on Qwen3.5-0.8B. + +### Added — probe tools (`evalrx.probing`) + +Every registered analyzer is described as a tool: family, the question it +answers, access level, cost hint, JSON-schema arguments, and its availability +on a given model and batch with every reason it cannot run +(`describe_probes`, modelled on VLMProbe's `Prober.describe()`). +`ProbeRouter` ranks runnable probes for failure modes, families and access; +`ProbeToolbox.call(name, params)` runs one with schema-checked arguments and +`.tools()` returns `evalrx.core.tool.Tool` objects. + +The agentic loop gains `list_probes` and `run_probe` accepts `name` + `params` +(or `probes: [...]`). Named calls run exactly the named probes, add to the +evidence gathered so far, and are rejected before any model work when the +arguments or the probe do not fit. `ProbeAgent.probe` takes `analyzer_params` +and `exact`; a pinned re-run replays the arguments of the named call. +`run_probe` with no arguments is unchanged and still limited to 3 calls per +run; the tool's overall cap is 6 to leave room for named calls. + +### Added — weight-space programming (`evalrx.wsp`) + +Direct, training-free weight edits: `TensorEdit` / `WeightEdit` (rank-one rule +`w' = w - alpha u (u^T w)`, low-rank, dense delta, sparse overwrite), `applied()` +(writes the edit and restores the exact weights, also on error), controls +(`random_direction`, `random_columns`), `edit_structure`, `contrast_edit` +(an edit derived from failing vs passing cases), and a reader for the edit +files published with *Sparks of Weight-Space Programming*. + +New analyzers `weight_edit_structure` and `weight_edit_audit` (the edit beside +random-direction, random-neuron and steering controls). New opt-in L4 repair +candidate `rank_one_weight_edit` via `FixAgent(weight_edit=...)`, validated by +the existing paired test and recorded with a matched control. See +`docs/wsp_and_probe_tools.md`, including what has not been checked yet. + ### Fixed — delta-rule relevance on transformers 5.23 transformers 5.23 moved the grouped query/key head expansion of Qwen3-Next / diff --git a/README.md b/README.md index 6e21fa65..79a5a890 100644 --- a/README.md +++ b/README.md @@ -155,11 +155,29 @@ spec runs through a black-box API or a white-box local backend — only the available capability set changes. [Analyzer Zoo →](docs/analyzers.md) · [Architecture guide →](docs/architecture.md) +Analyzers are also callable as tools. Each one is described by the question it +answers, what it needs and the arguments it takes, and reports whether it can +run on the model and data in hand — and if not, why: + +```python +from evalrx.probing import ProbeToolbox + +box = ProbeToolbox(model, cases) +box.describe()["attention_knockout"]["missing"] # why a probe cannot run here +box.route(failure_modes=["visual_neglect"]).names # what to run first +box.call("logit_lens", {"top_k": 5}) # arguments checked against a schema +``` + +In the agentic loop the judge gets the same view: `list_probes` shows what can +run, and `run_probe` takes a probe's name and arguments. +[Probe tools →](docs/wsp_and_probe_tools.md#probe-tools) + ## Docs & Examples [Quickstart](docs/quickstart.md) · [CLI](docs/cli.md) · [Exploratory Analysis](docs/m2_analysis.md) · [Intervention & Verification](docs/intervention.md) · +[Weight Edits & Probe Tools](docs/wsp_and_probe_tools.md) · [Analyzer Zoo](docs/analyzers.md) · [Architecture](docs/architecture.md) · [Extending EvalRX](docs/extending.md) · [Roadmap](docs/roadmap.md) — all live, searchable, at [evalvitals.github.io/evalrx](https://evalvitals.github.io/evalrx/overview/). @@ -203,11 +221,47 @@ raising it rather than climbing on its own. | **L2** | Scaffolds around an unchanged model — multi-call, tools, aggregation | ✅ | | **L3a** | Read internals — attention-guided cropping, contrastive decoding | ✅ | | **L3b** | Write internals — attention reweighting, activation steering | ✅ | -| **L4** | **Parameter space — build a dataset, fine-tune, re-test** | ✅ LoRA on the LLM ([`fix_internals.py`](evalrx/eval_agent/stages/fix/internals.py)); other recipe shapes recorded, not yet executed | +| **L4** | **Parameter space — build a dataset, fine-tune, re-test; or write a training-free weight edit** | ✅ LoRA on the LLM ([`internals.py`](evalrx/eval_agent/stages/fix/internals.py)); ✅ rank-one weight edit, opt-in ([`weight_edit.py`](evalrx/eval_agent/stages/fix/weight_edit.py)); other recipe shapes recorded, not yet executed | **L3b and L4 only exist for open weights** — you cannot modify a forward pass or fine-tune through somebody's API. +### Weight edits + +A weight edit changes a model by writing new values into a few tensors, with no +training: for example `w' = w - alpha * u (u^T w)` on some columns of one MLP +output projection. `evalrx.wsp` builds, applies and restores such edits, reads +their structure from the weights, and audits them against controls: + +```python +from evalrx.analyzers.weights import WeightEditContinuationAudit +from evalrx.wsp import WeightEdit, applied, edit_structure + +edit = WeightEdit.load("path/to/edit") +edit_structure(model, edit) # where it moved, its rank, its alpha +with applied(model, edit): # written in, then restored exactly + answer = model.generate(case.inputs) +WeightEditContinuationAudit(edit=edit).run(model, cases) # vs random direction, other neurons, steering +``` + +As a repair it is one more L4 candidate, validated by the same paired test and +recorded beside a matched random-direction control: +`FixAgent(max_tier="L4", weight_edit=True, finetune_pool=...)`. + +Checked on Qwen3.5-0.8B with the edit published in +[*Sparks of Weight-Space Programming*](https://rikiyat.github.io/blog/weight-space-programming/): + +| | Unedited | Edited | Random-direction control | +|---|---|---|---| +| GSM8K correct (64 problems, 16,384-token budget) | 6 | 37 | 7 | +| Left the thinking block | 10.9% | 92.2% | 9.4% | +| Ended in a verbatim loop | 87.5% | 1.6% | 90.6% | + +31 fixed, 0 broken. That edit was written by another system; deriving one from +the model's own behaviour recovered about half of its neurons and not its +direction. [Full results and limits →](docs/validation/astra_p50c_20261115.md) · +[Weight edits guide →](docs/wsp_and_probe_tools.md#weight-edits) + ## Project Status EvalRX is an early-stage research toolkit; interfaces may evolve, and some diff --git a/docs/analyzers.md b/docs/analyzers.md index 4d2c4703..fc1cf1e1 100644 --- a/docs/analyzers.md +++ b/docs/analyzers.md @@ -26,9 +26,10 @@ model, not by a black-box / white-box label. | [Lens](#lens) | `lens/` | 6 | 0 | | [Patching](#patching) | `patching/` | 6 | 0 | | [Geometry](#geometry) | `geometry/` | 2 | 0 | +| [Weights](#weights) | `weights/` | 3 | 0 | | [Reasoning](#reasoning) | `reasoning/` | 7 | 0 | | [Agent](#agent) | `agent/` | 7 | 0 | -| **Total** | | **54** | **2** | +| **Total** | | **57** | **2** | How to read the columns: @@ -215,6 +216,36 @@ encoders; on a decoder-only MLLM, read them as within-stream layer similarity. | Linear probe | `linear_probe` | `HIDDEN_STATES` | text + image | [Alain & Bengio 2016](https://arxiv.org/abs/1610.01644) | ✓ | +### Weights + +`evalrx/analyzers/weights/`: what a direct weight edit changes, and whether the +change is specific (`LOGITS` on a local model; the edit is written into the +module for the measurement and restored exactly afterwards). Both analyzers take +the edit as a constructor argument, an `evalrx.wsp.WeightEdit`, so a selector +only offers them when the host passes a constructed instance. See +[Weight edits and probe tools](wsp_and_probe_tools.md). + +`weight_edit_structure` needs no cases: it reports where the weights moved, how +far, the rank of the change, and for a rank-one change the strength `alpha` of +the rule `w' = w - alpha u (u^T w)` that reproduces it. `weight_edit_audit` +measures the edit on the next-token distribution beside a random direction of +the same size, the same rule on other neurons, and a constant activation shift +with the weights untouched. + +`weight_edit_continuation_audit` takes the same measurement along recorded +generations (`case.observed` appended to the prompt, teacher-forced), split into +positions inside a repeated stretch and positions outside one. Use it for an +edit whose effect only appears during generation, such as one that breaks +repetition loops. + + +| Analyzer | Key | Requires | Modality | Paper | Status | +|---|---|---|---|---|---| +| Weight-edit structure | `weight_edit_structure` | `LOGITS` | text | [Takehi et al. 2026 (WSP)](https://rikiyat.github.io/blog/weight-space-programming/) | ✓ | +| Weight-edit audit | `weight_edit_audit` | `LOGITS` | text | [Takehi et al. 2026 (WSP)](https://rikiyat.github.io/blog/weight-space-programming/); [Zhang et al. 2026 (EditLens)](https://xxlya.github.io/editlens/) | ✓ | +| Weight-edit continuation audit | `weight_edit_continuation_audit` | `LOGITS` | text | [Takehi et al. 2026 (WSP)](https://rikiyat.github.io/blog/weight-space-programming/); [Zhang et al. 2026 (EditLens)](https://xxlya.github.io/editlens/) | ✓ | + + ### Reasoning `evalrx/analyzers/reasoning/`: text-chain diagnostics, all black-box. Read the diff --git a/docs/architecture.md b/docs/architecture.md index 9e050fb8..51212135 100755 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -13,6 +13,8 @@ evalrx/ +-- specs.py # model identity registry +-- models/ # model composition, runtime backends, compatibility shims +-- analyzers/ # analyzers grouped by capability (attention, lens, uncertainty, …) ++-- probing/ # analyzers as tools: catalog, availability, routing, calls by name ++-- wsp/ # weight edits: build, apply/restore, controls, structure +-- datasets/ # loaders that produce FailureCase / CaseBatch +-- stats/ # statistical tests: McNemar, e-value, bootstrap CI, Friedman `-- eval_agent/ # automated diagnosis loop + selective-inference orchestration diff --git a/docs/intervention.md b/docs/intervention.md index 95a9bd87..83556ff0 100644 --- a/docs/intervention.md +++ b/docs/intervention.md @@ -63,6 +63,11 @@ it: | L3b | internals (write) | Modifies the forward pass (attention reweighting, sink suppression, activation steering). | | L4 | parameter space | Fine-tune recipe, always recorded for a human decision. v1 additionally *executes* one shape — LoRA on the language model — when you pass `FixAgent(finetune_pool=...)` (a diagnosis-only `CaseBatch`, disjoint from what's being validated); other recipe shapes stay recorded-only. | +L4 also has a training-free option: `FixAgent(weight_edit=True, finetune_pool=...)` +proposes a rank-one weight edit built from the diagnosis pool, validated by the +same paired test and recorded beside a matched random-direction control. See +[Weight Edits and Probe Tools](wsp_and_probe_tools.md#as-a-repair-fix-tier-l4). + ```python outcome = loop.run_fix(report, failure_cases, auto_escalate=True) # steps L2 → L3a → L3b ``` diff --git a/docs/roadmap.md b/docs/roadmap.md index b82a05cb..eb3cd225 100755 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -28,6 +28,8 @@ surfaces and planned interfaces. | **RunContext** | **Single owner of a run's output directory (`report/`, `figures/`, `artifacts/`, `experiments/`, `fixes/`, `manifest.json`, auto-generated `README.txt`); per-trial folders (`new_trial()`) for fix candidates and M5 experiments.** | | **FixAgent** | **Tiered post-loop repair (L1 prompt → L2 scaffold → L3a/b internals → L4 parameter space); paired McNemar + e-value validation; `max_repair_rounds` feedback-driven retry with candidate dedup; `loop.run_fix(auto_escalate=True)` steps the tier ladder.** | | **Result image overlays** | **`RelativeAttentionResult.overlay()`/`.image_overlays()` — CAM-style heatmap-on-source-image blending; duck-typed hook picked up by `RunLogger` into `figures/`.** | +| **Probe tools** | **`evalrx.probing`: every analyzer as a tool (family, question, access, cost, JSON-schema arguments), availability with reasons, `ProbeRouter`, `ProbeToolbox.call(name, params)`; agentic `list_probes` and named `run_probe`.** | +| **Weight edits (WSP)** | **`evalrx.wsp`: `WeightEdit` / `TensorEdit` (rank-one, low-rank, dense, sparse), reversible `applied()`, controls, structure read-out, contrast-derived edits; `weight_edit_structure` / `weight_edit_audit` analyzers; opt-in L4 `rank_one_weight_edit` repair. Tested on a planted-defect model and tiny random modules, not yet on a full-size model.** | | Contract tests | Parametrized pyod-style suite covering all 26 registered analyzers (554 unit tests). | ## Stage 2 stubs (registered, `_run` raises `NotImplementedError`) diff --git a/docs/validation/astra_p50c_20261115.md b/docs/validation/astra_p50c_20261115.md new file mode 100644 index 00000000..820326b6 --- /dev/null +++ b/docs/validation/astra_p50c_20261115.md @@ -0,0 +1,127 @@ +# Published weight edit on Qwen3.5-0.8B: audit and repair + +Date: 2026-11-15. Model: `Qwen/Qwen3.5-0.8B`, revision +`2fc06364715b967f1860aea9cf38778875588b17`. Edit: p50c from *Sparks of +Weight-Space Programming* (Takehi, Teoh, Zhang & Khattab, 2026), 64 columns of +`mlp.down_proj` in each of layers 15 and 19. Texts and reference numbers: +[EditLens](https://github.com/assassin808/edit-lens). No closed model was +called. Hardware: one RTX PRO 6000; transformers 5.23.0, torch 2.15. + +What was run: the weight-edit tools of `evalrx.wsp` on a real checkpoint, with +an edit written by someone else. The structure read-out, the teacher-forced +audit and the generation-level repair reproduce the published findings. An +attempt to DERIVE a comparable edit from the model's own behaviour recovered +part of the neurons and neither the direction nor the layers. + +## 1. Structure, read from the weights + +`python tools/check_astra_edit_qwen35.py --edit-dir /astra_logs/exp8_math/runs/p50c/edit` + +| | Layer 15 | Layer 19 | +|---|---:|---:| +| Base hash / rebuilt hash match the manifest | yes / yes | yes / yes | +| Changed columns / rows | 64 / 1024 | 64 / 1023 | +| Relative change | 0.1064 | 0.0811 | +| Effective rank | 1.036 | 1.046 | +| Rank-one residual | 0.0023 | 0.0029 | +| Alpha (± spread over columns) | 6.0006 ± 0.0018 | 6.0004 ± 0.0021 | +| Projection selectivity | 4.57 | 3.29 | + +The loader rebuilds the evaluated checkpoint bit for bit. The change is the +rule `w' = w - 6 u (u^T w)` on 64 columns per layer; \|cos\| between the two +layers' directions is 0.849. + +## 2. Teacher-forced audit along the unedited model's generations + +`python tools/audit_astra_edit_qwen35.py --edit-lens ` + +Twelve GSM8K generations of the unedited model that never left the thinking +block, first 4,096 tokens, 48,099 scored positions. A position counts as +in-loop when the 32 tokens before the predicted token occurred earlier. Values +are the change in NLL of the recorded next token, loop / productive. + +| State | This run | EditLens (6 texts) | +|---|---|---| +| Unedited NLL | 0.022 / 0.488 | 0.021 / 0.505 | +| p50c edit | +0.079 / +0.047 | +0.093 / +0.053 | +| Random direction, same columns and norms | +0.011 / +0.010 | +0.006 / +0.011 | +| Same rule, 64 other columns | -0.000 / +0.000 | +0.001 / +0.001 | +| Constant activation shift | +0.029 / +0.125 | +0.038 / +0.125 (their c = 1) | + +Dose curve of the fitted rule on twelve other generations +(`tools/recover_astra_edit_qwen35.py`): + +| Alpha | This run | EditLens | +|---:|---|---| +| 1 | +0.003 / +0.004 | +0.005 / +0.005 | +| 2 | +0.008 / +0.009 | +0.012 / +0.012 | +| 6 | +0.082 / +0.046 | +0.093 / +0.053 | +| 10 | +0.707 / +0.150 | +0.744 / +0.150 | + +The edit is specific to its direction and its neurons, acts more inside loops, +and is not free on ordinary text: the most likely token changes at 4.9% of +productive positions. Removing the component along `u` (alpha = 1) does almost +nothing; the effect comes from the overshoot. A constant activation shift of +the edit's mean size does less in loops and more damage outside them. + +## 3. Generation through the L4 fix path + +`python tools/generate_astra_edit_qwen35.py --edit-lens ` + +64 GSM8K test problems (every 20th), one sample each, thinking on, T = 0.6, +top_p = 0.95, top_k = 20, presence penalty 1.5, up to 16,384 new tokens, run as +`FixAgent(weight_edit={"edit_dir": ...})`. 74 minutes. + +| | Unedited | p50c edit | Random-direction control | +|---|---:|---:|---:| +| Correct | 6 / 64 | 37 / 64 | 7 / 64 | +| Left the thinking block | 10.9% | 92.2% | 9.4% | +| Stopped on its own | 10.9% | 85.9% | 9.4% | +| Ran to the token cap | 89.1% | 14.1% | 90.6% | +| Ended in a verbatim loop | 87.5% | 1.6% | 90.6% | +| Mean share of loop tokens | 57.4% | 7.6% | 58.6% | +| Mean new tokens | 15,077 | 9,509 | 15,282 | + +Paired validation, edit against the unedited model: 31 fixed, 0 broken, effect ++0.484 (CI +0.375 to +0.609), e-value 6.7e7 against a threshold of 20. The +published accuracies at a 32,768-token budget are 13.8% and 67.1%. + +## 4. Deriving the edit from loop vs productive positions + +`python tools/recover_astra_edit_qwen35.py --edit-lens ` + +Derived on the twelve generations of section 2, audited on twelve others. + +| | Result | +|---|---| +| Layers the contrast would pick | 9 and 12; the published layers rank 10th and 16th of 24 | +| \|cos\| with the published direction | 0.351 (layer 15), 0.444 (layer 19) | +| Published neurons in the derived top 64 (chance: 1.1) | 33 (layer 15), 22 (layer 19) | +| The same with the published direction supplied | 45, 34 | +| Derived edit at layers 15 + 19, alpha 6 | +0.050 / +0.026 | +| Random direction on the derived neurons | +0.187 / +0.022 | +| Derived edit at its own layers 9 + 12, alpha 6 | +0.088 / +0.004, on 6 of 12 texts only | + +The contrast finds which neurons matter. Its direction is not validated: on the +derived neurons a random direction disturbs loops more than the derived one. A +positive loop NLL change is cheap, because the unedited loop NLL is near zero +and can only rise; it means something only against controls. The layer 9 + 12 +edit has no controls yet. + +## Limits + +- One model, one edit, one benchmark. Sections 2 and 4 use 12 + 12 texts; section + 3 uses 64 problems, one sample, one seed and one control draw. +- Section 3 differs from the original evaluation: half the token budget, + Hugging Face `generate` instead of vLLM, numeric grading of the last boxed + answer instead of math-verify. +- "0 broken" rests on the 6 problems the unedited model got right. Harm on tasks + the model already does well is not measured here. +- The unedited model writes a boxed answer in 43.8% of runs and is correct in + 9.4%; most of those boxes sit inside an unfinished thinking block. Whether the + numeric grader misses correct ones has not been checked. +- Batched generation pads prompts on the left. In float32 the padded-vs-alone gap + is 0.016 against 0.004 for one prompt repeated; in bfloat16, batching alone + moves the output by 0.13 to 0.25. All arms share the same batches. +- Sections 2 and 4 are teacher-forced on text the unedited model wrote. Positions + within one generation are correlated. diff --git a/docs/wsp_and_probe_tools.md b/docs/wsp_and_probe_tools.md new file mode 100644 index 00000000..8bafa536 --- /dev/null +++ b/docs/wsp_and_probe_tools.md @@ -0,0 +1,301 @@ +# Weight Edits and Probe Tools + +Two additions that work together: + +- **Probe tools** (`evalrx.probing`): every registered analyzer described as a + tool — what it answers, what it needs, what arguments it takes, and whether + it can run on the model and data in hand. The agentic loop can list them and + call one by name. +- **Weight-space programming** (`evalrx.wsp`): build, apply, control and + inspect direct weight edits such as `w' = w - alpha * u (u^T w)`, audit an + edit against controls (`evalrx.analyzers.weights`), and try one as a + training-free repair at fix tier L4. + +## Probe tools + +### Describe what can run + +```python +from evalrx.probing import ProbeToolbox + +box = ProbeToolbox(model, cases) +report = box.describe() +report["attention_knockout"] +# {"family": "causal", "access": "white_box", "cost": "heavy", "intervenes": True, +# "question": "Which internal route carries the effect, measured by intervening on it?", +# "parameters": {...JSON schema...}, +# "available": False, +# "missing": ["needs a local white-box model that provides: attention, logits"], +# "codes": ["capability"], ...} +``` + +Every probe is listed. An unavailable one says why, with all reasons rather +than the first, so "cannot run here" is never read as "ran and found nothing". +No forward pass is made. The reasons and their codes: + +| Code | Meaning | +|---|---| +| `capability` | The model does not provide a required capability. | +| `modality` | The probe applies to modalities the model does not declare. | +| `data_slot` | The batch fills none of the modality slots the probe reads. | +| `trajectories` | The probe reads agent trajectories; the batch has none. | +| `data_precondition` | The cases lack labels or metadata the probe reads. | +| `injection` | A constructor argument must come from the host (a judge model, a weight edit). | +| `not_implemented` | Registered placeholder. | + +The description is read from the analyzer class: family from its +`analyzers/` location, arguments from its constructor, signals from +`signal_docs`. An analyzer may set `question`, `probe_cost`, `probe_family` or +`probe_intervenes` to override what is derived. + +### Route a question to probes + +```python +route = box.route(failure_modes=["visual_neglect"], max_probes=4) +route.names # ranked: probes mapped to the failure mode first +route.unavailable # requested probes that cannot run, with reasons +print(route.to_prompt()) +``` + +Ranking is `StrategyProbe`'s (slot-composed priority plus failure-mode boosts), +restricted to probes that can run. Filters: `families`, `access` +(`"black_box"` / `"white_box"`), `allow_interventions`, `only_matching`. + +### Call a probe by name + +```python +result = box.call("logit_lens", {"top_k": 5}) # arguments checked against the schema +tools = box.tools() # evalrx.core.tool.Tool objects +tools[0].to_openai_schema() +``` + +A call with an unknown argument, a wrong type, or an unavailable probe raises +`ProbeCallError` before any model work, with the accepted arguments in the +message. Arguments that cannot be written as JSON are never part of a call; +pass a constructed analyzer through `overrides={name: analyzer}`. + +### In the agentic loop + +`AgenticDiagnoseLoop` gains one tool and extends another: + +| Tool | Arguments | Effect | +|---|---|---| +| `list_probes` | `failure_modes`, `families`, `access`, `allow_interventions`, `only_matching`, `max_probes` | Shows the ranked runnable probes with arguments and reasons. Runs no model. Defaults to the failure modes of the current hypotheses. | +| `run_probe` | none | As before: the probe agent selects the analyzers. | +| `run_probe` | `name` + `params`, or `probes: [{name, params}]` | Runs exactly the named probes. Results are added to the evidence gathered so far. | + +A named call runs what was named and nothing else: `max_analyzers` does not cap +it and no substitute is run for a probe that turns out inapplicable. A rejected +call does no model work. When the held-out confirmation re-runs the same probes +on the confirm split, it replays the arguments of the named call. + +## Weight edits + +### Build and apply + +```python +from evalrx.wsp import TensorEdit, WeightEdit, applied, edit_structure + +edit = WeightEdit([ + TensorEdit.rank_one("layers.15.mlp.down_proj.weight", u, alpha=6.0, cols=columns), +]) +edit_structure(model, edit) # size, location, rank, the alpha the change follows + +with applied(model, edit) as report: + answer = model.generate(case.inputs) +# the original weights are back, also if the block raised +``` + +A `TensorEdit` holds one tensor's change in the form it was written in: the +rank-one rule, low-rank factors, a dense delta, or a sparse overwrite. +`alpha = 1` removes the component along `u`, `alpha = 2` reflects it, larger +values reflect and amplify. + +`applied` computes every edited tensor before writing any, so a bad name or +shape leaves the model untouched. `verify_base=True` requires the live tensors +to match the hashes recorded in the edit, and `max_relative_change` refuses an +edit above a size limit. + +Tensor names match a parameter exactly, or by the suffix starting at the +indexed container, so an edit written for +`model.language_model.layers.15.mlp.down_proj.weight` applies to a module that +names it `model.layers.15.mlp.down_proj.weight`. The layer index is never +dropped, and a name that fits two parameters is an error. + +`WeightEdit.save(dir)` / `WeightEdit.load(dir)` persist an edit. +`WeightEdit.from_astra(dir)` reads the `edit.npz` + `manifest.json` layout +published with *Sparks of Weight-Space Programming*. + +### Derive an edit from failing and passing cases + +```python +from evalrx.wsp import contrast_edit + +edit = contrast_edit(model, fail_cases, pass_cases, n_layers=2, n_columns=64, alpha=1.0) +``` + +The direction at a layer is the difference between the mean MLP output on +failing and on passing cases; the neurons are the columns that push hardest +along it on failing cases. No gradient step is taken. This proposes an edit; it +does not show that the edit helps. Activations are read at the last prompt +position, so a failure that only develops during generation is not visible to +it. + +For a failure that only exists during generation, contrast positions instead of +cases. `continuation_contrast_edit` appends each case's recorded output to its +prompt and contrasts positions inside a repeated stretch with positions outside +one: + +```python +from evalrx.wsp import continuation_contrast_edit + +edit = continuation_contrast_edit(model, looping_cases, repeat_ngram=32, n_columns=64, alpha=6.0) +``` + +`collect_continuation_stats` returns the per-layer group means on their own, and +`rank_columns` the direction with each neuron's push, projection and activation +difference, for comparing ranking signals. + +### Compare and re-express edits + +```python +from evalrx.wsp import edit_agreement, fit_rank_one + +rule = fit_rank_one(model, published) # u, columns and alpha that reproduce the edit +rule.metadata["fit"] # alpha and the unexplained share, per tensor +edit_agreement(model, mine, published) # shared tensors, direction cosine, neuron overlap +``` + +An edit stored as overwritten entries cannot be scaled; its rank-one fit can, +which gives a dose curve over `alpha`. Check `rank_one_residual` before treating +the fit as the edit. + +### Audit an edit + +```python +from evalrx.analyzers.weights import WeightEditAudit, WeightEditStructure + +WeightEditStructure(edit=edit).run(model, cases).findings +audit = WeightEditAudit(edit=edit, target="gold", n_seeds=3).run(model, cases).findings +``` + +`weight_edit_audit` measures the edit on the next-token distribution at the end +of the prompt, beside three alternatives on the same cases: + +| Finding | Reading | +|---|---| +| `specificity_random_direction` | Output movement of the edit over that of an equally large change along a random direction. Near 1: the direction is not what matters. | +| `specificity_random_columns` | The same ratio against the same rule on other neurons. Near 1: the choice of neurons is not what matters. | +| `steer_recovery` | Share of the edit's per-case margin change reproduced by a constant activation shift with the weights untouched. Near 1: the edit acts like a fixed bias. | +| `mean_delta_edit_pass`, `frac_flips_pass` | Side effects on cases the model already got right. | + +The controls are random draws; with few draws or few cases a ratio is an +estimate, not a test. Both analyzers are registered, so they appear in the +probe catalog under the `weight_edit` family once the host passes a constructed +instance: + +```python +box = ProbeToolbox(model, cases, overrides={"weight_edit_audit": WeightEditAudit(edit=edit)}) +box.call("weight_edit_audit", {"target": "gold", "n_seeds": 5}) +``` + +### Audit an edit along a generation + +Some edits act only during generation. `weight_edit_continuation_audit` appends +each case's recorded output (`case.observed`) to its prompt and scores every +continuation position on the token that actually came next: + +```python +from evalrx.analyzers.weights import WeightEditContinuationAudit + +findings = WeightEditContinuationAudit( + edit=edit, groups="repetition", repeat_ngram=32, n_seeds=2, +).run(model, cases).findings +``` + +A prediction counts as in-loop when the `repeat_ngram` tokens before the +predicted token already occurred earlier in the sequence, and as productive +otherwise. Per group the findings give the change in negative log-likelihood of +the recorded token (`mean_dnll__`), the KL divergence from the +unedited distribution, `specificity__` and +`steer_recovery_`; `dnll_edit_loop_minus_productive` is positive when the +edit makes looped text less likely than ordinary text. + +This is teacher-forced on text the unedited model wrote. It measures what each +state does to the probability of that text, not what an edited model would +write. Positions within one generation are not independent, so compare the +`per_case` rows before drawing a conclusion from the pooled means. The model +handle must tokenise the prompt exactly as it was when the text was generated. + +`tools/check_astra_edit_qwen35.py` and `tools/audit_astra_edit_qwen35.py` run +the structure check and this audit for the published p50c edit on Qwen3.5-0.8B; +`tools/recover_astra_edit_qwen35.py` derives an edit from the loop contrast and +compares it with the published one; `tools/generate_astra_edit_qwen35.py` lets +the baseline, the edited model and a control generate GSM8K answers through the +L4 fix path. + +### As a repair (fix tier L4) + +```python +agent = FixAgent( + judge=judge, max_tier="L4", score_fn=score, + finetune_pool=diagnosis_cases, # labelled, disjoint from what is validated + weight_edit={"alpha": 1.0, "n_columns": 64, "n_layers": 2}, +) +outcome = agent.propose_and_validate(model, cases, hypotheses) +``` + +`weight_edit` is opt-in (`True` for the defaults, or a dict of +`WeightEditSpec` fields). The candidate `rank_one_weight_edit`: + +- builds its edit from `finetune_pool` with `contrast_edit`, or loads one from + `edit_dir`; a pool that shares cases with the validated batch is refused; +- generates the batch with the edit written in, then restores the weights; +- is validated by the same paired test as every other tier, breaks included; +- also generates the batch under a matched random-direction control and records + both pass counts in `candidate.payload["audit"]`, with the edit's structure. + +Without a pool (and without `edit_dir`) the candidate is recorded with the +reason it did not run. The judge never writes the edit or any hook code. + +## What has and has not been checked + +- The edit arithmetic, the restore contract, the structure read-out and the + controls are tested on small tensors with known answers. +- The contrast heuristic, the audit and the fix path are tested on a hand-built + model with one planted rank-one defect (`tests/toy_wsp.py`): the derived edit + finds the planted layer, neuron and direction, repairs held-out cases, breaks + none, and a matched random direction does not repair them. +- Mechanics (hooks, name resolution, restoration) are tested on tiny + random-weight Qwen2.5-VL and Llama modules. +- The published-layout loader reproduces a re-implementation of the published + rebuild rule bit for bit on synthetic bfloat16 tensors, and reads the + published edit files. It has not been run against the original base + checkpoint. +- On Qwen3.5-0.8B with the published p50c edit, the base and rebuilt tensor + hashes match the manifest and the structure read-out gives 64 columns per + layer, rank one, alpha = 6.000 at layers 15 and 19. +- The continuation audit of p50c on Qwen3.5-0.8B (12 unfinished GSM8K + generations of the unedited model, 48,099 positions, teacher-forced) gives a + change in next-token NLL of +0.079 at loop positions and +0.047 at productive + positions; a matched random direction gives +0.011 / +0.010, the same rule on + other neurons -0.000 / +0.000, and a constant activation shift of the edit's + mean size +0.029 / +0.125. EditLens reports +0.093 / +0.053 for the edit on 6 + texts. The edit is specific and acts more in loops, and it is not free on + ordinary text: the most likely token changes at 4.9% of productive positions. +- Generated through the L4 fix path on 64 GSM8K problems, the same edit takes + the model from 6 to 37 correct, with 31 fixed and 0 broken; a matched random + direction gives 7. +- A loop-vs-productive contrast on that model recovers about half of the + published neurons at the published layers, and neither the direction nor the + layers. No edit PROPOSED by evalrx has been shown to repair a real model. + +Numbers, commands and limits: [validation report](validation/astra_p50c_20261115.md). + +## References + +- Takehi, Teoh, Zhang & Khattab (2026). *Sparks of Weight-Space Programming.* + +- Zhang, Shang & Li (2026). *EditLens: What Did Astra's Edit Actually Do?* + +- VLMProbe: diff --git a/evalrx/analyzers/__init__.py b/evalrx/analyzers/__init__.py index 2aeef9ac..4c5e8d53 100755 --- a/evalrx/analyzers/__init__.py +++ b/evalrx/analyzers/__init__.py @@ -20,6 +20,8 @@ reasoning/ extraction/termination hygiene; arithmetic, self-repair, step-value, knowledge split, contamination — GENERATE + weights/ weight-edit structure / weight-edit audit + (controls + steering comparison) — LOGITS (weights, write) agent/ loop / ignored-obs / first-error / rubric / counterfactual / reliability / tool-SHAP — Trajectory @@ -41,6 +43,7 @@ perturbation, reasoning, uncertainty, + weights, ) from evalrx.analyzers.base import Analyzer, Result diff --git a/evalrx/analyzers/weights/__init__.py b/evalrx/analyzers/weights/__init__.py new file mode 100644 index 00000000..20394262 --- /dev/null +++ b/evalrx/analyzers/weights/__init__.py @@ -0,0 +1,7 @@ +"""Weight-edit analyzers: what a direct weight edit changes, and whether it is specific.""" + +from evalrx.analyzers.weights.continuation_audit import WeightEditContinuationAudit +from evalrx.analyzers.weights.edit_audit import WeightEditAudit +from evalrx.analyzers.weights.edit_structure import WeightEditStructure + +__all__ = ["WeightEditAudit", "WeightEditContinuationAudit", "WeightEditStructure"] diff --git a/evalrx/analyzers/weights/continuation_audit.py b/evalrx/analyzers/weights/continuation_audit.py new file mode 100644 index 00000000..454b34a8 --- /dev/null +++ b/evalrx/analyzers/weights/continuation_audit.py @@ -0,0 +1,332 @@ +"""Weight-edit continuation audit — what an edit does along a recorded generation. + +:mod:`.edit_audit` measures an edit at one place, the end of the prompt. Some +edits act somewhere else: the edit studied in *Sparks of Weight-Space +Programming* does its work inside repetition loops that only appear thousands +of tokens into a generation. This analyzer follows the edit there. + +Each case supplies a prompt (``case.inputs.prompt``) and the text the model +produced for it (``case.observed``). The two are concatenated and run once per +model state, teacher-forced, so every position of the continuation is scored on +the token that actually came next. Per position it records + +* the change in negative log-likelihood of that next token (``dNLL`` > 0: the + state makes the recorded continuation LESS likely), and +* the KL divergence from the unedited next-token distribution, + +under the edit, under matched random-direction and random-neuron controls, and +under a constant activation shift with the weights untouched. + +Positions are then split into two groups. With ``groups="repetition"`` a +prediction counts as in-loop when the ``repeat_ngram`` tokens before the +predicted token have already occurred earlier in the sequence, and as +productive otherwise; prompt positions are never scored. Reading the result: + +* an edit that breaks loops raises the NLL of the repeated continuation at loop + positions and leaves productive positions alone + (``dnll_edit_loop_minus_productive`` well above zero); +* ``specificity_*`` near 1 in a group means a matched random change moves the + output there as much as the edit does; +* ``steer_recovery_*`` near 1 means a constant activation shift reproduces the + edit's per-position effect in that group. + +Limits. This is teacher-forced on text the UNEDITED model wrote: it says what +the edit does to the probability of that text, not what the edited model would +write instead. The prompt is tokenised by the model handle and the continuation +is appended as plain tokens, so the handle must be set up to produce exactly +the prompt the generation was made from (chat template included). Text-only +cases only. Memory: one pass holds two ``[tokens, vocab]`` logit tensors. + +References: +- Sparks of Weight-Space Programming — Takehi, Teoh, Zhang & Khattab, 2026. +- EditLens: What Did Astra's Edit Actually Do? — Zhang, Shang & Li, 2026. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from evalrx.core.analyzer import Analyzer +from evalrx.core.capability import Capability +from evalrx.core.registry import register_analyzer +from evalrx.core.result import Result +from evalrx.wsp.continuation import GROUPINGS, repetition_mask + +if TYPE_CHECKING: + from evalrx.core.case import CaseBatch + from evalrx.core.model import Model + +# Re-exported: the grouping rule lives with the sequence builder so that the +# audit and the continuation contrast label positions identically. +__all__ = ["GROUPINGS", "WeightEditContinuationAudit", "repetition_mask"] + + +@register_analyzer("weight_edit_continuation_audit") +class WeightEditContinuationAudit(Analyzer): + """Follow a weight edit along recorded generations: loop vs productive positions, against controls. + + Args: + edit: The :class:`~evalrx.wsp.edit.WeightEdit` to audit. + groups: ``"repetition"`` splits scored positions into ``loop`` and + ``productive`` (see module docstring); ``"none"`` keeps + them in one group, ``all``. + repeat_ngram: Window length of the repetition test. + controls: Control kinds to run (see :mod:`evalrx.wsp.controls`). + n_seeds: Random draws per control kind. + steering: Whether to run the constant-shift steering control. + max_tokens: Truncate prompt + continuation to this many tokens. + chunk: Positions scored at a time (bounds the float32 memory). + seed: Base seed of the control draws. + max_cases: Analyse at most this many cases (0 = all). + """ + + name = "weight_edit_continuation_audit" + requires = frozenset({Capability.LOGITS}) + question = ("Where along a generation does the edit act, and does a matched random " + "change act there too?") + signal_docs = { + "wc_frac_loop": ( + "Share of loop positions", + "Share of this generation's scored positions that sit inside a repeated " + "stretch. High means the model was looping.", + ), + "wc_dnll_edit_loop": ( + "Loop text made less likely", + "How much less likely the edit makes the recorded next token at positions " + "inside a repeated stretch (change in negative log-likelihood). Positive " + "means the edit pushes the model off the loop.", + ), + "wc_dnll_edit_productive": ( + "Normal text made less likely", + "The same change at positions outside any repeated stretch. Near zero means " + "the edit leaves ordinary text alone; positive is a side effect.", + ), + "wc_kl_edit_loop": ( + "Output shift in loops", + "How far the edit moves the next-token distribution at loop positions " + "(KL divergence from the unedited model).", + ), + "wc_kl_edit_productive": ( + "Output shift elsewhere", + "How far the edit moves the next-token distribution at positions outside " + "loops. Large values mean the edit also changes ordinary text.", + ), + } + + def __init__(self, edit: "Any", groups: str = "repetition", repeat_ngram: int = 32, + controls: "tuple[str, ...]" = ("random_direction", "random_columns"), + n_seeds: int = 1, steering: bool = True, max_tokens: int = 4096, + chunk: int = 256, seed: int = 0, max_cases: int = 0) -> None: + super().__init__(edit=edit, groups=groups, repeat_ngram=repeat_ngram, + controls=tuple(controls), n_seeds=n_seeds, steering=steering, + max_tokens=max_tokens, chunk=chunk, seed=seed, max_cases=max_cases) + + # ------------------------------------------------------------------ + def _run(self, model: "Model", cases: "CaseBatch") -> Result: + import torch + + from evalrx.wsp.continuation import encode_continuations + from evalrx.wsp.controls import CONTROL_KINDS, build_control + from evalrx.wsp.direction import output_shift + from evalrx.wsp.edit import applied, module_of, resolve_parameters + from evalrx.wsp.structure import edit_structure + + if self.groups not in GROUPINGS: + raise ValueError(f"{self.name}: groups must be one of {GROUPINGS}, got {self.groups!r}") + unknown = [k for k in self.controls if k not in CONTROL_KINDS] + if unknown: + raise ValueError(f"{self.name}: unknown control(s) {unknown}; choose from {CONTROL_KINDS}") + if int(self.n_seeds) < 1 or int(self.repeat_ngram) < 1 or int(self.chunk) < 1: + raise ValueError(f"{self.name}: n_seeds, repeat_ngram and chunk must be positive") + if int(self.max_tokens) < 3: + raise ValueError(f"{self.name}: max_tokens must be at least 3") + module = module_of(model) + resolved = resolve_parameters(module, self.edit) + device = next(module.parameters()).device + group_names = ("loop", "productive") if self.groups == "repetition" else ("all",) + + def logits_of(ids: "list[int]") -> Any: + with torch.no_grad(): + out = module(input_ids=torch.tensor([ids], device=device)) + return getattr(out, "logits", out)[0] + + # ---- Pass 1: build each case's token sequence and group labels ------------- + sequences, skipped = encode_continuations( + model, cases.stratified_head(self.max_cases), groups=self.groups, + repeat_ngram=int(self.repeat_ngram), max_tokens=int(self.max_tokens)) + items = [{"case": q.case, "ids": q.ids, "targets": q.targets, "labels": q.labels} + for q in sequences] + if not items: + raise ValueError(f"{self.name}: no case could be analysed (skipped: {dict(skipped)})") + + # ---- Controls, built once against the unedited weights --------------------- + states: list[tuple[str, Any]] = [("edit", self.edit)] + control_errors: dict[str, str] = {} + for kind in self.controls: + for draw in range(int(self.n_seeds)): + try: + states.append((kind, build_control(model, self.edit, kind, + seed=int(self.seed) + draw))) + except ValueError as exc: + control_errors[kind] = str(exc) + break + + # ---- Steering shift: the edit's mean effect over all scored positions ------- + owners = {f"{n}.weight": m for n, m in module.named_modules() + if isinstance(m, torch.nn.Linear)} + steerable = bool(self.steering) and all(key in owners for key, _ in resolved) + shifts: "dict[str, Any] | None" = None + if steerable: + act_sum: dict[str, Any] = {} + current: dict[str, Any] = {} + + def make(key: str): + def hook(_m: Any, args: Any) -> None: + x = args[0].detach()[0].double() # [tokens, d_in] + total = x[current["positions"]].sum(dim=0).cpu() + act_sum[key] = total if key not in act_sum else act_sum[key] + total + return hook + + handles = [owners[key].register_forward_pre_hook(make(key)) for key, _ in resolved] + n_scored = 0 + try: + for item in items: + # The input that produces the prediction of target i sits at i - 1. + current["positions"] = torch.tensor([i - 1 for i in item["targets"]], + device=device) + logits_of(item["ids"]) + n_scored += len(item["targets"]) + finally: + for handle in handles: + handle.remove() + delta = self.edit.as_delta(module) + shifts = {t.name: torch.from_numpy(t.delta).double() @ (act_sum[t.name] / n_scored) + for t in delta.tensors} + + # ---- Pass 2: per case, the clean run and every state ------------------------ + def score(clean: Any, other: Any, item: dict[str, Any]) -> "tuple[Any, Any, Any, Any]": + """Per target: clean NLL, NLL change, KL from clean, top-1 changed.""" + pos = torch.tensor([i - 1 for i in item["targets"]], device=device) + tgt = torch.tensor([item["ids"][i] for i in item["targets"]], device=device) + out = [[], [], [], []] + for a in range(0, len(pos), int(self.chunk)): + p, t = pos[a:a + int(self.chunk)], tgt[a:a + int(self.chunk)] + lp0 = torch.log_softmax(clean[p].float(), dim=-1) + lp1 = torch.log_softmax(other[p].float(), dim=-1) + nll0 = -lp0.gather(1, t[:, None])[:, 0] + nll1 = -lp1.gather(1, t[:, None])[:, 0] + out[0].append(nll0.double().cpu()) + out[1].append((nll1 - nll0).double().cpu()) + out[2].append((lp0.exp() * (lp0 - lp1)).sum(dim=-1).clamp_min(0).double().cpu()) + out[3].append((lp0.argmax(dim=-1) != lp1.argmax(dim=-1)).cpu()) + return tuple(torch.cat(part) for part in out) + + state_names = ["edit"] + [k for k in self.controls if k not in control_errors] + if shifts is not None: + state_names.append("steer") + # pooled[state][group] -> lists of per-position tensors across cases + pooled: dict[str, dict[str, dict[str, list]]] = { + s: {g: {"dnll": [], "kl": [], "flip": []} for g in group_names} for s in state_names} + clean_nll: dict[str, list] = {g: [] for g in group_names} + per_case: list[dict[str, Any]] = [] + applied_report: "dict[str, Any] | None" = None + + for item in items: + clean = logits_of(item["ids"]) + index = {g: torch.tensor([j for j, lab in enumerate(item["labels"]) if lab == g], + dtype=torch.long) for g in group_names} + row: dict[str, Any] = {"sample_id": item["case"].id, + "wc_n_positions": len(item["targets"])} + if self.groups == "repetition": + row["wc_frac_loop"] = round(len(index["loop"]) / len(item["targets"]), 4) + by_state: dict[str, list] = {} + for label, variant in states: + with applied(model, variant) as report: + if label == "edit" and applied_report is None: + applied_report = report.to_dict() + by_state.setdefault(label, []).append(score(clean, logits_of(item["ids"]), item)) + if shifts is not None: + with output_shift(module, shifts): + by_state["steer"] = [score(clean, logits_of(item["ids"]), item)] + del clean + + first = by_state["edit"][0] + for g in group_names: + if len(index[g]): + clean_nll[g].append(first[0][index[g]]) + row[f"wc_nll_clean_{g}"] = round(float(first[0][index[g]].mean()), 6) + for label, draws in by_state.items(): + # Average the draws position by position, then split by group. + dnll = torch.stack([d[1] for d in draws]).mean(dim=0) + kl = torch.stack([d[2] for d in draws]).mean(dim=0) + flip = torch.stack([d[3].double() for d in draws]).mean(dim=0) + for g in group_names: + if not len(index[g]): + continue + pooled[label][g]["dnll"].append(dnll[index[g]]) + pooled[label][g]["kl"].append(kl[index[g]]) + pooled[label][g]["flip"].append(flip[index[g]]) + row[f"wc_dnll_{label}_{g}"] = round(float(dnll[index[g]].mean()), 6) + row[f"wc_kl_{label}_{g}"] = round(float(kl[index[g]].mean()), 9) + per_case.append(row) + + # ---- Summaries over positions ------------------------------------------------ + def cat(parts: list) -> Any: + return torch.cat(parts) if parts else torch.zeros(0, dtype=torch.float64) + + findings: dict[str, Any] = { + "edit": self.edit.name, + "n_cases_analyzed": len(items), + "n_skipped": sum(skipped.values()), + "skip_reasons": dict(skipped), + "groups": self.groups, + "repeat_ngram": int(self.repeat_ngram) if self.groups == "repetition" else None, + "max_tokens": int(self.max_tokens), + "n_control_draws": int(self.n_seeds), + "max_relative_change": (applied_report or {}).get("max_relative_change"), + "n_positions": sum(len(i["targets"]) for i in items), + } + for g in group_names: + nll0 = cat(clean_nll[g]) + findings[f"n_positions_{g}"] = int(nll0.numel()) + if not nll0.numel(): + continue + findings[f"mean_nll_clean_{g}"] = float(nll0.mean()) + edit_dnll = cat(pooled["edit"][g]["dnll"]) + edit_kl = float(cat(pooled["edit"][g]["kl"]).mean()) + for label in state_names: + dnll = cat(pooled[label][g]["dnll"]) + kl = float(cat(pooled[label][g]["kl"]).mean()) + findings[f"mean_dnll_{label}_{g}"] = float(dnll.mean()) + findings[f"mean_kl_{label}_{g}"] = kl + findings[f"frac_flips_{label}_{g}"] = round( + float(cat(pooled[label][g]["flip"]).mean()), 4) + if label in self.controls: + # Output movement of the edit over that of the matched control. + findings[f"specificity_{label}_{g}"] = round(edit_kl / kl, 4) if kl > 0 else None + if label == "steer": + total = float((edit_dnll ** 2).sum()) + findings[f"steer_recovery_{g}"] = ( + round(1.0 - float(((edit_dnll - dnll) ** 2).sum()) / total, 4) + if total > 0 else None) + if self.groups == "repetition" and all( + f"mean_dnll_edit_{g}" in findings for g in group_names): + findings["dnll_edit_loop_minus_productive"] = ( + findings["mean_dnll_edit_loop"] - findings["mean_dnll_edit_productive"]) + if control_errors: + findings["controls_not_run"] = control_errors + if self.steering and shifts is None: + findings["steering_not_run"] = ( + "an edited tensor is not the weight of a linear layer, so its input " + "cannot be read to size the shift") + findings["structure"] = edit_structure(model, self.edit)["tensors"] + findings["per_case"] = per_case + findings["_caveat"] = ( + "Teacher-forced on text the unedited model wrote: this measures what each " + "state does to the probability of that text, not what an edited model would " + "write. Positions within one generation are not independent, so the " + "per-position means carry less evidence than their count suggests; compare " + "across cases (per_case) before drawing a conclusion. Controls are random " + "draws." + ) + return Result(analyzer=self.name, model=repr(model), cases=cases, findings=findings) diff --git a/evalrx/analyzers/weights/edit_audit.py b/evalrx/analyzers/weights/edit_audit.py new file mode 100644 index 00000000..d59539b8 --- /dev/null +++ b/evalrx/analyzers/weights/edit_audit.py @@ -0,0 +1,293 @@ +"""Weight-edit audit — does the edit change behaviour, and is the change specific? + +An edit that moves the model's answers has not yet been explained: a change of +the same size along a random direction, or the same rule on other neurons, may +move them just as much, and a constant shift of the activations may do the same +job without touching the weights. This analyzer measures the edit beside those +alternatives on the same cases and the same scalar. + +Per case, at the last prompt position (where the first answer token is +predicted), it records the next-token margin (see +:mod:`evalrx.analyzers._internals.targets`) and the KL divergence from the +unedited distribution under: + +* the edit itself; +* ``random_direction`` controls — same neurons and change size, random direction; +* ``random_columns`` controls — same rule, other neurons; +* a steering control — weights untouched, a constant vector added to each + edited layer's output equal to the edit's mean effect there. + +Reading the result: + +* ``specificity_*`` near 1 means a matched random change moves the output as + much as the edit: the direction (or the neurons) is not what matters. +* ``steer_recovery`` near 1 means a constant activation shift reproduces the + edit's effect on the margin: the edit acts like an input-independent bias. +* with labels, ``mean_delta_edit_pass`` and ``frac_flips_pass`` are the side + effects on cases the model already got right. + +Everything is measured teacher-forced on one position. An edit whose effect +only appears during generation (for example on repetition loops) needs a +generation-level evaluation, which the fix stage provides. + +References: +- Sparks of Weight-Space Programming — Takehi, Teoh, Zhang & Khattab, 2026. +- EditLens: What Did Astra's Edit Actually Do? — Zhang, Shang & Li, 2026. +""" + +from __future__ import annotations + +from collections import Counter +from typing import TYPE_CHECKING, Any + +from evalrx.core.analyzer import Analyzer +from evalrx.core.capability import Capability +from evalrx.core.registry import register_analyzer +from evalrx.core.result import Result + +if TYPE_CHECKING: + from evalrx.core.case import CaseBatch + from evalrx.core.model import Model + + +@register_analyzer("weight_edit_audit") +class WeightEditAudit(Analyzer): + """Measure a weight edit against random-direction, random-neuron and steering controls. + + Args: + edit: The :class:`~evalrx.wsp.edit.WeightEdit` to audit. + target: ``"prediction"`` (the clean run's top-1 vs top-2 token; label + free) or ``"gold"`` (the gold answer's first token vs the + strongest other token; a positive change moves toward gold). + controls: Control kinds to run (see :mod:`evalrx.wsp.controls`). + n_seeds: Random draws per control kind; their per-case results are + averaged. + steering: Whether to run the constant-shift steering control. + seed: Base seed of the control draws. + max_cases: Analyse at most this many cases (0 = all), stratified by label. + """ + + name = "weight_edit_audit" + requires = frozenset({Capability.LOGITS}) + question = "Does the edit change the answer, and more than a matched random change would?" + signal_docs = { + "we_delta_edit": ( + "Margin change (edit)", + "Change in the next-token margin caused by the edit. With the gold target, " + "positive means the edit moved the model toward the right answer.", + ), + "we_kl_edit": ( + "Output shift (edit)", + "How far the edit moves the next-token distribution from the unedited one " + "(KL divergence). Zero means the edit did nothing on this case.", + ), + "we_flips_top1": ( + "Top answer changed", + "Whether the edit changed the model's most likely next token on this case.", + ), + "we_kl_random_direction": ( + "Output shift (random dir.)", + "The same measurement for an equally large change along a random direction. " + "If it is as large as the edit's, the edit's direction is not what matters.", + ), + "we_kl_random_columns": ( + "Output shift (other neurons)", + "The same measurement for the same rule applied to other neurons. If it is as " + "large as the edit's, the choice of neurons is not what matters.", + ), + "we_delta_steer": ( + "Margin change (steering)", + "Change in the margin when the weights are left alone and a constant vector is " + "added to the edited layers' outputs instead. Close to the edit's own change " + "means the edit behaves like a fixed bias.", + ), + } + + def __init__(self, edit: "Any", target: str = "prediction", + controls: "tuple[str, ...]" = ("random_direction", "random_columns"), + n_seeds: int = 3, steering: bool = True, seed: int = 0, + max_cases: int = 0) -> None: + super().__init__(edit=edit, target=target, controls=tuple(controls), n_seeds=n_seeds, + steering=steering, seed=seed, max_cases=max_cases) + + def _run(self, model: "Model", cases: "CaseBatch") -> Result: + import torch + + from evalrx.analyzers._internals.targets import build_margin + from evalrx.core.case import Label + from evalrx.core.model import CaptureSpec + from evalrx.wsp.controls import CONTROL_KINDS, build_control + from evalrx.wsp.direction import output_shift + from evalrx.wsp.edit import applied, module_of, resolve_parameters + from evalrx.wsp.structure import edit_structure + + unknown = [k for k in self.controls if k not in CONTROL_KINDS] + if unknown: + raise ValueError(f"{self.name}: unknown control(s) {unknown}; choose from {CONTROL_KINDS}") + if int(self.n_seeds) < 1: + raise ValueError(f"{self.name}: n_seeds must be at least 1") + module = module_of(model) + resolved = resolve_parameters(module, self.edit) + spec = CaptureSpec(to_cpu=False) + + def last_logits(inputs: Any) -> Any: + trace = model.forward(inputs, capture={Capability.LOGITS}, spec=spec) + return trace.require(Capability.LOGITS)[-1:].detach().float().cpu() + + # Inputs of the edited linear layers at the last position, for the + # steering control: its shift is the edit's mean effect dW @ mean(input). + owners = {f"{n}.weight": m for n, m in module.named_modules() + if isinstance(m, torch.nn.Linear)} + steerable = bool(self.steering) and all(key in owners for key, _ in resolved) + act_sum: dict[str, Any] = {} + handles = [] + if steerable: + def make(key: str): + def hook(_m: Any, args: Any) -> None: + x = args[0].detach()[0, -1].double().cpu() + act_sum[key] = x if key not in act_sum else act_sum[key] + x + return hook + handles = [owners[key].register_forward_pre_hook(make(key)) for key, _ in resolved] + + # Pass 1: unedited logits and the margin each case is scored on. + kept: list[dict[str, Any]] = [] + skipped: Counter = Counter() + try: + for case in cases.stratified_head(self.max_cases): + try: + clean = last_logits(case.inputs) + except Exception: # noqa: BLE001 - an unreadable case is counted, not fatal + skipped["forward_failed"] += 1 + continue + margin, why = build_margin(self.target, clean, model=model, case=case) + if margin is None: + skipped[why] += 1 + continue + kept.append({"case": case, "clean": clean, "margin": margin, + "m0": margin.value(clean), "top1": int(clean[-1].argmax())}) + finally: + for handle in handles: + handle.remove() + if not kept: + raise ValueError(f"{self.name}: no case could be analysed (skipped: {dict(skipped)})") + n = len(kept) + + def measure(run: Any) -> "list[tuple[float, float, bool]]": + """(margin change, KL from clean, top-1 flipped) per kept case under *run*.""" + out = [] + for item in kept: + logits = run(item["case"].inputs) + p0 = torch.log_softmax(item["clean"][-1].double(), dim=-1) + p1 = torch.log_softmax(logits[-1].double(), dim=-1) + kl = float((p0.exp() * (p0 - p1)).sum().clamp_min(0.0)) + out.append((item["margin"].value(logits) - item["m0"], kl, + item["margin"].top1_changed(logits, item["top1"]))) + return out + + with applied(model, self.edit) as report: + applied_report = report.to_dict() + edited = measure(last_logits) + + control_rows: dict[str, list[list[tuple[float, float, bool]]]] = {} + control_errors: dict[str, str] = {} + for kind in self.controls: + for draw in range(int(self.n_seeds)): + try: + control = build_control(model, self.edit, kind, seed=int(self.seed) + draw) + except ValueError as exc: + control_errors[kind] = str(exc) + break + with applied(model, control): + control_rows.setdefault(kind, []).append(measure(last_logits)) + + steered = None + if steerable: + delta = self.edit.as_delta(module) + shifts = { + t.name: torch.from_numpy(t.delta).double() @ (act_sum[t.name] / n) + for t in delta.tensors if t.name in act_sum + } + with output_shift(module, shifts): + steered = measure(last_logits) + + per_case: list[dict[str, Any]] = [] + for i, item in enumerate(kept): + d_edit, kl_edit, flip = edited[i] + row: dict[str, Any] = { + "sample_id": item["case"].id, + "we_margin_clean": round(item["m0"], 4), + "we_delta_edit": round(d_edit, 6), + "we_kl_edit": round(kl_edit, 9), + "we_flips_top1": bool(flip), + } + for kind, draws in control_rows.items(): + row[f"we_delta_{kind}"] = round(sum(d[i][0] for d in draws) / len(draws), 6) + row[f"we_kl_{kind}"] = round(sum(d[i][1] for d in draws) / len(draws), 9) + if steered is not None: + row["we_delta_steer"] = round(steered[i][0], 6) + per_case.append(row) + + # Summaries are computed from the unrounded measurements: the rows are + # rounded for reading, and a ratio of two rounded small numbers is noise. + def avg(values: "list[float]") -> float: + return sum(values) / len(values) + + def control_mean(kind: str, field: int) -> float: + draws = control_rows[kind] + return avg([avg([d[i][field] for d in draws]) for i in range(n)]) + + kl_edit_mean = avg([e[1] for e in edited]) + findings: dict[str, Any] = { + "edit": self.edit.name, + "n_cases_analyzed": n, + "n_skipped": sum(skipped.values()), + "skip_reasons": dict(skipped), + "target": self.target, + "max_relative_change": applied_report["max_relative_change"], + "mean_delta_edit": avg([e[0] for e in edited]), + "mean_kl_edit": kl_edit_mean, + "frac_flips_top1": round(sum(bool(e[2]) for e in edited) / n, 4), + "n_control_draws": int(self.n_seeds), + } + for kind in control_rows: + kl_control = control_mean(kind, 1) + findings[f"mean_kl_{kind}"] = kl_control + findings[f"mean_delta_{kind}"] = control_mean(kind, 0) + # Ratio of output movement: edit over matched control. + findings[f"specificity_{kind}"] = ( + round(kl_edit_mean / kl_control, 4) if kl_control > 0 else None + ) + if control_errors: + findings["controls_not_run"] = control_errors + if steered is not None: + d_e = [e[0] for e in edited] + d_s = [x[0] for x in steered] + total = sum(x * x for x in d_e) + findings["mean_delta_steer"] = avg(d_s) + # 1 = the constant shift reproduces every per-case change; 0 = it + # explains none of it; negative = it moves cases the other way. + findings["steer_recovery"] = ( + round(1.0 - sum((a - b) ** 2 for a, b in zip(d_e, d_s)) / total, 4) + if total > 0 else None + ) + elif self.steering: + findings["steering_not_run"] = ( + "an edited tensor is not the weight of a linear layer, so its input " + "cannot be read to size the shift" + ) + for label, tag in ((Label.FAIL, "fail"), (Label.PASS, "pass")): + rows = [r for r, item in zip(per_case, kept) if item["case"].label == label] + if rows: + findings[f"n_{tag}"] = len(rows) + findings[f"mean_delta_edit_{tag}"] = avg([r["we_delta_edit"] for r in rows]) + findings[f"frac_flips_{tag}"] = round( + sum(bool(r["we_flips_top1"]) for r in rows) / len(rows), 4) + findings["structure"] = edit_structure(model, self.edit)["tensors"] + findings["per_case"] = per_case + findings["_caveat"] = ( + "Measured teacher-forced on the next-token distribution at the end of the " + "prompt. Controls are random draws: with few draws or few cases a specificity " + "ratio is an estimate, not a test. Effects that only appear during generation " + "are not visible here." + ) + return Result(analyzer=self.name, model=repr(model), cases=cases, findings=findings) diff --git a/evalrx/analyzers/weights/edit_structure.py b/evalrx/analyzers/weights/edit_structure.py new file mode 100644 index 00000000..264af3a7 --- /dev/null +++ b/evalrx/analyzers/weights/edit_structure.py @@ -0,0 +1,87 @@ +"""Weight-edit structure — describe an edit from the weights alone. + +No forward pass and no cases are needed: the analyzer applies the edit to a +copy of each tensor and reports where it moved, how large the move is, whether +it is low rank, and — for a rank-one change — the strength ``alpha`` of the +rule ``w' = w - alpha u (u^T w)`` that reproduces it. This is the first step of +auditing any edit, because every later explanation has to be an explanation of +THIS change. + +Reference: +- Sparks of Weight-Space Programming — Takehi, Teoh, Zhang & Khattab, 2026. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from evalrx.core.analyzer import Analyzer +from evalrx.core.capability import Capability +from evalrx.core.registry import register_analyzer +from evalrx.core.result import Result + +if TYPE_CHECKING: + from evalrx.core.case import CaseBatch + from evalrx.core.model import Model + + +@register_analyzer("weight_edit_structure") +class WeightEditStructure(Analyzer): + """Describe a weight edit from the weights: size, location, rank, and the rank-one rule it follows. + + Args: + edit: The :class:`~evalrx.wsp.edit.WeightEdit` to describe. + verify_base: Require the live tensors to match the hashes recorded in + the edit (an edit that overwrites entries is only + meaningful on the weights it was made on). + top_singular: How many leading singular values to report per tensor. + """ + + name = "weight_edit_structure" + # LOGITS stands in for "a local model whose parameters can be read": the + # capability vocabulary has no entry for parameter access, and every + # backend that provides full logits holds the module in process. + requires = frozenset({Capability.LOGITS}) + question = "What exactly did the edit change in the weights?" + probe_cost = "light" + probe_intervenes = False + signal_docs = { + "max_relative_change": ( + "Edit size", + "Largest relative change of any edited tensor (norm of the change over norm of " + "the original). Larger means a more invasive edit.", + ), + "max_rank_one_residual": ( + "Rank-one residual", + "Share of the change NOT explained by a single direction; 0 means the edit is " + "exactly rank one, higher means it has more structure than one direction.", + ), + } + + def __init__(self, edit: "Any", verify_base: bool = False, top_singular: int = 8) -> None: + super().__init__(edit=edit, verify_base=verify_base, top_singular=top_singular) + + def _run(self, model: "Model", cases: "CaseBatch") -> Result: + from evalrx.wsp.edit import applied + from evalrx.wsp.structure import edit_structure + + structure = edit_structure(model, self.edit, top_singular=self.top_singular) + # Entering the context proves the edit resolves, fits and restores. + with applied(model, self.edit, verify_base=self.verify_base) as report: + applied_report = report.to_dict() + rows = structure["tensors"] + residuals = [r["rank_one_residual"] for r in rows if "rank_one_residual" in r] + alphas = [r["alpha_estimate"] for r in rows if "alpha_estimate" in r] + findings: dict[str, Any] = { + "edit": self.edit.name, + "n_tensors": len(rows), + "changed_entries_total": sum(r.get("changed_entries", 0) for r in rows), + "max_relative_change": applied_report["max_relative_change"], + "max_rank_one_residual": max(residuals) if residuals else None, + "alpha_estimates": [round(a, 4) for a in alphas], + "base_verified": bool(self.verify_base), + "tensors": rows, + "direction_cosines": structure["direction_cosines"], + "edit_metadata": self.edit.metadata, + } + return Result(analyzer=self.name, model=repr(model), cases=cases, findings=findings) diff --git a/evalrx/eval_agent/__init__.py b/evalrx/eval_agent/__init__.py index 39cefaf1..8c08306d 100755 --- a/evalrx/eval_agent/__init__.py +++ b/evalrx/eval_agent/__init__.py @@ -79,6 +79,9 @@ FinetuneSpec + run_lora_repair (v1: LoRA on target="llm" only, trained on a caller-supplied finetune_pool; other recipe shapes recorded only) + weight_edit.py L4 training-free weight edit: WeightEditSpec + + run_weight_edit_repair (opt-in via + FixAgent(weight_edit=...); matched control recorded) repair_catalog.py discovery of pre-audited model repair capabilities hypothesis_tester.py M4 — HypothesisTester: statistical test + protocol consistency; stopping_criteria_met() drives the VLDiagnoseLoop exit @@ -170,6 +173,7 @@ from evalrx.eval_agent.stages.fix.tiers import FixTier, parse_tier, route_min_tier from evalrx.eval_agent.stages.fix.tools import PipelineSpec from evalrx.eval_agent.stages.fix.types import FixCandidate, FixOutcome, FixValidation +from evalrx.eval_agent.stages.fix.weight_edit import WeightEditSpec from evalrx.eval_agent.stages.hypothesis_tester import HypothesisTester, HypothesisTestResult from evalrx.eval_agent.stages.probe import ModelKind, StrategyProbe from evalrx.eval_agent.stages.probe_agent import ProbeAgent @@ -213,6 +217,7 @@ "INTERNALS_PRIMITIVES", "InternalsPrimitive", "FinetuneSpec", + "WeightEditSpec", # M2 "AnalysisModule", "AnalysisReport", diff --git a/evalrx/eval_agent/agentic/board.py b/evalrx/eval_agent/agentic/board.py index 55ff872a..fd4eadbf 100644 --- a/evalrx/eval_agent/agentic/board.py +++ b/evalrx/eval_agent/agentic/board.py @@ -48,6 +48,8 @@ class EvidenceBoard: protocol_summary: str = "" data_summary: dict[str, Any] = field(default_factory=dict) probe_findings: list[dict[str, Any]] = field(default_factory=list) + #: Rendered output of the last ``list_probes`` call (names, arguments, reasons). + probe_catalog: str = "" stats_findings: list[dict[str, Any]] = field(default_factory=list) stats_confirmatory: bool = False explore_takeaways: list[dict[str, Any]] = field(default_factory=list) @@ -74,6 +76,8 @@ def to_prompt(self, registry: "Any") -> str: if self.data_summary: lines.append(f"DATA: {self.data_summary}\n") + if self.probe_catalog: + lines.append(f"PROBE CATALOG (from list_probes):\n{self.probe_catalog}\n") lines.append(f"PROBE FINDINGS (M1): {self.probe_findings or 'none yet'}") lines.append( f"STATS FINDINGS (M2, confirmatory={self.stats_confirmatory}): " diff --git a/evalrx/eval_agent/agentic/tools.py b/evalrx/eval_agent/agentic/tools.py index 0e5b4fc4..e049d038 100644 --- a/evalrx/eval_agent/agentic/tools.py +++ b/evalrx/eval_agent/agentic/tools.py @@ -188,18 +188,154 @@ def _current_report(loop: Any, state: _RunState) -> "VLDiagnoseReport": ) -def _run_probe(loop: Any, state: _RunState) -> "Callable[[EvidenceBoard, dict], ToolOutcome]": +#: Full M1 passes (``run_probe`` with no probe named) allowed per run. +_MAX_UNNAMED_PROBE_RUNS = 3 + + +def _probe_overrides(loop: Any) -> "dict[str, Any]": + """Constructed analyzers the host configured on the loop's probe agent.""" + return dict(getattr(getattr(loop, "probe_agent", None), "_overrides", None) or {}) + + +def _requested_probes(params: "dict[str, Any]") -> "list[tuple[str, dict[str, Any]]] | str": + """Read ``run_probe`` arguments as ``[(name, params), ...]``. + + Accepts ``{"name": ..., "params": {...}}`` for one probe and + ``{"probes": [{"name": ..., "params": {...}}, ...]}`` for several. Returns + an error string when the shape is wrong, an empty list when the call names + no probe (the automatic-selection form). + """ + entries: list[Any] = [] + if params.get("probes") is not None: + if not isinstance(params["probes"], list): + return "params.probes must be a list of {name, params} objects" + entries.extend(params["probes"]) + if params.get("name") is not None: + entries.append({"name": params["name"], "params": params.get("params")}) + out: list[tuple[str, dict[str, Any]]] = [] + for entry in entries: + if isinstance(entry, str): + entry = {"name": entry} + if not isinstance(entry, dict) or not str(entry.get("name") or "").strip(): + return "each probe needs a name" + args = entry.get("params") or {} + if not isinstance(args, dict): + return f"params for {entry['name']!r} must be an object" + out.append((str(entry["name"]).strip(), dict(args))) + return out + + +def _list_probes(loop: Any, state: _RunState) -> "Callable[[EvidenceBoard, dict], ToolOutcome]": def handler(board: "EvidenceBoard", params: dict[str, Any]) -> ToolOutcome: - probe_results, artifact_pngs = loop._do_m1( - state.cycle, state.data, state.all_hypotheses, state.timings + from evalrx.probing import ProbeRouter + + modes = params.get("failure_modes") or [] + if isinstance(modes, str): + modes = [modes] + if not modes: + # Default to what the investigation currently suspects. + modes = [h.get("failure_mode") for h in board.hypotheses if h.get("failure_mode")] + families = params.get("families") + if isinstance(families, str): + families = [families] + router = ProbeRouter( + selector=getattr(getattr(loop, "probe_agent", None), "selector", None), + overrides=_probe_overrides(loop), ) + route = router.route( + loop.model, state.data, + failure_modes=modes or None, + families=families or None, + access=params.get("access"), + allow_interventions=bool(params.get("allow_interventions", True)), + only_matching=bool(params.get("only_matching", False)), + max_probes=int(params.get("max_probes", 12)), + ) + board.probe_catalog = route.to_prompt() + summary = f"{len(route.selected)} runnable probe(s)" + if route.unavailable: + summary += f", {len(route.unavailable)} requested but unavailable" + if not route.selected: + return ToolOutcome(False, summary + "; widen the filters", payload=route, error="empty") + return ToolOutcome(True, summary + ": " + ", ".join(route.names), payload=route) + + return handler + + +def _run_probe(loop: Any, state: _RunState) -> "Callable[[EvidenceBoard, dict], ToolOutcome]": + def handler(board: "EvidenceBoard", params: dict[str, Any]) -> ToolOutcome: + requested = _requested_probes(params or {}) + if isinstance(requested, str): + return ToolOutcome(False, requested, error="bad_params") + if not requested: + # No probe named: M1 selects (judge-guided or static), as before, + # and under the cap a full pass always had. The tool's own cap is + # higher only to leave room for the narrower named calls. + unnamed = sum( + 1 for a in board.action_log + if a.get("tool") == "run_probe" and a.get("ok") + and not _requested_probes(a.get("params") or {}) + ) + if unnamed >= _MAX_UNNAMED_PROBE_RUNS: + return ToolOutcome( + False, + f"run_probe without a probe name has reached its limit " + f"({_MAX_UNNAMED_PROBE_RUNS}); name the probes to run (see list_probes)", + error="max_calls", + ) + probe_results, artifact_pngs = loop._do_m1( + state.cycle, state.data, state.all_hypotheses, state.timings + ) + if not probe_results: + return ToolOutcome(False, "M1 produced no probe results", error="empty") + state.probe_results = probe_results + state.artifact_pngs = artifact_pngs + board.probe_findings = _summarize_probe(probe_results) + summary = f"ran {len(probe_results)} analyzer(s): {sorted(probe_results)}" + state.cycle += 1 + return ToolOutcome(True, summary, payload=probe_results) + + from evalrx.probing import ProbeCallError, availability, build_probe, spec_for + + overrides = _probe_overrides(loop) + names: list[str] = [] + call_params: dict[str, dict[str, Any]] = {} + for name, args in requested: + # Validate every call before any model work: one bad argument must + # not cost the probes named beside it, nor a slot of the call cap. + try: + build_probe(name, args, overrides=overrides) + except ProbeCallError as exc: + return ToolOutcome(False, str(exc), error=exc.code) + verdict = availability(spec_for(name), loop.model, state.data, overrides=overrides) + if not verdict.available: + return ToolOutcome( + False, f"{name} cannot run here: {'; '.join(verdict.missing)}", + error="unavailable", + ) + names.append(name) + if args: + call_params[name] = args + names = list(dict.fromkeys(names)) + try: + probe_results, artifact_pngs = loop._do_m1( + state.cycle, state.data, state.all_hypotheses, state.timings, + analyzers=names, analyzer_params=call_params or None, + ) + except ProbeCallError as exc: + return ToolOutcome(False, str(exc), error=exc.code) if not probe_results: - return ToolOutcome(False, "M1 produced no probe results", error="empty") - state.probe_results = probe_results - state.artifact_pngs = artifact_pngs - board.probe_findings = _summarize_probe(probe_results) - summary = f"ran {len(probe_results)} analyzer(s): {sorted(probe_results)}" + return ToolOutcome(False, f"no result from {names}", error="empty") + # Named calls ADD evidence: the judge is assembling a picture one probe + # at a time, so an earlier probe's findings must survive a later call. + state.probe_results = {**state.probe_results, **probe_results} + state.artifact_pngs = list(state.artifact_pngs) + list(artifact_pngs) + board.probe_findings = _summarize_probe(state.probe_results) state.cycle += 1 + missing = [n for n in names if n not in probe_results] + summary = f"ran {sorted(probe_results)}" + if missing: + summary += f"; no result from {missing} (failed at runtime or filtered)" return ToolOutcome(True, summary, payload=probe_results) return handler @@ -422,12 +558,63 @@ def handler(board: "EvidenceBoard", params: dict[str, Any]) -> ToolOutcome: def build_default_registry(loop: Any, state: _RunState) -> ToolRegistry: """Build the default M1-M4 tool registry for one :class:`AgenticDiagnoseLoop` run.""" registry = ToolRegistry() + registry.register(ToolSpec( + name="list_probes", + description=( + "List the probes that can run on this model and data, grouped by family, " + "with their arguments and why each is ranked; probes you asked for that " + "cannot run are listed with what they are missing. Runs no model. " + "params.failure_modes (list[str], default: the current hypotheses' modes), " + "params.families (list[str]), params.access ('black_box' | 'white_box'), " + "params.allow_interventions (bool, default true), " + "params.only_matching (bool, default false), params.max_probes (int, default 12)." + ), + params_schema={ + "type": "object", + "properties": { + "failure_modes": {"type": "array", "items": {"type": "string"}}, + "families": {"type": "array", "items": {"type": "string"}}, + "access": {"type": "string", "enum": ["black_box", "white_box"]}, + "allow_interventions": {"type": "boolean"}, + "only_matching": {"type": "boolean"}, + "max_probes": {"type": "integer"}, + }, + }, + handler=_list_probes(loop, state), + max_calls=4, + )) registry.register(ToolSpec( name="run_probe", - description="Run M1 analyzer probing to gather per-case findings on the model.", - params_schema={"type": "object", "properties": {}}, + description=( + "Gather per-case findings on the model (M1). With no params, the probe " + f"agent selects the analyzers (at most {_MAX_UNNAMED_PROBE_RUNS} times). " + "To call specific probes, pass params.name " + "(str) and params.params (object), or params.probes " + "(list of {name, params}); see list_probes for names and arguments. " + "Named calls add to the findings gathered so far." + ), + params_schema={ + "type": "object", + "properties": { + "name": {"type": "string"}, + "params": {"type": "object"}, + "probes": { + "type": "array", + "items": { + "type": "object", + "required": ["name"], + "properties": { + "name": {"type": "string"}, + "params": {"type": "object"}, + }, + }, + }, + }, + }, handler=_run_probe(loop, state), - max_calls=3, + # Named calls are narrower than a full M1 pass, so the cap allows a + # few of them; the run's max_actions still bounds the total. + max_calls=6, )) registry.register(ToolSpec( name="run_stats", diff --git a/evalrx/eval_agent/loop.py b/evalrx/eval_agent/loop.py index 5aeb865a..67111b69 100755 --- a/evalrx/eval_agent/loop.py +++ b/evalrx/eval_agent/loop.py @@ -591,6 +591,8 @@ def _get_diagnosis_agent(self) -> "Any": def _do_m1( self, cycle: int, data: "Any", all_hypotheses: "list[Any]", timings: "dict[str, float]", *, log: bool = True, + analyzers: "list[str] | None" = None, + analyzer_params: "dict[str, dict[str, Any]] | None" = None, ) -> "tuple[dict[str, Any], list]": """M1: protocol-guided probing + signal bridge. @@ -598,7 +600,11 @@ def _do_m1( when M1 produced nothing (caller decides whether to stop). When ``log`` is False the probe events are not written — used by :meth:`run_confirm` when it only needs to regenerate the stats the - tester reads (M1/M2 already belong to the analysis phase's log).""" + tester reads (M1/M2 already belong to the analysis phase's log). + + ``analyzers`` / ``analyzer_params`` name the probes and their arguments + explicitly, bypassing selection (see :meth:`ProbeAgent.probe`); the + agentic loop's ``run_probe`` tool passes them through.""" prior_modes = list(dict.fromkeys( h.predicted_failure_mode for h in all_hypotheses if getattr(h, "predicted_failure_mode", None) @@ -610,6 +616,9 @@ def _do_m1( protocol=self.protocol, prior_hypotheses=all_hypotheses or None, hint_failure_modes=prior_modes or None, + # Named calls run exactly what was named (no cap, no backfill). + **({"analyzers": analyzers, "exact": True} if analyzers is not None else {}), + **({"analyzer_params": analyzer_params} if analyzer_params else {}), ) _dt = time.monotonic() - _t0 timings["m1"] = timings.get("m1", 0.0) + _dt diff --git a/evalrx/eval_agent/prompts/agentic.py b/evalrx/eval_agent/prompts/agentic.py index a465ea57..f1f9852c 100644 --- a/evalrx/eval_agent/prompts/agentic.py +++ b/evalrx/eval_agent/prompts/agentic.py @@ -22,6 +22,9 @@ - Do not repeat a tool past its stated call limit. - Do not call a tool that is marked BLOCKED until its listed precondition(s) are met — run the blocking step first. +- To test a specific suspected mechanism, call "list_probes" to see which + probes can run and what arguments they take, then "run_probe" with the + probe's name; "run_probe" with no params lets the probe agent choose. - Prefer the order that makes evidentiary sense (e.g. you need probe findings before stats, stats before hypotheses, hypotheses before testing one) but you decide the sequence and when to stop, not a fixed script. diff --git a/evalrx/eval_agent/stages/fix/__init__.py b/evalrx/eval_agent/stages/fix/__init__.py index a28c005d..327faeb7 100644 --- a/evalrx/eval_agent/stages/fix/__init__.py +++ b/evalrx/eval_agent/stages/fix/__init__.py @@ -14,6 +14,7 @@ tools.py L2 tool catalog + PipelineSpec executor pipeline.py L2 coded-pipeline sandbox with the model bridge internals.py L3b intervention primitives, L4 LoRA repair + weight_edit.py L4 training-free weight edit (rank-one rule + matched control) repair_catalog.py discovery of pre-audited model repair capabilities Import ``FixAgent`` from :mod:`evalrx.eval_agent`. diff --git a/evalrx/eval_agent/stages/fix/agent.py b/evalrx/eval_agent/stages/fix/agent.py index 7428901d..6d790ef0 100755 --- a/evalrx/eval_agent/stages/fix/agent.py +++ b/evalrx/eval_agent/stages/fix/agent.py @@ -176,7 +176,19 @@ class FixAgent: the same cases is leakage, not a fix). ``None`` (default) means L4 candidates are recorded but not executed, same as before this executor existed. - See :func:`~.internals.run_lora_repair`. + See :func:`~.internals.run_lora_repair`. The + weight-edit executor reads its edit direction from + the same pool, under the same disjointness rule. + weight_edit: Opt in to a training-free weight edit at L4 + (:mod:`.weight_edit`): ``True`` for the default + settings, or a dict of + :class:`~.weight_edit.WeightEditSpec` fields + (``alpha``, ``n_columns``, ``n_layers``, ``layers``, + ``edit_dir``, ...). ``None`` (default) proposes no + weight edit. Needs a local white-box model, and a + ``finetune_pool`` unless ``edit_dir`` names a saved + edit; otherwise the candidate is recorded with the + reason it did not run. verbose: When ``True``, print this agent's own tier-routing / candidate-generation / validation-verdict narration to stdout (``evalrx.enable_console_logging()``). @@ -236,6 +248,7 @@ def __init__( paper_methods_only: bool = False, candidate_allowlist: "Iterable[str] | None" = None, finetune_pool: "CaseBatch | None" = None, + weight_edit: "bool | dict[str, Any] | None" = None, verbose: bool = False, baseline_generation_kwargs: "dict[str, Any] | None" = None, concurrency: int = 1, @@ -309,6 +322,7 @@ def __init__( scoring_note=scoring_note, floor_candidates=floor_candidates, deployed_spec=deployed_spec, + weight_edit=weight_edit, ) self.validator = CandidateValidator( score=score, @@ -717,7 +731,8 @@ def _signature(candidate: FixCandidate) -> "tuple[str, str, str]": key = str(p.get("code", "")) else: key = json.dumps( - {k: v for k, v in p.items() if k != "exec_error"}, sort_keys=True, default=str + {k: v for k, v in p.items() if k not in ("exec_error", "audit")}, + sort_keys=True, default=str, ) return (candidate.kind, candidate.name, key) diff --git a/evalrx/eval_agent/stages/fix/descriptions.py b/evalrx/eval_agent/stages/fix/descriptions.py index e28289c7..44827a94 100644 --- a/evalrx/eval_agent/stages/fix/descriptions.py +++ b/evalrx/eval_agent/stages/fix/descriptions.py @@ -74,6 +74,10 @@ "makes a yes-or-no call.", "pai_image_attention": "Makes the model weigh the picture more heavily and its own prior expectations less.", + "rank_one_weight_edit": + "Changes a small part of the model's own weights directly, without any training, " + "so that the internal signal seen on failing questions is removed, and compares " + "that against an equally large random change.", "salient_crop": "Crops the picture to its most eye-catching region before asking.", "self_consistency_5": diff --git a/evalrx/eval_agent/stages/fix/proposal.py b/evalrx/eval_agent/stages/fix/proposal.py index 15b08b33..229cef3e 100644 --- a/evalrx/eval_agent/stages/fix/proposal.py +++ b/evalrx/eval_agent/stages/fix/proposal.py @@ -389,6 +389,7 @@ def __init__( scoring_note: str = "", floor_candidates: "Iterable[str] | None" = ("self_consistency_5",), deployed_spec: "dict[str, Any] | None" = None, + weight_edit: "bool | dict[str, Any] | None" = None, ) -> None: self._judge = judge self.recorder = recorder @@ -407,6 +408,10 @@ def __init__( tuple(str(n) for n in floor_candidates) if floor_candidates else () ) self._deployed_spec = dict(deployed_spec) if deployed_spec else None + # None / False: no weight-edit candidate. True: default settings. + self._weight_edit: "dict[str, Any] | None" = ( + None if not weight_edit else ({} if weight_edit is True else dict(weight_edit)) + ) def propose( self, @@ -625,6 +630,7 @@ def propose( and (min_tier is None or min_tier <= FixTier.L4_PARAMETERS) ): candidates += self._l4_candidates(hyp_lines) + candidates += self._weight_edit_candidates(model, prior_names) if min_tier is not None: candidates = [c for c in candidates if c.tier >= min_tier] if self._candidate_allowlist is not None: @@ -1414,6 +1420,37 @@ def _l4_candidates(self, hyp_lines: str) -> "list[FixCandidate]": ) ] + def _weight_edit_candidates( + self, model: "Model", prior_names: "frozenset[str]" + ) -> "list[FixCandidate]": + """L4 training-free weight edit — opt-in, host-parameterised, never judge-authored.""" + from evalrx.eval_agent.stages.fix.weight_edit import ( + CANDIDATE_KIND, + CANDIDATE_NAME, + WeightEditSpec, + weight_edit_available, + ) + + if self._weight_edit is None or CANDIDATE_NAME in prior_names: + return [] + if not weight_edit_available(model): + logger.info("FixAgent: weight edit requested but %r has no editable MLP writes", model) + return [] + try: + spec = WeightEditSpec.from_payload(self._weight_edit) + except (TypeError, ValueError) as exc: + logger.warning("FixAgent: ignoring invalid weight_edit settings: %s", exc) + return [] + return [ + FixCandidate( + tier=FixTier.L4_PARAMETERS, + name=CANDIDATE_NAME, + kind=CANDIDATE_KIND, + payload=spec.to_dict(), + source="default", + ) + ] + def _ask_judge_object(self, prompt: str) -> "dict[str, Any]": """Single-JSON-object variant of :meth:`_ask_judge`.""" if self._judge is None: diff --git a/evalrx/eval_agent/stages/fix/tiers.py b/evalrx/eval_agent/stages/fix/tiers.py index fb7d126b..aec97ffa 100755 --- a/evalrx/eval_agent/stages/fix/tiers.py +++ b/evalrx/eval_agent/stages/fix/tiers.py @@ -14,7 +14,8 @@ confidence routing) L3b internals (write) modify the forward pass: attention reweighting, sink suppression, activation steering - L4 parameter space build a dataset, fine-tune, re-test + L4 parameter space build a dataset, fine-tune, re-test; or write a + training-free weight edit (opt-in) The allowed tier is an **input** to the fix module (default L2) — there is no automatic escalation. When every candidate within the allowed tier fails diff --git a/evalrx/eval_agent/stages/fix/validation.py b/evalrx/eval_agent/stages/fix/validation.py index d517f55e..0f1de24c 100644 --- a/evalrx/eval_agent/stages/fix/validation.py +++ b/evalrx/eval_agent/stages/fix/validation.py @@ -281,6 +281,7 @@ def scores( "primitive": self._run_primitive, "code": self._run_code, "finetune_spec": self._run_finetune, + "weight_edit": self._run_weight_edit, }.get(candidate.kind) if batch is not None: return batch(candidate, model, data, max_tokens_floor=max_tokens_floor) @@ -492,6 +493,23 @@ def _run_finetune( candidate.payload["exec_error"] = "" if result.ok else result.error return result.scores + def _run_weight_edit( + self, candidate: FixCandidate, model: "Model", data: "CaseBatch", **_: Any + ) -> "dict[str, Optional[bool]]": + """Generate *data* under a training-free weight edit (see :mod:`.weight_edit`).""" + from evalrx.eval_agent.stages.fix.weight_edit import run_weight_edit_repair + + model = self.recorder.instrument(model, data, f"fix_weight_edit:{candidate.name}") + result = run_weight_edit_repair(model, self.finetune_pool, data, candidate.payload, self.score) + for cid, output in result.outputs.items(): + self._record_output(cid, output) + if isinstance(candidate.payload, dict): + candidate.payload["exec_error"] = "" if result.ok else result.error + # What was applied and how the matched control fared travels with + # the candidate's record, next to its validation. + candidate.payload["audit"] = result.audit + return result.scores + def rates( self, candidate: FixCandidate, diff --git a/evalrx/eval_agent/stages/fix/weight_edit.py b/evalrx/eval_agent/stages/fix/weight_edit.py new file mode 100644 index 00000000..ec4f18af --- /dev/null +++ b/evalrx/eval_agent/stages/fix/weight_edit.py @@ -0,0 +1,230 @@ +"""L4 fix executor — a training-free weight edit (weight-space programming). + +Fine-tuning is not the only way to change a model's parameters. A weight edit +writes a small, structured change straight into a few tensors: here, the +rank-one rule ``w' = w - alpha * u (u^T w)`` on selected columns of some MLP +output projections. No gradient step is taken and no data is fitted beyond two +group means. + +What this executor does, and what keeps it honest: + +* The edit is built by :func:`evalrx.wsp.direction.contrast_edit` from a + DIAGNOSIS pool of labelled cases (failing vs passing), or loaded from a + directory the host supplies. The pool must be disjoint from the batch being + validated; an overlap is refused, because deriving the direction from the + cases it is then scored on is leakage, not repair. +* The edit is written in place for the duration of the generation pass and + restored exactly afterwards (:func:`evalrx.wsp.edit.applied`), also when a + case raises. +* The scores go back to the caller's paired validation unchanged, so a weight + edit is judged by the same test as every other tier, including the cases it + breaks. +* Unless disabled, the same batch is also generated under a matched + ``random_direction`` control. The control's pass count is recorded beside + the edit's. A repair that a random change of the same size reproduces is a + perturbation effect, and the record lets a reader see that. + +The judge never writes the edit or any hook code; it can only be given the +few numeric settings in :class:`WeightEditSpec`. +""" + +from __future__ import annotations + +import logging +from dataclasses import asdict, dataclass, field +from typing import TYPE_CHECKING, Any, Callable, Optional + +from evalrx.eval_agent.stages.fix.tools import score_to_bool + +if TYPE_CHECKING: + from evalrx.core.case import CaseBatch, FailureCase + from evalrx.core.model import Model + +logger = logging.getLogger(__name__) + +ScoreFn = Callable[["FailureCase", str], Optional[bool]] + +CANDIDATE_NAME = "rank_one_weight_edit" +CANDIDATE_KIND = "weight_edit" + + +@dataclass +class WeightEditSpec: + """Settings of one weight-edit repair candidate. + + Attributes: + alpha: Strength of ``w' = w - alpha u (u^T w)``. 1 removes the + selected neurons' write along the failure direction; values + above 1 reverse it. + n_columns: Neurons edited per layer. + n_layers: Layers edited when *layers* is not given (the ones whose + writes separate failing from passing cases most). + layers: Explicit decoder-layer indices. + edit_dir: Directory of a saved edit (``WeightEdit.save`` or the Astra + layout). When set, that edit is applied as written and the + contrast settings above are ignored. + verify_base: With *edit_dir*: require the live tensors to match the + hashes recorded in the edit. + control: Also generate under a matched random-direction control. + max_relative_change: Refuse an edit that changes any tensor by more + than this fraction of its norm. ``None`` = no limit. + max_pool_cases: Cap on failing and on passing pool cases read to build + the direction. + seed: Seed of the control draw. + """ + + alpha: float = 1.0 + n_columns: int = 64 + n_layers: int = 2 + layers: "list[int] | None" = None + edit_dir: str = "" + verify_base: bool = False + control: bool = True + max_relative_change: "float | None" = None + max_pool_cases: int = 64 + seed: int = 0 + + @classmethod + def from_payload(cls, payload: "dict[str, Any] | None") -> "WeightEditSpec": + """Build from a candidate payload, ignoring keys that are not settings.""" + known = {k: v for k, v in (payload or {}).items() if k in cls.__dataclass_fields__} + spec = cls(**known) + spec.alpha = float(spec.alpha) + spec.n_columns = int(spec.n_columns) + spec.n_layers = int(spec.n_layers) + if spec.layers is not None: + spec.layers = [int(i) for i in spec.layers] + if spec.n_columns < 1 or spec.n_layers < 1: + raise ValueError("n_columns and n_layers must be positive") + return spec + + def to_dict(self) -> "dict[str, Any]": + return asdict(self) + + +@dataclass +class WeightEditRepairResult: + """Outcome of one weight-edit generate-and-score pass. + + ``scores``/``ok``/``error`` mirror :class:`~.internals.LoraRepairResult`. + ``audit`` records what was applied and how the control fared. + """ + + scores: "dict[str, Optional[bool]]" = field(default_factory=dict) + outputs: "dict[str, str]" = field(default_factory=dict) + ok: bool = False + error: str = "" + audit: "dict[str, Any]" = field(default_factory=dict) + + +def weight_edit_available(model: "Model") -> bool: + """Whether *model* exposes a torch module with editable MLP output projections.""" + try: + from evalrx.wsp.direction import write_sites + + return bool(write_sites(model)) + except Exception: # noqa: BLE001 - no torch, no module, unknown architecture: not available + return False + + +def _generate_all(model: "Model", cases: "CaseBatch", score_fn: ScoreFn, + ) -> "tuple[dict[str, Optional[bool]], dict[str, str]]": + scores: "dict[str, Optional[bool]]" = {} + outputs: "dict[str, str]" = {} + for case in cases: + try: + out = str(model.generate(case.inputs)) + except Exception as exc: # noqa: BLE001 - one case must not abort the pass + logger.debug("weight-edit generate failed on %s: %s", case.id, exc) + scores[case.id] = None + continue + outputs[case.id] = out + scores[case.id] = score_to_bool(score_fn(case, out)) + return scores, outputs + + +def _pass_count(scores: "dict[str, Optional[bool]]") -> "dict[str, int]": + return {"n_pass": sum(1 for s in scores.values() if s is True), + "n_scored": sum(1 for s in scores.values() if s is not None)} + + +def run_weight_edit_repair( + model: "Model", + pool: "CaseBatch | None", + val_cases: "CaseBatch", + payload: "dict[str, Any] | None", + score_fn: ScoreFn, +) -> WeightEditRepairResult: + """Build (or load) a weight edit, generate *val_cases* under it, and score them. + + Never raises for an expected refusal: the reason comes back in ``error`` + with ``ok=False`` and the model untouched, so one candidate cannot abort a + fix run. + """ + try: + spec = WeightEditSpec.from_payload(payload) + except (TypeError, ValueError) as exc: + return WeightEditRepairResult(error=f"invalid weight-edit settings: {exc}") + if not weight_edit_available(model): + return WeightEditRepairResult( + error="needs a local white-box model with linear MLP output projections" + ) + + from evalrx.core.case import Label + from evalrx.wsp.controls import build_control + from evalrx.wsp.direction import contrast_edit + from evalrx.wsp.edit import WeightEdit, applied + from evalrx.wsp.structure import edit_structure + + if spec.edit_dir: + try: + edit = WeightEdit.load(spec.edit_dir) + except Exception as exc: # noqa: BLE001 + return WeightEditRepairResult(error=f"could not load edit from {spec.edit_dir!r}: {exc}") + else: + if pool is None or len(pool) == 0: + return WeightEditRepairResult( + error="no diagnosis pool configured — FixAgent(finetune_pool=...) supplies " + "the labelled cases the edit direction is computed from, never the " + "validation split" + ) + overlap = {c.id for c in pool} & {c.id for c in val_cases} + if overlap: + return WeightEditRepairResult( + error=f"the diagnosis pool shares {len(overlap)} case(s) with the batch " + "being validated; the edit direction must come from disjoint cases" + ) + fails = [c for c in pool if c.label == Label.FAIL][: spec.max_pool_cases] + passes = [c for c in pool if c.label == Label.PASS][: spec.max_pool_cases] + if not fails or not passes: + return WeightEditRepairResult( + error="the diagnosis pool needs both failing and passing cases " + f"(has {len(fails)} failing, {len(passes)} passing)" + ) + try: + edit = contrast_edit(model, fails, passes, layers=spec.layers, + n_layers=spec.n_layers, n_columns=spec.n_columns, + alpha=spec.alpha, name=CANDIDATE_NAME) + except Exception as exc: # noqa: BLE001 + return WeightEditRepairResult(error=f"could not build the edit: {exc}") + + audit: "dict[str, Any]" = {"edit": edit.describe(), "settings": spec.to_dict()} + try: + audit["structure"] = edit_structure(model, edit)["tensors"] + with applied(model, edit, verify_base=spec.verify_base, + max_relative_change=spec.max_relative_change) as report: + audit["applied"] = report.to_dict() + scores, outputs = _generate_all(model, val_cases, score_fn) + except Exception as exc: # noqa: BLE001 - refused before or while writing; weights restored + return WeightEditRepairResult(error=f"the edit could not be applied: {exc}", audit=audit) + audit["edit_result"] = _pass_count(scores) + + if spec.control: + try: + control = build_control(model, edit, "random_direction", seed=spec.seed) + with applied(model, control): + control_scores, _ = _generate_all(model, val_cases, score_fn) + audit["control_random_direction"] = _pass_count(control_scores) + except Exception as exc: # noqa: BLE001 - a missing control is reported, not fatal + audit["control_random_direction"] = {"error": str(exc)} + return WeightEditRepairResult(scores=scores, outputs=outputs, ok=True, audit=audit) diff --git a/evalrx/eval_agent/stages/probe_agent.py b/evalrx/eval_agent/stages/probe_agent.py index d8423ab9..d5a14f61 100755 --- a/evalrx/eval_agent/stages/probe_agent.py +++ b/evalrx/eval_agent/stages/probe_agent.py @@ -182,6 +182,11 @@ def __init__( self.max_cases_per_analyzer = int(max_cases_per_analyzer or 0) #: Names actually truncated by the cap, for the caller to report. self.capped_analyzers: dict[str, tuple[int, int]] = {} + #: Arguments each probe was last CALLED BY NAME with. A pinned re-run + #: (the held-out confirmation) replays them, so the confirm split is + #: probed the way the explore split was; rebuilding with defaults would + #: measure a different signal under the same name. + self._named_params: dict[str, dict[str, Any]] = {} # ------------------------------------------------------------------ # Public interface @@ -196,6 +201,8 @@ def probe( prior_hypotheses: list[Any] | None = None, hint_failure_modes: list[str] | None = None, analyzers: list[str] | None = None, + analyzer_params: "dict[str, dict[str, Any]] | None" = None, + exact: bool = False, ) -> dict[str, "Result"]: """Select analyzers and run each one, returning a ``{name: Result}`` dict. @@ -230,6 +237,21 @@ def probe( entirely (still filtered for applicability). Used by the loop's held-out M4 confirmation to re-run the same set on the confirm split. + analyzer_params: ``{name: constructor arguments}`` for this call + only, checked against each probe's schema + (:func:`~evalrx.probing.toolbox.build_probe`). + A call whose arguments are rejected raises + :class:`~evalrx.probing.toolbox.ProbeCallError` + before any analyzer runs. Used together with + *analyzers* by tool-style callers that name the + probe and its arguments themselves. + exact: With *analyzers*: run those and nothing else — + no ``max_analyzers`` cap and no backfill from + the static selector when one is inapplicable. + A caller that names its probes wants the + evidence it asked for, or to be told it is + missing; substitutes would be reported under + names it never requested. """ rationale: str self._last_need_custom = None @@ -240,7 +262,8 @@ def probe( # too — a fresh judge selection could pick a different set and turn # a real signal into a spurious "designated evidence not measured"). names = list(dict.fromkeys(analyzers)) - rationale = "Pinned by caller (held-out confirmation re-run)." + rationale = ("Called by name." if exact or analyzer_params else + "Pinned by caller (held-out confirmation re-run).") elif self.judge is not None and protocol is not None: names, rationale = self._llm_select(protocol, model, prior_hypotheses, data) else: @@ -251,15 +274,29 @@ def probe( data=data, ) rationale = _static_rationale(names, hint_failure_modes) - names, filtered = self._filter_applicable_names(names, model, data) + if exact and analyzers is not None: + filtered = [ + n for n in names + if not _analyzer_data_preconditions_met(n, data, model) or self._needs_injection(n) + ] + names = [n for n in names if n not in filtered] + else: + names, filtered = self._filter_applicable_names(names, model, data) if filtered: rationale += f" Filtered inapplicable analyzer(s): {', '.join(filtered)}." # Build (name, analyzer) pairs up front — _make_analyzer is cheap and # not thread-safe to call concurrently. + if exact and analyzers is not None: + for name in names: # a named call with no arguments means the defaults + self._named_params.pop(name, None) + if analyzer_params: + self._named_params.update({k: dict(v) for k, v in analyzer_params.items() if v}) + replay = self._named_params if analyzers is not None else {} tasks: list[tuple[str, "Analyzer"]] = [] for name in names: - analyzer = self._make_analyzer(name) + analyzer = self._make_analyzer( + name, (analyzer_params or {}).get(name) or replay.get(name)) if analyzer is not None: tasks.append((name, analyzer)) @@ -935,7 +972,15 @@ def _needs_injection(self, name: str) -> "list[str]": inspect.Parameter.VAR_KEYWORD) ] - def _make_analyzer(self, name: str) -> "Analyzer | None": + def _make_analyzer( + self, name: str, params: "dict[str, Any] | None" = None + ) -> "Analyzer | None": + if params: + # Explicit per-call arguments go through the probe's schema; a bad + # call is the caller's error and must surface, not be skipped. + from evalrx.probing.toolbox import build_probe + + return build_probe(name, params, overrides=self._overrides) if name in self._overrides: return self._overrides[name] cls = registry.analyzers.get(name) diff --git a/evalrx/probing/__init__.py b/evalrx/probing/__init__.py new file mode 100644 index 00000000..f06039a5 --- /dev/null +++ b/evalrx/probing/__init__.py @@ -0,0 +1,54 @@ +"""Probes as tools: discover, route and call analyzers by name. + + import evalrx + from evalrx.probing import ProbeToolbox + + box = ProbeToolbox(model, cases) + box.describe() # availability + reasons + box.route(failure_modes=["visual_neglect"]).names # what to run first + box.call("attention_knockout", {"queries": "last"}) + +See :mod:`evalrx.probing.catalog` for the manifest, +:mod:`evalrx.probing.router` for ranking, and +:mod:`evalrx.probing.toolbox` for execution. +""" + +from evalrx.probing.catalog import ( + FAMILY_QUESTIONS, + ProbeAvailability, + ProbeParam, + ProbeToolSpec, + availability, + describe_probes, + family_of, + probe_specs, + spec_for, +) +from evalrx.probing.router import ProbeRoute, ProbeRouter, RoutedProbe +from evalrx.probing.toolbox import ( + ProbeCallError, + ProbeToolbox, + build_probe, + check_params, + summarize_result, +) + +__all__ = [ + "FAMILY_QUESTIONS", + "ProbeAvailability", + "ProbeCallError", + "ProbeParam", + "ProbeRoute", + "ProbeRouter", + "ProbeToolSpec", + "ProbeToolbox", + "RoutedProbe", + "availability", + "build_probe", + "check_params", + "describe_probes", + "family_of", + "probe_specs", + "spec_for", + "summarize_result", +] diff --git a/evalrx/probing/catalog.py b/evalrx/probing/catalog.py new file mode 100644 index 00000000..48749ffc --- /dev/null +++ b/evalrx/probing/catalog.py @@ -0,0 +1,496 @@ +"""Probe tools — every registered analyzer described as a callable tool. + +An :class:`~evalrx.core.analyzer.Analyzer` already carries what a tool needs: +a registered name, constructor hyper-parameters, required capabilities, the +modalities it reads, and what its numbers mean. This module reads that off the +class and presents it in one uniform shape, so a caller (a person, the M1 +selector, or an LLM judge) can ask three questions without reading source: + +* what probes exist, and what does each one answer? (:func:`probe_specs`) +* which of them can run on THIS model and THIS batch, and if one cannot, + what exactly is missing? (:func:`describe_probes`) +* what arguments does a probe take? (:meth:`ProbeToolSpec.parameters_schema`) + +The availability report follows the convention of VLMProbe's +``Prober.describe()``: every method is listed, an unavailable one says why, and +"cannot run here" is never confused with "ran and measured zero". + +Nothing here imports torch, and nothing here runs a model. +""" + +from __future__ import annotations + +import ast +import inspect +import re +import textwrap +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Iterable, Mapping + +from evalrx.core.capability import Capability +from evalrx.core.registry import registry +from evalrx.core.tool import Tool + +if TYPE_CHECKING: + from evalrx.core.analyzer import Analyzer + from evalrx.core.model import Model + +# ---------------------------------------------------------------------- +# Families +# ---------------------------------------------------------------------- + +#: ``analyzers/`` -> family name. Packages not listed keep their name. +_PACKAGE_FAMILY = {"patching": "causal", "weights": "weight_edit"} + +#: What each family is for, in one sentence a judge can route on. +FAMILY_QUESTIONS: dict[str, str] = { + "perturbation": "Does the answer change when the input is changed in a controlled way?", + "uncertainty": "Does the model know when it is wrong?", + "hallucination": "Does the output assert things the input does not support?", + "reasoning": "Is the failure in extraction, termination, arithmetic, knowledge or the chain?", + "attention": "Where does the model look?", + "attribution": "Which input regions does the answer depend on?", + "lens": "What do intermediate states predict, layer by layer?", + "causal": "Which internal route carries the effect, measured by intervening on it?", + "geometry": "How are cases arranged in representation space?", + "weight_edit": "What does a direct weight edit change, and is the change specific?", + "agent": "Where does a multi-step agent run go wrong?", +} + +#: Families whose probes modify the forward pass or the parameters while they run. +_INTERVENING_FAMILIES = frozenset({"causal", "weight_edit"}) + +#: Capabilities a black-box (API) model can provide. +_BLACKBOX_CAPS = frozenset({Capability.GENERATE, Capability.LOGPROBS, Capability.TOOL_CALLS}) + +_JSON_TYPES: dict[type, str] = { + bool: "boolean", int: "integer", float: "number", str: "string", + list: "array", tuple: "array", dict: "object", +} +_ANNOTATION_TYPES = { + "bool": "boolean", "int": "integer", "float": "number", "str": "string", + "list": "array", "tuple": "array", "dict": "object", + "sequence": "array", "iterable": "array", "mapping": "object", +} + + +def family_of(cls: type) -> str: + """Family of an analyzer class, from its ``analyzers/`` location. + + A class attribute ``probe_family`` overrides the location, for analyzers + registered from outside ``evalrx.analyzers``. + """ + explicit = getattr(cls, "probe_family", None) + if explicit: + return str(explicit) + parts = (cls.__module__ or "").split(".") + if "analyzers" in parts: + idx = parts.index("analyzers") + if len(parts) > idx + 2: + package = parts[idx + 1] + return _PACKAGE_FAMILY.get(package, package) + return "custom" + + +# ---------------------------------------------------------------------- +# Parameters +# ---------------------------------------------------------------------- + +@dataclass(frozen=True) +class ProbeParam: + """One constructor argument of a probe. + + ``injected`` marks an argument that cannot be written as JSON (a model, a + callable, a weight edit): the host supplies it, a judge never does. + """ + + name: str + json_type: "str | None" = None + default: Any = None + required: bool = False + injected: bool = False + nullable: bool = False + + def schema(self) -> dict[str, Any]: + out: dict[str, Any] = {} + if self.json_type: + out["type"] = [self.json_type, "null"] if self.nullable else self.json_type + if not self.required and _is_jsonable(self.default): + out["default"] = list(self.default) if isinstance(self.default, tuple) else self.default + return out + + +def _is_jsonable(value: Any) -> bool: + if value is None or isinstance(value, (bool, int, float, str)): + return True + if isinstance(value, (list, tuple)): + return all(_is_jsonable(v) for v in value) + if isinstance(value, dict): + return all(isinstance(k, str) and _is_jsonable(v) for k, v in value.items()) + return False + + +def _annotation_type(annotation: Any) -> "tuple[str | None, bool, bool]": + """``(json_type, nullable, annotated)`` read from a constructor annotation. + + ``annotated`` is False when there is nothing to read; a present annotation + with no JSON counterpart (a callable, a model) returns ``(None, _, True)``. + """ + if annotation is inspect.Parameter.empty: + return None, False, False + if isinstance(annotation, type): + return _JSON_TYPES.get(annotation), False, True + text = str(annotation).strip().strip("\"'") + nullable = False + wrapped = re.fullmatch(r"(?:typing\.)?(Optional|Union)\[(.*)\]", text, re.DOTALL) + if wrapped: + nullable = wrapped.group(1) == "Optional" + text = wrapped.group(2) if nullable else wrapped.group(2).replace(",", "|") + if text in {"Any", "typing.Any", "object"}: + return None, True, False + members = [m.strip() for m in _split_top_level(text)] + nullable = nullable or any(m in {"None", "NoneType"} for m in members) + kinds = { + _ANNOTATION_TYPES.get(re.split(r"[\[\s]", m, maxsplit=1)[0].split(".")[-1].lower()) + for m in members if m not in {"None", "NoneType"} + } + if None in kinds: + return None, nullable, True # at least one member has no JSON form + if kinds == {"integer", "number"}: + kinds = {"number"} + return (next(iter(kinds)) if len(kinds) == 1 else None), nullable, True + + +def _split_top_level(text: str) -> list[str]: + """Split a union on ``|`` outside brackets (``dict[str, int | None] | None``).""" + parts, depth, start = [], 0, 0 + for i, ch in enumerate(text): + if ch in "[(": + depth += 1 + elif ch in "])": + depth -= 1 + elif ch == "|" and depth == 0: + parts.append(text[start:i]) + start = i + 1 + parts.append(text[start:]) + return parts + + +def _params_of(cls: type) -> "tuple[ProbeParam, ...]": + try: + signature = inspect.signature(cls.__init__) + except (TypeError, ValueError): + return () + params: list[ProbeParam] = [] + for name, p in signature.parameters.items(): + if name == "self" or p.kind in (inspect.Parameter.VAR_POSITIONAL, + inspect.Parameter.VAR_KEYWORD): + continue + required = p.default is inspect.Parameter.empty + json_type, nullable, annotated = _annotation_type(p.annotation) + default = None if required else p.default + if json_type is None and not required and default is not None: + json_type = _JSON_TYPES.get(type(default)) + if not required and default is None: + nullable = True + # Host-only arguments: a value that is not JSON, or an annotation that + # names something with no JSON form (a callable, a model, an edit). + injected = ( + not _is_jsonable(default) + or (json_type is None and (required or annotated)) + ) + params.append(ProbeParam(name=name, json_type=json_type, default=default, + required=required, injected=injected, nullable=nullable)) + return tuple(params) + + +# ---------------------------------------------------------------------- +# The spec +# ---------------------------------------------------------------------- + +def _first_line(text: "str | None") -> str: + for line in (text or "").strip().splitlines(): + line = line.strip() + if line: + return line + return "" + + +def is_stub(cls: type) -> bool: + """True when ``_run`` only raises ``NotImplementedError`` (a registered placeholder).""" + try: + tree = ast.parse(textwrap.dedent(inspect.getsource(cls._run))) + except (OSError, TypeError, SyntaxError, AttributeError): + return False + body = [n for n in tree.body[0].body + if not (isinstance(n, ast.Expr) and isinstance(n.value, ast.Constant))] + if len(body) != 1 or not isinstance(body[0], ast.Raise): + return False + exc = body[0].exc + if isinstance(exc, ast.Call): + exc = exc.func + return isinstance(exc, ast.Name) and exc.id == "NotImplementedError" + + +def _failure_modes_by_probe() -> dict[str, tuple[str, ...]]: + from evalrx.eval_agent.stages.probe import _FAILURE_MODE_TO_ANALYZERS + + inverse: dict[str, list[str]] = {} + for mode, names in _FAILURE_MODE_TO_ANALYZERS.items(): + for name in names: + inverse.setdefault(name, []).append(mode) + return {name: tuple(modes) for name, modes in inverse.items()} + + +@dataclass(frozen=True) +class ProbeToolSpec: + """A registered analyzer, described as a tool. + + Attributes: + name: Registered analyzer key; also the tool name. + family: Functional family (``lens``, ``causal``, ``uncertainty``, ...). + summary: One line saying what the probe does. + question: The question its result answers. + access: ``"black_box"`` (generation / logprobs only), ``"white_box"`` + (needs internals) or ``"data_only"`` (reads recorded data; + the model may be ``None``). + requires: Capability values the model must provide. + modalities: Modalities the probe applies to. + data_slots: Modality slots the batch must actually fill. + needs_trajectories: Whether it reads multi-step agent runs. + intervenes: Whether it changes the forward pass or weights while it + runs (always restored afterwards). + cost: ``"light"`` | ``"moderate"`` | ``"heavy"`` — a coarse hint + derived from access and family unless the analyzer class + sets ``probe_cost``. + params: Constructor arguments. + signals: ``metric -> meaning`` from the analyzer's ``signal_docs``. + failure_modes: Failure-mode tags this probe is routed to. + implemented: False for a registered placeholder. + """ + + name: str + family: str + summary: str + question: str + access: str + requires: tuple[str, ...] + modalities: tuple[str, ...] + data_slots: tuple[str, ...] + needs_trajectories: bool + intervenes: bool + cost: str + params: tuple[ProbeParam, ...] + signals: Mapping[str, str] = field(default_factory=dict) + failure_modes: tuple[str, ...] = () + implemented: bool = True + analyzer_cls: "type[Analyzer] | None" = field(default=None, repr=False, compare=False) + + # -- arguments ------------------------------------------------------ + @property + def injected_params(self) -> tuple[str, ...]: + """Required arguments only the host can supply.""" + return tuple(p.name for p in self.params if p.required and p.injected) + + @property + def callable_params(self) -> tuple[ProbeParam, ...]: + """Arguments a caller may pass as JSON.""" + return tuple(p for p in self.params if not p.injected) + + def parameters_schema(self) -> dict[str, Any]: + """JSON schema of the arguments a caller may pass.""" + schema: dict[str, Any] = { + "type": "object", + "properties": {p.name: p.schema() for p in self.callable_params}, + "additionalProperties": False, + } + required = [p.name for p in self.callable_params if p.required] + if required: + schema["required"] = required + return schema + + # -- views ---------------------------------------------------------- + def to_tool(self, fn: Any = None) -> Tool: + """The probe as a :class:`~evalrx.core.tool.Tool` (OpenAI / HF tool schema).""" + description = self.summary + if self.question and self.question != self.summary: + description = f"{self.summary} Answers: {self.question}" + return Tool(name=self.name, description=description, + parameters=self.parameters_schema(), fn=fn) + + def to_dict(self) -> dict[str, Any]: + return { + "name": self.name, + "family": self.family, + "summary": self.summary, + "question": self.question, + "access": self.access, + "requires": list(self.requires), + "modalities": list(self.modalities), + "data_slots": list(self.data_slots), + "needs_trajectories": self.needs_trajectories, + "intervenes": self.intervenes, + "cost": self.cost, + "parameters": self.parameters_schema(), + "injected_params": list(self.injected_params), + "signals": dict(self.signals), + "failure_modes": list(self.failure_modes), + "implemented": self.implemented, + } + + +def _signals_of(cls: type) -> dict[str, str]: + out: dict[str, str] = {} + for metric, doc in (getattr(cls, "signal_docs", None) or {}).items(): + out[str(metric)] = str(doc[1] if isinstance(doc, tuple) and len(doc) == 2 else doc) + return out + + +def _access_of(cls: type) -> str: + requires = frozenset(getattr(cls, "requires", frozenset())) + if not requires: + return "data_only" + return "black_box" if requires <= _BLACKBOX_CAPS else "white_box" + + +def _cost_of(cls: type, access: str, intervenes: bool) -> str: + explicit = getattr(cls, "probe_cost", None) + if explicit: + return str(explicit) + if access == "data_only": + return "light" + if intervenes or Capability.GRADIENTS in getattr(cls, "requires", frozenset()): + return "heavy" + return "moderate" + + +def spec_for(name: str, cls: "type[Analyzer] | None" = None, + failure_modes: "Mapping[str, tuple[str, ...]] | None" = None) -> ProbeToolSpec: + """Build the :class:`ProbeToolSpec` of one registered analyzer.""" + cls = cls or registry.analyzers.get(name) + family = family_of(cls) + summary = str(getattr(cls, "description", "") or _first_line(cls.__doc__) or name) + access = _access_of(cls) + intervenes = bool(getattr(cls, "probe_intervenes", family in _INTERVENING_FAMILIES)) + modes = failure_modes if failure_modes is not None else _failure_modes_by_probe() + return ProbeToolSpec( + name=name, + family=family, + summary=summary, + question=str(getattr(cls, "question", "") or FAMILY_QUESTIONS.get(family, summary)), + access=access, + requires=tuple(sorted(c.value for c in getattr(cls, "requires", frozenset()))), + modalities=tuple(sorted(getattr(cls, "applies_to_modalities", frozenset()))), + data_slots=tuple(sorted(getattr(cls, "requires_modalities", frozenset()) or ())), + needs_trajectories=bool(getattr(cls, "requires_trajectories", False)), + intervenes=intervenes, + cost=_cost_of(cls, access, intervenes), + params=_params_of(cls), + signals=_signals_of(cls), + failure_modes=tuple(modes.get(name, ())), + implemented=not is_stub(cls), + analyzer_cls=cls, + ) + + +def probe_specs(names: "Iterable[str] | None" = None) -> dict[str, ProbeToolSpec]: + """``{name: ProbeToolSpec}`` for every registered analyzer (or just *names*).""" + modes = _failure_modes_by_probe() + chosen = sorted(registry.analyzers.list()) if names is None else list(names) + return {name: spec_for(name, failure_modes=modes) for name in chosen} + + +# ---------------------------------------------------------------------- +# Availability +# ---------------------------------------------------------------------- + +@dataclass(frozen=True) +class ProbeAvailability: + """Whether a probe can run here, and every reason it cannot. + + ``codes`` are stable identifiers for programmatic use: ``not_implemented``, + ``capability``, ``modality``, ``data_slot``, ``trajectories``, + ``data_precondition``, ``injection``. + """ + + available: bool + missing: tuple[str, ...] = () + codes: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + return {"available": self.available, "missing": list(self.missing), + "codes": list(self.codes)} + + +def availability(spec: ProbeToolSpec, model: "Model | None", data: Any = None, *, + overrides: "Iterable[str]" = ()) -> ProbeAvailability: + """Check *spec* against a model and, when given, the batch about to be probed. + + Args: + spec: The probe. + model: The model under test (``None`` is valid for data-only probes). + data: The case batch, when known. Without it the data-shape checks + are skipped: no evidence is not evidence of absence. + overrides: Names for which the host already holds a constructed + analyzer; those never report a missing injected argument. + """ + missing: list[str] = [] + codes: list[str] = [] + + def fail(code: str, reason: str) -> None: + codes.append(code) + missing.append(reason) + + if not spec.implemented: + fail("not_implemented", "registered placeholder; the analysis is not implemented yet") + + provided = {c.value if isinstance(c, Capability) else str(c) + for c in (getattr(model, "capabilities", None) or ())} + lacking = [c for c in spec.requires if c not in provided] + if lacking: + where = "a local white-box model" if spec.access == "white_box" else "the model" + fail("capability", f"needs {where} that provides: {', '.join(lacking)}") + + if model is not None or spec.requires: + model_modalities = set(getattr(model, "modalities", None) or {"text"}) + if spec.modalities and not (set(spec.modalities) & model_modalities): + fail("modality", f"applies to {', '.join(spec.modalities)} models; " + f"this model declares {', '.join(sorted(model_modalities))}") + + if data is not None: + from evalrx.eval_agent.stages.probe import StrategyProbe, _carries_trajectories + + if spec.needs_trajectories and not _carries_trajectories(data): + fail("trajectories", "reads agent trajectories; the batch has none") + if spec.data_slots and spec.name in StrategyProbe.slot_starved({spec.name}, data, model): + fail("data_slot", f"the batch fills none of: {', '.join(spec.data_slots)}") + if not codes: + from evalrx.eval_agent.stages.probe_agent import _analyzer_data_preconditions_met + + if not _analyzer_data_preconditions_met(spec.name, data, model): + fail("data_precondition", "the cases lack the labels or metadata this probe reads") + + if spec.injected_params and spec.name not in set(overrides): + fail("injection", "the host must supply: " + ", ".join(spec.injected_params) + + " (pass a constructed analyzer via analyzer_overrides)") + + return ProbeAvailability(available=not codes, missing=tuple(missing), codes=tuple(codes)) + + +def describe_probes(model: "Model | None", data: Any = None, *, + overrides: "Iterable[str]" = (), + families: "Iterable[str] | None" = None) -> dict[str, dict[str, Any]]: + """Report every probe with its availability on *model* (and *data*). + + Returns ``{name: {...spec fields..., "available": bool, "missing": [...], + "codes": [...]}}``. No forward pass is made. + """ + wanted = None if families is None else {str(f) for f in families} + overrides = tuple(overrides) + report: dict[str, dict[str, Any]] = {} + for name, spec in probe_specs().items(): + if wanted is not None and spec.family not in wanted: + continue + report[name] = {**spec.to_dict(), + **availability(spec, model, data, overrides=overrides).to_dict()} + return report diff --git a/evalrx/probing/router.py b/evalrx/probing/router.py new file mode 100644 index 00000000..892fd05b --- /dev/null +++ b/evalrx/probing/router.py @@ -0,0 +1,220 @@ +"""Route a diagnostic question to the probes that can answer it. + +:class:`ProbeRouter` joins three things that were decided in separate places: + +* what can run — :func:`~evalrx.probing.catalog.availability`; +* what is worth running first — the slot-composed priority and failure-mode + boosts of :class:`~evalrx.eval_agent.stages.probe.StrategyProbe`; +* what the caller asked for — failure modes, families, access level. + +The result keeps the reason for every decision: each selected probe says why it +was ranked where it was, and each rejected probe that the request pointed at +says what it is missing. A caller that cannot get the evidence it wanted learns +that, instead of receiving a shorter list. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Iterable + +from evalrx.probing.catalog import ( + FAMILY_QUESTIONS, + ProbeToolSpec, + availability, + probe_specs, +) + +if TYPE_CHECKING: + from evalrx.core.model import Model + + +def normalize_mode(mode: str) -> str: + """Failure-mode tag in the form the routing table is keyed by.""" + return str(mode).strip().lower().replace(" ", "_").replace("-", "_") + + +def _short(value: Any, limit: int = 32) -> str: + text = repr(value) + return text if len(text) <= limit else text[: limit - 1] + "…" + + +@dataclass(frozen=True) +class RoutedProbe: + """One selected probe and why it was selected.""" + + name: str + spec: ProbeToolSpec + reasons: tuple[str, ...] = () + + def to_dict(self) -> dict[str, Any]: + return {"name": self.name, "family": self.spec.family, "access": self.spec.access, + "cost": self.spec.cost, "question": self.spec.question, + "reasons": list(self.reasons)} + + +@dataclass +class ProbeRoute: + """The outcome of one routing request.""" + + selected: list[RoutedProbe] = field(default_factory=list) + #: ``name -> reasons`` for probes the request pointed at that cannot run. + unavailable: dict[str, tuple[str, ...]] = field(default_factory=dict) + #: Failure-mode tags in the request that no probe is mapped to. + unknown_modes: tuple[str, ...] = () + + @property + def names(self) -> list[str]: + return [p.name for p in self.selected] + + def to_dict(self) -> dict[str, Any]: + return { + "selected": [p.to_dict() for p in self.selected], + "unavailable": {k: list(v) for k, v in self.unavailable.items()}, + "unknown_modes": list(self.unknown_modes), + } + + def to_prompt(self, *, with_params: bool = True) -> str: + """Render the route as a catalog an LLM judge can pick from.""" + lines: list[str] = [] + family = None + for probe in sorted(self.selected, key=lambda p: (p.spec.family,)): + if probe.spec.family != family: + family = probe.spec.family + lines.append(f"[{family}] {FAMILY_QUESTIONS.get(family, '')}".rstrip()) + bits = [f" - {probe.name} ({probe.spec.access}, {probe.spec.cost}): " + f"{probe.spec.summary}"] + if with_params and probe.spec.callable_params: + args = ", ".join( + p.name if p.required else f"{p.name}={_short(p.default)}" + for p in probe.spec.callable_params + ) + bits.append(f"params: {args}") + if probe.reasons: + bits.append("why: " + "; ".join(probe.reasons)) + lines.append(" | ".join(bits)) + if self.unavailable: + lines.append("UNAVAILABLE:") + for name, reasons in sorted(self.unavailable.items()): + lines.append(f" - {name}: {'; '.join(reasons)}") + if self.unknown_modes: + lines.append("No probe is mapped to: " + ", ".join(self.unknown_modes)) + return "\n".join(lines) + + +class ProbeRouter: + """Select and order probes for a model, a batch and a question. + + Args: + selector: The priority source. Defaults to a fresh + :class:`~evalrx.eval_agent.stages.probe.StrategyProbe`. + overrides: Names for which the host holds a constructed analyzer; they + are never rejected for a missing injected argument. + """ + + def __init__(self, selector: Any = None, overrides: "Iterable[str]" = ()) -> None: + if selector is None: + from evalrx.eval_agent.stages.probe import StrategyProbe + + selector = StrategyProbe() + self.selector = selector + self.overrides = tuple(overrides) + + def route( + self, + model: "Model | None", + data: Any = None, + *, + failure_modes: "Iterable[str] | None" = None, + families: "Iterable[str] | None" = None, + access: "str | None" = None, + allow_interventions: bool = True, + only_matching: bool = False, + max_probes: "int | None" = None, + ) -> ProbeRoute: + """Rank the probes that can run and answer the request. + + Args: + model: The model under test. + data: The batch about to be probed, when known. + failure_modes: Failure-mode tags (for example from M3 + hypotheses). Probes mapped to them lead. + families: Keep only these families. + access: ``"black_box"`` keeps probes an API model can + run; ``"white_box"`` keeps the ones that need + internals. ``None`` keeps both. + allow_interventions: When False, drop probes that modify the + forward pass or weights. + only_matching: When True and *failure_modes* is given, return + only probes mapped to those modes. + max_probes: Cap on the number of selected probes. + """ + from evalrx.eval_agent.stages.probe import _FAILURE_MODE_TO_ANALYZERS + + specs = probe_specs() + modes = [normalize_mode(m) for m in (failure_modes or ()) if str(m).strip()] + wanted_families = None if families is None else {str(f) for f in families} + + targeted: dict[str, list[str]] = {} + unknown: list[str] = [] + for mode in dict.fromkeys(modes): + mapped = _FAILURE_MODE_TO_ANALYZERS.get(mode) + if not mapped: + unknown.append(mode) + continue + for name in mapped: + targeted.setdefault(name, []).append(mode) + + def in_scope(spec: ProbeToolSpec) -> bool: + if wanted_families is not None and spec.family not in wanted_families: + return False + if access == "black_box" and spec.access == "white_box": + return False + if access == "white_box" and spec.access != "white_box": + return False + if not allow_interventions and spec.intervenes: + return False + return not (only_matching and modes and spec.name not in targeted) + + runnable: dict[str, ProbeToolSpec] = {} + route = ProbeRoute(unknown_modes=tuple(unknown)) + for name, spec in specs.items(): + if not in_scope(spec): + continue + verdict = availability(spec, model, data, overrides=self.overrides) + if verdict.available: + runnable[name] = spec + elif name in targeted or wanted_families is not None: + # Only report rejections the request actually pointed at; an + # unfiltered request would otherwise list half the registry. + route.unavailable[name] = verdict.missing + + order = self._priority(model, data, modes) + rank = {name: i for i, name in enumerate(order)} + slots = sorted(self._slots(model, data)) + ordered = sorted(runnable, key=lambda n: (rank.get(n, len(rank)), n)) + for name in ordered: + reasons: list[str] = [] + if name in targeted: + reasons.append("targets " + ", ".join(targeted[name])) + if name in rank: + reasons.append(f"priority {rank[name] + 1} for a {'+'.join(slots)} run") + else: + reasons.append("runnable, not in the priority list") + route.selected.append(RoutedProbe(name, runnable[name], tuple(reasons))) + if max_probes is not None: + route.selected = route.selected[: max(0, int(max_probes))] + return route + + # ------------------------------------------------------------------ + def _priority(self, model: Any, data: Any, modes: list[str]) -> list[str]: + try: + return list(self.selector.select(model, hint_failure_modes=modes or None, data=data)) + except Exception: # noqa: BLE001 - a model the selector cannot read still gets a route + return [] + + def _slots(self, model: Any, data: Any) -> set[str]: + try: + return set(self.selector.routed_slots(model, data)) + except Exception: # noqa: BLE001 + return {"text"} diff --git a/evalrx/probing/toolbox.py b/evalrx/probing/toolbox.py new file mode 100644 index 00000000..943c7a21 --- /dev/null +++ b/evalrx/probing/toolbox.py @@ -0,0 +1,226 @@ +"""Call a probe by name with JSON arguments. + +:func:`build_probe` turns ``(name, params)`` into a constructed analyzer after +checking the arguments against the probe's schema, and :class:`ProbeToolbox` +binds a model and a batch so each probe becomes a +:class:`~evalrx.core.tool.Tool` whose function runs it:: + + box = ProbeToolbox(model, cases) + box.describe()["attention_knockout"]["available"] + result = box.call("logit_lens", {"top_k": 5}) + tools = box.tools() # OpenAI / HF tool schemas, ready for an agent + +A bad call fails before any model work with a message that names the accepted +arguments, so an LLM caller can correct itself on the next turn. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any, Iterable, Mapping + +from evalrx.core.registry import registry +from evalrx.core.tool import Tool, ToolResult +from evalrx.probing.catalog import ( + ProbeParam, + ProbeToolSpec, + availability, + describe_probes, + probe_specs, + spec_for, +) +from evalrx.probing.router import ProbeRoute, ProbeRouter + +if TYPE_CHECKING: + from evalrx.core.analyzer import Analyzer + from evalrx.core.model import Model + from evalrx.core.result import Result + + +class ProbeCallError(ValueError): + """A probe call that cannot be executed as written. + + ``code`` is one of ``unknown_probe``, ``bad_params``, ``unavailable``. + """ + + def __init__(self, message: str, code: str = "bad_params") -> None: + super().__init__(message) + self.code = code + + +def _coerce(param: ProbeParam, value: Any, probe: str) -> Any: + """Check *value* against the declared JSON type; widen int to float only.""" + kind = param.json_type + if value is None: + if param.nullable or kind is None: + return None + raise ProbeCallError(f"{probe}: '{param.name}' may not be null (expected {kind})") + if kind is None: + return value + ok = { + "boolean": isinstance(value, bool), + "integer": isinstance(value, int) and not isinstance(value, bool), + "number": isinstance(value, (int, float)) and not isinstance(value, bool), + "string": isinstance(value, str), + "array": isinstance(value, (list, tuple)), + "object": isinstance(value, dict), + }[kind] + if not ok and kind == "integer" and isinstance(value, float) and value.is_integer(): + return int(value) # JSON has one number type; 3.0 for an integer field is 3 + if not ok: + raise ProbeCallError( + f"{probe}: '{param.name}' expects {kind}, got {type(value).__name__} ({value!r})" + ) + if kind == "number": + return float(value) if isinstance(param.default, float) else value + if kind == "array" and isinstance(param.default, tuple): + return tuple(value) + return value + + +def check_params(spec: ProbeToolSpec, params: "Mapping[str, Any] | None") -> dict[str, Any]: + """Validate JSON arguments for *spec*; return them ready for the constructor.""" + params = dict(params or {}) + accepted = {p.name: p for p in spec.callable_params} + unknown = sorted(set(params) - set(accepted)) + if unknown: + injected = [n for n in unknown if n in {p.name for p in spec.params}] + hint = (f" ({', '.join(injected)} must be supplied by the host, not in the call)" + if injected else "") + raise ProbeCallError( + f"{spec.name}: unknown argument(s) {unknown}{hint}; " + f"accepted: {sorted(accepted) or 'none'}" + ) + lacking = [p.name for p in accepted.values() if p.required and p.name not in params] + if lacking: + raise ProbeCallError(f"{spec.name}: missing required argument(s) {lacking}") + return {name: _coerce(accepted[name], value, spec.name) for name, value in params.items()} + + +def build_probe(name: str, params: "Mapping[str, Any] | None" = None, *, + overrides: "Mapping[str, Analyzer] | None" = None) -> "Analyzer": + """Construct the analyzer for one probe call. + + An entry in *overrides* supplies arguments a call cannot carry (a judge + model, a weight edit). With no *params* that instance is used as is; with + *params* a new instance of the same class is built from the override's + parameters updated by the call, so the override itself is never mutated. + """ + if not registry.analyzers.has(name): + raise ProbeCallError( + f"unknown probe {name!r}; registered: {registry.analyzers.list()}", "unknown_probe" + ) + spec = spec_for(name) + cleaned = check_params(spec, params) + override = (overrides or {}).get(name) + try: + if override is not None: + if not cleaned: + return override + return type(override)(**{**override.get_params(), **cleaned}) + if spec.injected_params: + raise ProbeCallError( + f"{name}: the host must supply {list(spec.injected_params)} " + "(pass a constructed analyzer via analyzer_overrides)", "unavailable", + ) + return spec.analyzer_cls(**cleaned) + except ProbeCallError: + raise + except (TypeError, ValueError) as exc: + raise ProbeCallError(f"{name}: rejected its arguments: {exc}") from exc + + +class ProbeToolbox: + """Probes bound to one model and (optionally) one batch. + + Args: + model: The model under test. + data: Default batch for :meth:`call`; also used for availability. + overrides: ``{name: analyzer}`` for probes that need host-supplied + constructor arguments. + max_cases: When positive, every call sees at most this many cases + (a stratified head of the batch). + """ + + def __init__(self, model: "Model | None", data: Any = None, *, + overrides: "Mapping[str, Analyzer] | None" = None, + max_cases: int = 0) -> None: + self.model = model + self.data = data + self.overrides: dict[str, "Analyzer"] = dict(overrides or {}) + self.max_cases = int(max_cases or 0) + + # -- discovery ------------------------------------------------------ + def specs(self) -> dict[str, ProbeToolSpec]: + return probe_specs() + + def describe(self, families: "Iterable[str] | None" = None) -> dict[str, dict[str, Any]]: + """Every probe with its availability here (see :func:`describe_probes`).""" + return describe_probes(self.model, self.data, overrides=self.overrides, + families=families) + + def available(self) -> list[str]: + return sorted(n for n, row in self.describe().items() if row["available"]) + + def route(self, **request: Any) -> ProbeRoute: + """Rank runnable probes for a request (see :meth:`ProbeRouter.route`).""" + return ProbeRouter(overrides=self.overrides).route(self.model, self.data, **request) + + # -- execution ------------------------------------------------------ + def call(self, name: str, params: "Mapping[str, Any] | None" = None, + data: Any = None) -> "Result": + """Run probe *name* with JSON *params* on *data* (default: the bound batch).""" + batch = self.data if data is None else data + if batch is None: + raise ProbeCallError(f"{name}: no data to probe", "unavailable") + analyzer = build_probe(name, params, overrides=self.overrides) + verdict = availability(spec_for(name), self.model, batch, overrides=self.overrides) + if not verdict.available: + raise ProbeCallError(f"{name} cannot run here: {'; '.join(verdict.missing)}", + "unavailable") + return analyzer.run(self.model, self._capped(batch)) + + def tools(self, names: "Iterable[str] | None" = None, *, + only_available: bool = True) -> list[Tool]: + """Probes as :class:`~evalrx.core.tool.Tool` objects bound to this toolbox.""" + report = self.describe() + chosen = list(names) if names is not None else sorted(report) + out: list[Tool] = [] + for name in chosen: + if name not in report: + raise ProbeCallError(f"unknown probe {name!r}", "unknown_probe") + if only_available and not report[name]["available"]: + continue + out.append(spec_for(name).to_tool(fn=self._tool_fn(name))) + return out + + # ------------------------------------------------------------------ + def _tool_fn(self, name: str): + def run(**params: Any) -> ToolResult: + result = self.call(name, params) + return ToolResult(text=summarize_result(result), meta={"result": result}) + + run.__name__ = name + return run + + def _capped(self, batch: Any) -> Any: + if self.max_cases <= 0: + return batch + head = getattr(batch, "stratified_head", None) + try: + if callable(head) and len(batch) > self.max_cases: + return head(self.max_cases) + except TypeError: + pass + return batch + + +def summarize_result(result: "Result", max_chars: int = 1500) -> str: + """Scalar findings of a result as compact text (no per-case rows).""" + findings = getattr(result, "findings", None) or {} + scalars = {k: v for k, v in findings.items() + if isinstance(v, (int, float, bool, str)) and not str(k).startswith("_")} + n_rows = len(findings.get("per_case", []) or []) + text = f"[{getattr(result, 'analyzer', '?')}] {scalars}" + if n_rows: + text += f" ({n_rows} per-case rows)" + return text if len(text) <= max_chars else text[: max_chars - 1] + "…" diff --git a/evalrx/wsp/__init__.py b/evalrx/wsp/__init__.py new file mode 100644 index 00000000..9783d055 --- /dev/null +++ b/evalrx/wsp/__init__.py @@ -0,0 +1,76 @@ +"""Weight-space programming: build, apply, control and inspect direct weight edits. + + from evalrx.wsp import WeightEdit, TensorEdit, applied, edit_structure + + edit = WeightEdit([TensorEdit.rank_one("layers.15.mlp.down_proj.weight", u, alpha=6.0, + cols=columns)]) + edit_structure(model, edit) # what changed, read from the weights + with applied(model, edit): # written in, then restored exactly + answer = model.generate(case.inputs) + +The behavioural audit lives with the analyzers +(:mod:`evalrx.analyzers.weights`), and the repair executor with the fix stage +(:mod:`evalrx.eval_agent.stages.fix.weight_edit`). +""" + +from evalrx.wsp.continuation import ( + ContinuationSequence, + ContinuationStats, + encode_continuation, + encode_continuations, + repetition_mask, +) +from evalrx.wsp.controls import CONTROL_KINDS, build_control +from evalrx.wsp.direction import ( + WriteSite, + collect_continuation_stats, + collect_write_stats, + continuation_contrast_edit, + contrast_edit, + edit_from_stats, + output_shift, + rank_columns, + write_sites, +) +from evalrx.wsp.edit import ( + AppliedEdit, + TensorEdit, + WeightEdit, + applied, + bf16_bits_to_f32, + module_of, + resolve_parameters, + tensor_sha256, +) +from evalrx.wsp.structure import delta_structure, edit_agreement, edit_structure, fit_rank_one + +__all__ = [ + "CONTROL_KINDS", + "AppliedEdit", + "ContinuationSequence", + "ContinuationStats", + "TensorEdit", + "WeightEdit", + "WriteSite", + "applied", + "bf16_bits_to_f32", + "build_control", + "collect_continuation_stats", + "collect_write_stats", + "continuation_contrast_edit", + "contrast_edit", + "delta_structure", + "edit_agreement", + "edit_from_stats", + "edit_structure", + "encode_continuation", + "encode_continuations", + "fit_rank_one", + "module_of", + "output_shift", + "rank_columns", + "repetition_mask", + "resolve_parameters", + "tensor_sha256", + "write_sites", +] diff --git a/evalrx/wsp/continuation.py b/evalrx/wsp/continuation.py new file mode 100644 index 00000000..a16b7ca1 --- /dev/null +++ b/evalrx/wsp/continuation.py @@ -0,0 +1,149 @@ +"""Prompt + recorded generation as one token sequence, with positions grouped. + +Some failures only exist during generation: a repetition loop starts thousands +of tokens after the prompt. To read or score the model there, the recorded +output of a case (``case.observed``) is appended to its prompt and the whole +sequence is run teacher-forced. This module builds that sequence and labels its +positions; the continuation audit and the continuation contrast both use it, so +they agree on what "a loop position" is. + +Conventions. ``ids`` is the full sequence, ``start`` the index of the first +continuation token. A *target* ``i >= start`` is a continuation token; it is +predicted from the model state at position ``i - 1``. Prompt tokens are never +targets. +""" + +from __future__ import annotations + +from collections import Counter +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Iterable + +if TYPE_CHECKING: + from evalrx.core.model import Model + +GROUPINGS = ("repetition", "none") + + +def repetition_mask(ids: "list[int]", n: int) -> "list[bool]": + """``mask[i]`` is True when the *n* tokens before index *i* occurred earlier. + + The window ``ids[i-n:i]`` is compared with every earlier window of the same + length, so ``mask[i]`` describes the context in which token *i* is + predicted: inside a stretch the sequence has already produced once. + """ + seen: set[tuple[int, ...]] = set() + mask = [False] * len(ids) + for i in range(n, len(ids)): + window = tuple(ids[i - n:i]) + mask[i] = window in seen + seen.add(window) + return mask + + +def continuation_text(case: Any) -> "str | None": + """The recorded output of *case* as text, or ``None`` when there is none.""" + observed = getattr(case, "observed", None) + if observed is None: + return None + text = observed if isinstance(observed, str) else str(observed) + return text if text.strip() else None + + +@dataclass +class ContinuationSequence: + """One case's token sequence with its grouped targets.""" + + case: Any + ids: "list[int]" + start: int + targets: "list[int]" + labels: "list[str]" + + def positions(self, group: "str | None" = None) -> "list[int]": + """State positions (``target - 1``) of every target, or of one group's.""" + return [i - 1 for i, lab in zip(self.targets, self.labels) + if group is None or lab == group] + + +def group_names(groups: str) -> "tuple[str, ...]": + if groups not in GROUPINGS: + raise ValueError(f"groups must be one of {GROUPINGS}, got {groups!r}") + return ("loop", "productive") if groups == "repetition" else ("all",) + + +def encode_continuation(model: "Model", case: Any, *, groups: str = "repetition", + repeat_ngram: int = 32, max_tokens: int = 4096, + ) -> "ContinuationSequence | str": + """Build the sequence for one case, or return the reason it cannot be built. + + The prompt is tokenised by the model handle (one short forward pass, so the + handle's own template and special-token rules apply); the continuation is + appended as plain tokens. Reasons: ``no_continuation``, ``forward_failed``, + ``has_media``, ``continuation_untokenizable``, + ``no_continuation_tokens_after_truncation``. + """ + from evalrx.analyzers._internals.access import media_positions, tokenizer_of + from evalrx.core.capability import Capability + + names = group_names(groups) + text = continuation_text(case) + if text is None: + return "no_continuation" + tokenizer = tokenizer_of(model) + if tokenizer is None: + raise ValueError("the model handle exposes no tokenizer to encode the continuation with") + try: + trace = model.forward(getattr(case, "inputs", case), capture={Capability.LOGITS}) + except Exception: # noqa: BLE001 - an unreadable case is reported, not fatal + return "forward_failed" + if media_positions(trace)[1]: + return "has_media" + prompt_ids = [int(t) for t in trace.token_ids] + try: + cont = tokenizer(text, add_special_tokens=False)["input_ids"] + except Exception: # noqa: BLE001 + return "continuation_untokenizable" + if cont and isinstance(cont[0], (list, tuple)): + cont = cont[0] + ids = (prompt_ids + [int(t) for t in cont])[: int(max_tokens)] + start = len(prompt_ids) + if len(ids) <= start: + return "no_continuation_tokens_after_truncation" + targets = list(range(max(start, 1), len(ids))) + if groups == "repetition": + mask = repetition_mask(ids, int(repeat_ngram)) + labels = [names[0] if mask[i] else names[1] for i in targets] + else: + labels = [names[0]] * len(targets) + return ContinuationSequence(case=case, ids=ids, start=start, targets=targets, labels=labels) + + +def encode_continuations(model: "Model", cases: Iterable[Any], **kwargs: Any, + ) -> "tuple[list[ContinuationSequence], Counter]": + """:func:`encode_continuation` over *cases*: ``(sequences, skip reasons)``.""" + sequences: list[ContinuationSequence] = [] + skipped: Counter = Counter() + for case in cases: + built = encode_continuation(model, case, **kwargs) + if isinstance(built, str): + skipped[built] += 1 + else: + sequences.append(built) + return sequences, skipped + + +@dataclass +class ContinuationStats: + """Mean MLP input and output per position group and layer. + + ``groups[name][layer] = {"in": [d_mlp], "out": [d_model], "n": positions}``. + """ + + groups: "dict[str, dict[int, dict[str, Any]]]" = field(default_factory=dict) + n_cases: int = 0 + skipped: "dict[str, int]" = field(default_factory=dict) + + def n_positions(self, group: str) -> int: + layers = self.groups.get(group) or {} + return next(iter(layers.values()))["n"] if layers else 0 diff --git a/evalrx/wsp/controls.py b/evalrx/wsp/controls.py new file mode 100644 index 00000000..18d173f4 --- /dev/null +++ b/evalrx/wsp/controls.py @@ -0,0 +1,107 @@ +"""Control edits — what an edit must be compared against before it is explained. + +An edit that changes behaviour proves little on its own: a change of the same +size in a random direction, or the same rule applied to other neurons, may do +the same. Each control here keeps everything about the edit except the one +property under test. + +``random_direction`` + Same columns, same per-column change norm, same sign pattern, but along a + random unit direction instead of the edit's own. Tests whether the + DIRECTION matters. +``random_columns`` + The same change moved to an equal number of other, randomly chosen columns. + For a rank-one rule the rule is re-applied to those columns; otherwise each + changed column's delta is moved. Tests whether the choice of NEURONS matters. + +Controls are built against the live weights and returned as dense-delta +:class:`~evalrx.wsp.edit.WeightEdit` objects, so they are applied by the same +code path as the edit they control for. +""" + +from __future__ import annotations + +from typing import Any + +from evalrx.wsp.edit import TensorEdit, WeightEdit, module_of, resolve_parameters + +CONTROL_KINDS = ("random_direction", "random_columns") + + +def _generator(seed: int) -> Any: + import torch + + return torch.Generator().manual_seed(int(seed)) + + +def _random_direction(d: Any, gen: Any) -> Any: + """Norm- and sign-matched change along one random unit direction.""" + import torch + + cols = torch.nonzero((d != 0).any(dim=0)).flatten() + if cols.numel() == 0: + return torch.zeros_like(d) + dc = d[:, cols] + u = torch.linalg.svd(dc, full_matrices=False)[0][:, 0] + signs = torch.sign(u @ dc) + signs[signs == 0] = 1.0 + r = torch.randn(d.shape[0], generator=gen, dtype=torch.float64) + r = r / r.norm() + out = torch.zeros_like(d) + out[:, cols] = torch.outer(r, dc.norm(dim=0) * signs) + return out + + +def _random_columns(w0: Any, d: Any, tensor_edit: TensorEdit, gen: Any) -> "Any | None": + """The edit's change on an equal number of other columns; ``None`` when there are too few.""" + import torch + + cols = torch.nonzero((d != 0).any(dim=0)).flatten() + free = torch.ones(d.shape[1], dtype=torch.bool) + free[cols] = False + pool = torch.nonzero(free).flatten() + if cols.numel() == 0 or pool.numel() < cols.numel(): + return None + picked = pool[torch.randperm(pool.numel(), generator=gen)[: cols.numel()]] + out = torch.zeros_like(d) + pure_rank_one = (tensor_edit.u is not None and tensor_edit.a is None + and tensor_edit.delta is None and tensor_edit.index is None) + if pure_rank_one: + u = torch.from_numpy(tensor_edit.u).double() + out[:, picked] = -tensor_edit.alpha * torch.outer(u, u @ w0[:, picked]) + else: + out[:, picked] = d[:, cols] + return out + + +def build_control(model: Any, edit: WeightEdit, kind: str, seed: int = 0) -> WeightEdit: + """Build one control for *edit* against the model's current weights. + + Raises ``ValueError`` for an unknown *kind*, a non-2-D edited tensor, or a + ``random_columns`` request on a tensor with no unchanged columns left. + """ + import torch + + if kind not in CONTROL_KINDS: + raise ValueError(f"unknown control {kind!r}; expected one of {CONTROL_KINDS}") + module = module_of(model) + gen = _generator(seed) + tensors: list[TensorEdit] = [] + with torch.no_grad(): + for tensor_edit, (key, param) in zip(edit.tensors, resolve_parameters(module, edit)): + if param.ndim != 2: + raise ValueError(f"{key}: controls are defined for 2-D weights") + w0 = param.detach().double().cpu() + d = tensor_edit.apply_to(param).double().cpu() - w0 + if kind == "random_direction": + control = _random_direction(d, gen) + else: + control = _random_columns(w0, d, tensor_edit, gen) + if control is None: + raise ValueError( + f"{key}: no unchanged columns left for a random_columns control" + ) + tensors.append(TensorEdit(name=key, form="delta", delta=control.float(), + shape=tuple(param.shape))) + return WeightEdit(tensors, name=f"{edit.name}:{kind}:{seed}", + metadata={"control_for": edit.name, "kind": kind, "seed": int(seed)}) diff --git a/evalrx/wsp/direction.py b/evalrx/wsp/direction.py new file mode 100644 index 00000000..4aa73983 --- /dev/null +++ b/evalrx/wsp/direction.py @@ -0,0 +1,417 @@ +"""Derive a rank-one weight edit from a contrast between two groups of cases. + +A weight edit needs a direction and a set of neurons. When no one has written +the edit by hand, both can be read from the model's own activations on labelled +cases, without any gradient step: + +* the DIRECTION at a layer is the difference between the mean MLP output on + failing cases and on passing cases (unit length) — the component of the + layer's write that separates the two groups; +* the NEURONS are the columns of the MLP output projection that push hardest + along that direction on failing cases relative to passing ones: + ``score_j = (mean_fail_j - mean_pass_j) * (u^T w_j)``. + +The resulting edit ``w_j' = w_j - alpha * u (u^T w_j)`` on those columns removes +(``alpha = 1``) or reverses (``alpha > 1``) exactly that push. + +This is a heuristic for PROPOSING an edit. It says nothing about whether the +edit helps: that is decided by validating it on cases the direction was not +computed from, against controls (:mod:`evalrx.wsp.controls`). + +Activations are read at the last prompt position, where the first answer token +is predicted. A failure that only develops during generation (for example a +repetition loop) is not visible there. +""" + +from __future__ import annotations + +from contextlib import contextmanager +from dataclasses import dataclass, field +from typing import TYPE_CHECKING, Any, Iterable, Iterator + +from evalrx.wsp.edit import TensorEdit, WeightEdit, module_of + +if TYPE_CHECKING: + from evalrx.core.model import Model + +#: Attribute paths of the MLP output projection inside a decoder layer, most +#: common first. The projection must be an ``nn.Linear`` whose weight is +#: ``[d_model, d_mlp]``. +_WRITE_PATHS = ("mlp.down_proj", "mlp.fc2", "mlp.c_proj", "mlp.fc_out", "feed_forward.w2", + "feed_forward.down_proj", "ffn.down_proj") + + +@dataclass(frozen=True) +class WriteSite: + """The MLP output projection of one decoder layer.""" + + layer: int + module: Any = field(repr=False, compare=False) + name: str = "" # full parameter name of the weight + + +def write_sites(model: Any) -> "list[WriteSite]": + """MLP output projections of the language decoder, one per layer that has one. + + Layers whose MLP is not a single linear write (for example a + mixture-of-experts block) are skipped rather than guessed at. + """ + import torch.nn as nn + + from evalrx.analyzers._internals.access import decoder_stack + + module = module_of(model) + stack = decoder_stack(module) + names = {id(p): n for n, p in module.named_parameters()} + sites: list[WriteSite] = [] + for index, layer in enumerate(stack.layers): + for path in _WRITE_PATHS: + obj = layer + for part in path.split("."): + obj = getattr(obj, part, None) + if obj is None: + break + if isinstance(obj, nn.Linear) and id(obj.weight) in names: + sites.append(WriteSite(layer=index, module=obj, name=names[id(obj.weight)])) + break + return sites + + +@contextmanager +def _capture_last(sites: "list[WriteSite]") -> "Iterator[dict[int, dict[str, Any]]]": + """Record each site's input and output at the last sequence position.""" + captured: dict[int, dict[str, Any]] = {s.layer: {} for s in sites} + + def make(layer: int): + def hook(_module: Any, args: Any, output: Any) -> None: + captured[layer]["in"] = args[0].detach()[0, -1].double().cpu() + captured[layer]["out"] = output.detach()[0, -1].double().cpu() + return hook + + handles = [s.module.register_forward_hook(make(s.layer)) for s in sites] + try: + yield captured + finally: + for handle in handles: + handle.remove() + + +def collect_write_stats(model: "Model", cases: Iterable[Any], *, + layers: "Iterable[int] | None" = None) -> "dict[int, dict[str, Any]]": + """Mean MLP input (neuron activations) and output at the last position. + + Returns ``{layer: {"in": mean activations [d_mlp], "out": mean write + [d_model], "n": cases}}``. Cases whose forward pass fails are skipped. + """ + from evalrx.core.capability import Capability + + sites = write_sites(model) + if layers is not None: + wanted = {int(i) for i in layers} + sites = [s for s in sites if s.layer in wanted] + sums: dict[int, dict[str, Any]] = {s.layer: {"in": None, "out": None, "n": 0} for s in sites} + for case in cases: + with _capture_last(sites) as captured: + try: + model.forward(getattr(case, "inputs", case), capture={Capability.LOGITS}) + except Exception: # noqa: BLE001 - one unreadable case must not end the pass + continue + for layer, got in captured.items(): + if "in" not in got: + continue + acc = sums[layer] + acc["in"] = got["in"] if acc["in"] is None else acc["in"] + got["in"] + acc["out"] = got["out"] if acc["out"] is None else acc["out"] + got["out"] + acc["n"] += 1 + return { + layer: {"in": acc["in"] / acc["n"], "out": acc["out"] / acc["n"], "n": acc["n"]} + for layer, acc in sums.items() if acc["n"] + } + + +def rank_columns(stats_a: "dict[str, Any]", stats_b: "dict[str, Any]", weight: Any, + ) -> "dict[str, Any]": + """Direction and per-neuron scores for one layer from two groups' mean activity. + + ``stats_a`` is the group to move away from (failing cases, loop positions), + ``stats_b`` the reference (passing cases, productive positions). Returns + + ``u`` unit difference of the mean writes, ``a - b``; + ``push`` ``(act_a - act_b) * (u^T w_j)`` — how much more neuron j + writes along ``u`` in group a than in group b; + ``projection`` ``u^T w_j`` — what neuron j writes along ``u`` when active; + ``activation`` ``act_a - act_b`` — how much more neuron j fires in group a; + ``separation`` ``||a - b||`` relative to the mean size of the two writes. + + ``u`` is ``None`` when the two groups write the same thing. + """ + gap = stats_a["out"] - stats_b["out"] + scale = 0.5 * (stats_a["out"].norm() + stats_b["out"].norm()) + separation = float(gap.norm() / scale) if float(scale) > 0 else 0.0 + if float(gap.norm()) == 0.0: + return {"u": None, "separation": separation} + u = gap / gap.norm() + projection = u @ weight.detach().double().cpu() + activation = stats_a["in"] - stats_b["in"] + return {"u": u, "push": activation * projection, "projection": projection, + "activation": activation, "separation": separation} + + +def edit_from_stats( + model: Any, + stats_a: "dict[int, dict[str, Any]]", + stats_b: "dict[int, dict[str, Any]]", + *, + layers: "Iterable[int] | None" = None, + n_layers: int = 2, + n_columns: int = 64, + alpha: float = 1.0, + name: str = "contrast_rank_one", + metadata: "dict[str, Any] | None" = None, +) -> WeightEdit: + """Build the rank-one edit that removes what group a writes beyond group b. + + The shared core of :func:`contrast_edit` and + :func:`continuation_contrast_edit`; *stats_a* / *stats_b* are + ``{layer: {"in", "out", "n"}}`` as the collectors return them. + """ + import torch + + sites = {s.layer: s for s in write_sites(model)} + if not sites: + raise ValueError("no MLP output projection found in the decoder layers") + chosen = None if layers is None else [int(i) for i in layers] + if chosen is not None: + unknown = sorted(set(chosen) - set(sites)) + if unknown: + raise ValueError(f"layers {unknown} have no MLP output projection " + f"(available: {sorted(sites)})") + shared = sorted(set(stats_a) & set(stats_b) & set(sites)) + if not shared: + raise ValueError("need at least one readable FAIL case and one readable PASS case") + ranked = {layer: rank_columns(stats_a[layer], stats_b[layer], sites[layer].module.weight) + for layer in shared} + separation = {layer: ranked[layer]["separation"] for layer in shared} + if chosen is None: + chosen = sorted(sorted(shared, key=lambda i: -separation[i])[: max(1, int(n_layers))]) + else: + chosen = [i for i in chosen if i in separation] + + tensors: list[TensorEdit] = [] + picked: dict[str, Any] = {} + for layer in chosen: + row = ranked[layer] + if row["u"] is None: + continue + push = row["push"] + k = min(int(n_columns), push.numel()) + cols = torch.topk(push, k).indices + cols = cols[push[cols] > 0] # only neurons that push TOWARD group a + if cols.numel() == 0: + continue + tensors.append(TensorEdit.rank_one(sites[layer].name, row["u"], alpha, + cols=cols.sort().values)) + picked[str(layer)] = {"n_columns": int(cols.numel()), + "separation": round(separation[layer], 6)} + if not tensors: + raise ValueError("the FAIL and PASS groups do not differ at any candidate layer") + return WeightEdit(tensors, name=name, metadata={ + "source": "contrast", "alpha": float(alpha), "layers": picked, + "layer_separation": {str(i): round(separation[i], 6) for i in shared}, + **(metadata or {}), + }) + + +def contrast_edit( + model: "Model", + fail_cases: Iterable[Any], + pass_cases: Iterable[Any], + *, + layers: "Iterable[int] | None" = None, + n_layers: int = 2, + n_columns: int = 64, + alpha: float = 1.0, + name: str = "contrast_rank_one", +) -> WeightEdit: + """Build a rank-one edit from the FAIL-minus-PASS contrast (see module docstring). + + Args: + model: White-box model handle. + fail_cases: Cases the model gets wrong. + pass_cases: Cases the model gets right. + layers: Decoder layers to edit. Default: the *n_layers* layers + whose mean writes differ most between the groups, relative + to the size of the write. + n_layers: How many layers to pick when *layers* is not given. + n_columns: Neurons edited per layer (capped at the layer's width). + alpha: Strength of ``w' = w - alpha u (u^T w)``. + + Raises: + ValueError: when either group is empty after the forward passes, or the + model has no recognisable MLP output projection. + """ + chosen = None if layers is None else [int(i) for i in layers] + available = {s.layer for s in write_sites(model)} + if not available: + raise ValueError("no MLP output projection found in the decoder layers") + if chosen is not None and set(chosen) - available: + raise ValueError(f"layers {sorted(set(chosen) - available)} have no MLP output " + f"projection (available: {sorted(available)})") + fail = collect_write_stats(model, fail_cases, layers=chosen) + passed = collect_write_stats(model, pass_cases, layers=chosen) + shared = sorted(set(fail) & set(passed)) + return edit_from_stats( + model, fail, passed, layers=chosen, n_layers=n_layers, n_columns=n_columns, + alpha=alpha, name=name, + metadata={"n_fail": min((fail[i]["n"] for i in shared), default=0), + "n_pass": min((passed[i]["n"] for i in shared), default=0), + "position": "last_prompt_token"}, + ) + + +def collect_continuation_stats( + model: "Model", + cases: Iterable[Any], + *, + groups: str = "repetition", + repeat_ngram: int = 32, + max_tokens: int = 4096, + layers: "Iterable[int] | None" = None, +) -> Any: + """Mean MLP input and output per position group along recorded generations. + + Each case's recorded output is appended to its prompt + (:mod:`evalrx.wsp.continuation`) and run once; at every layer the MLP + output projection's input and output are averaged over the positions of + each group (``loop`` / ``productive`` with ``groups="repetition"``). + + Returns a :class:`~evalrx.wsp.continuation.ContinuationStats`. + """ + import torch + + from evalrx.wsp.continuation import ContinuationStats, encode_continuations, group_names + + names = group_names(groups) + sites = write_sites(model) + if layers is not None: + wanted = {int(i) for i in layers} + sites = [s for s in sites if s.layer in wanted] + module = module_of(model) + device = next(module.parameters()).device + sequences, skipped = encode_continuations( + model, cases, groups=groups, repeat_ngram=repeat_ngram, max_tokens=max_tokens) + + sums = {g: {s.layer: {"in": None, "out": None, "n": 0} for s in sites} for g in names} + current: dict[str, Any] = {} + + def make(layer: int): + def hook(_module: Any, args: Any, output: Any) -> None: + x = args[0].detach()[0].double() + y = output.detach()[0].double() + for g, index in current["index"].items(): + if not len(index): + continue + acc = sums[g][layer] + xin, yout = x[index].sum(dim=0).cpu(), y[index].sum(dim=0).cpu() + acc["in"] = xin if acc["in"] is None else acc["in"] + xin + acc["out"] = yout if acc["out"] is None else acc["out"] + yout + acc["n"] += int(len(index)) + return hook + + handles = [s.module.register_forward_hook(make(s.layer)) for s in sites] + try: + for seq in sequences: + current["index"] = {g: torch.tensor(seq.positions(g), dtype=torch.long, device=device) + for g in names} + with torch.no_grad(): + module(input_ids=torch.tensor([seq.ids], device=device)) + finally: + for handle in handles: + handle.remove() + return ContinuationStats( + groups={g: {layer: {"in": acc["in"] / acc["n"], "out": acc["out"] / acc["n"], + "n": acc["n"]} + for layer, acc in per_layer.items() if acc["n"]} + for g, per_layer in sums.items()}, + n_cases=len(sequences), skipped=dict(skipped), + ) + + +def continuation_contrast_edit( + model: "Model", + cases: Iterable[Any], + *, + repeat_ngram: int = 32, + max_tokens: int = 4096, + layers: "Iterable[int] | None" = None, + n_layers: int = 2, + n_columns: int = 64, + alpha: float = 1.0, + name: str = "loop_contrast_rank_one", + stats: Any = None, +) -> WeightEdit: + """Build a rank-one edit from the contrast between loop and productive positions. + + The counterpart of :func:`contrast_edit` for a failure that only exists + during generation. Instead of failing vs passing cases at the end of the + prompt, it contrasts positions INSIDE a repeated stretch with positions + outside one, along the recorded generations of *cases* + (``case.observed``): the direction is the mean MLP write in loops minus the + mean write elsewhere, and the neurons are the ones that write along it more + in loops. + + Pass *stats* (from :func:`collect_continuation_stats`) to reuse one + collection pass for several settings. + + Like :func:`contrast_edit` this proposes an edit; it does not show the edit + helps. Validate it on generations it was not derived from. + """ + if stats is None: + stats = collect_continuation_stats(model, cases, groups="repetition", + repeat_ngram=repeat_ngram, max_tokens=max_tokens, + layers=layers) + loop, productive = stats.groups.get("loop") or {}, stats.groups.get("productive") or {} + if not loop or not productive: + raise ValueError( + "need both loop and productive positions; found " + f"{stats.n_positions('loop')} loop and {stats.n_positions('productive')} productive " + f"in {stats.n_cases} generation(s) (skipped: {stats.skipped})" + ) + return edit_from_stats( + model, loop, productive, layers=layers, n_layers=n_layers, n_columns=n_columns, + alpha=alpha, name=name, + metadata={"n_loop_positions": stats.n_positions("loop"), + "n_productive_positions": stats.n_positions("productive"), + "n_generations": stats.n_cases, "repeat_ngram": int(repeat_ngram), + "position": "continuation"}, + ) + + +@contextmanager +def output_shift(model: Any, shifts: "dict[str, Any]") -> Iterator[None]: + """Add a constant vector to the output of chosen linear layers, weights untouched. + + *shifts* maps a weight's full parameter name (as in + :attr:`WriteSite.name`) to a vector of the layer's output size. This is the + activation-steering counterpart of a weight edit: if a constant shift + reproduces the edit's effect, the edit did not need to depend on the input. + """ + import torch + + module = module_of(model) + owners = {f"{n}.weight" if n else "weight": m for n, m in module.named_modules() + if isinstance(m, torch.nn.Linear)} + handles = [] + try: + for key, vector in shifts.items(): + if key not in owners: + raise KeyError(f"no linear layer owns parameter {key!r}") + + def hook(_module: Any, _args: Any, output: Any, vector: Any = vector) -> Any: + return output + vector.to(device=output.device, dtype=output.dtype) + + handles.append(owners[key].register_forward_hook(hook)) + yield + finally: + for handle in handles: + handle.remove() diff --git a/evalrx/wsp/edit.py b/evalrx/wsp/edit.py new file mode 100644 index 00000000..219a9c34 --- /dev/null +++ b/evalrx/wsp/edit.py @@ -0,0 +1,520 @@ +"""Weight edits — direct, training-free changes to a model's parameters. + +Weight-space programming (WSP) changes a model by writing new values into a few +weight tensors instead of training. The edits studied so far are small and +structured: a rank-one change to some columns of one MLP output projection, +``w' = w - alpha * u (u^T w)``, or a sparse set of overwritten entries. + +This module is the data model and the one place such an edit touches a model: + +* :class:`TensorEdit` — the change to one tensor, stored in the form it was + written in (rank-one rule, low-rank factors, dense delta, sparse overwrite); +* :class:`WeightEdit` — a named set of tensor edits with provenance; +* :func:`applied` — a context manager that writes the edit into a live module + and restores the exact original values on exit, including after an error. + +Reading the edit files published with *Sparks of Weight-Space Programming* +(``edit.npz`` + ``manifest.json``) is supported by :meth:`WeightEdit.from_astra`. + +Torch is imported lazily; building, saving and loading edits needs numpy only. +""" + +from __future__ import annotations + +import hashlib +import json +import os +from contextlib import contextmanager +from dataclasses import dataclass, field +from typing import Any, Iterator, Mapping + +import numpy as np + +FORMAT_VERSION = 1 + + +def _f32(x: Any) -> "np.ndarray | None": + if x is None: + return None + if hasattr(x, "detach"): # torch tensor + x = x.detach().float().cpu().numpy() + return np.ascontiguousarray(x, dtype=np.float32) + + +def _index(x: Any) -> "np.ndarray | None": + if x is None: + return None + if hasattr(x, "detach"): + x = x.detach().cpu().numpy() + return np.ascontiguousarray(x, dtype=np.int64).reshape(-1) + + +def bf16_bits_to_f32(bits: Any) -> np.ndarray: + """bfloat16 bit patterns (uint16) -> float32 values.""" + return (np.asarray(bits).astype(np.uint32) << 16).view(np.float32) + + +@dataclass +class TensorEdit: + """The change to one weight tensor. + + The components are applied in this order, each one optional: + + 1. rank-one rule ``W[:, cols] -= alpha * u (u^T W[:, cols])`` + (``u`` is normalised to unit length; ``cols=None`` means every column); + 2. low-rank update ``W += A @ B`` (``A``: ``[rows, k]``, ``B``: ``[k, cols]``); + 3. dense delta ``W += delta``; + 4. sparse overwrite ``W.flat[index] = values``. + + The rank-one rule is defined for a 2-D weight whose ROWS index the space + ``u`` lives in. For an MLP output projection (``[d_model, d_mlp]``) that is + the residual stream, and each column is one neuron's write vector. + + Attributes: + name: Parameter name inside the module, or a unique suffix of it. + form: Label of the leading component: ``rank_one`` | ``low_rank`` | + ``delta`` | ``sparse_set``. Informational; application is driven + by which components are present. + base_sha256: When known, the hash of the tensor the edit was made on. + """ + + name: str + form: str = "delta" + u: "np.ndarray | None" = None + alpha: float = 0.0 + cols: "np.ndarray | None" = None + a: "np.ndarray | None" = None + b: "np.ndarray | None" = None + delta: "np.ndarray | None" = None + index: "np.ndarray | None" = None + values: "np.ndarray | None" = None + shape: "tuple[int, ...] | None" = None + base_sha256: str = "" + edited_sha256: str = "" + + def __post_init__(self) -> None: + self.u, self.a, self.b = _f32(self.u), _f32(self.a), _f32(self.b) + self.delta, self.values = _f32(self.delta), _f32(self.values) + self.cols, self.index = _index(self.cols), _index(self.index) + self.alpha = float(self.alpha) + if self.shape is not None: + self.shape = tuple(int(s) for s in self.shape) + if self.u is not None: + self.u = self.u.reshape(-1) + norm = float(np.linalg.norm(self.u)) + if not np.isfinite(norm) or norm == 0.0: + raise ValueError(f"{self.name}: the edit direction u must be finite and non-zero") + self.u = self.u / norm + if self.values is not None: + self.values = self.values.reshape(-1) + if (self.index is None) != (self.values is None): + raise ValueError(f"{self.name}: index and values must be given together") + if self.index is not None and len(self.index) != len(self.values): + raise ValueError(f"{self.name}: index and values differ in length") + if (self.a is None) != (self.b is None): + raise ValueError(f"{self.name}: low-rank factors a and b must be given together") + + # -- constructors --------------------------------------------------- + @classmethod + def rank_one(cls, name: str, u: Any, alpha: float, cols: Any = None) -> "TensorEdit": + """``W[:, cols] -= alpha * u (u^T W[:, cols])``. + + ``alpha=1`` removes the component along ``u``; ``alpha=2`` reflects it; + larger values reflect and amplify (``alpha=6``: flipped, five times larger). + """ + return cls(name=name, form="rank_one", u=u, alpha=alpha, cols=cols) + + # -- application ---------------------------------------------------- + @property + def is_empty(self) -> bool: + return (self.u is None and self.a is None and self.delta is None + and self.index is None) + + def apply_to(self, weight: Any) -> Any: + """Return the edited tensor (same dtype and device as *weight*; *weight* is untouched).""" + import torch + + if self.shape is not None and tuple(weight.shape) != self.shape: + raise ValueError( + f"{self.name}: the edit was made for shape {self.shape}, " + f"the tensor has shape {tuple(weight.shape)}" + ) + # float64 so that composing components does not accumulate rounding + # before the single cast back to the tensor's own dtype. + w = weight.detach().to(torch.float64).clone() + dev = w.device + + def t(x: np.ndarray) -> Any: + return torch.from_numpy(x).to(device=dev, dtype=torch.float64) + + if self.u is not None and self.alpha != 0.0: + if w.ndim != 2: + raise ValueError(f"{self.name}: the rank-one rule needs a 2-D weight") + u = t(self.u) + if u.numel() != w.shape[0]: + raise ValueError( + f"{self.name}: u has {u.numel()} entries, the weight has {w.shape[0]} rows" + ) + if self.cols is None: + w = w - self.alpha * torch.outer(u, u @ w) + else: + cols = torch.from_numpy(self.cols).to(dev) + if int(cols.max()) >= w.shape[1] or int(cols.min()) < 0: + raise ValueError(f"{self.name}: column index out of range") + sub = w[:, cols] + w[:, cols] = sub - self.alpha * torch.outer(u, u @ sub) + if self.a is not None: + w = w + (t(self.a) @ t(self.b)).reshape(w.shape) + if self.delta is not None: + w = w + t(self.delta).reshape(w.shape) + if self.index is not None: + flat = w.reshape(-1).clone() + idx = torch.from_numpy(self.index).to(dev) + if int(idx.max()) >= flat.numel() or int(idx.min()) < 0: + raise ValueError(f"{self.name}: flat index out of range") + flat[idx] = t(self.values) + w = flat.reshape(w.shape) + # Through float32 on the way down: edits are written as float32 values + # and then cast, and going straight from float64 to a 16-bit type can + # round a tie differently from that convention. + return w.to(torch.float32).to(weight.dtype) + + def scaled(self, factor: float) -> "TensorEdit": + """The same edit at *factor* times its strength. + + Exact for the rank-one, low-rank and dense components. A sparse + overwrite stores absolute values, not a change, so it cannot be scaled + without the base tensor and is refused. + """ + if self.index is not None: + raise ValueError( + f"{self.name}: a sparse overwrite cannot be scaled; convert it to a " + "dense delta against the base weights first (WeightEdit.as_delta)" + ) + return TensorEdit( + name=self.name, form=self.form, u=self.u, alpha=self.alpha * factor, cols=self.cols, + a=None if self.a is None else self.a * factor, b=self.b, + delta=None if self.delta is None else self.delta * factor, shape=self.shape, + ) + + # -- description ---------------------------------------------------- + def describe(self) -> dict[str, Any]: + out: dict[str, Any] = {"name": self.name, "form": self.form} + if self.shape is not None: + out["shape"] = list(self.shape) + if self.u is not None: + out["alpha"] = self.alpha + out["n_cols"] = None if self.cols is None else int(len(self.cols)) + if self.a is not None: + out["rank"] = int(self.a.shape[1]) if self.a.ndim == 2 else 1 + if self.index is not None: + out["n_overwritten"] = int(len(self.index)) + if self.base_sha256: + out["base_sha256"] = self.base_sha256 + return out + + _ARRAYS = ("u", "cols", "a", "b", "delta", "index", "values") + + +@dataclass +class WeightEdit: + """A set of tensor edits that together make one edit to a model. + + Attributes: + tensors: The per-tensor changes. + name: Short identifier for reports. + metadata: Free-form provenance (who wrote the edit, from what evidence). + """ + + tensors: list[TensorEdit] = field(default_factory=list) + name: str = "weight_edit" + metadata: dict[str, Any] = field(default_factory=dict) + + def __post_init__(self) -> None: + names = [t.name for t in self.tensors] + if len(set(names)) != len(names): + raise ValueError(f"{self.name}: a tensor is edited more than once: {sorted(names)}") + + def __len__(self) -> int: + return len(self.tensors) + + @property + def tensor_names(self) -> list[str]: + return [t.name for t in self.tensors] + + def scaled(self, factor: float) -> "WeightEdit": + return WeightEdit([t.scaled(factor) for t in self.tensors], + name=f"{self.name}_x{factor:g}", + metadata={**self.metadata, "scaled_by": factor}) + + def describe(self) -> dict[str, Any]: + """JSON-serialisable summary (no arrays).""" + return {"name": self.name, "n_tensors": len(self.tensors), + "tensors": [t.describe() for t in self.tensors], "metadata": self.metadata} + + # -- building from weights ------------------------------------------- + @classmethod + def from_state_dicts(cls, base: Mapping[str, Any], edited: Mapping[str, Any], *, + name: str = "weight_edit", atol: float = 0.0) -> "WeightEdit": + """The edit that turns *base* into *edited*, as one dense delta per changed tensor.""" + tensors: list[TensorEdit] = [] + for key, w0 in base.items(): + if key not in edited: + continue + d = _f32(edited[key]).astype(np.float64) - _f32(w0).astype(np.float64) + if d.shape and float(np.abs(d).max(initial=0.0)) > atol: + tensors.append(TensorEdit(name=key, form="delta", delta=d.astype(np.float32), + shape=tuple(d.shape))) + return cls(tensors, name=name, metadata={"source": "state_dict_difference"}) + + def as_delta(self, module: Any) -> "WeightEdit": + """This edit as dense deltas against the CURRENT weights of *module*. + + *module* may be a ``torch.nn.Module`` or a white-box model handle. + """ + module = module_of(module) + tensors = [] + for tensor_edit, (key, param) in zip(self.tensors, resolve_parameters(module, self)): + delta = (tensor_edit.apply_to(param).double() - param.detach().double()).float() + tensors.append(TensorEdit(name=key, form="delta", delta=delta, + shape=tuple(param.shape))) + return WeightEdit(tensors, name=self.name, metadata={**self.metadata, "as_delta": True}) + + # -- persistence ----------------------------------------------------- + def save(self, directory: str) -> str: + """Write ``manifest.json`` + ``edit.npz`` to *directory*; return the directory.""" + os.makedirs(directory, exist_ok=True) + arrays: dict[str, np.ndarray] = {} + manifest: dict[str, Any] = { + "format": "evalrx.weight_edit", "version": FORMAT_VERSION, + "name": self.name, "metadata": self.metadata, "tensors": [], + } + for i, t in enumerate(self.tensors): + entry: dict[str, Any] = { + "name": t.name, "form": t.form, "alpha": t.alpha, + "shape": None if t.shape is None else list(t.shape), + "base_sha256": t.base_sha256, "edited_sha256": t.edited_sha256, "arrays": [], + } + for attr in TensorEdit._ARRAYS: + value = getattr(t, attr) + if value is not None: + arrays[f"t{i}_{attr}"] = value + entry["arrays"].append(attr) + manifest["tensors"].append(entry) + np.savez_compressed(os.path.join(directory, "edit.npz"), **arrays) + with open(os.path.join(directory, "manifest.json"), "w", encoding="utf-8") as fh: + json.dump(manifest, fh, indent=1, default=str) + return directory + + @classmethod + def load(cls, directory: str) -> "WeightEdit": + """Read an edit written by :meth:`save` (or an Astra edit directory).""" + with open(os.path.join(directory, "manifest.json"), encoding="utf-8") as fh: + manifest = json.load(fh) + if manifest.get("format") != "evalrx.weight_edit": + return cls.from_astra(directory) + if int(manifest.get("version", 0)) > FORMAT_VERSION: + raise ValueError(f"{directory}: written by a newer format version") + with np.load(os.path.join(directory, "edit.npz"), allow_pickle=False) as z: + tensors = [] + for i, entry in enumerate(manifest["tensors"]): + arrays = {attr: z[f"t{i}_{attr}"] for attr in entry.get("arrays", [])} + tensors.append(TensorEdit( + name=entry["name"], form=entry.get("form", "delta"), + alpha=entry.get("alpha", 0.0), shape=entry.get("shape"), + base_sha256=entry.get("base_sha256", ""), + edited_sha256=entry.get("edited_sha256", ""), **arrays, + )) + return cls(tensors, name=manifest.get("name", "weight_edit"), + metadata=dict(manifest.get("metadata") or {})) + + @classmethod + def from_astra(cls, directory: str, *, name: "str | None" = None) -> "WeightEdit": + """Read an edit in the layout of the *Sparks of Weight-Space Programming* logs. + + ``manifest.json`` lists the edited tensors; ``edit.npz`` holds, per tensor + ``i``: for ``form == "lowrank"`` the factors ``t{i}_a`` (``[rows, k]``) and + ``t{i}_b`` (``[k, cols]``) added to the base; and for every form the flat + indices ``t{i}_idx`` whose final values are the bfloat16 bit patterns + ``t{i}_val``. Those overwritten entries are absolute values, so the edit + reproduces the evaluated checkpoint only on the base it was made on; the + recorded ``base_sha256`` is kept for :func:`applied` to check. + """ + with open(os.path.join(directory, "manifest.json"), encoding="utf-8") as fh: + manifest = json.load(fh) + tensors = [] + with np.load(os.path.join(directory, "edit.npz"), allow_pickle=False) as z: + for i, entry in enumerate(manifest["tensors"]): + lowrank = entry.get("form") == "lowrank" + tensors.append(TensorEdit( + name=entry["name"], + form="low_rank" if lowrank else "sparse_set", + a=z[f"t{i}_a"] if lowrank else None, + b=z[f"t{i}_b"] if lowrank else None, + index=z[f"t{i}_idx"] if f"t{i}_idx" in z.files else None, + values=(bf16_bits_to_f32(z[f"t{i}_val"]) + if f"t{i}_val" in z.files else None), + shape=entry.get("shape"), + base_sha256=entry.get("base_sha256", ""), + edited_sha256=entry.get("edited_sha256", ""), + )) + label = name or os.path.basename(os.path.dirname(os.path.abspath(directory))) or "astra" + return cls(tensors, name=label, + metadata={"source": "astra", "dtype": "bfloat16", + "changed_entries_total": manifest.get("changed_entries_total")}) + + +# ---------------------------------------------------------------------- +# Touching a live module +# ---------------------------------------------------------------------- + +def module_of(model: Any) -> Any: + """The ``torch.nn.Module`` to edit: *model* itself, or the one behind a white-box handle.""" + import torch.nn as nn + + if isinstance(model, nn.Module): + return model + from evalrx.analyzers._internals.access import torch_module + + return torch_module(model, "a weight edit") + + +def _anchored_suffix(name: str) -> str: + """The part of a parameter name that must match: from the indexed container on. + + ``model.language_model.layers.15.mlp.down_proj.weight`` anchors at + ``layers.15.mlp.down_proj.weight``. Wrapper prefixes before the container + differ between exports of the same model and may be dropped; the layer + index and everything after it may not, or an edit could land on another + layer. A name with no index has no droppable prefix. + """ + parts = name.split(".") + for i, part in enumerate(parts): + if part.isdigit(): + return ".".join(parts[max(i - 1, 0):]) + return name + + +def resolve_parameters(module: Any, edit: WeightEdit) -> "list[tuple[str, Any]]": + """``[(full parameter name, parameter), ...]`` for the tensors of *edit*, in order. + + A tensor name matches a parameter exactly, or — when the two differ only in + wrapper prefixes — by the suffix that starts at the indexed container (see + :func:`_anchored_suffix`). That lets an edit written against + ``model.language_model.layers.15.mlp.down_proj.weight`` apply to a module + that names the same tensor ``model.layers.15.mlp.down_proj.weight``. A name + that matches nothing, or more than one parameter, is an error. + """ + params = dict(module.named_parameters()) + out: list[tuple[str, Any]] = [] + for tensor_edit in edit.tensors: + if tensor_edit.name in params: + out.append((tensor_edit.name, params[tensor_edit.name])) + continue + suffix = _anchored_suffix(tensor_edit.name) + match = [k for k in params if k == suffix or k.endswith("." + suffix)] + if len(match) != 1: + problem = "matches no parameter" if not match else f"is ambiguous ({match[:3]})" + raise KeyError(f"weight edit tensor {tensor_edit.name!r} {problem} in " + f"{type(module).__name__}") + out.append((match[0], params[match[0]])) + if len({key for key, _ in out}) != len(out): + raise KeyError(f"{edit.name}: two tensor edits resolve to the same parameter") + return out + + +def tensor_sha256(tensor: Any) -> str: + """sha256 of a tensor's raw bytes in its own dtype (the hash edit manifests record).""" + import torch + + data = tensor.detach().cpu().contiguous() + if data.dtype == torch.bfloat16: + data = data.view(torch.int16) + return hashlib.sha256(data.numpy().tobytes()).hexdigest() + + +@dataclass +class AppliedEdit: + """What :func:`applied` did: one row per edited tensor.""" + + rows: list[dict[str, Any]] = field(default_factory=list) + + @property + def relative_change(self) -> float: + """Largest ``||dW|| / ||W||`` over the edited tensors.""" + return max((r["relative_change"] for r in self.rows), default=0.0) + + def to_dict(self) -> dict[str, Any]: + return {"tensors": self.rows, "max_relative_change": self.relative_change} + + +@contextmanager +def applied(model: Any, edit: WeightEdit, *, verify_base: bool = False, + max_relative_change: "float | None" = None) -> Iterator[AppliedEdit]: + """Write *edit* into the model for the duration of the block, then restore it. + + Every edited tensor is computed first; nothing is written if any of them + fails (a bad name, a shape mismatch, a change above *max_relative_change*). + On exit, normal or not, the original values are copied back and compared, + so a caller never continues with a half-edited or half-restored model. + + Args: + model: A ``torch.nn.Module`` or a white-box model handle. + edit: The edit to apply. + verify_base: Require every tensor's recorded ``base_sha256`` to + match the live tensor. A tensor with no recorded + hash is an error under this flag. + max_relative_change: Refuse an edit whose ``||dW|| / ||W||`` exceeds + this on any tensor. + + Yields: + :class:`AppliedEdit` with, per tensor, the resolved name, relative + change, and counts of changed entries, rows and columns. + """ + import torch + + module = module_of(model) + resolved = resolve_parameters(module, edit) + staged: list[tuple[Any, Any, Any]] = [] + report = AppliedEdit() + with torch.no_grad(): + for tensor_edit, (key, param) in zip(edit.tensors, resolved): + if verify_base: + if not tensor_edit.base_sha256: + raise ValueError(f"{tensor_edit.name}: verify_base needs a recorded base hash") + live = tensor_sha256(param) + if live != tensor_edit.base_sha256: + raise ValueError( + f"{key}: the live tensor differs from the one the edit was made on " + f"(sha256 {live[:12]} != {tensor_edit.base_sha256[:12]})" + ) + new = tensor_edit.apply_to(param) + if not bool(torch.isfinite(new).all()): + raise ValueError(f"{key}: the edit produces non-finite weights") + diff = (new.double() - param.double()) + base_norm = float(param.double().norm()) + rel = float(diff.norm()) / base_norm if base_norm > 0 else float("inf") + if max_relative_change is not None and rel > max_relative_change: + raise ValueError( + f"{key}: relative change {rel:.3g} exceeds the limit {max_relative_change:g}" + ) + changed = diff != 0 + row = {"name": key, "form": tensor_edit.form, "relative_change": rel, + "changed_entries": int(changed.sum())} + if changed.ndim == 2: + row["changed_rows"] = int(changed.any(dim=1).sum()) + row["changed_columns"] = int(changed.any(dim=0).sum()) + report.rows.append(row) + staged.append((param, param.detach().clone(), new)) + for param, _, new in staged: + param.copy_(new) + try: + yield report + finally: + with torch.no_grad(): + for param, backup, _ in staged: + param.copy_(backup) + broken = [i for i, (param, backup, _) in enumerate(staged) + if not torch.equal(param, backup)] + if broken: # pragma: no cover - copy_ of a clone is exact; this guards the contract + raise RuntimeError(f"{edit.name}: failed to restore {len(broken)} edited tensor(s)") diff --git a/evalrx/wsp/structure.py b/evalrx/wsp/structure.py new file mode 100644 index 00000000..28f5a7b1 --- /dev/null +++ b/evalrx/wsp/structure.py @@ -0,0 +1,189 @@ +"""What a weight edit changed — read from the weights alone. + +The first question about any edit is structural, and it needs no forward pass: +how many entries moved, in which rows and columns, and is the change low rank? +For a rank-one edit ``dW = u a^T`` the per-column coefficient ``a_j`` can be +compared with the column's old projection ``p_j = u^T w_j``: a constant ratio +``a_j / p_j = -alpha`` means the edit is the rule ``w' = w - alpha u (u^T w)`` +applied to those columns, and ``alpha`` can be read off. +""" + +from __future__ import annotations + +from typing import Any + +from evalrx.wsp.edit import TensorEdit, WeightEdit, module_of, resolve_parameters + + +def delta_structure(w0: Any, w1: Any, *, top_singular: int = 8) -> dict[str, Any]: + """Describe ``w1 - w0`` for one 2-D weight. + + Returns (all values JSON-serialisable): + + ``changed_entries`` / ``changed_rows`` / ``changed_columns`` + Where the tensor moved. + ``relative_change`` + ``||dW||_F / ||W0||_F``. + ``singular_values`` + Leading singular values of the change restricted to its changed columns. + ``effective_rank`` + Participation ratio ``(sum s)^2 / sum s^2`` of the singular values. + ``rank_one_residual`` + ``||dW - u a^T||_F / ||dW||_F`` for the top singular pair; 0 for an + exactly rank-one change. + ``alpha_estimate`` / ``alpha_spread`` + Mean and standard deviation of ``-a_j / p_j`` over the changed columns + (only columns whose old projection is not negligible). + ``projection_selectivity`` + Mean ``|u^T w_j|`` over changed columns divided by the mean over all + columns; above 1 means the edit chose columns that write along ``u``. + """ + import torch + + w0 = w0.detach().double() + w1 = w1.detach().double() + if w0.shape != w1.shape: + raise ValueError(f"shape mismatch: {tuple(w0.shape)} vs {tuple(w1.shape)}") + d = w1 - w0 + changed = d != 0 + out: dict[str, Any] = { + "shape": list(w0.shape), + "changed_entries": int(changed.sum()), + "relative_change": float(d.norm() / w0.norm()) if float(w0.norm()) > 0 else None, + } + if d.ndim != 2 or out["changed_entries"] == 0: + return out + cols = torch.nonzero(changed.any(dim=0)).flatten() + out["changed_rows"] = int(changed.any(dim=1).sum()) + out["changed_columns"] = int(cols.numel()) + dc = d[:, cols] + u_all, s, _ = torch.linalg.svd(dc, full_matrices=False) + u = u_all[:, 0] + a = u @ dc + out["singular_values"] = [float(v) for v in s[:top_singular]] + out["effective_rank"] = float(s.sum() ** 2 / (s ** 2).sum()) if float(s.sum()) > 0 else 0.0 + out["rank_one_residual"] = float((dc - torch.outer(u, a)).norm() / dc.norm()) + p_all = u @ w0 + p = p_all[cols] + scale = float(p_all.abs().mean()) + solid = p.abs() > 1e-3 * max(scale, 1e-30) + if bool(solid.any()): + ratio = -(a[solid] / p[solid]) + out["alpha_estimate"] = float(ratio.mean()) + out["alpha_spread"] = float(ratio.std()) if int(solid.sum()) > 1 else 0.0 + if scale > 0: + out["projection_selectivity"] = float(p.abs().mean() / scale) + return out + + +def edit_structure(model: Any, edit: WeightEdit, **kwargs: Any) -> dict[str, Any]: + """:func:`delta_structure` for every tensor of *edit* against the live weights. + + Also reports ``direction_cosines``: ``|cos|`` between the top left singular + vectors of each pair of edited tensors of equal row count, which says + whether the edit writes the same direction at several layers. + """ + import torch + + module = module_of(model) + rows: list[dict[str, Any]] = [] + directions: list[tuple[str, Any]] = [] + with torch.no_grad(): + for tensor_edit, (key, param) in zip(edit.tensors, resolve_parameters(module, edit)): + new = tensor_edit.apply_to(param) + row = {"name": key, "form": tensor_edit.form, **delta_structure(param, new, **kwargs)} + rows.append(row) + if row.get("changed_columns"): + d = new.double() - param.double() + directions.append((key, torch.linalg.svd(d, full_matrices=False)[0][:, 0])) + cosines = { + f"{a_key} ~ {b_key}": float((ua @ ub).abs()) + for i, (a_key, ua) in enumerate(directions) + for b_key, ub in directions[i + 1:] + if ua.numel() == ub.numel() + } + return {"edit": edit.name, "tensors": rows, "direction_cosines": cosines} + + +def _delta_and_direction(param: Any, tensor_edit: TensorEdit) -> "tuple[Any, Any, Any]": + """``(delta, changed columns, top left singular vector)`` of one tensor edit.""" + import torch + + d = tensor_edit.apply_to(param).double().cpu() - param.detach().double().cpu() + cols = torch.nonzero((d != 0).any(dim=0)).flatten() if d.ndim == 2 else torch.zeros(0) + if d.ndim != 2 or cols.numel() == 0: + return d, cols, None + return d, cols, torch.linalg.svd(d[:, cols], full_matrices=False)[0][:, 0] + + +def fit_rank_one(model: Any, edit: WeightEdit, *, name: "str | None" = None) -> WeightEdit: + """Re-express *edit* as the rank-one rule that best reproduces it on the live weights. + + For every edited 2-D tensor: ``u`` is the top left singular vector of the + change, the columns are the changed ones, and ``alpha`` is the mean of + ``-a_j / p_j`` (see :func:`delta_structure`). The result can be scaled + (:meth:`WeightEdit.scaled`), which an edit stored as overwritten entries + cannot. ``metadata["fit"]`` records, per tensor, the fitted ``alpha`` and the + share of the original change the rule leaves unexplained + (``rank_one_residual``); check it before treating the fit as the edit. + """ + import torch + + module = module_of(model) + tensors: list[TensorEdit] = [] + fit: list[dict[str, Any]] = [] + with torch.no_grad(): + for tensor_edit, (key, param) in zip(edit.tensors, resolve_parameters(module, edit)): + d, cols, u = _delta_and_direction(param, tensor_edit) + if u is None: + raise ValueError(f"{key}: no 2-D change to fit a rank-one rule to") + row = delta_structure(param.detach().cpu(), (param.detach().double().cpu() + d)) + if "alpha_estimate" not in row: + raise ValueError(f"{key}: the changed columns have no component along the " + "change direction, so no alpha can be read off") + tensors.append(TensorEdit.rank_one(key, u, row["alpha_estimate"], cols=cols)) + fit.append({"name": key, "alpha": row["alpha_estimate"], + "alpha_spread": row.get("alpha_spread"), + "rank_one_residual": row["rank_one_residual"], + "n_columns": int(cols.numel())}) + return WeightEdit(tensors, name=name or f"{edit.name}_rank_one", + metadata={"source": "rank_one_fit", "fit_of": edit.name, "fit": fit}) + + +def edit_agreement(model: Any, edit: WeightEdit, reference: WeightEdit) -> dict[str, Any]: + """How closely *edit* matches *reference*: same tensors, direction, neurons. + + For every tensor both edits change, reports ``direction_cosine`` (absolute + cosine between the top left singular vectors of the two changes; a + direction's sign is arbitrary), and the overlap of the changed columns + (``n_shared``, ``jaccard``, and ``expected_shared_by_chance`` for two random + column sets of those sizes). ``only_in_edit`` / ``only_in_reference`` list + tensors just one of them touches. + """ + import torch + + module = module_of(model) + mine = {key: (param, t) for t, (key, param) in zip(edit.tensors, + resolve_parameters(module, edit))} + theirs = {key: (param, t) for t, (key, param) in zip(reference.tensors, + resolve_parameters(module, reference))} + rows: list[dict[str, Any]] = [] + with torch.no_grad(): + for key in sorted(set(mine) & set(theirs)): + param = mine[key][0] + _, cols_a, u_a = _delta_and_direction(param, mine[key][1]) + _, cols_b, u_b = _delta_and_direction(param, theirs[key][1]) + a, b = set(cols_a.tolist()), set(cols_b.tolist()) + row: dict[str, Any] = { + "name": key, "n_columns_edit": len(a), "n_columns_reference": len(b), + "n_shared": len(a & b), + "jaccard": round(len(a & b) / len(a | b), 4) if a | b else None, + } + if param.ndim == 2 and param.shape[1]: + row["expected_shared_by_chance"] = round(len(a) * len(b) / param.shape[1], 3) + if u_a is not None and u_b is not None: + row["direction_cosine"] = float((u_a @ u_b).abs()) + rows.append(row) + return {"edit": edit.name, "reference": reference.name, "tensors": rows, + "only_in_edit": sorted(set(mine) - set(theirs)), + "only_in_reference": sorted(set(theirs) - set(mine))} diff --git a/mkdocs.yml b/mkdocs.yml index 6ce9e8da..bb0a7e4e 100755 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -37,12 +37,14 @@ nav: - Command-Line Interface: cli.md - Exploratory Analysis (M2/M3): m2_analysis.md - Verification & Intervention (M4/M5): intervention.md + - Weight Edits & Probe Tools: wsp_and_probe_tools.md - Stage Input/Output Reference: stage_io.md - Analyzer Zoo: analyzers.md - Architecture: architecture.md - Extending: extending.md - Roadmap: roadmap.md - Validation: + - Published weight edit on Qwen3.5-0.8B (2026-11-15): validation/astra_p50c_20261115.md - Gemini agentic run (2026-10-13): validation/gemini_agentic_20261013.md - Historical Gemma repairs (2026-08-30): validation/gemma_repair_20260830.md - Local Artifacts: local_artifacts.md diff --git a/tests/test_analyzers/test_analyzer_contract.py b/tests/test_analyzers/test_analyzer_contract.py index e1a363e7..9df62eaa 100755 --- a/tests/test_analyzers/test_analyzer_contract.py +++ b/tests/test_analyzers/test_analyzer_contract.py @@ -61,11 +61,17 @@ from evalrx.analyzers.uncertainty.logprob_entropy import LogprobEntropyAnalyzer from evalrx.analyzers.uncertainty.self_consistency import SelfConsistencyAnalyzer from evalrx.analyzers.uncertainty.verbalized_conf import VerbalizedConfidenceAnalyzer +from evalrx.analyzers.weights import ( + WeightEditAudit, + WeightEditContinuationAudit, + WeightEditStructure, +) from evalrx.core.capability import Capability, CapabilityError from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label, Step, StepRole, Trajectory from evalrx.core.registry import registry from evalrx.core.result import Result from evalrx.datasets import cases_from_records +from evalrx.wsp import TensorEdit, WeightEdit from tests.conftest import FakeModel # ── shared fixtures ──────────────────────────────────────────────────────────── @@ -234,6 +240,13 @@ def _long_prompt_batch() -> CaseBatch: "embed_lens": "needs a live torch module; see test_intervention_analyzers.py", "attention_sensitivity": "needs a live torch module; see test_intervention_analyzers.py", "head_attribution": "needs a live torch module; see test_intervention_analyzers.py", + # Weight-edit analyzers take the edit as a constructor argument and write it + # into a live torch module. Their result contract runs against a model with + # a planted defect in test_weight_edit_analyzers.py. + "weight_edit_structure": "needs a WeightEdit + live torch module; see test_weight_edit_analyzers.py", + "weight_edit_audit": "needs a WeightEdit + live torch module; see test_weight_edit_analyzers.py", + "weight_edit_continuation_audit": + "needs a WeightEdit + live torch module; see test_weight_edit_continuation.py", } @@ -866,6 +879,9 @@ def _check_coverage_verification_gap(f): _STUBS_IDS = [a.name for a, _, _ in _STUBS] +# A minimal edit for constructing the weight-edit analyzers (never applied here). +_WEIGHT_EDIT = WeightEdit([TensorEdit.rank_one("layers.0.mlp.down_proj.weight", [1.0, 0.0], 1.0)]) + _CLASS_FACTORIES: dict[str, Callable[[], Any]] = { "counterfactual": lambda: CounterfactualReplay( rerun_fn=lambda traj, idx, seed: True, @@ -877,6 +893,9 @@ def _check_coverage_verification_gap(f): run_with_tools=lambda case, names: None, tool_names=["a"] ), "trajectory_rubric": lambda: TrajectoryRubricJudge(judge=_RUBRIC_JUDGE), + "weight_edit_structure": lambda: WeightEditStructure(edit=_WEIGHT_EDIT), + "weight_edit_audit": lambda: WeightEditAudit(edit=_WEIGHT_EDIT), + "weight_edit_continuation_audit": lambda: WeightEditContinuationAudit(edit=_WEIGHT_EDIT), } diff --git a/tests/test_analyzers/test_weight_edit_analyzers.py b/tests/test_analyzers/test_weight_edit_analyzers.py new file mode 100644 index 00000000..f45e16ca --- /dev/null +++ b/tests/test_analyzers/test_weight_edit_analyzers.py @@ -0,0 +1,261 @@ +"""Weight-edit analyzers: structure from the weights, and the audit against controls. + +The audit is checked on a model with a planted defect, where the right reading +is known in advance: the true edit moves the failing cases toward the gold +answer and leaves the passing ones alone, a random direction does not, and a +constant activation shift cannot stand in for an edit that only acts on the +inputs that trigger the defect. +""" + +from __future__ import annotations + +import json + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.analyzers._internals.targets import build_margin # noqa: E402 +from evalrx.analyzers.weights import WeightEditAudit, WeightEditStructure # noqa: E402 +from evalrx.core.capability import Capability, CapabilityError # noqa: E402 +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label # noqa: E402 +from evalrx.core.registry import registry # noqa: E402 +from evalrx.core.result import Result # noqa: E402 +from evalrx.probing import ProbeToolbox, spec_for # noqa: E402 +from evalrx.wsp import TensorEdit, WeightEdit, applied, contrast_edit # noqa: E402 +from tests.conftest import FakeModel # noqa: E402 +from tests.toy_wsp import ( # noqa: E402 + DEFECT_LAYER, + WRONG_TOKEN, + D, + ToyModel, + echo_batch, + snapshot, + unchanged, +) + + +def _true_edit(model, alpha: float = 1.0) -> WeightEdit: + pool = echo_batch(4, 6) + return contrast_edit(model, [c for c in pool if c.label == Label.FAIL], + [c for c in pool if c.label == Label.PASS], + n_layers=1, n_columns=1, alpha=alpha) + + +def test_both_analyzers_are_registered_as_weight_edit_probes(): + for name in ("weight_edit_structure", "weight_edit_audit"): + assert registry.analyzers.has(name) + spec = spec_for(name) + assert spec.family == "weight_edit" + assert spec.injected_params == ("edit",) + assert "edit" not in spec.parameters_schema()["properties"] + assert spec_for("weight_edit_audit").intervenes is True + assert spec_for("weight_edit_structure").intervenes is False + + +def test_structure_reports_the_rule_and_leaves_the_model_alone(): + model = ToyModel() + snap = snapshot(model) + result = WeightEditStructure(edit=_true_edit(model, alpha=6.0)).run(model, echo_batch()) + assert isinstance(result, Result) and result.analyzer == "weight_edit_structure" + f = result.findings + json.dumps(f) + assert f["n_tensors"] == 1 + assert f["alpha_estimates"] == [6.0] + assert f["max_rank_one_residual"] < 1e-5 + assert f["tensors"][0]["changed_columns"] == 1 + assert f["max_relative_change"] > 0 + assert unchanged(model, snap) + + +def test_structure_can_require_the_recorded_base(): + model = ToyModel() + edit = WeightEdit([TensorEdit(name="layers.0.mlp.down_proj.weight", + delta=np.ones((D, D)), base_sha256="0" * 64)]) + with pytest.raises(ValueError, match="differs from the one the edit was made on"): + WeightEditStructure(edit=edit, verify_base=True).run(model, echo_batch()) + + +def test_audit_result_contract(): + model = ToyModel() + batch = echo_batch(4, 6) + snap = snapshot(model) + result = WeightEditAudit(edit=_true_edit(model), target="gold", n_seeds=2).run(model, batch) + f = result.findings + json.dumps(f) + assert f["n_cases_analyzed"] == len(batch) and f["n_skipped"] == 0 + assert len(f["per_case"]) == len(batch) + assert {r["sample_id"] for r in f["per_case"]} == {c.id for c in batch} + for row in f["per_case"]: + assert {"we_margin_clean", "we_delta_edit", "we_kl_edit", "we_flips_top1", + "we_delta_random_direction", "we_kl_random_direction", + "we_delta_random_columns", "we_kl_random_columns", "we_delta_steer"} <= set(row) + assert row["we_kl_edit"] >= 0 + documented = set(WeightEditAudit.signal_docs) + assert documented <= set(f["per_case"][0]) + assert f["n_fail"] == 4 and f["n_pass"] == 6 + assert unchanged(model, snap) + + +def test_audit_reads_the_planted_defect_correctly(): + model = ToyModel() + f = WeightEditAudit(edit=_true_edit(model), target="gold", n_seeds=3).run( + model, echo_batch(4, 6)).findings + # The edit moves failing cases toward gold, flips every one of them... + assert f["mean_delta_edit_fail"] > 1.0 + assert f["frac_flips_fail"] == 1.0 + # ...and has no side effect on the cases the model already got right. + assert abs(f["mean_delta_edit_pass"]) < 0.05 + assert f["frac_flips_pass"] == 0.0 + # A matched random direction, or the same rule on other neurons, moves the + # output far less than the edit does. + assert f["specificity_random_direction"] > 2.0 + assert f["specificity_random_columns"] > 2.0 + # The defect is input-dependent (it fires on trigger tokens only), so one + # constant shift for all cases cannot reproduce the edit. + assert f["steer_recovery"] < 0.9 + + +def test_a_random_edit_is_not_reported_as_specific(): + model = ToyModel() + torch.manual_seed(0) + arbitrary = WeightEdit([TensorEdit.rank_one( + f"layers.{DEFECT_LAYER}.mlp.down_proj.weight", torch.randn(D), 1.0, cols=[0, 2])]) + f = WeightEditAudit(edit=arbitrary, target="gold", n_seeds=5).run( + model, echo_batch(4, 6)).findings + assert f["frac_flips_top1"] == 0.0 + assert f["specificity_random_direction"] < 5.0 + + +def test_per_case_change_matches_an_independent_measurement(): + model = ToyModel() + batch = echo_batch(2, 2) + edit = _true_edit(model) + rows = {r["sample_id"]: r for r in WeightEditAudit( + edit=edit, target="gold", controls=(), steering=False).run(model, batch).findings["per_case"]} + for case in batch: + clean = model.forward(case.inputs, capture={Capability.LOGITS}).logits[-1:].float() + margin, _ = build_margin("gold", clean, model=model, case=case) + with applied(model, edit): + edited = model.forward(case.inputs, capture={Capability.LOGITS}).logits[-1:].float() + expected = margin.value(edited) - margin.value(clean) + assert rows[case.id]["we_delta_edit"] == pytest.approx(expected, abs=1e-5) + + +def test_a_null_edit_measures_zero(): + model = ToyModel() + null = _true_edit(model, alpha=0.0) + f = WeightEditAudit(edit=null, controls=(), steering=False).run(model, echo_batch()).findings + assert f["mean_kl_edit"] == 0.0 and f["mean_delta_edit"] == 0.0 + assert f["frac_flips_top1"] == 0.0 + + +def test_controls_and_steering_can_be_switched_off(): + model = ToyModel() + f = WeightEditAudit(edit=_true_edit(model), controls=("random_direction",), + steering=False, n_seeds=1).run(model, echo_batch()).findings + assert "specificity_random_direction" in f + assert "specificity_random_columns" not in f + assert "steer_recovery" not in f and "steering_not_run" not in f + assert "we_delta_steer" not in f["per_case"][0] + + +def test_a_control_that_cannot_be_built_is_reported(): + model = ToyModel() + whole = WeightEdit([TensorEdit.rank_one( + f"layers.{DEFECT_LAYER}.mlp.down_proj.weight", np.eye(D)[WRONG_TOKEN], 1.0)]) + f = WeightEditAudit(edit=whole, n_seeds=1).run(model, echo_batch()).findings + assert "random_columns" in f["controls_not_run"] + assert "specificity_random_direction" in f + + +def test_steering_is_skipped_with_a_reason_for_non_linear_tensors(): + model = ToyModel() + edit = WeightEdit([TensorEdit(name="embed.weight", delta=0.01 * np.ones((D, D)))]) + f = WeightEditAudit(edit=edit, controls=()).run(model, echo_batch()).findings + assert "steering_not_run" in f and "steer_recovery" not in f + + +def test_gold_target_skips_cases_without_a_gold_answer(): + model = ToyModel() + batch = CaseBatch([*echo_batch(2, 2), + FailureCase(inputs=Inputs(prompt="3 4 5"), label=Label.PASS)]) + f = WeightEditAudit(edit=_true_edit(model), target="gold", controls=(), + steering=False).run(model, batch).findings + assert f["n_cases_analyzed"] == 4 and f["skip_reasons"] == {"no_gold": 1} + + +def test_bad_settings_are_refused(): + model = ToyModel() + edit = _true_edit(model) + with pytest.raises(ValueError, match="unknown control"): + WeightEditAudit(edit=edit, controls=("shuffle",)).run(model, echo_batch()) + with pytest.raises(ValueError, match="n_seeds"): + WeightEditAudit(edit=edit, n_seeds=0).run(model, echo_batch()) + + +def test_the_model_is_restored_when_a_forward_fails_mid_audit(): + model = ToyModel() + snap = snapshot(model) + edit = _true_edit(model) # built before the forward pass is made flaky + original = model.forward + state = {"n": 0} + + def flaky(inputs, capture, spec=None): + state["n"] += 1 + if state["n"] == 8: # after the clean pass, inside the edited pass + raise RuntimeError("device lost") + return original(inputs, capture, spec) + + model.forward = flaky + with pytest.raises(RuntimeError, match="device lost"): + WeightEditAudit(edit=edit, controls=(), steering=False).run(model, echo_batch(3, 3)) + assert unchanged(model, snap) + for layer in model.module.layers: + assert not layer.mlp.down_proj._forward_hooks + assert not layer.mlp.down_proj._forward_pre_hooks + + +def test_models_without_internals_are_refused_clearly(): + edit = WeightEdit([TensorEdit(name="layers.0.mlp.down_proj.weight", delta=np.ones((D, D)))]) + with pytest.raises(CapabilityError): + WeightEditAudit(edit=edit).run(FakeModel(capabilities={Capability.GENERATE}), "x") + with pytest.raises(ValueError, match="white-box"): + WeightEditStructure(edit=edit).run(FakeModel(capabilities={Capability.LOGITS}), "x") + + +def test_audit_runs_as_a_probe_tool_with_json_arguments(): + model = ToyModel() + box = ProbeToolbox(model, echo_batch(), + overrides={"weight_edit_audit": WeightEditAudit(edit=_true_edit(model))}) + report = box.describe(families=["weight_edit"]) + assert report["weight_edit_audit"]["available"] is True + assert report["weight_edit_structure"]["codes"] == ["injection"] + result = box.call("weight_edit_audit", {"target": "gold", "n_seeds": 1, "steering": False}) + assert result.findings["target"] == "gold" and result.findings["n_control_draws"] == 1 + + +def test_audit_on_a_real_architecture(): + pytest.importorskip("transformers") + pytest.importorskip("PIL") + from evalrx.wsp import write_sites + from tests.tiny_whitebox import TinyVLM + + model = TinyVLM("qwen2_5_vl") + snap = snapshot(model) + site = write_sites(model)[3] + torch.manual_seed(0) + edit = WeightEdit([TensorEdit.rank_one(site.name, torch.randn(32), 6.0, cols=[1, 5, 9])]) + batch = CaseBatch([ + FailureCase(inputs=Inputs(prompt=f"what is {i}"), expected="b", + label=Label.FAIL if i % 2 else Label.PASS) + for i in range(4) + ]) + f = WeightEditAudit(edit=edit, n_seeds=1).run(model, batch).findings + json.dumps(f) + assert f["n_cases_analyzed"] == 4 + assert f["structure"][0]["alpha_estimate"] == pytest.approx(6.0, abs=1e-2) + assert f["mean_kl_edit"] > 0 + assert "steer_recovery" in f + assert unchanged(model, snap) diff --git a/tests/test_analyzers/test_weight_edit_continuation.py b/tests/test_analyzers/test_weight_edit_continuation.py new file mode 100644 index 00000000..f5d799ef --- /dev/null +++ b/tests/test_analyzers/test_weight_edit_continuation.py @@ -0,0 +1,338 @@ +"""Weight-edit continuation audit: the edit followed along recorded generations. + +Checked on the planted-defect model, where a "generation" can be written by +hand: after a trigger token the defective model emits the wrong token, so a +recorded text that alternates trigger / wrong token is this model's loop. The +true edit must make that repeated text less likely and leave trigger-free text +alone; a matched random change must do neither. +""" + +from __future__ import annotations + +import json + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.analyzers.weights import WeightEditContinuationAudit # noqa: E402 +from evalrx.analyzers.weights.continuation_audit import repetition_mask # noqa: E402 +from evalrx.core.capability import Capability, CapabilityError # noqa: E402 +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label # noqa: E402 +from evalrx.core.registry import registry # noqa: E402 +from evalrx.probing import ProbeToolbox, spec_for # noqa: E402 +from evalrx.wsp import TensorEdit, WeightEdit, applied, contrast_edit # noqa: E402 +from tests.conftest import FakeModel # noqa: E402 +from tests.toy_wsp import ( # noqa: E402 + DEFECT_LAYER, + WRONG_TOKEN, + D, + ToyModel, + echo_batch, + snapshot, + unchanged, +) + +# Trigger-free text first (never repeats a bigram), then the model's own loop. +_PLAIN = "3 4 5 6 2 5 3 6 4 2 6 3" +_LOOP = " ".join(["0 7"] * 8) +# A prompt whose bigram occurs nowhere in the texts above. +_PROMPT = "2 2" + + +def _true_edit(model) -> WeightEdit: + pool = echo_batch(4, 6) + return contrast_edit(model, [c for c in pool if c.label == Label.FAIL], + [c for c in pool if c.label == Label.PASS], n_layers=1, n_columns=1) + + +def _gen(prompt: str, continuation) -> FailureCase: + return FailureCase(inputs=Inputs(prompt=prompt), observed=continuation, label=Label.FAIL) + + +def _batch() -> CaseBatch: + return CaseBatch([_gen(_PROMPT, f"{_PLAIN} {_LOOP}"), + _gen(_PROMPT, f"{_PLAIN} {' '.join(['1 7'] * 8)}")]) + + +def _run(model, cases=None, **kwargs): + kwargs.setdefault("repeat_ngram", 2) + return WeightEditContinuationAudit(edit=_true_edit(model), **kwargs).run( + model, cases or _batch()).findings + + +# ---------------------------------------------------------------------- +# Repetition mask +# ---------------------------------------------------------------------- + +def test_repetition_mask_marks_contexts_already_seen(): + ids = [0, 7, 0, 7, 0, 7, 3, 4] + assert repetition_mask(ids, 2) == [False, False, False, False, True, True, True, False] + assert repetition_mask(ids, 9) == [False] * 8 + assert repetition_mask([], 2) == [] + assert repetition_mask([5, 5, 5, 5], 1) == [False, False, True, True] + + +# ---------------------------------------------------------------------- +# Contract +# ---------------------------------------------------------------------- + +def test_registered_as_a_weight_edit_probe(): + assert registry.analyzers.has("weight_edit_continuation_audit") + spec = spec_for("weight_edit_continuation_audit") + assert spec.family == "weight_edit" and spec.intervenes is True + assert spec.injected_params == ("edit",) + assert {"groups", "repeat_ngram", "max_tokens"} <= set(spec.parameters_schema()["properties"]) + + +def test_result_contract_and_restoration(): + model = ToyModel() + snap = snapshot(model) + f = _run(model, n_seeds=2) + json.dumps(f) + assert f["n_cases_analyzed"] == 2 and f["n_skipped"] == 0 + # 12 plain + 16 loop tokens per case, every continuation token scored once. + assert f["n_positions"] == 56 + assert f["n_positions_loop"] + f["n_positions_productive"] == 56 + assert len(f["per_case"]) == 2 + row = f["per_case"][0] + assert row["wc_n_positions"] == 28 and 0 < row["wc_frac_loop"] < 1 + assert set(WeightEditContinuationAudit.signal_docs) <= set(row) + assert unchanged(model, snap) + for layer in model.module.layers: + assert not layer.mlp.down_proj._forward_pre_hooks + assert not layer.mlp.down_proj._forward_hooks + + +def test_positions_are_grouped_by_the_repetition_mask(): + model = ToyModel() + f = _run(model, controls=(), steering=False) + # Per case: prompt (2) + plain (12) + "0 7" x 8. With bigram windows the + # first four loop tokens are each predicted in a context not seen before; + # the remaining twelve are predicted in a context that already occurred. + assert f["n_positions_loop"] == 2 * 12 + assert f["n_positions_productive"] == 2 * 16 + + +# ---------------------------------------------------------------------- +# Reading the planted defect +# ---------------------------------------------------------------------- + +def test_the_edit_acts_in_the_loop_and_not_on_plain_text(): + model = ToyModel() + f = _run(model, n_seeds=3) + # The repeated text becomes much less likely... + assert f["mean_dnll_edit_loop"] > 0.5 + # ...far more so than text outside the loop, + # (the productive group still holds the loop's first cycle, which cannot + # count as a repeat yet, so it is not exactly zero) + assert f["dnll_edit_loop_minus_productive"] > 0.4 + assert f["mean_kl_edit_loop"] > 3 * f["mean_kl_edit_productive"] + # a matched random direction or other neurons barely move the output, + assert f["specificity_random_direction_loop"] > 10 + assert f["specificity_random_columns_loop"] > 10 + assert abs(f["mean_dnll_random_direction_loop"]) < 0.1 + # and one constant shift does not reproduce an input-dependent edit. + assert f["steer_recovery_loop"] < 0.5 + + +def test_plain_text_alone_is_left_untouched(): + model = ToyModel() + cases = CaseBatch([_gen(_PROMPT, _PLAIN), _gen(_PROMPT, "6 5 4 3 2 3 5 2 4 6")]) + f = _run(model, cases, controls=(), steering=False) + assert f["n_positions_loop"] == 0 + assert "mean_dnll_edit_loop" not in f and "dnll_edit_loop_minus_productive" not in f + assert abs(f["mean_dnll_edit_productive"]) < 0.02 + assert f["frac_flips_edit_productive"] == 0.0 + + +def test_per_position_values_match_an_independent_measurement(): + model = ToyModel() + edit = _true_edit(model) + case = _gen("4 2", "0 7 0 7 3") + f = WeightEditContinuationAudit(edit=edit, groups="none", controls=(), + steering=False).run(model, CaseBatch([case])).findings + ids = torch.tensor([[4, 2, 0, 7, 0, 7, 3]]) + targets = [2, 3, 4, 5, 6] + + def nll(logits): + lp = torch.log_softmax(logits[0].double(), dim=-1) + return torch.stack([-lp[i - 1, int(ids[0, i])] for i in targets]) + + with torch.no_grad(): + clean = model.module(input_ids=ids) + with applied(model, edit): + edited = model.module(input_ids=ids) + assert f["n_positions_all"] == 5 + assert f["mean_nll_clean_all"] == pytest.approx(float(nll(clean).mean()), abs=1e-5) + assert f["mean_dnll_edit_all"] == pytest.approx(float((nll(edited) - nll(clean)).mean()), + abs=1e-5) + + +def test_a_null_edit_measures_zero_everywhere(): + model = ToyModel() + null = WeightEdit([TensorEdit.rank_one( + f"layers.{DEFECT_LAYER}.mlp.down_proj.weight", np.eye(D)[WRONG_TOKEN], 0.0)]) + f = WeightEditContinuationAudit(edit=null, repeat_ngram=2, controls=(), + steering=False).run(model, _batch()).findings + for group in ("loop", "productive"): + assert f[f"mean_dnll_edit_{group}"] == 0.0 and f[f"mean_kl_edit_{group}"] == 0.0 + + +# ---------------------------------------------------------------------- +# Options, skips, refusals +# ---------------------------------------------------------------------- + +def test_no_grouping_keeps_one_group(): + model = ToyModel() + f = _run(model, groups="none", controls=("random_direction",), steering=False) + assert f["n_positions_all"] == 56 and f["repeat_ngram"] is None + assert "specificity_random_direction_all" in f + assert "wc_frac_loop" not in f["per_case"][0] + + +def test_truncation_limits_the_scored_positions(): + model = ToyModel() + f = _run(model, max_tokens=10, controls=(), steering=False) + assert f["n_positions"] == 2 * (10 - 2) # two prompt tokens per case + + +def test_chunking_does_not_change_the_result(): + model = ToyModel() + a = _run(model, controls=(), steering=False, chunk=3) + b = _run(model, controls=(), steering=False, chunk=256) + for key in ("mean_dnll_edit_loop", "mean_kl_edit_loop", "mean_dnll_edit_productive"): + assert a[key] == pytest.approx(b[key], abs=1e-9) + + +def test_cases_without_a_recorded_generation_are_skipped_and_counted(): + model = ToyModel() + cases = CaseBatch([_gen("4 2", f"{_PLAIN} {_LOOP}"), _gen("4 2", None), _gen("4 2", " ")]) + f = _run(model, cases, controls=(), steering=False) + assert f["n_cases_analyzed"] == 1 and f["skip_reasons"] == {"no_continuation": 2} + with pytest.raises(ValueError, match="no case could be analysed"): + _run(model, CaseBatch([_gen("4 2", None)])) + + +def test_a_prompt_longer_than_the_limit_is_skipped(): + model = ToyModel() + f = _run(model, CaseBatch([_gen("4 2 3 5 6", "0 7"), _gen("4 2", f"{_PLAIN} {_LOOP}")]), + max_tokens=5, controls=(), steering=False) + assert f["skip_reasons"] == {"no_continuation_tokens_after_truncation": 1} + + +def test_a_control_that_cannot_be_built_is_reported(): + model = ToyModel() + whole = WeightEdit([TensorEdit.rank_one( + f"layers.{DEFECT_LAYER}.mlp.down_proj.weight", np.eye(D)[WRONG_TOKEN], 1.0)]) + f = WeightEditContinuationAudit(edit=whole, repeat_ngram=2).run(model, _batch()).findings + assert "random_columns" in f["controls_not_run"] + assert "specificity_random_direction_loop" in f + assert "mean_dnll_random_columns_loop" not in f + + +def test_steering_is_skipped_with_a_reason_for_non_linear_tensors(): + model = ToyModel() + edit = WeightEdit([TensorEdit(name="embed.weight", delta=0.01 * np.ones((D, D)))]) + f = WeightEditContinuationAudit(edit=edit, repeat_ngram=2, controls=()).run( + model, _batch()).findings + assert "steering_not_run" in f and "steer_recovery_loop" not in f + + +@pytest.mark.parametrize("kwargs,message", [ + ({"groups": "paragraph"}, "groups must be one of"), + ({"controls": ("shuffle",)}, "unknown control"), + ({"n_seeds": 0}, "must be positive"), + ({"repeat_ngram": 0}, "must be positive"), + ({"max_tokens": 2}, "max_tokens"), +]) +def test_bad_settings_are_refused(kwargs, message): + model = ToyModel() + with pytest.raises(ValueError, match=message): + _run(model, **kwargs) + + +def test_the_model_is_restored_when_a_forward_fails_mid_audit(): + model = ToyModel() + snap = snapshot(model) + edit = _true_edit(model) + module = model.module + original = module.forward + state = {"n": 0} + + def flaky(input_ids): + state["n"] += 1 + if state["n"] == 3: # case 1 clean, case 1 edited, then this one + raise RuntimeError("device lost") + return original(input_ids) + + module.forward = flaky + try: + with pytest.raises(RuntimeError, match="device lost"): + WeightEditContinuationAudit(edit=edit, repeat_ngram=2, controls=(), + steering=False).run(model, _batch()) + finally: + module.forward = original + assert unchanged(model, snap) + + +def test_models_without_internals_are_refused_clearly(): + edit = WeightEdit([TensorEdit(name="layers.0.mlp.down_proj.weight", delta=np.ones((D, D)))]) + with pytest.raises(CapabilityError): + WeightEditContinuationAudit(edit=edit).run( + FakeModel(capabilities={Capability.GENERATE}), "x") + with pytest.raises(ValueError, match="white-box"): + WeightEditContinuationAudit(edit=edit).run( + FakeModel(capabilities={Capability.LOGITS}), "x") + + +def test_runs_as_a_probe_tool_with_json_arguments(): + model = ToyModel() + box = ProbeToolbox(model, _batch(), overrides={ + "weight_edit_continuation_audit": WeightEditContinuationAudit(edit=_true_edit(model))}) + result = box.call("weight_edit_continuation_audit", + {"repeat_ngram": 2, "controls": ["random_direction"], "steering": False}) + assert result.findings["repeat_ngram"] == 2 + assert "specificity_random_direction_loop" in result.findings + + +# ---------------------------------------------------------------------- +# A real local model handle +# ---------------------------------------------------------------------- + +def test_on_a_real_local_model_handle(): + """Mechanics on ``HFLocalModel`` over a from-scratch Llama: prompt tokens come + from the handle, the continuation is appended, and the weights are restored. + The weights are random, so only the bookkeeping is asserted.""" + pytest.importorskip("transformers") + from evalrx.core.spec import ModelSpec + from evalrx.models.backends.base import RuntimeConfig + from evalrx.models.backends.hf.model import HFLocalModel + from evalrx.wsp import write_sites + from tests.test_eval_agent.test_fix_agent import _tiny_llama, _TinyWordTokenizer + + class _Tok(_TinyWordTokenizer): + def __call__(self, text, return_tensors=None, add_special_tokens=True): + ids = [self.vocab.get(w, self.vocab[""]) for w in text.strip().lower().split()] + if return_tensors == "pt": + input_ids = torch.tensor([ids or [3]], dtype=torch.long) + return {"input_ids": input_ids, "attention_mask": torch.ones_like(input_ids)} + return {"input_ids": ids} + + spec = ModelSpec(key="tiny-llama-cont", family="fake", model_type="fake_llm", hf_repo="") + model = HFLocalModel(spec, RuntimeConfig(device="cpu", dtype="float32", max_new_tokens=3)) + model._hf = (_tiny_llama(), _Tok()) + snap = snapshot(model) + site = write_sites(model)[1] + torch.manual_seed(0) + edit = WeightEdit([TensorEdit.rank_one(site.name, torch.randn(16), 6.0, cols=[1, 5, 9])]) + cases = CaseBatch([_gen("classify alpha", "yes no yes no yes no yes no beta alpha")]) + f = WeightEditContinuationAudit(edit=edit, repeat_ngram=2, n_seeds=1).run(model, cases).findings + json.dumps(f) + assert f["n_positions"] == 10 # ten continuation tokens after two prompt tokens + assert f["n_positions_loop"] == 5 and f["n_positions_productive"] == 5 + assert f["mean_kl_edit_loop"] > 0 + assert "steer_recovery_loop" in f + assert f["structure"][0]["alpha_estimate"] == pytest.approx(6.0, abs=1e-2) + assert unchanged(model, snap) diff --git a/tests/test_eval_agent/test_agentic_probe_tools.py b/tests/test_eval_agent/test_agentic_probe_tools.py new file mode 100644 index 00000000..615423af --- /dev/null +++ b/tests/test_eval_agent/test_agentic_probe_tools.py @@ -0,0 +1,206 @@ +"""The agentic loop's probe tools: list what can run, then call probes by name. + +``run_probe`` with no arguments keeps its old meaning (the probe agent selects). +With a name it runs exactly that probe with schema-checked arguments, adds the +result to the evidence gathered so far, and rejects a bad call before any model +work — so a judge can correct itself without spending a probe pass. +""" + +from __future__ import annotations + +import json + +from evalrx.core.capability import Capability +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label +from evalrx.eval_agent import AgenticDiagnoseLoop, DiagnosisAgent, HypothesisTester +from evalrx.eval_agent.agentic.actions import Action +from evalrx.eval_agent.agentic.board import EvidenceBoard +from evalrx.eval_agent.agentic.tools import ( + _requested_probes, + _RunState, + build_default_registry, +) +from evalrx.eval_agent.run_context import RunContext +from evalrx.eval_agent.stages.probe_agent import ProbeAgent +from evalrx.eval_agent.stages.protocol import ExperimentProtocol +from tests.conftest import FakeModel + +_WHITEBOX = {Capability.GENERATE, Capability.ATTENTION, Capability.HIDDEN_STATES, + Capability.LOGITS} + + +class _Judge(FakeModel): + def __init__(self, responses=()): + super().__init__(capabilities={Capability.GENERATE}) + self._responses = iter(responses) + self.calls: list[str] = [] + + def generate(self, inputs, **kwargs) -> str: + self.calls.append(str(inputs)) + return next(self._responses) + + +class _CountingModel(FakeModel): + def __init__(self, **kwargs): + super().__init__(**kwargs) + self.n_forward = 0 + + def forward(self, inputs, capture, spec=None): + self.n_forward += 1 + return super().forward(inputs, capture, spec) + + +def _cases() -> CaseBatch: + return CaseBatch([ + FailureCase(inputs=Inputs(prompt=f"q{i}"), label=Label.FAIL if i % 2 else Label.PASS) + for i in range(4) + ]) + + +def _setup(tmp_path, model=None, judge=None): + ctx = RunContext(tmp_path / "run", verbose=False) + loop = AgenticDiagnoseLoop( + model=model or _CountingModel(capabilities=_WHITEBOX, modalities={"text"}), + protocol=ExperimentProtocol( + description="probe tool test", task_domain="test", + failure_patterns="none", target_modalities=frozenset({"text"}), + ), + judge=judge or _Judge(), + probe_agent=ProbeAgent(max_workers=1, max_analyzers=2), + diagnosis_agent=DiagnosisAgent(judge=_Judge()), + hypothesis_tester=HypothesisTester(min_effect=0.05), + run_logger=ctx.logger, + ) + data = _cases() + state = _RunState(original_data=data, data=data) + return loop, state, build_default_registry(loop, state), EvidenceBoard() + + +def _call(registry, board, tool, params=None): + outcome = registry.dispatch(Action(tool=tool, params=params or {}, rationale="t"), board) + board.action_log.append({"step": len(board.action_log), "tool": tool, + "params": params or {}, "ok": outcome.ok, + "summary": outcome.summary}) + return outcome + + +def test_registry_offers_list_probes_and_run_probe(tmp_path): + _, _, registry, board = _setup(tmp_path) + assert {"list_probes", "run_probe"} <= registry.tool_names() + catalog = registry.catalog_for_prompt(board) + assert "list_probes" in catalog and "params.name" in catalog + + +def test_request_parsing_accepts_one_or_many(): + assert _requested_probes({}) == [] + assert _requested_probes({"name": "a", "params": {"k": 1}}) == [("a", {"k": 1})] + assert _requested_probes({"probes": [{"name": "a"}, "b"]}) == [("a", {}), ("b", {})] + assert isinstance(_requested_probes({"probes": "a"}), str) + assert isinstance(_requested_probes({"probes": [{"params": {}}]}), str) + assert isinstance(_requested_probes({"name": "a", "params": [1]}), str) + + +def test_list_probes_runs_no_model_and_fills_the_board(tmp_path): + loop, _, registry, board = _setup(tmp_path) + outcome = _call(registry, board, "list_probes", {"families": ["lens"], "max_probes": 5}) + assert outcome.ok is True + assert loop.model.n_forward == 0 + assert "logit_lens" in board.probe_catalog + assert "PROBE CATALOG" in board.to_prompt(registry) + assert all(p.spec.family == "lens" for p in outcome.payload.selected) + + +def test_list_probes_defaults_to_the_current_hypotheses(tmp_path): + _, _, registry, board = _setup(tmp_path) + board.hypotheses = [{"statement": "s", "failure_mode": "logit_lens", "status": "proposed"}] + outcome = _call(registry, board, "list_probes", {"only_matching": True}) + assert outcome.ok is True + assert "logit_lens" in outcome.payload.names + + +def test_list_probes_reports_an_empty_route(tmp_path): + _, _, registry, board = _setup(tmp_path) + outcome = _call(registry, board, "list_probes", {"families": ["no_such_family"]}) + assert outcome.ok is False and outcome.error == "empty" + + +def test_named_run_probe_runs_exactly_that_probe(tmp_path): + _, state, registry, board = _setup(tmp_path) + outcome = _call(registry, board, "run_probe", + {"name": "logit_lens", "params": {"top_k": 2}}) + assert outcome.ok is True, outcome.summary + assert set(state.probe_results) == {"logit_lens"} + assert [row["analyzer"] for row in board.probe_findings] == ["logit_lens"] + + +def test_named_calls_accumulate_evidence(tmp_path): + _, state, registry, board = _setup(tmp_path) + assert _call(registry, board, "run_probe", {"name": "logit_lens"}).ok + assert _call(registry, board, "run_probe", {"probes": [{"name": "attention"}]}).ok + assert set(state.probe_results) == {"logit_lens", "attention"} + assert {row["analyzer"] for row in board.probe_findings} == {"logit_lens", "attention"} + assert state.cycle == 2 + + +def test_bad_arguments_are_rejected_before_any_model_work(tmp_path): + loop, state, registry, board = _setup(tmp_path) + outcome = _call(registry, board, "run_probe", + {"probes": [{"name": "attention"}, + {"name": "logit_lens", "params": {"nope": 1}}]}) + assert outcome.ok is False and outcome.error == "bad_params" + assert "accepted" in outcome.summary + assert loop.model.n_forward == 0 + assert state.probe_results == {} and state.cycle == 0 + + +def test_unknown_and_unavailable_probes_are_rejected_with_reasons(tmp_path): + model = _CountingModel(capabilities={Capability.GENERATE}, modalities={"text"}) + _, _, registry, board = _setup(tmp_path, model=model) + unknown = _call(registry, board, "run_probe", {"name": "no_such_probe"}) + assert unknown.ok is False and unknown.error == "unknown_probe" + blocked = _call(registry, board, "run_probe", {"name": "logit_lens"}) + assert blocked.ok is False and blocked.error == "unavailable" + assert "hidden_states" in blocked.summary + + +def test_run_probe_without_a_name_still_lets_the_agent_select(tmp_path): + _, state, registry, board = _setup(tmp_path) + outcome = _call(registry, board, "run_probe") + assert outcome.ok is True + assert 1 <= len(state.probe_results) <= 2 # ProbeAgent(max_analyzers=2) chose + + +def test_stats_precondition_is_met_by_a_named_probe(tmp_path): + _, _, registry, board = _setup(tmp_path) + assert "BLOCKED" in [line for line in registry.catalog_for_prompt(board).splitlines() + if line.startswith("- run_stats")][0] + assert _call(registry, board, "run_probe", {"name": "logit_lens"}).ok + assert "BLOCKED" not in [line for line in registry.catalog_for_prompt(board).splitlines() + if line.startswith("- run_stats")][0] + + +def test_full_loop_drives_list_then_named_probe(tmp_path): + judge = _Judge([ + json.dumps({"tool": "list_probes", "params": {"families": ["lens"]}, + "rationale": "see what can run"}), + json.dumps({"tool": "run_probe", "params": {"name": "logit_lens", "params": {"top_k": 2}}, + "rationale": "read the layers"}), + json.dumps({"tool": "stop", "params": {"resolved": False, "reason": "enough"}, + "rationale": "done"}), + ]) + loop, _, _, _ = _setup(tmp_path, judge=judge) + report = loop.run(_cases()) + assert report.stopped_by == "agent_stop" + assert len(judge.calls) == 3 + # The second decision saw the catalog produced by the first. + assert "PROBE CATALOG" in judge.calls[1] and "logit_lens" in judge.calls[1] + + +def test_unnamed_runs_keep_their_own_limit_while_named_calls_continue(tmp_path): + _, _, registry, board = _setup(tmp_path) + for _ in range(3): + assert _call(registry, board, "run_probe").ok + fourth = _call(registry, board, "run_probe") + assert fourth.ok is False and fourth.error == "max_calls" + assert "name the probes" in fourth.summary + assert _call(registry, board, "run_probe", {"name": "logit_lens"}).ok diff --git a/tests/test_eval_agent/test_fix_weight_edit.py b/tests/test_eval_agent/test_fix_weight_edit.py new file mode 100644 index 00000000..b1c1d1aa --- /dev/null +++ b/tests/test_eval_agent/test_fix_weight_edit.py @@ -0,0 +1,302 @@ +"""L4 weight-edit repair: a training-free rank-one edit as a validated fix candidate. + +Run against a model with one planted defect (``tests/toy_wsp.py``), so every +expectation here is known in advance rather than hoped for: the edit derived +from a disjoint diagnosis pool repairs the held-out failures, breaks nothing, +is validated by the same paired test as every other tier, and is recorded +beside a matched random-direction control that does not repair them. +""" + +from __future__ import annotations + +import json + +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.core.capability import Capability # noqa: E402 +from evalrx.core.case import CaseBatch, Label # noqa: E402 +from evalrx.eval_agent.hypothesis import Hypothesis # noqa: E402 +from evalrx.eval_agent.stages.fix.agent import FixAgent # noqa: E402 +from evalrx.eval_agent.stages.fix.tiers import FixTier # noqa: E402 +from evalrx.eval_agent.stages.fix.weight_edit import ( # noqa: E402 + CANDIDATE_KIND, + CANDIDATE_NAME, + WeightEditSpec, + run_weight_edit_repair, + weight_edit_available, +) +from evalrx.wsp import TensorEdit, WeightEdit, contrast_edit # noqa: E402 +from tests.conftest import FakeModel # noqa: E402 +from tests.toy_wsp import ( # noqa: E402 + DEFECT_LAYER, + ToyModel, + echo_batch, + exact_score, + snapshot, + unchanged, +) + +_ONE = {"n_layers": 1, "n_columns": 1} + + +def _hyp() -> Hypothesis: + return Hypothesis(statement="one neuron writes the wrong answer on trigger inputs", + target_model="toy", predicted_failure_mode="prior", test_design="") + + +def _weight_edit(out): + found = [v for v in out.attempted if v.candidate.kind == CANDIDATE_KIND] + assert len(found) <= 1 + return found[0] if found else None + + +# ---------------------------------------------------------------------- +# Executor +# ---------------------------------------------------------------------- + +def test_executor_repairs_held_out_cases_and_restores_the_model(): + model = ToyModel() + snap = snapshot(model) + pool, val = echo_batch(4, 6), echo_batch(6, 8) + result = run_weight_edit_repair(model, pool, val, dict(_ONE), exact_score) + assert result.ok is True and result.error == "" + assert set(result.scores) == {c.id for c in val} + assert all(result.scores.values()) # every held-out case passes + assert set(result.outputs) == {c.id for c in val} + assert unchanged(model, snap) + json.dumps(result.audit) + + +def test_audit_records_what_was_applied_and_the_control(): + model = ToyModel() + pool, val = echo_batch(4, 6), echo_batch(6, 8) + audit = run_weight_edit_repair(model, pool, val, dict(_ONE), exact_score).audit + assert audit["edit"]["tensors"][0]["name"] == f"layers.{DEFECT_LAYER}.mlp.down_proj.weight" + assert audit["structure"][0]["changed_columns"] == 1 + assert audit["applied"]["max_relative_change"] > 0 + assert audit["edit_result"] == {"n_pass": 14, "n_scored": 14} + # The matched random direction leaves the six failures failing. + assert audit["control_random_direction"] == {"n_pass": 8, "n_scored": 14} + assert audit["settings"]["alpha"] == 1.0 + + +def test_control_can_be_switched_off(): + model = ToyModel() + before = model.n_generate + audit = run_weight_edit_repair(model, echo_batch(4, 6), echo_batch(2, 2), + {**_ONE, "control": False}, exact_score).audit + assert "control_random_direction" not in audit + assert model.n_generate - before == 4 # one pass over the batch, not two + + +@pytest.mark.parametrize("pool_factory,payload,message", [ + (lambda val: None, {}, "no diagnosis pool"), + (lambda val: CaseBatch([]), {}, "no diagnosis pool"), + (lambda val: val, {}, "must come from disjoint cases"), + (lambda val: CaseBatch([c for c in echo_batch(4, 6) if c.label == Label.FAIL]), {}, + "both failing and passing"), + (lambda val: echo_batch(4, 6), {"n_columns": 0}, "invalid weight-edit settings"), + (lambda val: echo_batch(4, 6), {"alpha": "strong"}, "invalid weight-edit settings"), + (lambda val: echo_batch(4, 6), {"layers": [99]}, "could not build the edit"), + (lambda val: echo_batch(4, 6), {"max_relative_change": 1e-9}, "could not be applied"), + (lambda val: echo_batch(4, 6), {"edit_dir": "/no/such/dir"}, "could not load edit"), +]) +def test_refusals_explain_themselves_and_touch_nothing(pool_factory, payload, message): + model = ToyModel() + snap = snapshot(model) + val = echo_batch(3, 3) + result = run_weight_edit_repair(model, pool_factory(val), val, payload, exact_score) + assert result.ok is False + assert message in result.error + assert result.scores == {} + assert model.n_generate == 0 + assert unchanged(model, snap) + + +def test_models_without_editable_weights_are_refused(): + api_like = FakeModel(capabilities={Capability.GENERATE}) + assert weight_edit_available(api_like) is False + result = run_weight_edit_repair(api_like, echo_batch(2, 2), echo_batch(2, 2), {}, exact_score) + assert result.ok is False and "white-box" in result.error + assert weight_edit_available(ToyModel()) is True + + +def test_a_failing_generate_scores_none_and_the_model_is_still_restored(): + model = ToyModel() + snap = snapshot(model) + val = echo_batch(3, 3) + doomed = list(val)[1].inputs.prompt + original = model.generate + + def flaky(inputs, **kwargs): + if inputs.prompt == doomed: + raise RuntimeError("generation failed") + return original(inputs, **kwargs) + + model.generate = flaky + result = run_weight_edit_repair(model, echo_batch(4, 6), val, {**_ONE, "control": False}, + exact_score) + assert result.ok is True + assert list(result.scores.values()).count(None) == 1 + assert unchanged(model, snap) + + +def test_a_saved_edit_is_applied_as_written_without_a_pool(tmp_path): + model = ToyModel() + pool = echo_batch(4, 6) + edit = contrast_edit(model, [c for c in pool if c.label == Label.FAIL], + [c for c in pool if c.label == Label.PASS], n_layers=1, n_columns=1) + directory = edit.save(str(tmp_path / "edit")) + val = echo_batch(5, 5) + result = run_weight_edit_repair(model, None, val, {"edit_dir": directory}, exact_score) + assert result.ok is True and all(result.scores.values()) + assert result.audit["edit"]["name"] == edit.name + + +def test_a_saved_edit_with_a_stale_base_hash_is_refused(tmp_path): + model = ToyModel() + snap = snapshot(model) + stale = WeightEdit([TensorEdit.rank_one( + f"layers.{DEFECT_LAYER}.mlp.down_proj.weight", [0, 0, 0, 0, 0, 0, 0, 1.0], 1.0)]) + stale.tensors[0].base_sha256 = "0" * 64 + directory = stale.save(str(tmp_path / "stale")) + result = run_weight_edit_repair(model, None, echo_batch(2, 2), + {"edit_dir": directory, "verify_base": True}, exact_score) + assert result.ok is False and "differs from the one the edit was made on" in result.error + assert unchanged(model, snap) + + +def test_settings_ignore_unknown_payload_keys(): + spec = WeightEditSpec.from_payload({"alpha": 3, "exec_error": "x", "audit": {}, "layers": (1,)}) + assert spec.alpha == 3.0 and spec.layers == [1] + assert "exec_error" not in spec.to_dict() + + +# ---------------------------------------------------------------------- +# Through the fix agent +# ---------------------------------------------------------------------- + +def test_fix_agent_validates_the_edit_with_the_shared_paired_test(): + model = ToyModel() + snap = snapshot(model) + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, score_fn=exact_score, + finetune_pool=echo_batch(4, 6), weight_edit=dict(_ONE)) + out = agent.propose_and_validate(model, echo_batch(16, 10), [_hyp()]) + + v = _weight_edit(out) + assert v is not None + assert v.candidate.tier is FixTier.L4_PARAMETERS and v.candidate.name == CANDIDATE_NAME + assert v.exec_error == "" + assert v.n_fixed == 16 and v.n_broken == 0 + assert v.verdict == "fixed" and v.e_value > v.e_threshold + assert len(v.outputs) == 26 # what it answered is kept per case + assert out.fixed is True and out.best.candidate.name == CANDIDATE_NAME + audit = v.candidate.payload["audit"] + assert audit["edit_result"]["n_pass"] == 26 + assert audit["control_random_direction"]["n_pass"] == 10 + assert unchanged(model, snap) + + +def test_weight_edit_is_opt_in(): + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, score_fn=exact_score, + finetune_pool=echo_batch(4, 6)) + out = agent.propose_and_validate(ToyModel(), echo_batch(8, 4), [_hyp()]) + assert _weight_edit(out) is None + + +def test_weight_edit_is_only_offered_at_the_parameter_tier(): + agent = FixAgent(judge=None, max_tier="L3b", allow_codegen=False, score_fn=exact_score, + finetune_pool=echo_batch(4, 6), weight_edit=True) + out = agent.propose_and_validate(ToyModel(), echo_batch(8, 4), [_hyp()]) + assert _weight_edit(out) is None + + +def test_without_a_pool_the_candidate_is_recorded_not_executed(): + model = ToyModel() + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, score_fn=exact_score, + weight_edit=True) + out = agent.propose_and_validate(model, echo_batch(8, 4), [_hyp()]) + v = _weight_edit(out) + assert v is not None and v.fixed is False + assert v.verdict == "not_executed" + assert "no diagnosis pool" in v.summary + + +def test_a_pool_that_overlaps_the_validated_batch_is_never_used(): + model = ToyModel() + snap = snapshot(model) + batch = echo_batch(8, 4) + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, score_fn=exact_score, + finetune_pool=batch, weight_edit=dict(_ONE)) + out = agent.propose_and_validate(model, batch, [_hyp()]) + v = _weight_edit(out) + assert v.verdict == "not_executed" and "disjoint" in v.summary + assert out.fixed is False + assert unchanged(model, snap) + + +def test_not_offered_for_a_model_without_internals(): + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, weight_edit=True, + finetune_pool=echo_batch(4, 6)) + out = agent.propose_and_validate( + FakeModel(capabilities={Capability.GENERATE}), echo_batch(8, 4), [_hyp()]) + assert _weight_edit(out) is None + + +def test_invalid_agent_settings_propose_nothing(): + agent = FixAgent(judge=None, max_tier="L4", allow_codegen=False, score_fn=exact_score, + finetune_pool=echo_batch(4, 6), weight_edit={"n_layers": 0}) + out = agent.propose_and_validate(ToyModel(), echo_batch(8, 4), [_hyp()]) + assert _weight_edit(out) is None + + +def test_the_candidate_has_a_plain_language_description(): + from evalrx.eval_agent.stages.fix.descriptions import BUILTIN_DESCRIPTIONS + + text = BUILTIN_DESCRIPTIONS[CANDIDATE_NAME] + assert "without any training" in text and "random" in text + + +# ---------------------------------------------------------------------- +# A real local model handle +# ---------------------------------------------------------------------- + +def test_executor_on_a_real_local_model_handle_restores_generation(): + """Mechanics on ``HFLocalModel`` over a from-scratch Llama: the edit is built, + applied during generation and fully removed. Whether it helps is not at + issue here — the weights are random.""" + pytest.importorskip("transformers") + from evalrx.core.case import FailureCase, Inputs + from evalrx.core.spec import ModelSpec + from evalrx.models.backends.base import RuntimeConfig + from evalrx.models.backends.hf.model import HFLocalModel + from tests.test_eval_agent.test_fix_agent import _tiny_llama, _TinyWordTokenizer + + spec = ModelSpec(key="tiny-llama-wsp", family="fake", model_type="fake_llm", hf_repo="") + model = HFLocalModel(spec, RuntimeConfig(device="cpu", dtype="float32", max_new_tokens=3)) + model._hf = (_tiny_llama(), _TinyWordTokenizer()) + assert weight_edit_available(model) is True + + def case(word, label): + return FailureCase(inputs=Inputs(prompt=f"classify {word}"), expected="yes", label=label) + + pool = CaseBatch([case("alpha", Label.FAIL), case("alpha", Label.FAIL), + case("beta", Label.PASS), case("beta", Label.PASS)]) + val = CaseBatch([case("alpha", Label.FAIL), case("beta", Label.PASS)]) + snap = snapshot(model) + before = [model.generate(c.inputs) for c in val] + + result = run_weight_edit_repair( + model, pool, val, {"alpha": 6.0, "n_columns": 8}, + lambda c, out: str(c.expected) in str(out)) + assert result.ok is True, result.error + assert set(result.scores) == {c.id for c in val} + names = [t["name"] for t in result.audit["applied"]["tensors"]] + assert names and all(n.endswith("mlp.down_proj.weight") for n in names) + assert all(row["alpha_estimate"] == pytest.approx(6.0, abs=1e-2) + for row in result.audit["structure"]) + assert "n_pass" in result.audit["control_random_direction"] + assert unchanged(model, snap) + assert [model.generate(c.inputs) for c in val] == before diff --git a/tests/test_probing/__init__.py b/tests/test_probing/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/test_probing/test_catalog.py b/tests/test_probing/test_catalog.py new file mode 100644 index 00000000..b3199b4a --- /dev/null +++ b/tests/test_probing/test_catalog.py @@ -0,0 +1,217 @@ +"""The probe catalog: every analyzer described as a tool, with honest availability.""" + +from __future__ import annotations + +import json + +import pytest + +import evalrx # noqa: F401 - registers the analyzers +from evalrx.core.analyzer import Analyzer +from evalrx.core.capability import Capability +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label +from evalrx.core.registry import registry +from evalrx.core.result import Result +from evalrx.core.tool import Tool +from evalrx.probing import ( + FAMILY_QUESTIONS, + availability, + describe_probes, + family_of, + probe_specs, + spec_for, +) +from evalrx.probing.catalog import _annotation_type, _params_of, is_stub +from tests.conftest import FakeModel + + +def _text_batch(n: int = 4) -> CaseBatch: + return CaseBatch([ + FailureCase(inputs=Inputs(prompt=f"question {i}"), + label=Label.FAIL if i % 2 else Label.PASS) + for i in range(n) + ]) + + +def test_every_registered_analyzer_has_a_spec(): + specs = probe_specs() + assert set(specs) == set(registry.analyzers.list()) + for name, spec in specs.items(): + assert spec.name == name + assert spec.summary, name + assert spec.question, name + assert spec.access in {"black_box", "white_box", "data_only"}, name + assert spec.cost in {"light", "moderate", "heavy"}, name + + +def test_every_family_is_documented(): + families = {spec.family for spec in probe_specs().values()} + assert families <= set(FAMILY_QUESTIONS), families - set(FAMILY_QUESTIONS) + + +def test_family_comes_from_the_package_with_renames(): + assert family_of(registry.analyzers.get("logit_lens")) == "lens" + assert family_of(registry.analyzers.get("attention_knockout")) == "causal" + assert family_of(registry.analyzers.get("weight_edit_audit")) == "weight_edit" + + class Outside(Analyzer): + def _run(self, model, cases): # pragma: no cover - never run + return Result(analyzer="x", model="m") + + assert family_of(Outside) == "custom" + Outside.probe_family = "lens" + assert family_of(Outside) == "lens" + + +def test_access_follows_required_capabilities(): + specs = probe_specs() + assert specs["self_consistency"].access == "black_box" + assert specs["logit_lens"].access == "white_box" + assert specs["loop_detect"].access == "data_only" + assert specs["attention_knockout"].intervenes is True + assert specs["logit_lens"].intervenes is False + + +def test_schemas_are_valid_json_and_reject_extra_arguments(): + for name, spec in probe_specs().items(): + schema = spec.parameters_schema() + json.dumps(schema) # every default must be JSON-serialisable + assert schema["type"] == "object", name + assert schema["additionalProperties"] is False, name + for param in spec.callable_params: + assert not param.injected, (name, param.name) + + +def test_host_only_arguments_stay_out_of_the_schema(): + spec = spec_for("first_error_judge") + assert "judge" in {p.name for p in spec.params} + assert "judge" not in spec.parameters_schema()["properties"] + assert spec.injected_params == () # optional: the probe can run without it + + required = spec_for("trajectory_rubric") + assert required.injected_params == ("judge",) + + +@pytest.mark.parametrize("annotation,expected", [ + ("int", ("integer", False, True)), + ("float | None", ("number", True, True)), + ("Optional[int]", ("integer", True, True)), + ("dict[str, int | None] | None", ("object", True, True)), + ("tuple[str, ...]", ("array", False, True)), + ("int | float", ("number", False, True)), + ("Optional[Callable[[Any], str]]", (None, True, True)), + ("'Model | None'", (None, True, True)), + ("Any", (None, True, False)), +]) +def test_annotation_parsing(annotation, expected): + assert _annotation_type(annotation) == expected + + +def test_parameter_types_fall_back_to_the_default_value(): + class Probe(Analyzer): + def __init__(self, k=3, ratio=0.5, flag=False, tag="x", fn=len): + super().__init__(k=k, ratio=ratio, flag=flag, tag=tag, fn=fn) + + def _run(self, model, cases): # pragma: no cover + return Result(analyzer="x", model="m") + + params = {p.name: p for p in _params_of(Probe)} + assert params["k"].json_type == "integer" + assert params["ratio"].json_type == "number" + assert params["flag"].json_type == "boolean" + assert params["tag"].json_type == "string" + assert params["fn"].injected is True + + +def test_to_tool_produces_an_openai_schema(): + tool = spec_for("logit_lens").to_tool() + assert isinstance(tool, Tool) + schema = tool.to_openai_schema() + assert schema["type"] == "function" + assert schema["function"]["name"] == "logit_lens" + assert schema["function"]["parameters"]["type"] == "object" + + +def test_failure_modes_are_the_inverse_of_the_routing_table(): + from evalrx.eval_agent.stages.probe import _FAILURE_MODE_TO_ANALYZERS + + specs = probe_specs() + for mode, names in _FAILURE_MODE_TO_ANALYZERS.items(): + for name in names: + if name in specs: + assert mode in specs[name].failure_modes + + +def test_placeholders_are_reported_as_not_implemented(): + stubs = [name for name, spec in probe_specs().items() if not spec.implemented] + for name in stubs: + assert is_stub(registry.analyzers.get(name)) + model = FakeModel(capabilities=set(Capability)) + for name in stubs: + verdict = availability(spec_for(name), model) + assert verdict.available is False + assert "not_implemented" in verdict.codes + + +def test_missing_capability_names_what_is_missing(): + model = FakeModel(capabilities={Capability.GENERATE}) + verdict = availability(spec_for("logit_lens"), model, _text_batch()) + assert verdict.available is False + assert verdict.codes == ("capability",) + assert "hidden_states" in verdict.missing[0] + + +def test_all_reasons_are_reported_not_just_the_first(): + model = FakeModel(capabilities={Capability.GENERATE}) + verdict = availability(spec_for("attention_knockout"), model, _text_batch()) + assert {"capability", "modality", "data_slot"} <= set(verdict.codes) + assert len(verdict.missing) == len(verdict.codes) + + +def test_trajectory_probes_need_trajectories_in_the_batch(): + model = FakeModel(capabilities={Capability.GENERATE}) + verdict = availability(spec_for("loop_detect"), model, _text_batch()) + assert verdict.codes == ("trajectories",) + # Without a batch there is no evidence either way: nothing is assumed missing. + assert availability(spec_for("loop_detect"), model).available is True + + +def test_injected_argument_is_satisfied_by_an_override(): + model = FakeModel(capabilities={Capability.GENERATE}) + spec = spec_for("trajectory_rubric") + assert "injection" in availability(spec, model).codes + assert "injection" not in availability(spec, model, overrides=["trajectory_rubric"]).codes + + +def test_describe_probes_lists_everything_with_a_verdict(): + model = FakeModel(capabilities={Capability.GENERATE}) + report = describe_probes(model, _text_batch()) + assert set(report) == set(registry.analyzers.list()) + for name, row in report.items(): + assert isinstance(row["available"], bool) + assert row["available"] == (not row["missing"]), name + assert len(row["missing"]) == len(row["codes"]), name + json.dumps(report) + assert report["self_consistency"]["available"] is True + assert report["logit_lens"]["available"] is False + + +def test_describe_probes_filters_by_family(): + report = describe_probes(FakeModel(), families=["lens"]) + assert report and all(row["family"] == "lens" for row in report.values()) + + +def test_describe_runs_no_model_call(): + class Counting(FakeModel): + calls = 0 + + def generate(self, inputs, **kwargs): + type(self).calls += 1 + return "x" + + def forward(self, inputs, capture, spec=None): + type(self).calls += 1 + return super().forward(inputs, capture, spec) + + describe_probes(Counting(capabilities=set(Capability)), _text_batch()) + assert Counting.calls == 0 diff --git a/tests/test_probing/test_router_toolbox.py b/tests/test_probing/test_router_toolbox.py new file mode 100644 index 00000000..7c83b592 --- /dev/null +++ b/tests/test_probing/test_router_toolbox.py @@ -0,0 +1,302 @@ +"""Routing a question to probes, and calling a probe by name with JSON arguments.""" + +from __future__ import annotations + +import pytest + +import evalrx # noqa: F401 - registers the analyzers +from evalrx.analyzers.agent.trajectory_rubric import TrajectoryRubricJudge +from evalrx.core.capability import Capability +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label +from evalrx.core.result import Result +from evalrx.core.tool import ToolResult +from evalrx.eval_agent.stages.probe import StrategyProbe +from evalrx.eval_agent.stages.probe_agent import ProbeAgent +from evalrx.probing import ( + ProbeCallError, + ProbeRouter, + ProbeToolbox, + build_probe, + check_params, + spec_for, + summarize_result, +) +from tests.conftest import FakeModel + +_WHITEBOX = {Capability.GENERATE, Capability.ATTENTION, Capability.HIDDEN_STATES, + Capability.LOGITS, Capability.LOGPROBS} + + +def _batch(n: int = 6) -> CaseBatch: + return CaseBatch([ + FailureCase(inputs=Inputs(prompt=f"question {i}"), + label=Label.FAIL if i % 2 else Label.PASS) + for i in range(n) + ]) + + +# ---------------------------------------------------------------------- +# Router +# ---------------------------------------------------------------------- + +def test_route_selects_only_runnable_probes(): + model = FakeModel(capabilities={Capability.GENERATE}) + route = ProbeRouter().route(model, _batch()) + assert route.selected + for probe in route.selected: + assert probe.spec.access != "white_box" + + +def test_route_order_matches_the_strategy_selector(): + model = FakeModel(capabilities=_WHITEBOX) + data = _batch() + route = ProbeRouter().route(model, data) + expected = StrategyProbe().select(model, data=data) + runnable = set(route.names) + assert route.names == [n for n in expected if n in runnable] + + +def test_failure_modes_lead_and_carry_the_reason(): + model = FakeModel(capabilities=_WHITEBOX) + route = ProbeRouter().route(model, _batch(), failure_modes=["Overconfidence"]) + assert route.names[:2] == ["verbalized_confidence", "calibration"] + assert "targets overconfidence" in route.selected[0].reasons[0] + + +def test_requested_but_unavailable_probes_are_reported_with_reasons(): + model = FakeModel(capabilities={Capability.GENERATE}) + route = ProbeRouter().route(model, _batch(), failure_modes=["logit_lens"]) + assert "logit_lens" in route.unavailable + assert "hidden_states" in route.unavailable["logit_lens"][0] + assert "logit_lens" not in route.names + + +def test_unknown_failure_modes_are_named_not_dropped(): + route = ProbeRouter().route(FakeModel(), _batch(), failure_modes=["no_such_mode"]) + assert route.unknown_modes == ("no_such_mode",) + + +def test_only_matching_keeps_just_the_targeted_probes(): + model = FakeModel(capabilities=_WHITEBOX) + route = ProbeRouter().route(model, _batch(), failure_modes=["overconfidence"], + only_matching=True) + assert set(route.names) == {"verbalized_confidence", "calibration"} + + +def test_filters_by_family_access_and_intervention(): + model = FakeModel(capabilities=_WHITEBOX) + lens = ProbeRouter().route(model, _batch(), families=["lens"]) + assert lens.selected and all(p.spec.family == "lens" for p in lens.selected) + + black = ProbeRouter().route(model, _batch(), access="black_box") + assert all(p.spec.access != "white_box" for p in black.selected) + + white = ProbeRouter().route(model, _batch(), access="white_box") + assert white.selected and all(p.spec.access == "white_box" for p in white.selected) + + passive = ProbeRouter().route(model, _batch(), allow_interventions=False) + assert all(not p.spec.intervenes for p in passive.selected) + + +def test_max_probes_caps_the_selection(): + route = ProbeRouter().route(FakeModel(capabilities=_WHITEBOX), _batch(), max_probes=3) + assert len(route.selected) == 3 + + +def test_prompt_rendering_lists_names_arguments_and_rejections(): + model = FakeModel(capabilities={Capability.GENERATE}) + text = ProbeRouter().route(model, _batch(), failure_modes=["overconfidence", "logit_lens"], + max_probes=4).to_prompt() + assert "verbalized_confidence" in text + assert "params:" in text + assert "UNAVAILABLE:" in text and "logit_lens" in text + + +# ---------------------------------------------------------------------- +# Arguments +# ---------------------------------------------------------------------- + +def test_unknown_arguments_are_rejected_with_the_accepted_list(): + with pytest.raises(ProbeCallError) as err: + check_params(spec_for("logit_lens"), {"topk": 3}) + assert "accepted" in str(err.value) + assert err.value.code == "bad_params" + + +def test_wrong_types_are_rejected(): + with pytest.raises(ProbeCallError, match="expects integer"): + check_params(spec_for("self_consistency"), {"n": "five"}) + with pytest.raises(ProbeCallError, match="expects integer"): + check_params(spec_for("self_consistency"), {"n": True}) + + +def test_whole_floats_are_accepted_for_integer_fields(): + cleaned = check_params(spec_for("self_consistency"), {"n": 4.0}) + assert cleaned == {"n": 4} and isinstance(cleaned["n"], int) + with pytest.raises(ProbeCallError, match="expects integer"): + check_params(spec_for("self_consistency"), {"n": 4.5}) + + +def test_host_only_argument_in_a_call_is_explained(): + with pytest.raises(ProbeCallError) as err: + check_params(spec_for("first_error_judge"), {"judge": "gpt"}) + assert "supplied by the host" in str(err.value) + + +def test_build_probe_constructs_with_the_given_arguments(): + analyzer = build_probe("self_consistency", {"n": 2}) + assert analyzer.name == "self_consistency" + assert analyzer.get_params()["n"] == 2 + + +def test_build_probe_unknown_name(): + with pytest.raises(ProbeCallError) as err: + build_probe("no_such_probe") + assert err.value.code == "unknown_probe" + + +def test_build_probe_needs_an_override_for_injected_arguments(): + with pytest.raises(ProbeCallError) as err: + build_probe("trajectory_rubric") + assert err.value.code == "unavailable" + + judge = FakeModel(capabilities={Capability.GENERATE}) + override = TrajectoryRubricJudge(judge=judge) + assert build_probe("trajectory_rubric", overrides={"trajectory_rubric": override}) is override + + # Per-call arguments build a NEW instance; the override is not mutated. + before = dict(override.get_params()) + rebuilt = build_probe("trajectory_rubric", {"max_cases": 2}, + overrides={"trajectory_rubric": override}) + assert rebuilt is not override + assert rebuilt.get_params()["max_cases"] == 2 + assert rebuilt.get_params()["judge"] is judge + assert override.get_params() == before + + +# ---------------------------------------------------------------------- +# Toolbox +# ---------------------------------------------------------------------- + +def test_call_runs_the_probe_and_returns_a_result(): + box = ProbeToolbox(FakeModel(capabilities=_WHITEBOX), _batch()) + result = box.call("logit_lens", {"top_k": 2}) + assert isinstance(result, Result) + assert result.analyzer == "logit_lens" + + +def test_call_refuses_an_unavailable_probe_before_running(): + box = ProbeToolbox(FakeModel(capabilities={Capability.GENERATE}), _batch()) + with pytest.raises(ProbeCallError) as err: + box.call("logit_lens") + assert err.value.code == "unavailable" + assert "hidden_states" in str(err.value) + + +def test_call_without_data_is_refused(): + with pytest.raises(ProbeCallError): + ProbeToolbox(FakeModel()).call("attention") + + +def test_available_matches_describe(): + box = ProbeToolbox(FakeModel(capabilities={Capability.GENERATE}), _batch()) + report = box.describe() + assert box.available() == sorted(n for n, row in report.items() if row["available"]) + + +def test_tools_are_bound_and_callable(): + box = ProbeToolbox(FakeModel(capabilities=_WHITEBOX), _batch()) + tools = {tool.name: tool for tool in box.tools()} + assert set(tools) == set(box.available()) + out = tools["logit_lens"].fn(top_k=2) + assert isinstance(out, ToolResult) + assert "logit_lens" in out.text + assert isinstance(out.meta["result"], Result) + + +def test_tools_can_include_unavailable_probes_on_request(): + box = ProbeToolbox(FakeModel(capabilities={Capability.GENERATE}), _batch()) + names = {tool.name for tool in box.tools(["logit_lens"], only_available=False)} + assert names == {"logit_lens"} + assert box.tools(["logit_lens"]) == [] + + +def test_max_cases_caps_what_a_probe_sees(): + seen = {} + + class Counting(FakeModel): + def generate(self, inputs, **kwargs): + seen[str(inputs)] = True + return "x" + + box = ProbeToolbox(Counting(capabilities={Capability.GENERATE}), _batch(6), max_cases=2) + result = box.call("verbalized_confidence") + assert len(result.cases) == 2 + + +def test_summarize_result_keeps_scalars_and_counts_rows(): + result = Result(analyzer="a", model="m", findings={ + "mean": 0.5, "_caveat": "hidden", "per_case": [{"x": 1}, {"x": 2}], "nested": {"k": 1}, + }) + text = summarize_result(result) + assert "'mean': 0.5" in text and "2 per-case rows" in text + assert "_caveat" not in text and "nested" not in text + + +# ---------------------------------------------------------------------- +# ProbeAgent pass-through +# ---------------------------------------------------------------------- + +def test_probe_agent_runs_named_probes_with_arguments(): + agent = ProbeAgent(max_workers=1) + model = FakeModel(capabilities=_WHITEBOX) + results = agent.probe(model, _batch(), analyzers=["logit_lens"], + analyzer_params={"logit_lens": {"top_k": 2}}) + assert set(results) == {"logit_lens"} + assert "Called by name" in agent.last_schema.rationale + + +def test_exact_calls_are_neither_capped_nor_backfilled(): + model = FakeModel(capabilities=_WHITEBOX) + capped = ProbeAgent(max_workers=1, max_analyzers=1) + both = capped.probe(model, _batch(), analyzers=["logit_lens", "attention"], exact=True) + assert set(both) == {"logit_lens", "attention"} # the cap does not apply + + roomy = ProbeAgent(max_workers=1, max_analyzers=3) + only = roomy.probe(model, _batch(), analyzers=["logit_lens"], exact=True) + assert set(only) == {"logit_lens"} # nothing was added to fill the slots + + # An inapplicable named probe is dropped and reported, not replaced. + none = roomy.probe(model, _batch(), analyzers=["loop_detect"], exact=True) + assert none == {} + assert "loop_detect" in roomy.last_schema.rationale + + +def test_probe_agent_surfaces_a_bad_call(): + agent = ProbeAgent(max_workers=1) + with pytest.raises(ProbeCallError): + agent.probe(FakeModel(capabilities=_WHITEBOX), _batch(), analyzers=["logit_lens"], + analyzer_params={"logit_lens": {"nope": 1}}) + + +def test_pinned_rerun_replays_the_arguments_of_the_named_call(): + built = [] + agent = ProbeAgent(max_workers=1) + original = agent._make_analyzer + + def spy(name, params=None): + built.append((name, params)) + return original(name, params) + + agent._make_analyzer = spy + model = FakeModel(capabilities=_WHITEBOX) + agent.probe(model, _batch(), analyzers=["logit_lens"], exact=True, + analyzer_params={"logit_lens": {"top_k": 2}}) + # The held-out confirmation pins the same names without arguments. + agent.probe(model, _batch(), analyzers=["logit_lens"]) + assert built == [("logit_lens", {"top_k": 2}), ("logit_lens", {"top_k": 2})] + + # A later named call with no arguments goes back to the defaults. + agent.probe(model, _batch(), analyzers=["logit_lens"], exact=True) + agent.probe(model, _batch(), analyzers=["logit_lens"]) + assert built[2:] == [("logit_lens", None), ("logit_lens", None)] diff --git a/tests/test_tools/__init__.py b/tests/test_tools/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/test_tools/test_generate_astra_edit.py b/tests/test_tools/test_generate_astra_edit.py new file mode 100644 index 00000000..4b034a15 --- /dev/null +++ b/tests/test_tools/test_generate_astra_edit.py @@ -0,0 +1,78 @@ +"""Grading and loop signals of tools/generate_astra_edit_qwen35.py. + +The script's generation path needs a GPU and a real checkpoint; what decides its +numbers on any machine is how an answer is read and compared, so that part is +tested here. +""" + +from __future__ import annotations + +import importlib.util +import sys +from pathlib import Path + +import pytest + +TOOLS = Path(__file__).resolve().parents[2] / "tools" + + +@pytest.fixture(scope="module") +def gen(): + sys.path.insert(0, str(TOOLS)) + try: + spec = importlib.util.spec_from_file_location( + "generate_astra_edit_qwen35", TOOLS / "generate_astra_edit_qwen35.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + finally: + sys.path.remove(str(TOOLS)) + + +def test_answer_is_read_after_the_thinking_block(gen): + text = "maybe \\boxed{7}\nThe answer is \\boxed{18}." + assert gen.answer_text(text) == "The answer is \\boxed{18}." + assert gen.is_correct(text, "18") is True + # An answer boxed only inside an unfinished thinking block still counts as given: + # there is no later text to prefer. + assert gen.is_correct(" so \\boxed{18}", "18") is True + + +@pytest.mark.parametrize("text,expected", [ + ("\\boxed{18}", "18"), + ("first \\boxed{1} then \\boxed{2}", "2"), + ("\\boxed{\\frac{1}{2}}", "\\frac{1}{2}"), + ("\\boxed {5}", "5"), + ("\\boxed{12", None), + ("\\boxed 5", None), + ("no box", None), +]) +def test_last_boxed(gen, text, expected): + assert gen.last_boxed(text) == expected + + +@pytest.mark.parametrize("value,expected", [ + ("18", 18.0), ("-3.5", -3.5), ("$1,250$", 1250.0), ("\\$70000", 70000.0), ("45\\%", 45.0), + ("18\\text{ dollars}", 18.0), ("\\frac{1}{2}", 0.5), ("\\dfrac{3}{4}", 0.75), ("3/4", 0.75), + ("x=5", None), ("eighteen", None), ("", None), (None, None), ("1/0", None), +]) +def test_to_number(gen, value, expected): + assert gen.to_number(value) == expected + + +def test_is_correct_compares_numbers_not_strings(gen): + assert gen.is_correct("\\boxed{18.0}", "18") is True + assert gen.is_correct("\\boxed{\\$18}", "18") is True + assert gen.is_correct("\\boxed{19}", "18") is False + assert gen.is_correct("the answer is 18", "18") is False # no box, no credit + assert gen.is_correct("\\boxed{many}", "18") is False + + +def test_verbatim_loop_end(gen): + block = "so the total is forty two dollars and we check again" + looping = "We start. " + " ".join([block] * 6) + " so the tot" + assert gen.verbatim_loop_end(looping) is True + assert gen.verbatim_loop_end("a plain answer with no repetition at all, the end.") is False + # Three repeats of a short block do not cover enough words. + assert gen.verbatim_loop_end("x y z x y z x y z cut") is False + assert gen.verbatim_loop_end("") is False diff --git a/tests/test_wsp/__init__.py b/tests/test_wsp/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/test_wsp/test_continuation_contrast.py b/tests/test_wsp/test_continuation_contrast.py new file mode 100644 index 00000000..2966a74d --- /dev/null +++ b/tests/test_wsp/test_continuation_contrast.py @@ -0,0 +1,246 @@ +"""Deriving an edit from loop vs productive positions, fitting a rank-one rule to +an edit, and comparing two edits — on the model with a known planted defect.""" + +from __future__ import annotations + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.core.case import FailureCase, Inputs, Label # noqa: E402 +from evalrx.wsp import ( # noqa: E402 + TensorEdit, + WeightEdit, + applied, + collect_continuation_stats, + continuation_contrast_edit, + edit_agreement, + encode_continuation, + encode_continuations, + fit_rank_one, + rank_columns, + write_sites, +) +from tests.toy_wsp import ( # noqa: E402 + DEFECT_LAYER, + DEFECT_NEURON, + N_LAYERS, + WRONG_TOKEN, + D, + ToyModel, + snapshot, + unchanged, +) + +_PLAIN = "3 4 5 6 2 5 3 6 4 2 6 3" +_PROMPT = "2 2" +_KEY = f"layers.{DEFECT_LAYER}.mlp.down_proj.weight" + + +def _gen(continuation, prompt: str = _PROMPT) -> FailureCase: + return FailureCase(inputs=Inputs(prompt=prompt), observed=continuation, label=Label.FAIL) + + +def _looping(trigger: int = 0, cycles: int = 8) -> FailureCase: + return _gen(f"{_PLAIN} {' '.join([f'{trigger} {WRONG_TOKEN}'] * cycles)}") + + +def _reference(alpha: float = 6.0) -> WeightEdit: + return WeightEdit([TensorEdit.rank_one(_KEY, np.eye(D)[WRONG_TOKEN], alpha, + cols=[DEFECT_NEURON])], name="reference") + + +# ---------------------------------------------------------------------- +# Sequences +# ---------------------------------------------------------------------- + +def test_sequence_has_prompt_then_continuation_with_grouped_targets(): + model = ToyModel() + seq = encode_continuation(model, _gen("0 7 0 7 0 7 3"), repeat_ngram=2) + assert seq.ids == [2, 2, 0, 7, 0, 7, 0, 7, 3] and seq.start == 2 + assert seq.targets == [2, 3, 4, 5, 6, 7, 8] + assert seq.labels == ["productive"] * 4 + ["loop"] * 3 + # A target is predicted from the state one position earlier. + assert seq.positions() == [1, 2, 3, 4, 5, 6, 7] + assert seq.positions("loop") == [5, 6, 7] + + +def test_sequence_reasons_and_truncation(): + model = ToyModel() + assert encode_continuation(model, _gen(None)) == "no_continuation" + assert encode_continuation(model, _gen("0 7"), max_tokens=2) == ( + "no_continuation_tokens_after_truncation") + assert len(encode_continuation(model, _gen("0 7 0 7"), max_tokens=4).ids) == 4 + assert encode_continuation(model, _gen("0 7"), groups="none").labels == ["all", "all"] + with pytest.raises(ValueError, match="groups must be one of"): + encode_continuation(model, _gen("0 7"), groups="paragraph") + sequences, skipped = encode_continuations(model, [_gen("0 7"), _gen(None), _gen(" ")]) + assert len(sequences) == 1 and dict(skipped) == {"no_continuation": 2} + + +# ---------------------------------------------------------------------- +# Statistics and the derived edit +# ---------------------------------------------------------------------- + +def test_stats_cover_every_layer_and_count_positions(): + model = ToyModel() + stats = collect_continuation_stats(model, [_looping(0), _looping(1)], repeat_ngram=2) + assert stats.n_cases == 2 + assert stats.n_positions("loop") == 24 and stats.n_positions("productive") == 32 + for group in ("loop", "productive"): + assert sorted(stats.groups[group]) == list(range(N_LAYERS)) + for row in stats.groups[group].values(): + assert row["in"].shape == (D,) and row["out"].shape == (D,) + for site in write_sites(model): + assert not site.module._forward_hooks + + +def test_stats_match_a_direct_computation(): + model = ToyModel() + case = _gen("0 7 0 7 0 7 3") + stats = collect_continuation_stats(model, [case], repeat_ngram=2) + captured = {} + site = write_sites(model)[DEFECT_LAYER] + handle = site.module.register_forward_hook( + lambda m, args, out: captured.update(x=args[0].detach()[0].double(), + y=out.detach()[0].double())) + with torch.no_grad(): + model.module(input_ids=torch.tensor([[2, 2, 0, 7, 0, 7, 0, 7, 3]])) + handle.remove() + loop = stats.groups["loop"][DEFECT_LAYER] + assert torch.allclose(loop["in"], captured["x"][[5, 6, 7]].mean(dim=0)) + assert torch.allclose(loop["out"], captured["y"][[5, 6, 7]].mean(dim=0)) + assert loop["n"] == 3 + + +def test_the_planted_neuron_fires_more_inside_loops(): + model = ToyModel() + stats = collect_continuation_stats(model, [_looping(0), _looping(1)], repeat_ngram=2) + inside = float(stats.groups["loop"][DEFECT_LAYER]["in"][DEFECT_NEURON]) + outside = float(stats.groups["productive"][DEFECT_LAYER]["in"][DEFECT_NEURON]) + assert inside > 3 * outside + + +def test_loop_contrast_recovers_the_planted_layer_neuron_and_direction(): + model = ToyModel() + snap = snapshot(model) + edit = continuation_contrast_edit(model, [_looping(0), _looping(1)], repeat_ngram=2, + n_layers=1, n_columns=1, alpha=6.0) + assert edit.tensor_names == [_KEY] + assert edit.tensors[0].cols.tolist() == [DEFECT_NEURON] + assert edit.metadata["position"] == "continuation" + assert edit.metadata["n_loop_positions"] == 24 + # The planted layer separates loops from ordinary text most clearly. + separation = edit.metadata["layer_separation"] + assert max(separation, key=separation.get) == str(DEFECT_LAYER) + agreement = edit_agreement(model, edit, _reference())["tensors"][0] + assert agreement["direction_cosine"] > 0.99 + assert agreement["n_shared"] == 1 and agreement["jaccard"] == 1.0 + assert unchanged(model, snap) + + +def test_precomputed_stats_give_the_same_edit(): + model = ToyModel() + cases = [_looping(0), _looping(1)] + stats = collect_continuation_stats(model, cases, repeat_ngram=2) + direct = continuation_contrast_edit(model, cases, repeat_ngram=2, n_layers=1, n_columns=2) + reused = continuation_contrast_edit(model, [], n_layers=1, n_columns=2, stats=stats) + assert reused.tensor_names == direct.tensor_names + assert np.allclose(reused.tensors[0].u, direct.tensors[0].u) + assert reused.tensors[0].cols.tolist() == direct.tensors[0].cols.tolist() + + +def test_both_position_groups_are_required(): + model = ToyModel() + with pytest.raises(ValueError, match="need both loop and productive positions"): + continuation_contrast_edit(model, [_gen(_PLAIN)], repeat_ngram=2) + with pytest.raises(ValueError, match="need both loop and productive positions"): + continuation_contrast_edit(model, [_gen(None)], repeat_ngram=2) + + +def test_rank_columns_reports_each_signal(): + model = ToyModel() + stats = collect_continuation_stats(model, [_looping(0), _looping(1)], repeat_ngram=2) + weight = write_sites(model)[DEFECT_LAYER].module.weight + row = rank_columns(stats.groups["loop"][DEFECT_LAYER], + stats.groups["productive"][DEFECT_LAYER], weight) + assert int(row["push"].argmax()) == DEFECT_NEURON + assert int(row["projection"].abs().argmax()) == DEFECT_NEURON + assert int(row["activation"].argmax()) == DEFECT_NEURON + assert torch.allclose(row["push"], row["activation"] * row["projection"]) + same = rank_columns(stats.groups["loop"][0], stats.groups["loop"][0], weight) + assert same["u"] is None and same["separation"] == 0.0 + + +# ---------------------------------------------------------------------- +# Fitting a rank-one rule, comparing edits +# ---------------------------------------------------------------------- + +def test_fit_rank_one_recovers_the_rule_behind_a_dense_edit(): + model = ToyModel() + snap = snapshot(model) + dense = _reference(alpha=6.0).as_delta(model) + assert dense.tensors[0].form == "delta" + fit = fit_rank_one(model, dense) + row = fit.metadata["fit"][0] + assert row["alpha"] == pytest.approx(6.0, abs=1e-4) + assert row["rank_one_residual"] < 1e-6 and row["n_columns"] == 1 + assert fit.tensors[0].form == "rank_one" and fit.tensors[0].cols.tolist() == [DEFECT_NEURON] + with applied(model, dense): + target = snapshot(model) + with applied(model, fit): + assert all(torch.allclose(v, target[k], atol=1e-5) + for k, v in model.module.state_dict().items()) + assert unchanged(model, snap) + + +def test_a_fitted_edit_can_be_scaled_where_the_original_cannot(): + model = ToyModel() + weight = write_sites(model)[DEFECT_LAYER].module.weight.detach() + with applied(model, _reference(6.0)): + edited = write_sites(model)[DEFECT_LAYER].module.weight.detach().clone() + changed = torch.nonzero((edited != weight).reshape(-1)).flatten() + overwrite = WeightEdit([TensorEdit(name=_KEY, form="sparse_set", index=changed, + values=edited.reshape(-1)[changed])]) + with pytest.raises(ValueError, match="cannot be scaled"): + overwrite.scaled(0.5) + half = fit_rank_one(model, overwrite).scaled(0.5) + assert half.tensors[0].alpha == pytest.approx(3.0, abs=1e-4) + with applied(model, half) as small, applied(model, _reference(3.0)): + pass + assert small.relative_change > 0 + + +def test_fit_refuses_an_edit_with_nothing_to_fit(): + model = ToyModel() + null = WeightEdit([TensorEdit(name=_KEY, delta=np.zeros((D, D)))]) + with pytest.raises(ValueError, match="no 2-D change"): + fit_rank_one(model, null) + + +def test_agreement_separates_direction_from_neuron_overlap(): + model = ToyModel() + reference = _reference() + same_direction_other_neuron = WeightEdit([TensorEdit.rank_one( + _KEY, np.eye(D)[WRONG_TOKEN], 6.0, cols=[DEFECT_NEURON, 0])]) + row = edit_agreement(model, same_direction_other_neuron, reference)["tensors"][0] + assert row["direction_cosine"] > 0.99 + assert row["n_shared"] == 1 and row["jaccard"] == 0.5 + assert row["expected_shared_by_chance"] == pytest.approx(2 * 1 / D) + + tilted = np.eye(D)[WRONG_TOKEN] + np.eye(D)[2] # 45 degrees off the reference + other_direction = WeightEdit([TensorEdit.rank_one(_KEY, tilted, 6.0, cols=[DEFECT_NEURON])]) + row = edit_agreement(model, other_direction, reference)["tensors"][0] + assert row["n_shared"] == 1 and row["jaccard"] == 1.0 + assert row["direction_cosine"] == pytest.approx(2 ** -0.5, abs=1e-3) + + +def test_agreement_lists_tensors_only_one_edit_touches(): + model = ToyModel() + elsewhere = WeightEdit([TensorEdit.rank_one("layers.0.mlp.down_proj.weight", + np.eye(D)[1], 1.0, cols=[1])]) + report = edit_agreement(model, elsewhere, _reference()) + assert report["tensors"] == [] + assert report["only_in_edit"] == ["layers.0.mlp.down_proj.weight"] + assert report["only_in_reference"] == [_KEY] diff --git a/tests/test_wsp/test_direction.py b/tests/test_wsp/test_direction.py new file mode 100644 index 00000000..73992506 --- /dev/null +++ b/tests/test_wsp/test_direction.py @@ -0,0 +1,183 @@ +"""Deriving an edit from a FAIL/PASS contrast, on a model with a known planted defect.""" + +from __future__ import annotations + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.core.capability import Capability # noqa: E402 +from evalrx.core.case import Label # noqa: E402 +from evalrx.wsp import ( # noqa: E402 + applied, + build_control, + collect_write_stats, + contrast_edit, + edit_structure, + output_shift, + write_sites, +) +from tests.toy_wsp import ( # noqa: E402 + DEFECT_LAYER, + DEFECT_NEURON, + N_LAYERS, + WRONG_TOKEN, + D, + ToyModel, + echo_batch, + exact_score, + snapshot, + unchanged, +) + + +def _groups(batch): + return ([c for c in batch if c.label == Label.FAIL], + [c for c in batch if c.label == Label.PASS]) + + +def _accuracy(model, batch) -> float: + return sum(exact_score(c, model.generate(c.inputs)) for c in batch) / len(batch) + + +def test_write_sites_finds_one_projection_per_layer(): + model = ToyModel() + sites = write_sites(model) + assert [s.layer for s in sites] == list(range(N_LAYERS)) + params = dict(model.module.named_parameters()) + for site in sites: + assert site.name == f"layers.{site.layer}.mlp.down_proj.weight" + assert params[site.name] is site.module.weight + + +def test_write_stats_have_the_layer_shapes_and_remove_their_hooks(): + model = ToyModel() + fails, _ = _groups(echo_batch()) + stats = collect_write_stats(model, fails) + assert sorted(stats) == list(range(N_LAYERS)) + for row in stats.values(): + assert row["in"].shape == (D,) and row["out"].shape == (D,) and row["n"] == len(fails) + for site in write_sites(model): + assert not site.module._forward_hooks + only = collect_write_stats(model, fails, layers=[DEFECT_LAYER]) + assert list(only) == [DEFECT_LAYER] + + +def test_the_planted_neuron_fires_on_failing_cases_only(): + model = ToyModel() + fails, passes = _groups(echo_batch()) + on = collect_write_stats(model, fails)[DEFECT_LAYER]["in"][DEFECT_NEURON] + off = collect_write_stats(model, passes)[DEFECT_LAYER]["in"][DEFECT_NEURON] + assert float(on) > 1.0 and abs(float(off)) < 0.5 + + +def test_contrast_edit_recovers_the_planted_layer_neuron_and_direction(): + model = ToyModel() + fails, passes = _groups(echo_batch()) + edit = contrast_edit(model, fails, passes, n_layers=1, n_columns=1) + assert len(edit) == 1 + tensor = edit.tensors[0] + assert tensor.name == f"layers.{DEFECT_LAYER}.mlp.down_proj.weight" + assert tensor.cols.tolist() == [DEFECT_NEURON] + expected = np.zeros(D) + expected[WRONG_TOKEN] = 1.0 + assert np.allclose(tensor.u, expected, atol=0.05) + assert edit.metadata["n_fail"] == len(fails) and edit.metadata["n_pass"] == len(passes) + + +def test_the_edit_repairs_unseen_cases_and_breaks_none(): + model = ToyModel() + pool, held_out = echo_batch(4, 6), echo_batch(6, 8) + snap = snapshot(model) + assert _accuracy(model, held_out) == pytest.approx(8 / 14) + edit = contrast_edit(model, *_groups(pool), n_layers=1, n_columns=1) + with applied(model, edit): + assert _accuracy(model, held_out) == 1.0 + assert unchanged(model, snap) + assert _accuracy(model, held_out) == pytest.approx(8 / 14) + + +def test_a_matched_random_direction_does_not_repair(): + model = ToyModel() + pool, held_out = echo_batch(4, 6), echo_batch(6, 8) + edit = contrast_edit(model, *_groups(pool), n_layers=1, n_columns=1) + for seed in range(5): + with applied(model, build_control(model, edit, "random_direction", seed=seed)): + assert _accuracy(model, held_out) < 1.0 + + +def test_a_wider_edit_still_repairs_without_side_effects(): + model = ToyModel() + pool, held_out = echo_batch(4, 6), echo_batch(6, 8) + edit = contrast_edit(model, *_groups(pool), n_layers=2, n_columns=4) + assert len(edit) == 2 + for tensor in edit.tensors: + assert 1 <= len(tensor.cols) <= 4 + with applied(model, edit): + assert _accuracy(model, held_out) == 1.0 + + +def test_structure_of_the_derived_edit_reads_back_as_rank_one(): + model = ToyModel() + edit = contrast_edit(model, *_groups(echo_batch()), n_layers=1, n_columns=1, alpha=1.0) + row = edit_structure(model, edit)["tensors"][0] + assert row["changed_columns"] == 1 + assert row["rank_one_residual"] < 1e-5 + assert row["alpha_estimate"] == pytest.approx(1.0, abs=1e-3) + + +def test_explicit_layers_are_respected_and_checked(): + model = ToyModel() + fails, passes = _groups(echo_batch()) + edit = contrast_edit(model, fails, passes, layers=[DEFECT_LAYER], n_columns=2) + assert edit.tensor_names == [f"layers.{DEFECT_LAYER}.mlp.down_proj.weight"] + with pytest.raises(ValueError, match="no MLP output projection"): + contrast_edit(model, fails, passes, layers=[99]) + + +def test_both_groups_are_required(): + model = ToyModel() + fails, passes = _groups(echo_batch()) + with pytest.raises(ValueError, match="FAIL case and one readable PASS"): + contrast_edit(model, [], passes) + with pytest.raises(ValueError, match="FAIL case and one readable PASS"): + contrast_edit(model, fails, []) + + +def test_identical_groups_give_no_edit(): + model = ToyModel() + _, passes = _groups(echo_batch()) + with pytest.raises(ValueError, match="do not differ"): + contrast_edit(model, passes[:1], passes[:1]) + + +def test_output_shift_changes_the_forward_pass_only_inside_the_block(): + model = ToyModel() + case = list(echo_batch())[4] + key = f"layers.{DEFECT_LAYER}.mlp.down_proj.weight" + clean = model.forward(case.inputs, capture={Capability.LOGITS}).logits[-1] + bump = torch.zeros(D) + bump[WRONG_TOKEN] = 5.0 + with output_shift(model, {key: bump}): + shifted = model.forward(case.inputs, capture={Capability.LOGITS}).logits[-1] + after = model.forward(case.inputs, capture={Capability.LOGITS}).logits[-1] + assert float(shifted[WRONG_TOKEN] - clean[WRONG_TOKEN]) > 4.0 + assert torch.equal(after, clean) + with pytest.raises(KeyError): + with output_shift(model, {"layers.9.mlp.down_proj.weight": bump}): + pass + + +def test_write_sites_on_a_real_architecture(): + pytest.importorskip("transformers") + pytest.importorskip("PIL") + from tests.tiny_whitebox import TinyVLM + + model = TinyVLM("qwen2_5_vl") + sites = write_sites(model) + assert len(sites) == 6 + params = dict(model._loaded[0].named_parameters()) + for site in sites: + assert site.name.endswith(f"layers.{site.layer}.mlp.down_proj.weight") + assert params[site.name].shape == (32, 64) diff --git a/tests/test_wsp/test_edit.py b/tests/test_wsp/test_edit.py new file mode 100644 index 00000000..a9d16e08 --- /dev/null +++ b/tests/test_wsp/test_edit.py @@ -0,0 +1,392 @@ +"""Weight edits: the arithmetic of each form, and the apply/restore contract.""" + +from __future__ import annotations + +import hashlib +import json + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.wsp import ( # noqa: E402 + TensorEdit, + WeightEdit, + applied, + bf16_bits_to_f32, + resolve_parameters, + tensor_sha256, +) + + +class _Block(torch.nn.Module): + def __init__(self, d: int = 6, h: int = 10) -> None: + super().__init__() + self.up_proj = torch.nn.Linear(d, h, bias=False) + self.down_proj = torch.nn.Linear(h, d, bias=False) + + +class _Net(torch.nn.Module): + def __init__(self, n: int = 3) -> None: + super().__init__() + torch.manual_seed(0) + self.layers = torch.nn.ModuleList([_Block() for _ in range(n)]) + + +def _snapshot(net): + return {k: v.detach().clone() for k, v in net.state_dict().items()} + + +def _same(net, snap) -> bool: + return all(torch.equal(v, snap[k]) for k, v in net.state_dict().items()) + + +# ---------------------------------------------------------------------- +# Arithmetic +# ---------------------------------------------------------------------- + +def test_rank_one_alpha_one_removes_the_projection_on_chosen_columns(): + torch.manual_seed(1) + w = torch.randn(6, 10) + u = torch.randn(6) + cols = [1, 4, 7] + out = TensorEdit.rank_one("w", u, alpha=1.0, cols=cols).apply_to(w) + unit = u / u.norm() + assert torch.allclose(unit @ out[:, cols], torch.zeros(3), atol=1e-6) + untouched = [c for c in range(10) if c not in cols] + assert torch.equal(out[:, untouched], w[:, untouched]) + + +def test_rank_one_alpha_two_is_a_reflection(): + torch.manual_seed(2) + w = torch.randn(6, 10) + u = torch.randn(6) + out = TensorEdit.rank_one("w", u, alpha=2.0).apply_to(w) + unit = u / u.norm() + assert torch.allclose(out.norm(dim=0), w.norm(dim=0), atol=1e-5) + assert torch.allclose(unit @ out, -(unit @ w), atol=1e-5) + + +def test_rank_one_alpha_six_flips_and_scales_by_five(): + torch.manual_seed(3) + w = torch.randn(6, 10) + u = torch.randn(6) + out = TensorEdit.rank_one("w", u, alpha=6.0).apply_to(w) + unit = u / u.norm() + assert torch.allclose(unit @ out, -5.0 * (unit @ w), atol=1e-4) + + +def test_direction_is_normalised_and_must_be_usable(): + edit = TensorEdit.rank_one("w", [3.0, 4.0], alpha=1.0) + assert np.allclose(edit.u, [0.6, 0.8]) + with pytest.raises(ValueError): + TensorEdit.rank_one("w", [0.0, 0.0], alpha=1.0) + with pytest.raises(ValueError): + TensorEdit.rank_one("w", [float("nan"), 1.0], alpha=1.0) + + +def test_low_rank_dense_and_sparse_components_compose_in_order(): + w = torch.zeros(2, 3) + edit = TensorEdit( + name="w", form="low_rank", + a=np.array([[1.0], [2.0]]), b=np.array([[1.0, 0.0, 1.0]]), # [[1,0,1],[2,0,2]] + delta=np.full((2, 3), 0.5), # +0.5 everywhere + index=[0, 5], values=[9.0, -9.0], # overwrite last + ) + out = edit.apply_to(w) + assert out.tolist() == [[9.0, 0.5, 1.5], [2.5, 0.5, -9.0]] + + +def test_apply_to_does_not_touch_its_input_and_keeps_dtype(): + w = torch.randn(4, 5, dtype=torch.float64) + before = w.clone() + out = TensorEdit.rank_one("w", torch.randn(4), 3.0).apply_to(w) + assert torch.equal(w, before) + assert out.dtype == torch.float64 and out is not w + + +@pytest.mark.parametrize("edit,message", [ + (TensorEdit.rank_one("w", [1.0, 0.0, 0.0], 1.0), "rows"), + (TensorEdit.rank_one("w", [1.0, 0.0, 0.0, 0.0], 1.0, cols=[99]), "column index"), + (TensorEdit(name="w", index=[999], values=[1.0]), "flat index"), + (TensorEdit(name="w", delta=np.zeros((4, 5)), shape=(9, 9)), "shape"), +]) +def test_edits_that_do_not_fit_are_refused(edit, message): + with pytest.raises(ValueError, match=message): + edit.apply_to(torch.zeros(4, 5)) + + +def test_malformed_components_are_refused_at_construction(): + with pytest.raises(ValueError): + TensorEdit(name="w", index=[0, 1], values=[1.0]) + with pytest.raises(ValueError): + TensorEdit(name="w", index=[0]) + with pytest.raises(ValueError): + TensorEdit(name="w", a=np.zeros((2, 1))) + with pytest.raises(ValueError): + WeightEdit([TensorEdit(name="w", delta=np.zeros(1)), + TensorEdit(name="w", delta=np.zeros(1))]) + + +def test_scaling_is_exact_for_rules_and_refused_for_overwrites(): + torch.manual_seed(4) + w = torch.randn(6, 10) + u = torch.randn(6) + half = TensorEdit.rank_one("w", u, 6.0).scaled(0.5).apply_to(w) + assert torch.allclose(half, TensorEdit.rank_one("w", u, 3.0).apply_to(w)) + dense = TensorEdit(name="w", delta=np.ones((6, 10))).scaled(2.0).apply_to(w) + assert torch.allclose(dense, w + 2.0) + with pytest.raises(ValueError, match="cannot be scaled"): + TensorEdit(name="w", index=[0], values=[1.0]).scaled(2.0) + + +# ---------------------------------------------------------------------- +# Resolving names +# ---------------------------------------------------------------------- + +def test_names_resolve_exactly_or_by_unique_suffix(): + net = _Net() + exact = WeightEdit([TensorEdit(name="layers.1.down_proj.weight", delta=np.zeros((6, 10)))]) + assert resolve_parameters(net, exact)[0][0] == "layers.1.down_proj.weight" + # An edit written against a differently rooted model still finds its tensor. + foreign = WeightEdit([TensorEdit(name="model.language_model.layers.2.down_proj.weight", + delta=np.zeros((6, 10)))]) + assert resolve_parameters(net, foreign)[0][0] == "layers.2.down_proj.weight" + + +def test_a_wrong_layer_index_never_lands_on_another_layer(): + net = _Net(n=1) # only layers.0 exists + edit = WeightEdit([TensorEdit(name="model.layers.9.down_proj.weight", + delta=np.zeros((6, 10)))]) + with pytest.raises(KeyError, match="no parameter"): + resolve_parameters(net, edit) + + +def test_names_without_an_index_have_no_droppable_prefix(): + net = _Net() + with pytest.raises(KeyError, match="no parameter"): + resolve_parameters(net, WeightEdit([TensorEdit(name="x.down_proj.weight", + delta=np.zeros((6, 10)))])) + + +def test_a_name_that_fits_two_towers_is_ambiguous(): + class Two(torch.nn.Module): + def __init__(self) -> None: + super().__init__() + self.vision = _Net(n=1) + self.text = _Net(n=1) + + edit = WeightEdit([TensorEdit(name="model.layers.0.down_proj.weight", + delta=np.zeros((6, 10)))]) + with pytest.raises(KeyError, match="ambiguous"): + resolve_parameters(Two(), edit) + + +# ---------------------------------------------------------------------- +# apply / restore +# ---------------------------------------------------------------------- + +def _edit(alpha: float = 6.0) -> WeightEdit: + torch.manual_seed(5) + return WeightEdit([ + TensorEdit.rank_one("layers.0.down_proj.weight", torch.randn(6), alpha, cols=[0, 3]), + TensorEdit.rank_one("layers.2.down_proj.weight", torch.randn(6), alpha, cols=[1]), + ], name="t") + + +def test_applied_writes_then_restores_exactly(): + net = _Net() + snap = _snapshot(net) + with applied(net, _edit()) as report: + assert not _same(net, snap) + rows = {r["name"]: r for r in report.rows} + assert rows["layers.0.down_proj.weight"]["changed_columns"] == 2 + assert rows["layers.2.down_proj.weight"]["changed_columns"] == 1 + assert report.relative_change > 0 + assert _same(net, snap) + + +def test_applied_restores_when_the_block_raises(): + net = _Net() + snap = _snapshot(net) + with pytest.raises(RuntimeError, match="boom"): + with applied(net, _edit()): + raise RuntimeError("boom") + assert _same(net, snap) + + +def test_nothing_is_written_when_any_tensor_fails(): + net = _Net() + snap = _snapshot(net) + bad = WeightEdit([ + TensorEdit.rank_one("layers.0.down_proj.weight", torch.randn(6), 6.0), + TensorEdit.rank_one("layers.1.down_proj.weight", torch.randn(7), 6.0), # wrong size + ]) + entered = False + with pytest.raises(ValueError): + with applied(net, bad): + entered = True + assert entered is False + assert _same(net, snap) + + +def test_size_limit_refuses_a_large_edit(): + net = _Net() + snap = _snapshot(net) + with pytest.raises(ValueError, match="exceeds the limit"): + with applied(net, _edit(alpha=50.0), max_relative_change=0.01): + pass + assert _same(net, snap) + with applied(net, _edit(alpha=1e-6), max_relative_change=0.01): + pass + + +def test_non_finite_results_are_refused(): + net = _Net() + snap = _snapshot(net) + edit = WeightEdit([TensorEdit(name="layers.0.down_proj.weight", + delta=np.full((6, 10), np.inf, dtype=np.float32))]) + with pytest.raises(ValueError, match="non-finite"): + with applied(net, edit): + pass + assert _same(net, snap) + + +def test_verify_base_checks_the_recorded_hash(): + net = _Net() + param = net.layers[0].down_proj.weight + good = WeightEdit([TensorEdit(name="layers.0.down_proj.weight", delta=np.ones((6, 10)), + base_sha256=tensor_sha256(param))]) + with applied(net, good, verify_base=True): + pass + stale = WeightEdit([TensorEdit(name="layers.0.down_proj.weight", delta=np.ones((6, 10)), + base_sha256="0" * 64)]) + with pytest.raises(ValueError, match="differs from the one the edit was made on"): + with applied(net, stale, verify_base=True): + pass + unhashed = WeightEdit([TensorEdit(name="layers.0.down_proj.weight", delta=np.ones((6, 10)))]) + with pytest.raises(ValueError, match="recorded base hash"): + with applied(net, unhashed, verify_base=True): + pass + + +# ---------------------------------------------------------------------- +# Building from weights, persistence +# ---------------------------------------------------------------------- + +def test_from_state_dicts_reproduces_the_edited_model(): + net = _Net() + base = _snapshot(net) + with applied(net, _edit()): + edited = _snapshot(net) + diff = WeightEdit.from_state_dicts(base, edited) + assert sorted(diff.tensor_names) == ["layers.0.down_proj.weight", "layers.2.down_proj.weight"] + with applied(net, diff): + assert all(torch.allclose(v, edited[k], atol=1e-6) for k, v in net.state_dict().items()) + + +def test_as_delta_is_equivalent_on_the_weights_it_was_built_from(): + net = _Net() + edit = _edit() + with applied(net, edit): + target = _snapshot(net) + with applied(net, edit.as_delta(net)): + assert all(torch.allclose(v, target[k], atol=1e-6) for k, v in net.state_dict().items()) + + +def test_save_and_load_round_trip(tmp_path): + edit = _edit() + edit.metadata["note"] = "round trip" + loaded = WeightEdit.load(edit.save(str(tmp_path / "e"))) + assert loaded.name == "t" and loaded.metadata["note"] == "round trip" + assert loaded.tensor_names == edit.tensor_names + for a, b in zip(edit.tensors, loaded.tensors): + assert np.array_equal(a.u, b.u) and np.array_equal(a.cols, b.cols) + assert a.alpha == b.alpha and a.form == b.form + net = _Net() + with applied(net, edit): + target = _snapshot(net) + with applied(net, loaded): + assert _same(net, target) + + +def test_describe_is_json_serialisable_and_carries_no_arrays(): + text = json.dumps(_edit().describe()) + assert "alpha" in text and "n_cols" in text + + +# ---------------------------------------------------------------------- +# The published edit layout +# ---------------------------------------------------------------------- + +def _bf16_bits(x32: np.ndarray) -> np.ndarray: + """float32 -> bfloat16 bits, round to nearest even (reference implementation).""" + u = np.ascontiguousarray(x32, dtype=np.float32).view(np.uint32).astype(np.uint64) + u = u + 0x7FFF + ((u >> 16) & 1) + return (u >> 16).astype(np.uint16) + + +def _sha(bits: np.ndarray) -> str: + return hashlib.sha256(np.ascontiguousarray(bits).tobytes()).hexdigest() + + +def _write_astra_dir(tmp_path, base_bits, form): + """Write an edit directory in the published layout; return (dir, expected bits).""" + rng = np.random.default_rng(0) + rows, cols = base_bits.shape + arrays = {} + expected = base_bits.copy().reshape(-1) + if form == "lowrank": + a = rng.normal(size=(rows, 2)).astype(np.float32) + b = rng.normal(size=(2, cols)).astype(np.float32) + w = bf16_bits_to_f32(base_bits).astype(np.float64) + for k in range(2): + w = w + np.outer(a[:, k], b[k]) + expected = _bf16_bits(w.astype(np.float32)).reshape(-1) + arrays["t0_a"], arrays["t0_b"] = a, b + idx = np.array([0, 7, 13], dtype=np.int32) + val = _bf16_bits(np.array([0.25, -1.5, 3.0], dtype=np.float32)) + expected[idx] = val + arrays["t0_idx"], arrays["t0_val"] = idx, val + directory = tmp_path / form / "edit" + directory.mkdir(parents=True) + np.savez(directory / "edit.npz", **arrays) + manifest = {"tensors": [{ + "name": "model.language_model.layers.0.down_proj.weight", + "shape": [rows, cols], "dtype": "bfloat16", "form": form, + "base_sha256": _sha(base_bits), + "edited_sha256": _sha(expected.reshape(rows, cols)), + }], "changed_entries_total": 3} + (directory / "manifest.json").write_text(json.dumps(manifest)) + return str(directory), expected.reshape(rows, cols) + + +@pytest.mark.parametrize("form", ["sparse", "lowrank"]) +def test_published_layout_rebuilds_the_checkpoint_bit_for_bit(tmp_path, form): + net = _Net().to(torch.bfloat16) + param = net.layers[0].down_proj.weight + base_bits = param.detach().view(torch.int16).numpy().astype(np.uint16) + directory, expected_bits = _write_astra_dir(tmp_path, base_bits, form) + + edit = WeightEdit.from_astra(directory) + assert edit.tensors[0].form == ("low_rank" if form == "lowrank" else "sparse_set") + assert edit.metadata["source"] == "astra" + with applied(net, edit, verify_base=True): + assert tensor_sha256(param) == _sha(expected_bits) + assert tensor_sha256(param) == _sha(base_bits) + # WeightEdit.load recognises the layout without being told. + assert WeightEdit.load(directory).tensors[0].base_sha256 == _sha(base_bits) + + +def test_published_edit_is_refused_on_a_different_base(tmp_path): + net = _Net().to(torch.bfloat16) + base_bits = net.layers[0].down_proj.weight.detach().view(torch.int16).numpy().astype(np.uint16) + directory, _ = _write_astra_dir(tmp_path, base_bits, "sparse") + other = _Net() + with torch.no_grad(): + other.layers[0].down_proj.weight.add_(1.0) + other = other.to(torch.bfloat16) + with pytest.raises(ValueError, match="differs from the one the edit was made on"): + with applied(other, WeightEdit.from_astra(directory), verify_base=True): + pass diff --git a/tests/test_wsp/test_structure_controls.py b/tests/test_wsp/test_structure_controls.py new file mode 100644 index 00000000..4eed8efa --- /dev/null +++ b/tests/test_wsp/test_structure_controls.py @@ -0,0 +1,169 @@ +"""Reading an edit from the weights, and building the controls it is compared with.""" + +from __future__ import annotations + +import numpy as np +import pytest + +torch = pytest.importorskip("torch") + +from evalrx.wsp import ( # noqa: E402 + CONTROL_KINDS, + TensorEdit, + WeightEdit, + applied, + build_control, + delta_structure, + edit_structure, +) + + +class _Net(torch.nn.Module): + def __init__(self, rows: int = 12, cols: int = 40) -> None: + super().__init__() + torch.manual_seed(0) + self.a = torch.nn.Linear(cols, rows, bias=False) + self.b = torch.nn.Linear(cols, rows, bias=False) + + +def _rank_one(alpha: float = 6.0, cols=(2, 5, 9, 17)): + torch.manual_seed(1) + w0 = torch.randn(12, 40) + u = torch.randn(12) + w1 = TensorEdit.rank_one("w", u, alpha, cols=list(cols)).apply_to(w0) + return w0, w1, u / u.norm() + + +def test_structure_recovers_a_rank_one_rule(): + w0, w1, _ = _rank_one(alpha=6.0) + s = delta_structure(w0, w1) + assert s["changed_columns"] == 4 and s["changed_rows"] == 12 + assert s["rank_one_residual"] < 1e-5 + assert s["effective_rank"] == pytest.approx(1.0, abs=1e-4) + assert s["alpha_estimate"] == pytest.approx(6.0, abs=1e-3) + assert s["alpha_spread"] < 1e-3 + assert 0 < s["relative_change"] < 5 + + +@pytest.mark.parametrize("alpha", [1.0, 2.0, 3.5]) +def test_alpha_is_read_off_for_other_strengths(alpha): + w0, w1, _ = _rank_one(alpha=alpha) + assert delta_structure(w0, w1)["alpha_estimate"] == pytest.approx(alpha, abs=1e-3) + + +def test_structure_flags_a_change_that_is_not_rank_one(): + torch.manual_seed(2) + w0 = torch.randn(12, 40) + w1 = w0.clone() + w1[:, :6] += torch.randn(12, 6) + s = delta_structure(w0, w1) + assert s["rank_one_residual"] > 0.3 + assert s["effective_rank"] > 2.0 + assert len(s["singular_values"]) == 6 + + +def test_structure_of_no_change_and_of_a_vector(): + w = torch.randn(4, 4) + assert delta_structure(w, w.clone())["changed_entries"] == 0 + bias = delta_structure(torch.ones(5), torch.ones(5) * 2) + assert bias["changed_entries"] == 5 and "singular_values" not in bias + with pytest.raises(ValueError): + delta_structure(torch.ones(2, 2), torch.ones(3, 3)) + + +def test_selectivity_is_high_when_columns_were_chosen_by_projection(): + torch.manual_seed(3) + w0 = torch.randn(12, 40) + u = torch.randn(12) + unit = u / u.norm() + top = torch.topk((unit @ w0).abs(), 4).indices.tolist() + w1 = TensorEdit.rank_one("w", u, 6.0, cols=top).apply_to(w0) + assert delta_structure(w0, w1)["projection_selectivity"] > 1.5 + + +def test_edit_structure_reports_shared_directions_across_tensors(): + net = _Net() + u = torch.randn(12) + shared = WeightEdit([TensorEdit.rank_one("a.weight", u, 6.0, cols=[1, 2]), + TensorEdit.rank_one("b.weight", u, 6.0, cols=[3, 4])]) + report = edit_structure(net, shared) + assert len(report["tensors"]) == 2 + assert list(report["direction_cosines"].values())[0] == pytest.approx(1.0, abs=1e-5) + + +def _edit(cols=(2, 5, 9, 17)) -> WeightEdit: + torch.manual_seed(4) + return WeightEdit([TensorEdit.rank_one("a.weight", torch.randn(12), 6.0, cols=list(cols))], + name="e") + + +def _delta_of(net, edit): + before = net.a.weight.detach().clone() + with applied(net, edit): + return (net.a.weight.detach() - before).double() + + +def test_random_direction_keeps_columns_and_norms_but_not_the_direction(): + net = _Net() + edit = _edit() + d = _delta_of(net, edit) + c = _delta_of(net, build_control(net, edit, "random_direction", seed=0)) + assert torch.equal((c != 0).any(dim=0), (d != 0).any(dim=0)) + assert torch.allclose(c.norm(dim=0), d.norm(dim=0), atol=1e-5) + u_edit = torch.linalg.svd(d, full_matrices=False)[0][:, 0] + u_ctrl = torch.linalg.svd(c, full_matrices=False)[0][:, 0] + assert float((u_edit @ u_ctrl).abs()) < 0.95 + assert float(torch.linalg.svd(c, full_matrices=False)[1][1]) < 1e-6 # still rank one + + +def test_random_columns_applies_the_same_rule_elsewhere(): + net = _Net() + edit = _edit() + d = _delta_of(net, edit) + c = _delta_of(net, build_control(net, edit, "random_columns", seed=0)) + edit_cols = set(torch.nonzero((d != 0).any(dim=0)).flatten().tolist()) + ctrl_cols = set(torch.nonzero((c != 0).any(dim=0)).flatten().tolist()) + assert len(ctrl_cols) == len(edit_cols) and not (ctrl_cols & edit_cols) + # Same rule: the control is the alpha = 6 edit along the same u on those columns. + w0 = net.a.weight.detach().double() + u = torch.from_numpy(edit.tensors[0].u).double() + cols = sorted(ctrl_cols) + assert torch.allclose(c[:, cols], -6.0 * torch.outer(u, u @ w0[:, cols]), atol=1e-5) + + +def test_random_columns_moves_the_change_for_other_edit_forms(): + net = _Net() + delta = np.zeros((12, 40), dtype=np.float32) + delta[:, [0, 1]] = 1.0 + edit = WeightEdit([TensorEdit(name="a.weight", delta=delta)]) + c = _delta_of(net, build_control(net, edit, "random_columns", seed=3)) + cols = torch.nonzero((c != 0).any(dim=0)).flatten().tolist() + assert len(cols) == 2 and not ({0, 1} & set(cols)) + assert torch.allclose(c[:, cols], torch.ones(12, 2, dtype=torch.float64), atol=1e-6) + + +def test_controls_are_seeded(): + net = _Net() + edit = _edit() + for kind in CONTROL_KINDS: + a = _delta_of(net, build_control(net, edit, kind, seed=7)) + b = _delta_of(net, build_control(net, edit, kind, seed=7)) + c = _delta_of(net, build_control(net, edit, kind, seed=8)) + assert torch.equal(a, b) and not torch.equal(a, c) + + +def test_control_errors_are_explicit(): + net = _Net() + with pytest.raises(ValueError, match="unknown control"): + build_control(net, _edit(), "shuffle") + everything = _edit(cols=range(40)) + with pytest.raises(ValueError, match="no unchanged columns"): + build_control(net, everything, "random_columns") + + +def test_building_a_control_leaves_the_model_untouched(): + net = _Net() + before = {k: v.clone() for k, v in net.state_dict().items()} + for kind in CONTROL_KINDS: + build_control(net, _edit(), kind) + assert all(torch.equal(v, before[k]) for k, v in net.state_dict().items()) diff --git a/tests/toy_wsp.py b/tests/toy_wsp.py new file mode 100644 index 00000000..e833f667 --- /dev/null +++ b/tests/toy_wsp.py @@ -0,0 +1,155 @@ +"""A hand-built model with one planted rank-one defect, behind the ``Model`` contract. + +The weight-edit code has to be tested where the right answer is known. This +model is small enough to reason about by hand: + +* tokens ``0..7`` embed as scaled basis vectors of an 8-dimensional stream; +* each decoder layer adds ``down_proj(relu(up_proj(h)))`` to the stream, with + no attention, so the output at a position depends on that position's token; +* the unembedding is the identity, so the predicted token is the largest + stream coordinate. The clean model echoes its last input token. + +The defect lives in layer :data:`DEFECT_LAYER`: neuron :data:`DEFECT_NEURON` +fires on the trigger tokens :data:`TRIGGERS` and writes strongly along +coordinate :data:`WRONG_TOKEN`. On a trigger token the model therefore answers +:data:`WRONG_TOKEN` instead of echoing; on every other token it is correct. + +Removing that neuron's write along coordinate 7 (a rank-one edit with +``alpha = 1`` on one column) repairs the model, and nothing else in the model +is needed to explain the failure. Small random weights everywhere else keep +the contrast from being trivially exact. +""" + +from __future__ import annotations + +from typing import Any + +import torch + +from evalrx.core.capability import Capability +from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label +from evalrx.core.model import Model, Trace + +D = 8 +VOCAB = 8 +N_LAYERS = 3 +DEFECT_LAYER = 1 +DEFECT_NEURON = 5 +TRIGGERS = (0, 1) +WRONG_TOKEN = 7 +SAFE_TOKENS = (2, 3, 4, 5, 6) + + +class _Mlp(torch.nn.Module): + def __init__(self) -> None: + super().__init__() + self.up_proj = torch.nn.Linear(D, D, bias=False) + self.down_proj = torch.nn.Linear(D, D, bias=False) + + def forward(self, h: torch.Tensor) -> torch.Tensor: + return self.down_proj(torch.relu(self.up_proj(h))) + + +class ToyDecoderLayer(torch.nn.Module): + def __init__(self) -> None: + super().__init__() + self.mlp = _Mlp() + + def forward(self, h: torch.Tensor) -> torch.Tensor: + return h + self.mlp(h) + + +class ToyLM(torch.nn.Module): + def __init__(self, noise: float = 0.02, seed: int = 0) -> None: + super().__init__() + torch.manual_seed(seed) + self.embed = torch.nn.Embedding(VOCAB, D) + self.layers = torch.nn.ModuleList([ToyDecoderLayer() for _ in range(N_LAYERS)]) + with torch.no_grad(): + self.embed.weight.copy_(2.0 * torch.eye(D)) + for layer in self.layers: + layer.mlp.up_proj.weight.mul_(0).add_(noise * torch.randn(D, D)) + layer.mlp.down_proj.weight.mul_(0).add_(noise * torch.randn(D, D)) + mlp = self.layers[DEFECT_LAYER].mlp + mlp.up_proj.weight[DEFECT_NEURON].zero_() + for token in TRIGGERS: + mlp.up_proj.weight[DEFECT_NEURON, token] = 1.0 + mlp.down_proj.weight[:, DEFECT_NEURON].zero_() + mlp.down_proj.weight[WRONG_TOKEN, DEFECT_NEURON] = 3.0 + + def forward(self, input_ids: torch.Tensor) -> torch.Tensor: + h = self.embed(input_ids) # [1, seq, D] + for layer in self.layers: + h = layer(h) + return h # identity unembedding: logits are the stream + + +class _DigitTokenizer: + """``"3 0 5"`` -> ids ``[3, 0, 5]``; the gold answer of a case is a digit.""" + + def __call__(self, text: str, add_special_tokens: bool = False) -> dict: + return {"input_ids": [int(w) % VOCAB for w in str(text).split()]} + + +class ToyModel(Model): + """:class:`ToyLM` under the evalrx ``Model`` contract (white-box, text only).""" + + def __init__(self, **kwargs: Any) -> None: + self.capabilities = frozenset({Capability.GENERATE, Capability.LOGITS}) + self.modalities = frozenset({"text"}) + self._hf = (ToyLM(**kwargs).eval(), _DigitTokenizer()) + self.n_generate = 0 + + @property + def _loaded(self): + return self._hf + + @property + def module(self) -> ToyLM: + return self._hf[0] + + def _ids(self, inputs: Any) -> list[int]: + prompt = inputs.prompt if isinstance(inputs, Inputs) else str(inputs) + return self._hf[1](prompt)["input_ids"] + + def forward(self, inputs: Any, capture: "set[Capability]", spec: Any = None) -> Trace: + ids = self._ids(inputs) + with torch.no_grad(): + logits = self.module(torch.tensor([ids]))[0] + return Trace(tokens=[str(i) for i in ids], token_ids=ids, + provided={Capability.LOGITS}, logits=logits) + + def generate(self, inputs: Any, **kwargs: Any) -> str: + self.n_generate += 1 + trace = self.forward(inputs, capture={Capability.LOGITS}) + return str(int(trace.logits[-1].argmax())) + + +def echo_case(last: int, context: tuple[int, ...] = (4, 2)) -> FailureCase: + """A case whose correct answer is its last token; FAIL iff that token is a trigger.""" + prompt = " ".join(str(t) for t in (*context, last)) + return FailureCase(inputs=Inputs(prompt=prompt), expected=str(last), + label=Label.FAIL if last in TRIGGERS else Label.PASS) + + +def echo_batch(n_fail: int = 4, n_pass: int = 6) -> CaseBatch: + """*n_fail* trigger cases and *n_pass* safe ones, each a distinct case object.""" + fails = [echo_case(TRIGGERS[i % len(TRIGGERS)], context=(3 + i % 4, 2 + i % 5)) + for i in range(n_fail)] + passes = [echo_case(SAFE_TOKENS[i % len(SAFE_TOKENS)], context=(2 + i % 5, 6 - i % 4)) + for i in range(n_pass)] + return CaseBatch(fails + passes) + + +def exact_score(case: FailureCase, output: str) -> bool: + return str(output).strip() == str(case.expected).strip() + + +def snapshot(model: Any) -> dict: + module = model._loaded[0] if hasattr(model, "_loaded") else model + return {k: v.detach().clone() for k, v in module.state_dict().items()} + + +def unchanged(model: Any, snap: dict) -> bool: + module = model._loaded[0] if hasattr(model, "_loaded") else model + return all(torch.equal(v, snap[k]) for k, v in module.state_dict().items()) diff --git a/tools/audit_astra_edit_qwen35.py b/tools/audit_astra_edit_qwen35.py new file mode 100644 index 00000000..b70e7e89 --- /dev/null +++ b/tools/audit_astra_edit_qwen35.py @@ -0,0 +1,263 @@ +"""Follow a published weight edit along the base model's own looping generations. + +Runs ``weight_edit_continuation_audit`` with the p50c edit of "Sparks of +Weight-Space Programming" on Qwen3.5-0.8B, on the texts EditLens used: GSM8K +generations of the UNEDITED model that never left the thinking block (no +````) and contain repetition loops. + +For every position of those generations the script measures, teacher-forced, +how much less likely the recorded next token becomes (dNLL) and how far the +next-token distribution moves (KL), under + + * the edit, + * a random direction of the same size on the same neurons, + * the same rule on other neurons, + * a constant activation shift with the weights untouched, + +separately for loop positions and productive positions. + +EditLens's own teacher-forced table (results/02_loops/steering/ +q4_simpler_intervention.md, 6 texts) has the edit raise the NLL of the repeated +continuation by +0.093 inside loops and by +0.053 at productive positions: a +larger effect in loops, with measurable collateral outside them. The script +prints what it measures; it does not assume that outcome. + +Requirements: this branch of evalrx installed, a GPU with ~12 GB free (two +``[4096, vocab]`` logit tensors plus float32 work chunks), and a clone of +https://github.com/assassin808/edit-lens for the edit and the recorded texts. + + python tools/audit_astra_edit_qwen35.py --edit-lens ~/Github/edit-lens + +Run ``tools/check_astra_edit_qwen35.py`` first: this script refuses to continue +if the loaded weights are not the ones the edit was made on. +""" + +from __future__ import annotations + +import argparse +import glob +import gzip +import json +import os +import sys +import time + +BASE_REPO = "Qwen/Qwen3.5-0.8B" +BASE_REVISION = "2fc06364715b967f1860aea9cf38778875588b17" +# The suffix EditLens appends to each GSM8K question (scripts/q4_tf_interventions.py). +QUESTION_SUFFIX = " Please reason step by step, and put your final answer within \\boxed{}." + + +def load_model(model: str, revision: "str | None", device: str): + import torch + from transformers import AutoModelForCausalLM, AutoTokenizer + + kwargs = {"revision": revision} if revision else {} + tokenizer = AutoTokenizer.from_pretrained(model, **kwargs) + module = AutoModelForCausalLM.from_pretrained(model, dtype=torch.bfloat16, **kwargs) + return module.to(device).eval(), tokenizer + + +def as_evalrx_model(module, tokenizer): + """An evalrx model handle that tokenises the prompt verbatim (no template added).""" + import evalrx + + try: + handle = evalrx.wrap(module, tokenizer) + if getattr(handle.runtime, "apply_chat_template", False): + raise RuntimeError("the handle would re-apply the chat template") + print(f"model handle: evalrx.wrap -> {type(handle).__name__}") + return handle + except Exception as exc: # noqa: BLE001 - fall back to a minimal handle, and say so + print(f"evalrx.wrap not used ({type(exc).__name__}: {exc}); using a minimal handle") + + import torch + + from evalrx.core.capability import Capability + from evalrx.core.model import Model, Trace + + class Handle(Model): + def __init__(self) -> None: + self.capabilities = frozenset({Capability.GENERATE, Capability.LOGITS}) + self.modalities = frozenset({"text"}) + self._hf = (module, tokenizer) + + @property + def _loaded(self): + return self._hf + + def generate(self, inputs, **kwargs) -> str: # not used by the audit + raise NotImplementedError + + def forward(self, inputs, capture, spec=None) -> Trace: + ids = tokenizer(getattr(inputs, "prompt", str(inputs)))["input_ids"] + device = next(module.parameters()).device + with torch.no_grad(): + logits = module(input_ids=torch.tensor([ids], device=device)).logits[0] + return Trace(tokens=[str(i) for i in ids], token_ids=list(ids), + provided={Capability.LOGITS}, logits=logits) + + return Handle() + + +def looping_cases(edit_lens: str, tokenizer, *, n_texts: int, max_tokens: int, + repeat_ngram: int, min_loop_frac: float, offset: int = 0): + """Unfinished GSM8K generations of the unedited model with enough loop positions. + + *offset* skips that many qualifying generations first, so two calls with + ``offset=0`` and ``offset=n`` return disjoint sets. + """ + from evalrx.analyzers.weights.continuation_audit import repetition_mask + from evalrx.core.case import CaseBatch, FailureCase, Inputs, Label + + questions = {} + with open(os.path.join(edit_lens, "data", "gsm8k_test.jsonl"), encoding="utf-8") as fh: + for line in fh: + row = json.loads(line) + questions[row["id"]] = row["question"] + pattern = os.path.join(edit_lens, "astra_logs", "exp8_math", "eval", "unedited", "parts", + "gsm8k_s*", "gsm8k_first_samples.jsonl.gz") + files = sorted(glob.glob(pattern)) + if not files: + raise SystemExit(f"no recorded generations found at {pattern}") + cases, seen, unfinished = [], 0, 0 + for path in files: + with gzip.open(path, "rt", encoding="utf-8") as fh: + for line in fh: + row = json.loads(line) + seen += 1 + if "" in row["text"] or row["id"] not in questions: + continue + unfinished += 1 + prompt = tokenizer.apply_chat_template( + [{"role": "user", "content": questions[row["id"]] + QUESTION_SUFFIX}], + tokenize=False, add_generation_prompt=True, enable_thinking=True) + prompt_ids = tokenizer(prompt)["input_ids"] + text_ids = tokenizer(row["text"], add_special_tokens=False)["input_ids"] + ids = (list(prompt_ids) + list(text_ids))[:max_tokens] + mask = repetition_mask(ids, repeat_ngram)[len(prompt_ids):] + if not mask or sum(mask) / len(mask) <= min_loop_frac: + continue + cases.append(FailureCase( + inputs=Inputs(prompt=prompt), observed=row["text"], label=Label.FAIL, + id=str(row["id"]), metadata={"source": os.path.basename(os.path.dirname(path))})) + if len(cases) >= offset + n_texts: + break + if len(cases) >= offset + n_texts: + break + cases = cases[offset:] + print(f"recorded generations read: {seen}; unfinished (no ): {unfinished}; " + f"kept with loop share > {min_loop_frac}: {len(cases)} (after skipping {offset})") + if not cases: + raise SystemExit("no looping generation found; lower --min-loop-frac or --repeat-ngram") + return CaseBatch(cases) + + +def fmt(value, spec: str = "+.4f") -> str: + return "n/a" if value is None else format(value, spec) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--edit-lens", required=True, help="path to a clone of edit-lens") + parser.add_argument("--edit-dir", default="", + help="edit directory (default: /astra_logs/exp8_math/runs/p50c/edit)") + parser.add_argument("--model", default=BASE_REPO) + parser.add_argument("--revision", default=BASE_REVISION, + help="model revision; pass an empty string for a local directory") + parser.add_argument("--device", default="cuda") + parser.add_argument("--n-texts", type=int, default=12) + parser.add_argument("--max-tokens", type=int, default=4096) + parser.add_argument("--repeat-ngram", type=int, default=32) + parser.add_argument("--min-loop-frac", type=float, default=0.03) + parser.add_argument("--n-seeds", type=int, default=2, help="random draws per control") + parser.add_argument("--no-verify-base", action="store_true", + help="skip the base-hash check (only for a dry run on a stand-in model)") + parser.add_argument("--out", default="astra_edit_continuation_audit.json") + args = parser.parse_args() + + from evalrx.analyzers.weights import WeightEditContinuationAudit + from evalrx.wsp import WeightEdit, applied + + edit_dir = args.edit_dir or os.path.join(args.edit_lens, "astra_logs", "exp8_math", "runs", + "p50c", "edit") + edit = WeightEdit.from_astra(edit_dir) + module, tokenizer = load_model(args.model, args.revision or None, args.device) + print(f"loaded {args.model} as {type(module).__name__} on {args.device}") + + if not args.no_verify_base: + try: + with applied(module, edit, verify_base=True) as report: + print("base hashes match; edit applies to:", + ", ".join(r["name"] for r in report.rows)) + except (ValueError, KeyError) as exc: + print(f"REFUSED: {exc}") + return 1 + + model = as_evalrx_model(module, tokenizer) + cases = looping_cases(args.edit_lens, tokenizer, n_texts=args.n_texts, + max_tokens=args.max_tokens, repeat_ngram=args.repeat_ngram, + min_loop_frac=args.min_loop_frac) + + started = time.time() + findings = WeightEditContinuationAudit( + edit=edit, groups="repetition", repeat_ngram=args.repeat_ngram, + controls=("random_direction", "random_columns"), n_seeds=args.n_seeds, + steering=True, max_tokens=args.max_tokens, + ).run(model, cases).findings + print(f"audit finished in {time.time() - started:.0f}s") + + f = findings + print(f"\ncases: {f['n_cases_analyzed']} (skipped {f['n_skipped']}: {f['skip_reasons']})") + print(f"positions: {f['n_positions']:,} = {f.get('n_positions_loop', 0):,} loop + " + f"{f.get('n_positions_productive', 0):,} productive (ngram {f['repeat_ngram']})") + states = [s for s in ("edit", "random_direction", "random_columns", "steer") + if f"mean_kl_{s}_productive" in f or f"mean_kl_{s}_loop" in f] + header = f"{'':<18}" + "".join(f"{s:>18}" for s in states) + for group in ("loop", "productive"): + if f"mean_nll_clean_{group}" not in f: + print(f"\n[{group}] no positions") + continue + print(f"\n[{group}] clean NLL of the recorded next token: " + f"{f[f'mean_nll_clean_{group}']:.4f}") + print(header) + print(f"{'dNLL':<18}" + "".join( + f"{fmt(f.get(f'mean_dnll_{s}_{group}')):>18}" for s in states)) + print(f"{'KL from clean':<18}" + "".join( + f"{fmt(f.get(f'mean_kl_{s}_{group}'), '.3e'):>18}" for s in states)) + print(f"{'top-1 changed':<18}" + "".join( + f"{fmt(f.get(f'frac_flips_{s}_{group}'), '.4f'):>18}" for s in states)) + print(f" specificity vs random direction : " + f"{fmt(f.get(f'specificity_random_direction_{group}'), '.2f')}") + print(f" specificity vs other neurons : " + f"{fmt(f.get(f'specificity_random_columns_{group}'), '.2f')}") + print(f" steering recovery : " + f"{fmt(f.get(f'steer_recovery_{group}'), '.3f')}") + if "dnll_edit_loop_minus_productive" in f: + print(f"\nedit dNLL, loop minus productive: {f['dnll_edit_loop_minus_productive']:+.4f}") + for key in ("controls_not_run", "steering_not_run"): + if key in f: + print(f"{key}: {f[key]}") + + print("\nper case (edit dNLL: loop / productive, share of loop positions):") + for row in f["per_case"]: + print(f" {row['sample_id']:>16} " + f"{fmt(row.get('wc_dnll_edit_loop')):>9} / {fmt(row.get('wc_dnll_edit_productive')):>9}" + f" loop share {row.get('wc_frac_loop', float('nan')):.3f}" + f" n={row['wc_n_positions']}") + n_loop_up = sum(1 for r in f["per_case"] + if r.get("wc_dnll_edit_loop") is not None + and r.get("wc_dnll_edit_productive") is not None + and r["wc_dnll_edit_loop"] > r["wc_dnll_edit_productive"]) + print(f"cases where the edit raises NLL more in loops than outside: " + f"{n_loop_up} of {len(f['per_case'])}") + + with open(args.out, "w", encoding="utf-8") as fh: + json.dump({"model": args.model, "revision": args.revision, "edit": edit.describe(), + "settings": vars(args), "findings": findings}, fh, indent=1, default=str) + print(f"\nfull result written to {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/check_astra_edit_qwen35.py b/tools/check_astra_edit_qwen35.py new file mode 100644 index 00000000..b98574b0 --- /dev/null +++ b/tools/check_astra_edit_qwen35.py @@ -0,0 +1,192 @@ +"""Check a published weight edit against the real Qwen3.5-0.8B base model. + +Two steps, both read-only with respect to the files on disk: + + 1. Load the edit (``edit.npz`` + ``manifest.json``) and apply it in memory with + the base-hash check on. This passes only if every edited tensor of the + loaded model is bit-identical to the tensor the edit was made on. While the + edit is applied, the edited tensors are hashed and compared with the + ``edited_sha256`` recorded in the manifest, which tests that the rebuild is + bit-exact. + 2. Read the structure of the edit from the weights: changed columns, rank, + and the strength ``alpha`` of the rule ``w' = w - alpha * u (u^T w)``. + +For the p50c edit of "Sparks of Weight-Space Programming", EditLens reports 64 +changed columns of ``mlp.down_proj`` in each of layers 15 and 19, a rank-one +change per layer, and alpha = 6. The script prints what it measures next to +those numbers; it does not assume them. + +Requirements: the evalrx branch with ``evalrx.wsp`` installed, torch, +transformers (a release that knows Qwen3.5), and a clone of +https://github.com/assassin808/edit-lens for the edit files. + + python tools/check_astra_edit_qwen35.py \ + --edit-dir /path/to/edit-lens/astra_logs/exp8_math/runs/p50c/edit + +No GPU is needed for these two steps; a 0.8B model fits in CPU memory. +""" + +from __future__ import annotations + +import argparse +import json +import sys + +BASE_REPO = "Qwen/Qwen3.5-0.8B" +# The revision the edit was made on, as stated in the EditLens README. +BASE_REVISION = "2fc06364715b967f1860aea9cf38778875588b17" + +# What EditLens reports for p50c. Used only to label the printed comparison. +EXPECTED = {"changed_columns": 64, "alpha": 6.0, "layers": (15, 19)} + + +def load_module(model: str, revision: "str | None", device: str): + """Load the checkpoint in its stored dtype (bfloat16) without changing any value.""" + import torch + import transformers + + kwargs = {"dtype": torch.bfloat16} + if revision: + kwargs["revision"] = revision + errors = [] + # Only the parameters are used, so any class that loads the checkpoint will do. + for cls_name in ("AutoModelForImageTextToText", "AutoModelForCausalLM", "AutoModel"): + cls = getattr(transformers, cls_name, None) + if cls is None: + continue + try: + module = cls.from_pretrained(model, **kwargs) + print(f"loaded {model} with {cls_name} ({type(module).__name__})") + return module.to(device).eval() + except Exception as exc: # noqa: BLE001 - try the next loader, report all at the end + errors.append(f"{cls_name}: {type(exc).__name__}: {exc}") + raise SystemExit("could not load the model:\n " + "\n ".join(errors)) + + +def step1_verify(module, edit) -> "list[dict]": + """Apply the edit with the base-hash check; compare the rebuilt tensors' hashes.""" + from evalrx.wsp import applied, resolve_parameters, tensor_sha256 + + resolved = resolve_parameters(module, edit) + rows = [] + for tensor_edit, (name, param) in zip(edit.tensors, resolved): + rows.append({ + "edit_name": tensor_edit.name, + "resolved_name": name, + "dtype": str(param.dtype).replace("torch.", ""), + "base_hash_matches": tensor_sha256(param) == tensor_edit.base_sha256, + }) + if not all(r["base_hash_matches"] for r in rows): + return rows # applying would be refused; report which tensors differ + + with applied(module, edit, verify_base=True) as report: + for row, tensor_edit, (_, param) in zip(rows, edit.tensors, resolved): + row["edited_hash_matches"] = ( + tensor_sha256(param) == tensor_edit.edited_sha256 + if tensor_edit.edited_sha256 else None + ) + for row, applied_row in zip(rows, report.rows): + row["changed_entries"] = applied_row["changed_entries"] + row["relative_change"] = applied_row["relative_change"] + # Outside the block the original weights are back. + for row, tensor_edit, (_, param) in zip(rows, edit.tensors, resolved): + row["restored"] = tensor_sha256(param) == tensor_edit.base_sha256 + return rows + + +def step2_structure(module, edit) -> dict: + from evalrx.wsp import edit_structure + + return edit_structure(module, edit) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--edit-dir", required=True, + help="directory with edit.npz and manifest.json (e.g. .../runs/p50c/edit)") + parser.add_argument("--model", default=BASE_REPO, + help="Hugging Face repo id or local directory of the base model") + parser.add_argument("--revision", default=BASE_REVISION, + help="model revision; pass an empty string for a local directory") + parser.add_argument("--device", default="cpu") + parser.add_argument("--out", default="astra_edit_check.json", + help="where to write the full result as JSON") + args = parser.parse_args() + + from evalrx.wsp import WeightEdit + + edit = WeightEdit.from_astra(args.edit_dir) + print(f"edit '{edit.name}': {len(edit)} tensor(s)") + for tensor in edit.tensors: + n_overwritten = 0 if tensor.index is None else len(tensor.index) + print(f" {tensor.name} form={tensor.form} overwritten={n_overwritten:,}") + + module = load_module(args.model, args.revision or None, args.device) + + # ---- Step 1 ------------------------------------------------------- + print("\n[1] base-hash check and bit-exact rebuild") + try: + verify = step1_verify(module, edit) + except KeyError as exc: + print(f" FAILED to resolve a tensor name: {exc}") + return 2 + for row in verify: + print(f" {row['resolved_name']}") + print(f" stored dtype : {row['dtype']}") + print(f" base hash matches : {row['base_hash_matches']}") + if "edited_hash_matches" in row: + print(f" rebuilt hash matches : {row['edited_hash_matches']}") + print(f" changed entries : {row['changed_entries']:,}") + print(f" relative change : {row['relative_change']:.4f}") + print(f" restored after exit : {row['restored']}") + base_ok = all(r["base_hash_matches"] for r in verify) + rebuilt_ok = base_ok and all(r.get("edited_hash_matches") is not False for r in verify) + if not base_ok: + print(" -> the loaded weights are NOT the ones the edit was made on " + "(wrong revision, a re-export, or a dtype other than bfloat16). " + "Step 2 is skipped: the overwritten entries are absolute values and " + "mean nothing on another base.") + json.dump({"edit": edit.describe(), "verify": verify}, open(args.out, "w"), indent=1) + return 1 + + # ---- Step 2 ------------------------------------------------------- + print("\n[2] structure of the edit, read from the weights") + structure = step2_structure(module, edit) + for row in structure["tensors"]: + print(f" {row['name']}") + print(f" changed columns : {row.get('changed_columns')}" + f" (EditLens: {EXPECTED['changed_columns']})") + print(f" changed rows : {row.get('changed_rows')}") + print(f" effective rank : {row.get('effective_rank', float('nan')):.4f}" + " (1 = rank one)") + print(f" rank-one residual : {row.get('rank_one_residual', float('nan')):.4g}" + " (0 = exactly rank one; bfloat16 rounding keeps it above 0)") + alpha = row.get("alpha_estimate") + spread = row.get("alpha_spread") + if alpha is not None: + print(f" alpha estimate : {alpha:.4f} +/- {spread:.4f}" + f" (EditLens: {EXPECTED['alpha']})") + print(f" projection selectivity : {row.get('projection_selectivity', float('nan')):.3f}" + " (>1: the edited columns write along u more than average)") + print(f" top singular values : " + f"{[round(v, 4) for v in row.get('singular_values', [])[:4]]}") + for pair, cosine in structure["direction_cosines"].items(): + print(f" |cos| between edit directions: {cosine:.4f} ({pair})") + + result = { + "model": args.model, "revision": args.revision, + "edit": edit.describe(), "verify": verify, "structure": structure, + "base_ok": base_ok, "rebuilt_ok": rebuilt_ok, + } + with open(args.out, "w", encoding="utf-8") as fh: + json.dump(result, fh, indent=1, default=str) + print(f"\nfull result written to {args.out}") + if not rebuilt_ok: + print("NOTE: the base matched but a rebuilt tensor's hash did not; " + "the loader's rebuild is not bit-exact for this edit.") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/generate_astra_edit_qwen35.py b/tools/generate_astra_edit_qwen35.py new file mode 100644 index 00000000..91af1a40 --- /dev/null +++ b/tools/generate_astra_edit_qwen35.py @@ -0,0 +1,621 @@ +"""Generation-level check of a published weight edit through the L4 fix path. + +The teacher-forced audits say what the edit does to the probability of text +the unedited model wrote. This script lets the models write: GSM8K problems are +generated by the unedited Qwen3.5-0.8B, by the model with the p50c edit of +"Sparks of Weight-Space Programming" written in, and by the model with a +matched random-direction control written in. The comparison runs through +evalrx's own repair path: + + FixAgent(weight_edit={"edit_dir": ...}) -> rank_one_weight_edit candidate + baseline generation, edit generation, control generation + paired McNemar + e-value on correct / incorrect + +and, per arm, reports whether the model left its thinking block, whether it +stopped on its own, whether it ended in a loop, and accuracy. + +Protocol. As in the original evaluation: chat template with thinking ON, user +turn = problem + newline + "Please reason step by step, and put your final +answer within \\boxed{}.", sampling at temperature 0.6, top_p 0.95, top_k 20, +presence penalty 1.5. Differences from the original, which matter: + +* Token budget. The original allowed 32,768 new tokens per answer; the + unedited model used 29,289 on average and the edited one 11,993. This script + defaults to 16,384. An answer that needs more counts as unfinished here, so + absolute accuracies are below the published ones for both arms. +* Engine. Hugging Face ``generate`` instead of vLLM, one sample per problem + instead of two, and the presence penalty implemented here (on generated + tokens only). +* Grading. Numeric comparison of the last boxed answer after the thinking + block with the gold number, not math-verify. + +Speed. Answers are generated in batches, cached per weight state, and handed to +the fix path one by one. Batching pads prompts on the left; before any long +generation the script checks that a padded batch reproduces unpadded results on +this architecture. The check runs in float32, because in the checkpoint's 16-bit +dtype batching alone already moves the output by rounding. If padding is not +exact, problems are chosen so that every batch holds prompts of equal length. Run with ``--timing-only`` first: it performs that check, measures +tokens per second, prints a time estimate and exits. + + python tools/generate_astra_edit_qwen35.py --edit-lens ~/Github/edit-lens --timing-only + python tools/generate_astra_edit_qwen35.py --edit-lens ~/Github/edit-lens +""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +import time + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import audit_astra_edit_qwen35 as base # noqa: E402 + +INSTRUCTION = "Please reason step by step, and put your final answer within \\boxed{}." +ARMS = ("baseline", "edit", "control") + + +# ---------------------------------------------------------------------- +# Grading and output signals +# ---------------------------------------------------------------------- + +def answer_text(text: str) -> str: + """The part of a response the answer is read from: after the last ````.""" + return text.split("")[-1].strip() if "" in text else text + + +def last_boxed(text: str) -> "str | None": + """Content of the last ``\\boxed{...}`` (brace matched), or ``None``.""" + start = text.rfind("\\boxed") + if start < 0: + return None + open_at = text.find("{", start) + if open_at < 0 or text[start + len("\\boxed"):open_at].strip(): + return None + depth = 0 + for i in range(open_at, len(text)): + if text[i] == "{": + depth += 1 + elif text[i] == "}": + depth -= 1 + if depth == 0: + return text[open_at + 1:i] + return None + + +def to_number(value: "str | None") -> "float | None": + """A boxed answer as a number, or ``None`` when it is not one.""" + if value is None: + return None + s = value.strip() + s = re.sub(r"\\text\{[^}]*\}|\\mathrm\{[^}]*\}", "", s) + s = s.replace("\\$", "").replace("$", "").replace("\\%", "").replace("%", "") + s = s.replace("\\!", "").replace("\\,", "").replace(",", "").replace(" ", "") + frac = re.fullmatch(r"\\d?frac\{(-?[\d.]+)\}\{(-?[\d.]+)\}", s) + try: + if frac: + return float(frac.group(1)) / float(frac.group(2)) + if re.fullmatch(r"-?[\d.]+/-?[\d.]+", s): + a, b = s.split("/") + return float(a) / float(b) + match = re.match(r"-?\d+(?:\.\d+)?", s) + return float(match.group(0)) if match and match.end() == len(s) else None + except (ValueError, ZeroDivisionError): + return None + + +def is_correct(text: str, gold: str) -> bool: + pred, target = to_number(last_boxed(answer_text(text))), to_number(gold) + return pred is not None and target is not None and abs(pred - target) < 1e-6 + + +def verbatim_loop_end(text: str, max_period: int = 200, min_repeats: int = 3, + min_span: int = 40) -> bool: + """Whether the text ends with one block of words repeated back to back. + + The last word is dropped (a response cut at the cap usually stops inside + one). A block of 1..max_period words must repeat at least *min_repeats* + times and cover at least *min_span* words. + """ + words = text.split()[:-1] + n = len(words) + for period in range(1, max_period + 1): + reps = max(min_repeats, -(-min_span // period)) + if reps * period > n: + break + block = words[n - period:] + if all(words[n - (j + 1) * period:n - j * period] == block for j in range(1, reps)): + return True + return False + + +# ---------------------------------------------------------------------- +# A model handle that generates in batches and remembers which arm it is in +# ---------------------------------------------------------------------- + +def build_handle(module, tokenizer, watched, args): + """An evalrx ``Model`` over *module* that batches and caches generation. + + The fix path asks for one answer at a time. On the first request in a given + weight state the handle generates every prepared prompt in batches and + serves the rest from memory. The weight state is identified by the sums of + the *watched* tensors (the ones the edit touches), which also tells the + handle whether it is producing the baseline, the edit or the control. + """ + import torch + from transformers import LogitsProcessor, LogitsProcessorList + + from evalrx.core.capability import Capability + from evalrx.core.model import Model, Trace + + class PresencePenalty(LogitsProcessor): + """Subtract *penalty* from every token already generated in this answer.""" + + def __init__(self, penalty: float, prompt_len: int) -> None: + self.penalty, self.prompt_len = float(penalty), int(prompt_len) + + def __call__(self, input_ids, scores): + generated = input_ids[:, self.prompt_len:] + if generated.shape[1] == 0: + return scores + seen = torch.zeros_like(scores, dtype=torch.bool) + seen.scatter_(1, generated, True) + return scores - self.penalty * seen + + device = next(module.parameters()).device + eos = module.generation_config.eos_token_id + eos_ids = set(eos if isinstance(eos, (list, tuple)) else [eos]) - {None} + if tokenizer.eos_token_id is not None: + eos_ids.add(tokenizer.eos_token_id) + pad_id = (tokenizer.pad_token_id if tokenizer.pad_token_id is not None + else (min(eos_ids) if eos_ids else 0)) + + class Handle(Model): + def __init__(self) -> None: + self.capabilities = frozenset({Capability.GENERATE, Capability.LOGITS}) + self.modalities = frozenset({"text"}) + self._hf = (module, tokenizer) + self.prompts: list[str] = [] + self.padding_ok = True + self.states: list[tuple] = [] # fingerprints, in order of appearance + self.records: dict[str, dict[str, dict]] = {} # arm -> prompt -> record + self.max_new_tokens = int(args.max_new_tokens) + + @property + def _loaded(self): + return self._hf + + def forward(self, inputs, capture, spec=None) -> Trace: + ids = tokenizer(getattr(inputs, "prompt", str(inputs)))["input_ids"] + with torch.no_grad(): + logits = module(input_ids=torch.tensor([ids], device=device)).logits[0] + return Trace(tokens=[str(i) for i in ids], token_ids=list(ids), + provided={Capability.LOGITS}, logits=logits) + + # -- weight state ------------------------------------------------ + def arm(self) -> str: + state = tuple(float(p.detach().double().sum()) for p in watched) + if state not in self.states: + self.states.append(state) + index = self.states.index(state) + return ARMS[index] if index < len(ARMS) else f"state_{index}" + + # -- generation -------------------------------------------------- + def batches(self) -> "list[list[int]]": + """Prompt indices per batch: any prompts when padding is exact, else equal lengths.""" + order = list(range(len(self.prompts))) + if self.padding_ok: + return [order[i:i + args.batch_size] for i in range(0, len(order), args.batch_size)] + by_len: dict[int, list[int]] = {} + for i in order: + by_len.setdefault(len(tokenizer(self.prompts[i])["input_ids"]), []).append(i) + return [group[i:i + args.batch_size] for group in by_len.values() + for i in range(0, len(group), args.batch_size)] + + def generate_batch(self, prompts: "list[str]", *, max_new_tokens: int, seed: int, + min_new_tokens: int = 0) -> "list[dict]": + encoded = [tokenizer(p)["input_ids"] for p in prompts] + width = max(len(e) for e in encoded) + ids = torch.full((len(encoded), width), pad_id, dtype=torch.long) + mask = torch.zeros((len(encoded), width), dtype=torch.long) + for row, e in enumerate(encoded): # left padding + ids[row, width - len(e):] = torch.tensor(e) + mask[row, width - len(e):] = 1 + kwargs = dict(max_new_tokens=max_new_tokens, pad_token_id=pad_id) + if eos_ids: + kwargs["eos_token_id"] = sorted(eos_ids) + if min_new_tokens: + kwargs["min_new_tokens"] = min_new_tokens + if args.greedy: + kwargs["do_sample"] = False + else: + kwargs.update(do_sample=True, temperature=args.temperature, top_p=args.top_p, + top_k=args.top_k) + processors = LogitsProcessorList() + if args.presence_penalty and not args.greedy: + processors.append(PresencePenalty(args.presence_penalty, width)) + torch.manual_seed(seed) + with torch.no_grad(): + out = module.generate(input_ids=ids.to(device), attention_mask=mask.to(device), + logits_processor=processors, **kwargs) + records = [] + for row in range(len(encoded)): + new = out[row, width:].tolist() + stop = next((i for i, t in enumerate(new) if t in eos_ids), None) + tokens = new if stop is None else new[:stop] + records.append({ + "text": tokenizer.decode(tokens, skip_special_tokens=False), + "n_tokens": len(tokens), "stopped": stop is not None, "token_ids": tokens, + }) + return records + + def fill(self, arm: str) -> None: + self.records[arm] = {} + batches = self.batches() + started = time.time() + for number, batch in enumerate(batches): + # The same seed for the same batch in every arm: the arms differ + # by their weights, not by the random numbers they draw. + done = self.generate_batch([self.prompts[i] for i in batch], + max_new_tokens=self.max_new_tokens, + seed=args.seed + number) + for i, record in zip(batch, done): + self.records[arm][self.prompts[i]] = record + elapsed = time.time() - started + print(f" [{arm}] batch {number + 1}/{len(batches)} " + f"({sum(r['n_tokens'] for r in done):,} tokens, {elapsed:.0f}s elapsed)", + flush=True) + + def generate(self, inputs, **kwargs) -> str: + prompt = getattr(inputs, "prompt", str(inputs)) + arm = self.arm() + if arm not in self.records: + self.fill(arm) + if prompt not in self.records[arm]: + record = self.generate_batch([prompt], max_new_tokens=self.max_new_tokens, + seed=args.seed)[0] + self.records[arm][prompt] = record + return self.records[arm][prompt]["text"] + + # -- is left padding exact here? ----------------------------------- + def _padded(self, prompts: "list[str]"): + encoded = [tokenizer(p)["input_ids"] for p in prompts] + width = max(len(e) for e in encoded) + ids = torch.full((len(encoded), width), pad_id, dtype=torch.long) + mask = torch.zeros((len(encoded), width), dtype=torch.long) + for row, e in enumerate(encoded): + ids[row, width - len(e):] = torch.tensor(e) + mask[row, width - len(e):] = 1 + return encoded, width, ids.to(device), mask.to(device) + + def _gap(self, prompts: "list[str]") -> "tuple[float, bool, int]": + """Largest top-5 log-prob gap between each prompt alone and in one batch.""" + encoded, width, ids, mask = self._padded(prompts) + with torch.no_grad(): + batched = module(input_ids=ids, attention_mask=mask).logits[:, -1].float() + worst, same_top = 0.0, True + for row, e in enumerate(encoded): + alone = module(input_ids=torch.tensor([e], device=device)).logits[0, -1].float() + a = torch.log_softmax(alone, -1) + b = torch.log_softmax(batched[row], -1) + top = torch.topk(a, 5).indices + worst = max(worst, float((a[top] - b[top]).abs().max())) + same_top = same_top and int(a.argmax()) == int(b.argmax()) + return worst, same_top, sum(len(e) < width for e in encoded) + + def check_padding(self, prompts: "list[str]") -> dict: + """Is a left-padded batch the same computation as each prompt alone? + + Measured in float32. In the checkpoint's own 16-bit dtype, running a + prompt alone and in a batch already differs by rounding, which can + both hide a masking error and look like one. Three gaps are reported: + + ``float32_padded`` prompts of different lengths, padded — the test; + ``float32_unpadded`` one prompt repeated — the floor for float32; + ``native_unpadded`` the same in the checkpoint's dtype — how much + batching alone moves the output there. + """ + dtype = next(module.parameters()).dtype + before = [float(p.detach().double().sum()) for p in watched] + native_floor, _, _ = self._gap([prompts[0]] * len(prompts)) + native_padded, native_top, n_padded = self._gap(prompts) + module.float() + try: + f32_floor, _, _ = self._gap([prompts[0]] * len(prompts)) + f32_padded, f32_top, _ = self._gap(prompts) + finally: + module.to(dtype) + after = [float(p.detach().double().sum()) for p in watched] + if before != after: # a 16-bit value survives the trip through float32 exactly + raise RuntimeError("the weights changed during the padding check") + ok = f32_top and f32_padded < max(args.padding_tolerance, 10 * f32_floor) + return {"ok": bool(ok), "n_prompts": len(prompts), "n_padded": n_padded, + "float32_padded": f32_padded, "float32_unpadded": f32_floor, + "float32_same_top1": bool(f32_top), "native_padded": native_padded, + "native_unpadded": native_floor, "native_same_top1": bool(native_top), + "native_dtype": str(dtype).replace("torch.", "")} + + return Handle() + + +# ---------------------------------------------------------------------- +# Data +# ---------------------------------------------------------------------- + +def gsm8k_cases(edit_lens: str, tokenizer, n_cases: int, stride: int, *, + equal_length_batches: int = 0): + """GSM8K test problems as cases. + + Default: every *stride*-th problem. With ``equal_length_batches=B``: + problems are instead taken from the prompt lengths (in tokens) shared by at + least B problems, B per length, so every batch can be formed without + padding. That narrows the sample to common prompt lengths; the lengths used + are printed. + """ + from evalrx.core.case import CaseBatch, FailureCase, Inputs + + rows = [] + with open(os.path.join(edit_lens, "data", "gsm8k_test.jsonl"), encoding="utf-8") as fh: + for line in fh: + rows.append(json.loads(line)) + + def prompt_of(row: dict) -> str: + return tokenizer.apply_chat_template( + [{"role": "user", "content": row["question"] + "\n" + INSTRUCTION}], + tokenize=False, add_generation_prompt=True, enable_thinking=True) + + if equal_length_batches: + by_len: dict[int, list[dict]] = {} + for row in rows: + by_len.setdefault(len(tokenizer(prompt_of(row))["input_ids"]), []).append(row) + full = sorted((n for n, group in by_len.items() if len(group) >= equal_length_batches), + key=lambda n: (-len(by_len[n]), n)) + needed = -(-n_cases // equal_length_batches) + if len(full) < needed: + raise SystemExit( + f"only {len(full)} prompt lengths have {equal_length_batches} problems; " + f"{needed} are needed. Lower --batch-size or --n-cases.") + used = sorted(full[:needed]) + chosen = [row for n in used for row in by_len[n][:equal_length_batches]][:n_cases] + print(f"problems chosen by prompt length so that batches need no padding: " + f"lengths {used} tokens, {equal_length_batches} problems each") + else: + chosen = rows[::max(1, stride)][:n_cases] + cases = [] + for row in chosen: + gold = row["answer"].split("####")[-1].strip().replace(",", "") + cases.append(FailureCase(inputs=Inputs(prompt=prompt_of(row)), expected=gold, + id=str(row["id"]))) + return CaseBatch(cases) + + +def arm_summary(records: "dict[str, dict]", cases, max_new_tokens: int, ngram: int) -> dict: + from evalrx.wsp import repetition_mask + + rows = [] + for case in cases: + record = records.get(case.inputs.prompt) + if record is None: + continue + mask = repetition_mask(record["token_ids"], ngram) + rows.append({ + "id": case.id, "correct": is_correct(record["text"], str(case.expected)), + "left_thinking": "" in record["text"], "stopped": record["stopped"], + "boxed": last_boxed(answer_text(record["text"])) is not None, + "n_tokens": record["n_tokens"], "hit_cap": record["n_tokens"] >= max_new_tokens, + "verbatim_loop_end": verbatim_loop_end(record["text"]), + "loop_share": sum(mask) / len(mask) if mask else 0.0, + }) + n = len(rows) + if not n: + return {"n": 0, "rows": []} + + def share(key: str) -> float: + return sum(bool(r[key]) for r in rows) / n + + return {"n": n, "accuracy": share("correct"), "left_thinking": share("left_thinking"), + "stopped_on_its_own": share("stopped"), "boxed_answer": share("boxed"), + "hit_token_cap": share("hit_cap"), "verbatim_loop_end": share("verbatim_loop_end"), + "mean_new_tokens": sum(r["n_tokens"] for r in rows) / n, + "mean_loop_share": sum(r["loop_share"] for r in rows) / n, "rows": rows} + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--edit-lens", required=True, help="path to a clone of edit-lens") + parser.add_argument("--edit-dir", default="") + parser.add_argument("--model", default=base.BASE_REPO) + parser.add_argument("--revision", default=base.BASE_REVISION) + parser.add_argument("--device", default="cuda") + parser.add_argument("--n-cases", type=int, default=64) + parser.add_argument("--stride", type=int, default=20, + help="take every stride-th GSM8K test problem") + parser.add_argument("--max-new-tokens", type=int, default=16384) + parser.add_argument("--batch-size", type=int, default=16) + parser.add_argument("--temperature", type=float, default=0.6) + parser.add_argument("--top-p", type=float, default=0.95) + parser.add_argument("--top-k", type=int, default=20) + parser.add_argument("--presence-penalty", type=float, default=1.5) + parser.add_argument("--greedy", action="store_true", + help="greedy decoding instead of the original sampling settings") + parser.add_argument("--seed", type=int, default=0) + parser.add_argument("--no-control", action="store_true", + help="skip the random-direction control arm (saves a third of the time)") + parser.add_argument("--no-verify-base", action="store_true") + parser.add_argument("--padding-tolerance", type=float, default=0.01, + help="largest float32 log-prob gap accepted between padded and unpadded runs") + parser.add_argument("--exact-batches", action="store_true", + help="never pad: choose problems whose prompts have equal length") + parser.add_argument("--repeat-ngram", type=int, default=32) + parser.add_argument("--timing-only", action="store_true", + help="check batching, measure speed, print a time estimate and exit") + parser.add_argument("--out", default="astra_edit_generation.json") + parser.add_argument("--save-texts", default="", + help="also write every generated answer to this .jsonl file") + args = parser.parse_args() + + from evalrx.eval_agent.hypothesis import Hypothesis + from evalrx.eval_agent.stages.fix.agent import FixAgent + from evalrx.wsp import WeightEdit, applied, resolve_parameters + + edit_dir = args.edit_dir or os.path.join(args.edit_lens, "astra_logs", "exp8_math", "runs", + "p50c", "edit") + edit = WeightEdit.from_astra(edit_dir) + module, tokenizer = base.load_model(args.model, args.revision or None, args.device) + print(f"loaded {args.model} as {type(module).__name__} on {args.device}") + if not args.no_verify_base: + try: + with applied(module, edit, verify_base=True): + print("base hashes match") + except (ValueError, KeyError) as exc: + print(f"REFUSED: {exc}") + return 1 + watched = [param for _, param in resolve_parameters(module, edit)] + model = build_handle(module, tokenizer, watched, args) + cases = gsm8k_cases(args.edit_lens, tokenizer, args.n_cases, args.stride) + model.prompts = [c.inputs.prompt for c in cases] + print(f"{len(cases)} GSM8K problems (every {args.stride}th), " + f"up to {args.max_new_tokens:,} new tokens each, batch size {args.batch_size}, " + + ("greedy" if args.greedy else + f"sampling T={args.temperature} top_p={args.top_p} top_k={args.top_k} " + f"presence_penalty={args.presence_penalty}")) + + # ---- Is left-padded batching exact on this architecture? ------------------------- + probe = sorted(model.prompts, key=len) + probe = probe[:2] + probe[-2:] if len(probe) >= 4 else probe + padding = model.check_padding(probe) + print(f"padding check ({padding['n_prompts']} prompts, {padding['n_padded']} padded):") + print(f" float32: padded vs alone {padding['float32_padded']:.5f}, " + f"same prompt repeated vs alone {padding['float32_unpadded']:.5f} " + f"(largest top-5 log-prob gap; same top-1: {padding['float32_same_top1']})") + print(f" {padding['native_dtype']}: padded vs alone {padding['native_padded']:.5f}, " + f"same prompt repeated vs alone {padding['native_unpadded']:.5f}") + padding["exact"] = padding["ok"] + if args.exact_batches: + padding["ok"] = False + if padding["ok"]: + print(" -> the float32 padded gap is within tolerance; batching any prompts " + f"together. Compare it with the {padding['native_dtype']} gap of one prompt " + "repeated: batching alone moves the output more than padding does.") + else: + print(" -> padding is not used (" + + ("--exact-batches" if padding["exact"] else + "it changes the computation on this architecture") + + "). Selecting problems whose prompts have equal length instead.") + cases = gsm8k_cases(args.edit_lens, tokenizer, args.n_cases, args.stride, + equal_length_batches=args.batch_size) + model.prompts = [c.inputs.prompt for c in cases] + model.padding_ok = padding["ok"] + batches = model.batches() + print(f"{len(cases)} problems in {len(batches)} batches per arm " + f"(sizes {sorted({len(b) for b in batches})})") + + # ---- Speed ----------------------------------------------------------------------- + first = batches[0] + started = time.time() + probe_tokens = 256 + model.generate_batch([model.prompts[i] for i in first], max_new_tokens=probe_tokens, + seed=args.seed, min_new_tokens=probe_tokens) + seconds = time.time() - started + rate = len(first) * probe_tokens / seconds + arms = 2 if args.no_control else 3 + worst = arms * len(cases) * args.max_new_tokens / rate + print(f"speed: {rate:,.0f} tokens/s on a batch of {len(first)} " + f"({probe_tokens} tokens each, {seconds:.1f}s)") + print(f"upper bound for {arms} arms if every answer ran to the cap at this speed: " + f"{worst / 3600:.1f} h. Generation slows as answers get longer, and answers that " + "stop early shorten it; treat this as a rough guide.") + if args.timing_only: + return 0 + + # ---- The L4 fix path --------------------------------------------------------------- + def score(case, output) -> bool: + return is_correct(str(output), str(case.expected)) + + agent = FixAgent( + judge=None, max_tier="L4", allow_codegen=False, score_fn=score, floor_candidates=None, + candidate_allowlist={"rank_one_weight_edit"}, + weight_edit={"edit_dir": edit_dir, "verify_base": not args.no_verify_base, + "control": not args.no_control, "seed": args.seed}, + ) + hypothesis = Hypothesis( + statement="The model enters repetition loops while reasoning and never reaches an answer.", + target_model=args.model, predicted_failure_mode="degenerate_repetition", test_design="") + started = time.time() + outcome = agent.propose_and_validate(model, cases, [hypothesis]) + print(f"fix path finished in {(time.time() - started) / 60:.1f} min") + + found = [v for v in outcome.attempted if v.candidate.kind == "weight_edit"] + if not found: + print("the weight-edit candidate was not attempted; candidates:", + [(v.candidate.name, v.verdict) for v in outcome.attempted]) + return 1 + v = found[0] + audit = v.candidate.payload.get("audit") or {} + + # ---- Report ------------------------------------------------------------------------ + summaries = {arm: arm_summary(model.records.get(arm, {}), cases, args.max_new_tokens, + args.repeat_ngram) for arm in ARMS if arm in model.records} + keys = [("accuracy", "accuracy"), ("left_thinking", "left the thinking block"), + ("stopped_on_its_own", "stopped on its own"), ("boxed_answer", "gave a boxed answer"), + ("hit_token_cap", "ran to the token cap"), ("verbatim_loop_end", "ended in a verbatim loop"), + ("mean_loop_share", "mean share of loop tokens"), ("mean_new_tokens", "mean new tokens")] + print(f"\n{'':<28}" + "".join(f"{arm:>12}" for arm in summaries)) + for key, label in keys: + cells = [] + for arm in summaries: + value = summaries[arm].get(key) + cells.append("n/a" if value is None else + (f"{value:,.0f}" if key == "mean_new_tokens" else f"{value:.3f}")) + print(f"{label:<28}" + "".join(f"{c:>12}" for c in cells)) + + print("\npaired test, edit vs baseline (evalrx fix validation):") + print(f" verdict {v.verdict}; fixed {v.n_fixed}, broken {v.n_broken}, pairs {v.n_pairs}; " + f"effect {v.effect}; e-value {v.e_value} (needs {v.e_threshold}); " + f"execution error: {v.exec_error or 'none'}") + print(f" {v.summary}") + print(f" fix outcome: fixed={outcome.fixed}") + if "control_random_direction" in audit: + print(f" correct answers: edit {audit.get('edit_result')}, " + f"random-direction control {audit['control_random_direction']}") + if "baseline" in summaries and "edit" in summaries: + b = {r["id"]: r for r in summaries["baseline"]["rows"]} + e = {r["id"]: r for r in summaries["edit"]["rows"]} + shared = sorted(set(b) & set(e)) + left = sum(1 for i in shared if e[i]["left_thinking"] and not b[i]["left_thinking"]) + lost = sum(1 for i in shared if b[i]["left_thinking"] and not e[i]["left_thinking"]) + print(f" left the thinking block only under the edit: {left}; only at baseline: {lost} " + f"(of {len(shared)})") + + result = { + "model": args.model, "revision": args.revision, "settings": vars(args), + "padding_check": padding, "tokens_per_second_probe": rate, + "arms": {arm: {k: s[k] for k in s if k != "rows"} for arm, s in summaries.items()}, + "per_case": {arm: s["rows"] for arm, s in summaries.items()}, + "validation": {"verdict": v.verdict, "n_fixed": v.n_fixed, "n_broken": v.n_broken, + "n_pairs": v.n_pairs, "effect": v.effect, "e_value": v.e_value, + "e_threshold": v.e_threshold, "summary": v.summary, + "exec_error": v.exec_error, "fixed_cases": v.fixed_cases, + "broken_cases": v.broken_cases, "outcome_fixed": bool(outcome.fixed)}, + "audit": audit, + } + with open(args.out, "w", encoding="utf-8") as fh: + json.dump(result, fh, indent=1, default=str) + print(f"\nfull result written to {args.out}") + if args.save_texts: + with open(args.save_texts, "w", encoding="utf-8") as fh: + for arm, records in model.records.items(): + for case in cases: + record = records.get(case.inputs.prompt) + if record is not None: + fh.write(json.dumps({"arm": arm, "id": case.id, "text": record["text"], + "n_tokens": record["n_tokens"], + "stopped": record["stopped"]}) + "\n") + print(f"generated answers written to {args.save_texts}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/recover_astra_edit_qwen35.py b/tools/recover_astra_edit_qwen35.py new file mode 100644 index 00000000..d8879e68 --- /dev/null +++ b/tools/recover_astra_edit_qwen35.py @@ -0,0 +1,228 @@ +"""Can a loop-vs-productive contrast find the published edit on its own? + +The p50c edit of "Sparks of Weight-Space Programming" was written by an AI +system. This script asks whether evalrx can PROPOSE a comparable edit from the +base model's own behaviour, with no access to that edit: + + 1. Derive: on one set of the unedited model's looping GSM8K generations, + contrast the MLP writes at loop positions with those at productive + positions (``continuation_contrast_edit``). That yields, per layer, a + direction and a ranking of neurons. + 2. Compare with the published edit: where its two layers rank among all + layers, the cosine between the derived and the published direction at + those layers, and how many of the published 64 neurons the derived ranking + recovers (against the number expected by chance). + 3. Audit on a DISJOINT set of generations (``weight_edit_continuation_audit``): + the published edit and the derived one, each at several strengths alpha. + +Nothing here is assumed to succeed. The contrast finds the direction along +which loops differ from ordinary text; the published edit works by overshooting +(alpha = 6, where alpha = 1 does almost nothing), and its direction need not be +that difference. + + python tools/recover_astra_edit_qwen35.py --edit-lens ~/Github/edit-lens +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +import time + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import audit_astra_edit_qwen35 as base # noqa: E402 + + +def with_alpha(edit, alpha: float, name: str): + """The same rank-one edit (direction, neurons) at another strength.""" + from evalrx.wsp import TensorEdit, WeightEdit + + return WeightEdit([TensorEdit.rank_one(t.name, t.u, alpha, cols=t.cols) + for t in edit.tensors], name=name, metadata=dict(edit.metadata)) + + +def audit(model, cases, edit, args, *, controls=(), steering=False, n_seeds=1) -> dict: + from evalrx.analyzers.weights import WeightEditContinuationAudit + + findings = WeightEditContinuationAudit( + edit=edit, groups="repetition", repeat_ngram=args.repeat_ngram, controls=controls, + n_seeds=n_seeds, steering=steering, max_tokens=args.max_tokens, + ).run(model, cases).findings + findings.pop("structure", None) + return findings + + +def summary_row(label: str, f: dict) -> str: + def g(key: str, spec: str = "+.4f") -> str: + return "n/a" if f.get(key) is None else format(f[key], spec) + + return (f"{label:<34}{g('mean_dnll_edit_loop'):>10}{g('mean_dnll_edit_productive'):>10}" + f"{g('mean_kl_edit_loop', '.5f'):>10}{g('mean_kl_edit_productive', '.5f'):>10}" + f"{g('frac_flips_edit_loop', '.4f'):>10}{g('frac_flips_edit_productive', '.4f'):>10}") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--edit-lens", required=True, help="path to a clone of edit-lens") + parser.add_argument("--edit-dir", default="") + parser.add_argument("--model", default=base.BASE_REPO) + parser.add_argument("--revision", default=base.BASE_REVISION) + parser.add_argument("--device", default="cuda") + parser.add_argument("--n-derive", type=int, default=12, + help="generations the direction and neurons are derived from") + parser.add_argument("--n-audit", type=int, default=12, + help="other generations the edits are audited on") + parser.add_argument("--max-tokens", type=int, default=4096) + parser.add_argument("--repeat-ngram", type=int, default=32) + parser.add_argument("--min-loop-frac", type=float, default=0.03) + parser.add_argument("--alphas", type=float, nargs="+", default=[1.0, 2.0, 6.0, 10.0]) + parser.add_argument("--no-verify-base", action="store_true") + parser.add_argument("--out", default="astra_edit_recovery.json") + args = parser.parse_args() + + import torch + + from evalrx.wsp import ( + WeightEdit, + applied, + collect_continuation_stats, + continuation_contrast_edit, + edit_agreement, + fit_rank_one, + rank_columns, + resolve_parameters, + write_sites, + ) + + edit_dir = args.edit_dir or os.path.join(args.edit_lens, "astra_logs", "exp8_math", "runs", + "p50c", "edit") + published = WeightEdit.from_astra(edit_dir) + module, tokenizer = base.load_model(args.model, args.revision or None, args.device) + print(f"loaded {args.model} as {type(module).__name__} on {args.device}") + if not args.no_verify_base: + try: + with applied(module, published, verify_base=True): + print("base hashes match") + except (ValueError, KeyError) as exc: + print(f"REFUSED: {exc}") + return 1 + model = base.as_evalrx_model(module, tokenizer) + + common = dict(max_tokens=args.max_tokens, repeat_ngram=args.repeat_ngram, + min_loop_frac=args.min_loop_frac) + print("\nderive set:") + derive = base.looping_cases(args.edit_lens, tokenizer, n_texts=args.n_derive, **common) + print("audit set:") + held_out = base.looping_cases(args.edit_lens, tokenizer, n_texts=args.n_audit, + offset=len(derive), **common) + assert not ({c.id for c in derive} & {c.id for c in held_out}) + + # The published edit as an explicit rank-one rule, so it can be compared and re-scaled. + reference = fit_rank_one(module, published, name="published") + sites = {s.name: s for s in write_sites(model)} + ref_layers = [sites[key].layer for key, _ in resolve_parameters(module, reference)] + n_cols = max(int(len(t.cols)) for t in reference.tensors) + print(f"\npublished edit as a rank-one rule: layers {ref_layers}, " + + ", ".join(f"alpha={r['alpha']:.3f} residual={r['rank_one_residual']:.4f}" + for r in reference.metadata["fit"])) + + # ---- 1. Derive ---------------------------------------------------------------- + started = time.time() + stats = collect_continuation_stats(model, derive, groups="repetition", + repeat_ngram=args.repeat_ngram, max_tokens=args.max_tokens) + print(f"\n[1] contrast from {stats.n_cases} generations: {stats.n_positions('loop'):,} loop " + f"vs {stats.n_positions('productive'):,} productive positions " + f"({time.time() - started:.0f}s)") + free = continuation_contrast_edit(model, derive, stats=stats, n_layers=len(ref_layers), + n_columns=n_cols, alpha=6.0, name="derived_free_layers") + forced = continuation_contrast_edit(model, derive, stats=stats, layers=ref_layers, + n_columns=n_cols, alpha=6.0, name="derived_same_layers") + separation = {int(k): v for k, v in free.metadata["layer_separation"].items()} + order = sorted(separation, key=lambda i: -separation[i]) + print(" layers by loop-vs-productive separation of the MLP write (largest first):") + print(" ", " ".join(f"L{i}:{separation[i]:.3f}" for i in order[:8])) + for layer in ref_layers: + print(f" published layer L{layer}: rank {order.index(layer) + 1} of {len(order)} " + f"(separation {separation[layer]:.3f})") + free_layers = sorted(int(k) for k in free.metadata["layers"]) + print(f" layers the contrast would pick on its own: {free_layers}") + + # ---- 2. Compare with the published edit ------------------------------------------ + print("\n[2] agreement with the published edit at its own layers") + agreement = edit_agreement(module, forced, reference) + recovery: list[dict] = [] + for tensor, row in zip(reference.tensors, agreement["tensors"]): + layer = sites[row["name"]].layer + weight = sites[row["name"]].module.weight + ranked = rank_columns(stats.groups["loop"][layer], stats.groups["productive"][layer], + weight) + published_cols = set(tensor.cols.tolist()) + k = len(published_cols) + u_pub = torch.from_numpy(tensor.u).double() + signals = { + "push (derived direction)": ranked["push"], + "|projection| (derived direction)": ranked["projection"].abs(), + "activation difference": ranked["activation"], + "push (published direction)": + ranked["activation"] * (u_pub @ weight.detach().double().cpu()), + "|projection| (published direction)": (u_pub @ weight.detach().double().cpu()).abs(), + } + overlap = {name: len(set(torch.topk(score, k).indices.tolist()) & published_cols) + for name, score in signals.items()} + chance = k * k / weight.shape[1] + print(f" L{layer}: |cos(derived u, published u)| = {row['direction_cosine']:.4f}") + print(f" neurons recovered in the top {k} (chance: {chance:.1f}):") + for name, hits in overlap.items(): + print(f" {name:<36} {hits:>3} of {k}") + recovery.append({"layer": layer, "direction_cosine": row["direction_cosine"], + "top_k": k, "chance": chance, "overlap": overlap}) + + # ---- 3. Audit on held-out generations -------------------------------------------- + print(f"\n[3] audit on {len(held_out)} held-out generations " + "(dNLL of the recorded next token; KL from the unedited model; top-1 changed)") + print(f"{'edit':<34}{'dNLL loop':>10}{'dNLL prod':>10}{'KL loop':>10}{'KL prod':>10}" + f"{'flip loop':>10}{'flip prod':>10}") + audits: dict[str, dict] = {} + started = time.time() + for alpha in args.alphas: + label = f"published rule, alpha={alpha:g}" + audits[label] = audit(model, held_out, with_alpha(reference, alpha, label), args) + print(summary_row(label, audits[label])) + for alpha in args.alphas: + label = f"derived, same layers, alpha={alpha:g}" + audits[label] = audit(model, held_out, with_alpha(forced, alpha, label), args) + print(summary_row(label, audits[label])) + label = "derived, own layers, alpha=6" + audits[label] = audit(model, held_out, free, args) + print(summary_row(label, audits[label])) + label = "published edit as stored" + audits[label] = audit(model, held_out, published, args) + print(summary_row(label, audits[label])) + + print("\n derived edit (same layers, alpha=6) against its own controls:") + controlled = audit(model, held_out, forced, args, + controls=("random_direction", "random_columns"), steering=True, n_seeds=2) + audits["derived, same layers, alpha=6, with controls"] = controlled + for group in ("loop", "productive"): + print(f" [{group}] specificity vs random direction " + f"{controlled.get(f'specificity_random_direction_{group}')}, vs other neurons " + f"{controlled.get(f'specificity_random_columns_{group}')}, steering recovery " + f"{controlled.get(f'steer_recovery_{group}')}") + print(f" ({time.time() - started:.0f}s)") + + with open(args.out, "w", encoding="utf-8") as fh: + json.dump({ + "model": args.model, "revision": args.revision, "settings": vars(args), + "derive_ids": [c.id for c in derive], "audit_ids": [c.id for c in held_out], + "published_fit": reference.metadata["fit"], "published_layers": ref_layers, + "layer_separation": separation, "layers_picked_by_contrast": free_layers, + "recovery": recovery, "agreement": agreement, "audits": audits, + }, fh, indent=1, default=str) + print(f"\nfull result written to {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main())