Repository navigation
Add probe tools and Add WSP option in L4 - #149
Merged
Merged
Conversation
Probe tools (evalrx.probing) - Describe every registered analyzer as a tool: family, the question it answers, access level, cost hint, JSON-schema arguments. - describe_probes(): availability on a model and batch with every reason a probe cannot run, following VLMProbe's Prober.describe() convention. - ProbeRouter ranks runnable probes for failure modes, families and access. - ProbeToolbox.call(name, params) runs a probe with schema-checked arguments; .tools() returns evalrx.core.tool.Tool objects. - Agentic loop: new list_probes tool; run_probe accepts name + params or a list of probes. Named calls run exactly what was named, add to the evidence gathered so far, and are rejected before any model work when they do not fit. run_probe with no arguments is unchanged and keeps its limit of 3. - ProbeAgent.probe gains analyzer_params and exact; a pinned re-run replays the arguments of the named call. Weight-space programming (evalrx.wsp) - TensorEdit / WeightEdit: rank-one rule w' = w - alpha u (u^T w), low-rank, dense delta, sparse overwrite; save/load; reader for the edit layout published with "Sparks of Weight-Space Programming". - applied(): writes an edit and restores the exact weights, also on error; optional base-hash check and size limit. - Controls (random direction, random columns), structure read-out, and contrast_edit (an edit derived from failing vs passing cases). - Analyzers weight_edit_structure and weight_edit_audit (the edit beside random-direction, random-neuron and steering controls). - Fix tier L4: opt-in rank_one_weight_edit candidate via FixAgent(weight_edit=...), validated by the existing paired test and recorded with a matched random-direction control. Tested on a hand-built model with a planted rank-one defect and on tiny random-weight modules; not yet run on a full-size model. See docs/wsp_and_probe_tools.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
tools/check_astra_edit_qwen35.py loads an edit in the published edit.npz + manifest.json layout, checks the base hashes against the loaded Qwen3.5-0.8B weights, rebuilds the edited tensors and compares their hashes with the manifest, then prints the structure of the edit (changed columns, rank, alpha) next to the numbers EditLens reports. Dry-run only on a tiny bfloat16 Llama with a synthetic edit; not yet run on Qwen3.5-0.8B. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
weight_edit_continuation_audit appends each case's recorded output to its prompt and scores every continuation position, teacher-forced, on the token that actually came next: change in NLL and KL from the unedited model, under the edit, random-direction and random-neuron controls, and a constant activation shift. Positions are split into those inside a repeated stretch (the n tokens before the predicted token already occurred) and those outside. The existing weight_edit_audit reads the end of the prompt only, which cannot show an edit that acts inside repetition loops during generation. tools/audit_astra_edit_qwen35.py runs it for the published p50c edit on the unfinished GSM8K generations of the unedited model. Dry-run on a tiny stand-in model with the real recorded texts; not yet run on Qwen3.5-0.8B. docs: record the structure check result on Qwen3.5-0.8B (hashes match; 64 columns per layer, rank one, alpha = 6.000 at layers 15 and 19). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
continuation_contrast_edit derives a rank-one edit from the contrast between positions inside a repeated stretch and positions outside one, along recorded generations. It is the counterpart of contrast_edit for a failure that only exists during generation; both now share one core, edit_from_stats. collect_continuation_stats and rank_columns expose the per-layer group means and the per-neuron push, projection and activation signals. fit_rank_one re-expresses an edit as the rank-one rule that reproduces it, so an edit stored as overwritten entries can be scaled over alpha. edit_agreement compares two edits: shared tensors, direction cosine, neuron overlap against chance. The sequence building and the loop / productive labelling move to evalrx.wsp.continuation, so the audit and the contrast label positions identically. tools/recover_astra_edit_qwen35.py derives an edit on one set of looping generations, compares it with the published p50c edit, and audits both over alpha on a disjoint set. Dry-run on a tiny stand-in model, where no recovery is expected and none is found; not yet run on Qwen3.5-0.8B. docs: record the continuation audit of p50c on Qwen3.5-0.8B, and correct the audit script's description of what EditLens reports for productive positions (+0.053, not "largely alone"). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
tools/generate_astra_edit_qwen35.py generates GSM8K answers with the
unedited Qwen3.5-0.8B, with the published p50c edit written in, and with a
matched random-direction control, through FixAgent(weight_edit={"edit_dir"}).
It reports the paired validation and, per arm, whether the model left its
thinking block, stopped on its own, ended in a loop, and was correct.
Protocol follows the original evaluation (thinking on, T=0.6, top_p=0.95,
top_k=20, presence penalty 1.5) with three stated differences: a 16,384-token
default budget instead of 32,768, Hugging Face generate with one sample per
problem instead of vLLM with two, and numeric grading of the last boxed
answer instead of math-verify.
Answers are generated in batches and cached per weight state, so the fix
path's one-at-a-time requests stay cheap. Before any long generation the
script checks that left-padded batching reproduces unpadded predictions on
the architecture and otherwise batches only prompts of equal length.
--timing-only runs that check, measures speed and prints a time estimate.
Dry-run on a tiny stand-in model; not yet run on Qwen3.5-0.8B. The grading
and loop-end helpers are unit tested.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
On Qwen3.5-0.8B the first run of the check reported a top-5 log-prob gap of 0.2455 between padded and unpadded predictions and accepted it against a tolerance of 0.25. That number cannot be read: in bfloat16, running a prompt alone and in a batch already differs by rounding, so the gap may be a masking error or may be noise, and the tolerance was set where it could not tell. The check now runs in float32 (the weights are cast for the check and restored, with the edited tensors verified unchanged) and reports the padded gap beside the gap of one prompt repeated, in float32 and in the checkpoint's own dtype. Padding is used only when the float32 gap is below 0.01. When padding is not exact, or with --exact-batches, the script selects GSM8K problems from prompt lengths shared by a full batch of problems, so every batch is formed without padding. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
README: probes as tools under "Use it as a library"; the L4 row of the repair ladder now lists the training-free weight edit; a "Weight edits" section with the result on Qwen3.5-0.8B and a statement of what was not achieved (the edit was published by another system; deriving one recovered about half of its neurons and not its direction). docs/validation/astra_p50c_20261115.md records the four runs on Qwen3.5-0.8B with the p50c edit: structure, teacher-forced audit with dose curve, generation through the L4 fix path (6 -> 37 of 64 correct, control 7; 31 fixed, 0 broken), and the loop-contrast recovery attempt, with limits. The generation script no longer calls padding "exact": in float32 the padded gap was 0.016 against 0.004 for one prompt repeated. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
xxlya
requested review from
Nanboy-Ronan,
evalvitals and
jiaq-liu
as code owners
November 16, 2026 00:11
Nanboy-Ronan
previously approved these changes
Nov 16, 2026
ubuntu-latest has started resolving to Ubuntu 26.04. Playwright 1.55.0 does not support Chromium there, so `npx playwright install --with-deps chromium` exits 1 before any frontend test runs (run 43778858430, job 133011251167). The job last passed on main on 2026-11-13 on the previous image. Only the frontend job is pinned; lint, docs and the Python test jobs pass on the new image and stay on ubuntu-latest. Unpin when @playwright/test is upgraded to a release that supports 26.04. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KoBWQhSUCKmU2rGbRh5ALP
Nanboy-Ronan
approved these changes
Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Type of change
Checklist
ruff check .passesmypy evalrxpassespytest -m "not gpu"passes (all fast unit tests green)FakeModelCHANGELOG.mdupdated under[Unreleased]