[frahlg]

Every change to Core should meet a simulated house with a real fuse (#1772). On 21 October a rig outside the repo ran the real ftw binary against srcfl/device-simulator and found four bugs that review and go/internal/sitesim missed. The release check was manual. This PR makes it a maintained part of the stack. Core is unchanged.

The split (Fredrik, 22 October)

The house lives in one place, srcfl/device-simulator; FTW keeps only what is FTW's.

	srcfl/device-simulator	srcfl/ftw (this PR)
House	plant per phase, IEC fuse (fuse/ package), Sungrow battery following EMS writes, OCPP chargers and the car, models keyed by device-drivers driver id with per-model timing
Faults	command delays, start delay, silence, outages, reconnects, late and glitched meter, signed currents, load-balancer cap
Runs	simulator run -spec: port base, timeline, /api/run/event, record (plant.jsonl, OCPP frames), seeded random timelines
Scenarios		which cases, FTW's config, FTW's requests, criteria read from FTW's API, known failures, catalog provenance
Entry points		make siterig, siterig-full, siterig-random, siterig-coverage
Where it runs		siterig-runner on its own machine, posting siterig/short, siterig/full, siterig/random commit statuses. No GitHub Actions.
Release		the beta gate reads siterig/full (#1798, proposal)

Paired PRs, all merged into device-simulator's main: srcfl/device-simulator#8 (the six gaps the first rig patched around, driver ids, model parameters), #9 (the thermal fuse package), #10 (headless runs) and #11 (faults and random timelines). go/test/site/devsim.version pins main at a3030d3.

What is here

• go/test/site (test files only): TestSite starts simulator run with a house spec per scenario, starts ftw against it, drives FTW's API, records /api/status and /api/loadpoints once a second, and judges each run from what flowed through the fuse (the house's record) and what FTW said. A criterion in a scenario's known map must still fail; when a fix makes it pass, the run fails until the fix deletes the line. A record with a hole over 15 s (a sleeping laptop) fails as "host paused".
• Judging again: make siterig-judge DIR=… judges a recorded run with the criteria as they are now, so a changed criterion or known line is checked against a 56-minute run in a second.
• Lanes: FTW_SITE_LANES=n runs n scenarios at once, 100 ports apart from FTW_SITE_PORT_BASE. FTW_SITE_SHARD=k/n splits a set over machines.
• Random mode: FTW_SITE=random:<first seed>:<count>. Each seed draws FTW's side (fuse, house, phase mode and steps, Charge now, pause, plan, Core restart, import ceiling at random times); the house draws its own timeline from the same seed. Eight invariants every second: the fuse holds; over the limit no longer than the fastest lever + 2 s + 3 s after an outside step, and at most 3 s after FTW's own raise; Home use measured while every device reads; Charge now charges when there is room; no offer above the charger's rating; no discharge asked at the battery's floor; no raise on a stale meter. A failing seed is cut a minute after its first violation and written to found/<seed>.json; copied to testdata/found/, it runs in the full set, known until triaged. make siterig-random SEED=n replays a seed.
• The fault catalog: each scenario names the entries of the catalog (reports/felkatalog/catalog.yaml in the FTW stack workspace) it covers and its sources; make siterig-coverage CATALOG=… prints which faults have a scenario, per area. Today 29 of 840 entries have one; in fuse_charging 14 of 112 faults, 7 of its 34 open ones.
• siterig-runner (go/test/site/runner, stdlib only): polls master and open pull requests with gh api, runs the short set on PR heads that touch control, loadpoint, OCPP or telemetry code and the full set on each new master commit, a night's random seeds once a day, and posts the result on the exact SHA with counts in the description. It skips forks unless -allow-forks: the rig runs the pull request's code. launchd and systemd files and install steps in its README. This Mac can run it today, a dedicated box later; never the home energy box.

Results

The full set on this Mac, three lanes, Core = master at bbeac31 (7202ace only adds tests), device-simulator at 76e3fdf, judged again with this branch's criteria (make siterig-judge; s5t rerun with the charger that switches phases): 16 pass, 7 fail as known, 0 unexpected (23 scenarios; s7s runs in the short set only). It took 72 minutes; s12 alone takes 56, so four lanes would not be faster.

Scenario	Case	Criteria	Master	Time
s0-25	OCPP charger adopted, no car: Home use stays measured	✓ Home use measured in 95 % of samples ✓ no OCPP disconnects	pass	5 min
s0b-25	the same at the charger's default heartbeat	✓ Home use measured in 95 % of samples ✓ no OCPP disconnects	pass	5 min
s1-16	Charge now, house 3 A per phase, 16 A fuse	✓ target within 60 s ✓ holds 95 % of the time ✓ fuse holds	pass	4 min
s1-25	Charge now, house 3 A per phase, 25 A fuse	✓ target within 60 s ✓ holds 95 % of the time ✓ fuse holds ✓ no fuse_limit with room to spare	pass	4 min
s2-16	Charge now during 9 A per phase of PV export, 16 A	✗ car starts within 60 s ✓ fuse holds	known: export counted as import, fix in the next beta	3 min
s3-16	house load steps to near the fuse while car and battery charge, 16 A	✓ fuse holds ✓ over fuse - 1 A no longer than the response time + 3 s after the step	pass	7 min
s3-25	the same on 25 A	✓ fuse holds ✓ over fuse - 1 A no longer than the response time + 3 s after the step	pass	7 min
s4-25	surplus_unlock_bat_soc saved but not armed, then Charge now	✓ no stops ✓ no offer drops to 0 W ✓ car starts ✓ full power over the last minute	pass	4 min
s4b-25	the same without a phase mode	✓ no stops ✓ no offer drops to 0 W ✓ car starts ✓ full power over the last minute	pass	4 min
s5-25	surplus only, phase mode auto, about 9 kW of surplus	✓ no offer above the charger's rating ✓ fuse holds ✓ only valid steps (6-16 A) ✗ charges at least the minimum 90 % of the time	known: surplus without steps never starts, next beta	5 min
s5s-25	the same with a step table	✓ no offer above the charger's rating ✓ fuse holds ✓ only valid steps (6-16 A) ✗ charges at least the minimum 90 % of the time	known: auto stays on 1p under 9 kW surplus, next beta	5 min
s6-25	PV inverter offline for 5 minutes	✓ Home use shown throughout the outage ✓ Home use marked estimated throughout the outage ✓ measured before the outage ✓ measured after the outage ✓ never estimated while every driver reads	pass	8 min
s7-25	Core restart of about 20 s while charging	✓ the day's Home use is kept ✓ Home use back within 30 s ✓ Charge now never stops ✓ full power within 420 s	pass	23 min
s8-60-25	charger applies profiles 60 s late	✓ fuse holds ✓ over fuse - 1 A no longer than the charger's delay + 7 s	pass	7 min
s8-120-25	charger applies profiles 120 s late	✓ fuse holds ✓ over fuse - 1 A no longer than the charger's delay + 7 s	pass	12 min
s9-25	Charge now switched from 3p to 1p at the same watts	✓ fuse holds ✓ over fuse - 1 A at most 3 s on 3p ✓ over fuse - 1 A at most 3 s after the switch to 1p	pass	5 min
s9r-25	s9 reversed: 1p raised to 3p beside L2 at 20 A (house 2/20/2 A)	✓ fuse holds ✓ over fuse - 1 A at most 3 s on 3p ✓ over fuse - 1 A at most 3 s after the switch to 1p	pass	5 min
s10-25	two chargers: one raised while the other has just been cut	✓ fuse holds ✓ over fuse - 1 A at most 3 s after the step	pass	8 min
s11-25	pause and restart; charger declares 180 s, applies in 5 s	✓ an offer within 60 s of the restart ✓ over fuse - 1 A at most 3 s ✓ fuse holds	pass	5 min
s5t-25	s5s with a goal on a surplus-only loadpoint, a charger that switches phases	✓ no offer above the charger's rating ✓ fuse holds ✗ charges at least 6 A on one phase 90 % of the time	known until #1799 (B)	5 min
s12-20	the field night: plan on auto with Easee's steps, 3x20 A, charger applies in 40 s	✓ fuse holds ✓ planned charging never orders one phase ✗ no order cut while every phase has room ✓ at least 90 % of the plan by the ready time	known until #1799 (A); 90 % tolerated	56 min
s13-25	Charge now across a 10 s charger socket reconnect	✗ Charge now goes on after the charger returns ✓ no unplug seen	known until #1788	5 min
s14-25	a one-phase car on an 11 kW charger under a plan	✗ delivers 3.4-3.8 kW over the last 10 min ✗ never called not following	known until fix D	20 min

What the new cases found on master:

• s12, the field night. Every start from 0 W goes: 16 A, cut to 0 W 2 s later, three times 32 s apart, then about 130 s of nothing, then a ramp from 7 A to 13 A in 6 s steps. Each start costs about 3.5 min. One start in the window left 93 % of the plan; two starts (the planner paused mid-window) left 87 %. So "90 % of the plan" is tolerated, not known, until fix(loadpoint): planned charging runs on three phases, and room never cuts a running order #1799 (A); the cuts are known. The plan here needs 7-10 kW, more than one phase gives, so this case tests the cuts and the shortfall, not the field's one-phase choice (fix(loadpoint): planned charging runs on three phases, and room never cuts a running order #1799 cause 1), which needs a lighter plan.
• s14, one-phase car. Core orders 14 A on three phases (9 660 W); the car takes 3.2 kW on L1 and Core calls it not_following (power_below_target). Known until fix D.
• s5t never starts under 4.14 kW: master orders 6 A on three phases and cuts it for fuse_limit (see fix(loadpoint): planned charging runs on three phases, and room never cuts a running order #1799 below). s13 ends Charge now at a socket reconnect (Should a charger socket reconnect end Charge now? #1788).
• s9r passes: L2 at 20 A stays under its limit when the 1p order moves to 3p.
• s7 first failed on a rig fault: the sample right after Core came back had no status yet and counted as a lost day total. Only samples with a status count now; the run passes.

On the pin as pushed (26430d2, with Codex's eight fixes): the short set plus s9, s9r, s11 and s13 gave the same verdicts as on 76e3fdf: 10 pass, s13 known (33 minutes, two lanes).

Against #1799's head (74603d0): s5t still fails. Core now orders 16 A on one phase, then cuts it to 0 W within 4 s for fuse_limit, while each phase exports 5.5 A and /api/status shows phase_amps [5.5, 5.5, 5.5] with phase_powers −1 270 W each: export read as import, the same fault s2 keeps known. So s5t's known line names both. s12, on the other hand, is fixed there: no cuts, 6 862 Wh of the 6 600 planned, the first order 57 s after the plan (8 A, rising to 15 A on three phases), so its known line points at the right fix and goes once this branch is on a master with #1799.

Random mode: seed 1 (52 events over 20 min) tripped "no discharge asked at the battery's floor" for 902 s (catalog ev-099); random cases tolerate that finding until it is fixed.

Update: master 2e84710 and a one-phase charger (22 October, afternoon)

The branch is now on master 8ac2750 (#1799, #1795, #1802, #1801, #1804). The full set on 2e84710 (72 min, three lanes; device-simulator 26430d2, drivers 5cc7bb1a), judged with this branch's criteria:

• 16 pass, 5 known: s2 and s5t (export read as import), s5 and s5s, s13 (Should a charger socket reconnect end Charge now? #1788).
• s12 fixed by fix(loadpoint): planned charging runs on three phases, and room never cuts a running order #1799: no cuts with room, 99 % of the plan. Its known and tolerated lines are gone. The judge no longer counts a pause the plan asked for as a cut (one such pause between slots on 2e84710), and it records fuse_limit_with_room_s, seconds of fuse_limit with 6 A of room: each ramp step against the 40 s charger waits up to 144 s.
• s14, new failure: fix(loadpoint): a one-phase car on a three-phase charger is seen, planned and ordered as one #1801 ended not_following, but the one-phase car now stops. After a replan (reactive-load in one run, loadpoint_soc_changed in a second) the plan has no EV energy, commanded_w 0, reason plan: 0 W and 128 W over the last 10 min. Before fix(loadpoint): a one-phase car on a three-phase charger is seen, planned and ordered as one #1801 (62ed7ea) the car kept 3.1 kW. Its known lines are gone, so the set fails until the fix (in progress) lands.

New cases, with device-simulator#11's wired_phases: 1 (a charger wired on L1 gives an order's current there, up to its rating):

Case	2e84710 (before #1804)	8ac2750
s15-25 one-phase charger, Charge now as the app sends it (short set)	fail: the API reports 3 phases, the 3p order gives 0 W with fuse_limit	pass: API 1 phase, 3 657 W after 1.7 s
s15b-25 the same, phase mode 1p		pass, 1.4 s
s15c-25 the same, phase mode auto		pass, 3.1 s
s15p-25 one-phase charger, plan for 600 Wh		pass: 1 000 Wh, at most 3 680 W, never not following

(At the time, devsim.version pinned #11's head, cba7142.) The full set is now about 270 scenario-minutes: about 90 min in three lanes, 70 in four.

Release v0.155.1-beta.1: 16c4ad0 (22 October)

The full set on the release commit: 28 scenarios, four lanes, 63 minutes, drivers 5cc7bb1a, device-simulator cba7142. Judged with this branch at 4d7b0b1: 24 pass, 4 known, 0 fail; siterig/full success is posted on 16c4ad0.

	Scenarios
pass	s0, s0b, s1-16, s1-25, s3-16, s3-25, s4, s4b, s5t, s6, s7, s8-60, s8-120, s9, s9r, s10, s11, s12, s14, s15, s15b, s15c, s15bat, s15p
known	s2-16 (export counted as import), s5-25 (surplus without steps never starts), s5s-25 (auto stays on 1p under 9 kW surplus), s13-25 (#1788)
fixed since this morning	s12 (#1799), s14 (#1806, after #1801 had stopped the car), s5t (between 2e84710 and 16c4ad0), s15 (#1804), s15bat (#1805)

s5t's known line was the only failure, because it passes now; it is gone. s15bat-16 (one-phase charger, 16 A, Charge now while Core's battery raise is on its way; added for #1805): 8ac2750 fails (car charging after 40 s), 5b697a5 and 16c4ad0 pass (3.1 s and 2.1 s). It needs the battery to follow 8 s late; with the model's 2 s, Charge now catches a raise in flight only by chance.

The thermal fuse (23 October)

device-simulator#9 merged: each phase's fuse is now a thermal model, within IEC 60269-1's gates and following ABB's gG curves for 16-63 A. The stack #8 → #10 → #11 is rebased on it.

• devsim.version pinned Docs: per-package CLAUDE.md for IO + plumbing layers #11's head, 3bf737d; it now pins device-simulator's main, a3030d3, the same tree.
• A house spec can set site.fuse_ambient_c and site.fuse_preheat_a, so a scenario can start with warm fuses; no scenario uses them yet.
• The short set on this pin: 8 of 8 pass (23 minutes, two lanes, device-simulator fetched by the pin).

Ready (23 October)

On master 7cccf99 (with #1812) and device-simulator main a3030d3, drivers 219c462e: the short set passes 8 of 8 (s0, s1-16, s1-25, s3-16, s3-25, s7s, s8-60, s15; 23 minutes, two lanes, device-simulator fetched by the pin). The beta gate (#1798) stays a draft until the own-machine runner has run steadily for a few days.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-23T06:31:37.738289Z	b2c4fc4	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1d09cf0c17

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Reject missing simulator measurements before judging safety

If the simulator's site endpoint fails during a judged window, the recorder still writes timestamped samples containing site_err, but sample discards that error and this fallback turns the missing measurement into zero current. blown likewise returns false for a missing site. Consequently, safety-only judges such as S3, S8, S9 and S10 can report success without observing the fuse or phase currents; the longest-gap check does not catch this because samples continue arriving. Validate measurement completeness before evaluating these criteria, and fail the run when required simulator readings are unavailable.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Assert that both chargers reach the load-sharing setup

If multi-charger dispatch stops charging one or both cars, S10 still passes: its only criteria check fuse safety, while pre_step_w is merely recorded. plug confirms connection and charger recognition, not power delivery, and successful manual-hold requests do not establish that both cars reached 10 A before the step. This allows the test to pass without exercising the delayed release of capacity it is intended to cover, even when the single-charger scenarios remain healthy. Require both chargers to reach the intended pre-step draw before applying and judging the cut/raise sequence.

Useful? React with 👍 / 👎.

[frahlg]

Back to draft: direction changed (Fredrik, 22 Oct). The rig must not run in GitHub Actions; a runner on its own machine will publish siterig/short and siterig/full commit statuses instead, and the house moves into srcfl/device-simulator (headless scenario runner, fault injection, importable fuse model), leaving FTW the scenario pack, make siterig and the runner. This PR will be reworked to that shape; site.yml goes away. The CI results above stay valid as evidence of the scenarios and criteria.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2c4fc4f1c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Fail random runs when an FTW action cannot be applied

If a generated pause, plan, ceiling change or restart fails, the run only records ftw:error and continues successfully. judgeInvariants never checks that mark, so the seed can pass without exercising its requested actions—for example, a rejected plan leaves an idle, fuse-safe house, and a failed restart can leave FTW unavailable without violating the safety-only checks. Return the action error or add a mandatory action-success criterion so broken API requests and restart failures cannot produce a successful random status.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Propagate recorded-run failures to the judge test result

When rejudging a recording produces failed criteria or a loading error, TestJudge only logs and saves the outcome; it never marks the test failed. Consequently, make siterig-judge exits successfully even when the revised criteria reject the recording, including when a known criterion now passes and requires its exemption to be removed. Check o.Error and o.Failures as TestSite does so the command's exit status reflects the rejudged results.

Useful? React with 👍 / 👎.