[stakach]

Why

#65 added work at startup that grows with the size of the estate, all at once:

• Every Module reconciled in one batch. Each system the router mapped as it loaded counted as changed (the mappings start empty), so every Module in the estate was queued for reconcile. The first pass then read every Module's redis state and held all the values before publishing any of them. This was on top of the initial sync, which already publishes everything a page at a time with backpressure. On the e2e stack this showed as reconciled retained state … added: "278" straight after boot.
• Quadratic loading. Mapping a system scanned every Module's mappings: three times since fix(mqtt): keep retained state topics accurate #65, once before it. Loading 20,000 systems of 5 modules took 9.8s for mapping alone, and that grows with the square of the estate.
• The broker sweep held two copies of every topic: everything the broker retains, and everything published, compared as two sets.

Changes

• Initial load: a system read by the routers' initial load is mapped from that row (no second query per system) and queues no reconcile. Status events haven't started at that point, nothing is published yet that needs removing, and the initial sync publishes every Module against these mappings. Changes after startup reconcile exactly as before.

• Indexes: Mappings::State keeps a reverse index of system → modules, maintained by map_module / unmap_system / unmap_module, and Published keeps one of system → modules. Per-system lookups no longer scan everything. Mapping 20,000 systems: 9.8s → 0.04s.

• Reconcile: status names are read for the whole set (HKEYS), which is still needed so a removal never deletes a topic another Module now owns. Values are read one Module at a time as they're published, so payloads never accumulate. For a large zone or driver change, peak memory is now the affected Modules' topic names, not their values.

• Sweep: each retained topic is checked against the registry as the broker streams it, through its system with a small cache of recent systems, and only stale-looking topics are kept. They're re-checked in the processing fiber before removal, as before, so a status published mid-sweep isn't removed. Peak memory is the stale topics plus a few systems' topics.

• Router loading: the Module and Driver routers used to ask the database whether each row was in scope as they loaded. That meant each Module's systems and their zones, and for a Driver every one of its Modules and then theirs. Every system and Module is already mapped by then, so loading now asks the mappings. A system being loaded is mapped from the zones already read with it.

• Metadata: it's broadcast inline instead of from a fiber per model, so the bounded queues hold loading back. Before, every system and zone at boot got a fiber holding its JSON. It's also published once per model instead of once per top-level zone, since the topic doesn't depend on the zone.

• Table loading: this is in fix: page the initial load by key, not by offset place-labs/resource#20 (keyset paging instead of OFFSET, which walked about n²/128 rows per table: 10.4s of database time against 0.04s on 100k rows). Released as placeos-resource 3.2.0, which shard.lock now uses.

At scale

3,000 systems, 15,000 modules and 165,312 statuses in redis, with core stopped and an anonymous mosquitto (the e2e broker checks every publish with rest-api):

	master (#65)	this branch + resource#20
routers loaded	25.8s	4.0s
initial sync complete	48.9s	9.2s
sweep complete	not after 20 min: 100% CPU, broker showed no connected client	11.3s
peak memory	1.1 GiB	73 MiB

A warm restart (165k topics retained) took 11.7s at a 72 MiB peak. Master published every value twice (a 165,312-value reconcile on top of the initial sync).

Known limitation: the sweep can't see a whole large broker

mosquitto drops messages for a client whose outgoing queue passes max_queued_messages (1000 by default), and a subscription delivers every retained topic at once. A client that drains slower than the broker sends gets only part of them. After deleting 2,500 modules (about 27,500 stale topics) while source was down, the sweep saw and removed 385, and the broker logged dropped messages for source's client. mosquitto_sub, which drains faster, received all 14,965 connected topics. #65's sweep has the same limit. Changes made while source is running are reconciled exactly and don't depend on the sweep. This only affects stale topics left from before a start.

Testing

• ./test: 69 examples, 0 failures. New spec/published_spec.cr, new mappings index specs, and a router spec showing that a loaded system queues no reconcile but a change after startup does.
• e2e (PlaceOS/user-interfaces stagehand local suite) against an image of this branch: 15/15. Startup now logs the initial sync and sweep with no batch reconcile.
• Sweep, by hand: I planted a wrong-index topic in a live system and a topic for a system that doesn't exist, then restarted source. It logged checked: "2", removed: "2", and the live topic was kept.

🤖 Generated with Claude Code