Repository navigation
fix(mqtt): keep retained state topics accurate - #65
Merged
Merged
Conversation
Brokers held retained state that no longer applied, and missed state that did. Clients that trust the broker (stagehand's hot list) alerted on deleted modules and on a module's old index after a reorder. - number modules by the system's own module order, not database order - record what is published per module, and after any system, module, driver or zone change remove topics that no longer apply and publish current values at new ones - handle changes to one system in order, re-reading it from the database - publish state written before the module's system was known - only touch the systems a zone change applies to - send publishes and deletions through one ordered queue - remove a cleared status from the broker rather than retaining null - republish everything when a broker reconnects, and retry a broker that is down at startup with backoff - sweep retained state topics nothing maps to after the initial sync Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 14, 2026
stakach
added a commit
that referenced
this pull request
Oct 14, 2026
* fix(mqtt): keep startup memory and time flat on large estates #65 made startup reconcile every Module in the estate in one batch: each system mapped as the routers loaded queued a reconcile, so the first pass read every Module's state into memory at once, on top of the initial sync that already publishes it with backpressure. Mapping each system also scanned every Module, so loading was quadratic in the estate's size. - systems read by the initial load are mapped from that row and queue no reconcile; the initial sync covers them - index system modules by system, and the published registry by system, so per-system work no longer scans everything - reconcile reads status names up front and values a Module at a time - the broker sweep checks each retained topic as it arrives and keeps only the stale ones, rather than comparing two sets of every topic Mapping 20,000 systems of 5 modules: 9.8s before, 0.04s after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix: answer scope from the mappings while the routers load The Module and Driver routers asked the database whether each row was in scope as they loaded: a Module's systems and their zones, and for a Driver every one of its Modules, then theirs. Every system and Module is already mapped by then, so loading asks the mappings instead. Metadata is broadcast inline rather than from a fiber per model, so the bounded publisher queues hold loading back instead of thousands of fibers each holding a model's JSON, and is published once per model rather than once per top-level zone, as the topic does not depend on the zone. A system being loaded is mapped from the zones already read with it. With the rest of this branch and place-labs/resource#20, at 3,000 systems and 15,000 modules the routers finish loading in 4s (26s on master) and boot peaks at 73MiB (1.1GiB on master). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(shard.lock): bump placeos-resource to 3.2.0 for keyset loading Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Clients that trust the broker's retained state got the wrong picture. Stagehand's hot list alerted on modules that had been deleted, and after a system's modules were reordered it alerted on the old index. Stagehand hides this today by re-checking every alert over REST, which makes it slow. With this fix the broker can be trusted and that re-check can be removed.
Bugs fixed
Published). After any system, module, driver or zone change it removes topics that no longer apply (keeping any another module now owns) and publishes current values at new ones.null. They are now removed from the broker.statetopics and removes the ones nothing maps to. This cleans up what production brokers hold today.Behaviour changes
null.Testing
./test: 62 examples, 0 failures. Newspec/reconcile_spec.cr, plus new cases in the control system and zone router specs.e2e/local). The specs change systems the way an admin would and require the broker's retained topics to match exactly what the system's modules, names, zones and redis state imply. They cover reorder, removal from a system, module delete, system delete, rename, moving a system to another level and a cleared status. All fail againstplaceos-2.2610.1and pass with this branch (15/15 on repeated runs).Not covered
A status set with a TTL expires out of redis without an event, so the broker keeps its last value.
🤖 Generated with Claude Code