Skip to content

chore(profiles): derive the discovery rows from the OWNd profile (prep for OWNd #142) - #808

Draft
gdluck wants to merge 2 commits into
OpenWebNet-HA:v2-phase1-architecturefrom
gdluck:chore/profiles-companion-ownd-142
Draft

gdluck wants to merge 2 commits into
OpenWebNet-HA:v2-phase1-architecturefrom
gdluck:chore/profiles-companion-ownd-142

Conversation

@gdluck

@gdluck gdluck commented Oct 24, 2026 •

Copy link
Copy Markdown

Requested by A

Draft: the MyHOME companion of OpenWebNet-HA/OWNd#142 (fix(profiles): drop WHO 16 from the F461 and MyHomeServer1 profiles). Sequencing: merge this → merge OpenWebNet-HA/OWNd#142 → OWNd release → bump the OWNd pin here, and scripts/sync_documentation.py --update regenerates the two docs rows together with the pinned_whos fallback below (one edit in scripts/update_gateway_profiles.py).

Review follow-up (CI red on the first push)

  • Cause: the quality-scale, test-coverage and docs jobs run scripts/verify_ha_standards.py / scripts/sync_documentation.py --check without OWNd installed ([OWND] Client library verification skipped (OWNd not in path: No module named 'OWNd') in the same log). The first push removed the static queried_whos and printed unknown (OWNd profile not available) for every row when OWNd is not importable, so the regenerated table differed from the committed one: [RULE_DOCS_ANTI_DRIFT] Gateway Discovery table out of sync in docs/getting-started/hardware-compatibility.md. Locally the check had only been run with OWNd on PYTHONPATH.
  • Fix: each discovery row carries pinned_whos, the row the pinned OWNd (OWNd==2.0.0b11 in manifest.json) yields; it is used only when OWNd is not importable, so the generated table is byte-identical with and without OWNd. With OWNd importable the row is still derived from the installed profile. Reproduced and fixed locally: sync_documentation.py --check now passes both in a venv without OWNd (pyyaml + ruff only, as in the CI jobs) and with OWNd master on PYTHONPATH; tests/test_gateway_profiles.py::test_discovery_table_without_ownd_matches_the_committed_docs pins the fallback to the committed table and ::test_discovery_table_with_ownd_matches_the_committed_docs pins the derived table to it, so installed profile == committed table == pinned_whos is asserted explicitly.
  • Oracle double-check (the maintainer refreshed own-firmware-oracle main to 72759e8, fix(oracle): write a past date in who13-clock so the DIM 1 verdict does not depend on the day own-firmware-oracle#61 and fix(results): refresh 7 stale sound-who22 TSVs and MCP index (#58) own-firmware-oracle#62): the WHO 16 verdicts for the F461 020011 and the MyHomeServer1 028206 are unchanged, see "What does the Firmware Oracle / Encyclopedia say" below.
  • Also: verify_profiles_against_oracle() now flags a discovery row whose profile_name resolves to OWNd's generic profile (get_gateway_profile() never returns None, so the earlier is None check could not fire).

What is the problem

scripts/update_gateway_profiles.py keeps a static queried_whos list per startup discovery row next to the list it derives from the installed OWNd profile, and verify_profiles_against_oracle() compares the two. The derived list is what the generated docs/getting-started/hardware-compatibility.md table shows and what the runtime sends (the startup gate in gateway.py asks profile.supports_who(who) for every discovery WHO; #794 reads the same flag for the refused requests). When an OWNd profile changes, the static copy goes stale: with OpenWebNet-HA/OWNd#142 the F461 and MyHomeServer1 rows become [1, 2, 4, 18], the oracle check reports Discovery row **F461** queried_whos=[1, 2, 4, 16, 18] disagrees with F461 profile [1, 2, 4, 18], sync_documentation.py --check fails and tests/test_gateway_profiles.py asserts the old literal row. Hard-coding the new rows instead would be red against the pinned OWNd 2.0.0b11 today.

Refs OpenWebNet-HA/OWNd#142; #794 (the runtime gate); #424 / #459 (the F461 / MyHomeServer1 plants that speak WHO 22 only, and the MyHomeServer1 "firmware discards 5, 16, 22" note). The combined test build (combined/all-prs-2026-10-24, commit 4a79e8cb) showed which files and tests are involved; its hard-coded rows are not used here.


What is the solution

  1. scripts/update_gateway_profiles.py:
    • GATEWAY_DISCOVERY_METADATA rows carry model, profile_name, pinned_whos (the no-OWNd fallback, see below) and notes; no queried_whos.
    • New discovery_whos_for(profile): [w for w in STARTUP_DISCOVERY_WHOS if profile.supports_who(w)], the rule MyHOMEGatewayHandler._profile_supports_who applies at startup; [] for anything without supports_who.
    • calibrate_discovery_with_ownd() fills queried_whos from the installed profile for every row; the F461 and MyHomeServer1 rows carry a notes_without_who16 text that replaces notes when the row does not carry WHO 16 ("Sound is WHO 22 only (no bt_difson; every WHO 16 frame refused)"), so the wording follows the profile too.
    • Without an importable OWNd (the quality-scale, test-coverage and docs CI jobs) every row falls back to pinned_whos, the row the pinned OWNd 2.0.0b11 yields, so the generated table is the same either way; nothing else is guessed. At a pin bump that changes a profile, pinned_whos is edited together with sync_documentation.py --update (the tests below fail until both are done).
    • verify_profiles_against_oracle() no longer compares a static list with the profile; it checks that every discovery row names a real OWNd profile (not the generic fallback), and the per-gateway supports_who / audio_measured / auth_measured checks against firmware_oracle.json are unchanged.
  2. tests/test_gateway_profiles.py: the F461 and MyHomeServer1 cells are asserted against the installed profile (right before and after fix(profiles): drop WHO 16 from the F461 and MyHomeServer1 profiles OWNd#142), the other rows stay literal; discovery_whos_for mirrors the runtime gate for six profiles; a profile double without WHO 16 yields [1, 2, 4, 18] and the WHO 22 wording for both rows while the F454 row is unchanged; without OWNd every row carries exactly its pinned_whos and the generated block equals the committed one; with OWNd the derived block equals the committed one; pinned_whos is ignored when OWNd is importable.
  3. docs/getting-started/hardware-compatibility.md: unchanged in this PR (against the pinned OWNd the generated table is byte-identical). After the pin bump the F461 and MyHomeServer1 rows regenerate to WHO 1, 2, 4, 18 with the WHO 22 note.
  4. .github/release-notes/2.0.0b16.md: one line under "Other changes".

Not changed here, named as follow-ups: the per-zone *#16*ZONE*5## media-player poll (media_player.py) on a gateway whose profile lacks WHO 16 (one refused frame per configured WHO 16 zone; those entities cannot work on these gateways anyway, and the planned WHO 22 backend is the right gate); and the topology side effect of OpenWebNet-HA/OWNd#142 itself (topology.py delegation: an F461 primary lets a WHO-16-capable follower own WHO 16 and 22, and a same-tier F454 / F461 pair is now ordered by WHO count rather than by MAC), which is OpenWebNet-HA/OWNd#142's to document and which this PR does not alter.


Companion PR: OpenWebNet-HA/OWNd#142 (fix(profiles): drop WHO 16 from the F461 and MyHomeServer1 profiles, draft). MyHOME #794 is the runtime half (startup requests gated on profile.supports_who, with a MyHOME fallback table until the OWNd release).


State Diagram

stateDiagram-v2
    direction LR
    state "Is OWNd importable" as Check
    state "Installed OWNd profile" as Profile {
        [*] --> Pinned : 2.0.0b11 - F461 and MyHomeServer1 supported_who includes 16
        [*] --> AfterPinBump : OWNd release with 142 - no 16
    }
    state "Discovery row - docs table, same rule as the startup gate" as Row {
        Pinned --> Row16 : discovery_whos_for gives 1 2 4 16 18 - notes sound unmeasured class default
        AfterPinBump --> Row18 : discovery_whos_for gives 1 2 4 18 - notes_without_who16 Sound is WHO 22 only
    }
    state "No OWNd - quality-scale, test-coverage and docs CI jobs" as Fallback {
        [*] --> PinnedRows : pinned_whos per row - the pinned OWNd rows, edited at the pin bump with update
        PinnedRows --> Row16 : today, identical to the committed table
    }
    [*] --> Check
    Check --> Profile : yes
    Check --> Fallback : no
Loading

Hardware Verification


How is it tested

  • tests/test_gateway_profiles.py::test_generate_gateway_discovery_table_and_verify_oracle (updated), ::test_discovery_whos_follow_the_runtime_gate, ::test_discovery_row_drops_who16_when_the_profile_does, ::test_discovery_table_without_ownd_uses_the_pinned_rows, ::test_discovery_table_without_ownd_matches_the_committed_docs, ::test_discovery_table_with_ownd_matches_the_committed_docs, ::test_pinned_rows_are_ignored_when_ownd_is_importable (new); 18 tests in the file.
  • Fail-before / pass-after: on the base script 4 of the first-push tests fail (no discovery_whos_for, rows hard-code their WHOs); on the first-push script (CI red) 3 fail: the table test (rows carry no pinned_whos), the no-OWNd rows test and the no-OWNd-equals-committed-docs test (unknown (OWNd profile not available) cells). All 18 pass after.
  • scripts/sync_documentation.py --check and scripts/verify_ha_standards.py: pass without OWNd (a venv with pyyaml + ruff only, the CI jobs' shape; before the fix: Gateway Discovery table out of sync) and with OWNd master (f04d8aa) on PYTHONPATH.
  • Full suite against OWNd master (f04d8aa, the behaviour of the 2.0.0b11 pin), rebased on v2-phase1-architecture 31ea9fd: 4514 passed, 1 skipped (first push on ddc6d60: 4421 passed, 1 skipped); scripts/verify_ownd_coverage.py 100 % (16,080 / 16,080 statements, 0 missing).
  • Against the fix(profiles): drop WHO 16 from the F461 and MyHomeServer1 profiles OWNd#142 branch (gdluck/OWNd fix/profiles-f461-mhs1-no-who16, 95dc8a5) on PYTHONPATH: tests/test_gateway_profiles.py + tests/test_firmware_oracle_conformance.py + tests/test_trace_oracle_validation.py + tests/test_golden_conformance.py: 711 passed, 1 skipped before the two docs-equality tests were added; verify_profiles_against_oracle() returns no violation, and the generator yields | **F461** | WHO 1, 2, 4, 18 | Lighting, cover, climate, and energy meter 51 queries. Sound is WHO 22 only (no bt_difson; every WHO 16 frame refused), alarm query omitted. | and the same for the MyHomeServer1. As intended, sync_documentation.py --check, ::test_discovery_table_with_ownd_matches_the_committed_docs and tests/test_sync_documentation.py::test_check_all_documentation_real_tree then report the docs drift that the pin bump resolves (--update + pinned_whos).
  • scripts/typing_ratchet.py 0 errors (baseline 0), ruff check ., scripts/verify_ha_standards.py, scripts/manage_translations.py check pass.
  • Note for the sequencing: pypi_standards.yml installs OWNd from git (OWND_REF, the branch named like the MyHOME branch, else master), so once fix(profiles): drop WHO 16 from the F461 and MyHomeServer1 profiles OWNd#142 is on OWNd master that job reports the same docs drift on v2-phase1-architecture until the pin bump lands; this was already the case before this PR (the generator derived the rows from the installed profile when importable).

Live retest (2026-10-24)

Not exercised live: docs generator and tests only, no bus effect of its own (the gate it documents was confirmed for this startup configuration, above).


Where are the traces

Subsystem Capture Plant Frames verified by this PR
Docs generator No physical trace applies: this change touches no bus frame (configuration / diagnostics / validation only) — none
WHO 16 refused by the gateway (the fact OpenWebNet-HA/OWNd#142 encodes and this table will show) tests/fixtures/traces/f461_live_2026-10-23/ run run1_oracle_replay_readonly step oracle_who16; tests/fixtures/traces/myhomeserver1_live_2026-10-24/ run run2_reads_thermo_sound_alarm_lights step read_who16 (trace PR: #799, merged); upstream tests/fixtures/traces/issue_466/ F461 2.0.11; MyHomeServer1 3.87.13; MyHomeServer1 (#466) *#16*0*5## NACK 5 ms / 3 ms, *#16*101*5## NACK 6 ms / 3 ms (note: who16_refused_locally)

What does the Firmware Oracle / Encyclopedia say

Re-checked against the refreshed results/mcp_index.json at own-firmware-oracle main 72759e8 (schema 1.2.0, 529 inputs; after OpenWebNet-HA/own-firmware-oracle#61 and OpenWebNet-HA/own-firmware-oracle#62): the F461 020011 and the MyHomeServer1 028206 answer nack / silent to all 86 WHO 16 inputs (77 in sound-who16-source, 9 in trace-corpus), *#16*0*5## at sound-who16-source.tsv line 16 and *#16*101*5## at line 28 on both, no bus frame, no emitted frame; identical to the 53f9514 rows mirrored in tests/golden/firmware_oracle.json (schema 1.1.0, 509 inputs). The refresh added the 20 who13-clock inputs and four sound-who22 rows per gateway and changed nine MH202 lights-area0 / interface-routing / trace-corpus replies (nack → ack); nothing about WHO 16 changed. The golden mirror is left as is: it is refreshed by hand in its own commit together with tests/test_firmware_oracle_conformance.py (which pins schema_version to 1.0.0 / 1.1.0) and tests/test_trace_oracle_validation.py (which cites 53f9514), not something this PR should fold in.

Published rows (results/<P>/<V>/oracle/full/sound-who16-source.tsv, index 53f9514 and 72759e8 alike), *#16*0*5##: nack / silent on F450 020010, F459 020105, F460 020012, F461 020011, MH202 010024, MyHomeServer1 028206; ack / out (bus 24 30 34 42 35 30 30 38 30 38 46 0d) on F453 010100, F453AV 030014, F454 020051, H4684 020054, MH200 020102, MH200N 010108. OpenWebNet-HA/OWNd#142 drops WHO 16 from the F461 and MyHomeServer1 profiles on the strength of all 86 WHO 16 inputs being refused there; it keeps the MH202 (a live MH202 relays WHO 16 state frames from an F441M matrix, MyHOME #669), so the generated MH202 row does not change either. This PR adds no verdict of its own: verify_profiles_against_oracle() keeps checking audio_measured against this row and supports_who against the WHO 1 / 2 / 3 / 4 / 25 probe rows, unchanged.


Checklist

🤖 Generated with Claude Code

@GreenGrassBlueOcean

Copy link
Copy Markdown
Contributor

@gdluck i double check the firmware oracle and made a fix for the stale who16 please double check with the oracle and fix failing ci

…p for OWNd OpenWebNet-HA#142)

scripts/update_gateway_profiles.py carried a static `queried_whos` list per
discovery row next to the derived one, and the oracle check compared the two,
so an OWNd profile change (OWNd OpenWebNet-HA#142 drops WHO 16 from the F461 and the
MyHomeServer1) turned the check and the docs test red until the list was edited
by hand. The rows now come only from the installed OWNd profile over
STARTUP_DISCOVERY_WHOS, the rule the startup gate in gateway.py applies, and
the sound wording of a row follows `supports_who(16)`. Green against the pinned
OWNd today; after OWNd OpenWebNet-HA#142 lands and the pin is bumped,
`sync_documentation.py --update` regenerates the two rows and nothing else
needs an edit. Tests pin both shapes with a profile double.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gdluck
gdluck force-pushed the chore/profiles-companion-ownd-142 branch from b06248b to 09dd4b5 Compare October 24, 2026 21:10
…tical with and without OWNd (CI follow-up)

The quality-scale, test-coverage and docs CI jobs run scripts/verify_ha_standards.py and
scripts/sync_documentation.py --check without OWNd installed ("[OWND] Client library
verification skipped (OWNd not in path: No module named 'OWNd')"). The first push derived
the startup discovery rows from the installed OWNd profile only and printed
"unknown (OWNd profile not available)" otherwise, so those jobs regenerated a table that
differed from the committed docs/getting-started/hardware-compatibility.md:
"[RULE_DOCS_ANTI_DRIFT] Gateway Discovery table out of sync".

Each discovery row now carries pinned_whos, the row the pinned OWNd 2.0.0b11 yields; it is
used only when OWNd is not importable, so the generated table is byte-identical with and
without OWNd. With OWNd importable the row is still derived from profile.supports_who over
STARTUP_DISCOVERY_WHOS, as the startup gate does, and notes_without_who16 applies whenever
the row does not carry WHO 16. verify_profiles_against_oracle() flags a row whose
profile_name resolves to the generic profile (get_gateway_profile() never returns None).

Tests: without OWNd every row carries exactly its pinned_whos and the generated block equals
the committed one; with OWNd the derived block equals the committed one (installed profile ==
committed table == pinned_whos); pinned_whos is ignored when OWNd is importable. Fail-before
on the first-push script: 3 of 18 fail; all pass after. sync_documentation.py --check passes
in a venv without OWNd and with OWNd master (f04d8aa) on PYTHONPATH. Against the OWNd OpenWebNet-HA#142
branch (95dc8a5) the two docs-equality tests report the drift that the pin bump resolves
with --update together with pinned_whos.

Oracle re-check at own-firmware-oracle main 72759e8 (results/mcp_index.json, schema 1.2.0,
529 inputs): F461 020011 and MyHomeServer1 028206 still answer nack/silent to all 86 WHO 16
inputs (*OpenWebNet-HA#16*0*5## sound-who16-source.tsv line 16, *OpenWebNet-HA#16*101*5## line 28), unchanged from
53f9514.

Refs OpenWebNet-HA/OWNd#142.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@gdluck

gdluck commented Oct 24, 2026

Copy link
Copy Markdown
Author

Checked with the refreshed oracle and fixed the CI (head 88ed3fa, rebased on the current base, still draft).

CI cause and fix. The quality-scale, test-coverage and docs jobs run the docs check without OWNd installed; this PR's first push derived the discovery rows from the installed profile and, with no OWNd, printed "unknown" rows, so the regenerated table no longer matched docs/getting-started/hardware-compatibility.md. Each discovery row now carries pinned_whos (the rows the pinned OWNd 2.0.0b11 yields: F461 / MyHomeServer1 still WHO 1, 2, 4, 16, 18), used only when OWNd is not importable; with OWNd the row is still derived from profile.supports_who. Tests pin installed-profile table == committed table == pinned_whos, and verify_profiles_against_oracle() now flags a profile name that resolves to the Generic profile (the previous is None check was vacuous). Both shapes pass sync_documentation.py --check locally (with and without OWNd).

Oracle re-check (main 72759e8, after #62). F461 020011 and MyHomeServer1 028206 still answer nack / silent to all 86 WHO 16 inputs (77 in sound-who16-source, 9 in trace-corpus; *#16*0*5## line 16, *#16*101*5## line 28), no bus frames: nothing changed for WHO 16, so OpenWebNet-HA/OWNd#142 stands. The refresh added the who13-clock inputs, 4 sound-who22 rows per gateway and flipped 9 MH202 replies in other suites. tests/golden/firmware_oracle.json (1.1.0 / 509) was not synced here: it is refreshed by hand in its own commit together with the conformance test (which pins the schema version), so that belongs in a separate PR if you want 72759e8 in.

Suite 4514 passed at 100 % coverage against the pinned OWNd; against the OWNd #142 branch the profile, conformance and trace-oracle tests pass (711) and only the docs-equality checks report the expected pin-bump drift, resolved by --update at that time. CI on 88ed3fa was still queued when I wrote this.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@GreenGrassBlueOcean

Copy link
Copy Markdown
Contributor

@gdluck please rebase

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants