diff --git a/examples/message-codegen/EVIDENCE.md b/examples/message-codegen/EVIDENCE.md index 743c025b824..53564bcc168 100644 --- a/examples/message-codegen/EVIDENCE.md +++ b/examples/message-codegen/EVIDENCE.md @@ -129,3 +129,1184 @@ sign-in in the isolated browser; no credentials were entered. Its decoding libraries pass independently, but that does not establish application UI acceptance. Rerun's direct-file screen capture also needs final confirmation of pose visualization and complete semantic playback before task 3.5 is closed. + +## Runtime transports and browser cutover (2026-09-21) + +The runtime work is still in progress. This evidence covers typed transports, +web schema decoding, and the migrated built-in web adapters; it does not establish +that all DimOS consumers or native SDKs have completed the cutover. + +- `demo_pubsub.py`: generated `LineSegments3D` and 921,600-byte `Image` exchanged + over raw LCM, separate loopback TCP Zenoh sessions, and CPU shared memory. + Decoded bytes and integer nanosecond timestamps matched on every path. +- Typed pub/sub pattern, core CDR transport, and CLI checks: 67 passed. A glob + subscription regression was fixed: `/sensor/*` must match the logical topic + before the slash-separated `#package/msg/Type` suffix. +- WebRTC plus core transport checks: 34 passed. Explicit data-frame channel/type + routing distinguishes `Point` and `Vector3` despite their identical CDR layouts. + Truncated, trailing, raw/unframed, foreign-type, foreign-channel, and wrong-codec + frames are dropped; the next valid frame is delivered. +- Python web codecs: 48 passed. Cockpit authoring plus generated array/helper + checks: 94 passed. Costmap and map codec checks: 17 passed. +- Browser SDK: 322 passed. Cockpit: 154 passed. Both TypeScript checks passed. + The CDR decoder uses pinned Foxglove parser/serialization libraries. Fixtures + cover standard and custom messages in both byte orders, raw images, fixed and + dynamic arrays, empty messages, Unicode, and integers beyond JavaScript's safe + number range. Former LCM codecs and their browser fixtures were removed. +- Chromium opened `demo_cdr.html` and displayed the generated custom message's + endpoints, weight 4, and `stamp.nanosec = 500000123`. Switching to the image + and big-endian CDR displayed both pixels. Reading the rendered canvas yielded + RGBA `[0, 1, 2, 255, 253, 254, 255, 255]` as expected. + Screenshots: ignored `build/message-codegen/viewers/cdr-browser-custom.png` + and `cdr-browser-image.png`. Browser recording: + `/tmp/dimos-cdr-browser-harness/agent-workspace/recordings/session-20260921-192504` + (6 frames). The task-owned browser and Vite server were stopped afterward. + +The combined Python runtime/web subset passed 318 tests before adding the explicit +WebRTC sequence regression; that WebRTC file then passed all 29 tests. The final +SDK and cockpit runs remained 322 and 154 tests, with both type checks passing. +Mypy passed on 12 changed runtime/helper/codec modules. The external packaging +relay and installed typing check also passed after making the bundled generator +a regular package so an installed DimOS cannot shadow it. + +The subsequent relay consumer migration passed the entire `dimos/web` suite +plus the generated image/grid helper tests: **677 passed in 35.59 s**. This +includes forkserver blueprint deployment, real relay session/transport tests, +image and map delivery, cached-map replay after unsubscribing, relay respawn, +teleoperation stop/watchdog behavior, and lease handover. The watchdog's old +`Twist.zero()` call was replaced with the generated zero-valued `Twist()`. +The new array-to-image helper also passed strided input, independent copy, +explicit encoding, and big-endian depth tests. + +### Native SDK CDR relay + +The C++ SDK suite passed all **76 tests** using real generated messages in its +default-codec tests. The Rust SDK passed **143 tests**, including TF, both CDR +byte orders, malformed input, and propagation of publish encoding errors without +enqueuing a payload. C++ and Rust ping/pong examples and the custom/image relays +built successfully. The generated Rust crate also passed Cargo source-package +verification: a build from the packaged archive, with only Python's standard +library used for generation and no preexisting generated source. + +The native demo used the real SDK processes, generated message packages, stdin +launch protocol, and Python typed transports: + +```text +lcm: Python weight=4 → C++ weight=5 → Rust weight=6 → Python verified +lcm: 921,600 image bytes and source nanoseconds=1700000000123456789 match +zenoh: Python weight=4 → C++ weight=5 → Rust weight=6 → Python verified +zenoh: 921,600 image bytes and source nanoseconds=1700000000123456789 match +``` + +Full output and process logs are under `build/message-codegen/demo/evidence`. +This does not complete the native consumer, Nix build matrix, coordinator +blueprint, or live Rerun migration; stage 4 remains open. + +### Lidar and mapping message consumers + +After switching their generated fields and codecs, the Livox Rust suite passed +24 tests, voxel mapping passed 60, and the MLS planner passed 94. The planner's +new regression checks that graph endpoints and cost survive CDR in explicit +`LineSegments3D` fields. The SDK suite passed 146 tests after extracting the shared +point-cloud reader, including padded organized rows, mixed float32/float64 XYZ, +both byte orders, malformed layouts, and nonfinite-coordinate filtering. + +The Python structured point-cloud helper passed 21 tests; together with the +Livox configuration tests, the focused Python run passed 26. The helper and +three native Python module declarations passed mypy with the generated stubs. +`demo_pointcloud.py` displayed both byte orders with organized shape `(2, 1)`, +row/point strides `(32, 20)`, and preserved two-value tags. Editing its explicit +copy left the borrowed source unchanged. Output is recorded in +`build/message-codegen/demo/evidence/pointcloud-layouts.txt`. + +The Livox live loopback E2E also passed against the released CDR binaries +(`test_live_loopback_handshake_and_stream`, 3.03 s). It exercised the virtual +Mid-360 handshake, actual UDP sensor traffic, native CDR publication, Python +schema decoding, a 5 m point ring, per-point deskew offsets, and gravity IMU. +The physical-sensor recording test remains deselected; no robot was required. + +### Remaining lidar and camera native adapters + +Point-LIO's Rust suite passed 7 tests after moving to generated CDR. Its point +reader now preserves deskew offsets for padded organized rows in either byte +order. RealSense's Rust suite passed 3 tests, including the generated `ImuInfo` +round trip that replaces its handwritten fingerprint and LCM encoder; its Python +configuration suite also passed 3. The hardware-free RGBD benchmark built and +encoded 407,040, 101,760, and 45,280 points at pixel strides 1, 2, and 3. +These are synthetic conversion checks, not a physical-camera acceptance test. + +The dimSLAM Rust adapter passed 7 tests, covering generated odometry/cloud +conversion, exact and negative timestamps, depth byte-order normalization with +row padding, BGR-to-RGB conversion, and invalid layouts/backend dimensions. +The Python camera/lidar declarations and geometry helper passed mypy. +The new odometry-to-TF helper passed zero, negative, and nanosecond-precise source +timestamp cases and preserves the message's declared parent/child frames. + +The C++ SDK suite passed 77 tests including the production lidar cloud builder. +Its standalone `dimos_lidar_cdr_demo` visibly decoded two XYZI points in each CDR +byte order, preserving frame `lidar`, timestamp `1s + 250000000ns`, and intensity +`0.5`. Output is in `build/message-codegen/demo/evidence/lidar-cpp.txt`. + +Both full C++ lidar Nix builds completed successfully with the generated CMake +package: Point-LIO and FAST-LIO. Their flake inputs/locks and CMake configurations +no longer fetch `dimos-lcm` headers. The generated package itself built from the +pinned source definitions in Nix. The root Rust workspace passed `cargo check +--workspace --locked`; the independently built RealSense and dimSLAM crates +passed the tests reported above. + +Clippy passed with warnings denied for the SDK and migrated root-workspace +modules, and independently for RealSense and dimSLAM. Generated explicit Default +implementations retain a targeted lint annotation: the same emitter supports +nonzero `.msg` defaults and fixed arrays longer than 32 elements. + +### Independently generated messages in the native SDKs + +The external package relay was rebuilt after aligning generated Rust codec errors +with SDK port signatures. Both the installed file relay and the actual native +SDK relay passed. The external application changed only its local `.msg` copy; +its Rust ports used the external type's generated methods directly: + +```text +lcm: external Python → native C++ → native Rust → Python: added-locally/cpp-native/rust-native +zenoh: external Python → native C++ → native Rust → Python: added-locally/cpp-native/rust-native +SDK codecs accepted the external generated type directly; no handwritten codec or upstream schema PR. +``` + +All other fields and the exact timestamp matched after both native processes. +The external Rust consumer test also verified SDK-compatible function signatures, +`InvalidInput` on bounded-field encode failure, and `InvalidData` on truncated +CDR. The built-in custom-message/image SDK relay still passed on both transports, +and the Rust SDK suite still passed all 146 tests. Mypy passed for both runtime +demo entry points using their separately generated type stubs. + +### TF through the real module coordinator + +The generated-message Python/Rust TF example now runs through +`ModuleCoordinator.build`, Python worker deployment, native process startup, +and typed stream wiring. Both LCM and Zenoh checks passed, each collecting four +composed `a → d` transforms with `x=1.5`, `y² + z²=1`, and changing positions. +Both runs returned normally after interruption and stopped their native workers. +The test logs are captured in pytest's temporary directory; the test summary is +`build/message-codegen/rust-tf-blueprint-tests.log`. + +This verifies coordinator wiring beyond the direct native SDK relay. It does not +complete the three-language blueprint with live viewers required by task 4.10. + +### Raw LCM fragment validation + +The retained Rust raw transport passed 18 unit tests and its compile doctest after +adding bounded reassembly. Checks cover reordered/duplicate fragments, conflicting +metadata, invalid offsets/counts, overlapping or missing bytes, invalid channels, +expiry, aggregate payload limits, and continuing reception after malformed UDP +packets. Clippy passed with warnings denied. + +The Python ↔ Rust raw transport demo still exchanged 128, 65,536, and 1,048,576 +bytes exactly. After rebuilding the Rust native examples, the three-language SDK +relay passed on LCM and Zenoh again, preserving 921,600 image bytes and timestamp +`1700000000123456789`. Transport limits are documented in the raw crate README. + +### Three-language coordinator blueprint + +`demo_blueprint.py` now deploys the Python producer/verifier and C++/Rust relays +through the actual module coordinator. Its parametrized E2E test passed on both +LCM and Zenoh, checking three distinct samples each. Two additional standalone +Zenoh launches each verified five samples. Every sample changes its custom +segment fields, 921,600 RGB image bytes, and exact source nanoseconds; Python +compares the complete returned messages against the expected native edits. +Mypy and Ruff passed for the demo. + +The Zenoh demo uses a temporary loopback router and explicit client endpoints. +Initial runs using default multicast peer discovery intermittently returned +neither stream despite successful native startup. Explicit routing passed the +repeated launches, but this does not establish the cause or fix default peer +discovery. Keep that finding open for the final runtime checks. Logs are +`build/message-codegen/three-language-blueprint-{tests,zenoh,zenoh-repeat}.log`. + +This completes the three-language coordinator portion of the human demo; +live Rerun/browser image and pose integration is still required for task 4.10. + +### Generated Python TF and recorded-stream lookup + +The live TF buffer and `StreamTF` now use generated `TransformStamped` and +`TFMessage` values. Composition/inversion/Euler conversion live in geometry +helpers backed by the existing SciPy dependency, outside generated classes. +The time index keys transforms by integer nanoseconds, snapshots incoming data, +and returns independent lookup values. + +The migrated TF suite and geometry helper suite passed 68 checks together: +live/in-memory/SQLite lookup parity, composition/inversion, nearest-time and +retention behavior, exact/negative timestamps, distinct adjacent nanoseconds, +copy independence, and invalid rotations/frames. Mypy passed on the TF buffer, +recorded-stream lookup, geometry helpers, and native TF demo. The coordinator +TF check passed on both LCM and Zenoh, now verifying Python and Rust composition +of the same chain. + +Correction to the earlier quaternion diagnosis: the pinned Jazzy +`geometry_msgs/msg/Quaternion.msg` defines `w=1`; the generated default is an +identity rotation. A zero quaternion must be constructed explicitly to test +rejection. The initial missing TF replies were not evidence of a zero default. +Robot-specific publishers and other legacy convenience-class consumers remain +to be updated; this is not completion of task 4.2 or 4.6. + +### Static publishers and robot mount callers + +`StaticTfPublisher` and its frame-tree helper now use generated stamped transforms +and exact `time.time_ns()` stamps. The Go2/Mid-360, Alfred URDF, RealSense, and +Mid-360/RealSense mount declarations use those values. Go2's Zenoh mount tree and +odometry-to-TF callback were converted as well; its camera/video consumers still +need their separate runtime migration. + +Spot's URDF mount composition and optical-frame rotation now use the geometry +helpers. Its SDK odometry boundary publishes generated Odometry and a matching +TFMessage, and replay's TF/odometry declarations use the generated types. A +synthetic SDK-value test checks negative source time, pose, velocity, frame names, +and CDR round trips without importing or connecting to the robot SDK. + +Eleven mount/helper/Spot boundary checks passed. The cumulative TF demo now +includes the production periodic `StaticTfPublisher` for `b → c`; Python publishes +`a → b`, Rust publishes `c → d`, and both language buffers compose `a → d`. +Both LCM and Zenoh E2E checks passed, including shutdown. This is hardware-free +message/geometry verification, not physical robot or camera acceptance. + +### Camera calibration and webcam cutover + +Calibration construction/YAML loading now returns generated CameraInfo values. +The Go2 YAML helper was moved out of the connection module so the Zenoh-side +module no longer imports the robot SDK just to obtain calibration. Go2's periodic +calibration publisher uses integer nanoseconds and leaves its stored template +unchanged. The broader Go2 connection still has legacy image/cloud/pose paths; +its two generated perception-interface type mismatches remain to be resolved by +that consumer migration. + +The webcam and CameraModule use generated Image, CameraInfo, and TFMessage values. +Stereo crops preserve RGB pixels and exact source headers. Metadata uses a shared +stamp, declared frame names, and independent copies of configured mounts. Image +sharpness is now an explicit helper used by the existing quality-barrier operator. + +Forty-four camera/helper/view tests passed, plus the focused Go2 publication test. +Checks cover the actual 1280x720 equidistant calibration, CDR round trips, invalid +matrix dimensions/intrinsics/FOV, synthetic BGR capture and stereo crops, metadata +frame/stamp consistency, and sharpness ordering across visual encodings. Mypy +passed on eight changed helper/camera/calibration/demo modules. Physical capture +and the complete Go2 stack were not run. + +`demo_message_helpers.py` visibly prints the decoded calibration matrix and four +distortion coefficients with source timestamp `1700000000123456789`; output is +`build/message-codegen/demo/evidence/message-helpers.txt`. + +### Go2 sensor values and simulation lidar IPC + +The WebRTC video/lidar conversion now constructs generated Image/PointCloud2 +values directly. Hardware-free tests exercise the actual reactive conversion +using synthetic driver events, checking RGB pixels, XYZ coordinates, declared +frames, exact host-arrival stamps, and CDR round trips. The new XYZ factory +preserves organized shape, copies source storage, and rejects finite coordinates +that cannot fit float32. Missing points remain NaN/infinity with `is_dense=False`. + +MuJoCo video/calibration construction and its lidar shared-memory boundary use +generated messages. Lidar IPC now carries CDR instead of pickle. Odd/even sequence +checks discard an observed in-progress or changed write before decoding. The +standalone `demo_mujoco_lidar_shm.py` passed with separate producer/consumer +processes and printed two expected XYZ points and timestamp +`1700000000123456789`. No engine or robot was launched. + +The focused cloud, WebRTC conversion, connection lifecycle, and shared-memory +suite passed **51 tests**, with one hardware-recording test deselected. Log: +`build/message-codegen/go2-sensor-cutover-tests.log`. Mypy passed on eight changed +production modules. Pytest ran with `-o addopts=''` because this local environment +lacks the configured xdist/timeout/coverage plugins. Repository conftest emitted +its existing subprocess ResourceWarning; the standalone IPC demo exited cleanly. + +This resolves the Go2 image/cloud interface type mismatches recorded above. +Go2/WebRTC/MuJoCo pose and TF callers still need their generated-value cutover; +DimSim's external browser producer and complete live simulation remain unverified. + +### Go2 and simulation pose/TF values + +WebRTC odometry and TF streams now return generated PoseStamped and +TransformStamped. The raw converter preserves the device header; the live +connection explicitly replaces it with a host-arrival header, preserving the +existing timing policy without a float timestamp round trip. Go2's publication +copies the received pose before overriding its configured parent frame. Its +mount chain and TFMessage retain that pose's exact stamp. + +MuJoCo's pose boundary maps simulator WXYZ quaternions into generated XYZW fields. +DimSim's TF builder and G1 simulation ports/mounts also use generated values. G1's +camera metadata is restamped per publication without mutating the template. +The Unitree legacy Odometry subclass still serves unconverted recording-fixture +callers; it has no new fallback role and remains scheduled for deletion. + +The focused geometry, device conversion, Go2/WebRTC, MuJoCo lifecycle/pose, and +simulation TF suites passed **43 tests**. Simulator startup dependencies are +stubbed in lifecycle/pose tests; shared-memory odometry uses real regions. No +engine or robot was run. Eight changed production modules passed mypy. +Logs: `build/message-codegen/unitree-pose-tests.log` and +`build/message-codegen/unitree-simulation-tf-tests.log`. + +The human-facing `demo_robot_tf.py` passed and printed the decoded namespaced +camera chain with source timestamp `1746565669448350564`. Captured output: +`build/message-codegen/demo/evidence/robot-tf.txt`. + +The broader Go2 blueprint topology test could not collect because PyTorch is +absent from this environment. Its generated-pose fixture has been updated, but +that test is not counted as passing. Full blueprint/runtime acceptance remains +open, as do recorder and legacy-fixture cutovers. + +The cumulative Python/Rust coordinator TF demo was rerun on both LCM and Zenoh: +**2 E2E tests passed** in 8.68 seconds, including changing transform lookups and +clean worker shutdown. Log: `build/message-codegen/unitree-pose-tf-e2e.log`. + +### Python storage codecs and generated-message replay + +Generated messages now select CDR instead of falling through to pickle. SQLite +persists/reconstructs `cdr` or `lz4+cdr`; old `lcm`, `lz4+lcm`, and private `jpeg` +codec IDs are rejected. The obsolete Python LCM and JPEG storage codecs are +removed. Raw/depth Image values remain lossless; compressed images are explicit +CompressedImage values stored through the same CDR codec. + +The generic MCAP reader resolves installed types from `cdr`/`ros2msg` schema names. +Unknown schemas remain raw bytes, and Python payload-module metadata does not +trigger imports. Duplicate compatible channels on a topic aggregate counts; +conflicting schemas/timing policies and stream-name collisions are rejected. +Generic dataset opening now uses McapStore, leaving the robot-specific DDS preset +available for callers that explicitly choose it. + +TransportRecorder now accepts generated-message contracts and reads source time +from Header. A zero source stamp is preserved; unstamped values use receipt time. +Observation pose conversion accepts generated nested PoseStamped/TransformStamped +and returns generated values. Replay ports use the recorded generated class +directly; the old rich-subclass-to-base-type substitution is removed. + +**47 focused tests passed**, covering exact values/bytes after SQLite reopening, +raw/depth/compressed images, codec reconstruction and obsolete-ID rejection, +MCAP schema discovery/time ordering, pose metadata, recorder timestamps, and real +module replay. Log: `build/message-codegen/storage-cdr-tests.log`. +Tests ran with `--noconftest` because the memory conftest eagerly imports optional +CLIP/PyTorch dependencies. Two broader checks were deselected after their imports +failed: Rust-recorder delegation needs PyTorch, and the full Rerun blueprint needs +Numba. They are not counted as passing. Seven changed production modules passed +mypy. Checking the demo's full import graph additionally hits the installed MCAP +writer's `from .__init__ import __version__`, which mypy reports as duplicate +module names; the demo itself was executed successfully. + +`demo_storage_replay.py` passed: SQLite and MCAP both returned weights `[4, 5, 6]`, +and module ports replayed stamps `1700000000123456789` through +`1700000000123456791`. Output is +`build/message-codegen/demo/evidence/storage-replay.txt`. Temporary files and module +resources were cleaned up. The Rust recorder and remaining legacy fixtures still +need conversion; this does not close stage 4 or stage 5 acceptance. + +### Native recorder configuration dependency audit + +The shared OnExisting policy moved to `dimos.memory.recording_policy`; its callers +now import it there. Native recorder configuration no longer imports the Python +embedding module just to obtain this enum. The previously blocked Rust-session +delegation test passes without PyTorch, and the policy/native configuration +modules pass mypy. + +Running both native-recorder Python configuration suites now reaches their real +storage behavior: **23 passed, 9 failed**. Every failure requests the obsolete +`lcm` storage codec. This is an outstanding coordinated Python/Rust recorder +cutover, not an accepted regression result. Log: +`build/message-codegen/rust-recorder-config-audit.log`. + +### Native recorder CDR and external schemas + +The coordinated native recorder cutover resolves the nine obsolete-codec failures +above: **32 Python configuration checks, 16 Rust unit tests, five native process +E2E tests, and one external-package recording E2E test passed**. Production Python +configuration modules pass mypy; the Rust recorder passes Clippy with warnings +denied. Root workspace `cargo check --workspace --locked --offline` passes. + +The recorder now receives complete generated schemas, writes ROS2-profile MCAP +with `cdr` channels and embedded `ros2msg` definitions, and uses chunk compression. +SQLite accepts `cdr` and `lz4+cdr`; obsolete LCM/JPEG wrappers are removed. Raw +image pixels remain exact. The root Cargo lockfile no longer depends on the +external `dimos-lcm` message repository; the in-tree raw LCM transport remains. + +Recognized stamped messages retain integer source nanoseconds, including the E2E +MCAP value `1700000000123456789`. Zero stamps remain zero; invalid nanosecond +fields are rejected. MCAP rejects negative timestamps instead of silently +clamping them. SQLite converts timestamps to seconds only at its existing query +API boundary. Known ordinary message payloads preserve their original CDR bytes, +including byte order; TF arrays split into generated single-transform TFMessages. + +`demo_native_recording.py` records three changing external Telemetry messages on +each transport with the same Rust binary. The locally added `application_note` +field survives along with every payload timestamp. The unknown custom timestamp +layout uses reception time for MCAP indexing. Foxglove's independent decoder +reads all six samples from embedded schemas without the custom package installed +in its environment. This is decoder evidence, not Foxglove/Rerun UI acceptance. + +Review artifacts and transcripts: + +- `build/message-codegen/demo/evidence/external-native-recording-{lcm,zenoh}.mcap` + and matching `.log` files. +- `build/message-codegen/demo/evidence/native-recording.txt` and + `native-recording-foxglove.txt`. +- `build/message-codegen/rust-recorder-cdr-tests.log`, + `rust-recorder-config-cdr-tests.log`, `rust-recorder-cdr-e2e.log`, and + `external-native-recording-e2e.log`. +- `build/message-codegen/rust-recorder-cdr-clippy.log` and + `recorder-workspace-check.log`. + +The Python recorder checks use `--noconftest` to avoid unrelated optional +CLIP/PyTorch imports. This leaves an unregistered `self_hosted` marker warning. +The host filesystem filled during the first workspace/Nix build attempts; those +attempts failed and are not counted as passing. Removing only this worktree's +reproducible compiler caches allowed the workspace check and executable build +to pass. The isolated Nix retry then passed, including all 16 Rust tests and +installation/fixup. An offline Nix build confirmed the completed output. The +installed Nix executable also passed the external custom-message demo on both +transports; its transcript is +`build/message-codegen/demo/evidence/nix-native-recording.txt`, with MCAP/log +artifacts in the adjacent `nix-native-recording/` directory. Full stage 4/5 +acceptance, legacy consumers/fixtures, and live viewer demos remain outstanding. + +### Optional ROS bridge uses CDR directly + +Removed the LCM intermediate types, recursive field-copy conversion, renamed +time fields, complex-type allowlist, and old type-import caches. The bridge +resolves canonical `package/msg/Type` names and uses installed ROS serialization +with generated DimOS encode/decode. Mismatched message identities are rejected +even when their wire layouts match. Importing the conversion module remains +possible without ROS; actually requesting conversion reports the missing +optional dependency. + +**25 checks passed** against the installed Jazzy type support in the isolated +reference container. Coverage includes seven real pub/sub cases, nested poses, +empty/nonempty TF arrays, raw/compressed image bytes, padded big-endian point +cloud data, camera calibration arrays, Unicode frames, exact epoch/zero/negative +stamps, invalid names, and wrong-type rejection. On the ROS-free host, four +contract checks pass and 21 ROS-dependent checks skip. The container's minimal +pytest reports three unknown optional-plugin configuration warnings. + +`demo_ros_bridge.py` passed with three changing image/pose pairs on actual ROS +topics. Transcript: `build/message-codegen/demo/evidence/ros-bridge.txt`. The +local existing reference image needed `typing_extensions.py` mounted from the +host virtualenv; the updated reference Dockerfile explicitly installs that +dependency and pytest. The CI job now downloads the standalone Python binding +and runs the same bridge checks and demo; this is workflow wiring, not a claim +that the remote job has passed. + +Galaxea depth/lidar/IMU and compressed-image callers now use generated sensor +types. Compressed images preserve their source format and header rather than +forcing JPEG and replacing zero timestamps with wall time. Other Galaxea +geometry/control consumers still need conversion; robot hardware was not used. +Mypy passes on the converter, Galaxea connection, and demo. The full runtime +cutover remains incomplete. + +### Galaxea generated control and feedback messages + +The R1Pro connection now uses generated MotorCommandArray, JointState, Twist, +PoseStamped, Odometry, and TFMessage alongside the previously converted sensor +streams. No legacy message imports remain in this connection module. Commands +use generated arrays directly and require 18 entries in every declared array +before publishing. ROS command conversion uses the CDR bridge; unsupported +base axes remain zero. + +Wheel integration computes delta time from integer source nanoseconds and +publishes nested ROS2 pose/twist fields. Pose, odometry, and both TF edges retain +the same exact stamp. Joint-state aggregation retains the oldest segment stamp, +including zero, without converting through floating-point seconds or replacing +it with wall time. The publisher's interval wait now responds to its stop event. + +**13 focused checks passed**, covering integration and clock jumps, exact +zero/epoch joint stamps, all 18 joint values, three command segments, tracking +velocity sentinels, and rejection of each malformed command array. Tests use +real generated messages and mock only the ROS/output boundaries. Mypy passes on +the connection and demo. Log: `build/message-codegen/galaxea-cdr-tests.log`. +The test launcher emitted a subprocess ResourceWarning; the reported process +was no longer live when checked after the run. + +`demo_galaxea_messages.py` passed using the actual module output subscriptions. +It printed base x coordinates `0.100000`, `0.199980`, and `0.299900`, the complete +lidar TF chain, and exact stamps `1700000000223456789` through +`1700000000423456789`. Transcript: +`build/message-codegen/demo/evidence/galaxea-messages.txt`. No robot hardware or +ROS was required. This validates the connection's generated-message boundary; +remaining downstream whole-body/navigation consumers and hardware acceptance +are not claimed complete. + +### Whole-body adapter generated streams + +TransportWholeBodyAdapter now declares generated JointState, Imu, and +MotorCommandArray types. Commands receive an explicit wall-clock header. Five +focused tests pass for disconnected behavior, preservation of complete feedback +when a short frame arrives, and XYZW-to-WXYZ IMU conversion. The adapter and demo +pass mypy. + +The hardware-free demo passed on LCM and independent loopback Zenoh sessions, +exchanging two-joint feedback, IMU, and full hybrid motor commands. Transcript: +`build/message-codegen/demo/evidence/whole-body-transport.txt`. The automatic +subprocess E2E check exercises the same assertions and cleanup. The initial demo +Zenoh factory incorrectly passed a leading slash to the raw transport; it now +uses the same `dimos/` topic mapping as the production transport factory. +No production topic compatibility path was added. + +### G1 and shared coordinator joint-state cutover + +G1WholeBodyConnection now uses generated MotorCommandArray, JointState, and Imu +messages. It constructs one explicit integer-nanosecond arrival header for each +feedback sample, shared by the joint and IMU messages. Unitree WXYZ is explicitly +copied to ROS XYZW. Every command array must contain all 29 joints before any +DDS command is written. Existing damping-first soft start, mode-machine, and +CRC behavior remain covered by the command tests. + +The shared coordinator, tick-loop outputs, velocity/trajectory consumers, and +IK task joint states now use generated JointState. IK command snapshots use +explicit deep copies; generated sequence names are compared as lists. Internal +coordinator snapshot timestamps remain floating seconds, converted explicitly +to Header.stamp at publication. G1/GROOT and Galaxea coordinator transport maps +and declared joint ports now reference the same generated types. Other legacy +pose/trajectory consumers remain outstanding. + +**12 G1 tests and all 369 tests in `dimos/control` passed.** The narrower 138-test +routing/joint-state suite and 60-test IK suite passed before broadening to the +full directory. Pinocchio 4.1.0, pin-pink 4.3.0, and QP dependencies were installed +from the local cache; ONNX Runtime 1.24.1 was installed to collect the GROOT task +tests. Mypy passes on 12 changed production/blueprint/demo modules. + +Both human demos passed: G1 printed 29-joint samples with q0 `0.0`, `0.1`, `0.2` +and exact stamps `1700000000123456789` through `1700000000123456791`; the running +mock coordinator printed left `[0.1, 0.2]`, right `[0.3, 0.4]`, and the merged +four-joint state with matching tick stamps. Transcripts: + +- `build/message-codegen/demo/evidence/g1-messages.txt` +- `build/message-codegen/demo/evidence/coordinator-joints.txt` +- `build/message-codegen/g1-wholebody-cdr-tests.log` +- `build/message-codegen/control-all-cdr-tests.log` + +These checks exercise synthetic hardware boundaries and real coordinator ticks; +no physical G1 control or complete robot blueprint acceptance is claimed. + +### Generated base-motion adapter and coordinator commands + +TransportTwistAdapter now uses generated Twist/PoseStamped and reads nested +position/quaternion fields through the geometry yaw helper. The coordinator and +GROOT command contract, plus Go2/GROOT transport maps, use generated Twist. +Existing enable, stop, one/two/three-axis mapping, and odometry copy-isolation +behavior remain covered. + +**373 checks passed** (four base-adapter cases and all 369 control tests), plus +one subprocess E2E that runs the base demo on both LCM and independent Zenoh +sessions. Mypy passes on six changed production/blueprint/demo modules. +The demo received odometry `[2.0, -1.0, pi/2]`, observed the generated velocity +command `(0.5, -0.2, 0.3)`, and observed a zero Twist after disable. + +Logs: `build/message-codegen/base-control-cdr-tests.log`, +`base-transport-cdr-e2e.log`, and +`build/message-codegen/demo/evidence/base-transport.txt`. No hardware motion was +performed. Legacy benchmark/trajectory/pose consumers outside this adapter path +remain pending in the broader runtime cutover. + +### Manipulation planning consumes generated JointState + +Planning-group projection/target normalization, joint-space validation, planning +protocol/model declarations, and RobotStateMonitor now use generated JointState. +Projection and normalization explicitly copy the source Header instead of +inventing a new timestamp. RobotStateMonitor preserves that header when syncing +its reordered state into the world and invoking callbacks; arrival-time freshness +tracking remains separate. Generated sequence data is converted explicitly at +the NumPy joint-space boundary. + +**47 tests passed** across planning groups, planning specs/model validation, and +the state monitor. Added checks cover negative/epoch source stamps, header copy +isolation, reordered positions/velocities, incomplete feedback, and stopped +monitor behavior. Six production/demo modules pass mypy. The yourdfpy 0.0.60 +dependency was installed to run model-validation tests rather than omit them. + +The hardware-free planning demo passed: generated CDR feedback selected left-arm +positions `[-0.2, 0.3]`, normalized target `[0.1, 0.5]`, validated joint limits, +and retained source nanoseconds `1700000000123456789`. Logs: +`build/message-codegen/planning-joints-cdr-tests.log` and +`build/message-codegen/demo/evidence/planning-joints.txt`. + +Planner backend implementations, manipulation module entry points, and the +interactive manipulation visualizer still have legacy message consumers. These +checks do not claim the full manipulation stack is converted. + + +### Planner backend JointState and PointCloud2 self-filter cutover + +All planning backend JointState imports now use generated values, including +Drake/Jacobian/Pink IK, RRT and RoboPlan, world synchronization, path utilities, +and trajectory parametrizer inputs. Dictionary constructors became explicit +keyword construction; the parametrizer copies generated inputs with deepcopy. +Array/list conversions are explicit at numerical and sequence API boundaries. +Pose and trajectory message types remain part of the pending cutover. + +The robot self-filter now consumes generated PointCloud2 and TransformStamped. +A separate selection helper retains complete point records, including arbitrary +fields, field counts, point padding, and byte order. It removes row padding when +forming an unorganized result and preserves the source Header. Clear masks copy +the source stamp into a world-frame header. Transform matrices come from an +explicit geometry helper. + +**253 tests passed with no skips** across the complete planning subtree and the +geometry/point-cloud helper suites. Installed pinned RoboPlan 0.6.0 and Drake +1.49.0 allowed actual backend tests to run. Added selection tests cover both byte +orders, custom array fields, point padding, header preservation, copy isolation, +empty output, and invalid masks. Nineteen production/demo modules pass mypy. + +Both human-facing terminal demos passed: `demo_planning_backends.py` printed five +CDR-decoded, collision-checked joint waypoints in a real Drake world; +`demo_self_filter.py` printed three changing arm positions, one retained point +per capture, and the original nanosecond stamp on both outputs. Logs: +`build/message-codegen/planner-backends-cdr-tests.log`, +`planner-backends-mypy.log`, and +`build/message-codegen/demo/evidence/{planning-backends,self-filter}.txt`. + +These local results do not claim that manipulation entry points, legacy poses +and trajectories, interactive visualizers, or the full repository are converted. + + +### Generated trajectories from planning through execution + +Trajectory generators, parametrizers, previews, execution splitting, joint and +planar-base tasks, coordinator RPC contracts, and the legacy threaded controller +now use generated ROS2 JointTrajectory and JointTrajectoryPoint values. Waypoint +times are Duration fields; validation compares normalized integer nanoseconds, +including adjacent nanoseconds beyond floating-point resolution. Interpolation +and duration calculation live in `dimos.msgs.trajectory`. Position-only joint +commands are accepted with zero velocity feed-forward. Joint/base splitting +preserves all optional point fields and the original Header. Anchoring a running +trajectory also preserves its metadata and optional fields. + +Execution feedback now uses generated `dimos_msgs/msg/TrajectoryStatus` with +Duration elapsed/remaining fields and a wall-clock Header. The internal enum +references generated wire constants, and wire state comparisons use equality. +The three handwritten trajectory codecs were deleted, with no legacy imports +remaining in Python source. The typed test spy uses the generated decode API and +coordinator tests use canonical schema names. + +The remaining manipulation JointState consumers (module, SDK, operator, Viser, +and controllers) now use generated values, explicit copies, and list conversion +at sequence API boundaries. Model-state reordering preserves the incoming Header. +Snapshot reprs still expose joint values for interactive inspection. + +Validation: + +- **968 tests passed** across all manipulation and control tests, trajectory + helper tests, the transport demo subprocess test, and LeRobot's host module + contract tests. No skips. This includes real Drake/RoboPlan and Viser tests. +- **Three Galaxea planar-preview tests passed**, including moving its mock base + through generated trajectory execution. +- **Five process-level coordinator E2E tests passed on Zenoh and the same five on + LCM**: RPC startup, execution, published feedback, cancellation, and dual-arm + execution. No coordinator or pytest processes remained afterward. +- **26 focused checks passed** after the final header-preservation and demo edits. +- **28 production/demo/test-support modules passed mypy.** +- The isolated LeRobot runtime's joint/trajectory inputs and corresponding tests + were updated and syntax-checked. Its full runtime suite is still pending; the + separate torch/lerobot environment is not installed here. Its legacy image API + also remains part of the broader image-consumer cutover. + +`demo_trajectory.py` passed on LCM and independent loopback Zenoh sessions. It +transmitted 51 generated points, preserved source stamp +`1700000000123456789`, printed changing commands from `[0, 0]` to `[0.3, -0.2]`, +and received generated COMPLETED status with zero remaining duration. The demo's +synthetic clock exercises the actual task without moving hardware. + +Logs: `build/message-codegen/trajectory-cdr-tests.log`, +`trajectory-cdr-mypy.log`, `trajectory-planar-preview-tests.log`, +`trajectory-blueprint-tests.log` (Zenoh), `trajectory-blueprint-lcm-tests.log`, +`trajectory-final-focused-tests.log`, and +`build/message-codegen/demo/evidence/trajectory.txt`. + +Local setup added the pinned optional xacro, Viser, MuJoCo, websocket-client, and +pytest plugins needed for the existing suites. These results do not establish +full-repository CI or completion of the remaining pose/image/viewer cutover. + +### Generated pose, grasp, and path consumers (runtime cutover in progress) + +Manipulation planning, FK/IK, preview, dispatch, the SDK, and Viser now use +nested generated Pose/PoseStamped values. Matrix/Euler/local-offset operations +live outside the generated classes. Robot state snapshots retain the source +Header atomically; FK-derived poses and their TF edges retain exact source +nanoseconds. Static mount publication copies its configured transform before +setting the publication time. + +GraspCandidate and GraspCandidateArray now come from the generated DimOS schema +package. The two handwritten pickle codecs are removed. The heuristic and +isolated GraspGenX adapter accept generated clouds, preserve their source +headers, and return generated ranked proposals. Pick-and-place validates finite +scores and valid pose geometry before issuing a gripper or motion command. +The GraspGenX tests use a mocked inference backend, not a loaded GPU model. + +Base path followers, benchmarking, and coordinator scalar/stamped-twist commands +use generated messages. SVG and Rerun scene rendering reads generated points, +poses, paths, detection boxes, and occupancy grids. The occupancy origin rotation +is applied in both renderers. Existing occupancy borrowing and block reduction +helpers remain in use; renderers validate physical resolution separately. + +Human-facing demonstrations: + +- `demo_planning_backends.py`: real Drake/RRT, five FK tool poses and CDR TF edges; + the first source stamp is `1700000000.123456789`. +- `demo_grasp_proposals.py`: a four-point CDR cloud becomes a CDR ranked grasp at + `[0.5, 0, 0.15]`, with downward approach and unchanged nanoseconds. +- `demo_path_following.py`: 41 generated path waypoints and CDR Twist commands + drive the existing base simulator to `arrived` in 38 ticks. The SVG in + `build/message-codegen/demo/evidence/path-following.svg` overlays reference + and executed paths. + +Validation logs are under `build/message-codegen/`: `pose-combined-tests.log`, +`pose-space-tests.log`, `pose-combined-mypy.log`, `pose-blueprint-lcm-tests.log`, +and `pose-blueprint-zenoh-tests.log`. The combined manipulation, control, geometry, grasp, object, and transform suite +passed 1080 tests. Both coordinator E2E runs passed all five tests. The +renderer/occupancy/view suite passed 61 tests, including actual +headless Rerun SDK logging and rotated-grid geometry. + +This checkpoint is not an accepted stage-4 completion. The broad type check of +127 modules still reports four errors at legacy point-cloud boundaries: the +older grasp provider, the scene-to-pick cloud interface, and cloud-derived +occupancy rendering. Their producers must be converted; no compatibility reader +or fallback was added. The temporal object-registration suite also fails +collection because the optional Hydra/segmentation environment is absent. +Focused generated-object CDR and spatial-deduplication tests pass. Full viewer UI +acceptance and the remaining inventory retain their unchecked OpenSpec tasks. + +### Generated cloud and occupancy renderers (runtime cutover in progress) + +Cloud-to-map algorithms, inflation, gradient/Voronoi maps, obstacle operations, +and navigation-map composition now use generated PointCloud2/OccupancyGrid values. +Map images and path footprints use generated Image/Path values. Space accepts +CDR point clouds directly in both SVG and Rerun. Packed rgb/rgba conversion handles +FLOAT32 color bits, UINT32, both byte orders, and padded rows. + +- `occupancy-renderer-tests.log`: 92 generated mapping, color, view, and renderer + checks passed, including real headless Rerun SDK output and rotated-grid paths. +- `occupancy-fixtures-tests.log`: 10 existing image/XML fixture checks passed. + Three gradient image fixtures were regenerated for the ROS float32 resolution: + 0.05 is stored as 0.05000000074505806. A direct comparison reproduced the old + fixture exactly with float64 0.05 and identified 718 changed pixels with the + declared float32 value. Seven other fixture comparisons passed unchanged. +- `occupancy-cutover-mypy.log`: all 15 checked production/demo modules passed. +- `demo_occupancy.py`: 1,200 CDR points, 44 occupied cells before inflation and + 208 afterward; exact source timestamp 1700000000123456789 ns retained. SVG + output is `build/message-codegen/demo/evidence/occupancy.svg`. + +These checks do not complete the broader navigation cutover. A* and path +resampling/masking callers and their fixtures still require conversion, as do +remaining perception cloud/image producers. The demo SVG was generated and +validated programmatically; human visual review is deferred. + +### Generated A* and path processing (runtime cutover in progress) + +A* consumes generated occupancy buffers in both Python and the compiled C++ +extension and returns generated Path/PoseStamped values with the source header. +Resampling and path masks use nested ROS fields. Coordinate helpers apply the +full grid-origin transform. Sub-nanocell rotation roundoff no longer assigns an +exact cell corner to the preceding cell, and negative out-of-map positions are +rejected. Resampling writes pose edits back into generated sequences and merges +a numerically duplicated final point rather than reversing the last heading. + +- `path-all-tests.log`: 38 checks passed, covering the complete occupancy test + directory, recorded-map A*, native/Python parity, generated CDR paths, rotated + grids, mask exclusion, source-header retention, and input isolation. +- Mypy passed on the four changed production modules and the demo. +- The native A* extension was compiled locally; generated-pipeline tests assert + its availability before exercising the native branch. +- Six path-image fixtures were regenerated. On both recorded input maps, all + 161/107 generated A* waypoints exactly matched the old implementation when + both were given the declared float32 resolution. Image expectations now also + reflect generated path coordinates and the corrected final-point handling. +- `demo_grid_path.py`: both planners produced 41 waypoints, resampled to 122 + CDR-decoded poses, with exact source headers and no occupied mask cells. + `grid-path-python.svg` and `grid-path-cpp.svg` are the human review artifacts. + +Goal validation, higher-level global/local planners, and navigation module +wiring still require conversion; this evidence does not accept stage 4. + +### Generated replanning navigation module (runtime cutover in progress) + +Goal validation, global/local replanning, navigation maps, clearance masks, +position tracking, retry limiting, and the module's stream/spec declarations now +use generated messages. Odometry and clicked points convert explicitly to nested +PoseStamped values. Robot-footprint clearing copies and replaces the read-only +grid buffer. Clearance masks invalidate when map metadata changes. + +- `navigation-tests.log`: 67 focused checks passed, including all replanning A*, + occupancy, and geometry tests plus the two-transport subprocess navigation E2E. +- `navigation-mypy.log`: 16 production/demo modules passed. +- The arrival regression initially failed because an aligned robot at the first + path point skipped directly to final rotation. It now checks the last path + point, and the regression requires physical displacement before arrival. +- Cancellation joins the local control thread and sends generated zero Twist; + its regression verifies the captured thread is stopped. +- Goal-validator fixture coordinates changed by one cell at the float32 map + resolution. Running the old validator with that resolution reproduced all + three new coordinates exactly; obstacle-search behavior is retained. +- `demo_navigation.py` passed on LCM and independent explicit-loopback Zenoh + sessions: each run reached (2.815, 1.000) for a (3, 1) goal within the configured + 0.2 m tolerance, with 38 commands and 21 published path poses. Source timestamp + 1700000000123456789 ns survived map processing and path publication. + `navigation-lcm.svg` and `navigation-zenoh.svg` are the review artifacts. + +This verifies the navigation module with synthetic transport inputs, not a full +robot blueprint. Other navigation implementations, map producers, and perception +consumers still need conversion before stage 4 can be accepted. + +### Generated voxel and cost-map producers (runtime cutover in progress) + +CostMapper and the voxel accumulator now consume generated PointCloud2. Both +voxel implementations read declared fields through the shared point-cloud +helper; obsolete wrapper/Open3D conversion functions are removed. Generated +voxel output preserves the latest source stamp exactly, including zero stamps, +and initial-safe-radius changes replace an independent occupancy array after +releasing the borrowed view. The safety disc respects the map-origin rotation. + +- `map-producer-combined-tests.log`: 88 checks passed across voxel/cost-map + producers, cloud-to-occupancy, replanning navigation, occupancy operations, + geometry, and the expanded two-transport subprocess E2E. +- `map-producer-mypy.log`: seven production/demo modules passed. +- Packed CPU and Open3D-on-CPU backends passed union/column-carving and missing + point checks. CUDA hardware behavior was not exercised. +- The cumulative `demo_navigation.py` now sends 576 source points through CDR, + accumulates 576 voxels, runs the real CostMapper module to produce a 62×62 + grid, and drives the navigation module to arrival on both LCM and independent + explicit-loopback Zenoh sessions. Exact source nanoseconds survive the pipeline. + Terminal output is in `map-navigation-demo.log`; existing navigation SVGs are + regenerated by the expanded demo. + +The VoxelGridMapper memory-stream wrapper's generated declarations are updated, +but runtime import remains unverified here: the existing memory module imports +optional PyTorch, which is absent. The old recorded voxel tests still depend on +LegacyPickleStore/Go2Moment fixtures; replacing those recordings remains part of +the full cutover. These limitations do not affect the verified direct voxel → +CostMapper → navigation transport demo, and do not complete stage 4. + +### Generated basic follower and odometry history (runtime cutover in progress) + +The basic path follower, public navigation protocol, and OdometryHist now use +generated messages. TF lookups retain the generated PoseStamped shape. History +uses integer nanoseconds for publication throttling and copies source headers +and nested poses into generated Path output. Its Rerun override constructs the +archetype directly, retaining the actual Z coordinate. + +- `history-combined-tests.log`: 97 checks passed across follower/history, map + producers, navigation, occupancy, geometry, and the two-transport module E2E. +- `history-mypy.log`: four production/demo modules passed. +- Nine focused follower/history tests cover TF mount composition and lookup + throttling, generated velocity/arrival/empty-path behavior, one-nanosecond + publication intervals at epoch time, copy isolation, bounded history, frame + overrides, replay reset to zero, and Rerun path coordinates. +- `demo_odometry_history.py`: fixed-step simulation reached x=1.700 within the + basic follower's 0.3 m tolerance, emitted zero Twist on arrival, and retained + 35 history poses stamped 1700000000123456789 through 1700000000123456823 ns. + The generated Rerun line retains z=0.25 m. The SVG review artifact is + `build/message-codegen/demo/evidence/odometry-history.svg`. + +Exploration and patrol consumers still require generated-message conversion. +This checkpoint does not complete the runtime or viewer acceptance gates. + +### Generated frontier exploration (runtime cutover in progress) + +Wavefront exploration now consumes generated OccupancyGrid/PoseStamped/Bool and +publishes generated goals. Coordinate conversion uses the full map origin; +frontier centroids retain the map plane's height. Stop goals preserve the +current odometry header. Perimeters smaller than one cell no longer admit +empty frontier groups. + +- `frontier-combined-tests.log`: 101 checks passed across frontier exploration, + follower/history, map producers, navigation, occupancy, geometry, and the + cumulative two-transport navigation E2E. +- `frontier-mypy.log`: the explorer and demo pass mypy. +- Four frontier regressions verify translated/rotated map centroids, original + pose headers on stop, copy isolation, and an actual exploration-loop goal + decoded from generated CDR with the map's exact frame and nanoseconds. +- `demo_frontiers.py`: selected (5.00, 4.75) m at the known/unknown boundary; + stamp 1700000000123456789 ns retained. The review artifact is + `build/message-codegen/demo/evidence/frontiers.svg`. + +Patrol and other remaining consumers still need conversion. This checkpoint +does not complete the full exploration-to-robot blueprint or stage 4 acceptance. + +### Generated patrol routing + +The patrol module and all three routers now consume generated occupancy maps and +nested poses. Coordinate conversion uses the shared map-origin helpers, and +selected goals retain the map frame and integer timestamp. The existing office +coverage fixture now constructs generated clouds and maps directly. + +- `build/message-codegen/patrol-tests.log`: **9 passed**, including six generated + CDR cases covering all routers with ordinary and rotated/translated maps, plus + all three existing office-map coverage tests. +- Mypy passed for 11 checked source files (patrol production code and demo). +- `build/message-codegen/patrol-demo.log`: all routers produced five CDR goals; + three reviewable SVGs were written under `demo/evidence/patrol-*.svg`. +- This verifies goal selection and coverage calculations. It does not claim + patrol lifecycle or hardware motion acceptance, or completion of stage 4. + +### Generated movement manager + +The click-to-goal relay and teleop/navigation velocity arbiter use generated +PointStamped, Twist, and Bool. Clicks retain their source header; cancellation +headers use integer wall-clock nanoseconds. Component scaling uses explicit +keyword fields. Existing cancellation behavior is unchanged. + +- `build/message-codegen/movement-patrol-tests.log`: **14 passed**, comprising + five movement tests with CDR-decoded outputs and nine patrol checks. +- Mypy passed for the movement module and its demo. +- `demo_movement.py` printed the preserved click timestamp, teleop suppression, + and navigation resumption and completed its assertions. + +### Generated 2D visual-servo controller + +VisualServoing2D uses generated CameraInfo's lowercase `k` intrinsic matrix and +returns generated Twist values. Five parameterized checks passed for forward, +stationary, reverse, turning, and invalid-width detections after CDR decoding. +The terminal `demo_visual_servo.py` passed all four visible scenario assertions; +mypy passed for controller and demo. Person-follow and security-demo callers, +and their detection/cloud pipelines, remain part of the unfinished coordinated +runtime cutover; these controller checks do not establish their acceptance. + +### Generated detection-cloud filters + +Height, statistical, radius, and visibility filters now consume generated +PointCloud2, CameraInfo, and TransformStamped. Open3D performs geometric +selection; original point records are selected by index so custom fields are +not discarded when reconstructing the output. Four checks passed, verifying +CDR round trips, exact source headers, field definitions, and byte-equivalent +selected records carrying custom point IDs. Mypy passed for filters and demo. +`demo_cloud_filters.py` passed both visible 101-to-100-point scenarios. +Detection3DPC and its higher-level consumers still require conversion; these +results establish filter behavior, not acceptance of the person-follow pipeline. + +### Generated 3D detection projection and target calculation + +Detection3DPC and its batch projection wrapper now accept generated depth, +clouds, camera calibration, and stamped transforms. Depth projection preserves +the depth stamp and target frame; cloud projection selects original records. +Centroid poses use the cloud header. Open3D bounding calculations are explicit. +DetectionNavigation now consumes these generated values and produces generated +Twist commands using the shared quaternion helper. + +Seven projection/filter checks passed, covering translated depth unprojection, +image/behind-camera rejection, bounding extents, robust target selection, CDR +velocity output, and field-preserving filters. Mypy passed for four production +modules. `demo_detection_projection.py` passed and printed the expected world +centroid (11.5, 1.5, 2.0) with stamp 1700000000123456789 ns. The 2D detection image +API and person-follow/security callers remain unfinished; no full pipeline or +model-inference acceptance is claimed. + +### Generated 2D bounding-box image and wire boundary + +Detection2DBBox now crops generated image views, annotates an independent BGR +copy, and emits generated vision_msgs/Detection2D with the original image header. +Class identifiers use the ROS string field on the wire. Eight combined generated +bbox/projection/filter checks passed. The bbox case checks crop pixels, RGB/BGR +conversion, source immutability, integer timestamp preservation, and decoded +identifiers. Mypy passed for bbox/base and the earlier projection demo. +`demo_detection_bbox.py` generated the annotated review PNG and passed its CDR +assertions. The model-backed existing bbox fixture was not run; its generated +image shape assertion was updated, and model/image fixture conversion remains +part of the remaining perception work. + +### Generated detection collections + +ImageDetections and ImageDetections2D use generated images and Detection2DArray; +array headers preserve the source image stamp and frame. Person/segmentation +constructors now derive their application-level floating timestamp from generated +headers. Shared `image_to_bgr` produces an independent drawing array and serves +both individual and collection annotation. The annotation demo now exercises the +collection's CDR array and rendered output. + +Nine combined generated detection/projection/filter checks passed. Mypy passed +for the shared image helper, collection base, and bbox. Broader checking remains +incomplete: Torch typing is unavailable and the person class has pre-existing +unparameterized NumPy annotations. Model-backed inference and person-follow +lifecycle acceptance are still outstanding. + +### Generated vision-provider and tracker image boundaries + +Person-follow streams, EdgeTAM preprocessing, visual queries, and Qwen/OpenAI +providers now use generated messages. Provider images use explicit JPEG encoding; +legacy ndarray query compatibility was removed. VLM resize and warmup paths use +explicit array helpers. Point detections preserve generated image headers. + +Sixteen combined generated vision/detection/projection/filter tests passed. +Provider tests intercept API calls and decode their JPEGs to verify dimensions, +colors, MIME type, and single/batch request structure. CPU tracker tests verify +normalization, temporary image cleanup, and mask conversion; model construction +is substituted, so no weights or inference are exercised. The offline +`demo_vl_images.py` passed. Mypy passed for person-follow and the three VLM modules. + +Local verification installed OpenAI 2.21.0, Hydra 1.3.2, and CPU Torch 2.7.1. +Person-follow imports successfully. Actual tracking inference, simulation camera +producers, and the full person-follow lifecycle remain unverified and unfinished. + +### Generated person-follow thread lifecycle + +Two standalone tests pass for the real control thread with deterministic tracker +output: successful detection produces motion then a final zero command, and +immediate tracking loss produces only zero commands. Both join the thread within +two seconds and clean up the module. Commands are decoded from CDR before +assertion. `demo_person_follow.py` passed the visible motion/stop scenario. + +The tests use `--noconftest` because the unrelated agent conftest requires missing +LangGraph. Model creation and tracker inference are substituted; neither actual +weights nor robot actuation are verified. Inspection confirms MuJoCo and DimSim +camera producers already use generated CameraInfo and Image; no producer change +was needed for this increment. Simulation execution remains a separate gate. +The standalone control-thread demo does not enter an MCP skill-call context; +its final tool-progress update logs an unknown-tool warning. MCP tool-stream +lifecycle is not part of this check. + +### Generated 3D bbox and marker messages + +3D bbox, marker, and array conversions now emit generated vision messages and +poses. Image headers supply exact timestamps. Marker hypotheses are assigned back +to the generated result sequence after editing, preserving dictionary-qualified +identity. Six converted existing/extended tests passed for bbox geometry, marker +identity after CDR, array behavior, annotation, exact stamps, and copy isolation. +Mypy passed for three production modules. `demo_marker_messages.py` passed its +visible two-marker CDR check. Upstream fiducial producers still need conversion; +these checks do not establish the full marker detection pipeline. + +### Generated fiducial frame detection + +Marker calibration, image conversion, Rodrigues pose conversion, and world-pose +composition now use generated messages and shared helpers. The stamped transform +constructor accepts a Header directly, retaining integer timestamps. Ten combined +marker geometry, actual rendered-marker detection, and wire/array checks passed. +Mypy passed for both production modules. The ArUco demo detected DICT_4X4_50:7 at +world x=2.000 m, z=0.402 m, retained the exact image stamp, and wrote an annotated +PNG; the rendered output was inspected. Fiducial stream modules/transformer and +hardware-camera integration remain to be converted or verified. + +### Generated fiducial observation stream + +DetectMarkers, MarkersPerFrame, and MarkerDetectionStreamModule now use generated +images, camera calibration, transforms, and arrays. The live input path preserves +zero image stamps instead of replacing them with wall-clock time. Shared SpeedLimit +now computes distance and angular separation with generated geometry helpers. + +Nineteen marker/stream checks passed, including intrinsics refresh during +smoothing, grouping/empty frames, quality and speed gating, quaternion sign +invariance, and zero timestamp lookup. Mypy passed for four changed production +modules. The ArUco demo now also runs image observation → marker observation → +CDR array and retains the exact image stamp. Shared fiducial test helpers were +converted; the separate MarkerTfModule tests and producer still need conversion. + +### Generated marker TF publication and desk blueprint + +MarkerTfModule publishes generated TFMessage/TransformStamped values directly +from generated detections, retaining exact array timestamps. The desk blueprint +uses standalone calibration loading and generated static transforms. Converted +fixture verification renders a twelve-marker board and checks its published TF +chain. All **38 fiducial tests passed**, including the existing LCM input +integration, namespace/identity cases, empty arrays, static TF thread, calibration +loading, and generated stream checks (`build/message-codegen/fiducial-tests.log`). +Mypy passed for the TF publisher and desk blueprint. No legacy message imports or +LCM codec calls remain in fiducial Python files. The cumulative ArUco demo now +prints CDR TF edges world → markers → marker_7 and passed. Physical camera +acquisition and actual robot use were not exercised. ReportLab 4.5.0 was installed +locally to run the existing generated-board tests. + +### Generated bbox navigation + +BBoxNavigationModule now consumes generated CameraInfo/Detection2DArray and +publishes generated PoseStamped with the exact detection header. Its existing +forward/left/up projection convention is unchanged. One integration test passed +with real LCM input channels, CDR output decoding, calibration gating, empty +arrays, and exact timestamps. Mypy passed for the production module. +`demo_bbox_navigation.py` passed and printed the expected projected goal. This +is a message/projection check, not physical navigation or TF-frame validation. + +Recent holonomic, MLS, ray-tracing, and evaluation results are in +[EVIDENCE-navigation.md](EVIDENCE-navigation.md). + + +### Offline replay, PGO cloud inputs, recorder and metadata consumers + +The generated-message batch converts B1 odometry forwarding, M20 video/camera/TF +metadata, offline replay and marker rendering, map reconstruction/CDR export, +pose-fill mounts, and PGO point-cloud input/submap/ICP boundaries. Existing +GTSAM and Open3D numerical algorithms remain; their internal numeric transforms +still require retirement review. CollectionRecorder now records generated image +and joint-state values and preserves the timestamp of application EpisodeStatus +messages instead of substituting wall-clock time. ObjectDB cleanup and convex +hull extraction use generated point buffers. Moondream fixtures use real bbox +values so generated Detection2DArray construction exercises the actual conversion. + +The local synthetic regression batch passed **51 tests**, covering geometry, +pose-fill rotation/unmatched frames, PGO body-to-world submap placement and merge, +CLI import cost, B1 forwarding, M20 metadata, and Rerun camera/cloud modes. Scoped +pre-commit hooks passed. Local perception/collection test collection is blocked +by missing hydra, torch and h5py; no extra model or dataset was downloaded. Native +GTSAM/Open3D full-pipeline validation remains dependent on CI. + +Before this batch, exact published commit +42c3f4411a4d19135952dc22df8cdee927dff235 passed message-codegen standalone and +Jazzy-reference jobs, C++/Rust native builds, Rust and Web CI. Main CI still failed: +lint reported 34 errors in 12 files; Python 3.12 reported 6267 passed, 82 skipped, +3 failed and 3 errors. Other matrix jobs were cancelled by fail-fast. The fixes +above address a subset of those failures; these previous results do not prove +this batch or the entire migration complete. + +WebXR control-path edits and full memory-document execution remain paused after +automatic approval rejection. Eight pending files are excluded from this batch. +No hardware, network configuration, or live-control acceptance test was performed. + + +### Generated relocalization and offline memory viewers + +The next batch completes runtime CDR premap loading (`.pc2.cdr`), generated TF +publication/placement, lidar scan-window merging, and Go2 premap/live-map merging. +Matching thresholds and native algorithms remain unchanged. Offline tuning and +map viewing now use generated point-cloud APIs. The existing synthetic map-view +case moved out of the self-hosted asset suite and writes an RRD in ordinary tests. +Go2 DDS offline odometry/IMU integration and path/camera/cloud rendering use +nested generated values; measured extrinsics retain the previous SciPy +normalization of the six-decimal rotation. + +Memory-store Rerun output dispatches through generated-message helpers outside +message classes and keeps explicit application adapters. Camera pairing and +first-TF timing are preserved. Space camera rendering and near queries now use +generated calibration/image/nested pose fields. Query radius boundary behavior +is unchanged, and prior path/image/cloud snapshots remain unmodified. + +**55 focused checks passed** in the batch: 29 relocalization/DDS/map-view/CLI and +camera/cloud bridge tests, 12 memory-render tests, 8 generated near-query cases, +2 actual Rerun scene/camera tests, and 4 PGO/codebase-constructor checks. The pure +memory tests were run with scoped confcutdir to avoid unrelated parent CLIP +fixtures; they do not establish full model-dependent memory behavior. Two SVG +cases remain locally blocked by missing numba. Strict scoped mypy passed for +15 production files. Native mesh/perception fixtures still require CI because +Open3D/hydra are absent locally. + +Previous exact published commit 31800fc679ecd5e904ca860ca49bd1cab511d191 passed +message-codegen, Native C++/Rust, Rust and Web CI. Main CI failed: lint had 5 errors +in 2 files; ARM had 6057 passed, 229 skipped and 3 failed. Other Python lanes were +cancelled by fail-fast. This batch repairs those five consumer type errors and +three identified fixture failures; the new exact-commit CI remains required. +md-babel still fails on old memory documentation codecs; the paused eight files +are excluded again. Historical `.pc2.lcm` map assets must be regenerated or +explicitly migrated before using the CDR loader; none was downloaded or rewritten. +Final legacy API/dependency retirement and viewer UI acceptance remain open. + +For the subsequent compressed-image/runtime adapter batch, see +[evidence/compressed-image-runtime.md](evidence/compressed-image-runtime.md). + +Recorder/graph boundaries and the isolated Python example are covered in +[evidence/recorder-graph-runtime.md](evidence/recorder-graph-runtime.md). + +[Unitree/Spot/camera runtime evidence](evidence/unitree-spot-camera-runtime.md) +covers the next consumer batch. + +[Camera/export, PGO, approved WebXR and memory documentation evidence](evidence/camera-export-runtime.md) +covers the subsequent offline consumer batch. + +[Hosted arm, robot and simulation consumer evidence](evidence/robot-simulation-consumers.md) +covers the next offline migration batch and its remaining gates. + +[Geometry, simulation, eval and experimental consumer evidence](evidence/experimental-consumers.md) +records terminal CI for the previous batch and standalone Python 3.9 encoding. + +## Remaining consumer cutover + +See [final-consumer-cutover.md](evidence/final-consumer-cutover.md) for generated +entity/control/PGO/eval migration, exact previous CI failures, offline evidence +and remaining legacy retirement boundaries. diff --git a/examples/message-codegen/README.md b/examples/message-codegen/README.md index dc7c1d37229..a10a5bf4eae 100644 --- a/examples/message-codegen/README.md +++ b/examples/message-codegen/README.md @@ -5,6 +5,9 @@ reading, a position, defaults, fixed and variable arrays, and a bounded string. This example generates Python, C++, and Rust from that definition without ROS. The native processes print the fields they receive and each adds its own hop. +See [Add and use a message](/docs/development/messages.md) for the new-type and +Python/C++/Rust user stories, including a bounded runnable local example. + ## Build dependencies Use Python 3.12+, a C++17 compiler, CMake 3.20+, and Rust 1.92+. Install Python @@ -225,3 +228,1098 @@ Stop the Rerun process with Ctrl-C and close its browser tab. The script starts no persistent services. Remove `build/message-codegen/viewers/` to clean up recordings and `examples/message-codegen/viewer-checker/node_modules/` to clean up the Node dependencies. + +## Runtime cutover foundations + +The generated set also includes the in-tree `dimos_msgs` definitions. To inspect +weighted segments and trajectory duration fields without a running robot: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_message_helpers.py +``` + +The source timestamp survives CDR at full nanosecond precision. Segment endpoints +and weights are explicit fields, and controller-facing floating seconds are an +explicit helper conversion from ROS Duration. The generated classes carry data +and codecs; helpers do not add methods or compatibility properties to them. +This is preparation for stage 4, whose runtime consumer cutover is still in +progress. + +The typed transport demo exercises DimOS's actual transport classes. It sends a +custom weighted-segment message and a 921,600-byte RGB image through LCM, two +separate Zenoh sessions connected by loopback TCP, and CPU shared memory: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_pubsub.py +``` + +Use `--backend lcm`, `--backend zenoh`, or `--backend shm` to select one path. +The demo chooses temporary ports and topics and closes its sessions, sockets, +threads, and shared-memory handles on exit. It prints the received custom field, +pixel count, and exact source timestamp. This is a transport demonstration; +the complete three-language blueprint and live viewer cutover remain stage-4 +acceptance work. LCM limits its physical channel name, including the type suffix, +to 63 UTF-8 bytes; overlong names fail explicitly before sending. + +### Browser schema decoding during the runtime cutover + +After building the generated Python module, regenerate the shared vectors: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python -m dimos.web.relay_bridge.gen_cdr_fixtures +cd web +deno install +cd sdk +deno task fixture --host 127.0.0.1 +``` + +Open `http://127.0.0.1:5174/demo_cdr.html`. Select `custom_segments` to inspect the +nested endpoints, weight, and exact timestamp. Select `image` to see the decoded +pixels, and switch byte order to verify both generated encodings. Each selection +uses the SDK's actual decoder registry and the complete schema from its manifest +record. The viewer has no generated message bindings. Stop Vite with Ctrl-C. +This is a fixture-based browser demonstration; live three-language blueprint +integration remains a separate acceptance gate. + +Automated checks for this part of the cutover: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/pytest \ + dimos/core/test_cdr_transport.py \ + dimos/protocol/pubsub/impl/webrtc/test_transport.py \ + dimos/web/test_cdr_codec.py dimos/web/test_codecs.py dimos/web/test_cockpit.py \ + dimos/web/relay_bridge/test_map_codecs.py \ + dimos/web/relay_bridge/test_costmap_encoding.py \ + dimos/msgs/test_generated_views.py -o addopts='' -q +cd web/sdk +deno task test +deno task check +cd ../cockpit +deno task test +deno task check +``` + +Generated messages keep their ROS fields; conversion operations live in helpers. +For example, image arrays require an explicit encoding and header: + +```python +from dimos.msgs.image import image_from_array, image_view +from dimos.msgs.time import header_now + +message = image_from_array(pixels, encoding="rgb8", header=header_now("camera")) +borrowed = image_view(message) # read-only, retains its owner; honors row padding and byte order +editable = borrowed.copy() # an explicit independent mutable copy +``` + +`image_from_array` copies the supplied pixels and validates their dtype and shape. +It accepts non-contiguous inputs. `occupancy_view` provides the corresponding +read-only grid view using `info.width` and `info.height`. Pose consumers access +`message.pose.position`; planar heading is `yaw(message.pose.orientation)` from +`dimos.msgs.geometry`. Creating a zero velocity command is simply generated +`Twist()`, including in watchdog and stop paths. + +### Native SDK relay during the runtime cutover + +With the C++ SDK's LCM, Zenoh C/C++, JSON, and PFR build dependencies installed, +build the generated package and native relays: + +```bash +bash scripts/setup_message_codegen.sh +bash scripts/install_native_messages.sh +cmake -S native/cpp -B build/native-cpp -DDIMOS_NATIVE_BUILD_TESTS=ON \ + -DCMAKE_PREFIX_PATH="$PWD/build/message-codegen/install" +cmake --build build/native-cpp -j 2 +ctest --test-dir build/native-cpp --output-on-failure +cmake -S examples/native-modules/cpp -B build/native-cpp-examples \ + -DCMAKE_PREFIX_PATH="$PWD/build/message-codegen/install" +cmake --build build/native-cpp-examples -j 2 +cargo build -p dimos-native-module-examples +cargo test -p dimos-module --lib +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_native.py +``` + +The generated Python extension comes from the initial generation/build demo. +The `native` job in `.github/workflows/ci.yml` lists the pinned Zenoh archives +and system dependencies. Add their installation prefix to `CMAKE_PREFIX_PATH` +and their pkg-config directory to `PKG_CONFIG_PATH` when installed locally. + +The demo starts actual SDK subprocesses, supplies their stdin launch configuration, +and sends a custom `LineSegments3D` and a 640×480 RGB image through +Python → C++ → Rust → Python, first over LCM and then over loopback Zenoh TCP. +The segment weight changes from 4 to 5 to 6. The full image and exact source +nanoseconds must survive both relays. LCM fragments the image along each hop. +Use `--backend lcm` or `--backend zenoh` for one transport. Temporary topics and +ports isolate runs; subprocesses and sessions close automatically, including on +failure. Native logs go to `build/message-codegen/demo/evidence/native-*.log`. +This exercises the SDK launch protocol; the full coordinator blueprint and live +Rerun integration are still separate acceptance work. + +The in-tree Rust message dependency generates only inside Cargo's `OUT_DIR`. +Its source package bundles the parser, generator, schemas, and licenses. Cargo +requires Python 3.10+ during the build (`DIMOS_CODEGEN_PYTHON` can select it); +the resulting Rust binaries need neither Python nor ROS. Verify the actual source +archive with `cargo package -p dimos-generated-messages --allow-dirty --offline`. + +### Point-cloud layouts + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_pointcloud.py +``` + +This displays an organized cloud with field padding, row padding, and a two-value +`tags` field, in both byte orders. The helper borrows a read-only structured view; +`pointcloud_xyz` explicitly copies XYZ coordinates. The demo edits a separate +copy and checks that the borrowed source remains unchanged. No processes or +viewer windows remain after it exits. + +### C++ lidar message construction + +The native C++ test build also produces a hardware-free demonstration of the +production lidar cloud builder: + +```bash +cmake --build build/native-cpp -j 2 +build/native-cpp/tests/dimos_lidar_cdr_demo +``` + +It prints the decoded XYZI point count, frame, timestamp, and intensity for both +CDR byte orders. It also verifies negative timestamp normalization and rejects +invalid timestamps and negative cloud sizes. The executable exits on its own. + +### External messages through native SDK ports + +After the external packaging demo and the native SDK dependency setup above: + +```bash +bash scripts/test_message_packages.sh +bash scripts/test_external_native_messages.sh +``` + +The first command adds `application_note` to a local copy of `Telemetry.msg` and +builds separate Python/CMake/Cargo packages. The second builds C++ and Rust SDK +modules against those packages and exchanges the extended message through +Python → C++ → Rust → Python over LCM and Zenoh. The displayed note becomes +`added-locally/cpp-native/rust-native`; every other field and the source timestamp +must remain unchanged. It uses the generated Rust `Telemetry::encode` and +`Telemetry::decode` directly in port declarations. No SDK-specific codec adapter +or change to the built-in definition registry is needed. + +Generated Rust codec entry points return `std::io::Result`: invalid values at +encode time report `InvalidInput`, while malformed CDR reports `InvalidData`. +The external consumer test checks both errors against the SDK's function types. +Subprocesses and sessions close on exit. Logs and visible results are under +`build/message-codegen/external-native/evidence`. + +### Python/Rust TF through the module coordinator + +Build the Rust examples, then run the TF blueprint using either backend: + +```bash +cargo build --release -p dimos-native-module-examples +DIMOS_TRANSPORT=zenoh PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/native-modules/rust_tf.py +# Repeat with DIMOS_TRANSPORT=lcm on a host configured for multicast. +``` + +The Python worker publishes `a → b → c` using generated `TransformStamped` +messages, and the Rust broadcaster publishes `c → d`. The Rust listener prints +`x=1.5` and changing `y=cos(t), z=sin(t)` for the composed `a → d` transform. +The pinned ROS quaternion definition defaults `w=1`; generated types preserve +that identity rotation, and this example also sets it explicitly. Stop with Ctrl-C; the coordinator +stops both native processes and its Python workers. + +The automated check runs the same blueprint, verifies four changing composed +transforms per backend, and checks clean shutdown. It requires the normal DimOS +runtime dependencies (including Open3D). Pytest disables interactive host system +configuration, and the LCM run reserves a temporary local UDP port: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/pytest \ + examples/native-modules/test_rust_tf.py -m native_e2e -o addopts='' -v +``` + +### Three-language module coordinator blueprint + +After building the native relays above, run the complete worker-based graph: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_blueprint.py --transport zenoh +# On a multicast-configured host: +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_blueprint.py --transport lcm +``` + +```text +Python worker → C++ native worker → Rust native worker → Python worker + weight n n + 1 n + 2 validate fields + RGB image ──────────────────────────────────────── validate all pixels +``` + +Each of five samples changes the custom weight, endpoint, image pixels, and exact +source timestamp. The final Python worker validates the full encoded messages; +the terminal reports each result. `--samples N` changes the sample count. Native +processes and workers stop automatically after the finite run, including failures. + +The Zenoh demo starts a temporary loopback router and connects every worker to +its explicit endpoint, with multicast discovery disabled. The topics and module +names have a unique namespace. The LCM run uses `LCM_DEFAULT_URL`, so set it to an +isolated multicast port when other local blueprints are running. As with the TF +demo, normal DimOS runtime dependencies are required. Automated checks choose a +temporary LCM port and run the same command on each transport: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/pytest \ + examples/message-codegen/test_blueprint.py -m native_e2e -o addopts='' -v +``` + +This finite demo exercises coordinator deployment, remapping, native startup, +and generated streams. Live Rerun/browser integration remains a later addition. + +### Generated TF values and geometry helpers + +The live TF buffer and recorded-stream lookup return generated +`geometry_msgs/msg/TransformStamped` values. Read the parent frame and timestamp +from `value.header`, the child from `value.child_frame_id`, and the translation +and quaternion from `value.transform`. The separate helpers in +`dimos.msgs.geometry` compose, invert, or convert those values: + +```python +from dimos.msgs.geometry import compose_transforms, inverse_transform, pose_from_transform + +world_to_tool = compose_transforms(world_to_arm, arm_to_tool) +tool_to_world = inverse_transform(world_to_tool) +world_pose = pose_from_transform(world_to_tool) +``` + +Composition requires matching intermediate frame names and preserves the first +edge's exact source stamp. Inversion swaps frames and preserves the stamp. The +TF time index uses integer nanoseconds and owns snapshots of received transforms; +mutating a received value or a lookup result cannot change buffered history. +Float-second lookup arguments remain for existing timing APIs. + +The Python/Rust TF demo above now logs the composed transform from both Python +and Rust. Its automated check verifies both results. Robot-specific publishers +and downstream consumers are being converted to these fields as part of the +remaining runtime cutover. + +The TF demo's fixed `b → c` edge is published by the production +`StaticTfPublisher` in its own worker. This exercises periodic generated +`TFMessage` publication alongside the dynamic Python and Rust edges. Static mount +implementations return `list[TransformStamped]`; `frames_to_edge_transforms` +constructs them from `(child, parent, xyz, fixed-axis-rpy)` entries. A `None` parent +marks the root and produces no edge. The publisher supplies current integer +nanosecond stamps on every cycle. + +### Camera calibration and webcam messages + +`dimos.msgs.camera_info` constructs generated `CameraInfo` values from pinhole +intrinsics, field of view, or a ROS calibration YAML file. Construction requires +an explicit `Header`; message defaults do not invent a capture time. YAML loading +checks the matrix dimensions and finite values. The generated fields are lowercase +`k`, `d`, `r`, and `p`. `intrinsic_matrix(info)` returns an independent mutable +NumPy copy. + +The existing helper demo now loads the packaged Go2 calibration, encodes and +decodes it with CDR, and prints its dimensions, distortion model, matrix, +coefficients, and exact source timestamp: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_message_helpers.py +``` + +The webcam path emits generated RGB images; stereo slicing happens on pixels +before message construction. Camera metadata and both TF edges share one exact +stamp without mutating the configured calibration or mount. Hardware-free tests +feed known BGR pixels into the capture boundary and verify the decoded RGB crops. + +### Simulation lidar shared memory (no engine required) + +Run the actual simulation lidar producer and consumer in separate processes: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_mujoco_lidar_shm.py +``` + +The demo prints decoded XYZ coordinates, frame `world`, and the exact timestamp +`1700000000123456789`. It writes generated PointCloud2 CDR bytes through the +production shared-memory buffer and cleans up its regions on exit. It does not +launch MuJoCo or require robot hardware. Live simulation and Go2 pose/TF consumers +remain separate acceptance checks. + +### Robot pose and TF conversion (no hardware required) + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_robot_tf.py +``` + +This runs the WebRTC device-pose converter and Go2's TF-chain builder on a sample +packet, round-trips generated TFMessage through CDR, and prints the three edges +and their composed camera frame. The robot-local frames receive `robot0/`; the +source parent stays `odom`. Timestamp `1746565669448350564` survives unchanged. +The live WebRTC connection deliberately supplies a host-arrival header instead, +matching its previous timing policy with integer-nanosecond precision. + +### Generated-message storage and module replay + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_storage_replay.py +``` + +The demo records custom weighted segments to SQLite and a ROS2-profile MCAP, +reopens both by their stored type/schema, and replays SQLite through a real +module's `Out` port. It prints weights `4, 5, 6` and consecutive source nanoseconds +starting at `1700000000123456789`. Temporary recordings are removed automatically. +This checks Python storage and replay. + +### Record an external custom message with the Rust recorder + +First complete the external-package demo above, including its locally added +`application_note` field. Build the recorder once, then record that external type +over both transports using the same executable: + +```bash +cargo build -p dimos-memory-recorder --locked +PYTHONPATH=.:build/message-codegen/demo/cpp/build:build/message-codegen/external-app/venv/lib/python3.12/site-packages \ + .venv/bin/python examples/message-codegen/demo_native_recording.py +``` + +Adjust the external virtualenv's Python version in the path when necessary. +The demo prints `locally-added-field-0` through `locally-added-field-2` and exact +payload stamps on LCM and Zenoh. It stops each recorder and closes its transport +resources, retaining `external-native-recording-{lcm,zenoh}.mcap` and `.log` files +under `build/message-codegen/demo/evidence/` for inspection. Use `--output` to +choose another artifact directory and `--executable` for a packaged recorder. + +With the viewer-checker dependencies installed as described above, independently +decode both recordings from their embedded schemas: + +```bash +node examples/message-codegen/viewer-checker/check-native-recording.mjs \ + build/message-codegen/demo/evidence/external-native-recording-lcm.mcap +node examples/message-codegen/viewer-checker/check-native-recording.mjs \ + build/message-codegen/demo/evidence/external-native-recording-zenoh.mcap +``` + +The Foxglove decoder requires no custom message package. Arbitrary custom payloads +are preserved byte-for-byte; their MCAP publish time uses reception time because +the recorder has no compiled knowledge of their timestamp layout. Recognized +standard stamped messages use exact source nanoseconds. Delete the four retained +MCAP/log files when finished reviewing them. These terminal checks complement +the separate Foxglove and Rerun UI acceptance demos. + +### Optional ROS2 bridge (isolated verification) + +DimOS generation and runtime remain ROS-free. To check the existing optional +bridge against ROS2 Jazzy after building the standalone Python binding, use the +independent reference container: + +```bash +docker build -f .github/docker/message-codegen-reference.Dockerfile \ + -t dimos-cdr-jazzy-reference .github/docker +docker run --rm -v "$PWD:/source:ro" dimos-cdr-jazzy-reference ' + set -e + source /opt/ros/jazzy/setup.bash + export PYTHONPATH=/source:/source/build/message-codegen/demo/cpp/build:$PYTHONPATH + cd /source + python3 -m pytest -o addopts="" --noconftest --import-mode=importlib -p no:cacheprovider \ + dimos/protocol/pubsub/impl/test_rospubsub_conversion.py \ + dimos/protocol/pubsub/impl/test_rospubsub.py -q + python3 examples/message-codegen/demo_ros_bridge.py +' +``` + +The demo prints three poses (`x=0, 1, 2`) and RGB pixels (`[0,128,255]`, +`[1,128,254]`, `[2,128,253]`) received through real ROS publishers/subscribers. +Source nanoseconds `123456789` through `123456791` survive unchanged. Nodes, +timers, subscriptions, and the container are cleaned up on exit. With ROS +already installed and sourced, run the same Python demo directly. + +The bridge uses ROS's serialization/type support and generated DimOS CDR codecs. +It requires matching ROS message packages only in the bridge environment; it +does not translate through LCM, rename fields, or load legacy message classes. +The independent Jazzy CI job runs these checks using the binding produced by +the preceding ROS-free build. + +### Galaxea feedback without hardware or ROS + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_galaxea_messages.py +``` + +This feeds synthetic chassis-speed samples into the actual Galaxea connection +callback and subscribes to its generated pose, odometry, and TF output streams. +It prints three changing positions and the `odom -> base_link -> lidar_chassis_left_link` +chain, with exact source nanoseconds on every edge. It requires neither ROS nor +a robot; it cleans up subscriptions and the module on exit. The neighboring +`test_connection_messages.py` also checks joint-state aggregation, command array +splitting, tracking-speed sentinels, and malformed-command rejection. + +### Whole-body transport feedback and commands + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_whole_body_transport.py +``` + +This connects the actual whole-body adapter to synthetic robot-side publishers +and a command subscriber, first over LCM and then through two explicit loopback +Zenoh sessions. It prints feedback positions `[0.25, -0.5]`, commanded positions +`[0.75, -0.25]`, and the hardware-facing WXYZ quaternion. The demo checks command +gains/torques and generated headers, then closes transports, subscriptions, and +sessions. `test_whole_body_transport.py` runs the same exchange automatically. + +### G1 whole-body feedback and coordinator joint streams + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_g1_messages.py +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_coordinator_joints.py +``` + +The G1 demo feeds three synthetic 29-motor snapshots through the actual output +ports. It prints changing joint positions, the converted ROS XYZW orientation, +and consecutive exact source nanoseconds shared by joint and IMU messages. It +does not initialize the Unitree SDK or connect to a robot. + +The coordinator demo runs two mock arms and prints aggregate and per-arm joint +streams, verifying names, positions, ROS header frames, CDR round trips, and one +shared tick stamp. Its existing internal snapshot clock remains in floating +seconds and is explicitly converted at the message boundary. Both demos close +subscriptions and stop their modules on exit. + +### Base-motion transport commands, odometry, and stop + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_base_transport.py +``` + +The actual base adapter receives generated PoseStamped odometry over LCM and +then independent loopback Zenoh sessions. It prints planar position `(2, -1)` +and yaw `pi/2`, sends velocity `(0.5, -0.2, 0.3)` after enabling, then verifies +a generated zero Twist arrives when disabled. It closes the adapter and all +transport resources. `test_base_transport.py` runs this exchange automatically. + +### Generated joint feedback in manipulation planning + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_planning_joints.py +``` + +This decodes a generated coordinator JointState, selects the left-arm joints in +canonical order, normalizes a differently ordered target, and validates both +against declared joint limits. It prints selected positions `[-0.2, 0.3]`, target +`[0.1, 0.5]`, and the preserved source stamp `1700000000123456789`. No robot, +planning backend, ROS installation, or persistent artifact is required. + + +### Planner backends and robot self exclusion + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_planning_backends.py +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_self_filter.py +``` + +The backend demo requires the manipulation dependencies (including Drake). It +creates a temporary two-joint URDF, plans between generated JointState values, +interpolates five waypoints, and checks every CDR-decoded waypoint against the +real Drake world. The printed shoulder positions run from -0.4 through 0 to 0.4. +This is an unobstructed joint-path demo; it does not exercise obstacle avoidance +or the still-pending generated trajectory/pose cutover. + +The self-filter demo requires yourdfpy. It moves a modeled cube through x=0, 1, +and 2, showing that the cube's return disappears while a distant point remains. +Both the retained cloud and the world-frame clear mask round-trip through CDR +with their original integer nanosecond timestamps. The demos remove temporary +models and dispose module resources on exit; neither needs robot hardware or ROS. + + +### Generated trajectories and execution status + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_trajectory.py +``` + +The trapezoid generator produces 51 ROS2 JointTrajectoryPoint values for two +joints. The demo sends the generated JointTrajectory over LCM and then over +independent loopback Zenoh sessions, preserving source nanoseconds +`1700000000123456789`. A real JointTrajectoryTask executes it against synthetic +feedback; printed commands progress from `[0, 0]` to `[0.3, -0.2]`. Generated +`dimos_msgs/msg/TrajectoryStatus` crosses the return channel and reports +`COMPLETED`, progress 1, and zero remaining duration. Each transport and session +closes on exit. `test_trajectory.py` runs the exchange as a subprocess test. + +Import trajectories from `dimos_generated.trajectory_msgs.msg`; waypoint times +use `builtin_interfaces/Duration`. Use `duration_from_seconds` and `to_seconds` +from `dimos.msgs.time` at floating-time boundaries. `sample_trajectory` and +`trajectory_duration` in `dimos.msgs.trajectory` provide interpolation and total +time outside the generated message classes. Absent velocities mean zero +feed-forward for joint-position execution. Generated status state fields are +integers with constants on TrajectoryStatus; compare them with `==`. + +The old handwritten JointTrajectory, TrajectoryPoint, and TrajectoryStatus +classes are removed. Generated sequences return value copies for message +entries: edit a point and assign it back with `trajectory.points[index] = point` +when changing a waypoint. + +### Generated poses and grasp proposals + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_planning_backends.py +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_grasp_proposals.py +``` + +The Drake demo now prints each waypoint's forward-kinematics tool position and +its CDR TF edge. Derived poses retain the joint sample's exact timestamp; the +first printed stamp is `1700000000.123456789`. The synthetic URDF is removed +when the demo exits. + +The grasp demo needs neither hardware nor a GPU. Four generated PointCloud2 +points cross a CDR round trip into the heuristic provider. The generated +`dimos_msgs/msg/GraspCandidateArray` crosses another round trip and prints a +ranked grasp at `[0.5, 0.0, 0.15]`, a downward approach axis, score `1.0`, and the +unchanged source timestamp `1700000000123456789` nanoseconds. The provider stops +on exit. + +Use nested ROS fields (`stamped.pose.position`, `stamped.header.stamp`) and the +functions in `dimos.msgs.geometry` for pose matrices, Euler angles, and local +translation. A pose-matrix conversion rejects non-finite or non-rigid geometry. +Generated configuration poses keep a zero timestamp unless a source supplies +one. Access grasp entries through `.candidates`; the former handwritten pickle +codecs have been removed. Grasp consumers validate scores and geometry before +commanding motion. + +For base motion, run: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_path_following.py +``` + +This demo decodes a generated 41-waypoint Path, runs the path follower against +DimOS's base simulator, and round-trips each Twist command through CDR. It prints +`arrived` and writes `build/message-codegen/demo/evidence/path-following.svg` +with the reference and executed paths. Remove that generated SVG to clean up. +Space's SVG and Rerun renderers now read generated poses, points, paths, detection +boxes, and occupancy grids directly. Both apply the occupancy origin rotation; +headless tests exercise the actual Rerun SDK as well as SVG output. + +### Generated cloud-to-map and inflation demo + +Run after building the generated Python messages above (the normal mapping environment +also requires NumPy, SciPy, Numba, and the visualization dependencies): + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_occupancy.py +``` + +The demo sends 1,200 synthetic points through CDR, projects them into an occupancy +map, inflates the obstacle by 0.2 m, and round-trips the generated grid through CDR. +It prints the occupied-cell counts and checks the exact source header. Open +`build/message-codegen/demo/evidence/occupancy.svg` to inspect the resulting map. +The demo needs no robot, ROS installation, viewer login, or running transport. +Remove that SVG to clean up its output. + +### Generated map → native/Python planner → path demo + +The normal native extension build provides A*. To build just that extension for +this demo, with the compiler and pybind11 already installed: + +```bash +c++ -O3 -shared -std=c++17 -fPIC \ + $(.venv/bin/python -m pybind11 --includes) \ + dimos/navigation/replanning_a_star/min_cost_astar_cpp.cpp \ + -o dimos/navigation/replanning_a_star/min_cost_astar_ext$(.venv/bin/python -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))') +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_grid_path.py +``` + +Both planners route around the same synthetic wall using a CDR-decoded grid. +The demo resamples each result, sends it through generated Path CDR, checks exact +source headers and obstacle exclusion, and exports `grid-path-python.svg` and +`grid-path-cpp.svg` under `build/message-codegen/demo/evidence/`. Open the SVGs to +compare the routes. The native extension is required; the demo fails explicitly +if it is missing. Remove the two SVGs to clean up the demo output. + +### Run the navigation module over both transports + +After building generated messages and the native A* extension above: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_navigation.py +``` + +This starts the real `CostMapper` and `ReplanningAStarPlanner` modules and a +synthetic drive loop. A generated CDR cloud is accumulated by the CPU voxel +mapper, then sent to CostMapper over LCM and separate loopback Zenoh sessions. +The resulting generated OccupancyGrid feeds navigation alongside generated +Odometry and PointStamped inputs. The drive loop consumes generated Twist +commands and sends new odometry until the module publishes a generated arrival +Bool. It checks the published Path's exact source header and the final stop +command. The simulation advances 0.1 seconds per control command at an accelerated +100 Hz wall-clock rate; it needs no robot or ROS installation. + +The terminal prints source/voxel counts, costmap dimensions, final position, +command count, and source timestamp. +Open `build/message-codegen/demo/evidence/navigation-lcm.svg` and +`navigation-zenoh.svg` to inspect the planned route. Module threads, subscriptions, +and transport sessions stop automatically; remove the two SVGs to clean up output. +`test_navigation.py` runs this same demo as a bounded subprocess E2E check. + +### Basic follower and exact odometry history + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_odometry_history.py +``` + +This hardware-free, fixed-step simulation passes a generated CDR path through the +basic follower's control step, applies its generated Twist commands, and checks +its generated arrival Bool. Each simulated Odometry sample passes through CDR +and the real OdometryHist handler. The demo checks consecutive nanosecond stamps +and true-height Rerun geometry, then exports the reference and traveled paths to +`build/message-codegen/demo/evidence/odometry-history.svg`. Open that file for +visual review; remove it to clean up the demo output. This exercises control +steps directly; it does not start a robot or the follower's timed control thread. + +### Generated frontier goals + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_frontiers.py +``` + +The demo decodes an occupancy map from CDR, detects the frontier between its +known and unknown halves, and round-trips the selected pose through CDR while +checking the exact source header. It prints the selected coordinates and writes +`build/message-codegen/demo/evidence/frontiers.svg`. Open that SVG to inspect +the selected goal; remove it to clean up. No robot, ROS installation, or viewer +login is required. + +### Generated patrol goals + +After the generation/build setup above, run: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_patrol.py +``` + +The demo prints five goals each for random, coverage, and frontier patrol, and +writes `build/message-codegen/demo/evidence/patrol-{random,coverage,frontier}.svg`. +Each goal crosses a CDR encode/decode boundary and retains the map's exact header. +The red markers show selected goals; this demo does not simulate travel between +them. No robot or ROS installation is required. Remove those three SVG files to +clean up the demo outputs. + +### Generated movement arbitration + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_movement.py +``` + +This terminal demo forwards a stamped click through CDR, shows teleop overriding +navigation, then shows navigation resuming after cooldown. It uses in-process +module callbacks and CDR output round trips; it does not actuate a robot or +exercise network discovery. No output files require cleanup. + +### Generated visual-servo commands + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_visual_servo.py +``` + +This terminal demo decodes generated CameraInfo and prints CDR-decoded velocity +commands for four synthetic detections: far away, at target distance, too close, +and right of center. It needs no camera, robot, or ROS installation and writes no +files. It exercises the 2D controller, not the complete person-follow pipeline. + +### Generated detection-cloud filters + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_cloud_filters.py +``` + +This terminal demo filters a synthetic cloud containing a dense object and one +isolated low point. Height and radius filters each retain 100 of 101 points; +results cross CDR boundaries and preserve the source header. It requires Open3D +from the development environment, no sensor or ROS, and writes no files. + +### Generated depth-to-detection projection + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_detection_projection.py +``` + +The terminal demo projects sixteen depth pixels into a translated world frame, +then prints the generated CDR centroid and exact source timestamp. It uses +synthetic depth and needs no sensor, robot, or ROS installation. No files are +written. This exercises projection, not model inference or the person-follow +module lifecycle. + +### Generated image detection annotation + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_detection_bbox.py +``` + +Open `build/message-codegen/demo/evidence/detection-bbox.png` to inspect the +synthetic target's bounding box and label. The terminal prints CDR-decoded track +and class identifiers and checks the exact image header. Remove the PNG to clean +up. This uses synthetic pixels, with no model download or sensor required. + +### Generated vision-model image boundary + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_vl_images.py +``` + +This offline terminal demo resizes a generated image and converts a fixed model +response into a CDR detection array. It prints the dimensions and verifies the +source header. It does not call an API or run model inference, and writes no files. + +### Person-follow control thread with generated messages + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_person_follow.py +``` + +This hardware-free demo starts the actual person-follow control thread with +fixed tracker output. It prints a CDR-decoded forward command, requests stopping, +checks the final zero command, and joins the thread. Model creation and tracker +inference are substituted; this is not a model-accuracy or robot demo. It needs +the optional Torch/Hydra imports but downloads no weights and writes no files. + +### Generated 3D marker messages + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_marker_messages.py +``` + +This terminal demo serializes two synthetic marker detections and prints their +CDR-decoded dictionary-qualified identities and world positions. It verifies the +exact source timestamp. It performs no marker inference or camera acquisition +and writes no files. + +### Detect an ArUco marker from a generated image + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_aruco_detection.py +``` + +This demo renders marker 7, runs OpenCV ArUco detection and pose estimation, +composes a translated world pose, and decodes the generated CDR detection. It +writes `build/message-codegen/demo/evidence/aruco-detection.png` with the detected +bbox and label. Expected world coordinates are x approximately 2 m and z +approximately 0.402 m. No camera or model weights are needed; delete the PNG to +clean up. + +### Generated bounding-box navigation goal + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_bbox_navigation.py +``` + +This terminal demo sends generated calibration and detections over LCM, then +prints a CDR-decoded goal (2.00, -0.40, -0.40) with the original detection header. +It retains the module's existing forward/left/up coordinate convention and does +not perform TF conversion. It needs no robot and writes no files. + +### Holonomic tracking through CDR + +After building the generated demo bindings, run: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_holonomic_tracking.py +``` + +The terminal shows a synthetic robot accelerating toward `(2, 1)` and settling +at the target. Reference poses, measured poses, and limited velocity commands +round-trip through generated CDR codecs each tick. This exercises the tracking +law and limiter, without a transport, path planner, hardware, or ROS install. +No processes or output files require cleanup. + +### Threaded holonomic path following + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_holonomic_path.py +``` + +The real follower thread drives a synthetic robot along a one-metre path, +prints decoded velocity and position, reports arrival, and publishes zero +velocity. Path, odometry, and commands use generated CDR round-trips. The demo +closes its thread and subscriptions automatically; it needs no robot or ROS. +This is an in-process control demo, not network transport verification. + +The holonomic path demo also exercises `DanLocalPlanner`'s core: it smooths and +resamples the incoming CDR path and visibly suppresses a duplicate replan inside +the configured commit window before starting the follower. + +### Native MLS planning from generated clouds + +With the generated demo bindings already built, install the native Python +planner into the checkout environment and run the terrain demo: + +```bash +uv pip install --python .venv/bin/python 'maturin>=1,<2' +source .venv/bin/activate +maturin develop --release --uv -m dimos/navigation/nav_3d/mls_planner/rust/py/Cargo.toml +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_mls_transformer.py +``` + +The demo prints a TF-derived starting pose, then sends a CDR-decoded synthetic +floor to the native MLS planner and prints the CDR-decoded path waypoints and +exact source timestamp. It requires no robot, model download, or ROS install. +It runs in one process and writes no output artifacts; native build products +remain in Cargo's ignored target directory. + +The MLS terrain demo also writes +`build/message-codegen/demo/evidence/mls-planner.rrd`, containing the floor, +raised path nodes, and connecting segments through the planner's Rerun adapters. +Open it with `rerun build/message-codegen/demo/evidence/mls-planner.rrd`. +For headless verification, use `rerun rrd verify` on that file. The recording is +an ignored build artifact and may be deleted after review. + +### Recorded navigation evaluation + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_navigation_recording.py +``` + +The demo writes generated lidar and odometry messages to temporary SQLite +streams, reads them through the navigation evaluator, and prints sensor-to-world +point registration and accumulated trajectory distance. It removes the temporary +database automatically. No recording download or hardware is required. + +The terrain demo also exercises native ray tracing and records its local map, +TF edges, and colormap annotations. Install its additional native extension with: + +```bash +maturin develop --release --uv -m dimos/mapping/ray_tracing/rust/py/Cargo.toml +``` + +The recorded-navigation demo now also invokes the actual native ray-tracing CLI +on its generated SQLite lidar/TF streams. Install the ray-tracing extension as +above. It writes `build/message-codegen/demo/evidence/raytrace-cli.rrd`; inspect +that file with Rerun or check it with `rerun rrd verify`. The temporary SQLite +source is still removed automatically. + +The terrain recording's TF hierarchy is now logged through the live Rerun +bridge's generated-message callback, including its frame axes. This demo invokes +the callback directly and does not start a network subscription or viewer. + +### Generated camera messages through the bridge + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_camera_bridge.py +rerun rrd verify build/message-codegen/demo/evidence/camera-bridge.rrd +``` + +Open `camera-bridge.rrd` in Rerun to inspect a red/green image, its compressed +copy, calibration, and two depth images that both represent one metre. The demo +uses generated CDR messages and the bridge callback; it does not start a network +subscription. Its only output is the ignored RRD file. + +The camera demo also logs a generated 3D box labelled `demo-box id=4` in the +camera's optical frame, exercising the same bridge used for image dispatch. + +The terrain demo sends the generated terrain cloud through the shared bridge, +which renders explicit height colors. The same adapter preserves standard packed +RGB colors when supplied by a cloud producer. + +The terrain demo's `world/planned_path` entity now comes from the shared bridge's +generated Path adapter. It is displayed 0.5m above the original route for clarity; +the path message retains its original coordinates. + +The terrain recording also includes `world/occupancy`, a small generated grid +with free, occupied, unknown, and intermediate-cost cells. The shared adapter +places its textured plane using the grid's origin pose. + +### Live LCM/Zenoh-to-Rerun bridge + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_live_bridge.py --transport lcm +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_live_bridge.py --transport zenoh +``` + +This starts the real bridge headlessly on an available local gRPC port, publishes +a generated image over the selected transport, waits for decoded/rendered delivery, and writes +`build/message-codegen/demo/evidence/live-bridge-lcm.rrd`. It closes the subscriptions +and transport automatically and never opens a window. A bounded subprocess test +runs the demo and verifies the recording with Rerun. Multicast loopback must be +available in the test environment. + +The Zenoh variant writes `live-bridge-zenoh.rrd`. It uses separate publisher and +subscriber sessions connected explicitly over loopback TCP with scouting and +gossip disabled. This verifies the transport-to-bridge route without claiming +default peer-discovery reliability. Both variants have bounded subprocess tests. + +Run the viewer control protocol without a GUI: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_viewer_controls.py +``` + +This opens a local WebSocket, sends a click, velocity, and stop, and prints the +verified generated CDR values. It requires no robot or ROS installation. + +Inspect command-center JSON and a generated click goal without a GUI: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_command_center.py +``` + +The demo exercises registered command-center handlers directly and prints the +state prepared for the browser, including a compressed costmap. + +Inspect a VLM request image without credentials or model inference: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_vlm_image.py +``` + +The demo writes `build/message-codegen/demo/evidence/vlm-request.jpg` from a +generated CDR image through the agent's request builder, using a stubbed model. + +Exercise named-location navigation without hardware or inference: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_navigation_skill.py +``` + +The demo tags generated odometry and verifies that a retrieved CDR goal preserves +position and orientation, with memory and navigation RPCs stubbed. + +Observe a synthetic camera over LCM/CDR and inspect its MCP image response: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_mcp_observe.py +``` + +The demo saves `build/message-codegen/demo/evidence/mcp-observe.jpg`. It invokes +MCP request handling directly and needs no model credentials or robot. + +Check keyboard commands and immediate key-release stopping without a display: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_keyboard_cdr.py +``` + +Synthetic pygame events drive the real handlers; subscribers verify generated +CDR velocity and gripper values. No hardware commands are sent. + +Inspect the hosted operator's PNG map and compact odometry payload: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_hosted_map.py +``` + +This writes `build/message-codegen/demo/evidence/hosted-map.png` from CDR inputs +through local callbacks, with no hosted connection or robot. + +### Explicit generated video telemetry + +After rebuilding the generated messages with the current in-tree definitions: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_video_stats.py +``` + +The demo converts browser health JSON into `dimos_msgs/msg/VideoStats`, crosses +CDR and SQLite boundaries, and prints an exact dropped-frame counter of +4294967297. Counters use integer schema fields rather than Joy's float32 axes. +The temporary database is removed automatically; no browser, robot or hosted +connection is required. The browser JSON plane retains its named metric fields. + +### Phone browser command CDR + +With the generated extension built, start an isolated loopback preview: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_phone_cdr.py +``` + +Open the printed loopback URL and click **Connect**. No sensor permission is +needed: the page sends its default values. The demo validates five generated +TwistStamped messages and exits, shutting down the web server. It never starts +the module control loop or publishes robot commands. + +The browser fetches `/teleop/schema`, uses pinned generic Foxglove ROS2/CDR +libraries to encode each advertised type, and wraps bytes in the existing relay +data frame with an explicit channel, qualified type and `cdr` encoding. The +backend rejects unknown channels, mismatched types and unframed legacy bytes. +This preview requires access to esm.sh for the pinned browser libraries. + +### Hosted Go2 command boundaries + +Inspect CDR velocity and navigation commands with mocked streams and driver RPC: + +```sh +PYTHONPATH=.:build/message-codegen/demo/cpp/build .venv/bin/python examples/message-codegen/demo_go2_command_cdr.py +``` + +The handler harness accepts two CDR commands one nanosecond apart, displays +the clamped velocity and a generated world-frame navigation goal, and checks +that E-STOP suppresses publication. It asserts that no driver RPC was called. +This is a command-boundary demo; it does not start a blueprint or connect to Go2. + +## Inspect generated camera compositing + +Run the bounded handler demo from the checkout's prepared environment: + +```bash +PYTHONPATH=.:build/message-codegen/demo/cpp/build \ + .venv/bin/python examples/message-codegen/demo_camera_mux_cdr.py +``` + +It decodes two generated RGB images, combines and scales them, then encodes +the result as CDR and JPEG. It asserts RGB ordering and the latest source +timestamp `1700000000123456790ns`. Ports are mocked; it starts no camera, +video encoder, control loop or robot connection. + +## Record and replay the native blueprint + +With the runtime installed and the native relay/recorder binaries built, run: + +```bash +python examples/message-codegen/demo_blueprint_recording.py --transport zenoh \ + --samples 3 --output build/message-codegen/demo/evidence/blueprint-recording.mcap +``` + +The output must be a new filename. The demo launches a loopback Zenoh router, +Python producer, C++ relay, Rust relay, and Rust MCAP recorder. It checks custom +line edits, exact image bytes, and standard poses, stops the producers, then +independently decodes the embedded ROS schemas and replays all three streams +through DimOS. Replay bytes must match the recording in each stream's order. +Samples have 100 ms timestamp spacing and retain a nonzero nanosecond remainder; +this avoids collapsing adjacent nanoseconds in the existing float-second replay +scheduler. Host tuning is checked only, never applied by this demo. + +The same command accepts `--transport lcm`; this path requires working multicast +on the host and has not passed the current host's default-interface gate. +The recording remains for viewer inspection; no producer or router remains. +See [recording evidence](evidence/blueprint-recording.md) for the verified run +and remaining viewer/transport acceptance gates. + +### Build the release wheel matrix without publishing + +The `release-build-check.yml` workflow accepts a manual `full-matrix=true` +dispatch on the proposal branch. It builds and runs the installed-wheel smoke +test for Python 3.10, 3.11 and 3.12 on Linux x86-64, Linux ARM64 and macOS ARM64, +and checks the source archive. Successful wheels are retained as review artifacts. +It has no package-publishing or release-tag jobs. Ordinary pull requests retain +the single-wheel smoke check. The release workflow's `dry-run` publishes to +TestPyPI and is not a substitute for this build-only check. diff --git a/examples/message-codegen/demo_blueprint_recording.py b/examples/message-codegen/demo_blueprint_recording.py new file mode 100644 index 00000000000..457fab89131 --- /dev/null +++ b/examples/message-codegen/demo_blueprint_recording.py @@ -0,0 +1,173 @@ +# Copyright 2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Record the native relay blueprint to MCAP, stop it, then replay every CDR value.""" + +import argparse +from contextlib import ExitStack +from functools import partial +from pathlib import Path +import socket +import threading +from typing import Any +from unittest.mock import patch + +from demo_blueprint import PythonRelayDemo, blueprint +from dimos_generated.dimos_msgs.msg import LineSegments3D +from dimos_generated.geometry_msgs.msg import PoseStamped +from dimos_generated.sensor_msgs.msg import Image +from mcap.reader import make_reader +import numpy as np +from rosbags.typesys import Stores, get_types_from_msg, get_typestore + +from dimos.core.coordination.module_coordinator import ModuleCoordinator +from dimos.core.stream import In +from dimos.experimental.memory.rust_recorder import RustMcapStoreConfig, RustRecorder +from dimos.memory.recording_policy import OnExisting +from dimos.memory.replay_module import replay_module +from dimos.memory.store.mcap import McapStore +from dimos.msgs.time import to_nanoseconds +from dimos.protocol.service.system_configurator.base import configure_system +from dimos.protocol.service.zenohservice import ZenohConfig, ZenohSessionPool + + +class RelayRecorder(RustRecorder): + l2: In[LineSegments3D] + i2: In[Image] + p0: In[PoseStamped] + + +def verify_and_replay(artifact: Path, samples: int) -> None: + expected: dict[str, list[bytes]] = {} + independent = get_typestore(Stores.ROS2_JAZZY) + with artifact.open("rb") as source: + for schema, channel, row in make_reader(source).iter_messages(): + assert schema is not None and schema.encoding == "ros2msg" + assert channel.message_encoding == "cdr" + independent.register(get_types_from_msg(schema.data.decode(), schema.name)) + independent.deserialize_cdr(row.data, schema.name) + with McapStore(path=str(artifact)) as store: + for name in ("l2", "i2", "p0"): + messages = [obs.data for obs in store.stream(name)] + expected[name] = [message.encode() for message in messages] + observed_samples = set() + for message in messages: + delta = to_nanoseconds(message.header.stamp) - 1700000000123456789 + sample, remainder = divmod(delta, 100_000_000) + assert remainder == 0 + assert 0 <= sample < samples + observed_samples.add(sample) + if name == "l2": + assert message.segments[0].weight == sample + 2 + assert message.segments[0].start.x == sample + elif name == "p0": + assert ( + message.pose.position.x, + message.pose.position.y, + message.pose.position.z, + ) == (sample, 0, 0) + assert message.pose.orientation.w == 1 + else: + assert message.encoding == "rgb8" + assert (message.width, message.height, message.step) == (640, 480, 1920) + np.testing.assert_array_equal( + message.data.view(), np.full(921600, sample % 256, dtype=np.uint8) + ) + assert observed_samples == set(range(samples)), (name, observed_samples) + print( + f"MCAP {name}: verified {len(messages)} values, samples={sorted(observed_samples)}" + ) + + module = replay_module(str(artifact))(dataset=str(artifact)) + received: dict[str, list[bytes]] = {name: [] for name in expected} + done = threading.Event() + lock = threading.Lock() + + def receive(name: str, message: Any) -> None: + with lock: + received[name].append(message.encode()) + if all(len(received[key]) >= len(values) for key, values in expected.items()): + done.set() + + with ExitStack() as stack: + for name in expected: + stack.callback( + module.outputs[name].subscribe(lambda message, name=name: receive(name, message)) + ) + stack.callback(module.stop) + module.start() + assert done.wait(15), {name: len(values) for name, values in received.items()} + assert received == expected + print("Replay: every recorded CDR value matched in per-stream order after producers stopped") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--transport", choices=("lcm", "zenoh"), default="zenoh") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--samples", type=int, default=3) + parser.add_argument("--cpp", type=Path, default=Path("build/native-cpp-examples/cdr_relay")) + parser.add_argument("--rust", type=Path, default=Path("target/debug/cdr_relay")) + parser.add_argument("--recorder", type=Path, default=Path("target/debug/dimos-memory-recorder")) + args = parser.parse_args() + if args.samples < 1: + parser.error("--samples must be positive") + if args.output.exists(): + parser.error("--output must be a new file; existing recordings are preserved") + for executable in (args.cpp, args.rust, args.recorder): + if not executable.is_file(): + parser.error(f"Build the native executable first: {executable}") + args.output.parent.mkdir(parents=True, exist_ok=True) + recording = RelayRecorder.blueprint( + executable=str(args.recorder.resolve()), + build_command="", + cwd=str(args.recorder.resolve().parent), + store=RustMcapStoreConfig(path=str(args.output.resolve())), + on_existing=OnExisting.ERROR, + record_tf=False, + encoding_threads=2, + ) + with ExitStack() as stack: + stack.enter_context( + patch( + "dimos.protocol.service.system_configurator.base.configure_system", + partial(configure_system, check_only=True), + ) + ) + bp = blueprint(args.transport, args.cpp, args.rust, recording=recording) + if args.transport == "zenoh": + with socket.socket() as reservation: + reservation.bind(("127.0.0.1", 0)) + endpoint = f"tcp/127.0.0.1:{reservation.getsockname()[1]}" + pool = ZenohSessionPool() + stack.callback(pool.close_all) + pool.acquire( + ZenohConfig( + mode="router", listen=[endpoint], connect=[], multicast=False, gossip=False + ) + ) + bp = bp.global_config( + zenoh_mode="client", zenoh_connect=endpoint, zenoh_multicast=False + ) + coordinator = ModuleCoordinator.build(bp) + stack.callback(coordinator.stop) + producer = coordinator.get_instance(PythonRelayDemo) + for sample in range(args.samples): + print(producer.exchange(sample), flush=True) + verify_and_replay(args.output.resolve(), args.samples) + print(f"Retained self-describing {args.transport} recording: {args.output}") + + +if __name__ == "__main__": + main() diff --git a/examples/message-codegen/demo_native_recording.py b/examples/message-codegen/demo_native_recording.py new file mode 100644 index 00000000000..9e433be7bb4 --- /dev/null +++ b/examples/message-codegen/demo_native_recording.py @@ -0,0 +1,158 @@ +# Copyright 2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Record an independently generated custom type with an unchanged Rust binary.""" + +import argparse +from contextlib import ExitStack +import json +import os +from pathlib import Path +import socket +import subprocess +import threading +from typing import Any +import uuid + +from demo_native import free_port, stop_process +from external_telemetry.demo_msgs.msg import Telemetry +from mcap.reader import make_reader + +from dimos.core.transport import LCMTransport, ZenohTransport +from dimos.experimental.memory.rust_recorder import RustStreamSpec +from dimos.protocol.service.zenohservice import ZenohConfig, ZenohSessionPool + + +def record(backend: str, executable: Path, output: Path) -> None: + endpoint = f"tcp/127.0.0.1:{free_port(socket.SOCK_STREAM)}" + url = f"udpm://239.255.76.67:{free_port(socket.SOCK_DGRAM)}?ttl=0" + ready, written = threading.Event(), threading.Event() + transcript: list[str] = [] + with ExitStack() as stack: + pool = ZenohSessionPool() + stack.callback(pool.close_all) + topic = f"demo/{uuid.uuid4().hex[:8]}" + publisher: Any + if backend == "lcm": + publisher = LCMTransport(topic, Telemetry, url=url) + else: + publisher = ZenohTransport( + topic, + Telemetry, + session_pool=pool, + scouting=False, + multicast=False, + gossip=False, + listen=[endpoint], + connect=[], + ) + stack.callback(publisher.stop) + publisher.start() + spec = RustStreamSpec.from_type( + port="telemetry", name="telemetry", payload_type=Telemetry, codec="cdr" + ) + session = ZenohConfig( + mode="client", + connect=[endpoint], + listen=[], + multicast=False, + gossip=False, + scouting=False, + connect_timeout=10, + ).to_wire() + process = stack.enter_context( + subprocess.Popen( + [str(executable.resolve())], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + env={ + **os.environ, + "DIMOS_TRANSPORT": backend, + "LCM_DEFAULT_URL": url, + "RUST_LOG": "info,dimos_memory_recorder=debug", + }, + ) + ) + + def read_logs() -> None: + assert process.stdout is not None + for raw in process.stdout: + line = raw.decode(errors="replace") + transcript.append(line) + if "memory recorder ready" in line: + ready.set() + if "memory recorder batch written" in line: + written.set() + + reader = threading.Thread(target=read_logs, name="recorder-demo-logs") + reader.start() + stack.callback(reader.join, 5) + stack.callback(stop_process, process) + assert process.stdin is not None + launch = { + "topics": {"telemetry": publisher.channel}, + "session": session if backend == "zenoh" else {}, + "config": { + "store": {"kind": "mcap", "path": str(output.resolve())}, + "encoding_threads": 2, + "streams": [spec.model_dump()], + }, + } + process.stdin.write(json.dumps(launch).encode() + b"\n") + process.stdin.close() + assert ready.wait(15), "".join(transcript) + expected = [] + for index in range(3): + message = Telemetry(application_note=f"locally-added-field-{index}") + message.header.stamp.sec = 1700000000 + message.header.stamp.nanosec = 123456789 + index + expected.append(message) + written.clear() + publisher.publish(message) + assert written.wait(5), "".join(transcript) + stop_process(process) + assert process.returncode == 0, "".join(transcript) + with output.open("rb") as stream: + records = list(make_reader(stream).iter_messages()) + assert len(records) == len(expected) + for (schema, channel, row), original in zip(records, expected, strict=True): + assert schema is not None and schema.encoding == "ros2msg" + assert b"application_note" in schema.data + assert channel.message_encoding == "cdr" + assert row.publish_time == row.log_time # Unknown stamp layout uses reception time. + decoded = Telemetry.decode(row.data) + assert decoded == original + print( + f"{backend}: {decoded.application_note}, payload stamp={decoded.header.stamp.sec}.{decoded.header.stamp.nanosec:09d}" + ) + output.with_suffix(".log").write_text("".join(transcript)) + print(f"Recorded {len(records)} independently generated messages with full schema: {output}") + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--executable", type=Path, default=Path("target/debug/dimos-memory-recorder") + ) + parser.add_argument("--output", type=Path, default=Path("build/message-codegen/demo/evidence")) + args = parser.parse_args() + args.output.mkdir(parents=True, exist_ok=True) + for backend in ("lcm", "zenoh"): + record(backend, args.executable, args.output / f"external-native-recording-{backend}.mcap") + print("PASS: custom field recorded on both transports without a recorder decoder or rebuild") + + +if __name__ == "__main__": + main() diff --git a/examples/message-codegen/demo_storage_replay.py b/examples/message-codegen/demo_storage_replay.py new file mode 100644 index 00000000000..cd90f21a6d8 --- /dev/null +++ b/examples/message-codegen/demo_storage_replay.py @@ -0,0 +1,96 @@ +# Copyright 2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Record generated values to SQLite and MCAP, then replay SQLite through module ports.""" + +from pathlib import Path +from tempfile import TemporaryDirectory +from threading import Event, Lock +from typing import Any + +from dimos_generated.dimos_msgs.msg import LineSegment3D, LineSegments3D +from dimos_generated.geometry_msgs.msg import Point +from dimos_generated.std_msgs.msg import Header + +from dimos.memory.codecs.base import codec_for, codec_id +from dimos.memory.replay_module import replay_module +from dimos.memory.store.mcap import McapStore +from dimos.memory.store.sqlite import SqliteStore +from dimos.msgs.time import time_from_nanoseconds, to_nanoseconds +from dimos.protocol.cdr_mcap import CdrMcapWriter + + +def main() -> None: + messages = [ + LineSegments3D( + header=Header(frame_id="map", stamp=time_from_nanoseconds(1700000000123456789 + index)), + segments=[LineSegment3D(start=Point(x=index), end=Point(y=2), weight=index + 4)], + ) + for index in range(3) + ] + with TemporaryDirectory(prefix="dimos-cdr-storage-") as directory: + database = Path(directory) / "memory.db" + artifact = Path(directory) / "recording.mcap" + with SqliteStore(path=str(database)) as store, CdrMcapWriter(artifact) as writer: + stream = store.stream("segments", LineSegments3D) + for index, message in enumerate(messages): + stream.append(message, ts=1 + index * 0.05) + writer.write( + "segments", + message.encode(), + schema_name=message.msg_name, + schema=message.schema, + log_time_ns=1000000000 + index * 50000000, + publish_time_ns=to_nanoseconds(message.header.stamp), + ) + print(f"SQLite default codec: {codec_id(codec_for(LineSegments3D))}") + for source in [ + SqliteStore(path=str(database), must_exist=True), + McapStore(path=str(artifact)), + ]: + with source: + decoded = [observation.data for observation in source.stream("segments")] + assert decoded == messages + print( + f"{type(source).__name__}: weights={[value.segments[0].weight for value in decoded]}" + ) + replay = replay_module(str(database))(dataset=str(database)) + received: list[Any] = [] + ready = Event() + lock = Lock() + + def receive(message: LineSegments3D) -> None: + with lock: + received.append(message) + print( + f"Module replay: weight={message.segments[0].weight}, stamp={to_nanoseconds(message.header.stamp)}" + ) + if len(received) == len(messages): + ready.set() + + unsubscribe = replay.outputs["segments"].subscribe(receive) + try: + replay.start() + assert ready.wait(5), "Replay did not deliver all messages" + assert received == messages + finally: + unsubscribe() + replay.stop() + print( + "PASS: CDR persisted, reopened by type/schema, replayed through Out ports; temporary files removed" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/message-codegen/evidence/blueprint-recording.md b/examples/message-codegen/evidence/blueprint-recording.md new file mode 100644 index 00000000000..fe1d61c68d4 --- /dev/null +++ b/examples/message-codegen/evidence/blueprint-recording.md @@ -0,0 +1,186 @@ +# Native blueprint recording and replay + +Verified locally on 2026-09-30 using the current runtime batch on top of +`f4a269ce4bc766a1ff876ed2b6f5ac432e6d3900`. + +- 33 recorder, SQLite path, stamped-message and TF regression tests passed. +- Strict scoped mypy passed for RustRecorder and the SQLite replay helper. +- The actual Python -> C++ -> Rust -> Python Zenoh relay produced three + custom LineSegments3D samples, three 640x480 RGB images, and three PoseStamped + samples. Native edits incremented line weight twice; image bytes were exact. +- Rust recorder reported received=9, written=9, encode_errors=0 and exited cleanly. +- MCAP had CDR channels and embedded ros2msg schemas. Independent rosbags Jazzy + decoding succeeded for every row; all three streams contained samples 0,1,2. +- After all producers stopped, DimOS ReplayModule emitted every recorded value + with byte equality and per-stream order preserved. + +The retained local artifact is +`build/message-codegen/demo/evidence/blueprint-current-zenoh-v3.mcap`; terminal +log is `/tmp/cdr-blueprint-recording-v3.log`. Earlier failed artifacts were +preserved. This demo uses installed built-in custom messages; arbitrary installed +external-message capture has separate acceptance requirements. + +The first recorder run exposed inherited TF wiring being passed despite +`record_tf=False`. RustRecorder now filters native launch topics to declared +recording streams; regression tests cover TF enabled and disabled. Another run +exposed float-second replay treating 1 ns spaced epoch timestamps as static. +The demo now uses realistic 100 ms intervals while checking exact nanoseconds; +sub-float-resolution scheduling is not claimed. + +No hardware ran and host tuning was check-only. Default-interface LCM, human +Foxglove/Rerun inspection, full dependency retirement and final installed-package +acceptance remain open. This evidence does not mark stages 5 or 6 complete. + +## Follow-up regression batch + +The published parent `f4a269ce4` codegen workflow 36763634296 succeeded. Main +workflow 36763630343 ended cancelled after ARM reported 6170 passed, 228 skipped +and one documentation-branding failure. Both offending spellings in +`docs/usage/lcm.md` are repaired; other Python jobs were cancelled, not passed. + +Follow-up local checks: 15 Joy/Header/LineSegments3D/branding tests and 53 +stamped-covariance/time tests passed. Migrated suites retain numeric payloads, +empty and large sequences, covariance patterns, copy independence, explicit +clock and datetime conversions, and add independent ROS CDR checks. Legacy +convenience constructors, custom string formatting, inheritance, Path-based +line packing and implicit timestamps are intentionally retired. Generated ROS +time overflow now reports an explicit ValueError; both int32 boundaries and +values immediately outside them are covered. Scoped mypy passed time.py. + +## Future cross-stack compatibility note + +Parent review on 2026-09-30 identified EpisodeStatus in the independent open +chain #3854 -> #3855 -> #3921 -> #3931 -> #3942. It is used by typed live Out, +collection recorder input/source timestamp extraction, offline episode boundaries +and TUI RPC. That type is absent from this CDR checkout. Its eventual integration +would need a coordinated generated CDR design if that proposal is adopted; a JSON substitution would not satisfy +the typed CDR recorder contract. The user has not selected the integration design. +No code or commits from that chain were imported or changed. This is an explicit +future compatibility note, not a regression introduced by this branch. +CDR is not a prerequisite for merging that separate stack: it targets current +main and remains independent. No merge is authorized for this CDR draft. + +## Image and value-type retirement follow-up + +Local verification after `0bac7f454`: 52 Path/basic-covariance checks, five +CameraInfo calibration checks, and 58 generated-image/compression/detection/heavy-import +checks passed. Strict mypy passed image.py, Rerun message_helpers.py and both +manual message tools. The original JPEG/JXL mean-error and compressed-size +thresholds, lossless PNG/JXL depth, codec effort comparison, camera calibration +values, Rerun box geometry/labels and path sequence behavior are retained. + +PNG/JXL encoding and decoding live in external functions over generated values. +Source headers and 16UC1 depth units survive, including big-endian inputs. Raw +64FC1, 16SC1 and 32FC3 views remain supported; unsupported compression types are +rejected explicitly. The local venv was missing the existing lockfile dependency +imagecodecs 2026.6.26; its 27.5 MiB wheel was installed without changing project +dependencies or system packages. + +Image file/selection tests use deterministic temporary pixels and real sharpness +calculation in virtual-time windows. The manual pickle tool retains only its +raw-array input path, converting to generated images after loading; no old typed +message decoding was added. Manual publishers and hardware tools were not run. + +## Geometry and map follow-up + +Local checks: 42 wrench/geometry, 12 odometry, 78 point-cloud/viewer and 32 occupancy +tests passed. Strict scoped mypy passed geometry.py, pointcloud.py, occupancy.py +and the Rerun helper. Force/torque arrays and axis-aligned finite cloud bounds +are external functions; generated types stay value containers. Point-field tests +retain intensity, zero offset_time, tag, line and all original overlap scenarios. +Occupancy tests retain unknown/free/occupied counts, threshold filtering, origin +yaw, coordinate conversion and obstacle-preserving reduction. Borrowed views +remain read-only; tests mutate declared message data explicitly. ROS float32 +resolution is compared with numerical tolerances instead of assuming Python +float64 storage. + +## Generated geometry regression continuation + +The Vector3, Quaternion, Pose, Transform, and Twist suites now use generated +values and explicit external math. Non-unit Hamilton products/inverses, zero +errors, rotated local offsets, frame composition, independent ROS decoding, +and original numerical tolerances remain covered. Constructor coverage now +checks explicit nested fields, deep-copy isolation, covariance lengths and +layout, zero/default timestamps, exact nanoseconds, and rejection of retired +polymorphic fields. Stamped values are distinct from their payloads. + +Offline message-directory validation: **614 passed** with only +`tf2_msgs/test_TFMessage_lcmpub.py` excluded. The attempted whole-directory run +also produced 614 passes, but that real multicast test failed host LCM self-test +and thread teardown; this remains an environment-dependent gate, not a pass. +No host routing/firewall configuration was changed. Strict scoped mypy for +`dimos/msgs/geometry.py` and repository hooks passed. + +Published commit `0bac7f454bcd2f61766c0246e38a40016c62746c` has terminal success +for main CI 36765815202 and codegen CI 36765824915. Self-hosted lanes were +skipped. These results do not cover subsequent local commits until their own +exact-HEAD workflows complete. + +## Python dependency retirement and remaining consumers + +The 47 obsolete Python message implementation files have been removed after +migrating their remaining Python imports. The root project now declares raw +`lcm-dimos-fork` directly and no longer depends on `dimos-lcm`. The task venv's +installed `dimos-lcm` was also uninstalled: **624 offline message/voxel tests +passed without it**. The lock removes only `dimos-lcm` and its unused +`foxglove-websocket` dependency; no remaining package versions changed. +Re-resolution required restricting the existing optional `a750-control` wheel +to Python 3.12, its only published/locked ABI, instead of requesting it on 3.10/3.11. + +The retired typed-pickle voxel regression fixture is now deterministic CDR +geometry, with exact voxel counts at three resolutions, repeated-ingestion +invariance, numerical coordinates, and range preservation. Raw PLY/image +occupancy fixtures remain supported. Full local test collection reached 6555 +tests with 16 collection errors for absent optional dependencies (coacd, h5py, +open_clip, plotext, reportlab, requests_mock, trimesh, ultralytics, yourdfpy), +not obsolete message imports. This is not a full-suite pass. + +Remaining examples and the optional LeRobot runtime use generated image, +pose, point-cloud and UInt32 button fields. The runtime copies image pixels +before retaining an observation. Its own scoped mypy configuration passes, +but its behavioral suite still requires the absent LeRobot environment. +The virtual robot's pure unicycle integration passed a numerical check; +the C++ controller compiled against the installed CMake message package and +raw LCM, without being run. The obsolete Lua and TypeScript LCM-codec demos +are retired explicitly; Lua generation is deferred and the browser CDR SDK +is the supported web path. Transform documentation passed 5 executable Python +blocks; eval documentation passed all 6 blocks (4 no-result, 2 with results). + +Outstanding dependency gate: the M20 onboard DrDDS/Zenoh C++ bridge still +fetches/uses old message headers. The local environment lacks the vendor DrDDS +SDK, and its `sensor_msgs::msg`/`nav_msgs::msg` types collide with generated C++ +type names. A verified vendor boundary/codec migration is still required; +no board build, deployment or robot operation has been performed. Historical +manual replay fixtures and final viewer/transport acceptance also remain open. + +The shared Go2 moment fixture now takes an explicit CDR SQLite path; its tests +create generated recordings instead of resolving the old pickle dataset name. +The manual memory import/query tool now copies 1024 deterministic CDR samples +into an isolated destination. Its original >1000 counts, >10-second duration, +pagination, ordering, overlap, lazy decoding and pose assertions remain. +All **18 tool checks passed**, including cached CPU CLIP embedding/search with +model network access disabled; no new weight download was needed. These +synthetic image results validate the pipeline, not real-scene recognition. + +The separate real detection fixture `unitree_go2_lidar_corrected` is absent +locally and its LFS archive is 1,212,727,745 bytes. Regenerating it while retaining +its semantic assertions is still an explicit outstanding fixture gate. + +### DimSim standalone codec audit + +The remaining browser/Deno `@dimos/msgs` consumers now use pinned Foxglove CDR +codecs and a standalone full-schema bundle exported from canonical generated +messages. RGB publishes raw RGBA Image, depth publishes 16UC1 Image, and lidar, +odometry and cmd_vel use PointCloud2/PoseStamped/Twist with ROS 2 channel names. +The existing LCM transport and LC02 WebSocket envelope remain. JPEG stays only +in eval/sidebar previews. No legacy decoding fallback was introduced. + +Verified locally: Vite production build; complete Deno CLI type check with a +frozen lock preserving unrelated original dependency versions; three socket-free +codec tests (large image envelope, all velocity components, malformed input); +canonical-schema drift pytest; strict Python exporter typing; pre-commit checks. +Four Deno-encoded payloads (Twist, PoseStamped, Image and PointCloud2) decoded in +both generated Python and independent rosbags with exact values/source stamps; +a generated Python Twist decoded in Deno with exact command components. +Full browser/physics simulation acceptance remains unverified. These checks +neither operate hardware nor prove host multicast or end-to-end rendering. diff --git a/examples/message-codegen/test_native_recording.py b/examples/message-codegen/test_native_recording.py new file mode 100644 index 00000000000..616bf2b3a74 --- /dev/null +++ b/examples/message-codegen/test_native_recording.py @@ -0,0 +1,44 @@ +# Copyright 2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import importlib.util +from pathlib import Path +import subprocess +import sys + +import pytest + + +@pytest.mark.native_e2e +def test_external_schema_records_on_both_transports_with_unchanged_binary(tmp_path): + executable = Path("target/debug/dimos-memory-recorder").resolve() + if not executable.is_file() or importlib.util.find_spec("external_telemetry") is None: + pytest.skip("Build the recorder and install the external message demo package first") + result = subprocess.run( + [ + sys.executable, + str(Path(__file__).with_name("demo_native_recording.py")), + "--executable", + str(executable), + "--output", + str(tmp_path), + ], + capture_output=True, + text=True, + timeout=60, + ) + assert result.returncode == 0, result.stdout + result.stderr + assert "lcm: locally-added-field-2" in result.stdout + assert "zenoh: locally-added-field-2" in result.stdout + assert "PASS:" in result.stdout diff --git a/examples/message-codegen/viewer-checker/check-native-recording.mjs b/examples/message-codegen/viewer-checker/check-native-recording.mjs new file mode 100644 index 00000000000..efb5cb7f03e --- /dev/null +++ b/examples/message-codegen/viewer-checker/check-native-recording.mjs @@ -0,0 +1,40 @@ +import assert from 'node:assert/strict'; +import { readFile } from 'node:fs/promises'; +import { McapIndexedReader } from '@mcap/core'; +import { parse, fixupTypes } from '@foxglove/rosmsg'; +import { MessageReader } from '@foxglove/rosmsg2-serialization'; +import zstd from '@foxglove/wasm-zstd'; + +await zstd.isLoaded; +const bytes = await readFile(process.argv[2]); +const reader = await McapIndexedReader.Initialize({ + readable: { + size: async () => BigInt(bytes.length), + read: async (offset, size) => bytes.subarray(Number(offset), Number(offset + size)), + }, + decompressHandlers: { zstd: (data, size) => zstd.decompress(data, Number(size)) }, +}); +assert.equal(reader.header.profile, 'ros2'); +const decoders = new Map(); +for (const schema of reader.schemasById.values()) { + assert.equal(schema.encoding, 'ros2msg'); + const definitions = parse(new TextDecoder().decode(schema.data), { ros2: true, skipTypeFixup: true }); + definitions[0].name = schema.name.replace('/msg/', '/'); + fixupTypes(definitions); + decoders.set(schema.id, new MessageReader(definitions)); +} +let count = 0; +for await (const message of reader.readMessages({ validateCrcs: true })) { + const channel = reader.channelsById.get(message.channelId); + assert.equal(channel.messageEncoding, 'cdr'); + assert.equal(channel.topic, 'telemetry'); + const value = decoders.get(channel.schemaId).readMessage(message.data); + assert.equal(value.application_note, `locally-added-field-${count}`); + assert.equal(value.header.stamp.sec, 1700000000); + assert.equal(value.header.stamp.nanosec, 123456789 + count); + assert.equal(message.logTime, message.publishTime); + console.log(`Foxglove decoded ${value.application_note} from embedded schema.`); + count += 1; +} +assert.equal(count, 3); +console.log('PASS: native recorder custom messages need no viewer-side message package');