Skip to content

p1_display plotting performance improvements. - #444

Merged
adamshapiro0 merged 23 commits into
masterfrom
plot-perf
Oct 7, 2026
Merged

adamshapiro0 merged 23 commits into
masterfrom
plot-perf

Conversation

@adamshapiro0

@adamshapiro0 adamshapiro0 commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

New Features

  • Add InputDataWrapperMessage.count_wrapped_data() to total up wrapped packets by type straight from a file, without deserializing each message

Changes

  • Generate plots roughly 1.6x faster and with less than half the peak memory on a multi-hour log
    • Skip building per-message debug/trace strings when they won't be printed
    • Reduce MixedLogReader progress check call frequency
    • Limit unnecessary, expensive enum __call__() operations with new lookup_value() function
    • Improved p1_display summary reset event table generation speed
    • Reuse satellite and signal ID instances across epochs
    • Release decoded GNSSSignals messages when not needed
    • Build constant trace values as numpy arrays rather than lists
    • Add orjson package for faster Plotly JSON encoding
  • Reduced plot HTML file sizes (up to 40% or more for long logs)
    • Share a single customdata across multiple traces in a plot
    • Write one shared plotly.min.js alongside the figures instead of embedding a copy in each one, so an individual plot file now needs the rest of the output directory to render
    • Shrink the generated HTML by emitting each figure's shared hover time data once rather than once per trace
    • Write the UTC time axis at millisecond resolution, which is all a browser can display
    • Encode UTC X-axis values as base64-encoded float instead of date strings (the Plotly default)
    • Share identical X-axis arrays between Plotly traces in the generated HTML output

Fixes

  • Fix EventNotificationMessage.to_numpy() raising OverflowError on numpy 2 when a log event carries a negative severity
  • Fix logger.trace() raising TypeError when passed printf-style arguments
  • Fall back to any source ID when plotting map if no IDs found in pose source range
  • Pin Plotly version <7 to keep the generated map loading from disk (file:// no longer works with Plotly 7)

@bhers4 bhers4 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Comment thread python/fusion_engine_client/analysis/analyzer.py Outdated
Comment thread python/fusion_engine_client/analysis/analyzer.py
Comment thread python/fusion_engine_client/parsers/mixed_log_reader.py Outdated
Comment thread python/fusion_engine_client/utils/trace.py
Base automatically changed from decoder to master October 7, 2026 11:48
A long log can contain millions of InputDataWrapper messages, and unwrapping
them completely in order to generate a summary can be very slow.
…array.

numpy 2 raises instead of wrapping around when an unsigned array is built
from a negative value, which the signed severity of a LOG event produced.
Plotly writes each value out as a literal date string, and Javascript's Date
cannot represent the sub-millisecond digits that were being emitted.
…d one.

Plotly only writes `plotly.min.js` into the output directory when it is not
already there, so a directory written by an earlier run went on serving its
original library no matter which version was installed.
A long log holds millions of per-signal objects that the plots never look at
once the Numpy arrays exist, which accounted for a third of peak memory.
Plotly validates a list one element at a time, which dominated the signal
status plot for a log with millions of signal observations.
Plotly 7 bundles a MapLibre build whose web worker is an ES module, which
Chrome refuses to start from a `file://` page, leaving the map blank.
@adamshapiro0
adamshapiro0 force-pushed the plot-perf branch 2 times, most recently from 780dbe1 to c1e289f Compare October 7, 2026 13:40
@adamshapiro0
adamshapiro0 merged commit dec2859 into master Oct 7, 2026
27 checks passed
@adamshapiro0
adamshapiro0 deleted the plot-perf branch October 7, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants