Skip to content

[BREAKING][feat] Use native MessagePack for Controller metadata - #221

Open
0oshowero0 wants to merge 17 commits into
Ascend:mainfrom
0oshowero0:feat/controller-msgpack-metadata
Open

0oshowero0 wants to merge 17 commits into
Ascend:mainfrom
0oshowero0:feat/controller-msgpack-metadata

Conversation

@0oshowero0

@0oshowero0 0oshowero0 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

The Rust Controller request engine proposed in #220 needs a metadata wire format it can read and write directly without Python pickle. This PR switches Controller metadata (BatchMeta and field_schema) across ZMQ RPC boundaries to native MessagePack.

Changes

  • Native MessagePack BatchMeta (CUSTOM_TYPE_BATCH_META = 6): Encode BatchMeta (global_indexes, partition_ids, field_schema, production_status, custom_meta, _custom_backend_meta, _links) as native MessagePack inside msgpack.Ext(6, ...), raising RuntimeError if non-MessagePack values are present rather than silently falling back to pickle. Normalize integer global_indexes via operator.index in BatchMeta.__init__ and BatchMeta.__setstate__.
  • Canonical field_schema wire codec: Add encode_field_schema, decode_field_schema, and parse_torch_dtype to transport canonical dtype names ("int64", "bfloat16", "bool", etc.) and shape lists across NOTIFY_DATA_UPDATE, KV_PREPARE_UPDATE, and VALIDATE_DUMP_SCHEMA, restoring torch.dtype and tuple shapes in Python. Normalize global_indexes and custom_backend_meta sample keys in StorageManager.notify_data_update.
  • BatchMeta.extra_info & single-pass SET_CUSTOM_META grouping: Omit BatchMeta.extra_info from the Controller's MessagePack payload while keeping it intact in Python pickle / copy.deepcopy / Ray RPC serialization. Group partition_custom_meta in a single pass in AsyncTransferQueueClient.async_set_custom_meta instead of calling chunk_by_partition().
  • ZMQ decode safety & legacy checkpoint migration: Wrap malformed envelope fields in ZMQMessageDecodeError and catch response serialization errors in the Controller request loop. Normalize legacy string dtypes and list shapes in TransferQueueController.load_checkpoint with a FutureWarning (scheduled for removal in 0.2.0).

Breaking change

  • Upgrade the Controller, TransferQueueClient, and StorageManager processes together to the same TransferQueue version.
  • Sample tags (tag / BatchMeta.custom_meta) and per-field backend metadata (_custom_backend_meta) carried in BatchMeta must be native MessagePack values (dict with UTF-8 str keys, list, str, bytes, 64-bit int, float, bool, None; nested maps and lists are supported, and tuple decodes as list). Convert NumPy or PyTorch scalars with .item(), and avoid "__tq_tensordict__": True in user metadata dicts.
  • BatchMeta.extra_info is omitted from Controller ZMQ replies (reconstructs as {}), while Python pickle and Ray RPC preserve it.
  • Automatic migration of legacy string dtypes and list shapes in Controller checkpoints is deprecated and will be removed in 0.2.0.

Performance

Local microbenchmark (200 warmup + 2,000 timed iterations, median p50 latency and ZMQ frame byte size) comparing origin/main (32276bb, pickle-backed BatchMeta and torch.dtype) against HEAD (750a772, native MessagePack) on 256 samples across 4 fields (2 regular + 2 jagged per_sample_shapes) and 4 backend strategy strings per sample:

ZMQ Message / Path Encode p50 (main → HEAD) Decode p50 (main → HEAD) Wire Size (main → HEAD)
GET_META_RESPONSE (empty poll) 5.5 µs → 3.0 µs (1.8x) 3.3 µs → 2.8 µs (1.2x) 317 B → 190 B (-40.1%)
GET_META_RESPONSE (256 samples, no tags) 59.1 µs → 21.1 µs (2.8x) 51.8 µs → 78.4 µs 41,165 B → 48,116 B (+16.9%)
GET_META_RESPONSE (256 samples, with tags) 78.7 µs → 31.2 µs (2.5x) 76.5 µs → 114.2 µs 53,223 B → 62,979 B (+18.3%)
KV_PREPARE_UPDATE_RESPONSE (256 samples) 19.8 µs → 7.6 µs (2.6x) 50.6 µs → 57.0 µs 4,140 B → 3,941 B (-4.8%)
NOTIFY_DATA_UPDATE (256 samples) 32.0 µs → 42.7 µs 250.5 µs → 266.5 µs 48,207 B → 48,099 B (-0.2%)
SET_CUSTOM_META (256 samples, empty tags) 258.3 µs → 11.4 µs (22.7x) 36.9 µs → 36.5 µs 1,168 B → 1,168 B (0.0%)
SET_CUSTOM_META (256 samples, with tags) 1,561.9 µs → 357.3 µs (4.4x) 165.8 µs → 173.8 µs 15,767 B → 15,767 B (0.0%)

Controller-side response serialization (the single-threaded bottleneck in _process_request_loop) is 1.8x–2.8x faster (BatchMeta encode), while client-side BatchMeta decode (distributed across client processes) adds ~27–38 µs and ~17–18% wire bytes because self-describing MessagePack maps write string keys explicitly rather than relying on Python pickle memo tables.

Validation

  • ruff check, ruff format --check, mypy transfer_queue, compileall, and license header checks passed.
  • Unit and E2E pytest suites covering serial_utils, BatchMeta, TransferQueueClient, TransferQueueController, KV interface, data dump, and checkpoint save/restore passed, including live ZMQ round-trips and legacy checkpoint FutureWarning migration tests.

Encode BatchMeta with MessagePack extension 6 and encode standalone field schemas with native values. Carry per-row tags and per-field backend metadata as codec-tagged blobs.

Cover the wire format and round trips over real ZMQ. Client and Controller use the same package version.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Restore uniform and per-sample shapes when constructing Python BatchMeta from MessagePack. Keep scalar shapes and missing shapes covered by the round-trip tests.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Send canonical dtype names in metadata RPCs and restore Torch dtypes at Python entry points. Resolve known legacy dtype names when loading Controller checkpoints and keep protocol tests focused.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Encode standalone tags and backend metadata as typed extensions so the shared codec can restore them. Build SET_CUSTOM_META from one tag snapshot grouped by partition.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-20T08:31:44.658657Z 401e74f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

0oshowero0 and others added 11 commits November 20, 2026 17:47
Use the existing codec and payload pair at metadata RPC boundaries to drop the extra MessagePack extension layer.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Warn once when restoring legacy dtype names and tell callers to re-save with the current release.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Share field schema encoding and decoding across metadata RPCs. Restore tuple shapes before recording data and when loading older Controller checkpoints.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Move nested accelerator tensors to CPU during metadata pickling. Apply the same rule to BatchMeta when a message falls back to pickle.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Include the frame summary when a message omits a required envelope field.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Accept scalar and flat-list tag values. Validate tags before writing or restoring state and pass empty metadata values through natively.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Normalize BatchMeta indexes at the wire boundary so custom samplers do not trigger whole-message pickle fallback.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Reuse schema dtype parsing during checkpoint preflight so an invalid legacy dtype cannot replace Controller state or restore storage payloads.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Send backend values directly, normalize indexes before transport, and validate row tags before they can enter the wire codec or storage. Keep BatchMeta.extra_info in Python and Ray serialization.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Prepare the Controller snapshot once before restoring storage. Apply that snapshot on success or release it on failure.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Remove custom tag/backend validation types and helpers, restore original client/interface and single-step checkpoint restore flows, and fail at runtime when BatchMeta cannot be encoded as native MessagePack.

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
… deprecations

Keep per_sample_shapes list-to-dict indexing inside StorageManager.notify_data_update, clarify the legacy checkpoint FutureWarning, and tag all FutureWarning removals with TODO(0.2.0).

Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant