Repository navigation
[BREAKING][feat] Use native MessagePack for Controller metadata - #221
Open
0oshowero0 wants to merge 17 commits into
Open
0oshowero0 wants to merge 17 commits into
0oshowero0 wants to merge 17 commits into
Conversation
Encode BatchMeta with MessagePack extension 6 and encode standalone field schemas with native values. Carry per-row tags and per-field backend metadata as codec-tagged blobs. Cover the wire format and round trips over real ZMQ. Client and Controller use the same package version. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Restore uniform and per-sample shapes when constructing Python BatchMeta from MessagePack. Keep scalar shapes and missing shapes covered by the round-trip tests. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Send canonical dtype names in metadata RPCs and restore Torch dtypes at Python entry points. Resolve known legacy dtype names when loading Controller checkpoints and keep protocol tests focused. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Encode standalone tags and backend metadata as typed extensions so the shared codec can restore them. Build SET_CUSTOM_META from one tag snapshot grouped by partition. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Use the existing codec and payload pair at metadata RPC boundaries to drop the extra MessagePack extension layer. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Warn once when restoring legacy dtype names and tell callers to re-save with the current release. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Share field schema encoding and decoding across metadata RPCs. Restore tuple shapes before recording data and when loading older Controller checkpoints. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Move nested accelerator tensors to CPU during metadata pickling. Apply the same rule to BatchMeta when a message falls back to pickle. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Include the frame summary when a message omits a required envelope field. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Accept scalar and flat-list tag values. Validate tags before writing or restoring state and pass empty metadata values through natively. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Normalize BatchMeta indexes at the wire boundary so custom samplers do not trigger whole-message pickle fallback. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Reuse schema dtype parsing during checkpoint preflight so an invalid legacy dtype cannot replace Controller state or restore storage payloads. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Send backend values directly, normalize indexes before transport, and validate row tags before they can enter the wire codec or storage. Keep BatchMeta.extra_info in Python and Ray serialization. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Prepare the Controller snapshot once before restoring storage. Apply that snapshot on success or release it on failure. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com>
Remove custom tag/backend validation types and helpers, restore original client/interface and single-step checkpoint restore flows, and fail at runtime when BatchMeta cannot be encoded as native MessagePack. Signed-off-by: 0oshowero0 <o0shower0o@outlook.com> Co-authored-by: Cursor <cursoragent@cursor.com>
… deprecations Keep per_sample_shapes list-to-dict indexing inside StorageManager.notify_data_update, clarify the legacy checkpoint FutureWarning, and tag all FutureWarning removals with TODO(0.2.0). Signed-off-by: 0oshowero0 <o0shower0o@outlook.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The Rust Controller request engine proposed in #220 needs a metadata wire format it can read and write directly without Python
pickle. This PR switches Controller metadata (BatchMetaandfield_schema) across ZMQ RPC boundaries to native MessagePack.Changes
BatchMeta(CUSTOM_TYPE_BATCH_META = 6): EncodeBatchMeta(global_indexes,partition_ids,field_schema,production_status,custom_meta,_custom_backend_meta,_links) as native MessagePack insidemsgpack.Ext(6, ...), raisingRuntimeErrorif non-MessagePack values are present rather than silently falling back topickle. Normalize integerglobal_indexesviaoperator.indexinBatchMeta.__init__andBatchMeta.__setstate__.field_schemawire codec: Addencode_field_schema,decode_field_schema, andparse_torch_dtypeto transport canonical dtype names ("int64","bfloat16","bool", etc.) and shape lists acrossNOTIFY_DATA_UPDATE,KV_PREPARE_UPDATE, andVALIDATE_DUMP_SCHEMA, restoringtorch.dtypeandtupleshapes in Python. Normalizeglobal_indexesandcustom_backend_metasample keys inStorageManager.notify_data_update.BatchMeta.extra_info& single-passSET_CUSTOM_METAgrouping: OmitBatchMeta.extra_infofrom the Controller's MessagePack payload while keeping it intact in Pythonpickle/copy.deepcopy/ Ray RPC serialization. Grouppartition_custom_metain a single pass inAsyncTransferQueueClient.async_set_custom_metainstead of callingchunk_by_partition().ZMQMessageDecodeErrorand catch response serialization errors in the Controller request loop. Normalize legacy stringdtypes andlistshapes inTransferQueueController.load_checkpointwith aFutureWarning(scheduled for removal in0.2.0).Breaking change
TransferQueueClient, andStorageManagerprocesses together to the same TransferQueue version.tag/BatchMeta.custom_meta) and per-field backend metadata (_custom_backend_meta) carried inBatchMetamust be native MessagePack values (dictwith UTF-8strkeys,list,str,bytes, 64-bitint,float,bool,None; nested maps and lists are supported, andtupledecodes aslist). Convert NumPy or PyTorch scalars with.item(), and avoid"__tq_tensordict__": Truein user metadata dicts.BatchMeta.extra_infois omitted from Controller ZMQ replies (reconstructs as{}), while Pythonpickleand Ray RPC preserve it.dtypes andlistshapes in Controller checkpoints is deprecated and will be removed in0.2.0.Performance
Local microbenchmark (200 warmup + 2,000 timed iterations, median p50 latency and ZMQ frame byte size) comparing
origin/main(32276bb,pickle-backedBatchMetaandtorch.dtype) againstHEAD(750a772, native MessagePack) on 256 samples across 4 fields (2 regular + 2 jaggedper_sample_shapes) and 4 backend strategy strings per sample:main→HEAD)main→HEAD)main→HEAD)GET_META_RESPONSE(empty poll)5.5 µs→3.0 µs(1.8x)3.3 µs→2.8 µs(1.2x)317 B→190 B(-40.1%)GET_META_RESPONSE(256 samples, no tags)59.1 µs→21.1 µs(2.8x)51.8 µs→78.4 µs41,165 B→48,116 B(+16.9%)GET_META_RESPONSE(256 samples, with tags)78.7 µs→31.2 µs(2.5x)76.5 µs→114.2 µs53,223 B→62,979 B(+18.3%)KV_PREPARE_UPDATE_RESPONSE(256 samples)19.8 µs→7.6 µs(2.6x)50.6 µs→57.0 µs4,140 B→3,941 B(-4.8%)NOTIFY_DATA_UPDATE(256 samples)32.0 µs→42.7 µs250.5 µs→266.5 µs48,207 B→48,099 B(-0.2%)SET_CUSTOM_META(256 samples, empty tags)258.3 µs→11.4 µs(22.7x)36.9 µs→36.5 µs1,168 B→1,168 B(0.0%)SET_CUSTOM_META(256 samples, with tags)1,561.9 µs→357.3 µs(4.4x)165.8 µs→173.8 µs15,767 B→15,767 B(0.0%)Controller-side response serialization (the single-threaded bottleneck in
_process_request_loop) is 1.8x–2.8x faster (BatchMetaencode), while client-sideBatchMetadecode (distributed across client processes) adds~27–38 µsand~17–18%wire bytes because self-describing MessagePack maps write string keys explicitly rather than relying on Pythonpicklememo tables.Validation
ruff check,ruff format --check,mypy transfer_queue,compileall, and license header checks passed.serial_utils,BatchMeta,TransferQueueClient,TransferQueueController, KV interface, data dump, and checkpoint save/restore passed, including live ZMQ round-trips and legacy checkpointFutureWarningmigration tests.