Repository navigation
Description
TL;DR: Keep
TransferQueueControllerdeployed and discovered as a named Ray actor, and replace its single-threaded Python ZeroMQ (ZMQ) loop with a multi-threaded Rust metadata engine embedded in the same actor process via PyO3 (abi3). Under TransferQueue's non-blocking polling model, consumers repeatedly pollGET_METAwhile producers pushNOTIFY_DATA_UPDATE, funneling high-frequency RPCs onto one GIL-bound Python thread that payspicklecodec and sparsetorch.Tensorscan overhead on every call. The Rust engine parallelizes wire codec across worker threads and executes partition operations concurrently over compactu64bitsets. Built-in samplers run natively in Rust while custom PythonBaseSamplersubclasses run through an on-demand PyO3 callback once a batch is ready, preserving full API, checkpoint, and Grafana compatibility so the legacy Python loop can be retired after rollout.
1. Background and Bottlenecks
tq.init() uses Ray only to launch, discover, and configure TransferQueueController; all runtime metadata traffic goes directly over ZMQ (interface.py). Once TransferQueueClient and StorageManager obtain the ZMQ endpoints, they send metadata RPCs directly to TransferQueueController, while storage backends move tensor payloads independently.
Non-blocking consumer polling combined with producer updates funnels high-frequency ZMQ traffic onto a single Python thread (_process_request_loop, controller.py), creating two bottlenecks at scale. Because GET_META(mode="fetch") returns BatchMeta.empty() immediately whenever ready samples are below batch_size (controller.py), waiting consumers continuously poll TransferQueueController alongside NOTIFY_DATA_UPDATE and KV_* requests:
- Single-thread GIL serialization: Multipart ZMQ I/O, message serialization, readiness scanning, and index allocation are CPU-bound in-memory operations executed sequentially on one core; adding Python threads or
asynciostill contends on the GIL. - Sparse
torch.Tensorstate andpicklewire overhead: Each partition tracks readiness and consumption intorch.int8tensors sized up tomax(global_index) + 1(controller.py), allocating temporary mask tensors even on empty polls, whileBatchMetaandtorch.dtypefall back to Pythonpickleon the wire (serial_utils.py).
2. Architecture and Python–Rust Boundary
Keep the named Ray actor for cluster placement and discovery, and embed the Rust metadata engine in-process via PyO3 (abi3) rather than spawning an external subprocess. An in-process PyO3 extension shares the Ray actor's lifecycle and RSS memory accounting, and invokes user-defined Python BaseSampler subclasses in microseconds without cross-process IPC—avoiding permanent dual maintenance of two TransferQueueController implementations.
flowchart TB
Callers["Caller Processes<br/>tq.init() · TransferQueueClient · StorageManager"]
Backends["Storage Backends (Data Plane Unchanged)<br/>SimpleStorage · MooncakeStore · Yuanrong"]
subgraph Actor["TransferQueueController (Single Ray Actor Process)"]
direction LR
PyShell["Python Actor Shell (Cold Path)<br/>Ray Discovery & Config · /metrics Exporter · Custom Sampler Adapter"]
RustEngine["Rust Metadata Engine (Hot Path)<br/>ZMQ ROUTER · Parallel Codec · Partition Bitsets · Native Samplers"]
PyShell <-- "PyO3 (lifecycle / metrics snapshot / custom sample)" --> RustEngine
end
Callers -- "Ray .remote() (startup, config, discovery)" --> PyShell
Callers <-- "Direct ZMQ (polling & metadata RPCs)" --> RustEngine
Callers <-- "Tensor Payloads" --> Backends
Python and Rust split responsibilities across four explicit interfaces:
- Ray Bootstrap & Lifecycle (Python Actor Shell): Preserves actor registration (
TransferQueueController),store_config/get_config,get_zmq_server_info, and teardown. On__init__, the Python actor shell starts the Rust metadata engine, waits forhandshake_socketandrequest_handle_socket(ROUTER) to bind, and publishes their endpoints. - ZMQ Wire Codec (Native MsgPack Frame + Opaque User Metadata): Framework metadata—
BatchMeta,field_schema(canonicaldtypestrings and integershapearrays), and production/consumption status arrays—uses pure MsgPack inserial_utils.pyinstead ofpickle(metadata.py), allowing Rust worker threads to decode requests and encode replies natively. Per-sample user dictionaries (custom_meta/tagandcustom_backend_meta) are encoded byTransferQueueClientas self-contained MsgPack byte slices; the Rust metadata engine stores and splices the byte slices verbatim by row without invoking Python (inspecting scalar keys liketotal_lengthsonly when required by a built-in sampler), preserving arbitrary Python objects transparently. - Sampler Execution (Native Fast Path + On-Demand PyO3 Callback): Built-in samplers (
SequentialSampler,GRPOGroupNSampler,RankAwareSampler,SeqlenBalancedSampler) run natively in Rust with zero GIL acquisition. User-defined PythonBaseSamplersubclasses run through a PyO3 adapter inside the same engine: all producer updates,KV_*calls, and unreadyfetchpolls return purely in Rust without touching the GIL; Rust acquires the GIL only once a partition has at leastbatch_sizeready samples to callsampler.sample(ready_indexes, batch_size, partition=PartitionView, **sampling_config), wherePartitionViewlazily decodescustom_metaonly if queried. - Checkpointing & Prometheus/Grafana Observability:
- Checkpointing:
SAVE_CONTROLLER_CHECKPOINTandLOAD_CONTROLLER_CHECKPOINT(controller.py) write and restore a versioned binary snapshot ofPartitionIndexManagerand partition state directly from Rust, calling PyO3 once to include Pythonsampler.save_checkpoint()state when a custom sampler is active (with legacy.pklimport during migration). - Prometheus & Grafana: Python's
TQMetricsExporter(metrics.py) remains the/metricsHTTP endpoint and continues scrapingSimpleStorageUnitmetrics. The Rust metadata engine records per-op_typecounts and latency histograms in lock-freeAtomicU64buckets alongside partition and index gauges, syncing snapshots toTQMetricsExportereveryTQ_METRICS_COLLECT_INTERVALseconds sografana_dashboard.jsonrequires zero changes.
- Checkpointing:
3. Concurrency and State Design
- Decouple ZMQ I/O from parallel worker-pool codec. Non-blocking ZMQ I/O tasks read and write multipart frames on
handshake_socketandrequest_handle_socket(controller.py), dispatching raw frames to a multi-threaded Rust worker pool so MsgPack decoding andBatchMetaencoding scale across CPU cores outside state locks. - Shard execution by
partition_idover compactlocal_rowbitsets. Requests targeting different partitions execute concurrently against per-partition state, while single-partition mutations remain atomic so readiness checks, sampling, andmark_consumedstay race-free. Cross-partition coordination is limited to O(1) per-unit FIFO queues inPartitionIndexManager(controller.py) and shared-field links on__public__. Within each partition, sparseglobal_indexvalues map bidirectionally to denselocal_rowslots (0 .. N-1), storing field readiness and per-task consumption inu64bitsets so polling scans reduce to bitwiseANDover machine words while ZMQ replies keepglobal_index.
4. Migration and Retirement Plan
- PyO3 Skeleton & Pure-MsgPack
BatchMetaCodec: Add native MsgPack encoding forBatchMetaandtorch.dtypeinserial_utils.py, bring up the PyO3 Rust metadata engine behindcontroller.backend="rust", and pass startup,HANDSHAKE,GET_LIST_PARTITIONS, andtq.close()tests. [BREAKING][feat] Use native MessagePack for Controller metadata #221 - Core Read/Write Loop & Concurrency Benchmark: Implement
PartitionIndexManager,local_rowbitsets,GET_META(insert,fetch,force_fetch),NOTIFY_DATA_UPDATE, cleanup RPCs, and built-in samplers. Verify behavioral parity againsttests/test_controller.pyand benchmark multi-client polling and write QPS/P99 latency. - Full Feature Parity (KV, Custom Sampler Adapter, Checkpoint, Metrics): Port
KV_*andLINKRPCs, add the PyO3 customBaseSampleradapter, wire atomic metrics snapshots intoTQMetricsExporter, and support checkpoint save/load (including legacy.pklimport). - Default Flip & Legacy Cleanup: Ship prebuilt
abi3wheels in CI, flipcontroller.backenddefault from"python"to"rust"after pilot validation, and remove the legacy Python request loop and tensor state tables in the following release.
Activity
0oshowero0 commentedon Nov 20, 2026
1. 背景与性能瓶颈
tq.init()仅利用 Ray 完成TransferQueueController的拉起、寻址与配置同步,运行期的元数据请求全部直接走 ZMQ(interface.py)。TransferQueueClient和StorageManager获取 ZMQ 地址后直接向TransferQueueController收发元数据请求,样本 Tensor 数据则由各存储后端独立搬运。非阻塞消费轮询叠加生产写入,使高频 ZMQ 请求全部汇聚到单条 Python 线程(
_process_request_loop,controller.py)上串行执行,在大规模场景下形成两处瓶颈。由于GET_META(mode="fetch")在就绪样本不足batch_size时立即返回空BatchMeta(controller.py),等待数据的消费进程会持续轮询TransferQueueController,与生产方的NOTIFY_DATA_UPDATE、KV_*请求并发竞争:asyncio仍受同一把 GIL 串行限制。torch.Tensor状态表与pickle编解码开销:每个分区直接以稀疏global_index为行下标分配长度为max(global_index) + 1的torch.int8状态矩阵(controller.py),连高频空轮询也要临时构造掩码 Tensor;同时BatchMeta与torch.dtype在传输层回退到 Pythonpickle(serial_utils.py)。2. 总体架构与 Python–Rust 工作界面
保留命名 Ray Actor 负责集群调度与发现,采用 PyO3(
abi3)同进程嵌入而非独立子进程拉起 Rust 元数据引擎。同进程扩展与 Ray Actor 共享生命周期及 RSS 内存统计,且能以微秒级开销回调用户自定义 PythonBaseSampler,避免因自定义采样器而长期双写维护两套TransferQueueController。flowchart TB Callers["调用方进程<br/>tq.init() · TransferQueueClient · StorageManager"] Backends["存储后端(数据面不变)<br/>SimpleStorage · MooncakeStore · Yuanrong"] subgraph Actor["TransferQueueController(单个 Ray Actor 进程)"] direction LR PyShell["Python 外壳(冷路径)<br/>Ray 发现与配置 · /metrics 导出 · 自定义 Sampler 适配"] RustEngine["Rust 元数据引擎(热路径)<br/>ZMQ ROUTER · 并行编解码 · 分区位图状态 · 内置 Sampler"] PyShell <-- "PyO3(启停 / 监控快照 / 按需采样回调)" --> RustEngine end Callers -- "Ray .remote()(启动、配置、地址发现)" --> PyShell Callers <-- "ZMQ 直连(高频轮询与元数据请求)" --> RustEngine Callers <-- "样本 Tensor 读写" --> BackendsPython 外壳与 Rust 元数据引擎的工作界面按四个维度明确切分:
TransferQueueController的 Actor 名称、store_config/get_config、get_zmq_server_info及关闭逻辑。Python 外壳在__init__中通过 PyO3 启动 Rust 元数据引擎,等待handshake_socket与request_handle_socket两个 ZMQROUTERsocket 绑定就绪后对外发布地址。serial_utils.py中为BatchMeta和torch.dtype补齐纯 MsgPack 编码分支(如dtype编码为标准字符串、shape编码为整数数组,不再回退pickle,metadata.py),使 Rust 工作线程能够原生解码请求并编码响应。对于用户自定义的custom_meta(tag)与custom_backend_meta,TransferQueueClient按样本编码为自包含的 MsgPack 字节块,Rust 元数据引擎按行原样存储字节块并在构造BatchMeta时直接拼回响应(仅在内置SeqlenBalancedSampler需要时读取total_lengths等标量),无需调用 Python 即可透明兼容任意用户 Python 对象。Sampler执行(内置原生快路径 + 自定义 PyO3 按需回调):内置采样器(SequentialSampler、GRPOGroupNSampler、RankAwareSampler、SeqlenBalancedSampler)在 Rust 元数据引擎内原生实现,全程零 GIL。用户自定义 PythonBaseSampler子类通过 PyO3 适配器接入同一引擎:所有写入通知、KV_*请求以及数据未齐时的空轮询fetch仍在 Rust 内无 GIL 直接返回;仅当分区就绪样本达到batch_size时,Rust 元数据引擎才获取一次 GIL 调用sampler.sample(ready_indexes, batch_size, partition=PartitionView, **sampling_config)(其中只读代理PartitionView仅在被调用get_custom_meta()时按需懒解码对应行数据)。SAVE_CONTROLLER_CHECKPOINT/LOAD_CONTROLLER_CHECKPOINT协议(controller.py),由 Rust 元数据引擎直接读写PartitionIndexManager与分区状态的带版本头紧凑二进制快照;若启用了自定义 PythonSampler,保存与恢复时通过 PyO3 同步一次sampler.save_checkpoint()状态(灰度期兼容导入旧版.pkl快照)。TQMetricsExporter(metrics.py)对外提供/metricsHTTP 端点及SimpleStorageUnit指标采集。Rust 元数据引擎用无锁AtomicU64记录各op_type请求计数与固定分桶延迟直方图,连同分区和索引统计每TQ_METRICS_COLLECT_INTERVAL秒同步一次快照给TQMetricsExporter,现有grafana_dashboard.json无需任何改动。3. 并发模型与状态设计
handshake_socket与request_handle_socket的多帧收发(controller.py),将原始帧分发至多线程 Rust 工作池,使 MsgPack 解码与BatchMeta编码在状态锁外利用多核并行执行。partition_id分片并发读写紧凑local_row位图。 不同分区的请求在各自分区状态上并发执行;同一分区内的状态读写保持原子性,保证就绪扫描、采样与mark_consumed严格一致。跨分区同步仅限PartitionIndexManager的 O(1) 存储单元 FIFO 队列(controller.py)及__public__共享字段引用检查。每个分区内部将稀疏global_index双向映射为稠密的local_row(0 .. N-1),以u64位图存储字段就绪状态与各任务消费状态;高频轮询扫描简化为机器字按位与运算,对外响应和存储路由继续使用global_index。4. 分阶段推进与旧实现下线计划
BatchMeta编解码:在serial_utils.py中补齐BatchMeta与torch.dtype的纯 MsgPack 编码,打通controller.backend="rust"下的 PyO3 启停、HANDSHAKE、GET_LIST_PARTITIONS与tq.close()生命周期。 [BREAKING][feat] Use native MessagePack for Controller metadata #221PartitionIndexManager、局部local_row位图、GET_META(insert、fetch、force_fetch)、NOTIFY_DATA_UPDATE、清理接口及内置Sampler,对齐tests/test_controller.py语义并对比多客户端高频轮询与写入场景下的 QPS 及 P99 延迟。Sampler适配、Checkpoint、监控):迁移KV_*与LINK系列接口,接入 PyO3 自定义BaseSampler适配器、TQMetricsExporter原子指标快照及 Checkpoint 读写(含旧版.pkl导入)。abi3wheel,完成灰度验证后将controller.backend默认值切换为"rust",并在下一版本移除旧 Python 请求循环与稀疏 Tensor 状态表代码。