Skip to content

[RFC] A Concurrent Rust Request Engine for TransferQueueController #220

Description

@0oshowero0

TL;DR: Keep TransferQueueController deployed and discovered as a named Ray actor, and replace its single-threaded Python ZeroMQ (ZMQ) loop with a multi-threaded Rust metadata engine embedded in the same actor process via PyO3 (abi3). Under TransferQueue's non-blocking polling model, consumers repeatedly poll GET_META while producers push NOTIFY_DATA_UPDATE, funneling high-frequency RPCs onto one GIL-bound Python thread that pays pickle codec and sparse torch.Tensor scan overhead on every call. The Rust engine parallelizes wire codec across worker threads and executes partition operations concurrently over compact u64 bitsets. Built-in samplers run natively in Rust while custom Python BaseSampler subclasses run through an on-demand PyO3 callback once a batch is ready, preserving full API, checkpoint, and Grafana compatibility so the legacy Python loop can be retired after rollout.

1. Background and Bottlenecks

tq.init() uses Ray only to launch, discover, and configure TransferQueueController; all runtime metadata traffic goes directly over ZMQ (interface.py). Once TransferQueueClient and StorageManager obtain the ZMQ endpoints, they send metadata RPCs directly to TransferQueueController, while storage backends move tensor payloads independently.

Non-blocking consumer polling combined with producer updates funnels high-frequency ZMQ traffic onto a single Python thread (_process_request_loop, controller.py), creating two bottlenecks at scale. Because GET_META(mode="fetch") returns BatchMeta.empty() immediately whenever ready samples are below batch_size (controller.py), waiting consumers continuously poll TransferQueueController alongside NOTIFY_DATA_UPDATE and KV_* requests:

  1. Single-thread GIL serialization: Multipart ZMQ I/O, message serialization, readiness scanning, and index allocation are CPU-bound in-memory operations executed sequentially on one core; adding Python threads or asyncio still contends on the GIL.
  2. Sparse torch.Tensor state and pickle wire overhead: Each partition tracks readiness and consumption in torch.int8 tensors sized up to max(global_index) + 1 (controller.py), allocating temporary mask tensors even on empty polls, while BatchMeta and torch.dtype fall back to Python pickle on the wire (serial_utils.py).

2. Architecture and Python–Rust Boundary

Keep the named Ray actor for cluster placement and discovery, and embed the Rust metadata engine in-process via PyO3 (abi3) rather than spawning an external subprocess. An in-process PyO3 extension shares the Ray actor's lifecycle and RSS memory accounting, and invokes user-defined Python BaseSampler subclasses in microseconds without cross-process IPC—avoiding permanent dual maintenance of two TransferQueueController implementations.

flowchart TB
    Callers["Caller Processes<br/>tq.init() · TransferQueueClient · StorageManager"]
    Backends["Storage Backends (Data Plane Unchanged)<br/>SimpleStorage · MooncakeStore · Yuanrong"]

    subgraph Actor["TransferQueueController (Single Ray Actor Process)"]
        direction LR
        PyShell["Python Actor Shell (Cold Path)<br/>Ray Discovery & Config · /metrics Exporter · Custom Sampler Adapter"]
        RustEngine["Rust Metadata Engine (Hot Path)<br/>ZMQ ROUTER · Parallel Codec · Partition Bitsets · Native Samplers"]
        PyShell <-- "PyO3 (lifecycle / metrics snapshot / custom sample)" --> RustEngine
    end

    Callers -- "Ray .remote() (startup, config, discovery)" --> PyShell
    Callers <-- "Direct ZMQ (polling & metadata RPCs)" --> RustEngine
    Callers <-- "Tensor Payloads" --> Backends
Loading

Python and Rust split responsibilities across four explicit interfaces:

  • Ray Bootstrap & Lifecycle (Python Actor Shell): Preserves actor registration (TransferQueueController), store_config / get_config, get_zmq_server_info, and teardown. On __init__, the Python actor shell starts the Rust metadata engine, waits for handshake_socket and request_handle_socket (ROUTER) to bind, and publishes their endpoints.
  • ZMQ Wire Codec (Native MsgPack Frame + Opaque User Metadata): Framework metadata—BatchMeta, field_schema (canonical dtype strings and integer shape arrays), and production/consumption status arrays—uses pure MsgPack in serial_utils.py instead of pickle (metadata.py), allowing Rust worker threads to decode requests and encode replies natively. Per-sample user dictionaries (custom_meta / tag and custom_backend_meta) are encoded by TransferQueueClient as self-contained MsgPack byte slices; the Rust metadata engine stores and splices the byte slices verbatim by row without invoking Python (inspecting scalar keys like total_lengths only when required by a built-in sampler), preserving arbitrary Python objects transparently.
  • Sampler Execution (Native Fast Path + On-Demand PyO3 Callback): Built-in samplers (SequentialSampler, GRPOGroupNSampler, RankAwareSampler, SeqlenBalancedSampler) run natively in Rust with zero GIL acquisition. User-defined Python BaseSampler subclasses run through a PyO3 adapter inside the same engine: all producer updates, KV_* calls, and unready fetch polls return purely in Rust without touching the GIL; Rust acquires the GIL only once a partition has at least batch_size ready samples to call sampler.sample(ready_indexes, batch_size, partition=PartitionView, **sampling_config), where PartitionView lazily decodes custom_meta only if queried.
  • Checkpointing & Prometheus/Grafana Observability:
    • Checkpointing: SAVE_CONTROLLER_CHECKPOINT and LOAD_CONTROLLER_CHECKPOINT (controller.py) write and restore a versioned binary snapshot of PartitionIndexManager and partition state directly from Rust, calling PyO3 once to include Python sampler.save_checkpoint() state when a custom sampler is active (with legacy .pkl import during migration).
    • Prometheus & Grafana: Python's TQMetricsExporter (metrics.py) remains the /metrics HTTP endpoint and continues scraping SimpleStorageUnit metrics. The Rust metadata engine records per-op_type counts and latency histograms in lock-free AtomicU64 buckets alongside partition and index gauges, syncing snapshots to TQMetricsExporter every TQ_METRICS_COLLECT_INTERVAL seconds so grafana_dashboard.json requires zero changes.

3. Concurrency and State Design

  • Decouple ZMQ I/O from parallel worker-pool codec. Non-blocking ZMQ I/O tasks read and write multipart frames on handshake_socket and request_handle_socket (controller.py), dispatching raw frames to a multi-threaded Rust worker pool so MsgPack decoding and BatchMeta encoding scale across CPU cores outside state locks.
  • Shard execution by partition_id over compact local_row bitsets. Requests targeting different partitions execute concurrently against per-partition state, while single-partition mutations remain atomic so readiness checks, sampling, and mark_consumed stay race-free. Cross-partition coordination is limited to O(1) per-unit FIFO queues in PartitionIndexManager (controller.py) and shared-field links on __public__. Within each partition, sparse global_index values map bidirectionally to dense local_row slots (0 .. N-1), storing field readiness and per-task consumption in u64 bitsets so polling scans reduce to bitwise AND over machine words while ZMQ replies keep global_index.

4. Migration and Retirement Plan

  1. PyO3 Skeleton & Pure-MsgPack BatchMeta Codec: Add native MsgPack encoding for BatchMeta and torch.dtype in serial_utils.py, bring up the PyO3 Rust metadata engine behind controller.backend="rust", and pass startup, HANDSHAKE, GET_LIST_PARTITIONS, and tq.close() tests. [BREAKING][feat] Use native MessagePack for Controller metadata #221
  2. Core Read/Write Loop & Concurrency Benchmark: Implement PartitionIndexManager, local_row bitsets, GET_META (insert, fetch, force_fetch), NOTIFY_DATA_UPDATE, cleanup RPCs, and built-in samplers. Verify behavioral parity against tests/test_controller.py and benchmark multi-client polling and write QPS/P99 latency.
  3. Full Feature Parity (KV, Custom Sampler Adapter, Checkpoint, Metrics): Port KV_* and LINK RPCs, add the PyO3 custom BaseSampler adapter, wire atomic metrics snapshots into TQMetricsExporter, and support checkpoint save/load (including legacy .pkl import).
  4. Default Flip & Legacy Cleanup: Ship prebuilt abi3 wheels in CI, flip controller.backend default from "python" to "rust" after pilot validation, and remove the legacy Python request loop and tensor state tables in the following release.

Activity

  1. self-assigned this
    on Nov 20, 2026
  2. 0oshowero0 commented on Nov 20, 2026

    @0oshowero0
    CollaboratorAuthor

    TL;DR:保持 TransferQueueController 作为命名 Ray Actor 的部署与发现方式不变,通过 PyO3(abi3)在同一 Actor 进程内嵌入多线程 Rust 元数据引擎,替代现有的单线程 Python ZMQ 请求循环。在当前全量非阻塞轮询机制下,消费方高频轮询 GET_META 叠加生产方 NOTIFY_DATA_UPDATE,全部挤在单条受 GIL 限制的 Python 线程中执行 pickle 编解码与稀疏 torch.Tensor 扫描。Rust 元数据引擎通过工作线程池并行编解码,并基于局部紧凑 u64 位图按分区并发处理请求;内置 Sampler 在 Rust 原生运行,自定义 Python BaseSampler 通过按需 PyO3 回调在同一引擎内兼容,同时无缝衔接现有 Checkpoint 与 Grafana 监控,灰度验证稳定后即可下线旧 Python 循环,无需长期维护两套 TransferQueueController。

    1. 背景与性能瓶颈

    tq.init() 仅利用 Ray 完成 TransferQueueController 的拉起、寻址与配置同步,运行期的元数据请求全部直接走 ZMQ(interface.py)。TransferQueueClient 和 StorageManager 获取 ZMQ 地址后直接向 TransferQueueController 收发元数据请求,样本 Tensor 数据则由各存储后端独立搬运。

    非阻塞消费轮询叠加生产写入,使高频 ZMQ 请求全部汇聚到单条 Python 线程(_process_request_loop,controller.py)上串行执行,在大规模场景下形成两处瓶颈。由于 GET_META(mode="fetch") 在就绪样本不足 batch_size 时立即返回空 BatchMeta(controller.py),等待数据的消费进程会持续轮询 TransferQueueController,与生产方的 NOTIFY_DATA_UPDATE、KV_* 请求并发竞争:

    1. 单线程 GIL 串行化:ZMQ 多帧收发、消息编解码、就绪状态扫描与索引分配均为内存级 CPU 操作,在 Python 内加线程或改用 asyncio 仍受同一把 GIL 串行限制。
    2. 稀疏 torch.Tensor 状态表与 pickle 编解码开销:每个分区直接以稀疏 global_index 为行下标分配长度为 max(global_index) + 1 的 torch.int8 状态矩阵(controller.py),连高频空轮询也要临时构造掩码 Tensor;同时 BatchMeta 与 torch.dtype 在传输层回退到 Python pickle(serial_utils.py)。

    2. 总体架构与 Python–Rust 工作界面

    保留命名 Ray Actor 负责集群调度与发现,采用 PyO3(abi3)同进程嵌入而非独立子进程拉起 Rust 元数据引擎。同进程扩展与 Ray Actor 共享生命周期及 RSS 内存统计,且能以微秒级开销回调用户自定义 Python BaseSampler,避免因自定义采样器而长期双写维护两套 TransferQueueController。

    flowchart TB
        Callers["调用方进程<br/>tq.init() · TransferQueueClient · StorageManager"]
        Backends["存储后端(数据面不变)<br/>SimpleStorage · MooncakeStore · Yuanrong"]
    
        subgraph Actor["TransferQueueController(单个 Ray Actor 进程)"]
            direction LR
            PyShell["Python 外壳(冷路径)<br/>Ray 发现与配置 · /metrics 导出 · 自定义 Sampler 适配"]
            RustEngine["Rust 元数据引擎(热路径)<br/>ZMQ ROUTER · 并行编解码 · 分区位图状态 · 内置 Sampler"]
            PyShell <-- "PyO3(启停 / 监控快照 / 按需采样回调)" --> RustEngine
        end
    
        Callers -- "Ray .remote()(启动、配置、地址发现)" --> PyShell
        Callers <-- "ZMQ 直连(高频轮询与元数据请求)" --> RustEngine
        Callers <-- "样本 Tensor 读写" --> Backends
    
    Loading

    Python 外壳与 Rust 元数据引擎的工作界面按四个维度明确切分:

    • Ray 启动与生命周期(Python 外壳):保留 TransferQueueController 的 Actor 名称、store_config / get_config、get_zmq_server_info 及关闭逻辑。Python 外壳在 __init__ 中通过 PyO3 启动 Rust 元数据引擎,等待 handshake_socket 与 request_handle_socket 两个 ZMQ ROUTER socket 绑定就绪后对外发布地址。
    • ZMQ 消息编解码(框架元数据原生化 + 用户元数据字节透传):在 serial_utils.py 中为 BatchMeta 和 torch.dtype 补齐纯 MsgPack 编码分支(如 dtype 编码为标准字符串、shape 编码为整数数组,不再回退 pickle,metadata.py),使 Rust 工作线程能够原生解码请求并编码响应。对于用户自定义的 custom_meta(tag)与 custom_backend_meta,TransferQueueClient 按样本编码为自包含的 MsgPack 字节块,Rust 元数据引擎按行原样存储字节块并在构造 BatchMeta 时直接拼回响应(仅在内置 SeqlenBalancedSampler 需要时读取 total_lengths 等标量),无需调用 Python 即可透明兼容任意用户 Python 对象。
    • Sampler 执行(内置原生快路径 + 自定义 PyO3 按需回调):内置采样器(SequentialSampler、GRPOGroupNSampler、RankAwareSampler、SeqlenBalancedSampler)在 Rust 元数据引擎内原生实现,全程零 GIL。用户自定义 Python BaseSampler 子类通过 PyO3 适配器接入同一引擎:所有写入通知、KV_* 请求以及数据未齐时的空轮询 fetch 仍在 Rust 内无 GIL 直接返回;仅当分区就绪样本达到 batch_size 时,Rust 元数据引擎才获取一次 GIL 调用 sampler.sample(ready_indexes, batch_size, partition=PartitionView, **sampling_config)(其中只读代理 PartitionView 仅在被调用 get_custom_meta() 时按需懒解码对应行数据)。
    • Checkpoint 与 Prometheus/Grafana 监控:
      • 状态保存与恢复:沿用现有 ZMQ SAVE_CONTROLLER_CHECKPOINT / LOAD_CONTROLLER_CHECKPOINT 协议(controller.py),由 Rust 元数据引擎直接读写 PartitionIndexManager 与分区状态的带版本头紧凑二进制快照;若启用了自定义 Python Sampler,保存与恢复时通过 PyO3 同步一次 sampler.save_checkpoint() 状态(灰度期兼容导入旧版 .pkl 快照)。
      • Grafana 监控无缝衔接:直接复用 Python 外壳中的 TQMetricsExporter(metrics.py)对外提供 /metrics HTTP 端点及 SimpleStorageUnit 指标采集。Rust 元数据引擎用无锁 AtomicU64 记录各 op_type 请求计数与固定分桶延迟直方图,连同分区和索引统计每 TQ_METRICS_COLLECT_INTERVAL 秒同步一次快照给 TQMetricsExporter,现有 grafana_dashboard.json 无需任何改动。

    3. 并发模型与状态设计

    • 解耦非阻塞 ZMQ 收发与多线程并行编解码。 独立的非阻塞 I/O 任务负责 handshake_socket 与 request_handle_socket 的多帧收发(controller.py),将原始帧分发至多线程 Rust 工作池,使 MsgPack 解码与 BatchMeta 编码在状态锁外利用多核并行执行。
    • 按 partition_id 分片并发读写紧凑 local_row 位图。 不同分区的请求在各自分区状态上并发执行;同一分区内的状态读写保持原子性,保证就绪扫描、采样与 mark_consumed 严格一致。跨分区同步仅限 PartitionIndexManager 的 O(1) 存储单元 FIFO 队列(controller.py)及 __public__ 共享字段引用检查。每个分区内部将稀疏 global_index 双向映射为稠密的 local_row(0 .. N-1),以 u64 位图存储字段就绪状态与各任务消费状态;高频轮询扫描简化为机器字按位与运算,对外响应和存储路由继续使用 global_index。

    4. 分阶段推进与旧实现下线计划

    1. PyO3 骨架与纯 MsgPack BatchMeta 编解码:在 serial_utils.py 中补齐 BatchMeta 与 torch.dtype 的纯 MsgPack 编码,打通 controller.backend="rust" 下的 PyO3 启停、HANDSHAKE、GET_LIST_PARTITIONS 与 tq.close() 生命周期。 [BREAKING][feat] Use native MessagePack for Controller metadata #221
    2. 核心读写链路与并发压测:实现 PartitionIndexManager、局部 local_row 位图、GET_META(insert、fetch、force_fetch)、NOTIFY_DATA_UPDATE、清理接口及内置 Sampler,对齐 tests/test_controller.py 语义并对比多客户端高频轮询与写入场景下的 QPS 及 P99 延迟。
    3. 全特性对齐(KV、自定义 Sampler 适配、Checkpoint、监控):迁移 KV_* 与 LINK 系列接口,接入 PyO3 自定义 BaseSampler 适配器、TQMetricsExporter 原子指标快照及 Checkpoint 读写(含旧版 .pkl 导入)。
    4. 默认切换与旧代码清理:在 CI 构建预编译 abi3 wheel,完成灰度验证后将 controller.backend 默认值切换为 "rust",并在下一版本移除旧 Python 请求循环与稀疏 Tensor 状态表代码。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions