Skip to content

[Perf]: Reuse Mooncake buffer pool for RDMA puts - #216

Open
chenjr97 wants to merge 1 commit into
Ascend:mainfrom
chenjr97:chenjr/put_buffer
Open

chenjr97 wants to merge 1 commit into
Ascend:mainfrom
chenjr97:chenjr/put_buffer

Conversation

@chenjr97

@chenjr97 chenjr97 commented Nov 11, 2026 •

Copy link
Copy Markdown

overview

Extend the pre-registered buffer pool approach from #175, which optimized GET, to CPU RDMA PUT.

Implementation

  • Reuse the existing Mooncake BufferPool, backed by the local buffer registered during store setup.
  • Lease a buffer, copy payloads into it, and release it after synchronous batch upsert and any retries complete.
  • Copy uniform, contiguous tensor groups in one strided operation and ragged groups per tensor. Route serialized non-tensor values through the same PUT worker.
  • Fall back to direct registration when the pool is missing or unavailable, a group exceeds the per-worker limit, or a batch contains duplicate keys.

Evaluation

Setup

Benchmarks ran on a single node with the PUT client and storage segment colocated, using RDMA with either 1 or 8 HCAs. Each HCA port reported an active Ethernet link rate of 200 Gb/s.

Setting Value
Mooncake 0.3.12.post1
Payload Dense CPU uint8 tensors
Local buffer / storage segment 10 GiB / 16 GiB
Batch size limit / maximum batch workers 400 / 4
Samples per variant, case, and HCA configuration 5 warmups + 10 measured runs

Latency measures await client.async_put(metadata=None), including metadata, storage, and controller work. Payload preparation, initialization, subsequent GET verification, and partition clearing run outside the timed region. The figure below presents the complete benchmark results.

PUT latency

The table below includes the single-key, 1 GiB total, and largest tested cases for each payload size, with duplicates removed. Speedup is baseline median latency divided by pooled median latency.

Package size / key Keys / PUT Total / PUT 1 HCA speedup 1 HCA reg + unreg saved (ms) 8 HCA speedup 8 HCA reg + unreg saved (ms)
4 MiB 256 1 GiB 1.40× 36.98 2.20× 91.13
16 MiB 64 1 GiB 1.53× 40.57 2.09× 74.87
16 MiB 256 4 GiB 1.36× 162.34 3.22× 475.54
64 MiB 16 1 GiB 1.66× 44.66 3.50× 93.62
64 MiB 64 4 GiB 1.47× 177.99 2.75× 509.66
64 MiB 128 8 GiB 1.55× 386.22 3.20× 1011.39
256 MiB 4 1 GiB 1.95× 84.01 3.26× 102.36
256 MiB 16 4 GiB 1.47× 196.08 3.29× 506.37
256 MiB 32 8 GiB 1.65× 412.80 3.27× 998.25
1024 MiB 1 1 GiB 1.45× 71.00 3.02× 155.74
1024 MiB 4 4 GiB 1.45× 198.54 3.08× 491.34
1024 MiB 8 8 GiB 1.46× 404.62 2.97× 948.90
put_buffer_latency

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T16:45:24.878143Z 7a5245b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a5245b2dd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".


batch_ptrs, batch_sizes, _ = self._preprocess_tensors_for_put(batch_tensors)
batch_ptrs, batch_sizes, batch_tensors = self._preprocess_tensors_for_put(batch_tensors)
if self._buffer_pool is not None and len(set(batch_keys)) == len(batch_keys):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep NPU tensors on the direct registration path

When protocol="rdma" is used with an Ascend/NPU TensorDict, _preprocess_tensors_for_put preserves NPU tensors (it only calls .cpu() for CUDA), yet this unconditional pool branch sends them to _copy_between_buffer_and_tensors, whose torch.frombuffer destination is CPU memory. The previous path registered each NPU data_ptr() directly (and _bind_npu_device exists to make those registrations work); now every pool-sized NPU value pays a synchronous device-to-host copy and loses the direct-device transfer path, while only oversized or pool-exhausted groups retain the old behavior. Restrict pooling to CPU-backed tensors so NPU puts keep the registered-pointer path.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chase-Rong Please help to verify whether this comment is true?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chase-Rong Please help to verify whether this comment is true?

Thanks for flagging this. I'm not very familiar with the NPU path and don't have NPU hardware available to verify this behavior. If confirmed, we can restrict pooling to CPU tensors and let NPU tensors fall back to the existing register/unregister path.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chase-Rong Please help to verify whether this comment is true?

OK, I can verify this

@chenjr97 chenjr97 changed the title [perf]:Reuse Mooncake buffer pool for RDMA puts [perf]: Reuse pre-registered buffers for Mooncake RDMA puts Nov 11, 2026
@chenjr97 chenjr97 changed the title [perf]: Reuse pre-registered buffers for Mooncake RDMA puts [perf]: Reuse Mooncake buffer pool for RDMA puts Nov 11, 2026
@chenjr97 chenjr97 changed the title [perf]: Reuse Mooncake buffer pool for RDMA puts [Perf]: Reuse Mooncake buffer pool for RDMA puts Nov 11, 2026

@0oshowero0 0oshowero0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Review] Pooled PUT send-buffer lifetime

except Exception as release_error:
logger.warning(f"Returning the pooled send buffer failed: {release_error}")
raise
lease.release()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Review] Keep the send buffer leased until RDMA has finished using it

In Mooncake 0.3.12.post1, used for the benchmark, a transfer can still be pending when the 60-second wait returns. The classic freeBatchID() returns BatchBusy for unfinished tasks, and the destructor discards that result. When this code releases the lease, the next PUT group can immediately overwrite the same registered source address. If a queued write from the old group is posted later, it can send the new group's bytes to the old destination. Timeout path · Batch cleanup

Mooncake #4491 reproduced a late write overwriting a reused destination under RDMA fault injection. I verified the immediate source address reuse with a fake store. The combined scenario still needs hardware verification. Mooncake #4491

Before enabling pooled PUT by default, please establish that batch_upsert_from() leaves the source buffer safe to reuse after every attempt, including an attempt that times out before a retry succeeds. Mooncake can also retain the lease until all attempts have stopped accessing the source.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review.

Mooncake < 0.3.13 (incl. 0.3.12.post1 used in the benchmark): a native wait timeout is returned to TQ as success, so TQ never retries. This doesn't depend on pooling and affects main too. Details in #218.

Mooncake >= 0.3.13: the timed-out key returns -800, and _batch_upsert_with_retry retries it from the same buffer. _write_via_buffer_pool keeps the lease across all attempts and releases it only after the final result, so the buffer is not reused while retries are running. The case above, where a later group overwrites the source, cannot happen during retry.

Given this, I suggest raising the Mooncake lower bound from >=0.3.10.post2 to >=0.3.13.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[AI Review] Keep the send buffer leased until timed-out transfers have stopped

I agree that raising the minimum Mooncake version to 0.3.13 addresses #218. TQ also correctly keeps the same lease throughout the retry loop.

The remaining case is an initial attempt that times out, followed by a successful retry. TQ then releases the lease, and the next group in the same PUT can overwrite the source address. If the first attempt reads that address later, it can send the next group's bytes. Mooncake 0.3.14 defers cleanup of the native batch descriptor. TQ still releases its lease when the retry loop returns. Mooncake deferred cleanup

I confirmed immediate source-address reuse with a fake store after both a successful retry and retry exhaustion. Mooncake #4491 demonstrated a late write to a reused remote destination under RDMA fault injection. Corruption caused by local source-address reuse in #216 still needs hardware verification. Mooncake #4491

Before enabling pooled PUT by default, please ensure that the lease stays active until every attempt has stopped accessing the source buffer. If Mooncake can return earlier, the native operation needs to retain the lease until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants