Repository navigation
Conversation
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a5245b2dd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| batch_ptrs, batch_sizes, _ = self._preprocess_tensors_for_put(batch_tensors) | ||
| batch_ptrs, batch_sizes, batch_tensors = self._preprocess_tensors_for_put(batch_tensors) | ||
| if self._buffer_pool is not None and len(set(batch_keys)) == len(batch_keys): |
There was a problem hiding this comment.
Keep NPU tensors on the direct registration path
When protocol="rdma" is used with an Ascend/NPU TensorDict, _preprocess_tensors_for_put preserves NPU tensors (it only calls .cpu() for CUDA), yet this unconditional pool branch sends them to _copy_between_buffer_and_tensors, whose torch.frombuffer destination is CPU memory. The previous path registered each NPU data_ptr() directly (and _bind_npu_device exists to make those registrations work); now every pool-sized NPU value pays a synchronous device-to-host copy and loses the direct-device transfer path, while only oversized or pool-exhausted groups retain the old behavior. Restrict pooling to CPU-backed tensors so NPU puts keep the registered-pointer path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
@Chase-Rong Please help to verify whether this comment is true?
There was a problem hiding this comment.
@Chase-Rong Please help to verify whether this comment is true?
Thanks for flagging this. I'm not very familiar with the NPU path and don't have NPU hardware available to verify this behavior. If confirmed, we can restrict pooling to CPU tensors and let NPU tensors fall back to the existing register/unregister path.
There was a problem hiding this comment.
@Chase-Rong Please help to verify whether this comment is true?
OK, I can verify this
0oshowero0
left a comment
There was a problem hiding this comment.
[AI Review] Pooled PUT send-buffer lifetime
| except Exception as release_error: | ||
| logger.warning(f"Returning the pooled send buffer failed: {release_error}") | ||
| raise | ||
| lease.release() |
There was a problem hiding this comment.
[AI Review] Keep the send buffer leased until RDMA has finished using it
In Mooncake 0.3.12.post1, used for the benchmark, a transfer can still be pending when the 60-second wait returns. The classic freeBatchID() returns BatchBusy for unfinished tasks, and the destructor discards that result. When this code releases the lease, the next PUT group can immediately overwrite the same registered source address. If a queued write from the old group is posted later, it can send the new group's bytes to the old destination. Timeout path · Batch cleanup
Mooncake #4491 reproduced a late write overwriting a reused destination under RDMA fault injection. I verified the immediate source address reuse with a fake store. The combined scenario still needs hardware verification. Mooncake #4491
Before enabling pooled PUT by default, please establish that batch_upsert_from() leaves the source buffer safe to reuse after every attempt, including an attempt that times out before a retry succeeds. Mooncake can also retain the lease until all attempts have stopped accessing the source.
There was a problem hiding this comment.
Thanks for the review.
Mooncake < 0.3.13 (incl. 0.3.12.post1 used in the benchmark): a native wait timeout is returned to TQ as success, so TQ never retries. This doesn't depend on pooling and affects main too. Details in #218.
Mooncake >= 0.3.13: the timed-out key returns -800, and _batch_upsert_with_retry retries it from the same buffer. _write_via_buffer_pool keeps the lease across all attempts and releases it only after the final result, so the buffer is not reused while retries are running. The case above, where a later group overwrites the source, cannot happen during retry.
Given this, I suggest raising the Mooncake lower bound from >=0.3.10.post2 to >=0.3.13.
There was a problem hiding this comment.
[AI Review] Keep the send buffer leased until timed-out transfers have stopped
I agree that raising the minimum Mooncake version to 0.3.13 addresses #218. TQ also correctly keeps the same lease throughout the retry loop.
The remaining case is an initial attempt that times out, followed by a successful retry. TQ then releases the lease, and the next group in the same PUT can overwrite the source address. If the first attempt reads that address later, it can send the next group's bytes. Mooncake 0.3.14 defers cleanup of the native batch descriptor. TQ still releases its lease when the retry loop returns. Mooncake deferred cleanup
I confirmed immediate source-address reuse with a fake store after both a successful retry and retry exhaustion. Mooncake #4491 demonstrated a late write to a reused remote destination under RDMA fault injection. Corruption caused by local source-address reuse in #216 still needs hardware verification. Mooncake #4491
Before enabling pooled PUT by default, please ensure that the lease stays active until every attempt has stopped accessing the source buffer. If Mooncake can return earlier, the native operation needs to retain the lease until then.
overview
Extend the pre-registered buffer pool approach from #175, which optimized GET, to CPU RDMA PUT.
Implementation
BufferPool, backed by the local buffer registered during store setup.Evaluation
Setup
Benchmarks ran on a single node with the PUT client and storage segment colocated, using RDMA with either 1 or 8 HCAs. Each HCA port reported an active Ethernet link rate of 200 Gb/s.
0.3.12.post1uint8tensorsLatency measures
await client.async_put(metadata=None), including metadata, storage, and controller work. Payload preparation, initialization, subsequent GET verification, and partition clearing run outside the timed region. The figure below presents the complete benchmark results.PUT latency
The table below includes the single-key, 1 GiB total, and largest tested cases for each payload size, with duplicates removed. Speedup is baseline median latency divided by pooled median latency.