Skip to content

[refactor, transfer] Introduce PayloadTransfer abstraction for SimpleStorage - #214

Open
shun001 wants to merge 3 commits into
Ascend:mainfrom
shun001:208-nixl
Open

shun001 wants to merge 3 commits into
Ascend:mainfrom
shun001:208-nixl

Conversation

@shun001

@shun001 shun001 commented Nov 11, 2026 •

Copy link
Copy Markdown

Summary

Introduce a PayloadTransfer interface for SimpleStorage and adapt the
existing ZMQ backend to it.

PUT, GET, and UPDATE now use a shared request interface. Requests pass
unencoded field values to the backend, and GET with recv=None returns
decoded field values directly. This preserves the existing codec path
without introducing an additional encode/decode round trip.

The adapter reuses the existing ZMQ multipart protocol and socket pool.
Storage-side parsing, reads, writes, and updates run through a shared
StorageUnit handler, providing an entry point for additional transport
backends.

This change follows the interface direction proposed in
RFC #208.

Changes

  • Add request headers, response headers, and transport error handling.
  • Move ZMQ payload request construction and socket leasing into
    ZmqPayloadTransfer.
  • Route PUT, GET, and UPDATE through the shared storage handler.
  • Preserve Manager-side batching, routing, and result assembly.

Validation

Functional and fault recovery tests

The focused suite passed on both Linux hosts, with 77 tests passing
on each host
. It covers the adapter, socket pool, Manager retry
behavior, and shared ZMQ context.

Additional cross-host fault injection passed in both directions between
the two hosts, covering:

  • Delayed replies arriving after a timeout.
  • Server socket closure and recovery after rebinding the same port.
  • Request cancellation while a reply remains outstanding.
  • An UPDATE applied successfully with its reply deliberately dropped.
  • Eight concurrent GET requests after faults with a pool capacity of one.

The checks confirmed that failed sockets were discarded, late replies
were isolated from subsequent requests, and the pool remained usable.
The lost-reply UPDATE was applied once, reported a timeout, and its
result was verified by subsequent GET requests.

Faults were injected through an isolated ROUTER service using the
production codec, adapter, and storage handler.

Performance comparison

The dual-node comparison used identical benchmark code and
configuration for main and this implementation.

  • Writer and reader on one host; two StorageUnits on the other.
  • CPU data, eight threads, metrics disabled, and a 1 Gb/s network.
  • Four workloads: 1 MiB, 288 MiB, 1 GiB, and approximately 240 MiB of
    mixed tensor and string fields.
  • Six steady-state samples per version and workload.
Workload PUT median latency change GET median latency change
1 MiB +0.20% +1.19%
288 MiB −0.09% −0.98%
1 GiB −0.02% −0.09%
Mixed fields +0.43% −0.19%

Positive values indicate higher latency. Median changes remained within
−0.98% to +1.19% across the tested operations and workloads. The largest
mean latency increase was 3.09% for mixed-field PUT.

No material end-to-end performance regression was observed in the
tested dual-node configuration.
These measurements describe the
current network and workloads; large transfers are constrained by the
1 Gb/s link.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T07:12:10.015103Z d394bf9 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d394bf99bd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

raise TransferTimeout("NIXL storage request timed out", wire_bytes=wire_bytes)
await asyncio.sleep(0.001)
response = reply.result()
except Exception as exc:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Treat cancelled writes as outcome-unknown after sending

When a PUT/UPDATE task is cancelled after send_notification but before the reply, asyncio.CancelledError inherits BaseException and therefore bypasses this except Exception; the storage unit may still apply the write while the caller sees only cancellation. A caller using asyncio.wait_for and retrying can consequently apply a non-idempotent UPDATE twice, so this post-send cancellation path needs to finish determining the outcome or surface WriteOutcomeUnknown just like the timeout path.

Useful? React with 👍 / 👎.

logger.exception("NIXL storage request %s failed", request_id)
reply["error"] = str(exc)[:4096]
try:
packed = self._pack_control(reply)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prevent oversized update acknowledgements after committing

When an UPDATE produces enough per-sample stored_shapes to make its reply exceed MAX_CONTROL_BYTES (1 MiB), _handle has already called apply_update before this packing attempt. _pack_control then falls into the outcome-unknown reply, so AsyncSimpleStorageManager.update_data raises before notify_data_update; for a newly added or shape-changing field, storage is committed while controller metadata remains stale. Large valid updates therefore deterministically leave the planes inconsistent; reject or preflight before applying, or carry the acknowledgement body outside the capped notification.

Useful? React with 👍 / 👎.

@ji-huazhong

Copy link
Copy Markdown
Collaborator

cc @Ginray

@shun001 shun001 changed the title [WIP]Support nixl PayloadTransfer for SimpleStorage Support ZMQ PayloadTransfer for SimpleStorage Nov 20, 2026
@0oshowero0 0oshowero0 changed the title Support ZMQ PayloadTransfer for SimpleStorage [refactor, transfer] Introduce PayloadTransfer abstraction for SimpleStorage Nov 20, 2026
Signed-off-by: lanzeshun <962034936@qq.com>
Signed-off-by: lanzeshun <962034936@qq.com>
Signed-off-by: lanzeshun <962034936@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants