[OneSizeFitsQuorum]

JIRA: https://issues.apache.org/jira/browse/THRIFT-6436
Client: py

TTransportBase.readAll repeatedly copies the accumulated bytes when a read completes through multiple short reads. Collect chunks and join them once instead. A short inline comment explains why chunks are joined once.

The patch is limited to the accumulation loop and seven regression tests in the existing transport test file. They cover fragmented and single reads, remaining request sizes, nonpositive sizes, EOF, exception propagation, consecutive reads, and a fragmented frame.

Local performance validation

A separate Python TSimpleServer process returned a prebuilt 30 MiB binary payload over loopback TCP, using TBinaryProtocol and TFramedTransport. Only the client's readAll implementation differed. Each implementation had seven alternating rounds of three measured RPCs, with one warm-up per round. Values below are medians over 21 RPCs; timing excludes connection setup and payload verification. Frame/string limits were explicitly configured for the large test response.

Environment: CPython 3.10.12, Linux 6.8.0-136-generic x86_64, Intel Xeon Platinum 8362 @ 2.80 GHz, eight logical CPUs visible. Baseline: db8caa7.

Socket read mode	RPC wall time, before → after	Client CPU time, before → after	Payload throughput, before → after
Unrestricted	76.44 → 50.96 ms	39.56 → 14.88 ms	392.5 → 588.7 MiB/s
Requests capped at 64 KiB	732.53 → 49.96 ms	695.42 → 15.30 ms	41.0 → 600.5 MiB/s

Unrestricted frame-body read counts ranged from 11–14 before and 136–437 after, so fragmentation was not identical. The explicitly controlled 64 KiB case used exactly 481 body reads for both implementations. These are local synthetic-service measurements, not HBase throughput results or predictions for other deployments. The RPC harness and raw results remain local.

Standalone reproduction of fragmented-read cost

This is an in-memory check of the accumulation cost, independent of the TCP measurements above. After checking out this PR:

(cd lib/py && python3 setup.py build)
git show HEAD^:lib/py/src/transport/TTransport.py > /tmp/readall-before.py
PYTHONPATH=lib/py/test python3 - <<'PY'
import importlib.util
import statistics
import timeit

import _import_local_thrift
from thrift.transport.TTransport import TTransportBase

spec = importlib.util.spec_from_file_location('baseline', '/tmp/readall-before.py')
baseline = importlib.util.module_from_spec(spec)
spec.loader.exec_module(baseline)
data = bytes(range(256)) * (30 * 1024 * 1024 // 256)
width = (len(data) + 54) // 55
chunks = [data[i:i + width] for i in range(0, len(data), width)]


class Fragmented(TTransportBase):
    def __init__(self):
        self.chunks = iter(chunks)

    def read(self, size):
        return next(self.chunks, b'')


for label, implementation in (
    ('before', baseline.TTransportBase.readAll), ('after', TTransportBase.readAll)
):
    assert implementation(Fragmented(), len(data)) == data
    times = timeit.repeat(lambda: implementation(Fragmented(), len(data)), number=1, repeat=7)
    print(label, round(statistics.median(times) * 1000, 2), 'ms')
PY

Validation

• The seven new regression tests were added and run against the original implementation before applying the runtime change.

• All 14 transport tests passed after the change, along with the local TCP A/B measurements above.

• All 26 Python-specific CI checks passed for the final commit, covering Python 3.10–3.14 on Linux and the macOS matrix, plus Python static analysis.

• flake8 --max-line-length=100, codespell and git diff --check passed for the final patch.

• make style is unavailable in this unconfigured checkout because there is no root Makefile.

• CI environment note: the Smalltalk setup job fails while downloading the Pharo image (curl exit 7), before its tests run. Repository administrator permissions are required to rerun that job.

Generated-by: OpenAI Codex (GPT-6)

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open issues

• struct.pack is used in the new test but there is no import struct shown in the updated imports.… · New
• The benchmark assumes only the 4-byte frame header uses readAll(4) and treats any `readAll(sz <=… · New
• spec_from_file_location() can return None, and spec.loader can be None (e.g., if… · New

What changed in this PR

Improves Python client transport performance by making TTransportBase.readAll accumulate chunks and join() once, avoiding repeated bytes concatenation for fragmented reads.

Changes:

• Refactor TTransportBase.readAll to collect read chunks in a list and b''.join() at the end.
• Add regression tests for readAll behavior across fragmentation, EOF, exceptions, and framed payloads.
• Add a loopback RPC benchmark to compare baseline vs updated readAll behavior and record socket read fragmentation.

File	Description
lib/​py/​src/​transport/​TTransport.py	Switch readAll from incremental bytes concatenation to list accumulation + join for better performance on fragmented reads.
lib/​py/​test/​thrift_transport.py	Add regression tests and a small fragmented transport helper to verify readAll semantics.
lib/​py/​test/​benchmark_readall.py	Add a standalone benchmark to measure before/after readAll behavior over a framed RPC on loopback TCP.

_{Review effort: Lite}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

6 open issues

• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,… · New
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,… · New
• struct.pack is used in the new test but there is no import struct shown in the updated imports.…
• server.kill() is not available on older Python 3 versions (it was added after the earliest 3.x… · New
• spec_from_file_location() can return None, and spec.loader can be None (e.g., if…
• The benchmark assumes only the 4-byte frame header uses readAll(4) and treats any `readAll(sz <=…

_{Review effort: Lite}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

7 open issues

• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• struct.pack is used in the new test but there is no import struct shown in the updated imports.…
• This changes the accepted element types in readAll: b''.join(...) accepts general bytes-like… · New
• server.kill() is not available on older Python 3 versions (it was added after the earliest 3.x…
• spec_from_file_location() can return None, and spec.loader can be None (e.g., if…
• The benchmark assumes only the 4-byte frame header uses readAll(4) and treats any `readAll(sz <=…

_{Review effort: Lite}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

10 open issues

• struct.pack is used but this diff doesn't add an import struct in the module imports shown… · New
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• struct.pack is used in the new test but there is no import struct shown in the updated imports.…
• This changes the accepted element types in readAll: b''.join(...) accepts general bytes-like…
• server.kill() is not available on older Python 3 versions (it was added after the earliest 3.x…
• spec_from_file_location() can return None, and spec.loader can be None (e.g., if…
• The benchmark assumes only the 4-byte frame header uses readAll(4) and treats any `readAll(sz <=…
• The new comment states a behavioral requirement about mutation, but read() in Python transports… · New
• On EOF (chunkLen == 0), the code appends an empty chunk to buff and then immediately raises.… · New

_{Review effort: Lite}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

• The inline comment at line 86 is misleading where it sits (inside the loop) because the join… · New
• self.assertIs(type(result), bytes) is an unidiomatic type assertion in unittest. Prefer… · New

10 resolved since last review

• struct.pack is used but this diff doesn't add an import struct in the module imports shown…
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• test_benchmark_readall.py (and benchmark_readall.py) uses Python 3-only stdlib features (e.g.,…
• struct.pack is used in the new test but there is no import struct shown in the updated imports.…
• This changes the accepted element types in readAll: b''.join(...) accepts general bytes-like…
• server.kill() is not available on older Python 3 versions (it was added after the earliest 3.x…
• spec_from_file_location() can return None, and spec.loader can be None (e.g., if…
• The benchmark assumes only the 4-byte frame header uses readAll(4) and treats any `readAll(sz <=…
• On EOF (chunkLen == 0), the code appends an empty chunk to buff and then immediately raises.…
• The new comment states a behavioral requirement about mutation, but read() in Python transports…

_{Review effort: Lite}

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

No open issues 🎉

2 resolved since last review

• self.assertIs(type(result), bytes) is an unidiomatic type assertion in unittest. Prefer…
• The inline comment at line 86 is misleading where it sits (inside the loop) because the join…

_{Review effort: Lite}