Implement native adjacent byte range coalescing in cat_ranges

[yuxin00j]

Summary

This PR implements native byte range coalescing for GCSFileSystem.cat_ranges and ExtendedGcsFileSystem.cat_ranges to optimize chunk-based workload downloads (such as PyTorch Distributed Checkpoint / DCP payload loading).

Adjacent and near-adjacent byte requests on the same object are coalesced into singular block reads natively, reducing request amplification and leveraging zero-copy memoryview slicing.

Key Changes

• Core Coalescing & Unpacking Helpers (gcsfs/core.py):

  • _coalesce_ranges: Groups and merges overlapping, contiguous (max_gap=0), and near-contiguous (gap <= max_gap) range requests for a given file, generating relative slice offsets mapped to caller indices.
  • _merge_file_ranges & _is_coalesce_enabled: Unifies conditional coalescing vs 1-to-1 range mapping.
  • _unpack_range_results: Safely populates caller results with bytes when coalescing is inactive (max_gap is None or < 0) or memoryview slices when active (max_gap >= 0).
  • Safely handles bounded and unbounded ranges (end=None), empty inputs, scalar broadcasting, and separate non-overlapping clusters.

• GCSFileSystem._cat_ranges (gcsfs/core.py):

  • Adds native support for coalesced byte range reads across multiple files.
  • Executes chunked reads using asyn._run_coros_in_chunks respecting batch_size.
  • Normalizes iterable/generator inputs for starts and ends.
  • Supports both on_error="return" and on_error="raise" error modes.

• ExtendedGcsFileSystem._cat_ranges (gcsfs/extended_gcsfs.py):

  • Bulk Zonal Downloads via MRD: Connects range coalescing with Zonal (RAPID) buckets via AsyncMultiRangeDownloader (MRD).
  • Dynamic Multi-MRD Pool Concurrency: Sizes pool_size = min(max_batches, max(1, concurrency)) to stream multiple batches in parallel across pooled MRDs, avoiding head-of-line blocking and single-stream throughput bottlenecks on large multi-gigabyte transfers.
  • Unified Coroutine Scheduling: Combines Zonal MRD batch downloads and Non-Zonal HTTP GETs into a single coroutine pool scheduled via asyn._run_coros_in_chunks, providing global concurrency throttling.
  • Lazy Metadata Lookup: Queries object metadata (_info()) only when unbounded ranges (end=None or negative offsets) require file size resolution.
  • Resource Cleanup: Ensures all acquired MRDPool instances are cleanly closed in finally blocks upon completion or failure.

Testing

• Unit Tests in gcsfs/tests/test_core.py:
  • Parameterized tests for _coalesce_ranges covering single, contiguous, overlapping, unordered, unbounded, and multi-cluster cases.
  • Integration tests for input validation, scalar broadcasting, contiguous coalescing (max_gap=0), multi-file coalescing, and error modes.
• Unit Tests in gcsfs/tests/test_extended_gcsfs.py:
  • Tests for non-zonal delegation, zonal coalescing with MockMRD, mixed zonal/non-zonal paths in a single call, partial batch failure preservation, and lazy metadata validation.
  • Tests for dynamic MRD pool concurrency, chunk-based batch size limits via _run_coros_in_chunks, return types (bytes vs memoryview), and pool cleanup upon error.

[gemini-code-assist]

Code Review

This pull request implements _cat_ranges and cat_ranges in both GCSFileSystem and ExtendedGcsFileSystem to fetch multiple byte ranges efficiently, supporting range coalescing and leveraging AsyncMultiRangeDownloader for zonal buckets. It also adds comprehensive unit tests. The review feedback suggests extracting the duplicated input validation logic into a shared helper and removing an unnecessary fallback default value of 64 for the batch size.

[codecov]

Codecov Report

❌ Patch coverage is 93.72385% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.26%. Comparing base (8a3d3f5) to head (2e472db).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
gcsfs/core.py	94.73%	8 Missing ⚠️
gcsfs/extended_gcsfs.py	91.95%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1024      +/-   ##
==========================================
+ Coverage   90.10%   90.26%   +0.15%     
==========================================
  Files          16       16              
  Lines        3679     3902     +223     
==========================================
+ Hits         3315     3522     +207     
- Misses        364      380      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[yuxin00j]

/gcbrun

[yuxin00j]

/gcbrun