Skip to content

Mirror readinto from the pyarrow stream onto ArrowFile - #2111

Open
hampsterx wants to merge 1 commit into
fsspec:masterfrom
hampsterx:fix/arrow-file-readinto
Open

Mirror readinto from the pyarrow stream onto ArrowFile#2111
hampsterx wants to merge 1 commit into
fsspec:masterfrom
hampsterx:fix/arrow-file-readinto

Conversation

@hampsterx

Copy link
Copy Markdown

ArrowFile mirrors a fixed method list from the pyarrow stream it wraps, and readinto is not on
it, though pyarrow.NativeFile implements it. That makes it the one file class here without one:
AbstractBufferedFile implements readinto (fsspec/spec.py), so anything written
against the usual fsspec handle contract breaks on an Arrow-backed filesystem and nowhere else.

LibArchiveFileSystem is the in-repo case, and it fails silently. custom_reader documents that
"the file object must support the standard readinto and 'seek' methods" and calls it from a
ctypes callback. An AttributeError there is swallowed by ctypes, libarchive sees a zero-length
read, and ls() returns an empty archive rather than raising:

>>> fs = ArrowFSWrapper(LocalFileSystem())
>>> LibArchiveFileSystem(fo=fs.open("a.tar", "rb")).ls("", detail=False)
Exception ignored on calling ctypes callback function ...
AttributeError: 'ArrowFile' object has no attribute 'readinto'
[]                      # on master; ['hello.txt'] with this patch

(The swallow itself is pre-existing and independent of this patch: any readinto failure in that
callback yields an empty archive rather than an error. This just removes the trigger.)

Buffered reads raise properly at least:

>>> io.BufferedReader(fs.open("a.txt", "rb")).read(3)
AttributeError: 'ArrowFile' object has no attribute 'readinto'

A whole-file read() is fine, since that path goes to read() on the raw object. A sized read,
peek() and read1() each fill the buffer through readinto.

Gzip is how it turned up. fsspec/compression.py registers isal's IGzipFile as the gzip codec
when isal imports and the stdlib GzipFile only when it does not, and isal decompresses through
readinto. So a .gz file read through any Arrow-backed filesystem fails in an environment that
merely carries the package, and reads fine everywhere else. A transitive xopen dependency brought
isal in on x86-64 and AArch64, and every .gz read started raising.

One entry on the mirror list, following #1154 (seekable) and #1944 (size).

Tests

test_readinto covers the handle directly on both the seekable and non-seekable paths;
test_readinto_supports_a_buffered_reader is parametrized over read(3), peek(3) and read1(3).
All eight cases fail on master with the AttributeError above and pass with the one-line change.

Neither the gzip nor the libarchive case is asserted: isal is not guaranteed on CI and libarchive is
optional, and a combined libarchive + pyarrow fixture does not exist here. Both were reproduced by
hand against master and this branch. One note for anyone reproducing the buffered cases: the
pure-Python _pyio.BufferedReader uses raw.read for sized reads and so does not show the bug,
only the C io.BufferedReader does.

A downstream workaround, for anyone hitting this before a release: panodata/omniload#314 mixes a
_open override into its Arrow-backed filesystem classes to expose the stream's own readinto.

`ArrowFile` mirrors a fixed method list from the pyarrow stream it wraps and
`readinto` is not on it, though `pyarrow.NativeFile` implements it. That leaves
it the one file class here without one, so code written against the
`AbstractBufferedFile` contract breaks on an Arrow-backed filesystem and
nowhere else.

`LibArchiveFileSystem` is the in-repo case, and it fails silently. Its
`custom_reader` calls `readinto` from a ctypes callback, which swallows the
`AttributeError`, so libarchive sees a zero-length read and `ls()` returns an
empty archive instead of raising. Buffered reads at least raise: a sized `read`,
`peek` and `read1` each fill the buffer through `readinto`, while an unsized
`read()` falls back to `read(-1)` and is unaffected.

Gzip is how it turned up. `fsspec.compression` registers isal's `IGzipFile` as
the `gzip` codec when `isal` imports, and that reader decompresses through
`readinto`, so reading a `.gz` file through an Arrow-backed filesystem fails in
an environment that happens to carry the package and reads fine everywhere
else.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant