Process succeeds when importing from fileystem yields empty results

[amotl]

Hi there. We just did some basic user testing and discovered a little flaw. With the following incantation, the transfer process succeeds, while I think it should fail because the source file foo/bar/baz.csv does not exist.

docker run --rm --publish 4566:4566 docker.io/floci/floci:latest

omniload ingest \
  --source-uri 's3://?endpoint_url=http://localhost:4566&access_key_id=test&secret_access_key=test' \
  --source-table 'foo/bar/baz.csv' \
  --dest-uri 'duckdb:///example.db' \
  --dest-table 'testdrive.test'

[amotl]

Can I humbly ask you to look into this, @hampsterx?
Do you have any suggestions already, @coderabbitai?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== candidate source files =='
fd -t f . | sed -n '1,240p'

printf '%s\n' '== ingest/S3/source-table references =='
rg -n -i -C 3 \
  '(?:\bingest\b|source[-_ ]table|source_uri|source-uri|s3://|S3|endpoint_url)' \
  -g '!**/node_modules/**' -g '!**/dist/**' -g '!**/build/**' . | sed -n '1,500p'

Length of output: 45332

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== implementation and test candidates =='
fd -t f . src/dlt_filesystem tests/dlt_filesystem | sort

printf '%s\n' '== outlines =='
for file in \
  src/dlt_filesystem/source/impl/remote.py \
  src/dlt_filesystem/source/base.py \
  src/dlt_filesystem/source/core.py \
  tests/dlt_filesystem/test_source_local.py
do
  if [ -f "$file" ]; then
    echo "--- $file"
    ast-grep outline "$file" --items all || true
  fi
done

printf '%s\n' '== relevant symbols and error behavior =='
rg -n -i -C 5 \
  '(?:class\s+(?:S3Source|.*Source)|def\s+(?:get|read|load|tables|resources)|glob\s*\(|exists\s*\(|FileNotFound|NoSuch|empty|yield)' \
  src/dlt_filesystem tests/dlt_filesystem | sed -n '1,700p'

Length of output: 59775

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

for spec in \
  'src/dlt_filesystem/source/impl/remote.py:90,205p' \
  'src/dlt_filesystem/source/adapter.py:90,145p' \
  'src/dlt_filesystem/source/core.py:1,115p' \
  'src/dlt_filesystem/source/model.py:1,220p' \
  'src/dlt_filesystem/error.py:1,180p' \
  'src/dlt_filesystem/source/error.py:1,180p' \
  'tests/dlt_filesystem/test_source_remote.py:1,360p'
do
  file="${spec%%:*}"
  range="${spec#*:}"
  range="${range#*:}"
  if [ -f "$file" ]; then
    echo "===== $file ($range) ====="
    sed -n "$range" "$file"
  fi
done

printf '%s\n' '== filesystem resource and locator call sites =='
rg -n -C 4 \
  '(?:filesystem\(|FilesystemLocator\(|infer_resource\(|readers\()' \
  src/dlt_filesystem tests/dlt_filesystem | sed -n '1,500p'

Length of output: 49437

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '== glob_files definition/imports =='
rg -n -C 8 \
  '(?:from .* import glob_files|def glob_files\b|glob_files\s*=)' \
  src tests | sed -n '1,360p'

printf '%s\n' '== adapter and source test fixtures =='
rg -n -C 6 \
  '(?:filesystem\(|resource_for_reader\(|ArrowFSWrapper|MemoryFileSystem|LocalFileSystem|pytest\.raises)' \
  tests/dlt_filesystem | sed -n '1,600p'

printf '%s\n' '== dependency versions =='
rg -n -C 2 '^\s*"?dlt|^\s*"?fsspec|^\s*"?s3fs' pyproject.toml

Length of output: 36129

---

@amotl, yes. I traced this to the shared filesystem source path.

S3CompatibleSource turns foo/bar/baz.csv into bucket foo plus glob bar/baz.csv. The filesystem lister then iterates glob_files(...); when no object matches, it yields nothing and raises nothing. The reader receives no input, so dlt treats the run as a successful empty ingestion.

Suggested fix

1. In src/dlt_filesystem/source/adapter.py, make the listing resource raise a dedicated error (for example, NoFilesMatchedError) when glob_files(...) finds zero files.
2. Include the bucket URL and requested glob in the error message.
3. Keep the check at file-discovery level—not on row counts—so an existing empty CSV remains valid and loads zero rows.
4. Add regression tests for:
   • nonexistent literal file → failure;
   • unmatched glob → failure;
   • existing empty file → success with zero rows.

Implementing this in the shared adapter fixes S3 and other filesystem-backed sources consistently, rather than patching only S3CompatibleSource.

🐇🔎

[amotl]

The filesystem lister then iterates glob_files(...); when no object matches, it yields nothing and raises nothing. The reader receives no input, so dlt treats the run as a successful empty ingestion.

@coderabbitai: Ah right, I suspected this already. Thanks for confirming.

Now, do we think it is good DWIM like this, or should omniload exit differently? I suggest to either fail on empty sources, or if that is too harsh, only fail if a concrete file is specified, i.e. when not using wildcards, if that is possible. I think it is acceptable to succeed if a wildcard match does not find any files, but not in other cases. wdyt?

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@amotl, I agree: fail for an unmatched concrete source path, but allow an unmatched glob to succeed with zero discovered files.

That is the better DWIM boundary:

• foo/bar/baz.csv means “load this required object” → fail if absent.
• foo/bar/*.csv means “load whatever currently matches” → zero matches can be legitimate.
• An existing but empty CSV still succeeds with zero rows.

Implement it at discovery time: detect whether the requested path contains filesystem glob metacharacters, then raise a clear FileNotFoundError/domain-specific error only when a non-glob resolves to no files. Include the original source URI/path in the message.

A small caveat: a non-glob path may also denote a directory/prefix in some source modes. If directory ingestion is supported, only fail after confirming that neither the exact object nor an applicable directory/prefix resolution exists.

🐇🔍

---

✏️ Learnings added

Learnt from: amotl
URL: https://github.com/panodata/omniload/issues/260

Timestamp: 2026-07-28T20:46:06.355Z
Learning: For Omniload filesystem-backed ingestion, an unmatched concrete source path should fail because it usually indicates a missing required input, while an unmatched wildcard/glob source may validly succeed with zero discovered files. This decision must be made during file discovery rather than based on row counts, so existing empty files remain valid zero-row loads.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[amotl]

Maybe adding an --accept-empty-source command-line option is required to better tell things apart? Of course, it would be so sweet without.

[amotl]

@hampsterx: Can you possibly improve this topic with support from your clankers if you agree that improvement is a good idea, in the same spirit of thoroughness like all your other patches, solving all possible edge cases in a single patch without much ado? 🙏

[amotl]

@hampsterx: Thank you for submitting a patch right away. 👍

• fix(filesystem): fail missing concrete sources #265

Another upstream issue and patch read remotely related to this topic.

• not all globstrings are supported in open_files fsspec/filesystem_spec#957
• Expand ?/[] globs for a single-string path in read mode fsspec/filesystem_spec#2080