eval-under

Reproducible test harness for running arbitrary commands under a temporarily-mounted filesystem. Purpose-built for catching filesystem-specific behaviour bugs (rename semantics, close-on-exec, timestamp granularity, locking, permissions, adjusted-branch fallbacks, root_squash interactions) in tools like git-annex, DataLad, and rsync -- classes of bugs that don't show up in plain-ext4 CI.

git-annex is the immediate demo target: this repo runs git annex test -- plus git's own testsuite, pjdfstest, and a curated set of stress-ng filesystem stressors -- against every backend on every push. The harness itself is both backend- and suite-agnostic: new filesystems drop in as bin/eval-under-<name> scripts, new suites as bin/ci/target-<name>.sh (see below).

Read GOTCHAS.md before drawing conclusions from a red cell. It records the exact mkfs / mount / export settings each backend uses -- results only mean something relative to those -- and the root cause of every failure already run down, so nobody re-investigates a known one.

CI status

Backend	git-annex test	git testsuite	stress-ng	pjdfstest
BeeGFS 7.4.6
BeeGFS 8.1.0
NFS (localhost)
Loop vfat
Loop ext4

Rows are backends (which filesystem the work happens on), columns are targets (which suite is run on it). All 20 cells are one job matrix in .github/workflows/test.yaml, fanned out from .github/matrix.yaml -- adding a filesystem or a suite is a data edit, not a code edit.

The badges are ours, not GitHub's: GitHub publishes one badge per workflow file, so a per-cell grid is not something it can serve for a single matrix workflow. Each run renders an SVG per cell onto the gh-pages branch, and each badge links to that cell's own job log via the status page -- GitHub has no stable URL for "the latest job of this matrix cell", so the page's #<cell> anchor supplies the indirection. The page also carries what a badge cannot: which run produced the result, how long ago, and why a cell is red on purpose.

State lives in status.json on that branch and is merged across runs, so a partial run -- "Re-run failed jobs" being the common one -- updates only its own cells instead of blanking the rest.

Run bin/ci/gen-readme-matrix.sh after editing the matrix to refresh the table.

A red cell is not automatically a bug: vfat has no symlinks, ownership, or xattrs, and NFS has its own locking and close-to-open rules. The matrix exists to make which filesystem breaks which layer visible at a glance. GOTCHAS.md lists the cells that are red for a known reason, with the reason.

Test targets

Budgets below are what the suite itself costs on the loop-ext4 cell; a sync-heavy backend (NFS, BeeGFS) runs the same work considerably slower, which is why each target carries its own timeout in bin/ci/matrix.sh.

Target	What it runs	Prep	Budget
git-annex	The full git annex test suite	Daily build from con/git-annex	tens of minutes
git	174 scripts of git's own testsuite (t0*.sh t1*.sh, ~10k assertions)	Source build, pinned tag	~4 min build + ~2 min on ext4
stress-ng	20 curated filesystem stressors, one at a time, --verify	apt install stress-ng	~2 min
pjdfstest	POSIX conformance: 238 scripts, ~8800 assertions	Source build, pinned commit	~3.5 min on ext4

Why these four, in order of how deep they sit:

• stress-ng is the cheapest and the bluntest: it hammers rename, link/symlink, locka/lockf, xattr, chmod/chown, utime with verification on, so a filesystem that returns success while doing the wrong thing is caught before anything is built on top of it. Each stressor runs separately, so "vfat has no xattrs" reports as a skip rather than poisoning the run.
• pjdfstest is the specification check: it tells you which syscall returned which errno where POSIX says otherwise. Where git annex test says "something is wrong", this says rename returns EEXIST instead of ENOTEMPTY.
• git is the integration check for the layer git-annex actually stands on -- index, refs, object store, attributes. Git's suite is the canonical filesystem-picky testbed and is maintained far more rigorously than anything hand-written here. Only the trash directories (where the tests' file operations happen) live on the mount; the build itself stays on the runner's disk. It is driven through prove, git's TAP harness, rather than the default make test target -- under make the first failing script aborts the run before the totals are ever printed, whereas prove finishes every script and ends with a summary naming each failure.
• git-annex is the original motivation and the top of the stack.

Pinned upstream refs live in bin/ci/matrix.sh and are bumped deliberately: with a moving testsuite, a newly-red cell is ambiguous -- did the filesystem regress, or did upstream add a test? git is pinned to a release tag. pjdfstest is pinned to a commit rather than its one upstream tag (0.1, 2016), which no longer compiles: major(), minor() and makedev() moved to <sys/sysmacros.h> in glibc 2.28 and the tree builds with -Werror, so those implicit declarations are hard errors on any current toolchain. Master builds clean, so we pin a commit on it instead of carrying a patch.

Motivation

A 2024 report of 35+ git-annex test failures on BeeGFS 7.4.6 tracked back to a rename-semantics quirk that BeeGFS exposes but ext4/tmpfs usually mask. That's not a new class of story: DataLad's CI has for years included NFS and vfat-loop flavours precisely because those filesystems break git-annex assumptions in ways plain ext4 doesn't (eval_under_nfs, eval_under_testloopfs).

This repo consolidates and generalises that pattern: one dispatcher (bin/eval-under), one CI workflow driven by an external matrix definition, a small backend script per filesystem, and a small target script per test suite. Both axes slot in uniformly; the git-annex-under-BeeGFS coverage that motivated the repo is now one cell of that matrix.

CLI usage

# git-annex smoke under BeeGFS 7.4.6, HOME on the mount
sudo bin/eval-under beegfs --set-home -- bash -c '
  cd "$HOME" && git init t && cd t && git annex init && git annex test
'

# Same, under a 200 MB xfs loop
sudo bin/eval-under loop --fs xfs --size 200 --set-home -- \
  bash -c 'cd "$HOME" && git annex test'

# Under a localhost NFS export (async by default; --sync to reproduce
# the fsync-heavy slow path)
sudo bin/eval-under nfs --set-home -- bash -c 'cd "$HOME" && git annex test'

# Skip teardown to poke around after a failure
sudo bin/eval-under beegfs --set-home --keep -- some-failing-command

# Discover backends / read backend help
bin/eval-under --list
bin/eval-under nfs --help

All backends accept --mount-point, --set-home, --keep, and their own backend-specific options. See bin/eval-under BACKEND --help for the full flag / env-var / default table per backend.

File layout

Path	Purpose
Vagrantfile + provision/	Ubuntu 24.04 libvirt VM with docker + BeeGFS + NFS + loop deps + git-annex
bin/eval-under	Dispatcher: routes to bin/eval-under-<backend>
bin/eval-under-beegfs	BeeGFS backend (containerised cluster + kernel client mount)
bin/eval-under-nfs	NFS backend (localhost loopback export)
bin/eval-under-loop	Loop-device backend (dd + losetup + mkfs. + mount)
fixtures/beegfs/docker-compose-v7.yml	BeeGFS v7 test cluster (mgmtd + meta + storage), network_mode: host
fixtures/beegfs/docker-compose-v8.yml	Same, for BeeGFS v8.x (different mgmtd command style / gRPC control plane)
fixtures/beegfs/beegfs-*.conf.template	Minimal client + helperd confs for the throwaway cluster
.github/matrix.yaml	Single source of truth: backends x targets, pinned upstream refs, per-target knobs
bin/ci/matrix.sh	Shell accessors over .github/matrix.yaml, sourced by every other bin/ci script
bin/ci/matrix-json.sh	Renders that file as the workflow's matrix: value (via fromJson)
bin/ci/install-target.sh	Runner-side prep for a target (apt package, or source build at a pinned tag)
bin/ci/target-<target>.sh	The suite itself, run inside the mount by bin/ci/run-under.sh
bin/ci/gen-readme-matrix.sh	Regenerates the README badge grid from .github/matrix.yaml
bin/ci/render-badge.sh	Renders one status badge as a self-contained SVG
bin/ci/update-status.py	Merges a run's per-cell results into the persistent status.json
bin/ci/render-report.py	Renders status.json into the badge set + the report page
bin/ci/publish-status.sh	Ties those together and pushes the site to gh-pages
.github/workflows/test.yaml	The whole matrix: one matrix job, 20 test cells, one publish job
drafts/git-annex-test-beegfs.yaml	Copy-target workflow for con/git-annex (external PR target)

Local iteration (VM)

The dev container this repo is usually edited in lacks CAP_SYS_MODULE (no BeeGFS kmod) and doesn't run its own NFS server. Use the Vagrant VM:

vagrant up                              # first time: ~10 min
vagrant ssh
cd /vagrant

# Iterate on any backend:
sudo bin/eval-under beegfs --set-home -- bash -c '
  cd "$HOME" && git init t && cd t && git annex init && git annex fsck
'
sudo bin/eval-under nfs    --set-home -- git annex test
sudo bin/eval-under loop --fs vfat --set-home -- git annex test

# Or run a whole CI cell exactly as the runner would. install-target.sh
# is the one-off prep (source builds land in $EVAL_UNDER_SRC_DIR, not on
# the mount); run-under.sh then wraps the suite in the backend.
bin/ci/install-target.sh pjdfstest
sudo -E bin/ci/run-under.sh loop ext4 pjdfstest
sudo -E bin/ci/run-under.sh nfs  n/a  stress-ng

Optional: install act in the VM to replay the GitHub workflow locally.

VAGRANT_INSTALL_ACT=1 vagrant provision
vagrant ssh -c 'cd /vagrant && act -j test'

act runs the workflow in a container, so it can validate the YAML flow but cannot exercise the BeeGFS kernel module or the host's NFS server -- useful for shaking out workflow bugs, not for actual filesystem testing.

Adding a new backend

1. Drop bin/eval-under-<newbackend> next to the existing backends. It's picked up automatically by bin/eval-under --list.
2. Follow the pattern: set -eu, ${SUDO[@]} arrays for root, trap teardown EXIT, a here-doc'd usage(), EVAL_UNDER_<BACKEND>_* env vars for backend-specific options, and the common --mount-point / --set-home / --keep flags on top.
3. At the end, run the wrapped command with TMPDIR, DATALAD_TESTS_TEMP_DIR, and (if --set-home) HOME pointing at the mount.
4. Add a row to backends: in .github/matrix.yaml, then run bin/ci/gen-readme-matrix.sh to refresh the README grid above. The workflow picks the new cells up on its own. Commit the result.
5. Teach bin/ci/install-backend.sh how to install its client packages.
6. Update provision/setup.sh if the backend needs new host packages.

Adding a new test target

1. Write bin/ci/target-<name>.sh. It runs inside the mount, with TMPDIR (and HOME, with --set-home) already pointing at the filesystem under test. Exit non-zero on failure; skip -- loudly -- rather than fail on operations the filesystem genuinely cannot do.
2. Teach bin/ci/install-target.sh how to prepare it on the runner. Build source trees into $EVAL_UNDER_SRC_DIR (the runner's own disk), never onto the mount: only the suite's I/O should exercise the filesystem under test. Pin any upstream checkout to a tag.
3. Add an entry to targets: in .github/matrix.yaml with its label, timeout, loop-size-mb, needs-root, and needs-git-annex.
4. Run bin/ci/gen-readme-matrix.sh and commit the new README column.
5. shellcheck bin/ci/*.sh bin/eval-under* before committing.

Upstream targets

• bin/eval-under-beegfs + fixtures/beegfs/* + a copy of drafts/git-annex-test-beegfs.yaml -> PR to con/git-annex once validated, either as a new workflow or as a matrix flavour of test-annex in build-ubuntu.yaml.
• Optionally, a slimmed-down smoke workflow -> PR to ThinkParQ/beegfs-containers addressing their issue #21.

Requirements (host, for Vagrant)

• libvirt + vagrant-libvirt plugin (Debian/Ubuntu: apt install vagrant libvirt-daemon-system then vagrant plugin install vagrant-libvirt)
• ~30 GB free disk, ~6 GB free RAM for the VM

Default box is cloud-image/ubuntu-24.04 (Canonical's official image, has a libvirt provider). To use virtualbox instead: VAGRANT_DEFAULT_PROVIDER=virtualbox vagrant up -- that switches the box to bento/ubuntu-24.04 automatically.

Licensing

Machine-readable per the REUSE specification:

• License texts live in LICENSES/ (MIT for project-native files, Apache-2.0 for the ThinkParQ-derived BeeGFS Docker Compose fixtures).
• Path-to-license mappings live in REUSE.toml.
• Absorbed files preserve upstream attribution: bin/eval-under-nfs and bin/eval-under-loop credit the DataLad developers (MIT); fixtures/beegfs/docker-compose-v{7,8}.yml credit ThinkParQ GmbH (Apache-2.0).
• Validate with uvx --from reuse reuse lint (or pip install reuse && reuse lint). This project ships 100% REUSE-compliant.

New files contributed to this repo don't need per-file SPDX headers; they are covered by the REUSE.toml catchall block. Add per-file headers (or an additional [[annotations]] block) only if the file carries a different license or additional attribution.

References

• Bug that motivated the BeeGFS coverage: https://git-annex.branchable.com/bugs/35_failed_tests_on_beegfs/
• Datalad's original per-filesystem wrappers: tools/eval_under_nfs, tools/eval_under_testloopfs
• BeeGFS containerisation guidance: https://doc.beegfs.io/latest/advanced_topics/containers.html
• BeeGFS CSI driver e2e workflow (client install patterns): https://github.com/ThinkParQ/beegfs-csi-driver/blob/main/.github/workflows/build-test-publish.yaml