Test runner reports PASS with exit 0 when make fails before any TAP output #139

The test runner in Postgres-Extensions/pgxntool-test (.claude/skills/test/scripts/run-tests.sh) reports STATUS: PASS and exits 0 when the make target fails before producing any TAP output.

Related: pgxntool-test #95 reports the same symptom for a missing test/bats submodule. This issue covers the underlying cause, which also hides failures that happen after bats output has finished.

Reproduction

Makefile in an empty directory:

test-all:
	@echo "make: ../pgxntool: No such file or directory" >&2
	@exit 2

Running bash <pgxntool-test>/.claude/skills/test/scripts/run-tests.sh test-all prints:

STATUS: PASS
SUITES: 0  TOTAL: 0  PASSED: 0  FAILED: 0  SKIPPED: 0
EXIT_CODE: 0

and the script exits 0.

Cause

• Line 80: $CMD 2>&1 || EXIT_CODE=$? runs inside the { ... } | tee | while pipeline, so the assignment happens in a subshell and is lost.
• Line 147 corrects EXIT_CODE only when FAILED > 0.
• Line 172 sets STATUS from the TAP counts alone. It checks neither EXIT_CODE nor whether TOTAL is zero.

Impact: any make failure that emits no not ok line is reported as a pass.

Suggested fix

Capture the status of $CMD outside the pipeline (for example from PIPESTATUS[0], or from a status file written by the subshell). Set STATUS=FAIL when that status is non-zero or TOTAL is zero.

🤖 Generated with Claude Code