Skip to content

pipeline-sql: run one SQL query across a stage as a merged table - #4

Merged
tarzan merged 1 commit into
mainfrom
maarten/pipeline-cmd-psql-output-layout
Sep 23, 2026
Merged

tarzan merged 1 commit into
mainfrom
maarten/pipeline-cmd-psql-output-layout

Conversation

@tarzan

@tarzan tarzan commented Sep 23, 2026

Copy link
Copy Markdown
Member

Running SQL through pipeline-cmd ('pg:psql -c ...') squeezes each app's own psql table into the appname | output layout, which is unreadable. This adds pipeline-sql <pipeline> <stage> ("<sql>" | --file=<path>), which runs one query against every app's DATABASE_URL via pg:psql -f and prints a single merged table with the app name as the first column: aligned on a terminal, appname;col1;col2 when piped, buffered and sorted by app.

psql is driven entirely through meta-commands in the script file (heroku's pg:psql exposes no psql flags): unaligned tab-separated output, no footer, ON_ERROR_STOP, and a sentinel \echo line so anything ~/.psqlrc prints first is discarded. stdout and stderr are captured separately, so psql NOTICEs never become rows and are forwarded to stderr with the app name; failed apps get their error in place of rows, zero-row apps are skipped with a count (or shown as one empty row with -a), and a header that differs between apps is reported as schema drift.

The chunked fan-out core of run_pipeline_workers is extracted into make_run_tmpdir / fan_out_workers / report_skipped so all three stage commands share it; pipeline-cmd and config-replace output is unchanged and their tests are untouched. 22 new bats tests cover the command (69 total, shellcheck clean), and it was exercised end-to-end against a real local psql through a stubbed heroku.

🤖 Generated with Claude Code

Running SQL through pipeline-cmd ('pg:psql -c ...') squeezes each app's
own psql table into the appname | output layout, which is unreadable.
pipeline-sql runs a single query against every app's DATABASE_URL via
pg:psql -f and prints ONE table with the app name as the first column
(aligned on a terminal, appname;col1;col2 when piped).

psql is driven entirely through meta-commands in the script file, since
heroku's pg:psql exposes no psql flags: unaligned tab-separated output,
no footer, ON_ERROR_STOP, and a sentinel \echo line that lets the worker
discard anything ~/.psqlrc printed first. stdout and stderr are captured
separately so NOTICEs never become rows; they are forwarded to stderr
with the app name. Failed apps get their error in place of rows, zero-row
apps are skipped with a count (or one empty row with -a), and a header
that differs between apps is reported as schema drift.

The chunked fan-out core of run_pipeline_workers is extracted into
make_run_tmpdir / fan_out_workers / report_skipped so all three commands
share it; pipeline-cmd and config-replace output is unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@tarzan
tarzan merged commit a8e54ed into main Sep 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant