pipeline-sql: run one SQL query across a stage as a merged table - #4
Merged
Merged
Conversation
Running SQL through pipeline-cmd ('pg:psql -c ...') squeezes each app's
own psql table into the appname | output layout, which is unreadable.
pipeline-sql runs a single query against every app's DATABASE_URL via
pg:psql -f and prints ONE table with the app name as the first column
(aligned on a terminal, appname;col1;col2 when piped).
psql is driven entirely through meta-commands in the script file, since
heroku's pg:psql exposes no psql flags: unaligned tab-separated output,
no footer, ON_ERROR_STOP, and a sentinel \echo line that lets the worker
discard anything ~/.psqlrc printed first. stdout and stderr are captured
separately so NOTICEs never become rows; they are forwarded to stderr
with the app name. Failed apps get their error in place of rows, zero-row
apps are skipped with a count (or one empty row with -a), and a header
that differs between apps is reported as schema drift.
The chunked fan-out core of run_pipeline_workers is extracted into
make_run_tmpdir / fan_out_workers / report_skipped so all three commands
share it; pipeline-cmd and config-replace output is unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Running SQL through
pipeline-cmd('pg:psql -c ...') squeezes each app's own psql table into theappname | outputlayout, which is unreadable. This addspipeline-sql <pipeline> <stage> ("<sql>" | --file=<path>), which runs one query against every app'sDATABASE_URLviapg:psql -fand prints a single merged table with the app name as the first column: aligned on a terminal,appname;col1;col2when piped, buffered and sorted by app.psql is driven entirely through meta-commands in the script file (heroku's
pg:psqlexposes no psql flags): unaligned tab-separated output, no footer,ON_ERROR_STOP, and a sentinel\echoline so anything~/.psqlrcprints first is discarded. stdout and stderr are captured separately, so psql NOTICEs never become rows and are forwarded to stderr with the app name; failed apps get their error in place of rows, zero-row apps are skipped with a count (or shown as one empty row with-a), and a header that differs between apps is reported as schema drift.The chunked fan-out core of
run_pipeline_workersis extracted intomake_run_tmpdir/fan_out_workers/report_skippedso all three stage commands share it;pipeline-cmdandconfig-replaceoutput is unchanged and their tests are untouched. 22 new bats tests cover the command (69 total, shellcheck clean), and it was exercised end-to-end against a real local psql through a stubbedheroku.🤖 Generated with Claude Code