Retry deploy scripts when Postgres has no free connection slots - #2055
Draft
RhysSullivan wants to merge 2 commits into
Draft
RhysSullivan wants to merge 2 commits into
RhysSullivan wants to merge 2 commits into
Conversation
RhysSullivan
force-pushed
the
migrate-retry-too-many-connections
branch
from
September 18, 2026 18:41
adb7b28 to
faf3e51
Compare
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-marketing | e1eb96c | Commit Preview URL Branch Preview URL |
Sep 18 2026, 07:16 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
executor-cloud | e1eb96c | Sep 18 2026, 07:17 PM |
Contributor
Cloudflare preview
Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes. |
@executor-js/cli
@executor-js/config
@executor-js/execution
@executor-js/sdk
@executor-js/codemode-core
@executor-js/runtime-quickjs
@executor-js/plugin-file-secrets
@executor-js/plugin-graphql
@executor-js/plugin-keychain
@executor-js/plugin-mcp
@executor-js/plugin-onepassword
@executor-js/plugin-openapi
executor
commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The production deploy on 2026-09-18 failed twice in the
Migrate databasejob (run 35372933403, attempts 1 and 2, at 17:14Z and 17:23Z) with SQLSTATE 53300,remaining connection slots are reserved for roles with the SUPERUSER attribute, on the first statement (CREATE SCHEMA IF NOT EXISTS "drizzle"). Nothing had been applied; the 18:10Z rerun passed. Both refusals came after Worker redeploys. The deploy scripts open direct connections that compete for the server's slots with the Hyperdrive pools (through PSBouncer), operator sessions, and CI; the connection count at the time was not measured.This PR is a mitigation for a transient 53300, not the capacity fix. The capacity changes were made on the database side the same day: the production branch now has
pgconf.max_connections=50with pgbouncerdefault_pool_size=12andmax_db_connections=12(changed 2026-09-18 18:41Z), and the bound Hyperdrive configplanetscale-executor-main-axubconnects through PSBouncer on port 6432 withorigin_connection_limit12 (modified 18:42Z).What it does: the deploy scripts wait for a slot instead of failing the deploy.
src/db/too-many-connections.ts: classify 53300 anywhere in the Drizzle/postgres.js cause chain and retry only that failure: 30 retries, 30 s apart, about fifteen minutes (longer than the nine minutes the two refusals spanned). Any other failure surfaces unchanged on the first attempt.retryTooManyConnectionsis the Effect core;retryWhileTooManyConnectionswraps a Promise for the scripts that throw at their boundary;waitForConnectionSlotopens a script's connection with a guardedSELECT 1.scripts/migrate.ts: both steps over the direct connection (Drizzle migrations and code migrations) are guarded, so a 53300 on a later statement after a reconnect also retries.scripts/ensure-workos-mirror-ready.ts: every readiness read is guarded.scripts/backfill-workos-mirror.tsandscripts/drain-workos-events.ts(spawned by the gate, each with its own connection): wait for a slot before their work. Their store failures are classified asWorkOsMirrorErrorwithout the driver code, so the guard sits on opening the connection, which is where 53300 is raised; a connection dropped and reopened mid-run is not guarded there.Verified:
bun run --cwd apps/cloud typecheck;src/db/too-many-connections.test.ts(7 tests, including the production schedule underit.effect's TestClock: an attempt every 30 s, the 31st and last at 15:00, then the retry stops);oxfmt,oxlint, andbun run format:checkfrom the repo root. Not verified: the scripts against a server that actually refuses connections.