Skip to content

Retry deploy scripts when Postgres has no free connection slots - #2055

Draft
RhysSullivan wants to merge 2 commits into
mainfrom
migrate-retry-too-many-connections
Draft

RhysSullivan wants to merge 2 commits into
mainfrom
migrate-retry-too-many-connections

Conversation

@RhysSullivan

@RhysSullivan RhysSullivan commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

The production deploy on 2026-09-18 failed twice in the Migrate database job (run 35372933403, attempts 1 and 2, at 17:14Z and 17:23Z) with SQLSTATE 53300, remaining connection slots are reserved for roles with the SUPERUSER attribute, on the first statement (CREATE SCHEMA IF NOT EXISTS "drizzle"). Nothing had been applied; the 18:10Z rerun passed. Both refusals came after Worker redeploys. The deploy scripts open direct connections that compete for the server's slots with the Hyperdrive pools (through PSBouncer), operator sessions, and CI; the connection count at the time was not measured.

This PR is a mitigation for a transient 53300, not the capacity fix. The capacity changes were made on the database side the same day: the production branch now has pgconf.max_connections=50 with pgbouncer default_pool_size=12 and max_db_connections=12 (changed 2026-09-18 18:41Z), and the bound Hyperdrive config planetscale-executor-main-axub connects through PSBouncer on port 6432 with origin_connection_limit 12 (modified 18:42Z).

What it does: the deploy scripts wait for a slot instead of failing the deploy.

  • src/db/too-many-connections.ts: classify 53300 anywhere in the Drizzle/postgres.js cause chain and retry only that failure: 30 retries, 30 s apart, about fifteen minutes (longer than the nine minutes the two refusals spanned). Any other failure surfaces unchanged on the first attempt. retryTooManyConnections is the Effect core; retryWhileTooManyConnections wraps a Promise for the scripts that throw at their boundary; waitForConnectionSlot opens a script's connection with a guarded SELECT 1.
  • scripts/migrate.ts: both steps over the direct connection (Drizzle migrations and code migrations) are guarded, so a 53300 on a later statement after a reconnect also retries.
  • scripts/ensure-workos-mirror-ready.ts: every readiness read is guarded.
  • scripts/backfill-workos-mirror.ts and scripts/drain-workos-events.ts (spawned by the gate, each with its own connection): wait for a slot before their work. Their store failures are classified as WorkOsMirrorError without the driver code, so the guard sits on opening the connection, which is where 53300 is raised; a connection dropped and reopened mid-run is not guarded there.

Verified: bun run --cwd apps/cloud typecheck; src/db/too-many-connections.test.ts (7 tests, including the production schedule under it.effect's TestClock: an attempt every 30 s, the 31st and last at 15:00, then the retry stops); oxfmt, oxlint, and bun run format:check from the repo root. Not verified: the scripts against a server that actually refuses connections.

@RhysSullivan
RhysSullivan force-pushed the migrate-retry-too-many-connections branch from adb7b28 to faf3e51 Compare September 18, 2026 18:41
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 18, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
executor-marketing e1eb96c Commit Preview URL

Branch Preview URL
Sep 18 2026, 07:16 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Sep 18, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
executor-cloud e1eb96c Sep 18 2026, 07:17 PM

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Cloudflare preview

Console https://executor-preview-pr-2055.executor-e2e.workers.dev
MCP https://executor-preview-pr-2055.executor-e2e.workers.dev/mcp
Deployed commit e1eb96c

Sign-in is Cloudflare Access (one-time PIN to an allowed email). The preview has its own database and encryption key; it is destroyed when this PR closes.

@pkg-pr-new

pkg-pr-new Bot commented Sep 18, 2026

Copy link
Copy Markdown

Open in StackBlitz

@executor-js/cli

npm i https://pkg.pr.new/@executor-js/cli@2055

@executor-js/config

npm i https://pkg.pr.new/@executor-js/config@2055

@executor-js/execution

npm i https://pkg.pr.new/@executor-js/execution@2055

@executor-js/sdk

npm i https://pkg.pr.new/@executor-js/sdk@2055

@executor-js/codemode-core

npm i https://pkg.pr.new/@executor-js/codemode-core@2055

@executor-js/runtime-quickjs

npm i https://pkg.pr.new/@executor-js/runtime-quickjs@2055

@executor-js/plugin-file-secrets

npm i https://pkg.pr.new/@executor-js/plugin-file-secrets@2055

@executor-js/plugin-graphql

npm i https://pkg.pr.new/@executor-js/plugin-graphql@2055

@executor-js/plugin-keychain

npm i https://pkg.pr.new/@executor-js/plugin-keychain@2055

@executor-js/plugin-mcp

npm i https://pkg.pr.new/@executor-js/plugin-mcp@2055

@executor-js/plugin-onepassword

npm i https://pkg.pr.new/@executor-js/plugin-onepassword@2055

@executor-js/plugin-openapi

npm i https://pkg.pr.new/@executor-js/plugin-openapi@2055

executor

npm i https://pkg.pr.new/executor@2055

commit: e1eb96c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant