Skip to content

fix(web): start the webhook cleanup job at server startup - #507

Closed
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/4l1lzj5n11mg
Closed

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/4l1lzj5n11mg

Conversation

@polylane

@polylane polylane Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Webhook delivery history was never cleaned up, so the shared database grew with every delivery for every team. The server now starts the daily cleanup job that removes records older than the retention window, which bounds the table's size. The job registers its schedule when its module loads, and no code imported it, while every other background job starts in the same function. Adding that import starts the job. The retention window and the database schema are unchanged.

Production impact at 2ff11b3: Startup adds a daily cleanup job; no schema or request-path change

Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:23). Recorded for commit 2ff11b3cf3c56aca43337659d83d1c623833086e.

Full analysis
flowchart LR
  R["instrumentation.ts register()"] -->|now imports| J["webhook-cleanup-job.ts"]
  J --> S["upsertJobScheduler<br/>daily 03:00 UTC"]
  J --> W["Worker on webhook-cleanup queue"]
  S --> W
  W --> D["db.webhookCall.deleteMany<br/>createdAt < 30 days"]
  D --> T["WebhookCall table bounded"]
Loading
Detection evidence

Affected: int_13ee3c91c001c2c8xfdj06a9

Out of scope
  • The 30-day retention window and the WebhookCall schema stay as they are; only the job wiring changes.
  • Row counts and growth rate: the connected Railway account is read-only and returned Not Authorized, so they were not measured.

Production impact

I checked the deployed Railway service unsend (production, 1 replica), which this repository deploys to. The changed function only starts background jobs at boot and is on no request path. The new job registers one BullMQ schedule and deletes rows older than 30 days once daily: additive, no schema change, removes nothing. Railway metrics for the service returned 403 Not Authorized with the read-only token, so its traffic was not readable; the change carries no production mechanism to degrade it.

Affected resources

  • unsend

Observability

  • Startup already logs Registering instrumentation in apps/web/src/instrumentation.ts#register.
  • The enabled job logs [WebhookCleanupJob]: Deleted old webhook calls with the deleted count and cutoff, plus [WebhookCleanupJob]: Job completed / Job failed.
  • This change adds no new telemetry; it starts existing logging.

Polylane assessed this pull request at 2ff11b3.

Causal chain
  • Signal (alert): Webhook delivery records are never deleted: the cleanup job is never started
  • Surfacing site: usesend/useSend at apps/web/src/instrumentation.ts#register
  • Mechanism: register() starts every other background job but never imports ~/server/jobs/webhook-cleanup-job, so the module's daily BullMQ scheduler and Worker are never registered and WebhookCall rows older than 30 days are never deleted.
  • Producer: apps/web/src/instrumentation.ts#register, at apps/web/src/instrumentation.ts#register
  • Last observed: 2026-11-12T23:07:00.000Z
  • Why the fix is still needed while the signal is quiet: This is a code-path finding, not a runtime metric. The module is still unimported on the default branch at commit 89d5ff4, verified by inspection in this run: the causal path is unchanged and the defect remains even though the scan's signal timestamp is older than a day.
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: none named
  • Producer evidence:
    • apps/web/src/instrumentation.ts lines 11-46 start EmailQueueService.init, usage-job, initDomainVerificationJob, cleanup-email-bodies and CampaignSchedulerService.start but not ~/server/jobs/webhook-cleanup-job.
    • Repository-wide search for webhookCleanupQueue, WEBHOOK_CLEANUP_QUEUE, WEBHOOK_RETENTION_DAYS and webhook-cleanup returns only apps/web/src/server/jobs/webhook-cleanup-job.ts and apps/web/src/server/queue/queue-constants.ts:8, so nothing else imports or prunes the job.
    • apps/web/src/server/jobs/webhook-cleanup-job.ts registers its schedule at module load (line 40) and deletes rows older than 30 days (lines 19-26), so it acts only when imported.
    • apps/web/src/server/service/webhook-service.ts calls db.webhookCall.create for each delivery attempt on the dispatch and test paths, so the table grows per event for every team.
    • PR fix(smtp): install the SMTP server from the monorepo lockfile in every build #505, the only PR merged after the scan, changes the SMTP Dockerfile, .dockerignore, publish.yml and pnpm-lock.yaml and does not touch the causal path.
2 files changed (+65/-0)
  • apps/web/src/instrumentation.ts: modified, +2/-0
  • apps/web/src/instrumentation.unit.test.ts: added, +63/-0
Lint output
ys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/ui:lint: $ eslint . --max-warnings 0
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing bdfdba0250238d1e
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.

 Tasks:    1 successful, 3 total
Cached:    0 cached, 3 total
  Time:    3.486s 
Failed:    usesend-js#lint

[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint

Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry

usesend-js#lint:  ERROR  command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
 ERROR  run failed: command  exited (1)

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.


Summary by cubic

Starts the webhook cleanup job at server startup so webhook delivery history no longer grows unbounded in the shared database.

The cleanup job registers its daily scheduler and worker only when its module is imported, and nothing imported it before. This adds that import to the server boot sequence and adds unit tests covering both configured and missing Redis URLs. The 30-day retention window and schema are unchanged.

Written for commit 2ff11b3. Summary will update on new commits.

View guided diff Turn on auto-fix

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added polylane severity:medium Polylane autofix severity: medium labels Nov 15, 2026
@vercel

vercel Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unsend-marketing Ready Ready Preview Nov 15, 2026 5:39am UTC

@coderabbitai

coderabbitai Bot commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1eecb97b-20ef-44f7-8e06-fb01ca097ed8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying usesend with  Cloudflare Pages  Cloudflare Pages

Latest commit: 2ff11b3
Status: ✅  Deploy successful!
Preview URL: https://0f7faa06.usesend.pages.dev
Branch Preview URL: https://polylane-autofix-4l1lzj5n11m.usesend.pages.dev

View logs

@KMKoushik

Copy link
Copy Markdown
Member

Thanks, confirmed: the job was never imported, and production logs show no run since webhooks launched. I'm replacing this with #521, because its first run would delete about 10 months of rows in one DELETE on a database whose disk is about 78% full.

The replacement deletes in batches of 1,000, each with lock and statement timeouts and a 10-minute nightly budget. It only deletes finished calls not touched in 30 days, so a call someone just retried is kept. It's cloud only, so self-hosters don't lose history on upgrade. It adds an integration test against real Postgres.

@KMKoushik KMKoushik closed this Nov 16, 2026
KMKoushik added a commit that referenced this pull request Nov 16, 2026
…in small batches (cloud) (#521)

* fix(web): start the webhook cleanup job, and delete old calls in small batches

The nightly webhook cleanup job was never imported, so it never ran:
webhook call history has grown since webhooks shipped in January.

- Start it at server startup with the other Redis-backed jobs.
- Delete in batches of 1000 ids (each its own short transaction) with a
  short pause between batches, stopping after 10 minutes; the next run
  carries on. A single DELETE of ~10 months of rows would write a burst
  of WAL and hold a long transaction on a database whose disk is ~78%
  full.

Supersedes #507.

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

* fix(web): only delete finished, untouched webhook calls, with per-batch timeouts, on cloud only

- Delete only DELIVERED/FAILED/DISCARDED calls created and last updated
  over 30 days ago; the DELETE repeats these conditions, so a call
  retried after it was selected is kept.
- Each batch runs in its own transaction with lock_timeout 5s and
  statement_timeout 60s; the deadline is checked before each batch; a
  failed batch stops the run, keeping what was already deleted.
- Start the job on cloud only; self-hosted installs keep their history
  until there's a setting to opt in.
- Log progress every 50 batches and the run's elapsed time; comments
  no longer claim the cleanup shrinks the database files.

* fix(web): give each cleanup batch a Prisma transaction limit above its SQL timeouts; test against postgres

Prisma's default interactive-transaction timeout (5s) would have cut a
slow batch off before its 60s statement_timeout, possibly failing on the
same batch every night. Adds an integration test that runs the cleanup
against a real Postgres (CI): only finished calls created and last
updated over 30 days ago are deleted.

* test(web): register the cleanup-job mock per test, so the self-hosted check can fail

---------

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 2ff11b3c Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant