Repository navigation
feat(web): 90-day-ready email content cleanup: batched, safe on large tables, runs on cloud when enabled - #524
Conversation
…d explain it in the dashboard The email content cleanup (EMAIL_CLEANUP_DAYS) only ran on self-hosted installs, as one UPDATE over every old email. Cloud kept every email's html, text, attachments and headers forever: 3.7 GB of the 7.1 GB database. - Runs wherever EMAIL_CLEANUP_DAYS is set (cloud or self-hosted). Nothing changes until it's set. - Walks forward by createdAt with a Redis watermark, so each night only reads emails it hasn't looked at; clears up to 100,000 emails a night in batches of 500, each a short transaction with lock/statement timeouts. - Vacuums Email after a run (TRUNCATE FALSE, PARALLEL 0), so the freed space is reused instead of the table growing. - Never clears scheduled or queued emails, or ones scheduled after the cutoff; doesn't change updatedAt (the campaign page sorts by it). - Email details explains removed content instead of 'No content to preview'. Tested on Postgres 16 with 400k production-shaped emails: a night of 50k took 23 s, wrote 41-110 MB of WAL, and grew the table by ~10 MB, then 3 MB once vacuum let the space be reused.
…, exact nightly cap, tune autovacuum instead of vacuuming from the app - Clear content only when an email was created AND last updated before the cutoff: sending updates the row, so an email that waited in a queue keeps its content for N days after it went out. - Emails looked at but not clearable yet go on a pending list (Redis set) that later nights check again by id, so the watermark passing them doesn't keep their content forever. - Each batch looks at no more rows than the nightly cap has left. - No VACUUM from the app (it had no timeout and ran on a pooled connection). Instead, a migration sets autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor to 0.02 on Email, so autovacuum runs after each night's changes. On 400k production-shaped rows, two nights of 100k left the table 278 MB smaller with no dead tuples left. - Cutoff is exact (N x 24 h), not calendar days.
…ut on the autovacuum migration - Pending emails are a Redis sorted set scored by when each becomes eligible (90 days after it was sent; daily for scheduled/queued), so each is checked when it's due, not at random. - The recheck pass shares the run's row cap and time limit, with pauses. - The run result reports how many emails are still pending. - The autovacuum migration runs in a transaction with lock_timeout 10s, so a deploy can't wait indefinitely behind a VACUUM/ANALYZE/CIC on Email. - Production Redis is maxmemory-policy noeviction with no maxmemory, so the pending set can't be evicted on its own.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Deploying usesend with
|
| Latest commit: |
7bea979
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://b0463596.usesend.pages.dev |
| Branch Preview URL: | https://feat-cloud-email-body-retent.usesend.pages.dev |
|
No actionable comments were generated in the recent review. WalkthroughThe cleanup job now runs whenever email cleanup is enabled. It clears eligible email content in bounded batches, tracks progress and deferred checks in Redis, and preserves delivery status and event history. The email query returns the configured retention period, which the preview uses to show a retention notice when applicable. The migration adjusts autovacuum settings, and the documentation describes cleanup timing, limits, and behavior. Suggested reviewers: Change: Feature Merge Risk:
|
Why
Cloud keeps every email's HTML, text, attachments and headers forever. That's 3.7 GB of the 7.1 GB production database, and it grows about 250 MB a month. We're setting a 90-day retention for email content on cloud. Delivery status and events are kept.
This PR changes nothing on its own. The cleanup only runs where
EMAIL_CLEANUP_DAYSis set, and on cloud it'll be set only after customers are told. The one thing that applies on deploy is the autovacuum tuning onEmail, which changes a table setting and touches no data.What it does
html,text,attachmentsandheadersof emails sent more thanEMAIL_CLEANUP_DAYSago, meaning created and last updated before the cutoff. Sending updates the row, so an email that waited in a queue keeps its content for N days after it went out. SES status updates don't touchupdatedAt.createdAtfrom a saved position, so it never rereads emails it has already looked at. It clears at most 100,000 emails a night, in batches of 500. Each batch is its own transaction withlock_timeout 5sandstatement_timeout 60s, there's a 15-minute limit per run, and short pauses between batches.noevictionwith nomaxmemory, so the set can't be evicted on its own.updatedAt, which the campaign page sorts by.VACUUMfrom the app. A migration setsautovacuum_vacuum_scale_factorandautovacuum_analyze_scale_factorto 0.02 onEmail, inside a transaction withlock_timeout 10s. That makes autovacuum visit after about 2% of rows change, instead of 20%.UPDATE. The self-hosting docs are updated.Tested
cleanup-email-bodies.integration.test.ts, runs in CI):updatedAtis untouched.createdAtacross batches are all handled.updatedAtcondition fails 3 tests.Index Scan BackwardoncreatedAtplus anIncremental Sort, about 30 pages per batch.prisma migrate diff(database → schema) is empty.Turning it on (cloud, later)
EMAIL_CLEANUP_DAYS=90on Railway and redeploy.[EmailCleanupJob]logs, dead tuples, WAL and free disk.The backlog of about 1.2M emails clears in about 12 nights.
Ask Polylane for an impact assessment of this pull request:
Summary by cubic
Rebuilds the email content cleanup so it works on cloud at scale, and explains removed content in the email dashboard.
The old job was a single
UPDATEover all old emails and only ran on self-hosted installs. The new job runs whereverEMAIL_CLEANUP_DAYSis set and works in small nightly batches, so nothing changes until that env var is set. The only change that applies on deploy is a migration that tunes autovacuum onEmail— it alters a table setting and touches no data.createdAtwith a saved position, clearing at most 100,000 per night in batches of 500, each batch its own transaction withlock_timeout5s,statement_timeout60s, and a 15-minute run limit.html,text,attachments, andheadersonly for emails created and last updated before the cutoff, so scheduled, queued, and late-sent emails keep their content; delivery status and events are always kept.updatedAt, which the campaign page sorts by.VACUUMwith autovacuum tuning onEmail(autovacuum_vacuum_scale_factorandautovacuum_analyze_scale_factorat 0.02).Written for commit 7bea979. Summary will update on new commits.
Summary by CodeRabbit