Skip to content

feat(web): 90-day-ready email content cleanup: batched, safe on large tables, runs on cloud when enabled - #524

Merged
KMKoushik merged 3 commits into
mainfrom
feat/cloud-email-body-retention
Nov 17, 2026
Merged

KMKoushik merged 3 commits into
mainfrom
feat/cloud-email-body-retention

Conversation

@KMKoushik

@KMKoushik KMKoushik commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Why

Cloud keeps every email's HTML, text, attachments and headers forever. That's 3.7 GB of the 7.1 GB production database, and it grows about 250 MB a month. We're setting a 90-day retention for email content on cloud. Delivery status and events are kept.

This PR changes nothing on its own. The cleanup only runs where EMAIL_CLEANUP_DAYS is set, and on cloud it'll be set only after customers are told. The one thing that applies on deploy is the autovacuum tuning on Email, which changes a table setting and touches no data.

What it does

  • Rule: clear html, text, attachments and headers of emails sent more than EMAIL_CLEANUP_DAYS ago, meaning created and last updated before the cutoff. Sending updates the row, so an email that waited in a queue keeps its content for N days after it went out. SES status updates don't touch updatedAt.
  • Never cleared: scheduled or queued emails, and emails scheduled after the cutoff.
  • Nightly pace: walks forward by createdAt from a saved position, so it never rereads emails it has already looked at. It clears at most 100,000 emails a night, in batches of 500. Each batch is its own transaction with lock_timeout 5s and statement_timeout 60s, there's a 15-minute limit per run, and short pauses between batches.
  • Skipped emails: emails looked at but not clearable yet go into a Redis sorted set, scored by when they become eligible, and are rechecked by id once due. Production Redis is noeviction with no maxmemory, so the set can't be evicted on its own.
  • Untouched fields: doesn't change updatedAt, which the campaign page sorts by.
  • Vacuum: no VACUUM from the app. A migration sets autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor to 0.02 on Email, inside a transaction with lock_timeout 10s. That makes autovacuum visit after about 2% of rows change, instead of 20%.
  • Dashboard: the email page explains removed content ("removed N days after sending; status and events are kept") instead of "No content to preview".
  • Self-hosted: the same job replaces the old single unbatched UPDATE. The self-hosting docs are updated.

Tested

  • Real Postgres (cleanup-email-bodies.integration.test.ts, runs in CI):
    • The right emails are cleared, and recent, scheduled, queued and late-sent ones are kept.
    • updatedAt is untouched.
    • A late-sent email is rechecked exactly when due (not at 70 days, cleared at 81 days on a simulated clock).
    • Ties on createdAt across batches are all handled.
    • The cap holds exactly, in both the recheck and scan phases.
    • Removing the updatedAt condition fails 3 tests.
  • Scale: Postgres 16 with 400k production-shaped emails (~4 KB TOASTed bodies).
    • One night of 50k took 23 s and wrote 41–110 MB of WAL.
    • With autovacuum tuned, two nights of 100k left no dead tuples, and the table was 278 MB smaller.
    • The keyset batch query is an Index Scan Backward on createdAt plus an Incremental Sort, about 30 pages per batch.
  • Migrations: apply on a fresh database, and prisma migrate diff (database → schema) is empty.
  • Suites: unit (376) and tRPC (23) tests pass, and the production build passes.
  • Review: Astra reviewed three rounds, and every finding is addressed.

Turning it on (cloud, later)

  1. Customer notice, plus privacy policy and docs updates (drafts to be approved separately).
  2. Set EMAIL_CLEANUP_DAYS=90 on Railway and redeploy.
  3. Watch the first nights: [EmailCleanupJob] logs, dead tuples, WAL and free disk.

The backlog of about 1.2M emails clears in about 12 nights.

Ask Polylane for an impact assessment of this pull request:

Assess with Polylane


Summary by cubic

Rebuilds the email content cleanup so it works on cloud at scale, and explains removed content in the email dashboard.

The old job was a single UPDATE over all old emails and only ran on self-hosted installs. The new job runs wherever EMAIL_CLEANUP_DAYS is set and works in small nightly batches, so nothing changes until that env var is set. The only change that applies on deploy is a migration that tunes autovacuum on Email — it alters a table setting and touches no data.

  • Walks emails forward by createdAt with a saved position, clearing at most 100,000 per night in batches of 500, each batch its own transaction with lock_timeout 5s, statement_timeout 60s, and a 15-minute run limit.
  • Clears html, text, attachments, and headers only for emails created and last updated before the cutoff, so scheduled, queued, and late-sent emails keep their content; delivery status and events are always kept.
  • Emails looked at but not ready yet are tracked in a Redis sorted set (scored by when they become eligible) and rechecked by id once due.
  • Does not touch updatedAt, which the campaign page sorts by.
  • Replaces the app-level VACUUM with autovacuum tuning on Email (autovacuum_vacuum_scale_factor and autovacuum_analyze_scale_factor at 0.02).
  • The email details page explains that content was removed after N days instead of showing "No content to preview", and the self-hosting docs are updated.

Written for commit 7bea979. Summary will update on new commits.

View guided diff Turn on auto-fix

Summary by CodeRabbit

  • New Features
    • Email content is cleared after the configured retention period, while delivery status and event history are retained.
    • Scheduled or queued emails keep their content until sending. Older eligible emails may be cleaned up gradually, with a nightly limit.
    • Email previews show a retention notice when content has been removed.
  • Documentation
    • Updated self-hosting guidance with details about retention, cleanup limits, and freed storage.

…d explain it in the dashboard

The email content cleanup (EMAIL_CLEANUP_DAYS) only ran on self-hosted
installs, as one UPDATE over every old email. Cloud kept every email's
html, text, attachments and headers forever: 3.7 GB of the 7.1 GB database.

- Runs wherever EMAIL_CLEANUP_DAYS is set (cloud or self-hosted). Nothing
  changes until it's set.
- Walks forward by createdAt with a Redis watermark, so each night only
  reads emails it hasn't looked at; clears up to 100,000 emails a night in
  batches of 500, each a short transaction with lock/statement timeouts.
- Vacuums Email after a run (TRUNCATE FALSE, PARALLEL 0), so the freed
  space is reused instead of the table growing.
- Never clears scheduled or queued emails, or ones scheduled after the
  cutoff; doesn't change updatedAt (the campaign page sorts by it).
- Email details explains removed content instead of 'No content to preview'.

Tested on Postgres 16 with 400k production-shaped emails: a night of 50k
took 23 s, wrote 41-110 MB of WAL, and grew the table by ~10 MB, then 3 MB
once vacuum let the space be reused.
…, exact nightly cap, tune autovacuum instead of vacuuming from the app

- Clear content only when an email was created AND last updated before
  the cutoff: sending updates the row, so an email that waited in a queue
  keeps its content for N days after it went out.
- Emails looked at but not clearable yet go on a pending list (Redis set)
  that later nights check again by id, so the watermark passing them
  doesn't keep their content forever.
- Each batch looks at no more rows than the nightly cap has left.
- No VACUUM from the app (it had no timeout and ran on a pooled
  connection). Instead, a migration sets autovacuum_vacuum_scale_factor
  and autovacuum_analyze_scale_factor to 0.02 on Email, so autovacuum runs
  after each night's changes. On 400k production-shaped rows, two nights
  of 100k left the table 278 MB smaller with no dead tuples left.
- Cutoff is exact (N x 24 h), not calendar days.
…ut on the autovacuum migration

- Pending emails are a Redis sorted set scored by when each becomes
  eligible (90 days after it was sent; daily for scheduled/queued), so each
  is checked when it's due, not at random.
- The recheck pass shares the run's row cap and time limit, with pauses.
- The run result reports how many emails are still pending.
- The autovacuum migration runs in a transaction with lock_timeout 10s, so
  a deploy can't wait indefinitely behind a VACUUM/ANALYZE/CIC on Email.
- Production Redis is maxmemory-policy noeviction with no maxmemory, so the
  pending set can't be evicted on its own.
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unsend-marketing Ready Ready Preview Nov 17, 2026 11:32am UTC

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying usesend with  Cloudflare Pages  Cloudflare Pages

Latest commit: 7bea979
Status: ✅  Deploy successful!
Preview URL: https://b0463596.usesend.pages.dev
Branch Preview URL: https://feat-cloud-email-body-retent.usesend.pages.dev

View logs

@coderabbitai

coderabbitai Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The cleanup job now runs whenever email cleanup is enabled. It clears eligible email content in bounded batches, tracks progress and deferred checks in Redis, and preserves delivery status and event history. The email query returns the configured retention period, which the preview uses to show a retention notice when applicable. The migration adjusts autovacuum settings, and the documentation describes cleanup timing, limits, and behavior.

Suggested reviewers: depak7, ntwcklng

Change: Feature

Merge Risk: Minimal · up to 7bea9

The change adds bounded, opt-in email content cleanup and a retention notice in the dashboard. No blocking issue remains. If an operator later lowers the retention period, some already-deferred emails keep their content until their earlier scheduled check, which delays cleanup but does not lose data.

Pre-merge checks: Passed 4

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names a feature (email content cleanup) and several real traits of the change: batching, safety on large tables, and cloud enablement. The changeset matches these points. The title is long a…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names a feature (email content cleanup) and several real traits of the change: batching, safety on large tables, and cloud enablement. The changeset matches these points. The title is long at 15 words and 104 characters, and it packs several qualifiers into one line, but it still identifies the main change accurately.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Warning

Some tools did not complete. Review the errors below.

ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

apps/web/src/app/(dashboard)/emails/email-details.tsx

ESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox.

apps/web/src/server/api/routers/email.ts

ESLint skipped: the matched ESLint configuration already failed (missing-dependency).

apps/web/src/server/jobs/cleanup-email-bodies.integration.test.ts

ESLint skipped: the matched ESLint configuration already failed (missing-dependency).

  • 1 others

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6f380988-2c65-4f11-822d-8717bbda0af6
Commits

Reviewing files that changed from the base of the PR and between fd2d8f4 and 7bea979.

Files selected for processing: 6

  • apps/docs/self-hosting/overview.mdx
  • apps/web/prisma/migrations/20261117120000_email_autovacuum_tuning/migration.sql
  • apps/web/src/app/(dashboard)/emails/email-details.tsx
  • apps/web/src/server/api/routers/email.ts
  • apps/web/src/server/jobs/cleanup-email-bodies.integration.test.ts
  • apps/web/src/server/jobs/cleanup-email-bodies.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

@KMKoushik
KMKoushik merged commit d311e81 into main Nov 17, 2026
7 checks passed
@KMKoushik
KMKoushik deleted the feat/cloud-email-body-retention branch November 17, 2026 11:43

This branch was successfully deployed

1 active deployment
Preview — 7bea9793 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant