Skip to content

fix(email): retry transient SES send failures instead of dropping them - #518

Open
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/f90a192mhud4
Open

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/f90a192mhud4

Conversation

@polylane

@polylane polylane Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

When Amazon SES refused an outgoing mail for a temporary reason — a throttle, a server error, or a dropped connection — the email worker caught the error, recorded the mail as failed, and returned as if the send had finished. The queue never tried again, so the message was lost for good. With this change the worker throws those temporary errors back to the queue, which retries the email a bounded number of times with a growing delay, and only records a permanent failure for errors that will not clear on their own. The catch block in executeEmail recorded every send error as terminal and returned, and the email jobs had no attempts, so a transient failure ended the mail permanently. The analogous webhook delivery path already retries transient failures with attempts and backoff. The rethrow only covers errors SES marks transient, so a rejected message keeps its immediate FAILED record.

This change is a declared stopgap: it contains the symptom while the producer's fix is tracked as a follow-up on the issue timeline.

Production impact at bf4fa63: Email retry change is bounded and additive to error handling

Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:27). Recorded for commit bf4fa6394a90392f5464701cd65d849c89c5967f.

Full analysis
flowchart TD
  A["queueEmail / queueBulk add the job with EMAIL_JOB_OPTIONS"] --> B["worker runs executeEmail"]
  B --> C["sendRawEmail throws"]
  C --> D{"isRetryableSendError and attempts remain?"}
  D -->|"yes"| E["throw: BullMQ retries after backoff"]
  E --> B
  D -->|"no"| F["catch records EmailEvent FAILED and returns"]
  G["observed before the fix: every throw was recorded FAILED once, with no retry"] -.-> F
  H["producer left unchanged: SES refuses the send under its quota; no send-rate pacing in the worker"] -.-> C
Loading
Detection evidence

Affected: Railway account 4e7500cc-fd24-403f-8734-843433c85cea

Production impact

The change touches only the outbound email worker in the production unsend app. The retry applies to errors SES marks transient, adds at most two extra sends per email with exponential backoff, and leaves the terminal-failure path unchanged. I could not read the service's live metrics: the Railway connection returns 403 and the digest is frozen at 2026-10-21, so the load check is empty rather than clear.

Affected resources

  • unsend
Checked and not affected (1)
  • Redis

Observability

  • "Failed to send email" error log in apps/web/src/server/aws/ses.ts#sendRawEmail (existing) shows a send that SES rejected.
  • New warn log [EmailQueueService]: Transient send failure, the queue will retry the email with emailId, attempt, attempts shows a retry handed back to the queue.
  • EmailEvent rows with status: "FAILED" (existing) remain the record of a terminal failure.

Polylane assessed this pull request at bf4fa63.

Causal chain
  • Signal (alert): Failed SES send is marked FAILED and never retried unless the error is a throttle
  • Surfacing site: usesend/useSend web app, outbound email worker at usesend/useSend:apps/web/src/server/service/email-queue-service.ts#executeEmail
  • Mechanism: executeEmail wraps the send in one try/catch. The catch block writes an EmailEvent with status FAILED, sets Email.latestStatus = FAILED and returns without a throw, so BullMQ treats the job as completed. The email jobs are added with DEFAULT_QUEUE_OPTIONS, which has no attempts value, so the job runs once. A transient failure therefore ends the attempt for that email permanently.
  • Producer: EmailQueueService outbound send worker, at usesend/useSend:apps/web/src/server/service/email-queue-service.ts#executeEmail
  • Trigger: A transient SES or AWS SDK failure: a throttling error, an SES 5xx, or a connection or socket error
  • What happens to the failed unit today: BullMQ worker contract: a thrown error marks the job failed and lets BullMQ retry it under the job's attempts and backoff; a return marks the job completed.
  • Last observed: 2026-10-21T09:44:00.000Z
  • Why the fix is still needed while the signal is quiet: The send path emits no log or metric that counts a send failure, and the Railway connection has refused every read with 403 Not Authorized since the sync paused at 2026-10-25T17:45:08Z, so the telemetry digest is frozen at 2026-10-21. No change merged since then touches email-queue-service.ts, queue-constants.ts, or the send retry path, and inspected code at HEAD 9ae9a23 still records a retryable send as terminal with no retry.
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: Queued customer emails that meet a transient SES failure are marked FAILED and are never delivered or retried
  • Producer evidence:
    • email-queue-service.ts catch block: on any send error it creates an EmailEvent with status FAILED, updates Email.latestStatus to FAILED, and returns; it does not rethrow
    • apps/web/src/server/queue/queue-constants.ts DEFAULT_QUEUE_OPTIONS sets only removeOnComplete and removeOnFail, with no attempts, and both queueEmail and queueBulk spread it into the job options
    • apps/web/src/server/aws/ses.ts#sendRawEmail logs and rethrows the SES/SDK error, which the catch in executeEmail then records as terminal
    • The analogous webhook path (webhook-service.ts) sets attempts and exponential backoff and rethrows so BullMQ retries a transient delivery failure
2 files changed (+307/-2)
  • apps/web/src/server/service/email-queue-service.ts: modified, +50/-2
  • apps/web/src/server/service/email-queue-service.unit.test.ts: added, +257/-0
Lint output
keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/ui:lint: $ eslint . --max-warnings 0
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing 62295f12be2ad8a7
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.

 Tasks:    1 successful, 3 total
Cached:    0 cached, 3 total
  Time:    3.9s 
Failed:    usesend-js#lint

[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint

Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry

usesend-js#lint:  ERROR  command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
 ERROR  run failed: command  exited (1)

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.


Summary by cubic

Retries transient SES send failures in the email worker instead of recording them as permanent failures, so a temporary throttle or server error no longer loses the message.

  • Email jobs now use three attempts with exponential backoff; terminal errors (e.g. rejected recipients) still mark the email FAILED at once.
  • Only errors SES marks transient are rethrown to the queue; the existing terminal-failure path is unchanged.
  • Adds unit tests covering throttling, transient server errors, last-attempt failures, terminal errors, and successful sends.

Written for commit bf4fa63. Summary will update on new commits.

View guided diff Turn on auto-fix

…FAILED

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added polylane severity:medium Polylane autofix severity: medium labels Nov 16, 2026
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unsend-marketing Ready Ready Preview Nov 16, 2026 9:19pm UTC

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 807202a7-3ec4-4266-bfe4-a935b9a565a5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying usesend with  Cloudflare Pages  Cloudflare Pages

Latest commit: bf4fa63
Status: ✅  Deploy successful!
Preview URL: https://9dc4f6f7.usesend.pages.dev
Branch Preview URL: https://polylane-autofix-f90a192mhud.usesend.pages.dev

View logs

This branch was successfully deployed

1 active deployment
Preview — bf4fa639 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant