Repository navigation
fix(email): retry transient SES send failures instead of dropping them - #518
Open
polylane[bot] wants to merge 1 commit into
Open
polylane[bot] wants to merge 1 commit into
polylane[bot] wants to merge 1 commit into
Conversation
…FAILED Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
Deploying usesend with
|
| Latest commit: |
bf4fa63
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9dc4f6f7.usesend.pages.dev |
| Branch Preview URL: | https://polylane-autofix-f90a192mhud.usesend.pages.dev |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When Amazon SES refused an outgoing mail for a temporary reason — a throttle, a server error, or a dropped connection — the email worker caught the error, recorded the mail as failed, and returned as if the send had finished. The queue never tried again, so the message was lost for good. With this change the worker throws those temporary errors back to the queue, which retries the email a bounded number of times with a growing delay, and only records a permanent failure for errors that will not clear on their own. The catch block in
executeEmailrecorded every send error as terminal and returned, and the email jobs had noattempts, so a transient failure ended the mail permanently. The analogous webhook delivery path already retries transient failures with attempts and backoff. The rethrow only covers errors SES marks transient, so a rejected message keeps its immediate FAILED record.This change is a declared stopgap: it contains the symptom while the producer's fix is tracked as a follow-up on the issue timeline.
Production impact at
bf4fa63: Email retry change is bounded and additive to error handlingRepository lint:
pnpm run lint(declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.Fix walkthrough
Watch the fix explained (1:27). Recorded for commit
bf4fa6394a90392f5464701cd65d849c89c5967f.Full analysis
flowchart TD A["queueEmail / queueBulk add the job with EMAIL_JOB_OPTIONS"] --> B["worker runs executeEmail"] B --> C["sendRawEmail throws"] C --> D{"isRetryableSendError and attempts remain?"} D -->|"yes"| E["throw: BullMQ retries after backoff"] E --> B D -->|"no"| F["catch records EmailEvent FAILED and returns"] G["observed before the fix: every throw was recorded FAILED once, with no retry"] -.-> F H["producer left unchanged: SES refuses the send under its quota; no send-rate pacing in the worker"] -.-> CDetection evidence
Affected: Railway account
4e7500cc-fd24-403f-8734-843433c85ceaProduction impact
The change touches only the outbound email worker in the production
unsendapp. The retry applies to errors SES marks transient, adds at most two extra sends per email with exponential backoff, and leaves the terminal-failure path unchanged. I could not read the service's live metrics: the Railway connection returns 403 and the digest is frozen at 2026-10-21, so the load check is empty rather than clear.Affected resources
unsendChecked and not affected (1)
RedisObservability
"Failed to send email"error log inapps/web/src/server/aws/ses.ts#sendRawEmail(existing) shows a send that SES rejected.warnlog[EmailQueueService]: Transient send failure, the queue will retry the emailwithemailId,attempt,attemptsshows a retry handed back to the queue.EmailEventrows withstatus: "FAILED"(existing) remain the record of a terminal failure.Polylane assessed this pull request at
bf4fa63.Causal chain
2 files changed (+307/-2)
apps/web/src/server/service/email-queue-service.ts: modified, +50/-2apps/web/src/server/service/email-queue-service.unit.test.ts: added, +257/-0Lint output
Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.
Summary by cubic
Retries transient SES send failures in the email worker instead of recording them as permanent failures, so a temporary throttle or server error no longer loses the message.
Written for commit bf4fa63. Summary will update on new commits.