Repository navigation
fix(web): keep a campaign sending after one batch job fails - #517
Closed
polylane[bot] wants to merge 1 commit into
Closed
polylane[bot] wants to merge 1 commit into
polylane[bot] wants to merge 1 commit into
Conversation
Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
Deploying usesend with
|
| Latest commit: |
1eebff9
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://116d797b.usesend.pages.dev |
| Branch Preview URL: | https://polylane-autofix-jhdqcw4b73g.usesend.pages.dev |
Member
|
Thanks, the bug is real, and I reproduced it against real Redis. I'm replacing this with #520 for production-safety reasons:
#520 also adds a test that runs the real batch worker: it fails after sending, then reruns, and no contact is emailed twice. |
KMKoushik
added a commit
that referenced
this pull request
Nov 16, 2026
* fix(web): keep a campaign sending after one of its batches fails Campaign batches use a fixed BullMQ job id, and failed jobs are kept for 30 days. BullMQ ignores add() for an id that exists, so after one failed batch the scheduler could never queue the next one: the campaign stayed RUNNING and its remaining recipients never got the email. Nothing was logged, because the batch worker had no failed handler. - queueBatch uses requeueJob: a failed batch is retried by the scheduler after a minute (not on every 1.5s tick), and a batch that failed more than a day ago is not retried automatically, so an old campaign never starts sending again by surprise. - Resuming a campaign clears a kept failed batch, so the owner can still continue an old one. - A failed batch is logged as an error. - requeueJob gains retryFailedAfterMs and maxFailedAgeMs options; webhook Retry doesn't use them, so its behaviour is unchanged. Supersedes #517. Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com> * fix(web): pause a campaign whose batch failed long ago, and check job state without list scans - A batch that failed more than a day ago now pauses its campaign in the database. Skipping it was not enough: BullMQ deletes the failed job after 30 days, and the campaign would then restart on the next tick. - requeueJob checks isFailed()/isCompleted() (single ZSCORE lookups) instead of getState(), which scans the wait and active lists; the campaign scheduler calls it every 1.5s per running campaign. - Tests: the real batch worker, re-run after a part-way failure, emails only the contacts not yet emailed, once each, then advances the cursor; resume keeps the campaign paused if the failed batch can't be cleared. * fix(web): let explicit send and schedule clear an old failed batch, and never let a stale pause undo a resume - sendCampaign and scheduleCampaign clear a kept failed batch, like resume, so a campaign the owner just rescheduled isn't paused again. - The scheduler's pause only applies if the campaign's updatedAt is unchanged since it was read, so it can't undo a resume or reschedule made at the same moment; without that read it doesn't pause at all. - requeueJob still sends add() when the job is neither failed nor completed, in case it finished during the checks (BullMQ ignores it while the job exists). - Tests: a batch that fails after sending and runs again emails nobody twice (checked to fail if the already-sent check is removed); schedule clears an old failed batch; no pause without a fresh read. * fix(web): save the new schedule before clearing an old failed batch Clearing first left a moment where the scheduler could start an early batch for a running campaign being rescheduled. Saving the schedule first means the worst case on failure is a paused campaign, not an early send. Resume keeps its order: the campaign is still paused while the failed batch is cleared. * fix(web): treat a failure without a finish time as fresh, and re-check before pausing a campaign - requeueJob: a job fetched while running that fails just after has no finishedOn on the snapshot; treat it as a fresh failure, never an old one, so it can't pause a campaign by mistake. - queueBatch looks at the failed batch again right before pausing, so a resume, schedule or send that cleared it wins. A clear landing between that check and the update can still pause; nothing is sent in that case. --------- Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A campaign can silently stop delivering when one of its batch jobs fails. The batch is queued under a fixed job id, and BullMQ keeps a failed job under that id for 30 days, so the scheduler can never queue the next batch and the remaining recipients never get the email. After this change the next batch is queued and the campaign runs to completion. Inspected code shows a failed batch job under the fixed id makes every later
add()a no-op, and a campaign advances only through that batch worker, so one transient failure stops delivery for good. Clearing a finished job before the enqueue lets the scheduler continue, and it leaves a batch that is still running untouched.Production impact at
1eebff9: Campaign enqueue restores only a finished job, no production regressionRepository lint:
pnpm run lint(declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.Fix walkthrough
Watch the fix explained (1:09). Recorded for commit
1eebff9b8c8ecc4e5143db004c32e25f187ee323.Full analysis
flowchart TD A["Scheduler tick ~1.5s"] --> B["queueBatch: add jobId campaign-batch-ID"] B --> C{"Job key exists?"} C -- "no" --> D["Worker processes one batch, advances lastCursor"] D -- "ok" --> E["removeOnComplete frees the id"] E --> F["Campaign reaches SENT"] D -- "throws" --> G["Job FAILED, retained 30 days"] G --> C C -- "yes" --> H["handleDuplicatedJob: add() no-op"] H --> I["Campaign stuck in RUNNING, contacts unsent"] J["Fix: queueBatch clears a completed or failed job before add()"] -.-> BDetection evidence
Affected:
int_13ee3c91c001c2c8xfdj06a9Out of scope
apps/web/src/server/service/webhook-service.tsandemail-queue-service.ts; each is a separate producer tracked on its own issue.Production impact
I checked the deployed Railway service
unsend(production, serves app.usesend.com), which this repository deploys to. The change adds agetJob/getStateread and clears a job only when it iscompletedorfailed, then enqueues the next batch; anactiveorwaitingbatch is left untouched, so an in-flight batch is never disturbed and no extra load is added. Live metrics are frozen at 2026-10-21 because the Railway reads return 403, so the production observability read is empty.Checked and not affected (2)
unsenduseSendObservability
getJob/getState/removecalls on the BullMQ campaign queue.campaign-batch-<id>job is left infailedstate while its campaign stays RUNNING, andcampaign.lastSentAtkeeps advancing.campaign-scheduler-job.tslogs its tick, andcampaign-service.tslogs the defensive window skip, using the repository's existing pinologger.Polylane assessed this pull request at
1eebff9.Causal chain
campaign-${campaignId}, ..., { jobId:campaign-batch-${campaignId}, ...DEFAULT_QUEUE_OPTIONS })2 files changed (+69/-2)
apps/web/src/server/service/campaign-service.ts: modified, +17/-1apps/web/src/server/service/campaign-service.unit.test.ts: modified, +52/-1Lint output
Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.
Summary by cubic
A campaign can silently stop delivering when one of its batch jobs fails. BullMQ keeps the failed job under the fixed
campaign-batch-<id>key for 30 days, and every lateradd()becomes a no-op, so the scheduler can never queue the next batch and remaining recipients never get the email. This change clears a retained finished job before enqueueing the next batch, letting the campaign run to completion. An active or waiting batch is left untouched, so an in-flight batch is never disturbed.getJob/getStatebeforeremove.Written for commit 1eebff9. Summary will update on new commits.