Skip to content

fix(web): keep a campaign sending after one batch job fails - #517

Closed
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/jhdqcw4b73gh
Closed

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/jhdqcw4b73gh

Conversation

@polylane

@polylane polylane Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

A campaign can silently stop delivering when one of its batch jobs fails. The batch is queued under a fixed job id, and BullMQ keeps a failed job under that id for 30 days, so the scheduler can never queue the next batch and the remaining recipients never get the email. After this change the next batch is queued and the campaign runs to completion. Inspected code shows a failed batch job under the fixed id makes every later add() a no-op, and a campaign advances only through that batch worker, so one transient failure stops delivery for good. Clearing a finished job before the enqueue lets the scheduler continue, and it leaves a batch that is still running untouched.

Production impact at 1eebff9: Campaign enqueue restores only a finished job, no production regression

Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:09). Recorded for commit 1eebff9b8c8ecc4e5143db004c32e25f187ee323.

Full analysis
flowchart TD
  A["Scheduler tick ~1.5s"] --> B["queueBatch: add jobId campaign-batch-ID"]
  B --> C{"Job key exists?"}
  C -- "no" --> D["Worker processes one batch, advances lastCursor"]
  D -- "ok" --> E["removeOnComplete frees the id"]
  E --> F["Campaign reaches SENT"]
  D -- "throws" --> G["Job FAILED, retained 30 days"]
  G --> C
  C -- "yes" --> H["handleDuplicatedJob: add() no-op"]
  H --> I["Campaign stuck in RUNNING, contacts unsent"]
  J["Fix: queueBatch clears a completed or failed job before add()"] -.-> B
Loading
Detection evidence

Affected: int_13ee3c91c001c2c8xfdj06a9

Out of scope
  • The same retained-job-id pattern in apps/web/src/server/service/webhook-service.ts and email-queue-service.ts; each is a separate producer tracked on its own issue.

Production impact

I checked the deployed Railway service unsend (production, serves app.usesend.com), which this repository deploys to. The change adds a getJob/getState read and clears a job only when it is completed or failed, then enqueues the next batch; an active or waiting batch is left untouched, so an in-flight batch is never disturbed and no extra load is added. Live metrics are frozen at 2026-10-21 because the Railway reads return 403, so the production observability read is empty.

Checked and not affected (2)
  • unsend
  • useSend

Observability

  • The change adds no new signal; it uses the existing getJob/getState/remove calls on the BullMQ campaign queue.
  • Operators see the corrected behavior in the BullMQ campaign queue: no campaign-batch-<id> job is left in failed state while its campaign stays RUNNING, and campaign.lastSentAt keeps advancing.
  • campaign-scheduler-job.ts logs its tick, and campaign-service.ts logs the defensive window skip, using the repository's existing pino logger.

Polylane assessed this pull request at 1eebff9.

Causal chain
  • Signal (alert): Campaign sending stalls permanently after one failed batch job
  • Surfacing site: campaign batch queue (BullMQ) in the useSend web app at usesend/useSend:apps/web/src/server/service/campaign-service.ts#CampaignBatchService.queueBatch
  • Mechanism: queueBatch adds the batch with the fixed jobId campaign-batch-${campaignId}. When a batch job throws, createWorkerHandler rethrows, BullMQ marks it FAILED, and DEFAULT_QUEUE_OPTIONS retains it for 30 days. BullMQ's addStandardJob script returns handleDuplicatedJob when the job key exists in any state, so every later add() stores no job and the scheduler never refills the batch.
  • Producer: CampaignBatchService.queueBatch on the BullMQ campaign queue, at usesend/useSend:apps/web/src/server/service/campaign-service.ts#CampaignBatchService.queueBatch
  • Trigger: Any batch job that throws: a DB/Redis error in db.contact.findMany or db.campaign.update, or another unexpected worker error.
  • What happens to the failed unit today: DEFAULT_QUEUE_OPTIONS: removeOnComplete true, removeOnFail age 30 days, no attempts; the fixed jobId is used to serialize one batch per campaign.
  • Cadence check: Scheduler refills a batch about every 1.5s; the fixed jobId blocks each refill once a failed job is retained.
  • Last observed: 2026-11-16T17:30:06Z
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: Emails to the remaining subscribed contacts of a stalled campaign are delayed indefinitely; the campaign stays in RUNNING.
  • Producer evidence:
    • apps/web/src/server/service/campaign-service.ts: add(campaign-${campaignId}, ..., { jobId: campaign-batch-${campaignId}, ...DEFAULT_QUEUE_OPTIONS })
    • apps/web/src/server/queue/queue-constants.ts DEFAULT_QUEUE_OPTIONS: removeOnComplete true, removeOnFail age 30 days, no attempts
    • node_modules/.pnpm/bullmq@5.79.3/.../addStandardJob-9.lua:90 EXISTS jobIdKey then handleDuplicatedJob; handleDuplicatedJob.lua returns jobId without storing
    • apps/web/src/server/jobs/campaign-scheduler-job.ts is the only refill path; apps/web/src/server/queue/bullmq-context.ts createWorkerHandler does not catch, so a worker throw marks the job FAILED
2 files changed (+69/-2)
  • apps/web/src/server/service/campaign-service.ts: modified, +17/-1
  • apps/web/src/server/service/campaign-service.unit.test.ts: modified, +52/-1
Lint output
on is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing 62295f12be2ad8a7
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.

 Tasks:    1 successful, 3 total
Cached:    0 cached, 3 total
  Time:    4.395s 
Failed:    usesend-js#lint

[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint

Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry

usesend-js#lint:  ERROR  command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
 ERROR  run failed: command  exited (1)

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.


Summary by cubic

A campaign can silently stop delivering when one of its batch jobs fails. BullMQ keeps the failed job under the fixed campaign-batch-<id> key for 30 days, and every later add() becomes a no-op, so the scheduler can never queue the next batch and remaining recipients never get the email. This change clears a retained finished job before enqueueing the next batch, letting the campaign run to completion. An active or waiting batch is left untouched, so an in-flight batch is never disturbed.

  • Clears only completed or failed jobs, verified via getJob/getState before remove.

Written for commit 1eebff9. Summary will update on new commits.

View guided diff Turn on auto-fix

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added polylane severity:medium Polylane autofix severity: medium labels Nov 16, 2026
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unsend-marketing Ready Ready Preview Nov 16, 2026 5:44pm UTC

@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d24babd9-a023-4f2c-897f-60cebb5d82d9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying usesend with  Cloudflare Pages  Cloudflare Pages

Latest commit: 1eebff9
Status: ✅  Deploy successful!
Preview URL: https://116d797b.usesend.pages.dev
Branch Preview URL: https://polylane-autofix-jhdqcw4b73g.usesend.pages.dev

View logs

@KMKoushik

Copy link
Copy Markdown
Member

Thanks, the bug is real, and I reproduced it against real Redis. I'm replacing this with #520 for production-safety reasons:

#520 also adds a test that runs the real batch worker: it fails after sending, then reruns, and no contact is emailed twice.

@KMKoushik KMKoushik closed this Nov 16, 2026
KMKoushik added a commit that referenced this pull request Nov 16, 2026
* fix(web): keep a campaign sending after one of its batches fails

Campaign batches use a fixed BullMQ job id, and failed jobs are kept for
30 days. BullMQ ignores add() for an id that exists, so after one failed
batch the scheduler could never queue the next one: the campaign stayed
RUNNING and its remaining recipients never got the email. Nothing was
logged, because the batch worker had no failed handler.

- queueBatch uses requeueJob: a failed batch is retried by the scheduler
  after a minute (not on every 1.5s tick), and a batch that failed more
  than a day ago is not retried automatically, so an old campaign never
  starts sending again by surprise.
- Resuming a campaign clears a kept failed batch, so the owner can still
  continue an old one.
- A failed batch is logged as an error.
- requeueJob gains retryFailedAfterMs and maxFailedAgeMs options; webhook
  Retry doesn't use them, so its behaviour is unchanged.

Supersedes #517.

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

* fix(web): pause a campaign whose batch failed long ago, and check job state without list scans

- A batch that failed more than a day ago now pauses its campaign in the
  database. Skipping it was not enough: BullMQ deletes the failed job
  after 30 days, and the campaign would then restart on the next tick.
- requeueJob checks isFailed()/isCompleted() (single ZSCORE lookups)
  instead of getState(), which scans the wait and active lists; the
  campaign scheduler calls it every 1.5s per running campaign.
- Tests: the real batch worker, re-run after a part-way failure, emails
  only the contacts not yet emailed, once each, then advances the cursor;
  resume keeps the campaign paused if the failed batch can't be cleared.

* fix(web): let explicit send and schedule clear an old failed batch, and never let a stale pause undo a resume

- sendCampaign and scheduleCampaign clear a kept failed batch, like
  resume, so a campaign the owner just rescheduled isn't paused again.
- The scheduler's pause only applies if the campaign's updatedAt is
  unchanged since it was read, so it can't undo a resume or reschedule
  made at the same moment; without that read it doesn't pause at all.
- requeueJob still sends add() when the job is neither failed nor
  completed, in case it finished during the checks (BullMQ ignores it
  while the job exists).
- Tests: a batch that fails after sending and runs again emails nobody
  twice (checked to fail if the already-sent check is removed); schedule
  clears an old failed batch; no pause without a fresh read.

* fix(web): save the new schedule before clearing an old failed batch

Clearing first left a moment where the scheduler could start an early
batch for a running campaign being rescheduled. Saving the schedule
first means the worst case on failure is a paused campaign, not an early
send. Resume keeps its order: the campaign is still paused while the
failed batch is cleared.

* fix(web): treat a failure without a finish time as fresh, and re-check before pausing a campaign

- requeueJob: a job fetched while running that fails just after has no
  finishedOn on the snapshot; treat it as a fresh failure, never an old
  one, so it can't pause a campaign by mistake.
- queueBatch looks at the failed batch again right before pausing, so a
  resume, schedule or send that cleared it wins. A clear landing between
  that check and the update can still pause; nothing is sent in that case.

---------

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 1eebff9b Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant