Skip to content

fix(web): re-dispatch a webhook call when it is retried - #516

Closed
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/3ecm3nov2vyf
Closed

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/3ecm3nov2vyf

Conversation

@polylane

@polylane polylane Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Retrying a failed webhook delivery returned success but sent nothing. The retry reused a job id that BullMQ still held for the failed attempt, so the new job was dropped and the call stayed pending. The retry now queues a fresh job, so the subscriber receives the event. BullMQ ignores add() when a job id already exists, and failed jobs are kept for 30 days, so the retry was dropped. The resolved BullMQ add script and the retained-job option show the collision; successful calls are unaffected because completed jobs are removed immediately.

Production impact at 3c7e57a: Webhook retry change cannot degrade the web service

Repository lint: pnpm run lint (declared in package.json) fails over the whole repository (the declared target takes no file arguments), so the failure is not attributed to this change and did not block it; run it before merging.

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:33). Recorded for commit 3c7e57a1c8476962fa605d07d77a7580863d885d.

Full analysis
flowchart TD
  A["Manual retry"] --> B["retryCall: call set to PENDING"]
  B --> C["enqueueCall(callId)"]
  C --> D["queue.remove(callId) clears the retained job"]
  D --> E["queue.add(jobId = callId)"]
  E --> F["job runs and the event is re-sent"]
  G["failed job retained for 30 days"] -. "before this fix add() was a no-op" .-> E
Loading
Detection evidence

Affected: int_13ee3c91c001c2c8xfdj06a9

Out of scope
  • The per-webhook lock still consumes a delivery attempt on contention. That is why calls fail and reach the retry path, and it is unchanged here.

Production impact

The changed path is the web service's webhook producer (apps/web/src/server/service/webhook-service.ts). The only new runtime effect is a Queue.remove(jobId) before the existing Queue.add; BullMQ's remove returns 0 without throwing for a missing or locked job, so no error path is added. I read the production unsend Railway service (tier 2, serves app.usesend.com) and could not read live metrics because the Railway account returns 403 Not Authorized, so the read is empty.

Affected resources

  • unsend

Observability

  • [WebhookQueueService]: Worker error logs a failed BullMQ job; the changed enqueueCall is upstream of it.
  • Webhook call <id> completed successfully and webhookCall.status transitions (PENDING, IN_PROGRESS, DELIVERED, FAILED) show whether a retry was delivered.
  • [WebhookQueueService]: Call not found and the discarded-call log cover the non-active paths.

Polylane assessed this pull request at 3c7e57a.

Causal chain
  • Signal (alert): Retrying a failed webhook call never re-dispatches it
  • Surfacing site: usesend/useSend web app at apps/web/src/server/service/webhook-service.ts#retryCall
  • Mechanism: retryCall resets the call to PENDING and calls enqueueCall, which calls queue.add with jobId=callId. BullMQ still retains the FAILED job under that id for 30 days (removeOnFail age), so add() returns the existing job and stores no new job, and no delivery runs.
  • Producer: WebhookQueueService.enqueueCall, at usesend/useSend:apps/web/src/server/service/webhook-service.ts#enqueueCall
  • Trigger: Manual retry of a webhook call whose job reached the final attempt and is retained as FAILED.
  • What happens to the failed unit today: retryCall exists to re-dispatch a failed delivery; BullMQ add() is a no-op for an existing job id.
  • Last observed: 2026-11-16T00:04:16.000Z
  • Blast radius: unknown other resource(s), unknown other tenant(s); data at risk: A manual retry never delivers the webhook event; the subscriber misses that event and the call row stays PENDING.
  • Producer evidence:
    • apps/web/src/server/service/webhook-service.ts#retryCall resets the call and calls WebhookQueueService.enqueueCall(call.id, teamId).
    • apps/web/src/server/service/webhook-service.ts#enqueueCall adds the job with { jobId: callId }.
    • apps/web/src/server/queue/queue-constants.ts DEFAULT_QUEUE_OPTIONS removeOnFail.age = 30243600, spread into the webhook dispatch queue defaultJobOptions with attempts: WEBHOOK_MAX_ATTEMPTS.
    • bullmq 5.79.3 dist/cjs/scripts/addStandardJob-9.js: for a custom jobId that EXISTS, the script returns handleDuplicatedJob, which emits a 'duplicated' event and returns the existing id without storing a new job.
2 files changed (+82/-1)
  • apps/web/src/server/service/webhook-service.ts: modified, +5/-0
  • apps/web/src/server/service/webhook-service.unit.test.ts: modified, +77/-1
Lint output
ys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
@usesend/ui:lint: $ eslint . --max-warnings 0
@usesend/lib:lint: $ eslint . --max-warnings 0
usesend-js:lint: cache miss, executing 62295f12be2ad8a7
usesend-js:lint: [WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
usesend-js:lint: $ eslint . --max-warnings 0
usesend-js:lint: sh: 1: eslint: not found
usesend-js:lint: [ELIFECYCLE] Command failed.
@usesend/ui:lint: [ELIFECYCLE] Command failed.

 Tasks:    1 successful, 3 total
Cached:    0 cached, 3 total
  Time:    3.566s 
Failed:    usesend-js#lint

[ELIFECYCLE] Command failed with exit code 1.
[WARN] The "pnpm" field in package.json is no longer read by pnpm. The following keys were ignored: "pnpm.overrides". See https://pnpm.io/settings for the new home of each setting.
$ turbo lint

Attention:
Turborepo now collects completely anonymous telemetry regarding usage.
This information is used to shape the Turborepo roadmap and prioritize features.
You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
https://turborepo.dev/docs/telemetry

usesend-js#lint:  ERROR  command (/workspace/usesend/useSend/packages/sdk) /root/.local/share/pnpm/.tools/pnpm/11.11.0/bin/pnpm run lint exited (1)
 ERROR  run failed: command  exited (1)

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.


Summary by cubic

Retrying a failed webhook delivery returned success but sent nothing. BullMQ keeps failed jobs for 30 days under the call id, and add() ignores a job id that already exists, so the retry never queued a new delivery. The retry now removes the retained job before enqueuing a fresh one, and completed deliveries are unaffected because those jobs are removed immediately.

  • Adds Queue.remove(callId) before the existing Queue.add in enqueueCall, with tests covering the retry and non-retry paths.

Written for commit 3c7e57a. Summary will update on new commits.

View guided diff Turn on auto-fix

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added polylane severity:medium Polylane autofix severity: medium labels Nov 16, 2026
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
unsend-marketing Ready Ready Preview Nov 16, 2026 5:42am UTC

@polylane
polylane Bot requested a review from KMKoushik November 16, 2026 05:42
@coderabbitai

coderabbitai Bot commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e1005063-edd7-41c3-8ed9-8afa49db6882

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying usesend with  Cloudflare Pages  Cloudflare Pages

Latest commit: 3c7e57a
Status: ✅  Deploy successful!
Preview URL: https://b72a9a7a.usesend.pages.dev
Branch Preview URL: https://polylane-autofix-3ecm3nov2vy.usesend.pages.dev

View logs

@KMKoushik

Copy link
Copy Markdown
Member

Thanks, the bug is real, and I reproduced it against real Redis. I'm replacing this with #519 for one production reason.

queue.remove(callId) runs before every add(), including for every new webhook call. For an id that doesn't exist, BullMQ 5.79.3's remove script (removeJobFromAnyState) runs LREM over the wait, paused and active lists inside Redis: blocking work on the hot path that grows with the backlog. It also deletes delayed or waiting jobs before re-adding them.

#519 leaves new calls on a plain add() and handles only the Retry path. A kept failed job is moved back with BullMQ's atomic job.retry(), attempts reset; a job that is still queued or running is left alone. It also adds an integration test against real Redis.

@KMKoushik KMKoushik closed this Nov 16, 2026
KMKoushik added a commit that referenced this pull request Nov 16, 2026
…#519)

* fix(web): make webhook Retry deliver a call that used up its attempts

BullMQ ignores add() when a job with the same id exists, and webhook jobs
are kept for 30 days after they fail. So pressing Retry on a call that
had used up its attempts reset the row to PENDING but queued nothing.

Retry now goes through requeueJob: a kept failed job is moved back to the
queue with its attempts reset, in one atomic step; a job that is still
waiting, delayed or running is left alone; otherwise a new job is added.
New calls keep the plain add(), so the hot path does no extra Redis work.

Supersedes #516, which removed the job before every add(), including for
every new call, where BullMQ's remove scans the wait list.

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

* test(web): keep the requeue integration tests from hanging or timing out on slow CI

* fix(web): keep a webhook call retryable when Retry can't queue it

* fix(web): only undo Retry's own reset when queueing fails

---------

Co-authored-by: Polylane <polylane[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 3c7e57a1 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant