[d-klotz]

A queue handler that hits a provider rate limit no longer gives its message back to SQS for a 30 minute redelivery that spends one of its three receives. The queue worker puts the message back so it runs at the time the provider said. This is phase 2 of ADR-049 (#655).

Stacked on #656. Merge #656 first. This PR needs RateLimitError and retryAt from it, and GitHub retargets this PR to next when #656 merges.

What problem this fixes

• A long limit costs all three receives. After the Requester's in-process ladder, a throttled handler rethrows and SQS redelivers 30 minutes later. The queue allows 3 deliveries, so a limit that lasts two hours sends the message to the DLQ, which only logs.
• A sync split into many messages then stays open forever. One lost message means the run is never completed and never failed.
• The user sees a run that looks stuck. Nothing says "waiting for the provider limit until 23:00 UTC".

What this PR does

When a handler throws an error with isRateLimited and a retryAt (a RateLimitError from the Requester, or one a module throws), Worker.run puts the message back so it runs at retryAt:

Wait	What the worker does	Record
Up to 900 s	Sends the same body again with DelaySeconds, then acknowledges the old message	handled
Over 900 s, SCHEDULER_ROLE_ARN set	Creates a one-time EventBridge schedule at retryAt that sends the body to the queue, then acknowledges	handled
Over 900 s, no scheduler	Extends the visibility timeout to retryAt (12 h less 20 minutes at most: SQS counts the 12 h from the receive, and an extension does not reset it). This uses one receive	reported failed

• The new message goes out before the old one is acknowledged. Delivery stays at least once, as today. A crash between the two can run the work twice, so handlers stay idempotent.

• A deferred body carries _frigg: { deferrals, firstDeferredAt, deferralId }. It is reserved; handlers never see it. deferralId is the SQS id of the first message that was put back, and later deferrals keep it. The new message starts at receive count 1.

• Two caps bound it: FRIGG_QUEUE_MAX_DEFERRALS (default 30) and FRIGG_QUEUE_MAX_DEFERRED_MS (default 26 h, from the first deferral to the retry time). The HubSpot and Salesforce modules probe a daily limit every hour, so the defaults cover one full day of hourly probes. Core owns both ends of these numbers, so devtools injects nothing.

• Past a cap, with no retryAt, with no event source ARN, or on a FIFO queue, the record fails like any retryable error does, and delivery.isLastAttempt (feat(core): tell queue handlers their last delivery attempt #653) applies.

• Run state. With a processId in the message, the worker writes, through two use cases (RecordRateLimitWait, ClearRateLimitWait), Process.context.rateLimit = { status: 'WAITING' | 'EXHAUSTED', mechanism, retryAt, reason, module, deferrals, deferralId, updatedAt }. After a deferred or redelivered message ran, a WAITING value becomes null at once when that message set it (same deferralId), and for any other message only when its retryAt is past or within 5 s. So a job that resumes before its retryAt clears its own wait, and cannot clear another message's wait. applyProcessUpdate also keeps the process updatedAt fresh while it waits.

• A throttled token endpoint defers too. OAuth2Requester replaces non-definitive refresh errors with a fresh Error on purpose (the original message can hold the client secret). The fresh error now keeps isRateLimited, retryAt, waitMs, reason and module, none of which is secret.

• Devtools. The one-time scheduler is provisioned when any API module declares static rateLimit. The base role gains sqs:ChangeMessageVisibility.

• One log record per outcome. Acknowledged: WARN record_deferred. Not acknowledged before the last receive: one WARN (record_deferral_capped, record_visibility_extended, or record_failed with deferral.skipped and the cause as an error). Not acknowledged on the last receive: one ERROR record_lost_rate_limited with the same details, and no WARN.

• A scheduler that cannot be built is handled like a failed schedule: the record falls back to the visibility tier with scheduleError, instead of failing the whole batch.

• The scheduler client honours the local endpoint. The EventBridge adapter now takes the same awsConfigOptions() as the SQS clients. Before, a local run with SCHEDULER_ROLE_ARN set sent a long deferral to the real AWS endpoint with the machine's own credentials; the local end-to-end run found it.

Handler contract

Check the last attempt first, then rethrow:

async processBatch({ data, delivery }) {
    try { await this.syncPage(data); }
    catch (error) {
        if (delivery?.isLastAttempt) return this.failRun(data.processId, error);
        throw error; // a RateLimitError too: core puts the message back at retryAt
    }
}

The delay and schedule tiers send a new message, so they never reach the last attempt. The visibility tier and a cap use a receive. On the last receive the worker writes an ERROR, frigg.worker.record_lost_rate_limited, and the handler has that one chance to end the run. (ADR-049 Appendix A.10 rethrows a rate-limited error first; that order leaves the run open in those two cases, so the check moves first.)

Design choices worth a look

• The mock scheduler is never used for a deferral. The factory returns it for STAGE=dev, and a schedule kept in memory would be lost with the acknowledged message. The worker builds the EventBridge adapter explicitly, and only when SCHEDULER_ROLE_ARN is set and SCHEDULER_PROVIDER is not mock.
• The queue URL comes from record.eventSourceARN (GetQueueUrl by name, memoized per invocation). The base Worker has no integration name for a <NAME>_QUEUE_URL, and a composed URL breaks LocalStack and other partitions.
• The Worker's SQS client now takes the offline and endpoint options that QueuerUtil has. Deferral puts GetQueueUrl, SendMessage and ChangeMessageVisibility on the normal path; without it, frigg start and LocalStack would reach real AWS on every deferral.
• sqs:ChangeMessageVisibility is granted to every stack, not only to one with a declaring module. feat(core): honour Retry-After and let API modules declare rate limits #656 honours Retry-After for every module, so any queue worker can meet a long wait. The statement is already scoped to <service>--<stage>-*Queue; the template gains one action string, and no new resource, function or environment variable. ADR-049 §8 asks for a byte-identical template when no module declares rateLimit; this is the one deviation.
• @aws-sdk/client-scheduler is now a declared dependency of core. Core already required it (the scheduler adapter is exported from the package root) and got it only because admin-scripts hoists it. The lockfile gains one line; npm ci --dry-run passes.

Caveats

• Not run against real AWS. Every tier ran end to end against LocalStack, and is unit-tested with aws-sdk-client-mock. Before relying on it in production, run one throttled message per tier on a dev stack and check the queue, the schedule group and CloudTrail.
• ReportBatchItemFailures and our own ChangeMessageVisibility. The visibility tier reports the record failed after extending its timeout. As far as I know Lambda does not reset the visibility of a reported failure, but I have not verified it. If it does, that tier falls back to today's 30 minute redelivery.
• A stack whose queue is not its own (ownership.queue: 'external') needs sqs:ChangeMessageVisibility and scheduler access granted separately. Pre-existing and wider now: createSchedulerExecutionRole names <Name>Queue for every integration, so such an app cannot deploy once the scheduler is on.
• A wait over 26 hours (a monthly quota) is capped at once and falls back to delivery.isLastAttempt. Raise FRIGG_QUEUE_MAX_DEFERRED_MS for an app that wants longer.
• The handler contract ends a run on the last receive, also when core could still defer. A handler that checks delivery.isLastAttempt first ends the run and returns, so on the third receive a RateLimitError that the delay or schedule tier could still put back ends the run instead. A message reaches its third receive only after two failures that were not deferred, so this is rare.
• The run state is read, then written. Two workers that write in the same instant can keep the earlier retryAt. It is a UI signal only, and the next write corrects it.
• Twenty workers on one limit each defer their own copy. Nothing paces them yet.

Not in this PR

The RATE_LIMITED message for the user, pacing, and the policies of the HubSpot, Salesforce and Pipedrive modules.

Test plan

• core: npm test in packages/core — 2,817 pass (with feat(core): honour Retry-After and let API modules declare rate limits #656 merged in); the 63 failures (DocumentDB, Prisma decryption, Mongo memory server) are the same 63 that fail on origin/next in this environment
• core/Worker.deferral.test.js: 44 tests — the three tiers, the two caps, the 12 h margin, every skip reason, one log record per outcome also on the last receive, a scheduler that cannot be built, the hooks, the SQS client options
• queues/queue-deferral.test.js: 22 tests. oauth-2.test.js: 2 new. integration-defined-workers.test.js: the hook wiring and the clear after a deferred run; the merge rules in record-rate-limit-wait.test.js and clear-rate-limit-wait.test.js
• devtools: 9 new tests (scheduler switch, IAM action, and a template with no declared rate limit gains no scheduler resource, function or env var); the 36 failures in infrastructure/domains are the same on origin/next
• aws-sdk-client-mock is now a declared dev dependency of core: one lockfile line, npm ci --dry-run passes, one copy
• type declarations checked with tsc --strict against a Worker subclass that overrides the hooks
• local end-to-end run (frigg-local-test harness: real Postgres, LocalStack SQS and Scheduler, a real QueueWorker from createQueueWorker, and an integration whose API module calls a local fake provider, with the feat(core): honour Retry-After and let API modules declare rate limits #656, feat(core): put a rate-limited queue message back at its retry time #657 and feat(core): tell the integration when a rate limit is too long to wait for #658 core combined):
  • the delay tier sends a new message with _frigg.deferrals: 1 that arrives 3 s later as receive 1, and the run state goes from WAITING to null after it runs
  • the visibility tier hides the message with a real ChangeMessageVisibility
  • the schedule tier creates frigg-defer-<messageId> with an at(...) time, the queue as target and the deferred body, and acknowledges the record
  • an earlier wait does not replace a later one in the run state
• local full-app run (frigg start on LocalStack SQS and Scheduler and Postgres; 4 fake API-key modules that throttle like HubSpot, Salesforce, Quo and Clockwork; the Management API creates the integrations and starts 6-page syncs; feat(core): honour Retry-After and let API modules declare rate limits #656, feat(core): put a rate-limited queue message back at its retry time #657 and feat(core): tell the integration when a rate limit is too long to wait for #658 combined): 29/29 checks. It covers in-process waits, the 5 min cap, the delay and schedule tiers (the runner fires due schedules, because LocalStack 3.8.1 does not), one warning per long wait, each page fetched exactly once, and a run state that is cleared also when a job resumes 60 min before its retryAt
• eslint: 0 errors. Warnings per file match the base, except 4 camelcase warnings in oauth-2.test.js on the grant_type and refresh_token constructor params, which the 58 existing warnings in that file share
• CI build (22.x) fails on feat(core): add queryMappings to IntegrationMappingRepository #651, feat(core): tell queue handlers their last delivery attempt #653 and feat(core): structured logging (ADR-048) #654 too (frigg-cli, devtools infrastructure and management-ui suites, and a missing libcrypto.so.1.1 for the Mongo memory server), so it does not gate this PR

Release notes

• Upgrade @friggframework/core and @friggframework/devtools together. Devtools grants the IAM action and switches on the scheduler; core does the deferral.
• Redeploy the stack for the IAM action and, when a module declares rateLimit, for the scheduler resources.

🤖 Generated with Claude Code

📦 Published PR as canary version: 2.0.0--canary.657.f16d20b.0

✨ Test out this PR locally via:

npm install @friggframework/admin-scripts@2.0.0--canary.657.f16d20b.0
npm install @friggframework/core@2.0.0--canary.657.f16d20b.0
npm install @friggframework/devtools@2.0.0--canary.657.f16d20b.0
npm install @friggframework/eslint-config@2.0.0--canary.657.f16d20b.0
npm install @friggframework/prettier-config@2.0.0--canary.657.f16d20b.0
npm install @friggframework/schemas@2.0.0--canary.657.f16d20b.0
npm install @friggframework/serverless-plugin@2.0.0--canary.657.f16d20b.0
npm install @friggframework/test@2.0.0--canary.657.f16d20b.0
npm install @friggframework/ui@2.0.0--canary.657.f16d20b.0
# or 
yarn add @friggframework/admin-scripts@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/core@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/devtools@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/eslint-config@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/prettier-config@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/schemas@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/serverless-plugin@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/test@2.0.0--canary.657.f16d20b.0
yarn add @friggframework/ui@2.0.0--canary.657.f16d20b.0

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-28T23:44:40.707723Z	ebe5c60	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ebe5c602d3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[d-klotz]

Addressed the automated review in the latest commit:

• Shared context.rateLimit (Codex P2): the value is one per process, so an unrelated message that succeeded could clear it while another message still waited. The queue worker now compares times. A new wait replaces the stored one only when its retryAt is later, a WAITING value never hides an EXHAUSTED one, and after a deferred or redelivered message ran the worker clears the value only when it is WAITING and its retryAt is past or within 5 s (the scheduler fires up to a second early). Both hooks now read the process first; a failed read is logged and never changes the outcome. 15 tests cover it.
• SonarCloud: defer() is split into one helper per tier, which brings its cognitive complexity under the limit. The two no-op hooks have a body. The redelivery check no longer negates a comparison, because !(receiveCount > 1) is not receiveCount <= 1 when the count is unknown.

This branch also merges the fixes made to #656 (the review of the Requester), so the stack stays in sync.

Full packages/core run: 2,769 pass, and the same 63 failures as origin/next in this environment.

[claude]

Beyond the inline findings, I also checked two other candidate issues and ruled them out: context.rateLimit (backend-utils.js) is a brand-new field with no prior usage anywhere in the codebase, so there's no overwrite risk for existing integration data; and the visibility-extension deferral path for externally-owned/non-scheduler queues is still bounded by SQS's native receive-count/isLastAttempt check (independent of the body-embedded deferral counter), so it can't extend visibility forever even though that counter itself doesn't advance on redelivery.

Extended reasoning...

This PR is a large, security/reliability-sensitive change to the queue worker's rate-limit deferral logic (Worker.js, backend-utils.js, oauth-2.js, scheduler-builder.js) with three confirmed findings already flagged inline (a scheduler IAM/deploy gap, a lost module name on OAuth rate limits, and a rate-limit-state clearing race across sibling messages). I independently traced two additional candidate concerns from the ruled-out list — a potential field-overwrite in backend-utils.js and a potential unbounded-visibility-extension loop for external queues in Worker.js — and confirmed via the code (grep for prior context.rateLimit usage, and the delivery.isLastAttempt gate at Worker.js's visibility branch) that neither holds up as a bug.

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

Code review completed

Nothing new to post: everything this review found is already covered by existing comments on this pull request or didn't merit a separate one.

[claude]

⚠️ Code review skipped — your organization has no extra usage available to pay for this review.

If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.

Once extra usage is available, comment @claude review on this pull request, or push a new commit, to trigger a review.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cursor]

Frigg 2.0.1 review (Group 2) — COMMENT only

Checklist

• Requeue / deferral decision lives in the worker or eventing, not the API module: pass
  • Worker.js + queues/queue-deferral.js catch RateLimitError, SQS DelaySeconds ≤15m, EventBridge Scheduler beyond that, visibility fallback if scheduler unavailable.
  • API/Requester still only waits in-process or throws (from #656).

Verdict (would): would approve

Findings

None blocking. One nit: ensure scheduler IAM / SCHEDULER_ROLE_ARN failure modes are called out in operator docs (tests cover visibility fallback). Stack on #656 as intended.

[claude]

⚠️ Code review skipped — your organization's extra usage balance is too low to start another review, so this review was not started.

An organization admin can add extra usage credits at claude.ai/admin-settings/usage.

Once credits are available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.

[sonarqubecloud]

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud