diff --git a/README.md b/README.md index 06870d8..851dd9f 100644 --- a/README.md +++ b/README.md @@ -475,7 +475,9 @@ the others checks or relies on Reply's own state rather than repeating an effect **Retries.** Reads are retried on errors. A write is retried only when it can't have reached Reply: the connection was never made, or a `429` asked to wait 3 seconds or less. A longer `429` comes back -as `rate_limited` with `retry_after`. +as `rate_limited` with `retry_after`. A `429` that names a `reference` or says a daily or weekly limit +was reached is never resent, whatever it asks to wait: Reply may have recorded the attempt, and a repeat +would record another. Each comes back at once as the refusal it is. ## Raw API access @@ -508,7 +510,8 @@ A write through `reply api` (POST, PUT, PATCH, DELETE) is never resent on a 5xx connection that broke after sending. That last case exits 3: the write may have happened. `reply api` goes by the method alone, so this includes a POST that only reads, such as the `…/filter` calls; run it again yourself if it failed. A `429` is waited out only when `Retry-After` asks for 3 seconds or -less; a longer one is printed as the `429` it is. +less; a longer one is printed as the `429` it is, and so is one whose body names a `reference` or a +daily or weekly limit. ### Windows: Git Bash rewrites the path diff --git a/skills/reply-adapter/reply-api/SKILL.md b/skills/reply-adapter/reply-api/SKILL.md index ca68e30..3692b97 100644 --- a/skills/reply-adapter/reply-api/SKILL.md +++ b/skills/reply-adapter/reply-api/SKILL.md @@ -6,7 +6,7 @@ description: > Use before calling any non-trivial v3 endpoint, when choosing an endpoint for a task, or when an API call fails and needs interpretation. metadata: - version: 1.5.0 + version: 1.5.1 pack: reply-adapter category: execution maturity: reviewed @@ -73,7 +73,9 @@ per-page `.md` fetches; the spec is large. has no feed yet answers a bare 404 — page through the inbox threads instead. - **Rate limits:** 100 req/min and 3000 req/hour per user; reporting and sequence-stats endpoints are stricter. On 429 the CLI already retried honoring `Retry-After` — if it still fails, wait - and reduce concurrency (sequential calls for bulk work). + and reduce concurrency (sequential calls for bulk work). The exception is a 429 that names a + `reference` or says a daily or weekly limit was reached: it is never resent and comes back at + once as the refusal it is. - **Direct sends** (from reply-cli 0.17.0): `--send-after ` takes a moment with its offset and sends it as UTC, and Reply holds the send until then. `--idempotency-key ` goes as the `Idempotency-Key` header: the same request repeated under it answers the send Reply took rather @@ -192,7 +194,7 @@ Record endpoint paths used, entity IDs created/modified, background-job IDs and | 403 `insufficient_scope` | key lacks the endpoint's scope | tell the user which scope is missing; don't retry | | 403/401 `TEAM_REQUIRED`·`TEAM_NOT_ACCESSIBLE`·`USER_REQUIRED`·`USER_NOT_FOUND` | team/user resolution | see reply-auth skill | | 404 | wrong/stale ID | re-list the entity, re-resolve the ID — don't invent IDs | -| 429 | rate limit | respect `Retry-After`; go sequential | +| 429 | rate limit | respect `Retry-After`; go sequential; one that names a `reference` or says a daily or weekly limit was reached is never resent and comes back at once as the refusal it is | | 5xx | transient | already retried; wait, retry once, then report the API may be down | ## Safety @@ -207,6 +209,9 @@ skill's confirmation rules — especially anything that sends messages or starts ## Changelog +- 1.5.1 (2026-11-05): *Conventions* and the error table say that a 429 naming a `reference` or + saying a daily or weekly limit was reached is never resent and comes back at once as the refusal + it is. - 1.5.0 (2026-10-30): *Conventions* says how a direct send's text is filled in as a template: on an email, a LinkedIn message and an InMail; a send that goes now refused by reply-cli 0.17.2's three codes, with the names Reply lists; a held send filled in when it goes and failed then; the diff --git a/skills/reply-adapter/reply-cli/SKILL.md b/skills/reply-adapter/reply-cli/SKILL.md index 25814d9..c5651ec 100644 --- a/skills/reply-adapter/reply-cli/SKILL.md +++ b/skills/reply-adapter/reply-cli/SKILL.md @@ -6,7 +6,7 @@ description: > Use when executing any Reply.io operation from a shell, setting up authentication, or when another Reply skill needs commands executed. metadata: - version: 1.3.0 + version: 1.3.1 pack: reply-adapter category: execution maturity: reviewed @@ -116,7 +116,9 @@ reply api /v3/whoami --verbose # full request/response trace on s Rules: the path is used **exactly as written in the API docs** (starts with `/v3`; query string goes in the path). `--method GET|POST|PUT|PATCH|DELETE` overrides the default. Output is always `{"code": , "data": }` on stdout — check `code` yourself; stdout stays clean -for piping. Transient failures (429/5xx) retry automatically honoring `Retry-After`. +for piping. Transient failures (429/5xx) retry automatically honoring `Retry-After`, except a 429 +that names a `reference` or says a daily or weekly limit was reached, which is never resent and +comes back at once as the refusal it is. ### Exit codes & errors @@ -163,6 +165,8 @@ explicit confirmation its Safety section requires. ## Changelog +- 1.3.1 (2026-11-05): `reply api` says that a 429 naming a `reference` or saying a daily or weekly + limit was reached is never resent and comes back at once as the refusal it is. - 1.3.0 (2026-08-10): said plainly that reaching the whole API is not reaching the whole job — the per-operation answer is in `reply-operations-mapping` — because "exposes the entire API v3" was the sentence an agent was reading as "anything the user asks for is reachable". The read-back in diff --git a/src/__tests__/operations/direct-send.test.ts b/src/__tests__/operations/direct-send.test.ts index bd69535..0de2c9c 100644 --- a/src/__tests__/operations/direct-send.test.ts +++ b/src/__tests__/operations/direct-send.test.ts @@ -1,8 +1,8 @@ import fs from 'fs'; import os from 'os'; import path from 'path'; -import {describe, it, expect, beforeEach} from 'vitest'; -import {mock_fetch, json, ctx, sent, answer, capture} from './helpers'; +import {describe, it, expect, beforeEach, afterEach} from 'vitest'; +import {mock_fetch, json, ctx, sent, answer, capture, instant_timers, real_timers} from './helpers'; import {handle_direct_send, type Direct_send_name, type Direct_send_opts} from '../../operations/direct-send'; const SUBJECT = 'Quick question, Marta'; @@ -230,6 +230,67 @@ describe('the LinkedIn sends', ()=>{ } }); + describe('a limit is one attempt at the send', ()=>{ + beforeEach(()=>{ instant_timers(); }); + afterEach(()=>{ real_timers(); }); + + // Every call gets the same refusal, so a retry shows up in the call count rather than in a request left unanswered. + const always = (body: Record, status = 429, headers: Record = {})=>{ + mock_fetch.mockImplementation(async()=>json(body, status, headers)); + }; + const LIMITED = {title: 'Too Many Requests', status: 429}; + const details_of = (e: unknown)=>(e as {details?: unknown}).details; + + it('a provider limit that recorded a failed send is one call, and names that send', async()=>{ + always({ + ...LIMITED, code: 'directLinkedIn.weeklyLimitReached', detail: 'The send failed.', + reference: 'linkedin-1042', state: 'failed', sendAfter: '2026-11-03T14:00:00Z', + }); + const e = await failure('linkedin-connect', {note: NOTE}, LINKEDIN_OPTS); + expect(e).toMatchObject({code: 'linkedin.limit_reached', exit_code: 1, status: 429, reply_code: 'directLinkedIn.weeklyLimitReached'}); + expect(details_of(e)).toStrictEqual({limit: 'weekly', reference: 'linkedin-1042', state: 'failed', send_after: '2026-11-03T14:00:00Z'}); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + + it('each allowance is one call, with no reference and no Retry-After to go by', async()=>{ + for (const [reply_code, limit] of [['directLinkedIn.dailyLimitReached', 'daily'], ['directLinkedIn.weeklyLimitReached', 'weekly']]) + { + mock_fetch.mockReset(); + always({...LIMITED, code: reply_code}); + const e = await failure('linkedin-message', {body: TEXT}, LINKEDIN_OPTS); + expect(e, reply_code).toMatchObject({code: 'linkedin.limit_reached', details: {limit}}); + expect(mock_fetch, reply_code).toHaveBeenCalledTimes(1); + } + }); + + it('passes on a short Retry-After rather than waiting it out', async()=>{ + always({...LIMITED, code: 'directLinkedIn.dailyLimitReached'}, 429, {'Retry-After': '2'}); + expect(await failure('linkedin-message', {body: TEXT}, LINKEDIN_OPTS)).toMatchObject({code: 'linkedin.limit_reached', retry_after: 2}); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + + it('a 429 naming a send under a code that is no limit is one call too, and stays rate_limited', async()=>{ + always({...LIMITED, code: 'directLinkedIn.somethingNew', reference: 'linkedin-1042', state: 'failed'}); + const e = await failure('linkedin-message', {body: TEXT}, LINKEDIN_OPTS); + expect(e).toMatchObject({code: 'rate_limited', details: {reference: 'linkedin-1042', state: 'failed'}}); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + + it('a plain rate limit is still resent, and the send goes through', async()=>{ + answer(json(LIMITED, 429), activity('LinkedInConnect')); + expect(await run('linkedin-connect', {note: NOTE}, LINKEDIN_OPTS)).toMatchObject({status: 'sent', activity_id: 1042}); + expect(mock_fetch).toHaveBeenCalledTimes(2); + }); + + it('at any other status an allowance code stays reply.refused, and still names the send', async()=>{ + always({title: 'Bad Request', status: 400, code: 'directLinkedIn.weeklyLimitReached', reference: 'linkedin-1042', state: 'failed'}, 400); + const e = await failure('linkedin-connect', {note: NOTE}, LINKEDIN_OPTS); + expect(e).toMatchObject({code: 'reply.refused', reply_code: 'directLinkedIn.weeklyLimitReached'}); + expect(details_of(e)).toStrictEqual({reference: 'linkedin-1042', state: 'failed'}); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + }); + it('holds a text to the length Reply publishes, counting characters as its document does, before any call', async()=>{ const cases: [Direct_send_name, Record, string, number][] = [ ['linkedin-message', {}, 'body', 8000], diff --git a/src/__tests__/utils/client.test.ts b/src/__tests__/utils/client.test.ts index f419299..e193ad1 100644 --- a/src/__tests__/utils/client.test.ts +++ b/src/__tests__/utils/client.test.ts @@ -283,6 +283,90 @@ describe('utils/client', ()=>{ expect(mock_fetch).toHaveBeenCalledTimes(2); }); + describe('a 429 that settles the request is one attempt', ()=>{ + const LIMIT_WITH_RECORD = { + title: 'Too Many Requests', status: 429, code: 'directLinkedIn.weeklyLimitReached', detail: 'The send failed.', + reference: 'linkedin-1042', state: 'failed', sendAfter: '2026-11-03T14:00:00Z', + }; + const once_only = async(body: unknown, headers: Record = {})=>{ + instant_timers(); + mock_fetch.mockImplementation(async()=>err_res(429, body, headers)); + const {request_raw} = await import('../../utils/client'); + const r = await request_raw(BASE, 'tok', 'POST', '/x', {a: 1}); + expect(r.status).toBe(429); + expect(mock_fetch).toHaveBeenCalledTimes(1); + return r; + }; + + it.each([ + ['a reference and a state', {reference: 'linkedin-1042', state: 'failed'}], + ['a reference under a code that names no allowance', {code: 'directLinkedIn.unknown', reference: 'email-7'}], + ['the weekly allowance', {code: 'directLinkedIn.weeklyLimitReached'}], + ['the daily allowance', {code: 'directLinkedIn.dailyLimitReached'}], + ['the weekly allowance in another family', {code: 'directEmail.weeklyLimitReached'}], + ['the daily allowance in another family', {code: 'supportRequest.dailyLimitReached'}], + ['the weekly allowance with no family', {code: 'weeklyLimitReached'}], + ['the daily allowance with no family', {code: 'dailyLimitReached'}], + ])('%s is not resent', async(_name, body)=>{ + await once_only({title: 'Too Many Requests', ...body}); + }); + + it('hands the caller the whole answer of that one attempt, with a short Retry-After not waited out', async()=>{ + const r = await once_only(LIMIT_WITH_RECORD, {'Retry-After': '2'}); + expect(r.data).toEqual(LIMIT_WITH_RECORD); + expect(r.response_headers['retry-after']).toBe('2'); + }); + + it('holds for a read too: it goes by the answer, not by the call', async()=>{ + instant_timers(); + mock_fetch.mockImplementation(async()=>err_res(429, {code: 'directLinkedIn.dailyLimitReached'})); + const {request_raw} = await import('../../utils/client'); + expect((await request_raw(BASE, 'tok', 'GET', '/x')).status).toBe(429); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + + it('throws from request as any other refusal, after that one attempt', async()=>{ + instant_timers(); + mock_fetch.mockImplementation(async()=>err_res(429, LIMIT_WITH_RECORD)); + const err = await get(BASE, 'tok', '/x').catch(e=>e); + expect(err).toBeInstanceOf(Api_error); + expect(err.status).toBe(429); + expect(err.code).toBe('directLinkedIn.weeklyLimitReached'); + expect(mock_fetch).toHaveBeenCalledTimes(1); + }); + }); + + describe('an ordinary 429 keeps its retry', ()=>{ + it.each([ + ['no body', ''], + ['a title alone', {title: 'Too Many Requests', status: 429}], + ['a code that names no allowance', {code: 'rateLimit.exceeded'}], + ['a code that only resembles an allowance', {code: 'linkedInCommentPreview.dailySampleLimitReached'}], + ['a null reference', {code: 'rateLimit.exceeded', reference: null, state: null}], + ['an empty reference', {reference: ''}], + ['a reference that is not text', {reference: 1042}], + ['a body of JSON null', 'null'], + ['a body of a JSON list', '[]'], + ['a body that is not JSON', 'API calls quota exceeded!'], + ])('%s is resent up to the limit', async(_name, body)=>{ + instant_timers(); + mock_fetch.mockImplementation(async()=>err_res(429, body)); + const {request_raw} = await import('../../utils/client'); + expect((await request_raw(BASE, 'tok', 'POST', '/x', {a: 1})).status).toBe(429); + expect(mock_fetch).toHaveBeenCalledTimes(4); // initial + 3 retries + }); + + it('waits out a short Retry-After as before, and takes the answer after it', async()=>{ + instant_timers(); + mock_fetch.mockReset(); + mock_fetch.mockResolvedValueOnce(err_res(429, {title: 'Too Many Requests'}, {'Retry-After': '2'})) + .mockResolvedValueOnce(json_res({ok: true})); + const {request_raw} = await import('../../utils/client'); + expect((await request_raw(BASE, 'tok', 'POST', '/x', {a: 1})).status).toBe(200); + expect(mock_fetch).toHaveBeenCalledTimes(2); + }); + }); + const broken_body = ()=>new Response(new ReadableStream({ start(c){ c.error(new TypeError('terminated')); }, }), {status: 200}); diff --git a/src/utils/client.ts b/src/utils/client.ts index c857bcc..c3d7cad 100644 --- a/src/utils/client.ts +++ b/src/utils/client.ts @@ -15,6 +15,8 @@ const RETRY_BASE_MS = 500; // The longest Retry-After waited out here. A longer one goes back to the caller as the 429 it // is: a wait inside one call spends a time budget the caller may not have. const MAX_WAIT_SECONDS = 3; +// The allowances a 429 can say have run out, as opposed to a pace that was too fast. +const ALLOWANCE_CODES = ['weeklyLimitReached', 'dailyLimitReached']; // Node's fetch reports every transport failure as TypeError('fetch failed') with the system // error on `.cause`. These are failures to connect at all, so the request never left: no route, // no answer to the connect, a URL that can't be parsed, or a TLS handshake that failed. @@ -127,6 +129,22 @@ const parse_retry_after = (raw: string | null | undefined): number | null=>{ const backoff_ms = (attempt: number): number=>RETRY_BASE_MS * 2 ** attempt; +// Whether a 429 answers the request instead of asking for it later, read from the body alone so +// a route that answers in the same shape is covered too. A reference means Reply recorded this +// attempt, so sending it again records another. An allowance that has run out won't come back +// within any wait made here. A code is matched on its last part: the part before the dot is only +// the family of the route that answered. +const settles_request = (text: string): boolean=>{ + const body = parse_body(text) as {reference?: unknown; code?: unknown} | string | null; + if (typeof body !== 'object' || body === null) + { + return false; + } + const {reference, code} = body; + return (typeof reference === 'string' && reference !== '') + || (typeof code === 'string' && ALLOWANCE_CODES.includes(code.slice(code.lastIndexOf('.') + 1))); +}; + // One request under the retry policy, body included: an answer that breaks off mid-body is a // failure after sending like any other. It returns the final Response and its text for any HTTP // status. It throws RuntimeError('network') when Reply could not be reached and nothing was sent, @@ -160,7 +178,7 @@ const send = async(url: string, init: RequestInit, kind: Call_kind): Promise<{re await sleep(backoff_ms(attempt)); continue; } - if (res.status === 429 && can_retry) + if (res.status === 429 && can_retry && !settles_request(text)) { const wait = parse_retry_after(res.headers.get('Retry-After')); if (wait !== null && wait > MAX_WAIT_SECONDS)