Repository navigation
Import from Resend, SendGrid and Postmark - #155
Merged
Merged
Conversation
importSuppressions takes an optional reason and date per entry, importContacts per-entry topic choices (opt-outs apply; opt-ins never undo an opt-out), and createWebhookEndpoint can create an endpoint disabled. Migrations from other providers use them; the API and CSV imports are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The migration and migration_item tables hold imports from other providers (Resend, SendGrid, Postmark): the provider's key encrypted with the webhook secrets key ring until the import finishes, the inventory, the customer's choice, progress and one item per imported object. Finished imports are kept 90 days and unstarted ones a day. reencryptWebhookSecrets covers the stored keys, Sentry scrubbing masks Resend/SendGrid keys and Postmark token headers, and dashboard.migration_connections limits key checks per organization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coss/migrate reads a customer's Resend, SendGrid or Postmark account through their APIs (throttled below the providers' shared limits, retries honoring Retry-After) and the worker's runner imports it into one environment in leased, resumable steps. Objects are recorded before they're created, so retries never duplicate them; what exists is left alone. Templates are converted to coss.com's template language and published only when nothing was lost; webhook endpoints are created disabled with new secrets. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An environment's Import page connects the old provider with an API key (checked against a per-organization daily budget), shows what the worker found, starts the import with the chosen kinds (domains are created right there, which only the dashboard may do) and follows its progress to a report with next steps. The worker's migrations loop runs the imports, and the retention loop purges old reports and unstarted imports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/docs/migrate explains the dashboard's Import page (what each kind becomes, how the provider's key is handled, re-running an import), and a page per provider lists what's imported and how it maps, what isn't, and how to switch the code to the coss.com SDK. AGENTS.md describes @coss/migrate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Phases checkpoint after every unit of work (an object, a batch, a page): phase, cursor and counts are saved together, fenced on the lease and the import still running, so a cancel or a lost lease stops the phase before its next write, and a failed step resumes without counting anything twice. Finished phases are saved as finished. - The provider client reserves each request's start before waiting, so concurrent requests keep their spacing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The start action creates the chosen domains before queueing the import, so a failure leaves it startable and starting again retries the domain left pending (each is recorded, none created twice). - Postmark's domains are read in pages. - Membership and suppression pages stop at the step's deadline between batches, from their checkpointed offset, like contacts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An inline default (SendGrid's insert … "default=…") becomes the variable's default for every use on coss.com, so a template that also uses the variable without it (a plain tag, a condition, a loop) is no longer lossless: it is imported as a draft with a warning instead of published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A template the import created is recorded on its item before it is published, so a retry after an interrupted attempt publishes it instead of taking it for an existing one. - An item an attempt finished without checkpointing is counted when the retry meets it again (claimItem returns finished items too). - A SendGrid key that can't read unsubscribe groups still imports the other suppression lists; the inventory says the groups are left out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Resend contact's topic choices are read through every page, so no opt-out is lost. - Domain creation re-checks before each domain that the import is still ready: a cancel stops it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Contacts retried without their unreadable property values go through the same plan-limit handling as the first attempt: reaching the limit stops the contacts phase alone, and the rest of the import runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- When Postmark's template validation (its inferred data model) is throttled or unavailable the error propagates, so the runner retries; when it fails otherwise, a template with sections is left a draft with a warning instead of being converted on a guess and published. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The Import page disables contacts as "Nothing to import" only when the inventory counted exactly zero and found no lists, fields or topics (an estimate of zero, when counting ran out of time, keeps it selectable); suppressions likewise need an exact zero. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Every source's inventory reads each kind on its own: a kind the key may not read (403) or the account doesn't offer (404) is marked unavailable instead of failing the inspection, while authentication and transient errors still propagate. Shared by Resend, SendGrid and Postmark (sources/sections.ts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Custom fields, topics and lists are looked up before they are created, so an environment at its cap (50 properties, 100 topics) still resolves the ones it has, and their values and opt-outs are imported; a create that loses a race to the same key still finds the winner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Removing Handlebars whitespace control (~) can change what a template sends (a space left inside a link), so a template that uses it is imported as a draft to review instead of published. - The SendGrid guide says so, and that an inline default only converts exactly when every use of the variable has it (round 3). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Imported webhook URLs are normalized the way endpoints store them before they are matched and recorded, so an endpoint already there (or created by an interrupted attempt) is found however the provider writes its URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Resend segment members are read from /segments/{id}/contacts, the path
Resend's own SDK lists a segment's contacts with, so an audience only gets
that segment's contacts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A provider's webhooks that post to the same URL (Postmark's, one per stream) are imported as one endpoint subscribed to all of their events, with a note saying they were combined, instead of keeping the first one's events only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The throttle reserved each request's start when the call began waiting, so a timer firing late (a loaded CI runner) let two requests start together. Requests now take turns: each waits for the previous one to start, then for the spacing from that start. Seen as a CI failure of the spacing test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- claimItem looks up an already recorded item before allocating a name, so a retry no longer reserves a name it never stores (which pushed later objects to suffixed names like foo_2_2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Imports from Resend, SendGrid and Postmark, dashboard only (an environment's Developers → Import page, plus a link in "Get started"). The customer pastes the old provider's API key; coss.com reads the account and brings over what they choose:
d-…id as the alias.How it works
@coss/migrate:Retry-After; SendGrid's contact exports are downloaded through a public-address-pinned, size-capped fetch.migrationsloop: leased, resumable phases.Supporting changes
importSuppressionstakes a reason and date per entry,importContactsper-entry topic choices (opt-ins never undo an opt-out), andcreateWebhookEndpointanenabled: false. API and CSV imports are unchanged.migration/migration_item(migration0087).WEBHOOK_SECRETS_KEYand covered byreencryptWebhookSecrets. It is deleted when the import finishes or is canceled; unstarted imports are purged after a day and reports after 90 days (new retention rule).dashboard.migration_connectionslimits key checks per organization (20/day), so coss.com can't be used to test stolen keys./docs/migrate, with a page per provider.No new env vars or infra; the
migrationsloop runs in the existing worker.Testing
bun run lint && bun run typecheck && bun run test && bun run buildgreen.@coss/migrate's 83 tests:Review
Codex reviewed the branch in 19 rounds; rounds 16, 17 (after rebasing on main) and 19 were clean.
~whitespace control keeps the template a draft/segments/{id}/contacts(raised in round 5, re-raised with Resend's SDK as evidence)Rejected:
/contactshas nosegment_idfilter/segments/{id}/contacts), so it was then fixed.GET /webhooksreturns only theoutboundstream'soutbounddefault is only for creating one.🤖 Generated with Claude Code