Skip to content

Import from Resend, SendGrid and Postmark - #155

Merged
pumfleet merged 21 commits into
mainfrom
migrate-imports
Oct 19, 2026
Merged

pumfleet merged 21 commits into
mainfrom
migrate-imports

Conversation

@pumfleet

@pumfleet pumfleet commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Imports from Resend, SendGrid and Postmark, dashboard only (an environment's Developers → Import page, plus a link in "Get started"). The customer pastes the old provider's API key; coss.com reads the account and brings over what they choose:

  • Domains: added to the organization (DNS records to add; Domain Connect where available). Created by the dashboard's start action, since only its IAM policy may create SES identities and tenants.
  • Templates: converted from SendGrid Handlebars, Postmark Mustachio (layouts merged) and Resend to the template language. Published only when the conversion is exact; otherwise left as drafts with warnings by line and column. SendGrid templates keep their d-… id as the alias.
  • Contacts and lists: custom fields → contact properties, lists/segments → audiences, unsubscribe groups/topics → topics, with each contact's topic choices.
  • Suppressions: keep the provider's reason and date. Unsubscribe-group members become topic opt-outs, or category-scoped suppressions when they aren't contacts.
  • Webhook endpoints: events mapped to coss.com's; created disabled with new signing secrets.

How it works

  • New package @coss/migrate:
    • Provider sources: fixed hosts, throttled below providers' shared limits, retries honoring Retry-After; SendGrid's contact exports are downloaded through a public-address-pinned, size-capped fetch.
    • Template converters.
    • The worker's new migrations loop: leased, resumable phases.
  • Every unit of work is checkpointed with its cursor and counts, fenced on the lease. This makes cancels stop the import, retries never duplicate or double-count, and what already exists is left alone.
  • A report lists what was imported, what needs review (drafts, variable renames, disabled endpoints) and the next steps.

Supporting changes

  • importSuppressions takes a reason and date per entry, importContacts per-entry topic choices (opt-ins never undo an opt-out), and createWebhookEndpoint an enabled: false. API and CSV imports are unchanged.
  • Tables migration / migration_item (migration 0087).
  • The provider key is encrypted with WEBHOOK_SECRETS_KEY and covered by reencryptWebhookSecrets. It is deleted when the import finishes or is canceled; unstarted imports are purged after a day and reports after 90 days (new retention rule).
  • Sentry scrubbing masks Resend/SendGrid keys and Postmark token headers.
  • dashboard.migration_connections limits key checks per organization (20/day), so coss.com can't be used to test stolen keys.
  • Docs at /docs/migrate, with a page per provider.

No new env vars or infra; the migrations loop runs in the existing worker.

Testing

  • bun run lint && bun run typecheck && bun run test && bun run build green.
  • @coss/migrate's 83 tests:
    • the runner against a fake provider: full import, resuming in short steps, re-runs finding what exists, cancel mid-step, failed steps resuming without double counts, interrupted template publication, plan limits, topic caps, revoked keys, rate limits, held organizations, purges;
    • each source against recorded API responses;
    • the template converters.
  • Email/webhooks tests cover the import extensions, re-encryption of the stored keys and the scrubbing patterns.
  • In the dashboard (browser): connect, inventory, start (with domains), progress and the report, at desktop and phone width. A run against SendGrid's real API with an invalid key failed cleanly and deleted the key.
  • Not yet verified against real provider accounts: SendGrid export file format and columns, SendGrid event-webhook field names, Resend triple-brace escaping, Postmark's suggested template model. Worth a run per provider before announcing.

Review

Codex reviewed the branch in 19 rounds; rounds 16, 17 (after rebasing on main) and 19 were clean.

Round Fixed
1 Writes fenced against cancel or lease loss (fenced per-unit checkpoints); throttle race; double counts on retries
2 Domain creation retried before queueing; Postmark domains paged; page loops stop at the step deadline
3 An inline Handlebars default with bare uses keeps the template a draft
4 Interrupted template publication resumed; counts recovered on replay; SendGrid suppressions without unsubscribe-group access
6 Resend topic choices paged; domain creation stops on cancel
7 Plan limit on the property-free retry stops only the contacts phase
8 Postmark templates with sections stay drafts when its validation fails
9 An incomplete contact count of 0 doesn't disable contacts
10 Inventory isolates unreadable kinds (all providers)
11 Existing fields, topics and lists found at the environment's caps
12 ~ whitespace control keeps the template a draft
13 Webhook URLs normalized before dedupe
14 Resend segment members read from /segments/{id}/contacts (raised in round 5, re-raised with Resend's SDK as evidence)
15 Webhooks sharing a URL merged into one endpoint
CI The spacing test failed on a loaded runner: requests now take turns, spaced from the previous one's actual start
18 A retried item no longer reserves a name it never stores

Rejected:

Round Finding Why
5 Resend /contacts has no segment_id filter Rejected at first, because Resend's List Contacts reference documents the filter (the user agreed). Codex brought new evidence in round 14 (the official SDK uses /segments/{id}/contacts), so it was then fixed.
8 Postmark GET /webhooks returns only the outbound stream's Postmark's reference for List webhooks: "If no stream is provided, will return all webhooks for the server." The outbound default is only for creating one.

🤖 Generated with Claude Code

pumfleet and others added 21 commits October 19, 2026 18:09
importSuppressions takes an optional reason and date per entry, importContacts
per-entry topic choices (opt-outs apply; opt-ins never undo an opt-out), and
createWebhookEndpoint can create an endpoint disabled. Migrations from other
providers use them; the API and CSV imports are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The migration and migration_item tables hold imports from other providers
(Resend, SendGrid, Postmark): the provider's key encrypted with the webhook
secrets key ring until the import finishes, the inventory, the customer's
choice, progress and one item per imported object. Finished imports are kept
90 days and unstarted ones a day. reencryptWebhookSecrets covers the stored
keys, Sentry scrubbing masks Resend/SendGrid keys and Postmark token headers,
and dashboard.migration_connections limits key checks per organization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coss/migrate reads a customer's Resend, SendGrid or Postmark account through
their APIs (throttled below the providers' shared limits, retries honoring
Retry-After) and the worker's runner imports it into one environment in
leased, resumable steps. Objects are recorded before they're created, so
retries never duplicate them; what exists is left alone. Templates are
converted to coss.com's template language and published only when nothing
was lost; webhook endpoints are created disabled with new secrets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An environment's Import page connects the old provider with an API key
(checked against a per-organization daily budget), shows what the worker
found, starts the import with the chosen kinds (domains are created right
there, which only the dashboard may do) and follows its progress to a report
with next steps. The worker's migrations loop runs the imports, and the
retention loop purges old reports and unstarted imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/docs/migrate explains the dashboard's Import page (what each kind becomes,
how the provider's key is handled, re-running an import), and a page per
provider lists what's imported and how it maps, what isn't, and how to switch
the code to the coss.com SDK. AGENTS.md describes @coss/migrate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Phases checkpoint after every unit of work (an object, a batch, a page):
  phase, cursor and counts are saved together, fenced on the lease and the
  import still running, so a cancel or a lost lease stops the phase before
  its next write, and a failed step resumes without counting anything twice.
  Finished phases are saved as finished.
- The provider client reserves each request's start before waiting, so
  concurrent requests keep their spacing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The start action creates the chosen domains before queueing the import, so
  a failure leaves it startable and starting again retries the domain left
  pending (each is recorded, none created twice).
- Postmark's domains are read in pages.
- Membership and suppression pages stop at the step's deadline between
  batches, from their checkpointed offset, like contacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An inline default (SendGrid's insert … "default=…") becomes the variable's
  default for every use on coss.com, so a template that also uses the
  variable without it (a plain tag, a condition, a loop) is no longer
  lossless: it is imported as a draft with a warning instead of published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A template the import created is recorded on its item before it is
  published, so a retry after an interrupted attempt publishes it instead of
  taking it for an existing one.
- An item an attempt finished without checkpointing is counted when the retry
  meets it again (claimItem returns finished items too).
- A SendGrid key that can't read unsubscribe groups still imports the other
  suppression lists; the inventory says the groups are left out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Resend contact's topic choices are read through every page, so no
  opt-out is lost.
- Domain creation re-checks before each domain that the import is still
  ready: a cancel stops it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Contacts retried without their unreadable property values go through the
  same plan-limit handling as the first attempt: reaching the limit stops the
  contacts phase alone, and the rest of the import runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- When Postmark's template validation (its inferred data model) is throttled
  or unavailable the error propagates, so the runner retries; when it fails
  otherwise, a template with sections is left a draft with a warning instead
  of being converted on a guess and published.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The Import page disables contacts as "Nothing to import" only when the
  inventory counted exactly zero and found no lists, fields or topics (an
  estimate of zero, when counting ran out of time, keeps it selectable);
  suppressions likewise need an exact zero.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Every source's inventory reads each kind on its own: a kind the key may not
  read (403) or the account doesn't offer (404) is marked unavailable instead
  of failing the inspection, while authentication and transient errors still
  propagate. Shared by Resend, SendGrid and Postmark (sources/sections.ts).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Custom fields, topics and lists are looked up before they are created, so
  an environment at its cap (50 properties, 100 topics) still resolves the
  ones it has, and their values and opt-outs are imported; a create that
  loses a race to the same key still finds the winner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Removing Handlebars whitespace control (~) can change what a template
  sends (a space left inside a link), so a template that uses it is imported
  as a draft to review instead of published.
- The SendGrid guide says so, and that an inline default only converts
  exactly when every use of the variable has it (round 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Imported webhook URLs are normalized the way endpoints store them before
  they are matched and recorded, so an endpoint already there (or created by
  an interrupted attempt) is found however the provider writes its URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Resend segment members are read from /segments/{id}/contacts, the path
  Resend's own SDK lists a segment's contacts with, so an audience only gets
  that segment's contacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A provider's webhooks that post to the same URL (Postmark's, one per
  stream) are imported as one endpoint subscribed to all of their events,
  with a note saying they were combined, instead of keeping the first one's
  events only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The throttle reserved each request's start when the call began waiting, so
a timer firing late (a loaded CI runner) let two requests start together.
Requests now take turns: each waits for the previous one to start, then for
the spacing from that start. Seen as a CI failure of the spacing test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- claimItem looks up an already recorded item before allocating a name, so a
  retry no longer reserves a name it never stores (which pushed later
  objects to suffixed names like foo_2_2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@pumfleet
pumfleet merged commit 4c85057 into main Oct 19, 2026
11 checks passed
@pumfleet
pumfleet deleted the migrate-imports branch October 19, 2026 23:27
@linear-code

linear-code Bot commented Oct 19, 2026

Copy link
Copy Markdown

ENG-6500

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant