webhook-delivery

A multi-tenant webhook delivery service — the infrastructure that sits between "an event happened" and "the customer's server heard about it, reliably."

Webhooks sound trivial (just POST the payload) until receivers are slow, down, or flaky. Then you need durable queueing, retries that don't DDoS the recovering receiver, deduplication, sender authentication, and per-tenant isolation. This project builds that infrastructure from first principles on PostgreSQL and Redis — no Kafka, no managed queue — to show exactly where each guarantee comes from.

---

Architecture

flowchart LR
    C[Client] -->|"POST /events + Idempotency-Key"| A[FastAPI Ingestion]
    A -->|"atomic transaction"| P[(PostgreSQL Outbox)]
    W1["Worker Pool (TaskGroup)"] -->|"FOR UPDATE SKIP LOCKED"| P
    W1 -->|"HMAC-signed POST"| R[Receiver]
    W1 -.->|"token bucket (Lua)"| K[(Redis)]

Loading

One Delivery Lifecycle (with Retry & Eventual Success)

sequenceDiagram
    participant C as Client
    participant A as API
    participant P as Postgres
    participant W as Worker
    participant R as Receiver
    C->>A: POST /events (Idempotency-Key)
    A->>P: BEGIN: insert event + pending delivery
    A-->>C: 202 Accepted (event_id, replayed)
    W->>P: claim (SELECT ... FOR UPDATE SKIP LOCKED)
    W->>R: POST (X-Webhook-Signature, X-Webhook-Timestamp)
    R-->>W: 500 Internal Server Error
    W->>P: attempt #1 recorded, next_attempt_at = now + jitter
    Note over W: exponential backoff with full jitter
    W->>R: POST (retry, same signature scheme)
    R-->>W: 200 OK
    W->>P: status = succeeded, COMMIT

Loading

---

Quickstart

1. Start the Stack

docker compose up --build -d

All 5 services (db, redis, api, worker, receiver) will boot and pass their health checks.

2. End-to-End Walkthrough

# 1. Create a tenant (returns an API key — save it)
export API_KEY=$(curl -s -X POST localhost:8000/tenants \
  -H 'Content-Type: application/json' \
  -d '{"name": "acme"}' | python3 -c "import sys,json; print(json.load(sys.stdin)['api_key'])")

# 2. Register a webhook endpoint pointing to the example receiver
export ENDPOINT_DATA=$(curl -s -X POST localhost:8000/endpoints \
  -H "Authorization: Bearer $API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url": "http://receiver:9000/webhook"}')
export ENDPOINT_ID=$(echo $ENDPOINT_DATA | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])")
export WEBHOOK_SECRET=$(echo $ENDPOINT_DATA | python3 -c "import sys,json; print(json.load(sys.stdin)['secret'])")

# 3. Configure the receiver with the shared signing secret
RECEIVER_SECRET=$WEBHOOK_SECRET docker compose up -d receiver

# 4. Fire an event with an Idempotency-Key
curl -s -X POST localhost:8000/events \
  -H "Authorization: Bearer $API_KEY" \
  -H "Idempotency-Key: $(uuidgen)" \
  -H 'Content-Type: application/json' \
  -d "{\"endpoint_id\": \"$ENDPOINT_ID\", \"event_type\": \"order.created\", \"payload\": {\"order_id\": 42}}" | python3 -m json.tool

# 5. Check real-time delivery metrics
curl -s localhost:8000/metrics | python3 -m json.tool

# 6. Check the receiver's idempotent processing stats
curl -s localhost:9000/stats | python3 -m json.tool

---

API Reference

Endpoint	Auth	Method	Description
/tenants	None	POST	Create tenant, returns API key (shown once)
/whoami	Bearer	GET	Verify tenant authentication
/endpoints	Bearer	POST	Register receiver URL, returns signing secret
/endpoints	Bearer	GET	List active endpoints (secrets omitted)
/events	Bearer + Idempotency-Key	POST	Ingest event → 202 Accepted (or 200 if replayed)
/deliveries?status=	Bearer	GET	Browse deliveries (pending, delivering, succeeded, dead)
/deliveries/{id}/retry	Bearer	POST	Manually revive a dead-lettered delivery
/metrics	None	GET	Queue depth, retry & DLQ rates, p50/p95/p99 latency
/health	None	GET	Container liveness check

---

Load Test Results (Locust on Local Compose)

Load test executed against a deliberately flaky receiver (30% simulated 500 error rate) using 50 concurrent simulated producers over 2 minutes:

Metric	Measured Value	Analysis
Ingestion Throughput	315.1 req/s sustained	37,566 events ingested in 120s with 0 ingestion failures
Ingestion Latency (p95)	190 ms (Median: 120 ms)	Cheap atomic INSERT + flush keeps ingestion path responsive
Delivery Latency (p95)	5.0 ms (p50: 2.0 ms, p99: 14.0 ms)	Fast local HTTP dispatch via httpx.AsyncClient pool
Eventual Delivery Success	100%	Retried failed events until delivered; 0 permanently dropped
Retry Rate	30.2%	Matches the configured 30% receiver failure rate
Dead-Letter (DLQ) Rate	0.0%	Backoff allowed receiver to recover before reaching 8-attempt cap

Full methodology, worker concurrency scaling analysis (5 → 20 workers), and geometric retry distributions are documented in docs/loadtest.md.

---

Key Design Decisions

1. At-Least-Once Delivery + Idempotent Consumer

Exactly-once delivery over HTTP is fundamentally impossible due to the Two Generals Problem (network partitions during ACK). The platform guarantees at-least-once delivery, while receivers enforce deduplication via event IDs for effectively-once processing. An example proving this ships in examples/receiver.py.

2. Transactional Outbox Pattern

Events and their corresponding delivery jobs are committed in the same PostgreSQL transaction. This guarantees an accepted event (202 Accepted) is never lost to an application crash between ingestion and queue enqueue (solving the dual-write problem).

3. Postgres as the Queue (FOR UPDATE SKIP LOCKED)

Workers claim available jobs using SELECT ... FOR UPDATE SKIP LOCKED. Concurrent worker tasks never lock the same row or block each other, and uncommitted claims automatically roll back if a worker crashes.

4. HMAC-SHA256 Payload Signing

Every webhook request carries X-Webhook-Signature (v1=<hex>) computed over timestamp.body using a unique per-endpoint secret. This prevents payload tampering and replay attacks.

5. Exponential Backoff with Full Jitter

Retry delays are calculated using uniform(0, min(cap, base × 2^attempt)). Full jitter prevents the thundering herd problem by desynchronizing retries after receiver downtime. Jobs transition to dead status after 8 failed attempts.

6. Per-Tenant Rate Limiting (Redis Token Bucket)

An atomic Redis Lua script enforces per-tenant delivery rate limits so one tenant's event storm cannot starve deliveries for other tenants. Rate limiting at delivery time preserves lossless ingestion.

---

Production Readiness & Deployment

Environment Configuration

Variable	Default	Description
DATABASE_URL	postgresql+asyncpg://postgres:postgres@localhost:5432/webhooks	Async SQLAlchemy database URL
REDIS_URL	redis://localhost:6379/0	Redis connection URL
WORKER_CONCURRENCY	5	Number of concurrent worker tasks per worker process
CLAIM_BATCH_SIZE	10	Rows claimed per SKIP LOCKED query
DELIVERY_TIMEOUT_S	10.0	HTTP request timeout for receiver dispatch
MAX_ATTEMPTS	8	Maximum retry attempts before routing to DLQ
BUCKET_CAPACITY	10	Token bucket capacity per tenant
BUCKET_REFILL_PER_S	5.0	Token bucket refill rate (events/second)

Deploying to Render / Railway

1. Database: Provision a managed PostgreSQL instance and run migrations:

   python -m alembic upgrade head

2. Cache: Provision a managed Redis instance.
3. API Service: Deploy container with command:

   uvicorn app.main:app --host 0.0.0.0 --port 8000

4. Worker Service: Deploy background worker with command:

   python -m app.worker

---

Development & Testing

# Install dependencies
uv sync

# Run linter
uv run ruff check .

# Run test suite (requires local Postgres on 5433 or set PGPORT)
PGPORT=5433 uv run pytest -v

# Run Locust load test
API_KEY="whsk_..." ENDPOINT_ID="..." uv run locust --headless --host http://localhost:8000 -u 50 -r 10 --run-time 2m --html results/load_test_report.html

---

Project Structure

app/
├── main.py              # FastAPI app setup and router aggregation
├── config.py            # Pydantic settings management
├── db.py                # Async SQLAlchemy engine and session factory
├── models.py            # Declarative models: Tenant, Endpoint, Event, Delivery, DeliveryAttempt
├── auth.py              # Cryptographic API key generation and Bearer token dependency
├── outbox.py            # Transactional outbox pattern implementation
├── delivery.py          # HTTP delivery engine with HMAC signing and attempt logging
├── worker.py            # Asyncio TaskGroup worker pool with SKIP LOCKED claiming
├── signing.py           # HMAC-SHA256 signature generator and verification helper
├── backoff.py           # Full-jitter exponential backoff calculator
├── ratelimit.py         # Redis Lua token bucket implementation
├── logging.py           # Structlog JSON and console formatting
└── routers/
    ├── tenants.py       # POST /tenants, GET /whoami
    ├── endpoints.py     # POST /endpoints, GET /endpoints
    ├── events.py        # POST /events (Idempotency-Key dedup)
    ├── deliveries.py    # GET /deliveries, POST /deliveries/{id}/retry
    └── metrics.py       # GET /metrics (SQL aggregation)
alembic/versions/        # Database schema migrations
examples/receiver.py     # Flaky receiver demonstrating idempotent consumption
locustfile.py            # Locust load test scenario
docs/
├── architecture.md      # Schema design, failure modes, and scaling analysis
└── loadtest.md          # Full load testing methodology and findings