fix(recon): replace stub with claims and payout reconciliation engine

docs/runbook-reconciliation.md

@@ -201,11 +201,119 @@ adapter before relying on reconciliation for live balance verification.
 
 ---
 
+## Claims & Payout Reconciliation (Automated & On-Demand)
+
+### Purpose & Architecture
+
+The Claims & Payout Reconciliation engine periodically compares on-chain claim events emitted by the Soroban prediction market contracts against database payout and prediction records. Unlike the single-market position diff tool, payout reconciliation inspects settlement distributions across time windows and reports persisted, auditable reports.
+
+When discrepancies are detected, the system:
+1. Records a fingerprint-hashed reconciliation report in `reconciliation_reports` with status `discrepancies_found`.
+2. Persists granular discrepancy rows in `reconciliation_discrepancies`.
+3. Increments Prometheus metrics `reconciliation_discrepancies_total{kind="<kind>"}`.
+4. Emits structured audit log entries (`reconciliation.payouts.completed`) with severity metadata for incident alerting.
+
+---
+
+### Discrepancy Kinds & Severities
+
+| Kind | Severity | Description | Immediate Impact |
+|---|---|---|---|
+| `claimed_on_chain_missing_in_db` | **CRITICAL** | A claim was settled on-chain via Soroban event, but no corresponding payout record exists in the backend DB. | Indexer lag, missed webhook, or untracked contract interaction. |
+| `db_paid_missing_on_chain` | **CRITICAL** | DB marks payout as paid/settled, but no matching on-chain event exists within the reconciliation window. | Potential phantom credit, failed transaction that was marked paid, or RPC desync. |
+| `amount_mismatch` | **CRITICAL** | On-chain claim amount differs from DB payout amount for the same claim identifier / user prediction. | Calculation error, rounding bug, or unexpected smart contract fee deduction. |
+| `claim_for_losing_prediction` | **CRITICAL** | On-chain claim event references a prediction row whose resolved outcome does not match the winning outcome. | Contract logic bug, market resolution dispute divergence, or unauthorized drain attempt. |
+| `duplicate_on_chain_claim` | **CRITICAL** | Multiple on-chain claim events detected for the same unique prediction ID. | Smart contract replay bug or missing contract-level payout check. |
+| `stale_unclaimed` | **MEDIUM / INFO** | DB prediction is marked eligible for winning claim, but has remained unclaimed beyond the configured grace period (e.g. 7+ days). | User did not claim tokens yet, or claiming UI has issues. Low operational risk, no fund loss. |
+
+---
+
+### Time Windows & Lookback Parameters
+
+Reconciliation runs against parameterized time boundaries:
+- `windowStart`: Beginning of the evaluation period (ISO timestamp or Date).
+- `windowEnd`: End of the evaluation period (defaults to execution timestamp).
+- `settlementGracePeriodMs`: Configurable buffer (default 1 hour) to allow in-flight transactions and indexer ingestion to settle before flagging a missing DB row or missing on-chain claim.
+
+---
+
+### Querying Reports via Admin API
+
+Operators can inspect reconciliation reports and their granular discrepancies using the Admin API:
+
+#### 1. List Reconciliation Reports
+```bash
+curl --request GET \
+  --url "https://<host>/api/admin/reports?limit=20&status=discrepancies_found" \
+  --header 'Authorization: Bearer <admin-jwt>'
+```
+- Requires permission scope: `report:read` (or `role: "admin"`).
+- Supports keyset pagination using `cursor` and `limit`.
+- Response includes `id`, `windowStart`, `windowEnd`, `status`, `discrepancyCount`, `fingerprint`, and `createdAt`.
+
+#### 2. Get Detailed Reconciliation Report
+```bash
+curl --request GET \
+  --url "https://<host>/api/admin/reports/<REPORT_ID>" \
+  --header 'Authorization: Bearer <admin-jwt>'
+```
+- Returns the complete report along with the `discrepancies` array containing:
+  - `kind`: One of the 6 discrepancy types
+  - `marketId`: Target prediction market
+  - `predictionId`: Affected prediction ID (if resolved)
+  - `claimantAddress`: Stellar account address
+  - `onChainAmount` & `dbAmount`: Exact Stroop-level amounts
+  - `severity`: `"critical"` | `"medium"` | `"info"`
+  - `details`: JSON object with execution context and transaction hashes
+
+---
+
+### Operator Triage & Remediation Runbook
+
+When a critical discrepancy alert is raised (`reconciliation_discrepancies_total > 0`):
+
+#### Scenario A: `claimed_on_chain_missing_in_db`
+1. Fetch report details: `GET /api/admin/reports/:id`.
+2. Extract the transaction hash and claimant address from `details`.
+3. Check indexer health probe: `GET /api/admin/health/detail` to verify if the Soroban indexer is lagging.
+4. If indexer is lagging, run gap backfill:
+   ```bash
+   POST /api/admin/reindex { "ledger": <start_ledger> }
+   ```
+5. If indexer is healthy, verify whether the event was skipped due to malformed payload in the dead-letter queue:
+   ```bash
+   GET /api/admin/webhooks/dlq
+   ```
+
+#### Scenario B: `db_paid_missing_on_chain`
+1. Query the payout transaction hash in Stellar Horizon / Soroban RPC.
+2. If the transaction failed on-chain or never submitted:
+   - Check the outbox queue (`docs/side-effect-outbox.md`).
+   - Revert DB status from `paid` to `pending` or re-queue the payout worker job.
+3. If transaction succeeded on-chain but was outside the reconciliation time window, adjust `windowStart` or increase `settlementGracePeriodMs`.
+
+#### Scenario C: `amount_mismatch`
+1. Compare `onChainAmount` vs `dbAmount` in the discrepancy details.
+2. Note: Amounts are stored as exact 7-decimal Stroop values (e.g. `10000000` = `1.0000000 XLM`).
+3. If fees or pool split caused deviation, verify contract formula against `payoutCalculationService.ts`.
+4. If DB over-credited, freeze account payout withdrawals and conduct ledger balance review.
+
+#### Scenario D: `claim_for_losing_prediction`
+1. **HALT TRADING & PAYOUTS**: Immediately inspect market resolution status.
+2. Verify Soroban contract state using `soroban contract invoke` for the affected `marketId`.
+3. Audit oracle resolution events (`docs/ORACLE_SETTLEMENT_GUARD.md`) to verify if an erroneous resolution was executed on-chain.
+
+#### Scenario E: `duplicate_on_chain_claim`
+1. Inspect both claim transactions on Horizon to check if the contract allowed a double payout.
+2. If verified duplicate, engage security escalation protocol immediately to pause contract withdrawals.
+
+---
+
 ## Related resources
 
-- `src/routes/admin/reconciliation.ts` — HTTP handler
-- `src/services/reconciliationService.ts` — diff logic and audit
-- `src/workers/reconciliationWorker.ts` — async BullMQ worker
-- `docs/job-queue.md` — queue design and enqueue patterns
-- `POST /api/admin/reindex` — trigger a ledger backfill
-- `GET /api/admin/health/detail` — check indexer lag before reconciling
+- `src/routes/admin/reconciliation.ts` — Admin reconciliation HTTP endpoints (`GET /api/admin/reports`, `GET /reports/:id`)
+- `src/services/reconciliationService.ts` — Diff engine, payout rules, and report persistence
+- `src/workers/reconciliationWorker.ts` — Scheduled BullMQ background worker
+- `docs/job-queue.md` — BullMQ queue design and enqueue patterns
+- `POST /api/admin/reindex` — Trigger Soroban ledger backfill
+- `GET /api/admin/health/detail` — Check indexer lag before reconciling

drizzle/0028_add_reconciliation_reports.sql

@@ -0,0 +1,46 @@
+-- Migration: add reconciliation_reports and reconciliation_discrepancies tables
+-- These tables persist nightly and on-demand payout reconciliation runs,
+-- tracking discrepancies between winning predictions, claims ledger rows, and indexed win_clm events.
+
+CREATE TABLE IF NOT EXISTS "reconciliation_reports" (
+  "id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
+  "fingerprint" text NOT NULL UNIQUE,
+  "status" text NOT NULL DEFAULT 'ok',
+  "since" timestamp with time zone,
+  "until" timestamp with time zone,
+  "total_keys" integer NOT NULL DEFAULT 0,
+  "matched_keys" integer NOT NULL DEFAULT 0,
+  "unmatched_keys" integer NOT NULL DEFAULT 0,
+  "discrepancy_count" integer NOT NULL DEFAULT 0,
+  "summary" jsonb NOT NULL DEFAULT '{}'::jsonb,
+  "created_at" timestamp with time zone NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS "reconciliation_reports_created_at_id_idx"
+  ON "reconciliation_reports" ("created_at", "id");
+
+CREATE TABLE IF NOT EXISTS "reconciliation_discrepancies" (
+  "id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
+  "report_id" uuid NOT NULL REFERENCES "reconciliation_reports"("id") ON DELETE CASCADE,
+  "kind" text NOT NULL,
+  "user_id" uuid REFERENCES "users"("id") ON DELETE SET NULL,
+  "stellar_address" text,
+  "market_id" text NOT NULL,
+  "prediction_id" uuid,
+  "claim_id" uuid,
+  "tx_hash" text,
+  "db_amount" text,
+  "on_chain_amount" text,
+  "difference" text,
+  "details" jsonb NOT NULL DEFAULT '{}'::jsonb,
+  "created_at" timestamp with time zone NOT NULL DEFAULT now()
+);
+
+CREATE INDEX IF NOT EXISTS "reconciliation_discrepancies_report_id_idx"
+  ON "reconciliation_discrepancies" ("report_id");
+
+CREATE INDEX IF NOT EXISTS "reconciliation_discrepancies_kind_idx"
+  ON "reconciliation_discrepancies" ("kind");
+
+CREATE INDEX IF NOT EXISTS "reconciliation_discrepancies_report_kind_idx"
+  ON "reconciliation_discrepancies" ("report_id", "kind");

drizzle/meta/_journal.json

@@ -50,6 +50,13 @@
       "when": 1785400000000,
       "tag": "0027_claims_prepare_submit",
       "breakpoints": true
+    },
+    {
+      "idx": 28,
+      "version": "7",
+      "when": 1785450000000,
+      "tag": "0028_add_reconciliation_reports",
+      "breakpoints": true
     }
   ]
 }