model-router

An inference gateway that picks a model, proves the call is affordable before making it, falls back when a provider fails, and explains every one of those decisions in the response — section 3 of PLATFORM_SPEC.md.

The idea this repository exists to demonstrate: every routing decision is explainable, and the cost ceiling is enforced before the call rather than discovered after it.

Both halves are load-bearing. A router that picks a model but cannot say why is not observable, and the eval harness downstream scores routing decisions using exactly that explanation. A router that adds up costs after the responses arrive has not enforced a budget; it has written a receipt.

Zero runtime dependencies. Node 20.11+. TypeScript.

---

The ordering that is the whole design

flowchart TB
  cp["agent-control-plane<br/><i>InferenceRequest — spec §3</i>"]

  subgraph dispatch["Router#dispatch — src/router.ts"]
    direction TB
    v["<b>1. validate</b><br/><i>validate.ts</i><br/>InvalidRequest names the field"]
    p["<b>2. plan</b><br/><i>selection.ts · registry/</i><br/>hard filter → total deterministic rank"]
    c{"<b>3. cache</b><br/><i>cache.ts</i><br/>sha256 of messages+model+params"}

    subgraph gate["<b>4. gate</b> — per ranked candidate, no await anywhere in here"]
      direction TB
      g1["<b>a. THE COST CEILING</b><br/><i>money.ts</i> · estimate vs max_cost_usd"]
      g2["<b>b. the breaker</b><br/><i>breaker.ts</i> · per provider"]
      g3["<b>c. the ledger</b><br/><i>ledger.ts</i> · synchronous reserve"]
      g1 --> g2 --> g3
    end

    d["<b>5. dispatch</b><br/>Transport.invoke · deadline · AbortSignal"]
    s["<b>6. settle</b><br/>charge the real cost · record health · write cache"]
  end

  cp --> v --> p --> c
  c -->|"hit — free, nothing charged"| out
  c -->|"miss"| g1
  g3 -->|"reserved"| d --> s --> out
  d -.->|"failure — release the hold in full"| fb["<b>7. fallback</b><br/>next ranked candidate<br/>fallbacks_used += 1"]
  fb -.-> g1

  g1 -.->|"over_ceiling"| refuse
  g2 -.->|"circuit_open"| refuse
  g3 -.->|"over_budget"| refuse

  out["InferenceResponse<br/><b>+ route explanation</b>"]
  refuse["BudgetExceeded · CircuitOpen<br/>NoEligibleModel · AllProvidersFailed<br/>UpstreamTimeout<br/><b>+ route explanation</b>"]

  out --> tel["span — spec §5 → agent-telemetry"]
  refuse --> tel

  classDef gateBox fill:#7c2d12,stroke:#fb923c,color:#fff
  classDef exit fill:#14532d,stroke:#22c55e,color:#fff
  classDef bad fill:#7f1d1d,stroke:#ef4444,color:#fff
  class g1,g2,g3 gateBox
  class out exit
  class refuse bad

Loading

Steps 4a to 4c all happen before the transport is touched, and src/router.ts lines 178 to 271 — the entire path from the start of dispatch to the reservation — contains no await at all. That gap is where a concurrency bug would live, so there isn't one. Ten concurrent route() calls each run synchronously through their reservation before any of them suspends.

Full reasoning in ADR 0001.

Why the estimate is a bound and not a guess

Cost is input_tokens × input_price + output_tokens × output_price. Before the call, only one of those four numbers is known for certain. This router makes a second one certain: it reserves an explicit output allowance and hands the transport the same cap it reserved budget for. A provider that honours the cap cannot produce a call that costs more than the estimate on the output side.

Rounding points the same way. Estimates round up, budgets and ceilings round down, settled actuals round to nearest — so rounding can never make a call look affordable when it is not (ADR 0002).

The input side is exact only relative to the injected TokenEstimator. The default is a character heuristic that leans high (3.5 characters per token, against the ~4 English prose usually gets); inject a real tokenizer to make it exact. Where the provider's own count disagrees, settle charges the real number and the difference is recorded as an overdraft — visible in the ledger snapshot, and enough to refuse the run's next call. A breach that gets absorbed silently is worse than one that gets charged loudly.

---

The signature idea, and the tests that attack it

test/signature.budget.test.ts is written to break the claim, not to demonstrate it. Each test names the specific way the guarantee could be fake:

Test	The lie it is looking for
refuses without calling any provider when the estimate breaches max_cost_usd	the check happens, but the call went out anyway — asserts transport.callCount === 0
refuses on the estimate even when the call would actually have come in under the ceiling	the "check" is a post-hoc reconciliation wearing a hat — the scripted provider would have returned 10 output tokens
explains the refusal: route.chosen is null and the numbers are in the error	a refusal that cannot be explained
caps the provider call at the same output allowance it reserved budget for	the estimate is a guess, because nothing bounds the output
lets through only as many concurrent calls as the run budget can afford	the ledger is not atomic — 10 concurrent requests, budget for 3.5, transport gated open mid-flight
holds the line when many runs share one router	one run's spend leaks into another's budget
reports cache_hit, charges nothing, and does not call the provider	a free path is charged
serves a cache hit even when the run has no budget left	a free path is gated by a check that exists to stop spending
records the overdraft and refuses the next call	a provider that ignores the cap invalidates the check silently
is refused before any call when requireRunBudget is set	an undeclared budget quietly means unlimited

The concurrency test asserts more than a count: exactly 3 of 10 reach the provider, 7 reject with BudgetExceeded, reserved never exceeds the budget mid-flight, every reservation is released or settled afterwards, and spent + remaining == budget exactly, in integer nano-USD.

Checked by mutation, not just by passing. Changing the one line in CostLedger#reserve that subtracts in-flight reservations — the classic version of this bug, where only settled spend counts — fails exactly 3 of the 116 tests, 2 of them in the signature file: lets through only as many concurrent calls as the run budget can afford (10 calls reached the transport instead of 3) and holds the line when many runs share one router (4 instead of 2). The line was then restored and the suite re-run green.

The lint was verified the same way, against a file carrying one violation of each rule.

---

How it fits the platform

Five repositories, one control plane. This is the inference leg.

flowchart LR
  cp["<b>agent-control-plane</b><br/>Python<br/>runs, DAG, budgets"]
  mr["<b>model-router</b><br/>TypeScript<br/><i>you are here</i>"]
  sb["agent-sandbox<br/>Python"]
  tl["<b>agent-telemetry</b><br/>Python<br/>span schema"]
  ev["<b>agent-evals</b><br/>Python<br/>replay · score"]

  cp -->|"§3 InferenceRequest<br/>→ response or typed error"| mr
  cp -->|"§4 actions"| sb
  mr -->|"§5 inference span"| tl
  sb -->|"§5 action span"| tl
  tl -->|"§6 recorded run, JSONL"| ev
  ev -.->|"scores route from §3"| mr

  classDef me fill:#1e3a8a,stroke:#60a5fa,color:#fff
  class mr me

Loading

Contract	Section	What this repository implements
Identifiers	§1	Consumes run_id, step_id, trace_id as opaque strings; mints span_id as 16 hex, the OTel format. Nothing is an autoincrement.
Run state machine	§2 rule 3	"A run enters BUDGET_EXHAUSTED when the router reports the budget ledger would go negative. The check happens before the call, not after." This is the sentence the repository is built around. The router reports it as BudgetExceeded; the transition itself belongs to the control plane.
Inference contract	§3	The whole of it. Request shape, response shape, the mandatory route object, and the five typed error names — NoEligibleModel, BudgetExceeded, AllProvidersFailed, UpstreamTimeout, CircuitOpen, each set as the class's name so a Python consumer can switch on the serialised form.
Span schema	§5	Emits one kind: "inference" span named model_router.route per request, with cost_usd as a first-class attribute so a run total is a sum over spans. Both of the section's enforced-in-code rules: bodies withheld by default, and whatever was withheld named in redacted.
Recorded-run format	§6	Carries attributes.request_digest as sha256:… so a recorded inference correlates on replay, and keeps candidate ordering total and deterministic so a replay reaches the same decision.

Talks to: agent-control-plane (inbound, §3) and agent-telemetry (outbound, §5). Does not talk to: agent-sandbox — both are called by the control plane and neither calls the other. agent-evals reads this router's decisions second-hand, through the recorded run.

Takes no dependency on any of them. The span shape is duplicated as a local interface in src/spans.ts rather than imported, which is the price of the empty dependencies block and is paid deliberately.

---

Install and run

npm install      # devDependencies only: typescript, @types/node
npm run ci       # clean, lint, typecheck, build, test  ← the whole gate
npm test         # build, then node --test
npm run demo     # a narrated offline tour, examples/demo.ts

Using it

import { Registry, Router } from 'model-router';

const router = new Router({
  registry: Registry.fromFile('./my-catalog.json'),
  transports: [anthropicTransport, openaiTransport], // you write these
  breaker: { failureThreshold: 5, cooldownMs: 30_000 },
  spans: mySpanSink,
});

router.openRun('run_01J...', 0.25); // USD for the whole run

const response = await router.route({
  run_id: 'run_01J...',
  step_id: 'step_01J...',
  trace_id: '4bf92f3577b34da6a3ce929d0e0e4736',
  messages: [{ role: 'user', content: '...' }],
  requirements: {
    capabilities: ['tool_use', 'json_mode'],
    max_cost_usd: 0.05,
    max_latency_ms: 30_000,
    prefer: 'cost',
  },
  cache: { mode: 'read_write' },
});

Every response carries the explanation:

response.route;
// {
//   considered: ['citrine-fast', 'acme-small', 'borealis-pro', 'acme-large'],
//   chosen: 'citrine-fast',
//   reason: 'cheapest of 4 eligible candidates meeting capabilities [tool_use, json_mode] '
//         + 'and latency ceiling 30000ms; estimated $0.003085 for 25 input tokens and up to '
//         + '2048 output tokens, checked before the call',
//   fallbacks_used: 0,
//   rejected: [{ model: 'borealis-lite', reason: 'missing capability tool_use' }, ...],
//   skipped: [],
//   strategy: 'cost',
// }

route is on the failures too. RouterError has a route field and a toJSON(); a BudgetExceeded reports chosen: null plus every estimate and the ceiling each one breached. A refusal you cannot explain is as opaque as a choice you cannot explain.

The Transport is the only thing you implement

interface Transport {
  readonly provider: string;
  invoke(request: TransportRequest): Promise<TransportResult>;
}

It is also the only thing that touches a network. TransportRequest carries max_output_tokens — the cap the estimate paid for; forward it to the provider or the bound becomes a guess — a deadline_ms, and an AbortSignal that fires when the deadline passes. Reasoning in ADR 0003; the obligations this places on your transport are in the threat model, T8.

The registry

Pricing is data, in src/registry/models.json, and src/registry/catalog.ts is the only file in the package that reads it. A test fails the build if any other source file so much as mentions usd_per_mtok.

The shipped catalogue is synthetic: invented providers (acme, borealis, citrine) and placeholder prices chosen to give the strategies a meaningful spread. Every entry says so in pricing.pricing_source, Registry#placeholderPricedModels() lists them, and new Router({ requireRealPricing: true }) refuses to start while any remain. Supply your own with Registry.fromFile(path).

Field	Meaning
capabilities	hard filter against requirements.capabilities
pricing	per-million-token input and output prices, plus their provenance
typical_latency_ms	hard filter against max_latency_ms, and the latency score
max_output_tokens	the output allowance reserved when the request names none
context_window	hard filter against estimated input plus output allowance
quality_score	operator-declared preference weight for prefer: 'quality'. Not a measurement — this router does not benchmark models

Selection

Two phases, kept apart so the decision stays explainable. The hard filter (capabilities, latency ceiling, context window, whether a transport exists at all) removes models with a reason attached; they appear in route.rejected. The survivors are put in a total, deterministic order by prefer — every comparator ends in a model-id tie-break, so the same request against the same registry always routes the same way, which is what makes a recorded run replayable.

Cost is deliberately not a hard filter. Affordability depends on what the run has already spent, so it is decided against the ledger at dispatch and reported in route.skipped with the numbers attached — otherwise "your ceiling is too low" arrives disguised as "nothing matched your capabilities". ADR 0004.

---

Failure, designed

What happens when a dependency is down, slow, or lying.

What breaks	What happens
A provider returns an error	The attempt fails, the reservation is released in full (nothing charged), the breaker counts it, and the chain walks to the next ranked candidate. route.fallbacks_used counts the walk.
A provider hangs	The per-attempt deadline (max_latency_ms, else defaultTimeoutMs = 60s) fires an AbortController. Whichever promise wins the race, a fired deadline is reported as UpstreamTimeout rather than as whatever the HTTP client called it. A lone breach surfaces as UpstreamTimeout; with somewhere to fall back to it is just another failure in the chain.
A provider keeps failing	Five consecutive failures (default) open its breaker. Further calls are refused without being made until a 30s cooldown admits exactly one probe — not the whole backlog. The probe closes it or re-opens it with the cooldown restarted. Every transition is published to onBreakerTransition with from, to, reason, at. ADR 0005.
Every candidate is behind an open breaker	CircuitOpen, no call made.
Every candidate is unaffordable	BudgetExceeded, no call made, every estimate and the ceiling in the explanation.
The chain is exhausted	AllProvidersFailed, with each attempt's model, provider and error in details.failures.
A provider lies about usage, or ignores the output cap	The real cost is charged, the excess is recorded as overdraft_usd, remaining_usd goes negative, and the run's next call is refused. Charged loudly rather than absorbed. Under-reporting is believed — see threat model T2.
The registry lists a model with no transport	Hard-rejected at plan time with no transport is registered for provider X, visible in route.rejected, rather than surfacing as a surprise at dispatch.
Nothing in the registry fits	NoEligibleModel, with a per-model reason for every rejection.
The catalogue is malformed	CatalogError at load, naming the offending field. The router never starts on half a catalogue.
The caller sends a bad request	InvalidRequest, naming the field. Deliberately not a RouterError, so isRouterError(e) never swallows a caller bug.

Every row is asserted in the suite; none of it is aspiration.

---

Observability

Every request emits one span shaped to spec section 5 (kind: "inference", name: "model_router.route"), carrying the routing decision, the usage, the cost and the request digest. Two of the section's rules are enforced in code rather than documented as advice:

• Prompt and response bodies are not exported by default. capturePrompts: true opts in.
• What was withheld is named in redacted, so the gap is visible rather than silent.

router.breakerSnapshot() and router.ledgerSnapshot(run_id) are readable at any time; the ledger snapshot reports both USD and exact integer nano-USD, and the nano fields are the authoritative ones.

---

Tests

Offline. No network, no credentials, no live model, nothing to mock at the HTTP layer: time is a ManualClock and providers are FakeTransports. test/hygiene.test.ts makes that a property rather than a promise — it asserts that nothing under src/ imports anything but node: builtins and relative paths, and that no network API (fetch, node:http, node:net, node:dns, node:tls, node:dgram) is reachable from the source at all.

$ npm run ci
lint: 33 files, 12 rules, no findings
...
ℹ tests 116
ℹ suites 25
ℹ pass 116
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 447.3728

116 tests, 25 suites, 0 failures. Size, from find src -name "*.ts" | xargs wc -l: 2,243 lines of TypeScript across 18 source files, plus a 97-line JSON catalogue. Tests are a further 1,960 lines across 10 test files and one fixture module.

CI

.github/workflows/ci.yml, on Node 20.x and 22.x. Every step was run locally before it was written into the workflow:

Step	Command	Can it fail?
lint	npm run lint	Yes — verified against a probe file violating six rules; exit 1, six findings
types	npm run typecheck	Yes — strict, noUnusedLocals, noUnusedParameters, noUncheckedIndexedAccess, exactOptionalPropertyTypes
build	npm run build	Yes
tests	node --test "dist/test/**/*.test.js"	Yes — verified by mutating CostLedger#reserve; 3 failures
dependency audit	npm audit --omit=dev --audit-level=low	Yes — currently an empty runtime tree, 0 vulnerabilities, so this asserts it stayed empty
example	node dist/examples/demo.js	Yes

No badge, because there is no remote for it to point at.

The linter is scripts/lint.mjs, a zero-dependency script holding twelve invariants a type-checker cannot express — among them no console in a library, no any, no focused test, no TODO marker, no process.env read in src/, relative imports carrying their .js, and a module header on anything that declares something. Not ESLint, which would have meant a dependency and a config (ADR 0003).

Documentation

• docs/adr/ — five decisions where a real alternative was rejected
• docs/threat-model.md — assets, seven trust boundaries, thirteen threats, and which of them are not mitigated

---

Limitations

Built from what was actually not done.

• No linter in the conventional sense. scripts/lint.mjs enforces this repository's own invariants and tsconfig covers the generic ones, but neither is ESLint, and neither does type-aware analysis of the kind @typescript-eslint would.
• In-memory, single-process. The ledger, the cache and the breakers live in one process. Two gateway instances each enforce the budget they can see, which is not the same as enforcing it once. ResponseCache is an interface you can back with something shared; the ledger and breakers are not yet abstracted that way.
• Run accounts have no TTL, eviction or persistence. They grow until closeRun is called, and a restart forgets every budget.
• No cache stampede protection. N identical concurrent requests all miss and all call the provider. Single-flight would fix it and is not implemented.
• The input-token estimate is a heuristic unless you inject a tokenizer. The output side of the bound is exact; the input side is exact only relative to the estimator. The overdraft mechanism is what catches the difference.
• No streaming. Transport returns a complete result. Streaming would move output-cap enforcement from a request parameter to mid-stream accounting, which deserves its own design rather than a flag.
• No retries within a provider. A failure falls back to the next candidate. Retrying the same provider after a backoff on a 429 is a real strategy and is not implemented.
• The deadline is per attempt, not per request, so a three-candidate chain can take three times max_latency_ms in the worst case.
• The breaker counts consecutive failures, not a failure rate. A provider failing 40% of calls may never trip, because one success resets the counter.
• parent_span_id is always null. The section 3 contract passes a trace_id but not the control plane's step span id, and the contract was not extended unilaterally.
• Cache lookup is exact-match only, probing candidates in rank order. No semantic or prefix similarity, and nothing clever about which cached model would have been preferable.
• The shipped catalogue is synthetic. Invented providers, placeholder prices, every entry declaring pricing_source: "placeholder". No claim is made anywhere in this repository about any real provider's pricing.
• Three security gaps are known and unfixed, documented rather than quietly carried: a transport error message reaches the span stream verbatim and is not redacted (T8); a catalogue priced at zero passes validation and silently disables every ceiling (T5); a SpanSink that throws turns a paid, settled call into an error (T11). See docs/threat-model.md.

Licence

MIT. See LICENSE.