From cc4ce6096c950f51c15572abbe65351ec4f5cd6b Mon Sep 17 00:00:00 2001 From: Andrey Golovanov Date: Sat, 31 Oct 2026 09:25:30 +0000 Subject: [PATCH 1/2] Preserve development plans and roadmap in the repository --- README.md | 5 + docs/design/README.md | 55 + docs/design/archive/README.md | 29 + docs/design/archive/network-layer-r17.md | 1876 +++++++ docs/design/archive/protocols-r8.md | 4305 +++++++++++++++++ docs/design/archive/routing-policy-r4.md | 267 + docs/design/archive/routing-policy-reviews.md | 90 + docs/design/protocols.md | 514 ++ docs/design/routing-policy.md | 636 +++ docs/plan.md | 199 + 10 files changed, 7976 insertions(+) create mode 100644 docs/design/README.md create mode 100644 docs/design/archive/README.md create mode 100644 docs/design/archive/network-layer-r17.md create mode 100644 docs/design/archive/protocols-r8.md create mode 100644 docs/design/archive/routing-policy-r4.md create mode 100644 docs/design/archive/routing-policy-reviews.md create mode 100644 docs/design/protocols.md create mode 100644 docs/design/routing-policy.md create mode 100644 docs/plan.md diff --git a/README.md b/README.md index 0916af1..c8ac7fa 100644 --- a/README.md +++ b/README.md @@ -231,6 +231,11 @@ CI runs Python 3.11 to 3.14 and the free-threaded 3.14t build. Reports on each development slice, with their evidence and measurements, are in [docs/history](docs/history/README.md). +Planned work (the Open/R and BGP foundation and the shared route-policy +language) is ordered in the roadmap, [docs/plan.md](docs/plan.md). Its +contracts are in [docs/design](docs/design/README.md); they describe future +behaviour, not features available today. + ## License MIT diff --git a/docs/design/README.md b/docs/design/README.md new file mode 100644 index 0000000..cb5a49c --- /dev/null +++ b/docs/design/README.md @@ -0,0 +1,55 @@ +# Design + +These pages specify behaviour NetLab-Sim does not have yet: the foundation for +Open/R and BGP, and the shared route-policy language. They are for maintainers +who implement or review that work. + +## Which document decides what + +| Question | Where the answer lives | +|---|---| +| What the code does now | The code and its tests, then the [README](../../README.md) and the [reference](../reference.md). [History reports](../history/README.md) record what each delivered slice changed and how it was checked, as of their date | +| What future work must guarantee | The design pages below. Each statement is a requirement for planned code, or an internal guarantee or decision that implemented code still keeps; neither is user documentation | +| What is planned, in which order, and its state | The [roadmap](../plan.md), the only place with status | +| Where a requirement came from | The [archive](archive/README.md): frozen design documents kept for extraction and provenance. They are not current rules and do not override anything above | + +When a design page and the code disagree, the code describes today and the +page describes the target; the plan row says which slice closes the gap. When +the archive and a design page disagree, the design page wins. + +## Pages + +- [Protocol foundation](protocols.md): the common design decisions, the + contracts for F0 and F5, the requirements for F1a, C6, S1a and S1b, and the + list of pending contracts with their archived sources. +- [Routing policy](routing-policy.md): the route-policy language, attachment + points, per-node configuration, lifecycle and the acceptance examples E01 to + E24. + +## Pending contracts + +The roadmap records whether a slice's contract is ready +([status values](../plan.md#status-values)). Reconciling a pending contract +means: collect every requirement from the archived sections and from all later +amendments of them, write them as exact rules with the examples and tests that +check them, and resolve or record each conflict. A pending contract is +reconciled before its implementation starts, not all at once: work on the next +slice needs only its own contract and those it depends on. + +Rules for these pages: + +- One home per requirement. The plan holds order and state; a design page holds + the rule; neither repeats the other. +- A design page names no status and no schedule. Planned names (records, + modules, settings) are written as code-formatted names and are not links; + links to code point only at files that exist. +- A behaviour taken from another implementation (FRR, Open/R) names its pinned + source. Such a statement is a vendor reference until a reproducer or pinned + fixture confirms it. +- When a slice lands, the behaviour users rely on moves into the README or the + reference in the same change. Internal guarantees, their proofs and + decisions that still bind later work may stay on the design page; anything + that is no longer true or no longer binding is removed. + +How the roadmap is kept current is under +[Maintaining the roadmap](../plan.md#maintaining-the-roadmap). diff --git a/docs/design/archive/README.md b/docs/design/archive/README.md new file mode 100644 index 0000000..af16da6 --- /dev/null +++ b/docs/design/archive/README.md @@ -0,0 +1,29 @@ +# Design archive + +Frozen copies of the design documents written before the plan moved into this +repository. They are kept byte for byte as they were last revised, so that +every requirement can be traced to its source. They are not current rules: the +[design pages](../README.md) and [docs/plan.md](../../plan.md) override them, +and their status statements (what is implemented, what can start, test counts, +review outcomes, measurements) describe their own date and older commits. + +The project was called NetSim, with the package `netsim`, and from +2026-10-02 NetGraph-Sim (`ngraph_sim`); it is now NetLab-Sim (`netlab_sim`). +The documents keep the old names, file paths and links. Links to files +outside this repository (research notes, local checkouts, other documents +under their old names) are not repaired and may not resolve; nothing current +depends on them. + +| File | Original name | Content | Revision and date | SHA-256 | +|---|---|---|---|---| +| [network-layer-r17.md](network-layer-r17.md) | `hazy-foraging-cloud.md` | Network-layer design of record: Gates A, B and C | Revision 17 (Gate C implemented), with a rename note of 2026-10-02 | `22bf74952aaeb9c975b428b88fcd1bbf4f9f5202b4a4f49cd7d5ec42112aca5e` | +| [protocols-r8.md](protocols-r8.md) | `netsim-protocols.md` | Open/R and BGP design and plan; slice table §15.9 amended by §16.9; S0 delivery in §17 | Revision 8, 2026-09-27, with S0 updates of 2026-10-02 | `995c2d5b0c1ef986804125cd5737971e01beedce7d7329105e6b8ed1caaf53a9` | +| [routing-policy-r4.md](routing-policy-r4.md) | `routing-policy.md` | Routing-policy language and hooks | Revision 4, 2026-09-27 | `2319e0a49de9aa29fefafe92bc757a9db15bfb655f1db55aa977c4db1c8b36d3` | +| [routing-policy-reviews.md](routing-policy-reviews.md) | `routing-policy-reviews.md` | Review passes that produced routing-policy revision 4 | 2026-09-27 | `8e2f9d5759ea133f4fe77ee90e1b049cc01ca8c1884ba78366fbdf72d742283a` | + +Within `protocols-r8.md`, later sections override earlier ones (§16 over §15, +§15 over the body). `routing-policy-r4.md` links to its review record by its +original file name, which is kept here. + +To check the copies, run `shasum -a 256` on the four files in this directory +and compare with the table. diff --git a/docs/design/archive/network-layer-r17.md b/docs/design/archive/network-layer-r17.md new file mode 100644 index 0000000..32b8fe9 --- /dev/null +++ b/docs/design/archive/network-layer-r17.md @@ -0,0 +1,1876 @@ +# NetSim network layer: design (revision 17: Gate C implemented and confirmed) + +> **Rename, 2026-10-02:** the project is now NetGraph-Sim: distribution `ngraph-sim`, package `ngraph_sim`, CLI `ngraph-sim`, scenario keys `attrs.sim` / `data.sim`, workflow step `SimStudy`. Paths `netsim/...` and the old names below refer to the same code, now in `ngraph_sim/`. The repository is github.com/networmix/NetGraph-Sim and the checkout `~/ws/NetGraph-Sim`. Legacy and compatibility paths described below (Study `settle` mode, `keep_*` aliases, top-level NetGraph attrs, `strict=False`, `Simulation.recorder`, unbound NHT keys, untagged deliveries) were removed on 2026-10-02. + + +## Context and scope + +NetSim started as a zero-dependency discrete-event engine (`netsim/core.py`, +`netsim/resources.py`); Gates A and B below are implemented on `main` +(2026-09-20). The goal is a network layer that simulates devices +with a RIB and a FIB, three interface kinds (Loopback, Ethernet, +PortChannel with min-links), IPv4/IPv6/MAC addressing and UP/DOWN events, +and uses that forwarding state to **place flows (demands) and compute +capacities and utilization** as topology and routing state change over +time, including **traffic engineering with SRv6** (flows on pre-defined +paths expressed as SR Policies, micro-SIDs included). Packets (Ethernet + +IPv4/IPv6 header objects with an SRH) assist placement: hash-based path +selection for a microflow, loop, blackhole, MTU and TTL detection, tracing +and SRv6 header processing. They are not a per-packet queueing simulation. + +The design is delivered in three **gates**; each gate is a shippable, +tested simulator, and nothing from a later gate is a dependency of an +earlier one: + +- **Gate A**: immutable state tree, explicit scheduler rounds, + interfaces and LAG with physical-failure accounting, static and oracle + RIB/FIB, the plain-IP forwarding records and action interpreter + (`forwarding.py`), plain-IP FLUID and HASH utilization under failures, + adapters with `srv6=False`. +- **Gate B**: the same interpreter extended with exactly the supported SR + behaviours. **Authoritative Gate B behaviour set:** H.Encaps and + H.Encaps.Red at the headend; End and End.X with the PSP and USD flavors + (USP rejected); default-table End.DT46; NEXT-C-SID applies to End, + End.X and End.DT46 (uN, uA, uDT46). Rejected in Gate B with explicit + errors: on-wire B6 (End.B6.Encaps, uB6), nested encapsulation of an + already encapsulated packet, End.DT4, End.DT6, End.DX4, End.DX6, VRF + table selection. Local BSID-to-policy steering (a next hop resolving to + the headend's own BSID) is supported and is distinct from an on-wire B6 + endpoint. Gate B includes the minimal owner-scoped SID and policy + operations (`SidClient`, `PolicyClient` with one owner per policy); + Gate C adds agent execution and transport. One-owner SR Policies and + tested rate and capacity accounting complete the gate. Every table, + pseudocode block, adapter and acceptance row below is read under this + set; behaviours listed for completeness but not in the set are + post-Gate-C work. +- **Gate C**: scoped client and NHT contract, one minimal scoped FIFO + connection service (incarnations, timeouts) plus link channels, a + minimal link-state reference agent with cold-start tests. + +Later gates extend existing modules; no later gate supplies a +dependency an earlier gate needs. + +Deferred beyond Gate C, with named extension points: transport features +beyond the enumerated connection service, full Open/R and BGP, stateful +controllers, multi-client BSIDs, nested B6, resilient hashing, +REPLACE-C-SID, midpoint protection, VRFs, runtime checkpoint and restore, +broad adapter round trips, in-band protocol transport. + +## Provenance + +| NetSim concept | FBOSS | FRRouting | SONiC | Cumulus / Linux | +|---|---|---|---|---| +| Immutable versioned state tree; deltas | `SwitchState`, `StateDelta`, `HwSwitch::stateChanged` | | CONFIG_DB / APPL_DB / STATE_DB | | +| Derivations in dependency order | `StateObserver`s | zebra meta-queue | orchagent PortsOrch → IntfsOrch → NeighOrch → RouteOrch | | +| Oper state values | | | | netdev `operstate` = RFC 2863 | +| Routes keyed by client with distance | `RouteNextHopsMulti` by `ClientID`, `AdminDistance` | `zapi_route(type, instance, distance, metric)`, `connect` / `local` types | STATIC_ROUTE `distance` | | +| Next-hop kinds; SRv6 on the next hop | `NextHop` (`srv6SegmentList`, `tunnelType`) | `NEXTHOP_TYPE_*`, `blackhole_type`, staticd `segments` | ROUTE_TABLE `nexthop / ifname / weight / segment / seg_src` | `encap seg6` | +| Next-hop groups with ids | | `nhg_hash_entry` | NEXTHOP_GROUP_TABLE | kernel nexthop objects | +| Asynchronous RIB → FIB with status | | dataplane, `ROUTE_ENTRY_QUEUED / INSTALLED` | fpmsyncd → orchagent → ASIC_DB | kernel FIB, `switchd` | +| Client route API, NHT | `FbossCtrl` / `FibService`: `addUnicastRoutes(clientId)`, `syncFib`, `registerNextHops`, `streamNextHopStatus` | ZAPI `ROUTE_ADD/DELETE`, `NEXTHOP_REGISTER/UPDATE`, `REDISTRIBUTE_*`, `INTERFACE_*`, SRv6 SID manager | | | +| State store with subscriptions; agents never talk directly | FSDB path pub/sub | | Redis tables | Arista EOS Sysdb (same pattern) | +| LAG with min-links, member forwarding state | `AggregatePort.minimumLinkCount`, `portToFwdState` | | PORTCHANNEL `min_links`, member `status` | bonding `min_links`, `xmit_hash_policy` | +| ECMP / LAG hashing, resilient hashing | `LoadBalancer` (ECMP, AggregatePort) | | | per-switch seed; resilient nexthop groups | +| Local SIDs, micro-SIDs | `MySidFields` (uN, uA, uDT46, uB6), SAI `MY_SID_ENTRY` | locators, `format usid-f3216`, `seg6local` | SRV6_MY_LOCATORS / SRV6_MY_SIDS, `Srv6Orch` | `seg6local` actions and flavors | +| Unnumbered | | | | BGP unnumbered (RFC 8950) | + +## Decision: topology representation + +NetSim owns an interface-centric model in its own state tree. NetGraph +`Network` has no interface, address, LAG or SID concept and costs ~0.27 s +plus networkx, pandas, jsonschema and the compiled Core extension to +import; NetGraph-Core is an immutable int-indexed compute kernel with one +`int64` per edge. NetSim's history records the pain of a hard ngraph +dependency. Both are optional extras (`netsim[ngraph]`): NetGraph for +authoring and interchange, Core as an accelerator and an independent +cross-check for plain-IP placement. SONiC CONFIG_DB JSON is a +dependency-free configuration format. The core package keeps zero +runtime dependencies. + +## Architecture: four layers + +``` +netsim.core / netsim.resources DES engine: clock, heap, Process, resources (existing; + DEFERRED = 8) +netsim.model clock-free: state tree, deltas, handles, derivations, packet actions, converge() +netsim.runtime binds the model to the clock: rounds, kinds, observation, agents, transport +netsim.adapters NetGraph, NetGraph-Core, SONiC CONFIG_DB +``` + +Dependency direction is strict: `core ← model ← runtime ← adapters`. + +``` +netsim/ + core.py + DEFERRED = 8 (base of the reserved settle band); nothing else changes + model/ + addressing.py AddressFamily, MacAddress, EUI-64, address normalization (ints inside the model) + lpm.py PrefixTable (dict per prefix length; swappable) + packets.py EtherType constants, EthernetFrame, IPv4Packet, IPv6Packet, SRH (validated), L4Header, FlowKey, + PacketTemplate, wire size of a header stack + hashing.py LoadBalancer (field sets, seed, HASH_THRESHOLD | RESILIENT), flow_hash, region select, tables + state.py NetworkState, DeviceState, records, PMap (owning, transitively immutable), StateDelta, paths, + canonicalization, generations + interfaces.py enums; interface config and oper records; derive_ethernet / derive_lag (component two-pass) + links.py LinkConfig, LinkOper, physical transfer availability + routing.py Nexthop, Route, RibState (sharded), ResolutionPolicy, resolve_fib(), per-row outcomes, epochs + srv6.py SidStructure, Locator, allocator, LocalSid, compression (§6.2), SegmentList, SrPolicy, validation + forwarding.py Adjacency, NexthopGroup, FibEntry, Fib, NeighborTable, the action program (forward step) + flows.py Demand, DemandSet, TrafficClass, forwarding-equivalence classes, leg walk (SCC), capacity models, + PlacementReport + igp.py RouteSource: per-device Dijkstra, ECMP, locators + contracts.py ClientId, RibClient, PolicyClient, SidClient, NhtClient, agent / source / controller protocols, + AgentContext / GlobalContext, AgentOutput, SessionEvent, SrDbView, op records + derive.py every derivation and the kind order (used by converge() and runtime/pipeline.py) + entities.py Device, Interface, Link handles (generation-checked) and the builder / operations API + network.py Network: state, update(), clock and observation hooks, validation, converge(), place(), trace() + runtime/ + pipeline.py Coordinator: rounds, kinds (COALESCE | DEBOUNCE), StageEvent, ROUND_END, retry, convergence limit + events.py observation queue (SETTLED deliveries with snapshot roots), Stats + timeline.py Timeline: Record(seq, time, round, origin) + flat typed events per delta, bounded delta/root retention, + series from placement arrays, rows()/to_csv() export + simulation.py Simulation: bind, settle(), run_until(), at(), send(), retry(), reset_agent() + agents.py context views, inboxes and outboxes, AGENT and IGP kinds, run_delay, RNG and stats buffering + transport.py LinkChannel, Session (scoped endpoints, incarnations, sequence numbers), liveness, timers + agents/ reference.py (Gate C); openr.py, bgp.py (deferred) + adapters/ sonic.py (no dependencies), ngraph.py, core.py +``` + +## Data model: the state tree + +### Principle + +All simulated state lives in one immutable tree, `NetworkState` (FBOSS +`SwitchState`). Nodes are frozen slotted records; maps are `PMap` (an +owning, unexposed backing dict copied on set; content `__eq__`; structural +sharing at the map level; batch edits go through a private builder that +copies each touched shard once per commit). An update produces a new root +and a `StateDelta(old, new)`; unchanged subtrees are the same objects so +diffs are identity-first. Within a node, **config** (intended) and +**oper** (derived) are separate records (RFC 8342), and every oper record +or client-keyed config entry has exactly one writer. + +**Transitive immutability.** Every committed leaf is transitively +immutable: `PMap` copies or rejects mutable inputs; numeric report data +is committed as `bytes` plus immutable shape and type metadata, and +accessors return a **fresh** read-only `memoryview` (or an immutable +numeric wrapper) per call, so no caller ever receives the object stored +in a snapshot and releasing a returned view cannot invalidate the report +or another reader; never `array` or `list`; agent state, messages and +payloads follow the same contract (frozen records, tuples, `PMap`, +`bytes`), checked in debug mode by a recursive validator; RNG state is an +immutable value committed with the record. Nothing aliases a caller-owned +container. + +**Canonicalization.** A derivation compares its recomputed record with the +old one by value and returns the old object when equal; per-row records +are reused the same way; `version` fields bump only on content change; +`since` fields carry forward when the value is unchanged. `update()` +computes the delta before commit and returns `None` for an empty content +delta, so a run on unchanged inputs commits nothing. Agent state is +compared by identity only (agents return the same object when nothing +changed). + +**Tree equality** for tests is modulo `version` and every `since` field. + +**Handles.** `Device`, `Interface`, `Link` are cached handles keyed by +(kind, key, generation); every entity gets an immutable generation at +creation; identity, equality and hash use (network, kind, key, +generation); every access validates the generation, `exists` is false for +an obsolete generation even if the name was reused; handles carry +`__dict__` for user attributes (as resources do in +`tests/test_invariants.py`). + +**Paths.** Every tree location has a canonical path of typed components +(`("devices", "R1", "interfaces", "eth1", "oper")`) with a reversible +escaped string form; imported names (NetGraph uses `/`) are preserved +without changing path identity; globs are a separate query syntax. +`StateDelta.changed_paths()` is lazy; subscriptions (kinds, agents, +recorder, tooling) match by path prefix through a prefix index. + +### Tree and nodes + +``` +NetworkState(version, devices, links, demands, traffic_classes, placement | None, transport, + allocators: Allocators(next_mac, next_link_index, generations, per-device next_ifindex, SID allocators)) +DeviceState(config, oper: DeviceOper(router_id), interfaces, ribs[af], fibs[af], resolver_epoch[af], + neighbors, nexthop_groups, load_balancers, srv6_config, srv6_sids, srv6_policies, + agents: PMap[name, AgentNode(config, state, srdb_view)], nht) +InterfaceNode = LoopbackNode | EthernetNode | PortChannelNode identity (index, generation), config, oper +LinkNode(id, index, generation, a, b, config: LinkConfig, oper: LinkOper(state, since)) +StateDelta sections: links, config(device), interface_oper(device), rib_rows(device, af), neighbors(device), + srv6_sids(device), srv6_policies(device), nexthop_groups(device), fibs(device, af), nht(device), + agents(device), transport, demands, placement +``` + +| Node | Config (intended) | Oper (derived) | Writer | +|---|---|---|---| +| Device | enabled (+ since), seed, fib_delay, resolution policy, router_id, srv6 source address, srv6 hop limit, srdb_source | router_id | operations / L3 | +| Loopback | admin (+ since), description, ipv4, ipv6 | oper, reason, since | L3 | +| Ethernet | admin (+ since), description, mtu, speed, metric, ipv4, ipv6, unnumbered, forwarding per AF, aggregate_id, carrier_delay_up / down | oper, reason, since, carrier_raw, carrier_raw_since | CARRIER | +| PortChannel | admin, description, mtu, metric, ipv4, ipv6, unnumbered, forwarding per AF, min_links, member_delay_up / down | oper, reason, since, members: PMap[name, MemberOper(candidate, candidate_since, active)], bandwidth | LAG | +| Link | capacity override, delay, risk_groups | state (UP / FAILED), since | operations (fault injection) | +| RibState (per AF) | rows by client key (sharded by prefix length and a radix sub-shard) | | clients through the one door | +| Resolver input epoch (per AF) | | advanced atomically by `Network`'s commit machinery with every relevant input commit | commit machinery | +| Resolver outcome (per AF) | | processed epoch (the input epoch consumed), per-row outcomes | FIB | +| NeighborTable | | (interface, scope, address) → mac | L3 | +| Srv6Sids | locators, local SID config, SID requests | allocated SIDs with adjacency state, My-SID view | L3 | +| Srv6Policies | policies and candidate paths (one owner per policy in v1), steering entries | policy states | FIB | +| NexthopGroupTable, Fib (per AF) | | groups; entries, version | FIB | +| LoadBalancers | ecmp, lag: field sets, seed, mode, buckets | | operations | +| Agent | agent config | opaque state, srdb_view | AGENT | +| NHT | registrations by (client, af, address, scope, options) | results | FIB | +| Transport | sessions opened by agents | channel liveness, session connection state | TRANSPORT | +| Demands, TrafficClasses | rows | | operations | +| PlacementReport | | read-only arrays per directed edge, per-demand results, caches, version | PLACEMENT | + +Conventions: membership on the member (`aggregate_id`); addresses only on +L3 interfaces; `index` (ifIndex, creation order) is the canonical sort +key and comes from the tree's allocators so a failed update leaves no +gaps; loads live only in the `PlacementReport`; a route is one frozen +row; names may not contain `|`, `:`, `--` or `/` unless imported (then +escaped). + +### Addressing and LPM + +`MacAddress(int)` (validated 48-bit, parse/format, multicast and local +bits, `eui64()`, `link_local()`). IP addresses are `ipaddress` objects at +the API boundary only; **no `ipaddress` object is used inside +derivations, the forward step or placement**, and `hash()` of one is +never used. `PrefixTable[T]` in `lpm.py`: dict per prefix length, +descending tuple of populated lengths, per-instance masks; O(k) probes; +brute-force reference test. + +## Event model + +### `Network.update(fn, origin) -> StateDelta | None` + +`fn` is a pure function from the current root to a candidate root. +`update()` validates and canonicalizes the candidate, computes the delta +and timestamps (`Network.clock()`, default `lambda: 0`, the runtime +installs `env.now`) **before** commit, returns `None` for an empty content +delta, commits once (bumps `version`), then hands the delta to the +observation queue. Internal notification preparation completes before +publication; observer exceptions after commit are reported as observer +failures and never roll back the committed root. On any exception before +commit the root and the allocators are unchanged and the exception +propagates. A nested `update()` from inside a derivation or observer +raises `RuntimeError`; observers may only mark kinds dirty, append to the +log and enqueue deliveries. + +### Kinds and modes + +One `StageKind` per band, not one stage per entity. A kind keeps pending +work per entity and runs the entities due at a time in sorted order in +one `update()`, so each kind emits one delta per round. + +- `COALESCE` (LAG, L3, IGP, AGENT, FIB, TRANSPORT, PLACEMENT): one pending + deadline per entity. A cause at `t` with delay `d` requests `t + d`; the + entity keeps the **earliest** requested pending deadline: a later + request joins the pending run, an earlier request advances it (the + implemented rule); the deadline clears when a run claims the work, and + causes arriving during or after the claim schedule the next run. Entities sharing a deadline share one `StageEvent`. +- `DEBOUNCE` (CARRIER, LAG member timers): absolute deadlines. On a raw + input change the kind records `raw` and `raw_since = now` (one observed + transition per operations batch), picks `delay(new raw value)` and + schedules at `scheduled_at = max(now, raw_since + delay)`; a run applies + the change only if `now >= scheduled_at` recomputed from the current + tree, else reschedules at the new deadline. Same-timestamp raw changes + collapsed by one batch are one transition. + +Durations are validated finite and non-negative; event times finite; +asynchronous message and timer deadlines must satisfy `scheduled_at > +now` after conversion to the engine's float time (a delay that does not +advance the float clock is rejected). Tests use one supported time +quantum (integer or dyadic seconds). + +`StageEvent` is an `Event` subclass with `__slots__` scheduled through +`env.schedule(event, priority, delay)`, the pattern `run(until=t)` uses; +`Timeout` is untouched; `tests/test_invariants.py` scans `netsim.runtime` +for Event subclasses. + +### Rounds (the coordinator owns ordering; priorities only order within a round) + +``` +NORMAL = 1 operations, user processes, deliveries, timer firings +CARRIER = DEFERRED+0 LAG = +1 L3 = +2 IGP = +3 AGENT = +4 FIB = +5 TRANSPORT = +6 PLACEMENT = +7 +ROUND_END = DEFERRED+8 coordinator sentinel +SETTLED = DEFERRED+9 observation deliveries +``` + +The heap orders events by (time, priority, eid); it does not know +dependencies. The runtime **coordinator** turns that into rounds: + +1. A round at time `t` is the set of kind runs scheduled at `t` before + its `ROUND_END` sentinel. Kinds run in priority order. **Exactly one + round is active at a timestamp.** `StageEvent`s and `ROUND_END` carry + a round generation; events of a stale generation do nothing. +2. Each entity's pending **deadline is computed first** (COALESCE or + DEBOUNCE rule). Work due after `now` belongs to that future timestamp + and is not part of any current round. For work due now: a forward + invalidation (a kind of higher priority than the running one) joins + the current round; a backward or already-executed invalidation (lower + or equal priority, or a kind that already ran in this round: AGENT SID + requests → L3, FIB → AGENT via NHT and policy states, TRANSPORT → + AGENT, PLACEMENT → controllers) joins the **successor** round at `t`. + FIB → AGENT with `run_delay = 0.001` is therefore work at `t + 0.001`, + not a round at `t`. +3. `ROUND_END` finishes the round: if successor work exists the + coordinator schedules those kinds and a new `ROUND_END` at `t` with the + next generation; otherwise it releases the observation deliveries for + `t` at `SETTLED`. +4. Every round, including SID-allocation rounds, counts against + `Simulation.max_rounds_per_timestamp` (default 10 000); exceeding it + raises `ConvergenceError(time, rounds, last_origins)`. With positive + delays a timestamp normally needs one round. + +Non-band events (NORMAL, URGENT) can interleave with an active round: an +operation, a user process resumed by a condition, a resource completion +or a process termination may schedule NORMAL or URGENT events at `now`. +A state change from such an event uses the same rule as any other cause +(forward work joins the active round, backward work its successor); it +never opens a second concurrent round. The band itself never schedules a +NORMAL event for the current time. The log records `(time, round, origin, +delta)`. + +### Dependency and invalidation matrix + +The runtime keeps an **explicit invalidation matrix** from input paths to +owning kinds, and every edge has an insertion, withdrawal, replacement and +recovery test. Until an incremental rule is proven, invalidation is +conservative: every SR consumer on any SR-DB change; every local AF on any +resolver-input change; every session whose recorded dependency set +intersects the change. Dependency records track **lookup queries, +including failed lookups**, not only returned prefixes, so a newly +inserted more-specific route or a formerly absent SID re-triggers. + +| Kind (entity) | Inputs | +|---|---| +| CARRIER (Ethernet e) | link state, link replacement (old and new endpoints); admin and enabled of e's device and of the peer's device; e.config; peer config | +| LAG (PortChannel p) | oper of p's members; p.config; oper of the members' peers; config of peer bundles; link capacity of members | +| L3 (Device d) | oper and config of d's interfaces; peer interface config (addresses, MAC, forwarding enablement) and endpoint replacement where they affect scoped neighbors; d's config (enabled, router_id, srv6); SID requests | +| IGP (Device d, AF) | interface oper, metric, addresses, unnumbered and forwarding flags of every device's L3 interfaces; device enabled; links; locators | +| AGENT (Device d, agent) | the agent's subscribed local paths; its inbox (deliveries, timers, session events) | +| FIB (Device d, AF) | rib rows of d (both AFs); interface oper and forwarding flags of d; neighbors; NHT registrations; load balancer config; **every config value captured in its compiled programs** (SRv6 source address, hop limit, encapsulation behavior); srv6_sids, policy config, `srdb_view` and srdb_source; SR-DB inputs recorded in `srdb_deps[d]` (conservative: all SR-DB changes) | +| TRANSPORT (session s) | interface oper and link config of link channels; **physical link state and endpoint availability**, fibs, groups, policy states, steering, interface oper and link delay of every device in `s.deps` (devices visited by the previous run's forward steps, both directions) | + +Dependencies are invalidated **directly** from the changed input, even +when an intermediate derivation returns an unchanged record. +| PLACEMENT (class c) | the class cache's recorded dependencies: consulted FIBs, groups, interfaces, links (physical state and capacity), SID and policy actions, steering, load balancers, demands, traffic classes; the previous PlacementReport | + +### Failures and retry + +A derivation exception propagates out of `env.run()`; the engine has +already popped the event and cleared its callbacks. The failed kind +retains its captured work and marks itself retryable without +rescheduling; `Simulation.retry()` is explicit (called after the +configuration is corrected; `settle()` does not call it) and schedules a +fresh `StageEvent` for each retryable kind. Work is claimed before the run +and consumed by publication: a failure **before** publication (the +derivation raised) keeps the claimed work retryable; a failure **after** +publication (an observer raised; `Network.update` re-raises it marked as +published, see `published_failure()`) leaves the committed root in place and the work consumed, +so a retry never re-executes a published run. A retried agent run consumes +the same captured inbox prefix once. + +### Observation and quiescence + +`EventBus.next()` / `wait_for()` deliveries are **separate events per +subscriber**, released at `SETTLED` after the last round of the +timestamp. Each delivery carries its triggering delta(s), whose +historical roots stay as committed (a CARRIER delta's root is pre-LAG), +plus `settled_root`, captured immediately before that delivery after all +currently due rounds finished. Internal commit hooks cannot call +`update()`; public SETTLED deliveries run outside that guard and may +write, in which case the write starts new round work before the next +delivery is processed. Multiple callbacks on one engine event share no +barrier (the engine runs them back to back); the observation queue is +therefore the only settled view, and tests distinguish cause roots from +`settled_root`. `Timeline` retains the last root per timestamp for the +most recent `keep_roots` timestamps (bounded by default) plus a bounded +deque of `(time, round, origin, delta)`; `snapshot_at(t)`, +`series(path)`, `utilization_series(edge)`. A `Stats` store outside the +tree keeps timestamped counters. + +`Simulation.settle(max_steps=100_000)` steps while `env.peek() == +env.now` (the cap limits dispatched events, not instructions inside a +callback). `run_until(t)` = `env.run(until=t)` if `t > now`, then +`settle()`; `env.run(until=t)` alone stops at a newly inserted URGENT +event, so previously scheduled URGENT events at `t` with smaller eids +still run and NORMAL ones stay pending. `at(t, fn)` schedules an +operation. Binding: `Simulation(env, net)` raises if already bound, +installs the clock, runs a synchronous full derivation with delays +ignored (timeline root 0). `Network.converge()` (clock-free) runs +`model/derive.py` kinds in order ignoring delays, RouteSources included, +DeviceAgents and Controllers excluded, repeating until clean with an +iteration cap. + +Tree snapshots support inspection and state-delta replay only. Restarting +execution would need a runtime checkpoint (time, scheduler ids, pending +work, timer and session generations, inboxes and outboxes, dirty rounds, +RNG and stats); that is deferred, and snapshots are not advertised as +protocol replay. + +## Interfaces, links and physical failure + +Enums are API-boundary values; hot-path oper fields are small ints. +`OperState` uses RFC 2863 values (UP, DOWN, LOWER_LAYER_DOWN, NOT_PRESENT). + +Rules: + +- Loopback: UP iff admin UP and device enabled. +- Ethernet: admin DOWN or device disabled → DOWN immediately; no link → + NOT_PRESENT; raw carrier = link UP and peer admin UP and peer enabled; + effective carrier after debounce; UP iff effective carrier, else DOWN + with CARRIER or PEER_ADMIN. +- PortChannel, computed **per connected component of bundles** in three + passes from one snapshot of interface oper and config: (1) candidates + and candidate stability timestamps for every bundle in the component + (member oper UP, peer port bundled); (2) every bundle's `min_links_ok` + (admin UP, enabled, `len(candidates) >= min_links`); (3) active sets + (candidates whose peer bundle has `min_links_ok` and whose member delay + elapsed), oper (DOWN if admin DOWN or disabled; UP iff `len(active) >= + min_links`; else LOWER_LAYER_DOWN with MIN_LINKS / PARTNER / MEMBERS) and + bandwidth (sum of active members' link capacities; recomputed on + capacity changes). The component commits together, so bundle sort order + cannot change partner results. +- `l3_usable(af)` = oper UP, not bundled, and forwarding enabled for that + AF (explicit per-AF flag, default: enabled iff the interface has a + global address of that AF or is `unnumbered`; an IPv4-only link + therefore does not forward IPv6 unless enabled explicitly, which the + Gate-B fixture does). +- A PortChannel whose members terminate on different peer bundles is + **rejected** at configuration (or, if imported, kept as invalid + configuration with no candidates and reason `MULTIPLE_PARTNERS`); an + active L3 bundle always has one unambiguous peer bundle. + +**Physical transfer availability** is separate from observed oper state. +A directed edge can carry traffic iff the link is UP and both endpoint +interfaces are physically usable (admin UP, device enabled, present). +Every transmission, in the forward step, in FLUID legs and in timed +messages, checks this and rejects with `LINK_DOWN` or `RX_DOWN` while the +attempted transmission is still charged as `offered`. Routing may stay +stale through carrier delay and FIB delay; that transient is what the +utilization series shows. + +`Link`: capacity = override or min of speeds, `delay`, `fail()` / +`restore()`. Validation: unique names; one link per Ethernet on different +devices; members are Ethernets on the same device, unbundled, without +addresses; no duplicate or overlapping addresses per device; MTU +≥ 1280 with IPv6; positive metrics, speeds and capacities; `min_links ≥ +1`; effective member and bundle MTUs consistent; unique MACs; single peer +bundle per PortChannel; SRv6 rules. + +## Routing + +### Clients and rows + +``` +ClientId(name, instance=0) identity +ClientProfile(client, distance, protocol_origin: int (RFC 9256 §2.6: CLI 30, BGP SR-TE 20, PCEP 10), originator) +Route(prefix, source: ClientId, distance, metric=0, nexthops, distinguisher=(), tag=None, attrs=(), table=0) +Nexthop(interface=None, address=None, scope=None, special=None, srv6=None, policy=None, weight=1, onlink=False) + interface only | interface + address (link-local requires scope = interface) | address only (recursive, resolved + in the RIB of the address's family) | special BLACKHOLE | UNREACHABLE | PROHIBIT | RECEIVE | SRV6_LOCAL(behavior) + | srv6 = Srv6Encap(segments (symbolic or literal), behavior H_ENCAPS_RED | H_ENCAPS, source) | policy = PolicyRef +``` + +`ClientId(name="static", instance=0)` with profile distance 1 is +reserved for operations (`Device.add_route(prefix, nexthops=[...], +distance=1, distinguisher=())` is sugar over one row with possibly +several next-hops); `ClientId(name="igp", instance=0)` with distance 110 +is the oracle; agents register their own identity and profile. Examples +in this document that write `("static", 1)` or `("igp", 110)` name +(identity, distance). `RibClient.add_routes` upserts by +(prefix, source, distinguisher) and is a no-op for an identical row; +`delete_routes(keys)` ignores missing keys; `sync(rows)` replaces all rows +of the client in the AF in one update reusing unchanged rows; rows are +validated before commit (family, source, next-hop shape); an output holds +per AF either `sync` or delete-then-add. Duplicate client ids are +rejected at registration; ties between different sources at equal +(distance, metric) break by `ClientId` ascending. + +### Route status and outcomes + +Each (device, AF) has a **resolver input epoch**, advanced by the commit +machinery atomically with any FIB input commit (rows, interface oper, +neighbors, registrations, SR inputs), and a **processed epoch** written +by the FIB kind together with the outcomes, recording the input epoch it +consumed; writing FIB outputs never re-invalidates its inputs. A row is `PENDING` iff the +input epoch is ahead of the processed epoch (an unchanged static row is +pending when an interface change queues work); otherwise its immutable +**per-row outcome** from the last run says `INSTALLED` (with the group it +contributed to) or `NOT_INSTALLED(reason)`, and `SELECTED` means it is in +the group `best_groups()` would choose. Programming state and current +selection are exposed separately; `INSTALLED` does not imply live egress. +Owner notifications cover row deletion and interface- or policy-driven +outcome changes as well as FIB entry changes. + +### Selection and resolution (`resolve_fib`) + +Best-first groups of equal (distance, metric, source) with unioned +next-hops; rows from different sources never merge. Within a group the +action precedence is a **simulator policy**, not an RFC rule: a +homogeneous special action overrides forwarding legs; different special +actions in one equal-ranked group are rejected with `AMBIGUOUS_ACTION`. +The first group with a resolvable leg (or all legs under +`ALL_OR_NOTHING`) is active; otherwise `NOT_INSTALLED` and LPM falls +through to a covering prefix (`lpm_fallthrough=True`, kernel and zebra +behaviour; fallback across candidates and across prefixes are separate +documented steps). + +``` +resolve_nexthop(owner, nh): + interface only / interface + address: require l3_usable(egress AF) first (cross-AF next hops also require the + relevant neighbor-discovery enablement); then require a usable scoped adjacency: peer MAC on the p2p link + or bundle for interface-only (connected-prefix recursion keeps the final destination), the scoped neighbor + entry for that address otherwise; onlink bypasses prefix-membership validation only and never fabricates a + MAC; failure is an unresolved leg before candidate selection + address only: LPM in the RIB of the address's family excluding the gray stack (never through itself); + RECEIVE -> unresolvable (never a recursive forwarding or drop result); DROP_* -> unresolvable, or a drop leg + only when resolve_via_drop; SRV6_LOCAL(B6) -> resolve_policy; FORWARD -> child legs + srv6 encap: compile the action program (below); first-entry feasibility uses the same local-action dispatch as + transit (a matched local SID returns its continuation), not the address-only cases + policy: valid lists of the active path, each an encap leg weighted by list weight +weights: child shares are normalized within each parent, multiplied by the parent's weight, equivalent legs + merged, then truncated to max_ecmp_paths in a stable order (interface index, next-hop, encap) and renormalized +outcome: FibEntry(prefix, action program | group id, contributing row keys, rejected rows with reasons, depends_on: + queried prefixes incl. failed lookups, interfaces, policies, SIDs) +``` + +`ResolutionPolicy` defaults: `resolve_via_default=False`, +`unresolved=PARTIAL`, `max_recursion_depth=8`, `max_ecmp_paths=64`, +`lpm_fallthrough=True`, `resolve_via_drop=False`, `validate_all_sids=True`. + +### Neighbors, AF enablement, unnumbered (RFC 8950) + +The L3 kind derives neighbor entries for configured peer addresses on the +attached p2p link or bundle (numbered) and for the peer's EUI-64 +link-local when IPv6 forwarding is enabled (bundle MAC for bundles). +Entries are keyed (interface, scope, address). IPv4 prefixes may use +interface-only next-hops or interface + scoped IPv6 link-local next-hops; +the frame uses EtherType 0x0800 with the MAC from the IPv6 entry. +Numbered, unnumbered and cross-AF paths are tested separately. + +### `igp.py` (oracle RouteSource) + +Per device and AF: Dijkstra over interfaces with `l3_usable(af)` and +their metrics, ECMP successors, destinations = every device's loopback and +interface prefixes plus locator prefixes; rows `("igp", 110)` via `sync`. +Runs in the IGP band for dirty AFs; NetGraph-Core `spf_to` accelerates it +when installed. This makes unnumbered NetGraph imports routable in Gate A. + +## Forwarding as an action program (`model/forwarding.py`) + +Every forwarding decision compiles to an ordered **action program** over +a packet with stage metadata (`ORIGINATED | TRANSIT | AFTER_ENCAP | +AFTER_DECAP`): `L2_CHECK`, `STEER`, `LOOKUP`, `ENCAP(list)`, +`LOCAL_SID(behavior)`, `TTL`, `SELECT_GROUP`, `SELECT_MEMBER`, `MTU`, +`TRANSMIT(edge)`, `DELIVER`, `DROP(reason)`. Probes, the timed driver, +HASH placement and the resolver's compiled entries all execute the same +program interpreter, and compiled entries must be trace-equivalent to the +uncompiled step (a property test). + +Order for a packet arriving on an interface: + +1. `L2_CHECK`: ingress oper UP else `IFACE_DOWN`; `dst MAC ∈ {ingress MAC, + bundle MAC, multicast}` else `MAC_MISMATCH`; EtherType else + `UNSUPPORTED_ETHERTYPE`; tagged → `VLAN_UNSUPPORTED`. +2. `STEER` (packets entering the forwarding path and inner packets after + decapsulation; never after a shift or End re-lookup): precedence is + the explicit demand override (consumed at the demand's initial + ingress, never re-applied), then the matching ingress steering rule + (ordered by longest dst prefix, exact DSCP before wildcard, exact sport + before wildcard, then stable rule id), then ordinary FIB. A + decapsulating node may apply its own table once for that ingress + context. A match yields `ENCAP`. ENCAP → DECAP → ENCAP cycles are + caught by a per-packet transition budget over (device, action state), + independent of the nested-header check. +3. `LOOKUP`: LPM; RECEIVE → `DELIVER`; DROP_* → `DROP`; `SRV6_LOCAL` → + `LOCAL_SID` (the behaviour performs its own hop-limit checks and + decrements; no generic `TTL` afterwards); miss → `NO_ROUTE`; a + `PolicyRef` or encap entry executes wherever that FIB entry is used, + on any device. +4. `TTL`: for TRANSIT packets only, ≤ 1 → `TTL_EXPIRED` else −1; + ORIGINATED packets are not decremented; `ENCAP` performs the RFC 8986 + §5.1 S05 inner decrement exactly once for transit packets and + initializes the outer hop limit from `srv6_hop_limit` without a headend + transit decrement. +5. `ENCAP(list)`: choose the list (weighted hash on the inner flow key), + build the outer header (flow label from the inner key, DSCP copied), + then dispatch the first outer entry through the **same local-action + dispatch as transit**. `LOCAL_SID` returns an explicit continuation: + `RELOOKUP` (End / uN: look the new DA up on this device), + `CROSS_CONNECT(adjacency)` (End.X / uA: after shift and decrement, + proceed through the bound adjacency to egress validation, member + selection, MTU and transmission, with **no further local DA lookup**), + `DECAP_LOOKUP`, `DELIVER` or `DROP`; `ENCAP` follows that continuation. + A compiled adjacency keeps this whole header and TTL transformation + and must be trace-equivalent to the interpreted run. +6. `SELECT_GROUP`: hash on the outermost header as it exists now (L4 + fields only for a transport next header); egress `l3_usable(af)` else + `EGRESS_DOWN`; missing neighbor → `ADJ_UNRESOLVED`. With the device + setting `fast_failover` (data-plane pruning, as ASIC fast failover does) + legs whose port has no raw carrier or is unusable are skipped before the + hash and the flow re-hashes over the live legs; off, the flow drops until + the control plane reprograms the group (`fib_delay`). FLUID placement + applies the same rule. `SELECT_MEMBER` for + bundles by the LAG hash. Cross-connects skip `SELECT_GROUP` but not the + rest. +7. `MTU`: transmitted length from the actual header stack (outer 40, SRH + 8 + 16·entries only when present, inner) against the egress interface + or bundle MTU; IPv6 never fragments; IPv4 the same in v1. +8. `TRANSMIT(edge)`: physical transfer availability (above) else + `LINK_DOWN` / `RX_DOWN`; re-encapsulate (`src` = egress MAC, `dst` = + neighbor MAC) and hand to the peer. + +SRv6 behaviours (RFC 8986 pseudocode with the RFC 9800 §4.1.1 insertion), +each validating the SRH shape and bounds first (negative or oversized +`segments_left`, `last_entry` inconsistent with the entry count, reduced +SRH with SL = LE + 1 allowed per RFC 8754 §4.3.1) and returning modeled +drops (`SRH_MALFORMED`): + +- NEXT_CSID pre-step (uN, uA; uB6 post-Gate-C), before the SL == 0 test whether or not + an SRH is present: `arg` = DA bits after `lbl + lnfl` of the matched SID; + `arg != 0` → hop limit ≤ 1 → `TTL_EXPIRED`; shift; hop limit −1; SL + unchanged; uN re-looks up on the same device, uA cross-connects, uB6 + continues to its encapsulation; `arg == 0` → classic processing. +- END: SL == 0 or no SRH → USP pops the SRH if flavored, then upper layer: + USD → decapsulate and re-lookup the inner packet, else + `UPPER_LAYER_NOT_ALLOWED`; otherwise hop limit check, −1, SL −1, DA = + entries[SL], PSP pops the SRH when SL reaches 0. +- END_X: as END, then cross-connect (USD: decapsulate and cross-connect). +- END_DT46 (Gate B): SL != 0 → `SRH_SL_NONZERO`; decapsulate any inner + family and re-look up in the default table. END_DT4 / DT6 and + END_DX4 / DX6 (post-Gate-C): the same shape with family checks (DT4 + rejects an IPv6 payload) and DX cross-connecting; configuring them in + Gate B is rejected at validation. +- END_B6_ENCAPS / uB6 (post-Gate-C): pre-step if uB6, else SL == 0 or no + SRH → `UPPER_LAYER_NOT_ALLOWED`, hop limit check, −1, SL −1, DA = + entries[SL]; then encapsulate with a valid list of the bound policy and + look up the new outer DA. In Gate B a packet whose active SID is a B6 + behaviour drops with `SRV6_UNSUPPORTED` and configuring a B6 SID is + rejected. Nested encapsulation of an already encapsulated packet is + rejected (`NESTED_ENCAP_UNSUPPORTED`); one hop-limit budget per header. +- Header chain: an outer `IPv6Packet` carries `next_header` (43 for a + routing header, else the transport or inner-IP protocol number) and the + `SRH` record carries `next_header` for what follows it; the typed payload + must agree with those values or the packet is `SRH_MALFORMED`. Hash + fields come from the outermost IP header plus the transport header that + the chain reaches (through the SRH); an encapsulated inner packet + contributes only through the flow label the headend derived from it. +- Hop limit is decremented once per SID *behaviour that decrements* (End, + End.X, pre-step); decapsulation paths do not decrement. + +Drop reasons: IFACE_DOWN, MAC_MISMATCH, UNSUPPORTED_ETHERTYPE, +VLAN_UNSUPPORTED, NO_ROUTE, DROP_BLACKHOLE, DROP_UNREACHABLE, +DROP_PROHIBIT, TTL_EXPIRED, EGRESS_DOWN, ADJ_UNRESOLVED, MTU_EXCEEDED, +LINK_DOWN, RX_DOWN, SID_UNKNOWN, SRH_MALFORMED, SRH_SL_NONZERO, +UPPER_LAYER_NOT_ALLOWED, NESTED_ENCAP_UNSUPPORTED, POLICY_DOWN, LOOP, +CONGESTION. + +`Network.trace(device, packet, max_hops=64)` is synchronous; +`Simulation.send()` yields `link.delay` per hop. + +## SRv6 (`model/srv6.py`) + +Standards: RFC 8402, 8754, 8986, 9256, 9602, 9800. + +### SID structure, locators, allocation + +`SidStructure(lbl, lnl, fl, al)`; C-SID = the `lnl + fl` bits after the +block; a container = block, then C-SID bits back to back, then zeros +(C-SID 0 = End-of-Container). + +| Format | SID | Structure | Installed prefix length | +|---|---|---|---| +| UNCOMPRESSED | LOC:FUNCT:ARG | e.g. (48, 16, 16, 48) | lbl + lnl + fl (convention) | +| F3216 bare | uN = block ‖ node id (GIB) | (32, 16, 0, 80) | /48 | +| F3216 bare | uA, uB6 = block ‖ function (LIB) | (32, 0, 16, 80); WLIB (32, 0, 32, 64) | /48; /64 | +| F3216 bare | uDT4/6/46, uDX4/6 (terminal) | (32, 0, 16, 0) | /48 | +| F3216 composite | block ‖ node ‖ function (FRR / SONiC form) | (32, 16, 16, 64) | /64 | + +Bare and composite forms are **distinct SIDs with distinct forwarding** +(a composite SID reaches its node through the routed /48 and then executes +locally; a bare local C-SID executes wherever the packet is); both are +supported natively and preserved literally by adapters. Default ranges +(Cisco/FRR): GIB 0x0001–0xDFFF, LIB 0xE000–0xFFF6, WLIB 0xFFF7–0xFFFF. +Default block `5f00::/32` inside RFC 9602's `5f00::/16`. The default GIB +holds 57,343 node identifiers per block before reservations; that is a +property of the configured block, not a network-size limit: larger +networks model several blocks (a multi-block allocation, exhaustion and +collision test is part of Gate B), and not every simulated device needs +a global SID. + +Allocation is a **persistent allocator in the root** with reserved, +disjoint sub-ranges per purpose (automatic adjacency SIDs, terminal SIDs, +BSIDs, client requests) and non-zero node ids; allocation fails +atomically on collision or exhaustion; allocated ids persist across oper +changes and until explicit deletion of the interface or device (a +re-created interface gets a new id); `uA function = adjacency sub-range +base + interface index` while it fits, else the allocator. A `srv6-local` +RIB row is installed for a SID only while it is usable (uA: interface +oper UP); the SID stays in the SR-DB with `adjacency_up=False`. With +`drop_unknown_local=True` (default for F3216) the LIB and WLIB ranges are +covered by `srv6-local` UNREACHABLE rows computed as an **exact prefix +cover of the configured ranges** (`B:E000::/35` for the defaults), so an +unknown local C-SID drops as `SID_UNKNOWN` without shadowing summaries. + +Validation per block: gib, lib and wlib pairwise disjoint; node ids +unique and in gib; local functions in lib ∪ wlib; no device's srv6-local +prefix equals or covers another device's uN prefix; classic locators +disjoint; a locator block never covers a configured interface or loopback +address; exhaustion and rollback are tested. + +### Compression and encapsulation (RFC 9800 §6.2, RFC 8986 §5) + +RFC 9800 §6.2 presents S01–S16 "for illustrative purposes" and allows any +encoding of equal or shorter length; NetSim adopts S01–S16 as its +**deterministic encoding contract** and verifies it with independently +hand-checked vectors (there is no official golden suite). `compress(list, +sr_db)` implements §6.2 S01–S16: validate each flavored +SID's structure (invalid or unknown structure is not compressible); +initialize a container from its first compressible SID with remaining +capacity = that SID's `al` (80 bits for F3216); append further +compressible SIDs (NEXT_CSID flavor, Argument 0, same Locator-Block) while +`lnfl` fits; then admit at most one following SID with a known structure, +the same block and `lnl + fl + al ≤ remaining` (a terminal uDT46 with +`al = 0` fits), copying its Locator-Node, Function and Argument bits and +consuming it; push and continue. Tests cover mixed widths and blocks, +non-zero arguments, full containers and the terminal fit boundary. + +`SRH(entries, segments_left, last_entry, flags, tag)` in RFC 8754 order; +Segments Left counts entries. Encapsulation: outer `IPv6Packet` (`src` = +SRv6 source or loopback, `dst` = first entry, hop limit `srv6_hop_limit`, +DSCP copied, flow label from the inner key); `H_ENCAPS`: SRH with all n +entries reversed, SL = LE = n − 1; `H_ENCAPS_RED` (default): entries 2..n +reversed, LE = n − 2, SL = n − 1. RFC 8986 §5.2 says the SRH *may* be +omitted when the policy has one segment and no flag, tag or TLV is +needed; NetSim's policy: omitted exactly then (n = 1, flags = 0, tag = 0, +no TLVs), else pushed. Reduced SRH validation accepts SL = LE + 1 (RFC +8754 §4.3.1.1). Transit inner TTL: H.Encaps decrements the inner hop +limit or TTL once for transit packets (RFC 8986 §5.1 S05, RFC 2473) and +never for packets originated on the headend. + +### SR Policies (RFC 9256), ownership, validation, steering + +``` +Segment = literal SID | AdjSeg(device, interface) | NodeSeg(device) | TermSeg(device, behavior) +SegmentList(segments, weight=1, name=None) +CandidatePath(preference=100, segment_lists, name, protocol_origin, originator, discriminator=0) +SrPolicy(owner: ClientId, color, endpoint, name, bsid, candidate_paths, fallback=IGP | DROP) +PolicyState(active_path | None, valid_lists: tuple[(list, wire entries)], status, reasons) +``` + +In v1 a policy (color, endpoint) has **one owner** and one BSID; +conflicting writes are rejected; candidate keys (origin, originator, +discriminator) map to exactly one registered client. Active path: highest +preference among valid paths; ties by higher protocol origin, then +(`prefer_installed`, default **false** for history-independent +convergence) already active, then lower originator, then higher +discriminator. W-ECMP across lists by weight. + +Validation has three separate layers: (1) **basic validity** (RFC 9256 +§5.1, which declares a segment list invalid when it is empty, its weight +is 0, it mixes SR-MPLS and SRv6, the headend cannot resolve the *first* +SID to outgoing interfaces and next hops, any *non-first symbolic* SID +(types C–K; in NetSim `AdjSeg`, `NodeSeg`, `TermSeg`) cannot be resolved +to a SID, or explicitly requested verification fails): non-empty, weight +> 0, no SR-MPLS mix, every symbolic segment resolves to an owner; literal +IPv6 SIDs beyond the first are not checked by this layer, and unknown or +invalid compression metadata keeps a literal SID literal and uncompressed +without invalidating it (ownership, behaviour or strict-profile checks +may still reject it); (2) **headend first-entry resolution** by compiling +the action program with the local-action dispatch, including the +headend's own local SIDs; (3) the **strict simulator profile** +(`validate_all_sids`, default on; deliberately stronger than the RFC's +rule, which checks only the first SID's path and the symbolic SIDs' +resolution): terminal behaviour last (`TERMINAL_NOT_LAST`), last SID +decapsulates or carries USD (`ENDPOINT_NO_DECAP`), **terminal SID owner +equals the policy endpoint** (`ENDPOINT_MISMATCH`), and an **end-to-end +walk**: `RIB_RESOLVABLE` means local recursive egress feasibility only; +end-to-end validation follows the selected underlay egresses **hop by +hop** through each visited device's RIB-derived forwarding until the SID +owner is reached (a preferred blackhole on a transit device is not +reachability), with loop detection, an explicit partial-ECMP rule (a list +is valid only if every ECMP branch reaches the owner; partial reachability +is reported), and all queries and failures recorded as dependencies. A +global or composite SID first reaches its owner before its behaviour +updates the current node; a bare local C-SID is resolved in the current +node's scope and must be owned there with its adjacency UP; uA/End.X move +to the adjacency peer (a bundle's peer bundle); uB6 (post-Gate-C) +requires the referenced policy to have a valid list by the same walk and +moves to its endpoint. With `srdb_source=("agent", name)` layer 3 validates only that +agent's published `srdb_view` claims and never upgrades local +resolvability to an oracle proof. **Policy status exposes five things +separately:** basic validity (layer 1), strict-profile validity (layer +3), the selected candidate path, the programmed forwarding version +(PENDING while `fib_delay` has not elapsed: the old program still +forwards, and a path that became invalid meanwhile keeps forwarding +through it), and observed delivery from placement or probes. Actual +delivery is always determined by installed forwarding and physical state; +a drop during the programming interval stays visible in the timeline. + +Steering forms: BSID (a next-hop resolving to the BSID), per-destination +(`Nexthop(policy=PolicyRef)`; executed wherever that FIB entry is used), +per-flow through the ingress `SteeringTable`, and `Demand.steer`. +Precedence and matching as in the action program. `fallback=DROP` +installs an explicit discard when the policy is DOWN; `fallback=IGP` +performs the lookup **excluding the triggering policy or steering +action**, so it cannot re-select the same policy-bearing route. After +decapsulation the inner packet is steered once; repeated encapsulation is +guarded (`NESTED_ENCAP_UNSUPPORTED`). + +## Hashing (`model/hashing.py`) + +`LoadBalancer(id: ECMP | AGGREGATE_PORT, ipv4_fields, ipv6_fields, +transport_fields, seed, mode: HASH_THRESHOLD | RESILIENT, buckets=256)` +with LAG presets. `flow_hash` is keyed BLAKE2b from a per-Network +prototype over the selected fields of the **outermost header as it exists +at that selection**; H.Encaps sets `outer.flow_label = flow_hash(inner +key, domain seed) & 0xFFFFF` (RFC 6437/6438). + +- `HASH_THRESHOLD` (RFC 2992): region selection over the 64-bit hash with + documented integer rounding of region boundaries; FLUID uses the ideal + rational shares, so the two differ by the rounding only. +- `RESILIENT` (deferred to Gate C+): bucket table on the FibEntry; on + removal only vanished buckets are reassigned; on **addition or weight + change** the minimum number of buckets moves from over-represented + adjacencies to satisfy the documented rounded target shares (a newly + added adjacency therefore receives buckets); deletion-only stability and + addition/reweight behaviour have separate tests; FLUID uses actual + bucket counts; rounding error is bounded only after rebalancing. +- FLUID approximates HASH only under explicitly independent selection + domains (per-device seeds, distinct ECMP and LAG personalization) and + representative flow-key sampling; correlated revisits and polarization + studies use HASH. Rational routing fractions are exact; report doubles + carry a stated tolerance. + +## Flow placement (`model/flows.py`) + +``` +Demand(id, source, dst, rate, *, template=None, payload_size=1000, mode=FLUID, dscp=0, klass=None, steer=None, + flows=1, priority=0, tag=None) frozen; rate = IP-payload bit/s; time in seconds +PacketTemplate(af, src, dst, protocol, sport, dport, dscp, flow_label, payload_size, ttl, hop_limit) + deterministic defaults from the source device (loopback address, protocol 17, ports from a seeded generator); + a microflow generator maps index i into specified header fields (ports, flow label) before hashing, so the + auxiliary index is never an implicit wire field; every template field that affects steering, size or forwarding + is part of the FLUID equivalence class +DemandSet.matrix(sources, dsts, rate, mode=PAIRWISE | COMBINE, tag=...); DemandSet.pair(fwd, rev) for bidirectional + demands (an explicit reverse demand with its own source, dst and rate; nothing is inferred) +TrafficClass(name, dscp, priority); CapacityModel: UNCONSTRAINED | LOSSY +PlacementReport(time, version, per directed edge: offered_wire, carried_wire, congestion_dropped, capacity; + per demand: payload_delivered, drops by (reason, device or edge); classes and caches; detail) + views: utilization(edge), interface_tx/rx, bundle_utilization, member_utilization, oversubscribed(), + bottlenecks(k), by_risk_group(), by_tag() (available in FULL detail; SUMMARY keeps per-class aggregates + and per-demand totals and states which queries are unavailable) +``` + +**Rate and wire accounting.** `rate` is IP-payload bit/s and +`payload_size` is bytes after the inner IP header (the template states +any included transport header); packet rate `q = rate / (8 × +payload_size)`. Frame bytes on an edge = payload + inner IP header + +outer headers and SRH when present + 14 Ethernet bytes; FCS, preamble +and inter-frame gap are excluded explicitly. Attempted wire load on edge +e = `q × 8 × frame_bytes(e)`, charged as `offered` even if the physical +transfer fails; a physical failure carries zero. IP MTU compares IP +bytes only (excluding Ethernet) against the egress MTU in FLUID +(deterministic `MTU_EXCEEDED`). Reports distinguish payload delivery from +wire load, and tests use these exact units. + +**Forwarding-equivalence classes.** FLUID demands are grouped only into +classes for which forwarding is provably identical: (destination, dscp, +steering outcome at ingress, packet template, and the packet-state +continuation for SR chains). Each class walk is a destination-driven +**leg chain** over nodes keyed by (device, action state, header or +segment continuation), so encapsulation, local SID execution and +decapsulation are explicit transitions, not device-only nodes. HASH +demands never use classes: each microflow (`flows = N`, keys +`(flow_key, i)`, `rate / N`) is an explicit packet template executed by +the action program. + +**Leg walk.** Discover the class graph from the seed devices (LOOKUP → +group → members → `TRANSMIT` edges, with physical availability checked +per edge), compute strongly connected components over forwarding states and apply +the **loop-cut model**: a transition that stays inside a cyclic SCC +(including a self-loop) is charged as offered on that edge and recorded +as `LOOP` loss, exit transitions of the component are propagated, and +inflow is pushed through the condensation DAG in topological order, so +healthy inflow to nodes downstream of a looping branch is preserved +(the diamond fixture with the cycle {R1, R2} and the exit R1→R3→R4 +delivers 50 of 100 and loses 50 as `LOOP`). This is a conservative +approximation, not a TTL simulation: real packets re-circulate and a +fraction exits per pass; that geometric model (`loop_model=AMPLIFY`, with +the per-header hop-limit bound) stays deferred and changing the default +requires its own approval. HASH and probes execute finite per-header hop +limits and drop `TTL_EXPIRED`. Splits use the FLUID +rule, then equal shares across active bundle members; drops for +NO_ROUTE, DROP_*, EGRESS_DOWN, POLICY_DOWN, LINK_DOWN, RX_DOWN, +MTU_EXCEEDED are recorded at the node or edge where they occur; TTL is +not modelled in FLUID (HASH and probes carry one budget per header). A +directed edge traversed by k legs carries the rate k times; +conservation holds per leg and per demand (delivered + drops = rate). + +**Gate B forwarding identity and accounting contract.** Gate A keys the +walk's decisions by device and applies one wire factor per class, which +is exact for plain IP. Gate B keys vertices by (device, forwarding- +relevant packet and action state: header stack, active SID continuation) +so that R1→R2→R1 with a different continuation, or encapsulation followed +by decapsulation, are distinct vertices; every transmission carries its +own frame length and the packet rate, repeated visits of one physical +edge add up, MTU applies to that edge's IP stack and LOSSY clipping to its +wire load, and when several legs of one class consume the same residual +the order is the class's deterministic leg order. Reference numbers for +100 Mbit/s of 1000-byte IPv4 payload (no FCS, preamble or IFG): plain +framing 103.4 Mbit/s; outer IPv6 without SRH 107.4; outer IPv6 with a +one-entry SRH 109.8; after decapsulation 103.4 again. Probes, HASH and +FLUID share one action interpreter: FLUID expands alternatives, HASH +selects one, and the headend executes its own local SID before +transmitting. + +**Caches.** A class's result (sparse per-edge fractions and per-source +contributions, kept only in FULL detail) is stored in the frozen +`PlacementReport` with its **dependency record**: every consulted FIB +entry and failed lookup, group, interface, link (state and capacity), +SID and policy action, steering rule and load balancer. A class is +recomputed when any dependency's content changed; capacity-only changes +reuse UNCONSTRAINED fractions but recompute utilization and all LOSSY +allocation; only pure rate changes under UNCONSTRAINED rescale; changing +source, flows, mode, template or steering is a new class. Aggregate +inflow per class is O(V + E); per-source attribution is budgeted +separately and off by default. + +**Capacity models.** The walk propagates **packet rate** `q` (equivalently +original payload rate), never carried wire bit/s, so overhead is +recomputed per edge across ENCAP and DECAP. `UNCONSTRAINED`: no capacity +clipping (physical and MTU drops still apply), utilization may exceed 1. +`LOSSY`: demands in (−priority, id) order, lists by (weight desc, name), +legs in chain order, allocated per demand and directed edge in a stable +traversal: incoming `q` is split by the routing fractions; on edge e +`q_out = min(q, residual_wire(e) / (8 × frame_bytes(e)))`, the residual +is decremented by `q_out × 8 × frame_bytes(e)`, only `q_out` reaches +successors, the lost original payload rate is recorded as a congestion +drop on e, node and action drops are recorded separately, and original +payload demand is conserved. Strict-sequential by construction. +NetGraph-Core `EQUAL_BALANCED_LOSSY` parity holds only for plain-IP +single-leg demands without LAGs (or `expand_lags=True`) and at unit +frame overhead. + +## Determinism and free-threading + +Sorted iteration everywhere; never `hash()` or `id()` order; allocators in +the tree; `Device.seed` from the name; hot-path values are small ints; no +`ipaddress` in hot paths; no `lru_cache`; caches only inside a run or in +the tree. **Ownership boundary**: each `Network`/`Simulation` and its +mutable caches (handle cache, heaps, dirty sets, inboxes, observation +queue) has one owning thread; independent simulations run concurrently; +immutable snapshots may be transferred after safe publication; +cross-thread writes are queued to the owner; concurrent updates to one +Network are not supported. Verified on 3.14t by deterministic independent +runs and throughput measured against the same workload on one quiet +machine. + +## Standards basis + +| Area | Standard | Kept | Simplified | +|------|----------|------|------------| +| Datastores | RFC 8342 | config vs oper, one writer per key | | +| Interfaces | RFC 2863, 8343, 8344 | oper values, admin/oper, speed, mtu, addresses | statistics are rates | +| Ethernet, aggregation | IEEE 802.3, RFC 7042, IEEE 802.1AX, OpenConfig `min-links` / `aggregate-id` | frame fields, members, partner check, min-links | no FCS, no LACP PDUs | +| IPv4 / IPv6 | RFC 791, 2474, 1812, 8200, 4291, 6437, 6438 | headers, LPM, TTL rules, flow label at encapsulation | no checksum, fragmentation, ICMP | +| Unnumbered | RFC 8950 | interface or scoped link-local next hops for IPv4 | no BGP | +| RIB / FIB | RFC 8349, 8430 | distance then metric, next-hop kinds, cross-AF recursion, weighted groups | one table per AF | +| ECMP | RFC 2991, 2992 | region selection, resilient variant later | keyed BLAKE2b | +| Segment routing | RFC 8402, 8754, 8986, 9256, 9602, 9800 | SRH with containers, H.Encaps / Red, behaviours and flavors, NEXT-C-SID with F3216 (bare and composite), GIB / LIB / WLIB, §6.2 compression, 5f00::/16, one-owner SR Policy, W-ECMP, BSID, steering | no HMAC / TLVs / OAM / PCEP; nested B6 and REPLACE-C-SID deferred | +| BGP semantics used by NHT | RFC 4271 §9.1.2.1 | prospective resolvability with exclusion context | no BGP | + +## Protocol agents and integration (Gate C contract, revision 16) + +Refined on 2026-09-20 after an adversarial review of the implemented Gates +A and B against this section (scratchpad `codex-review-14-gatec.md`, copy +of `/private/tmp/netsim-gate-c-review-20260920/review.md`). Reproduced on +the tree: a kind run rejected by an observer after publication was +re-executed by `retry()` (fixed: an observer's exception is re-raised marked as published, `published_failure()`, and the claimed work is consumed); +`Study.iterations` drained the whole heap, which a recurring timer never +empties (fixed: the observation window, then `Simulation.run_derivations()` +which finishes only DEFERRED-band work such as a delayed FIB stage and +leaves timers alone); a metric-only RIB change +leaves the FIB object identical (so NHT cannot be keyed on FIB identity); +`changed_paths()` reports coarse device sections (a naive prefix matcher +on `.../oper` misses an interface oper change); a failed physical link +keeps both interfaces oper UP during carrier delay (a channel keyed on +oper state would pass traffic through a dead wire); the engine's heap +orders by delivery time, so a datagram sent later with a shorter delay +overtakes an earlier one; an unrelated loopback addition re-validates +every policy headend (conservative SR invalidation is a commit-path cost); +`srdb_source=("agent", name)` is ignored by policy validation, which +always reads the oracle. + +### Principles + +1. **One door** for routes, policies and SIDs through clients; agents and + sources register a `ClientId` and profile. +2. **Locality.** `AgentContext` is a projection, never a `Device` handle, + the model `_View` or a `NetworkState`. It exposes the device's + configured values and *effective observations* (oper state after + debounce, neighbors, its own RIB view with status, its installed + forwarding, session states, delivered messages with their scoped sender + identity, timers, NHT and SID results); it excludes raw carrier, remote + endpoints, root access and any oracle-derived inventory as a discovery + mechanism. The only global role is the `RouteSource` (pure function of + the tree, run by `converge()` too). `srdb_source=("agent", name)` + validates SR policies against that agent's *advertised* immutable + `srdb_view`; claims it has not advertised are unknown; only the + explicit oracle mode reads remote model state, and an unsupported mode + is rejected rather than silently falling back to the oracle. Remote + failures cost real delivery immediately where physical forwarding + requires it, while an agent's knowledge changes only through local + observations, messages and timeouts. Stateful `Controller`s stay + deferred past Gate C. +3. **Agent state in the tree**, opaque and identity-compared, with + validated immutable representations (incremental trusted builders, + debug whole-tree checks); inboxes, outboxes and timers are runtime + state owned by the `Simulation`. Callback objects carry immutable + configuration and no hidden evolving state; a frozen envelope may not + contain a mutable message or RNG. +4. **Subscriptions** are device-scoped, typed paths kept in an index: an + ancestor replacement or removal invalidates descendant subscriptions, + a descendant change invalidates ancestor subscriptions, and a coarse + delta section is filtered at field level before it wakes anyone. + Registration supplies an initial snapshot; causes are deduplicated; + only affected agents are scheduled; wakeups use commit-time internal + dispatch, not delayed public `SETTLED` observers. A one-device change in + a 100k-agent idle fixture examines only the matched branches. +5. **Protocols are plugins**; the contract lives in `model/contracts.py`. + +### Client APIs + +`RibClient` (above); `PolicyClient` (one owner per policy; candidate keys → +one client; steering entries keyed by client); `SidClient` +(`request_sid(request_id, behavior, args)`, results in `ctx.sid_results`). + +`NhtClient` is a local routing query service with three separate answers: +`resolve(candidate, exclude_rows, options)` is a pure RIB query returning +eligibility, the selected resolution prefix and source, the IGP cost to the +queried next hop (or unavailable) with its provenance (never a sum of +unrelated protocol metrics through recursion), resolved legs, the queries +made (including failed lookups) and the input epoch; installed forwarding +is exposed separately with the processed epoch, FIB version and PENDING +status; registrations are keyed by (owner, AF, address, interface +generation and scope, `connected_only`, `resolve_via_default`) and notify +on every semantic change of the answer, including cost-only changes (a +metric change does not change the FIB object) and formerly unsuccessful +queries that become resolvable. Registration never makes unrelated route +programming pending. Prospective questions (RFC 4271 §9.1.2) are asked +with an exclusion context that also excludes the candidate's own row. Exit +matrix: IPv4 and IPv6; identical link-local addresses on different +interfaces; connected-only and default controls; self-covering candidate +and mutual recursion; same-prefix alternative rows; failed lookup followed +by a more-specific insertion; cost-only change; delayed FIB; withdrawal +and recovery. Router id is derived by L3. + +### Agent contract + +``` +class DeviceAgent(Protocol): + client: ClientId; profile: ClientProfile; config: AgentConfig (immutable: peers, AS, timers, run_delay finite and + non-negative, default 1e-3, zero permitted and guarded by max_rounds_per_timestamp; + processing_delay; retransmit_timeout; listen ports; inbox and queue limits) + def subscriptions(self) -> tuple[Path, ...] # local paths only + def on_init(self, ctx) -> AgentOutput # first AGENT run after registration or reset + def on_run(self, ctx) -> AgentOutput + +AgentContext (read-only projection): now; device name; device config; agent_config; router_id; interfaces (index, + name, kind, config, effective oper, l3_usable per AF, link-local, MAC, bundle members); own link attributes + (index, delay, capacity as configured locally); neighbors (scoped); rib_view (rows by source with status); + lookup(af, address, scope), installed forwarding version and status; route_notifications; nht results; + sid_results; srdb_local; policy states; agent_state; inbox (the captured prefix); connection states of own + sessions; causes; rng (private generator seeded from committed state); stats (buffered counters) +AgentOutput (frozen): agent_state; operation records (route_ops, policy_ops, sid_requests, nht ops) produced by the + context's clients and published only by the runtime; messages (channel, port, immutable payload; route + advertisements batched as one immutable payload, never one event per prefix); sessions (open / close / abort); + timers (arm(name, delay > 0) / cancel); srdb_view | None +``` + +**Publication.** Every captured run has a receipt `(device generation, +agent generation, run id)`. All runs of a round prepare against one +immutable input snapshot; their accepted operation records are applied in +canonical owner order onto one staging root with transaction-local +allocator reservations and a per-output journal, so a conflicting or +invalid output is undone alone (two accepted SID requests can never reserve +the same value; a failed peer output never discards a successful one). The +round publishes once (one AGENT delta with the successful receipts inside). +A published receipt consumes its captured inbox prefix and publishes its +outbox (messages, session operations, timers) exactly once, even if an +observer raises afterwards (marked as published); a rejected receipt keeps its +captured inbox prefix, pre-run agent state and RNG, and `retry()` re-runs +only rejected receipts against a local view rebuilt from the corrected +committed root at the current time, with later arrivals kept for the next +run. A messages-only output is a success even when its model edit is a +no-op. Same-timestamp feedback produces successor rounds; positive delays +must advance simulated time. Restart: `Simulation.reset_agent(device, +name, purge=False)` invalidates the old generation's receipt, cancels its +timers, sessions, pending runs and inbox and outbox entries, resets state +and runs `on_init`; retention is defined separately for routes, policies, +SIDs and NHT registrations (rows stay until the first `sync` unless +purged). Registration, unregistration and client ownership are atomic and +scoped by device and agent generation. + +**Bounds.** Agent simulations expose a global time and event budget, +per-agent inbox limits and per-connection queued message and byte limits +with explicit overflow outcomes (rejection or connection failure, never +silent truncation), counts of live versus stale scheduled events, and +periodic compaction of lazy queues. Stats are counters and aggregates by +default with opt-in bounded samples. `run_delay` is a batching latency, +not a modeled CPU queue. Correctness never depends on retained history. + +### Transport (`runtime/transport.py`) + +- `LinkChannel(device, interface)` (datagram): a send attempt requires + local effective eligibility (the interface l3-usable for the family, or + bundle UP) **and** physical transfer and receive availability of the + wire (a stale oper UP during carrier delay cannot pass traffic through a + failed link); channel metadata never reveals raw failure to the agent. + A channel is directed and carries an endpoint incarnation; delivery at + `now + link.delay + processing_delay` must be a strictly future + representable time; the same physical checks apply at delivery and a + stale endpoint generation drops the datagram. Gate C guarantees FIFO + for surviving datagrams within one incarnation by **ordered release** + (a later datagram never overtakes an earlier one even when the delay + changed), and never retransmits lost datagrams. A bundle channel + samples its delay from the active members at send time; membership + changes do not reorder in-flight datagrams. +- `Session` (a reliable-message abstraction, not TCP): endpoints are (AF, + address, interface scope when link-local, port); a **connection** has a + distinct id, initiating endpoint and port, endpoint generations and + per-direction sequence numbers; states LISTEN, CONNECTING, ESTABLISHED, + DOWN; active open against a listener, passive-only peers accepted, + refused → `SessionEvent(DOWN, REFUSED)`. Simultaneous opens yield two + distinct connections that the protocol may keep or collapse (RFC 4271 + §6.8 decides by BGP identifier after OPEN); coalescing by endpoint order + is an opt-in policy of the reference protocol, never the only behaviour. + Close has two forms: abort (invalidate every pending, in-flight and + queued entry of the incarnation) and drain-then-close (a final message + may be sent first). Directly connected scoped link-local streams need + no routes (cold start). **Path reachability is separate from connection + state**: derived by the TRANSPORT kind with the forward step (a stable + per-connection probe key, both directions, dependencies including + negative queries recorded); a transient outage stalls delivery in one + ordered queue per direction and releases in sequence when the path + returns. The no-progress timeout starts at the first undelivered entry + and resets on delivery; it is distinct from protocol hold timers and + from the optional BFD-like immediate notification; expiry closes with + `SessionEvent(DOWN, TIMEOUT)`. Each direction has a bounded message and + byte admission with an explicit rejection or connection-failure result; + accepted data is never dropped silently. Control traffic is not charged + to demand utilization unless explicitly modeled. +- All delays strictly positive and validated; deliveries and timer + firings are NORMAL events at later times; the inbox is a runtime deque + whose captured prefix is consumed by publication. + +### Reference protocol (`agents/reference.py`) + +A documented single-area link-state teaching protocol, not Open/R +compatibility: neighbor discovery over scoped link channels without +oracle peer identities, bidirectional adjacency with hold timers, immutable +versioned adjacency and prefix records synchronized over sessions, flood +deduplication, expiry or withdrawal of stale origins across partition and +restart, ECMP SPF only when the semantic LSDB changes, client-owned route +deltas, and a small SID and locator advertisement so learned SR validation +has a fixture. It is accepted **with the oracle disabled**: normalized +prefixes, metrics, next-hop sets and final forwarding equal a separate +oracle network, ignoring ownership and row identity; an observed local +interface DOWN reacts immediately while a muted hello channel with oper +still UP withdraws after the hold time; cold start, partition and heal, +origin restart, duplicate and out-of-order advertisements, stale +withdrawal and metric-only change are separate tests; a heartbeat-only +interval triggers no SPF, FIB or placement work; an agent that reads +`oracle_igp` fails the locality test even when its routes match. + +### Studies with agents + +`settle()` means same-timestamp quiescence only and `run_derivations()` +finishes pending derivation-band work without touching timers. Agent +studies use an +explicit warm-up and observation horizon, a maximum event budget and a +named stability predicate over selected routing, programming and delivery +outputs (liveness refreshes do not reset it unless they change those +outputs), and report `converged`, `deadline_exceeded` or +`budget_exceeded`; convergence is never inferred from the last commit. +Each iteration constructs a fresh runtime from configuration and performs +deterministic warm-up; a model fork is not a warm protocol restart +(runtime checkpoint and restore stay deferred); warm-up cost is reported +as study cost. Oracle studies keep the fast fork path. + +### Scale qualification + +Separate oracle and protocol workload profiles (devices, links, areas, +advertised prefixes, per-device RIB and LSDB entries, peers and sessions, +demands and classes, message rate, duration, retention) accompany every +scale claim. A flat domain with one advertised prefix per device costs +about N(N − 1) remote rows (100 million at 10k devices) whatever the +implementation; a bounded routing scenario is a declared input, not a +consequence of a faster language. Gate C validates protocol semantics on +small topologies and scheduling and locality on large idle bounded-state +fixtures; the conservative SR consumer invalidation (every headend on any +RIB or interface change) is a named commit-path cost to be scoped by the +resolver's dependency records. Pure SPF, resolution and placement +functions remain the acceleration seams; native kernels are decided from +end-to-end profiles after the protocol workload exists. + +### Gate C slices + +| Slice | Deliverable | Exit evidence | +|---|---|---| +| C0 contracts and lifecycle | typed `AgentNode`, `AgentContext`, `AgentOutput`, operation records, run receipts, startup, reset and fork rules, the immutable payload boundary; this section reconciled with the code | publication versus observer failure unambiguous; a messages-only output works; cross-agent conflicts isolate only rejected outputs; runtime objects cannot leak through the context | +| C1 agent scheduler | the AGENT kind, indexed local subscriptions, captured inboxes, timers, RNG and stat buffering, partial success and retry | two-agent failure and retry matrix, successor rounds, same-time versus positive-delay limits, reset and deletion generations, no wakeups or scans of unrelated idle agents | +| C2 local query and NHT | candidate overlay and exclusion query, cost and resolution epochs, installed-state distinction, scoped registrations and dependencies | the NHT exit matrix above; metric-only changes notify although the FIB object is unchanged; misses become resolved on insertion | +| C3 transport | physical link datagrams, scoped route-free sessions, routed reliable-message sessions, bounded queues and connection lifecycle | cold-start link-local exchange; physical loss during carrier delay; FIFO under delay changes; one-way outage, refusal, timeout, close, old incarnation and backpressure | +| C4 reference protocol | discovery, finite semantic LSDB, flood, sync and restart, SPF and route diffs, learned SR view test | oracle-free cold start and failure and heal; normalized final equivalence on small random graphs; delayed FIB drops observable; unchanged hellos cause no SPF or FIB work | +| C5 studies and qualification | fresh-runtime warm-up, explicit observation and stability windows, streaming convergence and loss metrics, workload benchmarks | periodic timers never hang an iteration; failures beyond the deadline report non-convergence; repeatable outputs across interpreters and seeds; bounded queue and history memory; declared scale envelope | + +Not in Gate C: full Open/R or BGP (only their contract probes: discovery +without prior routes, passive and simultaneous opens, BGP-style +prospective and cost-sensitive NHT), controllers, on-wire B6, resilient +hashing, VRFs, in-band control traffic accounting, runtime checkpoint and +restore, parallel execution of one `Simulation`. + +### Open/R and BGP on the contract (deferred implementations) + +Open/R: Spark hellos on link channels; KvStore over link-local scoped +sessions to discovered neighbors first, loopback sessions after routes +exist; LinkMonitor from interface subscriptions; Decision SPF with +interface + link-local next-hops; PrefixManager; Fib = `RibClient` with +`sync`. BGP: listener and active-open sessions, FSM driven by +`SessionEvent`s and timers, Adj-RIB-In/Loc-RIB/Adj-RIB-Out in sharded +state, best path, `NhtClient.resolvable` for §9.1.2.1, distances 20/200, +redistribution from `rib_view`, color communities to `PolicyRef` +steering. Both are Gate C+ work items with cold-start tests. + +## Runtime API and the diamond + +``` +env = netsim.Environment(); net = netsim.model.Network(srv6_block="5f00::/32") +R = {n: net.add_device(n) for n in ("R1", "R2", "R3", "R4")} +for i, r in enumerate(R.values(), 1): + r.add_loopback("lo0", ipv4=f"10.0.0.{i}/32", ipv6=f"2001:db8::{i}/128") +net.add_p2p(R["R1"], "eth3", R["R3"], "eth1", ipv4=("10.1.13.0/31", "10.1.13.1/31"), speed=100e6) +net.add_p2p(R["R2"], "eth3", R["R4"], "eth1", ipv4=("10.1.24.0/31", "10.1.24.1/31"), speed=100e6) +net.add_p2p(R["R3"], "eth2", R["R4"], "eth2", ipv4=("10.1.34.0/31", "10.1.34.1/31"), speed=100e6) +net.add_lag(R["R1"], "Po1", ["eth1", "eth2"], R["R2"], "Po1", ["eth1", "eth2"], + ipv4=("10.1.12.0/31", "10.1.12.1/31"), min_links=2, speed=100e6) +R["R1"].add_route("10.0.0.2/32", nexthops=[("Po1", "10.1.12.1")]); R["R1"].add_route("10.0.0.3/32", nexthops=[("eth3", "10.1.13.1")]) +R["R1"].add_route("10.0.0.4/32", nexthops=["10.0.0.2", "10.0.0.3"]) # one row, two recursive next-hops +R["R2"].add_route("10.0.0.4/32", nexthops=[("eth3", "10.1.24.1")]); R["R3"].add_route("10.0.0.4/32", nexthops=[("eth2", "10.1.34.1")]) +net.add_demand(Demand("d1", source="R1", dst="10.0.0.4", rate=100e6)) +sim = netsim.runtime.Simulation(env, net) +sim.at(10, lambda: net.links["R1:eth1--R2:eth1"].fail()) +sim.run_until(20) +report = sim.timeline.snapshot_at(10).placement; report.utilization(net.links["R1:eth3--R3:eth1"].edge("R1")) +sim.timeline.summary(10); sim.timeline.utilization_series(net.links["R1:eth3--R3:eth1"].edge("R1")) +``` + +Addresses use /31 pairs (RFC 3021), so no broadcast address is used as a +next hop. `add_route` takes a list of next-hops for one row; distinct +rows need explicit distinguishers. Clock-free: `net.converge()`, +`net.place()`. + +## Adapters + +`sonic.py`: CONFIG_DB load/dump preserving **literal** SID values, +prefixes, structures and behaviours (bare and composite forms are +distinct; no naming conversion); round-trip guarantees restricted to a +versioned supported schema; APPL_DB export in wire form. `core.py`: +`build_graph` (`ext_edge_id = 2*index + direction`, masks), IGP +acceleration, FLUID vs `EQUAL_BALANCED` cross-check on random simple +graphs (plain IP, unit overhead), pinned-path cross-check. + +### Running a NetGraph-built network (`adapters/ngraph.py`, Gate A) + +A network built with NetGraph (Python API or scenario YAML with +blueprints) is a supported input for a full simulation, not only a +topology source. The adapter maps the whole scenario: + +| NetGraph | NetSim | +|---|---| +| `Node(name, disabled, risk_groups, attrs)` | `Device` (name escaped if it contains `/`), `enabled = not disabled`, `lo0` with loopback addresses from `attrs["loopback_ipv4" / "loopback_ipv6"]` or deterministic pools in name order; a locator when `srv6=True` (Gate B) | +| `Link(source, target, capacity, cost, disabled, risk_groups, attrs, id)` | one `Link` between two auto-created Ethernets (`attrs["source_interface" / "target_interface"]` or `eth{n}` in link-id order); `capacity × capacity_unit` → speed (bit/s, default unit 1 Gbit/s); `cost` → interface metric both ends (`attrs["reverse_cost"]` for asymmetry); `disabled` → FAILED; risk groups kept on the link; parallel links tagged `attrs["lag"]` → one PortChannel per side with `attrs["min_links"]` | +| addressing | `unnumbered` (default): loopbacks only, interface next-hops, oracle routes; `p2p`: /31 and /127 pairs from pools | +| routes | `igp.py` (oracle) installs shortest-path ECMP routes with NetGraph costs as metrics, so `flow_policy: SHORTEST_PATHS_ECMP` maps to FLUID over the oracle; WCMP or TE flow policies are not mapped in Gate A (Gate B expresses pinned paths as SR Policies) | +| `demands` (`TrafficDemand`: source and target regexes, volume, mode combine or pairwise) | `DemandSet.matrix(sources, dsts, rate=volume × capacity_unit, mode, tag=demand set name)`; destinations are the matched devices' loopbacks | +| `failures` (`FailurePolicy`, seeded modes and rules) | a **failure schedule**: `FailureManager.compute_exclusions(policy, seed_offset)` yields excluded nodes and links per iteration; the adapter turns each iteration into operations at a chosen time (`device.enabled = False`, `link.fail()`), optionally restored after a dwell, so a Monte Carlo study becomes one long timed run or N independent runs with the scenario seed | +| `seed` | `Network` seed for MAC pools, hash seeds and the failure schedule | +| results | `to_network(net, placement)` writes per-link utilization and offered, carried and dropped wire load into link `attrs` and per-device drops into node `attrs`; `Timeline.utilization_series` gives utilization over time; a `results.json` export mirrors NetGraph's shape (per iteration: exclusions, delivered, oversubscribed links) so existing NetGraph tooling can read it | + +`from_scenario(scenario, *, capacity_unit=1e9, addressing="unnumbered", +srv6=False)` returns `(Network, DemandSet, FailureSchedule)`; +`from_network(network, ...)` is the topology-only subset. The plain-IP +FLUID results of a NetGraph-built network must match NetGraph's own +`SHORTEST_PATHS_ECMP` placement per directed edge (the +`EQUAL_BALANCED` cross-check) whenever no demand is dropped, which is the +acceptance test for the import. Caveats stated to users: NetGraph +capacities are abstract units (choose `capacity_unit`); NetGraph costs +must be non-negative integers (they are, by Core's constraint); node +names containing `/` are escaped in paths; a NetGraph link is +bidirectional with the same capacity each way, which maps to one +full-duplex NetSim link. + +## Acceptance scenario (drives the tests) + +Diamond as in the example (100 Mbit/s links, /31 pairs, loopbacks, +locators in `5f00::/32`; oracle routes where statics are absent). The +Gate-A variant is IPv4-only and leaves IPv6 forwarding disabled on links; +the Gate-B fixture explicitly enables IPv6 forwarding on every Ethernet +and PortChannel L3 interface (scoped link-local neighbors derived) and +verifies locator reachability before creating policies. + +| Step | Expected | +|---|---| +| Build; `converge()` and `Simulation` + `run_until()` (no device agents, `prefer_installed=False`) | trees equal modulo version/since; R1 10.0.0.4/32 INSTALLED with group `[(eth3, 10.1.13.1), (Po1, 10.1.12.1)]`; contributing row key recorded | +| FLUID 100 Mbit/s R1 → 10.0.0.4 | payload: R1→R3 50, each Po1 member 25, R2→R4 50, R3→R4 50; wire load adds 34 bytes per 1000-byte packet; delivered 100 | +| t=10: fail R1:eth1 link, `min_links=2` / `1` | Po1 LOWER_LAYER_DOWN both ends, connected /31 withdrawn, static via R2 NOT_INSTALLED, all 100 via R3 / Po1 UP, remaining member 50 | +| Physical failure during debounce: `carrier_delay_down=1` on **both** endpoints of R1:eth1–R2:eth1, `carrier_delay_up=0`, `min_links=2`, UNCONSTRAINED | during 10..11 routing is stale: Po1 still splits to eth1; on that edge offered payload 25 Mbit/s (offered wire 25.85 Mbit/s at 1000-byte payload), carried 0, payload drop 25 (`LINK_DOWN`), delivered 75; at 11 the transition applies, delivered returns to 100 and the remaining path offers 103.4 Mbit/s wire. Asymmetric partner failure (one endpoint delayed) is a separate test with its immediate bundle consequences | +| Rounds at t=10 (statics only, no agents) | one round: link; CARRIER {R1.eth1, R2.eth1}; LAG {R1.Po1, R2.Po1}; L3 {R1, R2}; FIB {R1, R2}; PLACEMENT; then SETTLED deliveries | +| `fib_delay=0.05` on R1 | at 10 R1 rows PENDING (epochs), placement still into a down Po1 with `EGRESS_DOWN`; at 10.05 INSTALLED and corrected | +| Carrier delay: `delay_down=1`, fail at 10, restore at 10.5 | no oper transition (two raw transitions logged) | +| Carrier delay: `delay_up=2`, `delay_down=0.1`; raw UP at 10, DOWN at 10.5, UP at 10.6 | UP at 12.6; a `carrier_delay_up` reconfiguration to 1 at 11 reschedules to `max(11, 10.6 + 1)` | +| Asymmetric delays: R1.eth1 `delay_down=0.2`, R2.eth1 0 | R2.Po1 shrinks at 10; at 10.2 R1.Po1 `min_links_ok` false and R2.Po1 active shrinks again (component two-pass) | +| Reactive process waiting on eth1 DOWN | its delivery is a separate SETTLED event with the settled root; its failing of a second link opens another round | +| Failed derivation | an exception propagates from `run_until`; after fixing, `settle()` retries the retained work once | +| IPv6 and unnumbered variants | identical payload loads | +| HASH probe IPv4 5-tuple | one path, ttl 64 at R1 (originated), 63 at transit, delivered; TTL=1 drops `TTL_EXPIRED` at the second device | +| Mutual statics R1/R2 for 10.0.0.9/32 plus a healthy branch S→B→D | FLUID: the cyclic SCC absorbs its inflow as `LOOP`, the healthy branch delivers; HASH: `TTL_EXPIRED` | +| `resolve_via_default=True`; blackhole; mixed special+forwarding group | resolvable via default; `DROP_BLACKHOLE`; special wins (documented) | +| Gate B: SR Policy (color 10, endpoint R4), paths pref 200 strict R1→R2→R4 and pref 100 strict R1→R3→R4, demand steered by class | all 100 on Po1 (50 per member) and R2→R4; wire load on the encapsulated hops 107.4 Mbit/s without SRH and 109.8 with a one-entry SRH (1000-byte payload), 103.4 after decapsulation | +| Fail Po1 (`min_links=2`) | R1.uA→R2 adjacency DOWN in the SR-DB → list invalid → path 100 active | +| `fallback=DROP`, only path 200 | explicit discard installed, `POLICY_DOWN`; equals NetGraph-Core pinned-path result | +| W-ECMP lists via R2 (3) and via R3 (1) | 75 / 25; HASH microflows ≈ 3:1 | +| UNCOMPRESSED probe, list `[R2.End.X→R4, R4.End.DT46]` | H.Encaps.Red at R1 (SRH with one further entry, SL = 1, LE = 0), End.X at R2 (SL 0), DT46 at R4 | +| uSID strict path `[R1.uA→R2, R2.uA→R4, R4.uDT46]` (F3216 bare) | one container, no SRH; **R1 executes its own uA before transmitting** (DA shifted to `B:A2:D4::`, outer hop limit 63 on the wire), R2 shifts (62) and forwards, R4 decapsulates; compiled entry trace-equivalent to the step-by-step run | +| uSID with SRH: 7 C-SIDs (loose path with uN and uA) | DA = container 1, SRH = [container 2], SL = 1; after the sixth C-SID `arg == 0` → SL 0, DA = container 2 (PSP pops the SRH) | +| uSID validity | `[R2.uA→R4, R4.uDT46]` from R1 invalid under the strict profile and forwards wrongly under basic validation (trap test); `[..., R4.uN]` without USD invalid; `[R1.uA→R2, R2.uA→R3, R3.uDT46]` for endpoint R4 invalid (`ENDPOINT_MISMATCH`); a literal SID list without structure metadata is accepted uncompressed | +| Composite vs bare | `B:N:F::/64` and `B:F::/48` are distinct SIDs; the SONiC round trip preserves each literally | +| BSID steering | static via BSID → policy legs; H.Encaps at R1, no BSID on the wire | +| Gate C: reference protocol with the oracle disabled | after convergence the agent's rows equal the oracle rows of a separate oracle network under **normalized** comparison (prefix, metric, next-hop set, final forwarding; ownership and row identity ignored); an observed local interface DOWN at t=10 reacts immediately, a muted hello channel with oper still UP withdraws after the hold time, then re-flood, SPF and the same final placement; one AGENT delta per round with the successful receipts inside; cold start converges over link-local sessions; a heartbeat-only interval causes no SPF, FIB or placement work | +| Round overflow | a same-time subscribed-state feedback loop with `run_delay = 0` (permitted, guarded) raises `ConvergenceError`; a positive-delay message storm is bounded by a time/event budget instead | +| Agent batch failure | one agent's invalid output (including an allocation conflict with another agent's accepted output) leaves the other agents' outputs committed in the round's single AGENT delta, propagates an aggregate failure, and `retry()` re-runs only the rejected receipt with its captured inbox prefix; an observer failure after publication re-runs nothing | +| NHT | R1 registers 10.0.0.2; with every alternate route removed the result becomes UNRESOLVED and the agent withdraws dependents; a self-covered next hop reports UNRESOLVED with `via_prefix == P`; a metric-only change of the resolving route notifies with the new cost although the FIB object is unchanged; a formerly failed lookup notifies when a more-specific row appears | +| Sessions | U1, U2 sent during a transient outage arrive in order after recovery; a datagram sent later with a shorter link delay never overtakes an earlier one on the same channel; a failed wire under carrier delay drops datagrams although oper is UP; a no-progress outage longer than the timeout closes the connection, both agents get `SessionEvent(DOWN, TIMEOUT)`, late events of the old incarnation are no-ops; no listener → `REFUSED`; simultaneous opens yield two connections; a full admission queue rejects explicitly | +| Client contract | identical row add and missing-key delete yield `None`; `sync` from one client leaves another's rows untouched; a bad output raises, leaves rows, state, outbox and RNG unchanged, and the retry consumes the same inbox | +| NetGraph scenario (Gate A) | the README's dual-site Clos scenario (blueprints, `count: N` parallel links, `demands`, `failures`, `seed`) imports through `from_scenario`; with the oracle the per-directed-edge FLUID loads equal NetGraph's `SHORTEST_PATHS_ECMP` placement at 1e-9 when nothing is dropped; each failure iteration applied as a timed schedule reproduces NetGraph's per-iteration delivered volume; `to_network` writes utilization back and `results.json` loads in NetGraph tooling | + +## Probabilistic failures and the study entry point (revision 13) + +Decided on 2026-09-20 from experiments (scripts in the session scratchpad: +`exp_mc.py`, `exp_avail.py`, `exp_leases.py`, `exp_ng_mc.py`, +`exp_ng_vs_ns.py`, `exp_parallel.py`, `exp_profile_iter.py`). + +### What was measured + +| Experiment | Result | +|---|---| +| NetGraph draws vs NetSim draws, same scenario and step seed (`square_mesh`, 6 iterations) | identical failure patterns in the same order, identical `occurrence_count`, identical `failure_id` (blake2s of sorted nodes `|` sorted links) | +| `backbone_clos.yml` (127 nodes, 756 links), 50 iterations of `weighted_modes` | NetGraph static MC: 0.3 ms per iteration; NetSim fork + converge: 2.4 s per iteration; 41/41 failure ids matched | +| Profile of one NetSim iteration on that scenario | oracle IGP 4.1 s (Dijkstra for every device and AF, twice), FIB 2.8 s (32k prefix resolutions), both full recomputes for one failed link | +| Monte Carlo on Clos 16x4: rebuild vs `Network.fork()` vs timed fail/restore | 69 ms / 56 ms / 88 ms per iteration; derivation dominates, not construction | +| Forked iterations on 3.14t threads (Clos 16x4, 64 iterations) | 2.3x on 4 threads, flat at 8 (shared-root refcount traffic); 1.0x on the GIL build | +| Renewal process on Clos 8x4, 32 links, MTBF 1e4 s, MTTR 100 s, 2027 failures | 171 s wall (85 ms per failure, about 10 records each), 194k timeline events in 92 MB, availability computable from `delivered_series` | +| Policy-style events with durations, Poisson arrivals, up to 9 overlapping | lease counting keeps "link UP iff no lease held" with 0 violations over 577 failures | +| Delivered volumes vs NetGraph | comparable only under the same capacity model; NetGraph ECMP admission is lossless per demand, NetSim UNCONSTRAINED is not, so the check stays scoped to "nothing dropped" as in the acceptance row | + +### Conclusions + +1. **Static Monte Carlo stays NetGraph's job.** NetSim is four orders of + magnitude slower per settled iteration and cannot compete with the + C++ core on combinatorial studies; it should consume NetGraph's draws + and add what NetGraph cannot express: the transient (loss during + convergence under carrier delay, FIB delay and fast failover), time-based + availability, and protocol behaviour. The one performance lever is + incremental IGP and FIB (and Core-accelerated SPF), not the failure + machinery. +2. **NetGraph failure policies are the selection language.** A policy draw + is `FailurePolicy.apply_failures_typed` seeded per draw; NetSim reuses + `FailureManager.compute_exclusions(policy, seed_offset=effective_seed + i)` + bit-for-bit, so an iteration in NetSim and in NetGraph share the + `failure_id`. Risk groups resolve to members inside NetGraph; NetSim + applies nodes as `enabled=False` and links as `fail()`, never as masks, + so the DES sees transitions. +3. **Time is NetSim's addition.** NetGraph has no duration, repair or + schedule concept and its schema rejects unknown keys in `failures`, but + policy, node, link, risk-group and demand `attrs` are free-form and + workflow steps accept arbitrary kwargs. NetSim timing parameters + therefore live in `attrs.netsim` (per entity: `mtbf`, `mttr`, + `carrier_delay_down`, `fib_delay`, `fast_failover`, interface names, + addresses) and in the NetSim workflow step's kwargs (per study: rate, + duration distribution, horizon, model, retention). + +### Failure sources (`runtime/failures.py`) + +Every source yields fault events `(entities, start, duration | None)` +applied through one **lease registry**: an entity fails when its first +lease starts and restores when its last lease ends, so overlapping events +(a risk group and one of its members, two concurrent draws) compose +correctly. Entities are devices, links and risk groups (a group is the +transitive member set resolved once). + +| Source | What it is | Seeds | Use | +|---|---|---|---| +| `Schedule` | explicit timed events from Python, a table, or a NetGraph `results.json` replayed by `failure_id` | none | reproductions, worst-case patterns found by NetGraph | +| `Draws` | one failure set per iteration: a NetGraph policy with `effective_seed + i` (NetGraph-compatible) or an enumeration (`all single links`, `all pairs`, `all risk groups`) | NetGraph's | transient studies per failure pattern, dedup by `failure_id` with `occurrence_count` as the weight | +| `Process` | a point process over a horizon: per-entity renewal (TTF and TTR distributions from `attrs.netsim.mtbf/mttr` or defaults; exponential, lognormal, Weibull) or policy-driven arrivals (Poisson rate per mode, each arrival draws a set through the policy and holds it for a drawn duration) | derived from the scenario seed with NetGraph's `SeedManager` formula and the components `("netsim", kind, name)`, so streams are reproducible and independent of NetGraph's | availability, loss-time, concurrent-failure distributions, recovery times | + +Validation tests: seed compatibility (failure ids equal NetGraph's for a +scenario and step seed), lease invariant under random overlap, renewal +unavailability converging to `MTTR / (MTBF + MTTR)` per entity, schedule +replay reproducing a NetGraph pattern, results round trip. + +### Iteration semantics + +An iteration forks the converged baseline (`Network.fork()`: shared +immutable root, no copy), applies the set at `t0`, runs until settled (the +transient is on the timeline), optionally restores and settles again (the +recovery transient), and records: NetGraph's `FlowIterationResult` fields +(`failure_id`, `failure_state`, `occurrence_count`, `flows`, `summary`) +plus `data.netsim` (settle time, bits lost during the transient, drop +reasons, event counts, per-demand downtime). Iterations are independent, +so they parallelize by process (roots are immutable; threads give 2.3x on +3.14t today). + +### Entry points + +1. **Python one-off**: `Network` + `Simulation` + a failure source, zero + dependencies. `Simulation.failures(Process(...))` and + `Simulation.failures(Schedule(...))` schedule the events. +2. **NetGraph `Scenario` (recommended for studies)**: the expanded + scenario is the description of record (network, demands, policies, + risk groups, seed); `Study.from_scenario(scenario, **options)` builds + the baseline through the adapter and exposes `iterations(draws, ...)` + (a `Draws` source from a policy or an enumeration), `enumerate(...)`, + `process(source, horizon)`, `replay(results, step, select)`, each + returning a `StudyResult` with `to_ngraph()` (NetGraph result shape) + and `rows()`; iterations run serially in this revision. NetGraph + `combine` demands become one anycast destination announced by every + target with an even share per source; `pairwise` splits the volume over + the pairs; NetGraph priorities (lower first) map to NetSim priorities + (higher first) with the original value kept for exports. NetSim does + not grow its own DSL. +3. **Workflow step**: `netsim.adapters.ngraph` registers `NetSimStudy` + with `register_workflow_step`, so a scenario YAML can carry a + `type: NetSimStudy` step with NetSim kwargs; `netsim run scenario.yaml` + imports both packages and writes NetGraph's `results.json` shape with + the NetSim step beside NetGraph's steps, and `ngraph run` works once + `netsim.adapters.ngraph` is imported. + +### Work item (before Gate B) + +`runtime/failures.py` (leases, Schedule, Draws with enumeration, Process +with distributions), `study.py` (Study, StudyResult, availability and +loss metrics from the timeline, process parallelism), adapter additions +(policy draws with NetGraph seeding, `attrs.netsim` mapping, `NetSimStudy` +step, results export and replay), a `netsim run` CLI, and the tests above. +Incremental IGP and FIB are a separate performance item that this work +does not depend on. + +## Scale: what changes now (revision 14) + +An independent Codex scale review (report at +`/private/tmp/netsim-scale-review-20260920/report.md`, critique of this +section in the session scratchpad `codex-review-4.md`) measured the Gate A +tree at 1k to 100k devices. Every headline claim was reproduced on the +live tree on 2026-09-20 (`verify_scale.py`, `verify_converge.py`, +`verify_pending.py`, `proto_shards.py`, `proto_igp_rib.py` in the +session scratchpad). What the numbers say, and what they do not: + +| Claim | Reproduced | Root cause in the tree | +|---|---|---| +| one-device commit costs 16 ms at 100k devices | yes: 16.5 ms = `bump_epochs` 6.5 ms + `StateDelta.is_empty` 8.7 ms; the top-map copy itself is 0.24 ms | both walk every device although only one changed (`derive.bump_epochs`, `state.diff_pmap`) | +| RIB `rows(prefix)` is linear in the shard | yes: 631 µs per lookup at 10k rows; the resolver is quadratic | `RibState.rows` sorts and filters the whole prefix-length shard | +| pipeline pending scan is quadratic in distinct deadlines | yes: 0.8 → 5.3 ms per deadline from 250 to 2000 debounce deadlines | `_run_kind` scans all `kind.pending` on every run | +| `round_end_scheduled` never shrinks | yes | fixed: popped when the round closes | +| Tarjan recursion fails at chain length 1000 | yes | fixed: explicit-stack Tarjan, tested to 20k with a reference | +| `FrozenPrefixTable.shards()` leaks writable inner dicts | yes (introduced with the timeline) | fixed: inner tables wrapped too | +| initial convergence 512 devices 50 s; repeat without change 4 s at 256 | yes: 8.4 s / 4.1 s at 256 on the live tree | `derive_fib` re-resolved every (device, AF) even when the resolver epoch had not moved; fixed: epoch gate, now 5.6 s / 1.1 s; the remaining no-op cost is the oracle SPF for every device | +| RIB index and per-pass egress cache give only 1.3–1.5x on convergence | yes: prototype 8.4 → 6.0 s at 256 | the dominant cost is resolving every prefix on every event and running SPF for every device, not lookups | +| 100k devices need 430 MB of configuration | plausible, not re-measured | ~4.3 KB per device across five interface and link records | +| N² RIB rows in a flat all-to-all domain | arithmetic, correct | the oracle installs a route to every loopback on every device; hierarchy or aggregation is a scenario property, not a code fix | + +Not accepted from the report: the "P0" framing of keyed agent +subscriptions, observation byte budgets, backend contracts and a full +point resolver as blockers before Gate B. Agents do not exist yet, the +observation policy already has retention knobs, and the resolver +acceptance matrix is the Gate C incremental-resolution item. They stay +P1/P2 by profile. + +### The rule that fixes the class of issues + +Bookkeeping must cost in proportion to the changed shards and inputs; +derivation must cost in proportion to the affected dependency closure +plus the output it necessarily produces (a bridge failure legitimately +changes every device's routes, and no data structure removes that +output). Any wider conservative scan is named, measured and bounded, +never implicit. Analytical and protocol-simulation scale results are +published separately against a workload passport. The Gate A tree +already provides the two ingredients: identity-first deltas and resolver +epochs. Where a step still walks everything that did not change, the fix +is structural, not a cache. + +1. **Sharded persistent top-level maps** (`state.py`). `PMap` keeps its + API but the `devices`, `links` and `demands` maps (later any per-device + table that grows with N) become two-level: 2^k shards by key hash, a + `set` copies one shard and the shard vector, and `diff_pmap`, + `bump_epochs` and `changed_paths` walk only shards whose identity + differs. Prototype at 100k keys: `set` 0.24 → 0.003 ms, one-key diff + 5.3 → 0.02 ms, `get` 0.09 → 0.15 µs. This removes the O(N) commit, + the O(N) epoch bump, the O(N) delta and the 3.85 MB per snapshot at + 100k at once, with no transaction API change. Builder semantics stay + (detach on build). Canonicalization returning old objects is what makes + shard identity meaningful, so derivations must keep returning the old + record when nothing changed. +2. **`Network.batch()`**: a scoped transaction that applies many builder + operations to one candidate root, validates once and commits once, for + bulk construction and the adapters (10k `add_device` calls take 6 s + today because every call is a global commit). +3. **RIB prefix index** (`routing.py`): `prefixes` becomes + `FrozenPrefixTable[tuple[Route, ...]]` (prefix → rows sorted by rank), + maintained by `rib_apply`, so `rows(prefix)` is O(1) and `rows_of` uses + a client index. Prefix-length shards remain the storage. +4. **Per-kind deadline heap** (`pipeline.py`): `(deadline, generation, + entity)` with lazy invalidation (an entry is stale when + `pending[entity]` no longer equals its deadline); due work pops while + `deadline <= now`. DEBOUNCE reschedules push a new entry and leave the + old one to expire. `scheduled` and `round_end_scheduled` entries are + removed when consumed. +5. **Oracle IGP reuse** (`igp.py`): the adjacency and egress structure is + built once per (invocation, AF) and shared by every source's SPF; a + family is skipped only when its whole destination set is empty, and an + empty result is still synced to withdraw stale rows. Memoizing across + invocations on input identity (inside an owned source object, never a + module-level cache, never shared through `fork()`) stays open together + with the convergence certificate. +6. **Epoch-gated derivation** (done): `derive_fib` skips a (device, AF) + whose processed epoch equals its input epoch, so a no-op converge and + `Simulation()` on a converged network resolve nothing. Any RIB change + of a device advances **both** families' epochs and schedules both FIB + runs, because a row may resolve recursively through the other family + (RFC 8950); until dependency-selected invalidation exists this is the + conservative rule. +7. **Correctness now** (done): iterative Tarjan; sealed prefix-table + views and a constructor that copies its inputs; bounded round + bookkeeping; resolver dependencies scoped per prefix frame (memoized + children merge their dependencies into the parent) so `depends_on` is + exact and the total is linear in the table instead of quadratic (3000 + blackhole routes: 4.5 M references → 3018); a kind whose own commit + dirties it again gets its successor round (due work is claimed before + the run); adjacencies carry the next hop's family. + +Contracts the Codex critique added to the items above: + +- Sharding is O(S + N/S) per mutation and O(S + qN/S) per diff with q + changed shards: linear in N with a small coefficient, not sublinear; + benchmark 256 and 512 shards; keep small maps compact; mix the key hash + (aligned integer keys such as `i << 8` fall into one shard under a plain + mask); include `allocators.next_ifindex`; make `PMap.__eq__` shard-aware + so `canon()` and record comparisons do not restore full scans; define + iteration order explicitly (insertion order is no longer natural) and + keep set-like `keys()` equality; never canonicalize with `tree_equal`, + which ignores timestamps and epochs. +- `Network.batch()` must stage into builders and freeze once (persistent + `set` per operation still copies shards), must not rebuild the prefix + index per imported route, and must invalidate provisional handles on + abort (generations roll back with the tree, so an escaped handle could + become valid again): either non-reused generations or explicit + provisional invalidation. +- The deadline heap keeps a monotonically increasing per-kind ticket, + separate from round generations, as the invalidation key and the tie + breaker (entities may be incomparable); COALESCE keeps the earliest + deadline, DEBOUNCE replaces with the latest request even if earlier; + retry keeps an in-flight batch without overwriting newer requests; + stale entries are compacted against live pending work, and stale + `StageEvent`s already in the engine queue are accounted for. +- Convergence shortcuts need a certificate, not a tree version: a + version-only skip would keep a 2 Gbit/s delivery on a 1 Gbit/s link + after `set_capacity_model(LOSSY)`, and `add_source` and the evaluated + time also change results without a version bump. A certificate covers + root identity, derivation-configuration revision and time, is issued + only after a full convergence, and may be copied to a fork. +- Oracle reuse: skip a family only when the whole destination set is + empty (an addressless device is valid unnumbered transit), sync an empty + result to withdraw stale rows, and never key a memo inside a source + object that `fork()` shares by numeric epoch alone. +- Observation: `Timeline` appends records unconditionally, so disabling + extraction does not establish a memory plateau; long runs need a record + budget (P1). + +**Status (2026-09-20, branch `network-layer-gate-a`):** items 1–7 and +the revision 13 failure/study layer are implemented, each on its own +Codex worker branch with A/B/A numbers under `benchmarks/` and +`tests/runtime/*.md`, merged with FIB/placement fingerprints unchanged +(`tests/test_fingerprints.py`), and hardened by two adversarial review +rounds (scratchpad `codex-review-5.md`; fixes: batch epoch consumption +after an in-batch converge, validator rejects the mutable prefix-table +builder, Study tolerance and single-pass metrics, lease consistency under +observer exceptions, NetGraph `combine` as anycast and priority direction, +backward-band invalidations to the successor round, RIB delete ownership, +incremental client/prefix indexes, linear carrier invalidation; third +round: `converge()`/`place()` inside `batch()` raise (a batch is one commit +of builder operations, derivations run on committed roots only), +`update(fn)` inside a batch advances epochs for its own changes, retry +keeps successor-round work out of the current round, every observer runs +before an observer exception is re-raised, anycast allocation is +collision-checked and fork-safe, adapter metadata rolls back on abort and +is copied by `fork()`, `strict` import rejects untranslated NetGraph demand +options). +Integrated measurements on the same machine: one-device commit at 100k +devices 16.5 ms → 0.076 ms; 64 retained roots at 100k 246 MB → 1 MB; +`rows(prefix)` at 10k rows 631 µs → 0.3 µs; one-row RIB add or delete at +10k rows 1.5 ms → 17 µs; 256-device ring initial convergence 8.4 s → +2.0 s, no-op 4.1 s → 0.29 s, one failure 1.6 s → 0.47 s; 10k `add_device` +6.8 s → 0.09 s in a batch; NetGraph backbone import plus convergence +2.43 s → 0.66 s and 2.4 s → 0.52 s per failure iteration; 2000 carrier +deadlines on one device 10.5 s → 2.4 s (1.2 ms each, mildly rising); +Monte Carlo iteration on Clos 16x4 56 → 35 ms. Lookups cost 1.8x on +sharded maps (117 ns vs 64 ns); bulk RIB sync is 1.4x slower for the +index maintenance. Unchanged by design: the per-event cost on small +topologies (Clos 8x4 renewal study still 83 ms per failure, dominated by +the oracle SPF for every device and full dirty-FIB resolution), which is +the incremental-resolution and oracle-memo item below. + +Acceptance for 1–5, each with an old-vs-new measurement on one quiet +machine and fingerprint equality of FIBs and placements before and after: +a one-device commit at 100k devices touches one shard (< 0.1 ms); 10k +devices build in a batch in under 1 s; 10k-row `rows(prefix)` in under 5 +µs; 2000 distinct deadlines run in time linear in the count; no-op +converge at 256 devices under 0.2 s (measured 0.29 s: the remaining cost is the oracle SPF for every device, kept as the target of the oracle memo item); convergence fingerprints unchanged +for 32 to 256 devices. + +### What remains the bottleneck afterwards, and its plan + +After 1–5 the per-event cost is the oracle SPF for every device plus the +FIB re-resolution of every prefix of every device whose rows changed. +That is the **incremental resolution** item (deferred until a measured +Gate C workload requires it; Gate C itself delivers scoped point queries +and indexed notifications, see slice C2): rows reused by identity +through `sync` already mark what changed; the resolver then re-resolves +only changed prefixes and the dependents recorded in `depends_on` +(interfaces, queried prefixes including failed lookups), with a measured +fallback to a full rebuild when the closure is large, and the acceptance +matrix from the design (batch add/withdraw, competing sources, covering +prefix change, recursive chain and cycle, ECMP leg down, equality with +the full resolver after every event). Native SPF (NetGraph-Core, 10 ms +per 100k-vertex SPF vs 70 ms in Python) comes after that, by profile, +and only together with a storage that does not materialize one Python +object per route: an N² flat domain is a scenario property that no +language change removes. + +Two modes, one forwarding model: the analytical mode (oracle, instant +convergence, lazy routes allowed behind an explicit interface) and the +protocol mode (local views, messages, timers, stale FIBs preserved) share +interfaces, transfer availability, RIB/FIB contracts, SRv6 semantics, +probes, accounting and the timeline schema; they may differ in how state +is materialized. Sizing a study needs a workload passport (devices, +degree, routes per device, demands and classes, sessions, event rate, +history detail), and 100k support is claimed only against such a passport +with published time and memory, never from the ability to build the +configuration. + +## Gates and phases + +1. **Gate A**: `core.py` `DEFERRED`; `model/addressing.py`, `lpm.py`, + `packets.py` (templates, wire size), `hashing.py` (threshold only), + `state.py` (owning PMap, immutability validator, generations, paths, + canonicalization), `interfaces.py` and `links.py` (physical + availability, LAG component two-pass), `routing.py` (rows, epochs, + outcomes, resolver), `forwarding.py` (plain-IP records, the action + interpreter, probes and timed forwarding), `igp.py`, `contracts.py` + (clients only), `derive.py`, `entities.py`, `network.py`; `flows.py` + plain IP (classes, SCC legs, wire accounting, UNCONSTRAINED and LOSSY, + report); `runtime/pipeline.py` (rounds with generations, kinds, retry, + convergence limit), `events.py`, `simulation.py`; diamond rows through + "mixed group"; NetGraph adapter (`from_scenario`: topology, demands, + failure schedule, seed; `to_network` and `results.json` export; + `srv6=False`) and Core adapter for the cross-check. +2. **Gate B** — **implemented on 2026-09-20 on `main`** (slices B0–B4 as + seven Codex xhigh worker branches `gateb/g1..g7` plus follow-ups, + integrated with the plain-IP fingerprint oracle unchanged; 1238 tests on + both interpreters, 134 against the real NetGraph and Core packages; + three adversarial review rounds (scratchpad `codex-review-10-gateb1.md`, + `codex-review-11-gateb2.md`) whose 16 findings were all fixed by the + owning workers or the integrator; remaining cost: plain-IP convergence + about 6% slower (clos 8x4 converge plus three failures 30.6 → 32.6 ms; + 1250 tests after the fourth review round) from packet-state vertices + and SR hooks in placement). Profile of the remaining per-event cost on + the 256-device ring, one link failure plus converge, 870 ms wall: the + oracle runs SPF for every device and family twice (the fixed-point + confirmation pass repeats it: 1024 SPF calls, 0.7 s profiled) and syncs + every device's rows (3072 `rib_apply` calls, 0.39 s), while the FIB + re-resolves every prefix of the 92 dirty (device, family) pairs (11776 + prefix resolutions, 0.6 s). The next performance items are therefore, + in order: skip the oracle's second pass when its inputs are identical + (the convergence certificate), avoid re-syncing unchanged devices, and + the incremental resolver.: `srv6.py` (structures, allocator, compression, policies + with one owner, validation layers), the SR extension of the action + program in `forwarding.py` with the authoritative Gate-B behaviour + set, SR legs in `flows.py`, steering precedence, SONiC adapter with + literal SIDs; diamond Gate-B rows; hand-checked RFC 9800 §6.2 vectors. + Delivered in acceptance-gated slices, each keeping plain-IP + fingerprints unchanged and clock-free convergence equal to settled + timed execution: + - **B0** behaviour and flavor matrix as the authoritative table; typed + SR records and the header-chain contract; owner-scoped SID and + policy operations; allocator and invalidation wiring. Exit: every + unsupported combination rejected; collision, exhaustion and import + failures atomic; forks detached; local, remote and failed-query + dependencies tested. Invalidation scenarios that must schedule the + consumer: a remote adjacency SID goes down without changing the + headend's RIB; a missing SID appears; the SRv6 source or hop-limit + setting changes while rows stay identical; an owner deletes and + recreates a SID; a batch changes and restores an input (committed + dirty epoch). Every changed SR input advances the affected consumers' + epochs and runtime work; negative queries are recorded; bare local + SIDs are scoped by device and block; resolution reads one immutable + snapshot regardless of device iteration order; the oracle originates + locator prefixes; conservative all-SR-consumer invalidation is + acceptable at first. + - **B1** uncompressed literal SIDs, locator routing, H.Encaps and + H.Encaps.Red, End, End.X, End.DT46, headend-local execution. Exit: + hand-checked traces for IPv4 and IPv6 inner packets; malformed SRH, + TTL, MTU, flavors and physical failures; compiled and interpreted + execution agree on the same encoded input. + - **B2** one-owner policies, weighted lists, steering and fallback + (excluding the triggering action so IGP fallback cannot re-select + it), stateful SR FLUID and HASH with capacity. Exit: timed diamond + failover and restore; validity, programming and delivery reported + separately; per-edge frame sizes; repeated-device visits and loop-cut + cases; conservation including tiny rates. + - **B3** NEXT-C-SID compression, bare and composite forms, several + blocks and widths. Exit: independently checked encoding boundaries + (block changes, mixed widths, non-zero arguments, unknown structures, + capacity limits, terminal-tail fit); local-scope and cross-connect + cases (End.X's shifted cross-connect checked apart from End's lookup + continuation); compressed and uncompressed equivalents deliver along + the intended physical path when TTL, MTU and capacity do not + constrain either, with each encoding's header sizes and hop-limit + transitions tested separately. + - **B4** versioned, bounded SONiC subset; explicit-path NetGraph SR + translation; policy timeline and Study export. Exit: literal SID + preservation; strict errors for unsupported input; destination, + priority and unit exports; repeatable study results with measured + scaling limits. + Across slices: nonzero FIB and carrier delays, successor-round + invalidation, retry, observer exceptions, change-and-revert batches, + owner conflicts and failed allocation are exercised; LOSSY keeps its + qualified tests (no unconstrained superposition assertions) and HASH + convergence to FLUID is asserted only for independent hash domains. +3. **Gate C** — **implemented on 2026-09-20 on `main`** (slices C0–C5: + the C0 contract written by the integrator, then five Codex xhigh worker + branches `gatec/c1-agents` (scheduler, subscriptions, inboxes, timers, + transactional publication and retry), `gatec/c2-nht` (local query and + NHT, `srdb_source` agent mode, scoped SR consumer invalidation), + `gatec/c3-transport` (link channels, sessions, indexed reachability), + `gatec/c4-reference` (the oracle-free reference link-state protocol with + 44 acceptance tests and a locality harness) and `gatec/c5-study` (study + windows, fresh-runtime warm-up, streaming metrics, workload passport), + each integrated with the plain-IP fingerprint oracle unchanged and + verified on both interpreters and against the real NetGraph; reports in + `tests/model/c2*-handoff.md`, `docs/history/c4-reference-agent.md`, + `docs/history/c5-study-windows.md`). Integration decisions recorded in the + commit history: a fresh `Simulation` restarts every agent node another + runtime initialized (a fork is not a warm protocol restart); + `AgentConfig.processing_delay` defaults to 1 ms so deliveries over + zero-delay links advance the clock; opaque agent values compare by + identity; the context exposes no path reachability. The first + adversarial review round (scratchpad `codex-review-15-gatec1.md`: 4 P0, + 5 P1, all reproduced and accepted) was fixed by the owning workers and + merged on 2026-09-21 (main `a2370a0`, 1588 tests on both interpreters): + per-output commit-time validation so an invalid output is rejected + alone, publication status fixed before any finalization step, control + notifications (session events, rejections, timers) admitted into a + bounded allowance whose exhaustion fails explicitly, a physically lost + datagram admitted and lost silently (only local admission errors are + rejections), detached prospective NHT queries on the context + (`ctx.resolve`, `ctx.installed` over the device subtree only), scoped + registrations bound to the interface generation (`SCOPE_STALE` after + re-creation), whole-tree state validation only under `debug_validate`, + dependency- and owner-indexed session dispatch with bounded DOWN + retention, and aggregate stats with opt-in bounded samples; the + reviewer's ten probes re-run clean on the merged tree. Measured on the + same machine, 64-router reference-protocol cold start: 0.625 s simulated + in 14.1 s wall before the fixes (53% transitive validation of agent + state on every run) and 6.4 s after (4224 session messages, 2176 SPF + runs, 4858 events); one link failure 1.09 s → 0.41 s wall (2972 → 1111 + µs per event). The second review round (scratchpad + `codex-review-16-gatec2.md`) confirmed eight of the nine fixes, found + one partial (the shallow admission check admitted a frozen record + wrapping a list: fixed on 2026-09-21 by `validate_admitted`, an + incremental walk that trusts subtrees reused by identity from the + previously admitted value, so a new state costs only its changed + subtrees and whole-root walks stay debug-only; 32-router cold start + 1.11 → 1.32 s wall for the guarantee) and two new items assigned to the + owners: the agent-study baseline must come from the warmed protocol + runtime rather than the cold synchronous tree (C5), and rejected + receipts must be parked outside the runnable heap until `retry()` + (C1); both merged (98a0615, 5e0ef63). Confirmation rounds 3 to 9 + (scratchpad `codex-review-17..23`; round 9 on main `380691a` found + no new finding and no remaining blocker within the stated boundary) + confirmed every earlier fix and + closed the admission theme step by step: the immutability walks are now + fail-closed through one classifier (exact leaf types; leaf, tuple and + frozenset subclasses only without attribute storage read from real + descriptors, which admits `MacAddress` and `NamedTuple`; enum members + only with the enum machinery's own attributes and their values walked; + only classes decorated as frozen dataclasses themselves, inheriting + only frozen dataclasses and storage-free mixins, with no instance + storage beyond their fields and every inherited slot an effective + field; `PMap` keys + and values; frozen prefix tables with exact-int keys, masks and bits; + `Fraction` and its storage-free subclasses with exact-int components; + `FloatArray` owning exact bytes; declared-immutable types), and a subscription or NHT cause that lands + while a receipt is parked gets its own run after a successful retry. + Boundary stated for the record: enum members and declared-immutable + types are class-level singletons trusted by declaration at admission; + mutating them afterwards is a program-level contract violation the + snapshot invariant cannot police. Scoped point queries and indexed + notifications are in; full incremental FIB reconstruction stays + deferred unless a measured Gate C workload requires it (the remaining + 64-router cost is dominated by full local FIB rebuilds: 164k prefix + resolutions in the qualification run). +4. **After Gate C** (each its own review). Open/R and BGP are planned in + `/Users/networmix/.claude/plans/netsim-protocols.md` (revision 8, + 2026-09-27, after internal adversarial reviews). It targets modern FRR, + keeps FRR 10.5 as a settings preset, and adds a per-node settings system. + It integrates the routing-policy plan in `NetSim/docs/plans/` (a shared + policy engine) and a FIB programming and acknowledgement contract. That plan also + records a confirmed resolver defect: answers depend on traversal order under + recursion cycles and memoized depth truncation, and slice F1a fixes it. + Remaining items: Open/R, BGP, resilient hashing, + nested B6, REPLACE-C-SID, multi-client BSIDs, NHT-based incremental + resolution, midpoint protection, VRFs, runtime checkpoint, in-band + transport, `loop_model=AMPLIFY`. + +## Verification + +- `make qt`; `make check-ci` (lint, pyright, coverage ≥ 85 %) before each + gate; `make check-ft`; benchmarks before any scale claim (route batches + of 100k rows, report size, retained memory, update/delta cost, 8-leaf + smoke within the 10 s timeout, 64-leaf marked `slow` with its own + timeout), old vs new on one quiet machine. +- Scheduler invariants: kinds run in priority order within a round; + backward invalidations land in the next round; `ROUND_END` and SETTLED + ordering; convergence limit; retry after failure; positive-delay rule; + `run(until)` and `settle()` semantics against `core.py`. +- State invariants: transitive immutability (validator), identity of + unchanged subtrees, empty delta on no-op, one writer per key (origin + tags), generations on handles, allocators unchanged after failed + updates, sharded state keeps other shards' identity. +- Routing and SR: resolver fixtures pinned against NOS behaviour + (precedence, weights, truncation), epoch-based status, neighbor and AF + enablement matrix, invalidation matrix edge tests, RFC 9800 compression + vectors, behaviour tests per RFC branch incl. malformed SRH, compiled vs + interpreted trace equivalence, validation layer tests. +- Placement property tests: Kirchhoff per node and per demand on random + acyclic FIBs; superposition and scaling under UNCONSTRAINED; Core + `EQUAL_BALANCED` cross-check on random simple graphs (plain IP); SCC + loop regression (healthy and looping branches merging); physical + failure during carrier delay; LOSSY monotonicity; HASH with many + microflows converging to FLUID on independent-domain cases. +- Agents: output validation and atomicity, locality of the view, FIFO + and incarnation rules, cold start, reference agent equals oracle on + random graphs. diff --git a/docs/design/archive/protocols-r8.md b/docs/design/archive/protocols-r8.md new file mode 100644 index 0000000..77e20f7 --- /dev/null +++ b/docs/design/archive/protocols-r8.md @@ -0,0 +1,4305 @@ +# NetSim protocols: Open/R and BGP (design and plan, revision 8, 2026-09-27) + +> **Rename, 2026-10-02:** the project is now NetGraph-Sim: distribution `ngraph-sim`, package `ngraph_sim`, CLI `ngraph-sim`, scenario keys `attrs.sim` / `data.sim`, workflow step `SimStudy`. Paths `netsim/...` and the old names below refer to the same code, now in `ngraph_sim/`. The repository is github.com/networmix/NetGraph-Sim and the checkout `~/ws/NetGraph-Sim`. Legacy and compatibility paths described below (Study `settle` mode, `keep_*` aliases, top-level NetGraph attrs, `strict=False`, `Simulation.recorder`, unbound NHT keys, untagged deliveries) were removed on 2026-10-02. + + +Companion to the network-layer design of record, +`/Users/networmix/.claude/plans/hazy-foraging-cloud.md` (revision 17, Gates A, +B and C implemented). This document plans the item "After Gate C: Open/R, BGP" +of that design. Research reports, benchmarks and their citations live in +`/Users/networmix/.claude/plans/netsim-protocols-research/`: + +- `research-openr.md`: Open/R at `facebook/openr` commit `81dae4c` (MIT), cloned at `~/ws/openr`. +- `research-frr-decision.md`: FRR bgpd and zebra decision process, multipath, NHT, next-hop rules, install, profiles, pipeline. Read at master `f1d7bec` (10.8.0-dev), cloned at `~/ws/frr`. +- `research-frr-rr-confed.md`: peer sort, route reflection, confederations, AS_PATH, attribute propagation, update groups, FSM, graceful restart (same commit). +- `research-bgp-options.md`: FRR, BIRD, GoBGP, OpenBGPD, ExaBGP, Batfish, C-BGP, Holo, RustyBGP and bgpsim compared; FRR topotests as a corpus; licensing; the FRR differential lane. +- `bench_bgp_scale.py`, `bench_agent_ctx.py`, `bench_pmap.py`: the measurements in section 1. + +The default FRR dialect is **`frr-modern`**: master `f1d7bec` (10.8.0-dev), +cloned at `~/ws/frr`, with one suspect master change quarantined (Z5, section +2B.2). The **`frr-10.5`** dialect covers what SONiC builds (frr-10.5.4, tag +`4cb6d9e`, `~/ws/frr-10.5.4`). The full history with tags is at `~/ws/frr-git`. +Section 11 records what changed in each revision. + +FRR `file:line` citations refer to master `f1d7bec` unless marked `10.5.4:`. +Open/R citations refer to `81dae4c`, and NetSim citations to main `0e5b299`. + +--- + +## 0. Summary + +NetSim gets a faithful Open/R IGP and an FRR-dialect BGP, both as protocol +agents on the Gate C contract. Nothing gets a privileged path into the model: +routes go through `RibClient`, next hops through NHT, messages through the +transport, and configuration through the tree. Protocols meet only through the +RIB, NHT and published device-local views. + +Four phases: + +1. **Foundation (S0, F0–F8, RP1, RP3, RP4).** In order of the list: + - S0 scale hygiene: five measured fixes that make builds linear and idle + agent runs cheap (section 14); + - contract additions; + - a fix for a confirmed defect: the resolver's answers depend on traversal + order; + - fixes for the measured O(table) costs that would make BGP-sized tables + unusable; + - the transport, underlay and toolkit pieces both protocols need; + - the settings system (per-node overrides, dialects, profiles; section 2A); + - the forwarding programming contract with FIB acknowledgement gating (F8, + section 5.13); + - the shared route-policy engine and its RIB, FIB and resolver hooks (RP1, + RP3, RP4; section 5.8, from the reviewed routing-policy plan). +2. **Open/R (OR1–OR6) and BGP (BG1–BG7)** as two parallel tracks. BGP tests use + a domain-scoped oracle IGP underlay, so neither track waits on the other. +3. **Integration (IX)**: IGP plus BGP scenarios for every deployment shape in + the brief. + +Section 12 is the implementation roadmap: milestones M0–M6, the test harness, +and per-slice files, signatures, tests-first lists and exit checks. Section 14 +is the scale review: measurements on the current tree, experiments proving the +fixes, prototypes at 1M prefixes, the honest scale target with passports, and +the streamlined slice set (S0 first). Section 15 applies the Codex review of +that revision: the immutable `PackedTable` contract, corrected passports and +measurements, the per-leg cycle rule, the activation-stage `ip protocol` +filter, group versions for programming, and the authoritative slice and +dependency table (§15.9), which supersedes the earlier ones. Section 13 +answers each part of the brief directly: IGP domain shapes, session types, +hop-by-hop iBGP instead of an IGP, flavor swapping, RR and confederations, RFC +annotation and direct versus indirect next hops. +4. **Later (BG8 and beyond)**: more decision flavors, add-path, graceful restart, + BFD, SR color steering, an analytical BGP solver. + +| Decision | Choice | +|---|---| +| IGP reference | The Open/R open-source daemon, module for module. Features absent upstream stay absent unless labelled `netsim_ext` | +| BGP reference | RFCs first; modern FRR (master, 10.8.0-dev) as the default dialect where the RFCs leave room, with `frr-10.5` (SONiC) as a settings preset; generic version differences are settings, bugs are not emulated; clean-room reimplementation with no FRR code | +| Differential oracle | Optional containerlab lane running FRR images: `10.7.1` release, the `master` nightly pinned by digest, and `10.5.4` for the SONiC dialect. It compares converged state only; goldens are committed as NetSim-generated data | +| Configuration | Tree state: agent configuration changes in place within a generation; commands are exec-only operations | +| Observability | Agents publish immutable operational views; tests, the timeline and studies read views, never agent internals | +| Modularity | One package per protocol and one submodule per real module. Submodules are pure functions over frozen sub-states, joined by explicit worklist queues | +| Replaceable parts | Frozen specs resolved against built-in tables with citations (decision flavors, multipath rules, Open/R selection mode, policy dialect), plus a settings system in which every setting is overridable per node, with dialects and presets as data (section 2A) | +| Domains | Configuration, not topology: a device runs 0..k IGP instances (more than one is `netsim_ext`) and at most one BGP instance | +| Scale rule | Cost proportional to change is a performance exit criterion for every protocol slice (§8), backed by operation-count and ratio benchmarks | +| Next hops | Direct and indirect exactly as FRR and zebra handle them: single-hop eBGP is non-recursive with a connected check, everything else is recursive, and BGP-over-BGP is allowed | +| Routing policy | One typed native language and evaluator shared by BGP, Open/R and the RIB, FIB and resolver hooks; exact vendor adapters (FRR subset first); bindings per node (D13) | +| Installation | An asynchronous programming contract with generation-guarded completions; FIB acknowledgement gating and inactive export per node (D14) | + +--- + +## 1. Where the project stands + +**Implemented** on main `0e5b299`: `make qt` passes 1648 tests, skips 11 and +deselects 13 slow ones (2026-09-26); CI runs Python 3.11–3.14 and 3.14t. + +- Gate A: immutable state tree, rounds, interfaces and LAG, static and oracle + routing, the FIB resolver with recursion and RFC 8950, placement, the NetGraph + and Core adapters. +- Gate B: SRv6 with micro-SIDs, SR policies, steering, the SONiC subset. +- Gate C: + - the agent contract (`netsim/model/contracts.py`); + - NHT (`netsim/model/nht.py`); + - transport (`netsim/runtime/transport.py`): link-channel datagrams, + route-free scoped link-local sessions, routed sessions, simultaneous open + giving two connections (RFC 4271 §6.8), abort versus drain, no-progress + timeout, bounded queues; + - the reference link-state protocol (`netsim/agents/reference.py`); + - study windows with fresh-runtime warm-up. + +**What already fits:** + +- Several named agents per device, each with its own `ClientId(name, instance)` + (`netsim/model/network.py:483-545`). +- Per-row distance: `rib_apply` never uses the profile distance, so one BGP + client can install at 20 and at 200. +- Recursive next hops with a depth limit and cross-family resolution. +- NHT with `connected_only` and `resolve_via_default`. +- Scoped link-local sessions for cold start, unnumbered BGP and KvStore peering. +- Advertised SR-DB views. +- Fingerprints hash flattened FIB group content, not group ids + (`tests/test_fingerprints.py:56-83`). + +**Gaps.** Measured on this tree with the scripts in the research folder +(same machine, CPython 3.14), or confirmed in code: + +| # | Gap | Evidence | Consequence | +|---|---|---|---| +| G1 | The FIB rebuilds every prefix of a (device, AF) on any input change (`netsim/model/routing.py:834-934`) | 4-router diamond with N recursive `/24`s via a loopback. One unrelated row added: 1.4 / 15.7 / 177 / 1709 ms at N = 0 / 1k / 10k / 50k. One IGP metric change: 1.6 / 16.5 / 187 / 1803 ms (≈ 35 µs per row) | Every event on a BGP-sized table costs O(table) | +| G2 | The agent context copies every RIB row of every client on every run (`netsim/runtime/agents.py:511-529`) | An idle agent whose run only re-arms a timer: 0.19 / 1.23 / 12.9 / 130 ms per run at 0 / 1k / 10k / 50k rows on its device | Every keepalive and hello costs O(RIB) | +| G3 | Admission re-walks every key of a changed `PMap` (`netsim/model/state.py:735-790`, `for k, v in o.items()`) | Agent state holding a `PMap` of N paths, one entry changed. `PMap.set` takes 1.5 / 3.4 µs, `validate_admitted` 20 / 236 ms, at 10k / 100k | Any large agent table costs O(size) per run | +| G4 | `CAUSE_ROUTES` scans every outcome row of each agent's client on a changed device (`netsim/runtime/agents.py:338-365`) | 8.5 / 270 ms per agent per commit at 10k / 50k rows, for every agent on the device | O(rows) per commit per agent; the largest per-agent cost measured | +| G5 | NHT per-owner views and notifications scan all registrations (`agents.py:366-391`); every epoch bump refreshes every registration (`nht.py:327-350`, `derive.py:904-911`) | code | O(registrations × agents) per commit | +| G6 | Listeners bind exact endpoints; sessions carry no TTL or GTSM (`netsim/runtime/transport.py:805-925`) | code | Cannot model single-hop versus multihop eBGP, per-peer TTL on one listening socket, or `ttl-security` | +| G7 | The oracle IGP is one flat domain over all devices (`netsim/model/igp.py:191-218`) | code | Multi-AS and multi-domain BGP tests cannot use the fast underlay | +| G8 | `AgentConfig` is fixed per generation, yet the runtime reads `run_delay`, the inbox limits and `listen_ports` from it (`agents.py:234, 424`; `transport.py:339-349`); reset and fresh runtimes rebuild from it (`agents.py:267, 1046`) | code | A configuration change would need a restart, which drops sessions | +| G9 | NHT reports a cost only for link-state sources and only when a single source resolves (`netsim/model/nht.py:180-196`); the answer is prospective (RIB), not installed | code | FRR's IGP-metric step uses the metric of the installed resolving route from any protocol: a BGP route contributes its MED, a connected or static route contributes 0 (`research-frr-decision.md` §3.4) | +| G10 | Agents can publish only `srdb_view` into the tree (`contracts.py:697`) | code | Tests, `Timeline.series` and study stability predicates would have to read opaque agent state. BFD-like cooperation has no channel | +| G11 | The reference protocol is monolithic and re-scans the whole LSDB on every run | `netsim/agents/reference.py` | Not a base for Open/R. It stays as the teaching protocol and contract probe | +| G12 | **Defect: the full resolver gives order-dependent answers.** The memo is consulted before the stack and depth checks, and results that were stack-excluded or depth-truncated are memoized (`routing.py:446-471`; lookups exclude everything on the stack, `:503`) | Reproduced (scale review `order_dep.py`, `order_dep2.py`). A two-route cycle A ↔ B under a covering `/8` resolves both to A's next hop or both to B's, depending on which is visited first. With `max_recursion_depth=4`, adding unrelated deeper chain members uninstalls a valid chain | An incremental resolver cannot be checked against the full one, and today's results depend on traversal order | +| G13 | `FibEntry.depends_on` is transitive and has no runtime consumer (only `forwarding.py:92` and tests) | After an IGP metric change every recursive entry changes (`interfaces` goes from `('e2','e3')` to `('e3',)`) and the group id changes. About 1.5 KB per row across RIB, FIB and outcomes (74.6 MB for 50k rows) | A hierarchical FIB cannot leave entries untouched | +| G14 | The Simulation path has further O(table) steps. The resolver builds fresh per-length dicts (`lpm.py:223-227`), so timeline FIB-event extraction (`timeline.py:832-866`) and the unchanged-FIB check (`routing.py:921`) walk everything. NHT refresh recomputes every registration on every RIB change, up to three times per commit (`derive.py:760-763, 810-812, 906-910`) | One added row at 10k / 50k: timeline FIB events 31 / 168 ms; unchanged-FIB check 5.9 / 102 ms; G4 8.5 / 270 ms per agent on the device. NHT refresh 40–49 µs per registration at 50k rows. One-row agent run 1.0 ms at 0 rows, 411 ms at 10k | Fixing only `resolve_fib` leaves the Simulation path O(table) | +| G15 | Conservative downstream invalidation. Placement recomputes every class through a device on any FIB change (`pipeline.py:643`; `flows.py:929-971`). Transport re-traces every connection through a device on any FIB change (`transport.py:608-660`). Both families' epochs are bumped on any RIB change (`pipeline.py:613`; `derive.py:869-879`), so PENDING is device-wide | code | Every BGP FIB commit re-places demands and re-traces sessions, and `rib_view` shows the whole table PENDING during `fib_delay` | +| G17 | **Quadratic network build.** `_validate_interface_config` walks every device per interface add (`network.py:738`) | 5.5 / 21 / 31 ms per device at 2k / 8k / 10k devices; linear at 95–125 µs once indexed (section 14.2) | 100k devices cannot be built | +| G18 | The MT RNG state (625 ints) is committed per agent run (`agents.py:792`) | 27 KB retained per idle run; 1.8 KB without it | Long runs with many agents grow without bound | +| G19 | Every agent run commits, because `runs`, `receipt` and `rng` change | 160–200 µs per idle tick; the DES core alone costs about 1.5 µs per timeout | 100k agents cannot tick | +| G20 | Per-table mask tuples (`lpm.py:193`) and about ten resolver objects per prefix | 25 KB per device with no routes; 1.5 KB per recursive row | Memory at 100k devices and 1M prefixes | +| G16 | Garbage collection multiplies allocation-heavy paths | G4 at 50k rows: 219 ms with GC, 54 ms without (778k tracked objects) | Passports must report heap object counts. Removing per-entry `DependsOn` saves about 8 objects per FIB entry | + +--- + +## 2. Decisions + +**D1. References, pins and licensing.** + +- **Open/R** (MIT) is followed module for module at `81dae4c`: messages, FSMs, + merge rules, timers and the initialization chain. Upstream facts that + contradict common assumptions are honoured (`research-openr.md`, headline): + - there is no domain name; + - Decision implements only shortest-path ECMP; + - DUAL flood optimisation is not wired in; + - the default metric is RTT-based; + - a fresh node boots soft-drained; + - area import policy is a stub; + - metric-based best-route selection is off by default. + + Anything NetSim adds is named `netsim_ext` and is off by default. +- **BGP.** The RFCs come first: 4271, 4456, 5065, 6793, 4760, 2545, 8950, 7606, + 8212, 5004, 6286, 1997, 4360, 8092, 5082, and later 7911 and 4724. Where the + RFCs leave room, the dialect is **modern FRR**. Three reasons: + - FRR covers everything in scope; + - it publishes `bgp_path_selection_reason` codes that make selection traces + diffable; + - SONiC runs FRR, and NetSim has a SONiC adapter. + + Section 2B carries the analysis of 10.5 against modern: + - the default dialect `frr-modern` follows master `f1d7bec`, except the + quarantined Z5 tie-break; + - `frr-10.5` reproduces SONiC's release defaults and generic behaviour + choices, not its bugs; + - every generic version difference is a setting with per-dialect defaults + (section 2A), never a version check in code. + + Profiles (FRR `frr defaults`): `traditional` and `datacenter`. Distribution + overlay: `sonic` (section 2B.4), each with provenance. +- **Licensing.** FRR is GPL-2.0+, so NetSim reimplements it clean-room. + Behaviour comes from RFCs, FRR's documentation and FRR black-box runs. + Comments cite FRR function names, steps and reason enums. No code or + transliteration is copied. +- **Rejected as reference:** + - GoBGP: no IGP-metric step; + - OpenBGPD: no confederations; + - Batfish: a fixed point with an approximate MED rule; + - C-BGP: unmaintained since 2013; + - Holo: no RR or confederations; + - bgpsim: no confederations; + - FRR under DES time: impossible, because it uses the wall clock and kernel TCP. + + BIRD 3 is kept as an optional second opinion when NetSim and FRR disagree. +- **Differential lane.** The FRR differential lane compares converged state + only, and is opt-in and off `make check`. Its images and pins are in section + 2A.5 rule 7. FRR topotest data is read at run time + from a pinned FRR checkout and never vendored; the data files carry no licence + header. + +**D2. Protocols are agents; no privileged path.** Agents see only what +`AgentContext` gives them and act only through `AgentOutput`. They interact only +through three channels: + +- the RIB, through `RibClient` and the RIB feed; +- NHT; +- published views (`AgentOutput.views`, F0), which other agents on the same + device may read. This is the FSDB/Sysdb principle. + +No agent reads another agent's state. + +**D3. Structure mirrors the real implementation, joined by explicit queues.** + +- Each protocol is a package with one submodule per real module. +- Each submodule is a pure function over frozen records and `PMap`s: + `(sub_state, queue_inputs, projection) -> (sub_state', queue_outputs, + timer_ops)`. +- **Declared projections.** Each submodule receives only the declared slice of + the context it needs, never the whole `ctx`. A guarded proxy checks this in + tests. +- **Worklist.** The agent run drains a worklist of explicit module queues in a + fixed priority order until every queue is empty, with a bound that raises if + exceeded. This matters because Open/R's queues form cycles: + - LinkMonitor ↔ Spark; + - KvStore → Decision → Fib → PrefixManager → KvStore. +- **Programming-gated edges.** Edges that in reality wait for programming + consume the F0 owner notification (`RowNotice INSTALLED`) instead of + handing off within the run: Open/R Fib → PrefixManager, and FULL_SYNC → + PREFIX_DB_SYNCED. +- **Timers.** Timers are namespaced `module/key`. Per-key deadlines (KvStore + TTL, per-peer MRAI, hold timers) are multiplexed through one armed timer per + module plus a deadline heap in state, as Open/R's `ttlCountdownQueue` does. +- **Randomness.** Each module draws from its own RNG stream, derived by name + from `ctx.rng`, so a new draw in one module does not shift the jitter of + another. + +**D4. Replaceable parts are frozen specs with provenance.** + +- Decision flavors are **named code implementations** of a small interface + (`compare`, `select`, `multipath`) over narrow immutable inputs: `frr` and + `rfc4271` ship built in, and a scenario may add a class at deployment. The + setting `bgp.decision` stores the name, so it is overridable per node. There + is no step interpreter and no per-run step compilation (§15.8, §16). Knobs + that FRR exposes as `bgp bestpath ...` are ordinary settings. +- Multipath rules, the Open/R selection mode and the policy dialect are + likewise named implementations selected by settings. +- User-defined steps are immutable classes with a purity contract and `refs`, + passed by value inside the spec. There is no global mutable registry. +- Every built-in and user step declares `refs`: an RFC section, an FRR + function, step or reason, an Open/R `file:line`, or a vendor document URL. An + invariant test rejects a missing reference, which turns "annotated with RFCs" + into a checked design rule. +- The flavor implementation is looked up by name from `ctx.agent_config` on + each run (an O(1) dict lookup in an immutable per-agent table); callables + never live in state or configuration (`validate_immutable`), and shared + mutable caches are banned by the free-threading rule in AGENTS.md. + +**D5. Domains are configuration.** An IGP domain is the set of IGP instances +that can form adjacencies. Four things decide it: + +- which interfaces each instance owns; +- area membership (interface and neighbour regexes); +- interface exclusions; +- ports, which are per-domain configuration with equal defaults. + +Consequences: + +- A device runs 0..k IGP instances with distinct `ClientId('openr', i)` and + disjoint interface sets; overlapping sets are rejected. More than one instance + is `netsim_ext.multi_instance` (a per-node setting, off by default), because upstream Open/R hard-codes FIB client + 786 (`Fib.h:392`), so two daemons would overwrite each other. Each instance + drops hellos on interfaces it does not own, and only one instance's + `srdb_view` backs the device's SR-DB. +- A device runs at most one BGP instance (VRFs stay deferred). +- A single-device domain is either an instance whose interfaces are not + discovered, or no IGP at all (connected, static and BGP routes only). +- Area `"0"` matches any area during negotiation (`Spark.cpp:2112-2121`), so + domains separated only by area id still peer if either side says `"0"`. The + deploy helpers never rely on area names for separation. + +**D6. Cost proportional to change is a performance exit criterion.** Functional +work starts once F0, F2a and F5 land (§8). Every protocol slice's performance +exit ships two benchmarks, as operation counts in CI and wall-time ratios in +slow runs: + +- a one-prefix change costs the same at 10× the table size, within a stated + factor; +- an idle run admits O(1) changed records (for example a small Spark liveness + record) and does no SPF, FIB or placement work, whatever the table size. + +A faithful Spark increments its sequence number on every hello, so "idle runs +return the same state object" is not the claim. Large tables are always `PMap`s +updated with `set`, never tuples or frozensets, which admission walks in full. + +**D7. Timing fidelity through defaults and presets.** Defaults are faithful: +Open/R's `OpenrConfig.thrift`, FRR's `traditional` and `datacenter` profiles, +and the `sonic` preset. The `lab` preset shortens timers but keeps the order of +events. Protocol presets and dialects require `run_delay > 0`, so feedback +between agents on one device can never run away inside one timestamp. Jitter comes from per-module RNG streams +and is deterministic per seed. + +**D8. Next hops and sessions follow FRR and zebra.** + +- **Single-hop eBGP** (TTL 1, no `disable-connected-check`) registers NHT with + `connected_only`. Connection attempts wait for that answer + (`bgp_fsm.c:2704-2717`). The route installs non-recursive: interface plus + address, taken from the NHT legs. +- **Recursive install.** iBGP, confed, multihop and GTSM eBGP routes install + recursively: address-only next hops resolved by F1's hierarchical resolver. + BGP-over-BGP is allowed (`research-frr-decision.md` §3.4, §3.8, §5). +- **Confed peers** default to TTL 1, like eBGP (`peer_ttl_update`, + `bgpd.c:2532-2552`; in 10.5.4 the same rule is inline in `peer_create`, + `10.5.4:bgpd.c:2098`, and `peer_as_change`, `10.5.4:bgpd.c:2201-2205`). They get no connected check, install recursively, and use + distance 200. Loopback confed peering more than one hop away needs + `ebgp-multihop`. +- **Fast external failover** is on by default. An interface going down stops + every peer on it with TTL 1 or `ttl-security hops 1` (`bgp_zebra.c:286-299`; + `10.5.4:bgp_zebra.c:273-286`). + iBGP and multihop sessions survive the loss of their next hop until the hold + timer expires, unless `bgp fast-convergence` is set. +- **IGP metric.** The decision process uses the metric of the installed + resolving route (`NhtResult.via_metric` under `mode='installed'`, F0), + matching zebra's NHT (G9). + +**D9. Keys are ready for add-path and VRFs.** Paths are keyed +`(peer, path_id)` and destinations `(vrf, afi, safi, prefix)` from day one, +with `path_id = 0` and the default VRF until those features land. + +**D10. No analytical BGP in this phase.** Import policy, decision and export +stay pure, so a later timeless solver (C-BGP style, bounded iterations with +non-convergence detection) can reuse them for Monte Carlo studies. + +**D11. Configuration is tree state; commands are exec-only.** + +Users change settings through the layered settings system (section 2A). It +resolves per device and writes the results with the primitive below. + +- **In-place configuration.** `Network.configure_agent(device, name, config)`, + also exposed as `Simulation.configure`, replaces `AgentNode.config` within the + same generation in one `update()`. + - It adds `Cause(CAUSE_CONFIG)` and is validated at submission by the + plugin's pure `validate_config`. + - Reset and fresh runtimes keep the latest configuration. + - Plugins read configuration only from `ctx.agent_config`, enforced by an + invariant test. The reference agent's `self.config.params` read is fixed to + match. +- **Persistent state.** `AgentNode.persistent` is an immutable record that + survives reset unless purged. It models Open/R's PersistentStore: drain, + overload and metric overrides. +- **Commands.** `Command`s are exec-only operations, such as clear, soft + in/out, route refresh and the Open/R drain APIs (which write `persistent` + through `AgentOutput.persistent`). + - Each is validated synchronously by the plugin's pure `validate_command` at + `Simulation.command`. + - Errors at run time are reported in a view, never raised, so a bad command + cannot stall retry. + - Commands are control-class inbox entries. A command addressed to a + generation reset before it runs is dropped and counted. +- **Policy edits.** A validated definition is active for new inputs at commit. + The owner schedules reevaluation of existing routes by + `policy.reevaluation.mode` and `.delay_s`, the `REEVALUATE` apply action of + section 5.8. In the FRR dialects the default delay is the 5 s + `route-map delay-timer` (`bgpd.c:9561`; `bgp_routemap.c:5191-5194`), and + FRR's delay 0 means manual. + +**D12. Restart semantics are explicit.** + +- `reset_agent(purge=True)` models a plain daemon restart: zebra removes a + disconnected client's routes. +- `reset_agent(purge=False)` models retention: Open/R's FibService keeps routes + until the first `syncFib`, and BGP graceful restart (BG8). +- Open/R persisted drain state survives either reset. +- FRR's default GR helper mode advertises the capability without address + families (`bgp_open.c:1684-1745`), so FRR-to-FRR sessions flush routes on + loss. Deferring GR is therefore safe for default configurations, but not for + SONiC ToR routers, whose template enables GR. That is a recorded divergence + of the `sonic` preset until BG8 (section 2B.4). + +**D13. Routing policy is one shared, typed engine in the model layer.** It +adopts the reviewed routing-policy plan (`NetSim/docs/plans/routing-policy.md`, +revision 4) with the integration corrections of section 5.8: +- FRR `ip protocol route-map` maps to a candidate-next-hop **activation + filter** in the resolver, evaluated once per original activated next hop + (§15.4, §16.4), not to submission-time admission and not to `FIB_INSTALL`; +- FRR `ip nht route-map` maps to `RESOLUTION_IMPORT(scope='nht')` with no + fall-through. + +BGP, Open/R and the device hooks all use this one evaluator; BGP keeps only an +access adapter. + +**D14. Forwarding installation is an explicit asynchronous contract, F8.** +Desired and installed actions are separate, with generation-guarded completions +at a named boundary. FIB acknowledgement gating and inactive-route export are +first-class, per-node behaviours (section 5.13). + +--- + +## 2A. Settings: per-node overrides, dialects and presets + +**Requirement.** Every setting can be overridden per node. Behavioural +differences between FRR releases that are generic and reasonable become +settings with per-dialect defaults; protocol code never checks a version. +Maintainability comes first: the system must stay auditable as FRR moves. + +### 2A.1 The schema + +Each setting is declared once, in a fragment owned by its consumer package +(`netsim/agents/bgp/settings.py`, `.../openr/settings.py`, +`netsim/model/device_settings.py`). The engine `netsim/settings/` sits in the +model layer and never imports agents. A composition root assembles the +catalog from the fragments by explicit import, with no runtime registration. + +```python +Setting(key='bgp.bestpath.med.missing_as_worst', type=bool, + scopes=(INSTANCE,), # where an explicit value may be written + default=DefaultsSpec(False), # joint first-match entries on (dialect range, profile) + since=None, # first dialect that has the knob; the resolver rejects it earlier + unit=None, # override unit (e.g. 'bgp.timers' covers keepalive and hold) + apply=RECALC_BESTPATH, + refs=('FRR bgp_vty.c `bgp bestpath med missing-as-worst`', 'RFC 4271 §9.1.2.2(c)'), + doc='Treat a missing MED as the worst value instead of 0.') +``` + +**Key grammar.** Keys carry object indices: +- `device.fib_delay`; +- `zebra.nht.resolve_via_default`; +- `bgp.timers`; +- `bgp.af[ipv6-unicast].nexthop_prefer_global`; +- `bgp.peer_group[FABRIC].timers`; +- `bgp.neighbor[10.0.0.1].af[ipv4-unicast].allowas_in`; +- `openr[dc1].spark.hello_time_s`. + +FRR's global `bgp …` commands map onto instance scope, since each device runs +one instance. + +**Defaults.** A `DefaultsSpec` is FRR's `FRR_CFG_DEFAULT` form: an ordered list +of `(dialect range?, profile?, value)` entries, where the first match wins. One +joint list covers entries such as `BGP_IMPORT_CHECK`, which needs "traditional +and version < 7.4". Gates below 10.5 are dropped, because no NetSim dialect +predates it. + +**Availability.** `since` and `allowed_values_by_dialect` record when a knob or +value exists. The generic resolver rejects out-of-range writes, so protocol code +never sees a dialect. + +**Override units.** A unit such as `timers` (keepalive and hold) or +maximum-prefix (limit, threshold, restart) is written whole. A partial write is +a validation error, as in FRR's CLI. Structured and list values (Open/R regex +lists such as `static_interface_metrics`, route-map terms, flavor specs) are +replaced, never merged. + +**Apply actions** follow modern FRR in every dialect: + +| Action | Meaning | +|---|---| +| `LIVE` | Takes effect on next use | +| `NEXT_SESSION` | Takes effect at the next session (values negotiated at OPEN) | +| `RECALC_BESTPATH` | Re-runs best-path selection (`bgp_recalculate_all_bestpaths`) | +| `SOFT_IN` | Re-applies inbound policy | +| `SOFT_OUT` | Re-runs outbound policy under duplicate suppression; a forced re-send happens only on `clear soft out` or ROUTE-REFRESH (AP-36) | +| `REEVALUATE` | Policy edits: the definition is active for new inputs at commit; the owner schedules reevaluation of existing routes by `policy.reevaluation.mode` and `.delay_s` (section 5.8). Replaces the former fixed 5 s `DELAYED` | +| `SESSION_RESET` | Resets the session. Keys that change the peer sort (remote-as, local-as, confederation peers) also re-derive the TTL (AP-19) | +| `CAPABILITY_ELSE_RESET` | Signals a CAPABILITY message when dynamic capability was negotiated, otherwise resets (BG2 models this) | +| `RESTART_INSTANCE` | Restarts the protocol instance | +| `DEPLOY_ONLY` | Read only when an object is created, as FRR's `peer_new` snapshot of `bgp default dynamic-capability` does (`bgpd.c:1836`) | + +**Upstream constants** that are not user settings (for example Open/R's +hard-coded throttles) live in a per-package `constants.py`, each with refs. + +### 2A.2 Dialects and presets + +**Dialects** are immutable pins that supply defaults and availability: +- `frr-master-f1d7bec`, with the alias `frr-modern`; +- `frr-10.5.4`, with the alias `frr-10.5`; +- `native`: today's NetSim defaults. It applies to devices with no FRR agent and + keeps current fingerprints unchanged. For example, NetSim's + `ResolutionPolicy.resolve_via_default` is False while FRR traditional defaults + zebra's to True. + +An alias is resolved to its pin when a layer is stored, so a stored scenario +keeps its behaviour when the alias moves, as FRR's `frr version` line does. +There is no `frr-10.7` dialect: every master-only delta is a fix or out of +scope, so its column would equal `frr-modern`. + +**Profiles** are FRR's `frr defaults`: `traditional` or `datacenter`. + +**Presets** are ordered named bundles of meta-keys and values: +- `sonic` (section 2B.4); +- `lab`, with short timers that keep event order, which gives D7 a home. + +**Meta-keys.** `device.dialect`, `device.profile` and `device.presets` are +device-scoped and resolved first, through the user layers only. +- A device has a single dialect, shared by its zebra and BGP. +- Changing the dialect is `RESTART_INSTANCE` (a software upgrade). + +### 2A.3 Layers and precedence + +Layers are immutable records in the tree (`NetworkState.settings`), so forks, +studies and resets carry them. From lowest to highest: + +1. the network layer; +2. group layers, each with a selector (a device-name regex and/or labels from + `DeviceState.labels`, a new F0 field that adapters fill from NetGraph + attributes and roles) and a priority; +3. the device layer. + +**Resolution per device:** + +1. Resolve the meta-keys (dialect, profile, presets) from the layers. +2. For each (object, key), the explicit value comes from the highest layer that + writes that exact (object, key). Layer order only ranks writers of the same + (object, key). +3. FRR inheritance runs over objects, not layers: instance → AF → peer-group → + neighbor → neighbor-AF (section 2B.5). A value set on + `peer_group[FABRIC].timers` by a group layer therefore wins over an + instance-level `timers` from the device layer for FABRIC members, as FRR + templates behave. +4. Absent explicit values fall back to presets, then the `DefaultsSpec` for the + device's dialect and profile. + +**Validation.** +- A layer that targets an object absent from a device is a validation error. +- Two equal-priority layers that write the same (object, key) on the same device + are rejected by the `update()` that creates the conflict: adding the layer, + adding the device, or changing its labels. + +### 2A.4 One write path; resolution output + +**The resolver is the only writer of:** +- `AgentNode.config`, meaning the whole `AgentConfig`: the runtime fields + (`run_delay`, `processing_delay`, inbox, queue and byte limits, + `listen_ports`) as well as `params`; +- the settings fields of `DeviceConfig`. + +`enabled`, `enabled_since` and `seed` are not settings; operations and failure +sources own them (`failures.py:207`). `Network.configure_agent` is internal to +the resolver. `add_agent(device, BgpAgent(BgpConfig(...)))` is sugar that writes +a device layer and object layers. A `debug_validate` invariant checks that +re-resolving the layers reproduces the stored configurations. + +**Output** is typed frozen records per consumer module (`BgpTimers`, +`DecisionSettings`, `NeighborSettings`, `SparkSettings`): +- when a new record equals the device's previous one, the previous object is + reused; +- hashable leaf records are interned in a dict local to one resolution call (no + module-level table, per the free-threading rule; `PMap` is unhashable, + `state.py:222`); +- validation uses `validate_admitted(new, old)` (`state.py:735`). + +**Provenance** is computed on demand, by re-resolving one device with tracing: +`net.settings.explain('leaf7', 'bgp.neighbor[10.0.0.1].timers')` returns the +value and its chain of layers. No side map is stored. + +**Change.** A layer change re-resolves the affected devices in the same +`update()`. It delivers `CAUSE_CONFIG` with the changed keys (F2a), and the +timeline records the settings diff (F1 owns `timeline.py`). The agent receives +one per-owner action plan compiled from the effective-record diff (§15.5), not +a set of per-key actions. + +### 2A.5 Rules enforced by tests + +1. Resolved-record classes declare no field defaults (AST check). Upstream + constants live only in `constants.py` files with refs. +2. Resolved records carry no dialect, so protocol code cannot branch on one; the + rule holds by construction. +3. Every setting has scopes, a type or validator, a unit if it belongs to one, + an apply action, refs and a doc string. +4. Setting keys and resolved-record fields are in a declared bijection (checked). +5. A new setting needs evidence that it changes behaviour, and a test marked + `@covers('key')` that exercises its values. A meta-test fails for uncovered + keys. Spec-valued keys are exempt from value enumeration. +6. `docs/settings.md` and the dialect table are generated from the catalog. + CI fails if they are stale. +7. The FRR lane checks whole-dialect behaviour. + - Golden source for `frr-modern`: `quay.io/frrouting/frr:master` pinned by + digest and mirrored. + - `:10.7.1` is informative, with a known-difference list (AP-01, AP-19, + AP-23, AP-36). + - `:10.5.4` covers `frr-10.5`. + - Avoid 10.5.0–10.5.3 and 10.6.0, which carry the IPv6 neighbour-address + tie-break bug (D6). +8. Adding a dialect means a new pin with data rows justified by the + per-subsystem diff review used for 10.5 → modern + (`netsim-protocols-research/frr-diff-*.md`). + +### 2A.6 Adapters write layers + +| Source | Becomes | +|---|---| +| NetGraph `attrs.netsim` | Labels plus device layers. The existing keys `fib_delay` and `fast_failover` (`netsim/adapters/ngraph.py:181-183`) move into `device.*`, and the old spellings stay accepted. Risk-group mtbf/mttr are study parameters, not settings | +| SONiC CONFIG_DB | The `sonic` preset, plus role-conditional device and object layers written by the adapter: confederation on disaggregated T2, FrontEnd `redistribute connected` route-maps, GR on ToR, GR disabled with LLGR on UpperRegionalHub | +| `frr.conf` | Device and object layers. `frr version` selects the dialect pin, and versions below 10.5 are rejected. `frr defaults` selects the profile. Dumping writes every explicit value and its inheritance or unset intent (an explicit value equal to a default is still written), so the adapter round-trips and a later default change cannot alter a pinned value | + +Scenario files with migrations are deferred until a stored format exists. + +### 2A.7 What is not a setting, and what is only linted + +- **Not settings:** + - oracle IGP domains (`Network.sources`, off-tree route sources); + - Simulation and transport globals (`max_rounds_per_timestamp`, retention); + - failure parameters (studies); + - Open/R drain and overload, which are `AgentNode.persistent`, changed by + commands; + - device `enabled`, `enabled_since` and `seed`. +- **Client profiles are per node** (routing-policy plan P0; + `RFCs/PER-NODE-SETTINGS-PLAN.md`). Today `Network.profiles` is keyed only by + `ClientId`, and `add_agent` rejects the same client with a different distance + on another node (probe reproduced by the policy research). Identity stays + shared, for attribution, policy matching and metrics. The behavioural + defaults become node settings: `rib.client[/].distance`, + `.protocol_origin` and `.link_state`. The network registry keeps identity + plus default templates only. Explicit per-row distances still win. +- **Routing policy is settings too.** + - Definitions and named sets are immutable, revisioned values + (`policy.definition[]`, `policy.set[]`), pinned by content + digest. + - Bindings are keys per attachment point, for example + `bgp.neighbor[10.0.0.1].af[ipv4-unicast].policy.import`, + `rib.admission[]` and `resolution.nht[]`. + - Each resolves to exactly one `RoutePolicyBinding`. Its value is replaced + whole, never merged. + - `policy.reevaluation.mode`, `policy.reevaluation.delay_s` and + `bgp.policy.on_replay_unavailable` are per node. +- **Consistency lints (warnings, not constraints):** + - Open/R ports per domain; + - confederation identifier and peer lists across members; + - RR cluster ids; + - dialect mixing across peers, which is allowed and realistic. +- **Mapped settings.** + - `zebra.nht.resolve_via_default` drives both NHT registrations and + `ResolutionPolicy`. + - `zebra.multipath_num` caps FIB groups, replacing the fixed + `max_ecmp_paths=64`, which truncated FRR's packaged 256 and SONiC's 514. + Values: `native` 64, `frr-modern` 256, `frr-10.5` 514 for SONiC. + +### 2A.8 Failure modes of settings systems, and the countermeasures + +| Failure mode | Countermeasure | +|---|---| +| Setting sprawl: knobs nobody tests | Rule 5 (evidence plus `@covers`); rule 4 (bijection); combinations are covered by dialect scenarios, not pairwise | +| Precedence surprises | Per-(object, key) writer ranking, FRR object inheritance, conflicts rejected at `update()`, `explain()` provenance, settings diffs in the timeline | +| Defaults drift from the reference | Defaults exist only in `DefaultsSpec` entries with refs into pinned trees. A dialect is a new pin plus diff rows, and the documentation is generated | +| Version checks spread through code | Resolved records carry no dialect (rule 2) | +| A second write path reverts or bypasses settings | The resolver is the only writer, and `debug_validate` checks re-resolution | +| Per-node configuration costs memory and validation at 10k devices | Record reuse when equal, per-call interning, `validate_admitted`, re-resolution of affected devices only | +| Run-time change semantics re-implemented ad hoc | Apply actions declared per key, one test per apply class, modern FRR semantics in every dialect | +| Inheritance differs from FRR | Section 2B.5 rules, tested through the `frr.conf` adapter and the lane | +| Hidden global state and free-threading contention | Immutable fragments, a composition-root catalog, per-call interning, frozen outputs | + +### 2A.9 Where it lives and who builds it + +- **F0** adds: + - the `NetworkState.settings` stub, `DeviceState.labels`, and the + `Network.settings` API skeleton; + - the `Command`, `CAUSE_CONFIG` and `persistent` records. +- **F7** builds: + - the engine (`netsim/settings/`: schema, layers, resolution, inheritance, + provenance, documentation generators); + - the composition root; + - the `native` dialect with the `device.*` and `zebra.*` fragments; + - NetGraph label translation. + + Existing behaviour moves onto it first, with fingerprints unchanged. +- **F2a** delivers `CAUSE_CONFIG` with changed keys; **F1** adds timeline + settings events. +- **BG1 and OR1** own their fragments, units, polarity and Open/R regex scopes. +- **BG7** writes the SONiC and `frr.conf` layers. Each later slice adds the keys + it implements (the §2B.3 rows name their slices). + +--- + +## 2B. FRR versions: 10.5 versus modern, and what becomes a setting + +Source: three per-subsystem diffs of frr-10.5.4 against master `f1d7bec`. Each +difference is attributed to its release through the full history +(`~/ws/frr-git`) and checked by function-level code comparison, not by commit +hashes: +- `netsim-protocols-research/frr-diff-decision-nht.md`; +- `netsim-protocols-research/frr-diff-advertise-policy.md`; +- `netsim-protocols-research/frr-diff-sessions-defaults.md`. + +About 1,330 bgpd, 550 zebra and 390 lib commits separate the two trees. + +### 2B.1 Findings + +- **The core is unchanged from 10.5.4 to master.** The following all match: + - best-path steps 1–15, and sorted-list insertion (in FRR since 10.1); + - the 50 ms bgpd and 10 ms zebra queue holds; + - install flags, distances, zebra recursion, the connected check, fast + external failover; + - the RR matrix and attribute writing, next-hop rules, confed checks, + RFC 8212, communities; + - coalesce, MRAI defaults, collision, timers and the route-map delay. +- **About 60 differences are fixes, where 10.5 is simply wrong.** Examples: + - a martian NEXT_HOP resets the session instead of treat-as-withdraw + (RFC 7606); + - a GR-stale path with a dead next hop stays best; + - LLGR-stale routes leak; + - link bandwidth is clamped at 34 Gbps; + - `maximum-paths 1` reinstalls on every run; + - TTL and local-as edge cases; + - duplicate UPDATEs after policy changes; + - BFD strict mode delays teardown by 30 s. +- **A few are genuine behaviour choices** (section 2B.3). +- **Master-only semantic changes:** + - withdrawals before announcements in one UPDATE (a fix); + - TTL re-derivation on every sort change (a fix); + - non-transitive extended communities stripped toward eBGP (a fix); + - no forced re-send after a policy edit (a fix); + - UPA, out of scope. Its receive-side marking is unconditional in master, so + `frr-modern` diverges on UPA-marked paths (recorded); + - the rs-client OTC check, out of scope until RFC 9234 roles; + - the zebra tie-break inversion (Z5), whose code contradicts its commit + message. +- **Corrections.** + - `bgp deterministic-med` stopped affecting best path in 10.4.0 (83ad94694, + #18210; also 10.2.2 and 10.3.1). + - "Withdrawn first" is master-only; "MRAI gates withdrawals" holds from + 10.6.0. +- **Conflicts between the reports, resolved:** + - N1 (a fix) over S24 (proposed as a setting): the GR-stale deletion is + followed; + - the suppress-fib-pending delay range is 0–10000 ms (`bgp_vty.c:2122`), not + 0–1000; + - AP-20 over S18: a member's remote-as override is accepted, and the + `frr-10.5` adapter warns. + +### 2B.2 Target, quarantine and quirks (revises D1) + +- **`frr-modern`** is master `f1d7bec`. + - Only Z5 is quarantined: the zebra tie-break defers to NetSim's F1a rule + until a black-box run on the pinned master image decides. + - All other master deltas are fixes and are followed. + - The pin moves at 10.8.0 by repeating the diff review. +- **`frr-10.5`** reproduces SONiC's release defaults and generic behaviour + choices, never its bugs. The 10.5 bugs are documented divergences. When a + 10.5.4 lane run disagrees on one of them, triage cites the divergence list. +- **Quirks.** A `quirk.*` flag is allowed only when all of these hold: + - it materially changes converged state or timing; + - it ships in a release users run; + - a test pins it. + + Quirks default to off everywhere; a preset may enable one with a stated + reason. There are currently no candidates: + - N6 (`bandwidth ignore` falls through to link-bandwidth weighting, since + 10.6.0) contradicts FRR's documented plain ECMP, so a quirk for it would + be off in all dialects. It is revisited with UCMP in BG8; + - the 10.5.4 link-bandwidth clamp fails the rule, because SONiC's template + configures no link bandwidth. + +### 2B.3 Behaviour choices that become settings + +Only true choices carry per-dialect defaults. Knobs that exist only in newer +releases are expressed through `since`, not as dialect columns. + +| Key | Values | `frr-10.5` | `frr-modern` | Scopes | Apply | Slice | Rows | +|---|---|---|---|---|---|---|---| +| `bgp.bestpath.redist_admin_distance` | bool | False | True | instance (NetSim knob; FRR has none) | RECALC_BESTPATH | BG3 | D1 | +| `bgp.out.mrai_withdrawals` | `bypass`, `gated` | `bypass` (writes at once unless the peer's MRAI timer is already running, `10.5.4:bgp_packet.c:448`) | `gated` (RFC 4271 §9.2.1.1) | instance (so it stays out of the update-group key) | LIVE | BG4 | AP-35, S03 | +| `bgp.bestpath.deterministic_med` | bool | profile (dc True, trad False) | profile | instance | RECALC_BESTPATH | BG3 | D3 | +| `bgp.bestpath.dmed_semantics` | `frr-current`, `classic` | `frr-current` | `frr-current` | instance | RECALC_BESTPATH | BG3 | D3. `classic` means pre-10.1 FRR: grouping, and only group winners compete, when `deterministic_med` is True | +| `bgp.gr.select_defer_time_s` | seconds | 240 | 120 | instance | LIVE | BG8 | P4, S01 | + +Knobs with `since=frr-modern`, following the FRR scope and default: + +| Key | Default | Scope | Slice | +|---|---|---|---| +| `bgp.af[ipv6].nexthop_prefer_global` | False | AF | BG3 | +| `bgp.neighbor[*].af[*].allowas_in.route_map` | None | peer-group, neighbor-AF | BG4 | +| `bgp.advertisement_delay_s` | None | instance | BG4 | +| `bgp.suppress_fib_pending.adv_delay_ms` | 1000; range 0–10000, fixed at 1000 in 10.5 | instance | BG4 | +| `bgp.use_underlay_nexthop_weight` | False | instance | BG8 | + +Knobs with the same default in both dialects: + +| Key | Default | Scope | Slice | +|---|---|---|---| +| `bgp.suppress_fib_pending` | off; on in `sonic`; maps to the `frr-suppress-fib-pending` gate profile of section 5.13 | instance | BG4 (after F8) | +| `bgp.export.inactive` | True in FRR dialects (inactive best paths are advertised unless suppress-fib-pending), False in `native` | instance | BG4 | +| `bgp.suppress_duplicates` | True, modern semantics (AP-33) | instance | BG4 | +| `bgp.ucmp.link_bandwidth_mode` | `default` | instance | BG8 | +| `bgp.default.dynamic_capability` | profile (dc True, trad False); DEPLOY_ONLY | instance | BG2 (with CAPABILITY messages) | + +`zebra.rib_tiebreak` (Z5) is not a key. F1a records NetSim's tie rule for equal +distance and metric (today: `ClientId` order, `routing.py:225`). An FRR arrival +order ("oldest" or "newest") would need a per-row sequence in the RIB, so it is +added only if the black-box run shows the difference matters. + +**suppress-fib-pending changes converged state, not only timing.** When a +selected path is not FIB-installed (`FAIL_INSTALL`, `BETTER_ADMIN_WON`), bgpd +withdraws it from peers (`bgp_check_withdrawal`; `bgp_route.h:771-806`; +`bgp_zebra.c:3128-3160`; identical in 10.5.4). Example: a redistributed /31 +that loses to the connected route is never re-advertised. A selected `network` +static is exempt unless another non-selected non-NORMAL path exists at that destination (`bgp_route.h:779-801`). It uses the F0 owner notifications. + +### 2B.4 Profile defaults and the `sonic` preset + +| Key | datacenter | traditional | +|---|---|---| +| keepalive / hold | 3 / 9 s | 60 / 180 s | +| connect retry | 10 s | 30 s | +| `ebgp-requires-policy` (RFC 8212) | False | True | +| `deterministic-med` | True | False | +| `suppress-duplicates` | True | True | +| `graceful-restart notification` (RFC 8538) | True | True | +| `hard-administrative-reset` | True | True | +| `enforce-first-as` | True | True | +| `network import-check` | True | True | +| dynamic capability | True | False | +| link-local capability; RR allow-outbound-policy; `bestpath aigp`; IPv6 `nexthop prefer-global` | False | False | +| zebra `ip nht resolve-via-default` (per VRF) | False | True | + +Values are unchanged 10.5.4 → master (`FRR_CFG_DEFAULT_*`). + +**`sonic` preset**, verified against SONiC master: +- **Profile: traditional.** The generated `frr.conf` has no `frr defaults` + line, the daemons start without `-F` (`supervisord.conf.common.j2`), and + FRR's packaging does not enable datacenter. +- **Dialect:** `frr-10.5` (FRR 10.5.4, `rules/frr.mk`). +- **Values** (`bgpd.main.conf.j2`, saved in the research folder): + - `bgp suppress-fib-pending`; + - `no bgp ebgp-requires-policy`; + - `no bgp default ipv4-unicast`; + - `maximum-paths 64` (IPv4 and IPv6) when `constants.bgp.maximum_paths` is + enabled, capped by `MULTIPATH_NUM` 514; + - `bgp bestpath as-path multipath-relax` when enabled. +- **ASIC-offload notification.** zebra runs with + `--asic-offload=notify_on_offload`, so INSTALLED follows ASIC programming. + `device.fib_delay` models that delay under suppress-fib-pending. +- **Role-conditional lines** are written by the SONiC adapter as device layers: + - GR on ToR routers when enabled (restart-time 240 s, select-defer 45 s, + preserve-fw-state); + - `coalesce-time 10000` on dual-ToR; + - UpperRegionalHub `graceful-restart-disable` with LLGR stale-time 864000; + - confederation on disaggregated T2; + - FrontEnd `redistribute connected` route-maps. +- **Recorded divergence until BG8.** GR is not modelled, so where SONiC enables + GR (ToR), T1 helpers retaining ToR routes is not reproduced. + +### 2B.5 Mirroring FRR's inheritance machinery + +From `frr-diff-sessions-defaults.md` §5–§6: + +| FRR | NetSim rule | +|---|---| +| Override bits (`flags_override`, `af_flags_override`, `filter_override`) with invert polarity for default-on flags | Each object stores only its explicit values; polarity is a property of the key | +| `no` on a member restores the inherited value (`peer_flag_inherit`, `bgpd.c:1153-1170`); group edits skip overriding members | Removing a layer's explicit value restores inheritance; group values never override explicit member values | +| The override unit is the flag group (`timers` covers keepalive and hold; maximum-prefix covers limit, threshold, restart) | Units are written whole (2A.1) | +| Role, GR mode and BFD are copied group → member without override tracking (last writer wins) | An explicit member value always wins (recorded deviation) | +| One group per neighbor; a member remote-as override is allowed since 10.7.0 | One group per neighbor; the override is accepted in every dialect, and the `frr-10.5` adapter warns | +| Instance defaults applied at creation; capability flags copied into peers at creation | `DEPLOY_ONLY` apply action for those keys | +| `peergroup_flag_check` reads the member's override bit rather than its effective flag (link-local capability, strict BFD) | Use the effective value (recorded FRR quirk) | +| `DFLT_*` resolved against the configuration's `frr version`, `SAVE_*` against the running version | Dialect pins resolved at storage; dumps write explicit values and inheritance intent, never only differences from defaults | + +--- + +## 3. Foundation + +### F0. Contract additions (integrator; records, defaults, stubs) + +All additions go in `netsim/model/contracts.py` (and `network.py` for +`configure_agent`). Each has a default, so existing agents are unaffected. + +- **Configuration, persistence, commands.** As D11: + - `Network.configure_agent`; + - `CAUSE_CONFIG`; + - `AgentNode.persistent` and `AgentOutput.persistent`; + - `Command(time, name, payload, generation)`; + - `Simulation.command`; + - plugin hooks `validate_config` and `validate_command`. +- **Views.** `AgentOutput.views: PMap[str, immutable]` is stored in `AgentNode` + and compared by identity. + - `ctx.view(agent, name)` reads another agent's view on the same device. + - Subscriptions may target `('agents', name, 'views', key)`. + - The timeline records view changes; tests, `Timeline.series` and study + stability predicates read views only. + - Open/R and BGP publish bounded show-records: neighbours, adjacencies, + sessions, best path and reason per prefix, command results. +- **RIB feed** (zebra `REDISTRIBUTE_*` analogue). The op is + `WatchOp(af, sources, kind)`. + - Watches are stored in `DeviceState` as (owner, af, sources, kind). + - `sources` is a set of `ClientId`s or `ALL`. `ALL` excludes the owner's own + client, and naming it is rejected, because zebra has no redistribute-self. + - `kind='selected'` covers rows the resolver installed. The FIB kind emits + `RouteChange(watch, af, epoch, upserts, removals)` when it commits + processed epoch `e`. The changed set is the prefixes whose FIB entry + changed, plus installed prefixes whose contributing rows changed content + (from `rib_apply`'s changed set), so metric-only and tag-only changes reach + the feed: + - upserts are the `Route` objects of `FibEntry.contributing` from the RIB + that run consumed, filtered to the watched sources; + - removals are keys that left the set; + - nothing is emitted while PENDING. + + This matches zebra, which redistributes after dataplane success. Rows carry + `tag`, `metric` and `distance`, so injection policies can match and set tags + against two-border redistribution loops. + - `kind='all'` covers every row of the watched sources whatever its + selection. It emits at the RIB commit of those rows (input epoch), + independent of FIB processing. It serves Open/R-style injection, which + upstream receives through thrift `advertisePrefixes` whatever the RIB + selected. + - A snapshot is delivered on registration and on every generation change. + - `RouteChange` is a control-class entry. At most one uncaptured entry exists + per watch, coalesced latest-state-wins and never merged into a captured or + parked prefix. + - Rows taken from the tree skip re-validation. +- **Owner notifications.** `ctx.route_notifications` holds + `RowNotice(key, status, reason, zapi)` for changed own rows. + - It carries NetSim's native status and reason (`INSTALLED` or + `NOT_INSTALLED`, with `SHADOWED`, `UNRESOLVED`, `AMBIGUOUS_ACTION` or + `LOOP_DETECTED`), plus + the derived ZAPI code: `INSTALLED`, `FAIL_INSTALL`, `BETTER_ADMIN_WON` or + `REMOVED`. + - Metric-only row changes count as changes. +- **NHT.** + - `NhtKey.mode ∈ {'rib', 'installed'}`. `installed` answers from the last + processed FIB (zebra NHT accepts only installed next hops), and F1's + recursive-next-hop table backs it. + - `NhtResult.via_metric` is the metric of the selected resolving row at the + first level (zebra `nhr->metric`), next to the strict `cost`. + - Notifications fire only on answer fields: `eligible`, `via_prefix`, + `via_source`, `via_metric`, `cost`, `legs`. They no longer fire on + `queries` or `interfaces`, which are dependencies. + - An exact-prefix mode serves BGP `network` import-check (zebra IMPORT_CHECK): + the prefix must be present exactly and resolved by a row that is not from a + BGP client (`bgp_nht.c:699-714`, `nhr->type == ZEBRA_ROUTE_BGP`). +- **Change-key semantics.** `ctx.nht_changes` and `ctx.route_notifications` list + the keys captured by this receipt. + - That means the union since the generation's previous published run, kept + on rejection. A key may have changed back since. + - Values are read from the snapshot. + - `on_init` receives empty lists and reconciles from `ctx.nht` and + `ctx.rows_of(own client)`. +- **Route views.** `RouteView` is split into `rib_selected` (first resolvable + group at the input epoch; zebra `ZEBRA_FLAG_SELECTED`) and `installed` + (processed epoch, `None` while PENDING). This replaces today's field, which + reads the installed FIB entry even while the row is PENDING + (`agents.py:521-525`). +- **Transport.** + - Wildcard listeners use `Endpoint(address=None)` for a family, scoped to an + interface set so that two instances never collide. + - Each listener carries an immutable accept table keyed by remote address (a + prefix for BG8 listen ranges) → `(ttl, min_ttl)`. A remote not in the table + is REFUSED, as FRR's `bgp_accept` closes unconfigured peers. + - `SessionOp.ttl` and `SessionOp.min_ttl` apply to the open. + `SessionOp('set_opts', connection, ttl, min_ttl)` changes them later. + - `ConnectionView.local` always reports the concrete local endpoint. +- **Lazy accessors, all O(1) over immutable records.** They are built from the + detached `LocalContext` (`nht.py:57-122`), never from `DeviceState`, which + holds other agents' state, and they keep no mutable cache (the context must + pass `validate_immutable`, `tests/runtime/test_agents.py:213-259`). + - `ctx.rows_of(af, client)` returns the existing + `RibState.clients[client]` map. + - `ctx.route(af, prefix)` reads through `prefixes`. + - `ctx.row_status(af, key)` reads the outcome. + - `ctx.rib_view(af)` keeps its meaning and is computed on demand. +- **Records F1 and F2 share.** + - `ResolverOutcome.rows` becomes `PMap[ClientId, PMap[RowKey, RowOutcome]]`. + - NHT registrations are held in per-owner maps. + - `RecursiveNexthop` and group records. + - `FibGroupEvent` for the timeline. + + With per-client maps, G4 and G5 become identity checks per client, and + `RowNotice` becomes a `diff_pmap` of the client's sub-map, so F2 does not wait + for F1's incremental resolver. + +- **Routing policy, installation and client-profile records** (contracts only; + owners in section 8): + - `RoutePolicyBinding` storage in settings; + - the `RouteSubmission` helper signature; + - the F8 records (`ProgrammingRequest`, `ProgrammingResult`, desired and + installed action, `POLICY_EXCLUDED`); + - node-local client-profile settings; + - a regression test that fails today: the same client with distance 20 on + node A and 200 on node B. + +Exit: contract tests for every new record and rule; existing agents and tests +unchanged; the locality rules extend to every new accessor. + +### F1. Resolver: deterministic semantics, persistent FIB, then incremental + +Files: `netsim/model/routing.py`, `netsim/model/forwarding.py`, +`netsim/model/lpm.py`, `netsim/model/derive.py` (FIB kind, `bump_epochs`, NHT +refresh calls), `netsim/runtime/timeline.py`. It is split in three, with a +design-review gate before F1c. + +**F1a. Order-independent resolver semantics** (fixes G12, a defect today). + +- *Fingerprints first.* Extend the fingerprint oracle before touching anything + (`tests/test_fingerprints.py:55-67` covers only prefix, action, interface, + next hop and weight on oracle-IGP topologies). + - New fields: outcomes (status, reason), MAC, encap, adjacency AF, NHT + answers, selection and PENDING. + - New scenarios: recursive, cyclic, cross-family (RFC 8950), `lpm_fallthrough`, + `resolve_via_default` and SR. + - Record baselines and list F1a's intended diffs explicitly. +- *Rules.* Each prefix's answer becomes a function of the RIB only: + - depth is the length of the next-hop chain (kept per recursive entry), not + stack height; + - a depth-truncated result is never memoized; + - a prefix excludes only itself from its own resolution; + - recursion is decided per leg: a leg resolves when it has a finite witness + within the depth bound; a leg whose resolution closes a cycle with no + escape fails with `LOOP_DETECTED`; the route's other legs are unaffected + and weights are normalized over the survivors (§15.4). + + Each rule is pinned by a test that fails on the old code. The reproducers are + `netsim-protocols-research/scale-review/order_dep.py` and `order_dep2.py`. +- *Free fix.* `DeviceContext.rib` uses truthiness (`derive.py:728-729`), which + calls `RibState.__len__` twice per prefix, about 7% of resolver time. It + becomes `is None`. + +**F1b. Persistent FIB storage and a non-transitive FIB** (fixes G13, G14). + +- *Two-level next-hop groups*, as zebra and hierarchical FIBs use. + - A per-(device, next-hop family) table of recursive entries, + `RecursiveNexthop(af, address, interface, interface_generation, + connected_only, resolve_via_default)`: the physical resolution key carries + every resolution-changing scope, including the interface identity and + generation for link-local addresses (§15.4). Each entry holds the resolved + legs with a depth **per leg**, the resolving prefix and source, + `via_metric`, and the transitive dependencies, failed lookups included. + Owner and client projections (policy binding, query mode) are keyed + separately and applied per row, never to a shared group. + - A `FibEntry.depends_on` holds only its own prefix plus references to + recursive-entry ids; the transitive dependencies live on the recursive + entries. The 23 tests that read `depends_on` are updated. +- *Group key:* + - the route AF; + - the unresolved next-hop set with weights; + - the route's own prefix, only when it covers one of its recursive next-hop + addresses (so 10/8 via 10.1.1.1 and 20/24 via 10.1.1.1 do not share a group + while resolving differently). + + Group ids are stable and reference-counted with garbage collection, so churn + does not leak groups. Group content is the flattened legs, truncated to + `max_ecmp_paths` and normalized exactly as `_group_from_legs` + (`routing.py:819`) does now. Forwarding, placement and fingerprints stay + one-level. +- *Storage and diffs.* + - FIB entries move to persistent per-length `PMap` tables, as the RIB uses. + The lookup slowdown (117 vs 64 ns on the design record's numbers) is + measured against placement and trace. + - The unchanged-FIB check and the timeline diff use shard identity and + `diff_pmap`. +- *Timeline contract.* An underlay change alters only group content, so the + timeline gains `FibGroupEvent(device, af, group, nexthops)` and keeps a + per-prefix `FibEvent` only when the entry itself changes. This is a + documented change to `tests/runtime/test_timeline.py:141-143`. Expanding to + per-prefix events would be O(dependents): 50k events cost at least 150 ms. +- *NHT refresh* is selected through the prefix watch index, using each + registration's recorded queries, instead of refreshing every registration. +- *One RIB submission boundary.* Every production writer calls a single + model-owned `submit_rows(...)` that wraps `rib_apply`: + - `derive.py` connected, local and SR-local; + - `igp.py` sources; + - `entities.py` `RibClient`; + - `network.py` batch reconciliation; + - `runtime/agents.py` `RouteOp`s. + + `rib_apply` stays the pure table primitive. The boundary emits the + changed-row sets F1c, F2b and RP3 need, and an audit test fails on any new + direct `rib_apply` call from production code. +- *Prefix watch index* (zebra `rnh` longest-match watchers). It keeps sorted + watched addresses per family and answers "which watched addresses fall inside + a newly inserted prefix" by bisect. A per-length scan would cost O(watchers) + per insertion, about 5M checks in a 50k-prefix load with 100 watchers. + +**F1c. Incremental resolution with the full resolver as oracle.** + +- *Dirty sets.* + - `rib_apply` reports changed prefixes. + - Interface and neighbour changes reach direct legs through an index. + - The watch index invalidates recursive entries. + - A recursive entry that flips between resolved and unresolved (including + partial resolution under `all_or_nothing`) re-queues every dependent prefix, + because the prefix falls through to the next row group. + - Dirty sets are processed in sorted order, since `str`-hashed sets are + randomized per process. + - A measured fallback runs a full rebuild when the closure is large. +- *SR entries* stay on the full path first and migrate last. +- *Oracle.* The full resolver runs on every commit in F1 tests and in seeded + random RIB sequences, and is sampled at quiescence elsewhere. Running it + everywhere would roughly double FIB time across the suite. + +**Exit.** + +- F1a: fingerprint baselines and the intended diffs; new rule tests fail on the + old code. +- F1b and F1c: the resolver acceptance matrix of the design of record. +- Operation-count assertions in CI: `resolve_prefix` calls, entries touched, + NHT refreshes. +- Wall-time ratio benchmarks, slow and deselected, on both harnesses: + - `net.converge()`: with 50k recursive rows, the FIB step for an IGP change + goes from 1.80 s to ≤ 5 ms with O(1) entries touched, and a one-row add + from 1.71 s to ≤ 1 ms. + - Simulation: the FIB, timeline and NHT-refresh share of the one-row agent + run at 10k rows is ≤ 3 ms. The whole run at ≤ 5 ms (from 411 ms) is a joint + F1c + F2c exit. +- Existing converge benchmarks not more than 5% slower. +- The end-to-end BGP cost of a next-hop metric change stays O(paths on that next + hop): about 0.5 s at 50k paths. It is published in passport (b), not claimed + as a FIB target. + +### F2. Agent runtime at scale + +Three sub-slices, so functional protocol work is not blocked on scale work: + +- **F2a**: configuration, persistence, commands, views and change keys. It + follows F0 and runs in parallel with F3–F5. +- **F2b**: the RIB feed, owner notifications and the NHT `installed` mode with + `via_metric`. It needs F1b. +- **F2c**: the scale work for G2–G5 and admission. + +Files: `netsim/runtime/agents.py`, `validate_admitted` and the `PMap` builder +in `netsim/model/state.py`, NHT delivery in `netsim/model/nht.py`, and F0 +delivery. + +- **Lazy context** as F0 defines it. Interface views are cached in the runtime, + holding strong references to nodes, and never keyed by `id()`. +- **Per-client identity notifications** (G4, G5). F0's per-client outcome map + and per-owner NHT maps turn the scans into O(1) identity checks per client, + and `RowNotice` into a `diff_pmap` of the client's sub-map. Row content is + tracked, so a metric-only change counts. +- **Shard-identity admission** (G3). Published shard dicts are never mutated: + `set` and `remove` copy, the builder copies on first write and forgets its + shards on `build` (`state.py:371-429`), and `_wrap` owns only fresh dicts, + with its callers confined to `state.py`, `lpm.py` and `rib_apply`. The rule: + - pair new with previous only when `o._mask == prev._mask`, otherwise walk + everything (promotion from flat to sharded at 513 entries); + - skip a shard when `o._shards[i] is prev._shards[i]`; + - in a changed shard, trust an entry only when `prev_shard.get(k, MISSING) is + v` and `k` is an exact `int` or `str`, or an exact tuple of such leaves. + Otherwise validate both key and value; + - for `FrozenPrefixTable`, compare `o._tables[plen] is prev._tables[plen]`, + never `shards()`, which allocates fresh proxies (`lpm.py:88-93`). A + persistent prefix-table builder makes identity meaningful. + + Guard tests: + - the visited-node count is at most N/256 + c (the old code visits at least N); + - a mutable value in a changed shard raises; + - removing then re-adding a key as a tuple subclass with storage raises; + - promotion walks fully; + - `PMap(dict(old))` walks fully; + - a builder `set` after `build()` leaves the published shard unchanged. + + Protocol tables are nested (per peer, per area, per AF), so each map stays at + or below about 50k entries. A changed-key log emitted by the `PMap` builder is + the follow-up option for O(k) admission. +- **F0 behaviour implemented,** delivered through the inbox with the existing + bounds: + - configuration, persistence, commands, views and change keys (F2a); + - the RIB feed (`selected` at the processed epoch, `all` at the input epoch), + owner notifications, and the NHT `installed` mode with `via_metric`, read + from F1b's recursive-next-hop table (F2b). +- **`connected_only` parity.** FRR's connected check accepts a CONNECT or STATIC + resolving route, not LOCAL (`zebra_rnh.c:646-667`). This touches + `_CandidateResolver` in `routing.py`, so it is coordinated with F1. It is + pinned by test, and the install-time difference is reproduced: a + non-recursive route over a static stays inactive in zebra. + +Exit (operation counts in CI, wall-time ratios in slow benchmarks): + +- G2: an idle run at 50k rows goes from 130 ms to ≤ 0.3 ms. +- G3: one-entry admission at 100k goes from 236 ms to ≤ 100 µs, with the + visited-count ratio asserted. +- G4 and G5 become per-client identity checks. +- The Gate C agent suites pass on both interpreters. + +### F3. Transport for BGP + +Files: `netsim/runtime/transport.py`, `netsim/runtime/channels.py`. + +- **Wildcard listeners** with interface-set scoping and socket precedence: a + specific binding beats the wildcard. +- **Accept tables.** Per-remote TTL and `min_ttl` on accepted connections, with + `set_opts`. +- **Hop limit.** A routed session derives its hop count from the path trace it + already computes. A path whose forwarding hops exceed the TTL is unreachable, + so the connect or the queue stalls until the no-progress timeout, as a kernel + that drops the packets behaves. +- **GTSM** (RFC 5082). The sender uses TTL 255 and the receiver requires + `min_ttl = 256 − hops` (FRR `ttl-security hops N` sets `IP_MINTTL`). + +Exit (TTL counts forwarding hops, so a TTL-1 packet still reaches the adjacent +router's loopback): + +- With `disable-connected-check`, a TTL-1 session to the adjacent loopback + establishes. +- A loopback two hops away needs `ebgp-multihop 2`. +- GTSM with `hops 1` rejects a two-hop path. +- Wildcard versus specific listener precedence; the accept table refuses an + unknown remote. +- Existing transport suites unchanged. + +"No single-hop eBGP session to a loopback without multihop" is a BG2 test, +because the NHT connected gating is what blocks it in FRR. + +### F4. Domain-scoped oracle IGP + +File: `netsim/model/igp.py`. + +- **Factory.** `oracle_igp_domains(domains)` returns a `RouteSource`. Each + `IgpDomain(name, devices, include, exclude, passive, instance, distance, + metric_mode)` runs its own SPF. + - Passive interfaces advertise their prefixes without forming adjacencies. + - Each domain installs rows as `ClientId('igp', instance)`. +- **Purpose.** The fast analytical underlay for BGP tests and mixed studies. The + global `oracle_igp` equals one domain spanning every device. + +Exit: + +- Two ASes, each with its own domain and the inter-AS links excluded: no + inter-domain rows. +- The single-domain variant reproduces the current fingerprints. + +### F5. Shared protocol toolkit + +Location: `netsim/agents/common/`. + +- `selection.py`: the `DecisionFlavor` protocol (`compare`, `select`, + `multipath`), `PathView`, `DecisionContext`, `Verdict`, `Selection`, and the + small shared comparison helpers (AS_PATH length, MED comparability, router-id + substitution) that both built-in flavors use, each with refs. No interpreter. +- `spf.py`: pure ECMP Dijkstra (distances, first hops, predecessor sets) over + integer node ids. +- `timers.py`: + - `ExponentialBackoff` (Open/R semantics); + - `AsyncDebounce` (`AsyncDebounce.h:43-67`); + - throttle; + - jitter from per-module RNG streams; + - the per-module deadline heap multiplexer of D3. +- `routeops.py`: a RIB delta emitter keyed by changed-key sets. +- `sizes.py`: modelled message sizes. +- `refs.py`: the `@ref(...)` decorator (an immutable `__refs__` tuple on rule + functions and steps) and `@covers_ref(...)` for tests, feeding the generated + `docs/traceability.md` (section 13.8). +- `net.py`: the source-address selection helper (primary address of the egress + interface toward a destination), used by BGP when no `update-source` is set, + mirroring FRR's `getsockname` (`research-frr-decision.md` §4.2). +- `tests/agents/harness.py`: + - the locality harness; + - guarded submodule projection proxies; + - twin comparisons and normalizers; + - view-based assertions. + +### F6. Downstream invalidation (measured, optional) + +G15's conservative scans (placement on any FIB change, transport re-trace, +device-wide epochs) are narrowed with F1b's FIB diff: changed prefixes and +changed groups. This happens only if IX profiles show them dominating. Until +then, IX scenarios measure placement at declared points (`settle_delay` or +measurement-point placement) and passports report the cost. + +### F8. Forwarding programming contract (D14, section 5.13) + +Files: the FIB kind in `model/derive.py` and the programming records in +`model/forwarding.py` (F1 owner), with completion notifications in +`runtime/agents.py` (F2 owner). F8 follows F1b. + +- **State.** Each (device, table, family, prefix) holds a desired action and an + installed action. Requests carry `(device, table, af, prefix, generation, + request_id, action)`. Completion arrives after `device.fib.latency_s`, or + fails per `device.fib.failure_model` or `device.fib.capacity`, at the + configured boundary. +- **Guards.** + - Stale-generation completions are ignored. + - A withdrawal cancels obligations. + - Same-time completion and withdrawal are ordered deterministically. + - `POLICY_EXCLUDED` desired absence keeps the old installed entry until its + DELETE completes. +- **Default cost.** Today's `fib_delay` becomes the zero-failure special case, + so existing fingerprints hold. +- **Exit.** G01–G04 (gate-independent parts), E09, E14, the stale-ACK tests, and + unchanged fingerprints with default settings. + +### RP1. Route-policy language (routing-policy plan P1) + +- **Files:** `netsim/model/route_policy/` (`types.py`, `compile.py`, + `evaluate.py`), plus the capability table. +- **Timing:** after F0 and F7, in parallel with F1b/F2b. It must land before + BG3's import hook, BG4 and OR5. +- **Exit:** E01–E03, E10, E12 and E19–E22; the flow, default and absence + truth tables; fingerprint canonicalisation. + +### RP3. RIB admission with retained submissions (P3) + +- **Files:** retained submission and admitted views in `model/routing.py`; the + `RIB_ADMISSION` hook, including per-next-hop mode, on F1b's submission + boundary. +- **Timing:** after F1b and F2b. +- **Exit:** + - E04 (device side) and E13; + - deny → allow and allow → deny; + - generation purge; + - `ip protocol` per-next-hop semantics: leg pruning, and fallback to the next + candidate when no leg survives (`zebra_nhg.c:2880-2921`). + +### RP4. Resolver and FIB policy hooks (P4) + +- **Scope:** `RESOLUTION_IMPORT` with `scope='nht'` (per-client NHT answers, no + fall-through, as `zebra_rnh.c:515-576`) and `scope='forwarding'`; the + `FIB_INSTALL` exclusion. +- **Timing:** after F1c and F8. +- **Exit:** E05, E06, E09, E11 and E14; mixed node profiles; inactive × FIB gate + × policy. + +RP2 (BGP hooks) is part of BG3 and BG4. RP5 (the JSON codec, the typed FRR +adapter and explanations) is part of BG7. RP6 (acceptance, including E07, E23 +and E24) is part of M5. P0 (ownership and client profiles) is part of F0 and +F7. + +--- + +## 4. Open/R + +### 4.1 Fidelity target + +**Reproduce.** The eight behaviour groups of `research-openr.md` §10: + +1. the Spark FSM, area negotiation, GR and timers with jitter; +2. the metric pipeline; +3. LinkMonitor backoff, throttles and the adjacency hold; +4. KvStore merge, TTL, refresh, self-originated override, 3-way sync, peer FSM, + flood loop checks and per-area isolation; +5. Decision debounce and route computation; +6. PrefixManager selection, `area_stack` and cross-area rewrite; +7. the initialization chain; +8. the Fib delete delay and programming boundary. + +**Simplify.** Thrift encoding, KvStore transport internals, the ctrl server, +counters, dead features, FabricHelper, NeighborMonitor and FSDB, netlink resync, +and the RTT step detector (the first sample sets the metric; an optional step +model covers changes). + +### 4.2 Package + +``` +netsim/agents/openr/ + config.py OpenrConfig and AreaConfig (fields the daemon reads), resolved from settings.py (the fragment + whose DefaultsSpec holds upstream defaults) and the lab preset + types.py Spark messages, NeighborEvent, PeerEvent, Adjacency, AdjacencyDatabase, PrefixMetrics, + PrefixEntry, PrefixDatabase, KvValue, Publication, sync records, InitializationEvent + spark.py neighbor FSM per (interface, neighbor), area deduction, RTT, GR, fast init + link_monitor.py interfaces with flap backoff, adjacency DB per area, metric pipeline, drain and + overload, KvStore peer specs, interface-address redistribution + kvstore.py per-area KvStoreDb: merge, TTL (deadline heap), flooding, peer FSM, 3-way sync, + self-originated keys + decision.py LinkState per area, SPF, PrefixState, per-prefix routes, debounce + prefix_manager.py prefix types and selection, originated prefixes, cross-area redistribution, + injection from the RIB feed + fib.py AWAITING / SYNCING / SYNCED, RouteOps, delete delay, FULL_SYNC after RowNotice INSTALLED + init.py initialization event gates + agent.py OpenrAgent: worklist of module queues, timer namespaces, sessions, views +``` + +State: `OpenrState(init, spark, spark_liveness, link_monitor, kv: PMap[area, +KvStoreDb], decision, prefix_manager, fib)`. Liveness counters (sequence numbers, +reflected timestamps) live in a small record, so a hello run admits only that +record. Drain, overload and metric overrides live in `AgentNode.persistent`. + +### 4.3 Mapping to NetSim primitives + +| Open/R | NetSim | +|---|---| +| Spark over UDP `ff02::1:6666` per interface, hop limit 255 (`Spark.cpp:484-566, 1024-1034`) | `Datagram(interface, link_local=True, port=spark_port)` on the link channel. Links are point-to-point, so there is no LAN fan-out | +| KvStore thrift over TCP to `%:2018`, connect 500 ms, request 2500 ms (`KvStore-inl.h:1716-1790`) | A scoped session to the neighbour's link-local address on `kvstore_port`. Both sides open their own client connection, giving two connections as with thrift. Requests and responses are `Message`s with request ids; timeouts are timers | +| Inter-module queues (`Main.cpp`) | The D3 worklist. Explicit throttles and debounces are timers. Fib → PrefixManager waits for `RowNotice INSTALLED` | +| FIB client 786 (`Fib.h:392`); Linux handler protocol 99, priority 10 (`Platform.thrift:128-138`) | `RouteOp` under `ClientId('openr', i)`. Distance 10, matching Open/R's priority 10 and FBOSS `AdminDistance.OPENR = 10`; configurable per row | +| Route metric | **Declared modelling choice**, the setting `openr.fib.metric_mode` (a `netsim.*`-style choice, per node). `metric_mode='igp_cost'` (default) sets the row metric to `igpCost`, so FRR hot potato over Open/R is observable. `metric_mode='linux'` uses a constant 10, as the kernel handler programs, so hot potato disappears. FRR's redistribution MED follows the row metric (`bgp_route.c:11951`) | +| Netlink interface events | A subscription on `('interfaces',)`; `InterfaceView` supplies oper state, link-local address and addresses | +| Thrift drain APIs | `Command`s writing `AgentNode.persistent` (D11) | +| RTT from reflected hello timestamps (`Spark.cpp:889-975`) | The same arithmetic on simulated timestamps. Metric = `max(rtt_us/100, 1)` after rounding down to whole ms with a 1 ms floor (`LinkMonitor.cpp:33-35`; `Spark.cpp:872-886`). Links with an RTT under 2 ms therefore all get metric 10 | +| `enable_v4`, `v4_over_v6_nexthop` (RFC 5549 / RFC 8950) | IPv4 routes use `Nexthop.via(interface, neighbor LL, af=6)` or the neighbour's IPv4 transport address | + +### 4.4 Domains, areas, instances + +- **Deploy helpers.** Open/R's default area `"0"` with empty regexes discovers + nothing (`Config.cpp:34-39, 85-90`), so `openr.deploy` always writes explicit + `AreaConfig`s. +- **Multi-area.** The lowest matching area wins (`Spark.cpp:2802-2841`). A node + in several areas redistributes through PrefixManager: `distance + 1`, + `area_stack`, destination areas exclude the next-hop areas + (`PrefixManager.cpp:1669-1786`). +- **Separation.** Interfaces are owned by one instance or excluded from + discovery. Never rely on area names, because of the `"0"` wildcard (D5). +- **Multiple instances** are `netsim_ext.multi_instance` (D5). They exchange + routes only through an explicit RIB-feed redistribution, with tags against + two-border loops. +- **Single-device domain.** An instance with no discovering interfaces behaves + as `research-openr.md` §9 describes: KVSTORE_SYNCED on the first empty + PeerEvent, an empty computed RIB, FIB_SYNCED, and its own keys in its own + store. + +### 4.5 Behaviour by module + +Each rule carries its citation in code. The key rules: + +- **Spark.** + - The FSM (`Spark.cpp:103-172`). + - The incarnation check on WARM → NEGOTIATE. + - Handshakes every 500 ms under a 2 s negotiate timer. + - Negotiated hold and GR times are the minimum of both sides. + - One-way teardown. + - Hello 20 s ±20%; fast-init 500 ms for six periods. + - Heartbeat every hold/3 ±20%; hold 30 s; GR 30 s with restarting hellos. + - `holdAdjacency` and `adjOnlyUsedByOtherNode`. + - NEIGHBOR_DISCOVERED after 2 s minimum, 15 s maximum. + - Check that an adjacency brought up from WARM through a handshake can carry + RTT 0 (`getRttMetric(0)` returns 1); reproduce whatever upstream does. +- **LinkMonitor.** + - Flap backoff of 60–300 s. + - The metric pipeline, including soft-drain increments, which boot at +100 + when `assume_drained` (`LinkMonitor.cpp:243-254`). + - Adjacency up throttled 1 s, down immediate. + - `adj:` persisted per area. + - One KvStore peer per neighbour per area. + - `adj_hold_time_s` 4 s. + - Interface addresses redistributed as `LOOPBACK` (path_pref 1000, + source_pref 200). +- **KvStore.** + - The merge order of `KvStoreUtil.cpp:344-584`. + - TTL decrement and the 500 ms threshold. + - Floods go to INITIALIZED peers except the sender, with a `nodeIds` loop + check. + - Pending keys for syncing peers. + - 3-way sync, the peer FSM, backoff of 4–256 s, parallel sync limit 2 → 32. + - Self-originated keys: version bump, refresh at ttl/4, withdrawal by + version + 1 then expiry. + - Optional `flood_rate`. +- **Decision.** + - Bidirectional (node, ifName, otherIfName) matching. + - Overloaded nodes are not expanded; link cost is the maximum of both + directions. + - SPF is memoized per topology. + - The steps of `SpfSolver.cpp:184-319`. + - Debounce 10 → 250 ms. + - A full rebuild on topology change; prefix-only changes recompute only those + prefixes. + - With `enable_best_route_selection=false` (the default), `bestNodeArea` is + the lexicographically lowest (node, area) pair. It supplies + `bestPrefixEntry` and `bestArea` independently of which areas' next hops + are used (`SpfSolver.cpp:390-399, 543-560`; `SpfSolver.h:44`). +- **PrefixManager.** + - Type tie-break and `selectBestPrefixMetrics`. + - Per-area writes skip `area_stack` areas and send delete markers. + - `minimum_supporting_routes`. + - Cross-area rewrite: append `bestArea`, `distance + 1`, type RIB, reset + non-transitive fields; runs only with more than one area. + - RIB-feed injection with a configured `PrefixType` (for example `BGP`). +- **Fib.** + - AWAITING → SYNCING → SYNCED. + - Deletes held back 1 s. + - FULL_SYNC publishes once `RowNotice INSTALLED` confirms programming. + - On restart, routes are retained until the first sync (D12). +- **Initialization.** The chain of `research-openr.md` §1, every gate with its + timeout. + +### 4.6 Performance design + +- **Work per run.** O(merged keys) plus SPF only on a topology change. A + hello or heartbeat run admits only the liveness record, runs no SPF and emits + no route ops. +- **Indexes.** LSDB indexes (adjacency by node, prefix by prefix) are updated + from merged deltas. +- **Passport.** 256-node three-tier Clos in two areas, one loopback per node, + `lab` and `openr-default` profiles, expected to take minutes. It reports + cold-start and single-failure wall time, events, messages, SPF runs, memory + and heap objects. With the default RTT metric and sub-2 ms links, the metric + is hop count × 10. The fixed cost of about 0.2 ms per agent run plus a commit + per hello interval is part of the budget. + +### 4.7 Acceptance + +- **Oracle equivalence.** + - Configuration: a single area, `use_rtt_metric=False`, static metrics equal + to symmetric interface metrics, `assume_drained=False` (or an undrain + command), and `enable_v4=True` or `v4_over_v6_nexthop=True`. + - On random graphs of 6–10 nodes, normalized routes, next hops, FIB groups + and placement equal a domain oracle restricted to the redistributed + prefixes. +- **Faithful defaults.** + - RTT metrics come from link delays. + - Soft-drain at boot shows +100 until an undrain command. + - Cold start is no faster than the modelled gates allow. +- **Multi-area.** + - ABR redistribution; two ABRs do not loop. + - A case where `bestArea` differs from the areas that supply next hops. +- **Separation.** + - Excluded boundaries leak nothing. + - An area-`"0"` wildcard case documents the pitfall. + - Multi-instance isolation. +- **Spark.** + - The FSM matrix. + - An area mismatch sends NEGOTIATE back to WARM. + - GR within the GR time keeps routes. + - One-way teardown. + - A muted channel with oper still UP expires on the hold timer. +- **KvStore.** + - Merge property tests (random interleavings converge). + - TTL expiry after a partition. + - 3-way sync after heal. + - Self-originated override. + - Flood loop check. +- **Decision and drain.** + - Hard drain keeps the drained node reachable but removes it as transit. + - Soft-drain increments apply. + - `minNexthop`. + - `drain_metric` propagates across areas. +- **Fib.** + - The delete delay is observable in placement. + - Redistribution waits for programming. +- **Locality and regression.** The locality harness passes; submodule + projection proxies pass; both interpreters; fingerprints unchanged for + non-Open/R networks. + +### 4.8 Slices + +| Slice | Deliverable | Exit evidence | +|---|---|---| +| OR1 | config, types, Spark, agent shell (worklist, timers, views) | FSM matrix; area negotiation; RTT metric; GR; jitter determinism; liveness-only admission | +| OR2 | LinkMonitor and KvStore, single area | LSDB convergence properties; partition, heal, expiry; peer FSM | +| OR3 | Decision, PrefixManager and Fib (single area), initialization chain, PrefixManager's programmed-prefix readiness gate (membership, as `PrefixManager.cpp:650`) | Oracle equivalence; delete delay; gate timings; readiness withdrawal on programming loss (G10 Open/R part); ratio benchmarks | +| OR4 | multi-area, `netsim_ext.multi_instance`, boundaries | ABR, `area_stack`, `bestArea` case; separation tests | +| OR5 | drain and overload through commands and `persistent`, overrides, RIB-feed injection | Drain matrix; restart keeps drain; injected prefix from a static stand-in | +| OR6 | scale passport, study integration, `netsim_ext` SRv6 locator and SID advertisement for learned SR validation | Passport numbers; learned SR policy over Open/R | + +--- + +## 5. BGP + +### 5.1 Scope + +**In scope:** + +- IPv4 and IPv6 unicast over MP-BGP, with 4-byte AS numbers. +- eBGP, iBGP and confederation peers; route reflection. +- Session modes: interface addresses, loopbacks (update-source, multihop, + GTSM) and unnumbered IPv6 link-local with ENHE. +- A typed route-map subset for policy, with FRR's delay timer. +- `network`, `redistribute`, `maximum-prefix`. +- Soft reset: replay with soft-reconfiguration inbound, otherwise + ROUTE-REFRESH. +- Named decision flavors and multipath. +- FRR pipeline timing (`frr-modern` defaults; `frr-10.5` differences are settings, section 2B). + +**Out of scope for now:** EVPN, VPN, labeled unicast, flowspec, RPKI, +dampening, BMP, ORF, multi-access LANs. NetSim links are point-to-point. + +### 5.2 Package + +``` +netsim/agents/bgp/ + config.py global, neighbor, peer-group, AF config; validate_config; profiles + frr-traditional, frr-datacenter, sonic, lab + types.py AFI/SAFI, NLRI, path ids, OPEN / UPDATE / KEEPALIVE / NOTIFICATION / ROUTE-REFRESH / EoR, + capabilities, NOTIFICATION codes + attrs.py PathAttributes (shared by identity), communities, ORIGINATOR_ID, CLUSTER_LIST, + next hops (IPv4; IPv6 global [+ link-local]), modelled sizes (RFC 4271 §4) + aspath.py SEQ/SET/CONFED_SEQ/CONFED_SET; hop count (confed excluded, a set counts 1); confed ops; + loop checks with allowas-in; AS4 semantics + peer.py peer_sort, effective local AS, local-as, router-id selection, negotiated capabilities + fsm.py Idle, Connect, Active, OpenSent, OpenConfirm, Established, Clearing; timers; collision; + OPEN validation (identifier, RFC 6286); NOTIFICATION; fast external failover + policy.py BGP adapter to the shared route-policy engine (section 5.8): PolicyAccess for BGP attributes, + hook wiring (import, export, table-map, redistribute), reevaluation scheduling, RFC 8212 + inbound.py UPDATE processing in FRR's check order; outcome classes (session reset, + treat-as-withdraw, attribute discard, filtered); soft-reconfiguration store; maximum-prefix + nexthop.py next-hop cache keyed (address, interface for link-local); NHT registrations + (mode installed); validity, via_metric; back-index next hop -> paths; connect gating + decision/ + frr.py bgp_path_info_cmp (modern; version differences are settings), hand-written with refs per reason; sorted-list insertion + rfc4271.py §9.1.1 preference, §9.1.2 exclusions, §9.1.2.2 elimination, RFC 4456 §9, RFC 5065 §5.2/§5.3 + multipath.py maximum-paths per profile, equal-cluster-length, as-path relax, peer-type relax + locrib.py Dest (below); dirty queue; 50 ms process hold + outbound.py update groups keyed by compiled export-policy fingerprint and peer facts; announce check in FRR order; next-hop rules with a self sentinel patched + per peer; Adj-RIB-Out per group; duplicate suppression; source echo; packing by attribute + set; withdrawals first; per-peer MRAI; coalesce timer; EoR + rr.py reflection matrix, ORIGINATOR_ID and CLUSTER_LIST writing, inbound loop checks + confed.py confederation data, send-side AS_PATH construction, receive checks + zebra.py BGP_RIB_EXPORT (table-map) hook, then Loc-RIB -> RouteOps (distances, recursion, legs, weights, metric = MED, + reinstall conditions, latest-state-wins queue) + origination.py network with exact-prefix, non-BGP import-check; redistribute from the RIB feed with policy and tags + agent.py BgpAgent: worklist composition, timers, sessions, views, commands +``` + +### 5.3 Sessions + +| Mode | Configuration | Transport | Next hop sent | Receiver NHT and install | +|---|---|---|---|---| +| eBGP single-hop on link addresses | `neighbor 10.1.12.1 remote-as X` | routed, TTL 1; source address from `agents/common/net.py` | self (the local address) | `connected_only`, connect gated on it; non-recursive install (interface + address) | +| iBGP over loopbacks | `remote-as internal`, `update-source lo` | TTL 255; needs a route to the peer (IGP, static or BGP) | unchanged for eBGP-learned routes unless next-hop-self | recursive; `via_metric` (installed) feeds the IGP step | +| eBGP multihop or GTSM on loopbacks | `ebgp-multihop N` or `ttl-security hops N` | TTL N, or TTL 255 with `min_ttl` | self (a loopback, indirect for the receiver) | recursive | +| Confed peer | peer AS in `bgp confederation peers` | TTL 1 by default (`bgpd.c:2532-2552`); `ebgp-multihop` for loopbacks further away | unchanged; next-hop-self applies | no connected check; recursive; distance 200 | +| iBGP on link addresses | `neighbor remote-as internal` | TTL 255 | unchanged; with `next-hop-self force` on reflected routes, self | recursive (resolves over the connected route) | +| Unnumbered | `neighbor eth1 interface remote-as external` | scoped link-local session; the peer's link-local comes from `ctx.neighbors` | ENHE: IPv4 NLRI over an IPv6 link-local next hop (RFC 8950) | driven by interface state, never registered with zebra (`bgp_nht_ifp_up/down`) | +| Passive; dynamic peers (BG8) | `passive`; `bgp listen range` | listener with accept table | by sort | by sort | + +- **Collision** (RFC 4271 §6.8, FRR): the connection initiated by the speaker + with the higher BGP identifier survives; on equal identifiers, the higher AS + wins. The loser gets Cease/7. An Established or Clearing connection always + survives. +- **Router id** follows zebra: the configured value; else the highest IPv4 + loopback address; else the highest IPv4 interface address + (`zebra/router-id.c`). IPv6-only fabrics need an explicit id. An iBGP OPEN + carrying our own identifier is rejected (RFC 6286). +- **Teardown.** Fast external failover (D8). Hold expiry sends NOTIFICATION + 4/0, flushes the Adj-RIB-In and withdraws from other peers. Recovery goes + through idle hold, connect retry and the coalesce timer. +- **Unnumbered simplifications, recorded.** + - FRR learns the peer's link-local from RAs with a 10 s interval + (`bgpd.h:68`); NetSim knows it as soon as the L3 neighbour entry exists. + - The final tie-break compares the IPv6 scope id, which needs NetSim's stable + ifindex. + +### 5.4 Next hops and attributes on send and receive + +**Send side** (`research-frr-decision.md` §4.5; `research-frr-rr-confed.md` §5): + +| Route → target | NEXT_HOP | +|---|---| +| local `network`, aggregate, redistributed connected → any | self | +| any → eBGP | self, unless the next hop shares a connected subnet with the target (RFC 4271 §5.1.3 third party) | +| eBGP-learned → iBGP | unchanged, unless next-hop-self or force | +| iBGP → iBGP (reflected) | unchanged; plain next-hop-self ignored; `force` sets self | +| any → confed member peer | unchanged; next-hop-self applies | +| confed-learned → iBGP | unchanged; plain next-hop-self applies (this is not reflection) | +| `attribute-unchanged next-hop` / route-map `unchanged` | kept | +| a link-local next hop placed in the global field | always reset to self (`bgp_route.c:3205-3217`) | + +Other attributes on send: + +- MED is stripped toward eBGP unless the route is locally originated or + `attribute-unchanged med` is set. +- LOCAL_PREF is sent to iBGP and confed peers, filled with the default when + absent. +- ORIGINATOR_ID and CLUSTER_LIST are written only iBGP → iBGP. +- Communities follow the well-known rules. +- IPv6 carries a global plus link-local next hop only under RFC 2545 §3. + +**Receive side.** + +- Checks run in FRR's order (`research-frr-rr-confed.md`, implications §5): + 1. local-as; + 2. AS loop with allowas-in; + 3. originator; + 4. cluster; + 5. RFC 8212; + 6. filters; + 7. route-map; + 8. BLACKHOLE and GSHUT; + 9. own next hop (`bgp_update_martian_nexthop`: filtered). + + A martian IPv4 NEXT_HOP (0/8, 127/8, class D/E) is caught earlier, at + attribute parse, and is treat-as-withdraw (RFC 7606 §7.3). 10.5.4 reset the + session instead; this is a fix, not modelled. +- Additional semantic checks: + - confed sanity: a confed peer's path must start with a confed segment, and + an eBGP path may contain none; a violation is treat-as-withdraw; + - `enforce-first-as` (on by default); + - AS 0 from eBGP; + - `reject-as-sets` (on by default). +- Every loop-check failure is an implicit withdraw, never a NOTIFICATION. +- Validity comes from NHT. When a next hop changes, only the paths on it are + re-evaluated (FRR `evaluate_paths` over `bnc->paths`). + +**Install.** + +- Distances: eBGP 20, iBGP 200, confed 200. +- Recursion is allowed except for single-hop eBGP. +- Metric = MED. +- `network` and redistributed winners are not installed. +- For an unchanged best path with unchanged attributes and no clear or + add-path in effect, a route is reinstalled only on an IGP-metric, multipath or + bandwidth change (`bgp_route.c:4439-4462`); otherwise the complete emitted + local row is compared and replaced when its semantics change (a MED edit + changes the installed metric). `zebra.py` never re-emits identical rows: each costs about 7.8 µs of `RouteOp` + validation plus about 2 µs of `rib_apply` equality checking. + +### 5.5 Decision process and flavors + +**Per-destination state.** Everything a flavor needs across runs is explicit and +immutable: + +```python +Dest(key=(vrf, afi, safi, prefix), paths: PMap[PathKey, Path], order: tuple[PathKey, ...], + unsorted: tuple[PathKey, ...], # in bgp_process order: each changed path moves to the head + removed_best: PathKey | None, # a removed old best survives one round + best: PathKey | None, multipath: tuple[PathKey, ...], reason: str, + flags: frozenset[str], # multipath_changed, igp_changed, attr_changed + flavor_memo: Any) # reset on any flavor or knob change (full recalculation) +PathView(attrs, peer, sort, received_seq, established, valid, via_metric, stale, ...) +DecisionContext(config knobs, router id, confederation, prev_best) +``` + +**Candidate filter** (all flavors). Excluded: holddown paths (invalid next hop, +removed) and paths from peers that are not Established, except local paths and +paths in GR helper wait. GR-stale paths stay and lose late, unless their next hop +becomes unreachable, in which case modern FRR deletes them (RFC 4724 §4.2; +N1 in section 2B). + +**`frr`**: modern `bgp_path_info_cmp`, hand-written with a reference per reason. + +- **Pre-steps**, in order: + - LLGR_STALE loses, ahead of weight, and sets no reason (quirk kept); + - step 0, admin distance between a redistributed path and an installable BGP + path, gated by `bgp.bestpath.redist_admin_distance`: on in `frr-modern` + (since 10.6.0), off in `frr-10.5`; + - UPA is master-only and out of scope. +- **Main steps:** + - 1–3: weight (32768 for local routes); local-pref; local route (all local + sub-types tie). + - 3.5: AIGP (off by default). + - 4: AS_PATH length (confed excluded, a set counts as 1). 5: origin. + - 6: MED, under the leftmost-AS rules with the confed variants and + missing-as-worst. + - 7: peer type (confed counts as internal). + - 8: IGP metric, held. + - 9: cluster-list length as an IGP substitute, only with + `equal-cluster-length`. + - 10: confed-external over confed-internal. It returns immediately, + overriding step 8, except under `bgp bestpath peer-type multipath-relax`, + where it is deferred past the multipath check like step 7 + (`bgp_route.c:1654-1684`). + - 11: multipath check. + - 12: keep the current eBGP best (RFC 5004), unless `compare-routerid`. + - 13: router id, with ORIGINATOR_ID substituted. + - 14: cluster-list length. + - 15: stale, locally configured, lower neighbour address (IPv4 before IPv6, + then scope id). +- **Selection is sorted-list insertion**, with three rules that decide results: + - (a) within one 50 ms batch, changed paths are inserted in reverse arrival + order, because each moves to the head; + - (b) withdrawals and holddown do not re-sort the remaining list; the new + best is the head of the existing order; + - (c) step 12 reads the previous best, so history matters. +- **Reason strings** equal FRR's `selectionReason`. +- **`dmed`** (`bgp.bestpath.dmed`). + - `'frr-current'` is FRR since 10.4.0 (also 10.2.2 and 10.3.1): grouping + feeds only add-path. + - `'classic'` is pre-10.1 FRR: only the neighbouring-AS group winners compete. + - The default in both dialects is `'frr-current'`. + +**`rfc4271`**, in three phases: + +- §9.1.1 Phase 1 assigns the degree of preference: LOCAL_PREF for iBGP-learned + routes, a policy value for eBGP-learned routes. +- §9.1.2 Phase 2 excludes unresolvable next hops (§9.1.2.1) and AS loops, then + keeps the highest preference. +- §9.1.2.2 a–g eliminates over the whole set: + - (a) AS_PATH length, with an AS_SET counting as 1; + - (b) origin; + - (c) MED per neighbour AS, with a missing MED as 0; `neighborAS` for + iBGP-learned routes as the section defines it; + - (d) eBGP over iBGP; + - (e) interior cost, skipped when no cost can be determined; + - (f) lowest BGP identifier; + - (g) lowest peer address. + +Amendments: + +- RFC 4456 §9: ORIGINATOR_ID replaces the identifier; the shorter CLUSTER_LIST + rule sits between (f) and (g). +- RFC 5065 §5.2: MED comparison skips confed segments. +- RFC 5065 §5.3, all four rules: + 1. a path with only confed segments has the local AS as its neighbour AS; + 2. otherwise the neighbour AS is the leftmost AS of the first AS_SEQUENCE; + 3. confed segments are not counted in path length; + 4. confed peers count as internal at (d). + +Locally originated routes compete through a documented NetSim choice, since the +RFC does not say. There is no multipath by default (RFC 4271 defines none); the +FRR rule can be enabled explicitly. The result does not depend on order. + +**Later flavors (BG8).** `cisco-ios` and `junos` (section 10), each verified +against the vendor document for the targeted release. + +**Custom flavors.** A `FlavorSpec` holds built-in step names and user step +instances with refs and is the value of the `bgp.decision` setting, so it is +overridable per node. Knob arguments to `FlavorSpec.use(...)` write the +corresponding `bgp.bestpath.*` settings. + +**Multipath (FRR).** + +- `maximum-paths` defaults to `MULTIPATH_NUM` per profile: + - 16 for a bare `./configure`; + - 256 for FRR's packages (`debian/rules:84`; `redhat/frr.spec.in:21`); + - SONiC: patch 0016 raises `MULTIPATH_NUM` to 514, which caps the value; the + SONiC template configures `maximum-paths 64`. +- Candidates tie with the best path through step 11. iBGP needs an identical + AS_PATH; eBGP and confed need the same configured neighbour AS, unless + `as-path multipath-relax`. +- `equal-cluster-length` applies to iBGP. +- `maxpaths` is chosen by the best path's peer sort; the set follows sort order. +- Link-bandwidth UCMP comes in BG8. + +### 5.6 Route reflection (RFC 4456, FRR dialect) + +- **Reflection matrix.** It applies only when source and target are both IBGP. + - A client route goes to clients (unless + `no bgp client-to-client reflection`), non-clients, confed and eBGP peers. + - A non-client route goes to clients, confed and eBGP peers. + - eBGP, confed and local routes go to everyone. +- **Attributes.** ORIGINATOR_ID is set from the source peer's identifier + **only when absent** (`bgp_route.c:2820-2825`), and the cluster id is + prepended to CLUSTER_LIST, iBGP → iBGP only. Both are stripped toward confed + and eBGP peers. The cluster id defaults to the router id. +- **Inbound loop checks.** Our identifier as ORIGINATOR_ID, or our cluster id in + CLUSTER_LIST, is an implicit withdraw. +- **Outbound policy on reflected routes.** Only permit and deny apply; set + actions are dropped unless `bgp route-reflector allow-outbound-policy`, in + which case a route-map may rewrite the next hop and that explicit rewrite + prevents the automatic reset (`bgp_route.c:2963-2981, 3127-3149`). Plain + `next-hop-self` is ignored on reflected routes; `force` applies. +- **Echo.** FRR suppresses sending a route back to its source only for + `neighbor X solo` peers (`PEER_FLAG_LONESOUL`, `bgp_route.c:2471, 2612`). + Echoes to eBGP sources and RR clients are therefore sent and dropped by the + receiver. They are modelled, because they change message counts. + +### 5.7 Confederations (RFC 5065, FRR dialect) + +- **Peer sort.** A peer AS listed in `bgp confederation peers` is CONFED. A + peer whose AS equals our confederation identifier is iBGP (10.7.0 fix; 10.5.4 + classed it eBGP). + - TTL 1 by default. + - Distance 200. + - No connected check. + - RFC 8212 exempts confed peers in FRR, although the RFC names + confederation boundaries (recorded deviation). +- **Sending to a confed peer.** The member AS is prepended as + AS_CONFED_SEQUENCE. LOCAL_PREF is sent, MED is kept and the next hop is + unchanged. ORIGINATOR_ID and CLUSTER_LIST are dropped. NO_EXPORT_SUBCONFED + blocks the route. +- **Sending to a true external peer.** Confed segments are stripped and the + confederation id is prepended. +- **Receiving.** + - The confed sanity checks of §5.4. + - The loop check covers the member AS in any segment and the confed id in + non-confed segments. + - Any path containing the confed id is suppressed on send. +- **Decision.** + - Step 7 counts confed peers as internal. + - Step 10 prefers confed-external over confed-internal before the IGP metric. + - `med confed`; `as-path confed`. + +### 5.8 Routing policy: one shared engine (integrates `docs/plans/routing-policy.md`) + +The BGP-local route-map evaluator of earlier revisions is replaced by the shared +route-policy engine specified in the reviewed routing-policy plan +(`NetSim/docs/plans/routing-policy.md`, revision 4; review record +`routing-policy-reviews.md`, 36 findings). That document is the contract for +the language, evaluator, attachment capabilities, lifecycle and fixtures +E01–E24. This section states how it fits this plan and records the corrections +made during integration (section 11, revision 5). + +**Language and engine.** `netsim/model/route_policy/` (`types.py`, +`compile.py`, `evaluate.py`; a codec later) is model-layer code shared by BGP, +Open/R injection, RIB admission, FIB installation and resolver filtering. +- **Records:** `RoutePolicy(name, terms)`; `Term(name, when, actions, flow)`; + `RoutePolicyBinding(point, policies, fallback, match_input)`; + `RoutePolicyRef` / `RouteSetRef` pinned by revision and content digest. + `RoutePolicyRef` is distinct from the SR `PolicyRef`. +- **Flow values**, exactly four: `ACCEPT`, `REJECT`, `NEXT_TERM`, + `NEXT_POLICY`. `ACCEPT` and `REJECT` end the whole chain, and the fallback at + the end of a chain is explicit. An absent binding is distinct from an empty + chain (RFC 8212 needs the difference). A `REJECT` term cannot carry actions, + and a rejection discards tentative edits. +- **Match input:** `ORIGINAL` means this attachment's input; `WORKING` is the + default. Actions always operate on the working route. +- **Predicates:** typed leaves composed with `AllOf`, `AnyOf` and `Not`. + - Quantifiers are explicit, and an absent attribute is not zero (`Present`). + - `PrefixListPermits` gives ordered first-match prefix lists. + - BGP AS-path predicates count hops as section 5.5 does: confederation + segments excluded, an AS_SET counted once. +- **Excluded from the language:** loops, calls, jumps, runtime name resolution, + Python callbacks and raw regular expressions. +- **Access to route fields:** a code-owned `PolicyAccess[T]` (`read`, + `read_context`, `apply`) over the owner's own immutable route type, with a + closed field and action catalog and one capability table per attachment + point. There is no universal route record and no `set(field, Any)`. +- **Failure handling:** compilation rejects the whole configuration on any + error. Failures in an access adapter are invariant violations that abort the + owner's transaction; they never become a policy deny. + +**Attachment points and their FRR mappings.** Integration checked each FRR +mapping in source and corrected two. The FRR adapter in BG7 uses this table, +and it rejects every construct it cannot lower exactly. + +| Point | Owner and meaning | FRR construct it translates, verified in source | +|---|---|---| +| `BGP_IMPORT` | BGP inbound; filter or modify a received candidate | `neighbor X route-map M in` | +| `BGP_EXPORT` | BGP outbound, per peer; suppress or modify an advertisement | `neighbor X route-map M out` | +| `BGP_RIB_EXPORT` | BGP → RIB submission; may rewrite the local row's metric and next hops or omit it; the selected BGP path is untouched | `table-map M`. Filtering behaviour is to be confirmed by a lane fixture | +| `REDISTRIBUTE` | Origination into BGP from RIB rows | `redistribute P route-map M` | +| `RIB_ADMISSION` | Model-owned admission of a submitted row into the main RIB, keeping a retained submission view | **None directly.** FRR's `ip protocol P route-map M` is not a submission filter: zebra evaluates it per original next hop during next-hop activation (`zebra_nhg.c:2880-2921`, after `nexthop_active` fills the interface), deactivating a denied next hop and its resolved contribution; a route left with no active next hop is ineligible and zebra selects the next candidate; system routes and family mismatches bypass it. It maps to the resolver's candidate-next-hop activation filter (RP3), which reevaluates on underlay change without a new submission | +| `FIB_INSTALL` | Model programming stage; desired absence (`POLICY_EXCLUDED`) without fallback to another candidate and without leg pruning | **None.** Native only, and the FRR adapter never targets it; `ip protocol` belongs to `RIB_ADMISSION`, per the row above. The EOS documentation conflict stays a named fixture | +| `RESOLUTION_IMPORT` | Resolver; excludes candidates for a query. **Integration adds a scope:** `nht` (next-hop tracking answers for one client) or `forwarding` (recursive resolution for installation) | **`ip nht P route-map M` = scope `nht`, `fallthrough='never'`.** zebra first picks the resolving route by longest match, then filters that route's next hops for that client protocol. If all are filtered, the client is told "filtered/unresolved" and no less specific route is tried (`zebra_rnh.c:515-576`). Forwarding recursion is unaffected. The native `forwarding` scope, with fall-through obeying `lpm_fallthrough`, has no FRR equivalent | + +**Where the pieces live.** +- `agents/bgp/policy.py` becomes the BGP field and action adapter plus hook + wiring. It is not an evaluator. +- Open/R's `prefix_manager.py` uses `REDISTRIBUTE`-style bindings when RIB-feed + injection lands (OR5), through a destination-owned capability row. +- The device hooks (`RIB_ADMISSION`, `FIB_INSTALL`, `RESOLUTION_IMPORT`) belong + to F1/F2 and the new F8 programming contract. +- Policy bindings are per-node settings (section 2A): exactly one resolved + binding per (node, point, owner or instance, family, table, peer). + +**Retained inputs.** +- `RIB_ADMISSION` keeps the unfiltered submission view next to the admitted + view once a family has a binding, so deny → allow can restore a row. +- Every production RIB writer goes through one model-owned submission helper + (F1b): `derive.py` connected, local and SR-local; `igp.py`; `entities.py` + `RibClient`; `network.py` batch reconciliation; `runtime/agents.py` + `RouteOp`s. Trusted bypass writers update both views. +- BGP import reevaluation replays the soft-reconfiguration store when one is + kept, else sends ROUTE-REFRESH (RFC 2918). Enhanced route refresh (RFC 7313, + BoRR/EoRR) gives a completion boundary. With neither, the state is + `needs_session_reset`, never reported as settled; the per-node setting + `bgp.policy.on_replay_unavailable` is `report` or `reset_session`. + +**Edit timing** replaces the old fixed 5 s `DELAYED` action. +- A validated definition is active for new inputs at commit. Reevaluation of + existing routes follows `policy.reevaluation.mode` (`automatic` or `manual`) + and `policy.reevaluation.delay_s`. +- The `native` default is `automatic`, 0 s. The FRR dialects default to + `automatic`, 5 s, the `bgp route-map delay-timer` default. +- The FRR adapter maps a delay-timer of 0 to `manual`, because FRR's zero + disables automatic route updates and flushes a pending timer + (`bgp_set_route_map_delay_timer`). Zero never maps to native immediate + reevaluation. +- Scheduled work captures the owner generation and target revision, and reads + current inputs when it runs. + +**Update groups** are keyed by the compiled export-binding fingerprint plus the +declared peer and context facts the policy reads, never by policy name. + +**Regular expressions.** None in the native core. The FRR adapter lowers only +anchored single-ASN AS-path patterns (`^A_`, `_A$`, `_A_` and their +equivalents) to typed predicates, and rejects the rest with a report. A bounded +regex grammar is a separate later slice with source-backed semantics. + +**RFC 8212 and profiles.** With `bgp.ebgp_requires_policy` True (the +`traditional` profile) an otherwise valid eBGP session still reaches Established +and exchanges control messages; per AF and direction, a missing import binding +discards received routes and a missing export binding suppresses advertisements +(FRR filters in announce and UPDATE processing, `bgp_route.c:2692-2702, +6265-6275`, and reports the state operationally). The configuration and the +OPEN are never rejected for a missing policy; views report the missing-policy +state; an attached empty binding is distinct and follows its fallback. Not +enforced under `datacenter`, or under the `sonic` preset, which writes `no bgp +ebgp-requires-policy`. Confed peers are exempt (a recorded FRR deviation). + +**`maximum-prefix`.** Cease/1, with restart-after or warning-only. + +### 5.9 Pipeline timing (`frr-modern` defaults) + +| Stage | Behaviour | Default | +|---|---|---| +| inbound | withdrawn NLRI (IPv4, then MP_UNREACH) before announced; a duplicate of the interned attributes triggers nothing. Master-only (#23420); 10.5.4 applies IPv4 announce first, so a prefix in both fields ends withdrawn. That is a fix, not modelled for `frr-10.5` | – | +| best-path scheduling | one selection per destination per burst; the work queue holds the first enqueue | 50 ms hold | +| install | latest-state-wins queue to `RouteOp`s, then the device's `fib_delay`. zebra's own 10 ms RIB queue hold (`zebra_router.h:185`) is folded into `fib_delay` (recorded simplification) | – | +| new subgroup | coalesce timer; the initial table bypasses MRAI. "Peers" counts the peers configured in the instance at each creation | `min(10 s, 1 s + 50 ms × peers)` | +| MRAI | per peer; withdrawals gated per `bgp.out.mrai_withdrawals` (`gated` in modern, `bypass` in `frr-10.5`) | 0 s for eBGP and iBGP | +| install queue | two classes: destinations that resolve tracked next hops are drained first (10.7.0) | – | +| suppress-fib-pending / FIB ACK gate | the F8 programming contract and gate profiles of section 5.13; FRR's `suppress-fib-pending` = the limited-scope `frr-suppress-fib-pending` profile (new peer-learned routes gated; a selected path not installed is withdrawn, except a selected `network` static) | off; on in `sonic`; batch 1000 ms | +| inactive export | whether a BGP best path that lost the main-RIB election is advertised (`bgp.export.inactive`, section 5.13) | on in FRR dialects unless suppress-fib-pending | +| advertisement-delay | hold UPDATEs N s after the first Established peer; selection and FIB proceed | off | +| GR select-defer | restarter deferral (BG8) | 120 s modern, 240 s `frr-10.5`, 45 s `sonic` ToR | +| keepalive / hold | negotiated as the minimum hold; keepalive capped at hold/3 | 60/180 s traditional, 3/9 s datacenter | +| connect retry | doubles to 120 s unless set explicitly | 30 s traditional, 10 s datacenter | +| idle hold | 1 s for the first start; doubled before re-arming after an error (so 2 s first), up to 120 s; reset on Established | 1 s | +| policy edits | active for new inputs at commit; reevaluation per `policy.reevaluation.mode`/`.delay_s`; import replay needs a soft-reconfiguration store, ROUTE-REFRESH or reports `needs_session_reset` (section 5.8) | automatic, 5 s in FRR dialects; 0 s in `native` | +| update-delay | plugs selection until timer or EoR from all peers | off | + +### 5.10 State and performance + +- **Tables.** + - `Dest` per key in a sharded `PMap`. + - A next-hop back-index `PMap[nh_key, PMap[(dest, path), None]]`. + - Adj-RIB-Out as `PMap[group, PMap[dest, advertised attrs]]`. + - Per-peer deadlines in the module deadline heap. +- **Attribute sharing.** Attributes are shared by identity: an UPDATE's + attribute object is delivered by reference. Import policy is evaluated per + route and context; compiled bindings are reused, results are not (a policy + may depend on the prefix, so attribute identity is not a valid cache key). +- **Change-proportional targets.** + - An UPDATE with k prefixes costs O(k × affected groups). + - A next-hop change costs O(paths on that next hop). + - A keepalive run admits O(1) records. +- **Cost model.** Primitive costs measured on this tree: + - `PMap.set` 1.5 / 3.4 µs at 10k / 100k; + - `rib_apply` 9 µs per row in a batch, 23 µs for a single row at 50k; + - `RouteOp` validation 7.8 µs per row; + - incremental FIB about 30 µs per prefix; + - admission about 30 µs per touched shard at 50k; + - a fixed 0.2 ms per agent run. Each route op also triggers a second + `CAUSE_ROUTES` run. + + So a BGP path arrival costs roughly 150–250 µs per router, plus about 40 µs + per update group. Keepalives with the datacenter profile cost about 670 runs + per simulated second per 1000 sessions (0.13–0.33 s of wall time). +- **Passports** are defined in section 14.4 (P-A analytical 100k devices, P-B + BGP full table, P-C wide protocol fabric). The earlier list below is kept + for the cost model only; each publishes wall time, events, messages, memory + and heap object counts (GC multiplies allocation-heavy paths by about 4×, + G16), with `keep_records` bounded: + - (a) RFC 7938 Clos, 128 devices, eBGP unnumbered. Plausible, but minutes, + not seconds: a 64-neighbour spine holds about 70k paths, and MRAI 0 path + hunting multiplies that. + - (b) An AS with 2 RRs, 10k external prefixes on 32 clients plus 50k on 4 + clients. Gated on the cost of a single-prefix flap. The revision 1 size + (50k × 32) was about 3.2M path arrivals, roughly 8 minutes and at least + 4 GB, beyond a CI runner. + - (c) Hop-by-hop iBGP RR with next-hop-self force on a 64-device ring and Clos. + +### 5.11 Acceptance scenarios + +Each scenario records its provenance: an RFC section and the FRR topotest it is +written after, where one exists. Topotests are not copied. + +1. **eBGP single-hop, numbered, IPv4 and IPv6.** The next hop is direct and + installs non-recursive. An interface going down triggers fast external + failover: + - the session is stopped; + - the Adj-RIB-In is flushed; + - withdrawals go to the other peers; + - recovery follows idle hold, connect retry and coalesce. + + With `no bgp fast-external-failover`, the path is invalidated by the + connected check and the session survives until hold expiry. +2. **Unnumbered eBGP with RFC 8950** (after `bgp_unnumbered`, + `bgp_ipv4_over_ipv6`). +3. **iBGP full mesh over loopbacks, first on a domain oracle, then on Open/R** + with `metric_mode='igp_cost'`. + - Hot potato by IGP metric, reason "IGP Metric" (after + `bgp_path_selection`). + - An IGP failure reprocesses only the paths on the affected next hops (count + assertion). + - `metric_mode='linux'` removes hot potato, as it would on a real device. +4. **eBGP over loopbacks.** + - With the loopback learned from an IGP and no multihop, there is no + session: NHT connected gating holds the connect. + - With a static route to the loopback, the session establishes at TTL 1, + because zebra's connected check accepts STATIC, but the routes stay + inactive (§9 quirk). + - `disable-connected-check` at TTL 1 establishes and installs recursively + (`10.5.4:bgp_zebra.c:1570-1574`). + - `ebgp-multihop` establishes it. + - GTSM rejects a longer path. +5. **Route reflection** (after `bgp_rr_ibgp`). + - Single cluster; redundant RRs with the same and with different cluster + ids. + - Client-to-client reflection off. + - ORIGINATOR_ID and CLUSTER_LIST loops handled as implicit withdraws. + - Reflected next hops unchanged, so an IGP or next-hop-self force is + required. + - Echo message counts. +6. **Hop-by-hop iBGP, every router an RR for its neighbours, + `next-hop-self force`, no IGP.** A NetSim scenario; `bgp_cluster_length` + covers the related case without force, with next hops resolved BGP-over-BGP. + - With `maximum-paths ibgp N equal-cluster-length`, this equals shortest-hop + ECMP. + - Without it, step 9 is skipped. The multipath set is every neighbour whose + best path does not contain the receiver's cluster id, so A→C and C→A can + coexist. The forwarding loop is asserted by trace. +7. **Confederations** (after `bgp_confed1` and + `bgp_confederation_stripped_to_external`). + - Externals see the confed id. + - LOCAL_PREF is kept across sub-ASes. + - Confed-external wins before the IGP metric. + - `med confed`. + - Loopback confed peering one hop away establishes at TTL 1 (no connected + check); two hops away needs `ebgp-multihop 2`. +8. **Decision flavors.** + - Per-step unit tables per flavor. + - Order dependence with controlled batching: all iBGP paths A = (AS1, + MED 0, rid 3), B = (AS1, MED 10, rid 1), C = (AS2, MED 5, rid 2). + - Under `frr` with one path per batch, arrival A, B, C selects C, and + B, C, A selects A. + - The same paths delivered in one batch follow rule (a). + - `rfc4271` selects the same path for every order. + - `compare-routerid` versus RFC 5004. + - A unit test that a withdrawal does not re-sort. +9. **Policy** (the routing-policy plan's fixtures E01–E24 apply in full; the + BGP-side ones belong to this scenario). + - RFC 8212 per profile, with absent and empty bindings distinguished. + - local-pref, MED, communities, prepend. + - NO_EXPORT, NO_ADVERTISE, NO_EXPORT_SUBCONFED. + - Edit reevaluation (automatic, manual, FRR delay 0) and soft reset in, all + three paths: replay, ROUTE-REFRESH and `needs_session_reset` + (E08, E15–E17). + - Update-group isolation by policy fingerprint and peer facts (E18). + - `table-map` and redistribute hooks. + - `maximum-prefix`. +13. **Installation gating and inactive export** (section 5.13, G01–G10). + - An ACK-gated first advertisement. + - Failure, capacity limits and a lost completion. + - A stale ACK after replacement. + - A withdrawal while a request is pending. + - A static or IGP route winning the RIB, with inactive export on and off. + - The three gate profiles. + - The FRR suppress-fib-pending limited-scope profile. +10. **Dynamics.** Expected results are per flavor. + - RFC 3345 §2 MED oscillation: `rfc4271` cycles, and the study reports + `deadline_exceeded`. `frr` may settle in a stable, suboptimal state, + because withdrawals do not re-sort; the arrival order decides, and the FRR + lane confirms. + - BAD GADGET (Griffin–Shepherd–Wilfong) diverges under both flavors. + - DISAGREE settles in one of two stable states depending on timing and + seed. +11. **Timing.** + - MRAI 0 versus 30 s. + - The coalesce delay dominates initial convergence. + - Hold expiry on a stalled transport. + - Connect-retry and idle-hold backoff. +12. **Origination.** + - `network` with exact-prefix import-check. + - `redistribute` through the RIB feed. + - Tag-based loop prevention between two borders (IX). + +### 5.12 Slices + +| Slice | Deliverable | Exit evidence | +|---|---|---| +| BG1 | `types`, `attrs`, `aspath`, `peer` (sort, router id), `config`, profiles | Semantic vectors re-derived from RFC 4271 §9.1.2.2, RFC 5065 §5.2/§5.3 and RFC 6793 (FRR `tests/bgpd/test_aspath.c` as an edge-case checklist only); sizes; profile provenance | +| BG2 | `fsm` and session modes (needs F2a, F3), capabilities, collision, timers, fast external failover, NHT connect gating | Establishment matrix for every row of §5.3; collision; hold negotiation; backoffs; teardown and flush | +| BG3 | `inbound`, `nexthop`, `frr` and `rfc4271` flavors, multipath, `locrib`, `zebra` (needs F2b; performance exit F1c, F2c) | Decision tables; scenario 8; direct and indirect next hops; BGP-over-BGP; `via_metric`; ratio benchmarks | +| BG4 | `outbound`, the `policy` adapter and hooks (RP2: export, table-map, redistribute, reevaluation), MRAI and coalesce (with `bgp.out.mrai_withdrawals`), FIB ACK gate profiles and inactive export on F8, advertisement-delay, `origination`, soft reset and refresh paths, `maximum-prefix` | Scenarios 1–4, 9, 11, 12, 13 (except IX); message counts (echo, withdrawals first); `sonic` preset timing and withdrawal effects of suppress-fib-pending; `@covers` tests for every key BG4 introduces | +| BG5 | route reflection | Scenarios 5, 6 | +| BG6 | confederations | Scenario 7 | +| BG7 | conformance and dynamics: scenario 10; RP5 (the canonical JSON policy codec, explanations, and the typed FRR route-map subset adapter with the corrected `ip protocol` and `ip nht` mappings and anchored AS-path regex lowering); `netsim/adapters/frr.py` (a bounded `frr.conf` subset: load and dump); SONiC BGP tables (`BGP_NEIGHBOR`, `BGP_GLOBALS`, `DEVICE_METADATA.bgp_asn`); the optional FRR containerlab lane (10.5.4, 10.7.1, master by digest) with committed goldens; the Z5 and N6 black-box checks that decide the quarantined items | Scenario pass list; lane report; goldens replayed by `make check` without dependencies | + +### 5.13 FIB programming, acknowledgement gating and advertising inactive routes + +These two functions are required (`RFCs/FIB-ACK-AND-INACTIVE.md`): gating an +advertisement on forwarding installation, and advertising a valid BGP best path +that another protocol displaced in the main RIB (RFC 4277 §11). The +routing-policy plan's `FIB_INSTALL` fixtures E09 and E14 depend on the same +programming contract, so it becomes foundation slice F8. + +**Programming contract (F8).** +- Each (device, table, family, prefix) keeps a desired action and an installed + action. +- Requests carry `(device, table, family, prefix, generation, request_id, + action)`, and completions name a configured boundary (`control`, `kernel` or + `hardware`). +- Delay, capacity failure, explicit error, lost completion, retry and + withdrawal are observable. A timeout is never turned into success. +- A completion for an old generation, another table or an obsolete request is + ignored, and a withdrawal cancels outstanding obligations. +- The hardware model is per-node settings: `device.fib.latency_s`, which + supersedes the current `fib_delay`, and `device.fib.capacity`, + `device.fib.failure_model` and `device.fib.multipath_completion` (partial or + atomic). +- `RowNotice` carries the request generation, so stale completions cannot + satisfy a gate. + +**Settings** (per node; the bundle is one override unit named +`fib_advertisement`): + +| Key | Values | Default | +|---|---|---| +| `bgp.export.inactive` | bool: may a BGP best path that lost the main-RIB election be exported? | True in the FRR dialects, which advertise the best path regardless of the zebra winner unless suppress-fib-pending (`bgp_route.h:771-806`); False in `native`, as Junos defaults | +| `fib.ack.gate` | `off`, `selected`, `effective`, `strict` | `off`. The three gate profiles follow the analysis: `selected` gives an inactive route no own-install obligation (recorded as `not_applicable`); `effective` waits for the forwarding action that will carry attracted traffic; `strict` requires the route's own installation | +| `fib.ack.boundary` | `control`, `kernel`, `hardware` | `kernel`, and `hardware` under `sonic`, because zebra runs with `--asic-offload=notify_on_offload` | +| `fib.ack.post_delay_s` | seconds | 0; 1 s under FRR suppress-fib-pending batching | +| `fib.ack.loss_action` | `withdraw`, `keep` | `withdraw` | + +**FRR compatibility.** `bgp suppress-fib-pending` maps to the profile +`frr-suppress-fib-pending`: `strict` scope, limited as FRR documents it. +- Only newly peer-learned routes are gated. Later attribute or best-path changes + and redistributed routes are excluded, and a later generic dataplane deletion + does not cause a withdrawal (FRR 10.5 manual). +- The withdrawal predicate returns "withdraw" for any non-selected non-NORMAL path before it reaches the selected-`network`-static exemption (`bgp_route.h:779-801`); the exemption applies only when no such other path exists. +- When a better-administrative-distance route wins, the BGP route is withdrawn + (`bgp_check_withdrawal`), so inactive export is effectively off. + +The exact 10.5.4 and master behaviour is a lane fixture before it earns a +compatibility label. + +**Open/R.** PrefixManager's readiness gate checks membership of programmed +prefixes (`PrefixManager.cpp:650`, withdrawal near line 710). It does not check +an exact next-hop generation, and the model reproduces that weaker gate for +Open/R. + +**Acceptance.** Fixtures G01–G10 of the analysis, plus the coupled pairs to test +explicitly: +- FIB gate × inactive export; +- startup hold × GR; +- MED strategy × reflection; +- resolver source policy × route leaking (when VRFs land); +- weight source × missing weights. + +--- + +## 6. Integration scenarios (IX) + +These cover every deployment shape in the brief. Each runs a failure, reports +convergence timing (Open/R debounce, FIB delay, NHT, BGP process hold, MRAI) +and transient loss from placement, and compares against an oracle where one +exists. + +1. **One IGP domain over every device** (Open/R), with iBGP over loopbacks in a + full mesh, then through RRs. Next hops are indirect. +2. **Per-device domains (no IGP)** with eBGP per hop, as in RFC 7938: + unnumbered, multipath-relax, next hops direct. +3. **Several IGP domains** (per AS or region), eBGP between them, iBGP inside. + A failure in one domain reprocesses only that domain's BGP paths. +4. **Hop-by-hop iBGP RR with `next-hop-self force` instead of an IGP**, plus an + overlay RR peering over loopbacks learned from BGP (BGP-over-BGP). +5. **A confederation**, first with an IGP domain per sub-AS (confed peering on + link addresses, or on loopbacks with `ebgp-multihop` when not adjacent), + then with one IGP + across the confederation. +6. **Open/R multi-area** with BGP injected into Open/R (`PrefixType.BGP`, RIB + feed `kind='all'`) and Open/R redistributed into BGP. A two-border case + shows that the injection loops without tags and holds with them. The same + prefix learned over BGP and redistributed locally is decided by + `bgp.bestpath.redist_admin_distance`, run under both dialects. +7. **Per-node settings.** One fabric runs a network layer, a spine group layer + and a single-leaf override, and `explain()` output is asserted. A mixed + deployment puts the `sonic` preset (`frr-10.5`, traditional) on the leaves + and `frr-modern` with the datacenter profile on the spines. It shows the + timing effect of suppress-fib-pending and its advertisement effect: a + selected but not installed path is withdrawn. A run-time settings change on one device re-resolves + only that device, and its apply action is visible in the timeline. + +### 6.1 Configuration sketches + +Names are indicative. + +```python +from netsim.agents import bgp, openr +from netsim.agents.common.selection import FlavorSpec +from netsim.model.igp import IgpDomain, oracle_igp_domains + +# (1) One Open/R domain over the fabric; iBGP over loopbacks to two RRs. +net.settings.layer('fabric', values={'device.dialect': 'frr-modern', 'device.profile': 'datacenter'}) +openr.deploy(net, devices=net.devices, # upstream defaults: RTT metric, soft-drain at boot + enable_v4=True, # upstream default is off + areas=(openr.AreaConfig('0', include_interface_regexes=('eth.*', 'Po.*'), + redistribute_interface_regexes=('lo.*',)),)) +clients = bgp.PeerGroup('clients', remote_as='internal', update_source='lo', + route_reflector_client=True) +for rr in ('RR1', 'RR2'): + net.add_agent(rr, bgp.BgpAgent(bgp.BgpConfig( + asn=65000, cluster_id='10.0.0.100', peer_groups=(clients,), # sugar: writes device and object layers + neighbors=tuple(bgp.Neighbor(loopback(c), group='clients') for c in edge)))) + +# (2) No IGP: eBGP per hop over unnumbered links (RFC 7938), SONiC-like leaves. +net.settings.group('sonic-leaves', select=label('leaf'), priority=10, values={'device.presets': ('sonic',)}) +for dev in fabric: + net.add_agent(dev, bgp.BgpAgent(bgp.BgpConfig( + asn=asn_of(dev), + neighbors=tuple(bgp.Neighbor.interface(p, remote_as='external', + activate=('ipv4-unicast', 'ipv6-unicast')) # sonic: no default ipv4-unicast + for p in ports(dev)), + redistribute=(bgp.Redistribute('connected', policy='LOOPBACKS'),)))) + +# (3) Two IGP domains on the fast oracle underlay, eBGP between them. +net.add_source(oracle_igp_domains(( + IgpDomain('as1', devices=as1, exclude=('ext.*',), instance=1), + IgpDomain('as2', devices=as2, exclude=('ext.*',), instance=2)))) + +# (4) Hop-by-hop iBGP instead of an IGP: every router reflects to its neighbours. +for dev in fabric: + net.add_agent(dev, bgp.BgpAgent(bgp.BgpConfig( + asn=65000, networks=(loopback(dev),), + neighbors=tuple(bgp.Neighbor(peer_address(dev, p), remote_as='internal', + route_reflector_client=True, next_hop_self='force') + for p in ports(dev)), + multipath=bgp.Multipath(ibgp=8, equal_cluster_length=True)))) + +# (5) Confederation: member AS 65001 of AS 100, peering with member AS 65002. +bgp.BgpConfig(asn=65001, confederation=bgp.Confederation(identifier=100, peers=(65002,)), ...) + +# (6) Two Open/R domains on one border router (netsim_ext.multi_instance): disjoint interfaces. +net.add_agent('B1', openr.OpenrAgent(cfg_dc1, instance=0), name='openr-dc1') +net.add_agent('B1', openr.OpenrAgent(cfg_dc2, instance=1), name='openr-dc2') + +# Decision flavors: by name with knob overrides; custom specs carry their own refs. +bgp.BgpConfig(..., decision=FlavorSpec.use('rfc4271')) +bgp.BgpConfig(..., decision=FlavorSpec.use('frr', always_compare_med=True, # knobs are sugar for settings + deterministic_med=True, dmed_semantics='classic')) +bgp.BgpConfig(..., decision=FlavorSpec( # a custom flavor: built-in steps plus a user step + 'lab-prefer-short-cluster', selection='elimination', + steps=('local_pref', 'as_path_length', 'origin', 'med', 'ebgp_over_ibgp', + ShortestClusterListFirst(refs=('RFC 4456 §9',)), # frozen, pure, carries refs + 'igp_metric', 'router_id', 'peer_address'), + refs=('NetSim plan §5.5',))) + +# Settings are layered and overridable per node (section 2A); changes re-resolve only affected devices. +net.settings.layer('dc-defaults', values={'device.dialect': 'frr-modern', 'device.profile': 'datacenter', + 'bgp.timers': (3, 9)}) # a unit is written whole +net.settings.group('spines', select=label('spine'), priority=10, + values={'bgp.af[ipv4-unicast].maximum_paths': 64, + 'bgp.bestpath.as_path.multipath_relax': True}) +net.settings.device('leaf7', values={'bgp.neighbor[10.0.0.1].timers': (10, 30)}) +print(net.settings.explain('leaf7', 'bgp.neighbor[10.0.0.1].timers')) # value and its layer chain +# Changes are tree updates (CAUSE_CONFIG with changed keys and apply actions); commands are exec-only. +sim.at(30, lambda: net.settings.device('RR1', values={'bgp.neighbor[10.0.0.7].shutdown': True})) +sim.at(35, lambda: net.settings.device('R9', values={'device.dialect': 'frr-10.5'})) # upgrade/downgrade: RESTART_INSTANCE +sim.at(40, lambda: sim.command('R3', 'openr', 'set_node_overload', True)) +sim.at(50, lambda: sim.command('R1', 'bgp', 'clear_soft_in', '10.1.12.1')) +``` + +--- + +## 7. Verification + +- **Invariants** (additions to `tests/test_invariants.py`): + - protocol packages import only contracts, addressing, routing records, + `state.PMap`/`record` and `agents/common`, checked at symbol level; + - every step and profile value has refs, and every public rule function in + the protocol packages carries `@ref` (section 13.8); + - `docs/traceability.md` (RFC section → functions → tests) is fresh, and + every cited section has a test; + - plugins read configuration only from `ctx.agent_config`; + - agent states, views and messages are frozen; + - no module-level mutable state in protocol packages; + - flavors are deterministic given inputs and history. +- **Locality and projections.** The locality harness runs for both agents, and + guarded projections run per submodule. +- **Views only.** Tests assert on published views and on RIB/FIB/placement, + never on agent state. +- **Property tests.** + - KvStore merge convergence. + - BGP under Gao–Rexford-safe policies converges to the unique stable state of + a small static solver in the tests. + - RR equals a full mesh, and a confederation equals a full mesh, where RFC 4456 + and RFC 5065 guarantee it. + - The `rfc4271` flavor gives the same result for any arrival order. +- **Conformance.** The scenario list and the FRR lane goldens. + - The lane needs privileged Docker and about 50–100 MB per FRR container, and + polls until converged. + - Order-dependent scenarios (RFC 3345 under `frr`, DISAGREE) get no goldens; + they assert the set of admissible outcomes instead. +- **Performance.** + - CI asserts operation counts, never wall time: `resolve_prefix` calls, + entries touched, admission visits, NHT refreshes, agent runs per event. + - Wall-time ratios live in slow, deselected benchmarks and passports. + - Performance claims need old-versus-new runs on the same quiet machine. +- **Generators.** `hypothesis` is not a dependency. Property tests use seeded + generators with fixed seeds and bounded sizes, each under 5 s, well inside the + 60 s per-test hang guard. +- **Determinism.** + - CI runs 3.11–3.14 and 3.14t (`.github/workflows/python-test.yml:12`). + - Dirty sets are processed in sorted order. + - A test runs key scenarios under two `PYTHONHASHSEED` values and compares + fingerprints and event streams. +- **Checks.** `make check-ci` (coverage ≥ 85%) and `make check-ft` for every + slice; local checks do not replace CI. + +--- + +## 8. Execution plan + +1. **Design review.** Three internal reviewers on revision 1 (section 11), then + Codex through the Superset CLI on revision 4. The Gate C reviewer terminal + has ended, so a fresh Codex reviewer is used. Findings are applied before F0. +2. **F0** by the integrator: contract records, defaults and stubs, committed + first as the skeleton. F0 includes the records F1 and F2 share (per-client + outcome map, per-owner NHT maps, recursive-next-hop and group records, + `FibGroupEvent`). It also includes the settings stubs every worker builds on: + `NetworkState.settings`, `DeviceState.labels`, the `Network.settings` API + skeleton, `CAUSE_CONFIG`, `Command` and `AgentNode.persistent`. +3. **Foundation workers** on isolated worktree branches. Merge order: + F0 → F2a, F3, F4, F5, F7 (parallel) → F1a and RP1 → F1b → F2b and F8 → F2c, + F1c and RP3 (parallel) → RP4. A design review gate sits after F1b, before F1c. + F6 runs only if IX profiles require it. + + | Worker | Files owned | + |---|---| + | F1 (a, b, c) | `model/routing.py`, `model/forwarding.py`, `model/lpm.py`, `model/derive.py` (FIB kind, `bump_epochs`, NHT refresh selection), `runtime/timeline.py`, `tests/test_fingerprints.py` | + | F2 (a, b, c) | `runtime/agents.py`, `validate_admitted` and the `PMap` builder in `model/state.py`, NHT delivery, `installed` mode and `via_metric` computation in `model/nht.py` (reading F1b's recursive-next-hop table), F0 delivery in `runtime/simulation.py` | + | F3 | `runtime/transport.py`, `runtime/channels.py` | + | F4 | `model/igp.py` | + | F5 | `agents/common/`, `tests/agents/harness.py` | + | F8 | programming records in `model/forwarding.py` and the FIB kind in `model/derive.py` (with the F1 owner); completion notifications in `runtime/agents.py` (with the F2 owner) | + | RP1 | `netsim/model/route_policy/` (types, compiler, evaluator, capability table) and `tests/model/route_policy/` | + | RP3 | retained submission and admitted views and the `RIB_ADMISSION` hook in `model/routing.py` (coordinated with F1b's submission boundary) | + | RP4 | `RESOLUTION_IMPORT` scopes in the resolver and NHT, and `FIB_INSTALL` exclusion on F8 | + | F7 | the `netsim/settings/` engine (schema, layers, resolution, inheritance, provenance, documentation generators), the composition-root catalog, the `native` dialect with `device.*` and `zebra.*` fragments, NetGraph label translation in `adapters/ngraph.py`. The `NetworkState.settings` and `DeviceState.labels` stubs and the `Network.settings` skeleton come from F0 | + + `connected_only` parity touches `_CandidateResolver` in `routing.py`; the + F1 owner makes that change and F2 supplies the tests. Integration uses the + extended fingerprints (F1a) as the equivalence oracle, runs full checks on + both interpreters, and ends with an adversarial review round. +4. **Two tracks.** Every protocol slice has separate functional and performance + exits, so functional work never waits for F1c or F2c. + + | Slice | Needs (functional) | Needs (performance exit) | + |---|---|---| + | OR1, OR2 | F0, F2a, F5, F7 | F2c | + | OR3 | OR2, F2b | F1b, F2c | + | OR4 | OR3 | F2c | + | OR5 | OR3, RP1 (F2a commands and persistence, F2b RIB feed) | F2c | + | OR6 | OR5 | F1c, F2c | + | BG1 | F5, F7 | – | + | BG2 | BG1, F2a, F3 | F2c | + | BG3 | BG2, F2b, F4, RP1 (import hook) | F1c, F2c | + | BG4 | BG3, RP1, F8 (suppress-fib-pending, ACK gate, inactive export) | F1c, F2c | + | BG5–BG6 | BG3 | F1c, F2c | + | BG7 | BG6 | – | + + - OR1–OR6 own `agents/openr/`. + - BG1–BG7 own `agents/bgp/`, `adapters/frr.py` and the BGP tables in + `adapters/sonic.py`. + - Reviews run after OR3 and BG4 and at each track's end. +5. **IX**, then README and reference documentation, then a final review. +6. **Orchestration** as in Gates B and C: + - Codex xhigh workers created with `superset workspaces create --local + --project --branch --base-branch main --agent codex + --effort xhigh --prompt ...`; + - progress read with `superset terminals read`; + - findings routed back to the owning workers; + - commits and pushes only when the user asks. + +--- + +## 9. Risks and open questions + +- **FRR sorted-list semantics** (deterministic-MED, no re-sort on withdrawal) + rest on source reading. The lane confirms them, and both `dmed` semantics are + offered. +- **Master is a moving target.** `frr-modern` resolves to the immutable pin + `frr-master-f1d7bec`. Z5 is quarantined until a black-box run, and N6 is + revisited with UCMP in BG8. At 10.8.0 the per-subsystem diff review repeats, and the dialect + table changes only by data rows. +- **Dialects do not emulate bugs.** A 10.5.4 lane run will disagree wherever + 10.5.4 is buggy (section 2B.1). Those disagreements are triaged against the + divergence list, and a `quirk.*` flag is added only under the 2B.2 rule. +- **Other FRR quirks** are recorded as named deviations: + - the lower neighbour address wins, contrary to the documentation; + - `bandwidth ignore` still weights; + - a static route passes the connected check, yet the non-recursive install + over it stays inactive; + - no send-side check for a next hop equal to the receiver's address; + - RFC 8212 exempts confed peers. +- **Open/R route metric** is a declared modelling choice (`metric_mode`) that + decides whether hot potato over Open/R is observable. +- **Open/R defaults** (RTT metric, soft-drain at boot, empty default area, + `enable_v4` off, the `"0"` wildcard) make naive comparisons fail. Profiles and + deploy helpers handle this. +- **SONiC's FRR.** The template is verified (`bgpd.main.conf.j2`, saved in the + research folder: suppress-fib-pending, no RFC 8212, `maximum-paths 64`, GR on + ToR only). The patch series still needs a review for behaviour changes, for + example the 10.5.5 backports it may carry. +- **Python performance** at 100k prefixes × many peers. Passports set + expectations; native kernels stay deferred. +- **Hierarchical FIB.** Group-id semantics change; SR entries migrate last. +- **Licensing.** No FRR code, and topotest data is not vendored. +- **No multi-access LANs.** + +--- + +## 10. Deferred (named extension points) + +- **BGP (BG8 and beyond):** + - `cisco-ios`, from Cisco document 13753: weight; local origin; oldest eBGP + path; router id before cluster length. + - `junos`, from Juniper "Understanding BGP Path Selection": no weight; + strictly internal paths first; oldest external path; router id before + cluster length; MED grouped by neighbouring AS unless + `cisco-non-deterministic`. + - add-path (RFC 7911). + - graceful restart, EoR and update-delay (RFC 4724, 8538). + - BFD as an agent publishing views (RFC 5880, 5883). + - link-bandwidth UCMP. + - aggregates. + - dynamic neighbours. + - SR color steering (RFC 9256 §8) and SRv6 services (RFC 9252, which needs + VRFs). + - an analytical BGP solver. + - RFC 9107 optimal route reflection (absent in FRR too). +- **Open/R:** KSP2 and UCMP (absent upstream); the RIB policy API; an RTT step + detector model. +- **Common:** a control-plane CPU model; incremental SPF; native kernels by + profile. + +--- + +## 11. Review log + +**Revision 1 → 2 (2026-09-27).** Three internal adversarial reviews, each +checked against source; spot checks reproduced by the integrator. + +- **Architecture and contract fit** (16 findings; 1 P0, 11 P1, 4 P2, all + accepted): + - P0: configuration becomes tree state (D11), because the runtime reads + `AgentNode.config`, and commands become exec-only. + - The RIB feed has a precise definition (processed epoch, installed rows, + snapshots, coalescing, kinds, tags). + - Change-key semantics per receipt. + - `RouteView` split. + - `run_delay > 0` for protocol profiles. + - Flavors become frozen specs, since `validate_immutable` rejects callables; + `Dest` history is explicit. + - Open/R worklist with programming-gated edges. + - The idle claim is restated. + - Listener accept tables. + - Views (G10). + - Multi-instance becomes `netsim_ext`, separated by interface ownership. + - Lazy accessors from `LocalContext`, and NHT costs (G5). + - Module projections, RNG streams, timer multiplexing. + - The iBGP-on-link row corrected. + - Restart semantics (D12). +- **Protocol fidelity** (25 findings; 2 P0, 13 P1, 10 P2, all accepted): + - P0: the FRR pin is frr-10.5.4, where step 0 is absent and the LLGR_STALE + pre-step is present. + - P0: MED-oscillation expectations are per flavor. + - The sorted-list rules are written down. + - The `rfc4271` flavor is completed (§9.1.1, §9.1.2, RFC 5065 §5.2 and all + four §5.3 rules). + - `MULTIPATH_NUM` per profile (16 / 256 / 514). + - Fast external failover. + - The F3 TTL test is corrected; NHT gating moves to BG2. + - Confed TTL 1. + - Router-id rule. + - Receive-side AS_PATH checks. + - Route-map delay timer, soft reset, `maximum-prefix`. + - Multi-instance labelled. + - Open/R route metric declared as a modelling choice. + - `bestNodeArea` default rule. + - Oracle-equivalence configuration. + - Scenario 6 expected set. + - Idle-hold doubling. + - zebra queue simplification recorded. + - Unnumbered simplifications recorded. + - Send-side MED, LOCAL_PREF and link-local rules. + - Echo wording. + - Area `"0"` warning. + - RTT floor. + - Restart semantics. + - Research report corrections appended as errata sections: + `research-frr-decision.md` (version pin, finding 1 holds for bare builds + only, missing fast external failover and confed TTL); + `research-frr-rr-confed.md` (§2.6 row 9, the §3.3 RFC section, idle hold); + `research-openr.md` (§6 step 4, distance source). +- **Scale, tests and execution** (13 findings; 3 P0, 8 P1, 2 P2, all accepted; + the P0s reproduced by the integrator): + - P0: the full resolver is order-dependent (G12), so F1a comes first, with + fingerprints extended beforehand. + - P0: `depends_on` is transitive (G13), so F1b makes the FIB non-transitive, + with a refined group key and reference-counted group garbage collection. + - P0: the missed Simulation-path O(table) costs (G14): persistent FIB storage, + `diff_pmap` timeline diffs, `FibGroupEvent`, NHT refresh by watch index. + - The ≤ 5 ms target is scoped to the FIB step, with both harnesses stated. + - `zebra.py` suppresses identical rows. + - Exact shard-identity admission rule and guard tests; target ≤ 100 µs at + 100k with nested maps. + - Lazy context without a cache; per-client outcome maps and per-owner NHT maps + in F0. + - GC multiplier and heap counts in passports. + - Cost model; passport (b) re-scoped. + - Operation-count assertions in CI; seeded generators; `PYTHONHASHSEED` + determinism; CI matrix corrected. + - FRR lane needs; no goldens for order-dependent scenarios. + - Merge order F0 → F3/F4/F5 → F1a → F1b → F2 ∥ F1c, file ownership fixed, + functional versus performance exits per slice. + - F6 for the conservative downstream scans (G15). + - Watch-index range queries by bisect. + +**Revision 3 → 4 (2026-09-27).** + +- **Verification pass on revision 3** (7 P1, 14 P2, all applied): + - F2 split into F2a, F2b and F2c, with real functional dependencies; + - G2 and G3 numbers made consistent; + - the F1 Simulation target scoped to the FIB share; + - the RIB feed now carries metric-only and tag-only changes, with the + `kind='all'` trigger defined; + - citation convention stated, with 10.5.4 citations for `peer_ttl_update` and + fast external failover; + - SONiC `maximum-paths 64` capped by 514; + - the confed loopback statement corrected; + - "entry criterion" reworded as a performance exit; + - `FlavorSpec.use` defined, the no-cache flavor resolution, a custom-flavor + example; + - per-watch coalescing; + - non-BGP import-check; + - scenario 4 static and `disable-connected-check` cases; + - `enable_v4` in the Open/R example; + - G12 wording; + - `LOOP_DETECTED` in `RowNotice`; + - NHT `installed` mode ownership; + - the confed-learned → iBGP send row. +- **User direction (2026-09-27):** target the most modern FRR and keep a precise + record of 10.5 differences; generic differences become settings; every setting + is overridable per node; plan for maintainability. + - Three per-subsystem diffs of frr-10.5.4 against master, each difference + attributed to its release through the full history + (`frr-diff-decision-nht.md`, `frr-diff-advertise-policy.md`, + `frr-diff-sessions-defaults.md`). + - Section 2B: findings; dialects `frr-modern` (master `f1d7bec` with Z5 and + N6 quarantined) and `frr-10.5` (defaults and choices, not bugs); the quirk + rule; the settings table; profile defaults; FRR's defaults and inheritance + machinery mirrored. + - Section 2A: the settings system (schema, dialects, layers in the tree, + FRR inheritance, typed interned resolution with provenance, apply actions, + test-enforced rules, adapters, failure modes), delivered as slice F7. + - Corrections: + - `deterministic-med` changed in 10.4.0, not 10.5.0; + - "withdrawn first" and "MRAI gates withdrawals" are modern-only; + - martian NEXT_HOP is treat-as-withdraw in modern; + - a peer AS equal to the confed id is iBGP in modern; + - GR-stale paths with dead next hops are deleted in modern; + - SONiC's template enables suppress-fib-pending, now in BG4 scope. +- **Review of sections 2A and 2B** (2 P0, 15 P1, 7 P2; all applied; the SONiC + traditional-profile claim verified: no `-F`, no `frr defaults` line): + - P0: precedence is defined per (object, key), with dialect, profile and + presets as device meta-keys resolved first, and FRR inheritance over + objects; + - P0: a single write path: the resolver is the only writer of `AgentConfig` + and the `DeviceConfig` settings fields, `add_agent` configurations are + sugar, and a `debug_validate` re-resolution check; + - interning via record reuse and per-call tables, not identity memos; + - selectors use `DeviceState.labels`; + - a `native` dialect keeps today's defaults and fingerprints; + - the Apply enum is completed (`DEPLOY_ONLY`, `CAPABILITY_ELSE_RESET`, + sort-changing `SESSION_RESET`, `SOFT_OUT` semantics); + - `since` and `allowed_values_by_dialect` replace per-dialect rejection; + - dialect ids are immutable pins behind aliases, and `frr-10.7` is dropped; + - only Z5 is quarantined; N6 and the link-bandwidth clamp are not quirks; + - `zebra.rib_tiebreak` is deferred to F1a; + - suppress-fib-pending withdraws selected but not installed paths; + - the `sonic` preset corrected: traditional profile, conditional + `maximum-paths`, role-conditional lines written by the adapter, GR on ToR as + a divergence, ASIC-offload notification; + - distribution replaced by ordered presets, and `lab` becomes a preset; + - partial unit writes rejected; + - the whole `AgentConfig` is resolved, including run-time fields; + - non-settings and lints listed; + - `deterministic_med` boolean kept, with `dmed_semantics`; + - slice ownership fixed (F0 stubs, F2a, F1 timeline, BG7 adapters, fragments + in BG1 and OR1); + - enforcement rules rewritten as an AST check and a key-field bijection with + `@covers`; + - DefaultsSpec joint first-match with no gates below 10.5; + - `mrai_withdrawals` wording and instance scope; + - structured values replace; + - provenance on demand; + - the key grammar; + - the composition root; + - report conflicts recorded (N1 over S24, the 0–10000 ms range, AP-20 over + S18, UPA receive marking). + + + +**Revision 4 → 5 (2026-09-27): integration of the reviewed routing-policy plan.** + +Sources: +- `NetSim/docs/plans/routing-policy.md`, revision 4, with + `routing-policy-reviews.md` (36 findings), written by a separate session. + Both are untracked in the NetSim repository; committing them is the user's + decision. +- `RFCs/FIB-ACK-AND-INACTIVE.md`. +- `RFCs/PER-NODE-SETTINGS-PLAN.md`. + +**Adopted as written:** +- the shared typed language and one evaluator in the model layer (D13, section + 5.8), replacing the BGP-local route-map evaluator; +- four flows, explicit fallback, absent versus empty binding, + `ORIGINAL`/`WORKING` match input; +- the closed field and action catalog, with owner-supplied `PolicyAccess`; +- the capability table per point; +- no regex, loops, calls or callbacks in the core; +- digest-pinned revisions; +- retained submission views for RIB admission, and the single RIB submission + boundary with its callsite audit (F1b, RP3); +- desired versus installed FIB with generation-guarded completions (F8, D14); +- owner-defined reevaluation replacing the fixed 5 s delay (`REEVALUATE`), with + import replay needing retained input or ROUTE-REFRESH, else + `needs_session_reset`; +- update groups keyed by compiled fingerprint plus peer facts; +- `RoutePolicyRef` distinct from SR `PolicyRef`; +- node-local client profiles (P0), which moves client profiles from the + non-settings list into per-node settings; +- fixtures E01–E24; +- the FIB acknowledgement gate and inactive-export contract with fixtures + G01–G10. + +**Corrected during integration, checked in FRR source:** +1. **`ip protocol P route-map`.** It is evaluated per next hop inside zebra's + next-hop activation (`zebra_nhg.c:2880-2921`). A deny deactivates that leg, + and a route with no active leg is ineligible, so zebra selects the next + candidate. The routing-policy plan's native `FIB_INSTALL` (whole action, no + fallback) cannot represent it. The FRR construct maps to `RIB_ADMISSION` + with a new `per_nexthop=True` mode, and `FIB_INSTALL` stays native only. +2. **`ip nht P route-map`.** It filters the next hops of the resolving route + zebra has already chosen, for one client protocol. If all are filtered, the + client is told "filtered/unresolved" with no less-specific fallback + (`zebra_rnh.c:515-576`), and forwarding recursion is unaffected. + `RESOLUTION_IMPORT` gains an explicit scope (`nht` or `forwarding`), and the + FRR adapter uses `nht` with `fallthrough='never'`. +3. **Inactive export default.** FRR advertises its BGP best path regardless of + the zebra winner unless suppress-fib-pending, in which case + `bgp_check_withdrawal` withdraws non-installed selected paths + (`bgp_route.h:771-806`). So `bgp.export.inactive` defaults to True in the FRR + dialects and False in `native`. +4. **SONiC completion boundary.** SONiC's zebra runs with + `--asic-offload=notify_on_offload`, so the `sonic` preset uses + `fib.ack.boundary='hardware'`. +5. **Anchored AS-path regexes.** Common SONiC and FRR configurations use + anchored AS-path regexes. The FRR adapter lowers the anchored single-ASN + shapes to typed predicates instead of rejecting every regex; other patterns + are rejected with a report. +6. **FRR deny entries with `set` clauses.** FRR ignores the sets. The adapter + drops them and records a deviation, because native `REJECT` terms cannot + carry actions. +7. **Reevaluation defaults per dialect.** The FRR dialects use 5 s, the + `route-map delay-timer` default. `native` uses 0 s. FRR's delay-timer of 0 + maps to `manual`. + +**Scheduling.** +- P0 goes to F0 and F7. +- P1 becomes RP1 in M1. +- P2 goes to BG3 and BG4. +- P3 becomes RP3 and P4 becomes RP4, both in M2, with the new F8. +- P5 goes to BG7. +- P6 goes to M5. +- Dependencies, file ownership, milestones and roadmap signatures were updated + accordingly (sections 3, 8 and 12). +- Scenario 9 now includes E01–E24's BGP-side fixtures; new scenario 13 covers + installation gating. +- Sections 13.11 and 13.12 answer the two new requirements. + + +**Revision 5 → 6 (2026-09-27): scale review with experiments and prototypes** +(section 14; scripts in `scale-experiments/`). +- Measured: 1.5 KB and 84 µs per recursive row; quadratic build (G17); 27 KB + retained and 160–200 µs per idle agent run (G18, G19); 25 KB per device with + no routes (G20); `PMap` admission 3.15 s per change at 1M; DES core 0.9 µs + per event. +- Proved: linear build by indexing the locator check; 1.8 KB retention without + the RNG commit; shard-identity admission 1.5 ms at 1M, hence a changed-key + log; a columnar hierarchical FIB at 13 B per route with 5 µs underlay changes; + a struct-of-arrays Loc-RIB at 17 B per path; columnar LPM rejected. +- Decided: S0 scale hygiene first; zero-commit idle runs (runtime bookkeeping + out of `AgentNode`); columnar FIB and BGP payloads with interned attributes; + lazy analytical mode at 100k devices; the honest passports P-A, P-B, P-C; + streamlining (F2a into F0, F7 in two steps, RP4 and parts of F8 deferred, + JSON codec dropped, lane on two images). + + +**Revision 6 → 7 (2026-09-27): Codex adversarial review of revision 6** +(`codex-review-rev6.md`; 1 P0, 17 P1, 4 P2; all accepted; source claims +re-verified by the integrator; the P0 answered with `proto_chunked.py`). +- P0: packed storage gets an immutable persistent contract (`PackedTable`, + §15.1): 16 B per row, 10 µs per single-row update, 0.1 ms admission. +- Passports re-budgeted with every index and component (P-A ≤ 4 GB, P-B + ≤ 3 GB); measurement labels corrected (DES 1.5 µs per timeout; `RowKey`, not + `Route`; allocation increments); the columnar-LPM rejection withdrawn. +- Zero-commit publication becomes Gate C slice C6; the `PMap` changed-key + certificate and `PackedTable` become slice S1; the SRv6 consumer copy joins + S0; lazy analytical mode becomes slice A1 with a query contract. +- Per-leg cycle rule (F1a); `ip protocol` as an activation-stage filter; scope + in shared recursive keys; no import-policy memoization. +- Root settings delta and dump of explicit intent; one action plan per + configuration transaction; preset precedence examples. +- Group versions and ERROR completion in minimal F8; Open/R sync completion + including the empty sync; scoped listener accept identity. +- BGP corrections: reinstall guard, ORIGINATOR_ID only if absent and outbound + policy next-hop rewrite, confed step deferral under peer-type relax, + suppress-fib-pending predicate order, multipath ceiling as package data. +- Streamlining: two explicit selection functions instead of a flavor + interpreter; smaller settings start; §15.9 is the single authoritative + dependency table; readiness: S0 and additive F0 now, everything else after + the `PackedTable` and C6 gates. + + +**Revision 7 → 8 (2026-09-27): second Codex review** (`codex-review-rev7.md`; +1 P0, 14 P1, 1 P2; all accepted; body contradictions fixed in place; §16 records +the decisions). +- P0: positional identity across compaction; the first `PackedTable` is + append-only with logical or generation-checked references and a bound + change certificate. +- Index choice per device class after update benchmarks; the common RIB + assigned to the X owner; BG3 without CSR; budgets as hypotheses with + per-component reporting. +- Certificates as immutable evidence; S1 split into S1a and S1b. +- C6 publication seam before root observers; `initialized` stays in the tree; + per-stream PRNG counters (sent to worker W3). +- Per-leg witness recurrence; activation filter on the original next hop. +- Route and group obligations; sync operation identity with empty and total + failure; per-key feed coalescing with bounded payloads. +- F0 owns the root settings delta; preset precedence decided; dumps keep + explicit intent; one action plan per transaction. +- RFC 8212 discards and suppresses rather than refusing. +- §15.9 edges and owners amended (§16.9); readiness gates must pass, not + merely exist. + + +**S0 delivered (2026-09-27)** on `main` locally: see §17. Two Codex review +rounds ran in parallel with the implementation (§15, §16). + +--- + +## 12. Implementation roadmap: how each slice is built + +This section turns the design into buildable work. Each slice lists files, +the key types and function signatures, the tests written first, and the exit +check. Signatures are Python and indicative, but binding in shape: pure +functions over frozen records, no runtime objects in protocol modules. Size +estimates are rough lines of code (implementation + tests) for scheduling only. + +### 12.1 Milestones + +| Milestone | Slices | Gate to pass | Rough size | +|---|---|---|---| +| M0 skeleton | F0 | contract tests green, all existing tests green, stubs importable | 0.8k | +| M1 foundation | F1a, F2a, F3, F4, F5, F7, RP1 (parallel after F0) | resolver order tests, extended fingerprints, settings engine on `native` with fingerprints unchanged, transport TTL/accept tests, toolkit unit tests, route-policy truth tables (E01–E03, E10, E12, E19–E22), the two-node client-profile regression passing; review round | 8k | +| M2 scale foundation | F1b → F2b and F8, then F1c ∥ F2c ∥ RP3, then RP4; design gate before F1c | op-count assertions and G1–G5 ratio benchmarks; full-vs-incremental oracle; stale-ACK and programming tests (G01–G04, E09, E14); RIB admission (E04, E13); resolver and FIB policy (E05, E06, E11) | 8k | +| M3 protocol cores | OR1–OR3 ∥ BG1–BG3 | Open/R oracle equivalence (single area); BGP decision tables, session matrix, direct and indirect next hops; review round after OR3 and BG3/BG4 | 9k | +| M4 protocol breadth | OR4–OR5 ∥ BG4–BG6 | multi-area, drain; outbound, policy, RR, confederations; scenarios 1–9, 11, 12 | 7k | +| M5 conformance and integration | OR6, BG7, IX | passports, FRR lane report, IX 1–7 | 5k | +| M6 release | docs, README, final review | generated `docs/settings.md`, reference pages, review with no open P0/P1 | 1k | + +Each milestone ends with an integration merge on main (full `make check-ci` and +`make check-ft`), an adversarial review round (internal reviewers, optionally +Codex), and a memory/plan update. Nothing is pushed without the user's request. + +### 12.2 Test harness (built once, in F5 and F7, used by every slice) + +``` +tests/ + agents/ + harness.py locality guard, submodule projection proxy, twin comparison, view assertions, + op counters (resolve_prefix calls, admission visits, agent runs per event) + topo.py declarative fixtures: build(TopoSpec) -> Network (ring, clos(k), diamond, random(n, seed), + numbered | unnumbered | dual-stack, labels for settings selectors) + openr/ test_spark.py, test_kvstore.py, test_decision.py, ..., test_oracle_equivalence.py + bgp/ test_aspath.py, test_decision_frr.py, test_decision_rfc4271.py, test_sessions.py, + test_nexthop.py, test_outbound.py, test_rr.py, test_confed.py, scenarios/ + settings/ test_schema.py, test_precedence.py, test_inheritance.py, test_apply.py, test_docs_fresh.py + model/ test_resolver_order.py, test_fib_hierarchical.py, test_resolver_incremental.py +tools/frr_lane/ render.py (frr.conf + containerlab topology), run.sh, collect.py, normalize.py, compare.py +``` + +Test layers, cheapest first: + +1. **Pure module tests.** Feed frozen records into one submodule and assert the + returned records. No `Network`, no runtime. Most protocol logic is covered + here (decision tables, merge rules, FSM transitions, AS_PATH vectors). +2. **Agent tests on small fixtures.** `topo.build(...)` plus agents plus + `Simulation`, asserting published views, RIB/FIB and placement only. +3. **Scenario tests.** Named scenarios (section 5.11, section 6) as Python data: + topology, layers, events, expected views at given times. +4. **Equivalence tests.** Protocol twin against the domain oracle; incremental + resolver against the full resolver; `rfc4271` flavor against permuted + arrival orders. +5. **Seeded property tests** (no `hypothesis`): fixed seeds, bounded sizes, + each under 5 s. +6. **Performance.** Op-count assertions in CI; wall-time ratio benchmarks and + passports marked slow and deselected. +7. **FRR lane** (opt-in, Linux with Docker): `tools/frr_lane/` renders the same + scenario for FRR, collects `show ... json`, normalizes, and writes goldens + that the scenario tests replay without dependencies. + +Every test that pins a defect fails on the old code (AGENTS.md); each new +setting has an `@covers('key')` test (rule 5). + +### 12.3 Foundation slices + +**F0: contract and settings skeleton** (integrator). + +- `netsim/model/contracts.py` + ```python + CAUSE_CONFIG = 'config' + @record class Command: time: float; name: str; payload: Any; generation: int | None = None + @record class WatchOp: kind: str; af: int; sources: frozenset[ClientId] | None; watch: str # kind: 'selected'|'all' + @record class RouteChange: watch: str; af: int; epoch: int; upserts: tuple[Route, ...]; removals: tuple[RowKey, ...] + @record class RowNotice: key: RowKey; status: str; reason: str | None; zapi: str + # NhtKey.mode: str = 'rib' ('rib' | 'installed' | 'exact'); NhtResult.via_metric: int | None = None + # AgentOutput: views: PMap[str, Any] = empty; persistent: Any = None; watches: tuple[WatchOp, ...] = () + # AgentNode: views, persistent; SessionOp: ttl: int | None = None; min_ttl: int | None = None; kind 'set_opts' + # Endpoint.address: int | None (None = wildcard, listen only); Listener.accept: PMap[RemoteId, AcceptOpts] # RemoteId(af, address, interface, interface_generation | None) + ``` +- `netsim/model/state.py`: `NetworkState.settings: Any = None` (the layers + record, F7 fills it), `DeviceState.labels: frozenset[str] = frozenset()`. +- `netsim/model/network.py`: `Network.settings` property returning a + `SettingsAPI` whose methods (`layer`, `group`, `device`, `explain`) raise + `NotImplementedError` until F7; `Device.label(*names)`. +- `netsim/runtime/simulation.py`: `Simulation.command(device, agent, name, + payload) -> None` (stub raising until F2a). +- Shared F1/F2 records: `ResolverOutcome.rows: PMap[ClientId, PMap[RowKey, + RowOutcome]]` accessor shim over today's flat map; `RecursiveNexthop`, + `GroupKey`, `FibGroupEvent` record definitions. +- Tests: `tests/model/test_contract_f0.py` (construction, validation, + immutability of every new record; defaults keep old behaviour). + +**F1a: order-independent resolver** (`routing.py`, `tests/test_fingerprints.py`). + +1. Extend the fingerprint (outcomes, MAC, encap, adjacency AF, NHT answers, + selection, PENDING) and add recursive, cyclic, cross-family, fallthrough, + resolve-via-default and SR scenarios; record baselines. +2. Tests first (`tests/model/test_resolver_order.py`): the A↔B cycle yields + `LOOP_DETECTED` for both under every order; a depth-4 chain resolves the same + with and without unrelated deeper members; a seeded fuzz resolves 200 random + RIBs in 20 random prefix orders and asserts identical outcomes. +3. Implementation: `_Resolver.resolve_prefix(af, prefix)` computes chain depth + per recursive next hop (`depth(nh) = 1 + depth(resolving prefix)`), never + memoizes a truncated result, excludes only the prefix itself, and marks + a leg whose only witnesses close a cycle `LOOP_DETECTED`, while the route's + other legs keep their finite witnesses (per-leg rule, §15.4 and §16.4). +4. Fix `DeviceContext.rib` truthiness (`derive.py:728-729`). + +**F1b: hierarchical, persistent FIB** (`forwarding.py`, `lpm.py`, `routing.py`, +`derive.py`, `timeline.py`, `nht.py` refresh selection). + +```python +@record class RecursiveNexthop: key: tuple[int, int, str | None, int | None, bool, bool] # (af, address, interface, if_gen, connected_only, via_default) + legs: tuple[tuple[Adjacency, int], ...] # (leg, depth of its witness) + via_prefix: Prefix | None; via_source: ClientId | None + via_metric: int | None; deps: DependsOn +@record class GroupKey: af: int; nexthops: tuple[Nexthop, ...]; self_prefix: Prefix | None +class PersistentPrefixTable(Generic[T]): set(net, plen, v) -> Self; remove(net, plen) -> Self; lookup(addr) -> ... + diff(other) -> Iterator[tuple[Prefix, T | None, T | None]] +class WatchIndex: add(af, address, key) -> WatchIndex; covered_by(af, prefix) -> Iterator[key] # bisect ranges +def resolve_fib(rib, ctx, policy, version, processed_epoch, old=None) -> tuple[Fib, ResolverOutcome, FibChanges] +``` + +Tests: `tests/model/test_fib_hierarchical.py` (group reuse by key, refcount GC, +self-covering key split, IGP change touches one recursive entry and no FIB +entry, flattened adjacencies equal the old resolver's), timeline +`FibGroupEvent` test replacing `test_timeline.py:141-143`, NHT selective +refresh counts. + +**F1c: incremental resolution** (`routing.py`, FIB kind in `derive.py`). + +```python +@record class Dirty: prefixes: frozenset[Prefix]; interfaces: frozenset[str]; neighbors: frozenset[tuple[str, int]] + recursive: frozenset[tuple] # RecursiveNexthop keys +def resolve_fib_incremental(rib, ctx, policy, old_fib, old_outcome, dirty, *, fallback_ratio=0.2 + ) -> tuple[Fib, ResolverOutcome, FibChanges] +``` + +Tests: `tests/model/test_resolver_incremental.py` runs the design-of-record +matrix and a seeded random event stream, asserting equality with the full +resolver after every event; op-count assertions (entries touched, recursive +entries recomputed). + +**F2a: configuration, commands, views** (`runtime/agents.py`, +`runtime/simulation.py`, `model/network.py` internal `configure_agent`). + +- `AgentRuntime.deliver_command(device, agent, Command)` with synchronous + `plugin.validate_command(name, payload, agent_config)`. +- `AgentRuntime._publish` stores `views` and `persistent`; `ctx.view(agent, + name)`; subscriptions on `('agents', name, 'views', key)`. +- `CAUSE_CONFIG` carries `changed_keys: frozenset[str]` computed by F7. +- Tests: `tests/runtime/test_agent_config.py` (config change without new + generation, reset keeps latest config, bad command rejected at submit, run-time + command error reported in a view, stale-generation command dropped and + counted, views readable by a second agent on the device, persistent survives + reset unless purged). + +**F2b: RIB feed, owner notifications, NHT installed mode** (FIB kind hook, +`model/nht.py`, `runtime/agents.py`). + +```python +def route_changes(dev_old, dev_new, af, fib_changes, rib_changed) -> tuple[tuple[WatchOwner, RouteChange], ...] +def row_notices(outcome_old, outcome_new, client) -> tuple[RowNotice, ...] # diff_pmap of the client sub-map +def resolve_installed(ctx, key) -> NhtResult # reads RecursiveNexthop table +``` + +Tests: `tests/runtime/test_rib_feed.py` (selected vs all, metric-only and +tag-only changes, PENDING suppression, snapshot on registration and on reset, +per-watch coalescing, own-client exclusion, two-border tag loop), +`test_row_notice.py`, `test_nht_installed.py` (installed answer lags RIB +answer by `fib_delay`; `via_metric` from a BGP resolver equals its MED). + +**F2c: runtime scale** (`runtime/agents.py`, `model/state.py`). + +- Lazy context accessors backed by `RibState.clients` and per-client outcome + maps; interface-view cache holding strong node references. +- `validate_admitted` shard-identity rule (section 3, F2) and a `PMap` + visited-node counter for tests. +- Tests: guard tests of section 3 F2; G2–G5 op-count assertions; ratio + benchmarks (slow). + +**F3: transport** (`runtime/transport.py`, `runtime/channels.py`). + +```python +def listen(device, agent, gen, endpoint: Endpoint, scope: frozenset[str] | None, + accept: PMap[RemoteId, AcceptOpts]) -> None | Rejection # RemoteId scoped by ingress interface for link-local +def _path(state, device, local, remote, connection, *, ttl: int, min_ttl: int | None) -> _Path # hop-limited trace +``` + +`agents/common/net.py`: `source_address(ctx, af, destination) -> int | None`. +Tests: the F3 exit list (section 3) in `tests/runtime/test_transport_ttl.py`. + +**F4: domain oracle** (`model/igp.py`). + +```python +@record class IgpDomain: name: str; devices: frozenset[str] | None; include: tuple[str, ...] = ('.*',) + exclude: tuple[str, ...] = (); passive: tuple[str, ...] = (); instance: int = 0 + distance: int = 110; metric_mode: str = 'interface' +def oracle_igp_domains(domains: Sequence[IgpDomain]) -> RouteSource +``` + +Tests: two-AS isolation, passive interfaces, single-domain fingerprint equality. + +**F5: protocol toolkit** (`netsim/agents/common/`). + +```python +# selection.py +class DecisionFlavor(Protocol): + name: str; refs: tuple[str, ...] + def compare(self, new: PathView, exist: PathView, ctx: DecisionContext) -> Verdict: ... + def select(self, dest: Dest, changed: tuple[PathKey, ...], ctx: DecisionContext) -> Selection: ... + def multipath(self, dest: Dest, best: PathKey, ctx: DecisionContext) -> tuple[PathKey, ...]: ... +BUILTIN_FLAVORS: Mapping[str, DecisionFlavor] # 'frr', 'rfc4271'; a deployment may pass more by name +# spf.py +def ecmp_spf(adjacency: Mapping[int, tuple[tuple[int, int, int], ...]], source: int) -> SpfResult # (peer, metric, link id) +# timers.py +class DeadlineHeap: ... # one shared persistent priority structure (F5): arm/cancel O(log n), due(now) O(k log n), stale entries bounded; never a tuple rebuilt per edit +@record class ExponentialBackoff: initial: float; maximum: float; current: float; last_error: float | None +@record class AsyncDebounce: min_s: float; max_s: float; current: float | None +def jitter(rng: random.Random, value: float, pct: float) -> float +# routeops.py +def route_ops(old: PMap[RowKey, Route], new: PMap[RowKey, Route], changed: frozenset[RowKey], af: int, *, + sync_threshold: int) -> tuple[RouteOp, ...] +``` + +Tests: unit tests per helper; `tests/agents/harness.py` self-tests (the +projection proxy rejects an undeclared attribute read). + +**F7: settings engine** (`netsim/settings/`, `netsim/model/device_settings.py`, +`adapters/ngraph.py`). + +```python +@record class DefaultsSpec: entries: tuple[tuple[str | None, str | None, Any], ...]; fallback: Any # (dialect range, profile, value) +@record class Setting: key: str; type: Any; scopes: tuple[str, ...]; default: DefaultsSpec; apply: str + unit: str | None = None; since: str | None = None; refs: tuple[str, ...] = (); doc: str = '' +@record class Fragment: owner: str; settings: tuple[Setting, ...]; records: Mapping[str, type] # key -> resolved field +@record class Layer: name: str; priority: int; selector: Selector | None; values: PMap[str, Any] +@record class Selector: names: str | None = None; labels: frozenset[str] = frozenset() +def catalog_for(device_fragments: tuple[Fragment, ...]) -> Catalog # model fragments + fragments of agents on the device +def resolve_device(catalog, layers, device_state, previous: Resolved | None) -> Resolved +@record class Resolved: dialect: str; profile: str; device_config: DeviceConfig + agent_configs: PMap[str, AgentConfig]; changed_keys: frozenset[str] +def explain(catalog, layers, device_state, key) -> tuple[tuple[str, Any], ...] # (layer or default, value) chain +``` + +The catalog is per device: model-owned fragments plus the immutable +`settings_fragment()` of each agent plugin present on that device, so no global +registry exists and the model layer never imports agent packages. + +Tests: `tests/settings/` precedence matrix (network, group, device, object; +equal-priority conflict rejected at `update()`; absent-object target rejected), +inheritance (peer-group, override units, unset restores), dialect pins and +aliases, `since` rejection, apply-action classes, `explain` chains, record +reuse, generated docs freshness, `native` fingerprints unchanged, NetGraph +`attrs.netsim` keys migrated. + +**F8: forwarding programming contract.** + +```python +@record class ProgrammingRequest: device: str; table: int; af: int; prefix: Prefix; generation: int + request_id: int; action: Any # desired FIB action or DELETE +@record class ProgrammingResult: request_id: int; generation: int; outcome: str # OK|ERROR|CAPACITY|LOST + boundary: str # control|kernel|hardware +@record class FibSlot: desired: Any | None; installed: Any | None; pending: ProgrammingRequest | None + reason: str | None # e.g. POLICY_EXCLUDED +def program(slot: FibSlot, desired: Any, gen: int, now: float, hw: FibHardware) -> tuple[FibSlot, tuple[Event, ...]] +def complete(slot: FibSlot, result: ProgrammingResult) -> FibSlot # ignores stale generations +``` + +Tests: `tests/model/test_programming.py` (G01–G04 without a gate, E09, E14, +deterministic same-time ordering, fingerprints unchanged with default +settings). + +**RP1: route-policy language.** + +```python +# netsim/model/route_policy/types.py +class Flow(Enum): ACCEPT; REJECT; NEXT_TERM; NEXT_POLICY +@record class RoutePolicy: name: str; terms: tuple[Term, ...] +@record class Term: name: str; when: Expr; actions: tuple[Action, ...]; flow: Flow +@record class RoutePolicyBinding: point: str; policies: tuple[RoutePolicyRef, ...]; fallback: str + match_input: str = 'WORKING'; per_nexthop: bool = False + scope: str | None = None; fallthrough: str = 'inherit' # RESOLUTION_IMPORT only +@record class RoutePolicyRef: name: str; revision: str; digest: str +class PolicyAccess(Protocol[T]): + def read(self, route: T, field: Field) -> Any: ... + def read_context(self, context: Any, field: Field) -> Any: ... + def apply(self, route: T, actions: tuple[Action, ...]) -> T: ... +# compile.py / evaluate.py +def compile_binding(binding, definitions, sets, capability: CapabilityRow) -> CompiledBinding # raises on any error +def evaluate(compiled: CompiledBinding, route: T, context: Any, access: PolicyAccess[T], + trace: TraceSink | None = None) -> PolicyResult # pure +``` + +Tests: `tests/model/route_policy/` (truth and flow tables, absence rules, +quantifiers, `PrefixListPermits`, capability violations at compile time, +canonical fingerprints, E01–E03, E10, E12, E19–E22). + +**RP3 and RP4: device hooks.** + +```python +def submit_rows(dev: DeviceState, af: int, client: ClientId, op: RouteOp, *, bypass: bool = False + ) -> tuple[DeviceState, RowChanges] # F1b boundary; RP3 adds the admission views +def admit(submitted: RibState, bindings, access) -> RibState # per_nexthop mode prunes legs; zero legs = not admitted +def nht_filtered(answer: NhtResult, binding, client) -> NhtResult # scope 'nht': filter the chosen resolver, no fallthrough +``` + +Tests: E04, E05, E06, E11 and E13; `ip protocol` leg pruning and fallback to +the next candidate; `ip nht` returning filtered with no fall-through. + +### 12.4 Open/R slices + +**OR1: config, types, Spark, agent shell.** + +- `settings.py` fragment (upstream defaults from `OpenrConfig.thrift` with refs; + area and interface-regex scopes); `config.py` resolved records + (`OpenrSettings`, `AreaSettings`, `SparkSettings`). +- `types.py`: `SparkHello`, `SparkHandshake`, `SparkHeartbeat`, `NeighborEvent`. +- `spark.py`: + ```python + @record class SparkNeighbor: name: str; interface: str; area: str; state: str; seq: int; rtt_us: int | None + hold: float; gr: float; established_at: float | None + def on_hello(st: SparkState, iface: str, msg: SparkHello, now: float, cfg) -> tuple[SparkState, SparkOut] + def on_handshake(st, iface, msg, now, cfg) -> tuple[SparkState, SparkOut] + def on_heartbeat(st, iface, msg, now, cfg) -> tuple[SparkState, SparkOut] + def on_timer(st, name, now, cfg) -> tuple[SparkState, SparkOut] + def deduce_area(areas, iface, neighbor_name) -> str | None + ``` +- `agent.py`: `OpenrAgent(instance=0)`; worklist runner; timer namespace; + views (`neighbors`, `adjacencies`). +- Tests first: FSM transition table (one test per row), area mismatch, GR + window, one-way teardown, RTT arithmetic and floor, hello jitter determinism, + liveness-only admission count. + +**OR2: LinkMonitor and KvStore.** + +```python +def merge_key(local: KvValue | None, incoming: KvValue, *, sender_is_originator: bool) -> MergeResult +def full_sync_request(db: KvStoreDb) -> KvSyncRequest # hashes only +def dump_difference(db, req) -> KvSyncResponse # values + tobe_updated keys +def flood(db, delta, exclude_peer, node_id) -> tuple[Message, ...] +def ttl_tick(db, now, heap) -> tuple[KvStoreDb, Publication] +def adjacency_db(lm: LinkMonitorState, area, cfg, persistent) -> AdjacencyDatabase +``` + +Tests: merge-rule table (version, originator, value, ttlVersion), seeded +interleaving convergence property, 3-way sync after partition, TTL expiry, +self-originated override, flood loop check, metric pipeline table, flap backoff. + +**OR3: Decision, PrefixManager, Fib, initialization.** + +```python +def link_state(adj_dbs: PMap[str, AdjacencyDatabase]) -> LinkState # bidirectional (node, if, otherIf) +def compute_routes(ls_by_area, prefix_state, me, cfg, changed: frozenset[Prefix] | None) -> RibDelta +def select_best_node_area(entries, cfg) -> tuple[str, str] # lowest (node, area) when selection off +def redistribute_cross_area(programmed: RibDelta, areas) -> tuple[PrefixEntry, ...] +def fib_step(fib: FibState, delta: RibDelta, notices: tuple[RowNotice, ...], now) -> tuple[FibState, RouteOps, Timers] +``` + +Tests: oracle equivalence twins (section 4.7 configuration), delete delay in +placement, init-gate timings, change-proportional op counts. + +**OR4–OR6.** Multi-area and multi-instance (`deduce_area`, `area_stack`, +`bestArea` case, separation tests); drain and overload commands writing +`persistent`, RIB-feed injection (`kind='all'`); passport and study +integration, the `netsim_ext` SRv6 advertisement. + +### 12.5 BGP slices + +**BG1: data types, AS_PATH, peers, settings.** + +```python +@record class Segment: kind: int; asns: tuple[int, ...] # SEQ, SET, CONFED_SEQ, CONFED_SET +@record class AsPath: segments: tuple[Segment, ...] +def hop_count(p: AsPath, *, count_confed: bool = False) -> int # a set counts 1, confed excluded (RFC 5065 §5.3) +def prepend(p, asn, *, confed: bool = False, times: int = 1) -> AsPath +def strip_confed(p) -> AsPath; def left_as(p, *, skip_confed: bool = True) -> int | None +def loop_count(p, asn, *, exclude_confed: bool = False) -> int # FRR aspath_loop_check vs _confed +@record class PathAttributes: origin: int; as_path: AsPath; next_hop: NextHop; med: int | None; local_pref: int | None + atomic_aggregate: bool; aggregator: tuple[int, int] | None; communities: frozenset[int] + ext_communities: frozenset[bytes]; large_communities: frozenset[tuple[int, int, int]] + originator_id: int | None; cluster_list: tuple[int, ...]; aigp: int | None +def peer_sort(inst: InstanceSettings, nbr: NeighborSettings, open_as: int | None) -> str # IBGP|EBGP|CONFED|INTERNAL +def router_id(configured: int | None, interfaces: PMap[str, InterfaceView]) -> int | None # zebra rule +def update_size(attrs, nlri: int, afi: int, *, extended: bool) -> int # RFC 4271 §4 +``` + +Tests: AS_PATH vectors re-derived from RFC 4271/5065/6793; peer-sort matrix +(incl. peer AS == confed id → IBGP; `remote-as auto`); router-id selection; +fragment coverage. + +**BG2: sessions.** + +```python +@record class PeerFsm: state: str; conn: int | None; caps: Caps | None; hold: float; keepalive: float + connect_retry: float; idle_hold: float; last_error: str | None +def fsm_step(fsm: PeerFsm, ev: FsmEvent, now: float, cfg: NeighborSettings) -> tuple[PeerFsm, tuple[FsmAction, ...]] +def resolve_collision(ours: PeerFsm, theirs_conn: int, my_id: int, peer_id: int, my_as: int, peer_as: int) -> int +def negotiate(local: Caps, remote: Caps) -> Caps +def session_endpoints(ctx, nbr: NeighborSettings) -> tuple[Endpoint, Endpoint, int, int | None] # local, remote, ttl, min_ttl +``` + +Events include `InterfaceDown` (fast external failover) and `NhtChanged` +(connect gating). Tests: establishment matrix for every row of section 5.3, +collision, hold negotiation and expiry, backoff sequences, fast external failover +flush, dynamic-capability CAPABILITY-vs-reset. + +**BG3: inbound, next hops, decision, Loc-RIB, install.** + +```python +def process_update(st: BgpState, peer: PeerKey, upd: Update, now, cfg) -> tuple[BgpState, frozenset[DestKey]] +def nexthop_register(st, path) -> tuple[BgpState, tuple[NhtOp, ...]] +def on_nht_changes(st, keys: tuple[NhtKey, ...], ctx) -> tuple[BgpState, frozenset[DestKey]] +# decision/frr.py and decision/rfc4271.py +def compare(new: PathView, exist: PathView, ctx: DecisionContext) -> Verdict +def select(dest: Dest, changed: tuple[PathKey, ...], ctx: DecisionContext) -> Selection +def select_multipath(dest: Dest, best: PathKey, ctx) -> tuple[PathKey, ...] +def process_dirty(st, dirty: frozenset[DestKey], ctx) -> tuple[BgpState, tuple[LocRibChange, ...]] # after 50 ms hold +def install_ops(changes, st, cfg) -> tuple[RouteOp, ...] # distances, recursion, legs, metric = MED +``` + +Decision tests are data tables (one row per step and flavor, with the expected +winner and reason string), plus scenario 8 order dependence, direct and +indirect next hops, BGP-over-BGP and `via_metric`. + +**BG4–BG6: outbound, policy, RR, confederations.** + +```python +@record class UpdateGroupKey: sort: str; policy_out: str | None; nh_self: str | None; rr_client: bool; ... +def announce_check(route: LocRibEntry, src: PeerKey, grp: UpdateGroupKey, st, cfg) -> Advert | None +def per_peer_nexthop(advert: Advert, peer: PeerKey, st) -> NextHop +def queue_and_pack(adj_out: AdjRibOut, adverts, withdraws, now, mrai) -> tuple[AdjRibOut, tuple[Update, ...]] +# policy: BGP's PolicyAccess over its own route type; evaluation is route_policy.evaluate (RP1) +def bgp_policy_access() -> PolicyAccess[BgpRoute] +def apply_binding(binding: CompiledBinding, route: BgpRoute, context: BgpPolicyContext) -> PolicyResult +def reflect_to(src_sort, src_client: bool, dst_sort, dst_client: bool, c2c: bool) -> bool # RFC 4456 +def rr_stamp(attrs, src_router_id: int, cluster_id: int) -> PathAttributes +def confed_outbound(attrs, dst_sort: str, member_as: int, confed_id: int) -> PathAttributes # RFC 5065 +``` + +Tests: scenarios 1–7, 9, 11, 12; message-count assertions; settings `@covers` +tests for each key the slice introduces. + +**BG7: conformance.** `netsim/adapters/frr.py` (load and dump a bounded +`frr.conf` subset into layers), SONiC BGP tables into layers, `tools/frr_lane/` +scripts, goldens, the Z5 and N6 black-box checks, scenario 10 dynamics. + +### 12.6 Worker assignment template + +Each slice is one worker branch (`protocols/`), created as in Gates B +and C. The prompt contains: the slice section of this plan (sections 3, 4.8, +5.12 and 12), the owned files, the tests to write first, the exit check, the +rule "no edits outside owned files except through the integrator", and the +checks to run (`make check-ci`, `make check-ft`, the slice's benchmarks). The +integrator merges in the order of section 8, runs the fingerprint oracle and +full checks, and routes review findings back to the owning worker. + +### 12.7 First actions after approval + +1. F0 commit by the integrator (records, stubs, `test_contract_f0.py`). +2. Launch F1a, F2a, F3, F4, F5 and F7 workers in parallel. +3. Integrate M1, run the review round, update this plan and memory. +4. Launch F1b, then F2b; hold the F1c design gate; launch OR1 and BG1 as soon + as F5 and F7 are merged. + +--- + +## 13. How the plan answers each part of the brief + +One subsection per requirement. Each gives the mechanism, where it lives in +code, how a user configures it, and the test that proves it. + +### 13.1 A proper Open/R IGP, following the GitHub source + +- **Mechanism.** `netsim/agents/openr/` mirrors the daemon module for module + (Spark, LinkMonitor, KvStore, Decision, PrefixManager, Fib, initialization) + at `facebook/openr` `81dae4c`, cloned in `~/ws/openr`. Each submodule is a + pure function over its frozen state; the agent drains the modules' queues as + a worklist in the daemon's order. Programming-gated edges wait for FIB + confirmation (D3). +- **Fidelity.** The eight behaviour groups of section 4.1 are reproduced with + upstream defaults and `file:line` refs. Upstream facts that surprise people + are kept as upstream has them: no domain name, SP_ECMP only, RTT metric by + default, soft-drain at boot, area policy a stub. Anything added is + `netsim_ext` and off by default. +- **Proof.** Section 4.7: oracle equivalence on random graphs, FSM and merge + tables, partition and heal, multi-area, drain, delete delay in placement. + +### 13.2 Which BGP to follow + +- **Decision.** RFCs first; FRR as the dialect where the RFCs leave room; + clean-room Python on NetSim's agent contract (D1, research + `research-bgp-options.md`). +- **Why FRR.** It implements everything asked (RR, confederations, add-path, + ENHE, unnumbered), publishes selection reason codes that make traces + diffable, and is what SONiC runs. +- **Why not run FRR itself.** FRR is wall-clock and kernel-TCP bound, so it + cannot run under simulated time, and it is GPL, so its code cannot be ported + into MIT NetSim. +- **Rejected.** GoBGP (no IGP-metric step), OpenBGPD (no confederations), + Batfish (fixed point with an approximate MED rule), C-BGP (unmaintained), + Holo (no RR or confederations), bgpsim (no confederations). +- **Version.** Modern FRR (master `f1d7bec`), with FRR 10.5 (SONiC) as a + settings preset; generic version differences are settings (section 2B). +- **Checked against real FRR.** An optional containerlab lane compares + converged state; its goldens are replayed by the normal test suite. + +### 13.3 IGP domain size: one device, all devices, several domains + +An IGP domain is not a declared object. It is the set of IGP instances that +can form adjacencies, which follows from configuration (D5). + +| Shape | How it is configured | What happens | +|---|---|---| +| All devices in one domain | `openr.deploy(net, devices=all, areas=(AreaConfig('0', include_interface_regexes=('eth.*','Po.*'), redistribute_interface_regexes=('lo.*',)),))` | Spark discovers every included interface, KvStores flood one LSDB, Decision computes routes to every loopback | +| Several domains | Deploy Open/R per domain with device sets whose boundary interfaces are excluded from discovery (`exclude_interface_regexes`), or leave boundary devices without Open/R on those interfaces | No Spark hello crosses the boundary, so no adjacency and no flooding. Each domain has its own KvStore contents. Never rely on area names, because area `"0"` is a wildcard (`Spark.cpp:2112-2121`) | +| A border device in two domains | Two agents, `OpenrAgent(instance=0)` and `OpenrAgent(instance=1)`, with disjoint interface sets (`netsim_ext.multi_instance`) | Each instance owns its interfaces and drops hellos on the others. Routes cross only by explicit RIB-feed redistribution with tags | +| Multi-area inside one domain | Several `AreaConfig`s selected by interface and neighbour regexes | Per-area LinkState and KvStore; ABRs redistribute through PrefixManager with `area_stack` loop prevention (section 4.4) | +| One device as its own domain | An Open/R instance whose interfaces are not discovered, or no IGP at all | The instance originates its own prefixes into its own store and computes nothing remote (`research-openr.md` §9). Without an IGP, the device relies on connected, static and BGP routes | +| Fast analytical underlay | `oracle_igp_domains((IgpDomain('as1', devices=..., exclude=('ext.*',)), ...))` (F4) | Per-domain SPF with passive interfaces; used by BGP tests and studies that do not need protocol timing | + +- **Code.** `openr/spark.py` `deduce_area(areas, iface, neighbor_name)`, + interface ownership checks in `openr/link_monitor.py`, `model/igp.py` + `oracle_igp_domains`. +- **How BGP sees domains.** BGP resolves next hops through whatever the device's + RIB holds, so a next hop inside another domain is unresolvable unless + something (BGP itself, a static route, redistribution) provides a route. + That is the real behaviour. +- **Proof.** Section 4.7 separation tests; IX 1, 2, 3, 5 and 6. + +### 13.4 BGP sessions: iBGP, eBGP, confederation; loopbacks or interface addresses + +Session setup is one pure function, `bgp/fsm.py` `session_endpoints(ctx, nbr)`, +evaluated from the resolved neighbour settings: + +1. **Peer sort** (`bgp/peer.py` `peer_sort`): IBGP when the remote AS equals the + local AS; CONFED when it is listed in `bgp confederation peers`; EBGP + otherwise. `remote-as internal|external|auto` is resolved at OPEN. A peer AS + equal to our confederation id is IBGP. +2. **Remote endpoint.** The configured neighbour address, or for + `neighbor eth1 interface` the peer's link-local address from `ctx.neighbors` + on that interface (unnumbered, RFC 8950 ENHE turned on). +3. **Local endpoint.** `update-source` gives that interface's address, typically + the loopback. Otherwise the egress interface's primary address toward the + peer (`agents/common/net.py` `source_address`, as FRR's `getsockname`). For + unnumbered, the interface's link-local address, scoped. +4. **TTL.** 255 for IBGP; 1 for EBGP and CONFED; N with `ebgp-multihop N`; 255 + plus `min_ttl = 256 - hops` with `ttl-security hops N` (F3). +5. **Connect gating.** Single-hop EBGP registers NHT with `connected_only` and + does not connect until the peer address is on a connected network + (`bgp_fsm.c:2704-2717`). +6. **Passive side.** A wildcard listener on port 179 scoped to the device's + interfaces, with an accept table of configured peers giving each peer's TTL + and `min_ttl`; unknown remotes are refused (`bgp_accept`). +7. **Collision.** Both sides may open, and transport keeps both connections; the + FSM keeps the one initiated by the higher BGP identifier (RFC 4271 §6.8). + +What each combination therefore does, with no special cases in code: + +| Configuration | Outcome | +|---|---| +| eBGP on link addresses | TTL 1, connected check passes, session up; next hop direct | +| iBGP on loopbacks with `update-source lo` | TTL 255; the session comes up once an IGP, static or BGP route reaches the peer loopback; next hops indirect | +| eBGP on loopbacks without multihop | Held by the connected check (no session), unless a static route satisfies it (then up, but routes stay inactive, a recorded FRR quirk) | +| eBGP on loopbacks with `ebgp-multihop 2` or GTSM | Up across one transit router; GTSM rejects longer paths | +| Confederation peers | TTL 1 by default: adjacent loopbacks work, farther ones need `ebgp-multihop` | +| iBGP on link addresses | TTL 255 across one hop; next hops unchanged unless next-hop-self | +| Unnumbered eBGP | Scoped link-local session with no routes needed; IPv4 routes use IPv6 link-local next hops | + +- **Proof.** BG2 establishes one session per row of section 5.3; scenarios 1, 2 + and 4; IX 1, 2, 4 and 5. + +### 13.5 iBGP hop by hop with RR and next-hop-self instead of an IGP + +- **Configuration** (sketch 4 in section 6.1). On every router: + - one iBGP session per link, on link addresses; + - every neighbour is a route-reflector client; + - the cluster id defaults to the router id, so every router is its own + cluster; + - `next-hop-self force` on every neighbour; + - `network `; + - `maximum-paths ibgp N equal-cluster-length`. +- **Why it works like an IGP:** + - Loop prevention: each reflection prepends the reflector's cluster id to + CLUSTER_LIST, and a router drops a route carrying its own cluster id or its + own ORIGINATOR_ID as an implicit withdraw (RFC 4456 §8; `rr.py` + `inbound_loop`). + - Path choice: all copies of a loopback carry the same ORIGINATOR_ID, so the + router-id step ties (RFC 4456 §9 substitution) and the shorter CLUSTER_LIST + wins (step 14). The result is shortest-hop routing. + - Next hops: `force` rewrites the next hop to the local link address on every + hop (plain next-hop-self is ignored on reflected routes). So every next hop + is direct, resolved through the connected route, and no IGP is needed. + - ECMP: with `equal-cluster-length`, only equal-hop paths join the multipath + set. +- **Recorded pitfall.** Without `equal-cluster-length`, FRR admits longer + cluster-list paths into the multipath set, which can form forwarding loops. + The simulator reproduces this and the test detects the loop by trace + (scenario 6). +- **Overlay on top.** Loopback sessions, for example to a central RR, then + resolve over these BGP-learned loopback routes: BGP over BGP, allowed as in + zebra (IX 4). +- **Proof.** Scenario 6 (with and without `equal-cluster-length`); IX 4 against + a hop-count oracle. + +### 13.6 Replaceable, named decision flavors + +- **What a flavor is.** A frozen `FlavorSpec(name, selection, steps, refs)` + (`agents/common/selection.py`). `selection` is `insertion` (FRR's sorted + list), `elimination` (RFC 4271 whole-set elimination) or `grouped(key)`. + `steps` names built-in steps or carries user step objects. Each step is a + `PairStep` (returns win, tie, or a held result released later, which FRR's + IGP-metric and confederation steps need) or a `FilterStep` (set + elimination). Steps may mark the multipath checkpoint. +- **Where it plugs in.** The flavor is the value of the per-node setting + `bgp.decision`. `locrib.process_dirty` calls + `select(dest, changed, ctx)` for each changed destination only, using the + flavor resolved from `ctx.agent_config` (a pure O(steps) lookup, nothing + cached). Multipath is a separate rule object (`multipath.py`). +- **What flavors see.** Everything they may depend on is explicit per-prefix + state in `Dest`: the ordered path list, unsorted paths in arrival order, the + previous best, arrival sequence numbers. So history-dependent flavors (FRR's + RFC 5004 step, vendors' oldest-path steps) are expressible and deterministic. +- **Built in.** `frr` (hand-written from `bgp_path_info_cmp`, step by step) and + `rfc4271`. `cisco-ios` and `junos` come in BG8. +- **Adding a named flavor.** + 1. List built-in step names and any new step classes (frozen, pure, with + refs). + 2. Choose the selection algorithm. + 3. Give it a name and refs. + 4. Set it as `bgp.decision` on a network, group or device layer. + + No change to the agent is needed (example in section 6.1). +- **Knobs versus flavors.** Knobs such as `always-compare-med` are ordinary + settings; `FlavorSpec.use('frr', always_compare_med=True)` is sugar that + writes them. +- **Proof.** Per-step data tables per flavor, the order-dependence test + (scenario 8), and an invariant that every step carries refs. + +### 13.7 Route reflection and confederations + +- **Route reflection** (`bgp/rr.py`), RFC 4456 in FRR's dialect (section 5.6). + - Hooks: `reflect_to(...)` in the outbound announce check (the reflection + matrix); `rr_stamp(...)` writes ORIGINATOR_ID and prepends CLUSTER_LIST on + iBGP → iBGP only; `inbound_loop(...)` in inbound processing; decision steps + 13 (ORIGINATOR_ID as router id) and 14 (CLUSTER_LIST length), plus step 9 + with `equal-cluster-length`. + - Per-node settings: client flag per neighbour and AF, `cluster_id`, + client-to-client reflection, allow-outbound-policy, next-hop-self force. + - Proof: scenarios 5 and 6. +- **Confederations** (`bgp/confed.py`), RFC 5065 in FRR's dialect (section 5.7). + - Hooks: `peer_sort` returns CONFED; `confed_outbound(...)` prepends + AS_CONFED_SEQUENCE toward members and strips confederation segments and + prepends the confederation id toward true external peers; receive checks + (segment sanity, loops over member AS and confederation id); decision steps + 4 (confed segments not counted), 6 (MED rules), 7 (confed counts as + internal) and 10 (confed-external before the IGP metric); distance 200; TTL + 1 by default. + - Proof: scenario 7, IX 5. + +### 13.8 RFC annotations + +- **In code.** A decorator `@ref(...)` from `agents/common/refs.py` records + citations on every rule-bearing function and step, for example + `@ref('RFC 4456 §8', 'FRR bgp_route.c:6235-6255 inbound cluster check')`. It + stores an immutable `__refs__` tuple: no registry, no runtime cost. + - Module docstrings cite the governing RFC sections. + - Behaviour borrowed from FRR or Open/R cites function, step or reason names + and `file:line` in the pinned tree. +- **In settings.** Every `Setting` and `DefaultsSpec` entry carries refs + (section 2A.1). +- **Enforced.** An invariant test scans the protocol packages (`decision/`, + `aspath.py`, `inbound.py`, `outbound.py`, `rr.py`, `confed.py`, `fsm.py`, + `nexthop.py`, `openr/*`) and fails on any public rule function or step + without refs. +- **Traceability.** Tests carry `@covers_ref('RFC 4456 §8')`. A generator writes + `docs/traceability.md` (RFC section → functions → tests), and CI fails if a + cited section has no test or the file is stale. + +### 13.9 Direct and indirect next hops + +The same path through the code, from receipt to forwarding (D8, section 5.4, +F1b): + +1. **Receive** (`bgp/inbound.py`, `bgp/nexthop.py`). The next hop is classified + by the session: + - single-hop eBGP (TTL 1, no `disable-connected-check`) registers NHT with + `connected_only=True`, so it must be on a connected network: **direct**; + - everything else (iBGP, confed, multihop, GTSM) registers a recursive NHT + key, which may resolve over IGP, static, connected or BGP routes: + **indirect**; + - link-local next hops (unnumbered) are keyed by interface and follow + interface state, never registered with zebra. +2. **Validity and metric.** The path is valid when the NHT answer is eligible + (installed mode). The decision process's IGP-metric step uses `via_metric`: + the metric of the installed resolving route, which is the IGP cost for an + IGP route, 0 for connected or static, and the MED for a BGP route. +3. **Install** (`bgp/zebra.py`). + - Direct routes install as `Nexthop.via(interface, address)`, taken from the + NHT legs (non-recursive, as FRR sends single-hop eBGP). + - Indirect routes install as `Nexthop.recursive(address, af)` with recursion + allowed. + - Distances: eBGP 20, iBGP and confed 200. +4. **Forwarding** (F1b hierarchical FIB). Every recursive next hop becomes one + `RecursiveNexthop` entry shared by all routes that use it. A FIB group is + keyed by the unresolved next-hop set, and its content is the flattened legs. + An IGP change recomputes one recursive entry and the groups using it; the + 50k BGP entries keep their group id, so the change costs O(affected next + hops), not O(routes) (G1: from 1.8 s to a target of 5 ms). +5. **Next-hop change.** NHT notifies the changed keys, and BGP re-evaluates only + the paths on those next hops (FRR `evaluate_paths` over `bnc->paths`). +6. **Send side** (`bgp/outbound.py`). + - To eBGP, the next hop is self unless third party (RFC 4271 §5.1.3). + - To iBGP, it is unchanged unless next-hop-self; on reflected routes plain + next-hop-self is ignored and `force` applies, and a permitted outbound + route-map may rewrite it explicitly. + - To confed peers, it is unchanged. + - Self is a sentinel patched per peer with that session's local address. + + The sender's configuration therefore decides whether the receiver sees a + direct or an indirect next hop. +- **Proof.** Scenarios 1 (direct, fast external failover), 3 (indirect, hot + potato, reprocessing counts), 4 (static quirk, `disable-connected-check`); + BGP over BGP in IX 4; the G1 and op-count benchmarks. + +### 13.10 Modularity and flexibility, in one place + +- **Layers.** `core ← model ← runtime ← agents / adapters`. Protocol packages + import only contracts, addressing, routing records, `state.PMap`/`record` + and `agents/common` (invariant test). +- **Inside a protocol.** One submodule per real module, each a pure function + over frozen state with a declared context projection, joined by explicit + queues (D3). Adding behaviour means adding or replacing a submodule, not + editing the agent. +- **Replaceable parts.** + - decision flavors and multipath rules as values of settings; + - policy as typed route-map records; + - Open/R route selection mode as a setting; + - dialects and presets as data. +- **Configuration.** Every setting is per node through layers (section 2A), and + run-time changes follow declared apply actions. +- **Cooperation.** Protocols meet only through the RIB (with the redistribution + feed), NHT and published views (D2). So Open/R, BGP, statics, the analytical + oracle and future IGPs (IS-IS) mix on any device without knowing each other. +- **Extending.** + - A new IGP reuses `agents/common` (SPF, timers, routeops, settings, + harness). + - A new BGP feature (add-path, GR, BFD) is a submodule plus settings; keys + are already add-path and VRF ready (D9). + +### 13.11 Routing policy (integrated from the reviewed routing-policy plan) + +- **One engine for every hook.** A small typed language and one pure evaluator + (`netsim/model/route_policy/`, RP1) serve every attachment point: BGP import, + export, table-map, redistribution, RIB admission, FIB installation and + resolution. +- **Per-node bindings.** Bindings are per-node settings that resolve to exactly + one binding per scope. Definitions are immutable and pinned by digest. +- **Access by owners.** Each owner supplies a typed `PolicyAccess` for its own + route type, and one capability table per point limits which fields may be + read and written. +- **Vendor fidelity.** Exact adapters, FRR first (RP5 in BG7). Integration + corrected two mappings from source: `ip protocol route-map` is per-next-hop + RIB admission with fallback to the next candidate, and `ip nht route-map` is + NHT-scoped with no fall-through. +- **Edits.** Changes follow owner-defined reevaluation, and import replay + reports `needs_session_reset` when neither retained input nor ROUTE-REFRESH + is available. +- **Proof.** E01–E24 across RP1, RP3, RP4, BG3/BG4 and M5. + +### 13.12 Installation gating and advertising inactive routes + +- **Mechanism.** The F8 programming contract keeps desired and installed actions + separate, with generation-guarded completions at a named boundary. + - The gate profiles `selected`, `effective` and `strict`, plus + `bgp.export.inactive`, are per-node settings. + - FRR's `suppress-fib-pending` is a limited-scope profile. + - Open/R's readiness gate is reproduced as upstream has it, on + programmed-prefix membership. +- **Proof.** G01–G10, scenario 13, and the coupled pair FIB gate × inactive + export. + +--- + +## 14. Scale review: experiments, prototypes and the streamlined plan (revision 6) + +Goal set on 2026-09-27: review the plan critically, prove its points with +experiments and prototypes, and make sure the implementation stays simple to +maintain and performs for more than 100k devices and 1M prefixes. Scripts are in +`netsim-protocols-research/scale-experiments/`, all run on main `0e5b299`, +CPython 3.14.5, one machine. No repository code changed. + +### 14.1 What the current tree costs + +| Measurement | Result | Script | +|---|---|---| +| One device with 200k recursive BGP-style rows, converged | 296 MB and 16.9 s: about **1.5 KB and 84 µs per row** | `mem_probe.py` | +| Where the bytes go | RIB 504 B per row (360 B of it `PMap` shard overhead, 80 B `Route`, 56 B prefix tuple). Converge adds 981 B per row: about ten objects per prefix in the resolver (`FibEntry`, `RowOutcome`, `DependsOn`, `_Leg`, memo and frame tuples) | `mem_breakdown.py` | +| Building a network | **Quadratic**: 5.5 ms per device at 2k devices, 21 ms at 8k, 31 ms at 10k (312 s for 10k). Cause: `_validate_interface_config` (`network.py:738`) walks every device on every interface add for the SRv6 locator-overlap check; 22.5M `items()` calls for 9k interfaces | `build_phases.py`, `prof_build.py` | +| Converge with no routes | 1.7 ms and **25 KB per device**. 13 KB of it is `addressing.py:178`: every `PrefixTable` allocates a tuple of 33 or 129 mask ints (`lpm.py:193`), four tables per device | `converge_mem.py` | +| Idle agent tick (timer re-arm, state returned by identity) | **160–200 µs per run** and **27 KB retained per run** | `build_phases.py`, `tick_retention.py` | +| Cause of the retention | `AgentNode.rng = ctx.rng.getstate()` (`agents.py:792`): the 625-int Mersenne Twister state is committed on every run and kept by every retained root | `tick_retention.py` | +| Cause of the run cost | Every run is a full tree commit, because `runs`, `receipt` and `rng` change on every run. The commit pays delta diffing, `bump_epochs`, SRv6 `consumers_affected`, output validation and `AgentNode.__eq__` | `prof_ticks.py` | +| Raw DES core, 100k periodic processes | about 1.5 µs per timeout firing (712k firings, not 1M: the periods exceed 1 s), 776 B per process. The engine is not the bottleneck | `des_core_100k.py` | +| `PMap` at 1M entries | 104 B per entry, 1.35 s to build, `set` 40 µs. `validate_admitted` of one changed entry: **3.15 s** | `proto_pmap_1m.py` | + +### 14.2 Experiments that prove the fixes + +| Fix | Experiment | Result | +|---|---|---| +| Index the locator check (skip it when no device has locators; otherwise a block index) | monkeypatched `_validate_interface_config` | Build becomes linear: **95–125 µs per device**, 20k devices in 2.5 s; 100k projected at about 15 s instead of hours (`exp_build_fix.py`) | +| Stop committing the MT state | `getstate()` returns `None` | Retention falls from 26.9 KB to **1.8 KB per run** (`exp_rng_fix.py`) | +| Replace mask tuples by shifts (`(addr >> (bits − plen)) << (bits − plen)`) | arithmetic; per-device tables hold no mask objects | Saves about 13 KB per device (1.3 GB at 100k), and removes a shared object from the lookup hot path, which also helps free-threading | +| Shard-identity admission | prototype over the changed shard only | 1 change in 1M: **1.5 ms** instead of 3.15 s; but still O(N / 256) per change, so a changed-key log from the `PMap` builder (O(k)) is the mechanism, not the follow-up | + +### 14.3 Prototypes at 1M prefixes + +| Prototype | Result | Conclusion | +|---|---|---| +| Columnar route table: `array('Q')` prefixes, `array('B')` lengths, `array('I')` group ids; groups keyed by unresolved next-hop set; recursive next hops with dependent-group sets (`proto_hfib.py`) | 1M routes in 240 ms and **13 B per route** (plus 106 B per route for a dict exact-match index). An underlay change touches one recursive entry and its groups and **zero routes: 4.8 µs**. A route add: 9 µs. A trivial O(N) scan of 1M routes: 51 ms | A mutable lower bound: the hierarchical dependency structure avoids touching routes on an underlay change, and the payload is about 100× smaller than the object model (the exact-match index is not, at 106 B per row). The immutable design is §15.1 with the §16 corrections. Any per-event O(N) walk is at least 50 ms at 1M, so "cost proportional to change" is a hard requirement | +| Columnar LPM: sorted arrays per length with `bisect` (`proto_columnar_lpm.py`) | 117 B per prefix, 4.6 µs per lookup, 52 µs per insert (memmove), against the current dict-per-length table at 104 B and 1.3 µs | Withdrawn in §15.2: the probe retained temporaries and kept duplicates. Corrected, packed columns hold 11.8 MB against 98.6 MB for the dict-per-length table at 1M, with 4.1 µs against 1.2 µs lookups. The index is chosen per device class (§16.2) | +| BGP Loc-RIB, 1M prefixes × 4 paths with interned attributes (`proto_locrib.py`) | Frozen path objects: 650 MB, 9 s, 162 B per path. Struct-of-arrays (CSR offsets, peer, attribute id, sequence, flags): **68 MB, 1.7 s, 17 B per path**; a simple selection over one destination 0.4 µs | Object-per-path is unaffordable at 1M. Attribute interning plus arrays is the design | + +### 14.4 The scale target, stated honestly + +"100k devices and 1M prefixes" has two readings, and only one is feasible on +one machine in any language: + +- **Every device holds the full 1M table:** 10^11 route entries. At 16 B each + that is 1.6 TB. Not feasible. +- **1M prefixes network-wide, bounded per-device tables:** leaves hold locals + plus a default or summaries; spines hold pod summaries; a few dozen border, + route-reflector or super-spine devices hold full tables. Total rows about + 10^8, which is 2–3 GB in columnar form. Feasible, with the changes below. + +The design of record already states the N(N − 1) flat-domain limit. Two more +limits are measured now: + +- **Full per-device SPF at 100k devices** is 100k Dijkstra runs. In Python + that is hours; with the NetGraph-Core kernel (10 ms per 100k-vertex SPF, per + the design of record) it is about 1,000 s. So the analytical mode at 100k + devices must be **demand-driven and lazy**: compute forwarding only on the + devices and prefixes a query or a demand touches, which the design of record + already allows ("lazy routes behind an explicit interface"). +- **Initial load of a 1M-prefix BGP table** on one device is 4M path arrivals. + At 20 µs per path that is 80 s per device; at 2–5 µs, with bulk array + operations per UPDATE, 8–20 s. Bulk paths need array-level work, and + optionally a numpy or native kernel behind the same pure interface, like the + Core kernel today. + +**Passports** (section 5.10 replaces its earlier list): + +| Passport | Shape | Targets | +|---|---|---| +| P-A analytical, 100k devices | Clos with summaries and defaults; 1M prefixes network-wide; full tables on ≤ 64 devices; demand-driven forwarding | build ≤ 60 s; ≤ 8 KB per device baseline (≤ 0.8 GB); one failure re-derives only affected devices and prefixes | +| P-B protocol, BGP full table | 16 devices (2 RRs, 14 clients or peers), 1M prefixes × 4 paths | ≤ 1.5 GB total; initial load ≤ 5 min; one prefix flap ≤ 1 ms per device; one next-hop change ≤ O(paths on that next hop) | +| P-C protocol, wide fabric | 4k devices, Open/R or eBGP unnumbered, 40k prefixes, `lab` timers | cold start ≤ 10 min; idle tick ≤ 40 µs and 0 B retained; one link failure ≤ 5 s wall | + +Every claim beyond these passports is unproven. The user confirmed on +2026-09-27 that the target is the bounded-table reading: about 1M prefixes +network-wide, per-device tables of locals plus aggregates and defaults, and +full tables on a few dozen devices (passport P-A). + +### 14.5 Plan changes (revision 6) + +**S0, scale hygiene: before everything else in M1.** Five small fixes, each +with a benchmark that fails on the old code: + +1. Locator-overlap check through an index; build is linear. +2. A small deterministic PRNG in place of `random.Random` state: a 64-bit + counter-based generator whose state is one int, with per-module streams + derived by name (D3). The 25 KB per run goes away. +3. Masks by shift arithmetic; no mask tuples per table. +4. Timeline retention bounded by default (`keep_records`, `keep_events`) so a + long run cannot grow without limit. +5. A `PMap` builder changed-key log, used by admission (F2c) and by RIB + change sets (F1b). Admission becomes O(changed keys). + +**Zero-commit idle runs (F2a, a design change to Gate C).** `runs`, `receipt` +and `rng` move from `AgentNode` into runtime bookkeeping keyed by +(device, agent, generation); the tree keeps `state`, `views`, `persistent` and +`config`. A run whose output is identity state plus timers or messages commits +nothing: no delta, no `bump_epochs`, no SRv6 invalidation, no timeline record. +Target: ≤ 40 µs and 0 B retained per idle run, from 160–200 µs and 27 KB. +Receipts stay observable through views and the runtime's budget API. + +**Packed payloads (F1b, renamed "persistent packed FIB").** Storage is the +immutable `PackedTable` of §15.1 (bytes chunks, copy-on-write builder, +changed-row log), never mutable arrays. The prefix index stays a dict per +length for bounded devices and becomes packed sorted key columns for +full-table devices. The payloads become packed rows: +- FIB: per family, `entries` = arrays of (prefix, length, group id, action); + groups interned by unresolved next-hop set; recursive next hops in one table + with dependent-group sets; no per-entry `DependsOn` object. +- RIB: rows stay objects at the API (`Route` is the contract), but the per-row + outcome becomes an array indexed by row position, and `RowOutcome` objects + are created only on read. + + Target: ≤ 60 B per FIB entry, from about 1 KB. + +**BGP tables (BG3).** Loc-RIB as struct-of-arrays with interned attributes: +CSR offsets per destination; per path: peer id, attribute id, sequence, flags. +`PathView` objects are created only for the paths a decision compares. +Adj-RIB-Out per update group as arrays of (destination, attribute id). Target: +≤ 24 B per path. + +**Bulk paths.** Initial table load, `sync`, and full best-path recomputation +after a knob change operate on arrays per UPDATE or per table, behind the same +pure functions. A numpy or native kernel may back them later, chosen by +profile, as with the Core kernel. + +**Analytical mode at scale (new item in F4).** `oracle_igp_domains` gains a +lazy mode: routes for a (device, prefix) are computed on first query from a +per-domain SPF cache keyed by source, and demands drive which sources are +computed. The mode is an explicit per-domain setting; it is never chosen by device count. + +**PMap fixed shard count.** 256 shards make every map operation O(N/256). +Above about 1M entries the count should grow with the map (for example +√N rounded to a power of two, decided at promotion). This is a contained change +in `state.py` with the existing `PMap` tests as the oracle. + +### 14.6 Streamlining: what to merge, defer or drop + +The plan grew to twelve foundation slices, two protocol tracks and a policy +engine. The following keeps the requirements and cuts the surface: + +| Change | Reason | +|---|---| +| Merge F2a into F0 as "contract and delivery"; the runtime side of the new records is small once idle runs are zero-commit | One owner for the contract and its delivery avoids a hand-off | +| F7 settings engine lands in two steps: **F7a** (schema, device layer, dialects and profiles as data, `native` dialect, no groups) with M1; **F7b** (group layers with selectors, presets, `explain`, generated docs) with M3 | Protocol slices need only per-device resolution at first; groups and provenance are conveniences | +| Defer RP4 (`RESOLUTION_IMPORT` forwarding scope and `FIB_INSTALL`) to M4; keep the `nht` scope, which FRR needs, in RP3 | No scenario before M4 uses them; F8 is the dependency | +| F8 delivers desired versus installed actions and **group versions** with generation-guarded completions, an explicit ERROR completion and a deterministic injection seam, and the sync completion event; the `effective` gate profile and stochastic failure models come with BG8 | Pending, error and success must be distinguishable from the start; the two profiles FRR and Open/R need (`strict` and `selected`) cover scenario 13 | +| Drop the JSON policy codec from BG7; keep the FRR adapter and explanations | Nothing consumes JSON policies yet; the schema exists in the records | +| The FRR lane runs 10.7.1 and master by digest routinely; the pinned 10.5.4 fixtures run whenever `frr-10.5` behaviour changes | Two images cover `frr-modern`; a different release cannot certify the `frr-10.5` dialect | +| One test harness module (`tests/agents/harness.py`) rather than per-package fixtures | Already planned; stated as a rule | +| Decision flavors are two explicit functions plus a code-level named-class seam, not a step interpreter (§15.8) | Two known algorithms do not justify an interpreter; named replaceable flavors remain | + +Simplicity rules added to section 7's invariants: +- no unrelated-table scan for a local edit (an op-count test per kind); affected closure, bulk configuration and SPF work, and amortized compaction are budgeted explicitly; +- no Python object per route, path or FIB entry outside API boundaries + (a memory-per-row benchmark per table); +- no per-run tree write for identity outputs. + +### 14.7 Revised milestone M1 and first actions + +M1 now starts with S0 (one worker, one week) and F0 with delivery, then +F1a, RP1, F3, F4, F5 and F7a in parallel. Exit adds: linear build to 100k +devices under 60 s, idle run ≤ 40 µs with 0 B retained, admission O(k), +fingerprints unchanged. + +First actions after approval: +1. S0 with its five benchmarks (each failing on the old code), as the first + commit series. +2. F0 with zero-commit idle runs and the changed-key log records. +3. The rest of M1 in parallel. + +--- + +## 15. Codex review of revision 6: dispositions and the contracts it required (revision 7) + +Report: `netsim-protocols-research/codex-review-rev6.md` (GPT-6-Astra, xhigh; +26 minutes; every `scale-experiments` script re-run; four new probes, copied to +`scale-experiments/codex-probes/`). Verdict: S0 conditional go as a small +staged series; the expanded F0 of §14.6 no-go until the storage, publication, +scope, settings-delta and completion contracts exist. All 22 findings are +accepted; the integrator verified the source claims (ORIGINATOR_ID, confed +deferral, reinstall guard, SRv6 consumer copy, Open/R empty sync) and answered +the P0 with a new prototype. Where this section conflicts with earlier text, +this section wins; §15.9 is the authoritative dependency table. + +### 15.1 P0: packed storage now has an immutable, incremental contract + +**Finding.** The §14.3 prototypes mutate `array` and `dict` objects. The tree +admits only transitively immutable values (`validate_immutable` rejects +`array`; a read-only `memoryview` over a mutable buffer is not immutable); the +only safe packed pattern in the tree today is bytes-backed `FloatArray` +(`state.py:437-474`). Freezing a whole array per edit is O(table bytes). + +**Contract: `PackedTable`** (model layer, `netsim/model/packed.py`), a +persistent vector of fixed-width rows: +- **Storage.** `chunks: tuple[bytes, ...]` of `CH` rows each (4096), a row + struct per table kind (FIB row 16 B: prefix u64, plen u8, flags u8, group + u32; path row 24 B: peer u32, attr u32, seq u64, flags u8 and padding), plus + `count`. A 4096-row chunk is therefore 64 KB for 16 B rows and 96 KB for + 24 B rows. + IPv6 prefixes use a 16-byte key field or a prefix-id indirection (below). +- **Identity.** Positions are private to one table generation. External + references use logical keys (prefix, `(peer, path_id)`), or + `(table_generation, slot, slot_generation)` checked on dereference; bare + positions never cross a publication boundary (queued requests, dirty sets, + back-indexes and completions carry logical identity or their snapshot). + The **first implementation is append-only**: no slot reuse or compaction + within a live generation, an explicit memory limit, and rebuilding at a + generation boundary as one owner transaction that publishes the table, every + index, the pools and the allocator together with an explicit old-to-new + remap (a storage change, never a route withdrawal). Bounded churn is not + claimed for that version. Old snapshots keep old chunks. +- **Builder.** Private, copy-on-write per chunk: the first write to a chunk + copies 64 KB; further writes to that chunk are in place on the private copy; + `build()` freezes touched chunks to `bytes` and emits a **change + certificate**: logical adds, updates and deletes, bound to the exact base and + result generations and naming its coordinate system and any remap. The + trusted builder also validates row schema, bounds and references (generic + immutability does not). No published chunk is ever mutated. +- **Admission.** `bytes` are leaves, so validating a 1M-row table costs 0.1 ms; + no walk over rows. +- **Widths.** Attribute ids u32 (not the prototype's u16); peer ids u32; + sequence u64; overflow is a validation error at the builder. +- **Index.** Small and full-table indexes implement the same exact lookup, + LPM, exclusion and last-write-wins semantics. Bounded devices keep persistent + per-length maps. For full-table devices the candidate is ordered + `(network_key, row_reference)` entries in bounded sorted blocks behind a + persistent block directory, with a defined 128-bit IPv6 ordering and family + separation; a full per-length rebuild is a budgeted bulk operation, never + the one-prefix edit. That index is adopted only after insertion, withdrawal, + replacement, LPM and publication are measured under held snapshots + (§16.2). The prototype's 0.24 µs bisect over a mutable key array is a lower + bound, not that design. + +**Measured on the prototype** (`scale-experiments/proto_chunked.py`, 1M rows): +16 B per row; one persistent single-row update 10 µs; held old snapshot +unchanged; 10k updates in one builder 4.7 ms; `validate_immutable` 0.1 ms. +Changed rows come from the builder log, not from diffing chunks (a diff of 2000 +changes took 138 ms). + +**Gate.** Before any protocol depends on it: tests holding an old snapshot +across insert, update, delete and compaction; admission; changed-row log +correctness against a brute-force diff; free-threaded readers on one snapshot; +edit and publication cost and retained-version memory measured. + +### 15.2 P1s on measurements and passports + +- **Passport arithmetic corrected (finding 2).** The 106 B per row dict index + dominated and was omitted from the 2–3 GB figure. Revised budgets, per + component, current plus retained history: + - **P-A** (100k devices, 1M prefixes network-wide, ≤ 64 full-table devices): + bounded devices at ≤ 8 KB each = 0.8 GB; full-table devices at ≤ 40 B per + row (packed row 16 B, packed index 8 B, outcomes and group refs 16 B) = + 64 × 1M × 40 B = 2.6 GB; shared prefix dictionary and attribute pools + 0.2 GB; history bounded by retention settings. **Target ≤ 4 GB**, split by + device class in the report. + - **P-B** (16 BGP devices, 1M prefixes × 4 paths per device): Loc-RIB paths + 16 × 4M × 24 B = 1.5 GB; FIB and RIB rows 16 × 1M × 40 B = 0.6 GB; + Adj-RIB-Out per group as packed (dest, attr) pairs; attribute pool shared. + **Target ≤ 3 GB**; initial load measured across all 16 devices, not one + table. + - **P-C** unchanged in shape, with the workload named (areas, degree, prefix + distribution, timers) and "idle" defined as identity-state timer work. + + Every passport reports `sum(submitted rows)`, selected rows, installed rows, + paths, distinct attribute sets, update groups, queued NLRI and retained + generations, and budgets current, peak-build and retained memory separately. +- **Measurement labels corrected (findings 3, 4).** Section 14.1's "80 B + `Route`" is a `RowKey` tuple; the 296 MB is an allocation increment; the + 25 KB per device includes one loopback route; the DES probe fired 712k + events, not 1M, so the raw core costs about **1.5 µs per timeout**; the + "route add" in `proto_hfib.py` replaced an existing index entry. "Proved" in + §14.3 means the mutable lower bound; §15.1 is the immutable design. The + columnar-LPM rejection is withdrawn: packed columns save memory and cost + lookup time, so the index is chosen per device class (above). + +### 15.3 P1s on runtime and state semantics + +- **Zero-commit idle runs are a Gate C slice, not F0 work (finding 5).** New + slice **C6** after F0: pre-run RNG, counter and initialization state are + captured in the per-generation journal; runtime state is published at the + same success boundary as the tree, including the no-delta case; captured + prefixes are consumed before external callbacks; parked rejection and + explicit retry are preserved; receipts are exposed as a current view plus + bounded history; reset, remove and generation purge clear every runtime + map; `Network.fork()` does not clone a running scheduler. Gate: all Gate C + tests plus no-delta RNG use, mixed batches, failed outer commit and + post-publication callback failure. The S0 step is smaller: a versioned + compact PRNG whose state (one 64-bit counter) changes only when drawn, with + module streams derived by a fixed hash of `(seed, device, agent, module)`, + never Python `hash()`. +- **Changed-key certificate for `PMap` (finding 6).** Its own reviewed state + slice **S1**: the builder emits a delta (changed and deleted keys) certified + against the exact base map identity; composition across staged edits; + delete-then-reinsert and equal-but-different keys validate in full; an + unrelated or rebuilt map falls back to full validation; the certificate is + consumed on admission and never retains a parent chain. The shard count stays + 256 until a measured workload justifies bulk sizing. +- **More linear commit-path work (finding 7).** `srv6.consumer_index` copies + the whole consumer frozenset on any device change (`srv6.py:1323-1332`; + 3.5 ms at 100k consumers): S0 keeps the object when membership is + unchanged. `DeadlineHeap` becomes a shared persistent priority structure with + stated update cost, not a tuple rebuilt per edit; watch indexes state their + insertion cost. The invariant is reworded: no unrelated-table scan for a + local edit; affected closure, bulk configuration and SPF work, and amortized + compaction are budgeted explicitly. Op-count tests cover unchanged consumers, + timer-only runs and one-key edits. +- **Lazy analytical mode gets a query contract (finding 8).** A pure + `forwarding_query(snapshot, domain, device, prefix)` shared by trace and + placement; memoized SPF as disposable runtime data keyed by input epochs with + an eviction budget; old-snapshot answers preserved; never reachable from + agents; enumeration, absent-route, aggregate and default semantics and + invalidation defined; mode selection explicit, never by device count. It is + its own analytical slice (**A1**) and does not block protocol work. P-A + names the demanded source count. + +### 15.4 P1s on the resolver and policy hooks + +- **Cycle rule (finding 9).** Blanket `LOOP_DETECTED` for every member of a + dependency cycle over-rejects: a route with a cyclic recursive leg **and** an + independent direct leg must keep the direct leg. F1a's rule becomes per leg: + a leg resolves when it has a finite witness within the depth bound; a leg + whose resolution closes a cycle with no escape fails with `LOOP_DETECTED`; + other legs of the same route are unaffected; weights are normalized over + surviving legs. Tests: closed cycle, cycle plus direct leg (`resolver_escape.py`), + cycle plus lower-priority candidate, blocked less-specific fallback, under + every enumeration order. +- **`ip protocol` stage (finding 10).** The filter runs after next-hop + activation, when the resolving interface is known, not at submission: a + named **candidate-next-hop activation filter** in the resolver, per row, + with FRR-equivalent activated-next-hop facts; positive and negative underlay + dependencies recorded; reevaluated on their change without a new submission; + fallback to another candidate when no leg survives. Native `RIB_ADMISSION` + (submission) and native `FIB_INSTALL` stay distinct. The shared evaluator + is reused with its own capability row. +- **Scope in shared keys (finding 11).** `RecursiveNexthop` keys include + interface identity and generation for link-local addresses and every + resolution-changing scope; owner and client projections (policy binding, + query mode) are keyed separately and applied per row, never to a shared + group. `nht_filtered` receives the selected row's tag, metric, distance and + source and its activated next hops from the processed snapshot. +- **No import-policy memoization (finding 12).** Policies depend on the + prefix and context, not only the attributes; §5.10's memoization is removed. + Compiled bindings are reused; results are not. + +### 15.5 P1s on settings and configuration + +- **Root settings delta (finding 13).** `NetworkState.settings` is added to + the root fields `StateDelta.changed_paths()` enumerates, and to batch and + fork preservation; a settings-only update is never an empty delta. Desired + layer changes and effective changes are tracked separately, and only the + latter schedule protocol work. Dumps serialize explicit values and + inheritance intent, not only differences from dialect defaults. Tests: + equal-effective edit followed by a parent change; clear and unset; dump and + load; aborted batch; fork; reset. +- **One action plan per transaction (finding 14).** A configuration + transaction resolves the complete candidate, diffs old and new effective + records once, and compiles one per-owner plan with a dominance order: reset + or restart supersedes session work; recalculation, export and replay + coalesce against the new revision; `NEXT_SESSION` keeps its pending intent. + Preset precedence is stated with two conflicting examples (a preset's + neighbor value against an explicitly inherited instance value; preset + profile metadata against user layers) and unsupported combinations are + rejected. Minimal provenance sufficient to explain each winner ships in F7a. + +### 15.6 P1s on programming, sync and sessions + +- **Group versions (finding 15).** Installed forwarding dereferences + **installed** group content. An underlay change programs the group once and + advances its installed version at the completion boundary; prefixes keep + their group reference and are not re-requested. Prefix obligations name the + installed route or group generation they need; obsolete completions cannot + release a gate; installed content survives a pending or failed DELETE. + Minimal F8 keeps an explicit ERROR completion and a deterministic injection + seam, because pending, error and success must be distinguishable. +- **Open/R sync completion (finding 16).** An owner- and generation-scoped + sync completion event with batch identity and explicit successful and failed + members, including the empty set (Open/R publishes an empty FULL_SYNC, + `Fib.cpp:386-401, 849-859`). PrefixManager's gate applies only to entries + that carry next hops (`PrefixManager.cpp:650-662`). OR3 depends on this + minimal F8 contract. +- **Listener accept identity (finding 17).** The accept table is keyed by + scoped remote identity: AF, address and ingress interface identity and + generation where the address is link-local; that scope flows through + connection lookup, accept options and collision identity. Listen ranges are + a separate matching mode for globally scoped addresses. + +### 15.7 BGP corrections (findings 18, 19, 20) + +- **Reinstall.** For an unchanged best path with unchanged attributes and no + clear or add-path in effect, use FRR's reduced change flags + (`bgp_route.c:4439-4462`); otherwise compare the complete emitted local row + and replace it when its semantics change. A MED edit changes the installed + metric. +- **Route reflection.** ORIGINATOR_ID is set only if absent + (`bgp_route.c:2820-2825`); `force` affects ordinary next-hop-self on + reflection; with `allow-outbound-policy` a route-map may rewrite the next + hop and an explicit rewrite prevents the automatic reset. Tests: two RRs in + series; an allowed route-map rewrite. +- **Confederation step.** Step 10 returns immediately by default and defers + under `bgp bestpath peer-type multipath-relax` (`bgp_route.c:1654-1684`); + both are tested. +- **suppress-fib-pending.** `bgp_check_withdrawal` returns "withdraw" for any + non-selected non-NORMAL path before the selected-static exemption + (`bgp_route.h:779-801`); the FRR profile gets an explicit event and state + projection with source-backed fixtures (selected `network` plus a + non-selected local path; delayed failure; better-admin; generic removal) and + is not reduced to `strict` plus flags. +- **Multipath ceiling.** The build ceiling is package data: FRR's Debian + packaging configures 256 for both 10.5.4 and master; SONiC's 514 is SONiC's + own build value; the template configures 64. `frr-10.5` behaviour changes + run the pinned 10.5.4 fixtures. + +### 15.8 Streamlining, revised (findings 21, 22) + +- **Decision flavors are two explicit functions**, `frr_select` and + `rfc4271_select`, over the same narrow immutable inputs, sharing small + comparison helpers with source references. No step interpreter, no + hold/release slots. The extension seam is code: a flavor is a named class + implementing `DecisionFlavor` (`compare`, `select`, `multipath`) added to the + per-agent flavor table at deployment; configuration selects by name. This + keeps "named, replaceable flavors" without an interpreter. +- **Settings start small:** typed desired and effective records, explicit + resolution, a small action planner, unknown keys rejected, precedence and + default tests. Generated-documentation enforcement, the key-field bijection + check and broad selector syntax move to F7b or later. +- **Policy evaluator stays as reviewed:** four flows, typed capability table, + no result cache, no regex, VM or callbacks. +- **Kept early because they prevent rewrites:** source-exact decision + fixtures, one settings transaction, one submission boundary, installed + versus desired group semantics, scoped NHT, immutable snapshots, exactly-once + publication, ERROR completion, empty sync. + +### 15.9 Authoritative slices and dependencies (supersedes §8, §12.1 and §14.7) + +| Slice | Content | Needs | +|---|---|---| +| S0 | locator index; arithmetic masks; bounded retention defaults; SRv6 consumer index without copy; compact PRNG (state changes only when drawn) | – | +| S1a | `PMap` changed-key certificate (§16.2) | S0 | +| S1b | `PackedTable` (append-only first version) and its index gate (§16.1, §16.2) | S0 | +| F0 | additive contract records only: commands, views, persistent, RIB feed, `RowNotice`, NHT modes, transport options, settings root field with delta, labels, packed-table and programming record types, F1/F2 shared records | – | +| C6 | zero-commit publication and off-tree runtime bookkeeping (§15.3) | F0 | +| F1a | order-independent resolver with the per-leg cycle rule; extended fingerprints | F0 | +| F2a | delivery of F0 records (commands, config, views, change keys) | F0, C6 | +| F3 | transport: scoped accept identity, TTL, GTSM, wildcard listeners | F0 | +| F4 | domain oracle (materialized); A1 lazy mode is separate | F0 | +| F5 | toolkit: spf, timers (shared persistent priority structure), routeops, sizes, refs, harness | – | +| F7a | settings: schema, dialects and profiles as data, device layer, transaction and action plan, root delta, minimal provenance | F0 | +| RP1 | route-policy language and evaluator | F0, F7a | +| F1b + F2b + F8 | one integration boundary with one merge owner for `derive.py`, `forwarding.py`, row and group completion: persistent packed FIB with group versions; RIB submission boundary; RIB feed and owner notifications; NHT installed mode; programming contract with ERROR completion and sync completion | S1, F1a, F2a | +| F1c | incremental resolution with the full resolver as oracle | F1b, design gate | +| F2c | runtime scale (lazy context, per-client identity notifications, admission via S1a) | F2a, S1a; performance exit after X | +| RP3 | RIB admission with retained submissions; candidate-next-hop activation filter (`ip protocol`, per original activated next hop); `nht` scope | F1b/F2b/F8, RP1 | +| RP4 | forwarding-scope resolution filter; `FIB_INSTALL` exclusion | F1c, F8, RP1 | +| A1 | lazy analytical forwarding query | F4, F1b | +| F7b | group selectors, presets, `explain`, generated docs | F7a | +| OR1 | Spark and the agent shell | F0, F2a, F5, F7a | +| OR2 | LinkMonitor and KvStore | OR1 | +| OR3 | Decision, PrefixManager, Fib with sync completion | OR2, F1b/F2b/F8 | +| OR4 | multi-area | OR3 | +| OR5 | drain, overload, RIB-feed injection | OR4, RP1 | +| OR6 | passport, study integration, learned SR | OR5; performance exit F1c, F2c | +| BG1 | types, AS_PATH, peers, settings fragment | F5, F7a | +| BG2 | sessions and FSM; a session-only NHT registration (`rib` mode) for connect gating until installed mode lands in X | BG1, F2a, F3 | +| BG3 | inbound, next hops and path tracking, the `frr` and `rfc4271` flavors, Loc-RIB on `PackedTable` with stable path slots, install; differential source fixtures | BG2, F1b/F2b/F8, F4, RP1, S1b; performance exit F1c, F2c | +| BG4 | outbound, policy hooks, MRAI, gate profiles, origination | BG3; performance exit F1c, F2c | +| BG5, BG6 | RR, confederations | BG4 | +| BG7 | conformance, FRR adapter, lane | BG5, BG6 | +| IX | integration scenarios | OR6, BG7, RP4, A1, F7b | + +Milestones: **M0** S0, F0, F5 · **M1** S1, C6, F1a, F2a, F3, F4, F7a, RP1 · **M2** the +F1b/F2b/F8 boundary, then F1c, F2c, RP3, A1 · **M3** OR1–OR3, BG1–BG3 · **M4** +OR4–OR5, BG4–BG6, RP4, F7b · **M5** OR6, BG7, IX · **M6** docs and final review. +Reviews after M0+M1, after the F1b/F2b/F8 boundary, after M3 and at the end. + +### 15.10 Readiness + +S0 can start now, as a staged series with reproducers first. F0's additive +records can start with S0. S1 and C6 start by writing their own gates; F3, F4, +F5 and F7a do not wait for `PackedTable`. Consumers of `PackedTable` and of +zero-commit publication merge only after the corresponding gates **pass** +(§16.9). + +--- + +## 16. Codex review of revision 7: dispositions (revision 8) + +Report: `netsim-protocols-research/codex-review-rev7.md` (GPT-6-Astra, xhigh; +17 minutes; 1 P0, 14 P1, 1 P2; probes copied to +`scale-experiments/codex-probes-rev7/`). All 16 findings are accepted. The body +text they contradicted has been corrected in place (D4, F5, F0, F1b, the F3 +signature, §5.8, §5.13, §13.9, §14, §15.1 and §15.10); this section records the +decisions and the contracts that remain to be built. Where it conflicts with +§15, §16 wins. + +### 16.1 P0: identity across compaction (finding 1) + +Positions are private to a table generation; external references are logical +keys or `(table_generation, slot, slot_generation)`; the first `PackedTable` is +append-only with rebuild at a generation boundary in one owner transaction +(table, indexes, pools, allocator, remap). The change certificate is bound to +exact base and result generations and names its coordinate system. Gate: +held-snapshot tests across insert, update, delete and rebuild; stale queued +references; remap failure rolling back everything; certificate correctness +against a brute-force logical diff. + +### 16.2 Indexes, budgets and admission (findings 2, 3, 4) + +- No mandated packed sorted column. Bounded devices keep persistent per-length + maps; full-table devices adopt bounded sorted blocks behind a persistent + directory only after their update benchmark passes; 128-bit IPv6 ordering + and `(network_key, row_reference)` entries are required; numeric prefix ids + are not address-order keys. +- The 40 B per row and 24 B per path figures are hypotheses for named layouts. + The **common submitted and admitted RIB with its three indexes** is assigned + to the X boundary owner (F1b/F2b/F8); if it stays an object RIB, the P-A + ceiling is revised upward and stated. BG3 uses stable path slots plus a + bounded per-destination membership and order structure, never monolithic + CSR. Passports report current, builder-peak, rebuild-peak and retained + history separately; P-A names links, interfaces and local rows per device; + P-C is instantiated (areas, degree, distribution, timers) before it can pass. +- Generic immutability and semantic admission are distinct. Certificates are + immutable evidence that never mutate a published map or table; an + owner-local admission transaction composes edits back to the admitted base; + a missing or mismatched certificate means full validation; no unbounded + ancestor chain. The same lifecycle covers `set`, `remove`, `update`, + `build` and the packed builder. S1 splits into **S1a** (`PMap` certificate) + and **S1b** (`PackedTable` and its index), each with its own gate. + +### 16.3 Publication, initialization, PRNG (findings 5, 6, 7) + +- **C6 publication seam.** A non-user-code finalizer commits accepted runtime + successors at the root publication point, **before root observers**, and + performs the same transition on accepted no-delta batches; a failed outer + commit publishes neither side and keeps retry captures; captured prefixes + are consumed before external callbacks; the pipeline distinguishes prepare, + abort, publish and external finalization. Tests place an observer ahead of + the runtime observers and inject an outer commit failure. `after_run` alone + is not the publication hook. +- **Initialization stays in the tree.** `AgentNode.initialized` remains a + one-time tree transition on a successful `on_init`; only per-run bookkeeping + (`runs`, `receipt`, `rng`) leaves the tree. Fresh runtimes keep today's + generation and reset semantics. Current receipts come from an explicit + `Simulation`/`AgentRuntime` API as freshly frozen values; bounded + `AgentRunEvent`s are emitted at runtime publication even without a root + delta; old snapshots never acquire newer receipts. +- **PRNG.** A named, versioned generator with golden vectors; **one counter per + declared module stream**, persisted across accepted runs, advanced only when + drawn; stream keys from a fixed encoding hashed with blake2b of network seed, + device identity, agent identity, generation and module; rejected runs + restore every counter; a stated supported API and reset rule; the change + from MT seed sequences documented. This is the S0 W3 contract. + +### 16.4 Resolver and policy hooks (findings 8, 9) + +- **Per-leg witnesses.** F1a defines a finite, non-self-supporting resolution + witness with its candidate and exclusion context; depth is checked per + witness, not by one scalar per record; cyclic branches contribute no weight + unless an explicitly permitted acyclic escape exists; lower-ranked and + less-specific fallback obey the selection and `lpm_fallthrough` rules; + `all_or_nothing` is preserved; deduplication and weight combination are + deterministic; cached physical results are reused only where exclusion + context and remaining depth cannot change the answer. Tests add + cycle-with-two-exits, unequal branch depths, cycle plus lower-ranked + candidate, all-or-nothing, repeated adjacency with unequal weights and a + blocked covering prefix, under every enumeration order. +- **Activation filter.** The FRR attachment evaluates once per **original + candidate next hop** after activation, with that next hop's activated type, + address, ifindex and recursion state plus permitted row facts; it preserves + the system-route and family bypasses; a deny deactivates the original next + hop and its resolved contribution, never each flattened child independently + (`zebra_nhg.c:2403-2406, 2622-2628, 2693-2716, 2909-2917`; + `zebra_routemap.c:70-87`). Native submission admission and native + whole-action FIB policy stay separate capability rows. + +### 16.5 Programming, sync, feeds (findings 10, 11, 12) + +- **Route and group obligations.** Separate route and group request + identities, desired and installed generations, and explicit dependency + completion: a route becomes installed only when its group content is + installed, and the prior installed action survives until then. Group-only + updates avoid per-prefix programming only while selection and eligibility + are unchanged; loss of eligibility triggers the budgeted affected-prefix + re-election (F1c). Supersession is resolved once at the group obligation; + obsolete acknowledgements never satisfy newer intents. Reclamation counts + desired, installed and in-flight references; snapshots own their data. + `installed=None while PENDING` is a desired-row projection and never erases + a retained installed action. Installed NHT and forwarding read installed + content and matching provenance. +- **Sync operations.** An accepted sync carries owner, generation and batch id + even when its RIB edit is identity (`rib_apply(empty, sync=...)` returns the + same RIB today). The programming boundary tracks the batch's upserts and + deletions, including cleanup of retained installed routes, and emits one + immutable completion with per-operation success and failure or a distinct + total failure; an empty no-work batch completes successfully; failed + removals are never omitted. Partial success reaches Open/R with failed + members scheduled for retry; total failure does not advance initial sync. + These records belong to F0 and the X owner, not to OR3 reconstruction. +- **Route feed coalescing.** Messages identify baseline and result epochs; + coalescing composes per-key final upserts and removals relative to the last + captured baseline (disjoint edits are preserved, net no-ops cancel); + registration and reset snapshots are distinct; large snapshots and change + sets use bounded pages or immutable table-backed views with declared byte + limits and an explicit resync outcome; entry count is not a memory bound. + +### 16.6 Settings decisions (finding 13) + +- Root settings delta integration is **F0**; resolution and the per-owner + action plan are **F7a**. Desired-layer edits are preserved when effective + values are equal. +- Preset precedence, decided: an explicitly inherited user instance value + (100) beats a preset's neighbor value (200); only absent explicit inheritance + falls back to presets. An explicit user profile (`datacenter`) beats a preset + profile (`traditional`); without a user profile the preset supplies it. + Preset recursion and conflicting meta-key combinations are rejected. + +### 16.7 RFC 8212 (finding 15) + +Missing policy discards received routes and suppresses advertisements per AF +and direction; the session still establishes; the state is reported in views; +configuration and OPEN are never rejected for it. Scenario 9 pins Established +with zero accepted and advertised prefixes, independent directions and AFs, and +behaviour after attaching policy. + +### 16.8 Body cleanup and A1 (finding 16) + +Applied in place: no step interpreter (D4, F5, §12), scoped listener and +recursive keys, a shared timer facility instead of a tuple heap, corrected §14 +rows, §13.9 reflection wording, the 10.5.4 fixture rule, and the exemption +wording in §5.13. Deferred by §14.6 and now consistent: JSON codec and +stochastic capacity or lost-completion models; ERROR, empty sync and stale-ACK +tests stay. **A1 mini-contract**, required before A1 starts: address-LPM and +exact-prefix queries; enumerable materialized state; absent, default and +aggregate answers; domain identity; cache lifetime; old-snapshot behaviour; +never a silent replacement of materialized RIB semantics. + +### 16.9 Slices, edges and owners (finding 14; amends §15.9) + +Edges added: RP1 → RP3, RP4, OR5; OR1 → OR2; OR4 and OR5 → OR6; BG5 → BG7; +F2c's performance exit waits for X (its lazy-context work may start after F2a +and S1a); BG2 owns a session-only NHT registration in `rib` mode until the +installed mode lands in X, and BG3 owns path tracking. Functional and +performance gates are separate for every protocol slice (BG3 and BG4 keep +their F1c/F2c performance gates). IX depends on OR6, BG7, RP4, A1 and F7b. + +Owners: one state owner for S1a and S1b; one C6 owner for `agents.py`, +`pipeline.py`, `network.py` publication and runtime receipt events; F0 owns +root settings-delta integration; F7a owns its resolver and planner; the X +merge owner coordinates `routing.py`, `lpm.py`, `nht.py`, `derive.py`, +`forwarding.py` and `timeline.py` against one reviewed storage and programming +contract; F2c consumes S1 and X. The worker template distributes §15.9 as +amended here. + +Milestones: **M0** S0, F0, F5 · **M1** S1a, S1b gate, C6, F1a, F2a, F3, F4, F7a, +RP1 · **M2** X (F1b/F2b/F8), then F1c, F2c, RP3, A1 · **M3** OR1–OR3, BG1–BG3 · +**M4** OR4–OR5, BG4–BG6, RP4, F7b · **M5** OR6, BG7, IX · **M6** docs and +final review. + +### 16.10 Readiness + +S0 continues as a staged series (W1 locator index and SRv6 consumer copy, +W2 masks and retention, W3 PRNG per §16.3). S1a and S1b start by writing their +gates; F0's additive records proceed, with the listener, recursive-resolution, +programming, sync and packed-table records frozen only after §16.1–16.5 are +in their signatures; C6 starts with its seam design and tests; F3, F4, F5 and +F7a do not wait. + +--- + +## 17. S0 delivered (2026-09-27) + +Three Codex xhigh workers on isolated worktree branches, integrated on `main` +locally (not pushed) after `make check-ci` and `make check-ft` on the merged +tree: **1,759 passed, 14 skipped, coverage 95.40 %** on CPython 3.14.5 and +3.14.5t. Handoff reports: `docs/history/s0-w1-locator-index.md`, +`docs/history/s0-w2-masks-retention.md`, `docs/history/s0-w3-prng.md`. + +| Fix | Change | Regression test (fails on `0e5b299`) | +|---|---|---| +| Locator scan (G17) | An off-tree immutable index `(device, locator) → block`, maintained by identity diff at every `update()` boundary, inside batches and on fork; validation is O(locators) and skipped for IPv4-only interfaces | `tests/model/test_s0_locator_consumer.py` (25 cases: linear operation count, atomic rejection, rollback, fork, root replacement) | +| SRv6 consumer copy | `consumer_index` returns the previous frozenset when membership is unchanged | same file: identity kept across an unrelated commit | +| Mask tuples (G20) | Shift arithmetic; `FrozenPrefixTable` and `PrefixTable` hold no masks; the `masks` parameter removed from `_owned` and its callers | `tests/model/test_lpm_scale.py` (1,000 tables: 1.5–6.9 MB → 0.15 MB) | +| Unbounded retention | `Simulation` and `Timeline` default to `keep_records=10_000`, `keep_events=100_000`; `Study` follows; explicit `None` stays unbounded; documented in `docs/reference.md` | `tests/runtime/test_timeline_retention.py` | +| MT state per run (G18) | `netsim/model/prng.py`: versioned SplitMix64 with per-stream counters, blake2b-derived stream keys, identity reuse when nothing is drawn; `ctx.rng` is a `Prng` with the documented API; `AgentNode.rng` holds a sorted tuple of `(stream, counter)` or `None` | `tests/runtime/test_prng.py`, `test_agent_prng.py`, `test_agent_prng_retention.py` (golden vectors, stream isolation, `PYTHONHASHSEED` independence, O(1) retention) | + +Old/new/old on the same machine (load about 2.3, fresh sequential processes, +`scale-experiments`-style probe at 2k and 6k devices): + +| Metric | `0e5b299` | S0 | +|---|---|---| +| build, 2k / 6k devices | 3.8 s / 34 s (1.9 / 5.7 ms per device, quadratic) | 0.13 s / 0.40 s (66 / 67 µs per device, linear) | +| converge, no protocol routes | 25.4 KB per device | 9.5 KB per device | +| idle agent run, retained | 26.9 KB per run | 1.8 KB per run | +| idle agent run, time (under tracemalloc) | 563 µs | 461 µs; the per-run tree commit remains and is the C6 target | + +W1's and W2's provisional numbers (build 52 s → 0.6 s at 8k; consumer helper +3.5 ms → 1.5 µs at 100k; converge 50.9 → 19.2 MB at 2k) are consistent with +these. Fingerprints are unchanged. Not done in S0, by design: the `PMap` +changed-key certificate (S1a), packed storage (S1b) and zero-commit publication +(C6). + +**S0 review fix (2026-10-02).** A Codex xhigh adversarial review of the merged S0 +diff (`netsim-protocols-research/codex-review-s0.md`) confirmed one P2 and no +P0/P1: `Device.configure(seed=...)` or a `Network.seed` edit on a live agent +re-keyed its `Prng` family mid-generation, changing the sequence without a +reset and accumulating counters (64 edits → 65 entries). Fix: the v1 committed +state is now `(key, counters)`; the key is hashed once at the generation's +first publication (`Prng.for_agent`) and restored by `Prng.restore` afterwards, +so seed edits take effect only at `reset_agent`/`restart_all`, as under the +old MT seeding. `AgentNode.rng` is `None` only before the first publication. +Also applied: `Study` with `keep={'arrays': True}` or `{'timeline': True}` +selects unbounded events unless `events` is given; `_randbelow` rejects +non-positive bounds; `lookup` is pinned to `lookup_iter`; the workload +retention tests assert explicit defaults. Lifecycle tests in +`tests/runtime/test_agent_prng.py` (seed edit before/after first publication, +64 edits with change-back, rejection then edit then retry, reset and restart +adopting new seeds). diff --git a/docs/design/archive/routing-policy-r4.md b/docs/design/archive/routing-policy-r4.md new file mode 100644 index 0000000..2e0724b --- /dev/null +++ b/docs/design/archive/routing-policy-r4.md @@ -0,0 +1,267 @@ +# Routing policy for NetSim + +Status: reviewed implementation proposal, revision 4, 27 September 2026. No routing implementation is added by this document. Scope: language choice, policy semantics, shared engine, node configuration, RIB/FIB/resolver integration, implementation sequence and acceptance. Review record: [routing-policy-reviews.md](routing-policy-reviews.md). + +## 1. Decision + +Use a small, typed **NetSim route-policy language**: Boolean matches, ordered actions, ordered terms and chains, explicit continuation and terminal allow/deny. Author it as immutable Python records; support a canonical JSON representation using the same schema. Share one pure evaluator between routing protocols, RIB admission, FIB programming and resolver filtering. Keep each attachment point's effects in its owning module. + +Use RFC 9067/OpenConfig as structural guidance and eventual interchange formats. Translate an explicitly supported FRR route-map subset first. Do not use a vendor CLI, a YANG tree, Python callbacks, Rego or CEL as the execution language. Do not build a universal vendor-language virtual machine. Reject unsupported foreign semantics instead of silently approximating them. + +This narrows the earlier [research guide](/Users/networmix/ws/RFCs/ROUTING-POLICY-GUIDE.md): a shared representation is useful; arbitrary configurable combinations of all vendor control-flow rules are not. Native semantics are a versioned contract. Vendor compatibility describes a tested translation subset, not the name of an entire operating system. + +Every modeled behavioral setting and policy binding is overridable per node. Shared definitions are immutable values; they never make a policy update network-global by accident. Preserve current behavior for nodes with no new bindings. + +## 2. Evidence and alternatives + +The current worktree was inspected at `0e5b299225372f394dd3505a09cd92127b33dfa7`. The live protocol plan now contains a substantial settings system (§2A), immutable specifications, in-place agent configuration, and a BGP-local first-match route-map subset (§5.8). Its latest captured [snapshot](/Users/networmix/ws/RFCs/evidence/policy-plan-review/claude-protocol-plan.md) and [provenance](/Users/networmix/ws/RFCs/evidence/policy-plan-review/inputs.json) are the comparison boundary; this document does not edit that live plan. + +| Approach | Useful properties | Reason it should not be the core execution contract | +|---|---|---| +| FRR/EOS route maps | Familiar, compact, directly relevant to SONiC/FRR | Similar syntax hides different continuation, matching and action-order behavior; BGP-first ownership cannot serve model-layer hooks | +| Junos policy statements | Clear terms, policy chains, reusable sets and explicit flow control | Defaults depend on context; policy expressions and route-filter lookup need additional semantics | +| IOS XR RPL | Readable conditionals, parameterization and reusable policies | `pass` continues and a later drop can override it; modification can confer acceptance. A simple terminal-permit translation is wrong | +| BIRD filters | Typed route values, conditionals and reusable functions | This is a router-specific programming language; implementing its runtime and attributes adds unnecessary scope | +| IETF/OpenConfig models | Common vocabulary, schema validation and interchange | They do not erase device capability and attachment differences; a management tree is a poor hot-path representation | +| RPSL/RPSLng | Inter-AS intent and registry-based filter generation | Does not provide the local programming/ACK/resolution lifecycle required here | +| CEL | Typed, side-effect-free expressions | Could express predicates, but NetSim would still need route effects, chaining, attachment validation and lifecycle; a second evaluator adds little initially | +| OPA/Rego | Declarative policy over structured data | Its declarative query model does not directly supply ordered route transformations; adding an external engine does not remove the routing-specific design | +| Arbitrary Python callables | Immediate flexibility | Hidden state, nondeterminism, nonserializable closures and unknown dependencies conflict with tree admission and replay | +| Small native records | Explicit semantics, direct validation, stable serialization, bounded evaluation | NetSim must maintain a small language contract; keep its scope disciplined and tests observable | + +Primary evidence: [RFC 9067 §§4–6](https://www.rfc-editor.org/rfc/rfc9067.html), [OpenConfig](https://openconfig.net/projects/models/schemadocs/yangdoc/openconfig-routing-policy.html), [FRR pinned route maps](/Users/networmix/ws/RFCs/vendor/frr/10.8.0-dev/doc/user/routemap.rst), [Junos chains](https://www.juniper.net/documentation/us/en/software/junos/routing-policy/topics/concept/policy-routing-policies-chain-evaluation-method.html), [IOS XR RPL](https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/routing/configuration-guide/routing-config-cisco8000/routing-policy-language-wrapper/routing-policy-language-structure.html), [BIRD 3.1 filters](https://bird.nic.cz/doc/bird-3.1.0.html#filters), [CEL implementation](https://github.com/cel-expr/cel-go), [Rego](https://www.openpolicyagent.org/docs/policy-language), [RPSL](https://www.rfc-editor.org/rfc/rfc2622.html). + +RFC 9067 explicitly exposes original-versus-modified attribute matching as implementation-specific read-only state. A configurable NetSim binding is a simulator feature, not a writable interpretation of that YANG leaf. FRR's pinned `route_map_apply_ext()` also distinguishes match, no-match and NOOP and applies different call/exit rules; translating every native feature into an ordinary first-match list would lose behavior. The first adapter rejects those unsupported constructs. + +## 3. Existing code and ownership + +| Current seam | Consequence for implementation | +|---|---| +| [Route, RibState, rib_apply](../../netsim/model/routing.py) | Route identity is `(prefix, source, distinguisher)` within a family; local route attributes differ from BGP wire attributes. Preserve identity and provenance across policy | +| [ResolutionPolicy](../../netsim/model/routing.py) | Already owns default-route, fallthrough, depth and ECMP behavior. Extend this owner rather than creating a second resolver-settings engine | +| [DeviceConfig / immutable records](../../netsim/model/state.py) | Use admitted frozen records, tuples and PMaps. Functions and compiled opaque objects do not belong in published configuration | +| [AgentConfig.params](../../netsim/model/contracts.py) | Holds immutable protocol settings; share policy definition types without importing BGP into model code | +| [Network.add_agent](../../netsim/model/network.py), [RibClient](../../netsim/model/entities.py), [agent route submission](../../netsim/runtime/agents.py) | Current client profiles are network-global. The node-local profile split must cover all consumers, not only agent registration | +| [NHT refresh](../../netsim/model/nht.py), [derive_fib](../../netsim/model/derive.py) | Current resolver policy is device-wide; client-scoped filters need explicit identity and invalidation | +| [SR policies](../../netsim/model/srv6.py), `Device.policy_client()` | These are segment-routing policies. Do not rename or overload them with route filtering | + +Put the shared package at `netsim/model/route_policy/`, using the explicit prefix `RoutePolicy` for public entry points and references; short syntax names such as `Term` remain scoped to this module. Initial modules: `types.py` for immutable syntax and results; `compile.py` for validation/reference resolution; `evaluate.py` for pure execution. Protocol-specific access code and scheduling stay with BGP/Open/R; the existing resolver owns resolution effects. Add a codec module only when the JSON surface is delivered. Vendor adapters live under `netsim/adapters/`, not inside the evaluator. + +No new generic settings framework: integrate with the protocol plan's §2A schema and precedence. No duplicated BGP policy evaluator: `agents/bgp/policy.py` becomes the BGP field/action adapter, attachment wiring and refresh scheduling. No changes to `SrPolicy` semantics. + +## 4. Language contract + +The following are intended record shapes, not existing callable APIs: + +```python +RoutePolicy(name, terms: tuple[Term, ...]) +Term(name, when: Expr, actions: tuple[Action, ...], flow: Flow) +RoutePolicyBinding(point, policies: tuple[RoutePolicyRef, ...], + fallback: ALLOW | DENY, match_input: ORIGINAL | WORKING) +RoutePolicyRef(name, revision) +RouteSetRef(name, revision) +PolicyResult(verdict: ALLOW | DENY, route, reason) +``` + +`Flow` has exactly four values: `ACCEPT`, `REJECT`, `NEXT_TERM`, `NEXT_POLICY`. A term's flow is explicit. A chain's fallback is explicit in resolved configuration. An absent attachment is represented separately from an attached empty chain; adapters can enforce protocol requirements such as RFC 8212 without confusing an empty policy with no policy. + +Evaluation starts with immutable original input and a working route initially identical to it. Process policies and terms in their written order. A false condition advances to the next term. A true condition executes its ordered actions, then its flow. `NEXT_TERM` continues this policy; `NEXT_POLICY` skips its remaining terms. End of a policy continues the chain. `ACCEPT` and `REJECT` terminate the entire chain. End of the chain applies its fallback. Acceptance returns the working route; rejection returns no publishable route and discards tentative edits. A later policy cannot undo terminal rejection by accepting. + +A `REJECT` term must have no actions; reject that combination at compilation rather than quietly ignoring side effects. A terminal acceptance applies to the whole binding, not merely the named policy. On every invocation, ORIGINAL means the input to **this attachment**, before any term in its chain; it is not necessarily the original BGP UPDATE. Earlier attachment results are already part of this input. Reapplication obtains that attachment's retained pre-policy input. + +Actions always read the current working route. Conditions read either original or working route according to the binding's `match_input`, fixed for that complete invocation. Native default is `WORKING`. There is no hidden per-term profile lookup. Context facts such as peer identity cannot be changed by actions. A profile change is validated and applied through node configuration. + +There are no loops, arbitrary jumps, recursive calls, runtime name resolution, general expressions in actions or arbitrary user functions. Reuse comes from named sets, ordered chains and construction-time helpers that produce immutable records. Helpers are ordinary scenario-authoring code, never serialized callbacks. An initially sufficient language must express the examples in §8 before adding more control flow. + +### Matches + +Use typed leaves composed with `AllOf`, `AnyOf` and `Not`. Empty conjunction/disjunction are rejected; use explicit `Always`/`Never`. Leaves cover prefix membership with exact/min/max length, address family, source protocol/client, origin kind, peer/role, tag, next-hop address/interface, community membership and scalar comparison. BGP leaves add AS-path contains/first/last/length, origin and local attributes. Named-set predicates carry a `RouteSetRef`; both definitions and their referenced sets are pinned. A per-node set change constructs a new policy revision/reference through authoring helpers rather than mutating a globally shared name. + +Prefix sets are unordered unions of typed prefix ranges. Separately provide `PrefixListPermits(ref: RouteSetRef)`, a typed predicate over ordered `(range, permit|deny)` entries with first-match result and implicit false at exhaustion. This small domain predicate lets the FRR adapter preserve common mixed permit/deny prefix lists without expanding them into quadratic Boolean expressions. It is not an arbitrary nested policy call. Neither construct claims Junos longest-match/walkup semantics; reject unsupported foreign filters. IPv4 and IPv6 never match across families. Community sets specify any/all/none; replace/add/delete are distinct actions. AS-path predicates operate on the protocol's structured path, preserving AS_SET/confederation distinctions. + +Use separate field identifiers for BGP MED and local RIB metric, BGP LOCAL_PREF and local distance, protocol NEXT_HOP and resolved egress, and source client versus peer. Fields are a closed tagged union, not arbitrary string paths into `attrs`. A field unavailable at an attachment is a compile error even inside an unreachable branch. A legal but absent optional attribute is not zero: `Present(field)` tests presence; ordinary comparisons on absence are false; `Not` negates that Boolean normally. Operators, AFs, literal types and ranges are checked at compile time, including rejecting bool where an integer is required. Typed next hops retain family/interface scope; an IPv6 link-local address without required scope is invalid. + +Set-valued predicates state their quantifier. Community ANY means nonempty intersection, ALL means every member of the nonempty configured set is present, and NONE means disjointness. Empty configured match sets are rejected. Next-hop address/interface predicates select proposed or resolved values and use explicit ANY or nonempty-ALL over that collection; no hidden choice of the first ECMP leg. A FIB allow/deny applies to the complete selected action, not per-leg pruning. BGP protocol next-hop rewriting and replacing a local row's complete next-hop tuple are distinct action variants. + +Raw regular expressions are excluded from the initial native core. Typed AS predicates cover the required initial examples. The live plan's unspecified regex subset needs a separate bounded grammar and source-backed semantics before acceptance; do not silently run Python `re` on an invented AS-path rendering. This is a deliberate scope revision to §5.8, not an implied implementation of its regex promise. + +### Actions and typed access + +Initial actions: set local preference, MED, weight, origin or tag; add/delete/replace communities; prepend an explicit AS sequence; set an explicit next hop; set local RIB distance/metric where that hook permits it. FIB and resolution hooks initially need allow/deny only. Additional programming attributes can be added with a concrete scenario and attachment validation. No generic `set(field_name, Any)` escape hatch is public. + +The capability table is declared once and drives validation and documentation: + +| Point | Read domain | Permitted writes in first delivery | +|---|---|---| +| BGP_IMPORT | BGP candidate attributes, common prefix/source facts, ingress peer facts | BGP attributes/local weight and local tag; next hop only with typed family/scope | +| BGP_EXPORT | Selected BGP attributes/common facts and egress peer facts | Advertised BGP attributes and advertised next hop; no RIB distance/metric or identity changes | +| BGP_RIB_EXPORT | Selected BGP attributes and proposed local Route | Local row metric and next hops; selected BGP path is immutable | +| REDISTRIBUTE to BGP | Source common Route facts and explicit destination-BGP defaults | Derived BGP attributes and local tag; source row and protocol ownership are immutable | +| RIB_ADMISSION | Local Route prefix/family/source/origin, distance, metric, tag and proposed next hops | Local distance, metric, tag and next hops; no prefix/source/key/table rewrite | +| FIB_INSTALL | Selected route common facts and resolved forwarding facts | None; allow/deny only | +| RESOLUTION_IMPORT | Candidate common facts plus explicit resolver-query facts | None; allow/deny only | + +Redistribution to other protocols needs a destination-owned capability row when that protocol's origination feature lands. A generic attachment name does not grant BGP setters to OSPF/Open/R. AS-path first/last matches return false for an empty path or an ambiguous endpoint set, and examine the non-confederation external endpoint. Native path length means BGP hops: sequence ASNs count individually, a nonempty AS_SET counts once, and confederation segments do not count. Contains requires `NON_CONFED` or `ALL_SEGMENTS`, with membership including applicable AS_SET members. The FRR adapter translates only the pinned meanings it can prove. Local-pref/MED are unsigned 32-bit values; origin is a closed enum; the local distance domain follows the model's validated routing contract rather than a guessed vendor-wide range. + +The evaluator does not manufacture a giant universal route record. A trusted, code-owned `PolicyAccess[T]` supplies typed field reads and ordered action application over the owner's immutable route representation. These functions are passed by the owning module at execution, never supplied by configuration or stored in the state tree. A closed field/action catalog validates every combination. BGP owns BGP attribute reads and transformations; the model owns local route fields. Apply transformations to an invocation-local working value, publishing only an accepted result. + +The access protocol has three operations: `read(route, field)`, `read_context(context, field)` and `apply(route, ordered_actions)`. Inputs contain only explicit immutable local facts; no NetworkState, clocks, RNGs, external lookups or hidden NHT queries are available. Reads are pure and total for the declared field domain. No-op application returns the input by identity. The compiler records the fields read/written, including named-set dependencies. Trusted implementations are code-owned, but are still tested with guarded inputs for locality and mutation. + +Compile-time errors reject the entire candidate configuration. A legal absent attribute follows the presence rules above; it is not an error. Unexpected access/type failures are invariant violations that abort the owner's transaction without partial publication and retain a diagnostic. They are never converted silently into ordinary policy denial. Protocol limits on an accepted transformed route, such as outgoing BGP message size, remain explicit host validation after policy and before emission. + +Compilation resolves named sets, validates ranges and attachment legality, computes a canonical semantic fingerprint, and produces immutable terms with resolved references. It does not generate Python code or introduce bytecode. The interpreter dispatches over a closed set of typed operations; additions require one semantic contract and tests. + +The public schema version owns semantics; two language versions cannot silently share a compiled fingerprint. Serialize tagged unions with fixed field names and canonical numeric/address forms. Preserve chain/term/action/prefix-list order; sort only mathematically unordered sets. Compute stable fingerprints from canonical bytes, not Python `hash()`, object identity or metadata labels. Missing references, duplicate definitions/term identifiers, reference cycles and malformed expression trees fail compilation. Resolve references once, without recursive textual expansion. Each term executes at most once per invocation; expression and set costs are bounded by their explicit input sizes. No unmeasured latency promise follows from that bound. + +## 5. Attachment points and stage effects + +These hooks are separate even when a vendor uses the same word, such as import or export, for several of them. + +| Point | Subject and owner | Meaning of rejection | +|---|---|---| +| `BGP_IMPORT` | Received candidate; BGP inbound | Candidate is filtered; any previously admitted incarnation is withdrawn appropriately | +| `BGP_EXPORT` | Eligible path + target peer; BGP outbound | Suppress/withdraw advertisement for this peer only | +| `BGP_RIB_EXPORT` | Selected BGP path plus proposed local Route; BGP zebra adapter | Withdraw/omit that protocol submission, without altering BGP's selected path | +| `REDISTRIBUTE` | Source RIB route + target protocol; originator | Do not originate the derived destination-protocol route | +| `RIB_ADMISSION` | Protocol-selected row submitted to main RIB; submitting owner | Keep protocol selection separate from main-RIB admission | +| `FIB_INSTALL` | Selected/resolved action; model programming stage | Desired action is absent by policy; remove any older installed action; do not try the next RIB candidate implicitly | +| `RESOLUTION_IMPORT` | Potential underlay route + resolver context; resolver | Exclude this candidate from this query's resolver inputs | + +`BGP_RIB_EXPORT` is the FRR `table-map` boundary. Its read view includes BGP attributes and the proposed local row, but its writable result is **only that row**: local metric and next hops initially. `SetMed` is not a legal alias for `SetRibMetric` here. The BGP owner retains the selected path and regenerates the submission after policy edits; a denied new submission deletes a previously submitted row. This avoids pushing BGP wire-attribute interpretation into the generic device RIB. + +`RIB_ADMISSION` follows that boundary and applies to all supported submitting sources using common local Route fields. Add a single model-owned submission helper used by agent RouteOps, `RibClient`, route-source updates and generated-route writers. Keep `rib_apply` as the pure table primitive underneath; it must not unexpectedly execute policy for unrelated internal uses. Audit every production `rib_apply` callsite and require explicit submission/bypass ownership. Generated connected/local and SR-local routes remain trusted, unbound internal sources in the first delivery; attempts to attach an admission filter to them are explicit unsupported configurations, not ignored policies. Their protocol-specific correctness rules are not an optional routing knob. + +When the first device RIB-admission binding becomes active for a family, retain the existing unfiltered RibState as that family's immutable **submission view**, and derive the admitted view separately. Thereafter add/delete/sync operations update the submission view first, even for rejected routes; the affected admitted rows are regenerated from those inputs. Reuse the existing RibState/ClientId indexes rather than inventing another raw-table format. With no bindings, keep the old single-view fast path. On last binding removal, the submitted table becomes the admitted table and the extra view can be released. Shared unchanged row objects avoid copying every route payload; measure index overhead. Owner/generation deletion purges retained inputs as well as admitted rows, so a later allow cannot resurrect a dead agent's route. + +Trusted bypass writers update **both** views while retention is active. Otherwise detaching the last policy could resurrect an old connected/SR route. Batch collapse/equality and sync reconciliation operate against submitted rows, not admitted rows; a new version of a still-denied route must not disappear as a no-op. Persist raw-only changes and verify StateDelta remains nonempty, while avoiding unnecessary FIB work if the admitted output is unchanged. The current `changed_paths()` iterates DeviceState fields, so reuse that mechanism rather than inventing an unrelated event bus. + +The submission audit includes `derive.py` connected/local and SR-local writers; `igp.py` analytical IGP synchronization; `entities.py` RibClient; `network.py` batch reconciliation; and `runtime/agents.py` RouteOps. These are the production callsites found in the inspected tree. New route sources must use the same boundary. An explicit bypass means trusted generation semantics, not permission to omit retention maintenance. + +The generic admission policy sees local distance, metric, tag, source, prefix and proposed next hops. BGP community/AS matching at this boundary is unavailable; use `BGP_RIB_EXPORT` while those attributes are still owned by BGP. FIB policy reads selected/resolved common facts. Resolver policy reads candidate route facts plus the query context. This deliberate capability split avoids a second universal attribute store. + +For resolver filtering, explicitly identify the route source view (selected RIB or installed forwarding state), VRF/table, family, client and query mode. Filtering does not copy routes across tables. Ordered table fallback and less-specific fallback remain resolver settings, not arbitrary policy control flow. Policy has no authority to disable recursion-loop detection, scope link-local addresses globally or fabricate an installed next hop. Rejected and absent candidates are dependencies too. + +The binding owner selects a valid underlay view before evaluation. In installed mode, rejected/noninstalled rows are never brought back merely because policy allows them. The resolver tests the permitted candidates at its lookup boundary; a denied more-specific can try a less-specific only when the effective `lpm_fallthrough` rule allows it. Resolution does not invoke the route's import policy again. Existing code stores RIBs by family and its RowKey omits the Route.table field: do not claim arbitrary VRF/table coexistence is implemented. Carry table scope in the design, but reject bindings for an unsupported table/VRF until the owning routing model supports that scope without aliasing. + +Do not apply route policy to packets, LSDB flooding or SPF topology events. Packet PBR, SR steering and route leaking remain their own owners. If route leaking is implemented, it can reuse the evaluator through a typed future attachment with provenance/loop rules; a policy record alone does not implement a table or VPN feature. + +Export acceptance cannot bypass protocol eligibility, AS/cluster/originator loop protection, session/AF capability, inactive-route eligibility or a FIB gate. A next-hop export action changes an advertised copy; it does not change the locally installed next hop. Hardware programming exclusion, capacity failure, pending completion and later removal are separate states. See the existing [FIB ACK/inactive analysis](/Users/networmix/ws/RFCs/FIB-ACK-AND-INACTIVE.md). + +Keep desired and actual forwarding separate when policy removes an installed route. Rejection sets desired absence with reason `POLICY_EXCLUDED`; an existing entry persists until the modeled DELETE completes. No ADD is pending and a DELETE ACK is not an install success. On failure of deletion, expose the retained actual entry and retry/failure state. Do not replace a FIB-policy exclusion with an implicit discard action or a fallback candidate. + +FIB request identity includes node/owner generation, table/family/prefix and desired-action generation. A binding edit that changes the desired action, its eligibility or gate contract supersedes pending requests. An unchanged action/eligibility/gate can keep its existing request after explicit equivalence checking; do not reprogram identical forwarding merely to carry a new source label. Old completion for a superseded request cannot satisfy the current gate. This uses F1/F2's programming contract, not a second policy-specific ACK mechanism. + +The EOS documentation conflict recorded in the research guide remains unresolved. Do not use either conflicting passage as the generic default. Native `FIB_INSTALL` policy affects installation only; export changes only through a separately configured gate or eligibility policy. A compatibility adapter requires a pinned executable fixture before asserting EOS behavior. + +## 6. Configuration, lifecycle and cost + +Definitions and named sets are immutable, revisioned values. A node's effective binding references exact revisions. Reusing a definition does not subscribe every node to a mutable name. A configuration transaction may explicitly replace bindings on several selected nodes, but node edits remain local. Follow the existing settings precedence; replace ordered chains and definitions as whole values rather than merging terms by name. + +There is exactly one resolved binding for `(node, attachment, owner/instance, family, supported table, peer when applicable)`. Group/instance/peer layers choose or replace that binding through the existing schema; the evaluator never runs each inheritance layer as an extra policy. Chains provide composition explicitly. Node defaults, individual bindings, match-input mode, automatic/manual reevaluation, delay and replay-unavailable behavior all have node-local override paths. Scope restrictions are type constraints: a FIB-only field need not accept peer scope. Default field values and provenance are provided by §2A's resolver once, not a second merge algorithm. + +Use the planned settings tree for desired definitions/references. Materialized device-hook bindings live with DeviceConfig; materialized protocol-hook bindings live with the corresponding AgentConfig.params. Equal immutable definition payloads may be shared by reference. BGP sweep/refresh status belongs to that agent's immutable runtime state; device-hook work belongs to the model/pipeline's existing per-device scheduling state. No counters, deadlines or executable adapter functions enter a shared definition. P0 names these records with the settings/F0 owner; no external mutable registry becomes authoritative. + +Resolve all references and validate the complete candidate configuration before publishing it. One bad field, missing reference or illegal action rejects the configuration transaction. Scenario defaults and node/instance/AF/peer overrides compile into concrete bindings once. Explanations retain provenance separately from semantic fingerprints. Serialization records the native language revision, foreign source/profile identity and any explicit deviation from compatibility defaults. + +Maintain three concepts: committed configuration, active definition for new inputs, and completion of reevaluation of older inputs. Changing a definition atomically does not make a remote route refresh instantaneous. For the native behavior, activate a validated definition for new input at configuration commit and schedule existing-route reevaluation. The node-local default is automatic reevaluation with zero delay; compatibility profiles may delay or require manual reevaluation. This is outside the evaluator. + +Use `reevaluation_mode=automatic|manual` and `reevaluation_delay_s`, owned per node/protocol attachment domain. In automatic mode, successive edits replace pending work with the newest revision and restart the debounce deadline; manual mode keeps a visible pending revision until a command or new route event processes it. A scheduled work item captures owner generation and target revision, reads the latest valid input when executed, and cannot publish an obsolete result after a withdrawal or replacement. BGP import replay, outbound duplicate suppression, RIB refresh, resolver invalidation and FIB work remain separate owner operations. No global scan/timer crosses nodes. + +**FRR nuance:** the inspected `bgp_route_map_mark_update()` restarts a debounce timer; a value of zero disables route updates while refreshing references with `route_update=false`. Changing an already-running timer to zero executes that pending update first (`bgp_vty.c`). Translate this into a mode plus delay and an explicit transition action; never turn vendor zero into native immediate reevaluation. Existing map mutation and create/delete pointer refresh are not proven to have identical new-input timing. Exact pre-expiry behavior requires the pinned daemon fixture; native activation-at-commit is a declared simplification until that fixture establishes an adapter-specific need. Sources: [update scheduling](/Users/networmix/ws/RFCs/vendor/frr/10.8.0-dev/bgpd/bgp_routemap.c), functions `bgp_route_map_mark_update` and `bgp_route_map_process_update`; [CLI transition](/Users/networmix/ws/RFCs/vendor/frr/10.8.0-dev/bgpd/bgp_vty.c), `bgp_set_route_map_delay_timer`. + +Import reevaluation needs complete pre-policy inputs or peer refresh. With retained input, replay once from that input. With negotiated ROUTE-REFRESH, request it and process returned routes under the active revision. With neither, report `needs_session_reset`; new inputs use the active definition but the old-route sweep remains incomplete. A node's `on_replay_unavailable=report|reset_session` controls whether to request a reset automatically; native default is report. The view must never call this mixed revision state settled. Enhanced route refresh provides BoRR/EoRR boundaries; a basic refresh alone does not provide that completion marker, so report completion as unverified unless another modeled protocol mechanism proves it. Do not infer completion from an arbitrary quiet interval or confuse EoR with EoRR. [RFC 7313](https://www.rfc-editor.org/rfc/rfc7313.html). + +For native device RIB/FIB/resolver hooks, retained local inputs allow complete reevaluation at the scheduled node event; any derived programming remains asynchronous. A raw-only change and a same-result revision adoption are real state changes but need not emit new BGP advertisements or FIB writes. Reset/fork/fresh-runtime reconstruction carries definitions, active revisions and pending work according to the existing owner lifecycle; replay must not lose or duplicate a pending edit. + +Start with no policy-result cache and no global interning service. Cache compiled immutable bindings within their owning effective configuration. Reevaluate the routes attached to an affected binding; node-local conservative work is the correctness baseline. Add fine-grained route indexes only if measurements show they matter, preserving a full local reevaluation oracle. Do not build dependency indexes for fields that cannot affect a decision. + +BGP update-group sharing must include the compiled export-binding semantic fingerprint and every relevant peer/context input. A policy name is insufficient. Initially split groups by the complete declared peer/context projection; optimize sharing only when equal policy inputs imply equal outputs. Resolver query identity similarly includes its policy binding and scope. With no result cache, correctness still depends on scheduling reevaluation after edits and underlay changes; removing caching does not remove negative dependencies. + +Store no mutable counters inside shared policy definitions. Optional bounded traces record input identity, attachment, matched terms, edits, final reason and downstream eligibility. Full traces are opt-in. All routing outcomes must be identical with tracing off/on. Published records obey `validate_immutable`; retain current free-threaded ownership rules. No performance improvement is claimed before measurement. + +Use the same evaluator for normal execution and explanation; trace collection is an optional sink outside published route state, not another interpreter. `explain_effective_policy` reports the winning configuration sources, exact definition revisions, fallback, capabilities and active/reevaluated status. `explain_route_policy` additionally reports the local input and term/action path. Limit output and count omissions rather than retaining an unbounded route history. + +## 7. Compatibility policy + +Initially implement a typed FRR adapter for first-match permit/deny maps using the proven predicate/action subset, including ordered prefix-list predicates. It produces native records and a translation report: exact mappings, rejected constructs and intended deviations. It does not claim full CLI parsing. At first, reject call, continue/goto, NOOP-sensitive matchers and unsupported filter semantics unless a dedicated exact lowering is delivered and tested. The adapter must normalize FRR's documented grouping of matches and action order; preserving text order by assumption is insufficient. Exact grouping/order fixtures are part of P5. + +The canonical native language does not grow merely to claim more import coverage. A new foreign feature must have an operational use case, pinned expected behavior, a compact exact representation and tests for tricky composition. IETF/OpenConfig serialization is a later adapter over the supported subset. Reject IETF condition-subroutines until their Boolean return and side effects can be preserved; FRR called-deny and Junos policy-expression results are not equivalent substitutes. + +Use pinned corrected FRR behavior for current modeling, with 10.5 differences as deliberate behavior settings where useful. Keep historical bugs out of normal settings. The earlier [FRR comparison](/Users/networmix/ws/RFCs/FRR-VERSION-DIFFERENCES.md) records exact snapshots and limits; “10.8” here is a development commit, not a released standard of correctness. + +## 8. Expressiveness and adversarial examples + +All examples below are required design fixtures, not tests already executed. + +| ID | Policy/scenario | Expected result | +|---|---|---| +| E01 | Customer import: allowed prefix AND acceptable path; set LOCAL_PREF 200, add customer community, accept; fallback deny | Only permitted customer routes admitted; shared received attributes unchanged | +| E02 | Tag in one policy and match that tag in the next | Working mode sees the new tag; original mode does not; actions still operate on working state | +| E03 | Reject bogons, then select outbound communities/AS prepend for one peer | Rejected routes never reach later transforms; another peer is unchanged | +| E04 | Protocol-selected BGP route denied by RIB admission | BGP candidate/export eligibility preserved; no admitted main-RIB row | +| E05 | FIB-deny a selected /24 with an installed covering /16 | /24 programming excluded; packets may follow /16. This is not an explicit discard route | +| E06 | Same next hop, two node/client resolution filters: allow IGP versus deny that IGP source | Distinct results and complete dependencies; no cache cross-talk | +| E07 | Two nodes share a policy revision; an unmatched prefix reaches fallback ALLOW on A and node-overridden DENY on B | Node B override changes no state or decision on A | +| E08 | Reapply an edited import policy after prior prepend | New policy runs once on original received path, not the already prepended value | +| E09 | FIB policy changes while an older programming request is in flight | Old success cannot release an advertisement for a new desired action | +| E10 | Term transforms then `NEXT_POLICY`; next policy rejects | No tentative edit reaches a route table or a peer | +| E11 | Filtered resolver /32 becomes permitted without a new BGP UPDATE | Relevant next-hop users are reevaluated, including prior negative results | +| E12 | Attach BGP LOCAL_PREF action to FIB installation | Configuration rejected before any node state changes | +| E13 | Replace a denied submitted row, then allow it; detach final policy after a connected-route deletion | Latest submitted row appears; deleted generated route does not reappear | +| E14 | Change installed route to FIB-denied, with delayed/failed DELETE | Actual old forwarding remains observable until successful removal; no false install ACK | +| E15 | Edit twice during debounce, then withdraw a route before the sweep | Latest revision runs on current inputs; route is not resurrected by stale scheduled work | +| E16 | Route-map timer set to zero with a pending timer, then edited again | Pinned FRR transition and disabled automatic processing are distinguished from native zero-delay mode | +| E17 | No retained pre-policy input and no refresh capability | New-input revision is visible; existing-route completion remains reset-needed, never falsely settled | +| E18 | Same export policy name/revision but different peer facts read by the policy | Update-group sharing does not give both peers the first peer's result | +| E19 | Prefix list denies a /24 before permitting its covering /16 | /24 is false and another /24 within /16 is true; union-of-permits translation is rejected | +| E20 | Same semantic policy with reordered unordered sets versus reordered actions | Set reordering preserves fingerprint/result; action reordering may change both | +| E21 | Missing MED compared with zero, then Not and Present compositions | Missing is not numeric zero; negation/presence follows the explicit native truth rules | +| E22 | Later REJECT versus a throwing access adapter after an earlier tentative transform | No partial route publication; policy deny and invariant failure remain distinct | +| E23 | Node fork/reset with an active revision and pending delayed sweep | Correct lifecycle reconstruction, one current sweep, no cross-fork state or stale generation | +| E24 | No policy bindings on an existing plain-IP scenario | Existing default behavior/fingerprints and single-view RIB path preserved | + +Illustrative construction (names are proposed, not current APIs): + +```python +customer = RoutePolicy('customer-in', terms=( + Term('authorized', + AllOf(PrefixIn(RouteSetRef('customer-prefixes', 'r1')), AsPathFirst(64512)), + actions=(SetLocalPref(200), AddCommunities((StandardCommunity(65000, 100),))), + flow=ACCEPT), +)) +binding = RoutePolicyBinding( + point=BGP_IMPORT, policies=(RoutePolicyRef('customer-in', 'r1'),), + fallback=DENY, match_input=WORKING) +``` + +`StandardCommunity` validates two unsigned 16-bit components; large and extended communities use distinct typed constructors. Source protocol/client, peer and route origin remain separate fields; redistribution must not overwrite provenance to make matching convenient. `RoutePolicyRef` is deliberately distinct from the existing SR `PolicyRef`. + +Revision labels such as `r1` are human-facing in this illustration. Resolved references and serialized manifests also pin the canonical content digest; reuse of a label for different bytes is rejected. A materialized configuration is self-contained for replay and does not fetch a later policy/set revision from disk or the network. + +## 9. Implementation slices and gates + +| Slice | Deliverable and dependency | Exit evidence | +|---|---|---| +| P0 ownership | Coordinate with F0/F7 settings owners: node-local client profiles, node bindings, source retention contract; confirm F1 resolver/programming boundaries | Two-node/same-client/different-profile reproducer becomes a passing regression; no client-identity workaround | +| P1 language | Types, compiler, pure evaluator, typed prefix/community/AS predicates and immutable results | E01–E03, E10, E12, E19–E22; explicit flow/default/absence cases; no mutable admission escape | +| P2 BGP | BGP access adapter, import/export/RIB-export/origination hooks, refresh and edit scheduling | E04 at BGP boundary, E08, E15–E18, peer isolation, policy × reflection/next-hop transforms, RFC 8212; depends on BG1/BG2 attributes/session contract | +| P3 RIB | Admission for agents and existing route sources, retained originals and restore-on-policy-edit | E04 at device boundary, E13, distance/metric changes, deny→allow and allow→deny, generation purge | +| P4 resolver/FIB | F1/F2 integration, resolver filters, installation exclusion, reasoned state and ACK causality | E05/E06/E09/E11/E14, mixed node profiles, inactive × FIB-gate × policy interaction | +| P5 interfaces | Canonical JSON codec, typed FRR subset adapter, explanations | E16/E19 source comparisons, round-trip equality, precise errors, grouping/action-order fixtures; reject unsupported constructs | +| P6 acceptance | Integration, replay/fork/reset, free-threaded correctness and measurements | E07/E23/E24 plus entire matrix, scope documentation, all required checks and measured local work limits | + +P1 belongs before BG4 policy work; BGP must consume it rather than building an independent evaluator. P0 and P4 coordinate with F1's resolver redesign instead of applying ad hoc filtering around the current combined selection/FIB function. Device-side policy hooks are part of this plan's first complete delivery, not optional future work. + +Coordinate through owned contracts, not simultaneous edits to shared files: settings/F0 owner declares binding storage and node-profile changes; policy owner supplies types/compiler/evaluator and tests; BGP owner implements its access/hooks; F1/F2 owner implements submission, resolver/programming integration and notifications. Freeze the small type/capability contract before parallel implementation. This is an ownership plan, not an instruction to launch agents or modify their live plans. + +Minimum changes to the captured protocol plan: replace the §5.8 BGP-local evaluator with this shared package; add BGP_RIB_EXPORT to the zebra adapter; add P0/P3/P4 contracts to F0/F1/F2/F7; replace the undifferentiated `DELAYED = 5 s` action with owner-defined reevaluation semantics; revise the unspecified regex promise; key update groups by effective semantics and relevant peer input rather than policy name alone. Do not make P5 full vendor CLI support a dependency of native routing-policy integration. + +Validation must cover: full evaluator truth/control-flow cases; each hook's legal fields/actions; source retention and dynamic updates; fallback versus discard; negative dependencies; policy revision changes with equal outputs; stale ACKs; same prefix across node/table/client scope; fork/reset/replay; unchanged plain-IP fingerprints for unconfigured nodes. A standalone policy result is insufficient evidence for a hook's integrated behavior. + +For implementation, follow [AGENTS.md](../../AGENTS.md): source-backed reproducer and failing old-code regression for defects, `make check-ci`, and `make venv-ft` then `make check-ft`. Local checks do not replace CI. Measure same quiet machine old/new/old for unconfigured baseline, bulk initial policies, one-node policy edits and route churn; report absolute time, memory, affected-route counts and equivalence. Do not promise a numeric speedup from this plan. + +Measure with representative 1k/10k-node fabrics and route/peer counts from the protocol plan's scale passports, plus small adversarial topologies. Separate policy evaluation cost, retained-input/index memory, binding compilation and downstream routing work. A one-node edit must never enumerate unrelated nodes; within its affected attachment, full reevaluation is permitted as the initial correctness baseline. Every material regression requires investigation and a documented acceptance decision before release, not an invented universal percentage threshold. Vendor differential fixtures compare normalized decisions and event boundaries against exact builds; source inspection alone never earns a conformance label. + +## 10. Completion boundaries + +This planning task is complete when the language choice has evidence, the execution contract is precise, all hooks have ownership and lifecycle rules, per-node configuration is explicit, the implementation sequence is actionable, and adversarial reviews have been applied with findings dispositioned. Implementation is complete only after P0–P6 and their acceptance evidence. Full vendor CLI coverage, a universal compiler, policy-driven packet forwarding, route-leaking/VPN features, general regexes and a YANG management server are separate expansions. diff --git a/docs/design/archive/routing-policy-reviews.md b/docs/design/archive/routing-policy-reviews.md new file mode 100644 index 0000000..a3a27d1 --- /dev/null +++ b/docs/design/archive/routing-policy-reviews.md @@ -0,0 +1,90 @@ +# Routing-policy plan: adversarial reviews and refinements + +27 September 2026. These are sequential self-review passes with different acceptance lenses, not claims of independent reviewers or vendor conformance runs. The implementation target is [routing-policy.md](routing-policy.md). Earlier drafts and source provenance are retained in [research evidence](/Users/networmix/ws/RFCs/evidence/policy-plan-review/). + +## Review method + +Review the actual draft against current source, primary language descriptions and concrete route/event traces. For every finding, classify the claim and change the plan or record why no change is justified. A heading/link check proves document integrity only. It does not prove future implementation correctness. The final audit maps the user's requirements to the revised artifact and its evidence. + +## Pass 1 — semantics and expressiveness + +Input: revision 1, SHA-256 `9cf5f3ef55047f2ff063ea6629094748a4efc7f19e514cc21f0bf754263a3ff5`. Review focused on composition, attachment effects and edits that must restore previously filtered information. + +| ID | Finding / adversarial case | Disposition and refinement | +|---|---|---| +| S01 | RIB admission had no concrete retained-input owner. Deny→allow cannot reconstruct a rejected row | Confirmed; retain submission and admitted views, reuse RibState indexes, centralize submission handling (§5) | +| S02 | A single generic admission hook cannot both remain protocol-neutral and inspect full BGP attributes for table-map | Confirmed; separate `BGP_RIB_EXPORT` from generic `RIB_ADMISSION`, with exact read/write boundaries (§5) | +| S03 | ORIGINAL could mean pre-UPDATE or pre-attachment, producing different results after redistribution/export | Confirmed ambiguity; define original relative to each invocation and require pre-policy input for replay (§4) | +| S04 | Terminal reject with actions suggests meaningful side effects before rejection | Confirmed simplification opportunity; reject that syntax, discard earlier tentative edits on later rejection (§4) | +| S05 | Proposed PolicyRef collides with existing SR policy terminology | Confirmed; public name is RoutePolicyRef; existing SrPolicy/PolicyClient remain unchanged | +| S06 | Absent MED could be confused with zero, and a BGP-only field might be read at a device hook | Confirmed; separate unsupported-at-hook errors from legal absence; add Present and explicit comparison semantics (§4) | +| S07 | Resolution permission could reintroduce a noninstalled path into installed-mode lookup | Confirmed ambiguity; source-view selection precedes policy; permission cannot fabricate eligibility (§5) | +| S08 | Policy rejection could accidentally trigger less-specific fallback irrespective of resolver settings | Confirmed; reuse effective fallthrough and ordered-table rules (§5) | +| S09 | Mentioning VRF/table keys implied current model support despite family-keyed RIBs and RowKey shape | Confirmed; retain design scope, reject unsupported live bindings until routing owner supports them (§5) | +| S10 | Import replay without raw routes or ROUTE-REFRESH cannot completely enact a policy edit | Confirmed; explicit pending/reset-needed state, old effective binding retained unless reset is authorized by configuration (§6) | +| S11 | Example community literal looked like an encoded integer and was ambiguous | Confirmed; typed StandardCommunity(65000, 100) and distinct community kinds (§8) | +| S12 | “Just use a broader language” would remove these lifecycle problems | Rejected claim; language syntax does not supply retained input, owner/generation purge or installation boundaries | + +Result: revision 2 incorporates S01–S11. The pure language remains small; new complexity was assigned to the stage that already owns the affected state. + +## Pass 2 — lifecycle, concurrency and current-code integration + +Input: revision 2, SHA-256 `0c6d19f99a17074432d0cfeba7f5b7fcaf176f7a12751bffd3e51707b57a2fc6`. Reread production submission callsites, StateDelta, NHT/derive entry points and pinned FRR timer code. Reran the existing read-only two-node ClientProfile probe: the same client with distance 20 on node A and 200 on node B is still rejected. This confirms a current ownership requirement gap, not a regression claim. + +| ID | Finding / adversarial case | Disposition and refinement | +|---|---|---| +| L01 | Trusted generated-route bypass plus retained raw RIB can restore deleted connected routes when policy is detached | Confirmed; every bypass updates both views, and all production writers are enumerated (§5) | +| L02 | Batch comparison against admitted rows can discard changes to a still-denied route | Confirmed; batch/sync uses submission view; raw-only changes persist; no unnecessary FIB programming (§5) | +| L03 | FIB-deny on an already installed route was described only as blocking requests | Confirmed; desired absence schedules DELETE, with actual forwarding visible until completion/failure (§5) | +| L04 | Changing every policy revision could force needless FIB reprogramming, while accepting any old ACK is unsafe | Confirmed; action/eligibility/gate equivalence permits request reuse; superseded request cannot release current gate (§5) | +| L05 | “Apply after five seconds” conflates definition activation with an old-route sweep | Confirmed; distinct active and reevaluated revisions; exact FRR pre-expiry timing remains a named compatibility fixture (§6) | +| L06 | Zero-delay normalization would mis-model FRR zero, including disabling a pending timer | Confirmed directly in bgp_routemap.c / bgp_vty.c; mode plus delay and explicit pending-timer transition (§6) | +| L07 | Revision 2's keep-old-binding rule for unavailable replay creates a second activation model and hides new-input behavior | Refined S10; native activation is uniform, incomplete historical sweep is explicit, per-node report/reset action (§6) | +| L08 | Basic ROUTE-REFRESH cannot by itself justify a fully settled status | Confirmed from RFC 7313 boundary distinction; report unverified completion unless a modeled marker/mechanism proves it | +| L09 | Delayed work can replay withdrawn routes or old revisions | Confirmed; generation/revision guard plus current-input read at execution; E15 | +| L10 | Export grouping keyed by policy name omits peer-dependent matches and changed semantics | Confirmed; compiled fingerprint and declared peer/context projection; E18 | +| L11 | A code-owned access adapter could still read global state or silently turn failures into denial | Confirmed contract gap; minimal explicit local inputs, guarded access tests, transactional invariant failure (§4) | +| L12 | New raw-RIB fields necessarily require a separate event system | Rejected; current StateDelta iterates DeviceState fields; verify coverage and reuse it | + +Result: revision 3 incorporates L01–L11 and adds E13–E18. These are implementation requirements and design traces; no new daemon/runtime feature was executed. + +## Pass 3 — maintainability, minimality and implementation readiness + +Input: revision 3, SHA-256 `d1a774b2eee190b069572a52fb7e012fe162edb04c2b9bb77b845bedefed1530`. Review challenged every proposed abstraction: does it avoid duplication, preserve needed expressions and give an implementer an unambiguous owner/contract? + +| ID | Finding / challenge | Disposition and refinement | +|---|---|---| +| M01 | Rejecting every ordered prefix list makes the initial FRR subset needlessly weak; Boolean expansion can be quadratic | Confirmed; one domain-specific first-match PrefixListPermits predicate, E19. No general calls/jumps added | +| M02 | “Typed actions at legal hooks” was underspecified and could lead to duplicate capability checks | Confirmed; one explicit read/write capability table used for validation/documentation (§4) | +| M03 | A second universal route DTO or dynamic dictionary would duplicate BGP state and weaken types | Rejected alternative; retain owner-specific immutable route types and a narrow trusted access protocol; no arbitrary set(field, Any) | +| M04 | Per-node layering could accidentally execute every inherited binding, applying transforms twice | Confirmed; exactly one resolved binding per scoped point; composition only through its explicit chain (§6) | +| M05 | Semantic fingerprints could depend on Python hash, set order or comments | Confirmed; canonical tagged serialization, semantic order preservation and fixed language revision (§4) | +| M06 | Named policies were pinned but example named sets were not, allowing later set changes to alter old revisions | Confirmed; RouteSetRef, transitive digest pins, self-contained materialized configuration (§4/§8) | +| M07 | Matchers over community/multipath values could choose accidental quantifiers; AS-path segment treatment was vague | Confirmed; explicit quantified collection behavior and native AS-path definitions (§4) | +| M08 | A separate trace interpreter could disagree with actual decisions or retain unbounded state | Confirmed risk; same evaluator with bounded optional trace sink and two explanation views (§6) | +| M09 | Fine-grained route caches/global interning look attractive but complicate correctness before a workload justifies them | Confirmed decision to omit initially; compiled-binding reuse and local full reevaluation are the baseline (§6) | +| M10 | The plan's integration prose did not say which Claude-plan sections must change or who owns shared files | Confirmed; explicit section-level deltas and owner contract before parallel implementation (§9) | +| M11 | “Performance tests” without workload or attribution are not actionable | Confirmed; scale-passport sizes, baseline/edit/churn cases, separate memory/evaluation/downstream costs and old/new/old measurement (§9) | +| M12 | Expanding to a full vendor VM would remove unsupported constructs from the plan | Rejected scope expansion; exact supported subsets are explicit, and native expressiveness is exercised by E01–E24 | + +Result: revision 4. Core control flow still has four outcomes, no loops/calls/jumps, no general-purpose external evaluator, no duplicate settings framework and no policy-result cache. Added specificity is concentrated in attachment contracts, state ownership and acceptance, where it prevents implementation forks. + +## Final completion audit + +The final artifact was reread against the objective, current worktree and captured protocol plan. The live Claude plan still matched the captured snapshot at the closing comparison. NetSim HEAD remained `0e5b299225372f394dd3505a09cd92127b33dfa7`; the only NetSim working-tree additions were the two documents in this directory. + +| Required outcome | Authoritative evidence inspected | Assessment | +|---|---|---| +| Critically review policy languages | Plan §2 compares FRR/EOS, Junos, IOS XR, BIRD, IETF/OpenConfig, RPSL, CEL, Rego, callbacks and native records, with primary sources and explicit tradeoffs | Complete for the design decision; no claim of a complete vendor feature survey | +| Propose the best approach for NetSim | §§1/3/4/7 choose a small native core, model-layer ownership, exact adapters and explicit unsupported features | Complete recommendation with reasons and limits | +| Write an actionable plan | §9 provides P0–P6, dependencies, code owners, source callsites, required protocol-plan changes and acceptance gates | Complete planning artifact | +| Run deep reviews and refine | Three captured input revisions, S01–S12/L01–L12/M01–M12, and final revision 4 | Complete: 36 findings dispositioned, including rejected alternatives and a revised earlier decision | +| Maintainable implementation | Existing settings/state machinery reused; typed capability table; one evaluator; no universal route DTO, dynamic registry or second ACK system | Covered by explicit contracts and review findings | +| Streamlined implementation | Four control outcomes; bounded typed predicates; no calls/jumps/general regexes/external interpreter/result cache initially | Explicitly bounded core; restrictions do not remove required device hooks | +| Expressive policy behavior | E01–E24 cover common route transformations, composition, ordered prefix filters and adversarial lifecycle cases | Required behavior specified; proposed implementation tests, not passing test results | +| Per-node configuration and isolation | §6 binding identity/overrides/revision pins, E07/E18/E23; current two-node profile probe rerun | Requirement and current prerequisite are explicit | +| RIB/FIB/resolution plus ACK/inactive interactions | Separate hooks in §5, retained submissions, actual versus desired FIB, gate causality and P4 acceptance | Covered without equating absence, exclusion, failure and discard | + +Verification: [document verification](/Users/networmix/ws/RFCs/evidence/policy-plan-review/document-verification.json) checked local links, contiguous E01–E24/P0–P6 IDs, unique review IDs and whitespace. Source snapshots/downloads are included in the RFC library integrity seal. These checks support artifact integrity and traceability, not semantic correctness of code that has not been written. + +Remaining work is implementation P0–P6 and their specified tests, benchmarks and CI. Exact FRR pre-expiry edit behavior and the conflicting EOS FIB documentation require the named compatibility fixtures before those behaviors receive a vendor conformance label. They do not prevent adopting the fully specified native design. No production source, live Claude plan or agent session was modified by this planning work. diff --git a/docs/design/protocols.md b/docs/design/protocols.md new file mode 100644 index 0000000..7830f71 --- /dev/null +++ b/docs/design/protocols.md @@ -0,0 +1,514 @@ +# Protocol foundation contracts + +This page states what the foundation for Open/R and BGP must guarantee. It +holds the common design decisions, the minimum contracts for the next slices +(F0 and F5) and the requirements that F1a, C6, S1a and S1b must meet. Later +slices keep their requirements in the archived design and are listed under +[Pending contracts](#pending-contracts). Nothing here is an available API: the +records, modules and names below are planned. Order and state are in +[docs/plan.md](../plan.md); how this page relates to the code and the archive +is in [the design index](README.md). + +Two kinds of statement appear here: + +- **Requirements** are what the implementation must do. Each slice's + contract is complete only when its requirements are written as exact rules + with their tests. +- **Vendor references** cite Open/R at commit `81dae4c` or FRR at master + `f1d7bec` (10.8.0-dev; `10.5.4:` marks the SONiC release) by file and line. + They record a source reading made for the archived design and have not been + reproduced. A vendor behaviour becomes a guarantee only with a reproducer or + a pinned fixture. + +## Common decisions + +These hold for every slice below and in the archived design. + +- **References.** Open/R is followed module for module at `81dae4c`, including + upstream facts that contradict common assumptions (no domain name; + shortest-path ECMP only; a fresh node boots soft-drained; the default metric + is RTT-based; area import policy is a stub). BGP follows the RFCs first and + the modern FRR dialect where they leave room; `frr-10.5` (what SONiC builds) + is a settings dialect. Generic version differences are settings with + per-dialect defaults, never a version check in code, and vendor bugs are not + emulated. FRR is GPL-2.0+, so its behaviour is reimplemented clean-room from + RFCs, documentation and black-box runs; no code is copied. Anything + NetLab-Sim adds beyond upstream Open/R is labelled as an extension and is off + by default. +- **Differential oracle.** An optional lane runs pinned FRR images and compares + converged state only; it is opt-in and outside the standard checks. Its + goldens are committed as data generated by NetLab-Sim, and FRR test data is + read from a pinned FRR checkout at run time, never vendored. +- **No privileged path.** Protocols are agents. They see only their agent + context and act only through their output. They meet only through the RIB + (route submission and the RIB feed), next-hop tracking (NHT) and views + published on the same device. No agent reads another agent's state. +- **Structure mirrors the real implementation.** One package per protocol and + one submodule per real module. A submodule is a pure function over frozen + records and persistent maps, receives only the declared projection of the + context it needs, and exchanges explicit queues with the others. A run drains + the queues in a fixed priority order with a bound that raises when exceeded. + Edges that wait for programming in reality wait for the owner notification + of installation. Per-key deadlines go through one armed timer per module. + Each module draws randomness from its own named stream. +- **Replaceable parts are named code, not an interpreter.** Decision flavors + (`frr`, `rfc4271`, and classes a scenario adds at deployment) implement + `compare`, `select` and `multipath` over narrow immutable inputs and are + selected by a per-node setting. Callables never live in state or + configuration. Every built-in rule declares its references (an RFC section, + a vendor function or reason, or a document URL), and a test rejects a + missing reference. +- **Domains are configuration.** A device runs 0..k IGP instances with + disjoint interface sets (more than one instance is an extension, off by + default) and at most one BGP instance. VRFs are deferred. +- **Cost proportional to change** is a performance exit criterion for every + protocol slice: a one-prefix change costs the same at ten times the table + size within a stated factor, and an idle run admits O(1) changed records and + does no SPF, FIB or placement work. Large tables are persistent maps or + packed tables, never tuples or frozensets. No unrelated-table scan for a + local edit; affected closure, bulk configuration, SPF work and amortized + compaction are budgeted explicitly. No Python object per route, path or FIB + entry outside API boundaries, with one stated exception: the common + submitted and admitted RIB at X may stay object-based if the memory ceiling + is revised upward and stated ([S1b](#s1b-packedtable-and-its-index), + Memory). No per-run tree write for identity outputs. +- **Timing.** Defaults are faithful to the reference profiles; a `lab` preset + shortens timers but keeps the order of events. Protocol presets require a + positive agent run delay, so feedback between agents on one device cannot run + away within one timestamp. +- **Next hops follow FRR and zebra.** Single-hop eBGP registers NHT with + `connected_only`, waits for that answer before connecting, and installs + non-recursively. iBGP, confederation, multihop and GTSM eBGP install + recursively through the resolver; BGP over BGP is allowed. The decision + process uses the metric of the installed resolving route. +- **Keys are ready for add-path and VRFs**: paths keyed `(peer, path_id)`, + destinations `(vrf, afi, safi, prefix)`, with `path_id = 0` and the default + VRF until those features land. +- **No analytical BGP in this phase.** Import policy, decision and export stay + pure, so a later timeless solver can reuse them. +- **Configuration is tree state; commands are exec-only.** Agent configuration + changes in place within a generation and records a configuration cause; + reset and fresh runtimes keep the latest configuration; plugins read + configuration only from their context. Persistent agent state survives reset + unless purged. Commands (clear, soft reset, route refresh, drain) are + validated synchronously when submitted, report run-time errors in a view and + are dropped and counted when their generation was reset first. +- **Restart semantics are explicit**: a purging reset models a plain daemon + restart (zebra removes the client's routes); a retaining reset models Open/R's + route retention until first sync and, later, BGP graceful restart. +- **Routing policy** is one typed engine in the model layer, shared by BGP, + Open/R and the device hooks: [routing-policy](routing-policy.md). +- **Forwarding installation** is an asynchronous contract: desired and + installed actions are separate, completions are generation-guarded, and FIB + acknowledgement gating and inactive-route export are per-node behaviours. + +## F0: contract additions + +F0 adds records and fields only. Every addition has a default, so existing +agents, scenarios and results are unchanged. Its exit is a contract test for +every new record and rule, the existing suites unchanged, and the locality +rules extended to every new accessor. + +**Freeze rule.** Records with no open obligation can land first. The listener, +recursive-resolution, programming, sync and packed-table records are frozen +only after the obligations of [S1a and S1b](#s1a-and-s1b-certificates-and-packed-tables), +[C6](#c6-zero-commit-publication), [F1a](#f1a-order-independent-resolver), +[programming, sync and feeds](#programming-sync-and-feeds), and routing +policy's [next-hop activation](routing-policy.md#next-hop-activation) and +[recursive next-hop keys](routing-policy.md#resolver-filtering) are expressed +in their fields. This page states those obligations; it does not fix field lists +or signatures. + +### What F0 adds + +- **Configuration, persistence, commands**: in-place agent configuration with + a configuration cause; persistent agent state in the agent node and the agent + output; a command record with its time, name, payload and target generation; + a simulation entry point to submit commands; plugin hooks that validate a + configuration and a command. Delivery is F2a. +- **Views**: agents publish immutable named views, stored in the agent node and + compared by identity. Another agent on the same device reads them; a + subscription can target one view key; the timeline records view changes; + tests, timeline series and study stability predicates read views only. +- **RIB feed** (the analogue of zebra redistribution). A watch names a family, + a set of source clients or all clients, and a kind. "All" excludes the + owner's own client, and naming the own client is rejected. + - Kind `selected` covers rows the resolver installed. It reports at the + commit of a processed FIB epoch: upserts are the contributing routes of the + changed entries, filtered to the watched sources, including installed + prefixes whose contributing rows changed content only (metric or tag); + removals are keys that left the set; nothing is reported while the FIB is + pending. + - Kind `all` covers every row of the watched sources whatever its selection, + at the RIB commit of those rows. + - A snapshot is delivered on registration and on every generation change. + Feed entries are control-class inbox entries, coalesced as + [programming, sync and feeds](#programming-sync-and-feeds) states. +- **Owner notifications**: for each changed own row, a `RowNotice` with the + native status (`INSTALLED` or `NOT_INSTALLED`) and reason (`SHADOWED`, + `UNRESOLVED`, `AMBIGUOUS_ACTION`, `LOOP_DETECTED`) plus the derived ZAPI code + (`INSTALLED`, `FAIL_INSTALL`, `BETTER_ADMIN_WON`, `REMOVED`). A metric-only + change counts. +- **NHT**: a mode per key, `rib` (today's answer) or `installed` (from the last + processed FIB, as zebra answers), and an exact-prefix mode for BGP `network` + import check (present exactly, resolved by a row that is not from BGP). The + answer carries the metric of the selected first-level resolving row next to + the strict cost. Notifications fire only on answer fields (`eligible`, + `via_prefix`, `via_source`, `via_metric`, `cost`, `legs`), not on + dependencies (`queries`, `interfaces`). +- **Change keys**: the NHT and row-notice change lists name the keys captured + by this receipt: the union since the generation's previous published run, + kept on rejection. Values are read from the snapshot; a key may have changed + back. The initialization run receives empty lists and reconciles from the + context. +- **Route views** separate the RIB-selected group at the input epoch (the + desired projection) from the actual installed forwarding at the processed + epoch. The desired row's installed projection may be absent while + programming is pending; the actual installed content stays visible to + forwarding and to installed-mode NHT until its replacement or deletion + succeeds (see [programming, sync and feeds](#programming-sync-and-feeds)). + Today one field reads the installed entry even while the row is pending. +- **Transport options**: wildcard listeners for a family scoped to an + interface set; a per-listener accept table keyed by scoped remote identity + with TTL and minimum TTL, refusing an unknown remote; TTL and minimum TTL on + session open and a later option change; the connection view reports the + concrete local endpoint. F3 implements them. +- **Lazy accessors** for own rows, one route, one row status and the RIB view, + O(1) over immutable records, built from the detached local context and never + from device state; no mutable cache. +- **Records F1 and F2 share**: resolver outcomes keyed by client, then row; + NHT registrations held per owner; recursive next-hop and group records; a + FIB group timeline event. +- **Packed-table records**: the record types whose obligations are under + [S1b](#s1b-packedtable-and-its-index); S1b implements them. +- **Settings root**: a settings field on the network state and labels on + devices, with the delta rules of [settings](#settings-delta-and-precedence). +- **Routing policy, programming and client profiles**: binding storage in + settings, the route submission helper, the programming request, result, + desired and installed action records and the `POLICY_EXCLUDED` reason, and + node-local client-profile settings with the regression below. + +### Per-node client profiles + +A client profile (for example a route source's administrative distance) is +registered network-wide today. Two devices that run the same client with the +same profile are accepted; if one device uses distance 20 and the other 200, +the second registration is rejected because the client "is registered with a +different profile" (`netlab_sim/model/network.py`). Every protocol setting must +be overridable per node, so F0 adds node-local client-profile settings with a +regression test that fails on today's code: the same client with distance 20 on +node A and 200 on node B. No change of client identity may be used to work +around it. + +### Programming, sync and feeds + +These obligations bind the programming, sync and feed records that F0 defines +and X implements. + +- **Route and group obligations.** Route requests and group requests have + separate identities, desired and installed generations, and explicit + dependency completion: a route becomes installed only when its group content + is installed, and the previously installed action survives until then. + Installed forwarding reads installed group content. An underlay change + programs the group once and advances its installed version at completion; + prefixes keep their group reference and are not re-requested. A group-only + update avoids per-prefix programming only while selection and eligibility are + unchanged; loss of eligibility triggers the budgeted re-election of affected + prefixes (F1c). Supersession is resolved once, at the group obligation; an + obsolete acknowledgement never satisfies a newer intent or releases a gate. + Reclamation counts desired, installed and in-flight references; snapshots + own their data. "Installed is absent while pending" is a projection of the + desired row and never erases a retained installed action; installed content + survives a pending or failed DELETE. Installed NHT and forwarding read + installed content and its matching provenance. Pending, error and success + are distinguishable from the first version: an explicit ERROR completion and + a deterministic failure-injection seam. +- **Sync operations.** An accepted sync carries owner, generation and batch + identity even when its RIB edit is the identity: an empty sync over an + already-empty RIB, or a sync with no RIB delta, returns the same RIB today. + (An empty full sync over an owner's existing rows withdraws them and is not + an identity edit.) The programming boundary tracks the batch's upserts and + deletions, including cleanup of retained installed routes, and emits one + immutable completion with per-operation success and failure, or a distinct + total failure. An empty batch completes successfully; failed removals are + never omitted. Partial success reaches Open/R with the failed members + scheduled for retry; total failure does not advance initial sync. Open/R + publishes an empty full sync (vendor reference: `Fib.cpp:386-401, 849-859`), + and its PrefixManager gate applies only to entries that carry next hops + (`PrefixManager.cpp:650-662`). These records belong to F0 and X, not to the + Open/R slices. +- **Route feed coalescing.** Feed messages identify their baseline and result + epochs. Coalescing composes per-key final upserts and removals relative to the + last captured baseline: disjoint edits are preserved and net no-ops cancel. + At most one uncaptured entry exists per watch, and it is never merged into a + captured or parked one. Registration and reset snapshots are distinct. Large + snapshots and change sets use bounded pages or immutable table-backed views + with declared byte limits and an explicit resync outcome; an entry count is + not a memory bound. +- **Listener identity.** The accept table is keyed by scoped remote identity: + family, address, and the ingress interface identity and generation where the + address is link-local. That scope flows through connection lookup, accept + options and collision identity. Listen ranges are a separate matching mode + for globally scoped addresses. + +### Settings delta and precedence + +- The settings root is one of the root fields a state delta enumerates and is + preserved by batches and forks; a settings-only update is never an empty + delta. F0 owns this root integration; resolution and the per-owner action + plan are F7a. +- Desired-layer edits and effective changes are tracked separately. An edit is + preserved even when the effective value is equal; only effective changes + schedule protocol work. Dumps serialize explicit values and inheritance + intent, not only differences from dialect defaults. Tests: an + equal-effective edit followed by a parent change; clear and unset; dump and + load; aborted batch; fork; reset. +- Preset precedence: an explicitly inherited user instance value (for example + 100) beats a preset's neighbor value (200); only absent explicit inheritance + falls back to presets. An explicit user profile (`datacenter`) beats a + preset's profile (`traditional`); without a user profile the preset supplies + it. Preset recursion and conflicting meta-key combinations are rejected. +- A configuration transaction resolves the complete candidate, diffs old and + new effective records once, and compiles one plan per owner: reset or + restart supersedes session work; recalculation, export and replay coalesce + against the new revision; work deferred to the next session keeps its + pending intent. Minimal provenance that explains each winning value ships + with F7a. + +## F5: protocol toolkit + +Shared pieces both protocols use, so neither builds its own: + +- **Selection**: the decision-flavor protocol (`compare`, `select`, + `multipath`) with its narrow path, destination and context views, and small + shared comparison helpers (AS_PATH length, MED comparability, router-id + substitution), each with references. No interpreter and no hold or release + slots. +- **SPF**: a pure ECMP Dijkstra over integer node ids returning distances, + first hops and predecessor sets. Today the oracle IGP + (`netlab_sim/model/igp.py`) and the reference agent + (`netlab_sim/agents/reference.py`) each have their own; the toolkit gives + protocols one. +- **Timers**: Open/R exponential backoff, async debounce, throttle, jitter + drawn from a module's own stream of the agent's PRNG, and a per-module + deadline structure that is one shared persistent priority structure with + stated costs (arm and cancel O(log n), due O(k log n), stale entries + bounded), never a tuple rebuilt per edit. +- **Route operations**: a RIB delta emitter keyed by changed-key sets. +- **Sizes**: modelled message sizes. +- **References**: a decorator that attaches an immutable reference tuple to rule + functions, and one for tests that cover a reference. +- **Source address**: the primary address of the egress interface toward a + destination, used by BGP when no update source is set. +- **Test harness**, one module for every protocol: the locality harness, + guarded projection proxies (an undeclared read fails), twin comparisons with + normalizers, and view-based assertions. + +Exit: unit tests per helper and harness self-tests, including a projection +proxy rejecting an undeclared attribute read. + +## F1a: order-independent resolver + +Today the recursive resolver's answer for a prefix can depend on unrelated +routes and on the order in which prefixes are visited: depth is measured as +stack height, and results cut short by the stack or by the depth limit are +memoized. F1a makes each prefix's answer a function of the RIB only. It is the +one foundation slice that fixes a wrong result. + +### Two separating examples + +Both are reproduced on the baseline commit; their expected answers follow from +the rules below. + +1. **A sibling cycle with an escape.** `eth1` is a numbered point-to-point + link whose neighbour is `10.9.0.1`. Route A, `10.1.0.0/16`, has two next + hops: out of `eth1` via `10.9.0.1`, and recursively via `10.2.0.1`. Route + B, `10.2.0.0/16`, has one next hop, recursively via `10.1.0.1`. A's + recursive leg leads through B back to A, so it closes a cycle and must fail + with `LOOP_DETECTED`; A keeps its direct leg. B's leg resolves through A's + direct leg, a finite witness that does not involve B. Expected: A and B + both install out of `eth1` via `10.9.0.1`, in every visiting order. Today, + resolving A first leaves B unresolved, and resolving B first installs + both. Without A's recursive leg both install today. +2. **Unrelated routes change a chain's answer.** On the same link, recursive + static routes Xi = `10.0.0.i/32` via `10.0.0.(i+1)` for i = 1..9, and X10 + = `10.0.0.10/32` out of `eth1` via `10.9.0.1`, with the default maximum recursion + depth of 8. X5..X10 depend only on X6..X10. Expected: their + answers are the same whether X1..X4 exist or not. Today they all resolve + without X1..X4, and X5..X8 become unresolved when X1..X4, which no other + route uses, are added. + +### Rules + +- Fingerprints come first: before the resolver changes, the fingerprint oracle + (`tests/test_fingerprints.py`) covers outcomes (status and reason), MAC, + encapsulation, adjacency family, NHT answers, selection and pending state, + with recursive, cyclic, cross-family (RFC 8950), `lpm_fallthrough`, + `resolve_via_default` and SR scenarios. Baselines are recorded and F1a's + intended differences listed. +- Depth is the length of the next-hop chain, kept per recursive entry, not + stack height. A depth-truncated result is never memoized. A prefix excludes + only itself from its own resolution. +- **Per-leg witnesses.** A leg resolves when it has a finite resolution witness + that does not support itself, with its candidate and exclusion context, + within the depth bound; depth is checked per witness, not by one number per + record. A leg whose only witnesses close a cycle fails with `LOOP_DETECTED`; + cyclic branches contribute no weight unless an explicitly permitted acyclic + escape exists. The route's other legs are unaffected, and weights are + normalized over the surviving legs. +- Lower-ranked and less-specific fallback obey the selection and + `lpm_fallthrough` rules; `all_or_nothing` is preserved; deduplication and + weight combination are deterministic. A cached physical result is reused only + where the exclusion context and the remaining depth cannot change the answer. +- The resolver's device context tests for an absent RIB by identity, not by + truthiness (which measures the table on every prefix). + +Tests fail on the old code and run under every enumeration order: closed +cycle; cycle plus a direct leg (example 1); cycle with two exits; unequal +branch depths; cycle plus a lower-ranked candidate; `all_or_nothing`; repeated +adjacency with unequal weights; blocked covering prefix and blocked +less-specific fallback; a chain with and without unrelated members (example +2); and a seeded fuzz of random RIBs in random prefix orders asserting identical +outcomes. + +## C6: zero-commit publication + +An agent run that changes nothing must publish nothing to the network tree, +while initialization, receipts and the PRNG stay exact. + +### Current behaviour + +An agent whose run returns the same state and no operations still replaces the +network root and adds a timeline record, because the per-run bookkeeping (`runs`, +`receipt`, `rng`) is stored in the agent node of the tree +(`netlab_sim/model/contracts.py`, `netlab_sim/runtime/agents.py`). This is +planned work, not a fault of the current API. + +### Requirements + +- **Publication seam.** A finalizer that runs no user code commits accepted + runtime successors at the root publication point, before root observers, and + performs the same transition on accepted batches with no root delta. A failed + outer commit publishes neither side and keeps the retry captures. Captured + prefixes are consumed before external callbacks. The pipeline distinguishes + prepare, abort, publish and external finalization. The after-run hook alone is + not the publication hook. Tests place an observer ahead of the runtime + observers and inject an outer commit failure. +- **Initialization stays in the tree.** The node's initialized flag remains a + one-time tree transition on a successful initialization run; only the per-run + bookkeeping (`runs`, `receipt`, `rng`) leaves the tree. Fresh runtimes keep + today's generation and reset semantics. +- **Runtime state.** Pre-run PRNG, counter and initialization state are captured + in the per-generation journal and published at the same success boundary as + the tree, including the no-delta case. Parked rejection and explicit retry + are preserved. Reset, removal and generation purge clear every runtime map. + Forking a network does not clone a running scheduler. +- **Receipts.** Current receipts come from an explicit simulation or agent + runtime API as freshly frozen values, plus bounded history; bounded run events + are emitted at runtime publication even without a root delta; an old snapshot + never acquires a newer receipt. +- **PRNG.** The S0 generator keeps its contract: named and versioned with + golden vectors, one counter per declared stream persisted across accepted + runs and advanced only when drawn, stream keys from a fixed encoding hashed + with BLAKE2b, every counter restored on a rejected run. The implementation is + `netlab_sim/model/prng.py`; its design and evidence are in the + [S0 W3 report](../history/s0-w3-prng.md). + +Gate: every Gate C test, plus PRNG use in a no-delta run, mixed batches, a +failed outer commit and a callback failure after publication. + +## S1a and S1b: certificates and packed tables + +### S1a: changed-key certificate for `PMap` + +- The persistent-map builder emits a certificate of changed and deleted keys, + bound to the exact base map identity and result. +- Certificates compose across staged edits. Delete-then-reinsert and equal but + different keys validate in full. An unrelated or rebuilt map falls back to full + validation. The certificate is consumed on admission and never retains a + parent chain. +- Generic immutability and semantic admission are distinct. Certificates are + immutable evidence and never mutate a published map. An owner-local admission + transaction composes edits back to the admitted base. A missing or mismatched + certificate means full validation. The same lifecycle covers set, remove, + update, build and the packed builder. +- The shard count stays 256 until a measured workload justifies another size. + +### S1b: `PackedTable` and its index + +- **Storage.** A persistent vector of fixed-width rows in immutable byte chunks, + with a row layout per table kind. Admission treats the bytes as leaves, with + no walk over rows. No published chunk is ever mutated; old snapshots keep old + chunks. +- **Identity across compaction.** Positions are private to one table generation. + External references are logical keys or `(table_generation, slot, + slot_generation)` checked on dereference; a bare position never crosses a + publication boundary (queued requests, dirty sets, back-indexes and + completions carry logical identity or their snapshot). +- **First version is append-only**: no slot reuse or compaction within a live + generation, an explicit memory limit, and rebuilding at a generation boundary + as one owner transaction that publishes the table, every index, the pools and + the allocator together with an explicit old-to-new remap. A rebuild is a + storage change, never a route withdrawal. Bounded churn is not claimed for + this version. +- **Builder.** Private and copy-on-write per chunk. Building freezes touched + chunks and emits a change certificate of logical adds, updates and deletes, + bound to the exact base and result generations and naming its coordinate + system and any remap. Changed rows come from the builder's log, not from + diffing chunks. The builder validates row schema, bounds and references. + Attribute and peer ids are 32-bit and sequence numbers 64-bit; overflow is a + validation error at the builder. +- **Index.** Small and full-table indexes implement the same exact lookup, LPM, + exclusion and last-write-wins semantics. No packed sorted column is mandated. + Bounded devices keep persistent per-length maps. Full-table devices adopt + bounded sorted blocks of `(network_key, row_reference)` entries behind a + persistent directory, with a defined 128-bit IPv6 ordering and family + separation, only after their insertion, withdrawal, replacement, LPM and + publication costs pass a benchmark under held snapshots. Numeric prefix ids + are not address-order keys. A full per-length rebuild is a budgeted bulk + operation, never a one-prefix edit. +- **Memory.** Per-row byte figures are hypotheses for named layouts. Reports + state current, builder-peak, rebuild-peak and retained-history memory + separately. The common submitted and admitted RIB with its indexes belongs to + X; if it stays an object RIB, the memory ceiling is revised upward and stated. + BGP's Loc-RIB uses stable path slots plus a bounded per-destination + membership and order structure, never one monolithic compressed table. + Scale passports name links, interfaces and local rows per device, and the + protocol-churn passport fixes its workload (areas, degree, prefix + distribution, timers) before it can pass. + +Gate, before any protocol depends on it: held-snapshot tests across insert, +update, delete and rebuild; stale queued references; a remap failure rolling +back everything; admission; certificate and changed-row log correctness against +a brute-force logical diff; free-threaded readers on one snapshot; edit and +publication cost and retained-version memory measured. + +## Pending contracts + +These slices keep their requirements in the archived design until each is +reconciled here or on its own page before implementation starts. The sections +named are the extraction sources; later sections of that document override +earlier ones where they conflict, and the requirements above already apply. +Package and module names in the archive are proposals, written with the old +project name. + +| Slice | Requirements to reconcile | Archived sources | +|---|---|---| +| F2a | Delivery of configuration, commands, views and change keys | [F2](archive/protocols-r8.md#f2-agent-runtime-at-scale), [roadmap](archive/protocols-r8.md#123-foundation-slices) | +| F3 | Wildcard listeners and precedence, accept tables, hop limit from the path trace, GTSM (RFC 5082) | [F3](archive/protocols-r8.md#f3-transport-for-bgp), [listener identity](archive/protocols-r8.md#156-p1s-on-programming-sync-and-sessions) | +| F4 | Domain-scoped oracle IGP; one spanning domain reproduces today's fingerprints | [F4](archive/protocols-r8.md#f4-domain-scoped-oracle-igp) | +| F7a, F7b | Settings schema, layers and precedence, one write path, dialects and presets, FRR inheritance, the FRR 10.5 versus modern differences; F7a starts small: generated-documentation enforcement, the key-field bijection check and broad selector syntax wait for F7b or later | [settings](archive/protocols-r8.md#2a-settings-per-node-overrides-dialects-and-presets), [FRR versions](archive/protocols-r8.md#2b-frr-versions-105-versus-modern-and-what-becomes-a-setting), [revision 7](archive/protocols-r8.md#155-p1s-on-settings-and-configuration), [streamlining](archive/protocols-r8.md#158-streamlining-revised-findings-21-22) | +| X (F1b, F2b, F8) | Two-level next-hop groups, persistent FIB storage, timeline group events, one RIB submission boundary, prefix watch index; RIB feed and NHT installed mode; programming contract and acknowledgement gating; the obligations above | [F1](archive/protocols-r8.md#f1-resolver-deterministic-semantics-persistent-fib-then-incremental), [F2](archive/protocols-r8.md#f2-agent-runtime-at-scale), [F8](archive/protocols-r8.md#f8-forwarding-programming-contract-d14-section-513), [installation gating](archive/protocols-r8.md#513-fib-programming-acknowledgement-gating-and-advertising-inactive-routes), [group versions](archive/protocols-r8.md#156-p1s-on-programming-sync-and-sessions) | +| F1c | Dirty sets, sorted processing, measured full-rebuild fallback, the full resolver as oracle; design gate first | [F1](archive/protocols-r8.md#f1-resolver-deterministic-semantics-persistent-fib-then-incremental) | +| F2c | Lazy context, per-client identity notifications, shard-identity admission and its guard tests | [F2](archive/protocols-r8.md#f2-agent-runtime-at-scale) | +| A1 | Address-LPM and exact-prefix queries; enumerable materialized state; absent, default and aggregate answers; domain identity; cache lifetime; old-snapshot behaviour; never a silent replacement of materialized RIB semantics | [query contract](archive/protocols-r8.md#153-p1s-on-runtime-and-state-semantics), [A1](archive/protocols-r8.md#168-body-cleanup-and-a1-finding-16) | +| OR1–OR6 | Open/R fidelity target, module mapping, areas, behaviour by module, performance design, acceptance | [Open/R](archive/protocols-r8.md#4-openr), [roadmap](archive/protocols-r8.md#124-openr-slices) | +| BG1–BG7 | Sessions, next hops and attributes, decision process and flavors, route reflection, confederations, pipeline timing, state and performance, acceptance scenarios, RFC 8212 | [BGP](archive/protocols-r8.md#5-bgp), [corrections](archive/protocols-r8.md#157-bgp-corrections-findings-18-19-20), [RFC 8212](archive/protocols-r8.md#167-rfc-8212-finding-15), [roadmap](archive/protocols-r8.md#125-bgp-slices) | +| IX | Integration scenarios and their configuration sketches | [IX](archive/protocols-r8.md#6-integration-scenarios-ix) | +| All | Verification approach, scale passports, risks and open questions | [verification](archive/protocols-r8.md#7-verification), [scale review](archive/protocols-r8.md#14-scale-review-experiments-prototypes-and-the-streamlined-plan-revision-6), [passports](archive/protocols-r8.md#152-p1s-on-measurements-and-passports), [risks](archive/protocols-r8.md#9-risks-and-open-questions) | + +Measurements and test counts in the archive are observations of their own +revision and workload; they say nothing about the current code until measured +again under the repository's benchmark protocol. Only figures the archive names +as targets or hypotheses (exit budgets, passport ceilings, per-row layout +sizes) have a role in future work, and only as such. diff --git a/docs/design/routing-policy.md b/docs/design/routing-policy.md new file mode 100644 index 0000000..7bd6974 --- /dev/null +++ b/docs/design/routing-policy.md @@ -0,0 +1,636 @@ +# Routing policy contract + +NetLab-Sim will get one small, typed route-policy language and one pure +evaluator, shared by BGP, Open/R route injection and the device's RIB, FIB and +resolver hooks. This page is the contract for that language, its attachment +points, per-node configuration, lifecycle and acceptance examples. It is a +planned feature: none of the records or names below exist in the package yet. +Order and state are in [docs/plan.md](../plan.md), and the slices that deliver +this page are listed under [Slices](#slices). + +The page carries the reviewed routing-policy design (revision 4 of 27 September +2026, [archived](archive/routing-policy-r4.md) with its +[review record](archive/routing-policy-reviews.md)) and the later amendments of +the protocol design +([§5.8](archive/protocols-r8.md#58-routing-policy-one-shared-engine-integrates-docsplansrouting-policymd), +[§15.4](archive/protocols-r8.md#154-p1s-on-the-resolver-and-policy-hooks), +[§16.4–16.7](archive/protocols-r8.md#164-resolver-and-policy-hooks-findings-8-9)): +activation per original candidate next hop, separate forwarding and NHT +resolution scopes, group and acknowledgement causality, settings storage and +precedence, regular-expression lowering, RFC 8212, and no JSON codec in the +first delivery. Where the two disagreed, the amendment is what this page says. + +The existing SR policies (`SrPolicy`, `Device.policy_client()`) are +segment-routing policies. They solve a different problem, keep their meaning +and are not route policies. + +## Decision + +- A small, typed **NetLab-Sim route-policy language**: Boolean matches, + ordered actions, ordered terms and chains, explicit continuation and terminal + allow or deny. It is authored as immutable Python records. +- One pure evaluator shared by routing protocols, RIB admission, FIB + programming and resolver filtering. Each attachment point's effects stay in + the module that owns them. +- RFC 9067 and OpenConfig guide the structure and are possible later + interchange formats. An explicitly supported FRR route-map subset is + translated first. A vendor CLI, a YANG tree, Python callbacks, Rego or CEL are + not the execution language, and there is no universal vendor-language + virtual machine. Unsupported foreign semantics are rejected, never silently + approximated. +- Native semantics are a versioned contract. Vendor compatibility describes a + tested translation subset, not the name of an operating system. +- Every modelled behavioural setting and policy binding is overridable per + node. Shared definitions are immutable values; a policy update is never + network-wide by accident. Nodes with no new bindings keep today's behaviour. + +## Alternatives + +| Approach | Useful properties | Why it is not the execution contract | +|---|---|---| +| FRR and EOS route maps | Familiar, compact, directly relevant to SONiC and FRR | Similar syntax hides different continuation, matching and action-order rules; BGP-first ownership cannot serve model-layer hooks | +| Junos policy statements | Clear terms, chains, reusable sets, explicit flow control | Defaults depend on context; policy expressions and route-filter lookup need more semantics | +| IOS XR RPL | Readable conditionals, parameters, reusable policies | `pass` continues and a later drop can override it; modification can confer acceptance; a terminal-permit translation is wrong | +| BIRD filters | Typed route values, conditionals, functions | A router-specific programming language; its runtime and attributes add scope | +| IETF and OpenConfig models | Common vocabulary, schema validation, interchange | Do not erase device capability and attachment differences; a management tree is a poor hot-path form | +| RPSL and RPSLng | Inter-AS intent, registry-based filter generation | No local programming, acknowledgement or resolution lifecycle | +| CEL | Typed, side-effect-free expressions | Route effects, chaining, attachment validation and lifecycle would still be needed; a second evaluator adds little | +| OPA and Rego | Declarative policy over structured data | No ordered route transformations; an external engine does not remove the routing design | +| Python callables | Immediate flexibility | Hidden state, nondeterminism, unserializable closures and unknown dependencies conflict with tree admission and replay | +| Small native records | Explicit semantics, direct validation, stable serialization, bounded evaluation | NetLab-Sim maintains a small language contract with disciplined scope | + +Primary sources: [RFC 9067](https://www.rfc-editor.org/rfc/rfc9067.html) +§§4–6, +[OpenConfig routing policy](https://openconfig.net/projects/models/schemadocs/yangdoc/openconfig-routing-policy.html), +[Junos policy chains](https://www.juniper.net/documentation/us/en/software/junos/routing-policy/topics/concept/policy-routing-policies-chain-evaluation-method.html), +[IOS XR RPL](https://www.cisco.com/c/en/us/td/docs/iosxr/cisco8000/routing/configuration-guide/routing-config-cisco8000/routing-policy-language-wrapper/routing-policy-language-structure.html), +[BIRD 3.1 filters](https://bird.nic.cz/doc/bird-3.1.0.html#filters), +[CEL](https://github.com/cel-expr/cel-go), +[Rego](https://www.openpolicyagent.org/docs/policy-language), +[RPSL](https://www.rfc-editor.org/rfc/rfc2622.html); FRR route maps as +documented in `doc/user/routemap.rst` at FRR master `f1d7bec`. + +RFC 9067 exposes original-versus-modified attribute matching as +implementation-specific read-only state; a configurable binding here is a +simulator feature, not a writable reading of that leaf. FRR distinguishes +match, no-match and NOOP and applies different call and exit rules +(`route_map_apply_ext()`); lowering every native FRR feature into a plain +first-match list would lose behaviour, so the first adapter rejects those +constructs. + +## Code it builds on + +| Current code | Consequence | +|---|---| +| `Route`, `RibState`, `rib_apply` in [`routing.py`](../../netlab_sim/model/routing.py) | Route identity is `(prefix, source, distinguisher)` within a family; local route attributes differ from BGP wire attributes. Policy preserves identity and provenance | +| `ResolutionPolicy` in [`routing.py`](../../netlab_sim/model/routing.py) | Already owns default-route, fall-through, depth and ECMP behaviour; it is extended rather than duplicated | +| `DeviceConfig` and immutable records in [`state.py`](../../netlab_sim/model/state.py) | Published configuration holds admitted frozen records, tuples and persistent maps; no functions or compiled opaque objects | +| `AgentConfig.params` in [`contracts.py`](../../netlab_sim/model/contracts.py) | Holds immutable protocol settings; policy types are shared without importing BGP into model code | +| `Network.add_agent` in [`network.py`](../../netlab_sim/model/network.py), `RibClient` in [`entities.py`](../../netlab_sim/model/entities.py), agent route submission in [`agents.py`](../../netlab_sim/runtime/agents.py) | Client profiles are network-wide today; the node-local split covers every consumer, not only agent registration ([per-node profiles](protocols.md#per-node-client-profiles)) | +| NHT in [`nht.py`](../../netlab_sim/model/nht.py), FIB derivation in [`derive.py`](../../netlab_sim/model/derive.py) | Resolver policy is device-wide today; client-scoped filters need explicit identity and invalidation | + +The planned package is `netlab_sim.model.route_policy`, with public entry +points prefixed `RoutePolicy`; short syntax names such as `Term` stay scoped to +it. Its first modules hold immutable syntax and results, validation and +reference resolution, and pure execution. Protocol-specific access code and +scheduling stay with BGP and Open/R; the resolver owns resolution effects; +vendor adapters live with the other adapters, not in the evaluator. There is no +new generic settings framework (bindings use the [settings system](protocols.md#settings-delta-and-precedence)) +and no second BGP evaluator: BGP keeps only its field and action adapter, +attachment wiring and refresh scheduling. + +## Language + +These are intended record shapes, not callable APIs: + +```python +RoutePolicy(name, terms: tuple[Term, ...]) +Term(name, when: Expr, actions: tuple[Action, ...], flow: Flow) +RoutePolicyBinding(point, policies: tuple[RoutePolicyRef, ...], + fallback: ALLOW | DENY, match_input: ORIGINAL | WORKING) +RoutePolicyRef(name, revision) +RouteSetRef(name, revision) +PolicyResult(verdict: ALLOW | DENY, route, reason) +``` + +### Evaluation + +- `Flow` has exactly four values: `ACCEPT`, `REJECT`, `NEXT_TERM`, + `NEXT_POLICY`. A term's flow is explicit, and a chain's fallback is explicit + in resolved configuration. An absent attachment is distinct from an attached + empty chain, so adapters can enforce rules such as RFC 8212. +- Evaluation starts with the immutable original input and a working route + equal to it. Policies and terms run in written order. A false condition + advances to the next term. A true condition runs its ordered actions, then its + flow. `NEXT_TERM` continues this policy; `NEXT_POLICY` skips its remaining + terms; the end of a policy continues the chain. `ACCEPT` and `REJECT` end the + whole chain; the end of the chain applies the fallback. Acceptance returns the + working route; rejection returns no publishable route and discards tentative + edits. A later policy cannot undo a terminal rejection. +- A `REJECT` term with actions is a compilation error. A terminal acceptance + applies to the whole binding, not only the named policy. +- `ORIGINAL` means the input to **this attachment**, before any term of its + chain, not necessarily the original BGP UPDATE; earlier attachment results + are part of it. Reapplication uses that attachment's retained pre-policy + input. +- Actions always read the current working route. Conditions read the original + or the working route by the binding's `match_input`, fixed for the whole + invocation; the native default is `WORKING`. Context facts such as peer + identity cannot be changed by actions; a profile change goes through node + configuration. +- There are no loops, arbitrary jumps, recursive calls, runtime name + resolution, general expressions in actions or user functions. Reuse comes + from named sets, ordered chains and construction-time helpers that produce + immutable records; helpers are scenario-authoring code, never serialized + callbacks. The language grows only when the examples below need it. + +### Matches + +- Typed leaves composed with `AllOf`, `AnyOf` and `Not`. Empty conjunctions and + disjunctions are rejected; `Always` and `Never` are explicit. +- Leaves cover prefix membership with exact, minimum and maximum length, + address family, source protocol or client, origin kind, peer and role, tag, + next-hop address and interface, community membership and scalar comparison. + BGP adds AS-path contains, first, last and length, origin and local + attributes. +- Named-set predicates carry a `RouteSetRef`; definitions and the sets they + reference are pinned. A per-node set change builds a new revision and + reference through the authoring helpers instead of mutating a shared name. +- Prefix sets are unordered unions of typed prefix ranges. + `PrefixListPermits(ref)` is a separate predicate over ordered + `(range, permit | deny)` entries with first-match result and false at + exhaustion. It is a domain predicate, not a nested policy call, and claims no + Junos longest-match semantics. IPv4 and IPv6 never match across families. +- Community sets state ANY (nonempty intersection), ALL (every member of the + nonempty configured set is present) or NONE (disjoint); empty configured + match sets are rejected. Replace, add and delete are distinct actions. +- AS-path predicates work on the structured path and keep AS_SET and + confederation distinctions. First and last return false for an empty path or + an ambiguous endpoint set and examine the non-confederation external + endpoint. Native path length counts BGP hops: each sequence ASN, a nonempty + AS_SET once, confederation segments not at all (as the decision process + counts). Contains states `NON_CONFED` or `ALL_SEGMENTS`, with AS_SET members + included. +- BGP MED and the local RIB metric, LOCAL_PREF and the local distance, the + protocol NEXT_HOP and the resolved egress, and the source client and the peer + are separate fields. Fields are a closed tagged union, not string paths. A + field unavailable at an attachment is a compile error even in an unreachable + branch. +- A legal absent optional attribute is not zero: `Present(field)` tests + presence, ordinary comparisons on absence are false, and `Not` negates that + Boolean normally. Operators, families, literal types and ranges are checked at + compile time, including rejecting a Boolean where an integer is required. + Typed next hops keep family and interface scope; an IPv6 link-local address + without its scope is invalid. +- Next-hop predicates select proposed or resolved values and use explicit ANY + or nonempty ALL over that collection; there is no hidden choice of the first + ECMP leg. A FIB allow or deny applies to the complete selected action, not + per leg. +- Raw regular expressions are not in the native core. Typed AS predicates + cover the required examples; Python `re` is never run on an invented AS-path + rendering. + +### Actions and typed access + +- Initial actions: set local preference, MED, weight, origin or tag; add, + delete or replace communities; prepend an explicit AS sequence; set an + explicit next hop; set local RIB distance or metric where the hook permits. + BGP protocol next-hop rewriting and replacing a local row's complete + next-hop tuple are distinct actions. There is no public generic + `set(field, Any)`. A further programming attribute needs a concrete scenario + and attachment validation. +- `StandardCommunity` validates two unsigned 16-bit parts; large and extended + communities have their own constructors. LOCAL_PREF and MED are unsigned + 32-bit; origin is a closed enum; the local distance follows the model's + routing contract. Source, peer and origin stay separate fields; redistribution + never overwrites provenance. `RoutePolicyRef` is distinct from the SR + `PolicyRef`. +- A code-owned `PolicyAccess[T]` supplies typed reads and ordered action + application over the owner's own immutable route type, with three operations: + `read(route, field)`, `read_context(context, field)` and + `apply(route, ordered_actions)`. The owner passes it at execution; it is never + configured or stored in the tree. Inputs are explicit immutable local facts: + no network state, clocks, RNGs, external lookups or hidden NHT queries. Reads + are pure and total for their declared domain; a no-op application returns + its input by identity. Trusted implementations are still tested with guarded + inputs for locality and mutation. +- Compilation errors reject the entire candidate configuration. An access or + type failure at run time is an invariant violation that aborts the owner's + transaction without partial publication and keeps a diagnostic; it never + becomes a policy deny. Protocol limits on an accepted route (such as BGP + message size) are validated by the host after policy. +- Compilation resolves named sets once (no textual expansion), validates ranges + and attachment legality, records the fields each binding reads and writes, + including its named-set dependencies, computes a canonical semantic + fingerprint and produces immutable terms with resolved references. It generates no code; the + interpreter dispatches over a closed set of typed operations. Missing + references, duplicate definitions or term ids, reference cycles and malformed + trees fail compilation. Each term runs at most once per invocation; costs are + bounded by explicit input sizes, with no latency promise. +- The schema version owns semantics; two language versions never share a + fingerprint. Canonical serialization uses tagged unions with fixed field names + and canonical number and address forms, keeps chain, term, action and + prefix-list order, and sorts only mathematically unordered sets. Fingerprints + come from canonical bytes, never Python `hash()`, identity or labels. Resolved + references pin a content digest as well as a revision label; reusing a label + for different content is rejected. A materialized configuration is + self-contained for replay. + +### Capabilities + +One table, declared once, drives validation and documentation. + +| Point | Read domain | Writes in the first delivery | +|---|---|---| +| `BGP_IMPORT` | BGP candidate attributes, common prefix and source facts, ingress peer facts | BGP attributes, local weight and local tag; next hop only with typed family and scope | +| `BGP_EXPORT` | Selected BGP attributes, common facts, egress peer facts | Advertised BGP attributes and advertised next hop; no RIB distance, metric or identity change | +| `BGP_RIB_EXPORT` | Selected BGP attributes and the proposed local route | Local row metric and next hops; the selected BGP path is immutable | +| `REDISTRIBUTE` to BGP | Source route common facts and explicit destination-BGP defaults | Derived BGP attributes and local tag; source row and ownership immutable | +| `RIB_ADMISSION` | Local route prefix, family, source, origin, distance, metric, tag, proposed next hops | Local distance, metric, tag and next hops; no prefix, source, key or table rewrite | +| Candidate-next-hop activation filter | One original candidate next hop after activation (its activated type, address, interface and recursion state) plus permitted row facts | None; allow or deny for that next hop | +| `FIB_INSTALL` | Selected route common facts and resolved forwarding facts | None; allow or deny | +| `RESOLUTION_IMPORT` | Candidate common facts plus explicit resolver query facts | None; allow or deny | + +Redistribution into another protocol needs a capability row owned by that +protocol when its origination lands; a generic attachment name grants no BGP +setters to Open/R. The activation filter's point name is fixed when RP3's +contract is reconciled. + +## Attachment points + +These hooks are separate even where a vendor uses one word, such as import or +export, for several of them. + +| Point | Subject and owner | Meaning of rejection | FRR construct | +|---|---|---|---| +| `BGP_IMPORT` | Received candidate; BGP inbound | The candidate is filtered; a previously admitted incarnation is withdrawn | `neighbor X route-map M in` | +| `BGP_EXPORT` | Eligible path and target peer; BGP outbound | Advertisement suppressed or withdrawn for this peer only | `neighbor X route-map M out` | +| `BGP_RIB_EXPORT` | Selected BGP path plus proposed local route; BGP's RIB adapter | That submission is withdrawn or omitted; BGP's selection is unchanged | `table-map M`; its filtering behaviour needs a pinned fixture | +| `REDISTRIBUTE` | Source RIB route and target protocol; originator | The derived route is not originated | `redistribute P route-map M` | +| `RIB_ADMISSION` | Protocol-selected row submitted to the main RIB; model-owned admission | No admitted main-RIB row; protocol selection is unaffected | None | +| Activation filter | Each original candidate next hop during next-hop activation; resolver | That next hop and its resolved contribution are deactivated; a route left with no active next hop is ineligible and the next candidate is selected | `ip protocol P route-map M` | +| `FIB_INSTALL` | Selected and resolved action; model programming stage | Desired action is absent by policy (`POLICY_EXCLUDED`); an older installed action is removed; no fallback to another candidate, no leg pruning | None; native only | +| `RESOLUTION_IMPORT` | Potential underlay route and resolver context; resolver; scope `nht` or `forwarding` | The candidate is excluded from this query's inputs | `ip nht P route-map M` is scope `nht` with no fall-through; scope `forwarding` has no FRR equivalent | + +The FRR column is a vendor reference from a reading of FRR master `f1d7bec`; +each mapping is confirmed by a pinned fixture in the FRR lane before it is +called conformant. + +### BGP to RIB + +`BGP_RIB_EXPORT` is the FRR `table-map` boundary. It reads BGP attributes and +the proposed local row, but writes **only that row**: local metric and next +hops at first. `SetMed` is not an alias for setting the RIB metric. BGP keeps +the selected path and regenerates its submission after a policy edit; a denied +new submission deletes the previously submitted row. + +### RIB admission and retained submissions + +- `RIB_ADMISSION` applies to all supported submitting sources through common + local route fields. Every production RIB writer goes through one + model-owned submission helper: connected, local and SR-local routes in + `derive.py`, the oracle IGP in `igp.py`, `RibClient` in `entities.py`, batch + reconciliation in `network.py` and agent route operations in + `runtime/agents.py`. `rib_apply` stays the pure table primitive and never runs + policy for unrelated internal uses; an audit test fails on a new direct call + from production code. New route sources use the same boundary. +- Generated connected, local and SR-local routes stay trusted, unbound sources + in the first delivery; attaching an admission filter to them is rejected as + unsupported, not ignored. +- When a family's first admission binding becomes active, the unfiltered RIB + is kept as that family's immutable **submission view** and the admitted view + is derived from it. Add, delete and sync operations update the submission + view first, even for rejected routes; affected admitted rows are regenerated. + The existing RIB and client indexes are reused. With no bindings the single + view stays; on the last binding's removal the submitted table becomes the + admitted table. Unchanged row objects are shared; index overhead is measured. +- Trusted bypass writers update **both** views while retention is active, so + detaching the last policy cannot resurrect a deleted connected or SR route. + Batch collapse, equality and sync reconciliation compare submitted rows, so a + new version of a still-denied route is not lost as a no-op. A raw-only change + is a real state change (the state delta stays nonempty) but causes no FIB work + when the admitted output is unchanged. Owner or generation deletion purges + retained inputs as well as admitted rows, so a later allow cannot resurrect a + dead agent's route. +- Admission sees local distance, metric, tag, source, prefix and proposed next + hops. BGP communities and AS paths are not available there; use + `BGP_RIB_EXPORT` while BGP owns them. + +### Next-hop activation + +FRR's `ip protocol` is not a submission filter. The activation filter runs in +the resolver after next-hop activation, when the resolving interface is known, +once per **original** candidate next hop, with that next hop's activated type, +address, interface index and recursion state plus permitted row facts. A deny +deactivates that original next hop and its resolved contribution, never each +flattened child independently. System routes and family mismatches bypass it. +Positive and negative underlay dependencies are recorded, and the filter is +evaluated again when they change, without a new submission. When no next hop +survives, the route is ineligible and the next candidate is tried. Vendor +reference: `zebra_nhg.c:2403-2406, 2622-2628, 2693-2716, 2880-2921, +2909-2917`, `zebra_routemap.c:70-87`. Native submission admission, this filter +and whole-action FIB policy are separate capability rows. + +### Resolver filtering + +- The binding names its source view (selected RIB or installed forwarding + state), table, family, client and query mode. Filtering does not copy routes + between tables. +- Scope `nht` filters one client's next-hop tracking answers: the resolving + route is chosen by longest match, then its next hops are filtered for that + client; if none remains, the client is told filtered or unresolved and no less + specific route is tried (vendor reference: `zebra_rnh.c:515-576`). Forwarding + recursion is unaffected. Scope `forwarding` filters recursive resolution for + installation, and a denied more-specific falls through to a less specific one + only when the effective `lpm_fallthrough` rule allows it. +- The source view is chosen before evaluation. In installed mode a rejected or + uninstalled row never returns because policy allows it. Resolution does not + run the route's import policy again. Ordered table fallback and less-specific + fallback stay resolver settings. Policy cannot disable recursion-loop + detection, make a link-local address global or fabricate an installed next + hop. Rejected and absent candidates are dependencies too. +- Recursive next-hop keys include the interface identity and generation for + link-local addresses and every scope that changes resolution. Owner and + client projections (policy binding, query mode) are keyed separately and + applied per row, never to a shared group. A filtered NHT answer receives the + selected row's tag, metric, distance and source and its activated next hops + from the processed snapshot. +- The RIB is stored per family and its row key has no table field, so bindings + for a table or VRF the routing model does not support are rejected until it + does. + +### Installation and acknowledgements + +- Export acceptance cannot bypass protocol eligibility, AS, cluster or + originator loop protection, session or family capability, inactive-route + eligibility or a FIB gate. A next-hop export action changes the advertised + copy, not the locally installed next hop. Programming exclusion, capacity + failure, pending completion and later removal are separate states. +- When policy removes an installed route, desired and installed state stay + separate: rejection sets desired absence with reason `POLICY_EXCLUDED`; the + existing entry stays until the modelled DELETE completes; no ADD is pending + and a DELETE acknowledgement is not an install success; a failed deletion + shows the retained entry and its retry or failure state. FIB-policy + exclusion is never replaced by an implicit discard or a fallback candidate. +- A programming request's identity includes node and owner generation, table, + family, prefix and desired-action generation. A binding edit that changes + the desired action, its eligibility or its gate contract supersedes pending + requests; an unchanged action, eligibility and gate keeps its request after + an explicit equivalence check, so a new source label does not reprogram + identical forwarding. An old completion never satisfies the current gate. + Group obligations follow the + [programming contract](protocols.md#programming-sync-and-feeds); there is no + policy-specific acknowledgement mechanism. +- Native `FIB_INSTALL` affects installation only; export changes only through a + separately configured gate or eligibility policy. Published Arista EOS + documentation describes this hook in two conflicting ways; neither is the + native default, and an EOS adapter needs a pinned executable fixture first. + +### Out of scope for policy + +Route policy does not apply to packets, LSDB flooding or SPF topology events. +Packet policy-based routing, SR steering and route leaking keep their own +owners. Route leaking may later reuse the evaluator through a typed attachment +with its own provenance and loop rules; a policy record alone implements no +table or VPN feature. + +## Configuration and lifecycle + +- Definitions and named sets are immutable, revisioned values. A node's + effective binding references exact revisions; reusing a definition does not + subscribe nodes to a mutable name. One transaction may replace bindings on + several nodes; node edits stay local. Chains and definitions are replaced as + whole values, never merged by term name. +- Exactly one resolved binding exists per (node, attachment point, owner or + instance, family, supported table, peer where applicable). Group, instance + and peer layers choose or replace that binding through the settings schema; + the evaluator never runs inherited layers as extra policies. Node defaults, + individual bindings, match-input mode, reevaluation mode and delay, and the + replay-unavailable action all have node-local overrides. Scope restrictions + are type constraints (a FIB-only field need not accept a peer scope). +- Bindings are stored in the settings tree. The highest layer that writes an + exact (object, key) wins; absent explicit values fall back to presets, then + to the dialect's defaults (archived + [layers and precedence](archive/protocols-r8.md#2a3-layers-and-precedence), + reconciled with F7a). Preset precedence is in + [settings](protocols.md#settings-delta-and-precedence). Materialized + device-hook bindings live with the device configuration and protocol-hook + bindings with the agent configuration. Equal immutable definitions may be + shared by reference. BGP refresh status belongs to that agent's runtime + state; device-hook work to the existing per-device scheduling. No counters, + deadlines or executable functions enter a shared definition, and no + external mutable registry is authoritative. +- The complete candidate configuration is resolved and validated before it is + published; one bad field, missing reference or illegal action rejects the + transaction. Scenario defaults and node, instance, family and peer overrides + compile into concrete bindings once. Explanations keep provenance apart from semantic fingerprints. + Serialization records the language revision, the foreign source or profile + and any deviation from compatibility defaults. +- Three things are distinct: committed configuration, the definition active for + new inputs, and completion of reevaluation of older inputs. A validated + definition is active for new inputs at commit; existing routes are + reevaluated by `policy.reevaluation.mode` (`automatic` or `manual`) and + `policy.reevaluation.delay_s`, owned per node and attachment domain. The + native default is automatic with zero delay; the FRR dialects default to + automatic with 5 s, FRR's route-map delay timer. +- In automatic mode successive edits replace pending work with the newest + revision and restart the debounce deadline; in manual mode a visible pending + revision waits for a command or a new route event. Scheduled work captures the + owner generation and target revision, reads current inputs when it runs, and + cannot publish an obsolete result after a withdrawal or replacement. BGP + import replay, outbound duplicate suppression, RIB refresh, resolver + invalidation and FIB work stay separate owner operations. No global scan or + timer crosses nodes. +- **FRR delay timer.** FRR's zero delay disables automatic route updates while + references refresh, and changing a running timer to zero first runs the + pending update (vendor reference: `bgp_route_map_mark_update`, + `bgp_route_map_process_update` in `bgp_routemap.c`; + `bgp_set_route_map_delay_timer` in `bgp_vty.c`). The FRR adapter maps zero to + `manual` with an explicit transition action; zero never means native + immediate reevaluation. Native activation at commit is a declared + simplification until a pinned daemon fixture shows FRR's exact timing before + expiry. +- Import reevaluation needs complete pre-policy input or a peer refresh. With + retained input, replay once from it. With negotiated ROUTE-REFRESH + (RFC 2918), request it and process the returned routes under the active + revision. With neither, the state is `needs_session_reset`: new inputs use + the active definition, the old-route sweep stays incomplete, and the view + never calls this settled. The per-node setting + `bgp.policy.on_replay_unavailable` is `report` (native default) or + `reset_session`. Enhanced route refresh + ([RFC 7313](https://www.rfc-editor.org/rfc/rfc7313.html)) gives BoRR and EoRR + boundaries; a basic refresh alone does not, so completion is reported as + unverified unless another modelled mechanism proves it. A quiet interval is + not completion, and EoR is not EoRR. +- Device RIB, FIB and resolver hooks reevaluate completely from retained local + inputs at the scheduled node event; derived programming stays asynchronous. + A raw-only change or a same-result revision adoption is a real state change + but need not emit BGP advertisements or FIB writes. Reset, fork and + fresh-runtime reconstruction carry definitions, active revisions and pending + work through the owner lifecycle; replay neither loses nor duplicates a + pending edit. + +### Cost and observability + +- No policy-result cache and no global interning at first; compiled bindings + are reused within their effective configuration, results are not (import + policy depends on the prefix and context, not only the attributes). The + routes attached to an affected binding are reevaluated; node-local + conservative work is the correctness baseline. Finer route indexes come only + when measurements show they matter, with a full local reevaluation as their + oracle, and never for fields that cannot affect a decision. +- BGP update groups are keyed by the compiled export-binding fingerprint and + every declared peer and context fact the policy reads, never by policy name. + Groups split by the complete declared projection first; sharing is optimized + only where equal policy inputs imply equal outputs. Resolver query identity + includes its policy binding and scope. Without a result cache, correctness + still needs reevaluation after edits and underlay changes, including + negative dependencies. +- Shared definitions hold no mutable counters. Optional bounded traces record + input identity, attachment, matched terms, edits, final reason and downstream + eligibility; full traces are opt-in, and outcomes are identical with tracing + on or off. Published records pass immutability validation and the + free-threading ownership rules. No speedup is claimed before measurement. +- Explanations use the same evaluator with an optional trace sink. + `explain_effective_policy` reports the winning configuration sources, exact + revisions, fallback, capabilities and active or reevaluated status; + `explain_route_policy` adds the local input and the term and action path. + Output is limited and omissions counted. + +## Compatibility + +- The first adapter is a typed FRR adapter for first-match permit and deny + maps over the proven predicate and action subset, including ordered prefix + lists. It produces native records and a translation report (exact mappings, + rejected constructs, intended deviations) and claims no full CLI parsing. It + rejects call, continue and goto, NOOP-sensitive matches and unsupported filter + semantics unless an exact lowering is delivered and tested. It normalizes + FRR's documented grouping of matches and order of actions; text order is not + assumed. +- The FRR adapter lowers only anchored single-ASN AS-path patterns (`^A_`, + `_A$`, `_A_` and their equivalents) to typed predicates and rejects every + other regular expression with a report. A bounded regular-expression grammar + with source-backed semantics is a separate later slice. +- The native language does not grow to claim more import coverage. A new + foreign feature needs an operational use, pinned expected behaviour, a compact + exact representation and tests for tricky composition. IETF and OpenConfig + serialization is a later adapter over the supported subset; IETF condition + subroutines are rejected until their Boolean return and side effects can be + kept. FRR called-deny and Junos policy-expression results are not + substitutes for each other. +- Pinned, corrected FRR behaviour is the model; FRR 10.5 differences are + behaviour settings where useful, and historical bugs stay out of normal + settings. FRR master is a development commit, not a released standard. +- The canonical JSON policy codec is not part of the first delivery; nothing + consumes JSON policies yet, and the record schema already defines the + canonical form. + +## RFC 8212 + +With `bgp.ebgp_requires_policy` set (the FRR `traditional` profile), a missing +import binding discards received routes and a missing export binding +suppresses advertisements, per address family and direction. The eBGP session +still reaches Established and exchanges control messages; the configuration and +the OPEN are never rejected for a missing policy; views report the +missing-policy state; behaviour changes once a policy is attached. An attached +empty binding is distinct and follows its fallback. The rule is not enforced +under the `datacenter` profile or the `sonic` preset, and confederation peers +are exempt (a recorded FRR behaviour). Vendor reference: +`bgp_route.c:2692-2702, 6265-6275`. + +## Examples + +These are required design fixtures, not tests that exist or have run. + +| ID | Policy or scenario | Expected result | +|---|---|---| +| E01 | Customer import: allowed prefix AND acceptable path; set LOCAL_PREF 200, add customer community, accept; fallback deny | Only permitted customer routes admitted; shared received attributes unchanged | +| E02 | Tag in one policy and match that tag in the next | Working mode sees the new tag; original mode does not; actions still operate on working state | +| E03 | Reject bogons, then select outbound communities and AS prepend for one peer | Rejected routes never reach later transforms; another peer is unchanged | +| E04 | Protocol-selected BGP route denied by RIB admission | BGP candidate and export eligibility preserved; no admitted main-RIB row | +| E05 | FIB-deny a selected /24 with an installed covering /16 | /24 programming excluded; packets may follow the /16. This is not an explicit discard route | +| E06 | Same next hop, two node or client resolution filters: allow an IGP versus deny that IGP source | Distinct results and complete dependencies; no cache cross-talk | +| E07 | Two nodes share a policy revision; an unmatched prefix reaches fallback ALLOW on A and a node-overridden DENY on B | B's override changes no state or decision on A | +| E08 | Reapply an edited import policy after a prior prepend | The new policy runs once on the original received path, not the prepended value | +| E09 | FIB policy changes while an older programming request is in flight | An old success cannot release an advertisement for a new desired action | +| E10 | A term transforms, then `NEXT_POLICY`; the next policy rejects | No tentative edit reaches a route table or a peer | +| E11 | A filtered resolver /32 becomes permitted without a new BGP UPDATE | Relevant next-hop users are reevaluated, including prior negative results | +| E12 | Attach a BGP LOCAL_PREF action to FIB installation | Configuration rejected before any node state changes | +| E13 | Replace a denied submitted row, then allow it; detach the final policy after a connected-route deletion | The latest submitted row appears; the deleted generated route does not reappear | +| E14 | Change an installed route to FIB-denied, with a delayed or failed DELETE | The old forwarding stays observable until successful removal; no false install acknowledgement | +| E15 | Edit twice during debounce, then withdraw a route before the sweep | The latest revision runs on current inputs; stale scheduled work does not resurrect the route | +| E16 | Route-map timer set to zero with a pending timer, then edited again | The pinned FRR transition and disabled automatic processing are distinguished from native zero-delay mode | +| E17 | No retained pre-policy input and no refresh capability | The new-input revision is visible; existing-route completion stays reset-needed, never falsely settled | +| E18 | Same export policy name and revision but different peer facts read by the policy | Update-group sharing does not give both peers the first peer's result | +| E19 | A prefix list denies a /24 before permitting its covering /16 | The /24 is false and another /24 within the /16 is true; a union-of-permits translation is rejected | +| E20 | Same semantic policy with reordered unordered sets versus reordered actions | Set reordering keeps fingerprint and result; action reordering may change both | +| E21 | Missing MED compared with zero, then `Not` and `Present` compositions | Missing is not zero; negation and presence follow the native truth rules | +| E22 | A later REJECT versus a throwing access adapter after an earlier tentative transform | No partial publication; policy deny and invariant failure stay distinct | +| E23 | Node fork or reset with an active revision and a pending delayed sweep | Correct lifecycle reconstruction, one current sweep, no cross-fork state or stale generation | +| E24 | No policy bindings on an existing plain-IP scenario | Existing behaviour, fingerprints and the single-view RIB path preserved | + +An illustrative construction (proposed names, not current APIs): + +```python +customer = RoutePolicy('customer-in', terms=( + Term('authorized', + AllOf(PrefixIn(RouteSetRef('customer-prefixes', 'r1')), AsPathFirst(64512)), + actions=(SetLocalPref(200), AddCommunities((StandardCommunity(65000, 100),))), + flow=ACCEPT), +)) +binding = RoutePolicyBinding( + point=BGP_IMPORT, policies=(RoutePolicyRef('customer-in', 'r1'),), + fallback=DENY, match_input=WORKING) +``` + +## Slices + +The route-policy work is delivered inside the slices of +[docs/plan.md](../plan.md), which orders them. The device-side hooks are part of +the first complete delivery, not optional. + +| Part | Delivered in | Content | Exit evidence | +|---|---|---|---| +| P0 ownership | F0, F7a | Node-local client profiles, binding storage in settings, the retained-submission contract; the resolver and programming boundaries agreed with X | The two-node, same-client, different-profile regression passes; no client-identity workaround | +| P1 language | RP1 | Types, compiler, pure evaluator, typed prefix, community and AS predicates, immutable results, the capability table | E01–E03, E10, E12, E19–E22; flow, default and absence truth tables; fingerprint canonicalization; no mutable admission escape | +| P2 BGP | BG3 (import), BG4 (export and hooks) | BGP access adapter; import, export, RIB-export and origination hooks; refresh and edit scheduling | E04 at the BGP boundary, E08, E15–E18; peer isolation; policy with reflection and next-hop transforms; RFC 8212 | +| P3 RIB | RP3 | Admission for agents and existing sources, retained submissions, restore on policy edit; the activation filter; `RESOLUTION_IMPORT` scope `nht` | E04 at the device boundary, E13; distance and metric changes; deny → allow and allow → deny; generation purge; activation per original next hop, with fallback to the next candidate when none survives | +| P4 resolver and FIB | RP4, with X | `RESOLUTION_IMPORT` scope `forwarding`, `FIB_INSTALL` exclusion, reasoned state and acknowledgement causality on the programming contract | E05, E06, E09, E11, E14; mixed node profiles; inactive route × FIB gate × policy | +| P5 interfaces | BG7 | Typed FRR subset adapter with the `ip protocol` and `ip nht` mappings and AS-path lowering; explanations | E16 and E19 source comparisons; grouping and action-order fixtures; precise errors; unsupported constructs rejected | +| P6 acceptance | M5 | Integration, replay, fork and reset, free-threaded correctness, measurements | E07, E23, E24 and the whole matrix; scope documentation; the repository's checks; measured local work limits | + +P1 lands before BGP's import hook, BG4 and OR5; BGP consumes it instead of +building its own evaluator. P0 and P4 follow the resolver redesign instead of +filtering around today's combined selection and FIB function. The small type +and capability contract is frozen before parallel work starts. + +### Validation and measurement + +- Validation covers: the evaluator's truth and control-flow cases; each hook's + legal fields and actions; source retention and dynamic updates; fallback + versus discard; negative dependencies; revision changes with equal outputs; + stale acknowledgements; the same prefix across node, table and client scope; + fork, reset and replay; unchanged plain-IP fingerprints for unconfigured + nodes. A standalone policy result is not evidence for a hook's integrated + behaviour. +- Implementation follows the repository's [development rules](../../AGENTS.md): + a source-backed reproducer and a failing regression for defects, the standard + and free-threaded checks, and CI. +- Measurements compare old and new on the same quiet machine for the + unconfigured baseline, bulk initial policies, one-node policy edits and route + churn, reporting absolute time, memory, affected-route counts and + equivalence, on representative 1k- and 10k-node fabrics with the protocol + design's route and peer counts plus small adversarial topologies. Policy + evaluation, retained-input and index memory, binding compilation and + downstream routing work are reported separately. A one-node edit never + enumerates unrelated nodes; within its attachment, full reevaluation is the + initial baseline. A material regression needs investigation and a recorded + decision before release, not a universal percentage threshold. Vendor + differential fixtures compare normalized decisions and event boundaries + against exact builds; source reading alone never earns a conformance label. + +## Outside the first delivery + +Full vendor CLI coverage, a universal compiler, policy-driven packet +forwarding, route leaking and VPN features, general regular expressions, a YANG +management server and the canonical JSON codec are separate later work. diff --git a/docs/plan.md b/docs/plan.md new file mode 100644 index 0000000..5e880ba --- /dev/null +++ b/docs/plan.md @@ -0,0 +1,199 @@ +# Roadmap + +This page is for maintainers. It is the rolling product roadmap and the only +place that records the state of planned work: what each slice of work is, what +it needs first, whether its contract is ready and whether it is implemented. +The user documentation +([README](../README.md) and [reference](reference.md)) describes what the +code does today. The future behaviour that this plan builds is specified in +[docs/design](design/README.md); nothing on those pages is available until its +row here says so. + +## Goal + +NetLab-Sim gets a faithful Open/R IGP and a BGP that follows the RFCs first +and modern FRR where the RFCs leave room. Both run as protocol agents on the +existing agent contract, with no privileged path into the model. They share a +foundation: stable storage and change certificates for large tables, agent +runs that publish nothing when nothing changed, an order-independent recursive +resolver, a persistent FIB with an asynchronous programming contract, a +per-node settings system, one route-policy language with its device hooks, and +a protocol toolkit. Integration scenarios combine the IGP and BGP in the +deployment shapes the design names. + +## Boundaries + +- The plan adds behaviour. Scenarios a slice does not affect keep the + behaviour the [README](../README.md) and [reference](reference.md) describe. + A slice may change an existing answer only as its contract states: to + correct a demonstrated defect, with the reproducer reviewed (F1a's + [examples](design/protocols.md#two-separating-examples)), or as a documented + interface change with its intended differences reviewed (for example, X's + group timeline events). +- Contracts live in [docs/design](design/README.md). This page holds order and + state, not requirements. +- Functional and performance gates are separate for every protocol slice. +- Items listed under [Optional and deferred](#optional-and-deferred) are not + scheduled. + +## Status values + +**Contract** + +- `pending`: requirements exist but are not yet reconciled and reviewed into a + contract an implementation can start from. The link names the current + source: a design page, or an archived plan section listed on it. +- `ready`: the linked design page holds a reviewed contract for this slice. +- `—`: delivered before this roadmap; the current code and user documentation + define the behaviour. + +**Implementation** + +- `planned`: none of the slice is in the product. +- `partly implemented`: part of the slice is in the product; the evidence + names which part. +- `implemented`: the slice is in the product. + +## What this page records + +The rows state product facts: what the slice is, what it depends on, whether +its contract is ready and how much of it the product contains, with in-repo +witnesses (tests, history reports, user documentation). They do not record +how the work is run: an effort's plan revision, task acceptance, reviews and +CI results live in that effort's own records, outside the repository. Items +that landed before this roadmap keep their known landing commits as history. + +## Maintaining the roadmap + +- One row per slice. IDs never change and are never reused; a split gets new + IDs (S1 became S1a and S1b) and the old ID disappears from the table. +- Change a row in the same commit that changes the slice's scope or + dependencies, makes its contract ready, or lands part or all of its + implementation. Its evidence names the in-repo witnesses of that change; a + row never tries to name the commit that changes it. +- Status words live only in this table. Design pages state requirements, and + history reports and the archive keep the facts of their date. +- When this table and a design page disagree on order or dependencies, this + table wins; when they disagree on behaviour, the design page wins. + +The rows were assessed at commit `4514276` (31 October 2026); each later row +changes in the commit that changes the product. + +## Slices + +The protocol design's authoritative slice table, with its later amendments of +edges, applied. `X` is the integration boundary F1b + F2b + F8, which lands as +one change. + +| ID | Work | Needs | Contract | Implementation | Evidence | +|---|---|---|---|---|---| +| Gate A | Plain-IP simulator: state tree, interfaces, static and oracle routing, recursive FIB resolver, placement, studies, NetGraph and Core adapters; scale fixes T1–T7 | – | [—](../README.md#network-layer) | implemented | Landed `c2d658e`…`01591de`; reports [T1](history/t1-pmap-sharding.md), [T2](history/t2-scoped-batches.md), [T2B](history/t2b-convergence-boundaries.md), [T2C](history/t2c-committed-root-derivations.md), [T3](history/t3-rib-indexes.md), [T3B](history/t3b-rib-updates.md), [T4](history/t4-deadline-heap.md), [T6](history/t6-failure-studies.md), [T7](history/t7-scoped-l3.md) | +| Gate B | SRv6: SIDs and micro-SIDs, SR policies and steering, NEXT-C-SID, packet chains, SONiC subset | – | [—](../README.md#segment-routing) | implemented | Landed `ecd32d6`…`02c9fbb`; no history report | +| Gate C | Agent contract, NHT, transport and sessions, reference link-state agent, study windows | – | [—](../README.md#protocol-agents) | implemented | Landed `9032ea8`…`380691a`; reports [C2](history/c2-nht.md), [C2F](history/c2f-nht-runtime-seams.md), [C2R](history/c2r-nht-detached-queries.md), [C4](history/c4-reference-agent.md), [C5](history/c5-study-windows.md) | +| S0 | Scale hygiene: locator index, SRv6 consumer index without copy, arithmetic masks, bounded retention defaults, compact PRNG | – | [—](reference.md) | implemented | Landed `27e92cd`…`b10be0e`, seed lifecycle fix `9d53746`; reports [W1](history/s0-w1-locator-index.md), [W2](history/s0-w2-masks-retention.md), [W3](history/s0-w3-prng.md) | +| S1a | `PMap` changed-key certificate | S0 | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | +| S1b | `PackedTable`, append-only first version, and its index gate | S0 | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | +| F0 | Additive contract records: commands, views, persistent state, RIB feed, `RowNotice`, NHT modes, transport options, settings root field with delta, labels, packed-table and programming record types, records F1 and F2 share | – | [pending](design/protocols.md#f0-contract-additions) | planned | Known gap: [client profiles are network-wide](design/protocols.md#per-node-client-profiles) | +| C6 | Zero-commit publication and off-tree runtime bookkeeping | F0 | [pending](design/protocols.md#c6-zero-commit-publication) | planned | Known gap: [an idle agent run still commits](design/protocols.md#current-behaviour) | +| F1a | Order-independent resolver with the per-leg cycle rule; extended fingerprints | F0 | [pending](design/protocols.md#f1a-order-independent-resolver) | planned | Known defect: [answers depend on unrelated routes and order](design/protocols.md#two-separating-examples) | +| F2a | Delivery of F0 records: commands, configuration, views, change keys | F0, C6 | [pending](design/protocols.md#pending-contracts) | planned | | +| F3 | Transport: scoped accept identity, TTL, GTSM, wildcard listeners | F0 | [pending](design/protocols.md#pending-contracts) | planned | | +| F4 | Domain-scoped oracle IGP, materialized | F0 | [pending](design/protocols.md#pending-contracts) | planned | | +| F5 | Protocol toolkit: SPF, timers on a shared persistent priority structure, route operations, sizes, references, test harness | – | [pending](design/protocols.md#f5-protocol-toolkit) | planned | | +| F7a | Settings: schema, dialects and profiles as data, device layer, transaction and action plan, root delta, minimal provenance | F0 | [pending](design/protocols.md#pending-contracts) | planned | | +| RP1 | Route-policy language and evaluator | F0, F7a | [pending](design/routing-policy.md) | planned | | +| X | F1b + F2b + F8: persistent packed FIB with group versions; RIB submission boundary; RIB feed and owner notifications; NHT installed mode; programming contract with ERROR completion and sync completion | S1a, S1b, F1a, F2a | [pending](design/protocols.md#pending-contracts) | planned | | +| F1c | Incremental resolution with the full resolver as oracle | X, design gate | [pending](design/protocols.md#pending-contracts) | planned | | +| F2c | Runtime scale: lazy context, per-client identity notifications, admission via S1a | F2a, S1a; performance exit after X | [pending](design/protocols.md#pending-contracts) | planned | | +| RP3 | RIB admission with retained submissions; candidate-next-hop activation filter per original next hop; `nht` resolution scope | X, RP1 | [pending](design/routing-policy.md) | planned | | +| RP4 | Forwarding-scope resolution filter; `FIB_INSTALL` exclusion | F1c, X, RP1 | [pending](design/routing-policy.md) | planned | | +| A1 | Lazy analytical forwarding query | F4, X | [pending](design/protocols.md#pending-contracts) | planned | | +| F7b | Group selectors, presets, `explain`, generated settings docs | F7a | [pending](design/protocols.md#pending-contracts) | planned | | +| OR1 | Open/R Spark and the agent shell | F0, F2a, F5, F7a | [pending](design/protocols.md#pending-contracts) | planned | | +| OR2 | LinkMonitor and KvStore | OR1 | [pending](design/protocols.md#pending-contracts) | planned | | +| OR3 | Decision, PrefixManager, Fib with sync completion | OR2, X | [pending](design/protocols.md#pending-contracts) | planned | | +| OR4 | Multi-area | OR3 | [pending](design/protocols.md#pending-contracts) | planned | | +| OR5 | Drain, overload, RIB-feed injection | OR4, RP1 | [pending](design/protocols.md#pending-contracts) | planned | | +| OR6 | Passport, study integration, learned SR | OR4, OR5; performance exit F1c, F2c | [pending](design/protocols.md#pending-contracts) | planned | | +| BG1 | BGP types, AS_PATH, peers, settings fragment | F5, F7a | [pending](design/protocols.md#pending-contracts) | planned | | +| BG2 | Sessions and FSM; a session-only NHT registration in `rib` mode for connect gating until the installed mode lands in X | BG1, F2a, F3 | [pending](design/protocols.md#pending-contracts) | planned | | +| BG3 | Inbound, next hops and path tracking, the `frr` and `rfc4271` decision flavors, Loc-RIB on `PackedTable` with stable path slots, install; differential source fixtures | BG2, X, F4, RP1, S1b; performance exit F1c, F2c | [pending](design/protocols.md#pending-contracts) | planned | | +| BG4 | Outbound, policy hooks, MRAI, gate profiles, origination | BG3; performance exit F1c, F2c | [pending](design/protocols.md#pending-contracts) | planned | | +| BG5 | Route reflection | BG4 | [pending](design/protocols.md#pending-contracts) | planned | | +| BG6 | Confederations | BG4 | [pending](design/protocols.md#pending-contracts) | planned | | +| BG7 | Conformance, FRR adapter and route-policy interfaces, differential lane | BG5, BG6 | [pending](design/protocols.md#pending-contracts) | planned | | +| IX | Integration scenarios | OR6, BG7, RP4, A1, F7b | [pending](design/protocols.md#pending-contracts) | planned | | + +## Milestones + +- **M0**: S0, F0, F5. Open: F0 and F5 are planned. +- **M1**: S1a, the S1b gate, C6, F1a, F2a, F3, F4, F7a, RP1. +- **M2**: X, then F1c, F2c, RP3, A1. +- **M3**: OR1–OR3, BG1–BG3. +- **M4**: OR4–OR5, BG4–BG6, RP4, F7b. +- **M5**: OR6, BG7, IX. +- **M6**: user documentation and final review. + +Design reviews run after M0 and M1 together, after X, after M3 and at the +end. Route-policy work maps onto +these slices as stated in [routing-policy](design/routing-policy.md#slices). + +## Landing rules + +- S1a and S1b each start by writing their gate. Consumers of `PackedTable` and + of zero-commit publication merge only after the corresponding gate passes. +- F0's additive records can land early. The listener, recursive-resolution, + programming, sync and packed-table records are frozen only after the + [identity, publication, resolver and programming obligations](design/protocols.md#f0-contract-additions) + are in their signatures. +- C6 starts with its publication-seam design and tests. F3, F4, F5 and F7a do + not wait for `PackedTable`. +- S1a and S1b change the same state layer and land as one coordinated series, + not as independent parallel edits. C6 alone changes the agent publication + path in `netlab_sim/runtime/agents.py`, `netlab_sim/runtime/pipeline.py` and + `netlab_sim/model/network.py` and the runtime receipt events. +- F0 owns the root settings-delta integration; F7a owns settings resolution and + the action planner. +- X lands as one change against one reviewed storage and programming contract: + `routing.py`, `lpm.py`, `nht.py`, `derive.py`, `forwarding.py` (in + `netlab_sim/model/`) and `netlab_sim/runtime/timeline.py` change together. + F2c consumes S1 and X. +- BG2 owns the temporary `rib`-mode session registration; BG3 owns path + tracking. + +## Backlog outside the protocol plan + +Identified work with a stable ID, not part of the slice table or its +dependencies, and not scheduled: + +- **NG-026**: port the `netlab-sim run` CLI to NetGraph 0.26. + `netlab_sim/cli.py` calls `Scenario.run()`; NetGraph 0.26 replaced that + pattern with `ngraph.prepare(...)` and `ngraph.run(...)` + ([NetGraph 0.26 API, Results and export](https://github.com/networmix/NetGraph/blob/v0.26.0/docs/reference/api.md#results-and-export)). + Whether the optional NetGraph adapter needs other changes is part of the + port. + +## Optional and deferred + +Not in the table, and not scheduled: + +- **F6**, downstream invalidation narrowed by FIB diffs: optional, only if + integration profiles show the conservative scans dominating + ([source](design/archive/protocols-r8.md#f6-downstream-invalidation-measured-optional)). +- **Route policy**: the canonical JSON policy codec; full vendor CLI coverage, + a universal compiler, policy-driven packet forwarding, route leaking and VPN + features, general regular expressions, a YANG management server + ([routing-policy](design/routing-policy.md#outside-the-first-delivery)). +- **Programming**: the `effective` gate profile and stochastic capacity or + lost-completion models, with BGP extensions. +- **BGP beyond BG7**: further decision flavors (`cisco-ios`, `junos`), + add-path, graceful restart with EoR and update-delay, BFD, link-bandwidth + UCMP, aggregates, dynamic neighbours and listen ranges, SR color steering and + SRv6 services, an analytical BGP solver, optimal route reflection + ([source](design/archive/protocols-r8.md#10-deferred-named-extension-points)). +- **Open/R**: KSP2 and UCMP, the RIB policy API, an RTT step detector model. +- **Common**: a control-plane CPU model, incremental SPF, native kernels. +- **Network layer**: resilient hashing, nested B6, REPLACE-C-SID, multi-client + BSIDs, midpoint protection, VRFs, runtime checkpoint, in-band transport, + `loop_model=AMPLIFY` + ([source](design/archive/network-layer-r17.md#gates-and-phases)). From dfde24243fe2b0fcdd6316f86e933706b4b1e7e3 Mon Sep 17 00:00:00 2001 From: Andrey Golovanov Date: Sat, 31 Oct 2026 10:13:36 +0000 Subject: [PATCH 2/2] Schedule stabilization before protocol foundation --- README.md | 4 +- docs/design/README.md | 9 +++- docs/design/stabilization.md | 85 ++++++++++++++++++++++++++++++++++++ docs/plan.md | 51 ++++++++++++---------- 4 files changed, 122 insertions(+), 27 deletions(-) create mode 100644 docs/design/stabilization.md diff --git a/README.md b/README.md index c8ac7fa..23ce899 100644 --- a/README.md +++ b/README.md @@ -231,8 +231,8 @@ CI runs Python 3.11 to 3.14 and the free-threaded 3.14t build. Reports on each development slice, with their evidence and measurements, are in [docs/history](docs/history/README.md). -Planned work (the Open/R and BGP foundation and the shared route-policy -language) is ordered in the roadmap, [docs/plan.md](docs/plan.md). Its +Planned work (stabilization of the current product, then the Open/R and BGP +foundation and the shared route-policy language) is ordered in the roadmap, [docs/plan.md](docs/plan.md). Its contracts are in [docs/design](docs/design/README.md); they describe future behaviour, not features available today. diff --git a/docs/design/README.md b/docs/design/README.md index cb5a49c..0311b69 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -1,8 +1,10 @@ # Design These pages specify behaviour NetLab-Sim does not have yet: the foundation for -Open/R and BGP, and the shared route-policy language. They are for maintainers -who implement or review that work. +Open/R and BGP, and the shared route-policy language. The stabilization page +also states what the review, the NetGraph 0.26 port and the defect fixes of +the implemented product must guarantee. They are for maintainers who implement +or review that work. ## Which document decides what @@ -19,6 +21,9 @@ the archive and a design page disagree, the design page wins. ## Pages +- [Stabilization](stabilization.md): the review of the implemented product, + the NetGraph 0.26 port and the fixes of confirmed defects, including today's + resolver defects. - [Protocol foundation](protocols.md): the common design decisions, the contracts for F0 and F5, the requirements for F1a, C6, S1a and S1b, and the list of pending contracts with their archived sources. diff --git a/docs/design/stabilization.md b/docs/design/stabilization.md new file mode 100644 index 0000000..e1431a2 --- /dev/null +++ b/docs/design/stabilization.md @@ -0,0 +1,85 @@ +# Stabilization + +This page states what the stabilization work must guarantee: a review of the +implemented product (RV), the port to NetGraph 0.26 and the NetGraph-Core +release it requires (NG-026), and the fixes for confirmed defects (FX). Order, +ownership and state are in [docs/plan.md](../plan.md). + +## What counts as a defect + +- A defect is a wrong result, a crash or a wrong error in a scenario the + product supports today: the README, the reference, the tests or the domain's + first principles give a different answer. It is shown by the source and a + reproducer on a pinned commit. +- Behaviour that a planned slice adds is not a defect unless a current + contract is shown to be violated. Two examples: an idle agent run still + commits ([C6](protocols.md#current-behaviour)), and client profiles are + network-wide ([F0](protocols.md#per-node-client-profiles)). + +## RV: review of the implemented product + +- The object is Gate A, Gate B, Gate C and S0 at a pinned commit. RV assesses + implemented behaviour, architecture and code, tests and documentation, the + supported integrations, and scale risks in proportion. Expected answers are + derived independently of the code. A passing suite does not show the + absence of defects, and tests skipped for a missing optional package count + as not run. +- The NetGraph and NetGraph-Core adapters are reviewed against their + documented translation rules; NG-026 qualifies their code on the new + releases. +- Every finding has a reproducer, a grade (blocks an existing supported + scenario, does not block, or is not a defect) and an owning slice. +- The known resolver defects ([two separating examples](protocols.md#two-separating-examples)) + are confirmed again on the pinned commit. +- The findings are recorded in a history report. + +## NG-026: NetGraph 0.26 port + +`netlab-sim run` calls `Scenario.run()`, which NetGraph 0.26 replaced with +preparing a scenario and running the prepared scenario. The port qualifies +every boundary between NetLab-Sim and NetGraph or NetGraph-Core, not only the +CLI, against the public interfaces of the pinned releases. For NetGraph these +are documented in the +[NetGraph 0.26 API reference](https://github.com/networmix/NetGraph/blob/v0.26.0/docs/reference/api.md) +(sections "Custom steps and exporters", "Failure cases", "Results and +export"): + +- **Dependencies**: the exact supported NetGraph release and the NetGraph-Core + release it requires, installed clean into a fresh environment, and a test + lane that runs the adapter tests against them. +- **CLI**: `netlab-sim run` runs and exports through the public path. +- **Workflow step**: `SimStudy` works as a public custom workflow step: + registration, declared needs, and its result through the step output and an + exporter. +- **Network and Core**: network import and export (topology, capacities, + `attrs.sim` settings, risk groups) and the Core adapter (`edge_arrays`, + `build_graph`) work on the pinned releases. +- **Inputs**: demand sets and failure policies come from public scenario + interfaces. Each demand keeps its declared volume and the expansion the + reference states; an unsupported shape gets a clear error. +- **Failures**: NetGraph's failure cases are deduplicated, with occurrence + counts; they are not an ordered stream of draws. The iterations, process and + replay modes each state their draw order, seed derivation and treatment of + repeated patterns. +- The README and the reference change in the same commit as the port. + +## FX: confirmed defect fixes + +- FX fixes and confirms the blockers found by RV or NG-026 that the + [roadmap](../plan.md#landing-rules) assigns to it. Each fix has a regression + test that fails on the old code and an independent confirmation of the fix + and of its intended differences. +- A deferred finding that does not block names its owning slice and the + reason. A finding is not closed by being graded. +- **Resolver.** FX corrects today's cycle, depth and order defects for the + inputs the product already accepts: both + [separating examples](protocols.md#two-separating-examples) and the F1a + rules that existing static routes, links and next-hop types can express. + Before the change, FX meets F1a's fingerprint rule + ([Rules](protocols.md#rules), first item) on today's inputs: it extends + `tests/test_fingerprints.py` to the outputs and scenarios listed there, + records the baselines, and lists and confirms every difference as intended. + The correction changes the one resolver in place, with no fallback or + second resolver. +- FX is complete when every blocker assigned to it is fixed and confirmed and + every other finding has an owner and a reason. diff --git a/docs/plan.md b/docs/plan.md index 5e880ba..7e664f9 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -26,7 +26,7 @@ deployment shapes the design names. - The plan adds behaviour. Scenarios a slice does not affect keep the behaviour the [README](../README.md) and [reference](reference.md) describe. A slice may change an existing answer only as its contract states: to - correct a demonstrated defect, with the reproducer reviewed (F1a's + correct a demonstrated defect, with the reproducer reviewed (the resolver [examples](design/protocols.md#two-separating-examples)), or as a documented interface change with its intended differences reviewed (for example, X's group timeline events). @@ -82,8 +82,9 @@ changes in the commit that changes the product. ## Slices The protocol design's authoritative slice table, with its later amendments of -edges, applied. `X` is the integration boundary F1b + F2b + F8, which lands as -one change. +edges, applied, preceded by the stabilization slices RV, NG-026 and FX. `MS` +in the Needs column is the whole [MS milestone](#milestones). `X` is the +integration boundary F1b + F2b + F8, which lands as one change. | ID | Work | Needs | Contract | Implementation | Evidence | |---|---|---|---|---|---| @@ -91,15 +92,18 @@ one change. | Gate B | SRv6: SIDs and micro-SIDs, SR policies and steering, NEXT-C-SID, packet chains, SONiC subset | – | [—](../README.md#segment-routing) | implemented | Landed `ecd32d6`…`02c9fbb`; no history report | | Gate C | Agent contract, NHT, transport and sessions, reference link-state agent, study windows | – | [—](../README.md#protocol-agents) | implemented | Landed `9032ea8`…`380691a`; reports [C2](history/c2-nht.md), [C2F](history/c2f-nht-runtime-seams.md), [C2R](history/c2r-nht-detached-queries.md), [C4](history/c4-reference-agent.md), [C5](history/c5-study-windows.md) | | S0 | Scale hygiene: locator index, SRv6 consumer index without copy, arithmetic masks, bounded retention defaults, compact PRNG | – | [—](reference.md) | implemented | Landed `27e92cd`…`b10be0e`, seed lifecycle fix `9d53746`; reports [W1](history/s0-w1-locator-index.md), [W2](history/s0-w2-masks-retention.md), [W3](history/s0-w3-prng.md) | -| S1a | `PMap` changed-key certificate | S0 | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | -| S1b | `PackedTable`, append-only first version, and its index gate | S0 | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | -| F0 | Additive contract records: commands, views, persistent state, RIB feed, `RowNotice`, NHT modes, transport options, settings root field with delta, labels, packed-table and programming record types, records F1 and F2 share | – | [pending](design/protocols.md#f0-contract-additions) | planned | Known gap: [client profiles are network-wide](design/protocols.md#per-node-client-profiles) | +| RV | Review of Gate A, Gate B, Gate C and S0: graded findings with reproducers and owners | – | [pending](design/stabilization.md#rv-review-of-the-implemented-product) | planned | | +| NG-026 | Port the NetGraph and NetGraph-Core adapters and the `netlab-sim run` CLI to the public interfaces of NetGraph 0.26 and the NetGraph-Core release it requires, with pinned versions | – | [pending](design/stabilization.md#ng-026-netgraph-026-port) | planned | | +| FX | Fixes of confirmed defects in the implemented product outside NG-026, including today's resolver cycle, depth and order defects, with regressions and independent confirmation | RV | [pending](design/stabilization.md#fx-confirmed-defect-fixes) | planned | | +| S1a | `PMap` changed-key certificate | S0, MS | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | +| S1b | `PackedTable`, append-only first version, and its index gate | S0, MS | [pending](design/protocols.md#s1a-and-s1b-certificates-and-packed-tables) | planned | | +| F0 | Additive contract records: commands, views, persistent state, RIB feed, `RowNotice`, NHT modes, transport options, settings root field with delta, labels, packed-table and programming record types, records F1 and F2 share | MS | [pending](design/protocols.md#f0-contract-additions) | planned | Known gap: [client profiles are network-wide](design/protocols.md#per-node-client-profiles) | | C6 | Zero-commit publication and off-tree runtime bookkeeping | F0 | [pending](design/protocols.md#c6-zero-commit-publication) | planned | Known gap: [an idle agent run still commits](design/protocols.md#current-behaviour) | -| F1a | Order-independent resolver with the per-leg cycle rule; extended fingerprints | F0 | [pending](design/protocols.md#f1a-order-independent-resolver) | planned | Known defect: [answers depend on unrelated routes and order](design/protocols.md#two-separating-examples) | +| F1a | Order-independent resolver with the per-leg cycle rule; extended fingerprints | F0 | [pending](design/protocols.md#f1a-order-independent-resolver) | planned | Known defect: [answers depend on unrelated routes and order](design/protocols.md#two-separating-examples); FX owns its correction for current inputs | | F2a | Delivery of F0 records: commands, configuration, views, change keys | F0, C6 | [pending](design/protocols.md#pending-contracts) | planned | | | F3 | Transport: scoped accept identity, TTL, GTSM, wildcard listeners | F0 | [pending](design/protocols.md#pending-contracts) | planned | | | F4 | Domain-scoped oracle IGP, materialized | F0 | [pending](design/protocols.md#pending-contracts) | planned | | -| F5 | Protocol toolkit: SPF, timers on a shared persistent priority structure, route operations, sizes, references, test harness | – | [pending](design/protocols.md#f5-protocol-toolkit) | planned | | +| F5 | Protocol toolkit: SPF, timers on a shared persistent priority structure, route operations, sizes, references, test harness | MS | [pending](design/protocols.md#f5-protocol-toolkit) | planned | | | F7a | Settings: schema, dialects and profiles as data, device layer, transaction and action plan, root delta, minimal provenance | F0 | [pending](design/protocols.md#pending-contracts) | planned | | | RP1 | Route-policy language and evaluator | F0, F7a | [pending](design/routing-policy.md) | planned | | | X | F1b + F2b + F8: persistent packed FIB with group versions; RIB submission boundary; RIB feed and owner notifications; NHT installed mode; programming contract with ERROR completion and sync completion | S1a, S1b, F1a, F2a | [pending](design/protocols.md#pending-contracts) | planned | | @@ -126,6 +130,11 @@ one change. ## Milestones +- **MS**: RV, NG-026, FX. RV and NG-026 proceed independently; FX starts + from RV's findings and also fixes findings of NG-026 while the port is in + progress. MS closes when all three are done and every blocker they found, + including late port findings, is fixed and confirmed. The remaining M0 + work, and every new foundation slice, waits for MS. - **M0**: S0, F0, F5. Open: F0 and F5 are planned. - **M1**: S1a, the S1b gate, C6, F1a, F2a, F3, F4, F7a, RP1. - **M2**: X, then F1c, F2c, RP3, A1. @@ -149,9 +158,10 @@ these slices as stated in [routing-policy](design/routing-policy.md#slices). - C6 starts with its publication-seam design and tests. F3, F4, F5 and F7a do not wait for `PackedTable`. - S1a and S1b change the same state layer and land as one coordinated series, - not as independent parallel edits. C6 alone changes the agent publication - path in `netlab_sim/runtime/agents.py`, `netlab_sim/runtime/pipeline.py` and - `netlab_sim/model/network.py` and the runtime receipt events. + not as independent parallel edits. Among the protocol slices, C6 alone + changes the agent publication path in `netlab_sim/runtime/agents.py`, + `netlab_sim/runtime/pipeline.py` and `netlab_sim/model/network.py` and the + runtime receipt events. - F0 owns the root settings-delta integration; F7a owns settings resolution and the action planner. - X lands as one change against one reviewed storage and programming contract: @@ -160,18 +170,13 @@ these slices as stated in [routing-policy](design/routing-policy.md#slices). F2c consumes S1 and X. - BG2 owns the temporary `rib`-mode session registration; BG3 owns path tracking. - -## Backlog outside the protocol plan - -Identified work with a stable ID, not part of the slice table or its -dependencies, and not scheduled: - -- **NG-026**: port the `netlab-sim run` CLI to NetGraph 0.26. - `netlab_sim/cli.py` calls `Scenario.run()`; NetGraph 0.26 replaced that - pattern with `ngraph.prepare(...)` and `ngraph.run(...)` - ([NetGraph 0.26 API, Results and export](https://github.com/networmix/NetGraph/blob/v0.26.0/docs/reference/api.md#results-and-export)). - Whether the optional NetGraph adapter needs other changes is part of the - port. +- NG-026 owns fixes local to the NetGraph and NetGraph-Core adapters and the + CLI; FX owns every other confirmed defect of the implemented product (code, + tests, documentation, the SONiC adapter), including those the port exposes. + An FX change that delivers part of F1a (such as the extended fingerprints or + the resolver correction for current inputs) updates F1a's row to that + subset, with its evidence; F1a keeps its F0 dependency and delivers only the + remaining guarantees. ## Optional and deferred