From 570897d05df2d3d0c4b53327ce1ec72c3ded908b Mon Sep 17 00:00:00 2001 From: Andrey Golovanov Date: Sun, 25 Oct 2026 01:27:11 +0100 Subject: [PATCH] Clarify documentation and explain the scenario pipeline --- CHANGELOG.md | 4 + README.md | 126 +++++++++------ dev/PIPELINE.md | 233 ++++++++++++++++++++++++++++ dev/STREAMLINING.md | 363 -------------------------------------------- 4 files changed, 317 insertions(+), 409 deletions(-) create mode 100644 dev/PIPELINE.md delete mode 100644 dev/STREAMLINING.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 808192f..0ad3c47 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Docs**: Explain installation, configuration and scenario generation in a standalone pipeline guide. + ## [0.5.1] - 2026-10-25 ### Changed diff --git a/README.md b/README.md index 6049635..1331277 100644 --- a/README.md +++ b/README.md @@ -2,14 +2,17 @@ [![CI](https://github.com/networmix/TopoGen/actions/workflows/python-test.yml/badge.svg?branch=main)](https://github.com/networmix/TopoGen/actions/workflows/python-test.yml) -TopoGen generates backbone network scenarios for the contiguous United States. -It selects metro locations from Census urban areas, uses highway paths as link -routes, and adds sites, routers, traffic and failure scenarios for -[NetGraph](https://github.com/networmix/NetGraph). +TopoGen creates backbone network scenarios for the contiguous United States. +It chooses metropolitan areas from Census data and follows highways to connect +them. It then adds network sites, routers, traffic demands and failure policies. +The result is a YAML scenario for [NetGraph](https://github.com/networmix/NetGraph). + +There are two steps: `generate` chooses the locations and routes; `build` turns +those routes into a network. NetGraph runs the analysis of that network. ## Install -Requires Python 3.11+. +Use Python 3.11 or later. Run these commands in a terminal: ```bash git clone https://github.com/networmix/TopoGen @@ -19,21 +22,26 @@ source venv/bin/activate python -m pip install -e . ``` -## Download data +This also installs NetGraph and its computation engine, NetGraph-Core, in the +versions this release supports. [pyproject.toml](pyproject.toml) lists those +version ranges and the other dependencies. + +## Download the geographic data -Create a `data/` directory and save these Census files there. Keep the ZIP files -as downloaded; no need to unpack them. +Create a `data/` directory in the repository and save these files there. +Keep the ZIP files as downloaded; do not unpack them. - [Urban areas (2020)](https://www2.census.gov/geo/tiger/TIGER2020/UAC/tl_2020_us_uac20.zip) - [Primary roads (2024)](https://www2.census.gov/geo/tiger/TIGER2024/PRIMARYROADS/tl_2024_us_primaryroads.zip) - [State boundaries (2024)](https://www2.census.gov/geo/tiger/GENZ2024/shp/cb_2024_us_state_500k.zip) -The example configs already use these filenames. To store data elsewhere, change -`data_sources` in the config. Paths are relative to your working directory. +The example configurations use these filenames. To store the files elsewhere, +change `data_sources` in your configuration. Relative paths start at the directory +where you run the command. -## Run +## Build a network -Start with the five-metro example. Run these commands from the repository root: +Run this five-metro example from the repository root: ```bash cp examples/small_baseline.yml config.yml @@ -42,56 +50,81 @@ topogen generate config.yml -o output topogen build config.yml -o output ``` -`info` shows the config and whether data files exist. `generate` finds highway -routes between metros. `build` adds the site topologies, hardware and traffic, -then validates the NetGraph scenario. +`info` checks the configuration and shows whether its data files exist. +`generate` selects the metros: the urban areas you name, then the largest +remaining ones by land area. It then finds highway routes between nearby metros. +A route between two metros is called a **corridor**. + +`build` adds points of presence (PoPs), where backbone routers are located, and +data-center sites. A **blueprint** describes the devices and links inside a site. +The build also assigns hardware, creates traffic demands and checks the scenario +with NetGraph before saving it. -Files in `output/` use the config filename as their prefix: +Files in `output/` use the configuration filename as their prefix: | File | Contents | | --- | --- | -| `config_integrated_graph.json` | Metro locations and highway routes | -| `config_scenario.yml` | NetGraph scenario, ready to run | -| `config_*.jpg` | Network maps and site diagrams | +| `config_integrated_graph.json` | Metro locations and corridors from `generate`; `build` reads it | +| `config_scenario.yml` | The network, traffic, failure policies and analysis steps | +| `config_network_graph.json` | Sites and the links between them, with their capacities | +| `config_*.jpg` | Maps and site diagrams enabled in the configuration | -Map backgrounds need an internet connection. To run the capacity, failure and -cost analyses defined in the scenario: +Map backgrounds need an internet connection. To run the analysis steps saved in +the scenario: ```bash ngraph run output/config_scenario.yml --output output ``` -## Configure +## Change the design -Edit `config.yml`. These are the main settings: +Edit `config.yml`. Capacities and traffic volumes in the supplied examples use +Gbps; geographic distances use kilometers. -| Setting | Controls | +| Setting | What it controls | | --- | --- | -| `clustering.metro_clusters` | Number of metros | -| `clustering.override_metro_clusters` | Metros to include by name | -| `corridors` | Nearest neighbors, route count and distance limits | -| `build.build_defaults` | PoPs, data centers, site blueprints and link capacities | +| `clustering.metro_clusters` | Number of metro locations | +| `clustering.override_metro_clusters` | Name patterns of urban areas to include; they count toward `metro_clusters` | +| `corridors` | Nearby metros to connect, route count and distance limits | +| `build.build_defaults` | PoPs, data-center sites, blueprints, link capacities and costs | | `build.build_overrides` | Different settings for selected metros | -| `build.tm_sizing` | Link sizing from traffic, including headroom | -| `traffic` | Traffic volume and model: uniform, gravity or hose | -| `components` | Router and optic assignments | - -See [examples/](examples/) for baseline, Clos and Dragonfly configs, and the -[schema](topogen/schemas/topogen_config.json) for all settings. - -Custom definitions go in `lib/blueprints.yml`, `lib/components.yml`, -`lib/failure_policies.yml` or `lib/workflows.yml`. Each file maps names to -definitions; matching names replace built-ins. - -After changing geography, rerun `generate` and `build`. For site, hardware or -traffic changes, rerun `build`. To process all example configs: +| `build.tm_sizing` | Link capacities calculated from traffic, with spare capacity | +| `traffic` | Traffic volume and distribution model | +| `components` | Router types and optics for link endpoints | +| `failure_policies` | Failures to model in the scenario | +| `workflows` | Analysis steps for NetGraph to run | +| `visualization` | Maps and diagrams to export | + +The traffic models are `uniform` (equal traffic between data-center sites), +`gravity` (traffic weighted by site size and distance), and `hose` (traffic fitted +to each site's send and receive totals). + +Start with the [baseline, Clos and Dragonfly examples](examples/). +The [configuration schema](topogen/schemas/topogen_config.json) lists every +accepted field and its allowed values. + +Your own sites, hardware, failures and analyses go in `lib/blueprints.yml`, +`lib/components.yml`, `lib/failure_policies.yml` and `lib/workflows.yml`. +TopoGen reads `lib/` in the directory where you run the command. Each file maps +names to definitions; a name that matches a built-in definition replaces it. +The repository's `lib/` already adds failure policies and workflows, such as +`hose_samples_3` for three sampled traffic matrices. See +[how the pipeline works](dev/PIPELINE.md) for the build rules, Python entry +points and validation steps. + +After changing locations or highway routing, run `generate` and `build` again. +After changing sites, hardware or traffic, run `build` again. + +## Build several configurations ```bash ./build.sh examples output ``` -This writes each config's results to its own subdirectory. Use `--force` to -regenerate saved graphs or `--build-only` to rebuild scenarios from them. +Each configuration gets a subdirectory, such as `output/small_baseline/`. +The script reuses a saved corridor graph when one exists. Use `--force` to +regenerate the graphs, or `--build-only` to build scenarios from saved graphs. +A failed configuration makes the command return a failure status. ## Development @@ -100,7 +133,8 @@ python -m pip install -e '.[dev]' make check-ci ``` -In a Superset workspace, use `bash .superset/workspace.sh setup` to install the -development environment. +This checks formatting, lint, types, example configurations and tests. +In a Superset workspace, `bash .superset/workspace.sh setup` installs the +development environment; the Run button runs these checks. -[Changes and migration notes](CHANGELOG.md) · [MIT license](LICENSE) +[Changelog](CHANGELOG.md) · [MIT license](LICENSE) diff --git a/dev/PIPELINE.md b/dev/PIPELINE.md new file mode 100644 index 0000000..5c4d2a2 --- /dev/null +++ b/dev/PIPELINE.md @@ -0,0 +1,233 @@ +# How TopoGen builds a scenario + +TopoGen first chooses metro locations and the highway routes between them. +It saves this geography so you can change the network design without processing +the road data again. A second step adds sites, devices and traffic, and produces +a NetGraph scenario. + +Run commands from the repository root. Relative data and `lib/` paths start at +the working directory, even when the configuration file is in another directory. + +## 1. Choose locations and corridors + +```bash +topogen generate examples/small_baseline.yml -o output +``` + +`TopologyConfig.from_yaml` loads the settings and checks them against the +configuration schema. The geographic step then works in five stages: + +1. It reads the Census urban areas, primary roads and state boundaries and keeps + urban areas in the contiguous United States. Coordinates are projected to + `projection.target_crs`, which must measure distance in meters. +2. It picks `clustering.metro_clusters` metros. Each pattern in + `override_metro_clusters` selects the largest urban area whose name contains + it, ignoring case. The remaining places go to the largest urban areas by land + area. A metro is a point inside its urban area with a radius derived from its + land area, capped at `max_uac_radius_km`. +3. It builds a graph of the selected highway classes and attaches each metro to + its nearest highway point. That point must be within + `validation.max_metro_highway_distance_km`. +4. It pairs each metro with its `corridors.k_nearest` nearest metros, counting + only those within `max_edge_km` in a straight line. For each pair it keeps up + to `k_paths` shortest highway routes of at most `max_corridor_distance_km`. +5. It gives each route a named risk group. A highway segment carries the risk + group of every route that uses it, and a corridor collects the risk groups of + all its segments. Corridors that share a highway segment therefore share a + risk group, so a failure policy can fail them together. By default, segments + inside a metro's radius are left out (`risk_groups.exclude_metro_radius_shared`). + +The output is `output/small_baseline_integrated_graph.json`. Its graph is a +NetworkX `MultiGraph`: two metros can have several separate corridors. The edge +key is the route index, and key 0 is the shortest route. Corridor distances, +routes and risk groups belong to those edges. Do not convert it to a simple +graph, which would lose parallel corridors. + +Python entry points are `topogen.build_integrated_graph`, +`topogen.save_to_json` and `topogen.load_from_json`. Serialization stores the +coordinate reference system with the graph. The build command checks it against +`projection.target_crs`; regenerate the graph if the projection changes. + +## 2. Add sites, devices and traffic + +```bash +topogen build examples/small_baseline.yml -o output +``` + +The build reads `output/small_baseline_integrated_graph.json`, the saved graph +with the same configuration filename prefix. + +### Sites and links + +Each metro gets `pop_per_metro` PoPs and `dc_regions_per_metro` data-center +sites, from `build.build_defaults` or from a `build_overrides` entry that names +the metro. Three kinds of site links connect them: + +| Link type | Connects | +| --- | --- | +| `intra_metro` | Every pair of PoPs in a metro | +| `dc_to_pop` | Each data-center site to every PoP in its metro | +| `inter_metro_corridor` | The PoPs of two metros, once for each corridor between them | + +A corridor connects every PoP pair of its two metros with `mode: mesh`. With +`mode: one_to_one` it connects PoP 1 to PoP 1, PoP 2 to PoP 2, and so on. + +A link's routing `cost` is its length in whole kilometers, rounded up, unless +the configured link `cost` is larger. Corridors use their highway length; links +inside a metro use distances around a circle of the metro's radius. The rounded +length is also kept as `distance_km`. For example, a 412.3 km corridor with +configured cost 1 gets `cost: 413` and `distance_km: 413`. + +### Traffic + +Each data-center site has `traffic.mw_per_dc_region` megawatts of power and +offers `gbps_per_mw` Gbps per megawatt. The small baseline has three +data-center sites of 150 MW at 200 Gbps per MW. Each site offers 30,000 Gbps, +and the traffic matrix totals 90,000 Gbps. The model decides which site pairs +carry that traffic: + +- `uniform` divides it equally among all ordered pairs of distinct sites. +- `gravity` weights a pair by the product of the two sites' power, divided by + their distance; `alpha` and `beta` are the exponents of those terms. +- `hose` starts from random pair weights, optionally favoring short distances, + and fits them so each site sends and receives its offered volume. + +`traffic.gravity.rounding_gbps`, when above zero, rounds the emitted pair volumes +of both the gravity and the hose model to that step. Rounding can shift a site's +totals slightly away from its offered volume. `priority_ratios` divides the +volume into priority classes. The demand set is +named `traffic.matrix_name`. With the hose model and `samples` above 1, the build +writes one set per sample: `_1`, `_2` and so on. The +built-in workflow and traffic-based sizing use `` itself, so select +a workflow that names the sample sets and set `build.tm_sizing.matrix_name` to +one of them. The repository's `lib/workflows.yml` has `hose_samples_3` and +`hose_samples_10` for `matrix_name: baseline_traffic_matrix`. + +### Link sizing from traffic + +When `build.tm_sizing.enabled` is true, TopoGen sizes links for the demand set +named by `tm_sizing.matrix_name`, which defaults to `traffic.matrix_name`: + +1. It treats each metro as one node and routes every metro-to-metro demand over + the corridor links by cost. `flow_placement` sets how traffic splits across + equal-cost paths. +2. Each corridor link that carries traffic gets its peak load in either direction, + multiplied by `headroom` and rounded up to a multiple of `quantum_gbps`. +3. A PoP's egress is the total capacity of its corridor links. A data-center link + gets `alpha_dc_to_pop` times the egress of its PoP. A PoP-to-PoP link in a + metro gets `beta_intra_pop` times the smaller egress of its two PoPs. Both are + rounded up to a multiple of `quantum_gbps`. +4. With `respect_min_base_capacity`, no link falls below its configured capacity. + +For example, take two metros with one PoP and one 25 MW data-center site each, +200 Gbps per MW and uniform traffic. Each direction carries 5,000 Gbps. With +headroom 1.3 and a 3,200 Gbps quantum, the corridor needs 6,500 Gbps, which +rounds up to 9,600 Gbps. Each PoP's egress is then 9,600 Gbps, so with +`alpha_dc_to_pop: 1.2` each data-center link needs 11,520 Gbps and gets 12,800. + +### Devices and hardware + +The build expands the site blueprints into devices and links. A site link's +capacity is a budget for that pair of sites; it is divided equally among the +device links that connect them. `components.hw_component` assigns a router +platform to each device role. + +`components.optics` maps `local_role->remote_role` to the optic installed at the +local end of a link. Each end gets enough optics to carry its link: the link +capacity divided by the optic capacity, rounded up. In the example above, one +9,600 Gbps device link with `800G-ZR+` optics (800 Gbps each) needs 12 optics at +each end. When `optics` is set, every link end needs a mapping: a link between +`core` and `leaf` routers needs both `core->leaf` and `leaf->core`, or the build +stops with an error. + +The scenario YAML contains `seed`, `blueprints`, `components`, the `network`, +`risk_groups`, failure policies under `failures`, traffic under `demands` and the +analysis `workflow`. Building the scenario does not execute that workflow. + +## 3. Validate and save + +The CLI loads the scenario with NetGraph, which checks the YAML schema, the +workflow steps and the failure policies. It then checks the network for isolated +devices and audits hardware, ports, optics, roles and data-center capacity. For +each demand set, a data-center site's total sent traffic, and separately its total +received traffic, must fit within the capacity of the links that leave the site. +The CLI replaces the destination YAML only after validation succeeds, so a failed +build keeps the existing file. + +`topogen.scenario.build_scenario` returns YAML without validating it. This script +builds, validates and writes a scenario from Python. It runs the same checks as +the CLI, but writes the file directly instead of replacing it atomically: + +```python +from pathlib import Path + +from topogen import RunContext, TopologyConfig, load_from_json +from topogen.scenario import build_scenario +from topogen.validation import validate_scenario_yaml + +config = TopologyConfig.from_yaml(Path("examples/small_baseline.yml")) +context = RunContext(Path("output"), "small_baseline") +graph_path = context.path("integrated_graph.json") +graph, crs = load_from_json(graph_path) +assert crs == config.projection.target_crs + +scenario_yaml = build_scenario(graph, config, context=context) +issues = validate_scenario_yaml( + scenario_yaml, + integrated_graph_path=graph_path, + hw_component_map=config.components.hw_component, + optics_map=config.components.optics, +) +if issues: + raise SystemExit("\n".join(issues)) +context.path("scenario.yml").write_text(scenario_yaml) +``` + +`validate_scenario_yaml` returns a list of problems; an empty list means the +scenario passed. The two component maps enable the hardware and optic coverage +checks that the CLI runs. + +`topogen.RunContext` sets the output directory, filename prefix and optional +debug directory for one invocation. `RunContext(Path("output"), "small_baseline")` +names files with the `small_baseline_` prefix. Output paths are separate from +topology settings. With a context, `build_scenario` also writes +`small_baseline_network_graph.json` and the maps and diagrams enabled in the +configuration. + +## Libraries and output + +The `lib/` directory supplies custom blueprints, hardware components, failure +policies and workflows. Each YAML file maps names to definitions; custom names +are added and matching built-in names are replaced. Referenced site blueprints +and their dependencies are included in the scenario. Unknown references and +cyclic blueprint dependencies are errors. + +`workflows.assignments.default` chooses a workflow. Each step uses `type` for its +analysis class and `demand_set` when it needs traffic. Built-in workflow steps use +`traffic.matrix_name`; workflows from `lib/workflows.yml` keep the names they list. +In the built-in workflow, `MaximumSupportedDemand` searches for the largest +multiplier of the traffic that the network can carry. `TrafficMatrixPlacement` +then places the traffic, scaled by that multiplier, under each sampled failure +of its `failure_policy`. + +The `failures` section contains the policy named by +`failure_policies.assignments.default` and every policy that a workflow step +names. File prefixes come from the configuration filename, not the names inside +the YAML. The batch script adds one subdirectory per configuration. + +## Working on the code + +| Area | Main files | +| --- | --- | +| Settings and CLI | `topogen/config.py`, `topogen/cli.py`, `topogen/context.py` | +| Geographic processing | `topogen/metro_clusters.py`, `topogen/highway_graph.py`, `topogen/corridors.py`, `topogen/integrated_graph.py` | +| Scenario assembly | `topogen/scenario/assembly.py`, `sizing.py`, `expansion.py`, `striping.py` | +| Traffic models | `topogen/traffic_matrix.py` | +| Scenario checks | `topogen/validation/` | +| Maps and diagrams | `topogen/visualization.py`, `topogen/blueprint_viz.py` | + +Run `make check-ci` after changes. It validates the example configurations and +runs formatting, lint, type checks and tests. Integration tests exercise scenario +building and NetGraph validation without downloading the national datasets. +To check geographic generation, download the files listed in the +[README](../README.md#download-the-geographic-data) and run both commands on this page. diff --git a/dev/STREAMLINING.md b/dev/STREAMLINING.md deleted file mode 100644 index 73ee62c..0000000 --- a/dev/STREAMLINING.md +++ /dev/null @@ -1,363 +0,0 @@ -# TopoGen streamlining - -Objective: simplify the topology pipeline, make its configuration and graph -contracts explicit, fix correctness gaps, and improve measured performance. -This is one cleanup with implementation gates, not a sequence of releases. - -Baseline: `68fd680`; `make check-ci` passes (315 tests). Official Census inputs -and reproducible baseline artifacts are in ignored `data/` and `output/`. -Before-change profiles and all four sample scenarios are saved in -`output/streamline/before/`. Dragonfly assembly expands NetGraph DSL 28 times -for 27 site adjacencies; capacity counting accounts for over half of profiled -assembly time. - -## Gates - -1. **One network expansion for assembly.** Generate demands once; retain their - identity through sizing and emission. Expand the complete network once, - including node rules; index concrete links by site edge. Use that index for - capacity allocation and optics. Never group distinct capacity budgets by a - shared adjacency family. Verify heterogeneous capacities, striping, custom - blueprints, and semantic equivalence of the four examples. -2. **Explicit configuration and execution context.** Separate serializable - settings from paths and export context. Remove hidden config attributes, - no-op controls, duplicated link parsing, and fallback logic that conceals - invalid state. Keep CLI errors actionable and only report success after - validation. Document intentional contract changes. -3. **Coherent geographic graph contract.** Fix nearest-neighbor adjacency - dependence on metro IDs; make path multiplicity honest throughout discovery, - serialization, site construction, and risks. Replace whole-road-graph scans - per corridor with path/index traversal; preserve geographic provenance. - Verify on adversarial small graphs and the downloaded national datasets. -4. **Validation and delivery.** Preserve node rules in audits, avoid repeated - expansion where the same network suffices, and keep construction, hardware, - capacity and failure-policy checks. Run `make check-ci`, real generation and - build for all example designs, before/after semantic comparisons and - repeatable timings. Update README and Unreleased changelog with the final - contracts, evidence and limitations. No push is part of this task. - -Progress is recorded below as evidence becomes available. Existing behavior is -not a correctness oracle where a regression demonstrates a bug. Timing claims -must separate computation from optional map rendering and network tile access. - - -## Completed evidence - -All four gates are complete. The package now has one explicit corridor graph -contract and one assembly path; removed APIs and formats have no compatibility -adapter. Required migration steps are in [README](../README.md#current-contracts-and-migration). - -- Assembly generates traffic once and expands the complete device DSL once, - including node rules. Capacities and endpoint optics use the actual concrete - links for each site edge. Heterogeneous PoP budgets, asymmetric optics, missing - reverse assignments and caller link attributes have regression coverage. -- Typed `RunContext` owns output paths; configuration has no hidden export state. - Schema checks reject removed controls and formats. Library parsing is shared; - malformed libraries and computation/render failures propagate. CLI validation - precedes atomic YAML replacement, including `--print`. -- Corridor identity survives contraction, k-nearest pairing, k-path extraction, - JSON and site expansion. Risks use the owning path's length and reject unknown - owners. Tests cover parallel routes of different lengths, ID-order dependence, - JSON round trips and shared risks. Road copies and scans were removed from - contraction and corridor extraction where ownership/indexing suffices. -- `make check-ci`: **324 passed**, 87.61% coverage; Ruff formatting/lint, Pyright - and all four example schemas pass. Pyright reports one existing dependency - typing warning for Contextily (no type stub), with zero errors. Full log: - `output/streamline/final-ci.log`. Real NetGraph parsing and reduced workflows - execute in integration tests. Published changelog history is unchanged. - -### Repeatable assembly measurement - -Recorded for the initial cleanup, before the follow-up review below. These -timings were not rerun after the audit corrections. - -Local Python 3.13.13, NetGraph 0.24.0, NetGraph-Core 0.11.0. Compare baseline -`68fd680` in an isolated checkout with this implementation, using the **same -frozen geographic graph**, corresponding example configs and one virtualenv. -One warmup, seven timed iterations; median milliseconds. Optional maps and -artifact I/O are excluded. This is an assembly measurement, not a claim about -national geographic generation or NetGraph simulation throughput. - -| Example | Device nodes / links | Before, ms | After, ms | Build speedup | -| --- | ---: | ---: | ---: | ---: | -| small_baseline | 13 / 27 | 32.2 | 30.1 | 1.07x | -| small_clos | 127 / 756 | 55.3 | 37.4 | 1.48x | -| small_dragonfly | 133 / 739 | 83.5 | 43.9 | 1.90x | -| small_dragonfly_custom | 103 / 612 | 64.2 | 39.9 | 1.61x | - -Evidence: `output/streamline/compare_assembly.py`, `benchmark-before/` and -`benchmark-after/` under `output/streamline/`. Every iteration passed scenario -validation. All four parsed scenario documents are equal after accounting for -explicit `:path0` adjacency identities and newly preserved `euclidean_km` / -`detour_ratio` attributes. Comparison script and result: -`verify_semantics.py`, `semantic-comparison.json` in the same directory. -Corrected geographic routing can intentionally change distances when regenerating -from Census inputs; the frozen-graph comparison isolates assembly changes. - -### Real-data artifacts - -Official Census ZIPs, provenance and integrity/schema/CRS checks are in `data/` -and `data/SOURCES.json`: 2,645 urban areas, 17,522 primary-road features and 56 -state/territory boundaries. All four examples were generated from these data, -then built and validated. Each has 5 metros and 8 corridors. Current artifacts -are in `output/` and `output/streamline/final/`; old outputs are retained under -`output/streamline/before/`. - -```bash -venv/bin/python -m topogen generate examples/small_baseline.yml -o output -venv/bin/python -m topogen build examples/small_baseline.yml -o output -``` - -The same commands succeed for `small_clos`, `small_dragonfly` and -`small_dragonfly_custom`. Maps and blueprint diagrams were rendered; the -baseline geographic/site maps and Clos diagram were visually inspected. -The real example builds with maps took about 12 seconds per CLI invocation; -Contextily tile access and rendering dominate compared with assembly alone. -Full Monte Carlo workflows on these real artifacts were not run. No remote CI, -commit, push or release was performed. - - -## Follow-up correctness review - -The earlier passing suite did not establish absence of defects. Added behavioral -witnesses before fixes; failures and verification logs are in `output/recheck/`. -No compatibility path or format was added during this review. - -- **DC demand accounting:** separate demand sets were summed, regex selectors - bypassed checks, and DSL rules were used instead of actual external link - capacity. The audit now resolves demands through NetGraph, checks every set - independently, excludes internal/disabled links and reports unmatched demands. - Dictionary-only validation now explicitly covers metadata/references, while - capacity auditing belongs to the NetGraph-backed path. -- **Hose fitting:** impossible margins could silently become a different profile. - Both fitting stages now share one vectorized fitter with feasibility and - convergence checks; exactly saturated margins use their unique star solution. -- **Disabled rounding:** gravity and hose rounded to two decimals even with - rounding disabled, erasing small demands. They now preserve unquantized values. -- **Numeric/config contracts:** integer strings above 2^53 lost precision through - float conversion, and NaN/infinity could bypass schema bounds. Exact parsing - and recursive finite-number checks reject these cases. -- **Geographic contracts:** projected metre units are now required; degree/foot - CRS values previously reached metre-based calculations. Duplicate metro names - could merge site identities; names/IDs and coordinate values are now checked. - PoP coordinates are cross-checked even when a metro has no DC group. -- **Validation robustness:** malformed network/attribute/reference shapes report - issues instead of aborting the validator. -- **Component inventory:** hardware overridden in a blueprint remained a library - dependency. Inventory now scans finalized blueprints and reuses that section. -- **Cleanup:** removed an unused graph-filter argument, a silent role-map - substitution and duplicated IPF loops; corrected library and audit API docs. - -Final `make check-ci`: **341 passed**, **87.77% coverage**, Ruff and all four -example schemas pass, Pyright has zero errors and the existing Contextily stub -warning. Added 17 regression cases in `tests/test_review_regressions.py`. -`git diff --check` passes; published changelog history is preserved. - -Additional behavioral evidence: - -- Contraction preserves the first five shortest simple-path lengths on 163 - connected random graphs (`contraction-differential.txt`). -- Sixty seeded hose profiles at scales 1e-6, 1 and 1e6 preserve every DC total - within 1e-8 relative error (`hose-numerics.txt`). -- All four real-data examples are rebuilt and validated with no change to their - parsed scenario documents. All four workflow steps execute on each example - with iteration counts reduced to 3; MSD alpha is positive (1.6875). Results and - the reproducible runner are `real-workflow-results.json` and - `run-real-scenarios.py` in `output/recheck/`. - -This verifies local construction, validation and reduced workflow execution. -Full Monte Carlo runs and remote CI were not run. Changes remain uncommitted; -no push or release was performed. - - -## Full-source simplification and performance audit (completed) - -Objective: inspect all package code and support scripts for straightforward -control flow, duplication, defects and unnecessary computation. Preserve no -legacy compatibility paths. Previous test success is a baseline, not completion. - -Coverage gates: - -1. Traffic models: shared pair selection/rounding/emission, deterministic PRNG, - disabled-debug overhead, bounded fitting and conservation witnesses. -2. Site graph and sizing: endpoint/stripe selection, distinct corridor identities, - repeated expansion/I/O, topology modes, capacity and serialization contracts. -3. Configuration and libraries: one validation boundary, exact conversion, - overrides/references, coherent public loading API. -4. Geographic pipeline: metro selection, road anchoring/contraction, corridor - discovery/risks, serialization and representative real-data execution. -5. Remaining modules: CLI/context, audits, visualization/helpers, entry points, - packaging/dev/CI/workspace scripts and test adequacy. -6. Final evidence: full current checks, adversarial witnesses, before/after - measurements on identical inputs, real examples and package build. - -Inspection and performance artifacts are in `output/full-review/`; the 341-test -result in `output/recheck/` is the starting baseline. Current completion evidence -for every coverage gate is recorded below. - -### Full-source audit completion - -All six coverage gates are complete for the current local source. This is a -behavioral/code audit, not a proof that no possible defect remains. No legacy -adapter or compatibility re-export was introduced. - -| Coverage gate | Inspected scope and outcome | -| --- | --- | -| Traffic | All three models, common inventory, partner selection, fitting, quantization, PRNG and output. Shared emission replaces duplicate branches; debug formatting is conditional. | -| Sites and sizing | All adjacency families, roles, stripe selection, expansion, capacity routing, risk ownership and serialization. Construction, sizing and network serialization have separate modules. Ordered metro-pair demands share one routing operation and one capacity update rule. | -| Config and libraries | Every dataclass/schema section and all four libraries. Schema owns scalar bounds; exact key conversion and cross-field checks remain explicit. Libraries include nested blueprint dependencies, detect cycles, and are finalized once before stripe selection. Workflow definitions load once per assembly. | -| Geography | UAC filtering/overrides, naming, coordinates, road loading/snapping, contraction, anchoring, nearest-neighbor pairs, corridor extraction/risk and JSON. The four national-data examples were regenerated and compared with saved graphs. | -| Remaining package and support | CLI/context/logging/public exports, all validation audits, maps/blueprint diagrams, Makefile, batch/analyzer/workspace scripts, packaging and CI configuration. Tests were migrated to current internal APIs; no compatibility wrappers. | -| Verification | Full local CI target, baseline-failing witnesses, traffic differential/performance checks, real generation/build/validation/reduced workflow runs, package build/metadata checks and installed-wheel smoke test. | - -Additional defects isolated and fixed during this audit: - -- Gravity top-K selection depended on alphabetical metro-name order; jitter used - the global PRNG. Zero-share classes could also enter invalid normalization. -- Uniform traffic expanded across devices, introducing intra-DC traffic and - changing site shares with blueprint size. Explicit `group_pairwise` emission - now matches DC-level sizing. -- Striping ignored `one_to_one` and custom endpoint matches, collapsed equal - device basenames from distinct groups and treated literal names as regexes. - Stripe rules now address exact full paths and merge attributes per device. - Disabled corridor risks no longer leak into emitted adjacencies. -- Config priority conversion truncated fractions; normalized duplicates and - repeated metro overrides overwrote previous entries. Unknown stripe keys and - unmatched forced metros were silently ignored. They now fail explicitly. -- Configured link cost was ignored for geometric links. It now acts as a minimum - routing cost while `distance_km` retains the geographic value. -- Sizing accepted partial endpoint patterns and non-finite/negative volume. - Those inputs are rejected before routing. -- Nested blueprint definitions omitted their dependencies. Both build and stripe - selection now use the same finalized library with recursive dependencies. -- Blueprint diagrams implemented a second, permissive DSL parser; dictionary - selectors created phantom groups. Both panels now use actual expanded devices, - with group counts and summed link capacities. -- The UAC boundary mask assumed NAD83 regardless of source CRS. Metro coordinates - were NumPy scalars, causing direct `generate -> build` Python calls to fail YAML - emission even though a JSON round trip succeeded. The loader now returns native - floats and uses the actual source CRS for filtering. -- Role audits accepted `None`, boolean and numeric roles as strings; chassis - capacity audits accepted NaN/infinity. These malformed values now produce issues. -- Batch builds returned exit 0 after stage failures. The analyzer called k-core - routines on unsupported MultiGraphs. Both have current-contract behavior; - vertex analysis uses simple adjacency and preserves reported corridor counts. - -Final evidence (all paths below are relative to `output/full-review/`): - -- `final-check-ci.log`: **375 passed**, **89.52% coverage**; Ruff and four config - validations pass; Pyright has **0 errors**, with one existing missing Contextily - stub warning. `git diff --check` and shell syntax checks pass. -- The `*-before-tests.log` files retain failing witnesses; `real-examples-before.log` - records the direct Python/YAML scalar failure before its fix. -- `traffic-parity.log`: **72 profiles** cover gravity/hose, multiple inventory - sizes and all rounding policies. Non-volume fields match exactly; directional - volumes match within 1e-12 relative / 1e-10 absolute tolerance. -- `traffic-before-bench.json` and `traffic-final-bench.json`: seven-run local - medians on identical inputs, logging disabled, allocation peak measured - separately. Gravity (80 DCs, 18,960 entries): **90.97 -> 6.50 ms** and - **29,040,021 -> 9,586,183 bytes**. Hose (40 DCs, 3 samples, 14,040 entries): - **12.56 -> 5.33 ms** and **9,911,842 -> 6,815,101 bytes**. These are generator - microbenchmarks, not end-to-end or cross-machine performance claims. -- `real/results.json`: all four examples regenerate **5 metros / 8 corridors**; - device/link counts are **13/27**, **127/756**, **133/739**, **103/612**. - Four workflow steps run on each with Monte Carlo iterations reduced to **3**; - MSD alpha remains **1.6875**. Full Monte Carlo runs were not performed. -- `network-parity.json`: all four expanded networks match the saved baseline - exactly in device attrs and link capacities/costs/risks/hardware. The scenario - text changes only by explicit `logic: or` on single-condition role selectors; - the real examples use hose, so the uniform contract change is exercised by - separate heterogeneous-DC regression tests. -- Fresh maps and blueprint diagrams are under `real/`; the final Clos diagram - was visually inspected for counts, capacities, group layout and readable labels. -- `package-build.log`, `package-check.log`, `pip-check.log`, `wheel-smoke.log`: - wheel/sdist build, Twine metadata checks and dependency checks pass. Importing - the installed wheel (outside the source package path) loads its schema and - builds/validates the real baseline. - -Released changelog history remains byte-for-byte unchanged. Work is local on -`explain-topology-pipeline-2cd9ea05`, based on `68fd680`; changes remain uncommitted. -No push, remote CI, release or deployment was performed. - -### Contextily typing follow-up (2026-10-04) - -Added `typings/contextily/__init__.pyi` for the `add_basemap` arguments used by -TopoGen, checked against installed Contextily 1.7.1. This is an explicitly limited -static description, with no runtime wrapper, `Any` escape hatch or diagnostic -suppression. Pyright uses the explicit `typings` stub path. - -`make lint` now reports **0 errors / 0 warnings**. A separate temporary type probe -accepts the real call and rejects incorrect alpha, attribution and Axes types -(`output/full-review/contextily-typing/probe-result.json`). No runtime source was -changed; the earlier 375-test result remains the latest full-suite run. - -### Code and architecture recheck (2026-10-04) - -This follow-up supersedes the earlier test count. It checks current production -modules, repository callers, tests, comments/docstrings, CLI/batch scripts and -example configuration. GeoPandas, Matplotlib, Contextily and its local stub remain. - -Removed twelve helpers whose only repository consumers were their own tests: -the four point/transform/bearing functions, three MetroCluster convenience -methods, TopologyConfig.summary, and four singular/list/filter component APIs. -The effective component library remains available through get_builtin_components; -topogen info remains the configuration summary command. These are intentional -API removals with no compatibility aliases, documented in README and Unreleased. - -Removed the unused corridor_path_ids edge tags and three redundant CorridorPath -fields. Scenario construction now reads the corridor MultiGraph directly instead -of copying every edge to an intermediate list of dictionaries. The shared site -edge identity lives in naming rather than forcing serialization to depend on -device expansion. Sizing resolves the Core flow-placement enum directly and -combines capacity adjustment, aggregate reporting and PoP egress accumulation. -Removed unused loggers, an empty TYPE_CHECKING block, obsolete type/noqa -suppression comments and dead batch-runner state. Production Python is smaller by -253 lines relative to this review's starting snapshot. - -Two misleading diagnostics were reproduced before changes: - -- A DC-to-PoP YAML rule received an “Inter-metro corridors” heading because the - comment inserter searched the next 15 lines. Deleted this text heuristic; - link_type attributes remain authoritative. Witness: before-build.log. -- Largest-component filtering legitimately removed a metro anchor, but raised - a RuntimeError alleging a contraction bug. It now raises ValueError identifying - highway filtering and the relevant settings. The new regression fails against - the starting source (anchor-before.log) and passes after correction. - -Corrected descriptions of full-mesh PoP adjacency with ring-arc costs, -representative points versus centroids, hose same-metro weighting, component -filtering and geometry cleanup counts. Reused the single dissolved CONUS geometry -without dissolving it again. Duplicate snapped road segments have the same length -by construction, so their unreachable shorter-edge branch was removed. - -CLI tests no longer restore global print or mock sys.exit with another SystemExit -implementation. They now require the expected exceptions. The quiet-mode test -uses valid CLI syntax, checks successful dispatch, and demonstrates that the same -stub prints when quiet mode is absent. Previously it passed on argparse rejecting -the unsupported -c option. Ruff now checks unused noqa directives. The two SciPy -imports retain narrow missing-stub suppressions; removing them was checked and -does produce real missing-stub diagnostics. Contextily needs no suppression. - -Evidence is under output/architecture-review/: - -- final-check-ci.log: 348 tests pass, coverage 89.69%, all four configuration - schemas validate. The count is 375 minus 28 cases belonging to deleted helpers - plus one new anchor-filtering regression. No remaining behavior tests were - removed. An optional unused-type-ignore check exposed a warning inside - Pyright's bundled typeshed; that option was not retained. Final lint is recorded - separately in final-lint.log. -- focused.log: 61 geographic, sizing and assembly checks pass. -- cli-check.log: 18 CLI and batch checks pass, including failure propagation. -- verify_examples.py, example-parity.json: actual national geographic inputs - regenerate the exact saved graph (5 metros, 8 corridors). All four example - geographic configurations were first checked equal, then each scenario rebuilt - and validated. Parsed YAML and expanded node/link attributes, capacities, costs, - risks and hardware exactly match the pre-review snapshots. Device/link counts: - baseline 13/27, Clos 127/756, Dragonfly 133/739, custom Dragonfly 103/612. -- A production symbol-reference scan found no further unreferenced functions - beyond the PyYAML ignore_aliases override, which is invoked by the serializer. - This is repository-level evidence, not proof about unknown external consumers. - -No new workflow simulation, tile download or performance benchmark was needed for -this pass. Shell syntax and git diff --check pass; released changelog history is -unchanged. This review was completed before committing and opening the PR.