Simplify topology generation and remove legacy code

CHANGELOG.md

@@ -9,6 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Diagnostics**: Report metro anchors removed by highway filtering as a configuration/data error; correct ring-mesh, representative-point and geometry-cleanup descriptions. Stop inserting heuristic YAML section comments that could label the wrong links.
+- **Traffic**: Keep top-K gravity partners independent of metro-name ordering; seed jitter locally and distribute uniform traffic between DC sites rather than individual devices.
+- **Striping**: Respect one-to-one site mode and endpoint matches; identify devices by full literal paths and preserve disabled risk-group settings.
+- **Config**: Reject fractional/duplicate priority keys, duplicate metro overrides, unknown striping fields and unmatched forced metro patterns. Apply configured link cost as a minimum without changing geographic distances.
+- **Blueprints**: Include recursive dependencies and reject cycles; derive diagrams from resolved devices instead of a duplicate DSL interpreter.
+- **Geography**: Filter urban areas using their actual CRS and return YAML-safe coordinates for direct Python generation/build calls.
+- **Validation**: Reject non-string device roles and non-finite chassis capacities.
+- **Tools**: Return failure from batch builds when any configuration fails; analyze vertex connectivity and k-cores on simple metro adjacency while preserving corridor counts.
+
+- **Traffic**: Reject infeasible or unconverged hose fits; preserve small demands when rounding is disabled.
+- **Validation**: Audit each demand set against expanded external DC capacity; resolve regex selectors, report unmatched demands, and handle malformed network structures without crashing.
+- **Contracts**: Preserve exact integer strings; reject non-finite config values, non-metre/geographic CRS and duplicate metro identities; check PoP coordinates without DC groups.
+- **Components**: Collect blueprint component dependencies after hardware overrides, removing replaced platform references.
 - **TM Sizing**: Split uniform traffic over ordered DC pairs, including same-metro pairs, to match NetGraph demand expansion.
 - **TM Sizing**: Size corridors for peak directional load even when `respect_min_base_capacity` is disabled.
 - **TM Sizing**: Preserve separate corridors with repeated edge keys across different PoP pairs.
@@ -18,9 +31,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Workflows**: Bind built-in demand references to `traffic.matrix_name`.
 - **Failures**: Make the `empty` policy a valid no-failure mode.
 - **Docs**: Install from GitHub; the `topogen` name on PyPI belongs to an unrelated project.
+- **Geography**: Preserve alternate highway routes, isolated rings and all `k_paths` through corridor JSON and site expansion; form nearest-neighbor pairs independently of metro ID order. Resolve risk distance from its owning path and reject unknown risk references.
+- **Hardware**: Resolve optic counts for each site edge and endpoint; preserve caller link attributes and geographic corridor metadata.
+- **Validation**: Preserve node rules in expansion audits, require scenario-local component definitions, and audit declared optic port usage.
+- **CLI**: Validate before atomically replacing scenario YAML, including `--print`; scope quiet output to the command.
 
 ### Changed
 
+- **Internal**: Remove duplicate corridor metadata and intermediate edge copying; decouple site identifiers from device expansion and consolidate sizing capacity passes.
+- **Checks**: Detect unused Ruff suppressions; make CLI tests verify successful quiet dispatch and required exception propagation.
+- **Typing**: Describe the Contextily API used by TopoGen in a scoped local stub; keep missing-stub diagnostics enabled.
+- **Internal**: Share traffic pair selection/rounding/emission, skip disabled debug formatting, route aggregated metro-pair demand once, reuse finalized blueprint libraries and build workflows once.
+- **Internal**: Separate site construction, sizing and serialization; centralize schema bounds and metro naming. Remove the unused metro-loader formatting argument; configuration dataclasses reject undeclared attributes.
+
 - **BREAKING**: **Scenarios**: Emit NetGraph `demands` and `failures` sections; update consumers of generated YAML.
 - **BREAKING**: **Failures**: Custom rules must use `scope`, `mode` and `match`; migrate `lib/failure_policies.yml`.
 - **BREAKING**: **Blueprints**: Link selectors in `lib/blueprints.yml` use NetGraph `$var`/`${var}` placeholders; `{var}` is no longer expanded.
@@ -32,13 +55,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Workflows**: Use `parallelism: auto` in built-in and sample workflows.
 - **Validation**: Construct the full NetGraph scenario to check workflow arguments and failure policies before topology audits.
 - **Internal**: Remove redundant validation and scenario code, the PyPI publish workflow, the pre-commit Pyright hook and benchmark flags.
+- **BREAKING**: **Contracts**: Require corridor `MultiGraph` JSON with explicit edge keys; regenerate saved graphs. Use `topogen.scenario.build_scenario`, `RunContext` and public visualization settings.
+- **BREAKING**: **Config**: Remove ignored metadata, road-processing controls, `edge_select`, component assignments/library fields and scenario overrides; reject invalid integer quantities.
+- **BREAKING**: **Optics**: Require explicit `local->remote` endpoint mappings; remove implicit reverse and legacy delimiter behavior. Link role selectors use `role|role` strings.
+- **Assembly**: Generate traffic and expand the device network once per build; reuse the result for budgets, hardware and artifacts.
+- **Internal**: Consolidate link parsing and user-library loading; remove format guessing, swallowed computation/render errors and duplicated graph copies.
 
 ### Added
 
 - **Superset**: Add setup, teardown and run commands to install dev dependencies, copy local `.env` files and run checks.
 
 ### Removed
 
+- **BREAKING**: Remove unused point/bearing utilities, `MetroCluster` distance/overlap/array helpers, `TopologyConfig.summary()` and singular/list/filter component helpers. Use the merged `get_builtin_components()` mapping and `topogen info`.
 - **Dependencies**: Unused `seaborn`, `rich`, `nbformat`, `nbconvert`, `ipykernel`, `itables`, `rasterio` and `scikit-learn` requirements.
 
 ## [0.4.0] - 2026-03-15

Makefile

@@ -126,11 +126,7 @@ qt:
 
 validate:
 	@echo "📋 Validating TopoGen config YAMLs..."
-	@if $(PYTHON) -c "import jsonschema" >/dev/null 2>&1; then \
-		$(PYTHON) -c "import json, yaml, jsonschema, pathlib; from importlib import resources as res; f=res.files('topogen.schemas').joinpath('topogen_config.json').open('r', encoding='utf-8'); schema=json.load(f); f.close(); cfg_dirs=['examples']; cfg_files=[]; [cfg_files.extend(list(pathlib.Path(d).rglob('*.yaml'))+list(pathlib.Path(d).rglob('*.yml'))) for d in cfg_dirs if pathlib.Path(d).exists()]; [jsonschema.validate(yaml.safe_load(open(fp)), schema) for fp in cfg_files]; print(f'✅ Validated {len(cfg_files)} TopoGen config YAML files')"; \
-	else \
-		echo "⚠️  jsonschema not installed. Skipping schema validation"; \
-	fi
+	@$(PYTHON) -c "from pathlib import Path; from topogen.config import TopologyConfig; files=sorted(p for p in Path('examples').rglob('*') if p.suffix in {'.yml', '.yaml'}); [TopologyConfig.from_yaml(p) for p in files]; print(f'Validated {len(files)} TopoGen configs')"
 
 # Build and Package
 build:

README.md

@@ -45,12 +45,13 @@ ngraph run output/config_scenario.yml
 ```
 
 `generate` writes `output/config_integrated_graph.json`, a metro-to-metro corridor
-graph, and a preview JPEG. `build` reads the graph and writes
+multigraph. Preview JPEGs are written when their export settings are enabled. `build` reads the graph and writes
 `output/config_scenario.yml`. File prefixes come from the config filename.
 
 `build` checks the schema, NetGraph Scenario construction, topology, and hardware.
 `ngraph run` executes the workflows. Add `--print` to `topogen build` to also print
-the YAML and skip scenario validation.
+the validated YAML. A failed build preserves the previous scenario YAML; successful
+writes replace it atomically.
 
 To process a folder of configs, use `./build.sh examples output`. It reuses saved
 graphs; pass `--force` to regenerate them or `--build-only` to rebuild scenarios.
@@ -60,18 +61,34 @@ See `./build.sh --help` for filename filters.
 
 ```python
 from pathlib import Path
-from topogen import TopologyConfig, build_integrated_graph, save_to_json
+from topogen import RunContext, TopologyConfig, build_integrated_graph, save_to_json
+from topogen.scenario import build_scenario
+from topogen.validation import validate_scenario_yaml
 
 config = TopologyConfig.from_yaml(Path("config.yml"))
-graph = build_integrated_graph(config)
+context = RunContext(Path("output"), stem="config")
+graph = build_integrated_graph(config, context=context)
 save_to_json(
     graph,
-    Path("output/config_integrated_graph.json"),
+    context.path("integrated_graph.json"),
     config.projection.target_crs,
     config.output.formatting,
 )
+scenario_yaml = build_scenario(graph, config, context=context)
+issues = validate_scenario_yaml(
+    scenario_yaml,
+    hw_component_map=config.components.hw_component,
+    optics_map=config.components.optics,
+)
+assert not issues, issues
 ```
 
+Without `context`, the Python generation/build functions return their results
+without exporting files. Configuration holds topology and visualization settings;
+`RunContext` holds output paths, the filename stem and the optional debug directory.
+The CLI supplies this context automatically. `--debug-dir` exports the exact traffic
+matrices used for sizing and emission, for all traffic models.
+
 ## Configuration
 
 Configs are checked against [the JSON schema](topogen/schemas/topogen_config.json).
@@ -95,8 +112,83 @@ workflow arguments when the library is loaded.
 Traffic models are `uniform`, `gravity`, and `hose`. With `build.tm_sizing.enabled`,
 TopoGen routes traffic on a collapsed metro graph and sizes corridors for the
 larger directional load, then applies headroom and capacity increments. Uniform
-traffic includes same-metro DC pairs in the total split. Sizing models one demand
-node per DC region and rejects unsupported endpoint patterns.
+traffic uses `group_mode: group_pairwise`: each ordered pair of distinct DC sites
+receives an equal share, including same-metro sites. Device counts within a DC do
+not change its share. Sizing aggregates traffic by ordered metro pair before
+routing and rejects unsupported endpoint patterns.
+
+### Current contracts and migration
+
+There is one supported format at each stage; obsolete inputs are rejected.
+
+- Regenerate saved graphs with `topogen generate`: JSON requires
+  `graph_type: corridors` and an explicit key for every parallel edge. The Python
+  build API accepts an undirected `networkx.MultiGraph` of metro nodes only.
+  Each metro supplies `name`, `name_orig`, `metro_id`, `x`, `y` and `radius_km`.
+  Names and IDs must be unique nonempty strings; coordinates must be finite.
+  The target CRS must be projected with metre units.
+- Every discovered path survives extraction, JSON and site expansion. Chain
+  contraction preserves alternate routes and both arcs of a ring. Risk tags are
+  aggregated along each path, including risks shared with other paths.
+- Import `build_scenario` from `topogen.scenario`; `topogen.scenario_builder` has
+  been removed. Replace private configuration output attributes with `RunContext`
+  and public `config.visualization` fields.
+- Use `get_builtin_components()` and ordinary dictionary lookup/filtering for the
+  effective component library. The unused singular/list/filter helpers,
+  `TopologyConfig.summary()`, standalone point/bearing helpers, and unused
+  `MetroCluster` distance/overlap/array helpers have been removed. `topogen info`
+  remains the configuration summary command.
+- Delete `highway_processing.min_cycle_nodes`, `validation_sample_size`,
+  `build.tm_sizing.edge_select`, `output.scenario_metadata`,
+  `components.assignments`, `components.library`, and failure/workflow
+  `assignments.scenario_overrides`. These settings did not affect generation.
+  Library definitions belong in the corresponding `lib/*.yml` file.
+- Link `role_pairs` use strings such as `core|leaf`. Optics mappings explicitly
+  name the **local and remote roles**, using `local->remote`. Specify both
+  directions when both ends need optics; the module types may differ:
+
+  ```yaml
+  components:
+    optics:
+      leaf->spine: 800G-DR4
+      spine->leaf: 1600G-2xDR4
+  ```
+
+  There is no implicit reverse mapping. Old `a-b` and `a|b` optics keys are rejected.
+- Scenario hardware definitions must be present in the scenario's `components`.
+  The builder includes components referenced by configured assignments and used
+  blueprints. Audits never substitute the local built-in component library.
+  Port audits count declared modules; optics audits check their capacity.
+- Capacity/cost integer inputs must be finite and integral; booleans and fractional
+  values are errors. Integer strings retain exact precision, and non-finite
+  configuration numbers are rejected. Unknown routing policies and malformed library files fail
+  explicitly. Optional libraries may be absent, but a present file must be a mapping.
+
+Link `cost` is a minimum routing cost; the geometric distance remains separately
+available as `distance_km`. Striping accepts only `width` (with optional
+`mode: width`) or `mode: by_attr` plus `attribute`. It respects `one_to_one`,
+endpoint matches and full device paths, including literal regex metacharacters.
+Duplicate normalized priorities or metro overrides, unknown striping options and
+unmatched forced metro patterns are errors. Nested blueprint dependencies are
+included recursively; reference cycles are rejected.
+
+Assembly generates traffic once and expands the complete device network once.
+Capacity and optics are resolved per site edge against that expansion. Validation
+constructs one full NetGraph scenario; coverage audits additionally expand tagged
+rules and probe blueprint definitions, retaining node/link rules and variables.
+Risk definitions use the length of their owning corridor path; references without
+a corresponding path are errors.
+DC capacity audits resolve NetGraph selectors and check each demand set separately
+against external device-link capacity. `run_ngraph=False` checks metadata and
+references only. Combined demand endpoints spanning multiple DCs require workflow
+execution to determine their per-DC placement.
+Hose generation rejects infeasible margins or a fit that does not converge.
+With `rounding_gbps: 0`, gravity and hose preserve unquantized demand values.
+Gravity jitter uses the scenario seed without changing Python's global PRNG.
+Blueprint diagrams aggregate the already expanded devices and link capacities;
+they do not implement a second version of the NetGraph DSL.
+Map rendering uses network tiles through Contextily and may dominate wall time;
+render failures propagate to the caller.
 
 ## Development
 

build.sh

@@ -106,17 +106,8 @@ mkdir -p "$OUTPUT_DIR_RAW" || die "Cannot create output directory: $OUTPUT_DIR_R
 CONFIGS_DIR=$(abs_path "$CONFIGS_DIR_RAW")
 OUTPUT_DIR=$(abs_path "$OUTPUT_DIR_RAW")
 
-# Detect TopoGen invoker once: prefer installed CLI; otherwise python -m.
-TOPGEN_INVOKE=(topogen)
-if ! command -v "${TOPGEN_INVOKE[0]}" >/dev/null 2>&1; then
-  if command -v python3 >/dev/null 2>&1; then
-    TOPGEN_INVOKE=(python3 -m topogen)
-  elif command -v python >/dev/null 2>&1; then
-    TOPGEN_INVOKE=(python -m topogen)
-  else
-    die "Neither 'topogen' nor Python found on PATH. Activate your venv or install TopoGen."
-  fi
-fi
+# Use the installed CLI from the active environment.
+command -v topogen >/dev/null 2>&1 || die "Activate the TopoGen venv or install TopoGen."
 
 echo "🚀 TopoGen batch run"
 OUTPUT_DIR_PRINT=${OUTPUT_DIR%/.}
@@ -213,11 +204,10 @@ while IFS= read -r -d '' cfg; do
 
     if [[ "$run_generate" == "true" ]]; then
       # Run generate writing artefacts directly to workdir via -o
-      ("${TOPGEN_INVOKE[@]}" generate "$cfg_abs" -o "$workdir") 2>&1 | tee "$workdir/generate.log"
+      topogen generate "$cfg_abs" -o "$workdir" 2>&1 | tee "$workdir/generate.log"
       gen_ec=${PIPESTATUS[0]}
     else
       # Use cached artefacts
-      : # artefacts already in workdir from previous run
       echo "⏭️  Skipping generate: found existing ${stem}_integrated_graph.json" | tee "$workdir/generate.log" >/dev/null
       gen_ec=100  # special code for 'cached'
     fi
@@ -238,27 +228,24 @@ while IFS= read -r -d '' cfg; do
   fi
 
   # Build after successful generation or when a saved graph is available.
-  build_icon="⏭️"
-  build_note="skipped"
   build_ec=-1
   scenario_out="$workdir/${stem}_scenario.yml"
   if [[ $gen_ec -eq 0 || $gen_ec -eq 100 ]]; then
     # Run build writing scenario into workdir via -o
-    ("${TOPGEN_INVOKE[@]}" build "$cfg_abs" -o "$scenario_out") 2>&1 | tee "$workdir/build.log"
+    topogen build "$cfg_abs" -o "$scenario_out" 2>&1 | tee "$workdir/build.log"
     build_ec=${PIPESTATUS[0]}
     case "$build_ec" in
       0)
-        build_icon="✅"; build_note="ok"; build_ok=$((build_ok + 1));;
+        build_ok=$((build_ok + 1));;
       3)
-        build_icon="⚠️"; build_note="validation failed"; build_validation=$((build_validation + 1));;
+        build_validation=$((build_validation + 1));;
       2)
-        build_icon="❌"; build_note="config error"; build_config=$((build_config + 1));;
+        build_config=$((build_config + 1));;
       1)
-        build_icon="❌"; build_note="runtime error"; build_runtime=$((build_runtime + 1));;
+        build_runtime=$((build_runtime + 1));;
       *)
-        build_icon="❌"; build_note="exit $build_ec"; build_other=$((build_other + 1));;
+        build_other=$((build_other + 1));;
     esac
-    :
   else
     build_skipped=$((build_skipped + 1))
   fi
@@ -269,7 +256,6 @@ while IFS= read -r -d '' cfg; do
   if [[ $gen_ec -eq 100 ]]; then gen_col="⏭️ cached"; fi
   if [[ $gen_ec -ne 0 && $gen_ec -ne 2 && $gen_ec -ne 100 ]]; then gen_col="❌ runtime"; fi
 
-  build_col="$build_icon"
   case "$build_ec" in
     0) build_col="✅" ;;
     3) build_col="⚠️ validation" ;;
@@ -314,4 +300,8 @@ echo "   • Generate: ✅ $gen_ok  | ⏭️ $gen_cached (cached)  | ❌ $gen_fa
 echo "   • Build:    ✅ $build_ok  | ⚠️ $build_validation (validation)  | ❌ $build_runtime (runtime)  | ❌ $build_config (config)  | ⏭️ $build_skipped (skipped)  | ❓ $build_other (other)"
 echo "======================"
 
+if (( build_ok != total )); then
+  echo "Batch failed: $((total - build_ok)) config(s) did not build successfully." >&2
+  exit 1
+fi
 echo "Done. ✨"