[PERF] Reduce cold graph extraction work on large Python projects

[TristanKruse]

[PERF] Reduce cold graph extraction work on large Python projects

Problem

The current 5,000-leaf-module comparison corpus contains 5,005 Python files
and 13,738 import edges. The published end-to-end fresh-process median for
one ArchUnitPython boundary rule was 3.59 s (v1.5.0, commit
09c9d199dec095aaa60be227bfd7b5c2b7ffb107, Python 3.13.4, Windows 11).

Exploratory timing of one direct rule.check() found graph extraction taking
roughly 1.8–2.6 s, depending on machine load. Within that phase,
_extract_located_imports() accounted for about 1.2–1.7 s over 5,005 files,
and import resolution about 0.37–0.51 s over 13,738 imports. These components
are cumulative instrumentation measurements, not independent benchmark bars.

The source suggests two specific sources of avoidable work:

• Each parsed file is walked separately to find TYPE_CHECKING blocks,
  conditional import ranges, and actual imports/calls.
• Absolute imports are resolved through repeated filesystem existence checks,
  even though the scan already has the set of project Python files.

Proposed investigation and implementation

1. Add phase timings and peak memory to the regression benchmark in [Test] Benchmark for ArchUnitPython #55.
2. Try a single AST visitor that records contextual import kinds and imports
   in one traversal. Compare it with current behavior on nested conditions.
3. Build a module/file lookup from the discovered project files and use it
   for internal absolute-import resolution; retain current semantics for
   package __init__.py, relative imports, parent roots, and external modules.
4. Profile again before considering parallel parsing or a persistent disk
   cache. A disk cache needs explicit invalidation semantics and a separate
   correctness test matrix.

Acceptance criteria

• Existing tests and seeded direct-boundary/cycle correctness gates pass.
• Tests cover relative and absolute imports, TYPE_CHECKING, optional imports,
  literal dynamic imports, ignore directives, excluded paths, and malformed
  Python files.
• Report before/after fresh-process p50/p95 at 100/1,000/5,000 modules,
  extraction phase time, and peak memory. Preserve zero runtime dependencies.

[TristanKruse]

Implementation progress, now in PR #60 (not yet merged or released):

• Tried a single AST walk and a per-graph absolute-import lookup; neither produced a repeatable gain on the generated corpus, so those experiments were reverted.
• A larger win came from direct file-dependency rules: discover all project files for correct internal-target classification, but parse imports only from files selected by the rule's source filters. Parsed per-file edges are shared within the process, so later direct rules or a full-graph rule can reuse them. Clearing the graph cache also clears this session.
• Four paired in-process checks on the 5,005-file corpus parsed 1,251 files with selection versus 5,005 with the full graph. Full/selective times were 6.37/3.13, 6.73/3.14, 6.26/3.51, and 7.20/3.18 seconds. Both paths returned the same result. A separate tracemalloc diagnostic measured 12.98 MiB versus 5.25 MiB peak Python-tracked allocations; this is not total process RSS, and its instrumented timings are not suitable as speed results.
• The fresh-process ArchUnitPython harness (one warm-up, five samples per size) measured p50/p95 of 0.830/0.868 s at 100 modules, 1.008/1.194 s at 1,000, and 1.743/1.892 s at 5,000. The previous matcher-cache-only 5,000-module median was 2.826 s, so the separate-run median is 38% lower. The clean, seeded forbidden-edge, and cycle correctness gates passed.
• In the 10-additional-rule diagnostic, elapsed time fell from 7.62 to 5.58 s, with the same 7,494 violations. Full suite: 299 passed; Ruff and Mypy pass.

These were preliminary local trials on the older 1.5.0 checkout, not a claim that ArchUnitPython is fastest. PR #60 is based on current 1.7.0, includes an eight-pair alternating comparison against clean 1.7.0, and passes CI on Python 3.10–3.13. No persistent disk cache or new runtime dependency was added.