Skip to content

perf: bound parallel import extraction and trim AST traversal - #61

Open
TristanKruse wants to merge 1 commit into
perf/selective-direct-dependenciesfrom
perf/import-ast-traversal
Open

TristanKruse wants to merge 1 commit into
perf/selective-direct-dependenciesfrom
perf/import-ast-traversal

Conversation

@TristanKruse

@TristanKruse TristanKruse commented Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Stacked on #60; the base is perf/selective-direct-dependencies, so this PR contains only the next extraction optimization and its tests. No public API, parser dependency, or new runner is introduced.

  • Collect only import/context-relevant AST nodes, preserving the standard walk's relative order. Prune only known terminal node types; unknown syntax nodes still traverse their children.
  • For selections with at least 64 uncached files, overlap independent source extraction with an eight-task executor window. Do not enqueue or retain every file's results at once.
  • Consume results in the original file order. Target resolution, graph assembly, and session/cache mutation stay on the calling thread. Small and already-cached selections stay serial; executor imports are lazy.
  • Keep Python's parser, filesystem target checks, import kinds, ignore directives, and explicit cache invalidation unchanged.

Validation

  • 326 tests passed on Python 3.13.
  • 326 tests passed on Python 3.12 with concurrency forced on even for small fixtures.
  • Ruff and strict Mypy passed; git diff --check passed.
  • GitHub CI passed on Python 3.10, 3.11, 3.12, and 3.13 (tests, lint, typing, release metadata and package build).
  • Dedicated tests cover out-of-order worker completion, ordered graphs, main-thread-only resolution, selected-file-only extraction, partial/full cache reuse, no executor on cached checks, the pending-task bound, and exception propagation/executor cleanup.
  • A differential check of 1,868 standard-library/repository files preserved all 13,071 located imports, including order and classification, against the complete ast.walk reference.
  • Every fresh-process size passed clean, forbidden-import, and cycle gates in both variants. The ten-rule workload returned 7,494 violations in both variants. The real-project check preserved all 385 ordered edges and import kinds (identical graph fingerprints across every run).

Local performance gates

Eight randomized pairs per workload, Python 3.13 on the same Windows host, compared against #60 at 1a6d5ed. These are workload-specific medians, not universal speed claims. Filesystem caches were not flushed. Host load varied; absolute values must not be compared with earlier sessions.

Workload #60 This PR Change
Fresh pytest process, 100 modules 1.529 s 1.585 s +3.7%
Fresh pytest process, 1,000 modules 1.743 s 1.805 s +3.6%
Fresh pytest process, 5,000 modules 2.765 s 2.654 s -4.0%
First rule, 5,000 modules, startup excluded 1.146 s 0.967 s -15.6%
Ten additional rules, 5,000 modules 3.285 s 2.551 s -22.3%
Full graph of the same real library source 0.295 s 0.243 s -17.7%

The library-only multi-rule measurements were uninstrumented. The generated corpus contains 5,005 Python files; the direct rule selects 1,251 source files. The small fresh-process results do not show a win and are included explicitly. This optimization does not eliminate process/test-runner startup cost or establish a cross-platform performance claim. Linux/macOS performance and very large individual-file memory behavior deserve follow-up measurement; the bounded window is a task-count limit, not a byte-level memory cap.

On the same ten-rule corpus, a one-run tracemalloc diagnostic measured peak Python-tracked allocations of 10.21 MiB before and 11.06 MiB after (+0.85 MiB / 8.3%), with 7,494 violations in both variants. This is not total process RSS and does not include native thread-stack memory. The speed improvement has a small measured Python-allocation cost; it is not a memory-reduction claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant