Skip to content

perf: avoid unnecessary work in file dependency checks - #60

Open
TristanKruse wants to merge 6 commits into
mainfrom
perf/selective-direct-dependencies
Open

TristanKruse wants to merge 6 commits into
mainfrom
perf/selective-direct-dependencies

Conversation

@TristanKruse

@TristanKruse TristanKruse commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Speed up direct file-dependency checks without changing their results:

  • Cache source/target filter results per distinct path label for the duration of one rule check.
  • Inventory all project files for correct internal-import resolution, but parse imports only from files selected by a direct rule's source filters. Cache per-file edges within the process so later rules and full-graph checks reuse them. Empty source filters still request the full graph, and cache clearing invalidates both caches.
  • For glob filters used with search(), avoid matching a redundant leading * prefix. The original compiled regex remains available on Filter.regexp; only the internal search path uses the equivalent faster form. Custom compiled regexes are unchanged.
  • Walk each parsed syntax tree once, then reuse the node sequence to classify conditional/type-checking imports and collect imports.
  • Skip per-file evaluation of default exclusions that cannot match a .py filename; directory pruning, custom file exclusions, and .archignore still apply.

Refs #58 and #59. The reusable benchmark work remains tracked in #55.

Verification

On a generated project with four areas and a direct api → infrastructure prohibition, I alternated clean 1.7.0 and commit 2f7efbf in eight fresh-process pairs per size. Both versions passed the clean, seeded forbidden-import, and cycle correctness gates before timing:

Leaf modules 1.7.0 median 2f7efbf median Change Faster pairs
100 0.831 s 0.784 s -5.6% 6/8
1,000 1.357 s 0.878 s -35.3% 8/8
5,000 3.559 s 1.349 s -62.1% 8/8

The final exclusion change was then measured independently against 2f7efbf, again in eight fresh-process pairs with the same correctness gates:

Leaf modules 2f7efbf median Current branch median Change
100 0.790 s 0.778 s -1.5%
1,000 0.903 s 0.838 s -7.2%
5,000 1.391 s 1.206 s -13.3%

These are local Windows 11 / Python 3.13.4 results for one synthetic direct-boundary rule, not a universal ranking. At 5,000 leaf modules, selective extraction parses 1,251 of the 5,005 project files. The two tables are separate paired experiments; their absolute times should not be combined across runs.

  • Local full suite: 313 passed on Python 3.12 and 3.13; Ruff and Mypy pass.
  • Tests cover partial/full graph equivalence in both call orders, namespace-package aliases, import-kind merging, type-only/conditional/dynamic imports, combined source filters, empty filters, no matches, explicit exclusions, .archignore, cache invalidation, and preservation of glob search and Filter identity semantics.
  • Full graph edge counts and hashes matched clean 1.7.0 on the generated 1,000-module project and the sample-project fixture.

Change Type

  • Refactoring

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant