Skip to content

feat: warn when running on an unsupported Python version - #133

Merged
not-matthias merged 1 commit into
masterfrom
cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python
Sep 4, 2026
Merged

feat: warn when running on an unsupported Python version#133
not-matthias merged 1 commit into
masterfrom
cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python

Conversation

@not-matthias

Copy link
Copy Markdown
Member

Print a warning in the pytest header when the running interpreter is outside the
Python version range pytest-codspeed claims to support:

WARNING: Python 3.16 is not officially supported by pytest-codspeed (supported:
3.9 to 3.15). Support is experimental and untested, measurements may be unreliable.

Customers upgrading Python currently get silent, unvalidated measurements: nothing
tells them the interpreter they benchmark on was never tested. Surfacing it in the
header means it lands in the CI log right under the existing codspeed: <version>
line, next to the other notices.

The bounds live in pytest_codspeed.utils and mirror requires-python plus the
Programming Language :: Python classifiers in pyproject.toml and the CI matrix.
The issue did not specify a list, and the sources disagreed: the classifiers and CI
say 3.15, while the README badge says 3.14 and the docs badge 3.13. The badges are
stale (docs still claimed 3.13 well after 3.14 classifiers shipped), so the bounds
follow the classifiers. Warning on 3.15 would contradict the package's own metadata
and redden the 3.15 CI legs.

test_get_python_version_warning_supported asserts no warning fires on the
interpreter running the suite, so adding a Python version to the CI matrix without
bumping the constant fails loudly instead of silently warning users.

Not included: the "warning on the Run page" half of the issue is platform-side. The
plugin already ships full interpreter metadata via get_environment_metadata and
collect_and_write_python_environment, so no extra plumbing is needed here.

Refs COD-3400

@codspeed-hq

codspeed-hq Bot commented Aug 27, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 1.17%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 16 improved benchmarks
❌ 23 (👁 23) regressed benchmarks
✅ 238 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Memory test_multiprocessing_map[10] 4.1 MB 3.2 MB +27.26%
WallTime test_multiprocessing_map[1000] 59.2 ms 49.8 ms +18.96%
WallTime test_multiprocessing_map[100000] 419.4 ms 364.9 ms +14.96%
WallTime test_sum_of_squares[sum_of_squares_sum_labmda_power] 350.4 µs 312.5 µs +12.13%
WallTime test_multiprocessing_map[10000] 59.8 ms 56.2 ms +6.36%
WallTime test_sudoku[initial_grid0] 8.3 µs 7.9 µs +4.97%
WallTime test_open_knight_tour[1] 4.3 µs 4.1 µs +4.66%
Memory test_multiprocessing_map[100000] 6.2 MB 6 MB +3.63%
WallTime test_open_close_fd[10] 96.8 µs 93.4 µs +3.61%
WallTime test_make_lowpass 5.6 µs 5.5 µs +2.75%
WallTime test_make_peak 6.4 µs 6.3 µs +2.7%
WallTime test_hamilton_cycle[graph0] 25.5 µs 24.8 µs +2.56%
WallTime test_color[graph0-3] 31.4 µs 30.7 µs +2.39%
WallTime test_open_close_fd[100] 901.6 µs 880.6 µs +2.37%
WallTime test_recursive_fibo_20 5.3 ms 5.2 ms +2.1%
WallTime test_pipe_communication[100] 6.3 ms 6.2 ms +2.09%
👁 WallTime test_noop_pass_decorated 658.4 ns 804.6 ns -18.16%
👁 WallTime test_make_lowshelf 7.3 µs 7.7 µs -5.18%
👁 WallTime test_make_highpass 5.5 µs 5.7 µs -3.82%
👁 WallTime test_generate_sum_of_subsets_soln[nums0-9] 26.1 µs 29.3 µs -10.98%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python (14b0bb9) with master (c7e9981)

Open in CodSpeed

@not-matthias
not-matthias force-pushed the cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python branch from e09ec2d to 2345b56 Compare August 27, 2026 16:16
@not-matthias

Copy link
Copy Markdown
Member Author

@greptileai review

@not-matthias
not-matthias marked this pull request as ready for review August 27, 2026 16:26
@not-matthias
not-matthias force-pushed the cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python branch from 2345b56 to 4e49265 Compare August 27, 2026 16:30
@not-matthias
not-matthias marked this pull request as draft August 27, 2026 16:32
@greptile-apps

greptile-apps Bot commented Aug 27, 2026

Copy link
Copy Markdown

Greptile Summary

The PR adds a pytest-header notice when the active interpreter falls outside the supported Python 3.9–3.15 range.

  • Defines supported-version bounds and formats local or GitHub Actions warnings.
  • Adds unit and pytest integration coverage for supported, unsupported, and annotation output.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
src/pytest_codspeed/plugin.py Adds supported-version bounds, warning formatting, and inclusion of the warning in pytest's report header.
tests/test_pytest_plugin.py Adds tests for supported and unsupported versions, GitHub Actions annotations, and header integration.

Reviews (2): Last reviewed commit: "feat: warn when running on an unsupporte..." | Re-trigger Greptile

Comment thread src/pytest_codspeed/utils.py Outdated
@not-matthias
not-matthias marked this pull request as ready for review August 27, 2026 17:08

@GuillaumeLagrange GuillaumeLagrange left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could use a hint of deslop for the tests, olgtm.

I wont highjack this one for the perf trampoline, let's merge this I'll make another PR

Comment thread src/pytest_codspeed/plugin.py Outdated
Comment thread src/pytest_codspeed/plugin.py Outdated
Comment thread src/pytest_codspeed/plugin.py Outdated
Comment thread tests/test_pytest_plugin.py Outdated
Print a warning in the pytest header when the interpreter is outside the version
range pytest-codspeed claims to support, so customers upgrading Python are told
their measurements run on an untested interpreter instead of getting silent,
unvalidated results.

On GitHub Actions the warning is emitted as a `::warning` workflow command so it
becomes an annotation shown outside the job log; on GitLab CI, which has no
annotation mechanism, the line is coloured instead. Mirrors the `warnCi` helper
of the codspeed-node counterpart, including the percent-encoding of
workflow-command syntax.

The bounds mirror `requires-python` and the `Programming Language :: Python`
classifiers in pyproject.toml, and the CI matrix.

Refs COD-3400
@not-matthias
not-matthias force-pushed the cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python branch from 65a48fb to 14b0bb9 Compare September 4, 2026 14:28
@not-matthias
not-matthias merged commit 14b0bb9 into master Sep 4, 2026
57 checks passed
@not-matthias
not-matthias deleted the cod-3400-warn-when-running-pytest-codspeed-with-unsupported-python branch September 4, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants