fix(ci): run each matrix leg on its own Python version

[runpod-Henrik]

The quality-gates matrix declares four Python versions but ran 3.11 on every leg. This makes the matrix real, and fixes the three tests that turned out to be broken on 3.10 once 3.10 actually ran.

Updated 2026-09-22 after merging main. This PR originally carried a second, independent fix for a leaked test thread. #371 has since landed a different fix for that same leak, so that part is gone — see Dropped from this PR below. The matrix fix is unaffected.

1. The Python version matrix was decorative

quality-gates declares ['3.10', '3.11', '3.12', '3.13'], but every leg ran Python 3.11. make dev runs a bare uv sync, and uv resolves its interpreter from .python-version (pinned to 3.11) rather than from the one actions/setup-python just installed.

Confirmed across four CI runs on three unrelated branches, the oldest from 2026-08-10 — every leg reports Using CPython 3.11.x and builds .venv/lib/python3.11:

Run	Branch	Leg that failed	Interpreter
31410374670	deanq/sls-python-parity-by-default	Quality Gates (3.10)	Python 3.11.15
32861648254	fix/367-app-delete-removes-endpoint	Quality Gates (3.12)	Python 3.11.16
32869479106	fix/365-deploy-empty-resources	Quality Gates (3.12)	Python 3.11.16
33012747552	Henrik/ci-coverage-artifact	Quality Gates (3.13)	Python 3.11.16

Fix: UV_PYTHON: ${{ matrix.python-version }}, which takes precedence over the pin file. Set at job level rather than on the install step, because make ci-quality-github runs uv run pytest, which resolves the interpreter again.

uv.lock was stale, and 3.13 could not resolve without it

Re-locked in the same commit. On main it is wrong in two ways:

• requires-python = ">=3.10, <3.13" while pyproject.toml says >=3.10,<3.14 — the lock never covered 3.13 at all
• tomlkit>=0.13.0 is declared in pyproject.toml, but the lock's runpod-flash entry omits it from both dependencies and requires-dist (the package block itself is present, pulled in by another dependency)

uv lock --check   on main        -> "The lockfile at `uv.lock` needs to be updated"
uv lock --check   on this branch -> Resolved 136 packages, clean

CI never caught the drift: pre-check uses --frozen, which does not verify freshness, and make dev uses plain uv sync, which silently re-resolves on the runner.

2. Three tests only ever passed because they only ever ran on 3.11

Turning the matrix on is what surfaced these. All three pass on 3.11+ and fail on 3.10, so they were green for as long as the matrix was decorative. Each failure is a version-dependent difference in how mocks behave, not a product bug.

test_run_server_helpers.py — test_allows_plain_dict_body, test_allows_empty_dict_body

call_with_body → _map_body_to_params calls inspect.signature(func), and signature() of a Mock is version-dependent: on 3.10 it raises TypeError: 'Mock' object is not subscriptable (Mock auto-creates a __signature__ child, which inspect then tries to use), while on 3.11+ it reports (*args, **kwargs). Passing spec= does not help — the failure is in signature(), not in spec resolution.

call_with_body catches Exception and converts it into a 500 JSONResponse, so on 3.10 these failed as assert <JSONResponse object> == {'ok': True}, with the real TypeError visible only inside the response body.

Fix: wrap the mock in a real async def (*args, **kwargs) — the same signature 3.11+ inferred from the Mock, so both branches of _map_body_to_params behave exactly as before, and delegating to the mock keeps the call assertions.

test_resource_manager_extended.py — the legacy-state-file test

Whether cloudpickle can pickle a Mock is version-dependent; on 3.10 this failed with Could not pickle object as excessively deep recursion required. Replaced the MagicMock with a module-level _LegacyResource stub. Only .config_hash is read here (via _refresh_config_hashes), and the absence of get_resource_key is what keeps the legacy key un-migrated — which is exactly what the test asserts. Module-level rather than defined in the test body, because cloudpickle serialises a function-local class by value, which is the thing being avoided.

Dropped from this PR

The original second half of this PR fixed test_deploy_all_background leaking a daemon thread into unrelated tests (surfacing as _pickle.PicklingError: args[0] from __newobj__ args has the wrong class on whichever test happened to be running). It stubbed threading.Thread.

#371 has since landed on main with a different fix for the same leak — it joins the real thread and additionally asserts mock_deploy.await_count == 3. Merging main conflicted on exactly that test, and this PR takes main's version: it asserts strictly more, and a CI-plumbing PR should not revert a landed fix. tests/unit/test_deployment.py is now byte-identical to main.

The commit that introduced the stub is still in this branch's history; its effect is reverted by the merge, so the net diff against main contains no change to that test.

Verification

Run on the merge commit:

Check	Result
make test-coverage, parallel pass	2692 passed, 1 skipped, 1 xfailed
make test-coverage, serial pass (-m serial)	53 passed, 1 skipped
Exit status	0 — coverage 86.59% against the 65 gate
ruff check + ruff format --check, 260 tracked files	clean
uv lock --check	clean

And on CI, the check that actually matters for this fix — each leg now reports a different interpreter, where previously all four reported 3.11.x (run 35759535135):

Leg	Interpreter
Quality Gates (3.10)	Using CPython 3.10.21
Quality Gates (3.11)	Using CPython 3.11.16
Quality Gates (3.12)	Using CPython 3.12.14
Quality Gates (3.13)	Using CPython 3.13.15

All four pass, which is also the first evidence that the suite is green on 3.10, 3.12 and 3.13 at all.

Not addressed here

Two deeper faults that made the leaked-thread bug possible are still present — #371 fixed the symptom in the test, not these:

• ResourceManager._resources / _resource_configs are class variables, so instance mutation writes through to global state — while _load_resources() uses assignment, which shadows them with instance attributes. The manager silently switches between class-level and instance-level state depending on whether the state file existed.
• conftest.worker_flash_dir is scope="session", so every test in a worker shares one resources.pkl, and reset_singletons forces a reload from it on every test.

🤖 Generated with Claude Code

[runpod-Henrik]

What the real matrix found on 3.10, and the fix

Making the matrix real (259c527) turned the 3.10 leg into an actual 3.10 run for the first time, and it failed immediately:

Using CPython 3.10.21
platform linux -- Python 3.10.21
= 3 failed, 2627 passed, 1 skipped, 1 xfailed in 127.20s

All three failures are pre-existing and 3.10-specific — nothing to do with the matrix change itself, which only stopped hiding them. All three reproduce on a real 3.10 interpreter independently of this branch. 0625cc9 fixes them; both are test-only changes.

1 & 2. test_run_server_helpers.py — inspect.signature() on a Mock

FAILED TestCallWithBodyEmptyInputValidation::test_allows_plain_dict_body
       AssertionError: assert <starlette.responses.JSONResponse object> == {'ok': True}
FAILED TestCallWithBodyEmptyInputValidation::test_allows_empty_plain_dict_body

call_with_body → _map_body_to_params calls inspect.signature(func), and signature() of a Mock is version-dependent:

Python	inspect.signature(AsyncMock())
3.10.19	raises TypeError: 'Mock' object is not subscriptable
3.13.11	returns (*args, **kwargs)

Mock auto-creates a __signature__ child attribute, which inspect then tries to use. Passing spec= does not help — verified; the failure is inside signature(), not in spec resolution.

This was double-masked. call_with_body catches Exception and converts it into a 500 JSONResponse, so the visible failure was the uninformative assert <JSONResponse object> == {'ok': True}. The real TypeError was readable only inside the response body:

returned: JSONResponse
body: {"error":"'Mock' object is not subscriptable"}
_map_body_to_params raised TypeError: 'Mock' object is not subscriptable   # py3.10
_map_body_to_params -> {'args': {'key': 'value'}}                         # py3.13

Fix: wrap the mock in a real async def (*args, **kwargs). That is the same signature 3.11+ inferred from the Mock, so both branches of _map_body_to_params behave exactly as before — the non-empty body maps to the first parameter, the empty body spreads as kwargs — and delegating to the mock keeps assert_called_once_with().

3. test_resource_manager_extended.py — pickling a Mock

FAILED TestLoadResources::test_loads_legacy_dict_format
       _pickle.PicklingError: Could not pickle object as excessively deep recursion required.

The test does cloudpickle.dump({"key1": MagicMock(config_hash="hash1")}). Whether cloudpickle can pickle a Mock is version-dependent; 3.10 rejects it.

Fix: a module-level _LegacyResource stand-in. Only .config_hash is read (via _refresh_config_hashes), and the absence of get_resource_key is what keeps the legacy key un-migrated — which is exactly what the test asserts. Module-level rather than defined in the test body, because cloudpickle serialises a function-local class by value.

Verification

CI on 0625cc9 — the first run in this repo's history where all four legs use four different interpreters:

Leg	Result
Quality Gates (3.10)	pass
Quality Gates (3.11)	pass
Quality Gates (3.12)	pass
Quality Gates (3.13)	pass
Validation	pass

Locally, before pushing, both affected files were run against three real interpreters:

Python	Result
3.10	40 passed (previously 3 failed)
3.11	40 passed
3.13	40 passed

ruff format --check and ruff check clean.

Worth a separate look

The blanket except Exception: return JSONResponse(status_code=500, ...) in call_with_body turns genuine errors into silent 500s. It is what made a one-line TypeError take three steps to diagnose, and in production it would do the same to a real bug. Not changed here — it is behaviour, not a test fix.