feat(pytest): use commandOverride for RunTests and DiscoverTest - #123
Conversation
PyTest.RunTests and DiscoverTests hardcoded 'python -m pytest', ignoring commandOverride (from --command or DD_TEST_OPTIMIZATION_RUNNER_COMMAND env var). Use commandOverride when set so callers can run 'pytest' (console script) instead of 'python -m pytest'. This fixes the service inference issue: 'python -m pytest' sets sys.argv[0] to pytest/__main__.py, which detect_service treats as a test path → service='pytest'. 'pytest' (console script) sets argv[0]='pytest' (skipped by detect_service → correct service).
…ests Two new tests: - TestPyTest_RunTests_WithCommandOverride: verifies RunTests uses commandOverride (command='pytest', args=[]) instead of hardcoded 'python -m pytest'. - TestPyTest_DiscoverTests_WithCommandOverride: verifies DiscoverTests uses commandOverride (command='pytest', args=['tests/test_foo.py']) instead of hardcoded 'python -m pytest'. Both tests pass: the mock executor captures command='pytest' (no '-m pytest' in args) when commandOverride is set.
GetSettings() stored the raw HTTP response body (c.settingsRawResponse)
before checking the status code. On 401 (no valid API key),
the error body ('data') was stored and then written to
.testoptimization/cache/http/settings.json by StoreCacheAndExit().
The subprocess ddtrace CI Visibility plugin then read this invalid
cache and logged 'Error parsing cached settings file: data'
to stderr, breaking test_span_schematization's
'assert err == b""'.
Fix: on non-2xx status, clear settingsRawResponse and return
error before storing the raw response. Only cache successful
responses.
… (401)" This reverts commit 8503f12.
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: deaaec3 | Docs | View more details | Give us feedback! |
There was a problem hiding this comment.
A multi-part PyTest command override can share argument storage across parallel workers. Workers can replace each other's test file lists, so some tests can run twice while other tests do not run.
🤖 Datadog Autotest · Commit deaaec3 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
| args := []string{"-m", "pytest"} | ||
| if len(p.commandOverride) > 0 { | ||
| command = p.commandOverride[0] | ||
| args = p.commandOverride[1:] |
There was a problem hiding this comment.
Parallel workers share override argument storage
Some tests can run twice while other tests do not run.
Assertion details
- Input: Use parallel workers with a multi-part override such as
uv run pytestwhen the override slice has spare capacity. - Expected:
Each worker must use a private argument slice. Clone the override arguments in RunTests before append. Apply the same safe handling in DiscoverTests, and add a multi-part parallel override test. - Actual:
argsshares the command-override backing array. Concurrent workers append different test files to this shared storage and can replace each other's arguments.
Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
The ddtest PR (DataDog/ddtest#123) is merged. Revert to downloading the released binary instead of building from source (which required installing Go in the CI job). curl -fsSL "https://github.com/DataDog/ddtest/releases/${DDTEST_VERSION:-latest}/download/ddtest-linux-amd64" -o bin/ddtest What: download ddtest binary from latest release. Why: ddtest PR merged; no longer need to build from source. E2E testing: ddtest is downloaded from the latest GitHub release.
The ddtest PR (DataDog/ddtest#123) is merged to main but not yet released. Build from main instead of downloading a release binary (which doesn't have the commandOverride change yet). git clone --depth 1 https://github.com/DataDog/ddtest.git /tmp/ddtest-src (cd /tmp/ddtest-src && go build -o "${CI_PROJECT_DIR}/bin/ddtest" .) Once a new release is published, revert to the curl download with DDTEST_VERSION. What: build ddtest from main (merged, not released). Why: PR merged but no release yet; need the commandOverride change. E2E testing: ddtest is built from main with the commandOverride fix.
PR #123 made PyTest.RunTests and DiscoverTests honor commandOverride (set via --command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of hardcoding 'python -m pytest'. Update settings, README, running, and best_practices docs to describe the override and retain PYTEST_ADDOPTS guidance for passing extra flags.
ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of hardcoding 'python -m pytest'. Update the configuration and best_practices pages to describe the override and retain PYTEST_ADDOPTS guidance for passing extra flags.
PR #123 made PyTest.RunTests and DiscoverTests honor commandOverride (set via --command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of hardcoding 'python -m pytest'. Update settings, README, running, and best_practices docs to describe the override and retain PYTEST_ADDOPTS guidance for passing extra flags.
ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of hardcoding 'python -m pytest'. Update the configuration and best_practices pages to describe the override and retain PYTEST_ADDOPTS guidance for passing extra flags.
* docs(test_parallelization): reflect pytest --command support since 1.7.0 ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of hardcoding 'python -m pytest'. Update the configuration and best_practices pages to describe the override and retain PYTEST_ADDOPTS guidance for passing extra flags. * Replace "Since ..." with "For versions ... and later" Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com> --------- Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>
What
PyTest.RunTestsandDiscoverTestshardcodedpython -m pytest, ignoring thecommandOverrideloaded from--commandorDD_TEST_OPTIMIZATION_RUNNER_COMMAND. This PR makes them usecommandOverridewhen set, so callers can runpytest(the console script) instead ofpython -m pytest.Why
python -m pytestsetssys.argv[0]to pytest's__main__.py, which ddtrace'sdetect_serviceincorrectly treats as a test file path →service='pytest'instead of the module name (e.g.'tests.contrib.graphql').pytest(console script) setsargv[0]='pytest'whichdetect_serviceskips → correct service inference.This is needed for the dd-trace-py ddtest dogfooding: ddtest runs
python -m pytestby default, which breaks snapshot tests. With this change, dd-trace-py setsDD_TEST_OPTIMIZATION_RUNNER_COMMAND=pytestand ddtest runspytest <files>instead.How to test
DD_TEST_OPTIMIZATION_RUNNER_COMMAND=pytest ./bin/ddtest run -p python -f pytest --ci-node 0→sys.argv[0]='pytest'→detect_serviceskips it → correct module name.Two new tests:
TestPyTest_RunTests_WithCommandOverride: verifies RunTests uses commandOverrideTestPyTest_DiscoverTests_WithCommandOverride: verifies DiscoverTests uses commandOverride