Skip to content

feat(pytest): use commandOverride for RunTests and DiscoverTest - #123

Merged
gnufede merged 5 commits into
mainfrom
gnufede/ddtest-dogfooding
Aug 27, 2026
Merged

feat(pytest): use commandOverride for RunTests and DiscoverTest#123
gnufede merged 5 commits into
mainfrom
gnufede/ddtest-dogfooding

Conversation

@gnufede

@gnufede gnufede commented Aug 27, 2026

Copy link
Copy Markdown
Member

What

PyTest.RunTests and DiscoverTests hardcoded python -m pytest, ignoring the commandOverride loaded from --command or DD_TEST_OPTIMIZATION_RUNNER_COMMAND. This PR makes them use commandOverride when set, so callers can run pytest (the console script) instead of python -m pytest.

Why

python -m pytest sets sys.argv[0] to pytest's __main__.py, which ddtrace's detect_service incorrectly treats as a test file path → service='pytest' instead of the module name (e.g. 'tests.contrib.graphql'). pytest (console script) sets argv[0]='pytest' which detect_service skips → correct service inference.

This is needed for the dd-trace-py ddtest dogfooding: ddtest runs python -m pytest by default, which breaks snapshot tests. With this change, dd-trace-py sets DD_TEST_OPTIMIZATION_RUNNER_COMMAND=pytest and ddtest runs pytest <files> instead.

How to test

DD_TEST_OPTIMIZATION_RUNNER_COMMAND=pytest ./bin/ddtest run -p python -f pytest --ci-node 0sys.argv[0]='pytest'detect_service skips it → correct module name.

Two new tests:

  • TestPyTest_RunTests_WithCommandOverride: verifies RunTests uses commandOverride
  • TestPyTest_DiscoverTests_WithCommandOverride: verifies DiscoverTests uses commandOverride

PyTest.RunTests and DiscoverTests hardcoded 'python -m pytest', ignoring
commandOverride (from --command or DD_TEST_OPTIMIZATION_RUNNER_COMMAND
env var). Use commandOverride when set so callers can run
'pytest' (console script) instead of 'python -m pytest'.

This fixes the service inference issue: 'python -m pytest' sets
sys.argv[0] to pytest/__main__.py, which detect_service treats as a
test path → service='pytest'. 'pytest' (console script) sets
argv[0]='pytest' (skipped by detect_service → correct service).
…ests

Two new tests:
- TestPyTest_RunTests_WithCommandOverride: verifies RunTests uses
  commandOverride (command='pytest', args=[]) instead of
  hardcoded 'python -m pytest'.
- TestPyTest_DiscoverTests_WithCommandOverride: verifies
  DiscoverTests uses commandOverride (command='pytest',
  args=['tests/test_foo.py']) instead of hardcoded
  'python -m pytest'.

Both tests pass: the mock executor captures command='pytest'
(no '-m pytest' in args) when commandOverride is set.
GetSettings() stored the raw HTTP response body (c.settingsRawResponse)
before checking the status code. On 401 (no valid API key),
the error body ('data') was stored and then written to
.testoptimization/cache/http/settings.json by StoreCacheAndExit().
The subprocess ddtrace CI Visibility plugin then read this invalid
cache and logged 'Error parsing cached settings file: data'
to stderr, breaking test_span_schematization's
'assert err == b""'.

Fix: on non-2xx status, clear settingsRawResponse and return
error before storing the raw response. Only cache successful
responses.
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Tests  Code Coverage

All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 86.47% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: deaaec3 | Docs | View more details | Give us feedback!

@gnufede gnufede changed the title Gnufede/ddtest dogfooding feat(pytest): use commandOverride for RunTests and DiscoverTest Aug 27, 2026
@gnufede
gnufede marked this pull request as ready for review August 27, 2026 08:24
@gnufede
gnufede requested a review from a team as a code owner August 27, 2026 08:24
@gnufede
gnufede merged commit c58409c into main Aug 27, 2026
19 checks passed
@gnufede
gnufede deleted the gnufede/ddtest-dogfooding branch August 27, 2026 08:27

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

A multi-part PyTest command override can share argument storage across parallel workers. Workers can replace each other's test file lists, so some tests can run twice while other tests do not run.

Open Bits AI session

🤖 Datadog Autotest · Commit deaaec3 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

args := []string{"-m", "pytest"}
if len(p.commandOverride) > 0 {
command = p.commandOverride[0]
args = p.commandOverride[1:]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Parallel workers share override argument storage

Some tests can run twice while other tests do not run.

Assertion details
  • Input: Use parallel workers with a multi-part override such as uv run pytest when the override slice has spare capacity.
  • Expected: Each worker must use a private argument slice. Clone the override arguments in RunTests before append. Apply the same safe handling in DiscoverTests, and add a multi-part parallel override test.
  • Actual: args shares the command-override backing array. Concurrent workers append different test files to this shared storage and can replace each other's arguments.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

gnufede added a commit to DataDog/dd-trace-py that referenced this pull request Aug 27, 2026
The ddtest PR (DataDog/ddtest#123) is merged. Revert to
downloading the released binary instead of building from
source (which required installing Go in the CI job).

  curl -fsSL "https://github.com/DataDog/ddtest/releases/${DDTEST_VERSION:-latest}/download/ddtest-linux-amd64" -o bin/ddtest

What: download ddtest binary from latest release.
Why: ddtest PR merged; no longer need to build from source.
E2E testing: ddtest is downloaded from the latest GitHub release.
gnufede added a commit to DataDog/dd-trace-py that referenced this pull request Aug 27, 2026
The ddtest PR (DataDog/ddtest#123) is merged to main but not yet
released. Build from main instead of downloading a release
binary (which doesn't have the commandOverride change yet).

  git clone --depth 1 https://github.com/DataDog/ddtest.git /tmp/ddtest-src
  (cd /tmp/ddtest-src && go build -o "${CI_PROJECT_DIR}/bin/ddtest" .)

Once a new release is published, revert to the curl download
with DDTEST_VERSION.

What: build ddtest from main (merged, not released).
Why: PR merged but no release yet; need the commandOverride change.
E2E testing: ddtest is built from main with the commandOverride fix.
gnufede added a commit that referenced this pull request Aug 28, 2026
PR #123 made PyTest.RunTests and DiscoverTests honor commandOverride
(set via --command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of
hardcoding 'python -m pytest'. Update settings, README, running, and
best_practices docs to describe the override and retain PYTEST_ADDOPTS
guidance for passing extra flags.
gnufede added a commit to DataDog/documentation that referenced this pull request Aug 28, 2026
ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests
honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND)
instead of hardcoding 'python -m pytest'. Update the configuration and
best_practices pages to describe the override and retain PYTEST_ADDOPTS
guidance for passing extra flags.
gnufede added a commit that referenced this pull request Aug 28, 2026
PR #123 made PyTest.RunTests and DiscoverTests honor commandOverride
(set via --command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND) instead of
hardcoding 'python -m pytest'. Update settings, README, running, and
best_practices docs to describe the override and retain PYTEST_ADDOPTS
guidance for passing extra flags.
gnufede added a commit to DataDog/documentation that referenced this pull request Aug 28, 2026
ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests
honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND)
instead of hardcoding 'python -m pytest'. Update the configuration and
best_practices pages to describe the override and retain PYTEST_ADDOPTS
guidance for passing extra flags.
gnufede added a commit to DataDog/documentation that referenced this pull request Sep 1, 2026
* docs(test_parallelization): reflect pytest --command support since 1.7.0

ddtest PR DataDog/ddtest#123 made PyTest.RunTests and DiscoverTests
honor commandOverride (--command / DD_TEST_OPTIMIZATION_RUNNER_COMMAND)
instead of hardcoding 'python -m pytest'. Update the configuration and
best_practices pages to describe the override and retain PYTEST_ADDOPTS
guidance for passing extra flags.

* Replace "Since ..." with "For versions ... and later"

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>

---------

Co-authored-by: Heston Hoffman <hestonhoffman@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants