Skip to content

e2e_tests.md

Latest commit

 

History

History
128 lines (103 loc) · 6.55 KB

e2e_tests.md

File metadata and controls

128 lines (103 loc) · 6.55 KB
Raw

End-to-End Tests

End-to-end tests exercise the running MPT API and cover the full request/response lifecycle. They live under tests/e2e/ and rely on live credentials and configurable endpoints.

Directory Layout

tests/
└── e2e/
    ├── conftest.py      # Shared E2E fixtures, one client per token scope
    ├── helper.py        # Resource setup and teardown helpers shared across domains
    ├── test_access.py   # Authentication and access coverage, not tied to one domain
    ├── accounts/
    ├── audit/
    ├── billing/
    ├── catalog/
    ├── commerce/
    ├── exchange/
    ├── helpdesk/
    ├── integration/
    ├── notifications/
    ├── program/
    ├── spotlight/
    └── streaming/

conftest.py builds one client fixture per token scope — mpt_vendor, mpt_ops and mpt_client — plus an async counterpart for each: async_mpt_vendor, async_mpt_ops and async_mpt_client. Ask for the scope the endpoint requires; the operations-scoped pair is what the streaming suites use.

Most directories mirror an API domain. streaming/ is the exception: it covers the platform streaming read mode itself — the contract stream_snapshot() implements — rather than one domain, and uses whichever collection makes the case observable.

Running Tests

make e2e                                  # run the E2E suite
make e2e path=tests/e2e/catalog           # run a subset of E2E tests
make e2e args="-k test_create_variant"    # pass extra pytest options

The target always passes a test path (tests/e2e unless path is set), so args only carries options. Passing options alone used to drop the path and fall back to testpaths, which made CI run the unit suite again after the E2E tests. E2E runs also disable coverage (--no-cov) so they do not overwrite the unit coverage.xml that SonarCloud reads.

E2E tests need live API credentials and run against the real API, so they have their own target: make test covers tests/unit only, and make check-all does not run them.

Streaming Memory Coverage

tests/e2e/streaming/test_sync_streaming.py and its async twin assert the property stream_snapshot() exists for: consuming it keeps memory bounded however many records the export carries, the bound streaming.md documents. It is the one suite whose result depends on the size of the live dataset, so it carries requirements the rest of tests/e2e/ does not.

It streams operations-scoped catalog.items, which holds well over the 20,000 records the largest single read exports. The measurement is a tracemalloc allocation peak — Python allocations rather than process resident size, so it does not move with allocator or kernel behaviour — taken over four whole reads: a 500-record warmup that pays for connection setup, lazy imports and the first read buffers outside every comparison, then a 2,000-record export, a 20,000-record export, and the same 20,000-record export buffered with list(). Only the last three are measured; the warmup is a separate read rather than a prologue of a measured one, so nothing can hide in it. Ten times the records must not cost materially more memory, and buffering the same export must cost far more, which is what shows the measurement is sensitive enough to see buffering reintroduced.

That makes the suite the heaviest consumer of live data in tests/e2e/: 42,500 records per case (500 + 2,000 + 20,000 + 20,000) for the sync suite, so roughly 85,000 including the async twin. Size the run against a rate-limited or metered environment from that figure, not from the 20,000 of the largest single read.

The comparison is against the peaks measured in the same run rather than an absolute byte threshold, so it does not need recalibrating per environment. If the environment holds fewer records than the coverage exports, the fixture fails with that count rather than passing on a dataset too small to mean anything. tests/e2e/streaming/memory_probe.py documents the protocol and the reasoning behind each constant.

Streaming Integrity Coverage

tests/e2e/streaming/test_sync_streaming_integrity.py and its async twin cover the parts of the streaming contract that only a live platform can demonstrate: a $meta.deleted deletion stub, that same stub withheld but still counted under skip_deleted, the declared MPT-Item-Count reaching a progress receiver, and an early close not reporting an incomplete export.

The stub cases provoke the contract's own race rather than simulating it: create a product, open a stream ordered oldest-first so that row is emitted last, then hard-delete it once the first record has arrived — the response headers are out by then, so the export's key snapshot already contains it. This needs a collection large enough that the platform cannot pre-buffer the body; on a small one the row streams back as a full record and no stub is produced. The assertions therefore pin the stub itself, so a race that was lost fails rather than passing on an absence.

Truncated transports and count mismatches are deliberately not covered here. Neither can be requested from a live platform — one needs the API to abort mid-body, the other contradicts the contract — so a test shaped like coverage for either would pass without exercising the path. They belong at a layer that can control the server.

Environment Variables

Variable Description
MPT_API_BASE_URL MPT API base URL
MPT_API_TOKEN_VENDOR Vendor API token
MPT_API_TOKEN_CLIENT Client API token
MPT_API_TOKEN_OPERATIONS Operations API token
MPT_API_TOKEN_EXTENSION Extension API key (installation token / extension framework auth)

Optional ReportPortal Integration

Variable Description
RP_API_KEY ReportPortal API key
RP_ENDPOINT ReportPortal endpoint URL
RP_LAUNCH ReportPortal launch name

Configuration

E2E test configuration lives in e2e_config.test.json. Set the required environment variables before invoking the suite to avoid credential validation failures.

Test results are published to Report Portal.