[!!!][FEATURE] Share test instances between identically configured test cases - #740
Open
bmack wants to merge 2 commits into
Open
[!!!][FEATURE] Share test instances between identically configured test cases#740bmack wants to merge 2 commits into
bmack wants to merge 2 commits into
Conversation
…st cases Functional test instances are identified by sha1(static::class), so every test case class provisions its own instance: its own directory tree, its own compiled dependency injection container and its own database schema. Across the TYPO3 core corpus that is 764 instances, while only 207 distinct instance configurations exist. Measured on ext:core, provisioning accounts for roughly a third of the suite's wall clock, and a cold bootstrap costs about 25 times a warm one because the container has to be compiled again. Instances are now identified by what actually shapes them - the extensions to load, the paths to link and provide, the configuration overrides, the folders to create and whether the database is initialised - so test cases configured identically share one instance. It is provisioned once, its container is compiled once and its schema is created once. Sharing an instance between test case classes exposes state that used to be private simply because every test case owned an instance. Six such channels were found and are addressed here: * The instance cache directory is shared, because the package dependent cache identifier derives from the project path. The whole "core" cache group is dropped when one test case class hands the instance to the next, so a cached TCA schema or a backend module registry written by one test case cannot be seen by another. The compiled container is deliberately kept: it depends only on the active package set, and keeping it is what makes sharing worthwhile. * Database snapshots were keyed on the instance and created on the first test of the instance rather than of the test case. They are now scoped to the test case class, so a test case that did not provision the instance no longer restores a snapshot it never created. * Test cases write to typo3conf/system/settings.php, for example through the "configuration:set" command. A pristine copy is kept at provisioning time and restored at a test case class boundary. * Test cases that override how the database is provisioned - the core schema test cases start from a database with no tables - rely on being the test case that provisions the instance. Such test cases now get their own instance family, keyed on the declaring class of initializeTestDatabase() and initializeTestDatabaseAndTruncateTables(). Breaking changes: * getInstanceIdentifier() and getInstancePath() are no longer static. They cannot be, because test cases may assign the configuration properties in their own setUp() before calling parent::setUp(), and a static method cannot see those assignments. Both are marked @internal, and getInstancePath() already documented that it may break at any time. Migration, for test cases that touch them: - Calling them: use $this->getInstanceIdentifier() and $this->getInstancePath() instead of self:: or static::. - Overriding them: drop the "static" keyword from the declaration. A subclass that still declares them static will fail to load. - Overriding getInstanceIdentifier() to pin an instance is usually the wrong hook now. Anything that shapes what the instance *contains* belongs in getInstanceConfiguration(), so that test cases configured the same way keep sharing an instance; an overridden identifier opts the test case out of sharing entirely. Keeping the methods static was considered, by computing the identifier in setUp() and handing it out through a static property. That preserves both static call sites and static overrides, but it puts mutable static state back into FunctionalTestCase - the same class of coupling as the six channels fixed above - and leaves a static signature whose result is only meaningful between setUp() and tearDown(). The honest break was preferred. Neither typo3/cms nor the testing framework itself calls these statically any more. * DatabaseSnapshot::initialize() gains an optional third argument naming the snapshot separately from the instance. Existing two argument calls behave as before. Validated against the full functional suite of TYPO3 core - all system extensions - comparing assertion counts and failing test sets, not just totals: DBMS tests assertions baseline shared delta sqlite 12392 72711 1341.6 s 847.8 s -36.8 % MariaDB 12427 72966 identical outcome, timing per notes Postgres 12388 72693 2109.8 s 1672.4 s -20.7 % Identical failure sets in every case. A controlled back to back measurement of ext:core on MariaDB, three runs in one job, shows 563.6 s against 450.0 s, a 20.2 per cent improvement with a 3.6 per cent spread between repeats.
The test instance directory and the test database are named after the instance identifier alone, so two functional test runs sharing a working copy address the same instance directory and the same database. A run that provisions an instance then deletes it while the other run is still using it, and both collapse. Reproduced on an unmodified checkout by running one test case twice at the same time: 41 errors and 10 failures in one run, 34 errors and 12 failures in the other, one shared instance directory. Setting TYPO3_TESTING_WORKER to a different value per run appends it to the instance identifier, so the runs get their own instance directory and their own database. The same two runs then pass with 70 tests each. The variable is unset by default and naming is then unchanged. It becomes part of database names, so the value is restricted to one to eight alphanumeric characters. An invalid value is rejected rather than sanitised: silently folding "1-a" and "1a" onto one instance would reintroduce exactly the collision this prevents. This is a prerequisite for running the suite with several workers, but it is useful on its own - it is what makes it possible to run two database backends, or one suite and one test file, against a single working copy.
bmack
force-pushed
the
layer0-share-test-instances
branch
from
August 10, 2026 21:21
544a014 to
c5369a3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Functional test instances are identified by
sha1(static::class), so every test case classprovisions its own instance: its own directory tree, its own compiled dependency injection
container and its own database schema. Across the TYPO3 core corpus that is 764 instances,
while only 207 distinct instance configurations exist.
This series keys instances on what actually shapes them — the extensions to load, the paths to
link and provide, the configuration overrides, the folders to create, whether the database is
initialised — so test cases configured identically share one instance. It is provisioned once,
its container is compiled once, its schema is created once.
Blast radius
Probably the first thing you want to know, since this carries
[!!!].getInstanceIdentifier()andgetInstancePath()become non-static. They cannot stay static: atest case may assign the configuration properties in its own
setUp()before callingparent::setUp(), and a static method cannot see those assignments.@internal, andgetInstancePath()'s docblock already stated "This may breakanytime."
typo3/cmsnor this repository calls them statically any more — core's call siteswere converted upstream in
17e81da849d.@internalmethod. Thefirst commit carries a migration note covering the three cases such a test case can be in.
Keeping them static via a holder that
setUp()populates was considered and rejected — it wouldput mutable static state back into
FunctionalTestCase, which is the same class of coupling asthe pollution channels this commit fixes, and would leave a static signature whose result is only
meaningful between
setUp()andtearDown(). The reasoning is in the commit message.What to look at
Most of the diff is mechanism. The two decisions worth a reviewer's attention:
getInstanceConfiguration()). Anything that shapes aninstance's content but is missing from that list would let two genuinely different instances
share an identifier — a silent wrong-green. The list is the load-bearing part.
whole
corecache group is dropped, butdiis kept, because it depends only on the activepackage set — and keeping it is what makes sharing worth doing at all. If that reasoning is
wrong, the speedup mostly evaporates.
Sharing an instance between test case classes exposes state that was previously private simply
because every test case owned an instance. Six such channels were found and are fixed here; the
commit message lists them individually.
Measurements
Layer 0 in isolation, full core corpus, measured 2026-07-31 against the then-current
main:A more recent full-corpus run (2026-08-04, with the composer-mode work of the follow-up PR also
applied, but in classic mode where that work is largely inert) measured 1379.2 s → 945.9 s,
−31.4 % on sqlite.
Every comparison was made on assertion counts and failing test sets, not just totals. A suite
that stays green while quietly doing less work would otherwise look like a win.
Verification
Each commit was verified green individually against the full TYPO3 core functional corpus, with
the upstream testing framework run as a control arm in the same job. On the current base both
commits produce results identical to upstream: 12473 tests, 72962 assertions, 0 errors, the same
6 pre-existing failures.
Follow-up
A second PR provisions functional test instances in composer mode and builds on this one. The
TYPO3 core side of that work is prepared and waits on a release containing this PR. That is the
change this series exists for; the speedup here is what makes it affordable.