-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Server Tooling & Methods
Currently, unit tests are always executed as follows:
- Fixtures alphabetical order
- Within a fixture, order of declaration in test files
This ordering implies that - given the presence of a global shared state - unit tests can potentially depend in a deterministic way on each other. It would be good to randomize the execution order to force programmers to write proper setup/teardown to cleanup "reused" state.
I am quite sure that a random shuffle of fixture names and tests ordering within files would currently result in some tests failing. Even in the improbable case everything would work, it is still important to address this ticket to have the warranty that future tests would not hit this problem.
An example of the described problem is the bug solved in SERVER-49969: the only way to catch the error was by changing the fixture name.