[SERVER-49972] Randomize unit tests execution order Created: 29/Jul/20 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Pierlauro Sciarelli | Assignee: | Backlog - Server Tooling and Methods (STM) (Inactive) |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | tig-qwin-eligible | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Server Tooling & Methods
|
| Participants: |
| Description |
|
Currently, unit tests are always executed as follows:
This ordering implies that - given the presence of a global shared state - unit tests can potentially depend in a deterministic way on each other. It would be good to randomize the execution order to force programmers to write proper setup/teardown to cleanup "reused" state. I am quite sure that a random shuffle of fixture names and tests ordering within files would currently result in some tests failing. Even in the improbable case everything would work, it is still important to address this ticket to have the warranty that future tests would not hit this problem. An example of the described problem is the bug solved in SERVER-49969: the only way to catch the error was by changing the fixture name. |
| Comments |
| Comment by Steven Vannelli [ 10/May/22 ] |
|
Moving this ticket to the Backlog and removing the "Backlog" fixVersion as per our latest policy for using fixVersions. |
| Comment by Robert Guo (Inactive) [ 15/Oct/20 ] |
|
Bumping this ticket to the backlog for now as we don't plan to get to it in the near future. Feel free to pick it up if you'd like it sooner. STM would be happy to review the code. |
| Comment by Billy Donahue [ 29/Jul/20 ] |
|
To acm's point, I added randomization to the MONGO_INITIALIZER order but since the seed isn't printed out or reproducible it has much less practical benefit than it could. Init order bugs it turns up in testing will just look like flakes. Wrote SERVER-48377 to fix this. The same could happen here unless the test order seed is carefully managed and injected by the CI system on reruns. |