[SERVER-49972] Randomize unit tests execution order Created: 29/Jul/20  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: Backlog - Server Tooling and Methods (STM) (Inactive)
Resolution: Unresolved Votes: 0
Labels: tig-qwin-eligible
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Tooling & Methods
Participants:

 Description   

Currently, unit tests are always executed as follows:

  • Fixtures alphabetical order
  • Within a fixture, order of declaration in test files

This ordering implies that - given the presence of a global shared state - unit tests can potentially depend in a deterministic way on each other. It would be good to randomize the execution order to force programmers to write proper setup/teardown to cleanup "reused" state.

I am quite sure that a random shuffle of fixture names and tests ordering within files would currently result in some tests failing. Even in the improbable case everything would work, it is still important to address this ticket to have the warranty that future tests would not hit this problem.

An example of the described problem is the bug solved in SERVER-49969: the only way to catch the error was by changing the fixture name.



 Comments   
Comment by Steven Vannelli [ 10/May/22 ]

Moving this ticket to the Backlog and removing the "Backlog" fixVersion as per our latest policy for using fixVersions.

Comment by Robert Guo (Inactive) [ 15/Oct/20 ]

Bumping this ticket to the backlog for now as we don't plan to get to it in the near future. Feel free to pick it up if you'd like it sooner. STM would be happy to review the code.

Comment by Billy Donahue [ 29/Jul/20 ]

To acm's point, I added randomization to the MONGO_INITIALIZER order but since the seed isn't printed out or reproducible it has much less practical benefit than it could. Init order bugs it turns up in testing will just look like flakes. Wrote SERVER-48377 to fix this.

The same could happen here unless the test order seed is carefully managed and injected by the CI system on reruns. 

Generated at Thu Feb 08 05:21:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.