This ticket aims to catalog our existing test so that we can place under the following categories:

• Fully deterministic tests:
  • These are tests where the expected steps, behavior and results are exactly the same from run-to-run. For WT, this will typically only apply to unit tests calling sequential code.
    • Example: calls to functions such as __wt_ispo2
• Predictable tests:
  • Some multi-threaded activity may not be 100% deterministic ‘under the hood’, and tests where the code is fully deterministic are preferred. However, if the results are fully predictable, then that is still very useful.
    • Example: create a database, and write the integers 1 to 100 to a table, and then read them back.
    • Note: if the behavior/results of a test is dependent on, for example, when eviction happens to take place, then that test is non-deterministic.
• Non-deterministic tests:
  • These are tests where the expected behavior and results can vary from run-to-run.
    • These include tests where the sequence of operations is determined by a random number generator with a variable seed.
• Optionally-predictable tests:
  • These are tests which can be deterministic, predictable, or non-predictable, depending on an initial seed to a random number generator and/or the test configuration.
    • E.g. Some non-deterministic tests can be made optionally-predictable simply by providing the same random number seed every time. This may require changing the test code to be able to set the seed, for example.
    • E.g. An optionally-predictable test, when combined with a particular seed and/or configuration, becomes either predictable or unpredictable.

Each deterministic test in Evergreen will be tagged as deterministic. Each optionally-predictable test which has been made predictable by the appropriate choice in Evergreen of a random seed and/or configuration will also be labeled as a predictable test.

The ticket will clearly split post-merge tests into deterministic and non-deterministic tests, so that it is clear whether a developer has broken something or if they’ve been ‘unlucky’ to hit a non-deterministic bug