Test/format is one of the major stress-testing frameworks we leverage, which has been proven effective in capturing issues. We have a script format.sh that invokes the test/format program and maintains N jobs running constantly (one job completes the next is started). This job scheduling mode simulates constant load scenarios and is the default mode currently employed by our Evergreen testing.

Another job scheduling mode is to simulate the "thundering herd" effect by starting N jobs simultaneously and waiting for all jobs completed before starting another N jobs. This job scheduling mode was employed by sue.loverso@mongodb.com in her local testing, which managed to capture/reproduce many issues that Evergreen testing failed to, thanks to the load and timing differences compared to the other scheduling mode. See WT-14206 for additional details.

This ticket is to introduce this new job scheduling mode to the format.sh script and exercise this new mode in our Evergreen testing.