UPDATE: In the interest of not declaring implementation, I just wanted to highlight that the goal here is twofold:

1) Reducing the number of failures showing up for the buildbaron.
2) Taking scheduled->finish patch build execution time below 2.5 hours.

How we get there (splitting test suites, deleting tests, running in parallel, etc.) I leave to whomever owns this.

===========

I'm not certain this is even the place to discuss this, but wanted to at least start the discussion. We've noticed a few jumps in test timing, and it seems like we should at least consider reducing those jumps where possible. This ticket just represents the first test timing regression that I've run down.

Enterprise RHEL 6.2 - sharding_WT

This commit (and the ones before it) take 39 minutes
This commit takes 48 minutes
I’m rerunning them now, but it looks like one of these two commits by Kal caused
mongos_rs_shard_failure_tolerance.js to go from 1 minutes to 8 minutes.
https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_sharding_WT_a707b9852bf6e03e7d6e6ef3ad464dbd28d690fa_15_11_11_15_18_34
https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_sharding_WT_67b68b5f094d88753ae2fe14f6d708c9e5b4bfbd_15_11_11_15_18_34

Windows 2008R2 DEBUG

Same thing:
The first commit (and the ones before it) take 3 hours and 10 minutes
The second commit takes 4 hours.
Looks like one of those two commits increased the timing for several tests within the suite.