1) Not setting nojournal appears to cause mongod to do its preallocation checks, in spite of noprealloc being set. Adding nojournal to the set of default flags for creating a mongod results in the following runtime improvements:
Finished in 220.794634234 seconds.
Finished in 56.069549861 seconds.
Finished in 176.140051997 seconds.
Finished in 15.343051209 seconds.
Net result is a runtime reduction of ~30%. Given that test data is ephemeral by definition, I don't see any obvious advantage to leaving journaling on for these checks. I can open a PR with this fix if you guys agree that it's not necessary.
2) Setting SO_REUSEADDR on the port checks in MongoConfig::port_available? allows the test suite to more easily reuse existing replica sets; this helps to prevent sequential test runs from failing due to conflicts in replica set runs (and prevents us from having to do the expensive initial allocation).
3) Finally, it seems like the replica test runs seem to stall out because:
1) The RS is spun up
2) The RS elects a primary
3) The primary is killed
4) The RS has to wait a full ~30 seconds (const time_t LeaseTime = 30 in consensus.cpp) to elect a new master.
I don't see an immediate way around this since the lease time is hardcoded in the mongo daemon, but it makes me nervous, because it looks like it would be possible for the cluster to be unavailable for up to 30 seconds if you were to kill the primary immediately after an election. I think this probably requires resolution in the mongo server proper, but it seems like when attempting to cast a `yea` vote, the server should check if the member it previously voted for is still available, and if not, it should release that lease and cast a new `yea` vote. Am I interpreting that correctly, and if so, should I open a server ticket to that effect?