Major - P3 The following test is failing intermittently (multiple times per patch) on evergreen:
[2019/06/13 16:02:34.810] ERROR [10.255s]: test_cmap_wait_queue_fairness_must_issue_Connections_to_threads_in_the_order_that_the_threads_entered_the_queue (test_cmap.TestCMAP) [2019/06/13 16:02:34.810] ---------------------------------------------------------------------- [2019/06/13 16:02:34.810] Traceback (most recent call last): [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/__init__.py", line 410, in wrap [2019/06/13 16:02:34.810] return f(*args, **kwargs) [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/test_cmap.py", line 383, in run_scenario [2019/06/13 16:02:34.810] self.run_scenario(scenario_def, test) [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/test_cmap.py", line 291, in run_scenario [2019/06/13 16:02:34.810] self.run_operations(test['operations']) [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/test_cmap.py", line 218, in run_operations [2019/06/13 16:02:34.810] self.run_operation(op) [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/test_cmap.py", line 213, in run_operation [2019/06/13 16:02:34.810] meth(op) [2019/06/13 16:02:34.810] File "/data/mci/f666f1928e5891ef9a16b87cce969eae/src/test/test_cmap.py", line 173, in wait_for_thread [2019/06/13 16:02:34.810] raise thread.exc [2019/06/13 16:02:34.810] ConnectionFailure: Timed out while checking out a connection from connection pool with max_size 1 and wait_queue_timeout 1.0
I believe that the 1 second wait_queue_timeout might be too short. We should increase it to 5 or 10 seconds.