The concurrency_sharded*.yml test suites run against a 2-shard, 3-node replica set shard cluster (sometimes with a 1-node CSRS and sometimes with a 3-node CSRS) with 2 mongos processes. They currently run on the -small distros, which typically correspond to c4.xlarge and have 7.5GB of memory. The 6+ mongod processes tend not to leave much headroom for the other MongoDB processes, resmoke.py buffering log messages, or gdb attaching to a process. We have recently been seeing the OOM killer killing processes on the machine and causing the Evergreen task to fail. Moving to the -large distros, which typically correspond to c4.4xlarge and have 30GB of memory, should address these failures. This involves doing the following:
- Adding the "large" Evergreen task tag to all of the concurrency_sharded* task definitions.
- Explicitly specifying the distros option to the concurrency_sharded* tasks listed on the Enterprise RHEL 6.2 (large transactions format) and Enterprise RHEL 6.2 (majority read concern off) build variants. Note that the concurrency_sharded_causal_consistency_gen task specifies use_large_distro: "true" already and can be left alone.