[SERVER-47428] Move concurrency_sharded* Evergreen tasks to -large distros Created: 09/Apr/20  Updated: 29/Oct/23  Resolved: 15/Apr/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.4.0-rc2, 4.7.0, 4.2.10

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: sharding-4.4-stabilization
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4, v4.2
Sprint: Sharding 2020-04-20
Participants:
Linked BF Score: 18

 Description   

The concurrency_sharded*.yml test suites run against a 2-shard, 3-node replica set shard cluster (sometimes with a 1-node CSRS and sometimes with a 3-node CSRS) with 2 mongos processes. They currently run on the -small distros, which typically correspond to c4.xlarge and have 7.5GB of memory. The 6+ mongod processes tend not to leave much headroom for the other MongoDB processes, resmoke.py buffering log messages, or gdb attaching to a process. We have recently been seeing the OOM killer killing processes on the machine and causing the Evergreen task to fail. Moving to the -large distros, which typically correspond to c4.4xlarge and have 30GB of memory, should address these failures. This involves doing the following:

  1. Adding the "large" Evergreen task tag to all of the concurrency_sharded* task definitions.
  2. Explicitly specifying the distros option to the concurrency_sharded* tasks listed on the Enterprise RHEL 6.2 (large transactions format) and Enterprise RHEL 6.2 (majority read concern off) build variants. Note that the concurrency_sharded_causal_consistency_gen task specifies use_large_distro: "true" already and can be left alone.


 Comments   
Comment by Githook User [ 10/Aug/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-47428 Move concurrency_sharded* tasks to -large distros.

Not all build variants run the concurrency* tasks with the "large" task
tag on a -large distro. The RHEL 6.2 and SUSE 12 build variants were
specifically modified to do so because OOM failures have been observed
on them.

(cherry picked from commit 6202c52a254ced5238eaf7f77117f7c4865aed56)
(cherry picked from commit 6c5be09f0f56e390ee0eb652b6c723fb3e484d7f)
Branch: v4.2
https://github.com/mongodb/mongo/commit/d4e970c26ded41593ea5c008526f7c2ff1d56d5a

Comment by Githook User [ 15/Apr/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-47428 Move concurrency_sharded* tasks to -large distros.

Not all build variants run the concurrency* tasks with the "large" task
tag on a -large distro. The RHEL 6.2 and SUSE 12 build variants were
specifically modified to do so because OOM failures have been observed
on them.

(cherry picked from commit 6202c52a254ced5238eaf7f77117f7c4865aed56)
Branch: v4.4
https://github.com/mongodb/mongo/commit/6c5be09f0f56e390ee0eb652b6c723fb3e484d7f

Comment by Githook User [ 15/Apr/20 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-47428 Move concurrency_sharded* tasks to -large distros.

Not all build variants run the concurrency* tasks with the "large" task
tag on a -large distro. The RHEL 6.2 and SUSE 12 build variants were
specifically modified to do so because OOM failures have been observed
on them.
Branch: master
https://github.com/mongodb/mongo/commit/6202c52a254ced5238eaf7f77117f7c4865aed56

Generated at Thu Feb 08 05:14:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.