[SERVER-33406] Reduce concurrency for s390x compile and compile_all builds to avoid OOM failures Created: 20/Feb/18  Updated: 29/Oct/23  Resolved: 25/Feb/18

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.4.14, 3.6.4, 3.7.3

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Ian Whalen (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-29442 Reduce concurrency for Solaris builds... Closed
is related to SERVER-30516 Reduce concurrency for Windows compil... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6, v3.4
Participants:
Linked BF Score: 0

 Description   

The compile and compile_all Evergreen tasks has been failing intermittently due to one of the ld processes being killed by OOM killer since Feb 2017. As a workaround—similar to what was done in SERVER-29442 and SERVER-30516—we should reduce the value for the --jobs option specified to SCons when our s390x builders are running the compile and compile_all task to avoid further Evergreen failures. Given that an individual linker process may use >3.5GB of memory of 16GB total, we should prevent SCons from running more than 3 of them as the same time by specifying -j3 rather than -j$(grep -c ^processor /proc/cpuinfo) or -j$(( $(grep -c ^processor /proc/cpuinfo) / 2 )) in each build variant's ${compile_flags} expansion.

[2018/02/16 20:07:33.596] mci-exec  9539  3.3 22.5 3896412 3660772 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccoOHhDJ
[2018/02/16 20:07:33.596] mci-exec  9837  3.4 22.6 3896592 3685484 ?     D    19:52   0:31 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/cc8EneGh
[2018/02/16 20:07:33.597] mci-exec 13693  5.0 22.2 3883392 3625496 ?     D    19:57   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccLraHsk
[2018/02/16 20:07:33.597] mci-exec 14395  5.5 22.3 3885152 3640884 ?     D    19:58   0:30 /opt/mongodbtoolchain/v2/lib/gcc/s390x-mongodb-linux/5.4.0/../../../../s390x-mongodb-linux/bin/ld @/tmp/ccekp0ZZ

https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_ubuntu1604_s390x_compile_all_537e479caf3c6a2db67c0fa1c5d6769c9aea8786_01_01_01_00_00_00



 Comments   
Comment by Githook User [ 26/Feb/18 ]

Author:

{'email': 'ian.whalen@gmail.com', 'name': 'Ian Whalen', 'username': 'IanWhalen'}

Message: SERVER-33406 Reduce compile concurrency in s390x buildvariants
Branch: v3.6
https://github.com/mongodb/mongo/commit/cd452da61d5375983b5ccdd9965068f741f5191a

Comment by Githook User [ 26/Feb/18 ]

Author:

{'email': 'ian.whalen@gmail.com', 'name': 'Ian Whalen', 'username': 'IanWhalen'}

Message: SERVER-33406 Reduce compile concurrency in s390x buildvariants
Branch: v3.4
https://github.com/mongodb/mongo/commit/7e32d6883aa6b7c141e6dbf230879902dcf1d1fe

Comment by Githook User [ 25/Feb/18 ]

Author:

{'email': 'ian.whalen@gmail.com', 'name': 'Ian Whalen', 'username': 'IanWhalen'}

Message: SERVER-33406 Reduce compile concurrency in s390x buildvariants
Branch: master
https://github.com/mongodb/mongo/commit/0a0597248fa8234414c07ec6cd5a4fef51169148

Generated at Thu Feb 08 04:33:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.