[SERVER-82612] Use all available CPU cores for scons in compilation tasks Created: 31/Oct/23  Updated: 02/Feb/24

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Tommaso Tocci Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-74287 create scons OOM retry tool Closed
Assigned Teams:
Correctness
Sprint: Dev Tools 2020-04-06
Participants:

 Description   

Currently we use two evergreen global expansion variables to limit the amount the number of CPU cores SCons will use in compile tasks. In particular:

num_scons_link_jobs_available: "0.25"
num_scons_unit_cc_jobs_available: "0.75"

We only use 25% of available cores for link jobs and 75% for cc jobs.

I run a patch with those two variables set to 1 and all compile tasks in all variants simply succeeded without issues.

I think that rather than limiting by default the number of cores, we should instead use all cores by default and cap the number of jobs only in the variants that hit Out of memory limit. (e.g. here)



 Comments   
Comment by Tommaso Tocci [ 31/Oct/23 ]

I did some experiments that shows how changing those two global default variables to 1 can bring up to 40% execution time improvements for compile tasks.

Generated at Thu Feb 08 06:49:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.