[SERVER-74287] create scons OOM retry tool Created: 22/Feb/23  Updated: 31/Oct/23  Resolved: 15/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.0.0-rc0, 4.4.20, 5.0.16, 6.0.6

Type: Improvement Priority: Major - P3
Reporter: Daniel Moody Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-82612 Use all available CPU cores for scons... Open
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0, v5.0, v4.4, v4.2
Participants:
Linked BF Score: 31

 Description   

We frequently get failed builds in evergreen due to OOM and we have tried to manage this by limiting the concurrency for certain tasks, but this is not a very precise solution and has resulted in continued failed builds and maintenance. 

 

We should create a scons tool which will modify link and compile jobs so that if the underlying command failed, we parse the stdout/stderr and identify the OOM error message depending on the compiler/platform. We then wait some random time, and try again up to some max number of retries.



 Comments   
Comment by Githook User [ 21/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74287 add oom retry tool

(cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da)
Branch: v6.0
https://github.com/mongodb/mongo/commit/5e53037e3612791e1662f3fa4b04b72de8839d86

Comment by Githook User [ 21/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74287 add oom retry tool

(cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da)
(cherry picked from commit f13c679a03aad1ea09da6dcd131fbee959d9039b)
Branch: v5.0
https://github.com/mongodb/mongo/commit/099317518e6da031512251c99ed39bd5ce39f6ae

Comment by Githook User [ 21/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74287 add oom retry tool

(cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da)
(cherry picked from commit f13c679a03aad1ea09da6dcd131fbee959d9039b)
(cherry picked from commit 7f3f01822da6ceb877d6944a2a5de8df0d403c0b)
Branch: v4.4
https://github.com/mongodb/mongo/commit/8b65a42f2385e590b2fef76dec4e30a573039951

Comment by Githook User [ 15/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74287 add oom retry tool
Branch: master
https://github.com/mongodb/mongo/commit/3949b2a325ac9f421a4b29074bc037c27293c5da

Generated at Thu Feb 08 06:27:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.