[SERVER-74287] create scons OOM retry tool Created: 22/Feb/23 Updated: 31/Oct/23 Resolved: 15/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0-rc0, 4.4.20, 5.0.16, 6.0.6 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Moody | Assignee: | Daniel Moody |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Backport Requested: |
v6.0, v5.0, v4.4, v4.2
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 31 | ||||||||||||||||
| Description |
|
We frequently get failed builds in evergreen due to OOM and we have tried to manage this by limiting the concurrency for certain tasks, but this is not a very precise solution and has resulted in continued failed builds and maintenance.
We should create a scons tool which will modify link and compile jobs so that if the underlying command failed, we parse the stdout/stderr and identify the OOM error message depending on the compiler/platform. We then wait some random time, and try again up to some max number of retries. |
| Comments |
| Comment by Githook User [ 21/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da) |
| Comment by Githook User [ 21/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da) |
| Comment by Githook User [ 21/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: (cherry picked from commit 3949b2a325ac9f421a4b29074bc037c27293c5da) |
| Comment by Githook User [ 15/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: |