[SERVER-74316] break up variants which cache unrelated builds Created: 23/Feb/23 Updated: 29/Oct/23 Resolved: 17/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Moody | Assignee: | Daniel Moody |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Participants: | |||||||||
| Description |
|
Several compile variants have several sets of tasks which compile different things with different flags. There could be some optimization here by standardizing the flags and aggregating the tasks to a fewer tasks.
Investigate the main compile variants and if the different tasks which compile tasks can be combined for:
For example, the linux-x86-dynamic-compile-required has tasks like:
Some of these tasks have different compile flags. Could the targets among these tasks all be built in a single task? |
| Comments |
| Comment by Githook User [ 17/Mar/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}Message: |
| Comment by Daniel Moody [ 28/Feb/23 ] |
|
The question is what do we really need in the variant? In the current state of the yml, these are compile variants aimed at created binaries for test variants, yet many of these tasks also perform testing.
Also in terms of priority, the linux-x86-dynamic-compile-required is one of the most run variants. Do we really need the ancillaries like crypt and stich which have special build flags here? could the live in another suggested compile variant? |
| Comment by Daniel Moody [ 28/Feb/23 ] |
|
In the main variant here, linux-x86-dynamic-compile-required, we are building and caching 4 separate builds due to flags, we would expect one build generated in the variant, so I will list 3 builds that do not conform: crypt_create_lib: Use of specific flags SHLINKFLAGS_EXTRA= The other problem is that many of the tasks build the same targets, but use the cache as a distributed compiler to increase concurrency. This has the effect of pulling down entire builds repeatedly from the cache, where instead, the same mci could be used, or the test could download the binaries from s3 bucket. Also do the nature of dependency chains in the build there is not much randomness so the desired "distributed compiler" effect ends up not being very effective. |
| Comment by Alex Neben [ 27/Feb/23 ] |
|
I think they are built in multiple tasks for performance reasons. I bet though that there is no reason for different compile flags for most of these tasks though. |