[SERVER-74316] break up variants which cache unrelated builds Created: 23/Feb/23  Updated: 29/Oct/23  Resolved: 17/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.0.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Daniel Moody Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-74624 Make sure commit-queue build times re... Closed
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Participants:

 Description   

Several compile variants have several sets of tasks which compile different things with different flags. There could be some optimization here by standardizing the flags and aggregating the tasks to a fewer tasks.

 

Investigate the main compile variants and if the different tasks which compile tasks can be combined for:
linux-x86-dynamic-compile-required
amazon-linux2-arm64-compile
linux-debug-aubsan-compile-required
linux-x86-dynamic-debug-compile-required
commit-queue

 

For example, the linux-x86-dynamic-compile-required has tasks like:

  1. standard compile_* tasks
  2. crypt_* tasks
  3. stitch_* tasks
  4. integration_* tasks
  5. libdeps_graph_linting
  6. other various tasks

Some of these tasks have different compile flags. Could the targets among these tasks all be built in a single task?



 Comments   
Comment by Githook User [ 17/Mar/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'daniel.moody@mongodb.com', 'username': 'dmoody256'}

Message: SERVER-74316 split out crypt and stitch builds to separate variants
Branch: master
https://github.com/mongodb/mongo/commit/4f403ab89f26fabedb5221574ef1ae15c3a0d936

Comment by Daniel Moody [ 28/Feb/23 ]

The question is what do we really need in the variant? In the current state of the yml, these are compile variants aimed at created binaries for test variants, yet many of these tasks also perform testing. 

 

Also in terms of priority, the linux-x86-dynamic-compile-required is one of the most run variants. Do we really need the ancillaries like crypt and stich which have special build flags here? could the live in another suggested compile variant?

Comment by Daniel Moody [ 28/Feb/23 ]

In the main variant here, linux-x86-dynamic-compile-required, we are building and caching 4 separate builds due to flags, we would expect one build generated in the variant, so I will list 3 builds that do not conform:

crypt_create_lib: Use of specific flags SHLINKFLAGS_EXTRA=Wl,-Bsymbolic -Wl,-no-gnu-unique CCFLAGS=-fno-gnu-unique
crypt_debug_create_lib: use of debug flags
stich_create_lib: use of dynamic-sdk (static build that is commonly confused as dynamic build)

The other problem is that many of the tasks build the same targets, but use the cache as a distributed compiler to increase concurrency. This has the effect of pulling down entire builds repeatedly from the cache, where instead, the same mci could be used, or the test could download the binaries from s3 bucket. Also do the nature of dependency chains in the build there is not much randomness so the desired "distributed compiler" effect ends up not being very effective.
 
The other variants suffer from the same problems, but to lesser degrees.

Comment by Alex Neben [ 27/Feb/23 ]

I think they are built in multiple tasks for performance reasons. I bet though that there is no reason for different compile flags for most of these tasks though.

Generated at Thu Feb 08 06:27:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.