[SERVER-65040] Reuse build artifacts across builders which differ only in server test execution parameters Created: 29/Mar/22 Updated: 29/Oct/23 Resolved: 27/Oct/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.2.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Andrew Morrow (Inactive) | Assignee: | Robert Guo (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Dev Platform 2022-05-30, Dev Platform 2022-06-13, Dev Platform 2022-06-27, Dev Platform 2022-07-11, Dev Platform 2022-07-25 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 60 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
We have lots of builders which differ only in the parameters under which the tests execute. For instance ubuntu1804-sbe-yielding-debug and ubuntu1804-debug-suggested. Currently, those builders each declare the full complement of build and test stages: compile_dist_test, run_unittests. However, the results of compile_dist_test are identical across those two builders, and the unit tests aren't parameterized. We could arrange things so that these builders (and builders in similar relationships) simply re-use the compilation artifacts from the "canonical" builder. Eliminating this duplicated work would reduce artifact cache pressure, eliminate duplicate BFs when things are broken, avoid spending time executing unit tests which should give identical results, etc. |
| Comments |
| Comment by Githook User [ 24/Oct/22 ] | |||
|
Author: {'name': 'Rob Guo', 'email': 'robert.guo@mongodb.com'}Message: | |||
| Comment by Robert Guo (Inactive) [ 08/Apr/22 ] | |||
|
Yep. Thanks for the reminder Max! I believe we could do something similar to | |||
| Comment by Max Hirschhorn [ 07/Apr/22 ] | |||
|
Does this include having burn_in_tags_gen not generate a compile_dist_test task to build the server for the build variants it generates? For some historical context: When | |||
| Comment by Iryna Zhuravlova [ 04/Apr/22 ] | |||
|
This work should be done after | |||
| Comment by Richard Samuels (Inactive) [ 29/Mar/22 ] | |||
|
We'd need to audit the buildvariants and find those with identical flags (this could be done with a python script). Using the above example, tasks currently dependent on archive_dist_test will instead need a slightly different depends_on:
(same for archive_dist_test_debug) Then we'd have to make the portion of the yaml that downloads the binaries/debug symbols aware of that change. That might be as simple as writing some python that generates a yaml file with values for the expansions set in &set_task_expansion_macros (currently set in etc/evergreen_yml_definitions/definitions.yml) Then we'd need to also update the symbolizer and hang analyzer to be aware of the new locations for download debug symbols. I suspect this could be achieved in one change with an update to db-contrib-tool. A further audit of uses of the ${build_variant} expansion would be required to make sure we don't miss anything else. I suspect we'll encounter a complication due to |