[SERVER-50632] Building and installing tests takes a very long time Created: 28/Aug/20 Updated: 28/Aug/23 Resolved: 28/Aug/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.5 Desired |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | Ryan Egesdahl (Inactive) | Assignee: | [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive) |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Server Development Platform
|
| Participants: |
| Description |
|
I noticed that building the tests under the install-all-meta target takes a very long time to complete. Some of that time is due to linking, but an even greater portion appears to be SCons scanning its cache for dependencies of unittests.txt and integration-tests.txt (among others?). What we have essentially done is to pin every object file node as a dependency to a single SCons node that then outputs a single text file for each test class. This is probably not what we really wanted to do if we care about performance. We should rearrange this so these text files get appended to atomically somehow as part of the test build, or potentially some other way that accomplishes the same thing without involving the SCons dependency graph. |
| Comments |
| Comment by Andrew Morrow (Inactive) [ 31/Aug/20 ] |
|
Currently, the mongo_test_list.py tool causes the test list it generates to have a hard Depends edge on ever test. When building in a mode that uses content-signatures, this is potentially expensive. Instead, these edges could be weakened to Require edges. Unfortunately, this would break things as currently written, because those hard Depends edges are what cause the test list to be regenerated if the space of tests changes. What should be happening is something more along the lines of accumulating the list of tests into a SCons Value node, and then having the test list file depend on that list. Additionally, we can probably eliminate the calls that look like env.Alias("$UNITTEST_ALIAS", result[0]), since these were a legacy of the pre-hygienic build system. All tests should already be arriving in the correct {{Alias}es that explicit connection. Finally, we could eliminate the need to alias the test list into the test alias by generating the test list and then using AutoInstall to install it with the correct COMPONENT and ROLE annotations, which would automatically incorporate it into the correct dependencies, as well as stage it into the installation directory, where we would want it to go anyway as part of making a self testable installation. |