[SERVER-50632] Building and installing tests takes a very long time Created: 28/Aug/20  Updated: 28/Aug/23  Resolved: 28/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.5 Desired
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Ryan Egesdahl (Inactive) Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Development Platform
Participants:

 Description   

I noticed that building the tests under the install-all-meta target takes a very long time to complete. Some of that time is due to linking, but an even greater portion appears to be SCons scanning its cache for dependencies of unittests.txt and integration-tests.txt (among others?). What we have essentially done is to pin every object file node as a dependency to a single SCons node that then outputs a single text file for each test class.

This is probably not what we really wanted to do if we care about performance. We should rearrange this so these text files get appended to atomically somehow as part of the test build, or potentially some other way that accomplishes the same thing without involving the SCons dependency graph.



 Comments   
Comment by Andrew Morrow (Inactive) [ 31/Aug/20 ]

Currently, the mongo_test_list.py tool causes the test list it generates to have a hard Depends edge on ever test. When building in a mode that uses content-signatures, this is potentially expensive. Instead, these edges could be weakened to Require edges.

Unfortunately, this would break things as currently written, because those hard Depends edges are what cause the test list to be regenerated if the space of tests changes.

What should be happening is something more along the lines of accumulating the list of tests into a SCons Value node, and then having the test list file depend on that list.

Additionally, we can probably eliminate the calls that look like env.Alias("$UNITTEST_ALIAS", result[0]), since these were a legacy of the pre-hygienic build system. All tests should already be arriving in the correct {{Alias}es that explicit connection.

Finally, we could eliminate the need to alias the test list into the test alias by generating the test list and then using AutoInstall to install it with the correct COMPONENT and ROLE annotations, which would automatically incorporate it into the correct dependencies, as well as stage it into the installation directory, where we would want it to go anyway as part of making a self testable installation.

Generated at Thu Feb 08 05:23:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.