[SERVER-20315] Separate compile and unit test execution tasks in evergreen config Created: 08/Sep/15 Updated: 05/Feb/16 Resolved: 08/Oct/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Build |
| Affects Version/s: | None |
| Fix Version/s: | 3.1.9 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Spencer Brody (Inactive) | Assignee: | Jonathan Reams |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | Build A (10/09/15) | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Right now the more unit tests we add, the longer the compile phase runs, and thus the longer before other suites get started. No reason we couldn't run the unit tests in parallel with the jstests and start getting feedback faster. |
| Comments |
| Comment by Andrew Morrow (Inactive) [ 07/Oct/15 ] | ||||||
|
Might as well go ahead and add the 'enterprise' into the name here, as well. | ||||||
| Comment by Kamran K. [ 07/Oct/15 ] | ||||||
|
jonathan.reams: Can you add the 'unittests' task to the "~ ASAN SSL Ubuntu 14.10" variant before you close this ticket? It wasn't added as part of the above commit. | ||||||
| Comment by Githook User [ 07/Oct/15 ] | ||||||
|
Author: {u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'}Message: | ||||||
| Comment by Spencer Brody (Inactive) [ 28/Sep/15 ] | ||||||
|
Conclusion of offline conversation is that we will strip the debug symbols and compress the unit tests, then add them to s3 to allow the unit test execution to run as a different task (potentially on a different machine) than the compile task. | ||||||
| Comment by Andrew Morrow (Inactive) [ 28/Sep/15 ] | ||||||
|
I'm not thrilled about having the compile step not build the whole tree. I really want to know that everything compiles before we do anything else. | ||||||
| Comment by Max Hirschhorn [ 28/Sep/15 ] | ||||||
|
Per suggestion from acm, I ran a patch build that skips building the unit tests by removing it from the all alias. It takes about 8 minutes to build the remaining targets.
For comparison, it took about 13 minutes on a recent commit.
| ||||||
| Comment by Spencer Brody (Inactive) [ 28/Sep/15 ] | ||||||
|
Agreed, I phrased the original description poorly. I'm most interested in making sure we get feedback from jstests as soon as possible, so unit tests taking long or more importantly failing shouldn't stop you from seeing if the jstests pass. | ||||||
| Comment by Max Hirschhorn [ 25/Sep/15 ] | ||||||
|
I'd be in favor of splitting out running the C++ unit tests from the compile task as long as we don't make the unit tests passing a prerequisite for running any of the other tests. The situation I've seen a few times is a bit different from what Spencer described: someone commits code to master that breaks one of the unit tests and causes a bunch of commits that queue up after it to not run any additional tests. The unit tests take approximately 30 seconds to run with a sufficient --jobs argument to resmoke.py. From that perspective it wouldn't be worthwhile to split the unit tests out into their own task because scheduling a new task in Evergreen takes around that long. Additionally, we'd need to store the unit test binaries in S3 so that another task could fetch them. My proposal is to make compiling and running the unit tests its own task. This has the additional benefit of decreasing the time for the compile task because it won't need to build the unittests target anymore. If the code is so horribly broken, then running jsCore should catch it (~4 minutes on Linux 64-bit), and given the changes proposed in | ||||||
| Comment by Spencer Brody (Inactive) [ 25/Sep/15 ] | ||||||
|
I personally don't think so. 9 times out of 10 unit tests failures are due to someone forgetting to update the unit tests along with their code change. In that case, it'd be nice to know that all the jstests passed and so when you fix the unit test you could just run the unit test suite before being able to push. | ||||||
| Comment by Ernie Hershey [ 25/Sep/15 ] | ||||||
|
Is this in opposition to I see the point about this making the cost of new unit tests higher than it should be - delaying jstest starts. | ||||||
| Comment by Michael O'Brien [ 08/Sep/15 ] | ||||||
|
spencer moved this to SERVER, since this would require a change to the .evergreen.yml file and not a change in evergreen itself. |