[SERVER-20315] Separate compile and unit test execution tasks in evergreen config Created: 08/Sep/15  Updated: 05/Feb/16  Resolved: 08/Oct/15

Status: Closed
Project: Core Server
Component/s: Build
Affects Version/s: None
Fix Version/s: 3.1.9

Type: Improvement Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Jonathan Reams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-18718 Move unit tests into a separate task Closed
Related
related to SERVER-20826 Add unittests task to non-debug ASan ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Build A (10/09/15)
Participants:

 Description   

Right now the more unit tests we add, the longer the compile phase runs, and thus the longer before other suites get started. No reason we couldn't run the unit tests in parallel with the jstests and start getting feedback faster.



 Comments   
Comment by Andrew Morrow (Inactive) [ 07/Oct/15 ]

Might as well go ahead and add the 'enterprise' into the name here, as well.

Comment by Kamran K. [ 07/Oct/15 ]

jonathan.reams: Can you add the 'unittests' task to the "~ ASAN SSL Ubuntu 14.10" variant before you close this ticket? It wasn't added as part of the above commit.

Comment by Githook User [ 07/Oct/15 ]

Author:

{u'username': u'jbreams', u'name': u'Jonathan Reams', u'email': u'jbreams@mongodb.com'}

Message: SERVER-20315 Run unit tests in separate MCI task
Branch: master
https://github.com/mongodb/mongo/commit/b536dc562f6bfadab75c8b9d2bc0693553350c1d

Comment by Spencer Brody (Inactive) [ 28/Sep/15 ]

Conclusion of offline conversation is that we will strip the debug symbols and compress the unit tests, then add them to s3 to allow the unit test execution to run as a different task (potentially on a different machine) than the compile task.

Comment by Andrew Morrow (Inactive) [ 28/Sep/15 ]

I'm not thrilled about having the compile step not build the whole tree. I really want to know that everything compiles before we do anything else.

Comment by Max Hirschhorn [ 28/Sep/15 ]

Per suggestion from acm, I ran a patch build that skips building the unit tests by removing it from the all alias. It takes about 8 minutes to build the remaining targets.

[2015/09/28 14:23:15.411] scons: Building targets ...
...
[2015/09/28 14:31:26.031] scons: done building targets.

For comparison, it took about 13 minutes on a recent commit.

[2015/09/28 13:26:01.131] scons: Building targets ...
...
[2015/09/28 13:39:00.381] scons: done building targets.

Comment by Spencer Brody (Inactive) [ 28/Sep/15 ]

Agreed, I phrased the original description poorly. I'm most interested in making sure we get feedback from jstests as soon as possible, so unit tests taking long or more importantly failing shouldn't stop you from seeing if the jstests pass.

Comment by Max Hirschhorn [ 25/Sep/15 ]

I'd be in favor of splitting out running the C++ unit tests from the compile task as long as we don't make the unit tests passing a prerequisite for running any of the other tests. The situation I've seen a few times is a bit different from what Spencer described: someone commits code to master that breaks one of the unit tests and causes a bunch of commits that queue up after it to not run any additional tests.

The unit tests take approximately 30 seconds to run with a sufficient --jobs argument to resmoke.py. From that perspective it wouldn't be worthwhile to split the unit tests out into their own task because scheduling a new task in Evergreen takes around that long. Additionally, we'd need to store the unit test binaries in S3 so that another task could fetch them.

My proposal is to make compiling and running the unit tests its own task. This has the additional benefit of decreasing the time for the compile task because it won't need to build the unittests target anymore. If the code is so horribly broken, then running jsCore should catch it (~4 minutes on Linux 64-bit), and given the changes proposed in SERVER-20581, we wouldn't run many further tests.

Comment by Spencer Brody (Inactive) [ 25/Sep/15 ]

I personally don't think so. 9 times out of 10 unit tests failures are due to someone forgetting to update the unit tests along with their code change. In that case, it'd be nice to know that all the jstests passed and so when you fix the unit test you could just run the unit test suite before being able to push.

Comment by Ernie Hershey [ 25/Sep/15 ]

Is this in opposition to SERVER-20581? Should any jstests wait for unit tests to finish?

I see the point about this making the cost of new unit tests higher than it should be - delaying jstest starts.

Comment by Michael O'Brien [ 08/Sep/15 ]

spencer moved this to SERVER, since this would require a change to the .evergreen.yml file and not a change in evergreen itself.

Generated at Thu Feb 08 03:53:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.