[SERVER-31778] Test master with WiredTiger develop Created: 08/May/17 Updated: 30/Oct/23 Resolved: 29/Jan/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | 3.7.2 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Ramon Fernandez Marina | Assignee: | Luke Chen |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Sprint: | Storage 2017-12-04, Storage 2018-01-01, Storage 2018-01-15, Storage 2018-01-29 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
This ticket is to add a variant to evergreen.yml in master that tests against the develop branch in WiredTiger. The goal is to provide early signal on WT changes before we import them into master and other branches. |
| Comments |
| Comment by Ian Whalen (Inactive) [ 29/Jan/18 ] |
|
I don't think we need to change anything - just giving tess.avitabile a heads up that this landed and any failures on the new buildvariant can probably be sent straight to backlog-server-execution. |
| Comment by Luke Chen [ 29/Jan/18 ] |
|
ian.whalen The changes just landed into master branch. Failures specific to the 2 new WiredTiger develop variants are expected to go to WiredTiger team. Will BB process needs to be updated to reflect this change? |
| Comment by Githook User [ 29/Jan/18 ] |
|
Author: {'email': 'luke.chen@mongodb.com', 'name': 'Luke Chen', 'username': 'lukech'}Message: |
| Comment by Ian Whalen (Inactive) [ 19/Jan/18 ] |
|
agreed, I'm really excited about this! |
| Comment by Luke Chen [ 19/Jan/18 ] |
|
Ramón helped to point out the wtdevelop directory (from wiredtiger git repo) does not contain SConscript and build_* directories which are essential for MongoDB build. The solution is to selectively copy those source code directories from wtdevelop to wiredtiger directory. Here is the successful patch build (https://evergreen.mongodb.com/version/5a61693dc9ec44723d001d1f) for the 2 new WiredTiger develop variants:
|
| Comment by Luke Chen [ 12/Jan/18 ] |
|
After another discussion with ramon.fernandez, alexander.gorrod and I were well convinced to start with the solution of adding a couple of new wtdevelop variants in mongodb-mongo-master project, which has the benefits of less efforts, relatively less noise to BB (and reuse of BB tools), ease of failure correlation (by comparing with other non-wtdevelop variants), etc. Though not providing full-run capability (all mongodb-mongo-master variants) with wtdevelop codebase, it is a good starting point we can get to implement quickly, and evaluate later based on team experiences. Thanks much ramon.fernandez! |
| Comment by Ramon Fernandez Marina [ 03/Jan/18 ] |
|
In terms of setup, my gut feeling is that having a forked repo and a new Evergreen project is an overly complicated setup, since I believe we can achieve what we need with just a new variant (maybe two if we wanted to test Windows as well) in the existing repo. When I first started this work, max.hirschhorn's suggestion was to (and I'm paraphrasing here) make better groups of tests and then use those groups in different variants to simplify maintenance and have comprehensive coverage. My initial patch worked, but didn't do any test fine-tuning or cleanup of tasks/groups. |
| Comment by Luke Chen [ 22/Dec/17 ] |
|
I discussed with alexander.gorrod about test set required:
It would be clear to have a new Evergreen project created for the WiredTiger develop branch testing:
In order to keep up with periodic variant/task updates of etc/evergreen.yml in MongoDB (master) Evergreen project, we are thinking of writing a script to do below tasks against the fork repo, and let it run once-per-hour:
max.hirschhorn and ramon.fernandez, your comments are very welcome. |
| Comment by Michael Cahill (Inactive) [ 31/Oct/17 ] |
|
ian.whalen, I think we could just convert this to a SERVER ticket, make sure max.hirschhorn is a reviewer and get on with it. I doubt that anyone who hasn't already participated here will have an opinion about this work. |
| Comment by Ramon Fernandez Marina [ 31/Oct/17 ] |
|
Tentatively assigning this to luke.chen after a conversation with michael.cahill. Luke, there's a code review above that should be a reasonable starting point (in terms of Evergreen mechanics). The rest of the work is coming up with a sane subset of changes to the test subset. I'd argue that we should not let best be the enemy of good here and just start testing the subset in that code review, then iterate, but now it's up to you to convince people this is a good way forward – good luck |
| Comment by Alexander Gorrod [ 11/Jul/17 ] |
Nifty - is that to ensure builds work with the enterprise modules?
Ideally, but any step in this direction is going to be helpful, so if that makes more work, I'm happy not to include it for now.
Your suggestion makes sense to me. |
| Comment by Max Hirschhorn [ 10/Jul/17 ] |
Just to mention it explicitly: The existing Evergreen stepback semantics around the manifest should give you this behavior, as earlier versions of the wiredtiger/wiredtiger/develop branch would have been associated with earlier mongodb/mongo/master commits on the 1-hour batchtime builders and them stepping back.
alexander.gorrod, does that mean you'd want a Windows version (--dbg=on or otherwise) as well?
If you're concerned about needing to maintain an explicitly list of tasks, then my suggestion would be to define a YAML anchor for the list of tasks run on the "Enterprise RHEL 6.2 (inMemory)" builder and then cheat by having the "Enterprise RHEL 6.2 WiredTiger develop" builder's ${test_flags} expansion set --storageEngine=wiredTiger to ensure we don't run with the MMAPv1 storage engine. I typically request engineers add new tasks to the "Enterprise RHEL 6.2 (inMemory)" builder so I'm less worried about failing to maintain that list. (See this code review comment for my canonical list of build variants.) |
| Comment by Alexander Gorrod [ 10/Jul/17 ] |
|
Thanks ramon.fernandez - the approach looks good to me. max.hirschhorn asked some philosophilcal questions in the code review, I'm going to give my opinion here:
The reasons I'd like to have this variant run regularly in Evergreen are three fold: I'd request that it is run daily, since there are generally less than 5 commits into WiredTiger develop each day. Or even more ideally triggered by commits into WiredTiger develop, but at most once every 8 hours.
I agree that excluding MMAPv1 tasks would be nice. I used to maintain a set of WiredTiger specific test cases, but it's difficult to maintain as test names and suites evolve over time. As with all automated testing there is tension between missing test coverage and wasting testing resources. Maintaining a manual set of tasks leads to quite a high level of risk that we won't get the best test coverage possible. |
| Comment by Ramon Fernandez Marina [ 07/Jul/17 ] |
|
FTR, my suggested change only addresses correctness. I'll look at sys-perf next week. |
| Comment by Alexander Gorrod [ 19/May/17 ] |
|
That makes me a bit sad - it would be helpful to have the functionality available leading up to the 3.6 code freeze deadline, but I understand we missed the project planning boat. |
| Comment by Ian Whalen (Inactive) [ 19/May/17 ] |
|
@watchers on this ticket: The Perf team has discussed and believes that the work necessary to accomplish this is uncertain enough in scope that it should go into the Project backlog and get prioritized/scheduled like any other work there. That means we'll come back to prioritizing this in the mid-July planning session unless someone gives a shout that this should get prioritized up. |
| Comment by Alexander Gorrod [ 08/May/17 ] |
The WiredTiger sys-perf jobs are the ones we are most interested in. An example of the last patch build I ran is:
I'd say daily, as long as there have been changes since the previous run would be ideal. I'd like the ability to manually trigger if I'm keener as well (as I can do with Evergreen at the moment). For non-develop branches I expect it to run rarely, since we only update the content of those branches rarely.
I'd like it automated, ideally I won't need to start patch builds when doing drops from WiredTiger into MongoDB master any more.
I think I'd like it the same, it can probably share the mainline baselines as well. |
| Comment by David Daly [ 08/May/17 ] |
|
Let's talk about frequency and goals for this. We could enable for sys-perf and or microbenchmarks. It will get a little messy as we will be tracking mongo master and wiredtiger for it, but there's no reason we can't do it. Questions:
|