[SERVER-31778] Test master with WiredTiger develop Created: 08/May/17  Updated: 30/Oct/23  Resolved: 29/Jan/18

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: 3.7.2

Type: New Feature Priority: Major - P3
Reporter: Ramon Fernandez Marina Assignee: Luke Chen
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-33647 Enterprise Windows 2008R2 WiredTiger ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Storage 2017-12-04, Storage 2018-01-01, Storage 2018-01-15, Storage 2018-01-29
Participants:

 Description   

This ticket is to add a variant to evergreen.yml in master that tests against the develop branch in WiredTiger.

The goal is to provide early signal on WT changes before we import them into master and other branches.



 Comments   
Comment by Ian Whalen (Inactive) [ 29/Jan/18 ]

I don't think we need to change anything - just giving tess.avitabile a heads up that this landed and any failures on the new buildvariant can probably be sent straight to backlog-server-execution.

Comment by Luke Chen [ 29/Jan/18 ]

ian.whalen The changes just landed into master branch. Failures specific to the 2 new WiredTiger develop variants are expected to go to WiredTiger team. Will BB process needs to be updated to reflect this change?

Comment by Githook User [ 29/Jan/18 ]

Author:

{'email': 'luke.chen@mongodb.com', 'name': 'Luke Chen', 'username': 'lukech'}

Message: SERVER-31778 Test master with WiredTiger develop
Branch: master
https://github.com/mongodb/mongo/commit/055b329dbd215d5020c34aa1696b4186abcfc9f4

Comment by Ian Whalen (Inactive) [ 19/Jan/18 ]

agreed, I'm really excited about this!

Comment by Luke Chen [ 19/Jan/18 ]

Ramón helped to point out the wtdevelop directory (from wiredtiger git repo) does not contain SConscript and build_* directories which are essential for MongoDB build. The solution is to selectively copy those source code directories from wtdevelop to wiredtiger directory.

Here is the successful patch build (https://evergreen.mongodb.com/version/5a61693dc9ec44723d001d1f) for the 2 new WiredTiger develop variants:

  • ~ Linux DEBUG WiredTiger develop
  • ~ Enterprise Windows 2008R2 WiredTiger develop
Comment by Luke Chen [ 12/Jan/18 ]

After another discussion with ramon.fernandez, alexander.gorrod and I were well convinced to start with the solution of adding a couple of new wtdevelop variants in mongodb-mongo-master project, which has the benefits of less efforts, relatively less noise to BB (and reuse of BB tools), ease of failure correlation (by comparing with other non-wtdevelop variants), etc. Though not providing full-run capability (all mongodb-mongo-master variants) with wtdevelop codebase, it is a good starting point we can get to implement quickly, and evaluate later based on team experiences. Thanks much ramon.fernandez!

Comment by Ramon Fernandez Marina [ 03/Jan/18 ]

In terms of setup, my gut feeling is that having a forked repo and a new Evergreen project is an overly complicated setup, since I believe we can achieve what we need with just a new variant (maybe two if we wanted to test Windows as well) in the existing repo.

When I first started this work, max.hirschhorn's suggestion was to (and I'm paraphrasing here) make better groups of tests and then use those groups in different variants to simplify maintenance and have comprehensive coverage. My initial patch worked, but didn't do any test fine-tuning or cleanup of tasks/groups.

Comment by Luke Chen [ 22/Dec/17 ]

I discussed with alexander.gorrod about test set required:

  • those critical ! and important * variants in MongoDB (master) Evergreen project are needed for an once-per-day run
  • other normal variants might be needed depending on the WiredTiger changes, and can be un-scheduled by default

It would be clear to have a new Evergreen project created for the WiredTiger develop branch testing:

In order to keep up with periodic variant/task updates of etc/evergreen.yml in MongoDB (master) Evergreen project, we are thinking of writing a script to do below tasks against the fork repo, and let it run once-per-hour:

  • rollback local changes (if any)
  • update master branch to be up-to-date with upstream (pulling latest changes of MongoDB master branch, including etc/evergreen.yml)
  • apply WiredTiger code changes (pulling latest changes of WiredTiger develop branch into src/third_party/wiredtiger/)
  • make etc/evergreen.yml adjustments (local changes), similar as Ramon's change for RHEL6.2 variant, but applies wtdevelop module to all variants, also make ! and * variants scheduled once per day, other variant un-scheduled

max.hirschhorn and ramon.fernandez, your comments are very welcome.

Comment by Michael Cahill (Inactive) [ 31/Oct/17 ]

ian.whalen, I think we could just convert this to a SERVER ticket, make sure max.hirschhorn is a reviewer and get on with it. I doubt that anyone who hasn't already participated here will have an opinion about this work.

Comment by Ramon Fernandez Marina [ 31/Oct/17 ]

Tentatively assigning this to luke.chen after a conversation with michael.cahill.

Luke, there's a code review above that should be a reasonable starting point (in terms of Evergreen mechanics). The rest of the work is coming up with a sane subset of changes to the test subset. I'd argue that we should not let best be the enemy of good here and just start testing the subset in that code review, then iterate, but now it's up to you to convince people this is a good way forward – good luck

Comment by Alexander Gorrod [ 11/Jul/17 ]

Just to mention it explicitly: The existing Evergreen stepback semantics around the manifest should give you this behavior

Nifty - is that to ensure builds work with the enterprise modules?

does that mean you'd want a Windows version (--dbg=on or otherwise) as well?

Ideally, but any step in this direction is going to be helpful, so if that makes more work, I'm happy not to include it for now.

If you're concerned about needing to maintain an explicitly list of tasks

Your suggestion makes sense to me.

Comment by Max Hirschhorn [ 10/Jul/17 ]

1) It will allow the WiredTiger team to see test failures uncovered only by MongoDB tests closer to when the commit that triggered them is made. Which makes it much faster to isolate the root cause, and reduces the risk that we end up with bad storage changes in master.

Just to mention it explicitly: The existing Evergreen stepback semantics around the manifest should give you this behavior, as earlier versions of the wiredtiger/wiredtiger/develop branch would have been associated with earlier mongodb/mongo/master commits on the 1-hour batchtime builders and them stepping back.

2) It will mean there is less manual work that needs to be done prior to updating the version of WiredTiger in MongoDB - to that end I'd ideally like this target to run on all the ! variants.

alexander.gorrod, does that mean you'd want a Windows version (--dbg=on or otherwise) as well?

I agree that excluding MMAPv1 tasks would be nice. I used to maintain a set of WiredTiger specific test cases, but it's difficult to maintain as test names and suites evolve over time. As with all automated testing there is tension between missing test coverage and wasting testing resources. Maintaining a manual set of tasks leads to quite a high level of risk that we won't get the best test coverage possible.

If you're concerned about needing to maintain an explicitly list of tasks, then my suggestion would be to define a YAML anchor for the list of tasks run on the "Enterprise RHEL 6.2 (inMemory)" builder and then cheat by having the "Enterprise RHEL 6.2 WiredTiger develop" builder's ${test_flags} expansion set --storageEngine=wiredTiger to ensure we don't run with the MMAPv1 storage engine. I typically request engineers add new tasks to the "Enterprise RHEL 6.2 (inMemory)" builder so I'm less worried about failing to maintain that list. (See this code review comment for my canonical list of build variants.)

Comment by Alexander Gorrod [ 10/Jul/17 ]

Thanks ramon.fernandez - the approach looks good to me.

max.hirschhorn asked some philosophilcal questions in the code review, I'm going to give my opinion here:

My impression is that we're going to run a patch build against this build variant before doing the WiredTiger import, so I'm not sure there's a reason to make this too often.

The reasons I'd like to have this variant run regularly in Evergreen are three fold:
1) It will allow the WiredTiger team to see test failures uncovered only by MongoDB tests closer to when the commit that triggered them is made. Which makes it much faster to isolate the root cause, and reduces the risk that we end up with bad storage changes in master.
2) It will mean there is less manual work that needs to be done prior to updating the version of WiredTiger in MongoDB - to that end I'd ideally like this target to run on all the ! variants.
3) It brings automated testing integration for the WiredTiger project one step closer.

I'd request that it is run daily, since there are generally less than 5 commits into WiredTiger develop each day. Or even more ideally triggered by commits into WiredTiger develop, but at most once every 8 hours.

I don't think we want to run everything that Enterprise RHEL 6.2 runs because
that includes tasks which run with the MMAPv1 storage engine. I would have liked
to recommend the set of tasks we use for the InMemory storage engine
(https://github.com/mongodb/mongo/blob/r3.5.9/etc/evergreen.yml#L8078-L8152) as
an alternate starting point; however, due to how some of the Evergreen task
dependencies work, we aren't consistent about using the "_WT"-suffixed tasks in
that list so some manual fix-up would be required.

I agree that excluding MMAPv1 tasks would be nice. I used to maintain a set of WiredTiger specific test cases, but it's difficult to maintain as test names and suites evolve over time. As with all automated testing there is tension between missing test coverage and wasting testing resources. Maintaining a manual set of tasks leads to quite a high level of risk that we won't get the best test coverage possible.

Comment by Ramon Fernandez Marina [ 07/Jul/17 ]

FTR, my suggested change only addresses correctness. I'll look at sys-perf next week.

Comment by Alexander Gorrod [ 19/May/17 ]

That makes me a bit sad - it would be helpful to have the functionality available leading up to the 3.6 code freeze deadline, but I understand we missed the project planning boat.

Comment by Ian Whalen (Inactive) [ 19/May/17 ]

@watchers on this ticket: The Perf team has discussed and believes that the work necessary to accomplish this is uncertain enough in scope that it should go into the Project backlog and get prioritized/scheduled like any other work there. That means we'll come back to prioritizing this in the mid-July planning session unless someone gives a shout that this should get prioritized up.

Comment by Alexander Gorrod [ 08/May/17 ]

Which projects?

The WiredTiger sys-perf jobs are the ones we are most interested in. An example of the last patch build I ran is:
https://evergreen.mongodb.com/version/59034a432fbabe26f5006754#?59034a432fbabe26f5006754=3.2.12-Baseline

How often?

I'd say daily, as long as there have been changes since the previous run would be ideal. I'd like the ability to manually trigger if I'm keener as well (as I can do with Evergreen at the moment).

For non-develop branches I expect it to run rarely, since we only update the content of those branches rarely.

Do we want it automated, or are patch builds sufficient?

I'd like it automated, ideally I won't need to start patch builds when doing drops from WiredTiger into MongoDB master any more.

Do we want to treat it different than the mainline projects

I think I'd like it the same, it can probably share the mainline baselines as well.

Comment by David Daly [ 08/May/17 ]

Let's talk about frequency and goals for this. We could enable for sys-perf and or microbenchmarks. It will get a little messy as we will be tracking mongo master and wiredtiger for it, but there's no reason we can't do it.

Questions:

  • Which projects?
  • How often?
    • Do we want it automated, or are patch builds sufficient?
    • Do we want to treat it different than the mainline projects (possibly because of difference of batch time, or how we want to triage)?
Generated at Thu Feb 08 04:28:09 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.