[SERVER-49786] Freeze DSI and Genny for non-master perf projects Created: 21/Jul/20  Updated: 29/Oct/23  Resolved: 12/Aug/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.2.10, 4.0.21

Type: Task Priority: Major - P3
Reporter: Ryan Timmons Assignee: Ryan Timmons
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-50133 Perf YAML Cleanups Closed
Backwards Compatibility: Fully Compatible
Sprint: STM 2020-08-10, STM 2020-08-24
Participants:
Story Points: 1

 Description   

PM-1822 will change the interface between DSI and sys-perf evergreen yamls. To minimize the risk of breaking non-master branches and the overhead of multiple backports during PM-1822, we wish to "freeze" the version of DSI in use on the following projects:

  1. sys-perf v4.4
  2. perf v4.4
  3. sys-perf v4.2
  4. perf v4.2
  5. sys-perf v4.0
  6. perf v4.0
  7. sys-perf v3.6
  8. perf v3.6

To do this we will create a "legacy" branch of DSI and modify non-master evergreen yamls to use this. This means that any changes to DSI will not be usable by old perf projects unless those changes are backported.

At the end of PM-1822 we could selectively bring the required projects back up to DSI master or leave these projects "forever frozen".



 Comments   
Comment by Githook User [ 10/Aug/20 ]

Author:

{'name': 'Ryan Timmons', 'email': 'ryan.timmons@10gen.com', 'username': 'rtimmons'}

Message: SERVER-49786 Freeze DSI and Genny to 'legacy' branch
Branch: v3.6
https://github.com/mongodb/mongo/commit/a7f239476e2e0b17fcfea38e3de3aee5fb9c6081

Comment by Githook User [ 10/Aug/20 ]

Author:

{'name': 'Ryan Timmons', 'email': 'ryan.timmons@10gen.com', 'username': 'rtimmons'}

Message: SERVER-49786 Freeze DSI and Genny to 'legacy' branch
Branch: v4.0
https://github.com/mongodb/mongo/commit/1fe24fa0a441d86155eb619256d79b288c82dd91

Comment by Githook User [ 10/Aug/20 ]

Author:

{'name': 'Ryan Timmons', 'email': 'ryan.timmons@10gen.com', 'username': 'rtimmons'}

Message: SERVER-49786 Freeze DSI and Genny to 'legacy' branch
Branch: v4.2
https://github.com/mongodb/mongo/commit/f6873198b2f89b5b624c9d6caccab86adf237266

Comment by Ryan Timmons [ 27/Jul/20 ]

Thanks all. I'm going to proceed with this unless any objections by COB.

If we had a new test to master, I'd like to at least get it to 4.4 (if it can run on 4.4).

Yes. I'll update the DSI PR template indicating that new workloads should be "backported" to the "legacy" branch. I'll also call it out in #performance slack.

For non breaking changes it would be nice to cherry-pick them to stable/legacy.

I'm less optimistic about this. It's very hard to know what's a breaking change--half the reason we're doing this project. I think we'll consider it on a case-by-case basis depending on what the changes are, but I don't want us to guess wrong and accidentally cause everything to turn purple.

This project isn't huge, and we'll kill 'legacy' at the end of it, so this is all temporary.

Comment by David Daly [ 27/Jul/20 ]

A few quick thoughts as I'm generally fine with this plan

  1. The reason we've kept one branch is to ensure we run exactly the same tests on each branch. This allows us to make fair comparisons on performance results. Note we don't run all tests on all branches, but if we do run a test, it's the same configuration, etc. This is also part of the reason we didn't embed DSI into the server repo at the start. 
  2. This has caused some stiffness on occasion, for instance with new mongodb configuration knobs needed for the server. I hope we could handle that problem without maintaining a branch per stable release. 
  3. I'd like to think more about how we handle 4.4 branch in this plan. We very regularly compare master to most recent stable and I don't want to lose that comparison. Two possible thoughts here: 
    1. If we had a new test to master, I'd like to at least get it to 4.4 (if it can run on 4.4). 
    2. For non breaking changes it would be nice to cherry-pick them to stable/legacy. I don't have the project in my head enough to have a sense of whether there will be clearly non-breaking changes, and other high risk ones, but I could imagine that there are. For instance, if the configuration files don't relocate or significantly change, changes to them should be fine. 
Comment by Eric Milkie [ 25/Jul/20 ]

Thanks Ryan – I like this plan.

Comment by Cristopher Stauffer [ 25/Jul/20 ]

I don't think I have used this recently enough to meaningfully weigh in on it.

Comment by Ryan Timmons [ 24/Jul/20 ]

Yup your summary is exactly right, milkie. Thanks for your input.

Is the lack of matching release branches stifling DSI development? Should we consider branching DSI in the future whenever we create new sys-perf and perf branches?

It has in the past for sure. The problem is the lack of an "API" between DSI and the server repo branches. This has meant we had to patch-test all server branches since each branch did things differently. With the API between DSI and evergreen perf yamls established (and tested) we can freely change the impl without fear of breaking pre-master branches of the server. This eliminates the need for branching DSI with every server release.

I suspect we'll bring v4.0+ up to the latest API once it's established in master. If we no longer need 3.6 at that point we can kill the "legacy" branch and continue like things are now.

Will there be a procedure for requesting backports of future DSI changes?

I was planning to add a note to the DSI PR template mentioning the distinct branches. I think that's sufficient for most DSI asks, but I could outline something more formal if it feels too non-obvious.

Comment by Eric Milkie [ 24/Jul/20 ]

It took me a while to understand what this proposal is. Let me summarize it and please tell me if I'm correct or off base:
1. Rather than branching DSI per server release, as is done for sys-perf and perf repos, we only have one branch (the master branch) that services all released consumers on every branch, and any changes made to DSI must be compatible with all sys-perf and perf branches all the way back to 3.6.
2. We wish to introduce a new change to DSI that will break compatibility with all branches of sys-perf and perf.
3. Rather than make changes to all branches of sys-perf and perf to be compatible with this proposed change, we are instead going to only make changes to the master branches of sys-perf and perf. To avoid making similar changes to older branches of sys-perf and perf, we plan to branch the DSI repo into two (we could have instead made a branch per release version, but this is expected to be overkill.).
4. Going forward, future changes to DSI can now be considered for backport from the master branch to the new legacy branch.

I think this plan is fine, but I have some questions:
1. Is the lack of matching release branches stifling DSI development? Should we consider branching DSI in the future whenever we create new sys-perf and perf branches?
2. Will there be a procedure for requesting backports of future DSI changes?

Comment by Robert Guo (Inactive) [ 24/Jul/20 ]

Some historical data:

for 4.2, we added the ValidateCmd Genny workload (SERVER-41909), sys-bench (SERVER-46746) and optimized linkbench (SERVER-45835) after GA

for 4.0, we added tpc-c (SERVER-35062), cursor manager (SERVER-40246), auth (SERVER-39869) and ValidateCmd after GA

Comment by Ian Whalen (Inactive) [ 24/Jul/20 ]

I don't know enough to give a helpful answer either way.

Comment by Ryan Timmons [ 24/Jul/20 ]

Adding milkie, ian.whalen, and cristopher.stauffer - do any of you have any concerns with freezing DSI for non-master perf projects like this ticket proposes?

Comment by Ryan Timmons [ 24/Jul/20 ]

david.daly, kelsey.schubert, and david.bradford - do any of you have any concerns with freezing DSI for non-master perf projects like this ticket proposes?

Comment by Brooke Miller [ 23/Jul/20 ]

We discussed in triaging that we should come up with a one-pager explaining the value that could be provided by disabling DSI. 

Generated at Thu Feb 08 05:20:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.