[SERVER-53951] Add a mode to resmoke to bisect changes Created: 21/Jan/21 Updated: 05/Apr/23 Resolved: 15/Jun/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Robert Guo (Inactive) | Assignee: | Robert Guo (Inactive) |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | dp-qp-stakeholder-request-2021-04, dp-qp-stakeholder-request-2021-07, tig-resmoke | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Participants: | |||||||||||||
| Story Points: | 0 | ||||||||||||
| Description |
|
Add a resmoke bisect sub-command that wraps around git bisect. It'll compile the Server and run resmoke with a repeat of 100 to capture flakiness tests under git-bisect. We need users to be able to configure the compile and resmoke commands based on git commits. Something like the following:
|
| Comments |
| Comment by Robert Guo (Inactive) [ 15/Jun/21 ] |
|
Apologies for the delay on the response. After thinking about the work involved here and mulling over the convo we had in March, I'm realizing that even for the MVP, there's quite a bit of work involved and we will likely need to prioritize this work at the project-level. In more detail, here's some of the complexity.
After chatting with charlie.swanson about the fuzzer minimizer project (PM-1880) a few weeks ago and how bisection would be useful there, I added If you have concerns or questions, please don't hesitate to reach out to me on Slack. Closing this ticket as a dupe of |
| Comment by Ian Whalen (Inactive) [ 28/May/21 ] |
|
Got it. TY for the context! |
| Comment by Ian Whalen (Inactive) [ 28/May/21 ] |
|
Hey all, I was a little sad to see the bisector get bumped back by a few sprints. We're definitely sure that the things pre-empting it are higher priority? |
| Comment by Robert Guo (Inactive) [ 31/Mar/21 ] |
|
To summarize the discussion from earlier this month, the problem Svilen described was that a problem exists in release A and we want to know if the problem also exists in release B. An analogous case is where an issue was solved in release A and we want to know if it was fixed in release B. Some practical considerations:
We will try to address a subset of the concerns first; the second one being the more urgent one. Some possible solutions include
|
| Comment by Raiden Worley (Inactive) [ 09/Mar/21 ] |
|
robert.guo and I discussed a design that would use parameterized patch builds. We would create an evergreen task that normally no-ops, but if given two commits and a test as arguments from the CLI would do the bisect. We could make this a faster by downloading binaries from existing compiles (where available). Resmoke's setup_multiversion subcommand already does this using the evergreen API, so we may be able to generalize the existing script. Not every commit has compilations, but we could use a custom bisect algorithm that privileges commits that have compiles, so we'd only have to actually compile at the end when bisecting between the last two waterfall compiles. svilen.mihaylov we were wondering if this would match your use case? And we were curious what the use case for a patch-build approach would be over using a spawned host or virtual workstation.
|
| Comment by Maria van Keulen [ 09/Mar/21 ] |
|
svilen.mihaylov james.wahlin While it seems plausible that offloading the compilation work to Evergreen would reduce execution time of the bisect, thanks to the recent compile optimizations, Evergreen itself does not have the ability to choose a singular resmoke test in a suite and execute it. Nor would it have the ability to execute a new resmoke test that hadn't existed at the time of a given commit in the bisect. Therefore, this feature would be separate from Evergreen, although we could explore offloading the compile logic to Evergreen and/or downloading compile artifacts as a future optimization. I believe we discussed in our meeting a while back that the MVP for this feature would be a script similar to the one Charlie has posted above, but with robust detection of changes in compile logic and resmoke logic across release versions. There are spawn host script optimizations (EVG-5856, EVG-13010) which can automate running of this script on a spawn host, assuming the platforms are compatible. In the short term, I imagine the workflow would be to paste the server spawn host setup script contents, followed by the resmoke bisect invocation, into the user-defined setup script box from EVG-5856. CC robert.guo |
| Comment by Brooke Miller [ 09/Mar/21 ] |
|
FYI james.wahlin, robert.guo will be setting up time with you to sync up on this next week. Also, in doing this ticket, we need to evaluate which team is responsible for the maintenance of this templated yaml file after this quick-win. We also should send an email to DB Contributors once this is in prod to make people aware of the new feature. |
| Comment by Charlie Swanson [ 29/Jan/21 ] |
|
Hi all. I just found my bisect script I have written and used recently. I uploaded it here. I think I used it to track down a change between 4.2 and 4.4 branches somewhere. It's pretty ugly and complex, but I think it serves well to illustrate some of the problems we run into when trying to run git bisect. In my case the 'resmoke.ini' file changed and caused issues, as did the name of the build targets and the build target install directories. |