[SERVER-53951] Add a mode to resmoke to bisect changes Created: 21/Jan/21  Updated: 05/Apr/23  Resolved: 15/Jun/21

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Robert Guo (Inactive)
Resolution: Duplicate Votes: 1
Labels: dp-qp-stakeholder-request-2021-04, dp-qp-stakeholder-request-2021-07, tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File bisect.sh    
Issue Links:
Duplicate
duplicates SERVER-57730 Resmoke.py should bisect non-multiver... Closed
Related
Participants:
Story Points: 0

 Description   

Add a resmoke bisect sub-command that wraps around git bisect. It'll compile the Server and run resmoke with a repeat of 100 to capture flakiness tests under git-bisect.

We need users to be able to configure the compile and resmoke commands based on git commits. Something like the following:

- startGitVersion: xyz
   setupCommand: source ./script-to-setup-venv-and-install-pip-dependencies
   compileCommand: scons ijk
   resmokeCommand: resmoke.py abc
- startGitVersion: xyz
   compileCommand: scons ijk
   resmokeCommand: resmoke.py abc

  • We will need to make a template of the config YAML with global changes to resmoke and compile steps, e.g. resmoke.py -> resmoke.py run.
  • The script will need to be tested on Ubuntu 1804 (which is the majority of virtual workstations). We'll make a best-effort to ensure it works cross platform by avoid using platform-specific code where possible. (We do not intend for this to be run outside of virtual workstations, as it would take a very long time due to the lack of icecream cluster access.)


 Comments   
Comment by Robert Guo (Inactive) [ 15/Jun/21 ]

Apologies for the delay on the response. After thinking about the work involved here and mulling over the convo we had in March, I'm realizing that even for the MVP, there's quite a bit of work involved and we will likely need to prioritize this work at the project-level. In more detail, here's some of the complexity.

  1. Evergreen tasks are defined in evergreen.yml, not resmoke, so to rerun a given task, we would have to parse the YAML for every commit to know exactly what gets run for a task for a given commit. This has unknowable complexity given the size of that file. Alternatively, we could check for resmoke's output for local invocation in the Evergreen logs, but we only retain logs of successful tasks for 30 days.
  2. Older environments, especially pip packages, are no longer available, so resmoke.py may or may not run for a given commit
  3. All of the multiversion tasks need the commit for the old version, which we don't pin and always use the latest version. This is not available after task logs are gone. Alternatively, we can analyze the completion time of Evergreen's push tasks to find the best approximate binary.
  4. Many tasks, including most of the background passthroughs, will not run correctly for a single file, even for somewhat deterministic failures, since the passthroughs do not necessarily run to completion on the first file.

After chatting with charlie.swanson about the fuzzer minimizer project (PM-1880) a few weeks ago and how bisection would be useful there, I added SERVER-57730 to the project to bisect a subset of fuzzer failures. In addition to fuzzers, this script should work across a fair number of other suites that have not changed over the years. This approach will allow us to better receive feedback on the most user-requested bisection features on top of the basic bisect script without having to have full support of all the edge cases (for the MVP) upfront, which is difficult for STM to prioritize given our limited resources.

If you have concerns or questions, please don't hesitate to reach out to me on Slack. Closing this ticket as a dupe of SERVER-57730

Comment by Ian Whalen (Inactive) [ 28/May/21 ]

Got it.  TY for the context!

Comment by Ian Whalen (Inactive) [ 28/May/21 ]

Hey all, I was a little sad to see the bisector get bumped back by a few sprints.  We're definitely sure that the things pre-empting it are higher priority?

Comment by Robert Guo (Inactive) [ 31/Mar/21 ]

To summarize the discussion from earlier this month, the problem Svilen described was that a problem exists in release A and we want to know if the problem also exists in release B. An analogous case is where an issue was solved in release A and we want to know if it was fixed in release B. Some practical considerations:

  • Versions A and B are likely on different branches
  • Version A may be many years old with expired Evergreen compiled binaries and possibly no shared supported OS versions with version B

We will try to address a subset of the concerns first; the second one being the more urgent one. Some possible solutions include

  • Raiden's idea above of relying on the retry logic in setup_multiversion to download existing binaries on nearby available commits
  • enforcing documentation of changes to the resmoke invocation, scons compile invocation and environment setup (we already do the latter 2 for fuzzer self-tests) so vanilla git-bisect can succeed.
Comment by Raiden Worley (Inactive) [ 09/Mar/21 ]

robert.guo and I discussed a design that would use parameterized patch builds. We would create an evergreen task that normally no-ops, but if given two commits and a test as arguments from the CLI would do the bisect.

We could make this a faster by downloading binaries from existing compiles (where available). Resmoke's setup_multiversion subcommand already does this using the evergreen API, so we may be able to generalize the existing script. Not every commit has compilations, but we could use a custom bisect algorithm that privileges commits that have compiles, so we'd only have to actually compile at the end when bisecting between the last two waterfall compiles.

svilen.mihaylov we were wondering if this would match your use case? And we were curious what the use case for a patch-build approach would be over using a spawned host or virtual workstation.

 

Comment by Maria van Keulen [ 09/Mar/21 ]

svilen.mihaylov james.wahlin While it seems plausible that offloading the compilation work to Evergreen would reduce execution time of the bisect, thanks to the recent compile optimizations, Evergreen itself does not have the ability to choose a singular resmoke test in a suite and execute it. Nor would it have the ability to execute a new resmoke test that hadn't existed at the time of a given commit in the bisect. Therefore, this feature would be separate from Evergreen, although we could explore offloading the compile logic to Evergreen and/or downloading compile artifacts as a future optimization.

I believe we discussed in our meeting a while back that the MVP for this feature would be a script similar to the one Charlie has posted above, but with robust detection of changes in compile logic and resmoke logic across release versions. There are spawn host script optimizations (EVG-5856, EVG-13010) which can automate running of this script on a spawn host, assuming the platforms are compatible. In the short term, I imagine the workflow would be to paste the server spawn host setup script contents, followed by the resmoke bisect invocation, into the user-defined setup script box from EVG-5856. CC robert.guo

Comment by Brooke Miller [ 09/Mar/21 ]

FYI james.wahlin, robert.guo will be setting up time with you to sync up on this next week.

Also, in doing this ticket, we need to evaluate which team is responsible for the maintenance of this templated yaml file after this quick-win. We also should send an email to DB Contributors once this is in prod to make people aware of the new feature.

Comment by Charlie Swanson [ 29/Jan/21 ]

Hi all. I just found my bisect script I have written and used recently. I uploaded it here. I think I used it to track down a change between 4.2 and 4.4 branches somewhere. It's pretty ugly and complex, but I think it serves well to illustrate some of the problems we run into when trying to run git bisect. In my case the 'resmoke.ini' file changed and caused issues, as did the name of the build targets and the build target install directories.

Generated at Thu Feb 08 05:32:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.