[SERVER-33972] Bind jstestfuzz version to server version Created: 19/Mar/18  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Backlog - Server Tooling and Methods (STM) (Inactive)
Resolution: Unresolved Votes: 0
Labels: stm
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Tooling & Methods
Participants:
Story Points: 2

 Description   

As the fuzzer becomes more dependent on code in the server repo, like the rollback fixture and the impending preamble hooks, it would be nice to not have to make a change in the server, wait a while, change the fuzzer, wait a bit more and go back to the server to remove code left in place to support the old fuzzer behavior.

We should be able to do by making jstestfuzz a module in evergreen.yml



 Comments   
Comment by Steven Vannelli [ 10/May/22 ]

Moving this ticket to the Backlog and removing the "Backlog" fixVersion as per our latest policy for using fixVersions.

Comment by Max Hirschhorn [ 02/May/19 ]

Every time I come back to think about this ticket, I'm skeptical about doing it. Now that EVG-727 is resolved, having the 10gen/jstestfuzz repository as a module would mean that every mongodb/mongo commit has the "correct" version of the rollback_test.js fixture. There wouldn't be any rough edges around stepback.

I acknowledge the dance required to be able to do SERVER-40469 and not break the rollback fuzzer is awkward. However, I worry as we add more fuzzers which run on more branches that having the 10gen/jstestfuzz repository be a module isn't enough to solve the underlying issue of how it and the mongodb/mongo are loosely coupled. For better or worse, we won't always backport the changes to the rollback_test.js so what should happen if the API across branches diverges? Maybe a different answer is "they shouldn't" and we only have a single version of rollback_test.js across all branches?

The only end state I really care about is that the fuzzer doesn't have separate branches corresponding to each branch of the mongodb/mongo repository. Maintaining several different versions of resmoke.py across all supported branches has proven to be challenging.

Comment by Robert Guo (Inactive) [ 04/Apr/18 ]

I'm moving the ticket over to 4.1 desired due to additional discussion that came out of the CR. We can revisit this again after 4.0

Comment by Robert Guo (Inactive) [ 30/Mar/18 ]

This ticket will result in some unintended side effects for changing the fuzzer and running fuzzer tests. The intention of this ticket is to solve the problem of having to go back and forth when making changes to the server that the fuzzer depends on. When making any change that requires coordination between the server and the fuzzer, the existing workflow requires the following steps

1. change the server code on master and keep the old server code
1a. backport the server change to other branches
2. wait until all fuzzer tasks for all previous commits have run
3. change the fuzzer code to support the new server code, keep the old fuzzer code for compatibility with older server branches if needed
4. remove the old server code to support the old fuzzer code, backport to all the branches in step 1a.

This ticket will allow fuzzer changes to follow the same workflow as the enterprise modules and remove the need for steps 2 and 4. The new workflow will look like:
1. Change server code
1a. At the same time, backport all changes
2. Immediately afterwards, change fuzzer code

This workflow is inherently racy and can result in failures in a small number of commits (usually 1) that start between when the server and the fuzzer code is pushed. A similar problem exists for the enterprise module and comes up infrequently. There is discussion on possible fixes in EVG-727 but no conclusion yet. None of the proposed solutions in EVG-727 so far affects the efficacy of this ticket.

A notable caveat to this new workflow is that the changes to server master, backports, and changes to the fuzzer repo need to happen simultaneously. Because the fuzzer repo is not versioned according to the server codebase like other modules. If the backports don't happen immediately, the existing 4 step workflow still needs to be followed.

As for unintended consequences of doing the work in this ticket, first is that people can no longer restart failing fuzzer tasks caused by fuzzer bugs (not server bugs) and have them succeed as a result of pulling in an updated fuzzer version. This should be the desirable behavior and is consistent with that of other modules.

Another side-effect/benefit is that we can now use evergreen patch-set-module for patch builds to the fuzzer repo instead of making manual changes to use a forked repo to evergreen,yml. This again seems desirable to me.

Finally, stepbacks need to be turned off for fuzzer tasks to prevent a server bug from stepping back to previous tasks from using an incompatible/outdated version of the fuzzer.

Generated at Thu Feb 08 04:35:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.