[SERVER-53886] jstestfuzz (mutational) fuzzer is forcing latest FCV during validate in multiversion tests Created: 19/Jan/21  Updated: 29/Oct/23  Resolved: 27/Oct/21

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 5.2.0

Type: Bug Priority: Major - P3
Reporter: Benety Goh Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 0
Labels: tig-multiversion
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-57003 Generate all tasks for a build varian... Closed
Related
is related to SERVER-53359 jstestfuzz (mutational) fuzzer is for... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: STM 2021-08-23, STM 2021-09-06, STM 2021-11-01
Participants:
Linked BF Score: 47
Story Points: 2

 Description   

The jstestfuzz*.yml changes in SERVER-53359 resulted in the fuzzer tests using the latest FCV (currently 4.9) in non-multiversion fuzzer test runs. Unfortunately, these configuration changes also affect the multiversion suites where last-lts might be a more suitable FCV to use during validate.



 Comments   
Comment by Githook User [ 27/Oct/21 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-53886 Don't force latest FCV during validate in jstestfuzz multiversion tests
Branch: master
https://github.com/mongodb/mongo/commit/dbc4453af53117b246b65541a84241a1ac044c17

Comment by Robert Guo (Inactive) [ 22/Sep/21 ]

Going to have to bump this out again. We still need to wait on SERVER-55857 so multiversion tasks and fuzzer tasks are not mutually exclusively generated.

Comment by Robert Guo (Inactive) [ 28/Jun/21 ]

There seems to be some errors in the multiversion task generation logic for fuzzers. I didn't look into it further since this code will largely be rewritten in SERVER-57003. FYI david.bradford

I'm going to commit the change on the jstestfuzz side and mark this ticket as dependent on SERVER-57003.

 [2021/06/28 16:56:53.914] Traceback (most recent call last):
 [2021/06/28 16:56:53.914]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/src/buildscripts/evergreen_gen_multiversion_tests.py", line 451, in <module>
 [2021/06/28 16:56:53.914]     main()  # pylint: disable=no-value-for-parameter
 [2021/06/28 16:56:53.914]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
 [2021/06/28 16:56:53.914]     return self.main(*args, **kwargs)
 [2021/06/28 16:56:53.914]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/venv/lib/python3.9/site-packages/click/core.py", line 782, in main
 [2021/06/28 16:56:53.915]     rv = self.invoke(ctx)
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
 [2021/06/28 16:56:53.915]     return _process_result(sub_ctx.command.invoke(sub_ctx))
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
 [2021/06/28 16:56:53.915]     return ctx.invoke(self.callback, **ctx.params)
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
 [2021/06/28 16:56:53.915]     return callback(*args, **kwargs)
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/src/buildscripts/evergreen_gen_multiversion_tests.py", line 380, in run_generate_tasks
 [2021/06/28 16:56:53.915]     generate_orchestrator.generate(evg_expansions)
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/src/buildscripts/evergreen_gen_multiversion_tests.py", line 328, in generate
 [2021/06/28 16:56:53.915]     generated_config = self.generate_fuzzer(evg_expansions)
 [2021/06/28 16:56:53.915]   File "/data/mci/cd1aa5b838fde20f5e60de5a183337c5/src/buildscripts/evergreen_gen_multiversion_tests.py", line 298, in generate_fuzzer
 [2021/06/28 16:56:53.915]     existing_tasks.add({ExistingTask(f"{suite}_multiversion_gen")})
 [2021/06/28 16:56:53.915] TypeError: unhashable type: 'set'

Comment by Robert Guo (Inactive) [ 16/Jun/21 ]

Yeah good point Max. Thanks for the suggestion.

 

For the impl: we can use the overrides introduced in SERVER-55856 to continue to reuse the existing suite definition with a new TestData field in the multiversion suite definition that the fuzzer-generated file can check to disable the setFeatureCompatibilityVersion command

Comment by Max Hirschhorn [ 17/May/21 ]

I could be missing something but I wouldn't expect the mutational (jstestfuzz) fuzzer to be able to change the feature compatibility version in the jstestfuzz_*_multiversion tasks without causing secondaries to crash.

  • If the primary is the newer binary version, then its latestFCV would cause an older binary version secondary to crash.
  • If the primary is the older binary version, then its lastStableFCV would cause a newer binary version secondary to crash.

Perhaps this ticket is really about preventing the mutational (jstestfuzz) fuzzer from running the setFeatureCompatibilityVersion command based around a TestData option only set in the jstestfuzz_*_multiversion tasks? If the fuzzer isn't capable of running the setFeatureCompatibilityVersion command, then there won't be any need to restore the feature compatibility version to a specific value during collection validation.

Comment by Brooke Miller [ 26/Jan/21 ]

In triaging, robert.guo  mentioned that we'll need to figure out how to override the FCV value for multiversion suites. There are a few places to set various flags, but it's difficult to determine where to grab the final value from. (More details to come in the design for the Multiversion project.)

Generated at Thu Feb 08 05:32:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.