[SERVER-53359] jstestfuzz (mutational) fuzzer is forcing last-lts FCV during validate instead of latest Created: 14/Dec/20  Updated: 29/Oct/23  Resolved: 15/Jan/21

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.9.0, 4.4.5

Type: Bug Priority: Minor - P4
Reporter: Max Hirschhorn Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
related to SERVER-53886 jstestfuzz (mutational) fuzzer is for... Closed
related to SERVER-52231 add timeseries to all-features builders Closed
related to SERVER-56961 [v4.4] Ensure cluster is in FCV 4.4 w... Closed
related to SERVER-57557 [v4.4] Support running checkFCV() she... Closed
is related to SERVER-51333 setFeatureCompatibilityVersion should... Closed
is related to SERVER-36718 Validation hook should upgrade before... Closed
is related to SERVER-46323 Update FCV constants throughout serve... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Execution Team 2021-01-25
Participants:
Linked BF Score: 47

 Description   

The changes from SERVER-36718 had reintroduced the TestData.forceValidationWithFeatureCompatibilityVersion option so that the jstestfuzz (mutational) fuzzer could upgrade the feature compatibility version before running the validate command. This is useful because certain versions of MongoDB have the validate command intentionally fail in last-lts FCV to allow users to be completely confident about downgrading.

It appears we forgot to change 4.2 -> 4.4 in the jstestfuzz*.yml test suites during the 4.4 release cycle. Observe that both the 4.2 and 4.4 branches are currently forcing the FCV to 4.2. Instead, the 4.4 branch should be forcing the feature compatibility version to 4.4. This hasn't been an issue because there isn't a difference in validate behavior on 4.4, and that'll likely be why the skipFCV concept from b45c07c as part of SERVER-51333 is safe on the 4.4 branch.

However, this in turn led us to change 4.2 -> 4.4 in c593d0f as part of SERVER-46323 when it should really be 4.9 on the master branch now.



 Comments   
Comment by Githook User [ 17/Feb/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-53359 update remaining jstestfuzz configs to validate at latest FCV

(cherry picked from commit fc1ee1497f904fc9d6a25813f5d012292af32a14)
Branch: v4.4
https://github.com/mongodb/mongo/commit/47922bc5faf1a2719bdb9d08b69bb7be0f65631d

Comment by Githook User [ 17/Feb/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-53359 temporarily disable jstestfuzz_multiversion tasks in ubuntu 18.04 builders

(cherry picked from commit c15915dac6c5fbdd35616a58d88e324d7261a424)
Branch: v4.4
https://github.com/mongodb/mongo/commit/20164c8b95c25c32cd04f52d54f621217aceab3d

Comment by Githook User [ 16/Feb/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-53359 jstestfuzz should validate at latest FCV

(cherry picked from commit 8cd628682a3db3a1ec97b884d876f4c6e9343ff8)
Branch: v4.4
https://github.com/mongodb/mongo/commit/6a600b1694f9be3a0f5c71ddf7b46efdc95404c5

Comment by Githook User [ 19/Jan/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-53359 temporarily disable jstestfuzz_multiversion tasks in ubuntu 18.04 builders
Branch: master
https://github.com/mongodb/mongo/commit/c15915dac6c5fbdd35616a58d88e324d7261a424

Comment by Githook User [ 15/Jan/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-53359 update remaining jstestfuzz configs to validate at latest FCV
Branch: master
https://github.com/mongodb/mongo/commit/fc1ee1497f904fc9d6a25813f5d012292af32a14

Comment by Githook User [ 15/Jan/21 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-53359 jstestfuzz should validate at latest FCV
Branch: master
https://github.com/mongodb/mongo/commit/8cd628682a3db3a1ec97b884d876f4c6e9343ff8

Comment by Max Hirschhorn [ 13/Jan/21 ]

Another option to avoid manually updating the jstestfuzz*.yml YAML suite files after each major release would be to change TestData.forceValidationWithFeatureCompatibilityVersion to TestData.forceValidationWithLatestFCV because that's the only way it is used. Then jstests/hooks/run_validate_collections.js and jstests/hooks/validate_collections.js can refer to the latestFCV variable that's compiled into the mongo shell without needing to use eval().

Comment by Max Hirschhorn [ 14/Dec/20 ]

The following changes would enable the jstestfuzz*.yml YAML suite files to refer to the latestFCV constant compiled into the mongo shell so we don't need to manually update them after each major release. It requires the collection validation hook allowing TestData.forceValidationWithFeatureCompatibilityVersion to be eval()'d instead of being interpreted as a constant string literal value.

diff --git a/buildscripts/resmokeconfig/suites/jstestfuzz.yml b/buildscripts/resmokeconfig/suites/jstestfuzz.yml
index b90c1c772c..85723a9afe 100644
--- a/buildscripts/resmokeconfig/suites/jstestfuzz.yml
+++ b/buildscripts/resmokeconfig/suites/jstestfuzz.yml
@@ -19,7 +19,7 @@ executor:
         TestData:
           skipValidationOnInvalidViewDefinitions: true
           skipValidationOnNamespaceNotFound: false
-          forceValidationWithFeatureCompatibilityVersion: "4.4"
+          forceValidationWithFeatureCompatibilityVersion: latestFCV
   fixture:
     class: MongoDFixture
     mongod_options:
diff --git a/jstests/hooks/run_validate_collections.js b/jstests/hooks/run_validate_collections.js
index 029e501cab..8f1445d6e3 100644
--- a/jstests/hooks/run_validate_collections.js
+++ b/jstests/hooks/run_validate_collections.js
@@ -34,12 +34,15 @@ if (topology.type === Topology.kStandalone) {
 }
 
 const adminDB = db.getSiblingDB('admin');
-const requiredFCV = jsTest.options().forceValidationWithFeatureCompatibilityVersion;
+let requiredFCV = jsTest.options().forceValidationWithFeatureCompatibilityVersion;
 
 let originalFCV;
 let originalTransactionLifetimeLimitSeconds;
 
 if (requiredFCV) {
+    requiredFCV = new Function(
+        `return typeof ${requiredFCV} === "string" ? ${requiredFCV} : "${requiredFCV}"`)();
+
     // Running the setFeatureCompatibilityVersion command may implicitly involve running a
     // multi-statement transaction. We temporarily raise the transactionLifetimeLimitSeconds to be
     // 24 hours to avoid spurious failures from it having been set to a lower value.
diff --git a/jstests/hooks/validate_collections.js b/jstests/hooks/validate_collections.js
index 856191ca51..4b8e02bdca 100644
--- a/jstests/hooks/validate_collections.js
+++ b/jstests/hooks/validate_collections.js
@@ -97,8 +97,10 @@ function CollectionValidator() {
                 return {ok: 1};
             }
 
-            const requiredFCV = jsTest.options().forceValidationWithFeatureCompatibilityVersion;
+            let requiredFCV = jsTest.options().forceValidationWithFeatureCompatibilityVersion;
             if (requiredFCV) {
+                requiredFCV = new Function(`return typeof ${requiredFCV} === "string" ? ${
+                    requiredFCV} : "${requiredFCV}"`)();
                 // Make sure this node has the desired FCV as it may take time for the updates to
                 // replicate to the nodes that weren't part of the w=majority.
                 assert.soonNoExcept(() => {

Generated at Thu Feb 08 05:30:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.