[SERVER-48081] Malformed document validators can cause a secondary to fail on oplog application Created: 11/May/20  Updated: 22/Jan/21  Resolved: 22/Jan/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Nicholas Zolnierz Assignee: Ted Tuckman
Resolution: Won't Fix Votes: 0
Labels: qopt-team
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-48035 Update doc_validation_encrypt_keyword... Closed
Operating System: ALL
Steps To Reproduce:

diff --git a/jstests/noPassthrough/invalid_collection_validator_at_startup.js b/jstests/noPassthrough/invalid_collection_validator_at_startup.js
index 2d453345a0..21a9e97e63 100644
--- a/jstests/noPassthrough/invalid_collection_validator_at_startup.js
+++ b/jstests/noPassthrough/invalid_collection_validator_at_startup.js
@@ -12,13 +12,20 @@ const collName = "collectionWithMalformedValidator";
 
 // Create a collection with an invalid regex using a fail point.
 (function createCollectionWithMalformedValidator() {
-    const conn = MongoRunner.runMongod({dbpath: dbpath});
-    assert.neq(null, conn, "mongod was unable to start up");
+    // const conn = MongoRunner.runMongod({dbpath: dbpath});
+    // assert.neq(null, conn, "mongod was unable to start up");
 
-    const testDB = conn.getDB("test");
+    const rst = new ReplSetTest({
+        nodes: 3,
+    });
+    rst.startSet();
+    rst.initiate();
+
+    // const testDB = conn.getDB("test");
+    const testDB = rst.getPrimary().getDB("test");
     assert.commandWorked(testDB[collName].insert({a: "hello world"}));
 
-    assert.commandWorked(conn.adminCommand(
+    assert.commandWorked(testDB.adminCommand(
         {configureFailPoint: 'allowSettingMalformedCollectionValidators', mode: 'alwaysOn'}));
 
     // Invalid because '*' indicates that repetitions should be allowed but it's preceded by a
@@ -29,13 +36,21 @@ const collName = "collectionWithMalformedValidator";
     assert.commandWorked(
         testDB.runCommand({collMod: collName, validator: {email: {$regex: invalidRegex}}}));
 
-    MongoRunner.stopMongod(conn);
+    // MongoRunner.stopMongod(conn);
+    rst.stopSet();
 })();
 
 (function startUpWithMalformedValidator() {
-    const conn = MongoRunner.runMongod({dbpath: dbpath, noCleanData: true});
-    assert.neq(null, conn, "mongod was unable to start up");
-    const testDB = conn.getDB("test");
+    // const conn = MongoRunner.runMongod({dbpath: dbpath, noCleanData: true});
+    // assert.neq(null, conn, "mongod was unable to start up");
+    // const testDB = conn.getDB("test");
+    const rst = new ReplSetTest({
+        nodes: 3,
+        nodeOptions: {noCleanData: true},
+    });
+    rst.startSet();
+    rst.initiate();
+    const testDB = rst.getPrimary().getDB("test");
 
     // Check that we logged a startup warning.
     const cmdRes = assert.commandWorked(testDB.adminCommand({getLog: "startupWarnings"}));
@@ -48,6 +63,7 @@ const collName = "collectionWithMalformedValidator";
     assert.commandWorked(testDB.someOtherCollection.insert({a: 1}));
     assert.eq(testDB.someOtherCollection.find().itcount(), 1);
 
-    MongoRunner.stopMongod(conn);
+    // MongoRunner.stopMongod(conn);
+    rst.stopSet();
 })();
 })();

Sprint: Query 2020-11-30, Query 2020-12-14, Query 2020-12-28, Query 2021-01-11, Query 2021-01-25
Participants:

 Description   

Found as part of SERVER-45514, which is testing that we can upgrade a replica set with a persisted document validator that is considered invalid in the "latest" binVersion (4.4 in this case). If one of the secondaries is lagging behind and does not yet apply the "create" oplog entry, then upon upgrade the secondary will fassert on oplog application.

Note that there are two distinct paths to a failure. One is the case described above, where a node is catching up after upgrade and encounters the malformed doc validator. The second is when the node starts up and already has the entry for the "create". In that case, it will happily build the in-memory Collection state with a non-OK validator, however any subsequent inserts to that collection will fail on oplog application.



 Comments   
Comment by Charlie Swanson [ 15/Oct/20 ]

ted.tuckman if you have some time next sprint can you pick up the investigation that katherine.wu started? Lower priority than all the other things in your sprint.

Comment by Nicholas Zolnierz [ 11/May/20 ]

max.hirschhorn if the Collection state already has a non-OK validator, then we fail before checking DisableDocumentValidation. It does seem like we should move that check up to the top of the function which AFAICT would fix the insert issue.

Comment by Max Hirschhorn [ 11/May/20 ]

Prior to that change, we used to run both validation and shard version checking.

Secondaries doing oplog application have always used DisableDocumentValidation to skip doing document validation. Are you suggesting there were cases where that wasn't sufficient?

Comment by Pierlauro Sciarelli [ 11/May/20 ]

Take a look to this commit relative to SERVER-47748.

 Validation can be disabled by using UnreplicatedWritesBlock.

Comment by Nicholas Zolnierz [ 11/May/20 ]

kaloian.manassiev oh that's interesting, do you have a ticket #?

Comment by Kaloian Manassiev [ 11/May/20 ]

FYI pierlauro.sciarelli made it so that validators will not run on secondaries at all. Prior to that change, we used to run both validation and shard version checking.

Generated at Thu Feb 08 05:16:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.