[SERVER-46557] Commit quorum value can get lost when the node gets restarted or rolled back. Created: 03/Mar/20  Updated: 29/Oct/23  Resolved: 09/Apr/20

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 4.4.0-rc2, 4.7.0

Type: Bug Priority: Major - P3
Reporter: Suganthi Mani Assignee: Suganthi Mani
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-46659 Make initial sync work with two phase... Closed
is depended on by SERVER-47461 Remove commit quorum value from Start... Closed
Duplicate
is duplicated by SERVER-46814 Invariant on "Commit quorum is missin... Closed
is duplicated by SERVER-46829 Make setIndexCommitQuorum cmd to beha... Closed
Related
related to SERVER-47463 SetIndexCommitQuorum cmd should be su... Closed
is related to SERVER-46814 Invariant on "Commit quorum is missin... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.4
Sprint: Execution Team 2020-03-23, Execution Team 2020-04-06, Execution Team 2020-04-20
Participants:
Linked BF Score: 50

 Description   

Currently, we persist the 'commitQuorum' value that user provided for createIndexes cmd, into "config.system.IndexBuilds collection, only on receiving first ready to commit vote by primary. If current primary gets restarted/rolled back before receiving any votes, and gets re-elected, then new primary would not know about the commit quorum value. Currently, we crash our system. As, our current contract is, two phase index build should have a commit quorum value set.

To be noted, all the nodes who are eligible to become primary should have the commit quorum value, as the 2 phase index build can survive state transition and any node can act as coordinator for the 2 phase index build.



 Comments   
Comment by Githook User [ 15/Apr/20 ]

Author:

{'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}

Message: SERVER-46557 Guarantees the commit quorum value gets persisted before start of the index build.

(cherry picked from commit 4c41ed37fd58c4e4d4d2707f63f797f23ff9b486)
Branch: v4.4
https://github.com/mongodb/mongo/commit/cead747b708c560c8487bf04051f554cceaea334

Comment by Githook User [ 09/Apr/20 ]

Author:

{'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}

Message: SERVER-46557 Guarantees the commit quorum value gets persisted before start of the index build.
Branch: master
https://github.com/mongodb/mongo/commit/4c41ed37fd58c4e4d4d2707f63f797f23ff9b486

Comment by Suganthi Mani [ 12/Mar/20 ]

Just a quick rewind on the history of this ticket, yes, we should persist the commit quorum value in the config.system.indexBuilds collection when we start the index build, i.e., when we start writing the startIndexBuild oplog entry. The question that raised when I was implementing the commit quorum POC is that, should we need to write to system.indexBuilds collection before or after the startIndexBuild oplog entry. The answer for that was we should definitely do it before writing the startIndexBuild oplog entry, else we would lead to same issue described in this server ticket.

Next question came was, if we use a separate oplog entry to write to system.indexBuilds collection about the commit quorum value, then we might end up with a case where a node might just have an entry to config.system.IndexBuilds collection with no index actually built for the uuid referred in that entry. So, I proposed that we should use startIndexBuild oplog entry to perform 2 things on secondaries, It should start the index build (setting catalog index entry ready false) and write to system.IndexBuilds about the commit quorum value in the same WT transaction . At that time, we felt this would be a corner case and should be addressed later.

Comment by Benety Goh [ 11/Mar/20 ]

Perhaps we should persist the commit quorum into the

{config.system.indexBuilds}

collection without a majority write concern at the beginning of the index build. This will allow us to move away from relying on the in-memory commit quorum, provided by the createIndexes command, in ReplIndexBuildState.

Generated at Thu Feb 08 05:11:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.