[SERVER-46557] Commit quorum value can get lost when the node gets restarted or rolled back. Created: 03/Mar/20 Updated: 29/Oct/23 Resolved: 09/Apr/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Storage |
| Affects Version/s: | None |
| Fix Version/s: | 4.4.0-rc2, 4.7.0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Suganthi Mani | Assignee: | Suganthi Mani |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v4.4
|
||||||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2020-03-23, Execution Team 2020-04-06, Execution Team 2020-04-20 | ||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 50 | ||||||||||||||||||||||||||||||||||||||||
| Description |
|
Currently, we persist the 'commitQuorum' value that user provided for createIndexes cmd, into "config.system.IndexBuilds collection, only on receiving first ready to commit vote by primary. If current primary gets restarted/rolled back before receiving any votes, and gets re-elected, then new primary would not know about the commit quorum value. Currently, we crash our system. As, our current contract is, two phase index build should have a commit quorum value set. To be noted, all the nodes who are eligible to become primary should have the commit quorum value, as the 2 phase index build can survive state transition and any node can act as coordinator for the 2 phase index build. |
| Comments |
| Comment by Githook User [ 15/Apr/20 ] |
|
Author: {'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}Message: (cherry picked from commit 4c41ed37fd58c4e4d4d2707f63f797f23ff9b486) |
| Comment by Githook User [ 09/Apr/20 ] |
|
Author: {'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com', 'username': 'smani87'}Message: |
| Comment by Suganthi Mani [ 12/Mar/20 ] |
|
Just a quick rewind on the history of this ticket, yes, we should persist the commit quorum value in the config.system.indexBuilds collection when we start the index build, i.e., when we start writing the startIndexBuild oplog entry. The question that raised when I was implementing the commit quorum POC is that, should we need to write to system.indexBuilds collection before or after the startIndexBuild oplog entry. The answer for that was we should definitely do it before writing the startIndexBuild oplog entry, else we would lead to same issue described in this server ticket. Next question came was, if we use a separate oplog entry to write to system.indexBuilds collection about the commit quorum value, then we might end up with a case where a node might just have an entry to config.system.IndexBuilds collection with no index actually built for the uuid referred in that entry. So, I proposed that we should use startIndexBuild oplog entry to perform 2 things on secondaries, It should start the index build (setting catalog index entry ready false) and write to system.IndexBuilds about the commit quorum value in the same WT transaction . At that time, we felt this would be a corner case and should be addressed later. |
| Comment by Benety Goh [ 11/Mar/20 ] |
|
Perhaps we should persist the commit quorum into the {config.system.indexBuilds}collection without a majority write concern at the beginning of the index build. This will allow us to move away from relying on the in-memory commit quorum, provided by the createIndexes command, in ReplIndexBuildState. |