[SERVER-53176] Return an error when commitQuorum includes voting buildIndexes:false members Created: 02/Dec/20  Updated: 29/Oct/23  Resolved: 13/Jan/21

Status: Closed
Project: Core Server
Component/s: Index Maintenance, Replication
Affects Version/s: 4.4.2
Fix Version/s: 4.9.0, 4.4.4

Type: Improvement Priority: Major - P3
Reporter: Konrad Baumgart Assignee: Louis Williams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

mongo db server and cli client 4.4.2


Issue Links:
Backports
Documented
is documented by DOCS-14132 Investigate changes in SERVER-53176: ... Closed
Problem/Incident
causes SERVER-63531 commitQuorum incorrectly includes bui... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.4
Sprint: Execution Team 2021-01-25
Participants:

 Description   

Given a replica set with config like

{
  _id: 'rs',
  members: [
    {
      _id: 0,
      host: '…',
      priority: 3,
    },
    {
      _id: 1,
      host: '…',
      priority: 1,
    },
    {
      _id: 2,
      host: '…',
      priority: 0,
      hidden: true,
      buildIndexes: false,
    },
  ]
} 

Building indexes on replica-set with default commitQuorum (votingMembers) seems to wait for build on member that doesn't build indexes (due to replica-set configuration)

I have to lower commitQuorum to majority, otherwise createIndex command doesn't finish.

votingMembers commitQuorum should mean: all data-bearing index-building voting replica set members.



 Comments   
Comment by Githook User [ 15/Jan/21 ]

Author:

{'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}

Message: SERVER-53176 Return an error when an index build commit quorum depends on voting buildIndexes:false nodes

(cherry picked from commit eed69840da689b56db5f4f7fc0da991c27f28fb6)
Branch: v4.4
https://github.com/mongodb/mongo/commit/8b2c1155aa0cab0d88cb6c7ad4e4e6126cffb933

Comment by Githook User [ 13/Jan/21 ]

Author:

{'name': 'Louis Williams', 'email': 'louis.williams@mongodb.com', 'username': 'louiswilliams'}

Message: SERVER-53176 Return an error when an index build commit quorum depends on voting buildIndexes:false nodes
Branch: master
https://github.com/mongodb/mongo/commit/eed69840da689b56db5f4f7fc0da991c27f28fb6

Comment by Louis Williams [ 11/Jan/21 ]

Throwing an error seems to be good solution, as it would help both future users and me (when executing commands via cli and forgetting about that commit qourum). It would be best to handle other commit quorums too: in my config using commitQurum 3 is also problematic and in some ill configurations "majority" may be also problematic.

Great! When determining whether a commit quorum is satisfiable, we will not consider voting, buildIndexes:false nodes.

I have only 3 nodes, so marking one of them as non-voting would make cluster less resilient.

True, making your hidden node non-voting means that your replica set will not have a majority if one of the other nodes fails. The other solution would be to add another regular node to your replica set so that you have 3 eligible primaries and 1 hidden node, cost permitting.

Comment by Konrad Baumgart [ 11/Jan/21 ]

I have only 3 nodes, so marking one of them as non-voting would make cluster less resilient. I handled my problem by modifying the lib like:

const CreateIndexesOperation = require('mongodb/lib/operations/create_indexes')
 
if (process.env.MONGODB_RS) {
 CreateIndexesOperation.prototype.executeOld = CreateIndexesOperation.prototype.execute
 CreateIndexesOperation.prototype.execute = function(...args) {
  this.options.commitQuorum = this.options.commitQuorum ?? 'majority'
  this.executeOld(...args)
 }
}

Throwing an error seems to be good solution, as it would help both future users and me (when executing commands via cli and forgetting about that commit qourum). It would be best to handle other commit quorums too: in my config using commitQurum 3 is also problematic and in some ill configurations "majority" may be also problematic.

I see there is already some check implemented for simplest unsatisfiable commit quorum, so extending it seems good.

rs0:PRIMARY> db.foo.createIndex({a:1}, {}, 28)
{
	"operationTime" : Timestamp(1610349191, 6),
	"ok" : 0,
	"errmsg" : "Commit quorum cannot be satisfied with the current replica set configuration",
	"code" : 278,
	"codeName" : "UnsatisfiableCommitQuorum",
	"$clusterTime" : {
		"clusterTime" : Timestamp(1610349191, 6),
		"signature" : {
			"hash" : BinData(0,"gKqHumdWJGuFXDzLS7IitRF2UKk="),
			"keyId" : NumberLong("6913888566116352003")
		}
	}
}

Comment by Louis Williams [ 08/Jan/21 ]

Hi, konrad.baumgart@erli.pl. I understand that this is confusing and undesirable behavior. To avoid the workaround in your code, you could reconfigure the buildIndexes: false node to be non-voting. Documentation here. That node is already hidden and has a priority of 0. If you set votes to also be 0, then it will not be included in the default commitQuorum. This means your createIndexes commands can remain topology-agnostic for testing.

Rather than changing the meaning of "votingMembers" or even editing the documentation, I think the most reasonable solution would be to prevent future users from running into this problem. That is, we should return an error when a user tries to create an index with the "votingMembers" commit quorum and there are voting,  buildIndexes: false nodes in the set. Otherwise, it is guaranteed that your index build will never complete.

Comment by Konrad Baumgart [ 21/Dec/20 ]

For me (and hopefully most other use-cases) it would be best, if I could call

db.indextest_empty.createIndex({a:1})

no matter if I'm on replicaset or standalone.

Currently I have to write code like

db.test.createIndex(
    { a: 1 },
    {
      partialFilterExpression: { b: true },
      ...(process.env.MONGODB_RS ? {commitQuorum: 'majority'} : {}),
    }
)

to run code both locally and in integration tests and in production without error 'Standalones can't specify commitQuorum'

Comment by Eric Sedor [ 02/Dec/20 ]

Thanks konrad.baumgart@erli.pl, I am able to reproduce this:

replset:PRIMARY> db.version()
4.4.2
replset:PRIMARY> rs.conf()
{
	"_id" : "replset",
	"version" : 5,
	"term" : 197,
	"protocolVersion" : NumberLong(1),
	"writeConcernMajorityJournalDefault" : true,
	"members" : [
		{
			"_id" : 0,
			"host" : "localhost:27017",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 3,
			"tags" : {
 
			},
			"votes" : 1
		},
		{
			"_id" : 1,
			"host" : "localhost:27018",
			"arbiterOnly" : false,
			"buildIndexes" : true,
			"hidden" : false,
			"priority" : 1,
			"tags" : {
 
			},
			"votes" : 1
		},
		{
			"_id" : 2,
			"host" : "localhost:27019",
			"arbiterOnly" : false,
			"buildIndexes" : false,
			"hidden" : true,
			"priority" : 0,
			"tags" : {
 
			},
			"votes" : 1
		}
	],
	"settings" : {
		"chainingAllowed" : true,
		"heartbeatIntervalMillis" : 2000,
		"heartbeatTimeoutSecs" : 10,
		"electionTimeoutMillis" : 10000,
		"catchUpTimeoutMillis" : -1,
		"catchUpTakeoverDelayMillis" : 30000,
		"getLastErrorModes" : {
 
		},
		"getLastErrorDefaults" : {
			"w" : 1,
			"wtimeout" : 0
		},
		"replicaSetId" : ObjectId("5f3e81852c45bf5a0d4cf74b")
	}
}
replset:PRIMARY> for (var i=0;i<1000;i++) { db.indextest.insert({a:1,b:1,c:1})}
WriteResult({ "nInserted" : 1 })
replset:PRIMARY> db.indextest_empty.createIndex({a:1})
{
	"createdCollectionAutomatically" : true,
	"numIndexesBefore" : 1,
	"numIndexesAfter" : 2,
	"commitQuorum" : "votingMembers",
	"ok" : 1,
	"$clusterTime" : {
		"clusterTime" : Timestamp(1606950535, 2),
		"signature" : {
			"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
			"keyId" : NumberLong(0)
		}
	},
	"operationTime" : Timestamp(1606950535, 2)
}
replset:PRIMARY> db.indextest.createIndex({a:1}) // hangs

It may not be that modifying the meaning of votingMembers is the best solution to this problem, but it is clear that marking a voting member with buildIndexes: false causes the default commitQuorum. I'll pass this on to an appropriate team to consider this behavior.

As you've already noted, if buildIndexes is set, you must also manually relax the commitQuorum to a number or setting that allows the index-building nodes to satisfy quorum.

Gratefully,
Eric

Generated at Thu Feb 08 05:30:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.