[SERVER-13175] Write command with invalid write concern crashes mongos in a mixed cluster. Created: 12/Mar/14  Updated: 11/Jul/16  Resolved: 17/Mar/14

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 2.6.0-rc1
Fix Version/s: 2.6.0-rc2

Type: Bug Priority: Blocker - P1
Reporter: Bernie Hackett Assignee: Randolph Tan
Resolution: Done Votes: 0
Labels: 26qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-13032 break early when resolving a replica ... Closed
Operating System: ALL
Steps To Reproduce:
  1. Set up a mixed version cluster as mentioned in the description.
  2. Shard a collection on _id.
  3. Insert enough documents into the sharded collection that chunks exist on both shards.
  4. Do a multi update using a $in query that matches documents on both shards using a write concern that not all shards support.

In my example both shards are standalone mongod. w > 1 write concern causes mongos to crash.

Participants:

 Description   

Using a mixed version cluster (2.6 mongos and config, 2.6 shard, 2.4 shard), a write operation that affects both shards causes mongos to crash if the write concern is invalid for one of the shards.

Commands in python:

>>> c.test.command('update', 'test', updates=[SON([('q', {'_id': {'$in': [2, 99997]}}), ('u', {'$set': {'foo': 'bar'}}), ('upsert', False), ('multi', True)])])
{u'nModified': 1, u'ok': 1, u'n': 2}
>>> 
>>> c.test.command('update', 'test', updates=[SON([('q', {'_id': {'$in': [3, 99996]}}), ('u', {'$set': {'foo': 'bar'}}), ('upsert', False), ('multi', True)])], writeConcern={'w': 2, 'wtimeout': 100})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pymongo/database.py", line 435, in command
    uuid_subtype, compile_re, **kwargs)[0]
  File "pymongo/database.py", line 335, in _command
    for doc in cursor:
  File "pymongo/cursor.py", line 1017, in next
    if len(self.__data) or self._refresh():
  File "pymongo/cursor.py", line 961, in _refresh
    self.__uuid_subtype))
  File "pymongo/cursor.py", line 885, in __send_message
    res = client._send_message_with_response(message, **kwargs)
  File "pymongo/mongo_client.py", line 1201, in _send_message_with_response
    raise AutoReconnect(str(e))
pymongo.errors.AutoReconnect: connection closed

mongos log:

2014-03-12T13:54:56.212-0700 [mongosMain] MongoS version 2.6.0-rc2-pre- starting: pid=92669 port=27017 64-bit host=devtop.local (--help for usage)
2014-03-12T13:54:56.212-0700 [mongosMain] db version v2.6.0-rc2-pre-
2014-03-12T13:54:56.213-0700 [mongosMain] git version: 27afe048247d1ca3d32858bd9047fc92e97efea8
...
2014-03-12T13:55:50.499-0700 [Balancer] distributed lock 'balancer/devtop.local:27017:1394657696:16807' unlocked. 
2014-03-12T13:55:52.067-0700 [conn1] ERROR: Uncaught std::exception: vector::_M_range_check, terminating
2014-03-12T13:55:52.067-0700 [conn1] dbexit:  rc:100

sh.status:

mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
	"_id" : 1,
	"version" : 4,
	"minCompatibleVersion" : 4,
	"currentVersion" : 5,
	"clusterId" : ObjectId("531fe9146e3028d5a974d0b6")
}
  shards:
	{  "_id" : "shard0000",  "host" : "localhost:37017" }
	{  "_id" : "shard0001",  "host" : "localhost:37018" }
  databases:
	{  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
	{  "_id" : "test",  "partitioned" : true,  "primary" : "shard0000" }
		test.test
			shard key: { "_id" : 1 }
			chunks:
				shard0001	3
				shard0000	4
			{ "_id" : { "$minKey" : 1 } } -->> { "_id" : 0 } on : shard0001 Timestamp(2, 0) 
			{ "_id" : 0 } -->> { "_id" : 15093 } on : shard0001 Timestamp(3, 0) 
			{ "_id" : 15093 } -->> { "_id" : 31137 } on : shard0001 Timestamp(4, 0) 
			{ "_id" : 31137 } -->> { "_id" : 47181 } on : shard0000 Timestamp(4, 1) 
			{ "_id" : 47181 } -->> { "_id" : 69233 } on : shard0000 Timestamp(3, 2) 
			{ "_id" : 69233 } -->> { "_id" : 95999 } on : shard0000 Timestamp(3, 4) 
			{ "_id" : 95999 } -->> { "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(3, 5) 



 Comments   
Comment by Randolph Tan [ 17/Mar/14 ]

Confirmed that SERVER-13032 fixes the issue.

Comment by Randolph Tan [ 13/Mar/14 ]

Some extra info about the crash:

Happens only in mixed clusters, shards all in 2.4 or 2.6 runs ok.

How the crash happens:

1. The broadcast write is broken down into child writes, one for each shard.
2. Mongos sends the writes.
3. Mongos gets the response one by one.
4. The response from the 2.6 shard is processed first, the parent op will be canceled since the 2.6 shard returns an { ok: 0 } response for the invalid write concern.

https://github.com/mongodb/mongo/blob/r2.6.0-rc1/src/mongo/s/write_ops/batch_write_op.cpp#L483-490

5. Then the response from 2.4 shard is processed. And since the getLastError response will give { ok: 1 }, it will continue on (as opposed to the early return at step #4).
5.a In debug builds, it will actually trigger a dassert here because the op was already cancelled at step #4:

https://github.com/mongodb/mongo/blob/r2.6.0-rc1/src/mongo/s/write_ops/batch_write_op.cpp#L531

6. It then crashes when it calls either noteWriteComplete or noteWriteError:

https://github.com/mongodb/mongo/blob/r2.6.0-rc1/src/mongo/s/write_ops/batch_write_op.cpp#L543-556

because both of them will try to access an element in the _childOps vector with an index and this vector was already cleared during the cancel call at step #4.

Current status

It is very likely that SERVER-13032 will be getting rid of the cancellation so this ticket is currently on hold.

Generated at Thu Feb 08 03:30:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.