[SERVER-27258] A v3.4 config server can crash with a core dump if it gets an unsupported shard key from mongo S. Created: 01/Dec/16 Updated: 05/Apr/17 Resolved: 28/Dec/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.4.2 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Ricardo Amendoeira | Assignee: | Nathan Myers |
| Resolution: | Done | Votes: | 0 |
| Labels: | crash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Backwards Compatibility: | Fully Compatible |
| Sprint: | Sharding 2017-01-02 |
| Participants: |
| Description |
|
Hello, First of all I'm not looking for support on this, I'm not even sure if it's considered a bug but it seems like it's a bit dangerous. I'm working on adding a new feature to mongodb and have been modifying the code for mongo S (not mongod/config server, which is the one I'm able to crash remotely). Essentially I modified the mongo/s/shard_key_pattern.cpp to also accept "2dsphere" as a valid sharding key and then from the mongo shell called shardCollection with a 2dsphere key (index already configured). The config server promptly crashed with a core dump and the following backtrace:
Like I said, I'm not sure if this is considered a problem or not. |
| Comments |
| Comment by Githook User [ 28/Dec/16 ] | |
|
Author: {u'name': u'Nathan Myers', u'email': u'ncm@asperasoft.com'}Message: | |
| Comment by Ricardo Amendoeira [ 13/Dec/16 ] | |
|
Hi Kaloian Manassiev, If I understood correctly you wanted a mongodump --oplog of the config server, right? Thanks, | |
| Comment by Kaloian Manassiev [ 12/Dec/16 ] | |
|
Thanks ric2b. We will use this ticket to add guardrails to the config server so it does not crash. Do you mind stopping the balancer by issuing sh.stopBalancer, then sharding the collection using the command above and your custom build and then taking a dump of the config server so we can have the data to reproduce the problem with? Thanks in advance. -Kal. | |
| Comment by Ricardo Amendoeira [ 09/Dec/16 ] | |
|
Ok, sorry for the delay. I just confirmed the issue, compiled mongod clean from the 3.4 tag and was still able to crash the config server by issuing
to the mongoS server. | |
| Comment by Ricardo Amendoeira [ 02/Dec/16 ] | |
|
I've only been building mongo (shell) and mongos, yes. I'll make sure I rebuild mongod from the 3.4 tag and get back to you, but I can't do it right now. | |
| Comment by Kaloian Manassiev [ 02/Dec/16 ] | |
|
The code, which is shown in the call stack runs both on mongos and mongod. Are you saying that you only built mongos, but not mongod (i.e., you're using 3.4.0 for these) and it still crashes, or you built all three? If stock 3.4 crashes, then can you please dump the config database and attach it? | |
| Comment by Ricardo Amendoeira [ 02/Dec/16 ] | |
|
Ok, thanks for the response. I just find it a bit weird that a server is able to crash another, unmodified, server. But I understand if the config server is supposed to be in a protected network and will (hopefully) only receive messages from trusted servers. | |
| Comment by Kaloian Manassiev [ 01/Dec/16 ] | |
|
What seems to be happening here is that the contents of the shard key BSON are empty. This is something, which would have been caught at an earlier stage, while parsing the collection description in CollectionType::fromBSON, which is eventually called by DBConfig::_loadIfNeeded. Is it possible that any of your changes caused the checks in there to be bypassed and the in-memory representation of the collection to end up with an invalid shard key? In either case, this is not a supported scenario and is more appropriate for the mongodb-users group or Stack Overflow with the mongodb tag. Please post it there for more discussion. Best regards, |