[SERVER-30344] prevent shards from implicitly creating a collection on createIndexes Created: 26/Jul/17 Updated: 30/Oct/23 Resolved: 26/Sep/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.5.10 |
| Fix Version/s: | 3.6.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Esha Maharishi (Inactive) | Assignee: | Esha Maharishi (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v3.4
|
||||||||||||||||||||||||||||||||||||||||
| Sprint: | Sharding 2017-07-31, Sharding 2017-08-21, Sharding 2017-09-11, Sharding 2017-10-02 | ||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||
| Description |
|
createIndexes on mongos is currently broadcast to all shards rather than only targeting shards which own data for the collection, since listIndexes expects the primary shard to have all the indexes for the collection (and the primary shard might not own any chunks for the collection at some particular time) ( However, this means the collection is implicitly created on shards which don't own chunks for the collection AND don't have an entry for the collection in their storage engine (e.g., any shard that has never owned chunks for the collection). The collection is implicitly created on these shards without the collection options, including the collection's UUID. To prevent this, 1) mongods running as --shardsvr should fail createIndexes with NamespaceNotFound rather than implicitly create the collection with the wrong options |
| Comments |
| Comment by Scott Glajch [ 02/Jan/18 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
This bug definitely exists on 3.4.9, which we are running. The base use case in which I can get this to happen is:
This, in and of itself, doesn't seem to result in errors, as once you add enough data to the collection for the balancer to move a chunk to the new shard, it seems to create all the remaining missing indexes on the new shard. However I'm wondering if this is linked to chunk migration issues we've been having for a long time. We have actually been having issues with chunks being unable to move for years, and while the causes have varied, I'm thinking this might be one of the causes. One of the reasons we get for failed chunk moves is: "aborting migration, shard is missing 1 indexes and collection is not empty. Non-trivial index creation should be scheduled manually". That "1" can be N, depending on the number of indexes missing. Sometimes it's just missing 1 index, which feels like a symptom of the system getting into a weird state when adding 1 new index, and sometimes it's missing all but the _id index, which feels more like a symptom of the system getting into a weird state after having added a shard to the cluster (which we have done over the years as our customer base has grown). I'll post back here if we can get a reproducible test case (outside of our production environment), but we would be looking forward to a 3.4x backport of this if that happens.
| |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Esha Maharishi (Inactive) [ 17/Oct/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
I believe (but have not tested) that this bug exists on 3.4.9, because createIndexes in 3.4.9 extends AllShardsCollectionCommand, which targets all shards (not just shards that own data for the collection) if the namespace is sharded. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 26/Sep/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {'email': 'esha.maharishi@mongodb.com', 'name': 'Esha Maharishi', 'username': 'EshaMaharishi'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 28/Jul/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Yeah, I guess it's OK, esha.maharishi. I will say that we wouldn't need every write to an unsharded collection to take a trip to the config server; only the first write. But eliminating implicit collection creation on shards can wait for future work. | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Esha Maharishi (Inactive) [ 28/Jul/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Hmm, re "Can we?", I don't think so schwerin. For example, a write to a new collection will implicitly create the collection on the primary shard. The shard shouldn't just fail in this case... and we definitely don't want to add an extra round-trip between mongos and the primary shard for each unsharded write to check if mongos should explicitly create the collection first. I went through all the ways a collection can be created on mongod with a new UUID as of today:
Based on mongos's behavior today, only the ones in bold can cause a sharded collection to have different UUIDs across shards and/or the config server. schwerin, do you think it's fine to spot-fix just these? | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Andy Schwerin [ 26/Jul/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
createIndexes may not be the only risky behavior. Do we need to do something more affirmative, like prohibiting shard servers from implicitly creating collections entirely? Can we? | |||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Esha Maharishi (Inactive) [ 26/Jul/17 ] | |||||||||||||||||||||||||||||||||||||||||||||||
|
Minor backwards-breaking change: mongods running with --shardsvr will reject createIndexes for a collection they don't know about. Users shouldn't really be sending createIndexes directly to a shard anyway, which is why this is considered "minor." |