[SERVER-35932] Mongos saved docs in wrong shard Created: 02/Jul/18 Updated: 10/Jul/18 Resolved: 10/Jul/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.4.10 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Pierre Coquentin | Assignee: | Esha Maharishi (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 14.04 |
||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Sprint: | Sharding 2018-07-16 | ||||||||
| Participants: | |||||||||
| Description |
|
Each month, our system creates new empty sharded collections, then to balance the load, we manually moved newly created chunk to a dedicated shard using the commands:
So we have 2 shards (sigfoxSet and sigfoxSet-2), the first one (the primary shard) contains all collections not sharded and the second one contains all sharded collections. All works perfectly until it doesn't This month we ran into a problem when the system starts using one of the sharded collection. The mongos didn't save the docs in the right shard, it saves the docs in the primary shard like if we didn't move the chunks previously, but it was impossible to read them afterward as they were not in the correct shard, the collection on the mongos was empty. Usually, when the mongos start using a new sharded collection, we see in the logs something like that:
But this time, nothing in the logs until we detected the problem and forced a restart of the mongos. After the restart, all was working perfectly like nothing happens. We dumped directly the docs from the wrong shard by executing the mongodump on the replicatset of the primary shard and restored them using a mongos.
Feel free to ask for additional information if needed.
|
| Comments |
| Comment by Pierre Coquentin [ 10/Jul/18 ] |
|
Indeed, we had an unexpected election on the primary shard a few days after the move chunk, it fits all the symptoms. So, until we upgrade to the latest version, we have to not forget to restart mongoses if an election is triggered. Pierre |
| Comment by Esha Maharishi (Inactive) [ 09/Jul/18 ] |
|
tldr; If this is happening infrequently and only after sharding a collection (as opposed to after a regular migration), it may be due to elections on the primary shard triggering the bug ( PierreCoquentin, can you check if this issue only occurs if the primary shard had an election around the time the issue manifested? Note that we fixed this in 4.0 and backported the fix to 3.6.2 under _______________________________ Full explanation: The fact that this happens on a recurring basis after sharding a collection (i.e., only affects writes after a unsharded -> sharded transition, rather than writes after a migration) suggests it may have to do with the fact that we clear a shard's routing table cache when it steps up to primary or steps down to secondary. This fits the description, because it means a shard would accept a request from mongos sent with an UNSHARDED shardVersion, since a shard interprets not having a routing table cache entry as meaning that the collection is unsharded. This means the shard would not return a stale shardVersion error, which would have triggered mongos to refresh its routing table and re-route. It also fits because after sharding the collection, all mongos would be routing requests to the primary shard with the UNSHARDED shardVersion until they hear such a stale shardVersion error from the primary shard, or until they are restarted (because when mongos loads the routing table for a database for the first time, it also loads all the sharded collections). However, I would expect the mongoses ran the shardCollection and moveChunks to be able to target correctly without needing a restart, since a mongos marks its routing table entry for a collection as stale after sharding a collection and after sending a moveChunk for the collection. If there are many mongos, perhaps it just went unnoticed that writes through the (presumably, single) mongos through which shardCollection and moveChunk were run were actually targeted correctly? |
| Comment by Pierre Coquentin [ 04/Jul/18 ] |
|
A slight precision I forgot, our application is composed of multiple application nodes, each one has their own mongos, and during the incident, all of them was saving in the wrong shard until I restart them one by one. |
| Comment by Esha Maharishi (Inactive) [ 03/Jul/18 ] |
|
kaloian.manassiev, from what I can tell, the write path on 3.4.10 mongos does attach the UNSHARDED version:
However, not all reads are versioned in 3.4.10, particularly if mongos thinks the collection is unsharded. I have not enumerated these here since the issue seems to be with writes. PierreCoquentin, one thing I noticed you said was:
1) Does this mean you are moving all chunks for sharded collections to the second shard? Or are you letting the chunks be balanced evenly across your two shards? Also, you mentioned
Two further questions: 2) What kind of write do you mean by "save"? Insert, op-style update with upsert: true, or replacement-style update with upsert: true? 3) When you tried to read the docs, did you do so from the same mongos as you used to write the docs? If so, what kind of read did you use (aggregate, find, count?) |