[SERVER-68628] Retrying a failed resharding operation after a primary failover can lead to server crash or lost writes Created: 08/Aug/22 Updated: 29/Oct/23 Resolved: 09/Aug/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.11, 6.0.2, 6.1.0-rc0 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Max Hirschhorn | Assignee: | Max Hirschhorn |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | sharding-nyc-subteam1 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v6.0, v5.0
|
||||||||
| Sprint: | Sharding 2022-08-08, Sharding 2022-08-22 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 144 | ||||||||
| Story Points: | 3 | ||||||||
| Description |
|
While a resharding operation is ongoing, every write to the collection being resharded is amplified to a "db.system.resharding.<uuid>" sharded collection (aka temporary resharding collection). This is in part achieved by having ShardingWriteRouter::getReshardingDestinedRecipient() fill in a "destinedRecipient" field into the oplog entries for the inserts, updates, and deletes on the collection being resharded so these oplog entries can be later fetched by the appropriate recipient shard. ShardingWriteRouter calls CatalogCache::getCollectionRoutingInfo() to make this routing decision rather than CatalogCache::getCollectionRoutingInfoWithRefresh(). This is safe if the primary of the donor shard hasn't changed because it will have already refreshed the routing information for the temporary resharding collection earlier. However, if a new primary of the donor shard has been elected then the routing information for the temporary resharding collection may be arbitrarily stale. The routing information being stale is problematic for a couple reasons:
Running the flushRouterConfig command on all mongod --shardsvr processes before re-attempting a failed resharding operation will prevent the routing information for the temporary resharding collection from being stale. |
| Comments |
| Comment by Max Hirschhorn [ 09/Aug/22 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: The primary of a replica set shard may otherwise have arbitrarily stale (cherry picked from commit eda2f5d5d94ef0e7ec8016e5b9cad00d746f1787) |
| Comment by Githook User [ 09/Aug/22 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: The primary of a replica set shard may otherwise have arbitrarily stale (cherry picked from commit eda2f5d5d94ef0e7ec8016e5b9cad00d746f1787) |
| Comment by Githook User [ 08/Aug/22 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: The primary of a replica set shard may otherwise have arbitrarily stale |