[SERVER-68628] Retrying a failed resharding operation after a primary failover can lead to server crash or lost writes Created: 08/Aug/22  Updated: 29/Oct/23  Resolved: 09/Aug/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 5.0.11, 6.0.2, 6.1.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: sharding-nyc-subteam1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.0, v5.0
Sprint: Sharding 2022-08-08, Sharding 2022-08-22
Participants:
Linked BF Score: 144
Story Points: 3

 Description   

While a resharding operation is ongoing, every write to the collection being resharded is amplified to a "db.system.resharding.<uuid>" sharded collection (aka temporary resharding collection). This is in part achieved by having ShardingWriteRouter::getReshardingDestinedRecipient() fill in a "destinedRecipient" field into the oplog entries for the inserts, updates, and deletes on the collection being resharded so these oplog entries can be later fetched by the appropriate recipient shard. ShardingWriteRouter calls CatalogCache::getCollectionRoutingInfo() to make this routing decision rather than CatalogCache::getCollectionRoutingInfoWithRefresh(). This is safe if the primary of the donor shard hasn't changed because it will have already refreshed the routing information for the temporary resharding collection earlier. However, if a new primary of the donor shard has been elected then the routing information for the temporary resharding collection may be arbitrarily stale. The routing information being stale is problematic for a couple reasons:

  1. If the routing information for the temporary resharding collection says the collection is unsharded, then ShardingWriteRouter calling ChunkManager::findIntersectingChunkWithSimpleCollation() will result in a segmentation fault.
  2. If the routing information for the temporary resharding collection represents the chunk distribution from a prior resharding attempt, then the recipient shards may miss applying oplog entries and not end up consistent with the collection being resharded.

Running the flushRouterConfig command on all mongod --shardsvr processes before re-attempting a failed resharding operation will prevent the routing information for the temporary resharding collection from being stale.



 Comments   
Comment by Max Hirschhorn [ 09/Aug/22 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-68628 Refresh routing info on primary during resharding.

The primary of a replica set shard may otherwise have arbitrarily stale
routing information for the source collection and temporary resharding
collection based on the time of its last refresh.

(cherry picked from commit eda2f5d5d94ef0e7ec8016e5b9cad00d746f1787)
Branch: v6.0
https://github.com/mongodb/mongo/commit/8e55aa4ae2719be344a689a678f96531d4f82629

Comment by Githook User [ 09/Aug/22 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-68628 Refresh routing info on primary during resharding.

The primary of a replica set shard may otherwise have arbitrarily stale
routing information for the source collection and temporary resharding
collection based on the time of its last refresh.

(cherry picked from commit eda2f5d5d94ef0e7ec8016e5b9cad00d746f1787)
Branch: v5.0
https://github.com/mongodb/mongo/commit/6f918f3fed2333aa3d1ac213bd8176743a733895

Comment by Githook User [ 08/Aug/22 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-68628 Refresh routing info on primary during resharding.

The primary of a replica set shard may otherwise have arbitrarily stale
routing information for the source collection and temporary resharding
collection based on the time of its last refresh.
Branch: master
https://github.com/mongodb/mongo/commit/eda2f5d5d94ef0e7ec8016e5b9cad00d746f1787

Generated at Thu Feb 08 06:11:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.