[SERVER-61976] [Resharding] Shards can error while refreshing their shard version following step-up, stalling the resharding operation Created: 09/Dec/21 Updated: 29/Oct/23 Resolved: 21/Dec/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 5.0.0, 5.1.0, 5.2.0-rc0 |
| Fix Version/s: | 5.3.0, 5.1.2, 5.0.6, 5.2.0-rc5 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Brett Nawrocki |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | sharding-nyc-subteam1 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v5.2, v5.1, v5.0
|
||||||||
| Sprint: | Sharding 2021-12-27 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 34 | ||||||||
| Story Points: | 2 | ||||||||
| Description |
|
On step-up, shards will clear the filtering metadata and schedule a shard version refresh for the source collection and the temporary resharding collection. It is possible for the shard version refresh triggered through onShardVersionMismatch(..., boost::none /* shardVersionReceived */) to error and not complete the shard version refresh. This can leave a recipient shard waiting to learn all donor shards are prepared to donate or can leave a donor shard waiting to learn all recipient shards have finished cloning. Shards must reattempt calling onShardVersionMismatch() until it succeeds to ensure forward progress for the DonorStateMachines and RecipientStateMachines. Wrapping the call in an AsyncTry is the likely implementation solution.
|
| Comments |
| Comment by Githook User [ 07/Jan/22 ] |
|
Author: {'name': 'Brett Nawrocki', 'email': 'brett.nawrocki@mongodb.com', 'username': 'brettnawrocki'}Message: On step-up, shards will clear the filtering metadata and schedule a Therefore, shards now will retry on errors until the refresh (cherry picked from commit 70417bcbe6ca27b9e20455de5e77313ef68c648a) |
| Comment by Githook User [ 07/Jan/22 ] |
|
Author: {'name': 'Brett Nawrocki', 'email': 'brett.nawrocki@mongodb.com', 'username': 'brettnawrocki'}Message: On step-up, shards will clear the filtering metadata and schedule a Therefore, shards now will retry on errors until the refresh (cherry picked from commit 70417bcbe6ca27b9e20455de5e77313ef68c648a) |
| Comment by Githook User [ 07/Jan/22 ] |
|
Author: {'name': 'Brett Nawrocki', 'email': 'brett.nawrocki@mongodb.com', 'username': 'brettnawrocki'}Message: On step-up, shards will clear the filtering metadata and schedule a Therefore, shards now will retry on errors until the refresh (cherry picked from commit 70417bcbe6ca27b9e20455de5e77313ef68c648a) |
| Comment by Githook User [ 21/Dec/21 ] |
|
Author: {'name': 'Brett Nawrocki', 'email': 'brett.nawrocki@mongodb.com', 'username': 'brettnawrocki'}Message: The previous commit for |
| Comment by Githook User [ 20/Dec/21 ] |
|
Author: {'name': 'Brett Nawrocki', 'email': 'brett.nawrocki@mongodb.com', 'username': 'brettnawrocki'}Message: On step-up, shards will clear the filtering metadata and schedule a Therefore, shards now will retry on errors until the refresh |
| Comment by Pierlauro Sciarelli [ 14/Dec/21 ] |
|
is it worth Sharding NYC addressing I think it's better to solve the bugs under two separate tickets as they have different purposes. I am happy to review the adding of the AsyncTry if NYC has some bandwidth for it. By the way, I think it's worth mentioning there is also a flow that doesn't interrupt the RecoverRefreshThread: in case the stepdown-stepup process happens right between the beginning of its execution and before refreshing, the operation context would not get interrupted because we would miss the onStepdown/onStepUp interruption hooks. That seems way too unlikely to happen in case the same node is stepping down and up again since elections are not that fast. But it could totally happen in case a secondary is stepping up. |
| Comment by Max Hirschhorn [ 13/Dec/21 ] |
|
Randolph pointed out that it is odd for the newly-scheduled shard version refresh on step-up to have been killed because we would have already joined the RstlKillOpThread by the time resharding::clearFilteringMetadata() function is called. Max believes that we're seeing the effects of the primary joining a shard version refresh which had been initiated while the node was still secondary prior to the step-up. ShardServerCatalogCacheLoader::onStepUp() would have interrupted the OperationContext actively running forcePrimaryCollectionRefreshAndWaitForReplication() and propagated that interruption notification back through CatalogCache::getCollectionRoutingInfoWithRefresh() and the RecoverRefreshThread's future chain. joinShardVersionOperation() would have then propagated the interruption notification back through onShardVersionMismatch() and abandoned doing a new shard version refresh after the in-progress one was killed on step-up. pierlauro.sciarelli, kaloian.manassiev, is it worth Sharding NYC addressing |