[SERVER-31521] Shard primaries that step down can't force the new primary to refresh its routing table Created: 11/Oct/17 Updated: 30/Oct/23 Resolved: 19/Oct/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.6.0-rc1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Jack Mulrow | Assignee: | Dianna Hohensee (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Backwards Compatibility: | Fully Compatible | ||||
| Operating System: | ALL | ||||
| Sprint: | Sharding 2017-10-23 | ||||
| Participants: | |||||
| Linked BF Score: | 0 | ||||
| Description |
|
If a shard primary steps down, it will trigger the ShardServerCatalogCacheLoader::onStepDown function which sets the PrimarySteppedDown error code interrupt on the OperationContextGroup used by the ShardServerCatalogCacheLoader. This means that all operation contexts added to this group will be automatically interrupted until resetInterrupt() is called in the ShardServerCatalogCacheLoader::onStepUp hook, when/if the node transitions back to primary. The problem is that when a secondary that used to be primary receives a command that forces it to refresh its routing table, it will add an operation context to its OperationContextGroup and call _runSecondaryGetChunksSince, which will attempt to send a "forceRoutingTableRefresh" command to the primary, but because the operation context was automatically interrupted upon being added to the group, this will fail with PrimarySteppedDown, and the primary will never actually receive the refresh command. I think |
| Comments |
| Comment by Githook User [ 19/Oct/17 ] |
|
Author: {'email': 'dianna.hohensee@10gen.com', 'name': 'Dianna Hohensee', 'username': 'DiannaHohensee'}Message: |
| Comment by Kaloian Manassiev [ 16/Oct/17 ] |
|
Yes, this is correct. Changing the behaviour of OperationContextGroup to not have a persistent "interrupted" state also seems reasonable to me. |
| Comment by Dianna Hohensee (Inactive) [ 16/Oct/17 ] |
|
So the approach is to significantly change OperationContextGroup. It will continue to interrupt the OperationContexts that it holds at the time of interrupt, but it will no longer interrupt OperationContexts created afterwards. Removing concepts like OperationContextGroup::resetInterrupt(), so interrupt doesn't 'stick'. |
| Comment by Kaloian Manassiev [ 12/Oct/17 ] |
|
So the problem here is that the "interrupted" state of the OperationContextGroup is sticky until the node becomes primary again? Is this a problem for forceRoutingTableRefresh only - it seems to me that even the local reads to refresh from the local chunks collection will also fail due to interrupt, wouldn't they? If that is the case, I don't think I was looking at the usage of OperationContextGroup and given that we interrupt all active operation contexts and set the state to secondary atomically, we could remove the sticky interrupt state and just make the code here check for whether it is still primary (or the right term)? |