[SERVER-31521] Shard primaries that step down can't force the new primary to refresh its routing table Created: 11/Oct/17  Updated: 30/Oct/23  Resolved: 19/Oct/17

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 3.6.0-rc1

Type: Bug Priority: Major - P3
Reporter: Jack Mulrow Assignee: Dianna Hohensee (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sharding 2017-10-23
Participants:
Linked BF Score: 0

 Description   

If a shard primary steps down, it will trigger the ShardServerCatalogCacheLoader::onStepDown function which sets the PrimarySteppedDown error code interrupt on the OperationContextGroup used by the ShardServerCatalogCacheLoader. This means that all operation contexts added to this group will be automatically interrupted until resetInterrupt() is called in the ShardServerCatalogCacheLoader::onStepUp hook, when/if the node transitions back to primary.

The problem is that when a secondary that used to be primary receives a command that forces it to refresh its routing table, it will add an operation context to its OperationContextGroup and call _runSecondaryGetChunksSince, which will attempt to send a "forceRoutingTableRefresh" command to the primary, but because the operation context was automatically interrupted upon being added to the group, this will fail with PrimarySteppedDown, and the primary will never actually receive the refresh command.

I think SERVER-30148 will fix this, because the ShardingState's refresh logic shouldn't use an operation context from the ShardServerCatalogCacheLoader's interrupted OperationContextGroup.



 Comments   
Comment by Githook User [ 19/Oct/17 ]

Author:

{'email': 'dianna.hohensee@10gen.com', 'name': 'Dianna Hohensee', 'username': 'DiannaHohensee'}

Message: SERVER-31521 Make OperationContextGroup have a non-sticky interrupt and update ShardServerCatalogCacheLoader accordingly
Branch: master
https://github.com/mongodb/mongo/commit/88c39edf9275e47e5db4b1dbffb0bf51f9187731

Comment by Kaloian Manassiev [ 16/Oct/17 ]

Yes, this is correct. Changing the behaviour of OperationContextGroup to not have a persistent "interrupted" state also seems reasonable to me.

Comment by Dianna Hohensee (Inactive) [ 16/Oct/17 ]

So the approach is to significantly change OperationContextGroup. It will continue to interrupt the OperationContexts that it holds at the time of interrupt, but it will no longer interrupt OperationContexts created afterwards. Removing concepts like OperationContextGroup::resetInterrupt(), so interrupt doesn't 'stick'.

Comment by Kaloian Manassiev [ 12/Oct/17 ]

So the problem here is that the "interrupted" state of the OperationContextGroup is sticky until the node becomes primary again?

Is this a problem for forceRoutingTableRefresh only - it seems to me that even the local reads to refresh from the local chunks collection will also fail due to interrupt, wouldn't they? If that is the case, I don't think SERVER-30148 will fix it, because the control flow will eventually still get to having to read the local collection contents and (if the storage engine is a good citizen and obeys interruption) we will have the same problem.

I was looking at the usage of OperationContextGroup and given that we interrupt all active operation contexts and set the state to secondary atomically, we could remove the sticky interrupt state and just make the code here check for whether it is still primary (or the right term)?

Generated at Thu Feb 08 04:27:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.