[SERVER-36349] Handle the ShardingOperationFailedStatus in the OperationShardingState destructor when the OperationContext is not created in a ServiceEntryPoint Created: 30/Jul/18 Updated: 29/Oct/23 Resolved: 03/Dec/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 4.0.0, 4.1.1 |
| Fix Version/s: | 4.0.5, 4.1.6 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Cheahuychou Mao | Assignee: | Blake Oler |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Backport Requested: |
v4.0
|
||||||||
| Sprint: | Sharding 2018-11-05, Sharding 2018-11-19, Sharding 2018-12-03, Sharding 2018-12-17 | ||||||||
| Participants: | |||||||||
| Linked BF Score: | 57 | ||||||||
| Description |
|
Whenever an OperationContext is created, an OperationShardingState is created as a decoration on the OperationContext. When it's created through the regular process of a ServiceEntryPoint, a ScopedOperationCompletionShardingAction is created that will go out of scope at the end of ServiceEntryPoint::execCommandDatabase(). When an OperationContext is created outside of the regular process of a ServiceEntryPoint, a ScopedOperationCompletionShardingAction is not created. If an exception occurs during a WriteOp that matches StaleConfigInfo or CannotImplicitlyCreateCollectionInfo, the OperationShardingState will set a failed status in an error Status member variable. This Status is expected to be handled and reset by the destructor of ScopedOperationCompletionShardingActions. That is currently the only place where the Status could be reset. When the OperationShardingState is destructed, an invariant exists that asserts that the error Status member variable in OperationShardingState has been reset. In the case that an OperationContext is created outside of the ServiceEntryPoint codepath, a ScopedOperationCompletionShardingActions object will never exist to reset the error Status member variable. Upon destruction of the OperationContext, the destructor for the OperationShardingState will be called as well. If there was ever a StaleConfig or CannotImplicitlyCreateCollection status/exception, in a WriteOp, during the lifetime of the OperationContext, the destructor will trip the invariant. We should either relax the invariant, or think of other appropriate places to reset the OperationShardingState error Status member variable. |
| Comments |
| Comment by Githook User [ 10/Dec/18 ] |
|
Author: {'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}Message: (cherry picked from commit be673d00e71fe975fe9fe934ce719ff9d27a2712) |
| Comment by Githook User [ 03/Dec/18 ] |
|
Author: {'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}Message: |
| Comment by Blake Oler [ 13/Nov/18 ] |
|
In the logical session refresh, the opCtx for the ClusterWriter will never receive a ScopedOperationCompletionShardingAction entry. This is because when the ScopedOperationCompletionShardingAction code was written, it was under the assumption that we will always be writing through the command loop. The logical session cache refresh function isn't in the command loop. Whether or not the opCtx created can ever actually receive the StaleConfigInfo due to the ARS's asynchronicity is outside of the scope of my current understanding of how the ARS works. However, it still stands that if the logical session cache refresh receives a StaleConfigInfo error, it will not need to act on it. |
| Comment by Kaloian Manassiev [ 13/Nov/18 ] |
|
Just for my understanding, how does the OpCtx on which the ClusterWriter runs get a ScopedOperationCompletionShardingAction entry on it? The cluster writer should be using one OpCtx and the ARS (which is async), meaning the StaleConfigInfo should be happening on a different thread/OpCtx. |
| Comment by Randolph Tan [ 12/Nov/18 ] |
|
Approach sounds good to me. |
| Comment by Esha Maharishi (Inactive) [ 12/Nov/18 ] |
|
Sounds fine to me, as long as the logical session cache refresh really does not need to take action on StaleConfig or CannotImplicitlyCreateCollection, which I don't know enough about to say. |
| Comment by Blake Oler [ 12/Nov/18 ] |
|
The logical session cache's refresh is the only place where both create our own opCtx, and use the ClusterWriter from within a mongod. This means that the refresh is currently the only place where we could receive these errors outside of a service entry point's command loop. This particular invariant in the OperationShardingState asserts that we deal with both StaleConfigInfo and CannotImplicitlyCreateCollectionInfo errors. These are necessary to act upon during a command loop. However, we do not intend to deal with either of these errors if they bubble up directly from the refresh. If we receive a StaleConfigInfo error during using the ClusterWriter, we will simply ignore these errors and retarget on the next refresh cycle. If we receive a CannotImplicitlyCreateCollectionInfo error, then we will recreate the collection properly on the next refresh cycle. Because we don't care about acting upon these errors, we have a quick fix solution – create an ON_BLOCK_EXIT function that will directly clear any erroneous status from the OperationShardingState upon exit from the refresh. Since the OperationShardingState that's throwing the invariant is created by and directly attached to our created opCtx, this will have no greater impact on operations outside of the refresh itself. Looking to either esha.maharishi or renctan to LGTM this solution (since Misha is out this week) |
| Comment by Blake Oler [ 27/Sep/18 ] |
|
Pulling this out of the Logical Sessions Refactor epic, because the error is not unique to the Logical Session Cache. I am also putting this back into Needs Scheduling so that the Sharding team can assess the severity of this bug. |
| Comment by Misha Tyulenev [ 27/Aug/18 ] |
|
greg.mckeon yes, this and all logical sessions cache issues going to this epic. Updated the ticket. thanks! |
| Comment by Gregory McKeon (Inactive) [ 27/Aug/18 ] |
|
misha.tyulenev does this belong in your logical sessions refactor epic? |
| Comment by Gregory McKeon (Inactive) [ 03/Aug/18 ] |
|
misha.tyulenev can you talk with cheahuychou.mao about how you want to fix this? |