[SERVER-61482] Updates to config.reshardingOperations wait for PrimaryOnlyService to be rebuilt while holding oplog slot, stalling replication on config server indefinitely Created: 15/Nov/21 Updated: 29/Oct/23 Resolved: 17/Nov/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 5.0.0, 5.1.0 |
| Fix Version/s: | 5.2.0, 5.0.5, 5.1.1 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Max Hirschhorn | Assignee: | Max Hirschhorn |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | sharding-nyc-subteam1 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||
| Backport Requested: |
v5.1, v5.0
|
||||||||||||||||||||||||||||
| Sprint: | Sharding 2021-11-29 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Linked BF Score: | 159 | ||||||||||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||||||||||
| Description |
|
PrimaryOnlyService::onStepUp() waits for stepUpOpTime to become majority-committed before attempting to rebuild any Instances. New optimes becoming majority-committed depend on the ability for secondaries to successfully read new entries from a forward-scanning oplog cursor, which in turn depend on there not being any outstanding storage transactions with oplog slots still reserved (aka a hole in the oplog). ReshardingOpObserver::onUpdate() attempts to get the ReshardingCoordinator and its associated ReshardingCoordinatorObserver to update their in-memory states. Doing so must wait until the ReshardingCoordinatorService has finished rebuilding. However, ReshardingOpObserver::onUpdate() currently waits for the ReshardingCoordinatorService to have finished rebuilding with its storage transaction still active and after having acquired an oplog slot for the update to config.reshardingOperations. If the ReshardingCoordinatorService wasn't already rebuilt before the update to the config.reshardingOperations collection came in from the donor or recipient shard, then it won't ever finish rebuilding. Nor will replication on the config server be able to make progress with the oplog hole present. |
| Comments |
| Comment by Githook User [ 17/Nov/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: (cherry picked from commit 2d3592d80fb851e00b519bfeeea513aaff6d0c95) |
| Comment by Githook User [ 17/Nov/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: (cherry picked from commit 2d3592d80fb851e00b519bfeeea513aaff6d0c95) |
| Comment by Githook User [ 16/Nov/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: |