[SERVER-60774] Resharding may apply through reshardFinalOp without transitioning to strict consistency, stalling write operations on collection being resharded until critical section times out Created: 18/Oct/21 Updated: 29/Oct/23 Resolved: 21/Oct/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 5.1.0-rc0 |
| Fix Version/s: | 5.2.0, 5.0.4, 5.1.0-rc2 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Max Hirschhorn |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Backport Requested: |
v5.1, v5.0
|
||||||||||||||||||||
| Sprint: | Sharding 2021-11-01 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Story Points: | 1 | ||||||||||||||||||||
| Description |
|
Typically the ReshardingOplogFetcher stops fetching new donor oplog entries after it has retrieved the reshardFinalOp entry. However, upon resuming from a primary failover, the ReshardingOplogFetcher won't realize it has fetched the reshardFinalOp entry already and it'll continue to retrieve empty batches. (The donor shard won't write any new oplog entries destined for the recipient shard after the reshardFinalOp.) This leads the ReshardingOplogFetcher to insert no-op reshardProgressMark entries after the reshardFinalOp intos the recipient shard's local oplog buffer collection. The ReshardingDonorOplogIterator assumes the reshardFinalOp will be the last entry in a batch. The presence of these extra no-op reshardProgressMark entries after the reshardFinalOp will prevent it and the ReshardingOplogApplier from realizing the reshardFinalOp has been applied through. The recipient shard therefore never reaches the "strict-consistency" state. This leads the overall resharding operation to fail with a ReshardingCriticalSectionTimeout error response after writes been blocked for the collection being resharded for reshardingCriticalSectionTimeoutMillis (5 seconds by default).
|
| Comments |
| Comment by Githook User [ 21/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingOplogFetcher to return without doing any work when Also changes ReshardingDonorOplogIterator to throw if it ever sees an (cherry picked from commit 13093cdb3f878f20e8ebda8ac78f329d1b33a52f) |
| Comment by Githook User [ 21/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingOplogFetcher to return without doing any work when Also changes ReshardingDonorOplogIterator to throw if it ever sees an (cherry picked from commit 13093cdb3f878f20e8ebda8ac78f329d1b33a52f) |
| Comment by Githook User [ 20/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingOplogFetcher to return without doing any work when Also changes ReshardingDonorOplogIterator to throw if it ever sees an |
| Comment by Max Hirschhorn [ 20/Oct/21 ] |
|
Reopened because my changes above don't handle how after the ReshardingOplogFetcher has inserted the reshardFinalOp, upon resuming, it won't ever know to exit. I'll type up the changes to switch to have getOplogFetcherResumeId() detect whether the reshardFinalOp has already been inserted so the ReshardingOplogFetcher can know there's no work left for it to do. |
| Comment by Githook User [ 19/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingDonorOplogIterator to not assume the (cherry picked from commit d3b36587bc2222fdefa1c755bb253f78c894496c) |
| Comment by Githook User [ 19/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingDonorOplogIterator to not assume the (cherry picked from commit d3b36587bc2222fdefa1c755bb253f78c894496c) |
| Comment by Githook User [ 18/Oct/21 ] |
|
Author: {'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}Message: Changes the ReshardingDonorOplogIterator to not assume the |