[SERVER-34893] A source shard may successfully complete a migration without changing its shard version Created: 08/May/18 Updated: 29/Oct/23 Resolved: 24/May/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 4.0.0-rc1, 4.1.1 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Jack Mulrow | Assignee: | Randolph Tan |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v4.0
|
||||||||||||||||
| Sprint: | Sharding 2018-06-04 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 30 | ||||||||||||||||
| Description |
|
Currently, the config server invalidates its cached metadata for a collection after it hears a response to moveChunk from the source shard in a migration. However, the recipient shard in that migration is free to begin a new migration as soon as it completes _recvChunkCommit (called earlier during the critical section). If a request to move a chunk away from the recipient shard comes to the config server before it invalidates its cache, it will send a stale shard version to the recipient and the recipient may complete the forced refresh at the beginning of a migration without seeing the persisted metadata changes, so it will not know it has received a new chunk, but begin to drive a migration anyway. This is only a problem when the shard believes it only owns one chunk, because when it goes to commit the migration it will not send a control chunk, so its shard version will not change after the migration commits, because the version of the chunk it doesn't know it will still own won't be bumped. Then the shard will be able to accept reads from a stale mongos looking for the chunk that was just moved even after refreshing at the end of moveChunk, returning wrong results. I think a partial solution would be to have the config server invalidate its cached metadata during _configsvrCommitChunkMigration, to decrease the likelihood of this happening, but a full solution may require preventing the recipient shard in a migration from driving a new one until the migration it was a part of has completed. |
| Comments |
| Comment by Githook User [ 24/May/18 ] |
|
Author: {'username': 'renctan', 'name': 'Randolph Tan', 'email': 'randolph@10gen.com'}Message: (cherry picked from commit 58fb5cb7a948fc05175b757ce5b0098e77af9ce1) |
| Comment by Githook User [ 24/May/18 ] |
|
Author: {'username': 'renctan', 'name': 'Randolph Tan', 'email': 'randolph@10gen.com'}Message: |
| Comment by Kaloian Manassiev [ 23/May/18 ] |
|
This actually sounds like a great idea and indeed helps with concurrent migrations within a shard - LGTM. FYI - janna.golden/blake.oler. I think we can also make the config server just ignore the control chunk if it sees it, which makes backwards compatibility handling simpler. The only slight concern I have is to do with the critical section latency going up because it might be more expensive to find the "control chunk" through direct query instead of the cache. |
| Comment by Randolph Tan [ 21/May/18 ] |
|
Proposed solution: instead of having the shard tell the config server the 'control chunk' in commitChunk command, let the config server determine it and bump the version accordingly. The entire command is protected via the chunkOp resource mutex so we can guarantee that any reads and manipulation will be serialized and the config server will always be the absolute source of truth. This solution also has the benefit of not conflicting with allowing parallel migrations within a shard. |
| Comment by Gregory McKeon (Inactive) [ 15/May/18 ] |
|
kaloian.manassiev This is rc0, but isn't scheduled for work. Can we either bump to 4.0.0 or schedule for this week? |