[SERVER-29423] Sharding balancer schedules multiple migrations with the same conflicting source or destination Created: 02/Jun/17 Updated: 30/Oct/23 Resolved: 16/Jan/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 3.4.4 |
| Fix Version/s: | 3.4.11, 3.6.3, 3.7.2 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Kaloian Manassiev | Assignee: | Kaloian Manassiev |
| Resolution: | Fixed | Votes: | 4 |
| Labels: | bkp | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Requested: |
v3.6, v3.4
|
||||||||||||
| Sprint: | Sharding 2018-01-29 | ||||||||||||
| Participants: | |||||||||||||
| Case: | (copied to CRM) | ||||||||||||
| Description |
|
The sharding balancer policy avoids producing multiple migrations for the same shard. However, the policy runs per collection and doesn't save any state across collections. Because of this, if there are multiple collections which need balancing, it may produce overlapping shards. There is no correctness problem with this, but it will cause useless ConflictingOperationInProgress errors to pollute the config server and shard's logs on each balancer round. |
| Comments |
| Comment by Kaloian Manassiev [ 22/Jan/18 ] | |||||||
|
The effect of this fix cannot be easily quantified. It will not make individual migrations to go faster, but it will improve parallelism in the case where there are multiple collections, which all need to be re-balanced. Currently in the worst-case scenario only one migration could effectively run in a round, because of conflicts with other collections. So I expect the performance gain to be between 1 to the max possible parallel migrations in a system. | |||||||
| Comment by Aparna Shah (Inactive) [ 22/Jan/18 ] | |||||||
|
ramon.fernandez would you be able to provide some numbers on performance gain observed in balancing/chunk migration as a result of this fix? It might help us better manage customer expectations in https://jira.mongodb.org/projects/HELP/queues/issue/HELP-5680 | |||||||
| Comment by Githook User [ 16/Jan/18 ] | |||||||
|
Author: {'email': 'kaloian.manassiev@mongodb.com', 'name': 'Kaloian Manassiev', 'username': 'kaloianm'}Message: (cherry picked from commit b5ebe8a5492c4f5e33970c0f885b9ac51460b9dc) | |||||||
| Comment by Githook User [ 16/Jan/18 ] | |||||||
|
Author: {'email': 'kaloian.manassiev@mongodb.com', 'name': 'Kaloian Manassiev', 'username': 'kaloianm'}Message: (cherry picked from commit b5ebe8a5492c4f5e33970c0f885b9ac51460b9dc) | |||||||
| Comment by Githook User [ 16/Jan/18 ] | |||||||
|
Author: {'email': 'kaloian.manassiev@mongodb.com', 'name': 'Kaloian Manassiev', 'username': 'kaloianm'}Message: | |||||||
| Comment by Akira Kurogane [ 10/Jan/18 ] | |||||||
|
Some notes to share for anyone diagnosing the issue backwards from logs and searching through this JIRA. 1. The following sort of "Balancer move ... failed" log message with "caused by :: ConflictingOperationInProgress" will be prevalent on the primary config server's logs before each new { what: "balancer.round" } document is inserted to the actionlog collection
and spread between the shards there will be a >= number of { what: "moveChunk.error" } documents being inserted into the changelog collection. There may be other causes (ChunkRangeCleanupPending, ChunkTooBig, etc.) but the ConflictingOperationInProgress will outnumber them. 2. The race between moveChunks can lead to some shard pairs not having any migration at all, even though there were candidates that could have used those shard pairs. In the example above the { what: "balancer.round" } document has ..., errorOccured: false, candidateChunks: 14, chunksMoved: 2 but it was a six-shard cluster with candidates for all shard pairs, so it should have been chunksMoved: 3 in the typical balance round. The reason why 1 was missed was:
| |||||||
| Comment by Vishal Katikineni [ 22/Dec/17 ] | |||||||
|
ConflictingOperationInProgress error is also slowing down the chunk migration rate when there are multiple collections to be balanced. Looks like a shard can only participate in one chunk migration at a time ie either receive or donate. We should be able to receive/donate multiple chunks from the same shard provided they are different collections. | |||||||
| Comment by Oleg Rekutin [ 10/Jun/17 ] | |||||||
|
The impact of this problem is captured in | |||||||
| Comment by Oleg Rekutin [ 10/Jun/17 ] | |||||||
|
If many collections need balancing, this effectively degrades the config servers to constantly refresh the chunks due to ConflictingOperationInProgress. This heavily pollutes the logs, at a rate of ~600M/hr. I think part of the problem is that ConflictingOperationInProgress for the reason of "Unable to start new migration because this shard is currently receiving chunk" is not an operation that should merit a refresh. It's an operation that's conflicting with the act of transferring data, but not necessarily with this collection. So ConflictingOperationInProgress represents both "unable to transfer due to a shard issue" and "unable to transfer due to an another operation altering this collection's metadata." Is it possible that only the 2nd category needs a refresh retry in catalog_cache.cpp? If you look in the method CatalogCache::_scheduleCollectionRefresh_inlock, in this section:
you can see how ConflictingOperationInProgress operations due to shards transferring chunks is going to lead to heavy collection metadata refreshing, if lots of collections need balancing. |