[SERVER-68670] Cache imbalanced collections for balancing Created: 09/Aug/22 Updated: 29/Oct/23 Resolved: 23/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0-rc0, 6.0.6, 6.3.2 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Pierlauro Sciarelli | Assignee: | Silvia Surroca |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | balancer-perf-improvement | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||
| Backport Requested: |
v6.3, v6.0
|
||||||||||||||||||||||||
| Sprint: | Sharding EMEA 2022-09-19, Sharding EMEA 2022-10-03, Sharding EMEA 2022-10-17, Sharding EMEA 2022-10-31, Sharding EMEA 2022-11-14, Sharding EMEA 2022-11-28, Sharding EMEA 2022-12-12, Sharding EMEA 2022-12-26, Sharding EMEA 2023-01-09, Sharding EMEA 2023-01-23, Sharding EMEA 2023-02-06, Sharding EMEA 2023-02-20, Sharding EMEA 2023-03-06, Sharding EMEA 2023-03-20, Sharding EMEA 2023-04-03 | ||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||
| Linked BF Score: | 161 | ||||||||||||||||||||||||
| Description |
|
At the beginning of every balancing round, data size stats are collected from each shards in order to spot potential imbalance and schedule migrations. Currently, the balancer is requesting such stats in batches, collecting info for up to 20 collections per request. Purpose of this ticket is to rethink the current batching policy because this hardcoded limit may become source of scalability issues in case of thousands of sharded collections. [EDIT] REPURPOSED: this ticket also added a cache of imbalanced collections that lives between rounds in order to remember which collections had a migration scheduled during the previous round. |
| Comments |
| Comment by Githook User [ 24/Apr/23 ] |
|
Author: {'name': 'Pol Pinol Castuera', 'email': 'pol.pinol@mongodb.com', 'username': 'PolPinol'}Message: |
| Comment by Githook User [ 20/Apr/23 ] |
|
Author: {'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}Message: |
| Comment by Githook User [ 23/Mar/23 ] |
|
Author: {'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}Message: |
| Comment by Githook User [ 22/Mar/23 ] |
|
Author: {'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}Message: Revert " This reverts commit 9fe03fd6c85760920398b7891fde74069f5457db. |
| Comment by Githook User [ 21/Mar/23 ] |
|
Author: {'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}Message: |