[SERVER-40459] Optimize the construction of the balancer's collection distribution status histogram Created: 03/Apr/19 Updated: 21/Dec/23 Resolved: 19/Oct/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | 4.2.24, 4.4.24, 5.0.20 |
| Fix Version/s: | 4.2.25, 4.4.25, 5.0.22, 6.0.11, 7.0.3 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kaloian Manassiev | Assignee: | Tommaso Tocci |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | ShardingRoughEdges, balancer-round-perf, high-value | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Assigned Teams: |
Sharding EMEA
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Backport Requested: |
v7.1, v7.0, v6.0, v5.0, v4.4, v4.2
|
||||||||||||||||||||||||||||||||
| Sprint: | Sharding EMEA 2023-09-18, Sharding EMEA 2023-10-02, Sharding EMEA 2023-10-16, Sharding EMEA 2023-10-30 | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||
| Description |
|
On every round the sharding balancer constructs a histogram of where the chunks are places across the cluster. This histogram is produced through linear scan over all chunks in the chunk manager and because of this as the number of chunks grows, so grows its complexity. Without redesigning how this histogram is maintained, we could still get significant performance boost if we switch it to use KeyString like the ChunkManager. |
| Comments |
| Comment by Githook User [ 06/Nov/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: This effectively reverts GitOrigin-RevId: 1818719db3009bcc10addd016f08f35c2b6718d8 |
| Comment by Githook User [ 05/Nov/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: This effectively reverts GitOrigin-RevId: be9ea3f19e95fc92a6835c9c30381fb29962b45e |
| Comment by Githook User [ 05/Nov/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: This effectively reverts GitOrigin-RevId: 7a3c2a47cee449b5c02392b08dc5b33265cbe11b |
| Comment by Githook User [ 04/Nov/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: This effectively reverts GitOrigin-RevId: 595d8110122be5b1ca2d951c6f71cd8ecb29b8cb |
| Comment by Githook User [ 22/Oct/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: This effectively reverts GitOrigin-RevId: 5d091b472fa127626698ea9244e3a7fb5f727849 |
| Comment by Githook User [ 20/Oct/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: Revert " This reverts commit 56f414fe81c8fd7ae2ad87ada1ed0f9cb1299151. |
| Comment by Githook User [ 12/Oct/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 02/Oct/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 20/Sep/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 15/Sep/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 15/Sep/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 13/Sep/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}Message: |
| Comment by Githook User [ 13/Sep/23 ] |
|
Author: {'name': 'Tommaso Tocci', 'email': '58224719+toto-dev@users.noreply.github.com', 'username': 'toto-dev'}Message: |
| Comment by Pierlauro Sciarelli [ 26/Apr/22 ] |
|
Closing this ticket, as mentioned by Kal the referred code is going away under the "no more auto splitter" project. |
| Comment by Kaloian Manassiev [ 11/Feb/22 ] |
|
I believe that this ticket will no longer be necessary once we do No More Auto-Splitter, because the balancer will be relying on the data size coming from the Shards and not on chunks. |