[SERVER-40459] Optimize the construction of the balancer's collection distribution status histogram Created: 03/Apr/19  Updated: 21/Dec/23  Resolved: 19/Oct/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.2.24, 4.4.24, 5.0.20
Fix Version/s: 4.2.25, 4.4.25, 5.0.22, 6.0.11, 7.0.3

Type: Improvement Priority: Major - P3
Reporter: Kaloian Manassiev Assignee: Tommaso Tocci
Resolution: Fixed Votes: 1
Labels: ShardingRoughEdges, balancer-round-perf, high-value
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-37810 Optimise balancer performance with zo... Closed
Problem/Incident
causes SERVER-82325 Config server could invariant during ... Closed
Related
related to SERVER-82365 Optimize the construction of the bala... Closed
Assigned Teams:
Sharding EMEA
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.1, v7.0, v6.0, v5.0, v4.4, v4.2
Sprint: Sharding EMEA 2023-09-18, Sharding EMEA 2023-10-02, Sharding EMEA 2023-10-16, Sharding EMEA 2023-10-30
Participants:
Case:

 Description   

On every round the sharding balancer constructs a histogram of where the chunks are places across the cluster. This histogram is produced through linear scan over all chunks in the chunk manager and because of this as the number of chunks grows, so grows its complexity.

Without redesigning how this histogram is maintained, we could still get significant performance boost if we switch it to use KeyString like the ChunkManager.



 Comments   
Comment by Githook User [ 06/Nov/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-82325 Config server could invariant during balancer round

This effectively reverts SERVER-40459

GitOrigin-RevId: 1818719db3009bcc10addd016f08f35c2b6718d8
Branch: v4.4
https://github.com/mongodb/mongo/commit/f45ff415970f8244303c77a964948630ce3bc93b

Comment by Githook User [ 05/Nov/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-82325 Config server could invariant during balancer round

This effectively reverts SERVER-40459

GitOrigin-RevId: be9ea3f19e95fc92a6835c9c30381fb29962b45e
Branch: v4.4
https://github.com/mongodb/mongo/commit/1818719db3009bcc10addd016f08f35c2b6718d8

Comment by Githook User [ 05/Nov/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-82325 Config server could invariant during balancer round

This effectively reverts SERVER-40459

GitOrigin-RevId: 7a3c2a47cee449b5c02392b08dc5b33265cbe11b
Branch: v4.4
https://github.com/mongodb/mongo/commit/be9ea3f19e95fc92a6835c9c30381fb29962b45e

Comment by Githook User [ 04/Nov/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-82325 Config server could invariant during balancer round

This effectively reverts SERVER-40459

GitOrigin-RevId: 595d8110122be5b1ca2d951c6f71cd8ecb29b8cb
Branch: v4.4
https://github.com/mongodb/mongo/commit/7a3c2a47cee449b5c02392b08dc5b33265cbe11b

Comment by Githook User [ 22/Oct/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-82325 Config server could invariant during balancer round

This effectively reverts SERVER-40459

GitOrigin-RevId: 5d091b472fa127626698ea9244e3a7fb5f727849
Branch: v4.4
https://github.com/mongodb/mongo/commit/595d8110122be5b1ca2d951c6f71cd8ecb29b8cb

Comment by Githook User [ 20/Oct/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: Revert "SERVER-40459 Optimize the construction of the balancer's collection distribution status histogram"

This reverts commit 56f414fe81c8fd7ae2ad87ada1ed0f9cb1299151.
Branch: v6.0
https://github.com/mongodb/mongo/commit/e6230d4c34ba250a3129ccfa5da06ccbc5b6d536

Comment by Githook User [ 12/Oct/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Optimise the construction of the balancer's collection distribution status histogram
Branch: v4.2
https://github.com/mongodb/mongo/commit/5c793a76207e3f10798a85cff5157982af9c7d77

Comment by Githook User [ 02/Oct/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Optimize the construction of the balancer's collection distribution status histogram
Branch: v7.0
https://github.com/mongodb/mongo/commit/9b772f346351d9e567e2886ecb4a8cedd7d81ad4

Comment by Githook User [ 20/Sep/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Optimize the construction of the balancer's collection distribution status histogram
Branch: v6.0
https://github.com/mongodb/mongo/commit/56f414fe81c8fd7ae2ad87ada1ed0f9cb1299151

Comment by Githook User [ 15/Sep/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Restore old behavior for balancing chunks in the No-Zone
Branch: v4.4
https://github.com/mongodb/mongo/commit/3e18c4c56048ddf22a6872edc111b542521ad1d5

Comment by Githook User [ 15/Sep/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Restore old behavior for balancing chunks in the No-Zone
Branch: v5.0
https://github.com/mongodb/mongo/commit/e352dc0694fed5d04eeff6bf17e1479acc93d679

Comment by Githook User [ 13/Sep/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': 'tommaso.tocci@mongodb.com', 'username': 'toto-dev'}

Message: SERVER-40459 Optimise the construction of the balancer's collection distribution status histogram
Branch: v4.4
https://github.com/mongodb/mongo/commit/3afe2c1000160a52d8ba90694429ffd6b5a8f40b

Comment by Githook User [ 13/Sep/23 ]

Author:

{'name': 'Tommaso Tocci', 'email': '58224719+toto-dev@users.noreply.github.com', 'username': 'toto-dev'}

Message: SERVER-40459 Optimise the construction of the balancer's collection distribution status histogram
Branch: v5.0
https://github.com/mongodb/mongo/commit/2299c2520d0d410cc645289ee1c8d1de81b1ae88

Comment by Pierlauro Sciarelli [ 26/Apr/22 ]

Closing this ticket, as mentioned by Kal the referred code is going away under the "no more auto splitter" project.

Comment by Kaloian Manassiev [ 11/Feb/22 ]

I believe that this ticket will no longer be necessary once we do No More Auto-Splitter, because the balancer will be relying on the data size coming from the Shards and not on chunks.

CC pierlauro.sciarelli

Generated at Thu Feb 08 04:55:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.