[SERVER-68670] Cache imbalanced collections for balancing Created: 09/Aug/22  Updated: 29/Oct/23  Resolved: 23/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.0.0-rc0, 6.0.6, 6.3.2

Type: Bug Priority: Major - P3
Reporter: Pierlauro Sciarelli Assignee: Silvia Surroca
Resolution: Fixed Votes: 0
Labels: balancer-perf-improvement
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-71009 Make sure the select chunks operation... Closed
is depended on by SERVER-75082 Make balancer migrations throttling c... Closed
Problem/Incident
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.3, v6.0
Sprint: Sharding EMEA 2022-09-19, Sharding EMEA 2022-10-03, Sharding EMEA 2022-10-17, Sharding EMEA 2022-10-31, Sharding EMEA 2022-11-14, Sharding EMEA 2022-11-28, Sharding EMEA 2022-12-12, Sharding EMEA 2022-12-26, Sharding EMEA 2023-01-09, Sharding EMEA 2023-01-23, Sharding EMEA 2023-02-06, Sharding EMEA 2023-02-20, Sharding EMEA 2023-03-06, Sharding EMEA 2023-03-20, Sharding EMEA 2023-04-03
Participants:
Linked BF Score: 161

 Description   

At the beginning of every balancing round, data size stats are collected from each shards in order to spot potential imbalance and schedule migrations. Currently, the balancer is requesting such stats in batches, collecting info for up to 20 collections per request.

Purpose of this ticket is to rethink the current batching policy because this hardcoded limit may become source of scalability issues in case of thousands of sharded collections.

[EDIT] REPURPOSED: this ticket also added a cache of imbalanced collections that lives between rounds in order to remember which collections had a migration scheduled during the previous round.



 Comments   
Comment by Githook User [ 24/Apr/23 ]

Author:

{'name': 'Pol Pinol Castuera', 'email': 'pol.pinol@mongodb.com', 'username': 'PolPinol'}

Message: SERVER-68670 Collecting balancing stats must not impact scalability
Branch: v6.0
https://github.com/mongodb/mongo/commit/2bd08faf234eab61f838532998d021dafb50951e

Comment by Githook User [ 20/Apr/23 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: SERVER-68670 Collecting balancing stats must not impact scalability
Branch: v6.3
https://github.com/mongodb/mongo/commit/69c396199a7bcf409c02e456d192fa794f03aef7

Comment by Githook User [ 23/Mar/23 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: SERVER-68670 Collecting balancing stats must not impact scalability
Branch: master
https://github.com/mongodb/mongo/commit/3db39116e26f32e3a53ad55140783f4e0b01147f

Comment by Githook User [ 22/Mar/23 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: Revert "SERVER-68670 Collecting balancing stats must not impact scalability"

This reverts commit 9fe03fd6c85760920398b7891fde74069f5457db.
Branch: master
https://github.com/mongodb/mongo/commit/b4aec5316dc08fbb8c1d0b48c1477a5a8f086847

Comment by Githook User [ 21/Mar/23 ]

Author:

{'name': 'Silvia Surroca', 'email': 'silvia.surroca@mongodb.com', 'username': 'silviasuhu'}

Message: SERVER-68670 Collecting balancing stats must not impact scalability
Branch: master
https://github.com/mongodb/mongo/commit/9fe03fd6c85760920398b7891fde74069f5457db

Generated at Thu Feb 08 06:11:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.