[SERVER-58472] Implement a balancer merge chunks machinery Created: 13/Jul/21  Updated: 29/Oct/23  Resolved: 08/Sep/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.1.0
Fix Version/s: 5.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Simon Gratzer (Inactive) Assignee: Allison Easton
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File balancer_round.png    
Backwards Compatibility: Fully Compatible
Sprint: Sharding EMEA 2021-08-09, Sharding EMEA 2021-08-23, Sharding EMEA 2021-09-06
Participants:

 Description   

The balancer needs to support new functionality which will be required for the algorithm in SERVER-58471

After putting the balancer into a "merge chunks" mode via the per-collection command developed as part of SERVER-58270 ,
the balancer needs to work in a special mode where we attempt to merge and move chunks to shards where they can be merged.

The end goal is to reach a optimal number of chunks based on the default chunk size and the size of the data.
After the merge chunks algorithm signals convergence, the balancer needs to move back into the default mode.

 

  • Estimate chunk sizes by using the dataSize estimation API
  • Waiting for chunk size estimation should not completely block balancer, if sizes for chunks are unknown we need to queue the estimation operation and then move on to the next operation
  • Store chunk sizes in config.chunk documents for later processing stages
  • Balancer needs to perform merges operations which the merge algorithm decided on

See Kal’s script for proof of concept

Below a schematic of the new balancer round



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 02/Sep/21 ]

Author:

{'name': 'Paolo Polato', 'email': 'paolo.polato@mongodb.com', 'username': 'ppolato'}

Message: SERVER-58472 First Version of Balancer Commands Scheduler
Branch: master
https://github.com/mongodb/mongo/commit/53c498736bef89572d34a5201c3af749f65e68a2

Generated at Thu Feb 08 05:44:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.