[SERVER-37810] Optimise balancer performance with zone sharding Created: 30/Oct/18  Updated: 29/Sep/23  Resolved: 28/Sep/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Josef Ahmad Assignee: Tommaso Tocci
Resolution: Duplicate Votes: 4
Labels: ShardingRoughEdges, balancer-round-perf, high-value, shardingemea-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File onCPU-CSRS-primary.png    
Issue Links:
Duplicate
duplicates SERVER-40459 Optimize the construction of the bala... Closed
Related
is related to SERVER-32526 Use KeyString for the ChunkManager's ... Closed
Assigned Teams:
Sharding EMEA
Participants:
Case:
Story Points: 2

 Description   

Reproduced in MongoDB 3.4.16 and 4.0.3.

With a considerable number of chunks (1+ million), the balancer is observed to spend a large amount of time checking each chunk for belonging to a tag. This can lead to a situation where a balancer round spends most of its time finding a candidate chunk (e.g. one minute) rather than migrating a chunk. This can have a significant impact on the overall cluster balancing performance.

Below is the a repro where the balancer spends 90% of its time finding a candidate chunk, and only 10% of its time moving the chunk.

Off-CPU profiling suggests that the balancer thread is CPU-bound. Attached a 60-second flame graph of the 3.4.16 CSRS primary process. The CSRS primary is only balancing the cluster at that time.

Most CPU time is consumed in BSONObj:woCompare().



 Comments   
Comment by Garaudy Etienne [ 29/Sep/23 ]

To be explicitly clear: This issue is the same as SERVER-40459, which has now been fixed in 4.4+.

Comment by Matt Panton [ 28/Sep/23 ]

Balancer performance with zone sharding performance has increased with due to the following enhancements - SERVER-80488, SERVER-80703, and SERVER-40459.

With most of the poor performance due to SERVER-40459.

Comment by Kaloian Manassiev [ 30/Oct/18 ]

Thank you josef.ahmad for the detailed report and for the heat map!

Looking at it, this is effectively the same issue as SERVER-32526 and there are two things that can be optimized:

  • Use KeyString for the ShardToChunksMap in the DistributionStatus class
  • Reduce the number of times we construct DistributionStatus for the different stages of the balancer - 1) when selecting chunks to split and 2) when selecting chunks to move (if the previous operation didn't do any splits).
Generated at Thu Feb 08 04:47:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.