[SERVER-28981] Sharding balancer prefers shards in a specific order when moving chunks Created: 26/Apr/17  Updated: 30/Oct/23  Resolved: 16/Mar/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.2.13, 3.4.4, 3.5.6
Fix Version/s: 3.4.15, 3.6.4, 3.7.4

Type: Improvement Priority: Major - P3
Reporter: Matthew Kruse Assignee: Kevin Pulo
Resolution: Fixed Votes: 0
Labels: bkp
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File screenshot-1.png    
Issue Links:
Backports
Related
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6, v3.4
Sprint: Sharding 2018-03-12, Sharding 2018-03-26
Participants:
Case:

 Description   

When dealing with multiple collections and sharding the default mongo balancer behavior seems to pick the next shard to move to in alphabetic order. Long term for big data this behavior is terrible for cluster balancing as the majority of data tends to get dropped on rs0, then rs1, then rs2, etc.

In my test scenario, I have 10 databases each with there primary set specifically to one of the shards below. Each database has 1000 collections, as I went from 0 documents to 10+ billion chunk distribution across those shards did this with the default balancer behavior. For performance reasons on mongo I could not use a single collection as performance was 100x worse. Range scans on an index performed horribly compared to full collection scans on small collections due to random IO versus sequential IO.

mongos> db.chunks.aggregate([ { $group:{ _id: "$shard", cnt: { $sum: 1} } }, {$sort: {"_id": 1}} ]);
{ "_id" : "rs0", "cnt" : 28128 }
{ "_id" : "rs1", "cnt" : 18092 }
{ "_id" : "rs2", "cnt" : 7748 }
{ "_id" : "rs3", "cnt" : 6475 }
{ "_id" : "rs4", "cnt" : 5429 }
{ "_id" : "rs5", "cnt" : 4412 }
{ "_id" : "rs6", "cnt" : 4185 }
{ "_id" : "rs7", "cnt" : 3956 }
{ "_id" : "rs8", "cnt" : 3640 }
{ "_id" : "rs9", "cnt" : 3012 }
mongos> db.version();
3.4.2

I'm at a point were I'm going to have to turn off the default mongo balancer and essentially write my own to get data distributed properly. I would like the balancer to be data storage aware when picking a shard to move to rather then picking them in alphabetic order. Random would even be better as that would produce a better data distribution then what is currently occurring.



 Comments   
Comment by Githook User [ 26/Mar/18 ]

Author:

{'email': 'kevin.pulo@mongodb.com', 'name': 'Kevin Pulo', 'username': 'devkev'}

Message: SERVER-16802 SERVER-28981 Balancer consider shards and collections in random order

(cherry picked from commit 651b3e017ce880d9ddbebb400af621c61d8c7389)
Branch: v3.4
https://github.com/mongodb/mongo/commit/af4ea84a50cf35473c77ea2b27f19c63afd43bc1

Comment by Githook User [ 19/Mar/18 ]

Author:

{'email': 'kevin.pulo@mongodb.com', 'name': 'Kevin Pulo', 'username': 'devkev'}

Message: SERVER-16802 SERVER-28981 Balancer consider shards and collections in random order

(cherry picked from commit 9850f1f190f13fb5bfd229e35d55d8fee3adc58f)
Branch: v3.6
https://github.com/mongodb/mongo/commit/651b3e017ce880d9ddbebb400af621c61d8c7389

Comment by Githook User [ 16/Mar/18 ]

Author:

{'email': 'kevin.pulo@mongodb.com', 'name': 'Kevin Pulo', 'username': 'devkev'}

Message: SERVER-16802 SERVER-28981 Balancer consider shards and collections in random order
Branch: master
https://github.com/mongodb/mongo/commit/9850f1f190f13fb5bfd229e35d55d8fee3adc58f

Comment by Kaloian Manassiev [ 04/May/17 ]

Hi mkruse@adobe.com,

Than you for reporting this behaviour. You are correct that we can definitely randomize the selection of shards across all which are eligible. Currently the balancer gets them in sorted order from the shard cache (which happens to be an red-black tree std::map) and that's how it ends up selecting, but there is no good reason to do that.

Best regards,
-Kal.

Comment by Matthew Kruse [ 26/Apr/17 ]

To clarify, when things are tied I'd want the behavior to be different

Chunk counts
rs0:1
rs1:0
rs2:0

In the above scenario the balancer will always go to rs1 next if a chunk split happens on rs0. I'd like it to randomly choose between rs1 and rs2 or identify which one has less storage/documents on it and choose that shard.

Generated at Thu Feb 08 04:19:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.