[SERVER-30653] Randomize Chunk Locations with chunkLocationRandomizer Created: 14/Aug/17  Updated: 20/Sep/17  Resolved: 18/Aug/17

Status: Closed
Project: Core Server
Component/s: Performance, Sharding
Affects Version/s: 3.4.7
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Kevin Rice Assignee: Kelsey Schubert
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-5047 Better chunk selection when balancing Blocked
Participants:

 Description   

Current chunk migration strategies specify that some chunks will stay in a location until split. This can lead, over time and with imperfect choices of shard keys, to hot shards.

The problem can be solved, to a point, by creating a job that randomly relocates chunks.

The goal here is to eliminate any existing bias in where the chunks are located, and improve shard performance.

This could be enabled or disabled by a setting in the balancer, and changed the same way we change the setting for secondaryThrottle. This setting, chunkLocationRandomizer , could randomly pick a chunk and move it to a different server.

Of course, this could be made more intelligent by attempting to detect how many hot chunks there are, creating a plan that (if most are on one shard) moves them to another specific shard based on an optimized pattern. There would need to be hysteresis so it didn't constantly move the same chunk back and forth, but that wouldn't be difficult. But, it's enough to just relocate existing chunks into different shards to ensure uniform traffic.

In the short run, however, the randomizer could significantly minimze hot-shard problems caused by historical accident.

BTW, the reason I have a hot shard is that I can't have a shard key that's compound index of (field_one hashed and also field_two hashed-or-unhashed.). Instead, I have compound index, and it leads to a hot shard since field 2 is monotonically increasing.



 Comments   
Comment by Kelsey Schubert [ 18/Aug/17 ]

Hi kevin.rice@searshc.com,

Thank you for the improvement request. From your description, this feature appears present one possible solution to work described by SERVER-5047. Therefore, I'm linking this ticket so it'll be considered as part of that work. Please feel free to vote for SERVER-5047 and watch it for updates.

Kind regards,
Thomas

Generated at Thu Feb 08 04:24:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.