[SERVER-4313] sharded systems should throttle writes when chunk migrations cannot keep up Created: 17/Nov/11  Updated: 21/Dec/22  Resolved: 21/Dec/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Dwight Merriman Assignee: [DO NOT USE] Backlog - Sharding EMEA
Resolution: Won't Do Votes: 3
Labels: sharding-common-backlog
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Sharding EMEA
Participants:

 Description   

In some cases data may be arriving at a shard faster than it can be migrated elsewhere. The typical situation where this is seen is on a brand new collection which does not have many chunks yet, to which a very large bulk import is sent. In this situation, it is necessary to throttle the injection at the shard; if injest rate is, for a long period of time, higher than migration output rate, the node will "overflow" with either data or workload (often the latter). Presplitting is a good technique to avoid this situation currently.

Thus this ticket is to throttle writes s.t. migrations do not fall behind.

For "phase one" inserts are the only thing that needs throttling. We can make a phase 2 ticket after that.



 Comments   
Comment by Connie Chen [ 21/Dec/22 ]

For now, closing this as won't do, as we would not want to significantly impact a user's workload or application by throttling their writes.

Comment by Kevin J. Rice [ 15/Mar/13 ]

We have 24 shards each on 2 boxes (done because excessive lock contention)(24 shards is 24x faster). We start out with all the data in 1 shards, or maybe 2, even though there are 48 total (we have 2 boxes). So the shard counts are: 1, 40, 0, 0, 0, ... and all the writes are happening on shard 1 (and a small number on shard 0). Since we peg the IO immediately due to it being sized for 48 shards, no balancing happens.

So, the system should have some kind of radioactive-decay-curve for balancing: the longer it goes unbalanced, the more it throttles down the writes, until it has enough cycles to write. Or, the more unbalanced it is regardless of time, the more normal writes are throttled back.

I see behavior where excessively overloaded systems get progressively more unbalanced, as no migration can happen and new data is only written to the one shard where it was written before. This happens naturally in collections that are not pre-split, but can happen even in pre-split situations over time running at 100% IO utilization.

Generated at Thu Feb 08 03:05:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.