[SERVER-4313] sharded systems should throttle writes when chunk migrations cannot keep up Created: 17/Nov/11 Updated: 21/Dec/22 Resolved: 21/Dec/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Sharding |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Dwight Merriman | Assignee: | [DO NOT USE] Backlog - Sharding EMEA |
| Resolution: | Won't Do | Votes: | 3 |
| Labels: | sharding-common-backlog | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Sharding EMEA
|
| Participants: |
| Description |
|
In some cases data may be arriving at a shard faster than it can be migrated elsewhere. The typical situation where this is seen is on a brand new collection which does not have many chunks yet, to which a very large bulk import is sent. In this situation, it is necessary to throttle the injection at the shard; if injest rate is, for a long period of time, higher than migration output rate, the node will "overflow" with either data or workload (often the latter). Presplitting is a good technique to avoid this situation currently. Thus this ticket is to throttle writes s.t. migrations do not fall behind. For "phase one" inserts are the only thing that needs throttling. We can make a phase 2 ticket after that. |
| Comments |
| Comment by Connie Chen [ 21/Dec/22 ] |
|
For now, closing this as won't do, as we would not want to significantly impact a user's workload or application by throttling their writes. |
| Comment by Kevin J. Rice [ 15/Mar/13 ] |
|
We have 24 shards each on 2 boxes (done because excessive lock contention)(24 shards is 24x faster). We start out with all the data in 1 shards, or maybe 2, even though there are 48 total (we have 2 boxes). So the shard counts are: 1, 40, 0, 0, 0, ... and all the writes are happening on shard 1 (and a small number on shard 0). Since we peg the IO immediately due to it being sized for 48 shards, no balancing happens. So, the system should have some kind of radioactive-decay-curve for balancing: the longer it goes unbalanced, the more it throttles down the writes, until it has enough cycles to write. Or, the more unbalanced it is regardless of time, the more normal writes are throttled back. I see behavior where excessively overloaded systems get progressively more unbalanced, as no migration can happen and new data is only written to the one shard where it was written before. This happens naturally in collections that are not pre-split, but can happen even in pre-split situations over time running at 100% IO utilization. |