Currently only one shard can be removed at a time. I trust that this was changed for a good reason but I feel a distinction should be made between actively draining a shard (which was problematic to do for more than one shard at the same time apparently) and tagging a shard as draining.
The reason being is currently if we want to remove 5 shards for example and when the first of those 5 shards is draining it will potentially move its chunks to one of the other 4 shards that we want to remove anyway. This makes our cluster down scales unnecessarily slow since it's needlessly moving the same chunks around multiple times.
The proposed solution is as follows :
- Allow marking multiple shards as "draining"
- Balancer picks one of these draining shards as the one that is going to be drained completely
- Balancer avoids moving chunks to shards that are marked as "draining"