Currently there is no way to remove a shard from a cluster without migrating the chunks on that shard away to other shards. This makes it impossible to remove a shard which is entirely lost (because we can't move the chunks away because we don't have them). We also can't introduce a new repl set on top of that shard, because it'll have the wrong term.
We should provide a mechanism to update the routing table such that:
- A given shard is removed
- Chunk info for chunks on that shard is removed
- key space for sharded collections on that shard are re-allocated to other shards (so as to avoid leaving gaps in the routing table)
- the shard in question is completely unavailable
We will add a new option to the removeShard command that puts a shard into forced draining mode in order to provide the above.