ReshardingCoordinatorService thread-pool starvation when concurrent resharding operations run concurrently — post-strict-consistency continuation can be starved indefinitely

ReshardingCoordinatorService runs all active ReshardingCoordinator instances on one shared, fixed-size thread pool (reshardingCoordinatorServiceMaxThreadCount, default 5). Two of that pool's periodic per-operation tasks — ReshardingRecipientPostCloningDeltaCollector and the analogous donor delta collector — issue a broadcast RPC to every shard and block the calling worker thread synchronously on AsyncRequestsSender::get() instead of chaining asynchronously. With N resharding operations running concurrently, up to N pool threads can be tied up this way at once. Once free capacity drops too low, another operation's coordinator can be left permanently unable to get a thread to run its post-strict-consistency continuation — even though the data that continuation needs is already ready. The operation then sits in state: "blocking-writes" forever, with no error and no timeout (in test environments, reshardingCriticalSectionTimeoutMillis is often set very high).

Suggested fix

Make ReshardingRecipientPostCloningDeltaCollector (and the donor equivalent)'s shard fan-out non-blocking — chain a continuation off the AsyncRequestsSender future instead of calling .get() on it — so the collector doesn't occupy a whole pool thread for the RPC round-trip's duration. This removes the O(N)-threads-per-concurrent-operation consumption pattern entirely, rather than just raising the N at which it reappears.

The coordinator runs a delta collector for both a recipient and a donor per resharding operation. those collectors block for the duration of the critical section, pinning 2 threads per resharding operation. there is additional blocking work during the critical section, such as sendCommandToShards round-trips: these are the refresh, abort and establish-participant calls, plus the document-count calls, and each blocks one pool thread for its round trip.

If the delta collectors can be made to run on their own thread pools, or be made to run asynchronous, then more concurrent reshardings can run together without a cap.

SERVER-136621 removes the explicit gate that allowed only one resharding operation at a time, behind featureFlagConcurrentResharding. Once the flag is on, several coordinators, donors and recipients run in the same node at once. Each resharding primary-only service uses a single shared,fixed-size thread pool per node, set by the following startup parameters:

• {{reshardingCoordinatorServiceMaxThreadCount (default: }}10)
• {{reshardingDonorServiceMaxThreadCount (default: }}5)
• {{reshardingRecipientServiceMaxThreadCount (default: }}10)

  These defaults were sized when at most one resharding ran at a time. Several operations now compete for the same pools, and some of the work they run blocks a pool thread.