-
Type:
Task
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Cluster Scalability
-
ClusterScalability 28Sep-12Oct, ClusterScalability 12Oct-26Oct, ClusterScalability 26Oct-9Nov
-
None
-
None
-
None
-
None
-
None
-
None
-
None
The coordinator runs a delta collector for both a recipient and a donor per resharding operation. those collectors block for the duration of the critical section, pinning 2 threads per resharding operation. there is additional blocking work during the critical section, such as sendCommandToShards round-trips: these are the refresh, abort and establish-participant calls, plus the document-count calls, and each blocks one pool thread for its round trip.
If the delta collectors can be made to run on their own thread pools, or be made to run asynchronous, then more concurrent reshardings can run together without a cap.
SERVER-136621 removes the explicit gate that allowed only one resharding operation at a time, behind featureFlagConcurrentResharding. Once the flag is on, several coordinators, donors and recipients run in the same node at once. Each resharding primary-only service uses a single shared, fixed-size thread pool per node, set by the following startup parameters:
- {{reshardingCoordinatorServiceMaxThreadCount (default: }}10)
- {{reshardingDonorServiceMaxThreadCount (default: }}5)
- {{reshardingRecipientServiceMaxThreadCount (default: }}10)
These defaults were sized when at most one resharding ran at a time. Several operations now compete for the same pools, and some of the work they run blocks a pool thread.
- is duplicated by
-
SERVER-135291 making coordinator pool usage in resharding non-blocking (delta collectors)
-
- Backlog
-
- is related to
-
SERVER-136621 Remove explicit gate on concurrent reshardings in coordinator service to allow a capped number of concurrent reshardings
-
- Closed
-
- related to
-
SERVER-138009 change stream monitors on donors and recipients are blocking on PoS thread pool
-
- Backlog
-