Resharding change streams monitor can pin the donor service thread pool and block DDL for minutes

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Duplicate
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Cluster Scalability
    • ALL
    • v9.0
    • ClusterScalability 20Jul-3Aug, ClusterScalability 3Aug-17Aug
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      The resharding change streams monitor can starve the donor state machine for many minutes (see BF-44929).

      The monitor consumes change events in a synchronous loop that runs on the ReshardingDonorService executor, which has only 2 threads by default. A batch only ends on the final event, the batch size limit, or the batch time limit (default 5 minutes). On an idle collection that means one monitor pins an executor thread for the full 5 minutes.

      In the failing runs the balancer's random moveCollections keep aborting with DuplicateKey. The aborted operation's monitor would keep running and pin one thread, then the next moveCollection's monitor would pin the other. That operation's donor state transition would then stay queued for ~300s while holding the collection DDL lock. The DDL lock timeout is also 5 minutes, so a concurrent dropIndexes fails with LockBusy if it starts waiting just after the reshard takes the lock.

      This started with featureFlagReshardingVerification being enabled (SERVER-94478).

            Assignee:
            Kruti Shah
            Reporter:
            Henri Nikku
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: