Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34582

AsyncRequestsSender can block network threads during construction

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical - P2
    • Resolution: Fixed
    • Affects Version/s: 3.6.4
    • Fix Version/s: 3.6.5, 3.7.7
    • Component/s: Networking
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v3.6
    • Sprint:
      Platforms 2018-04-23, Platforms 2018-05-07
    • Linked BF Score:
      0

      Description

      • The AsyncRequestsSender holds a lock during construction and work scheduling.
      • This lock prevents callbacks from running if their response comes back during scheduling.
      • Scheduling can take a long time (up to 20 seconds per shard) if a read preference cannot be satisfied. This is done by a blocking call into the ReplicaSetMonitor

      The bad sequence of events is:

      1. Scatter gather request to two shards is dispatched
      2. The first host suceeds in targetting and runs
      3. The second host cannot satisfy it's read pref, blocking holding a lock
      4. The first request suceeds, blocking in running _handleResponse

      If you have enough of those, you can saturate all background networking workers, making your mongos completely unresponsive until targeting can succeed.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jason.carey Jason Carey
              Reporter:
              jason.carey Jason Carey
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: