Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29120

mongos connections increase indefinitely on chunk md refresh

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major - P3 Major - P3
    • None
    • 3.2.12
    • Networking, Sharding
    • None
    • ALL

    Description

      We have a pretty intensive process that stresses our mongod servers. When this process is running, we encounter the following issue: suddenly, there's a sudden increase in # of connections mongos opens, and then it starts taking more and more cpu until it becomes totally stuck. Please see the 2 graphs attached.

      When this happens, mongos logs shows A LOT of open/close connection messages (to the mongod servers). This trigger for this process seems to be a chunk move. When the move is complete, the ChunkManager refreshes - and then the connections starts opening and closing endlessly.

      Note that our mongod+mongos run on super strong machines, c4.4xlarge.

      Is this known / any workaround?

      SERVER-22611 (opened by myself) seems to be related, but fixed only in 3.4, we are using 3.2.12.

      Attachments

        1. dan1.png
          dan1.png
          121 kB
        2. dan2.png
          dan2.png
          97 kB

        Activity

          People

            kaloian.manassiev@mongodb.com Kaloian Manassiev
            yonido Yoni Douek
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: