Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34884

Replication executor creates connection storm which should be limited

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Networking, Replication
    • None
    • Fully Compatible
    • ALL

    Description

      In my procuct environment, member of shard create a lot of connections to peer nodes,  which cause the peer reach the maxIncomingConnections config(8000), causing the mongodb service unavailable.

      As described in  SERVER-29429  which limit the maxConnecting from mongos to mongod,  we should also limit the maxConnecting from mongod to mongod to prevent connection storm.

       

      Below is part of mongod's log 

       

      // code placeholder
       
      2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
      ... 
      more similar lines ...
       
      
      

       

      Attachments

        Activity

          People

            dmitry.agranat@mongodb.com Dmitry Agranat
            zyd_com@126.com Zhang Youdong
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: