Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-32916

mongos creates lots of connections to nongod nodes

    XMLWordPrintable

    Details

    • Type: Question
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Networking
    • Labels:
      None

      Description

      HI

      we ran mongodb sharded cluster for several weeks, it works well.
      but we found mongos would creates lots of connections to primary nods in a shard set.
      when this occured, the respone time of this cluster will increase sharply
      and bigger than the number we can accepted.

      we found some typical statement from logs, as following:

      logs from mongos log:

      2018-01-25T22:10:00.006+0800 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Connecting to 10.136.142.35:28000
      2018-01-25T22:10:00.008+0800 I ACCESS [conn2318094] Successfully authenticated as principal useeeeeeeer on admin
      2018-01-25T22:10:00.013+0800 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to 10.136.180.52:28000, took 10111ms (10 connections now open to 10.136.180.52:
      28000)
      2018-01-25T22:10:00.013+0800 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Failed to connect to 10.136.179.52:28000 - HostUnreachable: End of file
      2018-01-25T22:10:00.013+0800 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-12-0] Failed to close stream: Transport endpoint is not connected
      2018-01-25T22:10:00.014+0800 I ASIO [NetworkInterfaceASIO-TaskExecutorPool-8-0] Connecting to 10.136.5.44:28000

      logs from mongod:
      2018-01-25T22:10:00.002+0800 I NETWORK [thread2] connection accepted from 10.136.180.33:30285 #7564018 (10233 connections now open)
      2018-01-25T22:10:00.006+0800 I NETWORK [thread2] connection refused because too many open connections: 10240

      notes: 1. we sets the maxincomming conections of mongod is 10240.
      2. when the cluster goes into usual status, the conections of mongod is about several hunderds.

      we could offer the detail logs and ftdc logs so as to find the root cause.

      Thanks.

        Attachments

        1. metrics.tar.gz
          29.90 MB
        2. mongod.log.tar.gz
          32.34 MB
        3. mongos.log.tar.gz
          12.59 MB

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: