Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29206

MongoS [NetworkInterfaceASIO-TaskExecutorPool-0-0] ExceededTimeLimit: Operation timed out

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Duplicate
    • 3.2.12
    • None
    • Networking
    • None
    • Fully Compatible
    • ALL

    Description

      We continue to receive the below errors on our mongos logs; which causes connection issues to our mongo cluster. This happens throughout the day every day.

      We have 4 mongos nodes. About to have 8 to see if that helps. All 4 will see the issue.

      mongos.log:  2017-05-15T14:30:16.501+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r4-data-001.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.501+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r11-data-001.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.501+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r9-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.501+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r9-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.534+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r6-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.534+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r6-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:16.534+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r8-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      mongos.log:  2017-05-15T14:30:18.442+0000 I ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] Failed to connect to c0-r8-data-003.mongo-euc-1.com:27017 - ExceededTimeLimit: Operation timed out
      

      Below is our configuration file:

      ---
      net:
        port: '27017'
        bindIp: 0.0.0.0
      systemLog:
        path: "/var/log/mongodb/mongos.log"
        logAppend: true
        destination: file
      sharding:
        autoSplit: true
        configDB: c0-config-001.mongo-euc-1.com:27017,c0-config-002.mongo-euc-1.com:27017,c0-config-003.mongo-euc-1.com:27017
        chunkSize: 64
      setParameter:
        cursorTimeoutMillis: '1800000'
      security:
        keyFile: "*********"
      

      We have tried a variety of options none have helped:

      setParameter:
        cursorTimeoutMillis: '1800000'
        ShardingTaskExecutorPoolHostTimeoutMS: '300000000'
        ShardingTaskExecutorPoolMinSize: '50'
      

      setParameter:
        cursorTimeoutMillis: '1800000'
        ShardingTaskExecutorPoolRefreshTimeoutMS: 60000
        ShardingTaskExecutorPoolHostTimeoutMS: 300000000
        ShardingTaskExecutorPoolMinSize: 125
      

      When using any of the config options above; we see the errors occur pretty quickly.

      "opcounters" : {
              "insert" : 5369,
              "query" : 431092,
              "update" : 283293,
              "delete" : 5692,
              "getmore" : 88,
              "command" : 760326
          },
      

      Attachments

        Issue Links

          Activity

            People

              jason.carey@mongodb.com Jason Carey
              wegerer Wayne Egerer
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: