Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37936

Mongod can't handle more than 32K connectipn

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 3.2 Required
    • Fix Version/s: None
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide
      • Prepare 3 nodes for MongoDB & some nodes to conduct test connections 
      • Setup mongodb replica set of 3 nodes
      • Set increase ulimit in mongod.service 
      • Create concurrent connection as much as possible from test nodes
      Show
      Prepare 3 nodes for MongoDB & some nodes to conduct test connections  Setup mongodb replica set of 3 nodes Set increase ulimit in mongod.service  Create concurrent connection as much as possible from test nodes

      Description

      Mongod can't be access whenever mongod reach > 32K concurrent connections. Each time it reach the number, server can't be access

       

      # MONGOD SERVER'S CONSOLE
      $ uname -a
      bash: fork: Cannot allocate memory
       
      # OTHER SERVER TRIED TO CONNECT TO MONGOD SERVER
      $ ssh mongodb2
      ssh_exchange_identification: read: Connection reset by peer
       
      # MongoDB Log
      2018-11-06T05:21:09.810+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
      2018-11-06T05:21:09.810+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
      2018-11-06T05:21:09.855+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
      2018-11-06T05:21:09.855+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
      2018-11-06T05:21:09.921+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
      2018-11-06T05:21:09.921+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
       
      # Number connection was captured before the issue appears
      $ mongo --eval "db.serverStatus().connections"
      {                                                                                                                       
              "current" : 32272,                             
              "available" : 479728,                               
              "totalCreated" : NumberLong(32469)   
      }
       
      
      

       

      Mongod Environtment

      • Server: GCE (n1-standard-16)
      • OS: Debian 8
      • MongoDB version: 3.2.21

      $ uname -a
      Linux rz-tf-test-mongodb-3 3.16.0-6-amd64 #1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64 GNU/Linux
       
      $ more /etc/debian_version
      8.10
       
      $ dpkg -l|grep mongod
      ii  mongodb-org                           3.2.21                           amd64        MongoDB open source document-oriented database system (metapackage)
      ii  mongodb-org-mongos                    3.2.21                           amd64        MongoDB sharded cluster query router
      ii  mongodb-org-server                    3.2.21                           amd64        MongoDB database server
      ii  mongodb-org-shell                     3.2.21                           amd64        MongoDB shell client
      ii  mongodb-org-tools                     3.2.21                           amd64        MongoDB tools
       
       
      $ ps axuwww|grep mongod
      mongodb  23216 32.9  3.0 23508108 471380 ?     Ssl  05:16   7:13 /usr/bin/mongod --quiet --config /etc/mongod.conf
       
      $ more /proc/23216/limits
      Limit                     Soft Limit           Hard Limit           Units
      Max cpu time              unlimited            unlimited            seconds
      Max file size             unlimited            unlimited            bytes
      Max data size             unlimited            unlimited            bytes
      Max stack size            8388608              unlimited            bytes
      Max core file size        0                    unlimited            bytes
      Max resident set          unlimited            unlimited            bytes
      Max processes             640000               640000               processes
      Max open files            640000               640000               files
      Max locked memory         unlimited            unlimited            bytes
      Max address space         unlimited            unlimited            bytes
      Max file locks            unlimited            unlimited            locks
      Max pending signals       60235                60235                signals
      Max msgqueue size         819200               819200               bytes
      Max nice priority         0                    0
      Max realtime priority     0                    0
      Max realtime timeout      unlimited            unlimited            us
       
      $ more /etc/mongod.conf
      storage:
        dbPath: /var/lib/mongodb
        journal:
          enabled: true
        wiredTiger:
          engineConfig:
            cacheSizeGB: 12
            configString: "session_max=80000"
      operationProfiling:
         slowOpThresholdMs: 200
         mode: slowOp
      systemLog:
        verbosity: 0
        destination: file
        logAppend: true
        logRotate: reopen
        path: /var/log/mongodb/mongod.log
      net:
        port: 27017
        bindIp: 0.0.0.0
      replication:
        replSetName: cluster1
        oplogSizeMB: 100

       

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: