[SERVER-37936] Mongod can't handle more than 32K connectipn Created: 06/Nov/18  Updated: 06/Nov/18  Resolved: 06/Nov/18

Status: Closed
Project: Core Server
Component/s: Concurrency, Performance, Replication
Affects Version/s: 3.2 Required
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Rizal Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:
  • Prepare 3 nodes for MongoDB & some nodes to conduct test connections 
  • Setup mongodb replica set of 3 nodes
  • Set increase ulimit in mongod.service 
  • Create concurrent connection as much as possible from test nodes
Participants:

 Description   

Mongod can't be access whenever mongod reach > 32K concurrent connections. Each time it reach the number, server can't be access

 

# MONGOD SERVER'S CONSOLE
$ uname -a
bash: fork: Cannot allocate memory
 
# OTHER SERVER TRIED TO CONNECT TO MONGOD SERVER
$ ssh mongodb2
ssh_exchange_identification: read: Connection reset by peer
 
# MongoDB Log
2018-11-06T05:21:09.810+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
2018-11-06T05:21:09.810+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
2018-11-06T05:21:09.855+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
2018-11-06T05:21:09.855+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
2018-11-06T05:21:09.921+0000 I NETWORK  [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable
2018-11-06T05:21:09.921+0000 I NETWORK  [initandlisten] failed to create thread after accepting new connection, closing connection
 
# Number connection was captured before the issue appears
$ mongo --eval "db.serverStatus().connections"
{                                                                                                                       
        "current" : 32272,                             
        "available" : 479728,                               
        "totalCreated" : NumberLong(32469)   
}
 

 

Mongod Environtment

  • Server: GCE (n1-standard-16)
  • OS: Debian 8
  • MongoDB version: 3.2.21

$ uname -a
Linux rz-tf-test-mongodb-3 3.16.0-6-amd64 #1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64 GNU/Linux
 
$ more /etc/debian_version
8.10
 
$ dpkg -l|grep mongod
ii  mongodb-org                           3.2.21                           amd64        MongoDB open source document-oriented database system (metapackage)
ii  mongodb-org-mongos                    3.2.21                           amd64        MongoDB sharded cluster query router
ii  mongodb-org-server                    3.2.21                           amd64        MongoDB database server
ii  mongodb-org-shell                     3.2.21                           amd64        MongoDB shell client
ii  mongodb-org-tools                     3.2.21                           amd64        MongoDB tools
 
 
$ ps axuwww|grep mongod
mongodb  23216 32.9  3.0 23508108 471380 ?     Ssl  05:16   7:13 /usr/bin/mongod --quiet --config /etc/mongod.conf
 
$ more /proc/23216/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             640000               640000               processes
Max open files            640000               640000               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       60235                60235                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
 
$ more /etc/mongod.conf
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  wiredTiger:
    engineConfig:
      cacheSizeGB: 12
      configString: "session_max=80000"
operationProfiling:
   slowOpThresholdMs: 200
   mode: slowOp
systemLog:
  verbosity: 0
  destination: file
  logAppend: true
  logRotate: reopen
  path: /var/log/mongodb/mongod.log
net:
  port: 27017
  bindIp: 0.0.0.0
replication:
  replSetName: cluster1
  oplogSizeMB: 100

 



 Comments   
Comment by Danny Hatcher (Inactive) [ 06/Nov/18 ]

Hello Rizal,

I recommend checking the various kernel settings within your OS configuration to confirm that everything is set according to our Operations Checklist. Additionally, please note that MongoDB 3.2 has reached its end-of-life so we strongly recommend you upgrading versions when you are able.

If you have additional questions on this topic, please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

Because this is not a bug within MongoDB, I will now close this SERVER ticket.

Thank you,

Danny

Generated at Thu Feb 08 04:47:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.