-
Type: Bug
-
Resolution: Done
-
Priority: Major - P3
-
None
-
Affects Version/s: 3.2 Required
-
Component/s: Concurrency, Performance, Replication
-
Labels:None
-
ALL
-
Mongod can't be access whenever mongod reach > 32K concurrent connections. Each time it reach the number, server can't be access
# MONGOD SERVER'S CONSOLE $ uname -a bash: fork: Cannot allocate memory # OTHER SERVER TRIED TO CONNECT TO MONGOD SERVER $ ssh mongodb2 ssh_exchange_identification: read: Connection reset by peer # MongoDB Log 2018-11-06T05:21:09.810+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2018-11-06T05:21:09.810+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection 2018-11-06T05:21:09.855+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2018-11-06T05:21:09.855+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection 2018-11-06T05:21:09.921+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable 2018-11-06T05:21:09.921+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection # Number connection was captured before the issue appears $ mongo --eval "db.serverStatus().connections" { "current" : 32272, "available" : 479728, "totalCreated" : NumberLong(32469) }
Mongod Environtment
- Server: GCE (n1-standard-16)
- OS: Debian 8
- MongoDB version: 3.2.21
$ uname -a Linux rz-tf-test-mongodb-3 3.16.0-6-amd64 #1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64 GNU/Linux $ more /etc/debian_version 8.10 $ dpkg -l|grep mongod ii mongodb-org 3.2.21 amd64 MongoDB open source document-oriented database system (metapackage) ii mongodb-org-mongos 3.2.21 amd64 MongoDB sharded cluster query router ii mongodb-org-server 3.2.21 amd64 MongoDB database server ii mongodb-org-shell 3.2.21 amd64 MongoDB shell client ii mongodb-org-tools 3.2.21 amd64 MongoDB tools $ ps axuwww|grep mongod mongodb 23216 32.9 3.0 23508108 471380 ? Ssl 05:16 7:13 /usr/bin/mongod --quiet --config /etc/mongod.conf $ more /proc/23216/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 640000 640000 processes Max open files 640000 640000 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 60235 60235 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us $ more /etc/mongod.conf storage: dbPath: /var/lib/mongodb journal: enabled: true wiredTiger: engineConfig: cacheSizeGB: 12 configString: "session_max=80000" operationProfiling: slowOpThresholdMs: 200 mode: slowOp systemLog: verbosity: 0 destination: file logAppend: true logRotate: reopen path: /var/log/mongodb/mongod.log net: port: 27017 bindIp: 0.0.0.0 replication: replSetName: cluster1 oplogSizeMB: 100