Details
-
Bug
-
Resolution: Done
-
Major - P3
-
None
-
3.2 Required
-
None
-
ALL
-
Description
Mongod can't be access whenever mongod reach > 32K concurrent connections. Each time it reach the number, server can't be access
# MONGOD SERVER'S CONSOLE
|
$ uname -a
|
bash: fork: Cannot allocate memory
|
|
|
# OTHER SERVER TRIED TO CONNECT TO MONGOD SERVER
|
$ ssh mongodb2
|
ssh_exchange_identification: read: Connection reset by peer
|
|
|
# MongoDB Log
|
2018-11-06T05:21:09.810+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable |
2018-11-06T05:21:09.810+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection |
2018-11-06T05:21:09.855+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable |
2018-11-06T05:21:09.855+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection |
2018-11-06T05:21:09.921+0000 I NETWORK [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable |
2018-11-06T05:21:09.921+0000 I NETWORK [initandlisten] failed to create thread after accepting new connection, closing connection |
|
|
# Number connection was captured before the issue appears
|
$ mongo --eval "db.serverStatus().connections" |
{
|
"current" : 32272, |
"available" : 479728, |
"totalCreated" : NumberLong(32469) |
}
|
|
|
|
Mongod Environtment
- Server: GCE (n1-standard-16)
- OS: Debian 8
- MongoDB version: 3.2.21
$ uname -a
|
Linux rz-tf-test-mongodb-3 3.16.0-6-amd64 #1 SMP Debian 3.16.56-1+deb8u1 (2018-05-08) x86_64 GNU/Linux |
|
|
$ more /etc/debian_version
|
8.10
|
|
|
$ dpkg -l|grep mongod
|
ii mongodb-org 3.2.21 amd64 MongoDB open source document-oriented database system (metapackage) |
ii mongodb-org-mongos 3.2.21 amd64 MongoDB sharded cluster query router |
ii mongodb-org-server 3.2.21 amd64 MongoDB database server |
ii mongodb-org-shell 3.2.21 amd64 MongoDB shell client |
ii mongodb-org-tools 3.2.21 amd64 MongoDB tools |
|
|
|
|
$ ps axuwww|grep mongod
|
mongodb 23216 32.9 3.0 23508108 471380 ? Ssl 05:16 7:13 /usr/bin/mongod --quiet --config /etc/mongod.conf |
|
|
$ more /proc/23216/limits |
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes |
Max core file size 0 unlimited bytes |
Max resident set unlimited unlimited bytes
|
Max processes 640000 640000 processes |
Max open files 640000 640000 files |
Max locked memory unlimited unlimited bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 60235 60235 signals |
Max msgqueue size 819200 819200 bytes |
Max nice priority 0 0 |
Max realtime priority 0 0 |
Max realtime timeout unlimited unlimited us
|
|
|
$ more /etc/mongod.conf
|
storage:
|
dbPath: /var/lib/mongodb
|
journal:
|
enabled: true |
wiredTiger:
|
engineConfig:
|
cacheSizeGB: 12 |
configString: "session_max=80000" |
operationProfiling:
|
slowOpThresholdMs: 200 |
mode: slowOp
|
systemLog:
|
verbosity: 0 |
destination: file
|
logAppend: true |
logRotate: reopen
|
path: /var/log/mongodb/mongod.log
|
net:
|
port: 27017 |
bindIp: 0.0.0.0 |
replication:
|
replSetName: cluster1
|
oplogSizeMB: 100 |