I have a mongo server with high read/write in a short time. I used python and pymongo, when I wake up this morning I found no connection can make to mongod master cause it's connections reached 19992, its a pretty much scary number

even I stopped all the program, the connection number seems no change

I have 1 master and 7 slaves

insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time
localhost:27417 0 0 0 0 0 1 0 624m 910m 258m 0 0 0 0|0 7|0 62b 1k 19792 M 10:58:49
localhost:27418 *0 *0 *0 *0 0 1|0 0 624m 766m 183m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27419 *0 *0 *0 *0 0 1|0 0 624m 768m 184m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27420 *0 *0 *0 *0 0 1|0 0 624m 765m 174m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27421 *0 *0 *0 *0 0 1|0 0 624m 764m 171m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27422 *0 *0 *0 *0 0 1|0 0 624m 764m 174m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27423 *0 *0 *0 *0 0 1|0 0 624m 763m 172m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27424 *0 *0 *0 *0 0 1|0 0 624m 770m 179m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49

I checked the socket connections

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED
udp 0 0 0.0.0.0:48514 0.0.0.0:*
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432
unix 2 [ ] DGRAM 3651 @/org/kernel/udev/udevd
unix 5 [ ] DGRAM 16472048 /dev/log
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket
unix 2 [ ] DGRAM 16792651
unix 2 [ ] DGRAM 16472057
unix 2 [ ] DGRAM 16472052

insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time

localhost:27417 0 0 0 0 2 1 0 624m 2.87g 287m 0 0 0 0|0 7|0 162b 1k 19992 M 10:36:16