[SERVER-3146] mongodb didn't release the connections Created: 26/May/11  Updated: 30/Mar/12  Resolved: 29/May/11

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 1.8.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: davyzhang Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 2
Labels: connection
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

centos5.5


Operating System: Linux
Participants:

 Description   

I have a mongo server with high read/write in a short time. I used python and pymongo, when I wake up this morning I found no connection can make to mongod master cause it's connections reached 19992, its a pretty much scary number

even I stopped all the program, the connection number seems no change

I have 1 master and 7 slaves

insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time
localhost:27417 0 0 0 0 0 1 0 624m 910m 258m 0 0 0 0|0 7|0 62b 1k 19792 M 10:58:49
localhost:27418 *0 *0 *0 *0 0 1|0 0 624m 766m 183m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27419 *0 *0 *0 *0 0 1|0 0 624m 768m 184m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27420 *0 *0 *0 *0 0 1|0 0 624m 765m 174m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27421 *0 *0 *0 *0 0 1|0 0 624m 764m 171m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27422 *0 *0 *0 *0 0 1|0 0 624m 764m 174m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27423 *0 *0 *0 *0 0 1|0 0 624m 763m 172m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49
localhost:27424 *0 *0 *0 *0 0 1|0 0 624m 770m 179m 0 0 0 0|0 0|1 62b 1k 2 SLV 10:58:49

I checked the socket connections

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:60000 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28423 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:28424 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:38422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27417 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27418 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27419 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27420 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27421 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27422 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:27423 0.0.0.0:* LISTEN
tcp 0 0 222.73.61.42:27420 222.73.61.43:38249 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56699 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56698 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56697 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56696 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56702 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56701 ESTABLISHED
tcp 0 0 127.0.0.1:27417 127.0.0.1:56700 ESTABLISHED
tcp 0 0 222.73.61.42:27422 222.73.61.43:33616 ESTABLISHED
tcp 0 0 222.73.61.42:27417 222.73.61.43:60218 ESTABLISHED
tcp 0 0 222.73.61.42:27423 222.73.61.43:33035 ESTABLISHED
tcp 0 3324 222.73.61.42:58422 119.85.195.88:54295 ESTABLISHED
tcp 0 0 222.73.61.42:27424 222.73.61.43:55825 ESTABLISHED
tcp 0 0 222.73.61.42:54279 222.215.136.8:80 ESTABLISHED
tcp 0 0 222.73.61.42:27418 222.73.61.43:37093 ESTABLISHED
tcp 0 0 222.73.61.42:27419 222.73.61.43:38346 ESTABLISHED
tcp 0 0 127.0.0.1:56702 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56701 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56700 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56699 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56698 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56697 127.0.0.1:27417 ESTABLISHED
tcp 0 0 127.0.0.1:56696 127.0.0.1:27417 ESTABLISHED
tcp 0 0 222.73.61.42:27421 222.73.61.43:39843 ESTABLISHED
udp 0 0 0.0.0.0:48514 0.0.0.0:*
udp 0 0 222.73.61.42:50721 61.128.128.68:53 ESTABLISHED
udp 0 0 127.0.0.1:52274 127.0.0.1:52274 ESTABLISHED
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 9081684 /var/run/nscd/socket
unix 2 [ ACC ] STREAM LISTENING 18011686 /tmp/mongodb-27417.sock
unix 2 [ ACC ] STREAM LISTENING 18011689 /tmp/mongodb-27422.sock
unix 2 [ ACC ] STREAM LISTENING 18011691 /tmp/mongodb-28422.sock
unix 2 [ ACC ] STREAM LISTENING 18011697 /tmp/mongodb-27420.sock
unix 2 [ ACC ] STREAM LISTENING 18011702 /tmp/mongodb-28417.sock
unix 2 [ ACC ] STREAM LISTENING 18011693 /tmp/mongodb-27421.sock
unix 2 [ ACC ] STREAM LISTENING 18011695 /tmp/mongodb-28421.sock
unix 2 [ ACC ] STREAM LISTENING 18011699 /tmp/mongodb-28420.sock
unix 2 [ ACC ] STREAM LISTENING 18011710 /tmp/mongodb-27419.sock
unix 2 [ ACC ] STREAM LISTENING 18011713 /tmp/mongodb-28419.sock
unix 2 [ ACC ] STREAM LISTENING 18011716 /tmp/mongodb-27418.sock
unix 2 [ ACC ] STREAM LISTENING 18011719 /tmp/mongodb-28418.sock
unix 2 [ ACC ] STREAM LISTENING 18011722 /tmp/mongodb-27424.sock
unix 2 [ ACC ] STREAM LISTENING 18011725 /tmp/mongodb-28424.sock
unix 2 [ ACC ] STREAM LISTENING 18011728 /tmp/mongodb-27423.sock
unix 2 [ ACC ] STREAM LISTENING 18011731 /tmp/mongodb-28423.sock
unix 2 [ ACC ] STREAM LISTENING 12771288 /tmp/.s.PGSQL.5432
unix 2 [ ] DGRAM 3651 @/org/kernel/udev/udevd
unix 5 [ ] DGRAM 16472048 /dev/log
unix 2 [ ] STREAM CONNECTED 18706425 /var/run/nscd/socket
unix 2 [ ] DGRAM 16792651
unix 2 [ ] DGRAM 16472057
unix 2 [ ] DGRAM 16472052

insert query update delete getmore command flushes mapped vsize res faults locked % idx miss % qr|qw ar|aw netIn netOut conn repl time

localhost:27417 0 0 0 0 2 1 0 624m 2.87g 287m 0 0 0 0|0 7|0 162b 1k 19992 M 10:36:16



 Comments   
Comment by davyzhang [ 29/May/11 ]

Sorry.... I didn't keep the log for long, after I restart the mongodb the previous log was.... gone. Till now the problem didn't show up, I will submit the log if it shows up again. and is it possible to add a command to view the detail of connections like netstat , which ip is using this connection and the last query from this conection. Even manually kill the connection at server side. Many thanks!

Comment by Eliot Horowitz (Inactive) [ 27/May/11 ]

Can you attach server logs

Comment by davyzhang [ 27/May/11 ]

and today the same problem happened in my test server again I can not connect to db using mongo shell even

mongo localhost:27417/dt_base
MongoDB shell version: 1.8.1
connecting to: localhost:27417/dt_base
Fri May 27 10:51:15 MessagingPort recv() errno:104 Connection reset by peer 127.0.0.1:27417
Fri May 27 10:51:15 SocketException: remote: error: 9001 socket exception [1]
Fri May 27 10:51:15 DBClientCursor::init call() failed
Fri May 27 10:51:15 terminate() called in shell, printing stack:
0x80ad346 0x61d515 0x61d552 0x61d68a 0x810fc46 0x81217fe 0x812af06 0x80cd86d 0x815e115 0x81f4f5e 0x81f666f 0x82002a6 0x81f58e0 0x81bb751 0x81bb6a8 0x81bb5b2 0x816a48d 0x80c54e4 0x814fc87 0x80b0fe4
mongo(_Z11myterminatev+0x56) [0x80ad346]
/usr/lib/libstdc++.so.6 [0x61d515]
/usr/lib/libstdc++.so.6 [0x61d552]
/usr/lib/libstdc++.so.6 [0x61d68a]
mongo(_ZN5mongo9uassertedEiPKc+0xd6) [0x810fc46]
mongo(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x4ae) [0x81217fe]
mongo(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x96) [0x812af06]
mongo(_ZN5mongo10shellUtils9onConnectERNS_20DBClientWithCommandsE+0x19d) [0x80cd86d]
mongo(ZN5mongo26mongo_external_constructorEP9JSContextP8JSObjectjPlS4+0x145) [0x815e115]
mongo(js_Invoke+0xc34) [0x81f4f5e]
mongo(js_InvokeConstructor+0x295) [0x81f666f]
mongo(js_Interpret+0x9ad5) [0x82002a6]
mongo(js_Execute+0x2f4) [0x81f58e0]
mongo(JS_EvaluateUCScriptForPrincipals+0xa7) [0x81bb751]
mongo(JS_EvaluateUCScript+0x43) [0x81bb6a8]
mongo(JS_EvaluateScript+0x6c) [0x81bb5b2]
mongo(_ZN5mongo7SMScope4execERKNS_10StringDataERKSsbbbi+0x11d) [0x816a48d]
mongo(_ZN5mongo10shellUtils9initScopeERNS_5ScopeE+0xf4) [0x80c54e4]
mongo(_ZN5mongo12ScriptEngine8newScopeEv+0x37) [0x814fc87]
mongo(_Z5_mainiPPc+0x1d04) [0x80b0fe4]

Comment by davyzhang [ 26/May/11 ]

> db.serverStatus(1)
{
"host" : "42yl:27417",
"version" : "1.8.1",
"process" : "mongod",
"uptime" : 71732,
"uptimeEstimate" : 71470,
"localTime" : ISODate("2011-05-26T03:02:48.301Z"),
"globalLock" : {
"totalTime" : 71732232290,
"lockTime" : 149471421,
"ratio" : 0.002083741384148133,
"currentQueue" :

{ "total" : 0, "readers" : 0, "writers" : 0 }

,
"activeClients" :

{ "total" : 7, "readers" : 7, "writers" : 0 }

},
"mem" :

{ "bits" : 32, "resident" : 258, "virtual" : 910, "supported" : true, "mapped" : 624 }

,
"connections" :

{ "current" : 19792, "available" : 208 }

,
"extra_info" :

{ "note" : "fields vary by platform", "heap_usage_bytes" : 562688, "page_faults" : 965 }

,
"indexCounters" : {
"btree" :

{ "accesses" : 12789, "hits" : 12789, "misses" : 0, "resets" : 0, "missRatio" : 0 }

},
"backgroundFlushing" :

{ "flushes" : 1195, "total_ms" : 848633, "average_ms" : 710.1531380753138, "last_ms" : 101, "last_finished" : ISODate("2011-05-26T03:02:18.691Z") }

,
"cursors" :

{ "totalOpen" : 7, "clientCursors_size" : 7, "timedOut" : 0 }

,
"network" :

{ "bytesIn" : 685742402, "bytesOut" : 2742190274, "numRequests" : 3800041 }

,
"repl" :

{ "ismaster" : true }

,
"opcounters" :

{ "insert" : 104225, "query" : 9, "update" : 925044, "delete" : 45734, "getmore" : 1642979, "command" : 1119290 }

,
"asserts" :

{ "regular" : 0, "warning" : 56, "msg" : 0, "user" : 0, "rollovers" : 0 }

,
"writeBacksQueued" : false,
"ok" : 1
}

Generated at Thu Feb 08 03:02:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.