Details
-
Bug
-
Resolution: Done
-
Minor - P4
-
None
-
None
-
None
-
None
-
ALL
Description
(mongos startup log messages):
Wed Nov 24 05:17:20 /opt/mongodb/bin/mongos db version v1.7.3, pdfile
version 4.5 starting (--help for usage)
Wed Nov 24 05:17:20 git version:
fcb5790e7528a101a49c52e442ded0e2b7a043d6
Wed Nov 24 05:17:20 sys info: Linux [internal ec2 hostname]
2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64
BOOST_LIB_VERSION=1_41
Wed Nov 24 05:17:20 [mongosMain] waiting for connections on port 27017
After running mongos for a week and a half, I attempted to shut it
down before terminating its client (our app server via the 2.3 java
driver). That failed with the following stack trace:
Sat Dec 4 17:02:06 [mongosMain] connection accepted from
127.0.0.1:46213 #327
Sat Dec 4 17:02:20 [conn327] end connection 127.0.0.1:46213
Sat Dec 4 17:02:24 [mongosMain] connection accepted from
127.0.0.1:46214 #328
Sat Dec 4 17:02:24 [conn328] terminating, shutdown command received
Sat Dec 4 17:02:24 [conn328] dbexit: shutdown called rc:0 shutdown
called
Sat Dec 4 17:02:24 [mongosMain] dbexit: rc:0
Received signal 11
Backtrace: 0x51b195 0x3fb5832730 0x3fb5835c35 0x650e19 0x653aa5
0x3fb581ec1d 0x49b319 [weird chars removed here]
Backtrace: 0x51b195 0x3fb5832730 0x3fb58326b5 0x3fb5833e95
0x3fb586fb1b 0x3fb5875436 0x3fb5835ce2 0x650e19 0x59d849 0x56273c
0x617757 0x64a729 0x65862a 0x564592 0x67c930 0x3fb6407761
0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(exit+0x35)[0x3fb5835c35]
/opt/mongodb/bin/mongos[0x650e19]
/opt/mongodb/bin/mongos(main+0x2295)[0x653aa5]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3fb581ec1d]
/opt/mongodb/bin/
mongos(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x51)
[0x49b319]
===
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/lib64/libc.so.6(exit+0xe2)[0x3fb5835ce2]
/opt/mongodb/bin/mongos[0x650e19]
/opt/mongodb/bin/
mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb
+0x89)[0x59d849]
/opt/mongodb/bin/
mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE
+0x56c)[0x56273c]
/opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
+0x2d7)[0x617757]
/opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x299)[0x64a729]
/opt/mongodb/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x14a)[0x65862a]
/opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
+0x262)[0x564592]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Received signal 6
Backtrace: 0x51b195 0x3fb5832730 0x3fb58326b5 0x3fb5833e95
0x3fb586fb1b 0x3fb5875436 0x4f85ae 0x4f9c54 0x67c930 0x3fb6407761
0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
/lib64/libc.so.6[0x3fb5832730]
/lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
/lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
/lib64/libc.so.6[0x3fb586fb1b]
/lib64/libc.so.6[0x3fb5875436]
/opt/mongodb/bin/
mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE
+0x14e)[0x4f85ae]
/opt/mongodb/bin/
mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv
+0x74)[0x4f9c54]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Received signal 11
Backtrace: 0x51b195 0x3fb5832730 0x51ecd0 0x5207b8 0x521311 0x5544eb
0x668632 0x4f850c 0x4f9c54 0x67c930 0x3fb6407761 0x3fb58e0e7d
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
/lib64/libc.so.6[0x3fb5832730]
/opt/mongodb/bin/
mongos(_ZN5mongo16DBConnectionPool8onCreateEPNS_12DBClientBaseE+0x20)
[0x51ecd0]
/opt/mongodb/bin/
mongos(_ZN5mongo16DBConnectionPool13_finishCreateERKSsPNS_12DBClientBaseE
+0x58)[0x5207b8]
/opt/mongodb/bin/
mongos(_ZN5mongo16DBConnectionPool3getERKNS_16ConnectionStringE+0x111)
[0x521311]
/opt/mongodb/bin/mongos(_ZN5mongo15DistributedLock6unlockEv+0x6b)
[0x5544eb]
/opt/mongodb/bin/mongos(_ZN5mongo8Balancer3runEv+0x652)[0x668632]
/opt/mongodb/bin/
mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE
+0xac)[0x4f850c]
/opt/mongodb/bin/
mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv
+0x74)[0x4f9c54]
/opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
/lib64/libpthread.so.0[0x3fb6407761]
/lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
===
Sat Dec 4 17:04:13 [conn279] end connection [IP address]:60043
Sat Dec 4 17:04:13 [conn276] end connection [IP address]:60040
I killed the mongos process; it didn't complain when I restarted it,
and seemed to function normally. Recently, I tried to shut it down
again after bringing down our app server first. Although the mongos
process terminated, mongos.log revealed this shorter stack trace:
Tue Dec 14 00:09:36 [cursorTimeout] killing old cursor
3284674711636122215 idle for: 600030ms
Tue Dec 14 00:09:39 [conn714] terminating, shutdown command received
Tue Dec 14 00:09:39 [conn714] dbexit: shutdown called rc:0 shutdown
called
Tue Dec 14 00:09:39 CursorCache at shutdown - sharded: 106
passthrough: 6
Received signal 11
Backtrace: 0x51b195 0x3fb5832730 0x523726 0x523d5a 0x520874 0x520930
0x53d52e 0x565c31 0x56d677 0x64c7e1 0x64ff73 0x64fd64 0x64f2c2
0x3fb5835ce2 0x650e19 0x59d849 0x56273c 0x617757 0x64a729 0x65862a
/opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
/lib64/libc.so.6[0x3fb5832730]
/opt/mongodb/bin/
mongos(ZNSt8_Rb_treeISsSt4pairIKSsN5mongo11PoolForHostEESt10_Select1stIS4_ESt4lessISsESaIS4_EE13insert_uniqueESt17_Rb_tree_iteratorIS4_ERKS4
+0xb6)[0x523726]
/opt/mongodb/bin/
mongos(ZNSt3mapISsN5mongo11PoolForHostESt4lessISsESaISt4pairIKSsS1_EEEixERS5
+0x39a)[0x523d5a]
/opt/mongodb/bin/mongos(_ZN5mongo16DBConnectionPool4_getERKSs+0x34)
[0x520874]
/opt/mongodb/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x30)
[0x520930]
/opt/mongodb/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x47e)[0x53d52e]
/opt/mongodb/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x51)
[0x565c31]
/opt/mongodb/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev
+0x47)[0x56d677]
/opt/mongodb/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21)
[0x64c7e1]
/opt/mongodb/bin/
mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E
+0x2b3)[0x64ff73]
/opt/mongodb/bin/
mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E
+0xa4)[0x64fd64]
/opt/mongodb/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xe2)[0x64f2c2]
/lib64/libc.so.6(exit+0xe2)[0x3fb5835ce2]
/opt/mongodb/bin/mongos[0x650e19]
/opt/mongodb/bin/
mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb
+0x89)[0x59d849]
/opt/mongodb/bin/
mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE
+0x56c)[0x56273c]
/opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
+0x2d7)[0x617757]
/opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x299)[0x64a729]
/opt/mongodb/bin/
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
+0x14a)[0x65862a]
===