Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-2216

Check that mongos shutdowns cleanly even with ongoing operations

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor - P4 Minor - P4
    • None
    • None
    • None
    • None
    • ALL

    Description

      (mongos startup log messages):
      Wed Nov 24 05:17:20 /opt/mongodb/bin/mongos db version v1.7.3, pdfile
      version 4.5 starting (--help for usage)
      Wed Nov 24 05:17:20 git version:
      fcb5790e7528a101a49c52e442ded0e2b7a043d6
      Wed Nov 24 05:17:20 sys info: Linux [internal ec2 hostname]
      2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64
      BOOST_LIB_VERSION=1_41
      Wed Nov 24 05:17:20 [mongosMain] waiting for connections on port 27017

      After running mongos for a week and a half, I attempted to shut it
      down before terminating its client (our app server via the 2.3 java
      driver). That failed with the following stack trace:

      Sat Dec 4 17:02:06 [mongosMain] connection accepted from
      127.0.0.1:46213 #327
      Sat Dec 4 17:02:20 [conn327] end connection 127.0.0.1:46213
      Sat Dec 4 17:02:24 [mongosMain] connection accepted from
      127.0.0.1:46214 #328
      Sat Dec 4 17:02:24 [conn328] terminating, shutdown command received
      Sat Dec 4 17:02:24 [conn328] dbexit: shutdown called rc:0 shutdown
      called
      Sat Dec 4 17:02:24 [mongosMain] dbexit: rc:0
      Received signal 11
      Backtrace: 0x51b195 0x3fb5832730 0x3fb5835c35 0x650e19 0x653aa5
      0x3fb581ec1d 0x49b319 [weird chars removed here]
      Backtrace: 0x51b195 0x3fb5832730 0x3fb58326b5 0x3fb5833e95
      0x3fb586fb1b 0x3fb5875436 0x3fb5835ce2 0x650e19 0x59d849 0x56273c
      0x617757 0x64a729 0x65862a 0x564592 0x67c930 0x3fb6407761
      0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(exit+0x35)[0x3fb5835c35]
      /opt/mongodb/bin/mongos[0x650e19]
      /opt/mongodb/bin/mongos(main+0x2295)[0x653aa5]
      /lib64/libc.so.6(__libc_start_main+0xfd)[0x3fb581ec1d]
      /opt/mongodb/bin/
      mongos(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x51)
      [0x49b319]
      ===
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
      /lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
      /lib64/libc.so.6[0x3fb586fb1b]
      /lib64/libc.so.6[0x3fb5875436]
      /lib64/libc.so.6(exit+0xe2)[0x3fb5835ce2]
      /opt/mongodb/bin/mongos[0x650e19]
      /opt/mongodb/bin/
      mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb
      +0x89)[0x59d849]
      /opt/mongodb/bin/
      mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE
      +0x56c)[0x56273c]
      /opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
      +0x2d7)[0x617757]
      /opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x299)[0x64a729]
      /opt/mongodb/bin/
      mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
      +0x14a)[0x65862a]
      /opt/mongodb/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE
      +0x262)[0x564592]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===
      Received signal 6
      Backtrace: 0x51b195 0x3fb5832730 0x3fb58326b5 0x3fb5833e95
      0x3fb586fb1b 0x3fb5875436 0x4f85ae 0x4f9c54 0x67c930 0x3fb6407761
      0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
      /lib64/libc.so.6[0x3fb5832730]
      /lib64/libc.so.6(gsignal+0x35)[0x3fb58326b5]
      /lib64/libc.so.6(abort+0x175)[0x3fb5833e95]
      /lib64/libc.so.6[0x3fb586fb1b]
      /lib64/libc.so.6[0x3fb5875436]
      /opt/mongodb/bin/
      mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE
      +0x14e)[0x4f85ae]
      /opt/mongodb/bin/
      mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv
      +0x74)[0x4f9c54]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===
      Received signal 11
      Backtrace: 0x51b195 0x3fb5832730 0x51ecd0 0x5207b8 0x521311 0x5544eb
      0x668632 0x4f850c 0x4f9c54 0x67c930 0x3fb6407761 0x3fb58e0e7d
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
      /lib64/libc.so.6[0x3fb5832730]
      /opt/mongodb/bin/
      mongos(_ZN5mongo16DBConnectionPool8onCreateEPNS_12DBClientBaseE+0x20)
      [0x51ecd0]
      /opt/mongodb/bin/
      mongos(_ZN5mongo16DBConnectionPool13_finishCreateERKSsPNS_12DBClientBaseE
      +0x58)[0x5207b8]
      /opt/mongodb/bin/
      mongos(_ZN5mongo16DBConnectionPool3getERKNS_16ConnectionStringE+0x111)
      [0x521311]
      /opt/mongodb/bin/mongos(_ZN5mongo15DistributedLock6unlockEv+0x6b)
      [0x5544eb]
      /opt/mongodb/bin/mongos(_ZN5mongo8Balancer3runEv+0x652)[0x668632]
      /opt/mongodb/bin/
      mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE
      +0xac)[0x4f850c]
      /opt/mongodb/bin/
      mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv
      +0x74)[0x4f9c54]
      /opt/mongodb/bin/mongos(thread_proxy+0x80)[0x67c930]
      /lib64/libpthread.so.0[0x3fb6407761]
      /lib64/libc.so.6(clone+0x6d)[0x3fb58e0e7d]
      ===
      Sat Dec 4 17:04:13 [conn279] end connection [IP address]:60043
      Sat Dec 4 17:04:13 [conn276] end connection [IP address]:60040

      I killed the mongos process; it didn't complain when I restarted it,
      and seemed to function normally. Recently, I tried to shut it down
      again after bringing down our app server first. Although the mongos
      process terminated, mongos.log revealed this shorter stack trace:

      Tue Dec 14 00:09:36 [cursorTimeout] killing old cursor
      3284674711636122215 idle for: 600030ms
      Tue Dec 14 00:09:39 [conn714] terminating, shutdown command received
      Tue Dec 14 00:09:39 [conn714] dbexit: shutdown called rc:0 shutdown
      called
      Tue Dec 14 00:09:39 CursorCache at shutdown - sharded: 106
      passthrough: 6
      Received signal 11
      Backtrace: 0x51b195 0x3fb5832730 0x523726 0x523d5a 0x520874 0x520930
      0x53d52e 0x565c31 0x56d677 0x64c7e1 0x64ff73 0x64fd64 0x64f2c2
      0x3fb5835ce2 0x650e19 0x59d849 0x56273c 0x617757 0x64a729 0x65862a
      /opt/mongodb/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x51b195]
      /lib64/libc.so.6[0x3fb5832730]
      /opt/mongodb/bin/
      mongos(ZNSt8_Rb_treeISsSt4pairIKSsN5mongo11PoolForHostEESt10_Select1stIS4_ESt4lessISsESaIS4_EE13insert_uniqueESt17_Rb_tree_iteratorIS4_ERKS4
      +0xb6)[0x523726]
      /opt/mongodb/bin/
      mongos(ZNSt3mapISsN5mongo11PoolForHostESt4lessISsESaISt4pairIKSsS1_EEEixERS5
      +0x39a)[0x523d5a]
      /opt/mongodb/bin/mongos(_ZN5mongo16DBConnectionPool4_getERKSs+0x34)
      [0x520874]
      /opt/mongodb/bin/mongos(_ZN5mongo16DBConnectionPool3getERKSs+0x30)
      [0x520930]
      /opt/mongodb/bin/mongos(_ZN5mongo14DBClientCursorD0Ev+0x47e)[0x53d52e]
      /opt/mongodb/bin/mongos(_ZN5mongo21FilteringClientCursorD1Ev+0x51)
      [0x565c31]
      /opt/mongodb/bin/mongos(_ZN5mongo27ParallelSortClusteredCursorD0Ev
      +0x47)[0x56d677]
      /opt/mongodb/bin/mongos(_ZN5mongo19ShardedClientCursorD0Ev+0x21)
      [0x64c7e1]
      /opt/mongodb/bin/
      mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E
      +0x2b3)[0x64ff73]
      /opt/mongodb/bin/
      mongos(_ZNSt8_Rb_treeIxSt4pairIKxN5boost10shared_ptrIN5mongo19ShardedClientCursorEEEESt10_Select1stIS7_ESt4lessIxESaIS7_EE8_M_eraseEPSt13_Rb_tree_nodeIS7_E
      +0xa4)[0x64fd64]
      /opt/mongodb/bin/mongos(_ZN5mongo11CursorCacheD1Ev+0xe2)[0x64f2c2]
      /lib64/libc.so.6(exit+0xe2)[0x3fb5835ce2]
      /opt/mongodb/bin/mongos[0x650e19]
      /opt/mongodb/bin/
      mongos(_ZN5mongo11CmdShutdown3runERKSsRNS_7BSONObjERSsRNS_14BSONObjBuilderEb
      +0x89)[0x59d849]
      /opt/mongodb/bin/
      mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderE
      +0x56c)[0x56273c]
      /opt/mongodb/bin/mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE
      +0x2d7)[0x617757]
      /opt/mongodb/bin/mongos(_ZN5mongo7Request7processEi+0x299)[0x64a729]
      /opt/mongodb/bin/
      mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE
      +0x14a)[0x65862a]
      ===

      Attachments

        Activity

          People

            alerner Alberto Lerner
            alerner Alberto Lerner
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: