Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-5713

Can't shutdown mongodb

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 2.0.3
    • Sharding
    • None
    • Linux

    Description

      Using a sharded cluster consisting of 3 shards,when i shutdown mongodb ,first kill pid,but the mongodb can't shutdown after almost 2 hour,the log is

      -----------------------log info----------------------
      Thu Apr 26 17:57:11 [mongosMain] connection accepted from 202.173.8.34:33212 #1586431
      Thu Apr 26 17:57:11 [conn1586431] end connection 202.173.8.34:33212
      Thu Apr 26 17:57:13 [mongosMain] connection accepted from 202.173.8.35:44814 #1586432
      Thu Apr 26 17:57:13 [conn1586432] end connection 202.173.8.35:44814
      Thu Apr 26 17:57:14 [mongosMain] connection accepted from 202.173.10.200:41622 #1586433
      Thu Apr 26 17:57:14 [conn1586433] end connection 202.173.10.200:41622
      Thu Apr 26 17:57:15 [mongosMain] connection accepted from 1.8.101.35:50989 #1586434
      Thu Apr 26 17:57:15 [conn1586434] end connection 1.8.101.35:50989
      Thu Apr 26 17:57:16 [mongosMain] connection accepted from 202.173.8.34:33434 #1586435
      Thu Apr 26 17:57:16 [conn1586435] end connection 202.173.8.34:33434
      Thu Apr 26 17:57:18 [mongosMain] connection accepted from 202.173.8.35:45036 #1586436
      Thu Apr 26 17:57:18 [conn1586436] end connection 202.173.8.35:45036
      Thu Apr 26 17:57:21 [mongosMain] connection accepted from 202.173.8.34:33653 #1586437
      Thu Apr 26 17:57:21 [conn1586437] end connection 202.173.8.34:33653
      Thu Apr 26 17:57:23 [mongosMain] connection accepted from 202.173.8.35:45265 #1586438
      Thu Apr 26 17:57:23 [conn1586438] end connection 202.173.8.35:45265
      Thu Apr 26 17:57:26 [mongosMain] connection accepted from 202.173.8.34:33879 #1586439
      Thu Apr 26 17:57:26 [conn1586439] end connection 202.173.8.34:33879
      Thu Apr 26 17:57:27 [mongosMain] connection accepted from 202.173.8.34:33920 #1586440
      Thu Apr 26 17:57:27 [conn1586440] end connection 202.173.8.34:33920
      Thu Apr 26 17:57:28 [mongosMain] connection accepted from 202.173.8.35:45487 #1586441
      Thu Apr 26 17:57:28 [conn1586441] end connection 202.173.8.35:45487
      Thu Apr 26 17:57:32 [mongosMain] connection accepted from 202.173.8.34:34141 #1586442
      Thu Apr 26 17:57:32 [conn1586442] end connection 202.173.8.34:34141
      Thu Apr 26 17:57:33 [mongosMain] connection accepted from 202.173.8.35:45706 #1586443
      Thu Apr 26 17:57:33 [conn1586443] end connection 202.173.8.35:45706
      Thu Apr 26 17:57:37 [mongosMain] connection accepted from 202.173.8.34:45819 #1586444
      Thu Apr 26 17:57:37 [conn1586444] end connection 202.173.8.34:45819
      Thu Apr 26 17:57:38 [mongosMain] connection accepted from 202.173.8.35:45927 #1586445
      Thu Apr 26 17:57:38 [conn1586445] end connection 202.173.8.35:45927
      Thu Apr 26 17:57:42 [mongosMain] connection accepted from 202.173.8.34:46039 #1586446
      Thu Apr 26 17:57:42 [conn1586446] end connection 202.173.8.34:46039
      Thu Apr 26 17:57:43 [mongosMain] connection accepted from 202.173.8.35:46149 #1586447
      Thu Apr 26 17:57:43 [conn1586447] end connection 202.173.8.35:46149
      Thu Apr 26 17:57:47 [mongosMain] connection accepted from 202.173.8.34:46262 #1586448
      Thu Apr 26 17:57:47 [conn1586448] end connection 202.173.8.34:46262
      Thu Apr 26 17:57:48 [mongosMain] connection accepted from 202.173.8.35:46372 #1586449
      Thu Apr 26 17:57:48 [conn1586449] end connection 202.173.8.35:46372
      Thu Apr 26 17:57:52 [mongosMain] connection accepted from 202.173.8.34:46483 #1586450
      Thu Apr 26 17:57:52 [conn1586450] end connection 202.173.8.34:46483
      Thu Apr 26 17:57:53 [mongosMain] connection accepted from 202.173.8.35:46596 #1586451
      Thu Apr 26 17:57:53 [conn1586451] end connection 202.173.8.35:46596
      Thu Apr 26 17:57:57 [mongosMain] connection accepted from 202.173.8.34:46707 #1586452
      Thu Apr 26 17:57:57 [conn1586452] end connection 202.173.8.34:46707
      Thu Apr 26 17:57:58 [mongosMain] connection accepted from 202.173.8.35:46817 #1586453
      Thu Apr 26 17:57:58 [conn1586453] end connection 202.173.8.35:46817
      Thu Apr 26 17:58:02 [mongosMain] connection accepted from 202.173.8.34:46927 #1586454
      Thu Apr 26 17:58:02 [conn1586454] end connection 202.173.8.34:46927
      Thu Apr 26 17:58:03 [mongosMain] connection accepted from 202.173.8.35:47038 #1586455
      Thu Apr 26 17:58:03 [conn1586455] end connection 202.173.8.35:47038
      Thu Apr 26 17:58:07 [mongosMain] connection accepted from 202.173.8.34:47148 #1586456
      Thu Apr 26 17:58:07 [conn1586456] end connection 202.173.8.34:47148
      Thu Apr 26 17:58:08 [mongosMain] connection accepted from 202.173.8.35:47260 #1586457
      Thu Apr 26 17:58:08 [conn1586457] end connection 202.173.8.35:47260
      Thu Apr 26 17:58:09 [mongosMain] dbexit: received signal 15 rc:0 received signal 15

          • glibc detected *** ./mongos: free(): invalid pointer: 0x0000000000b18320 ***
            ======= Backtrace: =========
            /lib64/libc.so.6[0x3c41c71ce2]
            /lib64/libc.so.6(cfree+0x8c)[0x3c41c7590c]
            ./mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1b9)[0x525c69]
            ./mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x527c24]
            ./mongos(thread_proxy+0x80)[0x806640]
            /lib64/libpthread.so.0[0x3c42806367]
            /lib64/libc.so.6(clone+0x6d)[0x3c41cd30ad]
            ======= Memory map: ========
            00400000-008fe000 r-xp 00000000 08:03 39583752 /home/kx_monitor/monitor2/nosql/mongodb-linux-x86_64-2.0.3-rc0/bin/mongos
            00afe000-00b15000 rw-p 004fe000 08:03 39583752 /home/kx_monitor/monitor2/nosql/mongodb-linux-x86_64-2.0.3-rc0/bin/mongos
            00b15000-00b19000 rw-p 00b15000 00:00 0
            0ef6c000-10926000 rw-p 0ef6c000 00:00 0 [heap]
            40dc8000-40dc9000 ---p 40dc8000 00:00 0
            40dc9000-40ec9000 rw-p 40dc9000 00:00 0
            411f6000-411f7000 ---p 411f6000 00:00 0
            411f7000-412f7000 rw-p 411f7000 00:00 0
            412f7000-412f8000 ---p 412f7000 00:00 0
            412f8000-413f8000 rw-p 412f8000 00:00 0
            41e37000-41e38000 ---p 41e37000 00:00 0
            41e38000-42038000 rw-p 41e38000 00:00 0
            42038000-42039000 ---p 42038000 00:00 0
            42039000-42239000 rw-p 42039000 00:00 0
            42239000-4223a000 ---p 42239000 00:00 0
            4223a000-4243a000 rw-p 4223a000 00:00 0
            4243a000-4243b000 ---p 4243a000 00:00 0
            4243b000-4263b000 rw-p 4243b000 00:00 0
            4263b000-4263c000 ---p 4263b000 00:00 0
            4263c000-4283c000 rw-p 4263c000 00:00 0
            4283c000-4283d000 ---p 4283c000 00:00 0
            4283d000-42a3d000 rw-p 4283d000 00:00 0
            42a3d000-42a3e000 ---p 42a3d000 00:00 0
            42a3e000-42c3e000 rw-p 42a3e000 00:00 0
            42d3f000-42d40000 ---p 42d3f000 00:00 0
            42d40000-42e40000 rw-p 42d40000 00:00 0
            42f40000-42f41000 ---p 42f40000 00:00 0
            42f41000-43141000 rw-p 42f41000 00:00 0
            43141000-43142000 ---p 43141000 00:00 0
            43142000-43342000 rw-p 43142000 00:00 0
            43947000-43948000 ---p 43947000 00:00 0
            43948000-43a48000 rw-p 43948000 00:00 0
            43c4a000-43c4b000 ---p 43c4a000 00:00 0
            43c4b000-43d4b000 rw-p 43c4b000 00:00 0
            43f4d000-43f4e000 ---p 43f4d000 00:00 0
            43f4e000-4404e000 rw-p 43f4e000 00:00 0
            44250000-44251000 ---p 44250000 00:00 0
            44251000-44351000 rw-p 44251000 00:00 0
            4454d000-4454e000 ---p 4454d000 00:00 0
            4454e000-4474e000 rw-p 4454e000 00:00 0
            44b50000-44b51000 ---p 44b50000 00:00 0
            44b51000-44d51000 rw-p 44b51000 00:00 0
            44d51000-44d52000 ---p 44d51000 00:00 0
            44d52000-44e52000 rw-p 44d52000 00:00 0
            44e52000-44e53000 ---p 44e52000 00:00 0
            44e53000-44f53000 rw-p 44e53000 00:00 0
            44f53000-44f54000 ---p 44f53000 00:00 0
            44f54000-45054000 rw-p 44f54000 00:00 0
            45457000-45458000 ---p 45457000 00:00 0
            45458000-45658000 rw-p 45458000 00:00 0
            46762000-46763000 ---p 46762000 00:00 0
            46763000-46863000 rw-p 46763000 00:00 0
            46863000-46864000 ---p 46863000 00:00 0
            46864000-46964000 rw-p 46864000 00:00 0
            46e67000-46e68000 ---p 46e67000 00:00 0
            46e68000-47068000 rw-p 46e68000 00:00 0
            48b72000-48b73000 ---p 48b72000 00:00 0
            48b73000-48d73000 rw-p 48b73000 00:00 0
            49175000-49176000 ---p 49175000 00:00 0
            49176000-49376000 rw-p 49176000 00:00 0
            49376000-49377000 ---p 49376000 00:00 0
            49377000-49577000 rw-p 49377000 00:00 0
            49778000-49779000 ---p 49778000 00:00 0
            49779000-49979000 rw-p 49779000 00:00 0
            4a17d000-4a17e000 ---p 4a17d000 00:00 0
            4a17e000-4a37e000 rw-p 4a17e000 00:00 0
            4a37e000-4a37f000 ---p 4a37e000 00:00 0
            4a37f000-4a57f000 rw-p 4a37f000 00:00 0
            4a981000-4a982000 ---p 4a981000 00:00 0
            4a982000-4ab82000 rw-p 4a982000 00:00 0
            4c790000-4c791000 ---p 4c790000 00:00 0
            4c791000-4c991000 rw-p 4c791000 00:00 0
            518b9000-518ba000 ---p 518b9000 00:00 0
            518ba000-51aba000 rw-p 518ba000 00:00 0
            51aba000-51abb000 ---p 51aba000 00:00 0
            51abb000-51cbb000 rw-p 51abb000 00:00 0
            524bf000-524c0000 ---p 524bf000 00:00 0
            524c0000-526c0000 rw-p 524c0000 00:00 0
            3c41800000-3c4181c000 r-xp 00000000 08:03 21233969 /lib64/ld-2.5.so
            3c41a1b000-3c41a1c000 r--p 0001b000 08:03 21233969 /lib64/ld-2.5.so
            3c41a1c000-3c41a1d000 rw-p 0001c000 08:03 21233969 /lib64/ld-2.5.so
            3c41c00000-3c41d4c000 r-xp 00000000 08:03 21233970 /lib64/libc-2.5.so
            3c41d4c000-3c41f4c000 ---p 0014c000 08:03 21233970 /lib64/libc-2.5.so
            3c41f4c000-3c41f50000 r--p 0014c000 08:03 21233970 /lib64/libc-2.5.so
            3c41f50000-3c41f51000 rw-p 00150000 08:03 21233970 /lib64/libc-2.5.so
            3c41f51000-3c41f56000 rw-p 3c41f51000 00:00 0
            3c42400000-3c42482000 r-xp 00000000 08:03 21233985 /lib64/libm-2.5.so
            3c42482000-3c42681000 ---p 00082000 08:03 21233985 /lib64/libm-2.5.so
            3c42681000-3c42682000 r--p 00081000 08:03 21233985 /lib64/libm-2.5.so
            3c42682000-3c42683000 rw-p 00082000 08:03 21233985 /lib64/libm-2.5.so
            3c42800000-3c42816000 r-xp 00000000 08:03 21233980 /lib64/libpthread-2.5.so
            3c42816000-3c42a15000 ---p 00016000 08:03 21233980 /lib64/libpthread-2.5.so
            3c42a15000-3c42a16000 r--p 00015000 08:03 21233980 /lib64/libpthread-2.5.so
            3c42a16000-3c42a17000 rw-p 00016000 08:03 21233980 /lib64/libpthread-2.5.so
            3c42a17000-3c42a1b000 rw-p 3c42a17000 00:00 0
            3c51600000-3c5160d000 r-xp 00000000 08:03 21233986 /lib64/libgcc_s-4.1.2-20080825.so.1
            3c5160d000-3c5180d000 ---p 0000d000 08:03 21233986 /lib64/libgcc_s-4.1.2-20080825.so.1
            3c5180d000-3c5180e000 rw-p 0000d000 08:03 21233986 /lib64/libgcc_s-4.1.2-20080825.so.1
            3c54600000-3c546e6000 r-xp 00000000 08:03 24197013 /usr/lib64/libstdc++.so.6.0.8
            3c546e6000-3c548e5000 ---p 000e6000 08:03 24197013 /usr/lib64/libstdc++.so.6.0.8
            3c548e5000-3c548eb000 r--p 000e5000 08:03 24197013 /usr/lib64/libstdc++.so.6.0.8
            3c548eb000-3c548ee000 rw-p 000eb000 08:03 24197013 /usr/lib64/libstdc++.so.6.0.8
            3c548ee000-3c54900000 rw-p 3c548ee000 00:00 0
            2aaaac000000-2aaaad2b5000 rw-p 2aaaac000000 00:00 0
            2aaaad2b5000-2aaab0000000 ---p 2aaaad2b5000 00:00 0
            2aaab0000000-2aaab19ec000 rw-p 2aaab0000000 00:00 0
            2aaab19ec000-2aaab4000000 ---p 2aaab19ec000 00:00 0
            2aaab4000000-2aaab5dc6000 rw-p 2aaab4000000 00:00 0
            2aaab5dc6000-2aaab8000000 ---p 2aaab5dc6000 00:00 0
            2aaab8000000-2aaab9452000 rw-p 2aaab8000000 00:00 0
            2aaab9452000-2aaabc000000 ---p 2aaab9452000 00:00 0
            2aaabc000000-2aaabd2e7000 rw-p 2aaabc000000 00:00 0
            2aaabd2e7000-2aaac0000000 ---p 2aaabd2e7000 00:00 0
            2aaac0000000-2aaac164d000 rw-p 2aaac0000000 00:00 0
            2aaac164d000-2aaac4000000 ---p 2aaac164d000 00:00 0
            2b414e8d1000-2b414e8d3000 rw-p 2b414e8d1000 00:00 0
            2b414e8e9000-2b414e8ed000 rw-p 2b414e8e9000 00:00 0
            7fff5c1c3000-7fff5c1d8000 rw-p 7ffffffea000 00:00 0 [stack]
            ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
            Received signal 6
            Backtrace: 0x54e415 0x3c41c30280 0x3c41c30215 0x3c41c31cc0 0x3c41c6a7fb 0x3c41c71ce2 0x3c41c7590c 0x525c69 0x527c24 0x806640 0x3c42806367 0x3c41cd30ad
            ./mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e415]
            /lib64/libc.so.6[0x3c41c30280]
            /lib64/libc.so.6(gsignal+0x35)[0x3c41c30215]
            /lib64/libc.so.6(abort+0x110)[0x3c41c31cc0]
            /lib64/libc.so.6[0x3c41c6a7fb]
            /lib64/libc.so.6[0x3c41c71ce2]
            /lib64/libc.so.6(cfree+0x8c)[0x3c41c7590c]
            ./mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x1b9)[0x525c69]
            ./mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x527c24]
            ./mongos(thread_proxy+0x80)[0x806640]
            /lib64/libpthread.so.0[0x3c42806367]
            /lib64/libc.so.6(clone+0x6d)[0x3c41cd30ad]
            ===
            Logstream::get called in uninitialized state
            Thu Apr 26 17:58:17 [LockPinger] warning: removing distributed lock ping thread '202.173.10.35:25000/CST-BJ-10:30000:1329301940:1804289383'
            Logstream::get called in uninitialized state
            Logstream::get called in uninitialized state
            Logstream::get called in uninitialized state
            Logstream::get called in uninitialized state
            Logstream::get called in uninitialized state
            Thu Apr 26 18:01:47 [CheckConfigServers] dbexit: received signal 15 rc:0 received signal 15
            Logstream::get called in uninitialized state
            Logstream::get called in uninitialized state
            Thu Apr 26 18:06:29 [Balancer] dbexit: received signal 15 rc:0 received signal 15
            Logstream::get called in uninitialized state
            Thu Apr 26 18:22:18 [cursorTimeout] dbexit: received signal 15 rc:0 received signal 15
            Received signal 11
            Backtrace: 0x54e415 0x3c41c30280 0x3c41c7b82b 0x3c5469c200 0x3c5469c8ff 0x7c5c8d 0x3c41c30280 0x3c41cdee6e 0x3c41c76944 0x3c41c75901 0x3c5469db6a 0x57d8cf 0x55f102 0x565aa0 0x563560 0x3c41c33355 0x7c2cf9 0x7c5c27 0x3c41c30280 0x3c41ccc5f2
            ./mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e415]
            /lib64/libc.so.6[0x3c41c30280]
            /lib64/libc.so.6(memcpy+0x3eb)[0x3c41c7b82b]
            /usr/lib64/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x50)[0x3c5469c200]
            /usr/lib64/libstdc++.so.6(_ZNSsC1ERKSs+0x4f)[0x3c5469c8ff]
            ./mongos(_ZN5mongo10sighandlerEi+0x14d)[0x7c5c8d]
            /lib64/libc.so.6[0x3c41c30280]
            /lib64/libc.so.6[0x3c41cdee6e]
            /lib64/libc.so.6[0x3c41c76944]
            /lib64/libc.so.6(cfree+0x81)[0x3c41c75901]
            /usr/lib64/libstdc++.so.6(_ZNSsD1Ev+0x3a)[0x3c5469db6a]
            ./mongos(_ZN5mongo18DBClientConnectionD0Ev+0x9f)[0x57d8cf]
            ./mongos(_ZN5mongo11PoolForHostD1Ev+0x22)[0x55f102]
            ./mongos(_ZNSt8_Rb_treeIN5mongo16DBConnectionPool7PoolKeyESt4pairIKS2_NS0_11PoolForHostEESt10_Select1stIS6_ENS1_14poolKeyCompareESaIS6_EE8_M_eraseEPSt13_Rb_tree_nodeIS6_E+0x490)[0x565aa0]
            ./mongos(_ZN5mongo16DBConnectionPoolD1Ev+0x50)[0x563560]
            /lib64/libc.so.6(exit+0xe5)[0x3c41c33355]
            ./mongos[0x7c2cf9]
            ./mongos(_ZN5mongo10sighandlerEi+0xe7)[0x7c5c27]
            /lib64/libc.so.6[0x3c41c30280]
            /lib64/libc.so.6(__select+0x62)[0x3c41ccc5f2]
            ===
            Logstream::get called in uninitialized state
            Thu Apr 26 19:04:05 [PeriodicTask::Runner] dbexit: received signal 15 rc:0 received signal 15

      Attachments

        Activity

          People

            Unassigned Unassigned
            zender zender
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: