There was some memory corruption in mongod config (Libc memory corruption, as clear from the logs) and this make it go in a wait state for a lock within libc, which never get completed due to memory corruption. I think that there should be some hang detection mechanism for these services also.
Stack Trace is:
[root@ct-node-ft-112 logs]# gdb attach 4225
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
attach: No such file or directory.
Attaching to process 4225
Reading symbols from /usr/local/ct/depend/mongo/bin/mongod...(no debugging symbols found)...done.
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x42685940 (LWP 4291)]
[New Thread 0x41e84940 (LWP 4276)]
[New Thread 0x41683940 (LWP 4273)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00000036cfcdfe8e in __lll_lock_wait_private () from /lib64/libc.so.6
(gdb) bt
#0 0x00000036cfcdfe8e in __lll_lock_wait_private () from /lib64/libc.so.6
#1 0x00000036cfc770bd in _L_lock_5478 () from /lib64/libc.so.6
#2 0x00000036cfc71de3 in _int_free () from /lib64/libc.so.6
#3 0x00000036cfc728bb in free () from /lib64/libc.so.6
#4 0x00000036cfc33493 in exit () from /lib64/libc.so.6
#5 0x000000000088a418 in mongo:bexit(mongo::ExitCode, char const*, bool) ()
#6 0x0000000000a95553 in mongo::initAndListen(int) ()
#7 0x0000000000a9f17a in main ()
(gdb) detach
Detaching from program: /usr/local/ct/depend/mongo/bin/mongod, process 4225
(gdb) quit
Logs are :
Sat Mar 24 10:25:23 [initandlisten] build info: Linux ip-10-110-9-236 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_41
Sat Mar 24 10:25:23 [initandlisten] options:
Sat Mar 24 10:25:23 [initandlisten] User Assertion: 10296bpath (/u01/configdb1) does not exist
Sat Mar 24 10:25:23 [initandlisten] exception in initAndListen: 10296 dbpath (/u01/configdb1) does not exist, terminating
Sat Mar 24 10:25:23 dbexit:
Sat Mar 24 10:25:23 [initandlisten] shutdown: going to close listening sockets...
Sat Mar 24 10:25:23 [initandlisten] shutdown: going to flush diaglog...
Sat Mar 24 10:25:23 [initandlisten] shutdown: going to close sockets...
Sat Mar 24 10:25:23 [initandlisten] shutdown: waiting for fs preallocator...
Sat Mar 24 10:25:23 [initandlisten] shutdown: lock for final commit...
Sat Mar 24 10:25:23 [initandlisten] shutdown: final commit...
Sat Mar 24 10:25:23 [initandlisten] shutdown: closing all files...
Sat Mar 24 10:25:23 [initandlisten] closeAllFiles() finished
Sat Mar 24 10:25:23 dbexit: really exiting now
Sat Mar 24 10:25:23 BackgroundJob starting: DataFileSync
Sat Mar 24 10:25:23 Invalid access at address: 0
Sat Mar 24 10:25:23 Got signal: 11 (Segmentation fault).
Sat Mar 24 10:25:23 Backtrace:
0xa90c39 0xa91210 0x36d080eb10
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90c39]
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa91210]
/lib64/libpthread.so.0 [0x36d080eb10]
-
-
- glibc detected *** /usr/local/ct/depend/mongo/bin/mongod: corrupted double-linked list: 0x0000000002da6450 ***
======= Backtrace: =========
/lib64/libc.so.6[0x36cfc724d8]
/lib64/libc.so.6(cfree+0x4b)[0x36cfc728bb]
/lib64/libc.so.6(exit+0xe5)[0x36cfc334f5]
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo10abruptQuitEi+0x5ee)[0xa90e8e]
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220)[0xa91210]
/lib64/libpthread.so.0[0x36d080eb10]
======= Memory map: ========
00400000-00c25000 r-xp 00000000 08:02 10750214 /usr/local/ct/depend/mongo/bin/mongod
00e25000-00e44000 rw-p 00825000 08:02 10750214 /usr/local/ct/depend/mongo/bin/mongod
00e44000-0184c000 rw-p 00000000 00:00 0
02d63000-02dc7000 rw-p 00000000 00:00 0 [heap]
40e83000-40e84000 ---p 00000000 00:00 0
40e84000-41684000 rw-p 00000000 00:00 0
41684000-41685000 ---p 00000000 00:00 0
41685000-41e85000 rw-p 00000000 00:00 0
41e85000-41e86000 ---p 00000000 00:00 0
41e86000-42686000 rw-p 00000000 00:00 0
36cf800000-36cf81c000 r-xp 00000000 08:02 7438662 /lib64/ld-2.5.so
36cfa1b000-36cfa1c000 r--p 0001b000 08:02 7438662 /lib64/ld-2.5.so
36cfa1c000-36cfa1d000 rw-p 0001c000 08:02 7438662 /lib64/ld-2.5.so
36cfc00000-36cfd4e000 r-xp 00000000 08:02 7438663 /lib64/libc-2.5.so
36cfd4e000-36cff4e000 ---p 0014e000 08:02 7438663 /lib64/libc-2.5.so
36cff4e000-36cff52000 r--p 0014e000 08:02 7438663 /lib64/libc-2.5.so
36cff52000-36cff53000 rw-p 00152000 08:02 7438663 /lib64/libc-2.5.so
36cff53000-36cff58000 rw-p 00000000 00:00 0
36d0000000-36d0082000 r-xp 00000000 08:02 7438664 /lib64/libm-2.5.so
36d0082000-36d0281000 ---p 00082000 08:02 7438664 /lib64/libm-2.5.so
36d0281000-36d0282000 r--p 00081000 08:02 7438664 /lib64/libm-2.5.so
36d0282000-36d0283000 rw-p 00082000 08:02 7438664 /lib64/libm-2.5.so
36d0800000-36d0816000 r-xp 00000000 08:02 7438670 /lib64/libpthread-2.5.so
36d0816000-36d0a15000 ---p 00016000 08:02 7438670 /lib64/libpthread-2.5.so
36d0a15000-36d0a16000 r--p 00015000 08:02 7438670 /lib64/libpthread-2.5.so
36d0a16000-36d0a17000 rw-p 00016000 08:02 7438670 /lib64/libpthread-2.5.so
36d0a17000-36d0a1b000 rw-p 00000000 00:00 0
36e1e00000-36e1e0d000 r-xp 00000000 08:02 7438679 /lib64/libgcc_s-4.1.2-20080825.so.1
36e1e0d000-36e200d000 ---p 0000d000 08:02 7438679 /lib64/libgcc_s-4.1.2-20080825.so.1
36e200d000-36e200e000 rw-p 0000d000 08:02 7438679 /lib64/libgcc_s-4.1.2-20080825.so.1
7effc9615000-7effc9616000 rw-p 00000000 00:00 0
7effc9a19000-7effc9a1c000 rw-p 00000000 00:00 0
7effc9a1c000-7effc9b12000 r-xp 00000000 08:02 9934339 /usr/lib64/libstdc++.so.6.0.13
7effc9b12000-7effc9d12000 ---p 000f6000 08:02 9934339 /usr/lib64/libstdc++.so.6.0.13
7effc9d12000-7effc9d19000 r--p 000f6000 08:02 9934339 /usr/lib64/libstdc++.so.6.0.13
7effc9d19000-7effc9d1b000 rw-p 000fd000 08:02 9934339 /usr/lib64/libstdc++.so.6.0.13
7effc9d1b000-7effc9d32000 rw-p 00000000 00:00 0
7fff263ea000-7fff263ff000 rw-p 00000000 00:00 0 [stack]
7fff263ff000-7fff26400000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
Sat Mar 24 10:25:23 Got signal: 6 (Aborted).
- glibc detected *** /usr/local/ct/depend/mongo/bin/mongod: corrupted double-linked list: 0x0000000002da6450 ***
-
Sat Mar 24 10:25:23 Backtrace:
0xa90c39 0x36cfc302d0 0x36cfc30265 0x36cfc31d10 0x36cfc6a99b 0x36cfc724d8 0x36cfc728bb 0x36cfc334f5 0xa90e8e 0xa91210 0x36d080eb10
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90c39]
/lib64/libc.so.6 [0x36cfc302d0]
/lib64/libc.so.6(gsignal+0x35) [0x36cfc30265]
/lib64/libc.so.6(abort+0x110) [0x36cfc31d10]
/lib64/libc.so.6 [0x36cfc6a99b]
/lib64/libc.so.6 [0x36cfc724d8]
/lib64/libc.so.6(cfree+0x4b) [0x36cfc728bb]
/lib64/libc.so.6(exit+0xe5) [0x36cfc334f5]
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo10abruptQuitEi+0x5ee) [0xa90e8e]
/usr/local/ct/depend/mongo/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa91210]
/lib64/libpthread.so.0 [0x36d080eb10]
- related to
-
SERVER-5681 unable to start mongod give exception in initAndListen: 10079 bad .ns file length, cannot open database, terminating
- Closed