mongod (primary) crashed with "Uncaught std::exception: St9bad_alloc, terminating". Looked like a clean shutdown although the lock file was not removed. Upon restarting the mongod it complained of unclean shutdown.

Sat Feb 5 07:34:58 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b4150ba0457700056b'), accId: 2367, sId: 1 }

281ms
Sat Feb 5 07:34:58 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b4150ba08475000aad'), accId: 2424, sId: 1 }

228ms
Sat Feb 5 07:34:58 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b5150ba08b78000315'), accId: 3096, sId: 2 }

119ms
Sat Feb 5 07:34:59 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b5160ba087240004dc'), accId: 3199, sId: 8 }

137ms
Sat Feb 5 07:35:07 [conn54697] query sd.metrics ntoreturn:1000 reslen:153521 nscanned:3100738 { $query: { $nor: [ { accId: 2461, tA:

{ $gt: new Date(1294299300226) }

}, { accId: 2805, tA:

{ $gt: new Date(1296286500226) }

}, { tA:

{ $gt: new Date(1296632100228) }

} ] }, $orderby:

{ _id: 1 }

} nreturned:1000 6786ms
Sat Feb 5 07:35:08 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b8140ba02e680015ed'), accId: 3282, sId: 2 }

150ms
Sat Feb 5 07:35:09 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b8160ba051240005a6'), accId: 3441, sId: 4 }

155ms
Sat Feb 5 07:35:18 [conn38727] query sd.$cmd ntoreturn:1 command: { count: "sessions", query: {} } reslen:64 155ms
Sat Feb 5 07:35:18 [conn27442] query sd.$cmd ntoreturn:1 command: { count: "sessions", query: {} } reslen:64 191ms
Sat Feb 5 07:35:18 [conn54337] query sd.metrics ntoreturn:864 reslen:39938 nscanned:368 { $query: { accId: 3564, metric: "diskUsage", sId: 8, tA:

{ $gte: new Date(1296802215000), $lte: new Date(1296946799000) }

, tAG:

{ $in: [ 5, 60 ] }

}, $orderby:

{ tA: 1 }

} nreturned:281 2009ms
Sat Feb 5 07:35:20 [conn54697] query sd.metrics ntoreturn:1000 reslen:151508 nscanned:3100678 { $query: { $nor: [ { accId: 2461, tA:

{ $gt: new Date(1294299310928) }

}, { accId: 2805, tA:

{ $gt: new Date(1296286510929) }

}, { tA:

{ $gt: new Date(1296632110930) }

} ] }, $orderby:

{ _id: 1 }

} nreturned:1000 9124ms
Sat Feb 5 07:35:20 [conn54541] insert sd.metrics 113ms
Sat Feb 5 07:35:21 [conn54697] remove sd.metrics query:

{ _id: ObjectId('4d4455b9150ba04f77000589'), accId: 3963, sId: 2 }

353ms
Sat Feb 5 07:35:22 [conn5864] Uncaught std::exception: St9bad_alloc, terminating
Sat Feb 5 07:35:22 dbexit:
Sat Feb 5 07:35:22 [conn54749] got request after shutdown()
Sat Feb 5 07:35:22 [conn54495] got request after shutdown()
Sat Feb 5 07:35:22 [conn54768] got request after shutdown()
Sat Feb 5 07:35:22 [conn54697] got request after shutdown()
Sat Feb 5 07:35:22 [conn54313] got request after shutdown()
Sat Feb 5 07:35:22 [conn54536] got request after shutdown()
Sat Feb 5 07:35:22 [conn54478] got request after shutdown()
Sat Feb 5 07:35:22 [conn54488] got request after shutdown()
Sat Feb 5 07:35:22 [conn54489] got request after shutdown()
Sat Feb 5 07:35:22 [conn89] got request after shutdown()
Sat Feb 5 07:35:22 [conn54298] got request after shutdown()
Sat Feb 5 07:35:22 [conn54268] got request after shutdown()
Sat Feb 5 07:35:22 [conn54521] got request after shutdown()
Sat Feb 5 07:35:22 [conn5864] shutdown: going to close listening sockets...
Sat Feb 5 07:35:22 [conn5864] closing listening socket: 5
Sat Feb 5 07:35:22 [conn5864] closing listening socket: 6
Sat Feb 5 07:35:22 [conn5864] closing listening socket: 7
Sat Feb 5 07:35:22 [conn5864] closing listening socket: 8
Sat Feb 5 07:35:22 [conn5864] removing socket file: /tmp/mongodb-27018.sock
Sat Feb 5 07:35:22 [conn54319] got request after shutdown()
Sat Feb 5 07:35:22 [conn50493] got request after shutdown()
Sat Feb 5 07:35:22 [conn54338] got request after shutdown()
Sat Feb 5 07:35:22 [conn54501] got request after shutdown()
Sat Feb 5 07:35:22 [conn54765] got request after shutdown()
Sat Feb 5 07:35:22 [conn51892] got request after shutdown()
Sat Feb 5 07:35:22 [conn52318] got request after shutdown()
Sat Feb 5 07:35:22 [conn54490] got request after shutdown()
Sat Feb 5 07:35:22 [conn52304] got request after shutdown()
Sat Feb 5 07:35:22 [conn54763] got request after shutdown()
Sat Feb 5 07:35:22 [conn54753] got request after shutdown()
Sat Feb 5 07:35:22 [conn5864] removing socket file: /tmp/mongodb-28018.sock
Sat Feb 5 07:35:22 [conn5864] shutdown: going to flush diaglog...
Sat Feb 5 07:35:22 [conn5864] shutdown: going to close sockets...
Sat Feb 5 07:35:22 [conn54296] got request after shutdown()
Sat Feb 5 07:35:22 [conn52040] got request after shutdown()
Sat Feb 5 07:35:22 [conn54781] got request after shutdown()
Sat Feb 5 07:35:22 [conn53689] got request after shutdown()
Sat Feb 5 07:35:22 [conn54532] got request after shutdown()
Sat Feb 5 07:35:22 [conn53832] got request after shutdown()
Sat Feb 5 07:35:22 [conn54255] got request after shutdown()
Sat Feb 5 07:35:22 [conn54502] got request after shutdown()
Sat Feb 5 07:35:22 [conn54238] got request after shutdown()
Sat Feb 5 07:35:22 [conn54775] got request after shutdown()
Sat Feb 5 07:35:22 [conn54538] got request after shutdown()
Sat Feb 5 07:35:22 [conn54747] got request after shutdown()
Sat Feb 5 07:35:22 [conn2] got request after shutdown()
Sat Feb 5 07:35:22 [initandlisten] now exiting
Sat Feb 5 07:35:22 dbexit: ; exiting immediately
Sat Feb 5 07:35:22 [conn54778] got request after shutdown()
Sat Feb 5 07:35:22 [conn54277] got request after shutdown()
Sat Feb 5 07:35:22 [conn54540] got request after shutdown()
Sat Feb 5 07:35:22 [conn54510] got request after shutdown()
Sat Feb 5 07:35:22 [conn54317] got request after shutdown()
Sat Feb 5 07:35:22 [conn54242] got request after shutdown()
Sat Feb 5 07:35:22 [conn54230] got request after shutdown()
Sat Feb 5 07:35:22 [conn54303] got request after shutdown()
Sat Feb 5 07:35:22 [conn54771] got request after shutdown()
Sat Feb 5 07:35:22 [conn54507] got request after shutdown()
Sat Feb 5 07:35:22 [conn54757] got request after shutdown()
Sat Feb 5 07:35:22 [conn53017] got request after shutdown()
Sat Feb 5 07:35:22 [conn53741] got request after shutdown()
Sat Feb 5 07:35:22 [conn54759] got request after shutdown()
Sat Feb 5 07:35:22 [conn54267] got request after shutdown()
Sat Feb 5 07:35:22 [conn54539] got request after shutdown()
Sat Feb 5 07:35:22 [conn53209] got request after shutdown()
Sat Feb 5 07:35:22 [conn54297] got request after shutdown()
Sat Feb 5 07:35:22 [conn53896] got request after shutdown()
Sat Feb 5 07:35:22 [conn52641] got request after shutdown()
Sat Feb 5 07:35:22 [conn54492] got request after shutdown()
Sat Feb 5 07:35:22 [conn54783] got request after shutdown()
Sat Feb 5 07:35:22 [conn54527] got request after shutdown()
Sat Feb 5 07:35:22 [conn53834] got request after shutdown()
Sat Feb 5 07:35:22 [conn54746] got request after shutdown()
Sat Feb 5 07:35:22 [conn52130] got request after shutdown()
Sat Feb 5 07:35:22 [conn54548] got request after shutdown()
Sat Feb 5 07:35:22 [conn54236] got request after shutdown()
Sat Feb 5 07:35:22 [conn54274] got request after shutdown()
Sat Feb 5 07:35:22 [conn53749] got request after shutdown()
Sat Feb 5 07:35:22 [conn54292] got request after shutdown()
Sat Feb 5 07:35:22 [conn54557] got request after shutdown()
Sat Feb 5 07:35:22 [conn51187] got request after shutdown()
Sat Feb 5 07:35:22 [conn54755] got request after shutdown()
Sat Feb 5 07:35:22 [conn54786] got request after shutdown()
Sat Feb 5 07:35:22 [conn54246] got request after shutdown()
Sat Feb 5 07:35:22 [conn53657] got request after shutdown()
Sat Feb 5 07:35:22 [conn54272] got request after shutdown()
Sat Feb 5 07:35:22 [conn54517] got request after shutdown()
Sat Feb 5 07:35:22 [conn83] got request after shutdown()
Sat Feb 5 07:35:22 [conn54782] got request after shutdown()
Sat Feb 5 07:35:22 [conn53849] got request after shutdown()
Sat Feb 5 07:35:22 [conn53701] got request after shutdown()
Sat Feb 5 07:35:22 [conn54258] got request after shutdown()
Sat Feb 5 07:35:22 [conn54266] got request after shutdown()
Sat Feb 5 07:35:22 [conn54252] got request after shutdown()
Sat Feb 5 07:35:22 [conn54256] got request after shutdown()
Sat Feb 5 07:35:22 [conn5864] shutdown: waiting for fs preallocator...
Sat Feb 5 07:35:22 [conn5864] shutdown: closing all files...
Sat Feb 5 07:35:22 [conn54509] got request after shutdown()
Sat Feb 5 07:35:22 [conn53025] got request after shutdown()
Sat Feb 5 07:35:22 Got signal: 6 (Aborted).

Sat Feb 5 07:35:22 [conn54487] got request after shutdown()
Sat Feb 5 07:35:22 [conn54522] got request after shutdown()
Sat Feb 5 07:35:22 [conn54262] got request after shutdown()
Sat Feb 5 07:35:22 [conn54550] got request after shutdown()
Sat Feb 5 07:35:22 [conn53029] got request after shutdown()
Sat Feb 5 07:35:22 [conn38808] got request after shutdown()
Sat Feb 5 07:35:22 [conn54549] got request after shutdown()
Sat Feb 5 07:35:22 [conn27413] got request after shutdown()
Sat Feb 5 07:35:23 [conn54772] got request after shutdown()
Sat Feb 5 07:35:23 [conn54493] got request after shutdown()
Sat Feb 5 07:35:23 [conn50595] got request after shutdown()
Sat Feb 5 07:35:23 [conn54281] got request after shutdown()
Sat Feb 5 07:35:23 [conn54769] got request after shutdown()
Sat Feb 5 07:35:23 [conn423] got request after shutdown()
Sat Feb 5 07:35:23 [conn54744] got request after shutdown()
Sat Feb 5 07:35:23 [conn54485] got request after shutdown()
Sat Feb 5 07:35:23 [conn54551] got request after shutdown()
Sat Feb 5 07:35:23 [conn54343] got request after shutdown()
Sat Feb 5 07:35:23 [conn53721] got request after shutdown()
Sat Feb 5 07:35:23 [conn53703] got request after shutdown()
Sat Feb 5 07:35:23 [conn54756] got request after shutdown()
Sat Feb 5 07:35:23 [conn38854] got request after shutdown()
Sat Feb 5 07:35:23 [conn54544] got request after shutdown()
Sat Feb 5 07:35:23 [conn54750] got request after shutdown()
Sat Feb 5 07:35:23 [conn53823] got request after shutdown()
Sat Feb 5 07:35:23 [conn54767] got request after shutdown()
Sat Feb 5 07:35:23 [conn53623] got request after shutdown()
Sat Feb 5 07:35:23 [conn54558] got request after shutdown()
Sat Feb 5 07:35:23 [conn54751] got request after shutdown()
Sat Feb 5 07:35:23 [conn54761] got request after shutdown()
Sat Feb 5 07:35:23 [conn54300] got request after shutdown()
Sat Feb 5 07:35:23 [conn54259] got request after shutdown()
Sat Feb 5 07:35:23 [conn50608] got request after shutdown()
Sat Feb 5 07:35:23 [conn54339] got request after shutdown()
Sat Feb 5 07:35:23 [conn54335] got request after shutdown()
Sat Feb 5 07:35:23 [conn54231] got request after shutdown()
Sat Feb 5 07:35:23 [conn53098] got request after shutdown()
Sat Feb 5 07:35:23 [conn53920] got request after shutdown()
Sat Feb 5 07:35:23 [conn38768] got request after shutdown()
Sat Feb 5 07:35:23 Backtrace:
0x899789 0x3f7aa302d0 0x3f7aa30265 0x3f7aa31d10 0x3f7aa296e6 0x5491c4 0x5473c7 0x74fd22 0x7507a6 0x89ac76 0x8ad550 0x3f7b60673d 0x3f7aad3d1d
/home/david/mongodb/latest/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x899789]
/lib64/libc.so.6 [0x3f7aa302d0]
/lib64/libc.so.6(gsignal+0x35) [0x3f7aa30265]
/lib64/libc.so.6(abort+0x110) [0x3f7aa31d10]
/lib64/libc.so.6(__assert_fail+0xf6) [0x3f7aa296e6]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6rwlockC1ERKNS_6RWLockEbb+0x124) [0x5491c4]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo9MongoFile13closeAllFilesERSt18basic_stringstreamIcSt11char_traitsIcESaIcEE+0x77) [0x5473c7]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo14shutdownServerEv+0x592) [0x74fd22]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo6dbexitENS_8ExitCodeEPKcb+0x436) [0x7507a6]
/home/david/mongodb/latest/bin/mongod(_ZN5mongo10connThreadEPNS_13MessagingPortE+0xd36) [0x89ac76]
/home/david/mongodb/latest/bin/mongod(thread_proxy+0x80) [0x8ad550]
/lib64/libpthread.so.0 [0x3f7b60673d]
/lib64/libc.so.6(clone+0x6d) [0x3f7aad3d1d]

Sat Feb 5 07:35:23 [conn38693] got request after shutdown()
Sat Feb 5 07:35:24 [conn27509] got request after shutdown()