On a sharded cluster that has been working fine for a few weeks mongos has suddenly started crashing with intervalls of only a few minutes without any obvious reason . No changes has been made in the application since yesterday when the cluster was working fine.

Below is the log output from mongos when the process dies.

Thu Sep 1 08:49:57 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 0
Thu Sep 1 08:49:57 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 1
Thu Sep 1 08:49:58 [conn13] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 2
Thu Sep 1 08:49:58 [conn17] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 2
Thu Sep 1 08:49:58 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 2
Thu Sep 1 08:50:00 [conn13] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 3
Thu Sep 1 08:50:00 [conn17] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 3
Thu Sep 1 08:50:00 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 3
Thu Sep 1 08:50:03 [conn13] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 4
Thu Sep 1 08:50:03 [conn17] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 4
Thu Sep 1 08:50:03 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 4
Thu Sep 1 08:50:07 [conn13] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 5
Thu Sep 1 08:50:07 [conn17] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 5
Thu Sep 1 08:50:08 [conn15] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 5
Thu Sep 1 08:50:08 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 0
Thu Sep 1 08:50:08 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 1
Thu Sep 1 08:50:09 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 2
Thu Sep 1 08:50:11 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 3
Thu Sep 1 08:50:14 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 4
Thu Sep 1 08:50:18 [conn19] ns: fragments_20110825.exposure_fragments Strategy::doQuery attempt: 5
Thu Sep 1 08:50:41 [mongosMain] connection accepted from 127.0.0.1:59472 #22
Thu Sep 1 08:50:47 [conn22] end connection 127.0.0.1:59472
Thu Sep 1 08:51:37 [LockPinger] dist_lock pinged successfully for: richassembler04.byburt.com:1314865925:1804289383
Received signal 6
Backtrace: 0x52f8f5 0x7f7481c88af0 0x7f7481c88a75 0x7f7481c8c5c0 0x7f7481c81941 0x69d485 0x50454b 0x505e04 0x6a50a0 0x7f748278c9ca 0x7f7481d3b70d
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52f8f5]
/lib/libc.so.6(+0x33af0)[0x7f7481c88af0]
/lib/libc.so.6(gsignal+0x35)[0x7f7481c88a75]
/lib/libc.so.6(abort+0x180)[0x7f7481c8c5c0]
/lib/libc.so.6(__assert_fail+0xf1)[0x7f7481c81941]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo17WriteBackListener3runEv+0x1c15)[0x69d485]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b)[0x50454b]
/opt/mongodb-1.8.2/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x505e04]
/opt/mongodb-1.8.2/bin/mongos(thread_proxy+0x80)[0x6a50a0]
/lib/libpthread.so.0(+0x69ca)[0x7f748278c9ca]
/lib/libc.so.6(clone+0x6d)[0x7f7481d3b70d]
===
Received signal 11
Backtrace: 0x52f8f5 0x7f7481c88af0 0x532ea0 0x577654 0x577c71 0x6305fe 0x63c769 0x63d3e0 0x6682f2 0x67d187 0x580b7c 0x6a50a0 0x7f748278c9ca 0x7f7481d3b70d
Received signal 11
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52f8f5]
Backtrace: /lib/libc.so.6(+0x33af0)[0x7f7481c88af0]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x532ea0]
0x52f8f5 0x7f7481c88af0 0x532ea0 0x577654 0x577c71 0x6305fe 0x63c769 0x63d3e0 0x6682f2 0x67d187 0x580b7c 0x6a50a0 0x7f748278c9ca 0x7f7481d3b70d
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x577654]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x577c71]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x52f8f5]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo8Strategy6insertERKNS_5ShardEPKcRKNS_7BSONObjE+0x5e)[0x6305fe]
/lib/libc.so.6(+0x33af0)[0x7f7481c88af0]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo16DBConnectionPool11onHandedOutEPNS_12DBClientBaseE+0x20)[0x532ea0]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo15ShardConnection5_initEv+0x1b4)[0x577654]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo15ShardConnectionC1ERKNS_5ShardERKSs+0xa1)[0x577c71]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo8Strategy6insertERKNS_5ShardEPKcRKNS_7BSONObjE+0x5e)[0x6305fe]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo13ShardStrategy7_insertERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrINS_12ChunkManagerEEE+0x5c9)[0x63c769]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x260)[0x63d3e0]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo7Request7processEi+0x172)[0x6682f2]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x67d187]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x580b7c]
/opt/mongodb-1.8.2/bin/mongos(thread_proxy+0x80)[0x6a50a0]
/lib/libpthread.so.0(+0x69ca)[0x7f748278c9ca]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo13ShardStrategy7_insertERNS_7RequestERNS_9DbMessageEN5boost10shared_ptrINS_12ChunkManagerEEE+0x5c9)[0x63c769]
/lib/libc.so.6(clone+0x6d)[0x7f7481d3b70d]
===
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo13ShardStrategy7writeOpEiRNS_7RequestE+0x260)[0x63d3e0]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo7Request7processEi+0x172)[0x6682f2]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x77)[0x67d187]
/opt/mongodb-1.8.2/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x34c)[0x580b7c]
/opt/mongodb-1.8.2/bin/mongos(thread_proxy+0x80)[0x6a50a0]
/lib/libpthread.so.0(+0x69ca)[0x7f748278c9ca]
/lib/libc.so.6(clone+0x6d)[0x7f7481d3b70d]