Wed May 23 11:15:52 [Balancer] distributed lock 'balancer/php1.lan:27017:1337774228:1804289383' acquired, ts : 4fbcff2803c18271e4659717 Wed May 23 11:15:52 [Balancer] distributed lock 'balancer/php1.lan:27017:1337774228:1804289383' unlocked. Wed May 23 11:15:53 [mongosMain] connection accepted from 127.0.0.1:40663 #547 Wed May 23 11:15:55 [conn458] Socket say send() errno:32 Broken pipe 172.16.49.111:27017 Wed May 23 11:15:55 [conn458] DBException in process: could not initialize cursor across all shards because : socket exception @ Duego2/mongo2:27027,mongo3:27027 Wed May 23 11:16:00 [mongosMain] connection accepted from 172.16.49.112:24828 #548 Wed May 23 11:16:00 [conn548] end connection 172.16.49.112:24828 Wed May 23 11:16:03 [Balancer] distributed lock 'balancer/php1.lan:27017:1337774228:1804289383' acquired, ts : 4fbcff3203c18271e4659718 Wed May 23 11:16:03 [Balancer] distributed lock 'balancer/php1.lan:27017:1337774228:1804289383' unlocked. Wed May 23 11:16:05 [mongosMain] dbexit: received signal 15 rc:0 received signal 15 mongos: /opt/extra/include/boost/thread/pthread/pthread_mutex_scoped_lock.hpp:25: boost::pthread::pthread_mutex_scoped_lock::pthread_mutex_scoped_lock(pthread_mutex_t*): Assertion `!pthread_mutex_lock(m)' failed. Received signal 6 Backtrace: 0x54e475 CursorCache at0x7f26fef9c1b5 0x7f26fef9efc0 0x7f26fef95301 0x5ea004 0x7f26ffa7e8ca 0x7f26ff03992d /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e475] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] /lib/libc.so.6(gsignal+0x35)[0x7f26fef9c1b5] /lib/libc.so.6(abort+0x180)[0x7f26fef9efc0] /lib/libc.so.6(__assert_fail+0xf1)[0x7f26fef95301] /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0xa44)[0x5ea004] /lib/libpthread.so.0(+0x68ca)[0x7f26ffa7e8ca] /lib/libc.so.6(clone+0x6d)[0x7f26ff03992d] === Received signal 11 *** glibc detected *** /usr/bin/mongos: free(): corrupted unsorted chunks: 0x0000000000b74560 *** 0x54e475 0x7f26fef9c230 0x7f26fefa055f 0x7f26fefa0665 0x54e4db 0x7f26fef9c230 0x7f26fef9c1b5 0x7f26fef9efc0 0x7f26fef95301 0x5ea004 0x7f26ffa7e8ca 0x7f26ff03992d /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e475] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] /lib/libc.so.6(+0x3655f)[0x7f26fefa055f] /lib/libc.so.6(+0x36665)[0x7f26fefa0665] /usr/bin/mongos[0x54e4db] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] /lib/libc.so.6(gsignal+0x35)[0x7f26fef9c1b5] /lib/libc.so.6(abort+0x180)[0x7f26fef9efc0] /lib/libc.so.6(__assert_fail+0xf1)[0x7f26fef95301] /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0xa44)[0x5ea004] /lib/libpthread.so.0(+0x68ca)[0x7f26ffa7e8ca] /lib/libc.so.6(clone+0x6d)[0x7f26ff03992d] === Received signal 11 Backtrace: 0x54e475 0x7f26fef9c230 0xb79710 /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e475] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] [0xb79710] === Received signal 11 Backtrace: 0x54e475 0x7f26fef9c230 0x56742f 0x591704 0x594662 0x56825c 0x57e031 0x56b9a3 0x56c4f2 0x57e1aa 0x56a2ec 0x586112 0x587de7 0x588130 0x590191 0x52501f 0x527394 0x806720 0x7f26ffa7e8ca 0x7f26ff03992d /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e475] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] /usr/bin/mongos(_ZN5mongo15assembleRequestERKSsNS_7BSONObjEiiPKS2_iRNS_7MessageE+0xef)[0x56742f] /usr/bin/mongos(_ZN5mongo14DBClientCursor13_assembleInitERNS_7MessageE+0x84)[0x591704] /usr/bin/mongos(_ZN5mongo14DBClientCursor4initEv+0x92)[0x594662] /usr/bin/mongos(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x3ac)[0x56825c] /usr/bin/mongos(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1)[0x57e031] /usr/bin/mongos(_ZN5mongo17DBClientInterface5findNERSt6vectorINS_7BSONObjESaIS2_EERKSsNS_5QueryEiiPKS2_i+0xa3)[0x56b9a3] /usr/bin/mongos(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x72)[0x56c4f2] /usr/bin/mongos(_ZN5mongo18DBClientConnection10runCommandERKSsRKNS_7BSONObjERS3_i+0x7a)[0x57e1aa] /usr/bin/mongos(_ZN5mongo20DBClientWithCommands8isMasterERbPNS_7BSONObjE+0x6c)[0x56a2ec] /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor16_checkConnectionEPNS_18DBClientConnectionERSsbi+0x102)[0x586112] /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor5checkEb+0x97)[0x587de7] /usr/bin/mongos(_ZN5mongo17ReplicaSetMonitor8checkAllEb+0x250)[0x588130] /usr/bin/mongos(_ZN5mongo24ReplicaSetMonitorWatcher3runEv+0x41)[0x590191] /usr/bin/mongos(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xbf)[0x52501f] /usr/bin/mongos(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74)[0x527394] /usr/bin/mongos(thread_proxy+0x80)[0x806720] /lib/libpthread.so.0(+0x68ca)[0x7f26ffa7e8ca] /lib/libc.so.6(clone+0x6d)[0x7f26ff03992d] === Received signal 11 Backtrace: 0x54e475 0x7f26fef9c230 0x7f26fefdbeca 0x7f26fefde354 0x7f26fefe0a30 0x558e72 0x5e982d 0x7f26ffa7e8ca 0x7f26ff03992d /usr/bin/mongos(_ZN5mongo17printStackAndExitEi+0x75)[0x54e475] /lib/libc.so.6(+0x32230)[0x7f26fef9c230] /lib/libc.so.6(+0x71eca)[0x7f26fefdbeca] /lib/libc.so.6(+0x74354)[0x7f26fefde354] /lib/libc.so.6(__libc_malloc+0x70)[0x7f26fefe0a30] /usr/bin/mongos(_ZN5mongo13MessagingPort4recvERNS_7MessageE+0xa2)[0x558e72] /usr/bin/mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x26d)[0x5e982d] /lib/libpthread.so.0(+0x68ca)[0x7f26ffa7e8ca] /lib/libc.so.6(clone+0x6d)[0x7f26ff03992d] === Logstream::get called in uninitialized state Wed May 23 11:16:26 [LockPinger] warning: removing distributed lock ping thread 'config1.vox:27019,config2.vox:27019,config3.vox:27019/php1.lan:27017:1337774228:1804289383' Logstream::get called in uninitialized state Logstream::get called in uninitialized state Logstream::get called in uninitialized state Logstream::get called in uninitialized state