-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
Affects Version/s: 2.0.0
-
Component/s: Replication
-
Labels:None
-
ALL
See below. An unusual configuration – in addition members 1 and 2 were down at the time. Further, mongos was calling ismaster and replsetgetstatus thousands of times per second. thus it is likely a race condition?
Crash was not during a reconfig but later. Sharded environment.
PRIMARY> rs.conf()
{
"_id" : "zzz",
"version" : 3,
"members" : [
{
"_id" : 0,
"host" : "x:27162"
},
{
"_id" : 1,
"host" : "x:27192",
"votes" : 0,
"arbiterOnly" : true
},
{
"_id" : 2,
"host" : "x:27193",
"votes" : 0,
"arbiterOnly" : true
}
Mon Sep 26 20:13:36 [conn12] query mydb.Mapping_E13_MM ntoreturn:2 idhack:1 reslen:20 106ms Mon Sep 26 20:13:36 [conn75] query mydb.Mapping_E13_MM ntoreturn:2 idhack:1 reslen:112 106ms Mon Sep 26 20:13:36 [conn74] query mydb.Mapping_E13_MM ntoreturn:2 idhack:1 reslen:112 107ms Mon Sep 26 20:14:30 [clientcursormon] mem (MB) res:45634 virt:492036 mapped:245801 *** glibc detected *** /apps/mongodb/bin/mongod: malloc(): memory corruption: 0x00007f4ed8000088 *** ======= Backtrace: ========= Mon Sep 26 20:14:54 Invalid access at address: 0 /lib/libc.so.6(+0x71ad6)[0x7fc6ef293ad6] Mon Sep 26 20:14:54 Got signal: 11 (Segmentation fault). /lib/libc.so.6(+0x74b6d)[0x7fc6ef296b6d] /lib/libc.so.6(__libc_malloc+0x70)[0x7fc6ef298930] /usr/lib/libstdc++.so.6(_Znwm+0x1d)[0x7fc6efae66bd] /usr/lib/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x59)[0x7fc6efac2b29] /usr/lib/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x2b)[0x7fc6efac3aeb] /usr/lib/libstdc++.so.6(_ZNSs7reserveEm+0x3c)[0x7fc6efac405c] /usr/lib/libstdc++.so.6(_ZNSt15basic_stringbufIcSt11char_traitsIcESaIcEE8overflowEi+0xb1)[0x7fc6efabe021] /usr/lib/libstdc++.so.6(_ZNSt15basic_streambufIcSt11char_traitsIcEE6xsputnEPKcl+0x35)[0x7fc6efac2215] /usr/lib/libstdc++.so.6(_ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_l+0x1b5)[0x7fc6efab83b5] /usr/lib/libstdc++.so.6(_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+0x2f)[0x7fc6efab862f] /apps/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x15b)[0xa844db] /lib/libpthread.so.0(+0xef60)[0x7fc6efd3df60] Mon Sep 26 20:14:54 /lib/libc.so.6(memcpy+0x2f7)[0x7fc6ef2a1a47] Backtrace: 0xa83fc9 0xa845a0 0x7fc6efd3df60 0x7fc6efa8d173 0x67a375 0x985e77 0x973b49 0x97512f 0x95d725 0x9607b4 0x87e037 0x88485c 0xa96a46 0x635dd7 0x7fc6efd358ba 0x7fc6ef2f102d /apps/mongodb/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa83fc9] /apps/mongodb/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa845a0] /lib/libpthread.so.0(+0xef60) [0x7fc6efd3df60] /usr/lib/libstdc++.so.6(_ZSt18_Rb_tree_incrementPSt18_Rb_tree_node_base+0x13) [0x7fc6efa8d173] /apps/mongodb/bin/mongod(_ZN5mongo9MongoFile17totalMappedLengthEv+0xf5) [0x67a375] /apps/mongodb/bin/mongod(_ZN5mongo15CmdServerStatus3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xed7) [0x985e77] /apps/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a9) [0x973b49] /apps/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97512f] /apps/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x95d725] /apps/mongodb/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xee4) [0x9607b4] /apps/mongodb/bin/mongod() [0x87e037] /apps/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c) [0x88485c] /apps/mongodb/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xa96a46] /apps/mongodb/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x635dd7] /lib/libpthread.so.0(+0x68ba) [0x7fc6efd358ba] /lib/libc.so.6(clone+0x6d) [0x7fc6ef2f102d] /usr/lib/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x76)[0x7fc6efac3b36] /usr/lib/libstdc++.so.6(_ZNSsC1ERKSs+0x3c)[0x7fc6efac3e0c] /apps/mongodb/bin/mongod(_ZNK5mongo11ReplSetImpl16_summarizeStatusERNS_14BSONObjBuilderE+0x8da)[0x7ea7aa] /apps/mongodb/bin/mongod(_ZN5mongo19CmdReplSetGetStatus3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x118)[0x7dd448] /apps/mongodb/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a9)[0x973b49] /apps/mongodb/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff)[0x97512f] /apps/mongodb/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35)[0x95d725] /apps/mongodb/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xee4)[0x9607b4] /apps/mongodb/bin/mongod[0x87e037] /apps/mongodb/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x55c)[0x88485c] /apps/mongodb/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76)[0xa96a46] /apps/mongodb/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287)[0x635dd7] /lib/libpthread.so.0(+0x68ba)[0x7fc6efd358ba]