I woke up this morning to see one of my mongod shards had a segfault over night.
Mon Oct 15 13:25:10 [conn46] waiting till out of critical section Mon Oct 15 13:25:10 [conn235] waiting till out of critical section Mon Oct 15 13:25:10 [conn46] waiting till out of critical section Mon Oct 15 13:25:10 [conn235] waiting till out of critical section Mon Oct 15 13:25:10 [conn234] waiting till out of critical section Mon Oct 15 13:25:10 [conn42] waiting till out of critical section Mon Oct 15 13:25:10 [conn234] waiting till out of critical section Mon Oct 15 13:25:16 [conn235] waiting till out of critical section Mon Oct 15 13:25:16 [conn47] moveChunk migrate commit not accepted by TO-shard: { active: true, ns: "pb3.hourly_stats", from: "mongo3.foobar.com:27018", min: { _id: "123.123.123.123-2012-10-02_02:52:40.121883_123.123.123.123.log.gz-a1034eed7679ecb8c8add69d8167afe6" }, max: { _id: "123.123.123.123-2012-10-02_02:57:40.457075_184.169.161.72.log.gz-5b0d8915c0f603a15102ab19f6fc01bc" }, shardKeyPattern: { _id: 1 }, state: "fail", errmsg: "", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, errmsg: "", ok: 0.0 } resetting shard version to: 4699|65||5069e0edf896a39bd4a02ee3 Mon Oct 15 13:25:16 [conn16] serverStatus was very slow: { after basic: 0, middle of mem: 0, after mem: 0, after connections: 0, after extra info: 0, after counters: 0, after repl: 0, after asserts: 0, after dur: 4690, at end: 4690 } Mon Oct 15 13:25:16 Invalid access at address: 0 from thread: conn237 Mon Oct 15 13:25:16 Got signal: 11 (Segmentation fault). Mon Oct 15 13:25:16 Backtrace: 0xade6e1 0x5582d9 0x558862 0x7f659a678060 0x842976 0xa422ae 0xa42789 0x6571f1 0x658d5d 0x659c1e 0x6afa08 0x6b3241 0xb5ba7d 0xb5d052 0x56fa52 0x5dbf11 0x7f659a66fefc 0x7f6599a0159d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9] /usr/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x262) [0x558862] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7f659a678060] /usr/bin/mongod(_ZN5mongo16BSONArrayBuilder6appendINS_7BSONObjEEERS0_RKT_+0x156) [0x842976] /usr/bin/mongod(_ZN5mongo17MigrateFromStatus4xferEPSt4listINS_7BSONObjESaIS2_EERNS_14BSONObjBuilderEPKcRxb+0x52e) [0xa422ae] /usr/bin/mongod(_ZN5mongo19TransferModsCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0xb9) [0xa42789] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRNS_14BSONObjBuilderEb+0x51) [0x6571f1] /usr/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0xcfd) [0x658d5d] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x2ae) [0x659c1e] /usr/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x38) [0x6afa08] /usr/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0xbc1) [0x6b3241] /usr/bin/mongod() [0xb5ba7d] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x3a2) [0xb5d052] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x82) [0x56fa52] /usr/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x411) [0x5dbf11] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f659a66fefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6599a0159d] ubuntu@mongo3:~$ mongod --version db version v2.2.0, pdfile version 4.5 Mon Oct 15 17:40:29 git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
- related to
-
SERVER-7493 Possible for read starvation to cause migration to get stuck in critical section
- Closed