-
Type:
Bug
-
Resolution: Incomplete
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.6.5, 3.0.1
-
Component/s: Networking, Replication
-
None
-
ALL
-
None
-
3
-
None
-
None
-
None
-
None
-
None
-
None
We have 1 shard with replica set (3 members). Primary was crashed (maybe network problems).
Log:
2014-11-03T16:16:28.454-0800 [conn65746] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.101.251:35711] 2014-11-03T16:16:28.459-0800 [conn65836] 0x11e9b11 0x118b849 0x11703f6 0xed29bb 0xecc162 0xf3dc89 0xa0fa83 0xa1338e 0xa2939a 0xa2a7e2 0xa2c9a6 0xd5f83a 0xba1052 0xba2630 0x770d5f 0x119f93e 0x7ffb32105182 0x7ffb31409fbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11703f6] /usr/bin/mongod(_ZN5mongo13SlaveTracking18opReplicatedEnoughENS_6OpTimeERKSs+0x8b) [0xed29bb] /usr/bin/mongod(_ZN5mongo18opReplicatedEnoughENS_6OpTimeERKSs+0x22) [0xecc162] /usr/bin/mongod(_ZN5mongo19waitForWriteConcernERKNS_19WriteConcernOptionsERKNS_6OpTimeEPNS_18WriteConcernResultE+0x209) [0xf3dc89] /usr/bin/mongod(_ZN5mongo18WriteBatchExecutor12executeBatchERKNS_21BatchedCommandRequestEPNS_22BatchedCommandResponseE+0xba3) [0xa0fa83] /usr/bin/mongod(_ZN5mongo8WriteCmd3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3de) [0xa1338e] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2939a] /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x1042) [0xa2a7e2] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2c9a6] /usr/bin/mongod(_ZN5mongo11newRunQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x230a) [0xd5f83a] /usr/bin/mongod() [0xba1052] /usr/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x580) [0xba2630] /usr/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x9f) [0x770d5f] /usr/bin/mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x4ee) [0x119f93e] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ffb32105182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffb31409fbd] 2014-11-03T16:16:28.459-0800 [conn65836] command xsf-content-live.$cmd command: update { $msg: "query not recording (too large)" } ntoreturn:1 keyUpdates:0 numYields:0 reslen:187 27232ms 2014-11-03T16:16:28.459-0800 [conn65836] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [192.168.101.251:35877] 2014-11-03T16:16:29.300-0800 [rsBackgroundSync] Assertion: 16235:going to start syncing, but buffer is not empty 2014-11-03T16:16:29.314-0800 [rsBackgroundSync] 0x11e9b11 0x118b849 0x11703f6 0xe08c85 0xe0f8b5 0xe0f8ed 0x122e4a9 0x7ffb32105182 0x7ffb31409fbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11703f6] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync5startEv+0x2a5) [0xe08c85] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync15_producerThreadEv+0xc5) [0xe0f8b5] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync14producerThreadEv+0x2d) [0xe0f8ed] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ffb32105182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffb31409fbd] 2014-11-03T16:16:29.314-0800 [rsBackgroundSync] replSet sync source problem: 16235 going to start syncing, but buffer is not empty 2014-11-03T16:16:29.314-0800 [rsBackgroundSync] Assertion: 16235:going to start syncing, but buffer is not empty 2014-11-03T16:16:29.325-0800 [rsBackgroundSync] 0x11e9b11 0x118b849 0x11703f6 0xe08c85 0xe0f8b5 0xe0f8ed 0x122e4a9 0x7ffb32105182 0x7ffb31409fbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11703f6] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync5startEv+0x2a5) [0xe08c85] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync15_producerThreadEv+0xc5) [0xe0f8b5] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync14producerThreadEv+0x2d) [0xe0f8ed] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ffb32105182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffb31409fbd] 2014-11-03T16:16:29.325-0800 [rsBackgroundSync] Assertion: 16235:going to start syncing, but buffer is not empty 2014-11-03T16:16:29.329-0800 [rsBackgroundSync] 0x11e9b11 0x118b849 0x11703f6 0xe08c85 0xe0f8b5 0xe0f8ed 0x122e4a9 0x7ffb32105182 0x7ffb31409fbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11703f6] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync5startEv+0x2a5) [0xe08c85] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync15_producerThreadEv+0xc5) [0xe0f8b5] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync14producerThreadEv+0x2d) [0xe0f8ed] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ffb32105182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffb31409fbd] 2014-11-03T16:16:29.329-0800 [rsBackgroundSync] Assertion: 16235:going to start syncing, but buffer is not empty 2014-11-03T16:16:29.331-0800 [rsBackgroundSync] 0x11e9b11 0x118b849 0x11703f6 0xe08c85 0xe0f8b5 0xe0f8ed 0x122e4a9 0x7ffb32105182 0x7ffb31409fbd /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11703f6] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync5startEv+0x2a5) [0xe08c85] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync15_producerThreadEv+0xc5) [0xe0f8b5] /usr/bin/mongod(_ZN5mongo7replset14BackgroundSync14producerThreadEv+0x2d) [0xe0f8ed] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ffb32105182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ffb31409fbd]