-
Type:
Bug
-
Resolution: Incomplete
-
Priority:
Major - P3
-
None
-
Affects Version/s: 2.4.6
-
Component/s: Internal Code
-
Labels:
-
Environment:Ubuntu 10.04.4
32 model name : Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz / 127Gb memory
-
Linux
Hi.
We have RS mongodb:
> rs.conf()
{
"_id" : "mdb01",
"version" : 19959,
"members" : [
{
"_id" : 0,
"host" : "mdb01d:27018"
},
{
"_id" : 1,
"host" : "mdb01:27018"
},
{
"_id" : 2,
"host" : "mdb01g:27018"
},
{
"_id" : 3,
"host" : "mdb-backup01d:27018",
"votes" : 0,
"priority" : 0,
"hidden" : true
}
]
}
today, secondary node's from our Replica Set out of order.
Wed Nov 20 21:28:40.921 [conn20] replSet RECOVERING
Wed Nov 20 21:28:40.921 [conn20] replSet info voting yea for mdb01e:27018 (1)
Wed Nov 20 21:28:41.055 [conn7] end connection 1.1.1.1:59411 (28 connections now open)
Wed Nov 20 21:28:41.483 [rsHealthPoll] DBClientCursor::init call() failed
Wed Nov 20 21:28:41.484 [rsHealthPoll] replset info rty-mdb-backup01d:27018 heartbeat failed, retrying
Wed Nov 20 21:28:41.485 [rsHealthPoll] replSet info rty-mdb-backup01d:27018 is down (or slow to respond):
Wed Nov 20 21:28:41.485 [rsHealthPoll] replSet member rty-mdb-backup01dt:27018 is now in state DOWN
Wed Nov 20 21:28:41.487 [rsHealthPoll] replSet member rty-mdb01e:27018 is now in state PRIMARY
Wed Nov 20 21:28:41.488 [rsHealthPoll] DBClientCursor::init call() failed
Wed Nov 20 21:28:41.488 [rsHealthPoll] replset info rty-mdb01g:27018 heartbeat failed, retrying
Wed Nov 20 21:28:41.490 [rsHealthPoll] replSet info rty-mdb01g:27018 is down (or slow to respond):
Wed Nov 20 21:28:41.490 [rsHealthPoll] replSet member rty-mdb01g:27018 is now in state DOWN
Wed Nov 20 21:28:42.474 [rsBackgroundSync] replSet syncing to: rty-mdb01e:27018
Wed Nov 20 21:28:42.475 [rsSync] replSet still syncing, not yet to minValid optime 528ce54b:20
Wed Nov 20 21:28:42.565 [rsSync] replSet SECONDARY
Wed Nov 20 21:28:42.648 [repl writer worker 2] ERROR: writer worker caught exception: E11000 duplicate key error index: fotki-bazinga.onetimeJobs.$activeUniqueIdentifier_1 dup key: { : "refreshCounters_{"userId":100544310,"albumId":377350}" } on: { ts: Timestamp 1384965478000|3, h: -6072474236908557566, v: 2, op: "i", ns: "fotki-bazinga.onetimeJobs", o: { _id: { taskId: "refreshCounters", jobId: 1384965478119 }, scheduleTime: new Date(1384965508119), activeUniqueIdentifier: "refreshCounters_{"userId":100544310,"albumId":377350}", parameters: "{"userId":100544310,"albumId":377350}", status: "ready", workers: {}, priority: 20 } }
Wed Nov 20 21:28:42.648 [repl writer worker 2] Fatal Assertion 16360
0xc5e916 0xc23273 0xb14021 0xc30b59 0xc9bbac 0x7f5cb0dd69ca 0x7f5cb017d21d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xc5e916]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x63) [0xc23273]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x121) [0xb14021]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x279) [0xc30b59]
/usr/bin/mongod() [0xc9bbac]
/lib/libpthread.so.0(+0x69ca) [0x7f5cb0dd69ca]
/lib/libc.so.6(clone+0x6d) [0x7f5cb017d21d]
Wed Nov 20 21:28:42.651 [repl writer worker 2]
***aborting after fassert() failure
Wed Nov 20 21:28:42.651 Got signal: 6 (Aborted).
Wed Nov 20 21:28:42.653 Backtrace:
0xc5e916 0x70c044 0x7f5cb00c7ba0 0x7f5cb00c7b25 0x7f5cb00cb670 0xc232ae 0xb14021 0xc30b59 0xc9bbac 0x7f5cb0dd69ca 0x7f5cb017d21d
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x26) [0xc5e916]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x3c4) [0x70c044]
/lib/libc.so.6(+0x33ba0) [0x7f5cb00c7ba0]
/lib/libc.so.6(gsignal+0x35) [0x7f5cb00c7b25]
/lib/libc.so.6(abort+0x180) [0x7f5cb00cb670]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0x9e) [0xc232ae]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x121) [0xb14021]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x279) [0xc30b59]
/usr/bin/mongod() [0xc9bbac]
/lib/libpthread.so.0(+0x69ca) [0x7f5cb0dd69ca]
/lib/libc.so.6(clone+0x6d) [0x7f5cb017d21d]
Our RS switched to read only, because we have lost all secondary nodes than fully made the service unusable (