-
Type:
Bug
-
Resolution: Duplicate
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Replication
-
Labels:None
-
ALL
One of our mongod instances exited with following assertion failure:
Fri Mar 14 22:20:31.052 [rsSync] replSet error possible failover clock skew issue? 53234f76:6
Fri Mar 14 22:20:48.213 [repl writer worker 1] ERROR: writer worker caught exception: E11000 duplicate key error index: database.collection.$field_1 dup key: { : null } on: { ts: Timestamp 1394779641000|1, h: -3728626592722565358, v: 2, op: "u", ns: "database.collection", o2: { _id: ObjectId('52f2021a0f4faf44db7a3a4a') }, o: { $set: { foo.bar: 2 } } }
Fri Mar 14 22:20:48.219 [repl writer worker 1] Fatal Assertion 16360
0xc4b4d5 0xadc233 0xc5b9a4 0xce02df 0x803030274
0xc4b4d5 <_ZN5mongo13fassertFailedEi+165> at /usr/local/bin/mongod
0xadc233 <_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+307> at /usr/local/bin/mongod
0xc5b9a4 <_ZN5mongo10threadpool6Worker4loopEv+548> at /usr/local/bin/mongod
0xce02df <_ZN5boost11this_thread5sleepERKNS_10posix_time5ptimeE+1439> at /usr/local/bin/mongod
0x803030274 <pthread_getprio+308> at /lib/libthr.so.3
Fri Mar 14 22:20:48.387 [repl writer worker 1]
***aborting after fassert() failure
Fri Mar 14 22:20:48.432 Got signal: 6 (Abort trap: 6).
Fri Mar 14 22:20:48.432 Backtrace:
0x59733c 0x803034626
0x59733c <_ZN5mongo10abruptQuitEi+988> at /usr/local/bin/mongod
0x803034626 <_pthread_sigmask+886> at /lib/libthr.so.3
mongod 2.4.8 on FreeBSD
mongod setup: sharded environment with replication, each replicaset contains one primary, one secondary and one arbiter. error occurred on secondary.
The `duplicate key error` is also strange, we have no such error in primary server.
Also the primary server log contains this errors (starting from Mar 15 02:24:50 and ending in Mar 15 09:21:17)
Sat Mar 15 02:24:50.091 [migrateThread] warning: secondaryThrottle on, but doc insert timed out after 60 seconds, continuing Sat Mar 15 09:21:17.299 [migrateThread] warning: secondaryThrottle on, but doc insert timed out after 60 seconds, continuing
And after that, the log is full of these lines (about 45 lines per second!)
Sat Mar 15 09:21:19.446 [migrateThread] warning: secondaries having hard time keeping up with migrate
- duplicates
-
-
- Closed
-