-
Type: Bug
-
Resolution: Duplicate
-
Priority: Critical - P2
-
None
-
Affects Version/s: 2.2.0
-
Component/s: Replication
-
None
-
Environment:ubuntu/3.0.0-17
-
Linux
We first got this crash during initial sync of a secondary node, ended up downgrading to 2.0 because the sync could not resume, and upgrading to 2.2 once it completed. Now after three days of uptime we have the crash again:
Sat Sep 8 17:53:33 [repl writer worker 7] ERROR: writer worker caught exception: E11000 duplicate key error index: [collection].$_id_ dup key: { : ObjectId('50287481463c0b83d82f8dea') } on: { ts: Timestamp 1347152002000|57, h: -1992302823673752270, op: "u", ns: "[collection]", o2: { _id: ObjectId('50287481463c0b83d82f8dea'), tu: { $size: 6 } }, o: { $set: { a: ObjectId('5026b145003c0b2238a9ddf6') }, $push: { tu: ObjectId('0000000000000000002721eb') }, $set: { tuc: 7 } } } Sat Sep 8 17:53:33 [repl writer worker 7] Fatal Assertion 16360 0xade6e1 0x802e03 0x64f77d 0x77d3dd 0x7c3659 0x7f69a5e88efc 0x7f69a521a59d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0x802e03] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd] /usr/bin/mongod() [0x7c3659] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f69a5e88efc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a521a59d] Sat Sep 8 17:53:33 [repl writer worker 7] ***aborting after fassert() failure Sat Sep 8 17:53:33 Got signal: 6 (Aborted). Sat Sep 8 17:53:33 Backtrace: 0xade6e1 0x5582d9 0x7f69a516d420 0x7f69a516d3a5 0x7f69a5170b0b 0x802e3e 0x64f77d 0x77d3dd 0x7c3659 0x7f69a5e88efc 0x7f69a521a59d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9] /lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f69a516d420] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f69a516d3a5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f69a5170b0b] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0x802e3e] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd] /usr/bin/mongod() [0x7c3659] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f69a5e88efc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a521a59d]
And it crashes on startup with the same exception:
at Sep 8 17:57:08 [rsStart] replSet STARTUP2 Sat Sep 8 17:57:08 [rsHealthPoll] replSet member 192.168.1.40:27017 is up Sat Sep 8 17:57:08 [rsHealthPoll] replSet member 192.168.1.40:27017 is now in state PRIMARY Sat Sep 8 17:57:08 [rsHealthPoll] replSet member 192.168.1.20:27017 is up Sat Sep 8 17:57:08 [rsHealthPoll] replSet member 192.168.1.20:27017 is now in state ARBITER Sat Sep 8 17:57:09 [rsSync] replSet SECONDARY Sat Sep 8 17:57:13 [rsBackgroundSync] replSet syncing to: 192.168.1.40:27017 Sat Sep 8 17:57:13 [repl writer worker 8] ERROR: writer worker caught exception: E11000 duplicate key error index: [collection].$_id_ dup key: { : ObjectId('50287481463c0b83d82f8dea') } on: { ts: Timestamp 1347152002000|57, h: -1992302823673752270, op: "u", ns: "[collection]", o2: { _id: ObjectId('50287481463c0b83d82f8dea'), tu: { $size: 6 } }, o: { $set: { a: ObjectId('5026b145003c0b2238a9ddf6') }, $push: { tu: ObjectId('0000000000000000002721eb') }, $set: { tuc: 7 } } } Sat Sep 8 17:57:13 [repl writer worker 8] Fatal Assertion 16360 0xade6e1 0x802e03 0x64f77d 0x77d3dd 0x7c3659 0x7f993759cefc 0x7f993692e59d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0x802e03] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd] /usr/bin/mongod() [0x7c3659] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f993759cefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f993692e59d] Sat Sep 8 17:57:13 [repl writer worker 8] ***aborting after fassert() failure Sat Sep 8 17:57:13 Got signal: 6 (Aborted). Sat Sep 8 17:57:13 Backtrace: 0xade6e1 0x5582d9 0x7f9936881420 0x7f99368813a5 0x7f9936884b0b 0x802e3e 0x64f77d 0x77d3dd 0x7c3659 0x7f993759cefc 0x7f993692e59d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xade6e1] /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5582d9] /lib/x86_64-linux-gnu/libc.so.6(+0x36420) [0x7f9936881420] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f99368813a5] /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f9936884b0b] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0x802e3e] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12d) [0x64f77d] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0x77d3dd] /usr/bin/mongod() [0x7c3659] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f993759cefc] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f993692e59d]
- duplicates
-
SERVER-3407 oplog is not idempotent for array operators, which could lead to silent data corruption (without journalling)
- Closed
- is related to
-
SERVER-5961 $push array size constraint can cause incorrect replication of another field during initial sync
- Closed