-
Type: Bug
-
Resolution: Duplicate
-
Priority: Critical - P2
-
None
-
Affects Version/s: 2.4.8
-
Component/s: None
-
None
-
ALL
Cluster = 1 PRIMARY + 3 SECONDARY + 1 ARBITER
MASTER and ARBITER are ok, all 3 SECONDARY crashed, and still crashed after restart instance.
I saw some duplicated key error in mongodb.log sometimes, but this was first time they cashed.
mongodb logs:
SECONDARY1:
```
Wed May 14 15:01:58.452 [repl writer worker 1] info: indexing in foreground on this replica; was a background index build on the primary
Wed May 14 15:01:58.452 [repl writer worker 1] build index vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi
Wed May 14 15:01:58.457 [repl writer worker 1] ERROR: writer worker caught exception: E11000 duplicate key error index: vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi.$dongxiID_1 dup key: { : null } on: { ts: Timestamp 1400050387000|17, h: -1650836744667758, v: 2, op: "i", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.system.indexes", o: { name: "dongxiID_1", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi", background: true, unique: true, key:
{ dongxiID: 1 } } }
Wed May 14 15:01:58.457 [repl writer worker 1] Fatal Assertion 16360
0xde05e1 0xda03d3 0xc28f3c 0xdadf21 0xe28e69 0x7fb81feeae9a 0x7fb81f1fdccd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xda03d3]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fb81feeae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb81f1fdccd]
Wed May 14 15:01:58.459 [repl writer worker 1]
***aborting after fassert() failure
Wed May 14 15:01:58.459 Got signal: 6 (Aborted).
Wed May 14 15:01:58.461 Backtrace:
0xde05e1 0x6d0559 0x7fb81f1404a0 0x7fb81f140425 0x7fb81f143b8b 0xda040e 0xc28f3c 0xdadf21 0xe28e69 0x7fb81feeae9a 0x7fb81f1fdccd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fb81f1404a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fb81f140425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fb81f143b8b]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xda040e]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fb81feeae9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb81f1fdccd]
```
SECONDARY2:
```
Wed May 14 14:53:08.718 [repl writer worker 1] ERROR: writer worker caught exception: E11000 duplicate key error index: vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi.$dongxiID_1 dup key: { : null } on: { ts: Timestamp 1400050387000|17, h: -1650836744667758, v: 2, op: "i", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.system.indexes", o: { name: "dongxiID_1", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi", background: true, unique: true, key:
} }
Wed May 14 14:53:08.718 [repl writer worker 1] Fatal Assertion 16360
Wed May 14 14:53:08.724 [initandlisten] connection accepted from 10.124.57.3:41607 #15706255 (138 connections now open)
0xde05e1 0xda03d3 0xc28f3c 0xdadf21 0xe28e69 0x7fdfe6683e9a 0x7fdfe59963fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xda03d3]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdfe6683e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdfe59963fd]
Wed May 14 14:53:08.740 [repl writer worker 1]
***aborting after fassert() failure
Wed May 14 14:53:08.748 Got signal: 6 (Aborted).
Wed May 14 14:53:08.750 Backtrace:
0xde05e1 0x6d0559 0x7fdfe58d84a0 0x7fdfe58d8425 0x7fdfe58dbb8b 0xda040e 0xc28f3c 0xdadf21 0xe28e69 0x7fdfe6683e9a 0x7fdfe59963fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fdfe58d84a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fdfe58d8425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fdfe58dbb8b]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xda040e]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdfe6683e9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdfe59963fd]
```
SECONDARY3:
```
Wed May 14 14:53:07.291 [repl writer worker 13] info: indexing in foreground on this replica; was a background index build on the primary
Wed May 14 14:53:07.291 [repl writer worker 13] build index vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi
Wed May 14 14:53:07.353 [repl writer worker 13] ERROR: writer worker caught exception: E11000 duplicate key error index: vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi.$dongxiID_1 dup key: { : null } on: { ts: Timestamp 1400050387000|17, h: -1650836744667758, v: 2, op: "i", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.system.indexes", o: { name: "dongxiID_1", ns: "vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.Dongxi", background: true, unique: true, key:
{ dongxiID: 1 } } }
Wed May 14 14:53:07.353 [repl writer worker 13] Fatal Assertion 16360
0xde05e1 0xda03d3 0xc28f3c 0xdadf21 0xe28e69 0x7fbb61e0fe9a 0x7fbb611223fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xda03d3]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fbb61e0fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fbb611223fd]
Wed May 14 14:53:07.402 [repl writer worker 13]
***aborting after fassert() failure
Wed May 14 14:53:07.409 Got signal: 6 (Aborted).
Wed May 14 14:53:07.421 Backtrace:
0xde05e1 0x6d0559 0x7fbb610644a0 0x7fbb61064425 0x7fbb61067b8b 0xda040e 0xc28f3c 0xdadf21 0xe28e69 0x7fbb61e0fe9a 0x7fbb611223fd
/usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xde05e1]
/usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x6d0559]
/lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fbb610644a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fbb61064425]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fbb61067b8b]
/usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xda040e]
/usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x12c) [0xc28f3c]
/usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x281) [0xdadf21]
/usr/bin/mongod() [0xe28e69]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fbb61e0fe9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fbb611223fd]
```
I've attached db files of vo5xzp29vrev6q5e49nof23a7lljzgexvqm0u95p19jjysfx.
- duplicates
-
SERVER-12662 Two background index builds in parallel with unique key constraint violations crashes secondary
- Closed