-
Type: Bug
-
Resolution: Done
-
Priority: Critical - P2
-
Affects Version/s: 2.6.9
-
Component/s: Index Maintenance
-
None
-
Minor Change
-
-
RPL 4 06/05/15
We just had 2/3 of the servers in our cluster crash and and then continue to crash when restarted. The log says:
2014-11-28T15:30:01.479-0500 [repl writer worker 6] added index to empty collection 2014-11-28T15:30:01.486-0500 [repl writer worker 6] warning: newExtent 1427 scanned 2014-11-28T15:30:01.503-0500 [repl index builder 933] build index on: lotus_stats.temp.InfestedEventScoreT1 properties: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT1", sparse: false, background: true } 2014-11-28T15:30:01.503-0500 [repl index builder 933] building index in background 2014-11-28T15:30:01.504-0500 [repl index builder 933] build index done. scanned 297 total records. 0.001 secs 2014-11-28T15:30:01.533-0500 [repl writer worker 7] build index on: lotus_stats.temp.InfestedEventScoreT2 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "lotus_stats.temp.InfestedEventScoreT2" } 2014-11-28T15:30:01.534-0500 [repl writer worker 7] added index to empty collection 2014-11-28T15:30:01.535-0500 [repl writer worker 7] warning: newExtent 1427 scanned 2014-11-28T15:30:01.552-0500 [repl index builder 934] build index on: lotus_stats.temp.InfestedEventScoreT2 properties: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT2", sparse: false, background: true } 2014-11-28T15:30:01.552-0500 [repl index builder 934] building index in background 2014-11-28T15:30:01.552-0500 [repl writer worker 7] halting index build: { r: 1.0 } 2014-11-28T15:30:01.566-0500 [repl writer worker 7] halted 1 index build(s) 2014-11-28T15:30:01.576-0500 [repl writer worker 7] uh oh: 36864 2014-11-28T15:30:01.590-0500 [repl writer worker 7] lotus_stats.archived.InfestedEventScoreT2 Assertion failure n >= 0 && n < static_cast<int>(_files.size()) src/mongo/db/storage/extent_manager.cpp 109 2014-11-28T15:30:01.634-0500 [repl writer worker 7] lotus_stats.archived.InfestedEventScoreT2 0x11e9b11 0x118b849 0x116fb5e 0xefb319 0xefb7dd 0xf00739 0x8cf83d 0x9bdcff 0xa2939a 0xa2b151 0xa2c9a6 0xe547d3 0xeb8d9e 0xeb96b0 0x117f0ee 0x122e4a9 0x7fdd0b04fe9a 0x7fdd0a36331d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x116fb5e] /usr/bin/mongod(_ZNK5mongo13ExtentManager12_getOpenFileEi+0xc9) [0xefb319] /usr/bin/mongod(_ZNK5mongo13ExtentManager9recordForERKNS_7DiskLocE+0x1d) [0xefb7dd] /usr/bin/mongod(_ZNK5mongo7DiskLoc3objEv+0x19) [0xf00739] /usr/bin/mongod(_ZN5mongo8Database16renameCollectionERKNS_10StringDataES3_b+0x9cd) [0x8cf83d] /usr/bin/mongod(_ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38cf) [0x9bdcff] /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2939a] /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x19b1) [0xa2b151] /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2c9a6] /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x973) [0xe547d3] /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x4fe) [0xeb8d9e] /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb96b0] /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117f0ee] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdd0b04fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdd0a36331d] 2014-11-28T15:30:01.665-0500 [repl writer worker 7] warning: repl Failed command { renameCollection: "lotus_stats.temp.InfestedEventScoreT2", to: "lotus_stats.archived.InfestedEventScoreT2", dropTarget: true } on admin with status UnknownError exception: assertion src/mongo/db/storage/extent_manager.cpp:109 during oplog application 2014-11-28T15:30:01.666-0500 [repl index builder 934] index build failed. spec: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT2", sparse: false, background: true } error: 11601 operation was interrupted 2014-11-28T15:30:01.666-0500 [repl index builder 934] lotus_stats.temp.InfestedEventScoreT2 Fatal Assertion 17204 2014-11-28T15:30:01.670-0500 [repl index builder 934] lotus_stats.temp.InfestedEventScoreT2 0x11e9b11 0x118b849 0x116e37d 0x8e387f 0x8e56ee 0xb8b93f 0xb8c498 0x1171532 0x122e4a9 0x7fdd0b04fe9a 0x7fdd0a36331d /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11] /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849] /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x116e37d] /usr/bin/mongod(_ZN5mongo12IndexCatalog15IndexBuildBlock4failEv+0x14f) [0x8e387f] /usr/bin/mongod(_ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE+0xa5e) [0x8e56ee] /usr/bin/mongod(_ZNK5mongo12IndexBuilder5buildERNS_6Client7ContextE+0x54f) [0xb8b93f] /usr/bin/mongod(_ZN5mongo12IndexBuilder3runEv+0x728) [0xb8c498] /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1171532] /usr/bin/mongod() [0x122e4a9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdd0b04fe9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdd0a36331d] 2014-11-28T15:30:01.670-0500 [repl index builder 934] ***aborting after fassert() failure
uname -a Linux war-stats1 3.13.0-39-generic #66~precise1-Ubuntu SMP Wed Oct 29 09:56:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
- is duplicated by
-
SERVER-17927 Dropping collection during active background index build on secondary crashed the server
- Closed