Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16348

Assertion failure n >= 0 && n < static_cast<int>(_files.size()) src/mongo/db/storage/extent_manager.cpp 109

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Critical - P2 Critical - P2
    • 2.6.11
    • Affects Version/s: 2.6.9
    • Component/s: Index Maintenance
    • Labels:
      None
    • Minor Change
    • Hide

      If I had to guess this might be two different clients doing an background ensureIndex on the same collection.

      We updated to 2.6.6 from 2.4 a few weeks ago – the code in question has been running faithfully for about a year. We tend to do this a lot (create transient collections and delete them weekly with lazy ensureIndex on demand) so I'm terrified about this causing stability problems.

      Show
      If I had to guess this might be two different clients doing an background ensureIndex on the same collection. We updated to 2.6.6 from 2.4 a few weeks ago – the code in question has been running faithfully for about a year. We tend to do this a lot (create transient collections and delete them weekly with lazy ensureIndex on demand) so I'm terrified about this causing stability problems.
    • RPL 4 06/05/15

      We just had 2/3 of the servers in our cluster crash and and then continue to crash when restarted. The log says:

      2014-11-28T15:30:01.479-0500 [repl writer worker 6]      added index to empty collection
      2014-11-28T15:30:01.486-0500 [repl writer worker 6] warning: newExtent 1427 scanned
      2014-11-28T15:30:01.503-0500 [repl index builder 933] build index on: lotus_stats.temp.InfestedEventScoreT1 properties: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT1", sparse: false, background: true }
      2014-11-28T15:30:01.503-0500 [repl index builder 933]    building index in background
      2014-11-28T15:30:01.504-0500 [repl index builder 933] build index done.  scanned 297 total records. 0.001 secs
      2014-11-28T15:30:01.533-0500 [repl writer worker 7] build index on: lotus_stats.temp.InfestedEventScoreT2 properties: { v: 1, key: { _id: 1 }, name: "_id_", ns: "lotus_stats.temp.InfestedEventScoreT2" }
      2014-11-28T15:30:01.534-0500 [repl writer worker 7]      added index to empty collection
      2014-11-28T15:30:01.535-0500 [repl writer worker 7] warning: newExtent 1427 scanned
      2014-11-28T15:30:01.552-0500 [repl index builder 934] build index on: lotus_stats.temp.InfestedEventScoreT2 properties: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT2", sparse: false, background: true }
      2014-11-28T15:30:01.552-0500 [repl index builder 934]    building index in background
      2014-11-28T15:30:01.552-0500 [repl writer worker 7] halting index build: { r: 1.0 }
      2014-11-28T15:30:01.566-0500 [repl writer worker 7] halted 1 index build(s)
      2014-11-28T15:30:01.576-0500 [repl writer worker 7] uh oh: 36864
      2014-11-28T15:30:01.590-0500 [repl writer worker 7] lotus_stats.archived.InfestedEventScoreT2 Assertion failure n >= 0 && n < static_cast<int>(_files.size()) src/mongo/db/storage/extent_manager.cpp 109
      2014-11-28T15:30:01.634-0500 [repl writer worker 7] lotus_stats.archived.InfestedEventScoreT2 0x11e9b11 0x118b849 0x116fb5e 0xefb319 0xefb7dd 0xf00739 0x8cf83d 0x9bdcff 0xa2939a 0xa2b151 0xa2c9a6 0xe547d3 0xeb8d9e 0xeb96b0 0x117f0ee 0x122e4a9 0x7fdd0b04fe9a 0x7fdd0a36331d
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11]
       /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849]
       /usr/bin/mongod(_ZN5mongo12verifyFailedEPKcS1_j+0x17e) [0x116fb5e]
       /usr/bin/mongod(_ZNK5mongo13ExtentManager12_getOpenFileEi+0xc9) [0xefb319]
       /usr/bin/mongod(_ZNK5mongo13ExtentManager9recordForERKNS_7DiskLocE+0x1d) [0xefb7dd]
       /usr/bin/mongod(_ZNK5mongo7DiskLoc3objEv+0x19) [0xf00739]
       /usr/bin/mongod(_ZN5mongo8Database16renameCollectionERKNS_10StringDataES3_b+0x9cd) [0x8cf83d]
       /usr/bin/mongod(_ZN5mongo19CmdRenameCollection3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x38cf) [0x9bdcff]
       /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2939a]
       /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x19b1) [0xa2b151]
       /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2c9a6]
       /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x973) [0xe547d3]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x4fe) [0xeb8d9e]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb96b0]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117f0ee]
       /usr/bin/mongod() [0x122e4a9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdd0b04fe9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdd0a36331d]
      2014-11-28T15:30:01.665-0500 [repl writer worker 7] warning: repl Failed command { renameCollection: "lotus_stats.temp.InfestedEventScoreT2", to: "lotus_stats.archived.InfestedEventScoreT2", dropTarget: true } on admin with status UnknownError exception: assertion src/mongo/db/storage/extent_manager.cpp:109 during oplog application
      2014-11-28T15:30:01.666-0500 [repl index builder 934] index build failed. spec: { v: 1, unique: true, key: { r: 1.0 }, name: "r_1", ns: "lotus_stats.temp.InfestedEventScoreT2", sparse: false, background: true } error: 11601 operation was interrupted
      2014-11-28T15:30:01.666-0500 [repl index builder 934] lotus_stats.temp.InfestedEventScoreT2 Fatal Assertion 17204
      2014-11-28T15:30:01.670-0500 [repl index builder 934] lotus_stats.temp.InfestedEventScoreT2 0x11e9b11 0x118b849 0x116e37d 0x8e387f 0x8e56ee 0xb8b93f 0xb8c498 0x1171532 0x122e4a9 0x7fdd0b04fe9a 0x7fdd0a36331d
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e9b11]
       /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x118b849]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xcd) [0x116e37d]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog15IndexBuildBlock4failEv+0x14f) [0x8e387f]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog11createIndexENS_7BSONObjEbNS0_16ShutdownBehaviorE+0xa5e) [0x8e56ee]
       /usr/bin/mongod(_ZNK5mongo12IndexBuilder5buildERNS_6Client7ContextE+0x54f) [0xb8b93f]
       /usr/bin/mongod(_ZN5mongo12IndexBuilder3runEv+0x728) [0xb8c498]
       /usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0xd2) [0x1171532]
       /usr/bin/mongod() [0x122e4a9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fdd0b04fe9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fdd0a36331d]
      2014-11-28T15:30:01.670-0500 [repl index builder 934]
      
      ***aborting after fassert() failure
      
      uname -a
      Linux war-stats1 3.13.0-39-generic #66~precise1-Ubuntu SMP Wed Oct 29 09:56:49 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
      

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            glen.miner Glen Miner
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: