Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-15038

Multiple background index builds may not interrupt cleanly for commands, on secondaries

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.6.5
    • Affects Version/s: 2.6.4
    • Component/s: Replication
    • None
    • Fully Compatible
    • ALL

      Issue Status as of Aug 28, 2014

      On a replica set, when a secondary is building multiple background indices on a given collection and that collection is dropped, the index builds may not be interrupted properly and the secondary may fail with an assertion error as follows:

      2014-08-26T11:18:13.384-0400 [repl writer worker 1] Assertion: 17348:cannot dropAllIndexes when index builds in progress

      In some rare cases, this issue may crash secondary nodes.

      The affected collection and indices may not be successfully dropped on secondary nodes, so if one of these nodes later becomes primary, the collection and indices may still be present.

      If a secondary node crashes, it needs to be restarted. If the secondary has become stale, it needs to be resynchronized by removing its data and performing an initial sync.

      The workaround is to avoid the situation of multiple background index builds concurrent with collection drops:

      • by making the indexes foreground builds, or
      • by synchronizing collection drop with index build completion on secondaries through the use of write concern on the background index builds

      MongoDB 2.6 production releases up to 2.6.4 are affected by this issue.

      The fix is included in the 2.6.5 production release.

      After an index build yields and regains its lock, check for interrupt before recovering our cursor.

      Original description

      After the bugfix in 2.6.4 related to this bug https://jira.mongodb.org/browse/SERVER-14494 , we update to 2.6.4, but we still reproducing the error:

      2014-08-25T05:02:21.430+0200 [repl index builder 35135] build index on: 542-110-6347.keyword_adgroup properties: { v: 1, key: { adgroup: 1 }, name: "adgroup_1", ns: "542-110-6347.keyword_adgroup", background: true }
      2014-08-25T05:02:21.430+0200 [repl index builder 35135]          building index in background
      2014-08-25T05:02:21.537+0200 [repl writer worker 12] CMD: drop 542-110-6347.keyword_adgroup
      2014-08-25T05:02:21.537+0200 [repl writer worker 12] halting index build: { adgroup: 1 }
      2014-08-25T05:02:21.537+0200 [repl writer worker 12] halting index build: { campaign: 1 }
      2014-08-25T05:02:21.537+0200 [repl writer worker 12] SEVERE: Invalid access at address: 0x1c00736e3a
      2014-08-25T05:02:21.542+0200 [repl writer worker 12] SEVERE: Got signal: 11 (Segmentation fault).
      Backtrace:0x11e6111 0x11e54ee 0x11e55df 0x7f5367a15340 0x8d9077 0xb898a5 0xa2eaa9 0xa31380 0xa2889a 0xa2a651 0xa2bea6 0xe50dd3 0xeb9819 0xeb9d60 0x117b6ee 0x122aaa9 0x7f5367a0d182 0x7f5366d1230d 
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x11e6111]
       /usr/bin/mongod() [0x11e54ee]
       /usr/bin/mongod() [0x11e55df]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f5367a15340]
       /usr/bin/mongod(_ZN5mongo12IndexCatalog23killMatchingIndexBuildsERKNS0_17IndexKillCriteriaE+0x227) [0x8d9077]
       /usr/bin/mongod(_ZN5mongo12IndexBuilder23killMatchingIndexBuildsEPNS_10CollectionERKNS_12IndexCatalog17IndexKillCriteriaE+0x15) [0xb898a5]
       /usr/bin/mongod(_ZN5mongo7CmdDrop15stopIndexBuildsEPNS_8DatabaseERKNS_7BSONObjE+0xe9) [0xa2eaa9]
       /usr/bin/mongod(_ZN5mongo7CmdDrop3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x1c0) [0xa31380]
       /usr/bin/mongod(_ZN5mongo12_execCommandEPNS_7CommandERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x3a) [0xa2889a]
       /usr/bin/mongod(_ZN5mongo7Command11execCommandEPS0_RNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x19b1) [0xa2a651]
       /usr/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6c6) [0xa2bea6]
       /usr/bin/mongod(_ZN5mongo21applyOperation_inlockERKNS_7BSONObjEbb+0x973) [0xe50dd3]
       /usr/bin/mongod(_ZN5mongo7replset8SyncTail9syncApplyERKNS_7BSONObjEb+0x749) [0xeb9819]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x50) [0xeb9d60]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x19e) [0x117b6ee]
       /usr/bin/mongod() [0x122aaa9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f5367a0d182]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f5366d1230d]
      2014-08-25T08:34:21.426+0200 ***** SERVER RESTARTED *****
      2014-08-25T08:34:21.429+0200 [initandlisten] MongoDB starting : pid=1422 port=27017 dbpath=/var/lib/mongodb 64-bit host=lohap2
      2014-08-25T08:34:21.429+0200 [initandlisten] db version v2.6.4

        1. mongod.log
          54.73 MB

            milkie@mongodb.com Eric Milkie
            eduherraiz Edu Herraiz
            0 Vote for this issue
            9 Start watching this issue