Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-9904

Secondaries crash when too many indexes

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical - P2 Critical - P2
    • None
    • 2.4.4
    • None
    • None
    • ALL

    Description

      I upgraded the primary and one secondary on my four node + two arbiter cluster (one node nonvoting) yesterday, from 2.2.3 to 2.4.4

      This morning all of my secondaries crashed with this error:

      Tue Jun 11 14:26:20 [repl writer worker 15] ERROR: add index fails, too
      many indexes for
      appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides key:{
      toyStory: 1 }
      writer worker caught exception: add index fails, too many indexes for
      appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides key:{
      toyStory: 1 } on: { ts: Timestamp 1370960780000|22, h:
      2282539247784756123, v: 2, op: "i", ns: "appdata49.system.indexes", o: {
      name: "toyStory_1", ns:
      "appdata49.app_b054cda3-7bd2-4f42-8bd9-31ca03987684:DisneyRides", key: {
      toyStory: 1 }, background: true } }
      Tue Jun 11 14:26:20 [repl writer worker 15]   Fatal Assertion 16360
      0xb07561 0xacc8b3 0x9abaf6 0xadab5d 0xb4d3d9 0x7fa1bc2cee9a 0x7fa1bb5e1cbd
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xacc8b3]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjE
      SaIS2_EEPNS0_8SyncTailE+0x156) [0x9abaf6]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
       /usr/bin/mongod() [0xb4d3d9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fa1bc2cee9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fa1bb5e1cbd]
      Tue Jun 11 14:26:20 [repl writer worker 15]
       
      ***aborting after fassert() failure
       
       
      Tue Jun 11 14:26:20 Got signal: 6 (Aborted).
       
      Tue Jun 11 14:26:20 Backtrace:
      0xb07561 0x5598c9 0x7fa1bb5244a0 0x7fa1bb524425 0x7fa1bb527b8b 0xacc8ee
      0x9abaf6 0xadab5d 0xb4d3d9 0x7fa1bc2cee9a 0x7fa1bb5e1cbd
       /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb07561]
       /usr/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0x5598c9]
       /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7fa1bb5244a0]
       /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7fa1bb524425]
       /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7fa1bb527b8b]
       /usr/bin/mongod(_ZN5mongo13fassertFailedEi+0xde) [0xacc8ee]
       /usr/bin/mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjE
      SaIS2_EEPNS0_8SyncTailE+0x156) [0x9abaf6]
       /usr/bin/mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xadab5d]
       /usr/bin/mongod() [0xb4d3d9]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7fa1bc2cee9a]
       /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fa1bb5e1cbd]

      We've seen the "add index fails, too many indexes" error before many times on our 2.2 primaries, and it's never caused a problem. Is this a bug that was introduced in 2.4?

      We dynamically generate indexes, so this will make it effectively impossible for us to upgrade to 2.4 until it's fixed.

      Attachments

        Activity

          People

            scotthernandez Scott Hernandez (Inactive)
            charity@parse.com charity majors
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: