Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-12956

Stopping a secondary in phase 2/3 of an index build corrupts the index

XMLWordPrintableJSON

    • ALL
    • Hide

      Create a replica set
      Have enough data in the set to make the index build take 10s of seconds
      Build an index on the primary
      Kill -5 the secondary when it reaches phase 2/3 of the index build (you can follow the logs to find this)
      Start the secondary (it will not rebuild the index)
      Insert a document to the primary
      The secondary will crash

      Show
      Create a replica set Have enough data in the set to make the index build take 10s of seconds Build an index on the primary Kill -5 the secondary when it reaches phase 2/3 of the index build (you can follow the logs to find this) Start the secondary (it will not rebuild the index) Insert a document to the primary The secondary will crash

      Issue Status as of March 31, 2014

      ISSUE SUMMARY

      If a secondary node is shut down during phase 2 of an index build (constructing the B-Tree), it will not attempt to rebuild the index upon restart and leave behind a corrupted index catalog. Subsequent inserts on the primary that are replicated to this secondary node will crash the secondary node.

      USER IMPACT

      A secondary node affected in this way has to be resynced or repaired. This only happens during abnormal termination of the mongod process with an index build in progress. Normal shutdown will not trigger this issue.

      SOLUTION

      On startup, incomplete index builds need to be cleaned up. The index creation needs to be manually restarted.

      WORKAROUNDS

      None.

      AFFECTED VERSIONS

      All versions from 2.4.0 to 2.4.9 are affected by this bug. The 2.6 series is unaffected by the issue.

      PATCHES

      The fix is included in the 2.4.10 production release.

      Original Description

      If a secondary is stopped during phase 2 of an index build the member will not attempt to re-build the index (as it does in phase 1). Leaving a corrupt index on that secondary will cause crashes and other problems.

        1. mongod.log
          23 kB
        2. server12956_killsecindex.js
          2 kB

            Assignee:
            eliot Eliot Horowitz (Inactive)
            Reporter:
            david.hows David Hows
            Votes:
            6 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: