Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43882

Building indexes for startup recovery uses unowned RecordData after yielding its cursor

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: 4.2.0
    • Fix Version/s: 4.3.1, 4.2.2
    • Component/s: Indexing, Storage
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v4.2
    • Sprint:
      Execution Team 2019-11-04, Execution Team 2019-11-18

      Description

      Issue Status as of Jan 6, 2020

      ISSUE DESCRIPTION AND IMPACT
      The code for building indexes during the restart of a replica set secondary accesses an un-owned memory object. If the memory backing the object has been re-used by a new owner, the index build process accesses unexpected memory locations and is likely to fail when parsing BSON in those locations.

      This causes documents to be treated as corrupt when they are not, and can trigger SERVER-44904 which removes the documents without removing keys from all indexes that reference removed documents. This creates index inconsistencies on the node.

      In the extremely unlikely event that consistent BSON is parsed from inconsistent memory, index keys themselves can become corrupted and will not reflect values in the document. This would cause incorrect results for queries that use the new index.

      DIAGNOSIS AND AFFECTED VERSIONS
      Replica sets running versions 4.2.0 and 4.2.1 are affected by this bug.

      The presence of "Invalid BSON detected" errors in the server process logs during startup initialization indicate that this bug has had an impact, and that documents have been lost on that node. This data inconsistency between nodes can lead to data loss.

      The validate command detects index inconsistencies and key corruption.

      REMEDIATION AND WORKAROUNDS
      The fix is included in the 4.2.2 production release. All users running 4.2.0 or 4.2.1 should upgrade to this version.

      If a Secondary node crashes while building an index, do not restart the node immediately. Instead, perform an upgrade to MongoDB 4.2.2 starting with the impacted node.

      If you see "Invalid BSON detected" errors logged on a replica set secondary that was restarted during an index build, resync that node from an un-impacted node.

      Original description

      startBuildingIndexesForRecovery uses an unowned RecordData after saving, committing, and restoring its cursor.

      This is problematic because the RecordData can point to freed or overwritten memory.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              milkie Eric Milkie
              Reporter:
              louis.williams Louis Williams
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: