Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-43882

Building indexes for startup recovery uses unowned RecordData after yielding its cursor

    • Fully Compatible
    • ALL
    • v4.2
    • Execution Team 2019-11-04, Execution Team 2019-11-18

      Issue Status as of Jan 6, 2020

      ISSUE DESCRIPTION AND IMPACT
      The code for building indexes during the restart of a replica set secondary accesses an un-owned memory object. If the memory backing the object has been re-used by a new owner, the index build process accesses unexpected memory locations and is likely to fail when parsing BSON in those locations.

      This causes documents to be treated as corrupt when they are not, and can trigger SERVER-44904 which removes the documents without removing keys from all indexes that reference removed documents. This creates index inconsistencies on the node.

      In the extremely unlikely event that consistent BSON is parsed from inconsistent memory, index keys themselves can become corrupted and will not reflect values in the document. This would cause incorrect results for queries that use the new index.

      DIAGNOSIS AND AFFECTED VERSIONS
      Replica sets running versions 4.2.0 and 4.2.1 are affected by this bug.

      The presence of "Invalid BSON detected" errors in the server process logs during startup initialization indicate that this bug has had an impact, and that documents have been lost on that node. This data inconsistency between nodes can lead to data loss.

      The validate command detects index inconsistencies and key corruption.

      REMEDIATION AND WORKAROUNDS
      The fix is included in the 4.2.2 production release. All users running 4.2.0 or 4.2.1 should upgrade to this version.

      If a Secondary node crashes while building an index, do not restart the node immediately. Instead, perform an upgrade to MongoDB 4.2.2 starting with the impacted node.

      If you see "Invalid BSON detected" errors logged on a replica set secondary that was restarted during an index build, resync that node from an un-impacted node.

      Original description

      startBuildingIndexesForRecovery uses an unowned RecordData after saving, committing, and restoring its cursor.

      This is problematic because the RecordData can point to freed or overwritten memory.

            Assignee:
            milkie@mongodb.com Eric Milkie
            Reporter:
            louis.williams@mongodb.com Louis Williams
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: