Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-6366

Off-by-one overflow in block-modification bitmaps for incremental backup

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker - P1
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT3.2.2, 4.5.1, 4.2.8, 4.4.0-rc9
    • Component/s: None
    • Labels:
      None
    • Case:
    • Backport Requested:
      v4.4, v4.2

      Description

      Issue Status as of June 15, 2020

      ISSUE SUMMARY
      There is a risk of in-memory corruption on MongoDB 4.2.7 when incremental backup is enabled.

      USER IMPACT
      For this bug to occur:

      • The deployment must be running MongoDB 4.2.7.
      • The deployment is managed by Ops Manager 4.2.13+ or managed in Cloud Manager with MongoDB agent version v10.14.0+.
      • Incremental backups are enabled.
      • The deployment has collection files larger than 2GB on disk.
      • These collections are modified anytime after backups are enabled.

      If these conditions are met (and only if all these conditions are met), there is a chance that MongoDB will improperly overwrite memory. The most likely results of this bug are failed operations or process crashes. It is possible that a document in memory may be corrupted in a way that allows it to be returned to a client or persisted to disk without an explicit failure.

      SUGGESTED ACTION
      All users on MongoDB 4.2.7 with incremental backup should upgrade to MongoDB 4.2.8 to avoid encountering this issue. If you have experienced a process crash and your deployment meets the above conditions, it may be advisable to validate the collections on the process to ensure that the data is consistent.

      AFFECTED VERSIONS
      This issue affects version 4.2.7.

      FIX VERSION
      The fix is included in 4.2.8.

      Original Description

      Michael Cahill found an error the code for managing block bitmaps for incremental backups.  Specifically, when touch the bit one past the end of the current map, we fail to extend the bitmap in this case and wind up overflowing the existing bitmap buffer, probably with bad consequences.

      With a default incremental backup granularity of 16MB and a bitmap allocation that starts at 128 bits and grows 64-bits at a time, this would only trigger if somebody modifies a large (> 2GB) file on a GB boundary.  

       

        Attachments

        1. fix.diff
          1 kB
        2. repro.diff
          5 kB

          Issue Links

            Activity

              People

              Assignee:
              keith.smith Keith Smith
              Reporter:
              keith.smith Keith Smith
              Votes:
              1 Vote for this issue
              Watchers:
              23 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: