Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-23977

Failure of MapViewOfFileEx in MemoryMappedFile::remapPrivateView with "errno:487 Attempt to access invalid address"

    • Type: Icon: Bug Bug
    • Resolution: Won't Fix
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: MMAPv1, Storage
    • Labels:
      None
    • Storage Execution
    • ALL
    • Platforms 14 (05/13/16), Platforms 15 (06/03/16)
    • 15

      Issue Status as of Dec 22, 2016

      ISSUE SUMMARY
      MMAPv1 on Windows has a known race condition during flushing that is reported to the log file as:

      [durability] MapViewOfFileEx for ... failed with error errno:487 Attempt to access invalid address. (file size is ...) in MemoryMappedFile::remapPrivateView
      [durability] Fatal Assertion 16148
      

      USER IMPACT
      This race condition exists in Windows version of MongoDB's MMap storage engine, and prevents MongoDB from properly managing memory mapped files. MongoDB will shutdown automatically if it detects this result.

      WORKAROUNDS
      This race condition does not affect storage engines other than MMAPv1. Therefore, to resolve this issue, we recommend transitioning to WiredTiger. Alternatively, since this issue only affects Windows machines, using another operating system would resolve the issue.

      TECHNICAL DETAILS
      The MMAPv1 (Memory Mapped) Storage Engine relies on the operating system's memory management system to manage the caching of the database file in 4kb chunks (i.e. pages) in MongoDB's memory. On Linux, this is implemented with the mmap system call. The operating system automatically reads in pages into memory when accessed, and evicts pages from memory when the system runs low on physical memory.

      One of the features of the MMAPv1 storage engine is the journal. The journal is a sequentially written file used to provide write-ahead-logging (WAL). The journal is used to support recovery of commits in the case of crash in an efficient way.

      In order to support recovery, for each commit made to the database, a description of the change is written to the journal first, next the journal is written to disk, then the database pages are written in memory, and finally the user receives a notification their change is complete. Please note the exact point at which the notification is received depends on write concern.

      In order to ensure the journal is written to disk before the database pages are written to disk, a private copy of each database file is made using the Copy-On-Write (COW) feature of the operating system's memory manager. Modifications are written to this private copy (which is never written to disk). This is necessary since the memory mapped file can be written by the OS at any time due to page eviction. Later, after changes have been written to the journal, the changes to private copy are copied to the file-backed copy.

      Now, after the pages are copied, the private copy, and the file-backed copy will have the same contents. At this point, MongoDB needs to instruct OS that these two copies are the same, and it can safely discard the copies of the file-backed pages it made.

      To tell the OS these pages are the same, the storage engine calls x' = mmap(x) on the private copy, and requires that the mmap function call return the same address back for the private copy (i.e. "x` == x). On Linux, the mmap function has this behavior.

      On Windows, there is no direct equivalent of the mmap function with the required behavior. The solution is a two step operation on Windows via UnmapViewOfFile, and MapViewOfFileEx. Because it is a two step process, there is a window for other components in the MongoD process to make calls into the Windows memory management system, and prevent MapViewOfFileEx from being able to use the same address again. Examples of this are new thread creation, Windows Heap allocation, TCMalloc allocation, and Mozilla Javascript memory allocation. If MapViewOfFileEx fails to map the private copy at the requested address, MongoDB terminates with the following fassert:

      [durability] MapViewOfFileEx for ... failed with error errno:487 Attempt to access invalid address. (file size is ...) in MemoryMappedFile::remapPrivateView
      [durability] Fatal Assertion 16148
      

      Unfortunately, there is no known solution to this issue other than workarounds described above.

      Original description

      On Windows, in MemoryMappedFile::remapPrivateView, if the virtual address occupied by the private view is claimed by another thread between the time the private view is unmapped and the time that it is mapped again, the MapViewOfFileEx call will fail with an "invalid address" error, and mongod will terminate. Following messages are diagnostic of this problem:

      I CONTROL  [durability] MapViewOfFileEx for ... failed with error errno:487 Attempt to access invalid address. (file size is ...) in MemoryMappedFile::remapPrivateView
      I -        [durability] Fatal Assertion 16148
      

            Assignee:
            backlog-server-execution [DO NOT USE] Backlog - Storage Execution Team
            Reporter:
            bruce.lucas@mongodb.com Bruce Lucas (Inactive)
            Votes:
            4 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated:
              Resolved: