Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-61427

Unique index builds can cause a loss of availability during commit due to checking many false duplicates

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 5.2.0, 5.1.2, 5.0.6, 4.4.11, 4.2.18
    • Affects Version/s: 4.2.0, 4.4.0, 5.0.0
    • Component/s: None
    • Labels:
    • Fully Compatible
    • v5.1, v5.0, v4.4, v4.2
    • Execution Team 2021-11-29

      While an index build is in progress, we take any concurrent writes and insert them into the side writes table. Once the collection scan and bulk load, we drain the writes from the side writes table into the index. However, it is possible for these concurrent writes to also be seen by the collection scan. In this case, we end up inserting the same index key twice: once during collection scan/bulk load, and once while draining side writes. This is not a correctness issue; for non-unique indexes we simply ignore that the key already exists, and for unique indexes we write down the key as a potential duplicate. For the unique case, we then check all of these false "duplicates" under a MODE_X collection lock before committing the index. This can lead to a loss of availability if the index being built is on a large collection with many writes during the collection scan.

      A couple potential solutions:

      • Initially check duplicates under a MODE_IX lock and then later use a MODE_X lock, similarly to what we do for draining the side writes table
      • When draining the side writes table for a unique index, if the entire key (including the record id, as opposed to only the key prefix) already exists in the table, do not count as a potential duplicate

            gregory.noma@mongodb.com Gregory Noma
            gregory.noma@mongodb.com Gregory Noma
            0 Vote for this issue
            17 Start watching this issue