Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-53376

[4.4] dbHash can live lock an aborting index build

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.4.4
    • Component/s: None
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Sprint:
      Execution Team 2021-01-11
    • Linked BF Score:
      22

      Description

      dbHash is allowed to hold open storage snapshots indefinitely while waiting for collection locks. Multi-doc transactions do this, but they have lock acquisition deadlines.

      This behavior introduces a very specific live lock in 4.4:

      • dbHash opens a read snapshot using the current cluster time
      • An index build aborts. While holding an X collection lock, the index build attempts to set a ghost commit timestamp for the catalog write using the same cluster time.
      • Due to an assertion in WT, setting the ghost timestamp will fail because there is an open transaction (dbHash) reading at the same timestamp.
      • The index build retries indefinitely, waiting for the dbHash reader to finish.
      • The dbHash operation is unable to make progress because it is blocked by the X lock.

      In general, I believe we should impose a lock timeout such that dbHash cannot hold open snapshots and block indefinitely, much like we already do for multi-document transactions.

      The alternative to imposing a lock deadline would be to fix ghost timestamps, but only in 4.4. I believe a dbHash change will avoid the risk of modifying 4.4 index build code that has been removed in master. Adding a lock timeout to dbHash assumes there are no other consequences of the index build ghost timestamping behavior. This same bug applies to background validation, which will need to undergo the same lock timeout change (SERVER-53445).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              louis.williams Louis Williams
              Reporter:
              louis.williams Louis Williams
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: