Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-65821

Deadlock during setFCV when there are prepared transactions that have not persisted commit/abort decision

    XMLWordPrintableJSON

Details

    • Fully Compatible
    • ALL
    • v6.0, v5.3, v5.0, v4.4
    • Execution Team 2022-05-02, Execution Team 2022-05-16
    • 169

    Description

      Here are the steps to reproduce the deadlock:

      • Run a cross-shard transaction with two participant shards, shard0 and shard1 where shard0 is the coordinator shard. Pause the TransactionCoordinator thread right before the commit decision is written (i.e. after the transaction has entered the "prepared" state).
      • Run a setFCV command against shard0. Wait until the setFCV thread is blocked waiting to acquire the global S lock (i.e. waiting for prepared transactions that existed before the FCV change to commit or abort).
      • Unpause the TransactionCoordinator thread. The transaction cannot commit since the TransactionCoordinator is blocked waiting to acquire the IX lock for the config.transaction_coordinators collection to write the commit decision.
      • Both the setFCV thread and TransactionCoordinator thread now hang.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. SERVER-75205 Deadlock between stepdown and restoring locks after yielding when all read tickets exhausted

          • Blocker - P1 - Blocks development and/or testing work, production could not run.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-60682 TransactionCoordinator may block acquiring WiredTiger write ticket to persist its decision, prolonging transactions being in the prepared state

          • Critical - P2 - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. SERVER-57476 Operation may block on prepare conflict while holding oplog slot, stalling replication indefinitely

          • Major - P3 - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-66340 Improve distributed transaction commit locking behavior

          • Major - P3 - Major loss of function.
          • Backlog

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-66341 Improve journal flusher locking behavior

          • Major - P3 - Major loss of function.
          • Backlog

          Improvement - An improvement or enhancement to an existing feature or task. SERVER-66342 Remove resourceIdFeatureCompatibilityVersion

          • Major - P3 - Major loss of function.
          • Backlog
          related to

          Bug - A problem which impairs or prevents the functions of the product. SERVER-66719 dbCheck FCV lock upgrade causes deadlock with setFCV

          • Major - P3 - Major loss of function.
          • Closed

          Task - A task that needs to be done. SERVER-66213 setFCV may need to wait for transactionLifetimeLimitSeconds

          • Major - P3 - Major loss of function.
          • Open

          Activity

            People

              gregory.noma@mongodb.com Gregory Noma
              cheahuychou.mao@mongodb.com Cheahuychou Mao
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: