Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-59719

shardsvr{Commit, Abort}ReshardCollection may return unrecoverable error on stepdown, leading to fassert() on config server

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 5.0.2
    • 5.2.0, 5.0.5, 5.1.1
    • Sharding
    • Fully Compatible
    • ALL
    • v5.1, v5.0
    • Sharding 2021-09-06, Sharding 2021-10-04, Sharding 2021-10-18, Sharding 2021-11-01, Sharding 2021-11-15
    • 151
    • 1

    Description

      Background

      It is possible that if a stepdown occurs while the resharding operation is in progress, that the opCtx doing the commit will be killed before the opCtx handling the command does. Which means, for instance,  that the ShardsvrCommitReshardCollectionCommand could reach the final uassert even though the Resharding(Recipient/Donor)Service was not able to finish committing (because it was interrupted).

       
      The config server primary won't retry on the error returned by the shard and will lead the config server primary to fassert in the ReshardingCoordinatorService.

      Problem

      • The lock-free reads made it such that reads could happen concurrent to any stepdowns that were in progress.
      • This means that database reads that were reliant on _alwaysInterruptAtStepDownOrUp to verify that there wasn't a stepdown in progress such as ShardsvrCommitReshardCollectionCommand no longer work
      • This is because even if is _alwaysInterruptAtStepDownOrUp is set to true, the RSTL lock won't be acquired for a database read. Which means that the database read can complete before the opCtx would eventually be interrupted.
      • Before lock-free reads the database read would wait for the in-progress stepdown to complete and hence wouldwait for the opCtx to have been interrupted by the actively running stepdown. We want to replicate this behavior for our fix.
      • The current uassert being returned will not lead to the ReshardingCoordinatorService retrying the commit/abort command until completion. Instead it will lead to a fatal assertion.

      Proposed Solution

      Do a no-op write using doNoopWrite before performing the sanity check to assure that the state document has been deleted. This will make sure that the operation hasn't been interrupted before asserting that there are no state documents left.

      Attachments

        Issue Links

          Activity

            People

              brett.nawrocki@mongodb.com Brett Nawrocki
              luis.osta@mongodb.com Luis Osta (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: