Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-60969

ReplClientInfo::getLastOp may not be updated before being used to wait for replication in tenant migration

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.1.1, 5.2.0
    • Component/s: Replication
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Backport Requested:
      v5.1
    • Sprint:
      Server Serverless 2021-11-01, Server Serverless 2021-11-15
    • Linked BF Score:
      167

      Description

      waitUntilMigrationReachesReturnAfterReachingTimestamp updates the rejectReadsBeforeTimestamp in the recipient state doc, and then use ReplClientInfo::getLastOp()  to get the latest opTime written by this client. This opTime is later being used to wait for replication. However in some cases this opTime may not be initialized at all, so the wait for replication will return immediately. This causes the recipientSyncData command to return without replicating the state doc updates to majority nodes, which is not correct because the donor might incorrectly think the migration is finished.

      One scenario for this to happen:

      1. Recipient reaches consistent state
      2. Donor sends recipientSyncData command with returnAfterReachingTimestamp to recipient
      1. Donor starts failing over and the new donor primary sends another recipientSyncData command with the same returnAfterReachingTimestamp
      2. The first recipientSyncData command updates the rejectReadsBeforeTimestamp in the state doc, but fails to wait for replication due to being stepped down
      3. The second recipientSyncData also starts updating the rejectReadsBeforeTimestamp, but since the state doc was already updated in memory with the same rejectReadsBeforeTimestamp, this will be a no-op and we do not write anything to the oplog. This means that the ReplClientInfo::getLastOp() will not be updated. So we end up waiting on an uninitialized opTime since no writes ever happened on this client, and the wait for replication will return immediately.

      This pattern (getting opTime from ReplClientInfo::getLastOp() and wait on that opTime) is being used in multiple places in tenant migration, so this issue may happen elsewhere.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jesse A. Jesse Jiryu Davis
              Reporter:
              wenbin.zhu Wenbin Zhu
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: