Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-60774

Resharding may apply through reshardFinalOp without transitioning to strict consistency, stalling write operations on collection being resharded until critical section times out

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • 5.1.0-rc0
    • 5.2.0, 5.0.4, 5.1.0-rc2
    • Sharding
    • None
    • Fully Compatible
    • ALL
    • v5.1, v5.0
    • Sharding 2021-11-01
    • 1

    Description

      Typically the ReshardingOplogFetcher stops fetching new donor oplog entries after it has retrieved the reshardFinalOp entry. However, upon resuming from a primary failover, the ReshardingOplogFetcher won't realize it has fetched the reshardFinalOp entry already and it'll continue to retrieve empty batches. (The donor shard won't write any new oplog entries destined for the recipient shard after the reshardFinalOp.) This leads the ReshardingOplogFetcher to insert no-op reshardProgressMark entries after the reshardFinalOp intos the recipient shard's local oplog buffer collection.

      The ReshardingDonorOplogIterator assumes the reshardFinalOp will be the last entry in a batch. The presence of these extra no-op reshardProgressMark entries after the reshardFinalOp will prevent it and the ReshardingOplogApplier from realizing the reshardFinalOp has been applied through. The recipient shard therefore never reaches the "strict-consistency" state. This leads the overall resharding operation to fail with a ReshardingCriticalSectionTimeout error response after writes been blocked for the collection being resharded for reshardingCriticalSectionTimeoutMillis (5 seconds by default).

      if (!batch.empty()) {
          const auto& lastEntryInBatch = batch.back();
          _resumeToken = getId(lastEntryInBatch);
       
          if (isFinalOplog(lastEntryInBatch)) {
              _hasSeenFinalOplogEntry = true;
              // Skip returning the final oplog entry because it is known to be a no-op.
              batch.pop_back();
      

      Attachments

        Issue Links

          Activity

            People

              max.hirschhorn@mongodb.com Max Hirschhorn
              max.hirschhorn@mongodb.com Max Hirschhorn
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: