Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-46669

moveChunk may succeed but not respect waitForDelete=true if replica set shard primary steps down

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Won't Do
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.4.0, 3.6.0, 4.0.0, 4.2.0
    • Component/s: Sharding
    • Labels:
      None
    • Sharding EMEA
    • ALL
    • Hide

      Apply server42192.patch to allow the moveChunk command to be automatically retried in the presence of failovers and run the agg_with_chunk_migrations.js FSM workload. The --repeat is necessary because while this concurrency test reproduces the issue often, it doesn't happen every time.

      python buildscripts/resmoke.py --suite=concurrency_sharded_multi_stmt_txn_terminate_primary jstests/concurrency/fsm_workloads/agg_with_chunk_migrations.js --repeat=10
      Show
      Apply server42192.patch to allow the moveChunk command to be automatically retried in the presence of failovers and run the agg_with_chunk_migrations.js FSM workload. The --repeat is necessary because while this concurrency test reproduces the issue often, it doesn't happen every time. python buildscripts/resmoke.py --suite=concurrency_sharded_multi_stmt_txn_terminate_primary jstests/concurrency/fsm_workloads/agg_with_chunk_migrations.js --repeat=10

      The changes from cc8e8a1 as part of SERVER-26307 made it so a BalancerInterrupted error response is no longer returned when the moveChunk command fails due to a retryable error on the replica set shard primary. Additionally, the changes from 53efde3 as part of SERVER-25999 made it so an OperationFailed error status would be returned by MigrationManager::_processRemoteCommandResponse(); however, any non-BalancerInterrupted error status is converted to an ok=1 response so long as the chunk has successfully been migrated. It does not check if _waitForDelete=true had been specified in the moveChunk command request to realize that we may not have waited long enough for the range to be cleaned up.

      We should either (a) wait long enough, or (b) preserve the OperationFailed error response as a way to inform the user.

      Status commandStatus = _processRemoteCommandResponse(
    remoteCommandResponse, &statusWithScopedMigrationRequest.getValue());

// Migration calls can be interrupted after the metadata is committed but before the command
// finishes the waitForDelete stage. Any failovers, therefore, must always cause the moveChunk
// command to be retried so as to assure that the waitForDelete promise of a successful command
// has been fulfilled.
>if (chunk->getShardId() == migrateInfo.to && commandStatus != ErrorCodes::BalancerInterrupted) {
    return Status::OK();
}

        1. server42192.patch
          29 kB

            Assignee:
            backlog-server-sharding-emea [DO NOT USE] Backlog - Sharding EMEA
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: