Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34878

if setShardVersion for dropCollection not sent and collection re-created, migrations for new collection can fail indefinitely

XMLWordPrintableJSON

    • Sharding EMEA
    • ALL
    • Hide

      Apply the following patch to remove the setShardVersion from dropCollection (to deterministically reproduce this issue) and run jstests/sharding/upsert_sharded.js.

      diff --git a/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp b/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp
index 16dd7c4..8c4f39f 100644
--- a/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp
+++ b/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp
@@ -420,55 +420,6 @@ Status ShardingCatalogManager::dropCollection(OperationContext* opCtx, const Nam     LOG(1) << "dropCollection " << nss.ns() << " collection marked as dropped";-    for (const auto& shardEntry : allShards) {
-        auto swShard = shardRegistry->getShard(opCtx, shardEntry.getName());
-        if (!swShard.isOK()) {
-            return swShard.getStatus();
-        }
-
-        const auto& shard = swShard.getValue();
-
-        SetShardVersionRequest ssv = SetShardVersionRequest::makeForVersioningNoPersist(
-            shardRegistry->getConfigServerConnectionString(),
-            shardEntry.getName(),
-            fassertStatusOK(28781, ConnectionString::parse(shardEntry.getHost())),
-            nss,
-            ChunkVersion::DROPPED(),
-            true);
-
-        auto ssvResult = shard->runCommandWithFixedRetryAttempts(
-            opCtx,
-            ReadPreferenceSetting{ReadPreference::PrimaryOnly},
-            "admin",
-            ssv.toBSON(),
-            Shard::RetryPolicy::kIdempotent);
-
-        if (!ssvResult.isOK()) {
-            return ssvResult.getStatus();
-        }
-
-        auto ssvStatus = std::move(ssvResult.getValue().commandStatus);
-        if (!ssvStatus.isOK()) {
-            return ssvStatus;
-        }
-
-        auto unsetShardingStatus = shard->runCommandWithFixedRetryAttempts(
-            opCtx,
-            ReadPreferenceSetting{ReadPreference::PrimaryOnly},
-            "admin",
-            BSON("unsetSharding" << 1),
-            Shard::RetryPolicy::kIdempotent);
-
-        if (!unsetShardingStatus.isOK()) {
-            return unsetShardingStatus.getStatus();
-        }
-
-        auto unsetShardingResult = std::move(unsetShardingStatus.getValue().commandStatus);
-        if (!unsetShardingResult.isOK()) {
-            return unsetShardingResult;
-        }
-    }
-
     LOG(1) << "dropCollection " << nss.ns() << " completed";     catalogClient
      Show
      Apply the following patch to remove the setShardVersion from dropCollection (to deterministically reproduce this issue) and run jstests/sharding/upsert_sharded.js. diff --git a/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp b/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp index 16dd7c4..8c4f39f 100644 --- a/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp +++ b/src/mongo/db/s/config/sharding_catalog_manager_collection_operations.cpp @@ -420,55 +420,6 @@ Status ShardingCatalogManager::dropCollection(OperationContext* opCtx, const Nam LOG(1) << "dropCollection " << nss.ns() << " collection marked as dropped" ;- for ( const auto& shardEntry : allShards) { - auto swShard = shardRegistry->getShard(opCtx, shardEntry.getName()); - if (!swShard.isOK()) { - return swShard.getStatus(); - } - - const auto& shard = swShard.getValue(); - - SetShardVersionRequest ssv = SetShardVersionRequest::makeForVersioningNoPersist( - shardRegistry->getConfigServerConnectionString(), - shardEntry.getName(), - fassertStatusOK(28781, ConnectionString::parse(shardEntry.getHost())), - nss, - ChunkVersion::DROPPED(), - true ); - - auto ssvResult = shard->runCommandWithFixedRetryAttempts( - opCtx, - ReadPreferenceSetting{ReadPreference::PrimaryOnly}, - "admin" , - ssv.toBSON(), - Shard::RetryPolicy::kIdempotent); - - if (!ssvResult.isOK()) { - return ssvResult.getStatus(); - } - - auto ssvStatus = std::move(ssvResult.getValue().commandStatus); - if (!ssvStatus.isOK()) { - return ssvStatus; - } - - auto unsetShardingStatus = shard->runCommandWithFixedRetryAttempts( - opCtx, - ReadPreferenceSetting{ReadPreference::PrimaryOnly}, - "admin" , - BSON( "unsetSharding" << 1), - Shard::RetryPolicy::kIdempotent); - - if (!unsetShardingStatus.isOK()) { - return unsetShardingStatus.getStatus(); - } - - auto unsetShardingResult = std::move(unsetShardingStatus.getValue().commandStatus); - if (!unsetShardingResult.isOK()) { - return unsetShardingResult; - } - } - LOG(1) << "dropCollection " << nss.ns() << " completed" ; catalogClient
    • 27

      Let's say the setShardVersion for dropCollection is not sent, and the collection is re-created (and re-sharded). This means the old chunks remain as garbage in the shard's config.cache.chunks.<ns>. (This is because the shard will not refresh after the drop. If it had, it would have seen the collection had been dropped and scheduled a task to drop its local config.cache.chunks.<ns> collection).

      The setShardVersion for shardCollection for the new collection will cause the shard to refresh and find the new collection entry. Since the epoch is different from the collection entry in the shard's config.cache.collections, the shard will query for chunks with version $gte 0|0|<new epoch>.

      The shard will find the chunk for the new collection and persist it in its config.cache.chunks.<ns>, alongside the garbage chunks. As part of this, the shard will attempt to delete chunks in config.cache.chunks.<ns> with ranges that "overlap" the new chunk.

      However, the old chunks may not get deleted if their shard key sorts as "less than" the new shard key (see SERVER-34856).

      . . .

      Now, at the beginning of a migration, the donor shard forces a refresh.

      The refresh queries config.chunks for chunks with version $gte the highest version of any chunk persisted on the shard in config.cache.chunks.<ns> (notice that the new collection's epoch is not included in the local query).

      If there is a garbage chunk with a major version greater than 1 (that is, at least one migration had occurred), then it will be returned by the local query and its version will be used as the $gte for the remote query. So, the remote query will find no chunks.

      The ShardServerCatalogCacheLoader will return ConflictingOperationInProgress, which will cause the CatalogCache to retry the refresh 3 times and get the same result.

      Once the retries are exhausted, the donor will fail the moveChunk command with ConflictingOperationInProgress.

            Assignee:
            backlog-server-sharding-emea [DO NOT USE] Backlog - Sharding EMEA
            Reporter:
            esha.maharishi@mongodb.com Esha Maharishi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: