Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-54020

ShardInvalidatedForTargeting thrown by resharding's getDestinedRecipient() not being retried by mongos

    XMLWordPrintable

    Details

    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      Apply the following patch to delay when the donor shard performs its refresh of the temporary resharding collection.

      python buildscripts/resmoke.py run --suite=sharding repro_resharding_write_before_temp_ns_refresh.js
      

      diff --git a/repro_resharding_write_before_temp_ns_refresh.js b/repro_resharding_write_before_temp_ns_refresh.js
      new file mode 100644
      index 0000000000..7012fae165
      --- /dev/null
      +++ b/repro_resharding_write_before_temp_ns_refresh.js
      @@ -0,0 +1,33 @@
      +(function() {
      +"use strict";
      +
      +load("jstests/sharding/libs/resharding_test_fixture.js");
      +
      +const reshardingTest = new ReshardingTest({numDonors: 2, numRecipients: 2, reshardInPlace: true});
      +reshardingTest.setup();
      +
      +const donorShardNames = reshardingTest.donorShardNames;
      +const sourceCollection = reshardingTest.createShardedCollection({
      +    ns: "reshardingDb.coll",
      +    shardKeyPattern: {oldKey: 1},
      +    chunks: [
      +        {min: {oldKey: MinKey}, max: {oldKey: 10}, shard: donorShardNames[0]},
      +        {min: {oldKey: 10}, max: {oldKey: MaxKey}, shard: donorShardNames[1]},
      +    ],
      +});
      +
      +const recipientShardNames = reshardingTest.recipientShardNames;
      +reshardingTest.withReshardingInBackground(  //
      +    {
      +        newShardKeyPattern: {newKey: 1},
      +        newChunks: [
      +            {min: {newKey: MinKey}, max: {newKey: 10}, shard: recipientShardNames[0]},
      +            {min: {newKey: 10}, max: {newKey: MaxKey}, shard: recipientShardNames[1]},
      +        ],
      +    },
      +    () => {
      +        assert.commandWorked(sourceCollection.insert({_id: 0, oldKey: 5, newKey: 15}));
      +    });
      +
      +reshardingTest.teardown();
      +})();
      diff --git a/src/mongo/db/s/resharding/resharding_donor_service.cpp b/src/mongo/db/s/resharding/resharding_donor_service.cpp
      index 2a7200f38e..6ba5aa0973 100644
      --- a/src/mongo/db/s/resharding/resharding_donor_service.cpp
      +++ b/src/mongo/db/s/resharding/resharding_donor_service.cpp
      @@ -263,6 +263,8 @@ void ReshardingDonorService::DonorStateMachine::
                   ->assertNoIndexBuildInProgForCollection(_donorDoc.getExistingUUID());
           }
       
      +    sleepsecs(10);
      +
           // Recipient shards expect to read from the donor shard's existing sharded collection
           // and the config.cache.chunks collection of the temporary resharding collection using
           // {atClusterTime: <fetchTimestamp>}. Refreshing the temporary resharding collection on
      

      Show
      Apply the following patch to delay when the donor shard performs its refresh of the temporary resharding collection. python buildscripts/resmoke.py run --suite=sharding repro_resharding_write_before_temp_ns_refresh.js diff --git a/repro_resharding_write_before_temp_ns_refresh.js b/repro_resharding_write_before_temp_ns_refresh.js new file mode 100644 index 0000000000..7012fae165 --- /dev/null +++ b/repro_resharding_write_before_temp_ns_refresh.js @@ -0,0 +1,33 @@ +(function() { +"use strict"; + +load("jstests/sharding/libs/resharding_test_fixture.js"); + +const reshardingTest = new ReshardingTest({numDonors: 2, numRecipients: 2, reshardInPlace: true}); +reshardingTest.setup(); + +const donorShardNames = reshardingTest.donorShardNames; +const sourceCollection = reshardingTest.createShardedCollection({ + ns: "reshardingDb.coll", + shardKeyPattern: {oldKey: 1}, + chunks: [ + {min: {oldKey: MinKey}, max: {oldKey: 10}, shard: donorShardNames[0]}, + {min: {oldKey: 10}, max: {oldKey: MaxKey}, shard: donorShardNames[1]}, + ], +}); + +const recipientShardNames = reshardingTest.recipientShardNames; +reshardingTest.withReshardingInBackground( // + { + newShardKeyPattern: {newKey: 1}, + newChunks: [ + {min: {newKey: MinKey}, max: {newKey: 10}, shard: recipientShardNames[0]}, + {min: {newKey: 10}, max: {newKey: MaxKey}, shard: recipientShardNames[1]}, + ], + }, + () => { + assert.commandWorked(sourceCollection.insert({_id: 0, oldKey: 5, newKey: 15})); + }); + +reshardingTest.teardown(); +})(); diff --git a/src/mongo/db/s/resharding/resharding_donor_service.cpp b/src/mongo/db/s/resharding/resharding_donor_service.cpp index 2a7200f38e..6ba5aa0973 100644 --- a/src/mongo/db/s/resharding/resharding_donor_service.cpp +++ b/src/mongo/db/s/resharding/resharding_donor_service.cpp @@ -263,6 +263,8 @@ void ReshardingDonorService::DonorStateMachine:: ->assertNoIndexBuildInProgForCollection(_donorDoc.getExistingUUID()); }   + sleepsecs(10); + // Recipient shards expect to read from the donor shard's existing sharded collection // and the config.cache.chunks collection of the temporary resharding collection using // {atClusterTime: <fetchTimestamp>}. Refreshing the temporary resharding collection on
    • Sprint:
      Sharding 2021-03-22, Sharding 2021-04-05, Sharding EMEA 2021-05-03
    • Linked BF Score:
      39
    • Story Points:
      1

      Description

      For some reason, the ShardInvalidatedForTargeting exception is being propagated back to the client as a write error rather than being automatically retried by mongos.

      bool allowLocks = true;
      auto tempNssRoutingInfo = Grid::get(opCtx)->catalogCache()->getCollectionRoutingInfo(
          opCtx,
          constructTemporaryReshardingNss(sourceNss.db(), getCollectionUuid(opCtx, sourceNss)),
          allowLocks);
       
      uassert(ShardInvalidatedForTargetingInfo(sourceNss),
              "Routing information is not available for the temporary resharding collection.",
              tempNssRoutingInfo.getStatus() != ErrorCodes::StaleShardVersion);
       
      uassertStatusOK(tempNssRoutingInfo);
      

        Attachments

          Activity

            People

            Assignee:
            jordi.serra-torrens Jordi Serra Torrens
            Reporter:
            max.hirschhorn Max Hirschhorn
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: