[SERVER-54047] Resharding permits transactions to commit after minFetchTimestamp without having destined recipient filled in Created: 26/Jan/21  Updated: 29/Oct/23  Resolved: 02/Feb/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.9.0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Randolph Tan
Resolution: Fixed Votes: 0
Labels: PM-234-M2.5, PM-234-T-lifecycle
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-54094 Avoid taking strong lock on collectio... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

python buildscripts/resmoke.py run --suite=sharding repro_resharding_txn_missing_destined_recipient.js

repro_resharding_txn_missing_destined_recipient.js

(function() {
"use strict";
 
load("jstests/sharding/libs/resharding_test_fixture.js");
 
const reshardingTest = new ReshardingTest({numDonors: 2, numRecipients: 2, reshardInPlace: true});
reshardingTest.setup();
 
const donorShardNames = reshardingTest.donorShardNames;
const sourceCollection = reshardingTest.createShardedCollection({
    ns: "reshardingDb.coll",
    shardKeyPattern: {oldKey: 1},
    chunks: [
        {min: {oldKey: MinKey}, max: {oldKey: 10}, shard: donorShardNames[0]},
        {min: {oldKey: 10}, max: {oldKey: MaxKey}, shard: donorShardNames[1]},
    ],
});
 
const mongos = sourceCollection.getMongo();
const session = mongos.startSession({causalConsistency: false, retryWrites: false});
const sessionCollection = session.getDatabase(sourceCollection.getDB().getName())
                              .getCollection(sourceCollection.getName());
 
assert.commandWorked(sessionCollection.insert({_id: 0, oldKey: 5, newKey: 15, counter: 0}));
 
session.startTransaction();
assert.commandWorked(sessionCollection.update({_id: 0, oldKey: 5}, {$inc: {counter: 1}}));
 
const recipientShardNames = reshardingTest.recipientShardNames;
reshardingTest.withReshardingInBackground(  //
    {
        newShardKeyPattern: {newKey: 1},
        newChunks: [
            {min: {newKey: MinKey}, max: {newKey: 10}, shard: recipientShardNames[0]},
            {min: {newKey: 10}, max: {newKey: MaxKey}, shard: recipientShardNames[1]},
        ],
    },
    () => {
        assert.soon(() => {
            const coordinatorDoc = mongos.getCollection("config.reshardingOperations").findOne({
                nss: sourceCollection.getFullName()
            });
 
            return coordinatorDoc !== null && coordinatorDoc.fetchTimestamp !== undefined;
        });
 
        assert.commandWorked(session.commitTransaction_forTesting());
    });
 
reshardingTest.teardown();
})();

Sprint: Sharding 2021-02-08
Participants:
Story Points: 1

 Description   

This leads the writes from the multi-document transaction to not be applied by the recipient shard(s).

[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000 Error: [{
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000  "docsWithDifferentContents" : [
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000          {
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                  "original" : {
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "_id" : 0,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "oldKey" : 5,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "newKey" : 15,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "counter" : 1
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                  },
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                  "resharded" : {
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "_id" : 0,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "oldKey" : 5,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "newKey" : 15,
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.584+0000                          "counter" : 0
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000                  }
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000          }
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  ],
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  "docsExtraAfterResharding" : [ ],
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  "docsMissingAfterResharding" : [ ]
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000 }] != [{
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  "docsWithDifferentContents" : [ ],
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  "docsExtraAfterResharding" : [ ],
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000  "docsMissingAfterResharding" : [ ]
[js_test:repro_resharding_txn_missing_destined_recipient] 2021-01-26T12:22:44.585+0000 }] are not equal :



 Comments   
Comment by Githook User [ 02/Feb/21 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-54047 Resharding permits transactions to commit after minFetchTimestamp without having destined recipient filled in
Branch: master
https://github.com/mongodb/mongo/commit/4941c446e9091caa1f7151bd874c8d5b72c39bb8

Comment by Max Hirschhorn [ 28/Jan/21 ]

I discussed the approach of taking a collection S lock to wait for all earlier writes on the collection being resharding to have finished with Geert and Andy. We agreed it is safe and correct but not ideal for its impact on operations on the collection being resharded. I have also filed SERVER-54094 as an improvement to avoid taking the collection S lock as part of choosing the minFetchTimestamp. Geert had brought up index builds taking a collection X lock by comparison and so a user has the choice to not run reshardCollection if they cannot tolerate the temporary stall in operations on the collection being resharded.

It was also clarified on Slack that it is safe and correct to

  1. Acquire RSTL lock in MODE_IX
  2. Acquire global lock in MODE_IX
  3. Acquire database lock in MODE_IX
  4. Acquire collection lock in S
  5. Write an oplog entry.

AutoGetDb db(opCtx.get(), donorDoc.getNss().db(), MODE_IX);
Lock::CollectionLock collLock(opCtx.get(), donorDoc.getNss(), MODE_S);

and

AutoGetOplog oplogWrite(opCtx.get(), OplogAccessMode::kWrite);
AutoGetCollection coll(opCtx.get(), donorDoc.getNss(), MODE_S);

are two ways of expressing this concept. The order in the second way is very important; it is only safe (i.e. not prone to deadlock) to acquire a resource in MODE_IS while already holding the resource in MODE_IX, and it not safe to acquire a resource in MODE_IX while already holding the resource in MODE_IS. Also note that writing an oplog entry only requires the global IX lock.

Comment by Max Hirschhorn [ 26/Jan/21 ]

The resharding design document had called for acquiring a MODE_S lock on the existing sharded collection namespace while the minFetchTimestamp value is being generated. The implementation doesn't appear to be doing that. CC blake.oler

We likely face a similar issue to the one reported in this ticket with a multi-document transaction committing after the final oplog entries have been generated.

Generated at Thu Feb 08 05:32:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.