[SERVER-56127] Retryable update may execute more than once if chunk is migrated and shard key pattern uses nested fields Created: 15/Apr/21  Updated: 29/Oct/23  Resolved: 14/Oct/21

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.2.0, 4.4.0, 5.0.0, 5.1.0-rc0
Fix Version/s: 5.2.0, 5.1.2, 5.0.6, 4.2.23, 4.4.17

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Bobby Morck (Inactive)
Resolution: Fixed Votes: 0
Labels: sharding-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
is caused by SERVER-31031 Don't send oplog entries that are unr... Closed
Related
is related to SERVER-41074 Don't migrate prePostImageDoc session... Closed
is related to SERVER-55111 When using a nested shard key, a dele... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.1, v5.0, v4.4, v4.2
Steps To Reproduce:

python buildscripts/resmoke.py run --suite=sharding repro_retryable_update_multiple_execution_dotted_sk.js

repro_retryable_update_multiple_execution_dotted_sk.js

(function() {
"use strict";
 
load("jstests/sharding/libs/create_sharded_collection_util.js");
 
const st = new ShardingTest({mongos: 1, config: 1, shards: 2, rs: {nodes: 1}});
 
const db = st.s.getDB("test");
const collection = db.getCollection("mycoll");
CreateShardedCollectionUtil.shardCollectionWithChunks(collection, {"x.y": 1}, [
    {min: {"x.y": MinKey}, max: {"x.y": 0}, shard: st.shard0.shardName},
    {min: {"x.y": 0}, max: {"x.y": 10}, shard: st.shard0.shardName},
    {min: {"x.y": 10}, max: {"x.y": 20}, shard: st.shard1.shardName},
    {min: {"x.y": 20}, max: {"x.y": MaxKey}, shard: st.shard1.shardName},
]);
 
assert.commandWorked(collection.insert({_id: 0, x: {y: 5}, counter: 0}));
 
const session = st.s.startSession({causalConsistency: false, retryWrites: false});
const sessionCollection = session.getDatabase(db.getName()).getCollection(collection.getName());
 
const updateCmd = {
    updates: [{q: {"x.y": 5, _id: 0}, u: {$inc: {counter: 1}}}],
    txnNumber: NumberLong(0),
};
 
const firstRes = assert.commandWorked(sessionCollection.runCommand("update", updateCmd));
assert.eq({n: firstRes.n, nModified: firstRes.nModified}, {n: 1, nModified: 1});
 
assert.commandWorked(db.adminCommand(
    {moveChunk: collection.getFullName(), find: {"x.y": 5}, to: st.shard1.shardName}));
 
const secondRes = assert.commandWorked(sessionCollection.runCommand("update", updateCmd));
print(`secondRes: ${tojsononeline(secondRes)}`);
assert.eq(collection.findOne({_id: 0}), {_id: 0, x: {y: 5}, counter: 1});
 
st.stop();
})();

Sprint: QE 2021-10-04, QE 2021-10-18
Participants:
Linked BF Score: 165

 Description   

Donor shards filter oplog entries relevant for session migration as part of chunk migration by extracting the shard key value from the oplog entry. In particular, it attempts to extract the shard key value from the 'o' field for op='i' insert oplog entries, from the 'o2' field for op='u' update oplog entries, and from the 'o' field for op='d' delete oplog entries. However, the shard key value has already been extracted from the document for op='u' and op='d' oplog entries as part of generating the oplog entries. For example, with a shard key pattern {"x.y": 1}, the resulting oplog entry would contain {o2: {"x.y": 5, _id: 0}} for an update and {o: {"x.y": 5, _id: 0}} for a delete. Attempting to extract the shard key value from those 'o2' and 'o' objects with ShardKeyPattern::extractShardKeyFromDoc() would result in a shard key value {"x.y": null} and lead the donor shard to incorrectly conclude the oplog entries isn't relevant for the chunk actively being migrated. This causes the recipient shard to not know the statement(s) from that oplog entry have already executed and therefore allows them to execute for a second time after the chunk migration commits.

if (nextOplog->isCrudOpType()) {
    auto shardKey =
        _keyPattern.extractShardKeyFromDoc(nextOplog->getObjectContainingDocumentKey());
    if (!_chunkRange.containsKey(shardKey)) {
        continue;
    }
}

[js_test:repro_retryable_update_multiple_execution_dotted_sk] uncaught exception: Error: [{ "_id" : 0, "x" : { "y" : 5 }, "counter" : 2 }] != [{ "_id" : 0, "x" : { "y" : 5 }, "counter" : 1 }] are not equal :



 Comments   
Comment by Githook User [ 22/Dec/21 ]

Author:

{'name': 'Bobby Morck', 'email': 'bobby.morck@mongodb.com', 'username': 'bmorck'}

Message: SERVER-56127 Fixing retryable writes on update and delete commands to not execute more than once

(cherry picked from commit 6d8290297b563121037f8e9a9f2d37ec45ddb4bf)
Branch: v5.1
https://github.com/mongodb/mongo/commit/c58f8b2a02c5c6d43017ad8cf4afe8c25b7c960d

Comment by Githook User [ 22/Dec/21 ]

Author:

{'name': 'Bobby Morck', 'email': 'bobby.morck@mongodb.com', 'username': 'bmorck'}

Message: SERVER-56127 Fixing retryable writes on update and delete commands to not execute more than once

(cherry picked from commit 6d8290297b563121037f8e9a9f2d37ec45ddb4bf)
Branch: v5.0
https://github.com/mongodb/mongo/commit/c52c37bb2ac332cde41047ce1f8c16447b893361

Comment by Githook User [ 14/Oct/21 ]

Author:

{'name': 'Bobby Morck', 'email': 'bobby.morck@mongodb.com', 'username': 'bmorck'}

Message: SERVER-56127 Fixing retryable writes on update and delete commands to not execute more than once
Branch: master
https://github.com/mongodb/mongo/commit/6d8290297b563121037f8e9a9f2d37ec45ddb4bf

Comment by Githook User [ 13/Oct/21 ]

Author:

{'name': 'Ethan Zhang', 'email': 'ethan.zhang@mongodb.com', 'username': 'yzhang1991'}

Message: Revert "SERVER-56127 Fixing retryable writes on update and delete commands to not execute more than once"

This reverts commit fedd7fa7eaf29751d3573ff39f18c3f09abbf06c.
Branch: master
https://github.com/mongodb/mongo/commit/a0453fb2b7c8891b0088de7543137bc7ad37d902

Comment by Ethan Zhang (Inactive) [ 13/Oct/21 ]

Reopening this issue because it caused a BF and the original commit is being reverted.

Comment by Githook User [ 08/Oct/21 ]

Author:

{'name': 'Bobby Morck', 'email': 'bobby.morck@mongodb.com', 'username': 'bmorck'}

Message: SERVER-56127 Fixing retryable writes on update and delete commands to not execute more than once
Branch: master
https://github.com/mongodb/mongo/commit/fedd7fa7eaf29751d3573ff39f18c3f09abbf06c

Comment by Max Hirschhorn [ 06/Oct/21 ]

Max Hirschhorn Am I correct that we need to backport this all the way back to 4.2?

Yes, that's correct.

Comment by Ethan Zhang (Inactive) [ 06/Oct/21 ]

max.hirschhorn Am I correct that we need to backport this all the way back to 4.2?

Generated at Thu Feb 08 05:38:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.