[SERVER-68361] LogTransactionOperationsForShardingHandler::commit misses transferring documents from prepared and non-prepared transactions changing a document's shard key value Created: 27/Jul/22  Updated: 29/Oct/23  Resolved: 10/Nov/22

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.2.0, 4.4.0, 5.0.0, 6.0.0, 6.1.0-rc3
Fix Version/s: 6.1.1, 4.2.24, 4.4.19, 5.0.15, 6.0.4, 6.2.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Randolph Tan Assignee: Randolph Tan
Resolution: Fixed Votes: 0
Labels: sharding-nyc-subteam1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Problem/Incident
is caused by SERVER-39940 Model a shard key update as a delete ... Closed
Related
related to SERVER-71219 Migration can miss writes from prepar... Closed
related to SERVER-70152 PeekableCursor's definition of hasNex... Closed
is related to SERVER-40791 Chunk migration clone blocks behind p... Closed
is related to SERVER-71028 MigrationChunkClonerSourceLegacy::nex... Backlog
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.1, v6.0, v5.0, v4.4, v4.2
Sprint: Sharding 2022-08-08, Sharding 2022-08-22, Sharding 2022-09-05, Sharding 2022-09-19, Sharding NYC 2022-11-14
Participants:
Linked BF Score: 113
Story Points: 5

 Description   
Original summary

Investigate getPreImageDocumentKey calls in LogTransactionOperationsForShardingHandler::commit

Original description

It is called a few times around here:
https://github.com/mongodb/mongo/blob/119e6b5979e4b1e8fcf91db711673393c56b4020/src/mongo/db/s/migration_chunk_cloner_source_legacy.cpp#L240

But there appears to be no code setting the value.



 Comments   
Comment by Githook User [ 21/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership

(cherry picked from commit e9858528d75b8b56211799c982f836b21912e780)
Branch: v4.2
https://github.com/mongodb/mongo/commit/73bb4a80add2bd900f89025b231e54ed18167f85

Comment by Githook User [ 18/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership

(cherry picked from commit 051e9310e027a52591ae9100078c24efafebad47)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e9858528d75b8b56211799c982f836b21912e780

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership

(cherry picked from commit e8ee517f32043af220c8db1a739c2d80b2cb0969)
Branch: v5.0
https://github.com/mongodb/mongo/commit/051e9310e027a52591ae9100078c24efafebad47

Comment by Githook User [ 14/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership

(cherry picked from commit 2061d2244caf64e2fee1b42418cd3a557d028e8c)
Branch: v6.1
https://github.com/mongodb/mongo/commit/d7ae6e64f16f2b9951b2310c4ae10eaf8e594ba4

Comment by Githook User [ 14/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership

(cherry picked from commit 2061d2244caf64e2fee1b42418cd3a557d028e8c)
Branch: v6.0
https://github.com/mongodb/mongo/commit/e8ee517f32043af220c8db1a739c2d80b2cb0969

Comment by Githook User [ 10/Nov/22 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-68361 Make migration properly handle cases when shard key value modification also results to changes in chunk membership
Branch: master
https://github.com/mongodb/mongo/commit/2061d2244caf64e2fee1b42418cd3a557d028e8c

Comment by Max Hirschhorn [ 27/Jul/22 ]

In addition to Randolph's observation that ReplOperation::setPreImageDocumentKey() isn't being called, it also noticed isInRange() is designed to extract the shard key value from the full document but may be called with a document key instead for op=u and op=d. This may mean the behavior for a shard key pattern using nested fields is also improper (see SERVER-56127 as a classic example for how this goes wrong).

bool isInRange(const BSONObj& obj,
               const BSONObj& min,
               const BSONObj& max,
               const ShardKeyPattern& shardKeyPattern) {
    BSONObj k = shardKeyPattern.extractShardKeyFromDoc(obj);
    return k.woCompare(min) >= 0 && k.woCompare(max) < 0;
}

Generated at Thu Feb 08 06:10:35 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.