[SERVER-82353] Multi-document transactions can miss documents when movePrimary runs concurrently Created: 20/Oct/23  Updated: 08/Feb/24  Resolved: 23/Jan/24

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.0, 4.4.0, 5.0.0, 6.0.0, 7.0.0
Fix Version/s: 7.2.1, 7.3.0-rc0, 7.0.6, 6.0.14

Type: Bug Priority: Major - P3
Reporter: Jordi Serra Torrens Assignee: Jordi Serra Torrens
Resolution: Fixed Votes: 0
Labels: car-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 0001-Repro-SERVER-82353-b.patch     Text File 0001-Repro-SERVER-82353.patch    
Issue Links:
Backports
Depends
Problem/Incident
Related
related to SERVER-77506 Sharded multi-document transactions c... Closed
Assigned Teams:
Catalog and Routing
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.2, v7.0, v6.0, v5.0, v4.4
Sprint: CAR Team 2023-11-13, CAR Team 2023-11-27, CAR Team 2023-12-11, CAR Team 2023-12-25, CAR Team 2024-01-08, CAR Team 2024-01-22, CAR Team 2024-02-05
Participants:
Linked BF Score: 101
Story Points: 3

 Description   

Consider a multi-document transaction with readConcern=snapshot (without atClusterTIme provided by the client) involving an unsharded collection, and the following interleaving:

1. Mongos chooses the 'atClusterTime' at which the transaction will run. Let's say it choses TS100.
2. Concurrently, a movePrimary executes. The recipient finishes cloning documents at TS200, and the operation commits at TS210.
3. MovePrimary finishes and mongos becomes aware of the new db-primary shard.
4. Now mongos proceed with routing the transaction statement to the new primary, but with atClusterTime=TS100.
5. On the shard, the databaseVersion check will pass, but the transaction will execute with a data snapshot @TS100, so it won't see the documents.

This can cause reads to not see the expected data, and writes to not modify the expected documents.

Edit: A similar bug can occur with readConcerns other than snapshot. For instance, consider initially shard1 owns dbA, and shard2 owns dbB:
1. Mongos targets a first transaction statement for dbA to shard1. This opens a snapshot at T100 on that shard.
2. MovePrimary moves dbB to shard1, which commits at T200.
3. Mongos targets a second statement for dbB to shard1. DatabaseVersion check passes, but the snapshot used by the transaction on shard1 does not contain the expected data for dbB.



 Comments   
Comment by Githook User [ 08/Feb/24 ]

Author:

{'name': 'Jordi Serra Torrens', 'email': 'jordi.serra-torrens@mongodb.com', 'username': 'jordist'}

Message: SERVER-82353 Detect placement conflicts in multi-document transactions due to move primary (#17826)

(cherry picked from commit 7c91dbccd67012a190c44ad801255c64eb54885d)
Branch: v7.2
https://github.com/mongodb/mongo/commit/f003cf74df75f5420acecbdc44157a50cd6c1db6

Comment by Githook User [ 01/Feb/24 ]

Author:

{'name': 'Jordi Serra Torrens', 'email': 'jordist@users.noreply.github.com', 'username': 'jordist'}

Message: SERVER-82353 Detect placement conflicts in multi-document transactions due to move primary (#18271)

GitOrigin-RevId: 21fe3bc7470f7da95de7d589db9e328e63bf829b
Branch: v6.0
https://github.com/mongodb/mongo/commit/ef59fc55dc476e36f44b10d77e6af43bcd237249

Comment by Githook User [ 23/Jan/24 ]

Author:

{'name': 'Jordi Serra Torrens', 'email': 'jordi.serra-torrens@mongodb.com', 'username': 'jordist'}

Message: SERVER-82353 Detect placement conflicts in multi-document transactions due to move primary (#17826)

(cherry picked from commit 7c91dbccd67012a190c44ad801255c64eb54885d)

GitOrigin-RevId: f182850a582dfa472e8d6aa83396dbe7391a33d9
Branch: v7.0
https://github.com/mongodb/mongo/commit/eba66bd4d7b4185e7af2bccb9df3966dda45faf2

Comment by Githook User [ 23/Jan/24 ]

Author:

{'name': 'Jordi Serra Torrens', 'email': 'jordist@users.noreply.github.com', 'username': 'jordist'}

Message: SERVER-82353 Detect placement conflicts in multi-document transactions due to move primary (#17826)

GitOrigin-RevId: 7c91dbccd67012a190c44ad801255c64eb54885d
Branch: master
https://github.com/mongodb/mongo/commit/b347171d4ed07234a9ec54902ad0132564c54443

Comment by Jordi Serra Torrens [ 20/Oct/23 ]

SERVER-77506 (more specifically, 4086290c7cee228b9bf53ec0ecc6c7bc48f7e65b) causes this to occur for transactions without an explicit atClusterTime passed by the client. For this reason, the attached repro does not reproduce on versions earlier than v7.0. On all versions, I think this bug can also happen when the client passed an explicit atClusterTime.

Generated at Thu Feb 08 06:48:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.