[SERVER-71219] Migration can miss writes from prepared transactions Created: 09/Nov/22  Updated: 29/Oct/23  Resolved: 27/Jan/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 4.2.0, 4.4.0, 5.0.0, 6.0.0, 6.1.0-rc3
Fix Version/s: 4.4.19, 5.0.15, 6.3.0-rc0, 6.0.5

Type: Bug Priority: Critical - P2
Reporter: Randolph Tan Assignee: Randolph Tan
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
causes SERVER-78050 Chunk Migration Can Lose Data If Proc... Closed
Related
is related to SERVER-68361 LogTransactionOperationsForShardingHa... Closed
is related to SERVER-38284 Remove donor collection X-lock acquis... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.2, v6.0, v5.0, v4.4
Steps To Reproduce:

Uncomment this line in test

Sprint: Sharding NYC 2022-11-28, Sharding 2022-12-12, Sharding NYC 2022-12-26, Sharding NYC 2023-01-09, Sharding NYC 2023-01-23, Sharding NYC 2023-02-06
Participants:
Linked BF Score: 113

 Description   

Migration relies on LogTransactionOperationsForShardingHandler to capture writes that gets committed. However, the issue is that the commit handler only gets registered to the recovery unit during prepare. This means that if another node becomes the primary, no callbacks from LogTransactionOperationsForShardingHandler will get called when the transaction gets committed.



 Comments   
Comment by Githook User [ 08/Feb/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-71219 Migration can miss writes from prepared transactions

This commit does 2 things:
1. Make sure that we register the migration source op observer hook in all paths where transactions transitions into prepare.
2. If we don't have the post image doc, fetch the latest doc from storage.

(cherry picked from commit a20d97c8e30b805410d86ccdbeac2d3f05c407ba)
Branch: v4.4
https://github.com/mongodb/mongo/commit/58d5abbf8e6e809e7d3c126c74655d2553ceabc1

Comment by Githook User [ 08/Feb/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-71219 Migration can miss writes from prepared transactions

This commit does 2 things:
1. Make sure that we register the migration source op observer hook in all paths where transactions transitions into prepare.
2. If we don't have the post image doc, fetch the latest doc from storage.

(cherry picked from commit bd227ce5529b0423ca791aa926db48ec8b10c07d)
Branch: v5.0
https://github.com/mongodb/mongo/commit/a20d97c8e30b805410d86ccdbeac2d3f05c407ba

Comment by Githook User [ 03/Feb/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-71219 Migration can miss writes from prepared transactions

This commit does 2 things:
1. Make sure that we register the migration source op observer hook in all paths where transactions transitions into prepare.
2. If we don't have the post image doc, fetch the latest doc from storage.

(cherry picked from commit 2f708612dc39780410bf40d31a404cb121f653a9)
Branch: v6.0
https://github.com/mongodb/mongo/commit/bd227ce5529b0423ca791aa926db48ec8b10c07d

Comment by Githook User [ 27/Jan/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-71219 Migration can miss writes from prepared transactions

This commit does 2 things:
1. Make sure that we register the migration source op observer hook in all paths where transactions transitions into prepare.
2. If we don't have the post image doc, fetch the latest doc from storage.
Branch: master
https://github.com/mongodb/mongo/commit/2f708612dc39780410bf40d31a404cb121f653a9

Comment by Githook User [ 27/Jan/23 ]

Author:

{'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'}

Message: SERVER-71219 Migration can miss writes from prepared transactions
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/f7b83cbc0574d898f464bc664e94dbcb6d52db89

Generated at Thu Feb 08 06:18:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.