[SERVER-35879] Add support for reconstituting transactions in their correct state from the transaction table during startup recovery Created: 28/Jun/18  Updated: 29/Oct/23  Resolved: 14/Dec/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.1.7

Type: Task Priority: Major - P3
Reporter: Gregory McKeon (Inactive) Assignee: Samyukta Lanka
Resolution: Fixed Votes: 0
Labels: prepare_durability, todo_in_code
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-35307 Serially within their own batch, seco... Closed
is depended on by SERVER-35872 Reconstruct prepared transactions on ... Closed
is depended on by SERVER-36492 Reconstruct prepared transactions at ... Closed
Related
related to SERVER-42545 Complete TODO listed in SERVER-35879 Closed
Backwards Compatibility: Fully Compatible
Sprint: Repl 2018-09-10, Repl 2018-10-08, Repl 2018-10-22, Repl 2018-11-05, Repl 2018-11-19, Repl 2018-12-03, Repl 2018-12-17, Repl 2019-01-14
Participants:

 Comments   
Comment by Githook User [ 14/Dec/18 ]

Author:

{'username': 'lankas', 'email': 'samy.lanka@mongodb.com', 'name': 'Samy Lanka'}

Message: SERVER-35879 Reconstitute prepared transactions at end of startup recovery
Branch: master
https://github.com/mongodb/mongo/commit/221c9016b79a60de0b0d86ed2e5d331e2f0bfe7a

Comment by Githook User [ 13/Dec/18 ]

Author:

{'username': 'lankas', 'email': 'samy.lanka@mongodb.com', 'name': 'Samy Lanka'}

Message: SERVER-35879 Skip invalidating sessions that have a transaction when a node becomes primary
Branch: master
https://github.com/mongodb/mongo/commit/d917a363457eadd2086b6ef4abae1ce0892aab98

Comment by Githook User [ 23/Oct/18 ]

Author:

{'name': 'Samy Lanka', 'email': 'samy.lanka@mongodb.com', 'username': 'lankas'}

Message: SERVER-35879 Deal with commitTransaction oplog entries during startup recovery
Branch: master
https://github.com/mongodb/mongo/commit/1fea1df1ee21cac90a2217f219f1ba12244fc4fa

Comment by Githook User [ 09/Oct/18 ]

Author:

{'name': 'Samy Lanka', 'email': 'samy.lanka@mongodb.com', 'username': 'lankas'}

Message: SERVER-35879 Deal with abortTransaction and prepareTransaction oplog entries during startup recovery
Branch: master
https://github.com/mongodb/mongo/commit/1d6087b0314b0201cddf837e7edcf2703464ff73

Comment by Judah Schvimer [ 03/Oct/18 ]

LGTM with 2 edits.
1) I agree with siyuan.zhou regarding abortTransaction.
2) I think we want to compare the stableTimestamp to the commitTimestamp rather than the "commit oplog entry timestamp".

Comment by Siyuan Zhou [ 03/Oct/18 ]

LGTM. For abort, I guess you want to add the logic into the abortTransaction command application.

Comment by Samyukta Lanka [ 03/Oct/18 ]

I'm planning on splitting this ticket into a few different commits to simplify the code review process:

1. Dealing with the abort and prepare oplog entries during startup recovery. Each of these will include writing the oplog entries and modifying config.transactions. My plan is to add separate functions to apply_ops.cpp similar to what was added here and call them from applyOps like here. These changes will be tested via unit tests.

2. Dealing with commit oplog entries. This will involve writing the oplog entries, modifying config.transactions and traversing the oplog for the corresponding prepare oplog entry to apply all the operations in the transaction. One option suggested by siyuan.zhou for how to apply all the operations from the transaction is to transform the prepare oplog entry into a normal applyOps entry and use the existing applyOps function. This will also require getting the stable timestamp from replication_recovery.cpp and passing it down to the applier so that we can compare the stable timestamp with the commit oplog entry timestamp to know whether we need to apply all the operations from the transaction or not. These changes will be tested via unit tests and some jstests.

3. Reconstructing the prepared transactions at the end of recovery. This will involve iterating over all entries in config.transactions and for each transaction in prepare, getting the corresponding oplog entry, checking out the session, applying the operations and then preparing the transaction. These changes will be tested mainly via jstests although I may not be able to unblacklist prepare tests from passthrough suites that involve unclean shutdowns, since those sometimes trigger rollbacks.

judah.schvimer and siyuan.zhou let me know if you have any comments on this.

Comment by Siyuan Zhou [ 11/Sep/18 ]

judah.schvimer and samy.lanka, I assume this is for startup recovery. I'm referring to this ticket when blacklisting tests in replica_sets_kill_secondaries_jscore_passthrough. After finishing this ticket, please unblacklist them.

Comment by Judah Schvimer [ 13/Aug/18 ]

I expect this will use a lot of the same machinery as SERVER-35307.

Generated at Thu Feb 08 04:41:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.