[SERVER-39017] Allow prepared transaction statements to persist in-memory until commit Created: 15/Jan/19  Updated: 29/Oct/23  Resolved: 01/Feb/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 4.1.6
Fix Version/s: 4.1.8

Type: Task Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-38284 Remove donor collection X-lock acquis... Closed
is depended on by SERVER-39021 Switch migrations to observe multi-st... Closed
Related
related to SERVER-69478 remove unnecessary copy in Transactio... Closed
related to SERVER-69560 change return type of TransactionPart... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2019-01-28, Sharding 2019-02-11
Participants:

 Description   

Problem Summary

We would like to be able to observe all statements on transaction commit. This is so that migrations may take these statements on commit and add them to any current transferMods queue.

Currently, for prepared transactions, we discard statements on prepare, and don't persist these statements until commit time.

Proposed Approach

We will have to make changes to the transaction participant in order to persist these statements.

  • A new method retrieveOperations() will simply retrieve and return a reference to _transactionOperations. This method will require that the session is checked out, and that it is only called after the transaction has been prepared.
  • A new method endTransactionAndClearOperationsInMemory() will clear the _transactionOperations variable and set _transactionOperationBytes to zero.
  • The current usage of the method endTransactionAndRetrieveOperations() will be replaced with retrieveOperations() when we want the operations, combined with endTransactionAndClearOperationsInMemory() when we want to clear the transaction memory on commit.
  • Prepared transactions will no longer call endTransactionAndRetrieveOperations() on prepare. They will call both retrieveOperations() and endTransactionAndClearOperationsInMemory() on commit.
  • Non-prepared transactions will continue to persist the previous behavior of the previous method, calling both retrieveOperationsFor() and endTransactionAndClearOperationsInMemory() on commit.

NOTE: We have renamed retrieveOperationsForMigrate() to retrieveOperations(), because we have identified usage of said operations outside of migrate, including here.



 Comments   
Comment by Githook User [ 01/Feb/19 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-39017 Allow prepared transaction statements to persist in-memory until commit
Branch: master
https://github.com/mongodb/mongo/commit/b8bfe9ff391ebeb10a5c2fb86979d854d17d0fd5

Comment by Blake Oler [ 17/Jan/19 ]

A note: the transactions team decided to call the onCommit opObserver while an unprepared transaction would still be inProgress. This is so that any exceptions inside the opObserver would allow the transaction to abort instead of committing. As such, we will maintain the current invariant of either inProgress or Prepared for both functions. This will result in no change to current behavior for the prerequisite states.

Comment by Judah Schvimer [ 16/Jan/19 ]

lgtm, I defer to Kal on the lifecycle management.

Comment by Kaloian Manassiev [ 15/Jan/19 ]

Yeah, I think retrieveCompletedTransactionOperations might be a more appropriate name.

I would make retrieveOperations() invariant that the transaction's status is InPrepare or Committed so that callers are not allowed to obtain an intermediate view of the transaction.

Comment by Blake Oler [ 15/Jan/19 ]

kaloian.manassiev

  1. The method retrieveOperations() will have the same requirements as endTransactionAndRetrieveOperations(). That is – a session must be checked out. We can see this requirement at the onTransactionCommit() handler here. And as a result of the prepared transaction lifecycle, a commit must be after the participant has entered the prepared state, implying that the list of operations will have become static.
  2. As far as I am aware, we don't need to return a copy. A reference should just work. I will defer to using a reference unless something indicating otherwise comes up during implementation of this ticket.

The above information will be integrated into the ticket description.

kaloian.manassiev do we want to rename the method retrieveOperations() to indicate that the operations are now static? Maybe retrieveCompletedTransactionOperations().

Comment by Kaloian Manassiev [ 15/Jan/19 ]

I have a couple of questions:

A new method retrieveOperations() will simply retrieve and return a copy of _transactionOperations.

What are the lifetime requirements for this method? I.e., it is allowed to only be called with the session checked-out, and only after the transaction participant enters the prepared state, right?

With the lifetime above - does it need to return a copy or just reference would work? I think you can just return a reference. The reason why this is better is that this object can be up to 16MB in size.

Comment by Blake Oler [ 15/Jan/19 ]

judah.schvimer can I get your LGTM on this approach? It has integrated your comments on the ticket SERVER-38284.

Generated at Thu Feb 08 04:50:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.