[SERVER-72502] Implement oplog batcher changes for parallelizing prepared transaction application Created: 04/Jan/23  Updated: 29/Oct/23  Resolved: 05/Apr/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.0.0-rc0

Type: Task Priority: Major - P3
Reporter: Wenbin Zhu Assignee: Wenbin Zhu
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-72499 Make SplitPrepareSessionManager acces... Closed
Related
related to SERVER-71938 Avoid applying commitTransaction entr... Closed
Assigned Teams:
Replication
Backwards Compatibility: Fully Compatible
Sprint: Repl 2023-01-23, Repl 2023-02-20, Repl 2023-04-03, Repl 2023-04-17
Participants:

 Description   

This should change OplogBatcher::mustProcessIndividually so that prepare entries will no longer be required to be in their own batch. The commitTransaction/abortTransaction command will no longer be required to be in its own batch, if and only if that commitTransaction/abortTransaction refers to a session in the SplitPrepareSessionManager.

Note that currently OplogBatcher::mustProcessIndividually is the same for both secondary mode and initial sync/recovering mode. We should probably do the above only for secondary mode and keep the initial sync/recovering behavior.

 

Note: We decided to still have every commitTransaction/abortTransaction entry to be processed in its own batch. This is because:

1) Since we currently disallow a session to be split if it was already split and not yet released. This could be relaxed so that multiple splits on the same session are allowed as long as they refer to different txnNumbers, but before we do that, the following case can happen if we allow commits/aborts to be in the same batch with others:

  • batch 1: [prepare(sess1, txn1)]
  • batch 2: [commit(sess1, txn1), prepare(sess1, txn2)] 

When trying to split the second prepare, it fails due to this limitation since we only release the split sessions after the txn is committed and the batch is finished.

2) Allowing commits/aborts to be in the same batch with other ops makes it possible for more edge cases and requires more testings. And since applying a commitTransaction/abortTransaction entry is usually quick, we think it's fine to first disallow this and once we're confident that the new algorithm works fine, we can start an effort to optimize it.



 Comments   
Comment by Githook User [ 05/Apr/23 ]

Author:

{'name': 'Wenbin Zhu', 'email': 'wenbin.zhu@mongodb.com', 'username': 'WenbinZhu'}

Message: SERVER-72502 Implement OplogBatcher changes for parallelizing prepared transaction oplog application.
Branch: master
https://github.com/mongodb/mongo/commit/75228bcd11ef005a55816d76e6c336448a9e1931

Comment by Wenbin Zhu [ 11/Jan/23 ]

We should be careful with the isEndOfLargeTransaction() check in mustProcessIndividually(). This check evaluates to true for a prepareTransaction entry of  a large prepared transaction, which would make every prepareTransaction entry of  a large prepared transaction to be processed in its own batch. Our goal is to apply prepareTransaction entries concurrently with other entries, but it might be fine for prepareTransaction of  a large prepared transaction to still be in its own batch, since we already parallelize application within a prepared transaction, and a large transaction means that we are applying more than 16MB worth of data, so it's fine to cut the batch.

Generated at Thu Feb 08 06:21:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.