[SERVER-72502] Implement oplog batcher changes for parallelizing prepared transaction application Created: 04/Jan/23 Updated: 29/Oct/23 Resolved: 05/Apr/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Wenbin Zhu | Assignee: | Wenbin Zhu |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Replication
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | Repl 2023-01-23, Repl 2023-02-20, Repl 2023-04-03, Repl 2023-04-17 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
This should change OplogBatcher::mustProcessIndividually so that prepare entries will no longer be required to be in their own batch. Note that currently OplogBatcher::mustProcessIndividually is the same for both secondary mode and initial sync/recovering mode. We should probably do the above only for secondary mode and keep the initial sync/recovering behavior.
Note: We decided to still have every commitTransaction/abortTransaction entry to be processed in its own batch. This is because: 1) Since we currently disallow a session to be split if it was already split and not yet released. This could be relaxed so that multiple splits on the same session are allowed as long as they refer to different txnNumbers, but before we do that, the following case can happen if we allow commits/aborts to be in the same batch with others:
When trying to split the second prepare, it fails due to this limitation since we only release the split sessions after the txn is committed and the batch is finished. 2) Allowing commits/aborts to be in the same batch with other ops makes it possible for more edge cases and requires more testings. And since applying a commitTransaction/abortTransaction entry is usually quick, we think it's fine to first disallow this and once we're confident that the new algorithm works fine, we can start an effort to optimize it. |
| Comments |
| Comment by Githook User [ 05/Apr/23 ] |
|
Author: {'name': 'Wenbin Zhu', 'email': 'wenbin.zhu@mongodb.com', 'username': 'WenbinZhu'}Message: |
| Comment by Wenbin Zhu [ 11/Jan/23 ] |
|
We should be careful with the isEndOfLargeTransaction() check in mustProcessIndividually(). This check evaluates to true for a prepareTransaction entry of a large prepared transaction, which would make every prepareTransaction entry of a large prepared transaction to be processed in its own batch. Our goal is to apply prepareTransaction entries concurrently with other entries, but it might be fine for prepareTransaction of a large prepared transaction to still be in its own batch, since we already parallelize application within a prepared transaction, and a large transaction means that we are applying more than 16MB worth of data, so it's fine to cut the batch. |