[SERVER-44852] Have secondary oplog application handle transactions with commands Created: 26/Nov/19  Updated: 29/Oct/23  Resolved: 14/Jan/20

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: None
Fix Version/s: 4.3.3

Type: New Feature Priority: Major - P3
Reporter: Geert Bosch Assignee: Maria van Keulen
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-45368 Allow collection and index creation i... Closed
Related
related to SERVER-44139 Allow collection creation inside a mu... Closed
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2019-12-30, Execution Team 2020-01-13, Execution Team 2020-01-27
Participants:

 Description   

Transactions appear as applyOps oplog entries. To allow transactions to include both commands, such as index and collection creation, and other operations, ensure that the entire transaction is applied as a single batch, and ensure that the operations inside this batch are applied serially. The latter goal can be accomplished by changing the logic in OplogApplierImpl::fillWriterVectors to make sure that a single writer worker is used to apply the operations in the batch.

To ensure the entire transaction is applied as a single batch, we can either parse the entire oplog entry as part of the check for whether it must be processed individually to check for commands inside of transactions, or add information to the oplog entry to signify that it is a transaction with a command. The former may have be a slight performance hit due to the extra processing, but the latter is more difficult to implement. Provided the performance hit is not too high, the former solution is preferable.



 Comments   
Comment by Maria van Keulen [ 22/Jan/20 ]

suganthi.mani Yes, create_collection.js does createCollection alongside CRUD ops in the same transaction.

SERVER-45370 and SERVER-44409 will also add some more coverage of parallel transactions. Is there a specific test case you have in mind that isn't covered here? I'm not sure I see the value in adding the same create_collection.js test to noPassthrough.

Comment by Judah Schvimer [ 22/Jan/20 ]

I think it is worth writing such a targeted test regardless so it is easier to debug if we break that case specifically, passthroughs are great for bridging coverage gaps, but they are also harder to debug.

Comment by Suganthi Mani [ 21/Jan/20 ]

maria.vankeulen I do know we have jstests/core/txns/create_collection.js and some fsm workloads (like jstests/concurrency/fsm_workloads/create_database.js) will run createCollection cmd inside the repl transaction. Will those tests run the createCollection cmd along with the CRUD ops inside a single transaction or run the createCollection cmd in one transaction and CRUD ops in other transaction? If not, I think we should try to write a jstest which should tests running a repl transaction containing both createCollection cmd and CRUD ops. Let me know your thoughts.

CC judah.schvimer siyuan.zhou

Comment by Githook User [ 14/Jan/20 ]

Author:

{'name': 'Maria van Keulen', 'email': 'maria.vankeulen@mongodb.com', 'username': 'mvankeulen94'}

Message: SERVER-44852 Handle transactions with commands during oplog application
Branch: master
https://github.com/mongodb/mongo/commit/846c7aa84ac08dbccc7d727e9068406b7b2de033

Comment by Maria van Keulen [ 06/Jan/20 ]

I've split out the work to accommodate cross-shard transactions into its own ticket (SERVER-45368) since this present ticket has a fair amount of work scoped already.

Comment by Maria van Keulen [ 20/Dec/19 ]

Thanks for letting me know. In that case, I agree that the best solution is to not decompose the applyOps for unprepared transactions if they contain commands. I will leave the serverStatus tracking as potential future work.

Comment by Judah Schvimer [ 19/Dec/19 ]

they are all applied serially as part of one batch

This is only true for prepared transactions. Transactions that never go through prepare do not get applied in one batch. They get split onto writer threads as part of a larger batch.

Comment by Maria van Keulen [ 19/Dec/19 ]

Per a discussion with suganthi.mani, when we apply the operations inside a transaction, they are all applied serially as part of one batch. As such, applying both the commands and the operations inside the same batch should be fine. The work for this ticket should be to modify the existing transaction oplog application code as necessary to accommodate oplog application of the commands newly allowed inside transactions.

Comment by Judah Schvimer [ 27/Nov/19 ]

I think it would be a lot simpler to process transaction oplog entries that contain commands in their own batch. It would be a perf hit, but I expect these are not very common so it might be quite small. We'd have to either add an indicator to the oplog entry that it is a transaction that contains a command, or start looking at the entire transaction when batching oplog entries. I think it may be a good idea to start with a simpler solution and then add complexity if this is a significant bottleneck to secondary application in practice. Adding server status metrics counting how many transactions have commands in them could help us decide if this complexity is worth adding (and would be a useful metric in general).

Generated at Thu Feb 08 05:07:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.