[SERVER-44852] Have secondary oplog application handle transactions with commands Created: 26/Nov/19 Updated: 29/Oct/23 Resolved: 14/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Storage |
| Affects Version/s: | None |
| Fix Version/s: | 4.3.3 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Geert Bosch | Assignee: | Maria van Keulen |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | Execution Team 2019-12-30, Execution Team 2020-01-13, Execution Team 2020-01-27 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Transactions appear as applyOps oplog entries. To allow transactions to include both commands, such as index and collection creation, and other operations, ensure that the entire transaction is applied as a single batch, and ensure that the operations inside this batch are applied serially. The latter goal can be accomplished by changing the logic in OplogApplierImpl::fillWriterVectors to make sure that a single writer worker is used to apply the operations in the batch. To ensure the entire transaction is applied as a single batch, we can either parse the entire oplog entry as part of the check for whether it must be processed individually to check for commands inside of transactions, or add information to the oplog entry to signify that it is a transaction with a command. The former may have be a slight performance hit due to the extra processing, but the latter is more difficult to implement. Provided the performance hit is not too high, the former solution is preferable. |
| Comments |
| Comment by Maria van Keulen [ 22/Jan/20 ] |
|
suganthi.mani Yes, create_collection.js does createCollection alongside CRUD ops in the same transaction.
|
| Comment by Judah Schvimer [ 22/Jan/20 ] |
|
I think it is worth writing such a targeted test regardless so it is easier to debug if we break that case specifically, passthroughs are great for bridging coverage gaps, but they are also harder to debug. |
| Comment by Suganthi Mani [ 21/Jan/20 ] |
|
maria.vankeulen I do know we have jstests/core/txns/create_collection.js and some fsm workloads (like jstests/concurrency/fsm_workloads/create_database.js) will run createCollection cmd inside the repl transaction. Will those tests run the createCollection cmd along with the CRUD ops inside a single transaction or run the createCollection cmd in one transaction and CRUD ops in other transaction? If not, I think we should try to write a jstest which should tests running a repl transaction containing both createCollection cmd and CRUD ops. Let me know your thoughts. |
| Comment by Githook User [ 14/Jan/20 ] |
|
Author: {'name': 'Maria van Keulen', 'email': 'maria.vankeulen@mongodb.com', 'username': 'mvankeulen94'}Message: |
| Comment by Maria van Keulen [ 06/Jan/20 ] |
|
I've split out the work to accommodate cross-shard transactions into its own ticket ( |
| Comment by Maria van Keulen [ 20/Dec/19 ] |
|
Thanks for letting me know. In that case, I agree that the best solution is to not decompose the applyOps for unprepared transactions if they contain commands. I will leave the serverStatus tracking as potential future work. |
| Comment by Judah Schvimer [ 19/Dec/19 ] |
This is only true for prepared transactions. Transactions that never go through prepare do not get applied in one batch. They get split onto writer threads as part of a larger batch. |
| Comment by Maria van Keulen [ 19/Dec/19 ] |
|
Per a discussion with suganthi.mani, when we apply the operations inside a transaction, they are all applied serially as part of one batch. As such, applying both the commands and the operations inside the same batch should be fine. The work for this ticket should be to modify the existing transaction oplog application code as necessary to accommodate oplog application of the commands newly allowed inside transactions. |
| Comment by Judah Schvimer [ 27/Nov/19 ] |
|
I think it would be a lot simpler to process transaction oplog entries that contain commands in their own batch. It would be a perf hit, but I expect these are not very common so it might be quite small. We'd have to either add an indicator to the oplog entry that it is a transaction that contains a command, or start looking at the entire transaction when batching oplog entries. I think it may be a good idea to start with a simpler solution and then add complexity if this is a significant bottleneck to secondary application in practice. Adding server status metrics counting how many transactions have commands in them could help us decide if this complexity is worth adding (and would be a useful metric in general). |