[SERVER-65018] Batched writes must fit inside a single applyOps Created: 29/Mar/22  Updated: 29/Oct/23  Resolved: 08/Apr/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: Task Priority: Major - P3
Reporter: Haley Connelly Assignee: Josef Ahmad
Resolution: Fixed Votes: 0
Labels: PM-2227-M3
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
related to SERVER-64972 Generate change stream events for bat... Closed
related to SERVER-77302 ensure feature flag is disabled for B... Closed
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2022-04-18
Participants:
Linked BF Score: 168

 Description   

Unlike multi-document transactions which can span through multiple applyOps oplog entries, batched writes should generate one and only one applyOps. We should also ignore the multi-doc transactions specific maxNumberOfTransactionOperationsInSingleOplogEntry parameter. The reason for both decision is that the applyOps entry for a batched write lacks the multi-doc transactions specific txnNumber field which is needed to make multiple applyOps entries replicate atomically. There's no known use case for batched writes requiring to generate more than a single applyOps (16MB worth of oplog).

Original title: Investigate core/multi_batched_deletes.js running with 'maxNumberOfTransactionOperationsInSingleOplogEntry'
Original description: maxNumberOfTransactionOperationsInSingleOplogEntry: 2 causes the test to fail. It appears the test runs with 'partialTxn': true when the oplog is dumped. We expected batched multi deletes to fail completely when run in a transaction. It is unclear why this server parameter would cause the test to run batched deletes in a transaction. 



 Comments   
Comment by Githook User [ 08/Apr/22 ]

Author:

{'name': 'Josef Ahmad', 'email': 'josef.ahmad@mongodb.com', 'username': 'josefahmad'}

Message: SERVER-65018 Batched writes must fit inside a single applyOps
Branch: master
https://github.com/mongodb/mongo/commit/9ab71f9b2fac1e384529fafaf2a819ce61834228

Comment by Josef Ahmad [ 04/Apr/22 ]

An alternative solution would be to have the batched write context only allow batches within 16MB worth of oplog, which is well within the bounds of our expected usage. The solution would also ignore the maxNumberOfTransactionOperationsInSingleOplogEntry parameter which is specific to multi-document transactions. We'll explore the actual need and feasibility of bringing these aspects of the batched write context on par with the transaction API as part of a follow-up project represented by SERVER-64506.

Comment by Josef Ahmad [ 29/Mar/22 ]

Had a quick look at this ticket. Looks that when I implemented SERVER-63047 (transactional replication of WUOW's) I omitted an ingredient to handle the special case of splitting a transaction over multiple oplog entries: the prevOpTime oplog field. The good news is that I'm expecting this field, along with the ancillary txnNumber field, to not only enable replicating batched deletes of virtually any oplog size, but also to enable change streams (SERVER-64972).

Generated at Thu Feb 08 06:01:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.