[SERVER-70903] support multi-oplog format for batched operations Created: 27/Oct/22  Updated: 02/Feb/24  Resolved: 17/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Task Priority: Major - P3
Reporter: Benety Goh Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-70572 support applyOps oplog entry chaining... Closed
Related
related to SERVER-76409 enable feature flag for SERVER-70903 Open
related to SERVER-70904 use OplogApplierImpl to process batch... Closed
related to SERVER-72723 support rollback for multi-oplog batc... Closed
related to SERVER-72830 write multiversion js test for batche... Closed
related to SERVER-76029 remove 6.0/7.0 mixed cluster multiver... Closed
related to SERVER-72584 ensure partial transactions contain l... Closed
is related to SERVER-70572 support applyOps oplog entry chaining... Closed
is related to SERVER-51301 Have no-op writes for recording pre/p... Investigating
Backwards Compatibility: Fully Compatible
Sprint: Execution Team 2022-11-14, Execution Team 2022-12-12, Execution Team 2022-11-28, Execution Team 2022-12-26, Execution Team 2023-01-09, Execution Team 2023-01-23
Participants:

 Description   

Batched operations (see BatchedWriteContext) are currently limited to replicating as a single oplog entry. This places the responsibility of limiting the operations in a batched write on the caller. It would be nice to re-use the chained applyOps format used for large multi-document transactions to format large batched operations in a similar manner. Callers would no longer be constrained by the single applyOps format. However, for performance reasons, callers should still be aware of the implications of allowing unbounded set of operations to be batched and handle these issues accordingly.

This ticket is limited to having the primary emit a chain of applyOps oplog entries representing a large batched write. Processing this chain of applyOps oplog entries on the secondary will be addressed in a follow-up ticket.

This behavior will be guarded by a feature flag.



 Comments   
Comment by Benety Goh [ 23/Feb/23 ]

Batched writes that span multiple oplog entries will now be formatted similarly to large multi-doc transactions with a prevOpTime field linking each oplog entry in the chain to the previous entry. This is a flag-guarded behavior.

For backwards compatibility, batched writes that fit within a single oplog entry will omit the prevOpTime field.

Comment by Githook User [ 14/Jan/23 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-70903 keep prevOpTime field for large batched writes (multiple oplog entries)
Branch: master
https://github.com/mongodb/mongo/commit/2597c37f5bfa29f1ddc9b33d7212c7beecc5afb2

Comment by Githook User [ 13/Jan/23 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-70903 relax multi timestamp constraint for large batched writes spanning multiple entries
Branch: master
https://github.com/mongodb/mongo/commit/6ad7e5338b5068f65e19a565335880310df8a897

Comment by Githook User [ 13/Jan/23 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-70903 relax multi-oplog constraint for large batched writes
Branch: master
https://github.com/mongodb/mongo/commit/333f4b044ea2a1a9ad8340cc371b5706fe84211e

Comment by Benety Goh [ 11/Jan/23 ]

Supporting multi-oplog format for batched writes will requires relaxing the multi-timestamp constraint similar to what we do for unprepared multi-document transactions. SERVER-51301 describes a longer term strategy for RecoveryUnit::ignoreAllMultiTimestampConstraints().

Generated at Thu Feb 08 06:17:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.