[SERVER-65723] Add tunable parameter to improve batching on secondaries Created: 18/Apr/22  Updated: 29/Oct/23  Resolved: 12/May/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.10, 6.0.0-rc11, 6.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Matthew Russotto Assignee: Matthew Russotto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Documented
is documented by DOCS-15338 Investigate changes in SERVER-65723: ... Closed
Related
related to SERVER-53667 High rate of journal flushes on secon... Closed
is related to SERVER-54939 Investigate secondary batching behavi... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0, v5.0, v4.4
Sprint: Repl 2022-05-02, Repl 2022-05-16
Participants:

 Description   

The combination of ending oplog batches when the commit point moves and oplog exhaust results in very small oplog batches when the secondaries are nearly caught up. This results in more iops on the secondary. This ticket will add a knob to delay oplog batching when we're caught up, which will improve batching at the expense of some w:majority latency. Experimentally a modest delay of 20ms was enough to restore full-sized batches when the oplog is being continuously written (e.g. multi-update). The default for this parameter will be 0.



 Comments   
Comment by Alan Zheng [ 19/Dec/22 ]

The parameter is documented here: oplogBatchDelayMillis

Comment by Githook User [ 22/Jun/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65723 Add parameter oplogBatchDelayMillis to improve batching on caught-up secondaries

(cherry picked from commit 97dc4971f5894c070a029dd8d05b857062b5ddde)
Branch: v5.0
https://github.com/mongodb/mongo/commit/81a4764d6637fe6a645cf54bdba6984b7d466ea9

Comment by Githook User [ 22/Jun/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65723 Enhance waiting API of oplog buffer.
Add millisecond granularity, deadlines, and Interruptibles

(cherry picked from commit 6d128d3ac2197f120a8a5927e31c56bb6ab9d328)
Branch: v5.0
https://github.com/mongodb/mongo/commit/f60e7b0978053aa7908d7e7544e1bb14bd7f9b22

Comment by Matthew Russotto [ 16/Jun/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65273 Enhance waiting API of oplog buffer.
Add millisecond granularity, deadlines, and Interruptibles

(cherry picked from commit 6d128d3ac2197f120a8a5927e31c56bb6ab9d328)
Branch: v6.0
https://github.com/mongodb/mongo/commit/fdef6fd5099d2a8c0c3c8d70b617a50195e2e609

Comment by Githook User [ 15/Jun/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65723 Add parameter oplogBatchDelayMillis to improve batching on caught-up secondaries

(cherry picked from commit 97dc4971f5894c070a029dd8d05b857062b5ddde)
Branch: v6.0
https://github.com/mongodb/mongo/commit/fa286387d786bdd2927840ab5ede8573cb4583c7

Comment by Matthew Russotto [ 12/May/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65273 Enhance waiting API of oplog buffer.
Add millisecond granularity, deadlines, and Interruptibles
Branch: master
https://github.com/mongodb/mongo/commit/6d128d3ac2197f120a8a5927e31c56bb6ab9d328

Comment by Githook User [ 12/May/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-65723 Add parameter oplogBatchDelayMillis to improve batching on caught-up secondaries
Branch: master
https://github.com/mongodb/mongo/commit/97dc4971f5894c070a029dd8d05b857062b5ddde

Generated at Thu Feb 08 06:03:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.