[SERVER-66023] Do not constantly reset election and liveness timers Created: 27/Apr/22  Updated: 29/Oct/23  Resolved: 18/May/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.1, 5.0.11, 6.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Matthew Russotto Assignee: Matthew Russotto
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
Related
is related to SERVER-59776 50% regression in single multi-update Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0, v5.3, v5.0, v4.4
Sprint: Repl 2022-05-02, Repl 2022-05-16, Repl 2022-05-30
Participants:
Linked BF Score: 135

 Description   

We cancel the election timeout on secondaries whenever the primary liveness is updated, which potentially happens on every oplog batch. We cancel the liveness timeout on primaries whenever the oldest secondary liveness is updated, which potentially happens on every replSetUpdatePosition. It turns out cancelling a timer, at least on Linux, is quite expensive (likely system call overhead), and we do this in the replication lock, which increases contention on that already-hot mutex.

We can greatly reduce this with a class which handles "cancel and reschedule" by keeping track of the latest time of the reschedule, and then when the timeout occurs, reschedules at that point instead of immediately. This means we get no cancels and one reschedule every timeout interval (not every miniscule bump forward of the timer)



 Comments   
Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Use DelayableTimeoutCallback class to avoid resetting election and liveness timers.

(cherry picked from commit 94f976f475b7d26e75f60d0e0733aed99bb6f272)
Branch: v5.0
https://github.com/mongodb/mongo/commit/1b47d46c4f6f4a23392b0945601026c58f75792d

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Create a DelayableTimeoutCallback class to avoid constantly resetting timers.

(cherry picked from commit 466ddbc1784709f3421ed9ccc485210b1fe94e6e)
(cherry picked from commit 87d6ec40779e3750a123efd941e16d159bf938bb)
Branch: v5.0
https://github.com/mongodb/mongo/commit/7320052f50acc875e6ea5e74bb3aef92ce997d43

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Use DelayableTimeoutCallback class to avoid resetting election and liveness timers.

(cherry picked from commit 94f976f475b7d26e75f60d0e0733aed99bb6f272)
Branch: v6.0
https://github.com/mongodb/mongo/commit/baf9e3860e60e483f9cb23170456d2292c0d9abf

Comment by Githook User [ 27/Jul/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Create a DelayableTimeoutCallback class to avoid constantly resetting timers.

(cherry picked from commit 466ddbc1784709f3421ed9ccc485210b1fe94e6e)
(cherry picked from commit 87d6ec40779e3750a123efd941e16d159bf938bb)
Branch: v6.0
https://github.com/mongodb/mongo/commit/38130f58fbb884ec958b9d35981f8c868b2ef98b

Comment by Githook User [ 18/May/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Use DelayableTimeoutCallback class to avoid resetting election and liveness timers.
Branch: master
https://github.com/mongodb/mongo/commit/94f976f475b7d26e75f60d0e0733aed99bb6f272

Comment by Githook User [ 18/May/22 ]

Author:

{'name': 'Matthew Russotto', 'email': 'matthew.russotto@mongodb.com', 'username': 'mtrussotto'}

Message: SERVER-66023 Create a DelayableTimeoutCallback class to avoid constantly resetting timers.
Branch: master
https://github.com/mongodb/mongo/commit/466ddbc1784709f3421ed9ccc485210b1fe94e6e

Generated at Thu Feb 08 06:04:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.