[SERVER-40078] Add wall clock time corresponding to lastCommitted optime to optimes subdocument Created: 11/Mar/19  Updated: 29/Oct/23  Resolved: 16/Apr/19

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.1.11

Type: Task Priority: Major - P3
Reporter: Maria van Keulen Assignee: Maria van Keulen
Resolution: Fixed Votes: 0
Labels: todo_in_code
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-40080 Add lastApplied and lastDurable wall ... Closed
is depended on by SERVER-34598 Add millisecond-granularity wallclock... Closed
Related
related to SERVER-40353 Add wall clock time corresponding to ... Closed
related to SERVER-40659 Add regression tests for SERVER-40078 Closed
related to SERVER-40683 Make wall clock times in replication ... Closed
related to SERVER-40737 Use Date_t() as default wallTime valu... Closed
related to SERVER-42485 Remove FCV checks gating reporting w... Closed
related to SERVER-43465 Remove InvalidBSON try/catch blocks f... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.0, v3.6
Sprint: Storage NYC 2019-03-25, Storage NYC 2019-04-08, Storage NYC 2019-04-22
Participants:

 Description   

This ticket is split out from SERVER-34598. SERVER-40080 will now be used to track the patch to include wall clock times corresponding to the lastApplied and lastDurable optimes only. The tasks required for SERVER-40080 and SERVER-40078 are different, and the lastApplied/lastDurable changes are sufficiently complex on their own.

The current plan for implementing this follows. Since SERVER-40080 allows each member of a replica set to track its own last applied and last durable wall clock times, they can communicate this information to the primary similarly to how they communicate their OpTime information (i.e., through replSetUpdatePosition, heartbeats, etc.) so the majority committed wall clock time can be calculated similarly to the way the majority committed OpTime is calculated.



 Comments   
Comment by Maria van Keulen [ 17/Apr/19 ]

commit url: https://github.com/mongodb/mongo/commit/a738647a7281a892f1b35fba8b4e1cdf47bccf56

Comment by Maria van Keulen [ 27/Mar/19 ]

I am splitting out the work for adding the readConcernMajority wall clock time into SERVER-40353, since the patch to add the lastCommitted wall clock time is sufficiently complex on its own.

Comment by Maria van Keulen [ 19/Mar/19 ]

I've updated the ticket description with our current proposed solution.

Comment by Maria van Keulen [ 15/Mar/19 ]

schwerin I have a concern that threading wall time information through heartbeats as well as everywhere else the commit point can get updated, let alone introducing the upgrade/downgrade dependency, will make this ticket quite complex to both implement and test. I will spend some time scoping out what's necessary for this work so we can reach a decision on how to proceed.

Comment by Andy Schwerin [ 14/Mar/19 ]

Other than risk in the work, is there harm in using a solution that has upgrade/downgrade dependencies? I am not thrilled about having a data structure in memory whose size grows with the replication lag. Less so when that data structure also gets lost on process restart.

Comment by Matthew Russotto [ 14/Mar/19 ]

Unfortunately, SERVER-34728 notes that we do fill out this information from heartbeats.

Comment by Maria van Keulen [ 14/Mar/19 ]

schwerin IIUC, the primary's information on other nodes' last applied and last durable optimes can be updated via heartbeat responses from those nodes in addition to via replSetUpdatePosition. We would like to avoid having to make modifications to heartbeat responses for the purposes of this patch, since this would introduce an upgrade/downgrade dependency on this work. For SERVER-40080, we only update a member's own last applied and last durable wall clock times and do not transmit this information between nodes.

Is it possible to determine the last committed wall clock time from replSetUpdatePosition without information obtained from heartbeats?

CC matthew.russotto tess.avitabile

Comment by Andy Schwerin [ 14/Mar/19 ]

I had imagined that we would just decorate the MemberData in the TopologyCoordinator for each node with the wall clock time corresponding to that node’s “last applied” and “last durable” timestamps. That data is easily transmitted as part of replSetUpdatePosition. Then, you can compute the wall time corresponding to the majority commit optime trivially as part of computing the majority commit optime.

Could a technique like this work?

Generated at Thu Feb 08 04:53:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.