[SERVER-33743] Use all_committed to set lastApplied on primary nodes Created: 08/Mar/18  Updated: 29/Oct/23  Resolved: 20/Mar/18

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: None
Fix Version/s: 3.6.5, 3.7.4

Type: Task Priority: Major - P3
Reporter: Eric Milkie Assignee: Judah Schvimer
Resolution: Fixed Votes: 0
Labels: rollback-functional
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on WT-3961 The all_committed timestamp should be... Closed
is depended on by SERVER-29213 Have KVWiredTigerEngine implement Sto... Closed
Problem/Incident
Related
related to SERVER-34895 Stable timestamp can be set to timest... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v3.6, v3.4, v3.2
Sprint: Repl 2018-03-26
Participants:
Linked BF Score: 50

 Description   

Currently, for a node in PRIMARY state, lastApplied is set to each write's optime when each write commits, as long as the time being set moves forward. (It might not otherwise always move forward because writes can commit out of optime order.)
This method of setting lastApplied is problematic for consumers of this value, as it can be set ahead of writes that are still in-flight and have yet to commit. For one-voting-node replica sets, this can result in the majority point moving ahead of uncommitted, in-flight writes.

Instead, we can use WiredTiger's all_committed value to set lastApplied. After each write commits, we can query the current all_committed value and set lastApplied to that.
The all_committed value is the greatest (latest) timestamp value with no existing in-flight writes with lower (earlier) timestamps. This value has been used in the past to implement oplog visibility, preventing oplog readers from reading past "holes" in the oplog left by uncommitted, in-flight writes.



 Comments   
Comment by Githook User [ 23/Apr/18 ]

Author:

{'email': 'judah@mongodb.com', 'username': 'judahschvimer', 'name': 'Judah Schvimer'}

Message: SERVER-33743 removed unnecessary virtual keyword
Branch: v3.6
https://github.com/10gen/mongo-enterprise-modules/commit/99b82170690dfc88fa4dfb9ddbd820dec3d26af2

Comment by Githook User [ 23/Apr/18 ]

Author:

{'email': 'judah@mongodb.com', 'username': 'judahschvimer', 'name': 'Judah Schvimer'}

Message: SERVER-33743 Use all_committed to set lastApplied on primary nodes
Branch: v3.6
https://github.com/10gen/mongo-enterprise-modules/commit/b315eb586b4da04a3fc7f4820c90d0741ffa6e4b

Comment by Githook User [ 23/Apr/18 ]

Author:

{'email': 'judah@mongodb.com', 'username': 'judahschvimer', 'name': 'Judah Schvimer'}

Message: SERVER-33743 Use all_committed to set lastApplied on primary nodes

(cherry picked from commit 2ee6908f1c73dd50d6425e3462ccac2582deb2f3)
Branch: v3.6
https://github.com/mongodb/mongo/commit/51f11a4ca3442d0788a63b2c6d0836457cc08f3e

Comment by Githook User [ 20/Mar/18 ]

Author:

{'email': 'judah@mongodb.com', 'name': 'Judah Schvimer', 'username': 'judahschvimer'}

Message: SERVER-33743 Use all_committed to set lastApplied on primary nodes
Branch: master
https://github.com/mongodb/mongo/commit/2ee6908f1c73dd50d6425e3462ccac2582deb2f3

Generated at Thu Feb 08 04:34:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.