[SERVER-34279] Crash after upgrade before stable checkpoint can cause replication recovery to skip oplog entries Created: 03/Apr/18  Updated: 29/Oct/23  Resolved: 13/Apr/18

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 3.7.4

Type: Bug Priority: Major - P3
Reporter: Judah Schvimer Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: rollback-functional
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by WT-3959 Recovery timestamp set on restart sce... Closed
Duplicate
is duplicated by SERVER-34350 Properly initialize initial data time... Closed
Related
related to SERVER-34716 Add unittest of lastStableCheckpointT... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Repl 2018-04-23
Participants:

 Description   

Consider the following scenario:
1. Clean shutdown a 3.6-binary. It's appliedThrough value will be null.
2. Bring the node up with a 4.0 binary. Replication recovery will do nothing since we are consistent at the top of the oplog; there is no appliedThrough or recoveryTimestamp.
3. The node starts taking writes from ts=T1 to ts=T2, as a primary or secondary. These writes get written to the oplog, but only the oplog writes get journaled. The appliedThrough may move forward if it's a secondary, but those writes will also not be journaled.
4. Now, before we take a stable checkpoint, the node crashes.
5. Restart the 4.0 binary node. The node starts up with the same data as at step 2 (reflecting a consistent point at T1), but also with the oplog entries through T2 from step 3.
6. There is no recoveryTimestamp and the appliedThrough will be null, so we assume we're consistent at the top of the oplog, T2, when in reality we are consistent at T1. We then do not replay T1->T2.



 Comments   
Comment by Githook User [ 13/Apr/18 ]

Author:

{'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb', 'username': 'dgottlieb'}

Message: SERVER-34279: Perform an FCV upgrade to take stable checkpoints.
Branch: master
https://github.com/mongodb/mongo/commit/3ca72d037459c06eee20e80a754ae65420ce2a51

Comment by Githook User [ 13/Apr/18 ]

Author:

{'email': 'daniel.gottlieb@mongodb.com', 'name': 'Daniel Gottlieb', 'username': 'dgottlieb'}

Message: SERVER-34279: Ensure a minValid lastApplied exists for upgrade.
Branch: master
https://github.com/mongodb/mongo/commit/a3b00f53eaf0295eac183ba8009eda0b9dec95aa

Comment by Daniel Gottlieb (Inactive) [ 05/Apr/18 ]

I also believe the `initialDataTimestamp` needs to be set to the top of oplog when the `recoveryTimestamp` is 0. The storage engine currently sets the value when the recoveryTimestamp is non-zero. This work should consider doing this item, or ticket it for additional work, given it affects proper upgrade and rollback when the `recoveryTimestamp` is not present. See SERVER-34350

Amendment, SERVER-34350 is done as part of this ticket.

Generated at Thu Feb 08 04:36:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.