[SERVER-41333] Initial Sync and Shutdown write to the minValid document at the lastApplied Created: 28/May/19  Updated: 29/Oct/23  Resolved: 16/Jan/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: 4.3.3

Type: Bug Priority: Major - P3
Reporter: Pavithra Vetriselvan Assignee: Suganthi Mani
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by WT-4780 Enable assertion that commit timestam... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Repl 2019-11-18, Execution Team 2019-12-16, Repl 2019-12-02, Repl 2019-12-16, Execution Team 2020-01-13, Execution Team 2020-01-27, Execution Team 2019-12-30
Participants:
Linked BF Score: 9

 Description   

During shutdown and initial sync, we update the minValid document using the lastApplied as the timestamp of the write. Since the lastApplied has already been updated before we write to the minValid document, any reader can read at the lastApplied before the write commits.

This causes us to hit at WT invariant: commitTimestamp > latest active read timestamp (Note, the commitTimestamp is referring to the timestamp at which the write to the minValid document commits).

Initial Sync: We get the lastApplied from the repl coordinator here and update the minValid here using the lastApplied.

Shutdown: We get the lastApplied from the repl coordinator and update the minValid here.

This race doesn't seem to affect steady state replication as frequently. This is because sync tail ensures that the lastApplied does not change at the start of the batch and bgsync does not set the lastApplied unless it is unset.

UPDATE: The WT invariant was disabled. Enabling the WT invariant is blocked on this ticket.



 Comments   
Comment by Githook User [ 16/Jan/20 ]

Author:

{'username': 'smani87', 'name': 'Suganthi Mani', 'email': 'suganthi.mani@mongodb.com'}

Message: SERVER-41333 Make safer for initial Sync and Shutdown to write the minValid document at the lastApplied.
Branch: master
https://github.com/mongodb/mongo/commit/e34948fe7a90aa328237b54eddb75584a38ff52c

Comment by Tess Avitabile (Inactive) [ 29/May/19 ]

That sounds great, alexander.gorrod. We can see how hard it would be to avoid this issue for the minValid document.

Comment by Alexander Gorrod [ 28/May/19 ]

Thanks for the update tess.avitabile - I've adjusted the fix version for WT-4780. Forcing the conversation this far was the point of keeping the fix version at 4.2. If it would be helpful we can add a flag to the API so known violators of the assertion can be marked as such - let's make that decision when we come back to re-enabling the assertion after the release.

Comment by Tess Avitabile (Inactive) [ 28/May/19 ]

We may also hit the WT invariant when we set appliedThrough upon transitioning to secondary and beginning batch application.

Generated at Thu Feb 08 04:57:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.