[SERVER-38555] cappedTruncateAfter must not set oldest timestamp during startup recovery when enableMajorityReadConcern=false Created: 11/Dec/18  Updated: 29/Oct/23  Resolved: 13/Dec/18

Status: Closed
Project: Core Server
Component/s: Replication, Storage
Affects Version/s: None
Fix Version/s: 4.0.7, 4.1.7

Type: Bug Priority: Major - P3
Reporter: Tess Avitabile (Inactive) Assignee: Tess Avitabile (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive repro.zip    
Issue Links:
Backports
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:

Start up mongod with enableMajorityReadConcern:false and --replSet on the attached data files: repro.zip

Sprint: Repl 2018-12-17
Participants:
Linked BF Score: 33

 Description   

In cappedTruncateAfter, when majority read concern is disabled, we set the oldest timestamp to the truncate timestamp. This is incorrect when recovering from a stable timestamp at startup, since the truncate timestamp is the end of the oplog, and we will apply (and timestamp) writes from the recovery timestamp through the end of the oplog. This causes us to attempt to commit behind the oldest timestamp. We must not set the oldest timestamp to the truncate timestamp when recovering from a stable timestamp at startup.



 Comments   
Comment by Githook User [ 15/Feb/19 ]

Author:

{'name': 'Tess Avitabile', 'email': 'tess.avitabile@mongodb.com', 'username': 'tessavitabile'}

Message: SERVER-38555 cappedTruncateAfter must not set oldest timestamp during startup recovery when enableMajorityReadConcern=false

(cherry picked from commit bb47520a6ba133d2413e72160556910db5298caf)
Branch: v4.0
https://github.com/mongodb/mongo/commit/40d8548209240eab78112de265200c48cd7699c4

Comment by Githook User [ 13/Dec/18 ]

Author:

{'username': 'tessavitabile', 'email': 'tess.avitabile@mongodb.com', 'name': 'Tess Avitabile'}

Message: SERVER-38555 cappedTruncateAfter must not set oldest timestamp during startup recovery when enableMajorityReadConcern=false
Branch: master
https://github.com/mongodb/mongo/commit/bb47520a6ba133d2413e72160556910db5298caf

Comment by Daniel Gottlieb (Inactive) [ 12/Dec/18 ]

Following up an offline conversation. I think we should backport this. A note on severity: as far as I've reasoned, this never crash can only occur when:

  1. The mongod is running as a secondary with majority reads on when it crashes.
  2. The mongod is restarted with majority reads off.

Additionally, I believe this scenario never leads to any sort of data corruption (on 4.0). Finally, there are workarounds for getting mongod running again after this crash. Workarounds include:

  1. Restarting mongod may just work. If the truncate was persisted before the crash (I'm not sure it's guaranteed to be, but it's certainly possible/likely), the following run will not have to re-do the truncate, which won't mess the oldest timestamp value.
  2. Restarting with majority reads on is guaranteed to work. After replication recovery completes, a clean shutdown can be issued to restart the node with majority reads off.
Comment by Tess Avitabile (Inactive) [ 11/Dec/18 ]

daniel.gottlieb, this sounds like it might also be a problem on 4.0, since when we restart with enableMajorityReadConcern:false, we can recover from a stable timestamp at startup. Do you think we need to backport this change to 4.0?

Generated at Thu Feb 08 04:49:17 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.