[SERVER-81179] Backup cursor service reports backup cursor checkpointTimestamp that does not match the actual WT backup cursor checkpointTimestamp Created: 19/Sep/23  Updated: 29/Oct/23  Resolved: 20/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0, 7.1.0-rc4

Type: Bug Priority: Critical - P2
Reporter: Suganthi Mani Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-81032 Fix checkpoint detection while openin... Closed
is related to WT-11709 API to retrieve timestamp of a checkp... Blocked
is related to SERVER-81208 Use checkpoint cursor timestamp when ... Blocked
Assigned Teams:
Storage Execution NAMER
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v7.1
Sprint: Execution NAMR Team 2023-10-02
Participants:
Linked BF Score: 148

 Description   

After SERVER-81032, the  backup cursor service reports backup cursor checkpointTimestamp  that does not match the actual checkpointTimestamp at which WT opened a backup cursor, i.e, reported checkpointTimestamp   can be <= actual checkpointTimestamp) , instead of ==

Given the fact committing the checkpoint and updating txn_global.last_ckpt_timestamp (reported by getLastStableRecoveryTimestamp()) aren't atomic. This means, we can end up a scenario, like below

1) CKPT thread: WT checkpoint committed for TS(100) with ckptId:100
2) BackupService thread: Opens the _mdb_catalog cursor with read source as KCheckpoint.

  • This will open the checkpoint cursor on the latest checkpoint, ckptId:100

3) BackupService thread: Calls getLastStableRecoveryTimestamp() and reads the previous checkpoint  ts values , say TS(90).
4) CKPT thread: Updates the {{txn_global.last_ckpt_timestamp }} to TS(100)
5) BackupService thread: Opens the backup cursor
6) BackupService thread: Verifies if any checkpoint was taken between step #3 and #5 .

  • For which, It agains opens the checkpoint cursor on _mdb_catalog and reads checkpoint id as ckptId:100, and compares with step#2 checkpoint Id.

Since, step #2 and step#6 checkpoint Id are same, the sanity check in step#6 passes. However, now the backup cursor returns the `checkpointTimestamp` as TS(90) (ie, step #3 value) instead of actual checkpoint ts value at which WT opened backup cursor, which is TS(100).

Before SERVER-81032, given the fact WT takes checkpoint lock when opening the backup cursor (step #5) and for the entire checkpoint job, at step#6, calling the getLastStableRecoveryTimestamp() would guarantee to return at least TS(100), in the above case . And, I think, any new checkpoints between step#5 and #6 is uninteresting. So, it's ok, even if Step#6 reads the stale last checkpoint ts.

 

My proposal would be to make step 6 to use the original way , which is  using `getLastStableRecoveryTimestamp()`

 



 Comments   
Comment by Suganthi Mani [ 19/Sep/23 ]

Regression in 7.1

Comment by Suganthi Mani [ 19/Sep/23 ]

spoke offline and we agreed to have backup cursor service do the following steps

  1. Open checkpoint cursor on _mdb_catalog
  2. Read last stable recovery timestamp
  3. Open WT backup cursor
  4. Read last stable recovery timestamp
  5. Verify if a checkpoint occurred  between step #2  and step #3  by comparing steps #2 and #4 value.

This will address both BF-29621 and BF-30016  CC gregory.wlodarek@mongodb.com     gregory.noma@mongodb.com  

Generated at Thu Feb 08 06:45:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.