Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-7675

Query last ckpt timestamp changes without taking checkpoint

    • 5
    • Storage - Ra 2021-06-28
    • v5.0, v4.4, v4.2

      MDB backups of WT want to know the "checkpoint timestamp" of the data pinned by a backup cursor. IIRC sue.loverso offered providing a query_timestamp API for doing so, to which I responded "we're good".

      Instead MDB does optimistic concurrency control between opening backup cursors and taking checkpoints. We don't serialize the operations, but rather do the simple check + recheck:

      1. query_timestamp(get=last_checkpoint)
      2. open backup cursor
      3. query_timestamp(get=last_checkpoint)

      If the values are the same, we know the timestamp of the data being copied by the backup. If we are not taking a checkpoint during any of these calls to query_timestamp, we believe the returned timestamp should always match.

      We're observing cases where this assumption is not holding. We're seeing consecutive calls to query_timestamp returning different values, but no checkpoint was taken that would explain why. Unfortunately we don't have perfect logging of the values returned by "query_timestamp".

      As an example:

      D2 RECOVERY [WTCheckpointThread] Performing stable checkpoint. StableTimestamp: Timestamp(1623229619, 2), OplogNeededForRollback: Timestamp(1623229619, 2)
      D1 STORAGE  [WTCheckpointThread] Checkpoint took 57 seconds to complete.
      I  STORAGE  [conn29862] Opened backup cursor. ID: <snip> Term: 547
      I  STORAGE  [conn29862] Closed backup cursor. ID: <snip>
      <long log line saying an error was returned due to the timestamps not matching>
      D2 RECOVERY [WTCheckpointThread] Performing stable checkpoint. StableTimestamp: Timestamp(1623229735, 2), OplogNeededForRollback: Timestamp(1623229735, 2)

      sue.loverso noted that the conn->txn_global.last_ckpt_timestamp value is written to without a mutex which could explain this observation.

            ravi.giri@mongodb.com Ravi Giri
            daniel.gottlieb@mongodb.com Daniel Gottlieb (Inactive)
            0 Vote for this issue
            8 Start watching this issue