Checkpoint crash points after the checkpoint transaction commit have a timing-dependent outcome

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Checkpoints, Test Model
    • None
    • Storage Engines, Storage Engines - Persistence
    • 0.294
    • None
    • None

      Both checkpoint crash points reachable from debug=(checkpoint_crash_point=N) sit after the checkpoint transaction commits (__checkpoint_db_internal, src/checkpoint/checkpoint_txn.c). The commit puts the checkpoint's metadata updates into the logged metadata table, so with connection logging enabled whether the crashed checkpoint survives depends on whether the log server has already written those records.

      *wt_debug_crash() is kill(getpid(), SIGKILL) — a process kill, not a machine crash — so an unsynced write() survives just as well as a synced one. The FIXME-WT-15069 workaround that moves the crash ahead of *wt_log_flush therefore does not make the checkpoint unrecoverable; it only makes it rare. That is the WT-18299 flake.

      Current outcome matrix

      crash site logging off logging on
      per-tree loop (pre-commit) lost lost
      before_metadata_update (post-commit, pre-fsync) lost racy
      before_metadata_sync (post-fsync) lost recoverable

      Note the middle and bottom rows are identical with logging off: the only code between them is __checkpoint_log_stage(WT_TXN_LOG_CKPT_FLUSH), which returns immediately when logging is disabled, and the WT_TIMING_STRESS_CHECKPOINT_STOP point sits after both. So one of the two named phases contributes nothing in the non-logging case today.

      Secondary defect: the numeric mapping is inverted at the top

      __checkpoint_db_debug_crash_points() derives trigger points by subtracting the handle count:

      crash\_point = \(\(N \- 1\) \* \(handle\_next \+ 2\)\) / 999 \+ 1;
      if \(crash\_point > handle\_next\)
          crash\_trigger\_point = crash\_point \- handle\_next;
      

      Because CKPT_CRASH_BEFORE_METADATA_SYNC is 1 and CKPT_CRASH_BEFORE_METADATA_UPDATE is 2 but they fire in the opposite order, checkpoint_crash_point=1000 selects an earlier phase than 999 — contrary to what the setting documents ("higher values will result in crashes in the final phase"). This is a consequence of the arithmetic above, so it is fixed by removing it rather than separately.

      Proposal

      • Confine checkpoint_crash_point to the per-tree loop, which always precedes the commit, so its checkpoint is never recoverable with or without logging. Guard handle_next == 0: a checkpoint that gathered no handles has no crash site in that loop, and the configured crash point would otherwise survive to the __checkpoint_teardown assertion.
      • Reach the two final phases only through checkpoint_crash_trigger_point, and move before_metadata_update ahead of the commit as before_checkpoint_commit. That leaves one phase that never keeps the checkpoint and one that always keeps it with logging on, and adds a state nothing covered before: every data file fsynced, transaction not yet committed.
      • Refuse both settings at once — they name crash points on opposite sides of the commit and so disagree about the outcome.
      • Remove the FIXME-WT-15069 workaround.

      Definition of done

      • Every (crash point, logging) pair has exactly one outcome.
      • The FIXME-WT-15069 workaround is gone.
      • checkpoint_crash_point and checkpoint_crash_trigger_point together are rejected.
      • dist/api_data.py describes what the numeric setting actually does.

      Coverage note

      test/model is the only consumer of the numeric setting, and it only emits numbers, so on its own this change removes test/model's automatic coverage of the two final phases. WT-15069 restores it — at a controlled rate rather than one that varied with how many trees happened to be dirty — by teaching the workload format to name a phase. The phases stay fully reachable by name meanwhile, which is how test_key_provider_disagg02.py already drives its own crash points.

      Fix

      Written as the second commit of https://github.com/wiredtiger/wiredtiger/pull/14437, split out of WT-15069 because it stands alone and closes WT-18299 by itself.

            Assignee:
            Etienne Petrel
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: