Summary

__ckpt_get_blkmods() treats a missing checkpoint_backup_info key in a file's metadata record as a hard error. It is perfectly legitimate for a checkpoint to exist without that field — any file whose metadata was last written before the field was introduced will not have it. When incremental backup is configured, the first checkpoint of such a file fails with an unlogged WT_NOTFOUND, which is laundered into WT_PANIC by the block manager. The process aborts, and because the file's metadata is never rewritten, it aborts again on every subsequent checkpoint — an unrecoverable crash loop, with no diagnostic in the log.

The check that fails is a diagnostic aid added by WT-11178. It is not load-bearing: its purpose is to catch a bitmap regression, and there is no bitmap to regress against when the field is absent.

Root cause

The metadata read path has always tolerated this, and says so explicitly. src/meta/meta_ckpt.c:38-43 (line numbers at mongodb-8.0 / d860df9b91):

    /*
     * We could be reading in a configuration from an earlier release. If the string doesn't exist
     * then we're done.
     */
    if ((ret = \_\_wt_config_getones(session, config, "checkpoint_backup_info", &v)) != 0)
        return (ret == WT_NOTFOUND ? 0 : ret);

The check path added by WT-11178 does not. src/meta/meta_ckpt.c:1316-1317:

    WT_ERR(
      \_\_wt_config_getones(session, file_config, "checkpoint_backup_info", &backup_config_value));

Same key, same file, ~1280 lines apart, opposite handling.

__wt_config_getones() → __config_getraw() ends if (!found) return (WT_NOTFOUND); with no __wt_err() call, so nothing is logged.

How it reaches a panic

1. __ckpt_valid_blk_mods() (meta_ckpt.c:519) sets WT_BLOCK_MODS_VALID on the added checkpoint for every connection-level WT_BLKINCR_VALID entry. The file's own metadata is not consulted, so this fires even for a file that has never carried backup info.
2. __wt_meta_ckptlist_set() (meta_ckpt.c:1483) calls __ckpt_check_backup_blocks() unconditionally for every non-metadata dhandle.
3. __ckpt_check_backup_blocks() (meta_ckpt.c:1412) loops over the valid entries and calls __ckpt_get_blkmods(), which returns WT_NOTFOUND — silently.
4. That propagates out of __wt_meta_ckptlist_set() to __checkpoint_tree()'s err: label (src/txn/txn_ckpt.c:2451 → :2473), where resolve_bm is still true, so bm->checkpoint_resolve(bm, session, ret != 0) runs with failed == true.
5. __wt_block_checkpoint_resolve() (src/block/block_ckpt.c:1005-1011) then panics with a hardcoded EINVAL, and WT_TRET overwrites the original WT_NOTFOUND with WT_PANIC.

The user-visible result is two log lines and nothing else:

\_\_wt_block_checkpoint_resolve:1008:<file>.wt: the checkpoint failed, the system must restart: Invalid argument
\_\_wt_block_checkpoint_resolve:1008:the process must exit and restart: WT_PANIC: WiredTiger library panic

Note the absence of "File blkmods are not compatible with those in the checkpoint" — the message this code emits when the bitmap comparison genuinely fails. Its absence distinguishes this failure from the one WT-11178 was written to catch.

Why it is unrecoverable without rebuilding the file

__wt_config_collapse() discards any key not present in the first configuration string, so the record cannot regain the field through the non-use_base path in __ckpt_set(). The use_base path would heal it — meta_ckpt.c:291 supplies "checkpoint=(),checkpoint_backup_info=(),checkpoint_lsn=" as the base — but it sits after the failing check and is never reached. So every restart reproduces the failure identically.

Preconditions

1. The file's metadata record has no checkpoint_backup_info key. The field was added in WT-5206 (4.4.0), and 4.4+ writes an empty ,checkpoint_backup_info= even when incremental backup is off — so in practice this means a file whose last checkpoint was taken under 4.2 or earlier and which has not been checkpointed since.
2. At least one incremental backup ID is registered on the connection.
3. Something writes to the file, dirtying the btree. Clean btrees are skipped indefinitely (__checkpoint_lock_dirty_tree() → WT_BTREE_SKIP_CKPT, clean_ckpt_timer pinned to WT_BTREE_CLEAN_CKPT_FOREVER), which is why such a file can sit dormant for years and then fail the moment it is touched.

Affected versions

The check exists only on the 8.0 line. Verified by inspecting src/third_party/wiredtiger/src/meta/meta_ckpt.c at each tag (the file and tags exist at every ref checked, so the negatives are real):

WT-11178 was imported to mongodb/mongo as 4b7a8b79863 "for: 8.0.0-rc0". WT-12791 (be613c0bf5) removed the whole check on develop and was never backported to v8.0.

Proposed fix

Make __ckpt_get_blkmods() tolerate the missing key, matching __ckpt_load_blk_mods():

    WT_ERR_NOTFOUND_OK(
      \_\_wt_config_getones(session, file_config, "checkpoint_backup_info", &backup_config_value),
      false);

with output_item left cleared, so __ckpt_check_backup_blocks()'s if ((checkpoint_blkmods_buffer.size > 0) && (file_blkmods_buffer.size > 0)) guard skips the comparison — which is the correct semantic: there is no previous bitmap, therefore no bit can have been cleared.

Alternatively, backport WT-12791 to v8.0 and remove the check entirely, as develop already has.

Worth considering separately: __ckpt_check_backup_blocks() reports a genuine mismatch via WT_ASSERT_ALWAYS, i.e. abort() in a release build. If this check is retained, that is a harsh response to a diagnostic condition.

Related

• WT-11178 — introduced the check (post-mortem hardening for WT-10551, epic SEAI-128)
• WT-12791 — removed it on develop; not backported to v8.0
• WT-5206 — introduced checkpoint_backup_info (4.4.0)
• WT-13215 — earlier bug against this same comparison being over-strict; fix was to relax it
• WT-16780 — the missing error context that makes this class invisible; v8.0 only from 8.0.26
• WT-16697 — umbrella for the unexplained block-manager errors
• WT-6480 — earlier statement of the "checkpointed before backups were configured and never subsequently modified" case