Issue Summary

During chaos testing on disaggregated storage engines, significant FTDC (Full Time Diagnostic Data Capture) gaps (~2-3 seconds) are observed on the standby node, making t2 data unreadable. The primary node does not exhibit these gaps. Investigation points to slow stats collection on the oplog, specifically when fetching size statistics with statistics=(size) and statistics=(fast) for WT_STAT_DSRC_BLOCK_SIZE and WT_STAT_DSRC_BLOCK_REUSE_BYTES.

Context

• The FTDC logs show messages such as:

  [js_test:pali_chaos] d20077| 2026-06-24T05:42:23.758+00:00 W  FTDC     9761502     [ftdc] "FTDC collector was very slow","attr":{"collector":"local.oplog.rs.stats","durationMillis":2757}
  

• Analysis (with help from Claude) suggests that statistics=(size) uses a fast path that reads the stable checkpoint size from metadata without opening a dhandle, while statistics=(fast) opens the stable checkpoint dhandle, which on a follower involves reading checkpoint metadata from the disagg page service, causing a blocking ~2.7s read and resulting in the observed FTDC microgap.
• This behavior recurs every FTDC cycle, as the stable checkpoint name advances, forcing a fresh checkpoint dhandle open each second.
• Past related investigation: SERVER-113418
• The issue is consistently reproducible in every disagg_pali_chaos run on the standby, as shown in attached FTDC metrics files.

Proposed Solution

• Assign a WT engineer to investigate the slow stats collection path on the standby, focusing on the checkpoint dhandle open logic after a new checkpoint is installed.
• Consider adding artificial delays to the suspected code path to confirm the root cause.
• Review recent changes to functions related to statistics=(size) and statistics=(fast) for potential regressions.
• Aim to eliminate or mitigate the FTDC gaps on the standby to improve t2 data readability.

Original Slack thread: Slack Thread
This ticket was generated by AI from a Slack thread.