• Storage Engines - Persistence
    • 4.152
    • None
    • None

      Problem

      On the 7.0 branch, the stat checkpoint-cleanup: most recent duration on all eligible files (usecs) (cc_duration) is a per-checkpoint accumulator, not a per-cleanup-pass duration. It is zeroed at the start of each checkpoint and incremented by every internal-page traversal performed by the cleanup utility thread and by the inline cleanup path in the checkpoint. It therefore rises continuously between checkpoints (reaching ~10-12 s on a busy 7.0 primary), while a single cleanup pass takes hundreds of milliseconds. On 8.0.21+, 8.3, 9.0 and develop the same metric key is a gauge set once per full pass.

      What went wrong with the backport

      WT-16698 added the stats on develop (a827bdf691) by timing one full __checkpoint_cleanup_int() iteration and publishing it with WT_STAT_CONN_SET.

      The 7.0 backport eaa187c861 (PR #13339, BACKPORT-26883) was reworked because 7.0 can still run cleanup inline during a checkpoint as well as from the utility thread. It measured differently – WT_STAT_CONN_INCRV per internal-page traversal plus a reset in __wt_txn_checkpoint() – but kept the develop name and description unchanged.

      The semantic change was deliberate (see the comment on BACKPORT-26883), but it was never reflected in the stat description, the documentation, or the dashboards. Consumers therefore see the same field name with two different meanings depending on the server version.

      Secondary defects in the same backport:

      • The stats were added to conn_dsrc_stats, so a per-data-source copy is generated and published but never written (always 0).
      • cc_handle_processed has the same accumulation behaviour: incremented per btree/handle by both cleanup paths and reset per checkpoint, so it is not "most recent handles processed".

      Impact

      • FTDC/Atlas dashboards title this metric "Cleanup Pass Duration" and plot a sawtooth that resets every checkpoint; inside a window it looks like a cumulative counter and cannot be used to read per-pass duration on 7.0.
      • The metric key is identical on 7.0 and 8.0+, so the same field means different things across supported versions.
      • Cross-version comparisons of cleanup cost are misleading.

      Evidence

      FTDC from a 7.0.45 replica set, ~16.5 h, 52,630 samples:

      • 70 zeroings / 71 segments. Each segment climbs to ~10.5 M usec and then resets. Example segment: 2026-10-14 18:46:18 -> 19:14:44 (UTC+11), 1,973 -> 10,120,140 usec; reset at 19:14:45.
      • Window 2026-10-14T07:46:21Z -> 08:13:46Z: 16.392 ms -> 10,120.140 ms, 10,103.7 ms accrued; per-second increments p50 4.3 ms, p99 54.9 ms, max 274.6 ms.
      • On develop, 8 cleanup triggers give independent values (353, 508, 820, 857, 929, 1156, 835, 666 usec), confirming the intended per-pass gauge.

      Affected versions

      7.0.32 through 7.0.45 (stat introduced by BACKPORT-26883). Not affected: 8.0.21+, 8.2, 8.3, 9.0 and develop.

      Options

      1. Align 7.0 with develop: time one __checkpoint_cleanup_int() pass and WT_STAT_CONN_SET the result; remove the per-page increment and the per-checkpoint reset. Correct semantics and cross-version consistency; inline cleanup performed during a checkpoint is no longer counted (same as develop).
      2. Keep the 7.0 measurement but give it an accurate description (e.g. "checkpoint cleanup duration since the last checkpoint (usecs)"). This changes the serverStatus/FTDC key, so Atlas consumers must be migrated first.
      3. Keep the behaviour and the key, document the difference, and fix the dashboards/docs only. Cheapest, but the stat's description remains wrong.

      Recommend option 1 for correctness, or option 2 with a planned key migration if the inline coverage matters.

      Tasks

      • [ ] Decide between the options above.
      • [ ] Implement the chosen change on the 7.0 branch.
      • [ ] Remove the unused per-data-source copies of cc_duration / cc_handle_processed.
      • [ ] Add a test that triggers cleanup repeatedly and asserts the value does not accumulate across passes.
      • [ ] Update WT-16700 dashboards / documentation if the key or semantics change.

      Definition of done

      • The 7.0 value either matches the documented "most recent duration" semantics or is renamed/documented so its meaning is unambiguous.
      • A test covers the behaviour and fails on the current 7.0 code.
      • No silently different metric keys across supported versions.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: