Background compaction bypasses the eviction assist check because the compact server session ignores cache size

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Compaction
    • None
    • Storage Engines, Storage Engines - Persistence
    • 0.99
    • SE Persistence backlog
    • None

      Issue Summary

      The eviction-assist call in *wt_compact() is a no-op for the background compaction server, so background compaction never throttles itself against eviction once it has started working on a file. The compact server session is opened with WT_SESSION_IGNORE_CACHE_SIZE, and *wt_evict_app_assist_worker_check() returns immediately for any session carrying that flag. The only cache-related bail-out left inside the tree walk is __wt_evict_cache_stuck(), which requires eviction to be fully stuck and is therefore much stricter than the dirty trigger.

      This was found while investigating the following scenario: background compaction was enabled on a secondary, a very large collection was being compacted, the node was then elected primary during a maintenance window, and the node started hitting the 20% dirty cache trigger and impacting application operations. Compaction kept running throughout.

      Context

      The compact server session sets the flag at src/conn/conn_compact.c:

      /\*
       \* Compaction does enough I/O it may be called upon to perform slow operations for the block
       \* manager. Don't let the background compaction thread be pulled into eviction to limit
       \* performance impacts.
       \*/
      session\_flags = WT\_SESSION\_CAN\_WAIT | WT\_SESSION\_IGNORE\_CACHE\_SIZE;
      WT\_RET\(\_\_wt\_open\_internal\_session\(
        conn, "compact\-server", true, session\_flags, 0, &conn\->background\_compact.session\)\);
      

      __wt_evict_app_assist_worker_check() in src/evict/evict_inline.h then short-circuits:

      /\*
       \* Don't block the thread for eviction when holding the handle list, schema or table locks
       \* ... Some internal threads such as the sweep server and background compact will set the
       \* "ignore cache size" flag for this reason.
       \*/
      if \(F\_ISSET\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\) ||
        FLD\_ISSET\(session\->lock\_flags,
          WT\_SESSION\_LOCKED\_HANDLE\_LIST | WT\_SESSION\_LOCKED\_SCHEMA | WT\_SESSION\_LOCKED\_TABLE\)\)
          return \(0\);
      

      So the throttle added in [WT-13216] applies to foreground WT_SESSION::compact only. The description of [WT-14338] assumes the opposite ("note that this code is triggered from background compaction as well"), which should be corrected.

      Consequences, all inside a single file's compaction:
      * The per-file dirty/clean trigger pre-check in the server loop is the only dirty-cache gate, and it is not re-evaluated for the whole duration of the session->compact() call.
      * On cache pressure the server loop only backs off for full_iteration_wait_time (10 seconds) before retrying.
      * Compaction actively creates dirty content: *compact_page_inmem() calls *wt_page_modify_set() on clean in-memory pages selected for rewrite, and __compact_worker() forces two checkpoints per pass, up to 100 passes per file.
      * The default per-table timeout is 1200 seconds, and it is only evaluated at the same periodic check points, so a single large table can hold the server for ~20 minutes at a stretch.

      Related tickets:
      * [WT-14338] - investigation of compaction throttling thresholds (covers the wider design question).
      * [WT-13229] - give up compaction if the cache is under pressure for too long.
      * [WT-13216] - added the eviction check this ticket shows is bypassed for background compaction.
      * [WT-13080] - added the per-file pre-check in the background compact server loop.

      Proposed Solution

      Scope-clear WT_SESSION_IGNORE_CACHE_SIZE around the eviction-assist call in the periodic check block of __wt_compact() (src/btree/bt_compact.c), so background compaction pays its own eviction cost:

      orig\_flags = F\_MASK\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\);
      F\_CLR\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\);
      ret = \_\_wt\_evict\_app\_assist\_worker\_check\(session, false, false, false, false, &eviction\_happened\);
      F\_SET\(session, orig\_flags\);
      

      Notes on safety and scope:
      * That check point holds no flush lock and no block-manager resources, so the original rationale for the flag (not being pulled into eviction while performing slow block-manager operations) is preserved: the flag stays set everywhere else.
      * The walk does hold a hazard pointer on the current ref, which only forces busy=true in the assist check (evict what we can rather than block). This is the same state foreground compaction is already in today.
      * There is precedent for the save/restore idiom in src/conn/conn_layered_ingest.c.
      * Do not change the sweep server session or WTI_CHECKPOINT_SESSION_FLAGS, which use the same flag for different reasons.

      Follow-up work tracked elsewhere, listed here for context (do not implement in this ticket):
      * Threshold-based yield with hysteresis, plus a dedicated statistic and a WT_VERB_COMPACT message - [WT-13229] / [WT-14338]. Note that background_compact_fail_cache_pressure is currently gated on *wt_evict_cache_stuck() in the server loop, so any new dirty-trigger yield would be misattributed to plain background_compact_fail.
      * Generating less dirty content under pressure, e.g. skipping the in-memory selection path in *compact_page_inmem() when over the dirty target and rewriting only on-disk blocks.
      * An I/O capacity class for background compaction in src/conn/conn_capacity.c.

      Definition of Done

      • The eviction-assist check in __wt_compact() is effective for the background compaction server, verified by a test.
      • A Python test (in the style of test_compact07.py) runs background compaction against a table with a large amount of recoverable space, with a lowered eviction_dirty_trigger and a concurrent writer, and asserts that the compact thread performs eviction work (session_table_compact_eviction increases) rather than running unthrottled.
      • Foreground WT_SESSION::compact behaviour is unchanged.
      • The sweep server and checkpoint sessions are untouched.
      • A comment is added to [WT-14338] correcting the assumption that the bt_compact.c eviction check applies to background compaction.

            Assignee:
            [DO NOT USE] Backlog - Storage Engines Team
            Reporter:
            Etienne Petrel
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: