-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Compaction
-
None
-
Storage Engines, Storage Engines - Persistence
-
0.99
-
SE Persistence backlog
-
None
Issue Summary
The eviction-assist call in *wt_compact() is a no-op for the background compaction server, so background compaction never throttles itself against eviction once it has started working on a file. The compact server session is opened with WT_SESSION_IGNORE_CACHE_SIZE, and *wt_evict_app_assist_worker_check() returns immediately for any session carrying that flag. The only cache-related bail-out left inside the tree walk is __wt_evict_cache_stuck(), which requires eviction to be fully stuck and is therefore much stricter than the dirty trigger.
This was found while investigating the following scenario: background compaction was enabled on a secondary, a very large collection was being compacted, the node was then elected primary during a maintenance window, and the node started hitting the 20% dirty cache trigger and impacting application operations. Compaction kept running throughout.
Context
The compact server session sets the flag at src/conn/conn_compact.c:
/\* \* Compaction does enough I/O it may be called upon to perform slow operations for the block \* manager. Don't let the background compaction thread be pulled into eviction to limit \* performance impacts. \*/ session\_flags = WT\_SESSION\_CAN\_WAIT | WT\_SESSION\_IGNORE\_CACHE\_SIZE; WT\_RET\(\_\_wt\_open\_internal\_session\( conn, "compact\-server", true, session\_flags, 0, &conn\->background\_compact.session\)\);
__wt_evict_app_assist_worker_check() in src/evict/evict_inline.h then short-circuits:
/\* \* Don't block the thread for eviction when holding the handle list, schema or table locks \* ... Some internal threads such as the sweep server and background compact will set the \* "ignore cache size" flag for this reason. \*/ if \(F\_ISSET\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\) || FLD\_ISSET\(session\->lock\_flags, WT\_SESSION\_LOCKED\_HANDLE\_LIST | WT\_SESSION\_LOCKED\_SCHEMA | WT\_SESSION\_LOCKED\_TABLE\)\) return \(0\);
So the throttle added in [WT-13216] applies to foreground WT_SESSION::compact only. The description of [WT-14338] assumes the opposite ("note that this code is triggered from background compaction as well"), which should be corrected.
Consequences, all inside a single file's compaction:
* The per-file dirty/clean trigger pre-check in the server loop is the only dirty-cache gate, and it is not re-evaluated for the whole duration of the session->compact() call.
* On cache pressure the server loop only backs off for full_iteration_wait_time (10 seconds) before retrying.
* Compaction actively creates dirty content: *compact_page_inmem() calls *wt_page_modify_set() on clean in-memory pages selected for rewrite, and __compact_worker() forces two checkpoints per pass, up to 100 passes per file.
* The default per-table timeout is 1200 seconds, and it is only evaluated at the same periodic check points, so a single large table can hold the server for ~20 minutes at a stretch.
Related tickets:
* [WT-14338] - investigation of compaction throttling thresholds (covers the wider design question).
* [WT-13229] - give up compaction if the cache is under pressure for too long.
* [WT-13216] - added the eviction check this ticket shows is bypassed for background compaction.
* [WT-13080] - added the per-file pre-check in the background compact server loop.
Proposed Solution
Scope-clear WT_SESSION_IGNORE_CACHE_SIZE around the eviction-assist call in the periodic check block of __wt_compact() (src/btree/bt_compact.c), so background compaction pays its own eviction cost:
orig\_flags = F\_MASK\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\); F\_CLR\(session, WT\_SESSION\_IGNORE\_CACHE\_SIZE\); ret = \_\_wt\_evict\_app\_assist\_worker\_check\(session, false, false, false, false, &eviction\_happened\); F\_SET\(session, orig\_flags\);
Notes on safety and scope:
* That check point holds no flush lock and no block-manager resources, so the original rationale for the flag (not being pulled into eviction while performing slow block-manager operations) is preserved: the flag stays set everywhere else.
* The walk does hold a hazard pointer on the current ref, which only forces busy=true in the assist check (evict what we can rather than block). This is the same state foreground compaction is already in today.
* There is precedent for the save/restore idiom in src/conn/conn_layered_ingest.c.
* Do not change the sweep server session or WTI_CHECKPOINT_SESSION_FLAGS, which use the same flag for different reasons.
Follow-up work tracked elsewhere, listed here for context (do not implement in this ticket):
* Threshold-based yield with hysteresis, plus a dedicated statistic and a WT_VERB_COMPACT message - [WT-13229] / [WT-14338]. Note that background_compact_fail_cache_pressure is currently gated on *wt_evict_cache_stuck() in the server loop, so any new dirty-trigger yield would be misattributed to plain background_compact_fail.
* Generating less dirty content under pressure, e.g. skipping the in-memory selection path in *compact_page_inmem() when over the dirty target and rewriting only on-disk blocks.
* An I/O capacity class for background compaction in src/conn/conn_capacity.c.
Definition of Done
- The eviction-assist check in __wt_compact() is effective for the background compaction server, verified by a test.
- A Python test (in the style of test_compact07.py) runs background compaction against a table with a large amount of recoverable space, with a lowered eviction_dirty_trigger and a concurrent writer, and asserts that the compact thread performs eviction work (session_table_compact_eviction increases) rather than running unthrottled.
- Foreground WT_SESSION::compact behaviour is unchanged.
- The sweep server and checkpoint sessions are untouched.
- A comment is added to [WT-14338] correcting the assumption that the bt_compact.c eviction check applies to background compaction.