Motivation

In HELP-98158 a multi-tenant deployment hit sustained cache pressure driven by update memory (cache: bytes allocated for updates reaching 31% of a 176 GB cache). The question we could not answer from the field was the obvious one: which tables are holding that memory?

Nothing we ship today answers it on a live node:

• cache: bytes allocated for updates is connection-wide only. WT_BTREE.bytes_updates is maintained per tree (_wt_cache_page_inmem_incr) but is never published – _wt_btree_stat_init exposes cache_bytes_dirty_total and friends, not bytes_updates.
• __wt_verbose_dump_cache does print per-file update byte totals, but it gets them by walking every page of every open handle. That is far too expensive to run against a busy production cache, and in a release build it only fires automatically when the cache is already declared stuck (cache_stuck_timeout_ms elapsed with no progress) and verbose eviction logging happens to be enabled.

Publishing a per-table statistic is not a viable answer either: this deployment has a database per tenant and millions of collections, which rules out both a per-dhandle FTDC series and any approach that samples per-table statistics across the whole node.

Proposal

Report a bounded top-N list of the data handles consuming the most cache, cheap enough to run on a busy primary, ranked three ways:

• by update memory (WT_BTREE.bytes_updates) – which tables hold the pending-update bytes;
• by total resident bytes (WT_BTREE.bytes_inmem) – which tables hold the read working set; and
• by bytes read into cache (cache_bytes_read) – which tables churn the most data through the cache. Unlike the first two (levels), this is a cumulative counter, so ranking "top readers over an interval" means the consumer diffs two snapshots taken an interval apart; the report should expose the raw cumulative value.

The second and third rankings were added after further HELP-98158 analysis showed the read working set filling the cache was the underlying condition, with update bursts as the trigger; per-table resident and read-into-cache bytes exist in collStats today but are unusable for ranking across millions of collections. All three counters are already maintained incrementally, so producing the list is one pass over the dhandle list reading a few counters per handle – no tree walk, no page access, no hazard pointers. Cost scales with open handles rather than with cache size.

Open design questions:

• Delivery mechanism: a new WT_CONNECTION::debug_info category, a connection-level statistic holding the top entries, or an eviction verbose message emitted when the updates target is exceeded. Whatever we pick has to be reachable from mongod in the field – debug_info currently is not, which is a large part of why this was unanswerable.
• Whether to also rank by bytes_dirty_leaf.
• Handling of handles that are open but idle, and of the sweep interaction (sweep-closing a handle also resets its cumulative read counter, which consumers diffing snapshots need to tolerate).

Acceptance

On a running connection, and without walking the cache, an operator can obtain short ranked lists of the tables holding the most update memory, the most total cache memory, and the highest bytes read into cache, at a cost acceptable on a loaded production node.