Issue Summary

When oplog truncation exceeds available cache resources, WiredTiger can stall indefinitely, as observed in HELP-96307. The FTDC metrics show that WiredTiger is unable to make progress, with stable and other global timestamps stalling and not being updated. This leads to a state where the cache is full of dirty content, and the system remains stuck.

Context

• The server appears to have some throttling and logging in place, as indicated by log messages such as:

  "Flow control is engaged and the sustainer point is not moving" — replication commit point stalled
  "Unable to acquire a ticket within deadline, which indicates the system is stalled" — storage engine stalled
  

• However, WiredTiger itself does not clearly signal when it is overwhelmed or unable to make progress. There is no periodic log message or FTDC signal directly from WiredTiger indicating this stalled state.
• The root cause may vary, but the oplog truncate scenario is a reproducible case for testing.

Proposed Solution

• Implement mechanisms in WiredTiger to:
  • Flag to the server/admission control when it is overwhelmed (even if this does not resolve the issue directly, it provides visibility).
  • Return an error or force rollback on large internal transactions that cause cache exhaustion.
  • Periodically generate clear log messages from WiredTiger itself when it detects it cannot make progress, providing more context on the root cause.
  • Add an easily interpretable signal in FTDC indicating that WiredTiger is stalled.
• This would give a sense of ownership from WiredTiger and help with early detection and diagnosis of such issues.

Original Slack thread: Slack Thread
This ticket was generated by AI from a Slack thread.