SizeCountCheckpointFlusher::run busy-loops on shutdown interrupt, wedging shutdown

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      SizeCountCheckpointFlusher::run() can busy-loop forever when its operation context is interrupted by shutdown, wedging node shutdown. This surfaces with featureFlagSizeBasedOplogTruncationForDisagg enabled, since that flag starts the size-count checkpoint flusher.

      Root Cause

      In src/mongo/db/replicated_fast_count/size_count_checkpoint_flusher.cpp, the run() loop only returns from the thread when the caught exception is InterruptedDueToReplStateChange or NotWritablePrimary:

      } catch \(const DBException& ex\) {
          if \(ex.code\(\) == ErrorCodes::InterruptedDueToReplStateChange ||
              ex.code\(\) == ErrorCodes::NotWritablePrimary\) {
              LOGV2\_DEBUG\(12917804, 2, "...interrupted due to replication state", ...\);
              return;
          } else {
              incrementFlushFailureCount\(\);
              LOGV2\_WARNING\(12917805, "Exception handled in SizeCountCheckpointFlusher::run\(\)", ...\);
          }
      }
      

      On a normal shutdown the opCtx is killed with a shutdown error (e.g. InterruptedAtShutdown). That code is not one of the two handled above, so control falls into the else branch, logs a warning, and loops again. The opCtx is still interrupted, so waitForConditionOrInterrupt throws the same shutdown error immediately, and the loop spins without ever returning — the flusher thread never exits and shutdown hangs.

      Impact

      * A node with featureFlagSizeBasedOplogTruncationForDisagg enabled can hang during shutdown instead of exiting cleanly.
      * Discovered while adding rotation shutdown-safety tests under SERVER-131734: with the flag on (the disagg_storage suite enables it), the flusher wedged shutdown and hung the tests.

      Suggested Fix

      Treat shutdown errors like the existing repl-state-change / not-primary cases and return from run(), e.g. add ErrorCodes::isShutdownError(ex.code()) to the returning branch (or more simply, return on any interrupt that indicates the opCtx/node is going away).

      Workaround

      The rotation shutdown-safety jstests pin featureFlagSizeBasedOplogTruncationForDisagg: false to avoid the hang. That workaround should be removed once this is fixed.

      Affected Files

      * src/mongo/db/replicated_fast_count/size_count_checkpoint_flusher.cppSizeCountCheckpointFlusher::run()

            Assignee:
            Unassigned
            Reporter:
            Ken Martin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: