-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Execution
-
ALL
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Overview
SizeCountCheckpointFlusher::run() can busy-loop forever when its operation context is interrupted by shutdown, wedging node shutdown. This surfaces with featureFlagSizeBasedOplogTruncationForDisagg enabled, since that flag starts the size-count checkpoint flusher.
Root Cause
In src/mongo/db/replicated_fast_count/size_count_checkpoint_flusher.cpp, the run() loop only returns from the thread when the caught exception is InterruptedDueToReplStateChange or NotWritablePrimary:
} catch \(const DBException& ex\) { if \(ex.code\(\) == ErrorCodes::InterruptedDueToReplStateChange || ex.code\(\) == ErrorCodes::NotWritablePrimary\) { LOGV2\_DEBUG\(12917804, 2, "...interrupted due to replication state", ...\); return; } else { incrementFlushFailureCount\(\); LOGV2\_WARNING\(12917805, "Exception handled in SizeCountCheckpointFlusher::run\(\)", ...\); } }
On a normal shutdown the opCtx is killed with a shutdown error (e.g. InterruptedAtShutdown). That code is not one of the two handled above, so control falls into the else branch, logs a warning, and loops again. The opCtx is still interrupted, so waitForConditionOrInterrupt throws the same shutdown error immediately, and the loop spins without ever returning — the flusher thread never exits and shutdown hangs.
Impact
* A node with featureFlagSizeBasedOplogTruncationForDisagg enabled can hang during shutdown instead of exiting cleanly.
* Discovered while adding rotation shutdown-safety tests under SERVER-131734: with the flag on (the disagg_storage suite enables it), the flusher wedged shutdown and hung the tests.
Suggested Fix
Treat shutdown errors like the existing repl-state-change / not-primary cases and return from run(), e.g. add ErrorCodes::isShutdownError(ex.code()) to the returning branch (or more simply, return on any interrupt that indicates the opCtx/node is going away).
Workaround
The rotation shutdown-safety jstests pin featureFlagSizeBasedOplogTruncationForDisagg: false to avoid the hang. That workaround should be removed once this is fixed.
Affected Files
* src/mongo/db/replicated_fast_count/size_count_checkpoint_flusher.cpp — SizeCountCheckpointFlusher::run()