-
Type:
Bug
-
Resolution: Gone away
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Storage Execution
-
ALL
-
Storage Execution 2026-11-09
-
0
-
None
-
None
-
None
-
None
-
None
-
None
-
None
NB This is an AI-generated ticket. I read the description, and it sounds reasonable, but do send it back to Repl if it's a Repl issue.
Summary
A replica set node performing rollback of a drop of a timeseries collection can hang indefinitely inside KVDropPendingIdentReaper::immediatelyCompletePendingDrop()'s unbounded retry loop while re-creating the collection during the rollback reconstruct phase. No timeout value in the driving test hook can recover; the node only escapes when killed.
Found while investigating BF-46904.
Failure evidence
Build: replica_sets_reconfig_kill_primary_jscore_passthrough on rhel8-debug-aubsan-x86, mongodb-mongo-v9.0-staging @ 260d2adc (v9.0.5). The test itself passed; the CheckReplOplogs hook failed with awaitReplication timed out after 600s because one node entered rollback one second after the hook started and never finished.
The rolled-back operation was the test's final drop of a timeseries collection: jstests/core/timeseries/query/timeseries_insert_normalize_metadata.js drop of test.timeseries_insert_normalize_metadata at ts(1792527741, 28), t=1 (written to the node's rollback/local.oplog.rs/removed.2026-10-20T20-22-36.0.bson).
The hang-analyzer core dump of the rolling-back node (10 minutes into rollback) shows two threads wedged against each other:
BackgroundSync (rollback driver):
#0 pthread_cond_wait #2 mongo::ThreadPool::Impl::waitForIdle() #4 mongo::repl::OplogApplierImpl::_applyOplogBatch(...) #6 mongo::repl::ReplicationRecoveryImpl::_applyOplogOperations(...) #8 mongo::repl::ReplicationRecoveryImpl::_recoverFromStableTimestamp(...) #9 mongo::repl::ReplicationRecoveryImpl::recoverFromOplog(...) #10 mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(...) rollback_impl.cpp:756 #11 mongo::repl::RollbackImpl::runRollback(...) rollback_impl.cpp:285 #12 mongo::repl::BackgroundSync::_runRollbackViaRecoverToCheckpoint(...) bgsync.cpp:778
ReplWriterWorker-0 (the worker BackgroundSync waits on):
#0 nanosleep #1 mongo::sleepmillis #2 mongo::log_backoff_detail::logAndBackoffImpl(unsigned long) #3 mongo::KVDropPendingIdentReaper::immediatelyCompletePendingDrop(...) #4 mongo::StorageEngineImpl::immediatelyCompletePendingDrop(...) #5 mongo::durable_catalog::createStorage(...) #6 mongo::durable_catalog::createCollection(...) [re-creating the timeseries collection during rollback]
Root cause analysis
- KVDropPendingIdentReaper::immediatelyCompletePendingDrop() (src/mongo/db/storage/kv/kv_drop_pending_ident_reaper.cpp:386) retries in for (size_t retries = 1;; ++retries) with no deadline, exiting only when the drop succeeds or opCtx is interrupted. Every attempt returned ObjectIsBusy because something held the drop-pending ident (the old timeseries buckets collection ident).
- The blocker is not the transient cases the loop was written for: the node's Checkpointer thread was parked in FixedIntervalPolicy::waitUntilReady (idle) and its TimestampMonitor thread was idle in its periodic wait at hang time.
- The ident was pinned continuously for 10+ minutes with zero progress, so awaitReplication would time out identically at any timeout; the node was wedged until the hang analyzer killed it.
Suggested directions
- Determine what pins the drop-pending timeseries ident during rollback-via-recover-to-checkpoint (WiredTiger session/snapshot inspection of the archived core dumps would show it; dump_sessions/dump_recovery_units failed to symbolize on this core).
- Consider making immediatelyCompletePendingDrop bounded or interruptible when invoked from rollback, so a wedged drop cannot wedge the whole rollback sequence and the node's replication state indefinitely.
Repro artifacts
- BF: BF-46904
- Task: replica_sets_reconfig_kill_primary_jscore_passthrough_4-linux-enterprise
- Hang-analyzer core dumps and Core Analyzer results attached to the task (dump_mongod.131158.core.gz contains the stacks above).
- Data files artifact shows the rolled-back op in node0/rollback/local.oplog.rs/.
Mitigation on the test side (skip nodes in ROLLBACK/RECOVERING in ReplSetTest.awaitReplication so the hook doesn't burn its timeout) is not handled because the server-side fix should completely resolve the issue.