ExportXMLWordPrintableJSON

    • Type: Bug
    • Resolution: Gone away
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Storage Execution
    • ALL
    • Storage Execution 2026-11-09
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      NB This is an AI-generated ticket. I read the description, and it sounds reasonable, but do send it back to Repl if it's a Repl issue.

      Summary

      A replica set node performing rollback of a drop of a timeseries collection can hang indefinitely inside KVDropPendingIdentReaper::immediatelyCompletePendingDrop()'s unbounded retry loop while re-creating the collection during the rollback reconstruct phase. No timeout value in the driving test hook can recover; the node only escapes when killed.

      Found while investigating BF-46904.

      Failure evidence

      Build: replica_sets_reconfig_kill_primary_jscore_passthrough on rhel8-debug-aubsan-x86, mongodb-mongo-v9.0-staging @ 260d2adc (v9.0.5). The test itself passed; the CheckReplOplogs hook failed with awaitReplication timed out after 600s because one node entered rollback one second after the hook started and never finished.

      The rolled-back operation was the test's final drop of a timeseries collection: jstests/core/timeseries/query/timeseries_insert_normalize_metadata.js drop of test.timeseries_insert_normalize_metadata at ts(1792527741, 28), t=1 (written to the node's rollback/local.oplog.rs/removed.2026-10-20T20-22-36.0.bson).

      The hang-analyzer core dump of the rolling-back node (10 minutes into rollback) shows two threads wedged against each other:

      BackgroundSync (rollback driver):

      #0 pthread_cond_wait
      #2 mongo::ThreadPool::Impl::waitForIdle()
      #4 mongo::repl::OplogApplierImpl::_applyOplogBatch(...)
      #6 mongo::repl::ReplicationRecoveryImpl::_applyOplogOperations(...)
      #8 mongo::repl::ReplicationRecoveryImpl::_recoverFromStableTimestamp(...)
      #9 mongo::repl::ReplicationRecoveryImpl::recoverFromOplog(...)
      #10 mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(...)   rollback_impl.cpp:756
      #11 mongo::repl::RollbackImpl::runRollback(...)                                   rollback_impl.cpp:285
      #12 mongo::repl::BackgroundSync::_runRollbackViaRecoverToCheckpoint(...)         bgsync.cpp:778
      

      ReplWriterWorker-0 (the worker BackgroundSync waits on):

      #0 nanosleep
      #1 mongo::sleepmillis
      #2 mongo::log_backoff_detail::logAndBackoffImpl(unsigned long)
      #3 mongo::KVDropPendingIdentReaper::immediatelyCompletePendingDrop(...)
      #4 mongo::StorageEngineImpl::immediatelyCompletePendingDrop(...)
      #5 mongo::durable_catalog::createStorage(...)
      #6 mongo::durable_catalog::createCollection(...)  [re-creating the timeseries collection during rollback]
      

      Root cause analysis

      • KVDropPendingIdentReaper::immediatelyCompletePendingDrop() (src/mongo/db/storage/kv/kv_drop_pending_ident_reaper.cpp:386) retries in for (size_t retries = 1;; ++retries) with no deadline, exiting only when the drop succeeds or opCtx is interrupted. Every attempt returned ObjectIsBusy because something held the drop-pending ident (the old timeseries buckets collection ident).
      • The blocker is not the transient cases the loop was written for: the node's Checkpointer thread was parked in FixedIntervalPolicy::waitUntilReady (idle) and its TimestampMonitor thread was idle in its periodic wait at hang time.
      • The ident was pinned continuously for 10+ minutes with zero progress, so awaitReplication would time out identically at any timeout; the node was wedged until the hang analyzer killed it.

      Suggested directions

      • Determine what pins the drop-pending timeseries ident during rollback-via-recover-to-checkpoint (WiredTiger session/snapshot inspection of the archived core dumps would show it; dump_sessions/dump_recovery_units failed to symbolize on this core).
      • Consider making immediatelyCompletePendingDrop bounded or interruptible when invoked from rollback, so a wedged drop cannot wedge the whole rollback sequence and the node's replication state indefinitely.

      Repro artifacts

      Mitigation on the test side (skip nodes in ROLLBACK/RECOVERING in ReplSetTest.awaitReplication so the hook doesn't burn its timeout) is not handled because the server-side fix should completely resolve the issue.

            Assignee:
            Damian Wasilewicz
            Reporter:
            Brad Cater
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: