[SERVER-33186] Primary node may deadlock during shutdown Created: 08/Feb/18  Updated: 06/Dec/22  Resolved: 28/Jan/19

Status: Closed
Project: Core Server
Component/s: Replication, Stability
Affects Version/s: 3.6.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Igor Solodovnikov Assignee: Backlog - Replication Team
Resolution: Duplicate Votes: 0
Labels: neweng, shutdown
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File bugrep.txt    
Issue Links:
Duplicate
duplicates SERVER-36873 ReplicationCoordinatorExternalStateIm... Closed
Related
related to SERVER-28688 Deadlock between shutdown and stepdown Closed
Assigned Teams:
Replication
Operating System: ALL
Participants:

 Description   

Note: this deadlock is similar to SERVER-28688 but this is another one.
Note: I observed this deadlock in 3.6.1.

ReplicationCoordinatorExternalStateImpl::shutdown calls _taskExecutor->join() while having _threadMutex locked. In most cases there are no tasks for worker threads and _taskExecutor->join() returns immediately. But in some rare situations DropPendingCollectionReaper has some collections to drop and while these tasks are running signal processing thread keeps _threadMutex locked. If at this moment replication logic decides to stepdown then we have a deadlock because ReplicationCoordinatorExternalStateImpl::startProducerIfStopped tries to acquire _threadMutex while holding the global exclusive lock. After startProducerIfStopped starts its wait for _threadMutex drop collection tasks are also blocked by the global lock.

Attached file contains output of mongodb-waitsfor-graph, mongodb-show-locks, mongodb-uniqstack commands. In this file:

  • thread 2 (signalProcessingThread) owns _threadMutex lock (acquired in ReplicationCoordinatorExternalStateImpl::shutdown)
    and waits for shutdown of worker threads (_taskExecutor->shutdown(); _taskExecutor->join()
  • thread 47: "replexec-9" waits for _threadMutex (owned by thread 2)
    is processing _stepDownFinish event
    which calls _updateMemberStateFromTopologyCoordinator_inlock
    which calls startProducerIfStopped
    which tries to aquire _threadMutex
  • thread 48 (worker thread executing dropCollection task)
    waits for global lock owned by thread 47


 Comments   
Comment by Benety Goh [ 28/Jan/19 ]

Starting in 4.2, the second phase of collection drops will be handled internally within the storage engine and not at the replication layer. The deadlock described in this ticket involving the DropPendingCollectionReaper and the ReplicationCoordinatorExternalStateImpl _threadMutex should no longer happen.

Comment by Igor Solodovnikov [ 08/Feb/18 ]

One way to fix this is probably to disable stepdown if shutdown is in progress.

Generated at Thu Feb 08 04:32:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.