Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4471

Election in replica set can cause deadlock in monitor threads, leading to unavailability

    XMLWordPrintableJSON

Details

    • Fully Compatible
    • Not Needed

    Description

      We discovered blocking threads a couple of times in some of our Java processes shortly after an election in the replica set of 3 MongoDB instances, which make our services unresponsive:

      The Thread-Dump looks, in all processes with blocking threads, similar.

      First blocked thread:

       

      cluster-ClusterId{value='61f29e20622d627cc0e99c19', description='null'}-mongodb-instance-3:27017 id=24 state=BLOCKED
      		 
          - waiting to lock <0x0b2400cb> (a com.mongodb.internal.connection.MultiServerCluster)
      		 
           owned by cluster-ClusterId{value='61f29e20622d627cc0e99c19', description='null'}-mongodb-instance-1:27017 id=20
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster.onChange(AbstractMultiServerCluster.java:175)
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster.access$100(AbstractMultiServerCluster.java:50)
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster$DefaultServerDescriptionChangedListener.serverDescriptionChanged(AbstractMultiServerCluster.java:139)
      		 
          at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.updateDescription(DefaultSdamServerDescriptionManager.java:127)
      		 
          at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.update(DefaultSdamServerDescriptionManager.java:81)
      		 
          at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:165)
      		 
          at java.base@11.0.12/java.lang.Thread.run(Thread.java:829)    Locked synchronizers: count = 1
      		 
            - java.util.concurrent.locks.ReentrantLock$NonfairSync@32e56896

      Second blocked thread:

       

       

      cluster-ClusterId{value='61f29e20622d627cc0e99c19', description='null'}-mongodb-instance-2:27017 id=22 state=BLOCKED
      		 
          - waiting to lock <0x0b2400cb> (a com.mongodb.internal.connection.MultiServerCluster)
      		 
           owned by cluster-ClusterId{value='61f29e20622d627cc0e99c19', description='null'}-mongodb-instance-1:27017 id=20
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster.onChange(AbstractMultiServerCluster.java:175)
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster.access$100(AbstractMultiServerCluster.java:50)
      		 
          at com.mongodb.internal.connection.AbstractMultiServerCluster$DefaultServerDescriptionChangedListener.serverDescriptionChanged(AbstractMultiServerCluster.java:139)
      		 
          at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.updateDescription(DefaultSdamServerDescriptionManager.java:127)
      		 
          at com.mongodb.internal.connection.DefaultSdamServerDescriptionManager.update(DefaultSdamServerDescriptionManager.java:81)
      		 
          at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:165)
      		 
          at java.base@11.0.12/java.lang.Thread.run(Thread.java:829)    Locked synchronizers: count = 1
      		 
            - java.util.concurrent.locks.ReentrantLock$NonfairSync@166cbc02

       

      We renamed the host in the thread dump, but kept the instance number.

      The only way to fix this problem is to restart our processes.

      We discovered this problem since we upgraded from MongoDB Driver 4.2.3 (Spring Boot 2.5.6) to MongoDB Driver 4.4.0 (Spring Boot 2.6.2) and later MongoDB Driver 4.4.1 (Spring Boot 2.6.3)

      Our MongoDB Server instances run on version 4.2.2, what should be compatible according to https://docs.mongodb.com/drivers/java/sync/current/compatibility/ 

      How to Reproduce

      Unfortunately, we are not able to reproduce these phenomena. We just see them in irregular intervals and have to restart our services. 

      Additional Background

      • We use the synchronous Java Driver
      • Java 11
      • Linux 

      We can provide more information if needed.

      Thanks for your help!

       

      Attachments

        1. image-2022-02-03-17-15-21-102.png
          image-2022-02-03-17-15-21-102.png
          58 kB
        2. image-2022-02-03-17-15-43-755.png
          image-2022-02-03-17-15-43-755.png
          21 kB
        3. threaddump-anonymized.tdump
          431 kB

        Issue Links

          is caused by

          New Feature - A new feature of the product, which has yet to be developed. JAVA-3928 Connection pool paused state

          • Major - P3 - Major loss of function.
          • Closed
          links to

          Web Link PR1

          Web Link PR1 port to 4.4.x

          Web Link PR2

          Activity

            People

              valentin.kovalenko@mongodb.com Valentin Kavalenka
              robert.zilke@outlook.com Robert Zilke
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: