Details
-
Bug
-
Resolution: Done
-
Critical - P2
-
3.0.0
-
None
Description
When experiencing network issues between mongos and application server, we found that there is massive connection and thread leakage. In jstack this looks like this:
"cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5957 daemon prio=5 os_prio=0 tid=0x00007f75e039c000 nid=0x18185 waiting on condition [0x00007f6b00d4c000]
|
java.lang.Thread.State: TIMED_WAITING (parking)
|
at sun.misc.Unsafe.park(Native Method)
|
- parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
|
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
|
- locked <0x000000038c39b358> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
|
at java.lang.Thread.run(Thread.java:745)
|
|
|
"cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5956 daemon prio=5 os_prio=0 tid=0x00007f6be819a800 nid=0x18184 waiting on condition [0x00007f6b00a49000]
|
java.lang.Thread.State: TIMED_WAITING (parking)
|
at sun.misc.Unsafe.park(Native Method)
|
- parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
|
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
|
- locked <0x0000000396151050> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
|
at java.lang.Thread.run(Thread.java:745)
|
|
|
"cluster-ClusterId{value='577cd577c333491dceb8c8af', description='null'}-mongos01h.load.music.yandex.net:27017" #5948 daemon prio=5 os_prio=0 tid=0x00007f7620083800 nid=0x1817c waiting on condition [0x00007f6b00b4a000]
|
java.lang.Thread.State: TIMED_WAITING (parking)
|
at sun.misc.Unsafe.park(Native Method)
|
- parking to wait for <0x0000000386c3b1c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
|
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
|
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
|
at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
|
- locked <0x0000000396151230> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
|
at java.lang.Thread.run(Thread.java:745)
|
And up to 6-7 thousands of these useless threads per application server. At that point our mongos was killed by "too many opened files" error.
Attachments
Issue Links
- is duplicated by
-
JAVA-2290 Server monitor thread dies
-
- Closed
-