[JAVA-2290] Server monitor thread dies Created: 26/Aug/16  Updated: 11/Nov/17  Resolved: 26/Aug/16

Status: Closed
Project: Java Driver
Component/s: Cluster Management, Error Handling
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Running against MongoDB 3.0.9


Issue Links:
Duplicate
duplicates JAVA-2238 Connection and thread leakage in Defa... Closed

 Description   

My assumptions about how the (java) driver is modeled

Creating a MongoClient that connects to a replica set, a shared state object is created which contains all of the seeded/discovered members and the last known state of each member.

The state for each member is monitored by a thread e.g:

"cluster-ClusterId{value='57bdbb382f53757a079b2616', description='scrubbedReplName'}-scrubbedHostname:scrubbedPort" #96 daemon prio=5 os_prio=0 tid=0x00007f73cd07f800 nid=0x7a8a waiting on condition [0x00007f7321248000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000003c04d48e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForSignalOrTimeout(DefaultServerMonitor.java:238)
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.waitForNext(DefaultServerMonitor.java:219)
	at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:168)
	- locked <0x00000003c04d4408> (a com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable)
	at java.lang.Thread.run(Thread.java:745)

If one of those threads were to exit, that member's state would no longer be updated on replica set object's state.

What we observed

While our java application was running, the following log appeared:

2016-08-26T03:25:33.561+0000 [scrubbedThreadName] WARN  org.mongodb.driver.connection [SLF4JLogger.java.warn:86] - Got socket exception on connection [connectionId{localValue:3565883, serverValue:689402}] to scubbedHostname:scrubbedPort. All connections to scrubbedHostname:scrubbedPort will be closed.

That was immediately followed by some threads receiving the error:

com.mongodb.MongoSocketReadException: Prematurely reached end of stream
	at com.mongodb.connection.SocketStream.read(SocketStream.java:88) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:297) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:54) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.query(DefaultServerConnection.java:209) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:496) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo.execute(Mongo.java:772) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo$2.execute(Mongo.java:759) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:851) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:152) ~[mongo-java-driver-3.2.2.jar:na]
	at <ourcode> ~[classes/:na]
java.lang.IllegalThreadStateException: null
	at java.lang.Thread.start(Thread.java:705) ~[na:1.8.0_25]
	at com.mongodb.connection.DefaultServerMonitor.invalidate(DefaultServerMonitor.java:88) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.invalidate(DefaultServer.java:120) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.handleThrowable(DefaultServer.java:150) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.access$500(DefaultServer.java:38) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:161) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.query(DefaultServerConnection.java:209) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:496) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo.execute(Mongo.java:772) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo$2.execute(Mongo.java:759) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:851) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:152) ~[mongo-java-driver-3.2.2.jar:na]
	at <our code>
com.mongodb.MongoSocketReadException: Prematurely reached end of stream
	at com.mongodb.connection.SocketStream.read(SocketStream.java:88) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:491) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:221) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:102) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:435) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:297) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:54) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.query(DefaultServerConnection.java:209) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:496) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo.execute(Mongo.java:772) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo$2.execute(Mongo.java:759) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:851) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:152) ~[mongo-java-driver-3.2.2.jar:na]
	at <our code>
java.lang.IllegalThreadStateException: null
	at java.lang.Thread.start(Thread.java:705) ~[na:1.8.0_25]
	at com.mongodb.connection.DefaultServerMonitor.invalidate(DefaultServerMonitor.java:88) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.invalidate(DefaultServer.java:120) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.handleThrowable(DefaultServer.java:150) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer.access$500(DefaultServer.java:38) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:161) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.DefaultServerConnection.query(DefaultServerConnection.java:209) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:496) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation$1.call(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo.execute(Mongo.java:772) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo$2.execute(Mongo.java:759) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.initializeCursor(DBCursor.java:851) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCursor.hasNext(DBCursor.java:152) ~[mongo-java-driver-3.2.2.jar:na]
	at <our code>

Subsequent attempts to reach the primary resulted in the following:

com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches WritableServerSelector. Client view of cluster state is {type=REPLICA_SET, servers=[{address=scrubbedProblemHost:scrubbedPort, type=UNKNOWN, state=CONNECTING}, {address=scrubbedArbiter, type=REPLICA_SET_ARBITER, roundTripTime=7.8 ms, state=CONNECTED}, {address=scrubbedWorkingHost, type=REPLICA_SET_SECONDARY, roundTripTime=1.0 ms, state=CONNECTED}, {address=scrubbedWorkingHost, type=REPLICA_SET_SECONDARY, roundTripTime=0.9 ms, state=CONNECTED}]
	at com.mongodb.connection.BaseCluster.createTimeoutException(BaseCluster.java:369) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.connection.BaseCluster.selectServer(BaseCluster.java:101) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:75) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.<init>(ClusterBinding.java:71) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.binding.ClusterBinding.getWriteConnectionSource(ClusterBinding.java:68) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:219) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:111) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.operation.CreateIndexesOperation.execute(CreateIndexesOperation.java:66) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo.execute(Mongo.java:781) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.Mongo$2.execute(Mongo.java:764) ~[mongo-java-driver-3.2.2.jar:na]
	at com.mongodb.DBCollection.createIndex(DBCollection.java:1541) ~[mongo-java-driver-3.2.2.jar:na]

Server logs at the time of the first error shows:

2016-08-26T03:25:26.527+0000 I QUERY    [conn689402] assertion 13435 not master and slaveOk=false <query information>
2016-08-26T03:25:26.528+0000 I NETWORK  [conn689402] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server [10.20.0.204:48093]

Before restarting the application we got a thread dump which does not seem to show any threads for the problem host. We're conjecturing that server monitoring thread allowed an exception to escape, causing it to die and it was not replaced.

I can provide fuller logs upon request.



 Comments   
Comment by Jesse English [ 11/Nov/17 ]

Is there any solution to this issue? We are experiencing an identical issue to this.

Running this on a _localhost _ mongo (v3.4.1) connection on CentOS using morphia (v1.3.2) and java-mongo-driver (v3.4.2)
Java Exception:

om.mongodb.MongoSocketReadException: Prematurely reached end of stream
        at com.mongodb.connection.SocketStream.read(SocketStream.java:88)
        at com.mongodb.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:494)
        at com.mongodb.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:224)
        at com.mongodb.connection.UsageTrackingInternalConnection.receiveMessage(UsageTrackingInternalConnection.java:96)
        at com.mongodb.connection.DefaultConnectionPool$PooledConnection.receiveMessage(DefaultConnectionPool.java:440)
        at com.mongodb.connection.WriteCommandProtocol.receiveMessage(WriteCommandProtocol.java:262)
        at com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:104)
        at com.mongodb.connection.DeleteCommandProtocol.execute(DeleteCommandProtocol.java:64)
        at com.mongodb.connection.DeleteCommandProtocol.execute(DeleteCommandProtocol.java:38)
        at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
        at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
        at com.mongodb.connection.DefaultServerConnection.deleteCommand(DefaultServerConnection.java:162)
        at com.mongodb.operation.DeleteOperation.executeCommandProtocol(DeleteOperation.java:90)
        at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:139)
        at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:133)
        at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:422)
        at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:413)
        at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133)
        at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60)
        at com.mongodb.Mongo.execute(Mongo.java:845)
        at com.mongodb.Mongo$2.execute(Mongo.java:828)
        at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:342)
        at com.mongodb.DBCollection.remove(DBCollection.java:634)
        at org.mongodb.morphia.DatastoreImpl.delete(DatastoreImpl.java:176)
        at org.mongodb.morphia.DatastoreImpl.delete(DatastoreImpl.java:201)
        at com.eyesover.common.data.collections.WordCountCollection.deleteWordCloud(WordCountCollection.java:98)
        at com.eyesover.analyzer.wordcloud.WordCloudBuilder.run(WordCloudBuilder.java:62)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

mongo log

2017-11-10T20:32:43.176+0000 F -        [conn186] out of memory.
 
 0x558d06cc46b1 0x558d06cc3ce4 0x558d06c31a81 0x558d05fd5385 0x558d065968a8 0x558d06243175 0x558d06243d29 0x558d061bb57f 0x558d061bcc61 0x558d067d55c0 0x558d063d88a2 0x558d063da8a6 0x558d05fd7ffd 0x558d05fd892d 0x558d06c2c401 0x7f68f9830e25 0x7f68f955e34d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"558D0574C000","o":"15786B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"558D0574C000","o":"1577CE4","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"558D0574C000","o":"14E5A81","s":"_ZN5mongo12mongoReallocEPvm"},{"b":"558D0574C000","o":"889385","s":"_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi"},{"b":"558D0574C000","o":"E4A8A8","s":"_ZN5mongo26appendCursorResponseObjectExNS_10StringDataENS_9BSONArrayEPNS_14BSONObjBuilderE"},{"b":"558D0574C000","o":"AF7175"},{"b":"558D0574C000","o":"AF7D29"},{"b":"558D0574C000","o":"A6F57F","s":"_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE"},{"b":"558D0574C000","o":"A70C61","s":"_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE"},{"b":"558D0574C000","o":"10895C0","s":"_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE"},{"b":"558D0574C000","o":"C8C8A2"},{"b":"558D0574C000","o":"C8E8A6","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"558D0574C000","o":"88BFFD","s":"_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE"},{"b":"558D0574C000","o":"88C92D"},{"b":"558D0574C000","o":"14E0401"},{"b":"7F68F9829000","o":"7E25"},{"b":"7F68F9466000","o":"F834D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.2.2.el7.x86_64", "version" : "#1 SMP Tue Sep 12 22:26:13 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "558D0574C000", "elfType" : 3, "buildId" : "94C7FAB092E567C9338D13DB9B68751363D15EFD" }, { "b" : "7FFFDE466000", "elfType" : 3, "buildId" : "E980D6062AB656BDA6DC83B92828F3563BD4ABEA" }, { "b" : "7F68FA7CA000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "ED0AC7DEB91A242C194B3DEF27A215F41CE43116" }, { "b" : "7F68FA369000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "BC0AE9CA0705BEC1F0C0375AAD839843BB219CB1" }, { "b" : "7F68FA161000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "5B629F64AC6EA7AAC602BE56ED834BB6398C72AC" }, { "b" : "7F68F9F5D000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "8CC796BA7CA23193AD753D8625018B61264724BE" }, { "b" : "7F68F9C5B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5FAABA77B1848347CEC4B0CE7B31811D7D00D2FA" }, { "b" : "7F68F9A45000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "361D73E3AA2ACE6AF32B03D1B74A22E1FF68AB2D" }, { "b" : "7F68F9829000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "B8FBCA68CA56E79556BF7884DACF89504096ADEB" }, { "b" : "7F68F9466000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "C3F28802314AF4EE866BF8D2E1B506B7BBF34CF6" }, { "b" : "7F68FAA3C000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "962B8EEE329A2C57184AF2780756ED2035DEAAC0" }, { "b" : "7F68F9219000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DA322D74F55A0C4293085371A8D0E94B5962F5E7" }, { "b" : "7F68F8F31000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B69E63024D408E400401EEA6815317BDA38FB7C2" }, { "b" : "7F68F8D2D000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7F68F8AFA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A48639BF901DB554479BFAD114CB354CF63D7D6E" }, { "b" : "7F68F88E4000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7F68F86D6000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "6FDF5B013FD2739D304CFB9D723DCBC149EE03C9" }, { "b" : "7F68F84D2000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F68F82B8000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "FF4E72F4E574E143330FB3C66DB51613B0EC65EA" }, { "b" : "7F68F8091000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7F68F7E2F000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x558d06cc46b1]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x558d06cc3ce4]
 mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x558d06c31a81]
 mongod(_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi+0x55) [0x558d05fd5385]
 mongod(_ZN5mongo26appendCursorResponseObjectExNS_10StringDataENS_9BSONArrayEPNS_14BSONObjBuilderE+0xDC8) [0x558d065968a8]
 mongod(+0xAF7175) [0x558d06243175]
 mongod(+0xAF7D29) [0x558d06243d29]
 mongod(_ZN5mongo7Command3runEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS3_21ReplyBuilderInterfaceE+0x4FF) [0x558d061bb57f]
 mongod(_ZN5mongo7Command11execCommandEPNS_16OperationContextEPS0_RKNS_3rpc16RequestInterfaceEPNS4_21ReplyBuilderInterfaceE+0xF81) [0x558d061bcc61]
 mongod(_ZN5mongo11runCommandsEPNS_16OperationContextERKNS_3rpc16RequestInterfaceEPNS2_21ReplyBuilderInterfaceE+0x240) [0x558d067d55c0]
 mongod(+0xC8C8A2) [0x558d063d88a2]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x746) [0x558d063da8a6]
 mongod(_ZN5mongo23ServiceEntryPointMongod12_sessionLoopERKSt10shared_ptrINS_9transport7SessionEE+0x1FD) [0x558d05fd7ffd]
 mongod(+0x88C92D) [0x558d05fd892d]
 mongod(+0x14E0401) [0x558d06c2c401]
 libpthread.so.0(+0x7E25) [0x7f68f9830e25]
 libc.so.6(clone+0x6D) [0x7f68f955e34d]
-----  END BACKTRACE  -----

Comment by Jeffrey Yemin [ 26/Aug/16 ]

Closing as duplicate of JAVA-2238.

Comment by Jeffrey Yemin [ 26/Aug/16 ]

This is almost certainly another manifestation of JAVA-2238. The smoking gun is that IllegalThreadStateException thrown from Thread.start in DefaultServerMonitor. This could happen due to the lack of synchronization in DefaultServerMonitor#invalidate, which was addressed in the fix to that bug.

Generated at Thu Feb 08 08:56:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.