Uploaded image for project: 'Java Driver'
  1. Java Driver
  2. JAVA-4122

Getting Exceptions when CPU starving the driver

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Gone away
    • Icon: Unknown Unknown
    • None
    • 4.2.3
    • Scala
    • None

    Description

      Using the Scala driver (4.2.3) when we switched over to the use of an Atlas cluster we started noticing some instabilities in our application. 

      We where for instance getting

       com.mongodb.MongoInternalException: Unexpected runtime exception
      	at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:638)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:591)
      	at com.mongodb.internal.connection.InternalStreamConnection.access$1100(InternalStreamConnection.java:78)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:701)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:686)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:581)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:578)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$4(AsynchronousTlsChannel.java:122)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$5(AsynchronousTlsChannel.java:122)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doRead(AsynchronousTlsChannelGroup.java:569)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processRead$5(AsynchronousTlsChannelGroup.java:475)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.processRead(AsynchronousTlsChannelGroup.java:472)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.loop(AsynchronousTlsChannelGroup.java:407)
      	at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.IllegalArgumentException: capacity < 0: (-16 < 0)
      	at java.base/java.nio.Buffer.createCapacityException(Unknown Source)
      	at java.base/java.nio.ByteBuffer.allocate(Unknown Source)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.createNew(PowerOfTwoBufferPool.java:90)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.getByteBuffer(PowerOfTwoBufferPool.java:82)
      	at com.mongodb.internal.connection.PowerOfTwoBufferPool.getBuffer(PowerOfTwoBufferPool.java:77)
      	at com.mongodb.internal.connection.AsynchronousChannelStream.readAsync(AsynchronousChannelStream.java:111)
      	at com.mongodb.internal.connection.AsynchronousChannelStream.readAsync(AsynchronousChannelStream.java:107)
      	at com.mongodb.internal.connection.InternalStreamConnection.readAsync(InternalStreamConnection.java:578)
      	... 23 common frames omitted

      and

      com.mongodb.MongoInternalException: The reply message length 680237824 is greater than the maximum message length 48000000
      	at com.mongodb.internal.connection.MessageHeader.<init>(MessageHeader.java:41)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:700)
      	at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback.onResult(InternalStreamConnection.java:686)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:581)
      	at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:578)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250)
      	at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$4(AsynchronousTlsChannel.java:122)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$5(AsynchronousTlsChannel.java:122)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doRead(AsynchronousTlsChannelGroup.java:569)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processRead$5(AsynchronousTlsChannelGroup.java:475)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.processRead(AsynchronousTlsChannelGroup.java:472)
      	at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.loop(AsynchronousTlsChannelGroup.java:407)
      	at java.base/java.lang.Thread.run(Unknown Source) 

      It seems that the driver is throwing these exceptions when it's starved for CPU. We are running our application on K8s and have specific resources set. After doing some load testing the exception seems to disappear after we double the cpu resources to be able to handle 50% of the same traffic. 

      Can it be something is not thread safe?

      Why we need at least double the cpu resources I don't know yet.

      Attachments

        Activity

          People

            jeff.yemin@mongodb.com Jeffrey Yemin
            jork.zijlstra@nielsen.com Jork Zijlstra
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: