[JAVA-2408] java-mongodb-driver test case failure Created: 14/Dec/16  Updated: 18/Sep/17  Resolved: 18/Sep/17

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: ravi_ss Assignee: Ross Lawley
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File log_mongodbServer.txt     Text File log_run1.txt     Text File log_run2.txt     Text File log_withLeakDetect.txt    
Issue Links:
Duplicate

 Description   

I try to run testcases for java-mongodb-driver, before making any changes to the source code.
7 testcases are failed as shown below.

$ ./gradlew check -Dorg.mongodb.test.uri=mongodb://<ip>:27017/

:driver-core:test
i) One test with error "ns not found" as follow:
should deliver started and completed command events for split unacknowleded inserts

com.mongodb.MongoCommandException: Command failed with error -1: 'ns not found' on server <ip>:27017. The full response is

{ "ok" : 0.0, "errmsg" : "ns not found" }

at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
at com.mongodb.connection.WriteProtocolCommandEventSpecification.should deliver started and completed command events for split unacknowleded inserts(WriteProtocolCommandEventSpecification.groovy:145)

com.mongodb.MongoCommandException: Command failed with error -1: 'ns not found' on server <ip>:27017. The full response is

{ "ok" : 0.0, "errmsg" : "ns not found" }

at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
at com.mongodb.connection.WriteProtocolCommandEventSpecification.should deliver started and completed command events for split unacknowleded inserts(WriteProtocolCommandEventSpecification.groovy:145)

ii) Following 6 testcases failed with error as shown
should continue writing on write error when an unordered unacknowledged inserts must be split
should execute split acknowledged inserts
should execute split unacknowledged inserts
should not report write errors on split unacknowledged inserts
should report write errors on split acknowledged inserts
should stop writing on write error when an ordered unacknowledged inserts must be split

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 50367344, max: 67108864)
at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:624)
at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:578)
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68)
at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:625)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:265)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
at com.mongodb.connection.netty.NettyStream.getBuffer(NettyStream.java:93)
at com.mongodb.connection.InternalStreamConnection.getBuffer(InternalStreamConnection.java:514)
at com.mongodb.connection.ByteBufferBsonOutput.getByteBufferAtIndex(ByteBufferBsonOutput.java:91)
at com.mongodb.connection.ByteBufferBsonOutput.getCurrentByteBuffer(ByteBufferBsonOutput.java:86)
at com.mongodb.connection.ByteBufferBsonOutput.writeBytes(ByteBufferBsonOutput.java:62)
at org.bson.io.OutputBuffer.writeBytes(OutputBuffer.java:51)
at org.bson.BsonBinaryWriter.doWriteBinaryData(BsonBinaryWriter.java:163)
at org.bson.AbstractBsonWriter.writeBinaryData(AbstractBsonWriter.java:360)
at org.bson.codecs.BsonBinaryCodec.encode(BsonBinaryCodec.java:31)
at org.bson.codecs.BsonBinaryCodec.encode(BsonBinaryCodec.java:28)
at org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
at org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136)
at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115)
at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41)
at com.mongodb.connection.RequestMessage.addDocument(RequestMessage.java:253)
at com.mongodb.connection.RequestMessage.addCollectibleDocument(RequestMessage.java:219)
at com.mongodb.connection.InsertMessage.encodeMessageBodyWithMetadata(InsertMessage.java:73)
at com.mongodb.connection.RequestMessage.encodeWithMetadata(RequestMessage.java:160)
at com.mongodb.connection.WriteProtocol.execute(WriteProtocol.java:89)
at com.mongodb.connection.InsertProtocol.execute(InsertProtocol.java:68)
at com.mongodb.connection.InsertProtocol.execute(InsertProtocol.java:39)
at com.mongodb.connection.ProtocolTestHelper.execute(ProtocolTestHelper.java:31)
at com.mongodb.connection.WriteProtocolSpecification.should continue writing on write error when an unordered unacknowledged inserts must be split(WriteProtocolSpecification.groovy:201)

java.lang.ClassCastException: io.netty.util.internal.OutOfDirectMemoryError incompatible with java.lang.RuntimeException
at com.mongodb.async.FutureResultCallback.get(FutureResultCallback.java:77)
at com.mongodb.connection.ProtocolTestHelper.execute(ProtocolTestHelper.java:29)
at com.mongodb.connection.WriteProtocolSpecification.should continue writing on write error when an unordered unacknowledged inserts must be split(WriteProtocolSpecification.groovy:201)

Initially I used to get test search not found error, solved after mongodb server started with text search enabled.

Regards,

Hadi



 Comments   
Comment by Ross Lawley [ 18/Sep/17 ]

Hi ravi_ss,

We have been unable to reproduce the errors and as such we're closing this ticket as "Cannot Reproduce". We've been unable to identify a probable cause, so at this time have no way of taking this ticket forward.

If you are able to provide any new information or insight we'll happily reopen this ticket and investigate further.

Ross

Comment by Ross Lawley [ 09/Jan/17 ]

Hi ravi_ss,

I've not been able to reproduce the issue locally or via our test machines.

At the moment I'm blocked on this issue until I can either reproduce locally or on a test machine.

Ross

Comment by ravi_ss [ 21/Dec/16 ]

Hi Ross,
Any updates on this issue

Ravi_SS

Comment by ravi_ss [ 16/Dec/16 ]

Hi,
I have uploaded logs which were executed with -Dio.netty.leakDetection.level=advanced flag and also the mogodb server log

Thanks & regards,
Ravi_SS

Comment by Ross Lawley [ 16/Dec/16 ]

Hi ravi_ss,

Many thanks for all your help with trying to debug this issue, unfortunately, I'm still no closer to identifying the cause as I'm not able to replicate the failures locally, on travis or on our own CI servers.

Could you run the tests with extra netty leak detection on by passing the command line flag: -Dio.netty.leakDetection.level=advanced and attach the logs to the ticket. Could you also provide the MongoD logs for the duration of the test.

Perhaps that will help shed more light on the cause for the failures.

Ross

Comment by ravi_ss [ 16/Dec/16 ]

Hi Ross,

I am using netty 4.1.5.Final, Also I have tried with 4.1.6.Final facing the same issue.

Thanks & regards,
Ravi_SS

Comment by Ross Lawley [ 15/Dec/16 ]

Hi,

My apologies, I misread the clirr baseline version to be the driver version. Could you confirm the version of netty you are using? Travis is running against 4.1.5.Final without any memory leaks.

The 'ns not found' errors in the WriteProtocolCommandEventSpecification looks to be a race condition in the test when calling the cleanup.

Ross

Comment by ravi_ss [ 15/Dec/16 ]

Hi,
I have build the java mongodb version:r3.4.0
mongo-java-driver]$ git branch

  • (detached from r3.4.0)
    master

the mongodb server is

  1. ./mongod --version
    db version v2.4.9-rc0

I can see the similar log in https://travis-ci.org/mongodb/mongo-java-driver/jobs/183972445

Thanks & regards,
Ravi_SS

Comment by Ross Lawley [ 15/Dec/16 ]

Hi,

Could it be running against the driver version 3.2.0? I'm seeing the following in the logs:

All projects evaluated.
bson: baseline has been set to org.mongodb:bson:3.2.0
driver: baseline has been set to org.mongodb:mongodb-driver:3.2.0
driver-async: baseline has been set to org.mongodb:mongodb-driver-async:3.2.0
driver-core: baseline has been set to org.mongodb:mongodb-driver-core:3.2.0

There have been over 50 bugfixes since version 3.2.0. Including JAVA-2302, which is a fix for a memory leak with netty 4.1 and a warning for a leak can be observed in your tests.

Ross

Comment by ravi_ss [ 15/Dec/16 ]

Hi Ross,
I have built the latest version r3.4.0.

sorry for that,

Regards,
Ravi_SS

Comment by Ross Lawley [ 15/Dec/16 ]

Hi ravi_ss,

Can you confirm the version of the driver is r3.0.4 ? The test logs show a test in the WriteProtocolCommandEventSpecification is failing but that test case wasn't present in version 3.0.4 of the driver.

Ross

Comment by ravi_ss [ 15/Dec/16 ]

Hi Ross,
Thank you for your response,
I have built mongodb-driver for r3.0.4 version
platform s390x system
running the mongodb on remote server version 2.4, single node

Attached the logs of first and second run,
Increased the memory available for jdk by
export GRADLE_OPTS=-XX:MaxDirectMemorySize=10g

Thanks & regards,
Ravi_SS

Comment by Ross Lawley [ 15/Dec/16 ]

Thanks for that @ravi_ss,

Can you provide the other information to help shed more light on the cause:

  • What *driver version *are you running the tests on? The master branch or a tagged version?
  • What platform are you running it on?
  • What version of MongoDB are you running?
  • What topology are you running? single node, replicaSet, sharded cluster, sharded cluster with replicaSets

Ross

Comment by ravi_ss [ 15/Dec/16 ]

Attached the logs of first and second run,
Increased the memory available for jdk by
export GRADLE_OPTS=-XX:MaxDirectMemorySize=10g

Comment by Ross Lawley [ 14/Dec/16 ]

Hi ravi_ss,

Thanks for the ticket - you are correct the text search must be enabled for the test suite. Errors aren't expected when running the tests, we have a comprehensive continuous integration suite that runs the tests over various configurations. As such I have a few questions to help diagnose the cause:

  • What driver version are you running the tests on? The Master branch or a tagged version?
  • What platform are you running it on?
  • What version of MongoDB are you running?
  • What topology are you running? single node, replicaSet, sharded cluster, sharded cluster with replicaSets

Can you attach logs from the whole test suite? I would like to see where Spock errors when reporting the 'ns not found' issues.

The out of memory issues indicate you may need to increase the memory available to your JDK - you can pass the -XX:MaxDirectMemorySize= flag in the command line.

Ross

Generated at Thu Feb 08 08:57:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.