[JAVA-2179] com.mongodb.connection.ByteBufferBsonOutput generated negative buffer size by integer overflow Created: 28/Apr/16  Updated: 19/Oct/16  Resolved: 03/May/16

Status: Closed
Project: Java Driver
Component/s: BSON
Affects Version/s: 3.2.2
Fix Version/s: 0.8

Type: Bug Priority: Major - P3
Reporter: Jan S. Assignee: Jeffrey Yemin
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

In the class com.mongodb.connection.ByteBufferBsonOutput there is some strange buffer size calculation algorithm:

bufferList.add(bufferProvider.getBuffer(Math.min(INITIAL_BUFFER_SIZE << index, MAX_BUFFER_SIZE)));

for index=21 the part INITIAL_BUFFER_SIZE << index creates an *integer overflow*, resulting in the negative value: -2147483648
This causes the overall buffer size calculated in the presented line to become negative which will cause in the end the following exception:

{{java.lang.IllegalArgumentException: null
at java.nio.ByteBuffer.allocate(ByteBuffer.java:334) ~
at com.mongodb.internal.connection.PowerOfTwoBufferPool.createNew(PowerOfTwoBufferPool.java:84) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.internal.connection.PowerOfTwoBufferPool.getBuffer(PowerOfTwoBufferPool.java:76) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.SocketStream.getBuffer(SocketStream.java:69) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.InternalStreamConnection.getBuffer(InternalStreamConnection.java:511) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.UsageTrackingInternalConnection.getBuffer(UsageTrackingInternalConnection.java:89) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.DefaultConnectionPool$PooledConnection.getBuffer(DefaultConnectionPool.java:417) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.ByteBufferBsonOutput.getByteBufferAtIndex(ByteBufferBsonOutput.java:89) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.ByteBufferBsonOutput.getCurrentByteBuffer(ByteBufferBsonOutput.java:84) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.ByteBufferBsonOutput.writeByte(ByteBufferBsonOutput.java:73) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.io.OutputBuffer.write(OutputBuffer.java:150) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.io.OutputBuffer.writeInt32(OutputBuffer.java:59) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.io.OutputBuffer.writeInt(OutputBuffer.java:160) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.io.OutputBuffer.writeString(OutputBuffer.java:89) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.BsonBinaryWriter.doWriteString(BsonBinaryWriter.java:262) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.AbstractBsonWriter.writeString(AbstractBsonWriter.java:548) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.codecs.StringCodec.encode(StringCodec.java:31) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.codecs.StringCodec.encode(StringCodec.java:28) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:211) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.encodeMap(DBObjectCodec.java:220) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:196) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.encodeIterable(DBObjectCodec.java:269) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.writeValue(DBObjectCodec.java:198) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:128) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:61) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.CompoundDBObjectCodec.encode(CompoundDBObjectCodec.java:48) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.CompoundDBObjectCodec.encode(CompoundDBObjectCodec.java:27) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63) ~[mongo-java-driver-3.2.2.jar:na]
at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:101) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:43) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.BaseWriteCommandMessage.encodeMessageBodyWithMetadata(BaseWriteCommandMessage.java:129) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.RequestMessage.encodeWithMetadata(RequestMessage.java:160) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.WriteCommandProtocol.sendMessage(WriteCommandProtocol.java:212) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:101) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:115) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.InsertOperation.executeCommandProtocol(InsertOperation.java:76) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:141) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:133) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:230) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:221) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:133) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:60) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.Mongo.execute(Mongo.java:781) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.Mongo$2.execute(Mongo.java:764) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:333) ~[mongo-java-driver-3.2.2.jar:na]
at com.mongodb.DBCollection.insert(DBCollection.java:328) ~[mongo-java-driver-3.2.2.jar:na]
}}

Note this is not a theoretical bug report, the following exception was thrown on a productive system! Therefore please fix these strange and defect by design calculations...



 Comments   
Comment by Githook User [ 03/May/16 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Message: JAVA-2179: Avoid arithmetic overflow when writing to a large output buffer

When the byte buffer index exceeds 20, the buffer size overflows.
Branch: 3.2.x
https://github.com/mongodb/mongo-java-driver/commit/cdcaba813bf9aafc0f60e5e3180274ed30bb0c25

Comment by Githook User [ 03/May/16 ]

Author:

{u'username': u'jyemin', u'name': u'Jeff Yemin', u'email': u'jeff.yemin@10gen.com'}

Message: JAVA-2179: Avoid arithmetic overflow when writing to a large output buffer

When the byte buffer index exceeds 20, the buffer size overflows.
Branch: master
https://github.com/mongodb/mongo-java-driver/commit/67ece21a031fb3f7d59f60f6dd305b4d7258b0d1

Comment by Jan S. [ 28/Apr/16 ]

Yes, I can confirm that the document to-be inserted that triggered the exception had about 140MB of size.

The problem is that the only way to distinguish if a document exceeds the max bson size is trying to insert it and catch the BsonSerializationException. In such a case we "redirect" the document to GridFS.

Comment by Jeffrey Yemin [ 28/Apr/16 ]

Thank you for reporting this bug. I was able to reproduce it by attempting to insert a document whose BSON encoding is around 128MB, which is significantly larger that the largest that MongoDB allows (16MB).

I attempted to insert the following document:

new BasicDBObject("b", new byte[0x7FFFC00 - 150]);   // results in a message a shade under 128MB

and got this exception, as expected:

Exception in thread "main" org.bson.BsonSerializationException: Size 134216584 is larger than MaxDocumentSize 16793600

Then I attempted this one:

new BasicDBObject("b", new byte[0x7FFFC00]);   // results in a message a shade over 128MB

and I got the unexpected exception reported in the description of this issue.

Can you confirm whether the document(s) being inserted when that exception is thrown are significantly larger than 16MB? Here's one way to do it:

        MongoClient mongoClient = new MongoClient();
 
        DBCollection collection = mongoClient.getDB("test").getCollection("JAVA2179");
 
        BasicDBObject obj = new BasicDBObject("b", new byte[0x7FFFC00]);
 
        try {
            collection.insert(obj);
        } catch (IllegalArgumentException e) {
            BasicOutputBuffer buffer = new BasicOutputBuffer();
            new DBObjectCodec(mongoClient.getDefaultCodecRegistry())
                    .encode(new BsonBinaryWriter(buffer), obj, EncoderContext.builder().build());
            System.out.println(String.format("Size in hex: %x", buffer.getSize()));
        }

Generated at Thu Feb 08 08:56:32 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.