[JAVA-850] DBObject of size (smaller than 16 mb) is over Max BSON size 16777216 Created: 08/Jun/13  Updated: 31/Mar/15  Resolved: 25/Jun/13

Status: Closed
Project: Java Driver
Component/s: Connection Management
Affects Version/s: 2.11.0, 3.0.0
Fix Version/s: 3.0.0

Type: Bug Priority: Critical - P2
Reporter: rohit.nijhawan@10gen.com Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Java Driver on linux, osx with mongo 2.4.2


Attachments: Text File Rohit.java    

 Description   

Exception in thread "main" com.mongodb.MongoInternalException: DBObject of size 5242897 is over Max BSON size 16777216

Perhaps it IS the intended behaviour to not allow queries or remove operation's query parameters to exceed 4mb - the limit in Bytes.MAX_OBJECT_SIZE, but this message is very strange, and can easily throw off a developer.

Note that in OutMessage.java we check max on one thing _mongo.getConnector().getMaxBsonObjectSize()

...but report it on something else - which incidentally initializes it correctly.
_mongo.getMaxBsonObjectSize()

if (objectSize > Math.max(_mongo.getConnector().getMaxBsonObjectSize(), Bytes.MAX_OBJECT_SIZE))

{ throw new MongoInternalException("DBObject of size " + objectSize + " is over Max BSON size " + _mongo.getMaxBsonObjectSize()); }

 Comments   
Comment by Jeffrey Yemin [ 31/Mar/15 ]

Closing all resolved 3.0.0 issues, as 3.0.0 has been tagged and released.

Comment by Jeffrey Yemin [ 25/Jun/13 ]

Code is re-written in 3.0.x branch, and this is fixed by design.

Comment by Jeffrey Yemin [ 13/Jun/13 ]

OK, we'll definitely get this fixed, just not sure exactly when.

Comment by rohit.nijhawan@10gen.com [ 13/Jun/13 ]

A customer I am working with is performing a remove with field:{$in:[v1,v2,v3,v4,....]}
I was trying to simulate a completely different issue for them when I ran into this.
This code is not yet in production where their dataset could be much larger. Recently they confirmed that this remove operation is being done in batches with fixed array sizes once I reminded them that it could run into the 16 mb BSON limit.

It's a problem they've avoided thus far.

Comment by Jeffrey Yemin [ 12/Jun/13 ]

Do you have an actual use case for having such a large remove or update message?

Comment by rohit.nijhawan@10gen.com [ 12/Jun/13 ]

Yes I concur. If you run any other operation prior to it, things start and continue to work.
And, I have tested insert + find. Both of those will not fail. It's remove and update that throw the exception.

Comment by Jeffrey Yemin [ 11/Jun/13 ]

Thanks for the test case. I can see that this will only occur if the very first message you create is larger that 4MB. For example, this program works correctly:

public class JAVA850 {
    public static void main(String[] args) throws UnknownHostException {
        Mongo mongo = new MongoClient("localhost", 27017);
        DB db = mongo.getDB("test");
        DBCollection collection = db.getCollection("test");
 
        // send a small message first
        collection.findOne();
 
        DBObject doc = new BasicDBObject();
 
        int i = 0;
        char[] a = new char[1024 * 1024 * 5];
        while (i < a.length) {
            a[i] = 'a';
            i++;
        }
 
        String value = String.valueOf(a);
        doc.put("value", value);
        collection.remove(doc);
    }
}

Comment by rohit.nijhawan@10gen.com [ 11/Jun/13 ]

The reproducible case is in the file I attached.
The problem is not random. The code is in OutMessage.java

The stacktrace is pasted below
I have connected to a mongos and a mongod and in each case I got the same problem.

Exception in thread "main" com.mongodb.MongoInternalException: DBObject of size 5242906 is over Max BSON size 16777216
at com.mongodb.OutMessage.putObject(OutMessage.java:291)
at com.mongodb.OutMessage.writeRemove(OutMessage.java:189)
at com.mongodb.OutMessage.remove(OutMessage.java:69)
at com.mongodb.DBApiLayer$MyCollection.remove(DBApiLayer.java:265)
at com.mongodb.DBCollection.remove(DBCollection.java:251)
at com.mongodb.DBCollection.remove(DBCollection.java:273)
at com.a.Rohit.main(Rohit.java:29)

Comment by Jeffrey Yemin [ 08/Jun/13 ]

A few questions:

  • Do you have a reproducible case, or does it happen seemingly at random?
  • What's the full stack trace?
  • How are you connecting? Single ServerAddress, List of ServerAddress, and if the latter, are they mongos servers or replica set members?
Generated at Thu Feb 08 08:53:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.