[JAVA-556] Wrong exception on too large query when Mongo object is not reused Created: 20/Apr/12  Updated: 31/Mar/15  Resolved: 25/Jun/13

Status: Closed
Project: Java Driver
Component/s: Cluster Management
Affects Version/s: 2.7.2
Fix Version/s: 3.0.0

Type: Bug Priority: Major - P3
Reporter: Kay Agahd Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux 64 Bit



 Description   

I'm using the Java driver v2.7.2 to query a sharded system consisting of 4 shards, each 3 replicas, running linux-mongodb-v2.0.1, v2.0.2 and v2.0.4. All mongos are running v2.0.1.

The query I'm executing is of this type:
{_id:{$in:[1,2,3,4]}}

When the inlist becomes too big (more than 265,000 id's in my case), I get the following exception, where max BSON size is greater than 2.6 times of the DBObject size:

Exception in thread "main" com.mongodb.MongoInternalException:
DBObject of size 6442515 is over Max BSON size 16777216
at com.mongodb.OutMessage.putObject(OutMessage.java:147)
at com.mongodb.OutMessage._appendQuery(OutMessage.java:85)
at com.mongodb.OutMessage.query(OutMessage.java:44)
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

When I reuse the mongo object, I don't get the exception. Moreover, I can put up to 995,000 id's into the list without any problem.
Exceeding the inlist size results in an appropriate exception:

Exception in thread "main" com.mongodb.MongoException: could not
initialize cursor across all shards because : combinatorial limit of
$in partitioning of result set exceeded @ offerStoreDE4/
s115:27018,s132:27018,s125:27018
at com.mongodb.MongoException.parse(MongoException.java:82)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:312)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

However, when it's the first usage of the Mongo object then it's not yet reused and will fail as described above.

For more details, please see:
http://groups.google.com/group/mongodb-user/browse_thread/thread/3b96668396f0f9d4



 Comments   
Comment by Jeffrey Yemin [ 31/Mar/15 ]

Closing all resolved 3.0.0 issues, as 3.0.0 has been tagged and released.

Comment by Jeffrey Yemin [ 25/Jun/13 ]

See https://github.com/jyemin/mongo-java-driver/commit/fa0880c#L30R89

Comment by Jeffrey Yemin [ 20/Apr/12 ]

Ah, I mis-read the code. The first call is to Mongo.getConnector().getMaxBsonObjectSize(), while the second is to Mongo.getMaxBsonObjectSize(), which actually tries to fetch the value from the server.

Comment by Kay Agahd [ 20/Apr/12 ]

Yes, I can reproduce it at any time. The size of the DBObject in the error message may vary though as for example here:

Exception in thread "main" com.mongodb.MongoInternalException: DBObject of size 4208915 is over Max BSON size 16777216
at com.mongodb.OutMessage.putObject(OutMessage.java:147)
at com.mongodb.OutMessage._appendQuery(OutMessage.java:85)
at com.mongodb.OutMessage.query(OutMessage.java:44)
at com.mongodb.OutMessage.query(OutMessage.java:38)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:303)
at com.mongodb.DBCursor._check(DBCursor.java:369)
at com.mongodb.DBCursor._hasNext(DBCursor.java:498)
at com.mongodb.DBCursor.hasNext(DBCursor.java:523)

> Am I correct that you're connecting to a replica set (by giving a list of seed servers)?

No, I'm connecting to one mongos like this:
final Mongo m = new Mongo("sx210.ipx:27018")

Comment by Jeffrey Yemin [ 20/Apr/12 ]

Is this reproducible always? It looks like a race condition. What's happening is that Mongo.getMaxBsonObjectSize() will return 4MB when the Mongo instance is first created, since that was the default prior to 1.8.0. When using a replica set, the value is updated by a background thread. When running without a replica set, it's updated the first time the driver needs to find the primary/master. Am I correct that you're connecting to a replica set (by giving a list of seed servers)?

A possible workaround for you, until we can fix this, is to create the Mongo instance and then sleep in a loop until Mongo.getMaxBsonObjectSize() returns 16MB.

The confusing log message is just a symptom. Mongo.getMaxBsonObjectSize() is called again when constructing the exception, and by the time the second call is made, the value has been updated to 16MB.

Generated at Thu Feb 08 08:52:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.