[JAVA-1591] Setting DBCursor.batchSize() does not seem to work for certain queries. Created: 10/Dec/14  Updated: 10/Dec/14  Resolved: 10/Dec/14

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: 2.11.4, 2.12.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Severin Gehwolf Assignee: Ross Lawley
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Linux, x86_64, Fedora 20
$ mongod --version
db version v2.4.6
Wed Dec 10 12:59:53.180 git version: nogitversion


Attachments: Java Source File TestMongo.java    

 Description   

Disclaimer:
I'm not sure if this is a problem with the Java driver or a more general problem. It seems to reproduce with the mongo shell too. Feel free to re-assign as appropriate.

Problem:
DBCursor's which are the result of a query which selects documents via a range query ($lt and $gt) and has a sort on the same field applied don't work correctly when DBCursor.batchSize() has been used before calling DBCursor.hasNext()/next() the first time. It may be a more general problem related to setting the batch size.

I'm attaching a reproducer for this issue. The expected output should be similar to the following:

Before insert count: 0 (if > 0 all items got removed)
Inserting data (2004 items total) ..................... done.
Reached count 2004
TestRecord461
[...]
TestRecord1537
Cursor actual numGetMore() == 10. Expected 10.
Actual cursor size was 1077
Query was: { "$and" : [ { "timeStamp" : { "$gt" : 1418211812266}} , { "timeStamp" : { "$lt" : 1418211835526}}]}
QUERIES done! Test SUCCESSFUL! Got ALL 1077 results.
Test finished.

Actual output I get is:

Before insert count: 0 (if > 0 all items got removed)
Inserting data (2004 items total) ..................... done.
Reached count 2004
TestRecord461
[...]
TestRecord561
Cursor actual numGetMore() == 0. Expected 10.
Actual cursor size was 1077
Query was: { "$and" : [ { "timeStamp" : { "$gt" : 1418211812266}} , { "timeStamp" : { "$lt" : 1418211835526}}]}
QUERIES done! Test FAILED! Got only 100 results (out of 1077)
Test finished.

If the tester.removeData() line gets commented out (i.e. keep data in the DB) one can reproduce the same results with the mongo shell. Code as follows:

var query = { "$and" : [ { "timeStamp" : { "$gt" : 1418211812266}} , { "timeStamp" : { "$lt" : 1418211835526}}]};
var myCursor = db["test-mongo-java"].find(query).sort({"timeStamp": 1})
var count = 0; myCursor.batchSize(100).forEach(function(a) { print(a.name); count++; }); print(count)

That forEach() call also only prints 100 elements while myCursor.size() prints > 100 (e.g. 1077).

The Java reproducer can be run via (provided a mongod is running on local host on port 27518):

javac -cp path/to/mongo-java-driver.jar TestMongo.java
java TestMongo



 Comments   
Comment by Ross Lawley [ 10/Dec/14 ]

Hi,

This is not an issue with the Java driver but rather you are hitting a known bug with MongoDB 2.4 (SERVER-12438) which was fixed in MongoDB 2.6.

The bug was when using both batchSize and sort on a cursor where the sort was on an unindexed field. This caused the cursor to return fewer items than it should. Barring an upgrade of MongoDB if you remove the sort or index the timestamp field or use the default batchSize and then it will work as expected.

For future reference, a better place for questions regarding MongoDB usage or the Java driver specifics is the mongodb-user mailinglist or stackoverflow as you will reach a boarder audience there and should get a faster response. This JAVA project is specifically for driver based bugs or issues.

Hope that helps,

Ross

Generated at Thu Feb 08 08:55:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.