[JAVA-1174] java heap out of memory when using tailable cursor Created: 19/Apr/14  Updated: 21/May/14  Resolved: 21/May/14

Status: Closed
Project: Java Driver
Component/s: Query Operations
Affects Version/s: 2.11.2, 2.12.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: feng lu Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

java 1.6 and mongodb driver 2.11.2



 Description   

when I use this command to query the oplog of MongoDB, I got out of memory issue.

 cursor = oplogColl.find(query).sort(new BasicDBObject("$natural", 1)).addOption(Bytes.QUERYOPTION_TAILABLE).addOption(Bytes.QUERYOPTION_AWAITDATA);

and through a iterator to get all operation logs.

DBObject data = cursor.next();

Exception description

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2734)
        at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
        at java.util.ArrayList.add(ArrayList.java:351)
        at com.mongodb.DBApiLayer$Result.init(DBApiLayer.java:369)
        at com.mongodb.DBApiLayer$Result._advance(DBApiLayer.java:429)
        at com.mongodb.DBApiLayer$Result.hasNext(DBApiLayer.java:396)
        at com.mongodb.DBCursor._hasNext(DBCursor.java:464)
        at com.mongodb.DBCursor.hasNext(DBCursor.java:484)
        at com.patsnap.mongodb.dataload.MongoDbDataLoader.hasNext(MongoDbDataLoader.java:128)
        at com.patsnap.mongodb.dataload.MongoDbDataLoader.main(MongoDbDataLoader.java:86)

the reason is that the size type is List<Integer>, It will become big enough if you iterator more data and eventually out of memory

_sizes.add( res.size() );



 Comments   
Comment by Ross Lawley [ 21/May/14 ]

Closing on behalf of lufeng@zhihuiya.com as it wasn't an issue with the tailable cursor.

Comment by feng lu [ 21/May/14 ]

yes, you are right, this issue is not cause by tailable cursors.

but how can I close this issue?

Thanks.

Comment by feng lu [ 22/Apr/14 ]

Hi Jeff, it run more than 10 days and the size of dumped operation log larger than 300GB. My java heap size setting like his -Xmx1024m -Xms512m.

Thanks Jeff.

Comment by Jeffrey Yemin [ 22/Apr/14 ]

Do you have any sense of how long your cursor had been alive for, and how large that _sizes array had gotten? What did you have set as the maximum heap size?

Generated at Thu Feb 08 08:53:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.