[JAVA-3768] Memory Leak Created: 18/Jun/20  Updated: 27/Oct/23  Resolved: 18/Jun/20

Status: Closed
Project: Java Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Sharad Keer Assignee: Unassigned
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2020-06-16 at 2.50.47 PM.png     PNG File Screen Shot 2020-06-16 at 2.51.30 PM.png     PNG File Screenshot 2022-10-25 at 8.55.05 AM.png     PNG File Screenshot 2022-10-31 at 9.47.08 AM.png    

 Description   

We are experiencing trend in our production environment of high heap memory consumption. We did various performance correction via code optimizing, however pattern seems consistent and by inspecting heapdump we observed all primary suspect are due to tons of object being pilled up in the memory and not being GC correctly later. On looking more deeper using eclipse memory analyzer, we observed all the leak suspect has common thread pattern. For reference i have added the stack-trace below :-

http-nio-9021-exec-14
 at java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object; (HashMap.java:631)
 at java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (HashMap.java:612)
 at org.bson.BasicBSONObject.put(Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/Object; (BasicBSONObject.java:36)
 at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
 at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
 at com.mongodb.DBObjectCodec.readArray(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Ljava/util/List; (DBObjectCodec.java:335)
 at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:291)
 at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
 at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
 at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
 at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
 at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
 at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (DBObjectCodec.java:138)
 at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (DBObjectCodec.java:61)
 at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (CompoundDBObjectCodec.java:43)
 at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (CompoundDBObjectCodec.java:27)
 at com.mongodb.operation.CommandResultArrayCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonArray; (CommandResultArrayCodec.java:52)
 at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:53)
 at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
 at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
 at org.bson.codecs.configuration.LazyCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (LazyCodec.java:47)
 at org.bson.codecs.BsonDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (BsonDocumentCodec.java:101)
 at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:56)
 at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
 at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
 at com.mongodb.connection.ReplyMessage.<init>(Lcom/mongodb/connection/ResponseBuffers;Lorg/bson/codecs/Decoder;J)V (ReplyMessage.java:57)
 at com.mongodb.connection.CommandProtocol.getResponseDocument(Lcom/mongodb/connection/ResponseBuffers;Lcom/mongodb/connection/CommandMessage;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (CommandProtocol.java:139)
 at com.mongodb.connection.CommandProtocol.execute(Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (CommandProtocol.java:118)
 at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(Lcom/mongodb/connection/Protocol;Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (DefaultServer.java:168)
 at com.mongodb.connection.DefaultServerConnection.executeProtocol(Lcom/mongodb/connection/Protocol;)Ljava/lang/Object; (DefaultServerConnection.java:289)
 at com.mongodb.connection.DefaultServerConnection.command(Ljava/lang/String;Lorg/bson/BsonDocument;ZLorg/bson/FieldNameValidator;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (DefaultServerConnection.java:176)
 at com.mongodb.operation.QueryBatchCursor.getMore()V (QueryBatchCursor.java:207)
 at com.mongodb.operation.QueryBatchCursor.hasNext()Z (QueryBatchCursor.java:103)
 at com.mongodb.MongoBatchCursorAdapter.hasNext()Z (MongoBatchCursorAdapter.java:46)
 at com.mongodb.DBCursor.hasNext()Z (DBCursor.java:145)

 

I have attached the dominator tree for reference that shows retained object size in the heap. The used mongoDB driver version is 3.4.2 and currently we are operating in MSA architecture where hosting microservice has allocated heap size is 4GB.  Application built using JDK 8 and other supporting frameworks are Sping-OSS.

 



 Comments   
Comment by Jeffrey Yemin [ 02/Nov/22 ]

sanveer1995@gmail.com can you run the following command in your production environment and comment back with the output: java -XX:+PrintFlagsFinal -version | grep -i HeapSize. Or if you're setting the max heap size manually, what are you setting it to?

As an example, on my Mac Mini with 32 GB RAM, the output is:

$ java -XX:+PrintFlagsFinal -version | grep -i HeapSize
   size_t ErgoHeapSizeLimit                     = 0                                         {product} {default}
   size_t HeapSizePerGCThread              = 43620760                          {product} {default}
   size_t InitialHeapSize                            = 536870912                        {product} {ergonomic}
   size_t LargePageHeapSizeThreshold  = 134217728                         {product} {default}
   size_t MaxHeapSize                              = 8589934592                     {product} {ergonomic}
   uintx NonNMethodCodeHeapSize       = 5836300                            {pd product} {ergonomic}
   uintx NonProfiledCodeHeapSize          = 122910970                         {pd product} {ergonomic}
   uintx ProfiledCodeHeapSize                 = 122910970                         {pd product} {ergonomic}

So on this class of machine, 100 MB of retained memory is around 1% total available memory, given a max heap size of ~8GB.  So I wonder, is this really causing a problem in your production application? Are you encountering OutOfMemoryError exceptions?

 

Comment by Sanveer Singh [ 02/Nov/22 ]

Thanks for the response @Jeffrey Yemin.

Decreasing the batch size did work but querying large data is slower now. I guess due to increase in round trips to DB.

Not sure on the commas though but believe me that value is slightly more than 100 MB.

Comment by Jeffrey Yemin [ 31/Oct/22 ]

The 16MB refers to the size of the batch when encoded as BSON. When decoded into Java hash maps, the size is expected to increase considerably.

I suggest you either increase the size of your heap or decrease the batch size of the operations.

I'm also unclear what the number 10,56,01,944 even means. Why are there only two digits to the left of each comma?

Comment by Sanveer Singh [ 31/Oct/22 ]

@Jeffrey Yemin

According to the doc https://www.mongodb.com/docs/manual/tutorial/iterate-a-cursor/#cursor-batches, 
max size for data fetched in a batch should be 16MB

Operations of type find(), aggregate(), listIndexes, and listCollections return a maximum of 16 megabytes per batch. 
batchSize() can enforce a smaller limit, but not a larger one.   
 
find() and aggregate() operations have an initial batch size of 101 documents by default. 
Subsequent getMore operations issued against the resulting cursor have no default batch size, so they are limited only by the 16 megabyte message size. 

 

But in actual, the size of the data that is fetched in a batch is going way beyond 16MB 

In the below image of my heapdump, you can see that the sizes are going upto 100MB

We expect to run multiple jobs like this in parallel, so this is restricting the amount of load that we can hold in memory.

Comment by Jeffrey Yemin [ 18/Jun/20 ]

Hi sharad.keer@gmail.com

DBCursor does retain references to objects decoded from query results (in this case instances of BasicDBObject and the ArrayList instances contained within them), but they will eligible for garbage collection as soon as the DBCursor is.  So I do not think that the issue is with the driver. 

The evidence that you provided only shows that the driver created the objects that are being retained, but not that the driver is at fault for retaining them.  A few things to look at:

  1. Is the application releasing references to DBCursor instances as soon as possible?
  2. Is the application retaining references to objects returned from DBCursor.next(), perhaps in some cache?

I'm going to close this issue, but if you find more evidence that the driver is at fault, we can re-open it.

 

Generated at Thu Feb 08 09:00:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.