Details
-
Bug
-
Resolution: Works as Designed
-
Major - P3
-
None
-
None
-
None
-
None
Description
We are experiencing trend in our production environment of high heap memory consumption. We did various performance correction via code optimizing, however pattern seems consistent and by inspecting heapdump we observed all primary suspect are due to tons of object being pilled up in the memory and not being GC correctly later. On looking more deeper using eclipse memory analyzer, we observed all the leak suspect has common thread pattern. For reference i have added the stack-trace below :-
http-nio-9021-exec-14
|
at java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object; (HashMap.java:631)
|
at java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (HashMap.java:612)
|
at org.bson.BasicBSONObject.put(Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/Object; (BasicBSONObject.java:36)
|
at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
|
at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
|
at com.mongodb.DBObjectCodec.readArray(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Ljava/util/List; (DBObjectCodec.java:335)
|
at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:291)
|
at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
|
at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
|
at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
|
at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
|
at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
|
at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (DBObjectCodec.java:138)
|
at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (DBObjectCodec.java:61)
|
at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (CompoundDBObjectCodec.java:43)
|
at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (CompoundDBObjectCodec.java:27)
|
at com.mongodb.operation.CommandResultArrayCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonArray; (CommandResultArrayCodec.java:52)
|
at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:53)
|
at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
|
at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
|
at org.bson.codecs.configuration.LazyCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (LazyCodec.java:47)
|
at org.bson.codecs.BsonDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (BsonDocumentCodec.java:101)
|
at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:56)
|
at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
|
at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
|
at com.mongodb.connection.ReplyMessage.<init>(Lcom/mongodb/connection/ResponseBuffers;Lorg/bson/codecs/Decoder;J)V (ReplyMessage.java:57)
|
at com.mongodb.connection.CommandProtocol.getResponseDocument(Lcom/mongodb/connection/ResponseBuffers;Lcom/mongodb/connection/CommandMessage;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (CommandProtocol.java:139)
|
at com.mongodb.connection.CommandProtocol.execute(Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (CommandProtocol.java:118)
|
at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(Lcom/mongodb/connection/Protocol;Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (DefaultServer.java:168)
|
at com.mongodb.connection.DefaultServerConnection.executeProtocol(Lcom/mongodb/connection/Protocol;)Ljava/lang/Object; (DefaultServerConnection.java:289)
|
at com.mongodb.connection.DefaultServerConnection.command(Ljava/lang/String;Lorg/bson/BsonDocument;ZLorg/bson/FieldNameValidator;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (DefaultServerConnection.java:176)
|
at com.mongodb.operation.QueryBatchCursor.getMore()V (QueryBatchCursor.java:207)
|
at com.mongodb.operation.QueryBatchCursor.hasNext()Z (QueryBatchCursor.java:103)
|
at com.mongodb.MongoBatchCursorAdapter.hasNext()Z (MongoBatchCursorAdapter.java:46)
|
at com.mongodb.DBCursor.hasNext()Z (DBCursor.java:145)
|
I have attached the dominator tree for reference that shows retained object size in the heap. The used mongoDB driver version is 3.4.2 and currently we are operating in MSA architecture where hosting microservice has allocated heap size is 4GB. Application built using JDK 8 and other supporting frameworks are Sping-OSS.