XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Works as Designed
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None

    Description

      We are experiencing trend in our production environment of high heap memory consumption. We did various performance correction via code optimizing, however pattern seems consistent and by inspecting heapdump we observed all primary suspect are due to tons of object being pilled up in the memory and not being GC correctly later. On looking more deeper using eclipse memory analyzer, we observed all the leak suspect has common thread pattern. For reference i have added the stack-trace below :-

      http-nio-9021-exec-14
       at java.util.HashMap.putVal(ILjava/lang/Object;Ljava/lang/Object;ZZ)Ljava/lang/Object; (HashMap.java:631)
       at java.util.HashMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (HashMap.java:612)
       at org.bson.BasicBSONObject.put(Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/Object; (BasicBSONObject.java:36)
       at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
       at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
       at com.mongodb.DBObjectCodec.readArray(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Ljava/util/List; (DBObjectCodec.java:335)
       at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:291)
       at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
       at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
       at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
       at com.mongodb.DBObjectCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/lang/String;Ljava/util/List;)Ljava/lang/Object; (DBObjectCodec.java:288)
       at com.mongodb.DBObjectCodec.readDocument(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;Ljava/util/List;)Lcom/mongodb/DBObject; (DBObjectCodec.java:347)
       at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (DBObjectCodec.java:138)
       at com.mongodb.DBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (DBObjectCodec.java:61)
       at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lcom/mongodb/DBObject; (CompoundDBObjectCodec.java:43)
       at com.mongodb.CompoundDBObjectCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (CompoundDBObjectCodec.java:27)
       at com.mongodb.operation.CommandResultArrayCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonArray; (CommandResultArrayCodec.java:52)
       at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:53)
       at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
       at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
       at org.bson.codecs.configuration.LazyCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (LazyCodec.java:47)
       at org.bson.codecs.BsonDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (BsonDocumentCodec.java:101)
       at com.mongodb.operation.CommandResultDocumentCodec.readValue(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonValue; (CommandResultDocumentCodec.java:56)
       at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Lorg/bson/BsonDocument; (BsonDocumentCodec.java:84)
       at org.bson.codecs.BsonDocumentCodec.decode(Lorg/bson/BsonReader;Lorg/bson/codecs/DecoderContext;)Ljava/lang/Object; (BsonDocumentCodec.java:41)
       at com.mongodb.connection.ReplyMessage.<init>(Lcom/mongodb/connection/ResponseBuffers;Lorg/bson/codecs/Decoder;J)V (ReplyMessage.java:57)
       at com.mongodb.connection.CommandProtocol.getResponseDocument(Lcom/mongodb/connection/ResponseBuffers;Lcom/mongodb/connection/CommandMessage;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (CommandProtocol.java:139)
       at com.mongodb.connection.CommandProtocol.execute(Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (CommandProtocol.java:118)
       at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(Lcom/mongodb/connection/Protocol;Lcom/mongodb/connection/InternalConnection;)Ljava/lang/Object; (DefaultServer.java:168)
       at com.mongodb.connection.DefaultServerConnection.executeProtocol(Lcom/mongodb/connection/Protocol;)Ljava/lang/Object; (DefaultServerConnection.java:289)
       at com.mongodb.connection.DefaultServerConnection.command(Ljava/lang/String;Lorg/bson/BsonDocument;ZLorg/bson/FieldNameValidator;Lorg/bson/codecs/Decoder;)Ljava/lang/Object; (DefaultServerConnection.java:176)
       at com.mongodb.operation.QueryBatchCursor.getMore()V (QueryBatchCursor.java:207)
       at com.mongodb.operation.QueryBatchCursor.hasNext()Z (QueryBatchCursor.java:103)
       at com.mongodb.MongoBatchCursorAdapter.hasNext()Z (MongoBatchCursorAdapter.java:46)
       at com.mongodb.DBCursor.hasNext()Z (DBCursor.java:145)
      

       

      I have attached the dominator tree for reference that shows retained object size in the heap. The used mongoDB driver version is 3.4.2 and currently we are operating in MSA architecture where hosting microservice has allocated heap size is 4GB.  Application built using JDK 8 and other supporting frameworks are Sping-OSS.

       

      Attachments

        1. Screen Shot 2020-06-16 at 2.50.47 PM.png
          Screen Shot 2020-06-16 at 2.50.47 PM.png
          248 kB
        2. Screen Shot 2020-06-16 at 2.51.30 PM.png
          Screen Shot 2020-06-16 at 2.51.30 PM.png
          465 kB
        3. Screenshot 2022-10-25 at 8.55.05 AM.png
          Screenshot 2022-10-25 at 8.55.05 AM.png
          1.15 MB
        4. Screenshot 2022-10-31 at 9.47.08 AM.png
          Screenshot 2022-10-31 at 9.47.08 AM.png
          2.62 MB

        Activity

          People

            Unassigned Unassigned
            sharad.keer@gmail.com Sharad Keer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: