Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35642

We're seeing increasing memory usage on secondary members.

    • Type: Icon: Bug Bug
    • Resolution: Gone away
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.4
    • Component/s: Replication
    • Labels:
      None
    • ALL

      Basically, seeing the same symptoms as in https://jira.mongodb.org/browse/SERVER-26312
      after one of the apps using mongodb has been updated. Memory usage on primary is reasonable and stable. On secondary - too high and grows all the time. The only change in app I think could've caused it - one of the connections in updated app uses secondaryPreferred mode.

      On the attached CloudWatch chart blue line is master, orange is secondary. The third node isn't monitored atm, but it's res memory usage ( {{ db.serverStatus().mem.resident }} ) was around 2833Gb, which is very close to another secondary (orange).

      Memory allocated for each node - 4Gb (t2-medium). 
      This is server build information from master node, looks very similar to others.

      { 
          "version" : "3.6.4", 
          "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", 
          "modules" : [    ], 
          "allocator" : "tcmalloc", 
          "javascriptEngine" : "mozjs", 
          "sysInfo" : "deprecated", 
          "versionArray" : [
              NumberInt(3), 
              NumberInt(6), 
              NumberInt(4), 
              NumberInt(0)
          ], 
          "openssl" : {
              "running" : "OpenSSL 1.0.0-fips 29 Mar 2010", 
              "compiled" : "OpenSSL 1.0.1e-fips 11 Feb 2013"
          }, 
          "buildEnvironment" : {
              "distmod" : "amazon", 
              "distarch" : "x86_64", 
              "cc" : "/opt/mongodbtoolchain/v2/bin/gcc: gcc (GCC) 5.4.0", 
              "ccflags" : "-fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp", 
              "cxx" : "/opt/mongodbtoolchain/v2/bin/g++: g++ (GCC) 5.4.0", 
              "cxxflags" : "-Woverloaded-virtual -Wno-maybe-uninitialized -std=c++14", 
              "linkflags" : "-pthread -Wl,-z,now -rdynamic -Wl,--fatal-warnings -fstack-protector-strong -fuse-ld=gold -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro", 
              "target_arch" : "x86_64", 
              "target_os" : "linux"
          }, 
          "bits" : NumberInt(64), 
          "debug" : false, 
          "maxBsonObjectSize" : NumberInt(16777216), 
          "storageEngines" : [
              "devnull", 
              "ephemeralForTest", 
              "mmapv1", 
              "wiredTiger"
          ], 
          "ok" : 1.0, 
          "operationTime" : Timestamp(1529327365, 839), 
          "$clusterTime" : {
              "clusterTime" : Timestamp(1529327365, 839), 
              "signature" : {
                  "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA="), 
                  "keyId" : NumberLong(0)
              }
          }
      } 

      Hope it helps. Please, let me know if you need more information.

        1. memory.png
          119 kB
          Bruce Lucas
        2. mongodb-leak.png
          83 kB
          Ivan Filimonov
        3. screenshot-1.png
          49 kB
          Ivan Filimonov
        4. screenshot-2.png
          60 kB
          Ivan Filimonov

            Assignee:
            william.schultz@mongodb.com William Schultz (Inactive)
            Reporter:
            sunrize531@gmail.com Ivan Filimonov
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: