Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-35642

We're seeing increasing memory usage on secondary members.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: 3.6.4
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None
    • Operating System:
      ALL

      Description

      Basically, seeing the same symptoms as in https://jira.mongodb.org/browse/SERVER-26312
      after one of the apps using mongodb has been updated. Memory usage on primary is reasonable and stable. On secondary - too high and grows all the time. The only change in app I think could've caused it - one of the connections in updated app uses secondaryPreferred mode.

      On the attached CloudWatch chart blue line is master, orange is secondary. The third node isn't monitored atm, but it's res memory usage ( {{ db.serverStatus().mem.resident }} ) was around 2833Gb, which is very close to another secondary (orange).

      Memory allocated for each node - 4Gb (t2-medium). 
      This is server build information from master node, looks very similar to others.

      { 
          "version" : "3.6.4", 
          "gitVersion" : "d0181a711f7e7f39e60b5aeb1dc7097bf6ae5856", 
          "modules" : [    ], 
          "allocator" : "tcmalloc", 
          "javascriptEngine" : "mozjs", 
          "sysInfo" : "deprecated", 
          "versionArray" : [
              NumberInt(3), 
              NumberInt(6), 
              NumberInt(4), 
              NumberInt(0)
          ], 
          "openssl" : {
              "running" : "OpenSSL 1.0.0-fips 29 Mar 2010", 
              "compiled" : "OpenSSL 1.0.1e-fips 11 Feb 2013"
          }, 
          "buildEnvironment" : {
              "distmod" : "amazon", 
              "distarch" : "x86_64", 
              "cc" : "/opt/mongodbtoolchain/v2/bin/gcc: gcc (GCC) 5.4.0", 
              "ccflags" : "-fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -Werror -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-deprecated-declarations -Wno-unused-but-set-variable -Wno-missing-braces -fstack-protector-strong -fno-builtin-memcmp", 
              "cxx" : "/opt/mongodbtoolchain/v2/bin/g++: g++ (GCC) 5.4.0", 
              "cxxflags" : "-Woverloaded-virtual -Wno-maybe-uninitialized -std=c++14", 
              "linkflags" : "-pthread -Wl,-z,now -rdynamic -Wl,--fatal-warnings -fstack-protector-strong -fuse-ld=gold -Wl,--build-id -Wl,--hash-style=gnu -Wl,-z,noexecstack -Wl,--warn-execstack -Wl,-z,relro", 
              "target_arch" : "x86_64", 
              "target_os" : "linux"
          }, 
          "bits" : NumberInt(64), 
          "debug" : false, 
          "maxBsonObjectSize" : NumberInt(16777216), 
          "storageEngines" : [
              "devnull", 
              "ephemeralForTest", 
              "mmapv1", 
              "wiredTiger"
          ], 
          "ok" : 1.0, 
          "operationTime" : Timestamp(1529327365, 839), 
          "$clusterTime" : {
              "clusterTime" : Timestamp(1529327365, 839), 
              "signature" : {
                  "hash" : BinData(0, "AAAAAAAAAAAAAAAAAAAAAAAAAAA="), 
                  "keyId" : NumberLong(0)
              }
          }
      } 

      Hope it helps. Please, let me know if you need more information.

        Attachments

        1. memory.png
          memory.png
          119 kB
        2. mongodb-leak.png
          mongodb-leak.png
          83 kB
        3. screenshot-1.png
          screenshot-1.png
          49 kB
        4. screenshot-2.png
          screenshot-2.png
          60 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: