Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-30421

mongod "Cannot allocate memory" and crash

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Duplicate
    • Affects Version/s: 3.2.16
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Backwards Compatibility:
      Fully Compatible

      Description

      We just had a mongod crash after seeing this logged. We use sharding and replication.

      2017-07-31T04:29:57.670+0000 E STORAGE  [conn38291] WiredTiger (12) [1501475397:670620][1176:0x7fcb5f447700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.670+0000 E STORAGE  [conn31681] WiredTiger (12) [1501475397:670839][1176:0x7fccd90cc700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.670+0000 E STORAGE  [conn34966] WiredTiger (12) [1501475397:670984][1176:0x7fcc21259700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.682+0000 E STORAGE  [conn35450] WiredTiger (12) [1501475397:682359][1176:0x7fcef4660700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.695+0000 E STORAGE  [WTJournalFlusher] WiredTiger (12) [1501475397:695320][1176:0x7fd1676a8700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.715+0000 E STORAGE  [conn20441] WiredTiger (12) [1501475397:715160][1176:0x7fcf63c4f700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.737+0000 E STORAGE  [conn11932] WiredTiger (12) [1501475397:737176][1176:0x7fd12372e700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.787+0000 I -        [conn38291] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.787+0000 I -        [conn11932] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.787+0000 I -        [conn31681] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.788+0000 I -        [conn34966] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.787+0000 I -        [conn20441] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.788+0000 I -        [WTJournalFlusher] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.788+0000 I -        [conn35450] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.821+0000 E STORAGE  [conn31327] WiredTiger (12) [1501475397:821140][1176:0x7fcceda14700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.821+0000 I -        [conn31327] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.827+0000 E STORAGE  [conn33938] WiredTiger (12) [1501475397:827764][1176:0x7fcc58fd3700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.827+0000 I -        [conn33938] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.835+0000 E STORAGE  [conn37020] WiredTiger (12) [1501475397:835110][1176:0x7fcbaa7f6700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.835+0000 I -        [conn37020] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.864+0000 E STORAGE  [conn11921] WiredTiger (12) [1501475397:864452][1176:0x7fd124239700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.864+0000 I -        [conn11921] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.901+0000 E STORAGE  [conn36886] WiredTiger (12) [1501475397:901889][1176:0x7fcbb2b79700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.901+0000 I -        [conn36886] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.926+0000 E STORAGE  [conn37963] WiredTiger (12) [1501475397:926271][1176:0x7fcb72779700], WT_CONNECTION.open_session: only configured to support 20020 sessions (including 20 additional internal sessions): Cannot allocate memory
      2017-07-31T04:29:57.926+0000 I -        [conn37963] Invariant failure: conn->open_session(conn, NULL, "isolation=snapshot", &_session) resulted in status UnknownError: 12: Cannot allocate memory at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 54
      2017-07-31T04:29:57.971+0000 I CONTROL  [WTJournalFlusher] 
       0x1556b32 0x14ee2a4 0x14d4d53 0x126e8fb 0x126f665 0x126fd46 0x125aba3 0x14d7b77 0x1d00200 0x7fd16ca846ba 0x7fd16c7ba3dd
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"400000","o":"1156B32","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"10EE2A4","s":"_ZN5mongo10logContextEPKc"},{"b":"400000","o":"10D4D53","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j"},{"b":"400000","o":"E6E8FB","s":"_ZN5mongo17WiredTigerSessionC1EP15__wt_connectionmm"},{"b":"400000","o":"E6F665","s":"_ZN5mongo22WiredTigerSessionCache10getSessionEv"},{"b":"400000","o":"E6FD46","s":"_ZN5mongo22WiredTigerSessionCache16waitUntilDurableEb"},{"b":"400000","o":"E5ABA3","s":"_ZN5mongo18WiredTigerKVEngine24WiredTigerJournalFlusher3runEv"},{"b":"400000","o":"10D7B77","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"400000","o":"1900200"},{"b":"7FD16CA7D000","o":"76BA"},{"b":"7FD16C6B3000","o":"1073DD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.16", "gitVersion" : "056bf45128114e44c5358c7a8776fb582363e094", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.4.0-87-generic", "version" : "#110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B4C77D1B42936B23E28A2739927CB25274DB2D96" }, { "b" : "7FFC92910000", "elfType" : 3, "buildId" : "D4057BD712DABB1A8B5F4666C1CA1CD47E70FBA1" }, { "b" : "7FD16DA09000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "675F454AD6FD0B6CA2E41127C7B98079DA37F7B6" }, { "b" : "7FD16D5C5000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2DA08A7E5BF610030DD33B70DB951399626B7496" }, { "b" : "7FD16D3BD000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "F951C1E0765FCAE48F82CAFE35D1ADD36D6C9AF9" }, { "b" : "7FD16D1B9000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "0FC788F0861846257B5F1773FBD438E95DFC1032" }, { "b" : "7FD16CEB0000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "FF7A33D389E756CA381A8189291A968EA5E1F4F8" }, { "b" : "7FD16CC9A000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7FD16CA7D000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "27F189EF8DB8C3734C6A678E6EF3CB0B206D58B2" }, { "b" : "7FD16C6B3000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "088A6E00A1814622219F346B41E775B8DD46C518" }, { "b" : "7FD16DC72000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "9157F205547F0EB588E2AB1F2F120B74253A43EA" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1556b32]
       mongod(_ZN5mongo10logContextEPKc+0x174) [0x14ee2a4]
       mongod(_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j+0xF3) [0x14d4d53]
       mongod(_ZN5mongo17WiredTigerSessionC1EP15__wt_connectionmm+0x9B) [0x126e8fb]
       mongod(_ZN5mongo22WiredTigerSessionCache10getSessionEv+0xD5) [0x126f665]
       mongod(_ZN5mongo22WiredTigerSessionCache16waitUntilDurableEb+0x2F6) [0x126fd46]
       mongod(_ZN5mongo18WiredTigerKVEngine24WiredTigerJournalFlusher3runEv+0xB3) [0x125aba3]
       mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x1B7) [0x14d7b77]
       mongod(+0x1900200) [0x1d00200]
       libpthread.so.0(+0x76BA) [0x7fd16ca846ba]
       libc.so.6(clone+0x6D) [0x7fd16c7ba3dd]
      -----  END BACKTRACE  -----
      2017-07-31T04:29:57.975+0000 I -        [WTJournalFlusher] 
       
      ***aborting after invariant() failure
      

        Attachments

        1. overall_event.png
          overall_event.png
          449 kB
        2. workload_shift.png
          workload_shift.png
          169 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: