Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34861

Fatal Assertion 28832, "Failed to mlock: Cannot allocate memory", when using LDAP

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Gone away
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Security, Stability
    • Labels:
      None
    • Operating System:
      ALL
    • Case:

      Description

      A 3.6.3 enterprise version mongod instance using LDAP has crashed several times, at fairly regular intervals (4 and bit hours apart). The backtrace in each case is always like the below.

      2018-05-06T01:05:02.282-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
      2018-05-06T01:05:22.289-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
      2018-05-06T01:05:32.289-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
      2018-05-06T01:05:40.463-0400 I ACCESS   [conn368] Successfully authenticated as principal mms-monitoring-agent on admin
      2018-05-06T01:05:46.918-0400 F -        [conn3] Failed to mlock: Cannot allocate memory
      2018-05-06T01:05:46.918-0400 F -        [conn3] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
      2018-05-06T01:05:46.918-0400 F -        [conn3] 
       
      ***aborting after fassert() failure
       
       
      2018-05-06T01:05:46.934-0400 F -        [conn3] Got signal: 6 (Aborted).
       
      0x5fd75550 0x5fd74374 0x5fd749d4 0x3fffb1cf0478 0x3fffb0cf1f94 0x5deb322c 0x5fcd435c 0x5f7af07c 0x5f7afe34 0x5f7a65f4 0x5f799e14 0x5f79aba8 0x5e479cc4 0x5f7fd648 0x5f8005d0 0x5f813494 0x5e12157c 0x5e130184 0x5e12a954 0x5e12e4ac 0x5f527958 0x5e129134 0x5e12be24 0x5e12c8f4 0x5e12a9f8 0x5e12e4ac 0x5f528048 0x5fba7848 0x3fffb0ea8af4 0x3fffb0dd8814
      mongod.log.2018-05-06T05-05-56:----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"371E0000","o":"2865550","s":"_ZN5mongo15printStackTraceERSo"},{"b":"371E0000","o":"2864374"},{"b":"371E0000","o":"28649D4"},{"b":"3FFF9E650000","o":"478","s":"__kernel_sigtramp_rt64"},{"b":"3FFF9D610000","o":"41F94","s":"abort"},{"b":"371E0000","o":"9A322C","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"371E0000","o":"27C435C","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"371E0000","o":"229F07C"},{"b":"371E0000","o":"229FE34","s":"_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE"},{"b":"371E0000","o":"22965F4","s":"_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE"},{"b":"371E0000","o":"2289E14","s":"_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE"},{"b":"371E0000","o":"228ABA8","s":"_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE"},{"b":"371E0000","o":"F69CC4","s":"_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE"},{"b":"371E0000","o":"22ED648","s":"_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE"},{"b":"371E0000","o":"22F05D0","s":"_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE"},{"b":"371E0000","o":"2303494","s":"_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE"},{"b":"371E0000","o":"C1157C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"371E0000","o":"C20184","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A954","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2017958","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"371E0000","o":"C19134","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"371E0000","o":"C1BE24","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"371E0000","o":"C1C8F4","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A9F8","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2018048"},{"b":"371E0000","o":"2697848"},{"b":"3FFF9D800000","o":"8AF4"},{"b":"3FFF9D610000","o":"128814","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.17.1.el7.ppc64le", "version" : "#1 SMP Sun Jan 14 10:42:02 EST 2018", "machine" : "ppc64le" }, "somap" : [ { "b" : "371E0000", "elfType" : 3, "buildId" : "CB33058B673BEF7BA160427807DC1C100B94D6BA" }, { "b" : "3FFF9E650000", "elfType" : 3, "buildId" : "2C75A99E1CFE5BC193FFFC0EBAD200C84A78B68D" }, { "b" : "3FFF9E5A0000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D527B09409D52E9BFAE267B39C0FDF3D38E759A3" }, { "b" : "3FFF9E280000", "path" : "/lib64/libnetsnmpmibs.so.31", "elfType" : 3, "buildId" : "077A8F38393AE25F667612B8B59E9E77E409B52E" }, { "b" : "3FFF9E240000", "path" : "/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "54CC83D389C6B65799697562CA1DD3B6B02EE4E0" }, { "b" : "3FFF9E210000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "969ECD5474260DBCB4B2ABFB31F51C064AC4C6E9" }, { "b" : "3FFF9E170000", "path" : "/lib64/librpm.so.3", "elfType" : 3, "buildId" : "CB8D493850D9B9E3BAFC7DF944F9C0945AD24B19" }, { "b" : "3FFF9E110000", "path" : "/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "02DC64D69171CFCD3BE7346187970A1FA8F0FAC6" }, { "b" : "3FFF9E060000", "path" : "/lib64/libnetsnmpagent.so.31", "elfType" : 3, "buildId" : "089A3522498B9D14B0C252F4F50BE5A074799CB0" }, { "b" : "3FFF9E030000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "91974904A33B9779D30DA471FFE9057E7628C174" }, { "b" : "3FFF9DEE0000", "path" : "/lib64/libnetsnmp.so.31", "elfType" : 3, "buildId" : "FDFA734AB43A5C39A89FFB8F6D0C72C53824A8FC" }, { "b" : "3FFF9DE40000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6E0617AAB5BA600C362C0CAE397427AE26C729EB" }, { "b" : "3FFF9DB80000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2A87A1B2F916FB15784402846384B46628C6AD79" }, { "b" : "3FFF9DB30000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "3F93014C00BC39CE2CD5C38BFA246ED8E69ECD37" }, { "b" : "3FFF9DAA0000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "0988B930AA8F850B430045189134AD745A266E0C" }, { "b" : "3FFF9DA60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E82B0444E3C3C2A4F61D890764E5BE0DC533122F" }, { "b" : "3FFF9D9E0000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2C96E0BE57939D6BAF13E0BFB1B5BF8EBA59EC0C" }, { "b" : "3FFF9D8F0000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "68BDE3292C76B915D2D754B412C1EBC04F577651" }, { "b" : "3FFF9D8B0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "7812741EA4E2159DC508E42898665472C6975EB0" }, { "b" : "3FFF9D880000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "03B6F7D0C119E4E73F4FECA9348935B6FDD0C81C" }, { "b" : "3FFF9D840000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "CDC8EA09CEE9BD7C6FFCA5B32B6751034349039E" }, { "b" : "3FFF9D800000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "CB2DE03746A9E9FF42316A9B913624EFB7CC206E" }, { "b" : "3FFF9D610000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "9DAEB87441B1A97240F1F46F69886764BCADC31D" }, { "b" : "3FFF9E670000", "path" : "/lib64/ld64.so.2", "elfType" : 3, "buildId" : "6339F0F333D1B8C587C27E4E18F4A13C68E3A75E" }, { "b" : "3FFF9D5B0000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "1FBC9A52A46AFC63447524B7FC20DD6B6317DB9A" }, { "b" : "3FFF9D550000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "68B0A2A707ECE31BA3A5A8681B62BE3C4F262580" }, { "b" : "3FFF9D4D0000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "02DC767EC098B356F39CD6B5040DA5D821187934" }, { "b" : "3FFF9D480000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "1CB51DF2FC8DDBA92BD1550B7EC377B38B52C013" }, { "b" : "3FFF9D300000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "89B4886B985A919B1772A06E146390D44D1149A0" }, { "b" : "3FFF9D2A0000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "736ECBC237DE290F1EA3F96814DE4B252DFDFD35" }, { "b" : "3FFF9D270000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "EF7B4CAD945B5ACC48EBA382DC5BDAF648CB260A" }, { "b" : "3FFF9D240000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "831682C943C06075EE5E785631A739D4811C207E" }, { "b" : "3FFF9D1D0000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "1EF3D9EAAC25BD1AE78BFAE40BE1BF2FC9EF1A86" }, { "b" : "3FFF9D0A0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B55BD49FCC7D957739A3236377538F706E757D4A" }, { "b" : "3FFF9D040000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C1BAEA99E6FD27FA77BCB77C26B45184DC86F2A6" }, { "b" : "3FFF9D010000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D4B216A44EB7603EFA730565106921DB8F92A124" }, { "b" : "3FFF9CFD0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "11FCD88E67465947D0E20EEC65C8A499087EB34E" }, { "b" : "3FFF9CDF0000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "70D11256C4702E4FBAE6B58E876818C6A76476D5" }, { "b" : "3FFF9CDB0000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "9CCD982ECDFB582090DA0FFB4C40443F2E331ECF" }, { "b" : "3FFF9CD60000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "3ED0390673BCCA3EA49854F13CBBFDDC2EC83409" }, { "b" : "3FFF9CD30000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "8CDCAC8C2941127351284ADB8CC0C9D2A005CF8D" }, { "b" : "3FFF9CCF0000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "ADF50408432617067AD8B3D062CD57608A3CB2A6" }, { "b" : "3FFF9CCB0000", "path" : "/lib64/libelf.so.1", "elfType" : 3, "buildId" : "DDB70F20DE3E4632C3343AF17394D4ED2339622E" }, { "b" : "3FFF9CC50000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "1DAA639AA47F757D3637C04E54DDCE183823BD2D" }, { "b" : "3FFF9CC20000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "AFD04B59E402943412B1027AE2219299FEA875C8" }, { "b" : "3FFF9CBD0000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "71FA3939566953509C7FE36A83C73B9FE22A85CC" }, { "b" : "3FFF9CBA0000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "70879B51ACAF692786A561A359D86D8D89EB861E" }, { "b" : "3FFF9CB60000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "A1C181B034E65CFE4B12095775CAD9274B9BDF1B" }, { "b" : "3FFF9CB00000", "path" : "/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "871E8C147E5F3FA59D568CE41778920E3C771FDE" }, { "b" : "3FFF9C8F0000", "path" : "/lib64/libdb-5.3.so", "elfType" : 3, "buildId" : "A2B813A140BBBD02EB8935651719A22B5D82E736" }, { "b" : "3FFF9C8C0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "CCDA94380A7D7CC3EE0D311E2EA34267E9EEE809" }, { "b" : "3FFF9C890000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E6FCF4AFACA8AE007CDB4419DF42040D1C0E8CA0" }, { "b" : "3FFF9C860000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "43D98F1422C06A50678FEE88DB20A1067D77A449" }, { "b" : "3FFF9C7F0000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "88B77E47D365CB47FE391B128FD52B84868AD027" }, { "b" : "3FFF9C7C0000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8510E155A4490CFAE6014E03D89C0231541CD3E3" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x80) [0x39a45550]
       mongod(+0x2864374) [0x39a44374]
       mongod(+0x28649D4) [0x39a449d4]
       (__kernel_sigtramp_rt64+0x0) [0x3fff9e650478]
       libc.so.6(abort+0x2B4) [0x3fff9d651f94]
       mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x1C0) [0x37b8322c]
       mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x77C) [0x399a435c]
       mongod(+0x229F07C) [0x3947f07c]
       mongod(_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE+0x64) [0x3947fe34]
       mongod(_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE+0x224) [0x394765f4]
       mongod(_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE+0x74) [0x39469e14]
       mongod(_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE+0x528) [0x3946aba8]
       mongod(_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE+0x134) [0x38149cc4]
       mongod(_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE+0x88) [0x394cd648]
       mongod(_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE+0x620) [0x394d05d0]
       mongod(_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE+0x204) [0x394e3494]
       mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0xDC) [0x37df157c]
       mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0x104) [0x37e00184]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xC4) [0x37dfa954]
       mongod(+0xC1E4AC) [0x37dfe4ac]
       mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x208) [0x391f7958]
       mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x1C4) [0x37df9134]
       mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB04) [0x37dfbe24]
       mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x354) [0x37dfc8f4]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x168) [0x37dfa9f8]
       mongod(+0xC1E4AC) [0x37dfe4ac]
       mongod(+0x2018048) [0x391f8048]
       mongod(+0x2697848) [0x39877848]
       libpthread.so.0(+0x8AF4) [0x3fff9d808af4]
       libc.so.6(clone+0xE4) [0x3fff9d738814]
      -----  END BACKTRACE  -----
      

      The 'max locked memory' limit in effect on this mongod process was 64kb.

      The LDAP related options it was start with

      { ..., 
        security: { 
          authorization: "enabled", 
          keyFile: "...", 
          ldap: { authz: { queryTemplate: "{USER}?memberOf?base" }, bind: { method: "simple", queryPassword: "<password>", queryUser: "CN=...,OU=...,OU=...,DC=...,DC=..." }, servers: "....", transportSecurity: "none",   userToDNMapping: "[
                     {
                        match : "(.+)",
                        ldapQuery: "DC=...,DC=...??sub?(saMAccountName={0})"
                     }
                ]" } }, 
        setParameter: { authenticationMechanisms: "SCRAM-SHA-1,PLAIN" }, 
        ...
      }
      

      Can we understand if using LDAP means we need to raise the amount of available locked memory? Or can this be fixed to prevent >= 64kb of locked memory being used?

      At first this error looked like SERVER-29086, SERVER-28997, but those cases didn't use LDAP and should be fixed in 3.6.0+.

        Attachments

          Activity

            People

            Assignee:
            spencer.jackson Spencer Jackson
            Reporter:
            akira.kurogane Akira Kurogane
            Participants:
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: