[SERVER-34861] Fatal Assertion 28832, "Failed to mlock: Cannot allocate memory", when using LDAP Created: 07/May/18  Updated: 27/Oct/23  Resolved: 27/Jun/18

Status: Closed
Project: Core Server
Component/s: Security, Stability
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Akira Kurogane Assignee: Spencer Jackson
Resolution: Gone away Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:
Case:

 Description   

A 3.6.3 enterprise version mongod instance using LDAP has crashed several times, at fairly regular intervals (4 and bit hours apart). The backtrace in each case is always like the below.

2018-05-06T01:05:02.282-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:22.289-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:32.289-0400 I ACCESS   [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:40.463-0400 I ACCESS   [conn368] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:46.918-0400 F -        [conn3] Failed to mlock: Cannot allocate memory
2018-05-06T01:05:46.918-0400 F -        [conn3] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2018-05-06T01:05:46.918-0400 F -        [conn3] 
 
***aborting after fassert() failure
 
 
2018-05-06T01:05:46.934-0400 F -        [conn3] Got signal: 6 (Aborted).
 
0x5fd75550 0x5fd74374 0x5fd749d4 0x3fffb1cf0478 0x3fffb0cf1f94 0x5deb322c 0x5fcd435c 0x5f7af07c 0x5f7afe34 0x5f7a65f4 0x5f799e14 0x5f79aba8 0x5e479cc4 0x5f7fd648 0x5f8005d0 0x5f813494 0x5e12157c 0x5e130184 0x5e12a954 0x5e12e4ac 0x5f527958 0x5e129134 0x5e12be24 0x5e12c8f4 0x5e12a9f8 0x5e12e4ac 0x5f528048 0x5fba7848 0x3fffb0ea8af4 0x3fffb0dd8814
mongod.log.2018-05-06T05-05-56:----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"371E0000","o":"2865550","s":"_ZN5mongo15printStackTraceERSo"},{"b":"371E0000","o":"2864374"},{"b":"371E0000","o":"28649D4"},{"b":"3FFF9E650000","o":"478","s":"__kernel_sigtramp_rt64"},{"b":"3FFF9D610000","o":"41F94","s":"abort"},{"b":"371E0000","o":"9A322C","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"371E0000","o":"27C435C","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"371E0000","o":"229F07C"},{"b":"371E0000","o":"229FE34","s":"_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE"},{"b":"371E0000","o":"22965F4","s":"_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE"},{"b":"371E0000","o":"2289E14","s":"_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE"},{"b":"371E0000","o":"228ABA8","s":"_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE"},{"b":"371E0000","o":"F69CC4","s":"_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE"},{"b":"371E0000","o":"22ED648","s":"_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE"},{"b":"371E0000","o":"22F05D0","s":"_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE"},{"b":"371E0000","o":"2303494","s":"_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE"},{"b":"371E0000","o":"C1157C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"371E0000","o":"C20184","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A954","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2017958","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"371E0000","o":"C19134","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"371E0000","o":"C1BE24","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"371E0000","o":"C1C8F4","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A9F8","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2018048"},{"b":"371E0000","o":"2697848"},{"b":"3FFF9D800000","o":"8AF4"},{"b":"3FFF9D610000","o":"128814","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.17.1.el7.ppc64le", "version" : "#1 SMP Sun Jan 14 10:42:02 EST 2018", "machine" : "ppc64le" }, "somap" : [ { "b" : "371E0000", "elfType" : 3, "buildId" : "CB33058B673BEF7BA160427807DC1C100B94D6BA" }, { "b" : "3FFF9E650000", "elfType" : 3, "buildId" : "2C75A99E1CFE5BC193FFFC0EBAD200C84A78B68D" }, { "b" : "3FFF9E5A0000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D527B09409D52E9BFAE267B39C0FDF3D38E759A3" }, { "b" : "3FFF9E280000", "path" : "/lib64/libnetsnmpmibs.so.31", "elfType" : 3, "buildId" : "077A8F38393AE25F667612B8B59E9E77E409B52E" }, { "b" : "3FFF9E240000", "path" : "/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "54CC83D389C6B65799697562CA1DD3B6B02EE4E0" }, { "b" : "3FFF9E210000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "969ECD5474260DBCB4B2ABFB31F51C064AC4C6E9" }, { "b" : "3FFF9E170000", "path" : "/lib64/librpm.so.3", "elfType" : 3, "buildId" : "CB8D493850D9B9E3BAFC7DF944F9C0945AD24B19" }, { "b" : "3FFF9E110000", "path" : "/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "02DC64D69171CFCD3BE7346187970A1FA8F0FAC6" }, { "b" : "3FFF9E060000", "path" : "/lib64/libnetsnmpagent.so.31", "elfType" : 3, "buildId" : "089A3522498B9D14B0C252F4F50BE5A074799CB0" }, { "b" : "3FFF9E030000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "91974904A33B9779D30DA471FFE9057E7628C174" }, { "b" : "3FFF9DEE0000", "path" : "/lib64/libnetsnmp.so.31", "elfType" : 3, "buildId" : "FDFA734AB43A5C39A89FFB8F6D0C72C53824A8FC" }, { "b" : "3FFF9DE40000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6E0617AAB5BA600C362C0CAE397427AE26C729EB" }, { "b" : "3FFF9DB80000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2A87A1B2F916FB15784402846384B46628C6AD79" }, { "b" : "3FFF9DB30000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "3F93014C00BC39CE2CD5C38BFA246ED8E69ECD37" }, { "b" : "3FFF9DAA0000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "0988B930AA8F850B430045189134AD745A266E0C" }, { "b" : "3FFF9DA60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E82B0444E3C3C2A4F61D890764E5BE0DC533122F" }, { "b" : "3FFF9D9E0000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2C96E0BE57939D6BAF13E0BFB1B5BF8EBA59EC0C" }, { "b" : "3FFF9D8F0000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "68BDE3292C76B915D2D754B412C1EBC04F577651" }, { "b" : "3FFF9D8B0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "7812741EA4E2159DC508E42898665472C6975EB0" }, { "b" : "3FFF9D880000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "03B6F7D0C119E4E73F4FECA9348935B6FDD0C81C" }, { "b" : "3FFF9D840000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "CDC8EA09CEE9BD7C6FFCA5B32B6751034349039E" }, { "b" : "3FFF9D800000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "CB2DE03746A9E9FF42316A9B913624EFB7CC206E" }, { "b" : "3FFF9D610000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "9DAEB87441B1A97240F1F46F69886764BCADC31D" }, { "b" : "3FFF9E670000", "path" : "/lib64/ld64.so.2", "elfType" : 3, "buildId" : "6339F0F333D1B8C587C27E4E18F4A13C68E3A75E" }, { "b" : "3FFF9D5B0000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "1FBC9A52A46AFC63447524B7FC20DD6B6317DB9A" }, { "b" : "3FFF9D550000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "68B0A2A707ECE31BA3A5A8681B62BE3C4F262580" }, { "b" : "3FFF9D4D0000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "02DC767EC098B356F39CD6B5040DA5D821187934" }, { "b" : "3FFF9D480000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "1CB51DF2FC8DDBA92BD1550B7EC377B38B52C013" }, { "b" : "3FFF9D300000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "89B4886B985A919B1772A06E146390D44D1149A0" }, { "b" : "3FFF9D2A0000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "736ECBC237DE290F1EA3F96814DE4B252DFDFD35" }, { "b" : "3FFF9D270000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "EF7B4CAD945B5ACC48EBA382DC5BDAF648CB260A" }, { "b" : "3FFF9D240000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "831682C943C06075EE5E785631A739D4811C207E" }, { "b" : "3FFF9D1D0000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "1EF3D9EAAC25BD1AE78BFAE40BE1BF2FC9EF1A86" }, { "b" : "3FFF9D0A0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B55BD49FCC7D957739A3236377538F706E757D4A" }, { "b" : "3FFF9D040000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C1BAEA99E6FD27FA77BCB77C26B45184DC86F2A6" }, { "b" : "3FFF9D010000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D4B216A44EB7603EFA730565106921DB8F92A124" }, { "b" : "3FFF9CFD0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "11FCD88E67465947D0E20EEC65C8A499087EB34E" }, { "b" : "3FFF9CDF0000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "70D11256C4702E4FBAE6B58E876818C6A76476D5" }, { "b" : "3FFF9CDB0000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "9CCD982ECDFB582090DA0FFB4C40443F2E331ECF" }, { "b" : "3FFF9CD60000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "3ED0390673BCCA3EA49854F13CBBFDDC2EC83409" }, { "b" : "3FFF9CD30000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "8CDCAC8C2941127351284ADB8CC0C9D2A005CF8D" }, { "b" : "3FFF9CCF0000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "ADF50408432617067AD8B3D062CD57608A3CB2A6" }, { "b" : "3FFF9CCB0000", "path" : "/lib64/libelf.so.1", "elfType" : 3, "buildId" : "DDB70F20DE3E4632C3343AF17394D4ED2339622E" }, { "b" : "3FFF9CC50000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "1DAA639AA47F757D3637C04E54DDCE183823BD2D" }, { "b" : "3FFF9CC20000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "AFD04B59E402943412B1027AE2219299FEA875C8" }, { "b" : "3FFF9CBD0000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "71FA3939566953509C7FE36A83C73B9FE22A85CC" }, { "b" : "3FFF9CBA0000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "70879B51ACAF692786A561A359D86D8D89EB861E" }, { "b" : "3FFF9CB60000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "A1C181B034E65CFE4B12095775CAD9274B9BDF1B" }, { "b" : "3FFF9CB00000", "path" : "/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "871E8C147E5F3FA59D568CE41778920E3C771FDE" }, { "b" : "3FFF9C8F0000", "path" : "/lib64/libdb-5.3.so", "elfType" : 3, "buildId" : "A2B813A140BBBD02EB8935651719A22B5D82E736" }, { "b" : "3FFF9C8C0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "CCDA94380A7D7CC3EE0D311E2EA34267E9EEE809" }, { "b" : "3FFF9C890000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E6FCF4AFACA8AE007CDB4419DF42040D1C0E8CA0" }, { "b" : "3FFF9C860000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "43D98F1422C06A50678FEE88DB20A1067D77A449" }, { "b" : "3FFF9C7F0000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "88B77E47D365CB47FE391B128FD52B84868AD027" }, { "b" : "3FFF9C7C0000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8510E155A4490CFAE6014E03D89C0231541CD3E3" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x80) [0x39a45550]
 mongod(+0x2864374) [0x39a44374]
 mongod(+0x28649D4) [0x39a449d4]
 (__kernel_sigtramp_rt64+0x0) [0x3fff9e650478]
 libc.so.6(abort+0x2B4) [0x3fff9d651f94]
 mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x1C0) [0x37b8322c]
 mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x77C) [0x399a435c]
 mongod(+0x229F07C) [0x3947f07c]
 mongod(_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE+0x64) [0x3947fe34]
 mongod(_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE+0x224) [0x394765f4]
 mongod(_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE+0x74) [0x39469e14]
 mongod(_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE+0x528) [0x3946aba8]
 mongod(_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE+0x134) [0x38149cc4]
 mongod(_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE+0x88) [0x394cd648]
 mongod(_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE+0x620) [0x394d05d0]
 mongod(_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE+0x204) [0x394e3494]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0xDC) [0x37df157c]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0x104) [0x37e00184]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xC4) [0x37dfa954]
 mongod(+0xC1E4AC) [0x37dfe4ac]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x208) [0x391f7958]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x1C4) [0x37df9134]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB04) [0x37dfbe24]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x354) [0x37dfc8f4]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x168) [0x37dfa9f8]
 mongod(+0xC1E4AC) [0x37dfe4ac]
 mongod(+0x2018048) [0x391f8048]
 mongod(+0x2697848) [0x39877848]
 libpthread.so.0(+0x8AF4) [0x3fff9d808af4]
 libc.so.6(clone+0xE4) [0x3fff9d738814]
-----  END BACKTRACE  -----

The 'max locked memory' limit in effect on this mongod process was 64kb.

The LDAP related options it was start with

{ ..., 
  security: { 
    authorization: "enabled", 
    keyFile: "...", 
    ldap: { authz: { queryTemplate: "{USER}?memberOf?base" }, bind: { method: "simple", queryPassword: "<password>", queryUser: "CN=...,OU=...,OU=...,DC=...,DC=..." }, servers: "....", transportSecurity: "none",   userToDNMapping: "[
               {
                  match : "(.+)",
                  ldapQuery: "DC=...,DC=...??sub?(saMAccountName={0})"
               }
          ]" } }, 
  setParameter: { authenticationMechanisms: "SCRAM-SHA-1,PLAIN" }, 
  ...
}

Can we understand if using LDAP means we need to raise the amount of available locked memory? Or can this be fixed to prevent >= 64kb of locked memory being used?

At first this error looked like SERVER-29086, SERVER-28997, but those cases didn't use LDAP and should be fixed in 3.6.0+.



 Comments   
Comment by Akira Kurogane [ 07/May/18 ]

An extra observation: the crashes in this environment are happening 4.2hrs after each restart during the last 48hrs. Maybe exactly to the second - the error factor seems to be within a few ten's of seconds at most.

In the 48hrs before that time it was variable, between 7.0 and 8.4 hrs of uptime before each crash.

Generated at Thu Feb 08 04:38:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.