Major - P3 A 3.6.3 enterprise version mongod instance using LDAP has crashed several times, at fairly regular intervals (4 and bit hours apart). The backtrace in each case is always like the below.
Unable to find source-code formatter for language: txt. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
2018-05-06T01:05:02.282-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:22.289-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:32.289-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:40.463-0400 I ACCESS [conn368] Successfully authenticated as principal mms-monitoring-agent on admin
2018-05-06T01:05:46.918-0400 F - [conn3] Failed to mlock: Cannot allocate memory
2018-05-06T01:05:46.918-0400 F - [conn3] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2018-05-06T01:05:46.918-0400 F - [conn3]
***aborting after fassert() failure
2018-05-06T01:05:46.934-0400 F - [conn3] Got signal: 6 (Aborted).
0x5fd75550 0x5fd74374 0x5fd749d4 0x3fffb1cf0478 0x3fffb0cf1f94 0x5deb322c 0x5fcd435c 0x5f7af07c 0x5f7afe34 0x5f7a65f4 0x5f799e14 0x5f79aba8 0x5e479cc4 0x5f7fd648 0x5f8005d0 0x5f813494 0x5e12157c 0x5e130184 0x5e12a954 0x5e12e4ac 0x5f527958 0x5e129134 0x5e12be24 0x5e12c8f4 0x5e12a9f8 0x5e12e4ac 0x5f528048 0x5fba7848 0x3fffb0ea8af4 0x3fffb0dd8814
mongod.log.2018-05-06T05-05-56:----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"371E0000","o":"2865550","s":"_ZN5mongo15printStackTraceERSo"},{"b":"371E0000","o":"2864374"},{"b":"371E0000","o":"28649D4"},{"b":"3FFF9E650000","o":"478","s":"__kernel_sigtramp_rt64"},{"b":"3FFF9D610000","o":"41F94","s":"abort"},{"b":"371E0000","o":"9A322C","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"371E0000","o":"27C435C","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"371E0000","o":"229F07C"},{"b":"371E0000","o":"229FE34","s":"_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE"},{"b":"371E0000","o":"22965F4","s":"_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE"},{"b":"371E0000","o":"2289E14","s":"_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE"},{"b":"371E0000","o":"228ABA8","s":"_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE"},{"b":"371E0000","o":"F69CC4","s":"_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE"},{"b":"371E0000","o":"22ED648","s":"_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE"},{"b":"371E0000","o":"22F05D0","s":"_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE"},{"b":"371E0000","o":"2303494","s":"_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE"},{"b":"371E0000","o":"C1157C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"371E0000","o":"C20184","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A954","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2017958","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"371E0000","o":"C19134","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"371E0000","o":"C1BE24","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"371E0000","o":"C1C8F4","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A9F8","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2018048"},{"b":"371E0000","o":"2697848"},{"b":"3FFF9D800000","o":"8AF4"},{"b":"3FFF9D610000","o":"128814","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.17.1.el7.ppc64le", "version" : "#1 SMP Sun Jan 14 10:42:02 EST 2018", "machine" : "ppc64le" }, "somap" : [ { "b" : "371E0000", "elfType" : 3, "buildId" : "CB33058B673BEF7BA160427807DC1C100B94D6BA" }, { "b" : "3FFF9E650000", "elfType" : 3, "buildId" : "2C75A99E1CFE5BC193FFFC0EBAD200C84A78B68D" }, { "b" : "3FFF9E5A0000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D527B09409D52E9BFAE267B39C0FDF3D38E759A3" }, { "b" : "3FFF9E280000", "path" : "/lib64/libnetsnmpmibs.so.31", "elfType" : 3, "buildId" : "077A8F38393AE25F667612B8B59E9E77E409B52E" }, { "b" : "3FFF9E240000", "path" : "/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "54CC83D389C6B65799697562CA1DD3B6B02EE4E0" }, { "b" : "3FFF9E210000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "969ECD5474260DBCB4B2ABFB31F51C064AC4C6E9" }, { "b" : "3FFF9E170000", "path" : "/lib64/librpm.so.3", "elfType" : 3, "buildId" : "CB8D493850D9B9E3BAFC7DF944F9C0945AD24B19" }, { "b" : "3FFF9E110000", "path" : "/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "02DC64D69171CFCD3BE7346187970A1FA8F0FAC6" }, { "b" : "3FFF9E060000", "path" : "/lib64/libnetsnmpagent.so.31", "elfType" : 3, "buildId" : "089A3522498B9D14B0C252F4F50BE5A074799CB0" }, { "b" : "3FFF9E030000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "91974904A33B9779D30DA471FFE9057E7628C174" }, { "b" : "3FFF9DEE0000", "path" : "/lib64/libnetsnmp.so.31", "elfType" : 3, "buildId" : "FDFA734AB43A5C39A89FFB8F6D0C72C53824A8FC" }, { "b" : "3FFF9DE40000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6E0617AAB5BA600C362C0CAE397427AE26C729EB" }, { "b" : "3FFF9DB80000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2A87A1B2F916FB15784402846384B46628C6AD79" }, { "b" : "3FFF9DB30000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "3F93014C00BC39CE2CD5C38BFA246ED8E69ECD37" }, { "b" : "3FFF9DAA0000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "0988B930AA8F850B430045189134AD745A266E0C" }, { "b" : "3FFF9DA60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E82B0444E3C3C2A4F61D890764E5BE0DC533122F" }, { "b" : "3FFF9D9E0000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2C96E0BE57939D6BAF13E0BFB1B5BF8EBA59EC0C" }, { "b" : "3FFF9D8F0000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "68BDE3292C76B915D2D754B412C1EBC04F577651" }, { "b" : "3FFF9D8B0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "7812741EA4E2159DC508E42898665472C6975EB0" }, { "b" : "3FFF9D880000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "03B6F7D0C119E4E73F4FECA9348935B6FDD0C81C" }, { "b" : "3FFF9D840000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "CDC8EA09CEE9BD7C6FFCA5B32B6751034349039E" }, { "b" : "3FFF9D800000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "CB2DE03746A9E9FF42316A9B913624EFB7CC206E" }, { "b" : "3FFF9D610000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "9DAEB87441B1A97240F1F46F69886764BCADC31D" }, { "b" : "3FFF9E670000", "path" : "/lib64/ld64.so.2", "elfType" : 3, "buildId" : "6339F0F333D1B8C587C27E4E18F4A13C68E3A75E" }, { "b" : "3FFF9D5B0000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "1FBC9A52A46AFC63447524B7FC20DD6B6317DB9A" }, { "b" : "3FFF9D550000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "68B0A2A707ECE31BA3A5A8681B62BE3C4F262580" }, { "b" : "3FFF9D4D0000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "02DC767EC098B356F39CD6B5040DA5D821187934" }, { "b" : "3FFF9D480000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "1CB51DF2FC8DDBA92BD1550B7EC377B38B52C013" }, { "b" : "3FFF9D300000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "89B4886B985A919B1772A06E146390D44D1149A0" }, { "b" : "3FFF9D2A0000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "736ECBC237DE290F1EA3F96814DE4B252DFDFD35" }, { "b" : "3FFF9D270000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "EF7B4CAD945B5ACC48EBA382DC5BDAF648CB260A" }, { "b" : "3FFF9D240000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "831682C943C06075EE5E785631A739D4811C207E" }, { "b" : "3FFF9D1D0000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "1EF3D9EAAC25BD1AE78BFAE40BE1BF2FC9EF1A86" }, { "b" : "3FFF9D0A0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B55BD49FCC7D957739A3236377538F706E757D4A" }, { "b" : "3FFF9D040000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C1BAEA99E6FD27FA77BCB77C26B45184DC86F2A6" }, { "b" : "3FFF9D010000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D4B216A44EB7603EFA730565106921DB8F92A124" }, { "b" : "3FFF9CFD0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "11FCD88E67465947D0E20EEC65C8A499087EB34E" }, { "b" : "3FFF9CDF0000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "70D11256C4702E4FBAE6B58E876818C6A76476D5" }, { "b" : "3FFF9CDB0000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "9CCD982ECDFB582090DA0FFB4C40443F2E331ECF" }, { "b" : "3FFF9CD60000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "3ED0390673BCCA3EA49854F13CBBFDDC2EC83409" }, { "b" : "3FFF9CD30000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "8CDCAC8C2941127351284ADB8CC0C9D2A005CF8D" }, { "b" : "3FFF9CCF0000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "ADF50408432617067AD8B3D062CD57608A3CB2A6" }, { "b" : "3FFF9CCB0000", "path" : "/lib64/libelf.so.1", "elfType" : 3, "buildId" : "DDB70F20DE3E4632C3343AF17394D4ED2339622E" }, { "b" : "3FFF9CC50000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "1DAA639AA47F757D3637C04E54DDCE183823BD2D" }, { "b" : "3FFF9CC20000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "AFD04B59E402943412B1027AE2219299FEA875C8" }, { "b" : "3FFF9CBD0000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "71FA3939566953509C7FE36A83C73B9FE22A85CC" }, { "b" : "3FFF9CBA0000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "70879B51ACAF692786A561A359D86D8D89EB861E" }, { "b" : "3FFF9CB60000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "A1C181B034E65CFE4B12095775CAD9274B9BDF1B" }, { "b" : "3FFF9CB00000", "path" : "/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "871E8C147E5F3FA59D568CE41778920E3C771FDE" }, { "b" : "3FFF9C8F0000", "path" : "/lib64/libdb-5.3.so", "elfType" : 3, "buildId" : "A2B813A140BBBD02EB8935651719A22B5D82E736" }, { "b" : "3FFF9C8C0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "CCDA94380A7D7CC3EE0D311E2EA34267E9EEE809" }, { "b" : "3FFF9C890000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E6FCF4AFACA8AE007CDB4419DF42040D1C0E8CA0" }, { "b" : "3FFF9C860000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "43D98F1422C06A50678FEE88DB20A1067D77A449" }, { "b" : "3FFF9C7F0000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "88B77E47D365CB47FE391B128FD52B84868AD027" }, { "b" : "3FFF9C7C0000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8510E155A4490CFAE6014E03D89C0231541CD3E3" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x80) [0x39a45550]
mongod(+0x2864374) [0x39a44374]
mongod(+0x28649D4) [0x39a449d4]
(__kernel_sigtramp_rt64+0x0) [0x3fff9e650478]
libc.so.6(abort+0x2B4) [0x3fff9d651f94]
mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x1C0) [0x37b8322c]
mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x77C) [0x399a435c]
mongod(+0x229F07C) [0x3947f07c]
mongod(_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE+0x64) [0x3947fe34]
mongod(_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE+0x224) [0x394765f4]
mongod(_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE+0x74) [0x39469e14]
mongod(_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE+0x528) [0x3946aba8]
mongod(_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE+0x134) [0x38149cc4]
mongod(_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE+0x88) [0x394cd648]
mongod(_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE+0x620) [0x394d05d0]
mongod(_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE+0x204) [0x394e3494]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0xDC) [0x37df157c]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0x104) [0x37e00184]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xC4) [0x37dfa954]
mongod(+0xC1E4AC) [0x37dfe4ac]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x208) [0x391f7958]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x1C4) [0x37df9134]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB04) [0x37dfbe24]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x354) [0x37dfc8f4]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x168) [0x37dfa9f8]
mongod(+0xC1E4AC) [0x37dfe4ac]
mongod(+0x2018048) [0x391f8048]
mongod(+0x2697848) [0x39877848]
libpthread.so.0(+0x8AF4) [0x3fff9d808af4]
libc.so.6(clone+0xE4) [0x3fff9d738814]
----- END BACKTRACE -----
The 'max locked memory' limit in effect on this mongod process was 64kb.
The LDAP related options it was start with
{ ...,
security: {
authorization: "enabled",
keyFile: "...",
ldap: { authz: { queryTemplate: "{USER}?memberOf?base" }, bind: { method: "simple", queryPassword: "<password>", queryUser: "CN=...,OU=...,OU=...,DC=...,DC=..." }, servers: "....", transportSecurity: "none", userToDNMapping: "[
{
match : "(.+)",
ldapQuery: "DC=...,DC=...??sub?(saMAccountName={0})"
}
]" } },
setParameter: { authenticationMechanisms: "SCRAM-SHA-1,PLAIN" },
...
}
Can we understand if using LDAP means we need to raise the amount of available locked memory? Or can this be fixed to prevent >= 64kb of locked memory being used?
At first this error looked like SERVER-29086, SERVER-28997, but those cases didn't use LDAP and should be fixed in 3.6.0+.