[SERVER-39059] Mongos Segfault Created: 17/Jan/19  Updated: 01/Feb/19  Resolved: 01/Feb/19

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.0.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Ji Eun Lee Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Mongos segfault with this backtrace.

Got signal: 6 (Aborted).
 0x556d6cf68cf1 0x556d6cf67f09 0x556d6cf683ed 0x7fee63f6d6d0 0x7fee63bc7277 0x7fee63bc8968 0x556d6c3473af 0x556d6cefd2f7 0x556d6ca0bbcd 0x556d6ca0ca28 0x556d6c6fea70 0x556d6c7006f0 0x556d6c9aeda9 0x556d6c494bde 0x556d6c496037 0x556d6c399b27 0x556d6c3b720a 0x556d6c3b1ec7 0x556d6c3b56d1 0x556d6c811802 0x556d6c3b00df 0x556d6c3b3275 0x556d6c3b1607 0x556d6c3b1f4d 0x556d6c3b56d1 0x556d6c811d65 0x556d6cee4294 0x7fee63f65e25 0x7fee63c8fbad
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"556D6BE08000","o":"1160CF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"556D6BE08000","o":"115FF09"},{"b":"556D6BE08000","o":"11603ED"},{"b":"7FEE63F5E000","o":"F6D0"},{"b":"7FEE63B91000","o":"36277","s":"gsignal"},{"b":"7FEE63B91000","o":"37968","s":"abort"},{"b":"556D6BE08000","o":"53F3AF","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"556D6BE08000","o":"10F52F7","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"556D6BE08
000","o":"C03BCD","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE"},{"b":"556D6BE08000","o":"C04A28","s":"_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE"},{"b":"556D6BE08000","o":"8F6A70"},{"b":"556D6BE08000","o":"8F86F0"},{"b":"556D6BE08000","o":"BA6DA9","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBui
lderE"},{"b":"556D6BE08000","o":"68CBDE"},{"b":"556D6BE08000","o":"68E037","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"556D6BE08000","o":"591B27","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"556D6BE08000","o":"5AF20A","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"556D6BE08000","o":"5A9EC7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuard
ENS0_11ThreadGuardE"},{"b":"556D6BE08000","o":"5AD6D1"},{"b":"556D6BE08000","o":"A09802","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"556D6BE08000","o":"5A80DF","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"556D6BE08000","o":"5AB275
","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"556D6BE08000","o":"5A9607","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"556D6BE08000","o":"5A9F4D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"556D6BE08000","o":"5AD6D1"},{"b":"556D6BE08000","o":"A09D65"},{"b":"556D6BE08000","o":"10DC294"},{"b":"7FEE63F5E000","o":"7E25"},{"b":"7FEE63B91000","o":"FEBAD","s":"clone"}],"processInf
o":{ "mongodbVersion" : "4.0.0", "gitVersion" : "3b07af3d4f471ae89e8186d33bbb1d5259597d51", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.21.1.el7.x86_64", "version" : "#1 SMP Wed Mar 7 19:03:37 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "556D6BE08000", "elfType" : 3, "buildId" : "3DE9B27F71F9EF205D5F39F5DD71AA46C352D375" }, { "b" : "7FFFDAAAC000", "elfType" : 3, "buildId" : "228ADFE0D8C0852BF24F80B24803DA9E25F5B21E" }, { "
b" : "7FEE65171000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "2BDC2B6FF0B2C204CCE34D139A9EADA0272EB070" }, { "b" : "7FEE64D10000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "67CEB4EDD36BFE0EB31CD92DA2694ACA5377A599" }, { "b" : "7FEE64A9E000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6468A4E28A19CDD885A3CBC30E009589CA4C2E92" }, { "b" : "7FEE6489A000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5CD
B5A56336E7E2BD14FFA189411E44A834AFCD8" }, { "b" : "7FEE64692000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "D33989EC31EFE745EB0D3B68A92D19E77D7DDFDA" }, { "b" : "7FEE64390000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "F4CAE74047F9AA2D5A71FDEC67C4285D75753EBA" }, { "b" : "7FEE6417A000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C344A7E6783B19A5C763AC24746EC6BAD2607F28" }, { "b" : "7FEE63F5E000", "path" : "/lib64/libpthre
ad.so.0", "elfType" : 3, "buildId" : "F4C04BCE85D2D269D0A2AF4972FC69805B50345B" }, { "b" : "7FEE63B91000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "CB4B7554D1ADBEF2F001142DD6F0A5139FC9AA69" }, { "b" : "7FEE6538A000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D266B1F6650927E18108323BCCA8F7B68E68EB92" }, { "b" : "7FEE6397B000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "EA8E45DC8E395CC5E26890470112D97A1F1E0B65" }, { "b" : "7FEE65457000", "path" : "/usr/nhnkrb5/lib/libgssapi_krb5.so.2", "elfType" : 3 }, { "b" : "7FEE637E8000", "path" : "/usr/nhnkrb5/lib/libkrb5.so.3", "elfType" : 3 }, { "b" : "7FEE635E4000", "path" : "/lib64/libcom_err.so.2", "elfTyp
e" : 3, "buildId" : "A3832734347DCA522438308C9F08F45524C65C9B" }, { "b" : "7FEE634BE000", "path" : "/usr/nhnkrb5/lib/libk5crypto.so.3", "elfType" : 3 }, { "b" : "7FEE633B5000", "path" : "/usr/nhnkrb5/lib/libkrb5support.so.0", "elfType" : 3 }, { "b" : "7FEE6318E000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "A88379F56A51950A33198890D37F5F8AEE71F8B4" }, { "b" : "7FEE62F2C000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB
719CDB34BE800BF1641350D0A" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x556d6cf68cf1]
 mongos(+0x115FF09) [0x556d6cf67f09]
 mongos(+0x11603ED) [0x556d6cf683ed]
 libpthread.so.0(+0xF6D0) [0x7fee63f6d6d0]
 libc.so.6(gsignal+0x37) [0x7fee63bc7277]
 libc.so.6(abort+0x148) [0x7fee63bc8968]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x556d6c3473af]
 mongos(_ZN5mongo24secure_allocator_details8allocateEmm+0x737) [0x556d6cefd2f7]
 mongos(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE10_firstStepEPNS_16OperationContextENS_10StringDataE+0xBDD) [0x556d6ca0bbcd]
 mongos(_ZN5mongo24SaslSCRAMServerMechanismINS_15SCRAMSHA1PolicyEE8stepImplEPNS_16OperationContextENS_10StringDataE+0x98) [0x556d6ca0ca28]
 mongos(+0x8F6A70) [0x556d6c6fea70]
 mongos(+0x8F86F0) [0x556d6c7006f0]
 mongos(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_19CommandReplyBuilderE+0xD9) [0x556d6c9aeda9]
 mongos(+0x68CBDE) [0x556d6c494bde]
 mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x1B7) [0x556d6c496037]
 mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x637) [0x556d6c399b27]
 mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x556d6c3b720a]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x556d6c3b1ec7]
 mongos(+0x5AD6D1) [0x556d6c3b56d1]
 mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x556d6c811802]
 mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x556d6c3b00df]
 mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x556d6c3b3275]
 mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x357) [0x556d6c3b1607]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x556d6c3b1f4d]
 mongos(+0x5AD6D1) [0x556d6c3b56d1]
 mongos(+0xA09D65) [0x556d6c811d65]
 mongos(+0x10DC294) [0x556d6cee4294]
 libpthread.so.0(+0x7E25) [0x7fee63f65e25]
 libc.so.6(clone+0x6D) [0x7fee63c8fbad]
-----  END BACKTRACE  -----



 Comments   
Comment by Danny Hatcher (Inactive) [ 01/Feb/19 ]

Hi,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Thank you,

Danny

Comment by Danny Hatcher (Inactive) [ 17/Jan/19 ]

Hello,

Yes, please set your max locked memory to "unlimited" and see if the issue still occurs. In general, please try to follow our UNIX ulimit recommendations when possible. SERVER-37565 should help alleviate this specific issue in 4.2 but for now please set memlock to "unlimited".

Thank you,

Danny

Comment by Ji Eun Lee [ 17/Jan/19 ]

I found this too.

2019-01-07T03:41:54.741+0900 F - [conn4290722] Failed to mlock: Unknown error
2019-01-07T03:41:54.741+0900 F - [conn4290722] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246

 

Is it same issue?

https://jira.mongodb.org/browse/SERVER-36600

 

And do I need to set 'max locked memory' to 'unlimit'?

 

 

Generated at Thu Feb 08 04:50:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.