-
Type:
Bug
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 4.2.1
-
Component/s: None
-
Labels:
-
Environment:MongoDB Version: 4.2.1
Operating System: Ubuntu 18.04.3 LTS
Linux Kernel Info: Linux C-Mongos-1 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
-
ALL
-
I have three MongoDB sharded clusters, and all of them have experienced the following issue at different times: all mongos processes in the cluster suddenly start repeatedly restarting, approximately twice per minute, with the following log entries:
2023-10-09T07:09:33.788+0800 F - [conn51] terminate() called. No exception is active 0x5588507c2251 0x5588507c2008 0x5
588508cc636 0x5588508cc671 0x55884fa4b37c 0x55884fcc5fc9 0x55884fcc6542 0x55884fbeba90 0x55884fc0f1ac 0x55884fc0947f 0x55884fc0c85c 0x55884ff763d2 0x55884fc06ddd 0x55884fc0a133 0x55884fc084a7 0x55884fc093db 0x55884fc0c85c 0x55884ff7683b 0x558850669c34 0x7faff79356db 0x7faff765e88f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"564F1266F000","o":"12F4251","s":"_ZN5mongo15printStackTraceERSo"},{"b":"564F1266F000","o":"12F4008"},{"b":"564F1266F000","o":"13FE636","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"564F1266F000","o":"13FE671"},{"b":"564F1266F000","o":"57D37C"},{"b":"564F1266F000","o":"7F7FC9"},{"b":"564F1266F000","o":"7F8542","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"564F1266F000","o":"71DA90","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"564F1266F000","o":"7411AC","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"564F1266F000","o":"73B47F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"564F1266F000","o":"73E85C"},{"b":"564F1266F000","o":"AA83D2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"564F1266F000","o":"738DDD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"564F1266F000","o":"73C133","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"564F1266F000","o":"73A4A7","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"564F1266F000","o":"73B3DB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"564F1266F000","o":"73E85C"},{"b":"564F1266F000","o":"AA883B"},{"b":"564F1266F000","o":"119BC34"},{"b":"7F1C5129B000","o":"76DB"},{"b":"7F1C50EAA000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.1", "gitVersion" : "edf6d45851c0b9ee15548f0f847df141764a317e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-72-generic", "version" : "#81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "564F1266F000", "elfType" : 3, "buildId" : "3F81A0FC7E34954797F70C381FB39C1D2DFADD4A" }, { "b" : "7FFDE8D99000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "CB3D060291CD129C4F6FB35E537B9822932FB871" }, { "b" : "7F1C527EF000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "1C6BC2C0699CE0F7E848CA0B267E0CF07553F6AB" }, { "b" : "7F1C525D4000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F1C52109000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "68CECD8742C0C91CDC82709C3E52A8C9F9451FC5" }, { "b" : "7F1C51E7C000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "38AE5B4499DA422AA6D86BAED1902662DED5F730" }, { "b" : "7F1C51C78000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F1C51A70000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F1C516D2000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F1C514BA000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "41BDC55C07D5E5B1D8AB38E2C19B1F535855E084" }, { "b" : "7F1C5129B000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F1C50EAA000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F1C52A6E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F1C50C85000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F1C50A68000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "EE6E9462BA2491F4EE8C4E52C3323274A9366614" }, { "b" : "7F1C5084C000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F1C5063E000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F1C503F3000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "00F419F64B0E70D8C5EEF7050369AA40B2A6E090" }, { "b" : "7F1C501A1000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "70EEF126558D1559A0A4E334FB68E4E9AABE90CB" }, { "b" : "7F1C4FF93000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "C14042EC7BD22B9A07D2C16563FE3C2606F52AB7" }, { "b" : "7F1C4FD76000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F1C4F9F8000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "0E2784298E7D3F4D894FE130ACEFA77C3E624F72" }, { "b" : "7F1C4F693000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "E5AE5C31F804BE96532D0DB2091F19E472F2D4A0" }, { "b" : "7F1C4F45F000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F1C4F229000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F1C4EFA8000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F1C4ECD2000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "69FBCF425EE6DF03DE93B82FBC2FC33790E68A96" }, { "b" : "7F1C4EAA0000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "F400D5D643A7F9696DF0E6148FA99BEE6C1BDDF7" }, { "b" : "7F1C4E89C000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "17107881DF65C66B4C6D38CAB37C285FA44663BD" }, { "b" : "7F1C4E691000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "D78D71E8E016A534281B25B97CD7E5E9DB5FE00A" }, { "b" : "7F1C4E476000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F1C4E235000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F1C4DF06000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F1C4DCF3000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F1C4DAEF000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F1C4D8E6000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F1C4D659000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F1C4D3B7000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F1C4D181000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F1C4CF6B000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F1C4CD63000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F1C4CB3A000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F1C4C92B000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F1C4C6E1000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F1C4C3D8000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "A609DB077BDC3B54A8C64BBCF82C2B7BD1CC8B98" }, { "b" : "7F1C4C1A0000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" }, { "b" : "7F1C4BF8E000", "path" : "/lib/x86_64-linux-gnu/libnss_files.so.2", "elfType" : 3, "buildId" : "53BD03ADA320ECD75955334A752B75D0EE3B77F5" } ] }}
mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x5588507c2251]
mongos(+0x12F4008) [0x5588507c2008]
mongos(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x5588508cc636]
mongos(+0x13FE671) [0x5588508cc671]
mongos(+0x57D37C) [0x55884fa4b37c]
mongos(+0x7F7FC9) [0x55884fcc5fc9]
mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x1C2) [0x55884fcc6542]
mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3D0) [0x55884fbeba90]
mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x55884fc0f1ac]
mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x55884fc0947f]
mongos(+0x73E85C) [0x55884fc0c85c]
mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23S
erviceExecutorTaskNameE+0x182) [0x55884ff763d2]
mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlags
ENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x55884fc06ddd]
mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x843) [0x55884fc0a133]
mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x2E7) [0x55884fc084a7]
mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x55884fc093db]
mongos(+0x73E85C) [0x55884fc0c85c]
mongos(+0xAA883B) [0x55884ff7683b]
mongos(+0x119BC34) [0x558850669c34]
libpthread.so.0(+0x76DB) [0x7faff79356db]
libc.so.6(clone+0x3F) [0x7faff765e88f]
----- END BACKTRACE -----
I eventually discovered that simply restarting all mongo config processes automatically restores the normal operation of the mongos processes. But this cannot be a reliable, long-term solution. I haven't found any anomalies or anything higher than an 'Info' level message in the config and mongod logs.
Has anyone else encountered a similar problem? How should I proceed with further diagnosis and resolution? Any advice or guidance would be greatly appreciated. Thank you!
- is related to
-
-
- Closed
-