[SERVER-52918] Mongos router repeatedly crashes Created: 17/Nov/20 Updated: 18/Nov/20 Resolved: 18/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Gavin AIken | Assignee: | Edwin Zhou |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | Connect to mongos, terminate the connection - mongos crashes |
||||||||
| Participants: | |||||||||
| Description |
|
Today the mongos router process in our staging environment started crashing repeatedly. If we do not attempt to open any mongo connections via it, it will stay up. But when you try to connect to mongo via it, the connection hangs (eg with the mongo cli you never reach the prompt), and if you terminate the connection (eg ctrl-c the mongo cli) the mongos immediately crashes, with the following messages being logged:
2020-11-17T12:52:37.449-0700 I NETWORK [listener] connection accepted from 127.0.0.1:53468 #6 (1 connection now open) , driver: { name: "MongoDB Internal Client", version: "4.2.6" }, os: { type: "Linux", name: "CentOS Linux release 7.6.1810 (Core) ", architecture: "x86_64", version: "Kernel 3.10.0-957.21.3.el7.x86_64" } }}} ,{"b":"55CF309B2000","o":"137D536"},{"b":"55CF309B2000","o":"1488156","s":"ZN10cxxabiv111_terminateEPFvvE"},{"b":"55CF309B2000","o":"1488191"},{"b":"55CF309B2000","o":"59A85D"},{"b":"55CF309B2000","o":"82F425"},{"b":"55CF309B2000","o":"82F992","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"55CF309B2000","o":"7464C0","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55CF309B2000","o":"77E1AC","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55CF309B2000","o":"7798DF","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55CF309B2000","o":"77B56C"},{"b":"55CF309B2000","o":"B027F2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55CF309B2000","o":"7767AD","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55CF309B2000","o":"778023","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55CF309B2000","o":"778BD6","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55CF309B2000","o":"77983B","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55CF309B2000","o":"77B56C"},{"b":"55CF309B2000","o":"B02C5B"},{"b":"55CF309B2000","o":"1215A45"},{"b":"55CF309B2000","o":"1215AA4"},{"b":"7F20C8340000","o":"7DD5"},{"b":"7F20C7F73000","o":"FDEAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.6", "gitVersion" : "20364840b8f1af16917e4c23c1b5f5efd8b352f8", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.21.3.el7.x86_64", "version" : "#1 SMP Tue Jun 18 16:35:19 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55CF309B2000", "elfType" : 3, "buildId" : "E432730CDAA0AC3E3913E1A4B4160334D52CB1C9" }, { "b" : "7FFD563D6000", "elfType" : 3, "buildId" : "BC4D21950F4B2ADFB515DFBB0E082E2281689A0B" }, { "b" : "7F20C976D000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "9114859D3C4BEC47A03CA321EE367DCA799638CD" }, { "b" : "7F20C9554000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "C444AE61E7CBB716FD9C18A0B46A7FE8F4FCF3E5" }, { "b" : "7F20C90F2000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "3593FA778645A59EA272DBBB59D318C60940E792" }, { "b" : "7F20C8E80000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7F20C8C7C000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "357693C8F1F49D93010C4E31529C07CDD2BD3D08" }, { "b" : "7F20C8A74000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7F20C8772000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5B14BE4D749631673523A61074C10959D50F5455" }, { "b" : "7F20C855C000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "179F202998E429AA1215907F6D4C5C1BB9C90136" }, { "b" : "7F20C8340000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "96900CB0FF25B26F2BBDF247DE1408242E4773D8" }, { "b" : "7F20C7F73000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "EB9F22A3891E5FD3494DFD9ED199E20AE71BB08D" }, { "b" : "7F20C99D6000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "A527FE72908703C5972AE384E78D1850D1881EE7" }, { "b" : "7F20C7D40000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7F20C7B16000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "689404B6B895EACFAD65BA16E07E5BF0004F5E0C" }, { "b" : "7F20C78C4000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "2E28F6A705F2ECEA8460D4716D5D1C24B5DDA5E4" }, { "b" : "7F20C769D000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "8D0B4010959C321022DF9CE239277A9D7B34A76A" }, { "b" : "7F20C7370000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "F5A64BB37FA3972E545EF459A51310F0AB56FA56" }, { "b" : "7F20C7140000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "E0705772325A52C3372FFFB8BDE5F786E2E200D6" }, { "b" : "7F20C6F3C000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "084D2194302908913F68B9DCD27DE46FA5B50522" }, { "b" : "7F20C6D37000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "799B28AD9A5460D78376E2C11260F2E858B95DE3" }, { "b" : "7F20C6AF9000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "DE762A28174110911B273E175D54F222B313CFE0" }, { "b" : "7F20C68AC000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "BCC30853830CD911E58700591830DF51ABCBD7BA" }, { "b" : "7F20C65C3000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "45BAB0BB455BDFA960FDA22E4124CF17B67CC930" }, { "b" : "7F20C6390000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "A9B3906192687CC45D483AE3C58C8AF745A6726A" }, { "b" : "7F20C618C000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B4BE1023D9606A88169DF411BF94AF417D7BA1A0" }, { "b" : "7F20C5F7D000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7F20C5D28000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7F20C5B12000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7F20C5902000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "94B3BCB669126166B77CDCE6092679A6AA2004C8" }, { "b" : "7F20C56FE000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F20C54E1000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7F20C52BA000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7F20C5083000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7F20C4E21000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F20C4C1E000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" }, { "b" : "7F20C4A0B000", "path" : "/lib64/libnss_files.so.2", "elfType" : 3, "buildId" : "EB4032E5BEEFD1751F164AE026A99F3FEA8F7454" }, { "b" : "7F20C4804000", "path" : "/lib64/libnss_dns.so.2", "elfType" : 3, "buildId" : "1CBCAFE76C83D1C6B0B69B361D723B629F26141A" } ] }}}} , processManagement: { pidFilePath: "/var/run/mongodb/mongos.pid" }, security: { keyFile: "/var/run/mongodb/keyfile" }, sharding: { configDB: "stage-metric-config/slc-stage-mongoc11:27017" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongos.log" } }}} |
| Comments |
| Comment by Edwin Zhou [ 18/Nov/20 ] |
|
Thanks for including the duplicate ticket and providing your fix for the issue. I'll go ahead and close it as a duplicate of Best, Edwin |
| Comment by Gavin AIken [ 17/Nov/20 ] |
|
Sorry, looks like this is a duplicate of https://jira.mongodb.org/browse/SERVER-47553
We found that a restart of our config set fixed the issue with the mongos.
It also sounds like the issue is fixed in later 4.2.x versions (we have 4.2.6) so we will plan to upgrade ASAP.
I think this issue can be closed. |