[SERVER-65879] Mongos crashes while connecting to replica config set Created: 21/Apr/22  Updated: 26/Apr/22  Resolved: 26/Apr/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Wouter van der Meulen Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

My mongod config looks like this

# /etc/mongod.conf
storage:
  dbPath: /data
  journal:
    enabled: true
net:
  port: 27019
  bindIp: localhost, MY_IPS
 
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
 
systemLog:
  destination: file
  logAppend: true
  path: /log/mongod.log
  logRotate: reopen
 
sharding:
  clusterRole: configsvr
 
replication:
  replSetName: db-cnf-0 

And mongos

sharding:
  configDB: db-cnf-0/myIPS
net:
  port: 27017
  bindIp: 0.0.0.0, localhost 

Am I missing something obvious here?

Participants:

 Description   

I'm trying to set up a config server for my sharded cluster. I've set up my replica set, and am trying to start my mongos instance. I'm greeted with the following error.

Please note that there is nothing running on port 27017. 

This is running on a Ubuntu 20.04 arm64 server running MongoDB 4.2. The config replica set is spread out over 3 servers. I tried running mongos on the config server as well as on its own standalone server. Same result. 

2022-04-21T17:09:28.467+0000 I  SHARDING [mongosMain] mongos version v4.2.19
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] db version v4.2.19
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] git version: e68a7d47305e14e090cba9ce3d92533053299996
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] allocator: tcmalloc
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] modules: none
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] build environment:
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distmod: ubuntu1804
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distarch: aarch64
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     target_arch: aarch64
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] options: { config: "/etc/mongos.conf", net: { bindIp: "0.0.0.0, 127.0.0.1, localhost", port: 27017 }, sharding: { configDB: "OBFUSCATED" } }
2022-04-21T17:09:28.468+0000 I  NETWORK  [mongosMain] Starting new replica set monitor for db-cnf-0/OBFUSCATED
2022-04-21T17:09:28.468+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
2022-04-21T17:09:28.469+0000 I  SHARDING [thread1] creating distributed lock ping thread for process ip-10-2-4-249:27017:1650560968:8561466342769151073 (sleeping for 30000ms)
2022-04-21T17:09:28.471+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBBFUSCATED
2022-04-21T17:09:28.473+0000 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for db-cnf-0 is db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating sharding state with confirmed set db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating ShardRegistry connection string for shard config from: db-cnf-0/MYIPS to: db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
2022-04-21T17:09:28.475+0000 I  SHARDING [ShardRegistry] Received reply from config server node (unknown) indicating config server optime term has increased, previous optime { ts: Timestamp(0, 0), t: -1 }, now { ts: Timestamp(1650560964, 1), t: 1 }
2022-04-21T17:09:28.478+0000 I  SHARDING [mongosMain] No hosts found to pre-warm connections to
2022-04-21T17:09:28.480+0000 W  FTDC     [mongosMain] FTDC is disabled because neither '--logpath' nor set parameter 'diagnosticDataCollectionDirectoryPath' are specified.
2022-04-21T17:09:28.480+0000 I  FTDC     [mongosMain] Initializing full-time diagnostic data capture with directory ''
2022-04-21T17:09:28.482+0000 I  SH_REFR  [ConfigServerCatalogCacheLoader-0] Refresh for database config from version {} to version { uuid: UUID("a0685a8c-97ba-486d-9b4c-18e717336623"), lastMod: 0 } took 0 ms
2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on /tmp/mongodb-27017.sock
2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on 0.0.0.0
2022-04-21T17:09:28.482+0000 F  NETWORK  [listener] Error listening for new connections on 127.0.0.1:27017: Address already in use
2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Collection config.system.sessions is not sharded.
2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: Collection config.system.sessions is not sharded.
2022-04-21T17:09:28.483+0000 F  -        [listener] Fatal Assertion 31339 at src/mongo/transport/transport_layer_asio.cpp 785
2022-04-21T17:09:28.483+0000 F  -        [listener] \n\n***aborting after fassert() failure\n\n
2022-04-21T17:09:28.484+0000 W  SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document
2022-04-21T17:09:28.486+0000 F  -        [listener] Got signal: 6 (Aborted).
 0xaaaae65e2094 0xaaaae65e11d0 0xaaaae65e127c 0xffffb026578c 0xffffafbebc20 0xffffafbd8a14 0xaaaae596e6e0 0xaaaae5da5494 0xaaaae5da5528 0xaaaae66f2cc0 0xffffafd303f0 0xffffafc880dc
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"AAAAE5390000","o":"1252094","s":"_ZN5mongo15printStackTraceERSo"},{"b":"AAAAE5390000","o":"12511D0"},{"b":"AAAAE5390000","o":"125127C"},{"b":"FFFFB0265000","o":"78C","s":"__kernel_rt_sigreturn"},{"b":"FFFFAFBB8000","o":"33C20","s":"gsignal"},{"b":"FFFFAFBB8000","o":"20A14","s":"abort"},{"b":"AAAAE5390000","o":"5DE6E0","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"AAAAE5390000","o":"A15494","s":"_ZN5mongo9transport18TransportLayerASIO12_runListenerEv"},{"b":"AAAAE5390000","o":"A15528"},{"b":"AAAAE5390000","o":"1362CC0"},{"b":"FFFFAFD29000","o":"73F0"},{"b":"FFFFAFBB8000","o":"D00DC"}],"processInfo":{ "mongodbVersion" : "4.2.19", "gitVersion" : "e68a7d47305e14e090cba9ce3d92533053299996", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.13.0-1021-aws", "version" : "#23~20.04.2-Ubuntu SMP Thu Mar 31 11:41:36 UTC 2022", "machine" : "aarch64" }, "somap" : [ { "b" : "AAAAE5390000", "elfType" : 3, "buildId" : "2EB94AF923DA0022BAADADA909D64A710B44D9D7" }, { "b" : "FFFFB0265000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F6CBCB08C6FD188535BF35E0FD667AE153AC4C10" }, { "b" : "FFFFB01A0000", "path" : "/lib/aarch64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "B2CADB37BCFB7A3C4F32098D8BBC4513A331EA31" }, { "b" : "FFFFB017A000", "path" : "/lib/aarch64-linux-gnu/atomics/libresolv.so.2", "elfType" : 3, "buildId" : "0889980C6B53EEF60CBCD8072CF9E26A26418985" }, { "b" : "FFFFAFEEC000", "path" : "/lib/aarch64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "FE5BFD5C808C258529FBD489A0A07819DE99396F" }, { "b" : "FFFFAFE52000", "path" : "/lib/aarch64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "1003C55BEE90B5AC856F0D04E3F0B5671F9E95C5" }, { "b" : "FFFFAFE3E000", "path" : "/lib/aarch64-linux-gnu/atomics/libdl.so.2", "elfType" : 3, "buildId" : "34203F04F392612E4F618E3A82102DA0873B0EB8" }, { "b" : "FFFFAFE26000", "path" : "/lib/aarch64-linux-gnu/atomics/librt.so.1", "elfType" : 3, "buildId" : "C1A210BDB1735559129EFF77F3FE07EB6C53F6DC" }, { "b" : "FFFFAFD7C000", "path" : "/lib/aarch64-linux-gnu/atomics/libm.so.6", "elfType" : 3, "buildId" : "2C4D44877508959E5085120F0E21152758DA5EDA" }, { "b" : "FFFFAFD58000", "path" : "/lib/aarch64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "5CF1FF7964D00E3857E0C08D512E490AF56450AE" }, { "b" : "FFFFAFD29000", "path" : "/lib/aarch64-linux-gnu/atomics/libpthread.so.0", "elfType" : 3, "buildId" : "272A3E7C5BA785839BF9F8732B886916B65BBCBB" }, { "b" : "FFFFAFBB8000", "path" : "/lib/aarch64-linux-gnu/atomics/libc.so.6", "elfType" : 3, "buildId" : "E875E919BDD805AE22A034BE621D5BB2EA968753" }, { "b" : "FFFFB0235000", "path" : "/lib/ld-linux-aarch64.so.1", "elfType" : 3, "buildId" : "DC2D7183E0FC2CE2314B285B2344E623E3BA32EA" }, { "b" : "FFFFAFB81000", "path" : "/lib/aarch64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "15EE90CC314D70F871DA0737047EDBC7806186D1" }, { "b" : "FFFFAFB53000", "path" : "/lib/aarch64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "2B78B76D9DD313CCCB3C56DFE89FEF48C66AF7CA" }, { "b" : "FFFFAFB25000", "path" : "/lib/aarch64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B8FE622E6CF0F29342D9D43130F7390A98F4D914" }, { "b" : "FFFFAFAAB000", "path" : "/lib/aarch64-linux-gnu/libssh.so.4", "elfType" : 3, "buildId" : "7D6CD533828EEDA574A0C622D560CB4D31ED71FF" }, { "b" : "FFFFAFA8A000", "path" : "/lib/aarch64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "4A76EB926F1B0D062FDEE5DB9DF6A8287926DC5E" }, { "b" : "FFFFAFA32000", "path" : "/lib/aarch64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B24B6DFD002CF6551A1BAFD70BEAA622F6C74812" }, { "b" : "FFFFAF9CF000", "path" : "/lib/aarch64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3DC44F0F44BBB5F820D47C5B729B2907B47BEA6A" }, { "b" : "FFFFAF9B0000", "path" : "/lib/aarch64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "56A38255FB102798517DD7E5059EAB3EF67257BE" }, { "b" : "FFFFAF993000", "path" : "/lib/aarch64-linux-gnu/libbrotlidec.so.1", "elfType" : 3, "buildId" : "77EDB932EFC8A0C1264AC95C34E2C6D7A9E54019" }, { "b" : "FFFFAF969000", "path" : "/lib/aarch64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "B2808FA40EC84B5F676F66C40F2E5B79CC2EA410" }, { "b" : "FFFFAF7E0000", "path" : "/lib/aarch64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "F6121326AC6A81C3E4F79E0C7EF847F7189D206F" }, { "b" : "FFFFAF5EF000", "path" : "/lib/aarch64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "9F9D7D525B0BBBF14B37BB86B6C0CDBCAE29DC54" }, { "b" : "FFFFAF5A9000", "path" : "/lib/aarch64-linux-gnu/libhogweed.so.5", "elfType" : 3, "buildId" : "F04E2DD390CCF7FB5055093D7104EF998B35F7E2" }, { "b" : "FFFFAF561000", "path" : "/lib/aarch64-linux-gnu/libnettle.so.7", "elfType" : 3, "buildId" : "3BBDDDE3403BEF31550E7FF1D7C790B6740DBBA6" }, { "b" : "FFFFAF4DA000", "path" : "/lib/aarch64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "54232CDCFAAB88C52B5F0552734EA738CA98E834" }, { "b" : "FFFFAF3F2000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "5A7CFAD186F1E2CCE1AFE406CFE1ACAFF1CCFB65" }, { "b" : "FFFFAF3B5000", "path" : "/lib/aarch64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "A4AA186A505BF111FD2DA4BEFFC51627C9EBDEF9" }, { "b" : "FFFFAF3A1000", "path" : "/lib/aarch64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "6FBFF9336DAA2FCED9AE195D8570A590D79C3D5F" }, { "b" : "FFFFAF382000", "path" : "/lib/aarch64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0D51294A52168B81B137615C52A91DBDBEE17A5" }, { "b" : "FFFFAF357000", "path" : "/lib/aarch64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "DDA470DFAC99C4285D6A3076E1AE8FB1FF56C51E" }, { "b" : "FFFFAF309000", "path" : "/lib/aarch64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "C3433D5F07588222745E0C6895D712B386543520" }, { "b" : "FFFFAF2D8000", "path" : "/lib/aarch64-linux-gnu/libbrotlicommon.so.1", "elfType" : 3, "buildId" : "CB0C35C5F908F3F23F338BF0ECFDC3F7C9B646CE" }, { "b" : "FFFFAF18C000", "path" : "/lib/aarch64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "FD60D6D25EA33DAD6820D0B560D1B89544911339" }, { "b" : "FFFFAF167000", "path" : "/lib/aarch64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "8B9AE981258AC903E122AC93F717C29B31BDFB25" }, { "b" : "FFFFAF152000", "path" : "/lib/aarch64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "1A8415772833B993A593F6A42FFEECCCF36971D8" }, { "b" : "FFFFAF138000", "path" : "/lib/aarch64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "DB279FCB212692DF41F7BA16BAAF2D0BD9B773C3" }, { "b" : "FFFFAF09C000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "B2D822DC3C4EA03F11453483B503775C4D205C19" }, { "b" : "FFFFAEFF7000", "path" : "/lib/aarch64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "F7637D30B767964029BF8D3AED2D104F71B7E2D7" }, { "b" : "FFFFAEFAE000", "path" : "/lib/aarch64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "298A583F5B3035894AB4146592C65B9C4CE1A2BF" }, { "b" : "FFFFAEF88000", "path" : "/lib/aarch64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "4CDEFF83DFEF374F8BA07122F322E2A533A58C9B" }, { "b" : "FFFFAEF6F000", "path" : "/lib/aarch64-linux-gnu/libffi.so.7", "elfType" : 3, "buildId" : "095BA01481950C7652CB19EC8342F921B2EB8246" }, { "b" : "FFFFAEF36000", "path" : "/lib/aarch64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "DE842D5CDFF82E6C85AA7742FCB64BB752F7AE57" }, { "b" : "FFFFAEF17000", "path" : "/lib/aarch64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "069813AEB10E3FFE61191AC014EEBA8B17293986" }, { "b" : "FFFFAEEBB000", "path" : "/lib/aarch64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "5079CE381493D15064E9916E5880014CD456304A" }, { "b" : "FFFFAED89000", "path" : "/lib/aarch64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "DD2644987B96D32BC4CC61C6C884A416450B883E" }, { "b" : "FFFFAED40000", "path" : "/lib/aarch64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7DD2207B47D98892369A2F56C2B39179468BEF4C" }, { "b" : "FFFFAED19000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_files.so.2", "elfType" : 3, "buildId" : "0717293A95775EEB374442A0A201259056E89D48" }, { "b" : "FFFFAED03000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_dns.so.2", "elfType" : 3, "buildId" : "1A6829FF337426F86D047C089DECF92C969533E8" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x44) [0xaaaae65e2094]
 mongos(+0x12511D0) [0xaaaae65e11d0]
 mongos(+0x125127C) [0xaaaae65e127c]
 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffffb026578c]
 libc.so.6(gsignal+0xE0) [0xffffafbebc20]
 libc.so.6(abort+0xFC) [0xffffafbd8a14]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0xaaaae596e6e0]
 mongos(_ZN5mongo9transport18TransportLayerASIO12_runListenerEv+0x5FC) [0xaaaae5da5494]
 mongos(+0xA15528) [0xaaaae5da5528]
 mongos(+0x1362CC0) [0xaaaae66f2cc0]
 libpthread.so.0(+0x73F0) [0xffffafd303f0]
 libc.so.6(+0xD00DC) [0xffffafc880dc]
-----  END BACKTRACE  ----- 



 Comments   
Comment by Chris Kelly [ 26/Apr/22 ]

Hello Wouter,

I was able to reproduce your problem. In this case, it appears to be an easy fix. While you may not be running anything on 27017 on your servers, your mongos config is binding to both 0.0.0.0 and localhost. There is some nuance on the differences and similarities between these that are a good read here.

To oversimplify a bit, you can think of 0.0.0.0 including 127.0.0.1 in this context, which basically means you were telling it to bind to 127.0.0.1 twice. This is because 0.0.0.0 generally means to listen on all available network interfaces.

From the perspective of mongos, if told to listen on 0.0.0.0, it will accept connections on any interface including 127.0.0.1.

Try giving it a shot by removing 127.0.0.1 from your bindIp, and feel free to consult our reference on deploying sharded clusters here, or consult our MongoDB Developer Community Forums for more information.

Regards,
Christopher

Generated at Thu Feb 08 06:03:52 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.