Mongos crashes while connecting to replica config set

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL
    • Hide

      My mongod config looks like this

      # /etc/mongod.conf
storage:
  dbPath: /data
  journal:
    enabled: true
net:
  port: 27019
  bindIp: localhost, MY_IPS

processManagement:
  timeZoneInfo: /usr/share/zoneinfo

systemLog:
  destination: file
  logAppend: true
  path: /log/mongod.log
  logRotate: reopen

sharding:
  clusterRole: configsvr

replication:
  replSetName: db-cnf-0

      And mongos

      sharding:
  configDB: db-cnf-0/myIPS
net:
  port: 27017
  bindIp: 0.0.0.0, localhost

      Am I missing something obvious here?

      Show
      My mongod config looks like this # /etc/mongod.conf storage:   dbPath: /data   journal:     enabled: true net:   port: 27019   bindIp: localhost, MY_IPS processManagement:   timeZoneInfo: /usr/share/zoneinfo systemLog:   destination: file   logAppend: true   path: /log/mongod.log   logRotate: reopen sharding:   clusterRole: configsvr replication:   replSetName: db-cnf-0 And mongos sharding:   configDB: db-cnf-0/myIPS net:   port: 27017   bindIp: 0.0.0.0, localhost Am I missing something obvious here?
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      I'm trying to set up a config server for my sharded cluster. I've set up my replica set, and am trying to start my mongos instance. I'm greeted with the following error.

      Please note that there is nothing running on port 27017. 

      This is running on a Ubuntu 20.04 arm64 server running MongoDB 4.2. The config replica set is spread out over 3 servers. I tried running mongos on the config server as well as on its own standalone server. Same result. 

      2022-04-21T17:09:28.467+0000 I  SHARDING [mongosMain] mongos version v4.2.19
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] db version v4.2.19
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] git version: e68a7d47305e14e090cba9ce3d92533053299996
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] allocator: tcmalloc
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] modules: none
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] build environment:
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distmod: ubuntu1804
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distarch: aarch64
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     target_arch: aarch64
2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] options: { config: "/etc/mongos.conf", net: { bindIp: "0.0.0.0, 127.0.0.1, localhost", port: 27017 }, sharding: { configDB: "OBFUSCATED" } }
2022-04-21T17:09:28.468+0000 I  NETWORK  [mongosMain] Starting new replica set monitor for db-cnf-0/OBFUSCATED
2022-04-21T17:09:28.468+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
2022-04-21T17:09:28.469+0000 I  SHARDING [thread1] creating distributed lock ping thread for process ip-10-2-4-249:27017:1650560968:8561466342769151073 (sleeping for 30000ms)
2022-04-21T17:09:28.471+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBBFUSCATED
2022-04-21T17:09:28.473+0000 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for db-cnf-0 is db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating sharding state with confirmed set db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating ShardRegistry connection string for shard config from: db-cnf-0/MYIPS to: db-cnf-0/MYIPS
2022-04-21T17:09:28.473+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
2022-04-21T17:09:28.475+0000 I  SHARDING [ShardRegistry] Received reply from config server node (unknown) indicating config server optime term has increased, previous optime { ts: Timestamp(0, 0), t: -1 }, now { ts: Timestamp(1650560964, 1), t: 1 }
2022-04-21T17:09:28.478+0000 I  SHARDING [mongosMain] No hosts found to pre-warm connections to
2022-04-21T17:09:28.480+0000 W  FTDC     [mongosMain] FTDC is disabled because neither '--logpath' nor set parameter 'diagnosticDataCollectionDirectoryPath' are specified.
2022-04-21T17:09:28.480+0000 I  FTDC     [mongosMain] Initializing full-time diagnostic data capture with directory ''
2022-04-21T17:09:28.482+0000 I  SH_REFR  [ConfigServerCatalogCacheLoader-0] Refresh for database config from version {} to version { uuid: UUID("a0685a8c-97ba-486d-9b4c-18e717336623"), lastMod: 0 } took 0 ms
2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on /tmp/mongodb-27017.sock
2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on 0.0.0.0
2022-04-21T17:09:28.482+0000 F  NETWORK  [listener] Error listening for new connections on 127.0.0.1:27017: Address already in use
2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Collection config.system.sessions is not sharded.
2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: Collection config.system.sessions is not sharded.
2022-04-21T17:09:28.483+0000 F  -        [listener] Fatal Assertion 31339 at src/mongo/transport/transport_layer_asio.cpp 785
2022-04-21T17:09:28.483+0000 F  -        [listener] \n\n***aborting after fassert() failure\n\n
2022-04-21T17:09:28.484+0000 W  SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document
2022-04-21T17:09:28.486+0000 F  -        [listener] Got signal: 6 (Aborted).
 0xaaaae65e2094 0xaaaae65e11d0 0xaaaae65e127c 0xffffb026578c 0xffffafbebc20 0xffffafbd8a14 0xaaaae596e6e0 0xaaaae5da5494 0xaaaae5da5528 0xaaaae66f2cc0 0xffffafd303f0 0xffffafc880dc
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"AAAAE5390000","o":"1252094","s":"_ZN5mongo15printStackTraceERSo"},{"b":"AAAAE5390000","o":"12511D0"},{"b":"AAAAE5390000","o":"125127C"},{"b":"FFFFB0265000","o":"78C","s":"__kernel_rt_sigreturn"},{"b":"FFFFAFBB8000","o":"33C20","s":"gsignal"},{"b":"FFFFAFBB8000","o":"20A14","s":"abort"},{"b":"AAAAE5390000","o":"5DE6E0","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"AAAAE5390000","o":"A15494","s":"_ZN5mongo9transport18TransportLayerASIO12_runListenerEv"},{"b":"AAAAE5390000","o":"A15528"},{"b":"AAAAE5390000","o":"1362CC0"},{"b":"FFFFAFD29000","o":"73F0"},{"b":"FFFFAFBB8000","o":"D00DC"}],"processInfo":{ "mongodbVersion" : "4.2.19", "gitVersion" : "e68a7d47305e14e090cba9ce3d92533053299996", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.13.0-1021-aws", "version" : "#23~20.04.2-Ubuntu SMP Thu Mar 31 11:41:36 UTC 2022", "machine" : "aarch64" }, "somap" : [ { "b" : "AAAAE5390000", "elfType" : 3, "buildId" : "2EB94AF923DA0022BAADADA909D64A710B44D9D7" }, { "b" : "FFFFB0265000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F6CBCB08C6FD188535BF35E0FD667AE153AC4C10" }, { "b" : "FFFFB01A0000", "path" : "/lib/aarch64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "B2CADB37BCFB7A3C4F32098D8BBC4513A331EA31" }, { "b" : "FFFFB017A000", "path" : "/lib/aarch64-linux-gnu/atomics/libresolv.so.2", "elfType" : 3, "buildId" : "0889980C6B53EEF60CBCD8072CF9E26A26418985" }, { "b" : "FFFFAFEEC000", "path" : "/lib/aarch64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "FE5BFD5C808C258529FBD489A0A07819DE99396F" }, { "b" : "FFFFAFE52000", "path" : "/lib/aarch64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "1003C55BEE90B5AC856F0D04E3F0B5671F9E95C5" }, { "b" : "FFFFAFE3E000", "path" : "/lib/aarch64-linux-gnu/atomics/libdl.so.2", "elfType" : 3, "buildId" : "34203F04F392612E4F618E3A82102DA0873B0EB8" }, { "b" : "FFFFAFE26000", "path" : "/lib/aarch64-linux-gnu/atomics/librt.so.1", "elfType" : 3, "buildId" : "C1A210BDB1735559129EFF77F3FE07EB6C53F6DC" }, { "b" : "FFFFAFD7C000", "path" : "/lib/aarch64-linux-gnu/atomics/libm.so.6", "elfType" : 3, "buildId" : "2C4D44877508959E5085120F0E21152758DA5EDA" }, { "b" : "FFFFAFD58000", "path" : "/lib/aarch64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "5CF1FF7964D00E3857E0C08D512E490AF56450AE" }, { "b" : "FFFFAFD29000", "path" : "/lib/aarch64-linux-gnu/atomics/libpthread.so.0", "elfType" : 3, "buildId" : "272A3E7C5BA785839BF9F8732B886916B65BBCBB" }, { "b" : "FFFFAFBB8000", "path" : "/lib/aarch64-linux-gnu/atomics/libc.so.6", "elfType" : 3, "buildId" : "E875E919BDD805AE22A034BE621D5BB2EA968753" }, { "b" : "FFFFB0235000", "path" : "/lib/ld-linux-aarch64.so.1", "elfType" : 3, "buildId" : "DC2D7183E0FC2CE2314B285B2344E623E3BA32EA" }, { "b" : "FFFFAFB81000", "path" : "/lib/aarch64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "15EE90CC314D70F871DA0737047EDBC7806186D1" }, { "b" : "FFFFAFB53000", "path" : "/lib/aarch64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "2B78B76D9DD313CCCB3C56DFE89FEF48C66AF7CA" }, { "b" : "FFFFAFB25000", "path" : "/lib/aarch64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B8FE622E6CF0F29342D9D43130F7390A98F4D914" }, { "b" : "FFFFAFAAB000", "path" : "/lib/aarch64-linux-gnu/libssh.so.4", "elfType" : 3, "buildId" : "7D6CD533828EEDA574A0C622D560CB4D31ED71FF" }, { "b" : "FFFFAFA8A000", "path" : "/lib/aarch64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "4A76EB926F1B0D062FDEE5DB9DF6A8287926DC5E" }, { "b" : "FFFFAFA32000", "path" : "/lib/aarch64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B24B6DFD002CF6551A1BAFD70BEAA622F6C74812" }, { "b" : "FFFFAF9CF000", "path" : "/lib/aarch64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3DC44F0F44BBB5F820D47C5B729B2907B47BEA6A" }, { "b" : "FFFFAF9B0000", "path" : "/lib/aarch64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "56A38255FB102798517DD7E5059EAB3EF67257BE" }, { "b" : "FFFFAF993000", "path" : "/lib/aarch64-linux-gnu/libbrotlidec.so.1", "elfType" : 3, "buildId" : "77EDB932EFC8A0C1264AC95C34E2C6D7A9E54019" }, { "b" : "FFFFAF969000", "path" : "/lib/aarch64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "B2808FA40EC84B5F676F66C40F2E5B79CC2EA410" }, { "b" : "FFFFAF7E0000", "path" : "/lib/aarch64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "F6121326AC6A81C3E4F79E0C7EF847F7189D206F" }, { "b" : "FFFFAF5EF000", "path" : "/lib/aarch64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "9F9D7D525B0BBBF14B37BB86B6C0CDBCAE29DC54" }, { "b" : "FFFFAF5A9000", "path" : "/lib/aarch64-linux-gnu/libhogweed.so.5", "elfType" : 3, "buildId" : "F04E2DD390CCF7FB5055093D7104EF998B35F7E2" }, { "b" : "FFFFAF561000", "path" : "/lib/aarch64-linux-gnu/libnettle.so.7", "elfType" : 3, "buildId" : "3BBDDDE3403BEF31550E7FF1D7C790B6740DBBA6" }, { "b" : "FFFFAF4DA000", "path" : "/lib/aarch64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "54232CDCFAAB88C52B5F0552734EA738CA98E834" }, { "b" : "FFFFAF3F2000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "5A7CFAD186F1E2CCE1AFE406CFE1ACAFF1CCFB65" }, { "b" : "FFFFAF3B5000", "path" : "/lib/aarch64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "A4AA186A505BF111FD2DA4BEFFC51627C9EBDEF9" }, { "b" : "FFFFAF3A1000", "path" : "/lib/aarch64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "6FBFF9336DAA2FCED9AE195D8570A590D79C3D5F" }, { "b" : "FFFFAF382000", "path" : "/lib/aarch64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0D51294A52168B81B137615C52A91DBDBEE17A5" }, { "b" : "FFFFAF357000", "path" : "/lib/aarch64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "DDA470DFAC99C4285D6A3076E1AE8FB1FF56C51E" }, { "b" : "FFFFAF309000", "path" : "/lib/aarch64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "C3433D5F07588222745E0C6895D712B386543520" }, { "b" : "FFFFAF2D8000", "path" : "/lib/aarch64-linux-gnu/libbrotlicommon.so.1", "elfType" : 3, "buildId" : "CB0C35C5F908F3F23F338BF0ECFDC3F7C9B646CE" }, { "b" : "FFFFAF18C000", "path" : "/lib/aarch64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "FD60D6D25EA33DAD6820D0B560D1B89544911339" }, { "b" : "FFFFAF167000", "path" : "/lib/aarch64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "8B9AE981258AC903E122AC93F717C29B31BDFB25" }, { "b" : "FFFFAF152000", "path" : "/lib/aarch64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "1A8415772833B993A593F6A42FFEECCCF36971D8" }, { "b" : "FFFFAF138000", "path" : "/lib/aarch64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "DB279FCB212692DF41F7BA16BAAF2D0BD9B773C3" }, { "b" : "FFFFAF09C000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "B2D822DC3C4EA03F11453483B503775C4D205C19" }, { "b" : "FFFFAEFF7000", "path" : "/lib/aarch64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "F7637D30B767964029BF8D3AED2D104F71B7E2D7" }, { "b" : "FFFFAEFAE000", "path" : "/lib/aarch64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "298A583F5B3035894AB4146592C65B9C4CE1A2BF" }, { "b" : "FFFFAEF88000", "path" : "/lib/aarch64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "4CDEFF83DFEF374F8BA07122F322E2A533A58C9B" }, { "b" : "FFFFAEF6F000", "path" : "/lib/aarch64-linux-gnu/libffi.so.7", "elfType" : 3, "buildId" : "095BA01481950C7652CB19EC8342F921B2EB8246" }, { "b" : "FFFFAEF36000", "path" : "/lib/aarch64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "DE842D5CDFF82E6C85AA7742FCB64BB752F7AE57" }, { "b" : "FFFFAEF17000", "path" : "/lib/aarch64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "069813AEB10E3FFE61191AC014EEBA8B17293986" }, { "b" : "FFFFAEEBB000", "path" : "/lib/aarch64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "5079CE381493D15064E9916E5880014CD456304A" }, { "b" : "FFFFAED89000", "path" : "/lib/aarch64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "DD2644987B96D32BC4CC61C6C884A416450B883E" }, { "b" : "FFFFAED40000", "path" : "/lib/aarch64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7DD2207B47D98892369A2F56C2B39179468BEF4C" }, { "b" : "FFFFAED19000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_files.so.2", "elfType" : 3, "buildId" : "0717293A95775EEB374442A0A201259056E89D48" }, { "b" : "FFFFAED03000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_dns.so.2", "elfType" : 3, "buildId" : "1A6829FF337426F86D047C089DECF92C969533E8" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x44) [0xaaaae65e2094]
 mongos(+0x12511D0) [0xaaaae65e11d0]
 mongos(+0x125127C) [0xaaaae65e127c]
 linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffffb026578c]
 libc.so.6(gsignal+0xE0) [0xffffafbebc20]
 libc.so.6(abort+0xFC) [0xffffafbd8a14]
 mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0xaaaae596e6e0]
 mongos(_ZN5mongo9transport18TransportLayerASIO12_runListenerEv+0x5FC) [0xaaaae5da5494]
 mongos(+0xA15528) [0xaaaae5da5528]
 mongos(+0x1362CC0) [0xaaaae66f2cc0]
 libpthread.so.0(+0x73F0) [0xffffafd303f0]
 libc.so.6(+0xD00DC) [0xffffafc880dc]
-----  END BACKTRACE  -----

            Assignee:
            Chris Kelly
            Reporter:
            Wouter van der Meulen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: