Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-65879

Mongos crashes while connecting to replica config set

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • None
    • None
    • None
    • ALL
    • Hide

      My mongod config looks like this

      # /etc/mongod.conf
      storage:
        dbPath: /data
        journal:
          enabled: true
      net:
        port: 27019
        bindIp: localhost, MY_IPS
       
      processManagement:
        timeZoneInfo: /usr/share/zoneinfo
       
      systemLog:
        destination: file
        logAppend: true
        path: /log/mongod.log
        logRotate: reopen
       
      sharding:
        clusterRole: configsvr
       
      replication:
        replSetName: db-cnf-0 

      And mongos

      sharding:
        configDB: db-cnf-0/myIPS
      net:
        port: 27017
        bindIp: 0.0.0.0, localhost 

      Am I missing something obvious here?

      Show
      My mongod config looks like this # /etc/mongod.conf storage:   dbPath: /data   journal:     enabled: true net:   port: 27019   bindIp: localhost, MY_IPS   processManagement:   timeZoneInfo: /usr/share/zoneinfo   systemLog:   destination: file   logAppend: true   path: /log/mongod.log   logRotate: reopen   sharding:   clusterRole: configsvr   replication:   replSetName: db-cnf- 0 And mongos sharding:   configDB: db-cnf- 0 /myIPS net:   port: 27017   bindIp: 0.0 . 0.0 , localhost Am I missing something obvious here?

    Description

      I'm trying to set up a config server for my sharded cluster. I've set up my replica set, and am trying to start my mongos instance. I'm greeted with the following error.

      Please note that there is nothing running on port 27017. 

      This is running on a Ubuntu 20.04 arm64 server running MongoDB 4.2. The config replica set is spread out over 3 servers. I tried running mongos on the config server as well as on its own standalone server. Same result. 

      2022-04-21T17:09:28.467+0000 I  SHARDING [mongosMain] mongos version v4.2.19
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] db version v4.2.19
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] git version: e68a7d47305e14e090cba9ce3d92533053299996
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] allocator: tcmalloc
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] modules: none
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] build environment:
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distmod: ubuntu1804
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     distarch: aarch64
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain]     target_arch: aarch64
      2022-04-21T17:09:28.467+0000 I  CONTROL  [mongosMain] options: { config: "/etc/mongos.conf", net: { bindIp: "0.0.0.0, 127.0.0.1, localhost", port: 27017 }, sharding: { configDB: "OBFUSCATED" } }
      2022-04-21T17:09:28.468+0000 I  NETWORK  [mongosMain] Starting new replica set monitor for db-cnf-0/OBFUSCATED
      2022-04-21T17:09:28.468+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
      2022-04-21T17:09:28.469+0000 I  SHARDING [thread1] creating distributed lock ping thread for process ip-10-2-4-249:27017:1650560968:8561466342769151073 (sleeping for 30000ms)
      2022-04-21T17:09:28.471+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBBFUSCATED
      2022-04-21T17:09:28.473+0000 I  NETWORK  [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for db-cnf-0 is db-cnf-0/MYIPS
      2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating sharding state with confirmed set db-cnf-0/MYIPS
      2022-04-21T17:09:28.473+0000 I  SHARDING [Sharding-Fixed-0] Updating ShardRegistry connection string for shard config from: db-cnf-0/MYIPS to: db-cnf-0/MYIPS
      2022-04-21T17:09:28.473+0000 I  CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to OBFUSCATED
      2022-04-21T17:09:28.475+0000 I  SHARDING [ShardRegistry] Received reply from config server node (unknown) indicating config server optime term has increased, previous optime { ts: Timestamp(0, 0), t: -1 }, now { ts: Timestamp(1650560964, 1), t: 1 }
      2022-04-21T17:09:28.478+0000 I  SHARDING [mongosMain] No hosts found to pre-warm connections to
      2022-04-21T17:09:28.480+0000 W  FTDC     [mongosMain] FTDC is disabled because neither '--logpath' nor set parameter 'diagnosticDataCollectionDirectoryPath' are specified.
      2022-04-21T17:09:28.480+0000 I  FTDC     [mongosMain] Initializing full-time diagnostic data capture with directory ''
      2022-04-21T17:09:28.482+0000 I  SH_REFR  [ConfigServerCatalogCacheLoader-0] Refresh for database config from version {} to version { uuid: UUID("a0685a8c-97ba-486d-9b4c-18e717336623"), lastMod: 0 } took 0 ms
      2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on /tmp/mongodb-27017.sock
      2022-04-21T17:09:28.482+0000 I  NETWORK  [listener] Listening on 0.0.0.0
      2022-04-21T17:09:28.482+0000 F  NETWORK  [listener] Error listening for new connections on 127.0.0.1:27017: Address already in use
      2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Collection config.system.sessions is not sharded.
      2022-04-21T17:09:28.482+0000 I  CONTROL  [LogicalSessionCacheReap] Sessions collection is not set up; waiting until next sessions reap interval: Collection config.system.sessions is not sharded.
      2022-04-21T17:09:28.483+0000 F  -        [listener] Fatal Assertion 31339 at src/mongo/transport/transport_layer_asio.cpp 785
      2022-04-21T17:09:28.483+0000 F  -        [listener] \n\n***aborting after fassert() failure\n\n
      2022-04-21T17:09:28.484+0000 W  SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document
      2022-04-21T17:09:28.486+0000 F  -        [listener] Got signal: 6 (Aborted).
       0xaaaae65e2094 0xaaaae65e11d0 0xaaaae65e127c 0xffffb026578c 0xffffafbebc20 0xffffafbd8a14 0xaaaae596e6e0 0xaaaae5da5494 0xaaaae5da5528 0xaaaae66f2cc0 0xffffafd303f0 0xffffafc880dc
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"AAAAE5390000","o":"1252094","s":"_ZN5mongo15printStackTraceERSo"},{"b":"AAAAE5390000","o":"12511D0"},{"b":"AAAAE5390000","o":"125127C"},{"b":"FFFFB0265000","o":"78C","s":"__kernel_rt_sigreturn"},{"b":"FFFFAFBB8000","o":"33C20","s":"gsignal"},{"b":"FFFFAFBB8000","o":"20A14","s":"abort"},{"b":"AAAAE5390000","o":"5DE6E0","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"AAAAE5390000","o":"A15494","s":"_ZN5mongo9transport18TransportLayerASIO12_runListenerEv"},{"b":"AAAAE5390000","o":"A15528"},{"b":"AAAAE5390000","o":"1362CC0"},{"b":"FFFFAFD29000","o":"73F0"},{"b":"FFFFAFBB8000","o":"D00DC"}],"processInfo":{ "mongodbVersion" : "4.2.19", "gitVersion" : "e68a7d47305e14e090cba9ce3d92533053299996", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "5.13.0-1021-aws", "version" : "#23~20.04.2-Ubuntu SMP Thu Mar 31 11:41:36 UTC 2022", "machine" : "aarch64" }, "somap" : [ { "b" : "AAAAE5390000", "elfType" : 3, "buildId" : "2EB94AF923DA0022BAADADA909D64A710B44D9D7" }, { "b" : "FFFFB0265000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "F6CBCB08C6FD188535BF35E0FD667AE153AC4C10" }, { "b" : "FFFFB01A0000", "path" : "/lib/aarch64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "B2CADB37BCFB7A3C4F32098D8BBC4513A331EA31" }, { "b" : "FFFFB017A000", "path" : "/lib/aarch64-linux-gnu/atomics/libresolv.so.2", "elfType" : 3, "buildId" : "0889980C6B53EEF60CBCD8072CF9E26A26418985" }, { "b" : "FFFFAFEEC000", "path" : "/lib/aarch64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "FE5BFD5C808C258529FBD489A0A07819DE99396F" }, { "b" : "FFFFAFE52000", "path" : "/lib/aarch64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "1003C55BEE90B5AC856F0D04E3F0B5671F9E95C5" }, { "b" : "FFFFAFE3E000", "path" : "/lib/aarch64-linux-gnu/atomics/libdl.so.2", "elfType" : 3, "buildId" : "34203F04F392612E4F618E3A82102DA0873B0EB8" }, { "b" : "FFFFAFE26000", "path" : "/lib/aarch64-linux-gnu/atomics/librt.so.1", "elfType" : 3, "buildId" : "C1A210BDB1735559129EFF77F3FE07EB6C53F6DC" }, { "b" : "FFFFAFD7C000", "path" : "/lib/aarch64-linux-gnu/atomics/libm.so.6", "elfType" : 3, "buildId" : "2C4D44877508959E5085120F0E21152758DA5EDA" }, { "b" : "FFFFAFD58000", "path" : "/lib/aarch64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "5CF1FF7964D00E3857E0C08D512E490AF56450AE" }, { "b" : "FFFFAFD29000", "path" : "/lib/aarch64-linux-gnu/atomics/libpthread.so.0", "elfType" : 3, "buildId" : "272A3E7C5BA785839BF9F8732B886916B65BBCBB" }, { "b" : "FFFFAFBB8000", "path" : "/lib/aarch64-linux-gnu/atomics/libc.so.6", "elfType" : 3, "buildId" : "E875E919BDD805AE22A034BE621D5BB2EA968753" }, { "b" : "FFFFB0235000", "path" : "/lib/ld-linux-aarch64.so.1", "elfType" : 3, "buildId" : "DC2D7183E0FC2CE2314B285B2344E623E3BA32EA" }, { "b" : "FFFFAFB81000", "path" : "/lib/aarch64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "15EE90CC314D70F871DA0737047EDBC7806186D1" }, { "b" : "FFFFAFB53000", "path" : "/lib/aarch64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "2B78B76D9DD313CCCB3C56DFE89FEF48C66AF7CA" }, { "b" : "FFFFAFB25000", "path" : "/lib/aarch64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "B8FE622E6CF0F29342D9D43130F7390A98F4D914" }, { "b" : "FFFFAFAAB000", "path" : "/lib/aarch64-linux-gnu/libssh.so.4", "elfType" : 3, "buildId" : "7D6CD533828EEDA574A0C622D560CB4D31ED71FF" }, { "b" : "FFFFAFA8A000", "path" : "/lib/aarch64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "4A76EB926F1B0D062FDEE5DB9DF6A8287926DC5E" }, { "b" : "FFFFAFA32000", "path" : "/lib/aarch64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B24B6DFD002CF6551A1BAFD70BEAA622F6C74812" }, { "b" : "FFFFAF9CF000", "path" : "/lib/aarch64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "3DC44F0F44BBB5F820D47C5B729B2907B47BEA6A" }, { "b" : "FFFFAF9B0000", "path" : "/lib/aarch64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "56A38255FB102798517DD7E5059EAB3EF67257BE" }, { "b" : "FFFFAF993000", "path" : "/lib/aarch64-linux-gnu/libbrotlidec.so.1", "elfType" : 3, "buildId" : "77EDB932EFC8A0C1264AC95C34E2C6D7A9E54019" }, { "b" : "FFFFAF969000", "path" : "/lib/aarch64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "B2808FA40EC84B5F676F66C40F2E5B79CC2EA410" }, { "b" : "FFFFAF7E0000", "path" : "/lib/aarch64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "F6121326AC6A81C3E4F79E0C7EF847F7189D206F" }, { "b" : "FFFFAF5EF000", "path" : "/lib/aarch64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "9F9D7D525B0BBBF14B37BB86B6C0CDBCAE29DC54" }, { "b" : "FFFFAF5A9000", "path" : "/lib/aarch64-linux-gnu/libhogweed.so.5", "elfType" : 3, "buildId" : "F04E2DD390CCF7FB5055093D7104EF998B35F7E2" }, { "b" : "FFFFAF561000", "path" : "/lib/aarch64-linux-gnu/libnettle.so.7", "elfType" : 3, "buildId" : "3BBDDDE3403BEF31550E7FF1D7C790B6740DBBA6" }, { "b" : "FFFFAF4DA000", "path" : "/lib/aarch64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "54232CDCFAAB88C52B5F0552734EA738CA98E834" }, { "b" : "FFFFAF3F2000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "5A7CFAD186F1E2CCE1AFE406CFE1ACAFF1CCFB65" }, { "b" : "FFFFAF3B5000", "path" : "/lib/aarch64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "A4AA186A505BF111FD2DA4BEFFC51627C9EBDEF9" }, { "b" : "FFFFAF3A1000", "path" : "/lib/aarch64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "6FBFF9336DAA2FCED9AE195D8570A590D79C3D5F" }, { "b" : "FFFFAF382000", "path" : "/lib/aarch64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "E0D51294A52168B81B137615C52A91DBDBEE17A5" }, { "b" : "FFFFAF357000", "path" : "/lib/aarch64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "DDA470DFAC99C4285D6A3076E1AE8FB1FF56C51E" }, { "b" : "FFFFAF309000", "path" : "/lib/aarch64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "C3433D5F07588222745E0C6895D712B386543520" }, { "b" : "FFFFAF2D8000", "path" : "/lib/aarch64-linux-gnu/libbrotlicommon.so.1", "elfType" : 3, "buildId" : "CB0C35C5F908F3F23F338BF0ECFDC3F7C9B646CE" }, { "b" : "FFFFAF18C000", "path" : "/lib/aarch64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "FD60D6D25EA33DAD6820D0B560D1B89544911339" }, { "b" : "FFFFAF167000", "path" : "/lib/aarch64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "8B9AE981258AC903E122AC93F717C29B31BDFB25" }, { "b" : "FFFFAF152000", "path" : "/lib/aarch64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "1A8415772833B993A593F6A42FFEECCCF36971D8" }, { "b" : "FFFFAF138000", "path" : "/lib/aarch64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "DB279FCB212692DF41F7BA16BAAF2D0BD9B773C3" }, { "b" : "FFFFAF09C000", "path" : "/lib/aarch64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "B2D822DC3C4EA03F11453483B503775C4D205C19" }, { "b" : "FFFFAEFF7000", "path" : "/lib/aarch64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "F7637D30B767964029BF8D3AED2D104F71B7E2D7" }, { "b" : "FFFFAEFAE000", "path" : "/lib/aarch64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "298A583F5B3035894AB4146592C65B9C4CE1A2BF" }, { "b" : "FFFFAEF88000", "path" : "/lib/aarch64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "4CDEFF83DFEF374F8BA07122F322E2A533A58C9B" }, { "b" : "FFFFAEF6F000", "path" : "/lib/aarch64-linux-gnu/libffi.so.7", "elfType" : 3, "buildId" : "095BA01481950C7652CB19EC8342F921B2EB8246" }, { "b" : "FFFFAEF36000", "path" : "/lib/aarch64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "DE842D5CDFF82E6C85AA7742FCB64BB752F7AE57" }, { "b" : "FFFFAEF17000", "path" : "/lib/aarch64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "069813AEB10E3FFE61191AC014EEBA8B17293986" }, { "b" : "FFFFAEEBB000", "path" : "/lib/aarch64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "5079CE381493D15064E9916E5880014CD456304A" }, { "b" : "FFFFAED89000", "path" : "/lib/aarch64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "DD2644987B96D32BC4CC61C6C884A416450B883E" }, { "b" : "FFFFAED40000", "path" : "/lib/aarch64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "7DD2207B47D98892369A2F56C2B39179468BEF4C" }, { "b" : "FFFFAED19000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_files.so.2", "elfType" : 3, "buildId" : "0717293A95775EEB374442A0A201259056E89D48" }, { "b" : "FFFFAED03000", "path" : "/lib/aarch64-linux-gnu/atomics/libnss_dns.so.2", "elfType" : 3, "buildId" : "1A6829FF337426F86D047C089DECF92C969533E8" } ] }}
       mongos(_ZN5mongo15printStackTraceERSo+0x44) [0xaaaae65e2094]
       mongos(+0x12511D0) [0xaaaae65e11d0]
       mongos(+0x125127C) [0xaaaae65e127c]
       linux-vdso.so.1(__kernel_rt_sigreturn+0x0) [0xffffb026578c]
       libc.so.6(gsignal+0xE0) [0xffffafbebc20]
       libc.so.6(abort+0xFC) [0xffffafbd8a14]
       mongos(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0xaaaae596e6e0]
       mongos(_ZN5mongo9transport18TransportLayerASIO12_runListenerEv+0x5FC) [0xaaaae5da5494]
       mongos(+0xA15528) [0xaaaae5da5528]
       mongos(+0x1362CC0) [0xaaaae66f2cc0]
       libpthread.so.0(+0x73F0) [0xffffafd303f0]
       libc.so.6(+0xD00DC) [0xffffafc880dc]
      -----  END BACKTRACE  ----- 

      Attachments

        Activity

          People

            chris.kelly@mongodb.com Chris Kelly
            wouter@notifica.re Wouter van der Meulen
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: