upgrade mongodb from 3.0.2 to 3.2 SyncClusterConnection failed

XMLWordPrintableJSON

    • Type: Question
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 3.2.0
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi,

      I tried to upgrade on CentOS 6.5 VMs from MongoDB 3.0.2 to 3.2 with rpm packages. On the test system (same env, same config) everything works smooth. But on production I got the following messages from my mongos instances:

      2016-02-10T09:09:05.481+0000 I SHARDING [mongosMain] Updating config server connection string to: hostname1:27019,hostname2:27019,hostname4:27019
2016-02-10T09:09:05.483+0000 D SHARDING [mongosMain]  config string : hostname1:27019,hostname2:27019,hostname4:27019
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-TaskExecutor-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-4-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-5-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-7-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-6-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.485+0000 D NETWORK  [mongosMain] creating new connection to:hostname1:27019
2016-02-10T09:09:05.485+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.485+0000 D NETWORK  [mongosMain] connected to server hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.486+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.486+0000 D SHARDING [mongosMain] calling onCreate auth for hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.486+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.487+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPVRDb0d1RFFrUlB2K0VjM1ljUHdsdVQxdWpQN1ZtYjE1
2016-02-10T09:09:05.488+0000 D NETWORK  [mongosMain] sasl client input: cj1UQ29HdURRa1JQditFYzNZY1B3bHVUMXVqUDdWbWIxNVNVRnFURW9zVzFrVUxqK1RuazhmdEdrK0llU3NqdGYyLHM9aDRGbjQ2SVk3bmwzeFFOMmEzVzVuQT09LGk9MTAwMDA=
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9VENvR3VEUWtSUHYrRWMzWWNQd2x1VDF1alA3Vm1iMTVTVUZxVEVvc1cxa1VMaitUbms4ZnRHaytJZVNzanRmMixwPXByODZ0QmJVc0FKbGU4RzAvQjBOWDQ0ZWg4WT0=
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client input: dj1vb0c4bk9ma0FRNktHTEFXZWJwWmM3NGFZQ2s9
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.501+0000 D NETWORK  [mongosMain] creating new connection to:hostname2:27019
2016-02-10T09:09:05.503+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.503+0000 D NETWORK  [mongosMain] connected to server hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.504+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.504+0000 D SHARDING [mongosMain] calling onCreate auth for hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.504+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.505+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPTl3SENVdlRMSFdQcXJCSk9kcURvSWpaZTZ0RzdxRCtV
2016-02-10T09:09:05.506+0000 D NETWORK  [mongosMain] sasl client input: cj05d0hDVXZUTEhXUHFyQkpPZHFEb0lqWmU2dEc3cUQrVTNsSU8vMko1aWlnZ0YvWHZiTU1FUDNJYWgvcWFHb2VCLHM9ciswK1hVRUNPWjJiVTJoL2xJbHNUdz09LGk9MTAwMDA=
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9OXdIQ1V2VExIV1BxckJKT2RxRG9JalplNnRHN3FEK1UzbElPLzJKNWlpZ2dGL1h2Yk1NRVAzSWFoL3FhR29lQixwPUd5N1VWM3pSa2V6OXlDRzhUVVZMcVhMdTZIWT0=
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client input: dj1CdDQycENkc21hR3dBeUhDZDk5T0k3Y1JJU2M9
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.520+0000 D NETWORK  [mongosMain] creating new connection to:hostname4:27019
2016-02-10T09:09:05.522+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.524+0000 D NETWORK  [mongosMain] connected to server hostname4:27019 (10.140.208.114)
2016-02-10T09:09:05.525+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.525+0000 D SHARDING [mongosMain] calling onCreate auth for hostname4:27019 (10.140.208.114)
2016-02-10T09:09:05.525+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.526+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPTQvZElIOXEvYUhISTlRYVJvUVNKSzB6MkFTRzczc2Zw
2016-02-10T09:09:05.528+0000 D NETWORK  [mongosMain] sasl client input: cj00L2RJSDlxL2FISEk5UWFSb1FTSkswejJBU0c3M3NmcFJ2cDUrUlErR3gxZjQ2OWdQaVNFUHlsNThiekNubGpILHM9dkxha1dHWEJJejNUYktNdnRpUTBQUT09LGk9MTAwMDA=
2016-02-10T09:09:05.540+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9NC9kSUg5cS9hSEhJOVFhUm9RU0pLMHoyQVNHNzNzZnBSdnA1K1JRK0d4MWY0NjlnUGlTRVB5bDU4YnpDbmxqSCxwPWJzZHhZY1FIeS8raUZ2MkpDWjVuSUFRbFpUWT0=
2016-02-10T09:09:05.541+0000 D NETWORK  [mongosMain] sasl client input: dj1kbmVZclM0cHdQdnhlaUZVNEFCV3hzSkQ3Y289
2016-02-10T09:09:05.541+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.544+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname1:27019]
2016-02-10T09:09:05.544+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.544+0000 D NETWORK  [mongosMain] connected to server hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.545+0000 I NETWORK  [mongosMain] SyncClusterConnection connect fail to: hostname1:27019 :: caused by :: Surprised to discover that hostname1:27019 does not believe it is a config server
2016-02-10T09:09:05.545+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname2:27019]
2016-02-10T09:09:05.547+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.547+0000 D NETWORK  [mongosMain] connected to server hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.548+0000 I NETWORK  [mongosMain] SyncClusterConnection connect fail to: hostname2:27019 :: caused by :: Surprised to discover that hostname2:27019 does not believe it is a config server
2016-02-10T09:09:05.548+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname4:27019]
2016-02-10T09:09:05.549+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG

      Attached you find my config files. In total I have a setup of a sharded cluster with 2 replica sets. 3 config server 6 mongos instances and 3 rs members for each replica set.

      After a rollback to 3.0.2 everything is working fine again. As stated already on my test environment shared cluster with 2 repl sets, I had no issue...

      Thank you for your help!

      BR,

      Alex

        1. mongoc.conf
          0.4 kB
        2. mongod.conf
          0.4 kB
        3. mongos.conf
          0.4 kB

            Assignee:
            Kaloian Manassiev
            Reporter:
            Alexander Zima
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: