[SERVER-22549] upgrade mongodb from 3.0.2 to 3.2 SyncClusterConnection failed Created: 10/Feb/16  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Alexander Zima Assignee: Kaloian Manassiev
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File mongoc.conf     File mongod.conf     File mongos.conf    
Participants:

 Description   

Hi,

I tried to upgrade on CentOS 6.5 VMs from MongoDB 3.0.2 to 3.2 with rpm packages. On the test system (same env, same config) everything works smooth. But on production I got the following messages from my mongos instances:

2016-02-10T09:09:05.481+0000 I SHARDING [mongosMain] Updating config server connection string to: hostname1:27019,hostname2:27019,hostname4:27019
2016-02-10T09:09:05.483+0000 D SHARDING [mongosMain]  config string : hostname1:27019,hostname2:27019,hostname4:27019
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-TaskExecutor-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-ShardRegistry-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-0-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-1-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-2-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-3-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-4-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-5-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-7-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.484+0000 D ASIO     [NetworkInterfaceASIO-TaskExecutorPool-6-0] The NetworkInterfaceASIO worker thread is spinning up
2016-02-10T09:09:05.485+0000 D NETWORK  [mongosMain] creating new connection to:hostname1:27019
2016-02-10T09:09:05.485+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.485+0000 D NETWORK  [mongosMain] connected to server hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.486+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.486+0000 D SHARDING [mongosMain] calling onCreate auth for hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.486+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.487+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPVRDb0d1RFFrUlB2K0VjM1ljUHdsdVQxdWpQN1ZtYjE1
2016-02-10T09:09:05.488+0000 D NETWORK  [mongosMain] sasl client input: cj1UQ29HdURRa1JQditFYzNZY1B3bHVUMXVqUDdWbWIxNVNVRnFURW9zVzFrVUxqK1RuazhmdEdrK0llU3NqdGYyLHM9aDRGbjQ2SVk3bmwzeFFOMmEzVzVuQT09LGk9MTAwMDA=
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9VENvR3VEUWtSUHYrRWMzWWNQd2x1VDF1alA3Vm1iMTVTVUZxVEVvc1cxa1VMaitUbms4ZnRHaytJZVNzanRmMixwPXByODZ0QmJVc0FKbGU4RzAvQjBOWDQ0ZWg4WT0=
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client input: dj1vb0c4bk9ma0FRNktHTEFXZWJwWmM3NGFZQ2s9
2016-02-10T09:09:05.500+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.501+0000 D NETWORK  [mongosMain] creating new connection to:hostname2:27019
2016-02-10T09:09:05.503+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.503+0000 D NETWORK  [mongosMain] connected to server hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.504+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.504+0000 D SHARDING [mongosMain] calling onCreate auth for hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.504+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.505+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPTl3SENVdlRMSFdQcXJCSk9kcURvSWpaZTZ0RzdxRCtV
2016-02-10T09:09:05.506+0000 D NETWORK  [mongosMain] sasl client input: cj05d0hDVXZUTEhXUHFyQkpPZHFEb0lqWmU2dEc3cUQrVTNsSU8vMko1aWlnZ0YvWHZiTU1FUDNJYWgvcWFHb2VCLHM9ciswK1hVRUNPWjJiVTJoL2xJbHNUdz09LGk9MTAwMDA=
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9OXdIQ1V2VExIV1BxckJKT2RxRG9JalplNnRHN3FEK1UzbElPLzJKNWlpZ2dGL1h2Yk1NRVAzSWFoL3FhR29lQixwPUd5N1VWM3pSa2V6OXlDRzhUVVZMcVhMdTZIWT0=
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client input: dj1CdDQycENkc21hR3dBeUhDZDk5T0k3Y1JJU2M9
2016-02-10T09:09:05.519+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.520+0000 D NETWORK  [mongosMain] creating new connection to:hostname4:27019
2016-02-10T09:09:05.522+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.524+0000 D NETWORK  [mongosMain] connected to server hostname4:27019 (10.140.208.114)
2016-02-10T09:09:05.525+0000 D NETWORK  [mongosMain] connected connection!
2016-02-10T09:09:05.525+0000 D SHARDING [mongosMain] calling onCreate auth for hostname4:27019 (10.140.208.114)
2016-02-10T09:09:05.525+0000 D NETWORK  [mongosMain] sasl client input: 
2016-02-10T09:09:05.526+0000 D NETWORK  [mongosMain] sasl client output: biwsbj1fX3N5c3RlbSxyPTQvZElIOXEvYUhISTlRYVJvUVNKSzB6MkFTRzczc2Zw
2016-02-10T09:09:05.528+0000 D NETWORK  [mongosMain] sasl client input: cj00L2RJSDlxL2FISEk5UWFSb1FTSkswejJBU0c3M3NmcFJ2cDUrUlErR3gxZjQ2OWdQaVNFUHlsNThiekNubGpILHM9dkxha1dHWEJJejNUYktNdnRpUTBQUT09LGk9MTAwMDA=
2016-02-10T09:09:05.540+0000 D NETWORK  [mongosMain] sasl client output: Yz1iaXdzLHI9NC9kSUg5cS9hSEhJOVFhUm9RU0pLMHoyQVNHNzNzZnBSdnA1K1JRK0d4MWY0NjlnUGlTRVB5bDU4YnpDbmxqSCxwPWJzZHhZY1FIeS8raUZ2MkpDWjVuSUFRbFpUWT0=
2016-02-10T09:09:05.541+0000 D NETWORK  [mongosMain] sasl client input: dj1kbmVZclM0cHdQdnhlaUZVNEFCV3hzSkQ3Y289
2016-02-10T09:09:05.541+0000 D NETWORK  [mongosMain] sasl client output: 
2016-02-10T09:09:05.544+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname1:27019]
2016-02-10T09:09:05.544+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.544+0000 D NETWORK  [mongosMain] connected to server hostname1:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.545+0000 I NETWORK  [mongosMain] SyncClusterConnection connect fail to: hostname1:27019 :: caused by :: Surprised to discover that hostname1:27019 does not believe it is a config server
2016-02-10T09:09:05.545+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname2:27019]
2016-02-10T09:09:05.547+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG
2016-02-10T09:09:05.547+0000 D NETWORK  [mongosMain] connected to server hostname2:27019 (xxx.xxx.xxx.xxx)
2016-02-10T09:09:05.548+0000 I NETWORK  [mongosMain] SyncClusterConnection connect fail to: hostname2:27019 :: caused by :: Surprised to discover that hostname2:27019 does not believe it is a config server
2016-02-10T09:09:05.548+0000 I NETWORK  [mongosMain] SyncClusterConnection connecting to [hostname4:27019]
2016-02-10T09:09:05.549+0000 D COMMAND  [ConnectBG] BackgroundJob starting: ConnectBG

Attached you find my config files. In total I have a setup of a sharded cluster with 2 replica sets. 3 config server 6 mongos instances and 3 rs members for each replica set.

After a rollback to 3.0.2 everything is working fine again. As stated already on my test environment shared cluster with 2 repl sets, I had no issue...

Thank you for your help!

BR,

Alex



 Comments   
Comment by Alexander Zima [ 10/Feb/16 ]

worked, thank's a lot!!! so easy

BR,

Alex

Comment by Alexander Zima [ 10/Feb/16 ]

Hi Kal,

good point. On test I have it there. Didn't try until now, but I think that will solve it! As soon as I have a time slot again I will tell you the result, but I am pretty sure that's it.

Thank you very much!

BR,

Alex

Comment by Kaloian Manassiev [ 10/Feb/16 ]

Hi alexander.zima@bwinparty.com,

The following error message indicates that your config server is not running with the --configSvr option and as such is not reporting itself as sharding config server to mongos:

2016-02-10T09:09:05.548+0000 I NETWORK  [mongosMain] SyncClusterConnection connect fail to: hostname2:27019 :: caused by :: Surprised to discover that hostname2:27019 does not believe it is a config server

Looking at your config file mongoc.conf it looks like you are missing the --configSvr option. Can you please add it and let us know if this fixes your problem? This is the entry you need to add to your mongoc.conf:

sharding:
   clusterRole: configsvr

Best regards,
-Kal.

Generated at Thu Feb 08 04:00:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.