-
Type:
Question
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: 3.2.4
-
Component/s: Sharding
-
Labels:None
On AWS mongos seems to fail connecting to mongo config server after shard servers are added.
The setup is a test environment in aws
Config Server and the Mongos are running on two different machines in the same network ( different subnets )
Config Server # rsconf1/172.31..26.252 --> only 1 node in the replica set
Mongs service # 172.31.24.126 --> pointing to config server above
Shard1 # rs1/172.31.43.146
Shard2 # rs2/172.31.43.147
Shard # rs3/172.31.45.148
Only Shard1 add via mongos.
Mongs successfully connects to config server and then immediately fail to retrieve the list of shards
2016-04-21T21:57:05.979+0000 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to 172.31.26.252:27017
2016-04-21T21:57:06.188+0000 I SHARDING [Balancer] caught exception while doing balance: could not get updated shard list from config server due to Operation timed out
2016-04-21T21:57:06.189+0000 I SHARDING [Balancer] about to log metadata event into actionlog: { _id: "ip-172-31-24-126-2016-04-21T21:57:06.189+0000-57194cb2534cef5db40b623e", server: "ip-172-31-24-126", clientAddr: "", time: new Date(1461275826189), what: "balancer.round", ns: "", details: { executionTimeMillis: 30003, errorOccured: true, errmsg: "could not get updated shard list from config server due to Operation timed out" } }
---------------------------- $$$$$$$$$$$$$$$$$$$$$$$$ -------------------------
the operation times out on the config server , the same query passes when run directly connected to mongo config server ..
2016-04-21T21:59:08.690+0000 I COMMAND [conn29] command config.$cmd command: find { find: "shards", readConcern: { level: "majority", afterOpTime: { ts: Timestamp 1461274573000|2, t: 5 } }, maxTimeMS: 30000 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:92 locks:{} protocol:op_command 30401ms
2016-04-21T21:59:08.690+0000 I NETWORK [conn29] end connection 172.31.24.126:44130 (4 connections now open)
2016-04-21T21:59:38.084+0000 I NETWORK [initandlisten] connection accepted from 172.31.24.126:44133 #32 (5 connections now open)
----------------------------------- $$$$$$$$$$$$$$$$ -----------------------------
Note: mongo config server is running on an EBS backed storage. Even then it should take 30 seconds to return the sharded list, the same query when run on the terminal locally connected returns in fraction of a second.