Details
-
Bug
-
Resolution: Done
-
Major - P3
-
None
-
None
-
None
-
None
-
Config servers - 2.4.1, Shard routers are 2.4.6
-
ALL
Description
We had some temporary network issues for a few hours. After a few hours the shard routers fail to connect to the config server. We had to provision new shard routers which appear to work fine. I see a lot of these errors
"9001 socket exception [RECV_ERROR] server [162.243.34.127:27019]
Wed Oct 23 04:34:55.101 [conn4] call failed to: SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019 no data"
Wed Oct 23 03:51:07.624 [mongosMain] connection accepted from 24.16.96.170:61958 #2 (2 connections now open)
Wed Oct 23 03:51:07.800 [conn2] authenticate db: admin
Wed Oct 23 03:51:17.377 [conn2] creating WriteBackListener for: SG-edSpringProdSSDShard-1359.servers.mongodirector.com:27017 serverID: 5267474390aa292bd1d1aedc
Wed Oct 23 03:51:17.377 [conn2] creating WriteBackListener for: SG-edSpringProdSSDShard-1360.servers.mongodirector.com:27017 serverID: 5267474390aa292bd1d1aedc
Wed Oct 23 03:51:17.384 [conn2] creating WriteBackListener for: SG-edSpringProdSSDShard-1362.servers.mongodirector.com:27017 serverID: 5267474390aa292bd1d1aedc
Wed Oct 23 03:51:17.384 [conn2] creating WriteBackListener for: SG-edSpringProdSSDShard-1363.servers.mongodirector.com:27017 serverID: 5267474390aa292bd1d1aedc
Wed Oct 23 03:51:17.385 [conn2] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019]
Wed Oct 23 03:51:17.394 [conn2] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019]
Wed Oct 23 03:51:17.404 [conn2] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1355.servers.mongodirector.com:27019]
Wed Oct 23 03:53:37.396 [mongosMain] connection accepted from 24.16.96.170:62192 #3 (3 connections now open)
Wed Oct 23 03:53:37.585 [conn3] end connection 24.16.96.170:62192 (2 connections now open)
Wed Oct 23 03:54:51.748 [mongosMain] connection accepted from 24.16.96.170:62305 #4 (3 connections now open)
Wed Oct 23 03:54:51.956 [conn4] authenticate db: admin
Wed Oct 23 03:54:54.607 [conn4] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019]
Wed Oct 23 03:54:54.633 [conn4] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019]
Wed Oct 23 03:54:54.642 [conn4] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1355.servers.mongodirector.com:27019]
Wed Oct 23 04:09:24.390 [Balancer] Socket recv() errno:104 Connection reset by peer 162.243.34.126:27019
Wed Oct 23 04:09:24.390 [Balancer] SocketException: remote: 162.243.34.126:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.126:27019]
Wed Oct 23 04:09:24.391 [Balancer] DBClientCursor::init call() failed
Wed Oct 23 04:09:24.391 [Balancer] Detected bad connection created at 1382500164387060 microSec, clearing pool for SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019
Wed Oct 23 04:09:24.393 [Balancer] scoped connection to SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019,SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019,SG-edSpringP
rodSSDShard-1355.servers.mongodirector.com:27019 not being returned to the pool
Wed Oct 23 04:09:24.393 [Balancer] caught exception while doing balance: DBClientBase::findN: transport error: SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019 ns: admin.$cmd que
ry:
Wed Oct 23 04:09:30.393 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019]
Wed Oct 23 04:09:30.402 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019]
Wed Oct 23 04:09:30.428 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1355.servers.mongodirector.com:27019]
Wed Oct 23 04:11:17.889 [conn2] Socket recv() errno:104 Connection reset by peer 162.243.34.126:27019
Wed Oct 23 04:11:17.890 [conn2] SocketException: remote: 162.243.34.126:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.126:27019]
Wed Oct 23 04:11:17.890 [conn2] call failed to: SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019 no data
Wed Oct 23 04:12:40.329 [mongosMain] connection accepted from 127.0.0.1:53957 #5 (4 connections now open)
Wed Oct 23 04:12:40.341 [conn5] authenticate db: admin
Wed Oct 23 04:12:40.357 [conn5] end connection 127.0.0.1:53957 (3 connections now open)
Wed Oct 23 04:12:45.242 [mongosMain] connection accepted from 127.0.0.1:53958 #6 (4 connections now open)
Wed Oct 23 04:12:45.247 [conn6] authenticate db: admin
Wed Oct 23 04:12:45.259 [conn6] end connection 127.0.0.1:53958 (3 connections now open)
Wed Oct 23 04:12:50.985 [mongosMain] connection accepted from 127.0.0.1:53960 #7 (4 connections now open)
Wed Oct 23 04:12:50.987 [conn7] authenticate db: admin
Wed Oct 23 04:12:50.997 [conn7] end connection 127.0.0.1:53960 (3 connections now open)
Wed Oct 23 04:12:55.804 [mongosMain] connection accepted from 127.0.0.1:53961 #8 (4 connections now open)
Wed Oct 23 04:12:55.808 [conn8] authenticate db: admin
Wed Oct 23 04:12:55.822 [conn8] end connection 127.0.0.1:53961 (3 connections now open)
Wed Oct 23 04:14:55.097 [conn4] Socket recv() errno:104 Connection reset by peer 162.243.34.126:27019
Wed Oct 23 04:14:55.097 [conn4] SocketException: remote: 162.243.34.126:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.126:27019]
Wed Oct 23 04:14:55.097 [conn4] call failed to: SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019 no data
Wed Oct 23 04:29:30.463 [Balancer] Socket recv() errno:104 Connection reset by peer 162.243.34.126:27019
Wed Oct 23 04:29:30.463 [Balancer] SocketException: remote: 162.243.34.126:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.126:27019]
Wed Oct 23 04:29:30.463 [Balancer] DBClientCursor::init call() failed
Wed Oct 23 04:29:30.463 [Balancer] Detected bad connection created at 1382501370458915 microSec, clearing pool for SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019
Wed Oct 23 04:29:30.464 [Balancer] scoped connection to SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019,SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019,SG-edSpringP
rodSSDShard-1355.servers.mongodirector.com:27019 not being returned to the pool
Wed Oct 23 04:29:30.464 [Balancer] caught exception while doing balance: DBClientBase::findN: transport error: SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019 ns: admin.$cmd que
ry:
Wed Oct 23 04:29:36.465 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1353.servers.mongodirector.com:27019]
Wed Oct 23 04:29:36.475 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019]
Wed Oct 23 04:29:36.478 [Balancer] SyncClusterConnection connecting to [SG-edSpringProdSSDShard-1355.servers.mongodirector.com:27019]
Wed Oct 23 04:31:17.893 [conn2] Socket recv() errno:104 Connection reset by peer 162.243.34.127:27019
Wed Oct 23 04:31:17.893 [conn2] SocketException: remote: 162.243.34.127:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.127:27019]
Wed Oct 23 04:31:17.893 [conn2] call failed to: SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019 no data
Wed Oct 23 04:34:55.100 [conn4] Socket recv() errno:104 Connection reset by peer 162.243.34.127:27019
Wed Oct 23 04:34:55.101 [conn4] SocketException: remote: 162.243.34.127:27019 error: 9001 socket exception [RECV_ERROR] server [162.243.34.127:27019]
Wed Oct 23 04:34:55.101 [conn4] call failed to: SG-edSpringProdSSDShard-1354.servers.mongodirector.com:27019 no data
Wed Oct 23 04:49:13.309 [mongosMain] dbexit: received signal 15 rc:0 received signal 15
Attachments
Issue Links
- is duplicated by
-
SERVER-11321 db.locks.find() on the config server causes assertion
-
- Closed
-