[SERVER-34884] Replication executor creates connection storm which should be limited Created: 08/May/18  Updated: 15/Nov/21  Resolved: 24/Nov/19

Status: Closed
Project: Core Server
Component/s: Networking, Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Zhang Youdong Assignee: Dmitry Agranat
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-34684 --maxConns parameter limit both clien... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

In my procuct environment, member of shard create a lot of connections to peer nodes,  which cause the peer reach the maxIncomingConnections config(8000), causing the mongodb service unavailable.

As described in  SERVER-29429  which limit the maxConnecting from mongos to mongod,  we should also limit the maxConnecting from mongod to mongod to prevent connection storm.

 

Below is part of mongod's log 

 

// code placeholder
 
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
... 
more similar lines ...
 

 



 Comments   
Comment by Dmitry Agranat [ 06/Nov/19 ]

Hi zyd_com,

This request looks similar to SERVER-34684, please let us know if this is not the case.

Thanks,
Dima

Generated at Thu Feb 08 04:38:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.