Loading...

Type: Bug
Resolution: Duplicate
Priority: Major - P3
Fix Version/s: None
Affects Version/s: None
Component/s: Networking, Replication
Labels:
None

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Confidence Status:
None
Work Order:
0

Aha! Reference:
None
Tracking Level:
None
Risk Status:
None
Exec Notes:
None
Goal Name:
None
Goal Link:
None

In my procuct environment, member of shard create a lot of connections to peer nodes, which cause the peer reach the maxIncomingConnections config(8000), causing the mongodb service unavailable.

As described in ~~SERVER-29429~~ which limit the maxConnecting from mongos to mongod, we should also limit the maxConnecting from mongod to mongod to prevent connection storm.

Below is part of mongod's log

// code placeholder

2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.264+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.265+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.308+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
2018-05-08T02:19:03.309+0800 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 11.192.114.180:3033
... 
more similar lines ...

is duplicated by

SERVER-34684 --maxConns parameter limit both client and replica connections

Closed

Details

Description

Attachments

Issue Links

Activity

People

Dates