[SERVER-22280] Sockets/connections created and left hanging Created: 22/Jan/16  Updated: 26/Jan/16  Resolved: 26/Jan/16

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: 3.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Roberto Leibman Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

AWS VM Environment, 3 mongo replicaset across regions with IPSEC tunnel


Issue Links:
Duplicate
duplicates NODE-645 Sockets/connections created and left ... Closed
Related
Operating System: Linux
Steps To Reproduce:

The environment
----------------------
I have an environment split among two Amazon Region (OR and VA).
OR: 1 NAT, 1 SVC, 2 Mongo
VA: 1 NAT, 1 SVC, 1 Mongo (with priority = 0)
Both regions are connected by an ipsec tunnel managed by the 2 NAT machines, such that all traffic between regions goes through this tunnel,
The SVC machines each run 3 different services, each of these node services keeps a connection pool for their duration. The connection is made with the following url:

var url = 'mongodb://10.56.4.54:27017,10.72.3.114:27017,10.72.4.96:27017/mydatabase?maxPoolSize=100&replicaSet=replica-set-name'

It all seems to be running fine, until the IPSEC tunnel goes down. "mongostat --discover" shows roughly 600 connections spread accross all 3 mongo servers.

When the tunnel goes down (even if momentarily), the services on the OR region function correctly, the services on the VA region function for a little bit until their local Mongo realizes it's disconnected, at which point they just hang.

The problem is when the tunnel goes back up. At that point hundreds of connections ACCROSS the tunnel are made, almost as if each bad connection in each connection pool was causing a whole other connection pool. Of course, if my ulimit is set too low mongo stops accepting connections. Mongo sees these open connections as idle (dbCurrentOp(true) shows them as active=false). Using lsof (or netstat) I can see that the extra connections are TCP sockets in ESTABLISHED state accross the tunnel (from the mongo box they show as being opened to the ipsec box, NOT to their originating box on the other side of the tunnel). If I kill the services in the SVC boxes the connections do go away.

Participants:

 Description   

It's a fairly hard thing to reproduce, so bear with me... also, I'm not sure if the problem is in the node driver or in mongodb itself, so I'm going to try to log it in both places



 Comments   
Comment by Roberto Leibman [ 26/Jan/16 ]

Thomas,
Is there anything in my description that leads you to think it is not a bug in either the driver or the server?

It seems to me that when the server shows thousands of connections open and the client only requested a pool of 100 there's a bug somewhere.

Yes, I agree that the user group may give me some advice on how to work around the bug (and thanks for the suggestion, I just posted the question), but ultimately it's a matter of thousands of connections (with their sockets) open when I only asked for 100. You want to blame the driver, by all means, one way or the other, it's still a bug.

Comment by Kelsey Schubert [ 26/Jan/16 ]

Hi roberto@leibman.net,

I would reccomend posting this question on the mongodb-users group. The user group provides MongoDB-related support and may be able to help you narrow down this issue. If after further investigation there is evidence of a bug in the MongoDB server, please comment on this ticket or a open a new ticket.

Thank you,
Thomas

Generated at Thu Feb 08 03:59:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.