[SERVER-22280] Sockets/connections created and left hanging Created: 22/Jan/16 Updated: 26/Jan/16 Resolved: 26/Jan/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | 3.2.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Roberto Leibman | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
AWS VM Environment, 3 mongo replicaset across regions with IPSEC tunnel |
||
| Issue Links: |
|
||||||||||||
| Operating System: | Linux | ||||||||||||
| Steps To Reproduce: | The environment var url = 'mongodb://10.56.4.54:27017,10.72.3.114:27017,10.72.4.96:27017/mydatabase?maxPoolSize=100&replicaSet=replica-set-name' It all seems to be running fine, until the IPSEC tunnel goes down. "mongostat --discover" shows roughly 600 connections spread accross all 3 mongo servers. When the tunnel goes down (even if momentarily), the services on the OR region function correctly, the services on the VA region function for a little bit until their local Mongo realizes it's disconnected, at which point they just hang. The problem is when the tunnel goes back up. At that point hundreds of connections ACCROSS the tunnel are made, almost as if each bad connection in each connection pool was causing a whole other connection pool. Of course, if my ulimit is set too low mongo stops accepting connections. Mongo sees these open connections as idle (dbCurrentOp(true) shows them as active=false). Using lsof (or netstat) I can see that the extra connections are TCP sockets in ESTABLISHED state accross the tunnel (from the mongo box they show as being opened to the ipsec box, NOT to their originating box on the other side of the tunnel). If I kill the services in the SVC boxes the connections do go away. |
||||||||||||
| Participants: | |||||||||||||
| Description |
|
It's a fairly hard thing to reproduce, so bear with me... also, I'm not sure if the problem is in the node driver or in mongodb itself, so I'm going to try to log it in both places |
| Comments |
| Comment by Roberto Leibman [ 26/Jan/16 ] |
|
Thomas, It seems to me that when the server shows thousands of connections open and the client only requested a pool of 100 there's a bug somewhere. Yes, I agree that the user group may give me some advice on how to work around the bug (and thanks for the suggestion, I just posted the question), but ultimately it's a matter of thousands of connections (with their sockets) open when I only asked for 100. You want to blame the driver, by all means, one way or the other, it's still a bug. |
| Comment by Kelsey Schubert [ 26/Jan/16 ] |
|
I would reccomend posting this question on the mongodb-users group. The user group provides MongoDB-related support and may be able to help you narrow down this issue. If after further investigation there is evidence of a bug in the MongoDB server, please comment on this ticket or a open a new ticket. Thank you, |