[SERVER-21358] Sudden huge increase in number of connection and server is down Created: 09/Nov/15  Updated: 29/Nov/15  Resolved: 23/Nov/15

Status: Closed
Project: Core Server
Component/s: Admin
Affects Version/s: 2.6.11
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Romain Pechayre Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello,

We had an incident today with a 3 node mongo replica set in production running MongoDB 2.6.11.

The average number of connections on the primary is roughly 180. The number of connections increased to 1800 connections and the primary became unreachable. mongotop, mongostats and mongo command were just not connecting to the server.

After several minutes and absolutely no clue about where this was coming from, I killed the primary on the server to force an election. The election worked great (as always), and the replica set is now working perfectly again with a new primary and a normal number of open connections.

We first though there was an intrusion to one of our app servers (since mongo is not accessible from outside our EC2 security group), but since the problems disappeared when we changed the primary, it looks like Mongo suddenly stopped handling closed connection events.
We use no threads on application server side, but we do open/close connections very frequently because we use resque to handle background processes (which opens a connection upon every new job).

It took us time to take the decision to step down the primary because we really thought the cause of the huge increase in the number of connections was from our app servers, but it turned out te be the right thing to do.

Is this a known behaviour in MongoDB ?



 Comments   
Comment by Romain Pechayre [ 29/Nov/15 ]

Thank you for your answer. The reason why I reported this in the first place was because the issue disappeared immediately when I killed the primary in my replica set. So this was certainly not coming from the clients connected to the replica set.

Comment by Ramon Fernandez Marina [ 23/Nov/15 ]

Thanks for your report rpechayr. High connection count is often a symptom rather than an actual problem. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server.

For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources.

Regards,
Ramón.

Generated at Thu Feb 08 03:57:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.