[SERVER-21358] Sudden huge increase in number of connection and server is down Created: 09/Nov/15 Updated: 29/Nov/15 Resolved: 23/Nov/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 2.6.11 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Romain Pechayre | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
|
Hello, We had an incident today with a 3 node mongo replica set in production running MongoDB 2.6.11. The average number of connections on the primary is roughly 180. The number of connections increased to 1800 connections and the primary became unreachable. mongotop, mongostats and mongo command were just not connecting to the server. After several minutes and absolutely no clue about where this was coming from, I killed the primary on the server to force an election. The election worked great (as always), and the replica set is now working perfectly again with a new primary and a normal number of open connections. We first though there was an intrusion to one of our app servers (since mongo is not accessible from outside our EC2 security group), but since the problems disappeared when we changed the primary, it looks like Mongo suddenly stopped handling closed connection events. It took us time to take the decision to step down the primary because we really thought the cause of the huge increase in the number of connections was from our app servers, but it turned out te be the right thing to do. Is this a known behaviour in MongoDB ? |
| Comments |
| Comment by Romain Pechayre [ 29/Nov/15 ] |
|
Thank you for your answer. The reason why I reported this in the first place was because the issue disappeared immediately when I killed the primary in my replica set. So this was certainly not coming from the clients connected to the replica set. |
| Comment by Ramon Fernandez Marina [ 23/Nov/15 ] |
|
Thanks for your report rpechayr. High connection count is often a symptom rather than an actual problem. Please note that the SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your question will reach a larger audience. A question like this involving more discussion would be best posted on the mongodb-user group. See also our Technical Support page for additional support resources. Regards, |