[SERVER-24803] Mongo 3.0.12 WiredTiger Node Crashed After Massive Rise In Connections Created: 26/Jun/16 Updated: 20/Sep/16 Resolved: 27/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Concurrency, Replication, WiredTiger |
| Affects Version/s: | 3.0.12 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Avraham Kalvo | Assignee: | Unassigned |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Participants: |
| Description |
|
Dear Mongo Analyst, I'm issuing this issue with your team as we couldn't find a root cause that might explain how come last Friday saw a brand new upgraded replica set node The above has happened starting 12:00PM on Friday the 24th, as can be seen in the log file enclosed hereby. As per the application side, they had no significant processes taking place at the time as far as they claim, neither no changes in configuration (connection pool settings etc.) Enclosed please find is also the replica set configuration and current status. Note that the hosts generating the high rise in connections are the application hosts rather than the replica set nodes. |
| Comments |
| Comment by Avraham Kalvo [ 20/Sep/16 ] | ||||||||||||||||||||||||||||||
|
Thanks Ramon, | ||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 27/Jun/16 ] | ||||||||||||||||||||||||||||||
|
After examining the information you sent, what I see is that starting around 12pm on Jun 24 (as you pointed out) there's a huge spike on connections to this node:
Here's a list of how many connections came from which IP addresses:
Seeing how none of these IP addresses correspond to other nodes in your replica set, my only conclusion is that they came from application nodes. For some reason decided to connect to this node between 12:00pm and 12:35pm, and this node became overwhelmed. Since I haven't been able to find evidence of a bug in the server I'm going to close this ticket, since we keep the SERVER project for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag, where your questions will reach a larger audience. See also our Technical Support page for additional support resources. Regards, | ||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 27/Jun/16 ] | ||||||||||||||||||||||||||||||
|
Please update with progress status, Thanks, | ||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 27/Jun/16 ] | ||||||||||||||||||||||||||||||
|
The error accepted on app side is as follows: | ||||||||||||||||||||||||||||||
| Comment by Avraham Kalvo [ 26/Jun/16 ] | ||||||||||||||||||||||||||||||
|
Thanks Ramón, Enclosed please find. Thanks, | ||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 26/Jun/16 ] | ||||||||||||||||||||||||||||||
|
Sorry you had issues with MongoDB avrahamk, will wait for the logs to be uploaded to investigate if these issues were caused by a bug or by lack of machine resources. Regards, |