[SERVER-28001] Mongodb Crashed with the Got signal: 6 (Aborted) Created: 14/Feb/17 Updated: 31/May/17 Resolved: 21/Mar/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Admin |
| Affects Version/s: | 3.0.4 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Abhishek Manocha | Assignee: | Mark Agarunov |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
We have seen repeatedly the mongdb crashing running on Ubuntu Most recent stacktrace is here:
On the EC2 (we run this in AWS): |
| Comments |
| Comment by Mark Agarunov [ 21/Mar/17 ] | |||
|
Hello akmanocha We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket. Thanks, | |||
| Comment by Mark Agarunov [ 28/Feb/17 ] | |||
|
Hello akmanocha, My apologies, I overlooked the fact that you are using Mongodb version 3.0.4, which does not generate the diagnostic data since that was implemented starting with version 3.2. My recommendation would be to upgrade to a newer version, as there have been many fixes implemented since 3.0.4 and the behavior you're seeing may no longer be an issue in a more recent release. Alternatively, if you are unable to upgrade, please run the following commands and provide the ss.log and iostat.log files that are created:
Please leave this running until the issue happens again so that there is a complete log. Thanks, | |||
| Comment by Abhishek Manocha [ 28/Feb/17 ] | |||
|
What is diagnostic.data directory? I am not aware of it. What's the default for the same? | |||
| Comment by Mark Agarunov [ 21/Feb/17 ] | |||
|
Hello akmanocha, Thank you for the additional information and my apologies for the delay. We are still investigating this issue, however I suspect this may be related to Thanks, | |||
| Comment by Abhishek Manocha [ 21/Feb/17 ] | |||
|
Hey no update on this? Should I clone this / make it to a bug? | |||
| Comment by Abhishek Manocha [ 15/Feb/17 ] | |||
|
Hi Mark, Thanks for the input. How can I know that this is the number of open files issue? How do you get to that I mean if you can share the reasoning. My ulimit for mongouser (this specific mongo process owner) is 64000 hard and soft In the arbitar logs.txt attached. The very first line Can you pleas help me what this line means? Why secondary wants to become primary suddenly? And finally both go down at around 11:17 (9 mins later) with Got signal: 6 (Aborted) It can be open files issue, but root cause is not clear to me. Thanks | |||
| Comment by Mark Agarunov [ 14/Feb/17 ] | |||
|
Hello akmanocha, Thank you for the report. Looking over the provided output, this appears to be due to the number of open files being greater than what is allowed by your ulimits configuration. Please try increasing this limit as described in the documentation to see if this resolves the issue you are seeing. If this behavior is still present after increasing the open file limits, please provide the full logs from mongod and we will continue investigating this behavior. Thanks, |