[SERVER-15920] Segmentation fault on mongod processes and mongo shell Created: 03/Nov/14 Updated: 11/Aug/15 Resolved: 14/Nov/14 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Shell, Stability |
| Affects Version/s: | 2.6.5 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Anthony Brodard | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 1 |
| Labels: | crash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
OS : Gentoo Linux |
||
| Attachments: |
|
||||||||||||
| Issue Links: |
|
||||||||||||
| Operating System: | Linux | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Since we upgraded from 2.4.10 to 2.6.5, some nodes crash randomly by segmentation fault (mongod.txt). On these nodes, the mongo shell won't start and do a segfault on startup (mongo-shell.txt). Other nodes have the same configuration, and mongo runs perfectly. |
| Comments |
| Comment by Darko Luketic [ 20/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Thanks Ramon, I'll do so as soon as I find the time to.
I'll open a new ticket in the next few days. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 19/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
dluketic, the information in the link you sent shows a crash in V8, so the principal suspect is some configured limitation that's preventing V8 from operating properly. Please open a new ticket posting the stack trace, the output of 'paxctl -v /usr/bin/mongo', and any other information you think may be useful for us to track the issue down. Thanks, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Darko Luketic [ 19/Jan/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I'm not using a GRSEC enabled kernel and it's still happening. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 14/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Glad to hear you found the cause of the issue anthony@1000mercis.com, and thanks to ultrabug for the assist. Closing this ticket as a duplicate of | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Anthony Brodard [ 12/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi, This bug happens on GRSEC enabled kernels. This is related to this : https://jira.mongodb.org/browse/SERVER-12991 Thanks to Ramon and Ultrabug for your help, this issue can be marked as resolved. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ultrabug [ 09/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi For reference, we're not the only ones impacted. This is the related Gentoo bug providing other details if needed : https://bugs.gentoo.org/show_bug.cgi?id=526114 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Anthony Brodard [ 05/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi Ramon, We have built mongodb with the debugging option :
Now, the mongo shell display this error :
Tell me if you want the mongo and/or mongod binaries, or any other information. Anthony | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ultrabug [ 03/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
If it can help a bit, I'm attaching a strace of the mongo client segfault FYI we tried to reduce any system wise library dependency and rely only on bundled libs shipped with the sources of mongodb. The resulting scons command used to build mongo is : scons -j7 --variant-dir=build --cc=x86_64-pc-linux-gnu-gcc --cxx=x86_64-pc-linux-gnu-g++ --disable-warnings-as-errors --usev8 --ssl all And it has the same effect (the strace comes from a mongo shell client built using these scons options). Cheers | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 03/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I misread the logs earlier on, my apologies for that. Further examination shows that the crash is triggered when running mapReduce and that all crashes are happening inside V8 (v8::internal::OS::Allocate), so my recommendation would be to revisit your build procedure first. This could be a bug in V8, or some artifact triggered by the custom build. It could also be a bad interaction with the specific system libraries used for this build. You may also try running mongod/mongo under valgrind to see if this provides additional information. Another avenue to explore is building with debugging information; if the crash reproduces then would you be able to upload the binaries? If the answer is yes please let me know and I'll send you upload information. I'd also recommend you post this question in the mongodb-dev google group, where your question will reach a larger audience of MongoDB developers. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Anthony Brodard [ 03/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Yes, the authentication schema have been updated to the 2.6 format. We have followed all the upgrade procedure. Primary-rs1 :
Primary-rs2:
I can't send all the logs, but I can send specific parts you need after anonymization. Tell me what you need. I'm not sure that it's a mongo data or configuration related, because a segfault occurs on the mongo shell binarie too on the mentionned servers. The shell run correctly on the other servers. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 03/Nov/14 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
anthony@1000mercis.com, a quick glance at the mongod stack trace hints an issue with authentication. Did you upgrade your authentication schema to the 2.6 format? Did you follow the upgrade recommendations and checklists? It would be helpful if you could upload full logs of one of the mongod nodes from startup until the segfault. |