[SERVER-20027] MongoDB 3.0.3 Segfault after jscmd error Created: 19/Aug/15 Updated: 29/Jan/16 Resolved: 25/Aug/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | JavaScript, Stability |
| Affects Version/s: | 3.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Christopher Neill | Assignee: | Sam Kleinman (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Participants: |
| Description |
SunOS rs0db1.us-east-2 5.11 joyent_20150617T074149Z i86pc i386 i86pc Solaris |
| Comments |
| Comment by Ramon Fernandez Marina [ 29/Jan/16 ] |
|
For the record, |
| Comment by Nick Wisse [X] [ 26/Aug/15 ] |
|
Thanks Sam On Wed, Aug 26, 2015 at 5:29 PM, Sam Kleinman (JIRA) <jira@mongodb.org> – { name : "Nick Wisse", title : "Corporate Account Executive", phone : "+1 (866) 237-8815 Ext. 7143", email : "nick.wisse@MongoDB.com <nick.wisse@10gen.com>[image: Look up in Salesforce]", location : "New York, NY", twitter : ["@NickMongoDB <https://twitter.com/NickMongoDB>", "@M <https://twitter.com/mongodb>ongoDBinc"], facebook : ["MongoDB <https://www.facebook.com/mongodb>"] }Hot New News:
|
| Comment by Christopher Neill [ 26/Aug/15 ] |
|
Sorry to bother you, but none of those conditions apply, we are not using a newer compiler and certainly not Linux: Information for http://10.117.69.27/2015Q1/mongodb-3.0.3.tgz: |
| Comment by Sam Kleinman (Inactive) [ 25/Aug/15 ] |
|
Sorry for not getting back to you sooner. We see errors around AdjustAmountOfExternalAllocatedMemory in situations with newer compilers (e.g. GCC 5.2) that are incompatible with the version of V8 embedded in MongoDB, as well as deployments with some SELinux/grsecurity policies affecting memory use. You could experiment with your security policy settings and and the standard build if you are not using this build already. Starting in the 3.1.7 development release for the 3.2.0 stable release, MongoDB no longer embeds the V8 JavaScript engine, which should resolve this issue. Also, you may have some luck addressing questions like these to the mongodb-user forum for help with additional troubleshooting and support. I'm going to go ahead and close this issue, but feel free to reopen if you have more information, or are able to reproduce without security policies that restrict memory use and the standard build. |
| Comment by Christopher Neill [ 25/Aug/15 ] |
|
Requesting a status update. |
| Comment by Christopher Neill [ 25/Aug/15 ] |
|
From Joyent Support: Max Bruning (Joyent Support) Hi Chris, From running, in the global zone: $ dtrace -w -n 'proc:::signal-handle/pid == $target && args[0] == 11 && args[1]/ {print(*args[1]); ustack(); stop();}' -p 70914 CPU ID FUNCTION:NAME } } struct __file = { int __fd = 0 long __band = 0 } struct __prof = { short __syscall = 0 } After this, I ran $ gcore 70914 $ prun 70914 to get mongod to continue from its stopped state. Using mdb on the core file (and from the stack backtrace shown above via DTrace, it is dying at mongod`_ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20 _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20/ai The %rax register should be a pointer to thread specific data, but it is null. mongod is dying because of using a null pointer. _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20::dis call +0x189ef0 <_ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE> _ZN2v88internal6Thread14GetThreadLocalENS1_15LocalStorageKeyE::dis So pthread_getspecific is returning a NULL for some piece of thread specific data, and _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl+0x20 tries to indirect through that causing a segv. _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl is: > ::dem _ZN2v82V837AdjustAmountOfExternalAllocatedMemoryEl Searching for "mongodb AdjustAmountOfExternalAllocatedMemory segmentation violation" shows a few hits, but not quite the same. |
| Comment by Christopher Neill [ 19/Aug/15 ] |
|
FYI - per Joyent's recommendation, we doubled the size of the replica set members to go from 4GB to 8GB of RAM. Same thing: [root@logger1 us-east-2 /var/log/mongodb]# grep crit mongodb-remote.log , {"b":"400000","o":"10B55A3"}, {"b":"400000","o":"10B5CFE"}, {"b":"FFFFFD7FFF050000","o":"1179F6"}, {"b":"FFFFFD7FFF050000","o":"10A66B"}, {"b":"400000","o":"1148FE0"}, {"b":"400000","o":"1003454"}, {"b":"400000","o":"10038D2"}, {"b":"400000","o":"FFA4B1"}, {"b":"400000","o":"FFA4E1"}, {"b":"400000","o":"6807D9"}, {"b":"400000","o":"FF334C"}, {"b":"400000","o":"FF3DE1"}, {"b":"400000","o":"8AE6AF"}, {"b":"400000","o":"8B2CB9"}, {"b":"400000","o":"94538C"}, {"b":"400000","o":"9464DF"}, {"b":"400000","o":"94733E"}, {"b":"400000","o":"BCA785"},{"b":"400000","o" |