[SERVER-20204] Segmentation fault during index build on 3.0 secondary Created: 30/Aug/15 Updated: 13/Oct/15 Resolved: 02/Sep/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.5 |
| Fix Version/s: | 3.0.7, 3.1.8 |
| Type: | Bug | Priority: | Critical - P2 |
| Reporter: | Maksim Naumov | Assignee: | Geert Bosch |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | Linux | ||||||||||||
| Backport Completed: | |||||||||||||
| Sprint: | Quint 9 09/18/15 | ||||||||||||
| Participants: | |||||||||||||
| Description |
|
Hello
|
| Comments |
| Comment by Ramon Fernandez Marina [ 02/Sep/15 ] | ||||||||||||||||||||||||||
|
Thank you for reporting the problem fromyukki, and sorry you run into it. Feel free to post updates on this ticket after using 3.0.7-pre-, and to open new tickets if you run into further issues. | ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 02/Sep/15 ] | ||||||||||||||||||||||||||
|
Hello ramon.fernandez thank you for your support. I will try pre-build and then i will upgrade to 3.0.7. Thank you one more time. | ||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 02/Sep/15 ] | ||||||||||||||||||||||||||
|
fromyukki, we've identified the source of the problem and committed a fix. The fix will be in upcoming releases 3.0.7 and 3.1.8, to be released in the coming weeks. Unfortunately there's currently no workaround for this issue. You may try reducing the WiredTiger cache size, but depending on your data distribution you may run into If this issue is currently blocking you one thing you can try is using a 3.0.7-pre- build including the fix for this bug; the build will appear here soon. Regards, | ||||||||||||||||||||||||||
| Comment by Githook User [ 02/Sep/15 ] | ||||||||||||||||||||||||||
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}Message: (cherry picked from commit 0a6c20bdc7128d1f13e967a7cc6219b1dfc38b6b) | ||||||||||||||||||||||||||
| Comment by Githook User [ 01/Sep/15 ] | ||||||||||||||||||||||||||
|
Author: {u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}Message: | ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 01/Sep/15 ] | ||||||||||||||||||||||||||
|
Hello ramon.fernandez, I am sorry, but dmesg has nothing for August 30 | ||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 01/Sep/15 ] | ||||||||||||||||||||||||||
|
fromyukki, the stack trace seems to point to a failure to allocate memory. Can you please send us the output of
from this node? If there was a malloc(3) failure it may show up there. Depending on your syslog configuration, this information may have also been written to /var/log/kern.log or /var/log/syslog. If the output of dmesg -T doesn't show anything I'll ask you to send some of the log files I mentioned above (I'll create an upload portal so you can send this privately and securely). Thanks, | ||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
fromyukki, the reason you're seeing Note also that setting a high value for the WiredTiger cache may starve the OS buffer cache, which may have a negative impact on performance. That being said, this does not explain the segfault of course – we're still investigating the behavior you reported in this ticket. | ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
Yes sure. Mongo is running on virtual machine: 30Gb of memory, 5Tb of a disk and just 4 CPU. Segfault happened on the secondary node while it was syncing with the primary (very new node). After restart all the data was dropped and sync process was started from the very beginning. Here is the config file
In the log file nothing more (some megabytes of replication logs) and everything that I have, I posted as this issue description. | ||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
Thanks fromyukki – yes, I wanted to be sure there were no packaging-related issues as well, and the dpkg output clears that out. And yes, on to the segfault, for which we need some more information:
Thanks, | ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
Oh, yes my fault ... anyway, seg fault
| ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
Hi ramon.fernandez , I also saw this backtrace, but the fact is that I use the latest version from your repository. Maybe you compiled it with wrong version or repository has wrong one. | ||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 31/Aug/15 ] | ||||||||||||||||||||||||||
|
Hi fromyukki, I adjusted the "Affects Version/s" field as per the version in the stack trace:
Can you send the output of
? Thanks, | ||||||||||||||||||||||||||
| Comment by Maksim Naumov [ 30/Aug/15 ] | ||||||||||||||||||||||||||
|
It is written "Affects Version/s: 3.0.5", but I got this seg fault on 3.0.6. It says that the last version of mongo is installed But So strange |