[SERVER-17569] Got signal: 11 (Segmentation fault). Created: 12/Mar/15 Updated: 16/Nov/21 Resolved: 17/Mar/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Marcos Fernándex | Assignee: | Bruce Lucas (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
cat mongodb.log
|
| Comments |
| Comment by Bruce Lucas (Inactive) [ 17/Mar/15 ] | |
|
Hi Marcos, It turns out that we also recently reproduced this problem in an internal test, reported in Bruce | |
| Comment by Marcos Fernándex [ 16/Mar/15 ] | |
|
@Asya Kamsky oh thanks, wasnt aware of it, very useful, Im gonna try it now | |
| Comment by Asya Kamsky [ 16/Mar/15 ] | |
|
Note that you can specify different WiredTiger options to the collection when you create it: http://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options | |
| Comment by Bruce Lucas (Inactive) [ 16/Mar/15 ] | |
|
Thanks for the detailed information Marcos, and my apologies for missing your comment about OOM killer. I agree that neither segmentation fault nor the failure of --repair are the desired outcome, and furthermore I would expect that we should be able to survive OOM killer without this kind of corruption. We're looking into it. | |
| Comment by Marcos Fernándex [ 16/Mar/15 ] | |
|
As I said before (second comment), this broken state was produced by kernel OOM killing mongodb, in fact it taked me several minutes to figure out why mongodb was dissapearing and saw kern.log. WiredTiger is an expensive improvement if you use mongodb gridFS extensively, but unfortunately i'm not aware of how to configure wiredTiger/MMAPv1 per database. I have no logs of instances before, the long short history is:
My conclusions:
Thanks | |
| Comment by Bruce Lucas (Inactive) [ 16/Mar/15 ] | |
|
Hi Marcos, It looks like there's a problem in one of the journal files. We're investigating to see what we can determine about the cause. It will be helpful to have some information about the prior history of instance before you started observing this problem -
Thanks, | |
| Comment by Bruce Lucas (Inactive) [ 12/Mar/15 ] | |
|
Thanks Marcos. I will take a look and get back to you with my findings. | |
| Comment by Marcos Fernándex [ 12/Mar/15 ] | |
|
Upload complete. Thanks | |
| Comment by Ramon Fernandez Marina [ 12/Mar/15 ] | |
|
sombra2eternity, just hit enter when prompted for a password. | |
| Comment by Bruce Lucas (Inactive) [ 12/Mar/15 ] | |
|
Thanks Marcos. You can just hit enter when it asks for password. | |
| Comment by Marcos Fernándex [ 12/Mar/15 ] | |
|
scp -P 722 -r <filename> | |
| Comment by Marcos Fernándex [ 12/Mar/15 ] | |
|
ok, fair enought, im zipping it right now. Next I'll do: scp -P 722 -r <filename> right? | |
| Comment by Bruce Lucas (Inactive) [ 12/Mar/15 ] | |
|
Hi Marcos, We as a general policy don't access users' systems directly. In our experience, we have found that a 9 GB upload may take some time, but it is possible. Would you be willing to give it a try? We would very much like to look at the data to understand this issue. Thanks, | |
| Comment by J Rassi [ 12/Mar/15 ] | |
|
Assigning to bruce.lucas@10gen.com. | |
| Comment by Marcos Fernándex [ 12/Mar/15 ] | |
|
1. With a clean path it works, in fact it was working for a few days now correctly, then the daemon got killed by the kernel OOM (I think wiredTiger have a problem with memory too) and all got wrong. After I wrote this issue I just changed all binaries to the 3.0.1 release candidate to see if it could start, same error. Just give me an email address and I will send you ssh access to this server. Mongo is currently running with a clean /home/mongodb, old data is moved to /home/mongodb1 in order to check your above questions. | |
| Comment by J Rassi [ 12/Mar/15 ] | |
|
Hi, Thanks for the report, and sorry to hear that you're encountering this problem. We'll need to gather additional information in order to diagnose the issue:
If you are not comfortable posting your data or your logs publicly (or if either exceeds 100MB in total size), please upload them to our private drop box. Only MongoDB staff has access to data uploaded this way. You can do so as follows from the command line (when you are prompted for a password, please just press "enter"):
~ Jason Rassi | |
| Comment by Marcos Fernándex [ 12/Mar/15 ] | |
|
root@ns320462:/var/log/mongodb# mongod --version root@ns320462:/var/log/mongodb# uname -a ubuntu 14.10 |