[SERVER-18758] Segmentation fault on heavy unacknowledged bulk inserts with WiredTiger Created: 31/May/15 Updated: 04/Aug/15 Resolved: 03/Aug/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | WiredTiger |
| Affects Version/s: | 3.0.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Yan | Assignee: | Sam Kleinman (Inactive) |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
40 cores, 120g ram |
||
| Attachments: |
|
||||||||
| Issue Links: |
|
||||||||
| Operating System: | Linux | ||||||||
| Steps To Reproduce: | Create database, begin to emit heavy unacknowledged bulk writes and updates. |
||||||||
| Participants: | |||||||||
| Description |
|
Got segmentation fault while heavy multithreaded unacknowledged bulk writes to wiredTiger database. Acknowledges writes goes clear.
|
| Comments |
| Comment by Sam Kleinman (Inactive) [ 03/Aug/15 ] | |||||||||||||||||||||||||||||||
|
I'm going to go ahead and close this ticket, because you mentioned the issue seems to have resolved itself and there haven't been any updates in a while. If you encounter this isue, feel free to update this ticket and we can continue the exploration of this issue. Regards, | |||||||||||||||||||||||||||||||
| Comment by Sam Kleinman (Inactive) [ 02/Jul/15 ] | |||||||||||||||||||||||||||||||
|
Thanks your feedback. I'm glad to hear that things are working better for you right now. I'm going to set this ticket back to the waiting state. If you run into this report, we look forward to getting the output with the additional debug symbol data. Regards, | |||||||||||||||||||||||||||||||
| Comment by Yan [ 02/Jul/15 ] | |||||||||||||||||||||||||||||||
|
Ok, i will switch, but after 3.0.4 things are running smooth for now. | |||||||||||||||||||||||||||||||
| Comment by Sam Kleinman (Inactive) [ 02/Jul/15 ] | |||||||||||||||||||||||||||||||
|
Could you attempt to reproduce this issue using binaries with debug symbols? You can find these binaries for your linux release at https://www.mongodb.org/dl/linux. Thanks so much. Regards, | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 01/Jul/15 ] | |||||||||||||||||||||||||||||||
|
Hi lightket, apologies for the long delay, and thanks for uploading the binary you're using. Unfortunately this binary does not include debugging information, but it should help us track the debugging information on our end. Further attempts to reproduce have been unsuccessful, so we continue to investigate this issue – thank you for your patience. Regards, | |||||||||||||||||||||||||||||||
| Comment by Yan [ 11/Jun/15 ] | |||||||||||||||||||||||||||||||
|
Ramon, attaching binary file I'm sorry, but i cannot give you our code at this point. And it will be irrelevant anyway without source database (we developing migration tool). Lets just hope someone else comes up with this bug ang will be able to provide it. | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 09/Jun/15 ] | |||||||||||||||||||||||||||||||
|
lightket, I tried reproducing this on my end using the code below, with 100 threads doing both unordered and ordered bulk inserts of different sizes, but unfortunately I wasn't able to trigger the segfault you're observing. Therefore I'd like to emphasize the need for more details, and preferably a reproducer so we can find the root cause of this bug. Thanks in advance for your help,
| |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 08/Jun/15 ] | |||||||||||||||||||||||||||||||
|
Hi lightket, unfortunately the link you provided does not allow me to retrieve the exact binary you're using, which I need to decipher the stack trace. But knowing the version of RedHat you're using should help – what's the output of running
? Note that this may not be sufficient to find the root cause of this issue, and that ideally we'd like to reproduce this ourselves. Can you please provide more information on how you're triggering this? Is this restoring a large dataset or are you generating data? How large are these bulk inserts? Are you using YCSB or perhaps a program of your own to do these bulk inserts? If the latter it would be of great help if you could share that with us as requested above. Thanks, | |||||||||||||||||||||||||||||||
| Comment by Yan [ 08/Jun/15 ] | |||||||||||||||||||||||||||||||
|
Ramon, what will be our next action? | |||||||||||||||||||||||||||||||
| Comment by Yan [ 08/Jun/15 ] | |||||||||||||||||||||||||||||||
|
Fault again:
| |||||||||||||||||||||||||||||||
| Comment by Yan [ 31/May/15 ] | |||||||||||||||||||||||||||||||
|
Ramon, i have downloaded mongodb for linux from
| |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 31/May/15 ] | |||||||||||||||||||||||||||||||
|
lightket, can you also please provide more details about the binary you're using? Did you build it yourself? If the answer is no, where did you download it from? Are you using the OS' package manager? If you could send us the output of db.serverVersion() as well it would help us diagnose your problem. Thanks, | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 31/May/15 ] | |||||||||||||||||||||||||||||||
|
lightket, can you please share with us the code you're using to trigger this behavior? You can upload it privately (only accessible to MongoDB staff) and securely via scp as follows:
where <filename> is the file or directory to upload. When prompted for a password just press enter. Thanks, |