[SERVER-19754] mongod aborts on memory allocation failure after upgrade from 2.6.6 to 3.0.5 Created: 04/Aug/15 Updated: 07/Apr/23 Resolved: 25/Aug/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Internal Code, WiredTiger |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Deepak Shivamurthy | Assignee: | Ramon Fernandez Marina |
| Resolution: | Done | Votes: | 0 |
| Labels: | RF, WTmem | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Operating System: | Linux |
| Participants: |
| Description |
|
Hi Team, Recently I have updated mongodb from 2.6.6 to 3.0.5 (also moved from default storage to wiredtiger). I have not experience any issues till now, there was a lot of improvement in performance. But today one of the replica set was crashed with below errors (taken from log) when doing write operations (which creates temporary collections dynamically). I am not sure whether its a issue with wiredtiger.
|
| Comments |
| Comment by Ramon Fernandez Marina [ 24/Nov/15 ] | |||||||||||||||||||||||||||||||
|
ych.tiger@gmail.com, please take a look at this comment on If after increasing the limits for the number of open files the issue persists please open a new ticket. Thanks, | |||||||||||||||||||||||||||||||
| Comment by YANG Chenghu [ 24/Nov/15 ] | |||||||||||||||||||||||||||||||
|
3.0.6 also crashed
| |||||||||||||||||||||||||||||||
| Comment by Deepak Shivamurthy [ 12/Aug/15 ] | |||||||||||||||||||||||||||||||
|
I have upgraded to 3.0.6-rc0 (from 3.0.5), mongo services are not crashing now. I think now it seems to be resolved. I will let you know if I face any issues. Thanks for all your help, Ramon!! | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 12/Aug/15 ] | |||||||||||||||||||||||||||||||
|
deepak.shivamurth, I forgot to add that a binary replacement should be sufficient, but please take a look at the full documentation on upgrading to the latest revision of MongoDB, as it contains detailed information about sharded clusters, etc. Thanks, | |||||||||||||||||||||||||||||||
| Comment by Deepak Shivamurthy [ 12/Aug/15 ] | |||||||||||||||||||||||||||||||
|
thanks Ramon, I will have a look at it. I will update you if I find any issues. | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 12/Aug/15 ] | |||||||||||||||||||||||||||||||
|
deepak.shivamurth, this is to let you know that we've published the 3.0.6-rc0 release candidate containing a fix for If you're able to reproduce the problem, would you be able to test with 3.0.6-rc0 and let us know if the problem persists? Thanks, | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 07/Aug/15 ] | |||||||||||||||||||||||||||||||
|
deepak.shivamurth, if you're able to reproduce this problem (or if you've seen it more than once and expect it again) it would be very useful if you could collect server statistics that can help us find the root cause of the problem. To collect these statistics you'd need to run the following from a shell:
These commands will create two files, iostat.log and ss.log, containing statistics collected at 10 second intervals. This interval will allow you to leave these commands running for a while until the problem reproduces. If while collecting this data the problem doesn't reproduce it may still be useful for us to see the data: if it shows a pattern of increased memory usage that may allow us to pinpoint what part of the system may be responsible for the behavior you're seeing. In addition to the iostat.log and ss.log files we'd need the server logs producing during statistics collection time. You can upload files to this ticket at any time, or if you perfer you can use this private upload portal. Would it be possible for you to collect data about your server as described above? Thanks, | |||||||||||||||||||||||||||||||
| Comment by Deepak Shivamurthy [ 05/Aug/15 ] | |||||||||||||||||||||||||||||||
|
its a mongodb dedicated server and only one mongod instance is running. | |||||||||||||||||||||||||||||||
| Comment by Deepak Shivamurthy [ 05/Aug/15 ] | |||||||||||||||||||||||||||||||
|
Hi Ramon, Please find below vmstat output and also attached screenshot for CPU fyi, currently our mongod services are running with below configuration; RAM - 32
Regards, | |||||||||||||||||||||||||||||||
| Comment by Ramon Fernandez Marina [ 04/Aug/15 ] | |||||||||||||||||||||||||||||||
|
deepak.shivamurth, looks like mongod run out of memory. Can you please provide details on the memory configuration and usage on this node? Has the amount of memory available to mongod been limited in some way? Thanks, |