[SERVER-24709] mongod crashes on mongodump run(probably due to low memory) Created: 22/Jun/16 Updated: 21/Jun/17 Resolved: 24/May/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Stability, Tools |
| Affects Version/s: | 3.2.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Or Khafi [X] | Assignee: | Kelsey Schubert |
| Resolution: | Incomplete | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Operating System: | ALL |
| Steps To Reproduce: | Run mongodump and then stress the memory of the OS. |
| Participants: |
| Description |
|
Server - AWS EC2 t2.small, CentOS Linux release 7.2.1511 (Core) When running mongodump command on the secondary node, the db crashes in 10% of the times. I stressed the memory to reproduce, and looks like it happens due to low memory (though I have enough free memory when running/during the dump). The line when dump fails: The last messages in mongod.log:
|
| Comments |
| Comment by Kelsey Schubert [ 24/May/17 ] | |
|
Hi KafKafOwn, We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide the diagnostic.data with heapProfilingEnabled=true and we will reopen the ticket. Regards, | |
| Comment by Kelsey Schubert [ 06/Apr/17 ] | |
|
Hi mango, Thanks for reporting this issue. I've examined the metrics file you've uploaded and determined that this is a separate issue. So we can continue to investigate would you please open a new ticket? If you're able to reliably reproduce this issue, would you please enable the heap profiler as described above? After encountering this issue again, would you please upload the complete archive of the diagnostic.data as well as the complete log files? Thank you for your help, | |
| Comment by Jan S. [ 04/Apr/17 ] | |
|
I also have memory problems while dumping the database. But in difference to Or Khafi I am already using 3.4 , but with the MMAPv1 engine: db version v3.4.3 While dumping the database no other database operations were running. The system has 2GB (nearly unused) swap. | |
| Comment by Kelsey Schubert [ 13/Mar/17 ] | |
|
Hi KafKafOwn, Unfortunately, we have not yet been able to conclusively determine the root cause of this behavior from the diagnostic.data you provided. However, we have made significant improvements to MongoDB since 3.2.6 that may correct this issue. If this is still a concern for you, would you please upgrade to MongoDB 3.2.12? MongoDB 3.2.12 also includes a new parameter, which will allow us to collect additional diagnostic.data to better understand what is happening. If you are encounter this issue on MongoDB 3.2.12, would you please reproduce with the following parameter set?
Then, please upload the complete log files and the diagnostic.data to this new secure upload portal. Thank you again for your help, | |
| Comment by Kelsey Schubert [ 08/Dec/16 ] | |
|
Hi Paolo, I'd recommend upgrading to MongoDB 3.4 to take advantage of Kind regards, | |
| Comment by Paolo [X] [ 08/Dec/16 ] | |
|
Hi guys, any update on this? I'm having the same issue on Digital Ocean instance with SSD and 1GB RAM, and running: db version v3.2.10 | |
| Comment by Or Khafi [X] [ 22/Jun/16 ] | |
|
Uploaded. | |
| Comment by Kelsey Schubert [ 22/Jun/16 ] | |
|
Certainly, I've created this secure portal for you to use. Kind regards, | |
| Comment by Or Khafi [X] [ 22/Jun/16 ] | |
|
Hey Thomas, I know you are not collecting any secret data in your diagnostics, but can you please give me a private link to upload the data? Thanks, | |
| Comment by Kelsey Schubert [ 22/Jun/16 ] | |
|
Hi KafKafOwn, Thanks for the additional information. So we can continue to investigate, would you please archive (tar or zip) the $dbpath/diagnostic.data directory and attach it to this ticket? Also, would please attach the output of top so we can see what other processes are running on your machine? Thank you, | |
| Comment by Or Khafi [X] [ 22/Jun/16 ] | |
|
Out of memory: Kill process 16402 (mongod) score 765 or sacrifice child A few more details to help understand linux oom killer decision: | |
| Comment by Ramon Fernandez Marina [ 22/Jun/16 ] | |
|
KafKafOwn, can you check your system logs for the following string?
|