[SERVER-19620] Mongod killed because of OOM during index build Bottom up Created: 28/Jul/15  Updated: 28/Jul/15  Resolved: 28/Jul/15

Status: Closed
Project: Core Server
Component/s: Index Maintenance, WiredTiger
Affects Version/s: 3.0.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Endijs Lisovskis [X] Assignee: Ramon Fernandez Marina
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-18829 Cache usage exceeds configured maximu... Closed
Operating System: ALL
Steps To Reproduce:

I did not try to reproduce it (as that is live system), thus I'm not 100% sure it's easy to reproduce. But you can try to build compound index on 3 fields (1 string averaging 20 chars, but can be up to 100, 1 string ~10chars, date) on 8GB system which uses 4GB during normal day to day situation, on collection with 40M documents.

Participants:

 Description   

During index building Buttom up stage mongod was killed because of out of memory. Some stats:
4CPU cores, 8GB RAM (usually half is used)
MongoDB 3.0.4
Replica set with 2 members + arbiter
WiredTiger
Collection had 37302912 docs in it.
Compound index was build on 3 fields (string, string, date).
Index building stage finished fine. However during Bottom up mongod was killed. Last entry in mongod log:
2015-07-28T04:07:37.001-0400 I - [conn33853] Index: (2/3) BTree Bottom Up Progress: 23171800/37302912 62%
There were no stack traces.
However kernel log indicates that mongod was killed because of out of memory:
Out of memory: Kill process 1670 (mongod) score 921 or sacrifice child
Killed process 1670, UID 0, (mongod) total-vm:15740096kB, anon-rss:7576600kB, file-rss:300kB

Should mongod really be killed if there is not enough memory for bottom up? I doubt it. If not enough memory, just kill index building process.

As side effect of this - when mongod was started and synced I checked state of collection on which index was built. To my surprise its reported that it now has particular index. However as mongod was killed, info was not passed to secondary of that time (which is now master). Now i have new primary without index, secondary with index and i have no clue what to do with it. Is it correct, is it corrupt. How to get rid of it if its on secondary and not on master.



 Comments   
Comment by Ramon Fernandez Marina [ 28/Jul/15 ]

3.0.5 is available for download at https://www.mongodb.org/downloads

To remove the index on the secondary you can follow the procedure described here to add an index, and drop the index instead of creating it.

I'm going to close this ticket as a duplicate of SERVER-18829. If the problem persists with 3.0.5 please let us know.

Regards,
Ramón.

Comment by Endijs Lisovskis [X] [ 28/Jul/15 ]

OK. Looking forward for 3.0.5 release and then will try again. Any suggestion on how to handle "hanging" index on secondary? What is best practice in this situation? As I wrote in description i have now ended up with index on secondary and no index on primary. I guess best scenario is to get rid of index in secondary, to have identical collection setup. Is that so? If it is - how best to achieve this? Do i need to step down from current primary and once roles will be changed, drop index or there is different approach that should be used?

Comment by Ramon Fernandez Marina [ 28/Jul/15 ]

Hi Endijs, looks like you're running into SERVER-18829, sorry about that. This bug has been fixed in 3.0.5, which will be released later today.

Generated at Thu Feb 08 03:51:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.