[SERVER-19620] Mongod killed because of OOM during index build Bottom up Created: 28/Jul/15 Updated: 28/Jul/15 Resolved: 28/Jul/15 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Index Maintenance, WiredTiger |
| Affects Version/s: | 3.0.4 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Endijs Lisovskis [X] | Assignee: | Ramon Fernandez Marina |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | I did not try to reproduce it (as that is live system), thus I'm not 100% sure it's easy to reproduce. But you can try to build compound index on 3 fields (1 string averaging 20 chars, but can be up to 100, 1 string ~10chars, date) on 8GB system which uses 4GB during normal day to day situation, on collection with 40M documents. |
||||||||
| Participants: | |||||||||
| Description |
|
During index building Buttom up stage mongod was killed because of out of memory. Some stats: Should mongod really be killed if there is not enough memory for bottom up? I doubt it. If not enough memory, just kill index building process. As side effect of this - when mongod was started and synced I checked state of collection on which index was built. To my surprise its reported that it now has particular index. However as mongod was killed, info was not passed to secondary of that time (which is now master). Now i have new primary without index, secondary with index and i have no clue what to do with it. Is it correct, is it corrupt. How to get rid of it if its on secondary and not on master. |
| Comments |
| Comment by Ramon Fernandez Marina [ 28/Jul/15 ] |
|
3.0.5 is available for download at https://www.mongodb.org/downloads To remove the index on the secondary you can follow the procedure described here to add an index, and drop the index instead of creating it. I'm going to close this ticket as a duplicate of Regards, |
| Comment by Endijs Lisovskis [X] [ 28/Jul/15 ] |
|
OK. Looking forward for 3.0.5 release and then will try again. Any suggestion on how to handle "hanging" index on secondary? What is best practice in this situation? As I wrote in description i have now ended up with index on secondary and no index on primary. I guess best scenario is to get rid of index in secondary, to have identical collection setup. Is that so? If it is - how best to achieve this? Do i need to step down from current primary and once roles will be changed, drop index or there is different approach that should be used? |
| Comment by Ramon Fernandez Marina [ 28/Jul/15 ] |
|
Hi Endijs, looks like you're running into |