Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-19620

Mongod killed because of OOM during index build Bottom up

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.0.4
    • Component/s: Index Maintenance, WiredTiger
    • None
    • ALL
    • Hide

      I did not try to reproduce it (as that is live system), thus I'm not 100% sure it's easy to reproduce. But you can try to build compound index on 3 fields (1 string averaging 20 chars, but can be up to 100, 1 string ~10chars, date) on 8GB system which uses 4GB during normal day to day situation, on collection with 40M documents.

      Show
      I did not try to reproduce it (as that is live system), thus I'm not 100% sure it's easy to reproduce. But you can try to build compound index on 3 fields (1 string averaging 20 chars, but can be up to 100, 1 string ~10chars, date) on 8GB system which uses 4GB during normal day to day situation, on collection with 40M documents.

      During index building Buttom up stage mongod was killed because of out of memory. Some stats:
      4CPU cores, 8GB RAM (usually half is used)
      MongoDB 3.0.4
      Replica set with 2 members + arbiter
      WiredTiger
      Collection had 37302912 docs in it.
      Compound index was build on 3 fields (string, string, date).
      Index building stage finished fine. However during Bottom up mongod was killed. Last entry in mongod log:
      2015-07-28T04:07:37.001-0400 I - [conn33853] Index: (2/3) BTree Bottom Up Progress: 23171800/37302912 62%
      There were no stack traces.
      However kernel log indicates that mongod was killed because of out of memory:
      Out of memory: Kill process 1670 (mongod) score 921 or sacrifice child
      Killed process 1670, UID 0, (mongod) total-vm:15740096kB, anon-rss:7576600kB, file-rss:300kB

      Should mongod really be killed if there is not enough memory for bottom up? I doubt it. If not enough memory, just kill index building process.

      As side effect of this - when mongod was started and synced I checked state of collection on which index was built. To my surprise its reported that it now has particular index. However as mongod was killed, info was not passed to secondary of that time (which is now master). Now i have new primary without index, secondary with index and i have no clue what to do with it. Is it correct, is it corrupt. How to get rid of it if its on secondary and not on master.

            Assignee:
            ramon.fernandez@mongodb.com Ramon Fernandez Marina
            Reporter:
            Endijs Endijs Lisovskis [X]
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: