mongod killed by oom-killer when drop lots of collections.

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • ALL
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hello,

      We have a single replica set with 3 nodes running 3.4.14 with WiredTiger.

      After dropping lots of collections frequently, we found

      • the file sizeStorer.wt is growing up slowly
      • the mem of mongod grow slowly

      finally mongod was killed by oom-killer, and sizeStorer.wt grow up to 300MB

      Test script

      db = db.getSiblingDB("testdb");
      var count = 0;
      do{
        col =  db.getCollection("sizeStore");
        col.insert({"name":"aa"});
        doc=col.findOne();
        printjson(doc)
        db.sizeStore.drop();
        sleep(1);
        count ++;
        print("create table count " +count);
      }while(true)
      
      

      * sizeStorer.wt size:(create/drop table/ count 3130104)

      -rw------- 1 Ruby Ruby        49 Feb 25 02:59 WiredTiger
      -rw------- 1 Ruby Ruby        21 Feb 25 02:59 WiredTiger.lock
      -rw------- 1 Ruby Ruby      1011 Feb 28 07:20 WiredTiger.turtle
      -rw------- 1 Ruby Ruby    548864 Feb 28 07:20 WiredTiger.wt
      -rw------- 1 Ruby Ruby      4096 Feb 26 00:55 WiredTigerLAS.wt
      -rw------- 1 Ruby Ruby     36864 Feb 27 21:25 _mdb_catalog.wt
      drwx------ 4 Ruby Ruby      4096 Feb 25 02:59 admin
      drwx------ 2 Ruby Ruby      4096 Feb 28 07:20 diagnostic.data
      drwx------ 2 Ruby Ruby      4096 Feb 27 19:36 journal
      drwx------ 4 Ruby Ruby      4096 Feb 25 02:59 local
      -rw------- 1 Ruby Ruby         5 Feb 26 00:55 mongod.lock
      -rw------- 1 root root        81 Feb 26 01:58 monitor.sh
      -rw------- 1 Ruby Ruby 303792128 Feb 28 07:20 sizeStorer.wt
      -rw------- 1 Ruby Ruby        95 Feb 25 02:59 storage.bson
      drwx------ 4 Ruby Ruby      4096 Feb 25 10:28 testdb
      

      * OOM:

      Feb 27 21:24:49 localhost kernel: [239239.871767] Out of memory: Kill process 13316 (mongod) score 863 or sacrifice child
      Feb 27 21:24:49 localhost kernel: [239239.871801] Killed process 13316 (mongod) total-vm:6096592kB, anon-rss:3286352kB, file-rss:0kB
      Feb 27 21:24:49 localhost kernel: [239239.873442] systemd-journal: page allocation failure: order:0, mode:0x201da
      Feb 27 21:24:49 localhost kernel: [239239.873445] CPU: 1 PID: 451 Comm: systemd-journal Tainted: G           OE  ----V-------   3.10.0-327.62.59.83.h128.x86_64 #1
      Feb 27 21:24:49 localhost kernel: [239239.873446] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
      Feb 27 21:24:49 localhost kernel: [239239.873447] Call Trace:
      

      The above scenario is just a simplification of our business.
      In actual business, we create and delete almost 30000 collections per day.
      Approximately every 10 days, our business also meet database crash by OOM-killer.

      So I would like to know:
      How often does Mongodb support DDL operations?

      Thanks for your help. 

              Assignee:
              Unassigned
              Reporter:
              Mike sun
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: