[SERVER-46477] mongod killed by oom-killer when drop lots of collections. Created: 28/Feb/20  Updated: 28/Feb/20  Resolved: 28/Feb/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mike sun Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Participants:

 Description   

Hello,

We have a single replica set with 3 nodes running 3.4.14 with WiredTiger.

After dropping lots of collections frequently, we found

  • the file sizeStorer.wt is growing up slowly
  • the mem of mongod grow slowly

finally mongod was killed by oom-killer, and sizeStorer.wt grow up to 300MB

Test script

db = db.getSiblingDB("testdb");
var count = 0;
do{
  col =  db.getCollection("sizeStore");
  col.insert({"name":"aa"});
  doc=col.findOne();
  printjson(doc)
  db.sizeStore.drop();
  sleep(1);
  count ++;
  print("create table count " +count);
}while(true)

* sizeStorer.wt size:(create/drop table/ count 3130104)

-rw------- 1 Ruby Ruby        49 Feb 25 02:59 WiredTiger
-rw------- 1 Ruby Ruby        21 Feb 25 02:59 WiredTiger.lock
-rw------- 1 Ruby Ruby      1011 Feb 28 07:20 WiredTiger.turtle
-rw------- 1 Ruby Ruby    548864 Feb 28 07:20 WiredTiger.wt
-rw------- 1 Ruby Ruby      4096 Feb 26 00:55 WiredTigerLAS.wt
-rw------- 1 Ruby Ruby     36864 Feb 27 21:25 _mdb_catalog.wt
drwx------ 4 Ruby Ruby      4096 Feb 25 02:59 admin
drwx------ 2 Ruby Ruby      4096 Feb 28 07:20 diagnostic.data
drwx------ 2 Ruby Ruby      4096 Feb 27 19:36 journal
drwx------ 4 Ruby Ruby      4096 Feb 25 02:59 local
-rw------- 1 Ruby Ruby         5 Feb 26 00:55 mongod.lock
-rw------- 1 root root        81 Feb 26 01:58 monitor.sh
-rw------- 1 Ruby Ruby 303792128 Feb 28 07:20 sizeStorer.wt
-rw------- 1 Ruby Ruby        95 Feb 25 02:59 storage.bson
drwx------ 4 Ruby Ruby      4096 Feb 25 10:28 testdb

* OOM:

Feb 27 21:24:49 localhost kernel: [239239.871767] Out of memory: Kill process 13316 (mongod) score 863 or sacrifice child
Feb 27 21:24:49 localhost kernel: [239239.871801] Killed process 13316 (mongod) total-vm:6096592kB, anon-rss:3286352kB, file-rss:0kB
Feb 27 21:24:49 localhost kernel: [239239.873442] systemd-journal: page allocation failure: order:0, mode:0x201da
Feb 27 21:24:49 localhost kernel: [239239.873445] CPU: 1 PID: 451 Comm: systemd-journal Tainted: G           OE  ----V-------   3.10.0-327.62.59.83.h128.x86_64 #1
Feb 27 21:24:49 localhost kernel: [239239.873446] Hardware name: OpenStack Foundation OpenStack Nova, BIOS rel-1.10.2-0-g5f4c7b1-20181220_000000-szxrtosci10000 04/01/2014
Feb 27 21:24:49 localhost kernel: [239239.873447] Call Trace:

The above scenario is just a simplification of our business.
In actual business, we create and delete almost 30000 collections per day.
Approximately every 10 days, our business also meet database crash by OOM-killer.

So I would like to know:
How often does Mongodb support DDL operations?

Thanks for your help. 



 Comments   
Comment by Carl Champain (Inactive) [ 28/Feb/20 ]

Hi 2669706182@qq.com,

MongoDB 3.4 is already end-of-lifed, which means it is unsupported by us. We strongly recommend you upgrade to a supported version.
That said, the SERVER project is for bugs and feature suggestions for the MongoDB server. As this ticket does not appear to be a bug, I will now close it. If you need assistance troubleshooting, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Kind regards,
Carl
 

Generated at Thu Feb 08 05:11:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.