[SERVER-43170] MongoDB - Chat DB get stuck some time and repeat inifinitely when have to too many collections (800.000) Created: 05/Sep/19  Updated: 11/Sep/19  Resolved: 09/Sep/19

Status: Closed
Project: Core Server
Component/s: Performance, WiredTiger
Affects Version/s: 3.6.1, 3.6 Required
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Doanh Ngô Văn Assignee: Danny Hatcher (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongo_stuck.log     Text File mongo_stuck2.log     Text File mongo_stuck3.log    
Operating System: ALL
Participants:

 Description   

My DB get stuck some time about 6-10 second and repeat again inifinitely.

My DB design model:

  • DBName:
    + chatlogextension1
  • Collection names are mongo objectIds represent user ids - 200.000 collections per DB:
    5d21f0b2383cb27bfc908f3b
    5d21f0c2383cb27bfc908f72
    5d21f0dd383cb27bfc908fce
    5d21f0e9383cb27bfc909001
    5d21f10d383cb27bfc9090b1
    5d21f114383cb27bfc9090e1
    5d21f12d383cb27bfc909143
    ...

And i have about 5 chat DBs: chatlogextension1, chatlogextension2,...chatlogextension5.
When i query to these chat DBs, chat DB get stuck about per one minutes and repeat.

  • While other DB is not effected.
  • This is my OS Resources:
    + CPU: everage 15%
    + RAM: everage 72% (92/128 GB)
    + Max open files: 1000000 (ulimit -n)

May MongoDB handle about 800.000 collections or wiretiger  can handle 800.000 files?



 Comments   
Comment by Doanh Ngô Văn [ 11/Sep/19 ]

Thank for your great response!

Comment by Danny Hatcher (Inactive) [ 09/Sep/19 ]

Nothing stands out at me as an obvious bottleneck so it is likely that you are simply running into the limitations of the maximum number of open files that a mongod can reasonable handle. In situations like this, I recommend a re-architecture of your schema so that you are are not using so many collections in a short period of time. Additionally, you may wish to look into Sharding as it offers a relatively easy way to scare horizontally so any given process is not dealing with a million files. Finally, I recommend upgrading to the latest version of 3.6 (3.6.14 at the time of this comment) as we've made many performance improvements that may help your situation.

As this does not look like a bug in MongoDB, I'm going to close this ticket. If you need further assistance, I encourage you to ask our community by posting on the mongodb-user group or on Stack Overflow with the mongodb tag.

Comment by Doanh Ngô Văn [ 09/Sep/19 ]

the open files number my mongodb use is 728968 now

Comment by Doanh Ngô Văn [ 06/Sep/19 ]

I have update 3 log files at the time chatlogextension dbs are stuck.

Comment by Doanh Ngô Văn [ 06/Sep/19 ]

Thank for your reply! because Jira not allow to upload greater than 150MB size so i uploaded to my google drive. 
This is my diagnostic.data folder - zip file drive link: https://drive.google.com/file/d/1Z2Dfxn7PTzczFQcrt69p9-cEXfxb6bz8/view?usp=sharing

Comment by Danny Hatcher (Inactive) [ 05/Sep/19 ]

Having that many files open can cause problems but if you provide your "diagnostic.data" folder for the Primary node(s) I can take a look.

Generated at Thu Feb 08 05:02:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.