[SERVER-46701] Frequently facing issue with Too many open files Created: 06/Mar/20  Updated: 28/Apr/20  Resolved: 28/Apr/20

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 3.6.17
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Arun Kumar Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Participants:

 Description   

Hi Team,

Frequently we are facing production issue with MongoDB v3.6.17 

Mongod went down due to below errors.

What is does mean and let us know the solution for this.

Logs : 

NETWORK [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files

NETWORK [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files



 Comments   
Comment by Dmitry Agranat [ 28/Apr/20 ]

Hi arunkumar@mydbops.com,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 29/Mar/20 ]

Hi arunkumar@mydbops.com,

Hi,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please provide the requested information from my last comment?

Thanks,
Dima

Comment by Dmitry Agranat [ 13/Mar/20 ]

Hi arunkumar@mydbops.com,

Would you please archive (tar or zip) the $dbpath/diagnostic.data directory (the contents are described here) and mongod logs covering the time of this event and attach it to this ticket?

I've created a secure upload portal for you. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Also, can you provide the output from these commands executed on the server in question via the dbpath? This would give us the total amount of files.

ls | grep -iE 'index.*wt' | wc -l
ls | grep -iE 'collection.*wt' | wc -l

Comment by Arun Kumar [ 10/Mar/20 ]

Hi Dmitry,

Already we have tune those variables.

Could you please find my ulimit -a and limit.d output and let us know if anything need to change further.

Error : 

 

2020-03-10T03:46:20.001+0530 E - [ftdc] Assertion: 13538:couldn't open [/proc/16663/stat] Too many open files src/mongo/util/processinfo_linux.cpp 79
2020-03-10T03:46:20.658+0530 E STORAGE [conn1068439] WiredTiger error (24) [1583792180:658606][16663:0x7f508be41700], file:collection-2070-4001183421758999670.wt, WT_SESSION.open_cursor: __posix_open_file, 715:
 /data/db/mongo/collection-2070-4001183421758999670.wt: handle-open: open: Too many open files Raw: [1583792180:658606][16663:0x7f508be41700], file:collection-2070-4001183421758999670.wt, WT_SESSION.open_cursor:
 __posix_open_file, 715: /data/db/mongo/collection-2070-4001183421758999670.wt: handle-open: open: Too many open files
2020-03-10T03:46:20.658+0530 E STORAGE [conn1068439] An unsupported journal format detected - If you are trying to rollback from version 4.0 to 3.6, please re-start a 4.0 binary and cleanly shut it down so that
 the journal format will be downgraded.
2020-03-10T03:46:20.658+0530 F - [conn1068439] Invariant failure: ret resulted in status UnknownError: 24: Too many open files at src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp 130
2020-03-10T03:46:20.658+0530 F - [conn1068439]
***aborting after invariant() failure 

2020-03-10T03:46:20.723+0530 F - [conn1068439] Got signal: 6 (Aborted).

Logs : 

 

 

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64064
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 64064
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
/etc/security/limits.conf
# End of file
mongod soft nofile 55500
mongod hard nofile 65500
mongod soft nproc 55500
mongod hard nproc 65500

 

Comment by Dmitry Agranat [ 08/Mar/20 ]

Hi arunkumar@mydbops.com,

Please validate that your cluster is configured according to our Production Notes, specifically this section.

Thanks,
Dima

Generated at Thu Feb 08 05:12:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.