[SERVER-20128] too many open duplicate files my mongod with WT Created: 26/Aug/15  Updated: 28/Aug/15  Resolved: 28/Aug/15

Status: Closed
Project: Core Server
Component/s: WiredTiger
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Denis Assignee: David Hows
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive lsof_dumps.zip    
Participants:

 Description   

Hi , I have many open fd by mongod process.
lsof | grep mongod | grep "/disk2/mongodb_account/" |wc -l
66045
Over time, number fd is growing to 250k - 300k.
lsof | grep mongo | grep "disk2" | grep "/63--6528694272700620922.wt" | wc -l
111
mongod open file 63--6528694272700620922.wt 111 times .
what caused this behavior?

I use WT engine and 3.0.6 version.



 Comments   
Comment by Denis [ 28/Aug/15 ]

It's my mistake. Thanks for explain

Comment by David Hows [ 28/Aug/15 ]

Hi Denis,

The issue here is a misinterpretation of what lsof is outputting.

By default when you run lsof with no arguments it will report each instance of a held file handle by each thread in the system, this means you wind up with many duplicates of the same file handle being held by many threads within a process. You can see this by reviewing the FD column in lsof; for example, in the attached lsof_with_grep.txt file you will see that the second column is static (as this is the MongoDB PID), you will see that the third column is increasing, as this represents the thread ID within the given PID. Finally in the fifth column you will see that everything shares the same value of 15u. This 15u value states that this is the systems "15" file descriptor open in read/write mode (see the lsof man page for more details) and this single file handle is opened by each of the threads, this is expected behaviour in MongoDB all of the threads reported will want access to the underlying WiredTiger files.

Can you please re-run the lsof command and specify the MongoDB PID with the -p option to detail only the open file handles for the process rather than each thread. Eg, please run:

lsof -p 5467

or

lsof -p `pidof mongod`

Thanks,
David

Generated at Thu Feb 08 03:53:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.