[SERVER-70858] WiredTiger (24) opendir: Too many open files Created: 26/Oct/22  Updated: 28/Nov/22  Resolved: 28/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.10
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: brian wang Assignee: Chris Kelly
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Servers:
Red Hat Enterprise Linux Server release 6.7 (Santiago)

MongoDB:
db version v3.2.10
git version: 7909b3ab5ce2051c2726441120271008200317
OpenSSL version: OpenSSL 1.O. le-fips 11 Feb 2013
al locator: temalloc modules: none build environment:
distmod: rhe162
distarch: ×86_64
target_arch: ×86_64

ulimit -a Content:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 515665
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 1024
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


Attachments: Text File mongo-1.log    
Issue Links:
Related
Operating System: ALL
Participants:

 Description   

Hello. 
It's a system that's been working well for many years, and suddenly this issue has occurred. 

The service stopped after the first few query errors were output followed by a storage error. 

Please check. I wonder why these symptoms occur and what solutions are there?

The contents below are the contents of the log.
Thank you.

Query contents: 
"2022-10-21T14:59:47.386+0900 E QUERY    [conn1469211] Plan executor error during find command: FAILURE, stats: { stage: "SORT", nReturned: 0, executionTimeMillisEstimate: 420, works: 3704, advanced: 0, needTime: 3703, needYield: 0, saveState: 37, restoreState: 37, isEOF: 0, invalidates: 0, sortPattern:

{ create_date: -1.0 }

, memUsage: 33554957, memLimit: 33554432, inputStage: { stage: "SORT_KEY_GENERATOR", nReturned: 0, executionTimeMillisEstimate: 420, works: 3703, advanced: 0, needTime: 2, needYield: 0, saveState: 37, restoreState: 37, isEOF: 0, invalidates: 0, inputStage: { stage: "COLLSCAN", filter:

{ $and: [] }

, nReturned: 3701, executionTimeMillisEstimate: 400, works: 3702, advanced: 3701, needTime: 1, needYield: 0, saveState: 37, restoreState: 37, isEOF: 0, invalidates: 0, direction: "forward", docsExamined: 3701 } } }"

Storage Errors: 
"2022-10-21T23:19:15.942+0900 E STORAGE  [thread2] WiredTiger (24) [1666361955:942512][25242:0x7f45fbb27700], checkpoint-server: checkpoint server error: Too many open files
2022-10-21T23:19:15.942+0900 E STORAGE  [thread2] WiredTiger (-31804) [1666361955:942550][25242:0x7f45fbb27700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic"



 Comments   
Comment by brian wang [ 02/Nov/22 ]

Thank you for the guide.

 

Regards,

brian

Comment by Chris Kelly [ 26/Oct/22 ]

chingwen.wang@saltlux.com,

MongoDB 3.2 reached end of life in September 2018 and is no longer supported. In order to investigate the issue further, we would require you to be on a supported version such as 4.2 or above. 

If this issue persists for you on a supported version, we would be interested in investigating this further. After you upgrade, for each node in the replica set spanning a time period that includes the incident, please upload to the ticket:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)

Regards,

Christopher

 

 

 

 

Generated at Thu Feb 08 06:17:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.