[SERVER-54791] Excessive file descriptor usage during index build with external sort Created: 25/Feb/21  Updated: 29/Oct/23  Resolved: 28/Jul/21

Status: Closed
Project: Core Server
Component/s: Index Maintenance
Affects Version/s: 4.2.12
Fix Version/s: 5.0.3, 4.4.10, 5.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Dmitry Agranat Assignee: Gregory Noma
Resolution: Fixed Votes: 6
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Duplicate
is duplicated by SERVER-50709 Use boost::filesystem::path instead o... Closed
Problem/Incident
Related
related to SERVER-64659 Report in serverStatus number of file... Closed
related to SERVER-44984 Reduce index thread pool size and red... Closed
is related to SERVER-64659 Report in serverStatus number of file... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.0, v4.4, v4.2
Sprint: Execution Team 2021-07-12, Execution Team 2021-07-26, Execution Team 2021-08-09
Participants:
Case:
Linked BF Score: 151

 Description   

During the index build phase with the external sort, file descriptor usage can be in thousands or even tens of thousands. This might be problematic for workloads that already consume a significant number of file descriptors or OS's where increasing the equivalent to file descriptors might be a problem.



 Comments   
Comment by Githook User [ 23/Sep/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-54791 Use single file descriptor for external sort
Branch: v4.4
https://github.com/mongodb/mongo/commit/a9efdaf12d5f8a2c9ba9610d21fafdd0eeb8551e

Comment by Githook User [ 04/Aug/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-54791 Use single file descriptor for external sort

(cherry picked from commit 6e57f295d4fcf932f58cb402a056001c75fde9d2)
(cherry picked from commit 311b84df538a5ee9ab4db507f610d8b814bb2099)
Branch: v5.0
https://github.com/mongodb/mongo/commit/62e0f0c801e85d8938ec8548b624556d6f05e1fc

Comment by Githook User [ 28/Jul/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-54791 Fix `Sorter::File` cleanup
Branch: master
https://github.com/mongodb/mongo/commit/311b84df538a5ee9ab4db507f610d8b814bb2099

Comment by Githook User [ 28/Jul/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-54791 Use single file descriptor for external sort
Branch: master
https://github.com/mongodb/mongo/commit/6e57f295d4fcf932f58cb402a056001c75fde9d2

Generated at Thu Feb 08 05:34:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.