[SERVER-64659] Report in serverStatus number of file descriptors used during index builds Created: 18/Mar/22  Updated: 29/Oct/23  Resolved: 29/Apr/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.2.23, 6.0.0-rc6, 5.0.10, 4.4.16, 6.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Eric Sedor Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-68158 serverstatus_indexbulkbuilder.js shou... Closed
related to SERVER-54791 Excessive file descriptor usage durin... Closed
related to SERVER-60789 FTDC Collector for open files Closed
related to SERVER-65796 convert serverStatus metrics for inde... Closed
is related to SERVER-64660 Report in serverStatus number of file... Open
is related to SERVER-54791 Excessive file descriptor usage durin... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.0, v5.3, v5.0, v4.4, v4.2
Sprint: Execution Team 2022-05-02
Participants:
Linked BF Score: 28

 Description   

Currently, nodes can reach a "too many open files" state with a low number of connections and low wiredTiger data handle usage. Reporting file descriptors used as part of index builds will help close a diagnostic gap to identify index builds as a source of index file descriptors use.

Ideally, record this as two counters:

  • file handles opened
  • file handles closed


 Comments   
Comment by Githook User [ 29/Aug/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add multiple index spec test case for indexBulkBuilder server status section

(cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f)
Branch: v4.2
https://github.com/mongodb/mongo/commit/26d023b723fceba0214fddcf1cddbbe603295b14

Comment by Githook User [ 28/Aug/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add initial sync test case for indexBulkBuilder server status section

(cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c)
Branch: v4.2
https://github.com/mongodb/mongo/commit/def884b5dabc692ceb11e878f4b4ce28335f4804

Comment by Githook User [ 27/Aug/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add external sorter file handle metrics to indexBulkBuilder server status section

(cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64)
Branch: v4.2
https://github.com/mongodb/mongo/commit/8feeb48360ee87f396068ffe00cc1a5d7282e2e0

Comment by Githook User [ 26/Aug/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add indexBulkBuilder server status section with minimal creation metrics

(cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4)
Branch: v4.2
https://github.com/mongodb/mongo/commit/eb3adbe5a061ecb3c7fb123f42241bbae03f88bd

Comment by Benety Goh [ 25/Aug/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 Sorter::File accepts optional SorterFileStats for tracking file open/close operations

(cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f)
Branch: v4.2
https://github.com/mongodb/mongo/commit/26d341317559b63d21f07f25f7ff2f99373891a2

Comment by Githook User [ 07/Jul/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add multiple index spec test case for indexBulkBuilder server status section

(cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f)
Branch: v4.4
https://github.com/mongodb/mongo/commit/9827d2a5d878ff635285e394b98648231dbea268

Comment by Githook User [ 29/Jun/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add initial sync test case for indexBulkBuilder server status section

(cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c)
Branch: v4.4
https://github.com/mongodb/mongo/commit/a2f796d77b94ab24f27a04751522943bc12d8df8

Comment by Githook User [ 25/Jun/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add external sorter file handle metrics to indexBulkBuilder server status section

(cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64)
Branch: v4.4
https://github.com/mongodb/mongo/commit/32aff083f4698a2632f8a7037ce4150afa958e90

Comment by Githook User [ 24/Jun/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add indexBulkBuilder server status section with minimal creation metrics

(cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4)
Branch: v4.4
https://github.com/mongodb/mongo/commit/e163059b8d3d069d52a251970266cedc4a12fb54

Comment by Githook User [ 22/Jun/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 Sorter::File accepts optional SorterFileStats for tracking file open/close operations

(cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f)
Branch: v4.4
https://github.com/mongodb/mongo/commit/ead0d926edbbcbb795776a2f05f8e4245920c8ff

Comment by Githook User [ 22/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add multiple index spec test case for indexBulkBuilder server status section

(cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f)
Branch: v5.0
https://github.com/mongodb/mongo/commit/511684289723dc2114af9edddd10bbb81f772c88

Comment by Githook User [ 21/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add initial sync test case for indexBulkBuilder server status section

(cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c)
Branch: v5.0
https://github.com/mongodb/mongo/commit/1bb040c5be4866296ebe1e68930e76aae1cc3539

Comment by Githook User [ 20/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add external sorter file handle metrics to indexBulkBuilder server status section

(cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64)
Branch: v5.0
https://github.com/mongodb/mongo/commit/aecbb01f4bf1e34f0f299eab5b6ea950dd2a7a73

Comment by Githook User [ 19/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add resumable index build counter to indexBulkBuild server status section

(cherry picked from commit 0c40fc644df33161b34faa3708545e894757327e)
Branch: v5.0
https://github.com/mongodb/mongo/commit/856c10ff0da11b45e1a8462e48f93cf03381485c

Comment by Githook User [ 18/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add indexBulkBuilder server status section with minimal creation metrics

(cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4)
Branch: v5.0
https://github.com/mongodb/mongo/commit/776a79fd2e99e418e28e0cb5b875a859ee43c38f

Comment by Githook User [ 17/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 Sorter::File accepts optional SorterFileStats for tracking file open/close operations

(cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f)
Branch: v5.0
https://github.com/mongodb/mongo/commit/a470de123615559ab3ddc244f71e79e6c52c7812

Comment by Githook User [ 10/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add multiple index spec test case for indexBulkBuilder server status section

(cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f)
Branch: v6.0
https://github.com/mongodb/mongo/commit/62d427c2bba0a24e9205f27faecaf9cf04a1570f

Comment by Githook User [ 09/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add initial sync test case for indexBulkBuilder server status section

(cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c)
Branch: v6.0
https://github.com/mongodb/mongo/commit/e8f633678412d0faee1ce0c00685cfaa6d4001c7

Comment by Githook User [ 06/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add external sorter file handle metrics to indexBulkBuilder server status section

(cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64)
Branch: v6.0
https://github.com/mongodb/mongo/commit/8f2a387c7276734ef35daa463abfd1d16dfce6ad

Comment by Githook User [ 05/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add resumable index build counter to indexBulkBuild server status section

(cherry picked from commit 0c40fc644df33161b34faa3708545e894757327e)
Branch: v6.0
https://github.com/mongodb/mongo/commit/be69831f52c4e3b044584d5caf2d50b1e8c958f9

Comment by Githook User [ 04/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 fix lint
Branch: v6.0
https://github.com/mongodb/mongo/commit/b4950d6e53e4ebc85fc65c717649619a43ea13f6

Comment by Githook User [ 04/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add indexBulkBuilder server status section with minimal creation metrics

(cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4)
Branch: v6.0
https://github.com/mongodb/mongo/commit/5659527d24b3ae1d22a8e0fde26ac7d3a92310a0

Comment by Githook User [ 03/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 Sorter::File accepts optional SorterFileStats for tracking file open/close operations

(cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f)
Branch: v6.0
https://github.com/mongodb/mongo/commit/725536e39566658c224af146461b996541be1abe

Comment by Githook User [ 03/May/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 move sorter.h include to IndexAccessMethod cpp file

(cherry picked from commit dbbf2c06895528cbb96d1098ef00e8bca08e9784)
Branch: v6.0
https://github.com/mongodb/mongo/commit/919f4cc6ce28cc8b29c2f7d1383c73968e52ef6e

Comment by Githook User [ 29/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add multiple index spec test case for indexBulkBuilder server status section
Branch: master
https://github.com/mongodb/mongo/commit/e2abe094a39dda8d4b8f623de23423868c7efb8f

Comment by Githook User [ 29/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add initial sync test case for indexBulkBuilder server status section
Branch: master
https://github.com/mongodb/mongo/commit/4adfa1a54c9cba165525caf4a4200a891945637c

Comment by Githook User [ 29/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add external sorter file handle metrics to indexBulkBuilder server status section
Branch: master
https://github.com/mongodb/mongo/commit/bfc606f1cc5ad931f054b5c152993a57d30eef64

Comment by Githook User [ 29/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add resumable index build counter to indexBulkBuild server status section
Branch: master
https://github.com/mongodb/mongo/commit/0c40fc644df33161b34faa3708545e894757327e

Comment by Githook User [ 28/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 add indexBulkBuilder server status section with minimal creation metrics
Branch: master
https://github.com/mongodb/mongo/commit/adc47cc974960abf0bc10a1b15234dc95ee6fdd4

Comment by Githook User [ 28/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 Sorter::File accepts optional SorterFileStats for tracking file open/close operations
Branch: master
https://github.com/mongodb/mongo/commit/176747132c3be0b0e73adebf55ba4acf6211934f

Comment by Githook User [ 28/Apr/22 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-64659 move sorter.h include to IndexAccessMethod cpp file
Branch: master
https://github.com/mongodb/mongo/commit/dbbf2c06895528cbb96d1098ef00e8bca08e9784

Comment by Eric Sedor [ 08/Apr/22 ]

I think we can see cases where a large number of index builds occur at once, so I'm agreed that counters are best. Added this to the description

Comment by Kevin Arhelger [ 28/Mar/22 ]

eric.sedor,

I share the desire to track all file handles currently open (as well as non storage engine IO).

When a system runs out of file handles, its non-trivial to understand what contributed to the issue:

1. (monitored) 2*number of open connections SERVER-39147
2. (monitored) Number of dhandles
3. (unmonitored) Index builds
4. (unmonitored) allowDiskUseTrue
5. (unmonitored but small) journal, log, audit log, pid, lock.

On linux /proc/sys/fs/file-nr can be used to determine how close we are to the limit, but that doesn't show the contribution from the mongod process vs. other system activity.

As far as design I can see two approaches:

1. (Gauge) File handles currently open.
2. (Counter) File handles opened and file handles closed

Theres benefit for counters vs. gauges and I would prefer the counter approach.

Comment by Bruce Lucas (Inactive) [ 23/Mar/22 ]

What is the specific request - to report the current number of file descriptors currently in use across all index builds currently running at the point where serverStatus is called? This would be useful in FTDC if the offending index build(s) take substantially more than 1 second, which I think is likely to be the case, especially for an index build that uses a large number of files.

Comment by Eric Sedor [ 22/Mar/22 ]

Thanks Connie, I think that would change the diagnostic gap in 4.4+ to one where we are currently extrapolating the number of indexes being built at any one time from the logs, versus seeing the number of builds in FTDC. It's worth noting this number could still fluctuate very high for some use-cases and that may be useful signal. But no specific cases come to mind.

kevin.arhelger what's your sense of this?

Comment by Connie Chen [ 22/Mar/22 ]

eric.sedor this change would only be done for 4.2, is it still necessary since the newer versions (4.4+) only use one handle per index?

Generated at Thu Feb 08 06:00:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.