[SERVER-64659] Report in serverStatus number of file descriptors used during index builds Created: 18/Mar/22 Updated: 29/Oct/23 Resolved: 29/Apr/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 4.2.23, 6.0.0-rc6, 5.0.10, 4.4.16, 6.1.0-rc0 |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Eric Sedor | Assignee: | Benety Goh |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||
| Backport Requested: |
v6.0, v5.3, v5.0, v4.4, v4.2
|
||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2022-05-02 | ||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||
| Linked BF Score: | 28 | ||||||||||||||||||||||||||||||||
| Description |
|
Currently, nodes can reach a "too many open files" state with a low number of connections and low wiredTiger data handle usage. Reporting file descriptors used as part of index builds will help close a diagnostic gap to identify index builds as a source of index file descriptors use. Ideally, record this as two counters:
|
| Comments |
| Comment by Githook User [ 29/Aug/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f) |
| Comment by Githook User [ 28/Aug/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c) |
| Comment by Githook User [ 27/Aug/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64) |
| Comment by Githook User [ 26/Aug/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4) |
| Comment by Benety Goh [ 25/Aug/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f) |
| Comment by Githook User [ 07/Jul/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f) |
| Comment by Githook User [ 29/Jun/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c) |
| Comment by Githook User [ 25/Jun/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64) |
| Comment by Githook User [ 24/Jun/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4) |
| Comment by Githook User [ 22/Jun/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f) |
| Comment by Githook User [ 22/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f) |
| Comment by Githook User [ 21/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c) |
| Comment by Githook User [ 20/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64) |
| Comment by Githook User [ 19/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 0c40fc644df33161b34faa3708545e894757327e) |
| Comment by Githook User [ 18/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4) |
| Comment by Githook User [ 17/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f) |
| Comment by Githook User [ 10/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit e2abe094a39dda8d4b8f623de23423868c7efb8f) |
| Comment by Githook User [ 09/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 4adfa1a54c9cba165525caf4a4200a891945637c) |
| Comment by Githook User [ 06/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit bfc606f1cc5ad931f054b5c152993a57d30eef64) |
| Comment by Githook User [ 05/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 0c40fc644df33161b34faa3708545e894757327e) |
| Comment by Githook User [ 04/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 04/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit adc47cc974960abf0bc10a1b15234dc95ee6fdd4) |
| Comment by Githook User [ 03/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit 176747132c3be0b0e73adebf55ba4acf6211934f) |
| Comment by Githook User [ 03/May/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: (cherry picked from commit dbbf2c06895528cbb96d1098ef00e8bca08e9784) |
| Comment by Githook User [ 29/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 29/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 29/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 29/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 28/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 28/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Githook User [ 28/Apr/22 ] |
|
Author: {'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}Message: |
| Comment by Eric Sedor [ 08/Apr/22 ] |
|
I think we can see cases where a large number of index builds occur at once, so I'm agreed that counters are best. Added this to the description |
| Comment by Kevin Arhelger [ 28/Mar/22 ] |
|
I share the desire to track all file handles currently open (as well as non storage engine IO). When a system runs out of file handles, its non-trivial to understand what contributed to the issue: 1. (monitored) 2*number of open connections On linux /proc/sys/fs/file-nr can be used to determine how close we are to the limit, but that doesn't show the contribution from the mongod process vs. other system activity. As far as design I can see two approaches: 1. (Gauge) File handles currently open. Theres benefit for counters vs. gauges and I would prefer the counter approach. |
| Comment by Bruce Lucas (Inactive) [ 23/Mar/22 ] |
|
What is the specific request - to report the current number of file descriptors currently in use across all index builds currently running at the point where serverStatus is called? This would be useful in FTDC if the offending index build(s) take substantially more than 1 second, which I think is likely to be the case, especially for an index build that uses a large number of files. |
| Comment by Eric Sedor [ 22/Mar/22 ] |
|
Thanks Connie, I think that would change the diagnostic gap in 4.4+ to one where we are currently extrapolating the number of indexes being built at any one time from the logs, versus seeing the number of builds in FTDC. It's worth noting this number could still fluctuate very high for some use-cases and that may be useful signal. But no specific cases come to mind. kevin.arhelger what's your sense of this? |
| Comment by Connie Chen [ 22/Mar/22 ] |
|
eric.sedor this change would only be done for 4.2, is it still necessary since the newer versions (4.4+) only use one handle per index? |