[DOCS-15371] Investigate changes in SERVER-63261: Add metrics for wait time for requests to acquire egress connections Created: 28/May/22  Updated: 13/Nov/23  Resolved: 13/Jul/22

Status: Closed
Project: Documentation
Component/s: manual, Server
Affects Version/s: 6.1.0
Fix Version/s: 6.1.0-rc0, Server_Docs_20231030, Server_Docs_20231106, Server_Docs_20231105, Server_Docs_20231113

Type: Task Priority: Major - P3
Reporter: Backlog - Core Eng Program Management Team Assignee: Jason Price
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
documents SERVER-63261 Add metrics for wait time for request... Closed
Participants:
Days since reply: 1 year, 22 weeks ago
Epic Link: DOCSP-21764
Story Points: 3

 Description   
Original Downstream Change Summary

New keys added to connPoolStats command output.

Description of Linked Ticket

On sharded clusters, client requests and system operations need to be routed from mongos to mongod, and occasionally from mongod to mongod. These requests need to acquire an outbound connection from the source host to the target host. This process is asynchronous, and we don't know how long it takes for requests to acquire a connection. As a result, it's difficult for us to determine when egress connection pooling might be a bottleneck for servicing requests, or what the exact impact on request latency is when the connection pool is under strain. 

 

We should add metrics that answer the question: 'how much time does a request from (serverA) to (serverB) spend waiting to acquire a connection?' It might be best to use a histogram-based approach, in the style of SERVER-59858, where we maintain a histogram of wait-times for the last N connections/over the last X minutes. We could also 'rotate' the histograms, where we always keep one for the last (say) minute, and then have an 'aggregated' one of the last N minutes. It also would be ideal to collect the histograms on a per-targeted-host basis. 

Generally, egress connections are acquired by requests  here for the NITL-based task executors. ScopedDBConnection, defined here, is also sometimes used to acquire connections, namely in the old "scanning" RSM, dbclient_rs, and a 1 or 2 sharded commands, but it may not be worth it to collect metrics for this outdated component.  



 Comments   
Comment by Githook User [ 07/Sep/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: Docs-15371 metric egress connections (#1444)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: master
https://github.com/10gen/docs-mongodb-internal/commit/61f3d8bf82c9b1b92d7c6a717649fcf1afefdb4f

Comment by Githook User [ 07/Sep/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: Docs-15371 metric egress connections (#1444)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: master
https://github.com/10gen/docs-mongodb-internal/commit/61f3d8bf82c9b1b92d7c6a717649fcf1afefdb4f

Comment by Githook User [ 13/Jul/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: Docs-15371 metric egress connections (#1444)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: v6.1
https://github.com/10gen/docs-mongodb-internal/commit/61f3d8bf82c9b1b92d7c6a717649fcf1afefdb4f

Comment by Githook User [ 13/Jul/22 ]

Author:

{'name': 'jason-price-mongodb', 'email': '69260375+jason-price-mongodb@users.noreply.github.com', 'username': 'jason-price-mongodb'}

Message: Docs-15371 metric egress connections (#1444)

Co-authored-by: jason-price-mongodb <jshfjghsdfgjsdjh@aolsdjfhkjsdhfkjsdf.com>
Branch: v6.1
https://github.com/10gen/docs-mongodb-internal/commit/61f3d8bf82c9b1b92d7c6a717649fcf1afefdb4f

Comment by Education Bot [ 28/May/22 ]

Fix Version updated for upstream SERVER-63261:
6.1.0-rc0

Generated at Thu Feb 08 08:12:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.