[SERVER-44152] Pre-warm connection pools in mongos Created: 22/Oct/19  Updated: 30/Sep/21  Resolved: 13/Nov/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.3.2, 4.2.17

Type: New Feature Priority: Major - P3
Reporter: Randolph Tan Assignee: Anton Oyung
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Problem/Incident
causes SERVER-60344 Action plan on lagging setFCV replica... Closed
Related
related to SERVER-47169 Sharding initialization contacts conf... Closed
related to SERVER-59941 Refactor pre-warm connection pools in... Backlog
Backport Requested:
v4.2
Sprint: Sharding 2019-10-21, Sharding 2019-11-04, Sharding 2019-11-18
Participants:
Linked BF Score: 145

 Description   

Problem statement

When a mongos is initially brought up, it lacks connections to any of the cluster’s shard servers. This can add latency to user operations, as we require a request to a host before we start spooling a connections (even in the presence of minPoolSize).

Proposal

Provide a new startup parameter, which would cause mongos wait a configurable amount of time to establish at least minPoolSize connections to each shard server before accepting user connections. This would allow us to reboot mongos’, or add new ones, without making any initial user requests pay the overhead of waiting on connection establishment.

This should should be relatively simple. Cribbing from the test only mongos command: multicast

  1. Temporarily turn the host timeout to infinity
  2. Fetch all shard ids
  3. Foreach shard id
    1. Get the connection string
    2. Foreach host in the connection string
      1. Push the host onto a list
  4. Use the AsyncMulticaster, with the arbitrary executor, to multicast a ping to all hosts
  5. Wait for a configurable delay, periodically checking connpoolstats to see if hosts all have minPoolSize connections
    1. If all pools reach the desired size before the timeout, stop waiting
    2. If all pools don’t, continue at the end of the timeout
  6. Return the host timeout to its former value
  7. Start accepting client connections

Steps 1 and 7 may also be optional (if we’re waiting 10 seconds, the 5 minute host timeout may not matter).

In either case, the work should be relatively approachable, as it only involves being a regular client of the networking layer in mongos, rather than doing any development against the interior components.



 Comments   
Comment by Andrew Shuvalov (Inactive) [ 30/Sep/21 ]

The pre-warm feature is causing BF-22699, if you are interested to help with insight please comment in SERVER-60344.

Comment by Githook User [ 17/Sep/21 ]

Author:

{'name': 'Anton Oyung', 'email': 'anton.oyung@mongodb.com', 'username': 'AntonOyung'}

Message: SERVER-44152: Pre-warm connection pools in mongos
Branch: v4.2
https://github.com/mongodb/mongo/commit/be089838c55d33b6f6039c4219896ee4a3cd704f

Comment by Githook User [ 13/Nov/19 ]

Author:

{'name': 'Anton Oyung', 'username': 'AntonOyung', 'email': 'anton.oyung@mongodb.com'}

Message: SERVER-44152: Pre-warm connection pools in mongos
Branch: master
https://github.com/mongodb/mongo/commit/b375698b7fe1f4e69761559f1cad50c5e1f18014

Generated at Thu Feb 08 05:05:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.