[SERVER-25027] Configurable connection pools size for mongos Created: 13/Jul/16  Updated: 08/Jan/24  Resolved: 07/Nov/16

Status: Closed
Project: Core Server
Component/s: Networking, Sharding
Affects Version/s: None
Fix Version/s: 3.2.11, 3.4.0-rc3

Type: Improvement Priority: Critical - P2
Reporter: Dmitry Ryabtsev Assignee: Mira Carey
Resolution: Done Votes: 3
Labels: code-only
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Documented
is documented by DOCS-9479 Docs for SERVER-25027: Configurable c... Closed
Duplicate
is duplicated by SERVER-22082 Ability to specify minimum size for t... Closed
Related
related to SERVER-26740 Total connections is not stable and s... Closed
is related to SERVER-28564 How to maintain consistant connectio... Closed
is related to DOCS-10130 Documentation for connPoolStats is in... Closed
Backwards Compatibility: Fully Compatible
Backport Completed:
Sprint: Platforms 2016-10-10, Platforms 2016-10-31, Platforms 2016-11-21
Participants:
Case:

 Description   
Issue Status as of Apr 06, 2017

ISSUE DESCRIPTION
MongoDB 3.2.11 and 3.4.0 introduce a set of configurable parameters in mongos to help manage NetworkInterfaceASIO-based connection pools in sharded clusters. These parameters are:

TaskExecutorPoolSize
  • The number of NetworkInterfaceASIO-based connection pools to use for a given mongos
  • Defaults to 4 <= NUM_CORES <= 64
  • When tuned down, this will reduce the number of open connections in the system at some performance cost under a heavier load
  • A good default is generally either the actual default, or <16 if the the node running mongod has a large number of cores
ShardingTaskExecutorPoolHostTimeoutMS
  • How long will mongos wait before dropping all connections to a host that it hasn't communicated with
  • Default is 5 minutes
  • Turning this up will prevent a mongos from entirely severing a connection to a mongod
  • Longer durations will smooth connection spikes. Setting this to 3-4x the period of the connection spikes will generally prevent completely cold starts
ShardingTaskExecutorPoolMaxSize
  • The maximum number of connections to hold to a given mongod at any time.
  • Default is unlimited
  • Giving this is a maximum value, may be useful to set a limit for this value if some mongod nodes in the system are suffering from connection floods
  • Remember that setting this effectively limits the number of simultaneous operations from a mongos to a mongod at PoolMaxSize * TaskExecutorPoolSize
ShardingTaskExecutorPoolMinSize
  • The minimum number of connections to hold to a given host at any time.
  • Default is 1
  • If a system is experiencing latency issues because of cold starts, users should consider increasing this value
  • The cost is that mongos will keep these sessions open, which is a real fixed cost of TaskExecutorPoolSize * PoolMinSize * NUM_MONGOS per shard
ShardingTaskExecutorPoolRefreshRequirementMS
  • How long will mongos wait before attempting to heartbeat an idle connection in the pool
  • Default is 1 minute
  • Setting this higher or lower will increase or decrease heartbeat traffic, which will increase/decrease idle load
  • Lower values may reduce the number of transient network failures seen in a system (because non-user operations will see them earlier and transparently re-connect)
  • It is unlikely that the default value needs to be modified
ShardingTaskExecutorPoolRefreshTimeoutMS
  • How long will mongos wait before timing out a heartbeat
  • Default 20 seconds
  • If this value is too low, mongos won't be able to maintain connections in the pool
  • Tuning this value up may help if heartbeats actually take 20 seconds
Original description

It would be nice if users could configure a outbound connection pools in a way as most of the drivers work. The use case is to do throttling of the amount of connections that hit the shards on the mongos level - e.g. if a number of requestes/active connections to mongos exceeds the specified threshold, some connections will have to wait for their turn to be served.

Implementing this would mean that we need to expose the standard connection pool-related options such as:

  • maxPoolSize
  • waitQueueMultiple
  • waitQueueTimeoutMS

Given that mongos is based on a C++ driver I presume that should be doable.

Note on the implementation of this ticket

To clarify, this commit exposes the connection pooling options that currently exist in mongos, rather than driver equivalent options.

Added knobs include:

  • ShardingTaskExecutorPoolHostTimeoutMS
  • ShardingTaskExecutorPoolMaxSize,
  • ShardingTaskExecutorPoolMinSize,
  • ShardingTaskExecutorPoolRefreshRequirementMS
  • ShardingTaskExecutorPoolRefreshTimeoutMS


 Comments   
Comment by Githook User [ 17/Nov/16 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}

Message: SERVER-25027 Configurable connpool in mongos

Export server parameters for sharding connection pool

Not cherry-picked
Branch: v3.2
https://github.com/mongodb/mongo/commit/009580ad490190ba33d1c6253ebd8d91808923e4

Comment by Githook User [ 15/Nov/16 ]

Author:

{u'username': u'Machyne', u'name': u'Matt Cotter', u'email': u'matt.cotter@mongodb.com'}

Message: SERVER-25027 fix windows compile
Branch: v3.2
https://github.com/mongodb/mongo/commit/4030f8f8a72d40e535294fd4feb5646a52a9a729

Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}

Message: SERVER-25027 add units to exported server params

(cherry picked from commit 5a3a0afc8f27f38c2cd6a455316e5c65a94429fd)
Branch: v3.2
https://github.com/mongodb/mongo/commit/1b3b7381a34b65cb24fa1fa3e91afd3bac69cbdf

Comment by Githook User [ 14/Nov/16 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}

Message: SERVER-25027 Configurable connpool in mongos

Export server parameters for sharding connection pool

(cherry picked from commit b6c29702d8dcadb6c1ee90a876fac4117e0ca062)
Branch: v3.2
https://github.com/mongodb/mongo/commit/54c96a778a051ebe0d1471475039205009908327

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}

Message: SERVER-25027 add units to exported server params
Branch: master
https://github.com/mongodb/mongo/commit/5a3a0afc8f27f38c2cd6a455316e5c65a94429fd

Comment by Githook User [ 07/Nov/16 ]

Author:

{u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}

Message: SERVER-25027 Configurable connpool in mongos

Export server parameters for sharding connection pool
Branch: master
https://github.com/mongodb/mongo/commit/b6c29702d8dcadb6c1ee90a876fac4117e0ca062

Comment by Adam Flynn [ 04/Nov/16 ]

Thanks Waley - appreciate it!

Comment by Waley Chen [ 04/Nov/16 ]

Hey Adam,

We're reviewing one last thing before merging it into master and
we're currently working on the 3.2 backport.

Waley

Comment by Adam Flynn [ 03/Nov/16 ]

Saw this got merged in - thanks Waley!

Any chance we can get a 3.2 backport?

Adam

Comment by Adam Flynn [ 03/Oct/16 ]

Hi Waley,

Exposing the new connection pool should work well enough for my use-case.

Thanks.
Adam

Comment by Waley Chen [ 03/Oct/16 ]

Hi Adam,

We have two connection pools, a new one and the legacy one which we intend to deprecate by 3.5.
We use the new one for find queries and the legacy one for everything else.
We can expose the connection pool options for the new pool but not the old one.
Would that work still be relevant to you?

Thanks,
Waley

Comment by Adam Flynn [ 14/Jul/16 ]

For a little context on use-case: we have an application with ~20k processes talking to a MongoDB cluster. On average, each process uses <1 connection (average ~0.25-0.5 of a connection), but it's sometimes useful to burst to a few operations in parallel.

Without pool size limits in mongos, a client pool size limit of 3 is 60k connections. When the DB slows down, the connection pool pressure causes us to go from a steady state of 10k connections to 30-60k connections very quickly, making a bad situation worse. Because of this, we've had to set pool size to 1 everywhere.

Managing pool size in mongos gives us the double win of parallel operations when the DB is healthy and backpressure on the application when it isn't.

Adam

Comment by Kelsey Schubert [ 13/Jul/16 ]

Hi dmitry.ryabtsev,

Thanks for the improvement request. I'm marking this ticket to be considered during the next round of planning.

Kind regards,
Thomas

Generated at Thu Feb 08 04:08:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.