[SERVER-25027] Configurable connection pools size for mongos Created: 13/Jul/16 Updated: 08/Jan/24 Resolved: 07/Nov/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Networking, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.11, 3.4.0-rc3 |
| Type: | Improvement | Priority: | Critical - P2 |
| Reporter: | Dmitry Ryabtsev | Assignee: | Mira Carey |
| Resolution: | Done | Votes: | 3 |
| Labels: | code-only | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Backport Completed: | |||||||||||||||||||||||||||||||||||||
| Sprint: | Platforms 2016-10-10, Platforms 2016-10-31, Platforms 2016-11-21 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||
| Description |
|
Issue Status as of Apr 06, 2017 ISSUE DESCRIPTION TaskExecutorPoolSize
ShardingTaskExecutorPoolHostTimeoutMS
ShardingTaskExecutorPoolMaxSize
ShardingTaskExecutorPoolMinSize
ShardingTaskExecutorPoolRefreshRequirementMS
ShardingTaskExecutorPoolRefreshTimeoutMS
Original descriptionIt would be nice if users could configure a outbound connection pools in a way as most of the drivers work. The use case is to do throttling of the amount of connections that hit the shards on the mongos level - e.g. if a number of requestes/active connections to mongos exceeds the specified threshold, some connections will have to wait for their turn to be served. Implementing this would mean that we need to expose the standard connection pool-related options such as:
Given that mongos is based on a C++ driver I presume that should be doable. — Note on the implementation of this ticketTo clarify, this commit exposes the connection pooling options that currently exist in mongos, rather than driver equivalent options. Added knobs include:
|
| Comments |
| Comment by Githook User [ 17/Nov/16 ] |
|
Author: {u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}Message: Export server parameters for sharding connection pool Not cherry-picked |
| Comment by Githook User [ 15/Nov/16 ] |
|
Author: {u'username': u'Machyne', u'name': u'Matt Cotter', u'email': u'matt.cotter@mongodb.com'}Message: |
| Comment by Githook User [ 14/Nov/16 ] |
|
Author: {u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}Message: (cherry picked from commit 5a3a0afc8f27f38c2cd6a455316e5c65a94429fd) |
| Comment by Githook User [ 14/Nov/16 ] |
|
Author: {u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}Message: Export server parameters for sharding connection pool (cherry picked from commit b6c29702d8dcadb6c1ee90a876fac4117e0ca062) |
| Comment by Githook User [ 07/Nov/16 ] |
|
Author: {u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}Message: |
| Comment by Githook User [ 07/Nov/16 ] |
|
Author: {u'username': u'hanumantmk', u'name': u'Jason Carey', u'email': u'jcarey@argv.me'}Message: Export server parameters for sharding connection pool |
| Comment by Adam Flynn [ 04/Nov/16 ] |
|
Thanks Waley - appreciate it! |
| Comment by Waley Chen [ 04/Nov/16 ] |
|
Hey Adam, We're reviewing one last thing before merging it into master and Waley |
| Comment by Adam Flynn [ 03/Nov/16 ] |
|
Saw this got merged in - thanks Waley! Any chance we can get a 3.2 backport? Adam |
| Comment by Adam Flynn [ 03/Oct/16 ] |
|
Hi Waley, Exposing the new connection pool should work well enough for my use-case. Thanks. |
| Comment by Waley Chen [ 03/Oct/16 ] |
|
Hi Adam, We have two connection pools, a new one and the legacy one which we intend to deprecate by 3.5. Thanks, |
| Comment by Adam Flynn [ 14/Jul/16 ] |
|
For a little context on use-case: we have an application with ~20k processes talking to a MongoDB cluster. On average, each process uses <1 connection (average ~0.25-0.5 of a connection), but it's sometimes useful to burst to a few operations in parallel. Without pool size limits in mongos, a client pool size limit of 3 is 60k connections. When the DB slows down, the connection pool pressure causes us to go from a steady state of 10k connections to 30-60k connections very quickly, making a bad situation worse. Because of this, we've had to set pool size to 1 everywhere. Managing pool size in mongos gives us the double win of parallel operations when the DB is healthy and backpressure on the application when it isn't. Adam |
| Comment by Kelsey Schubert [ 13/Jul/16 ] |
|
Hi dmitry.ryabtsev, Thanks for the improvement request. I'm marking this ticket to be considered during the next round of planning. Kind regards, |