Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-25027

Configurable connection pools size for mongos

    • Fully Compatible
    • Platforms 2016-10-10, Platforms 2016-10-31, Platforms 2016-11-21

      Issue Status as of Apr 06, 2017

      ISSUE DESCRIPTION
      MongoDB 3.2.11 and 3.4.0 introduce a set of configurable parameters in mongos to help manage NetworkInterfaceASIO-based connection pools in sharded clusters. These parameters are:

      TaskExecutorPoolSize
      • The number of NetworkInterfaceASIO-based connection pools to use for a given mongos
      • Defaults to 4 <= NUM_CORES <= 64
      • When tuned down, this will reduce the number of open connections in the system at some performance cost under a heavier load
      • A good default is generally either the actual default, or <16 if the the node running mongod has a large number of cores
      ShardingTaskExecutorPoolHostTimeoutMS
      • How long will mongos wait before dropping all connections to a host that it hasn't communicated with
      • Default is 5 minutes
      • Turning this up will prevent a mongos from entirely severing a connection to a mongod
      • Longer durations will smooth connection spikes. Setting this to 3-4x the period of the connection spikes will generally prevent completely cold starts
      ShardingTaskExecutorPoolMaxSize
      • The maximum number of connections to hold to a given mongod at any time.
      • Default is unlimited
      • Giving this is a maximum value, may be useful to set a limit for this value if some mongod nodes in the system are suffering from connection floods
      • Remember that setting this effectively limits the number of simultaneous operations from a mongos to a mongod at PoolMaxSize * TaskExecutorPoolSize
      ShardingTaskExecutorPoolMinSize
      • The minimum number of connections to hold to a given host at any time.
      • Default is 1
      • If a system is experiencing latency issues because of cold starts, users should consider increasing this value
      • The cost is that mongos will keep these sessions open, which is a real fixed cost of TaskExecutorPoolSize * PoolMinSize * NUM_MONGOS per shard
      ShardingTaskExecutorPoolRefreshRequirementMS
      • How long will mongos wait before attempting to heartbeat an idle connection in the pool
      • Default is 1 minute
      • Setting this higher or lower will increase or decrease heartbeat traffic, which will increase/decrease idle load
      • Lower values may reduce the number of transient network failures seen in a system (because non-user operations will see them earlier and transparently re-connect)
      • It is unlikely that the default value needs to be modified
      ShardingTaskExecutorPoolRefreshTimeoutMS
      • How long will mongos wait before timing out a heartbeat
      • Default 20 seconds
      • If this value is too low, mongos won't be able to maintain connections in the pool
      • Tuning this value up may help if heartbeats actually take 20 seconds
      Original description

      It would be nice if users could configure a outbound connection pools in a way as most of the drivers work. The use case is to do throttling of the amount of connections that hit the shards on the mongos level - e.g. if a number of requestes/active connections to mongos exceeds the specified threshold, some connections will have to wait for their turn to be served.

      Implementing this would mean that we need to expose the standard connection pool-related options such as:

      • maxPoolSize
      • waitQueueMultiple
      • waitQueueTimeoutMS

      Given that mongos is based on a C++ driver I presume that should be doable.

      Note on the implementation of this ticket

      To clarify, this commit exposes the connection pooling options that currently exist in mongos, rather than driver equivalent options.

      Added knobs include:

      • ShardingTaskExecutorPoolHostTimeoutMS
      • ShardingTaskExecutorPoolMaxSize,
      • ShardingTaskExecutorPoolMinSize,
      • ShardingTaskExecutorPoolRefreshRequirementMS
      • ShardingTaskExecutorPoolRefreshTimeoutMS

            Assignee:
            mira.carey@mongodb.com Mira Carey
            Reporter:
            dmitry.ryabtsev@mongodb.com Dmitry Ryabtsev
            Votes:
            3 Vote for this issue
            Watchers:
            26 Start watching this issue

              Created:
              Updated:
              Resolved: