Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63045

Drop pooled connections to nodes no longer in sharded cluster after a topology change

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Cluster Scalability
    • 4

      After a repl set reconfig, drop pooled connections to the removed node.

      When a server leaves a sharded cluster, drop pooled connections to and from the removed server. 

      Right now, if a server is removed, hostname resolution is changed, and the server is added back with a new IP, pooled connections to the server that performed hostname resolution before the removal will be broken. The cluster will encounter errors when attempting to use those connections.

      Worse, if there is now a different MongoDB server at the original IP, then connections that were pooled before the removal will think they point at one MongoDB server when they really point to a different one (e.g. consider serverA with hostNameA. Originally, hostnameA resolves to IP_A. Then serverA is removed, and hostname resolution is changed so that hostnameA points to IP_B. Then serverA is re-added to the cluster, and serverB is started at IP_A. Connections that were pooled to serverA before it was removed will have already resolved hostNameA to IP_A, which is incorrect after it is removed. They will actually be talking to serverB). 

      Currently, we need to either wait for connection pools to age out or manually issue a dropConnections command on all cluster members to avoid this issue. If a user doesn't do this, they will encounter the above issues. We should automatically drop connections to and from removed nodes when we notice a topology change. 

       

      Note that SERVER-36417 completed this work for replica sets; i.e. nodes in a replica set that are removed from the config will have connections dropped as part of the reconfig process. This ticket tracks completing the work for sharded clusters/connections outside replica sets (i.e.shard to shard and shard to mongos) 

       

            Assignee:
            backlog-server-cluster-scalability [DO NOT USE] Backlog - Cluster Scalability
            Reporter:
            george.wangensteen@mongodb.com George Wangensteen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: