[CSHARP-3250] Connection not reused and keep growing Created: 05/Nov/20 Updated: 27/Oct/23 Resolved: 27/Nov/20 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | Connectivity |
| Affects Version/s: | 2.10.3 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Dennis Hoefakker | Assignee: | James Kovacs |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
c# driver in Azure VM hosting Service Fabric Cluster |
||
| Attachments: |
|
| Description |
|
We have a proces where all our tenant databases are processed. In our setup each tenant has it's own database (with own credentials). Currently we use the default of the c# driver (connection pool of 100). What i'm seeing is that the number of connections to our MongoBD server is only increasing. It looks like the driver is not reusing used connections from the pool. I use the following script
When running i see that the number of INACTIVE_CONNECTIONS is increasing.
and a little later
Is this normal, cause it feels like a bug, where a new connection (from pool) is created instead of reusing an inactive one.
|
| Comments |
| Comment by Dennis Hoefakker [ 01/Dec/20 ] |
|
James,
I kinda understand, but how come that when I call: MongoDB.Driver.ClusterRegistry.Instance.UnregisterAndDisposeCluster(mongoClient.Cluster) The connections are closed? The drivers seems to do the correct thing (at least that what the event API is telling me). Can I monitor anything server side? Or client side, to really see the "FIN -> ACK+FIN -> ACK" is executed? Dennis
|
| Comment by James Kovacs [ 27/Nov/20 ] |
|
Hi, Dennis, Thank you for confirming that the connection pools are working as expected from the drivers side. We agree that it is not recommended nor desirable to directly interact with the ClusterRegistry singleton. When a connection is closed by the driver, it closes the underlying TCP socket, which will proceed through the usual FIN -> ACK+FIN -> ACK sequence. If you are seeing TCP sockets lingering on the server, we recommend investigating further with your networking and server teams. If you have questions specific to connection handling by mongod or mongos, you may find one or more of the following support resources useful to you:
We have closed this case since this is not a bug in the .NET/C# driver. Sincerely, |
| Comment by Dennis Hoefakker [ 24/Nov/20 ] |
|
James, I ran some more checks, i do see:
I added a counter, and i see it maches the tenants i'm requesting data from, and reducing on connection remove. So from driver side it looks good (Y), BUT i don't see that reflected on server side (there i still see inactive connections) When i call MongoDB.Driver.ClusterRegistry.Instance.UnregisterAndDisposeCluster(mongoClient.Cluster) and i check the server the connections are removed/disposed. I really don't want to use it, cause it's not supported. Can this action be triggered after for example x time of inactivity automatically by the driver? Beside this, is it smart to do a short zoom call? maybe i'm misunderstandig, and we can allign things.
|
| Comment by James Kovacs [ 21/Nov/20 ] |
|
Hi, Dennis, Thank you for the update. In the .NET/C# driver, connection pools are shared between MongoClient instances with the same cluster key. It is for this reason that we do not provide a convenient way to dispose of MongoClient instances - the underlying connection pools may be shared between instances. Connection instances from a connection pool should be re-used and typically are re-used across operations. While an operation is inflight, that connection instance is checked out of the pool and is not available for re-use until it is returned to the pool on completion of the operation. Long-running operations tying up connections is a typical cause for increases in connection pool sizes. If you observe new connections being created while available connections are in the pool, this would be unexpected. If you observe this behaviour, please provide a repro demonstrating the problem so that we can investigate further with you. Regarding your question about closing or disposing a MongoClient... You can forcibly dispose the underlying cluster by calling MongoDB.Driver.ClusterRegistry.Instance.UnregisterAndDisposeCluster(mongoClient.Cluster). Note that this is not recommended nor supported as any use of a MongoClient using this cluster will throw a ObjectDisposedException. This includes MongoClient instances that happen to use the same cluster key. We only use this method in our test code, but it may work for you as a stopgap while we continue investigating this issue with you. Sincerely, |
| Comment by Dennis Hoefakker [ 19/Nov/20 ] |
|
James,
I ran some test, and see the connections not dropping. I do see the checking IN and OUT of connections so that seems to be working. What I also see is that connections are inactive and they stay like that. In our scenario it would be nice if the connections are freed (multiple tenants, multiple servers handling request). It looks like that the connections are freed only when the program / application is exited. What I try to do now is periodically close not used clients. I hoped that it would "free" the inactive connections (used by the client), but it doesn't. Looks like it's linked to the requesting IP. Also the Client doesn't have a .Close() / .Dispose() / .Stop() / .WhatEver() to shutdown and release resources. I ended up with null'ing the variabled but that doesn't free it. Is there another way to do this? Is there a hidden variable/option?
Cheers, Dennis |
| Comment by Dennis Hoefakker [ 14/Nov/20 ] |
|
James,
Thanks for the detailed description
Cheers, Dennis |
| Comment by James Kovacs [ 12/Nov/20 ] |
|
Hi, Dennis, Thank you for your followup. Your understanding is mostly correct. Let me clarify some terminology... When you create a MongoClient with a connection string and MongoClientSettings, we create a ClusterKey to identify a particular set of connection pools to use for that MongoClient. Things like authenticated user, timeouts, TLS/SSL configuration, and more are all part of the ClusterKey. Changing any of these options will result in a different set of connection pools within your application. When you perform an operation against your cluster, the driver performs server selection to decide which node to send a particular operation to. If you are performing a write or a read with primary read preference, then that operation will go to the primary. If you are performing a read from a secondary or nearest, then other factors come into play such as round-trip time to the nodes in the cluster. Once a node has been selected, we move into connection pooling. We look in the correct connection pool for that node (based on the ClusterKey) and attempt to check out a connection. If a connection is available, we use it. If a connection is not available, then we see if we are at maxPoolSize. If we are at maxPoolSize, we block waiting for an available connection. (This gets into wait queues and timeouts, which I won't go into details about.) If we are not at maxPoolSize, we open a new connection to this node, add it to the pool, and check it out for that operation. Once an operation is complete, that connection is returned to the pool for reuse. If you are seeing your connection count growing, we would suggest ensuring that your ClusterKey}}s (e.g. connection string and {{MongoClientSettings) are identical across calls. Also ensure that operations are complete and thus connections returned to their respective pools prior to the next operation. We provide an Eventing API that includes events for when connection pools are created, when a new connection is added to a pool, when a connection is checked out, when it is returned, etc. This should assist you in investigating when connections are created, destroyed, checked in/out of pools, etc. Please let us know if you have further questions. Sincerely, |
| Comment by Dennis Hoefakker [ 10/Nov/20 ] |
|
James,
Thanks for the answer. I kinda understand it, but still it seems to not be working ok. I understand each database connection creates a client and a corresponing connection pool, what i now understand is that there is a connectionpool for each of the nodes in the cluster. So when i trigger a action which checks each database, i understand that i get alot of connections, lot's of them are "inactive" after first run. When i then trigger a new run (so same databases / nodes) then i see the connections growing (inactive) where i expected the driver/client to reuse an inactive connection before opening a new one (from pool) and making that one inactive after usage. It seems that the driver opens a new connection from the pool (until the maxPoolSize) before reusing a closed/inactive connection?
I hope you understand what i mean, Dennis |
| Comment by James Kovacs [ 09/Nov/20 ] |
|
Hi, Dennis, Thank you for reaching out to us and reporting this issue. We understand that you have multiple databases, which are accessed with different credentials, and you are concerned that the connection pools are not reusing connections. Connection pools amortize the cost of connection setup by maintaining and reusing idle connections. The cost of connection setup not only includes the TCP socket and TLS/SSL handshake, but also authentication with the cluster nodes. If you change the authenticated user, you implicitly create another set of connection pools. Each connection pool is for a specific node in the cluster with a specific set of connection string parameters. So if you have a 3-node cluster and connect with 10 different users, you will have 30 connection pools each with potentially maxPoolSize (default 100) connections. Hopefully this explains the behaviour that you are observing. Common solutions include:
Please let us know if you have any additional questions. Sincerely, |
| Comment by Dennis Hoefakker [ 05/Nov/20 ] |
|
I see the script is not well formated db.currentOp(true).inprog.reduce( else { accumulator[ipaddressINActiveConnection] = (accumulator[ipaddressINActiveConnection] || 0) + 1; accumulator["INACTIVE_CONNECTION_COUNT"]++; } return accumulator;
I just reran with MaxConnectionPoolsize set to 10, but it's still growing
Could it perhaps be a server issue, where it's not cleaning up inactive connections ? |