[CSHARP-4064] Connection TCP ports are not free'd Created: 17/Feb/22  Updated: 27/Oct/23  Resolved: 09/Sep/22

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 2.13.3
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Dennis Hoefakker Assignee: James Kovacs
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive MongoTcpConnectionTest.zip    

 Description   

Summary

We upgraded to MongoDB c# driver 2.13.3 and noticed a huge increase in open ports. There doesn't seem to be a "cleanup" / "free" action running. This means the ports are only growing. And since windows has like a port limit (around 16K) that limit is reached after a few days

When analyzing, I see that ports are kept in state "Established", and never freed.

After downgrading to 2.10.3, I see that ports are used in the same way. But the biggest difference is that in 2.10.3 I see that ports are "freed" (put on Time Waiting) and then after some time freed to the system.

 __ 

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

How to Reproduce

You can reproduce it with a simple test app that needs to be in waiting mode so the connection idle event can close the connections. 

Use a tool like CurrPorts to see the port usage of the application. Switching the versions you will see the difference right away.

Additional Background

For us this is quite important since we are using a multi tenant database setup, where each customer has it's own database with credentials. This means we have a lot of connections (since the drivers creates a pool per clusterkey/database/username/password combination



 Comments   
Comment by PM Bot [ 09/Sep/22 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information.

Comment by James Kovacs [ 25/Aug/22 ]

Hi, dennis@valueblue.nl,

Thank you for the additional details regarding your multi-tenant setup.

As you note, connections are pooled based on the cluster key, which includes the authenticating user. Connection establishment involves many steps including TCP socket setup, TLS handshake, and authentication. Authentication is relatively expensive by design (involving many rounds of credential hashing) to discourage brute force attacks. Thus pooling connections irregardless of authenticating user and re-authenticating before every operation would increase the operation time for every MongoDB operation.

Regarding the monitoring threads used by the driver. MongoDB implemented streaming heartbeats in MongoDB 4.4 to reduce failover times during elections. This was implemented by the driver in 2.11 (CSHARP-2964) as Boris mentioned earlier. Using the new streaming heartbeats, a driver establishes a monitoring connection and then waits for push notifications of topology changes from the cluster members. The problem is that we have no way to measure round-trip time to cluster members, which is required when making server selection decisions. Thus a second monitoring thread (the RTT thread) is established which measures the round-trip time to the cluster member. That is why you saw the doubling of idle monitoring threads when moving from 2.10 (which doesn't support streaming heartbeats) to 2.13 (which does). Unfortunately there is no way to disable this behaviour in newer drivers.

We are planning to make MongoClient disposable in the 3.0.0 release (CSHARP-3431) but that release is a ways off and plans/priorities may change. A truly disposable MongoClient is likely the best solution for a multi-tenant situation such as yours as all resources associated with the MongoClient will be properly cleaned up and you can implement some sort of MongoClient caching mechanism based on your tenants' usage statistics.

In the meantime we do use a disposable MongoClient in our tests. See DisposableMongoClient in MongoDB.Driver.TestHelpers. I wouldn't recommend using this class directly, but more as an example of how the Dispose method cleans up resources - notably ClusterRegistry.Instance.UnregisterAndDisposeCluster(mongoClient.Cluster). I will admit that this method was designed with testing purposes in mind and not production. I would strongly recommend testing appropriately if you are going to use it in your multi-tenant application.

Hopefully this provides you with some guidance and ideas of how to address this issue both now and in the future.

Sincerely,
James

Comment by Dennis Hoefakker [ 17/Aug/22 ]

Hi @Boris,

Sorry for the late reply, this issue slipped, also caused we reverted back to driver 2.10.3.

You remark for "MongoClient per cluster" is something we already do. But our multi-tenant setup enforces us to have a user/password per database. And since the MongoClient is unique per server/database/user/password this causes a huge connection pool(s).

Is there a way to tell the new driver to use one connection, or tell the driver to free connections to the OS quicker. I also have an other bug  CSHAP-3250 which explains.

Comment by PM Bot [ 22/Mar/22 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to comment on this if you're able to provide more information.

Comment by Boris Dogadov [ 04/Mar/22 ]

Hello dennis@valueblue.nl

The described behavior is expected. Each MongoClient maintains two open connections for each server, in addition to the connections used by operations. This change was introduced in 2.11 CSHARP-2964, while in previous versions MongoClient maintained one connection per server.

For better performance, it’s highly recommended having a single MongoClient per cluster across your application.

Comment by Boris Dogadov [ 02/Mar/22 ]

Thank you dennis@valueblue.nl,
We will try to reproduce this issue.

Comment by Dennis Hoefakker [ 02/Mar/22 ]

Hi,

 

Sorry for the delay but i enclosed a simple example. I cleaned the solution's packages folder so you need to restore nuget packages and other refs using the following command:

Update-Package Microsoft.CodeDom.Providers.DotNetCompilerPlatform -r

 

When running the project, supply the connection string in the class "TenantMongoClientLocator" I didn't bother putting it in config files

trigger a DocumentCount using the request:

https://localhost:44333/api/test?databaseName=%YourDatabasename%&collectionName=%YourCollectionname%

 

What you will notice (after triggering the command above )using currPorts is that running 2.10.3 you see 2 connections to the db server, and after a while it's reduces to 1. When running the 2.13.3 you'll see 3 connections to the db server reducing to 2. Which is double open connections.

As stated before we are using multiple databases per customer with own credentials, running multiple microservices on a VM, where the micro's can have connections to the database.

MongoTcpConnectionTest.zip

Comment by Dennis Hoefakker [ 25/Feb/22 ]

Yes i'll set up an example, was a bit busy this week but will work on it next week!

Comment by Boris Dogadov [ 18/Feb/22 ]

Hi dennis@valueblue.nl , thank you for your bug report.

I failed to reproduce this issue with the following code:

 

var mongoUrl = MongoUrl.Create(@"mongodb://localhost:27017");
var clientSettings = MongoClientSettings.FromUrl(mongoUrl);
clientSettings.MaxConnectionLifeTime = TimeSpan.FromSeconds(60); // 5
clientSettings.MinConnectionPoolSize = 100; // 10
clientSettings.ClusterConfigurator += b =>
{
  b.ConfigureConnectionPool(s => s.With(maintenanceInterval: TimeSpan.FromSeconds(10)); // 60, 120
};
 
var client = new MongoClient(clientSettings); 
await Task.Delay(TimeSpan.FromMinutes(120));

I have tried 2.13.3 and 2.14.1 drivers, with various heartbeat intervals, max connection lifetimes and pool sizes. Running the sample for an hour, I can see that the connections are established and freed as expected via CurrPorts .

To investigate further it would be helpful to receive a self-contained sample code which reproduces the issue.

Thanks.

Generated at Wed Feb 07 21:47:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.