[CSHARP-4236] connecting to mongo instance 300+ times result with error: System.outofmemory exception Created: 28/Jun/22  Updated: 27/Oct/23  Resolved: 29/Jun/22

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: 2.15.0, 2.15.1, 2.16.0, 2.16.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: rgyu N/A Assignee: Dmitry Lukyanov (Inactive)
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Quarter: FY22Q3

 Description   

Summary

I have a C# program, basically, it connects to all mongo instances, and get the version information. to ensure that the mongodb version is as expected. The program works smoothly for driver version 2.14.1 for long time. Recently, I upgrade Mongo C# driver version from 2.14.1 to 2.15.0 and above, and program throw error below:

 

processing:    328 of 452
error: System.OutOfMemoryException:  type is “System.OutOfMemoryException”
   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(StackCrawlMark& stackMark)
   at System.Threading.Thread.Start()
   at MongoDB.Driver.Core.Servers.RoundTripTimeMonitor.Start()
   at MongoDB.Driver.Core.Servers.ServerMonitor.Initialize()
   at MongoDB.Driver.Core.Servers.DefaultServer.InitializeSubClass()
   at MongoDB.Driver.Core.Servers.Server.Initialize()
   at MongoDB.Driver.Core.Clusters.MultiServerCluster.Initialize()
   at MongoDB.Driver.ClusterRegistry.CreateCluster(ClusterKey clusterKey)
   at MongoDB.Driver.ClusterRegistry.GetOrCreateCluster(ClusterKey clusterKey)
   at MongoDB.Driver.MongoClient..ctor(MongoClientSettings settings)
   at MongoDB.Driver.MongoClient..ctor(MongoUrl url)
   at Ctrip.DBConfig.Business.MongoCheckVersion.check()

 

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

2.15.0,  2.15.1, 2.16.0 and 2.16.1  work smoothly for 2.14.1 and below.

How to Reproduce

 

            List<MongoClusterInfo> MongoClusterList = SQLMonitorV3.getMongoClusterList();
            for (int i = 0; i < MongoClusterList.Count; i++)
            {
                string cluster_name = MongoClusterList[i].cluster_name;
                string env_type = MongoClusterList[i].env_type;
                string cluster_type = MongoClusterList[i].cluster_type;
                int status = MongoClusterList[i].status;
                int is_dr = MongoClusterList[i].is_dr;

                
                string user = "username";
                string pwd = "password";
                string connString = "mongodb://" + user + ":" + pwd + "@";

                Console.WriteLine("processing:" + "\t" + i.ToString() + " of " + MongoClusterList.Count.ToString());

                if (cluster_type.Equals("sharding"))
                {
                    List<MongoInstanceInfo> MongoInstanceList = SQLMonitorV3.getMongoInstanceList(cluster_name + "mongos");
                    for (int j = 0; j < MongoInstanceList.Count; j++)
                   

{                         string mongos_ip = MongoInstanceList[j].ip_business;                         string mongos_port = MongoInstanceList[j].port;                         connString += mongos_ip + ":" + mongos_port + ",";                     }

                    connString = connString.Substring(0, connString.Length - 1);
                    connString += "/?authSource=admin";

                }
                else
                {

                    List<MongoInstanceInfo> MongoInstanceList = SQLMonitorV3.getMongoInstanceList(cluster_name);
                    for (int j = 0; j < MongoInstanceList.Count; j++)
                   

{                         string machine_name = MongoInstanceList[j].machine_name;                         string ip_business = MongoInstanceList[j].ip_business;                         string port = MongoInstanceList[j].port;                         string role = MongoInstanceList[j].role;                         string mongodb_version = MongoInstanceList[j].mongodb_version;                         if (role.ToLower().Equals("arbiter")) continue;                         if (role.ToLower().Equals("mongos")) continue;                         connString += ip_business + ":" + port + ",";                     }

                    connString = connString.Substring(0, connString.Length - 1);
                    connString += "/admin" + "?replicaSet=" + cluster_name;
                }
                
                try
                {
                    var mongourl = new MongoUrl(connString);
                    MongoClient client = new MongoClient(mongourl);
                    var db = client.GetDatabase("admin");
                    BsonDocument result = db.RunCommand<BsonDocument>(new BsonDocument("buildInfo", 1));
                    string version = (string)result.GetValue("version");
                    string allocator = (string)result.GetValue("allocator");

                    if (!(version.Equals("4.0.19")  || 
                        version.Equals("4.2.17") ||
                        version.Equals("4.2.19") ||
                        version.StartsWith("4.2.") ||
                        version.Equals("4.2.18")))
                   

{  Console.WriteLine("expected");                         continue;                     }

                    if (!allocator.Equals("tcmalloc"))
                   

{  Console.WriteLine("expected");                     }

                    
                }
                catch (Exception e)
               

{                     Console.WriteLine("error: "  + e.ToString());                     break;                 }

            }//end of for

 

Steps to reproduce. If possible, please include a Short, Self Contained, Correct (Compilable), Example.

 

  1. ensure we have more than 400 mongo instances to check.
  2. Run the above program, it will throw out error when in around iteration 330.
  3. Only the iteration number counts. I skip some iterations, and it will hit error later
  4. Note, once I revert driver back to 2.14.1. it works fine.

 

Additional Background

Please provide any additional background information that may be helpful in diagnosing the bug.



 Comments   
Comment by rgyu N/A [ 29/Jun/22 ]

thanks. I changed to use ClusterRegistry.Instance.UnregisterAndDisposeCluster(client.Cluster);, and observe the memory usage with Visual Studio diagnostic sessions.  Now the memory usage is flat. Previsouly, it is sharp increasing. thanks again, and the problem is resolved. 

Comment by Dmitry Lukyanov (Inactive) [ 29/Jun/22 ]

I'm not sure what you mean by posted code above. That code is related to the legacy driver. You didn't use it initially.
 

MongoClient client = new MongoClient(mongourl);
processing code...
client.Cluster.Dispose(); 

this is not enough. You also should unregister disposed cluster. You should use:

ClusterRegistry.UnregisterAndDisposeCluster(mongoClient.Cluster);  

Better if we can have an explict Open/Close like other database client drivers though.  Thank you very much for your suggestions.

we're planning to make our client disposable, but this will be a significant breaking change, so it will be done only in next major driver release, see this ticket for updates

 

 

Comment by rgyu N/A [ 29/Jun/22 ]

thank you very much for your response,  It seems the application reaches the server limit. Somehow I run the application, and hit the limitation below. the default _maxServerCount is 100. 

 

        internal static MongoServer Create(MongoServerSettings settings, IOperationExecutor operationExecutor)
        {
            lock (__staticLock)
            {
                MongoServer server;
                if (!__servers.TryGetValue(settings, out server))
                {
                    if (__servers.Count >= __maxServerCount)
                   

{                         var message = string.Format("MongoServer.Create has already created \{0}

servers which is the maximum number of servers allowed.", __maxServerCount);
                        throw new MongoException(message);
                    }
                    server = new MongoServer(settings, operationExecutor);
                    __servers.Add(settings, server);
                }
                return server;
            }
        }

 

I changed the program, and add the below code after each usage, the the patrol program runs fine again. 

 

var mongourl = new MongoUrl(connString);
MongoClient client = new MongoClient(mongourl);

processing code...

client.Cluster.Dispose();

 

Better if we can have an explict Open/Close like other database client drivers though.  Thank you very much for your suggestions. 

 

 

 

Comment by Dmitry Lukyanov (Inactive) [ 28/Jun/22 ]

Hey rgyu@trip.com,

I cannot reproduce this issue on my windows x64 machine. Here some notes that might bring some light on your issue.
1. The difference between 2.14.1 and 2.15 versions is that now we create (2 * Number of server endpoints) internal background thread instances for each mongo client where previously we used TPL tasks for this target. It was done in this way to handle some issues with suppressing execution context we had with the previous implementation.
Assuming that all your connection strings are different, effectively it means even for simple standalone server with single endpoint, in your 328 attempts you created 328 * 2 = 656 threads that ping server in addition to buildInfo requests you're doing in your application which somehow causes out of memory exception on your machine. If your server is replica set with 3 nodes, then a number of internal background threads for each mongo client will be 2 * 3 = 6.
2. If your application uses x32-bit, switching to x64 should help with it.
3. Although we do not usually recommend it, you may try using the below approach to release memory after each particular client that also should resolve your issue:

ClusterRegistry.UnregisterAndDisposeCluster(mongoClient.Cluster);

Generated at Wed Feb 07 21:47:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.