[CSHARP-2055] Sudden burst of "Exception = 'MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection." Created: 05/Oct/17  Updated: 27/Oct/23  Resolved: 13/Oct/17

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Manjunath Bhuyar Assignee: Robert Stam
Resolution: Works as Designed Votes: 0
Labels: question
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File cluster health.PNG    

 Description   

Hi,

Our ecosystem uses MongoDB as preferred storage engine (we are running 3.2 Enterprise version server) and c# driver 1.11. From last two days our system is throwing exceptions to establish connection
And it says "MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection."

We are using all default settings for establishing the connection.

After googling for a while, people suggested to increase the "MaxConnectionPoolSize" & "WaitQueueSize" settings, what are the ideal settings?

Regards
Manju

[Exception = 'MongoDB.Driver.MongoConnectionException: Too many threads are already waiting for a connection.
Server stack trace:
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(AcquireConnectionOptions options) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\MongoConnectionPool.cs:line 97
at MongoDB.Driver.MongoServerInstance.AcquireConnection() in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\MongoServerInstance.cs:line 465
at MongoDB.Driver.MongoServer.AcquireConnection(ReadPreference readPreference) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\MongoServer.cs:line 964
at MongoDB.Driver.MongoCursor`1.MongoCursorConnectionProvider.AcquireConnection() in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\MongoCursor.cs:line 1049
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 112
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 77
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)



 Comments   
Comment by Manjunath Bhuyar [ 06/Oct/17 ]

Hi Robert,

Found the issue, it was bad indexing on our part, our couple of collections were running on peak with multiple queries running very very slow. That made the server super slow to respond. Connection rates were running in 2.7K range, after applying proper index everything came back to normal and now connections are under 250

Thank you for the support and we can close the case.

Regards
Manju

Comment by Manjunath Bhuyar [ 06/Oct/17 ]

Attached is the cluster monitoring graphs, which show a sudden spike in activity in the last two days

Comment by Manjunath Bhuyar [ 06/Oct/17 ]

Hi Robert,

No luck with the setting of Poolsize to 300, still seeing connection failures. And the connection issues are happening on only one Database on the cluster and other DB's are having a different error
our ops person is checking the cluster for any issues in parallel

-Manju

Connection issue Stack trace
********************************************************
Server stack trace:
at MongoDB.Driver.Internal.MongoConnectionPool.AcquireConnection(AcquireConnectionOptions options) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\MongoConnectionPool.cs:line 161
at MongoDB.Driver.MongoServerInstance.AcquireConnection() in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\MongoServerInstance.cs:line 465
at MongoDB.Driver.MongoServer.AcquireConnection(ReadPreference readPreference) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\MongoServer.cs:line 964
at MongoDB.Driver.MongoCursor`1.MongoCursorConnectionProvider.AcquireConnection() in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\MongoCursor.cs:line 1049
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 112
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 77
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at HP.Sds.DeviceDataProcessor.Dal.DbGroup.ConsumableMongoDbGroup.GetActiveConsumablesForDevice(Int64 jamCustomerId, Int64 jamDeviceId) in C:\Agent_work\12\s\DeviceDataProcessor\src\Dal\DbGroup\ConsumableMongoDbGroup.cs:line 54

ReadPreference issue stack trace
*************************************************************
Exception = QueryFailure flag was could not find host matching read preference { mode: "primary", tags: [ {} ] } for set shard1 (response was { "$err" : "could not find host matching read preference { mode: \"primary\", tags: [ {} ] } for set shard1", "code" : 133 })., Stack Trace =
Server stack trace:
at MongoDB.Driver.Internal.MongoReplyMessage`1.ReadHeaderFrom(BsonBuffer buffer) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\Messages\MongoReplyMessage.cs:line 123
at MongoDB.Driver.Internal.MongoConnection.ReceiveMessage[TDocument](BsonBinaryReaderSettings readerSettings, IBsonSerializer serializer, IBsonSerializationOptions serializationOptions) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Communication\MongoConnection.cs:line 295
at MongoDB.Driver.Operations.QueryOperation`1.GetFirstBatch(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 119
at MongoDB.Driver.Operations.QueryOperation`1.Execute(IConnectionProvider connectionProvider) in d:\jenkins\workspace\mongo-csharp-driver-1.x-build\MongoDB.Driver\Operations\QueryOperation.cs:line 77
at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
at HP.Sds.ActionService.Dal.DbGroup.DeviceMongoDbGroup.AddDevice(Device entity) in C:\Agent_work\27\s\ActionService\src\Dal\DbGroup\DeviceMongoDbGroup.cs:line 26

Comment by Manjunath Bhuyar [ 05/Oct/17 ]

Thank you Robert, we will try your suggestion and double the values.

Regards
Manju

Comment by Robert Stam [ 05/Oct/17 ]

There is no single setting that is ideal for everyone.

You want MaxConnectionPoolSize to be big enough to handle spikes in load without:

1. Allowing too many connections to be opened (overloading the server with connections)
2. Having to wait too long for a connection to be released

You want WaitQueueSize to be big enough to handle spikes in load without:

1. Throwing an exception because the wait queue is full

So the ideal setting will depend on your load.

Given that you are seeing these exceptions we can deduce that your load has grown to the point where your current settings are no longer adequate.

Perhaps the easiest thing is to double each of the values and see whether that prevents further exceptions.

Generated at Wed Feb 07 21:41:29 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.