[CSHARP-4850] MongoDB.Driver.MongoWaitQueueFullException Created: 20/Nov/23  Updated: 13/Dec/23  Resolved: 13/Dec/23

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 2.18.0
Fix Version/s: None

Type: Improvement Priority: Unknown
Reporter: Mohamed BOUZIDI Assignee: Adelin Mbida Owona
Resolution: Works as Designed Votes: 0
Labels: external-user, performance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2023-11-20-11-51-31-616.png    
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

When the number of threads that trie to execute mongodb queries using MongoDb.Driver 2.18, we start to have some threads that hangs for a while (some seconds to 1 minute), then a lot of errors are logged which all point aout the same issue:

2023-11-16 12:05:53.579 +01:00 [ERR] An error occured while publishing message Domain.Events.IDomainEvent
MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server localhost:27017 is full.
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireWaitQueueSlot()
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.StartCheckingOut()
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.AcquireConnectionAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at Compliance.Wastewater.Infrastructure.PlantAnalysisReports.PlantAnalysisReportRepository.GetPlantAnalysisReportPersistentModel(Guid plantId, DateTime samplingStartDate, Guid finalityTypeId)

 

I tried to set Min and Max connection pool size to different values 100 to 500 but didn't resolve the issue:

I tried also to profile the application to find any long queries but I could not find out any. All the times I did it, The first thing that take time based on DotTrace profiler is the Acquire method of the connection pool (I will add some pictures)



 Comments   
Comment by Mohamed BOUZIDI [ 13/Dec/23 ]

The issue is fixed now after limiting the number of operations in //. Thank you for your help

Comment by Mohamed BOUZIDI [ 24/Nov/23 ]

adelin.mbidaowona@mongodb.com Clear for me, thank you again

Comment by Adelin Mbida Owona [ 22/Nov/23 ]

Hi mohamed.bouzidi.90@gmail.com, creating different instances of MongoClient per thread is not advisable. A MongoClient instance is thread safe and you will only need one instance of MongoClient even with multiple threads. 

When you instantiate a MongoClient, there are a variety of comparatively expensive operations that must be performed including DNS resolution (including TXT and SRV lookups if you are using mongodb+srv://), creating connection pools for each cluster member, establishing monitoring connections to each cluster member, etc. Much of this is done in the background, but it is the reason why we recommend creating your MongoClient instance during application bootstrapping and reusing that instance throughout the lifetime of the application. However, If you are connecting to multiple MongoDB clusters with different connection strings, you would have to create one MongoClient instance per cluster.

Comment by Mohamed BOUZIDI [ 22/Nov/23 ]

adelin.mbidaowona@mongodb.com Thank you for the full explanation. I will try to limit the number of concurrent threads that needs to connect to mongodb and let you know if that fixes the issue for us.

How about creating different instances of MongoClient without limiting the number of // threads, is it a bad thing to do ?

Comment by Adelin Mbida Owona [ 21/Nov/23 ]

Hi mohamed.bouzidi.90@gmail.com, I understand that you are experiencing MongoWaitQueueFullExceptions. This exception is not necessarily a bug. It just indicates that too many threads are trying to use MongoDB at once.

By default the connection pool has 100 connections. Threads pull connections from the connection pool and return them when they are done. Once all connections are in use, any further threads that want a connection must wait for some other thread to return a connection to the pool. The driver limits the number of threads that are allowed to wait for a connection at the same time. The default wait queue size is the WaitQueueMulitple (default 5) times the MaxPoolSize (default 100). Thus a default wait queue size of 500.

Also to help prevent connection storms, the maxConnecting setting was introduced, which limits the number of concurrent connection establishment requests to a cluster node. The default value is 2.

If you are not at maxPoolSize but are seeing MongoWaitQueueFullExceptions, you may be slow to establish new connections. 

The solution is to either decrease the number of threads trying to use MongoDB at the same time (decrease the degree of parallelism) or to configure the connection pool to have either more connections or a higher wait queue limit, or both.

You can also try increasing maxConnecting either via the connection string or via MongoClientSettings.MaxConnecting. I would suggest trying 4 (e.g. double the default value) and gradually increasing from there to see if it resolves the issue.

Hopefully these suggestions help solve your problem but if none of these work, this maybe a problem we would like to reproduce.

Comment by PM Bot [ 20/Nov/23 ]

Hi mohamed.bouzidi.90@gmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Wed Feb 07 21:49:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.