[CSHARP-4224] Prevent fill-in the threads queue under high load pressure (MongoWaitQueueFullException) Created: 21/Jun/22  Updated: 27/Oct/23  Resolved: 21/Jun/22

Status: Closed
Project: C# Driver
Component/s: Performance Benchmarking
Affects Version/s: 2.16.0
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Nicolas Rey Assignee: Robert Stam
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Case:

 Description   

Hi,

I had to implement an unique id generator, I chose to made it with MongoDb in C# (.net 6) using the official tutorial (https://www.mongodb.com/docs/v3.0/tutorial/create-an-auto-incrementing-field/).

I then decided to put the system under pressure, and I quickly being thrown the MongoWaitQueueFullException => the wait queue for acquiring a connection to server is full.

After searching a lot on this subject (tasks like https://jira.mongodb.org/browse/CSHARP-118 and a lot from SO), I came to the conclusion that too many threads (coming from the same client instance ?) leads to this behavior and besides possible client settings like increasing the WaitQueueSize property, which it's deprecated and that will disappear soon, it's not a long term solution.

One possibility is to manage incoming traffic, throttling threads (using SemaphoreSlim for example). It works perfectly but it's not ouf of the box from the driver side.

 

Then I tried with the exact same client parameters with a Go implementation. I expected to have the exact same kind of behavior under high load pressure, but it worked out of the box.

So my question is : what is the main difference between both C# and Go implementation that made use implementing a thread throttling in C# where in Go it's not necessary ? Is it possible to be explicit about it in the documentation with some examples for new comers in mongo C# driver ?

I create minimal reproductible samples here if you need to try by yourself :

https://github.com/NicolasREY69330/IdGenerator

 

Thank you guys for your great job



 Comments   
Comment by Robert Stam [ 21/Jun/22 ]

You're welcome!

Comment by Nicolas Rey [ 21/Jun/22 ]

Yes makes sense, I only see C# and Java having this exceptions thrown from my research on SO, I guess both implementations are close and based on this queue approach

Thank you Robert for your help

Comment by Robert Stam [ 21/Jun/22 ]

I think that the Go driver, being a newer driver, never implemented a wait queue in the first place.

Comment by Nicolas Rey [ 21/Jun/22 ]

Thanks for you quick reply.

Your approach completely makes sense to me, and I didn't understood that the queue itself will be removed.

By the way, is there a specificity on the Go driver explaining why the threads throttling is not necessary here ? I achieved really similar results from both C# (with 100 threads throttling)  and with Go. But with Go I didn't have to implement anything around the threads management, it worked out of the box under the exact same conditions.

Comment by Robert Stam [ 21/Jun/22 ]

When you have a lot more threads than you have connections in the connection pool you have high probability of getting `MongoWaitQueueFullException`.

The default size of the connection pool is 100. The default size of the connection pool wait queue is 500. So if you have over 600 threads there is always a possibility of a `MongoWaitQueueFullException`. The more threads the more likely. The more work each thread does before returning a connection to the connection pool the more likely.

The original rationale for having a wait queue is this: if you have a very long wait queue you are likely to get a whole lot of timeouts. Better to fail fast with a wait queue full exception than to fail slow (and much later) with a timeout. But our thinking on this has changed and we plan to remove the wait queue in a later release. See:

https://github.com/mongodb/specifications/blob/98db467cb8f2f7bf50e79fc563e59701a8b3e3bf/source/connection-monitoring-and-pooling/connection-monitoring-and-pooling.rst#why-are-waitqueuesize-and-waitqueuemultiple-deprecated

For now you can just configure a very large wait queue. As long as it is larger than the number of threads you have you will never get a `MongoWaitQueueFullException`.

Just to be clear, when we do remove the deprecated `WaitQueueSize` setting in a later release we will also remove the wait queue itself.

 

Generated at Wed Feb 07 21:47:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.