[CSHARP-4224] Prevent fill-in the threads queue under high load pressure (MongoWaitQueueFullException) Created: 21/Jun/22 Updated: 27/Oct/23 Resolved: 21/Jun/22 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | Performance Benchmarking |
| Affects Version/s: | 2.16.0 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Nicolas Rey | Assignee: | Robert Stam |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Case: | (copied to CRM) |
| Description |
|
Hi, I had to implement an unique id generator, I chose to made it with MongoDb in C# (.net 6) using the official tutorial (https://www.mongodb.com/docs/v3.0/tutorial/create-an-auto-incrementing-field/). I then decided to put the system under pressure, and I quickly being thrown the MongoWaitQueueFullException => the wait queue for acquiring a connection to server is full. After searching a lot on this subject (tasks like https://jira.mongodb.org/browse/CSHARP-118 and a lot from SO), I came to the conclusion that too many threads (coming from the same client instance ?) leads to this behavior and besides possible client settings like increasing the WaitQueueSize property, which it's deprecated and that will disappear soon, it's not a long term solution. One possibility is to manage incoming traffic, throttling threads (using SemaphoreSlim for example). It works perfectly but it's not ouf of the box from the driver side.
Then I tried with the exact same client parameters with a Go implementation. I expected to have the exact same kind of behavior under high load pressure, but it worked out of the box. So my question is : what is the main difference between both C# and Go implementation that made use implementing a thread throttling in C# where in Go it's not necessary ? Is it possible to be explicit about it in the documentation with some examples for new comers in mongo C# driver ? I create minimal reproductible samples here if you need to try by yourself : https://github.com/NicolasREY69330/IdGenerator
Thank you guys for your great job |
| Comments |
| Comment by Robert Stam [ 21/Jun/22 ] |
|
You're welcome! |
| Comment by Nicolas Rey [ 21/Jun/22 ] |
|
Yes makes sense, I only see C# and Java having this exceptions thrown from my research on SO, I guess both implementations are close and based on this queue approach Thank you Robert for your help |
| Comment by Robert Stam [ 21/Jun/22 ] |
|
I think that the Go driver, being a newer driver, never implemented a wait queue in the first place. |
| Comment by Nicolas Rey [ 21/Jun/22 ] |
|
Thanks for you quick reply. Your approach completely makes sense to me, and I didn't understood that the queue itself will be removed. By the way, is there a specificity on the Go driver explaining why the threads throttling is not necessary here ? I achieved really similar results from both C# (with 100 threads throttling) and with Go. But with Go I didn't have to implement anything around the threads management, it worked out of the box under the exact same conditions. |
| Comment by Robert Stam [ 21/Jun/22 ] |
|
When you have a lot more threads than you have connections in the connection pool you have high probability of getting `MongoWaitQueueFullException`. The default size of the connection pool is 100. The default size of the connection pool wait queue is 500. So if you have over 600 threads there is always a possibility of a `MongoWaitQueueFullException`. The more threads the more likely. The more work each thread does before returning a connection to the connection pool the more likely. The original rationale for having a wait queue is this: if you have a very long wait queue you are likely to get a whole lot of timeouts. Better to fail fast with a wait queue full exception than to fail slow (and much later) with a timeout. But our thinking on this has changed and we plan to remove the wait queue in a later release. See: For now you can just configure a very large wait queue. As long as it is larger than the number of threads you have you will never get a `MongoWaitQueueFullException`. Just to be clear, when we do remove the deprecated `WaitQueueSize` setting in a later release we will also remove the wait queue itself.
|