[CSHARP-204] Concurrency Issue on failure recovery Created: 15/Apr/11  Updated: 02/Apr/15  Resolved: 06/Jun/11

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: 1.0
Fix Version/s: 1.1

Type: Bug Priority: Major - P3
Reporter: Miguel Pilar Assignee: Robert Stam
Resolution: Done Votes: 1
Labels: concurrency
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 2003, IIS, connected to a single mongos instance


Attachments: Text File AcquireConnection-Concurrency-problem-fix.patch    
Backwards Compatibility: Fully Compatible

 Description   

Note: using git revision 1a730b for line numbers.

While doing high-concurrency load tests (~2000 connections) against our web service which calls mongo I ran into a concurrency issue that narrowed down to a problem between MongoServer.Disconnect() and MongoServer.AcquireConnection(MongoDatabase database,bool slaveOk).

The issue occurs when an exception occurs within a MongoConnection between the GetConectionPoolEndPoint call (MongoServer.cs:798) and the connectionPool.AcquireConnection(database) (MongoServer.cs:807) call in AcquireConnection(database, endPoint) (MongoServer.cs:799).

Since MongoConnection.HandleException() is forcing a server disconnect, when connectionPool.AcquireConnection(database) gets called, the result is eventually a InvalidOperationException("Attempt to get a connection from a closed connection pool") exception (MongoConnectionPool.cs:103).

As this is a concurrency issue that I've only seen in a very specific test, I have been unable to write a reliable test for it (we have to saturate the data link with several Gbps of data to make it fail semi-consistently).

Attached is the patch that I've implemented to fix this issue, since the lock() calls are reentrant we simply wrap both GetConectionPoolEndPoint and AcquireConnection with a lock(serverLock). So far this code has performed well, but it needs to be thoroughly reviewed to make sure my simple fix will not cause deadlocks on some scenario I have not taken into account.

This bug possibly affects previous versions.



 Comments   
Comment by Robert Stam [ 06/Jun/11 ]

Reviewed the patch and agreed with it. Thanks Miguel!

Comment by Grégoire Seux [ 23/May/11 ]

This issue also occured on Linux with mongo 1.8.1 and the last stable c# driver.

Generated at Wed Feb 07 21:36:07 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.