[CSHARP-3939] Unable to aquire connection Created: 28/Oct/21  Updated: 27/Oct/23  Resolved: 01/Nov/21

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Blocker - P1
Reporter: Arjav Jain Assignee: James Kovacs
Resolution: Works as Designed Votes: 0
Labels: c#, mongob, timeout
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Summary

Hi all,

I facing issue while connecting to mongodb from dotnet5 application. My application was able to connect to mongodb in the beginning and working fine but as we uploaded 30 million document to mongodb, now my application is not able to create connection to database, this issue is reproducible also like if i clean my mongodb, my application start working again and when i push 30 million document again, my application unable to acquire connection again.

From application I am receiving the following errors
 _- A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector

Unknown macro: { AllowedLatencyRange = 00}

, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "ReplicaSet", Type : "ReplicaSet", State : "Disconnected", Servers : [{ ServerId: "

Unknown macro: { ClusterId }

", EndPoint: "Unspecified/mongodb-usersegment1.fareportallan.local:27000", ReasonChanged: "Heartbeat", State: "Disconnected", ServerVersion: , TopologyVersion: , Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server._
 - Timed out waiting for a connection after 5000ms.

Connection string parameter - authSource=UserSegmentation&authMechanism=SCRAM-SHA-256&replicaSet=userProfiles&connect=replicaSet&safe=true&maxpoolsize=500&socketTimeoutMS=10000&connectTimeoutMS=5000&waitQueueTimeoutMS=5000&readPreference=secondaryPreferred&tls=true&ssl=true&tlsCAFile=/etc/ssl/prod-user-ca.pem

I am using singleton architecture, where single client is created in start-up file is used throughout the application.

I am also getting error from my health check
Health check code >
using var cursor = await _mongoDbConfiguration.MongoClient.GetDatabase(_mongoDbConfiguration.DatabaseName).ListCollectionNamesAsync(cancellationToken: cancellationToken);
await cursor.FirstAsync(cancellationToken);
return HealthCheckResult.Healthy("A healthy mongoDb.");

Error from Health check
 - at MongoDB.Driver.MongoDatabaseImpl.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
at MongoDB.Driver.MongoDatabaseImpl.ListCollectionNamesAsync(IClientSessionHandle session, ListCollectionNamesOptions options, CancellationToken cancellationToken)
at MongoDB.Driver.MongoDatabaseImpl.ExecuteReadOperationAsync[T](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.ListCollectionsOperation.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken)
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.EnteredPoolAsync(Boolean enteredPool, CancellationToken cancellationToken)
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.FinalizePoolEnterance(PooledConnection pooledConnection)

System.TimeoutException: Timed out waiting for a connection after 5000ms.

Thank you,
Arjav

 

Motivation

How does this affect the end user?

They are blocked

How likely is it that this problem or use case will occur?

Main path

If the problem does occur, what are the consequences and how severe are they?

Application fails

Is this issue urgent?

yes, i need to solve this within few days

Is this ticket only for tests?

No



 Comments   
Comment by James Kovacs [ 01/Nov/21 ]

Hi, arjav.jain@fareportal.com,

Thank you for letting us know that you found the root cause and that allocating sufficient CPU resources to your application resolved the issue. We have closed this issue.

Sincerely,
James

Comment by Arjav Jain [ 01/Nov/21 ]

Hi Guys,
I solve this problem, this is happening because we have not assigned appropriate CPU resource to our application and during Load we were seeing spikes in CPU consumption, increasing the CPU resource solve the problem
Thanks for the help.

Comment by Boris Dogadov [ 29/Oct/21 ]

Hi arjav.jain@fareportal.com,

Thank you for the additional information.
The fact that you are observing "wait queue is full", may indicate that there are too many operations that are waiting for a connection. As maxPoolSize is set to 500, the waitQueueSize would be 2500, translating to 2500 operations waiting for a connection. I would suggest checking whether this is an expected load.

Also there is a chance that this might be related to CSHARP-3885, it's worth checking whether this behaviour is reproducible in 2.12.4 driver version.
To investigate further it would very helpful to have a standalone reproduction code for this issue.

Comment by Arjav Jain [ 29/Oct/21 ]

Hello Boris Dogadov

  • This Does not start occurring after driver update
  • Yes, this become more severe as we add more document to mongoDb and after 30M, we are not able to push single doc to mongo
  • Yes, I am able to ping server and few connection are still active, on mongo we have ~100 connections
  • if we increase connection timeout then we are getting wait queue is full.
Comment by Arjav Jain [ 29/Oct/21 ]

Hello Dmitry Lukyanov, I am using mongo-csharp-driver","version":"2.13.2.0"

Comment by Boris Dogadov [ 28/Oct/21 ]

Hello arjav.jain@fareportal.com
It looks like all the available connections are in use and establishing new ones takes more than 5 seconds. This can happen due to variety of reasons.

If this behaviour is reproducible, could you please share a simple reproduction code?

Few questions:

  • Did this issue start occurring after particular driver version update?
  • Does this occur only after inserting 30M docs?
  • Are you able to ping the server when re-starting the application, with fully populated db?
  • What happens if the connection acquisition timeout is increased from 5 seconds to 60.
Comment by Dmitry Lukyanov (Inactive) [ 28/Oct/21 ]

Hello arjav.jain@fareportal.com , thank you for your report, please provide a driver version that you use.

Generated at Wed Feb 07 21:46:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.