[CSHARP-2649] mongo connection problem Created: 25/Jun/19  Updated: 25/Jun/20  Resolved: 25/Jun/20

Status: Closed
Project: C# Driver
Component/s: Connectivity
Affects Version/s: 2.8.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Mitereiter Balazs Zoltan Assignee: Wan Bachtiar
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

web server - OS: Debian 9.7.0, RAM: 6GB' CPU: 2x

  • .Net Core 2.2, web api, connection string with default values for connectivity

database server - OS: CentOS Linux 7 (Core), RAM: 96GB (96506 MB), CPU: 24x,
mongodb 4.0.10 replica set (2 data+1 arb), total database size: 75.8GB with WiredTiger snappy

servers are in the same internal network, 1000 Mbit/s



 Description   

We are running a web api on our mongo server, as we increased the traffic some of the following errors appeared.

2019-06-25 08:01:37	One or more errors occurred. (An exception occurred while opening a connection to the server.)
System.Private.CoreLib
GetResultCore
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
InnerException: MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.TimeoutException: Timed out connecting to 192.168.1.115:27017. Timeout was 00:00:10.
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)
 
2019-06-24 22:00:34	One or more errors occurred. (The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.)
System.Private.CoreLib
GetResultCore
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
InnerException: MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.CheckingOutConnection()
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)
 
2019-06-24 22:00:35	Timed out waiting for a connection after 120607ms.
MongoDB.Driver.Core
EnteredPool
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.EnteredPool(Boolean enteredPool)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnection(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.Create(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, Func`3 bulkWrite)
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.UpdatePartnerEntryMatching(PartnerEntry partnerEntry) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 2199

We thought we include some server stats too:

"uptime" : 59722,
"uptimeMillis" : NumberLong(59722298),
"connections" : {
		"current" : 235,
		"available" : 50965,
		"totalCreated" : 7522,
		"active" : 3
},
"network" : {
		"bytesIn" : NumberLong("102257868858"),
		"bytesOut" : NumberLong("278414574748"),
		"physicalBytesIn" : NumberLong("98858319509"),
		"physicalBytesOut" : NumberLong("223004861742"),
		"numRequests" : NumberLong(73497140),
		"compression" : {
				"snappy" : {
						"compressor" : {
								"bytesIn" : NumberLong("100211339649"),
								"bytesOut" : NumberLong("44584641783")
						},
						"decompressor" : {
								"bytesIn" : NumberLong("7926756110"),
								"bytesOut" : NumberLong("11610840700")
						}
				}
		},
		"serviceExecutorTaskStats" : {
				"executor" : "passthrough",
				"threadsRunning" : 235
		}
},
 
"opcounters" : {
		"insert" : 346427,
		"query" : 181740,
		"update" : 51700929,
		"delete" : 28074,
		"getmore" : 10177557,
		"command" : 11438426
},
"mem" : {
		"bits" : 64,
		"resident" : 41188,
		"virtual" : 54577,
		"supported" : true,
		"mapped" : 0,
		"mappedWithJournal" : 0
}

If you need any other info, we'll try to provide it.



 Comments   
Comment by Rachelle Palmer [ 25/Jun/20 ]

Hi there, thank you for reaching out to MongoDB. As this sounds more like a support issue, I wanted to give you some resources to get this question answered more quickly:

  • our MongoDB support portal, located at support.mongodb.com
  • our MongoDB community forums, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI.

Thank you!
Rachelle

Generated at Wed Feb 07 21:43:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.