Loading...

Type: Bug
Resolution: Done
Priority: Major - P3
Fix Version/s: None
Affects Version/s: 2.8.0
Component/s: Connectivity
Labels:
None
Environment:

Hide
web server - OS: Debian 9.7.0, RAM: 6GB' CPU: 2x
- .Net Core 2.2, web api, connection string with default values for connectivity

database server - OS: CentOS Linux 7 (Core), RAM: 96GB (96506 MB), CPU: 24x,
mongodb 4.0.10 replica set (2 data+1 arb), total database size: 75.8GB with WiredTiger snappy

servers are in the same internal network, 1000 Mbit/s

Show
web server - OS: Debian 9.7.0, RAM: 6GB' CPU: 2x - .Net Core 2.2, web api, connection string with default values for connectivity database server - OS: CentOS Linux 7 (Core), RAM: 96GB (96506 MB), CPU: 24x, mongodb 4.0.10 replica set (2 data+1 arb), total database size: 75.8GB with WiredTiger snappy servers are in the same internal network, 1000 Mbit/s

We are running a web api on our mongo server, as we increased the traffic some of the following errors appeared.

Unable to find source-code formatter for language: txt. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml

2019-06-25 08:01:37	One or more errors occurred. (An exception occurred while opening a connection to the server.)
System.Private.CoreLib
GetResultCore
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
InnerException: MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.TimeoutException: Timed out connecting to 192.168.1.115:27017. Timeout was 00:00:10.
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)

2019-06-24 22:00:34	One or more errors occurred. (The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.)
System.Private.CoreLib
GetResultCore
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
InnerException: MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.CheckingOutConnection()
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)

2019-06-24 22:00:35	Timed out waiting for a connection after 120607ms.
MongoDB.Driver.Core
EnteredPool
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.EnteredPool(Boolean enteredPool)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnection(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.Create(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, Func`3 bulkWrite)
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)
   at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.UpdatePartnerEntryMatching(PartnerEntry partnerEntry) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 2199

We thought we include some server stats too:

Unable to find source-code formatter for language: txt. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml

"uptime" : 59722,
"uptimeMillis" : NumberLong(59722298),
"connections" : {
		"current" : 235,
		"available" : 50965,
		"totalCreated" : 7522,
		"active" : 3
},
"network" : {
		"bytesIn" : NumberLong("102257868858"),
		"bytesOut" : NumberLong("278414574748"),
		"physicalBytesIn" : NumberLong("98858319509"),
		"physicalBytesOut" : NumberLong("223004861742"),
		"numRequests" : NumberLong(73497140),
		"compression" : {
				"snappy" : {
						"compressor" : {
								"bytesIn" : NumberLong("100211339649"),
								"bytesOut" : NumberLong("44584641783")
						},
						"decompressor" : {
								"bytesIn" : NumberLong("7926756110"),
								"bytesOut" : NumberLong("11610840700")
						}
				}
		},
		"serviceExecutorTaskStats" : {
				"executor" : "passthrough",
				"threadsRunning" : 235
		}
},

"opcounters" : {
		"insert" : 346427,
		"query" : 181740,
		"update" : 51700929,
		"delete" : 28074,
		"getmore" : 10177557,
		"command" : 11438426
},
"mem" : {
		"bits" : 64,
		"resident" : 41188,
		"virtual" : 54577,
		"supported" : true,
		"mapped" : 0,
		"mappedWithJournal" : 0
}

If you need any other info, we'll try to provide it.

Details

Description

Attachments

Activity

People

Dates