Details
-
Bug
-
Resolution: Done
-
Major - P3
-
None
-
2.8.0
-
None
-
web server - OS: Debian 9.7.0, RAM: 6GB' CPU: 2x
- .Net Core 2.2, web api, connection string with default values for connectivity
database server - OS: CentOS Linux 7 (Core), RAM: 96GB (96506 MB), CPU: 24x,
mongodb 4.0.10 replica set (2 data+1 arb), total database size: 75.8GB with WiredTiger snappy
servers are in the same internal network, 1000 Mbit/sweb server - OS: Debian 9.7.0, RAM: 6GB' CPU: 2x - .Net Core 2.2, web api, connection string with default values for connectivity database server - OS: CentOS Linux 7 (Core), RAM: 96GB (96506 MB), CPU: 24x, mongodb 4.0.10 replica set (2 data+1 arb), total database size: 75.8GB with WiredTiger snappy servers are in the same internal network, 1000 Mbit/s
Description
We are running a web api on our mongo server, as we increased the traffic some of the following errors appeared.
2019-06-25 08:01:37 One or more errors occurred. (An exception occurred while opening a connection to the server.)
|
System.Private.CoreLib
|
GetResultCore
|
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
|
at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
|
InnerException: MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.TimeoutException: Timed out connecting to 192.168.1.115:27017. Timeout was 00:00:10.
|
at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
|
--- End of inner exception stack trace ---
|
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
|
at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)
|
|
|
2019-06-24 22:00:34 One or more errors occurred. (The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.)
|
System.Private.CoreLib
|
GetResultCore
|
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
|
at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.InsertOrUpdateFields(CompanyId companyId, List`1 updates) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 105
|
InnerException: MongoDB.Driver.MongoWaitQueueFullException: The wait queue for acquiring a connection to server 192.168.1.115:27017 is full.
|
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.CheckingOutConnection()
|
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Servers.Server.GetChannelAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
|
at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionBase`1.UpdateOneAsync(FilterDefinition`1 filter, UpdateDefinition`1 update, UpdateOptions options, Func`3 bulkWriteAsync)
|
|
|
2019-06-24 22:00:35 Timed out waiting for a connection after 120607ms.
|
MongoDB.Driver.Core
|
EnteredPool
|
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnectionHelper.EnteredPool(Boolean enteredPool)
|
at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquireConnection(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Servers.Server.GetChannel(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Bindings.ServerChannelSource.GetChannel(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Bindings.ChannelSourceHandle.GetChannel(CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.RetryableWriteContext.Create(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
|
at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
|
at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
|
at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, Func`3 bulkWrite)
|
at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)
|
at RealTimeUpdaterApi.DataAccessLayer.MongoDBAccess.UpdatePartnerEntryMatching(PartnerEntry partnerEntry) in d:\Projects\NewProject\ImportOracleToMongo\RealTimeUpdaterApi\DataAccessLayer\MongoDBAccess.cs:line 2199
|
We thought we include some server stats too:
"uptime" : 59722,
|
"uptimeMillis" : NumberLong(59722298),
|
"connections" : {
|
"current" : 235,
|
"available" : 50965,
|
"totalCreated" : 7522,
|
"active" : 3
|
},
|
"network" : {
|
"bytesIn" : NumberLong("102257868858"),
|
"bytesOut" : NumberLong("278414574748"),
|
"physicalBytesIn" : NumberLong("98858319509"),
|
"physicalBytesOut" : NumberLong("223004861742"),
|
"numRequests" : NumberLong(73497140),
|
"compression" : {
|
"snappy" : {
|
"compressor" : {
|
"bytesIn" : NumberLong("100211339649"),
|
"bytesOut" : NumberLong("44584641783")
|
},
|
"decompressor" : {
|
"bytesIn" : NumberLong("7926756110"),
|
"bytesOut" : NumberLong("11610840700")
|
}
|
}
|
},
|
"serviceExecutorTaskStats" : {
|
"executor" : "passthrough",
|
"threadsRunning" : 235
|
}
|
},
|
|
|
"opcounters" : {
|
"insert" : 346427,
|
"query" : 181740,
|
"update" : 51700929,
|
"delete" : 28074,
|
"getmore" : 10177557,
|
"command" : 11438426
|
},
|
"mem" : {
|
"bits" : 64,
|
"resident" : 41188,
|
"virtual" : 54577,
|
"supported" : true,
|
"mapped" : 0,
|
"mappedWithJournal" : 0
|
}
|
If you need any other info, we'll try to provide it.