[CSHARP-2497] CPU consumption high when using await in ForEachAsync Created: 31/Jan/19  Updated: 03/Apr/23  Resolved: 25/Jun/20

Status: Closed
Project: C# Driver
Component/s: Performance
Affects Version/s: 2.7.3
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Dennis Hoefakker Assignee: Wan Bachtiar
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

I noticed this in our system. Our code was using a lot of CPU even when not "much" was going on. After some investigation i nailed it down to the short code snippets below.

I'm using MongoDB C# driver 2.7.2.

Both collections used have around 6k-10k documents in em.

I see around 8-10% of CPU consumption when i use the code below:

IMongoDatabase MyTenantMongoDatabase = MongoDB.MongoDatabase.Get(connectionString, "database");            
FindOptions MyFindOptions = new FindOptions();
MyFindOptions.NoCursorTimeout = true; //This can be a long running operation, so we don't want any timeouts!            
await MyTenantMongoDatabase.GetCollection<BsonDocument>("collection")
                         .Find(new BsonDocument(), MyFindOptions)
                         .ForEachAsync(async document =>
                         {
                             String MyLeftValue = document.GetValue("fieldX", String.Empty).AsString;
                             String MyRightValue = document.GetValue("fieldY", String.Empty).AsString;                             
                            FilterDefinition<BsonDocument> MyLeftObjectFilter = MyLeftObjectFilter = Builders<BsonDocument>.Filter.And( Builders<BsonDocument>.Filter.Eq("is_deleted", false),
                                                            Builders<BsonDocument>.Filter.Eq("title", MyLeftValue));                             
                            FilterDefinition<BsonDocument> MyRightObjectFilter = MyLeftObjectFilter = Builders<BsonDocument>.Filter.And( Builders<BsonDocument>.Filter.Eq("is_deleted", false),
                                                            Builders<BsonDocument>.Filter.Eq("title", MyRightValue));                             
                           FilterDefinition<BsonDocument> MyCountFilter = Builders<BsonDocument>.Filter.Or(MyLeftObjectFilter, MyRightObjectFilter);                            
                           Int64 MyObjectNeededForRelationCount = await MyTenantMongoDatabase.GetCollection<BsonDocument>("othercollection")
                                 .CountDocumentsAsync(MyCountFilter);                         
                          });

 

When i change it to the code snippet below i see around 0,5-1,5% of CPU consumption:

 IMongoDatabase MyTenantMongoDatabase = MongoDB.MongoDatabase.Get(connectionString, "database");            
FindOptions MyFindOptions = new FindOptions();
            
MyFindOptions.NoCursorTimeout = true; //This can be a long running operation, so we don't want any timeouts!            
await MyTenantMongoDatabase.GetCollection<BsonDocument>("collection")
                         .Find(new BsonDocument(), MyFindOptions)
                         .ForEachAsync(document =>
                         {
                             String MyLeftValue = document.GetValue("fieldX", String.Empty).AsString;
                             String MyRightValue = document.GetValue("fieldY", String.Empty).AsString;                             
                            FilterDefinition<BsonDocument> MyLeftObjectFilter = MyLeftObjectFilter = Builders<BsonDocument>.Filter.And( Builders<BsonDocument>.Filter.Eq("is_deleted", false),
                                                             Builders<BsonDocument>.Filter.Eq("title", MyLeftValue));                             
                             FilterDefinition<BsonDocument> MyRightObjectFilter = MyLeftObjectFilter = Builders<BsonDocument>.Filter.And( Builders<BsonDocument>.Filter.Eq("is_deleted", false),
                                                             Builders<BsonDocument>.Filter.Eq("title", MyRightValue));                             
                             FilterDefinition<BsonDocument> MyCountFilter = Builders<BsonDocument>.Filter.Or(MyLeftObjectFilter, MyRightObjectFilter);                            
                             Int64 MyObjectNeededForRelationCount = MyTenantMongoDatabase.GetCollection<BsonDocument>("othercollection")
                                 .CountDocuments(MyCountFilter);                         
                          });

 

So the change is to not use the async & await on Count in the ForEachAsync. This is a simplified version, in my other code i see a drop from 15-24 => 0,5 - 2.

If it's not clear please let me know!



 Comments   
Comment by Rachelle Palmer [ 25/Jun/20 ]

Hi there, thank you for reaching out to MongoDB. As this sounds more like a support issue, I wanted to give you some resources to get this question answered more quickly:

  • our MongoDB support portal, located at support.mongodb.com
  • our MongoDB community forums, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI.

Thank you!
Rachelle

Comment by Dennis Hoefakker [ 25/Apr/19 ]

@Artemov  ??? The BsonDocument are there so it's an easy reproducable piece of code, without classmaps / class structures.

I don't see how this comment solves / helps the issue. Could we stick to solving the issue.

Comment by Ivan Artemov [ 24/Apr/19 ]

P.S. Please, stop using BsonDocument everywhere!

Comment by Dennis Hoefakker [ 09/Apr/19 ]

@Wan because of your remark "async version because of less transitions and less synchronisation" i tried using .ConfigureAwait(false) on the async calls.  This didn't result in lesser CPU usage .  Also when i add ".GetAwaiter().GetResult()" to the async function and remove the async keyword it stays equal. 

I did a couple of runs to see the difference in total time between the Sync & ASync version.

SYNC Time
1 64084ms
2 63463ms
3 63542ms

The sync version used <1% CPU around 0,5% most of the time

ASYNC Time
1 64100ms
2 64301ms
3 65173ms

The ASync version used around 10% CPU with spikes to 15%

Hope it's clear, if you need more information please let me know. 

Comment by Dennis Hoefakker [ 20/Mar/19 ]

Hi Wan,

On my local dev environment i'm running Windows10, the services are running inside a ServiceFabric cluster on Azure running Windows OS.

I focused on CPU because i see a big increase in CPU usage on "heavy" (iterations on large sets) actions. I made a "simple" version of the example using count.

Does the answer:

  • So the change is to not use the async & await on Count in the ForEachAsync

Means that i must stop using await's in potential large iterations? Refactor to .Result() or use Sync versions?

The reason i went all async/await is because it's advised in this article : https://www.mongodb.com/blog/post/introducing-20-net-driver 

Do you want me to run both versions and see if there is a large difference it total complete time?

Dennis

 

Comment by Wan Bachtiar [ 06/Mar/19 ]

So the change is to not use the async & await on Count in the ForEachAsync

Hi Dennis,

The sync version of the snippet will consume less CPU than the async version because of less transitions and less synchronisation. Also calling sync count within async is likely to use less CPU because Task is blocked on the sync call.

Other than measuring the CPU consumption, did you measure the total time taken to complete the top level query for both versions ?

Also, which platform do you run this on ? i.e. Windows 10, Linux with dotnet core, etc.

Regards,
Wan.

Comment by Dennis Hoefakker [ 31/Jan/19 ]

I see the code snippets are  But i'm not able to edit it... 

Generated at Wed Feb 07 21:42:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.