[CSHARP-1297] Memory Leak 2.0 driver Created: 27/May/15  Updated: 05/Apr/19  Resolved: 04/Apr/16

Status: Closed
Project: C# Driver
Component/s: API, Performance
Affects Version/s: 2.0
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Fedor Finkenflugel Assignee: Unassigned
Resolution: Done Votes: 0
Labels: Bug, driver, question
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows


Backwards Compatibility: Minor Change

 Description   

I have a client application that inserts documents into Mongo at a very high speed, using the InsertAsync method. However, the connection pins the objects for too long with AsyncPinned handles, thus increasing my memory usage by a lot until it throws an OutOfMemory exception.

I solved this partly by implementing object pooling and using BulkWriteAsync, uploading about 5K documents at the same time. This allows the connection to unpin the documents before uploading the next bulk. This works fine for small documents but not for bigger ones. Mongo seems to perform slower with every BulkWrite, steadily increasing my memory again as my pools fill, and eventually throws a Timeout Exception.

Can this be fixed or is there any other way around it? The performance of this application is very important so I dont want to use the .Wait() option.



 Comments   
Comment by Robert Stam [ 27/May/15 ]

Regarding batching, it is always good for performance to insert documents in batches (though I would recommend you use InsertManyAsync instead of BulkWriteAsync). The reason is that each insert operation is a round trip to the server, so if we can insert multiple documents per round trip performance is better. It doesn't take very large batches to improve performance significantly. For example, a batch size of 100 potentially eliminates 99 out of 100 round trips (assuming the documents are small enough that 100 of them can fit in a single batch).

Regarding the out of memory exception, it sounds like you are using fire and forget inserts (I assume that's what you mean when you say you don't want to use .Wait). If that's the case, then you are probably creating fire and forget operations faster than they can be processed. Each operation requires memory, and that memory can't be freed until the operation completes. If more and more operations are started faster than they can complete the inevitable result is an out of memory exception. In this scenario there is no memory leak per se (the memory would eventually have been released when the operation completes), rather new memory is being allocated faster than old memory is being reclaimed.

If that hypothesis is correct, you will need to introduce some sort of throttling to keep from creating operations faster than they can be completed. Calling Wait (or using await in async code) is of course the simplest form of throttling. The best way to get the maximum insert performance is to have several Tasks (or threads) doing batched inserts in parallel, but having each Task wait for one operation to complete before starting the next. You may be able to achieve your required performance with a single Task/thread simply by batching the inserts.

If this explanation doesn't seem to fit your application please provide additional information and we can dig deeper.

Generated at Wed Feb 07 21:39:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.