[CSHARP-4669] clientEncryption.Encrypt in C# uses a lot of memory relative to a somewhat large file Created: 05/Jun/23 Updated: 28/Oct/23 Resolved: 06/Oct/23 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | Field Level Encryption |
| Affects Version/s: | None |
| Fix Version/s: | 2.22.0 |
| Type: | Bug | Priority: | Unknown |
| Reporter: | James Kovacs | Assignee: | Adelin Mbida Owona |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | new-eng, tech-debt | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Documentation Changes: | Not Needed |
| Documentation Changes Summary: | 1. What would you like to communicate to the user about this feature? |
| Description |
|
We currently have client side encryption setup to encrypt all our sensitive files stored in MongoDB/GridFS. However, we recently encountered a problem where encrypting a file of around 100-200MB will use up more than 2-3GB of memory per file, and this continues to accumulate in RAM usage until the GC is called. Of course, the memory usage isn’t linear, for example for around 100 files, the memory spike is only up to 10-15GB for example. But since we are running in kubernetes/docker, the GC doesn’t seem to know the actual limit of memory where it needs to be called, ending with the pod being deleted due to memory pressure on the node in certain cases. I’m not sure if we’re doing something wrong, or if there’s a way to limit the memory usage for encrypting those somewhat large files. Below is the code we are using for the encryption:
Created from this MongoDB Community Forums post. |
| Comments |
| Comment by Adelin Mbida Owona [ 09/Oct/23 ] |
|
The PR merged related to this ticket only improves memory usage by about 20%. Further savings will be made as part of the work involved in this ticket: CSHARP-4787 |
| Comment by Githook User [ 06/Oct/23 ] |
|
Author: {'name': 'Adelin Owona', 'email': '51498470+adelinowona@users.noreply.github.com', 'username': 'adelinowona'}Message: |
| Comment by Boris Dogadov [ 16/Jun/23 ] |
|
This behaviour is easily reproduced. This can be an issue with larger files, especially in containers with restricted RAM. |