[GODRIVER-2259] Excess memory consumption from Zstd compression Created: 20/Dec/21  Updated: 28/Oct/23  Resolved: 22/Jun/22

Status: Closed
Project: Go Driver
Component/s: None
Affects Version/s: None
Fix Version/s: 1.10.0

Type: Improvement Priority: Major - P3
Reporter: Thomas Henretty (Inactive) Assignee: Qingyang Hu
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2021-12-20 at 11.00.58 AM.png     PNG File profile006.png     PNG File zlib.png     PNG File zstd.png     PNG File zstd_encodeall.png     PNG File zstd_fixed.png    
Issue Links:
Related
Epic Link: Reduce Allocations
Quarter: FY23Q2, FY23Q3

 Description   

Original description:

Summary

We enabled zLib compression on ADL frontends and saw a large increase in memory consumed in the Go driver. A pprof heap profile taken while consumption was high is attached.

ADL was not under particularly heavy load however memory consumption remained high. A Grafana chart of our memory consumption is also attached. The left side shows memory consumption with compression enabled; the right side shows memory consumption after disabling compression and restarting. The pprof snapshot was taken at one of the peaks on the left side.

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

Driver was v1.7.4. Server version and topology varies widely.

How to Reproduce

Unknown. Perhaps run a large number of find / match / count queries over time with zLib compression enabled.

Additional Background

The linked ticket, MHOUSE-3597, contains a discussion around enabling / disabling compression. There was also some benchmarking results for all compression included in ticket comments that you may find interesting, although it is unlikely to shed any light on this ticket.

and were also attached with the original request.

The original request was to investigate excess memory usage when using zlib compression. However, the attached pprof snapshot, profile006.png, indicated the memory usage of zstd. Our initial investigation and profiling on a minimal reproduction running zlib and zstd side by side also implied that zstd consumed memory more significantly as shown in diagrams zstd.png and zlib.png. Thus, this ticket was amended to investigate excess memory usage when using zstd compression.



 Comments   
Comment by Githook User [ 22/Jun/22 ]

Author:

{'name': 'Qingyang Hu', 'email': '103950869+qingyang-hu@users.noreply.github.com', 'username': 'qingyang-hu'}

Message: Godriver-2259 Reduce memory consumption. (#985)
Branch: master
https://github.com/mongodb/mongo-go-driver/commit/27a230bae97e894d08015e3b8a548698fbf21dce

Generated at Thu Feb 08 08:38:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.