Original description:

Summary

We enabled zLib compression on ADL frontends and saw a large increase in memory consumed in the Go driver. A pprof heap profile taken while consumption was high is attached.

ADL was not under particularly heavy load however memory consumption remained high. A Grafana chart of our memory consumption is also attached. The left side shows memory consumption with compression enabled; the right side shows memory consumption after disabling compression and restarting. The pprof snapshot was taken at one of the peaks on the left side.

Please provide the version of the driver. If applicable, please provide the MongoDB server version and topology (standalone, replica set, or sharded cluster).

Driver was v1.7.4. Server version and topology varies widely.

How to Reproduce

Unknown. Perhaps run a large number of find / match / count queries over time with zLib compression enabled.

Additional Background

The linked ticket, MHOUSE-3597, contains a discussion around enabling / disabling compression. There was also some benchmarking results for all compression included in ticket comments that you may find interesting, although it is unlikely to shed any light on this ticket.

and were also attached with the original request.

The original request was to investigate excess memory usage when using zlib compression. However, the attached pprof snapshot, profile006.png, indicated the memory usage of zstd. Our initial investigation and profiling on a minimal reproduction running zlib and zstd side by side also implied that zstd consumed memory more significantly as shown in diagrams zstd.png and zlib.png. Thus, this ticket was amended to investigate excess memory usage when using zstd compression.