[SERVER-40710] Look into using intel ISA-L library Created: 18/Apr/19  Updated: 28/Aug/23

Status: Backlog
Project: Core Server
Component/s: Internal Code, Performance
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Mathias Stearn Assignee: Backlog - Service Architecture
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Service Arch
Participants:

 Description   

https://github.com/01org/isa-l

It has optimized implementations of crc and zlib-compatable compression and decompression. It seems to offer about 5x faster compression with a similar compression ratio.



 Comments   
Comment by Greg Tucker [ 10/May/19 ]

For the igzip cli tool we just chose a multi-MB io buffer for convenience.  There shouldn't be a performance issue with choosing a smaller buffer to reduce memory footprint if that's what you need.

If you have any questions about the ISA-L API or usage please let me know.

Comment by Geert Bosch [ 02/May/19 ]

I did testing on a 5-year-old AMD Opteron(tm) Processor 6344 at 2.6 MHz. I compiled igzip with our v3 toolchain.
Results are good. The table below list user times, best out of 3 runs. System times were small and comparable between both compressors, but noisy, so omitted.

The data sets are the statically linked mongod executable including debug info (1.2G), tarred up mongo-src.tar server sources (286M) and tarred up Enron dataset in BSON format (378M). The one notable downsize of igzip is larger memory footprint, though I don't think this is an issue.

file gzip -1 time (s) gzip -1 ratio gzip max rss igzip time(s) igzip ratio igzip max rss
mongod 15.93 6.75 1720 3.95 6.96 3928
mongo-src.tar 5.14 3.89 1736 1.42 3.89 3776
messages.bson 8.50 3.00 1696 2.54 3.14 3496
Comment by Keith Bostic (Inactive) [ 18/Apr/19 ]

redbeard0531, do you know if the library is known to work on AMD hardware?

Generated at Thu Feb 08 04:55:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.