[SERVER-40710] Look into using intel ISA-L library Created: 18/Apr/19 Updated: 28/Aug/23 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Internal Code, Performance |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Mathias Stearn | Assignee: | Backlog - Service Architecture |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Service Arch
|
| Participants: |
| Description |
|
https://github.com/01org/isa-l It has optimized implementations of crc and zlib-compatable compression and decompression. It seems to offer about 5x faster compression with a similar compression ratio. |
| Comments |
| Comment by Greg Tucker [ 10/May/19 ] | ||||||||||||||||||||||||||||
|
For the igzip cli tool we just chose a multi-MB io buffer for convenience. There shouldn't be a performance issue with choosing a smaller buffer to reduce memory footprint if that's what you need. If you have any questions about the ISA-L API or usage please let me know. | ||||||||||||||||||||||||||||
| Comment by Geert Bosch [ 02/May/19 ] | ||||||||||||||||||||||||||||
|
I did testing on a 5-year-old AMD Opteron(tm) Processor 6344 at 2.6 MHz. I compiled igzip with our v3 toolchain. The data sets are the statically linked mongod executable including debug info (1.2G), tarred up mongo-src.tar server sources (286M) and tarred up Enron dataset in BSON format (378M). The one notable downsize of igzip is larger memory footprint, though I don't think this is an issue.
| ||||||||||||||||||||||||||||
| Comment by Keith Bostic (Inactive) [ 18/Apr/19 ] | ||||||||||||||||||||||||||||
|
redbeard0531, do you know if the library is known to work on AMD hardware? |