Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-2880

Add support for Zstandard compression

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: WT2.9.0, 3.2.11, 3.4.0-rc2
    • Labels:
      None
    • # Replies:
      14
    • Last comment by Customer:
      true
    • Sprint:
      Storage 2016-10-31

      Description

      Hi Ramon,

      Looking at Facebook's new zstandard library release today and their compression numbers (see this blog post). It would be worth exploring if we can add zstandard as a new collection compression engine in WiredTiger. There is a wrapper library provide to make it zlib-compatible (see here) so it might be a useful short project to investigate.

      Kindest regards,
      Eoin

        Issue Links

          Activity

          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2880 Add Zstandard compression support (#3075)

          • Fix a bug found by inspection in LZ4 code: we're going to offset the destination buffer by sizeof(LZ4_PREFIX), so we need to offset the destination buffer length by the same amount.
          • Prettiness pass through the snappy driver so it and the zstd driver look the same, minor cleanup in zlib.
          • Add the compression_level configuration option to the zstd extension initialization function so it's possible to set the compression level from an application.
          • Fix a bug in zlib raw compression: the destination buffer size (dst_len), can be smaller than the target page size (page_max), use the smaller of the two values to set the target compression size.
          • The zlib raw compression function could return without calling deflateEnd on its z_stream structures, potentially leaking memory and scratch buffers.
          • If the default reserved bytes for zlib raw compression isn't sufficient, we fail on compression of what might be very large blocks. We don't have information on how many bytes need to be reserved in order to know the final deflate() will succeed, and the default value was experimentally derived, for all we know there are workloads where the default will fail
            a lot. Add a fallback before failing hard, try with 2x the default reserved space.
            Branch: mongodb-3.4
            https://github.com/wiredtiger/wiredtiger/commit/b486c5ce2f342f4ec659b31764320d740ab40f46
          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2880 Add Zstandard compression support (#3075) Fix a bug found by inspection in LZ4 code: we're going to offset the destination buffer by sizeof(LZ4_PREFIX), so we need to offset the destination buffer length by the same amount. Prettiness pass through the snappy driver so it and the zstd driver look the same, minor cleanup in zlib. Add the compression_level configuration option to the zstd extension initialization function so it's possible to set the compression level from an application. Fix a bug in zlib raw compression: the destination buffer size (dst_len), can be smaller than the target page size (page_max), use the smaller of the two values to set the target compression size. The zlib raw compression function could return without calling deflateEnd on its z_stream structures, potentially leaking memory and scratch buffers. If the default reserved bytes for zlib raw compression isn't sufficient, we fail on compression of what might be very large blocks. We don't have information on how many bytes need to be reserved in order to know the final deflate() will succeed, and the default value was experimentally derived, for all we know there are workloads where the default will fail a lot. Add a fallback before failing hard, try with 2x the default reserved space. Branch: mongodb-3.4 https://github.com/wiredtiger/wiredtiger/commit/b486c5ce2f342f4ec659b31764320d740ab40f46
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: ef9a7983ea47cea78400a4472a3d4e46735385c5 from branch mongodb-3.4

          ref: 6a31c2118c..ef9a7983ea
          for: 3.4.0-rc2

          WT-1592 Add ability to dump detailed cache information via statistics
          WT-2403 Enhance random cursor implementation for LSM trees
          WT-2880 Add support for Zstandard compression
          WT-2904 Fix a bug where the reported checkpoint size could be many times data size
          WT-2949 Add an option to wtperf to not close connection on shutdown
          WT-2954 Inserting multi-megabyte values can cause large in-memory pages
          WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks
          WT-2956 utility tests -h option is always overridden by the default setup
          WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized
          WT-2963 Race setting max_entries during eviction
          WT-2965 test_wt2323_join_visibility can hang on OSX
          WT-2974 lint
          WT-2976 Add a statistic tracking how long application threads spend doing I/O
          WT-2977 Csuite LSM Random test can occasionally fail
          WT-2985 Race during checkpoint can cause a core dump
          WT-2987 Fix a bug where opening a cursor on an incomplete table drops core
          WT-2988 Fix a bug where __wt_epoch potentially returns garbage values.
          Branch: master
          https://github.com/mongodb/mongo/commit/0609d0ce2ef563d7a4cde77d46396fe5c92c6df1

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: ef9a7983ea47cea78400a4472a3d4e46735385c5 from branch mongodb-3.4 ref: 6a31c2118c..ef9a7983ea for: 3.4.0-rc2 WT-1592 Add ability to dump detailed cache information via statistics WT-2403 Enhance random cursor implementation for LSM trees WT-2880 Add support for Zstandard compression WT-2904 Fix a bug where the reported checkpoint size could be many times data size WT-2949 Add an option to wtperf to not close connection on shutdown WT-2954 Inserting multi-megabyte values can cause large in-memory pages WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks WT-2956 utility tests -h option is always overridden by the default setup WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized WT-2963 Race setting max_entries during eviction WT-2965 test_wt2323_join_visibility can hang on OSX WT-2974 lint WT-2976 Add a statistic tracking how long application threads spend doing I/O WT-2977 Csuite LSM Random test can occasionally fail WT-2985 Race during checkpoint can cause a core dump WT-2987 Fix a bug where opening a cursor on an incomplete table drops core WT-2988 Fix a bug where __wt_epoch potentially returns garbage values. Branch: master https://github.com/mongodb/mongo/commit/0609d0ce2ef563d7a4cde77d46396fe5c92c6df1
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'}

          Message: WT-2880 Add Zstandard compression support (#3075)

          • Fix a bug found by inspection in LZ4 code: we're going to offset the destination buffer by sizeof(LZ4_PREFIX), so we need to offset the destination buffer length by the same amount.
          • Prettiness pass through the snappy driver so it and the zstd driver look the same, minor cleanup in zlib.
          • Add the compression_level configuration option to the zstd extension initialization function so it's possible to set the compression level from an application.
          • Fix a bug in zlib raw compression: the destination buffer size (dst_len), can be smaller than the target page size (page_max), use the smaller of the two values to set the target compression size.
          • The zlib raw compression function could return without calling deflateEnd on its z_stream structures, potentially leaking memory and scratch buffers.
          • If the default reserved bytes for zlib raw compression isn't sufficient, we fail on compression of what might be very large blocks. We don't have information on how many bytes need to be reserved in order to know the final deflate() will succeed, and the default value was experimentally derived, for all we know there are workloads where the default will fail
            a lot. Add a fallback before failing hard, try with 2x the default reserved space.
            Branch: mongodb-3.2
            https://github.com/wiredtiger/wiredtiger/commit/b486c5ce2f342f4ec659b31764320d740ab40f46
          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith.bostic@mongodb.com'} Message: WT-2880 Add Zstandard compression support (#3075) Fix a bug found by inspection in LZ4 code: we're going to offset the destination buffer by sizeof(LZ4_PREFIX), so we need to offset the destination buffer length by the same amount. Prettiness pass through the snappy driver so it and the zstd driver look the same, minor cleanup in zlib. Add the compression_level configuration option to the zstd extension initialization function so it's possible to set the compression level from an application. Fix a bug in zlib raw compression: the destination buffer size (dst_len), can be smaller than the target page size (page_max), use the smaller of the two values to set the target compression size. The zlib raw compression function could return without calling deflateEnd on its z_stream structures, potentially leaking memory and scratch buffers. If the default reserved bytes for zlib raw compression isn't sufficient, we fail on compression of what might be very large blocks. We don't have information on how many bytes need to be reserved in order to know the final deflate() will succeed, and the default value was experimentally derived, for all we know there are workloads where the default will fail a lot. Add a fallback before failing hard, try with 2x the default reserved space. Branch: mongodb-3.2 https://github.com/wiredtiger/wiredtiger/commit/b486c5ce2f342f4ec659b31764320d740ab40f46
          Hide
          xgen-internal-githook Githook User added a comment -

          Author:

          {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'}

          Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2

          ref: 9cf2f89d6d..b11ed312ce
          for: 3.2.11

          WT-1592 Dump detailed cache information via statistics
          WT-2403 Enhance random cursor implementation for LSM trees
          WT-2831 Skip creating a checkpoint if there have been no changes
          WT-2858 rename wtperf's CONFIG structure
          WT-2880 Add support for Zstandard compression
          WT-2895 Reduce the runtime of make check testing with disable long
          WT-2904 Fix a bug where the reported checkpoint size could be many times data size
          WT-2907 Bug in Java ConcurrentCloseTest case
          WT-2917 split wtperf's configuration into per-database and per-run parts
          WT-2920 Add statistic tracking application thread cache maintenance time
          WT-2931 Configure default in-memory dirty cache usage lower
          WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration
          WT-2933 Fix a race between named snapshots and checkpoints
          WT-2937 test_inmem01 aborts due to stuck cache
          WT-2938 Assembly files should end in .sx, not .S
          WT-2941 Improve test/format to use faster key-generation functions
          WT-2942 verbose strings don't need newline
          WT-2946 dist/s_docs incompatible with OS X Xcode installation
          WT-2948 simplify error handling by making epoch time return never fail
          WT-2949 Add an option to wtperf to not close connection on shutdown
          WT-2950 Inserting multi-megabyte values can cause large in-memory pages
          WT-2954 Inserting multi-megabyte values can cause large in-memory pages
          WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks
          WT-2956 utility tests -h option is always overridden by the default setup
          WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized
          WT-2963 Race setting max_entries during eviction
          WT-2965 test_wt2323_join_visibility can hang on OSX
          WT-2974 lint
          WT-2976 Add a statistic tracking how long application threads spend doing I/O
          WT-2977 Csuite LSM Random test can occasionally fail
          WT-2985 Race during checkpoint can cause a core dump
          WT-2987 Fix a bug where opening a cursor on an incomplete table drops core
          WT-2988 __wt_epoch potentially returns garbage values.
          Branch: v3.2
          https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a

          Show
          xgen-internal-githook Githook User added a comment - Author: {u'username': u'agorrod', u'name': u'Alex Gorrod', u'email': u'alexander.gorrod@mongodb.com'} Message: Import wiredtiger: b11ed312cedb905dec49dd2c9c262fabf64d13cd from branch mongodb-3.2 ref: 9cf2f89d6d..b11ed312ce for: 3.2.11 WT-1592 Dump detailed cache information via statistics WT-2403 Enhance random cursor implementation for LSM trees WT-2831 Skip creating a checkpoint if there have been no changes WT-2858 rename wtperf's CONFIG structure WT-2880 Add support for Zstandard compression WT-2895 Reduce the runtime of make check testing with disable long WT-2904 Fix a bug where the reported checkpoint size could be many times data size WT-2907 Bug in Java ConcurrentCloseTest case WT-2917 split wtperf's configuration into per-database and per-run parts WT-2920 Add statistic tracking application thread cache maintenance time WT-2931 Configure default in-memory dirty cache usage lower WT-2932 Allow applications to selectively ignore cache limit with in-memory configuration WT-2933 Fix a race between named snapshots and checkpoints WT-2937 test_inmem01 aborts due to stuck cache WT-2938 Assembly files should end in .sx, not .S WT-2941 Improve test/format to use faster key-generation functions WT-2942 verbose strings don't need newline WT-2946 dist/s_docs incompatible with OS X Xcode installation WT-2948 simplify error handling by making epoch time return never fail WT-2949 Add an option to wtperf to not close connection on shutdown WT-2950 Inserting multi-megabyte values can cause large in-memory pages WT-2954 Inserting multi-megabyte values can cause large in-memory pages WT-2955 Add statistics tracking the amount of time threads spend waiting for high level locks WT-2956 utility tests -h option is always overridden by the default setup WT-2959 Ensure WT_SESSION_IMPL is never used before it's initialized WT-2963 Race setting max_entries during eviction WT-2965 test_wt2323_join_visibility can hang on OSX WT-2974 lint WT-2976 Add a statistic tracking how long application threads spend doing I/O WT-2977 Csuite LSM Random test can occasionally fail WT-2985 Race during checkpoint can cause a core dump WT-2987 Fix a bug where opening a cursor on an incomplete table drops core WT-2988 __wt_epoch potentially returns garbage values. Branch: v3.2 https://github.com/mongodb/mongo/commit/ebbb4eb0b091fa185b06a060d24b68eb6761ba4a
          Hide
          pasette Dan Pasette added a comment - - edited

          This compression option is not currently supported up through the mongodb interface at this time. Currently, you can only use snappy, zlib or none.

          Show
          pasette Dan Pasette added a comment - - edited This compression option is not currently supported up through the mongodb interface at this time. Currently, you can only use snappy, zlib or none.

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                24 weeks, 6 days ago
                Date of 1st Reply:

                  Agile