[DRIVERS-1335] Improve Awareness of Network Compressors for Drivers Created: 15/Jul/20  Updated: 05/Feb/24

Status: Backlog
Project: Drivers
Component/s: Performance, Wire Protocol
Fix Version/s: None

Type: Epic Priority: Major - P3
Reporter: Rachelle Palmer Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to RUBY-2807 Warn at boot time if compression not ... Closed
is related to DRIVERS-600 Add support for Zstandard compression Closed
is related to WT-7814 Investigate zstd performance with ycs... In Progress
is related to SERVER-79842 Differentiate serverStatus.network.co... Closed
is related to SERVER-80150 Log negotiated network compressor wit... Closed
Product Manager: Alex Bevilacqua Alex Bevilacqua
Scope Cost Estimate: 0
Cost to Date: 0
Final Cost Estimate: 0
Cost Threshold %: 100

 Description   

Summary

How does this affect the end user?

Customers have complained that network transfer costs are extreme and unpredictable, leading to budget overruns of up to 12x their expected costs. We believe that enabling compression in drivers will help mitigate this issue. This problem likely affects all Atlas customers to some degree.

Is this issue urgent?

It is not urgent, but it is a problem we should solve - it is not a nice to have.

Is this ticket required by a downstream team?

Is this ticket only for tests?

Is this ticket have any functional impact, or is it just test improvements?



 Comments   
Comment by PM Bot [ 01/Aug/23 ]

Alex Bevilacqua renamed project from "Enable Compression by Default in all Drivers" to "Improve Awareness of Network Compressors for Drivers"

Comment by Jeffrey Yemin [ 16/Jul/21 ]

One other thing to consider.  If we enable it by default, we probably need to specify a way to explicitly disable it.  I suspect that mongodb://localhost/?compressors= is interpreted the same way as just mongodb://localhost in drivers, so couldn't be used to disable compression without some additional work.

Comment by Bernie Hackett [ 16/Jul/21 ]

Python can do this fairly easily. Like Java, an optional dependency just becomes required (which will annoy some users). However, we'll have to figure out if the python zstd and snappy modules work on s390x, POWER, ARM, etc, and not add the dependency if the architecture isn't supported.

Comment by Jeffrey Yemin [ 16/Jul/21 ]

This would be pretty simple in the Java driver.  The driver currently takes an optional dependency on https://github.com/luben/zstd-jni for zstd and https://github.com/xerial/snappy-java for snappy.  Both of these libraries offer JNI-based wrappers of the respective C implementations of those algorithms.  To enable either by default, we would just need to make the dependency required.  For both libraries, the C-based shared libraries are packaged along with the Java wrapper and deployed to Maven Central.  They typically track the C library releases closely and release a new wrapper every time the C-based library is released.  So there is no need for users of the driver to compile or download anything, unless they are running on an obscure platform for which there is no C library support (though the Snappy library offers a pure Java implementation that it uses as a fallback if it's unable to load the C library on the host platform).

The downsides are

  • We would have to be more careful about tracking and updating our dependencies on these libraries (currently we don't worry about that since applications have to opt in to the dependency and so are responsible for updating the dependencies).
  • the implementations wrap a C library so it makes users' applications susceptible to crashes in ways that they did not sign up for/opt in to. This could cause consternation for some users.
Generated at Thu Feb 08 08:23:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.