<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:01:50 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-3018] Compression of wire protocol</title>
                <link>https://jira.mongodb.org/browse/SERVER-3018</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Edited description:&lt;br/&gt;
Implement compression of wire protocol communication independent of running SSL.&lt;/p&gt;

&lt;p&gt;==================================&lt;br/&gt;
Old description:&lt;br/&gt;
Currently, the Mongo Wire protocol sends the data essentially in clear-text.  This has two implications for my user scenario.  First is that there&apos;s a lot of network traffic generated for queries.  When reports are run and many fields of data are retrieved, I get the same field name over and over.  Some compression here would speed up the delivery of the data.  The query itself is lightning fast, but the transaction is slowed down by the movement of the massive amount of data.&lt;/p&gt;

&lt;p&gt;Second, the clear-text has security implications.  Running SSL or some similar secure wire protocol could solve potentially both these issues.&lt;/p&gt;

&lt;p&gt;Thanks!&lt;/p&gt;</description>
                <environment></environment>
        <key id="16468">SERVER-3018</key>
            <summary>Compression of wire protocol</summary>
                <type id="2" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14711&amp;avatarType=issuetype">New Feature</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="jonathan.reams@mongodb.com">Jonathan Reams</assignee>
                                    <reporter username="rvanderwall23">Robert Vanderwall</reporter>
                        <labels>
                    </labels>
                <created>Fri, 29 Apr 2011 00:09:07 +0000</created>
                <updated>Mon, 24 Apr 2017 07:10:01 +0000</updated>
                            <resolved>Tue, 9 Aug 2016 16:26:50 +0000</resolved>
                                                    <fixVersion>3.3.11</fixVersion>
                                    <component>Networking</component>
                    <component>Security</component>
                                        <votes>77</votes>
                                    <watches>72</watches>
                                                                                                                <comments>
                            <comment id="1356456" author="ramon.fernandez" created="Sun, 14 Aug 2016 22:44:54 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sallgeud&quot; class=&quot;user-hover&quot; rel=&quot;sallgeud&quot;&gt;sallgeud&lt;/a&gt;, I did a quick search of the SERVER backlog and couldn&apos;t find a ticket about adding compression to clients &amp;#8211; care to create one? It will be a significant endeavor to add it to all drivers, but definitely a useful feature to have.&lt;/p&gt;</comment>
                            <comment id="1356439" author="sallgeud" created="Sun, 14 Aug 2016 18:57:43 +0000"  >&lt;p&gt;Is there a plan to do compression to clients at some point?&lt;/p&gt;</comment>
                            <comment id="1351629" author="jonathan.reams@10gen.com" created="Tue, 9 Aug 2016 16:26:50 +0000"  >&lt;p&gt;Wire protocol compression using Snappy has been implemented for MongoDB 3.3.11. Compression is off-by-default, but can be enabled in the server with &lt;tt&gt;&amp;#45;&amp;#45;networkMessageCompressors=snappy&lt;/tt&gt;. This feature is currently only supported for intra-cluster communication (e.g. mongod to mongod and mongod to mongos). More extensive documentation will be provided in the 3.4 documentation.&lt;/p&gt;</comment>
                            <comment id="1240887" author="linar-jether" created="Wed, 20 Apr 2016 06:42:14 +0000"  >&lt;p&gt;This feature is crucial for WAN replication, as it can easily compress the oplog for less replication lag&lt;/p&gt;</comment>
                            <comment id="1240850" author="sallgeud" created="Wed, 20 Apr 2016 05:41:59 +0000"  >&lt;p&gt;I&apos;d love to see some consideration for other algorithms like lz4 (for speed). Some modes of lz4 out perform snappy by 3-10 times. The most recent vuln aside, of love to see it as a compression option for wt, for those who want tons of speed with slightly less compression. &lt;/p&gt;

&lt;p&gt;We&apos;re getting 8.5:1 compression on snappy, which is amazing... But we&apos;d love to see twice the compression speed (including across the wire), at even 4:1 compression. &lt;/p&gt;

&lt;p&gt;Everyone&apos;s data compresses differently, so more options are always good. I&apos;m sure some would love even higher ratios, sacrificing CPU or responsiveness for saving space. &lt;/p&gt;</comment>
                            <comment id="1240827" author="nyxtom@gmail.com" created="Wed, 20 Apr 2016 04:10:24 +0000"  >&lt;p&gt;It&apos;s been a while since any reply on this issue. What&apos;s the status of getting this implemented?&lt;/p&gt;</comment>
                            <comment id="1106173" author="mgudas@hotwire.com" created="Fri, 4 Dec 2015 23:58:29 +0000"  >&lt;p&gt;I concur with Tomas Pecholt. Column names eating up lot of storage while resting, memory while processing and network while transiting. Data compression (not only column names but also data) in transit will free up network bandwidth. If one has app in his data center and MongoDB in AWS then to support 10x data volume, he has to beef up network bandwidth and other networking gear as well. It becomes more expensive.&lt;/p&gt;</comment>
                            <comment id="1065896" author="tdondich" created="Tue, 20 Oct 2015 17:17:47 +0000"  >&lt;p&gt;Please.  With cherry on top.  Add compression to the wire.&lt;/p&gt;</comment>
                            <comment id="1065776" author="sallgeud" created="Tue, 20 Oct 2015 16:05:07 +0000"  >&lt;p&gt;Now that you&apos;re using snappy in wiredtiger, it may be a a great implementation for the wire protocol as its been built with streaming in mind. I&apos;m curious how much this could help replicasets recover from large oplogs faster. &lt;/p&gt;

&lt;p&gt;It would be great that if implemented, the compression is not only between server and client, but also between servers in replicasets and shards.&lt;/p&gt;</comment>
                            <comment id="897435" author="redsand" created="Fri, 24 Apr 2015 15:39:48 +0000"  >&lt;p&gt;I appreciate the clarification Andreas, as you can see, the community is watching closely and it is imperative that some of the basics of functionality are covered (including wire compression).&lt;/p&gt;

&lt;p&gt;Please prioritize this, as this is a vital requirement for competing in the &quot;big data&quot; market, especially when many of your customers are still stuck on 1gbps switches.&lt;/p&gt;

&lt;p&gt;Tim&lt;/p&gt;</comment>
                            <comment id="897434" author="tdondich" created="Fri, 24 Apr 2015 15:39:15 +0000"  >&lt;p&gt;Thank you for re-opening. We at MaxCDN are more than happy to review any approaches to this ticket your team would like to take and test for you with our very large ingest volume.&lt;/p&gt;</comment>
                            <comment id="897425" author="andreas.nilsson@10gen.com" created="Fri, 24 Apr 2015 15:33:43 +0000"  >&lt;p&gt;Thanks for your swift responses. I will re-open this ticket and clarify the description.&lt;/p&gt;

&lt;p&gt;I closed it based on the description of the ticket as compression via OpenSSL. I will edit the description appropriately to reflect generic compression of the wire protocol.&lt;/p&gt;

&lt;p&gt;It is not my opinion, neither the opinion of MongoDB that compression via OpenSSL is a sufficient substitute to independent transport compression.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Andreas Nilsson&lt;/p&gt;</comment>
                            <comment id="897401" author="tdondich" created="Fri, 24 Apr 2015 15:17:21 +0000"  >&lt;p&gt;I absolutely agree with Tim Shelton.  This is a horrendous answer.  Really?  So you&apos;re saying we need to custom compile OpenSSL and have no control over the compression level?  It&apos;s this kind of response that makes me lose faith in MongoDB and look elsewhere for more efficient solutions.&lt;/p&gt;</comment>
                            <comment id="897383" author="redsand" created="Fri, 24 Apr 2015 15:05:39 +0000"  >&lt;p&gt;I feel like you are skirting responsibility for the solution.&lt;/p&gt;

&lt;p&gt;This is unacceptable and I feel you should re-review this with your leadership.  Compression should not be dependent upon SSL.   That is a completely absurd requirement.  Especially since Redhat removed OpenSSL compression.&lt;/p&gt;

&lt;p&gt;Are you serving the larger market? &lt;/p&gt;

&lt;p&gt;You should know that MOST legacy mongo installation do not have ssl on by default, and many are based upon redhat.&lt;/p&gt;

&lt;p&gt;I am also a vendor and I would never avoid something so imperative as this.&lt;/p&gt;</comment>
                            <comment id="897375" author="andreas.nilsson@10gen.com" created="Fri, 24 Apr 2015 14:59:46 +0000"  >&lt;p&gt;Mongo servers support SSL via OpenSSL. OpenSSL uses compression by default as long as the library is compiled with zlib or other compression support.&lt;/p&gt;

&lt;p&gt;After POODLE/CRIME etc. default distributions of OpenSSL are compiled without compression support. For instance RedHat disabled compression in OpenSSL by default in 2013 &lt;a href=&quot;https://rhn.redhat.com/errata/RHSA-2013-0587.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://rhn.redhat.com/errata/RHSA-2013-0587.html&lt;/a&gt; &lt;/p&gt;

&lt;p&gt;If you want to use wire compression with MongoDB please make sure that both the client and server side is compiled with compression. I will close this ticket as &quot;Works as Designed&quot;.&lt;/p&gt;

&lt;p&gt;Regards,&lt;br/&gt;
Andreas Nilsson&lt;/p&gt;</comment>
                            <comment id="621775" author="pelmam.dev@gmail.com" created="Sun, 15 Jun 2014 05:11:09 +0000"  >&lt;p&gt;I second that. Aggregated non-normalized documents are common practice with mongodb, so it&apos;s reasonable to expect the wire protocol to handle them efficiently.&lt;br/&gt;
We&apos;ve used various workarounds, mainly projection to extract smaller documents, and distributed cache. Unfortunately that&apos;s expensive code to maintain and &quot;micro manage&quot; - a distraction from the focus of business logic and customer experience (obviously here are cases where this can&apos;t be avoided, but a lot of it can be saved by simple compression).&lt;/p&gt;</comment>
                            <comment id="515336" author="tdondich" created="Thu, 13 Mar 2014 16:54:11 +0000"  >&lt;p&gt;We have a LOT of data attempting to be replicated to cross data-center replication to secondaries.  Compression would greatly solve our network utilization.  At this time, not compressing this data is causing us to not be able to effectively implement cross data-center replication for data redundancy.&lt;/p&gt;</comment>
                            <comment id="478784" author="rubencaro" created="Wed, 8 Jan 2014 08:50:32 +0000"  >&lt;p&gt;Network traffic volume itself can be a showstopper. It is for me on some projects. &lt;/p&gt;</comment>
                            <comment id="90772" author="tomas" created="Wed, 22 Feb 2012 10:32:19 +0000"  >&lt;p&gt;My 2$: While this would be a big improvement over current state, long column names still occupy a big part of memory and hard disk space. Ammount of memory is essential for mongodb performance and even hard disc space is not for free so i suggest to use column name compression on the server. Automatic decompression would be performed by client when needed. Because of this issue we currently assign one-letter names to all columns. That works but it doesn&apos;t look very descriptive when somebody uses rockmongo or similar sw and tries to view/modify db contents.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10320">
                    <name>Documented</name>
                                                                <inwardlinks description="is documented by">
                                        <issuelink>
            <issuekey id="336386">DOCS-9583</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="367209">SERVER-28442</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="309104">SERVER-25620</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="336845">SERVER-27310</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="11092">SERVER-524</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>19.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 22 Feb 2012 10:32:19 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 26 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-582</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>stephen.steneker@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 26 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10000" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Old_Backport</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10000"><![CDATA[No]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andreas.nilsson</customfieldvalue>
            <customfieldvalue>sallgeud</customfieldvalue>
            <customfieldvalue>jonathan.reams@mongodb.com</customfieldvalue>
            <customfieldvalue>linar-jether</customfieldvalue>
            <customfieldvalue>mgudas@hotwire.com</customfieldvalue>
            <customfieldvalue>pelmam.dev@gmail.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
            <customfieldvalue>rvanderwall23</customfieldvalue>
            <customfieldvalue>rubencaro</customfieldvalue>
            <customfieldvalue>tdondich</customfieldvalue>
            <customfieldvalue>nyxtom@gmail.com</customfieldvalue>
            <customfieldvalue>redsand</customfieldvalue>
            <customfieldvalue>tomas</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrp0qf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsod93:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4847</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1227">Platforms 2016-08-26</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrl9mf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>