<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:49:37 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-82551] Use parallel compressor to speedup binaries archival</title>
                <link>https://jira.mongodb.org/browse/SERVER-82551</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h2&gt;&lt;a name=&quot;Summary&quot;&gt;&lt;/a&gt;Summary&lt;/h2&gt;

&lt;p&gt;Using &lt;a href=&quot;https://zlib.net/pigz/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;pigz&lt;/a&gt; parallel compressor to create binary tarball would reduce &lt;tt&gt;archive_dist_test_debug&lt;/tt&gt; task runtime from &lt;tt&gt;~14&lt;/tt&gt; min to &lt;tt&gt;~5&lt;/tt&gt; min&lt;/p&gt;
&lt;h2&gt;&lt;a name=&quot;Longdescription&quot;&gt;&lt;/a&gt;Long description&lt;/h2&gt;

&lt;p&gt;The majority of evergreen tasks will run only after mongo binaries have been compiled, compressed and uploaded to S3.&lt;/p&gt;

&lt;p&gt;For amazon linux 2 variant these steps will take roughly:&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;&lt;tt&gt;16&lt;/tt&gt; min -&amp;gt; &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2_arm64_compile_compile_dist_test_patch_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_653e46321e2d1738a957b279_23_10_29_11_47_26/logs?execution=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;compile_dist_test&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;3.20&lt;/tt&gt; min -&amp;gt; &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/logs?execution=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;archive_dist_test&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;b&gt;&lt;tt&gt;14&lt;/tt&gt; min -&amp;gt;&lt;/b&gt; &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_debug_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/logs?execution=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;b&gt;archive_dist_test_debug&lt;/b&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;In &lt;tt&gt;archive_dist_test_debug&lt;/tt&gt; is mostly composed by two parts:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;tt&gt;12&lt;/tt&gt; min &lt;a href=&quot;https://parsley.mongodb.com/evergreen/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_debug_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/0/task?bookmarks=0,2658&amp;amp;filters=100Finished%2520command&amp;amp;shareLine=2466&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Compressing all the binaries into a tar file&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;tt&gt;2&lt;/tt&gt; min &lt;a href=&quot;https://parsley.mongodb.com/evergreen/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_debug_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/0/task?bookmarks=0,2472,2658&amp;amp;filters=100Finished%2520command&amp;amp;shareLine=2472&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;Uploading the tarball to S3&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;The compression is performed using the following tar command:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;/bin/tar -C build/install -T /data/mci/5098d994527fa548b1195cf0b5831e45/src/mongo-debugsymbols.tgz.filelist -czf mongo-debugsymbols.tgz&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;tar by default use single thread compression algorithm, this means that we are using only 1 out of the 16 cores available (we currently use amazon2-arm64-large for this task).&lt;/p&gt;

&lt;p&gt;It is possible to simply tell tar command to use the parallel compessor &lt;a href=&quot;https://zlib.net/pigz/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;pigz&lt;/a&gt; to make use of all the available core.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;A &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_debug_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/logs?execution=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;quick experiment &lt;/a&gt;showed how using pigz will reduce the tar command execution time from 9.22 min to 35 seconds.&lt;/b&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="2485092">SERVER-82551</key>
            <summary>Use parallel compressor to speedup binaries archival</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="tommaso.tocci@mongodb.com">Tommaso Tocci</assignee>
                                    <reporter username="tommaso.tocci@mongodb.com">Tommaso Tocci</reporter>
                        <labels>
                    </labels>
                <created>Sun, 29 Oct 2023 14:53:24 +0000</created>
                <updated>Sun, 12 Nov 2023 13:09:03 +0000</updated>
                            <resolved>Thu, 2 Nov 2023 17:10:13 +0000</resolved>
                                                    <fixVersion>7.2.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>4</watches>
                                                                                                                <comments>
                            <comment id="5873993" author="JIRAUSER1268551" created="Sun, 12 Nov 2023 13:09:03 +0000"  >&lt;p&gt;From my quick playing around with things this was the best query I could come up with. I just checked today and it doesn&apos;t seem to be making a big dent so my feeling is that we do not need to backport this.&lt;/p&gt;</comment>
                            <comment id="5856401" author="tommaso.tocci" created="Mon, 6 Nov 2023 11:48:00 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alex.neben%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;alex.neben@mongodb.com&quot;&gt;alex.neben@mongodb.com&lt;/a&gt; is your query monitoring &lt;a href=&quot;https://spruce.mongodb.com/task/mongodb_mongo_master_amazon_linux2_arm64_compile_archive_dist_test_debug_83fc0730a5fa6a581eb58ca3f9040a0a1ad7009e_23_10_28_16_42_48/logs?execution=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;&lt;b&gt;archive_dist_test_debug&lt;/b&gt;&lt;/a&gt; task specifically ? I think we won&apos;t see much improvements in the other `scons compile` steps if there are no binaries to archive.&lt;/p&gt;</comment>
                            <comment id="5855422" author="JIRAUSER1268551" created="Sun, 5 Nov 2023 17:44:44 +0000"  >&lt;p&gt;I made &lt;a href=&quot;https://ui.honeycomb.io/mongodb-4b/environments/production/result/7mvxbvtAHtC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this query&lt;/a&gt; that does a (hopefully) good job at teasing out your change to see if it has a major impact overall. If the answer is yes then&lt;br/&gt;
1. We can calculate how much $$$ we saved by this change&lt;/p&gt;

&lt;p&gt;2. We can decide on a backport based on these numbers.&lt;/p&gt;

&lt;p&gt;Either way this is an awesome change! Great work!&lt;/p&gt;</comment>
                            <comment id="5848543" author="xgen-internal-githook" created="Thu, 2 Nov 2023 08:23:18 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Tommaso Tocci&apos;, &apos;email&apos;: &apos;tommaso.tocci@mongodb.com&apos;, &apos;username&apos;: &apos;toto-dev&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-82551&quot; title=&quot;Use parallel compressor to speedup binaries archival&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-82551&quot;&gt;&lt;del&gt;SERVER-82551&lt;/del&gt;&lt;/a&gt; Use parallel compressor to speedup binaries archival&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/85da569702ecb99412a087a11d6ecb3e8501df0f&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/85da569702ecb99412a087a11d6ecb3e8501df0f&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5836013" author="JIRAUSER1268551" created="Tue, 31 Oct 2023 00:43:40 +0000"  >&lt;p&gt;I think whoever is on triage rotation this (or next) week should try to take this on. This is a huge win! &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tommaso.tocci%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;tommaso.tocci@mongodb.com&quot;&gt;tommaso.tocci@mongodb.com&lt;/a&gt; if you are already working on it I don&apos;t want to steal it from you so just communicate on this ticket if you want to take it over the finish line or not.&lt;/p&gt;</comment>
                            <comment id="5833323" author="JIRAUSER1271042" created="Mon, 30 Oct 2023 19:18:44 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=tommaso.tocci%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;tommaso.tocci@mongodb.com&quot;&gt;tommaso.tocci@mongodb.com&lt;/a&gt; Alright, I will assume the python package is just unrelated and broken. Thanks for the answer.&lt;/p&gt;</comment>
                            <comment id="5833229" author="tommaso.tocci" created="Mon, 30 Oct 2023 19:10:57 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=trevor.guidry%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;trevor.guidry@mongodb.com&quot;&gt;trevor.guidry@mongodb.com&lt;/a&gt; pigz &lt;a href=&quot;https://github.com/madler/pigz/releases/tag/v2.3&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;v2.3 was released 10 years ago&lt;/a&gt; 10 years ago and nowadays is shipped as standard package in many linux distros. I don&apos;t know for sure if pigz works correctly but I would assume so &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.mongodb.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="5831817" author="JIRAUSER1271042" created="Mon, 30 Oct 2023 16:20:41 +0000"  >&lt;p&gt;Is pigz reliable? I tried using a pigz python library before &lt;a href=&quot;https://github.com/bguise987/pigz-python&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/bguise987/pigz-python&lt;/a&gt; and some of the files that were compressed by it were corrupted. It is possible the python library is implemented incorrectly and the pigz binary is fine.&#160;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="486003" name="use_pigz_compressor.patch" size="756" author="tommaso.tocci@mongodb.com" created="Sun, 29 Oct 2023 14:56:02 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25130"><![CDATA[Server Development Platform]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 30 Oct 2023 16:20:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        12 weeks, 3 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>DEVPROD-1620</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alex.neben@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            12 weeks, 3 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alex.neben@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
            <customfieldvalue>trevor.guidry@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2xw7b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i2fr2c:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3741">Dev Tools 2020-04-06</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2xicn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>