<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:17:16 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-70839] Spawning dynamically linked mongod processes takes ~30 secs on EVG </title>
                <link>https://jira.mongodb.org/browse/SERVER-70839</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;h2&gt;&lt;a name=&quot;MotivationforRequest&quot;&gt;&lt;/a&gt;Motivation for Request&lt;/h2&gt;
&lt;p&gt;Reduce considerably core server EVG patch time.&lt;/p&gt;

&lt;h2&gt;&lt;a name=&quot;Context&quot;&gt;&lt;/a&gt;Context&lt;/h2&gt;
&lt;p&gt;As part of my Skunkwork project I tried to understand why starting up a sharded cluster using ShardingTest  takes so long. Surprisingly I discovered that just spawning the mongod processes can take up to ~30 secs on Evegreen.&lt;/p&gt;

&lt;p&gt;With &quot;spawning&quot; I mean the time that pass from when we &lt;span class=&quot;error&quot;&gt;&amp;#91;call execve| we call execve&amp;#93;&lt;/span&gt; to the time we &lt;a href=&quot;https://github.com/mongodb/mongo/blob/460f9e9ecdd209f68d2271241a8bc3b04d8f59d7/src/mongo/db/mongod_main.cpp#L1465&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;execute the first instruction  in mongod_main&lt;/a&gt;. &lt;br/&gt;
For instance in &lt;a href=&quot;https://evergreen.mongodb.com/lobster/build/98d630b2ce105acf7ddb8ccaf09ddcb2/test/62d6e06754f2487914b37399&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;this patch&lt;/a&gt; it took ~34 secs to spawn 1 mongod process:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:shardingtest_control_12_nodes] d22050| XOXO execve at time: Mon Jul 18 14:51:16 2022&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;[js_test:shardingtest_control_12_nodes] d22050| XOXO mongod_main start: Mon Jul 18 14:51:25 2022&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;As of today we run ~883 tests in every of the 10 sharding suites, each test take ~30 secs just to spawn the process required to setup the cluster. So only accounting for sharding tests, we are spending at least 73 AWS machine hours per patch per variant just to spawn those processes.&lt;/p&gt;

&lt;p&gt;It seems we have a huge space for improvement here.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2103439">SERVER-70839</key>
            <summary>Spawning dynamically linked mongod processes takes ~30 secs on EVG </summary>
                <type id="6" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14720&amp;avatarType=issuetype">Question</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="backlog-server-devplatform">[DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP)</assignee>
                                    <reporter username="tommaso.tocci@mongodb.com">Tommaso Tocci</reporter>
                        <labels>
                    </labels>
                <created>Mon, 1 Aug 2022 11:37:42 +0000</created>
                <updated>Wed, 14 Dec 2022 17:32:21 +0000</updated>
                            <resolved>Wed, 14 Dec 2022 17:32:21 +0000</resolved>
                                                                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="4925656" author="acm" created="Tue, 25 Oct 2022 15:16:36 +0000"  >&lt;p&gt;We need to do a little thinking about exactly which scenarios admit dropping &lt;tt&gt;z,now&lt;/tt&gt; but I agree we should do this. I also expect that &lt;tt&gt;--experimental-optimization=&amp;#43;{vishidden,fnsi&lt;/tt&gt;} may offer even another 30% reduction in startup times.&lt;/p&gt;</comment>
                            <comment id="4924593" author="tommaso.tocci" created="Tue, 25 Oct 2022 08:27:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=alexander.neben%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;alexander.neben@mongodb.com&quot;&gt;alexander.neben@mongodb.com&lt;/a&gt; I already did some comparison analysis and it seems that lazy symbol resolution would reduce overall EVG patch time.&lt;/p&gt;

&lt;p&gt;You can analyze the result by comparing:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;&lt;a href=&quot;https://spruce.mongodb.com/version/6352d72a61837d2bb0451e2f/task-duration?duration=DESC&amp;amp;page=0&amp;amp;statuses=success&amp;amp;variant=%5Eenterprise-rhel-80-64-bit-dynamic-required%24&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;A patch with lazy symbol resolution (without &lt;tt&gt;-z now&lt;/tt&gt;)&lt;/a&gt;&lt;/li&gt;
	&lt;li&gt;&lt;a href=&quot;https://spruce.mongodb.com/version/mongodb_mongo_master_9570d902706a9a1c456e1a6d78b555a92fbd23dc/task-duration?duration=DESC&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;The base patch without lazy symbol resolution&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;



&lt;p&gt;In RHEL 8.0 variant I observed tasks run time reduction that span between 5% and 20%&lt;/p&gt;</comment>
                            <comment id="4921437" author="tommaso.tocci" created="Mon, 24 Oct 2022 07:32:23 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=andrew.morrow%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;andrew.morrow@mongodb.com&quot;&gt;andrew.morrow@mongodb.com&lt;/a&gt; removing the -z now linking flag we have a ~30% startup time improvement, I&apos;ve opened &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-70797&quot; title=&quot;Reduce mongod startup time by using lazy symbol resolution&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-70797&quot;&gt;&lt;del&gt;DAG-2230&lt;/del&gt;&lt;/a&gt; to introduce the change.&lt;/p&gt;</comment>
                            <comment id="4729610" author="JIRAUSER1268551" created="Thu, 4 Aug 2022 18:53:07 +0000"  >&lt;p&gt;This might fall more on SDP than DAG and there are a ton of implications here.&lt;br/&gt;
For example, recently we just &quot;ran out of binary space&quot; and can no longer link statically without some other tradeoffs.&lt;/p&gt;

&lt;p&gt;Thank you for getting us this data and I hope there is something we can do about it.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="2164351">SERVER-70797</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1449563">SERVER-50476</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2104102">SERVER-68475</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="1665187">SERVER-55651</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2034615">SERVER-66014</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2164351">SERVER-70797</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="76292">SERVER-9761</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>4.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25130"><![CDATA[Server Development Platform]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 4 Aug 2022 18:53:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        1 year, 15 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-3103</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alex.neben@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            1 year, 15 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-devplatform</customfieldvalue>
            <customfieldvalue>alex.neben@mongodb.com</customfieldvalue>
            <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>tommaso.tocci@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i14jmf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i0ncp4:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i145rr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>