<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:18:34 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-28595] Understand the cause of slow incremental rebuild times</title>
                <link>https://jira.mongodb.org/browse/SERVER-28595</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description></description>
                <environment></environment>
        <key id="370522">SERVER-28595</key>
            <summary>Understand the cause of slow incremental rebuild times</summary>
                <type id="3" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14718&amp;avatarType=issuetype">Task</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="andrew.morrow@mongodb.com">Andrew Morrow</assignee>
                                    <reporter username="justin.cohler">Justin Cohler</reporter>
                        <labels>
                    </labels>
                <created>Mon, 3 Apr 2017 20:06:13 +0000</created>
                <updated>Wed, 8 Dec 2021 19:47:54 +0000</updated>
                            <resolved>Thu, 30 Jul 2020 15:32:56 +0000</resolved>
                                                                    <component>Build</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1580351" author="bdbaddog" created="Thu, 25 May 2017 20:31:19 +0000"  >&lt;h1&gt;&lt;a name=&quot;Notes%28allarefornullincrementalbuilds%29&quot;&gt;&lt;/a&gt;Notes (all are for null incremental builds)&lt;/h1&gt;
&lt;ul&gt;
	&lt;li&gt;Cached Configure is about 10% of null build runtime (approx 5 seconds)&lt;/li&gt;
	&lt;li&gt;Writing sconsign is about 20% of null build runtime (approx 10 seconds)&lt;/li&gt;
	&lt;li&gt;If we can safely enable caching changed() in FS.py that  5-10% runtime (approx 2-5 seconds)&lt;/li&gt;
	&lt;li&gt;Re regex called under get_content() this is due to logic to rebuild the command line to compare against previous build.&lt;/li&gt;
	&lt;li&gt;For 96k calls to subst, approximately 16k are for blank strings.. (typically via get_suffix and get_prefix)&lt;/li&gt;
	&lt;li&gt;Of all calls to subst for mongo build&lt;/li&gt;
	&lt;li&gt;24520 have $ in them (457 unique strings)&lt;/li&gt;
	&lt;li&gt;71517 are plain strings&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Looks like many places where micro-optimizations may help. &lt;br/&gt;
(Changing list uniquifying code for example)&lt;/p&gt;</comment>
                            <comment id="1555102" author="bdbaddog" created="Fri, 21 Apr 2017 19:07:40 +0000"  >&lt;p&gt;A few notes on what I&apos;ve investigated so far.&lt;br/&gt;
Tried commenting out sconsign writes.&lt;br/&gt;
Regular Null build: ~55s&lt;br/&gt;
With sconsign writing commented out: ~45s&lt;br/&gt;
Sadly pickle writes can&apos;t be done incrementally so in the case of a single file change unless sconsign is changed to some other method of persistance no improvement here.&lt;br/&gt;
I did try bumping to highest pickle version (From the current of 1), the difference (no my system) was not measurable (maybe +/- 1s)&lt;/p&gt;

&lt;p&gt;Also of note. dblite does have a &quot;dirty&quot; flag to indicate when something has been changed in what it expects to write and to only write when it&apos;s dirty.  So even with a null build, it&apos;s still getting set and writing. &lt;/p&gt;

&lt;p&gt;I did see a few benchmarks which noted that using JSON for persistance may be an order of magnitude faster.  May be worth investigating.&lt;/p&gt;</comment>
                            <comment id="1548815" author="acm" created="Thu, 13 Apr 2017 19:45:21 +0000"  >&lt;p&gt;We have cherry-picked that change in: &lt;a href=&quot;https://github.com/mongodb/mongo/commit/59681ee6603fc43f0f3209ec0f9c6a09476edfcc&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/59681ee6603fc43f0f3209ec0f9c6a09476edfcc&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1548805" author="bdbaddog" created="Thu, 13 Apr 2017 19:35:51 +0000"  >&lt;p&gt;Looks like you have scons-2.5.0 in your third_party sources.&lt;br/&gt;
Any reason you haven&apos;t upgraded to 2.5.1?&lt;br/&gt;
There is one performance update in 2.5.1.&lt;br/&gt;
  From Alexey Klimkin:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Use memoization to optimize PATH evaluation across all dependencies per&lt;br/&gt;
      node. (PR #345)&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="1541807" author="acm" created="Wed, 5 Apr 2017 18:19:40 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=bdbaddog&quot; class=&quot;user-hover&quot; rel=&quot;bdbaddog&quot;&gt;bdbaddog&lt;/a&gt; -&lt;/p&gt;

&lt;p&gt;I&apos;ve attached profiling data to this ticket, you should be able to view it with the above commands. Here are some observations:&lt;/p&gt;

&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;About 11% of the time spent in &lt;tt&gt;_build_targets&lt;/tt&gt; was spent writing back the sconsign file, even though nothing changed. Ideally, only objects which actually had changes should be re-written.&lt;/li&gt;
	&lt;li&gt;We spend about half of the time in &lt;tt&gt;changed&lt;/tt&gt;, over the course of about 20k calls. Many of these calls appear to be redundant. In particular, &lt;tt&gt;make_ready_current&lt;/tt&gt; causes two calls to &lt;tt&gt;changed&lt;/tt&gt; - one direct, and the other indirect, through &lt;tt&gt;release_target_info&lt;/tt&gt;. It seems likely that either &lt;tt&gt;release_target_info&lt;/tt&gt; does not need to call &lt;tt&gt;changed&lt;/tt&gt;, or that it could at least re-use the knowledge that &lt;tt&gt;make_ready_current&lt;/tt&gt; gleaned from its call to &lt;tt&gt;changed&lt;/tt&gt;. Also notable is that scan also calls into &lt;tt&gt;is_up_to_date&lt;/tt&gt;. This appears to be another potential avenue for re-use, since the &lt;tt&gt;scan&lt;/tt&gt; is reached from &lt;tt&gt;_find_next_ready_node&lt;/tt&gt;, and &lt;tt&gt;make_ready_current&lt;/tt&gt; is called from &lt;tt&gt;make_ready&lt;/tt&gt;, but both &lt;tt&gt;_find_next_ready_node&lt;/tt&gt; and &lt;tt&gt;make_ready&lt;/tt&gt; are direct children of &lt;tt&gt;next_task&lt;/tt&gt;&lt;/li&gt;
	&lt;li&gt;The 20k calls to to &lt;tt&gt;changed&lt;/tt&gt; are rather pernicious, as they result in 4.5 million calls to the decider subsystem via &lt;tt&gt;decide_source&lt;/tt&gt;. When that call chain is followed, only 9k calls resulted in a time spent talking to the filesystem via &lt;tt&gt;stat&lt;/tt&gt;, since the result is memoized. However, it appears that the memoization is performed at too deep a layer in the object structure. If the memoization was moved higher, a lot of python execution time could be removed.&lt;/li&gt;
	&lt;li&gt;About half of the calls to &lt;tt&gt;changed&lt;/tt&gt; spend significant time in &lt;tt&gt;get_contents&lt;/tt&gt;. In this case, we aren&apos;t actually getting the file contents, since we are using the MD5-Timestamp decider. Instead, we are just doing substitution. It is not clear why this is so expensive, though a lot of it seems to be time spent in regex. Perhaps a faster regex library than &lt;tt&gt;re&lt;/tt&gt; is in order? Or perhaps there is something we can do to reduce the substitution load we are asking of SCons?&lt;/li&gt;
	&lt;li&gt;Moving back up to &lt;tt&gt;build_targets&lt;/tt&gt;, we spend about 14% of our time in &lt;tt&gt;get_stored_implicit&lt;/tt&gt;, because we are building with &lt;tt&gt;--implicit-cache&lt;/tt&gt;. The majority of that time is spent in &lt;tt&gt;str_to_node&lt;/tt&gt;. Much like with the amplification in the calls to &lt;tt&gt;changed&lt;/tt&gt;, &lt;tt&gt;get_stored_implicit&lt;/tt&gt; makes 4k calls to &lt;tt&gt;prepare_dependencies&lt;/tt&gt;, which causes 1.5 million calls to &lt;tt&gt;str_to_node&lt;/tt&gt;. Improving both the runtime of &lt;tt&gt;str_to_node&lt;/tt&gt; itself, and if possible re-using state to avoid redundant calls to &lt;tt&gt;str_to_node&lt;/tt&gt; seems like a potential avenue for improvement.&lt;/li&gt;
	&lt;li&gt;Also of interest is to note that 12k calls to &lt;tt&gt;&lt;em&gt;find_next_ready_node&lt;/tt&gt; causes somewhat more than 3 million calls to &lt;tt&gt;&lt;/em&gt;&lt;em&gt;iter&lt;/em&gt;_&lt;/tt&gt;, for almost 14% of the total time in &lt;tt&gt;_build_targets&lt;/tt&gt;. It is not clear to me what part of &lt;tt&gt;_find_next_ready_node&lt;/tt&gt; is doing this or whether it can be improved, but it seems worth looking into.&lt;/li&gt;
&lt;/ul&gt;

</comment>
                            <comment id="1541729" author="acm" created="Wed, 5 Apr 2017 17:30:39 +0000"  >&lt;p&gt;I&apos;m attaching profile data for a no-op SCons rebuild, run as:&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;python -m cProfile -o profile_data.pyprof buildscripts/scons.py --implicit-cache --dbg=on --variables-files=etc/scons/xcode_macosx.vars --debug=time,count,memoizer all -j8&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;You can view this data with&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;pyprof2calltree -i ./profile_data.pyprof -k&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                        <issuelink>
            <issuekey id="143747">SERVER-14343</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="153317" name="profile_data.pyprof" size="1115112" author="andrew.morrow@mongodb.com" created="Wed, 5 Apr 2017 17:29:18 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>11.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 5 Apr 2017 17:30:39 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 37 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10857" key="com.pyxis.greenhopper.jira:gh-epic-link">
                        <customfieldname>Epic Link</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>PM-325</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 37 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>andrew.morrow@mongodb.com</customfieldvalue>
            <customfieldvalue>bdbaddog#1</customfieldvalue>
            <customfieldvalue>justin.cohler</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|ht57tz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr4mfz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1658">Platforms 2017-04-17</customfieldvalue>
    <customfieldvalue id="2838">Dev Tools 2019-03-25</customfieldvalue>
    <customfieldvalue id="2839">Dev Tools 2019-04-08</customfieldvalue>
    <customfieldvalue id="2910">Dev Tools 2019-05-06</customfieldvalue>
    <customfieldvalue id="2942">Dev Tools 2019-04-22</customfieldvalue>
    <customfieldvalue id="3112">Dev Tools 2019-08-12</customfieldvalue>
    <customfieldvalue id="3113">Dev Tools 2019-08-26</customfieldvalue>
    <customfieldvalue id="3246">Dev Tools 2019-09-09</customfieldvalue>
    <customfieldvalue id="3280">Dev Tools 2019-09-23</customfieldvalue>
    <customfieldvalue id="3281">Dev Tools 2019-10-07</customfieldvalue>
    <customfieldvalue id="3282">Dev Tools 2019-10-21</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs4can:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>