<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:07:07 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-24689] Support automatic compile bypass for non-source code changes in Evergreen patch builds</title>
                <link>https://jira.mongodb.org/browse/SERVER-24689</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;TL;DR - Patch builds can now automatically bypass compile step for non-source code changes, saving about 20-30 mins of patch build time. This would be used to modify evergreen.yml in a consistent manner when the compile step can be skipped for engineers working on test infrastructure and evergreen.yml changes.&lt;/p&gt;

&lt;p&gt;For patch builds, we implemented a method to automatically determine whether we can bypass the compilation step given the modified files in the patch. This feature has been requested for some time in order to save engineers time getting results back from patch builds. The idea is that if the files modified in the patch are deemed to not require a re-compilation of source files, then we should be able to retrieve pre-existing binaries (say from the base commit) and use those for any other tasks in the build variant. Saving the compile step time will reduce the time of patch builds by about 20-30 mins.&lt;/p&gt;

&lt;p&gt;We settled on an approach to use a whitelist of files and directories and exception lists to determine whether we should skip the compile step. This approach is a conservative method where we decide ahead of time which files and directories are considered to not require a compile. We include exception lists to explicitly call out files and directories that may overlap the whitelists or otherwise require the compile step. In essence, if all modified patch files belong to the whitelists and are not explicitly excepted, then we bypass the compile step. All other file changes will induce compilation.&lt;/p&gt;

&lt;p&gt;In evergreen.yml, we added 3 new functions to accomplish this feature.&lt;br/&gt;
1. &lt;tt&gt;&quot;get modified patch files&quot;&lt;/tt&gt;&lt;br/&gt;
2. &lt;tt&gt;&quot;bypass compile and fetch binaries&quot;&lt;/tt&gt; and&lt;br/&gt;
3. &lt;tt&gt;&quot;update bypass expansions&quot;&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;The &quot;get modified patch files&quot; function simply does a &quot;git diff HEAD --name-only&quot; to find all the files that were modified in your patch build and stores them in a file called &lt;tt&gt;patch_files.txt&lt;/tt&gt;. We check these files against the whitelist and exception lists to determine whether we bypass compile.&lt;/p&gt;

&lt;p&gt;The &quot;bypass compile and fetch binaries&quot; function takes the &lt;tt&gt;patch_files.txt&lt;/tt&gt; file and calls out to a new Python script called &lt;tt&gt;buildscripts/bypass_compile_and_fetch_binaries.py&lt;/tt&gt; that contains all the logic to determine whether to bypass the compile step and potentially generates 2 files used later in &lt;tt&gt;evergreen.yml&lt;/tt&gt;. If this function determines that compilation is required, then no files are generated; the default setting will cause the compile step to run. In the case where compilation is bypassed, one of the files generated is &lt;tt&gt;bypass_compile_expansions.yml&lt;/tt&gt; which contains the expansion macros used in evergreen.yml to bypass compile. The other file is &lt;tt&gt;artifacts.json&lt;/tt&gt; which contains the URL links to the base commit artifacts &lt;tt&gt;(binaries, mongo-shell and mongo-debugsymbols)&lt;/tt&gt; used in place of artifacts that would normally be generated during the compilation step. Of note, only certain files of the &lt;tt&gt;artifacts.tgz&lt;/tt&gt; &apos;artifact&apos; of the base commit are extracted; all others come from the patch build in order to preserve any modified patch files.&lt;/p&gt;

&lt;p&gt;The &quot;update bypass expansions&quot; function simply applies the expansions in &lt;tt&gt;bypass_compile_expansions.yml&lt;/tt&gt;. The main macro is &lt;tt&gt;bypass_compile&lt;/tt&gt; which is the knob that if enabled, bypasses the compile step in evergreen.yml.&lt;/p&gt;

&lt;p&gt;A caveat:&lt;br/&gt;
If you use a very recent commit as your base commit in your patch build, the artifacts may not be available yet and hence the bypass compile may be aborted, leading to compilation as usual.&lt;/p&gt;

&lt;p&gt;Other notes:&lt;br/&gt;
You can disable this compile bypass feature by commenting out the one line in the compile task that calls the function &lt;tt&gt;&quot;bypass compile and fetch binaries&quot;&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;We also tried the &lt;tt&gt;evergreen fetch&lt;/tt&gt; CLI command to download the base commit artifacts in one step. This afforded us parallel downloads, but ultimately we decided to go with an implementation using the new Evergreen command &lt;tt&gt;attach.artifacts&lt;/tt&gt; instead.&lt;/p&gt;

&lt;p&gt;The new Evergreen command &lt;a href=&quot;https://docs.google.com/document/d/1o8ylqJS3jmIk3l1p9RZezdNxBxHZHDfZMaYfjwN7L0A/edit#heading=h.bk9wg4dw6sr2&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;attach.artifacts&lt;/a&gt; gives us access to the &lt;tt&gt;task/&amp;lt;taskID&amp;gt;/files&lt;/tt&gt; endpoint and conveniently allows us to associate pre-existing URLs to a particular task. This saves us the duplication of storage and the cost of downloading these artifacts.&lt;/p&gt;
</description>
                <environment></environment>
        <key id="295550">SERVER-24689</key>
            <summary>Support automatic compile bypass for non-source code changes in Evergreen patch builds</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="eddie.louie">Eddie Louie</assignee>
                                    <reporter username="michael.grundy">Michael Grundy</reporter>
                        <labels>
                            <label>tig-evgconfig</label>
                    </labels>
                <created>Tue, 21 Jun 2016 17:22:24 +0000</created>
                <updated>Wed, 10 May 2023 18:34:46 +0000</updated>
                            <resolved>Tue, 20 Feb 2018 16:18:06 +0000</resolved>
                                                    <fixVersion>3.7.3</fixVersion>
                                    <component>Testing Infrastructure</component>
                                        <votes>1</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="1810239" author="xgen-internal-githook" created="Tue, 20 Feb 2018 15:49:49 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;email&apos;: &apos;eddie.louie@mongodb.com&apos;, &apos;name&apos;: &apos;Eddie Louie&apos;, &apos;username&apos;: &apos;elouie99&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24689&quot; title=&quot;Support automatic compile bypass for non-source code changes in Evergreen patch builds&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24689&quot;&gt;&lt;del&gt;SERVER-24689&lt;/del&gt;&lt;/a&gt; Support automatic compile bypass for non-source code changes in Evergreen patch builds&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/f159e5bae15513c24a2e9dbc953ce4988ea4be53&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/f159e5bae15513c24a2e9dbc953ce4988ea4be53&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1738121" author="max.hirschhorn@10gen.com" created="Thu, 30 Nov 2017 04:35:40 +0000"  >&lt;p&gt;The end result of this ticket should enable an engineer to flip a value in &lt;tt&gt;etc/evergreen.yml&lt;/tt&gt; and have pre-built binaries (e.g. from the compile task of the base commit, the last release on the branch) be downloaded by the &quot;fetch binaries&quot; function rather than compile them from scratch.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                        <issuelink>
            <issuekey id="505738">SERVER-33657</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1426586">SERVER-50078</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="2162818">SERVER-70710</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="493853">SERVER-33163</issuekey>
        </issuelink>
            <issuelink>
            <issuekey id="508074">SERVER-33764</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 30 Nov 2017 04:35:40 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 51 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>maria.vankeulen@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 51 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>eddie.louie</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.grundy</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk4tb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|htbugn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="2038">TIG 2018-1-15</customfieldvalue>
    <customfieldvalue id="2060">TIG 2018-1-1</customfieldvalue>
    <customfieldvalue id="2061">TIG 2017-12-18</customfieldvalue>
    <customfieldvalue id="2101">TIG 2018-1-29</customfieldvalue>
    <customfieldvalue id="2102">TIG 2018-02-12</customfieldvalue>
    <customfieldvalue id="2113">TIG 2018-02-26</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrku27:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>