<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:38:47 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-78603] Introduce a resmoke option to allow benchmarks to be run in parallel</title>
                <link>https://jira.mongodb.org/browse/SERVER-78603</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Currently, resmoke.py doesn&apos;t allow benchmarks to be run concurrently (-j must be 1) and rightfully so. However, there are cases where I want to confirm locally that some changes that I have made didn&apos;t break them (as in, didn&apos;t cause them to crash or fail in some unexpected way, as opposed to introduction of some performance regression).&lt;/p&gt;

&lt;p&gt;It would be nice to have a target called something like &quot;smoke-benchmarks&quot; or something like that which allows me to specify -j$(nproc --all) with which I can smoke test that they all complete without errors (just like unittests). &lt;/p&gt;</description>
                <environment></environment>
        <key id="2382457">SERVER-78603</key>
            <summary>Introduce a resmoke option to allow benchmarks to be run in parallel</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="alex.neben@mongodb.com">Alex Neben</assignee>
                                    <reporter username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</reporter>
                        <labels>
                    </labels>
                <created>Mon, 3 Jul 2023 11:07:58 +0000</created>
                <updated>Sun, 29 Oct 2023 21:19:18 +0000</updated>
                            <resolved>Tue, 11 Jul 2023 22:15:00 +0000</resolved>
                                                    <fixVersion>7.1.0-rc0</fixVersion>
                                                        <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="5557970" author="xgen-internal-githook" created="Tue, 11 Jul 2023 17:01:20 +0000"  >&lt;p&gt;Author: &lt;/p&gt;
{&apos;name&apos;: &apos;Alex Neben&apos;, &apos;email&apos;: &apos;alexander.neben@mongodb.com&apos;, &apos;username&apos;: &apos;IamXander&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-78603&quot; title=&quot;Introduce a resmoke option to allow benchmarks to be run in parallel&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-78603&quot;&gt;&lt;del&gt;SERVER-78603&lt;/del&gt;&lt;/a&gt; Remove single core limit on benchmarks&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/3a1d2d8b22eff8b0f4fa708cd85e977329cff0de&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/3a1d2d8b22eff8b0f4fa708cd85e977329cff0de&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="5556202" author="JIRAUSER1268551" created="Tue, 11 Jul 2023 00:37:13 +0000"  >&lt;p&gt;As mentioned in slack&lt;/p&gt;

&lt;p&gt;I have put up a&#160;&lt;a href=&quot;https://github.com/10gen/mongo/pull/14132&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;PR&lt;/a&gt; that removes the limit of -j=1. I have also included steps to run benchmarks and what the expected output should be. We believe that these benchmarks and not independent and running more than one at the same time causes them to fail.&lt;/p&gt;</comment>
                            <comment id="5555341" author="kaloian.manassiev" created="Mon, 10 Jul 2023 18:24:20 +0000"  >&lt;p&gt;What is PB?&lt;br/&gt;
Also, anything that&apos;s &quot;close to an hour&quot; will not satisfy my requirement.&lt;br/&gt;
My requirement is: I make some SConscript linking change or change some fundamental library and I want to very quickly, using all the resources of my virtual workstation (meaning all its 8 CPUs) to be able to validate that I didn&apos;t break the benchmarks.&lt;/p&gt;

&lt;p&gt;I really don&apos;t get it why can&apos;t you just allow -j to work for benchmarks for values greater than 1. I already can do something dumb like run benchmarks on a debug build and it emits a warning, so it&apos;s the same logic for -j8.&lt;/p&gt;</comment>
                            <comment id="5555158" author="JIRAUSER1274045" created="Mon, 10 Jul 2023 17:36:11 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev@mongodb.com&quot;&gt;kaloian.manassiev@mongodb.com&lt;/a&gt;: if we speed up the benchmark wallclock execution time in PB (say, to less than an hour), will that meet your needs? IIUC, that will let you verify reasonably quickly that your change does(n&apos;t) break benchmarks. WDYT?&lt;/p&gt;

&lt;p&gt;(Note: It&apos;s not theoretical; we are actively exploring ways to speed up benchmark testing in PB).&lt;/p&gt;</comment>
                            <comment id="5545027" author="JIRAUSER1253549" created="Wed, 5 Jul 2023 16:52:44 +0000"  >&lt;p&gt;I think there are two options we can do moving forward to help this situation.&lt;/p&gt;

&lt;p&gt;1. Create suites which could be used like &quot;--suites=concurrent_benchmarks,serial_benchmarks&quot; to allow some of the suites that can safely run in parallel to do so, and contain others which must be run serially.&lt;br/&gt;
2. Determine an acceptable and valid minimal --benchmark_min_time value to use, and add it as an option to resmoke or combined in to some specific resmoke benchmark suite config whose goal is speed (such as the proposed suites from 1).&lt;/p&gt;</comment>
                            <comment id="5545021" author="jason.chan" created="Wed, 5 Jul 2023 16:50:56 +0000"  >&lt;p&gt;Assigning back to SDP. Talking to &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.moody%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.moody@mongodb.com&quot;&gt;daniel.moody@mongodb.com&lt;/a&gt;, we think a solution could be to utilize the &lt;tt&gt;--benchmark_min_time&lt;/tt&gt; option to drastically shrink the run-time of of the benchmarks, and then have a concurrent test suite that runs them in parallel. There may be difficulties in the filesystem benchmarks which may require them to be run in a serial suite instead. &lt;/p&gt;</comment>
                            <comment id="5541477" author="kaloian.manassiev" created="Mon, 3 Jul 2023 20:45:17 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.moody%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.moody@mongodb.com&quot;&gt;daniel.moody@mongodb.com&lt;/a&gt;, this doesn&apos;t help me. What I need to test (and I have introduced a number of BFs recently) is that a change I make in SConscript for example or in some central decoration initialiser, doesn&apos;t cause any of the benchmarks to crash. This is as opposed to ensuring that the performance is the same.&lt;/p&gt;

&lt;p&gt;All I care about is that we remove the restriction of &lt;tt&gt;--j=1&lt;/tt&gt; for benchmarks. The rest of the tools and evergreen can continue running with 1.&lt;/p&gt;

&lt;p&gt;Does that make sense?&lt;/p&gt;</comment>
                            <comment id="5541427" author="JIRAUSER1253549" created="Mon, 3 Jul 2023 20:04:15 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=kaloian.manassiev%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;kaloian.manassiev@mongodb.com&quot;&gt;kaloian.manassiev@mongodb.com&lt;/a&gt; we have a high level target &quot;prove-benchmarks&quot; which will only run benchmarks which have incurred changes from that last time it was run, so essentially it will only rerun benchmarks which are possibly affected, even such as changing some include header. Rerunning with no changes, will quickly (&quot;scons&quot; quickly, so like 30-45 seconds) reprint the last test results (pass or fail).&lt;/p&gt;

&lt;p&gt;This does mean there is an upfront cost to run all the tests, but then after that, iterations should be faster and only run a small set of affected benchmarks (unless you working on some core header which affects everything). Is this helpful for your situation?&lt;/p&gt;</comment>
                            <comment id="5541425" author="billy.donahue" created="Mon, 3 Jul 2023 20:03:46 +0000"  >&lt;p&gt;The Google Benchmarks API doesn&apos;t really offer a &quot;check for correctness&quot; mode of operation, at least not directly.&lt;br/&gt;
The benchmark API reserves wide latitude on how much time the run will consume.&lt;/p&gt;

&lt;p&gt;The is a &lt;tt&gt;--benchmark_min_time&lt;/tt&gt;, documented directly in the code.&lt;br/&gt;
&lt;a href=&quot;https://github.com/10gen/mongo/blob/244f33b7433c74c470855864f3ae6b167decbd89/src/third_party/benchmark/dist/src/benchmark.cc#L62-L67&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/10gen/mongo/blob/244f33b7433c74c470855864f3ae6b167decbd89/src/third_party/benchmark/dist/src/benchmark.cc#L62-L67&lt;/a&gt;&lt;/p&gt;


&lt;p&gt;What the documentation doesn&apos;t mention about this flag is that it also informs the benchmark maximum time.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/google/benchmark/blob/main/docs/user_guide.md#runtime-and-reporting-considerations&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/google/benchmark/blob/main/docs/user_guide.md#runtime-and-reporting-considerations&lt;/a&gt;&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;In all cases, the number of iterations for which the benchmark is run is governed by the amount of time the benchmark takes. Concretely, the number of iterations is at least one, not more than 1e9, until CPU time is greater than the minimum time, or the wallclock time is 5x minimum time. The minimum time is set per benchmark by calling MinTime on the registered benchmark object.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;This &lt;tt&gt;MinTime&lt;/tt&gt; defaults to the &lt;tt&gt;--benchmark_min_time&lt;/tt&gt; given on the command line for the benchmark. The default value is 0.5 seconds, btw.&lt;br/&gt;
So I think there&apos;s a path for resmoke to try running benchmarks with this flag set to an &quot;epsilon&quot; value to produce &quot;correctness&quot; executions, and then maybe those can be parallelized by tweaking resmoke.py.&lt;/p&gt;

&lt;p&gt;So I think this is all about the command lines that resmoke uses to execute benchmarks, and wouldn&apos;t involve Service Architecture.&lt;br/&gt;
If the CLI doesn&apos;t actually work this way, I guess we&apos;d have to take another look, though.&lt;/p&gt;</comment>
                            <comment id="5541305" author="JIRAUSER1253549" created="Mon, 3 Jul 2023 18:54:24 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason.chan%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;jason.chan@mongodb.com&quot;&gt;jason.chan@mongodb.com&lt;/a&gt; even if we are talking about correctness, does that imply someone (or service arch) will have increased maintenance overhead for the extra mode for these tests? Lets say we do add &quot;concurrent correctness mode&quot;, if the benchmarks fail because they are running parallel, what does that imply or mean? How does one differentiate that failure? &lt;/p&gt;

&lt;p&gt;I am not opposed to the idea, but I am tying to understand exactly how much value is here.&lt;/p&gt;</comment>
                            <comment id="5541153" author="jason.chan" created="Mon, 3 Jul 2023 17:30:17 +0000"  >&lt;p&gt;I can assign this to Service Arch to triage to answer questions around whether we think there&apos;s anything interesting in regards to parallelizing the workloads, but my understanding of the request here is that it&apos;s mostly about completing the workloads without failure, as opposed to measuring actual performance. We can discuss the ownership of this once I gather feedback on &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.moody%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.moody@mongodb.com&quot;&gt;daniel.moody@mongodb.com&lt;/a&gt;&apos;s questions.&lt;/p&gt;

&lt;p&gt;I&apos;d be interested to get SDP&apos;s opinions as well on whether there are alternatives we haven&apos;t explored here around getting a better signal so developers can better catch things like this locally.&lt;/p&gt;</comment>
                            <comment id="5541133" author="JIRAUSER1253549" created="Mon, 3 Jul 2023 17:24:06 +0000"  >&lt;p&gt;The question is why is it intentionally serial in the first place? Are the benchmark tests sensitive to running in parallel? They seem to be measuring performance, if they are running concurrently will that impact the results?&lt;/p&gt;

&lt;p&gt;Yes the implementation of running some test concurrently is in resmoke, and other test list based tests (like unittests) already do this, but is that what we really want to do for these benchmark tests?&lt;/p&gt;</comment>
                            <comment id="5541119" author="JIRAUSER1274045" created="Mon, 3 Jul 2023 17:20:02 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=daniel.moody%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;daniel.moody@mongodb.com&quot;&gt;daniel.moody@mongodb.com&lt;/a&gt;, can you weigh in here? IIRC, the discussion centered on the desirability of parallelization in the first place...?&lt;/p&gt;</comment>
                            <comment id="5540859" author="JIRAUSER1274045" created="Mon, 3 Jul 2023 15:54:58 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=blake.oler%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;blake.oler@mongodb.com&quot;&gt;blake.oler@mongodb.com&lt;/a&gt; &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=jason.chan%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;jason.chan@mongodb.com&quot;&gt;jason.chan@mongodb.com&lt;/a&gt; Per SDP discussion, it sounds like this would be owned by service arch. Can y&apos;all advise?&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>14.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25130"><![CDATA[Server Development Platform]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 3 Jul 2023 15:54:58 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        30 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            30 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alex.neben@mongodb.com</customfieldvalue>
            <customfieldvalue>billy.donahue@mongodb.com</customfieldvalue>
            <customfieldvalue>daniel.moody@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
            <customfieldvalue>jason.chan@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>steve.gross@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2gb3z:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1ygwo:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                            <customfield id="customfield_22250" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Special Downgrade Instructions Required</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="23343"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2fx9b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>