<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:10:39 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-46161] Reduce frequency of yielding during query execution</title>
                <link>https://jira.mongodb.org/browse/SERVER-46161</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;By default, any auto-yielding queries will yield every &lt;a href=&quot;https://github.com/mongodb/mongo/blob/efa43f5170ab702319128770b0347df0bb2bd8b8/src/mongo/db/query/query_knobs.idl#L210-L215&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;128 work cycles&lt;/a&gt; or &lt;a href=&quot;https://github.com/mongodb/mongo/blob/efa43f5170ab702319128770b0347df0bb2bd8b8/src/mongo/db/query/query_knobs.idl#L217-L224&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;10 milliseconds&lt;/a&gt;, whichever comes first. Profile data shows that this yielding process is CPU intensive: we propagate &lt;tt&gt;saveState()&lt;/tt&gt; and &lt;tt&gt;restoreState()&lt;/tt&gt; throughout the execution tree, we call into a special lock manager functions to release and reacquire locks, and we save/restore any storage cursors.&lt;/p&gt;

&lt;p&gt;When MMAPv1 was supported, yielding frequently was necessary to allow writers to proceed. Now that the supported storage engines all support document-level concurrency, yielding is only necessary for the following:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;It&apos;s where our interrupt checks are housed in query execution.&lt;/li&gt;
	&lt;li&gt;It&apos;s where we call &lt;tt&gt;abandonSnapshot()&lt;/tt&gt;, allowing the storage engine to relinquish any resources necessary to hold open the snapshot.&lt;/li&gt;
	&lt;li&gt;Yielding our intent locks allows operations which require strong database or collection locks to proceed.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I&apos;ll address these points one-by-one. First, checking for interrupt regularly is still necessary. But there is no indication that the interrupt check itself is slow. We can simply check for interrupt in between every call to &lt;tt&gt;PlanExecutor::work()&lt;/tt&gt;, decoupling interrupt checking from yielding. Abandoning the snapshot is still necessary, but there&apos;s no evidence that doing it less frequently is problematic. Similarly, allowing operations which require strong locks to make progress is still necessary. But the storage team has been working to remove as many strong lock acquisitions as possible. Furthermore, operations which take strong locks are typically not on the hot path, and so it makes sense to block them for up to 10ms in order to allow query workloads to be faster.&lt;/p&gt;</description>
                <environment></environment>
        <key id="1149741">SERVER-46161</key>
            <summary>Reduce frequency of yielding during query execution</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13201">Fixed</resolution>
                                        <assignee username="david.storch@mongodb.com">David Storch</assignee>
                                    <reporter username="david.storch@mongodb.com">David Storch</reporter>
                        <labels>
                    </labels>
                <created>Fri, 14 Feb 2020 15:25:35 +0000</created>
                <updated>Sun, 29 Oct 2023 22:12:12 +0000</updated>
                            <resolved>Sat, 22 Feb 2020 16:32:32 +0000</resolved>
                                                    <fixVersion>4.3.4</fixVersion>
                                    <component>Querying</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="2890677" author="xgen-internal-githook" created="Sat, 22 Feb 2020 03:14:53 +0000"  >&lt;p&gt;Author:&lt;/p&gt;
{&apos;name&apos;: &apos;David Storch&apos;, &apos;username&apos;: &apos;dstorch&apos;, &apos;email&apos;: &apos;david.storch@mongodb.com&apos;}
&lt;p&gt;Message: &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-46161&quot; title=&quot;Reduce frequency of yielding during query execution&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-46161&quot;&gt;&lt;del&gt;SERVER-46161&lt;/del&gt;&lt;/a&gt; Increase number of PlanExecutor iterations before yielding to 1000.&lt;/p&gt;

&lt;p&gt;The new behavior is for queries to yield every 10ms or 1000&lt;br/&gt;
iterations, whichever comes first. This change improves&lt;br/&gt;
performance on many of our workloads, since yielding is&lt;br/&gt;
expensive and happens more frequently than needed on those&lt;br/&gt;
workloads.&lt;br/&gt;
Branch: master&lt;br/&gt;
&lt;a href=&quot;https://github.com/mongodb/mongo/commit/460acb836445e42960346bdb95a83553db8df018&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/460acb836445e42960346bdb95a83553db8df018&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="2879098" author="geert.bosch" created="Fri, 14 Feb 2020 18:28:27 +0000"  >&lt;p&gt;I have done experiments with the yield frequency at various points during the 3.4-3.6 timeframe, because I wondered whether this would help for simple queries, such as collection scans with simple predicates and whether we should have different defaults for MMAPv1 and WiredTiger. I did not see any significant benefits from yielding less often, so didn&apos;t pursue this. Of course this may have changed. How much time does a yield take compared to executing a work cycle?&lt;/p&gt;

&lt;p&gt;There is a balance between the cost of yielding that you described and the cost of not yielding, especially in update-heavy workloads where reads may block on I/O. So, I would advocate for stepwise/gradual change of the query knobs if supported by concrete benchmarking evidence. So, you might want to increase the number of work cycles to 256 or up to 1000. &lt;/p&gt;

&lt;p&gt;I&apos;m not sure it would really help to check for interruption at every work. Checking every 10ms seems fine-grained enough. &lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                            <issuelinktype id="10520">
                    <name>Problem/Incident</name>
                                            <outwardlinks description="causes">
                                                        </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2380797">SERVER-78556</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 14 Feb 2020 18:28:27 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 50 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_17050" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Downstream Team Attention</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="16941"><![CDATA[Not Needed]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 50 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>50.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>david.storch@mongodb.com</customfieldvalue>
            <customfieldvalue>geert.bosch@mongodb.com</customfieldvalue>
            <customfieldvalue>xgen-internal-githook</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwqox3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hwensf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="3292">Query 2020-02-24</customfieldvalue>
    <customfieldvalue id="3494">Query 2020-03-09</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hwqb6f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>