<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:43:33 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-80436] Reduce read-write contention on (query stats) partitions</title>
                <link>https://jira.mongodb.org/browse/SERVER-80436</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;&lt;tt&gt;$queryStats&lt;/tt&gt; is an aggregation query that does a read-only scan of the query stats cache, which is implemented as a collection of 16MB partitions. Even at a generous cache size of 300MB (the expected size on Atlas clusters), this results in 19 partitions.&lt;/p&gt;

&lt;p&gt;If there are many &lt;tt&gt;$queryStats&lt;/tt&gt; aggregation queries coming in to take a lock on each partition, the entire cache becomes a bottleneck for concurrent &lt;tt&gt;find&lt;/tt&gt; commands, which perform short-running read-update-writes to one specific partition.&lt;/p&gt;

&lt;p&gt;Currently, this is being mitigated by minimizing the lock time for the &lt;tt&gt;$queryStats&lt;/tt&gt; aggregation queries by copying the entire partition out, effectively creating a read-only view of the partition. This still requires copying 16MB under a lock, however, and the impact can be very visible. (See attachment, which shows massive performance degradation with 32 concurrent &lt;tt&gt;$queryStats&lt;/tt&gt; aggregations).&lt;/p&gt;

&lt;p&gt;Although we do not expect many concurrent &lt;tt&gt;$queryStats&lt;/tt&gt; aggregations at the moment, this can change in the future. And if it does, we will want to consider further improvements in the contention between the long-running read-only &lt;tt&gt;$queryStats&lt;/tt&gt; aggregations and the short-running read-update-write &lt;tt&gt;find&lt;/tt&gt; commands.&lt;/p&gt;

&lt;p&gt;Some basic ideas to possibly consider:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Finer grained locking for hash tables (to allow for more partitions) can decrease the contention between readers and writers. Since &lt;tt&gt;$queryStats&lt;/tt&gt; aggregation throughput is not particularly time-sensitive, but &lt;tt&gt;find&lt;/tt&gt; command throughput is, it&apos;s reasonable to trade long-running read-only performance losses for short-running read-write-update performance gains.&lt;/li&gt;
	&lt;li&gt;Lock-free data structures can provide some help here. Lock-free queues can help with LRU implementation.&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="2427053">SERVER-80436</key>
            <summary>Reduce read-write contention on (query stats) partitions</summary>
                <type id="4" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14710&amp;avatarType=issuetype">Improvement</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-query-integration">Backlog - Query Integration</assignee>
                                    <reporter username="william.qian@mongodb.com">William Qian</reporter>
                        <labels>
                            <label>former-pm-2885</label>
                            <label>qi-query-stats</label>
                            <label>query-skunkworks</label>
                    </labels>
                <created>Fri, 25 Aug 2023 17:37:14 +0000</created>
                <updated>Thu, 4 Jan 2024 20:36:54 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>2</watches>
                                                                                                                <comments>
                            <comment id="5752327" author="charlie.swanson" created="Wed, 4 Oct 2023 15:22:34 +0000"  >&lt;p&gt;flagging for scheduling and removing from PM-2885 - my recommendation would be to close as &quot;Won&apos;t Do.&quot; It is a good idea that can only help, but I don&apos;t see this as a huge problem for query stats except on multi-socket machines (which have their own problems).&lt;/p&gt;

&lt;p&gt;There are rumors from &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=william.qian%40mongodb.com&quot; class=&quot;user-hover&quot; rel=&quot;william.qian@mongodb.com&quot;&gt;william.qian@mongodb.com&lt;/a&gt; of some skunkworks ideas that could bring this back to life if they prove promising.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="471660" name="image.png" size="54612" author="william.qian@mongodb.com" created="Fri, 25 Aug 2023 17:32:05 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25467"><![CDATA[Query Integration]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 4 Oct 2023 15:22:34 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        18 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>brenda.rodriguez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            18 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-query-integration</customfieldvalue>
            <customfieldvalue>charlie.swanson@mongodb.com</customfieldvalue>
            <customfieldvalue>william.qian@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2ny1r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i25ydc:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2nk73:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>