<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:29:58 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-75353] Mongodb v5.0.3 scaling bottleneck/issue</title>
                <link>https://jira.mongodb.org/browse/SERVER-75353</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hi,&lt;/p&gt;

&lt;p&gt;I&apos;ve been studying the mongodb scaling impact on a high core count machines. I&apos;ve observed that the server doesn&apos;t scale linearly once it crosses 20 threads. This was tested against Mongodb v5.0.3 and YCSB client (readonly tests). Here is my observation,&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;Clients/Threads&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;4&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;8&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;10&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;20&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;30&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;40&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;Throughput per thread&#160;&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 1.00&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.99&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.98&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.94&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.94&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.80&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.59&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.43&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;&#160;&#160;&#160;&#160;&#160;&#160;&#160;&#160; 0.19&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;


&lt;p&gt;First row is self-explanatory. The second row show per thread throughput over single thread throughput.&#160; Also see the attachment showing the graph. In order to understand the issue, I collected some off-cpu data events. This data showed a lock/pause is the cause of scaling issue in two functions, viz. 1) mongo::ServiceContext::makeOperationContext() and 2) mongo::ServiceContext::__delistOperation(). I&apos;ve also attached flamegraph showing these bottlenecks.&lt;/p&gt;

&lt;p&gt;Unfortunately, I can&apos;t contribute to the project with actual code-contribution due to restrictions on my side but I can submit the issues hence this report. I&apos;m hoping someone from the mongodb community will take a look at this issue and provide a solution.&lt;/p&gt;

&lt;p&gt;I hope this helps as a a good starting point for further analysis.&lt;/p&gt;</description>
                <environment></environment>
        <key id="2299093">SERVER-75353</key>
            <summary>Mongodb v5.0.3 scaling bottleneck/issue</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-perf">Backlog - Performance Team</assignee>
                                    <reporter username="uttampc">NOVALUE Uttam</reporter>
                        <labels>
                            <label>perf-effort-medium</label>
                            <label>perf-urgency-soon</label>
                            <label>perf-value-essential</label>
                            <label>performance</label>
                            <label>scaling</label>
                    </labels>
                <created>Mon, 27 Mar 2023 20:33:35 +0000</created>
                <updated>Wed, 24 May 2023 09:27:56 +0000</updated>
                                                                                                <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="5447875" author="ger.hartnett" created="Wed, 24 May 2023 09:27:19 +0000"  >&lt;p&gt;Thank you for reporting this Uttam. We expect to start a project soon to investigate this and other similar issues.&#160;&lt;/p&gt;</comment>
                            <comment id="5304201" author="uttampc" created="Mon, 27 Mar 2023 20:36:52 +0000"  >&lt;p&gt;Here is the content of logback.xml (reduced output)&lt;/p&gt;

&lt;p&gt;$ cat logback.xml&#160;&lt;br/&gt;
&amp;lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&amp;gt;&lt;br/&gt;
&amp;lt;configuration&amp;gt;&lt;br/&gt;
&#160; &#160; &amp;lt;appender name=&quot;STDOUT&quot; class=&quot;ch.qos.logback.core.ConsoleAppender&quot;&amp;gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &amp;lt;encoder&amp;gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160; &#160; &amp;lt;pattern&amp;gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160; &#160; &#160; &#160; %d{HH:mm:ss.SSS} &#160;%-5level %logger{36} - %msg%n&lt;br/&gt;
&#160; &#160; &#160; &#160; &#160; &#160; &amp;lt;/pattern&amp;gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &amp;lt;/encoder&amp;gt;&lt;br/&gt;
&#160; &#160; &amp;lt;/appender&amp;gt;&lt;br/&gt;
&#160; &#160; &amp;lt;logger name=&quot;org.mongodb&quot; level=&quot;WARN&quot;&amp;gt;&lt;br/&gt;
&#160; &#160; &#160; &#160; &amp;lt;appender-ref ref=&quot;STDOUT&quot;/&amp;gt;&lt;br/&gt;
&#160; &#160; &amp;lt;/logger&amp;gt;&lt;br/&gt;
&amp;lt;/configuration&amp;gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                                        </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="439587" name="mongodb-scaling-cpu-util.png" size="141550" author="uttampc" created="Mon, 27 Mar 2023 20:21:32 +0000"/>
                            <attachment id="439588" name="mongodb-scaling-ipc.png" size="102418" author="uttampc" created="Mon, 27 Mar 2023 20:21:26 +0000"/>
                            <attachment id="439589" name="mongodb-thread-scaling.png" size="114620" author="uttampc" created="Mon, 27 Mar 2023 20:19:33 +0000"/>
                            <attachment id="439586" name="offcpu.80t.svg" size="110665" author="uttampc" created="Mon, 27 Mar 2023 20:28:39 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25124"><![CDATA[Product Performance]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 29 Mar 2023 19:34:31 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        37 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ger.hartnett@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            37 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-perf</customfieldvalue>
            <customfieldvalue>ger.hartnett@mongodb.com</customfieldvalue>
            <customfieldvalue>uttampc</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2214v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1kdv4:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Use ycsb client,&lt;br/&gt;
&#160;&lt;br/&gt;
&lt;tt&gt;$ python2.7 ./bin/ycsb load mongodb -jvm-args=&quot;-Dlogback.configurationFile=../logback.xml&quot; -s -P workloads/workloadc -threads 20  -p mongodb.url=mongodb://localhost/ycsb&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;{{}}&lt;/p&gt;

&lt;p&gt;&lt;tt&gt;$ python2.7 ./bin/ycsb run mongodb -jvm-args=&quot;-Dlogback.configurationFile=../logback.xml&quot; -s -P workloads/workloadc -threads 80  -p mongodb.url=mongodb://localhost/ycsb&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;{&lt;tt&gt;}&lt;/tt&gt;&lt;tt&gt;No. records: 40M&lt;/tt&gt;&lt;br/&gt;
&lt;tt&gt;Type of Request: Read-Only queries&lt;/tt&gt;&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i21na7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>