<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:28:41 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-31947] Performance degradation with more clients </title>
                <link>https://jira.mongodb.org/browse/SERVER-31947</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;While doing some benchmarks, I found out strange performance degradation for MongoDB for read only workload on YCSB (WorkloadC). You can see it on the right side of the graph where the number of clients is getting bigger:&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/170821/170821_throughput_mongodb.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;I tried to investigate using perf tool and figured out that the only two metrics that are growing significantly are number of cpu migrations and syscall `sched_yield`. On the next graph you can see the dynamics of cpu migration event, number of `sched_yield` also grew 3 times bigger.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/170820/170820_cpu_migrations.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;Judging from source code I can assume that it&apos;s somehow related to spin locks, since the only place where I found `sched_yield` is `spin_lock.cpp`.&lt;/p&gt;</description>
                <environment></environment>
        <key id="458084">SERVER-31947</key>
            <summary>Performance degradation with more clients </summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="13203">Gone away</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="erthalion">Dmitry Dolgov</reporter>
                        <labels>
                    </labels>
                <created>Mon, 13 Nov 2017 21:26:58 +0000</created>
                <updated>Fri, 27 Oct 2023 20:43:48 +0000</updated>
                            <resolved>Thu, 25 Oct 2018 19:00:45 +0000</resolved>
                                                                    <component>Concurrency</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>15</watches>
                                                                                                                <comments>
                            <comment id="2042438" author="thomas.schubert" created="Thu, 25 Oct 2018 19:00:45 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=erthalion&quot; class=&quot;user-hover&quot; rel=&quot;erthalion&quot;&gt;erthalion&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;m going to close this ticket for the time being since there have been a number of fixes that have landed that we expect to improve performance on this branch, please feel free to comment with an update after running new tests and we&apos;ll reopen the ticket.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1846145" author="erthalion" created="Tue, 27 Mar 2018 06:52:32 +0000"  >&lt;p&gt;Hi Kelsey,&lt;/p&gt;

&lt;p&gt;Thank you for the information. Yes, back then I was testing MongoDB 3.4.4. Soon I&apos;m going to do another round of benchmarks with new versions of all the databases, and we can compare the performance.&lt;/p&gt;</comment>
                            <comment id="1778426" author="thomas.schubert" created="Thu, 18 Jan 2018 22:27:36 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=erthalion&quot; class=&quot;user-hover&quot; rel=&quot;erthalion&quot;&gt;erthalion&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;My understanding is that these tests were executed with MongoDB 3.4.4, is that correct? I&apos;m curious whether you see the same behavior on a more recent version of MongoDB 3.4, which would included &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3345&quot; title=&quot;Improve rwlock scaling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3345&quot;&gt;&lt;del&gt;WT-3345&lt;/del&gt;&lt;/a&gt;. As you can see in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-3345&quot; title=&quot;Improve rwlock scaling&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-3345&quot;&gt;&lt;del&gt;WT-3345&lt;/del&gt;&lt;/a&gt;, we made significant improvements to the rwlocks in WiredTiger, which may affect the performance you&apos;re observing in these benchmarks.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                            <comment id="1761883" author="erthalion" created="Sat, 30 Dec 2017 08:37:26 +0000"  >&lt;p&gt;Hi Henrik,&lt;/p&gt;

&lt;p&gt;Thanks for your response. Yes, I&apos;m aware about that. But at the same time I made the same kind of test for PostgreSQL and MySQL, and performance degradation was not that significant there - that&apos;s why I thought it&apos;s strange and maybe worth mentioning.&lt;/p&gt;

&lt;p&gt; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;174462_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/174462/174462_select_btree_throughput.png&quot; title=&quot;select_btree_throughput.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;174462&quot; file-preview-title=&quot;select_btree_throughput.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/174462/_thumb_174462.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                            <comment id="1750075" author="henrik.edin" created="Tue, 12 Dec 2017 19:35:36 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=erthalion&quot; class=&quot;user-hover&quot; rel=&quot;erthalion&quot;&gt;erthalion&lt;/a&gt;, MongoDB uses a thread-per-connection model. This means that when the amount of connections increases the amount of context switches between threads also increases. As context switches aren&#8217;t free this is unfortunately a behavior that can be expected. There are several reasons why this can be expensive: cache misses, page faults, contention on spin locks, etc. &lt;/p&gt;

&lt;p&gt;In the public source depot, you can see experimentation on a thread pool model where different connections can execute on the same thread. It is too early to promise any results but you might find it worthwhile to follow that project.&lt;/p&gt;

&lt;p&gt;Henrik&lt;/p&gt;</comment>
                            <comment id="1739483" author="ian@10gen.com" created="Fri, 1 Dec 2017 15:26:03 +0000"  >&lt;p&gt;Thanks for filing this, Dmitry!  We believe that some upcoming work by the Platforms team on rate-limiting AsyncIO will improve the behavior you&apos;re seeing towards the right side of the graph.&lt;/p&gt;</comment>
                            <comment id="1725236" author="henrik.ingo@10gen.com" created="Tue, 14 Nov 2017 15:56:53 +0000"  >&lt;p&gt;Just adding a note that I met Dmitry at Highload++ and asked him to file this. (Thanks Dmitry!) I don&apos;t know more about this than is also described here, but his observation that a spinlock would be involved caught my interest.&lt;/p&gt;

&lt;p&gt;Also note that our current performance testing is on much more powerful instances than this, so we wouldn&apos;t experience these conditions (at least not with the same YCSB setup).&lt;/p&gt;</comment>
                            <comment id="1724568" author="thomas.schubert" created="Mon, 13 Nov 2017 21:34:20 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=erthalion&quot; class=&quot;user-hover&quot; rel=&quot;erthalion&quot;&gt;erthalion&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the report; I&apos;ve assigned this issue to the Storage Team for evaluation.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Kelsey&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="170820" name="cpu_migrations.png" size="23488" author="erthalion" created="Mon, 13 Nov 2017 21:19:40 +0000"/>
                            <attachment id="174462" name="select_btree_throughput.png" size="38274" author="erthalion" created="Sat, 30 Dec 2017 08:35:56 +0000"/>
                            <attachment id="170821" name="throughput_mongodb.png" size="36561" author="erthalion" created="Mon, 13 Nov 2017 21:19:27 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Mon, 13 Nov 2017 21:34:20 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        5 years, 15 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            5 years, 15 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>erthalion</customfieldvalue>
            <customfieldvalue>henrik.edin@mongodb.com</customfieldvalue>
            <customfieldvalue>henrik.ingo@mongodb.com</customfieldvalue>
            <customfieldvalue>ian@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htjw0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hra0nz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;Run YCSB/WorkloadC on MongoDB 3.4/3.2 (m4.xlarge).&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|htji4n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>