<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:08:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-25328] Performance issues with WiredTiger - huge latency peaks</title>
                <link>https://jira.mongodb.org/browse/SERVER-25328</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;We are storing a huge number of collections (like thousands) in our databases. We are planning to migrate our Mongo storages from MMAPv1 to WiredTiger, but before doing that, we did a bunch of performance tests on MongoDB 3.2.7 and 3.2.8. We created test dataset with a large number of collections (30 000) and written a test, which performs only read by id operations. The results showed latency peaks (please see the attached screenshot 1). The test was executed on the following hardware configurations: &lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;replica set with 3 nodes deployed on Amazon EC2 with SSD drives using XFS file system&lt;/li&gt;
	&lt;li&gt;one MongoDB instance run on MackBook Pro&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;We observed similar performance characteristic for both configurations used. After reading the docs and tuning WiredTiger configuration, we discovered, that the peaks are probably caused by periodic flush memory to disk (fsync). We tried to set syncdelay option to zero (which is actually not recommended) and noticed that performance was better but peaks are still there (please see attached screenshot 2). &lt;/p&gt;

&lt;p&gt;In order to reproduce the problem please use the attached zip file containing the following: &lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;Mongo shell script to easily load the test data&lt;/li&gt;
	&lt;li&gt;minimal REST service, that makes calls to Mongo&lt;/li&gt;
	&lt;li&gt;Gatling simulation, which makes calls to the REST service&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Steps to reproduce: &lt;br/&gt;
1. Load test data with MongoDB shell script &lt;br/&gt;
2. Run REST service &lt;br/&gt;
3. Run Gatling simulation &lt;br/&gt;
4. Notice the recurring peaks on latency charts For more detailed instructions on how to run it, please see README.txt in the zip.&lt;/p&gt;

&lt;p&gt;We also ran the above tests on Mongo 3.2.1. We conducted multiple tests both locally and on our machines on AWS and the performance is OK, there are no peaks. The results can be seen on screenshot 3.&lt;/p&gt;</description>
                <environment></environment>
        <key id="304995">SERVER-25328</key>
            <summary>Performance issues with WiredTiger - huge latency peaks</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="9">Done</resolution>
                                        <assignee username="xiangyu.yao@mongodb.com">Xiangyu Yao</assignee>
                                    <reporter username="p.bochynski@gmail.com">Piotr Bochynski</reporter>
                        <labels>
                            <label>3.7BackgroundTask</label>
                            <label>RF</label>
                    </labels>
                <created>Fri, 29 Jul 2016 11:42:25 +0000</created>
                <updated>Wed, 6 Dec 2017 21:16:15 +0000</updated>
                            <resolved>Tue, 7 Nov 2017 15:03:35 +0000</resolved>
                                                    <fixVersion>3.4.0-rc1</fixVersion>
                                    <component>WiredTiger</component>
                                        <votes>4</votes>
                                    <watches>17</watches>
                                                                                                                <comments>
                            <comment id="1716875" author="xiangyu.yao" created="Fri, 3 Nov 2017 19:33:50 +0000"  >&lt;p&gt;I retested this workload with mongod-3.2.8 on my local Linux machine and verified there are latency peaks for checkpoints.&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;170287_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/170287/170287_Screenshot-mongo3.2.8.png&quot; title=&quot;Screenshot-mongo3.2.8.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;170287&quot; file-preview-title=&quot;Screenshot-mongo3.2.8.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/170287/_thumb_170287.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;And then I retested this workload with our latest mongod-3.6 and got the results shown in the following screenshot. It seems the latency peaks are gone now.&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;170168_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/170168/170168_Screenshot-mongo3.6.png&quot; title=&quot;Screenshot-mongo3.6.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;170168&quot; file-preview-title=&quot;Screenshot-mongo3.6.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/170168/_thumb_170168.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;I also tested it on mongod-3.2.12 and mongod-3.4.10 and verified that they don&apos;t have this issue.&lt;br/&gt;
mongod-3.2.12:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;170322_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/170322/170322_Screenshot-mongo3.2.12.png&quot; title=&quot;Screenshot-mongo3.2.12.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;170322&quot; file-preview-title=&quot;Screenshot-mongo3.2.12.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/170322/_thumb_170322.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;

&lt;p&gt;mongod-3.4.10:&lt;br/&gt;
 &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;a id=&quot;170321_thumb&quot; href=&quot;https://jira.mongodb.org/secure/attachment/170321/170321_Screenshot-mongo3.4.10.png&quot; title=&quot;Screenshot-mongo3.4.10.png&quot; file-preview-type=&quot;image&quot; file-preview-id=&quot;170321&quot; file-preview-title=&quot;Screenshot-mongo3.4.10.png&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/thumbnail/170321/_thumb_170321.png&quot; style=&quot;border: 0px solid black&quot; role=&quot;presentation&quot;/&gt;&lt;/a&gt;&lt;/span&gt; &lt;/p&gt;</comment>
                            <comment id="1355045" author="alexander.gorrod" created="Fri, 12 Aug 2016 07:23:35 +0000"  >&lt;p&gt;The issue identified here should be improved in query only workloads by the change outlined in &lt;a href=&quot;https://jira.mongodb.org/browse/WT-2831&quot; title=&quot;Skip creating a checkpoint if there have been no changes&quot; class=&quot;issue-link&quot; data-issue-key=&quot;WT-2831&quot;&gt;&lt;del&gt;WT-2831&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1348179" author="p.bochynski@gmail.com" created="Fri, 5 Aug 2016 09:46:43 +0000"  >&lt;p&gt;Hi Thomas, &lt;br/&gt;
We ran the tests several times (without restarting mongod) and we always observed peaks during background flush with mongo 3.2.7. Interesting thing is that test performs only read operations (no writes at all). And the same test executed on mongo 3.2.1 gives much better results (10 ms vs 100 ms). &lt;br/&gt;
So right now we have stuck with version 3.2.1 and we cannot upgrade mongo.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Piotr&lt;/p&gt;</comment>
                            <comment id="1345512" author="thomas.schubert" created="Wed, 3 Aug 2016 02:02:01 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=p.bochynski%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;p.bochynski@gmail.com&quot;&gt;p.bochynski@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;d like to quickly explain what we&apos;re observing in this simulation. When a checkpoint begins it blocks access to the tables&apos; &quot;access points&quot; so it can get started. As there are a lot of tables in this dataset this is a little time consuming.&lt;/p&gt;

&lt;p&gt;This lock does not block existing connections from accessing tables that they have already touched, but can cause delays when each connection to MongoDB tries to gather data from a table it has not yet accessed. In our work with your test application, we saw that with subsequent executions of the suite the max access time was under 15ms.&lt;/p&gt;

&lt;p&gt;You can confirm this behavior by rerunning the test without restarting the mongod instance. &lt;/p&gt;

&lt;p&gt;I am moving this ticket to the WiredTiger team&apos;s backlog as we discuss next steps. Thank you again for excellent report; the reproduction greatly helped our investigation.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1341388" author="thomas.schubert" created="Fri, 29 Jul 2016 15:37:41 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=p.bochynski%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;p.bochynski@gmail.com&quot;&gt;p.bochynski@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for the very detailed bug report with clear reproduction steps. We are investigating this issue and will update this ticket when we know more.&lt;/p&gt;

&lt;p&gt;Best regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1341060" author="p.bochynski@gmail.com" created="Fri, 29 Jul 2016 11:47:12 +0000"  >&lt;p&gt;Tests to reproduce the issue.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="308483">WT-2831</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="132998" name="SERVER-25328.zip" size="2921680" author="p.bochynski@gmail.com" created="Fri, 29 Jul 2016 11:47:12 +0000"/>
                            <attachment id="170322" name="Screenshot-mongo3.2.12.png" size="57004" author="xiangyu.yao@mongodb.com" created="Mon, 6 Nov 2017 23:55:25 +0000"/>
                            <attachment id="170287" name="Screenshot-mongo3.2.8.png" size="132110" author="xiangyu.yao@mongodb.com" created="Mon, 6 Nov 2017 20:18:19 +0000"/>
                            <attachment id="170321" name="Screenshot-mongo3.4.10.png" size="56216" author="xiangyu.yao@mongodb.com" created="Mon, 6 Nov 2017 23:55:25 +0000"/>
                            <attachment id="170168" name="Screenshot-mongo3.6.png" size="69550" author="xiangyu.yao@mongodb.com" created="Fri, 3 Nov 2017 19:31:03 +0000"/>
                            <attachment id="132995" name="Screenshot1mongo3.2.7.png" size="46774" author="p.bochynski@gmail.com" created="Fri, 29 Jul 2016 11:43:04 +0000"/>
                            <attachment id="132996" name="Screenshot2mongo3.2.7syncdelay0.png" size="51875" author="p.bochynski@gmail.com" created="Fri, 29 Jul 2016 11:43:04 +0000"/>
                            <attachment id="132997" name="Screenshot3mongo3.2.1.png" size="81338" author="p.bochynski@gmail.com" created="Fri, 29 Jul 2016 11:43:04 +0000"/>
                            <attachment id="133088" name="checkpoints.png" size="108880" author="kelsey.schubert@mongodb.com" created="Fri, 29 Jul 2016 17:58:03 +0000"/>
                            <attachment id="133087" name="diagnostic-data-SERVER-25328.zip" size="60340" author="kelsey.schubert@mongodb.com" created="Fri, 29 Jul 2016 17:58:03 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>6.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 29 Jul 2016 15:37:41 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        6 years, 14 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            6 years, 14 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>p.bochynski@gmail.com</customfieldvalue>
            <customfieldvalue>xiangyu.yao@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk153:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr9s13:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1951">Storage 2017-11-13</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hseqe7:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>