<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:13:15 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26804] Pause in inserts for YCSB</title>
                <link>https://jira.mongodb.org/browse/SERVER-26804</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Running YCSB against a sharded cluster in our regression framework, we see a 10+ pause in inserts correlated with eviction in aggressive mode. The pause appears to be on the primary of the second shard of a three shard cluster. The test was run using c3.2xlarge instances, using local SSD for data and journal. The journal is on a separate device that the data. &lt;/p&gt;

&lt;p&gt;Here&apos;s a plot of some key stats during the pause:&lt;br/&gt;
&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/142612/142612_pause.png&quot; width=&quot;600&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;</description>
                <environment></environment>
        <key id="327297">SERVER-26804</key>
            <summary>Pause in inserts for YCSB</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="10038" iconUrl="https://jira.mongodb.org/images/icons/subtask.gif" description="">Backlog</status>
                    <statusCategory id="2" key="new" colorName="default"/>
                                    <resolution id="-1">Unresolved</resolution>
                                        <assignee username="backlog-server-storage-engines">Backlog - Storage Engines Team</assignee>
                                    <reporter username="david.daly@mongodb.com">David Daly</reporter>
                        <labels>
                            <label>3.7BackgroundTask</label>
                    </labels>
                <created>Thu, 27 Oct 2016 13:27:18 +0000</created>
                <updated>Tue, 6 Dec 2022 04:13:05 +0000</updated>
                                            <version>3.4.0-rc0</version>
                    <version>3.4.0-rc1</version>
                                                    <component>Performance</component>
                    <component>Storage</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>9</watches>
                                                                                                                <comments>
                            <comment id="1485889" author="david.daly" created="Thu, 26 Jan 2017 15:45:24 +0000"  >&lt;p&gt;I finally got something running &lt;a href=&quot;https://evergreen.mongodb.com/task/sys_perf_linux_3_node_replSet_industry_benchmarks_WT_a25baa98c2ba51b1bd10a39538911b43be4db64f_17_01_25_19_37_22#/%257B%2522compare%2522%253A%255B%257B%2522hash%2522%253A%2522a25baa98c2ba51b1bd10a39538911b43be4db64f%2522%257D%255D%257D&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;. For this test I dropped the YCSB collection between runs, but the server stayed up. YCSB uses the same document ids each time it runs, so if you don&apos;t drop the collection you get duplicate key errors. There is an option to change the start point of the documents, so we can work around this if needed. &lt;/p&gt;

&lt;p&gt;Looking at the results there are a few interesting things: &lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;The first two runs show drops to 0 lasting over 10 seconds. Twice on the first run (220 and 550 seconds in), and once on the second run (520 seconds in)&lt;/li&gt;
	&lt;li&gt;There are no 10 second drops to zero after the first two runs.&lt;/li&gt;
	&lt;li&gt;The throughput goes up dramatically across the first four runs&lt;/li&gt;
	&lt;li&gt;One of the secondaries falls off the oplog during the third run and stays in recovery for the rest of the run.&lt;/li&gt;
	&lt;li&gt;The performance looking at FTDC data looks much more stable a run or two after the third node falls off the oplog.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Does this data help confirm or deny your suspicions? It does seem to get much more stable over time, at least for the nodes that don&apos;t fall over. &lt;/p&gt;</comment>
                            <comment id="1454944" author="david.hows" created="Mon, 12 Dec 2016 04:49:02 +0000"  >&lt;p&gt;Hi David Daly,&lt;/p&gt;

&lt;p&gt;Sorry for the delay.&lt;/p&gt;

&lt;blockquote&gt;&lt;p&gt;What makes sense to do here? One simple experiment I could do is to run the load phase twice, either dropping the collection in between, or loading to a new collection the second time. Would that make sense?&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;It would make sense to try testing the load phase twice.&lt;/p&gt;

&lt;p&gt;I understand about the physical hardware.&lt;/p&gt;

&lt;p&gt;Let me know how that goes.&lt;/p&gt;
</comment>
                            <comment id="1446821" author="david.daly" created="Thu, 1 Dec 2016 14:27:09 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=david.hows&quot; class=&quot;user-hover&quot; rel=&quot;david.hows&quot;&gt;david.hows&lt;/a&gt; I can definitely do the pre-heating collections. What makes sense to do here? One simple experiment I could do is to run the load phase twice, either dropping the collection in between, or loading to a new collection the second time. Would that make sense? &lt;/p&gt;

&lt;p&gt;Testing this with physical hardware requires a fair amount more work to get a real cluster up and make sure it falls over again. If we want to try just standalone with oplog, I can run that locally on my machine. I don&apos;t know if it will reproduce there as it&apos;s just a different set of hardware than we&apos;re using in AWS. If you think there&apos;s something to learn from that experiment, I would be glad to do it. Thanks. &lt;/p&gt;</comment>
                            <comment id="1446460" author="david.hows" created="Thu, 1 Dec 2016 00:05:35 +0000"  >&lt;p&gt;To Henrik&apos;s comments.&lt;/p&gt;

&lt;p&gt;The issue here is not only about pre-heating the disk itself, although that could be a factor. It is about pre-heating the WiredTiger cache following a restart of the MongoDB instance. It is not unusual an instance to take some time in ramping to be able to keep up with eviction and there are other factors around the first usage of a collection, as we have to do things like initial writes.&lt;/p&gt;

&lt;p&gt;One thing worth testing here would be to heat up the instance and collection first, with an intial warm up pass to build the collection ahead of the workload. This should hopefully work around that worst case where we a large early checkpoint saturating the disk.&lt;/p&gt;

&lt;p&gt;To David Daly&apos;s comments.&lt;/p&gt;

&lt;p&gt;Understand where you are coming from. Are you able to look at doing some testing with physical hardware? Or pre-heating the collections as suggested above? Hopefully with some of those changes we can minimize the stalls.&lt;/p&gt;</comment>
                            <comment id="1424476" author="sue.loverso" created="Wed, 2 Nov 2016 20:06:07 +0000"  >&lt;p&gt;I looked at the FTDC from the run you showed.  The statistics I&apos;m viewing show a lot of IO going on during the stalls.  The number of active write system calls in progress goes up and stays up at 31 for the duration of the stall.  There is also 1 fsync and a checkpoint running for the entire time.  The system (2nd line) shows it is spending its time in iowait.  Coincidentally eviction is aggressive for the exact duration of the stall as well.&lt;/p&gt;</comment>
                            <comment id="1420398" author="david.daly" created="Fri, 28 Oct 2016 13:51:31 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=sue.loverso&quot; class=&quot;user-hover&quot; rel=&quot;sue.loverso&quot;&gt;sue.loverso&lt;/a&gt;, we&apos;re running YCSB from here: &lt;a href=&quot;https://github.com/mongodb-labs/YCSB/tree/evergreen&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb-labs/YCSB/tree/evergreen&lt;/a&gt; on the evergreen branch. &lt;br/&gt;
To set it up, clone the repo, and&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;cd YCSB/ycsb-mongo&lt;/li&gt;
	&lt;li&gt;./setup.sh&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;And we&apos;re running it with this command line&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;./bin/ycsb load mongodb -s -P workloads/workloadLongevity -p mongodb.url=mongodb://10.2.0.99:27017/ycsb -threads 32&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;
&lt;p&gt;Change the mongodb.url to the appropriate target for you. &lt;/p&gt;

&lt;p&gt;We were running the tests with a separate EC2 node (c3.2xlarge using local SSD) for the client, and each node in the cluster. I tried simplifying to 3 node repl set and standalone. The issue repeated on the 3 node repl set. The standalone shows drops in throughput, but none going to zero. I&apos;m not sure if that is random variation, or because there&apos;s no oplog to run with. I kicked of a 1node repl set run also to see what happens there. &lt;/p&gt;

&lt;p&gt;I think it makes sense to start with a 1 Node repl set, and see if you can reproduce with that locally. If it doesn&apos;t we can start pulling apart what&apos;s different between your local environment and our test environment. &lt;/p&gt;</comment>
                            <comment id="1419399" author="david.daly" created="Thu, 27 Oct 2016 13:38:55 +0000"  >&lt;p&gt;Attaching raw timeseries.html file for the primary of the second shard. &lt;br/&gt;
&lt;span class=&quot;nobr&quot;&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/attachment/142613/142613_timeseries.html&quot; title=&quot;timeseries.html attached to SERVER-26804&quot;&gt;timeseries.html&lt;sup&gt;&lt;img class=&quot;rendericon&quot; src=&quot;https://jira.mongodb.org/images/icons/link_attachment_7.gif&quot; height=&quot;7&quot; width=&quot;7&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/sup&gt;&lt;/a&gt;&lt;/span&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10011">
                    <name>Depends</name>
                                                                <inwardlinks description="is depended on by">
                                                        </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="142612" name="pause.png" size="280244" author="david.daly@mongodb.com" created="Thu, 27 Oct 2016 13:29:56 +0000"/>
                            <attachment id="142613" name="timeseries.html" size="994671" author="david.daly@mongodb.com" created="Thu, 27 Oct 2016 13:38:55 +0000"/>
                            <attachment id="143185" name="timeseries.p4.png" size="161377" author="sue.loverso@mongodb.com" created="Wed, 2 Nov 2016 20:06:07 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18555" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname># of Sprints</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25135"><![CDATA[Storage Engines]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 28 Oct 2016 13:24:07 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 2 weeks, 6 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>alexander.golin@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 2 weeks, 6 days ago
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_16465" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Linked BF Score</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0.0</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>backlog-server-storage-engines</customfieldvalue>
            <customfieldvalue>david.daly@mongodb.com</customfieldvalue>
            <customfieldvalue>david.hows</customfieldvalue>
            <customfieldvalue>sue.loverso@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjsfb:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hr70af:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_10557" key="com.pyxis.greenhopper.jira:gh-sprint">
                        <customfieldname>Sprint</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue id="1361">Storage 2016-11-21</customfieldvalue>
    <customfieldvalue id="1395">Storage 2016-12-12</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb-labs/YCSB&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;YCSB&lt;/a&gt; as run in longevity regression suite. 3 shard cluster in AWS using c3.2xlarge instances.  &lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hs0t0f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>