<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 03:57:46 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-21582] Sharding tests&apos; run time has doubled</title>
                <link>https://jira.mongodb.org/browse/SERVER-21582</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;The run time of the sharding suite has &lt;a href=&quot;https://evergreen.mongodb.com/task_timing/mongodb-mongo-master#/project=mongodb-mongo-master&amp;amp;buildVariant=enterprise-rhel-62-64-bit&amp;amp;taskName=sharding&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;doubled in time&lt;/a&gt; between these two commits:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/commit/e2eff50dba9396769caac67c78c7e7cc2968029b&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SERVER-20877 - fix linting errors&lt;/a&gt; - took &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_sharding_e2eff50dba9396769caac67c78c7e7cc2968029b_15_11_19_00_18_39&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;49 min&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://github.com/mongodb/mongo/commit/1944e28410ee687c7314e848d96582d5a9d54ff6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;SERVER-21078 - Fix race conditions by using the session cache more within the wtkv engine code&lt;/a&gt; - took &lt;a href=&quot;https://evergreen.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_62_64_bit_sharding_1944e28410ee687c7314e848d96582d5a9d54ff6_15_11_19_03_34_54&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;2hrs+&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The increase in time is not in any specific test, but is across the board. Tests, which took 1 minute to run, seem to be taking about 2 min now. Longer tests have gone up some too.&lt;/p&gt;</description>
                <environment></environment>
        <key id="240376">SERVER-21582</key>
            <summary>Sharding tests&apos; run time has doubled</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="alexander.gorrod@mongodb.com">Alexander Gorrod</assignee>
                                    <reporter username="kaloian.manassiev@mongodb.com">Kaloian Manassiev</reporter>
                        <labels>
                    </labels>
                <created>Fri, 20 Nov 2015 01:13:07 +0000</created>
                <updated>Thu, 14 Apr 2016 15:19:26 +0000</updated>
                            <resolved>Fri, 20 Nov 2015 22:42:26 +0000</resolved>
                                                                    <component>Performance</component>
                    <component>WiredTiger</component>
                                        <votes>0</votes>
                                    <watches>10</watches>
                                                                                                                <comments>
                            <comment id="1094817" author="dan@10gen.com" created="Fri, 20 Nov 2015 22:42:26 +0000"  >&lt;p&gt;fixed by change in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21078&quot; title=&quot;Segfault from race between getlasterror with fsync:true and clean database shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21078&quot;&gt;&lt;del&gt;SERVER-21078&lt;/del&gt;&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="1093987" author="alexander.gorrod" created="Fri, 20 Nov 2015 06:13:45 +0000"  >&lt;p&gt;After digging into this, it appears as though an applyOps cursor is being left open for a while. The new code in &lt;a href=&quot;https://github.com/mongodb/mongo/commit/1944e28410ee687c7314e848d96582d5a9d54ff6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://github.com/mongodb/mongo/commit/1944e28410ee687c7314e848d96582d5a9d54ff6&lt;/a&gt; now waits during shutdown for that session to be released.&lt;/p&gt;

&lt;p&gt;The original issue this change was fixing was a workload that constantly runs getLastError with fsync:true set, which could race with a WiredTiger connection close. The race was possible because the MongoDB clean shutdown takes the global lock, but the getLastError doesn&apos;t take that lock.&lt;/p&gt;

&lt;p&gt;I have coded up a patch that changes the approach to avoiding the race. The WiredTiger storage engine will wait for any fsync operation to complete before shutting down, rather than waiting for all sessions to be returned to the session cache.&lt;/p&gt;

&lt;p&gt;My testing shows that fixes the testing performance regression, and I haven&apos;t been able to reproduce the failure described in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21078&quot; title=&quot;Segfault from race between getlasterror with fsync:true and clean database shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21078&quot;&gt;&lt;del&gt;SERVER-21078&lt;/del&gt;&lt;/a&gt;.&lt;/p&gt;</comment>
                            <comment id="1093955" author="michael.cahill" created="Fri, 20 Nov 2015 05:05:02 +0000"  >&lt;p&gt;Moving to SERVER now we have confirmed that this is real.  Background syncs appear to have sessions open when they are waiting for activity.  There may also be other cases.&lt;/p&gt;

&lt;p&gt;Current thinking is to revert part of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-21078&quot; title=&quot;Segfault from race between getlasterror with fsync:true and clean database shutdown&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-21078&quot;&gt;&lt;del&gt;SERVER-21078&lt;/del&gt;&lt;/a&gt; and check inside WiredTiger that there is no flush in progress when &lt;tt&gt;WT_CONNECTION::close&lt;/tt&gt; is called.&lt;/p&gt;</comment>
                            <comment id="1093892" author="kaloian.manassiev" created="Fri, 20 Nov 2015 02:13:29 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=michael.cahill&quot; class=&quot;user-hover&quot; rel=&quot;michael.cahill&quot;&gt;michael.cahill&lt;/a&gt;:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;I suspect there is some sharding operation that keeps a session active...&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;Sharding tests enable the snapshot thread on the config server instances in order to use read concern majority, so this could be what is keeping active WT sessions.&lt;/p&gt;</comment>
                            <comment id="1093885" author="max.hirschhorn@10gen.com" created="Fri, 20 Nov 2015 01:51:09 +0000"  >&lt;blockquote&gt;&lt;p&gt;Interestingly, there is no increase in time on the &lt;a href=&quot;https://evergreen.mongodb.com/task_timing/mongodb-mongo-master#/project=mongodb-mongo-master&amp;amp;buildVariant=linux-64&amp;amp;taskName=sharding&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;linux-64 suite&lt;/a&gt;.&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;It seems like the execution time has increased across all build variants (see attached screenshots). Perhaps the &lt;tt&gt;sharding&lt;/tt&gt; and &lt;tt&gt;sharding_WT&lt;/tt&gt; tasks hadn&apos;t run on Linux 64-bit when you checked. It might be obvious, but I&apos;d like to mention explicitly that this affects both the &lt;tt&gt;sharding&lt;/tt&gt; and &lt;tt&gt;sharding_WT&lt;/tt&gt; tasks because CSRS run with &lt;tt&gt;&amp;#45;&amp;#45;storageEngine=wiredTiger&lt;/tt&gt;.&lt;/p&gt;

&lt;p&gt;Also, we are hitting the Evergreen 6-hour task execution time limit with the Windows DEBUG builders.&lt;/p&gt;</comment>
                            <comment id="1093883" author="michael.cahill" created="Fri, 20 Nov 2015 01:47:04 +0000"  >&lt;p&gt;The marker in the log is &quot;Waiting for WiredTiger operations to complete&quot; &amp;#8211; that is definitely kicking in with these all of the sharding tests.  I&apos;ll take a look today and let you know what I find.&lt;/p&gt;</comment>
                            <comment id="1093880" author="michael.cahill" created="Fri, 20 Nov 2015 01:37:48 +0000"  >&lt;p&gt;That change waits on shutdown for any WiredTiger operations that are in progress because otherwise mongod can segfault.&lt;/p&gt;

&lt;p&gt;It defines &quot;in progress&quot; as having a WiredTiger active (checked out of the cache).  I suspect there is some sharding operation that keeps a session active, which will make the shutdown very slow (it does eventually give up).&lt;/p&gt;

&lt;p&gt;We&apos;ll take a look and try to catch the state when this happens.&lt;/p&gt;</comment>
                            <comment id="1093869" author="kaloian.manassiev" created="Fri, 20 Nov 2015 01:17:11 +0000"  >&lt;p&gt;Interestingly, there is no increase in time on the &lt;a href=&quot;https://evergreen.mongodb.com/task_timing/mongodb-mongo-master#/project=mongodb-mongo-master&amp;amp;buildVariant=linux-64&amp;amp;taskName=sharding&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;linux-64 suite&lt;/a&gt;. &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=ernie.hershey&quot; class=&quot;user-hover&quot; rel=&quot;ernie.hershey&quot;&gt;ernie.hershey&lt;/a&gt;, is it possible that this is due to some change of hardware?&lt;/p&gt;</comment>
                            <comment id="1093868" author="kaloian.manassiev" created="Fri, 20 Nov 2015 01:15:37 +0000"  >&lt;p&gt;Aggregation tests have gone up in time as well, but that&apos;s probably because of those tests, which exercise sharding: &lt;a href=&quot;https://evergreen.mongodb.com/task_timing/mongodb-mongo-master#/project=mongodb-mongo-master&amp;amp;buildVariant=enterprise-rhel-62-64-bit&amp;amp;taskName=aggregation&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://evergreen.mongodb.com/task_timing/mongodb-mongo-master#/project=mongodb-mongo-master&amp;amp;buildVariant=enterprise-rhel-62-64-bit&amp;amp;taskName=aggregation&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="235761">SERVER-21078</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="99124" name="enterprise-rhel62_sharding.png" size="131799" author="max.hirschhorn@mongodb.com" created="Fri, 20 Nov 2015 01:40:19 +0000"/>
                            <attachment id="99125" name="linux64-sharding_WT.png" size="152288" author="max.hirschhorn@mongodb.com" created="Fri, 20 Nov 2015 01:40:19 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_10011" key="com.atlassian.jira.plugin.system.customfieldtypes:radiobuttons">
                        <customfieldname>Backwards Compatibility</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10038"><![CDATA[Fully Compatible]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Fri, 20 Nov 2015 01:37:48 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        8 years, 12 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            8 years, 12 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>alexander.gorrod@mongodb.com</customfieldvalue>
            <customfieldvalue>dan@mongodb.com</customfieldvalue>
            <customfieldvalue>kaloian.manassiev@mongodb.com</customfieldvalue>
            <customfieldvalue>max.hirschhorn@mongodb.com</customfieldvalue>
            <customfieldvalue>michael.cahill@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrko47:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hseizr:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrkzsn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>