<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 05:32:08 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-53899] TaskExecutor CPU fills up instantly</title>
                <link>https://jira.mongodb.org/browse/SERVER-53899</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;we have a shardCluster with 6 mongos and 3 shard.&#160;&lt;/p&gt;

&lt;p&gt;Each mongos use 8 Core controlled by cgroup, and&#160;taskExecutorPoolSize is 4&lt;/p&gt;

&lt;p&gt;Each shard has 1 primary , 4 secondaries and 1 hidden.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;we use ycsb to pressure test it with 48 thread.&lt;/p&gt;

&lt;p&gt;Anything goes OK,But there may be steep drop(both CPU&amp;amp;opcounters) appeared in mongos once or twice for a 10min pressure test,&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/296187/296187_image-2021-01-20-17-44-34-407.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;When all things go ok, the CPU like:&lt;br/&gt;
S 35.0 0.0 8:35.77 TaskExe.rPool-1&lt;br/&gt;
S 35.0 0.0 8:52.79 TaskExe.rPool-2&lt;br/&gt;
S 35.0 0.0 8:32.40 TaskExe.rPool-3&lt;br/&gt;
S 30.0 0.0 9:18.88 TaskExe.rPool-0&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;When the&#160;steep drop happened,someone TaskExecutor CPU fill up like:&lt;/p&gt;

&lt;p&gt;R 99.9 0.0 8:54.62 TaskExe.rPool-2&lt;br/&gt;
S 10.0 0.0 9:19.35 TaskExe.rPool-0&lt;br/&gt;
S 10.0 0.0 8:36.19 TaskExe.rPool-1&lt;br/&gt;
R 5.0 0.0 8:32.80 TaskExe.rPool-3&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;pstack result for TaskExecutor with CPU filled up:&lt;/p&gt;

&lt;p&gt;Thread 85 (Thread 0x7f01c6203700 (LWP 129527)):&lt;br/&gt;
#0 0x00007f01cedb86d0 in sha256_block_data_order_avx2 ()&lt;br/&gt;
#1 0x00007f01cedb9935 in SHA256_Update ()&lt;br/&gt;
#2 0x00007f01ced58817 in HMAC_Init_ex ()&lt;br/&gt;
#3 0x00007f01cec04e0f in mongo::SHA256BlockTraits::computeHmac( xxx )&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;perf top result when CPU full happened: about 7.32% for OPENSSL_cleanse.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;There are also slow log appeared in mongos, while secondary&apos;s log do not contain slow log.The reason is that the connectionPool in TaskExecutor with CPU filled up has many requests to be sent, the connectionPool stats log is :&lt;/p&gt;

&lt;p&gt;Updating controller for host:port with State: { requests: 19, ready: 0, pending: 2, active: 1, isExpired: false }&lt;/p&gt;

&lt;p&gt;The request continues to grow, and pending is always 2. I think it just the result of TaskExecutor CPU full&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;There 2 things worth mentioning&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;when we use PSH for every shard ,everything goes ok, mongos CPU just fill up but no steep drop&lt;/li&gt;
	&lt;li&gt;we use&#160;taskExecutorPoolSize=8, may ycsb 48 threads goes well(also steep drop sometimes), but 96 threads still have problems&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;How do I solved this problem?&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</description>
                <environment></environment>
        <key id="1594651">SERVER-53899</key>
            <summary>TaskExecutor CPU fills up instantly</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="bruce.lucas@mongodb.com">Bruce Lucas</assignee>
                                    <reporter username="wangxin201492@gmail.com">Xin Wang</reporter>
                        <labels>
                    </labels>
                <created>Wed, 20 Jan 2021 09:52:12 +0000</created>
                <updated>Mon, 8 Jan 2024 15:23:01 +0000</updated>
                            <resolved>Fri, 22 Jan 2021 15:42:34 +0000</resolved>
                                                                    <component>Sharding</component>
                                        <votes>1</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="3577849" author="bruce.lucas@10gen.com" created="Fri, 22 Jan 2021 15:42:07 +0000"  >&lt;p&gt;Thank you for the update. I&apos;ll close this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt; as you are already watching that ticket. Please feel free to re-open this or open a new ticket if &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt; doesn&apos;t resolve your problem.&lt;/p&gt;</comment>
                            <comment id="3577182" author="wangxin201492@gmail.com" created="Fri, 22 Jan 2021 05:06:41 +0000"  >&lt;p&gt;I also think this is the same issue with&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;&#160;, when our initial pressure test using taskExecutorPoolSize = 1&#160; and the slowlog &amp;amp; cpu filled up happend, we see mongos connect to all the secondaries with&#160;continuously rising connections, which is same with&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;&#160;,but a little difference is that&#160;all the second that ticket is only one secondary with connection rise&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;About test with&#160;taskExecutorPoolSize = 1 with&#160;ShardingTaskExecutorPoolMaxSize&amp;amp;ShardingTaskExecutorPoolMinSize = 20 , I have already done with it .&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/296570/296570_image-2021-01-22-12-50-18-538.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;All the 4 tests are using&#160;taskExecutorPoolSize = 1 and ShardingTaskExecutorPoolMaxSize&amp;amp;ShardingTaskExecutorPoolMinSize = 20&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;For the first test, the machine runs with YCSB is 8Core(all the previous pressure test is also 8Core), and we see everythings works well ( threads: 48)&lt;/li&gt;
	&lt;li&gt;I found that the trend graph I provided earlier did not achieve better performance as the number of threads increased, so I upgrate the machine runs with YCSB to 16 Core for the other 3 test(threads : 48 / 64 / 96), and we got benifit from it&#160;&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Summary YCSB result with all the tests Using&#160;ShardingTaskExecutorPoolMaxSize&amp;amp;ShardingTaskExecutorPoolMinSize = 20 :&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;taskExecutorPoolSize = 4 with YCSB 8Core :&#160;Operations - 38324742 ,&#160;AverageLatency(us) - 734 ,&#160;99thPercentileLatency(us) - 1877&lt;/li&gt;
	&lt;li&gt;taskExecutorPoolSize = 1 with YCSB 8Core :&#160;Operations - 38200525 ,&#160;AverageLatency(us) - 736 ,&#160;99thPercentileLatency(us) - 1893&lt;/li&gt;
	&lt;li&gt;taskExecutorPoolSize = 1 with YCSB 16Core :&#160;Operations - 39668980 ,&#160;AverageLatency(us) - 709 ,&#160;99thPercentileLatency(us) - 1856&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;As we see , there is no much difference (may be a little benifit from 16Core).&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I will pay close attention to&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt; and look forward to the solution of the problem&lt;/p&gt;</comment>
                            <comment id="3575602" author="bruce.lucas@10gen.com" created="Thu, 21 Jan 2021 14:48:45 +0000"  >&lt;p&gt;No problem &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wangxin201492%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;wangxin201492@gmail.com&quot;&gt;wangxin201492@gmail.com&lt;/a&gt; - I think your test with pool sizes gives us the confidence we needed that this ticket is the same issue.&lt;/p&gt;

&lt;p&gt;Regarding &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;, it is currently under active investigation, and we&apos;ll update the ticket when we have more information.&lt;/p&gt;

&lt;p&gt;Regarding settings in your production environment, we believe that setting taskExecutorPoolSize to a value larger than 1 is not useful starting in 4.2, and you might get better performance by leaving it at 1. Also I wonder if that would make the workaround of setting min and max pool size more feasible in your environment, as you would have one single pool instead of multiple smaller pools that might not individually be optimally sized.&lt;/p&gt;</comment>
                            <comment id="3575482" author="wangxin201492@gmail.com" created="Thu, 21 Jan 2021 14:01:59 +0000"  >&lt;p&gt;Sorry, I have no environment to test it on 4.4&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I found that&#160;&lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;&#160; is under investigation, any idea to find the root cause of the problem or any further plan about that?&#160;&lt;/p&gt;

&lt;p&gt;set&#160;ShardingTaskExecutorPoolMaxSize&amp;amp;ShardingTaskExecutorPoolMinSize to&#160; same value works well for pressure test, but it may not be applied in the production environment~&#160;&lt;/p&gt;

&lt;p&gt;hoping for the perfect soluation on 4.2&lt;/p&gt;</comment>
                            <comment id="3573762" author="wangxin201492@gmail.com" created="Wed, 20 Jan 2021 16:22:59 +0000"  >&lt;p&gt;Thanks for your reply.&lt;/p&gt;

&lt;p&gt;That&apos;s helpful for this issue. I&#160;follow by your suggestion and adjust ShardingTaskExecutorPoolMaxSize and ShardingTaskExecutorPoolMinSize both to&#160; 20, the shard cluster runs smoothly!&lt;/p&gt;

&lt;p&gt;&#160; &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/296265/296265_image-2021-01-21-00-16-26-376.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The number of threads for these three tests as seen in the picture are 48 / 64 / 96 , there is no steep drop mentioned above.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;I&apos;ll reply more after try more about&#160;ShardingTaskExecutorPoolMaxSize/ShardingTaskExecutorPoolMinSize/taskExecutorPoolSize. And for testing it on 4.4 , there is some difficult for me, I&apos;ll try my best to do it.&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;</comment>
                            <comment id="3573537" author="bruce.lucas@10gen.com" created="Wed, 20 Jan 2021 15:21:29 +0000"  >&lt;p&gt;This may be related to &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt; as there are several common elements:&lt;/p&gt;

&lt;p&gt;Version is 4.2.10&lt;/p&gt;

&lt;p&gt;The test is doing secondary reads:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;mongodb.readPreference=secondary&lt;/p&gt;&lt;/blockquote&gt;

&lt;p&gt;There are slow queries on mongos, but not mongod, indicating a bottleneck in mongos.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/296227/296227_conns.png&quot; width=&quot;100%&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;A-B and C-D are two periods when queuing is occurring in mongos, indicated by &quot;connections active&quot;, meaning that mongos would be seeing slow queries. During those periods, we see likely unncessary connections being created to the secondaries: &quot;totalInUse&quot; is considerably larger than &quot;connections active&quot;, suggesting that mongos incorrectly thinks some connections are in use when the can&apos;t be.&lt;/p&gt;

&lt;p&gt;The perf CPU reported may be associated with establishing connections.&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=wangxin201492%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;wangxin201492@gmail.com&quot;&gt;wangxin201492@gmail.com&lt;/a&gt;, would you be able to try one or both of the following to test this hypothesis?&lt;/p&gt;

&lt;ul&gt;
	&lt;li&gt;Set ShardingTaskExecutorPoolMaxSize and ShardingTaskExecutorPoolMinSize both to the same value, perhaps 20 based on what I see in your test, although you may have to experiment with this a bit. We think that doing this mitigates &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul&gt;
	&lt;li&gt;Try the test on 4.4, as we don&apos;t think it is impacted by &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-53853&quot; title=&quot;Large buildup of mongos to mongod connections and low performance with secondaryPreferred reads&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-53853&quot;&gt;&lt;del&gt;SERVER-53853&lt;/del&gt;&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="3573165" author="wangxin201492@gmail.com" created="Wed, 20 Jan 2021 11:22:10 +0000"  >&lt;p&gt;the attachment &quot;mongos.diagnostic.data.tar.gz&quot; is one of mongos diagnostic log with newest pressure test between 2020-01-20 18:15(+8) and&#160;2020-01-20 18:25(+8), and some slow log in it (2021-01-20T18:22:07.532+0800)&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1592884">SERVER-53853</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="1622584">SERVER-54504</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="296227" name="conns.png" size="342832" author="bruce.lucas@mongodb.com" created="Wed, 20 Jan 2021 15:20:12 +0000"/>
                            <attachment id="296187" name="image-2021-01-20-17-44-34-407.png" size="138064" author="wangxin201492@gmail.com" created="Wed, 20 Jan 2021 09:43:59 +0000"/>
                            <attachment id="296265" name="image-2021-01-21-00-16-26-376.png" size="125238" author="wangxin201492@gmail.com" created="Wed, 20 Jan 2021 16:15:48 +0000"/>
                            <attachment id="296570" name="image-2021-01-22-12-50-18-538.png" size="119840" author="wangxin201492@gmail.com" created="Fri, 22 Jan 2021 04:49:37 +0000"/>
                            <attachment id="296195" name="mongos.diagnostic.data.tar.gz" size="4642528" author="wangxin201492@gmail.com" created="Wed, 20 Jan 2021 11:18:30 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>7.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 20 Jan 2021 15:21:29 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        3 years, 2 weeks, 5 days ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            3 years, 2 weeks, 5 days ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>bruce.lucas@mongodb.com</customfieldvalue>
            <customfieldvalue>wangxin201492@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hyqarj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hybzd3:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;p&gt;6 mongos with&#160;taskExecutorPoolSize=4 and 8Core&lt;/p&gt;

&lt;p&gt;3 shard with PSSSSH&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;use YCSB to pressure it:&lt;/p&gt;

&lt;p&gt;./bin/ycsb run mongodb -P workloads/custom -s -threads 48 -p mongodb.url=xxx&lt;/p&gt;

&lt;p&gt;&#160;&lt;/p&gt;

&lt;p&gt;YCSB workload:&lt;/p&gt;

&lt;p&gt;recordcount=100000&lt;br/&gt;
operationcount=100000000&lt;br/&gt;
workload=site.ycsb.workloads.CoreWorkload&lt;br/&gt;
readallfields=true&lt;br/&gt;
readproportion=0.7&lt;br/&gt;
updateproportion=0.1&lt;br/&gt;
insertproportion=0.2&lt;br/&gt;
scanproportion=0&lt;br/&gt;
maxscanlength=1000&lt;br/&gt;
readmodifywriteproportion=0&lt;br/&gt;
insertorder=order&lt;br/&gt;
writeallfields=false&lt;br/&gt;
mongodb.readPreference=secondary&lt;br/&gt;
mongodb.maxconnections=200&lt;br/&gt;
requestdistribution=uniform&lt;br/&gt;
fieldlength=30&lt;br/&gt;
fieldcount=10&lt;br/&gt;
hotspotdatafraction=0.2&lt;br/&gt;
hotspotopnfraction=0.8&lt;br/&gt;
maxexecutiontime=600&lt;br/&gt;
table=pressure&lt;/p&gt;</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[bruce.lucas@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hypx0v:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>