<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 06:35:53 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-77539] Mongos Performance Issue</title>
                <link>https://jira.mongodb.org/browse/SERVER-77539</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Recently, a production cluster(WT 5.0 Version) come across a serious performance decrease when performing pure read workload. Using YCSB benchmark, we reproduce the problem.&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;EnvironmentSetup&quot;&gt;&lt;/a&gt;Environment Setup&lt;/h1&gt;

&lt;p&gt;Linux Kernel Version is 5.4.119.&lt;/p&gt;

&lt;p&gt;A sharded cluster(WT 5.0 Version) with three 8-cores 16-GB Mongos and five shards each with 8-cores 16-GB Mongod, a 1-core 2-GB ConfigSvr Replica. Using YCSB workload, we set {{&lt;/p&gt;
{field0: 1}
&lt;p&gt;}} as shard key, and perform point query based on &lt;tt&gt;_id&lt;/tt&gt; (&lt;b&gt;We query a key which is not shard key&lt;/b&gt;). According to our debugging and testing, we found some fun facts: &lt;b&gt;The performance dropping is caused by taskExecutorPoolSize configuration&lt;/b&gt;. Based on our experience, this configuration should be set to the number of cpu cores, which is surprised to harm the performance.&lt;br/&gt;
&#160;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&#160;&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&lt;b&gt;point search shard key&lt;/b&gt;&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&lt;b&gt;point search non-shard key&lt;/b&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;taskExecutorPoolSize: 1&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;5836.91 QPS&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;2770.74 QPS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;taskExecutorPoolSize:8&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;5279.16 QPS&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;1508.33 QPS&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;&#160;&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;FlameGraphAnalysis&quot;&gt;&lt;/a&gt;Flame Graph Analysis&lt;/h1&gt;

&lt;p&gt;We further record the flame graph, here is the result:&lt;/p&gt;

&lt;p&gt;When setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 8, there seem existing heavy lock contention&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452604/452604_image-2023-05-29-09-57-56-486.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;nearly every function call will invoke &lt;tt&gt;native_queued_spin_lock_slowpath&lt;/tt&gt; which wastes too much cpu to do useless works.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452603/452603_image-2023-05-29-09-59-05-893.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;When setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 1, things getting better&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452602/452602_image-2023-05-29-09-59-57-826.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452601/452601_image-2023-05-29-10-00-40-942.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;The lock content drop significantly, and SYSBENCH QPS improves from 1508.33 to 5279.16, but this is still far from our expectation, since WT 4.0 can achieve 13000+ QPS for this workload. So we further found this code block&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452600/452600_image-2023-05-29-10-01-32-972.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452599/452599_image-2023-05-29-10-01-44-996.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;WT 5.0 set &lt;tt&gt;use-diagnostic-latches&lt;/tt&gt; default to &lt;tt&gt;on&lt;/tt&gt; which will use &lt;tt&gt;detail_latch::Mutex&lt;/tt&gt;, but WT 4.0 use linux raw mutex. So we change this option to &lt;tt&gt;off&lt;/tt&gt; for further testing, and find that greatly improves the performance too.&lt;br/&gt;
&#160;&lt;/p&gt;
&lt;div class=&apos;table-wrap&apos;&gt;
&lt;table class=&apos;confluenceTable&apos;&gt;&lt;tbody&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&#160;&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&lt;b&gt;point search on shard key&lt;/b&gt; &lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;&lt;b&gt;point search on non-shard key&lt;/b&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;taskExecutorPoolSize: 1&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;24536.23 QPS&lt;/th&gt;
&lt;th class=&apos;confluenceTh&apos;&gt;10790.01 QPS&lt;/th&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;taskExecutorPoolSize:8&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;22579.78 QPS&lt;/td&gt;
&lt;td class=&apos;confluenceTd&apos;&gt;8044.38 QPS&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;&lt;/table&gt;
&lt;/div&gt;

&lt;p&gt;&#160;&lt;br/&gt;
We further analyze the flame graph when setting &lt;tt&gt;use-diagnostic-latches&lt;/tt&gt; to &lt;tt&gt;off&lt;/tt&gt;&lt;/p&gt;

&lt;p&gt;when setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 1, there are nearly no lock contention exists, and performance is the greatest. &lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452598/452598_image-2023-05-29-10-02-22-083.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;

&lt;p&gt;when setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 8, performance also increase but lower than above.&lt;/p&gt;

&lt;p&gt;&lt;span class=&quot;image-wrap&quot; style=&quot;&quot;&gt;&lt;img src=&quot;https://jira.mongodb.org/secure/attachment/452597/452597_image-2023-05-29-10-03-02-804.png&quot; style=&quot;border: 0px solid black&quot; /&gt;&lt;/span&gt;&lt;/p&gt;
&lt;h1&gt;&lt;a name=&quot;Conclusion&quot;&gt;&lt;/a&gt;Conclusion&lt;/h1&gt;

&lt;p&gt;Based on production cluster analysis and YCSB testing, we find two fact:&lt;/p&gt;
&lt;ol&gt;
	&lt;li&gt;Although setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to the number of cores is advised prior to WT 4.2, Setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 1 will gain the best performance in WT 5.0.&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;Using Linux default mutex gain the best performance, &lt;tt&gt;latch_detail::Mutex&lt;/tt&gt; wrapper class harms performance greatly.&lt;/li&gt;
&lt;/ol&gt;


&lt;h1&gt;&lt;a name=&quot;Question&quot;&gt;&lt;/a&gt;Question&lt;/h1&gt;
&lt;ol&gt;
	&lt;li&gt;According to this jira, is it recommanded to set &lt;tt&gt;taskExecutorPool&lt;/tt&gt; to 1 to gain the best performance under most circumstances?&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;After WT 4.0, why add &lt;tt&gt;use-diagnostic-latches&lt;/tt&gt; option to introduce a mutex wrapper which seems increasing lock contention and harming the performance? Should we leave it off in production environment to gain better performance?&lt;/li&gt;
&lt;/ol&gt;


&lt;ol&gt;
	&lt;li&gt;Can explain why setting &lt;tt&gt;taskExecutorPool&lt;/tt&gt; greater than 1 will cause such a difference?&lt;/li&gt;
&lt;/ol&gt;
</description>
                <environment></environment>
        <key id="2352548">SERVER-77539</key>
            <summary>Mongos Performance Issue</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="jim.oleary@mongodb.com">James O&apos;Leary</assignee>
                                    <reporter username="zhangwenjumlovercl@gmail.com">jum zhang</reporter>
                        <labels>
                            <label>mongos</label>
                            <label>perfomance</label>
                    </labels>
                <created>Mon, 29 May 2023 02:08:23 +0000</created>
                <updated>Thu, 19 Oct 2023 22:47:45 +0000</updated>
                            <resolved>Wed, 12 Jul 2023 09:32:21 +0000</resolved>
                                    <version>5.0.12</version>
                                                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="5559645" author="jim.oleary@10gen.com" created="Wed, 12 Jul 2023 09:20:24 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=zhangwenjumlovercl%40gmail.com&quot; class=&quot;user-hover&quot; rel=&quot;zhangwenjumlovercl@gmail.com&quot;&gt;zhangwenjumlovercl@gmail.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;It looks like you have independently discovered &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54504&quot; title=&quot;Disable taskExecutorPoolSize for Linux&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54504&quot;&gt;&lt;del&gt;SERVER-54504&lt;/del&gt;&lt;/a&gt; &apos;Disable taskExecutorPoolSize for Linux&apos;:&lt;/p&gt;
&lt;blockquote&gt;&lt;p&gt;This option is not useful for linux machines running 4.2 and beyond (and can cause problems if changed past the default of 1). So, we must disable this option for linux machines (It is always a value of 1).&lt;/p&gt;

&lt;p&gt;This parameter will still be available for Windows &amp;amp; OS X since we have found that it is useful for those machines, and that there are a non-trivial number of clusters running Windows in production.&lt;/p&gt;&lt;/blockquote&gt;
&lt;p&gt;As a result, I am going to mark this ticket as a duplicate of &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-54504&quot; title=&quot;Disable taskExecutorPoolSize for Linux&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-54504&quot;&gt;&lt;del&gt;SERVER-54504&lt;/del&gt;&lt;/a&gt; and have our documentation updated to deprecate this parameter on linux (see &lt;a href=&quot;https://jira.mongodb.org/browse/DOCS-16260&quot; title=&quot;Update taskExecutorPoolSize Documentation for older MongoDB versions &quot; class=&quot;issue-link&quot; data-issue-key=&quot;DOCS-16260&quot;&gt;&lt;del&gt;DOCS-16260&lt;/del&gt;&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
-Jim&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="1622584">SERVER-54504</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="2137909">DOCS-15632</issuekey>
        </issuelink>
                            </outwardlinks>
                                                                <inwardlinks description="is related to">
                                        <issuelink>
            <issuekey id="2389464">DOCS-16260</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="452604" name="image-2023-05-29-09-57-56-486.png" size="1864037" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 01:58:01 +0000"/>
                            <attachment id="452603" name="image-2023-05-29-09-59-05-893.png" size="713596" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 01:59:09 +0000"/>
                            <attachment id="452602" name="image-2023-05-29-09-59-57-826.png" size="2066914" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:00:03 +0000"/>
                            <attachment id="452601" name="image-2023-05-29-10-00-40-942.png" size="518854" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:00:43 +0000"/>
                            <attachment id="452600" name="image-2023-05-29-10-01-32-972.png" size="231327" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:01:35 +0000"/>
                            <attachment id="452599" name="image-2023-05-29-10-01-44-996.png" size="113901" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:01:46 +0000"/>
                            <attachment id="452598" name="image-2023-05-29-10-02-22-083.png" size="1159623" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:02:27 +0000"/>
                            <attachment id="452597" name="image-2023-05-29-10-03-02-804.png" size="1178503" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:03:06 +0000"/>
                            <attachment id="452595" name="lock_off_shard_key_not_same_task_executor_1.svg" size="9490332" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                            <attachment id="452596" name="lock_off_shard_key_not_same_task_executor_8.svg" size="9563733" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                            <attachment id="452592" name="lock_on_shard_key_not_same_task_executor_1.svg" size="9991308" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                            <attachment id="452591" name="lock_on_shard_key_not_same_task_executor_8.svg" size="11253620" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                            <attachment id="452593" name="lock_on_shard_key_same_task_executor_1.svg" size="14247501" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                            <attachment id="452594" name="lock_on_shard_key_same_task_executor_8.svg" size="14377008" author="zhangwenjumlovercl@gmail.com" created="Mon, 29 May 2023 02:08:14 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                <customfield id="customfield_12751" key="com.atlassian.jira.plugin.system.customfieldtypes:multiselect">
                        <customfieldname>Assigned Teams</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="25124"><![CDATA[Product Performance]]></customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_13552" key="com.go2group.jira.plugin.crm:crm_generic_field">
                        <customfieldname>Case</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[[5006R00001tJQpiQAG]]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Wed, 31 May 2023 15:21:49 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        30 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_22159" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Last Comment from Assignee</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[<p>Standup:</p>
<ul>
	<li>While trying to repro, I was having difficulty starting mongos processes on sys-perf with this setting.</li>
	<li>I found <a href="https://jira.mongodb.org/browse/SERVER-54504" title="Disable taskExecutorPoolSize for Linux" class="issue-link" data-issue-key="SERVER-54504"><del>SERVER-54504</del></a> which indicates that this parameter is not useful on linux for versions &gt; 4.2 and has been disabled for version 7.1 and greater.</li>
	<li>next steps: raise a docs ticket to have the online documentation updated to reflect the current recommendations.</li>
</ul>

]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            30 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>jim.oleary@mongodb.com</customfieldvalue>
            <customfieldvalue>zhangwenjumlovercl@gmail.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2b77b:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|i1tfzc:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10750" key="com.atlassian.jira.plugin.system.customfieldtypes:textarea">
                        <customfieldname>Steps To Reproduce</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>&lt;ol&gt;
	&lt;li&gt;run ycsb workload in default configuaration&lt;/li&gt;
	&lt;li&gt;setting &lt;tt&gt;taskExecutorPoolSize&lt;/tt&gt; to 8 will cause a greate performance decrease&lt;/li&gt;
	&lt;li&gt;Compile with &lt;tt&gt;use-diagnostic-latches&lt;/tt&gt; to off will cause a performance increate&lt;/li&gt;
&lt;/ol&gt;
</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                    <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|i2atcn:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>