<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:07:31 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-24824] Mongo 3.0.12 with MMAPv1 can&apos;t serve more than 1k qps</title>
                <link>https://jira.mongodb.org/browse/SERVER-24824</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;Hello All,&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The setup&lt;/b&gt;&lt;br/&gt;
I am running a replica set in production without sharding. All nodes are running latest stable mongo 2.6 except one hidden node which has mongo 3.0 with MMAPv1.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The data&lt;/b&gt;&lt;br/&gt;
I have around 4TB worth of data on each node (MMAPv1), with close to 7000 databases.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The plan&lt;/b&gt;&lt;br/&gt;
I decided to upgrade to 3.2 and as an intermediate step, I have to upgrade to 3.0 first. Initially I had used wiredTiger with this node, but encountered a problem when I sent prod traffic to that node. Here&apos;s the full description on the JIRA issue - &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-24514&quot; title=&quot;Global IX Lock for more than 4 minutes on Mongo 3.0.11 with wiredTiger&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-24514&quot;&gt;&lt;del&gt;SERVER-24514&lt;/del&gt;&lt;/a&gt;. For not being blocked on the issue, I decided to go ahead with 3.0.12 with MMAPv1 instead of wiredTiger. So to start with that, I added the aforementioned hidden member to the existing replica set. I started sending prod like read query traffic to this node to check if it will be able to withstand that much load. I did this for over a week.&lt;/p&gt;

&lt;p&gt;The plan was to roll out 3.0 on all secondaries if latencies and rps are close to prod like pattern.&lt;/p&gt;

&lt;p&gt;&lt;b&gt;The observation&lt;/b&gt;&lt;br/&gt;
It was observed that the node couldn&apos;t entertain the read traffic at a consistent rate. It can be seen that the node entertains ~1k queries per seconds, shoots to ~3.5k qps for a very brief moment of time and drops back to ~1k qps. (Please check Operations Per Second graph). This particular pattern is not observed when same traffic is sent to 2.6.x nodes. Those nodes can entertain same traffic at ~4k qps consistently.&lt;br/&gt;
In the process of understanding what exactly is happening, I ran db.currentOp()  on that node. I didn&apos;t find anything in particular. But some queries took ~200ms to return. For those queries &quot;timeAcquiringMicros&quot; is ~198ms. According to the docs it is the &quot;cumulative time in microseconds that the operation had to wait to acquire the locks&quot;. I would appreciate any help here.&lt;/p&gt;</description>
                <environment></environment>
        <key id="297129">SERVER-24824</key>
            <summary>Mongo 3.0.12 with MMAPv1 can&apos;t serve more than 1k qps</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="3">Duplicate</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="abhishek.amberkar">Abhishek Amberkar</reporter>
                        <labels>
                    </labels>
                <created>Tue, 28 Jun 2016 06:40:37 +0000</created>
                <updated>Wed, 17 Aug 2016 05:47:48 +0000</updated>
                            <resolved>Wed, 17 Aug 2016 04:02:55 +0000</resolved>
                                    <version>3.0.12</version>
                                                    <component>MMAPv1</component>
                    <component>Performance</component>
                                        <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="1359226" author="abhishek.amberkar" created="Wed, 17 Aug 2016 05:47:48 +0000"  >&lt;p&gt;Thank you Thomas.&lt;/p&gt;</comment>
                            <comment id="1359200" author="thomas.schubert" created="Wed, 17 Aug 2016 04:02:14 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=abhishek.amberkar&quot; class=&quot;user-hover&quot; rel=&quot;abhishek.amberkar&quot;&gt;abhishek.amberkar&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you for uploading the server status and iostat logs.&lt;/p&gt;

&lt;p&gt;After reviewing the data you provided, I have identified that the issue you are encountering is tracked in &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23798&quot; title=&quot;Increased ns file IO in 3.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23798&quot;&gt;&lt;del&gt;SERVER-23798&lt;/del&gt;&lt;/a&gt;. In MongoDB 3.0, a change was introduced that changed how ns files are loaded into memory. Please watch &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-23798&quot; title=&quot;Increased ns file IO in 3.0&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-23798&quot;&gt;&lt;del&gt;SERVER-23798&lt;/del&gt;&lt;/a&gt; for updates regarding this behavior.&lt;/p&gt;

&lt;p&gt;Thanks again for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1340844" author="abhishek.amberkar" created="Fri, 29 Jul 2016 05:25:22 +0000"  >&lt;p&gt;Hello Thomas,&lt;/p&gt;

&lt;p&gt;Here&apos;s a thread on mongodb-dev mailing list my colleague had opened few days ago. &lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://groups.google.com/d/msg/mongodb-dev/k1fYIgDd-ck/2Mv6hZK_BAAJ&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://groups.google.com/d/msg/mongodb-dev/k1fYIgDd-ck/2Mv6hZK_BAAJ&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Please let me know if you find anything from the logs and information provided which is relevant to my issue.&lt;/p&gt;

&lt;p&gt;Thank you.&lt;/p&gt;</comment>
                            <comment id="1333868" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:35:49 +0000"  >&lt;p&gt;Hello Thomas,&lt;/p&gt;

&lt;p&gt;I have attached all the files as requested. I have removed &quot;locks&quot; and &quot;recordStatus&quot; info from &quot;2.6&quot; logs, since ss2.log was running in GBs. Let me know if you need to have a look at that output as well. &lt;/p&gt;

&lt;p&gt;Also please check the description of the ticket where I have mentioned how I am running this whole setup. Let me know if you need any more information.&lt;/p&gt;

</comment>
                            <comment id="1331364" author="thomas.schubert" created="Thu, 21 Jul 2016 15:28:04 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=abhishek.amberkar&quot; class=&quot;user-hover&quot; rel=&quot;abhishek.amberkar&quot;&gt;abhishek.amberkar&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for uploading the logs - we&apos;re continuing to investigate this issue. To help us, would you please provide the following information?&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;Please run the following script on the 3.0.12 node for an hour
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus())); sleep(1000*$delay)}&quot; &amp;gt;ss3.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x $delay &amp;gt;iostat3.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt; &lt;/li&gt;
	&lt;li&gt;Similarly, please run the following script on the 2.6.x node for an hour
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus())); sleep(1000*$delay)}&quot; &amp;gt;ss2.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x $delay &amp;gt;iostat2.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;&lt;/li&gt;
	&lt;li&gt;Please upload ss2.log, iostat2.log, ss3.log, and iostat3.log.&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;Additionally, please provide the output of the following commands:&lt;/p&gt;
&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({buildInfo: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;db.adminCommand({hostInfo: true})&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Finally, if you could provide additional details about your workload, or could provide a workload that reproduces this issue, it would greatly speed our investigation.&lt;/p&gt;

&lt;p&gt;Thank you for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1330964" author="abhishek.amberkar" created="Thu, 21 Jul 2016 07:37:17 +0000"  >&lt;p&gt;Hello Ram&#243;n,&lt;/p&gt;

&lt;p&gt;Can I get any help here?&lt;/p&gt;

&lt;p&gt;I tried throwing more memory to this node and it started performing as expected. I wonder why 3.0.12 needs more memory than 2.6.x while serving the same amount of traffic. Could you please shed some light on this behaviour? &lt;/p&gt;

&lt;p&gt;Thank you.&lt;/p&gt;</comment>
                            <comment id="1308639" author="abhishek.amberkar" created="Tue, 28 Jun 2016 14:11:00 +0000"  >&lt;p&gt;Hello Ram&#243;n,&lt;/p&gt;

&lt;p&gt;I have attached the log as well as the memory graph. Hardware specs are same as 2.6 secondaries.&lt;/p&gt;</comment>
                            <comment id="1308503" author="ramon.fernandez" created="Tue, 28 Jun 2016 11:35:38 +0000"  >&lt;p&gt;&lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=abhishek.amberkar&quot; class=&quot;user-hover&quot; rel=&quot;abhishek.amberkar&quot;&gt;abhishek.amberkar&lt;/a&gt;, can you please upload the logs for this 3.0.12 node during the time you&apos;re conducting your measurements? Also, what are the hardware specs for this node? Are they the same as for your 2.6 secondaries?&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10010">
                    <name>Duplicate</name>
                                            <outwardlinks description="duplicates">
                                        <issuelink>
            <issuekey id="280756">SERVER-23798</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                            <issuelinktype id="10012">
                    <name>Related</name>
                                            <outwardlinks description="related to">
                                        <issuelink>
            <issuekey id="292899">SERVER-24514</issuekey>
        </issuelink>
                            </outwardlinks>
                                                        </issuelinktype>
                    </issuelinks>
                <attachments>
                            <attachment id="132224" name="host_info2.txt" size="2370" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                            <attachment id="132225" name="host_info3.txt" size="2487" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                            <attachment id="132226" name="iostat2.log" size="3805506" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                            <attachment id="132227" name="iostat3.log" size="3654572" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                            <attachment id="128752" name="mongo44.log" size="7043" author="abhishek.amberkar" created="Tue, 28 Jun 2016 14:10:00 +0000"/>
                            <attachment id="128747" name="mongo_resident_mem.png" size="44933" author="abhishek.amberkar" created="Tue, 28 Jun 2016 13:49:05 +0000"/>
                            <attachment id="128694" name="network_activity.png" size="31266" author="abhishek.amberkar" created="Tue, 28 Jun 2016 06:42:15 +0000"/>
                            <attachment id="128696" name="ops_per_second.png" size="39121" author="abhishek.amberkar" created="Tue, 28 Jun 2016 06:42:15 +0000"/>
                            <attachment id="128695" name="pagefaults_per_sec.png" size="31679" author="abhishek.amberkar" created="Tue, 28 Jun 2016 06:42:15 +0000"/>
                            <attachment id="132228" name="ss2.log" size="2353349" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                            <attachment id="132229" name="ss3.log" size="20554615" author="abhishek.amberkar" created="Mon, 25 Jul 2016 08:30:28 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>8.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Tue, 28 Jun 2016 11:34:22 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 26 weeks ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>abhishek.amberkar</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 26 weeks ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>abhishek.amberkar</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrk40f:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hsmyuf:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsetrz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>