<!-- 
RSS generated by JIRA (9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66) at Thu Feb 08 04:10:55 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>MongoDB Jira</title>
    <link>https://jira.mongodb.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>9.7.1</version>
        <build-number>970001</build-number>
        <build-date>13-04-2023</build-date>
    </build-info>


<item>
            <title>[SERVER-26021] Primary mongod instance stops serving traffic every few (5) hours</title>
                <link>https://jira.mongodb.org/browse/SERVER-26021</link>
                <project id="10000" key="SERVER">Core Server</project>
                    <description>&lt;p&gt;A running primary node&apos; throughput degrades over time. Switching primary&amp;lt;-&amp;gt;secondary solved the problem. As a precaution, we proactively shutdown the old primary once its secondary so that its ready for the next switch. MMS graph showing the degradation attached. This happens to primary nodes running WiredTiger. MMAP is not affected. We ran into this when we re-synced one of the secondaries nodes with storage engine set to WT and promoted it as primary&lt;/p&gt;</description>
                <environment></environment>
        <key id="314861">SERVER-26021</key>
            <summary>Primary mongod instance stops serving traffic every few (5) hours</summary>
                <type id="1" iconUrl="https://jira.mongodb.org/secure/viewavatar?size=xsmall&amp;avatarId=14703&amp;avatarType=issuetype">Bug</type>
                                            <priority id="3" iconUrl="https://jira.mongodb.org/images/icons/priorities/major.svg">Major - P3</priority>
                        <status id="6" iconUrl="https://jira.mongodb.org/images/icons/statuses/closed.png" description="The issue is considered finished, the resolution is correct. Issues which are closed can be reopened.">Closed</status>
                    <statusCategory id="3" key="done" colorName="success"/>
                                    <resolution id="4">Incomplete</resolution>
                                        <assignee username="kelsey.schubert@mongodb.com">Kelsey Schubert</assignee>
                                    <reporter username="francis@wizrocket.com">Francis Pereira</reporter>
                        <labels>
                    </labels>
                <created>Thu, 8 Sep 2016 08:48:42 +0000</created>
                <updated>Fri, 7 Apr 2023 14:15:58 +0000</updated>
                            <resolved>Mon, 13 Feb 2017 18:51:12 +0000</resolved>
                                    <version>3.0.12</version>
                                                    <component>Performance</component>
                    <component>WiredTiger</component>
                                        <votes>1</votes>
                                    <watches>13</watches>
                                                                                                                <comments>
                            <comment id="1489107" author="thomas.schubert" created="Tue, 31 Jan 2017 16:40:22 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;We haven&#8217;t heard back from you for some time, so I&#8217;m going to mark this ticket as resolved. Unfortunately, without additional information, we cannot continue to investigate this behavior. If this is still an issue for you, please provide iostats and diagnostic.data and we will reopen the ticket.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1415026" author="ramon.fernandez" created="Sat, 22 Oct 2016 13:17:02 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francis%40wizrocket.com&quot; class=&quot;user-hover&quot; rel=&quot;francis@wizrocket.com&quot;&gt;francis@wizrocket.com&lt;/a&gt;, have you had a chance to collect &lt;tt&gt;iostat&lt;/tt&gt; data as requested by Thomas to determine if the issue is indeed hardware limitations or something else?&lt;/p&gt;

&lt;p&gt;Please let us know.&lt;/p&gt;

&lt;p&gt;Thanks,&lt;br/&gt;
Ram&#243;n.&lt;/p&gt;</comment>
                            <comment id="1402393" author="thomas.schubert" created="Thu, 6 Oct 2016 20:33:29 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks again for providing the &lt;tt&gt;diagnostic.data&lt;/tt&gt;. After reviewing the data, it appears that this node is running into hardware limitations. The read/write load is stable and similar in both sets of diagnostic data. In addition, there is an average of 25 active filesystem read calls which suggests that the workload is I/O bound.&lt;/p&gt;

&lt;p&gt;To help us continue to investigate this issue, we will need to remove this hardware constraint. Here are two options&lt;/p&gt;
&lt;ul&gt;
	&lt;li&gt;Re-run this workload on a machine with more provisioned IOPs&lt;/li&gt;
	&lt;li&gt;Alternatively, can you share the code and data required to reproduce this issue, and we can run it on our machines?&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;If you choose to execute the workload yourself on a larger machine, during your repro please execute the following shell script to collect iostat data each second and upload it along with the &lt;tt&gt;diagnostic.data&lt;/tt&gt;&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x ${delay:?} &amp;gt;iostat.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;This will help us to correlate the I/O numbers to events recorded in the diagnostic.data. &lt;/p&gt;

&lt;p&gt;Thank you for your help,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1401506" author="thomas.schubert" created="Wed, 5 Oct 2016 22:44:56 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thank you very much for uploading the &lt;tt&gt;diagnostic.data&lt;/tt&gt; for both MongoDB 3.2.9 and 3.2.10. We are investigating this issue and will update you when we know more.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1400548" author="francis@wizrocket.com" created="Wed, 5 Oct 2016 06:47:37 +0000"  >&lt;p&gt;Diagnostics data from 3.2.9 and 3.2.10&lt;/p&gt;</comment>
                            <comment id="1398041" author="francis@wizrocket.com" created="Fri, 30 Sep 2016 19:49:10 +0000"  >&lt;p&gt;Thanks for the update @Thomas.Schubert. We have a way to reproduce this issue in an isolated environment. I will run our workload against  3.2.10-rc2  and submit diagnostics data. Expect data by Monday. &lt;/p&gt;</comment>
                            <comment id="1397987" author="thomas.schubert" created="Fri, 30 Sep 2016 19:03:04 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&apos;ve taken another look at your cloud group, and see that you have a large number of small databases, collections, and indexes. This schema design benefits MMAPv1 as DB level locks can be mitigated. However, the large number of files may hinder performance during WiredTiger checkpoints. As I mentioned previously, if you could provide the &lt;tt&gt;diagnostic.data&lt;/tt&gt;, we would be able to better determine what is going on here.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1396877" author="thomas.schubert" created="Thu, 29 Sep 2016 18:30:12 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francis%40wizrocket.com&quot; class=&quot;user-hover&quot; rel=&quot;francis@wizrocket.com&quot;&gt;francis@wizrocket.com&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;I&#8217;m sorry that you have encountered this issue, and that it has caused such trouble for your team. I see that you recently published a blog post that provides additional context around the issue that you are observing. &lt;/p&gt;

&lt;p&gt;As you have surmised, the problem is caused by WiredTiger&#8217;s behavior when its cache exceeds 95% utilization. A significant amount of work has gone into correcting this issue in MongoDB 3.2.10, which is in testing now and it&#8217;s scheduled for release in the coming days.&lt;br/&gt;
Please review &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26055?focusedCommentId=1394968&amp;amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-1394968&quot; class=&quot;external-link&quot; rel=&quot;nofollow&quot;&gt;this comment&lt;/a&gt; on &lt;a href=&quot;https://jira.mongodb.org/browse/SERVER-26055&quot; title=&quot;Server fills cache rapidly, even under replication only and then operates very slowing causing oplog backlog. When serving clients queries take ages to return once cache is &amp;#39;full&amp;#39;&quot; class=&quot;issue-link&quot; data-issue-key=&quot;SERVER-26055&quot;&gt;&lt;del&gt;SERVER-26055&lt;/del&gt;&lt;/a&gt; for technical details about the improvements included in MongoDB 3.2.10. I would recommend that you consider upgrading to MongoDB 3.2.10 to take advantage of these fixes when it is released. A MongoDB 3.2.10-rc2 release candidate is readily available for download if you&#8217;d like to start your testing right away.&lt;/p&gt;

&lt;p&gt;If you have diagnostic.data from the 3.2.9 node, I would be happy to examine it to confirm this diagnosis.&lt;/p&gt;

&lt;p&gt;Finally, I would like to thank you for sharing your experiences and recommending stronger safety practices to the community when performing major upgrades to MongoDB.&lt;/p&gt;

&lt;p&gt;Kind regards,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1382158" author="thomas.schubert" created="Mon, 12 Sep 2016 16:30:28 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for the additional information. Would you please run the following script until the issue occurs so we can get a better idea of what is going on here?&lt;/p&gt;

&lt;p/&gt;
&lt;div id=&quot;syntaxplugin&quot; class=&quot;syntaxplugin&quot; style=&quot;border: 1px dashed #bbb; border-radius: 5px !important; overflow: auto; max-height: 30em;&quot;&gt;
&lt;table cellspacing=&quot;0&quot; cellpadding=&quot;0&quot; border=&quot;0&quot; width=&quot;100%&quot; style=&quot;font-size: 1em; line-height: 1.4em !important; font-weight: normal; font-style: normal; color: black;&quot;&gt;
		&lt;tbody &gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;  margin-top: 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;delay=1&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;mongo --eval &quot;while(true) {print(JSON.stringify(db.serverStatus({tcmalloc:true}))); sleep(1000*${delay:?})}&quot; &amp;gt;ss.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
				&lt;tr id=&quot;syntaxplugin_code_and_gutter&quot;&gt;
						&lt;td  style=&quot; line-height: 1.4em !important; padding: 0em; vertical-align: top;&quot;&gt;
					&lt;pre style=&quot;font-size: 1em; margin: 0 10px;   margin-bottom: 10px;  width: auto; padding: 0;&quot;&gt;&lt;span style=&quot;color: black; font-family: &apos;Consolas&apos;, &apos;Bitstream Vera Sans Mono&apos;, &apos;Courier New&apos;, Courier, monospace !important;&quot;&gt;iostat -k -t -x ${delay:?} &amp;gt;iostat.log &amp;amp;&lt;/span&gt;&lt;/pre&gt;
			&lt;/td&gt;
		&lt;/tr&gt;
			&lt;/tbody&gt;
&lt;/table&gt;
&lt;/div&gt;
&lt;p/&gt;

&lt;p&gt;Afterwards please attach both the &lt;tt&gt;ss.log&lt;/tt&gt; and the &lt;tt&gt;iostats.log&lt;/tt&gt; to this ticket.&lt;/p&gt;

&lt;p&gt;Thank you,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1380315" author="francis@wizrocket.com" created="Fri, 9 Sep 2016 08:33:04 +0000"  >&lt;p&gt;I just figured out that a shutdown is not required as described earlier. When throughput degrades, switching P&amp;lt;-&amp;gt;S brings things back to normal. &lt;/p&gt;</comment>
                            <comment id="1379831" author="francis@wizrocket.com" created="Thu, 8 Sep 2016 18:20:18 +0000"  >&lt;p&gt;Hi Thomas, I have uploaded the log files from members of a replica set that showed this problem. I don&apos;t have diagnostic.data since I am running 3.0.12. &lt;/p&gt;</comment>
                            <comment id="1379568" author="thomas.schubert" created="Thu, 8 Sep 2016 15:20:17 +0000"  >&lt;p&gt;Hi &lt;a href=&quot;https://jira.mongodb.org/secure/ViewProfile.jspa?name=francispereira&quot; class=&quot;user-hover&quot; rel=&quot;francispereira&quot;&gt;francispereira&lt;/a&gt;,&lt;/p&gt;

&lt;p&gt;Thanks for reporting this issue. Would you please provide the following information so we can continue to investigate?&lt;/p&gt;

&lt;ol&gt;
	&lt;li&gt;the complete logs of the affected mongod&lt;/li&gt;
	&lt;li&gt;an archive (tar or zip) of the &lt;tt&gt;diagnostic.data&lt;/tt&gt; directory&lt;/li&gt;
&lt;/ol&gt;


&lt;p&gt;I&apos;ve created a secure upload portal for you to use &lt;a href=&quot;https://10gen-httpsupload.s3.amazonaws.com/upload_forms/132bdb06-5765-4c8e-9a2b-1bd0443be3bf.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;here&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Thanks again,&lt;br/&gt;
Thomas&lt;/p&gt;</comment>
                            <comment id="1379310" author="francis@wizrocket.com" created="Thu, 8 Sep 2016 08:59:49 +0000"  >&lt;p&gt;I can provide debugging info when this happens by  promoting the WT node as primary and letting it run for a few hours. Let me know what you need&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="140696" name="MongoDB-3.2.10-diagnostic.png" size="135579" author="kelsey.schubert@mongodb.com" created="Wed, 5 Oct 2016 22:38:47 +0000"/>
                            <attachment id="140697" name="MongoDB-3.2.9-diagnostic.png" size="138719" author="kelsey.schubert@mongodb.com" created="Wed, 5 Oct 2016 22:38:47 +0000"/>
                            <attachment id="137903" name="Screen Shot 2016-09-08 at 2.13.08 PM.png" size="763169" author="francispereira" created="Thu, 8 Sep 2016 08:50:41 +0000"/>
                            <attachment id="137945" name="Screen Shot 2016-09-09 at 12.00.12 AM.png" size="397005" author="francispereira" created="Thu, 8 Sep 2016 18:32:17 +0000"/>
                            <attachment id="140579" name="diagnostic.data.3.2.10.io1.tar.gz" size="15326993" author="francispereira" created="Wed, 5 Oct 2016 06:47:37 +0000"/>
                            <attachment id="140578" name="diagnostic.data.3.2.9.io1.tar.gz" size="13103645" author="francispereira" created="Wed, 5 Oct 2016 06:47:37 +0000"/>
                            <attachment id="140698" name="s26021_3210.png" size="71496" author="alexander.gorrod@mongodb.com" created="Wed, 5 Oct 2016 23:44:44 +0000"/>
                            <attachment id="140699" name="s26021_329.png" size="77593" author="alexander.gorrod@mongodb.com" created="Wed, 5 Oct 2016 23:44:44 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                <customfield id="customfield_10050" key="com.atlassian.jira.toolkit:comments">
                        <customfieldname># Replies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>13.0</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10055" key="com.atlassian.jira.ext.charting:firstresponsedate">
                        <customfieldname>Date of 1st Reply</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>Thu, 8 Sep 2016 15:20:17 +0000</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10052" key="com.atlassian.jira.toolkit:dayslastcommented">
                        <customfieldname>Days since reply</customfieldname>
                        <customfieldvalues>
                                        7 years, 2 weeks, 1 day ago
    
                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_18254" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Dependencies</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[]]></customfieldvalue>


                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_15850" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_10057" key="com.atlassian.jira.toolkit:lastusercommented">
                        <customfieldname>Last comment by Customer</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>true</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_10056" key="com.atlassian.jira.toolkit:lastupdaterorcommenter">
                        <customfieldname>Last commenter</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>luke.bonanomi@mongodb.com</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_11151" key="com.atlassian.jira.toolkit:LastCommentDate">
                        <customfieldname>Last public comment date</customfieldname>
                        <customfieldvalues>
                            7 years, 2 weeks, 1 day ago
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10032" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>Operating System</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10026"><![CDATA[ALL]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_10051" key="com.atlassian.jira.toolkit:participants">
                        <customfieldname>Participants</customfieldname>
                        <customfieldvalues>
                                        <customfieldvalue>francis@wizrocket.com</customfieldvalue>
            <customfieldvalue>kelsey.schubert@mongodb.com</customfieldvalue>
            <customfieldvalue>ramon.fernandez@mongodb.com</customfieldvalue>
    
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                        <customfield id="customfield_14254" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Product Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hrjwkv:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                <customfield id="customfield_12550" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2|hspjlj:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10558" key="com.pyxis.greenhopper.jira:gh-global-rank">
                        <customfieldname>Rank (Obsolete)</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>9223372036854775807</customfieldvalue>
                        </customfieldvalues>
                    </customfield>
                                                                                            <customfield id="customfield_23361" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Requested By</customfieldname>
                        <customfieldvalues>
                                

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            <customfield id="customfield_10053" key="com.atlassian.jira.ext.charting:timeinstatus">
                        <customfieldname>Time In Status</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                                                                        <customfield id="customfield_22870" key="com.onresolve.jira.groovy.groovyrunner:scripted-field">
                        <customfieldname>Triagers</customfieldname>
                        <customfieldvalues>
                                    <customfieldvalue><![CDATA[kelsey.schubert@mongodb.com]]></customfieldvalue>
    

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                    <customfield id="customfield_14350" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>serverRank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1|hsekbz:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                    </customfields>
    </item>
</channel>
</rss>